Reading Data: Text in PySpark: A Comprehensive Guide

What is Reading Text Files in PySpark?

Reading text files in PySpark means using the spark.read.text() method to load plain text files into a DataFrame, converting each line of text into a single-column structure within Spark’s distributed environment. You call this method on a SparkSession object—your entry point to Spark’s SQL capabilities—and provide a path to a text file, a directory of files, or a distributed source like HDFS or AWS S3. Spark’s architecture then takes over, distributing the file across its cluster, reading each line as a row in a DataFrame with a default column named value, and leveraging the Catalyst optimizer to prepare it for DataFrame operations like filter or custom parsing, or SQL queries via temporary views.

This functionality builds on Spark’s evolution from the legacy SQLContext to the unified SparkSession in Spark 2.0, offering a simple yet powerful way to handle unstructured text data. Text files—plain files with lines of characters, often logs, raw dumps, or custom formats—might come from system outputs, ETL pipelines, or manual exports, and spark.read.text() ingests them as-is, leaving further structuring to you via UDFs or string operations. Whether you’re processing a small file in Jupyter Notebooks or terabytes from Databricks DBFS, it scales effortlessly, making it a go-to for raw text ingestion.

Here’s a quick example to see it in action:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TextExample").getOrCreate()
df = spark.read.text("path/to/example.txt")
df.show(truncate=False)
# Assuming example.txt contains:
# Alice,25
# Bob,30
# Output:
# +-------+
# |value  |
# +-------+
# |Alice,25|
# |Bob,30  |
# +-------+
spark.stop()

In this snippet, we load a text file, and Spark reads each line into a DataFrame column named value, ready for parsing or analysis—a basic yet versatile starting point.

Parameters of spark.read.text()

The spark.read.text() method comes with a modest set of parameters, reflecting its focus on simplicity, but each one offers control over how Spark handles text files. Let’s break them down in detail, exploring their roles and impacts on the loading process.

path

The path parameter is the only required element—it tells Spark where your text file or files are located. You can pass a string pointing to a single file, like "data.txt", a directory like "data/" to load all text files inside, or a glob pattern like "data/*.txt" to target specific files. It’s flexible enough to work with local paths, HDFS, S3, or other file systems supported by Spark, depending on your SparkConf. Spark distributes the reading task across its cluster, efficiently processing one file or many in parallel.

wholeText

The wholeText parameter, when set to True (default is False), tells Spark to read each file as a single row in the DataFrame, with the entire file content in the value column and an additional path column for the file’s location. By default (False), Spark reads each line as a separate row, which suits line-delimited data like logs. Switching to True is ideal for small files or when you need the full text as one unit—say, for parsing a single document—but it’s less scalable for large files since it loads everything into memory per file.

lineSep

The lineSep parameter defines the line separator Spark uses to split the file into rows—defaulting to \n, \r\n, or \r based on the system. You can set it to a custom string, like "\t" or "|" (up to 128 characters), if your text uses a unique delimiter. It’s a subtle tweak—most files stick to standard newlines—but it’s essential for non-standard text where lines aren’t split by typical end-of-line markers, ensuring Spark aligns rows correctly.

encoding

The encoding parameter specifies the file’s character encoding—like UTF-8 (default), ISO-8859-1, or latin1—ensuring Spark reads text accurately, especially for non-English characters or special symbols. A mismatch can garble data, so it’s critical for diverse or legacy text files, aligning the read with the file’s actual encoding.

pathGlobFilter

The pathGlobFilter parameter isn’t explicitly listed for text() but can be passed via .option() as a glob pattern (e.g., "*.txt") to filter files within a directory. It refines the path parameter, ensuring only matching files are read—handy in mixed directories with non-text files.

Here’s an example using key parameters:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TextParams").getOrCreate()
df = spark.read.text("path/to/text_folder", wholeText=True, lineSep="|", encoding="UTF-8")
df.show(truncate=False)
# Assuming text_folder/example.txt contains:
# Alice|25|Engineer
# Output (with wholeText=True):
# +------------------+--------------------+
# |path              |value               |
# +------------------+--------------------+
# |path/to/example.txt|Alice|25|Engineer|
# +------------------+--------------------+
spark.stop()

This loads a folder of text files as whole files, split by | instead of newlines, using UTF-8 encoding, showing how parameters tailor the read.

Key Features When Reading Text Files

Beyond parameters, spark.read.text() offers features that enhance its simplicity and utility. Let’s explore these, with examples to highlight their value.

Spark reads text as-is, placing each line (or file with wholeText=True) into a single value column, leaving parsing to you—ideal for unstructured data like logs or free text. This raw approach contrasts with structured formats like Parquet, giving you flexibility to apply UDFs or string splits.

from pyspark.sql import SparkSession
from pyspark.sql.functions import split

spark = SparkSession.builder.appName("ParseText").getOrCreate()
df = spark.read.text("data.txt")
df_split = df.select(split("value", ",").alias("fields"))
df_split.show()
# Assuming data.txt: Alice,25
# Output:
# +----------+
# |    fields|
# +----------+
# |[Alice,25]|
# +----------+
spark.stop()

It also handles multiple files seamlessly—point path to a directory or glob pattern, and Spark combines them into one DataFrame, scaling for ETL pipelines or batch jobs with the Catalyst optimizer managing the load.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiFile").getOrCreate()
df = spark.read.text("text_folder/*.txt")
df.show()
spark.stop()

Compression support—.gz, .bz2—lets Spark read compressed text files natively, reducing storage and transfer costs without extra steps, perfect for logs or real-time analytics.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Compressed").getOrCreate()
df = spark.read.text("data.txt.gz")
df.show()
spark.stop()

Common Use Cases of Reading Text Files

Reading text files in PySpark fits into a variety of practical scenarios, leveraging its simplicity for raw data tasks. Let’s dive into where it shines with detailed examples.

Processing log files—like server or application logs—is a classic use. You read a directory of logs, parse with UDFs or string functions, and analyze with aggregate functions, turning raw text into insights for log processing.

from pyspark.sql import SparkSession
from pyspark.sql.functions import split, col

spark = SparkSession.builder.appName("LogRead").getOrCreate()
df = spark.read.text("logs/*.txt")
df_parsed = df.select(
    split("value", " ")[0].alias("timestamp"),
    split("value", " ")[1].alias("event")
)
df_parsed.groupBy("event").count().show()
# Assuming logs/event.txt: 2023-01-01 click
# Output:
# +-----+-----+
# |event|count|
# +-----+-----+
# |click|    1|
# +-----+-----+
spark.stop()

Ingesting unstructured data—like text dumps or documents—uses wholeText=True to read files as single rows, feeding ETL pipelines for cleaning or tokenization. A set of reports becomes a DataFrame for further structuring.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Unstructured").getOrCreate()
df = spark.read.text("reports/", wholeText=True)
df.show(truncate=False)
# Assuming reports/report.txt: Report content...
# Output:
# +----------------+---------------+
# |path            |value          |
# +----------------+---------------+
# |reports/report.txt|Report content...|
# +----------------+---------------+
spark.stop()

Exploratory analysis in Jupyter Notebooks benefits from quick text reads—load a file, inspect with show, and parse, ideal for ad-hoc insights from raw data.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Explore").getOrCreate()
df = spark.read.text("data.txt")
df.show(truncate=False)
spark.stop()

Handling custom formats—like pipe-delimited logs—uses lineSep to align rows, then splits fields for real-time analytics, adapting to non-standard text sources.

from pyspark.sql import SparkSession
from pyspark.sql.functions import split

spark = SparkSession.builder.appName("CustomFormat").getOrCreate()
df = spark.read.text("custom.txt", lineSep="|")
df_split = df.select(split("value", ",").alias("fields"))
df_split.show()
# Assuming custom.txt: Alice,25|Bob,30
# Output:
# +----------+
# |    fields|
# +----------+
# |[Alice,25]|
# |  [Bob,30]|
# +----------+
spark.stop()

FAQ: Answers to Common Questions About Reading Text Files

Here’s a detailed rundown of frequent questions about reading text in PySpark, with thorough answers to clarify each point.

Q: How does wholeText=True affect scalability?

With wholeText=True, each file becomes one row, loading its full content into memory—fine for small files but risky for large ones, as it can overwhelm executors. Default mode (False) splits by lines, scaling better for big datasets like logs, distributing rows across the cluster.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("WholeText").getOrCreate()
df = spark.read.text("data.txt", wholeText=True)
df.show(truncate=False)
spark.stop()

Q: Can I read structured text directly?

No—spark.read.text() loads raw lines into a value column; for structured text (e.g., CSV), use read.csv(). You’d parse the value column with split or UDFs to structure it.

from pyspark.sql import SparkSession
from pyspark.sql.functions import split

spark = SparkSession.builder.appName("StructText").getOrCreate()
df = spark.read.text("data.txt")
df.select(split("value", ",").getItem(0).alias("name")).show()
spark.stop()

Q: What’s the difference with spark.read.csv()?

spark.read.text() reads raw lines into one column, while read.csv() parses delimited fields into multiple columns with a schema. Text is for unstructured data; CSV is for structured, offering header and type inference options.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TextVsCSV").getOrCreate()
df_text = spark.read.text("data.txt")
df_csv = spark.read.csv("data.txt", header=True)
df_text.show()
df_csv.show()
spark.stop()

Q: How do I handle non-standard line separators?

Set lineSep to your delimiter—like "|"—and Spark splits rows accordingly. It’s limited to 128 characters, so complex separators need preprocessing or custom data sources.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LineSep").getOrCreate()
df = spark.read.text("data.txt", lineSep="|")
df.show()
# Assuming data.txt: Alice|Bob
# Output:
# +-----+
# |value|
# +-----+
# |Alice|
# |  Bob|
# +-----+
spark.stop()

Q: Does it support compressed files?

Yes—Spark reads .gz, .bz2, or .zip files natively, like "data.txt.gz", decompressing on the fly with no extra parameters, scaling for compressed logs or dumps efficiently.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Compressed").getOrCreate()
df = spark.read.text("data.txt.gz")
df.show()
spark.stop()

Reading Text Files vs Other PySpark Features

Reading text with spark.read.text() is a data source operation, distinct from RDD reads or ORC reads. It’s tied to SparkSession, not SparkContext, and feeds raw text into DataFrame operations.

More at PySpark Data Sources.

Conclusion

Reading text files in PySpark with spark.read.text() offers a simple, scalable entry to unstructured data, guided by key parameters. Elevate your skills with PySpark Fundamentals and master the read!