---
title: "PySpark: Splitting Text File into Columns Using Substring Function"
description: "In PySpark, use substring and select statements to split text file lines into separate columns of fixed length."
url: https://srinimf.com/2024/03/15/how-to-write-text-data-as-columns-pysprak/
date: 2024-03-15
modified: 2026-10-10
author: "Srini"
image: https://i0.wp.com/srinimf.com/wp-content/uploads/2024/03/image-7.png?fit=1024%2C768&ssl=1
categories: ["Data Science"]
tags: ["PySpark Interview questions"]
type: post
lang: en
---

# PySpark: Splitting Text File into Columns Using Substring Function

If you have a text file in PySpark where each line represents a record with no separators, and you need [to split](https://srinimf.com/2016/08/04/tip-to-split-input-file-with-outfil/) each line into separate columns based on a fixed length (e.g., 10 characters), you can do this using the **[`substring` function](https://srinimf.com/2024/02/12/top-5-pyspark-interview-questions-tredence-analytics/)** along with `select` statements. Here’s how you can do it.

## PySpark Code: Reading Text file & Write as Columns

### Input file: p.txt

```
12345oooooooQQQQQQQQQPPPPPPPP
```

### PySpark Code

```
from pyspark.sql import SparkSession 
from pyspark.sql.functions import substring
spark=SparkSession.builder.appName("Name").getOrCreate()

df=spark.read.text("/content/p.txt") 

df=df.select(
    substring(df.value, 1, 10).alias("col1"),
    substring(df.value, 11, 10).alias("col2"),
    substring(df.value, 21, 1000000).alias("col3")  ## you can give any big value: means till end of the row.
             
)

df.show()
```

**Recommended Books**

- [Data Pipeline Pocket Reference](https://www.amazon.in/Data-Pipelines-Pocket-Reference-Processing/dp/8194722934/?tag=srinimf-21)
- [Querying Databricks SQL](https://www.amazon.in/dp/9355518013/?tag=srinmf-21)
- [Azure Databricks Cookbook](https://www.amazon.in/dp/1789809711/?tag=srinmf-21)

Output

```
+----------+----------+---------+
|      col1|      col2|     col3|
+----------+----------+---------+
|12345ooooo|ooQQQQQQQQ|QPPPPPPPP|
+----------+----------+---------+
```

### Conclusion

- We first read the text file [into a DataFrame](https://srinimf.com/2024/03/26/2-easy-ways-to-read-multiple-files-into-a-dataframe-pyspark/) using spark.read.text(“/content/p.txt”). Then, we use the substring function to split each line into individual columns based on a fixed length of 10 characters.
- We use select statements to extract substrings from each line and alias them as separate columns (e.g., “col1”, “col2”, etc.).
- Finally, we show the resulting DataFrame where each line has been split into separate columns based on the fixed length. Adjust the substring parameters and alias names according to your specific requirements.

### Share this:

-
- [Tweet](https://twitter.com/share)
- [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2024%2F03%2F15%2Fhow-to-write-text-data-as-columns-pysprak%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2024%2F03%2Fimage-7.png%3Ffit%3D1024%252C768%26ssl%3D1&description=PySpark%3A%20Splitting%20Text%20File%20into%20Columns%20Using%20Substring%20Function)
-
- [Share on Tumblr](https://www.tumblr.com/share)
- [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2024/03/15/how-to-write-text-data-as-columns-pysprak/?share=jetpack-whatsapp)
- [More](#)
-

- [Print (Opens in new window) Print](https://srinimf.com/2024/03/15/how-to-write-text-data-as-columns-pysprak/?share=print)
- [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2024/03/15/how-to-write-text-data-as-columns-pysprak/?share=reddit)
- [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2024/03/15/how-to-write-text-data-as-columns-pysprak/?share=telegram)
- [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20PySpark%3A%20Splitting%20Text%20File%20into%20Columns%20Using%20Substring%20Function&body=https%3A%2F%2Fsrinimf.com%2F2024%2F03%2F15%2Fhow-to-write-text-data-as-columns-pysprak%2F&share=email)
-

### Like this:

Like Loading…