---
title: "PySpark Dataframe: Skipping First Rows and Counting Null Values"
description: "This PySpark guide covers skipping rows (beyond header) and counting NULLs for each column of a DataFrame."
url: https://srinimf.com/2024/04/23/pyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column/
date: 2024-04-23
modified: 2026-10-10
author: "Srini"
image: https://i0.wp.com/srinimf.com/wp-content/uploads/2024/04/pexels-photo-574080.jpeg?fit=1880%2C1245&ssl=1
categories: ["PySpark"]
tags: ["PySpark Interview questions"]
type: post
lang: en
---

# PySpark Dataframe: Skipping First Rows and Counting Null Values

In a Mphasis interview, two PySpark questions were asked about skipping the first two rows of a dataframe and counting the nulls in each column.

Related[5 Best Ways to Delete Rows in PySpark](https://srinimf.com/2024/03/25/5-best-ways-to-delete-rows-in-pyspark/)

![PySpark Interview Questions: Skip the First 2 Rows and Count the Nulls of Each Column](https://i0.wp.com/srinimf.com/wp-content/uploads/2024/04/45.10.114.jpg?resize=1536%2C1017&ssl=1)*[Pint Mug](https://www.brooklynmuseum.org/opencollection/objects/859) by Jacob Whitmore is licensed under [CC-BY 3.0](https://creativecommons.org/licenses/by/3.0/)*

## 1. Skipping the first two rows: PySpark

[In PySpark](https://srinimf.com/2024/12/22/calculate-match-percentage-in-pyspark-sql/), to skip the header you can use **option(“skipFirstLine”, “true”)**. 

Related](https://srinimf.com/2024/07/20/pyspark-when-check-numeric-of-column-tested/)

However, there’s no built-in option to skip additional lines beyond the header.

Related[SQL AVG Column Function: How to Use it](https://srinimf.com/2015/08/20/db2-avg-column-function-best-example/)

To skip the first two rows, you can use the skip option in combination with the drop method after reading the CSV file.

```
from pyspark.sql import SparkSession

# Create Spark session
spark = SparkSession.builder \
    .appName("Read CSV with Skip Rows") \
    .getOrCreate()

# Define CSV path
csv_path = "path/to/your/file.csv"

# Read CSV file into DataFrame
df = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv(csv_path)

# Drop the first two rows
df = df.drop(*df.head(2))

# Show DataFrame
df.show()
```

Related[Using conditions: 5 Top PySpark SQL Examples](https://srinimf.com/2023/09/15/how-to-add-conditions-in-pyspark-sql-5-examples/)

The *df.head(2), unpacks the list of row objects into separate Row objects.

## 2. Counting NULLs of each column: PySpark

In PySpark, you can count the number of [null values in](https://srinimf.com/2024/10/27/how-to-drop-columns-with-high-null-values-in-pyspark/) each column of a DataFrame using the `isNull()` method combined with a list comprehension to iterate over all columns.

```
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Create Spark session
spark = SparkSession.builder \
    .appName("Count Null Values Example") \
    .getOrCreate()

# Create a sample DataFrame
data = [(1, "A", None),
        (2, "B", 30),
        (3, "C", None),
        (4, None, 40),
        (5, "E", 50)]

columns = ["ID", "Name", "Age"]

df = spark.createDataFrame(data, columns)

# Count null values for each column
null_counts = [df.filter(col(column_name).isNull()).count() for column_name in df.columns]

# Create a dictionary to store column names and their corresponding null counts
null_counts_dict = dict(zip(df.columns, null_counts))

# Display null counts
for column, count in null_counts_dict.items():
    print(f"{column}: {count} null values")
```

Output

```
(6) Spark Jobs
df:pyspark.sql.dataframe.DataFrame = [ID: long, Name: string ... 1 more field]
ID: 0 null values
Name: 1 null values
Age: 2 null values
Command took 22.52 seconds -- by info@srinimf.com at 4/23/2024, 4:24:39 PM on my-cluster
```

Conclusion

The PySpark [interview questions](https://srinimf.com/2025/09/11/altimetrik-interview-questions-2025-python-sql-and-pyspark-explained/) shared here focus on two key areas: skipping the first two rows of a DataFrame and counting the null values of each column.

In the first scenario, the option to skip additional lines beyond the header in PySpark does not exist. However, it is possible to skip the first two rows by using the skip option in combination with the drop method after reading the CSV file. This involves dropping the first two rows using the drop method.

Moving on to the second question, to count the number of null values in each column of a DataFrame, the `isNull()` method combined with a list comprehension is leveraged to iterate over all columns. This enables the creation of a dictionary to store column names and their corresponding null counts, offering a comprehensive view of null value distribution.

Overall, these questions provide valuable insights into handling data preprocessing and management in PySpark, addressing common challenges encountered in data manipulation within the PySpark environment.

Related[DISTINCT Vs. COLLECT_SET: Top Differences](https://srinimf.com/2024/09/22/distinct-vs-collect_set-top-differences/)

### Share this:

-
- [Tweet](https://twitter.com/share)
- [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2024%2F04%2F23%2Fpyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2024%2F04%2Fpexels-photo-574080.jpeg%3Ffit%3D1200%252C795%26ssl%3D1&description=PySpark%20Dataframe%3A%20Skipping%20First%20Rows%20and%20Counting%20Null%20Values)
-
- [Share on Tumblr](https://www.tumblr.com/share)
- [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2024/04/23/pyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column/?share=jetpack-whatsapp)
- [More](#)
-

- [Print (Opens in new window) Print](https://srinimf.com/2024/04/23/pyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column/?share=print)
- [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2024/04/23/pyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column/?share=reddit)
- [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2024/04/23/pyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column/?share=telegram)
- [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20PySpark%20Dataframe%3A%20Skipping%20First%20Rows%20and%20Counting%20Null%20Values&body=https%3A%2F%2Fsrinimf.com%2F2024%2F04%2F23%2Fpyspark-interview-questions-skip-the-first-2-rows-and-count-the-nulls-of-each-column%2F&share=email)
-

### Like this:

Like Loading…