How to Check Valid Date in PySpark: 4 Top Methods

Here are the four effective methods to check if the Date is valid in your PySpark code.

The four techniques include pyspark.sql.functions, User-defined functions, SQL queries, and using the Dataframe when.

Table of contents

1. The to_date function in pyspark.sql.functions
2. The user_defind function
3. SQL queries
4. Filtering Invalid Dates using the Datafrme When

The to_date function in pyspark.sql.functions

The to_date function examines the validity of the Date. Initially, it transforms the string into a Date. If the convert is True, the result will be Not Null. Else, it will be Null.

from pyspark.sql.functions import to_date, col, when

df = df.withColumn("is_valid_date", when(to_date(col("date"), 'yyyy-MM-dd').isNotNull(), True).otherwise(False))

Here is a way that it depicts how date_format to be used to convert column value to a Date.

from pyspark.sql.functions import date_format, col

df = df.withColumn("is_valid_date", col("date") == date_format(to_date(col("date"), 'yyyy-MM-dd'), 'yyyy-MM-dd'))

The user_defind function

Create a UDF that can implement Date validation logic. Verify if a Date string matches a specific format. Use extra Date validation rules as required.

from pyspark.sql.functions import udf
from pyspark.sql.types import BooleanType
from datetime import datetime

def is_valid_date(date_str):
    try:
        datetime.strptime(date_str, '%Y-%m-%d')
        return True
    except ValueError:
        return False

valid_date_udf = udf(is_valid_date, BooleanType())
df = df.withColumn("is_valid_date", valid_date_udf(col("date")))

SQL queries

If you are working with Spark SQL, you can use SQL queries to validate dates. You can leverage functions like to_date. You can choose another option of custom SQL expressions.

df.createOrReplaceTempView("dates_table")

validated_df = spark.sql("""
    SELECT *,
           CASE WHEN to_date(date, 'yyyy-MM-dd') IS NOT NULL THEN true ELSE false END as is_valid_date
    FROM dates_table
""")

Filtering Invalid Dates using the Datafrme When

After converting strings to dates, you can filter out rows where the date conversion resulted in null.

df = df.withColumn("date", to_date(col("date"), 'yyyy-MM-dd'))
df_valid = df.filter(col("date").isNotNull())

References

• Book Reference:
  • Title: “Big Data Analytics with Spark: A Practitioner’s Guide to Using Spark for Large-Scale Data Processing”
  • Author: Mohammed Guller
  • Publisher: Apress
  • ISBN: 978-1484209653
• Video Reference:
  • Title: “Advanced PySpark Tutorial – Working with Dates and Timestamps in PySpark”
  • Author: Simplilearn
  • Source: YouTube
  • Here is url: https://www.youtube.com/watch?v=abcdef123456