---
title: "Master the New PySpark Features and Functions in Spark 3.5"
description: "Discover the latest PySpark functions and features in Spark 3.4 and 3.5, including Arrow-optimized UDFs, Python UDTFs, new array helpers, HyperLogLog aggregations, and enhanced streaming. Learn how to use them with practical examples."
url: https://srinimf.com/2025/08/10/master-the-new-pyspark-features-and-functions-in-spark-3-5/
date: 2025-08-10
modified: 2026-10-10
author: "Srini"
image: https://i0.wp.com/srinimf.com/wp-content/uploads/2025/08/pyspark-new-features-in-3.4-3.5.png?fit=2240%2C1260&ssl=1
categories: ["PySpark"]
tags: ["Array Functions", "HyperLogLog", "Spark 3.4", "Spark 3.5"]
type: post
lang: en
---

# Master the New PySpark Features and Functions in Spark 3.5

PySpark is constantly evolving, introducing powerful [new features](https://srinimf.com/2013/11/27/db2-11-for-zos-new-features/) that simplify development, enhance performance, and unlock new analytical possibilities. You can see significant improvements in Spark 3.4 and especially in Spark 3.5. Let’s explore the most exciting ones!

## **1. Arrow-Optimized Python UDFs**

One of the most impactful upgrades arrives with **Arrow-optimized Python UDFs**. When you enable it via:

```
spark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", True)
```

or use `@udf(..., useArrow=True)`, PySpark switches from traditional pickled UDFs to vectorized Arrow-based processing. This can deliver **up to 2× speed improvements** on modern CPUs due to efficient columnar data handling.

*Practical Tip:* Use this for any heavy UDF workloads—especially in data enrichment or feature engineering—where performance matters.

## **2. Python User-Defined Table Functions (UDTFs)**

PySpark now supports **Python UDTFs**, a significant upgrade that lets you create Python functions that return full tables rather than single column values. Defined in code and callable both in Python and via SQL, UDTFs open a powerful door in PySpark development:

```
from pyspark.sql.functions import udtf

class MyHelloUDTF:
    def eval(self, *args):
        yield "hello", "world"

test_udtf = udtf(MyHelloUDTF, returnType="c1: string, c2: string")
test_udtf().show()
```

*Use Case:* Exploding complex JSONs, unpacking embedded arrays, or returning multiple rows from a single input—without complex transformations.

## **3. Enhanced Testing API & Better Error Messages**

Testing becomes much cleaner with the **new PySpark testing utilities**:

```
from pyspark.testing import assertDataFrameEqual, assertPandasOnSparkEqual, assertSchemaEqual
```

These tools provide **color-coded, detailed diff output**, helping quickly spot schema or content mismatches during testing.

Moreover, PySpark now maps errors to structured **error classes and codes**, improving debuggability and integration with external monitoring.

## **4. New Array Helper Functions**

Spark 3.5 brings convenient array manipulation functions like `array_append` and `array_prepend` [to PySpark](https://srinimf.com/2024/03/11/how-to-create-udf-in-pyspark-best-example/)—eliminating common UDF or SQL hackiness.

```
from pyspark.sql.functions import array_append

df.withColumn("new_arr", array_append("arr_col", lit("new_val")))
```

*Why it matters:* Cleaner code, better readability, and faster development when working with nested or list-like data.

## **5. Expanded SQL Function Support + IDENTIFIER Clause**

Spark 3.5 marshaled 150 new SQL functions into the [PySpark DataFrame](https://srinimf.com/2024/09/10/common-pyspark-methods-for-dataframe-operations/) API—covering functions earlier only accessible via raw SQL strings.

It also introduced the **IDENTIFIER clause**, enabling safer template-based SQL queries by safely injecting table or column names, guarding against SQL injection risks.

Named argumen[1](#b2438f16-cea0-4fca-9b95-f62e2a1537a9)t support is now available for PySpark SQL function calls, making complex multi-param functions significantly more readable.

## **6. HyperLogLog Aggregations via Datasketches**

Spark 3.5 adds high-performance **HyperLogLog (HLL)** approximate aggregation functions powered by Apache Datasketches—enabling efficient estimations of distinct counts at scale. These functions persist and merge sketches effectively.

*Ideal for:* Real-time analytics, high cardinality data grouping, and memory-efficient summarization.

## **7. Structured Streaming Enhancements**

Spark Connect now supports structured streaming in Python, making streaming jobs more portable and client-enabled. Spark 3.5 introduces **watermark propagation** across operators and the `dropDuplicatesWithinWatermark` operation—making deduplication in event-time logic much smoother.

## **8. Other Notable PySpark API Upgrades**

Additional noteworthy features include:

- `.offset` method on DataFrame—for intuitive pagination.
- Enhanced `dir(df)` to list DataFrame columns directly.
- Support for nested timestamp types and `TimestampNTZType`.
- Runtime control over Python executables in UDFs.
- `assertDataFrameEqual` utility (also mentions in 3.5.0).
- Support for `fill_value` in pandas-on-Spark Series.

## Summary Table

| Feature | Benefit |
| --- | --- |
| Arrow-optimized UDFs | Faster UDF execution via vectorization |
| Python UDTFs | Table-level function output flexibility |
| Testing APIs + Error Classes | Easier, robust testing & debugging |
| Array helpers (append/prepend) | Cleaner list/array manipulation |
| SQL function expansion & IDENTIFIER | More Pythonic SQL and secure injection |
| HLL aggregations | Efficient approximate distinct counts |
| Streaming enhancements | Better streaming logic and dedup support |
| API convenience methods | Developer productivity gains |

## Final Thoughts

These updates make PySpark more powerful and developer-friendly than ever. Whether you’re optimizing performance with Arrow UDFs, tackling streaming deduplication, enriching your testing workflows, or exploring UDTFs, Spark 3.4–3.5 sets a solid foundation.

1. N**amed argument support** allows you to pass parameters using explicit names. This makes your code **more readable**, **easier to maintain**, and **less error-prone**. [↩︎](#b2438f16-cea0-4fca-9b95-f62e2a1537a9-link)

### Share this:

-
- [Tweet](https://twitter.com/share)
- [![](https://i0.wp.com/assets.pinterest.com/images/pidgets/pinit_fg_en_rect_gray_20.png)](https://www.pinterest.com/pin/create/link/?url=https%3A%2F%2Fsrinimf.com%2F2025%2F08%2F10%2Fmaster-the-new-pyspark-features-and-functions-in-spark-3-5%2F&media=https%3A%2F%2Fi0.wp.com%2Fsrinimf.com%2Fwp-content%2Fuploads%2F2025%2F08%2Fpyspark-new-features-in-3.4-3.5.png%3Ffit%3D1200%252C675%26ssl%3D1&description=Master%20the%20New%20PySpark%20Features%20and%20Functions%20in%20Spark%203.5)
-
- [Share on Tumblr](https://www.tumblr.com/share)
- [Share on WhatsApp (Opens in new window) WhatsApp](https://srinimf.com/2025/08/10/master-the-new-pyspark-features-and-functions-in-spark-3-5/?share=jetpack-whatsapp)
- [More](#)
-

- [Print (Opens in new window) Print](https://srinimf.com/2025/08/10/master-the-new-pyspark-features-and-functions-in-spark-3-5/?share=print)
- [Share on Reddit (Opens in new window) Reddit](https://srinimf.com/2025/08/10/master-the-new-pyspark-features-and-functions-in-spark-3-5/?share=reddit)
- [Share on Telegram (Opens in new window) Telegram](https://srinimf.com/2025/08/10/master-the-new-pyspark-features-and-functions-in-spark-3-5/?share=telegram)
- [Email a link to a friend (Opens in new window) Email](mailto:?subject=%5BShared%20Post%5D%20Master%20the%20New%20PySpark%20Features%20and%20Functions%20in%20Spark%203.5&body=https%3A%2F%2Fsrinimf.com%2F2025%2F08%2F10%2Fmaster-the-new-pyspark-features-and-functions-in-spark-3-5%2F&share=email)
-

### Like this:

Like Loading…