Introduction

If you’re learning Databricks Structured Streaming, you’ve likely come across the terms Window and Watermark. However, these are two of the most commonly misunderstood concepts because developers often use them together in streaming pipelines.

In this guide, you’ll learn:

• First, what a Window is
• Next, what a Watermark is
• Then, why a Watermark is needed
• After that, how they work together
• For example, real-world use cases
• Meanwhile, PySpark implementation details
• Finally, common interview questions to wrap up

Why Are Window and Watermark Important?

Streaming data never stops arriving. Unlike batch processing, Spark cannot wait for all the data before performing calculations. Therefore, it needs a strategy to manage the continuous flow.

For example, imagine you’re processing:

• Online orders
• Website clicks
• IoT sensor data
• Banking transactions

As a result, Spark needs a way to:

1. Group events into manageable chunks.
2. Decide how long to wait for delayed events.

That’s where Window and Watermark come into play.

What is a Window in Databricks?

A Window defines a fixed period of time that groups streaming records together so Spark can perform aggregate operations such as COUNT, SUM, AVG, MAX, and MIN.

Without a Window, Spark would never know when to calculate results because streaming data never ends. Therefore, Windows are essential.

Why Do We Need a Window?

Suppose you want to calculate the number of website visitors every 5 minutes.

Instead of calculating after every single event, Spark groups all events that occur within the same five-minute interval and then performs the aggregation once. Consequently, this approach improves performance and reduces computational overhead.

Window Timeline Diagram

Types of Windows in Databricks

Databricks supports three commonly used window types.

Tumbling Window

A Tumbling Window divides data into fixed, non-overlapping intervals.

Example:

• 10:00–10:05
• 10:05–10:10
• 10:10–10:15

As a result, each event belongs to exactly one window.

Sliding Window

A Sliding Window, meanwhile, overlaps with previous windows.

For example, a 10-minute window sliding every 5 minutes creates overlaps, allowing the same event to appear in multiple windows.

This is useful for moving averages and trend analysis.

Session Window

A Session Window groups events based on user activity.

For instance, if no event arrives within a specified timeout period, Spark automatically closes the session.

Developers commonly use this for user activity tracking and website analytics.

What is a Watermark in Databricks?

It tells Spark how long it should wait for late-arriving events before it considers a window complete.

A Watermark tells Spark how long to wait for late-arriving events before it considers a window complete. Additionally, it helps Spark manage delayed data while preventing excessive memory usage.

Why Do We Need a Watermark?

In real-world applications, data doesn’t always arrive on time. Therefore, delays can occur due to:

• Network latency
• Slow devices
• System failures
• Internet connectivity issues

Without a Watermark, Spark keeps every old window open indefinitely, causing memory usage to grow continuously. Therefore, Watermarks are critical for managing resource consumption.

Watermark Timeline Diagram

Event Time vs Processing Time

Before understanding Watermarks, you need to understand two different timestamps.

Event Time

Event Time records when the event actually occurred. For example, a customer placed an order at 10:02 AM.

Example:
A customer placed an order at 10:02 AM.

Processing Time

Processing Time, meanwhile, records when Spark receives the event.

For example, Spark receives the same order at 10:07 AM because of network delays.

Why Does Watermark Use Event Time?

Spark always calculates Watermarks using Event Time, not Processing Time. This ensures Spark handles late events correctly regardless of when they arrive.

As a result, Spark handles late events correctly regardless of when they arrive.

Watermark vs Window: How They Work Together

A Window groups records together. Meanwhile, a Watermark tells Spark when it’s safe to close that Window.

So, a Watermark tells Spark when it’s safe to close that Window.

Combined Timeline Diagram

(Insert a diagram showing windows, the latest event time, watermark position, and closed windows.)

Real-World Example

Imagine an online shopping application.

Most transactions arrive immediately. However, one transaction generated at 10:02 AM reaches Spark at 10:11 AM because of a temporary network issue.

For instance, a transaction generated at 10:02 AM subsequently reaches Spark at 10:11 AM due to a temporary network issue.

Therefore, if you configure the Watermark as 5 minutes, Spark accepts delayed events only until 10:07 AM.

As a result, Spark discards any event that arrives after that as too late.

PySpark Example

from pyspark.sql.functions import window
stream_df \
.withWatermark("event_time", "5 minutes") \
.groupBy(
    window("event_time", "5 minutes")
) \
.count()

Code Explanation

Therefore, in this example:

• Window duration is 5 minutes.
• Watermark delay is 5 minutes.
• Spark waits an additional 5 minutes for delayed events before closing the window.

Window vs Watermark

Common Mistakes Beginners Make

Confusing Window and Watermark

A Window groups data. However, a Watermark manages delayed data.

Meanwhile, a Watermark manages delayed data.

Assuming Watermark Creates Windows

Watermarks never create Windows. Therefore, you create Windows using the window() function.

In other words, you create Windows using the window() function.

Believing Watermark Uses Processing Time

Watermarks always work with Event Time. Therefore, this ensures Spark handles delayed events correctly.

Frequently Asked Interview Questions

Can Watermark exist without a Window?

Yes. You can also use Watermarks in stream-stream joins.

Does Watermark guarantee every delayed event is processed?

No. Spark only accepts events that arrive within the configured delay threshold.

Why does Watermark improve performance?

This is because Spark safely removes old state information after the watermark passes.

Best Practices

• Choose a Watermark duration based on expected data delays.
• Use Event Time instead of Processing Time.
• Monitor late-event metrics in production.
• Avoid excessively large Watermark values, as they increase memory usage.
• Test your streaming application with delayed events before deployment.

Easy Trick to Remember

Think of an examination hall.

• Window = Exam duration.
• Watermark = Grace period for late students.

Once the grace period ends, the invigilator allows no more students to enter.

Similarly:

• Window defines the aggregation period.
• Watermark defines how long Spark waits for late events.

Key Takeaways

• A Window groups streaming records into fixed time intervals.
• A Watermark determines how long Spark waits for delayed events.
• Windows perform aggregations.
• Watermarks improve memory efficiency by closing old windows.
• Both are essential for building efficient Structured Streaming applications.

Conclusion

Understanding the difference between Window and Watermark is essential for anyone working with Databricks Structured Streaming. Once you grasp these concepts, you’ll be better equipped to build reliable, scalable, and efficient real-time data pipelines. Consequently, whether you’re preparing for interviews or developing production-grade streaming applications, mastering Window and Watermark is a fundamental step in your Databricks journey.