Introduction
If you’re learning Databricks Structured Streaming, you’ve likely come across the terms Window and Watermark. These are two of the most commonly misunderstood concepts because they are often used together in streaming applications.
Many beginners think they perform the same function. In reality, they solve different problems.
In this guide, you’ll learn:
- What a Window is
- What a Watermark is
- Why Watermark is needed
- How they work together
- Real-world examples
- PySpark implementation
- Common interview questions
⭐ Srinimf Pics
Handpicked books, gadgets & learning resources
Why Are Window and Watermark Important?
Streaming data never stops arriving. Unlike batch processing, Spark cannot wait for all the data before performing calculations.
For example, imagine you’re processing:
- Online orders
- Website clicks
- IoT sensor data
- Banking transactions
Spark needs a way to:
- Group events into manageable chunks.
- Decide how long to wait for delayed events.
That’s where Window and Watermark come into play.
What is a Window in Databricks?
A Window is a fixed period of time that groups streaming records together so Spark can perform aggregate operations such as COUNT, SUM, AVG, MAX, and MIN.
Without a Window, Spark would never know when to calculate the results because streaming data never ends.
Why Do We Need a Window?
Suppose you want to calculate the number of website visitors every 5 minutes.
Instead of calculating after every single event, Spark groups all events occurring within the same five-minute interval and performs the aggregation once.
Window Timeline Diagram
(Insert a timeline diagram here showing three 5-minute windows.)
Types of Windows in Databricks
Databricks supports three commonly used window types.
Tumbling Window
A Tumbling Window divides data into fixed, non-overlapping intervals.
Example:
- 10:00–10:05
- 10:05–10:10
- 10:10–10:15
Each event belongs to exactly one window.
Sliding Window
A Sliding Window overlaps with previous windows.
For example, a 10-minute window sliding every 5 minutes creates overlapping windows that allow the same event to appear in multiple windows.
This is useful for moving averages and trend analysis.
Session Window
A Session Window groups events based on user activity.
If no event is received for a specified timeout period, Spark automatically closes the session.
This is commonly used for user activity tracking and website analytics.
What is a Watermark in Databricks?
A Watermark tells Spark how long it should wait for late-arriving events before considering a window complete.
It helps Spark manage delayed data while preventing excessive memory usage.
Why Do We Need a Watermark?
In real-world applications, data doesn’t always arrive on time.
Delays can occur because of:
- Network latency
- Slow devices
- System failures
- Internet connectivity issues
Without a Watermark, Spark would keep every old window open indefinitely, causing memory usage to grow continuously.
Watermark Timeline Diagram
(Insert a diagram showing the latest event time, watermark position, and late events being discarded.)
Event Time vs Processing Time
Before understanding Watermarks, it’s important to understand two different timestamps.
Event Time
Event Time is when the event actually occurred.
Example:
A customer placed an order at 10:02 AM.
Processing Time
Processing Time is when Spark receives the event.
Example:
Spark receives the same order at 10:07 AM because of network delays.
Why Does Watermark Use Event Time?
Watermarks are always calculated using Event Time, not Processing Time.
This ensures late events are handled correctly regardless of when they arrive.
How Window and Watermark Work Together
A Window groups records together.
A Watermark tells Spark when it’s safe to close that Window.
Combined Timeline Diagram
(Insert a diagram showing windows, the latest event time, watermark position, and closed windows.)
Real-World Example
Imagine an online shopping application.
Most transactions arrive immediately.
However, one transaction generated at 10:02 AM reaches Spark at 10:11 AM because of a temporary network issue.
If the Watermark is configured as 5 minutes, Spark accepts delayed events only until 10:07 AM.
Any event arriving after that is considered too late and is discarded.
PySpark Example
from pyspark.sql.functions import windowstream_df \.withWatermark("event_time", "5 minutes") \.groupBy( window("event_time", "5 minutes")) \.count()
Code Explanation
In this example:
- Window duration is 5 minutes.
- Watermark delay is 5 minutes.
- Spark waits an additional 5 minutes for delayed events before closing the window.
Window vs Watermark
| Feature | Window | Watermark |
|---|---|---|
| Purpose | Groups streaming records | Handles late-arriving records |
| Aggregation | Yes | No |
| Creates Time Buckets | Yes | No |
| Closes Windows | No | Yes |
| Improves Memory Usage | No | Yes |
| Uses Event Time | Yes | Yes |
Common Mistakes Beginners Make
Confusing Window and Watermark
A Window groups data.
A Watermark manages delayed data.
Assuming Watermark Creates Windows
Watermarks never create Windows.
Windows are created using the window() function.
Believing Watermark Uses Processing Time
Watermarks always work with Event Time.
Frequently Asked Interview Questions
Can Watermark exist without a Window?
Yes. Watermarks are also used in stream-stream joins.
Does Watermark guarantee every delayed event is processed?
No. Only events arriving within the configured delay threshold are accepted.
Why does Watermark improve performance?
Because Spark can safely remove old state information after the watermark passes.
Best Practices
- Choose a Watermark duration based on expected data delays.
- Use Event Time instead of Processing Time.
- Monitor late-event metrics in production.
- Avoid excessively large Watermark values, as they increase memory usage.
- Test your streaming application with delayed events before deployment.
Easy Trick to Remember
Think of an examination hall.
- Window = Exam duration.
- Watermark = Grace period for late students.
Once the grace period ends, no more students are allowed to enter.
Similarly:
- Window defines the aggregation period.
- Watermark defines how long Spark waits for late events.
Key Takeaways
- A Window groups streaming records into fixed time intervals.
- A Watermark determines how long Spark waits for delayed events.
- Windows perform aggregations.
- Watermarks improve memory efficiency by closing old windows.
- Both are essential for building efficient Structured Streaming applications.
Conclusion
Understanding the difference between Window and Watermark is essential for anyone working with Databricks Structured Streaming. Once you grasp these concepts, you’ll be better equipped to build reliable, scalable, and efficient real-time data pipelines. Whether you’re preparing for interviews or developing production-grade streaming applications, mastering Window and Watermark is a fundamental step in your Databricks journey.
Leave a Reply