AWS Glue crawlers: fix dynamic S3 folder paths with wildcards

Introduction

Beginners often face challenges with AWS Glue crawlers when S3 folder names change dynamically. Apps, APIs, or ETL jobs may create folders with random names, such as hashes or UUIDs. This makes it difficult for Glue crawlers to process the correct data.

For example, when working with AWS Glue crawler dynamic S3 folder paths, your S3 path may look like this:

s3://my-bucket/output/
   8d3f1a92/result/
   c2b7e5f1/result/
   9af1b2c8/result/

Here, the middle folder (8d3f1a92, c2b7e5f1) changes dynamically every execution, and therefore it cannot be treated as a stable part of the path.

Because of this constantly changing segment, the Glue crawler may:

create multiple unwanted partitions
fail to create a clean table
treat the hash folder as part of the schema

In this blog, we’ll therefore understand why this issue happens and how to fix it properly so your crawlers behave predictably.

AWS Glue Crawlers S3 Dynamic folder

To make this clearer, let’s say your actual folder structure is:

s3://project-bucket/results/
   123abc456/result/data.parquet
   789xyz111/result/data.parquet

In this structure, the folder names:

123abc456
789xyz111

are dynamic variables and, as a result, they change from run to run.

These may be, for example:

request IDs
hash values
UUIDs
timestamp-based folders

The issue is that the crawler instead expects stable folder structures.

So, when you crawl this path:

s3://project-bucket/results/

Glue then scans every level and may treat the dynamic folder as a partition column.

As a result, that makes querying difficult.

Why This Happens in AWS Glue

In general, AWS Glue crawler works best with predictable folder hierarchies.

For example:

s3://bucket/sales/year=2026/month=04/

This is perfect because:

folder names are meaningful
partitions are stable

However, dynamic folders like:

s3://bucket/output/9a8b7c/result/

have no business meaning in your data model.

So Glue doesn’t know what to do with them and, consequently, may create confusing partitions or tables.

Best Solution: Use Wildcard Path Pattern

In practice, this is the easiest and most effective solution.

Instead of creating the table from the top-level path, create it using a wildcard so that variable segments are handled automatically.

Specifically, use this path:

s3://project-bucket/results/*/result/

The * means:

“ignore any folder name at this level”

This automatically handles changing folder names, and therefore you don’t need to update the path for every new run.

Step-by-Step Fix

Step 1: Identify the Stable Folder Pattern

For instance, here are some example actual paths:

s3://project-bucket/results/abc123/result/
s3://project-bucket/results/def456/result/
s3://project-bucket/results/ghi789/result/

From these, a common stable pattern emerges:

results/*/result/

Step 2: Create Table Using Athena

To take advantage of this pattern, open Amazon Athena and run:

CREATE EXTERNAL TABLE result_table (
    customer_id string,
    amount double,
    created_date string
)
STORED AS PARQUET
LOCATION 's3://project-bucket/results/*/result/';

This creates one clean table, and as a result, you can query all dynamic folders together.

Step 3: Validate the Data

Next, run:

SELECT * FROM result_table LIMIT 10;

Now Athena reads data from all dynamic folders, while still presenting a single logical table.

Alternative Fix: Create a Curated Folder

For production systems, this is therefore the best practice.

Instead of querying dynamic folders directly, move data into a clean folder so that downstream queries remain simple.

For example:

raw/
   abc123/result/
   xyz456/result/curated/
   result/

How to Do This Using Glue Job

To do this efficiently, use a Glue PySpark job:

from pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()df = spark.read.parquet("s3://project-bucket/results/*/result/")df.write.mode("overwrite").parquet(
    "s3://project-bucket/curated/result/"
)

After this step, your crawler can safely crawl:

s3://project-bucket/curated/result/

This gives a stable table and, consequently, predictable query performance.

Why This Is Better

Overall, the benefits of a curated path include the following:

no dynamic folder confusion
easy crawler setup
faster queries
better performance
easier debugging

This is the standard approach used in real projects, and it therefore scales much better over time.

Common Beginner Mistake

However, many beginners try this:

s3://project-bucket/results/

This instead causes:

messy partitions
incorrect schema
multiple tables

To avoid these problems, avoid crawling unstable paths.

Best Practice Recommendation

Instead, use this approach:

Raw Layer
   results/hash/result/Processing Layer
   Glue JobCurated Layer
   curated/result/

This is a professional data engineering design and, as a result, simplifies maintenance and governance.

Final Conclusion

If your S3 path contains dynamically changing variables such as hashes, UUIDs, or run IDs, do not directly crawl the parent folder, because this leads to inconsistent tables and partitions.

Instead, use one of these approaches:

Quick Fix

First, use wildcard path:

*/result/

Production Fix

Second, move data to curated folder and crawl that path.

Finally, this keeps your Glue tables clean and query-ready, even when underlying folders change every run.

Srinimf

Subscribe for DAILY TIPS