Introduction
Amazon S3 is one of the most popular object storage services used to store anything from data backups to application assets. AWS Lambda is a serverless compute service that lets you run code in response to events. A common use case involves triggering a Lambda function after files are uploaded to an S3 bucket.
But what if you want to trigger Lambda only after 7 files have been uploaded?
This article walks you through how to set up such a scenario using S3, Amazon SQS, and Lambda together. We’ll explain why direct S3-to-Lambda won’t help you here, and how to build an event-driven, cost-effective, and scalable solution instead.
Why S3 Event Notifications Alone Won’t Work
By default, Amazon S3 event notifications trigger Lambda for each object uploaded to a bucket. That means if you upload 7 files, Lambda gets triggered 7 times—once for each file.
This is not efficient if:
- You want to process files in batches.
- You want to reduce the number of Lambda invocations (to save costs).
- Your logic depends on all 7 files being present before processing starts.
So, how do we fix this?
Solution Overview: S3 + SQS + Lambda
Here’s the architecture to trigger Lambda only after 7 files are uploaded:
- Configure Amazon S3 to send event notifications to an Amazon SQS queue.
- Set up a Lambda function to poll the queue.
- The Lambda function processes messages only when 7 files are received.
This design is scalable, decoupled, and follows serverless best practices.
Step-by-Step Implementation
Step 1: Create an Amazon SQS Queue
- Go to the Amazon SQS console.
- Create a new standard queue (not FIFO).
- Name it something like
s3-upload-queue. - Enable access permissions for S3 to send messages to it.
Step 2: Set Up S3 Event Notifications
- Go to your Amazon S3 bucket.
- In the “Properties” tab, scroll to Event notifications.
- Create a new event.
- Event type:
PUT(for new file uploads). - Destination: Your SQS queue.
- Event type:
- Save the changes.
Now, every time a file is uploaded to your bucket, a message will be sent to the SQS queue.
Step 3: Create and Configure a Lambda Function
- Go to the AWS Lambda console.
- Create a new function named
ProcessBatchOf7Files. - Set the trigger source as your SQS queue.
- In the function code:
- Collect messages.
- Check if there are 7 file upload events.
- Only then, proceed with processing.
- Optionally, store or checkpoint the state using Amazon DynamoDB to handle retries or partial batches.
Here’s some sample pseudocode for the Lambda function:
def lambda_handler(event, context):
messages = event['Records']
if len(messages) >= 7:
file_keys = [msg['body'] for msg in messages[:7]]
# Process files
print("Processing these 7 files:", file_keys)
# Delete processed messages (handled automatically if no error)
else:
print("Less than 7 files received. Waiting for more.")Additional Enhancements
- Use Amazon EventBridge if your architecture involves multiple event sources or complex rules.
- Store a count of uploaded files temporarily in DynamoDB to manage batches more flexibly.
- Set up DLQs (dead-letter queues) for Lambda and SQS to handle failed messages.
Benefits of This Approach
- ✅ Batch Processing: Trigger Lambda only when a batch of 7 files is ready.
- ✅ Cost-Effective: Fewer invocations, reduced Lambda duration.
- ✅ Scalable: Can easily modify the batch size or integrate with other services.
- ✅ Decoupled: SQS acts as a buffer and decouples the upload from the processing.
Conclusion
If your use case demands triggering a Lambda function only after 7 files are uploaded to an S3 bucket, relying solely on S3 event notifications won’t cut it. By integrating Amazon S3, SQS, and Lambda, you gain precise control over when and how your function is triggered. This serverless, event-driven architecture is ideal for batch processing, and it scales effortlessly with your workload.
Srinimf 