Modern applications often allow users to upload files—documents, invoices, images, or datasets. But a production-grade S3 file upload AWS Lambda architecture must be secure, scalable, and well-organized.

In this article, we will build a complete end-to-end architecture where:

1. A frontend user submits data
2. An API receives the request
3. The system validates the request against a database
4. A hashed folder key is generated
5. The file is securely uploaded to an object storage system

We will implement this using Amazon API Gateway, AWS Lambda, PostgreSQL, and Amazon S3.

This architecture is widely used in cloud-native applications and data platforms.

The Problem We Want to Solve

To understand why this architecture matters, consider a web application where users upload files associated with:

• Customer
• Site

Without a structured approach, files stored directly in a bucket become disorganized and difficult to manage.

Example (bad design):

s3://uploads-bucket/
   invoice1.csv
   invoice2.csv
   contract.pdf

To address this challenge, we generate a unique folder per customer + site combination using a hashed folder key.

To achieve this, we generate a hashed folder key using:

MD5(customer_id + "-" + site_id)

For example:

customer_id = 101
site_id = 22hash = 7e4c8b6a9adfbc13

Files are organized as:

s3://uploads-bucket/7e4c8b6a9adfbc13/invoice.csv

This approach ensures:

• clean structure
• deterministic folder naming
• no exposure of business IDs

High-Level Architecture

In our design, the complete system architecture works as follows:

Frontend Application
        │
        │ HTTP Request
        ▼
API Gateway
        │
        ▼
Lambda Function
        │
        │ Validate customer + site
        ▼
PostgreSQL Database
        │
        │ Generate MD5 folder
        ▼
Lambda generates S3 Presigned URL
        │
        ▼
Frontend uploads file
        │
        ▼
Amazon S3 bucket

This approach delivers several key benefits:

• database security
• scalable serverless backend
• direct upload to S3 without routing files through backend servers

Step 1: User Enters Data in Frontend

To initiate the S3 file upload process, users submit a file from the web application. The frontend collects:

• customer_id
• site_id
• file_name

Example request payload:

{
  "customer_id": 101,
  "site_id": 22,
  "file_name": "invoice.csv"
}

The frontend sends this request to the AWS Lambda-backed API endpoint:

POST /generate-upload-url

Step 2: API Gateway Receives Request

The request first reaches Amazon API Gateway, which acts as the entry point to your backend services.

At this stage, API Gateway manages authentication, request validation, and routing to AWS Lambda compute resources.

It performs:

• authentication
• request validation
• routing to backend compute

In this case, the route is configured as:

POST /generate-upload-url

This route triggers a backend AWS Lambda function that handles the core business logic.

Step 3: Lambda Processes the Request

Once the request reaches API Gateway, AWS Lambda takes over to process the incoming data.

The Lambda function is responsible for:

1. validating customer and site
2. generating the hashed folder
3. creating a presigned upload URL

Example Lambda logic:

import json
import hashlib
import boto3
import psycopg2s3 = boto3.client("s3")def lambda_handler(event, context):    body = json.loads(event["body"])
    customer_id = body["customer_id"]
    site_id = body["site_id"]
    file_name = body["file_name"]    folder_key = hashlib.md5(
        f"{customer_id}-{site_id}".encode()
    ).hexdigest()    s3_key = f"{folder_key}/{file_name}"

This process generates a unique hashed folder that determines where the file will be stored in S3.

Step 4: Validate Customer and Site in PostgreSQL

Before the Lambda function generates an upload link, it must verify that both the customer and site exist in the database.

To validate these identifiers, AWS Lambda queries PostgreSQL to confirm the customer-site relationship.

Example SQL query:

SELECT c.customer_id, s.site_id
FROM customer c
JOIN site s
ON c.customer_id = s.customer_id
WHERE c.customer_id = %s
AND s.site_id = %s;

If the record does not exist, the API returns an error, preventing unauthorized access to the S3 file upload system.

This validation step ensures:

• valid customer
• valid site
• no unauthorized uploads

Step 5: Generate S3 Presigned Upload URL

With the customer and site validated, AWS Lambda generates a presigned upload URL instead of routing files through the backend.

This design allows the frontend to upload directly to Amazon S3, bypassing the backend entirely for the file transfer.

Example code:

url = s3.generate_presigned_url(
    "put_object",
    Params={
        "Bucket": "customer-upload-bucket",
        "Key": s3_key
    },
    ExpiresIn=3600
)

The presigned URL is valid for 1 hour, balancing security with practical usability.

Step 6: API Response to Frontend

Lambda returns:

{
  "upload_url": "https://s3-presigned-url",
  "folder_key": "7e4c8b6a9adfbc13",
  "s3_key": "7e4c8b6a9adfbc13/invoice.csv"
}

With the presigned URL in hand, the frontend now knows exactly where to upload the file.

Step 7: Frontend Uploads File to S3

The frontend performs an HTTP PUT request:

PUT https://s3-presigned-url

The file content is sent directly to S3 via the presigned URL, ensuring secure and direct file upload to S3.

Resulting object:

s3://customer-upload-bucket/
    7e4c8b6a9adfbc13/
        invoice.csv

By uploading directly to S3, this architecture prevents backend servers from handling and storing large file uploads, improving scalability and reducing infrastructure costs.

Why Use Hash-Based Folder Keys?

Hash-based folder organization offers several advantages for managing large-scale S3 file upload systems:

1. Clean Storage Structure

bucket/
   7e4c8b6a9a/
   2ab881ef12/
   98cc112ab1/

2. Obfuscation

By hashing customer IDs and site IDs, you obscure business identifiers in S3 paths, adding a layer of obfuscation.

3. Deterministic Mapping

The same customer + site combination always maps to the same folder, enabling deterministic and predictable file organization.

4. Scalable File Organization

This approach scales efficiently, even when managing millions of files across multiple tenants.

Security Best Practices

When implementing an S3 file upload AWS Lambda architecture, follow these security best practices:

Never Expose Database to Frontend

Ensure your frontend never connects directly to PostgreSQL; instead, route all requests through the API Gateway and Lambda layer.

This enforcement ensures that database credentials remain secure and inaccessible to client-side code.

Use Presigned URLs

Presigned URLs are essential for secure Lambda S3 upload workflows because they:

• credential exposure
• unauthorized uploads

Validate IDs

In every Lambda execution, always verify:

customer_id
site_id

before generating an upload URL. This step prevents forged or unauthorized upload requests.

Use IAM Permissions

When deploying AWS Lambda functions for file upload operations, restrict Lambda role permissions to only:

s3:PutObject

for the required S3 bucket, following the principle of least privilege.

Final Architecture Overview

User uploads file
        │
        ▼
Frontend application
        │
        ▼
API Gateway endpoint
        │
        ▼
Lambda backend
        │
        │ Validate customer + site
        ▼
PostgreSQL
        │
        │ Generate MD5 folder
        ▼
Lambda generates presigned URL
        │
        ▼
Frontend uploads file
        │
        ▼
Amazon S3 bucket

Together, these components create an architecture that is secure, scalable, and cloud-native.

Conclusion

Building a robust upload system requires more than simply storing files. By integrating:

• API-driven architecture
• serverless compute
• database validation
• hash-based folder organization
• direct S3 uploads

you can build a highly scalable and secure solution for production environments.

Organizations that combine Amazon API Gateway, AWS Lambda, PostgreSQL, and Amazon S3 into a unified S3 file upload pipeline can implement production-grade solutions with minimal infrastructure management.

This design pattern—featuring serverless compute, presigned URLs, and hash-based organization—is widely adopted in modern cloud data platforms, SaaS applications, and enterprise systems.