How to Set Up Kinesis Firehose in AWS: Step-by-Step Guide

Discover the essentials of creating a Kinesis Firehose delivery stream in AWS. Follow our comprehensive guide for seamless configuration and setup.

Step 1: Open the Amazon Kinesis Console

1. Log in to your AWS Management Console.
2. Navigate to Amazon Kinesis under the “Analytics” section.

Step 2: Create a Firehose Delivery Stream

1. In the Kinesis console, choose Create delivery stream.
2. Provide a Delivery stream name (e.g., my-firehose-stream).

Step 3: Select the Source

Choose a data source:

• Direct PUT: Firehose will directly receive data from your applications.
• Kinesis Data Stream: You can use it as the source if you already have a Kinesis Data Stream.

Step 4: Configure the Destination

Select the destination where Firehose will deliver the data. In this case, choose Amazon S3.

S3 Destination Settings:

1. S3 Bucket: Select an existing bucket or create a new one (e.g., my-firehose-data-bucket).
2. Prefix: Define a prefix to organize the data in the S3 bucket (e.g., firehose-data/).
3. Error Output Prefix: Optionally define a prefix for failed records (e.g., firehose-errors/).

Scalable Data Streaming with Amazon Kinesis – Top Book

Step 5: Configure Data Transformation (Optional)

• If you want to process or transform the data before delivery:
  1. Enable Data Transformation.
  2. Specify a Lambda function for transforming incoming data.
• If not, you can skip this step.

Step 6: Configure Buffer and Compression Settings

1. Buffer Size and Interval:
   • Define the buffer size (default is 5 MB).
   • Define the buffer interval (default is 300 seconds).
2. Compression:
   • Choose a compression format (e.g., GZIP, Snappy, or leave it as None).

Step 7: Enable Data Encryption (Optional)

• Enable server-side encryption if required.
• Choose an AWS Key Management Service (KMS) key.

Step 8: Review IAM Permissions

Kinesis Firehose requires permission to write to your S3 bucket. Ensure:

1. An IAM role is automatically created for Firehose.
2. The IAM role has a AmazonS3FullAccess a policy or a custom policy granting access to the specified bucket.

Step 9: Review and Create

1. Review all the configurations.
2. Choose Create delivery stream.

Step 10: Test the Delivery Stream

1. Go to the newly created delivery stream.
2. Use the Test with Demo data feature to send sample records.
3. Verify that the records are delivered to the S3 bucket.

Step 11: Integrate with Your Applications

Use the AWS SDK, CLI, or API to send data to the Firehose delivery stream. For example:

Python Example with boto3:

import boto3

client = boto3.client('firehose', region_name='us-east-1')

response = client.put_record(
    DeliveryStreamName='my-firehose-stream',
    Record={
        'Data': b'{"field1": "value1", "field2": "value2"}\n'
    }
)

print(response)