How I Constructed A Cascading Information Pipeline Primarily based on AWS (Half 2) | by Memphis Meng | Aug, 2023

Automated, scalable, and highly effective

Previously, I shared my expertise in growing an information pipeline utilizing AWS CloudFormation know-how. It’s not an optimum method, although, as a result of it leaves behind 3 extra points awaiting decision:

1. The deployment needs to be imposed manually which might enhance the possibilities of errors;
2. All assets are created in a single single stack, with out correct boundaries and layers; as the event cycle goes on, the useful resource stack will probably be heavier, and managing it is going to be a catastrophe;
3. Many assets are speculated to be sustained and reused in different initiatives.

Briefly, we’re going to enhance the manageability and reusability of this challenge, in an agile method.

AWS allows customers to implement 2 kinds of CloudFormation structural patterns: cross-stack reference and nested stacking. Cross-stack reference stands for a designing fashion of growing cloud stacks individually, and normally independently, whereas the assets amongst all stacks could be interrelated primarily based on the reference relationship. Nested stacking means a CloudFormation stack composed of different stacks. It’s achieved through the use of the AWS::CloudFormation::Stack useful resource.

As a result of one in every of our missions we purpose to attain is to give you higher challenge administration, the challenge goes to be damaged down by layered separation and nested stacking is the one to assist. Nonetheless, in regard to the intrinsic interrelationship between the artifacts of the present stack, we’d additionally must take a drop of cross-stack reference.

We created 3 Lambda features, 3 DynamoDB tables, 1 IAM position together with its insurance policies hooked up, a number of SQS queues, and several other Cloudwatch alarms. As a result of complexity of the features themselves, on this model, they’re going to be outlined in separate templates, with the companies solely utilized by themselves together with alarms and lifeless letter queues. Other than these, IAM assets will probably be…