Skip to main content

· 9 min read

Developing for a data pipeline is very "open" to say the least, this is primarily due to the fact that a data pipeline essentially spins up an AWS EC2 instance which can effectively invoke or perform any action within the entire AWS stack using the AWS CLI, only limited by the instances permissions.

As a result the easier way to think for developing a AWS Data Pipeline is to rather look at what tasks pipelines are useful for and what tools you need to perform the duties you wish to carry out using the Data Pipeline.