CI/CD with AWS Serverless and Bitbucket Pipelines

If you are an AWS serverless engineer and are seeking an efficient solution to deploy AWS serverless components, then this is the solution for you. AWS Lambda and AWS Glue jobs are common serverless components that can be utilized in various data engineering projects or projects involving the generation and propagation of orbital leads to downstream components. Today, we will deploy these two AWS serverless resources using Bitbucket Pipelines.

AWS Lambda: This is one of the easiest solutions for serverless applications, where computation power is limited to 15 minutes. It serves as an excellent triggering point for downstream batch jobs or a series of different Lambdas.

AWS Glue Job: This is a powerful serverless component that is not limited by time constraints or computation power. It is primarily used for handling significant computations, time-consuming tasks, and processing heavy records.

Bitbucket Pipelines: Bitbucket, similar to GitHub, provides an efficient Git solution for managing code repositories and facilitating integrations for deploying code bases into various environments.

Bitbucket Pipes: Pipes are prebuilt containers made available by Bitbucket to simplify developers’ lives by offering pre-configured libraries. Developers only need to provide certain parameters for the pipe to function according to their requirements.

Architecture of the Automatic Deployment Procedures

Above is the architecture of the automated CI/CD process for AWS serverless applications. Essentially, whenever a developer commits their code to the release branch, the deployment pipeline will be triggered. Files will be packaged accordingly. If we wish to include only specific files, similar to working with a Linux-based system, we can write commands to create a new folder, copy specific files, and compress them for deployment.

Next, the code provided below should be updated in the sample Bitbucket Pipelines document mentioned here. All variables can be securely configured in the pipeline settings, which can be done manually in the repository’s settings.

# to deploy aws lambda
 - pipe: atlassian/aws-lambda-deploy:0.8.0
 variables:
   AWS_ACCESS_KEY_ID: $AWS_KEY
   AWS_SECRET_ACCESS_KEY: $AWS_SECRET
   AWS_DEFAULT_REGION: $AWS_REGION
   FUNCTION_NAME: $LAMBDA_ARN
   COMMAND: "update"
   ZIP_FILE: "lambda_code.zip"
   PUBLISH_FLAG: "--publish"
   DEBUG: "True"
# to deploy glue-scripts to s3
 - pipe: atlassian/aws-s3-deploy:1.1.0
 variables:
   AWS_ACCESS_KEY_ID: $AWS_KEY
   AWS_SECRET_ACCESS_KEY: $AWS_SECRET
   AWS_DEFAULT_REGION: $AWS_REGION
   S3_BUCKET: $S3_BUCKET/$S3_KEY
   LOCAL_PATH: "glue-scripts" #path of the local code folder

Results:

The results below show a successful run of the pipeline, deploying AWS serverless components such as Lambda and Glue. We use S3 deployment for Glue jobs, as the code for Glue jobs is always fetched from S3.

Reference:

[1]. https://support.atlassian.com/bitbucket-cloud/docs/deploy-to-aws-with-s3/

[2]. Sai Teja Makani, Streamlining AWS Serverless Deployments: A Bitbucket Pipeline Approach, International Journal of Computer Applications (IJCA), 4(1), 2023, pp. 1–8 https://iaeme.com/MasterAdmin/Journal_uploads/IJCA/VOLUME_4_ISSUE_1/IJCA_04_01_001.pdf