From manual file handling to automated processing: an AWS workflow connecting S3 events, containerized Batch jobs, and DynamoDB state tracking, delivered through Jenkins CI/CD.
From manual file handling to automated processing: an AWS workflow connecting S3 events, containerized Batch jobs, and DynamoDB state tracking, delivered through Jenkins CI/CD.
02 / Problem
Manual file detection and data processing required repeated human intervention. The workflow needed to detect incoming files, run processing workloads, and track state without relying on manual handoffs.
03 / Solution
At Tech Mahindra, I led a 3-engineer team to build an AWS Batch automation system from scratch. It connected S3 event triggers to Dockerized processing jobs, used DynamoDB for state tracking, and deployed through Jenkins CI/CD. Python was used to generate and configure AWS resources. I also used Infrastructure as Code (IaC) with Terraform to provision EC2 instances and supporting AWS resources, standardizing environment setup and reducing manual configuration.
04 / Architecture
S3 file arrival
Incoming files are stored in S3. Automated detection replaces the manual file-handling step.
05 / How it works
A file arrives in S3.
An S3 event initiates the automated workflow.
AWS Batch starts a Dockerized processing job using its ECR-hosted image.
The container runs the data-processing workload.
DynamoDB tracks processing state.
Jenkins CI/CD delivers changes to the system through a separate deployment workflow.
06 / Technology stack
S3
Provides the file-ingestion point and event triggers for automation.
AWS Batch
Runs the containerized processing workload.
Docker / ECR
Packages the job environment and stores the container images.
DynamoDB
Maintains processing state for the workflow.
Jenkins CI/CD
Automates delivery of changes to the system.
Python
Generates and configures AWS resources.
Terraform / IaC
Automates infrastructure provisioning and standardizes AWS environment configuration.
07 / Challenges & lessons learned
Engineering challenges addressed in the delivered workflow, with lessons drawn from the implementation.
CHALLENGE 01
Removing manual file handling
Problem
The existing data-processing workflow depended on manual intervention.
Approach
Led a 3-engineer team to connect S3 event triggers, AWS Batch jobs, and DynamoDB state tracking.
Result
The Batch system eliminated manual intervention through automated job triggers and state tracking.
Engineering takeaway
Automating the handoffs around a job is part of improving the complete workflow, alongside the processing logic itself.
CHALLENGE 02
Making processing state visible
Problem
An automated workflow needs a way to track processing state outside the running job.
Approach
Used DynamoDB for state tracking alongside Dockerized AWS Batch workloads.
Result
State tracking formed part of the delivered automation system.
Engineering takeaway
Job execution and state tracking are separate responsibilities. Explicit state gives the workflow a place to record progress.
CHALLENGE 03
Taking the workload into production
Problem
The processing solution needed to be packaged, deployed, and demonstrated to the client.
Approach
Packaged the jobs in Docker containers, deployed through Jenkins CI/CD, and demonstrated the automation directly to the client.
Result
The manual-processing automation was adopted into production, with a 70% reduction in processing time.
Engineering takeaway
Delivery includes both the deployment process and showing stakeholders how the system improves their work.
08 / Key features
Event-driven file detection
Containerized AWS Batch processing
DynamoDB state tracking
Docker images stored in ECR
Jenkins CI/CD delivery
3-engineer team leadership
Client demonstration and production adoption
Infrastructure as Code with Terraform
09 / Performance & results
70%
Less processing time in the Batch system
3
Engineers on the Batch project team I led
AWS Batch automation reduced manual data-processing time by 70% and was adopted into production after a client demonstration. I led a 3-engineer project team; my broader mentoring work at Tech Mahindra supported 4 junior engineers.
10 / Engineering takeaways
This work connected event-driven ingestion, containerized execution, and external state tracking into an automated processing workflow. The engineering responsibility extended from building the system with a team to demonstrating its value and delivering it into production.