Lead Data Engineer – Physical AI Platform

Caterpillar is hiring a Lead Data Engineer to design, build, and maintain scalable data pipelines, microservices, and cloud-based data platforms that support reliable, high-quality data for business and engineering teams. The role emphasizes cloud-native ingestion and streaming, data integration frameworks, operational monitoring, and data quality validation within an agile delivery environment.

Key Responsibilities

• Collaborate with Principal Software Engineers and Data Architects to define solution architecture.
• Lead solution design and optimization of scalable data pipelines and microservices in Python, supporting both real-time and batch data processing across enterprise platforms.
• Develop cloud-native data ingestion and streaming solutions using AWS services including Kinesis, S3, DynamoDB, EventBridge, and related technologies.
• Own the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives.
• Work with business, product, and engineering stakeholders to convert complex requirements into scalable data architectures, workflows, mappings, and system designs.
• Establish and enforce automated testing, data quality controls, and validation frameworks to support integrity, reliability, and compliance across distributed data ecosystems.
• Lead operational monitoring, performance tuning, and root-cause analysis for production data platforms using observability tools such as CloudWatch to maintain high availability and service reliability.

Required Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or a related field.
• 8+ years of experience in data engineering or related disciplines with increasing responsibility.
• Extensive experience working with modern, large-scale, complex data platforms.
• Strong foundation developing and deploying Python solutions in production environments.
• Experience leading teams to build high-throughput, scalable data pipelines.
• Hands-on experience with AWS data services including Kinesis, S3, DynamoDB, EventBridge, etc., at scale.
• Strong SQL skills, including data quality and validation practices.
• Experience deploying software using CI/CD tools such as Azure DevOps, Jira, Jenkins, etc.
• Experience developing microservices that support real-time data ingestion.
• Experience developing software applications using both relational and NoSQL databases.
• Ability to ensure data integrity across distributed and streaming systems.
• Experience with monitoring, testing, and automation in large-scale data environments.

Technologies

• Python, Java
• AWS: Kinesis, S3, DynamoDB, EventBridge, CloudWatch
• CI Autonomy
• CI/CD and tooling: Azure DevOps, Jira, Jenkins
• SQL, APIs, microservices
• Real-time and batch data processing

Location and Work Arrangement

• Full-time position working onsite at the Irving, TX office (Dallas).
• Domestic relocation assistance is available.
• Visa sponsorship is available.

Compensation

Summary pay range: USD 128,470 - 208,770 per year.

Benefits

• Medical, dental, and vision benefits.
• Paid time off plan (Vacation, Holidays, Volunteer, etc.).
• 401(k) savings plans.
• Health Savings Account (HSA).
• Flexible Spending Accounts (FSAs).
• Health Lifestyle Programs.
• Employee Assistance Program.
• Voluntary Benefits and Employee Discounts.
• Career Development.
• Incentive bonus.
• Disability benefits.
• Life Insurance.
• Parental leave.
• Adoption benefits.
• Tuition Reimbursement.

Additional Posting Details

Any offer of employment is conditioned upon successful completion of a drug screen.