Data Engineer Lead

The Data Engineering organization at AIG is looking for a hands-on Data Engineer Lead to design, build, and run production-grade data pipelines and platforms. This role applies engineering judgment, collaborates closely with product and stakeholders, and leads standards, reviews, mentorship, and technical documentation.

Key Responsibilities

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3, including integrated data quality checks and observability.
• Own end-to-end delivery of data products from requirements through production, working within an Agile/Scrum framework, including sprint planning, CI/CD, release management, and production readiness.
• Translate business and product requirements into technical designs, defining pipeline structure, data models, and SLAs.
• Apply knowledge of architecture and design patterns by evaluating trade-offs and communicating clear rationale to peers and stakeholders.
• Set and enforce engineering standards across the team, including coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks.
• Conduct thorough code reviews and pair with engineers on complex issues to raise overall technical quality.
• Use GenAI tooling throughout development, including Claude Code CLI and Snowflake Cortex, to improve code quality and delivery speed.
• Collaborate with product owners to refine requirements, surface trade-offs, and push back constructively when scope is not technically workable.
• Communicate design decisions and delivery updates clearly to both engineering peers and non-technical stakeholders.
• Mentor junior engineers through reviews, pairing, and knowledge sharing to build team capability alongside product delivery.
• Maintain high-quality technical documentation as a standard part of delivery, including Architecture Decision Records (ADR), runbooks, data dictionaries, and pipeline lineage notes.
• Evaluate and drive adoption of productivity tooling and best practices, establishing team norms for safe and effective use.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related technical field.
• 7+ years of hands-on data engineering experience with a consistent track record delivering production data pipelines end-to-end.
• Strong command of PySpark, Python, and SQL, including Spark optimization, partition management, and query performance tuning in Snowflake.
• Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch.
• Working knowledge of architecture and design principles, including Lakehouse patterns and fault-tolerant ingestion, with demonstrated ability to apply them in practice.
• Strong analytical and problem-solving skills, including decomposing complex requirements and arriving at practical, well-justified solutions through trade-off analysis.
• Clear communication skills, including explaining technical decisions and rationale in plain language to engineering peers and business stakeholders.
• Self-motivated learner with interest in staying current with evolving tools, frameworks, and engineering best practices.

Technologies

PySpark, Python, AWS EMR, AWS Glue, Amazon S3, AWS Aurora, AWS Lambda, IAM, CloudWatch, SQL, Snowflake, Snowflake Cortex, Claude Code, Claude Code CLI, GenAI, Agile/Scrum, CI/CD, Architecture Decision Record (ADR)

Preferred Qualifications

• AWS certification such as Solutions Architect Associate or equivalent.
• Domain familiarity with insurance or financial services data, including claims, policy, risk, or actuarial structures.
• Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team.
• Demonstrated use of GenAI tooling (for example, Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality.

Benefits

• Eligible for a bonus in accordance with the terms of the applicable incentive plan.
• Comprehensive benefits package including health, wellbeing, financial security, and professional development.
• Hybrid flexibility.
• Competitive compensation.

Base Salary Range

USD 125,000 - 135,000 per year.

Location and Work Model

Jersey City, NJ (onsite). In-person collaboration is valued as part of the culture, so team members are asked to be primarily in the office.

About the Team and Culture

The Data Office is made up of builders, thinkers, and pragmatic problem-solvers. The team emphasizes data engineering as a team sport, including careful documentation, collaborative design and review, and mentoring across levels while staying closely connected to business stakeholders. GenAI is encouraged as a practical force multiplier rather than a novelty.

Total Rewards

A Total Rewards Program is provided with benefits extending beyond time spent at work, including the comprehensive benefits package described above.