Senior Data Engineer

Delta Airlines is seeking a Senior Data Engineer in Atlanta, GA (onsite) to strengthen IT Service Assurance through secure, scalable, and AI-ready data solutions. In this role, you will lead the design and delivery of trusted data products while modernizing AWS data platforms and enabling operational intelligence across ITSM, CMDB, observability, telemetry, and enterprise platforms.

What you’ll do

• Lead independently across complex data initiatives by spotting opportunities for improvement, shaping solutions, and driving delivery from concept through production.
• Design and modernize secure, scalable AWS data platforms, pipelines, lakehouse patterns, and reusable data products aligned with enterprise architecture and governance standards.
• Provide senior technical leadership for ingestion, transformation, modeling, metadata, data quality, lineage, performance optimization, and production reliability.
• Build and optimize integrations spanning batch, incremental, event-driven, and near-real-time workflows across APIs, databases, files, ServiceNow, CMDB, observability, telemetry, and enterprise platforms.
• Develop efficient, maintainable Python and SQL for pipelines, orchestration, transformations, analytics layers, APIs, automation, and reusable engineering frameworks.
• Establish trusted data products using automated controls for completeness, accuracy, freshness, reconciliation, monitoring, alerting, and exception handling.
• Advance AI-ready architectures that prepare structured and unstructured data for Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases, semantic search, agents, and MCP-enabled solutions.
• Design retrieval and context patterns including retrieval, embedding, indexing, enrichment, and context management to improve usability, discoverability, and responsible AI outcomes.
• Drive operational excellence with CI/CD, Infrastructure as Code, automated testing, source control, code reviews, runbooks, production monitoring, and resilient deployment practices.
• Apply security and privacy requirements across the data lifecycle, including identity and access control, encryption, retention, data classification, and responsible AI requirements.
• Collaborate with stakeholders, mentor engineers, promote reusable standards, and evaluate emerging technologies to improve scalability, automation, cost, and business value.

Minimum qualifications

• At least 18 years of age and authorization to work in the United States.
• High school diploma, GED, or high school equivalency.
• 7+ years of IT experience, including 4+ years hands-on designing, developing, and supporting enterprise data pipelines, data platforms, lakehouses, or analytical data solutions.
• Demonstrated experience designing and implementing cloud-based data lake, data warehouse, or lakehouse architectures using AWS.
• Hands-on experience with Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, AWS EventBridge, AWS Lake Formation, AWS IAM, and related data services.
• Advanced proficiency in Python and SQL for ingestion, transformation, automation, analysis, data services, and performance optimization.
• Experience designing scalable ETL and ELT pipelines integrating APIs, databases, structured files, semi-structured content, and cloud services.
• Strong data modeling experience across dimensional, relational, normalized, denormalized, and analytics-oriented approaches, including layered raw, curated, published, and analytics architectures.
• Strong understanding of incremental ingestion and patterns such as change-data capture, watermarking, deduplication, record versioning, historical tracking, late-arriving data, schema evolution, and reconciliation.
• Experience with metadata, lineage, cataloging, source-to-target mapping, data-quality testing, freshness monitoring, logging, alerting, and exception handling.
• Working knowledge of Amazon SageMaker, Amazon Bedrock, foundation models, LLM application architectures, RAG, embeddings, vector search, semantic retrieval, and AI-ready data preparation.
• Experience preparing structured and unstructured data through parsing, chunking, enrichment, embedding generation, indexing, access filtering, and context management.
• Strong understanding of data governance, data classification, privacy, encryption, retention, identity, authorization, and secure handling of personal and sensitive information.
• Experience with Amazon Redshift development concepts including SQL optimization, workload performance, distribution and sort strategies, materialized views, stored procedures, access controls, and columnar formats such as Parquet.
• Experience with Git, automated testing, CI/CD, and Infrastructure as Code such as AWS CDK or CloudFormation.
• Ability to communicate complex technical concepts, data limitations, architecture decisions, AI data requirements, and implementation plans to technical and business audiences.
• Demonstrated ability to lead technical design, mentor engineers, conduct code and architecture reviews, establish reusable standards, and manage work across concurrent Agile initiatives.
• Consistent commitment to safety, security, responsible AI, operational reliability, and protection of personal and sensitive data.

Preferred qualifications

• Relevant AWS certification such as AWS Certified Data Engineer, AWS Certified Solutions Architect, AWS Certified Machine Learning Engineer, or another AWS credential.
• Hands-on experience with Amazon SageMaker, Amazon Bedrock (including Bedrock Knowledge Bases and Bedrock Agents or AgentCore), and Amazon OpenSearch Service, plus foundation and embedding models for production generative AI architectures.
• Experience implementing enterprise RAG, semantic search, hybrid retrieval, reranking, retrieval evaluation, grounding, citations, and AI-powered knowledge repositories.
• Experience designing or integrating vector databases and vector-capable platforms such as Amazon OpenSearch Service, PostgreSQL with pgvector, or comparable technologies.
• Experience with prompt engineering, model evaluation, guardrails, hallucination reduction, responsible AI controls, observability, and secure AI deployment patterns.
• Deep knowledge of ServiceNow data including APIs, table structures, reference fields, ITSM processes, configuration management data, and CMDB relationships.
• Experience integrating observability and operational platforms such as Dynatrace, PagerDuty, Amazon CloudWatch, ServiceNow, Sumo Logic, or similar services.
• Experience with OAuth 2.0 and OpenID Connect, identity federation, role-based access controls, and document- or row-level authorization for data and AI retrieval.
• Experience using automated data testing and quality frameworks such as AWS Glue Data Quality, Great Expectations, dbt tests, or comparable technologies.
• Demonstrated history leading data modernization, cloud migration, operational intelligence, platform engineering, generative AI, or AI-enablement initiatives.

Technologies you may work with

• AWS, Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases
• Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, AWS EventBridge, AWS Lake Formation, AWS IAM
• Python, SQL, ServiceNow, CMDB, Amazon CloudWatch, Dynatrace, PagerDuty, Sumo Logic
• Amazon OpenSearch Service, PostgreSQL, pgvector, OpenID Connect, OAuth 2.0
• CI/CD, Infrastructure as Code, AWS CDK, CloudFormation, Git
• AWS Glue Data Quality, Great Expectations, dbt, Parquet, ETL, ELT, Lakehouse, Lake Formation

Benefits

• Competitive salary, industry-leading profit sharing program, and performance incentives
• 401(k) with generous company contributions up to 9%
• Paid time off including vacation, holidays, paid personal time, maternity and parental leave
• Comprehensive health benefits including medical, dental, vision, short/long term disability, and life benefits
• Family care assistance including fertility support, surrogacy and adoption assistance, lactation support, subsidized back-up care, and programs supporting loved ones
• Holistic wellbeing programs including employee assistance program access, free financial coaching, and resources supporting mental health
• Domestic and International space-available flight privileges for employees and eligible family members
• Career development programs aligned to long-term goals
• World-wide partnerships focused on community service, sustainability, and reducing carbon footprint
• Business Resource Groups to promote inclusion and help implement strategies
• Recognition rewards and awards through Unstoppable Together
• Access to over 500 discounts and specialty savings through Deltaperks, including insurance, legal services, childcare, and other voluntary benefits