Senior Data Engineer - AI & Analytics Platform - Hybrid

The Senior Data Engineer for the AI analytics platform is a hybrid role focused on building and sustaining data pipelines and platform components on the Azure and Databricks stack to empower AI driven analytics. Key emphasis lies on data ingestion, architecture, security, and performance to support enterprise analytics solutions.

Overview

The position centers on developing scalable data engineering solutions within Allianz Life using Databricks Lakehouse technologies and Azure services. It requires collaboration with cross functional teams to deliver end to end data capabilities, ensure reliability, and promote best practices in data governance and security.

Responsibilities

• Design, implement, and maintain platform components including data connections, pipelines, Databricks Lakeflow jobs, and related GitHub Actions and Databricks Declarative Automation Bundle pipelines, along with monitoring frameworks.
• Create and sustain templated data engineering project repositories with a custom CLI, and manage Databricks DAB infrastructure as code repositories as needed.
• Ensure high availability, performance, observability, and data quality cost monitoring through proactive monitoring and incident response.
• Maintain comprehensive documentation for the internal platform and for external developers using the platform.
• Mentor junior data engineers, review pull requests, and ensure standardized, rigorous code quality at the platform level.
• Collaborate with internal teams, IT, data science, and analytics engineering to support end to end solution delivery, including provisioning Databricks DAB infrastructure for data ingestion and transformation, and supporting DevOps pipelines and processes.
• Provide support across the software development life cycle, including CI/CD, release management, and environment promotion.
• Offer ad hoc assistance for application deployment in the Databricks Apps space and Azure App Services; Kubernetes experience is preferred.
• Evaluate emerging Azure and Databricks services and AI/ML capabilities to foster innovation and continuous improvement.
• Contribute to data engineering platform governance covering data ethics, compliance, and security best practices.
• Mentor technical teams and promote architectural standards and reusable patterns.
• Utilize artificial intelligence tools and resources, including generative AI, to enhance platform capabilities.

Requirements

• 6+ years of experience in a mix of Data Engineering, DevOps, and application development roles.
• Experience configuring and maintaining Databricks Workspaces, ACLs, and other workspace artifacts.
• Proficiency with Databricks Lakehouse architecture, Spark, Unity Catalog, and Delta tables.
• Experience applying DevOps and SDLC practices in a regulated enterprise environment.
• Track record creating GitHub Actions workflows for code promotion across environments.
• Experience building data ingestion and data transformation pipelines in PySpark, preferably within Databricks.
• Must be legally authorized to work in the United States without requiring immigration sponsorship now or in the future, including holders of H-1B, H-4, L-1, L-2, TN, OPT, CPT, and other nonimmigrant visas.

Technologies

• Azure
• Databricks
• Databricks Lakeflow
• GitHub Actions
• Databricks Declarative Automation Bundle (DABs)
• Databricks DAB Infrastructure as Code
• Unity Catalog
• Delta tables
• Spark
• PySpark
• Databricks Workspaces
• ACLs
• Azure App Services
• Kubernetes

Location and Compensation

Location: Minneapolis, MN, with hybrid work arrangements. Compensation: USD 110,000 to 151,000 per year, dependent on experience and qualifications.

Benefits

• Comprehensive medical, dental, and vision plans
• Flexible spending accounts and health savings accounts
• Tuition reimbursement
• Student loan retirement program
• Generous annual paid leave
• Strong 401(k) company match
• Life insurance
• Onsite health center
• Child development center
• Fitness facility
• Convenience store
• Two cafeterias
• Employee Resource Groups