Senior CE Data Engineer

The Senior CE Data Engineer will build and operationalize the Eli Lilly LillyDirect Covered Entity (CE) Pharmacy data platform on Databricks. The role focuses on governed, auditable data at scale, including medallion pipeline implementation, data contracts, identity crosswalk logic, and PHI isolation controls.

Key Responsibilities

• Design and develop scalable Databricks pipelines that implement the CE Architect’s canonical data models across the medallion layers (bronze to silver to gold), operating entirely within the CE trust boundary.
• Assess and apply Databricks capabilities and integration patterns, including Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, and ingestion connectors, selecting approaches based on performance, cost, and scalability constraints.
• Implement identity crosswalk and PHI classification logic as defined by the CE Architect, using systems and tokens including Reltio MDM, Auth0/Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, and Transcend. Ensure merge and split integrity through code.
• Build to ODCS data contract requirements by enforcing schema, data quality, freshness/SLA, and lineage expectations as pipeline logic rather than documentation.
• Implement row and column-level security, masking, and tokenization boundaries to ensure PHI isolation is enforced at the platform layer, working alongside the Policy-as-Code Engineer and their OPA/Rego policies.
• Own the end-to-end pipeline development lifecycle for CE data products, from requirements and prototyping through production deployment and ongoing maintenance, in collaboration with Analytics Engineers and the DataHub Product Owner.
• Create and maintain CI/CD pipelines for CE data, contract, policy, and agent artifacts, using GitHub Actions and a Git-based promotion flow from development to testing to production.
• Automate data ingestion and product creation to reduce manual pipeline upkeep and improve onboarding time for new CE data sources.
• Partner with the CE Data Architect on reference architecture and patterns, providing implementation feedback to keep solutions buildable and performant at scale.
• Build data pipelines that enable CE Skills and agents to consume governed data while ensuring PHI classification and consent travel through agentic consumption paths.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a similar degree.
• Experience in data engineering, specifically building production data pipelines and data products.
• Advanced SQL and Python, with hands-on Databricks and Unity Catalog experience (including notebooks, Delta Lake, catalog/schema/grants, and Workflows).
• Authorization to work in the United States on a full-time basis. Lilly does not provide support for or sponsor work authorization or visas for this role, including F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.

Technologies and Tools

Databricks, Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, ingestion connectors, Reltio MDM, Auth0, Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, Transcend, ODCS, OPA/Rego, GitHub Actions, Git, Python, SQL, Unity Catalog catalog/schema/grants, Workflows, RBAC, ABAC, row- and column-level security, tokenization, masking, CI/CD, PySpark, DLT, pytest, DQX, Great Expectations, Terraform, MDM, identity resolution, Jira, Agile/Scrum, HIPAA, PHI, tokenization/de-identification, AI Skills, agents.

Benefits

• Company bonus (depending, in part, on company and individual performance).
• Eligibility to participate in a company-sponsored 401(k).
• Pension.
• Vacation benefits.
• Eligibility for medical, dental, vision, and prescription drug benefits.
• Flexible benefits (for example, healthcare and/or dependent day care flexible spending accounts).
• Life insurance and death benefits.
• Certain time off and leave of absence benefits.
• Well-being benefits (for example, employee assistance program, fitness benefits, and employee clubs and activities).

Compensation

USD 63,000 - 149,600 per year.

Location and Work Arrangement

Indianapolis, IN (onsite).