Databricks Engineer & AI Enablement Architect - Data Engineer III
Manager
Artificial Intelligence
Big Data
Bigdata
CI/CD
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Lineage
Data Management
Data Pipeline
Data Platform
Data Processing
Data Science
Data Science Ops
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Lakehouse
Delta Lake
DevOps
Devops Tools
ETL
Informatica
Information Technology (IT)
Integration
Service Principals
Software Development
Software Engineering
Spark
SQL
Job Description
Own secure Databricks platform governance and AI enablement for healthcare data engineering and GenAI/LLM production readiness.
- Lead governance, architecture, reliability, and day-to-day operation of secure, scalable Databricks environments supporting the Data Aggregation Contract (CDAC)
- Set up workspace governance and Unity Catalog structures, including access controls, service principals, and supporting policies to protect PHI and sensitive healthcare data
- Define medallion architecture standards and deliver reusable ingestion and transformation frameworks for CMS claims plus clinical, provider, quality, and oncology data
- Strengthen data quality and production reliability using validation, reconciliation, lineage, observability, SLA monitoring, dashboards, and proactive alerts
- Establish CI/CD standards, including review of code, data models, and pipeline designs
- Evaluate GenAI and LLM use cases for production readiness as part of delivery and operational practices
Requirements
- Bachelor’s degree
- Ability to obtain and maintain the required clearance for this role
- 4+ years of Databricks engineering experience
- 4+ years with Python, PySpark, SQL, Delta Lake, and Medallion Architecture
- 2+ years of experience in databricks architecture
- Ability to travel 10-15%, on average, based on work and client/industry needs
- Must be legally authorized to work in the United States without employer sponsorship, now or in the future
Technologies
- Databricks
- Unity Catalog
- Python
- PySpark
- SQL
- Delta Lake
- Medallion Architecture
- CI/CD
- GenAI
- LLM
- Service principals
Preferred
- Experience supporting federal healthcare clients, or working with Centers for Medicare & Medicaid Services (CMS) claims, Medicaid or Medicare claims, clinical and provider data, quality measures, or longitudinal patient data
- Databricks certifications such as Certified Data Engineer Professional, Data Analyst Associate, Machine Learning Professional, or Generative AI Engineer Associate
- Experience administering Databricks platforms, including Unity Catalog, workspace administration, cluster policies, service principals, secrets management, and role-based access controls
- Experience implementing Databricks Asset Bundles and Git-based CI/CD across development, test, and production environments
Team
- Part of Deloitte’s Government & Public Services (GPS) practice serving federal, state, and local government clients and public higher education institutions
- Deloitte’s AI & Data offering spans solutions for designing, developing, and operating Data and AI platforms, products, insights, and services
- Focus on helping clients innovate, enhance, and operate data, AI, and analytics capabilities with maturity and scale
Location
- Washington, DC (onsite)
Compensation
- Estimated wage range: $113,000 to $188,400 per year
- May be eligible for a discretionary annual incentive program (award depends on factors including individual and organizational performance)