AI Data Engineer

The AI Data Engineer will build and run the data-engineering foundation that powers HHMI’s AI Accelerator. The work focuses on transforming institutional content into governed, AI-ready datasets, with pipelines and infrastructure designed for reliable retrieval and downstream AI consumption. The role operates under the design authority of the Principal Knowledge & Data Architect, turning established patterns into production-ready systems.

Role focus

• Build AI-facing data pipelines that ingest from the landing zone provided by Operations Capabilities, transform data through bronze, silver, and gold layers, and deliver governed, AI-ready outputs for downstream use.
• Implement the medallion architecture using design patterns from the K&D Architect.
• Create Delta Lake table structures with partitioning, optimization, and schema evolution in mind.

Workflow orchestration and operations

• Own the workflow-orchestration framework, including Databricks Workflows and Delta Live Tables.
• Apply retry policies, alerting routes, run history, and cost tags to keep pipeline execution observable and dependable.

Governance and access control

• Implement governance patterns using Unity Catalog structure.
• Apply sensitivity classification, access control, and audit capabilities for AI-facing data assets.

Retrieval-supporting infrastructure

• Build and operate embedding pipelines and vector store maintenance.
• Support reindexing when models upgrade.
• Maintain retrieval evaluation frameworks to support quality and effectiveness.

Partnerships and data contracts

• Partner with Operations Capabilities on source-system contracts, including defining schema, cadence, SLA, and quality thresholds for what the AI Fabric consumes at the landing zone.
• Own the platform-side of that contract.
• Design and operate data quality processes, including data-quality checks, freshness monitoring, drift detection, and alerting that surfaces issues before AI users encounter them.

Reference patterns and team enablement

• Contribute to and consume a reference-pattern library that includes reusable pipeline patterns, code templates, and standards in the shared platform layer.
• Support AI developer velocity by providing the data engineering needed when AI Developers integrate into product teams and require specific datasets.

Minimum qualifications

• At least 4 years of hands-on production data engineering, including designing, building, and operating production data pipelines.
• Strong depth with Databricks and Spark technologies, including Delta Lake, medallion architecture, Delta Live Tables, Workflows, Databricks SQL, and Unity Catalog.
• Fluency in Python and SQL, including PySpark and ETL patterns, with SQL that runs at scale.
• Working familiarity with Git, CI/CD for data pipelines, and infrastructure-as-code using Terraform.
• AI-adjacent data engineering experience building the data foundation for AI use cases, including embedding pipelines, vector stores, chunking strategies, and retrieval evaluation.
• Practical workflow orchestration experience in Databricks Workflows or Airflow in production, including retry semantics, dependency management, and failure handling.
• Data quality and observability experience with tools such as Great Expectations or Databricks data-quality monitors (or equivalent).
• Governance discipline that includes Unity Catalog structures and sensitivity classification, with access control and audit considerations built from the first commit.
• AWS foundations at the level required for Databricks-on-AWS work, including IAM, S3, and KMS.
• Communication ability to work with the K&D Architect, AI Developers, and Operations Capabilities, including explaining data-engineering trade-offs to non-engineers.
• Bachelor’s degree or equivalent, plus at least 4 years of hands-on data engineering with meaningful exposure to AI or knowledge-management use cases.

Technologies

• Databricks Workflows, Delta Live Tables, Delta Lake, Databricks SQL, Unity Catalog
• Python, PySpark, SQL
• Git, CI/CD, Terraform
• Databricks data-quality monitors, Great Expectations
• Embedding pipelines, vector store maintenance, retrieval evaluation frameworks
• Databricks-on-AWS, IAM, S3, KMS, Airflow

Hybrid location and reporting

• Location: Chevy Chase, MD (hybrid)
• Schedule: three days per week in-person at HHMI’s offices in Chevy Chase, Maryland
• Reports to: Director of AI Enablement

Compensation and benefits

• Salary range: USD 128,816 - 161,021 per year (annual)
• Competitive pay
• Exceptional health benefits
• Retirement plans
• Time off
• Recognition and wellness programs

Equal opportunity and eligibility

• HHMI is an equal opportunity employer and uses E-Verify to confirm identity and employment eligibility of new hires.

Additional practical notes

• HHMI cannot sponsor a visa for this position at this time.
• This job description outlines principal duties and should not be considered exhaustive; essential duties and responsibilities are treated as essential functions unless otherwise noted.

Not this role

• Not a Data Platform Architect: platform design authority for Databricks-on-AWS sits with Platform Architects; this role builds AI-facing data pipelines on that platform.
• Not the Knowledge & Data Architect: the K&D Architect owns the knowledge and retrieval design; this role implements against that direction and focuses on hands-on delivery.
• Not an AI/ML engineer: AI Developers own model access, orchestration, and application delivery; this role supplies the data layer that AI Developers consume.
• Not a warehouse or BI data engineer: traditional analytics warehouses and BI-facing pipelines belong to Operations Capabilities; this role is AI-facing.

Nice to have

• Experience with knowledge graphs (Neo4j or comparable), entity resolution, or semantic data models.
• Experience with the modern data stack alongside Databricks-native tooling.
• Familiarity with LLM-based extraction, chunking, and evaluation frameworks.
• Background in research, academic, or mission-driven institutional environments.
• Experience with cross-platform data engineering such as Snowflake or BigQuery.

Physical requirements

• Ability to remain seated or standing for extended periods and perform reaching, grasping, and keyboard-based dexterity.
• Communication using spoken word and ability to see and hear within normal parameters.
• Mobility and ability to move materials weighing up to several pounds (for example, a laptop computer or tablet).