Deloitte is hiring an AI Engineer Consultant through its Project Delivery Talent Model to support AIOps and MLOps engineering efforts in an Azure + Databricks environment. This onsite role in St. Louis, MO centers on building reliable ML platforms, monitoring production workloads, and improving deployment resilience across the full model lifecycle.
In this position, you will oversee operational monitoring for Databricks jobs and clusters, maintain CI/CD delivery with Azure DevOps, and manage core ML platform components such as MLflow, Delta Lake, drift detection, and observability. You will also contribute to governance and security using Unity Catalog and support deployment workflows through Azure ML endpoints and Databricks model serving.
Responsibilities
- Monitor Databricks jobs and clusters by tracking job run status, cluster utilization, and autoscaling behavior through Databricks Jobs UI and Azure Monitor, addressing failed or delayed pipeline runs proactively.
- Build and maintain CI/CD pipelines using Azure DevOps with YAML to deploy notebooks, ML models, and Databricks workflows across dev, staging, and prod using Databricks Repos and Git integration.
- Operate MLflow for model lifecycle management, including experiment tracking, registering models in the MLflow Model Registry, managing transitions between staging and production, and maintaining versioning and lineage.
- Maintain Delta Lake pipelines with a focus on data quality, schema enforcement, and ACID compliance across bronze/silver/gold layers feeding training and inference workloads.
- Monitor model performance and detect drift by setting up automated drift detection (data and concept drift) via Databricks native monitoring or custom Azure ML integration, triggering retraining when thresholds are exceeded.
- Manage compute and cost optimization by configuring and right-sizing Databricks clusters (job clusters vs all-purpose), using autoscaling and spot instances, and reviewing Azure cost management dashboards to control spend.
- Implement observability using Azure Monitor and Log Analytics, setting up end-to-end logging and alerting across Databricks, Azure ML, and downstream services using Application Insights and Log Analytics workspaces.
- Support security, access, and governance by configuring Unity Catalog, managing service principals, handling secrets via Azure Key Vault, and administering RBAC across workspaces.
- Collaborate on model deployment via Azure ML endpoints, deploying models as real-time or batch endpoints using Azure ML Managed Endpoints or Databricks Model Serving with an emphasis on scalability and low-latency inference.
- Provide on-call support and incident response by troubleshooting pipeline failures, cluster crashes, or endpoint downtime, conducting root cause analysis and post-incident reviews to strengthen pipeline resilience.
Requirements
- 3-6+ years of experience in DevOps/MLOps/Data Engineering, including 1-2 years hands-on with Databricks and Azure.
- Strong proficiency in Python and/or Scala, plus SQL for data transformation and querying.
- Hands-on experience with Databricks Jobs, Workflows, Unity Catalog, Delta Lake, and Databricks Model Serving.
- Proficiency in the Azure ecosystem, including Azure DevOps, Azure ML, Azure Monitor, Azure Key Vault, and Azure Data Factory.
- Experience with MLflow including experiment tracking and model registry management.
- Working knowledge of CI/CD practices and Infrastructure as Code using Terraform or ARM/Bicep templates.
- Understanding of the ML lifecycle, including training, validation, deployment, monitoring, and retraining.
Education
- Bachelor's degree, preferably in Computer Sciences, Information Technology, Computer Engineering, or a related IT discipline.
Required
- Limited immigration sponsorship may be available.
- Ability to travel 10% on average, including possible overnight travel, based on work with clients and industries/sectors served.
Preferred
- Familiarity with containerization (Docker) and orchestration (such as Kubernetes, if applicable).
- Analytical ability to manage multiple projects and prioritize work into manageable deliverables.
- Ability to work independently or with minimum supervision.
- Excellent written and communication skills.
- Ability to deliver technical demonstrations.
Core technologies: Azure, Databricks, Azure Monitor, Databricks Jobs UI, Azure DevOps, YAML, Databricks Repos, Git, MLflow, MLflow Model Registry, Delta Lake, Databricks native monitoring, Azure ML, Databricks clusters, autoscaling, spot instances, Application Insights, Log Analytics workspaces, Unity Catalog, service principals, Azure Key Vault, RBAC, Azure ML Managed Endpoints, Databricks Model Serving, Python, Scala, SQL, CI/CD, Infrastructure as Code, Terraform, ARM, Bicep, Azure Data Factory, Azure ML endpoints.
