Lead AI Engineer

Vytwo is hiring a Lead AI Engineer in Dallas, TX to help build and run production-ready AI applications. The role focuses on designing end-to-end data and AI pipelines and deploying ML and Generative AI workloads on Azure Databricks and related Azure infrastructure.

You will take models from experimentation to production by implementing robust ETL/ELT workflows, CI/CD for Databricks, and the monitoring and governance needed for reliable operations.

What you’ll do

• Design and build end-to-end data and AI pipelines using Azure Databricks.
• Develop resilient ETL/ELT workflows with Python (PySpark) and SQL.
• Create CI/CD pipelines for Databricks deployments, including jobs, notebooks, and workflows.
• Integrate Databricks with Azure services such as Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
• Optimize Databricks workloads for performance, cost, and reliability.
• Build reusable, modular code for pipeline and AI components.
• Work with data scientists and platform teams to move models from experimentation to production.
• Implement logging, monitoring, and error handling for production pipelines.
• Develop and deploy ML and Generative AI models including LLMs, embeddings, and RAG pipelines for NLP, computer vision, and predictive analytics.
• Fine-tune LLMs using LoRA/QLoRA and integrate with Azure OpenAI or Hugging Face models.
• Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
• Apply responsible AI practices, including bias detection and model governance.

What you’ll need

• Azure Databricks experience with jobs, workflows, clusters, and Unity Catalog (preferred).
• Strong Python experience with a PySpark-heavy focus.
• SQL skills including complex joins, window functions, and analytical querying.
• Hands-on experience with Azure cloud services and concepts including ADLS Gen2, ADF, Key Vault, and IAM.
• Experience with pipeline orchestration and deployment including CI/CD and environment promotion.
• Azure DevOps experience.
• Strong understanding of ML lifecycle and MLOps best practices.
• Experience deploying models using MLflow or similar frameworks.

Tools and technologies

• Azure Databricks, Databricks workflows, Databricks jobs, Databricks notebooks, Unity Catalog
• Azure cloud infrastructure, Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions
• Python, PySpark, SQL, ETL/ELT
• CI/CD pipelines, logging, monitoring
• MLflow, LoRA, QLoRA, Hugging Face models
• FAISS, Azure Cognitive Search
• NLP, computer vision, predictive analytics, vector search, RAG pipelines
• Azure DevOps

Good to have (strong advantage)

• Experience with ML and Generative AI workloads on Databricks.
• RAG, embeddings, or inference pipeline experience.
• Experience with Terraform / ARM / Bicep.
• Databricks Asset Bundles experience.
• Experience with Airflow or ADF orchestration.
• Production monitoring and cost optimization experience.
• Knowledge of LangChain or similar AI application development frameworks.
• Experience with Azure AI services including Azure Machine Learning and Azure Cognitive Services.