Senior Azure Data Engineer
Senior
Azure
Azure Data Engineer
Azure Data Factory
Azure Data Lake
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Platform
Data Architecture
Data Engineer
Data Engineering
Data Factory
Data Factory Azure
Data Governance
Data Integration
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Security
Databricks
ETL
Microsoft Azure
Spark
SQL
Job Description
Yoh, A Day & Zimmermann Company is partnering with an enterprise-scale organization to add a Senior Azure Data Engineer to a modern lakehouse platform on Azure. The role focuses on designing, building, and optimizing scalable ETL/ELT workflows using Azure Databricks and a Medallion (Bronze/Silver/Gold) approach.
This position includes technical leadership across pipeline architecture, governance, security, and data reliability. You will work closely with analytics and business stakeholders, as well as data architecture, security, and infrastructure teams, to deliver trusted datasets, metrics, and KPIs.
Role and impact
- Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL.
- Develop ingestion workflows that bring data from multiple sources into Bronze/Silver/Gold layers (including Event Hub, APIs, Azure SQL, and Blob Storage).
- Implement best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance.
- Optimize pipeline performance and cost using techniques such as partitioning, caching, clustering, and Spark tuning.
- Integrate Databricks with enterprise Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps.
- Provide technical leadership through mentoring, code reviews, and architectural guidance.
Data governance, security, and reliability
- Implement secure access controls using managed identities, RBAC, and data protection practices such as masking.
- Collaborate with enterprise architecture and security teams to align with data privacy/security policies.
- Implement data quality checks, alerting, and lineage tracking using tools such as Great Expectations, Unity Catalog, and data observability platforms.
- Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines.
Required qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
- 5+ years of data engineering experience, including 3+ years with Azure Databricks / Apache Spark (PySpark).
- Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns.
- Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse.
- Strong coding skills in Python and SQL (Scala familiarity is a plus).
- Solid understanding of data modeling (OLTP/OLAP), star/snowflake schemas, and performance tuning.
- Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps.
- Databricks certification (for example, Databricks Certified Data Engineer Professional).
- Streaming/real-time pipeline experience including Structured Streaming and Event Hub (Kafka experience is listed as well).
- ML platform exposure such as MLflow or support for ML pipelines.
- Experience with reporting/consumption layers including Power BI, Purview, or Microsoft Fabric.
- Knowledge of governance/compliance standards including GDPR, HIPAA, and SOC2.
Technologies
- Azure Databricks, PySpark, Spark SQL
- Delta Lake, Unity Catalog
- Python, SQL, Scala
- ADLS Gen2, Azure Data Factory (ADF), Azure Synapse
- Event Hub, Key Vault, Azure DevOps
- Managed identities, RBAC, masking
- Great Expectations, data observability platforms
- Databricks notebooks, Databricks CLI, Terraform
- Structured Streaming, Kafka
- MLflow
- Power BI, Purview, Microsoft Fabric
Location and schedule
- Plano, TX (onsite)
- Hybrid schedule in Plano, TX
- Candidates must currently reside in the Dallas–Fort Worth (DFW) area
- Relocation is not available
- Candidates must be authorized to work in the United States without sponsorship now or in the future
Compensation
- USD 105,000 - 150,000 per year
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities