Senior Data Engineer (Azure & Databricks)

Senior Data Engineer role focused on Azure Databricks, Delta Lake, and SQL to design and scale a medallion-based data platform using PySpark and Databricks tooling in Bloomington, MN on a hybrid in-office schedule (Tue/Thu) for a 6+ month contract; US work authorization is required.

Responsibilities

• Design, develop, and optimize data pipelines in Azure Databricks using PySpark and SQL, applying Delta Lake and Unity Catalog best practices.
• Build modular, reusable libraries and utilities within Databricks to accelerate development and standardize workflows.
• Implement Medallion architecture (Bronze, Silver, Gold layers) for scalable, governed data zones.
• Integrate external data sources via REST APIs, SFTP file delivery, and SQL Server Managed Instance, implementing validation, logging, and schema enforcement.
• Utilize parameter-driven jobs and manage compute using Spark clusters and Databricks serverless.
• Collaborate with data analytics teams and business stakeholders to understand requirements and deliver analytics-ready datasets.
• Monitor and troubleshoot Azure Data Factory pipelines (jobs, triggers, activities, data flows) to identify and resolve job failures and data issues.
• Automate deployments and manage code using Azure DevOps for CI/CD, version control, and environment management.
• Contribute to documentation and architecture and drive continuous improvement of data engineering practices.
• Support the data platform for AI and machine learning initiatives and ensure readiness for advanced analytics.

Requirements

• Strong expertise with Azure Databricks, including PySpark, Delta Lake, Unity Catalog, and the ability to build reusable libraries, utility notebooks, and parameterized jobs.
• Advanced SQL skills with experience in Azure SQL Database and/or SQL Server Managed Instance.
• Experience designing, troubleshooting, and supporting data pipelines using Azure Data Factory.
• Proven ability to integrate external data sources, including REST APIs and SFTP.
• Working knowledge of Azure DevOps for CI/CD, version control, and parameterized deployments.
• Strong collaboration and communication skills, demonstrated by partnering with data analytics teams and business stakeholders.
• Interest or experience in preparing data platforms to support AI and machine learning initiatives.