Senior Data Engineer

The Senior Data Engineer will focus on delivering end-to-end data solutions and ETL/ELT pipelines leveraging Azure Data Factory and Databricks, with responsibilities spanning design, development, optimization, governance, and mentoring within the Project Delivery Model.

Location

Miami, FL onsite

Salary

USD 95,000 - 150,000 per year

Responsibilities

• Maintain ongoing communication with Engagement Managers (Directors), project teams, and representatives from multiple functional and technical groups, escalating issues that require management attention
• Design, build, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks
• Develop and fine-tune PySpark and Spark SQL notebooks for large-scale data transformations
• Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog
• Lead design discussions with client architects and other counterparts
• Collaborate with diverse teams to establish data contracts and agreed-upon schemas
• Head the design and optimization of high-volume data pipelines
• Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning
• Drive performance improvements through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
• Perform root cause analyses of production incidents and implement durable fixes
• Mentor junior and mid-level engineers via code reviews and pair programming
• Evaluate new technologies and advise on adoption, including DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server
• Experience with Azure components such as ADF, ADLS Gen2, Key Vault, and Azure Monitor
• Hands-on experience with Databricks including Delta Lake, Unity Catalog, and Workflows
• Familiarity with Apache Airflow
• Git and Azure DevOps
• Deep knowledge of Spark internals, including DAG optimization, spill analysis, and skew handling
• Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O
• Unity Catalog governance covering row and column security, external locations, and system tables
• Infrastructure as Code using Terraform and Azure ARM templates
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel approximately 10 percent on average, depending on client assignments

Technologies

• Python
• PySpark
• Spark SQL
• SQL Server
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2
• Key Vault
• Azure Monitor
• Databricks
• Delta Lake
• Unity Catalog
• Workflows
• Apache Airflow
• Git
• Azure DevOps
• Terraform
• Azure ARM templates
• Delta Live Tables (DLT)
• Auto Loader
• Serverless Compute
• Event Hubs
• DABs