Senior Data Engineer

Join Deloitte as a Senior Data Engineer in an on-site role based in Arlington Heights, IL. This position offers a competitive salary range of USD 95,000 – 150,000 per year and the opportunity to design, build, and optimize ETL and ELT pipelines on Azure Databricks. You will collaborate with client and engagement teams and help define data engineering standards within Deloitte's Project Delivery Model. The role supports professional growth through mentoring of junior engineers and active cross-team collaboration.

Responsibilities

• Maintain ongoing communication with Engagement Managers, project team members, and stakeholders across functional and technical groups, escalating issues to engagement management when necessary.
• Design, develop, and optimize end-to-end ETL and ELT pipelines using Azure Data Factory and Databricks.
• Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
• Architect data solutions across development, UAT, and production environments using Unity Catalog.
• Lead design discussions with client architects and other counterpart teams.
• Collaborate with multiple teams to establish data contracts and achieve schema agreements.
• Direct the design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Improve performance through adaptive query execution tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
• Perform root cause analyses for production incidents and implement lasting fixes.
• Mentor junior and mid-level engineers through code reviews and pair programming.
• Evaluate new technologies and recommend adoption of items such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs.

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server.
• Experience with Azure components including Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
• Familiarity with Databricks, Delta Lake, Unity Catalog, and Workflows.
• Experience with Apache Airflow.
• Hands-on work with Git or Azure DevOps.
• Solid understanding of Spark internals such as DAG optimization, spill analysis, and skew handling.
• Advanced Delta Lake features including time travel, deletion vectors, and predictive I/O.
• Unity Catalog governance covering row and column security, external locations, and system tables.
• Infrastructure as code experience with Terraform and Azure ARM templates.
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT field, or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel up to 10 percent depending on client work and engagements.

Technologies

• Python, PySpark, Spark SQL, SQL Server
• Azure Data Factory (ADF), ADLS Gen2, Key Vault, Azure Monitor
• Databricks, Delta Lake, Unity Catalog, Workflows
• Apache Airflow
• Git, Azure DevOps
• Terraform, Azure ARM templates
• Deep Spark internals (DAG optimization, spill analysis, skew handling)
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance (row/column security, external locations, system tables)
• Auto Loader, Serverless Compute, DABs, DLT, Event Hubs

Accommodation and accessibility

Information for applicants with a need for accommodation: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applicants.html