Databricks Engineer - Data Engineer III

The Databricks Engineer role supports Deloitte’s AI & Engineering team by building and optimizing data pipelines on the Databricks Lakehouse platform, while also overseeing governance, security, and performance.

Location

Washington, DC (onsite)

Compensation

USD 107,700 - 179,500 per year

Role Overview

This position focuses on Databricks engineering work including pipeline development, application of Medallion architecture patterns, performance tuning for Databricks and SQL workloads, and administration of governance capabilities in Unity Catalog such as row-level and column-level security, data lineage, and data sharing. The role also includes establishing CI/CD and testing practices for Databricks workflows.

Responsibilities

• Develop and optimize data pipelines using Python, PySpark, and SQL within the Databricks Lakehouse platform
• Use Medallion architecture patterns to organize, transform, and manage data across data layers
• Conduct performance tuning for Databricks workloads and SQL queries
• Configure row-level and column-level security, and manage data lineage and sharing in Unity Catalog
• Implement CI/CD and testing processes for Databricks workflows

Required Qualifications

• Bachelor’s degree
• Ability to obtain and maintain the required clearance for the role
• 2+ years of Databricks engineering experience
• 4+ years experience with Python, PySpark, and SQL
• Ability to travel 10-15%, on average, depending on work needs and client or industry/sector coverage
• Must be legally authorized to work in the United States without employer sponsorship, now or at any time in the future

Technologies

• Python
• PySpark
• SQL
• Databricks Lakehouse
• Medallion architecture
• Unity Catalog
• CI/CD

Skills for Success

• Ability to work independently and collaborate as part of a team
• Effective written and verbal communication skills
• Meticulous attention to detail and quality of work product
• Ability to build and sustain professional relationships
• Ability to lead projects or workstreams
• Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
• Strong interpersonal skills and professional demeanor
• Ability to meet deadlines
• Ability to provide clear guidance to others

Preferred Qualifications

• Experience using Medallion architecture in the Databricks Lakehouse platform
• Experience configuring row-level and column-level security, and managing data lineage and data sharing in Unity Catalog
• Experience implementing CI/CD and testing processes for Databricks
• Experience with performance tuning in Databricks