Databricks Data Engineer, Lead

Booz Allen Hamilton is seeking a Lead Databricks Data Engineer in Bethesda, Maryland, to support the design, build, and ongoing support of secure, scalable data pipelines. This role combines hands-on development in Databricks with technical leadership, mentorship, and platform oversight across Databricks and AWS.

Role focus

• Design, build, and maintain robust data pipelines within Databricks, contributing independently while providing technical leadership and mentorship to the team
• Create and optimize end-to-end data flows, including extract, transform, load (ETL) processes
• Design data models and improve performance and scalability for data processing and analytics
• Ensure data security and compliance throughout pipeline and platform operations
• Collaborate with cross-functional teams to support client outcomes and continuous improvement
• Mentor a multi-disciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment
• Oversee assessment, design, building, and maintenance of scalable platforms for clients

Experience and qualifications

• 8+ years designing, developing, deploying, and managing ETL processes and data pipelines with comprehensive logging and robust error handling to troubleshoot and resolve issues
• 3+ years building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools with Delta Lake, Databricks Delta, and Unity Catalog
• 3+ years writing Python and PySpark code for ETL scripts and custom transformations
• Experience with data modeling, including choosing between dimensional vs relation data models within a medallion architecture using Bronze, Silver, and Gold
• Experience with AWS services including AWS Glue, Amazon S3, and Amazon RDS
• Knowledge of ETL performance optimization techniques such as partitioning, bucketing, and caching, plus troubleshooting to improve query performance
• Ability to design and implement data models from schema definition through optimization for efficient queries and analytics
• Public Trust
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology

Technologies

• Databricks, AWS, Databricks notebooks, clusters, job orchestration tools
• Delta Lake, Databricks Delta, Unity Catalog
• Python, PySpark
• AWS Glue, Amazon S3, Amazon RDS

Benefits

• Health, life, disability, financial, and retirement benefits
• Paid leave
• Professional development and tuition assistance
• Work-life programs and dependent care
• Recognition awards program acknowledging employees for exceptional performance and demonstration of company values
• Full-time and part-time employees working at least 20 hours a week on a regular basis are eligible to participate in Booz Allen benefit programs

Additional information

• Location: Bethesda, MD (onsite)
• Work model: Onsite, with work primarily performed full-time at a customer facility
• Compensation: projected range of $77,600.00 to $176,000.00 (annualized USD)
• Applicants selected will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client
• Public Trust determination is required

Identity and interview policy

• Completing an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud
• Employees are expected to be on camera during interviews and assessments
• Booz Allen reserves the right to take your picture to verify identity and prevent fraud
• The use of AI or other tools to assist with responses during interviews is prohibited unless permission is explicitly provided

Nice to have

• Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch for serverless data processing architectures
• Experience with tools such as AWS Glue Data Catalog or Collibra for managing metadata, data lineage, and cataloging data assets
• Knowledge of data governance practices, data lineage, and compliance requirements (including GDPR or CCPA) supporting data quality, privacy, and security
• Ability to explain complex technical concepts to a non-technical audience, including architectural tradeoffs