Lead Databricks Data Engineer
Manager
Apache Spark
Artificial Intelligence
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Management
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Dlt
Databricks Lakehouse
Databricks Workflows
ETL
Informatica
Pyspark
Software Development
Spark
SQL
Job Description
EXL Service is seeking a Lead Databricks Data Engineer in New York, NY to lead the design, development, modernization, and optimization of enterprise-scale data platforms built on Databricks Lakehouse architecture. This role combines technical leadership with governance, delivery management, and hands-on engineering across large-volume batch and real-time workloads.
Key Responsibilities
- Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
- Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
- Define enterprise data models, ingestion frameworks, and transformation strategies.
- Lead implementation of Delta Lake, Unity Catalog, and enterprise data governance frameworks.
- Build and optimize scalable Spark and PySpark solutions for large-volume datasets.
- Drive cloud data platform modernization and migration initiatives.
- Serve as the technical lead for Databricks and Data Engineering programs.
- Establish engineering best practices, coding standards, and architecture guidelines.
- Perform architecture reviews, code reviews, and solution assessments.
- Mentor and guide Data Engineers, Technical Leads, and junior team members.
- Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
- Design cloud-native data solutions on Azure, AWS, or GCP.
- Implement secure, scalable, and resilient data architectures.
- Optimize costs, performance, reliability, and availability of data platforms.
- Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
- Define monitoring, observability, and operational excellence practices.
- Establish data governance, lineage, cataloging, and security controls.
- Ensure compliance with organizational and regulatory requirements.
- Implement data quality frameworks and monitoring solutions.
- Collaborate with security and compliance teams to enforce data protection standards.
- Partner with business leaders to understand strategic data requirements and translate them into technical solutions.
- Drive delivery planning, risk management, and resource coordination.
- Present architecture decisions and technical recommendations to leadership teams.
Required Skills and Technologies
- Databricks Lakehouse Platform, Databricks Workflows, Unity Catalog, Delta Lake, Structured Streaming
- Databricks SQL, Delta Live Tables (DLT), Databricks Asset Bundles
- Apache Spark, PySpark, Spark SQL
- Kafka, Real-Time Streaming Architectures
- Azure Databricks, Azure Data Factory (ADF), ADLS Gen2, Azure Synapse Analytics, Azure Key Vault
- Event Hub, S3, EMR, Glue, Redshift, Lambda
- IAM, BigQuery, Cloud Storage, Dataflow
- Pub/Sub, Composer, Python, SQL
- Scala (Preferred), Shell Scripting, Git
- Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, Kubernetes
- Data governance and quality: Data Governance Frameworks, Great Expectations or Deequ
- Data warehousing and modeling: Dimensional Modeling, Star & Snowflake Schema, Data Vault, Master Data Management
- Additional exposure: MLOps and AI Engineering, Snowflake and Databricks integration, Microsoft Fabric exposure, Data Mesh implementation
- Governance tools: Collibra, Alation, or Informatica
- Architecture: TOGAF or Cloud Architecture certifications are a plus
Experience and Education
- Experience: 10-12 years in total (10-12 Years specified) and 12-15 years of overall IT experience.
- Databricks experience: 6+ years hands-on.
- Spark/PySpark experience: 7+ years.
- Education: Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
- Certifications: Databricks Certified Data Engineer Professional, Azure Data Engineer Associate, AWS Data Analytics Specialty, and Google Professional Data Engineer Certification.
Employment Details
- Location: New York, NY (onsite)
- Employment type: Full-Time
- Compensation: USD 15 - 18 per hourly
- Technology Tower: Data & AI
- Practice Area: Data Engineering
- Primary Skill: Databricks Lakehouse Engineering
- Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps
Similar Jobs
