EngineerJobs.io
← Back to all jobs

Job Description

Capgemini is seeking a Senior Databricks Data Engineer to design and optimize scalable batch and real-time data solutions on the Databricks lakehouse. This onsite role focuses on building governed data pipelines that support analytics, machine learning, and generative AI.

Role Summary

In this position, you will develop ETL/ELT pipelines, enforce data governance and security with Unity Catalog, and support operationalization of machine learning workloads. Work includes building lakehouse architectures, improving pipeline reliability and performance, and preparing datasets for ML and generative AI use cases.

Key Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
  • Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
  • Implement data quality checks, monitoring, and pipeline optimization to ensure reliability and performance.
  • Design and manage Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
  • Develop and maintain Delta Lake tables with a focus on optimal query performance.
  • Conduct performance tuning using Databricks best practices such as Photon Engine, Liquid Clustering, and partitioning strategies.
  • Configure and administer Unity Catalog for enterprise-wide data governance.
  • Implement security controls across catalog, schema, table, row-level, and column-level scopes.
  • Enable data lineage, auditing, and compliance requirements across the data platform.
  • Develop and maintain workflows using Databricks Workflows.
  • Manage deployments using Databricks Asset Bundles (DABs).
  • Implement CI/CD pipelines leveraging GitHub, GitLab, and DevOps best practices.
  • Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies.
  • Prepare and curate datasets for machine learning and generative AI applications.
  • Support feature engineering and feature store implementations.
  • Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
  • Collaborate with Data Science and AI teams to operationalize ML workloads.

Required Qualifications

  • 5-8 years of experience in Data Engineering.
  • Strong proficiency in Python (PySpark) and SQL.
  • Hands-on expertise in the Databricks Lakehouse Platform.
  • Hands-on experience with Delta Lake and Unity Catalog.
  • Hands-on experience with Databricks Workflows.
  • Hands-on experience with Structured Streaming and Auto Loader.
  • Experience designing and supporting large-scale enterprise data platforms.
  • Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
  • Experience implementing CI/CD pipelines, version control, and automated testing practices.
  • Hands-on experience with Git, GitHub/GitLab, Terraform, and pytest.
  • Experience with at least one major cloud platform: Azure, GCP, or AWS.
  • Ability to work onsite at the customer location in Troy, MI for a minimum of four (4) days per week.

Tools and Technologies

PySpark, SQL, Delta Lake, Auto Loader, Databricks Lakehouse, Medallion Architecture, Photon Engine, Liquid Clustering, Unity Catalog, Databricks Workflows, Databricks Asset Bundles (DABs), GitHub, GitLab, DevOps, Terraform, Infrastructure-as-Code, pytest, Azure, GCP, AWS, Structured Streaming, Git, LLM, vector search, feature store.

Compensation

USD 89,400 - 166,398 per year.

Benefits

  • Paid time off based on employee grade (A-F), including vacation (12-25 days depending on grade), company paid holidays, personal days, and sick leave.
  • Medical, dental, and vision coverage (or provincial healthcare coordination in Canada).
  • Retirement savings plans (for example, 401(k) in the U.S., RRSP in Canada).
  • Life and disability insurance.
  • Employee assistance programs.
  • Other benefits as provided by local policy and eligibility.

Work Location

Troy, MI (Onsite Requirement). Candidates must be able to work onsite at the customer location in Troy, MI for a minimum of four (4) days per week.

Similar Jobs