Consultant, Data Engineer

Infinitive Inc is hiring a Consultant, Data Engineer to build and maintain Databricks-based data pipelines for client teams. The role is hybrid in McLean, Virginia, with on-site work at the client location, and focuses on moving and transforming data into Delta Lake using Python, PySpark, and SQL.

Role Overview

As a Data Engineering consultant, you will implement batch and incremental ETL processes in Databricks, designing data flows that ingest from relational databases, APIs, flat files, and cloud storage. You will structure outputs using medallion architecture (bronze, silver, gold) and support data quality, scheduling, monitoring, and operational documentation so client teams can run and maintain what you build.

Key Responsibilities

• Develop and maintain batch and incremental ETL pipelines in Databricks using Python, PySpark, and Spark SQL.
• Ingest data from relational databases, APIs, flat files, and cloud storage sources into Delta Lake tables.
• Apply medallion architecture patterns by building bronze, silver, and gold layers.
• Write, tune, and maintain SQL for data transformations, validation, and reporting tables.
• Add data quality checks, manage schema changes, and troubleshoot failed jobs.
• Schedule and monitor workloads using Databricks Workflows, following Unity Catalog permissions.
• Tune Spark jobs for performance and cost, including partitioning, file sizing, and cluster configuration.
• Use Git for version control, participate in code reviews, and contribute to CI/CD pipelines.
• Document pipelines, source-to-target mappings, and runbooks for client team operations.
• Join client meetings to clarify requirements, provide status updates, and walk through your work.

Required Qualifications

• 1 to 3 years of professional experience in data engineering, software engineering, or a related technical role.
• Bachelor’s degree in computer science, information systems, engineering, mathematics, or a related field, or equivalent experience.
• Hands-on experience with Databricks from a professional role or substantial academic project.
• Working proficiency in Python for data processing.
• Experience building transformations with Apache Spark and PySpark (DataFrames and Spark SQL).
• Strong SQL, including joins, window functions, CTEs, and aggregations.
• Understanding of ETL concepts such as incremental loads, slowly changing dimensions, dimensional modeling, and data quality checks.
• Familiarity with at least one cloud platform: AWS, Azure, or Google Cloud.
• Clear written and verbal communication, including the ability to explain technical work to non-technical stakeholders.
• Authorization to work in the United States (confirm sponsorship policy).

Technologies and Tools

• Databricks, Python, PySpark, Spark SQL, Delta Lake
• Medallion architecture, Amazon S3, Azure Data Lake Storage
• Databricks Workflows, Unity Catalog, Apache Spark, SQL, Git
• CI/CD, DataFrames, ETL, AWS, Azure, Google Cloud

Nice to Have

• Databricks Certified Data Engineer Associate certification.
• Experience with Delta Lake features such as MERGE and time travel, or with Lakeflow Declarative Pipelines (formerly Delta Live Tables).
• Exposure to Unity Catalog, dbt, Apache Airflow, or Azure Data Factory.
• Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps.
• Prior consulting or client-facing experience.
• Background in financial services, healthcare, or media data.

Location and Compensation

• Location: McLean, VA (hybrid)
• Salary: USD 75,000 to 120,000 per year

About the Engagement

• Build and maintain Databricks data pipelines for Infinitive clients.
• Work on project teams with senior engineers, architects, and business analysts.
• Write production Python, PySpark, and SQL code that moves data from source systems into tables used by client analysts and data scientists.