Senior Data Engineer

Red Cedar Consultancy, LLC is hiring a Senior Data Engineer / Databricks Specialist to support a federal project in Washington, DC (onsite).

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, Gold) with data integrity, completeness, and lineage tracking
• Administer Databricks workspaces, clusters, scheduled jobs, and access controls
• Work with Unity Catalog and notebooks to support governed data access
• Manage AWS services including S3, Lambda, IAM, Aurora, and Glue
• Provision and manage infrastructure using Terraform
• Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions
• Ingest and integrate structured, semi-structured, and unstructured data from APIs and databases
• Implement security and compliance controls in DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tooling
• Support FISMA/FedRAMP needs, including ATO reassessments and POA&M resolution
• Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking
• Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments and provide technical leadership plus production on-call support

Requirements

• 3+ years of dedicated, hands-on Databricks development experience
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake
• Strong proficiency in Python and SQL, including working knowledge of REST API-based data ingestion
• Hands-on production experience with AWS S3, IAM, Lambda, Glue, and Aurora
• Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins or GitLab CI (or equivalent)
• Experience working in Agile/Scrum delivery environments
• Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline
• Active or previous High-Risk Public Trust (Tier 4) clearance

Technologies

• AWS Databricks, Spark, Python, PySpark, SQL, Delta Lake
• Medallion Architecture, Unity Catalog, Databricks workspaces, Databricks clusters, Unity Catalog notebooks
• S3, Lambda, IAM, Aurora, Glue
• Terraform, Jenkins, GitLab CI
• REST API, Apache Spark
• Delta Live Tables (DLT), MLflow, Model Registry
• DevSecOps CI/CD, Databricks Scheduled Jobs, ETL/ELT
• Agile/Scrum

Benefits

• 401(k) matching
• Dental insurance
• Employee discount
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Professional development assistance
• Referral program
• Vision insurance

Preferred Qualifications

• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines
• Experience supporting federal government contracts and FISMA/FedRAMP environments

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect – Associate or Professional
• AWS Certified Data Analytics – Specialty

Application Question

• Do you have at least 3 years of hands-on experience with AWS Databricks, including PySpark, Spark SQL, and Delta Lake?

Security Clearance

• Confidential (Required)

Location: Washington, DC (onsite)
Salary: USD 110,000 - 115,000 per yearly
Minimum Experience: 3 years
Education: Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline