Senior Data Engineer
Job Description
IvoryCloud is seeking a Senior Data Engineer to design and optimize data pipelines and analytics platforms using Databricks for a Department of Energy engagement. Based in Rockville, Maryland with a hybrid work arrangement, you will collaborate with analytics, BI, and AI teams to deliver trusted data assets and AI-ready data at scale. This role centers on building scalable pipelines and advancing a modern data platform.
Responsibilities
- Design, develop, and maintain scalable data pipelines with Azure Databricks and Apache Spark.
- Build and optimize batch and streaming data processing solutions.
- Develop robust ETL/ELT frameworks for structured and semi‑structured data from SQL, APIs, SaaS apps, and Data Live Lake Storage.
- Implement transformation logic using PySpark, SQL, and Delta Lake within a Lakehouse / Medallion architecture.
- Integrate and model data from SQL, SaaS systems, and cloud storage into unified, analytics‑ready and AI‑ready structures.
- Partner with analytics, BI, and AI/ML teams to deliver trusted datasets, semantic layers, and AI‑ready data (features, vector‑ready patterns, embeddings).
- Improve data quality, observability, lineage, governance, and security in alignment with Azure and enterprise standards.
- Optimize data performance, storage costs, and processing efficiency in cloud environments.
- Participate in architecture discussions and contribute to the evolution of the organization’s data platform and engineering best practices.
Requirements
- U.S. Citizenship is required and non-negotiable.
- Two days per week in the Rockville, MD office are required and non-negotiable.
- Ability to pass multiple background and employment checks, including federal and local.
- Bachelor’s degree Technology, Business, or Related Field
Technologies
- Databricks
- Apache Spark
- PySpark
- SQL
- Delta Lake
- Data Live Lake Storage
- Python
- dbt
- Airflow
- Azure Data Factory
- Kafka
- Snowflake
- Git
- CI/CD Pipelines
- Azure, AWS, GCP
- Unity Catalog
- Delta Live Tables
- Workflows
- Lakeflow
- SQL Warehouse
- Mosaic AI
- Synapse pipelines
- Dagster
- Azure OpenAI
- Event Hubs
- Kinesis
- Redshift
- Synapse
Benefits
- 401(k)
- Health insurance
- Paid time off
- Vision insurance
What success looks like
- Deliver reliable, production-ready data pipelines supporting critical business use cases.
- Improve data processing performance and scalability.
- Establish reusable engineering patterns and best practices.
- Enable analytics and AI teams with high-quality, trusted datasets.
- Contribute to the modernization and growth of the organization’s data platform capabilities.
Technical Environment
- Databricks
- Apache Spark
- PySpark
- SQL
- Delta Lake
- Python
- dbt
- Airflow
- Azure Data Factory
- Kafka
- Snowflake
- Git
- CI/CD Pipelines
- Cloud Platforms (Azure, AWS, or GCP)
Education & Certifications
- B.S. in Business/Technology-related field
Legal
IvoryCloud is an Equal Opportunity Employer (EEO)
Pay
$155,000.00 - $200,000.00 per year
Experience
- Databricks: 4 years (Required)
Location
Rockville, MD (hybrid)
Application Questions
- This role requires U.S. Citizenship. Do you meet this requirement?
- Can you meet the hybrid requirement of 2 days a week in office in Rockville, Maryland?
- Are you able to come into the office for in-person interviews?
Similar Jobs
