Engineer Jobs in Doraville, GA

1 job found

Include remote jobs

5d ago

Lead Data Engineer

INSPYR Solutions Doraville, GA 🇺🇸 Remote

Doraville, GA

5d ago

$60 - $70/hr Contract Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

INSPYR Solutions seeks a Lead Data Engineer to steer the design and optimization of distributed data pipelines within a high-volume, mission-critical setting. The role focuses on building and refining a Lakehouse architecture with Delta Lake, enabling robust data governance, reproducible ML workflows, and dependable production systems, all within a contract framework based in Doraville, GA with remote options. The position offers an hourly rate of USD 60 to 70.

Responsibilities

Design, develop, and tune distributed data pipelines with Apache Spark to support high-demand, mission-critical workloads.
Create and maintain an enterprise Lakehouse using Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
Build automated ingestion frameworks across batch, streaming, and event-driven modes spanning multiple cloud services and integration points.
Prepare feature-ready datasets and establish reproducible ML deployment patterns to enable machine learning workflows.
Lead data quality initiatives, access control, and cataloging across the platform.
Apply cost optimization, cluster tuning, and performance engineering to maximize efficiency.
Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
Own production reliability, troubleshoot issues, and perform root cause analysis for data and ML pipelines.

Requirements

7+ years of advanced data engineering experience with distributed compute technologies.
Expert Spark engineering skills, including performance tuning, cluster configuration, partition strategies, and handling large datasets.
Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
Strong proficiency in Python and SQL for large-scale data transformations.
Experience supporting machine learning pipelines or model operationalization.
Proven ability to architect cloud-native data platforms on Azure, AWS, or GCP.
Ability to integrate diverse, complex data sources at enterprise scale.
Demonstrated track record of owning mission-critical production systems.
Experience with distributed streaming frameworks such as Kafka or Event Hubs.
Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
Background in data security, compliance controls, or audit-ready governance.
Experience automating data operations with CI/CD and infrastructure as code.

Technologies

Databricks, Python, SQL, Apache Spark, Delta Lake, MLflow, Notebooks
Hugging Face Transformers, LangChain, LlamaIndex, LLMs including Anthropic Claude, Meta LLaMA, Google Gemini
Kafka, Event Hubs, ADLS, S3, GCS
Git, GitHub, GitLab, Azure Repos, Databricks Repos, GitHub Actions, Azure DevOps
Unity Catalog, RBAC, REST APIs

Core Tools

Databricks (Spark, Delta Lake, MLflow, Notebooks)
Python & SQL
Apache Spark via Databricks
Delta Lake for lakehouse architecture

Cloud Platforms

Azure, AWS, or GCP
Cloud Storage: ADLS, S3, GCS

Data Integration

Kafka or Event Hubs (streaming)
Auto Loader (Databricks file ingestion)
REST APIs, AI/ML workflows, MLflow
Hugging Face Transformers, LangChain / LlamaIndex for LLM integration
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

Git ecosystems: GitHub, GitLab, Azure Repos
Databricks Repos
CI/CD: GitHub Actions, Azure DevOps

Security & Governance

Unity Catalog
RBAC

Benefits

Comprehensive medical benefits
Competitive pay
401(k) retirement plan

View full job page →

Get Engineer Jobs in Doraville, GA alerts

New Engineer Jobs in Doraville, GA, daily.