Store AI Agent - Onshore Data Engineer
Artificial Intelligence
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data & Ai
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Management
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Science
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Workflows
Delta Lake
ETL
Event Driven Architecture
Informatica
Pyspark
Spark
SQL
Streaming Data
Job Description
On-site contract opportunity in Dallas, TX focused on building Databricks-based data and machine learning pipelines.
Responsibilities
- Design, build, and maintain scalable data pipelines using Databricks, PySpark, and Python
- Create and optimize batch and real-time ingestion frameworks from diverse data sources
- Develop ETL/ELT solutions supporting enterprise data warehousing, analytics, and AI/ML workloads
- Implement data quality checks, validation, monitoring, and observability mechanisms
- Optimize Databricks jobs, notebooks, clusters, and workflows for performance and cost efficiency
- Design and maintain Delta Lake architectures, data models, and metadata management solutions
- Collaborate with Data Scientists and ML Engineers to prepare training datasets and feature stores
- Support development and deployment of deep learning solutions using PyTorch
- Build feature engineering pipelines and data transformation frameworks
- Enable model training, experimentation, and inference infrastructure on Databricks
- Integrate ML workflows with MLOps pipelines and CI/CD processes
- Apply DataOps and MLOps best practices
- Automate deployments using CI/CD pipelines
- Use cloud-native services for storage, security, monitoring, and orchestration
- Support infrastructure-as-code and environment automation initiatives
- Translate business, analytics, and data science requirements into technical solutions
- Ensure compliance with data governance, privacy, and security standards
- Participate in architecture reviews and technology evaluations
Requirements
- Strong experience with Databricks Platform
- Expertise in Python
- Strong hands-on experience with PySpark
- Experience with PyTorch for machine learning and deep learning workloads
- Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows
- Experience developing ETL/ELT pipelines
- Strong SQL and data modeling skills
- Experience with distributed computing and big data architectures
- Knowledge of REST APIs and data integration patterns
- Git version control and CI/CD tools
- Experience with one or more: Microsoft Azure (preferred), AWS, Google Cloud Platform (GCP)
- Experience with one or more: Spark, Kafka/Event Streaming, Azure Data Factory or equivalent integration tools, Data Lake architectures, Data Warehousing solutions, Feature Stores
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or related field
- 5+ years of experience in Data Engineering
- 3+ years of experience with Databricks and cloud-based data platforms
- Experience supporting machine learning and AI use cases using PyTorch
Technologies
- Databricks Platform
- Python
- PySpark
- PyTorch
- Delta Lake
- Unity Catalog
- Databricks Workflows
- ETL/ELT
- SQL
- REST APIs
- Git
- CI/CD
- Microsoft Azure
- AWS
- Google Cloud Platform (GCP)
- Spark
- Kafka/Event Streaming
- Azure Data Factory
- Feature Stores
Preferred Skills
- Experience with GenAI and LLM data pipelines
- Knowledge of MLflow and MLOps frameworks
- Hands-on experience with Docker and Kubernetes
- Experience with real-time streaming architectures
- Familiarity with vector databases and AI platforms
- Certifications in Databricks or Cloud platforms
Nice-to-Have Experience
- Retail, QSR, Restaurant Technology, Digital Commerce, or Customer Analytics domains
- Large-scale enterprise data modernization initiatives
- AI/ML and advanced analytics implementations
Benefits
- Medical
- Dental
- Vision
- Retirement benefits
Location: Dallas, TX, United States (onsite)
Job type: Contract
Pay range: $39.30/hour - $61.40/hour
Applications will be accepted on an ongoing basis.
Similar Jobs
S
