Data Engineer II
Big Data
Clickhouse
Cloud
Cloud Infrastructure
Cloud Native
Cloud Native Technologies
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Architecture
Data Build Tool
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Management
Data Modeling
Data Operations
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Database
Databases
DevOps
Devops Tools
DevSecOps
Engineering
Engineering Software
ETL
Exasol
Informatica
Kubernetes
Lakehouse
Platform Engineering
Programming Languages
Reporting and Analytics
Security Devops
Software Development
SQL
Job Description
Vylor is hiring a full-time Engineer II (Data Engineer II) to build scalable lakehouse and analytics data solutions for Seeds and Biotech domains and an enterprise data platform in Johnston, IA.
Responsibilities
- Design, develop, and maintain Lakehouse data pipelines, APIs, and data products for seed models, biotech research assets, enterprise applications, dashboards, and analytics
- Build scalable ETL/ELT pipelines using Python, SQL, dbt, Trino, and shell scripting to process and integrate large-scale agricultural data from enterprise systems, lab platforms, geographic datasets, drones, satellites, and IoT devices
- Create and maintain Bronze, Silver, and Gold layer data assets by integrating sources including Exasol, ClickHouse, MSSQL Server, and other enterprise systems
- Apply transformation, validation, reconciliation, and quality-control processes to support analytics and downstream usage
- Design Lakehouse datasets and transformation pipelines for seed, biotech, pathology, and disease analytics, including integration of laboratory, geographic, and historical research data
- Develop and optimize incremental and full-refresh dbt models and complex Trino SQL queries to improve performance, scalability, and maintainability
- Modernize legacy data applications by documenting business rules and source-to-target mappings, migrating ETL logic from legacy and Exasol-based environments to Trino and Lakehouse architectures, and deploying curated datasets to ClickHouse and Gold layer data products
- Develop and enhance PRISM and FRIP data integrations using Trino connectors and end-to-end comparison pipelines with reusable validation and reconciliation logic across PRISM, FRIP, Lakehouse, Exasol, and downstream application datasets
- Build reusable frameworks for logging, error handling, scheduling, automated testing, monitoring, deployment, and data-quality validation
- Troubleshoot application and pipeline integration issues and create analytics-ready datasets for statistical analysis, predictive modeling, regression, classification, clustering, dashboards, and business-intelligence reporting
Requirements
- Master’s degree (or equivalent) in Computer Science, Statistics, or a related field plus 2 years related experience
- 24 months experience designing and developing scalable ETL/ELT pipelines with Python, SQL, and shell scripting to extract, transform, validate, reconcile, and integrate large-volume structured, semi-structured, and unstructured data for enterprise R&D, analytical models, dashboards, and reporting systems
- 24 months experience developing ETL design mappings, source-to-target transformation rules, data reconciliation processes, data-quality controls, and complex SQL transformations to ensure accuracy and consistency across source and target platforms
- 24 months experience developing and optimizing complex Trino SQL queries and integrations among Lakehouse, Exasol, ClickHouse, MSSQL Server, and other databases to deliver curated, analytics-ready datasets for R&D models and enterprise applications (including phenomic pillar), dashboards, and reporting
- 24 months experience creating reusable Python- and Linux shell-based pipeline reliability and observability frameworks, including job scheduling, structured logging, exception handling, automated testing, monitoring, data-quality validation, cross-platform reconciliation, and application integration, plus using version control, containerization, and automated-deployment tools to deliver production data products
- 24 months experience analyzing and modernizing legacy data applications by documenting business rules and data mappings, migrating ETL logic to cloud-based or Lakehouse architecture, deploying curated data products, and using Git and Docker for source control, automated deployment, and containerization
Technologies
- Lakehouse
- Python
- SQL
- dbt
- Trino
- Shell scripting
- Exasol
- ClickHouse
- MSSQL Server
- Kubernetes
- Docker
- Git
- Linux
Benefits
- Health benefits for you and your family on your first day of employment
- Four weeks of paid time off and two weeks of well-being pay per year, plus paid holidays
- Excellent parental leave with a minimum of 16 weeks for mother and father
- Competitive retirement savings plan and tuition reimbursement program
- Numerous development opportunities to build skills
- Opportunity to contribute to making the world a better place
- Total rewards package information: Vylor Benefits
Telecommute Benefit
- Position includes a telecommute benefit within commuting distance to Johnston, IA Pioneer Hi-Bred/Vylor office, as directed
Location: Des Moines, IA (onsite)
Minimum Experience: 2 years