AI Data Engineer - AI Training data

The AI Data Engineer will design and operate Python-driven data processing pipelines that turn structured and unstructured sources into training-ready AI datasets. This onsite role in Seattle focuses on OCR and text workflows, large-scale batch processing, data quality validation, and dataset management within Azure, working closely with data science and engineering teams.

Responsibilities

• Design, build, maintain, and optimize Python-based pipelines for large-scale data ingestion, processing, and transformation.
• Create AI datasets from scratch by collecting, organizing, cleaning, validating, and preparing both structured and unstructured raw data.
• Develop OCR and text workflows, including text extraction, normalization, filtering, segmentation, and deduplication.
• Run and oversee large-scale batch processing jobs, troubleshoot job failures, and implement automated retry and recovery mechanisms.
• Implement automated data quality checks to detect missing or corrupted files, incomplete outputs, duplicate records, formatting issues, and metadata inconsistencies.
• Produce high-quality training datasets through filtering, tokenization, text segmentation, and conversion into standardized formats.
• Handle bulk data uploads, downloads, and synchronization between local processing environments and Azure storage platforms.
• Maintain organized datasets with consistent naming conventions, metadata management, version control, and source traceability.
• Optimize processing performance, memory usage, storage efficiency, and cloud resource consumption.
• Monitor dataset statistics, processing coverage, quality metrics, and exceptions, and generate operational reports.
• Collaborate with applied scientists and engineering teams to translate AI training requirements into scalable and repeatable data workflows.
• Maintain documented, tested, and version-controlled code while adhering to established data security and governance requirements.

Requirements

• 5+ years of professional experience in data engineering, data pipeline development, or data science with substantial hands-on programming responsibilities.
• Strong Python proficiency, including building reusable modules and working through debugging, testing, logging, and error handling.
• Experience building datasets from scratch, including ingestion, cleaning, transformation, validation, and preparation.
• Hands-on experience processing large-scale structured and unstructured datasets using memory-efficient, parallel, or distributed processing approaches.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk transfers, storage organization, and access troubleshooting.
• Experience with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Strong troubleshooting, analytical, and problem-solving skills with an emphasis on data quality and integrity.
• Ability to communicate technical challenges and collaborate across engineering and data science teams.

Preferred Qualifications

• Experience building OCR pipelines for scanned documents, multilingual text, or document-layout data.
• Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or similar workflow orchestration tools.
• Familiarity with Hugging Face Datasets, AI/ML tokenization, near-duplicate detection, and separation of training and evaluation data.
• Experience curating and preparing large-scale datasets for AI or machine learning model training.
• Knowledge of distributed data processing frameworks and technologies such as PySpark.
• Experience with dataset lineage, reproducibility, containerized environments, and CI/CD practices.
• Experience optimizing pipelines for performance, reliability, scalability, and cost efficiency.

Technologies

• Python
• Azure Blob Storage, Azure Data Lake Storage
• pandas, NumPy
• JSON, JSONL, CSV, Parquet
• SQL, Git, Linux, OCR
• tokenization
• Azure Databricks, Azure Machine Learning, Azure Data Factory
• Hugging Face Datasets
• PySpark
• CI/CD

Location and Schedule

Seattle, WA (onsite)

Schedule: 5 days onsite at Seattle, WA

Salary

USD 150,000 - 160,000 per year

For candidates based in Washington State, the anticipated salary range is $150,000–$160,000 USD/Annually.

Benefits

• Medical, dental, and vision coverage
• Flexible Spending Account
• 401k program
• Competitive PTO offerings
• Parental Leave
• Opportunities for professional growth and development

Applicant Accommodations

If you need a reasonable accommodation to participate in any part of the application or interview process, please contact [email protected].