EngineerJobs.io
← Back to all jobs

Job Description

Build, run, and improve end-to-end data pipelines for a large US-based global investment and technology company. This is a full-time, on-site role in McKinney, TX (Dallas project). You will work with financial and market-related data while designing reliable batch and streaming systems, strengthening data quality, and improving observability across the data platform.

What you’ll do

  • Own data pipelines end-to-end, covering ingestion, processing, validation, and publication.
  • Build and maintain batch and streaming pipelines for production use.
  • Implement reliable processing patterns, including idempotency, backfills, late-arriving data, and partitioning.
  • Develop automated data quality checks for completeness, timeliness, anomalies, and consistency across sources.
  • Monitor pipeline health and configure alerting to support fast incident response.
  • Investigate and resolve issues related to data quality and pipeline reliability.
  • Optimize pipeline performance and resource usage for large-scale workloads.
  • Onboard new datasets and data providers into the platform.
  • Work with data consumers by communicating incidents, data issues, changes, and availability.
  • Collaborate on APIs and internal libraries that enable data access and distribution.
  • Contribute to improved reliability, maintainability, and observability across the platform.

What you bring

  • 5+ years of professional experience in Data Engineering.
  • Strong hands-on experience with Python and SQL.
  • Strong hands-on experience with Apache Spark.
  • Experience with Hadoop / Hive or modern lakehouse formats such as Delta Lake or Apache Iceberg.
  • Experience building and maintaining batch and streaming pipelines.
  • Strong understanding of idempotent processing, backfills, late-arriving data, and partitioning strategies.
  • Experience implementing automated data quality checks.
  • Experience with monitoring and alerting for production data pipelines.
  • Experience working with large volumes of data and strong troubleshooting skills.
  • High attention to detail with a focus on data correctness.
  • Strong ownership and ability to manage data engineering tasks independently.
  • Strong communication skills with data consumers and technical stakeholders.
  • Valid US work authorization and ability to work on-site full-time in Dallas, TX.

Technologies you may use

  • Python, SQL, Apache Spark
  • Hadoop, Hive, Delta Lake, Apache Iceberg
  • Kafka, Airflow
  • Snowflake, Databricks
  • Java, Scala

Nice to have

  • Experience working with market data, including equities, macroeconomic data, or market data vendors.
  • Experience with Kafka or other streaming technologies.
  • Experience with Airflow or another workflow orchestration platform.
  • Experience with Snowflake or Databricks.
  • Experience developing APIs or reusable libraries for data distribution.
  • Experience with Java or Scala.
  • Previous experience in Investment Management / Financial Services.

Similar Jobs