Senior Data Engineer
Job Description
Build, run, and improve end-to-end data pipelines for a large US-based global investment and technology company. This is a full-time, on-site role in McKinney, TX (Dallas project). You will work with financial and market-related data while designing reliable batch and streaming systems, strengthening data quality, and improving observability across the data platform.
What you’ll do
- Own data pipelines end-to-end, covering ingestion, processing, validation, and publication.
- Build and maintain batch and streaming pipelines for production use.
- Implement reliable processing patterns, including idempotency, backfills, late-arriving data, and partitioning.
- Develop automated data quality checks for completeness, timeliness, anomalies, and consistency across sources.
- Monitor pipeline health and configure alerting to support fast incident response.
- Investigate and resolve issues related to data quality and pipeline reliability.
- Optimize pipeline performance and resource usage for large-scale workloads.
- Onboard new datasets and data providers into the platform.
- Work with data consumers by communicating incidents, data issues, changes, and availability.
- Collaborate on APIs and internal libraries that enable data access and distribution.
- Contribute to improved reliability, maintainability, and observability across the platform.
What you bring
- 5+ years of professional experience in Data Engineering.
- Strong hands-on experience with Python and SQL.
- Strong hands-on experience with Apache Spark.
- Experience with Hadoop / Hive or modern lakehouse formats such as Delta Lake or Apache Iceberg.
- Experience building and maintaining batch and streaming pipelines.
- Strong understanding of idempotent processing, backfills, late-arriving data, and partitioning strategies.
- Experience implementing automated data quality checks.
- Experience with monitoring and alerting for production data pipelines.
- Experience working with large volumes of data and strong troubleshooting skills.
- High attention to detail with a focus on data correctness.
- Strong ownership and ability to manage data engineering tasks independently.
- Strong communication skills with data consumers and technical stakeholders.
- Valid US work authorization and ability to work on-site full-time in Dallas, TX.
Technologies you may use
- Python, SQL, Apache Spark
- Hadoop, Hive, Delta Lake, Apache Iceberg
- Kafka, Airflow
- Snowflake, Databricks
- Java, Scala
Nice to have
- Experience working with market data, including equities, macroeconomic data, or market data vendors.
- Experience with Kafka or other streaming technologies.
- Experience with Airflow or another workflow orchestration platform.
- Experience with Snowflake or Databricks.
- Experience developing APIs or reusable libraries for data distribution.
- Experience with Java or Scala.
- Previous experience in Investment Management / Financial Services.