EngineerJobs.io
← Back to all jobs

Job Description

SkillStart is hiring a Senior Data Engineer for hybrid work in New York, NY 10114 to architect and run robust batch and streaming pipelines in a Capital Markets post-trade environment.

Responsibilities

  • Design, build, and maintain high-throughput batch and streaming data pipelines using Python, SQL, Spark, and streaming platforms.
  • Process and integrate post-trade data flows including trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting.
  • Develop and optimize relational data models, database schemas, security master structures, reference data models, and time-series financial datasets.
  • Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses on cloud platforms (preferably AWS, Snowflake, Redshift, Databricks).
  • Define data quality rules and data lineage coverage, with automated validation checks and reconciliation processes across complex flows.
  • Integrate real-time messaging architectures such as Kafka, RabbitMQ, and IBM MQ for ingestion, processing, and distribution of high-volume transactional trade data.
  • Apply data security, privacy controls, and regulatory compliance standards across ingestion and storage layers.
  • Use monitoring and observability tools (e.g., Datadog, Prometheus, Grafana) to track pipeline health, support SLA compliance, and accelerate incident response.

Requirements

  • Strong understanding of post-trade data flows, including allocations, confirmations, settlement, risk, finance, and regulatory reporting.
  • In-depth knowledge of securities data modeling and financial data structures, with hands-on experience across reference data, security master data, and transactional trade data.
  • Strong SQL skills, including designing, tuning, and optimizing relational data models and database schemas.
  • Proficiency in Python, with hands-on experience using data processing libraries and distributed frameworks such as Pandas and PySpark.
  • Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices.
  • Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar cloud-based platforms, preferably AWS.
  • Experience with stream and batch data processing technologies such as Kafka, Spark, Flink, or similar platforms.
  • Proven experience building ETL/ELT batch and streaming pipelines using Python and SQL.
  • Strong grounding in data quality assurance, data lineage tracking, data validation, and automated reconciliation.
  • Experience with messaging and event-driven platforms including Kafka, RabbitMQ, IBM MQ, or similar.
  • Clear understanding of data security, data privacy, and financial regulatory compliance requirements.
  • Familiarity with Datadog, Prometheus, and Grafana for monitoring and observability.

Technologies

  • Python, SQL, Spark
  • Pandas, PySpark
  • AWS, Snowflake, Redshift, Databricks
  • PostgreSQL, SQL Server
  • Kafka, RabbitMQ, IBM MQ, Flink
  • Datadog, Prometheus, Grafana

Preferred & Bonus Qualifications

  • Time-series data management and financial market data knowledge is a strong plus.
  • Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus.

Compensation

  • USD 122,000 - 148,000 per year

Work Location

  • Hybrid remote in New York, NY 10114

Similar Jobs