S
Senior Data Engineer
Backend Developer
Senior
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Operations
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
ETL
Event Driven Architecture
Informatica
Information Technology (IT)
Kafka
Programming Language
Programming Languages
Pyspark
Snowflake
Software Development
Spark
SQL
Stream Processing
Streaming Data
Job Description
SkillStart is hiring a Senior Data Engineer for hybrid work in New York, NY 10114 to architect and run robust batch and streaming pipelines in a Capital Markets post-trade environment.
Responsibilities
- Design, build, and maintain high-throughput batch and streaming data pipelines using Python, SQL, Spark, and streaming platforms.
- Process and integrate post-trade data flows including trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting.
- Develop and optimize relational data models, database schemas, security master structures, reference data models, and time-series financial datasets.
- Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses on cloud platforms (preferably AWS, Snowflake, Redshift, Databricks).
- Define data quality rules and data lineage coverage, with automated validation checks and reconciliation processes across complex flows.
- Integrate real-time messaging architectures such as Kafka, RabbitMQ, and IBM MQ for ingestion, processing, and distribution of high-volume transactional trade data.
- Apply data security, privacy controls, and regulatory compliance standards across ingestion and storage layers.
- Use monitoring and observability tools (e.g., Datadog, Prometheus, Grafana) to track pipeline health, support SLA compliance, and accelerate incident response.
Requirements
- Strong understanding of post-trade data flows, including allocations, confirmations, settlement, risk, finance, and regulatory reporting.
- In-depth knowledge of securities data modeling and financial data structures, with hands-on experience across reference data, security master data, and transactional trade data.
- Strong SQL skills, including designing, tuning, and optimizing relational data models and database schemas.
- Proficiency in Python, with hands-on experience using data processing libraries and distributed frameworks such as Pandas and PySpark.
- Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices.
- Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar cloud-based platforms, preferably AWS.
- Experience with stream and batch data processing technologies such as Kafka, Spark, Flink, or similar platforms.
- Proven experience building ETL/ELT batch and streaming pipelines using Python and SQL.
- Strong grounding in data quality assurance, data lineage tracking, data validation, and automated reconciliation.
- Experience with messaging and event-driven platforms including Kafka, RabbitMQ, IBM MQ, or similar.
- Clear understanding of data security, data privacy, and financial regulatory compliance requirements.
- Familiarity with Datadog, Prometheus, and Grafana for monitoring and observability.
Technologies
- Python, SQL, Spark
- Pandas, PySpark
- AWS, Snowflake, Redshift, Databricks
- PostgreSQL, SQL Server
- Kafka, RabbitMQ, IBM MQ, Flink
- Datadog, Prometheus, Grafana
Preferred & Bonus Qualifications
- Time-series data management and financial market data knowledge is a strong plus.
- Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus.
Compensation
- USD 122,000 - 148,000 per year
Work Location
- Hybrid remote in New York, NY 10114