The Lead Data Engineer - Pipelines, Spark Streaming and Spark Offline is an on site role in Tampa, FL with JPMorganChase. This position focuses on maintaining critical data pipelines and architectures across multiple technical areas within the Commercial & Investment Bank, delivering secure, scalable data solutions to support the firm’s objectives. As a core technical contributor, you will enhance, build, and deliver these pipelines and architectures within an agile team.
Responsibilities
- Collaborate with JPMorgan's lines of business and functions to deliver software solutions
- Design, architect, develop and productionize efficient data pipelines, data services and data platforms that support business needs
- Build highly scalable and reliable data processing pipelines and perform analyses to drive and optimize business results
- Design and develop features and entities for machine learning and rule-based use cases using Spark or other big data environments
- Seize opportunities to align physical, IT and data security architectures to manage access
- Incorporate reuse-first, AI-assisted practices into delivery and operations, ensuring traceability, auditability and adherence to resiliency and security standards
Requirements
- Formal training or certification in Data Engineering and 5+ years of applied experience
- Proven experience using enterprise authorized AI capabilities to support data engineering workflows, with strong validation habits and awareness of data sensitivity
- Ability to review and validate AI assisted outputs before use, escalating when uncertain and following data handling requirements
- Proficient programming skills in Python and PySpark
- Experience across the data lifecycle, building data frameworks and working with data lakes
- Experience with batch and real time data processing using Spark or Flink, and feature engineering in batch and real time contexts with Spark, Flink or Databricks
- Working knowledge of AWS Glue and EMR for data processing and real time data processing, with features using Flink or Databricks live tables or Spark streaming
- Experience with Databricks and Databricks live tables
- Experience building services with Glue, Lamida, EMR or Flask, and deploying them on AWS EKS or Kubernetes
- Working experience with both relational and NoSQL databases
- Experience in ETL pipelines for both batch and real time data processing, data warehousing and NoSQL databases
Technologies
- Python
- PySpark
- Spark
- Spark Streaming
- Flink
- Hadoop
- Kafka
- Databricks
- Databricks live tables
- AWS Glue
- AWS EMR
- Flask
- Lamida
- AWS EKS
- Kubernetes
- Docker
Benefits
- Base salary
- Commission-based pay
- Discretionary incentive compensation (cash and/or forfeitable equity)
- Comprehensive health care coverage
- On-site health and wellness centers
- Retirement savings plan
- Backup childcare
- Tuition reimbursement
- Mental health support
- Financial coaching
About Us
JPMorganChase is a long standing financial institution delivering innovative solutions to consumers, small businesses and large corporate, institutional, and government clients under the J.P. Morgan and Chase brands. With roots spanning more than two centuries, the firm leads in investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management.
We offer a competitive total rewards package based on role, experience, and location. Eligible employees may receive commission-based pay and discretionary incentive compensation in cash and/or forfeitable equity, in recognition of individual contributions. A range of benefits and programs is available based on eligibility to meet diverse needs.
We recognize that people are our strength and value diversity and inclusion across our global workforce. JPMorganChase is an equal opportunity employer and does not discriminate on the basis of protected attributes, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or other basis permitted by law.
JPMorgan Chase & Co. is an Equal Opportunity Employer, including Disability/Veterans.
About the Team
J.P. Morgan's Commercial & Investment Bank operates as a global leader across banking, markets, securities services and payments. Corporations, governments and institutions worldwide rely on our services in more than 100 countries. The team provides strategic advice, capital raising, risk management and liquidity solutions to clients globally.
