Capital One seeks a Data Engineer 5 (Senior Manager, IC) in Enterprise Data Tech (EDT) to lead a major data transformation. The role focuses on designing and delivering scalable, resilient, cloud-first data pipelines and platforms, leveraging technologies aligned with modern AI workloads.
Key Responsibilities
- Partner with and across Agile teams to design, develop, test, implement, and support technical solutions using full-stack development tools and technologies
- Provide technical influence to a team of developers, data analysts, and data scientists with expertise in machine learning, distributed microservices, lakehouse architecture, and full-stack systems
- Build solutions using Python and Spark, integrating open-source relational and NoSQL databases with cloud data warehousing platforms such as Databricks and Snowflake
- Stay current with data and platform trends by experimenting with and learning new technologies, contributing to internal and external technology communities, and mentoring other members of the data community
- Collaborate with product managers and software engineers to deliver robust cloud-first data solutions that support experiences used by millions of Americans
- Independently design, build, and deliver cloud data solutions and applications with minimal guidance
- Architect and enforce common data engineering design patterns to improve code quality, maintainability, and reusability across platforms and pipelines
- Design and build data pipelines and platforms with a focus on scalability, resilience, and operational efficiency, including performance under increasing data volume and business demand
- Act as a force-multiplier by combining hands-on engineering contributions and innovation with mentoring and skills elevation for peers and junior engineers
- Lead end-to-end, large-scale data transformation initiatives, including critical architectural decisions and platform evaluation (for example, Snowflake versus Databricks) based on technical and business requirements
- Serve as an ambassador for the data engineering team by clearly communicating technical concepts and data outcomes to internal and external stakeholders
Required Qualifications
- Bachelor’s degree or higher in Computer Science or a related quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Engineering)
- At least 6 years of experience in application development (internship experience does not apply)
- At least 4 years of experience in distributed data
- At least 4 years of experience with SQL
- At least 4 years of programming experience with at least one of the following: Python, Java, or Scala
- At least 4 years of experience designing and developing data pipelines
- At least 2 years of experience in data modeling and designing end-to-end data solutions using both relational and non-relational database systems
Technologies and Tools
Python, Spark, Databricks, Snowflake, SQL, NoSQL, relational databases, AWS, EMR, Glue, lakehouse architecture, microservices, Redshift, MongoDB, Cassandra, DynamoDB, Airflow, Dagster, Monte Carlo, Splunk, real-time, streaming data pipelines
Preferred Qualifications
- Master’s degree in Computer Science or a related field
- 8+ years of experience in data engineering
- 5+ years of hands-on experience designing, deploying, and operating data workloads in AWS
- 5+ years of experience building or supporting distributed data or compute workloads using tools such as EMR, Spark, Glue, or Databricks
- 5+ years of experience designing, implementing, and operating real-time or streaming data pipelines
- 5+ years of experience working with unstructured or semistructured data using NoSQL databases (for example: MongoDB, Cassandra, DynamoDB)
- 5+ years of experience designing, implementing, and supporting data warehousing solutions (for example: Snowflake, Redshift)
- 4+ years of data modeling experience
- 3+ years of experience working in an Agile development environment
- 3+ years of experience developing user-centric reusable data products
- 3+ years of experience working on data observability (for example: Monte Carlo, Splunk) or data orchestration tools (for example: Airflow, Dagster)
Location and Compensation
Location: New York, NY (onsite)
Estimated Salary: USD 250,800 - 286,200 per year
Benefits
- Health, financial, and other benefits supporting your total well-being
- Performance based incentive compensation, including cash bonus(es) and/or long term incentives (LTI)