Senior Data Engineer (Customer Data Products)

NBME is modernizing an AWS-based data platform to support analytics, AI, ML, and everyday business decision-making. In this hybrid role in Philadelphia, PA, you will help build data lakes and scalable pipelines, strengthen data quality and security, and lead technical work that enables broader adoption of emerging data and AI capabilities.

Salary: USD 127,337 - 160,000 per year
Location: Philadelphia, PA (hybrid)
Minimum experience: 7 years
Education: Bachelor’s Degree

Benefits

• Healthcare, Dental, Prescription, and Vision plans
• 401(k) with match
• Tuition Reimbursement Plan
• Commuter Benefit: Public Transit or Parking options
• Remote Friendly Workplace

Responsibilities

• Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, including feature stores and vector databases, aligned with best practices and security standards.
• Collaborate with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML training, fine-tuning, and feature extraction.
• Lead evaluation, research, and experimentation using batch and streaming data technologies, LLM data preparation frameworks, and MLOps tools.
• Serve as a technical lead to demonstrate capabilities of emerging AI and data technologies and support organization-wide adoption of modern data techniques.
• Contribute to the definition and refinement of processes and procedures for the data engineering practice.
• Educate and develop ETL developers on cloud-based initiatives to enable transition toward data engineering responsibilities.
• Support integrity and accuracy of corporate data, with emphasis on data security.
• Help ensure high data quality for Data Services, Analytics, and Master Data Management.
• Coordinate technical solutions and take ownership of designs, development, testing, and delivery.
• Build automated, scalable, test-driven data pipelines.
• Use software development practices including Git, CI/CD, and release management to enhance AWS CI/CD pipelines.
• Work with Data Engineers, DevOps engineers, and architects to improve DataOps tools and frameworks.

Requirements

• Bachelor’s Degree
• At least 7 years of experience in application development (internship experience does not apply)
• At least 4 years of experience in big data technologies
• At least 4 years of cloud computing experience using AWS
• 4+ years of application development experience including Python, SQL, Scala, or Java
• 4+ years of experience with distributed data and computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL, etc.)
• 4+ years of experience working on real-time data and streaming applications
• 4+ years of experience with NoSQL implementation (Mongo, Cassandra)
• 4+ years of data warehousing experience (Redshift)
• 6+ years of experience with UNIX/Linux including basic commands and shell scripting
• 7+ years of experience with Agile engineering practices
• 7+ years of experience with SQL optimization
• 4+ years of experience with PySpark
• 3+ years of experience with process orchestration including AirFlow, KubeFlow, AWS Step Functions, or Luigi

Technologies

• AWS
• Python, SQL, Scala, Java
• MapReduce, Hadoop, Hive, EMR
• Kafka, Spark
• MySQL, Mongo, Cassandra
• Redshift
• UNIX/Linux, shell scripting
• Git, CI/CD
• AirFlow, KubeFlow, AWS Step Functions, Luigi
• PySpark
• Feature stores
• Vector databases: Pinecone, Milvus, pgvector
• LLM data preparation frameworks
• MLOps tools
• DataOps tools
• APIs, Data-as-a-Service (DaaS), Amazon Redshift

Preferred Qualifications

• Proven experience implementing Generative AI, LLM data preparation pipelines, and Vector Databases (Pinecone, Milvus, pgvector)
• Strong experience building and maintaining Feature Stores for machine learning models
• Experience building highly scalable, secure, production-ready APIs and Data-as-a-Service (DaaS) platforms
• AWS Certified Data Engineer or AWS Certified Machine Learning – Specialty certifications
• 3+ year of experience with Machine Learning
• Experience with building a Data-as-a-service platform
• Experience with building APIs

Hybrid Position / Location and Onsite Requirements

• This role is designated as hybrid, with occasional onsite work for team collaboration or meetings.
• Applicants within a 50-mile radius from the Philadelphia office are currently prioritized.
• Onsite interviews and onboarding at the Philadelphia office may be required; advance notice will be provided where applicable.