Data Engineer (Ab Initio, OzoneCH, Flink)

The contract Data Engineer role in Berkeley Heights, NJ onsite focuses on building high-performance, event-driven data pipelines with the Ab Initio ecosystem for a large credit card processing system.

Location: Berkeley Heights, NJ (onsite) • Salary: USD 50 - 53 per hour • Type: Contract

Responsibilities

• Architect and develop intricate Ab Initio continuous flow graphs, incorporating stateful joins, sliding time windows, and aggregations.
• Construct event-driven data pipelines leveraging Kafka, MQ, and file streams to push updates to dashboards or downstream data stores in near real time.
• Guarantee the resilience of continuous flows through checkpointing and recovery to ensure exactly-once processing semantics.
• Utilize advanced Ab Initio components such as Reformat, Rollup, Join, and Partition to attain low-latency data processing.
• Proactively monitor live production streams to sustain 24/7 reliability and troubleshoot data issues as they arise.
• Develop ETL pipelines for both batch and real-time data ingestion and transformation.
• Implement and enforce data validation, security, integrity, and compliance across big data platforms.
• Diagnose and resolve performance issues in large-scale clusters.
• Collaborate with data scientists, analysts, and application teams to deliver robust data solutions.
• Automate workflows and improve operational efficiency through scripting and orchestration tools.

Requirements

• Strong understanding of credit card processing systems.
• Expertise with GDE (Graphical Development Environment), EME (Enterprise Meta Environment), Conduct>It, and Continuous Flows.
• Knowledge of Kafka, message queues, and real-time stateful services.
• Proficiency in Unix/Linux shell scripting, SQL, and databases such as Oracle and Teradata.
• Experience in Java, Scala, Python, or Kafka is a plus.
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of data security, governance, and compliance standards.
• Experience with cloud-based big data platforms.
• Exposure to containerization with Docker and Kubernetes for big data workloads.
• Knowledge of CI/CD pipelines for data engineering projects.

Technologies

• Ab Initio
• GDE (Graphical Development Environment)
• EME (Enterprise Meta>Environment)
• Conduct>It
• Kafka
• MQ
• Oracle
• Teradata
• Java
• Scala
• Python
• SQL
• Linux
• Unix
• Shell scripting
• Docker
• Kubernetes
• CI/CD pipelines
• Cloud-based big data platforms

Equal Employment Opportunity

Next Gen Software Solutions LLC is an Equal Opportunity Employer. We are committed to fostering an inclusive and diverse workplace where all applicants and employees are treated with respect and dignity. We do not discriminate on the basis of race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, genetic information, veteran status, or any other characteristic protected by applicable federal, state, or local laws.