Senior Data Engineer

Shield Consulting Solutions is hiring a Senior Data Engineer in Annapolis Junction, MD (onsite) to build and optimize end-to-end analytics data pipelines.

Responsibilities

• Apply extensive expertise in dataflow design, data transport mechanisms, and Apache Spark based distributed processing
• Design, implement, and optimize data ingress and egress pathways for efficient, scalable, and reliable processing of analytics workloads

Requirements

• Minimum 7 years of relevant experience
• Bachelor’s degree in a technical discipline
• Experience using the Linux CLI and Linux tools
• Experience developing Bash scripts to automate manual processes
• Recent software development experience using Python and Java
• Experience with Apache Airflow including DAG design, scheduling, operators, and sensors for orchestrating and monitoring complex workflows
• Experience with distributed big data processing engines, including Apache Spark
• Familiarity with SQL technologies including MySQL, MariaDB, and PostgreSQL for querying, joining, and aggregating large datasets
• Experience using Jupyter Notebook
• Experience with data wrangling and preprocessing using pandas and NumPy
• Experience working with structured, semi-structured, and unstructured data formats including Parquet, JSON, CSV, and XML
• Familiarity with data quality concepts, data validation, and anomaly detection
• Experience with Git source control
• Familiarity with HPC job scheduling tools including Slurm
• Experience with the Atlassian Tool Suite including JIRA and Confluence

Technologies

• Apache Spark
• Apache Airflow
• Linux CLI
• Bash
• Python
• Java
• MySQL
• MariaDB
• PostgreSQL
• Jupyter Notebook
• pandas
• NumPy
• Parquet
• JSON
• CSV
• XML
• Git
• Slurm
• JIRA
• Confluence

Compensation

• $230,000 - $240,000 per year (USD)

Benefits

• 25 days PTO
• 11 paid holidays
• 100% employer-paid healthcare for employees and dependents, available day 1
• 8% 401(k) employer match with immediate vesting