Senior Lead Data Engineer

The Senior Lead Data Engineer role at Information Technology Senior Management Forum (onsite in Plano, TX) focuses on architecting and delivering enterprise streaming data platforms, distributed pipelines, and analytics data stores that support timely, low-latency insights.

Responsibilities

• Design and build an enterprise-level scalable, low-latency, fault-tolerant streaming data platform to deliver meaningful, timely insights
• Develop next-generation distributed streaming data pipelines and analytics data stores using streaming frameworks such as Flink and Spark Streaming, with programming languages including Java, Scala, and Python
• Lead a group of engineers creating data pipelines using big data technologies, including Spark, Flink, Kafka, Snowflake, AWS Big Data Services, and Redshift, across medium to large scale datasets
• Establish and influence best practices for data pipeline design, data architecture, and processing of both structured and unstructured data
• Operate in a creative, collaborative environment aligned with agile methodologies, with emphasis on CI/CD and application resiliency standards, in partnership with Cyber & Security teams
• Collaborate with agile teams to design, develop, test, implement, and support technical solutions across full-stack development tools and technologies
• Lead developers with experience in machine learning, distributed microservices, and full-stack systems
• Apply Java, Scala, and Python along with open source RDBMS and NoSQL databases, and cloud data warehousing services such as Redshift and Snowflake
• Stay current with technology trends through experimentation and learning, participate in internal and external technology communities, and mentor engineers
• Collaborate with digital product managers to deliver robust cloud-based solutions that support financial empowerment for millions of Americans
• Perform unit testing and code reviews to ensure solutions are rigorously designed, clearly implemented, and tuned for performance

Requirements

• Bachelor’s Degree
• At least 6 years of experience in application development (internship experience does not apply)
• At least 2 years of experience in big data technologies
• At least 1 year of experience with cloud computing (AWS, Microsoft Azure, or Google Cloud)
• Master’s Degree
• 9+ years of experience in application development including Python, SQL, Scala, or Java
• 4+ years of experience in a public cloud (AWS, Microsoft Azure, Google Cloud)
• 5+ years of experience with distributed data or computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)
• 4+ years of experience working on real-time data and streaming applications
• 4+ years of experience with NoSQL implementation (Mongo, Cassandra)
• 4+ years of data warehousing experience (Redshift or Snowflake)
• 4+ years of experience with UNIX/Linux, including basic commands and shell scripting
• 2+ years of experience with agile engineering practices
• Experience leveraging interactive AI tooling to accelerate productivity, using capabilities beyond basic code completion

Technologies

• Flink, Spark Streaming, Spark, Kafka
• Java, Scala, Python
• Snowflake, AWS Big Data Services, Redshift
• Open Source RDBMS, NoSQL databases (Mongo, Cassandra)
• MapReduce, Hadoop, Hive, EMR, Gurobi, MySQL
• UNIX/Linux, shell scripting
• Agile methodologies, CI/CD
• SQL, AWS, Microsoft Azure, Google Cloud

Benefits

• Performance-based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
• Comprehensive, competitive, and inclusive set of health, financial, and other benefits supporting total well-being

Compensation

Plano, TX: $209,000 - $238,500 per year for Sr. Lead Data Engineer