Senior Data Engineer
Job Description
In this Senior Data Engineer role with the State of Illinois, you will lead the design, development, and continuous optimization of enterprise-grade data pipelines and distributed processing systems that support analytics, operations, and regulatory reporting. Working under administrative direction for the Department of Innovation & Technology (DoIT), you will architect scalable lakehouse and warehouse environments, strengthen DataOps and CI/CD automation, and help ensure data quality, governance, and observability across the enterprise.
Role overview
- Serve as a senior data engineering advisor for DoIT, performing highly complex professional and advisory functions.
- Lead enterprise data pipeline architecture and optimization, including scalable data platform design and expert guidance across the organization.
- Provide technical leadership across cross-functional data delivery squads, representing enterprise data engineering standards in agency-aligned feature development.
Responsibilities
- Design, build, and optimize enterprise ETL and ELT pipelines, distributed processing workloads, and semantic data models for analytics, business intelligence, and operational reporting.
- Drive technical governance for enterprise data lake architecture, supporting high availability, data quality, cost-efficient storage tiering, and end-to-end observability across the enterprise data asset inventory.
- Design and govern CI/CD automation frameworks for data engineering workloads, ensuring reliable and secure code promotion, environment parity, and production deployment.
- Implement and govern an enterprise role-based access control framework, ensuring auditable data access enforcement aligned with DoIT security standards and applicable state and federal data governance regulations.
- Engineer and optimize data workloads across relational, non-relational, and emerging data platforms, including large-scale RDBMS systems and NoSQL and graph databases.
- Continuously expand knowledge through online training platforms, meetings, training sessions, seminars, and conferences to stay current on products, vendors, techniques, and procedures.
- Perform other duties as required or assigned within the scope of the position.
Core technical focus
- Enterprise data platform engineering across environments and patterns, including lakehouse/warehouse governance and multi-layer integration.
- Distributed data processing optimization, including Spark execution engine tuning.
- Reusable data access layers and gateway patterns to abstract source complexity and reduce redundant data movement.
Minimum requirements
- Education: knowledge, skill, and mental development equivalent to four (4) years of college with coursework in Computer Science, Computer Engineering, Information Systems, Data Engineering, Software Engineering, Management Information Science, or a closely related field.
- Experience: five (5) years of professional experience in a related IT field, including data engineering, software engineering, database administration, business intelligence development, data architecture, systems integration, or a closely related technical discipline.
- Enterprise pipelines: five (5) years designing, building, and optimizing enterprise-scale data pipelines, fault-tolerant ETL/ELT patterns, and distributed data processing in a high-volume near-time production environment, including hands-on SQL performance tuning, execution plan analysis, index strategy, and workload optimization (examples include Microsoft SQL Server, Azure SQL Database, DB2, or equivalent).
- Spark optimization: five (5) years with distributed data processing and Spark execution engine optimization, including tuning large-scale batch, micro-batch, and streaming workloads (examples include Databricks, Apache Spark, Azure Synapse, Teradata Vantage, Snowflake, Amazon EMR, Starburst, Cloudera, or equivalent).
- Lakehouse architecture: five (5) years designing enterprise data platform and lakehouse architecture, including multi-zone lake/lakehouse patterns, open table format optimization, schema evolution, and integration of storage, compute, and catalog layers (examples include Microsoft Fabric, Databricks Unity Catalog, Snowflake, or equivalent).
- Data access layers: four (4) years engineering reusable data access layers and gateway patterns that abstract source complexity and deliver consistent, governed interfaces for downstream consumers and applications.
Preferred qualifications
- Experience matching the minimum enterprise pipeline, Spark optimization, and lakehouse architecture areas.
- Experience implementing role-based access control frameworks across enterprise data platforms aligned with formal enterprise data governance or security classification policy (examples include Databricks Unity Catalog, Microsoft Purview, Snowflake, or equivalent).
- Experience implementing enterprise data observability and data quality monitoring frameworks, including automated quality checks, schema drift detection, privacy monitoring incident leadership, data lineage tracking, anomaly detection, and pipeline alerting.
- Experience with DataOps, CI/CD pipeline engineering, and infrastructure-as-code for data workloads, including automated deployment of notebooks, Spark jobs, SQL deployments, and dbt models, and provisioning via tools such as Terraform, Azure Bicep, Databricks Asset Bundles, or GitHub Actions.
- Ability to provide expert peer code review and mentor junior and mid-level engineers.
- Strong communication skills for articulating technical concepts to both technical and non-technical audiences through documentation, presentations, and cross-functional collaboration.
Technologies
- SQL, ETL, ELT, CI/CD, DataOps, Spark
- Microsoft SQL Server, Azure SQL Database, DB2
- MongoDB, Couchbase, Redis, DynamoDB
- Neo4j, Cosmos DB Gremlin API
- Databricks, Apache Spark, Azure Synapse, Teradata Vantage, Snowflake, Amazon EMR, Starburst, Cloudera
- Microsoft Fabric, Databricks Unity Catalog
- Terraform, Azure Bicep, Databricks Asset Bundles, GitHub Actions
Benefits
- Competitive Group Insurance benefits including health, life, dental and vision plans.
- Flexible work schedules (when available and dependent upon position).
- 10-25 days of paid vacation time annually (10 days for first year of state employment).
- 12 days of paid sick time annually which carry over year to year.
- 3 paid personal business days per year.
- 13-14 paid holidays per year dependent on election years.
- 12 weeks of paid parental leave.
- Pension plan through the State Employees Retirement System.
- Deferred Compensation Program (voluntary supplemental retirement plan).
- Optional pre-tax programs: MCAP and DCAP.
- Tuition Reimbursement Program and eligibility for Federal Public Service Loan Forgiveness.
- For more information on State of Illinois benefits: https://www2.illinois.gov/cms/benefits/Pages/default.aspx
Job and application details
- Location: Chicago, IL (hybrid), with availability of 1 to 5 days per week in office depending on operational needs.
- Headquarters: 555 W Monroe St, Chicago, Illinois, 60661.
- Work schedule: 8:30 AM-5:00 PM Monday - Friday (hours may vary).
- Salary: anticipated starting salary $9,130 monthly; full range $9,130-$13,573.
- Job type: Salaried, Full Time; Number of vacancies: 1.
- Requisition ID: 59962.
- Opening date: 11/13/2026.
- Closing date: 12/01/2026.
- Agency: Department of Innovation and Technology.
- Class title: INFORMATION SYSTEMS ANALYST III - 21167.
- Skill option: System Services - Information Technology.
- Bargaining unit code: RC063.
- Union position: collective bargaining agreement/labor contract provisions apply.
- Agency contact: [email protected].
Conditions of employment
- Ability to verify identity.
- Employment authorization to accept a permanent full-time position with the State of Illinois.
- Successful completion of a position-specific, agency-required background check and self-disclosure of criminal history.
- Ability to travel occasionally, including overnight stays, as needed.
- Ability to work outside of normal business hours to meet deadlines.
- Ability to use agency-supplied equipment.
- Ability to lift and carry objects or equipment up to 10 pounds (sedentary work per U.S. Department of Labor definition).
- Compliance with revolving door restrictions under 5 ILCS 430/5-45.
- Compliance with Illinois Governmental Ethics Act requirements, including filing a Statement of Economic Interest.
Application requirements and submission method
- Apply electronically through illinois.jobs2web.com.
- State of Illinois employees should apply through the SuccessFactors employee career portal.
- Email or paper submissions (mail, fax, hand delivery) will not be considered.
- Proof of educational coursework is required, such as unofficial transcripts, copies of official transcripts, academic account records/screenshots, or similar documentation.
Additional information
- The State of Illinois does not provide sponsorship for employment visa status (for example, H-1B) and cannot provide OPT extensions under the STEM program.
- The State does not request job applicant wage or salary history in compliance with the Illinois Equal Pay Act.
- Reasonable accommodation is available for applicants and participants with disabilities; requests can be directed to the CMS Disability Resource Center at [email protected] or by phone at (217) 524-7514.