Data Engineer

The Data Engineer role at the Department of City Planning in New York City sits within the Information Technology Division, Geographic Data and Engineering unit. This onsite position in Manhattan focuses on designing, deploying, and scaling data infrastructure, building data products and pipelines, and collaborating across teams to publish high quality public geospatial datasets.

Responsibilities

• develop data engineering products, including:
  • operating data ingestion pipelines and overseeing data storage
  • executing transformations that standardize, geocode, merge, aggregate, and perform basic spatial manipulations
  • reviewing outputs with custom scripts or QAQC tools to identify errors, inconsistencies, and edge cases in datasets
• design new data products from concept to completion, collaborating on specifications with product owners, creating technical designs with teammates, assisting sprint planning, contributing to documentation, and delivering a finished product
• coordinate with other data engineers in sprint planning, design reviews, code reviews, and pair programming
• learn and modernize legacy data systems, bringing infrastructure into a contemporary development stack
• share accomplishments and lessons learned with NYC Planning, other city agencies, and broader audiences through talks, presentations, blogs, and other channels

Requirements

• baccalaureate degree in computer science, engineering, or a related field from an accredited college and at least four years of satisfactory full-time experience in IT automation engineering, monitoring engineering, or infrastructure management; or
• eight years of satisfactory full-time experience related to IT automation engineering, monitoring engineering, or infrastructure management; or
• education and/or experience equivalent to the above.

Technologies

• SQL databases (PostgreSQL, DuckDB, MySQL, SQL Server)
• Python
• Data Warehousing and Data Lakes
• dbt
• Cloud platforms: AWS, Google Cloud Platform, Azure
• Containerization
• CI/CD
• Git
• PostGIS
• GeoPandas
• BigQuery
• Snowflake
• Redshift
• Apache Spark
• Airflow
• Prefect
• Dagster

Benefits

• Public Service Loan Forgiveness

Location and Compensation

Location: 120 Broadway, New York, NY. Onsite in Manhattan. Salary: USD 100,000 to 112,000 per year.

Residency

New York City residency is not required for this position.