SHI’s IDCS team seeks a Software Engineer to develop and maintain software that runs on top of an integration and deployment automation platform. The work spans operator web applications, orchestration pipelines, inventory and addressing services, and CLI tooling that transform manual deployment and QC processes into observable, dependable software.
Responsibilities
- Design, build, and maintain an operator-facing web application, including rack views, pipelines, dashboards, work orders, and approvals.
- Develop backend services and REST APIs in Python using FastAPI or similar technologies to expose inventory, device actions, and run status.
- Build and extend the pipeline/DAG engine, including stage ordering, retries, parallel fan-out across devices, run history, and resume-from-failure behavior.
- Implement real-time capabilities using SSE or WebSockets for live run output, device status, and rack health.
- Create and maintain CLI tools and a command router that dispatches actions by customer, device type, and OEM.
- Build reusable libraries and frameworks with clear module boundaries and separated front-end pages, avoiding one-off scripts.
- Own the schema and data access layer for the inventory database, covering racks, devices, addressing, credentials, and run results.
- Maintain inventory as the single source of truth for addressing and device identity to ensure consistency across tools that read and write it.
- Implement reporting that generates QC coversheets, command logs, and audit evidence for each run.
- Integrate with vendor and infrastructure APIs, including Redfish, DHCP server APIs, and device CLIs over SSH.
- Support heterogeneous hardware through a common adapter model, including Dell, HPE, Lenovo, and Supermicro servers; Cisco, Arista, and Juniper switches; PDUs; and console servers.
- Develop validation, timeout, and error-handling logic to prevent hung or unreachable devices from stalling the platform.
- Create automated checks that scan racks and servers for health issues and open work orders.
- Implement role-based access, audit logging, and secure credential handling so plaintext secrets are not stored in code, logs, or API responses.
- Add approval gates so device-changing commands do not run without explicit operator approval.
- Write unit, integration, and end-to-end tests, using Git branching and code review to support safe production releases.
- Deploy and support services on RHEL using gunicorn/uvicorn behind nginx, systemd, and log analysis.
- Write documentation, runbooks, and SOPs.
- Mentor junior engineers and interns.
Requirements
- Minimum of 5 years of professional software engineering experience.
- 3+ years of hands-on Python development building production applications, services, or tooling.
- Experience designing, building, and supporting REST APIs and web applications in production.
- Experience designing and maintaining relational database schemas and the associated data access layer.
- Experience deploying and supporting services on Linux in production.
- Experience integrating with external systems via APIs and device interfaces (REST, SSH/CLI, or similar).
- Experience writing automated tests and shipping through Git-based code review.
- Strong Python skills, including typed, tested, production-quality code.
- REST API design experience using FastAPI, Flask, Django, or similar.
- Front-end experience with JavaScript, HTML, and CSS, including modern UI development without heavy framework dependence.
- SQL and relational databases, including MariaDB/MySQL.
- JSON/YAML.
- Git, including branching, pull requests, and code review.
- Asynchronous and concurrent programming, including threads, async I/O, and parallel fan-out.
- Linux administration with RHEL preferred.
- Bash/shell scripting.
- SSH, systemd services, and package management.
- Web server and reverse proxy fundamentals, including nginx and gunicorn/uvicorn.
- Log analysis and troubleshooting.
- TCP/IP, VLANs, and subnetting; DHCP/DNS; network troubleshooting.
- Understanding of enterprise server hardware, BIOS/UEFI, and firmware.
- Out-of-band management technologies, including BMC, IPMI, and Redfish.
- PXE/network boot concepts.
- Secure coding and secrets handling.
- Experience mentoring junior engineers.
Preferred Requirements
- Experience with Ansible, including playbooks, roles, and inventories.
- Experience with Paramiko, Netmiko, or similar network device automation libraries.
- Experience with Kea DHCP, including control agent APIs, reservations, and leases, as well as iPXE.
- Experience with Kubernetes and/or Docker.
- Experience with Prometheus and Grafana.
- Experience with real-time visualization technologies such as Canvas, WebGL, or three.js.
- Experience integrating AI/LLM tooling with human approval controls.
- Experience working with Dell, HPE, Lenovo, Supermicro, Cisco, Arista, Juniper, Raritan, Vertiv, or Avocent/OpenGear devices.
- Manufacturing or factory-integration experience.
Other Requirements
- Ability to work effectively alongside hardware, networking, integration, and QC teams in a factory integration environment.
- Ability to create reliable, reusable software and frameworks rather than one-off scripts.
- Ability to work across heterogeneous server, networking, power, and console-server technologies.
- Ability to develop and support secure, observable production applications and services.
- Ability to create clear technical documentation, runbooks, and SOPs.
- Demonstrated ability to mentor junior engineers and interns.
Technologies
Python, FastAPI, REST APIs, DAG engine, SSE, WebSockets, CLI, inventory database, Redfish, DHCP, SSH, unit tests, integration tests, end-to-end tests, gunicorn, uvicorn, nginx, systemd, RHEL, SQL, MariaDB, MySQL, Git, JSON, YAML, Linux, BMC, IPMI, PXE, TCP/IP, VLANs, BIOS/UEFI, Bash/shell scripting.
Benefits
- Health, wellness, and financial benefits to offer peace of mind to you and your family.
- 401K
- Flexible spending
- Medical, vision, dental
Compensation and Location
- Location: Piscataway, NJ (onsite)
- Salary: USD 140,000 - 180,000 per year
- Experience: 5+ years
