Engineers and Constructors International Inc. is hiring a Principal Data Engineer contract resource for an anticipated 7-month engagement in Houston, TX (onsite). In this role, you will lead Azure-based data lake architecture and governance, with hands-on delivery across Databricks Delta Live Tables, Structured Streaming, and high-volume data processing.
Role Summary
This Principal Data Engineer contract role focuses on building and governing Azure data lake capabilities, including data transformation pipelines, data lineage and metadata management, and production support for scalable analytics environments. You will provide technical leadership while guiding teams through capability roadmaps spanning Databricks and broader governance tooling.
Responsibilities
- Architect, design, and implement scalable Azure-based data lake solutions.
- Establish roadmaps for capability-building across Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.
- Provide technical leadership, ensuring adherence to best practices and quality standards.
- Lead development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
- Optimize Databricks resources for high-volume processing while staying within budget.
- Oversee Databricks Unity Catalog usage for governance, lineage, and metadata management.
- Develop and enforce robust data governance policies to support compliance, security, and data quality.
- Monitor data pipeline performance and proactively resolve production issues.
- Collaborate with cross-functional teams to align project objectives with business goals.
- Translate technical challenges into actionable insights for executive teams.
- Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.
- Facilitate knowledge transfer sessions to empower internal teams with cutting-edge methodologies.
Requirements
- Bachelor’s or Master’s degree in IT or a related discipline.
- 15-17 years of experience in the IT industry with expertise in delivering cost-effective cloud-based data analytics solutions.
- Proven track record building data lake architectures and optimizing performance in big data environments.
- Azure Data Engineering or Databricks certification (preferred).
- Strong knowledge of Azure technologies including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
- Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.
- Experience designing and developing scalable ETL pipelines and optimizing data workflows.
- Deep understanding of data migration from on-premises to cloud and cloud to cloud.
- Proven ability to lead diverse teams and manage end-to-end analytics projects.
- Exceptional communication and collaboration skills for working with executives and stakeholders.
- Strong sense of ownership, accountability, adaptability, and commitment to results.
- Expertise identifying risks and implementing effective mitigation strategies.
Technologies
- Azure, Qlik Replicate, Databricks Delta Live Tables, Unity Catalog, Databricks, Structured Streaming, SQL, Python, Scala, Spark-SQL
- ADF, ADLS, Stream Analytics, Synapse Analytics
- Hadoop, Kafka
- PowerBI, QlikSense
- AI/ML, NLP, GenAI
- AGILE, KANBAN
- MuleSoft, Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Products/TS - Products Investments
- VIP (Voyage Management System)
- IMOS, ETRMs
- MuleSoft
Additional Notes
- US Citizenship or Permanent US Residency is required to qualify for this position.
- The ECI Group uses AI-assisted tools in the early stages of application review, with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Group’s applicable data protection policies and Privacy Notice.
Nice to Have
- Familiarity with big data tools like Hadoop, Kafka, and Spark.
- Experience with visualization tools such as PowerBI or QlikSense.
- Knowledge of AI/ML, NLP, and GenAI concepts.
- Familiarity with AGILE and KANBAN methodologies.
Project and Business Context
The engagement includes work supporting the T& S Future Freight Platform – Jigsaw initiative, which involves building a new Freight Platform to replace RADAR and support freight business processes and customer needs. The project will start with a detailed design phase through Stage Gate 2.
Key deliverables include an approved solution architecture that supports an industry standard Voyage Management System (VIP) alongside other IT components to close capability gaps identified during the study phase. The platform will act as a hub to integrate necessary data between integration points, with attention to how the solution captures freight exposure management (including potential use of existing ETRMs), and how reporting needs will be met using existing capabilities such as IMOS.
Business Applications
- RADAR Bulk Upload, RADAR Mail, RADAR-SAFE, RadarMsgServer, RADAR_app
- VESON VIP T&S Pr| Data Modelling, Data Engineering, Data Integration
