Hillwood Development Company LLC is seeking a Senior Data Engineer to join its Corporate IT team in Dallas, TX (onsite). This role builds and evolves Hillwood’s enterprise Data Lakehouse on the Databricks platform, supporting secure, scalable, AI-ready data pipelines and governed data used for analytics and AI across the organization.
What you’ll do
- Design, develop, and maintain scalable Databricks data pipelines that integrate data from enterprise applications including JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, and others.
- Build and maintain data models across Bronze, Silver, and Gold layers in line with Hillwood’s medallion framework and enterprise data architecture.
- Implement incremental processing and change data capture (CDC) to keep datasets current, accurate, and cost-efficient.
- Administer and maintain Unity Catalog, including schema management, governance, permissions, and lineage tracking.
- Deliver production-ready pipelines with automated scheduling, monitoring, alerting, and recovery.
- Optimize workflows, jobs, and compute resources for improved performance, scalability, and cost.
- Establish engineering best practices including version control, code reviews, automated testing, and CI/CD.
- Create and maintain technical documentation, standards, and operational procedures to support long-term platform sustainability and knowledge transfer.
- Develop data quality frameworks, validation rules, reconciliation processes, and monitoring to ensure accuracy and reliability, including root-cause investigation for data quality issues.
- Partner with business stakeholders and data owners to define data standards, ownership, business definitions, and quality expectations, ensuring certified datasets align with governance requirements.
- Use AI-assisted development tools, and help establish standards for responsible use of AI-assisted software development.
- Prepare lakehouse assets for AI, machine learning, and natural language access, including feature stores, vector search, retrieval architectures, and document-based pipelines.
- Support AI workloads and platform capabilities such as model serving, agent frameworks, and emerging GenAI use cases, including integration of structured and unstructured data within Databricks.
- Identify and onboard new structured, unstructured, and third-party sources to expand enterprise platform value.
- Validate data outputs and business metrics with data owners prior to certification for business use, and communicate technical trade-offs to both technical and non-technical audiences.
- Serve as an internal subject matter expert for the architecture, operation, and ongoing evolution of Hillwood’s enterprise data lakehouse.
Key requirements
- Strong SQL, Python, PySpark, and modern data engineering development practices.
- Strong understanding of lakehouse architectures, data modeling principles, and enterprise data management.
- Ability to design scalable, reliable, maintainable data solutions for reporting, analytics, and AI.
- Knowledge of data governance, data quality, metadata management, and data lineage.
- Familiarity with source control, testing, code reviews, and CI/CD.
- Analytical and problem-solving skills to identify root causes and implement effective solutions.
- Ability to balance performance, scalability, maintainability, and cost.
- Effective written and verbal communication, including explaining technical concepts to non-technical audiences.
- Strong organizational skills and attention to detail, with the ability to manage multiple priorities.
- Ability to work independently while collaborating with business stakeholders, technical teams, and external partners.
- Curiosity and adaptability to evaluate emerging technologies, AI capabilities, and evolving data engineering practices.
- Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or related field required.
- Minimum five years of data engineering experience, including at least two years developing and supporting production solutions on Databricks using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
- Experience designing, building, and maintaining enterprise pipelines and lakehouse architectures in a cloud environment.
- Hands-on experience administering and using Unity Catalog, including metadata management, lineage, security, and access controls.
- Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization.
- Experience implementing data quality controls, reconciliation processes, and governance standards.
- Experience integrating data from ERP, financial, operational, or other enterprise systems.
- Experience with Microsoft Azure and Git-based tools including Azure DevOps and/or GitHub.
- Experience leveraging AI-assisted development tools as part of a professional software engineering workflow.
- Experience supporting AI/ML initiatives through data engineering such as feature engineering, vector search, RAG, and model-serving pipelines.
- Proven success partnering with business stakeholders to understand requirements, validate data, and deliver business-focused solutions.
Technologies you’ll work with
Databricks, SQL, Python, PySpark, Spark SQL, Delta Lake, Databricks Workflows, Unity Catalog, Bronze, Silver, Gold, medallion framework, change data capture (CDC), CI/CD, Git, Azure DevOps, GitHub, Microsoft Azure, JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, feature stores, vector search, retrieval architectures, document-based data pipelines, model serving, agent frameworks, GenAI, retrieval-augmented generation (RAG)
Benefits
- Medical Insurance
- Dental Insurance
- Vision Insurance
- Flexible Spending Accounts (FSA)
- Health Savings Accounts (HSA)
- Vacation time
- Sick time
- Holidays
- Paid family caregiver leave
- Mental health and wellness resources
- Life Insurance
- Disability Insurance
- 401(k) with company match
- Tuition assistance
- Additional programs to support Associates and their families
Preferred experience
- Databricks Data Engineer certification or equivalent Databricks credentials.
- Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms.
- Experience working with data in real estate, construction, energy, financial services, or related industries.
- Experience transitioning, stabilizing, and optimizing data solutions developed by third-party consultants or implementation partners.
- Experience supporting large language model (LLM), generative AI, or agent-based solution development.
- Experience designing solutions that combine structured and unstructured data for AI-enabled business applications.
