Sr Data Engineer

The Sr Data Engineer will join the founding team of Baker Tilly’s AI Product Group (APG), contributing to scalable, secure, and reliable data infrastructure for AI applications and intelligent agents at the Santa Clara, CA onsite location.

Role Focus

This position emphasizes governed data foundations and production-grade engineering for AI-ready datasets. Work includes lakehouse architecture, knowledge graph and retrieval capabilities, and evaluation-driven data engineering to improve grounding accuracy and continuous quality improvements.

Responsibilities

• Design and build scalable batch and incremental data pipelines that transform structured and unstructured information into clean, validated, business-ready datasets within a modern lakehouse environment.
• Develop and optimize lakehouse architecture, including transformation and modeling layers, medallion architecture, and integration of enterprise and legacy data sources.
• Ensure data quality, lineage, and provenance through automated validation, reconciliation, lineage tracking, and metadata management across the data lifecycle.
• Create data models and identity resolution capabilities using scalable models, data contracts, and entity-resolution processes to establish consistent client, engagement, and document identities.
• Implement data governance and access controls, supporting data cataloging, classification, permissions, and access management for sensitive information and authorized AI agent use.
• Build and maintain knowledge graph and retrieval capabilities, including knowledge graph structures, search indexes, relationships, and retrieval pipelines for relevant and properly governed enterprise information.
• Support AI data readiness by building pipelines that capture AI agent activity, practitioner feedback, reviews, and corrections to support ground-truth datasets and evaluation.
• Drive evaluation-based data engineering by establishing measurable quality standards, testing frameworks, benchmarks, and validation processes for extraction, retrieval, and grounding accuracy.
• Maintain data freshness and reliability using incremental processing, change data capture, versioning, and monitoring so data products, knowledge graphs, and search indexes remain current.
• Own production delivery and operational support by deploying, monitoring, troubleshooting, and maintaining production data pipelines with strong engineering practices, automated testing, code reviews, and operational documentation.
• Support knowledge transfer and technical ownership by working with external partners to transition solutions into internally owned, documented, and maintainable environments, including reusable code, tests, and runbooks.
• Collaborate across global and cross-functional teams, partnering with AI engineering, platform and production engineering, enterprise data teams, subject matter experts, and engineering teams in India.
• Participate in special projects and perform other duties as assigned to advance the firm’s AI and data platform capabilities.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.
• 3+ years of hands-on data engineering or software engineering experience, including demonstrated success delivering and maintaining production data pipelines.
• Experience independently building and operating production pipelines on a modern lakehouse platform, including ownership of data processing, transformation, and operational reliability.
• Experience implementing data quality, lineage, or data contracts in production pipelines, including automated testing, reconciliation, validation, or traceability.
• Strong proficiency in Python and SQL, with proven ability to develop, optimize, test, and maintain production-quality data engineering solutions.
• Strong understanding of data modeling principles for scalable and maintainable structures that accommodate evolving business requirements and source systems.
• Experience designing and developing batch and incremental pipelines for structured and unstructured enterprise data.
• Solid software engineering fundamentals, including version control, automated testing, code reviews, debugging, and reliable, repeatable workflow development.
• Ability to validate solutions through measurable outcomes, including automated tests, benchmarks, experiments, performance metrics, or comparative evaluations.
• Strong written and verbal communication skills, including documentation, explaining complex concepts, and collaborating with technical and nontechnical stakeholders.
• Ability to work with business subject matter experts and globally distributed stakeholders to translate requirements into dependable data solutions.
• Capability to manage competing priorities, adapt to evolving requirements, and deliver high-quality solutions in a fast-paced collaborative environment.
• Ability to incorporate AI-assisted and agentic development tools into daily development practices (prior AI experience is not required).
• No professional certifications or licenses required.

Preferred Technology and Domain Experience

• Modern lakehouse technologies such as Databricks, Apache Spark, Delta Lake, or medallion architecture.
• Cloud-based data platforms, preferably Microsoft Azure.
• Enterprise data governance capabilities, including data catalogs, classification, access controls, lineage, and tools such as Unity Catalog or comparable solutions.
• Knowledge graphs and retrieval-based applications, including graph databases, graph data modeling, vector search, hybrid search, or retrieval pipelines.
• Unstructured data processing experience such as document extraction, metadata management, content chunking, or search indexing.
• Entity resolution, master data management, change data capture, or incremental data processing experience.
• Understanding of data provenance and traceability requirements in complex enterprise or regulated environments.

Technologies

Python, SQL, Databricks, Apache Spark, Delta Lake, medallion architecture, Microsoft Azure, Unity Catalog, knowledge graphs, graph databases, graph data modeling, vector search, hybrid search, retrieval-based applications, document extraction, content chunking, search indexing, entity resolution, master data management, change data capture, incremental data processing, LLM systems

Pay

USD 185,000 - 230,000 per year. National pay rate range: $185,000 to $230,000. In California: $185,000 to $230,000.

Location

Santa Clara, CA (onsite)