Senior Data Engineer

Senior Data Engineer role at NTT Ltd in Charlotte, North Carolina (onsite). You will design and optimize scalable data pipelines using Databricks and related technologies to support AI-driven document processing and ontology-based data models, producing standardized outputs for downstream underwriting workflows.

What you’ll work on

Build end-to-end ingestion and processing flows that turn complex business documents into structured, validated data. You’ll integrate LLMs and AI models into extraction workflows, implement classification and reconciliation logic, and ensure outputs are dependable enough for production application consumption. You will also contribute to ontology mapping so insurance entities and concepts are represented consistently across the pipeline.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Create document ingestion, classification, extraction, validation, and transformation pipelines for PDFs, emails, attachments, Excel/XLSX files, insurance applications, Statements of Values (SOVs), and other business documents.
• Integrate LLMs and AI models into processing workflows to extract structured information from unstructured and semi-structured documents.
• Troubleshoot false positives, incorrect metadata, and inaccurate field values produced by LLM-based extraction pipelines.
• Implement document classification, source-specific extraction rules, and data validation, reconciliation, and quality-control mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
• Design and implement ontology-based data models covering insurance entities, business concepts, and relationships.
• Map document attributes to standardized ontology concepts and metadata structures.
• Optimize processing for large and complex Excel/XLSX and SOV files, including performance, scalability, and concurrency challenges.
• Implement reliable data persistence using PostgreSQL, Lakebase, or related technologies.
• Develop and maintain APIs and data products supporting downstream integration.
• Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
• Monitor production pipelines, troubleshoot issues, and continuously improve reliability, performance, and data quality.

Requirements

• 7+ years of experience in data engineering, software development, or production data pipeline development.
• 4+ years hands-on experience with Databricks and scalable data-processing pipelines.
• 5+ years experience with Python, PySpark, and SQL.
• 3+ years processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
• 2+ years integrating LLMs or AI models into document-processing or data-extraction pipelines.
• 3+ years implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
• Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
• 2+ years designing JSON outputs, APIs, and data products for downstream consumption.
• 2+ years experience with PostgreSQL, Lakebase, or similar relational database technologies.
• 2+ years troubleshooting production performance, scalability, concurrency, and data-processing issues.
• Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
• Strong analytical, problem-solving, communication, and cross-functional collaboration skills.

Technologies

• Databricks
• Python
• PySpark
• SQL
• LLMs
• AI models
• PostgreSQL
• Lakebase
• JSON
• APIs
• Excel/XLSX
• PDFs
• emails

Nice to have

• Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
• Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
• Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
• Experience building custom ontologies, knowledge graphs, or semantic data layers.
• Experience with document AI, OCR, information extraction, and document intelligence platforms.
• Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience optimizing large-scale document ingestion and distributed processing workloads.