Principal AI Engineer

Lead end-to-end AI and agentic solutions at Scotiabank in a hands-on Principal Engineer role based in Dallas, TX 75219 (onsite). You will help shape discovery, architecture, and delivery of AI capabilities across major banking domains, moving from early experimentation through production deployment and ongoing operations. Your work spans data engineering, AI/ML engineering, software engineering, and MLOps, with a clear focus on responsible AI and enterprise governance.

What you’ll deliver

• Partner with line-of-business stakeholders, product owners, and domain experts to understand business processes, operational challenges, and opportunities for AI-driven improvement.
• Lead discovery workshops to identify and prioritize AI and agentic use cases based on business value, technical feasibility, data readiness, risk, and delivery effort.
• Translate opportunities into solution designs, delivery roadmaps, technical backlogs, and measurable acceptance criteria.
• Determine when autonomous execution is appropriate and where deterministic controls, human review, or approval are required.
• Own end-to-end technical architecture across data ingestion, processing, model development, retrieval, agent orchestration, application integration, and production operations.
• Design agentic workflows that incorporate tool use, enterprise APIs, retrieval-augmented generation, state management, and human oversight.
• Establish reusable engineering patterns, reference architectures, and shared components to accelerate delivery across business lines.
• Lead architecture and code reviews, resolve complex engineering challenges, and guide teams through technical trade-offs.
• Build critical components and reference implementations while mentoring engineers and influencing technical decisions across teams.
• Design scalable batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, SQL, Python, and Azure-native services.
• Establish data quality, lineage, metadata, access controls, and curated data products for analytics, machine learning, and AI applications.
• Develop and integrate machine learning models, LLM-powered applications, retrieval services, and agentic workflows into enterprise systems.
• Define evaluation approaches for predictive performance, retrieval quality, response accuracy, tool execution, and end-to-end task completion.
• Establish lifecycle practices for experiments, models, prompts, agents, and evaluation datasets using MLflow and supporting engineering tools.
• Implement automated testing, CI/CD, infrastructure as code, deployment controls, rollback strategies, and environment management.
• Design and deploy containerized services using Docker and Kubernetes, including Azure Kubernetes Service (AKS).
• Define production monitoring for data and model drift, AI quality, latency, availability, cost, and operational failures.
• Partner with platform engineering, cybersecurity, and operations teams to deliver resilient services with clear ownership, support models, and service-level objectives.
• Embed privacy, security, auditability, explainability, and responsible AI requirements into solution design and delivery.
• Partner with model risk management, compliance, data governance, and cybersecurity teams throughout the delivery lifecycle.
• Implement appropriate controls for sensitive data access, prompt injection, unsafe tool execution, and unintended agent actions.
• Ensure production solutions maintain traceable decisions, evaluation evidence, and appropriate human accountability.

Required qualifications

• Extensive experience in software, data, or machine learning engineering, with demonstrated technical leadership across complex enterprise initiatives.
• Proven ability to take AI or machine learning solutions from business discovery through production deployment and operation.
• Strong hands-on proficiency in Python, SQL, API design, distributed systems, and modern software engineering practices.
• Deep experience with Databricks, Apache Spark, Delta Lake, and enterprise data pipeline architecture.
• Practical experience building generative AI and agentic applications, including retrieval, tool integration, orchestration, and evaluation.
• Experience with MLflow and MLOps practices, including experiment tracking, model registration, deployment, monitoring, and lifecycle governance.
• Strong knowledge of Azure architecture, identity, networking, security, and cloud-native application delivery.
• Experience with Docker, Kubernetes, CI/CD pipelines, automated testing, and infrastructure as code.
• Ability to communicate technical trade-offs clearly to executives, business stakeholders, architects, and engineering teams.
• Demonstrated ability to lead through influence and raise engineering standards across organizational boundaries.

Benefits

• Holistic well-being
• Flexible benefit programs designed to help support your unique family, financial, physical, mental, and social health needs

Preferred qualifications

• Experience in banking or financial services, particularly banking operations, capital markets, wealth management, fraud detection, or AML.
• Experience delivering AI solutions within regulated environments and collaborating with model risk and governance functions.
• Familiarity with Unity Catalog, enterprise metadata platforms, semantic models, vector search, and knowledge graphs.
• Experience with event-driven architectures, secure agent tool interfaces, and Model Context Protocol (MCP).
• Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Technology environment

• Data engineering: Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, Azure Data Lake Storage, Azure Data Factory, Azure Event Hubs.
• AI and machine learning: Azure-hosted AI models and services, MLflow, machine learning frameworks, agent orchestration frameworks, retrieval and vector search.
• Application engineering: Python, SQL, REST APIs, microservices, and event-driven integration.
• Cloud and runtime: Microsoft Azure, Docker, Kubernetes, AKS, Azure Functions, and Azure API Management.
• Engineering operations: Azure DevOps or GitHub Actions, Terraform or Bicep, Azure Monitor, Application Insights, Key Vault, and Microsoft Entra ID.

Requisition ID: 278902

Purpose: Scotiabank seeks a Principal Engineer to lead discovery, architecture, and delivery of AI and agentic solutions across Canadian Banking, International Banking, Global Banking and Markets (GBM), Global Wealth Management (GWM), Fraud, and Anti-Money Laundering (AML). This is a hands-on technical leadership role spanning data engineering, AI engineering, software engineering, and MLOps, partnering with business leaders to translate complex workflows into practical solutions from early experimentation through production deployment and ongoing operation. The role requires engineering judgment across traditional automation, machine learning, generative AI, and agentic workflows, with enterprise expectations for security, reliability, explainability, and measurable business value.

If your experience is closely related but doesn’t align perfectly with every qualification, Scotiabank encourages you to apply.