Scotiabank is seeking a Senior AI Engineer in Dallas, TX to design, build, and operate AI and agentic solutions supporting Canadian Banking, International Banking, GBM, GWM, Fraud, and AML use cases. This is a hands-on engineering role focused on dependable production delivery using Databricks and Azure-native tooling.
Responsibilities
- Join discovery sessions with business stakeholders to understand workflows, data sources, pain points, and opportunities for AI-enabled improvements.
- Convert requirements into technical designs, implementation plans, estimates, and testable acceptance criteria in partnership with product owners and technical leads.
- Evaluate data readiness, integration dependencies, and technical feasibility; develop focused prototypes to validate assumptions before production implementation.
- Define boundaries for agent autonomy, including human review, approvals, and deterministic business rule constraints.
- Build LLM-powered applications and agentic workflows using tool calling, enterprise APIs, retrieval-augmented generation, structured outputs, and state management.
- Implement document ingestion and processing, including parsing, chunking, embeddings, indexing, retrieval, and reranking to deliver permission-aware context.
- Integrate predictive models and business rules with AI workflows with data scientists and domain experts.
- Increase workflow reliability by implementing validation, timeouts, retries, idempotency, and fallback paths to prevent unintended repeated actions.
- Evaluate model and orchestration options against task quality, latency, cost, and operational requirements.
- Develop and maintain batch and streaming pipelines using Azure Databricks, Apache Spark, Delta Lake, Python, and SQL.
- Apply enterprise data quality standards, including schema validation, lineage, and access controls using Unity Catalog where applicable.
- Create tested APIs and services that connect AI applications to enterprise data, business systems, and approved tools.
- Write maintainable code with clear interfaces, unit and integration tests, documentation, and peer review; troubleshoot complex issues across application and data layers.
- Use MLflow to track experiments, register models, version prompts and configurations, and retain evaluation results and deployment evidence.
- Build evaluation datasets and automated checks for model performance, retrieval relevance, grounded responses, tool execution, and end-to-end task completion.
- Implement CI/CD pipelines, automated regression testing, environment configuration, and controlled releases with rollback support.
- Containerize services with Docker and deploy to Kubernetes, including Azure Kubernetes Service (AKS), using approved infrastructure-as-code patterns.
- Instrument applications for tracing, logs, and metrics; monitor AI quality, data and model drift where applicable, latency, availability, and consumption costs.
- Investigate production incidents, perform root-cause analysis, maintain runbooks, and support operations under the team’s agreed support model.
- Implement least-privilege access, secure secrets management, data protection, and audit logging using enterprise security and Azure identity standards.
- Apply safeguards for prompt injection, sensitive data disclosure, unsafe tool execution, and unauthorized agent actions.
- Implement human approval and escalation steps for sensitive decisions or actions, preserving traceability of inputs, outputs, and tool activity.
- Coordinate with cybersecurity, model risk, compliance, and data governance partners to address required controls and remediation items.
- Align component designs with enterprise architecture and reusable platform capabilities with Principal Engineers and architects.
- Collaborate with data engineers, data scientists, platform engineers, and business teams to resolve dependencies and deliver integrated solutions.
- Mentor engineers through code reviews, pairing, and knowledge sharing; contribute reusable components and practical engineering guidance.
- Communicate progress, blockers, technical trade-offs, and delivery risks clearly and early.
Requirements
- Strong professional experience in software, data, or machine learning engineering, including hands-on delivery of production AI or machine learning solutions.
- Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
- Hands-on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
- Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
- Experience with MLflow and MLOps practices including experiment tracking, model registration, versioning, deployment, and monitoring.
- Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
- Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, with the ability to deploy and troubleshoot services.
- Ability to independently deliver well-defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
- Understanding of AI evaluation, data privacy, secure tool use, human oversight, and limitations of probabilistic model outputs.
Technology Environment
Data engineering: Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, Azure Data Lake Storage, Azure Data Factory, and Azure Event Hubs.
AI and machine learning: Azure-hosted AI models and services, MLflow, machine learning frameworks, agent orchestration frameworks, retrieval, and vector search.
Software and cloud: Python, SQL, REST APIs, Microsoft Azure, Docker, Kubernetes, AKS, Azure Functions, and Azure API Management.
Engineering operations: Azure DevOps or GitHub Actions, Terraform or Bicep, Azure Monitor, Application Insights, Key Vault, and Microsoft Entra ID.
Technologies
Databricks, MLflow, Kubernetes, Azure, Azure Kubernetes Service (AKS), Azure-native stack, Azure Databricks, Apache Spark, Delta Lake, Python, SQL, Unity Catalog, Docker, CI/CD pipelines, Git, REST APIs, Azure Data Lake Storage, Azure Data Factory, Azure Event Hubs, Azure Functions, Azure API Management, Azure DevOps, GitHub Actions, Terraform, Bicep, Azure Monitor, Application Insights, Key Vault, Microsoft Entra ID, tool calling, retrieval-augmented generation, structured outputs, state management, retrieval, embeddings, indexing, reranking, vector search, document ingestion, chunking, semantic models, knowledge graph integrations, Model Context Protocol (MCP), event-driven architectures, agent orchestration frameworks, tracing, logs, metrics, audit logging, prompt injection, human approval.
Benefits
- Holistic well-being
- Flexible benefit programs to support unique family, financial, physical, mental, and social health needs
Preferred Qualifications
- Experience in financial services, particularly banking, capital markets, wealth management, fraud detection, or AML.
- Experience implementing retrieval and vector search, document processing, semantic models, or knowledge graph integrations.
- Familiarity with Model Context Protocol (MCP), event-driven architectures, and agent orchestration frameworks.
- Experience working in regulated environments with model risk, audit, and data governance requirements.
- Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.
Location
- United States, Texas, Dallas (onsite)
Hiring Manager
Prabhu Gururaj
