Multimodal AI Engineer (Robotics & Foundation Models)

Multimodal AI Engineer role focused on developing Vision-Language-Action (VLA) foundation models for autonomous robotic systems in a hybrid United States setting.

Responsibilities

• Develop, fine-tune, and adapt Vision Transformers (ViTs) and multimodal foundation models for robotic perception and action generation.
• Implement action-chunking architectures, including Action Chunking with Transformers (ACT) and Diffusion Policy, to support long-horizon robotic task planning.
• Build multi-camera fusion pipelines to combine visual inputs into coherent representations for VLA perception and decision-making.
• Create and optimize multimodal feature extraction pipelines that connect visual perception to downstream robotic planning and action-generation systems.
• Integrate computer vision workflows, including face recognition and perimeter threat assessment, into multimodal AI pipelines.
• Apply parameter-efficient fine-tuning (PEFT) approaches, including LoRA, to adapt foundation models for specific robotic applications.
• Use Transformer architectures, visual embeddings, and vector search techniques to improve multimodal representation and information retrieval.
• Profile and optimize multimodal models using pruning and other model compression techniques to improve inference efficiency.
• Prepare and optimize model weights for deployment using NVIDIA TensorRT and related inference optimization tools.
• Evaluate performance through systematic experiments, benchmarking, and iterative improvements.
• Collaborate with researchers and engineering teams across VLA, robotics controls, and edge AI to support end-to-end system development.

Requirements

• Master’s degree or Ph.D. in Computer Science, Artificial Intelligence, Robotics, Machine Learning, Computer Vision, or related fields.
• Approximately 3–5 years of relevant experience across multimodal AI, deep learning, computer vision, foundation models, or robotic learning.
• Strong programming skills in Python with hands-on experience using PyTorch.
• Experience developing, training, or fine-tuning Transformer-based, Vision Transformer, or multimodal deep learning models.
• Understanding of vision-language models, multimodal representation learning, and integrating visual inputs into decision-making systems.
• Practical experience with PEFT, including LoRA or similar techniques.
• Familiarity with robotic learning and action-generation approaches such as ACT, Diffusion Policy, imitation learning, or reinforcement learning.
• Experience with multi-camera perception, visual feature fusion, or similar computer vision pipelines.
• Knowledge of model profiling and optimization methods such as pruning and quantization.
• Familiarity with NVIDIA DeepStream, TensorRT, or GPU-accelerated inference is an advantage.
• Knowledge of embeddings and vector search, plus multimodal data processing, is a plus.
• Strong analytical and problem-solving skills, with the ability to run experiments and translate research results into engineering solutions.
• Ability to work hybrid and be available for on-site work at a designated research location if required for future projects.

Technologies

• Python, PyTorch
• Vision Transformers (ViTs)
• Transformer architectures
• Action Chunking with Transformers (ACT)
• Diffusion Policy
• LoRA, Parameter-efficient fine-tuning (PEFT)
• Vector search
• Pruning, model compression
• NVIDIA TensorRT, NVIDIA DeepStream
• GPU-accelerated inference
• Imitation learning, reinforcement learning

Why Join

• Work on emerging technologies at the intersection of multimodal AI, foundation models, and robotics.
• Develop AI systems that connect visual perception, planning, and physical execution.
• Explore advanced approaches to VLA model adaptation, action generation, and multimodal learning.
• Contribute to real-world AI applications involving autonomous systems and intelligent perception.
• Collaborate with researchers and engineers on challenging AI research and development projects.