Production AI Engineer - Vice President - Citi - #2166730

Discover your future at Citi
Working at Citi is far more than just a job. A career with us means joining a team of approximately 219,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.

Job Overview
The Production Engineer is a pivotal role within Citi's Technology organisation, responsible for designing, building, and operating the intelligent systems that underpin our global production environment. This is an engineering-first position at the intersection of software craftsmanship, AI-native development, and large-scale distributed systems.

As part of a multi-year transformation journey, the successful candidate will help define what production engineering looks like in an era of autonomous agentsa, generative AI, and self-ahealing infrastructure. You will be expected to write production-grade code daily, design agentic workflows, and contribute meaningfully to the evolution of our AI engineering practices across Citi's technology hub.

The role requires a comprehensive understanding of multiple areas within a function and how they interact to achieve the objectives of the function. Applies in-depth understanding of the business impact of technical contributions. Accountable for delivery of a full range of end-to-end projects. Excellent communication skills required to negotiate internally. Involved in short- to medium-term planning of actions and resources for own area.

Responsibilities:

• Designs, develops, and maintains production-grade software systems with a strong emphasis on reliability, scalability, and operational excellence across Citi's global technology estate.
• Architects and implements agentic AI workflows - building autonomous systems that can reason, plan, and act across production environments with minimal human intervention.
• Applies advanced prompt engineering techniques to integrate large language models (LLMs) into operational tooling, incident response pipelines, and developer productivity platforms.
• Leads the development of AI-native observability solutions - leveraging intelligent agents to detect anomalies, predict failures, and automate remediation before issues impact end users.
• Writes clean, well-tested, and well-documented code across the full stack; champions engineering best practices including code review, pair programming, and test-driven development.
• Drives Continuous Delivery and Automation efforts across supported applications by means of Root Cause Analysis reviews, knowledge management, performance tuning, and user training.
• Operates and evolves CI/CD pipelines, Infrastructure-as-Code tooling, and GitOps workflows to support rapid, safe delivery of software at scale.
• Collaborates with platform, data, and product engineering teams to embed AI capabilities into the production lifecycle - from deployment to decommission.
• Implements the Agile Framework through one of its implementations (SCRUM or Kanban) and ensures it integrates with overall organisation processes.
• Operates within a highly regulated financial environment, maintaining in-depth understanding of compliance requirements and their implications for system design and data handling.
• Coaches and mentors team members on AI engineering practices, prompt design patterns, and agentic system architecture - fostering a culture of continuous learning and technical excellence.
• Avidly communicates progress and project status across the organisation and ensures that stakeholders are managed appropriately throughout the execution period.
• Fosters a culture that promotes transparency and innovation for increased team productivity.

• Demonstrable experience in a critical software engineering or production engineering role with high business impact and a strong programming foundation (Java, Python, Go, or equivalent).
• Hands-on experience with AI/ML engineering - including working with LLM APIs (OpenAI, Anthropic, Gemini, or open-source equivalents), embedding models, and vector databases.
• Proven expertise in prompt engineering: designing, iterating, and evaluating prompts for production use cases including classification, summarisation, code generation, and autonomous decision-making.
• Experience designing and deploying agentic systems using frameworks such as LangChain, LangGraph, AutoGen, CrewAI, or equivalent - including multi-agent orchestration and tool-use patterns.
• Excellent engineering skills and strong understanding of Software Development Lifecycle, GitOps, and modern DevSecOps practices.
• Excellent working knowledge of key computer science concepts (networking, operating systems, virtualisation, containerisation, etc.).
• Polyglot full-stack developer mentality and ability to pick up new languages and skills.
• Excellent debugging and analytical skills: ability to isolate root cause across networking/infrastructure, application, and database stacks.
• Operational experience of deploying and running services at scale on top of Docker/Kubernetes stack and a service mesh (Istio or equivalent) is highly desirable.
• Operational experience with orchestration tools for CI/CD and Infrastructure-as-Code tooling (Terraform, CloudFormation, Pulumi, etc.) is highly desirable.
• Experience of delivering software using Agile delivery methodologies is a must (SCRUM/Kanban).
• Operational experience of using middleware technologies (MQ, Apache Kafka, etc.) to run services at scale is desirable.
• Strong experience with end-to-end observability stacks (Datadog, AppDynamics, Dynatrace, etc.) is desirable.
• Degree in Computer Science, Mathematics, Physics, or a related technical subject is desirable.
• Experience of senior stakeholder management.
• Consistently demonstrates clear and concise written and verbal communication skills.
• Ability to operate in a global environment with on-/near-/off-shore matrix reporting structures.