Software Engineer III - AI/ML Platform

Overview

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Software Engineer III at JPMorganChase within the Firmwide LLM Serving Platform team, you are an integral part of an agile team that designs, builds, and operates the services that make large language models usable at scale. This is an infrastructure-meets-ML role: you don't need to be an ML researcher, but you should be excited to learn how model architectures and inference constraints translate into real production systems. You will contribute to a living platform where we optimize performance — pushing down latency, increasing throughput, maximizing GPU utilization, and eliminating waste across the request lifecycle.

Responsibilities

• Build core backend services for LLM inference, including request routing, batching, scheduling, streaming responses, and quota/limits.
• Implement and maintain APIs and SDKs used by product and application teams across the firm.
• Profile and optimize performance end-to-end across CPU, memory, network, serialization, concurrency, GPU utilization, and caching.
• Improve reliability and operability through health checks, graceful degradation, autoscaling behaviors, incident follow-ups, and runbooks.
• Contribute to system design by breaking down ambiguous problems, proposing approaches, and making pragmatic tradeoffs.
• Add observability with metrics, tracing, logging, dashboards, and actionable alerts tied to SLOs.
• Support safe deployments through CI/CD improvements, canarying, feature flags, backward compatibility, and rollback plans.
• Learn LLM serving fundamentals — tokenization costs, KV cache, quantization, context length tradeoffs, throughput vs. latency.
• Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity across complex deliverables (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards; contributes learnings and reusable patterns to improve broader team effectiveness.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
• Add to team culture of diversity, opportunity, inclusion, and respect.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and applied experience.
• Bachelor's Degree in Computer Science or equivalent.
• Solid programming fundamentals: data structures, concurrency basics, debugging, testing.
• Comfort working in one or more of Go, Python, or TypeScript, with the ability to ramp up quickly on the others.
• Interest in distributed systems and system design, even if you haven't built large systems yet.
• Curiosity about LLMs and AI model architecture, with willingness to learn quickly.
• A measurement-driven mindset: you like profiling, benchmarking, and proving improvements with data.
• Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security.
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices.

Preferred qualifications, capabilities, and skills

• Experience with performance profiling tools such as pprof, flamegraphs, or distributed tracing systems.
• Familiarity with containers and orchestration (Docker, Kubernetes) and service-to-service networking.
• Understanding of inference concepts: batching, streaming tokens, GPU memory constraints, KV cache.
• Experience with high-throughput APIs (gRPC/HTTP), eventing/queues, or caching layers such as Redis.
• Exposure to reliability practices: SLOs/SLIs, on-call rotations, incident reviews.