Lead Data Engineer: Python, Databricks, AWS

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. Join our innovative External Regulatory Financial Control (ERFC)/Strategic Data team at JPMorganChase where we leverage cutting-edge technology to drive data-driven decision-making and enhance business performance. We are seeking a talented and motivated Software/Data Engineer to join our team and contribute to our mission of transforming data into actionable insights.

As a Lead Software Engineer at JPMorgan Chase within the External Regulatory Financial Control (ERFC)/Strategic Data team, you will play a crucial role in designing, developing, and maintaining scalable data processing solutions using Databricks, Python, and AWS. You will collaborate with cross-functional teams to deliver high-quality data solutions that support our business objectives.

Job responsibilities

• Execute creative, data-driven software solutions, including design, development, and technical troubleshooting, with the ability to think beyond routine approaches to solve technical problems.
• Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, and Databricks on AWS, processing and transforming large-scale financial datasets for analytics and reporting.
• Develop fact and dimension data models for reporting and analytics.
• Write secure, high-quality production code, and review and debug code written by others.
• Ensure data quality, consistency, security, and lineage throughout all stages of data processing and transformation, implementing monitoring and alerting mechanisms to maintain pipeline reliability.
• Support data migration and modernization initiatives, transitioning legacy systems to cloud-based data warehouses.
• Lead communities of practice across Software Engineering to promote awareness and adoption of new technologies. Foster a team culture of diversity, opportunity, inclusion, and respect.
• Mentor and guide Associate-level engineers, supporting their technical development and ensuring consistent delivery standards across the team.
• Collaborate with business stakeholders to develop data management strategies, transforming data into insights that drive strategic decisions.
• Document data flows, logic, and transformation rules to maintain transparency and facilitate knowledge sharing.
• Lead and Participate in the full Software Development Life Cycle (SDLC), including requirements gathering, design, development, testing, deployment, and maintenance
• Identify and automate remediation of recurring issues to improve operational stability, acting as both Production Support and SRE function for data pipelines and platform services.
• Utilise AI tools to accelerate development and testing of data pipelines (e.g. GitHub CoPilot, Claude Code).

Required qualifications, capabilities, and skills

• Proven experience in data management, ETL/ELT pipeline development, and large-scale data processing.

• Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.

• Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).

• Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).

• Strong understanding of data quality, security, and lineage best practices.

• Experience with cloud-based data warehouse migration and modernization.

• Proficient in CI/CD, continuous delivery methods (Jules/Jenkins, Spinnaker, Sonar), the full Software Development Life Cycle, and Agile methodologies.

• In-depth knowledge of the financial services industry and their IT systems.

• Excellent problem-solving, troubleshooting, and analytical skills with ability to investigate data issues, identify root causes, and implement solutions.

• Strong communication and documentation abilities, with the ability to collaborate effectively with business and technical stakeholders.

Preferred qualifications, capabilities, and skills

• Knowledge of regulatory reporting and financial data aggregation techniques.
• Experience with data orchestration tools (Airflow, Step Functions, etc.).
• Databricks or AWS certifications.
• Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).
• Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.