CrawlJobs Logo
Capital One Logo Capital One · IT - Software Development

Lead Software Engineer - Python + PySpark

United States, Richmond Employment contract 179400.00 - 225100.00 USD / Year · Job Posted July 03, 2026
Apply Position
Job Link Share

Job Description

Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Full Stack Software Engineers who are passionate about marrying data with emerging technologies. As a Capital One Lead Software Engineer, you'll have the opportunity to be on the forefront of driving a major transformation within Capital One.

Job Responsibility

  • Lead a portfolio of diverse technology projects and a team of developers with deep experience in distributed microservices, and full stack systems to create solutions that help meet regulatory needs for the company
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, mentoring other members of the engineering community
  • Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment
  • Utilize programming languages like JavaScript, Java, HTML/CSS, TypeScript, SQL, Python, and Go, Open Source RDBMS and NoSQL databases, Container Orchestration services including Docker and Kubernetes, and a variety of AWS tools and services

Requirements

  • Bachelor's Degree
  • At least 4 years of experience in software engineering (Internship experience does not apply)
  • At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

Nice to have

  • Master's Degree
  • 7+ years of experience in at least one of the following: Python, SQL, TypeScript, PySpark, JavaScript, Java or Go
  • 3+ years of experience with AWS, GCP, Microsoft Azure, or another cloud service
  • 4+ years of experience in open source frameworks
  • 1+ years of people management experience
  • 2+ years of experience in Agile practices

What we offer

  • Performance based incentive compensation
  • Health, financial and other benefits
Capital One - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Lead Software Engineer - Python + PySpark

8 matching positions

Lead Python Full Stack Data Engineer

We are assembling an A-team of highly skilled, autonomous, and visionary enginee...
Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of progressive, hands-on experience as a Senior/Lead Data Engineer
  • Expert-level proficiency in Python
  • Deep expertise in developing highly optimized, scalable, and production-grade PySpark applications
  • Deep architectural understanding and extensive hands-on experience with the entire Apache Spark ecosystem (Spark Core, Spark SQL, Spark Streaming, Spark MLlib)
  • Advanced proficiency with Hive for enterprise data warehousing
  • Expert knowledge of distributed computing fundamentals, HDFS, and other components of the Hadoop ecosystem
  • Master-level proficiency in SQL, complex query optimization, and advanced data warehousing concepts
  • Extensive experience with various data storage formats (e.g., Parquet, ORC, Avro) and leading data lake solutions (e.g., Delta Lake, Iceberg)
  • Proven experience with enterprise-grade NoSQL databases (e.g., Cassandra, MongoDB, HBase)
  • Expert-level experience with Apache Kafka
Job Responsibility
Job Responsibility
  • Lead and Architect end-to-end data solutions
  • Drive Strategic Initiatives within small, co-located squads
  • Act as a Player/Coach
  • Design, Develop, and Optimize highly efficient and resilient data ingestion, processing, and transformation pipelines using advanced Python and PySpark techniques
  • Architect and Implement sophisticated data storage solutions leveraging a diverse set of big data technologies
  • Champion Data Modeling and Governance
  • Strategically Engage with data consumers, data scientists, and business stakeholders
  • Lead the Implementation of real-time data streaming and complex event-driven architectures
  • Enforce and Evolve Best Practices in data engineering and software development
  • Exhibit High Autonomy and Agency
  • Fulltime
Read More
Arrow Right

Senior Software Engineer III

Robert Half is seeking a Senior Software Engineer III to join our Platform Engin...
Location
Location
United States , San Ramon
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong Python expertise (top priority)
  • Experience with PySpark and data processing workflows
  • Hands-on AWS experience (must be able to independently build and manage services like EC2)
  • Experience with CI/CD tools (e.g., Jenkins, Git) and DevOps practices
  • Solid understanding of cloud infrastructure, monitoring, and automation
  • Proven ability to troubleshoot production issues and maintain system reliability
  • 6+ years of experience in software engineering, platform engineering, or infrastructure
  • Experience working in large-scale, enterprise environments
  • Strong problem-solving skills and ability to manage competing priorities
  • Experience mentoring entry level engineers and leading technical initiatives
Job Responsibility
Job Responsibility
  • Design, build, and deploy scalable infrastructure and platform components supporting cloud-native applications
  • Lead development of CI/CD pipelines, infrastructure-as-code (IaC), and automation frameworks
  • Own platform systems end-to-end, ensuring reliability, scalability, and performance
  • Troubleshoot and resolve production issues, including outages, deployment failures, and infrastructure instability
  • Support on-call rotation (every 3 weeks), handling P1 incidents and coordinating with external vendors (e.g., Microsoft)
  • Collaborate across engineering, application, and security teams to improve platform capabilities
  • Mentor entry level engineers, conduct code reviews, and drive engineering best practices
  • Contribute to system design, documentation, and continuous improvement initiatives
What we offer
What we offer
  • Medical, vision, dental, and life and disability insurance
  • 401(k) plan
  • Fulltime
Read More
Arrow Right

Staff II Software Engineer AI/ML Ops

We're looking for a Lead Data Engineer to design, build, and optimize data pipel...
Location
Location
United States , Pleasanton
Salary
Salary:
245000.00 - 307000.00 USD / Year
blackline.com Logo
BlackLine
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong programming skills in languages such as Python, Java, or Scala
  • Expertise in ML frameworks (TensorFlow, PyTorch, scikit-learn) and orchestration tools (Airflow, Kubeflow, Vertex AI, MLflow)
  • Proven experience operating production pipelines for ML and LLM-based systems across cloud ecosystems (GCP, AWS, Azure)
  • Deep familiarity with LangChain, LangGraph, ADK or similar agentic system runtime management
  • Strong competencies in CI/CD, IaC, and DevSecOps pipelines integrating testing, compliance, and deployment automation
  • Hands-on with observability stacks (Prometheus, Grafana, Newrelic) for model and agent performance tracking
  • Understanding of governance frameworks for Responsible AI, auditability, and cost metering across training and inference workloads
  • Proficiency in containerization technologies (e.g., Docker, Kubernetes)
  • Proficient in scripting languages (e.g., Bash, python) for automation
  • Experience with workflow orchestration tools (e.g., Apache Airflow)
Job Responsibility
Job Responsibility
  • Lead data pipeline development: Build and maintain PySpark ETL pipelines with high data quality and performance
  • Manage integrations: Establish robust connections to client data sources via APIs and tools like FiveTran, Plaid, and BlackLine's own internal connector ecosystem
  • Ensure reliability: Monitor pipeline performance, automate testing, and validate data accuracy
  • Optimize for scale: Implement performance improvements (e.g., CDC mechanisms, indexing strategies) for large-scale datasets
  • Collaborate & innovate: Work with business stakeholders to refine data requirements and integrate cutting-edge AI and big data technologies
  • Partner with data science, security, and product teams to set evaluation and governance standards (Guardrails, Bias, Drift, Latency SLAs)
  • Mentor senior engineers and drive design reviews for ML pipelines, model registries, and agentic runtime environments
  • Lead incident response and reliability strategies for ML/AI systems
  • Collaborate with development teams to integrate AI solutions into existing workflows and applications
  • Ensure seamless integration with different platforms and technologies
What we offer
What we offer
  • Short-term and long-term incentive programs
  • Robust offering of benefit and wellness plans
  • Fulltime
Read More
Arrow Right
New

Senior Vice President - AI Lead Engineer - Banking Technology

Citi's Banking Technology organization is seeking an exceptional full stack hand...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep hands-on experience in engineering and executing scalable enterprise solutions
  • Expert-level proficiency in Python (e.g., FastAPI, Django, Flask, PySpark)
  • Solid understanding of core AI concepts, including knowledge representation, automated planning, decision-making under uncertainty, and multi-agent systems
  • Hands-on experience with relevant frameworks (e.g., Google ADK, LanGraph, LangChain, AutoGen, CrewAI, N8N)
  • Extensive experience with machine learning frameworks (e.g., TensorFlow, PyTorch) and relevant libraries (e.g., Scikit-Learn, NumPy, Pandas)
  • Proven experience in creating, deploying, and integrating MCPs (Model Context Protocol) into agentic AI systems
  • Deep familiarity with large language models (LLMs) such as ChatGPT, Claude, Gemini, and Llama, including their application within agentic systems
  • Demonstrated experience in designing and implementing robust APIs for AI services
  • Proficient in software development best practices, including version control (Git), CI/CD pipelines, comprehensive testing, and code reviews
  • Strong understanding of agile methodologies, application resiliency, and security principles applied to complex AI projects
Job Responsibility
Job Responsibility
  • Spearhead the end-to-end design, hands-on implementation, and deployment of scalable, robust agentic AI frameworks and generative AI solutions for critical banking use cases, ensuring high performance, reliability, and security
  • Build full-stack applications that seamlessly integrate state-of-the-art ML/LLM tools and services into comprehensive AI solutions
  • Champion an MVP-first approach, focusing on rapid iterations, hypothesis testing, and A/B experiments to achieve continuous improvement and quick 'time to value'
  • Design and implement rigorous metrics and evaluation strategies for AI system and agent performance, driving continuous optimization and behavioral improvement
  • Proactively explore, prototype (POC), and integrate the latest advancements in AI, particularly in agent-based systems, autonomous AI, and generative AI technologies
  • Provide hands-on technical leadership, guiding architectural decisions, and fostering collaboration with cross-functional teams including AI researchers, data scientists, product managers, and software engineers to integrate and scale AI solutions across Citi's products and services
  • Exhibit excellent communication and collaboration skills, effectively engaging with diverse stakeholders
What we offer
What we offer
  • Discover the top benefits offered to our global workforce, designed to support your well-being, growth and work-life balance
  • Learn About Global Benefits
  • Fulltime
Read More
Arrow Right

Senior Associate Software Engineer Databricks Developer

Amgen is looking for highly motivated expert Senior Data Engineer who can own th...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Any degree with 5 - 9 years of experience in Computer Science, IT or related field
  • Strong solution design and problem solving skills
  • Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, Redshift, and Scaled Agile methodologies
  • Proficiency in workflow orchestration, performance tuning on big data processing
  • Strong understanding of AWS services
  • Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures
  • Ability to quickly learn, adapt and apply new technologies
  • Strong problem-solving and analytical skills
  • Excellent communication and teamwork skills
  • Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices
Job Responsibility
Job Responsibility
  • Design, develop, and maintain scalable data pipelines using AWS (Redshift, S3, Glue, Lambda) and Databricks (Spark, Delta Lake) to support enterprise analytics and reporting
  • Architect and implement robust data models (dimensional and normalized) to enable high-performance querying and optimized reporting in Redshift
  • Build and optimize batch and real-time data processing frameworks, leveraging Spark Structured Streaming and cloud-native services
  • Lead data ingestion, transformation, and orchestration workflows ensuring data quality, reliability, and performance at scale
  • Perform advanced data analytics and root cause analysis to troubleshoot data discrepancies, performance issues, and pipeline failures
  • Ensure data security, compliance, and role-based access control (RBAC) across data environments
  • Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets
  • Develop and drive continuous improvements in CI/CD pipelines for automated data pipeline deployments, automated testing, version control, and monitoring for data platforms in a cloud-native environment
  • Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals
  • Stay up to date with emerging data technologies and best practices, ensuring continuous improvement
Read More
Arrow Right

AI Lead Engineer for Banking Technology

Citi's Banking Technology organization is seeking an exceptional full stack hand...
Location
Location
United States , Jersey City; New York
Salary
Salary:
176720.00 - 265080.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep hands-on experience in engineering and executing scalable enterprise solutions
  • Expert-level proficiency in Python (e.g., FastAPI, Django, Flask, PySpark) or Java (e.g., Spring Boot, Spring Cloud, Spring Security)
  • Proficiency in UI (e.g., Angular, React, Next.js, Node.js, TypeScript) for full-stack development
  • Proficiency in database technologies, such as Oracle, Postgres, or MongoDB
  • Solid understanding of core AI concepts, including knowledge representation, automated planning, decision-making under uncertainty, and multi-agent systems
  • Hands-on experience with relevant frameworks (e.g., LanGraph, LangChain, AutoGen, CrewAI, N8N)
  • Extensive experience with machine learning frameworks (e.g., TensorFlow, PyTorch) and relevant libraries (e.g., Scikit-Learn, NumPy, Pandas)
  • Proven experience in creating, deploying, and integrating MCPs (Model Context Protocol) into agentic AI systems
  • Deep familiarity with large language models (LLMs) such as ChatGPT, Claude, Gemini, and Llama, including their application within agentic systems
  • Demonstrated experience in designing and implementing robust APIs for AI services
Job Responsibility
Job Responsibility
  • Spearhead the end-to-end design, hands-on implementation, and deployment of scalable, robust agentic AI frameworks and generative AI solutions for critical banking use cases, ensuring high performance, reliability, and security
  • Build full-stack applications that seamlessly integrate state-of-the-art ML/LLM tools and services into comprehensive AI solutions
  • Champion an MVP-first approach, focusing on rapid iterations, hypothesis testing, and A/B experiments to achieve continuous improvement and quick "time to value."
  • Design and implement rigorous metrics and evaluation strategies for AI system and agent performance, driving continuous optimization and behavioral improvement
  • Proactively explore, prototype (POC), and integrate the latest advancements in AI, particularly in agent-based systems, autonomous AI, and generative AI technologies
  • Provide hands-on technical leadership, guiding architectural decisions, and fostering collaboration with cross-functional teams including AI researchers, data scientists, product managers, and software engineers to integrate and scale AI solutions across Citi's products and services
  • Exhibit excellent communication and collaboration skills, effectively engaging with diverse stakeholders
What we offer
What we offer
  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
  • discretionary and formulaic incentive and retention awards
  • Fulltime
Read More
Arrow Right

AI Lead Engineer for Banking Technology

Citi's Banking Technology organization is seeking an exceptional full stack hand...
Location
Location
United States , New York; Jersey City
Salary
Salary:
176720.00 - 265080.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Deep hands-on experience in engineering and executing scalable enterprise solutions
  • Expert-level proficiency in Python (e.g., FastAPI, Django, Flask, PySpark) or Java (e.g., Spring Boot, Spring Cloud, Spring Security)
  • Proficiency in UI (e.g., Angular, React, Next.js, Node.js, TypeScript) for full-stack development
  • Proficiency in database technologies, such as Oracle, Postgres, or MongoDB
  • Solid understanding of core AI concepts, including knowledge representation, automated planning, decision-making under uncertainty, and multi-agent systems
  • Hands-on experience with relevant frameworks (e.g., LanGraph, LangChain, AutoGen, CrewAI, N8N)
  • Extensive experience with machine learning frameworks (e.g., TensorFlow, PyTorch) and relevant libraries (e.g., Scikit-Learn, NumPy, Pandas)
  • Proven experience in creating, deploying, and integrating MCPs (Model Context Protocol) into agentic AI systems
  • Deep familiarity with large language models (LLMs) such as ChatGPT, Claude, Gemini, and Llama, including their application within agentic systems
  • Demonstrated experience in designing and implementing robust APIs for AI services
Job Responsibility
Job Responsibility
  • Spearhead the end-to-end design, hands-on implementation, and deployment of scalable, robust agentic AI frameworks and generative AI solutions for critical banking use cases, ensuring high performance, reliability, and security
  • Build full-stack applications that seamlessly integrate state-of-the-art ML/LLM tools and services into comprehensive AI solutions
  • Champion an MVP-first approach, focusing on rapid iterations, hypothesis testing, and A/B experiments to achieve continuous improvement and quick "time to value"
  • Design and implement rigorous metrics and evaluation strategies for AI system and agent performance, driving continuous optimization and behavioral improvement
  • Proactively explore, prototype (POC), and integrate the latest advancements in AI, particularly in agent-based systems, autonomous AI, and generative AI technologies
  • Provide hands-on technical leadership, guiding architectural decisions, and fostering collaboration with cross-functional teams including AI researchers, data scientists, product managers, and software engineers to integrate and scale AI solutions across Citi's products and services
  • Exhibit excellent communication and collaboration skills, effectively engaging with diverse stakeholders
What we offer
What we offer
  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
  • discretionary and formulaic incentive and retention awards
  • Fulltime
Read More
Arrow Right

Senior Software Engineer

Microsoft’s Azure Data engineering team is transforming analytics with products ...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • At least 10 years of software engineering experience building and shipping cloud-scale SaaS products
  • Strong programming skills in C#, Scala, Java, Python, or equivalent
  • Experience in SparkSQL , PySpark development
  • Demonstrated strength in establishing coding standards, conducting code reviews, and software design/architecture
  • Solid understanding of data engineering concepts and big data platforms, with eagerness to work deeply with Microsoft Fabric
  • Experience with Azure-based cloud services and distributed systems
  • Ability to debug complex distributed systems on Linux and/or Windows
  • Security Screening Requirements: Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Designing and building high-performance, scalable services for Microsoft Fabric
  • Developing new features and continuously improving our cloud-based SaaS platform, prioritizing efficiency, reliability, security, and operational excellence
  • Using AI-assisted development (e.g., LLM-based tools/agents) to accelerate design, implementation, testing, and code quality
  • Enhancing open-source components to meet enterprise requirements for performance, reliability, security, and supportability
  • Collaborating with cross-functional teams to integrate Fabric app capabilities with other Microsoft Fabric services
  • Serving as the Designated Responsible Individual (DRI)—the person on point for monitoring, incident response, diagnosing service degradation/downtime, and driving mitigations to restore and improve reliability
  • Leading continuous improvements to engineering processes, tools, and developer productivity
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right