CrawlJobs Logo
Capital One Logo Capital One · IT - Software Development

Lead Data Engineer - Data Publication and Transformation

United States, Richmond Employment contract · Job Posted June 28, 2026
Apply Position
Job Link Share

Job Description

Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. As a Capital One Lead Data Engineer, you’ll have the opportunity to be on the forefront of driving a major transformation within Capital One.

Job Responsibility

  • Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies
  • Work with a team of developers with deep experience in machine learning, distributed microservices, and full stack systems
  • Utilize programming languages like Java, Scala, Python and Open Source RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community
  • Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment
  • Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance

Requirements

  • Bachelor's Degree
  • At least 4 years of experience in application development (Internship experience does not apply)
  • At least 2 years of experience in big data technologies
  • At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

Nice to have

  • 7+ years of experience in application development including Python, SQL, Golang, Java or Scala
  • 4+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)
  • 4+ years experience with Distributed data/computing tools (MapReduce, Flink, EMR, Kafka, Spark, or MySQL)
  • 4+ year experience working on real-time data and streaming applications
  • 4+ years of experience with SQL implementation(PostgreSQL) and NoSQL implementation (DynamoDB)
  • 4+ years of data warehousing experience (Snowflake or Redshift)
  • 4+ years of experience with UNIX/Linux including basic commands and shell scripting
  • 2+ years of experience with Agile engineering practices
  • Experience leveraging interactive AI tooling to accelerate productivity, utilizing capabilities beyond basic code completion

What we offer

  • Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • Comprehensive, competitive, and inclusive set of health, financial and other benefits
Capital One - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Lead Data Engineer - Data Publication and Transformation

8 matching positions

Data Engineer – Lead

Data Engineer – Lead
Location
Location
India , Bengaluru Urban
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, OneLake, Pipelines, Dataflows Gen2, Notebooks, and Power BI integration
  • Expertise in ETL/ELT, data pipelines, distributed data processing, and cloud-scale data engineering
  • Strong SQL, Python, PySpark, and data modeling skills
  • Experience with Lakehouse, Warehouse, and Medallion Architecture
  • Understanding of Delta tables, dimensional modeling, star schema, facts, dimensions, and curated analytical datasets
  • Experience integrating structured, semi-structured, file-based, API-based, enterprise application, and cloud data sources
  • Experience with data quality, reconciliation, logging, monitoring, and error-handling frameworks
  • Experience leading technical teams and coordinating onshore/offshore delivery
  • Experience with Git, CI/CD, Azure DevOps, branching, code reviews, and release management
  • Good to Have: Experience with Azure Data Factory, Synapse, Databricks, ADLS Gen2, Azure SQL, Microsoft Purview, or related Azure services
Job Responsibility
Job Responsibility
  • Lead the design and implementation of scalable data pipelines and data processing frameworks in Microsoft Fabric
  • Define data engineering standards, development practices, naming conventions, coding guidelines, and reusable technical patterns
  • Lead implementation of Bronze, Silver, and Gold layers in the Medallion Architecture
  • Oversee ingestion, transformation, orchestration, validation, and publication of data from multiple enterprise, clinical, operational, and cloud-based sources
  • Guide development of Fabric Pipelines, Dataflows Gen2, Notebooks, Lakehouse tables, Warehouse objects, and curated datasets
  • Ensure scalability, performance, reliability, maintainability, security, monitoring, and optimization of data solutions
  • Define standards for data quality, reconciliation, logging, error handling, auditability, and lineage
  • Conduct technical design reviews, code reviews, performance reviews, and deployment readiness reviews
  • Mentor and guide data engineering teams across onshore/offshore locations
  • Collaborate with architects, platform engineers, BI teams, QA teams, AI/ML teams, functional consultants, and stakeholders
Read More
Arrow Right

Lead Data Migration Engineer

We are seeking an accomplished and detail-oriented Lead Data Migration Engineer ...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience in data engineering and data migration delivery, particularly within cloud environments
  • Strong focus on data pipeline testing, validation, and quality assurance
  • Ability to work across the full data lifecycle, with emphasis on migration and transformation
  • Strong analytical, problem-solving, and communication skills
  • Experience working in client-facing and delivery-focused environments
  • Ability to mentor junior engineers and contribute to team delivery
  • Strong hands-on experience with AWS cloud services, especially AWS Glue
  • Python / PySpark for data transformation
  • SQL for querying, validation, and reconciliation
  • YAML configuration for pipeline orchestration
Job Responsibility
Job Responsibility
  • Work as a senior engineer within data migration programmes, supporting delivery across complex transformation initiatives
  • Collaborate with architects and stakeholders to implement migration strategies and technical solutions
  • Contribute to project planning, estimation, and execution of migration deliverables
  • Design, build, and optimise data migration pipelines from legacy data warehouses to AWS-based Data Lakehouse platforms
  • Develop and maintain ETL/ELT pipelines using AWS Glue, Python/PySpark, SQL, and YAML-driven configurations
  • Implement bulk data migration processes followed by incremental/delta loads
  • Support pipeline repointing and replatforming to target cloud architectures
  • Test end-to-end ETL data solutions running on AWS services, including AWS Glue and Apache Iceberg
  • Validate data feed pipeline migrations transitioning to new AWS Data Lakehouse architectures
  • Test pipelines underpinned by AI ETL accelerators, Python/PySpark transformations, SQL-based logic, and YAML configuration-driven orchestration
What we offer
What we offer
  • Flexible work options
  • Support for physical, emotional, and financial wellbeing
  • Continuous growth and development opportunities
Read More
Arrow Right

Data & AI Transformation Lead Senior Vice President

We are seeking an exceptional Senior Vice President to serve as the senior techn...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 15+ years of progressive experience in technology, with a significant portion in senior technical leadership roles — and a continued, active hands-on involvement in software development
  • Java Expertise: Expert-level background in Java, including designing and building resilient, high-throughput distributed systems and microservices architectures. Must be comfortable writing and reviewing production code
  • Big Data Mastery: Deep, hands-on expertise in modern Big Data platforms using technologies such as Apache Spark, Kafka, Presto/Trino, and distributed data lakes or lakehouses
  • Transformation Track Record: Demonstrated success leading complex, multi-year legacy modernization and re-engineering programs from concept to production in a global enterprise environment
  • Change Leadership: A proven change agent with the ability to shift engineering culture, introduce modern practices, and bring teams along on a transformation journey
  • Leadership: Exceptional organizational leadership skills with experience managing and developing multiple teams of senior engineers, architects, and technical leads
  • Prompt Engineering & AI Tooling: Deep, practical experience with prompt engineering and hands-on use of AI-assisted development tools (e.g., GitHub Copilot, Claude, OpenAI Codex, Devin) to drive engineering productivity and code quality at scale
  • Financial Domain: Experience in financial services, particularly in trading systems, post-trade processing, sub-ledger accounting, or risk platforms
  • Cloud Platforms: Experience with public or private cloud platforms (AWS, GCP, Azure) and their associated data and compute services
  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent professional experience
Job Responsibility
Job Responsibility
  • Be an active, hands-on contributor to the codebase — designing, building, and reviewing new Java microservices that replace legacy components
  • Lead the re-engineering of the existing Oasys codebase, breaking down monolithic structures and migrating business logic into a modern, event-driven microservices architecture
  • Establish and enforce application methodologies and standards for program analysis, design, coding, testing, and implementation across the engineering organization
  • Analyze complex business processes, system workflows, and industry standards to define solutions to high-level problems, particularly in the context of post-trade processing and financial accounting
  • Serve as the organization's champion and change agent for AI-assisted software development, driving the adoption of tools such as GitHub Copilot, Claude, OpenAI Codex, and Devin across the engineering team
  • Bring deep, practical expertise in prompt engineering and apply it to accelerate development workflows, improve code generation quality, and automate repetitive engineering tasks
  • Lead by example — demonstrating how AI tooling can be integrated into day-to-day development practices to meaningfully improve productivity and code quality
  • Lead the design and delivery of the new Oasys platform in parallel with the Sybase IQ exit, overseeing the build-out of high-throughput data pipelines, distributed data stores, and scalable data processing frameworks
  • Drive the integration of new systems and functions to meet strategic goals, deploy new capabilities, and continuously enhance processes
  • Utilize advanced knowledge of the Oasys system flows and a comprehensive understanding of multiple technology domains to achieve transformation milestones on time and at scale
  • Fulltime
Read More
Arrow Right

Lead Data Migration Engineer with .NET

You will lead the data migration workstream for a compliance-critical public-sec...
Location
Location
Poland , Wrocław; Bielsko-Biała
Salary
Salary:
145.00 - 170.00 PLN / Hour
cshark.com Logo
Cshark
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience with large-scale data migration projects: ETL, transformation pipelines, cutover planning
  • Hands-on experience with event sourcing (relational-to-event model transformation): decomposing relational data into domain events with deterministic ordering
  • Experience with legacy database analysis: reverse-engineering un(der)documented schemas, understanding data semantics and relationships without complete documentation
  • Experience with data quality handling: nulls, duplicates, inconsistent formats, missing fields, validation rules
  • Familiarity with document/file migration at scale: metadata extraction, filename parsing, large-volume validation and dealing with corrupted or ambiguous data or metadata
  • Strong knowledge of backend design patterns (dependency injection, repository, mediator, factory)
  • English language skills (B2+ minimum): required for clear communication with stakeholders and documentation of architectural decisions
  • Hands-on experience with CQRS, Domain-Driven Design, and Event Sourcing
  • Ability to design and implement aggregates, entities, value objects, and bounded contexts
  • Skilled in building systems that enforce clear separation of concerns, scalability, and maintainability
Job Responsibility
Job Responsibility
  • Lead the data migration workstream for a compliance-critical public-sector registry system
  • Converting a fully normalised relational database into an event-sourced architecture, migrating large volumes of document files, and ensuring every data related decision is documented and approved by the client
What we offer
What we offer
  • 100% remote work
  • Flexible hours
  • International projects
  • Business english lessons
  • Participation in charity actions
  • In-house technology workshops
  • Fulltime
Read More
Arrow Right

Lead Software Engineer, Digital Transformation

As our Lead Software Engineer, you will collaborate with and shepherd software e...
Location
Location
United States , Boston
Salary
Salary:
Not provided
hbs.edu Logo
HBS
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum of seven years’ post-secondary education or relevant work experience
  • Bachelor’s/Advanced Degree in Mathematics, Physics, Computer Science, Engineering, Statistics, or 7+ years equivalent work experience
  • 3-5 years of experience in developing a variety of software products in a commercial environment
  • 2+ years of experience in building generative AI-based products
  • Strong Python and Typescript/JavaScript skills required
  • Expertise in multivariate statistical modelling (e.g., clustering, regression, principal components and factor analysis, time-series forecasting, Bayesian methods) and modern machine learning (ensemble methods, regularization, gradient boosting, representation learning, deep neural networks, transformers architectures, self-supervised learning, and MLOps fundamentals) required
  • Experience with cloud computing platforms (AWS, GCP, or other), or equivalent on-premise platform
  • Experience working in agile methodology is required
  • Prior experience in leading data science and machine learning based software development projects focused on solving business problems and seizing business opportunities
Job Responsibility
Job Responsibility
  • Develop production-ready software solutions that solve real business problems, taking into account business needs and technology/operations landscape
  • lead interaction with internal stakeholders and technology on specific projects and initiatives
  • Apply software engineering, data science, machine learning, and AI techniques to derive business value from the full range of internal and external data, leveraging Cloud-based technology
  • Design solutions, build initial code, and provide documentation with ways of working to minimize time to value and maximize re-usability
  • Translate complex data and methodology into strategic, operationally feasible insights and recommendations
  • automate implementation
  • Communicate clearly and effectively to technical and non-technical audiences, verbally and visually, to create understanding, engagement, and buy-in
  • Identify trends and opportunities to drive innovation, both in what we do and how we do it
  • evaluate new software engineering, data science, machine learning, and AI technologies and tools that can boost team performance, innovation, and business value
  • Manage, coach, and mentor a team of software engineers and data scientists, serving as a predominant technical expert
What we offer
What we offer
  • Generous paid time off including parental leave
  • Medical, dental, and vision health insurance coverage starting on day one
  • Retirement plans with university contributions
  • Wellbeing and mental health resources
  • Support for families and caregivers
  • Professional development opportunities including tuition assistance and reimbursement
  • Commuter benefits, discounts and campus perks
  • Fulltime
Read More
Arrow Right

Forward Deployed Engineer - Data Migration & Data Consolidation Platforms

As a Forward Deployed Engineer (FDE) for Data Migration & Data Consolidation Pla...
Location
Location
United States
Salary
Salary:
Not provided
rackspace.com Logo
Rackspace
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7-10+ years of progressive experience in enterprise data engineering, data migration, or large-scale system integration roles within complex, multi-platform environments
  • 3-5+ years directly leading end-to-end data migration or multi-system consolidation programs for Global Enterprises and Industry Leaders, with full ownership of technical delivery and client outcomes
  • Demonstrated client-facing experience serving as a trusted technical advisor to C-level executives, enterprise architecture teams, and cross-functional business stakeholders
  • Proven industry depth in at least two of the following verticals: Healthcare, Financial Services, Manufacturing, Retail, Energy & Utilities, or Public Sector
  • Hands-on migration complexity: successfully delivered programs involving at least 3+ heterogeneous source systems, 100M+ records, complex master data harmonization, and multi-phase cutover execution
  • Advanced proficiency in Python and SQL with working experience in PySpark and TypeScript/JavaScript
  • Hands-on expertise with modern ETL/ELT and data integration platforms (Informatica, Talend, Matillion, Fivetran, AWS Glue, Azure Data Factory)
  • Proven ability to build scalable, version-controlled data pipelines with error handling, incremental loading, and Change Data Capture (CDC)
  • Strong working knowledge of at least one major cloud provider (AWS, Azure, or GCP), including core infrastructure, managed data services, and security configurations
  • Experience with enterprise data warehouse and lakehouse platforms (Snowflake, Databricks, BigQuery, Redshift, Synapse Analytics, Delta Lake)
Job Responsibility
Job Responsibility
  • Migration Execution & Cloud Architecture: Lead end-to-end delivery of enterprise data migrations from corporate systems (SAP, Oracle, Epic ERP) to target cloud data platforms, including the design of cloud landing zones, data governance frameworks, and system rationalization strategies. Establish migration compliance controls, automated rollback procedures, and operational readiness gates while owning full technical accountability for 12–18+ month migration roadmaps
  • Data Pipeline Engineering & Transformation: Build production-grade data connectors to SAP (RFC, IDoc, BAPI, OData), Oracle (AQ, GoldenGate, APIs), and SQL/non-relational sources. Develop ETL/ELT pipelines with LLM-enabled transformation logic, multi-layer validation and reconciliation frameworks, and optimized throughput for datasets scaling from tens of millions to billions of records with built-in CDC and incremental loading
  • Ontology Layer Development & Schema Automation: Construct semantic ontology layers translating raw ERP structures into business-consumable objects (Customer, Order, Invoice, Product, Vendor, Asset). Deploy automated schema mapping agents for source-to-target analysis and transformation logic generation. Build unified master data models with row/column-level security, cross-system lineage tracking, and AI-ready semantic structures
  • Application & Workflow Delivery: Build operational dashboards, migration control centers, and agent-driven workflows for automated validation, exception handling, and anomaly detection using low-code platform tools. Generate TypeScript/Python SDKs for custom integrations and deliver real-time monitoring and self-service interfaces for migration progress, data quality KPIs, and compliance tracking
  • Multi-System Consolidation & Master Data Management: Lead consolidation of 5–15+ fragmented ERP instances into standardized master data models. Resolve complex entity resolution challenges including customer matching, product harmonization, and chart of accounts unification. Establish golden record frameworks, data quality scorecards, survivorship rules, and data stewardship workflows for post-migration governance
  • Client Engagement, Discovery & Modernization Advisory: Serve as primary technical advisor to C-suite and enterprise architecture stakeholders across all engagement phases. Deploy discovery agents to analyze legacy data estates, conduct assessment workshops, facilitate solution design sessions, and deliver executive briefings, go/no-go readiness assessments, and prioritized modernization roadmaps
  • Knowledge Transfer, Enablement & IP Development: Build reusable migration accelerators, playbooks, and reference architectures that scale across engagements. Lead knowledge transfer to upskill client teams for post-migration ownership and collaborate with internal product and sales engineering teams to feed field insights back into platform development and delivery methodology
  • Leadership & Executive Engagement: Operate autonomously in ambiguous, high-stakes client environments, driving outcomes with minimal oversight
  • translate deeply technical concepts into clear, business-level narratives for C-suite audiences through executive briefings and stakeholder communications
  • navigate organizational complexity, competing stakeholder priorities, and enterprise change management dynamics to maintain momentum across multi-workstream engagements
Read More
Arrow Right

Senior Engineer (Data)

We’re looking for a Senior Engineer (Data Engineer) to design and build the data...
Location
Location
United Kingdom , Warminster
Salary
Salary:
Not provided
4cstrategies.com Logo
4C Strategies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A capable data engineer who has independently led substantial pieces of pipeline, modelling or platform work from design through to production
  • Solid SQL and a working transformation language — Python is most common
  • Comfortable with version control, code review and writing tests for the data you produce
  • Comfortable across the Microsoft / Azure data stack (SQL Server, Data Factory, Synapse or Fabric, Power BI)
  • Sound judgement around data modelling and warehouse / lakehouse patterns
  • Pragmatic about quality: you instrument what you build and care about how it runs
  • A thoughtful collaborator who works well with engineers, project managers, designers and senior customer stakeholders
  • A mentor by instinct who lifts the people around you and helps spread good practice
  • UK security clearance: if you already hold UK SC, that’s a plus
  • A genuine interest in our mission — software that helps armed forces, public safety and resilience organisations train, prepare, and respond
Job Responsibility
Job Responsibility
  • Design, build and operate batch and streaming data pipelines that move training and exercise data between Exonaut, MoD systems and partner platforms
  • Lead data modelling and warehouse / lakehouse design decisions
  • Work primarily across the Microsoft / Azure data stack
  • Introduce event‑driven and CDC patterns (Kafka, Event Hubs and similar) where they help us keep data flowing in near‑real‑time across systems
  • Help raise our engineering standards around code quality, testing, observability and operability
  • Mentor and coach engineers across the team
  • Partner with project managers and designers to make sure the data we build genuinely serves the people who use it
What we offer
What we offer
  • 25 days holidays (plus bank holidays)
  • Private healthcare including dental
  • Pension Scheme with 5% employer contribution
  • Enhanced maternity and paternity benefits
  • Life Assurance
  • Cycle to work scheme
  • Work Extras Discounts at over 80 retailers
  • Company social events
  • Fulltime
Read More
Arrow Right

Data Engineer

The Data Engineer is accountable for developing high quality data products to su...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • First Class Degree in Engineering/Technology (4-year graduate course)
  • 12 to 14 years’ experience implementing data-intensive solutions using agile methodologies
  • Experience of relational databases and using SQL for data querying, transformation and manipulation
  • Experience of modelling data for analytical consumers
  • Ability to automate and streamline the build, test and deployment of data pipelines
  • Experience in cloud native technologies and patterns
  • A passion for learning new technologies, and a desire for personal growth, through self-study, formal classes, or on-the-job training
  • Excellent communication and problem-solving skills
  • An inclination to mentor
  • an ability to lead and deliver medium sized components independently
Job Responsibility
Job Responsibility
  • Developing and supporting scalable, extensible, and highly available data solutions
  • Deliver on critical business priorities while ensuring alignment with the wider architectural vision
  • Identify and help address potential risks in the data supply chain
  • Follow and contribute to technical standards
  • Design and develop analytical data models
  • Fulltime
Read More
Arrow Right