CrawlJobs Logo
Capital One Logo Capital One · IT - Software Development

Lead Data Engineer - Data Transformation (Modeling and Architecture)

United States, Richmond, Virginia Employment contract 179400.00 - 225100.00 USD / Year · Job Posted July 03, 2026
Apply Position
Job Link Share

Job Description

Lead Data Engineer - Data Transformation (Modeling and Architecture). Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who solve real problems and meet real customer needs. We are seeking Data Engineers who are passionate about marrying data with emerging technologies. As a Capital One Lead Data Engineer, you’ll have the opportunity to be on the forefront of driving a major transformation within Capital One.

Job Responsibility

  • Build and maintain comprehensive data models—spanning conceptual, logical, and physical layers—to ensure scalable architecture and high data integrity across enterprise systems
  • Lead design of the org data landscape by applying Consumer Driven design principles, ensuring that data structures reflect business realities and evolving organizational needs
  • Architect and implement robust data ecosystem solutions, including Data Lake and Data Warehouse patterns, to support diverse analytical and operational requirements
  • Support high-performance data pipelines and complex transformations that utilize SQL, Spark, and Python to process large-scale datasets efficiently
  • Define and Enforce rigorous data governance standards while managing metadata frameworks to ensure data compliance and discoverability
  • Translate complex technical concepts into actionable business insights, working independently to lead initiatives and collaborate with stakeholders to meet organizational goals
  • Contribute to the evolution of the data ecosystem by designing AI-ready architectures
  • Collaborate with and across Agile teams to design, develop, implement, and support technical solutions
  • Work with a team of developers with deep experience in machine learning, AI, distributed microservices, and full stack systems
  • Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community
  • Collaborate with digital product managers, and deliver robust cloud-based solutions that drive powerful experiences to help millions of Americans achieve financial empowerment

Requirements

  • Bachelor's Degree
  • At least 4 years of experience in application development
  • At least 2 years of experience in big data technologies
  • At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

Nice to have

  • 4+ years of experience in Data Architecture / Data Modeling
  • 7+ years of experience in application development including Python, SQL, Scala, or Java
  • 4+ years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)
  • 4+ years experience with Distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL)
  • 4+ year experience working on real-time data and streaming applications
  • 4+ years of experience with NoSQL implementation (Mongo, Cassandra)
  • 4+ years of data warehousing experience (Redshift or Snowflake)
  • 4+ years of experience with UNIX/Linux including basic commands and shell scripting
  • 2+ years of experience with Agile engineering practices
  • Experience leveraging interactive AI tooling to accelerate productivity, utilizing capabilities beyond basic code completion

What we offer

  • Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
  • Comprehensive, competitive, and inclusive set of health, financial and other benefits
Capital One - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Lead Data Engineer - Data Transformation (Modeling and Architecture)

8 matching positions

Digital Transformation and Data Science Lead of Live Operations

Lead the digital and analytics evolution of our live operations. As the Digital ...
Location
Location
United States , Salisbury
Salary
Salary:
128000.00 - 192000.00 USD / Year
perduefarms.com Logo
Perdue Farms
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree in Data Science, Computer Science, Engineering, Operations Research, or a related field
  • 5-8 years of experience
  • Strong experience with Dataiku and/or similar data science platforms
  • Advanced proficiency in Power BI or comparable data visualization tools
  • Demonstrated ability to solve complex problems using advanced analytical techniques and independent judgment
  • Experience leading large projects or processes with limited oversight
  • Proven ability to influence cross‑functional stakeholders and translate business needs into technical solutions
  • Strong communication skills with the ability to convey complex concepts to non‑technical audiences
Job Responsibility
Job Responsibility
  • Lead the strategy, design, development, and deployment of advanced analytics, AI, and data science solutions supporting live operations
  • Architect and optimize end‑to‑end data workflows and models using Dataiku or comparable platforms
  • Develop scalable dashboards, reporting frameworks, and visualization strategies in Power BI to support operational and executive decision‑making
  • Develop and support AI and machine learning models for forecasting and operational decision‑making
  • Identify opportunities to enhance performance through data insights, automation, and predictive analytics
  • Establish best practices, standards, and methodologies for analytics development and data governance
  • Lead and influence digital transformation initiatives across live operations, identifying and prioritizing high‑impact opportunities
  • Translate business needs into scalable digital solutions and drive alignment on approach, design, and implementation
  • Drive process improvement through automation, standardization, and advanced data utilization
  • Evaluate emerging technologies and recommend adoption strategies aligned to business objectives
What we offer
What we offer
  • medical/Rx
  • 401(k) with employer match after 1 year
  • critical illness
  • accident insurance
  • dental
  • vision
  • life insurance
  • optional group life insurance
  • short-term and long-term disability protection
  • flexible spending accounts
  • Fulltime
Read More
Arrow Right
New

Principal Engineer - Technical Lead (Gen AI and MACH Architecture)

At AKQA, we believe in the imaginative application of art and science to create ...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
akqa.com Logo
AKQA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum three years' experience with Generative AI, including proven experience shipping at least one AI solution to production in an enterprise context (beyond prototypes or sandbox experiments)
  • AI architecture for experience platforms: system design for intelligent products and services, including RAG, GraphRAG, agent orchestration and proximity / relevance evaluation
  • Hands-on implementation with Vercel AI SDK, Vercel AI Cloud and agentic cloud patterns for production AI experiences
  • AI content and data engineering: ingestion, cleansing, transformation, embedding, indexing and retrieval pipeline design
  • Integration across cloud AI services (e.g. GCP + Vertex AI, Vercel AI Cloud, AWS Bedrock, Azure OpenAI) within a broader MACH and API-led landscape
  • with preference for GCP and Vercel, or suitable equivalents on AWS and Azure, and the capability to draw parallels across platforms
  • GEO and LLM optimisation: experience designing intelligent experiences and content structures optimised for generative engine discovery and LLM retrieval
  • Model fine-tuning: experience customising and fine-tuning existing models for production use cases
  • including data preparation, evaluation, versioning and deployment within enterprise governance, cost and reliability constraints
  • AI cost-value analysis, observability, governance, testing and evaluation frameworks for production systems
Job Responsibility
Job Responsibility
  • Lead the technical direction of complex client programmes across MACH architectures and production Generative AI
  • spanning Design to Code workflows, solution design and API contracts through to cloud infrastructure, deployment and ongoing operation
  • Own architecture decisions, write and review production code, and mentor engineers, while remaining accountable for what ships, not just what is proposed
  • Designing, building and shipping enterprise-grade systems, with roughly 65 to 70% hands-on delivery and 30 to 35% technical leadership, architecture and team guidance
  • Work closely with Technical Managers, cross-discipline teams and on-shore, off-shore and hybrid engineering squads
  • Occasionally support pre-sales scoping or technical pitches
Read More
Arrow Right
New

Lead Data Engineer – AI & Foundation Models

Our Purpose Mastercard powers economies and empowers people in 200+ countries a...
Location
Location
Ireland , Dublin 18
Salary
Salary:
Not provided
mastercard.com Logo
Mastercard
Expiration Date
October 10, 2026
Flip Icon
Requirements
Requirements
  • Strong experience designing and building production‑grade data pipelines in large‑scale environments
  • Deep expertise with distributed data processing frameworks (e.g. Spark or equivalent) and SQL‑based analytics
  • Experience working with cloud data platforms and storage technologies (AWS, Azure, or GCP)
  • Solid understanding of data modeling, performance tuning, and cost‑efficient data architecture
  • Experience supporting machine learning and AI workloads, including training datasets, feature engineering, and inference data flows
  • Familiarity with data governance concepts, including lineage, data quality, access control, and auditability
  • Strong software engineering fundamentals, including version control, testing, CI/CD, and code quality standards
  • Ability to translate AI and product requirements into practical, scalable data solutions
  • Experience leading technical delivery and mentoring engineers, without formal line‑management responsibility
  • Clear, concise communicator able to collaborate effectively with engineers, data scientists, product managers, and stakeholders
Job Responsibility
Job Responsibility
  • Lead the design and implementation of scalable data pipelines supporting AI model training, inference, and experimentation
  • Own data ingestion, transformation, and aggregation patterns across batch and streaming workloads
  • Partner with AI engineers to enable feature engineering, feature stores, and training datasets aligned to model requirements
  • Ensure data pipelines meet enterprise standards for quality, availability, lineage, and governance
  • Drive best practices for data modeling, schema management, partitioning, and performance optimization
  • Implement robust data quality checks, validation, and monitoring to ensure trust in downstream AI systems
  • Collaborate with platform and infrastructure teams to build pipelines on cloud‑native and distributed data processing platforms
  • Support secure data access patterns, including environment isolation, access controls, and auditability
  • Lead code reviews and design reviews for data engineering deliverables across the program
  • Mentor and guide senior and mid‑level data engineers, providing technical direction and delivery oversight
  • Fulltime
Read More
Arrow Right

Data Engineer – Lead

Data Engineer – Lead
Location
Location
India , Bengaluru Urban
Salary
Salary:
Not provided
nttdata.com Logo
NTT DATA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, OneLake, Pipelines, Dataflows Gen2, Notebooks, and Power BI integration
  • Expertise in ETL/ELT, data pipelines, distributed data processing, and cloud-scale data engineering
  • Strong SQL, Python, PySpark, and data modeling skills
  • Experience with Lakehouse, Warehouse, and Medallion Architecture
  • Understanding of Delta tables, dimensional modeling, star schema, facts, dimensions, and curated analytical datasets
  • Experience integrating structured, semi-structured, file-based, API-based, enterprise application, and cloud data sources
  • Experience with data quality, reconciliation, logging, monitoring, and error-handling frameworks
  • Experience leading technical teams and coordinating onshore/offshore delivery
  • Experience with Git, CI/CD, Azure DevOps, branching, code reviews, and release management
  • Good to Have: Experience with Azure Data Factory, Synapse, Databricks, ADLS Gen2, Azure SQL, Microsoft Purview, or related Azure services
Job Responsibility
Job Responsibility
  • Lead the design and implementation of scalable data pipelines and data processing frameworks in Microsoft Fabric
  • Define data engineering standards, development practices, naming conventions, coding guidelines, and reusable technical patterns
  • Lead implementation of Bronze, Silver, and Gold layers in the Medallion Architecture
  • Oversee ingestion, transformation, orchestration, validation, and publication of data from multiple enterprise, clinical, operational, and cloud-based sources
  • Guide development of Fabric Pipelines, Dataflows Gen2, Notebooks, Lakehouse tables, Warehouse objects, and curated datasets
  • Ensure scalability, performance, reliability, maintainability, security, monitoring, and optimization of data solutions
  • Define standards for data quality, reconciliation, logging, error handling, auditability, and lineage
  • Conduct technical design reviews, code reviews, performance reviews, and deployment readiness reviews
  • Mentor and guide data engineering teams across onshore/offshore locations
  • Collaborate with architects, platform engineers, BI teams, QA teams, AI/ML teams, functional consultants, and stakeholders
Read More
Arrow Right

Senior Data Engineer Lead / Architect - Senior Vice President

At Citi Services - Global Trade Technology Organization, we are on a mission to ...
Location
Location
India , Pune, Maharashtra, India, Chennai, Tamil Nadu, India
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of professional experience in data engineering, with a proven track record of designing and building large-scale data systems
  • 3+ years in a technical leadership or architect role, with experience mentoring junior and senior engineers
  • Expert-level proficiency in at least one programming language (Python or Scala preferred) and exceptional SQL skills
  • Proven hands-on experience with Python or Scala for data manipulation, scripting, machine learning, and backend development
  • Deep, hands-on experience with a major cloud platform (AWS, GCP, or Azure) and its data ecosystem (e.g., S3/GCS, Redshift/BigQuery, EMR/Dataproc, Kinesis/Dataflow)
  • Extensive hands-on experience with modern big data technologies and Data streaming (like Hadoop, Hive, Impala, Apache Spark, Kafka, or Flink)
  • Proficiency with workflow orchestration tools such as Airflow, Dagster, or Prefect
  • Proficiency in designing and implementing microservices architectures, RESTful APIs, and event-driven systems with 'Data as a Product' Principle
  • Solid understanding of data modeling concepts and database design for both analytical (OLAP) and transactional (OLTP) workloads
  • Deep understanding and hands-on experience with relational databases (e.g., PostgreSQL, Oracle), NoSQL databases (e.g., MongoDB, Cassandra), data warehousing, and big data technologies (e.g., Spark, Kafka)
Job Responsibility
Job Responsibility
  • Architect & Design: Design, architect, and oversee the development of robust, scalable, and reliable data infrastructure, including data lakes, data warehouses, and real-time streaming platforms on the cloud
  • Build & Code: Act as a senior individual contributor and hands-on technical leader. Write clean, maintainable, and high-performance code for data ingestion, transformation, and serving layers (e.g., using Python, Scala, SQL, and Spark)
  • Lead & Mentor: Lead a team of data engineers, providing technical guidance, mentorship, and career development support. Foster a collaborative and inclusive team environment
  • Champion Culture: Define, document, and champion data engineering best practices across the organization, including CI/CD, data quality, testing frameworks, observability, and code review standards
  • Drive Strategy: Partner with leadership, product managers, data scientists, and analysts to understand data needs and develop a long-term data strategy and roadmap
  • Innovate & Evaluate: Stay at the forefront of data engineering technologies. Evaluate, prototype, and recommend new tools and frameworks to continuously improve our data platform
  • Ensure Governance: Implement and enforce robust data governance, security, and privacy policies in partnership with our security and compliance teams
  • Fulltime
Read More
Arrow Right

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...
Location
Location
North Macedonia
Salary
Salary:
Not provided
valtech.com Logo
Valtech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong proficiency in Python for building frameworks, automation tooling and reusable components
  • Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
  • Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
  • Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
  • Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
  • Experience with data engineering platform design, including scalable pipeline and workflow architectures
  • Strong understanding of software engineering principles (DRY, SOLID, modular design)
  • Experience working with version control systems and modern Git workflows
  • Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
  • Experience implementing automated testing strategies (unit, integration, data quality)
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
  • Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
  • Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
  • Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
  • Define and enforce engineering standards for testing, code quality, deployment and documentation
  • Identify and eliminate manual or repetitive processes through automation and tooling improvements
  • Integrate AI-assisted development tools into engineering workflows to improve productivity
  • Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
  • Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
  • Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents
What we offer
What we offer
  • Private health insurance
  • Education program
  • Wellbeing program
  • Free beverages
  • Events
  • Competitive conditions
  • Challenging projects
  • Cool colleagues
  • Honest feedback
  • Fulltime
Read More
Arrow Right

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...
Location
Location
Poland
Salary
Salary:
Not provided
valtech.com Logo
Valtech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong proficiency in Python for building frameworks, automation tooling and reusable components
  • Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
  • Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
  • Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
  • Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
  • Experience with data engineering platform design, including scalable pipeline and workflow architectures
  • Strong understanding of software engineering principles (DRY, SOLID, modular design)
  • Experience working with version control systems and modern Git workflows
  • Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
  • Experience implementing automated testing strategies (unit, integration, data quality)
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
  • Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
  • Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
  • Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
  • Define and enforce engineering standards for testing, code quality, deployment and documentation
  • Identify and eliminate manual or repetitive processes through automation and tooling improvements
  • Integrate AI-assisted development tools into engineering workflows to improve productivity
  • Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
  • Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
  • Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents
What we offer
What we offer
  • 24 working days of paid vacation
  • National holidays covered
  • Sick leave (up to 20/year)
  • Unpaid leave (up to 20/year)
  • Medical insurance
  • Multisport card OR Multikafeteria
  • Maternity & paternity leave support
  • Internal workshops & learning initiatives
  • Professional certifications reimbursement
  • Participation in professional local & global communities
Read More
Arrow Right

Lead Data Engineer (Platform)

As a Lead Data Engineer (Platform), you are passionate about engineering excelle...
Location
Location
Bulgaria , Sofia
Salary
Salary:
Not provided
valtech.com Logo
Valtech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong proficiency in Python for building frameworks, automation tooling and reusable components
  • Hands-on experience with Databricks (including notebooks, workflows, jobs and Unity Catalog)
  • Strong SQL skills and experience with distributed processing frameworks such as Apache Spark
  • Deep experience with dbt Core, including project structure, models, tests, macros and deployment at scale
  • Proven experience designing and maintaining CI/CD pipelines (e.g. GitHub Actions, Azure DevOps or GitLab CI)
  • Experience with data engineering platform design, including scalable pipeline and workflow architectures
  • Strong understanding of software engineering principles (DRY, SOLID, modular design)
  • Experience working with version control systems and modern Git workflows
  • Experience with cloud platforms (preferably AWS) and infrastructure-as-code concepts (e.g. Terraform)
  • Experience implementing automated testing strategies (unit, integration, data quality)
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams
  • Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery
  • Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency
  • Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity
  • Define and enforce engineering standards for testing, code quality, deployment and documentation
  • Identify and eliminate manual or repetitive processes through automation and tooling improvements
  • Integrate AI-assisted development tools into engineering workflows to improve productivity
  • Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations
  • Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale)
  • Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents
What we offer
What we offer
  • Flexibility, with hybrid work options and 25 vacation days for a healthy work-life balance
  • Co-subsidized transportation & Multisport cards
  • Premium health insurance for fast and easy access to top healthcare services
  • Training policy for technical and other skills-related events, courses and certifications
  • Personal career development roadmap guided by performance evaluations
  • Self-care program offering psychological consultations & discussions for you and the team
  • Cozy office space designed for comfort and productivity
  • Exciting team events and company gatherings
  • Fulltime
Read More
Arrow Right