CrawlJobs Logo

Senior Engineering Lead, Big Data Engineering & Regulatory Platforms

United States, Tampa Employment contract 141440.00 - 212160.00 USD / Year · Job Posted June 04, 2026
Apply Position
Job Link Share

Job Responsibility

  • Lead and guide a core technology team responsible for developing and maintaining regulatory reporting frameworks, with a focus on leveraging big data solutions like Hadoop, Hive, Spark, and Data Bricks.
  • Lead architecture governance process including architecture review boards, technology assessments, and conformance to key design principles
  • Build and maintain big data pipelines using technologies such as Apache Hadoop, Apache Kafka, Databricks and other cloud based big data tools.
  • Establish best practices for big data architecture including data governance, security and meta data requirement.
  • Evaluates and tests potential architecture solution designs to ensure they meet functional and non-functional requirements
  • Design, develop, and implement an end-to-end robust, scalable framework that addresses critical business needs, with a particular focus on regulatory compliance and reporting, performance ,security , reliability and operational efficiency , incorporating AI-driven solution.
  • Evaluate different technical options, conduct comparative analyses, and present clear recommendations to senior stakeholders.
  • Apply in-depth domain knowledge of financial services to comprehend complex business problems and develop appropriate solution options.
  • Drive opportunities for efficiency gains, enhanced controls, and extendable scalability through innovative architectural solutions.
  • Articulate and demonstrate architectural vision through various architectural artifacts.
  • Collaborate with high-performing teams to design enterprise-wide solutions across business, technical, and data architecture domains.
  • Engage with key stakeholders, including operations teams, product teams, and application development teams across different Lines of Business (LoBs) and functions, ensuring global engagement and alignment.
  • Provide technical leadership , mentoring junior engineers and fostering culture of innovation.

Requirements

  • 10+ years of proven experience in a senior-level architectural and engineering role
  • 5+ years experience in leading large scale enterprise architecture initiatives within banking, financial services industry
  • Experience with cloud platform (eg. AWS ,Azure ,GCP) in combination with big data technology such as Hadoop, Hive, Spark, and Data Bricks.
  • Knowledge of machine learning and advance analytical within big data environments.
  • Strong decision making, forward thinking and creative problem-solving skills to anticipate and respond quickly to technological/market influences
  • Comprehensive knowledge of design metrics, analytics tools, benchmarking activities and related reporting to identify best practices
  • In-depth domain knowledge of financial services.
  • Demonstrable commitment to architectural simplification, building sustainable, and future-ready platforms.
  • Results-oriented, innovative, and motivated individual.
  • Excellent communication and presentation skills for engaging with senior stakeholders.
  • Ability to work effectively in a global, multi-stakeholder environment.
  • Bachelor's degree/University degree or equivalent experience
  • Master's degree preferred

Nice to have

Master's degree preferred

What we offer

  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Senior Engineering Lead, Big Data Engineering & Regulatory Platforms

8 matching positions

Senior Data Engineer

We are looking for a Data Engineer to join our team and support with designing, ...
Location
Location
Salary
Salary:
Not provided
foundever.com Logo
Foundever
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum of 7 years plus experience in data engineering
  • Track record of deploying and maintaining complex data systems at an enterprise level within regulated environments
  • Expertise in implementing robust data security measures, access controls, and monitoring systems
  • Proficiency in data modeling and database management
  • Strong programming skills in Python and SQL
  • Knowledge of big data technologies like Hadoop, Spark, and NoSQL databases
  • Deep experience with ETL processes and data pipeline development
  • Strong understanding of data warehousing concepts and best practices
  • Experience with cloud platforms such as AWS and Azure
  • Excellent problem-solving skills and attention to detail
Job Responsibility
Job Responsibility
  • Design and optimize complex data storage solutions, including data warehouses and data lakes
  • Develop, automate, and maintain data pipelines for efficient and scalable ETL processes
  • Ensure data quality and integrity through data validation, cleansing, and error handling
  • Collaborate with data analysts, machine learning engineers, and software engineers to deliver relevant datasets or data APIs for downstream applications
  • Implement data security measures and access controls to protect sensitive information
  • Monitor data infrastructure for performance and reliability, addressing issues promptly
  • Stay abreast of industry trends and emerging technologies in data engineering
  • Document data pipelines, processes, and best practices for knowledge sharing
  • Lead data governance and compliance efforts to meet regulatory requirements
  • Collaborate with cross-functional teams to drive data-driven decision-making within the organization
What we offer
What we offer
  • Impactful work
  • Professional growth
  • Competitive compensation
  • Collaborative environment
  • Attractive salary and benefits package
  • Continuous learning and development opportunities
  • A supportive team culture with opportunities for occasional travel for training and industry events
Read More
Arrow Right

Testing Senior Group Manager

Citibank, N.A. seeks a Testing Senior Group Manager for its Rutherford, New Jers...
Location
Location
United States , Rutherford
Salary
Salary:
260000.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
June 16, 2026
Flip Icon
Requirements
Requirements
  • Requires a Bachelor’s degree or foreign equivalent in Engineering (any), Computer Science, Information Technology or related field and 8 years of progressive post-baccalaureate experience as a Team Lead, Manager, Software Engineer, Analyst, Programmer Analyst, QA Engineer, AVP, Associate or related position involving test automation strategy and developing test automation framework. 7 years of experience must include: Evaluating data quality and accuracy
  • Aggregation and reconciliation of data sets across multiple ETL pipelines using SQL and Excel
  • Python, Java and SQL for automation testing and building enterprise data quality engineering platforms
  • Establish controls and data standards on enterprise data platform
  • Hive, Impala, Apache Spark, Oracle, Relational Database system
  • Determining, implementing and executing testing strategy of ETL software applications
  • Impact analysis and defining functional validations for feature development and enhancements on Enterprise Data Platforms
  • and Hadoop, KDB, financial Institution data platforms and data sets.
Job Responsibility
Job Responsibility
  • Determine the approach to implement testing strategies and execute testing strategies
  • Structure the testing group to ensure the greatest impact, efficient use of resources, and alignment to business structures
  • Analyze business requirements from stakeholders and prepare functional test cases including the impact analysis to the current state application
  • Use Python, Java and SQL for automation testing and building enterprise data quality engineering platforms
  • Establish controls and data standards on enterprise data platform
  • Design and document technical solutions using Big Data stack of technologies, including aggregation and reconciliation of data sets across multiple ETL pipelines using SQL and Excel
  • Conduct architectural design reviews for automation testing, and lead data quality engineering teams to build enterprise data quality engine for regulatory, liquidity, and risk reporting testing
  • Evaluate data quality and accuracy and address complex issues using Hive, Impala, Apache Spark, Oracle, Relational Database system, Hadoop, KDB, and financial institution data platforms and data sets
  • Drive innovative testing strategies that support technological changes
  • Influence IT testing through planning, finances and budget, end results, and setting policies
What we offer
What we offer
  • medical, dental & vision coverage
  • 401(k)
  • life, accident, and disability insurance
  • wellness programs
  • paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
  • Fulltime
Read More
Arrow Right

Senior Manager, Data Engineering

Stanford Health Care is seeking a dynamic and experienced Senior Manager of Data...
Location
Location
United States , Palo Alto
Salary
Salary:
83.98 - 111.27 USD / Hour
stanfordhealthcare.org Logo
Stanford Health Care
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field. A Master’s degree is preferred
  • 8+ years of hands-on experience in data engineering, data warehousing, or a related field
  • 3+ years of experience in a leadership or management role, directly managing multiple technical teams
  • Experience in a healthcare or academic medical center environment is highly desirable
  • Proven ability to lead, mentor, and grow multiple high-performing engineering teams
  • Expert knowledge of data engineering principles, including ETL/ELT, data modeling, and data architecture
  • Strong strategic planning skills with experience in developing roadmaps for both data engineering and enterprise analytics
  • Deep understanding of cloud data platforms (GCP, AWS, Azure) and modern data warehouse technologies (Databricks, Snowflake, BigQuery, Redshift)
  • Proficiency with big data technologies (e.g., Spark, Kafka) and workflow orchestration tools (e.g., Airflow)
  • Expert-level SQL skills and proficiency in a programming language such as Python or Scala
Job Responsibility
Job Responsibility
  • Lead, mentor, and manage multiple teams of data engineers, providing technical guidance, career development, and performance management
  • Foster a culture of collaboration, innovation, and operational excellence
  • Manage resource allocation, project prioritization, and team workloads to ensure timely delivery of key initiatives
  • Recruit, coach, and motivate team members, developing leadership competencies within the team
  • Oversee the design, development, and maintenance of scalable and robust data pipelines (ETL/ELT) to support analytics and data science
  • Define and drive the technical roadmap for our cloud-based data platforms (e.g., GCP, AWS, Azure) and data warehousing solutions (e.g., Databricks)
  • Define and implement the strategy for enterprise analytics and operational reporting, in collaboration with key business and clinical stakeholders
  • Partner with enterprise architecture, applications, and data science teams to ensure sustainable and scalable data management and solution delivery
  • Partner with key stakeholders across Stanford Medicine (Stanford Health Care, Stanford School of Medicine, and Stanford Medicine Partners), including data scientists, analysts, clinical leaders, and researchers, to translate data needs into technical requirements
  • Manage the full lifecycle of data engineering projects, from conception and planning to execution and delivery
  • Fulltime
Read More
Arrow Right

Technology Lead Business Analyst

Location
Location
Canada , Mississauga
Salary
Salary:
120800.00 - 170800.00 USD / Year
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience as a Business Analyst, with at least 3 years focused on data-driven projects in the Capital Markets domain (e.g., trading systems, portfolio management, risk management, derivatives, equities)
  • Strong experience in implementing data control frameworks, data governance policies, and managing data accuracy within large data environments such as data lakes and big data projects
  • Proven experience managing data projects, working with cross-functional teams to ensure timely delivery of key milestones and meeting business requirements
  • Excellent ability to communicate complex technical concepts to business stakeholders and translate business needs into actionable technical requirements
  • Proven ability to define and enforce data quality standards, including methods for identifying and correcting data issues to ensure accuracy and completeness
  • Familiarity with data management tools, big data platforms (Hadoop, Spark, Hive), and reporting tools (e.g., Tableau, Power BI)
  • Knowledge of data integration techniques, ETL processes, and data visualization
  • Strong ability to manage relationships with key stakeholders, ensuring that business needs are met while aligning with technical capabilities and timelines
  • Strong analytical skills with the ability to review and solve complex data-related issues, providing innovative solutions to data challenges
Job Responsibility
Job Responsibility
  • Lead and manage data-related projects, particularly within the Capital Markets domain, ensuring alignment with business objectives and timely delivery
  • Oversee the successful implementation of data control processes and data governance practices for big data lake projects
  • Implement and enforce data control frameworks to ensure consistency, integrity, and accuracy of data across all stages of data lifecycle management
  • Ensure compliance with regulatory requirements related to data accuracy and reporting within the capital markets domain
  • Serve as the primary point of contact between business stakeholders and technical teams
  • Gather and define business requirements, translate them into technical specifications, and ensure that the final solutions meet the business needs
  • Develop and implement strategies to maintain high data quality standards, including data accuracy, completeness, and consistency, especially in large-scale data lake environments
  • Define, manage, and track data project timelines
  • Work closely with project managers to ensure milestones are met and project deliverables are completed on time, within scope, and within budget
  • Engage with internal and external stakeholders, including business units, data scientists, data engineers, and senior management, to ensure alignment on business needs and data requirements
  • Fulltime
Read More
Arrow Right

Senior Developer (Spark/Scala)

We are looking for a Senior Developer team with a strong background in Spark/Sca...
Location
Location
Poland , Warsaw
Salary
Salary:
Not provided
talan.com Logo
Talan
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in Big Data systems and data management
  • 3+ years of hands-on experience with Scala/Spark
  • Hands-on with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Hive
  • Excellent understanding of dimensional data and data modeling techniques
  • Experience implementing and deploying solutions in cloud environments
  • Ability to work independently and take ownership of project deliverables
  • Fluent in English (at least C1 level)
Job Responsibility
Job Responsibility
  • Lead end-to-end technical delivery of data engineering projects
  • Architect, design, and implement scalable data pipelines and platforms
  • Optimize workflows for data ingestion, processing, and storage (Spark, Airflow, Cloudera, DataBricks, Hive)
  • Develop and maintain data lake solutions for structured and unstructured data
  • Collaborate with global teams (business, data, and technology) to align solutions with organizational objectives
  • Oversee code quality, CI/CD pipelines, versioning, and release management
  • Write and review high-quality Scala/Spark code and contribute to automation
  • Mentor and support engineers, fostering technical excellence and growth
  • Ensure delivery is aligned with regulatory and compliance requirements
What we offer
What we offer
  • Smart Office Pack so that you can work comfortably from home
  • Training and career development
  • Private medical insurance
  • Life insurance
  • Possibility to be part of a multicultural team and work on international projects
  • Possibility to manage work-permits
  • Fulltime
Read More
Arrow Right

Senior Developer (Spark/Scala)

We are looking for a Senior Developer team with a strong background in Spark/Sca...
Location
Location
Poland , Warsaw
Salary
Salary:
Not provided
talan.com Logo
Talan
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in Big Data systems and data management
  • 3+ years of hands-on experience with Scala/Spark
  • Hands-on with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Hive
  • Excellent understanding of dimensional data and data modeling techniques
  • Experience implementing and deploying solutions in cloud environments
  • Ability to work independently and take ownership of project deliverables
  • Fluent in English (at least C1 level)
Job Responsibility
Job Responsibility
  • Lead end-to-end technical delivery of data engineering projects
  • Architect, design, and implement scalable data pipelines and platforms
  • Optimize workflows for data ingestion, processing, and storage (Spark, Airflow, Cloudera, DataBricks, Hive)
  • Develop and maintain data lake solutions for structured and unstructured data
  • Collaborate with global teams (business, data, and technology) to align solutions with organizational objectives
  • Oversee code quality, CI/CD pipelines, versioning, and release management
  • Write and review high-quality Scala/Spark code and contribute to automation
  • Mentor and support engineers, fostering technical excellence and growth
  • Ensure delivery is aligned with regulatory and compliance requirements
  • Hands-on experience with testing Big Data solutions - unit tests, integration tests, E2E tests
What we offer
What we offer
  • Smart Office Pack so that you can work comfortably from home
  • Training and career development
  • Private medical insurance
  • Life insurance
  • Possibility to be part of a multicultural team and work on international projects
  • Possibility to manage work-permits
  • Fulltime
Read More
Arrow Right

Senior Developer (Spark/Scala)

We are looking for a Senior Developer team with a strong background in Spark/Sca...
Location
Location
Poland , Warsaw
Salary
Salary:
Not provided
talan.com Logo
Talan
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in Big Data systems and data management
  • 3+ years of hands-on experience with Scala/Spark
  • Hands-on with Big Data technologies like Spark, Cloudera, Data Platform, Airflow, Hive
  • Excellent understanding of dimensional data and data modeling techniques
  • Experience implementing and deploying solutions in cloud environments
  • Ability to work independently and take ownership of project deliverables
  • Fluent in English (at least C1 level)
Job Responsibility
Job Responsibility
  • Lead end-to-end technical delivery of data engineering projects
  • Architect, design, and implement scalable data pipelines and platforms
  • Optimize workflows for data ingestion, processing, and storage (Spark, Airflow, Cloudera, DataBricks, Hive)
  • Develop and maintain data lake solutions for structured and unstructured data
  • Collaborate with global teams (business, data, and technology) to align solutions with organizational objectives
  • Oversee code quality, CI/CD pipelines, versioning, and release management
  • Write and review high-quality Scala/Spark code and contribute to automation
  • Mentor and support engineers, fostering technical excellence and growth
  • Ensure delivery is aligned with regulatory and compliance requirements
  • Hands-on experience with testing Big Data solutions - unit tests, integration tests, E2E tests
What we offer
What we offer
  • Smart Office Pack so that you can work comfortably from home
  • Training and career development
  • Private medical insurance
  • Life insurance
  • Possibility to be part of a multicultural team and work on international projects
  • Possibility to manage work-permits
  • Fulltime
Read More
Arrow Right

Senior Data Engineering Lead-Senior Vice President

The Applications Development Group Manager is a senior management level position...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 13+ years of relevant experience
  • Extensive experience in applications development
  • Experience in Wealth Management / Private Bank data space
  • Familiarity with key investment flows like Post Trade Flows, Portfolio and Risk Modelling, Performance, Market and Reference data including Client and Advisor and IC interactions
  • Sound knowledge of core investment products like Cash Equity, Fixed Income, Alternatives, Structured Notes and Derivatives
  • Experience with data technologies such as RDBMS and Hadoop (Cloudera) database
  • Experience in building cloud native data platforms (AWS & GCP) and exposure to modern data architecture
  • Experience in building real-time and batch data pipelines to support wide range of Analytics, Regulatory, Business and Operational use cases
  • Familiar with the big data technologies such as Hadoop, HDFS, Hive, HBase, Spark and Flink
  • Exposure with Snowflake and Databricks is preferred
Job Responsibility
Job Responsibility
  • Manage multiple teams of professionals to accomplish established goals and conduct personnel duties for team (e.g. performance evaluations, hiring and disciplinary actions)
  • Provide strategic influence and exercise control over resources, budget management and planning while monitoring end results
  • Utilize in-depth knowledge of concepts and procedures within own area and basic knowledge of other areas to resolve issues
  • Ensure essential procedures are followed and contribute to defining standards
  • Integrate in-depth knowledge of applications development with overall technology function to achieve established goals
  • Provide evaluative judgement based on analysis of facts in complicated, unique, and dynamic situations including drawing from internal and external sources
  • Influence and negotiate with senior leaders across functions, as well as communicate with external parties as necessary
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency, as well as effectively supervise the activity of others and create accountability with those who fail to maintain these standards
  • Fulltime
Read More
Arrow Right