CrawlJobs Logo

Data Lake SME

India, Bangalore · Job Posted March 04, 2026
Apply Position
Job Link Share

Job Description

We are looking for an experienced Data Lake / ETL Engineer with 7+ years of expertise in designing, developing, and managing large-scale data ingestion, transformation, and analytics pipelines. The role involves building scalable and secure data lake platforms, enabling business insights through efficient ETL/ELT frameworks, and ensuring data quality, performance, and governance across the enterprise ecosystem.

Job Responsibility

  • Design and implement data ingestion pipelines for structured, semi-structured, and unstructured data
  • Develop and manage ETL/ELT processes for large-scale data processing
  • Optimize storage and retrieval strategies across on-prem and cloud-based data lakes
  • Integrate data from multiple sources (databases, APIs, streaming platforms)
  • Implement real-time and batch processing using Apache Spark, Kafka, or Flink
  • Support metadata management, data lineage, and cataloging
  • Tune queries and pipelines for high performance and cost efficiency
  • Implement partitioning, indexing, and caching strategies for large datasets
  • Automate routine ETL/ELT workflows for reliability and speed
  • Ensure compliance with data governance, privacy, and regulatory standards (GDPR, HIPAA, etc.)
  • Implement encryption, masking, and role-based access control (RBAC)
  • Collaborate with cybersecurity teams to align with Zero Trust and IAM policies
  • Partner with data scientists, analysts, and application teams for analytics enablement
  • Provide L2/L3 support for production pipelines and troubleshoot failures
  • Mentor junior engineers and contribute to best practices documentation

Requirements

  • 7+ years of experience in data engineering, ETL/ELT development, or data lake management
  • Strong expertise in ETL tools (Informatica, Talend, dbt, SSIS, or similar)
  • Hands-on experience with big data ecosystems: Hadoop, Spark, Hive, Presto, Delta Lake, or Iceberg
  • Proficiency with SQL, Python, or Scala for data processing and transformation
  • Experience with cloud data platforms (AWS Glue, Redshift, Azure Synapse, GCP BigQuery)
  • Familiarity with workflow orchestration tools (Airflow, Temporal, Oozie)
  • Bachelor’s or Master’s degree in Computer Science, IT, or related field

Nice to have

  • Exposure to real-time data streaming (Kafka, Kinesis, Pulsar)
  • Knowledge of data modeling (Kimball/Inmon), star schema, and dimensional modeling
  • Experience with containerized deployments (Docker, Kubernetes)
  • AWS Certified Data Analytics – Specialty / Azure Data Engineer Associate / GCP Data Engineer
  • Informatica/Talend/dbt certifications

What we offer

  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Lake SME

8 matching positions

Data Lake SME

We are looking for an experienced Data Lake / ETL Engineer with 7+ years of expe...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.hpe.com/ Logo
Hewlett Packard Enterprise
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in data engineering, ETL/ELT development, or data lake management
  • Strong expertise in ETL tools (Informatica, Talend, dbt, SSIS, or similar)
  • Hands-on experience with big data ecosystems: Hadoop, Spark, Hive, Presto, Delta Lake, or Iceberg
  • Proficiency with SQL, Python, or Scala for data processing and transformation
  • Experience with cloud data platforms (AWS Glue, Redshift, Azure Synapse, GCP BigQuery)
  • Familiarity with workflow orchestration tools (Airflow, Temporal, Oozie)
Job Responsibility
Job Responsibility
  • Design and implement data ingestion pipelines for structured, semi-structured, and unstructured data
  • Develop and manage ETL/ELT processes for large-scale data processing
  • Optimize storage and retrieval strategies across on-prem and cloud-based data lakes
  • Integrate data from multiple sources (databases, APIs, streaming platforms)
  • Implement real-time and batch processing using Apache Spark, Kafka, or Flink
  • Support metadata management, data lineage, and cataloging
  • Tune queries and pipelines for high performance and cost efficiency
  • Implement partitioning, indexing, and caching strategies for large datasets
  • Automate routine ETL/ELT workflows for reliability and speed
  • Ensure compliance with data governance, privacy, and regulatory standards (GDPR, HIPAA, etc.)
What we offer
What we offer
  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion
  • Fulltime
Read More
Arrow Right

Applications Development Sr Programmer Analyst - Python Spark

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Chennai
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Strong expertise in Python (8+ years preferred)
  • Hands-on experience with Apache Spark / PySpark (4+ years)
  • Solid understanding of distributed computing concepts
  • Strong SQL skills and experience with relational databases
  • Experience with big data ecosystems (Hive, HDFS, Delta Lake, or similar)
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right

Senior Technical Product Marketing Manager

Fivetran is the data foundation for AI, enabling enterprises to scale analytics ...
Location
Location
United States , Denver
Salary
Salary:
154283.00 USD / Year
fivetran.com Logo
Fivetran
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4-8 years of experience in product marketing, data engineering, product management, sales engineering, and/or data and analytics consulting
  • Data industry experience and expertise: Either hands-on work with or a deep knowledge of data engineering with a focus on data integration, data lakes, open table formats (e.g. Delta Lake, Apache Iceberg), data warehouses, data catalogs and related technologies (e.g. SQL, dbt, python, Spark)
  • Technical aptitude and a desire to learn more: Knowledge of modern data infrastructure and tools, particularly in relation to data lakes and their role in analytics and storage. Has a keen interest in all things data integration and cloud destinations and a willingness to work on new product areas as they arise
  • Experience working with multiple different teams: Proven experience partnering and coaching across Product, Sales, Marketing, and Enablement teams to deliver impactful results
  • Strong in-person, virtual, and written communication: Exceptional verbal and written communication skills, capable of adapting messaging for different technical personas and mediums
  • Highly-organized with an excellent project management track record: Strong project management skills to juggle multiple initiatives and projects, maintain proactive communication with stakeholders, and meet deadlines
  • An understanding of our target customers: The ability to gain a deep understanding of the needs and challenges of data engineers, data scientists, data architects, and technical stakeholders
Job Responsibility
Job Responsibility
  • Content Development & Messaging: Create high-impact technical collateral across the Fivetran platform, including architecture diagrams, demos, white papers, blogs, pitch decks, technical guides, and other customer-facing content. Develop thought leadership that translates complex technical concepts into clear, value-based messaging for technical audiences. Review and edit materials across the product portfolio to ensure technical accuracy, consistency, and strategic alignment
  • Customer & Market Insights: Conduct research on market trends, customer needs, and competitive dynamics across the modern data ecosystem to inform product positioning and messaging. Partner closely with Product Management to deeply understand features, technical capabilities, and customer use cases across multiple product areas. Develop how-to guides, demo videos, case studies, and technical whitepapers that support adoption and expansion across key products
  • Cross-Functional Collaboration: Act as a technical marketing partner to GTM Product Marketers, Demand Generation, Regional Marketing, Sales, and Enablement to support launches, campaigns, webinars, field events, and sales plays across priority product areas. Collaborate with Partner Marketing to showcase integrations and ecosystem partnerships through joint initiatives such as webinars, workshops, and hands-on labs
  • Technical Expertise: Operationalize technical product marketing best practices across teams. Stay up to date with trends in data integration, cloud data platforms, AI, deployment models, governance, etc. Serve as technical SME on the Fivetran platform, articulating how it fits into customers’ broader data stacks and workflows. Provide best practices, reference architectures, and technical validation to support sales cycles and customer conversations
  • Thought Leadership: Represent Fivetran as a platform expert through events, presentations, webinars, and hands-on labs. Support analyst relations and strategic conversations by providing detailed technical insights and validating product innovation across the portfolio. Help position Fivetran as a foundational component of modern, scalable data infrastructure
What we offer
What we offer
  • 100% employer-paid medical insurance
  • Generous paid time-off policy (PTO), plus paid sick time, inclusive parental leave policy, holidays, and volunteer days off
  • RSU stock grants
  • Professional development and training opportunities
  • Company virtual happy hours, free food, and fun team-building activities
  • Monthly cell phone stipend
  • Access to an innovative mental health support platform that offers personalized care and resources in areas such as: therapy, coaching, and self-guided mindfulness exercises for all covered employees and their covered dependents
  • Fulltime
Read More
Arrow Right

Principal Engineer I – Senior Azure Databricks Administrator

Software Resources has an immediate, direct hire job opportunity for a Principal...
Location
Location
United States , Phoenix
Salary
Salary:
Not provided
softwareresources.com Logo
Software Resources
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of related experience in data analytics administration and development
  • 4+ years of Databricks related experience
  • Bachelor’s degree in related field required
  • Advanced proven experience in Azure Databricks (Workspace management, Clusters, Jobs, Unity Catalog, Delta Lake, User access management, Rest APIs and SDKs)
  • Knowledge of MLFlow & MLOps
  • Deep understanding of Azure infrastructure and data services, including Azure Data Lake, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, networking
  • Experience with CI/CD pipelines (Azure DevOps preferred)
  • Strong programming skills in SQL, Python, and/or PySpark
  • Advanced proven experience in leading cross-functional teams and managing multiple projects simultaneously
  • Advanced ability to see the big picture and align projects with organizational goals
Job Responsibility
Job Responsibility
  • Responsible for delivery and operations of technologies and platforms required to model, transform, analyze, report, visualize data
  • Provide SME expertise in designing, building, optimizing, streamlining and automating the Azure Databricks platform
  • Partner with ML engineers, data scientists, data analysts, and enterprise architects to provide frameworks, set standards, enforce best practices, train and enable users
  • Develop technical skills of one or more junior team-members
  • Take assignments that can be worked on individually without supervision, and manage work effort from concept to completion
  • Design, build, optimize, automate and maintain the Azure Databricks platform, ensuring scalability, security, governance and performance
  • Design, implement and manage Azure Databricks workspaces, clusters, jobs, access management
  • Design, implement and manage policies, monitoring and observability
  • Implement data analytics principles aimed at business enablement, reliability practices and sound recovery procedures
  • Ensure compliance with IT policies, procedures, and industry standards
What we offer
What we offer
  • Competitive salaries
  • An ownership stake in the company
  • Medical and dental insurance
  • Time off
  • A great 401k matching program
  • Tuition assistance program
  • An employee volunteer program
  • A wellness program
  • Fulltime
Read More
Arrow Right

Oracle ERP Service Owner

This is a new and exclusive opportunity for a Oracle ERP Service Owner to join a...
Location
Location
United Kingdom , London
Salary
Salary:
120000.00 - 150000.00 GBP / Year
socialvalueportal.com Logo
Social Value Portal Ltd
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive hands-on leadership in Oracle Cloud ERP/EPM design, configuration, and implementation within an international investment-banking environment
  • Deep practical knowledge of Oracle ERP/EPM modules, financial data integration, reporting tools, API-based integrations, ETL technologies, data lakes, accounting rules engines, and reference data
Job Responsibility
Job Responsibility
  • Establish and drive the long-term roadmap for Oracle Cloud Applications (ERP/EPM) to support financial transformation
  • Serve as the owner of the Oracle ERP/EPM product portfolio, leading module delivery and ensuring alignment with the Finance Technology Book of Work
  • Oversee ongoing and future EMEA strategic projects, ensuring governance, quality, and timely execution, while providing SME guidance throughout project lifecycles
  • Run the Oracle chapter for EMEA
  • Deliver, maintain, and enhance the Oracle Financials platform alongside key in-house finance systems to meet operational, regulatory, and strategic business needs across the region
What we offer
What we offer
  • home working hybrid 50/50%
  • Fulltime
Read More
Arrow Right

Senior Technical Product Marketing Manager

Fivetran is the data foundation for AI, enabling enterprises to scale analytics ...
Location
Location
United States , Oakland
Salary
Salary:
178069.00 - 222586.00 USD / Year
fivetran.com Logo
Fivetran
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4-8 years of experience in product marketing, data engineering, product management, sales engineering, and/or data and analytics consulting
  • Data industry experience and expertise: Either hands-on work with or a deep knowledge of data engineering with a focus on data integration, data lakes, open table formats (e.g. Delta Lake, Apache Iceberg), data warehouses, data catalogs and related technologies (e.g. SQL, dbt, python, Spark)
  • Technical aptitude and a desire to learn more: Knowledge of modern data infrastructure and tools, particularly in relation to data lakes and their role in analytics and storage. Has a keen interest in all things data integration and cloud destinations and a willingness to work on new product areas as they arise
  • Experience working with multiple different teams: Proven experience partnering and coaching across Product, Sales, Marketing, and Enablement teams to deliver impactful results
  • Strong in-person, virtual, and written communication: Exceptional verbal and written communication skills, capable of adapting messaging for different technical personas and mediums
  • Highly-organized with an excellent project management track record: Strong project management skills to juggle multiple initiatives and projects, maintain proactive communication with stakeholders, and meet deadlines
  • An understanding of our target customers: The ability to gain a deep understanding of the needs and challenges of data engineers, data scientists, data architects, and technical stakeholders
Job Responsibility
Job Responsibility
  • Content Development & Messaging: Create high-impact technical collateral across the Fivetran platform, including architecture diagrams, demos, white papers, blogs, pitch decks, technical guides, and other customer-facing content. Develop thought leadership that translates complex technical concepts into clear, value-based messaging for technical audiences. Review and edit materials across the product portfolio to ensure technical accuracy, consistency, and strategic alignment
  • Customer & Market Insights: Conduct research on market trends, customer needs, and competitive dynamics across the modern data ecosystem to inform product positioning and messaging. Partner closely with Product Management to deeply understand features, technical capabilities, and customer use cases across multiple product areas. Develop how-to guides, demo videos, case studies, and technical whitepapers that support adoption and expansion across key products
  • Cross-Functional Collaboration: Act as a technical marketing partner to GTM Product Marketers, Demand Generation, Regional Marketing, Sales, and Enablement to support launches, campaigns, webinars, field events, and sales plays across priority product areas. Collaborate with Partner Marketing to showcase integrations and ecosystem partnerships through joint initiatives such as webinars, workshops, and hands-on labs
  • Technical Expertise: Operationalize technical product marketing best practices across teams. Stay up to date with trends in data integration, cloud data platforms, AI, deployment models, governance, etc. Serve as technical SME on the Fivetran platform, articulating how it fits into customers’ broader data stacks and workflows. Provide best practices, reference architectures, and technical validation to support sales cycles and customer conversations
  • Thought Leadership: Represent Fivetran as a platform expert through events, presentations, webinars, and hands-on labs. Support analyst relations and strategic conversations by providing detailed technical insights and validating product innovation across the portfolio. Help position Fivetran as a foundational component of modern, scalable data infrastructure
What we offer
What we offer
  • 100% employer-paid medical insurance*
  • Generous paid time-off policy (PTO), plus paid sick time, inclusive parental leave policy, holidays, and volunteer days off
  • RSU stock grants*
  • Professional development and training opportunities
  • Company virtual happy hours, free food, and fun team-building activities
  • Monthly cell phone stipend
  • Access to an innovative mental health support platform that offers personalized care and resources in areas such as: therapy, coaching, and self-guided mindfulness exercises for all covered employees and their covered dependents
  • Fulltime
Read More
Arrow Right

Senior Principal Data Engineer

Your success is a train ride away! As we move America’s workforce toward the fut...
Location
Location
United States , Washington
Salary
Salary:
175427.00 - 215100.00 USD / Year
amtrak.com Logo
AMTRAK
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree or equivalent in Computer Science, Computer Information Systems, or a related field
  • 7 years of progressive, post-baccalaureate experience as a Senior Principal Data Engineer or any occupation related to Software Development
  • Experience building solutions using Informatica Data Quality, Informatica MDM, Erwin, and Informatica Power Center
  • Experience administering applications in UNIX, LINUX, and Windows environments
  • Experience writing and executing scripts in Perl
  • Experience implementing solutions on AWS leveraging identity resolutions, data lake, metadata, and data governance
  • Experience performing data migrations, cleansing, integration, and normalization for data warehouses
  • Experience creating customized Master Data Management (MDM) and IDD solutions
  • All positions require pre-employment background check verification and a pre-employment drug screen
Job Responsibility
Job Responsibility
  • Lead the technological advancement and architecture of data-driven solutions across multiple departments including Operations, Finance, Safety, Marketing, and IT
  • Spearhead the design, development, and delivery of end-to-end solutions on contemporary technology platforms
  • Oversee designing data ingestion, transfer, and consumption processes that are cost-effective and performance efficient
  • Act as a Subject Matter Expert (SME) for key and critical source systems to offer essential guidance and mentorship to the Development, Testing, Implementation, and Support teams
  • Adapt and enhance solutions to meet evolving requirements, ensuring alignment with business objectives
  • Fulltime
Read More
Arrow Right

Staff Software Engineer, Marketing Systems

Sequel Med Tech is seeking an experienced Staff Software Engineer to partner dir...
Location
Location
United States , Manchester; Marlborough
Salary
Salary:
170000.00 - 185000.00 USD / Year
sequelmedtech.com Logo
Sequel Med Tech
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelors’ degree in Computer Science or related field, or equivalent combination of education and work experience
  • 6+ years of experience in software engineering, demonstrating increasing technical breadth and responsibility
  • 5+ years of experience troubleshooting and training on enterprise email marketing platforms (SFMC preferred)
  • 5+ years of experience implementing SFMC solutions including complex and cross-feature implementations
  • Experience with building Pages, Microsites, Forms and form Processing
  • Expert level HTML/CSS (email), JavaScript, AMPscript/SSJS, SQL skills
  • Deep knowledge of Marketing Cloud/API’s, Data Extensions and Marketing Cloud SFTP
  • Deep knowledge of Salesforce Marketing Cloud AMPscript
  • Follows CI/CD, version control and test automation
  • CMS expertise in WordPress and Drupal (Webflow a plus)
Job Responsibility
Job Responsibility
  • Own end-to-end engineering across SFMC
  • define patterns, reusable templates, observability using AMPscript, SSJS, HTML/CSS And JavaScript
  • Create scalable architecture across journeys, automations, and CloudPages
  • Architect and own integrations with upstream/downstream systems via API’s, connectors, ETL/ELT jobs, and eventing
  • Define reliability standards, data integrity checks, and event-driven pipelines
  • Design and enforce deliverability strategy
  • own governance around sender reputation, DMARC, list hygiene, complaints/bounces
  • Architect CMS component libraries
  • enforce performance budgets
  • lead accessibility/SEO strategy
What we offer
What we offer
  • 401k plan with a 6% company match and 100% immediate vesting
  • Capped out-of-pocket insulin costs and GLP-1 coverage across all plans
  • Variety of Meritain health insurance plans
  • Flexible Spending Accounts (FSAs) or Health Savings Account (HSA)
  • Vision and dental coverage
  • Voluntary options such as long-term disability, accident, critical illness, hospital indemnity, and pet care discounts
  • Employer-paid short-term disability and life insurance
  • Flexible PTO
  • Generous paid holidays
  • Flex Time options
  • Fulltime
Read More
Arrow Right