CrawlJobs Logo
OptiSol Business Solutions Logo OptiSol Business Solutions · IT - Software Development

Senior Data Engineer – Data Engineering & AI Platforms

India, Chennai, Madurai, Coimbatore · Job Posted December 11, 2025
Apply Position
Job Link Share

Job Description

We are looking for a highly skilled Senior Data Engineer (L2) who can design, build, and optimize modern data platforms while contributing to solutioning and ODC delivery engagements. This opportunity is ideal for individuals who thrive in fast-paced environments, can balance hands-on engineering with leadership responsibilities, and want to grow in the Data & AI engineering space.

Job Responsibility

  • Design, build, and maintain scalable ETL/ELT pipelines across cloud and big data platforms
  • Contribute to architectural discussions by translating business needs into data solutions spanning ingestion, transformation, and consumption layers
  • Work closely with solutioning and pre-sales teams for technical evaluations and client-facing discussions
  • Lead squads of L0/L1 engineers—ensuring delivery quality, mentoring, and guiding career growth
  • Develop cloud-native data engineering solutions using Python, SQL, PySpark, and modern data frameworks
  • Ensure data reliability, performance, and maintainability across the pipeline lifecycle—from development to deployment
  • Support long-term ODC/T&M projects by demonstrating expertise during technical discussions and interviews
  • Integrate emerging GenAI tools where applicable to enhance data enrichment, automation, and transformations

Requirements

  • Strong hands-on expertise in cloud ecosystems (Azure / AWS / GCP)
  • Excellent Python programming skills with data engineering libraries and frameworks
  • Advanced SQL capabilities including window functions, CTEs, and performance tuning
  • Solid understanding of distributed processing using Spark/PySpark
  • Experience designing and implementing scalable ETL/ELT workflows
  • Good understanding of data modeling concepts (dimensional, star, snowflake)
  • Familiarity with GenAI/LLM-based integration for data workflows
  • Experience working with Git, CI/CD, and Agile delivery frameworks
  • Strong communication skills for interacting with clients, stakeholders, and internal teams

Nice to have

  • Demonstrated experience in leading technical teams or delivery squads
  • Prior involvement in architecture and solution design for data engineering programs
  • Certifications in cloud or data engineering (Azure DP-203, AWS Data Analytics, etc.)
  • Knowledge of data governance, quality, and compliance frameworks
  • Exposure to large-scale cloud data migrations or modernization initiatives

What we offer

  • Opportunity to work at the intersection of Data Engineering, Cloud, and Generative AI
  • Hands-on exposure to modern data stacks and emerging AI technologies
  • Collaboration with experts across Data, AI/ML, and cloud practices
  • Access to structured learning, certifications, and leadership mentoring
  • Competitive compensation with fast-track career growth and visibility
OptiSol Business Solutions - All Job Offers

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Suggested Tags: Frontend Developer Secretary Talent Manager Salesforce Administrator Data Analyst
Similar Jobs for

Senior Data Engineer – Data Engineering & AI Platforms

8 matching positions

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior AI Engineer (Agentic AI / LLM Engineering)

We’re partnering with a rapidly growing, innovation-focused organization that is...
Location
Location
United States
Salary
Salary:
Not provided
zeektek.com Logo
Zeektek
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of experience in software engineering, data engineering, or AI/ML engineering
  • Hands-on experience building applications using LLMs (e.g., GPT, open-source models)
  • Experience with agentic AI frameworks (e.g., LangChain, AutoGen, CrewAI, or similar)
  • Strong programming skills in Python
  • Experience working with large datasets and distributed data platforms
  • Familiarity with Databricks or similar modern data platforms
  • Experience building production-grade AI systems (not just POCs)
  • Strong understanding of prompt engineering and LLM optimization techniques
Job Responsibility
Job Responsibility
  • Design and build agentic AI systems leveraging large language models (LLMs)
  • Develop scalable AI solutions using modern data and AI platforms (Databricks preferred)
  • Translate business problems into production-ready AI workflows and applications
  • Collaborate with product and architecture teams to define AI use cases and technical approaches
  • Implement and optimize: Prompt engineering strategies
  • Token usage and cost efficiency
  • Model performance and response quality
  • Work with large-scale datasets to support training, fine-tuning, and inference workflows
  • Contribute to the development of AI engineering standards, frameworks, and best practices
  • Partner with data science teams to integrate models into production environments
What we offer
What we offer
  • Weekly Direct Deposit
  • 401K Matching
  • Competitive medical, dental and vision insurance
  • Consistent communication throughout your project
  • ZeekTek Referral Program
Read More
Arrow Right

Senior Data Engineer - AI and Analytics

We're building a world of health around every individual — shaping a more connec...
Location
Location
United States , Buffalo Grove
Salary
Salary:
101970.00 - 203940.00 USD / Year
https://www.cvshealth.com/ Logo
CVS Health
Expiration Date
June 24, 2026
Flip Icon
Requirements
Requirements
  • 3-5+ years of experience with SQL, NoSQL
  • 3-5+ years of experience with Python
  • 3+ years of experience with Data warehouses (such as data modeling and technical architectures) and infrastructure components
  • 3+ years of experience with ETL/ELT, and building high-volume data pipelines
  • 3+ years of experience with reporting/analytic tools
  • 3+ years of experience with Query optimization, data structures, transformation, metadata, dependency, and workload management
  • 3+ years of experience with Big data and cloud architecture
  • 3+ years of hands-on experience building modern data pipelines within a major cloud platform (preferably GCP, open to AWS or Azure)
  • 3+ years of experience with deployment/scaling of apps on containerized environment (i.e. Kubernetes, AKS)
  • 3+ years of experience with real-time and streaming technology (i.e. Azure Event Hubs, Azure Functions, Kafka, Spark Streaming)
Job Responsibility
Job Responsibility
  • Design, develop, and maintain optimal data pipelines to assemble large and intricate datasets
  • Cater to the business requirements of various CVS lines of business
  • Collaborate closely with teams to craft tools to provide actionable insights and integrate them with consumer touchpoints
  • Solve problems associated with large scale complex, structured and unstructured data
What we offer
What we offer
  • Medical, dental, and vision coverage
  • paid time off
  • retirement savings options
  • wellness programs
  • bonus, commission or short-term incentive program
  • Fulltime
Read More
Arrow Right

Senior Data Engineer - AI Infrastructure

We are building a large-scale data platform that transforms raw system logs into...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling, or data engineering OR Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in business analytics, data science, software development, data modeling, or data engineering OR equivalent experience.
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role.
  • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Design and implement large-scale data pipelines using PySpark and distributed processing frameworks
  • Build and maintain data models that accurately represent underlying system behavior and business logic
  • Ensure high standards of data correctness, completeness, and consistency across datasets
  • Develop validation, monitoring, and alerting mechanisms to detect data quality issues
  • Partner with data scientists to support experimentation and analytics use cases
  • Collaborate with platform engineers to ensure efficient data ingestion, processing, and storage
  • Optimize pipelines for performance, scalability, and cost efficiency
  • Define and enforce best practices for schema design, data transformations, and pipeline reliability
  • Fulltime
Read More
Arrow Right

Senior Data Engineer - AI Focused

At Doctolib, we're on a mission to transform healthcare through the power of AI....
Location
Location
France , Paris
Salary
Salary:
Not provided
doctolib.fr Logo
Doctolib
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master’s or Ph.D. degree in Computer Science, Data Engineering, or a related field
  • 5+ years of experience in Data Engineering, ideally supporting AI or ML workloads
  • Strong experience with the GCP data ecosystem
  • Proficiency in Python and SQL, with experience in data pipeline orchestration (e.g., Airflow, Dagster, Cloud Composer)
  • Deep understanding of NoSQL systems (e.g., MongoDB) and vector databases (e.g., FAISS, Vector Search)
  • Experience designing data architectures for RAG, embeddings, or model training pipelines
  • Knowledge of data governance, security, and compliance for sensitive or regulated data
  • Familiarity with W&B / MLflow / Braintrust / DVC for experiment tracking and dataset versioning (extract snapshots, change tracking, reproducibility)
  • Familiarity with containerized environments (Docker, Kubernetes) and CI/CD for data workflows
  • A collaborative mindset and passion for building the data foundations of next-generation AI systems
Job Responsibility
Job Responsibility
  • Ensure high standards of data quality for AI model inputs
  • Design, build, and maintain scalable data pipelines on Google Cloud Platform (GCP) for AI and machine learning use cases
  • Implement data ingestion and transformation frameworks that power Retrieval systems and training datasets for LLMs and multimodal models
  • Architect and manage NoSQL and Vector Databases to store and retrieve embeddings, documents, and model inputs efficiently
  • Collaborate with ML and platform teams to define data schemas, partitioning strategies, and governance rules that ensure privacy, scalability, and reliability
  • Integrate unstructured and structured data sources (text, speech, image, documents, metadata) into unified data models ready for AI consumption
  • Optimize performance and cost of data pipelines using GCP native services (BigQuery, Dataflow, Pub/Sub, Cloud Storage, Vertex AI)
  • Contribute to data quality and lineage frameworks, ensuring AI models are trained on validated, auditable, and compliant datasets
  • Continuously evaluate and improve our data stack to accelerate AI experimentation and deployment
What we offer
What we offer
  • Free comprehensive health insurance for you and your children
  • Parent Care Program: additional leave on top of the legal parental leave
  • Free mental health and coaching services through our partner Moka.care
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
  • Work from EU countries and the UK for up to 10 days per year, thanks to our flexibility days policy
  • Work Council subsidy to refund part of a sport club membership or a creative class
  • Up to 14 days of RTT
  • Lunch voucher with Swile card
  • Fulltime
Read More
Arrow Right

Senior AI & Data Engineer

The Senior AI & Data Engineer is an individual contributor role that acts as the...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
https://www.hpe.com/ Logo
Hewlett Packard Enterprise
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Data Science, AI/ML, Engineering, Mathematics, or a related technical discipline
  • PhD is a plus
  • 7 – 10 years of hands-on experience in AI/ML engineering, applied data science, or LLM engineering roles
  • Proven track record of delivering production AI systems
  • Deep expertise with at least two major LLM platforms (Claude, GPT, Gemini, or equivalent)
  • Significant experience with Collibra or an equivalent enterprise data governance platform
  • Demonstrated experience leading cross-functional AI initiatives and mentoring junior engineers
  • Strong ML fundamentals alongside modern generative AI skills
  • Experience with responsible AI practices, including fairness auditing, explainability, and content safety, is strongly preferred
Job Responsibility
Job Responsibility
  • Serve as the dual AI & data SME for the team and organization
  • Define and uphold engineering standards, design patterns, and best practices across both AI and data engineering disciplines
  • Lead technical discovery for new AI and data use cases
  • Participate in and lead cross-functional initiatives where AI and data strategy intersect
  • Mentor and upskill the Applied AI Engineer and AI Data Engineer
  • Architect and deliver complex agentic AI systems
  • Design and implement advanced RAG architectures
  • Lead LLM evaluation frameworks
  • Assess and implement LLM fine-tuning and alignment strategies
  • Own LLM integration architecture
What we offer
What we offer
  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion
Read More
Arrow Right