CrawlJobs Logo

Data Engineer - Python, AI

India, Pune · Job Posted May 28, 2026
Apply Position
Job Link Share

Job Description

We are looking for a mid-level Python Developer with combined experience in Data Engineering and AI/NLP engineering. The candidate will build NLP pipelines using libraries such as Flair, BERT, and LLM frameworks, and will also work on large-scale data processing using PySpark, Pandas, and related data tools. The role includes developing APIs, integrating with platform services, and supporting CI/CD deployments using GitHub and LightSpeed Enterprise.

Job Responsibility

  • Develop and optimize ETL/data processing jobs using PySpark, Pandas, PyArrow, and related libraries
  • Build and maintain NLP pipelines using Flair, BERT, and LLM-based models
  • Develop scalable ingestion and data transformation pipelines for AI and analytics use cases
  • Build and maintain Flask-based APIs for model inference and service integrations
  • Use regular expressions for text cleaning, parsing, and NLP preprocessing
  • Integrate caching and fast lookups using Redis
  • Manage and deploy ML models using MLflow for tracking and versioning
  • Support CI/CD workflows using GitHub, LightSpeed Enterprise, and deployment pipelines
  • Create and maintain Autosys JILs for job scheduling and automation
  • Use basic Linux commands for troubleshooting, operations, and deployment tasks
  • Monitor application and system health using ITRS Geneos
  • Write unit tests and improve automation test coverage (PyTest/unittest)
  • Work with REST APIs, microservices, and basic shell scripting
  • Work with cloud services (ECS), including boto3

Requirements

  • 10–12 years of hands-on Python programming experience
  • Strong fundamentals in Python, OOP, and design patterns
  • Experience with NLP libraries such as Flair, BERT, HuggingFace Transformers, or similar
  • Solid experience with PySpark, Pandas, PyArrow, and distributed data pipelines
  • Experience building APIs using Flask (FastAPI is a plus)
  • Experience with MLflow for model tracking and deployment
  • Good understanding of CI/CD practices and Git workflows
  • Experience working with Redis or similar in-memory stores
  • Experience with Autosys JILs for job scheduling
  • Comfortable with Linux command line and shell scripting
  • Strong debugging, problem-solving, and teamwork skills
  • Exposure to cloud services
  • AWS boto3 experience is an asset

Nice to have

  • Experience with Polars or Dask for high-performance data processing
  • Experience with PyTorch or TensorFlow for model training
  • Experience with Docker, Kubernetes, or containerized deployments
  • Experience with monitoring tools such as ITRS Geneos
  • Experience with FastAPI, Airflow, or Prefect

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Engineer - Python, AI

8 matching positions

Senior Python Engineer (Data Engineering & AI Agents)

Our client is a leading global investment management company headquartered in Lo...
Location
Location
Poland; Spain; United Kingdom
Salary
Salary:
Not provided
Intellias
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years building production software in Python, with strong engineering fundamentals (testing, performance, clean design)
  • Solid data engineering: SQL, columnar formats (e.g. Parquet), pipeline design, and handling datasets large enough that naive approaches don’t scale
  • Hands-on experience with at least one analytical or query engine (e.g. DuckDB, Trino, Spark, ClickHouse)
  • Real experience building LLM / agent applications: retrieval (RAG), vector databases, and tool / function calling
  • A working understanding of data governance: cataloguing, metadata, lineage, and access control (RBAC / ABAC)
  • An instinct for data quality and trustworthy "golden" sources
Job Responsibility
Job Responsibility
  • Build production-grade Python services and data pipelines over large data stores (columnar / time-series and relational), and the queries that join across them
  • Select and implement the right query or analytical engine for each workload, rather than defaulting to one
  • Build catalogue, metadata, lineage and semantic layers that make data discoverable and consistently understood across teams
  • Implement access control that travels with the data: fusing sensitivity and licensing scope, enforced at the point of use, including for AI agents
  • Build agent-facing data access: retrieval (RAG), vector search, and APIs / MCP servers, with permissions applied before context reaches the model
  • Apply LLMs pragmatically to data work (metadata generation, classification, entity resolution) with humans in the loop and evaluate the quality of what the agents produce
  • Help keep data trustworthy: establish golden sources, deduplication and data-quality checks at the source
  • Contribute to discovery and solutioning: assessing current state, weighing build-vs-adopt, and shaping pragmatic, costed plans
  • Fulltime
Read More
Arrow Right

Senior Python Engineer (Data Engineering & AI Agents)

Our client is a leading global investment management company headquartered in Lo...
Location
Location
Poland; Spain; United Kingdom
Salary
Salary:
Not provided
Intellias
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years building production software in Python, with strong engineering fundamentals (testing, performance, clean design)
  • Solid data engineering: SQL, columnar formats (e.g. Parquet), pipeline design, and handling datasets large enough that naive approaches don't scale
  • Hands-on experience with at least one analytical or query engine (e.g. DuckDB, Trino, Spark, ClickHouse)
  • Real experience building LLM / agent applications: retrieval (RAG), vector databases, and tool / function calling
  • A working understanding of data governance: cataloguing, metadata, lineage, and access control (RBAC / ABAC)
  • An instinct for data quality and trustworthy golden sources
Job Responsibility
Job Responsibility
  • Build production-grade Python services and data pipelines over large data stores (columnar / time-series and relational), and the queries that join across them
  • Select and implement the right query or analytical engine for each workload, rather than defaulting to one
  • Build catalogue, metadata, lineage and semantic layers that make data discoverable and consistently understood across teams
  • Implement access control that travels with the data: fusing sensitivity and licensing scope, enforced at the point of use, including for AI agents
  • Build agent-facing data access: retrieval (RAG), vector search, and APIs / MCP servers, with permissions applied before context reaches the model
  • Apply LLMs pragmatically to data work (metadata generation, classification, entity resolution) with humans in the loop and evaluate the quality of what the agents produce
  • Help keep data trustworthy: establish golden sources, deduplication and data-quality checks at the source
  • Contribute to discovery and solutioning: assessing current state, weighing build-vs-adopt, and shaping pragmatic, costed plans
  • Fulltime
Read More
Arrow Right

Senior Generalist (Python, Data) Engineer - Healthcare AI Platform

We are looking for a senior hands-on engineer to own and maintain a sophisticate...
Location
Location
United States
Salary
Salary:
Not provided
provectus.com Logo
Provectus
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years of hands-on engineering experience with production systems
  • Generalist mindset, comfortable across backend development, data pipelines, and cloud infrastructure
  • Strong production Python skills and solid software engineering fundamentals
  • Hands-on experience with AWS (S3, Lambda, ECS, SQS, Bedrock, RDS or similar)
  • Experience with analytical data pipelines (SQL, Spark) and data processing at scale
  • Experience with Infrastructure as Code (Terraform, CDK, or CloudFormation)
  • Experience with AI-assisted development workflows (Claude Code or alternatives), not just autocomplete
  • Experience maintaining and scaling systems in production, not just building POCs
  • Autonomous working style, you don't need a project manager to tell you what to do next
  • US-based, required due to project compliance and data security constraints
Job Responsibility
Job Responsibility
  • Maintain, improve, and extend an AI platform already running in production
  • Handle a mix of backend development, data engineering, DevOps, and infrastructure work
  • Translate business and product requirements into technical decisions independently
  • Work directly with internal stakeholders and consumer teams
  • Establish and maintain engineering best practices for the platform
  • Identify and resolve technical issues as the first line of defense
What we offer
What we offer
  • Full ownership of a production AI platform
  • Remote-first, fully flexible work environment
  • Annual performance bonus
  • Health, dental, vision, 401(k) with company match, unlimited PTO
  • Long-term engagement
  • this role extends beyond the initial project
Read More
Arrow Right

Senior Python Engineer (Data Engineering & AI Agents)

Our client is a leading global investment management company headquartered in Lo...
Location
Location
Poland; Spain; United Kingdom
Salary
Salary:
Not provided
Intellias
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 6+ years building production software in Python, with strong engineering fundamentals (testing, performance, clean design)
  • Solid data engineering: SQL, columnar formats (e.g. Parquet), pipeline design, and handling datasets large enough that naive approaches don't scale
  • Hands-on experience with at least one analytical or query engine (e.g. DuckDB, Trino, Spark, ClickHouse)
  • Real experience building LLM / agent applications: retrieval (RAG), vector databases, and tool / function calling
  • A working understanding of data governance: cataloguing, metadata, lineage, and access control (RBAC / ABAC)
  • An instinct for data quality and trustworthy 'golden' sources
Job Responsibility
Job Responsibility
  • Build production-grade Python services and data pipelines over large data stores (columnar / time-series and relational), and the queries that join across them
  • Select and implement the right query or analytical engine for each workload, rather than defaulting to one
  • Build catalogue, metadata, lineage and semantic layers that make data discoverable and consistently understood across teams
  • Implement access control that travels with the data: fusing sensitivity and licensing scope, enforced at the point of use, including for AI agents
  • Build agent-facing data access: retrieval (RAG), vector search, and APIs / MCP servers, with permissions applied before context reaches the model
  • Apply LLMs pragmatically to data work (metadata generation, classification, entity resolution) with humans in the loop and evaluate the quality of what the agents produce
  • Help keep data trustworthy: establish golden sources, deduplication and data-quality checks at the source
  • Contribute to discovery and solutioning: assessing current state, weighing build-vs-adopt, and shaping pragmatic, costed plans
Read More
Arrow Right

Ai Engineer Ii & Senior Ai Engineer - Getting Customers Ready For Ai

Security represents the most critical priorities for our customers in a world aw...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • Must pass the Microsoft background and Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Design, build, and improve AI-powered solutions, including machine learning models, LLM-based applications, RAG systems, and agentic workflows that solve customer and business challenges
  • Develop and operate scalable data pipelines, ETL processes, training datasets, and AI infrastructure to support model development, deployment, and lifecycle management
  • Deploy, monitor, and optimize AI systems using MLOps practices, leveraging telemetry and feedback to improve performance, reliability, security, and scalability
  • Transform large-scale, multi-source data into contextual intelligence, automation, and decision-support capabilities that drive customer and business outcomes
  • Build and integrate AI capabilities into applications, services, APIs, and platforms to deliver end-to-end customer experiences
  • Contribute to AI readiness initiatives through frameworks, metrics, telemetry, and solutions that help customers securely adopt and operationalize AI technologies
  • Demonstrate a builder mindset by rapidly prototyping, experimenting, and iterating on AI solutions while navigating ambiguity and evolving requirements
  • Partner across engineering, data science, product, and customer teams to translate complex problems into practical, AI-driven solutions
  • Write maintainable, well-tested code and contribute to engineering excellence through documentation, operational rigor, governance, and adherence to security, compliance, and Responsible AI practices
  • Embody our culture and values
  • Fulltime
Read More
Arrow Right

Data Engineer (Big Data, Python, Databricks) - Assistant Vice President

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Chennai, Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5-8 years of relevant handson experience in Big Data technologies like Cloudera, Python, HQL, Java/PySpark
  • Knowledge on Machine Learning, AI would be added advantage
  • Experience in systems analysis, data analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Bachelor’s degree/University degree or equivalent experience
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
  • Fulltime
Read More
Arrow Right

Ai Data Engineer - Kdb

We’re seeking an AI Data Engineer to lead on building next‑generation analytics ...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Seasoned engineer who has delivered impactful AI initiatives with demonstrable positive return on investment
  • Previous experience working at the forefront of technological evolution, adapting as paradigms have shifted
  • Clear vision for applying AI within front‑office trading systems, with a strong understanding of the practical challenges and intricacies involved
  • Hands-on development experience with Python
  • Deep understanding of front office trading workflows, low-latency architectures, market data dynamics, and the operational challenges of integrating AI into real-time trading environments
  • Demonstrated track record of evolving with emerging technologies and leading adoption of new paradigms, with the ability to translate cutting-edge AI capabilities into practical, production-ready solutions
Job Responsibility
Job Responsibility
  • Build and maintain systems that collect, store, process, and analyse data, such as data pipelines, data warehouses and data lakes to ensure that all data is accurate, accessible, and secure
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehouses and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - Data (Python, AWS)

Location
Location
Austria , Vienna
Salary
Salary:
Not provided
sportradar.com Logo
Sportradar
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years of data engineering experience with proven track record of leading complex data projects from conception to delivery
  • Exceptional communication skills and experience working in cross-functional teams with analysts, product managers, and business stakeholders
  • AWS & Data Engineering: Very strong hands-on experience with AWS services (S3, Lambda, Glue, Athena, Redshift, EMR, etc.) and proficiency with Apache Spark for large-scale data processing
  • Backend Development: Strong experience with Python for building data processing services and APIs, plus expert-level SQL for data processing and analytics
  • Infrastructure & DevOps: Hands-on experience with Docker, Terraform, and CI/CD pipelines with automation best practices for data systems
  • Clean Code Advocate: Strong commitment to writing clean, maintainable, well-documented code with comprehensive testing and deep knowledge of analytics/reporting requirements
  • Data Architecture: Experience designing scalable data architectures, data modeling, and optimizing data processing workflows
  • Dashboard Development: Experience creating and managing analytics dashboards in bi tools (Tableau, Qlik Sense, Quicksuite, Power BI) and data visualization solutions to present complex insights to stakeholders
  • GenAI Tools (beneficial): Experience working with contextual and prompt engineering with GitHub Copilot, Claude, or similar AI assistants
Job Responsibility
Job Responsibility
  • Scale & Performance Engineering: Processing and analyzing terabytes of advertising data with sub-second query performance while building and maintaining robust ETL pipelines using Spark and AWS services to handle massive data volumes daily
  • Data Pipeline Architecture & Development: Designing and building scalable data processing systems, developing backend APIs and microservices (Python or Go), architecting data flows that support both batch and real-time analytics requirements, and managing user-facing dashboards that visualize complex data insights
  • Infrastructure & Data Quality Operations: Implementing robust monitoring and alerting systems to detect data quality issues, managing AWS infrastructure using Terraform, implementing CI/CD best practices, and maintaining high coding standards across data processing systems
  • Cross-Functional Leadership & Collaboration: Leading large-scale data projects from requirements gathering through delivery, bridging technical implementation with business requirements, mentoring team members, and presenting technical concepts to stakeholders while challenging requirements constructively
  • End-to-End Data System Ownership: Taking complete ownership of complex data engineering projects while ensuring high availability and accuracy for both internal stakeholders and external clients, championing clean code principles, and serving as a knowledge leader who supports delivering the right data solutions
What we offer
What we offer
  • Collaborative environment with colleagues from all over the world (Engineering offices in Europe, Asia and US)
  • Ability to shape your own workday and career via a clearly defined professional and personal development plan
  • Opportunity to work with senior leadership, develop yourself and build your career within an inspiring and fast-growing company and digital sports environment
  • Vibrant and inclusive community, including Women in Tech and Pride groups which welcome all participants
  • Company culture that promotes social aspects, sports, physical exercise and fun
  • Innovative and cross-team challenges like ShipIt, office sports tournaments in Darts and Table Tennis and unique beer brewing competitions
  • Competitive salary and benefits (e.g. retirement pension and insurance plan)
  • Sportradar takes over the full costs of € 365.- for the Öffi-Ticket (Jahreskarte) for you
  • Fulltime
Read More
Arrow Right