CrawlJobs Logo

Software Engineer - Data Ingestion

Israel, Ramat Gan · Job Posted March 04, 2026
Apply Position
Job Link Share

Job Description

Coralogix is a modern, full-stack observability platform transforming how businesses process and understand their data. Our unique architecture powers in-stream analytics without reliance on expensive indexing or hot storage. We specialize in comprehensive monitoring of logs, metrics, trace and security events with features such as APM, RUM, SIEM, Kubernetes monitoring and more, all enhancing operational efficiency and reducing observability spend by up to 70%. We are seeking engineers with hands-on experience in developing and managing distributed systems and microservices in a production environment. The ideal candidate should have a solid background in infrastructure and operations. The team builds high-throughput ingestion and processing pipelines, efficient storage on object stores, and systems for data governance, usage reporting, and routing data to multiple physical locations. You will work on cloud-native, production systems written in Rust and Scala, operating at scale with Kafka, Postgres, Redis, and object storage, run on Kubernetes and operate in a multi-cloud environment.

Job Responsibility

  • Develop and operate distributed systems in production
  • Build Kafka-based ingestion and processing pipelines
  • Design systems for data governance, retention, deletion, and usage reporting
  • Work with Postgres and Redis, requiring solid database design and operational knowledge
  • Implement efficient persistence using column-oriented data formats and object storage

Requirements

  • Located in Israel
  • 5+ years of software development experience
  • Production experience with large-scale Apache Kafka or comparable distributed data streaming platforms
  • Strong understanding of distributed systems, databases, and production operations
  • Experience with Scala or Rust
  • B.Sc. in Computer Science or an equivalent field

Nice to have

  • Experience with Kafka Streams or similar frameworks
  • Experience with column-oriented data formats and large-scale analytical storage systems
  • Experience with Apache Arrow and DataFusion

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Software Engineer - Data Ingestion

8 matching positions

Senior Software Engineer, Data Ingestion

Fullstory’s mission is to help teams create amazing online experiences for their...
Location
Location
United States , Atlanta
Salary
Salary:
160000.00 - 180000.00 USD / Year
fullstory.com Logo
Fullstory
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • You possess a strong understanding the nuances of distributed architectures and tackling capacity and performance challenges when dealing with data at scale
  • You have experience writing Golang code in production
  • You have experience with Kubernetes and supporting highly available and reliable cloud based microservices in production
  • You have worked on asynchronous or streaming ingestion and processing systems and frameworks
  • You leverage AI tools to enhance work quality by implementing AI solutions that optimize efficiency
Job Responsibility
Job Responsibility
  • Engineer distributed systems that operate at tens to hundreds of thousands of requests per second using Go, Kubernetes, and GCP
  • Explore ideas about how to unlock new features through thoughtful architecture and framework designs
  • Ensure the quality and reliability of Fullstory's capture and extraction systems across many services and downstream applications both internally and externally
  • Collaborate with technical leaders and product experts to evolve the technical roadmap for Ingestion services, and to participate in collaborative development efforts across the Engineering organization
What we offer
What we offer
  • Flexibility and Connection
  • flexible PTO policy
  • annual company-wide closure
  • federal holidays
  • sponsored benefit packages for US-based Fullstorians
  • supplemental coverage options for international Fullstorians
  • professional development opportunities through training programs
  • annual learning subsidy for US and EMEA-based employees
  • monthly productivity stipend
  • team off-sites
  • Fulltime
Read More
Arrow Right

Staff Software Engineer - Data

Uber's mission is to reimagine the way the world moves for the better. Here, bol...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
uber.com Logo
Uber
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field
  • Experience Level: 10+ years of hands-on experience in Data Engineering, with a proven track record of delivering results at a Staff Engineer level (or equivalent scope) at a premier technology company
  • Expert SQL Competency: 10+ years of hands-on, expert-level SQL experience
  • Data Modeling & Warehousing: Extensive experience designing dimensional data models (Star/Snowflake schemas) and data warehouses
  • Software Engineering Fundamentals: Proficiency in at least one high-level programming language (Java, Scala, Python, or Go)
  • Big Data Ecosystem: 10+ years of experience working with distributed data systems (Hadoop, Hive, Spark) and MPP databases (Vertica, Redshift, etc.)
  • End-to-End Architecture: Experience designing full-lifecycle data systems, including logging, ingestion (Batch/Stream), quality frameworks, and monitoring
  • Technical Leadership: Excellent written and verbal communication skills
  • Mentorship & Growth: A strong passion for driving engineering excellence and mentoring engineers
Job Responsibility
Job Responsibility
  • Own the Technical Vision: You will own and drive the technical roadmap for the Payments data ecosystem, balancing long-term architectural scalability with short-term business critical deliveries
  • Navigate Ambiguity: Actively identify strategically important problems and inefficiencies without waiting for instruction
  • Drive Alignment: See the big picture and drive consensus on complex technical decisions across the organization
  • Architect at Scale: Design and implement resilient, cost-effective, and high-scale batch and streaming pipelines that power critical support operations and financial analytics
  • Elevate Data Standards: Define and enforce robust data modeling standards, data contracts, and governance frameworks
  • Optimize & Automate: Identify opportunities to automate manual workflows (like SLA tracking and issue detection) and optimize infrastructure efficiency to lower TCO
  • Raise the Bar: Champion sustainable engineering practices
  • Be a Trusted Mentor: Serve as a humble mentor and technical advisor to both junior engineers and peer leaders
  • Force Multiplier: Act as a role model for judgment and responsibility
  • Fulltime
Read More
Arrow Right

Software Engineer, Data Platform

Our systems operate at the massive speed and scale of algorithmic trading. We pr...
Location
Location
United States; United Kingdom; Netherlands , New York; London; Amsterdam
Salary
Salary:
115000.00 - 200000.00 USD / Year
helpcare.ai Logo
Helpcare AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent hands-on experience
  • 3-5 years of experience building internal data platform tooling or services that support large-scale data processing and analytics workflows (primarily using Python and SQL
  • Rust is a plus)
  • Experience with workflow orchestration systems used to manage complex data pipelines (Temporal, Airflow, Dagster, or similar)
  • Familiarity with columnar data formats such as Parquet or Arrow
  • Experience working with large-scale datasets and high-volume data pipelines in production environments
  • US citizen/visa only
Job Responsibility
Job Responsibility
  • Work with engineers and business teams to understand our datasets and build systems to store, process, and productionize them
  • Develop and maintain reliable data pipelines and workflow systems for large-scale data ingestion and transformation
  • Build internal services and tooling that support analytics, automation, and data operations
  • Implement ETL and data transformation processes for analytical databases and internal applications
  • Develop and maintain workflow orchestration for complex data processes
  • Optimize storage, retrieval, and processing performance across large datasets
  • Partner with analysts and researchers to support data modeling and analytics workflows
  • Monitor and troubleshoot data infrastructure to ensure reliability and data integrity
  • Continuously improve the platform through better system design, tooling, and engineering practices
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Data Pipeline

Fullstory’s mission is to help teams create amazing online experiences for their...
Location
Location
United States , Atlanta
Salary
Salary:
160000.00 - 180000.00 USD / Year
fullstory.com Logo
Fullstory
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong understanding of the nuances of distributed architectures and tackling capacity and performance challenges when dealing with data at scale
  • Experience writing Golang code in production
  • Experience with Kubernetes and supporting highly available and reliable cloud based microservices in production
  • Worked on asynchronous or streaming ingestion and processing systems and frameworks
  • Leverage AI tools to enhance work quality by implementing AI solutions that optimize efficiency
Job Responsibility
Job Responsibility
  • Engineer distributed systems that operate at tens to hundreds of thousands of requests per second using Go, Kubernetes, and GCP
  • Explore ideas about how to unlock new features through thoughtful architecture and framework designs
  • Ensure the quality and reliability of Fullstory's capture and extraction systems across many services and downstream applications both internally and externally
  • Collaborate with technical leaders and product experts to evolve the technical roadmap for Ingestion services, and to participate in collaborative development efforts across the Engineering organization
What we offer
What we offer
  • Flexibility and Connection with vibrant HQ in Atlanta and a tight-knit group in London
  • Flexible PTO policy and an annual company-wide closure, along with federal holidays
  • Sponsored benefit packages for US-based Fullstorians, and supplemental coverage options for international Fullstorians
  • Professional development opportunities through training programs and an annual learning subsidy for US and EMEA-based employees
  • Monthly productivity stipend for US and EMEA-based Fullstorians
  • Team Collaboration through team off-sites and an annual full-company meet-up
  • Paid parental leave
  • Bereavement leave, including miscarriage/pregnancy loss
  • Fulltime
Read More
Arrow Right

Software Engineer - Data Engineering

At Catawiki, data sits at the core of our decision-making, powering everything f...
Location
Location
Netherlands , Amsterdam
Salary
Salary:
Not provided
catawiki.com Logo
Catawiki
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years of hands-on experience building and operating data systems in production
  • Fluent in Python and SQL
  • Experience with data integration tools such as Fivetran and/or Airbyte
  • Experience with CI/CD, Infrastructure as Code (e.g. Terraform), and modern DataOps practices
  • Experience with cloud platforms (GCP is a plus)
  • Familiar with parts of the data stack such as BigQuery, PubSub, DataFlow, GKE, Airflow, Airbyte, FastAPI and Prometheus
  • Experience with streaming pipelines using technologies like Kafka, Pub/Sub, Dataflow, or Apache Beam
  • Keen to learn new tools, support data platform and machine learning engineering initiatives
  • Understand the importance of data privacy and GDPR
Job Responsibility
Job Responsibility
  • Build and Scale Data Pipelines: Maintain and develop reliable batch and streaming pipelines that ingest data from internal systems and third-party sources into Catawiki’s data warehouse
  • Empower Data Science and AI: Maintain and enhance the tools and platforms used by Data Scientists for analysis, experimentation, model training, and model deployment
  • Protect Data and Privacy: Ensure data is stored securely and that governance, access control, and privacy standards are consistently applied across the data platform
  • Run and Evolve the Data Platform: Maintain the infrastructure that hosts our data tools and applications, keeping it scalable, stable, and cost-effective
  • Own Core Data Tooling: Self-host and operate key data engineering tools such as Airflow and Airbyte on Kubernetes
  • Keep the Lights On: Provide operational support to ensure pipelines, platforms, and tools run smoothly and reliably for teams across the business
What we offer
What we offer
  • €100 Catavoucher when you join
  • €50 Catavoucher on your birthday
  • An extra day off each year to “Pursue Your Passion”
  • Additional leave for key work anniversaries and important life events
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Data

Patreon is looking for a Senior Software Engineer, Data to support our mission. ...
Location
Location
United States , San Francisco; New York
Salary
Salary:
200000.00 - 300000.00 USD / Year
patreon.com Logo
Patreon
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of experience in software development
  • At least 2+ years of experience in building scalable, production-grade data pipelines
  • Familiarity with SQL and distributed data processing tools like Spark, Flink, Kafka Streams, or similar
  • Strong programming foundations in Python or similar language, with good software engineering design patterns and principles (testing, CI/CD, monitoring)
  • Familiar with modern data lakes (eg: Delta Lake, Iceberg)
  • Familiar with data warehouses (eg: Snowflake, Redshift, BigQuery) and production data stores such as relational (eg: MySQL, PostgreSQL), object (eg: S3), key-value (eg: DynamoDB) and message queues (eg: Kinesis, Kafka)
  • Excellent collaboration and communication skills
  • comfortable partnering with non-technical stakeholders, writing crisp design docs, giving actionable feedback, and can influence without authority across teams
  • Understanding of data modeling and metric design principles
  • Passionate about data quality, system reliability, and empowering others through well-crafted data assets
Job Responsibility
Job Responsibility
  • Design, build, and maintain the pipelines that power all data use cases. This includes ingestion of raw data from production databases, object storage, and message queues, and vendors into our Data Lake, and building core datasets and metrics
  • Develop intuitive, performant, and scalable data models (facts, dimensions, aggregations) that support product features, internal analytics, experimentation, and machine learning workloads
  • Implement robust batch and streaming pipelines using Spark, Python, and Airflow
  • Build pipelines adhering to standards for accuracy, completeness, lineage, and dependency management. Build monitoring and observability so teams can trust what they’re using
  • Work with Product, Data Science, Infrastructure, Finance, Marketing, and Sales to turn ambiguous questions into well-scoped, high-impact data solutions
  • Pay down technical debt, improve automation, and follow best practices in data modeling, testing, and reliability. Mentor peers earlier in their career within the team
What we offer
What we offer
  • Equity plans
  • Healthcare
  • Flexible time off
  • Company holidays and recharge days
  • Commuter benefits
  • Lifestyle stipends
  • Learning and development stipends
  • Patronage
  • Parental leave
  • 401k plan with matching
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 6+ years of industry experience in software development
  • Experience with large web crawlers a plus
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 4+ years of industry experience in software development
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right