CrawlJobs Logo

Software Engineer, Data Platform

United States; United Kingdom; Netherlands, New York 115000.00 - 200000.00 USD / Year · Job Posted March 21, 2026
Apply Position
Job Link Share

Job Description

Our systems operate at the massive speed and scale of algorithmic trading. We process terabytes of data and billions of rows each day, and every layer of the platform must be engineered for performance and reliability. As a small team, we are continuously evolving our analytics platform to keep pace with growing datasets and increasingly complex analytical workloads. We are seeking a software engineer who enjoys building the systems that make large-scale data usable. This role sits at the intersection of data engineering and backend development, focused on building and improving the infrastructure that powers our analytics platform. That includes designing data pipelines, workflow orchestration, and internal tooling that supports high-volume, time-sensitive data processing. The ideal candidate is comfortable working with both real-time and historical data streams and has experience managing end-to-end data workflows. You will help design and maintain the services, pipelines, and platform infrastructure that support our analytics initiatives, working closely with both engineering and the business to turn complex datasets into usable systems.

Job Responsibility

  • Work with engineers and business teams to understand our datasets and build systems to store, process, and productionize them
  • Develop and maintain reliable data pipelines and workflow systems for large-scale data ingestion and transformation
  • Build internal services and tooling that support analytics, automation, and data operations
  • Implement ETL and data transformation processes for analytical databases and internal applications
  • Develop and maintain workflow orchestration for complex data processes
  • Optimize storage, retrieval, and processing performance across large datasets
  • Partner with analysts and researchers to support data modeling and analytics workflows
  • Monitor and troubleshoot data infrastructure to ensure reliability and data integrity
  • Continuously improve the platform through better system design, tooling, and engineering practices

Requirements

  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent hands-on experience
  • 3-5 years of experience building internal data platform tooling or services that support large-scale data processing and analytics workflows (primarily using Python and SQL
  • Rust is a plus)
  • Experience with workflow orchestration systems used to manage complex data pipelines (Temporal, Airflow, Dagster, or similar)
  • Familiarity with columnar data formats such as Parquet or Arrow
  • Experience working with large-scale datasets and high-volume data pipelines in production environments
  • US citizen/visa only

Nice to have

  • Experience working with analytics platforms, data providers, or financial data systems is a plus
  • Rust is a plus

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Software Engineer, Data Platform

8 matching positions

Software Engineer, Data Platform

Profound is on a mission to help companies understand and control their AI prese...
Location
Location
United States , New York City
Salary
Salary:
140000.00 - 260000.00 USD / Year
tryprofound.com Logo
Profound
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience building platforms for engineers and data scientists in areas such as product analytics and experimentation, data discovery and catalog, data quality and observability, or governance and access control
  • Experience designing and operating large-scale distributed data systems
  • Expertise with analytics and lakehouse technologies including ClickHouse, Tinybird, Snowflake, or Iceberg
  • Strong cloud experience with AWS, GCP, or Azure
  • Experience with data governance, reliability, and secure data operations
  • Excellent communication skills with the ability to influence across engineering and product teams
  • Experience with or interest in MLOps and supporting machine learning workflows in production
  • Strong problem-solving skills and comfort operating in a fast-paced, impact-driven environment
  • Motivated by access to rare and interesting AI data based on real human interactions with large language models
  • Excited by the opportunity to own and shape Profound’s data platform at an early stage
Job Responsibility
Job Responsibility
  • Design and build platform infrastructure that supports machine learning, analytics, and software engineers, as well as data scientists across the company
  • Develop tools and frameworks for data ingestion, transformation, quality, and observability
  • Architect scalable, reliable systems using Snowflake, Clickhouse, AWS, dbt, and Dagster
  • Build governance, access control, and data quality systems that scale with the organization
  • Enable MLOps by creating infrastructure that supports model development and productionization
  • Drive technical decisions around platform architecture, tool selection, and data patterns
  • Collaborate with product, engineering, and data teams to understand needs and deliver impactful solutions
What we offer
What we offer
  • equity
  • a full range of benefits and perks
  • Fulltime
Read More
Arrow Right

Software engineer 2 / Senior Software engineer - Azure Data

Microsoft's Azure Data engineering team is leading the transformation of analyti...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 3+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • Experience with the Azure stack including Storage, Compute, Networking, Fabric, Purview, Synapse, AKS, DevOps, Data Factory, or Power BI
  • Experience with big data technologies such as Spark, Kafka, Hadoop, or HBase
  • Experience building data lake or data engineering products, tools, or pipelines
  • Familiarity with container-based architectures (Docker, Kubernetes)
  • Ability to debug complex distributed systems on Linux and/or Windows platforms
Job Responsibility
Job Responsibility
  • Write extensible, maintainable code in C#, Java, Scala, or Python for Fabric Materialized Lake View services and HDInsight components
  • Use AI tools and coding best practices across the development lifecycle
  • Design data refresh, scheduling, and query optimisation features with minimal supervision
  • Review code from teammates for correctness, test coverage, security risks, and adherence to team standards
  • Coach junior engineers through code reviews
  • Debug complex issues in distributed systems running on Azure, Linux, and Windows
  • Run live site operations on a rotational, on-call basis
  • Integrate logging and instrumentation to gather telemetry on system health, performance, reliability, and security
  • Work with product managers, technical leads, and partners across geographies to define customer requirements for Materialized Lake View features
  • Fulltime
Read More
Arrow Right

Software Engineer | Data Platform & Self-Serve Enablement | Inter IKEA

We are on the journey to transform our digital capabilities, bringing core busin...
Location
Location
Sweden , Malmö
Salary
Salary:
Not provided
https://www.ikea.com Logo
IKEA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong experience in software engineering (Python, JS/TS, Go, or similar)
  • Experience working with data platforms or data-intensive systems
  • Hands-on experience with Databricks or similar distributed data frameworks (e.g., Spark)
  • Experience with cloud platforms (Azure and/or AWS)
  • Familiarity with DevOps practices, including CI/CD pipelines, infrastructure-as-code, and automation
  • Experience designing APIs, services, or reusable platform components
Job Responsibility
Job Responsibility
  • Build and evolve our data platform capabilities to support self-service analytics and data product thinking
  • Design and develop platform services, tooling, and frameworks that enable data producers and consumers
  • Work with Databricks as a core component of our modern data platform
  • Contribute to our knowledge graph / metadata layer, improving discoverability, lineage, and data understanding
  • Implement and promote DevOps practices (CI/CD, automation, testing, IaC) for data and platform engineering
  • Develop solutions across multi-cloud environments (Azure & AWS)
  • Collaborate with data engineers, analysts, and domain teams to remove friction and improve developer experience
  • Drive standardization, scalability, and best practices across the platform
Read More
Arrow Right

Senior Software Engineer - Data Platform, AI Infrastructure

We are building a large-scale, productized data platform that powers critical in...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
  • Strong programming experience in Python
  • Experience building and operating large-scale distributed systems
  • Hands-on experience with: Backend services or APIs (e.g., FastAPI, Flask, or similar)
  • Cloud-based infrastructure (Azure, AWS, or GCP)
  • Monitoring and observability systems (metrics, logging, alerting)
  • Experience designing systems with reliability, scalability, and operational clarity in mind
  • Proven ability to own and deliver production systems end-to-end
  • Ability to break down ambiguous problems, ask the right questions, and execute effectively
Job Responsibility
Job Responsibility
  • Design, build, and operate core components of a distributed data platform, including: Orchestration systems (e.g., Airflow or equivalent)
  • Backend services and APIs (Python/FastAPI or similar)
  • Monitoring, alerting, and reliability systems
  • Own the end-to-end lifecycle of platform components - from design through deployment, scaling, and maintenance
  • Ensure systems meet requirements for availability, performance, and data reliability at large scale
  • Define and enforce standardized patterns for infrastructure, deployment, and observability across the platform
  • Partner with data engineering teams to enable efficient, reliable data processing workflows
  • Diagnose and resolve complex issues in distributed systems, including performance bottlenecks and failure modes
  • Contribute to infrastructure-as-code and deployment systems to support reproducibility and operational excellence
  • Drive continuous improvements in system robustness, cost efficiency, and operational clarity
What we offer
What we offer
  • Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
  • Fulltime
Read More
Arrow Right

Senior Software Engineer, Data Platform

We’re looking for a Senior Data Platform Engineer to help design and scale the f...
Location
Location
United States , Los Angeles
Salary
Salary:
180000.00 - 220000.00 USD / Year
geniussports.com Logo
Genius Sports
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A proven track record, with 5+ years of hands-on experience in architecting distributed system architecture, guiding projects from initial ideation to successful production deployment
  • A product-minded approach—you understand how our software impacts user experience and business outcomes
  • Demonstrated mentorship and leadership experience, growing engineers and fostering a culture of technical excellence
  • Excellent collaboration and communication skills, with the ability to influence across teams
  • Broad expertise across the data systems engineering landscape, with deep experience in one or more of the following areas: Hands-on experience designing and operating modern data lakehouse systems (Iceberg, BigQuery, Clickhouse, etc.)
  • Expertise in distributed data processing and streaming architectures, including Spark, Pulsar, Kafka, and Flink, with a strong understanding of batch vs. real-time trade-offs
  • Experience working with large-scale datasets, including performance tuning, cost optimization, and observability in production environments
  • Solid understanding of data modeling, schema evolution, and data quality practices in distributed data platforms
Job Responsibility
Job Responsibility
  • Design, build, and maintain the Data Platform's foundational systems and features that enable internal teams to discover, access, and act on data quickly, reliably, and securely
  • Drive the modernization of our data infrastructure, navigating technical complexity and making sound architectural tradeoffs to strengthen our foundation and increase organizational productivity
  • Serve as a technical leader for large-scale Data Platform initiatives, driving architectural decisions and shaping engineering direction in close partnership with product and cross-functional stakeholders
  • Provide technical mentorship to junior and mid-level engineers, upskilling the team and preparing them to take on greater responsibilities
  • Stay up-to-date with emerging technologies and industry trends in data systems and platform development
What we offer
What we offer
  • Competitive salary
  • Range of benefits
  • Support for employee wellbeing
  • Help you grow your skills, experience and career
  • Fulltime
Read More
Arrow Right

Software Engineer - Data Platform

As a foundational software engineer on our data platform engineering team, you w...
Location
Location
United States , Los Angeles
Salary
Salary:
120000.00 - 200000.00 USD / Year
hadrian.co Logo
Hadrian Automation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Extensive experience shipping modern, data-centric applications (data systems use Argo-Workflows, Dagster, Superset, Aurora, RDS, S3, back-ends are Go and Python, with gRPC/Avro and Kafka as messaging platform)
  • Experience with IaC and GitOps tooling (use Terraform extensively, centralized on Kubernetes/Argo/Helm)
  • Extremely well versed with data querying techniques across NoSQL and SQL platforms
  • Bachelor's degree in Computer Science and/or equivalent experience
  • Solid understanding in building data architecture and pipelines
  • Are self-motivated and eager to get hands-on and tackle challenges independently while working collaboratively toward identified objectives
  • Work with a platform mentality -- driven to find the right architecture and plan up front and solve problems with the long term in mind
  • Take responsibility and ownership finding solutions no matter what
  • Deploy broad experience and big picture view to fix undreamed of problems with innovative solutions
  • Feel passionate about making things move in the real world with software
Job Responsibility
Job Responsibility
  • Scope, architect, implement, and deploy critical applications that will drive revenue and make a positive impact in the world
  • Build and manage a robust data warehouse and write software to coordinate and deploy data pipelines
  • Conceptualize and own the data architecture for multiple large-scale projects
  • Create and contribute to data frameworks that span on-premises and cloud infrastructure improve the efficacy of logging machine data, while working with data infrastructure to triage issues and resolve
  • Solve our most challenging machine data integration problems, utilizing optimal ETL patterns, frameworks, query techniques, sourcing from structured and unstructured data sources
  • Collaborate with machine engineers, product managers, and data scientists to understand data needs, representing key data insights visually in a meaningful way
  • Get to build alongside an incredible team of software engineers, mechanical engineers, operators, and the best machinists/CAM programmers in the world.
What we offer
What we offer
  • Medical, dental, vision, and life insurance plans for employees
  • 401k
  • Relocation support may be provided for certain situations, based on business need
  • Flexible vacation policy
  • Fulltime
Read More
Arrow Right

Senior Software Engineer - Data Platform

Help build the future of data by creating the technical "nervous system" of the ...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
multiverse.io Logo
Multiverse
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Significant experience in commercial software (Python, TypeScript, and Go)
  • Passion for modular, readable code
  • Understanding of how to treat "data as software"
  • Comfortable with cloud-native tools (AWS/Azure, Kubernetes)
  • Ability to explain complex architectural choices to both product managers and engineers
Job Responsibility
Job Responsibility
  • Architect for growth: define and refactor data models to ensure they stay fast and clear as we scale
  • Create a universal data layer: build GraphQL APIs and connectors so anyone can access data safely
  • Enable GenAI: build infrastructure for Vector Databases and AI-driven automation
  • Automate safety: implement "Privacy by Design" by automating security checks
What we offer
What we offer
  • 27 days holiday
  • 5 additional days off: 1 life event day, 2 volunteer days, 2 company-wide wellbeing days
  • 8 bank holidays per year
  • Private medical Insurance with Bupa
  • Medical cashback scheme
  • Life insurance
  • Gym membership & wellness resources through Wellhub
  • Access to Spill - mental health support
  • Work-from-anywhere scheme - up to 10 days per year
  • Kitchen that's always stocked
  • Fulltime
Read More
Arrow Right

Senior/Staff Software Engineer - Data Platform

Perplexity is looking for experienced Data Platform Engineers to design, build, ...
Location
Location
United States , San Francisco, Seattle, New York City
Salary
Salary:
250000.00 - 385000.00 USD / Year
perplexity.ai Logo
Perplexity
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years (Senior) or 8+ years (Staff) of software engineering experience
  • Strong experience building production data infrastructure systems
  • Hands-on experience with batch and/or streaming data processing at scale
  • Deep familiarity with data orchestration systems (Airflow, Dagster, or similar)
  • Proficiency in Python and at least one additional backend language (Go, TypeScript, etc.)
  • Strong systems thinking: you understand tradeoffs across reliability, latency, cost, and complexity
  • Experience supporting ML/AI workflows, training pipelines, or evaluation systems
  • Familiarity with data quality, lineage, observability, and governance tooling
  • Prior ownership of internal platforms used by many teams
Job Responsibility
Job Responsibility
  • Design and operate large-scale batch and streaming data pipelines supporting product features, AI training/evaluation, analytics, and experimentation
  • Build and evolve event-driven and streaming systems (e.g., Kafka/Kinesis/PubSub-style architectures) for real-time ingestion, transformation, and delivery
  • Own batch processing frameworks for backfills, aggregations, and offline computation
  • Lead the design and operation of data orchestration systems (e.g., Airflow, Dagster, or equivalent), including scheduling, dependency management, retries, SLAs, and observability
  • Establish strong guarantees around data correctness, freshness, lineage, and recoverability
  • Design systems that handle scale, partial failure, and evolving schemas
  • Build self-serve data platforms that empower engineers, data scientists, and analysts to safely create and operate pipelines
  • Improve developer experience for data work through better abstractions, tooling, documentation, and paved paths
  • Set standards for data modeling, testing, validation, and deployment
  • Drive architectural decisions across data infrastructure for storage, compute, orchestration, and APIs
What we offer
What we offer
  • Equity
  • Health
  • Dental
  • Vision
  • Retirement
  • Fitness
  • Commuter and dependent care accounts
  • Fulltime
Read More
Arrow Right