CrawlJobs Logo

Backend Software Engineer - Search, Crawler Team

perplexity.ai Logo

Perplexity

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

We are seeking an experienced Backend Software Engineer to join our Crawler team. In this role, you will design, develop, and operate systems that ingest, process, and manage web-scale data in support of our next generation of advanced search technologies. This is a critical, high-impact engineering position, requiring expertise across both backend and frontend components of our data acquisition stack.

Job Responsibility:

  • Take ownership of and lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems
  • Build, maintain, and optimize core backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management
  • Collaborate closely with Search and Infrastructure teams to ensure the reliable, high-quality ingestion and processing of billions of web pages
  • Architect and implement fullstack features and scalable distributed systems that handle high-load and real-time data operations
  • Rapidly iterate, experiment with novel approaches, and continuously enhance system performance, usability, and reliability

Requirements:

  • Minimum of 5 years of software development experience
  • Strong knowledge of data structures and algorithms in at least one of the following languages: Python, C++, Rust, or Go
  • Experience with large-scale web crawlers is highly desirable
  • Proven experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services
  • Deep understanding of cloud infrastructure, with hands-on experience in Kubernetes (K8s) and AWS
  • Demonstrated passion for writing clean, efficient, and scalable systems

Additional Information:

Job Posted:
February 21, 2026

Employment Type:
Fulltime
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Backend Software Engineer - Search, Crawler Team

Senior Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 6+ years of industry experience in software development
  • Experience with large web crawlers a plus
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Senior Consumer Insights Consultant

Lead end‑to‑end research and transform consumer data into sharp insights that gu...
Location
Location
Sweden , Stockholm
Salary
Salary:
Not provided
hayppgroup.com Logo
Haypp Group
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • At least 5 years’ experience in data analysis and insights creation
  • Experience from hands-on research. Ex. Drafting questionnaires, setting up surveys, conducting interviews, moderating focus groups, analyzing data and visualizing/presenting insights
  • Experience of client management – Relationship building, offer strategic guidance and up-sell
  • Understanding of latest and greatest within market/consumer research
  • Knowledge of both quantitative and qualitative methods and understanding how and when to use them
  • Expert in Excel and Power Point
  • Solid presentation skills
  • Fluency in Swedish & English
Job Responsibility
Job Responsibility
  • Translate research briefs into effective questionnaires, independently designing and scripting surveys that capture key learning objectives and generate actionable insights
  • End-to-end project management: take full responsibility for research projects from start to finish. Understanding client needs, developing proposals and selecting the appropriate methodology, coordinating fieldwork through our in-house data collection tool, analysing results, and delivering the final report
  • Convert data into insights in a relevant and appealing way
  • Client management: you will be fully responsible for building relationships with a set of clients and lead all projects from start to finish
  • Manage relationships with other internal departments: CRM, Customer Service, Category management, after-sales
  • Continuously building and developing the consumer insights offer
  • Support internal departments in their insights needs by conducting specific projects that help the growth of Haypp Group as a business
What we offer
What we offer
  • Premium ITP1 occupational pension savings plan
  • Insurance plan, including private healthcare
  • 30 days of paid vacation
  • Parental pay lift
  • Wellness allowance of 5,000 SEK per year
  • Lots of social happenings
  • Fulltime
Read More
Arrow Right

Principal Data Scientist

You’ll turn ambiguous business problems into durable, repeatable data science ap...
Location
Location
United States , Multiple Locations
Salary
Salary:
139900.00 - 274800.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience
  • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years data-science experience
  • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data science experience
  • OR equivalent experience.
Job Responsibility
Job Responsibility
  • Drives alignment between customer business priorities and data science strategy across complex engagements
  • Frames ambiguous business problems into scalable data science opportunities
  • Defines the data readiness strategy for complex engagements
  • Defines modeling strategies for ambiguous, high-impact business problems
  • Defines evaluation frameworks that connect model performance, business impact, operational health, and responsible AI requirements
  • Serves as a recognized technical and domain leader
  • Provides principal-level technical leadership in code quality, maintainability, production readiness, and debugging practices
  • Partners with customers and Microsoft cross-functional stakeholders to define strategic roadmaps
  • Acts as a trusted advisor to customer and Microsoft stakeholders.
  • Fulltime
Read More
Arrow Right

Registered Nurse

As a Registered Nurse, you play a vital role in delivering the highest standard ...
Location
Location
Australia , Warrawong
Salary
Salary:
47.81 - 60.96 AUD / Hour
warrigal.com.au Logo
Warrigal
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must be able to work Night shift and fill the below roster Week 1 Wednesday- Saturday Week 2 Wednesday- Thursday
  • A current AHPRA registration as a Registered Nurse (preferably in aged care, but not essential)
  • Clinical leadership skills and knowledge
  • Knowledge of legislation that affects aged care services
  • Resourceful, respectful, and passionate, with a love for sharing knowledge and mentoring team members
Job Responsibility
Job Responsibility
  • Delivering the highest standard of clinical care and support to ensure exceptional outcomes for our residents
  • Providing strong clinical leadership, guiding and mentoring the care team while supporting ongoing skill development
  • Being a trusted expert and a role model
  • Providing direction, reassurance, and compassionate, person‑centred care
  • Setting the tone for the entire care environment, ensuring residents receive dignity, respect, and high‑quality support
What we offer
What we offer
  • Penalty rates for nights and weekends
  • NFP salary packaging - pay up to $18,550 less tax per year
  • A fantastic on-boarding program, ongoing support and mentoring from clinical educators
  • A community that values your career and invests in your growth
  • Access to your local gym and 200+ more across the country with a Fitness Passport
  • Parttime
Read More
Arrow Right

Student Worker - Functional

The Student Worker - Functional supports marketing and promotional efforts for t...
Location
Location
United States , Lynchburg
Salary
Salary:
Not provided
aramark.co.uk Logo
Aramark UK
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Interest in marketing, communications, or related fields
  • Strong written and verbal communication skills
  • Ability to manage time effectively and work independently
  • Knowledge of campus culture and student dining preferences
  • Ability to work collaboratively with a diverse team
  • Availability to work flexible hours, including occasional events
Job Responsibility
Job Responsibility
  • Tabling at orientation, events, or meal plan promotions like DB reload
  • Tabling for social media awareness and acquisition
  • Execution of residential events upon request
  • Creates content for social media
  • Takes campus, student, and food photos in the dining locations and uploads to shared folders
  • Responsible for posting on Randolph Dining Instagram
  • Check in with GM and Sr District Marketing Manager weekly to discuss marketing needs and upcoming events
Read More
Arrow Right

Catering Assistant

Join our team in Queanbeyan as a Catering Assistant and help create a warm, welc...
Location
Location
Australia , Queanbeyan
Salary
Salary:
29.49 AUD / Hour
warrigal.com.au Logo
Warrigal
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience working in a commercial kitchen
  • Well‑developed interpersonal and communication skills
  • Demonstrated empathy and commitment to caring for older people
  • Understanding of infection control and safe work practices
  • Strong time‑management and organisational skills
  • Ability to work as part of a multidisciplinary team
  • Ability to follow workplace guidelines, procedures, and policies
Job Responsibility
Job Responsibility
  • Support day‑to‑day kitchen operations including food preparation, meal service, cleanliness, and maintaining safe and hygienic practices
  • Help create a warm, welcoming dining experience for our residents
  • Help create a homely atmosphere and bring a smile to our residents’ day
What we offer
What we offer
  • Superannuation + penalty rates + NFP salary Packaging
  • A fantastic on-boarding program to set you up for success, ongoing support
  • Subsidised access to your local gym and 200+ more across the country with a Fitness Passport
  • 24/7 access to mental, physical, social, and financial support via the LifeWorks app
  • Parttime
Read More
Arrow Right

Software Engineer

The Bing Multimedia team is building the next generation of AI-powered visual di...
Location
Location
United States , Redmond
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • 2+ years of experience building and shipping production software
  • Experience with backend services, distributed systems, or cloud-based applications
  • Ability to design, implement, and maintain scalable and reliable software components
  • Familiarity with AI/ML-enabled product development
  • Experience using data, metrics, or experimentation to drive engineering improvements
  • Experience building search, recommendation, or large-scale content discovery systems
  • Familiarity with LLMs, multimodal AI, or AI-powered product experiences
  • Experience with evaluation, experimentation, or quality metrics for user-facing systems
Job Responsibility
Job Responsibility
  • Build and improve AI-powered image search experiences that better understand user intent and context
  • Develop and ship production features and services across key product and platform areas
  • Apply LLM and multimodal AI capabilities to improve relevance, usefulness, and overall experience quality
  • Help create intelligent, adaptive experiences that support user exploration and decision-making
  • Design and operate scalable systems with sold performance, reliability, and efficiency
  • Use metrics, experiments, and user feedback to measure impact and guide improvements
  • Collaborate across engineering, product, design, and data science, while growing technical ownership and contributing to design and code reviews
  • Fulltime
Read More
Arrow Right

Pt patient services worker

Patient Services Workers are responsible for a variety of specialized duties rel...
Location
Location
United States , Dallas
Salary
Salary:
Not provided
aramark.co.uk Logo
Aramark UK
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must be able to speak, read, and write English
  • Bilingual abilities preferred, but not required
Job Responsibility
Job Responsibility
  • Delivers and collects patient trays during meal service
  • Delivers nourishments and/or snacks to patients as ordered by physician/dietitian in a timely manner
  • Visits patients to introduce the services of the Food and Nutrition Services Department
  • May obtain food preferences/dislikes from patients and/or family members
  • Communicate all patient food needs to the appropriate area of the Food and Nutrition Services Department
  • Ensures that nutrition diet care orders are provided to the patient by offering appropriate menu selections
  • Maintains and adheres to all sanitation standards by following assigned cleaning schedules
  • Completes other sanitation tasks as assigned by the Supervisor/Relief Supervisor
  • Inventories and re-stocks pantries, refrigerator, and freezers on assigned unit(s)
  • Maintains temperature logs for unit refrigerators and freezers
  • Parttime
Read More
Arrow Right