CrawlJobs Logo

Backend Software Engineer - Search, Crawler Team

perplexity.ai Logo

Perplexity

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

We are seeking an experienced Backend Software Engineer to join our Crawler team. In this role, you will design, develop, and operate systems that ingest, process, and manage web-scale data in support of our next generation of advanced search technologies. This is a critical, high-impact engineering position, requiring expertise across both backend and frontend components of our data acquisition stack.

Job Responsibility:

  • Take ownership of and lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems
  • Build, maintain, and optimize core backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management
  • Collaborate closely with Search and Infrastructure teams to ensure the reliable, high-quality ingestion and processing of billions of web pages
  • Architect and implement fullstack features and scalable distributed systems that handle high-load and real-time data operations
  • Rapidly iterate, experiment with novel approaches, and continuously enhance system performance, usability, and reliability

Requirements:

  • Minimum of 5 years of software development experience
  • Strong knowledge of data structures and algorithms in at least one of the following languages: Python, C++, Rust, or Go
  • Experience with large-scale web crawlers is highly desirable
  • Proven experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services
  • Deep understanding of cloud infrastructure, with hands-on experience in Kubernetes (K8s) and AWS
  • Demonstrated passion for writing clean, efficient, and scalable systems

Additional Information:

Job Posted:
February 21, 2026

Employment Type:
Fulltime
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Backend Software Engineer - Search, Crawler Team

Senior Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 6+ years of industry experience in software development
  • Experience with large web crawlers a plus
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right
New

Field Layout Technician

The Field Technician II is a mid-level technical role responsible for conducting...
Location
Location
United States , Phoenix
Salary
Salary:
Not provided
cecoconcrete.com Logo
Ceco Concrete Construction, LLC
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum of one year of experience in field engineering/lay-out
  • Good communication skills – written, verbal, interpersonal
  • Extensive travel up to 70%
  • Proficient at reading and understanding contract drawing, including but not limited to both structural and architectural drawings, with the ability to cross check between the two documents as needed
  • Ability to meet deadlines and multi-task in fast paced environment
  • Bilingual Spanish/English
  • Possess knowledge of construction field engineering from pre-construction to completion
Job Responsibility
Job Responsibility
  • Review project documents
  • Understand the objectives of the site survey and provides horizontal and vertical control using Total Stations
  • Participate in basic layout and field engineering activities
  • Establish control lines from control points
  • Layout Foundations
  • Layout vertical formwork such as columns and walls
  • Establish elevation control points
  • Layout decks and deck edges
  • Layout required embeds block-outs and other items required by contract
  • Assist in the performance of a variety of surveying activities
What we offer
What we offer
  • Inclusive Medical, Dental, Vision, Accident, and Illness insurance
  • Company paid AD&D and Life insurance
  • Health Savings Account contribution of up to $1,000 per year
  • 401(k) retirement savings program with a company match
  • Employee Assistance Program including discounts with major vendors & products
  • Mental and physical wellness programs
  • Career advancement opportunities with a stable well-established organization
  • Fulltime
Read More
Arrow Right
New

Receptionist

We are looking for a motivated and detail-oriented Receptionist to join a nonpro...
Location
Location
United States , Santa Barbara
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proficient in handling switchboards for phone lines
  • Advanced knowledge of Excel and related software
  • Strong organizational skills with high attention to detail
  • Ability to multitask and manage priorities effectively
  • Comfortable working in a nonprofit environment and engaging with its clients
  • Strong communication and interpersonal skills
  • A team-oriented mindset with a collaborative approach
  • Tech-savvy and adaptable to new tools and systems
Job Responsibility
Job Responsibility
  • Welcome and assist visitors, ensuring they have a positive experience
  • Respond to inquiries and provide exceptional customer service
  • Handle data entry tasks with accuracy and efficiency
  • Manage incoming mail, review its contents, and distribute it appropriately
  • Maintain office files and oversee general filing support
  • Order and restock office and kitchen supplies as needed
  • Support administrative staff with overflow tasks, including word processing, data entry, and online research
  • Demonstrate a strong sense of urgency when prioritizing tasks
  • Communicate effectively with team members and external stakeholders
  • Uphold excellent spelling, grammar, and social skills in all interactions
What we offer
What we offer
  • medical, vision, dental, and life and disability insurance
  • eligible to enroll in our company 401(k) plan
Read More
Arrow Right
New

Private Duty Nurse LPN

Join a Company That Puts People First! We are one of the largest private duty nu...
Location
Location
United States , Lumberton
Salary
Salary:
28.00 - 32.00 USD / Hour
aveanna.com Logo
Aveanna Healthcare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must have and maintain an active, unencumbered license (LPN/LVN) in the state in which the clinician will practice
  • Compact licenses must be transferred to your state of residence within 90 days
  • Current CPR certification (with hands-on component)
  • TB skin test (current within last 12 months)
  • Six months prior hands-on nursing experience preferred but not required
  • Must have reliable transportation
What we offer
What we offer
  • Health, Dental, Vision and Company-Paid Life Insurance
  • Paid Time Off Available
  • Flexible scheduling
  • 24/7 Local support from operators and clinicians
  • Tablet in each patient’s home allowing for electronic documentation
  • Career Pathing with opportunities for skill advancement
  • Weekly and/or Daily Pay
  • Employee Stock Purchase Plan with 15% discount
  • Employee Relief Fund
  • Thorough training
Read More
Arrow Right
New

Nursery practitioner

Join Our Team as a Nursery Practitioner! Are you passionate about creating brigh...
Location
Location
United Kingdom , Berkhamsted
Salary
Salary:
26436.00 GBP / Year
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • No qualifications required for this role
  • Previous experience working in a childcare setting would be beneficial
  • Must have right to work in the UK (unable to offer sponsorship)
  • Subject to enhanced DBS checks, satisfactory references, and verification of right to work in the UK and any essential qualifications
Job Responsibility
Job Responsibility
  • Create a warm, fun, and nurturing environment where children feel safe and thrive
  • Plan and deliver engaging, play-based activities aligned with the EYFS framework
  • Build meaningful relationships with children, families, and colleagues
  • Monitor and celebrate children’s development milestones, keeping records to share with parents
  • Support the day-to-day running of the nursery, maintaining high standards of hygiene, safety, and care
  • Be a team player, offering guidance to apprentices and volunteers while learning from others
What we offer
What we offer
  • Free childcare for the first month and 52% discount (based on working hours)
  • 31 days of holiday (inclusive of national bank holidays and the Christmas closure week)
  • Birthday off
  • Access to The National College full of training and courses
  • Perkbox discounts, wellbeing tools, and online medical services
  • Refer a friend or family for exciting bonuses
  • Long-service awards
  • 1 paid day off per year to volunteer
  • Fulltime
Read More
Arrow Right
New

Databricks Lead Engineer

An exciting opportunity has arisen for an experienced Databricks Lead Engineer t...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
harringtonstarr.com Logo
Harrington Starr
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong commercial experience with Databricks
  • Expertise in Delta Lake, Delta Tables, and Unity Catalog
  • Experience with Delta UniForm (Universal Format)
  • Hands-on experience with Apache Iceberg
  • Strong AWS experience (including S3 and Lake Formation)
  • Linux-based development experience
  • Familiarity with modern engineering practices (CI/CD, testing, version control)
  • Ability to work autonomously and drive delivery
  • Strong problem-solving and communication skills
Job Responsibility
Job Responsibility
  • Designing and implementing a scalable Delta Lake architecture
  • Modelling and partitioning high-cardinality datasets (e.g. order book data)
  • Implementing metadata management, compaction, and versioning strategies
  • Supporting migration from legacy data platforms to Delta Tables and Iceberg-compatible structures
  • Designing systems to support multiple downstream data delivery use cases
  • Establishing robust backup and recovery strategies (targeting 1-day RTO)
  • Acting as a technical lead while remaining hands-on in engineering delivery
What we offer
What we offer
  • Work on a high-impact, large-scale data transformation project
  • Opportunity to shape architecture decisions in a complex data environment
  • Hybrid working with a Central London base
  • Competitive day rate outside IR35
Read More
Arrow Right
New

Private Duty Nurse LPN

Join a Company That Puts People First! Licensed Practical / Vocational Nurse – L...
Location
Location
United States , Raeford
Salary
Salary:
28.00 - 32.00 USD / Hour
aveanna.com Logo
Aveanna Healthcare
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Must have and maintain an active, unencumbered license (LPN/LVN) in the state in which the clinician will practice
  • Compact licenses must be transferred to your state of residence within 90 days
  • Current CPR certification (with hands-on component)
  • TB skin test (current within last 12 months)
  • Six months prior hands-on nursing experience preferred but not required
  • Must have reliable transportation
What we offer
What we offer
  • Health, Dental, Vision and Company-Paid Life Insurance
  • Paid Time Off Available
  • Flexible scheduling
  • 24/7 Local support from operators and clinicians
  • Tablet in each patient’s home allowing for electronic documentation
  • Career Pathing with opportunities for skill advancement
  • Weekly and/or Daily Pay
  • Employee Stock Purchase Plan with 15% discount
  • Employee Relief Fund
  • Thorough training
Read More
Arrow Right
New

Director, Total Rewards Systems & Transformation

The Director, Total Rewards Systems & Transformation leads the strategy, design,...
Location
Location
United Kingdom , London
Salary
Salary:
Not provided
https://www.marriott.com Logo
Marriott Bonvoy
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong background in Total Rewards, Compensation & Benefits, or HR Transformation
  • Proven experience leading large-scale HR or rewards technology implementations
  • Deep knowledge of reward structures, salary ranges, bonus programs, and mobility frameworks
  • Demonstrated ability to partner with cross-functional teams and influence senior stakeholders
  • Advanced project management capabilities with the ability to influence at multiple levels
  • Strong technical acumen within HRIS and rewards systems
  • Excellent stakeholder management, communication, and presentation skills
  • High level of accuracy, attention to detail, and operational discipline
  • Ability to work effectively in a global environment and manage competing priorities
Job Responsibility
Job Responsibility
  • Lead continent-wide Total Rewards systems and transformation initiatives, driving innovation and digital enablement
  • Oversee systems, tools, and platforms supporting compensation and benefits administration, governance, and reporting
  • Manage complex cross-functional projects, ensuring milestones, deliverables, and risk mitigation are aligned and executed effectively
  • Partner with HR, Finance, and business leaders to modernize rewards processes and enhance operational efficiencies
  • Translate business needs into systems requirements and collaborate with technology partners to deliver scalable solutions
  • Ensure integrity of Total Rewards data, workflows, and system configurations
  • Conduct benchmarking and analysis to inform competitive compensation and benefits architecture
  • Lead change management and communication efforts for rewards system enhancements and transformation projects
  • Fulltime
Read More
Arrow Right