CrawlJobs Logo

Software Engineer – Web Crawling

woflow.com Logo

Woflow

Location Icon

Location:

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided

Job Description:

Woflow is a technology startup creating products and solutions to support a high-growth, on-demand economy. Our flagship product is an end-to-end platform that allows our customers to request and receive merchant data (think structured menu data, images, store information, etc) through a combination of web applications and public APIs. Behind the scenes, that data is created through a series of ML/AI models and workflow products along with a fully automated distributed workforce management platform. We are the world’s first Merchant Data Platform. Our customers include food delivery companies, online ordering platforms, and ecommerce marketplaces. We provide the data infrastructure to help these companies scale and grow. We are looking for a Software Engineer with at least 3 years of experience to join our Web Crawling team. This team plays a critical role in our infrastructure, ensuring high-quality data collection at scale. The right candidate will work closely with both the Web Crawling and Application teams but will primarily focus on writing and optimizing web crawlers. Our ideal candidate is someone who enjoys solving complex web scraping challenges, has strong reverse engineering skills, and thrives in a fast-paced, high-growth environment. This role is a great opportunity for an engineer looking to make a significant impact at a company where data is the product.

Job Responsibility:

  • Develop, enhance, and maintain web crawlers and scraping infrastructure
  • Optimize scraping techniques to handle anti-bot mechanisms, performance, and security challenges
  • Collaborate with a geographically distributed team to identify and resolve issues
  • Ensure high availability, efficiency, and reliability of crawling operations
  • Integrate AI solutions to enhance automation and data extraction accuracy

Requirements:

  • 3+ years of experience in software engineering with a focus on web crawling and data extraction
  • Strong expertise in Node.js (preferred) for web crawling applications
  • Deep understanding of HTML, JavaScript, and reverse engineering techniques
  • Hands-on experience with Playwright, Puppeteer, and Cheerio for automation and scraping
  • Knowledge of security and performance best practices related to web crawling

Nice to have:

  • Experience with Apify or Crawlee for large-scale crawling solutions
  • Proficiency in TypeScript
What we offer:
  • Unlimited PTO
  • Comprehensive medical, dental, and vision insurance plans
  • STD, LTD, AD&D, and life insurance coverage
  • Free membership to TalkSpace, Teladoc and Health Advocate
  • Free annual membership to One Medical in participating regions
  • 401(k) retirement plan with company matching
  • Pre-tax commuter benefits
  • Free equipment: laptop and home office stipend

Additional Information:

Job Posted:
February 21, 2026

Employment Type:
Fulltime
Work Type:
Remote work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Software Engineer – Web Crawling

Senior Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 6+ years of industry experience in software development
  • Experience with large web crawlers a plus
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Software Engineer, Data Acquisition

The Data Acquisition team within the Foundations organization at OpenAI is respo...
Location
Location
United States , San Francisco
Salary
Salary:
293000.00 - 385000.00 USD / Year
openai.com Logo
OpenAI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS/MS/PhD in Computer Science or a related field
  • 4+ years of industry experience in software development
  • Strong expertise in large stateful distributed systems and data processing
  • Proficiency in Kubernetes, and Infrastructure-as-Code concepts
  • Willingness and enthusiasm for trying new approaches and technologies
  • Ability to handle multiple tasks and adapt to changing priorities
  • Strong communication skills, both written and verbal
Job Responsibility
Job Responsibility
  • Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search
  • Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability
  • Work closely with the legal team to handle any compliance or data privacy-related matters
  • Develop and deploy highly scalable distributed systems capable of handling petabytes of data
  • Architect and implement algorithms for data indexing and search capabilities
  • Build and maintain backend services for data storage, including work with key-value databases and synchronization
  • Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks
  • Conduct and analyze experiments on data to provide insights into system performance
What we offer
What we offer
  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time (1 hour per 30 hours worked, or more, as required by applicable state or local law)
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Fulltime
Read More
Arrow Right

Tech Lead - Full Stack

We are looking for an experienced and visionary Tech Lead to lead our engineerin...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
rubick.ai Logo
Rubick AI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field
  • additional business or management qualifications are a plus
  • Proven experience (3-6 years) in a senior engineering leadership role, overseeing software development, with a successful track record of delivering complex projects
  • Strong technical background with expertise in software development methodologies, architecture, and tools
  • Experience in MERN stack
  • Strong experience in Python, Web Crawling, and infrastructure (Scrapy, Redis, Selenium)
  • Excellent leadership and people management skills with the ability to inspire and motivate teams, with 2+ years in a managerial position overseeing engineering teams
  • Expertise in large-scale system architecture, event-driven microservices, well-crafted APIs, and optimized algorithms
  • Strategic thinker with a problem-solving mindset and the ability to make sound decisions in a fast-paced environment
  • Effective communication skills with the ability to articulate technical concepts to non-technical stakeholders
Job Responsibility
Job Responsibility
  • Collaborate with the Engineering team to develop and refine processes for task prioritization, breakdown, estimation, and delegation
  • Mentor Engineers in dissecting challenges into actionable tasks and overseeing their execution
  • Develop and execute engineering plans, strategies, and roadmaps aligned with the company's goals and objectives
  • Drive the design, development, and implementation of scalable, high-quality software and hardware solutions
  • Collaborate with cross-functional teams including product management, design, and operations to deliver innovative and efficient products
  • Foster a culture of continuous improvement, innovation, and excellence in engineering practices and processes
  • Spearhead the strategy for quality assurance and testing within the team, ensuring high standards
  • Stay updated on industry trends, technological advancements, and best practices to drive innovation and maintain a competitive edge
What we offer
What we offer
  • Recharge with 24 Annual Paid Leaves — plenty of time to relax, travel, and refresh
  • Comprehensive Health Coverage — insurance worth ₹5,00,000 for you and your family’s peace of mind
  • Up to 6 Months of Maternity Leave — supporting you through life’s most special moments
  • Delicious Office Lunch — fuel your day with meals provided at the office
  • Performance-Based Incentives — your hard work earns the rewards it deserves
  • Fulltime
Read More
Arrow Right

Applied Scientist II, Selection Monitoring

Selection Monitoring team is responsible for making the biggest catalog on the p...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
amazon.de Logo
Amazon Pforzheim GmbH
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years of building models for business application experience
  • PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
  • Experience in patents or publications at top-tier peer-reviewed conferences or journals
  • Experience programming in Java, C++, Python or related language
  • Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing
Job Responsibility
Job Responsibility
  • Use AI, NLP and advances in LLMs/SLMs and agentic systems to create scalable solutions for business problems
  • Efficiently Crawl web, Automate extraction of relevant information from large amounts of Visually Rich Documents and optimize key processes
  • Design, develop, evaluate and deploy, innovative and highly scalable ML models, esp. leveraging latest advances in RL-based fine tuning methods like DPO, GRPO etc
  • Work closely with software engineering teams to drive real-time model implementations
  • Establish scalable, efficient, automated processes for large scale model development, model validation and model maintenance
  • Lead projects and mentor other scientists, engineers in the use of ML techniques
  • Publish innovation in research forums
Read More
Arrow Right

Research Grants AI Internship

Join our AI team as a Research Grants AI Intern and contribute to building intel...
Location
Location
Switzerland , Basel
Salary
Salary:
Not provided
mdpi.com Logo
MDPI
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Currently pursuing or recently completed a BSc or MSc in Data Science, Machine Learning, Software Engineering, or a related field
  • Initial practical experience in data acquisition, data processing, AI, or related areas
  • Basic knowledge of web scraping, API integration, and Python for data-intensive applications
  • Understanding of AI principles and interest in NLP or text mining
  • Familiarity with databases and data modeling concepts
  • Fluent in English (minimum B2)
  • Strong analytical and problem-solving skills
  • Ability to explain technical concepts to non-technical stakeholders
  • Comfortable working both independently and in an agile team environment
  • Curiosity about academic publishing and research ecosystems
Job Responsibility
Job Responsibility
  • Build workflows for crawling, extracting, and indexing grant information from public sources
  • Normalize and structure grant metadata (e.g., funding agency, grant IDs, investigators, timelines, keywords)
  • Develop and implement text-mining and metadata-matching methods to link grants with publications
  • Integrate grant data into internal databases and graph-based systems
  • Support the exploration of predictive models to identify research trends and funding opportunities
  • Contribute to potential integrations with internal analytics and editorial tools
What we offer
What we offer
  • The opportunity to contribute to the academic/scientific community
  • Flexible working hours
  • Team bond strengthening through team-building events
  • Professional growth opportunities with our global training system
  • Working in a collaborative and socially responsible team
  • Company retreat facility
  • Full-coverage insurance for accidents/daily sickness
  • Prime location near Basel train station and city center
  • Fulltime
Read More
Arrow Right

SEO Specialist

As an SEO Specialist, you will be responsible for maintaining and executing the ...
Location
Location
Philippines , Manila
Salary
Salary:
Not provided
staffvirtual.com Logo
STAFFVIRTUAL
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience in SEO with a focus on strategy and execution (2 to 4 years experience)
  • Strong understanding of search engine algorithms and ranking factors
  • Familiarity with SEO tools such as Ahrefs, Moz, SEMrush, etc.
  • Excellent analytical and problem-solving skills
  • Effective communication and collaboration skills
  • Strong written communication skills with attention to grammar and clarity
  • Detail-oriented with the ability to manage multiple tasks and deadlines
Job Responsibility
Job Responsibility
  • Local Search Optimization
  • Optimize and manage Google Business Profiles (GBP) for multiple locations
  • Improve local map pack rankings through strategic updates, category optimization, and posting
  • Keyword Research and Analysis
  • Conduct thorough keyword research to identify relevant search terms
  • Analyze keyword competitiveness and trends to inform content strategies
  • Perform audits and make strategic recommendations based on experience with the client
  • On-Page & Technical Local SEO
  • Collaborate with Content, Design, and Web Dev departments to optimize on-page elements such as meta tags, headers, content, and user experience
  • Implement structured data for local businesses, services, and FAQs
What we offer
What we offer
  • Competitive compensation and benefits package
  • HMO coverage starting on Day 1
  • Paid time offs
  • Training, career growth, and global exposure
  • A collaborative and supportive team culture
  • Fulltime
Read More
Arrow Right
New

National Sales Support

Are you detail-oriented and customer-focused? Help us power national sales succe...
Location
Location
United Kingdom , Watford
Salary
Salary:
Not provided
majestic.co.uk Logo
Majestic Wine Warehouses Ltd.
Expiration Date
April 14, 2026
Flip Icon
Requirements
Requirements
  • Excellent communication skills with the ability to build relationships and influence stakeholders at all levels of the business
  • High proficiency in Excel (VBA preferred) and the ability to handle complex data and reporting
  • Strong numerical and analytical skills with a proactive approach to solving problems
  • Exceptional attention to detail and a strong sense of accountability for your work
  • Ability to stay organised and deliver high-quality results in a fast-paced, changing environment
  • A dedicated team player who supports national sales goals and provides top-tier customer care
  • A fast learner who is proactive and comfortable taking initiative without constant supervision
Job Responsibility
Job Responsibility
  • Process orders accurately and within agreed deadlines
  • Work alongside the operations team in Hemel, the stores, CCC and supply chain
  • Develop strong relationships with customers, delivering exceptional service over telephone/ email
  • Effectively manage new business leads
  • Work alongside NSMs to ensure optimum retention rates and actively grow existing account base to help achieve sales growth and increased profitability
  • Key operational contact for on demand sales channel including Deliverect, Deliveroo and Uber Eats
  • Attend existing business meetings and events, work with the NSMs to deliver solutions to improve profitability
  • Create monthly bespoke statements and work with the credit control team and the NSMs to manage the debts (includes reviewing credit terms if needed)
  • Be the known office based point of contact for our customer base
  • Communicate accurately and efficiently answering all customer queries in a timely manner
What we offer
What we offer
  • A competitive salary & performance related bonus
  • Up to 20% Staff Discount
  • 29 days holiday, including public and bank holidays and we also have a Holiday Purchase Scheme
  • Life Assurance (Worth x2 times your annual salary)
  • A contributory Company Pension Plan
  • Career development opportunities
  • Fantastic incentives that take you around the world to explore our different vineyards
  • Free parking on site with electric charging points
  • Access to Retail Trust which includes: Retail Rewards including Instant savings with discounted e-vouchers, discounted reloadable shopping cards, gift vouchers and gift cards, Discounts of up to 30%, Access to free counselling and support phone line
  • Company maternity, paternity and adoption leave after 26 weeks
  • Fulltime
!
Read More
Arrow Right
New

Resort Safety Officer

Join our team as a Resort Safety Officer at South Seas! Imagine working where th...
Location
Location
United States , Captiva
Salary
Salary:
Not provided
southseas.com Logo
South Seas
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • High School diploma or equivalent and/or experience in a hotel or a related field preferred
  • Certificate of Completion in Safety and Security Training preferred
  • CPR/First Aid/AED/Heimlich Maneuver Certified
  • Must have a valid driver license for the applicable State and an acceptable MVR (Motor Vehicle Driving record)
Job Responsibility
Job Responsibility
  • Ensure resort safety by conducting regular patrols across the entire property
  • Maintain a strong, visible presence by patrolling all resort amenities consistently throughout the shift
  • Provide Pool Gate coverage to control access, check for prohibited items
  • Staff the Front Entrance Gate, issuing parking passes, verifying access credentials
  • Investigate and document incidents such as theft, auto accidents, guest complaints, and policy violations
  • Respond to medical emergencies and administer First Aid, CPR, and AED as needed
  • Monitor security systems and alarms
  • Maintain detailed and accurate daily activity logs and incident reports
  • Support the Front Desk with guest services
  • Uphold confidentiality when handling guest or employee incidents
What we offer
What we offer
  • Low-cost Medical, Dental, Vision Plans
  • Paid Life Insurance
  • Short- and Long-Term Disability
  • Paid Time Off & Holidays
  • 401(k) with 100% match up to 4 percent
  • Commuter and Company-paid Toll Programs
  • Unlimited Review Incentives
  • Free Daily Lunches
  • Fulltime
Read More
Arrow Right