CrawlJobs Logo

Senior Data & Automation Engineer

Canada, Toronto 90000.00 - 100000.00 CAD / Year · Job Posted January 20, 2026
Apply Position
Job Link Share

Job Description

Fluent is building the next generation advertising network, Partner Monetize & Advertiser Acquisition. Our vision is to build an ML/AI first network of advertisers and publishers to achieve a common objective, elevating relevancy in E-commerce for everyday shoppers. As a Senior Data & Automation Engineer, you will leverage your Databricks and Spark expertise to execute on building enterprise-grade data products that power Fluent’s business lines. These products serve as the foundation for sophisticated representations of customer journeys and marketplace activity across our ecosystem. You will partner with Data Architects, Data Scientists, and Product Managers to transform Enterprise Data Models into optimized physical data models and real-time pipelines. You will elevate standards across the team in code quality, observability, and architecture design—while actively contributing as a hands-on engineer. This role is fully remote in Ontario, with occasional travel to NYC or Toronto offices.

Job Responsibility

  • Design, build, and support scalable real-time and batch data pipelines using PySpark and Spark Structured Streaming on Databricks
  • Implement process automation and end-to-end workflows following Bronze → Silver → Gold architecture using Delta Lake best practices
  • Handle event-driven ingestion with Kafka and integrate into automated pipelines
  • Orchestrate workflows using Databricks Workflows/Jobs and CI/CD automation
  • Implement strong monitoring, observability, and alerting for reliability and performance (Databricks metrics, dashboards)
  • Collaborate cross-functionally in agile sprints with Product, Analytics, and Data Science teams
  • Translate enterprise logical data models into optimized physical and performance-tuned implementations
  • Write modular, version-controlled code in Git
  • contribute to code reviews and enforce quality standards
  • Implement robust logging, error handling, and data quality validation across automation layers
  • Utilize relevant AWS services (S3, IAM, Secrets Manager) and DevOps practices
  • Promote best practices through documentation, knowledge sharing, tech talks, and training

Requirements

  • 5+ years of professional experience in data engineering, including Spark (PySpark) and SQL
  • 3+ years of hands-on experience building pipelines on Databricks (Workflows, Notebooks, Delta Lake)
  • Deep understanding of Apache Spark distributed processing concepts and optimization
  • Strong experience with streaming architectures and Kafka
  • Familiarity with Databricks monitoring and observability tooling
  • Understanding of Lakehouse architecture, Unity Catalog, and governance principles
  • Proven proficiency in Git-based CI/CD workflows and automated deployment
  • Strong troubleshooting, optimization, and performance tuning skills
  • Experience designing and building large-scale, automated data pipelines

Nice to have

  • Experience with schema management (Schema Registry) and data validation frameworks (Great Expectations, Deequ)
  • Exposure to real-time ML systems and feature pipelines
  • Prior experience in startup or small agile teams
  • Familiarity with test-driven development in data engineering contexts

What we offer

  • Competitive compensation
  • Ample career and professional growth opportunities
  • New Headquarters with an open floor plan to drive collaboration
  • Health, dental, and vision insurance
  • Pre-tax savings plans and transit/parking programs
  • 401K with competitive employer match
  • Volunteer and philanthropic activities throughout the year
  • Educational and social events
  • Fully stocked kitchen
  • Catered lunch
  • Activity-filled events
  • Quarterly outings

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Senior Data & Automation Engineer

8 matching positions

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior QA Automation Engineer – Data & AI Platforms

We are looking for a Senior QA Automation Engineer with strong expertise in test...
Location
Location
Pakistan , Lahore; Karachi; Islamabad
Salary
Salary:
Not provided
northbaysolutions.com Logo
NorthBay
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Software Quality Assurance and Test Automation
  • Strong expertise in Test Planning, Test Strategy, and Test Execution
  • Hands-on experience with Selenium, Pytest, and Python-based automation
  • Solid experience in API Testing (REST APIs, JSON, Postman / similar tools)
  • Proven experience integrating automation with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, etc.)
  • Experience testing Data platforms, ETL pipelines, and large-scale datasets
  • Exposure to AI/ML pipeline testing, including model validation and data quality checks
  • Strong understanding of SDLC, Agile/Scrum, and DevOps practices
  • Excellent analytical, debugging, and communication skills
Job Responsibility
Job Responsibility
  • Lead test planning, test strategy, and test execution for complex web, API, data, and AI/ML-driven systems
  • Design, develop, and maintain automation frameworks using Selenium and Pytest
  • Perform API testing (RESTful services) including functional, regression, and integration testing
  • Validate data pipelines, ETL workflows, and data quality checks across large datasets
  • Test AI/ML pipelines, including model validation, data drift, feature validation, and inference testing
  • Integrate automated test suites into CI/CD pipelines to enable continuous testing
  • Collaborate with DevOps teams to ensure quality gates in build and deployment pipelines
  • Perform root cause analysis and drive defect resolution with engineering teams
  • Ensure adherence to QA best practices, SDLC, and Agile methodologies
  • Mentor junior QA engineers and contribute to improving overall QA maturity
What we offer
What we offer
  • Work on cutting-edge Data and AI/ML platforms
  • Hybrid work model across Lahore, Karachi, and Islamabad
  • Opportunity to influence quality strategy at scale
  • Collaborative engineering culture with strong focus on automation and innovation
  • Fulltime
Read More
Arrow Right

Senior Engineer - Data Engineer

Position - Data Engineer
Location
Location
India , Indore; Ahmedabad
Salary
Salary:
Not provided
arrow.com Logo
Arrow Electronics
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience: 4 to 8 years in software/data engineering
  • Proficiency in SQL, NoSQL databases (e.g., DynamoDB, MongoDB), ETL tools, and data warehousing solutions
  • Proficiency in Python is a must
  • Cloud Platforms: Azure, AWS (e.g., EC2, S3, RDS) or GCP
  • Experience with data visualization tools (e.g., Tableau, Power BI, Looker)
  • Knowledge of data governance and security practices
  • Experience with DevOps practices, including CI/CD pipelines and containerization (Docker, Kubernetes)
  • Excellent verbal and written communication skills in English
  • Experience working in Agile development environments
  • Understanding of AI and ML concepts, frameworks (e.g., TensorFlow, PyTorch), and practical applications
Job Responsibility
Job Responsibility
  • Design and development of real time software and Cloud/Web/mobile based software application
  • Analyze domain specific technical, high level or low level requirement and modification as per end customer or system requirement
  • Perform software testing including unit, functional and system level requirement including manual and automated
  • Perform code review following coding guidelines and static code analysis & troubleshoots software problems of limited difficulty
  • Document technical deliverable like software specifications, design document, code commenting, test cases and test report, Release note etc. throughout the project life cycle
  • Develop software solutions from established programming languages or by learning new language required for specific project
  • Fulltime
Read More
Arrow Right

Senior Data Engineer - Data Platform

We are looking for a Senior Data Engineer - Data Platform to join our Data & AI ...
Location
Location
France , Paris
Salary
Salary:
Not provided
doctolib.fr Logo
Doctolib
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • More than 7 years of experience as Site Reliability Engineer, Data Ops, Data Platform Engineer or in a similar role, with a proven track record of building and maintaining complex data infrastructures
  • Strong proficiency in data engineering and infrastructure tools and technologies, such as stream and events processing (Kafka, PubSub, Firehose) and Kubernetes
  • Expertise in programming languages like Python
  • Familiar with cloud infrastructure and services, preferably AWS, Azure, or GCP, and have experience with infrastructure-as-code tools such as Terraform
  • Excellent problem-solving skills with a focus on identifying and resolving data infrastructure bottlenecks and performance issues
Job Responsibility
Job Responsibility
  • Design and implement a scalable and reliable data infrastructure that supports the collection, processing, storage, and analysis of large-scale datasets while pushing security and privacy best practices
  • Build and maintain data pipelines that efficiently extract, transform, and load data from various sources into our data warehouse
  • Implement automation and orchestration tools to streamline infrastructure provisioning, data workflows, reduce manual effort, and improve operational efficiency
  • Monitor data platform for performance and reliability, identify and troubleshoot issues, and implement proactive solutions to ensure data quality and availability
  • Streamline and monitor platform costs, identify optimizations and saving opportunities while collaborating with data engineers, data scientists, and other stakeholders
What we offer
What we offer
  • Free comprehensive health insurance for you and your children
  • Parent Care Program: receive one additional month of leave on top of the legal parental leave
  • Free mental health and coaching services through our partner Moka.care
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
  • Work from EU countries and the UK for up to 10 days per year, thanks to our flexibility days policy
  • Up to 14 days of RTT
  • A subsidy from the work council to refund part of the membership to a sport club or a creative class
  • Lunch voucher with Swile card
  • Fulltime
Read More
Arrow Right

Senior Data Engineer – Data Engineering & AI Platforms

We are looking for a highly skilled Senior Data Engineer (L2) who can design, bu...
Location
Location
India , Chennai, Madurai, Coimbatore
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong hands-on expertise in cloud ecosystems (Azure / AWS / GCP)
  • Excellent Python programming skills with data engineering libraries and frameworks
  • Advanced SQL capabilities including window functions, CTEs, and performance tuning
  • Solid understanding of distributed processing using Spark/PySpark
  • Experience designing and implementing scalable ETL/ELT workflows
  • Good understanding of data modeling concepts (dimensional, star, snowflake)
  • Familiarity with GenAI/LLM-based integration for data workflows
  • Experience working with Git, CI/CD, and Agile delivery frameworks
  • Strong communication skills for interacting with clients, stakeholders, and internal teams
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable ETL/ELT pipelines across cloud and big data platforms
  • Contribute to architectural discussions by translating business needs into data solutions spanning ingestion, transformation, and consumption layers
  • Work closely with solutioning and pre-sales teams for technical evaluations and client-facing discussions
  • Lead squads of L0/L1 engineers—ensuring delivery quality, mentoring, and guiding career growth
  • Develop cloud-native data engineering solutions using Python, SQL, PySpark, and modern data frameworks
  • Ensure data reliability, performance, and maintainability across the pipeline lifecycle—from development to deployment
  • Support long-term ODC/T&M projects by demonstrating expertise during technical discussions and interviews
  • Integrate emerging GenAI tools where applicable to enhance data enrichment, automation, and transformations
What we offer
What we offer
  • Opportunity to work at the intersection of Data Engineering, Cloud, and Generative AI
  • Hands-on exposure to modern data stacks and emerging AI technologies
  • Collaboration with experts across Data, AI/ML, and cloud practices
  • Access to structured learning, certifications, and leadership mentoring
  • Competitive compensation with fast-track career growth and visibility
  • Fulltime
Read More
Arrow Right

Software Developer - Senior / Senior Data Engineer

Imagine being at the forefront of protecting financial integrity and enabling cr...
Location
Location
United States , Southlake, TX or Austin, TX
Salary
Salary:
55.92 - 62.13 USD / Hour
aquent.com Logo
Aquent
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field
  • 8+ years of hands-on experience in data management, including gathering data from diverse sources, consolidating it into centralized locations, and transforming it with business logic for consumption in visualization and data analysis
  • Strong expertise in BigQuery, Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, and related cloud platform services
  • Proficiency in Python and SQL for data processing and automation
  • Extensive experience with ETL processes and data pipeline design
  • Excellent problem-solving skills and meticulous attention to detail
  • Strong communication and collaboration skills, with the ability to actively listen, dialogue freely, and verbalize ideas effectively
  • Ability to thrive in an Agile work environment, delivering incremental value to customers by effectively managing and prioritizing tasks
Job Responsibility
Job Responsibility
  • Design, build, and maintain robust and scalable data pipelines using cloud platform tools such as BigQuery, Cloud Storage, Dataflow (Apache Beam), Cloud Composer (Airflow), and Pub/Sub
  • Develop high-performance, production-grade Python and SQL code, optimizing queries for efficient data extraction, transformation, and loading (ETL) processes
  • Implement complex data models in BigQuery, leveraging partitioning, clustering, and materialized views to achieve optimal performance
  • Collaborate closely with cross-functional teams, including business customers and Subject Matter Experts, to gather data requirements and deliver impactful solutions
  • Implement and uphold best practices for data quality, data governance, and data security
  • Proactively monitor and troubleshoot data pipeline issues, ensuring high availability and performance of critical data flows
  • Contribute to strategic data architecture decisions, providing recommendations for continuous improvement of data pipelines
  • Stay current with emerging trends and technologies in cloud-based data engineering and cyber security to drive innovation
  • Lead investigation and resolution efforts for identified data issues, taking ownership to resolve them in a timely manner
  • Document processes and procedures thoroughly for producing accurate metrics and ensuring operational clarity
What we offer
What we offer
  • subsidized health, vision, and dental plans
  • paid sick leave
  • retirement plans with a match
Read More
Arrow Right

Software Engineer / Senior Software Engineer - Data Engineering GitHub

As a Software Engineer at GitHub, you will enhance the collaboration experience ...
Location
Location
Czech Republic , Multiple Locations
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND experience in Data Engineering and coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Go, Ruby, Rust, or Python OR equivalent experience.
  • Master's Degree in Computer Science or related technical field AND engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Go, Ruby, Rust or Python OR Bachelor's Degree in Computer Science or related technical field AND engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Go, Ruby, Rust or Python OR equivalent experience.
Job Responsibility
Job Responsibility
  • Design, develop, test and ship high-quality technical solutions that scale across multiple GitHub services.
  • Collaborate with cross-functional teams to define and implement innovative solutions.
  • Provide technical leadership, mentorship, pairing opportunities, and code reviews to encourage the growth of others.
  • Own and advocate for the health and quality of the systems that the team builds, including participating in on-call and first responder rotations
  • Write architecture briefs and proposals, carry out code experiments, and build prototypes to learn how we can achieve planetary scale with our systems.
  • Design and implement APIs to facilitate seamless integration between software components.
  • Utilize CI/CD tools to set up automated pipelines for continuous integration and delivery.
  • Become intimately familiar with the systems you build and take pride in writing maintainable code.
  • Fulltime
Read More
Arrow Right