CrawlJobs Logo

Data Engineer

United States, Redmond Employment contract 155000.00 - 175000.00 USD / Year · Job Posted December 01, 2025
Apply Position
Job Link Share

Job Description

As a Data Engineer, you’ll build and refine the pipelines, data models, and services that make Microsoft’s research content discoverable and usable across modern applications and emerging AI scenarios. You’ll define the architecture and core data systems largely from scratch, creating the foundation for both web and AI-powered experiences. You’ll use existing AI models and out-of-the-box Microsoft tools to turn unstructured content into structured, high-quality data—not heavy ML research, but practical, model-assisted enrichment. You’ll partner with a PM and a full stack engineer while independently driving backend and data direction. The work includes an initial infrastructure lift to establish the environment, followed by iterative development as the MVP grows.

Job Responsibility

  • Build and maintain core data pipelines
  • Build and maintain end-to-end ingestion pipelines for documents, datasets, code repositories, videos, transcripts, and internal knowledge sources
  • Clean, normalize, structure, and store data in formats that support both web applications and AI-driven use cases
  • Use “out of the box” Microsoft tools—such as Fabric, Azure services, Cosmos DB, or Copilot Studio—to create reliable, maintainable systems
  • Enrich and model research data
  • Use AI models to transform unstructured content into structured metadata and durable knowledge assets
  • Design the architecture and foundational data systems, establishing the patterns and infrastructure for a new, scalable environment
  • Develop and refine embeddings, vector indexes, and retrieval components to support semantic search and grounding scenarios
  • Build backend and data services
  • Build data services, APIs, and backend components that power internal applications and agent-supported workflows
  • Iterate on systems after the initial MVP, improving reliability, performance, and scalability over time
  • Collaborate and translate requirements
  • Collaborate with a PM and full stack engineer to understand requirements and translate them into actionable data solutions
  • Work cross-functionally to define data needs and align systems with downstream consumers and discovery workflows

Requirements

  • Proven ability to design and build end-to-end data systems, from ingestion through cleaning, structuring, storage, and serving
  • Experience building and shipping data products that deliver practical value
  • Demonstrated impact using AI models in data workflows (applied use, not ML research)
  • 5+ years of software or data engineering experience, including at least 2 years of hands-on work with data pipelines
  • Comfortable defining architecture and starting systems from scratch, working independently in a small cross-functional team
  • Proficiency in Python, SQL, or similar languages used in data engineering workflows

Nice to have

  • Experience with Microsoft Fabric, Cosmos DB, Azure data services, or Copilot Studio
  • Background building data that supports embeddings, semantic search, or retrieval use cases
  • Familiarity with metadata frameworks, taxonomies, or knowledge modeling
  • Experience shaping ambiguous information into structured datasets and iterating quickly after an MVP

What we offer

  • Flexible time-off plan
  • 100% employer-paid medical, dental, and vision insurance
  • Employer-paid life insurance for those enrolled in medical coverage
  • 401(k) plan with company match
  • Fertility, surrogacy, and adoption benefits
  • Fitness and caregiver benefits
  • Employee Assistance Program
  • 100% employer-paid short- and long-term disability coverage

Looking for more opportunities?

Search for other job offers that match your skills and interests.

Similar Jobs for

Data Engineer

8 matching positions

New

Data Engineer

We believe in the power of ingenuity to build a positive human future. We challe...
Location
Location
United Kingdom , Belfast
Salary
Salary:
Not provided
paconsulting.com Logo
PA Consulting
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • You thrive in problem-solving and analytical thinking
  • You enjoy collaborating with multiple stakeholders in a fast-paced environment
  • Experience in the design and deployment of production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala, Spark, SQL
  • Experience performing tasks such as writing scripts, extracting data using APIs, writing SQL queries etc
  • Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources through ingestion and curation functions on AWS cloud using AWS native or custom programming
  • Be a British citizen or have been resident in the UK for the past 5 years
Job Responsibility
Job Responsibility
  • Work to agile best practices and cross-functionally with multiple teams and stakeholders
  • Use technical skills to problem solve with clients, as well as working on internal projects
  • Live in-person whiteboarding sessions to problem solve as a team, alongside asynchronous communication on Teams
  • Work alongside colleagues from across PA delivering transformative digital solutions
  • Design and build for the AWS cloud
What we offer
What we offer
  • Private healthcare for you and your family
  • 25 days annual leave (plus a bonus half day on Christmas Eve) with the opportunity to buy 5 additional days
  • Generous company pension scheme
  • Opportunity to get involved with community and charity-based initiatives
  • Annual performance-based bonus
  • PA share ownership
  • Tax efficient benefits (cycle to work, give as you earn)
  • Fulltime
Read More
Arrow Right

Data Engineer

Robert Half is seeking a Contract Data Engineer to support our client's data and...
Location
Location
United States , Nashville
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Proven experience as a Data Engineer or in a similar role
  • Strong proficiency in SQL and Python (or similar languages)
  • Experience with cloud platforms (AWS, Azure, GCP)
  • Hands-on experience with ETL / ELT tools (Airflow, dbt, Fivetran, Matillion, Glue, ADF, etc.)
  • Experience with data warehousing platforms such as Snowflake, Redshift, BigQuery, or Azure Synapse
  • Familiarity with streaming and big data tools (Kafka, Spark, Databricks, Hadoop) is a plus
  • Strong understanding of data modeling, performance tuning, and pipeline optimization
  • Experience with version control systems (Git) and agile development practices
  • Excellent problem-solving, analytical, and communication skills
Job Responsibility
Job Responsibility
  • Data Pipeline Development: Design, build, and maintain scalable ETL / ELT pipelines to ingest, transform, and deliver data from multiple sources
  • Data Architecture: Develop and optimize data models, schemas, and warehouse structures to support analytics, reporting, and business intelligence needs
  • Cloud Data Platforms: Work within cloud environments such as AWS, Azure, or GCP to deploy and manage data solutions
  • Data Warehousing: Design and support enterprise data warehouses using platforms such as Snowflake, Redshift, BigQuery, or Azure Synapse
  • Big Data Processing: Develop solutions using big data technologies such as Spark, Databricks, Kafka, and Hadoop when required
  • Performance Optimization: Tune queries, pipelines, and storage solutions for performance, scalability, and cost efficiency
  • Data Quality & Reliability: Implement monitoring, validation, and alerting processes to ensure data accuracy, integrity, and availability
  • Collaboration: Work closely with Data Analysts, Data Scientists, Software Engineers, and business stakeholders to understand requirements and deliver data solutions
  • Documentation: Maintain detailed documentation for pipelines, data flows, and system architecture
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
Read More
Arrow Right

Data Engineer

Our client is seeking a skilled Data Engineer to support the design, development...
Location
Location
United States , Miami
Salary
Salary:
Not provided
https://www.roberthalf.com Logo
Robert Half
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Information Systems, Engineering or related field
  • 3+ years of experience in a Data Engineer or similar role
  • Strong hands-on experience with SQL, Python and Snowflake
  • Experience developing and maintaining ETL workflows
  • Knowledge of data modeling concepts and best practices
  • Experience with Selenium for web automation or web scraping support
  • Strong analytical, problem-solving and troubleshooting skills
  • Ability to work independently and collaboratively in a team environment
Job Responsibility
Job Responsibility
  • Design, build and maintain scalable ETL pipelines to support data integration and reporting needs
  • Develop and optimize complex queries using SQL
  • Use Python to support data processing, transformation and automation tasks
  • Work within Snowflake to manage, transform and optimize cloud-based data solutions
  • Assist with automation efforts, including Selenium web scripting for web-based data extraction and process automation
  • Create and maintain logical and physical data models
  • Ensure data quality, integrity and consistency across multiple data sources
  • Collaborate with business stakeholders, analysts and technical teams to gather requirements and deliver data solutions
  • Monitor and troubleshoot data workflows and automation scripts
  • Document processes, workflows and technical specifications
What we offer
What we offer
  • medical
  • vision
  • dental
  • life and disability insurance
  • 401(k) plan
Read More
Arrow Right

Data Engineer

Join us as a Data Engineer at Bettys and Taylors Group - Our family business is ...
Location
Location
United Kingdom , Harrogate
Salary
Salary:
49500.00 - 60500.00 GBP / Year
jobs.360resourcing.co.uk Logo
360 Resourcing Solutions
Expiration Date
June 17, 2026
Flip Icon
Requirements
Requirements
  • Proven experience designing and building data pipelines and ETL/ELT processes using Microsoft Fabric or comparable cloud data platforms
  • Strong technical skills in SQL, DAX, M (Power Query), KQL and Python / PySpark for data engineering and analysis
  • Experience with data modelling techniques (e.g. star schema, dimensional modelling) and data warehousing concepts
  • Hands-on experience with Power BI, including dataset creation, semantic modelling, performance optimisation and building well-designed reports
  • Familiarity with CI/CD practices, version control (e.g. Git) and DevOps approaches for data engineering
  • Understanding of data governance, quality management, security and privacy, including GDPR and ISO27001 considerations
  • Excellent communication skills, with the ability to work confidently across technical and non-technical teams
  • A willingness to coach, mentor and upskill others, and a strong commitment to continuous learning
Job Responsibility
Job Responsibility
  • Design, develop and maintain scalable data pipelines using Microsoft Fabric and related technologies
  • Transform and curate data from multiple sources into structured, governed datasets
  • Optimise data pipeline performance and adhere to engineering standards such as CI/CD and version control
  • Collaborate on enterprise data models, Power BI solutions, and provision data for analytics and machine learning
  • Ensure data quality, compliance with governance standards, and support ITIL processes
What we offer
What we offer
  • A discretionary profit-share bonus scheme paid quarterly (11.9% average over the last three years)
  • Freshly cooked hot meals while at work, along with free drinks and snacks
  • Discounts on Bettys & Taylors products
  • A range of health benefits, including a health cash plan, health assessments, flu jabs and more
  • Wellbeing support, including an Employee Assistance Programme (free and independent service offering free counselling and health and wellbeing resources)
  • The option to buy and (in some cases) sell additional holidays
  • On-the-job learning
  • External training programmes
  • Apprenticeship opportunities
  • Internal 1:1 coaching schemes
  • Fulltime
!
Read More
Arrow Right

Data Engineer

A healthcare company based in the Midwest is looking to hire a remote Data Engin...
Location
Location
United States , Detroit
Salary
Salary:
85000.00 - 95000.00 USD / Year
strategicemployment.com Logo
Strategic Employment Partners
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 1+ years in stakeholder-facing Data Engineering/Data Analytics position
  • Consulting experience
  • Databricks, Foundry, or Snowflake
  • Python, PySpark, TypeScript
Job Responsibility
Job Responsibility
  • work closely with stakeholders on low code data engineering solutions
  • Fulltime
Read More
Arrow Right

Data Engineer

Wells Fargo is seeking a Data Engineer to design, build and optimize scalable da...
Location
Location
India , BENGALURU
Salary
Salary:
Not provided
https://www.wellsfargo.com/ Logo
Wells Fargo
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2+ years of Data Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
  • Strong experience in EDL pipelines using Spark & BigQuery on GCP
  • Hands-on expertise in GCP services (BigQuery, Dataflow, GCS, IAM) and cloud-native architecture
  • Solid understanding of enterprise data governance, data quality
  • Exposure to AI/ML or GenAI use cases, including integration with data platforms and awareness of AI governance standards
Job Responsibility
Job Responsibility
  • Participate in data warehouses across multiple databases and identify opportunity for developing table schemas within the Technology and Date area
  • Review and analyze basic data management business, operational, or technical assignments or challenges that require research, evaluation, and selection of alternatives and exercise independent judgment to guide medium risk deliverables
  • Present recommendations for resolving more complex data management situations and exercise independent judgment while developing expertise in the business data policies, procedures and compliance requirements
  • Collaborate and consult with technology colleagues, internal partners and stakeholders, including internal and external customers if applicable
  • Design and build scalable, production-grade data pipelines on GCP supporting analytics and AI workloads
  • Develop and manage secure, compliant cloud data platforms aligned with enterprise policies and controls
  • Optimize data processing for performance, cost efficiency, and reliability (Spark + BigQuery workloads)
  • Collaborate with AI/analytics teams to enable feature engineering, model consumption, and AI-driven insights
  • Fulltime
Read More
Arrow Right
New

Data Engineer

Join us as a Data Engineer at Barclays where you will spearhead the evolution of...
Location
Location
India , Bengaluru
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Hands-on experience in developing complex/medium/easy datasets using SQL tools (SQL/Snowflake)
  • Solid working knowledge of writing SQL queries
  • Working Knowledge on scripts (.bteq ) in Unix/ linux environments.
Job Responsibility
Job Responsibility
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models.
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Health and wellness
  • Wellness rooms
  • gyms and mental health support
  • Collaborative areas
  • Access to on-site cafeterias, fitness centers, and tech-equipped workstations
  • Fulltime
Read More
Arrow Right
New

Data Engineer

Join us as a Data Engineer at Barclays, responsible for supporting the successfu...
Location
Location
India , Pune
Salary
Salary:
Not provided
barclays.co.uk Logo
Barclays
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in ETL / data processing / data warehousing / data engineering / RDBMS / PySpark
  • Experience in Data Processing and ETL in On-prem and Cloud based systems
  • Preferred AWS cloud development, with a strong understanding of AWS services (e.g., EC2, S3, Lambda, RDS, DynamoDB, etc.)
  • Proficient in at least one Python programming language
  • Hands-on experience with Infrastructure as Code (IaC) tools like AWS CloudFormation, Terraform etc.
  • Experience with CI/CD tools and pipelines (Jenkins, GitLab CI, etc.) for automation and deployment
  • Good understanding of cloud security best practices and experience with IAM, VPC, security groups, and encryption mechanisms
  • Good knowledge of monitoring, logging, and troubleshooting tools (e.g., CloudWatch, CloudTrail)
  • Experience with version control systems (Git, GitHub, Bitbucket)
  • Strong communication skills and ability to work in a collaborative team environment
Job Responsibility
Job Responsibility
  • supporting the successful delivery of location strategy projects to plan, budget, agreed quality and governance standards
  • spearhead the evolution of our digital landscape, driving innovation and excellence
  • harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models
What we offer
What we offer
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Fulltime
Read More
Arrow Right