CrawlJobs Logo

Software Engineer II - Apache Spark

https://www.microsoft.com/ Logo

Microsoft Corporation

Location Icon

Location:
Canada , Vancouver

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

85000.00 - 166800.00 CAD / Year

Job Description:

Microsoft’s Azure Data engineering team is leading the transformation of analytics in the world of data with products like databases, data integration, big data analytics, messaging & real-time analytics, and business intelligence. The products our portfolio include Microsoft Fabric, Azure SQL DB, Azure Cosmos DB, Azure PostgreSQL, Azure Data Factory, Azure Synapse Analytics, Azure Service Bus, Azure Event Grid, and Power BI. Our mission is to build the data platform for the age of AI, powering a new class of data-first applications and driving a data culture. Within Azure Data, the big data analytics team provides a range of products that enable data engineers and data scientists to extract intelligence from all data – structured, semi-structured, and unstructured. We build the Data Engineering, Data Science, and Data Integration pillars of Microsoft Fabric. The Fabric Spark team is looking to push the frontiers of analytics performance and scale. You will have to help rebuild the way analytics engines process data. We build services that enable big data developers to build mission critical applications with high levels of SLA through auto detection and healing. We enable advanced data Insights and recommendations externally to our customers. As a Software Engineer II on our team, you will have an opportunity to work on and extend OSS technologies like Spark, Gluten, Velox, Delta and Hadoop as well as proprietary Microsoft services. You will help design & implement software components, participate in code reviews, design reviews, assist customers as needed to be able to leverage the service and keep the service running. You will work with a team of talented engineers aiming to solve performance issues by looking at an entire workload and learning from its execution history. Your goal is to make onboarding new workloads on to Fabric a breeze and to help keep them continuously tuned with changes in the Spark engine and user workload. This will require exploring the frontiers of what is possible. You should be AI forward in your approach to delivering impact.

Job Responsibility:

  • Work with a team of passionate engineers to deliver success for customers
  • Design, implement, test, and operate data services.
  • Release features on time, with high quality, meeting functional, performance, scalability, and compliance requirements.
  • Drive quality right from the design phase, incorporating best practices and engineering for testability.
  • Solve problems relating to mission critical services and create solutions to prevent problem recurrence.
  • Participate in product live site and operations.
  • Embody our culture and values

Requirements:

  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • 1+ year professional experience with Apache Spark, MapReduce, database internals, or compete technologies
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Nice to have:

  • Master's Degree in Computer Science or related technical field AND 3+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 5+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
  • Experience with database internals
  • Experience with OSS components like the Spark eco system
  • Experience with building scalable cloud services.

Additional Information:

Job Posted:
March 19, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Software Engineer II - Apache Spark

Software Engineer II

Security represents the most critical priorities for our customers in a world aw...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum of 4+ years of software development experience
  • Strong skills in distributed systems and data processing
  • Hands-on experience with cloud platforms such as Azure, AWS, or Google Cloud
  • Experience with Azure Services is a plus
Job Responsibility
Job Responsibility
  • Build cloud-scale services that process and analyze massive volumes of organizational signals in real time
  • Harness the power of Apache Spark for high-performance data processing and scalable pipelines
  • Apply machine learning to uncover subtle patterns and anomalies that signal insider threats
  • Craft intelligent user experiences using React and AI-driven insights to help security analysts act with confidence
  • Collaborate across disciplines—from data science to UX to cloud infrastructure—in a fast-paced, high-impact environment
  • Design and deliver end-to-end features including system architecture, coding, deployment, scalability, performance, and quality
  • Ensure engineering excellence by writing effective code, unit tests, debugging, code reviews, and building CI/CD pipelines
  • Troubleshoot and optimize Live Site operations, focusing on automation, reliability, and monitoring
  • Fulltime
Read More
Arrow Right

Software Engineer II - Delivery Search

We are seeking talented Senior Software Engineers to join our Engineering team, ...
Location
Location
United States , San Francisco, California; Sunnyvale, California
Salary
Salary:
171000.00 - 190000.00 USD / Year
uber.com Logo
Uber
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years of experience in software engineering
  • Expertise in big data technologies such as Apache Spark, Kafka, Flink, Airflow, Presto, or Snowflake
  • Familiarity with search retrieval, ranking techniques, query understanding, and text processing
  • Hands-on experience with search technologies such as Elasticsearch, OpenSearch, Solr, or Vespa
  • Strong experience with search and recommendation systems, working with Elasticsearch, OpenSearch, Solr, or similar technologies
  • Proficiency in distributed data processing frameworks and real-time streaming architectures
  • Deep understanding of data modeling, ETL pipelines, and data warehousing principles
  • Strong programming skills in Golan, Python, Scala, or Java
  • Experience with cloud platforms (AWS, GCP, or Azure) and modern data infrastructure tools
  • Ability to work on high-scale distributed systems and troubleshoot performance bottlenecks
Job Responsibility
Job Responsibility
  • Develop serving infrastructure to enhance system latency, throughput, and reliability
  • Enhance search relevance by improving indexing, retrieval, and ranking mechanisms
  • Develop and optimize search algorithms, ranking models, and query processing techniques
  • Implement and maintain scalable search pipelines and distributed indexing systems
  • Work with machine learning engineers to integrate AI-driven search ranking and personalization models
  • Analyze search performance metrics and run A/B experiments to measure improvements
  • Optimize latency, throughput, and scalability of search infrastructure
  • Contribute to system design and architecture decisions to improve search quality and efficiency
  • Write clean, efficient, and maintainable code in Go, Python or Java
  • Collaborate with cross-functional teams to enhance search relevance and user experience
What we offer
What we offer
  • Eligible to participate in Uber's bonus program
  • May be offered an equity award & other types of comp
  • All full-time employees are eligible to participate in a 401(k) plan
  • Eligible for various benefits
  • Fulltime
Read More
Arrow Right

Data Engineer II

Piper Companies is seeking a Data Engineer II to support a key federal healthcar...
Location
Location
Salary
Salary:
100000.00 - 115000.00 USD / Year
pipercompanies.com Logo
Piper Companies
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s Degree in Computer Science, Computer Engineering, or related technical discipline
  • 5+ years of experience in data engineering, backend engineering, or similar roles
  • Strong experience with Python and Apache Spark
  • Working knowledge of R
  • Solid understanding of data modeling, ETL processes, and distributed computing architectures
  • Strong foundation in software engineering fundamentals, including data structures, algorithms, and design patterns
  • Experience working within Agile/Scrum development teams
  • Ability to work independently and collaboratively
  • Strong analytical thinking and problem-solving skills
  • Excellent written and verbal communication abilities
Job Responsibility
Job Responsibility
  • Build and maintain PySpark data pipelines within the Databricks platform
  • Optimize Spark job performance, addressing bottlenecks, and improving resource utilization across distributed systems
  • Design, develop, and maintain backend data components and services that support large-scale data processing
  • Conduct research and develop proof-of-concepts for new tools, frameworks, and solutions in the data engineering ecosystem
  • Write clean, maintainable, and scalable code following best practices and coding standards
  • Perform code reviews to ensure quality, consistency, and reliability across the team
  • Debug and troubleshoot backend data issues, proactively identifying potential system risks
  • Develop and maintain thorough technical documentation
  • Actively participate in Agile ceremonies (standups, sprint planning, retrospectives)
  • Support estimation, task breakdown, and prioritization to meet delivery timelines
  • Fulltime
Read More
Arrow Right

Software Engineer II

We are the Microsoft Data Privacy Platform Team, a part of the Cloud Ecosystem S...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in computer science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, Python, Spark, C, C++, C#
  • Proven design, coding and debugging skills in Python, Spark, C# and PowerBI
  • Ability to work and communicate effectively across disciplines and teams
  • Expertise in distributed data processing and analysis using technologies such as Apache Spark
  • 4+ years of expertise in software development with a focus on designing and developing data engineering pipelines
  • Proven design and problem-solving skills, with an emphasis on quality and engineering excellence
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • 4+ years of experience in data engineering
  • Hands-on experience in architecting and implementing data engineering pipelines with cloud-based tools like Azure Synapse, Azure Data Factory, SQL, Azure Data Explorer, Power BI, and Apache Spark
  • Skilled in designing and developing Power BI reports
Job Responsibility
Job Responsibility
  • Provide new and improve upon existing data platform offerings with a fundamental understanding of the end-to-end scenarios
  • Collaborate with partner teams and customers to scope and deliver projects
  • Authoring and designing Big Data ETL pipelines in Python, Spark, Synapse, Kusto, etc
  • Developing and managing Power BI reports
  • Data extraction across a wide variety of data sources
  • Data cleaning, preprocessing, and transformation for further analysis by data analysts
  • Data Validation framework from source to endpoints ensuring data quality and integrity
  • Fulltime
Read More
Arrow Right

Software Engineer II

Imagine helping millions of users discover the best local businesses and service...
Location
Location
United States , Bellevue
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • 2+ years of experience in solving, design, coding, and debugging skills
  • 1+ with data engineering leveraging tools such as Apache Hadoop or Spark or equivalent experience
Job Responsibility
Job Responsibility
  • Works with appropriate stakeholders to determine user requirements for a set of features
  • Contributes to the identification of dependencies, and the development of design documents for a product area with little oversight
  • Creates and implements code for a product, service, or feature, reusing code as applicable
  • Contributes to efforts to break down larger work items into smaller work items and provides estimation
  • Acts as a Designated Responsible Individual (DRI) working on-call to monitor system/product feature/service for degradation, downtime, or interruptions and gains approval to restore system/product/service for simple problems
  • Remains current in skills by investing time and effort into staying abreast of current developments that will improve the availability, reliability, efficiency, observability, and performance of products while also driving consistency in monitoring and operations at scale
  • Fulltime
Read More
Arrow Right

Software Engineer II

Our team in Microsoft Security owns cloud services and AI powered product offeri...
Location
Location
United States , Redmond
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ year(s) technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
  • 2+ years of experience designing, developing, or maintaining secure software systems, with applied knowledge of authentication, data protection, access control, and secure coding practices
  • 2+ years of experience designing and building scalable data pipelines and analytics infrastructure using orchestration tools such as Apache Airflow, Azure Data Factory, or equivalent, with an emphasis on data quality, automation, pipeline reliability, and framework development
  • 2+ years of experience working with distributed data processing frameworks such as Apache Spark, Databricks, or similar technologies to transform and manage large-scale datasets
  • 2+ years of experience with cloud platforms such as Azure, AWS, GCP, or comparable environments, including building and operating scalable, highly available services and systems
  • 2+ years of experience programming in Python, Scala, or similar languages for data engineering tasks, including performance tuning, observability, and working with query languages such as KQL
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Designing, developing, and maintaining high-quality software solutions
  • Helping maintain and improve the accuracy and quality of the data that is collected to power services that help customers protect their digital footprint
  • Creating world-impacting solutions and reimagining the future of enterprise surface management through the power of modern platforms
  • Fulltime
Read More
Arrow Right

Software Engineer II - Distributed Systems

Microsoft’s Azure Data engineering team is leading the transformation of analyti...
Location
Location
Canada , Vancouver
Salary
Salary:
85000.00 - 166800.00 CAD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
  • OR equivalent experience
  • 1+ year experience with distributed messaging systems such as Azure Service Bus, Apache Kafka, RabbitMQ, Google Pub/Sub, or SQS/Kinesis or experience with stream processing or real-time data platforms such as Apache Spark, Flink, Storm, or similar frameworks
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Design and implement high-quality, scalable, and resilient components of distributed systems based on defined requirements and architectural guidance
  • Translate design specifications into clean, extensible, and maintainable code that meets performance, reliability, and security expectations
  • Continuously improve existing code through debugging, refactoring, and performance optimization to increase reliability and operational efficiency
  • Apply engineering best practices, metrics, and testing strategies to ensure code quality and long-term maintainability
  • Participate in code reviews by providing thoughtful feedback and incorporating feedback from peers to uphold team and Microsoft engineering standards
  • Collaborate with teammates, program managers, and partner teams throughout the development lifecycle, including planning, implementation, deployment, and live-site support
  • Contribute to the messaging infrastructure that supports event notification, asynchronous communication, telemetry, and diagnostics across Azure services
  • Fulltime
Read More
Arrow Right

Software Engineer II

Microsoft’s Fabric OneLake team is building the next generation of data experien...
Location
Location
United States , Redmond
Salary
Salary:
100600.00 - 199000.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Job Responsibility
Job Responsibility
  • Design and implement microservices for large-scale, cloud-native distributed systems that power OneLake’s next-generation data experiences.
  • Develop robust APIs and service components with SLAs, telemetry, and operational excellence for high availability and reliability.
  • Collaborate with cross-functional teams (PMs, engineers, and partner teams) to translate requirements into scalable and secure service designs.
  • Ensure service fundamentals—including reliability, scalability, security, cost optimization, and observability—are embedded in all deliverables.
  • Participate in architecture reviews, design documentation, and code reviews, contributing to best practices and technical standards.
  • Integrate with data platforms and frameworks such as Spark and OSS table formats (Delta Lake, Apache Iceberg) to enable seamless data workflows.
  • Explore and optionally contribute to advanced capabilities like semantic views, embedding pipelines, and vector indexing (FAISS, IVF-PQ, DiskANN).
  • Embody our culture and values.
  • Fulltime
Read More
Arrow Right