Cloud Data Ingestion Engineer Job at Beacon Hill (Charlotte)

Fabric Data Engineer / Azure Data Engineer

Location

Salary:

Not provided

Myticas Consulting

Expiration Date

Until further notice

Requirements

Hands-on Data Engineer with strong Microsoft Fabric experience, including Lakehouse, OneLake, Data Factory/Pipelines, Warehousing, and Fabric ecosystem services
Strong Python development skills for data engineering, data transformation, automation, and pipeline development
Experience working with Microsoft Fabric Lakehouse architecture, including data ingestion, transformation, storage, and analytics enablement
Strong understanding of data engineering fundamentals, including data modeling, partitioning, optimization, and performance tuning
Experience supporting Azure Data Platform technologies, including Azure Data Factory, Azure Synapse Analytics, Azure Storage, and related services
Ability to work within large-scale enterprise data environments supporting multiple business units and data domains
Knowledge of Medallion Architecture (Bronze, Silver, Gold) and practical implementation of modern data lakehouse solutions
Experience integrating and transforming structured and semi-structured data sources including APIs, databases, JSON, CSV, and cloud-based systems
Understanding of Fabric operational monitoring, capacity consumption monitoring, and platform governance best practices
Ability to clearly explain and demonstrate the distinction between Fabric Files, Tables, Lakehouses, Warehouses, and OneLake storage concepts

Job Responsibility

Design, build, and support scalable ETL/ELT data pipelines across enterprise Azure environments
Ability to support integration across multiple systems, applications, databases, and cloud platforms

New

Data Engineer (Cloud Architecture)

Data Careers is supporting a secure public sector organisation in the East Midla...

Location

United Kingdom , East Midlands

Salary:

Not provided

DataCareers

Expiration Date

Until further notice

Requirements

UK-based
Continuously resided in the UK for a minimum of five years
Eligible to obtain NPPV3 and SC clearance
Microsoft Fabric implementation, configuration or administration
Data engineering and ETL/ELT development
Data ingestion and integration from multiple source systems
Cloud-based data platform experience, ideally Azure
API integration and data pipeline development
SQL and data modelling expertise
Experience working with large and complex datasets

Job Responsibility

Design, develop and maintain modern cloud-based data pipelines
Build and support ETL and ELT processes across multiple source systems
Configure and administer Microsoft Fabric data environments
Design and implement data ingestion and landing strategies
Develop scalable data architectures using Medallion Architecture principles
Deliver API integrations and automated data workflows
Support data quality, governance and compliance requirements
Collaborate with analysts and stakeholders to develop effective data solutions
Contribute to the ongoing development of data engineering standards and best practice

Fulltime

Data Engineer – Cloud and Analytics Platforms

As a Data Engineer, you will be the foundational backbone of our data analytics ...

Location

Canada , Calgary

Salary:

Not provided

Randstad

Expiration Date

August 02, 2026

Requirements

Minimum of 3+ years in a dedicated Data Engineering role or a Software Developer role with a strong focus on data-backend development and complex transformations
Excellent, production-level knowledge of Python
Strong knowledge of SQL paired with a deep understanding of performance analysis and query optimization techniques
Rigid grasp of computer science fundamentals, including modularity, abstraction, data structures, and algorithms
Solid understanding of core data modeling concepts (normalized vs. denormalized data architectures, conceptual/logical/physical models, and star/snowflake schemas)
Proven experience building automated, production-grade ETL/ELT pipelines
Solid project experience leveraging various data storage technologies (including RDBMS, NoSQL, and Graph Databases)
Direct experience with cloud infrastructure provisioning and deployment automation on AWS

Job Responsibility

Build, improve, and maintain data infrastructure for ingesting, storing, and transforming data across advanced analytical workflows
Implement robust, scalable data pipelines supporting both stream and batch processing
Perform end-to-end ETL/ELT processes to extract and load data from a wide variety of structured and unstructured sources
Partner with enterprise architects and Information Services (IS) teams to design and evolve the overarching data platform environment in AWS
Write complex, automated SQL queries to manipulate data extracts and manage large-scale data flows
Perform continuous query optimization, performance analysis, and database tuning to minimize processing latency and cost
Assist with logical and physical data modeling, implementing optimized data structures (e.g., star and snowflake schemas)
Automate cloud deployments, data flows, and automated data quality validation checks
Collaborate closely with Data Scientists and ML developers to build, test, and operationalize automated ML pipelines
Design rigorous testing processes, formulating and executing comprehensive test cases for data validation

Fulltime

Senior Software Engineer, Data Ingestion

Fullstory’s mission is to help teams create amazing online experiences for their...

Location

United States , Atlanta

Salary:

160000.00 - 180000.00 USD / Year

Fullstory

Expiration Date

Until further notice

Requirements

You possess a strong understanding the nuances of distributed architectures and tackling capacity and performance challenges when dealing with data at scale
You have experience writing Golang code in production
You have experience with Kubernetes and supporting highly available and reliable cloud based microservices in production
You have worked on asynchronous or streaming ingestion and processing systems and frameworks
You leverage AI tools to enhance work quality by implementing AI solutions that optimize efficiency

Job Responsibility

Engineer distributed systems that operate at tens to hundreds of thousands of requests per second using Go, Kubernetes, and GCP
Explore ideas about how to unlock new features through thoughtful architecture and framework designs
Ensure the quality and reliability of Fullstory's capture and extraction systems across many services and downstream applications both internally and externally
Collaborate with technical leaders and product experts to evolve the technical roadmap for Ingestion services, and to participate in collaborative development efforts across the Engineering organization

What we offer

Flexibility and Connection
flexible PTO policy
annual company-wide closure
federal holidays
sponsored benefit packages for US-based Fullstorians
supplemental coverage options for international Fullstorians
professional development opportunities through training programs
annual learning subsidy for US and EMEA-based employees
monthly productivity stipend
team off-sites

Fulltime

Senior MLOps Engineer - Data Ingestion - Paris

We are looking for a Senior MLOps Engineer to join the Panda Team (Data & ML Ope...

Location

France , Paris

Salary:

Not provided

Doctolib

Expiration Date

Until further notice

Requirements

You have at least 7+ years as an MLOps Engineer or ML Platform Engineer with proven production model lifecycle management experience
You have expert-level experience with ML orchestration tools (MLflow, Braintrust, or similar) for batch processing and inference pipelines
You have a strong Site Reliability Engineering (SRE) foundation with focus on operations excellence, reliability, and observability
You have expertise in Python for automation and ML pipeline scripting
You have strong proficiency with infrastructure-as-code tools such as Terraform and container orchestration (Kubernetes)
You have experience with model evaluation frameworks and golden dataset management
You have a solid understanding of cloud infrastructure (preferably GCP, AWS, or Azure)
You have excellent problem-solving skills with focus on identifying and resolving infrastructure bottlenecks
You are fluent in English

Job Responsibility

Design and implement end-to-end ML model pipelines in production (LLM and custom models) with robust deployment, evaluation, and monitoring frameworks
Own data pseudo-anonymization architecture within ingestion services, converting Tier 0 (personal identifiers) to Tier 1 (anonymized data) while ensuring data quality and model performance
Build and maintain secure data export services with ML-based threat detection to prevent attack vectors (SQL injection, etc.) using adaptive models rather than manual rules
Manage golden datasets and implement production model evaluation frameworks to ensure anonymization quality and system reliability
Build and maintain data pipelines that efficiently extract, transform, and load data from various sources, handling multiple data formats (text, images, audio, video)
Implement automation and orchestration tools using ML orchestration platforms (MLflow, Braintrust, or similar) to streamline infrastructure provisioning and reduce manual effort
Monitor data and ML platforms for performance, reliability, and security
identify and troubleshoot issues proactively
Mentor team members on MLOps expertise and best practices to reduce knowledge silos and build organizational capability

What we offer

Free comprehensive health insurance for you and your children
25 days of paid vacation per year, plus up to 14 days of RTT
Free mental health and coaching services through our partner Moka.care
Work from abroad for up to 10 days per year thanks to our flexibility days policy
Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib
A subsidy from the work council to refund part of the membership to a sport club or a creative class
50% reimbursement of your public transport subscription
Parent Care Program: receive one additional month of leave on top of the legal parental leave
For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
Relocation support in case of international mobility

Fulltime

Cloud Data Engineer

We are looking for an experienced Cloud Data Engineer with strong expertise in b...

Location

India , Bangalore South

Salary:

Not provided

Wissen

Expiration Date

Until further notice

Requirements

Strong experience with AWS (ETL, data services)
Expertise with structured and unstructured (OCR) data
Knowledge of streaming data tools
Proficiency in SQL
Strong Python programming skills
Hands-on experience with PySpark
Experience with Airflow for workflow orchestration
Excellent communication skills

Job Responsibility

Design and develop scalable ETL pipelines using AWS cloud services
Build data ingestion frameworks for structured and unstructured (OCR-based) data
Develop and optimize SQL queries and data transformation logic
Orchestrate workflows using Airflow
Develop big data solutions using Python and PySpark
Build and maintain streaming data pipelines
Ensure data quality, performance, and reliability
Collaborate with cross-functional teams to gather requirements
Document architecture, workflows, and processes

Fulltime

Cloud Data Engineer

Hunkemöller is looking for a Cloud Data Engineer to take a core role in our digi...

Location

Netherlands , Hilversum

Salary:

Not provided

Hunkemöller

Expiration Date

Until further notice

Requirements

3 to 5 years of hands-on experience in data engineering, with a strong focus on data integration, APIs, and pipeline architecture
A strong desire to learn quickly and adapt to new technologies
Strong Python programming skills with a proven track record of building custom API extractors, handling pagination, rate limiting, and working with REST/GraphQL endpoints
Hands-on experience with Google Cloud Platform's ecosystem, specifically Cloud Composer, Dataflow, Cloud Run, and Firestore
Proficient in writing clean, well-documented, and tested code (e.g., pytest), with strong experience using Git, Docker, and CI/CD pipelines
Excellent written and verbal English communication skills

Job Responsibility

Build Ingestion Pipelines: Design, develop, and deploy robust data ingestion pipelines from various third-party APIs, webhooks, and source systems into Google Cloud
AI-Augmented Engineering: Actively leverage advanced AI coding assistants to accelerate pipeline development, generate boilerplate API connection code, debug complex scripts, and automate repetitive tasks
GCP Infrastructure & Orchestration: Build and manage data workflows using Cloud Composer (Airflow), and leverage Cloud Run and Dataflow for scalable, containerized data processing
Drive Reverse ETL: Architect and maintain the data pipelines that push refined data from BigQuery back into our operational platforms (marketing tools, CRM, etc.) to drive business action
Manage Operational Databases: Utilize Firestore and other NoSQL/relational databases to support operational data needs and microservices
Collaborate and Learn: Partner with our data modeling specialists to ensure smooth handoffs between ingestion and transformation
Participate in code reviews and continuously share new engineering best practices

Fulltime

Senior Google Cloud Data Engineer

We are seeking a Senior Google Cloud Data Engineer with a strong background in d...

Location

United States , New York City

Salary:

170000.00 - 240000.00 USD / Year

Valtech

Expiration Date

Until further notice

Requirements

Candidates must be legally authorized to work in the United States and must hold either U.S. citizenship or lawful permanent resident (Green Card) status.
Strong expertise in Python - for building data pipelines, processing tasks, and automation
SQL - Advanced SQL capabilities (nested fields, analytic functions)
Hands-on experience with Core GCP Data Stack: BigQuery: Expert-level SQL, performance tuning, and specifically BigQuery ML (BQML) for implementing logistic regression or k-means clustering on data in-place
Dataflow (Apache Beam): Strong proficiency in writing data pipelines (Python or Java) handling windowing, watermarks, and triggers for streaming data
Pub/Sub: Experience with event-driven architecture and message queuing
Hands-on experience with Visualization & BI: Looker Core – Advanced proficiency in LookML (not just drag-and-drop), including derived tables, explores, and Liquid syntax
Semantic Modeling: Develop robust LookML models to create a trusted layer of data governance, ensuring metrics are consistent across the organization
Dashboard Creation: Design intuitive, high-impact dashboards in Looker for two distinct audiences: Operational teams (real-time threat monitoring) and Executives (risk posture and compliance reporting
Familiarity with Alerting Frameworks

Job Responsibility

Drive projects independently, ensuring timely delivery and high-quality outcomes
Lead and execute projects related to cybersecurity, including anomaly detection in data to identify potential security threats
Develop dashboards and visuals for operational and executive use, leveraging Looker Core and BigQuery
Implement alerting systems to proactively detect anomalies and security issues
Design and implement data ingestion and transformation pipelines leveraging Google Cloud Dataflow / Apache Beam and BigQuery for analytics and large-scale querying
Implement Machine Learning models (specifically leveraging BigQuery ML or Vertex AI) to identify statistical outliers, potential intrusions, and fraudulent patterns in network traffic
Build fault-tolerant, self-healing, adaptive, and highly accurate streaming and batch data pipelines on GCP
Perform data manipulation and analysis to support threat detection and reporting
Collaborate closely with cross-functional teams to ensure data quality, availability, and reliability across cloud environments
Develop and maintain technical documentation for all assigned systems and projects

What we offer

Flexibility, with remote and hybrid work options (country-dependent)
Career advancement, with international mobility and professional development programs
Learning and development, with access to cutting-edge tools, training and industry experts
Medical, dental, and vision insurance for you and your family, plus employer contributions to Health Savings Accounts

Fulltime

Select Country

Cloud Data Ingestion Engineer

Job Description

Job Responsibility

Requirements

Looking for more opportunities?