Senior Data Engineer / Data Platform Architect

• Co-architect modular, reusable, and secure data platform components supporting Data Mesh principles
• Design optimal data storage, processing, and retrieval strategies across hybrid cloud environments, with a focus on Apache Iceberg-based architectures
• Integrate federated querying frameworks using Starburst and Stargate, ensuring high-performance access across data sources
• Build and optimize complex batch and streaming pipelines for diverse data domains, with an emphasis on performance, fault-tolerance, and scalability
• Enable schema evolution and transactional consistency in Iceberg tables, leveraging catalog services and metadata stores (e.g., Hive MetaStore)
• Drive the adoption and implementation of Apache Iceberg, Iceberg Catalogs, Hive MetaStore, and Starburst to unify analytics across cloud and on-prem data assets
• Operationalize data product templates, ingestion frameworks, and reusable patterns for use across multiple data domains
• Architect and deploy data ingestion and processing components that span AWS cloud (S3, Lambda, Glue, Redshift, Athena) and on-prem environments
• Implement data movement, governance, and monitoring frameworks that operate transparently across hybrid infrastructure
• Embed data quality, lineage, and metadata capture into the platform by default, ensuring compliance and traceability
• Enforce role-based access controls, encryption standards, and audit mechanisms aligned with corporate policies
• Guide junior engineers and developers through technical reviews, architecture guidance, and solution design
• Establish coding standards, testing patterns, CI/CD pipelines, and promote a culture of engineering excellence

• Expert in Python, SQL, and scripting for automation and data engineering
• Familiarity with Scala or Java is a plus
• Strong experience with Apache Spark, Kafka, Flink, or other distributed data processing engines
• Advanced knowledge of Apache Iceberg, including partitioning strategies, schema evolution, compaction, and ACID support
• Hands-on with AWS cloud services—S3, Glue, Lambda, Redshift, Athena, and EMR
• Experience architecting solutions across hybrid environments, integrating cloud-native and legacy systems
• Proficient with Hive MetaStore, Iceberg Catalogs, and metadata management
• Strong implementation experience with Starburst and Stargate for federated SQL access across disparate systems
• Working knowledge of Terraform, CloudFormation, Docker, GitHub Actions, or similar CI/CD and IaC tools
• Familiarity with data lineage, cataloging, RBAC, encryption, and compliance standards (GDPR, CCPA, etc.)
• Architectural Thinking – Ability to break down complex data platform needs into modular, scalable components
• Leadership & Mentorship – Demonstrated experience in guiding junior team members, setting technical direction, and reviewing solutions
• Business Orientation – Understands data’s role in business processes and can balance technical rigor with practical implementation
• Collaboration – Comfortable working across multiple teams, geographies, and functions, including architects, engineers, product owners, and business users
• Problem Solving – Strong analytical skills and a solution-oriented mindset for complex data challenges
• 13+ years of experience in data engineering, big data platforms, and cloud-based infrastructure
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related technical discipline
• Proven track record in building production-grade data platforms using Iceberg, Spark, and AWS services
• Deep understanding of federated architectures, hybrid integration patterns, and metadata-driven pipelines

Familiarity with Scala or Java