Data Architect/Databricks Consultant Job at Genzeon (Hyderabad)

Job Description

We are seeking a specialized Databricks Architect with deep expertise in cost optimization and migration strategies, particularly focused on transitioning away from Databricks platforms. The ideal candidate will have extensive experience in Spark cluster solutions and a proven track record of reducing Databricks operational costs while architecting successful migration paths to alternative platforms.

Job Responsibility

Conduct comprehensive cost analysis and auditing of existing Databricks deployments across multiple workspaces
Develop and implement aggressive cost reduction strategies targeting 30-50% savings through cluster optimization
Design and deploy automated cost monitoring solutions with real-time alerts and budget controls
Optimize cluster configurations, auto-scaling policies, and job scheduling to minimize compute costs
Implement spot instance strategies and preemptible VM usage for non-critical workloads
Establish cost allocation frameworks and implement chargeback mechanisms for business unit accountability
Create cost governance policies and developer guidelines to prevent cost overruns
Analyze and optimize storage costs including Delta Lake table optimization and data lifecycle management
Lead strategic initiatives to migrate workloads away from Databricks to cost-effective alternatives
Assess existing Databricks implementations and create detailed migration roadmaps to target platforms
Design migration architectures for transitioning to open-source Spark on Kubernetes, EMR, or other platforms
Develop automated migration tools and frameworks to minimize business disruption
Create comprehensive migration strategies including data export, job conversion, and dependency mapping
Establish parallel running environments to ensure zero-downtime migrations
Lead post-migration validation and performance benchmarking against original Databricks solutions
Document lessons learned and create reusable migration playbooks for future projects
Design high-performance, cost-optimized Spark cluster architectures outside of Databricks ecosystem
Implement custom Spark solutions on Kubernetes, YARN, and standalone cluster managers
Optimize Spark job performance through advanced tuning of memory management, serialization, and parallelism
Develop custom Spark operators and applications for specialized business use cases
Troubleshoot complex Spark performance bottlenecks and implement optimization strategies
Create cluster auto-scaling solutions and dynamic resource allocation frameworks
Design fault-tolerant Spark architectures with disaster recovery and high availability
Implement monitoring and alerting for Spark cluster health and job performance metrics
Collaborate with finance teams to develop multi-year cost reduction roadmaps
Evaluate and recommend alternative platforms based on cost-benefit analysis
Create business cases for migration projects with detailed ROI calculations
Establish technical debt reduction strategies related to Databricks dependencies
Partner with procurement teams on contract negotiations and vendor management

Requirements

8+ years of experience in big data architecture with focus on cost optimization
5+ years of hands-on Databricks experience with proven cost reduction achievements
Demonstrated experience architecting and executing complete platform migrations from Databricks to alternative solutions with successful outcomes
6+ years of advanced Apache Spark development and cluster management experience
Track record of achieving significant cost savings (minimum 40%+) in cloud data platforms
Expert knowledge of Databricks pricing models, compute types, and cost drivers
Experience with FinOps practices and cloud cost management tools
Proven ability to implement automated cost controls and budget management systems
Knowledge of alternative platforms and their cost structures (EMR, HDInsight, GCP Dataproc, etc.)
Deep expertise in migrating complex data workloads between different Spark platforms
Advanced knowledge of Spark internals, catalyst optimizer, and performance tuning
Experience with Kubernetes-based Spark deployments and container orchestration
Proficiency in infrastructure-as-code for multi-cloud Spark cluster provisioning
Strong background in data pipeline migration and ETL/ELT conversion strategies
Expert-level proficiency in Scala, Python, and Java for Spark development
Advanced SQL skills and experience with multiple database technologies
Experience with open-source alternatives to Databricks (Apache Spark, Delta Lake OSS, MLflow OSS)
Knowledge of streaming platforms (Kafka, Kinesis, Pulsar) and real-time architectures
Proficiency with monitoring tools (Prometheus, Grafana, ELK stack)
Bachelor's degree in Computer Science, Engineering, Information Technology, or related field

Nice to have

Databricks certifications combined with experience in competitive platforms
Cloud cost management certifications (AWS Cost Optimization, Azure Cost Management)
Experience with vendor negotiations and contract optimization
Background in building business cases for platform migrations
Knowledge of data governance during platform transitions
Experience with Apache Iceberg, Hudi, or other open table formats as Delta Lake alternatives
Master's degree preferred but not required with equivalent experience

Genzeon - All Job Offers

Select Country

Data Architect/Databricks Consultant

Job Description

Job Responsibility

Requirements

Nice to have

Looking for more opportunities?

Data Architect/Databricks Consultant

IT Training Lead

K Kitchen Representative

K Kitchen Representative

Restaurant Assistant Manager

Plant Operator - Crushing and Screen

Graduate Student Instructors

Shift Supervisor

Shift Supervisor

Our AI answers in your language