CrawlJobs Logo

Data Architect/Databricks Consultant

genzeon.com Logo

Genzeon

Location Icon

Location:
India , Hyderabad

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

We are seeking a specialized Databricks Architect with deep expertise in cost optimization and migration strategies, particularly focused on transitioning away from Databricks platforms. The ideal candidate will have extensive experience in Spark cluster solutions and a proven track record of reducing Databricks operational costs while architecting successful migration paths to alternative platforms.

Job Responsibility:

  • Conduct comprehensive cost analysis and auditing of existing Databricks deployments across multiple workspaces
  • Develop and implement aggressive cost reduction strategies targeting 30-50% savings through cluster optimization
  • Design and deploy automated cost monitoring solutions with real-time alerts and budget controls
  • Optimize cluster configurations, auto-scaling policies, and job scheduling to minimize compute costs
  • Implement spot instance strategies and preemptible VM usage for non-critical workloads
  • Establish cost allocation frameworks and implement chargeback mechanisms for business unit accountability
  • Create cost governance policies and developer guidelines to prevent cost overruns
  • Analyze and optimize storage costs including Delta Lake table optimization and data lifecycle management
  • Lead strategic initiatives to migrate workloads away from Databricks to cost-effective alternatives
  • Assess existing Databricks implementations and create detailed migration roadmaps to target platforms
  • Design migration architectures for transitioning to open-source Spark on Kubernetes, EMR, or other platforms
  • Develop automated migration tools and frameworks to minimize business disruption
  • Create comprehensive migration strategies including data export, job conversion, and dependency mapping
  • Establish parallel running environments to ensure zero-downtime migrations
  • Lead post-migration validation and performance benchmarking against original Databricks solutions
  • Document lessons learned and create reusable migration playbooks for future projects
  • Design high-performance, cost-optimized Spark cluster architectures outside of Databricks ecosystem
  • Implement custom Spark solutions on Kubernetes, YARN, and standalone cluster managers
  • Optimize Spark job performance through advanced tuning of memory management, serialization, and parallelism
  • Develop custom Spark operators and applications for specialized business use cases
  • Troubleshoot complex Spark performance bottlenecks and implement optimization strategies
  • Create cluster auto-scaling solutions and dynamic resource allocation frameworks
  • Design fault-tolerant Spark architectures with disaster recovery and high availability
  • Implement monitoring and alerting for Spark cluster health and job performance metrics
  • Collaborate with finance teams to develop multi-year cost reduction roadmaps
  • Evaluate and recommend alternative platforms based on cost-benefit analysis
  • Create business cases for migration projects with detailed ROI calculations
  • Establish technical debt reduction strategies related to Databricks dependencies
  • Partner with procurement teams on contract negotiations and vendor management

Requirements:

  • 8+ years of experience in big data architecture with focus on cost optimization
  • 5+ years of hands-on Databricks experience with proven cost reduction achievements
  • Demonstrated experience architecting and executing complete platform migrations from Databricks to alternative solutions with successful outcomes
  • 6+ years of advanced Apache Spark development and cluster management experience
  • Track record of achieving significant cost savings (minimum 40%+) in cloud data platforms
  • Expert knowledge of Databricks pricing models, compute types, and cost drivers
  • Experience with FinOps practices and cloud cost management tools
  • Proven ability to implement automated cost controls and budget management systems
  • Knowledge of alternative platforms and their cost structures (EMR, HDInsight, GCP Dataproc, etc.)
  • Deep expertise in migrating complex data workloads between different Spark platforms
  • Advanced knowledge of Spark internals, catalyst optimizer, and performance tuning
  • Experience with Kubernetes-based Spark deployments and container orchestration
  • Proficiency in infrastructure-as-code for multi-cloud Spark cluster provisioning
  • Strong background in data pipeline migration and ETL/ELT conversion strategies
  • Expert-level proficiency in Scala, Python, and Java for Spark development
  • Advanced SQL skills and experience with multiple database technologies
  • Experience with open-source alternatives to Databricks (Apache Spark, Delta Lake OSS, MLflow OSS)
  • Knowledge of streaming platforms (Kafka, Kinesis, Pulsar) and real-time architectures
  • Proficiency with monitoring tools (Prometheus, Grafana, ELK stack)
  • Bachelor's degree in Computer Science, Engineering, Information Technology, or related field

Nice to have:

  • Databricks certifications combined with experience in competitive platforms
  • Cloud cost management certifications (AWS Cost Optimization, Azure Cost Management)
  • Experience with vendor negotiations and contract optimization
  • Background in building business cases for platform migrations
  • Knowledge of data governance during platform transitions
  • Experience with Apache Iceberg, Hudi, or other open table formats as Delta Lake alternatives
  • Master's degree preferred but not required with equivalent experience

Additional Information:

Job Posted:
December 08, 2025

Icon
Genzeon - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Data Architect/Databricks Consultant

New

Production Systems Engineer

Meta is seeking a Production Systems Engineer to join our Hardware Design and Re...
Location
Location
Ireland , Dublin
Salary
Salary:
Not provided
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • An engineering degree is typical, or related technical discipline, or equivalent work experience
  • 4+ years experience coding in a higher-level language (Python, PHP, Java, Go, Rust, C++)
  • Experience building, maintaining and debugging production services or platforms - usually (but not necessarily) in a linux/unix environment
  • Knowledge of server architecture and components across Compute/Storage/AI Systems/Networking
Job Responsibility
Job Responsibility
  • Build and develop tooling solutions to automate business critical processes in service of managing the health of the Meta production hardware fleet
  • Troubleshoot, diagnose and root cause system failures, working with key partners to identify and deliver solutions
  • Proactively identify opportunities to fix or enhance tooling, hardware and processes
  • Build subject matter expertise in one or more of the specialist areas covered by the RTP (Release To Production) team in Dublin
  • Scientific approach to troubleshooting, root-cause analysis and investigation
Read More
Arrow Right
New

Regional Environmental Health and Safety Manager

Meta is seeking a hands on and experienced EHS Manager to lead the implementatio...
Location
Location
India , Bangalore
Salary
Salary:
Not provided
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Occupational Safety, Environmental Science, Engineering, or a related field
  • Minimum 10+ years of experience in EHS, with at least 5 years in a leadership or management role
  • Experience with regulatory requirements in India
  • Experience in semiconductor, chemical processing, life sciences, pharmaceutical, or manufacturing environments preferred
  • Experience of communication skills, with the ability to engage broad audiences
  • Ability to travel up to 25%
  • Languages (English + local languages)
Job Responsibility
Job Responsibility
  • Oversee and conduct site assessments, job and task risk assessments, and lead incident investigations to ensure compliance and continuous improvement
  • Collaborate with subject matter experts in areas such as lasers, hazardous chemicals, industrial hygiene, robotics, and occupational safety
  • Partner with Safety Engineering for the successful handover of new equipment and facilities
  • Ensure compliance with all applicable federal, state, and local EHS laws, regulations, and permits
  • Develop, maintain, and execute emergency response plans, serving as an emergency response coordinator when needed
  • Drive continuous improvement initiatives focused on learning from incidents and enhancing safety
  • Maintain legal registers, risk registers, and environmental impact assessments, ensuring critical controls are effective and communicated
  • Provide meaningful metrics and reporting to support decision-making by leadership and stakeholders
  • Lead risk management efforts including hazard identification, risk assessments, incident investigations, and corrective actions
  • Champion continuous improvement through audits, inspections, data analysis, and implementation of safety and environmental solutions
Read More
Arrow Right
New

Audio Systems Integration Engineer

Meta's Wearables team brings together industry leading experts to develop and sh...
Location
Location
United States , Redmond
Salary
Salary:
173000.00 - 245000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • BS degree in EE, Computer Science or equivalent experience
  • 8+ years of experience defining, prototyping, and demonstrating compelling audio experiences with end-to-end audio systems including acoustics, transducers, digital interfaces, audio ICs, and DSPs
  • 8+ years experience defining, tuning, and testing efficient real-time audio algorithms
  • 8+ years experience defining and executing objective and subjective test methodologies for consumer electronics devices and audio features such as voice communication, voice assistant, music playback, video capture
  • Experience with audio test systems such as SoundCheck, Audio Precision, or HEAD Acoustics ACQUA
  • Familiarity with typical DSP building blocks such as digital biquad filters, IIR, FIR, AGC, wide-band/multi-band DRC, loudspeaker thermal and excursion protection
  • Experience of effectively communicating new concepts and ideas to a broad range of partners and stakeholders, on a regular basis
Job Responsibility
Job Responsibility
  • Optimize and Debug Audio Systems
  • Analyze, define, tune, and debug real-time audio capture and render systems for forward-looking AR/VR and smart glasses products
  • Develop Measurement and Tuning Tools: Specify, develop, and document automated measurement tools and software modifications to enable efficient audio algorithm tuning and verification
  • Define KPIs and Audio performance
  • Use domain knowledge to define functional and experiential audio key performance indicators
  • Ensure the highest quality for user experience are met by defining and coordinating user experiments, for algorithm and system validation
  • Collaborate Cross-Functionally
  • Work closely with software, hardware, research, UXR, and product management teams to drive innovative solutions and best practices for audio systems
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right
New

Facility Technical Manager

Meta is seeking a data center Facility Technical Manager to join our Data Center...
Location
Location
United States , Newton County
Salary
Salary:
168000.00 - 234000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 10+ years of technical experience in Electrical, Mechanical, or Controls systems or a four-year technical degree such as engineering with 6+ years experience
  • 5+ years of experience in building, managing, and coordinating technical teams
  • General knowledge and understanding of applicable regulatory codes and standards
  • Experience communicating and presenting technical material effectively
  • Technical knowledge and experience with complex systems. Examples may include any of: Building Electrical systems, Electrical Distribution, Emergency Generators, Uninterruptible Power Supplies (UPS), High/Medium/Low voltage applications. Mechanical systems, including HVAC, Plumbing, Critical Water systems, Fire/Life Safety Protection Systems. Building Automation systems, PLC systems, DDC systems, and/or other automation/control systems
  • Experience troubleshooting complex problems and addressing root causes by analyzing data
  • Proven time management, organization, and prioritization skills
Job Responsibility
Job Responsibility
  • Lead a team consisting of Subject Matter Experts in the areas of: Mechanical, Electrical, Controls, Generator along with Environmental, Health, and Safety, Project Management, and Operations/Maintenance team members
  • Lead the team in providing technical solutions to operational challenges at the data center campus to include unexpected system responses, design issues, and response to equipment failures
  • Lead the team in providing maintenance planning, process workflows, document management, and safety programs to the Critical Operations teams
  • Report directly to the Data Center Campus Facility Manager (DCCFM) and provide direction on technical functions affecting physical infrastructure required throughout the campus. Establish team goals and prioritize resources to support the priorities of the Facility Operations organization
  • Provide mentorship and people management in accordance with company values including supporting career growth and providing performance feedback
  • Work with stakeholders to develop and implement operational improvement projects aligned with budget priorities
  • Work cross functionally with the Critical Operations Managers and Facility Support Lead to budget and prioritize work across the campus
  • Support and ensure team members collaborate with other cross functional teams to investigate technical issues and develop solutions throughout the lifecycle of the data center from startup through commissioning, modernization, and obsolescence
  • Provide leadership and support for investigating root causes of unplanned operational events. Coordinate, improve, and train on all programs related to maintenance planning and process development for the facility operations team
  • Collaborate with cross functional teams to provide support and technical expertise on facility retrofit and modernization projects
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right
New

Research Scientist Intern, Multimodal Contextual AI

At Reality Labs, our team brings novel experiences to life on Meta’s AR devices....
Location
Location
United States , Sunnyvale
Salary
Salary:
7313.00 - 12134.00 USD / Month
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Currently has, or is in the process of obtaining a PhD in Computer Science, Electrical Engineering, or a related field
  • Programming and simulation experience with languages such as C/C++ and Python
  • Experience with computer architecture and HW/SW co-design and co-optimization
  • Must obtain work authorization in the country of employment at the time of hire, and maintain on-going work authorization during employment
Job Responsibility
Job Responsibility
  • Build and characterize experimental HW+SW systems on AR devices and device prototypes
  • Develop embedded firmware and software in RTOS and mobile operating systems, e.g. AOSP
  • Collaborate with other researchers and engineers across various disciplines
Read More
Arrow Right
New

3IC

Step into a key leadership role as a 3IC at JD Sports! You'll lead by example, c...
Location
Location
Australia
Salary
Salary:
Not provided
jd-sports.com.au Logo
JD Sports Australia & New Zealand
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Experience in a customer-facing service role
  • Previous retail experience in a key holder or entry level leadership role
  • Results driven, with a demonstrated experience in leading teams to successfully deliver results and KPIs
  • A positive attitude, enabling you to motivate others and maintain a high-performing team culture
  • Collaborator with strong communication skills
  • Passionate to learn with a growth mindset
  • Loves sports fashion brands and is enthusiastic about sharing your knowledge with customers to build brand loyalty
  • Ability to work efficiently in a fast-paced team environment
Job Responsibility
Job Responsibility
  • Model great customer service, every customer, every time
  • Visibly driving service outcomes by coaching team to deliver exceptional service and a 'service over task' mentality
  • Support the Store Manager and Assistant Store Manager to drive sales and KPI performance
  • Utilise financial reporting to identify trade trends and opportunities and implement action plans
  • Assist the Management Team to complete weekly visual merchandising direction & promotional launches
  • Assist in recruiting, inducting and developing team members
  • Uphold company retail standards and deliver visual excellence
  • Work with BOH team to ensure product availability and operational excellence
What we offer
What we offer
  • Work in a dynamic business and engage with some of the best product and biggest sports fashion brands in the market
  • Training and development opportunities to kickstart, evolve and shape your career
  • Opportunity to progress across all areas of the business in a wide array of roles including Retail, Support Office, and our Distribution Centre
  • Access to our attractive staff discount! This can also be used for friends and family
  • Access to our Employee Assistance Program & Mental Health champions
  • One month paid parental leave for full-time employees
  • Fulltime
Read More
Arrow Right
New

Footwear Department Manager

Step into leadership as a Department Manager at JD Sports! Lead your team, deliv...
Location
Location
Australia , Mount Gravatt
Salary
Salary:
Not provided
jd-sports.com.au Logo
JD Sports Australia & New Zealand
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 2-3 years leadership experience within retail or like industry
  • Passionate about delivering an amazing service experience for customers
  • Positive attitude, with capacity to create a motivating environment for your team
  • Proven track record in driving and exceeding set KPI's and communicate well with all stakeholders
  • Excellent time management skills to ensure that daily priorities are executed in a timely manner
  • Thrives in a fast paced, ever-changing environment
  • Openly shares knowledge with the wider team to achieve common goals
  • Desire to develop yourself
  • your career and learn new skills
  • Strong interpersonal and communication skills, someone who thrives in social situations through engagement with customers and team members
Job Responsibility
Job Responsibility
  • Model exceptional customer service, every customer, every time
  • Coach team to deliver exceptional service and a ‘service over task’ environment
  • Actively gain, maintain and share product & brand knowledge for your department
  • Support management team to improve financial performance within your department
  • Work with back of house team to ensure product availability
  • Uphold company standards and deliver store visual excellence
What we offer
What we offer
  • Work in a dynamic business and engage with some of the best product and biggest sports fashion brands in the market
  • Training and development opportunities to kickstart, evolve and shape your career
  • Opportunity to progress across all areas of the business in a wide array of roles including Retail, Support Office, and our Distribution Centre
  • Access to our attractive staff discount! This can also be used for friends and family
  • Access to our Employee Assistance Program & Mental Health champions
  • One month paid parental leave for full-time employees
  • Fulltime
Read More
Arrow Right
New

Reliability Engineer

As a Reliability Engineer in Meta Reality Labs, you will take a critical role in...
Location
Location
United States , Sunnyvale
Salary
Salary:
144000.00 - 204000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Currently has, or is in the process of obtaining a Bachelor’s degree in Mechanical Engineering, Materials Engineering, Electrical Engineering, or a related technical field (or equivalent practical experience). Degree must be completed prior to joining Meta
  • At least 5 years of experience in reliability engineering or related field
  • Knowledge of mechanical design, material science, and failure mechanisms of consumer electronics
  • Experience in shipping products and familiarity with PDP cycles
  • Experience developing and performing reliability tests including (but not limited to) mechanical, shock, vibration, environmental, and characterization testing
  • Knowledge of industry test standards including ASTM, MIL, JEDEC, and IEC
  • Experience with failure analysis techniques and tools (but not limited to) optical microscopy, X-Ray, CT scan, SEM, EDX, and FTIR
  • Experience managing multiple tasks in a dynamic environment
Job Responsibility
Job Responsibility
  • Develop reliability requirements at system and module level for consumer facing hardware
  • Lead stress characterization of appropriate usage conditions
  • Lead and document FMEA (Failure Mode Effects Analysis)
  • Develop acceleration models and accelerated tests
  • Develop new reliability test specifications and procedures
  • Develop appropriate test plans
  • Apply design of experiments (DOE)
  • In-depth research into applicable failure mechanisms
  • Contribute to failure analysis and use correct methodology to identify root cause of failures
  • Manage reliability testing and analysis through product lifecycle
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right
Jobs by Category
Art and Entertainment Jobs Banking Jobs Call Center Jobs Construction Jobs Consulting Jobs Customer Service Jobs Education Jobs Energy Jobs Finance Jobs Gastronomy Jobs Health and Beauty Jobs Hospitality and Tourism Jobs Human Resources Jobs Insurance Jobs IT - Administration Jobs IT - Software Development Jobs Legal Jobs Logistics Jobs Manufacturing Jobs Marketing, Digital Marketing, SEO, SEM Jobs Media Jobs Nursing Jobs Office Administration Jobs Pharmacy Jobs Physical Work Jobs Public Relations Jobs Public Sector Jobs Purchasing Jobs Quality Control Jobs Real Estate Jobs Research and Development Jobs Sales Jobs
Job Positions Jobs by Country
United States Jobs United Kingdom Jobs India Jobs Canada Jobs Australia Jobs Germany Jobs Ireland Jobs Poland Jobs
Company
Blog About Us Contact
Follow Us
Facebook
X (Twitter)
LinkedIn
Instagram
CrawlJobs Group
ITFlashcards AllDevBlogs
© 2026 CrawlJobs Ltd. All Rights Reserved
Terms & Conditions Privacy Policy