Data Engineer - II

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using tools like Airflow, Airbyte and PySpark
• Collaborate with software engineers and analysts to ensure data availability and integrity for various applications
• Design and implement robust data pipelines to extract, transform, and load (ETL) data from various sources
• Utilize Airflow for orchestrating complex workflows and managing data pipelines
• Implement batch processing techniques using Airflow/PySpark to handle large volumes of data efficiently
• Develop ELT processes to optimize data extraction and transformation within the target data warehouse
• Leverage AWS services (e.g., S3, RDS, Lambda) for data storage, processing, and orchestration
• Ensure data security, reliability, and performance when utilizing AWS resources
• Work closely with developers, analysts, and other stakeholders to understand data requirements and provide the necessary data infrastructure
• Assist in troubleshooting and optimizing existing data workflows and queries
• Create and maintain documentation for data pipelines, workflows, and processes
• Adhere to best practices in data engineering, including version control and monitoring

• Bachelor’s degree in computer science, Engineering, Mathematics, a related field, or equivalent practical experience
• 3-5 years of experience in data engineering or a similar mid-level role
• Proficiency in Python and SQL
• experience with Java is a plus
• Hands-on experience with AWS, Airbyte, DBT, PostgreSQL, MongoDB, Airflow, and Spark
• Familiarity with data storage solutions such as PostgreSQL, MongoDB
• Experience with BigQuery (setup, management and scaling)
• Strong understanding of data modeling, ETL/ELT processes, and database systems
• Experience with data extraction, batch processing and data warehousing
• Excellent problem-solving skills and a keen attention to detail
• Ability to work effectively in a collaborative team environment with software engineers and analysts

experience with Java is a plus

• Competitive salary and benefits package
• Comprehensive Health Care benefits (best in the country, includes IPD+OPD, covers Employee, Spouse and two children)
• Growth and advancement opportunities within a rapidly expanding company