Job Description:
Data Engineer - Data Integration, IBM Corporation, Armonk, NY and various unanticipated client sites throughout the US: Manage the end-to-end delivery of data migration projects, implementing ETL/ELT concepts and leveraging ETL tools such as Informatica and DataStage, and cloud platforms like Google Cloud. Design and build end-to-end data pipelines to extract, integrate, transform, and load data from diverse source systems into target environments such as databases, data warehouses, or data marts. Collaborate with clients to define data mapping and transformation rules, ensuring accurate application prior to loading. Normalize data and establish relational structures to support system migrations. Develop processes for data cleaning, filtering, aggregation, and augmentation to maintain data integrity. Implement validation checks and data quality controls to ensure accuracy and consistency across systems. Create, maintain, and optimize SQL procedures, functions, triggers, and ETL/ELT processes. Develop, debug, and maintain ETL jobs while applying query optimization techniques - such as indexing, clustering, partitioning, and use of analytical functions - to enhance performance on large datasets. Partner with data analysts, data scientists, and business stakeholders to understand requirements and ensure delivery of the right data. Capture fallouts and prepare reports using Excel, Power BI, Looker, Crystal Reports, etc. Perform root cause analysis and resolution. Monitor and maintain pipelines to ensure stability and efficiency of data pipelines through regular monitoring, troubleshooting, and performance optimization. Maintain thorough and up-to-date documentation of all data integration processes, pipelines, and architectures. Analyze current trends, tools, and technologies in data engineering and integration. Utilize: Google Cloud Platform (Google Big Query, Cloud Storage, Google Looker), Procedural language/Structured Query Language (PL/SQL), Informatica, DataStage, Data Integration, Data Warehousing, Database Design / Modelling, Data Visualization (Power BI/ Crystal reports).
