Loading. Please Wait....
Sr. Data Engineer
Gurgaon, Tech | 1+ Year
Roles and Responsibilities
Tests, monitors, manage and validate data warehouse activity including data extraction, transformation, movement, loading, cleansing, and updating processes.
Building and maintaining optimized and highly available data pipelines that facilitate deeper analysis and reporting.
Collaborate with stakeholders to understand, define and document business applications, data sources/relationships, and needs (metrics, dimensions, charts and dashboards).
Analyze and Identify new data sources as needed per stakeholder requirements.
Working closely with analytics team and other business stakeholders to create data processing frameworks that help the business make sense of growing data.
Manage and monitor existing data pipelines and work with other team members to ensure best practices for ETL and data warehousing.
Work on optimization of data storage and processing infrastructure to reduce costs.
Facilitate integration of different BI tools as needed by business stakeholders.
Assist in the design and documentation of data infrastructure and technologies.
Proficient in languages: Python.
Minimum 2+ years’ experience as Data Engineer.
Experience in AWS Stack - Glue, Athena, Quick sight, RDS, Redshift, Kafka, pySparkExperience setting up data pipelines, archiving data, data lakes.
Bachelors or Master’s degree in computer science/Maths/statistics or related field.
Expertise with designing complex Data Models and Data Engineering solutions.
Write complex SQL for processing raw data, data transformations as well as data validations.
Experience with infrastructure as code using shell scripting.
Hands on experience in working with big data on AWS environments including cleaning/transforming/cataloging/mapping etc.
Experience in setup of data warehouse using Amazon Redshift, creating Redshift clusters and perform data analysis queries.
Experience in ETL and data modeling on AWS ecosystem components - AWS Glue, Redshift, DynamoDB.
Familiarity with AWS data migration tools such as AWS DMS, Amazon EMR, and AWS Data Pipeline.
Certifications on AWS will be an added plus.