PrateekDubey

DATA ENGINEERING MANAGER/ PRINCIPAL DATA ENGINEER

• Managing a team of Data Engineers covering Data Engineering, DevOps, Architecture, Governance, Observability and Cloud.
• Working with FWD Insurance as a Principal Data Engineer to rollout LLM powered voice to voice conversational AI Avatar using Llama3, ElevenLabs, etc. Scaling LLM applications on EKS to serve 1000s of users across South East Asia.
• Working with Laconic as a Principal Data Engineer to rollout a scalable data stack using DBT, EKS, Spark, Airflow, Open Metadata.
• Working with Temasek as a Principal Data Engineer to migrate Informatica BDM to Snowflake to support T+1 trading transactions.

DATA ENGINEERING MANAGER/ PRINCIPAL DATA ENGINEER

• Worked with Dafiti (LATAM) on data strategy, restructuring, prioritisation, roadmap planning, resolving customer pain points, and adopting data mesh architecture by treating data as a product for business deliverables
• Managed a team of 8 Engineers based out of Vietnam (FTE) and India (Contractual) covering Data Engineering, DevOps, Solution Architecture, Data Governance, Observability and Cloud for GSF Data
• Principal Engineer/ Architect for Global Data Platform using OSS CNCF, Linux & AI, and ASF tech stack on Kubernetes (AWS EKS). GDP is a Central Data Platform for GFG, Zalora, The Iconic and Dafiti to run their workloads including Model Serving.
• Technical Leader for GDP (Infrastructure as a Service), DaaS (Data as a Service) projects. Worked closely with regional teams from Zalora (SEA), and The Iconic (ANZ) to deliver data and ML solutions for Pricing Engine, Recommendation, Personalisation.
• Responsible for setting quarterly OKR’s, weekly 1:1s, technical delivery of projects, setting up roadmap for Data Platform, etc.
• Participate in deep technical, architectural discussions about design and code review PRs.
• Helped Zalora, Iconic, Dafiti migrate their Seller Dashboards to GDP to build a subscription based data as a product.
• Led the efforts to shutdown Snowflake and Redshift Cloud DWHs to cut cost and migrate to Trino backed Lakehouse.
• Ingested GA3 (The Iconic, Dafiti) and Segments (Zalora) events into GDP for behavioural data recommendation.
• Improved GDP observability and availability using Loki, Thanos, Grafana, and Prometheus with Slack and Email alerts.
• Incorporated Git-Ops methodology, setup CI/CD processes using GitHub Actions, Argo CD, Ansible and Terraform.
• Setup Data Governance for our Iceberg Lakehouse backed by Trino using Apache Ranger and Datahub.
• Setup Data Quality checks by leveraging Great Expectations.
• Setup ML-Ops solution using Mlflow (Experimentation), Feast (Feature Store), KServe (Model Serving), Kong (API Gateway).

DATA ENGINEERING MANAGER/ LEAD DATA ENGINEER

• Managed and lead efforts in Data Engineering, DevOps, Solution Architecture, Data Governance, Observability and Cloud.
• Built Telco Cyber Data Lake & Platform using Hybrid Cloud for Cyber Analytics and Threat Intelligence.
• Lead Architect and Engineer for Telco CDL Kubernetes Platform running On-Premise using Rancher and on AWS using EKS.
• Processed TBs of data using Spark on K8s stored in Kafka, Ceph and S3 using our Data and ML Platform on AWS and On-Premise.
• Built Streaming & Batch data pipelines using Apache Kafka and Spark Streaming. Orchestrated data pipelines using Airflow.
• Setup requisite logging and monitoring for our system and services using Elasticsearch, FluentD, Grafana, and Prometheus.
• Setup up DevOps, CI/CD processes using Gitlab CI, Argo CD, Ansible and Terraform.
• Setup Data Governance for Kafka, Ceph, Hive, & S3 using Apache Atlas, Apache Ranger and Amundsen.
• Setup ML-Ops solution for Telco CDL by leveraging Mlflow to track Experiments, log metrics & parameters.

LEAD DATA & ML ENGINEER

• Technical lead for ML Infrastructure team. Responsible for looking after Cloud Architectural design solutions over AWS and Azure for Data Science. Involved in ML-Ops, DevOps, Cloud, Data Engineering, and Data Science.
• Led a team of ML Engineers to build, manage and support ML Platforms on AWS and Azure along with supporting teams in their ML projects by building jobs related to Data Engineering, Feature Engineering, Model Training, Model Deployment etc.
• Data Science PIC for vendor engagement with AWS, Microsoft and Databricks and also owner for all our DS Cloud Accounts.
• Implemented Data Governance and Infrastructure Governance on our ML Platforms on AWS and Azure.
• Worked with GrabFood and Azure CAT team to develop GrabFood restaurant behavioral recommendation system on Azure Cloud Platform thus increasing Click through Rate by 170% compared to existing recommendation systems at Grab.
• Developed a Multi-User JupyterHub Data Science ML Platform on AWS EKS for ML and Analytics.
• Used Kubeflow Pipelines to automate Machine Learning pipelines on AWS EKS.
• Led AI Adoption across the organization by pitching adoption of Databricks as a preferred ML Platform on Azure.
• Automated Cloud Infrastructure using Terraform and Azure ARM. Used Helm as a K8s package manager on EKS and AKS.
• Led the team to Architect and develop multiple solutions for our ML Platform like Spark UI Proxy, JupyterHub resource tracking, AWS monitoring, Datadog monitoring for EKS, Agartha – Docker build Automation using Git Pull Requests.
• Developed AWS Access Key Rotation process in Data Science using Lambda, SNS, SES, CloudWatch.
• Developed STS Token generation service using Flask, Nginx and Python and deployed on EKS.

SENIOR DATA ENGINEER

• Worked as part of Cloud Engineering and was responsible for migrating Athena SAS DWH onto AWS for Risk & Finance Solutions.
• Designed Data Lake on AWS which was utilized by Data Engineering & Analytics for ETL, Analytics and Machine Learning.
• Automated AWS Cloud Infrastructure using Terraform for Risk and Finance Solutions.
• Designed and developed a Python Parser to auto-convert HiveQL codes into equivalent PySpark (Spark SQL) jobs to leverage the Spark capabilities on AWS EMR, thus reducing conversion time by over 90%.
• Reduced job workflow creation time by 80% through an automated Oozie workflow creation framework.
• Designed services for seamless monitoring like monitor Active EMR Clusters and SageMaker instances running across all regions. Used Boto3 library and deployed solution on Lambda. Business notifications configured via SES and scheduled via CloudWatch.
• Enabled Amazon SageMaker for Risk Modelling (Machine Learning) to leverage Spark on SageMaker both locally and through remote EMR Cluster. Persisted SageMaker Jupyter Notebooks in S3 instead of local EBS volume.
• Installed and setup Apache Airflow to schedule our Core Spark ETL Workflows.
• Worked on a framework for orchestration & monitoring of our Core EMR Cluster using Lambda, CloudWatch, & SNS.

SENIOR DATA ENGINEER

• Worked as part of Decision Analytics team which was spearheading Big Data adoption for Barclays thus reducing operational cost and delivering better insights from data.
• Worked on migrating SAS Datawarehouse onto Big Data platform - Cloudera for Risk Solutions vertical of Barclays.
• Designed an On-Premise data lake using Cloudera CDH, which was utilized by Data Engineering & Analytics for ETL workloads.
• Developed ETL batch data pipelines by converting SAS to PySpark jobs.
• Developed Real-Time data pipelines using Spark streaming, PySpark and HBase.
• Reduced metadata migration time from Teradata to Hadoop by 95% through an automation suite created using UNIX.
• Reduced historical data migration time from SAS to Hadoop by 70% through an automation suite created using UNIX.
• Trained colleagues on Hadoop and Spark technology stack along with Unix & Shell scripting.

SENIOR DATA ENGINEER

• Worked on multiple End to End Big Data project deliverables involving data ingestion, ETL, designing a data lake, data migration, creating unit test plans, proof of concepts etc.
• Subject matter expert for Legacy Data warehouse running on Unix and SQL platform using IBM Informix database.
• Designed and developed HRBPO Attendance Tracking System using Hive and Hadoop used for tracking logging hours of CSR’s.
• Designed and developed an automation framework using Hive and Unix to capture the pre and post CDH upgrade checks, thus reducing the testing time from 5 business working days to an hour.
• Carried out POC’s on varied Hadoop technologies like HDFS Snapshots, High Availability of HDFS/Hive/YARN/HUE, NoSQL database - HBase, Active Directory implementation on Hadoop Servers, Hive on Spark, Scala on Spark etc.
• Supported BI Infrastructure team with CDH upgrades and setting up QC cluster for Big Data platform.
• Responsible for production ETL workload job monitoring using Control-M, ETL fixes and ETL enhancements raised by business stakeholders and users. Designed and developed ETL codes in PIG Latin, HiveQL, Unix and SQL.

Global Data Platform

Built GDP using K8s on AWS. Created End-to-End Data Platform to support GFG, Zalora and Iconic data workloads. GDP is used to support data ingestion from varied data sources for behavioural data recommendations, search and personalisation.

Telco Cyber Data Lake and Platform

Built Telco CDL and Platform using K8s. Used Rancher K8s On-Premise along with EKS to containerize applications and services. Processed TB’s of network logs from Singtel for Threat Intelligence and Cyber Analytics using Apache Spark.

AWS & Azure ML Platforms

Led a team of MLE’s and developed ML Platforms at Grab for Data Science on both AWS and Azure Cloud. Responsible for supporting ML platforms for all production workloads and managed DS Cloud accounts. Used AWS EKS and Azure Databricks for core capabilities. Used Spinnaker for Model Deployment on EKS and Kubeflow for Model Training.

Athena Re-Platforming – SAS Datawarehouse Migration to AWS

Worked on migrating Risk and Finance solutions from legacy DWH platform SAS to AWS using EMR, Glue, Athena, Quicksight and Sagemaker. All SAS codes were re-written in Pyspark and scheduled via Airflow on EMR Spark Clusters.

Barclays Bureau – SAS Datawarehouse Migration to Cloudera

Worked as part of EXL Barclays Decision Analytics team and helped Risk Solutions to move their legacy DWH platform from SAS to Cloudera Big Data Platform. All SAS codes were re-written in Pyspark and scheduled via Oozie on Cloudera.

Informix Shutdown – Legacy Datawarehouse Migration to Cloudera

Worked as a Hadoop Developer in LDW Migration team wherein migrated over 2000 codes written in Unix & SQL by rewriting them in Pig Latin and porting them over to Cloudera Big Data Platform.

University of Pune, Maharashtra, India

Army Public School, Dhaula Kuan, New Delhi, India

Army Public School, Delhi Cantt, New Delhi, India

AWS Certified Developer Associate