Core Technical Skills
Cloud Platforms & Orchestration: Azure (ADF, ADLS Gen2, SHIR, Azure SQL, Blob Storage), Databricks
Big Data & Processing: PySpark, Spark SQL, Delta Lake, Parquet, Data Lakehouse Architecture
ETL/ELT Frameworks: Metadata-Driven Pipelines
Database & Migration: MySQL, Microsoft SQL Server, PostgreSQL, Schema Drift Management, Database Migration
Data Warehousing & BI: Delta/Curated Zones, Stored Procedures
Languages & Formats: Python (PySpark), SQL, JSON, CSV, Parquet.
Recent Project
ExperienceEnterprise Metadata-Driven Ingestion & Database Migration PlatformMetadata-Driven Pipeline Architecture:
Architected an end-to-end, metadata-driven ETL pipeline on Azure Data Factory to ingest multi-source data (CSV, Excel, JSON, REST APIs, and MySQL) into ADLS Gen2 without hardcoded logic.
4-Layer Pipeline Hierarchy:
Designed a modular pipeline hierarchy (Master rightarrow Source Type rightarrow Cloud Provider rightarrow File Type) using nested Lookup and Switch activities driven by SQL configurations to trigger downstream Databricks processing.
Database Migration & Secure Extraction:
Executed secure database migration strategies, implementing Self-Hosted Integration Runtime (SHIR) with ADF to seamlessly extract transactional data from MySQL databases into Azure Cloud infrastructure.