KDN-Solaris
KDN Solaris is a data modernization initiative focused on building a centralized enterprise data platform using Microsoft Fabric. The project integrates data from multiple source systems across HR, Finance, and Headcount domains into a unified Lakehouse architecture. By leveraging OneLake and Fabric, the platform provides a single source of truth for enterprise data, enabling efficient data management, governance, and seamless reporting through Power BI.
Case Study-1
Automated Real Estate Data Ingestion and File Lifecycle Management:-
This pipeline orchestrates an automated data movement workflow to process real estate metrics for Grandeur Property. First, the get_data activity extracts property listings, enquiries, and scheduling metrics from raw CSV files located in a landing folder, appends a dynamic Ingestion_timestamp using @utcNow(), and overwrites the structured data directly into a Lakehouse Delta table (Grandeure_property). Once ingestion completes successfully, a second copy activity safeguards the raw source files by transferring them from the active landing directory into a dedicated Archive folder as .txt files. Finally, a downstream Delete activity automatically purges the original files from the landing directory to prevent data duplication in future pipeline runs and maintain a clean file infrastructure.
Case Study-2
This pipeline orchestrates a metadata-driven incremental ETL workflow to process updates for a dynamic shipping log system. The process initiates with the Lookup1 activity, which queries a central control table (watermark_table) to retrieve the last successfully processed timestamp value. Using this value, the incremental_data copy activity filters raw JSON source files within the Case_Study_2_Tarun landing directory by enforcing a strict time window bounded between the retrieved watermark and the current @pipeline().TriggerTime. Only the newly modified files within this window are extracted and cleanly appended to the target Delta Lake destination table (shipping_log). Upon successful data ingestion, the downstream Notebook_test activity executes to post-process the data and update the watermark log table using the runtime execution timestamp, ensuring seamless tracking for subsequent pipeline runs.
Case Study-4
