Topic hub
Enterprise Data Platforms
Enterprise data platforms are where cost, trust and delivery speed get decided long before anyone sees a dashboard. These articles look at Azure and Databricks from the architecture and governance side: what the platform enforces, what your team still owns, and which fundamentals are worth understanding properly.
Azure · Databricks · architecture · data engineering · governance
Databricks Architecture & Governance
Fundamentals, Unity Catalog, pipelines and loading patterns, in reading order.
Databricks and the lakehouse
The platform, its governance layer and production pipelines.
- Fundamentals
Begin with Databricks: Lakehouse Fundamentals Explained
Why Databricks is better understood as an operating model for data and AI than as another analytics tool, from Delta tables to MLflow and streaming.
- Architecture
Unity Catalog: Overrated or Misunderstood?
What Unity Catalog should enforce as a governance control plane, and what remains outside its control boundary.
- Architecture
Databricks Pipelines: Beyond the Notebook
A notebook that works has proven the logic, not that the enterprise can depend on it: state, late data and transactional recovery.
Data engineering fundamentals
Loading patterns and file formats, explained accurately.
- Back2Front
Full Load vs Incremental Load: Azure Data Factory + Azure Databricks Quick Study Guide
Full load starts over; incremental load remembers where it left off. That memory makes it efficient and creates the need for stronger controls.
- Fundamentals
Apache Parquet Explained: Columnar Storage, Compression & Analytics
How columnar storage lets analytical engines read only the columns a query needs, and why that suits large datasets.
- Field notes
I Tried Data Loading Patterns in Microsoft Fabric Using Kaggle Data. Here's What Broke First.
Testing loading patterns on a Microsoft Fabric trial, and how quickly unclear patterns become trust and accountability problems.
- Guide
GDPR Compliance Guide for Data Engineers
What GDPR means for the people who build pipelines, data lakes and distributed systems.
Azure identity and application architecture
A four-part series on connecting an application to Azure without over-granting access.
- Series · 1 of 4
Part 1: Why I Wouldn't Build Microsoft Authentication Into My Web Application
Connecting a web app to Azure starts with the wrong question. The right one separates identity, authorization and product responsibility.
- Series · 2 of 4
Part 2: Azure Subscription, Entra ID, Tenant and App Registration: What Does Each One Actually Do?
Terms that sound like they overlap but each solve a distinct problem in the identity architecture.
- Series · 3 of 4
Part 3: Identity Is Not Authorization: API Permissions, Access Tokens and Azure RBAC Explained
A valid identity is not unlimited authorization. Each of these adds a distinct layer of control.
- Series · 4 of 4
Part 4: From Controlled Access to Visual Cloud Understanding
The series' identity and authorization patterns applied to a real product, APRIS AVA.
Field notes
Practical fixes from working on the platform.