Why the export is plain Parquet

Modified on Fri, 21 Aug at 1:22 PM

ILAP Analytics writes its export as plain Parquet files, not as Delta tables. This comes up as soon as anyone shortcuts the container into a lakehouse, because Delta would have made the last step shorter. This article explains what the choice costs you, why it is the way it is, and what would have to change for Delta to make sense. It is for data engineers evaluating the export, and assumes you know what a Delta table is and how it differs from a folder of Parquet files.

What it means in practice

A Table-section shortcut will not work. In Microsoft Fabric, a shortcut placed in a Lakehouse's Tables section must point at a Delta table. The export is plain Parquet, so the shortcut goes in Files/ instead, and something has to materialise Delta tables into Tables before a semantic model can read them.

That is why the Fabric path involves a notebook at all. The notebook is not there for convenience or for transformation — it is structurally required by this choice. See Reading Parquet from Fabric for what it does.

Nothing else about consumption changes. Any engine that reads Parquet reads the export directly: Spark, DuckDB, Polars, pandas, Synapse, Databricks, a Fabric pipeline. Delta would have removed one step in one product, not opened up a class of consumer that is otherwise shut out.

Why not Delta

One Delta table is one schema, and a snapshot is not one schema

A planning snapshot folder holds fourteen entities and a reporting folder ten — activities, resources, calendars, structures, profiles and the rest, each with its own columns. A Delta table has a single schema and its own transaction log, so "export as Delta" is not a matter of swapping the writer. The layout would have to become one Delta table per entity:

rs=1042/planning/Live_20260601000000/activity/_delta_log/…
                                     activity/part-0000-….parquet
                                     resource/_delta_log/…
                                     resource/part-0000-….parquet
                                     …

That is a change to the shape of the archive, and it reaches every piece of code that touches a path: the writers, the readers that reconstruct data after a purge, the validator that re-verifies stored bytes, and the lifecycle jobs that delete superseded snapshots. The real cost is the layout change, not the writer rewrite.

Consolidating snapshots into one table would defeat the point

The obvious Delta design — one evolving table per entity, with each import appending or merging — is the one thing the archive must not do. The export exists so that successive generations of a plan stay separate and comparable: the schedule as it stood in January, and again in March, and again in June, each intact after the source rows have been purged from SQL. Merging them into one table with time-travel history trades that structure for a version log, and a version log answers "what did this table look like at commit 47", not "what did this plan look like at the March cut-off".

Keeping snapshots as separate folders keyed on (reportScheduleId, revisionType, cutoffToken) makes the generation boundary the primary structure rather than something recoverable from metadata.

Delta's log does not replace the fingerprints

Delta records file sizes and column statistics. It does not hash file contents. The export's _complete.json records a SHA-256 and a row count for every file, so a validation job can re-download the stored bytes and prove they are unchanged.

For an archive holding the only surviving copy of purged data, that is not an optional extra — detecting silent corruption or alteration is a large part of why the archive is trusted at all. So the fingerprint file would survive a move to Delta regardless, and Delta would not be replacing it.

The problems Delta would solve are already solved

Delta's transaction log would subsume the hand-rolled commit marker and the write lease that keeps two exports from interleaving. That is a genuine simplification of the write path — but it is a simplification of code that exists, works and is tested, bought at the price of the layout change above. It is not a reason on its own.

What would make Delta worth doing

The case for Delta is not "it would shorten the Fabric example". It is "customers want to consume this from their own lakehouse, and plain Parquet is the friction that stops them". That is a product question about how the export is actually used, and it deserves to be decided on evidence rather than on the convenience of any one demo.

If it is ever taken up, the shape is reasonably clear: one Delta table per entity inside each snapshot folder, keeping the snapshot as the unit of generation, with _complete.json retained for content verification. A Table-section shortcut would then work directly and the notebook would no longer be required. Until then, the notebook is one cell of boilerplate and the export stays readable by everything.