Parquet export

Modified on Fri, 21 Aug at 1:22 PM

ILAP Analytics can export planning and reporting data as Parquet files into a storage account you own, one folder per snapshot, so the data survives the retention purge that eventually removes it from the operational database. This article describes what those files are, how they are laid out, and the handful of rules a lake consumer has to get right. It is for data engineers and report developers building on the export from a lakehouse, and assumes familiarity with Parquet and with columnar file layouts. To consume the export from Microsoft Fabric, see Reading Parquet from Fabric.

What the export is for

The API serves the data that is currently in ILAP Analytics. The Parquet export serves the data that was. Each import writes a snapshot of the schedule as it stood at its cut-off, and that snapshot stays in the storage account after the source rows are purged from SQL. Successive snapshots of the same report schedule are therefore a history of how a plan moved — dates slipping, hours growing, baselines being re-set — which no query against live data can reconstruct.

Two properties follow from that purpose, and they shape everything below:

Snapshots are separate generations, never merged. Each is a self-contained folder. Nothing rewrites an earlier snapshot to reflect a later one.
The archive is a system of record. Every file's row count and SHA-256 are recorded when it is written, so the stored bytes can be re-validated later. Consumers should have read access and nothing more.

Availability

The export is off by default and is turned on with deployment parameters — a switch, the storage account it writes to, and the container name that lake consumers shortcut into. Ask whoever operates your ILAP Analytics deployment whether it is enabled, and for the container URL.

An authenticated caller can also check:

GET /api/Configuration/ParquetExportAvailability
Authorization: bearer <DataReader token>

{ "Available": true }

Available is true only when the feature is switched on and the API can actually reach the storage account, so a false here covers both "not enabled" and "misconfigured".

How the files are laid out

Everything sits under one container, in a fixed three-level layout:

<container>/
  rs=1042/
    planning/
      Live_20260601000000/
        _complete.json
        _manifest.json
        activity.parquet
        resource.parquet
        ... 12 more
      Baseline_20260112000000/
        ...
      Live_nocutoff_318/
        ...
    reporting/
      Live_20260601000000/
        reportActivity.parquet
        reportActivityPeriodized.parquet
        ... 8 more

rs=<reportScheduleId> is Hive-style, so Spark discovers it as a partition column without being told.
planning and reporting are the two datasets. Planning is the schedule as posted; reporting is the derived, denormalised reporting view.
<revisionType>_<cutoffToken> is not Hive-style and has to be parsed out of the folder name.

Reading a snapshot folder name

Split on the first underscore, not the last. Revision types — Live, Baseline, OriginalBaseline, Current, Revised — contain no underscore. The token after it can, so it is everything that follows the first underscore, however many more the name holds.

Live_nocutoff_318 is revision Live with the token nocutoff_318, which is one token meaning this snapshot has no cut-off — the 318 is the planning schedule id that keeps coexisting no-cut-off snapshots apart, not a cut-off. Split on the last underscore instead and you get a revision type of Live_nocutoff, which does not exist, and 318 read as a cut-off, which it is not.

The token is one of:

Token	Meaning
`yyyyMMddHHmmss`	The snapshot's cut-off date and time. Stored raw, never timezone-converted, so the folder key cannot drift with the host's DST. Sorts correctly as a string.
`nocutoff`	No cut-off. The state before actuals were tracked.
`nocutoff_<planningScheduleId>`	No cut-off, in a planning folder, where several no-cut-off snapshots of one revision type can coexist. The suffix is the planning schedule id that tells them apart — it carries no time meaning.

A no-cut-off snapshot is the earliest, not one outside time. Where a report schedule has both, the no-cut-off snapshot is the state before expended hours and progress were being reported, so it precedes every dated cut-off. This makes plain string sorting exactly backwards: 'n' sorts after '9', so nocutoff lands after every date when it belongs before them, and a naïve max() over the raw tokens returns the pre-tracking snapshot as the most recent one. Any comparison for recency needs an ordering that ranks no-cut-off first.

Note the asymmetry in the table above: a reporting folder keeps the bare nocutoff, because only the active revision owns reporting and there is at most one. So pairing a reporting folder with its planning sibling is not string equality — nocutoff pairs with nocutoff_318.

What a snapshot means

The cut-off is the time axis, and there is exactly one version of each revision type per cut-off. The folder key is (reportScheduleId, revisionType, cutoffToken); the export time is not part of it. Re-importing the same schedule, revision and cut-off is a correction that supersedes the previous one and overwrites in place — not a second generation.

Two consequences worth planning for:

A consumer's snapshot dimension has grain (reportScheduleId, revisionType, cutoffToken) exactly. No export-time column, and never more than one row per key. Two complete folders resolving to one key means corruption or a layout change, and is worth failing on rather than quietly picking one.
GeneratedAtUtc, recorded inside _manifest.json and _complete.json, is the only visible signal that a correction replaced a generation. A cut-off whose GeneratedAtUtc has moved since the last read has been re-exported.

Not every planning snapshot has a reporting sibling. Only the active revision carries reporting; a non-active revision is exported planning-only. A planning folder without a matching reporting folder is normal, not a partial export.

A schedule's active revision can change over its life, and the reporting folder is named for whichever revision was active. A reporting series pinned to a fixed revision type therefore truncates silently at the changeover. Key a reporting time series on the cut-off alone — there is exactly one reporting set per cut-off — and treat the revision type as a descriptive attribute of the snapshot.

Which files a snapshot holds

Planning — fourteen files, the schedule as posted:

File	Holds
`activity.parquet`	Activities: dates, durations, work hours, progress
`successor.parquet`	Activity-to-activity links and their lag
`resource.parquet`	Resources available to the schedule
`resourceAssignment.parquet`	Resources assigned to activities, with quantities
`resourceAvailability.parquet`	Windows and rates at which a resource is available
`calendar.parquet`	Working calendars
`calendarOperation.parquet`	One rule within a calendar — working or exception
`weeklyRepeatingPeriod.parquet`	A calendar rule that repeats on a weekday, with start and finish times
`connectedPeriod.parquet`	A calendar rule covering one continuous date range
`structure.parquet`	Breakdown structures — the WBS and other activity hierarchies
`structureElement.parquet`	Nodes within a structure, with their parent
`activityStructureElement.parquet`	Which activities hang off which structure elements
`profile.parquet`	Distribution profiles used to spread quantities over time
`profilePoint.parquet`	The `(x, y)` points that define a profile

Reporting — ten files, the derived reporting view:

File	Holds
`reportActivity.parquet`	One row per activity: dates, work hours and progress, with the reference revisions carried as columns
`reportActivityPeriodized.parquet`	One row per activity per point in time: the time-phased hours series
`reportSuccessor.parquet`	Links, in reporting form
`reportResource.parquet`	Resources, in reporting form
`reportResourceAssignment.parquet`	Resource assignments, in reporting form
`reportCalendar.parquet`	Calendars, in reporting form
`reportStructure.parquet`	Structures, in reporting form
`reportStructureElement.parquet`	Structure elements, in reporting form
`reportActivityStructureElement.parquet`	Activity-to-structure-element membership, in reporting form
`reportProfile.parquet`	Profiles, in reporting form

The two reporting files that carry the numbers

Almost every report is built from two files, and they are not interchangeable.

reportActivity.parquet carries the dates. Start and Finish, plus StartBaseline / FinishBaseline, StartBaselineOriginal / FinishBaselineOriginal, StartCurrent / FinishCurrent and StartRevised / FinishRevised, alongside ActualStart and ActualFinish. Work-hour totals appear here too — TotalWorkHours and its four reference variants — as do PlannedProgress, CurrentProgress and TotalFloatHours. Date drift is measured here and nowhere else.

reportActivityPeriodized.parquet carries hours only, one row per activity per PointInTime: PlannedHours, PlannedLoad, CalendarHours, RemainingHours and their Baseline, OriginalBaseline, Current and Revised variants, with EarnedHours and ActualHours as single series. It holds no activity dates at all. What it shows of a date movement is the consequence — the planned-hours curve shifting along the time axis — never the dates themselves. Sourcing a date question from this file yields a plausible curve that answers a different question.

Revisions are columns; cut-offs are rows. Both files denormalise the revision comparison into columns, so comparing Live against a baseline at one cut-off is a comparison between columns of a single snapshot — no join, no filter. Comparing across time means varying the cut-off. Varying both at once — Live at March against Baseline at June — is not a third analysis; it produces a plausible, wrong answer with no error to warn you.

Two caveats on those columns:

The prefix order differs between the two files, which will trip up measure authoring: reportActivity uses StartBaselineOriginal, while reportActivityPeriodized uses OriginalBaselinePlannedHours.
Ignore the Baseline-suffixed earned-hours columns. BaselineEarnedHours and EarnedWorkHoursAtCutoff*Baseline are still written, but progress is only ever reported against live scope, so they carry no meaning. Actuals — earned hours, expended hours, current progress — are valid for Live only. Estimates — dates, planned hours, planned load — are comparable across all five revisions.

`_complete.json` — the commit signal

_complete.json is written last, only after every file in the snapshot has been written and validated. Its presence is what distinguishes a finished snapshot from a partial write.

{
  "SchemaVersion": 1,
  "GeneratedAtUtc": "2026-06-01T02:14:07.113Z",
  "Files": [
    { "Path": "rs=1042/planning/Live_20260601000000/activity.parquet",
      "RowCount": 8412,
      "Sha256": "9f2b…" }
  ]
}

Three things about it are easy to get wrong:

It lives in the planning folder only. A reporting folder carries no marker of its own; its readability comes from its planning sibling.
Gate every read on it. A folder without the marker is a partial write, and a consumer that globs the container without checking will read torn snapshots.
It is deleted first when a snapshot is rewritten, deliberately, so a re-import in progress reads as incomplete rather than as a mixture of old and new files. A snapshot that is skipped as incomplete on one run and reappears on the next was mid-import, and that is working as intended. Worth logging rather than swallowing.

Files records each file's container-relative path, row count and SHA-256 as written, so the stored bytes can be re-downloaded and verified later. Tamper detection on what may be the only surviving copy of purged data is the point of recording them.

`_manifest.json` — and why column names are not identifiers

Alongside the fixed schema of each entity, a snapshot carries a column per applicable metadata field. _manifest.json describes them:

{
  "ReportScheduleId": 1042,
  "ReportScheduleName": "Topside Module M14",
  "RevisionType": "Live",
  "Cutoff": "2026-06-01T00:00:00",
  "GeneratedAtUtc": "2026-06-01T02:14:07.113Z",
  "MetadataFieldsByFile": {
    "reportActivity.parquet": [
      { "ColumnName": "Description_2",
        "FieldName": "Description",
        "MetadataFieldId": 5514,
        "IlapId": "ILAP-DISCIPLINE-DESC",
        "SourceType": "Activity",
        "ParquetType": "string" }
    ]
  },
  "MetadataFields": [ "…the same, flattened across files…" ]
}

The physical column name is not a stable identifier, and treating it as one fails silently. Metadata columns are named against the set of names the fixed columns have already claimed, so:

A metadata field named Description on a report activity becomes Description_2, because the domain model claimed the name first.
The _2 / _3 suffixes are handed out in iteration order, so adding one field upstream can turn Location_2 into Location_3 in the next export.
Name sanitisation is many-to-one: Cost/Unit and Cost-Unit both reduce to Cost_Unit, and the loser is suffixed.

None of that raises an error. Union two snapshots on physical column names and values quietly appear under the wrong heading.

The stable identity is IlapId, and this manifest is the only route from a column back to it. Resolve every metadata column to a name derived from IlapId — ann_<IlapId> is a reasonable convention — before combining snapshots. Where two report schedules define different field sets, a model built this way carries the union and reads missing fields as null, which is safe precisely because the resolution is by identity rather than by position.

_manifest.json sits inside each planning snapshot folder, beside _complete.json. Snapshots written before that was the case carry a single manifest at the schedule root, rs=<id>/_manifest.json, instead — worth tolerating if you are reading an archive that goes back far enough.

Prefer MetadataFieldsByFile over the flat MetadataFields list. The flat list is deduplicated by column name, so a column name shared by two entities keeps only one identity; the per-file map preserves each file's own, and exists specifically to stop a cross-entity name collision decoding a value under the wrong field. Fall back to the flat list only for older snapshots written before the per-file map existed. If a manifest has the map but does not list a file, that file genuinely carries no metadata columns — falling back there would invent columns that are not in it.

Where IlapId is empty, identity falls back to MetadataFieldId, which is unique within one database and meaningless outside it. A model built on those ids will not survive being pointed at another tenant's export.