Import Focus Parquet from AWS S3 with Tags

[DMarshall-1]

Hi all,

I’m working on importing AWS cost data from an S3 bucket into Azure FinOps Hub using the FinOps Toolkit.

Following the Microsoft guidance here:
https://learn.microsoft.com/en-us/cloud-computing/finops/toolkit/hubs/deploy?tabs=azure-portal%2Cadx-dashboard#ingest-from-other-data-sources

I have successfully built and tested:

• Linked Service to AWS S3
• Source Dataset pointing to Parquet files in S3
• Azure Data Factory pipeline that:
  • Copies the data
  • Generates the FinOps manifest
  • Triggers ingestion into FinOps Hub

This works correctly while AWS cost allocation Tags are disabled.

Problem

When Tags are enabled on the AWS FOCUS export to Parquet, the pipeline / ingestion fails.

From investigation:

• AWS exports Tags as a mapped / complex type
• The FinOps Hub ingestion (via ADF) appears to expect Tags as JSON (string)
• As a result, Parquet ingestion fails when the Tags column is included

Question

Has anyone successfully imported AWS FOCUS Parquet data with Tags enabled into FinOps Hub?

Specifically:

• Is there a supported or recommended way to transform the Tags column (e.g. flattening or converting to JSON) prior to ingestion?
• Is this a known limitation of the current FinOps Toolkit ingestion pipeline?
• Are there examples or patterns (ADF Data Flow, preprocessing step, etc.) that others have used as a workaround?

Any guidance or examples would be greatly appreciated.

Thanks in advance.

[Roland Krummenacher (RolandKrummenacher)]

Investigation notes

Root cause

The FinOps Hub ingestion pipeline directly ingests Parquet files into Azure Data Explorer using a .ingest KQL command (Analytics/app.bicep:1385-1388). There is no intermediate ADF Copy Activity or Data Flow that transforms column types.

The Costs_raw table schema defines Tags as string (IngestionSetup_RawTables.kql:417), and the ingestion mapping maps the Parquet field directly to the column (IngestionSetup_RawTables.kql:567):

{ "Column": "Tags", "Properties": { "Field": "Tags" } }

Azure Cost Management exports Tags as a JSON string in Parquet (e.g., {"env":"prod"}), so this works for Azure data. AWS FOCUS exports encode Tags as a native Parquet MAP<string,string> type. ADX cannot ingest a Parquet MAP into a string column, which causes the failure.

However, ADX can ingest Parquet MAP types into a dynamic column (docs). In fact, MAP, LIST, and STRUCT can only be mapped to dynamic — no other target type is supported.

FOCUS spec status

The FOCUS specification defines Tags with a logical data type of "JSON" and requires the value to be a "serialized JSON string" (Key-Value Format). However, it does not specify the physical Parquet serialization type — leaving providers free to choose between a STRING column containing JSON or a native MAP<string,string>.

This is a known cross-provider incompatibility tracked in the FOCUS spec repo:

• FOCUS_Spec#680 (closed as dup) — "AWS exports TAGS to MAP type which causes issues with Microsoft Fabric"
• FOCUS_Spec#1079 (open) — Active tracking issue: "[FR] Standardize tag export format across FOCUS datasets." Maintainer assessment notes that "no physical serialization guidance exists for JSON-typed columns in Parquet exports" and that "JSON string is the most portable option."
• FOCUS_Spec#979 (open) — Related request from Google Cloud for MAP type flexibility.

Potential fix: Change Tags to dynamic in Costs_raw

Changing Tags: string → Tags: dynamic in Costs_raw (IngestionSetup_RawTables.kql:417) should be safe. The downstream transform Costs_transform_v1_2() already calls parse_json(Tags) (IngestionSetup_v1_2.kql:492) before writing to the final table (where Tags is already dynamic at line 971).

This handles both providers correctly:

• Azure JSON string Tags → ingested into dynamic as a string → parse_json() parses it into a property bag → works as before
• AWS MAP Tags → ingested into dynamic as a property bag → parse_json() is a no-op on an already-parsed dynamic → works correctly

How to test

If you are able to test this on a test hub instance (not production), we would appreciate validation with both Azure and AWS data. On a fresh test hub, run the following in your ADX database:

// Change the Tags column type from string to dynamic
.alter column Costs_raw.Tags type = dynamic

Then ingest both Azure and AWS FOCUS Parquet data (with Tags enabled) and verify:

1. Azure ingestion still works as before
2. AWS ingestion with Tags no longer fails
3. Downstream transforms produce correct Tags in the final tables

You can validate with:

// Check Tags in the raw table
Costs_raw
| where isnotempty(Tags)
| project Tags, TagType = gettype(Tags)
| take 10

// Check Tags in the final table
Costs_final_v1_2
| where isnotempty(Tags)
| project Tags, TagType = gettype(Tags)
| take 10

[DMarshall-1]

thank you very much for this, unfortunately I don't have a test instance at this time to verify it.

If we could make the change in Live and then revert if we encounter any issues I can make the change and test it?

Would it just be

.alter column Costs_raw.Tags type = string

to revert?

[Roland Krummenacher (RolandKrummenacher)]

Problem is that "Using the .alter command incorrectly may lead to data loss." see

https://learn.microsoft.com/en-us/kusto/management/alter-table-command?view=microsoft-fabric

[Roland Krummenacher (RolandKrummenacher)]

However, as the Costs_raw table doesn't keep data anyway (or only for a limited period of time) you should be fine.