Added Architecture & Data model. by croadfeldt · Pull Request #7 · dcm-project/dcm-project.github.io

croadfeldt · 2026-03-26T02:31:45Z

No description provided.

sourcery-ai

Sorry @croadfeldt, your pull request is larger than the review limit of 150000 diff characters

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…, layers, sovereignty controls. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…e model, posture groups, compliance domain groups Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…ability. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…h level, concurrent rehydration, discovered state retention. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

… deployment bootstrap info. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…alidat, policy review, governance, grouping, relationship role validation, information providers. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…tion, commit log capacity, system initiated records, distributed hash chains. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…on metadata, operational analysis. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…nty pre-filter, audit provenance, universal groups, information providers. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

machacekondra · 2026-03-27T11:07:07Z

@@ -0,0 +1,645 @@
+---


Do I understand correctly that this is suppose to be an k8s operator that is a service provider?

Yes, I was looking into the feasibility for f pulling in K8S operators as providers. It's not complete or vetted as an implementation yet though. I should make a section on topics like that.

machacekondra · 2026-03-27T11:07:52Z

+    OperationCreate   Operation = "CREATE"
+    OperationRead     Operation = "READ"
+    OperationUpdate   Operation = "UPDATE"
+    OperationDelete   Operation = "DELETE"


Why do we define here imperative operations, if it's k8s operator, which is declarative?

Great point. This is incomplete, I need to mark it as such and we should talk through this idea further.

machacekondra · 2026-03-27T11:09:57Z

+    // Returns the DCM-assigned provider UUID on success.
+    Register(ctx context.Context, reg ProviderRegistration) (string, error)
+
+    // ReportStatus sends a realized state payload to DCM.


So this is called just once, or every time there is a change? "realized state" might be strange name here, should it be just current state or something?

machacekondra · 2026-03-27T11:10:37Z

+    // ReportStatus sends a realized state payload to DCM.
+    ReportStatus(ctx context.Context, resourceID string, status RealizedState) error
+
+    // ReportEvent sends a lifecycle event to DCM.


What is difference between reporting status and event?

machacekondra · 2026-03-27T11:11:34Z

+    // ConfirmDecommission acknowledges a decommission request from DCM.
+    // Required for Level 3.
+    ConfirmDecommission(ctx context.Context, resourceID string, confirmation DecommissionConfirmation) error
+}


What about some health report, or is it meant to be part of capacity report?

machacekondra · 2026-03-27T11:12:08Z

+
+    // ConfirmDecommission acknowledges a decommission request from DCM.
+    // Required for Level 3.
+    ConfirmDecommission(ctx context.Context, resourceID string, confirmation DecommissionConfirmation) error


Why there must be a separate call for this? Why it can't be part of ReportEvent?

machacekondra · 2026-03-27T11:14:20Z

+```go
+// SDK is the primary interface for the DCM Operator SDK.
+// Operator developers interact with DCM through this interface.
+type SDK interface {


I don't understand this interface, why we need it?

machacekondra · 2026-03-27T11:21:29Z

+
+---
+
+## 10. Open Questions


I really didn't figure out, while reading the spec, what is the CR this operator reconciles? It's not defined anywhere and it's main communication point and spec between from DCM to operator.

machacekondra · 2026-03-27T11:25:09Z

+
+DCM is **not a provisioning tool**. It is the management plane that sits above
+provisioning tools, governing what gets requested, approved, built, owned, and
+decommissioned. Provisioning tools (Ansible, Terraform, Kubernetes operators)


So, are we suppose use Ansible, Terraform and K8s operators as provisioners?

Those can be tools used, yes. DCM is not prescriptive in what tools are used to effect the change, it may come with some though to bootstrap usage or as examples of how to build providers.

Many providers and provisioning tools exist already, the goal is to reuse feasible tooling and encourage the adoption of the DCM provider interface.

…s vs unique processing responsibilities. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…curity is top priority and extensible. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…ocumentation / specs. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…andling depedencies better, health endpoints. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

machacekondra · 2026-03-30T10:45:46Z

+intent-store/
+└── a1b2c3d4-tenant-uuid/
+    └── Compute/
+        └── VirtualMachine/


So you define intent.yaml per resource, how do you define the dependencies of the intent, then? Would be awesome to change this example to work with three tier app + DNS + IP(DHCP/static), requiring single VM is too simple use case, that don't really show how everything works together.

…cy run applied. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

pkliczewski · 2026-03-30T10:29:16Z

+
+## 2. Data Classification
+
+Data classification is a **first-class field-level metadata property** in the DCM data model. Every field in every payload carries a `data_classification` value. This classification is the primary axis of the authorization matrix and is the key input to sovereignty and compliance enforcement.


how do we want to verify this and which component should be responsible for processing data_classification?

pkliczewski · 2026-03-30T10:30:23Z

+
+**Classification is declared in three places:**
+- **Resource Type Specification** — default classification per field for all instances of that type
+- **Data Layer** — classification applied across a domain (e.g., an org layer that marks all cost_center fields as `confidential`)


how do we map it to org structure like cost_center?

pkliczewski · 2026-03-30T11:16:07Z

+
+### 3.1 What Accreditation Is
+
+An **Accreditation** is a formal, versioned, time-bounded attestation that a DCM component — a Service Provider, a Policy Provider, a Storage Provider, a Notification Provider, or a DCM deployment itself — satisfies the requirements of a specific compliance framework. Accreditations are issued by an **Accreditor** and registered with DCM as first-class artifacts.


please provide flow diagram for accreditation. it is not clear to me how it should work

pkliczewski · 2026-03-30T11:17:27Z

+### 3.4 Accreditation Lifecycle
+
+```
+Accreditation submitted (via API or GitOps PR)


do should we handle gitops? for now we only have an api.

pkliczewski · 2026-03-30T11:19:09Z

+---
+
+
+> **Architecture Update:** Section 4 of this document (Data/Capability Authorization Matrix) has been superseded by the **Unified Governance Matrix** ([doc 27](27-governance-matrix.md)). The governance matrix provides a more powerful, unified model that replaces the standalone matrix described here. The accreditation model (Sections 2-3) and zero trust interaction model (Section 5) remain current and are consumed by the governance matrix as inputs.


as this is work in progress, feel free to remove obsolete parts. No need for such updates

pkliczewski · 2026-03-30T12:09:21Z

+1. **Authentication** — is this identity who they claim to be?
+2. **Authorization** — what is this identity permitted to do?
+
+Every authentication mode DCM supports — static API key, local users, GitHub OAuth, LDAP, FreeIPA, Active Directory, OIDC, mTLS — is an Auth Provider implementation. The built-in Auth Provider ships with DCM and requires zero external configuration, enabling immediate home lab and evaluation use. External Auth Providers are registered artifacts, versioned, GitOps-managed, and audited.


we need to decide what we want to support

pkliczewski · 2026-03-30T12:10:11Z

+
+### 4.1 Built-In Auth Provider (zero configuration)
+
+Ships with DCM. Always registered. Cannot be deregistered — only deprioritized.


always registered means to me that this could be deployment config

pkliczewski · 2026-03-30T12:11:11Z

+
+---
+
+## 6. Credential Provider


what is the use case for credential provider? can we have a user flow?

pkliczewski · 2026-03-30T12:12:51Z

+
+## 1. The Core Model
+
+### 1.1 What an Authority Tier Is


what is the use case for this?

gabriel-farache · 2026-03-30T13:14:49Z

+  → New Data triggers new Events → repeat
+```
+
+See [Foundational Abstractions](data-model/foundations/) for the complete model.


the link to the fondations document is broken

gabriel-farache · 2026-03-30T13:23:53Z

+- **[Foundational Abstractions](data-model/foundations/)** — Data, Provider, Policy — read this first
+- **[Unified Provider Contract](data-model/provider-contract/)** — base contract + 11 typed extensions
+- **[Unified Policy Contract](data-model/policy-contract/)** — base contract + 7 output schemas
+- **[Federated Contribution Model](data-model/federated-contribution-model/)** — who contributes what and how


links broken, (Capabilities Matrix as well) I think all links should be reviewed

gabriel-farache · 2026-03-30T13:41:16Z

+Event (Data state change)
+  → Policy Engine evaluates all matching Policies


IIUC, if a policy changes, it means that all existing and previously executed requests have to go through that flow again?
Isn't the scope too broad here? Should it be limited to a specific set of data change (like incoming requests)?

gabriel-farache · 2026-03-30T14:01:58Z

+| **Resource Entity** | A realized infrastructure resource; the primary managed thing | Realized Store |
+| **Process Entity** | An ephemeral execution (job, playbook, pipeline) | Realized Store |
+| **Composite Entity** | A Meta Provider composition of Resource Entities | Realized Store |
+| **Intent State** | Consumer's raw declaration before processing | Intent Store (GitOps) |


Why would we store every intent in Git instead of a DB? Does the intent == the offering of DCM (our current catalog item) or is it the request sent to DCM to create/provision a resource?

If it's the 1st, then makes sense to persist it in Git but based on my understanding it's not the case. (I understand that the state is based on providers PoV, not DCM as the "requested state" is after the policies were evaluated)
Otherwise, storing the intent in the DB would be enough, no? Is there a need for community (by community I mean several people) reviews for each request

gabriel-farache · 2026-03-30T14:08:32Z

+| **Service Provider** | Realizes infrastructure resources | DCM → Provider → DCM |
+| **Information Provider** | Serves authoritative external data | DCM queries → Provider responds |
+| **Storage Provider** | Persists DCM state | DCM reads/writes ↔ Provider |
+| **Meta Provider** | Composes multiple providers | DCM → Meta → Children → DCM |


Does it means that only the Children talk to DCM? Shouldn't this falls on the Meta only? In case there is any translation to do from the children? Or is the meta just a boilerplate?

gabriel-farache · 2026-03-30T14:09:10Z

+| **Meta Provider** | Composes multiple providers | DCM → Meta → Children → DCM |
+| **Policy Provider** | Evaluates policies externally | DCM sends payload → Provider decides |
+| **Credential Provider** | Manages secrets and credentials | DCM requests → Provider issues |
+| **Auth Provider** | Authenticates identities | DCM verifies → Provider confirms |


can/should this provider be used by other providers as well?

gabriel-farache · 2026-03-30T14:16:41Z

+- **Enforcement level** — hard (cannot be overridden) or soft (can be tightened by more-specific policies)
+- **Domain precedence** — policies at more-specific domains win within their concern type; system > platform > tenant > resource_type > entity
+- **Lifecycle** — every Policy follows the standard artifact lifecycle (developing → proposed → active → deprecated → retired)
+- **Shadow mode** — proposed Policies execute against real traffic without applying results; safe validation before activation


so this is a specific type of policy that does some pre-flight validations? What is the point of evaluating without applying? Is it not duplication as the policy will be evaluated later?
Or is the GateKeeper or Validation applied anyway?

gabriel-farache · 2026-03-30T14:17:21Z

+| **Governance Matrix Rule** | Any cross-boundary interaction | `ALLOW / DENY / ALLOW_WITH_CONDITIONS / STRIP_FIELD / REDACT / AUDIT_ONLY` |
+| **Lifecycle Policy** | Relationship events | `action` on the related entity (save, destroy, notify, cascade) |
+
+**The unified Policy base contract** is defined in [B-policy-contract.md](B-policy-contract.md). All seven Policy types implement this base contract. What varies is the output schema.


B-policy-contract.md is broken

gabriel-farache

are the data models files duplicated between architecture/data-model/ and data-model/?

jenniferubah

@croadfeldt , I see there are two different folders for the data model. And the contents looks similar. For example, the concept of four-states are in both:
data-model under doc folder and data-model under architecture folder
Not sure if it is intentional and if yes, what are difference?

jenniferubah · 2026-03-30T17:16:53Z

+
+      - component_id: dns-primary
+        resource_type: DNS.Record
+        provided_by: self        # This Meta Provider handles DNS


I am unclear about the concept of provided_by: self field. With this field set to "self", it seems this particular consituent (dns-primary) already has a provider before it reaches policy? Is it supposed to by-pass policy which should handle the selection of provider each consituent? Or why pass this consituent in the payload at all since it already has its provider?

jenniferubah · 2026-03-30T17:28:48Z

+
+| Classification | Failure effect |
+|----------------|---------------|
+| `required` | DCM halts the compound request; triggers Recovery Policy; unrealized constituents are not dispatched |


Failure effect on a required would also mean a rollback/deletion of other successfully dispatches required/partial/optional consistuents, right?

jenniferubah · 2026-03-30T17:37:00Z

+| Classification | Failure effect |
+|----------------|---------------|
+| `required` | DCM halts the compound request; triggers Recovery Policy; unrealized constituents are not dispatched |
+| `partial` | DCM notes the failure; compound service continues; final status may be `DEGRADED` |


A bit unclear about the final status, why do we set it to DEGRADED and not sure a user would know what this implies? Also, why not just required and optional that way DCM knows to either apply recovery policy or not. I'm not sure about the partial aspect.

machacekondra · 2026-03-30T19:26:35Z

+
+The compound service definition is the Meta Provider's primary contribution to DCM. It is declared at registration and stored in the Resource Type Registry as a compound Resource Type Specification.
+
+### 2.1 Constituent Declaration


AFAIK this can't work. Just please try add here simple example. three tier app + DNS. Vm - backend is using VM2 database, and one using DHCP, one static IP, and both then are defined in DNS. To me this is imposible to define with this meta-provider specification. You here only define one thing depend on the other explicitly, this must be IMHO implicit based on the requirment of specific attribute. Otherwise it will be maintenence nightmare.

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

…project, added some details on who,what,why,etc.. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

… 15 specs, 4 OpenAPI) Addresses all PR dcm-project#7 review comments: - Eliminated duplicate data-model directories - Fixed all broken cross-references - Removed 116 duplicate files - Updated provider count (12), policy count (8), classification enum (8) - Synced all OpenAPI YAMLs with canonical schemas

… 15 specs, 4 OpenAPI) Addresses all PR dcm-project#7 review comments: - Eliminated duplicate data-model directories - Fixed all broken cross-references - Removed 116 duplicate files - Updated provider count (12), policy count (8), classification enum (8) - Synced all OpenAPI YAMLs with canonical schemas Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

croadfeldt · 2026-04-01T19:18:02Z

Superceded by PR #8

sourcery-ai Bot reviewed Mar 26, 2026

View reviewed changes

Added Architecture & Data model.

76781d4

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

croadfeldt force-pushed the main branch from 9d653ca to 76781d4 Compare March 26, 2026 16:46

Added 2nd round of updates. Mostly policy and relationship mapping.

f652375

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

croadfeldt force-pushed the main branch from 13771a8 to f652375 Compare March 26, 2026 16:47

croadfeldt added 15 commits March 26, 2026 13:29

Added auditing specifications, deployment specifications

21b8d31

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Added auth providers and webhooks

8911368

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Added git pr ingestion, additional detail on auth providers, webhooks…

44ecdd3

…, layers, sovereignty controls. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Update service dependency model, resource entity details.

af3eb2b

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Added some advanced information provider details, add DCM federation.

463ee18

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Provenanage models, validation, policy review. Two dimensional profil…

21e58cc

…e model, posture groups, compliance domain groups Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

resolve group and relationship gaps

45aa375

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Reframe confidence scoring model, add observability stream.

133954c

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Resolve override control, Constraint schema and post-realization edit…

4d620ad

…ability. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Updated states document, UUID preservation on rehydration, pinned aut…

8505e50

…h level, concurrent rehydration, discovered state retention. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Terminology replacement, cache model update, signal priority updates,…

5182be0

… deployment bootstrap info. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

consolidate answers into other relevant sections. Provenance model, v…

83863a4

…alidat, policy review, governance, grouping, relationship role validation, information providers. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Add SCIM, failover chain, MFA, pluggable storage, Hash chain verifica…

3178a5e

…tion, commit log capacity, system initiated records, distributed hash chains. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Updated policy profile registry and submission lifecycle, certificati…

489f85c

…on metadata, operational analysis. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Updated DCM Feneration, cert rotation, request routing flow, sovereig…

fb0235e

…nty pre-filter, audit provenance, universal groups, information providers. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

machacekondra reviewed Mar 27, 2026

View reviewed changes

machacekondra requested a review from pkliczewski March 27, 2026 11:26

croadfeldt added 5 commits March 29, 2026 09:15

Added details on meta providers. Clarrified them as compound provider…

2460a43

…s vs unique processing responsibilities. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Clarified design criterias and security posture modeling to ensure se…

dcd548d

…curity is top priority and extensible. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

More details on the webhook spec

e87c46c

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Cleaned up admin api spec. Another consistency run done through all d…

df4da1a

…ocumentation / specs. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Updated Auth revocation handling, ssl handling, document standards, h…

d959b60

…andling depedencies better, health endpoints. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

machacekondra reviewed Mar 30, 2026

View reviewed changes

Massive update with AEP applied, API specs coming together, Consisten…

b319b7d

…cy run applied. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

pkliczewski reviewed Mar 30, 2026

View reviewed changes

gabriel-farache reviewed Mar 30, 2026

View reviewed changes

jenniferubah reviewed Mar 30, 2026

View reviewed changes

machacekondra reviewed Mar 30, 2026

View reviewed changes

croadfeldt added 3 commits March 31, 2026 09:30

Updated docs, consolidated updates into core documents.

f1e1cf6

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Added more example use cases;

c600c77

Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

Attempting to complete the pieces needed to pull the trigger on this …

bbcfd2b

…project, added some details on who,what,why,etc.. Signed-off-by: Chris Roadfeldt <chris@roadfeldt.com>

croadfeldt mentioned this pull request Apr 1, 2026

Replace content with fixed, deduplicated architecture. #8

Open

croadfeldt closed this Apr 1, 2026


		## 2. Data Classification

		Data classification is a first-class field-level metadata property in the DCM data model. Every field in every payload carries a `data_classification` value. This classification is the primary axis of the authorization matrix and is the key input to sovereignty and compliance enforcement.


		### 3.1 What Accreditation Is

		An Accreditation is a formal, versioned, time-bounded attestation that a DCM component — a Service Provider, a Policy Provider, a Storage Provider, a Notification Provider, or a DCM deployment itself — satisfies the requirements of a specific compliance framework. Accreditations are issued by an Accreditor and registered with DCM as first-class artifacts.

		---


		> Architecture Update: Section 4 of this document (Data/Capability Authorization Matrix) has been superseded by the Unified Governance Matrix ([doc 27](27-governance-matrix.md)). The governance matrix provides a more powerful, unified model that replaces the standalone matrix described here. The accreditation model (Sections 2-3) and zero trust interaction model (Section 5) remain current and are consumed by the governance matrix as inputs.


		### 4.1 Built-In Auth Provider (zero configuration)

		Ships with DCM. Always registered. Cannot be deregistered — only deprioritized.

		Event (Data state change)
		→ Policy Engine evaluates all matching Policies


		The compound service definition is the Meta Provider's primary contribution to DCM. It is declared at registration and stored in the Resource Type Registry as a compound Resource Type Specification.

		### 2.1 Constituent Declaration

Conversation

croadfeldt commented Mar 26, 2026

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machacekondra Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabriel-farache Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

machacekondra Mar 30, 2026 •

edited

Loading

gabriel-farache Mar 30, 2026 •

edited

Loading

jenniferubah Mar 30, 2026 •

edited

Loading

jenniferubah Mar 30, 2026 •

edited

Loading

machacekondra Mar 30, 2026 •

edited

Loading