Skip to content

Add Customer 360 industry demo - #1472

Open
amising6 wants to merge 15 commits into
databrickslabs:mainfrom
amising6:customer360-dq-example
Open

Add Customer 360 industry demo#1472
amising6 wants to merge 15 commits into
databrickslabs:mainfrom
amising6:customer360-dq-example

Conversation

@amising6

Copy link
Copy Markdown

This PR adds a Customer 360 industry demo using fully synthetic data.

The demo shows how DQX can validate common Customer 360 data quality rules, including customer identity, email format, non-negative revenue, ticket count validation, date validation, and active customer flag consistency.

The contribution is limited to a new demo file under demos/dqx_demo_industry and does not modify core package code.

No real customer, client, employer, or proprietary data is included.

@amising6
amising6 requested a review from a team as a code owner August 14, 2026 16:42
@amising6
amising6 requested review from gergo-databricks and removed request for a team August 14, 2026 16:42
@CLAassistant

CLAassistant commented Aug 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

Copy link
Copy Markdown
Contributor

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

@amising6
amising6 force-pushed the customer360-dq-example branch from 9d75471 to 0983c4a Compare August 14, 2026 16:54
@amising6

Copy link
Copy Markdown
Author

Thank you. I have signed the CLA and updated the PR with a signed commit.

Please let me know if you would like me to adjust the demo structure, naming, or any DQX rule usage to better align with the existing industry demos.

@ghanse ghanse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amising6 dqx_customer_360_demo.py is empty. Do you need to push it?

@amising6
amising6 force-pushed the customer360-dq-example branch from 0983c4a to 3f56858 Compare August 14, 2026 18:31
@amising6

Copy link
Copy Markdown
Author

Thanks for catching that. I had created the file but missed saving/pushing the actual notebook content. I have updated the branch with the Customer 360 demo content now.

Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py
@ghanse

ghanse commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Thanks for adding this — a Customer 360 accelerator is a nice fit next to the other industry demos, and using fully synthetic data is the right call. A few things to sort out before it's ready:

Blockers (the demo won't run as-is):

  • is_unique needs columns: as a list, not column: — this fails at validate_checks and stops the notebook.
  • name: is nested under check: on all seven checks, so DQX ignores it. It belongs at the top level next to criticality.

Worth fixing:

  • The duplicate CUST-001 row means your "valid active customer" gets quarantined by the uniqueness check — give the dupe its own id.
  • Swap the magic is_in_range ceilings for is_not_less_than on the non-negative checks.
  • A couple of rows commented "Invalid" are warn-level and stay in valid_df; a one-line note would save new users some confusion.

Small stuff:

  • Add a trailing newline at the end of the file (trips some linters).
  • Add this demo to docs/dqx/docs/demos.mdx alongside the other industry accelerators so people browsing the docs actually find it.
  • The "Why this matters" and "Practical Takeaway" sections repeat the same point — trim one.

Good concept and structure — just needs the two YAML fixes to actually execute, plus the cleanups above.


This comment was generated with GitHub MCP.

@ghanse ghanse added the needs-changes Changes required after review label Aug 29, 2026
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments. In general, I would expect this demo to show example of some more comprehensive checks.

@mwojtyczka mwojtyczka added the under-review This PR is currently being reviewed by one of DQX maintainers. label Sep 1, 2026
Thanks for the detailed review. I have addressed all the comments:

Updated is_unique to use columns as a list.
Moved all rule names to the top level next to criticality.
Replaced the range checks with is_not_less_than.
Kept CUST-001 valid and added a separate duplicate ID pair to demonstrate the uniqueness check.
Clarified that warning-level records remain in valid_df.
Shortened the repetitive Practical Takeaway section.
Added the trailing newline.

I also validated the updated YAML successfully. Please review the latest changes when you get a chance.
@amising6

amising6 commented Sep 8, 2026

Copy link
Copy Markdown
Author

Thank you for the detailed review. I have pushed an expanded version of the Customer 360 demo and addressed the requested changes.

The demo now includes 12 checks covering identity completeness and uniqueness, customer name and email completeness, email validity, customer status completeness and allowed values, non-negative revenue and ticket counts, purchase and campaign date validation, and cross-field active-status consistency.

I also corrected the DQX check syntax, removed the artificial range ceilings, separated the duplicate test records, clarified warning-level behavior, removed redundant documentation, and verified that validate_checks completes without errors.

Please take another look when convenient.

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline review comments from a correctness pass. The main issues (first two) are that the demo teaches the opposite of DQX's actual split semantics; the rest are smaller.

Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
Comment thread docs/dqx/docs/demos.mdx Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes. 1 old comment is still not fully addressed and I added few more.

Comment thread docs/dqx/docs/demos.mdx
@amising6

amising6 commented Sep 9, 2026

Copy link
Copy Markdown
Author

Hi, I have pushed the requested updates and addressed the latest review comments.

The changes include correcting the warning and quarantine explanation, using runtime-relative dates, updating the active-status flag description, and pinning the demo documentation link to v0.16.0.

I have also re-requested the review. Thanks again for the detailed feedback.

@amising6
amising6 requested a review from mwojtyczka September 9, 2026 23:43
Comment thread demos/dqx_demo_industry/dqx_customer_360_demo.py Outdated
mwojtyczka and others added 3 commits September 10, 2026 14:26
…split narrative

- Use arguments.column form for the customer_id presence check to match
  the sibling not-null rules (drops the single-element for_each_column).
- Rewrite active_customer_flag_consistency so it only flags genuine
  ACTIVE/INACTIVE contradictions; unsupported statuses (e.g. UNKNOWN) no
  longer trip a second warning on top of the is_in_list error.
- Rewrite the split narrative to describe valid_df/invalid_df directly
  (the demo never calls get_valid()) and note warn-only rows land in both.

Co-authored-by: Isaac <no-reply@databricks.com>
Runs demos/dqx_demo_industry/dqx_customer_360_demo.py as a notebook job,
mirroring the banking and fashion industry-demo e2e tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Read back and show the persisted quarantine table so the demo surfaces
the saved invalid records inline.

Co-authored-by: Isaac <no-reply@databricks.com>

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - i added the demo to the e2e tests

@mwojtyczka mwojtyczka added Approved to Merge When PR is reviewed and approved. To be merged once all tests pass and removed under-review This PR is currently being reviewed by one of DQX maintainers. needs-changes Changes required after review labels Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved to Merge When PR is reviewed and approved. To be merged once all tests pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants