Add Customer 360 industry demo - #1472
Conversation
|
All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits |
9d75471 to
0983c4a
Compare
|
Thank you. I have signed the CLA and updated the PR with a signed commit. Please let me know if you would like me to adjust the demo structure, naming, or any DQX rule usage to better align with the existing industry demos. |
0983c4a to
3f56858
Compare
|
Thanks for catching that. I had created the file but missed saving/pushing the actual notebook content. I have updated the branch with the Customer 360 demo content now. |
|
Thanks for adding this — a Customer 360 accelerator is a nice fit next to the other industry demos, and using fully synthetic data is the right call. A few things to sort out before it's ready: Blockers (the demo won't run as-is):
Worth fixing:
Small stuff:
Good concept and structure — just needs the two YAML fixes to actually execute, plus the cleanups above. This comment was generated with GitHub MCP. |
mwojtyczka
left a comment
There was a problem hiding this comment.
Left some comments. In general, I would expect this demo to show example of some more comprehensive checks.
Thanks for the detailed review. I have addressed all the comments: Updated is_unique to use columns as a list. Moved all rule names to the top level next to criticality. Replaced the range checks with is_not_less_than. Kept CUST-001 valid and added a separate duplicate ID pair to demonstrate the uniqueness check. Clarified that warning-level records remain in valid_df. Shortened the repetitive Practical Takeaway section. Added the trailing newline. I also validated the updated YAML successfully. Please review the latest changes when you get a chance.
|
Thank you for the detailed review. I have pushed an expanded version of the Customer 360 demo and addressed the requested changes. The demo now includes 12 checks covering identity completeness and uniqueness, customer name and email completeness, email validity, customer status completeness and allowed values, non-negative revenue and ticket counts, purchase and campaign date validation, and cross-field active-status consistency. I also corrected the DQX check syntax, removed the artificial range ceilings, separated the duplicate test records, clarified warning-level behavior, removed redundant documentation, and verified that validate_checks completes without errors. Please take another look when convenient. |
mwojtyczka
left a comment
There was a problem hiding this comment.
Inline review comments from a correctness pass. The main issues (first two) are that the demo teaches the opposite of DQX's actual split semantics; the rest are smaller.
mwojtyczka
left a comment
There was a problem hiding this comment.
Thanks for the fixes. 1 old comment is still not fully addressed and I added few more.
added time date to fix hard code tim value
|
Hi, I have pushed the requested updates and addressed the latest review comments. The changes include correcting the warning and quarantine explanation, using runtime-relative dates, updating the active-status flag description, and pinning the demo documentation link to I have also re-requested the review. Thanks again for the detailed feedback. |
…split narrative - Use arguments.column form for the customer_id presence check to match the sibling not-null rules (drops the single-element for_each_column). - Rewrite active_customer_flag_consistency so it only flags genuine ACTIVE/INACTIVE contradictions; unsupported statuses (e.g. UNKNOWN) no longer trip a second warning on top of the is_in_list error. - Rewrite the split narrative to describe valid_df/invalid_df directly (the demo never calls get_valid()) and note warn-only rows land in both. Co-authored-by: Isaac <no-reply@databricks.com>
Runs demos/dqx_demo_industry/dqx_customer_360_demo.py as a notebook job, mirroring the banking and fashion industry-demo e2e tests. Co-authored-by: Isaac <no-reply@databricks.com>
Read back and show the persisted quarantine table so the demo surfaces the saved invalid records inline. Co-authored-by: Isaac <no-reply@databricks.com>
mwojtyczka
left a comment
There was a problem hiding this comment.
LGTM - i added the demo to the e2e tests
This PR adds a Customer 360 industry demo using fully synthetic data.
The demo shows how DQX can validate common Customer 360 data quality rules, including customer identity, email format, non-negative revenue, ticket count validation, date validation, and active customer flag consistency.
The contribution is limited to a new demo file under demos/dqx_demo_industry and does not modify core package code.
No real customer, client, employer, or proprietary data is included.