CMR-11242: Adding Toggle for Granule collections Reference validation by eudoroolivares2016 · Pull Request #2446 · nasa/Common-Metadata-Repository

eudoroolivares2016 · 2026-06-18T04:20:42Z

Overview

What is the objective?

This adds a feature toggle which when set to false will allow ingesting granules even if the references between keywords (namely, projects, platforms, instruments, instrument characterists , and operational modes). Instead, those will be returns as warnings from ingest response

What are the changes?

Major change: Creating the feature toggle in the common as it needed to be used in both ingest and the umm-lib. Threading that up from the validation so that warnings can be propagated down to the end user response.

Minor Changes:
Creating the feature toggle in the common as it needed to be used in both ingest and the umm-lib there were some additional validation which was dead-code that I am removing. The linter picked up a few changes that I allowed it to leave in there namely spacing. Changing imports to just make this more consistent.

What areas of the application does this impact?

Granule ingest and granule validation

Required Checklist

New and existing unit and int tests pass locally and remotely
clj-kondo has been run locally and all errors in changed files are corrected
I have commented my code, particularly in hard-to-understand areas
I have made changes to the documentation (if necessary)
My changes generate no new warnings

Additional Checklist

I have removed unnecessary/dead code and imports in files I have changed
I have cleaned up integration tests by doing one or more of the following:
- migrated any are2 tests to are3 in files I have changed
- de-duped, consolidated, removed dead int tests
- transformed applicable int tests into unit tests
- reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

…rms; and sensors

codecov-commenter · 2026-06-18T04:29:54Z

Codecov Report

❌ Patch coverage is 40.54054% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.97%. Comparing base (d304270) to head (73fbae2).

Files with missing lines	Patch %	Lines
...src/cmr/ingest/services/ingest_service/granule.clj	5.88%	16 Missing ⚠️
ingest-app/src/cmr/ingest/api/core.clj	20.00%	12 Missing ⚠️
...ngest-app/src/cmr/ingest/validation/validation.clj	20.00%	12 Missing ⚠️
...int-test/src/cmr/system_int_test/data2/granule.clj	14.28%	6 Missing ⚠️
ingest-app/src/cmr/ingest/api/granules.clj	28.57%	5 Missing ⚠️
...m-spec-lib/src/cmr/umm_spec/validation/granule.clj	88.57%	4 Missing ⚠️
...r/umm_spec/validation/umm_spec_validation_core.clj	33.33%	4 Missing ⚠️
ingest-app/src/cmr/ingest/api/collections.clj	50.00%	2 Missing ⚠️
...ngest-app/src/cmr/ingest/api/generic_documents.clj	0.00%	1 Missing ⚠️
ingest-app/src/cmr/ingest/api/services.clj	0.00%	1 Missing ⚠️
... and 3 more

❗ There is a different number of reports uploaded between BASE (d304270) and HEAD (73fbae2). Click for more details.

HEAD has 21 uploads less than BASE

Flag BASE (d304270) HEAD (73fbae2)

24 3

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2446       +/-   ##
===========================================
- Coverage   58.20%   28.97%   -29.23%     
===========================================
  Files        1069     1007       -62     
  Lines       74278    70705     -3573     
  Branches     2165     1253      -912     
===========================================
- Hits        43233    20488    -22745     
- Misses      29028    49015    +19987     
+ Partials     2017     1202      -815

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

eudoroolivares2016 · 2026-06-18T15:43:22Z

      ;; Log the successful ingest, with the metadata size in bytes.
      (api-core/log-concept-with-metadata-size concept-to-log request-context)
-      (api-core/generate-ingest-response headers save-granule-result))))
+      ;; (api-core/generate-ingest-response headers save-granule-result)


eudoroolivares2016 · 2026-06-18T15:45:18Z

-   (let [parsed-dif (c/parse-collection all-fields-collection-xml)]
-     (is (empty? (v/validate-collection parsed-dif))))))


This was the only place that c/parse-collection was called so its not actually being used in production

eudoroolivares2016 · 2026-06-18T16:39:56Z

  [variable validations-rules]
   (validation-errors->path-errors (v/validate validations-rules variable)))
+
+(comment (def foo (validation-errors->path-errors (v/validate vg/granule-validation-warnings g1)))


Remove the remaining rich comments

eudoroolivares2016 · 2026-06-18T19:29:05Z

    (v/validate (cons vv/variable-validations additional-validations)
                variable))))

-(defn validate-variable-warnings


This wasn't being used at all anywhere

looks like it is used in umm-spec-lib/src/cmr/umm_spec/validation but it looks like perhaps that func is not being used anywhere either...

eudoroolivares2016 · 2026-06-18T19:29:09Z

  [variable validations-rules]
   (validation-errors->path-errors (v/validate validations-rules variable)))
+
+(comment (def foo (validation-errors->path-errors (v/validate vg/granule-validation-warnings g1)))


Remove the remaining rich comments

jmaeng72 · 2026-06-19T18:30:17Z

  {:default []
   :parser #(map (comp keyword string/trim) (string/split % #","))})

+;; TODO we gotta make sure this is set to true before merging


the default is already true, why would it not be true?

Also this is a reminder to remove this ;; TODO comment before merging

jmaeng72 · 2026-06-19T18:31:58Z

 (def ENABLE_UMM_C_VALIDATION_HEADER "cmr-validate-umm-c")
 (def TESTING_EXISTING_ERRORS_HEADER "cmr-test-existing-errors")
+(def COLLECTION_WARNING_CONTEXT "After translating item to UMM-C the metadata had the following issue(s): ")
+(def COLLECTION_ERROR_CONTEXT "After translating item to UMM-C the metadata had the following existing error(s): ")


will there ever be a 'regular' error context in the future? If so, renaming this to be COLLECTION_EXISTING_ERROR_CONTEXT could be more beneficial

jmaeng72 · 2026-06-19T18:32:27Z

+            #{"[:TemporalExtents 0 :RangeDateTimes 0] BeginningDateTime [2001-01-01T12:00:00.000Z] must be no later than EndingDateTime [2000-05-11T12:00:00.000Z]"}})
+
+  (format-and-contextualize-warnings-existing-errors foo)
+  )


new line please

jmaeng72 · 2026-06-19T18:36:03Z

   [cmr.common-app.api.enabled :as common-enabled]
   [cmr.common-app.api.launchpad-token-validation :as lt-validation]
   [cmr.common.log :refer [info]]
+   [cmr.common.util :as util]


can you make this alphabetical, so move below cmr.common.services.errors

jmaeng72 · 2026-06-19T18:37:25Z

+      ;; (api-core/generate-ingest-response headers save-granule-result)
+      (api-core/generate-ingest-response headers (util/remove-nil-keys
+                                                (api-core/format-and-contextualize-warnings-existing-errors save-granule-result GRANULE_WARNING_CONTEXT nil)
+                                                               ))


fix paranthesis formatting

jmaeng72 · 2026-06-19T18:47:32Z

-                     (vu/has-parent-validator :short-name "Instrument short name")
-                     (v/every instrument-ref-validations)]})
+(def ^:private consistency-validations
+  "Granule-vs-parent-collection consistency validations. Enforced as errors


rename to Granule-vs-parent-collection-consistency-validations or granule-consistency-validations

jmaeng72 · 2026-06-19T18:48:04Z

-     set-geometries-spatial-representation
-     {:geometries (v/every sv/spatial-validation)
-      :orbit (v/when-present sv/spatial-validation)})])
+    ;; The spatial representation has to be set on the geometries before the conversion because


is this the right formatting? I would check all the files you changed to see if the indents are correct

jmaeng72 · 2026-06-19T18:48:57Z

+                                         :sensor-refs s-refs
+                                         :operation-modes ["Mode1" "Mode2"]})]
+    (g/map->PlatformRef {:short-name plat-short-name
+                         :instrument-refs [instr-ref]})))


new line between funcs

jmaeng72 · 2026-06-19T18:49:07Z

+                :existing-errors (map str ["e1" "e2"])}
+               "W: " "E: ")]
+      (is (= ["W: w1;; w2"] (:warnings out)))
+      (is (= ["E: e1;; e2"] (:existing-errors out))))))


new line at end

jmaeng72 · 2026-06-19T18:49:37Z

          ["<example>data</example>"]
          "<example>data</example>")))
+
+(deftest format-and-contextualize-warnings-existing-errors-test


when you build, make sure this test is actually executed and you see them in the logs

eudoroolivares2016 added 10 commits June 12, 2026 12:47

CMR-11242: Removing projects granule dependency

4efd103

CMR-11242: Removing hard granule dependencies for instruments; platfo…

1084b0d

…rms; and sensors

CMR-11242: Add warnings for granule collection references

7d5f7ad

CMR-11242: Starting cleanup

63b1ae6

CMR-11242: temp disable unit tests to get CI test build

fe6275c

CMR-11242: rebuild

0903752

CMR-11242: Remove debugging code to run unit tests

ed061c0

CMR-11242: Potentially fix some issues on the tests commenting out taps

791eb29

CMR-11242: Adding ingest test

73fbae2

CMR-11242: Start clean up

1fc5668

eudoroolivares2016 changed the title ~~CMR-11242: Adding Toggle for Granule collections validation~~ CMR-11242: Adding Toggle for Granule collections Reference validation Jun 18, 2026

eudoroolivares2016 added 2 commits June 18, 2026 11:06

CMR-11242: Cleanup

6a100ce

CMR-11224: Cleanup

f3027d6

eudoroolivares2016 commented Jun 18, 2026

View reviewed changes

eudoroolivares2016 requested review from daniel-zamora, eereiter, jceaser, jmaeng72 and zimzoom June 18, 2026 16:23

CMR-11242: Polishing

db2cd73

eudoroolivares2016 commented Jun 18, 2026

View reviewed changes

CMR-11242: Cleanup

61e2845

eudoroolivares2016 commented Jun 18, 2026

View reviewed changes

jmaeng72 requested changes Jun 19, 2026

View reviewed changes

		(let [parsed-dif (c/parse-collection all-fields-collection-xml)]
		(is (empty? (v/validate-collection parsed-dif))))))

Conversation

eudoroolivares2016 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What is the objective?

What are the changes?

What areas of the application does this impact?

Required Checklist

Additional Checklist

Uh oh!

codecov-commenter commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eudoroolivares2016 commented Jun 18, 2026 •

edited

Loading

codecov-commenter commented Jun 18, 2026 •

edited

Loading