Misc import updates#26
Open
dogboat wants to merge 142 commits into
Open
Conversation
…x/3.0.200-3.1.0-dev Release: Merge back 3.0.200 into bugfix from: master-into-bugfix/3.0.200-3.1.0-dev
…ion (DefectDojo#14931) * Include analysis.detail from Dependency Track FPF in finding description When Dependency Track sends findings via the Finding Packaging Format, the analysis.detail field is now forwarded alongside analysis.state. This appends the audit detail text to the finding description under an "Audit Detail:" label, making analyst notes visible without switching back to Dependency Track. * Move audit detail to top of finding description * Revert "Move audit detail to top of finding description" This reverts commit c3494e1. --------- Co-authored-by: valentijnscholten <valentijnscholten@gmail.com>
…efectDojo#15104) Replace the outdated, Findings-only add_files.md (which incorrectly claimed image-only support) with an audience-scoped OS/Pro pair, following the OS__/PRO__ convention already used by the risk_acceptance docs. - OS__add_files.md: Manage Files workflow for Findings, Engagements, and Tests; corrected the supported file-type list to match DD_FILE_UPLOAD_TYPES; added fresh screenshots. - PRO__add_files.md: documents the Pro UI flow (gear > Add File, the Files tab, and the row menu's Edit File Name / Delete File). - Add OS_/PRO_ prefixed screenshots; preserve the old /add_files/ URL via an alias on the OS page. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…efectDojo#15136) The v3 Product->Asset relabel (DefectDojo#13155) centralises filter copy in dojo/asset/labels.py, gated by ENABLE_V3_ORGANIZATION_ASSET_RELABEL (default on). The Asset-level AND tag filter on the finding list (test__engagement__product__tags_and) was left with a hardcoded "Product Tags (AND)" label and help text, so it kept the legacy wording even with the relabel enabled -- inconsistent with its siblings "Test Tags (AND)" / "Engagement Tags (AND)" and with the OR variant, whose label is already set dynamically to "Tags (Asset)". Wire it to the edition-aware labels by adding ASSET_FILTERS_TAGS_ASSET_AND_LABEL and ASSET_FILTERS_TAGS_ASSET_AND_HELP to both vocabulary branches, so the field reads "Asset Tags (AND)" in v3 and remains "Product Tags (AND)" in the legacy edition. Adds a regression test asserting the rendered FindingFilter form field. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…DefectDojo#15137) Since bootstrap.min.css is no longer loaded, the essential Bootstrap dropdown rules were re-implemented by hand in components/tailwind.css. That compat block omitted the `.dropdown-menu-right` utility (right: 0; left: auto), so every menu marked `dropdown-menu-right` (42 usages across 30+ templates, e.g. the finding actions menu) fell back to left-alignment and overflowed off the right edge of the page, clipping its labels and forcing horizontal page scroll. Restore the rule in the Bootstrap-compat block and rebuild tailwind-out.css (tailwindcss 4.2.4, matching the committed artifact). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ATURE_LOCATIONS (DefectDojo#15139) Creating a Finding Template from a finding or merging findings crashed with NotImplementedError when V3_FEATURE_LOCATIONS is enabled and the finding still carries legacy Endpoint rows (kept as backup by the locations migration). Both sites in dojo/finding/views.py iterated finding.endpoints.all() without the Endpoint.allow_endpoint_init() escape hatch used by the other legacy endpoint call sites. Fixes DefectDojo#15123. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…v1.35.6 (.github/workflows/k8s-tests.yml) (DefectDojo#14892) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* make impact_path deterministic * docs: add 3.1 upgrade note for deterministic JFrog Xray impact paths The JFrog Xray API Summary Artifact parser now sorts impact paths before selecting the first one, so file_path, description, and unique_id_from_tool (the deduplication key) stay stable across re-imports. Previously JFrog's arbitrary impact_path ordering could change the dedup key between scans and cause a single CVE to be re-imported as multiple separate findings. Document the behavior change and cleanup guidance in the 3.1 upgrade notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…xing (DefectDojo#15236) Master switch (default True, so OSS behaviour is unchanged). When False: watson models are not registered (no post-save index writes), the watson app + AsyncSearchContextMiddleware are stripped from settings, /simple_search returns 410, and installwatson is skipped (both the complete_initialization command and entrypoint-first-boot.sh). Lets deployments that have replaced watson (e.g. DefectDojo Pro's native Postgres search) stop paying the indexing cost and reclaim watson_searchentry. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
docs: redesign documentation site UI
…n caching (DefectDojo#15066) * feat(cache): shared two-tier (L1+L2) read-through cache for singleton config - new dojo/caching.py: dojo_settings_cache(key, timeout) read-through over an in-process L1 tier (off|thread|process, TTL'd) on top of django.core.cache (L2). Resolves L1 -> L2 -> getter; None is never cached. invalidate_dojo_settings_cache; model_to_cache_dict/cache_dict_to_model (store dicts, not pickled instances). Configurable: DD_SETTINGS_CACHE_ENABLED / DD_CACHE_L1_MODE / DD_CACHE_L1_TTL / DD_CACHE_L2_TTL (-1 disables a tier). - System_Settings read path served through the cache (dict, rebuilt per call); write paths (no_cache=True) unchanged; busted on save via post_save receiver. - disabled in unit tests (compose overrides) so assertNumQueries counts stay deterministic; the cache itself is covered by unittests/test_caching.py. No new dependencies; no behavior change on non-cache paths. * feat(cache): wire shared Redis cache backend for L2 (cross-process invalidation) The L1+L2 singleton cache (dojo/caching.py) needs its L2 tier to be a cross-process store so cache invalidation (e.g. on System_Settings save in uwsgi) propagates to celery workers. Without it Django defaults to per-process LocMemCache, so a worker keeps serving a stale System_Settings (e.g. enable_deduplication=False) and async dedup silently no-ops -> dedupe/ close_old_findings_dedupe/questionnaire integration tests fail. - Add DD_CACHE_URL env; when set, configure django.core.cache RedisCache. - Default DD_CACHE_URL=redis://valkey:6379/1 in compose (uwsgi/celery/beat). - Mount source into the integration celeryworker so it runs the same code as uwsgi (enables local reproduction of worker-side cache behavior). * fix(ci): drop integration celeryworker source mount (media volume race) Sharing defectdojo_media_integration_tests between uwsgi and celeryworker races on mkdir media/threat at startup (file exists), flaking the worker container. The mount was only a local-repro aid and is unnecessary in CI, where the worker runs the image built from PR source. Revert to the original worker definition; the Redis L2 fix is unaffected. * fix(cache): don't hit L2 backend on invalidate when L2 disabled Unit tests run with no Redis (DD_SETTINGS_CACHE_L2_TTL=-1) but inherited DD_CACHE_URL from base compose, so System_Settings post_save invalidation called cache.delete against a nonexistent valkey -> rest-framework setUpClass ConnectionError. invalidate_dojo_settings_cache now skips cache.delete when L2 is disabled (consistent with the read path, which already skips L2). Also blank DD_CACHE_URL in the unit-test overrides so the backend is LocMemCache. Adds test_invalidate_skips_l2_when_disabled. * refactor(cache): slim System_Settings middleware to L1-reset + error banner The decorator (dojo.caching) now owns all System_Settings caching, so the middleware no longer keeps its own per-request thread-local instance. It only: (a) resets the request-scoped L1 tier at request start, (b) surfaces a DB-read error for the banner. Renamed DojoSytemSettingsMiddleware -> DojoSettingsManagerMiddleware (it governs the shared settings cache lifecycle, not just System_Settings). Manager.get() reads straight through the cache. To keep assertNumQueries deterministic without the thread-local, unit tests now run with L1 ON (request/test-scoped) and L2 OFF instead of both disabled: L1 is reset per request (middleware) and per test (dojo_test_case setUp via the new refresh_system_settings_cache helper, replacing middleware.load()). This mirrors the old thread-local's per-request memoization. Rewrote the middleware tests for the new contract (cache via decorator, instance rebuilt per call, L1 reset, error banner). * test: rebaseline query counts for per-task L1 cache reset The L1 settings cache is reset per celery task (DojoAsyncTask), so eager-mode tests now faithfully mirror real async workers: each task re-reads System_Settings (from L2) instead of sharing one request-wide read as the old middleware thread-local did. assertNumQueries baselines shift accordingly: - Single cached reads now save a query (metrics 27->25/41->40, fp_history 7->6). - Bulk create/import where many per-finding tasks run inline read once per task (tag_inheritance create_100 3124->3224; zap import/reimport +3-4; importer performance +2-8). In production these run in workers, off the request path. Counts verified locally against the CI unit-test cache config (L1 on, L2 off). * test: pin L2 off via override_settings on query-count tests The query-count assertions must measure in-process (L1) behavior deterministically, independent of whether a shared L2 (Redis) cache is configured/warm in the run environment. Relying on the unit-test compose env (DD_SETTINGS_CACHE_L2_TTL=-1) left the counts env-dependent (e.g. running via run-unittest in dev mode, or the manage.py-test perf path, would see L2 on). Add an explicit @override_settings(SETTINGS_CACHE_L1_TTL=30, SETTINGS_CACHE_L2_TTL=-1) on the count-asserting classes so they are self-contained. No count change (compose already disabled L2 in CI). * feat(cache): drop L2 (shared) tier for singleton settings cache; L1-only The singleton settings cache (dojo/caching.py) becomes a single in-process L1 tier; the shared django.core.cache (L2) tier is removed. Cross-process/worker freshness is provided entirely by resetting L1 at each request and task boundary (middleware + DojoAsyncTask), so each request/task reads a singleton from the DB at most once and never serves a value cached during a prior request/task. Why no L2 for settings singletons: - The dedupe correctness fix only needs each task to see current config, which the per-task L1 reset already guarantees by re-reading the DB -- a shared L2 + cross-process invalidation is not required for correctness. - L2 added real cost and complexity for little gain on these tiny, hot singletons: a Redis dependency, pickled-vs-dict storage and version-keying, cross-process invalidation, and Redis-down fragility -- while the headline win (avoiding repeated reads within a bulk loop) comes from L1, not L2. - Trade-off: one small indexed DB read per singleton per request/task instead of amortising across processes via Redis. Acceptable for low-cardinality config. Removes SETTINGS_CACHE_L2_TTL and the L2 read/set/delete paths. DD_CACHE_URL and the Redis default django cache backend stay (used by other framework caching). Unit-test overrides drop the now-defunct L2 TTL var. test_caching rewritten to the L1-only contract. * feat(cache): guard against saving cache-derived System_Settings snapshots System_Settings.objects.get() returns a read-only snapshot rebuilt from the in-process (L1) cache; saving it could overwrite concurrent changes with stale field values. Tag such instances and make System_Settings.save() fail loudly instead of silently persisting a stale snapshot. - dojo/caching.py: add READ_ONLY_CACHE_MARKER, ReadOnlyCachedInstanceError, and a read_only= param on cache_dict_to_model (default off, so tuner/features keep saving their rebuilt instances for their read-modify-write flows). - System_Settings.save() raises ReadOnlyCachedInstanceError when the instance is tagged; get_cached_system_settings() tags the snapshot read_only=True. - Migrate get()->modify->save() writer sites (incl. the system_settings() test helper) to objects.get(no_cache=True) for a fresh, saveable row. - Document the singleton manager .get() semantics (ignores filter kwargs, never raises DoesNotExist; data migrations use apps.get_model, so get_or_create is unaffected). Fix stale L1/L2 comments left by the drop-L2 change. - test_caching: cover the save-guard and the read_only flag. * feat(cache): cache authorizations in Celery tasks via cache_for_request_or_task @cache_for_request is a no-op in Celery (no request), so per-user authorization filtering re-runs on every call inside a task -- notably the rules engine, which runs authorized querysets under impersonate(rule.owner). Add a request-or-task read-through cache and move the authorized-queryset/permission getters onto it. - dojo/request_cache: cache_for_request_or_task resolves the store as request cache -> task cache -> no caching. The task cache is a per-thread store active only inside a DojoAsyncTask (begin/end), so management commands / shells that never hit a task boundary do not cache. The key folds get_current_user().pk so results keyed on user=None (resolved downstream) cannot leak between users on a reused worker thread or within a task that impersonates multiple owners. - dojo/celery: DojoAsyncTask installs a fresh task cache before each task and drops it afterwards (finally), bounding staleness to a single task. - Migrate all get_authorized_* / authorized_*_id_set / get_*_permissions getters from @cache_for_request to @cache_for_request_or_task. - test_caching: cover task-scope caching, per-user isolation, and reset. * test(perf): align importer perf query counts to CI-measured values The counts resolved during the rebase onto dev were the branch's pre-rebase values; ~90 commits of upstream drift shifted the actuals by +1..+3 per step. Update TestDojoImporterPerformanceSmall / *SmallLocations expected_num_queries to the values reported by the CI test-performance job. (The auth-getter cache migration does not touch importer paths, so it contributes no delta here.) * feat(cache): add opt-in cache_none to dojo_settings_cache By default a None result is not cached (so the next call retries). Pass cache_none=True for getters where None is a legitimate steady-state answer (e.g. "no default row configured"): it caches the None via a sentinel too, so a missing row costs one DB read per request/task instead of one per call site. Safe only when the getter is signal-invalidated on the row's create/save. * test: correct ZAP import query counts after dev merge (CI-verified) The additive merge estimate for EXPECTED_ZAP_IMPORT_V2/V3 was off by one on the import path; set to the values reported by CI (294/318). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ectDojo#15238) Add FINDING_FALSE_POSITIVE_HISTORY_CANDIDATE_FILTER_METHOD hook to do_false_positive_history_batch. When configured, the per-finding candidate list resolved by the deduplication algorithm is passed through the plugin method before the FP decision, letting a plugin (Pro) narrow candidates by fields excluded from the hash string but compared per pair (set-match tokens on vulnerability_ids / CWEs). Resolved once via get_custom_method; a no-op when unset, so default behavior is unchanged. Tests: hook can suppress matches (drops candidates -> no FP replication) and passthrough matches the default path. Co-authored-by: Valentijn Scholten <valentijnscholten@gmail.com>
…efectDojo#15242) Standardize every reference to the company name as "DefectDojo, Inc." across the docs site content, config, and layout templates. Previously the name appeared as "DefectDojo Inc" or "DefectDojo Inc." without the comma. Possessive uses become "DefectDojo, Inc.'s". Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…efectDojo#15233) ImportScanForm and ReImportScanForm declared their finding-group fields (group_by, create_finding_groups_for_all_findings) in the class body guarded by is_finding_groups_enabled(). That helper reads a DB-backed system setting, so merely importing dojo.forms issued a SELECT. When the module is imported during app initialization (e.g. from an AppConfig ready() hook), Django raises: RuntimeWarning: Accessing the database during app initialization is discouraged. Move the conditional field construction into each form's __init__, so importing the module never touches the database. Behavior is unchanged: instances still gain the fields when finding groups are enabled, the fields keep their original position (appended last), and the empty default choice is still inserted. Evaluating the setting per-instance instead of once at import is also more correct if the setting changes. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o#15210) Covers instance setup (REST API key, From user email, regional API base URL), the service ID mapping identifier, the urgency vs account-priority severity mapping modes, status mappings, and the PagerDuty-specific caveats (resolved is final; title/body cannot be edited after creation). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…al runs) (DefectDojo#15234) * fix(docker): silence shell auto-import banner in DB-reach / first-boot scripts reach_database.sh and entrypoint-first-boot.sh pipe short scripts into `manage.py shell` (a DB connectivity check and superuser creation). Since Django 5.1 the shell command auto-imports every model on startup and prints a banner like: 36 objects could not be automatically imported: dojo.auditlog.services.ObjectsProductTags ... 237 objects imported automatically (use -v 2 for details). The "could not be imported" entries are dynamically-generated Tagulous tag models and auditlog proxies that aren't importable by their dotted path, so the list looks alarming on every container start even though nothing is wrong. Both scripts import exactly what they need, so pass --no-imports (Django >= 5.2) to skip the auto-import and drop the noise. The piped code still runs and exit codes are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(shell): drop non-importable models from shell auto-import Manual `manage.py shell` runs still showed the auto-import failure banner (the docker --no-imports change only covered the DB-reach / first-boot scripts). Override the shell command's get_auto_imports() to keep only paths that actually import, so dynamically-generated Tagulous tag models and auditlog proxy models no longer appear as "could not be automatically imported". Real models still auto-import; the filter is generic (tries the import, keeps what succeeds) so it also covers Pro's proxy models. Adds unittests/test_shell_command.py asserting every returned path imports, real models are kept, and the dropped paths are all genuinely non-importable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…15260) * docs: redesign documentation site UI Presentation-only redesign of docs.defectdojo.com. The rendered text of all 612 pages is byte-identical to the previous build (verified by diffing extracted page text across before/after builds); no content files are touched. Highlights: - Token-based design system (CSS custom properties for light and dark) using the DefectDojo brand palette, with navy header and footer shared with trust.defectdojo.com - Code blocks restyled as navy cards in both color modes by retheming the doks expressive-code frame variables: language labels in the header band, terminal dots for shell fences, styled copy button, and a brand-tuned Chroma syntax palette - The Open Source / Pro version toggle is rebuilt from a select dropdown into an accessible segmented control (radiogroup semantics). Behavior is unchanged: same localStorage key and values, anti-flash reveal, edition-aware "Model Your Assets" nav link, and instances stay in sync across the desktop sidebar and mobile offcanvas - Reading experience: 16px root font size (was 85 percent), 72ch measure, refined heading scale, framed screenshots, styled tables and callouts - Mobile fixes: the header now shows the brand on small screens, and the offcanvas navigation actually opens. The site disables the full Bootstrap bundle (doks.bootstrapJavascript = false), so the offcanvas plugin never loaded and mobile nav buttons were dead on the live site; custom.js now imports just the Offcanvas plugin - Homepage: navy hero with blueprint grid, prominent DocSearch trigger, quick-nav cards overlapping the hero edge; all copy unchanged Note: netlify.toml pins HUGO_VERSION=0.125.1, which cannot build doks-core 1.8.3 (it requires Hugo 0.146+ for layouts/_partials). The production pipeline evidently uses a newer Hugo already; this change was built and verified with Hugo 0.148.1 extended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix active nav link contrast on navy header The main nav row lives inside #offcanvasNavMain but outside .offcanvas-body in this header, so the white active-link rule never matched and the theme's Fuji Blue applied, which is hard to read on the navy band. Scope the desktop rules to .navbar-nav directly (active link is now white on a blue tint) and correct the mobile offcanvas scope the same way (link blue on the light panel). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: remove baked-in canvas from dark-mode logo The dark logo SVG is an auto-trace that carries its background as nine opaque fill paths (#212529), invisible on the old dark-gray header but visible as a box on the navy redesign. Delete the background paths and recolor the seven letter counters and pixel notches to the navy surface color so the wordmark renders clean in the header and footer. Same fix as applied to the trust-center copy of this logo; the SVG is still not truly transparent inside the letter counters, so reusing it on non-navy backgrounds would need a proper even-odd hole merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * stop os/pro switch flicker on page load * docs: let hero search button take its accessible name from visible text AccessLint flagged the label-in-name mismatch: aria-label said "Search documentation" while the visible text says "Search the docs", breaking voice-control activation. Drop the aria-label so the visible label is the accessible name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix sidebar clipping and focus-ring artifacts The theme bleeds .docs-links 24px past its column (margin-right: -24px), which pushed every sidebar item outside the version-menu scroll wrapper; combined with the wrapper's overflow-x clipping, long labels, Pro badges, and the active pill were cut off at the right edge. Zero the bleed inside the wrapper and add slight right padding. Also: clicked summaries keep :focus-visible in most browsers, and the clipped outline ring rendered as stray horizontal lines around collapsed groups in dark mode. Replace the ring with a filled focus state, and ease group-label letter-spacing so long edition names wrap cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: keep breadcrumbs on a single line Bootstrap breadcrumbs prefix each crumb with its "/" separator, so when the trail wrapped the separator dangled at the start of the next line. Keep the trail on one line and ellipsize long crumbs instead: "Home" never shrinks, ancestors compress to readable stubs, and the current page keeps roughly twice the room. The active item renders as a block because text-overflow does not draw an ellipsis on a flex container. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update versions in application files * docs: add Pro changelog entry for 3.1.100 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump django from 5.2.14 to 5.2.15 (DefectDojo#15230) Bumps [django](https://github.com/django/django) from 5.2.14 to 5.2.15. - [Commits](django/django@5.2.14...5.2.15) --- updated-dependencies: - dependency-name: django dependency-version: 5.2.15 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matt Tesauro <mtesauro@gmail.com> * docs: add ServiceDesk Plus to the Pro Integrations tool reference (DefectDojo#15217) Covers both editions (on-premises technician key and cloud Zoho OAuth with the self-client walkthrough and regional token endpoints), the group-name mapping identifier, priority/status name mappings, and the SDP-specific behaviors (full content sync on update, close-on-delete, mandatory-on-closure caveat). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> * docs: add Zendesk to the Pro Integrations tool reference (DefectDojo#15215) Covers instance setup (agent email + API token basic auth, subdomain URL), the numeric group ID mapping identifier, priority and status mappings, and the Zendesk-specific caveats (immutable description, solve-on-delete, closed is terminal). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> * feat(search): add DD_WATSON_SEARCH_ENABLED toggle to gate watson indexing (DefectDojo#15236) Master switch (default True, so OSS behaviour is unchanged). When False: watson models are not registered (no post-save index writes), the watson app + AsyncSearchContextMiddleware are stripped from settings, /simple_search returns 410, and installwatson is skipped (both the complete_initialization command and entrypoint-first-boot.sh). Lets deployments that have replaced watson (e.g. DefectDojo Pro's native Postgres search) stop paying the indexing cost and reclaim watson_searchentry. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * add OS Findings documentation (DefectDojo#15235) * docs: use "DefectDojo, Inc." consistently in company name references (DefectDojo#15242) Standardize every reference to the company name as "DefectDojo, Inc." across the docs site content, config, and layout templates. Previously the name appeared as "DefectDojo Inc" or "DefectDojo Inc." without the comma. Possessive uses become "DefectDojo, Inc.'s". Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(forms): avoid DB access at import time in Import/ReImport forms (DefectDojo#15233) ImportScanForm and ReImportScanForm declared their finding-group fields (group_by, create_finding_groups_for_all_findings) in the class body guarded by is_finding_groups_enabled(). That helper reads a DB-backed system setting, so merely importing dojo.forms issued a SELECT. When the module is imported during app initialization (e.g. from an AppConfig ready() hook), Django raises: RuntimeWarning: Accessing the database during app initialization is discouraged. Move the conditional field construction into each form's __init__, so importing the module never touches the database. Behavior is unchanged: instances still gain the fields when finding groups are enabled, the fields keep their original position (appended last), and the empty default choice is still inserted. Evaluating the setting per-instance instead of once at import is also more correct if the setting changes. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add PagerDuty to the Pro Integrations tool reference (DefectDojo#15210) Covers instance setup (REST API key, From user email, regional API base URL), the service ID mapping identifier, the urgency vs account-priority severity mapping modes, status mappings, and the PagerDuty-specific caveats (resolved is final; title/body cannot be edited after creation). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> * fix(shell): silence the shell model auto-import banner (docker + manual runs) (DefectDojo#15234) * fix(docker): silence shell auto-import banner in DB-reach / first-boot scripts reach_database.sh and entrypoint-first-boot.sh pipe short scripts into `manage.py shell` (a DB connectivity check and superuser creation). Since Django 5.1 the shell command auto-imports every model on startup and prints a banner like: 36 objects could not be automatically imported: dojo.auditlog.services.ObjectsProductTags ... 237 objects imported automatically (use -v 2 for details). The "could not be imported" entries are dynamically-generated Tagulous tag models and auditlog proxies that aren't importable by their dotted path, so the list looks alarming on every container start even though nothing is wrong. Both scripts import exactly what they need, so pass --no-imports (Django >= 5.2) to skip the auto-import and drop the noise. The piped code still runs and exit codes are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(shell): drop non-importable models from shell auto-import Manual `manage.py shell` runs still showed the auto-import failure banner (the docker --no-imports change only covered the DB-reach / first-boot scripts). Override the shell command's get_auto_imports() to keep only paths that actually import, so dynamically-generated Tagulous tag models and auditlog proxy models no longer appear as "could not be automatically imported". Real models still auto-import; the filter is generic (tries the import, keeps what succeeds) so it also covers Pro's proxy models. Adds unittests/test_shell_command.py asserting every returned path imports, real models are kept, and the dropped paths are all genuinely non-importable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update versions in application files --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Greg Anderson <greg.anderson@owasp.org> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Paul Osinski <posinski34@gmail.com> Co-authored-by: DefectDojo release bot <dojo-release-bot@users.noreply.github.com> Co-authored-by: Ross E Esposito <ross@defectdojo.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Paul Osinski <42211303+paulOsinski@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matt Tesauro <mtesauro@gmail.com> Co-authored-by: Blake Owens <76979297+blakeaowens@users.noreply.github.com> Co-authored-by: dangoelz <dangoelz@gmail.com>
….1.101-3.2.0-dev Release: Merge back 3.1.101 into dev from: master-into-dev/3.1.101-3.2.0-dev
…hub/workflows/validate_docs_build.yml) (DefectDojo#15256) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…thub/workflows/close-stale.yml) (DefectDojo#15254) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…o#15253) Bumps [django-polymorphic](https://github.com/django-commons/django-polymorphic) from 4.11.5 to 4.11.6. - [Release notes](https://github.com/django-commons/django-polymorphic/releases) - [Commits](django-commons/django-polymorphic@v4.11.5...v4.11.6) --- updated-dependencies: - dependency-name: django-polymorphic dependency-version: 4.11.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [django-htmx](https://github.com/adamchainz/django-htmx) from 1.27.0 to 1.28.0. - [Changelog](https://github.com/adamchainz/django-htmx/blob/main/docs/changelog.rst) - [Commits](adamchainz/django-htmx@1.27.0...1.28.0) --- updated-dependencies: - dependency-name: django-htmx dependency-version: 1.28.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…thub/workflows/pr-labeler.yml) (DefectDojo#15247) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
… v3.0.2 (.github/workflows/release-x-manual-helm-chart.yml) (DefectDojo#15246) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
….0 to 9.1.0-alpine (docker-compose.yml) (DefectDojo#15245) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
….6 to 3.14.6-slim-trixie (dockerfile.integration-tests-debian) (DefectDojo#15244) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…2 (.github/workflows/validate_docs_build.yml) (DefectDojo#15243) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…fectdojo/chart.yaml) (DefectDojo#15255) * chore(deps): update valkey docker tag from 0.23.0 to v0.24.1 (helm/defectdojo/chart.yaml) * update Helm documentation --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.50 to 3.1.51. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.50...3.1.51) --- updated-dependencies: - dependency-name: gitpython dependency-version: 3.1.51 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.20 to 0.15.21. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.20...0.15.21) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.21 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [django-filter](https://github.com/carltongibson/django-filter) from 25.2 to 26.1. - [Release notes](https://github.com/carltongibson/django-filter/releases) - [Changelog](https://github.com/carltongibson/django-filter/blob/main/CHANGES.rst) - [Commits](carltongibson/django-filter@25.2...26.1) --- updated-dependencies: - dependency-name: django-filter dependency-version: '26.1' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ence behavior (DefectDojo#15241) Two behaviors surfaced in review of the Pro dedup work: - Two different tools set to the same instance-wide algorithm (Global Component or Global Vulnerability ID) share a constant grouping hash and become mutual deduplication candidates (F23) — document as intended cross-tool behavior, and add the missing Global Vulnerability ID algorithm entry. - A tool's saved Deduplication Settings stay in effect regardless of feature-flag state; disabling a flag does not retroactively revert dedup (F20). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…metry + location docs (DefectDojo#15171) * feat(locations): LocationData.code factory + get_locations() hash symmetry Adds a LocationData.code() factory so parsers can emit static-analysis code coordinates as first-class locations: identity is file_path (+ line), while volatile context (snippet, end_line, sink/source metadata) is carried separately and omitted when unset, so downstream location subtypes can route it onto the finding reference instead of the location identity. Also fixes an asymmetry in Finding.get_locations(): the saved path filters a finding's location references to URL type before feeding the "endpoints" hash ingredient, but the unsaved path hashed every unsaved location type (dependencies included). A finding whose hash was computed before save could therefore never match its own hash recomputed after save. The unsaved path now applies the same URL-type filter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(locations): document source code locations and location drift matching Two new pages plus cross-links from the existing docs: - asset_modelling/locations/PRO__source_code_locations.md — the Code location subtype: file+line identity, scan-managed lifecycle, the All Source Code / asset-scoped views, the movement-history trail, and the toggle-independent reference status sync. - triage_findings/finding_deduplication/PRO__location_drift_matching.md — the full feature guide: the churn problem, enabling the per-tool toggle, the two-stage matching model (stable identity + evidence passes per location type), severity re-score tolerance, scan-owned field refresh with human-edit protection, scan-pure hash adoption, location history, upgrade/enablement guidance for existing data, hash-field guidance (all-volatile fallback, location-in-title), the dependency dedupe stable-key interaction, the historical churn consolidation command, and the safeguards/limits. - PRO__deduplication_tuning.md — Content Fingerprint hash field (with the backfill-before-select ordering) and the Track Findings as Locations Change toggle, plus best-practice entries. - PRO__locations_overview.md — Source Code Locations join the subtype list. - import_data/import_intro/reimport.md — cross-link from the "line number shift" troubleshooting note to the drift matching page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(locations): parsers emit LocationData.code for source-code findings Follows the method PR DefectDojo#14395 established for dependencies: rather than downstream synthesis from finding scalars alone, static parsers now emit their code coordinates as LocationData.code() into unsaved_locations, gated on V3_FEATURE_LOCATIONS, alongside the unchanged scalar fields. Context the parser already extracts rides along (snippet, end_line, source/sink objects, taint source file/line) so location subtypes can route it onto the finding reference instead of the location identity. Coverage: 45 parser directories converted (~60 emission sites) — SAST, secret scanners, and IaC scanners. SCA parsers are deliberately skipped (their file_path is a dependency manifest; they already emit LocationData.dependency), as are dynamic findings and non-source paths (container-image filesystem hits, JIRA/Confluence URLs, cloud service names). Mixed parsers (trivy, checkmarx, aws_inspector2, wizcli, xygeni, mobsf, deepfence, rusty_hog, snyk_issue_api) emit only for their code-shaped findings. Multi-location: SARIF explodes results into one finding per location, each with its own emission; xygeni secrets emits one location per leaked line. Fabricated line numbers (trufflehog's line=0 dedup placeholder) are never propagated — real coordinates only. Secret scanners never pass matched secret content as snippets. unittests/test_code_location_emission.py exercises nine representative parsers against real fixtures: emission coherence with the finding scalars, snippet context, SARIF per-location behavior, and a gate-off class proving the sweep is inert without V3 (full unittests.tools suite: 1531 tests green with V3 off). Known follow-ups (out of scope): mend and meterian have no dependency emission from DefectDojo#14395 either; github_vulnerability's code-scanning branch parses no location fields at all today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(locations): LocationData.code robust to unhashable context; progpilot scalars CI (the V3-on rest-framework matrix variant) caught two coupled defects the local V3-off run could not, since the emission only fires under V3: - LocationData.code filtered unset context with `value not in {"", None}` — a SET membership test that hashes `value`, so any unhashable value crashed with "cannot use 'list' as a set element". Switched to tuple membership (`in ("", None)`), which compares by equality and never hashes — robust for every call site, present and future. - progpilot reports taint-source fields as single-element arrays (source_name=["$sql"], source_line=[610]); the sweep passed those lists straight into the context kwargs. Now collapsed to scalars via a _first() helper, with str()/int guards. Audited every other converted parser's context kwargs: all pass .text/int/ string scalars (checkmarx pathnode elements, sarif get_snippet, etc.) — progpilot was the only list-valued source. Verified against the full unittests.tools suite run with V3_FEATURE_LOCATIONS=True. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(locations): lint-clean the unset-context filter Replace the crash-prone set-membership filter with a truthiness dict comprehension: never hashes the value (crash-robust for any parser input), and satisfies the repo's ruff rules (no manual loop, no != "" comparison). Drops None/""/0/empty — all "no data" for the code location's context fields. Same fix, lint-clean form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(locations): pin the get_locations() unsaved-path URL filter with a raw AbstractLocation instance The existing symmetry tests feed LocationData, which the URL-only LocationManager already drops during cleaning — they pass even without the filter. Only a raw AbstractLocation instance (a plugin's Dependency/CodeLocation model object, passed through untyped by make_abstract_locations) reaches the hash comprehension, so this case is the one that actually fails without the fix: without the type filter its value leaks into the endpoints ingredient and drifts the pre-save hash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(lint): hoist test imports to module top for CI ruff (PLC0415) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
114 commits of dev drift resolved: - default_importer/default_reimporter: dev partitioned post-processing dispatch into per-push_to_jira batches; this branch's product_grading_option=not self.defer_product_grading carried into the partitioned loop. - middleware: kept BOTH independent additions — this branch's suppress_intermediate_flush() thread-local kill-switch AND dev's new watson_search_context_for_task(); the suppression check ported into dev's instance-level add_to_context_with_flush (the old class-level singleton guard is obsolete under dev's install). - test files: kept this branch's suppress test; merged both mock import styles in test_importers_closeold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test