Skip to content

refactor(go-website): connect list page to database - #5839

Merged
michaelkedar merged 2 commits into
🕸️🚫🐍-💽from
🕸️🚫🐍-💽2
Aug 25, 2026

Hidden character warning

The head ref may contain hidden characters: "\ud83d\udd78\ufe0f\ud83d\udeab\ud83d\udc0d-\ud83d\udcbd2"
Merged

refactor(go-website): connect list page to database#5839
michaelkedar merged 2 commits into
🕸️🚫🐍-💽from
🕸️🚫🐍-💽2

Conversation

@michaelkedar

Copy link
Copy Markdown
Member

Connect the list/search page to Datastore.

  • Made a new VulnerabilitySearchStore to provide needed methods for ecosystem counts, search results, and autocomplete.
    • have not connected Redis to the ecosystem counts, so these take a long time to populate (but they are locally cached). Is done in follow-up PR (next in stack)
  • Changed how the next-page logic works to use a after=timestamp_ID instead of a page number - page numbers require re-iterating through all previous results in the database to arrive at the next page, which becomes very inefficient later pages. Swapping to a timestamp/id lets us use the database more correctly.
    • Maintained page=N functionality (with a warning log) so we can see if people have been manually scraping our website

Stack created with GitHub Stacks CLIGive Feedback 💬

@michaelkedar
michaelkedar force-pushed the 🕸️🚫🐍-💽2 branch 2 times, most recently from 314405d to 31d27eb Compare August 24, 2026 05:48
@michaelkedar
michaelkedar force-pushed the 🕸️🚫🐍-💽 branch 2 times, most recently from 66453ec to 0669d9d Compare August 24, 2026 05:53

@another-rex another-rex left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments

<turbo-frame id="vulnerability-table-page-{{ .NextAfter }}" data-turbo-action="advance" target="_top" class="next-page-frame">
<div class="next-page-container">
<a class="next-page-button link-button" data-turbo-frame="_self" href="/list?page={{ add .Page 1 }}{{ if $.Query }}&q={{ $.EncodedQuery }}{{ end }}{{ if $.SelectedEcosystem }}&ecosystem={{ $.EncodedEcosystem }}{{ end }}">
<a class="next-page-button link-button" data-turbo-frame="_self" href="/list?after={{ .NextAfter }}{{ if $.Query }}&q={{ $.Query }}{{ end }}{{ if $.SelectedEcosystem }}&ecosystem={{ $.SelectedEcosystem }}{{ end }}">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it fine to not encode the query anymore? Why was this changed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment if this is intentional.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

html/template actually automatically encodes these, and we don't want to double-encode them.

}

// 3. Cold start fallback: fetch synchronously
return s.refreshEcosystemCounts(ctx)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a good place for singleflight, so we don't have a bunch of refreshes all running at the start of the server.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the singleflight change in #5840 (to avoid merge conflicts)

@michaelkedar
michaelkedar disabled the stack merge August 25, 2026 03:00
@michaelkedar
michaelkedar disabled the stack merge August 25, 2026 03:05
@michaelkedar
michaelkedar merged commit 336637c into master Aug 25, 2026
38 checks passed
@michaelkedar
michaelkedar deleted the 🕸️🚫🐍-💽2 branch August 25, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants