Experiment with pagefind for search

[savetheclocktower]

It might seem like this PR’s existence implies that I think #43 is lacking in some way. Not at all; I was just compelled to try out Pagefind and was pleasantly surprised at how well it solved this problem.

Here's what this PR does:

• Makes it so that every page build (via npm run build or npm run serve) also regenerates the search index (which is incredibly fast)
• Adds a search field to the page header
• Integrates Pagefind's UI into our existing theme (assigning its CSS variables to analogous variables in our CSS theme, whether we're in light mode or dark mode)
• Ensures only the main content of pages (rather than sidebars, page headers, or page footers) gets indexed
• Sets up Pagefind’s plugin for highlighting search terms after you click on results
• Allows it to crawl the latest version of the API docs… but not every version of the API docs, lest you get 17 near-identical results

I was also gratified that I managed to get this done without sprinkling any of Pagefind’s custom data attributes throughout the HTML. They say this is the “preferred” way of opting into indexing of certain pages and opting out of indexing others, but I was determined to avoid it if it could be helped. (Instead, I use CSS selectors to exclude certain portions of each page from being indexed, and use a silly glob to ensure only the latest API docs are indexed).

I owe a review of #43, so I'll get to that soon. But if we run into any snags with the search service, we might want to deploy this version as a stopgap, since it's better than having no search capability whatsoever!

[Daeraxa]

I'm a little confused by the intent of this PR I just want to be sure I understand:

• It is titled "experiment" - so is ready to land this for the live site or is it just to demonstrate in which case it should be a draft PR/WIP?
• Or is the intent to get this merged, as you say, as a stopgap whilst Add Search Capability #43 is still waiting for completion?
  • In which case would we be looking to integrate this into Add Search Capability #43 or is it to just be replaced entirely?

Either way I need to get back into the website and docs so I'll be having a proper look at this.

[Daeraxa]

First major issue I have noticed is that it reads the actual html underneath our platform switcher when showing the result preview:

<p>
  You can open the 
  <code>init.js</code> file in an editor from the 
    <span class="platform-linux"><em>Atom &gt; Init Script</em></span>
    <span class="platform-mac"><em>File &gt; Init Script</em></span>
    <span class="platform-win"><em>Edit &gt; Init Script</em></span>
  menu.
</p>

You can open the `init.js` file in an editor from the <span class="platform-linux">_Atom > Init Script_</span><span class="platform-mac">_File > Init Script_</span><span class="platform-win">_Edit > Init Script_</span> menu.

(On an unrelated note there is a docs error on this very line that has gone unnoticed for all this time that I shall be correcting imminently...)

[Daeraxa]

I guess this isn't necessarily unexpected but it isn't immediately clear whether a result is from the docs or from the API.

And I realise this might be beyond the goals of this PR if only meant to act as a stopgap but it might be worth being able to actually filter or group results by API or docs as it isn't going to be a common thing that you want to find results for both at the same time.
For example adding a docs: filter as either default or explicit when on the docs pages but api: when on the API page (or whatever that might look like in pagefind).

[Daeraxa]

I feel this one is kind of silly a bit bit petty but couldn't resist adding - dismiss at your leisure.

We could style the trigger keys to look a little more like rest of the <kbd> style we use (although it adds them as two separate spans). I just crudely pasted in parts of the rule that styles <kbd> in the dev tools:

[Daeraxa]

The "found" result markers don't really fit that well with the rest of the page theme but also their style is a bit of an issue where it overlaps nearby results (bottom left):

[savetheclocktower]

I'm a little confused by the intent of this PR I just want to be sure I understand:

• It is titled "experiment" - so is ready to land this for the live site or is it just to demonstrate in which case it should be a draft PR/WIP?

• Or is the intent to get this merged, as you say, as a stopgap whilst Add Search Capability #43 is still waiting for completion?

  • In which case would we be looking to integrate this into Add Search Capability #43 or is it to just be replaced entirely?

Either way I need to get back into the website and docs so I'll be having a proper look at this.

I started out just wanting to play around with this… and then thought the results were so good that it warranted feedback from others on whether it was good enough to ship. I figured it was possible @confused-Techie might've preferred it to what they'd been working on.

I think it's good enough to ship, but wouldn't want to do so without @confused-Techie’s feedback first (not trying to step on toes!).

[savetheclocktower]

I guess this isn't necessarily unexpected but it isn't immediately clear whether a result is from the docs or from the API.

Yeah, this is technically a bug in the pulsardoc generation, since the page titles just have the name of the class they're documenting and don't contextualize it.

[savetheclocktower]

First major issue I have noticed is that it reads the actual html underneath our platform switcher when showing the result preview:

There is one way of fixing this that would be pretty ridiculous, but it's at least possible:

pagefind allows you to create different “instances” of a search index (like if you wanted one for your blog and one for your docs, so that you could search each one in isolation). And UI widgets can be bound to specific instances.

So we could generate three different versions of the docs site search index — one for each platform — and could use the instance attribute on the custom elements to dynamically choose the instance corresponding to whichever platform is selected in the platform switcher. The version we generate for Linux, for example, could add .platform-win and .platform-mac to its ignored selectors list.

Pretty silly, but (a) it takes less than a second to generate a single index; (b) this problem will be present no matter which solution we pursue, so even a silly fix might be worth it!

[savetheclocktower]

The "found" result markers don't really fit that well with the rest of the page theme but also their style is a bit of an issue where it overlaps nearby results (bottom left):

It was just something I threw together, so it can certainly be improved. (But it does literally use theme colors!)

[savetheclocktower]

And I realise this might be beyond the goals of this PR if only meant to act as a stopgap but it might be worth being able to actually filter or group results by API or docs as it isn't going to be a common thing that you want to find results for both at the same time. For example adding a docs: filter as either default or explicit when on the docs pages but api: when on the API page (or whatever that might look like in pagefind).

This might eventually be solved by building different indexes for the API docs and the narrative docs, then selectively deciding whether to merge them together based on context. (I don't know yet whether a keyword like docs: or blog: is possible, but if nothing else we could certainly filter by instance if we figured out the UI.)

[Daeraxa]

I think it's good enough to ship, but wouldn't want to do so without @confused-Techie’s feedback first (not trying to step on toes!).

Agree, it is certainly an improvement over no search at all and it seems to work - all my comments are relatively minor.

It sounds like generating multiple indices would solve a few issues so if it isn't going to cause significant problems then I'd say go for it.

It was just something I threw together, so it can certainly be improved. (But it does literally use theme colors!)

Honestly I think its only really the tweak to the padding for the overlap issue but the colours, I'm guess I mean in terms of "brand" themeing as it seems the colour is the alert colour. If I was to pick I'd say stick close to the Pulsar themeing for find-and-replace with the one themes (similar to what has been done in terminal).

[savetheclocktower]

Yeah, this is technically a bug in the pulsardoc generation, since the page titles just have the name of the class they're documenting and don't contextualize it.

I fixed this; an API documentation page now says (API documentation) in the search result!

[savetheclocktower]

Honestly I think its only really the tweak to the padding for the overlap issue but the colours, I'm guess I mean in terms of "brand" themeing as it seems the colour is the alert colour. If I was to pick I'd say stick close to the Pulsar themeing for find-and-replace with the one themes (similar to what has been done in terminal).

We don't have existing colors like that in the shared theme across our three sites (main, blog, docs), so I'll invent one.