robots.txt contains a Disallow /search?facets to prevent crawlers infinitely spinning out into facets, but that same pattern also prevents them from accessing pager pages. This is maybe preventing them from indexing beyond the first page of Browse results.
The sitemap_generator gem is used on the BTAA Geoportal (blacklight) to solve this, but could probably work here too https://rubygems.org/gems/sitemap_generator/versions/5.1.0
Geoportal's integration builds slugs with Blacklight /catalog/#{slug}, UMedia could do similarly with /item#{slug}
https://github.com/BTAA-Geospatial-Data-Project/geoportal/blob/develop/config/sitemap.rb
Geoportal's sitemap.xml.gz is about 650k - UMedia's might end up much much larger.
robots.txt contains a
Disallow /search?facetsto prevent crawlers infinitely spinning out into facets, but that same pattern also prevents them from accessing pager pages. This is maybe preventing them from indexing beyond the first page of Browse results.The
sitemap_generatorgem is used on the BTAA Geoportal (blacklight) to solve this, but could probably work here too https://rubygems.org/gems/sitemap_generator/versions/5.1.0Geoportal's integration builds slugs with Blacklight
/catalog/#{slug}, UMedia could do similarly with/item#{slug}https://github.com/BTAA-Geospatial-Data-Project/geoportal/blob/develop/config/sitemap.rb
Geoportal's sitemap.xml.gz is about 650k - UMedia's might end up much much larger.