I want to write a routine in C that fetches a URI.

[ArthurPeabody]

I browse with lynx. It does not support JS. Many sites serve their pages only with JS, pages that don't require JS to render, just to fetch. I wrote a short JS script that fetches such pages, which I run with node, then render with lynx. I would like to embed this in lynx.

[shwetankg07]

One thing worth knowing before writing any C: Node's fetch doesn't run JavaScript either. It downloads the raw HTML, same as lynx. So if your Node script gets the page and lynx doesn't, the site isn't actually demanding JS. It's rejecting something about lynx's request, and the usual suspect is the User-Agent (plenty of sites block unknown or text-mode browsers), with TLS or HTTP/2 handling after that.

So try the cheap fix first:

lynx -useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" https://the-site

If that works, you can set it permanently in lynx.cfg (USERAGENT:).

If you still want a C fetcher, libcurl is the standard way and it's short. This fetches a URL, follows redirects, sends a browser User-Agent, decodes gzip/br, and prints the body to stdout:

#include <stdio.h>
#include <curl/curl.h>

int main(int argc, char **argv) {
    if (argc < 2) { fprintf(stderr, "usage: %s URL\n", argv[0]); return 2; }
    CURL *c = curl_easy_init();
    if (!c) return 1;
    curl_easy_setopt(c, CURLOPT_URL, argv[1]);
    curl_easy_setopt(c, CURLOPT_FOLLOWLOCATION, 1L);
    curl_easy_setopt(c, CURLOPT_USERAGENT,
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36");
    curl_easy_setopt(c, CURLOPT_ACCEPT_ENCODING, "");  /* accept gzip/br and decode */
    CURLcode rc = curl_easy_perform(c);                /* body goes to stdout by default */
    if (rc != CURLE_OK) fprintf(stderr, "fetch failed: %s\n", curl_easy_strerror(rc));
    curl_easy_cleanup(c);
    return rc != CURLE_OK;
}

cc fetch.c -o fetch $(curl-config --cflags --libs)
./fetch https://the-site | lynx -stdin

-stdin makes lynx render whatever you pipe into it, so you can plug this in without patching lynx itself. The same pipe works with your existing script: node fetch.js URL | lynx -stdin. (I compiled and ran the C above against a test page before posting.)

The one caveat: pages that really do build their content with JS (an empty <div id="root"> plus a bundle) won't render with any of these. That takes a headless browser, not a fetcher.

[ArthurPeabody]

Thanks. I was taking their word when they responded, 'Please enable JavaScript to continue.', and noticed that, when I disable JS in Firefox I can't access the same site. curl also doesn't fetch. I've been using lynx for 30 years, know about modifying the useragent header. Mine looked like your recommended but with the addition of l_y_n_x which lynx warns against removing and which I had followed but seems to be the sticking point for 1 domain at least.