Skip to content

fix: kill hung queries at 25 seconds and stop tile requests hanging forever on database errors #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arnoldcastro5000 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix: kill hung queries at 25 seconds and stop tile requests hanging forever on database errors #113

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72e87b2
fix: add statement_timeout and propagate errors from buildMapInstance
arnoldcastro5000 Jul 3, 2026
3905914
Lower default statement_timeout to 25s so it fires before the 30s Clo…
arnoldcastro5000 Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 37 additions & 19 deletions greenstand/app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,28 @@ const {Config} = require("./config");

const connectionString = process.env.DB_URL;
const max = process.env.PG_POOL_SIZE && parseInt(process.env.PG_POOL_SIZE) || 10;
log.info("pool settings:db:%s; pool size: %d", connectionString, max);
const pool = new Pool({
//default must stay below the CloudFront edge timeout (30s) so the DB gives up first and the edge does not retry the same query
const statementTimeout = process.env.STATEMENT_TIMEOUT && parseInt(process.env.STATEMENT_TIMEOUT) || 25000;
log.info("pool settings:db:%s; pool size: %d; statement_timeout: %d", connectionString, max, statementTimeout);
const pool = new Pool({
connectionString,
max,
});

//kill any query that runs longer than statementTimeout on our side,
//instead of waiting for the database server's 600s limit
pool.on('connect', (client) => {
client.query(`SET statement_timeout = ${statementTimeout}`).catch(e => {

@dadiorchen dadiorchen Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we set timeout on the db side, then it could make the situation worse, because sql failed and no pg pool cache for this and next, we still try to request and failed again, it could make this failing infinitely

log.error("failed to set statement_timeout:", e);
});
});

//an idle client that errors (e.g. its connection is dropped) makes the pool
//emit 'error'; without a listener that event crashes the whole process
pool.on('error', (e) => {
log.error("idle db client error:", e);
});

const config = new Config(pool);


Expand Down Expand Up @@ -43,25 +59,27 @@ app.use('/viewer/images', express.static(images));


async function buildMapInstance(x, y, z, params){
const map = await new Promise(async (res, rej) => {
//do the async work OUTSIDE the Promise executor: a throw/rejection inside
//an async executor does not reject the outer promise, it leaves it pending
//forever and the http request hangs with no response
const bboxDb = mercator.xyz_to_envelope_db_buffer(//x, y, z, false);
parseInt(x),
parseInt(y),
parseInt(z),
false,
100,
);
const bounds = bboxDb.join(",");
log.debug("bounds:", bounds);
const xmlString = await config.getXMLString({
zoomLevel: z,
bounds,
...params,
});

const map = await new Promise((res, rej) => {
const mapInstance = new mapnik.Map(256, 256);
mapInstance.registerFonts(path.join(__dirname, '../test/data/map-a/'), {recurse:true});

const bboxDb = mercator.xyz_to_envelope_db_buffer(//x, y, z, false);
parseInt(x),
parseInt(y),
parseInt(z),
false,
100,
);
const bounds = bboxDb.join(",");
log.debug("bounds:", bounds);
const xmlString = await config.getXMLString({
zoomLevel: z,
bounds,
...params,
});

mapInstance.fromString(xmlString, {
strict: true,
base: __dirname,
Expand Down