Should we add better tests to oflm test so that it actually catches things? The ones inherited from flm never catch anything.
A few from the last two weeks:
Asking for a model that doesn't exist got you a different model's answer, with the name you asked for echoed back (#52)
Every error response came back as HTTP 200 with a JSON parser message as the body, so the real error text never reached a client (#52)
Every document sent to /v1/embeddings was embedded as a query. Right shape, right length, deterministic, wrong (#52)
An answer cut off at max_tokens said it had finished normally (#52)
Every tool result arrived four tokens short, so a ticket code came back missing its last character and 18 degrees was reported as 15 (#59)
One client asking for low reasoning effort turned thinking on for everybody else (#59)
Should we add better tests to oflm test so that it actually catches things? The ones inherited from flm never catch anything.
A few from the last two weeks:
Asking for a model that doesn't exist got you a different model's answer, with the name you asked for echoed back (#52)
Every error response came back as HTTP 200 with a JSON parser message as the body, so the real error text never reached a client (#52)
Every document sent to /v1/embeddings was embedded as a query. Right shape, right length, deterministic, wrong (#52)
An answer cut off at max_tokens said it had finished normally (#52)
Every tool result arrived four tokens short, so a ticket code came back missing its last character and 18 degrees was reported as 15 (#59)
One client asking for low reasoning effort turned thinking on for everybody else (#59)