WebArena Shopping evaluator issues

[imenelydiaker]

There is an issue with some WebArena shopping tasks:

• On task 275: it's a search task where the agent is asked to search for "xbox". So the reference URL is __SHOPPING__/catalogsearch/result/?q=xbox. The agent (GenericAgent) gets to that URL correctly but is rewarded 0.
• Same thing for task 274 and probably other tasks.

@xhluca do you have other task IDs with the same failure?

[xhluca]

those are the two i observed but i imagine it's worth looking at various cases with shoping

[ZLZLGe]

There is an issue with some WebArena shopping tasks:

• On task 275: it's a search task where the agent is asked to search for "xbox". So the reference URL is __SHOPPING__/catalogsearch/result/?q=xbox. The agent (GenericAgent) gets to that URL correctly but is rewarded 0.
• Same thing for task 274 and probably other tasks.

@xhluca do you have other task IDs with the same failure?

I also encountered this issue, and it needs to be fixed by modifying the environment variable.
Change
export WA_SHOPPING="$BASE_URL:8082/"
to
export WA_SHOPPING="$BASE_URL:8082"

In task 274, the failure occurred because if WA_SHOPPING is not modified, the reference URL will be set to
http://localhost:8082//catalogsearch/result/?q=usb+wifi,
while the page.url obtained from the website is
http://localhost:8082/catalogsearch/result/?q=usb+wifi.
As you can see, there is an extra “/”, so we need to update WA_SHOPPING to fix this mismatch.
@xhluca

[FazeelUsmani]

Hey, can I take this one?
Looked into it and I think ZLZLGe is right. In task.py the URL templating is just a str.replace on __SHOPPING__ etc., so if WA_SHOPPING ends with a / you get //catalogsearch/... and the evaluator's ref URL won't match page.url.
The easiest fix is probably stripping the trailing slash from WebArenaInstance so it doesn't depend on how the user sets the env var, plus fixing the example in the webarena README. Can also add a small test for the substitution.
Let me know if that sounds right or if you'd rather just patch the README.

[imenelydiaker]

@FazeelUsmani feel free to open a PR with a fix! Thank you!