Wt

This will serve as an overarching ticket, to which many subtickets related to handling bot requests will be attached.
The reason for this ticket is that lately there has been an uptick in bot agents scraping the web. Not only search engines now do so, but also AI companies looking for resources.

If these bot requests are not handled correctly this can have a big impact on the performance of the server.
In general Wt already handles bot requests such that, if the user agent can be matched against a list of bots, the session is automatically terminated. This ensures that bots do not take up valuable server resources (cpu/memory).

We notice lately though, that this is not enough. The public widget gallery for example, still has certain cases where much more resources are taken up than expected. This due to Wt's approach to sessions, and not being aggressive enough against bots.