Project

General

Profile

Actions

Bug #13568

open

Bot agents can recursively call main page

Added by Matthias Van Ceulebroeck about 1 month ago. Updated 5 days ago.

Status:
Implemented @Emweb
Priority:
Normal
Target version:
Start date:
02/24/2025
Due date:
% Done:

100%

Estimated time:

Description

In case of dynamic resources being present on the page, this is requested from the application using the application's entrypoint, with request=resource&resource={id}.
Of course, by that point the application has already been shut down for bots. This will cause a new WApplication being spun up, to serve the non-existing resource.
In some cases, this will result in a 404, but if the entrypoint would match an WApplication path, that can cause a whole page being served.

Many bots will then iteratively go over the response again, and again find those resources, leading to them spamming a server with requests.
It has been decided that bots will get 404 responses in case of dynamic resources. They are still able to request static resources.

Actions #1

Updated by Matthias Van Ceulebroeck about 1 month ago

  • Status changed from InProgress to Review
  • Assignee deleted (Matthias Van Ceulebroeck)

This implementation was not changed, rather for the widget gallery specifically a robots.txt file was added.

Actions #2

Updated by Matthias Van Ceulebroeck 5 days ago

  • Status changed from Review to Implemented @Emweb
  • Assignee set to Matthias Van Ceulebroeck
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF