Wt

Hello Michael,

from the perspective of the library a request coming from a user or a search engine is identical. This may lead to the search engine receiving a bootstrapped request, containing very basic HTML and then a subsequent JavaScript request that actually sets up the whole Wt framework on the client side, and renders its widgets (managed with the configuration option progressive-bootstrap.
Wt is also able to render all widgets and content on the server's side, and then send it to the client. This will be much easier to interpret for search engine indexers.

You can configure which user agents get this treatment, in wt_config.xml, with the <user-agents type="bot"> value.

A small example may look like this:

<server>
  <application-settings location="*">
    <user-agents type="bot">
      <user-agent>.*spider.*</user-agent>
      <user-agent>.*bot.*</user-agent>
      <user-agent>.*Bot.*</user-agent>
      <!-- Any agent you can find that you suspect is a bot -->
    </user-agents>
  </application-settings>
</server>

This will serve a plain HTML session to any user agent that contains spider, bot or Bot in its name. As indicated, these names ARE case sensitive.

Note that this is also good for performance, since the server doesn't set up a complete session for the new connection. This avoids some memory being allocated, and the server trying to match incoming connections to its relevant session. The indexer will not send back those cookies, so for each request a new session will be created. This means that if you are serving a lot of different "pages", this will lead to a lot of requests, and thus sessions being created, resulting in unnecessary overhead.

Marking them as bots avoids this, and will make your server happier.

I hope this helps!

Best regards,
Matthias

Thanks for your help Matthias.