Wt

Hi Nikita,

I am not sure how you plan to implement that, so I cannot really say if this is a good idea or not. But if you do not yet have good idea on how to implement it inside of a WApplication (in which you will not have access to the request) , I would recommend you to use Haproxy. You will likely have much more options than if you do it inside of Wt.

Hey Romain,

after some struggle, I think I've got something that looks more or less acceptable.

Indeed, initially I set it up with Haproxy, but wanted to explore what could be accomplished in Wt. I came up with 4 features, 3 of which do not require access to the underlying HTTP requests, and the last one seems to function with a few workarounds. I would really like this to work out, but still having doubts that I might have gotten something wrong which could cause trouble later.

1. Vulnerability scanners. As far as I understand, before creating a WApplication instance we can simply check what WEnvironment::internalPath returns, check if it begins with one of the common paths for other web software, like /phpmyadmin, and simply ban the IP address. It seems I still have to create a WApplication instance and call quit on it, because just returning an empty std::unique_ptr <WApplication> I get an error reported.

2. Legitimate bots that disobey robots.txt and rel="nofollow". Serve the robots.txt as WResource, create a WAnchor with some internal path, like /begone-you-foul-bot, then call Widget::hide to ensure that no human user can normally visit it. Then check WEnvironment::agentIsSpiderBot if the link was followed, the misbehaving bot will ignore the restrictions getting caught visiting the forbidden link.

3. More sophisticated bots pretending to be humans. With WApplication::internalPathChanged and WTimer it can be calculated how frequently the links are being followed, so I guess an overly zealous bot can be caught that way. The issue here is WTimer code apparently split between the server and the client part, so I am seeing new HTTP requests in log every time the timer fires. Which is completely redundant in this case of course, since there is no intention to update the page on the client's side. Gotten used to the convenience of timer API like this and QTimer, I am not quite sure what approach to take to ensure that the custom timer won't interfere with Wt's own event loop.

4. The automatic anti-bot challenge. When the one challenge you use on your own website held long enough it had a link to that project and I paid it a visit. Turns out it's just a small template HTML file coupled with a piece of Haproxy configuration. We can generate this HTML in Wt (I have stripped it down to bare minimum just to test the client request, without the challenge itself):

std::unique_ptr <WApplication> makeChallenge (const WEnvironment& environment)
{
	constexpr char challenge_form [] =
	R"S(
		<span>Do you happen to be one of those pesky bots? Let's find out, shall we?</span>
		<form method="get" action="{1}" name="challenge">
			<input type="hidden" name="challenge_result">
		</form>
	)S";

	constexpr char challenge_code [] =
	R"S(
		async function startChallenge (form)
		{
			await new Promise (r => setTimeout(r, 5000));
			form.challenge_result.value = "{2}";
			form.submit();
		}
		startChallenge (document.forms.challenge);
	)S";

	std::unique_ptr <WApplication> test = std::make_unique <WApplication> (environment);
	test->root()->addNew <WText> (WString (challenge_form).arg(environment.internalPath()), TextFormat::UnsafeXHTML);
	test->doJavaScript (WString (challenge_code).arg(12345).toUTF8());
	test->quit();
	return test;
}

WApplication::quit coupled with setting the HTML form's action to internal path where the client is supposed to go in the first place creates an illusion of redirection. Initially I wanted the WResource to generate this HTML page, which was supposed to go into WServer::addEntryPoint as EntryPointType::StaticResource, but the internal path could not be captured that way. And one somewhat unpleasant disadvantage here compared to Haproxy is that the GET request had to be used instead of the POST. Because with the POST in this case refreshing the page after the challenge completes always triggers that browser warning about some form inputs. So I always have to call this to clean up the URL from the no longer needed variables of the GET method later:

wapplication_after_challenge_completed->doJavaScript ("window.history.replaceState (null, '', window.location.pathname);");

Finally, I have tried not to post much code to not cause confusion, but here's the entry point function that I have written to put things into perspective (without other functions that I have written, but I think their names are more or less self-explanatory):

auto entry_point = [] (const WEnvironment& environment) -> std::unique_ptr <WApplication>
{
	static constexpr char bot_trap_path [] = "/begone-you-foul-bot";

	if (wpp::antibot::pathBeginsWith (environment.internalPath(), wpp::antibot::common_web_software_paths) or
	    wpp::antibot::pathBeginsWith (environment.internalPath(), bot_trap_path))
	{
		wpp::antibot::banUser (environment);
		return wpp::antibot::makeStub (environment);
	};
		
	if (wpp::antibot::challengeAlreadyPassed (environment))
		return makeWebsite (environment, bot_trap_path);

	switch (wpp::antibot::checkChallenge (environment))
	{
		case WPP::Antibot::ChallengeStatus::NeedsToPass:
			return wpp::antibot::makeChallenge (environment);
		case WPP::Antibot::ChallengeStatus::Passed:
			wpp::antibot::addUserToChallengePassedList (environment);
			return makeWebsite (environment, bot_trap_path);
		case WPP::Antibot::ChallengeStatus::Failed:
			wpp::antibot::banUser (environment);
			return wpp::antibot::makeStub (environment);
	}
};

What do you think, is this all worth giving a shot?