Wt Freeze on multiple concurrent Sessions (maybe DDoS)
Added by Christian Meyer over 6 years ago
Hi there
I have troubles with my Application after I hit about 1000 Sessions that are spawned roughly at the same time.
It seems like I am under attack, as the new Sessions are spawned in a few minutes interval, waiting a bit and then come another wave of new connections.
The real trouble is when I hit the mentioned 1000 Sessions (not always the same, found log with 1010 and 1011)
Then I get a error log saying:
[2018-Sep-09 18:16:51.135941] 1401 - [info] "Wt: session created (#sessions = 1010)"
[2018-Sep-09 18:16:51.136081] 1401 [/ hzmEcQ8vkGqtAHLW] [info] "WEnvironment: UserAgent: Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G950F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/7.4 Chrome/59.0.3071.125 Mobile Safari/537.36"
[2018-Sep-09 18:16:51.136388] 1401 [/ hzmEcQ8vkGqtAHLW] [info] "Wt-Version used: 3.3.10;"
[2018-Sep-09 18:16:51.136571] 1401 [/ hzmEcQ8vkGqtAHLW] [error] "WMessageResources: Could not read: /home/wt/live/Website/approot/templates/widget_templates.xml"
[2018-Sep-09 18:16:51.136589] 1401 [/ hzmEcQ8vkGqtAHLW] [error] "WMessageResources: Could not read: /home/wt/live/Website/approot/language_files/language.xml"
[2018-Sep-09 18:16:51.136603] 1401 [/ hzmEcQ8vkGqtAHLW] [error] "WMessageResources: Could not read: /home/wt/live/Website/approot/ModelCollection/ModelStrings.xml"
[2018-Sep-09 18:16:51.136615] 1401 [/ hzmEcQ8vkGqtAHLW] [error] "WMessageResources: Could not read: /home/wt/live/Website/approot/website_config.xml"
[2018-Sep-09 18:16:51.138113] 1401 [/ hzmEcQ8vkGqtAHLW] [error] "Wt: fatal error: unable to open database file"
After that, No new Sessions spawn, when the Timeout is hit (600s) the Sessions are being removed.
But I only get a response from the Application if I restart it:
[2018-Sep-10 10:27:48.584865] 1365 [/ hzmzfkA8SVE0tolV] [info] "Wt: session destroyed (#sessions = 1)"
[2018-Sep-10 10:27:53.585079] 1365 [/ hzmdz6HDFR9nXodL] [info] "WebController: timeout: expiring"
[2018-Sep-10 10:27:53.589500] 1365 [/ hzmdz6HDFR9nXodL] [info] "Wt: session destroyed (#sessions = 0)"
# killing pid, respawned from control-script
[2018-Sep-10 11:21:36.109590] 1365 - [info] "WebController: shutdown: stopping 0 sessions."
[2018-Sep-10 11:21:36.109647] 1365 - [info] "WServer/wthttp: Shutdown: stopping web server."
[2018-Sep-10 11:22:02.014419] 7810 - [info] "config: reading Wt config file: /home/wt/live/Website/approot/wt_config.xml (location = '/home/wt/live/Website/build/source/Website.live.wt')"
[2018-Sep-10 11:22:02.014850] 7810 - [info] "WServer/wthttp: initializing built-in wthttpd"
[2018-Sep-10 11:22:02.016177] 7810 - [info] "wthttp: started server: http://0.0.0.0:8090"
[2018-Sep-10 11:22:48.983237] 7810 - [info] "Wt: session created (#sessions = 1)"
I have found that there is a limit on how many files can be opened on Ubuntu and that was set to 1024.
So my guess is, that this Limitation threw Wt out of it's track record of being the best thing ever =)
If the App would have crashed, I would have been able to just restart the Application,
as there is a control-script in place that checks whether or not the App is running.
Is that intended behaviour? Or is that a bug?
What is the Wt-Way of handling a DDoS Attack?
Thank you for your input!
Christian
Replies (2)
RE: Wt Freeze on multiple concurrent Sessions (maybe DDoS) - Added by Roel Standaert over 6 years ago
The open files limit indeed defaults to 1024. You can increase that limit if necessary. That's not unusual for very actively used web applications.
However, the question remains whether you actually have that many real sessions. Usually, when there's something like a DDoS attack, many plain HTML sessions are being created instead of Ajax sessions. Also, sessions will always start out as plain and will then upgrade to Ajax, so if many sessions are created at the same time. You can use <plain-ajax-sessions-ratio-limit> to determine how many plain sessions you allow (which also controls how many sessions can spawn at the same time). The default is 1 (100), which means that plain sessions will only be rejected if they are more than 100 of all sessions (which of course will never happen). So you'll have to lower that number to some fraction. For example, if it is set to 0.5, that means that only 50% of all sessions are allowed to be plain HTML sessions.
It is peculiar that Wt doesn't really recover from that when the sessions expire, though.
RE: Wt Freeze on multiple concurrent Sessions (maybe DDoS) - Added by Christian Meyer over 6 years ago
Thank you for the ratio limit hint!
I set the limit to .3 and there's hardly the amount of sessions than before!
It usually sticks around 100.
Looking forward to getting my actual usercount up though. In this case it was good to run into file limits this early!