Project

General

Profile

Actions

Bug #8610

closed

Wt 4.5.0: segfault in WebController::generateNewSessionId

Added by Chris Sykes over 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
06/01/2021
Due date:
% Done:

100%

Estimated time:

Description

Summary

We recently hit a segfault in WebController::generateNewSessionId().

The crash appears to have occured while a user was logging in.

  • GCC 9.3.0 armv7l from Yocto Dunfell
  • Wt 4.5.0 (Release build)

So far it's a one-off for us, we do not have a way to reproduce the crash.

Here's a partial stack trace of the crashing thread taken from the core dump:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI___libc_free (mem=0x1) at malloc.c:3102
3102    malloc.c: No such file or directory.
[Current thread is 1 (LWP 691)]


(gdb) bt
#0  __GI___libc_free (mem=0x1) at malloc.c:3102
#1  0x76009004 in operator delete (ptr=<optimized out>)
    at ../../../../../../../../../work-shared/gcc-9.3.0-r0/gcc-9.3.0/libstdc++-v3/libsupc++/del_op.cc:49
#2  0x76dee10e in __gnu_cxx::new_allocator<char>::deallocate (this=0x1795344, __p=<optimized out>)
    at /usr/include/c++/9.3.0/ext/new_allocator.h:119
#3  std::allocator_traits<std::allocator<char> >::deallocate (__a=..., __n=<optimized out>, __p=<optimized out>)
    at /usr/include/c++/9.3.0/bits/alloc_traits.h:470
#4  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (__size=<optimized out>, 
#5  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x1795344)
    at /usr/include/c++/9.3.0/bits/basic_string.h:232
#6  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x1795344, 
    __in_chrg=<optimized out>) at /usr/include/c++/9.3.0/bits/basic_string.h:658
#7  std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >::~pair (this=0x1795344, __in_chrg=<optimized out>) at /usr/include/c++/9.3.0/bits/stl_pair.h:208
#8  __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > (this=0x1795330, __p=0x1795344)
    at /usr/include/c++/9.3.0/ext/new_allocator.h:153
#9  std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > (__a=..., __p=0x1795344)
    at /usr/include/c++/9.3.0/bits/alloc_traits.h:497
#10 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_destroy_node (this=0x1795330, __p=0x1795334) at /usr/include/c++/9.3.0/bits/stl_tree.h:642
#11 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_drop_node (this=0x1795330, __p=0x1795334) at /usr/include/c++/9.3.0/bits/stl_tree.h:650
#12 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_erase_aux (this=this@entry=0x1795330, __position=..., Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.: 
__position@entry=...) at /usr/include/c++/9.3.0/bits/stl_tree.h:2511
#13 0x76ded996 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >) (Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.: 
__position=..., this=0x1795330)
    at /usr/include/c++/9.3.0/bits/stl_tree.h:1215
#14 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Wt::WebSession>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >) (Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.: 
__position=..., this=0x1795330) at /usr/include/c++/9.3.0/bits/stl_map.h:1037
#15 Wt::WebController::generateNewSessionId[abi:cxx11](std::shared_ptr<Wt::WebSession> const&) (this=0x17952b8, 
    session=std::shared_ptr<class Wt::WebSession> (use count 7, weak count 4) = {...})
    at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebController.C:833
#16 0x76df588e in Wt::WebSession::generateNewSessionId (this=0x724ebf88) at /usr/include/c++/9.3.0/bits/shared_ptr_base.h:251
#17 0x76b7d892 in Wt::WApplication::changeSessionId (this=<optimized out>)
    at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WApplication.C:596
#18 0x76cede3e in Wt::Auth::AuthWidget::onLoginChange (this=0x18bbd68)
    at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/Auth/AuthWidget.C:291
#19 0x76b64548 in std::function<void ()>::operator()() const (this=0x18a6864)
    at /usr/include/c++/9.3.0/bits/std_function.h:683

Frame 15 is the interesting part WebController.C:833:

  816 std::string
  817 WebController::generateNewSessionId(const std::shared_ptr<WebSession>& session)
  818 {
  819 #ifdef WT_THREADED
  820   std::unique_lock<std::recursive_mutex> lock(mutex_);
  821 #endif // WT_THREADEDâ‹„â‹„
  822 
  823   std::string newSessionId;
  824   do {
  825     newSessionId = conf_.generateSessionId();
  826     if (!conf_.registerSessionId(session->sessionId(), newSessionId))
  827       newSessionId.clear();
  828   } while (newSessionId.empty());
  829 
  830   sessions_[newSessionId] = session;
  831 
  832   SessionMap::iterator i = sessions_.find(session->sessionId());
* 833   sessions_.erase(i);
  834 
  835   if (!singleSessionId_.empty())
  836     singleSessionId_ = newSessionId;
  837 
  838   return newSessionId;
  839 }
(gdb) p newSessionId
$1 = "bMETm7O6IUX392TY"

(gdb) p i
$2 = 
    {first = <error: Cannot access memory at address 0x1>, second = std::shared_ptr<Wt::WebSession> (use count -1, weak count -1) = {get() = 0x0}}

(gdb) p sessions_
$3 = std::map with 1 element = {["bMETm7O6IUX392TY"] = std::shared_ptr<Wt::WebSession> (use count 7, weak count 4) = {
    get() = 0x724ebf88}}

I notice there is no check for i == sessions_.end().

Perhaps this is intentional, and sessions_.find is never expected to fail?

If failure is (sometimes) expected, suggest making the call to erase conditional on i != sessions_.end(), or, as i is otherwise unused, use the size_type std::map::erase( const key_type& key ) overload which handles this for you.

Happy to provide more information if needed.

Actions #1

Updated by Korneel Dumon over 3 years ago

Hi,

this is pretty surprising, but you make a strong case ... I wouldn't expect the call to sessions_.find to fail.
The only thing I can imagine is some bad request arrived in the middle of event handling, causing the session to be killed. Could you check WebSession::state()?

Actions #2

Updated by Chris Sykes over 3 years ago

I hope this is what you were asking for:

(gdb) p session->state_
$2 = Wt::WebSession::State::Loaded

Most other threads in the app/pool are not up to much:

(gdb) thread 2
[Switching to thread 2 (LWP 478)]
#0  __libc_do_syscall () at libc-do-syscall.S:49
49      libc-do-syscall.S: No such file or directory.
(gdb) bt
#0  __libc_do_syscall () at libc-do-syscall.S:49
#1  0x4a1bae20 in __GI___sigtimedwait (set=set@entry=0x7e8fe198, info=info@entry=0x7e8fe0c8, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#2  0x760dd8c4 in __sigwait (set=set@entry=0x7e8fe198, sig=sig@entry=0x7e8fe164) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#3  0x76c62ac0 in Wt::WServer::waitForShutdown () at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WServer.C:380
#4  0x0004a8aa in boost::wrapexcept<boost::system::system_error>::~wrapexcept (this=0x7e8fe2bc, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at /usr/include/boost/exception/exception.hpp:402
#5  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) thread 3
[Switching to thread 3 (LWP 546)]
#0  __libc_do_syscall () at libc-do-syscall.S:48
48      in libc-do-syscall.S
(gdb) bt
#0  __libc_do_syscall () at libc-do-syscall.S:48
#1  0x4a224d9a in __GI___ppoll64 (fds=0x74928ea8, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:47
#2  0x4a224e78 in __ppoll (fds=fds@entry=0x74928ea8, nfds=nfds@entry=1, timeout=timeout@entry=0x752c2220, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:78
#3  0x76303fea in qt_ppoll (timeout_ts=0x752c2220, nfds=1, fds=0x74928ea8) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/kernel/qcore_unix.cpp:164
#4  qt_safe_poll (fds=0x74928ea8, nfds=1, timeout_ts=timeout_ts@entry=0x752c2260) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/kernel/qcore_unix.cpp:164
#5  0x763051a4 in QEventDispatcherUNIX::processEvents (this=<optimized out>, flags=...) at ../../include/QtCore/../../../git/src/corelib/tools/qvector.h:88
#6  0x762c6b6c in QEventLoop::exec (this=this@entry=0x752c2300, flags=..., flags@entry=...) at ../../include/QtCore/../../../git/src/corelib/global/qflags.h:136
#7  0x76184ab6 in QThread::exec (this=<optimized out>) at ../../include/QtCore/../../../git/src/corelib/global/qflags.h:118
#8  0x76185772 in QThreadPrivate::start (arg=0x7e8fe2dc) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/thread/qthread_unix.cpp:342
#9  0x760d4978 in start_thread (arg=0xe16c3d9e) at pthread_create.c:477
#10 0x4a22b1fc in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /home/chris.sykes/tmp/debug/210519/sysroot/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) thread 4
[Switching to thread 4 (LWP 549)]
#0  __libc_do_syscall () at libc-do-syscall.S:48
48      libc-do-syscall.S: No such file or directory.
(gdb) bt
#0  __libc_do_syscall () at libc-do-syscall.S:48
#1  0x760d9f98 in futex_wait_cancelable (private=0, expected=0, futex_word=0x725396ec) at ../sysdeps/nptl/futex-internal.h:183
#2  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=<optimized out>, cond=0x725396c0) at pthread_cond_wait.c:508
#3  __pthread_cond_wait (cond=0x725396c0, mutex=0x725396a4) at pthread_cond_wait.c:638
#4  0x760231d0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /usr/src/debug/gcc-runtime/9.3.0-r0/arm-fluidity-linux-gnueabi/libstdc++-v3/include/arm-fluidity-linux-gnueabi/bits/gthr-default.h:865
#5  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../../../../../../work-shared/gcc-9.3.0-r0/gcc-9.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53
 ...

But the stack trace from thread 5 might be of interest:

(gdb) thread 5
[Switching to thread 5 (LWP 550)]
#0  __libc_do_syscall () at libc-do-syscall.S:48
48      in libc-do-syscall.S
(gdb) bt
#0  __libc_do_syscall () at libc-do-syscall.S:48
#1  0x760dccaa in __lll_lock_wait (futex=futex@entry=0x724ebf90, private=0) at lowlevellock.c:52
#2  0x760d680c in __GI___pthread_mutex_lock (mutex=0x724ebf90) at pthread_mutex_lock.c:80
#3  0x76df4130 in __gthread_mutex_lock (__mutex=0x724ebf90) at /usr/include/c++/9.3.0/arm-fluidity-linux-gnueabi/bits/gthr-default.h:749
#4  std::mutex::lock (this=0x724ebf90) at /usr/include/c++/9.3.0/bits/std_mutex.h:100
#5  std::unique_lock<std::mutex>::lock (this=0x740fd150, this=0x740fd150) at /usr/include/c++/9.3.0/bits/unique_lock.h:141
#6  Wt::WebSession::Handler::Handler (this=0x740fd138, session=..., lockOption=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebSession.C:789
#7  0x76dea3e4 in Wt::WebController::expireSessions (this=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebController.C:220
#8  0x76c62bd6 in Wt::WServer::expireSessions (this=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WServer.C:428
#9  0x769c1676 in http::server::Server::expireSessions (this=0x180b718, ec=...) at /usr/src/debug/wt/4.5.0-r1/git/src/http/Server.C:677
#10 0x769c6c3c in boost::asio::detail::executor_function_base::complete (this=<optimized out>) at /usr/include/boost/asio/detail/executor_function.hpp:32
#11 boost::asio::executor::function::operator() (this=<synthetic pointer>) at /usr/include/boost/asio/impl/executor.hpp:69
#12 boost::asio::asio_handler_invoke<boost::asio::executor::function> (function=<synthetic pointer>...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69

Looks like it was expiring old sessions when the crash occurred?

Actions #3

Updated by Korneel Dumon over 3 years ago

  • Status changed from New to InProgress
  • Assignee set to Korneel Dumon
  • Target version set to 4.6.0
Actions #4

Updated by Korneel Dumon over 3 years ago

  • Status changed from InProgress to Review
Actions #5

Updated by Roel Standaert about 3 years ago

  • Status changed from Review to Implemented @Emweb
Actions #6

Updated by Roel Standaert about 3 years ago

  • % Done changed from 0 to 100
Actions #7

Updated by Roel Standaert about 3 years ago

  • Status changed from Implemented @Emweb to Resolved
Actions #8

Updated by Roel Standaert almost 3 years ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF