Bug #8610
closedWt 4.5.0: segfault in WebController::generateNewSessionId
100%
Description
Summary¶
We recently hit a segfault in WebController::generateNewSessionId().
The crash appears to have occured while a user was logging in.
- GCC 9.3.0 armv7l from Yocto Dunfell
- Wt 4.5.0 (Release build)
So far it's a one-off for us, we do not have a way to reproduce the crash.
Here's a partial stack trace of the crashing thread taken from the core dump:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __GI___libc_free (mem=0x1) at malloc.c:3102
3102 malloc.c: No such file or directory.
[Current thread is 1 (LWP 691)]
(gdb) bt
#0 __GI___libc_free (mem=0x1) at malloc.c:3102
#1 0x76009004 in operator delete (ptr=<optimized out>)
at ../../../../../../../../../work-shared/gcc-9.3.0-r0/gcc-9.3.0/libstdc++-v3/libsupc++/del_op.cc:49
#2 0x76dee10e in __gnu_cxx::new_allocator<char>::deallocate (this=0x1795344, __p=<optimized out>)
at /usr/include/c++/9.3.0/ext/new_allocator.h:119
#3 std::allocator_traits<std::allocator<char> >::deallocate (__a=..., __n=<optimized out>, __p=<optimized out>)
at /usr/include/c++/9.3.0/bits/alloc_traits.h:470
#4 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (__size=<optimized out>,
#5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x1795344)
at /usr/include/c++/9.3.0/bits/basic_string.h:232
#6 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x1795344,
__in_chrg=<optimized out>) at /usr/include/c++/9.3.0/bits/basic_string.h:658
#7 std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >::~pair (this=0x1795344, __in_chrg=<optimized out>) at /usr/include/c++/9.3.0/bits/stl_pair.h:208
#8 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > (this=0x1795330, __p=0x1795344)
at /usr/include/c++/9.3.0/ext/new_allocator.h:153
#9 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > (__a=..., __p=0x1795344)
at /usr/include/c++/9.3.0/bits/alloc_traits.h:497
#10 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_destroy_node (this=0x1795330, __p=0x1795334) at /usr/include/c++/9.3.0/bits/stl_tree.h:642
#11 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_drop_node (this=0x1795330, __p=0x1795334) at /usr/include/c++/9.3.0/bits/stl_tree.h:650
#12 std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::_M_erase_aux (this=this@entry=0x1795330, __position=..., Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.:
__position@entry=...) at /usr/include/c++/9.3.0/bits/stl_tree.h:2511
#13 0x76ded996 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >) (Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.:
__position=..., this=0x1795330)
at /usr/include/c++/9.3.0/bits/stl_tree.h:1215
#14 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<Wt::WebSession>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> > >) (Python Exception <class 'gdb.error'> No type named std::_Rb_tree_node<struct std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<Wt::WebSession> >>.:
__position=..., this=0x1795330) at /usr/include/c++/9.3.0/bits/stl_map.h:1037
#15 Wt::WebController::generateNewSessionId[abi:cxx11](std::shared_ptr<Wt::WebSession> const&) (this=0x17952b8,
session=std::shared_ptr<class Wt::WebSession> (use count 7, weak count 4) = {...})
at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebController.C:833
#16 0x76df588e in Wt::WebSession::generateNewSessionId (this=0x724ebf88) at /usr/include/c++/9.3.0/bits/shared_ptr_base.h:251
#17 0x76b7d892 in Wt::WApplication::changeSessionId (this=<optimized out>)
at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WApplication.C:596
#18 0x76cede3e in Wt::Auth::AuthWidget::onLoginChange (this=0x18bbd68)
at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/Auth/AuthWidget.C:291
#19 0x76b64548 in std::function<void ()>::operator()() const (this=0x18a6864)
at /usr/include/c++/9.3.0/bits/std_function.h:683
Frame 15 is the interesting part WebController.C:833
:
816 std::string
817 WebController::generateNewSessionId(const std::shared_ptr<WebSession>& session)
818 {
819 #ifdef WT_THREADED
820 std::unique_lock<std::recursive_mutex> lock(mutex_);
821 #endif // WT_THREADEDâ‹„â‹„
822
823 std::string newSessionId;
824 do {
825 newSessionId = conf_.generateSessionId();
826 if (!conf_.registerSessionId(session->sessionId(), newSessionId))
827 newSessionId.clear();
828 } while (newSessionId.empty());
829
830 sessions_[newSessionId] = session;
831
832 SessionMap::iterator i = sessions_.find(session->sessionId());
* 833 sessions_.erase(i);
834
835 if (!singleSessionId_.empty())
836 singleSessionId_ = newSessionId;
837
838 return newSessionId;
839 }
(gdb) p newSessionId
$1 = "bMETm7O6IUX392TY"
(gdb) p i
$2 =
{first = <error: Cannot access memory at address 0x1>, second = std::shared_ptr<Wt::WebSession> (use count -1, weak count -1) = {get() = 0x0}}
(gdb) p sessions_
$3 = std::map with 1 element = {["bMETm7O6IUX392TY"] = std::shared_ptr<Wt::WebSession> (use count 7, weak count 4) = {
get() = 0x724ebf88}}
I notice there is no check for i == sessions_.end()
.
Perhaps this is intentional, and sessions_.find
is never expected to fail?
If failure is (sometimes) expected, suggest making the call to erase
conditional on i != sessions_.end()
, or, as i
is otherwise unused, use the size_type std::map::erase( const key_type& key )
overload which handles this for you.
Happy to provide more information if needed.
Updated by Korneel Dumon over 3 years ago
Hi,
this is pretty surprising, but you make a strong case ... I wouldn't expect the call to sessions_.find
to fail.
The only thing I can imagine is some bad request arrived in the middle of event handling, causing the session to be killed. Could you check WebSession::state()
?
Updated by Chris Sykes over 3 years ago
I hope this is what you were asking for:
(gdb) p session->state_
$2 = Wt::WebSession::State::Loaded
Most other threads in the app/pool are not up to much:
(gdb) thread 2
[Switching to thread 2 (LWP 478)]
#0 __libc_do_syscall () at libc-do-syscall.S:49
49 libc-do-syscall.S: No such file or directory.
(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:49
#1 0x4a1bae20 in __GI___sigtimedwait (set=set@entry=0x7e8fe198, info=info@entry=0x7e8fe0c8, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#2 0x760dd8c4 in __sigwait (set=set@entry=0x7e8fe198, sig=sig@entry=0x7e8fe164) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#3 0x76c62ac0 in Wt::WServer::waitForShutdown () at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WServer.C:380
#4 0x0004a8aa in boost::wrapexcept<boost::system::system_error>::~wrapexcept (this=0x7e8fe2bc, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at /usr/include/boost/exception/exception.hpp:402
#5 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 3
[Switching to thread 3 (LWP 546)]
#0 __libc_do_syscall () at libc-do-syscall.S:48
48 in libc-do-syscall.S
(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:48
#1 0x4a224d9a in __GI___ppoll64 (fds=0x74928ea8, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:47
#2 0x4a224e78 in __ppoll (fds=fds@entry=0x74928ea8, nfds=nfds@entry=1, timeout=timeout@entry=0x752c2220, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:78
#3 0x76303fea in qt_ppoll (timeout_ts=0x752c2220, nfds=1, fds=0x74928ea8) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/kernel/qcore_unix.cpp:164
#4 qt_safe_poll (fds=0x74928ea8, nfds=1, timeout_ts=timeout_ts@entry=0x752c2260) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/kernel/qcore_unix.cpp:164
#5 0x763051a4 in QEventDispatcherUNIX::processEvents (this=<optimized out>, flags=...) at ../../include/QtCore/../../../git/src/corelib/tools/qvector.h:88
#6 0x762c6b6c in QEventLoop::exec (this=this@entry=0x752c2300, flags=..., flags@entry=...) at ../../include/QtCore/../../../git/src/corelib/global/qflags.h:136
#7 0x76184ab6 in QThread::exec (this=<optimized out>) at ../../include/QtCore/../../../git/src/corelib/global/qflags.h:118
#8 0x76185772 in QThreadPrivate::start (arg=0x7e8fe2dc) at /usr/src/debug/qtbase/5.14.2+gitAUTOINC+3a6d8df521-r0/git/src/corelib/thread/qthread_unix.cpp:342
#9 0x760d4978 in start_thread (arg=0xe16c3d9e) at pthread_create.c:477
#10 0x4a22b1fc in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /home/chris.sykes/tmp/debug/210519/sysroot/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) thread 4
[Switching to thread 4 (LWP 549)]
#0 __libc_do_syscall () at libc-do-syscall.S:48
48 libc-do-syscall.S: No such file or directory.
(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:48
#1 0x760d9f98 in futex_wait_cancelable (private=0, expected=0, futex_word=0x725396ec) at ../sysdeps/nptl/futex-internal.h:183
#2 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=<optimized out>, cond=0x725396c0) at pthread_cond_wait.c:508
#3 __pthread_cond_wait (cond=0x725396c0, mutex=0x725396a4) at pthread_cond_wait.c:638
#4 0x760231d0 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /usr/src/debug/gcc-runtime/9.3.0-r0/arm-fluidity-linux-gnueabi/libstdc++-v3/include/arm-fluidity-linux-gnueabi/bits/gthr-default.h:865
#5 std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../../../../../../work-shared/gcc-9.3.0-r0/gcc-9.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53
...
But the stack trace from thread 5 might be of interest:
(gdb) thread 5
[Switching to thread 5 (LWP 550)]
#0 __libc_do_syscall () at libc-do-syscall.S:48
48 in libc-do-syscall.S
(gdb) bt
#0 __libc_do_syscall () at libc-do-syscall.S:48
#1 0x760dccaa in __lll_lock_wait (futex=futex@entry=0x724ebf90, private=0) at lowlevellock.c:52
#2 0x760d680c in __GI___pthread_mutex_lock (mutex=0x724ebf90) at pthread_mutex_lock.c:80
#3 0x76df4130 in __gthread_mutex_lock (__mutex=0x724ebf90) at /usr/include/c++/9.3.0/arm-fluidity-linux-gnueabi/bits/gthr-default.h:749
#4 std::mutex::lock (this=0x724ebf90) at /usr/include/c++/9.3.0/bits/std_mutex.h:100
#5 std::unique_lock<std::mutex>::lock (this=0x740fd150, this=0x740fd150) at /usr/include/c++/9.3.0/bits/unique_lock.h:141
#6 Wt::WebSession::Handler::Handler (this=0x740fd138, session=..., lockOption=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebSession.C:789
#7 0x76dea3e4 in Wt::WebController::expireSessions (this=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/web/WebController.C:220
#8 0x76c62bd6 in Wt::WServer::expireSessions (this=<optimized out>) at /usr/src/debug/wt/4.5.0-r1/git/src/Wt/WServer.C:428
#9 0x769c1676 in http::server::Server::expireSessions (this=0x180b718, ec=...) at /usr/src/debug/wt/4.5.0-r1/git/src/http/Server.C:677
#10 0x769c6c3c in boost::asio::detail::executor_function_base::complete (this=<optimized out>) at /usr/include/boost/asio/detail/executor_function.hpp:32
#11 boost::asio::executor::function::operator() (this=<synthetic pointer>) at /usr/include/boost/asio/impl/executor.hpp:69
#12 boost::asio::asio_handler_invoke<boost::asio::executor::function> (function=<synthetic pointer>...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69
Looks like it was expiring old sessions when the crash occurred?
Updated by Korneel Dumon over 3 years ago
- Status changed from New to InProgress
- Assignee set to Korneel Dumon
- Target version set to 4.6.0
Updated by Korneel Dumon over 3 years ago
- Status changed from InProgress to Review
Updated by Roel Standaert about 3 years ago
- Status changed from Review to Implemented @Emweb
Updated by Roel Standaert about 3 years ago
- Status changed from Implemented @Emweb to Resolved
Updated by Roel Standaert almost 3 years ago
- Status changed from Resolved to Closed