1347928152 Q * nlm Ping timeout: 480 seconds 1347933090 J * fisted_ ~fisted@xdsl-87-78-190-2.netcologne.de 1347933313 Q * fisted Read error: Operation timed out 1347935475 Q * clopez Quit: Leaving 1347935568 J * clopez ~clopez@17.28.165.83.dynamic.mundo-r.com 1347936060 Q * clopez Ping timeout: 480 seconds 1347936433 N * Bertl_zZ Bertl 1347936459 M * Bertl awake again ... 1347937198 M * Jb_boin :) 1347946151 J * ghislain ~AQUEOS@adsl2.aqueos.com 1347946633 M * Bertl off to bed ... 1347946637 N * Bertl Bertl_zZ 1347948090 Q * ensc|w Remote host closed the connection 1347948099 J * ensc|w ~ensc@www.sigma-chemnitz.de 1347948134 Q * nakacya Remote host closed the connection 1347948157 J * nakacya ~nakacya@KD118152083243.ppp-bb.dion.ne.jp 1347948700 Q * nakacya Ping timeout: 480 seconds 1347950741 Q * jeroen__ Quit: Ex-Chat 1347953474 J * vspas ~vspas@87.213.36.165 1347954450 J * BenG ~bengreen@cpc10-aztw24-2-0-cust114.aztw.cable.virginmedia.com 1347956148 J * kir ~kir@swsoft-msk-nat.sw.ru 1347957196 J * jeroen__ ~jeroen@mail.nl02.searchtrends.eu 1347958313 Q * ncopa Quit: Leaving 1347959188 Q * BenG Quit: I Leave 1347960062 Q * WMP Quit: ZNC - http://znc.in 1347960071 J * oftc ~oftc@auburn.sored.pl 1347960071 N * oftc Guest7426 1347960071 N * Guest7426 WMP 1347960393 J * nakacya ~nakacya@KD118152083243.ppp-bb.dion.ne.jp 1347961797 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:a163:6ddb:f7c2:7f0e 1347961803 Q * thierryp Remote host closed the connection 1347961910 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:a163:6ddb:f7c2:7f0e 1347962293 J * clopez ~clopez@fanzine.igalia.com 1347963259 P * kir PING 1347963259 1347964314 J * nlm ~nlm@host150.190-137-109.telecom.net.ar 1347965668 Q * thierryp Remote host closed the connection 1347966275 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:e048:ee1b:927c:7502 1347966671 J * ncopa ~ncopa@3.203.202.84.customer.cdi.no 1347967326 Q * Romster Quit: Geeks shall inherit properties and methods of object earth. 1347967563 Q * thierryp Remote host closed the connection 1347967593 Q * nlm Remote host closed the connection 1347967622 J * Romster ~romster@202.168.100.149.dynamic.rev.eftel.com 1347967761 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:6deb:fea4:f6ac:2033 1347970080 J * ard ~ard@shell2.kwaak.net 1347970957 J * kir ~kir@swsoft-msk-nat.sw.ru 1347971047 P * kir 1347972979 Q * thierryp Remote host closed the connection 1347973063 N * Bertl_zZ Bertl 1347973066 M * Bertl morning folks! 1347973280 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:281d:6465:4b1f:2709 1347973315 Q * thierryp Remote host closed the connection 1347974207 J * thierryp ~thierry@home.parmentelat.net 1347974763 Q * ghislain Read error: Connection reset by peer 1347974822 J * ghislain ~AQUEOS@adsl2.aqueos.com 1347974918 Q * thierryp Remote host closed the connection 1347975640 J * ViruSzZ ~root@stavrovski.net 1347975828 M * ViruSzZ hey there... this has been happening for a while on and off and I really do not know when and why is this happening. so, I tried to start a vserver and got this stuck for more than 10min. is this normal, should I wait or the process simply hung? 1347975832 M * ViruSzZ root 24509 0.0 0.0 66196 1768 pts/3 Ss 08:26 0:00 | \_ -bash 1347975834 M * ViruSzZ root 28579 0.0 0.0 164 32 pts/3 S+ 08:30 0:00 | \_ /usr/sbin/vspace --mount --fs --new -- /usr/sbin/vserver ----nonamespace _GUEST_NAME_ start 1347975837 M * ViruSzZ root 28586 0.0 0.0 64748 2244 pts/3 S+ 08:30 0:00 | \_ /bin/bash /usr/sbin/vserver ----nonamespace handsont start 1347975840 M * ViruSzZ root 28598 0.0 0.0 152 16 pts/3 S+ 08:30 0:00 | \_ /usr/lib64/util-vserver/lockfile /var/lock/vserver.etcvservers_GUEST_NAME_.startup /tmp/vserver-lock.n28595 1347975844 M * ViruSzZ root 28666 0.0 0.0 172 44 pts/3 D+ 08:30 0:00 | \_ /usr/lib64/util-vserver/secure-mount -a --chroot --fstab /etc/vservers/_GUEST_NAME_/fstab --rootfs no 1347975909 J * thierryp ~thierry@82.226.190.44 1347975909 M * ViruSzZ before I start the vserver I did chxid it using chxid -c CID -Rx /vservers/_vs_dir_/ 1347975972 M * daniel_hozac what mounts do you have in your fstab? 1347976024 M * ViruSzZ this is in the guest's fstab 1347976024 M * ViruSzZ none /proc proc defaults 0 0 1347976025 M * ViruSzZ none /dev/pts devpts gid=5,mode=620 0 0 1347976035 M * ViruSzZ and here's the hosts's one: 1347976047 M * ViruSzZ /dev/sda3 / ext3 defaults,data=journal 1 1 1347976050 M * ViruSzZ /dev/sda5 /vservers ext3 defaults,tag,noatime,data=journal 1 2 1347976053 M * ViruSzZ /dev/sda1 /boot ext3 defaults,data=journal 1 2 1347976056 M * ViruSzZ tmpfs /dev/shm tmpfs defaults 0 0 1347976058 M * ViruSzZ devpts /dev/pts devpts gid=5,mode=620 0 0 1347976061 M * ViruSzZ sysfs /sys sysfs defaults 0 0 1347976062 M * daniel_hozac what does strace -p 28666 show? 1347976063 M * ViruSzZ proc /proc proc defaults 0 0 1347976066 M * ViruSzZ /dev/sda2 swap swap defaults 0 0 1347976068 M * ViruSzZ cpuset /dev/cpuset cpuset defaults 0 0 1347976071 M * ViruSzZ vserver /dev/cgroup cgroup cpu,cpuset,memory 0 0 1347976096 M * ViruSzZ let me check that 1347976111 M * ViruSzZ # strace -p 28666 1347976111 M * ViruSzZ Process 28666 attached - interrupt to quit 1347976119 M * ViruSzZ that's it 1347976135 M * daniel_hozac how about cat /proc/28666/wchan? 1347976175 M * ViruSzZ # cat /proc/28666/wchan 1347976175 M * ViruSzZ call_rwsem_down_write_failed 1347976224 M * ViruSzZ and if I try to terminate the processes the one on D+ will stay there and will most likely increase hosts loadavg and the only solution will be to reboot the host machine. 1347976242 M * daniel_hozac do you have /proc/sysrq-trigger? 1347976255 M * ViruSzZ yup 1347976320 M * daniel_hozac try echo w > /proc/sysrq-trigger and check dmesg 1347976405 M * ViruSzZ this was logged only => http://pastebin.com/k0eb5ppn 1347976440 M * ViruSzZ nothing changed.... 1347976546 M * daniel_hozac okay, try echo t > /proc/sysrq-trigger then 1347976750 M * ViruSzZ tried that.. the situation have not changed and I think the same result is in dmesg. anyhow, do I need to provide the output of dmesg? 1347976825 M * daniel_hozac that is the idea. 1347976832 M * ViruSzZ ok, please wait. 1347976842 M * daniel_hozac you only really need the data from that process 1347976858 M * ViruSzZ ok, let me do that... 1347976938 M * ViruSzZ ok I've executed the latest echo t again and redirected whole dmesg to a file using # dmesg > /tmp/dmesg.out 1347976948 M * ViruSzZ then tried to grep the pid of the process and returned null 1347977026 M * daniel_hozac dmesg usually doesn't have a buffer big enough for it. 1347977034 M * daniel_hozac check /var/log/messages or similar on your system 1347977057 M * ViruSzZ didn't know that.... thanks 1347977105 M * ViruSzZ now here's something maybe usefull to some who knows C and I know that's not me yet :)_ 1347977108 M * ViruSzZ # grep -i 28666 /var/log/messages 1347977110 M * ViruSzZ Sep 18 08:52:08 vanadium kernel: [2515217.454751] secure-mount D ffff88063fd90900 0 28666 28586 0x00000004 1347977113 M * ViruSzZ Sep 18 08:58:17 vanadium kernel: [2515528.646196] .se->sum_exec_runtime : 1728666.904379 1347977143 M * daniel_hozac you'll want the lines around the first one. 1347977231 M * ViruSzZ i think i do not understand you what exactly you were saying. anyhow i retried: 1347977231 M * ViruSzZ # echo w > /proc/sysrq-trigger 1347977232 M * ViruSzZ # grep -i 28666 /var/log/messages 1347977232 M * ViruSzZ Sep 18 08:52:08 vanadium kernel: [2515217.454751] secure-mount D ffff88063fd90900 0 28666 28586 0x00000004 1347977236 M * ViruSzZ Sep 18 08:58:17 vanadium kernel: [2515528.646196] .se->sum_exec_runtime : 1728666.904379 1347977238 M * ViruSzZ Sep 18 09:06:09 vanadium kernel: [2516057.580614] secure-mount D ffff88063fd90900 0 28666 28586 0x00000004 1347977241 M * ViruSzZ # echo t > /proc/sysrq-trigger 1347977244 M * ViruSzZ # grep -i 28666 /var/log/messages 1347977246 M * ViruSzZ Sep 18 08:52:08 vanadium kernel: [2515217.454751] secure-mount D ffff88063fd90900 0 28666 28586 0x00000004 1347977249 M * ViruSzZ Sep 18 08:58:17 vanadium kernel: [2515528.646196] .se->sum_exec_runtime : 1728666.904379 1347977252 M * ViruSzZ Sep 18 09:06:09 vanadium kernel: [2516057.580614] secure-mount D ffff88063fd90900 0 28666 28586 0x00000004 1347977263 M * ViruSzZ aham ... ok got it let me look into the log file 1347977379 M * ViruSzZ here's the relevant section triggered when the sysreq was done => http://pastebin.com/YjGBAcj1 1347977486 M * daniel_hozac the lines after that are actually the most interesting. 1347977499 M * ViruSzZ ok. let me post them for you 1347977733 M * ViruSzZ http://pastebin.com/Z1sMw0yt 1347977755 J * nkukard ~nkukard@41.177.70.149 1347978425 M * ViruSzZ is there anything else that may be tried to get any useful info for debugging this as I think the only thing I can do right now is to reboot the host machine because it's been almost an hour and nothing is happening with this process .... 1347978473 M * daniel_hozac echo d > /proc/sysrq-trigger 1347978533 Q * Aiken Read error: Operation timed out 1347978574 M * ViruSzZ kernel: [2517418.914463] SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) 1347978592 M * ViruSzZ that's from /var/log/messages 1347978600 M * ViruSzZ right after i executed echo d > 1347978607 M * Bertl what kernel/patch is that? 1347978617 M * ViruSzZ 3.5.2-2.3.4.1 1347978623 M * ViruSzZ 0.30.216-pre3034 1347978633 M * ViruSzZ CentOS release 5.8 1347978961 J * fisted ~fisted@xdsl-87-78-183-17.netcologne.de 1347979106 M * daniel_hozac kernel configured without CONFIG_LOCKDEP, i presume? 1347979151 M * ViruSzZ let me check that 1347979163 M * ViruSzZ # zgrep -i CONFIG_LOCKDEP /proc/config.gz 1347979163 M * ViruSzZ CONFIG_LOCKDEP_SUPPORT= 1347979170 M * ViruSzZ CONFIG_LOCKDEP_SUPPORT=y 1347979380 Q * fisted_ Ping timeout: 480 seconds 1347980468 M * ViruSzZ is there any theoretical explanation what may be causing this issue and how can I for example help in order to track it down? or maybe is something hardware related? 1347980490 M * daniel_hozac it looks like a locking problem 1347981116 Q * sid3windr Ping timeout: 480 seconds 1347981497 J * sid3windr luser@bastard-operator.from-hell.be 1347981868 M * Bertl off to grab some groceries .. bbl 1347981881 N * Bertl Bertl_oO 1347982067 J * thierryp_ ~thierry@2a01:e35:2e2b:e2c0:d938:409d:65cf:a709 1347982318 Q * fisted Quit: leaving 1347983918 Q * thierryp_ Remote host closed the connection 1347983954 J * thierryp_ ~thierry@2a01:e35:2e2b:e2c0:e10a:64b5:ca4d:2894 1347984235 Q * vspas Ping timeout: 480 seconds 1347984572 J * bonbons ~bonbons@2001:960:7ab:0:21ff:3077:1584:97d9 1347985774 N * Bertl_oO Bertl 1347985780 M * Bertl back now ... 1347986296 Q * thierryp Remote host closed the connection 1347987188 J * thierryp ~thierry@home.parmentelat.net 1347987221 J * kir ~kir@swsoft-msk-nat.sw.ru 1347987312 P * kir 1347987872 Q * thierryp Remote host closed the connection 1347988893 Q * thierryp_ Remote host closed the connection 1347990528 J * hparker ~hparker@2001:470:1f0f:32c:beae:c5ff:fe01:b647 1347991435 J * sannes ~ace@cm-84.211.87.28.getinternet.no 1347994548 Q * sannes Remote host closed the connection 1347994919 J * hijacker ~hijacker@cable-84-43-134-121.mnet.bg 1347995867 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:9c2e:2a1f:a0ff:532f 1347996144 J * Aiken ~Aiken@2001:44b8:2168:1000:21f:d0ff:fed6:d63f 1347996348 Q * thierryp Ping timeout: 480 seconds 1347996762 Q * clopez Read error: Operation timed out 1347997495 Q * ViruSzZ Quit: leaving 1347997510 J * fisted ~fisted@xdsl-87-78-183-17.netcologne.de 1347999670 Q * hijacker Quit: Leaving 1348000067 Q * bonbons Quit: Leaving 1348004987 J * clopez ~clopez@17.28.165.83.dynamic.mundo-r.com 1348006720 Q * nkukard Read error: Connection reset by peer 1348006745 J * nkukard ~nkukard@41.177.102.30 1348007578 J * vspas ~vspas@82-169-243-226.ip.telfort.nl 1348007765 J * vspas1 ~vspas@82-169-243-226.ip.telfort.nl 1348007767 N * ensc Guest7519 1348007777 J * ensc ~irc-ensc@p54ADF561.dip.t-dialin.net 1348007943 J * vspas2 ~vspas@82-169-243-226.ip.telfort.nl 1348008066 J * vspas3 ~vspas@82-169-243-226.ip.telfort.nl 1348008182 Q * Guest7519 Ping timeout: 480 seconds 1348008210 Q * vspas Ping timeout: 480 seconds 1348008339 Q * vspas1 Ping timeout: 480 seconds 1348008441 J * vspas ~vspas@82-169-243-226.ip.telfort.nl 1348008514 Q * vspas2 Ping timeout: 480 seconds 1348008520 J * vspas1 ~vspas@82-169-243-226.ip.telfort.nl 1348008760 Q * vspas3 Ping timeout: 480 seconds 1348008789 J * vspas2 ~vspas@82-169-243-226.ip.telfort.nl 1348008882 J * vspas3 ~vspas@82-169-243-226.ip.telfort.nl 1348008922 Q * vspas Ping timeout: 480 seconds 1348009068 J * vspas ~vspas@82-169-243-226.ip.telfort.nl 1348009182 Q * vspas1 Ping timeout: 480 seconds 1348009213 J * vspas1 ~vspas@82-169-243-226.ip.telfort.nl 1348009270 Q * vspas2 Ping timeout: 480 seconds 1348009366 Q * vspas3 Ping timeout: 480 seconds 1348009439 J * vspas2 ~vspas@82-169-243-226.ip.telfort.nl 1348009499 J * vspas3 ~vspas@82-169-243-226.ip.telfort.nl 1348009552 Q * vspas Ping timeout: 480 seconds 1348009817 Q * vspas1 Ping timeout: 480 seconds 1348009849 J * vspas ~vspas@82-169-243-226.ip.telfort.nl 1348009920 Q * vspas2 Ping timeout: 480 seconds 1348010038 J * vspas1 ~vspas@82-169-243-226.ip.telfort.nl 1348010100 J * vspas2 ~vspas@82-169-243-226.ip.telfort.nl 1348010282 Q * vspas3 Ping timeout: 480 seconds 1348010332 Q * vspas Ping timeout: 480 seconds 1348010416 J * vspas ~vspas@82-169-243-226.ip.telfort.nl 1348010522 Q * vspas1 Ping timeout: 480 seconds 1348010762 Q * vspas2 Ping timeout: 480 seconds 1348010956 J * vspas1 ~vspas@82-169-243-226.ip.telfort.nl 1348011229 J * vspas2 ~vspas@82-169-243-226.ip.telfort.nl 1348011320 Q * vspas Ping timeout: 480 seconds 1348011637 Q * vspas1 Ping timeout: 480 seconds 1348011715 Q * vspas2 Ping timeout: 480 seconds 1348012441 Q * DoberMann Remote host closed the connection 1348012643 J * DoberMann ~james@2a01:e35:8b44:84c0::2