1347321979 Q * nakacya Remote host closed the connection 1347322002 J * nakacya ~nakacya@KD118152083243.ppp-bb.dion.ne.jp 1347322488 Q * nakacya Ping timeout: 480 seconds 1347323227 J * bergerx_ ~bergerx@178.233.12.223 1347323675 Q * bergerx Ping timeout: 480 seconds 1347326088 Q * Jb_boin Ping timeout: 480 seconds 1347326130 J * Jb_boin ~dedior@proxad.eu 1347326845 Q * AndrewLee Read error: Connection reset by peer 1347327526 J * AndrewLee ~andrew@n201.enc.hlc.edu.tw 1347327640 Q * clopez Read error: Operation timed out 1347330473 Q * fisted Read error: Operation timed out 1347330553 J * fisted ~fisted@xdsl-87-78-190-248.netcologne.de 1347331745 Q * nlm Ping timeout: 480 seconds 1347331888 Q * nkukard Ping timeout: 480 seconds 1347331914 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347333511 Q * fisted Quit: brb 1347333544 J * fisted ~fisted@xdsl-87-78-190-248.netcologne.de 1347337899 J * vn585 R3h@89.205.104.146 1347338283 Q * vn585 Read error: Connection reset by peer 1347338295 J * vn585 R3h@89.205.104.146 1347340121 M * Bertl off to bed now ... have a good one everyone! 1347340125 N * Bertl Bertl_zZ 1347341203 Q * nkukard Quit: Leaving 1347345932 J * ghislain ~AQUEOS@adsl2.aqueos.com 1347352596 J * kir ~kir@swsoft-msk-nat.sw.ru 1347353297 Q * FireEgl Read error: Connection reset by peer 1347353906 J * nakacya ~nakacya@s1106021.xgsspn.imtp.tachikawa.spmode.ne.jp 1347354328 J * FireEgl FireEgl@2001:470:e5ad:1:d576:3206:e173:bca4 1347354846 P * kir PING 1347354846 1347356014 J * clopez ~clopez@fanzine.igalia.com 1347356756 Q * FireEgl Ping timeout: 480 seconds 1347356813 Q * Guy- Ping timeout: 480 seconds 1347357159 J * vspas ~vspas@87.213.36.165 1347357397 J * Guy- ~korn@elan.rulez.org 1347357887 Q * Guy- Ping timeout: 480 seconds 1347358273 J * Guy- ~korn@elan.rulez.org 1347358321 M * WMP http://www.change.org/petitions/youtube-googlede-allow-third-party-recording-tools-for-youtube-freedomonyoutube 1347358746 Q * nakacya Remote host closed the connection 1347360922 J * BenG ~bengreen@cpc10-aztw24-2-0-cust114.aztw.cable.virginmedia.com 1347363277 J * vn131 R3h@89.205.104.146 1347363368 Q * vn585 Ping timeout: 480 seconds 1347363703 J * nlm ~nlm@host77.190-30-39.telecom.net.ar 1347364235 Q * Aiken Remote host closed the connection 1347366405 J * vn585 R3h@89.205.104.146 1347366545 Q * vn131 Ping timeout: 480 seconds 1347367405 Q * vn585 Ping timeout: 480 seconds 1347367951 J * fleischergesell ~fleischer@p4FDF1637.dip.t-dialin.net 1347368093 M * fleischergesell Hey, we use hosts with a 3.2.4 kernel and seem unable to make VIRT_CPU working in guests 1347368134 M * fleischergesell both, VIRT_LOAD and VIRT_MEM work just fine, but for cpus we have had no luck so far 1347368169 M * fleischergesell we use cgroups to limit guests to certain cores which does work like a charm, just the usage top or htop report in a guest will always display the overall host usage 1347368220 M * fleischergesell is this a known issue for linux-vserver? if not, how can we debug and fix it for our environment? 1347368250 J * vn585 R3h@89.205.104.146 1347369112 J * FireEgl FireEgl@2001:470:e5ad:1:5cf2:64c:97a9:ee1e 1347369463 Q * vn585 Ping timeout: 480 seconds 1347369499 J * nakacya ~nakacya@KD118152083243.ppp-bb.dion.ne.jp 1347369886 Q * ensc|w Remote host closed the connection 1347369895 J * ensc|w ~ensc@www.sigma-chemnitz.de 1347370228 J * fleischergesell1 ~fleischer@p5B0A0709.dip.t-dialin.net 1347370363 J * vn585 R3h@89.205.104.146 1347370445 Q * fleischergesell Ping timeout: 480 seconds 1347371308 N * Bertl_zZ Bertl 1347371313 M * Bertl morning folks! 1347371329 M * Bertl fleischergesell1: what Linux-VServer patch? 1347373964 M * fback morning Bertl :) 1347373983 M * fback Bertl: I know your answer ;-) but I have grsec related question 1347374044 M * fback Bertl: our admin team gave a try this new grsec patch, and they found devtmpfs doesn't work 1347374099 M * fback maybe you have any idea where to start looking at? 1347374406 M * Bertl the grsec patch, I presume :) 1347374427 M * Bertl well, the question is, what exactly isn't working? 1347374476 J * fisted_ ~fisted@xdsl-87-78-186-22.netcologne.de 1347374618 Q * fisted Read error: Operation timed out 1347374903 M * fback /dev/ after mount is empty 1347375020 M * daniel_hozac like it should be? 1347375155 M * fleischergesell1 Bertl: We are using VS-API: 0x00020308 1347375181 M * fback daniel_hozac: I hate to be between our admins and the channel... As far as I understood, kernel should create inodes/devices as required, and what's more new udev relies on this feature 1347375242 M * daniel_hozac the kernel did it with devfs. "devtmpfs" is just a tmpfs that is populated by userspace. 1347375286 M * fleischergesell1 should be patch version "vs2.3.2.7.diff" 1347375328 M * fback daniel_hozac: they say kernel with plain vserwer patch works as expected, the problem is only with vserver+grsec combo 1347375391 M * daniel_hozac check that grsec doesn't switch nodev on by default for tmpfs perhaps? 1347375927 M * Bertl yes, devfs is kind of back, but it needs to be enabled in the kernel 1347378257 J * uranus ~uranus@ip-2-205-164-241.web.vodafone.de 1347378280 M * uranus Bertl, the problem still exists with memcg-fix06: http://paste.linux-vserver.org/22935 1347378318 M * Bertl which suggests that it is not Linux-VServer related 1347378361 M * Bertl in the trace you uploaded the Linux-VServer code doesn't show up, which seems to confirm this 1347378417 M * Bertl but if you can, please annotate this trace and if you can recreate the issue easily, enable mutex debugging and try again 1347378458 M * uranus i dunno how to recreate it, most of the time it happens when in which case ever a guest is restarted 1347378483 M * uranus how do I enable mutex debugging? 1347379219 M * Bertl CONFIG_DEBUG_MUTEXES 1347379388 M * uranus thx 1347379595 J * bonbons ~bonbons@2001:960:7ab:0:6d07:8a11:3711:e358 1347379707 Q * ghislain Quit: Leaving. 1347380360 Q * vspas Ping timeout: 480 seconds 1347381094 M * uranus Bertl, CONFIG_DEBUG_MUTESXES was already enabled in my kernel build 1347381311 M * Bertl ah, yes, I missed the last 3 lines in the trace 1347381386 M * Bertl btw, what is directly above the trace (in your log file) i.e. what happened right before [24800.516811]? 1347381435 M * uranus any hints where i can start debugging this issue? 1347381765 M * uranus in dmesg nothing 1347381777 M * uranus but 2 minutes before that i executed vserver stop 1347381858 M * Bertl so the trace starts without any 'warning' or similar message? 1347381871 M * uranus yes 1347381930 M * uranus to ssh inside the guest was no connection possible 1347381974 M * Bertl well, to be honest, I think the trace is incomplete (well, not the trace but the output at least) 1347381980 M * uranus all processes inside the guest were in State R 1347381989 M * Bertl how did you capture it? 1347382268 M * uranus missed these 2 lines (big sorry): 1347382271 M * uranus INFO: task awk:21772 blocked for more than 120 seconds. 1347382271 M * uranus "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 1347382294 M * Bertl ah, so again it is the soft lockup causing the trace 1347382334 M * Bertl did you try to simply disable this feature or at least set the time longer? 1347382347 M * uranus hmm why is this thing not deactivated :/ 1347382360 M * daniel_hozac 2 minutes seems pretty long for it to be stuck in the same place. 1347382444 M * Bertl yes, but I've seen this many times in all the 3.x kernels (with and without Linux-VServer) 1347382480 M * Bertl maybe the measurement is flawed, maybe it overruns at some point, or maybe the scheduler is buggy 1347382483 M * uranus hmm this awk is the awk spawned by util vserver which does check /proc/cgroups 1347382500 M * uranus Detect Hard and Soft Lockups is not activcated 1347382516 M * uranus only "Detect Hung Tasks" is activated with 120 seconds 1347382536 M * Bertl yeah, that's the one triggering here 1347382590 M * uranus so i also disable detect hung tasks 1347382609 M * daniel_hozac well, it's an easy way to get traces of tasks that hang. 1347382699 M * uranus but if that feature is causing my awk (form vserver stop) hung in state d and no further vserver action is possible i will try to deactivate that "feature" 1347382806 M * daniel_hozac i doubt it is causing that, it's most likely just showing it 1347382812 J * ghislain ~AQUEOS@adsl2.aqueos.com 1347382842 M * uranus as soon this happens my kernel goes south :/ 1347382848 M * Bertl yes, but AFAICT, once it shows, it breaks the kernel 1347382898 M * uranus the build with this deactivated just runs - i'll report back 1347383007 Q * ghislain 1347383183 M * uranus now offline, maybe till tomorrow :) 1347383197 Q * uranus Quit: Verlassend 1347383233 M * Bertl daniel_hozac: do you have the hung task detection enabled anywhere on a somewhat recent kernel? 1347383427 M * daniel_hozac hmm, it's not enabled on my recent kernel machines, no. 1347383451 M * daniel_hozac i'll enable it in the next update. 1347383558 M * daniel_hozac (those are kvm hosts though, so unlikely to see much action) 1347384468 M * Bertl okay ... off for a nap .. bbl 1347384480 N * Bertl Bertl_zZ 1347385022 J * ghislain ~AQUEOS@adsl2.aqueos.com 1347386709 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347388349 J * sannes ~ace@cm-84.211.87.28.getinternet.no 1347388585 Q * clopez Ping timeout: 480 seconds 1347389034 M * fback uranus: I've had similar issues some time ago, replacing mobo solved it 1347390190 Q * BenG Quit: I Leave 1347390342 N * ensc Guest6703 1347390352 J * ensc ~irc-ensc@p54ADEB6D.dip.t-dialin.net 1347390760 Q * Guest6703 Ping timeout: 480 seconds 1347391435 Q * Jb_boin Read error: Connection reset by peer 1347391905 Q * Chlorek Ping timeout: 480 seconds 1347392039 J * Chlorek chlorek@chlorek.com 1347394508 J * Jb_boin ~dedior@proxad.eu 1347394515 Q * sannes Remote host closed the connection 1347395564 N * Bertl_zZ Bertl 1347395985 Q * bonbons Quit: Leaving 1347397313 Q * ghislain Quit: Leaving. 1347397614 Q * nkukard Read error: Connection reset by peer 1347397665 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397678 Q * nkukard Read error: Connection reset by peer 1347397704 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397720 Q * nkukard Read error: Connection reset by peer 1347397770 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397781 Q * nkukard Read error: Connection reset by peer 1347397816 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397829 Q * nkukard Read error: Connection reset by peer 1347397876 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397886 J * uranus ~uranus@62.152.161.117 1347397889 Q * nkukard Read error: Connection reset by peer 1347397941 M * uranus fback, thx for your reply: i have this issue on more boxes one with a core i7, one with a xeon E56 and one with a xeon e5-, so i think it is not related to a mobo 1347397949 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397960 Q * nkukard Read error: Connection reset by peer 1347397989 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347397998 Q * nkukard Read error: Connection reset by peer 1347398023 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398023 M * uranus Bertl, btw change to kernel 3.2.28 did not solve the issue, i'll test now without "hung task detection" 1347398038 Q * nkukard Read error: Connection reset by peer 1347398063 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398078 Q * nkukard Read error: Connection reset by peer 1347398113 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398124 Q * nkukard Read error: Connection reset by peer 1347398148 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398166 Q * nkukard Read error: Connection reset by peer 1347398207 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398220 Q * nkukard Read error: Connection reset by peer 1347398233 M * Bertl uranus: do they have something in common except for Linux-VServer? 1347398244 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398255 M * uranus no vanilla kernel + lv patch 1347398261 Q * nkukard Read error: Connection reset by peer 1347398285 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398300 Q * nkukard Read error: Connection reset by peer 1347398324 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398329 M * Bertl what about networkor mounts? io subsystem? 1347398337 M * Bertl *network 1347398342 Q * nkukard Read error: Connection reset by peer 1347398366 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398375 M * uranus network mounts are 2 nfs mounts for backup and "images", vserver are all on local storage 1347398382 Q * nkukard Read error: Connection reset by peer 1347398385 M * uranus local storage is either ext3 or ext4 1347398389 M * uranus issue happens on both 1347398404 M * Bertl those nfs mounts are on all problematic machines? 1347398419 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398431 Q * nkukard Read error: Connection reset by peer 1347398450 M * uranus Bertl, yes 1347398456 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398458 M * uranus i use nfsv3 1347398473 Q * nkukard Read error: Connection reset by peer 1347398497 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398512 Q * nkukard Read error: Connection reset by peer 1347398558 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398570 Q * nkukard Read error: Connection reset by peer 1347398592 M * Bertl I presume they are not strictly needed for normal operation, could you disable it completely on one machine next time you reboot? 1347398595 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398611 Q * nkukard Read error: Connection reset by peer 1347398635 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398651 Q * nkukard Read error: Connection reset by peer 1347398676 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398678 F * ChanServ +o Bertl 1347398691 Q * nkukard Read error: Connection reset by peer 1347398728 J * nkukard ~nkukard@41-133-138-36.dsl.mweb.co.za 1347398729 F * Bertl +b *!*nkukard@*.dsl.mweb.co.za 1347398738 Q * nkukard Read error: Connection reset by peer 1347398771 M * uranus it is commented out, next reboot is sheduled in about 9h 1347398822 M * uranus then i have one host without hung task detection and one host without hung task detection and without nfs mounts 1347398929 M * Bertl excellent! 1347398954 M * Bertl btw, could you upload your .config for the kernel, I presume it is pretty identical for those three as well 1347398967 M * uranus it completly identical 1347399107 M * Bertl the nfs will be interesting, as I'm using nfsv3 as well and I just checked, all the machines which exposed the issue had nfs mounts 1347399161 M * uranus first time if seen state d processes on host was von kernel 3.2 1347399174 M * uranus s/von/from/g 1347399347 M * uranus my config: http://www.your-filehosting.com/qan8b8bfs9k6/config.html 1347399513 M * uranus does this link work Bertl ? 1347399680 M * uranus the config was once based on the debian 3.2 kernel config 1347400410 Q * fleischergesell1 Ping timeout: 480 seconds 1347400605 M * Bertl link works, tx 1347400728 M * uranus btw it's the 3.4 config 1347401188 M * uranus Bertl, i'm off now, will read you irc log to answer to any questions on your side (if you have any) for now big thanks for your work! 1347402160 Q * vn585 Ping timeout: 480 seconds 1347402404 M * Bertl you're welcome! 1347403143 J * Aiken ~Aiken@2001:44b8:2168:1000:21f:d0ff:fed6:d63f