1438304951 J * fstd_ ~fstd@xdsl-87-78-16-240.netcologne.de 1438305111 Q * fstd Read error: Connection reset by peer 1438305111 N * fstd_ fstd 1438307317 Q * arekm Read error: Connection reset by peer 1438307327 J * arekm ~arekm@ixion.pld-linux.org 1438321439 Q * arekm Read error: Connection reset by peer 1438321439 J * arekm ~arekm@ixion.pld-linux.org 1438325287 J * derjohn_mob ~aj@x590c4633.dyn.telefonica.de 1438325395 J * Ghislain ~aqueos@adsl1.aqueos.com 1438326848 M * Ghislain hi 1438326946 M * Bertl_oO hey 1438327714 J * nikolayK ~nkichukov@199.91.137.248 1438328779 M * Ghislain hey bertl, sorry for the noise yesterday, vspace is ok, dont know if you saw my pityfull explanation ;) 1438328792 M * Bertl_oO yeah I got it :) 1438328804 M * Bertl_oO good to hear that this works as expected 1438328833 M * Ghislain great, i have some kernel message too , do not know if this is vserver related i sent them by email this morning 1438328866 M * Ghislain right now i launched a bigger loop to be sure, running rigth now and will certainly run a few hours before the end 1438328935 M * Ghislain hum not sent i did not hit the send button, now i do 1438328990 M * Ghislain my loop that manage files have a strange effect: 1438328994 M * Ghislain 20861 root 20 0 33824 3020 2700 R 100.2 0.0 491:34.53 sudo vserver wheezy641 enter 1438328994 M * Ghislain 5677 root 20 0 136m 128m 2572 S 2.0 0.8 2:16.89 bash test.sh 1438329019 M * Ghislain the bash test is 2% and the vserver enter is 100% cpu, not sure if this is normal ^^ 1438329085 M * Bertl_oO 'enter' is still a hack where you basically connect outside and inside, so I wouldn't wonder too much about that 1438329092 M * Ghislain i put on purpose the test to hit the dlimit so most of the wirte fails 150000 loop of 190 files where 47k are ok then hit the barrier 1438329108 M * Ghislain ok 1438329116 M * Bertl_oO good idea! 1438330849 Q * derjohn_mob Ping timeout: 480 seconds 1438331301 M * Ghislain i am really pissed off when you have a process the kernel CANNOT kill..really i can't think how this is possible that the kernel has so little control it cannot simply kill it 1438331385 M * Bertl_oO well, it could kill it, but the result would be desastrous 1438331386 M * Ghislain i have a umount and a vserver enter process unkillable, unfrozable etc 1438331401 M * Ghislain feels like windows: just reboot ! 1438331414 M * Bertl_oO should never be really necessary 1438331435 M * Bertl_oO the question is why the umount 'hangs' 1438331450 M * Bertl_oO typically this is because you told the kernel to wait :) 1438331450 M * Ghislain well the process is 100% cpu, cannot froze it with cgroup, cannot kill it etc.. 1438331480 M * Ghislain i speak about the vserver one, the umount is laying here idle 1438331504 M * Ghislain it was testfs on xfs volume that froze and i eventually ctrl-c it 1438331520 M * Bertl_oO an it still shows 100% cpu? 1438331539 M * Bertl_oO because to have 100% cpu, the process must be still active 1438331543 M * Bertl_oO i.e. killable 1438331582 M * Ghislain the vserver process at 100% cpu is in R state and we cannot kill it with kill -9 , i moved it into a cgroup and try to froze it also failed 1438331627 M * Bertl_oO strange, what process is that? 1438331648 M * Bertl_oO can you attach to it with strace? see what it is doing? 1438331649 M * Ghislain ohoh the server seems dead, i lost all access 1438331681 M * Bertl_oO as long as the context exists, you can also enter it 1438331694 M * Ghislain yes strace was just frozen with no response 1438331718 P * undefined 1438331802 M * Ghislain well i will hve to powercycle it, too bad this test server has no console 1438331878 M * Ghislain do you think we can release a patch for the previous LTS kernel so i can upgrade my servers while we work further into 4.1 ? for the inode issue ? 1438331936 M * Bertl_oO sure, the patch is actually trivial 1438331976 M * Ghislain cool, by the way, why those lines became obsolete ? 1438331996 M * Bertl_oO I think we accidentially added them after 3.14 1438332021 M * Bertl_oO (or maybe we had a reason to do so ... have to check the logs) 1438332048 M * Ghislain that will ease the fuzzy feeling inside my troubled heart ;p 1438332110 M * Ghislain not that kernel manipulation is putting any stress , i am not stress by magic 1438332112 M * Ghislain ;) 1438332166 M * Bertl_oO http://vserver.13thfloor.at/Experimental/delta-dlimit-fix03.diff 1438332185 M * Bertl_oO should apply (maybe with offset) to 3.14+ 1438332219 M * Ghislain k thanks will try later on my 3.4 series 1438332277 M * Ghislain yes i am still at 3.4 because i do not want to have a kernel too recent compared to the debian ones as the userland is too old sometimes 1438332868 M * Bertl_oO I do not remember that the linux kernel broke userspace compatibility ever (except maybe from 1.x to 2.x) 1438333198 M * Ghislain the kernel do not bothers me, userspace programs behaving like crap and using deprecated api for decades are 1438333499 M * Ghislain when i start/stop guest on the 4.1 test server i have real issues 1438333506 M * Ghislain 6046 root 20 0 180 4 0 R 98.0 0.0 0:27.91 vwait 1438333506 M * Ghislain 6455 root 20 0 4080 688 608 R 98.0 0.0 0:27.92 reboot 1438333519 M * Ghislain vwait and reboot process at 100% cpui 1438333545 M * Ghislain still the very same trace in kern logs 1438333586 M * Ghislain /usr/sbin/vwait --timeout 300 --status-fd 3 40192 1438333682 M * Ghislain bertl: seems there is a lock somewhere revealed by vwait 1438333729 M * Ghislain strace show nothing, lsof show only root directory, vwait binary dev/null and a bunch of /tmp/vwaitstats files 1438333822 M * Bertl_oO can you try it on a cleanly booted system and record all the kernel messages for me? 1438333839 M * Ghislain i just restarted it so it was clean 1438333853 M * Bertl_oO okay, then please upload the dmesg output 1438333853 M * Ghislain "reboot" command is runnign 100%cpu 1438334809 Q * FloodServ charon.oftc.net services.oftc.net 1438335083 J * derjohn_mob ~aj@2001:6f8:1337:0:4cc5:b74a:63c8:bc97 1438335297 J * FloodServ services@services.oftc.net 1438335613 Q * arekm Read error: Connection reset by peer 1438335636 J * arekm ~arekm@ixion.pld-linux.org 1438337062 J * undefined ~undefined@00011a48.user.oftc.net 1438338176 Q * arekm Read error: Connection reset by peer 1438338190 J * arekm ~arekm@ixion.pld-linux.org 1438338341 Q * FloodServ charon.oftc.net services.oftc.net 1438342846 M * Bertl_oO off for a nap ... bbl 1438342857 N * Bertl_oO Bertl_zZ 1438343960 Q * fstd Remote host closed the connection 1438343972 J * fstd ~fstd@xdsl-87-78-9-154.netcologne.de 1438345083 Q * Aiken Remote host closed the connection 1438350336 J * FloodServ services@services.oftc.net 1438350377 Q * FloodServ charon.oftc.net services.oftc.net 1438350558 J * Gremble ~Gremble@cpc29-aztw22-2-0-cust128.18-1.cable.virginm.net 1438351811 J * FloodServ services@services.oftc.net 1438352241 J * arekm_ ~arekm@ixion.pld-linux.org 1438352248 Q * arekm Read error: Connection reset by peer 1438352833 J * wicope ~wicope@0001fd8a.user.oftc.net 1438354533 N * Bertl_zZ Bertl 1438354535 M * Bertl back now ... 1438354740 Q * Gremble Quit: I Leave 1438355111 M * Ghislain server just came back 1438355122 M * Ghislain same issue trying to replicate 1438355126 M * Ghislain 4341 root 20 0 180 4 0 R 100.2 0.0 0:32.56 vwait 1438355126 M * Ghislain 4749 root 20 0 4080 732 656 R 100.2 0.0 0:32.56 reboot 1438355130 M * Ghislain both 100% 1438355291 M * Ghislain humm the dbg kernel package seems to not be good when i choose the debug options 1438355310 M * Ghislain so even with the kernel with debug i do not have the vmlinx file 1438355399 M * Ghislain whne i do it on the compile machine i have no symbols 1438355401 M * Bertl best compile the kernel the conventional way (i.e. by hand) 1438355421 M * Bertl you then get the vmlinux and you can also iterate faster on changes 1438355433 M * Bertl (as recompiles take only a few seconds) 1438356596 M * Ghislain addr2line MUST be run on the system ? i cannot run into the compile rig ? 1438356640 M * Bertl no, you can run it wherever you want 1438356652 M * Bertl as long as it has access to the correct vmlinux 1438356675 M * Ghislain buildwheezy64:~/linux-4.1.3# addr2line -e /root/linux-4.1.3/vmlinux ffffffffa8597dd8 1438356676 M * Ghislain ??:0 1438356700 M * Bertl check that the vmlinux was not stripped 1438356713 M * Bertl (and is the correct one, i.e. the one from the kernel you booted) 1438356791 M * Ghislain i hate this ^^ 1438356933 M * Bertl http://pastebin.com/raw.php?i=xYc4RVag 1438356978 M * Bertl undefined: ping? 1438358029 Q * derjohn_mob Ping timeout: 480 seconds 1438358913 M * Ghislain i found the pb this is your fault ! ;) 1438358923 M * Ghislain that dam tmpfs of 64m 1438359341 M * Bertl I don't think I have anything to do with your 64m tmpfs :) 1438359555 M * Ghislain well you might have to proove that statement ! 1438359588 M * Ghislain ;p 1438361210 M * Ghislain ok i give up i cannot have any debug symbol working 1438361526 Q * nikolayK Quit: Leaving 1438361557 M * Bertl well, if you provide me with your .config, I can build the kernel with debug information 1438361582 M * Bertl but please provide a cleaned up config (not the default debian kitchen sink one) as I have limited resources 1438371090 J * derjohn_mob ~aj@x590c4633.dyn.telefonica.de 1438373932 J * Aiken ~Aiken@d63f.h.jbmb.net 1438376045 J * derjohn_mobi ~aj@x4db244bb.dyn.telefonica.de 1438376174 Q * Aiken Ping timeout: 480 seconds 1438376463 Q * derjohn_mob Ping timeout: 480 seconds 1438376981 Q * AndrewLee Ping timeout: 480 seconds 1438377147 J * Aiken ~Aiken@quarry.jbmb.net 1438378194 Q * Ghislain Quit: Leaving. 1438378421 N * Bertl Bertl_oO 1438378475 J * AndrewLee ~andrew@210.240.39.201 1438378826 J * Ghislain ~aqueos@adsl1.aqueos.com 1438383111 Q * Ghislain Quit: Leaving. 1438383627 J * Ghislain ~aqueos@adsl1.aqueos.com 1438383702 Q * Ghislain 1438384231 J * Ghislain ~aqueos@adsl1.aqueos.com 1438384317 Q * Ghislain 1438385081 Q * wicope Remote host closed the connection 1438387160 Q * fstd Remote host closed the connection 1438387171 J * fstd ~fstd@xdsl-84-44-236-89.netcologne.de