1229990594 Q * dowdle Remote host closed the connection 1229993551 J * ghislainocfs2 ~Ghislain@adsl2.aqueos.com 1229993838 Q * ghislainocfs21 Ping timeout: 480 seconds 1229995191 Q * harry Server closed connection 1229995192 J * harry ~harry@d51A461B4.access.telenet.be 1229995207 Q * takeru Quit: takeru 1229996463 J * takeru ~takeru@nttkyo586012.tkyo.nt.ftth.ppp.infoweb.ne.jp 1229997305 Q * takeru Quit: takeru 1229998182 Q * BobR_zZ Server closed connection 1229998184 J * BobR_zZ odie@IRC.13thfloor.at 1229998410 J * takeru ~takeru@nttkyo586012.tkyo.nt.ftth.ppp.infoweb.ne.jp 1230003158 J * ghislainocfs21 ~Ghislain@adsl2.aqueos.com 1230003460 Q * ghislainocfs2 Ping timeout: 480 seconds 1230004066 Q * matti Server closed connection 1230004068 J * matti matti@acrux.romke.net 1230008240 Q * hparker Quit: Read error: 104 (Peer reset by connection) 1230009694 J * hparker ~hparker@2001:470:1f0f:32c:212:f0ff:fe0f:6f86 1230012022 Q * eyck Server closed connection 1230012023 J * eyck WZpmo08R@nat03.nowanet.pl 1230012383 Q * balbir_ Read error: Operation timed out 1230013331 Q * geb Remote host closed the connection 1230015075 Q * padde Server closed connection 1230015079 J * padde ~padde@patrick-nagel.net 1230015188 Q * faheem synthon.oftc.net resistance.oftc.net 1230015188 Q * micah synthon.oftc.net resistance.oftc.net 1230015188 Q * AndrewLee synthon.oftc.net resistance.oftc.net 1230015188 Q * dkg synthon.oftc.net resistance.oftc.net 1230015188 Q * sardyno synthon.oftc.net resistance.oftc.net 1230015258 J * AndrewLee ~andrew@flat.iis.sinica.edu.tw 1230015258 J * faheem ~faheem@cpe-071-077-007-143.nc.res.rr.com 1230015258 J * micah ~micah@micah.riseup.net 1230015266 J * sardyno ~me@pool-96-235-18-120.pitbpa.fios.verizon.net 1230015266 J * dkg ~dkg@lair.fifthhorseman.net 1230015468 J * balbir_ ~balbir@59.145.136.1 1230015916 Q * mEDI_S Server closed connection 1230015977 J * mEDI_S ~medi@snipah.com 1230016083 J * sharkjaw ~gab@149-67-194.231210.adsl.tele2.no 1230016274 J * doener ~doener@i577BACCE.versanet.de 1230016379 Q * doener_ Ping timeout: 480 seconds 1230017282 J * chi6IT41 ~chigital@services.mivitec.net 1230017623 Q * balbir_ Ping timeout: 480 seconds 1230017743 J * mtg ~mtg@dialbs-088-079-143-204.static.arcor-ip.net 1230017884 Q * chi6IT41 Ping timeout: 480 seconds 1230018527 J * chi6IT41 ~chigital@services.mivitec.net 1230018531 J * balbir_ ~balbir@59.145.136.1 1230020784 Q * padde Remote host closed the connection 1230020813 J * padde ~padde@patrick-nagel.net 1230021725 Q * infowolfe Quit: Leaving 1230021745 J * dna ~dna@77-207-103-86.dynamic.dsl.tng.de 1230022599 Q * wibble Remote host closed the connection 1230023048 J * kir ~kir@swsoft-msk-nat.sw.ru 1230023259 J * infowolfe ~infowolfe@c-24-21-204-37.hsd1.or.comcast.net 1230023432 J * glen__ ~glen@elves.delfi.ee 1230023527 M * glen__ hia. i have like 34 ip addresses (eth0, eth1) assigned and like 23 vservers on 2.6.27.8-1, and having some network troubles: address not supported errors time to time 1230023530 M * glen__ some know problem? 1230023606 Q * infowolfe Read error: Operation timed out 1230023654 J * infowolfe ~infowolfe@c-24-21-204-37.hsd1.or.comcast.net 1230023774 J * mtgx ~mtg@dialbs-088-079-143-204.static.arcor-ip.net 1230023793 Q * mtg Ping timeout: 480 seconds 1230023950 M * Supaplex a word for word error message might help 1230024545 J * esa` ~esa@ip-87-238-2-45.static.adsl.cheapnet.it 1230024553 Q * esa Ping timeout: 480 seconds 1230025001 J * friendly ~friendly@ppp118-208-135-226.lns10.mel4.internode.on.net 1230026674 Q * nenolod Quit: Leaving 1230026701 J * nenolod nenolod@petrie.dereferenced.org 1230027318 J * larsivi ~larsivi@9.80-202-30.nextgentel.com 1230027867 Q * friendly Quit: Leaving. 1230027923 Q * larsivi Ping timeout: 480 seconds 1230028257 Q * sharkjaw Remote host closed the connection 1230029970 J * FireEgl FireEgl@173-16-9-10.client.mchsi.com 1230031846 Q * balbir_ Ping timeout: 480 seconds 1230033068 J * gnuk ~F404ror@pla93-3-82-240-11-251.fbx.proxad.net 1230034801 J * SlackLnx ~Lee@bl7-147-46.dsl.telepac.pt 1230036755 Q * SlackLnx Quit: I'll Be Back 1230038799 Q * bibabu Quit: Coyote finally caught me 1230038804 J * bibabu bibabu@status2k.de 1230038945 Q * bibabu 1230038998 J * bibabu bibabu@vserv2.de 1230039317 J * bibabu` bibabu@vserv2.de 1230039319 Q * bibabu` 1230039327 J * bibabu` bibabu@vserv2.de 1230039389 Q * Aiken Remote host closed the connection 1230039479 Q * bibabu Ping timeout: 480 seconds 1230040710 J * SlackLnx ~Lee@bl7-147-46.dsl.telepac.pt 1230040943 Q * hparker Quit: Read error: 104 (Peer reset by connection) 1230040982 M * maharaja Bertl_zZ: it seems to be vserver related. i get exactly the same errors with the current debian backports kerlen (no grsec) 1230041572 N * Bertl_zZ Bertl 1230041576 M * Bertl morning folks! 1230041608 M * Bertl maharaja: could you please try with a mainline kernel (i.e. non debian) and let me know if whatever you see happens there too? 1230041637 M * Bertl and if so, could you give me some details about the problems? 1230041902 M * mnemoc it's debian, that's the problem :) 1230041928 M * Bertl could be, could be unrelated .. we'll see 1230041944 M * maharaja might be hardening patches in debian lenny 1230042017 M * maharaja anyways, i planned to retest with 2.6.22.19-vs2.2.0.7 but today only had the debian one at hand 1230042034 M * maharaja is 2.6.22.19-vs2.2.0.7 ok? 1230042042 M * Bertl what kernel version is debian? 1230042091 M * Bertl i.e. the kernel you were seeing problems with? 1230042116 M * maharaja 2.6.22.19-grsec2.1.11-vs2.2.0.7 (mainline with patch from linux-vserver.org website) 1230042139 M * maharaja and today vmlinuz-2.6.26-bpo.1-vserver-amd64 1230042182 M * maharaja which should be one of the 2.3.0.35 patches 1230042218 M * Bertl so, let's take one step back first, and talk about the issues you are seeing yes? 1230042220 M * maharaja guests are 64bit 1230042223 M * maharaja ok 1230042330 M * maharaja the hardware is: intel core2duo E8400 on intel g31 chipset 1230042369 M * maharaja host is: debian etch, 64bit 1230042379 M * maharaja currently running 2.6.22.19-grsec2.1.11-vs2.2.0.7 1230042443 M * Bertl okay 1230042447 M * maharaja util-vserver is 0.30.216~r2772-6 from debian sid 1230042468 M * maharaja guest is: debian lenny, 64bit, basically build with the "vserver build" cmd 1230042540 M * Bertl okay 1230042573 M * maharaja i currently also have a lenny 32bit at hand (fyi) 1230042602 M * maharaja the first thing i noticed was that apache failed to start with "8671 Bus error $HTTPD ${APACHE_ARGUMENTS} -k $ARGV" 1230042624 M * Bertl okay, would point towards grsec 1230042645 M * maharaja digging into this problem i found out that it was not only apache2 1230042692 M * Bertl just for a check, that happens on a 64bit guest, yes? 1230042709 M * maharaja http://paste.linux-vserver.org/12669 1230042722 M * maharaja yes 1230042740 M * maharaja ldd /usr/sbin/apache2 gives a bus error 1230042758 M * maharaja ldd /usr/lib/libaprutil-1.so.0 too 1230042765 M * Bertl hehe, nice ... 1230042790 M * maharaja # ldd libldap-2.4.so.2 too 1230042797 M * maharaja see pastebin for more erros 1230042804 M * Bertl you are booted with that kernel atm? 1230042809 M * maharaja yes 1230042811 M * maharaja on the 32bit guest 1230042813 M * maharaja it seems ok 1230042824 M * Bertl okay, try the following from the host 1230042838 M * Bertl chroot /path/to/guest/ 1230042848 M * Bertl ldd /usr/lib/libaprutil-1.so.0 1230042865 M * maharaja /usr/bin/ldd: line 117: 10208 Bus error LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@" 1230042888 M * Bertl so, we get that on the 'host' too, which almost excludes Linux-VServer 1230042901 M * maharaja without chroot, it works 1230042907 M * maharaja # ldd /vservers/k000425//usr/lib/libaprutil-1.so.0 1230042917 Q * davidkarban Quit: Ex-Chat 1230042919 M * Bertl different library set 1230042988 M * Bertl but you can try to copy the ldd script into the guest just to verify that it isn't a subtle change to ldd (just to make sure) 1230043017 M * Bertl now, the options here are: 1230043032 M * Bertl - completely fubared library which /usr/lib/libaprutil-1.so.0 depends on 1230043051 M * Bertl - chroot 'breaks' some part of grsec 1230043072 M * Bertl - grsec setup changed and this bus error is kind-of intentional for that config :) 1230043098 M * Bertl - and finally some weird kernel/grsec/Linux-VServer bug 1230043113 M * maharaja Bertl: using ldd from the host does not change anything 1230043126 M * Bertl do you have enough space to make a copy of the host system into some dir? 1230043150 M * Bertl (based on the filesystem, rsync or dump should do the trick) 1230043285 M * maharaja by that, you mean that i should use the host system to create a guest, right? 1230043299 M * maharaja for the grsecurity thing: the 32bit guest - also lenny - does not have this issues 1230043321 M * Bertl well, basically not a guest, just a chroot with most of the host contents 1230043328 M * maharaja ok 1230043342 M * Bertl all I want you do to there is to chroot into it, and ldd one of the sensitive libraries 1230043361 M * Bertl might need to do a double chroot and a --bind mount to get to the guest data 1230043362 M * maharaja ok 1230043413 M * maharaja i would create a new LV for this task. this shouldn't be any issue, right? 1230043463 M * maharaja (i forgot that the fs stack is: sdX -> raid1 (md) -> lvm -> xfs 1230043533 M * Bertl well, create the volume, but do not use the guest creation mechanisms 1230043546 M * Bertl just rsync or dump/restore the host filesystem there 1230043570 M * maharaja dumping is allready in progress 1230043645 M * Bertl just as sidenote, wasn't there some bug with xfs mmap, some time ago?) 1230044698 M * mnemoc hi, dumb OT question, is the order of the mount options in /proc/mount always the same? more exactly, is rw/ro always the first? 1230044766 M * Bertl well, no and yes :) 1230045257 M * mnemoc *G* 1230045274 M * maharaja Bertl: sorry for the delay 1230045278 M * maharaja i did so 1230045290 M * mnemoc Bertl: the yes to the second makes me happy enough :) 1230045293 M * maharaja and there does not seem to be any problem when using chroot 1230045297 M * maharaja and then ldd'ing the files 1230045349 M * Bertl okay, so I presume it is the guest data which is messed up 1230045352 M * maharaja ldd /usr/sbin/apache2 works, 1230045369 M * maharaja ldd /usr/lib/libapr-1.so.0 too 1230045385 M * Bertl to double check that, it would make sense to install a new guest (from scratch) and check there 1230045401 M * Bertl if that works fine (with chroot/ldd) then compare the libraries 1230045426 M * maharaja well, the libraries from host and guest are supposted to be different as host = etch and guest = lenny 1230045454 M * maharaja i googled a little bit and the debian folks seem to have incorporated some memory protection stuff 1230045484 M * Bertl ah, and now it is biting them in the ass ... nice :) 1230045504 M * maharaja i don't know for sure thou :) 1230045507 M * Bertl okay, in this case, the best approach would be the followin: 1230045516 M * maharaja what bugs me is that the 32bit lenny runs fine but the 64bit lenny is messed up 1230045558 M * Bertl first, get a mainline, vanilla kernel, no patches at all (e.g. 2.6.22.19) and see if the chroot/ldd case with one of the lenny guests fails 1230045583 M * Bertl if that fails, you're done, debian managed to get it incompatible with that kernel 1230045601 M * Bertl if that succeeds, add the Linux-VServer patch, and retest, again with chroot/ldd 1230045612 M * Bertl if that fails, we have a problem to look into 1230045629 M * Bertl if that succeeds, continue adding patches (like grsec :) until it fails :) 1230045640 M * maharaja ok 1230045653 M * Bertl if that fails, send a bug report to debian, that their kernel is mysteriously broken :) 1230045677 M * Bertl (or whoever is doing the vserver+grsec thingy) 1230045748 M * maharaja btw - http://people.linux-vserver.org/~harry/patch-2.6.22.19-vs2.2.0.7-grsec2.1.11-20080317.diff is doing the vserver+grsec thingy 1230045755 M * maharaja ;) 1230045760 M * maharaja *setting up build env* 1230045783 M * Bertl okay, so you should be done after vanilla and vanilla+Linux-VServer :) 1230045822 M * maharaja right 1230046598 J * larsivi ~larsivi@9.80-202-30.nextgentel.com 1230047136 Q * FireEgl Read error: Connection reset by peer 1230047324 M * maharaja Bertl: ok, stock 2.6.22.19 has the same issues 1230047330 M * maharaja s/stock/vanilla/ 1230047360 M * Bertl good, so I'd now suggest to try two things 1230047390 M * Bertl one, while you are at it, try to debootstrap a lenny 64bit on that kernel into some dir 1230047411 M * Bertl and get one of the 'broken' libraries there too, just to check with ldd 1230047420 M * Bertl (to definitely rule out a broken guest) 1230047442 M * Bertl and two: check with a recent vanilla kernel, e.g. 2.6.27.10 1230047456 M * Bertl (again vanilla) if that fails, have a chat with the debian folks :) 1230047481 M * maharaja mhm 1230047482 M * maharaja ok 1230047501 M * maharaja btw - anything useful you can read from: http://paste.linux-vserver.org/12667 ? 1230047509 M * maharaja e.h. whom from debian to contact? 1230047563 M * Bertl the maintainer for 'lib64/ld-linux-x86-64.so.2' 1230047596 J * Pazzo ~ugelt@reserved-225136.rol.raiffeisen.net 1230048235 M * maharaja mhm ... i kind of fail to correctly use cdebootstrap ;) 1230048241 M * maharaja *thinking* *working it out* 1230048278 J * dowdle ~dowdle@scott.coe.montana.edu 1230048414 M * maharaja ok, cdebootstrap lacks the script for lenny, debootstrap does not ... weired but true 1230048790 J * geb ~geb@AOrleans-151-1-68-15.w90-21.abo.wanadoo.fr 1230049658 Q * klikz Server closed connection 1230049661 J * klikz mike@no.phear.eu 1230049700 M * maharaja Bertl: the newly created guest works 1230049735 M * maharaja i kind of "accidentally" ran ldconfig on the non-functional guest 1230049748 M * Bertl okay? 1230049748 M * maharaja and it showed some truncated libraries 1230049761 M * Bertl well, there you go ... silent xfs corruption :) 1230049795 M * maharaja http://paste.linux-vserver.org/12670 1230049810 M * maharaja mhmmmm 1230049851 M * maharaja damn it 1230049852 M * maharaja :) 1230049870 M * Bertl okay, does that account for all the issues you were seeing? 1230049885 M * maharaja im checking right now 1230049990 M * maharaja any hint on where to find information about this xfs issue? and/or when it has been fixed? 1230050041 M * Bertl well, that was actually a shot in the dark .. i.e. I know that xfs had some kind of fs corruption every now and then, and I think I remember reading something on lkml or so ... 1230050054 Q * chi6IT41 Ping timeout: 480 seconds 1230050055 M * maharaja the first hits i get are from 2002 or so ;) 1230050079 M * maharaja mhm 1230050080 M * maharaja i c 1230050086 M * Bertl if you want to investigate, search the kernel changelogs 1230050101 M * Bertl they should show xfs related fixes 1230050123 M * maharaja im currently trying my luck with the git.kernel.org 1230050124 M * maharaja hehe 1230050169 M * maharaja anyways ... so 2.6.22.19 means broken xfs and working shed_hard ... 2.6.27.x *should* mean working xfs but no shed_hard 1230050183 M * maharaja well .. bad luck then 1230050277 M * ktwilight maharaja, you'd be fine with xfs as long as there's no sudden power cut or crashes. nobarrier and turning off write cache helps mitigate those issues. 1230050281 J * mrfree ~mrfree@host1-89-static.40-88-b.business.telecomitalia.it 1230050729 M * pmjdebruijn maharaja: we've had some issues with xfs and 2.6.22.19 as well 1230051049 A * mnemoc is marrying ext4 1230051083 M * Bertl how is ext4 for you? 1230051120 M * mnemoc it feels good :) 1230051140 M * mnemoc feels even faster than reiserfs 1230051172 M * mnemoc but I haven't tried to kill it yet 1230051443 M * ktwilight matters worst when you have xfs+raid10 together in .22.19 :| 1230051493 M * maharaja on allmost all of your vservers, we run sdX -> raid1 (md) -> lvm -> xfs 1230051507 M * maharaja i've never really seen any corruption to the day thou 1230051515 M * maharaja maybe pure luck or it went unnoticed 1230051525 M * maharaja doesn't make me feel too good 1230051527 M * mnemoc i have seen quite a few xfs's die on fsck :( 1230051539 M * mnemoc self destructive fs :( 1230051577 M * maharaja so ext3 is still the fs of choice? 1230051654 J * balbir_ ~balbir@122.167.214.101 1230051757 M * pmjdebruijn mnemoc: we did have some corruption, but xfs_repair has always fixed things for us 1230051850 M * maharaja over the last 7 years, afairc, i encountered 1 or 2 corrupt xfs fs 1230051860 M * mnemoc pmjdebruijn: i haven't been blessed 1230051875 M * mnemoc *every* time i have used it, i get my fs blanked 1230051911 M * maharaja i always managed to recover with the help of xfsprogs 1230051985 M * maharaja and because of the nice online resize, it used to be my first choice for data lvs 1230052007 M * maharaja anyways 1230052020 M * maharaja todays experience does not make me sleep well 1230052039 M * pmjdebruijn mnemoc: maybe you're using an ancient version of xfsprogs? 1230052080 M * mnemoc pmjdebruijn: maybe 1230052091 M * pmjdebruijn mnemoc: Debian? 1230052098 M * mnemoc pmjdebruijn: but i don't run any xfs-based system anymore 1230052101 M * mnemoc pmjdebruijn: hand-made 1230052393 J * bonbons ~bonbons@2001:960:7ab:0:2c0:9fff:fe2d:39d 1230052600 M * maharaja hand-made stuff has often bitten me in the ass 1230052622 Q * mrfree Quit: Leaving 1230052929 M * pmjdebruijn maharaja: amen 1230053195 M * maharaja ktwilight: what is the performance impact of using nobarrier and no write cache? 1230053240 M * mnemoc maharaja: i'm probably a sadist undercover :p 1230053268 M * ktwilight maharaja, never measured really 1230053365 M * ktwilight http://rafb.net/p/sWZFa015.html hmm, not so interesting :) 1230053402 M * maharaja ktwilight: this is with said configuration? 1230053408 M * ktwilight yup 1230053434 M * ktwilight not exactly a great test, but a hdparm -Tt shows http://rafb.net/p/lsKXuH32.html 1230053498 M * ktwilight am sure sw raid has improved since .22.19, but i have yet to test it out :/ 1230053771 M * maharaja ok, using -o nobarriers when mounting the xfs LV does not have a big impact 1230053788 M * maharaja http://rafb.net/p/mbCAxL96.html 1230053802 M * maharaja ill have to check for disabling write cache later on 1230053811 M * maharaja (first ill have to find out how to disable it ;) ) 1230053819 M * ktwilight maharaja, xfs' FAQ ;) 1230053905 Q * kir Quit: Leaving. 1230053916 M * maharaja there are so many utils i've never heard of ... blktool for example 1230053951 M * maharaja hehe 1230053956 M * maharaja bbl - gotta fetch a xmas present 1230054179 Q * takeru Quit: takeru 1230055521 Q * balbir_ Ping timeout: 480 seconds 1230055761 J * chi6IT41 ~chigital@tmo-100-232.customers.d1-online.com 1230055866 J * cga ~weechat@94.36.114.123 1230056202 Q * mtgx Quit: Verlassend 1230056490 Q * Pazzo Quit: Ex-Chat 1230057213 J * hparker ~hparker@linux.homershut.net 1230058075 Q * chi6IT41 Ping timeout: 480 seconds 1230058816 Q * jsambrook Remote host closed the connection 1230058839 J * Mojo1978 ~Mojo1978@ip-88-152-63-152.unitymediagroup.de 1230058929 J * jsambrook ~jsambrook@aelfric.plus.com 1230060188 J * AlanCox ~Lee@bl7-147-46.dsl.telepac.pt 1230060464 Q * SlackLnx Ping timeout: 480 seconds 1230060503 J * SlackLnx ~Lee@bl9-68-136.dsl.telepac.pt 1230060679 Q * AlanCox Ping timeout: 480 seconds 1230060966 Q * hparker Quit: Quit 1230062131 Q * gnuk Remote host closed the connection 1230063330 J * hparker ~hparker@linux.homershut.net 1230063871 J * AlanCox ~Lee@bl7-140-18.dsl.telepac.pt 1230064034 Q * SlackLnx Ping timeout: 480 seconds 1230064204 J * SlackLnx ~Lee@bl7-150-72.dsl.telepac.pt 1230064484 Q * AlanCox Ping timeout: 480 seconds 1230064879 Q * SlackLnx Ping timeout: 480 seconds 1230064917 Q * hparker Quit: bbiab 1230065564 J * hparker ~hparker@2001:470:1f0f:32c:212:f0ff:fe0f:6f86 1230067091 Q * bibabu` Quit: Coyote finally caught me 1230067095 J * bibabu bibabu@vserv2.de 1230067860 Q * bibabu Quit: Coyote finally caught me 1230067863 J * bibabu bip@status2k.de 1230068306 J * Aiken ~Aiken@ppp118-208-39-59.lns3.bne1.internode.on.net 1230068328 Q * cga Quit: WeeChat 0.2.6 1230068917 Q * Hollow Server closed connection 1230068928 J * Hollow ~hollow@shiva.xnull.de 1230070230 Q * bonbons Quit: Leaving 1230071840 Q * dna Quit: Verlassend 1230072143 Q * ensc Server closed connection 1230072163 J * ensc ~irc-ensc@77.235.182.26 1230074093 Q * Mojo1978 Remote host closed the connection 1230075781 M * Bertl off to bed now ... have a good night everyone! cya! 1230075789 N * Bertl Bertl_zZ