1191112971 J * duckx ~Duck@tox.dyndns.org
1191113091 Q * onox Quit: zZzZ
1191115258 Q * sladen Ping timeout: 480 seconds
1191115375 J * sladen paul@starsky.19inch.net
1191116368 Q * Punkie Quit: ChatZilla 0.9.78.1 [Firefox 2.0.0.7/2007091417]
1191116957 Q * zLinux Ping timeout: 480 seconds
1191117610 J * zLinux ~zLinux@88.213.17.231
1191119228 J * coderanger_ ~coderange@c-65-96-210-168.hsd1.ma.comcast.net
1191119502 J * igraltist ~user4@kasten-edv.de
1191120484 Q * coderanger_ Quit: coderanger_
1191120755 Q * hparker Remote host closed the connection
1191123650 J * hparker ~hparker@linux.homershut.net
1191127665 Q * Supaplex Read error: Operation timed out
1191129916 Q * hparker Quit: *burp*.. It's broke
1191131911 J * hparker ~hparker@linux.homershut.net
1191131918 Q * zLinux Remote host closed the connection
1191134832 J * _jmcaricand_zzz ~jmcarican@d77-216-200-218.cust.tele2.fr
1191135434 J * Julius ~julius@p57B24586.dip0.t-ipconnect.de
1191135463 M * Julius Guten Morgen
1191135748 Q * _jmcaricand_zzz Quit: KVIrc 3.2.4 Anomalies http://www.kvirc.net/
1191137238 Q * Julius Ping timeout: 480 seconds
1191137369 J * Julius ~julius@p57B24586.dip0.t-ipconnect.de
1191137547 Q * ktwilight_ Ping timeout: 480 seconds
1191138007 J * dna ~dna@255-239-dsl.kielnet.net
1191138588 J * ktwilight ~ktwilight@103.92-66-87.adsl-dyn.isp.belgacom.be
1191139567 Q * mstrobert Ping timeout: 480 seconds
1191140159 J * JonB ~NoSuchUse@kg0-231.kollegiegaarden.dk
1191140160 J * mstrobert ~mstrobert@wkstn.wycliffe.ca
1191141004 Q * arachnist Quit: bbl
1191141029 J * bonbons ~bonbons@ppp-110-228.adsl.restena.lu
1191142281 Q * Julius Ping timeout: 480 seconds
1191142315 J * Julius ~julius@p57B24586.dip0.t-ipconnect.de
1191142351 J * julius_ ~julius@p57B24586.dip0.t-ipconnect.de
1191142804 Q * Julius Ping timeout: 480 seconds
1191144177 Q * dna Quit: Verlassend
1191144308 M * arekm er
1191144544 J * _jmcaricand_zzz ~jmcarican@d77-216-200-218.cust.tele2.fr
1191144834 N * _jmcaricand_zzz jmcaricand
1191145694 M * arekm Bertl_zZ: wake up and back to debugging ;-)
1191145871 M * JonB heh
1191148372 J * ema ~ema@rtfm.galliera.it
1191149730 J * ntrs ~ntrs@79.125.240.136
1191150083 J * ntrs_ ~ntrs@79.125.250.49
1191150499 Q * ntrs Read error: Operation timed out
1191150718 A * bzed gives arekm a sun with lenny - have fun :)
1191151786 N * Bertl_zZ Bertl
1191151791 M * Bertl morning folks!
1191151800 M * JonB hey be
1191151802 M * JonB Bertl: 
1191151816 M * Bertl arekm: don't you know that my brain even works when I'm asleep :)
1191151943 A * bzed hands Bertl's serial brain connection to arekm 
1191151946 M * bzed morning Bertl :)
1191152168 M * arekm damn, EFI doesn't support fetching files over network
1191152243 M * arekm Bertl: going to test something. If much older userspace stuff is able to crash vserver
1191152273 J * arekmx ~arekm@chello089076024040.chello.pl
1191152294 Q * arekm Quit: leaving
1191152401 M * bzed arekmx: efi on an apple? you can use a firewire cable and target mode on them
1191152412 M * bzed if that helps
1191152432 M * arekmx bzed: on intel server
1191152446 M * arekmx bzed: which is 300km away from my location. wanted to do firmware upgrade
1191152459 M * arekmx unfortunately they didn't provide DOS based upgrade. Only EFI based ;/
1191152518 M * bzed eww
1191152558 M * Bertl arekmx: a few ideas on my side:
1191152575 M * bzed me still prefers sun's OBP or even better the stuff they have on the alpha machines
1191152593 M * Bertl - it looks like an SMP/SMT issue, but we haven't proven that yet, a test with smp disabled would be interesting
1191152705 M * Bertl - we see that the use count goes down (in certain cases even below zero, which I do not really understand) which would suggest that we have a serialization issue
1191152883 M * Bertl - OTOH, we always hit the SHUTDOWN case, and never ever did hit the BUG_ON(atomic_read(&vxi->vx_usecnt)) one, which suggests, that except for the shutdown the counts would be correct (maybe reorder the BUG_ON()s for a test?)
1191152963 M * daniel_hozac hmm, i thought we were hitting the usecnt?
1191152990 M * Bertl hmm, no, line 193 is the shutdown here
1191153031 M * daniel_hozac ah, i was looking at an older tree, nevermind.
1191153048 M * Bertl but it would make sense to hit the shutdown, when the usecount reaches zero before the task count
1191153076 M * daniel_hozac right.
1191153109 M * Bertl thus, IMHO, it would be interesting to see if we have a case where task count is zero, but shutdown isn't set
1191153220 M * Bertl so I think it would be better to test for the counts first
1191153231 M * Bertl and for the shutdown later ... I'll prepare a patch for that
1191153441 M * arekmx recompiling kernel without smp, unfortunately nosmp cmdline option causes mptsas go mad so can't test that way
1191153455 N * esa eSa|
1191153503 N * julius_ Julius
1191153507 M * arekmx Userspace doesn't matter. Tested with 2 year old glibc/gcc/binutils and it still crashed
1191153529 M * arekmx (I mean test compilation in such environment)
1191153536 M * Bertl arekmx: interesting (the mptsas part)
1191153688 M * Bertl arekmx: do you see any chance to run a simple test without the mptsas stuff? e.g. via initramfs or so?
1191153712 M * arekmx I'm almost sure that I have second machine which also has the same problem (2 x single core 3.6GHz Xeon with HT) because I remember having oopses with it with newer kernel and I'm back to 2.6.17 there. Don't have oopses unfortunately. megaraid_mbox SCSI there
1191153742 M * arekmx Bertl: but 2.1.1 works fine with mptsas on the same kernel where 2.0 or 2.2 doesn't work also with mptsas
1191153775 M * Bertl yes, but that might be related to the way we handle tasklets and softirqs
1191153792 M * Bertl (which is one of the changes we found between the versions you tested)
1191153816 J * pmenier ~pmenier@ACaen-152-1-13-33.w83-115.abo.wanadoo.fr
1191153840 M * Bertl I think we could even re-introduce the enter/leave part for softirqs on x86_64 for a test
1191153893 A * arekmx doing single test on 2x xeon machine (it's used in production unfortunately so only 1 test :)
1191153915 Q * ema Quit: leaving
1191153957 M * bzed I could do any tests on a dualcore AMD box or on a 4 CPU sparc machine (although sparc has bad futex bugs at the moment), but if I can help you, le tme know
1191153991 J * Punkie ~Punkie@home.pekelny.net
1191153992 M * Bertl bzed: yes, would be interesting, as we do not see the issue here yet
1191153997 M * Bertl wb Punkie!
1191154012 M * Punkie hi Bertl :)
1191154053 M * arekmx bzed: fetch 2.6.18.5 kernel + patch-2.6.18.5-vs2.0.3-rc1.diff, boot, chcontext --xid 666 -- bash and do make -j6 some kernel compilation there
1191154067 M * Bertl bzed: nah, let's use 2.3.0.24 for that
1191154084 M * Bertl bzed: maybe even with the following patches:
1191154110 M * Bertl http://vserver.13thfloor.at/Experimental/delta-mminit-fix01.diff
1191154179 M * bzed uh, no chance on sparc with 2.6.18, that one doesn;t boot at all on ultrasparc III machines
1191154181 M * Bertl http://vserver.13thfloor.at/Experimental/delta-free-fix01.diff
1191154210 M * Bertl bzed: go for 2.6.22.x it shows the very same issue on arekmx's machine
1191154248 M * bzed Bertl: anything to patch before trying it?
1191154248 A * arekmx didn't test free-fix01.diff
1191154257 M * Bertl bzed: (preferable 2.6.22.6, because the latest patch applies cleanly there)
1191154272 M * Bertl arekmx: yep, doesn't change anything, just a different error reporting sequence
1191154379 J * julius_ ~julius@p57B25DCF.dip.t-dialin.net
1191154460 M * bzed I can build a 2.6.22.9, that one should have a part of the futex bugs fixed, so I guess i could build somethign on the US III machine using it
1191154497 M * bzed it'll take some time though as I need to build on an US II machine
1191154510 M * arekmx Bertl: bad news. reproduced on second machine (2 x xeon) - http://pastebin.com/m4ea8ad63
1191154539 M * arekmx goind to test on third one - 1 x dual core xeon
1191154543 M * daniel_hozac what OS are you running on your guests?
1191154557 M * Bertl arekmx: wait, that's on line 194 now
1191154570 M * arekmx daniel_hozac: pld linux ac (ooold userspace) and th (very, very fresh, latest releases and so on)
1191154572 M * Bertl arekmx: please check what BUG_ON() is on your code line
1191154587 M * arekmx Bertl: 2x xeon has different kernel, 2.6.21, non vanilla
1191154625 M * daniel_hozac arekmx: just to get some sort of reproducibility, could you try with e.g. a Debian guest?
1191154636 M * bzed Bertl: I could just try to build a kernel on the US III now, it is running the normal debian vserver kernel with some patches
1191154653 M * arekmx daniel_hozac: if you have tarball with debian guest installed then yes
1191154667 M * daniel_hozac you can't build one yourself?
1191154685 M * arekmx will it build without dpkg?
1191154690 M * daniel_hozac yes.
1191154692 M * bzed arekmx: are you running an rpm based distro?
1191154696 M * arekmx bzed: yes
1191154705 M * bzed arekmx: I'll lik you to deboostrap...
1191154744 M * bzed http://bzed.de/_media/blog/debian/debootstrap-0.3.3.2-2.noarch.rpm
1191154751 M * bzed that should work
1191154755 M * bzed never tested it though
1191154756 M * daniel_hozac you don't have to install it.
1191154762 M * daniel_hozac the utils will download it for you.
1191154776 M * bzed ah even better
1191154777 M * arekmx oh, syslog logged even more than I saw on ipmi console on 2x xeon - http://pastebin.com/m41881c69
1191154787 Q * Julius Ping timeout: 480 seconds
1191154878 M * arekmx daniel_hozac: so -m debbootstrap?
1191154883 M * Bertl sec
1191154897 M * daniel_hozac yes.
1191154909 M * daniel_hozac (only spelled correctly)
1191154911 N * _Hunger Hunger
1191154912 M * daniel_hozac and -d etch or so.
1191154914 M * Bertl vserver etch32 build -m debootstrap --context 10104 --hostname etch32.debian.org --interface eth1:10.1.4.32/24 -- -d etch -m http://ftp.debian.org/debian -- --arch i386
1191154928 M * Bertl adjust the ip if necessary
1191154943 M * Bertl or for 64bit
1191154952 M * Bertl vserver etch64 build -m debootstrap --context 10114 --hostname etch64.debian.org --interface eth1:10.1.4.64/24 -- -d etch -m http://mirrors.kernel.org/debian -- --arch amd64
1191154953 A * arekmx installs wget, binutils... these needed by bootstrap, geez :)
1191154990 M * daniel_hozac all of which should be required by your util-vserver-build package...
1191155120 M * arekmx so I fixed deps in our pld cvs, too
1191155198 M * arekmx http://pastebin.com/m64a3613c ?
1191155289 M * bzed Bertl: there's no etch64.debian.org
1191155336 M * bzed debian.netcologne.de is a fast mirror
1191155361 M * arekmx Bertl: your cmdline works
1191155368 M * arekmx mine doesn't :>
1191155390 J * arachnist arachnist@088156185052.who.vectranet.pl
1191155412 Q * Aiken Quit: Leaving
1191155568 M * Bertl bzed: that is your _local_ host name :)
1191155604 M * bzed :P
1191155668 M * bzed Bertl: my local hostname is kinda different, there's a lot of debian stuff at my university ;)
1191155693 M * Bertl yeah, it's fine to name it differently :)
1191155782 M * arekmx how glibc headers package is named in debian?
1191155854 M * bzed libc6-dev
1191155985 M * Bertl daniel_hozac: the cannot create guest on newly mounted partition should be resolved in 0.30.213+, yes?
1191156008 M * Bertl daniel_hozac: (mailing list post from ADNET Ghislain)
1191156014 J * yarihm ~yarihm@84-74-20-183.dclient.hispeed.ch
1191156129 M * matti Bertl: :)
1191156237 M * Bertl matti: :)
1191156243 M * arekmx Sep 30 14:43:16 carme kernel: [  183.641869] vxW: xid=666 messing with the devpts.
1191156246 M * arekmx Sep 30 14:43:33 carme kernel: [  198.391834] vxW: xid=666 messing with the devpts.
1191156256 M * arekmx doing test compilation under etch now
1191156272 M * Bertl okay
1191156279 Q * JonB Quit: Leaving
1191156313 M * arekmx http://pastebin.com/m869a32f crashed
1191156340 M * arekmx hm, it likes to crash when I press ctrl+c on make -j6 compilation heh
1191156424 M * Bertl okay, that was now a debian guest, right? and what is on line 140?
1191156438 M * arekmx that was like this http://pastebin.com/m4cfc5e2b
1191156454 M * arekmx checking 140
1191156469 M * arekmx Bertl: and yes, it was etch 64bit guest
1191156561 M * arekmx void free_vx_info(struct vx_info *vxi)
1191156561 M * arekmx { /* context shutdown is mandatory */ BUG_ON(!vx_info_state(vxi, VXS_SHUTDOWN));
1191156564 M * arekmx BUG_ON(atomic_read(&vxi->vx_usecnt));
1191156571 M * arekmx 140 is         BUG_ON(!vx_info_state(vxi, VXS_SHUTDOWN));
1191156590 M * Bertl so you didn't add the reordering patch yet, right?
1191156594 M * arekmx no
1191156617 M * Bertl okay, np, could you try a 32bit kernel on that machine too?
1191156678 M * arekmx no, entire userspace is 64bit, that won't work
1191156692 M * Bertl okay, np, what about the 32bit debian guest?
1191156714 M * arekmx I have pld 32bit guest here, could try that
1191156728 M * arekmx we know that pld vs etch doesn't matter
1191156761 M * Bertl yeah, that's fine, just the debian guest is easier to recreate for us
1191156818 M * arekmx do you have >= 4 way machine anyway?
1191156844 M * arekmx I'm currently trying to reproduce the problem on third - 2 way machine with no luck
1191156844 M * Bertl no, unfortunately not :)
1191156871 M * matti arekmx: Czesc :)
1191156873 M * arekmx 2x dual core should be fine, too since 2 x single core with HT is enough to reproduce
1191156891 M * arekmx matti: hej.
1191156957 M * arekmx added reordering patch and trying 32bit guest now
1191157272 M * matti What is the "re-ordering" patch?
1191157310 M * Bertl http://vserver.13thfloor.at/Experimental/delta-free-fix01.diff
1191157327 A * arachnist evil
1191157347 A * arachnist has freebsd7/amd64 installed on his box
1191157368 M * matti BSD? ;p Yuck.
1191157369 M * matti ;p
1191157375 M * daniel_hozac Bertl: should be fixed since 12/31/05 11:58:18 according to svn.
1191157405 M * Bertl okay, which is definitely true for 0.30.213 I guess, probably even for 212, right?
1191157444 M * Bertl do you want to reply to that e-mail or shall I do that?
1191157479 M * daniel_hozac i'll reply once i find the version it's associated with.
1191157482 J * lilalinux ~plasma@dslb-084-058-208-116.pools.arcor-ip.net
1191157489 M * arekmx off our debugging topic: [  298.158786] vxW: xid=666 did hit the barrier.
1191157489 M * Bertl daniel_hozac: excellent! tx!
1191157491 M * arekmx what's that?
1191157513 M * Bertl arekmx: that means that something inside the guest tried to go outside the barrier flags
1191157543 M * arekmx that happens when I try to do chcontext --xid 666 -- chroot /vservers/carme-pld/ su - arekm
1191157544 M * Bertl usually the barrier is on /vservers when the guests are located in e.g. /vservers/guestA ...
1191157577 M * Bertl yep, do it like this: (cd /vservers/carme-pld/; chcontext --xid 666 -- chroot . ...)
1191157605 Q * arachnist Quit: brb/bbl
1191157645 M * arekmx Bertl: 32bit guest, oopsed http://pastebin.com/m1d4db948
1191157655 M * arekmx and it oopsed again when I hit ctrl+c
1191157673 M * arekmx ha! on third machine (1 x dual core) it also happened on ctrl+c
1191157688 M * Bertl okay, we didn't get the oops on UP kernel, right?
1191157722 M * arekmx phobos machine, 1 x dual core: http://pastebin.com/m6112ab92
1191157730 M * arekmx didn't test UP, testing now
1191157765 M * arekmx Bertl: http://pastebin.com/m1d4db948 this is with reordering patch applied
1191157780 M * arekmx and 140 line is...
1191157809 M * daniel_hozac what's the exact reproducer?
1191157820 M * Bertl kernel compiling
1191157823 M * daniel_hozac because i build kernels all the time on my dual core x86_64 box.
1191157829 M * arekmx daniel_hozac: make -j6, wait until it starts compiling *.c files, press ctrl+c then
1191157831 J * arachnist arachnist@088156184167.who.vectranet.pl
1191157876 A * matti have only smp machines... no multicore ones :<
1191157903 M * Bertl works fine here on my dualcore 64bit machine ...
1191157924 Q * lilalinux Remote host closed the connection
1191157928 M * Bertl let's see what the princeton machine says ...
1191157938 A * arachnist only has his own little athlon64 x2 4200}
1191157946 M * arachnist s/}/+/
1191157970 M * arekmx line 140 is         BUG_ON(atomic_read(&vxi->vx_tasks));
1191157976 Q * arachnist 
1191157984 M * Bertl arekmx: as expected, thanks!
1191157996 M * fb hello
1191158003 M * Bertl hello fb!
1191158070 A * arekmx sees the issue on 3 different machines now, there bug is real ;)
1191158083 M * daniel_hozac except, you're the only one seeing it.
1191158087 M * fb i have dual-core 64-bit cpu on my server, with sas too
1191158106 M * daniel_hozac have you tried building your kernel on another OS?
1191158108 M * fb but, unfortunately ;), it works quite stable
1191158128 M * fb maybe because there's only one physical cpu
1191158140 M * fb (and 4 reported by os)
1191158159 M * arekmx daniel_hozac: no, hmm, could test on that etch thing
1191158459 M * arekmx anyone of you using gcc 4.2?
1191158529 M * bzed sure
1191158551 M * arekmx bzed: cat /proc/version on vserver machine please then
1191158558 J * arachnist arachnist@088156185052.who.vectranet.pl
1191158566 M * bzed Linux version 2.6.22.5-vs2.2.0.3 (bzed@hal) (gcc version 4.2.1 20070712 (prerelease) (Debian 4.2-20070712-1)) #1 SMP PREEMPT Mon Aug 27 21:43:55 CEST 2007
1191158625 M * arekmx gcc (GCC) 4.2.1 20070828 (release) here, maybe something important was broken in middle, testing 4.2.2rc2 then
1191158656 M * arekmx and 4.1.2 from etch, too
1191159077 M * arekmx bad news - http://pastebin.com/m256cb195 oopsed even after using gcc from etch for its compilation
1191159103 M * arekmx I needed three make -j6 + ctrl+c to reproduce but it oopsed
1191159143 M * Bertl okay, I have a theory and an idea how to verify that
1191159157 M * Bertl give me a few minutes to prepare a patch
1191159182 M * Bertl arekmx: ah, do you think you could disable networking for a test?
1191159197 M * Bertl arekmx: i.e. compile the kernel without inet
1191159201 M * arekmx I could
1191159297 M * Bertl another thing, do you have the oopses you pasted to the pastebin stored somewhere as text? if so, could you make a tar.bz2 for me to download?
1191159326 M * arekmx no, but there is download link on pastebin which gets you text
1191159351 M * Bertl okay, perfect, tx
1191159471 M * arekmx daniel_hozac: do you login over console or network to that dualcore machine?
1191159485 M * Bertl matti: if you want to help here, please go through the irc logs and extract all the pastebin urls, then get the actual text into a file named like this: m256cb195.txt
1191159487 M * daniel_hozac network.
1191159514 M * matti Bertl: What period of time?
1191159525 M * Bertl last three days I'd say
1191159529 M * matti Aye.
1191159537 M * Bertl we are only interested in the pastings from arek*
1191159547 M * matti Aye.
1191159554 M * Bertl thanks in advance!
1191159580 M * arekmx matti: small shell script should be able to do that automaticly btw
1191159803 M * matti arekmx: I know.
1191159835 M * matti http://paste.romke.net/gJ1K - arekm' URLs.
1191159837 M * arekmx gcc 4.2.2rc2 compiled kernel also oopsed so we can rm -rf gcc from equation imo
1191159867 M * arekmx disabling network now
1191159899 M * daniel_hozac did you try 4.1?
1191159932 M * arekmx daniel_hozac: yes, http://pastebin.com/m256cb195 - etch gcc was used to compile this kernel
1191159992 M * arekmx and rest of toolchain (compilation was done in etch guest)
1191160051 M * arekmx daniel_hozac: maybe you could try to use my .config?
1191160175 M * arekmx daniel_hozac: http://ep09.pld-linux.org/~arekm/.config
1191160257 M * arekmx bzed: you too :)
1191160398 M * arekmx q: do you have CONFIG_TASKSTATS set?
1191160418 M * arekmx or CONFIG_TASK_*
1191160458 M * arekmx but hm, that doesn't matter, it'a new thing.
1191160762 M * arekmx Bertl: CONFIG_INET disabled (but ethX drivers not disabled) and it ooopsed
1191160776 M * Bertl okay, trace?
1191160801 M * arekmx http://pastebin.com/m6b1a1d4d
1191160852 M * matti Bertl: http://deadbeef.info/tmp/m256cb195.txt
1191160962 M * Bertl hmm, okay, could you make that one trace per file?
1191160972 M * matti Yes.
1191160989 M * Bertl okay, please try to name them after the ending of the pastebin urö
1191160991 M * Bertl *url
1191161021 M * matti Like: m256cb195.txt-abcdefgh
1191161021 M * matti ?
1191161041 M * Bertl more like http://pastebin.com/m256cb195 -> m256cb195.txt
1191161048 M * matti Oh.
1191161050 M * matti Aye.
1191161053 M * Bertl thanks
1191161058 M * matti :)
1191161060 M * matti No worries.
1191161072 M * Bertl and please put them all into a tar.bz2
1191161283 M * arekmx Bertl: new ideas?
1191161318 M * Bertl I'm working on it ...
1191161669 M * matti Bertl: http://deadbeef.info/tmp/vserver/
1191161674 M * matti Oh.
1191161676 M * matti tar.
1191161678 M * matti Forgot :)
1191161684 M * Bertl it's fine
1191161735 M * matti http://deadbeef.info/tmp/vserver/arekm.tar.bz2
1191161743 M * Bertl excellent! tx!
1191161756 M * matti Sorry for delay :)
1191161774 M * bzed arekmx: I'm still waiting for the first build :)
1191161778 M * matti arekmx: In this case it was perl one-liner :)
1191161781 M * matti arekmx: I love perl ;p
1191161809 M * arekmx one more oops http://pastebin.com/m46510cfd 
1191161814 A * arekmx hates perl ;P
1191161822 M * matti Heheh.
1191161851 M * Bertl arekmx: http://vserver.13thfloor.at/Experimental/delta-doubleref-debug01.diff
1191161873 M * Bertl okay, this one should show us if we have a ref counting issue or a flaw in the reference logic
1191161894 M * arekmx Bertl: does it need anything like sysctl setting?
1191161912 M * Bertl explanation: we currently assume that at least one use reference is taken as long as we have tasks referencing the vx_info
1191161935 M * Bertl this patch will take an additional usecnt reference for each task reference
1191161970 M * Bertl thus, if the usecnt ref counting is balanced, it should prevent the issue, but if it is not, we will get the same oops it just will take a little longer
1191161983 M * Bertl arekmx: no, no settings required
1191162188 J * ntrs ~ntrs@79.125.228.130
1191162484 M * arekmx Bertl: oopsed at first ctrl+c
1191162513 M * arekmx http://pastebin.com/m12b4dec2
1191162518 Q * ntrs_ Ping timeout: 480 seconds
1191162560 Q * Blissex Remote host closed the connection
1191162562 M * Bertl arekmx: and line 193 contains?
1191162729 M * arekmx         /* context shutdown is mandatory */
1191162730 M * arekmx         BUG_ON(!vx_info_state(vxi, VXS_SHUTDOWN));
1191162746 M * arekmx but this kernel doesn't have reorder thing
1191162751 M * Bertl hmm ... bad
1191162890 M * arekmx should I retest with reorder applied?
1191162926 M * Bertl can't hurt, but I gues sit will also hit the usecnt zero case
1191163153 M * Bertl this actually means that we get more put()s than get()s somehow
1191163265 M * Bertl and the fact that __mmdrop shows up that often, would suggest that this happens in the mm management
1191163601 M * arekmx http://pastebin.com/mf79abd3
1191163623 M * arekmx and 194 is...
1191163804 M * arekmx         BUG_ON(atomic_read(&vxi->vx_tasks));
1191163816 M * Bertl as I suspected
1191163871 M * Bertl now something in your setup or config has to be special, otherwise we would hit that almost permanently
1191163887 M * arekmx test my config then, please
1191163903 M * Bertl will do so shortly, no doubt
1191163961 M * arekmx three different hardware setups so something in config is more probable (well, all hw setups are intel) 
1191163991 M * Bertl do you have anything in common between those three?
1191164009 M * Bertl something like a special driver or special hardware?
1191164073 M * arekmx each has different controller (mtp sas, 3ware and lsi logic scsi raid); one has 1x dual core, other has 2x quad core and the third has 2x single core - all xeons
1191164092 M * Bertl what about the remote console?
1191164162 M * arekmx it's IPMI, 2 machines built in into mainboard, third as addon
1191164209 M * Bertl and you are working via console or via network logon by default?
1191164258 M * arekmx watching console over ipmi but doing everything else via network (like sshing + chcontext + test compilation)
1191164283 M * Bertl okay, could you try to enter a guest via ssh directly?
1191164294 M * arekmx note that ipmi doesn't require any operating system support, it's done by hardware
1191164305 M * Bertl i.e. run sshd inside the guest and try to trigger it from there?
1191164357 M * arekmx I was sshing directly but we tried to get minimal case, stopped guests and later switched to chcontext
1191164379 M * Bertl okay, so we had a panic with ssh directly to guest too, yes?
1191164407 M * arekmx yes (but I was still watching ipmi console then)
1191164428 M * Bertl well, I don't think that watching the console actually changes that much
1191164812 M * Bertl I think we should try to create an artificial test case with minimal noise
1191165053 M * matti arekmx: IPMI or iLOM?
1191165145 M * arekmx matti: ipmi
1191166564 M * arekmx better try that my config ;)
1191167047 J * ensc ~irc-ensc@p54B4DE3B.dip.t-dialin.net
1191167363 Q * arachnist Quit: bbl
1191167405 M * Bertl arekmx: so what was the chcontext line triggering it for you?
1191167430 M * arekmx chcontext --xid 666 -- chroot /vservers/carme-pld su - arekm then cd linux-source; make -j6
1191167556 J * arachnist arachnist@088156184167.who.vectranet.pl
1191167690 M * arekmx and ctrl+c
1191167715 M * Bertl yeah, building with your config atm (and -j6) and interrupting the build every 30 seconds or so :)
1191167718 M * daniel_hozac and if you don't ctrl+c, everything works fine?
1191167745 M * arekmx It still crashes just I have to wait for some time
1191167763 M * Bertl what I assume so far that happens is this:
1191167781 M * Bertl when a process exits, it is cleaned up by the rcu mechanisms
1191167800 M * Bertl those will pile up some reaping work when all cpus are under load
1191167833 M * Bertl thus, the task count is still high, while (for a yet unknown reason) the usecnt decrements 
1191167864 M * Bertl note that whatever causes this would probably be able to trigger it with two processes only
1191167902 M * Bertl so it would be really great if we could get some artifical test (probably memory related) to trigger this
1191167916 J * uiyr ~debian@ppp83-237-58-111.pppoe.mtu-net.ru
1191167929 M * Bertl welcome uiyr!
1191167947 M * uiyr i am new user debian
1191167962 M * Bertl nothing wrong with that! :)
1191168012 M * matti Heheh.
1191168073 M * uiyr P1-200MHz-MMX - ;)
1191168132 M * arekmx Bertl: so your machine is what exactly?
1191168160 M * Bertl I tested on a dual core (intel) and a dual cpu celeron D system
1191168252 M * Bertl arekmx: here is a kernel I compiled with your config, could you give that a spin?
1191168256 M * Bertl http://vserver.13thfloor.at/Stuff/AREKM/
1191168322 Q * uiyr Quit: õÈÏÖÕ
1191168419 M * daniel_hozac did doubleref-debug01 fix the problem?
1191168524 M * arekmx Bertl: need CONFIG_BLK_DEV_3W_XXXX_RAID=y and CONFIG_SCSI_3W_9XXX=y
1191168641 M * Bertl arekmx: that should be from the config you uploaded
1191168693 M * Bertl da	nope
1191168701 M * Bertl daniel_hozac: nope
1191168717 M * arekmx Bertl: yes but now I test on 1 x dual core machine :)
1191168729 M * arekmx Bertl: I left carme (2x quad) running, people wanted to use it
1191168759 M * Bertl okay, please upload a config for that one
1191168763 M * arekmx done
1191168858 M * Bertl same location?
1191168876 M * arekmx yes
1191168901 M * arekmx that one has CONFIG_SCSI_3W_9XXX=y in it
1191169103 M * arekmx Bertl: need bzImage only, don't need modules
1191169120 M * Bertl okay
1191169638 M * Bertl daniel_hozac: hmm, could you have a look at the posix timer stuff, I'm not sure it is safe to do the enter/leave there as we do it right now?
1191169667 J * dna ~dna@p54BCC928.dip.t-dialin.net
1191169682 M * Bertl arekmx: all machines have ntpd running on the host, yes?
1191169725 M * arekmx Bertl: yes
1191169745 M * Bertl anything else running on all machines?
1191169790 M * arekmx watchdog
1191169798 M * Bertl software?
1191169853 M * arekmx well, actually test kernel doesn't have watchdog compiled in so it doesn't matter
1191169855 M * Bertl filesystem on those machines is?
1191169858 M * arekmx xfs
1191169880 M * Bertl could we take that out of the equation somehow?
1191169904 M * arekmx you mean xfs?
1191169908 M * Bertl yep
1191169944 M * arekmx problematic
1191169955 M * Bertl do you have a swap partition?
1191169999 M * arekmx yes
1191170008 M * Bertl reasonably large?
1191170020 M * arekmx yes, could use it for testing
1191170028 M * arekmx so which fs you want?
1191170040 M * Bertl ext2 or ext3 should be fine, using ext3 here
1191170082 M * arekmx irqbalance also run here
1191170087 M * arekmx in daemon mode
1191170105 M * arekmx auditd also run
1191170123 M * Bertl okay, maybe try to disable those in a separate test
1191170149 A * Bertl is still compiling your kernel, btw :)
1191170535 M * arekmx hmmm
1191170631 M * arekmx tadam! when ntpd is not running then kernel doesn't oops
1191170638 M * Bertl okay, uploaded the bzImage
1191170649 M * Bertl aha, that is quite interesting ...
1191170656 M * arekmx Bertl: try ntpd on your machine
1191170659 M * Bertl let me try if I can trigger it with ntpd
1191170798 M * Bertl what ntpd version/flavor?
1191170815 M * arekmx 4.2.4p3
1191170855 M * arekmx http://pastebin.com/m4ccddd01 - such config
1191170875 M * arekmx run by ntpd -c config_file.conf
1191170950 M * Bertl will the servers answer to me too?
1191170963 M * arekmx yes, these are public ntp servers
1191171093 M * Bertl doesn't seem to cause anything bad here atm ...
1191171131 M * Bertl re-compiling your kernel with interruptions now :)
1191171141 A * arekmx doing more testing then
1191171304 M * Bertl daniel_hozac: would you consider it correct that at no time we should have vx_usecnt < vx_tasks ?
1191171318 J * adrien-modulis ~adrien@216.252.77.86
1191171325 M * Bertl wb adrien-modulis!
1191171451 M * arekmx again, didn't oops (tries ~10 times ctrl+c) then ntpd run and oopsed at 3 time
1191171496 M * Bertl okay, so ntpd seems to be some kind of catalyst in your case
1191171527 M * michal_ it might be interesting to know what kind of syscall() is beeing called just before...
1191171542 M * michal_ strace -Ff that ntp or sth
1191171593 M * arekmx Bertl: could I give you my ntpd binary?
1191171618 M * Bertl probably won't work because of library dependancies
1191171665 M * arekmx http://ep09.pld-linux.org/~arekm/ntpd
1191171684 M * arekmx can give missing libs, too
1191171689 M * arekmx just tell me which are missing
1191171734 M * Bertl make me a chroot to run it in, then I can start it
1191171762 M * arekmx ok
1191171802 Q * adrien-modulis Ping timeout: 480 seconds
1191171848 J * JonB ~NoSuchUse@192.38.8.25
1191172338 M * daniel_hozac Bertl: i think that's right.
1191172463 A * matti pokes michal_ 
1191172494 M * arekmx Bertl: http://ep09.pld-linux.org/~arekm/xxx.tar.bz2 (it contains much more than ntpd btw)
1191172518 A * michal_ hugs matti
1191172739 M * daniel_hozac Bertl: the POSIX timers look okay to me, makes sense to enter the context, IMHO.
1191172754 J * dna_ ~dna@p54BCEBA8.dip.t-dialin.net
1191172775 M * Bertl daniel_hozac: yeah, but can we guarantee that nothing will be using 'current->vx_info' at this time?
1191172785 M * daniel_hozac hmm?
1191172803 M * daniel_hozac like, that something isn't changing the context?
1191173061 M * arekmx Bertl: so it doesn't crash with ntpd run from chroot?
1191173109 M * arekmx michal_: probably clock_settime() syscall
1191173117 Q * dna Ping timeout: 480 seconds
1191173761 M * arekmx Bertl: you should see the crash. I now have ntpd which doesn't trigger this and probably know what's the difference, testing
1191173784 M * Bertl sounds good
1191173802 M * arekmx and I was wrong. also triggered the problem with "other" ntpd :/
1191173832 M * arekmx so waiting for your results
1191174265 Q * ntrs Read error: Connection reset by peer
1191174426 M * arekmx did  about 30 make -j6 + ctrl c and can't trigger the problem without ntpd
1191174553 J * bruno_ ~bonbons@2001:960:7ab:0:20b:5dff:fec7:6b33
1191174879 Q * bonbons Ping timeout: 480 seconds
1191175298 M * maddoc # vserver otho start
1191175298 M * maddoc vshelper.init: can not determine xid of vserver 'otho'; returned value was ''
1191175306 M * maddoc Any ideas?
1191175324 M * maddoc 2.6.22.6-grsec2.1.11-vs2.2.0.3
1191175350 M * daniel_hozac your guest isn't starting any services.
1191175426 M * daniel_hozac make it start a service, change the initstyle, or make the contexts persistent.
1191175523 M * maddoc I ran echo 101 > /etc/vserver/otho/context, if that's what you mean with persistent context.
1191175565 M * daniel_hozac no, that's a static context.
1191175575 M * maddoc Okay.
1191175577 M * daniel_hozac a persistent context is one that still exists after the last process exits.
1191175699 M * maddoc Hm, k. I created the vserver with " vserver otho build --force -n otho --hostname otho.srv.phelia.net --interface eth0:10.0.8.1/16 -m debootstrap -- -d etch -m http://ftp.port80.se/debian/" and I've done that on another host, and then it works just fine. 
1191175717 M * Bertl why --force?
1191175733 Q * pmenier Quit: KVIrc 3.2.0 'Realia'
1191175742 M * maddoc Got the same problem before, and tried just to reinstall the vserver.
1191175768 M * maddoc It complains about not being able to mount proc, but that's because of the grsec configuration.
1191175783 M * maddoc I got all set to yes except chroot_caps, can it have something to do with that?
1191175795 M * daniel_hozac do you get a complete guest?
1191175899 M * maddoc I dunno, it just stops after complaining about not being able to mount proc, but I think it did that on the other host too.
1191175917 M * daniel_hozac the debootstrap?
1191175924 M * maddoc Yeah.
1191175926 Q * JonB Quit: Leaving
1191175934 M * daniel_hozac you don't get the whole "retrieving package...." things?
1191175941 M * maddoc Yes, that's before.
1191175982 M * daniel_hozac so if you look around in /vservers/<guest>, does it look complete?
1191176038 M * maddoc I guess so, but it complained about /etc/default/rcS missing, so I just tried to fix it the ugly way and copy it from the host.
1191176086 M * arekmx daniel_hozac: could you test ntpd thing, too? http://ep09.pld-linux.org/~arekm/xxx.tar.bz2
1191176106 M * daniel_hozac i already run ntpd on my hosts.
1191176158 M * arekmx daniel_hozac: ntpd that uses clock_settime()?
1191176178 M * daniel_hozac ntp-4.2.2p1-5.el5
1191176196 M * arekmx it's configure time detected thing so version doesn't say anything
1191176222 J * meandtheshell ~markus@85.127.119.109
1191176310 M * Bertl arekmx: I've tried with your ntpd for some while now, no crashes so far
1191176352 M * arekmx Bertl: :-(
1191176397 M * daniel_hozac maddoc: have you enabled any service in the guest? i.e. when you start it, do you see it starting something?
1191176417 M * maddoc No, I just get that error.
1191176458 M * maddoc I've never manually enabled any service before.
1191176475 M * daniel_hozac which error?
1191176487 M * daniel_hozac oh, the vshelper.init one.
1191176493 M * maddoc Hang on a sec and I'll give you a complete log.
1191176500 M * daniel_hozac you'll have to chroot into it and do it.
1191176520 M * daniel_hozac (or just modify the symlinks manually)
1191176546 M * maddoc Do what? Sorry for my incompetence. >_<
1191176577 J * Piet ~piet@tor.noreply.org
1191176649 M * daniel_hozac enable a service.
1191176745 M * arekmx Bertl: maybe stop ntpd, set wrong time like 1/2 mins earlier (date -s '2007/09/30 19:22') and start ntpd again?
1191176769 M * Bertl let's see ...
1191176855 M * arekmx Bertl: few seconds better so ntpd has a chance to synchronize
1191177076 M * Bertl still looks fine here, what is your theory?
1191177172 M * arekmx Bertl: wondering if time sync done by ntpd changes something
1191177198 M * arekmx and ntpd doesn't do that sync immediately. it does that slowly etc etc
1191177313 M * maddoc daniel_hozac: http://ssh.ostruktur.com/~maddoc/vserver.txt
1191177378 J * virtuoso ~s0t0na@ppp91-122-94-47.pppoe.avangard-dsl.ru
1191177449 M * arekmx Bertl: http://pastebin.com/m1ca01fa6
1191177470 M * arekmx Bertl: your ntpd is from debian?
1191177580 J * JonB ~NoSuchUse@192.38.8.25
1191177592 M * Bertl no, my ntpd is from mandriva, but I'm running yours right now
1191177632 M * arekmx Bertl: watch logs when it synchronizes time and then try ctrl+c etc
1191177787 Q * virtuoso_ Ping timeout: 480 seconds
1191177887 Q * julius_ Ping timeout: 480 seconds
1191177887 Q * FireEgl Ping timeout: 480 seconds
1191178162 J * cedric ~cedric@rny93-2-82-66-66-30.fbx.proxad.net
1191178877 M * daniel_hozac maddoc: you may want to make sure you can get a clean guest first...
1191179046 J * Alissa ~Alissa@ppp-133-135.25-151.libero.it
1191179058 M * maddoc Okay.
1191179065 M * Alissa Hi all
1191179067 M * Alissa ^^
1191179191 M * daniel_hozac hello
1191179291 M * Alissa hi daniel_hozac
1191179590 J * ema ~ema@rtfm.galliera.it
1191179624 J * Aly_awai7a8i2 ~Alissa@ppp-192-139.25-151.libero.it
1191179781 Q * Piet Quit: Piet
1191179906 Q * Alissa Ping timeout: 480 seconds
1191179912 N * Aly_awai7a8i2 Alissa
1191180061 M * arekmx Bertl: if you find something then mail me at arekm/pld-linux.org or irc freenode: arekm
1191180087 M * arekmx disappearing now, seems that we are still nowhere with debugging
1191180092 Q * arekmx Quit: leaving
1191180281 M * Bertl welcome Alissa!
1191180318 J * CWC ~CWC@89-215-37-177.2073053861.ddns-lan.pl.ekk.bg
1191180692 Q * Alissa Ping timeout: 480 seconds
1191180728 Q * cedric Quit: cedric
1191180735 J * zLinux ~zLinux@88.213.35.36
1191181063 M * bzed Bertl: still not solved the bug?
1191181081 M * bzed Bertl: didn;t have the time to look into it yet :\
1191181113 M * Bertl problem is, we cannot reproduce it
1191181146 M * Bertl i.e. it seems to happen on arekm's machines, but it doesn't happen on any of our machines
1191181257 M * JonB Bertl: have arekm ship the machines to you ;-)
1191181272 M * Bertl yeah, good idea :)
1191181420 M * fb Bertl: is that a problem with 2.2/3 version only?
1191181460 M * daniel_hozac 2.0 too, apparently...
1191181471 M * Bertl it isn't a problem we actually see atm
1191181520 M * fb hm
1191181545 M * daniel_hozac arekm is the only one who has been able to cause it.
1191181546 M * fb i have 2.2 on smp with ntpd running
1191181557 M * fb for a year now, i think
1191181566 M * fb without even one warning
1191181577 M * fb it's debian etch
1191181583 M * Bertl well, we do kernel compiles in Linux-VServer guests (on SMP machines) all the time too
1191181600 M * daniel_hozac yeah.
1191181612 M * fb and afaik guys at our system administration prefer slackware and pld
1191181614 M * Bertl note: I'm not saying that this isn't an issue, it just seems very unlikely to trigger (usually)
1191181626 M * fb and have similar compaq servers
1191181648 M * fb never heard about similar problems from them
1191181672 M * fb and i also compile kernel inside the guest
1191181736 J * FireEgl FireEgl@4.0.0.0.1.0.0.0.c.d.4.8.0.c.5.0.1.0.0.2.ip6.arpa
1191181750 Q * CWC Quit: Client exiting
1191181859 M * fb and that makes it even more interesting :) I hope you'll find the issue :)
1191181880 M * Bertl we hope so too
1191181902 J * Alissa ~Alissa@151.81.9.165
1191181920 M * JonB daniel_hozac: arekm and static?
1191181949 M * daniel_hozac huh?
1191182170 M * JonB daniel_hozac: since arekm was the only one being able to hit the bug
1191182236 J * adrien-modulis ~adrien@216.252.77.86
1191182253 M * Alissa grrrrrrrr
1191182257 M * adrien-modulis Hi
1191182258 M * Alissa vkill not work >.>
1191182263 M * Alissa hi adrien-modulis
1191182273 M * daniel_hozac Alissa: what makes you say that?
1191182306 M * Alissa daniel_hozac I have a ghost of process of vserver
1191182319 M * Alissa (vserver-stop not work proprely)
1191182330 M * daniel_hozac how does vps faux list the processes?
1191182334 M * Alissa and vkill do not kill a process >.>
1191182352 M * Alissa daniel_hozac vserver-stat have the vserver on the list
1191182359 M * Alissa and vtop list a pricess
1191182362 M * Alissa *process
1191182372 M * daniel_hozac yes, but i'm more interested in the process state.
1191182383 M * daniel_hozac i.e. is it D? Z? R? S?
1191182409 M * Alissa USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1191182410 M * Alissa news     30917  0.5  0.0      0     0 ?        Ds   Sep29  17:34 [dnews]
1191182461 M * daniel_hozac that means you can't interrupt the process because it's doing something in kernel space.
1191182477 M * daniel_hozac do you have any kernel traces or similar in dmesg?
1191182491 M * Alissa wait...
1191182542 M * adrien-modulis Hi have a question, I've found cheap P4 servers but with limited scsi hard drive capacity (additionnals hd too expensive); so I plan to build a storage server with a desktop machine that has 3 sata 500go in software raid5. My question is finnaly will vserver run fine over NFS with this kind of setup.
1191182562 M * Alissa daniel_hozac no traces found...
1191182569 M * Bertl adrien-modulis: sure
1191182632 M * Bertl [dnews] looks like a kernel thread, no?
1191182661 M * Alissa [root@fc56390 ~]# dmesg | grep "dnews"
1191182661 M * Alissa [root@fc56390 ~]#
1191182694 M * Bertl try dmesg | grep oops
1191182696 M * daniel_hozac Bertl: ah, you're right.
1191182714 M * daniel_hozac that begs the question, how did the guest manage to start that?
1191182734 M * Bertl too many capabilities? :)
1191182750 M * daniel_hozac didn't we just add that capability?
1191182763 M * Alissa Bertl no oops found...
1191182768 M * Alissa empty list...
1191182790 M * adrien-modulis following question, in this setup the storage server becomes a single point of failure (I have 2 P4 servers, so I can run all the vserver on only one), any idea on how not to make this redundant ? I will ran into synchronisation problems If i have to storage server
1191182851 M * daniel_hozac you could use something like drbd.
1191182910 M * adrien-modulis thanks, it looks like it was I need
1191183097 M * Alissa uhmmm
1191183115 M * Alissa daniel_hozac only solution of solve my problem it's a reboot right?
1191183152 M * daniel_hozac unless you can figure out how to stop that kernel thread, i think so.
1191183181 M * Alissa uhmmm
1191183187 M * Alissa I not have idea... :/
1191183382 J * Punkie_ ~Punkie@home.pekelny.net
1191183533 J * Punkie__ ~Punkie@106.153.broadband2.iol.cz
1191183592 Q * Punkie__ 
1191183677 Q * Punkie Ping timeout: 480 seconds
1191183679 Q * Punkie_ Read error: No route to host
1191183953 M * meandtheshell hi folks  - I'd like to give Xen and Linux-VServer a shoot. Does anybody know how the kernel version I pick for dom0 effects the one in some domU? Do they need to be the same? If not what are the limitations? yes ... I already asked at #xen ;-]
1191184007 M * daniel_hozac #xen is probably the only place you'll get educated answers :)
1191184019 M * daniel_hozac but AFAIK, there is not direct relation between the two.
1191184189 J * Aiken ~james@ppp121-45-249-108.lns2.bne4.internode.on.net
1191184398 M * meandtheshell daniel_hozac: same for me "as far as I know ..." but I am not sure ;)
1191184419 M * daniel_hozac so, try it? :)
1191184457 M * meandtheshell will do
1191184766 Q * bruno_ Quit: Leaving
1191184808 Q * jmcaricand Quit: KVIrc 3.2.4 Anomalies http://www.kvirc.net/
1191184848 Q * yarihm Quit: Leaving
1191184886 P * adrien-modulis 
1191184968 M * bzed meandtheshell: you could try debian's xen-vserver kernels, they allow you to run vservers in xen guests :)
1191184987 A * meandtheshell knows that :)
1191185437 Q * meandtheshell Quit: Leaving.
1191185706 Q * FireEgl Ping timeout: 480 seconds
1191186224 J * FireEgl FireEgl@Sebastian.Atlantica.DollarDNS.Net
1191187073 J * Punkie ~Punkie@home.pekelny.net
1191187146 Q * nebuchadnezzar Read error: Connection reset by peer
1191187229 J * nebuchadnezzar ~nebu@zion.asgardr.info
1191187307 J * sladen_ paul@starsky.19inch.net
1191187341 Q * sladen Write error: connection closed
1191187437 N * sladen_ sladen
1191187944 Q * dna_ Quit: Verlassend
1191188438 J * Ashsong_ ~mstone@teach.laptop.org
1191188472 J * FBW ~fbw@fbw.cc
1191188495 Q * coderanger synthon.oftc.net oxygen.oftc.net
1191188495 Q * Darkglow synthon.oftc.net oxygen.oftc.net
1191188495 Q * Ashsong synthon.oftc.net oxygen.oftc.net
1191188893 J * Darkglow ~pdesnoyer@208.71.184.41
1191189437 M * Bertl okay, off to bed now .. have a good one everyone!
1191189442 N * Bertl Bertl_zZ
1191189826 J * Aly_away ~Alissa@151.81.6.52
1191189829 Q * Aly_away 
1191189901 Q * quasisane Quit: ERC Version 5.2 (IRC client for Emacs)
1191189977 Q * FBW Remote host closed the connection
1191190187 Q * Alissa Ping timeout: 480 seconds
1191190758 Q * JonB Ping timeout: 480 seconds
1191191825 Q * ema Quit: leaving
1191193227 J * quasisane ~sanep@c-76-118-191-64.hsd1.nh.comcast.net
1191194642 Q * ktwilight Ping timeout: 480 seconds
1191194914 Q * Punkie Quit: ChatZilla 0.9.78.1 [Firefox 2.0.0.7/2007091417]