1318896246 Q * karasz charon.oftc.net kilo.oftc.net
1318896266 J * karasz ~karasz@shell.opensde.net
1318901373 Q * ircuser-1_ Ping timeout: 480 seconds
1318902282 Q * hparker Quit: Quit
1318905229 J * fisted_ ~fisted@xdsl-87-78-218-25.netcologne.de
1318905253 Q * fisted Ping timeout: 480 seconds
1318910124 M * Bertl off to bed now ... have a good one everyone!
1318910130 N * Bertl Bertl_zZ
1318914174 Q * guerby Read error: Connection reset by peer
1318914274 J * guerby ~guerby@nc10d.tetaneutral.net
1318914408 J * sannes1 ~ace@cm-84.209.106.118.getinternet.no
1318915973 Q * FireEgl Remote host closed the connection
1318916080 J * derjohn_foo ~aj@213.238.45.2
1318916542 J * ghislain ~AQUEOS@adsl2.aqueos.com
1318916906 Q * derjohn_foo Remote host closed the connection
1318917212 J * derjohn_mob ~aj@213.238.45.2
1318917646 J * FireEgl ~FireEgl@173-16-9-169.client.mchsi.com
1318918140 J * ncopa ~ncopa@3.203.202.84.customer.cdi.no
1318920440 J * hijacker_ ~hijacker@cable-84-43-136-96.mnet.bg
1318922727 Q * hijacker_ Quit: Leaving
1318924432 Q * Aiken Quit: Leaving
1318926146 M * Mr_Smoke ghislain: any feedback on 3.0.4-2.3.1 yet ?
1318926153 M * Mr_Smoke and hi :)
1318927265 J * jeroen__ ~jeroen@imap.powerinternet.eu
1318928254 M * ghislain hi mr !
1318928288 M * ghislain i compiled it and need to put it on some machine now but customers do not let me any time right now,  i hope i will be able to put it soon
1318928308 M * ghislain all my machines are remote so i must do this carefully or loose them
1318928335 M * Mr_Smoke Same here
1318928345 M * Mr_Smoke And i just had another panic with the current one
1318928350 M * Mr_Smoke So I might jump right in, too
1318928366 M * Mr_Smoke Time for lunch, see ya
1318928370 M * ghislain i have some random panic with the one i use but i am unable to knwo from where it comes
1318928413 M * ghislain but they are so few and every 4/8 month separated so i cannot say if it is not related to anything 
1318930076 Q * FireEgl Read error: Connection reset by peer
1318930703 N * Bertl_zZ Bertl
1318930707 M * Bertl morning folks!
1318931039 J * FireEgl Sebtest@2001:470:e056:1:f85e:ed4e:abb6:e123
1318931340 M * renihs morning Bertl!
1318934463 J * kir ~kir@swsoft-msk-nat.sw.ru
1318934828 M * ser morning
1318934907 M * Bertl hey ser! LTNS!
1318934961 M * ser thesis writeup, hard times...
1318939708 M * Bertl off for now ... bbl
1318939713 N * Bertl Bertl_oO
1318940490 M * Mr_Smoke Argh
1318940493 M * Mr_Smoke Just missed bertl :p
1318940507 M * Mr_Smoke ghislain: aha, interesting
1318940523 M * Mr_Smoke ghislain: I have panics that are spaced across the same period, approx
1318940533 M * Mr_Smoke There is a consistency about them though
1318940561 M * Mr_Smoke First there's a paging error, then a panic
1318941113 M * Mr_Smoke Quite a few paging errors even
1318941904 M * Mr_Smoke I see a pattern now
1318941906 M * Mr_Smoke Time to mail
1318943806 Q * ncopa Quit: Leaving
1318945576 M * ghislain yes, i do not see anything as my hosting provider just reboot the servers...
1318946002 Q * AndrewLee Ping timeout: 480 seconds
1318946098 M * Mr_Smoke ghislain: who's your ISP ? OVH ?
1318946130 J * AndrewLee ~andrew@n201.enc.hlc.edu.tw
1318946152 J * ircuser-1 ~ircuser-1@025.205-93-216-nokia-dsl.dynamic.surewest.net
1318949057 Q * fisted_ Ping timeout: 480 seconds
1318949102 J * fisted ~fisted@xdsl-87-78-214-219.netcologne.de
1318949886 J * dowdle ~dowdle@scott.coe.montana.edu
1318950049 N * Bertl_oO Bertl
1318950053 M * Bertl back now ...
1318950055 J * thierryp ~thierry@zankai.inria.fr
1318950459 M * Mr_Smoke Hi Bertl 
1318950465 M * Mr_Smoke When you say "annotated stack trace"
1318950482 M * Mr_Smoke You mean you'd want to see the result for addr2line for each of the addresses, right ?
1318950845 M * Bertl yep, or at least the ones for the first few and the EIP
1318950894 M * Mr_Smoke Bertl: want the corresponding line of code from the kernel source too I guess ? :)
1318950926 M * Bertl if it differs from upstream + Linux-VServer patch, then yes
1318951088 M * Mr_Smoke hm it wouldn't, no
1318951096 M * Mr_Smoke shall I mail ot back to the list ?
1318951125 M * Mr_Smoke Or do you want it here ?
1318951140 M * Bertl either that, or better upload it somewhere where it can be viewed with a web browser
1318951162 M * Bertl (and then paste the url here and/or mail it to the list)
1318951182 M * Mr_Smoke Bertl: http://pastebin.com/HyMsRjNU
1318951198 M * Mr_Smoke So this is 2.6.38.7-vs2.3.0.37-rc15
1318951213 M * Mr_Smoke if in doubt on any line I can fetch it of course
1318951272 M * Bertl can you make that paste in such a way that the resolved addresses are in the same line as the stack address?
1318951282 M * Mr_Smoke sure
1318951294 M * Bertl that's a lot easier to read
1318951304 M * Bertl also, please include the full dump 
1318951311 M * Bertl i.e. not just bits and pieces
1318951339 M * Mr_Smoke um
1318951346 M * Bertl (otherwise everybody looking at it has to puzzle it back together)
1318951348 M * Mr_Smoke For the panic ? 
1318951349 J * derjohn_foo ~aj@213.238.45.2
1318951359 M * Mr_Smoke or for any page allocation failure ?
1318951384 M * Bertl just take the stack trace/panic/whatever as it comes out of dmesg or your console
1318951396 M * Bertl from start to end (cut here)
1318951412 M * Bertl and then annotate it with the resolved lines
1318951416 M * Mr_Smoke ok
1318951423 Q * derjohn_mob Read error: Connection reset by peer
1318951424 M * Mr_Smoke Gonna take a few mins, working on it
1318951432 M * Bertl i.e. no (...) no cutting of the head or tail, etc
1318951436 M * Mr_Smoke ok
1318951462 M * Bertl and make one paste/file per incident, if possible with a meaninful name
1318951510 M * Bertl something like incident, 30th September , 18:20, after guest start
1318951526 M * Mr_Smoke ok
1318951531 M * Mr_Smoke hm
1318951541 M * Mr_Smoke addr2line sometimes refers to /usr/src/linux/...
1318951543 M * Mr_Smoke and sometimes :
1318951554 M * Mr_Smoke to /usr/src/linux-2.6.38.7-vs2.3.0.37-rc15
1318951556 M * Mr_Smoke Is that bad ?
1318951588 M * Bertl probably not, just suggests that parts of that are in modules
1318951900 M * Mr_Smoke its a monolithic kernel, fwiw
1318951995 M * Mr_Smoke phew, 9 lines to go
1318952076 J * alpha_one_x86 ~kvirc@201.222.115.8
1318952108 M * alpha_one_x86 Hello, what is the more stable: vs2.3.0.37-rc17, vs2.3.1-pre9.2 or vs2.3.1 ?
1318952116 M * Bertl Mr_Smoke: then addr2line is somehow confused, but it doesn't really matter
1318952145 M * Bertl alpha_one_x86: isn't that clear from the nomenclature?
1318952162 M * alpha_one_x86 vs 2.3.1 is the release version?
1318952183 M * Bertl rc = release candidate, pre = prerelease, and vs2.3.1 is a release
1318952200 M * alpha_one_x86 ok, then the 2.3.1 is the most stable, thanks
1318952286 M * Mr_Smoke Bertl: http://pastebin.com/r4cXPyc4
1318952291 M * Bertl alpha_one_x86: doesn't mean it's the most tested version though
1318952292 M * Mr_Smoke pastebin is word wrapping though :/
1318952339 M * Bertl http://pastebin.com/raw.php?i=r4cXPyc4 is almost fine
1318952339 M * Mr_Smoke I'm trying pastebin.ca
1318952352 M * Mr_Smoke ok
1318952364 M * Bertl i.e. if you can get rid of the zigzag on the annotation, it would be perfect
1318952367 M * Mr_Smoke The tabbing went awry for some reason
1318952377 M * Mr_Smoke It's tabbed correctly here
1318952381 M * Mr_Smoke yep, can try
1318952391 M * Mr_Smoke Oh wait, can't, pastebin.com won't allow edits
1318952392 M * Bertl just run it through expand
1318952404 M * Bertl (before pasting)
1318952447 M * Bertl and the paste is still missing the information around that trace
1318952480 M * Bertl i.e. there should be a register dump before that and there is some more stuff after that
1318952547 Q * alpha_one_x86 Quit: KVIrc KVIrc Equilibrium 4.1.1, revision: 5816, sources date: 20110403, built on: 2011-06-11 13:01:00 UTC http://www.kvirc.net/
1318952557 M * Mr_Smoke Bertl: hows't that http://pastebin.ca/2091110
1318952577 M * Mr_Smoke Well, before the panic is a different trace for a page allocation stuff
1318952626 M * Mr_Smoke I'll add that in
1318952655 M * Mr_Smoke Hm it's not page alloc, it's a null ptr deref
1318952770 M * Bertl if there is no clear line/break between two traces, always combine them
1318952782 M * Bertl as most likely the second one is the result of the first one
1318953353 M * Mr_Smoke ok
1318953355 M * Mr_Smoke coming up
1318953368 M * Mr_Smoke Bertl: http://pastebin.ca/2091113
1318953375 M * Mr_Smoke That's as clean as pastebin.ca will let me tab it
1318953444 M * Mr_Smoke Is it usable ?
1318953507 M * Bertl guess it will do ...
1318953518 M * Mr_Smoke I'm sorry, I'm not very good at this
1318953521 M * Mr_Smoke First time :/
1318953562 M * Bertl np, you might also forget about pastebin* and just wrap them up into a tar.gz or similar, and upload to some upload service
1318953582 M * Mr_Smoke True
1318953591 M * Mr_Smoke duly noted for next time
1318953597 M * Bertl that way, I can download it and put it somewhere where it isn't mangled
1318953607 M * Mr_Smoke I might whip up some sed-based script that would do the addr2line stuff for me too
1318953628 M * Mr_Smoke that probably exists somewhere already
1318953820 M * Mr_Smoke The trace for this morning looks awfully similar, too
1318953877 Q * thierryp Remote host closed the connection
1318954106 M * Bertl you can pipe more than one address into addr2line
1318954135 M * Bertl and at some point, there was a script to do all the work, but it doesn't seem to work anymore, IIRC
1318954159 J * guifre ~guifre@aopcgr.uab.es
1318954160 M * Mr_Smoke Oh, I'll definitely look up the "more than 1 address" part then
1318954262 M * Mr_Smoke The hardest part is I couldn't isolate anything that would systematically cause the panic
1318954278 M * Mr_Smoke munin seems to be involved, probably because it keeps polling sysfs to do its job
1318954296 M * Mr_Smoke Also, for the record, this machine has had 65 call traces sicne June 16
1318954310 M * Mr_Smoke The other, almost identical machine, with the same kernel, has had NONE.
1318954322 M * Mr_Smoke This is why I'm suspicious about the hardware
1318954335 J * bonbons ~bonbons@2001:960:7ab:0:d9b3:3ce:39a6:3eda
1318954352 M * Mr_Smoke Also, the machine that had the panics hosts a program that causes a lot of these to happen :
1318954355 M * Mr_Smoke Sep 30 03:04:59 srv505 kernel: TCP: Possible SYN flooding on port 6809. Sending cookies.
1318954395 M * Mr_Smoke But I don't think that's related, they don't happen around the times that the kernel paniced
1318954506 M * Bertl my suggestion, as stated on the ML, would be to switch the entire software installation (or the hardware if that is easier to understand :)
1318954536 M * Bertl i.e. swap the disks or the disk content, and keep the hardware as is
1318954553 M * Mr_Smoke Not sure the ISP will agree to that
1318954564 M * Bertl if the issue suddenly appears on the other machine, it's very likely a software issue
1318954586 M * Mr_Smoke I'll see a diff between the kernel configs for starers
1318954612 M * Bertl alternatively, you can ask the ISP to switch the entire hardware for the problematic macine
1318954616 M * Bertl *machine
1318954658 M * Mr_Smoke Hm, identical kernel config, apart from debug
1318954664 M * Mr_Smoke Yeah, I can try that
1318954703 M * Mr_Smoke I'm gonna ask that they check the RAM, too
1318954996 M * Mr_Smoke Hah, they're preparing a new machine
1318955005 M * Mr_Smoke just in case
1318955037 M * Mr_Smoke Bertl: to give you a little more context, I've been using that kernel on that machine since may 26th
1318955083 M * Mr_Smoke the page alloc failures started appearing on june 16th
1318955103 M * Mr_Smoke So it is possible that the kernel might be at fault, but I have a hunch that it's not
1318955121 M * Mr_Smoke Unfortunately I have no history before that
1318955151 M * Mr_Smoke Oh wait, I have
1318955330 M * Mr_Smoke And I have older paging requests failures too
1318955440 M * Bertl hardware issues are often hard to diagnose from the panic/traces, especially if they depend on a specific software state
1318955476 M * Bertl e.g. if for example, hard disk activity triggers something in the hardware, it will look like the I/O subsystem is to blame
1318955570 M * Mr_Smoke Yeah, I get the idea
1318955598 M * Mr_Smoke Bertl: so in other words, if your analysis of the trace is inconclusive, it might indicate a probability of h/w failure then, right ?
1318955830 M * Bertl kind of
1318955861 M * Mr_Smoke ok
1318955886 M * Bertl the fact that one system has zero incidents, and the other a series of panics/traces is also a good indication of a hardware issue
1318955904 M * Mr_Smoke Indeed
1318955947 Q * derjohn_foo Ping timeout: 480 seconds
1318958344 M * daniel_hozac ugh.
1318958363 M * daniel_hozac trying to run guests on Fedora 15 is hard.
1318958390 M * daniel_hozac systemd is so great.....
1318958634 M * daniel_hozac ... and now i can't even communicate with it :-)
1318958878 Q * arekm Quit: leaving
1318959056 J * arekm ~arekm@ixion.pld-linux.org
1318959719 M * Bertl sounds like the real deal!
1318959786 M * Bertl what was the name of the other great replacement for sysv?
1318959913 M * Bertl ah, upstart ...
1318959935 A * arekm missed some interesting discussion ;P
1318959944 M * arekm upstart at least works in guest. does systemd?
1318959972 M * sannes1 arekm : without monkey patchinig it? :P
1318959984 M * daniel_hozac systemd in a guest is probably impossible.
1318959999 M * daniel_hozac i mean, even on a host it seems like a lot of work.
1318960243 J * hparker ~hparker@2001:470:1f0f:32c:beae:c5ff:fe01:b647
1318960662 Q * quasisane Quit: leaving
1318960756 M * arekm sannes1: without patching
1318961964 J * quasisane ~sanep@76.24.80.97
1318962050 P * kir Leaving.
1318964156 M * Bertl off for a nap ... bbl
1318964162 N * Bertl Bertl_zZ
1318967994 J * hijacker_ ~hijacker@cable-84-43-136-96.mnet.bg
1318970607 Q * sannes1 Remote host closed the connection
1318970862 Q * hijacker_ Quit: Leaving
1318971958 N * Bertl_zZ Bertl
1318971961 M * Bertl back now ...
1318973641 Q * bonbons Quit: Leaving
1318977915 Q * ghislain Quit: Leaving.
1318978617 Q * dowdle Remote host closed the connection