1357279227 M * Bertl off to bed now ... have a good one everyone!
1357279235 N * Bertl Bertl_zZ
1357279470 J * nkukard ~nkukard@41-133-139-74.dsl.mweb.co.za
1357283770 J * Ghislain ~aqueos@adsl1.aqueos.com
1357283928 J * alex3 ~alex@v1.fob.spline.inf.fu-berlin.de
1357288067 J * Hollow ~Hollow@217.111.2.114
1357288115 Q * Hollow 
1357288132 J * Hollow ~Hollow@217.111.2.114
1357290679 Q * nkukard Remote host closed the connection
1357293890 Q * FloodServ reticulum.oftc.net synthon.oftc.net
1357293890 Q * daniel_hozac reticulum.oftc.net synthon.oftc.net
1357293890 Q * fosco reticulum.oftc.net synthon.oftc.net
1357293890 Q * arekm reticulum.oftc.net synthon.oftc.net
1357293890 Q * macmaN reticulum.oftc.net synthon.oftc.net
1357293890 Q * cuba33ci reticulum.oftc.net synthon.oftc.net
1357293890 Q * theocrit1 reticulum.oftc.net synthon.oftc.net
1357293890 Q * AndrewLee reticulum.oftc.net synthon.oftc.net
1357293890 Q * _Shiva_ reticulum.oftc.net synthon.oftc.net
1357293890 Q * imachine_ reticulum.oftc.net synthon.oftc.net
1357293890 Q * tokkee reticulum.oftc.net synthon.oftc.net
1357293890 Q * transacid reticulum.oftc.net synthon.oftc.net
1357293890 Q * sladen reticulum.oftc.net synthon.oftc.net
1357293890 Q * aurel42 reticulum.oftc.net synthon.oftc.net
1357293890 Q * DelTree reticulum.oftc.net synthon.oftc.net
1357293890 Q * click reticulum.oftc.net synthon.oftc.net
1357293890 Q * Jb_boin reticulum.oftc.net synthon.oftc.net
1357293890 Q * grobie reticulum.oftc.net synthon.oftc.net
1357293890 Q * hijacker reticulum.oftc.net synthon.oftc.net
1357293890 Q * Rockj reticulum.oftc.net synthon.oftc.net
1357293890 Q * nyerup reticulum.oftc.net synthon.oftc.net
1357293890 Q * Bertl_zZ reticulum.oftc.net synthon.oftc.net
1357293890 Q * morrigan reticulum.oftc.net synthon.oftc.net
1357293890 Q * guerby reticulum.oftc.net synthon.oftc.net
1357293890 Q * ser_ reticulum.oftc.net synthon.oftc.net
1357293890 Q * ccxCZ reticulum.oftc.net synthon.oftc.net
1357293890 Q * DLange reticulum.oftc.net synthon.oftc.net
1357293890 Q * fisted reticulum.oftc.net synthon.oftc.net
1357293890 Q * geos_one reticulum.oftc.net synthon.oftc.net
1357293890 Q * ensc|w reticulum.oftc.net synthon.oftc.net
1357293890 Q * ircuser-1 reticulum.oftc.net synthon.oftc.net
1357293890 Q * _urbee reticulum.oftc.net synthon.oftc.net
1357293890 Q * BlackPanx reticulum.oftc.net synthon.oftc.net
1357293890 Q * Aiken reticulum.oftc.net synthon.oftc.net
1357293890 Q * Sith10 reticulum.oftc.net synthon.oftc.net
1357293890 Q * Guy- reticulum.oftc.net synthon.oftc.net
1357293890 Q * _nono_ reticulum.oftc.net synthon.oftc.net
1357293890 Q * mnemoc reticulum.oftc.net synthon.oftc.net
1357293890 Q * hparker reticulum.oftc.net synthon.oftc.net
1357293890 Q * Hunger reticulum.oftc.net synthon.oftc.net
1357293890 Q * alex3 reticulum.oftc.net synthon.oftc.net
1357293890 Q * MooingLemur reticulum.oftc.net synthon.oftc.net
1357293890 Q * nou reticulum.oftc.net synthon.oftc.net
1357293890 Q * eyck reticulum.oftc.net synthon.oftc.net
1357293890 Q * ntrs reticulum.oftc.net synthon.oftc.net
1357293890 Q * ard reticulum.oftc.net synthon.oftc.net
1357293890 Q * sid3windr reticulum.oftc.net synthon.oftc.net
1357293890 Q * ivan` reticulum.oftc.net synthon.oftc.net
1357293890 Q * yang reticulum.oftc.net synthon.oftc.net
1357293890 Q * mcp reticulum.oftc.net synthon.oftc.net
1357293890 Q * tolkor reticulum.oftc.net synthon.oftc.net
1357293890 Q * PowerKe_ reticulum.oftc.net synthon.oftc.net
1357293891 Q * harry reticulum.oftc.net synthon.oftc.net
1357293891 Q * Wonka reticulum.oftc.net synthon.oftc.net
1357293891 Q * Defaulttinen reticulum.oftc.net synthon.oftc.net
1357293891 Q * Hollow reticulum.oftc.net synthon.oftc.net
1357293891 Q * Ghislain reticulum.oftc.net synthon.oftc.net
1357293891 Q * Romster reticulum.oftc.net synthon.oftc.net
1357293891 Q * jrklein reticulum.oftc.net synthon.oftc.net
1357293891 Q * m_ueberall reticulum.oftc.net synthon.oftc.net
1357293891 Q * fichte` reticulum.oftc.net synthon.oftc.net
1357293891 Q * ex reticulum.oftc.net synthon.oftc.net
1357293891 Q * swenTjuln reticulum.oftc.net synthon.oftc.net
1357293891 Q * _are_ reticulum.oftc.net synthon.oftc.net
1357293891 Q * trippeh reticulum.oftc.net synthon.oftc.net
1357293891 Q * geb reticulum.oftc.net synthon.oftc.net
1357293891 Q * quasisane reticulum.oftc.net synthon.oftc.net
1357293891 Q * micah reticulum.oftc.net synthon.oftc.net
1357293891 Q * jrayhawk reticulum.oftc.net synthon.oftc.net
1357293891 Q * puck reticulum.oftc.net synthon.oftc.net
1357293891 Q * bzed reticulum.oftc.net synthon.oftc.net
1357293891 Q * brambles reticulum.oftc.net synthon.oftc.net
1357294147 J * ser ~ser@host1.tldp.ibiblio.org
1357294147 J * Hollow ~Hollow@217.111.2.114
1357294147 J * alex3 ~alex@v1.fob.spline.inf.fu-berlin.de
1357294147 J * Ghislain ~aqueos@adsl1.aqueos.com
1357294147 J * fisted ~fisted@xdsl-87-78-82-151.netcologne.de
1357294147 J * geos_one ~chatzilla@80.123.185.198
1357294147 J * ensc|w ~ensc@www.sigma-chemnitz.de
1357294147 J * ircuser-1 ~ircuser-1@35.222-62-69.ftth.swbr.surewest.net
1357294147 J * _urbee ~urbee@93-103-199-233.dynamic.t-2.net
1357294147 J * mnemoc ~amery@geeks.cl
1357294147 J * _nono_ ~gomes@licencieux.ircam.fr
1357294147 J * BlackPanx ~alen@91.209.18.70
1357294147 J * Hunger hunger@proactivesec.com
1357294147 J * Aiken ~Aiken@2001:44b8:2168:1000:21f:d0ff:fed6:d63f
1357294147 J * hparker ~hparker@2001:470:1f0f:32c:beae:c5ff:fe01:b647
1357294147 J * Sith10 xsx@156.17.70.157
1357294147 J * Guy- ~korn@elan.rulez.org
1357294147 J * transacid ~transacid@0001293b.user.oftc.net
1357294147 J * sladen ~paul@starsky.19inch.net
1357294147 J * MooingLemur ~troy@phx-pnap.pinchaser.com
1357294147 J * yang yang@yang.netrep.oftc.net
1357294147 J * arekm ~arekm@000161e0.user.oftc.net
1357294147 J * nou Chaton@causse.larzac.fr.eu.org
1357294147 J * Romster ~romster@202.168.100.149.dynamic.rev.eftel.com
1357294147 J * macmaN ~chezburge@138.167.190.90.dyn.estpak.ee
1357294147 J * cuba33ci ~cuba33ci@114-25-201-127.dynamic.hinet.net
1357294147 J * theocrit1 ~Hubert@kim.theocrite.org
1357294147 J * eyck ~eyck@nat08.nowanet.pl
1357294147 J * jrklein ~osx@2001:470:1f0f:572::250:160
1357294147 J * mcp ~mcp@wolk-project.de
1357294147 J * AndrewLee ~andrew@n201.enc.hlc.edu.tw
1357294147 J * tolkor ~rj@159.28.7.241
1357294147 J * Defaulttinen defaultti@lakka.kapsi.fi
1357294147 J * ivan` ~ivan`@000130ca.user.oftc.net
1357294147 J * imachine_ ~imachine@robot.greenhost24.pl
1357294147 J * ntrs ~ntrs@vault08.rosehosting.com
1357294147 J * sid3windr luser@bastard-operator.from-hell.be
1357294147 J * DLange ~DLange@dlange.user.oftc.net
1357294147 J * ard ~ard@shell2.kwaak.net
1357294147 J * _Shiva_ shiva@whatcha.looking.at
1357294147 J * harry ~harry@enzoverder.be
1357294147 J * tokkee tokkee@osprey.tokkee.org
1357294147 J * grobie ~grobie@tyr.schnuckelig.eu
1357294147 J * swenTjuln ~Marko@195.95.173.243
1357294147 J * click click@ice.vcon.no
1357294147 J * quasisane ~sanep@c-24-218-184-186.hsd1.nh.comcast.net
1357294147 J * Wonka produziert@chaos.in-kiel.de
1357294147 J * brambles lechuck@s0.barwen.ch
1357294147 J * trippeh atomt@t-1000.ugh.no
1357294147 J * hijacker ~hijacker@213.91.163.5
1357294147 J * micah ~micah@199.254.238.47
1357294147 J * Rockj rockj@rockj.net
1357294147 J * guerby ~guerby@nc10d-ipv6.tetaneutral.net
1357294147 J * daniel_hozac ~daniel@h149n2-spaa-a12.ias.bredband.telia.com
1357294147 J * fosco fosco@marx.wirefull.org
1357294147 J * geb ~geb@mars.gebura.eu.org
1357294147 J * FloodServ services@services.oftc.net
1357294147 J * nyerup irc@jespernyerup.dk
1357294147 J * PowerKe_ ~tom@94-226-105-17.access.telenet.be
1357294147 J * jrayhawk ~jrayhawk@nursie.omgwallhack.org
1357294147 J * ex ~ex@valis.net.pl
1357294147 J * DelTree ~deplagne@alcorak1.eric.deplagne.name
1357294147 J * fichte` ~fichte@bashpipe.de
1357294147 J * ccxCZ ~ccxCZ@156.200.broadband11.iol.cz
1357294147 J * aurel42 ~aurel@sid.a42.de
1357294147 J * morrigan morrigan@IRC.13thfloor.at
1357294147 J * bzed ~bzed@bzed.netrep.oftc.net
1357294147 J * _are_ ~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285
1357294147 J * m_ueberall dircproxy4@2a01:4f8:100:80e3:0:bc28:af65:5
1357294147 J * Jb_boin ~dedior@proxad.eu
1357294147 J * Bertl_zZ herbert@IRC.13thfloor.at
1357294147 J * puck ~puck@2404:130:0:1000::23:10
1357295520 Q * Hollow Quit: Hollow
1357296667 J * BenG ~bengreen@cpc29-aztw23-2-0-cust144.18-1.cable.virginmedia.com
1357297202 Q * fisted Remote host closed the connection
1357297262 J * fisted ~fisted@xdsl-87-78-81-212.netcologne.de
1357299522 M * ard http://paste.linux-vserver.org/23152
1357299547 M * ard up until line 777 everything is fine...
1357299571 M * ard and then it fails to find src addresses
1357299596 M * ard because it is waiting on a _raw_spin_lock
1357299610 Q * ircuser-1 Ping timeout: 480 seconds
1357299642 M * ard I don't know if this is an upstream issue or specific the vserver patch.
1357299667 M * ard It's hard to trigger, since the system had been running for 14 hours before this occured
1357300171 M * ard actually, it is trying to determine a source ip on an outgoing connection.
1357300276 M * ard +extern struct rtable *ip_v4_find_src(struct net *net, struct nx_info *,
1357300276 M * ard +       struct flowi4 *);
1357300287 M * ard is the function that is stuck on a _raw_spin_lock
1357300295 M * ard so there definetily is a race there
1357300950 Q * Aiken Remote host closed the connection
1357301866 J * Hollow ~Hollow@217.111.2.114
1357302359 N * alex3 AlexS
1357302395 N * AlexS AlexanderS
1357303121 J * ircuser-1 ~ircuser-1@35.222-62-69.ftth.swbr.surewest.net
1357303942 Q * ensc|w Remote host closed the connection
1357303952 J * ensc|w ~ensc@www.sigma-chemnitz.de
1357304250 Q * BenG Quit: I Leave
1357304539 J * locquest ~fede@190.193.53.25
1357306892 J * sannes ~ace@cm-84.211.93.82.getinternet.no
1357307391 Q * sannes Remote host closed the connection
1357307405 J * sannes ~ace@cm-84.211.93.82.getinternet.no
1357308973 M * disposable Bertl_zZ: i am also experiencing crashes. i don't even have to run a vserver for that. i installed a 3.7.0 kernel with vs2.3.5.3 patch, the installed the newest util-vserver (pre3038). When i typed reboot, i got http://i.imgur.com/h9OWd.png I'll install the kernel image with debugging enabled and see if i can provide more info than just that.
1357309821 M * disposable actually i can provide more info even now. http://paste.linux-vserver.org/23153 The crash (during shutdown) could actually be nfs related. unfortunately, i cannot be sure. All i know is that reboots worked fine before i installed util-vserver tools.
1357310024 N * Bertl_zZ Bertl
1357310027 M * Bertl morning folks!
1357310059 M * Bertl disposable: any reason why you use 3.7.0 with the vs2.3.5.3 patch?
1357310185 M * Bertl anyway, it seems like an nfs bug, probably triggered by the additional namespaces and/or sharing of that rpc_pipefs mount
1357311179 M * disposable Bertl: simply because when you wrote the 2.3.5.3 patch, it was teh latest kernel. 3.7.1 was not out yet
1357311262 M * Bertl hmm, vs2.3.5.3 was _only_ released for 3.7.1, 3.7 had vs2.3.5.1
1357311296 M * Bertl http://vserver.13thfloor.at/ExperimentalT/
1357311451 M * disposable are you sure? i played with it before it appeared on linux-vserver.org homepage
1357311487 M * Bertl that directory is the only one where I upload patches, and they are not removed from there either
1357311669 M * disposable Bertl: considering i wanted the single ip removal functinoality for 3.5 kernel and you said it should not be a problem, i don't see why running it on 3.7.0 should be bad.
1357311727 M * Bertl nah, no problem, just curious
1357311777 M * disposable anyway, i have a few questions about util-vserver now. what is util-vserver-legacy for? 
1357311801 M * Bertl for very old (ancient) kernels and kernel interfaces
1357311813 M * disposable thanks
1357311816 M * Bertl (nothing you want to use/install on any 2.6/3.x kernel)
1357311883 M * Bertl could you try a 3.6.x kernel/patch with your test system (vm) and see if the rpc_pipefs bug goes away?
1357311994 M * Bertl (same userspace, same kernel config as far as possible)
1357312328 M * disposable i have 2 kinds of vserver hosts (actually many kinds, but only 2 matter for this question). ubuntu 12.04 that were upgraded from ubuntu 10.04 and ubuntu 12.04 that are clean new install. on the upgraded ones, my existing vservers are running fine with the vs2.3.5.3 patched kernel and pre3038 utils. on the freshly installed ones, where i've just installed the kernel and utils, i can't even start a new vserver (haven't tried existing one). all i g
1357312360 M * disposable Bertl: i will play a lot more with nfs later so i'll come back with results
1357312393 M * Bertl okay, take your time
1357312508 M * disposable Bertl: what about this 'vshelper.init: can not determine xid of vserver 'superb'; returned value was '''? i don't think squeeze (my guest os) uses init-less init-style as the message suggests.
1357312791 M * Bertl it means that the guest exits before util-vserver completed, i.e. no services are started inside the guest, nothing keeps the context alive
1357312835 M * Bertl you need to either configure some services (when in sysv mode) or have an init running (in plain init style)
1357313188 M * Bertl a startup with --debug might shed some light though
1357313216 J * nkukard ~nkukard@41-133-139-74.dsl.mweb.co.za
1357314296 M * disposable how does one "go into sysv mode" and how does one have init running in "plain init style"? in the past, i started the vserver, entered and THEN installed/configure daemons. 
1357314335 M * Bertl at build time, you 'select' the init style, but you can also change it in the config anytime
1357314362 M * Bertl default is 'sysv' init style, which starts a bunch of services (they are configured by the post install scripts)
1357314398 M * Bertl selecting the 'plain' init style results in util-vserver only starting 'init' which has to start the services on its own
1357314469 M * Bertl btw, did you manage to contact daniel_hozac and send his part of the donation?
1357314487 M * disposable Bertl: yes, i did and he acknowledged it. ;)
1357314495 M * Bertl excellent!
1357314554 M * Bertl it's not that we get too many donations nowadays, so every bit helps/brings joy :)
1357314697 M * disposable when did this 'init' malarkey appear? i'd never had vservers refuse to start before and can't find a single line about this in vserver faq or documentation page.
1357315205 M * Bertl the init style was introduced several years ago (8-10 I guess?) when we decided to allow for a real init instead of just the sysv runlevel scripts
1357315229 M * Bertl as I said, there are defaults, and it is explained on the great flower page
1357315297 M * Bertl for the guest, I doubt it is 'refusing' to start (as I said, --debug will give more hints), I presume it starts, doesn't leave anything running, and thus the context is disposed before util-vserver completes
1357315554 M * disposable Bertl: thank you for the explanations and pointing at the flower page
1357316068 M * Bertl you're welcome!
1357316166 M * daniel_hozac just enabling/installing e.g. syslog should work well.
1357316363 M * Bertl usually does the trick for sysv init style, yes
1357316402 M * Bertl off to grab some groceries, bbl
1357316407 N * Bertl Bertl_oO
1357317582 Q * Hollow Quit: Hollow
1357318875 J * wmp ~wmp@2001:41d0:8:531b::1
1357318914 M * wmp hello
1357318981 M * wmp why one vserver use 300% cpu (from vserver-htop) but has this same cpu limit: http://wklej.org/id/914447
1357319072 J * bonbons ~bonbons@2001:960:7ab:0:587f:a6f5:44d9:55d9
1357319098 M * wmp after change /dev/cgroup/<guest name>/cpu.shares  i must restart vserver?
1357319225 M * daniel_hozac no, but cpu.shares isn't a limit.
1357319227 M * daniel_hozac it's a share.
1357319302 M * wmp daniel_hozac: yes, i understand
1357319322 M * wmp daniel_hozac: i have problem with one vserver, becouse use 300% cpu and i want limit this
1357319381 M * wmp daniel_hozac: maybe limit in /etc/security/limits.conf on guest?
1357319411 M * daniel_hozac no. get a kernel with hard CPU limits if that is what you want, and configure it.
1357319451 M * wmp daniel_hozac: kernel must have: CONFIG_CGROUP_NS ?
1357319476 M * daniel_hozac no.
1357319495 M * wmp vserver on /dev/cgroup type cgroup (rw,cpuset,cpu,cpuacct,memory,devices,freezer,net_cls,blkio)
1357319500 M * wmp this is good for cpu limiting?
1357319596 M * daniel_hozac if your kernel has it enabled, that should suffice.
1357319634 M * wmp daniel_hozac: what options must have my kernel?
1357321775 N * Bertl_oO Bertl
1357321778 M * Bertl back now ...
1357321968 M * ard Bertl : did you look at my console log? :-)
1357321974 M * ard [12:38] <ard> http://paste.linux-vserver.org/23152
1357321974 M * ard [12:39] <ard> up until line 777 everything is fine...
1357321974 M * ard [12:39] <ard> and then it fails to find src addresses
1357322002 M * ard If I am correct, there is a race bug somewhere at determining the outgoing source address in connect
1357322008 M * ard could not find out what yet
1357322028 M * ard problem is, that it only occured after 14 hours of intensive work ;-)
1357322066 M * ard As a matter of fact, someone is now trying siege on a comparable installation, and it still did not trigger that problem :-(
1357322132 M * ard maybe I am just looking at it the wrong way, and it's only a sympton that it cannot find the source address because somebody else has locked the routing table
1357322657 M * Bertl do you have the kernel build tree at hand?
1357322683 M * Bertl i.e. can you use addr2line to convert addresses into line numbers?
1357322722 M * ard Hmmmm...
1357322729 M * ard I have the build tree yes :-)
1357322746 M * Bertl okay, try addr2line -e vmlinux on some of the addresses
1357322757 M * Bertl e.g. ffffffff81537dd8
1357322781 M * ard Then I have to addr2line -e vmlinux ?
1357322794 M * Bertl addr2line -e vmlinux ffffffff81537dd8
1357322807 M * Bertl vmlinux is from the build tree
1357322831 M * ard If I do that (not on that system, but on the build system):
1357322833 M * ard ard@freeze7dev:/mnt/source/kernels/build-d64-i7/l-3.7.1-vs2.3.5.3$ addr2line -e vmlinux ffffffff81537dd8
1357322834 M * ard ??:0
1357322843 M * daniel_hozac did you not build it with DEBUG_INFO?
1357322848 M * Bertl means you are missing debug information
1357322868 M * ard ard@freeze7dev:/mnt/source/kernels/build-d64-i7/l-3.7.1-vs2.3.5.3$ grep DEBUG_INFO .config
1357322868 M * ard # CONFIG_DEBUG_INFO is not set
1357322871 M * ard :-(
1357322872 M * ard meeh
1357322914 M * ard This was a once in a lifetime bug :-(... I will try to setup a server and stress it...
1357322946 A * ard has seen enough once in a lifetime bugs :-)
1357322952 M * ard especially raid1 :-)
1357322976 M * ard cpu bugs 2 times already
1357323084 M * Bertl try to recompile with DEBUG_INFO enabled
1357323093 M * ard Hmmmm, if I recompile with the info, it should be the same, but with extra info
1357323096 M * Bertl then feed in the old addresses, sometimes that works
1357323140 M * daniel_hozac usually the addresses change though.
1357323148 M * ard :-(
1357323210 M * ard Will try anyway :-)
1357323230 M * ard and then reboot test server with that kernel, and look if addresses change
1357323674 M * ard it says: /mnt/source/kernels/build-d64-i7/l-3.7.1-vs2.3.5.3/include/linux/vs_inet.h:87
1357323682 M * ard I hope that is the correct line
1357323758 M * ard it looks sane, because a line earlier is a spin_lock
1357324146 M * Bertl looks good, now please do that for the other addresses in the stack traces as well, and annotate the stack trace with the file/line number starting after l-3.7.1-vs2.3.5.3/
1357324157 M * ard ah, ok :-)
1357324161 M * Bertl i.e. in this case, add include/linux/vs_inet.h:87 on the right side
1357324255 M * ard Well, there are 3 traces hapening (almost?) at once... CPU 2, 0 and 12
1357324618 M * ard http://paste.linux-vserver.org/23158
1357324630 M * ard I've removed the head to just the lockups
1357324712 M * ard so: it's just: include/linux/vs_inet.h:87 + include/net/route.h:283 in 2 cases, and once a single: include/linux/vs_inet.h:87
1357324773 M * ard ow, or you meant the calltrace? :-)
1357324785 M * ard which should make more sense...
1357324875 M * Bertl please do it like this: http://paste.linux-vserver.org/23159
1357324903 M * ard yes, will do that, sorry for my stupidity...
1357324910 M * Bertl and leave the rest of the dump/traces intact, i.e. do not cut anything out or off
1357324937 M * Bertl no problem, it's not easy for folks to get it right
1357324950 M * Bertl btw, I'd appreciate a script which does that automatically
1357324985 M * Bertl so if you feel like coding a little python/perl/whatever or know somebody who would like to do that for the community ...
1357325050 Q * Sith10 Ping timeout: 480 seconds
1357325733 M * ard http://paste.linux-vserver.org/23160
1357325742 M * ard first three call traces done...
1357325753 M * ard the second call trace is interesting I think
1357325936 M * Bertl yes, it is interesting, because it seems that we get called from an interrupt
1357326425 J * Aiken ~Aiken@2001:44b8:2168:1000:21f:d0ff:fed6:d63f
1357326773 M * ard So, to trigger it, I need outgoing as well as incoming connections...
1357327051 M * ard jups...
1357327055 M * ard that's it...
1357327071 M * ard I can ab from the server *and* ab to the server, and it is done...
1357327095 M * Bertl excellent, is the bmx2 involved?
1357327101 M * ard yes
1357327118 M * ard because it is the network card :-)
1357327124 M * Bertl are you able to try that somewhere/somehow without an bmx2?
1357327139 M * ard pfff...
1357327151 M * ard I need to compile one for this workstation I guess?
1357327174 M * Bertl just trying to figure if we trigger a hidden bmx2 bug or have a locking issue
1357327260 M * ard If I read the output correctly it's hard to trigger, because it's using a napi path
1357327290 M * ard (meaning it's already stressed)
1357327312 M * ard I can try an igb but then I have to do some work to set up that system...
1357327325 M * ard and igb's are extremely fast
1357327351 M * ard same kernel but different iron :-(
1357327366 M * Bertl yup, but it might trigger as well, as the napi path is exercised whenever the interrupt detects a second packet
1357327563 M * ard Hmmmm...
1357327661 M * ard I hope it's still triggerable without having any vservers ;-)
1357327685 M * Bertl well, doesn't matter, it happens in Linux-VServer code
1357327980 M * ard meeh... looking for a system, it's either production or unreachable somehow :-(
1357328102 M * ard If all else fails, I've got an atom server @home running 3.7.1 (while experimenting with the network namespaces), I can always use that, but I first have to hook op serial console
1357329293 J * Hollow ~Hollow@217.111.2.114
1357329857 A * ard found a server with a tg3 and an igb...
1357329935 Q * Guy- Ping timeout: 480 seconds
1357330062 J * Guy- ~korn@elan.rulez.org
1357330343 M * ard Nothing so far...
1357330364 M * ard difference is: tg3 instead of bnx2, no vservers (no network namespaces and no network contexts)
1357330368 M * ard just bare metal
1357330385 M * Bertl but with the Linux-VServer patch?
1357330387 A * ard guesses he needs some network contexts to at least make the lookup last
1357330391 M * ard yes, same kernel
1357330410 M * ard but compiled with DEBUG_INFO ;-)
1357330410 M * Bertl okay, yes, you need some contexts
1357330426 M * Bertl i.e. the code is not exercised when there is no context
1357330809 M * Bertl you could enable spinlock debugging on the bnx2 machine and see if that gives some clues
1357330883 M * Bertl but IMHO it isn't a real bug, more an unexpected lock contention
1357330925 M * Bertl (the soft lockup is with 23s extremely low, and I'd expect it to trigger on a 16? cpu machine under load quite easily (even without Linux-VServer)
1357330989 M * ard the problem is that all cpus eventually lock, and the system is not working anymore :-)
1357331009 M * Bertl yes, they lock when the soft lockup is reported
1357331026 M * Bertl that's a 'known' issue, at least known to me for several years now :)
1357331041 M * ard but it doesn't recover...
1357331048 M * Bertl yeah, I know
1357331127 M * ard But you mean that it wouldn't lockup if the soft lockup warning is disabled, or am I acting stupid again? :-)
1357331144 Q * nkukard Quit: Leaving
1357331165 M * Bertl no, that's correct, in my experience, 'no softlockup check' means that the system will recover
1357331397 M * ard well, the system is definitely dead for 20s before it start warning...
1357331408 M * ard and it is dead with a tg3 
1357331427 M * ard no network namespaces, just a simple vserver
1357331450 M * Bertl okay, do you have a trace from that system?
1357331466 M * ard just trrying to remember the magic break incantation
1357331777 M * ard Meeh, I got a No code specified on pastebin now :-(
1357331790 M * ard (the paste.linux-vserver.org)
1357331807 M * Bertl just use a different pastebin 
1357331923 M * ard LOL...
1357331933 M * ard pastebin.ca says: Sorry, an error has occurred. Reason: You did not provide any content. Please try again.
1357331945 M * Bertl :)
1357331958 M * ard what could be in my typescript that triggers that?
1357331962 M * ard ^Z perhaps? :-)
1357332007 M * ard Ah...
1357332010 M * Bertl maybe, maybe remove all special characters and leave just ascii
1357332016 M * ard I got 0 bytes in my serial console output
1357332029 M * ard I mean 0x0 characters.... ^@
1357332048 M * ard http://paste.linux-vserver.org/23162
1357332056 M * ard not annotated yet ...
1357332160 M * ard This one fortunately has only 4 cpus :-)
1357332459 J * isAAAc ~isaaac@breizh-entropy.org
1357332509 M * Bertl looks good, okay, let's try with spinlock debugging (on either system)
1357332731 M * Bertl hmm, are you adding/removing ips in your tests?
1357332827 M * ard no, this is a simple setup
1357332832 M * ard preconfigured
1357332846 M * ard apache2 in a vserver and from the vserver an ab to another system
1357332856 M * ard I do have the IP on dev lo with a mask of 32
1357332862 M * Bertl interesting
1357332905 M * Bertl I think I found a missing lock in ip_v4_find_src() but that can only trigger if you actually add/remove IPs
1357333625 A * ard is cleaning windows
1357333636 M * ard ard@c41901:~$ xlsclients |grep -c xterm
1357333636 M * ard 71
1357334023 M * ard shall I still annotate the dump?
1357334095 M * Bertl doesn't hurt, we might need it lateron
1357334201 A * ard is really impressed that the difference between a bonded interface, and a plain interface is not big
1357334211 M * ard actually none :-)
1357334692 M * ard Hmmmm, I got a single different line: include/linux/vs_inet.h:31...
1357334697 M * ard still busy though...
1357335435 Q * cuba33ci Read error: Connection reset by peer
1357335529 J * cuba33ci ~cuba33ci@114-36-233-2.dynamic.hinet.net
1357335805 M * ard http://paste.linux-vserver.org/23163
1357335870 M * ard on line 164 there is a different vs_inet.h line in the dump
1357336606 M * ard Bertl : I think that this path can only occur if CONFIG_TCP_MD5SIG is enabled...
1357336625 M * ard if you look at line 149 of http://paste.linux-vserver.org/23163
1357336677 M * ard tcp_v4_rcv is in net/ipv4/tcp_ipv4.c, and the only __inet_lookup_listener is done at line 2951 of tcp_ipv4.c
1357336810 M * ard forget what I said...
1357336826 M * ard Still: __inet_lookup_listener is only called from that part
1357336975 M * Bertl in lines 136-138, the annotation overwrites the original info (JFYI)
1357336990 M * Bertl do we have anything with spinlock debugging on?
1357336991 M * ard So, this is the irq time: tcp_v4_rcv in net/ipv4/tcp_ipv4.c is called, that probably calls tcp_v4_send_reset for some reason, tcp_v4_send_reset does MD5SIG checking, which does the __inet_lookup_listener
1357337026 M * Bertl I think we need to make those spinlocks irq safe
1357337042 M * Bertl I didn't expect the stack to be operated from interrupt context
1357337111 M * ard well, it is done in softirq time...
1357337118 M * ard (once it was hard-irq-time)
1357337134 M * Bertl doesn't matter, softirq needs special locking as well
1357337168 M * ard to do spinlock debugging I need to recompile the kernel?
1357337182 M * Bertl the only difference is, that if hardirqs are involved, we need to save the irq state as well
1357337185 J * BenG ~bengreen@cpc29-aztw23-2-0-cust144.18-1.cable.virginmedia.com
1357337199 M * Bertl yes, a compile is required
1357337199 M * ard that was once in 2.4...
1357337225 M * Bertl I'll prepare a patch to modify the locking accordingly
1357337250 M * Bertl i.e. try to trigger the issue with the current patch and spinlock debugging enabled
1357337256 A * ard needs to go to the train at 23:30
1357337283 M * Bertl to pay a visit, or to get home (last chance?)
1357337294 M * ard to get home :-)
1357337301 M * Bertl can you work from home?
1357337318 M * ard not the last chance, but it's the dutch public transport... they tend to just forget about the last train
1357337321 M * ard I can...
1357337333 M * Bertl well, then get home, and we continue from there
1357337341 M * ard ok :-)
1357338201 Q * Ghislain Quit: Leaving.
1357338343 Q * locquest Remote host closed the connection
1357338377 Q * Aiken Remote host closed the connection
1357340278 Q * BenG Quit: I Leave
1357340401 Q * fisted Remote host closed the connection
1357340468 J * fisted ~fisted@xdsl-87-78-191-68.netcologne.de
1357341627 Q * bonbons Quit: Leaving
1357342057 Q * isAAAc Quit: Konversation terminated!
1357342840 A * ard is eating
1357342869 M * Bertl enjoy your meal!
1357343027 M * Bertl http://vserver.13thfloor.at/ExperimentalT/delta-addrlock-fix02.diff
1357343458 M * ard Do you want to have the DEBUG_SPINLOCK output before I apply that patch?
1357343524 M * Bertl either before or after