1172361626 M * Bertl okay, when do you think you can confirm that this fixes your issue?
1172361687 M * daniel_hozac bonbons: well, i'm still not able to reproduce your issue.
1172361691 M * waldi maybe today
1172361719 M * daniel_hozac bonbons: have you tried disabling services to see if that's causing it?
1172361765 M * bonbons I've tried with a second guest that's just running ssh and getty, same effect
1172361777 M * daniel_hozac tried without the getty?
1172361791 M * daniel_hozac (and associated console device)
1172361819 M * bonbons no (but getty mutates to bash as it's from there I call halt -f)
1172362067 M * bonbons same thing with tty removed and getty disable, halt -f from ssh session
1172362278 M * bonbons also same with "vkill --xid 20 -s INT -- 1" from outside, simulating CTRL+ALT+DEL to guest
1172362281 M * daniel_hozac does it happen with vserver ... exec halt -f?
1172362439 M * bonbons same result
1172362468 M * daniel_hozac ok, so there has to be some difference between your guest and my guest, or the host systems.
1172362485 M * daniel_hozac i'm rebooting to 2.6.20-vs2.2.0-pre4 now.
1172362541 M * bonbons I would rather assume host system than guest, especially as I had the same behavior with a SuSE guest, which is definitely not Gentoo
1172362622 M * daniel_hozac what pipe is it that it
1172362625 M * daniel_hozac 's waiting for?
1172362700 M * bonbons cat /tmp/vshelper.<random>/pipe, called by vshelper
1172362723 M * bonbons cat /tmp/vshelper-stop-sync.<random>/pipe, called by vshelper
1172362739 M * bonbons forget the line without -stop-sync
1172362795 M * bonbons if you want to ps auxf, one "in progress", on with "wait for ever"
1172362849 M * daniel_hozac what bash do you have?
1172362898 M * bonbons 3.1.17 (gentoo 3.1_p17)
1172362987 M * daniel_hozac if you add an echo $VSHELPER_STOP_SYNC to vserver.stop, do you get that path?
1172363076 J * virtuoso ~s0t0na@80.253.205.251
1172363081 N * DoberMann DoberMann[ZZZzzz]
1172363095 M * bonbons Process lists: http://paste.linux-vserver.org/1202  (first is while vshelper is doing work, second half is when it reached the wait forever period)
1172363456 M * bonbons yes, it's same file
1172363534 M * daniel_hozac could you make vshelper run vserver ... stop with --debug, and upload the trace?
1172363679 M * bonbons where should I put the debug for it (next to --defaulttty?) line:     spawn $_VSERVER --defaulttty "$VSERVER" stop &
1172363913 M * bonbons here it is: http://homepage.internet.lu/BrunoP/vs-stop.log
1172364279 M * bonbons (that was with reboot -f from inside, but with halt -f it's not much different (except it doesn't try starting the guest again))
1172364499 M * daniel_hozac hmm. are you _sure_ all of your utils are the same version?
1172364534 M * daniel_hozac because that doesn't look like a vserver.stop that knows about the stop sync signal.
1172364545 M * Bertl btw, this is still one of the most common issues
1172364566 M * Bertl yesterday I installed rc3 and it kept complaining about the vshelper mismatch
1172364580 M * Bertl even when I executed the /proc/sys thing
1172364581 M * bonbons well, I removed rm -rf /usr/lib/util-vserver and emerge the rc3 again so there should be no old code left
1172364611 M * Bertl only executing the make distribution-install _again_ fixed it
1172364612 M * daniel_hozac bonbons: so grep doStopSync /usr/lib/util-vserver/vserver.stop returns one line?
1172364653 M * bonbons yep: line 92
1172364658 M * Bertl anyway ... nap attack ... back later I guess ...
1172364665 N * Bertl Bertl_zZ
1172364692 M * daniel_hozac bonbons: and you're sure those are the utils used by your vshelper?
1172364735 M * daniel_hozac because i don't see any of that in the trace.
1172364772 M * bonbons kernel says vshelper is /usr/lib/util-vserver/vshelper so... same for the ps output I posted here: http://paste.linux-vserver.org/1202
1172364961 M * daniel_hozac so if you add echo stopped >> "$VSHELPER_STOP_SYNC" after vshelper.doStopSync, there's no change? does it show in the trace then?
1172364963 M * bonbons here is the tbz of my installed util-vserver: http://homepage.internet.lu/BrunoP/util-vserver-0.30.213_rc3.tbz2
1172365318 M * daniel_hozac well, given that none of the stop sync stuff is showing up in the trace, i'm not sure what to think
1172365319 M * bonbons no change, and does no appear in the trace either... at least no new line and text remains constant except for mktemp filenames
1172365399 M * daniel_hozac what if you add something that doesn't parse/exist to it?
1172365412 M * daniel_hozac does it error then?
1172365603 M * bonbons I've added an inexistent command, but nothing bails out, though the echo Hello World on the line before shows up as expected
1172365611 M * bonbons no difference in the log
1172365659 M * daniel_hozac is stderr going somewhere else?
1172365691 M * daniel_hozac (though the debug output is sent to stderr, IIRC...)
1172365729 M * bonbons should not, I added    " 2>&1 | tee logfile" into vshelper
1172365883 M * daniel_hozac you should comment line 721 in scripts/functions, or stop vshelper from passing --defaulttty.
1172365894 M * daniel_hozac that's why we're not getting any interesting output.
1172366168 M * bonbons some change at the end, added return 0 at beginning of function setDefaultTTY
1172366195 M * daniel_hozac ok, please upload the new trace.
1172366254 M * bonbons here is the new trace: http://homepage.internet.lu/BrunoP/vs-stop-1.log
1172366323 M * daniel_hozac that looks much better.
1172366323 M * bonbons even more output difference if waiting long enough!
1172366383 M * daniel_hozac what does test -p /tmp/vshelper-stop-sync.KYvfGD/pipe && echo yes return?
1172366461 M * bonbons same place as the echo stopped > ...?
1172366497 M * daniel_hozac no, after it's "finished".
1172366510 M * daniel_hozac or hmm.
1172366516 M * daniel_hozac is /tmp a separate filesystem for you?
1172366539 M * daniel_hozac could you try adding /tmp to /etc/vservers/.defaults/namespace-cleanup-skip?
1172366553 M * bonbons yes, it is
1172366805 M * bonbons better results, except for a vwait that still has to timeout
1172366819 M * daniel_hozac what's it waiting for?
1172366822 M * bonbons for the rest the guest stops
1172366881 M * daniel_hozac ok, i've added /tmp to the list of paths to exclude as well...
1172366883 M * bonbons it's a child of vshelper (acording to ps auxf) that waits for the stopping context (/usr/sbin/vwait --timeout 30 --status-fd 3 20)
1172366920 M * daniel_hozac yeah, but what's still running in the guest at that point?
1172366942 M * bonbons halt -f in D state until vshelper returns
1172366957 Q * Piet_ Ping timeout: 480 seconds
1172366976 M * daniel_hozac hmm, well, with plain initstyle, vshelper should return once init has been signalled.
1172367183 M * bonbons and additionally CPU = 100% SYS for 30 seconds
1172367311 M * bonbons but it's (not immediate) child of forked vshelper waiting for it's vserver guest stop
1172367486 M * bonbons somehow the signal does not arrive at guest init, or that one ignores it as until end of those 30 seconds init, bash session and sshd remain alive
1172367671 M * daniel_hozac weird...
1172367708 M * bonbons will check again tomorrow without vanilla utils (that i without our testing hacks) - but I would prefer it doing a vserver name kill instead of vserver name stop for plain init style
1172367750 M * daniel_hozac for plain initstyle, that's what's it's doing, basically.
1172367861 M * bonbons so why is there the vserver name stop in the process tree and why is it waiting 30 seconds before finally killing? Possibly I'm missing something, or the hack altered behavior in wrong direction (omitting adding /tmp to namespace cleaning ignore-list)
1172367922 M * daniel_hozac it's waiting 30 seconds for it to shutdown on its own, as it ought to.
1172367966 M * daniel_hozac if it doesn't for whatever reason, it will be killed.
1172368058 M * bonbons that's when shutdown has not started yet, but when it has nearly finished but waits for virtual power-off an additional signal has not need to be processed anymore, thus nothing more will happen inside the guest
1172368128 M * bonbons especially if the init process calls the reboot() syscall itself and goes into infinite sleeping loop afterwards
1172368155 M * bonbons (thats not the case here, but very possible)
1172368245 M * bonbons anyhow, it's time to get to bed, let's finalize tomorrow :)
1172368250 M * daniel_hozac well, i'm not sure special-casing for that would be worth it.
1172368253 M * daniel_hozac yeah, good idea.
1172368258 M * daniel_hozac good night!
1172368272 M * bonbons thanks, same for you!
1172368281 Q * bonbons Quit: Leaving
1172372986 Q * bXi Ping timeout: 480 seconds
1172376393 Q * ensc Ping timeout: 480 seconds
1172376580 J * ensc ~irc-ensc@p54B4E98D.dip.t-dialin.net
1172379513 M * bored2sleep crud
1172381541 M * bored2sleep well, I got the xen-3.0.4.1 patch from fedora 7, applied to vanilla kernel 2.6.19.5, and then applied 2.2.0-rc14 over that.  everything patched up all pretty.  compiled.  booted the kernel up, which went fine.  
1172381607 M * bored2sleep the first thing I noticed was that reading /proc/virtual/.../limit is still very slow when the context is being heavily loaded (ie when it is going way past its cpu limit).  not a big deal, but still strange
1172381781 M * bored2sleep the second thing I noticed, was that I'm still having the "ps" problem of leaking dentries.
1172381821 M * bored2sleep memory usage is also going up, but that might just be from the dentries
1172381926 M * bored2sleep memory usage went from 55 mb to 105 mb, just by running "for i in seq 1 16000; do ps > /dev/null; done"
1172381964 M * bored2sleep and so far I've gotten dentry from around 100-150 up to over 70,000 
1172382019 M * bored2sleep logging out of ssh does not release the dentries either.
1172382050 M * bored2sleep the good news is that the VM limit/usage didn't go to -1 and crash this time :)
1172382199 M * bored2sleep when anybody wakes up again and wants to help, let me know :)
1172382405 M * bored2sleep btw, dentry count is up over 200,000 now, still climbing.  memory usage is over 210 mb
1172383006 M * bored2sleep if I run on the host "echo 2 > /proc/sys/vm/drop_caches", I have the interesting side effect that I get my memory back, though the dentry rlimit is still showing usage.  I suppose the rlimit is wrong.  still bad, since I have to echo a command to get memory back.
1172384196 J * Aiken ~james@ppp96-171.lns1.bne1.internode.on.net
1172385894 Q * gerrit Quit: Client exiting
1172386843 Q * bon Ping timeout: 480 seconds
1172387973 Q * Aiken Remote host closed the connection
1172388373 J * DoberMann_ ~james@AToulouse-156-1-126-98.w90-30.abo.wanadoo.fr
1172388478 Q * DoberMann[ZZZzzz] Ping timeout: 480 seconds
1172390670 Q * ZLinux Remote host closed the connection
1172390777 J * Aiken ~james@ppp96-171.lns1.bne1.internode.on.net
1172394601 J * DavidS ~david@chello062178045213.16.11.tuwien.teleweb.at
1172396500 N * DoberMann_ DoberMann
1172397830 J * bonbons ~bonbons@83.222.38.146
1172397836 J * dna ~naucki@169-196-dsl.kielnet.net
1172399929 M * daniel_hozac bored2sleep: as i said last time, we don't tag the dentries and as such the accounting will not be correct.
1172400035 M * bonbons morning daniel_hozac!
1172400042 M * daniel_hozac morning bonbons :)
1172400132 M * bonbons have done some new iterations this morning, adding a "vkill --xid <ixd> -s KILL" to doInternalMethod in vshelper just after vshelper.initStopSync does the job as I would expect it for plain init guests
1172400158 M * bonbons would just need not to do it for sysv oder other script-only guest...
1172400237 M * bonbons otherwise I still get my timeout with about 100% CPU for kernel during the 30 seconds (and guest does a "second" shutdown sequence)
1172400304 M * daniel_hozac it shouldn't actually be using the CPU. it's waiting for the context to go away.
1172400366 M * bonbons don't know what kernel-side numbercrunching is done... but at least gkrellm (running on the host) indicates 100% CPU load with really most of it being system
1172400428 M * bonbons and vtop is showing vwait with 100% CPU
1172400492 M * daniel_hozac that's odd.
1172400536 M * bonbons yep, I agree
1172400892 M * daniel_hozac could you try http://people.linux-vserver.org/~dhozac/p/uv/experimental/delta-plaininit-hack02.diff?
1172401106 M * bonbons with that the timeout does not happen, but it's still at 100% while the guest is doing it's internal shutdown sequence
1172401369 M * bonbons but shutdown from outside kill the guest instead of shutting it down properly
1172401411 M * daniel_hozac yeah, it's really a hack.
1172401419 J * sharkjaw ~gab@216-158-6.0503.adsl.tele2.no
1172401490 M * bonbons for vwait I assume the CPU load is possibly caused kernel-side, for the halt -f I would rather think the fix should go into vshelper...
1172401501 Q * sharkjaw 
1172401538 M * daniel_hozac vc_wait_exit doesn't do anything that would incur CPU load though, AFAICT.
1172401613 M * bonbons wanted to check with sysrq + t, but seems my kernel does not have sysrq enabled :(
1172401623 M * daniel_hozac heh.
1172401699 M * bonbons though it's enabled in kernel config... when I do the key sequence the computer just beeps: alt+sysrq+t
1172401760 M * daniel_hozac echo t > /proc/sysrq-trigger?
1172401786 M * bonbons cool, didn't know that one
1172401787 M * daniel_hozac so http://people.linux-vserver.org/~dhozac/p/uv/experimental/delta-plaininit-hack03.diff is more along the lines of what you're thinking, no?
1172402023 M * bonbons yes, but should just happen for guest with real process at pid = 1 as I understood that reboot -f or halt -f is the may of telling host to stop/resatrt guest from inside for sysv init style
1172402056 M * bonbons but shouldn't that hunk be before the spawn vserver ... stop?
1172402058 M * daniel_hozac yes, but after the stop sync has happened, the guest stop process is already done.
1172402128 M * daniel_hozac special-casing based on initstyle in vshelper would be _really_ ugly, so i'd prefer some generic method that should work for all of them...
1172402238 M * bonbons hmm, plain init style is 99% down when vshelper gets triggered, sysv init style is 0% down when vshlper gets triggered, correct?
1172402253 M * daniel_hozac yep.
1172402295 M * bonbons but currently the utils consider the 99% down as 0% down...
1172402297 M * daniel_hozac but by the time the stop sync signal comes, they should both be at 99% down.
1172402358 M * daniel_hozac does hack03 make a difference at all?
1172402394 M * bonbons have not run tested it yet, but I don't expect it to change anything
1172402445 M * daniel_hozac i guess hack01+hack03 should avoid the problems.
1172402653 M * bonbons well hack03 drops the vwait part
1172403218 Q * Aiken Quit: Leaving
1172403348 M * bonbons together the result looks much better
1172403413 M * bonbons stop from outside works, halt from inside shows not trace of init changing runlevel
1172405195 M * derjohn hi, 2.2.0-rc14/pre4 ? not rc14/pre5? 
1172405218 M * derjohn Bertl_zZ, is that the version from you tree ?
1172405222 M * derjohn *your
1172405227 J * ema ~ema@lart.galliera.it
1172405448 P * tzafrir Leaving
1172407157 M * daniel_hozac pre5 hasn't been uploaded yet.
1172407189 M * daniel_hozac bonbons: so hack01+hack03 is the magic combo?
1172407209 M * bonbons it looks like
1172407222 M * bonbons but you should test it with sysv style as well
1172407563 J * chand ~chand@vau75-3-82-224-128-66.fbx.proxad.net
1172411404 M * daniel_hozac seems to work fine here.
1172411583 N * DoberMann DoberMann[PullA]
1172411649 Q * FireEgl Quit: ...
1172411700 M * bonbons just tried with another guest and there it works fine too
1172412346 Q * ema Quit: leaving
1172412660 M * daniel_hozac http://people.linux-vserver.org/~dhozac/t/uv-testing/util-vserver-0.30.213-rc4.tar.bz2
1172413776 M * bonbons that one has an issue with init 6 from inside guest... timeout on shutdown before reboot
1172413788 M * daniel_hozac hmm?
1172413793 M * daniel_hozac -rc4?
1172413798 M * bonbons yes
1172413827 M * daniel_hozac that has hack01 and hack03 applied.
1172413829 M * bonbons again with 100% System CPU usage
1172413877 M * bonbons ok, add hack 3 to restart as well :)
1172413987 M * bonbons that fixes it
1172415334 M * daniel_hozac i'm not sure that's safe for legacy kernels.
1172415431 M * bonbons don't know, I have no such kernel I could test on...
1172415456 M * daniel_hozac would probably require SMP to make it easily triggerable as well.
1172415501 M * daniel_hozac the context could go away completely and the start could've been initiated by the time the killContext is reached.
1172415524 M * daniel_hozac which would cause it to kill the start processes.
1172415528 M * daniel_hozac or am i missing something?
1172415663 M * bonbons I already thought about that kind of race
1172415735 Q * meandtheshell Quit: Leaving.
1172415783 M * bonbons but don't know util-vserver good enough to know if that's possible or not (e.g. what it waits for before starting the guest again)
1172415805 M * daniel_hozac vwait would keep stop around until the context is completely gone.
1172415850 M * daniel_hozac (or until the timeout expires, after which it'll be forcibly killed. if it's still not gone, there must be an unkillable process in there)
1172416057 J * yarihm ~yarihm@84-74-16-109.dclient.hispeed.ch
1172416158 M * daniel_hozac for kernels without legacy support, it shouldn't be a problem as the kernel waits for vshelper to finish before it lets halt/reboot die.
1172416196 J * meandtheshel1 ~markus@85-125-193-102.dynamic.xdsl-line.inode.at
1172416204 M * daniel_hozac but for legacy kernels that run it asynchronously, there's no guarantee we haven't already proceeded to the start stage by the time killContext would be called.
1172416688 M * bonbons and none of the pipes or other lockfiles can help out for that?
1172416723 M * daniel_hozac adding yet another pipe/lock seems a bad idea, IMHO.
1172416756 M * bonbons I would rather think about the existing ones
1172416760 M * daniel_hozac i suppose we could make the stop sync pipe two-way, and have vshelper ack it before vserver.stop proceeds...
1172416845 M * bonbons I'll leave those details up to you as you know the implementation details of util-vserver and I don't
1172419178 N * Bertl_zZ Bertl
1172419188 M * Bertl good morning!
1172419190 M * daniel_hozac morning Bertl!
1172419223 M * derjohn moin Bertl !
1172419259 M * bonbons morning Bertl!
1172419315 M * Bertl daniel_hozac: what's your opinion on the lock change?
1172419485 M * daniel_hozac shouldn't both of the locks already be handled properly?
1172419506 M * daniel_hozac request should be tagged and inc'ed at flock_make_lock called from sys_flock.
1172419570 M * daniel_hozac if new_fl is going to stick around, we inc that after copying it.
1172419684 M * Bertl problem is, in flock_lock_file() we call locks_alloc_lock()
1172419697 M * daniel_hozac right.
1172419719 M * Bertl which doesn't inc/dec anything
1172419736 M * daniel_hozac we inc it if it's going to stick around.
1172419742 M * Bertl now what happens if we have a conflict?
1172419762 M * Bertl                 goto out;
1172419778 M * daniel_hozac where the lock is freed.
1172419786 M * Bertl and the count is decremented :)
1172419843 M * daniel_hozac but the lock isn't tagged yet.
1172419856 M * daniel_hozac so there's no context to attribute the dec to.
1172419914 M * Bertl good point
1172419953 M * Bertl so what we are actually missing is a tagging there
1172419975 M * Bertl otherwise it will have the one present in the cache before
1172419987 M * daniel_hozac well, it's tagged if it's going to be used.
1172419989 M * Bertl which explains most of the strange accounting :)
1172420010 M * Bertl the locks are allocated from the cache
1172420024 M * daniel_hozac ah, right... it's not initialized, just allocated.
1172420189 M * Bertl so we leave the inc where it was, after the copy
1172420201 M * Bertl and initialize the tag to zero, yes?
1172420207 M * daniel_hozac yeah, sounds good to me.
1172420232 M * daniel_hozac (though we initialize it to -1 in other places)
1172420239 M * daniel_hozac (like locks_init_lock)
1172420244 M * Bertl okay, we can do that too :)
1172420340 Q * sladen Ping timeout: 480 seconds
1172420376 J * sladen paul@starsky.19inch.net
1172420510 M * Bertl daniel_hozac: where do we set it to -1, btw?
1172420533 M * Bertl daniel_hozac: and another one: shouldn't we inc() in __setlease() after the copy too?
1172420693 M * daniel_hozac locks_init_lock.
1172420820 M * daniel_hozac and i think so...
1172420842 M * Bertl okay, then I suggest to try to move the inc() into the locks_copy_lock()
1172420871 M * Bertl and remove it from the other places we already do it after the copy
1172420965 M * Bertl hmm, probably not a good idea though
1172420967 M * daniel_hozac __posix_lock_file_conf looks rather complicated...
1172420981 M * Bertl yep, looking at that right now :)
1172421079 M * Bertl as locks and leases work(ed), I'll suggest to test with the initialization change first before we fix bugs which are not there ...
1172421108 M * daniel_hozac yeah, i think that should be sufficient...
1172421124 M * daniel_hozac but it's not.
1172421158 M * daniel_hozac after running chcontext --xid 42 ./test-lease test: vxW: !!! limit: d44de04c[LOCKS,10] = -2 on exit.
1172421173 M * daniel_hozac so i think we do have to add the inc in __setlease.
1172421182 M * Bertl interesting, we couldn't trigger that yesterday, no?
1172421203 M * daniel_hozac or waldi didn't have CONFIG_VSERVER_DEBUG :)
1172421244 M * Bertl ah, good point ... those highly efficient debian kernels, optimized for 486 .... they just can't be compiled with debugging enabled :)
1172421257 M * daniel_hozac hehe.
1172421303 M * Bertl could you paste me the urls of the test tools once again, please?
1172421318 M * daniel_hozac http://people.linux-vserver.org/~dhozac/t/tests/test-lease.c
1172421321 M * daniel_hozac http://people.linux-vserver.org/~dhozac/t/tests/test-flock.c
1172421328 M * Bertl tx
1172421448 M * Bertl what value is F_SETLEASE supposed to have?
1172421479 M * daniel_hozac hmm, no idea. -D_GNU_SOURCE should give it to you.
1172421499 M * Bertl right, tx
1172421719 M * daniel_hozac hmm, odd. adding the inc to __setlease doesn't make it go sub-zero, but the max is still 0.
1172421910 M * Bertl that is probably okay
1172421982 M * daniel_hozac hmm, seems we have the odd proc issues again/still with 2.6.20
1172422046 M * daniel_hozac /proc/virtual/42 exists, but all of the files are empty. no process is still in the context.
1172422110 M * Bertl how do you test/recreate this?
1172422143 M * daniel_hozac chcontext --xid 42 ./test-lease test
1172422152 M * daniel_hozac hmm, better i post to pastebin.
1172422158 M * Bertl okay :)
1172422221 M * daniel_hozac http://paste.linux-vserver.org/1203 is everything i've run since i rebooted.
1172422283 M * daniel_hozac 1020/1021 returned nothing.
1172422427 M * daniel_hozac 1018 killed the sleep process.
1172423257 J * FireEgl Proteus@68.220.222.136
1172423658 M * Bertl hmm, how did the %1 kill the sleep?
1172423669 M * Bertl here it kills the chcontext ...
1172423676 M * daniel_hozac it didn't :)
1172423680 M * daniel_hozac that's why i said 1018 ;)
1172423684 M * daniel_hozac (the vkill 2663)
1172423693 M * Bertl ah, okay
1172423864 Q * FireEgl Quit: ...
1172423892 M * Bertl even if I reproduce the steps here (except for the vkill, which need the context id here) it doesn't give me any hanging proc entries
1172423955 M * Bertl will now check why the max locks is not accounted, but I guess I have an idea
1172424050 M * Bertl yes, we have no actual avail checks in place there
1172424068 M * Bertl and as the max is stored lazily, it doesn't get updated
1172424223 M * daniel_hozac ah.
1172424236 M * daniel_hozac where is it that we're lacking avail checks?
1172424262 M * Bertl well, basically everywhere we vx_locks_inc()
1172424274 M * daniel_hozac i figured the locks_alloc_lock would handle that.
1172424300 M * Bertl it might as well do so right now, the max will be off by one too
1172424311 M * Bertl as it will check _before_ the allocation happens
1172424317 M * daniel_hozac right...
1172424350 M * Bertl I guess we could add a debug option to do the max/min stuff in inc/dec/add
1172424374 M * Bertl it's not really a problem with the current macros
1172424391 M * daniel_hozac i guess it doesn't matter much.
1172424723 M * Bertl so we should see if the changes fix it for waldi then ...
1172424828 M * Bertl am I right that the lock changes apply to 2.6.19 too? or did I miss something there?
1172424886 M * daniel_hozac it looks that way.
1172425625 J * Piet hiddenserv@tor.noreply.org
1172426654 J * cedric ~cedric@rny93-2-82-66-66-30.fbx.proxad.net
1172426750 M * matti Bertl: :)
1172426910 Q * Piet Remote host closed the connection
1172426916 M * cedric Hi all! I'm trying to install vserver on my server, I've compiled 2.6.20 with latest vserver 2.3.0.10 succesffuly, then util-vserver-0.30.212 and I keep getting a "ncontext: vc_net_create(): Invalid argument" error when trying to start a freshly created vserver ( http://paste.linux-vserver.org/1204 ), any idea what's wrong?
1172426931 J * Piet hiddenserv@tor.noreply.org
1172426943 M * cedric forgot to add the vserver has been created with debootstrap method 
1172427103 Q * chand Quit: chand
1172427347 M * daniel_hozac cedric: you forgot to specify --context when you built it.
1172427941 M * bored2sleep daniel_hozac: yes, I remember that you said that dentry account will not be perfect, but it shouldn't be completely insane either, should it?  Something is going wrong with "ps", and I'm using a much more recent/compatible kernel this time
1172428539 M * cedric daniel_hozac: added --context and it works nox! thanks! :)
1172429289 J * chand ~chand@vau75-3-82-224-128-66.fbx.proxad.net
1172429585 Q * dna Read error: Connection reset by peer
1172429609 J * dna ~naucki@169-196-dsl.kielnet.net
1172430406 Q * michal` Ping timeout: 480 seconds
1172430842 J * mmouse ~mmouse@office.haefft.de
1172430918 M * bored2sleep how can I remount /proc inside a guest?
1172430942 M * Bertl bored2sleep: I checked and I do not see this behaviour with a recent mainline kernel
1172430958 M * Bertl bored2sleep: I assume you are using a debian kernel?
1172431060 J * michal` ~michal@www.rsbac.org
1172431170 M * mmouse hello, I keep getting an error when trying to start a specific guest: 'vcontext: execvp("etc/init.d/rc"): No such file or directory' could anybody give me a hint what causes this? other guests are working fine.
1172431180 J * ray6 ~ray@vh5.gcsc2.ray.net
1172431190 M * ray6 reee :)
1172431208 M * Bertl mmouse: does the guest have an etc/init.d/rc? is it executable?
1172431209 M * daniel_hozac mmouse: are you using tagxid?
1172431217 M * Bertl ray6: welcome back! LTNS!
1172431242 M * ray6 Bertl: indeed :) 
1172431260 M * bored2sleep Bertl: I'm using vanilla 2.6.19.5 (just downloaded it myself) with fedora 7's most recent xen-3.0.4.1 patch and vs-2.2.0-rc14 (also just downloaded)
1172431300 M * Bertl bored2sleep: okay, so how do you trigger the DENT misbehaviour?
1172431319 M * bored2sleep Bertl: do you think it could be some interaction with something I've compiled into the kernel (ocfs, gfs, iscsi, aoe, sctp, dccp, etc)?
1172431320 M * mmouse bertl: yes, rc is there and x-bit is set
1172431355 Q * duckx Remote host closed the connection
1172431360 M * Bertl bored2sleep: IIRC, last time it was something like 'chcontext --xid <xid> -- ps auxw' which did leave you with 2 dentries or so?
1172431365 M * bored2sleep I'm not actually using any of those, but this is my test platform to see what gives the best performance/security/stability, so I'd like to play around with those
1172431373 M * mmouse daniel: no tags, just a separate dir for every guest (I hope - how can I check that?)
1172431401 M * daniel_hozac mmouse: cat /proc/mounts
1172431404 M * bored2sleep Bertl: currently I'm testing via ssh, but I tested before with vserver ... enter
1172431425 M * Bertl let's see if you can recreate it with the chcontext, yes?
1172431473 M * mmouse daniel: "/dev/mapper/vg0-lvomail /vhome/vomail xfs rw 0 0"
1172431481 M * mmouse daniel: /vhome/vomail is the guest dir
1172431529 M * Bertl bored2sleep: I'm also interested if the dentry count will stop around 2^17
1172431545 M * Bertl IIRC, last time that was not conclusive either ...
1172431563 M * bored2sleep Bertl: it didn't stop by 260,000 :)
1172431579 M * bored2sleep Bertl: and no, it doesn't happen via chcontext...
1172431668 M * mmouse daniel: I tried to migrate an existing (real) server to vserver, using something as http://linux-vserver.org/util-vserver:Howto_virtualize_an_exisiting_Linux_server
1172431675 M * bored2sleep Bertl: definitely does still happen via vserver ... exec
1172431690 M * bored2sleep and also via ssh
1172431712 M * Bertl okay, try to make the command shorter/lighter
1172431712 M * daniel_hozac mmouse: and how did you do that?
1172431726 M * Bertl bored2sleep: i.e. try to list specific proc entries for example
1172431737 M * Bertl and see if the counter increases
1172431758 M * bored2sleep Bertl: ok, brb then
1172431776 M * daniel_hozac Bertl: hmm, looks like the lock accounting is broken on 2.6.17 as well.
1172431797 M * daniel_hozac (not something we care about, just another data point)
1172431827 M * daniel_hozac i guess that automated test system needs to be created :)
1172431829 M * Bertl daniel_hozac: probably the same fix applies there
1172431838 M * daniel_hozac yeah, i expect it to.
1172431838 M * mmouse daniel: copied /etc/vserver/<name> from an existing (and running) vserver, adjusted the dirs
1172431859 M * daniel_hozac mmouse: try recreating the config using vserver ... build -m skeleton instead...
1172431872 M * mmouse copied the content of the existing real server to the vserver homedir
1172431887 M * daniel_hozac you configured the utils with --with-vrootdir=/vhome, right?
1172431919 M * mmouse copied tmp proc and dev from the working guest to the new one
1172431999 M * bored2sleep Bertl: '/bin/ls /proc/self/fd' does it
1172432012 M * mmouse daniel: good point. I thought so, but vserver-info tells "vserver-Rootdir: /vservers"
1172432069 M * mmouse daniel: could I set a symlink for that?
1172432108 M * Bertl bored2sleep: hmm ... sounds like a mainline issue then ... let me check that out ...
1172432127 N * DoberMann[PullA] DoberMann
1172432147 M * bored2sleep Bertl: I've never had this problem on any other system though... not that I've tried running "ps" 30,000 times on my normal systems, but still...
1172432157 M * daniel_hozac mmouse: just point /etc/vservers/<guest>/vdir to the correct directory.
1172432188 M * Bertl bored2sleep: okay, I can at least recreate that here
1172432196 M * mmouse daniel: that link is right, I checked
1172432284 M * daniel_hozac mmouse: so if you run chroot /etc/vservers/<guest>/vdir/ /bin/ls -l /etc/init.d/rc, you get the listing?
1172432309 M * bored2sleep Bertl: hooray! er, sort of  ;)
1172432348 M * Bertl but as I said, we do not touch code there, so it seems the kernel is leaking dentries there (or at least not cleaning them up in a timely fashion :)
1172432377 M * bored2sleep Bertl: hmm, strange, ls actually looks worse than ps... it adds two for each /proc/*/fd you do, even if you repeat the same one over and over again in the same command (ls /proc/self/fd /proc/self/fd)
1172432422 M * Bertl what about using /proc/<pid>/fd with an existing context?
1172432431 M * Bertl s/context/pid/
1172432452 M * Bertl i.e. find a process which does exist, and try with that one
1172432582 M * mmouse daniel: no... apparently /etc/init.d/rc is there, but I missed some other files while copying the guest. argh, sorry, I'll retry...
1172432626 M * Bertl like the bash or libs :)
1172432747 M * bored2sleep hmm, definitely happens in the host context as well (don't know where I can see dentries there, but I can definitely see memory usage go up)
1172432847 M * Bertl yes, as I said, I consider it a mainline issue
1172432862 M * Bertl but maybe we can figure where they are lost and why
1172432905 M * daniel_hozac are they lost? are dentries really expected to vanish as soon as nothing is using it anymore?
1172432917 M * daniel_hozac i thought the dentries were supposed to be cached.
1172432927 Q * yarihm Quit: Leaving
1172433013 M * bored2sleep daniel_hozac: you think "ls /proc/1/fd /proc/1/fd" should cache two more dentries than "ls /proc/1/fd"?
1172433036 M * bored2sleep btw, doesn't happen on centos4 base kernel (first other kernel I've tested)
1172433074 M * daniel_hozac CentOS4 uses a really patched 2.6.9 kernel, so that's not too interesting.
1172433083 M * Bertl yes, I suspect a subtle change when mainline got rid of the inode numbers
1172433120 J * ema ~ema@lart.galliera.it
1172433347 Q * cedric Quit: Quitte
1172433348 M * bored2sleep daniel_hozac: it just means it probably hasn't always been that way.  It was the quickest thing I could think to check
1172433417 M * bored2sleep it also doesn't happen in my normal xen0 kernel (2.6.16.33-xen0)
1172433490 M * bored2sleep it also doesn't happen in my normal xenU kernel (2.6.16.33-xenU)
1172433514 M * bored2sleep but it definitely does happen in 2.6.16.33-xenUvs (which was the kernel I was using when all this started)
1172433595 M * bored2sleep that suggests (to me anyway) that it is being caused by linux-vserver somehow.  if Bertl couldn't reproduce it, I'd say maybe my config changed in some unexpected way at some point, but it would be surprising if that happened for both of us
1172433669 M * Bertl interesting theory
1172433701 M * Bertl that would mean that it isn't caused by the inode changes
1172433714 M * Bertl (which would be even more interesting)
1172433718 M * Bertl checking now
1172433720 M * bored2sleep btw, if you want to test this really fast (or crash your system much faster, hehe) you can run "for i in `seq 0 999`;do echo -n '/proc/self/fd ';done>/testproc" followed by "free -m; bash -c 'time ls $(< /testproc ) > /dev/null'; free -m" (should see memory increase steadily by about 3mb each time 
1172433754 M * bored2sleep well, you don't need the 'bash -c' or "time " parts of that.  those were just part of my testing
1172433772 M * bored2sleep so "free -m; ls $(< /testproc ) >/dev/null; free -m"
1172433837 M * mmouse daniel: anyway, that wasn't the problem, it seems. For some reason I can't run binaries from this guest.
1172433893 M * bored2sleep the command also increases used dentries by 2000, if there's some place you can see that
1172433924 M * mmouse daniel: the guest is a debian woody (no comments please ;-) Are there any restrictions about the guest os?
1172433972 M * bored2sleep this command is much more "efficient" than the ps command.  good thought Bertl!
1172434204 M * bored2sleep btw, this actually is a pretty bad thing.  it means that an unprivileged user in a guest context can crash the guest and/or increase host memory usage until the guest hits the dentry rlimit
1172434295 M * Bertl yes, but that is equally true on any linux system
1172434318 M * Bertl even worse, an unprivilidged user on a normal Linux system can take down the machine via dentries quite easily
1172434396 M * bored2sleep Bertl: well, not entirely.  I can't reproduce this on centos or on my regular xen kernels
1172434402 M * Bertl but as I said, I think this is a general issue, although your previous statement would imply that it is a Linux-VServer specific issue
1172434428 M * Bertl bored2sleep: there are simple ways to keep the dentries around, so trust me, you _can_ do that on centos too
1172434444 M * bored2sleep I have specifically tested a xenU kernel with and without vserver patch, and I this particular bug doesn't normally happen
1172434476 M * bored2sleep Bertl: ok, but normally at least there is some accounting? no?
1172434483 M * Bertl nope
1172434486 M * bored2sleep :/
1172434524 M * bored2sleep how would you normally keep dentries around on a regular kernel? (since this bug doesn't help us there)
1172434530 M * Bertl but now that we know where we have to look, we are already half the way
1172434638 M * Bertl bored2sleep: a simple method is to build deep directory trees, while removing the ancestor
1172434810 M * Bertl bored2sleep: can you easily revert the Linux-VServer patch and verify with the same kernel and config?
1172434870 M * Bertl if it _is_ Linux-VServer related, I have a good idea where to look
1172434961 M * bored2sleep Bertl: I have two kernels that are identical except for the vs patch and the kernel config, which I've already tested.  I should be able to rebuild the latest kernel with just the vs patch removed, but same config
1172435131 M * Bertl daniel_hozac: we currently have a d_drop(dentry); in pid_revalidate()
1172435247 M * Bertl i.e. we put the dentry unconditional
1172435322 M * bored2sleep well, I've tested under low memory conditions, and it looks like the kernel is killing the dentries at least.  that means you can't bring down the host (though it is still using up *almost* all the memory, pushing other files out of cache.  at least it doesn't go to swap)
1172435385 M * bored2sleep it could definitely be worse, though for an actually loaded server, I'd bet this would ruin performance
1172435423 M * bored2sleep the guest is pretty messed up now though (can't load shared libraries and things, because of the rlimit)
1172435788 Q * shedi Quit: Leaving
1172436045 M * mmouse daniel: just for info - looks like it works now. I just made some stupid errors when using rsync to copy the guest. sorry.
1172436062 Q * mmouse 
1172436127 M * Bertl bored2sleep: okay, I guess I can fix this  .. i.e. bring it back to the 'original' behaviour
1172436216 M * Bertl bored2sleep: in fs/proc/base.c, line ~1052
1172436225 M * Bertl change the d_drop(dentry);
1172436229 M * Bertl to
1172436235 M * Bertl if (!ret)
1172436240 M * Bertl     d_drop(dentry);
1172436280 M * bored2sleep will try, as soon as my build system gets done with its current build
1172436357 M * Bertl this will not prevent that you 'lose' dentries with each 'ls /proc/self/fd' you run, but it will limit it to one dentry per pid, which is not really lost, just cached
1172436393 M * Bertl and which should be the current behaviour on vanilla too
1172436506 M * bored2sleep what I'm compiling now is vanilla+xenU, so I can test to see if this really eats memory there.  the configuration is exactly identical (except for vserver specific configuration, which was removed automatically)
1172436601 M * bored2sleep I'm also running my web benchmark right now to see how it affects performance if another guest is being obnoxious this way
1172436653 M * bored2sleep heh, ok, that was bad :)
1172436684 M * bored2sleep it is oopsing like crazy
1172436695 M * bored2sleep in the host, not the guests
1172436699 M * Bertl hmm?
1172436755 M * bored2sleep I'm still waiting for vanilla+xenU to compile, so I was testing the vanilla+xenU+vs-2.2.0-rc14 kernel as it was before
1172436756 M * Bertl you get a kernel panic on the host?
1172436780 M * bored2sleep if it stops scrolling any time, I'll let you know what it says ;)
1172436802 M * Bertl yes, please upload/photograph/copy ...
1172436816 M * bored2sleep heh, might need to be photograph at this point!
1172436868 M * bored2sleep although the benchmark showed only about 30% throughput drop... I'm not sure if that didnt finish before my console got sad
1172436919 M * bored2sleep the "good" news is that the guests are still responding normally... I have no idea what is happening on the host
1172436972 M * bored2sleep going to see if ssh is still up on the host, maybe I can get in that way and see something other than scrolling messages and beeps
1172437008 M * bored2sleep lol, nope, happens in ssh too... well, should be able to get the output from that at least
1172437035 J * Aiken ~james@ppp96-171.lns1.bne1.internode.on.net
1172437047 M * Bertl morning Aiken!
1172437080 M * bored2sleep unfortunately, I'm not sure how I'll get back the *first* oops, but I can post some random ones.
1172437105 M * Bertl dmesg and/or the /var/log/messages could contain that
1172437180 M * bored2sleep yeah, I can't access either right now, via console or ssh.  guess I should just kill the system and restart, hope I get something there
1172437541 M * bored2sleep heh, 91000 lines of oopses in /var/log/messages before I killed it
1172437561 M * Bertl let's get a few interesting ones uploaded then
1172437573 M * Bertl from the beginning if possible :)
1172437759 M * bored2sleep ok, I did the first two, in vserver pastebin 1206
1172437811 M * bored2sleep looks like they are basically the same (hard to tell in a small terminal window
1172437942 M * bored2sleep so, my test case was the earlier kernel, with one guest running "while sleep 1; do ls $(< /testproc ) $(< /testproc ) $(< /testproc ) $(< /testproc ) $(< /testproc ); done" and the other running apache, xinetd, and mysql, and getting lots of hits via apache.  when I'm not being abusive in the one guest, the other guest is completely fine
1172437958 M * bored2sleep now, let me go apply the small patch ;)
1172437992 J * chand_ ~chand@vau75-3-82-224-128-66.fbx.proxad.net
1172437993 Q * chand Read error: Connection reset by peer
1172438052 M * bored2sleep ok, modified two lines in function pid_revalidate
1172438085 M * bored2sleep looks like it is rebuilding ok
1172438249 Q * Aiken Quit: Leaving
1172438253 M * Bertl okay, nap attack ... back a little later ...
1172438260 N * Bertl Bertl_zZ
1172438375 M * bored2sleep ok
1172438410 M * bored2sleep btw, just tested vanilla+xenU, doesn't show the dentry caching behaviour.  this is exactly the same kernel, minus the vserver patch.
1172438662 M * bored2sleep ok, now I've loaded up vanilla+xenU+vserver-2.2.0-rc14+pid_revalidate, and I'm not seeing this behaviour. 
1172438724 M * bored2sleep this is exactly the same as the earlier kernel, except for that two line patch (same config, same xen build directory, everything)
1172438734 M * bored2sleep but now it works fine
1172438993 M * bored2sleep btw, I suppose I should have mentioned, before when I was getting the oopses, there were also a few errors showing up in the apache/php/mysql benchmark.  about 0.44% of the transactions were erroring out.  probably due to httpd being killed.  that also isn't happening now (because the bad guest isn't increasing dentry/memory usage)
1172439158 M * bored2sleep heh, I was just worried for a second when I sshed into the guest and did "ps" and the dentry count went up slightly, but the difference was that it only went up the first time.  the second and later times, it stayed the same.  this seems like correct behaviour
1172439877 Q * chand_ Quit: chand_
1172439979 Q * bonbons Quit: Leaving
1172440026 M * daniel_hozac Bertl_zZ: just to make sure i understand the problem, the d_drop will release the /proc/<pid> dentry, but leave the /proc/<pid>/* ones around?
1172440338 M * bored2sleep well, my reading is that if get_proc_task(dentry->d_inode) == NULL, then something bad happens if you immediately d_drop(dentry).  Unfortunately, I don't know what that actually means.
1172440391 M * bored2sleep er, wait, other way around.  != NULL
1172440479 M * bored2sleep if get_proc_task(dentry->d_inode) != NULL, then you don't d_drop(dentry)
1172440536 M * bored2sleep I'm not sure why doing d_drop(dentry) if a task exists uses up more memory, but I don't have any idea what any of these things do
1172440629 M * bored2sleep I'm just guessing, but maybe if you d_drop dentry to early, it gets removed from some structure, but can't be freed because a task is still using it... but if there is no task using it, then it gets freed normally.  something like that.
1172440839 M * bored2sleep like I notice that proc_flush_task does shrink_dcache_parent(dentry);d_drop(dentry);dput(dentry);, but only if dentry is not null.  maybe doing d_drop at the early time causes those other functions to not get run
1172440898 Q * infowolfe Read error: Connection reset by peer
1172440913 M * bored2sleep daniel_hozac: does that make any actual sense?
1172441432 J * infowolfe ~infowolfe@c-67-164-195-129.hsd1.ut.comcast.net
1172441631 M * daniel_hozac bored2sleep: IMHO d_drop just removes the dentry from the cache, without any further considerations. i'm not sure, but i imagine that if that dentry has children, those are now inaccessible too, so new cache entries have to be created each time.
1172441719 J * yarihm ~yarihm@84-75-123-221.dclient.hispeed.ch
1172442334 J * Aiken ~james@ppp96-171.lns1.bne1.internode.on.net
1172442438 N * DoberMann DoberMann[ZZZzzz]
1172442797 Q * yarihm Quit: Leaving
1172443270 M * bored2sleep daniel_hozac: yep, looks like shrink_dcache_parent is supposed to do that.  if we d_drop the dcache before we run shrink_dcache_parent, the children will be recached on the next run.
1172443302 M * bored2sleep normally the kernel lets proc_flush_task take care of both
1172443532 Q * meandtheshel1 Quit: Leaving.
1172444157 Q * dna Quit: Verlassend
1172444597 N * Bertl_zZ Bertl
1172444601 M * Bertl back now ...
1172444614 M * Bertl bored2sleep: so I assume that fixes it for you too?
1172444666 M * bored2sleep yup
1172444775 M * Bertl excellent, I'm thinking about using a soft limit (for the dentries) to trigger a cache walk, and explicitely getting rid of dentries at some point, but that is future stuff (i.e. not for 2.2.0 right now)
1172445073 M * bored2sleep yeah, that sounds like a good idea.  not so urgent, as long as it isn't quite so trivial to create problems.  btw, how else might you make trouble with dentries?  you were saying something about making deep directories?
1172445299 M * Bertl basically you create a directory, change into it, and remove the parent
1172445311 M * Bertl then you start all over ...
1172445594 M * bored2sleep ... I don't seem to be able to do that
1172445608 M * bored2sleep at least, not from the shell.
1172445850 M * bored2sleep it doesn't look like libc allows it either... 
1172445881 M * bored2sleep except maybe if you are root, and I'm not sure of that either.  definitely not for an unprivileged user
1172446080 M * Bertl hehe, you think you cannot create a directory in your home
1172446136 M * Bertl and then remove it, once you went down far enough?
1172446489 J * Aiken_ ~james@ppp96-171.lns1.bne1.internode.on.net
1172446585 M * bored2sleep Bertl: you can't remove a non-empty directory
1172446618 M * bored2sleep and if you remove your cwd, you can't then create new directories in it
1172446622 Q * DavidS Quit: Leaving.
1172446733 J * ZLinux ~ZLinux@88.213.57.11
1172446806 Q * Aiken Ping timeout: 480 seconds
1172446908 Q * cehteh Ping timeout: 480 seconds
1172446941 J * dreamind ~dreamind@C2107.campino.wh.tu-darmstadt.de
1172447245 M * bored2sleep looks  like the syscall itself won't let you remove a non-empty directory, according to my reading, so that isn't possible at all
1172447308 M * bored2sleep also, I tried just leaving the directories there, but it gets really slow, probably during pathwalks.  too slow to do much testing on, but I doubt it is a major concern
1172447678 J * test3 ~steady@ip70-160-26-12.hr.hr.cox.net