1221178019 Q * dowdle Remote host closed the connection 1221178187 J * filius ~filius@201.229.148.210 1221178218 M * filius hola 1221178270 Q * filius Quit: Saliendo 1221180042 Q * ViRUS Quit: If there is Artificial Intelligence, then there's bound to be some artificial stupidity. (Thomas Edison) 1221180678 Q * BrunoXLambert Quit: Leaving 1221190667 Q * derjohn_foo Ping timeout: 480 seconds 1221198783 J * derjohn_mob ~aj@e180205047.adsl.alicedsl.de 1221201860 J * ghislainocfs21 ~Ghislain@LPuteaux-151-41-11-129.w217-128.abo.wanadoo.fr 1221203039 J * ntrs ~ntrs@77.29.76.159 1221203351 J * cryptronic ~oli@p4FD2FE40.dip.t-dialin.net 1221203626 Q * ctrix Ping timeout: 480 seconds 1221203839 J * dna ~dna@73-224-dsl.kielnet.net 1221203928 Q * derjohn_mob Ping timeout: 480 seconds 1221204071 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221204597 J * larsivi ~larsivi@85.221.53.194 1221204718 J * hijacker ~hijacker@213.91.163.5 1221204728 M * hijacker hey fellows 1221204743 M * hijacker i have recently implemented some disk limits on a vserver 1221204752 M * hijacker however upon each reboot of the host server 1221204771 N * DoberMann[ZZZzzz] DoberMann 1221204875 M * hijacker the disk limits get removed and the guest seems some 400kb occupied space on its root partition 1221204883 M * hijacker *sees 1221204904 M * hijacker then i do: vdlimit --xid vs_name --remove /vservers 1221204916 M * hijacker chxid -URx -c vs_name /vservers/vs_name 1221204921 M * hijacker and all is fine 1221204962 M * daniel_hozac so you've got a /etc/vservers//dlimits entry, right? 1221204992 M * hijacker aye 1221205034 M * hijacker on the host: /dev/hda6 on /data type reiserfs (rw,tagxid) 1221205087 M * daniel_hozac so you're running 2.6.22.19-vs2.2.0.7? 1221205095 M * hijacker nope 1221205099 M * hijacker one sec 1221205111 M * hijacker uname -a 1221205111 M * hijacker Linux nevir 2.6.25.11-vs2.3.0.34.14nevir #3 Fri Sep 5 17:46:47 EEST 2008 i686 GNU/Linux 1221205175 M * daniel_hozac tagxid has been removed from 2.3... 1221205228 M * hijacker hmm 1221205242 M * hijacker so how do I accomplish disk limits then ? 1221205353 M * daniel_hozac -o tag. 1221205375 M * daniel_hozac do all testfs-tests succeed on reiserfs on your kernel? 1221205395 M * daniel_hozac i.e. -xyz 1221205493 M * derjohn interesting, tagxid has been removed? if I migrate from 2.x to 2.3 and habe disk limits set in /etc/vserver/.... will they be respected after the upgrade ? 1221205559 M * hijacker alright 1221205567 M * hijacker so daniel_hozac I shall remount with -o tag 1221205578 Q * ctrix Ping timeout: 480 seconds 1221205580 M * hijacker then I shall start testfs-tests with -xyz and will let you know the results 1221205599 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221205616 N * Bertl_zZ Bertl 1221205620 M * Bertl morning folks! 1221205623 M * daniel_hozac derjohn: tagxid is tag now. 1221205637 M * Bertl hijacker: you cannot 'remount' with tag 1221205656 M * derjohn Bertl, that early up ? things change with time :) 1221205656 M * Bertl hijacker: also, testing with testfs.sh will require an unused partition 1221205676 M * Bertl hijacker: i.e. do not run it on your existing partition, all data will be erased 1221205683 M * derjohn daniel_hozac, so it's renamed, not removed ? 1221205690 M * daniel_hozac yes. 1221205698 M * daniel_hozac tagxid has been removed, tag has been added. 1221205700 M * ktwilight_ hijacker, http://linux-vserver.org/Installation_on_Linux_2.6#Testing_your_setup that's a howto for the test 1221205701 M * Bertl derjohn: the early bird gets eaten by the worm ... or how was that? :) 1221205718 M * Bertl derjohn: tagxid has been obsoleted in 2.1 IIRC 1221205730 M * Bertl derjohn: i.e. it was in 2.2 for legacy support 1221205748 M * derjohn Bertl, that was with cthulhu IIRC ;) 1221205820 M * hijacker lool, thanks Bertl , I will not run the testfs.sh on my vserver partition then ;-) 1221205850 M * hijacker i have just unmounted and then remounted the partition with -o tag 1221205861 M * hijacker disk limits seem fine within the guest now 1221206070 J * wenchien ~wenchien@59.105.176.102 1221206090 M * hijacker i shall report if dlimits fail to properly account after host reboot 1221206098 M * hijacker thanks for the help all 1221206191 M * Bertl you're welcome! 1221206254 Q * ctrix Ping timeout: 480 seconds 1221206288 Q * quinq Quit: Quitte 1221206328 J * quinq ~quinq@quinq.eu.org 1221206348 Q * quinq Remote host closed the connection 1221206462 J * quinq ~quinq@quinq.eu.org 1221206616 Q * ntrs Ping timeout: 480 seconds 1221206791 J * blathijs ~matthijs@drsnuggles.stderr.nl 1221206863 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221206952 M * derjohn Bertl, does patch-2.6.26.5-vs2.3.0.35.5.diff also contain the ocfs2 patches? and: is the ocfs2 issue solved already ? 1221207024 M * daniel_hozac it does. 1221207056 M * Bertl derjohn: yep 1221207290 M * derjohn daniel_hozac, what abiut the pidspace stuff for the .206 utils ? 1221207349 Q * ctrix Ping timeout: 480 seconds 1221207369 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221207376 M * daniel_hozac not merged yet. 1221207390 M * Bertl will be soon in devel 1221207397 M * Bertl s/devel/experimental/ 1221207517 Q * hparker Quit: Read error: 104 (Peer reset by connection) 1221208213 Q * cryptronic Quit: Leaving. 1221212682 J * zbyniu ~zbyniu@host13-188.crowley.pl 1221212804 J * tramjoe_merin ~tramjoe@193.41.238.151 1221213217 M * awk do you know this server i'm on 1221213222 M * awk | server : saturn.oftc.net (Brisbane, Australia) 1221213235 M * awk is the most stable ircd server I have been on, I was idle for 42 days and never time out, etc.. 1221213554 M * Bertl congrats! 1221213588 M * sid3windr 42 days 1221213590 M * sid3windr perfect number! 1221213630 M * Bertl daniel_hozac: a friend of mine just reported that --bind mounts of files vie the guest's config fstab give a strange error (secure-mount: chdir(path): not a directory) 1221213638 M * Bertl *via 1221213776 Q * zbyniu Quit: leaving 1221215421 J * cryptronic ~oli@p4FD2FE40.dip.t-dialin.net 1221215649 M * cryptronic is there somewhere a doc howto limit a guest on only one cpu in a multicore system? 1221215785 M * cehteh google -> great flower page 1221215907 Q * lucrus Ping timeout: 480 seconds 1221215955 Q * quinq Remote host closed the connection 1221216491 J * lilalinux ~plasma@80.69.41.3 1221217285 Q * nkukard Quit: Leaving 1221217287 J * kir ~kir@swsoft-msk-nat.sw.ru 1221217378 P * kir 1221219367 Q * FireEgl Ping timeout: 480 seconds 1221219845 Q * ctrix Read error: Connection reset by peer 1221219845 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221219974 J * lucrus ~papo@host-84-223-100-175.cust-adsl.tiscali.it 1221220154 M * Bertl welcome lucrus! 1221220314 J * FireEgl FireEgl@173.16.9.10 1221220874 J * yarihm ~yarihm@whitehead2.nine.ch 1221221108 J * loddafnir ~mike@chello080108111137.5.11.univie.teleweb.at 1221221151 N * zbyniu_ zbyniu 1221221712 Q * Aiken Remote host closed the connection 1221221720 J * michal ~michal@www.rsbac.org 1221221859 Q * laptopnenolod Quit: this sh scripting is the language of the future 1221222354 Q * pisco_ Ping timeout: 480 seconds 1221222380 J * pisco ~pisco@tor.noreply.org 1221224516 J * hparker ~hparker@linux.homershut.net 1221226738 Q * dna Ping timeout: 480 seconds 1221227144 Q * lilalinux Remote host closed the connection 1221227457 Q * larsivi Quit: Konversation terminated! 1221228622 Q * esa Ping timeout: 480 seconds 1221229460 J * Walex ~Walex@82-69-39-138.dsl.in-addr.zen.co.uk 1221230270 M * cryptronic I have a question concering scheduling: what is the difference between fill-rate and fill-rate2 didn't get behind that with the CPU Schduling page 1221230329 M * Bertl hehe, it's quite simple, the scheduler has two sets of interval/fillrate pairs 1221230370 M * Bertl the one is in use when the machine is actively working on stuff (i.e. scheduling guests) 1221230387 M * Bertl the second pair is used when the machine would go idle (no runable guests) 1221230403 M * Bertl and time is artificially advanced 1221230413 M * Bertl (for those guests using idle-time) 1221230488 M * cryptronic so guests needing s lot of cpu on the same host are best set to use idle-time 1221230512 M * Bertl not necessarily ... 1221230521 M * daniel_hozac Bertl: it's a known bug. 1221230537 M * Bertl the mount? 1221230544 M * daniel_hozac yes. 1221230554 M * daniel_hozac file bind mounts are not supported right now. 1221230557 M * Bertl okay, fixed in trunk or planned? 1221230562 M * daniel_hozac planned. 1221230566 M * Bertl okay, np 1221230630 J * lilalinux ~plasma@80.69.41.3 1221230899 M * cryptronic must all guests on the same host have the same values for interval? 1221231050 J * dna ~dna@73-224-dsl.kielnet.net 1221231058 M * Bertl cryptronic: nope, not at all 1221231254 J * esa bip@62.123.8.23 1221231797 Q * dna Ping timeout: 480 seconds 1221232051 J * ntrs ~ntrs@77.29.64.33 1221232173 Q * FireEgl Read error: Operation timed out 1221233208 J * FireEgl FireEgl@adsl-220-214-234.bhm.bellsouth.net 1221233286 J * BrunoXLambert ~Bruno@modemcable188.10-70-69.static.videotron.ca 1221233651 J * quinq ~quinq@quinq.eu.org 1221233687 Q * lilalinux Remote host closed the connection 1221234157 Q * lucrus Ping timeout: 480 seconds 1221234194 J * lucrus ~papo@host-84-223-100-175.cust-adsl.tiscali.it 1221234772 Q * FireEgl Ping timeout: 480 seconds 1221235279 Q * tramjoe_merin Remote host closed the connection 1221235394 J * FireEgl FireEgl@173.16.9.10 1221236482 J * bonbons ~bonbons@2001:960:7ab:0:2c0:9fff:fe2d:39d 1221236504 J * dowdle ~dowdle@scott.coe.montana.edu 1221236515 Q * dowdle Remote host closed the connection 1221237799 J * blues_ blues@adr210.neoplus.adsl.tpnet.pl 1221237799 Q * ctrix Read error: Connection reset by peer 1221237801 J * ctrix ~8__D@81-174-32-147.static.ngi.it 1221237919 Q * blues Ping timeout: 480 seconds 1221238660 J * dowdle ~dowdle@scott.coe.montana.edu 1221239324 J * nkukard ~nkukard@dsl-245-119-234.telkomadsl.co.za 1221240190 J * edlinuxguru ~edlinuxgu@216.223.13.163 1221240459 M * edlinuxguru Hey all. You may have noticed a thread I was working on trying to get linux Vserver installed to a glusterFS shared two node disk. I have actually made great progress. I have a vserver running on a shared server. THe problem is it is painfully slow, and this is not the fault of the network disk. Does anyone have any clever diagnostics I can run to see what the hold up is 1221240503 M * daniel_hozac so running bonnie++ on it shows decent performance? 1221240531 M * edlinuxguru If I run from my system the performance is good....[root@jtg113 glustertest]# time find . > /dev/null 1221240531 M * edlinuxguru real 0m19.221s 1221240531 M * edlinuxguru user 0m0.018s 1221240531 M * edlinuxguru sys 0m0.042s 1221240532 M * daniel_hozac does it implement mmap? 1221240558 M * edlinuxguru But even vserver enter takes like 2 minutes 1221240576 M * daniel_hozac so the find was on the host, right? 1221241159 M * Bertl glusterFS is mostly userspace, no? 1221241323 J * larsivi ~larsivi@169.80-202-217.nextgentel.com 1221241342 M * ktwilight_ glusterfs itself has performance issues 1221241348 M * ktwilight_ edlinuxguru, which version are you using? 1221243135 N * DoberMann DoberMann[PullA] 1221243904 M * edlinuxguru I am using the latest stable 1221243931 M * edlinuxguru 1.3.12.. I had to patch it myself because it forced the nodev .. option but that is a minor change 1221243965 M * edlinuxguru It is not entierly a gluster performance issue If i run a test inside and outside a vserver the results are drastically different 1221243982 M * Bertl show us 1221244034 M * edlinuxguru Even vserver enter takes about 2 minutes 1221244064 M * Bertl what about a chroot /path/to/guest ? 1221244147 M * edlinuxguru That only seems to take a second or two. 1221244165 M * Bertl okay, try with vnamespace enter then 1221244241 M * ktwilight_ vserver kernel version? 1221244251 M * ktwilight_ though maybe it's irrelevant. 1221244254 M * edlinuxguru time vnamespace -e 40007 ls 1221244254 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221244254 M * edlinuxguru real 0m0.004s 1221244254 M * edlinuxguru user 0m0.000s 1221244254 M * edlinuxguru sys 0m0.002s 1221244259 M * edlinuxguru That works much better 1221244269 M * edlinuxguru uname -a 1221244269 M * edlinuxguru Linux jtg111.jointhegrid.com 2.6.22.19-vs2.3.0.34.1 #1 SMP Mon Mar 17 02:33:42 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux 1221244296 M * daniel_hozac time vserver ... exec ls 1221244328 M * edlinuxguru [root@jtg111 ~]# time vserver glustertest exec ls 1221244328 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221244328 M * edlinuxguru real 0m10.335s 1221244328 M * edlinuxguru user 0m0.051s 1221244328 M * edlinuxguru sys 0m0.029s 1221244349 M * Bertl time vnamespace -e 40007 ls /path/to/guest ? 1221244399 M * ktwilight_ hm i get very different results from you. 1221244407 M * edlinuxguru [root@jtg111 ~]# time vnamespace -e 40007 ls /opt/jtg/mnt/shared/vservers/glustertest/ 1221244407 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221244407 M * edlinuxguru real 0m0.004s 1221244407 M * edlinuxguru user 0m0.000s 1221244407 M * edlinuxguru sys 0m0.002s 1221244425 M * ktwilight_ http://rafb.net/p/sTd7qq49.html 1221244462 M * ktwilight_ is your vserver inside a glusterfs mount? 1221244478 M * edlinuxguru Yes root is a glusterfs mount 1221244491 M * edlinuxguru I patched gluster to allow 'dev' 1221244500 M * edlinuxguru It was forcing the nodev option 1221244515 M * ktwilight_ it could be 1.3's performance issue on small files 1221244547 M * ktwilight_ though am not sure why vnamespace is quicker. 1221244557 M * Loki|muh which network id should I use when using chbind to restrict a process to a single ip on the host? 1221244571 M * Bertl Loki|muh: a free one :) 1221244572 M * daniel_hozac Loki|muh: whichever you want that's not already taken in the 2-49151 range. 1221244608 M * edlinuxguru That is what I find weird. I know the gluster filesystem is not slow. 1221244617 M * ktwilight_ edlinuxguru, um, it is for small files. 1221244628 M * ktwilight_ and when i mean small files, i mean < 5MB 1221244634 M * Bertl edlinuxguru: patched how? 1221244646 M * ktwilight_ if you're mounting your entire vserver from a glusterfs, it can be a problem. 1221244674 M * Loki|muh how can I tell which ones are free? counting the # of vservers running on the host? and using anyone > # + 2? :-) 1221244688 M * Bertl ls /proc/virtnet 1221244707 M * edlinuxguru This is how it was patched I ---Currently I have committed a patch to solve this issue, as glusterfs explicitly specifies, 'dev, suid' options. But with whatever release you are trying, you may have to edit the sourecode a bit and compile to get it working 1221244707 M * edlinuxguru open xlators/mount/fuse/src/fuse-bridge.c, 1221244707 M * edlinuxguru goto function 'init()' 1221244707 M * edlinuxguru add 1221244707 M * edlinuxguru "-o", "dev", "-o", "suid", 1221244709 M * edlinuxguru to fuse_argv, and increment fuse_argc by 4 (from whatever value it has there). 1221244709 M * edlinuxguru compile glusterfs, install, and you should be good to solve this problem. 1221244743 M * edlinuxguru See kt here is the problem the gluster speed outside of the vserver is very different then the speed inside 1221244773 M * Loki|muh ah, thanks alot! :-) 1221244788 M * Bertl edlinuxguru: try to enter the network namespace and do the ls there 1221244838 M * fb edlinuxguru: does it support noatime option? if so, try to remount with it 1221244854 M * edlinuxguru 'i probably have to patch in noatime as well' 1221244883 M * edlinuxguru This is from mtab 1221244885 M * edlinuxguru glusterfs /opt/jtg/mnt/shared fuse rw,allow_other,default_permissions,max_read=1048576 0 0 1221245013 J * dna ~dna@73-224-dsl.kielnet.net 1221245064 M * edlinuxguru Sorry Bertl, command does not compute. You went a little over my head with that last request. Can you be more specific on how to issue the command from the network namespace 1221245107 M * daniel_hozac chbind --nid vnamespace -e ls 1221245131 M * Bertl prepend time :) 1221245164 M * daniel_hozac well, that's implied. 1221245167 M * edlinuxguru [root@jtg111 ~]# time chbind --nid 40007 vnamespace -e 40007 ls 1221245167 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221245167 M * edlinuxguru real 0m0.018s 1221245167 M * edlinuxguru user 0m0.013s 1221245167 M * edlinuxguru sys 0m0.005s 1221245200 M * Bertl so whatever take that long, it seems not to be directly Linux-VServer related 1221245213 M * fb maybe it's some issue with config? 1221245220 M * fb some strange limits or so? 1221245232 M * Bertl daniel_hozac: any ideas what to try from the enter sequence? 1221245266 M * edlinuxguru That is what I am trying to figure out... why 'time vserver glustertest exec ls' takes 10 seconds 1221245287 M * Loki|muh it seems to be a silly question, but would it be complicated to have the host always restricted to a specific ip? e.g. spawning init with chbind? ;-) 1221245313 M * Bertl Loki|muh: nope, not at all, but that will also restrict your guests to that IP :) 1221245339 M * ktwilight_ would be interesting to see what glsuterfs debug log spit out when executing these 3 different commands and compare what it's doing. 1221245414 M * Loki|muh Bertl: arghs ;-) 1221245432 M * daniel_hozac edlinuxguru: if you run with --debug, i assume the only thing that takes time is the long command, right? 1221245489 M * edlinuxguru Everything inside the vserver seems to take a long time... bash-3.2# time true 1221245489 M * edlinuxguru real 0m0.833s 1221245489 M * edlinuxguru user 0m0.000s 1221245489 M * edlinuxguru sys 0m0.000s 1221245553 M * daniel_hozac which brings me back to one of my initial questions. does it support mmap? 1221245581 M * edlinuxguru I dont know. Ill read up on it. 1221245599 M * ktwilight_ fuse doesn't support mmap IIRC 1221245613 M * daniel_hozac the hook is there, i know that much. 1221245625 M * daniel_hozac whether it's used or not is a different issue. 1221245632 M * Bertl edlinuxguru: try the --debug run, and see what command takes longer than expected 1221245668 Q * dna Ping timeout: 480 seconds 1221245693 M * edlinuxguru ++ exec /usr/bin/ionice -c2 -n0 /bin/nice -n 0 /usr/sbin/chbind --silent --secure --nid 40007 --ip 10.0.0.2/24 /usr/lib64/util-vserver/exec-ulimit /etc/vservers/glustertest/ulimits /usr/sbin/vnamespace --enter 40007 -- /usr/sbin/vspace --enter 40007 -- /usr/sbin/vtag --migrate --tag 40007 --silent -- /usr/sbin/vcontext --silent --migrate --chroot --xid 40007 --uid 0 -- ls 1221245711 M * edlinuxguru That seemed to be the bulk of it. 1221245729 N * DoberMann[PullA] DoberMann 1221245832 M * fb Bertl: maybe ionice + glusterfs issue? 1221245864 M * edlinuxguru I did not even try anything special with ionice. Its default 1221245978 M * edlinuxguru Also if I remove ionice from the above command it takes 10 seconds and then kicks me off the ssh :) 1221246004 M * edlinuxguru After producing the results 1221246038 M * ktwilight_ FWIW, am running bonnie++ on a glsuterfs mount inside of vserver, so far it's running well, no kicks no lags on other guests, everytihng is good so far. 1221246073 M * edlinuxguru I think it might have something to do with using gluster as the root file systems 1221246079 M * ktwilight_ on the other end, i see bonnie++ doing it's job on another vserver 1221246099 M * ktwilight_ i'd seoriusly doubt glusterfs before anything else. after all, it's sitll under heavy development ;) 1221246120 M * edlinuxguru For example if mount that same partition inside another vserver its fine 1221246123 M * ktwilight_ edlinuxguru, have you tried the latest tla checkout? or even the prereleases? 1221246136 M * ktwilight_ compile it with bdb, it might help. 1221246221 M * edlinuxguru I am all for giving those things ago, but I am looking at something related in how the two hook together, my disk performance is not bad. But using it as a root filesystem is lagging 1221246302 M * daniel_hozac i don't know what happens if it doesn't do mmap... 1221246325 M * daniel_hozac but emulating mmap using read/write would mean really bad performance, especially for executing things. 1221246400 M * edlinuxguru I follow what you are saying Daniel, but if I mounted the same partition at say /data and kept root on the local hardrive would that rule out the mmap issue ? 1221246453 M * daniel_hozac if you got the same bad performance? sure. 1221246502 M * edlinuxguru Ok i will test that 1221246542 Q * yarihm Quit: Leaving 1221246997 M * edlinuxguru Ok I did a bind mount /opt/jtg/mnt/shared/vservers/glustertest/ /data none bind 1221247029 M * edlinuxguru And it is not an issue real 0m0.003s 1221247120 M * edlinuxguru So maybe a bind mount go between can fix it 1221247289 M * Bertl sounds like a bug in fuse to me 1221247314 J * dennis ~dennis@dslb-084-059-211-166.pools.arcor-ip.net 1221247318 M * Bertl maybe it purges the buffers when you are outside the guest, and then it has to refetch everything 1221247320 P * dennis 1221247355 J * dennis ~dennis@dslb-084-059-211-166.pools.arcor-ip.net 1221247363 P * dennis 1221247371 J * dennis ~dennis@dslb-084-059-211-166.pools.arcor-ip.net 1221247374 P * dennis 1221247722 M * edlinuxguru Not that I am asking anyone to fix but, if anyone wants to take a closer I can grant access. 1221247885 M * Bertl try the extra long command from above with 'time' 1221247912 M * daniel_hozac in the guest's root directory on gluster. 1221247921 M * Bertl if that accounts for most of the 10 seconds, try removing the ionice 1221247963 M * Bertl if that doesn't fix it, remove the nice too 1221247968 M * Bertl then the ulimit command 1221248248 M * edlinuxguru time /usr/sbin/chbind --silent --secure --nid 40007 --ip 10.0.0.2/24 /usr/lib64/util-vserver/exec-ulimit /etc/vservers/glustertest/ulimits /usr/sbin/vnamespace --enter 40007 -- /usr/sbin/vspace --enter 40007 -- /usr/sbin/vtag --migrate --tag 40007 --silent -- /usr/sbin/vcontext --silent --migrate --chroot --xid 40007 --uid 0 -- ls 1221248248 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221248248 M * edlinuxguru real 0m10.187s 1221248248 M * edlinuxguru user 0m0.012s 1221248248 M * edlinuxguru sys 0m0.008s 1221248367 M * daniel_hozac replace ls with strace -fFr ls. 1221248541 M * edlinuxguru It is a couple of pages of output do you want me to post it all? 1221248577 M * edlinuxguru The left colume is the times correct? 1221248711 M * edlinuxguru 0.000000 execve("/bin/ls", ["ls"], [/* 19 vars */]) = 0 1221248711 M * edlinuxguru 4.454384 brk(0) = 0x615000 1221248711 M * edlinuxguru 0.001647 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ac04c927000 1221248711 M * edlinuxguru 0.000068 mmap(0x2b0a479e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x2b0a479e5000 1221248711 M * edlinuxguru 2.704024 mmap(0x2b0a479e7000, 13168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b0a479e7000 1221248711 M * edlinuxguru 0.000173 close(3) 1221248713 M * edlinuxguru 0.000069 mmap(0x2b0a4802f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x2b0a4802f000 1221248713 M * edlinuxguru 2.352279 mmap(0x2b0a48030000, 40256, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b0a48030000 1221248715 M * edlinuxguru 0.000174 close(3) 1221248752 M * Bertl looks like mmaps :) 1221248948 M * edlinuxguru That was like the first thing Daniel said :) 1221248948 M * Bertl could you trace the same ls on the host and upload to paste.linux-vserver.org (both traces actually) 1221249039 Q * Walex Remote host closed the connection 1221249153 M * edlinuxguru http://paste.linux-vserver.org/12466 1221249299 M * edlinuxguru http://paste.linux-vserver.org/12467 1221249433 M * Bertl hmm .. is the /tmp mounted inside the guest? 1221249539 M * Bertl I somewhat think the issue is kind-of self made with the dev hack you added 1221249573 M * Bertl edlinuxguru: for a test, try --bind mounting /dev from a normal filesystem into the guest 1221249607 M * edlinuxguru I needed to hack in dev otherwise the vserver would not start up at all. 1221249636 M * Bertl well, you could have mounted the /dev in the first place 1221249646 M * Bertl that probably would have made it start perfectly fine 1221249672 M * daniel_hozac those mmaps don't look filesystem related. 1221249693 M * Bertl mmap MAP_ANONYMOUS uses /dev/zero for mapping 1221249729 M * daniel_hozac surely the kernel has an internal reference to the cdev, and doesn't actually use the filesystem for that? 1221249942 M * Bertl probably 1221249994 M * edlinuxguru Is the the proper syntax is the vserver fstab /dev /dev none bind 0 0 ? 1221250008 M * daniel_hozac bind,dev 1221250102 M * edlinuxguru Starting with same slow speed. 1221250183 M * Bertl okay, let's remove more parts from the command sequence 1221250198 M * Bertl first, let's kill the tagging 1221250370 J * ntrs_ ~ntrs@77.29.76.93 1221250380 M * edlinuxguru time /usr/sbin/chbind --silent --secure --nid 40007 --ip 10.0.0.2/24 /usr/lib64/util-vserver/exec-ulimit /etc/vservers/glustertest/ulimits /usr/sbin/vnamespace --enter 40007 -- /usr/sbin/vspace --enter 40007 -- /usr/sbin/vcontext --silent --migrate --chroot --xid 40007 --uid 0 -- ls > /dev/null ---Same speed 1221250426 M * Bertl okay, now let's kill the ulimit part 1221250437 M * Bertl /usr/lib64/util-vserver/exec-ulimit /etc/vservers/glustertest/ulimits 1221250517 M * edlinuxguru Same result. One thing I want to mention the IP 10.0.0.2/24 is not routable no gateway. I just assigned a random one 1221250536 M * Bertl np, we'll remove the /usr/sbin/chbind now 1221250553 M * Bertl i.e. start with /usr/sbin/vnamespace ... 1221250663 Q * BrunoXLambert Quit: Leaving 1221250791 Q * ntrs Ping timeout: 480 seconds 1221250983 Q * edlinuxguru 1221251031 J * edlinuxguru ~edlinuxgu@216.223.13.163 1221251115 M * edlinuxguru That big command is almost down to nothing now 1221251136 M * Bertl same without the chbind? 1221251140 M * edlinuxguru usr/sbin/vspace --enter 40007 -- /usr/sbin/vcontext --silent --migrate --chroot --xid 40007 --uid 0 -- ls --still the same 1221251169 M * Bertl good :) 1221251199 M * edlinuxguru Good for you maybe :) 1221251201 M * Bertl daniel_hozac: suggestions for further reduction? 1221251844 M * Bertl try without the --chroot and --uid 0 1221252053 Q * bonbons Quit: Leaving 1221252196 M * edlinuxguru time /usr/sbin/vspace --enter 40007 -- /usr/sbin/vcontext --silent --migrate --xid 40007 --uid 0 -- ls 1221252196 M * edlinuxguru bin boot dev etc home lib lib64 media mnt opt proc root sbin selinux srv sys tmp usr var 1221252196 M * edlinuxguru real 0m0.003s 1221252196 M * edlinuxguru user 0m0.000s 1221252196 M * edlinuxguru sys 0m0.001s 1221252208 M * edlinuxguru That did it 1221252226 M * Bertl well, you are listing a different filesystem now 1221252243 M * Bertl adjust the ls to list the guest filesystem 1221252249 M * daniel_hozac no, it should be the same. 1221252258 M * daniel_hozac the cwd would need to be the guest's root in order for --chroot to work. 1221252293 M * Bertl ahprobably was, because you told him so (some time ago) 1221252302 M * Bertl -ah 1221252329 M * daniel_hozac right. 1221252339 M * edlinuxguru It is the mount. I confirmed it 1221252394 M * Bertl hmm? please elaborate 1221252420 M * edlinuxguru I put a file in the gluster mount. I am not listing a local directory. That is the vservers root. 1221252462 M * Bertl okay, so whatever the --chroot does, the fuse/glusterfs doesn't like it? 1221252466 M * daniel_hozac so the difference is definitely caused by executing things in the gluster filesystem. 1221252509 M * Bertl the question now is, why doesn't it hurt when edlinuxguru uses a normal chroot? 1221252539 M * Bertl edlinuxguru: you did try that, yes? 1221252822 M * edlinuxguru No I did not try that. It has the same performance 1221252845 M * Bertl so you get the same bad performance on the host without Linux-VServer stuff? 1221252948 M * edlinuxguru No if i run an ls of the mounted director it is fine .004 s. 1221252965 M * Bertl wait, you are running the ls from the host 1221252968 M * daniel_hozac as is chroot /path/to/guest ls? 1221252990 M * Bertl of course that is fine, you need to chroot into the guest dir to 'run it on the filesystem' 1221253041 M * edlinuxguru time chroot /opt/jtg/mnt/shared/vservers/glustertest/ ls --- 12s 1221253054 M * edlinuxguru time ls /opt/jtg/mnt/shared/vservers/glustertest/ .004s 1221253078 M * Bertl here you go, that _is_ the glusterfs performance 1221253120 M * Bertl now try the same thing on a non Linux-VServer kernel, and you are likely to get the same values 1221253156 M * edlinuxguru But the speed is fine outside a jail. 1221253178 M * Bertl look, in the first case, you are executing the ls from the 'jail' 1221253204 M * Bertl in the second case, you execute the ls from outside the filesystem (the host ls, with is _not_ on a glusterfs) 1221253218 M * edlinuxguru No that is on the gluster gs 1221253243 M * Bertl your host's / is on glusterfs? 1221253292 M * edlinuxguru No. It is not. 1221253323 M * Bertl okay, one explanation and an example how to 'make it as fast' inside a guest 1221253329 M * ktwilight_ i think he meant /opt/.../glustertest is on glusterfs 1221253355 M * Bertl edlinuxguru: when you do 'ls', then that loads the executable, and a bunch of libraries 1221253384 M * Bertl in the first case, that executable and the libraries come from the clusterfs (before any 'ls' action is taken) 1221253404 M * Bertl in the second case, they come from your host filesystem, whatever that is 1221253415 M * Bertl now, how to make that as fast inside a guest? 1221253433 M * Bertl simple: create a guest on your 'normal' filesystem 1221253446 M * Bertl and put the glusterfs part inside the guest's /opt 1221253462 M * Bertl then enter the guest, and do 'ls /opt' 1221253469 M * Bertl it will be as fast as on the host 1221253552 M * Bertl unfortunately glusterfs seems to take about 2.5 seconds to mmap an executable or library 1221253571 Q * loddafnir Remote host closed the connection 1221253596 M * edlinuxguru That could be more due to how it is deployed on the switchport then its actual performance 1221253601 M * edlinuxguru But I follow you 1221253616 M * ktwilight_ i'd file a bug to glusterfs :) 1221253648 M * edlinuxguru You are saying that since none of the libraries are local they all have to be fetched and nmapped each operation 1221253827 M * Bertl yes, and obviously that is a) slow, and b) doesn't get cached, for whatever reason 1221253871 M * ktwilight_ well, maybe enabling some performance (glusterfs) translators may help 1221253889 M * ktwilight_ s/may/would 1221253923 M * edlinuxguru I have iothreads/writebehind/read ahead 1221253948 M * edlinuxguru But they only have the default options 1221253960 M * ktwilight_ start tweaking 'em ;) 1221253967 M * ktwilight_ also, the order of which matters too 1221254057 M * edlinuxguru kt. What do you think would make this so much slower then say NFS. I have mounted directories and binary files across nfs and got better perofmance 1221254081 M * ktwilight_ for one, nfs doesn't use fuse. 1221254105 M * ktwilight_ 2) glusterfs is still young, so lots of work still needs to be done. 1221254116 M * ktwilight_ who knows, maybe 1.4 is a heck load difference to 1.3 1221254124 M * ktwilight_ 1.4 has some major architectural changes 1221254141 M * ktwilight_ s/some/some significant/ 1221254300 M * edlinuxguru Everyone thanks for the help. Sorry it was a deadend. Ill add something to the wiki. Maybe some troubleshooting tips. 1221254311 M * Bertl you're welcome! 1221254393 M * edlinuxguru Really it does work. By setup is two servers on an old switch. Maybe with gig-e transport two nodes on gluster does have decent performance 1221254418 M * ktwilight_ possible too. gigE is recommended for glusterfs 1221254431 J * Aiken ~Aiken@ppp118-208-124-125.lns4.bne4.internode.on.net 1221254678 M * edlinuxguru kt my email is irc @ gmail drop me a line. We can compare notes 1221254768 M * edlinuxguru is my irc @ gmail 1221254774 M * ktwilight_ edlinuxguru, sure 1221254788 M * ktwilight_ am also in #gluster in freenode 1221256134 J * laptopnenolod ~nenolod@pool-71-241-220-223.ptldme.east.verizon.net 1221256391 M * edlinuxguru I am getting ready for an IRC slap....I just loaded the gluster io cache translater. Its workable now 1221256403 M * edlinuxguru http://www.gluster.org/docs/index.php/GlusterFS_Translators_v1.3#IO-Cache_Translator 1221256871 J * doener_ ~doener@i577BA46A.versanet.de 1221256974 Q * doener Ping timeout: 480 seconds 1221257010 M * edlinuxguru Initially it has to load the cache. but once the cache gets loaded with the system libraries the speed is good 1221257164 M * Bertl excellent! 1221257546 Q * ntrs_ Ping timeout: 480 seconds 1221257659 M * edlinuxguru My goal here is to tie Gluster/ Vserver/ and Linux HA together. Make a really tight system that would compete with something like Vmware Infrastructure 1221257698 M * Bertl well, I'd consider OCFS2 or GFS(2) for that 1221257708 M * edlinuxguru I tried OCFS2 I did not like it 1221257738 M * Bertl but performance and stability are probably superior 1221257744 Q * cryptronic Quit: Leaving. 1221257752 M * edlinuxguru I tried OCFS2 over DRBD active/active 1221257758 M * edlinuxguru I had lots of problems 1221257762 M * Bertl but you said VMware(tm) so that is maybe not your goal :) 1221257821 M * edlinuxguru I found that DRDB requires partitioning. And your are essentially running 3 programs. DRBD HA OCFS2 1221257827 M * edlinuxguru That is a lot of configuration 1221257865 M * edlinuxguru Really in this case we dont need OCFS2 because I dont need an active/active filesystem. I need mirroring, but the vserver will only run on one side. 1221257913 M * Bertl then why use glusterfs? 1221257933 M * Bertl drbd and a failover setup with e.g. ext3 should do nicely 1221257997 M * edlinuxguru Yes true. I had bad luck with DRBD...First it did not compile. Then i had to update kernel. Often times it would lockup and needed a hard power cycle. Maybe it was just my bad luck 1221258036 M * edlinuxguru Gluster and FUSE are a little more userspace. I think that makes them more flexible. 1221258134 M * Bertl but unfortunatley also less performant 1221258298 M * edlinuxguru If memory serves DRBD has a device number in the kernel. Have you ever tried patching DRBD into the vserver kernel? 1221258319 M * Bertl it works fine with Linux-VServer 1221258561 M * edlinuxguru I have talked to people that have had good luck with it and bad. I did feal that OCFS2, heartbeat, and DRBD together most junior level people could not manage it. 1221258753 M * daniel_hozac it really isn't very hard. 1221258829 M * edlinuxguru Daniel, I agree to some extend. Imagine explaining over the phone in a crisis how to use the cibadmin commands :) 1221259079 M * daniel_hozac that'd be why you document things... 1221259370 M * edlinuxguru The HA and DRBD guys are doing a great job but the documentation has to be a little stronger on their end. 1221259397 M * edlinuxguru Even the newest release has wrong information right on the 5th line in the man page. 1221259400 M * daniel_hozac i meant _your_ documentation. 1221259411 M * daniel_hozac documenting _your_ setup. 1221259671 M * edlinuxguru I understand what you are saying I am playing devils advocate slightly. 1221259710 M * edlinuxguru Thanks again for the help everyone. 1221259730 M * Bertl np, keep us updated 1221259991 M * fb Bertl: btw, i never saw any project with such good support from the authors 1221260192 Q * edlinuxguru Ping timeout: 480 seconds 1221261771 Q * dowdle Remote host closed the connection 1221261856 M * Bertl fb: tx! 1221261866 M * Bertl off to bed now ... have a good one everyone! 1221261873 N * Bertl Bertl_zZ 1221262789 Q * quinq Remote host closed the connection 1221262935 J * quinq ~quinq@quinq.eu.org 1221263371 J * balbir_ ~balbir@bi01p1.co.us.ibm.com 1221263722 J * dowdle ~dowdle@scott.coe.montana.edu 1221263938 Q * balbir_ Read error: Operation timed out