1442188831 J * fstd ~fstd@xdsl-84-44-223-4.netcologne.de 1442189881 M * Bertl off to bed now ... have a good one everyone! 1442189888 N * Bertl Bertl_zZ 1442209325 Q * derjohn_mob Ping timeout: 480 seconds 1442211957 J * Ghislain ~aqueos@adsl1.aqueos.com 1442212765 M * snixor Hi! Is there any way to control disk cache inside vserver guest? I'd like to configure "vm.min_free_kbytes", but I'm not allowed - "permission denied". Any workarounds? 1442213685 J * opuk ~kupo@37.123.166.100 1442214149 J * nikolayK ~nkichukov@199.91.137.248 1442214202 J * opuk_ ~kupo@37.123.166.100 1442214312 Q * opuk Ping timeout: 480 seconds 1442214537 Q * gnarface Quit: Leaving 1442215956 M * Ghislain inside the guest it would not be a good idea to allow it to modify system limits 1442217483 M * snixor Are there other ways to limit page caching inside a guest? I have two mongo and mysql inside a single guest. Mongo will use all available memory for cache and when I issue a memory intensive sql query on a mysql the host starts dropping caches. This results in a temporary higher latency... 1442217583 M * snixor I would like to eliminate the need of dropping page caches 1442217929 M * Ghislain on my side yes i use cgroup for this 1442217955 M * Ghislain with cgroup you can limit the memory and the total memory of a guest ( mem+swap) 1442218414 M * snixor yes... and when you start a mongo process inside your guest, it will eat up all the memory and use it as cache 1442218428 M * snixor while this is good, it isn't good if you need some of it back 1442218453 M * snixor droping those caches results in some latency issues 1442218507 M * snixor i would like to prevent mongo from eating up all available ram for disk cache 1442218554 M * snixor i would like to keep 1G free, so when I need it, the system wount need to drop cache 1442218971 M * Ghislain look at memory.memsw.limit_in_bytes and memory.limit_in_bytes 1442218993 M * id is there no way to limit mongo itself? 1442219004 M * Ghislain you set them in /etc/vservers/xxxx/cgroup/memory.memsw.limit_in_bytes 1442219213 M * Ghislain you cannot differenciate between cache and other, you limit GLOBAL ram for the group/guest 1442219387 J * gnarface ~gnarface@108-227-52-42.lightspeed.irvnca.sbcglobal.net 1442221824 M * snixor id: I think not - http://stackoverflow.com/questions/4468873/how-to-release-the-caching-which-is-used-by-mongodb/4482465#4482465 1442221843 M * snixor Ghislain: swap and memory limits are set, but this doesn't help 1442221901 M * snixor it looks like it caused the problem.. I have now removed the cgroups. Now i am waiting if the system will honor hosts dirty_pages ratio 1442221927 M * snixor and prevent mongo from eating all the ram for page cache 1442222911 N * Bertl_zZ Bertl 1442222915 M * Bertl morning folks! 1442223138 M * nikolayK morning Bertl 1442223310 M * Bertl snixor: the problem is, that caches cannot be associated with a specific guest 1442223329 M * Bertl thus they cannot be properly limited on a per guest basis 1442223588 M * snixor thank you 1442223775 M * Bertl the host limit should work though, within the capabilities of the kernel 1442224602 N * jrayhawk_ jrayhawk 1442231041 M * Ghislain bertl: did you found any leads about the mysterious crash ? we see only a rise in disk activity of one of the guest beforehand 1442231055 M * Ghislain but nothing we can relate to 1442231082 M * Ghislain i am asking a complete hardware replacement to get rid of this possibility 1442231499 M * Bertl I have no idea so far 1442231506 Q * Aiken Remote host closed the connection 1442231642 M * Ghislain usualy i like mysteries ;p 1442231668 M * Ghislain well i will work the hardware part tso i be sure of it 1442231712 M * Bertl can't hurt to rule that out 1442231858 M * gnarface Ghislain: question, is bind running on that guest? 1442231895 M * gnarface bind9, specifically? 1442231912 M * gnarface (or rather, was it, at the time of the crash) 1442231960 Q * fstd Remote host closed the connection 1442231985 M * Ghislain hum guess so virtualmin runs it by default* 1442232031 J * fstd ~fstd@xdsl-87-78-180-5.netcologne.de 1442232039 M * gnarface a curious correlation 1442232090 M * Ghislain let me check 1442232237 M * Ghislain well bind9 is not suposed to crash the kernel in anyway. It seems not running and no trace in the logs so it does not seems to run on this guest but some of the guest runs it, not the one that get increased activity just before the crash 1442232274 M * gnarface hmm. well the one crash i managed to cause after sorting out all the crashes due to my flaky hardware, seemed to do with saturating disk/io 1442232306 M * gnarface but the guest i was copying data to was also running bind9 for the domain which i was using in the transfer command 1442232312 M * Ghislain compared to my old solaris day linux seems quite brittle when load increase 1442232381 M * gnarface could be related, but i copied the data differently (i think i scp'd an already created archive instead of piping stdout of tar to a sshfs mount) and the problem did not reoccur 1442232399 M * gnarface i forget what gave me the notion that it could be related to bind9 being on the same guest 1442232408 M * gnarface anyway probably just a red herring 1442232466 M * Ghislain i have an explosion of the system cpu timeat 11h00 then at 11h50 the server panic 1442232483 M * gnarface i think shortly after about kernel 3.14 or 3.16 general quality of the mainline kernel seems to have certainly fallen 1442232508 M * gnarface around 4.0 all kinds of bad stuff started happening 1442232533 M * Ghislain well i run 3.4.xxx here so 1442232545 M * gnarface ah, that's probably wise i think 1442232550 M * Ghislain what bertl will call JURASICK kernels ! ;p 1442232806 M * Ghislain i am looking at cpu at that time, context switch fall and interrupts/s double in a short time 1442233070 Q * Bertl Ping timeout: 480 seconds 1442234960 J * Gremble ~Gremble@cpc29-aztw22-2-0-cust128.18-1.cable.virginm.net 1442236554 M * Ghislain the crash is still kernel BUG at fs/dcache.c:537 1442236992 M * gnarface interesting 1442237004 M * gnarface unfortunately debugging kernel code is way beyond my area of expertise 1442237034 M * gnarface think it might matter what filesystem you use? 1442237501 M * Ghislain ext4 1442238316 M * gnarface wow there was ext4 support in 3.4? 1442238332 N * ensc Guest1670 1442238335 M * gnarface well ext4 has had a real bad year 1442238341 J * ensc ~irc-ensc@p54ADEE2F.dip0.t-ipconnect.de 1442238750 Q * Guest1670 Ping timeout: 480 seconds 1442239144 M * Ghislain well the major bug they found was in raid not in ext4 so not so bad 1442239153 J * derjohn_mob ~aj@nl12x.mullvad.net 1442239322 M * gnarface yea and not until later kernels, i thought 1442239353 M * gnarface though, i've had some severe data loss bugs (irretrievable filesystem corruptions) on ext4 with cheap flash cards 1442239361 M * gnarface NOT using raid 1442239406 M * gnarface i never really looked further into it. i switched filesystems and it stopped happening 1442239756 M * Ghislain do you changed the cheap cards also to not cheap one ? :) 1442240373 M * gnarface no actually, i kept them. they all still work fine, but they're slow 1442240387 M * gnarface actually with the exception of 1 1442240392 M * gnarface one can no longer be written to 1442240403 M * gnarface which might represent hardware failure 1442240435 M * gnarface however the manufacturers actually said some linux filesystems can trip that write-protect hardware lock and freeze it, without there being a total failure of all the free blocks 1442240440 M * gnarface they even released a tool to undo it 1442240453 M * gnarface but it only supports their newer devices :( 1442240475 M * gnarface (that brand is Patriot, btw, and i wouldn't recommend their cheap crap at any discount) 1442240485 M * Ghislain ^^ 1442240491 M * gnarface it can still read fine though 1442240500 M * gnarface i've got a hybrid iso live image on it that i still use in fact 1442240525 M * gnarface the other ones experienced the exact same behavior, even down to the write-protect error being triggered 1442240549 M * gnarface but in all those cases, reformatting them completely worked 1442240580 M * gnarface one actually corrupted repeatedly under ext4 1442240603 M * gnarface i had to reformat it i think 3 times before i eventually gave up on ext4 for all of them and switched filesystems 1442240676 M * gnarface the issue may have had something to do with swapping them around between different machines using different versions of ext4, but of course like i said i'm no kernel programmer so i don't really know. 1442240718 M * gnarface these are assorted SD, SDHC and USB flash keys in question 1442240730 M * gnarface so they were all also exposed to different ports/readers 1442240767 M * gnarface but it did seem as though the slower ones were the ones more prone to corruption quickly, so i certainly expect that low quality, or at least low speed, was a contributing factor 1442242468 J * Bertl herbert@IRC.13thfloor.at 1442242700 Q * derjohn_mob Ping timeout: 480 seconds 1442243658 J * derjohn_mob ~aj@2001:6f8:1337:0:fdab:8a87:9315:b487 1442243699 J * opuk ~kupo@37.123.166.100 1442243804 Q * opuk_ Ping timeout: 480 seconds 1442246078 J * derjohn_mobi ~aj@b2b-94-79-172-98.unitymedia.biz 1442246415 J * opuk_ ~kupo@37.123.166.100 1442246464 Q * derjohn_mob Ping timeout: 480 seconds 1442246522 Q * opuk Ping timeout: 480 seconds 1442246851 M * Ghislain from what i can read of the code in vserver the dentry is modified only to add a counter and inc/dec it so.. 1442246864 M * Ghislain nothign very bug prone, also i do not use dentry limits at all 1442246986 M * Bertl for a test, you can remove the accounting there (i.e. the relevant patch hunks) 1442247064 M * Ghislain is there a way to make patch remove the patch from dcache or must i do it manualy ? 1442247099 M * Bertl -R or --reverse 1442247115 Q * nikolayK Quit: Leaving 1442247178 M * Ghislain patch -R fs/dcache.c patch-3.4.106-vs2.3.3.9.diff => Unreversed patch detected! Ignore -R? 1442247398 M * Ghislain hum let me reload a fresh copy of the source and the lastest patch for 3.4.Xxx 1442247421 M * Ghislain is the delta-dlimit-fix03.diff relevant for 3.4 ? 1442247579 M * Ghislain dam i got a reject 1442247587 M * Ghislain Hunk #2 FAILED at 167. 1 out of 4 hunks FAILED -- saving rejects to file fs/libfs.c.rej 1442247654 M * undefined Ghislain: that looks like a failure i've already addressed on the mailing list 1442247682 M * Ghislain i assume as i compiled it before that i found a solution somewhere 1442247684 M * Ghislain :) 1442247714 M * undefined the stable updates are usually very similar for all the different versions, so i've addressed 3.10, 3.14, and 3.18 on the mailing list (as those are the ones i "support") 1442247782 M * Bertl I will update the patches tonight or tomorrow 1442247840 M * Ghislain k 1442247880 M * Ghislain i will wait your version then i will try to make a kernel without this dentry count 1442248067 M * Ghislain but really i dont see how it could make the dentry->d_count be zero 1442248128 M * Bertl it might be that we hit an already freed dentry there, in which case everything is possible 1442248276 M * Ghislain what i saw was not any code modifiying the dentry but perhaps i read badly ( remove the perhaps ;p ) 1442248351 M * undefined Bertl: my latest emails to the mailing list with fix-ups... 1442248354 M * undefined 3.10: http://archives.linux-vserver.org/201507/0000.html 1442248354 M * undefined 3.14: http://archives.linux-vserver.org/201508/0001.html 1442248354 M * undefined 3.18: http://list.linux-vserver.org/archive?mss:6969:201508:ehlceiakalnejibbanbb (i don't know why this email isn't in the mailing list archive at archive.linux-vserver.org?) 1442248433 M * Bertl thanks, no idea regarding the archives 1442248485 M * undefined i just prefer archives.linux-vserver.org because it references attached patches as separate files that can be specifically downloaded 1442248514 M * undefined unlike list.linux-vserver.org where any attached patches end up being appeneded to the end of the email 1442248673 M * Bertl yep 1442248744 M * Ghislain hum, i should reverse only on dcache.c or also on other files ? 1442248800 M * Bertl the short answer is only those affecting the d_count :) 1442248821 M * Bertl but I'll prepare a patch to revert those as well 1442248885 M * Ghislain ok thanks a lot ! i am reading the aptch but i do not see d_count modification just d_count test in the patch, i mena in line with + or - not the others 1442248986 M * Ghislain dcache.c and vs_limit.h are the only one but the .h is called by the .c i guess 1442248994 Q * derjohn_mobi Ping timeout: 480 seconds 1442250418 M * Ghislain i must go next episode tomorow :) 1442250487 M * Bertl cya 1442250977 Q * Gremble Quit: I Leave 1442251319 J * derjohn_mob ~aj@nl12x.mullvad.net 1442251347 M * Bertl off for a nap ... bbl 1442251350 N * Bertl Bertl_zZ 1442256179 J * derjohn_mobi ~aj@x4db0f5af.dyn.telefonica.de 1442256212 Q * derjohn_mobi 1442259279 Q * derjohn_mob Ping timeout: 480 seconds 1442260882 J * derjohn_mob ~aj@tmo-111-180.customers.d1-online.com 1442262525 J * FireEgl ~FireEgl@173-23-76-11.client.mchsi.com 1442263020 J * Aiken ~Aiken@d63f.h.jbmb.net 1442263455 N * Bertl_zZ Bertl 1442263457 M * Bertl back now ... 1442264114 J * bonbons ~bonbons@2001:a18:20c:4701:5155:67ad:b79c:d87a 1442265363 Q * Ghislain Quit: Leaving. 1442266790 Q * bonbons Quit: Leaving 1442273580 Q * jrayhawk Read error: No route to host 1442273613 J * jrayhawk ~jrayhawk@nursie.omgwallhack.org 1442275161 Q * fstd Remote host closed the connection