1355098557 J * imachine ~imachine@ip-178-216-200-202.e24cloud.com
1355104963 J * Aiken ~Aiken@2001:44b8:2168:1000:21f:d0ff:fed6:d63f
1355108055 M * imachine hello
1355108065 M * imachine I have some issues with running ubuntu inside a vserver
1355108135 M * imachine I followed the instructions for Lucid, pulled the initpost file inside /usr/lib/util-vserver/distributions/precise and built the vserver with --initstyle plain
1355108149 M * imachine the issue I have is with mounting of /run/lock
1355108235 M * imachine ubuntu mounts tmpfs at /run, then mounts /run/lock. I have the lines in /etc/vservers/myvserver/fstab but the mount point /run/lock doesn't exist (and I can't really create it since /run is tmpfs so recreates itself on each boot).
1355108241 M * imachine s/boot/mount ;)
1355108266 M * imachine if maybe there was some way to pass a script to vserver when issuing vserver-start
1355108294 M * imachine in order to mount /run, mkdir /run/lock, mount /run/lock, continue, profit
1355108453 M * imachine (I'd throw the mounting in rc.local inside the vserver, but mounting is not permitted from inside of vserver)
1355110016 M * Bertl there are several scripts at several stages you can utilize to do that
1355110057 M * Bertl in general, I'd suggest to keep the guest system from those 'strange' mounts because they seldom make sense in a host/guest setup
1355110214 M * imachine well ubuntu seems to need those. I'm stuck at upgrading initscripts right now because it tries to mount/umount during the configuration process which fails and hence I have unconfigured packages. weird that I had the same error a while back but it just went away by itself.
1355110238 M * imachine I think I need sleep or I'll be seeing double.
1355110247 M * imachine I'll check it out in the morning.
1355110249 M * imachine Cheers.
1355111430 Q * imachine Ping timeout: 480 seconds
1355121273 M * Bertl off to bed now ... have a good one everyone!
1355121279 N * Bertl Bertl_zZ
1355126157 Q * fisted Quit: bbl
1355126448 J * Ghislain ~aqueos@adsl1.aqueos.com
1355129047 J * nicholi ~nicholi@rrcs-76-79-196-34.west.biz.rr.com
1355130439 J * bonbons ~bonbons@2001:960:7ab:0:e1b1:2e3b:6571:2efc
1355131022 Q * ensc|w Remote host closed the connection
1355131031 J * ensc|w ~ensc@www.sigma-chemnitz.de
1355137513 Q * trippeh Ping timeout: 480 seconds
1355137717 J * trippeh atomt@t-1000.ugh.no
1355139450 J * thierryp ~thierry@home.parmentelat.net
1355139665 Q * ircuser-1 Ping timeout: 480 seconds
1355140309 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355140572 J * clopez ~clopez@242.17.60.213.dynamic.mundo-r.com
1355140770 M * imachine hey, if I change my vserver's IP, is it only in /etc/vservers/myvserver/interfaces/* that the IP address resides?
1355140796 M * imachine or should I change it somewhere else too - I changed my ip and now my vserver can't reach world.
1355140920 N * Bertl_zZ Bertl
1355140925 M * Bertl morning folks!
1355140971 M * Bertl imachine: did you change it when the guest was stopped?
1355141296 M * imachine Bertl, yea
1355141308 M * Bertl that's your problem
1355141318 M * imachine and the guest has the proper ip
1355141320 M * imachine I see
1355141333 M * Bertl ah, sorry, I read that you changed it while it was running
1355141343 M * imachine Bertl, nah, I stopped it first.
1355141354 M * imachine and from within the guest I can see the ip is there.
1355141360 M * Bertl yeah that's fine then, so the problem is most likely not with the guest
1355141372 M * imachine ok
1355141377 M * Bertl try, for a test, on the host to ping with that 'new' IP
1355141380 M * imachine I'll sort it out.
1355141391 M * Bertl i.e. ping -I <guest-ip> www.google.com
1355141428 M * imachine yeah I just did and it doesn't seem to work
1355141435 M * imachine so I guess it's another thing then
1355141436 M * imachine cheers
1355141444 Q * imachine Quit: leaving
1355141720 J * imachine ~imachine@178.216.200.235
1355141899 Q * imachine 
1355143152 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355143174 J * ircuser-1 ~ircuser-1@35.222-62-69.ftth.swbr.surewest.net
1355143245 Q * imachine 
1355143259 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355143262 Q * imachine 
1355143268 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355143699 J * Guy- ~korn@catv-176-63-136-219.catv.broadband.hu
1355144715 Q * imachine Quit: leaving
1355146366 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355146924 Q * imachine Remote host closed the connection
1355147073 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355147842 Q * imachine Quit: leaving
1355147845 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355147897 Q * imachine 
1355147972 J * imachine ~imachine@ip-178-216-200-235.e24cloud.com
1355148118 J * fisted ~fisted@xdsl-87-78-182-249.netcologne.de
1355149300 Q * tolkor Read error: Connection reset by peer
1355149306 J * tolkor ~rj@tdream.lly.earlham.edu
1355151772 Q * tolkor Remote host closed the connection
1355152408 J * BenG ~bengreen@cpc29-aztw23-2-0-cust144.18-1.cable.virginmedia.com
1355153444 J * tolkor ~rj@tdream.lly.earlham.edu
1355154443 Q * BenG Quit: I Leave
1355154503 J * benl ~benl@dockoffice.sonassihosting.com
1355154560 M * benl Hi All
1355154607 M * benl I recently added some more CPUs to a server this morning, and after a few hours - its performance fell through the floor - and the machine showed >100k context switches per second
1355154628 M * benl Removing the CPUs (or the reboot/shutdown) - appeared to solve the issue.
1355154634 M * Bertl what kernel?
1355154637 M * benl :(
1355154640 M * benl An old one.
1355154650 M * benl 2.6.32-5-vserver-amd64
1355154664 M * Bertl and a debian one too as it seems :)
1355154675 N * ensc Guest1003
1355154684 J * ensc ~irc-ensc@p54ADE2D9.dip.t-dialin.net
1355154687 M * DelTree looks pretty much like squeeze...
1355154716 M * benl Correct
1355154737 M * benl It went from 32 Cores / 64GB RAM - to - 64 Cores / 128GB RAM
1355154760 M * benl But there was ~8 hours between adding the components and the issue ocurring.
1355154780 M * benl So it could either be that it was highlighted under load, or a coincidence
1355154818 M * benl So the additional components were removed - and the machine powered back on, and the issue was gone
1355154827 M * Bertl more cores without any cpu groups or similar naturally increase the numnber of context switches
1355154829 M * benl Again, it could have been the reboot that fixed it, or the removal of the components
1355154849 M * Bertl more memory can cause similar issues and delays with older kernels
1355154857 M * benl Interesting.
1355154887 M * benl Would you advise using cpuset/cpugoups then?
1355154902 M * Bertl depends on your setup
1355155090 Q * Guest1003 Ping timeout: 480 seconds
1355155169 M * benl Ok
1355155199 M * Bertl for example, if you have 128 cpus, and 64 guests, and in average each guest needs/wants 2 cpus
1355155219 M * Bertl you can reduce the overhead dramatically by assigning 2 cpus to each guest
1355155240 M * Bertl because tasks will not be migrated away from those cpus
1355155282 M * benl Ah, I understand
1355155302 M * benl But it comes with the caveat that those guests cannot scale above 2 cores?
1355155320 M * Bertl and with many cpus, there comes locality of memory, so caches are local and efficient
1355155336 M * benl Absolutely, makes a lot of sense
1355155338 M * Bertl well, yes, you can do other combinations as well, like overlapping cpu sets
1355155361 M * Bertl e.g. assign 4 cpus to each guest with an overlap of two cpus between two guests
1355155513 M * benl Ok
1355155544 M * benl How come you guys don't backport newer kernels for Debian?
1355155552 M * benl Whats the "preferred" OS?
1355155572 M * Bertl we 'guys' do not backport any kernel for any distro
1355155591 M * Bertl mainly because the mainline/vanilla kernels/patches work for all distros
1355155682 M * benl :)
1355155828 M * Bertl also a big advantage of building your own kernels is that you can get rid of all the stuff you do not need
1355155892 M * benl I agree
1355155901 M * benl but my history of building kernels has been failure after failure
1355155908 M * DelTree like what ? driver for my keyboard ?
1355155912 M * trippeh And get to track all the security issues yourself :9
1355155919 M * trippeh Wee!
1355155948 M * DelTree even with make-kpkg I never get it right the first time... :|
1355156087 M * Guy- DelTree: newer kernels have make deb-pkg, fwiw
1355156115 M * DelTree ooch... yet another stupid change ?
1355156140 M * DelTree or did I get it all wrong ?
1355156155 M * trippeh Its just an alternative
1355156208 M * trippeh Here at $work we use neither, but our own debian/ to package kernels
1355156348 M * benl I can build them with make-kpkg
1355156356 M * benl But the resultant kernel is usually less than stable
1355156392 M * DelTree the true question is... what can I plug into my aptitude update so that I just have to aptitude full-upgrade to get a vserver kernel up and running... ^_^
1355156624 M * benl I've (generally) never had stability issues with the latest stable
1355157111 M * Bertl trippeh: do you have any idea when the debian stable vserver kernel received a security update?
1355157255 M * trippeh Bertl: Sun, 23 Sep 2012 04:22:37 +0900
1355157294 M * trippeh It is updated with the other non-vserver images
1355157432 M * Bertl and the last Linux-VServer related update/fix?
1355157467 M * trippeh [vserver] Update patch to 2.6.32.48-vs2.3.0.36.29.8 @ Fri, 23 Dec 2011 12:14:46 -0700
1355157470 M * trippeh heh
1355157493 M * trippeh Oldhat, but at 2.6.32, what can one expect :)
1355157514 M * Bertl so roughly 10-15 'fix' patches missing ... not so bad :)
1355157600 M * trippeh It does have the longterm -stable mainline patches after that tho.
1355157627 M * Bertl don't get me wrong, I understand that distro kernels give a certain warm cosy feeling compared to bleeding edge mainline kernels
1355157684 M * trippeh I get too restless like after 3-6 months after a distro release to run distro kernels :)
1355157695 M * trippeh Too much new hotness incoming
1355157697 M * Bertl the problem with distro kernels is that distro rules and the lack of good maintainers tend to make distro kernels worse than mainline
1355157704 M * trippeh Cant resist it
1355157704 M * trippeh ;)
1355157749 M * trippeh At work 3.7 is already in testing hehe
1355157797 M * Ghislain mainline kernel can be pain too as 32 showed. 34 for now seems a lot more stable and this is with the very same config :)
1355157855 M * Ghislain the issue with home made kernel is that every hoster and company will point toward this if any weird bug happen They will just say this is the kernel and drop support
1355157871 M * ard 3.5 and 3.6 hopelessly fail on my ARM based nas@home... and that's vanilla... 3.4 works perfectly, but @work I need something that's >=3.4 due to fusion-mpt bug fixes. So I am gonna compile 3.6.9
1355157911 M * trippeh 3.6.10 should be out later today (likely) fwiw
1355157927 M * Ghislain this is one big problem, not that i care as their support sucks anyways so i rely on people like  well me, vserver's team, exceliance and other that know their job ( me being an exception of that rule for pure ego )
1355157936 M * Ghislain 361 ouch so fast
1355157946 M * Ghislain need to rebuild everyday to keep up !
1355157954 M * Bertl ard: how are the stable debian kernels on arm?
1355158024 M * ard eh... dunno, I always build kernels myself. As vlan maintainer I occassionally got bug reports from faulty patches in the official debian release
1355158138 M * ard I did write a howto build a debian kernel package, so it should be somewhere on the internets. But building debian kernel packages is more easy than building the original way (unless you are cross-compiling)
1355158258 Q * Ghislain Quit: Leaving.
1355158365 J * Ghislain ~aqueos@adsl1.aqueos.com
1355158656 M * Bertl daniel_hozac: regarding mnt_is_reachable(), do you think we should simply drop the check or make it depend on a kernel config
1355158833 M * benl Bertl: if you want an ARM system to play with - I can give you a Raspberry P
1355158837 M * benl *Pi
1355158841 Q * sladen Quit: Changing server
1355158855 J * sladen ~paul@starsky.19inch.net
1355158875 M * Bertl benl: I won't say no to that, especially as I registered early (for one) and was 'forgotten' :)
1355158881 M * benl Ditto
1355158890 M * benl I forgot about it until recently
1355158895 M * benl Then ordered another one
1355158909 M * benl It took 4 weeks to arrive, but now I've got it, I've no idea what to do with it!
1355158931 M * benl I put the stock debian distro on it, ran `top`  - then got bored 5 minutes later
1355158939 M * Bertl hehe
1355158972 M * benl How about I swap you a Raspberry Pi for a super stable Debian kernel :)
1355159026 M * Bertl super stable debian kernel just means that it is at least 4 years old, right? :)
1355159038 A * ard compiled his own kernel for ubuntu
1355159050 M * ard after the upgrade the ubuntu 12.10 ended up in a reboot loop
1355159052 M * benl You can do a brand new one if you like
1355159064 M * benl super stable is the goal :)))
1355159086 M * Bertl I think 3.6.9+ will be super stable
1355159089 M * ard depends also on your hardware
1355159107 M * ard for e1000e adapters to work stable you also need a recent kernel :-)
1355159122 M * benl Really?
1355159130 M * benl Well, we don't have much that uses e1000e nowadays
1355159134 M * benl Almost everything is IGB
1355159137 M * ard there were some bugs...
1355159138 M * ard wow
1355159151 M * benl maybe ~5 servers with e1000e drivers
1355159155 M * ard now you mention igb, there were some bugs there too 8-D
1355159157 M * benl the Intel E5620 kit
1355159190 M * ard But most of the people don't trigger them, has to do with the buffering and queuemanagement if I recall correctly
1355159249 M * ard you can prevent triggering a lot of bugs by increasing the ammount of unused memory
1355159303 M * ard *reading backlog*
1355159313 M * ard wow... 64 cores... and only 128G ram :-)
1355159322 M * benl :)
1355159328 M * benl Just 2x 16GB DIMMS per CPU
1355159343 M * benl Its not for virtualisation in the strictest sense, its to run 1 website
1355159348 M * Bertl 2G per cpu seems fine to me, assuming that the guest systems have something to share
1355159355 M * ard ow, and what's also important is to always have the latest bios, because the microcode update utility on linux cannot update all microcode of the cpu
1355159419 M * ard I've never seen a shortage of CPU, only a shortage of RAM, or disk IOPS
1355159463 M * Bertl really depends on the workload
1355159470 M * ard jups...
1355159513 M * ard But we keep a tight control on the developers... If they program things wrong we are going to bofh them :-).
1355159751 M * benl This app is entirely CPU bound
1355159760 M * benl (given enough RAM)
1355159799 M * ard that must be something very interesting ;-)....
1355159811 M * benl Just e-commerce stores
1355159818 M * benl Well, 1 in particular
1355159823 M * ard multi store on a single platform?
1355159838 M * benl correct
1355159894 M * ard still I would say that a good store should not use a lot of cpu, but it all depends on your load/clients etc...
1355159905 M * benl Its the application I'm afraid
1355159908 M * benl Its "heavy"
1355159921 M * ard I've seen differences in drupal setups that makes me want to cry... (good setup vs very bad setup)
1355159924 M * benl And we're talking 250k daily unique visitors
1355159944 M * benl Around ~ 1000 orders per hour
1355159980 M * ard innodb or postgres? I would advice postgres, since I have a very disturbing aversion against innodb ;-).
1355159984 M * benl innodb
1355159992 M * benl many, many, many foreign keys
1355160003 M * benl and some MyISAM thrown in the mix for fun
1355160037 M * ard is it application engine bound or database engine bound?
1355160053 M * benl PHP
1355160072 M * ard Not that I want to keep you from your work... But I always like to know how people do stuff
1355160083 M * benl Np
1355160092 M * benl I ask enough random questions on here
1355160102 M * benl That Bertl will probably say aren't related to vserver
1355160116 M * ard Bertl en daniel_hozac are very helpful guys...
1355160125 M * benl I've got a lot of respect for them
1355160132 M * ard I don't think your context switch problem is vserver related...
1355160139 M * benl I'm not sure it is either
1355160149 M * benl The munin graphs show some interesting behavour
1355160151 M * ard a high context switch load is usually a process forking off, crashing forking etc...
1355160154 M * benl pre/post CPU addition
1355160164 M * benl Well, Nginx's fork rate did increase
1355160199 M * benl I can send a copy of the graphs if you want to have a look
1355160209 M * ard it might be that a process hit a limit, and then immidiately respawns...
1355160279 M * benl Really hard to saw
1355160280 M * benl Really hard to say
1355160288 M * Bertl regarding altered behaviour after changing the number of cpus or memory: keep in mind that many applications detect the number of cpus or the available memory nowadays and adjust accordingly ...
1355160337 M * benl Bertl: want a test rig with 32 Cores/64GB RAM :)
1355160341 M * benl ?
1355160356 M * Bertl if it has a serial console attached, why not?
1355160379 M * benl you're additicted to SOL
1355160384 M * benl Its missing HDDs at the minute
1355160984 M * benl our "old" kernel probably doesn't like having 64 cores :)
1355161380 M * ard When did you try the 64 cores?
1355161396 M * ard ah, monday
1355161419 M * benl This morning
1355161422 M * benl at ~1 am
1355161424 M * ard because you had the same issue on the 6th it seems
1355161472 M * ard the difference is that it probably faded out by itself :-)
1355161489 M * benl Thats what I think ...
1355161496 M * benl We even got Percona to consult
1355161505 M * benl As it was completely MySQL bound the whole time
1355161524 M * benl But Percona couldn't identify anything
1355161530 M * ard Hmmm, then you are using xtradb ;-)
1355161541 M * benl Yes
1355161554 M * ard that's better than innodb at least, those guys are good.
1355161564 M * ard But it still will have bugs...
1355161577 M * benl Un-identifyable ones it seems
1355161584 M * benl We were all pulling our hair out
1355161596 M * benl And cannot believe removing CPUs/RAM - solved a performance problem
1355161616 M * ard the problem with innodb is that it can change it's query solution on a whim...
1355161641 M * ard usually analyzing tables fixes that, but the most important thing is to take a look at the queries.
1355161659 M * ard But that should not ever lead to enormous amounts of context switches
1355161672 M * benl its so hard to identify
1355161677 M * benl Nginx/PHP was responsive
1355161686 M * benl The app itself (which connected to the DB), was not
1355161694 M * benl And MySQL was consuming 5000% CPU
1355161698 M * benl 50 cores ...
1355161738 M * ard ah, ok...
1355161749 M * ard You should set the max thread limit on innodb
1355161768 M * ard scale down the number of io threads and stuff like that...
1355161781 M * ard it will probably scale up due to the amount of cores
1355161807 M * benl They said something vaguely similar
1355161819 M * benl albeit, they had said to no set the innodb_thread_concurrency
1355161829 M * benl but the rest of the my.cnf was fine
1355161858 M * benl the underlying issue (they said) was that there was a large volume of slow queries using distinct - that were creating temp. tables on disk
1355161873 M * ard and you probably have ssd...
1355161891 M * benl 4x SSD in RAID10 on that system
1355161902 M * ard so it's doing a lot of writes to a device that can do 80k IOPS
1355161909 M * ard 160K IOPS then :-)
1355161918 M * benl I think these disks are only 40k
1355161928 M * ard in that case you do not have a real problem except for bad queries
1355161957 M * benl So the real mystery is
1355161964 M * benl Why did it suddenly stop ...
1355161979 M * benl We know the queries are greedy - and for now, cannot be adjusted.
1355161993 M * benl But how it started to generate this load, and why it disappeared - its a total mystery
1355162031 M * ard well, your conntrack is also peaking at those moments
1355162044 M * ard at least at the 6th
1355162078 M * benl Bertl said he didn't think it was a sysctl thing
1355162082 M * benl although I was convinced it was
1355162098 M * benl But he doesn't give me too much time (even when paid), until I change the Kernel
1355162111 M * benl and from experience, hand-built kernels are a disaster
1355162115 M * ard He is right on the kernel part :-)
1355162138 M * ard hand built, once known how to, will help you a lot
1355162144 M * benl I would agree
1355162149 M * ard you won't need any initrd/initramfs anymore
1355162157 M * benl I commissioned him to build a kernel ~ 5 months ago
1355162158 Q * thierryp Remote host closed the connection
1355162177 M * Bertl benl: ah, you can have all the time you want (if payed) but my advise was to switch to a recent kernel first before investigating further
1355162180 M * benl but the stupid network drivers regressed in the newer release!
1355162193 M * benl no no, I totally understand
1355162239 M * ard we are a debian company, and we use hand-build util-vserver and kernels...
1355162291 M * benl Make us a stable kernel
1355162294 M * benl And I'll take t
1355162296 M * benl And I'll take it
1355162928 Q * imcsk8 Remote host closed the connection
1355163069 J * imcsk8 ~ichavero@148.229.1.11
1355163082 A * ard is helping out a developer
1355166011 M * Jb_boin what kernel are you using right now?
1355166187 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:2431:c37c:2336:d27e
1355170716 T * * http://linux-vserver.org/ |stable 2.2.0.7,exp 2.3.2.7,grsec 2.3.0.36.38 | util-vserver-0.30.216-pre3029 | He who asks a question is a fool for a minute; he who doesn't ask is a fool for a lifetime -- share the gained knowledge on the Wiki, and we forget about the minute.
1355170716 T * ChanServ -
1355171535 J * hijacker_ ~hijacker@cable-84-43-134-121.mnet.bg
1355171713 Q * thierryp Remote host closed the connection
1355172785 Q * nou Ping timeout: 480 seconds
1355173146 J * nou Chaton@causse.larzac.fr.eu.org
1355173816 M * benl Bertl - whats the rysnc flags you use when cloing a vserver
1355173874 M * benl rsync -axHPSD --numeric-ids
1355173875 M * Bertl rsync -axHPSD --numeric-ids
1355173877 M * benl lol
1355173888 M * benl I just found them
1355173894 M * Bertl good :)
1355174392 Q * benl Quit:  HydraIRC -> http://www.hydrairc.com <- The professional IRC Client :D
1355175202 Q * brambles Remote host closed the connection
1355175603 J * brambles ~xymox@s0.barwen.ch
1355176149 Q * hijacker_ Quit: Leaving
1355176851 Q * brambles Remote host closed the connection
1355176864 J * brambles lechuck@s0.barwen.ch
1355180403 Q * fisted Remote host closed the connection
1355180403 Q * clopez Ping timeout: 480 seconds
1355180463 J * fisted ~fisted@xdsl-87-78-14-199.netcologne.de
1355181967 Q * nicholi Quit: leaving
1355182630 J * nicholi ~nicholi@rrcs-76-79-196-34.west.biz.rr.com
1355183270 Q * Ghislain Quit: Leaving.
1355183347 Q * bonbons Quit: Leaving