1515898213 M * daniel_hozac IPv6 has always been kinda random which address the host uses. 1515900301 M * Aiken I thought setting preferred_lft 0 on the host for the address in question should have told the host that address was depreciated an not to use it 1515901048 M * Aiken finally at the point of configuring 4.9.76, as my last was 3.4.17 is there anything I should look for? 1515901546 M * Bertl 1000+ new kernel options, but nothing on our side 1515901676 M * Aiken compiling now 1515907267 M * Aiken stopping a guest is fatal for the host 1515909420 M * Bertl hmm, in what way? 1515909433 M * Bertl do you get a kernel trace? 1515909499 M * Aiken 2 oops and vserver-info http://rilapy.jbmb.net/vserver/ 1515909558 M * Bertl can you run those addresses through addr2line for me please? 1515909739 M * Aiken maybe need to look for a debugging info option with the kernel, all aadr2line gives is ucount.c:? 1515909761 M * Bertl check this one: ffffffff8110402f 1515909772 M * Bertl addr2line -e vmlinux ffffffff8110402f 1515909791 M * Aiken even worse ??:? 1515909817 M * Bertl okay, check for CONFIG_DEBUG_INFO=y 1515909833 M * Aiken looking in System.map that last one is in free_cgroup_ns 1515909870 M * Bertl yeah, it seems to be caused by the cgroup namespace 1515909889 M * Aiken not set. A few years since I last did this. Testing on an alpha would have been the last time 1515909931 M * Bertl please enable it, it doesn't affect performance 1515909984 M * Aiken just debug info or add reduced, split, dwarf4 as well? 1515910042 M * Bertl we don't need dwarf4, reduced should be avoided, split is okay but might cause problems with some tools 1515910068 M * Aiken compiling with just info 1515910707 M * Bertl if possible, please upload your .config as well 1515910812 M * Aiken http://rilapy.jbmb.net/vserver/config-4.9.76 1515910819 M * Bertl tx 1515910903 J * fstd_ ~fstd@xdsl-78-34-243-132.netcologne.de 1515911108 M * Aiken finally compiled 1515911349 Q * fstd Ping timeout: 480 seconds 1515911350 N * fstd_ fstd 1515911566 M * Aiken tried the new kernel http://rilapy.jbmb.net/vserver/crash2.txt 1515911609 M * Aiken addr2line -e vmlinux ffffffff8109a254 1515911614 M * Aiken /usr/src/linux/kernel/ucount.c:166 1515911658 M * Aiken addr2line -e vmlinux ffffffff8110402f 1515911659 M * Aiken /usr/src/linux/kernel/cgroup.c:6391 1515911768 M * Bertl please try this one: http://vserver.13thfloor.at/Experimental/delta-nsproxy-fix02.diff 1515912019 M * Aiken no crash 1515912030 M * Bertl \o/ 1515912620 M * Aiken on the current vm I have been doing similar to ip -6 addr change preferred_lft 0 dev eth0 on the host to mark the guests ipv6 address as depreciated. At the moment on the test box the host seems to be sticking to it's ipv6 and the guest to it's 1515912824 M * Aiken looking at some old stuff I had started playing with netns when starting a guest but I do not remember why I stopped 1515914842 M * Aiken still no crashes, thanks 1515914851 M * Bertl thanks for testing! 1515921882 M * Bertl off to bed now ... have a good one everyone! 1515921884 N * Bertl Bertl_zZ 1515929358 J * yang_ ~yang@199.189.205.50 1515929389 M * yang_ oh my...I had to do "fsck" on a server, and now after its finished, I have noticed that /var/lib/vservers directory is empty... ? 1515930072 M * Ghislain is that a partition has not been mounted ? 1515930113 M * yang_ well I mounted back the partition after fsck and I think that the "y" to all just wiped out /var/lib/vservers or something, not sure if I can restore it back with fsck ? 1515930185 M * yang_ I mean the vservers contained like 100 GB of data at least, and now it says there is only about 20 GB of free space, so those files must still be somewhere 1515930231 M * Ghislain yes, i really dont know much about file recovery when filesystem screwup so cannot help here sorry 1515930237 M * yang_ ok 1515930291 M * yang_ ok well, not much I could do right now, I need to go to lunch, so back in about 3 hours, if anyone has any hint, I'd appreciate that 1515930317 M * Ghislain ok, good luck on this, pausing is also a good idea when you have those case ^ 1515930373 M * Ghislain Bertl_zZ: 4.4.111 with latest patch seems ok testfs and testme are ok 1515930391 M * Ghislain memeory reporting is still bad but the guest seems to run 1515930489 M * Guy- yang_: what fs did you have under /var/lib/vservers? doesn't it contain a lost+found directory with most of your stuff inside? also, why are there no backups? 1515930503 M * Ghislain daniel_hozack: thanks for looking at my wonderfull patchs ;p i have not been able to compile the new version it fail. I sent the errors and also the warnings in the ML 1515930527 M * yang_ Guy-: I dont know, I never made the backups in time 1515930559 M * Guy- yang_: what about the other questions? 1515930570 M * yang_ drwxr-xr-x 2 root root 60 Feb 17 2016 vservers 1515930579 M * yang_ filesystem is probably ext2 or ext3 1515930584 M * Ghislain daniel_hozac: thanks for looking at my wonderfull patchs ;p i have not been able to compile the new version it fail. I sent the errors and also the warnings in the ML 1515930595 M * Guy- yang_: well, maybe look at /proc/mounts, that should tell you what it is 1515930638 M * yang_ /proc is empty the server is offline, I was only able to mount partition in rescue console 1515930651 M * yang_ and I ran fsck there 1515930666 M * Guy- and you don't have /proc mounted? wow 1515930682 M * yang_ well the server isnt booted up 1515930695 M * Guy- still, even initramfs mounts /proc 1515930717 M * Guy- I don't know what that rescue console is, but not mounting /proc seems extreme 1515930740 M * yang_ ah yes I have that I see it now 1515930753 M * Guy- anyway, you can mount it yourself: mount -n -t procfs proc /proc 1515930760 M * Guy- sorry, mount -n -t proc proc /proc 1515930777 M * Ghislain tune2fs -l /dev/xxx will tell you if this is extx 1515930792 M * yang_ http://paste.debian.net/plainh/a748dfcf 1515930828 M * yang_ I think its this /dev/sda1 /mnt/disk1 ext4 rw,relatime,data=ordered 0 0 1515930861 M * Ghislain well recent vserver supports only extX so there is a BIG chance you use ext4 :) 1515930862 M * daniel_hozac Ghislain: i see no errors 1515930888 M * Ghislain ah damm 1515930891 M * Guy- yang_: pastebin output of "find /mnt/disk1 -ls; blkid /dev/sda1" (if it doesn't contain anything sensitive) 1515930896 M * yang_ which is strange, because the server is really old, and I have probablky used ext2 at that time, or ext3, not sure why it turned to ext4 1515930904 M * Ghislain again those double buffer in linux for cut adn paste... 1515930921 M * Guy- yang_: the ext4 driver can mount older versions of extX, I think 1515930926 M * Ghislain gimme a sec 1515931004 M * Guy- yang_: also "df -h /mnt/disk1" 1515931097 M * yang_ Guy-: that content is very large and would reveal all files, I am not able to paste that content 1515931105 M * Ghislain daniel_hozac: i sent it, do you got it ? /usr/include/linux/fs.h:366:23: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__kernel_rwf_t’ In file included from lib_internal/util-cleanupmount.c:25:0: 1515931144 M * yang_ Filesystem Size Used Avail Use% Mounted on 1515931146 M * yang_ /dev/sda1 290G 254G 22G 93% /mnt/disk1 1515931153 M * yang_ So there is like 22 GB free 1515931161 M * yang_ and vserver guests probably contain 100 GB 1515931168 M * yang_ so if they were wiped out 1515931176 M * yang_ the free data would be at least 120GB 1515931249 M * Guy- yang_: if the content of find -ls is very large, what exactly is missing? 1515931327 M * yang_ well probably the /var/lib/vservers ... and the directories with "guest vservers" in there ? 1515931339 M * yang_ or has that been reassigned elsewhere 1515931354 M * Guy- yang_: your fs is not mounted under /var/lib/vservers right now; you mounted it under /mnt/disk1 1515931379 M * Guy- yang_: so if your problem is that /var/lib/vservers is empty, then that's why 1515931406 M * yang_ so where is the content of lets say "guest1" , previously those were in /var/lib/vservers/guest1....but I dont recall if that only when system is booted up, or also when its being shut down ? 1515931448 M * Guy- yang_: currently, if /dev/sda1 indeed is the fs you normally have mounted under /var/lib/vservers, then guest1 would be under /mnt/disk1/guest1/ 1515931468 M * Guy- yang_: you just looked at what /mnt/disk1 contains -- didn't it strike you as familiar? 1515931562 M * yang_ well its just the / directory, but the guests should be somewhere too 1515931583 M * yang_ afaik in /var/lib/vservers, and that directory is just empty 1515931628 M * yang_ Thank you for your assistance, but I need to go to lunch now, will come back in about 2-3 hours 1515931794 M * Guy- yang_: I just explained why /var/lib/vservers was empty; if you're not going to pay attention to what I say, I'm not going to help 1515931812 M * Guy- if you don't understand something I say, ask 1515931839 M * Guy- but essentially, from what you've told us so far, everything should be fine after you reboot 1515931844 M * yang_ I ment /mnt/disk1/var/lib/vservers 1515931857 M * yang_ I will re-read what you wrote later again, and reply, ok 1515931861 M * yang_ afk now 1515931865 M * Guy- enjoy 1515933500 M * Ghislain daniel_hozac: it seems to compile on old debian but on recent one it fail with the error given 1515933522 M * Ghislain so perhaps a change in kernel or build chain perhaps dietlibc compatibility or such ? 1515933543 J * ntrs ~ntrs@vault08.rosehosting.com 1515933676 M * Ghislain daniel_hozac: on googlebrain a lot refers to dietlibc issue with recent code 1515933705 M * Ghislain i tried to get musl but the ./configure and Makefile are too complex for me to understand so i failled 1515934168 M * daniel_hozac musl should probably work if you just specify CC=musl-gcc to ./configure 1515934196 M * daniel_hozac 3131 should fix the last of that fs.h issue, could you give it a try? 1515934209 M * daniel_hozac not sure how i missed that instance, i did grep for all its uses... 1515934353 M * Ghislain sure uploading it now 1515934882 M * Ghislain in file included from lib_internal/util-cleanupmount.c:28:0:, /usr/include/linux/fs.h:366:23: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘__kernel_rwf_t’ 1515934921 M * Ghislain same error but 3 lines later :) 1515936606 M * Ghislain FYI musl do not work it fail before even starting the compile : checking for number of syscall 'vserver'... conftest.c:43:12: fatal error: asm/unistd.h: No such file or directory 1515937333 M * Ghislain bertl_zZ: for the memory used * 4096 is quite close to the difference between when the guest run or not, so i guess usage shown is pages not bytes 1515939587 J * Gremble ~Gremble@cpc1-aztw34-2-0-cust397.18-1.cable.virginm.net 1515940163 M * yang_ Guy-: to summorize...root@rescue:/mnt/disk1/var/lib/vservers# ls -la 1515940163 M * yang_ total 0 1515940188 M * Guy- yang_: yes, because whatever is normally there is not mounted 1515940189 M * yang_ guests should be in there "or not?" I maybe wiped them out with fsck ? 1515940192 M * yang_ ah 1515940209 M * yang_ so its only visible when the vserver machine is booted up, then these directories exist ? 1515940209 M * Guy- yang_: check /mnt/disk1/etc/fstab 1515940228 M * Guy- yang_: well, when you boot it up, I assume it will exist and be populated 1515940252 M * Guy- 13:10 < Guy-> but essentially, from what you've told us so far, everything should be fine after you reboot 1515940261 M * yang_ http://paste.debian.net/plainh/6b386e23 1515940344 M * yang_ well the thing is that, if I execute reboot,the machine doesnt come up, its in the datacentre, so I cant see the boot log, I will them to plug in the KVM... 1515940346 M * Guy- yang_: OK, it doesn't seem fstab has an entry for /var/lib/vservers 1515940380 M * yang_ I dont see what happens after reboot 1515940381 M * Guy- yang_: is there anything in /mnt/disk1/lost+found? 1515940426 M * yang_ http://paste.debian.net/plainh/8ace0e70 1515940470 M * Guy- yang_: that entry with the question marks doesn't look right 1515940487 M * Guy- the other directories could just as well be the guest directories that are missing 1515940492 M * yang_ right 1515940496 M * yang_ so how can I restore that ? 1515940513 M * Guy- take a look inside them; if you can identyify them, you can move them to /mnt/disk1/var/lib/vservers under the correct name 1515940518 M * yang_ ok 1515940523 M * Guy- when you are done with that, umount the fs and re-run fsck on it 1515940529 M * yang_ okay 1515940604 M * yang_ you are right, those directories are indeed the "guest" systems which were missing 1515940623 M * yang_ what about any special persmissions, or do I just do "mv" 1515940647 M * Guy- just mv 1515940918 M * yang_ mv /mnt/disk1/lost+found/#14090300 /mnt/disk1/var/lib/vservers/vserver1 if "vserver1" is the name of the guest, right ? 1515941109 M * yang_ Guy-: ^ 1515941483 M * Guy- yang_: yes 1515941506 M * Guy- yang_: you may have to escape the # character with a backslash, depending on your shell and settings 1515941710 M * yang_ right 1515942689 Q * Gremble Quit: Leaving 1515942744 M * yang_ Guy-: any idea how to fix the last lost+found directory ? I have moved the rest in place http://paste.debian.net/plainh/60b371e7 1515942759 M * yang_ I cannot see the content 1515943258 M * arekm wow, 4.9 patch 1515943612 M * Guy- yang_: 15:35 < Guy-> when you are done with that, umount the fs and re-run fsck on it 1515943624 M * Guy- yang_: fsck _may_ correct the last directory 1515943723 M * yang_ Guy-: running fsck with options "fsck -sV /dev/sda1" ? 1515943765 M * yang_ it's actually triggering "e2fsck" there on the rescue console the "fsck" 1515943791 M * Guy- yang_: I'd just run e2fsck -p 1515943797 M * Guy- (on /dev/sda1) 1515944142 M * yang_ http://paste.debian.net/plainh/3c042611 1515944310 M * yang_ Maybe I did not really guess the exact guest vserver name ? 1515944695 M * Guy- yang_: this is related to that unreadable entry 1515944705 M * Guy- yang_: retry e2fsck /dev/sda1, without options 1515944735 M * yang_ well 1515944780 M * yang_ unreadable entry is #4309096 and this error now relates to Inode 4309123, which is a different number 1515944793 M * yang_ but yes, it's probably related 1515944991 N * Bertl_zZ Bertl 1515944995 M * Bertl morning folks! 1515945153 M * Bertl Ghislain: the memory problem is on both 4.4.x and 4.9.x but not with 4.1.x yes? 1515945328 M * gnarface morning Bertl! 1515945397 M * yang_ Guy-: http://paste.debian.net/plainh/880f320a 1515945433 M * Guy- yang_: I'd say yes 1515946286 M * Bertl Ghislain: also, what's your guest's memory configuration? 1515946297 M * Bertl gnarface: how's going? 1515946352 M * gnarface Bertl: can't complain. 4.1.x has been very stable for me. 1515946371 M * Bertl good to hear! 1515946371 M * gnarface Bertl: i'd like to be able to upgrade to 4.9 but it's not important yet. 1515946446 M * yang_ Guy-: http://paste.debian.net/plainh/d4666474 1515946521 M * yang_ Bertl: is there a 4.9 patch released yet ? 1515946524 M * Guy- yang_: definitely yes 1515946538 M * Guy- yang_: as for 4.9 patch, no, it's still in alpha 1515946548 M * yang_ Unconnected directory inode 4309125 (/var/lib/vservers/???) 1515946558 M * yang_ now it keeps asking me more questions the fsck 1515946580 M * yang_ Connect to /lost+found? 1515946603 M * yang_ do I give "yes" to all ? 1515946610 Q * sannes Ping timeout: 480 seconds 1515946817 M * Guy- yes 1515946922 M * yang_ http://paste.debian.net/plainh/eb210fd7 1515946960 M * Guy- yang_: yes 1515947010 M * yang_ Inode 4310047 ref count is 21, should be 20. Fix ('a' enables 'yes' to all) ? 1515947023 M * yang_ just "yes" to all ? 1515947086 M * Guy- yes to all refcount correction questions 1515947100 M * Guy- but probably you'll want to say yes to all others too 1515947363 M * yang_ ok its finished http://paste.debian.net/plainh/5317086a 1515947377 M * yang_ Should I just reboot now ? 1515947408 M * yang_ It hasnt booted before, so maybe if something else is wrong it wont boot now either, it would maybe be smart to obtain remove KVM for it 1515947422 M * yang_ I mean remote KVM console 1515947478 M * yang_ I guess I could set up one of those remote boot outputs too, but I guess it has to get up until the IP stack 1515947490 M * yang_ "netconsole" 1515947674 M * Guy- yang_: don't reboot yet 1515947684 M * Guy- yang_: mount the fs again, and sort out the contents of lost+found 1515947732 M * yang_ ok 1515947823 M * yang_ ugh 1515947833 M * yang_ now there are even more directories inside than earlier 1515947855 M * yang_ Guy-: can we go into private chat, not to flood on this channel ? 1515947896 M * Bertl I don't mind, it's mildly entertaining ... 1515947933 M * yang_ Yeah, because I am so dumb 1515947965 M * Bertl well, because you are making all the 'common' mistakes in data rescue 1515947976 M * yang_ Yeah I havent really done that earlier 1515947988 M * yang_ http://paste.debian.net/plainh/8a13812c 1515948068 M * yang_ well 1515948088 M * yang_ I might have not assigned the right Guest names earlier....I only tried to do that by "memory" 1515948089 M * Bertl that looks very similar to what you had before btw. 1515948118 M * yang_ well now, I can see that 4 guests in /mnt/disk1/var/lib/vserver are there 1515948134 M * yang_ but there are like 2 of them which I have assigned there and arent there yet and are in /lost+found 1515948172 M * yang_ so I'll go through directories again 1515948398 M * yang_ what the ... http://paste.debian.net/plainh/27a269c4 1515948744 M * Bertl I'd say your guests are back in lost+found 1515948929 M * Bertl just a quick tip, if you can (network, second hdd, etc) make a backup of your data (the part which is still there) I would strongly suggest to do so 1515949136 M * yang_ ok 1515949175 M * yang_ yeah guests are back in the lost+found 1515949183 M * yang_ I dont know why 1515949586 M * yang_ http://paste.debian.net/plainh/f4683dfd 1515949619 M * yang_ I am rsyncing the whole partition now, first 1515949625 M * yang_ to backup 1515949923 M * yang_ What if the drive has bad sectors ? 1515951648 M * Bertl where is backup? 1515951714 M * yang_ I am backing up remotelly 1515951733 M * yang_ I didnt do backup before 1515951783 M * Bertl okay, just making sure that 'backup' is not a directory on the filesystem you are trying to rescue :) 1515951796 M * yang_ no :) 1515951822 M * yang_ The problem is I have little free time and now this... 1515951858 M * Bertl could have saved you a lot of time with a good backup plan 1515951862 M * yang_ indeed 1515951914 M * yang_ well I will probably get the vservers in a backup copy, so those could be ported on another vserver machine, and it would just work ok 1515951933 M * yang_ in case that the structure still has the files 1515951957 M * Bertl what are the rsync options you are using? 1515951958 M * yang_ but being clueless about fsck restore, then I get in such a mess 1515951964 M * yang_ I am using the ones you sugested 1515951975 M * Bertl including --numeric-ids ? 1515951976 M * yang_ 17:55 < Bertl> rsync -axXHPSD --numeric-ids 1515951976 M * yang_ 17:56 < Bertl> but I think -a already includes X and H, so not all are required 1515951977 M * yang_ 17:57 < Bertl> ah, no, I'm wrong, just checked with man, X and H are not in -a 1515951984 M * yang_ yes 1515951988 M * Bertl excellent! 1515952018 M * yang_ well I also add -e "ssh -p 22" ..... 1515952023 M * yang_ I hope that doesnt messes it up 1515952099 M * Bertl nah, that's fine, although it should be default 1515952126 M * yang_ default rsync is probably not in encrypted mode 1515952176 M * Bertl unless you use an rsync target, ssh is the default 1515952271 M * Bertl (i.e. if you use rsync://... then it will be just the rsync protocol, otherwise ssh) 1515952518 M * yang_ ok 1515953836 M * daniel_hozac Ghislain: could you get a complete output from that? 1515956071 M * Ghislain daniel_hozac: https://pastebin.com/raw/inG85nD5 1515956876 M * Ghislain is that what you where searching daniel? 1515956909 M * Ghislain it compile on a gcc6 but on gcc7 (debian buster) it does not and i have 6 warning, one error 1515957048 M * Bertl hey Ghislain, got my messages? 1515958226 M * Ghislain i must have missed some because theis WE i was on and off :) 1515958243 M * Bertl no problem ... let me repeat them 1515958245 M * Ghislain oh my memory config 1515958273 M * Ghislain i have 16Gb on the host, the guest use mem cgroup with a limit of 12gb meme and 12gb mem+swap 1515958332 M * Ghislain memory.memsw.limit_in_bytes and memory.limit_in_bytes 1515958332 M * Ghislain are 12884901888 1515958361 M * Ghislain with VIRT_MEM flag 1515958426 M * Ghislain any other i missed ? 1515958434 M * Bertl 16:52 < Bertl> Ghislain: the memory problem is on both 4.4.x and 4.9.x but not with 4.1.x yes? 1515958451 M * Ghislain ah 4.9 i had not tested yet 1515958501 M * Bertl but 4.4.111-vs2.3.9.5, yes? 1515958510 M * Ghislain 4.1 is fine: ie same config i got on an exemple: Mem:      12582912    4051520    8531392     445432          0          0 1515958529 M * Ghislain on 4.4.111 i have used= very very low 1515958594 M * Ghislain like this: Mem:       12582912         432    12582480       31832       30932    12498012 1515958614 M * Ghislain there si mysql spamd apache phpfpm etc.. so 452kb is unlikely :) 1515958621 M * Ghislain 432 1515958642 M * Ghislain and yes it is 4.4.111-vs2.3.9.5aq 1515958707 M * Ghislain so 4.1 ok, 4.4 , not ok, 4.9 not tested yet 1515958764 M * Bertl okay, tx 1515958830 M * Bertl off for now ... bbl 1515958834 N * Bertl Bertl_oO 1515969254 M * arekm Bertl_oO: booted 4.9, so far no problems 1515972993 Q * DoberMann Remote host closed the connection 1515973066 J * DoberMann ~james@2a01:e35:8b44:84c0::2