1409012952 M * Bertl off to bed now ... have a good one everyone! 1409012963 N * Bertl Bertl_zZ 1409016683 J * fisted_ ~fisted@xdsl-87-78-233-190.netcologne.de 1409017134 Q * fisted Ping timeout: 480 seconds 1409017135 N * fisted_ fisted 1409021322 Q * Romster Quit: Geeks shall inherit properties and methods of object earth. 1409022530 J * Romster ~Romster@202.168.100.149.dynamic.rev.eftel.com 1409029070 J * Ghislain ~aqueos@adsl1.aqueos.com 1409030439 Q * Romster Ping timeout: 480 seconds 1409033111 J * druschka_domaintechnik ~druschka@91-135-172-82.net.pr-link.at 1409033319 Q * druschka_domaintechnik 1409037182 J * Romster ~Romster@202.168.100.149.dynamic.rev.eftel.com 1409037582 J * redhat ~quassel@31.25.99.5 1409037598 J * _BWare_ ~itsme@31.25.99.5 1409037690 Q * BWare Read error: Operation timed out 1409037721 Q * webhat Read error: Operation timed out 1409042059 Q * hijacker Read error: Connection reset by peer 1409042071 J * hijacker ~hijacker@213.91.163.5 1409042276 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409045494 Q * thierryp Remote host closed the connection 1409047753 N * Bertl_zZ Bertl 1409047761 M * Bertl morning folks! 1409047936 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409048103 J * BenG ~bengreen@5751ac80.skybroadband.com 1409050469 Q * thierryp Remote host closed the connection 1409050490 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409050974 Q * thierryp Ping timeout: 480 seconds 1409053299 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409054539 M * adnauseam good day vserver folk 1409054606 M * Bertl and a good day to you too! 1409054721 M * adnauseam daniel_hozac: our network went down for 20 minutes and since then our vms aren't responding to external pings. (i'm trying to find out if they recevi pings from the host machine right now) 1409054740 M * adnauseam hows it going bertl! 1409054798 M * Bertl it's okay ... do you use network namespaces? 1409054967 M * adnauseam BenG: daniel_hozac: we tried new things today, all IPs were moved to an unused subnet, and they are pingable from the host machine 1409055126 M * adnauseam Bertl: i'm not aware that we do 1409055158 M * Bertl then 'pinging' the IP doesn't tell you much, it is basically pinging the host from the host 1409055160 M * adnauseam the ips ought to resolve to a host name though, if that's what you mean 1409055221 M * Bertl no, without network namespaces, all IPs reside on the host 1409055245 M * Bertl so basically pinging them from the host just verifies that the IP is there 1409055252 M * adnauseam i haven't seen the current configuration yet, but i'm talking to our admin as we speak 1409055277 M * adnauseam yeah, what's puzzling us is why they're not being reached externally, since the host machine can be 1409055291 M * Bertl well, network namespaces need to be configured, so if you don't know about them, it is unlikely that you are using them 1409055316 M * adnauseam oddly enough, after restarting the host yesterday, one VM did come back to life, but i'm not sure it's the case anymore after the new change 1409055331 M * Bertl it is simple to diagnose that, start a ping from outside, run tcpdump on the incoming interface 1409055333 M * adnauseam our admin knows more, i've asked him this and im wating for a reply 1409055356 M * Bertl maybe it would be a good idea to have your admin join here :) 1409055475 Q * thierryp Remote host closed the connection 1409055557 M * adnauseam trying to get him to get here, im not sure he uses irc ;p 1409055613 M * adnauseam i'm not sure he understood my question regarding namespaces, would not having a namespace cause this behaviour? the vms can ping the host, (i assume that's the host pinging itself?) but they can't ping external IPs 1409055680 M * renihs an admin, vserver admin even, that does _not_ use irc? 1409055695 M * Bertl yeah, very strange indeed :) 1409055712 M * Bertl anyway, let's take a step back and clarify the setup and problem 1409055728 M * Bertl adnauseam: so you have a host with a public IP I take it? 1409055748 M * Bertl and on that, a number of Linux-VServer guests, also with public IPs? 1409055881 M * adnauseam yes, a host with a public ip, and then several VMs with their own ips, which fail to reach external ips 1409055910 M * adnauseam our admin informed me he needs to go home, but will be available later 1409055929 M * Bertl LOL 1409055932 M * adnauseam he's setup a new VM with the config he knows to work, but that machine is also misbehaving 1409055949 M * Bertl what does VM mean here? 1409055953 M * adnauseam vserver ;p 1409055957 M * adnauseam vm, virtual machine 1409055962 M * adnauseam though i know it's a container, habbits 1409055988 M * Bertl okay, I thought he was maybe testing in a virtual machine 1409055998 M * adnauseam oh no, im the noob here :P 1409056005 M * Bertl so, the guests have public IPs as well, yes? 1409056032 M * adnauseam by guests do you mean the IPs sitting on the dummy device ? 1409056038 M * adnauseam or guests = vserver instances ? 1409056047 M * adnauseam well, they're all vserver instances really 1409056057 M * Bertl and the guest IPs are configured so that they show up on the host with the proper netmasks, etc (check with 'ip a l') 1409056062 M * adnauseam some sitting on eth0, some on dummy0, the ones on dummy0 are fine since they're natted though 1409056080 M * adnauseam will check as soon as the admin is back online 1409056092 M * adnauseam ip addr shows a "correct" configuration as far as documentation goes 1409056114 M * Bertl so you don't have access to the host? 1409056129 M * adnauseam no permissions yet, im the new guy :p 1409056163 M * Bertl but the one who knows how to IRC :) 1409056166 M * adnauseam but iworked yesterday with the admin and seen how it looks and can tell you what i know 1409056180 M * adnauseam aye, 90s kid ;p 1409056187 M * adnauseam dem darknet/dalnet dayz 1409056220 M * renihs without access to the problem box it will boil down to more or less educated guessing though 1409056273 M * adnauseam aye i realize that, i can only give the admin suggestions and he tries them out, yesterday we found out that the secondaries were not being promoted, so we have that fixed now. it was why the machines failed to begin with 1409056273 M * renihs so the guest (either on dummy or eth0) have "real" aka public ip addresses? 1409056288 M * renihs secondaries not being promoted? 1409056288 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409056298 M * adnauseam eth0 does, dummy0 is so we didnt look into it 1409056303 M * renihs of how old an kernel are we speaking here? :) 1409056322 M * adnauseam renihs: they are now, at least are set to. we've restarted the host and brought the vservers up, but the problem persisted 1409056359 M * adnauseam we then detached the physical link @ server room, thinking old configs were being cached somewhere, but that doesn't seem to have been the case 1409056368 M * adnauseam pretty old 1409056374 M * renihs cached? :) 1409056391 M * adnauseam well stored somewhere on the switch's chips ;p 1409056403 M * renihs well, i be sure to collect the output from like "ip addr show" and "ip route show table all" 1409056409 M * adnauseam it was a short in the dark, admin said it was once the case 1409056410 Q * BenG Quit: I Leave 1409056414 M * renihs and maybe "iptables-save -c" too :) 1409056417 M * adnauseam at least unplugging and replugging worked for that 1409056446 M * adnauseam rgr 1409056449 M * renihs if i were to bet i would say the problem is outside vserver :) 1409056453 M * adnauseam relayhing that 1409056474 M * renihs iptables-save -c just prints active rules and counters 1409056479 M * renihs doesnt do what it sounds like 1409056508 M * adnauseam we spoke with the network admin (not our department) and he said the logs show nothing wrong, that the host isn't blocked and that it's not misbehaving on their network, as far as they can tell 1409056550 M * Bertl as I said, it is rather simple to diagnose with tcpdump :) 1409056690 M * adnauseam never workd with tcpdump, ought to give that a go =} 1409056713 M * adnauseam networking usually baffles me. i tried to setup a network topology on a solatis machine and that alone was mind boggling 1409056718 M * Bertl I hope the admin has, otherwise I'm not too optimistic :) 1409056745 M * adnauseam he knows well more than i do, has worked much longer there as well and has a better general view. once he's home he'll join us here 1409056754 M * adnauseam i gave him the adress/channel 1409057019 M * renihs well, tcpdump is rather simple i suppose 1409057027 M * renihs interpreting might not be :p 1409057051 M * adnauseam reading a primer on it as we speak :} 1409057148 M * adnauseam btw, we're using squeezy, and the vserver version wehave is old, that much we know. what we also know is that it did work 2 days ago 1409057166 M * adnauseam what's going to happen to vserver now that it's deprecated ? 1409057178 M * adnauseam at least that;s what i've heard 1409057218 M * Bertl what is squeezy? 1409057282 M * renihs i think thats some bindistro codename 1409057284 M * adnauseam squeeze* debian's distribution 1409057296 M * renihs judging from the other admins behavioural pattern i would assume a debianeese 1409057301 M * adnauseam had both that and wheezy* in my head 1409057302 M * renihs oh 1409057309 M * adnauseam squeeze is old stable 1409060962 J * fisted_ ~fisted@xdsl-81-173-190-76.netcologne.de 1409061261 J * nelay ~circuser-@p5790491E.dip0.t-ipconnect.de 1409061283 Q * thierryp Remote host closed the connection 1409061306 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409061396 Q * thierryp Remote host closed the connection 1409061411 Q * fisted Ping timeout: 480 seconds 1409061411 N * fisted_ fisted 1409061416 J * thierryp ~thierry@lns-bzn-47f-62-147-212-202.adsl.proxad.net 1409061582 J * Wermwud ~Wermwud@69-29-150-18.stat.centurytel.net 1409061663 M * nelay I would like to provide more details to the problem user adnauseam was describing - with our vservers having routing issues. 1409061703 M * nelay the ip addresses configured seem to be fine with us. here is the output of "ip a l": 1409061715 M * nelay http://pastebin.com/wS0UQb5P 1409061742 M * nelay the public addresses starting with XXX.XX. are actual real public ip addresses owned by us 1409061830 M * nelay for example when i'm on the vserver using ip "XXX.XX..224.162" I cannot ping IP addresses in the internet. 1409061898 Q * thierryp Ping timeout: 480 seconds 1409061905 M * nelay the other vserver, which is configured the same way "XXX.XX.225.105" has no problems at all. 1409061946 M * nelay this is the weired thing. all the NATed vservers using the 10.0.0.xxx addresses are working fine. 1409062189 M * nelay when i issue a ping from one of the vservers and having tcpdump running on the host, i can see the ping request, however it never seems to reach its destination. 1409062227 M * nelay it seems to be that the problematic vservers cannot reach the default gateway 1409062312 J * BenG ~BenG@cpc29-aztw22-2-0-cust128.18-1.cable.virginm.net 1409062317 M * nelay but i have no idea why? and what to look after? any thoughts what i could try to troubleshoot the problem? 1409062790 M * nelay what could be a possible reason that one vserver cannot reach its gateway, while others can? 1409062823 M * DelTree just to be a fool for a minute... a service listening on 127.0.0.1 in a vserver cannot be reached from outside, right ? 1409063007 M * renihs DelTree, hum, yes it cant be reached 1409063023 M * DelTree how ? 1409063033 M * renihs it can not be reached :) 1409063036 M * renihs aka cant 1409063045 M * DelTree oops... 1409063058 M * renihs you said, "right" so i needed to add yes 1409063096 M * DelTree sorry... I just read too fast with no ' to catch my eye... _^ 1409063161 M * renihs nelay, hmm pastebin the output for "ip route show table all" so a better picture can be formed 1409063167 M * renihs i assume you cannot reach the default gw? 1409063194 M * nelay from the problematic vserver i cannot ping the default gw while on the others i can. 1409063263 M * renihs does the arp table (/proc/net/arp) show the physical address of the gw after pinging? 1409063372 M * nelay ip route show table all: http://pastebin.com/gmQW3nRJ 1409063488 M * nelay cat /proc/net/arp shows the same as on the host - so yes the MAC address for the gateway is also listed there 1409063551 M * renihs hmm abit confused why XXX.XX.225.109 has 2x src but looks fine else 1409063554 M * renihs so layer2 works as well 1409063596 M * renihs are there iptables rules in place? iptables-save -c might be usefull if 1409063601 M * renihs otherwise sniffing may shed some light 1409063645 M * nelay ah forget about that 109er - i just installed this vm for testing 1409063671 M * nelay with using prefix 32 - thats the reason it is listed twice right now 1409063689 M * renihs ah that explains 1409063778 M * renihs hmm well i would check fw rules now and probably target one vm with tcpdump and sniff the icmp session (ping of gw) 1409063824 M * nelay iptables: there are plenty of rules in place - however i also temporaily disabled all rules. and then the vservers should definitely be able to connect to the gateway. but the situation stayed the same. (of course the nat-ed vservers were not working any more because of the missing postrouting. 1409063839 M * renihs how did you disable? did you reset the POLICY? 1409063868 M * renihs trying to rule out really dumb things that can be overlooked easily 1409063896 M * renihs also, can the HOST reach the gw? (i assume yes since arp works) 1409063924 M * nelay we use the ferm package to confgure the firewall on that machine - stopping ferm should remove all rules and reset the policy. 1409063934 M * nelay iptables -L didnt show any rules as well. 1409063942 M * nelay host can reach the gw 1409063950 M * renihs and what did the policy say? do you have the output form the disabled session? 1409063967 M * nelay one of the bridged vservers can reach gateway as well 1409063981 M * nelay ALL nat-ed vservers can reach the gateway, too. 1409064016 M * renihs well, without knowing how the netfilter rules are flushed/reset its hard to say 1409064046 M * renihs if e.g Chain OUTPUT (policy DROP) is active after flushing/stopping ... 1409064086 M * renihs i am just trying to collect as much info as possible so when more knowledgable people wake up they can logread :) 1409064108 P * undefined 1409064109 M * nelay i will put the firewall rules on pastebin in a minute 1409064114 M * Bertl run tcpdump -vvnei eth0 1409064139 M * Bertl then use ping -I target 1409064150 M * Bertl and similar ping -I target 1409064153 M * Bertl (on the host) 1409064163 M * Bertl after that, upload the tcpdump output somewhere 1409064315 J * undefined ~undefined@00011a48.user.oftc.net 1409064375 M * nelay firewall rules: http://pastebin.com/mNqVJCqx 1409064584 M * renihs well, netfilter rules are not the cause (which i doubted but wanted to rule out) 1409064588 M * renihs nelay, what Bertl said :) 1409064649 M * renihs [4:240] -A INPUT -s XXX.XX.225.45/32 -p tcp -m tcp --dport 4949 -j ACCEPT <- indicates though that this has been hit 1409064672 M * renihs ahno i am blind, disregard, 1409064698 M * renihs why are these "-s"? 1409064713 M * renihs are you sure thats not supposed to be "--destination" 1409064727 M * nelay this port is only reachable by the givven source ip address. 1409064737 M * renihs yeah, its ok, all these XXX are confusing me 1409064746 M * nelay yeah - sorry about that... 1409064747 M * renihs anyhow, what bertl said :) 1409064764 M * nelay would be listening for icmp only be fine "tcpdump -vvnei eth0 icmp" 1409064774 M * nelay i have too much traffic on the machine right now :) 1409064925 M * renihs you can filter abit 1409064949 M * renihs tcpdump -vvnei eth0 icmp host and host 1409064951 M * renihs or like 1409064953 M * Bertl filter for icmp 1409065004 M * Bertl tcpdump -vvnei eth0 icmp 1409065292 Q * BenG Quit: I Leave 1409065379 M * nelay http://pastebin.com/pWV7MMYJ 1409065410 M * nelay to me it doesn't reveal much - when i ping from the host, i receive proper echo replies 1409065426 M * nelay when pinging from the problematic vserver ip address, i just don't receive any replies... 1409065469 M * Bertl well, it would help if you did what I asked you to do :) 1409065492 M * nelay didn't i? 1409065495 M * Bertl i.e. use ping -I on the host, once with the host IP (which works) and once with the guest IP (which doesn't work) 1409065502 M * nelay jup 1409065505 M * nelay i did that as well 1409065510 M * Bertl while having tcpdump running on the host 1409065517 M * nelay same output, but i can sent it again 1409065526 M * nelay i actually did both 1409065555 M * Bertl and if you do anonymize the output, please do it consistently 1409065589 M * renihs you didnt ping from the host with -I 1409065680 M * renihs well, i gotta go, should be back in abit i suppose 1409065685 M * nelay http://pastebin.com/evXdTCpi 1409065770 M * nelay the first part uses the problematic guest_ip => no reply 1409065783 M * nelay the second part in the pastebin uses the host_ip => correct replies 1409065829 M * nelay XXX.XXX. always stands for the same public prefix 1409065881 M * Bertl so the icmp request goes out quite fine, just nothing answers 1409065899 M * Bertl which implies that the routing back to the guest IP doesn't work 1409065903 M * nelay jepp 1409065918 M * Bertl what happens if you do the ping from outside to the guest IP? 1409065941 M * Bertl do you see anything with tcpdump coming in and what does the outside machine see? 1409065952 M * Bertl to me it looks like your network setup is broken 1409065963 M * Bertl i.e. the router/switch doesn't know where to send the reply 1409065988 M * Bertl but it could also be a firewall rule on the host which is supposed to reply (or somewhere on the way) 1409066018 Q * Aiken Ping timeout: 480 seconds 1409066019 M * nelay when i ping from the outside to the guest-ip, i see the request and also the reply in the tcpdump, however, it doesn't reach the external server from where i started the ping (=> 100% packet loss) 1409066128 M * nelay when i ping from the outside to one of the WORKING guest-ips, everything is fine 1409066170 M * nelay it is really weird... 1409066211 M * nelay maybe i have to talk again with the network operator. because so far I cannot identify that it is really a problem on my side (meaning on the host or the vserver configuration)... 1409066284 J * BenG ~bengreen@cpc29-aztw22-2-0-cust128.18-1.cable.virginm.net 1409066495 M * Bertl as you can see (with ping -I) it is unrelated to Linux-VServer 1409066511 M * Bertl i.e. you can even test that without a Linux-VServer enabled kernel and you will get the same results 1409066582 M * Bertl flushing the iptable rules and/or zeroing the counters and trying the ping (and watching the counters) rules out iptable magic 1409067694 M * nelay thanks to all of you for now! I will try to talk with the network operator tomorrow - so far we were sure the problem is related to linux-vserver, but thanks to your help, we understand that the problem might be really more general... 1409067835 M * Bertl you're welcome! 1409067855 M * Bertl off for a nap ... bbl 1409067869 N * Bertl Bertl_zZ 1409068930 J * renihs_ ~bla@chello084114030028.14.vie.surfer.at 1409069202 Q * nelay Remote host closed the connection 1409069341 J * bonbons ~bonbons@2001:a18:207:101:88d9:645b:72e9:4c2c 1409069672 J * nelay ~nelay@p5790491E.dip0.t-ipconnect.de 1409069681 Q * nelay 1409074364 J * thierryp ~thierry@home.parmentelat.net 1409074848 Q * thierryp Ping timeout: 480 seconds 1409075049 N * Bertl_zZ Bertl 1409075058 M * Bertl back ... 1409075321 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:24d7:72d4:3f2f:f1e7 1409076219 Q * Ghislain Quit: Leaving. 1409076509 Q * thierryp Remote host closed the connection 1409078922 Q * BenG Ping timeout: 480 seconds 1409079681 J * thierryp ~thierry@home.parmentelat.net 1409080163 Q * thierryp Ping timeout: 480 seconds 1409082442 Q * distemper Ping timeout: 480 seconds 1409082588 Q * Wermwud Quit: Leaving (Please imagine me slamming the door on my way out) 1409083281 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:799d:ab79:4a82:cf76 1409083762 Q * thierryp Ping timeout: 480 seconds 1409084521 J * Aiken ~Aiken@quarry.jbmb.net 1409084995 Q * bonbons Quit: Leaving 1409085081 J * thierryp ~thierry@home.parmentelat.net 1409085564 Q * thierryp Ping timeout: 480 seconds 1409088023 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:9d8c:d30b:769c:4aae 1409088507 Q * thierryp Ping timeout: 480 seconds 1409088682 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:41e6:61ff:340f:97b5 1409088790 J * thierryp_ ~thierry@2a01:e35:2e2b:e2c0:319f:3e52:cfb1:29a3 1409089162 Q * thierryp Ping timeout: 480 seconds 1409089273 Q * thierryp_ Ping timeout: 480 seconds 1409091775 Q * renihs_ Remote host closed the connection 1409092283 J * thierryp ~thierry@2a01:e35:2e2b:e2c0:2145:6e90:94f4:9df 1409092768 Q * thierryp Ping timeout: 480 seconds