Problem Solving Rush
Hello,
I’m fresh off a problem solving rush and I just felt like talking about it. A problem solving rush, in case it’s not obvious, is how I describe the great feeling you get from solving some problem that has been bugging you for a while without much help from external sources (ie a step by step guide). It doesn’t matter how simple the issue to those in the know, if it bugs you and you fix it, it feels great!
This afternoon I SSH’d into my router/web server and found that if I ran any commands that wrote a bunch of text to the console (ie most commands) the terminal would freeze and I would have to kill it. This happened on every machine I tried at work. When I got home I tried the same thing from inside my network and it worked fine. So from that I assumed that it was either a firewall issue or a bandwidth issue, now I am on DSL so I didn’t really believe the bandwidth possibility. So I started to analyze my firewall log to see if anything interesting showed up. After removing some extraneous log messages and retrying the problem by SSH’ing into work then back home again I was able to see some interesting entries show up. These entries where very weird.
May 20 19:46:33 DarkCloud INVALID INPUT IN=eth0 OUT= AC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0×00 PREC=0×00 TTL=64 ID=4550 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxx.xxx.xxx.xxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61201 FRAG:64 PROTO=TCP ] MTU=1492
immediately you can see some weird things like there is a packet within the packet (the [ ] part). The inner packet originates from my router and is going out to my work computer so I can send OK but what is the outer part? The outer packet is from my actual DSL router to my router so it seems my traffic is not leaving my house.
[ASIDE] Yes, in my house we have a DSL router that provides 4 connections, one to each roommates room. My connection goes into my router where I send it out over my local hub to my multitude of computers on my desk (3 including the router).[/ASIDE]
OK so if traffic is not leaving my house then there must be something wrong somewhere and it must have to do with my firewall somehow. The next thing I looked at was the PROTO=ICMP part. I went on google and searched around for ICMP and inner packet coming up with something called an ICMP Echo error packet. What? It turns out that if at a certain point in your packets journey to its destination has a problem with your packet it will drop it and sent an ICMP error back. OK so I am getting ICMP errors from my DSL router, woopidy-doo — what does it all mean Basil?
I figured I had good luck googling before lets try googling on TYPE=3 and CODE=4 to see if I can figure out what the reason is for the ICMP error response. I quickly found an error code reference that explained by way of a table that 3,4= fragmentation needed. Fragmentation needed? Huh? I thought fragmentation was bad!? What gives! So again to google to find out what this is all about. Again quickly I found a short paragraph on packet fragmentation, simple enough really. If a point on a packets journey does not accept packets of a certain size then it will tell the sending computer that in order for the traffic to pass that point it must be broken down into smaller packets before sending. This is allowed by the TCP/IP protocol but apparently is little known to the general public. So my console buffer was just too big to send through my DSL router. Now what? It suddenly occurred to me that my firewall was picking up these messages so they obviously were not going to the sshd service. Maybe it can react dynamically if it actually finds out about the problem. Hey, it’s worth a try!
I added a rule to accept fragmentation-needed ICMP error packets and all of a sudden ssh is working again without issue. RUSH STRIKES HERE!! Man that was a dooosey.
Well until next time. Oh, one more thing. I just wanted to share a quote I really enjoy with you so here it is.
“Against stupidity, the Gods themselves contend in vain.”
– Friedrich von Schiller
I can’t admit that I pulled this obscure reference out of some arcane tomb because I simple got it from Isaac Asimov. I’m a heavy Sci-Fi reader and one really good book is titled “The Gods Themselves” by Asimov. If you like Sci-Fi you need to read this book, it’s great.