Security - We will get there

Have you ever experienced that your remote desktop client could not connect to some of your windows servers? And most of the time the traffic was going through VPN tunnel. You have tried all kinds of troubleshooting tools such as ping, telnet etc and they all worked well. And your sniffer told you the tcp session on port 3389 was established too. But wait a second, some of the packets could not be seen from the other side. What is that?

If you look closer, these packets all share the same character. They are larger than others. Typically over 1400 bytes. Let's read why these larger packets get dropped.

"Some noncompliant routers silently drop IP datagrams that cannot be fragmented or do not correctly report their next-hop MTU.

To work around these problematic devices, changes can be made to the Windows Server 2003 TCP/IP stack by editing these registry values within the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters key:

• EnablePMTUBHDetect. This value adjusts the PMTU discovery algorithm to attempt to detect noncompliant routers, also called PMTU black hole routers. PMTU black hole detection is disabled by default but can be enabled by adding this value to the registry key and setting it to 1.

• EnablePMTUDiscovery. This value enables or disables the PMTU discovery mechanism, helping to diagnose problems with black hole routers. PMTU discovery is enabled by default but can be disabled by adding this value to the registry key and setting it to 0.

  PMTU discovery is enabled so that the two sides of a conversation can negotiate the most efficient MTU. When PMTU discovery is disabled, an MSS of 536 bytes and an MTU of 540 bytes are used for all non-local destination addresses.

  Note  On nonsecure networks, allowing PMTU discovery carries the risk that an attacker might force the MTU to a very small value and overwork the local system's TCP/IP stack."

This works across windows platforms including xp, vista, 2003 etc. Not tested on 2008 and windos 7 yet but they should apply the same principle if you encounter the similar issue.

One paticular note: this often happens in the vpn scenarios because VPN servers need to add header before the normal packet. And most of the network devices silently drop the over sized packet when don't fragment (DF) is set. I have seen this for so many times and it took me quite a while to fix it when I first saw it when troubleshooting a intranet application issue accross large partner's network.

Two options to fix this problem, I will normally try to set the EnablePMTUBHDetect first. If this doesn't solve the problem (sometimes it will happen), try to setup static MTU size on that network interface, normally it works.

You can contact me if you need to fix your network/infrastructure problems.