Insights Troubleshooting Blocked Ports

Troubleshooting Blocked Ports

One of the more common networking issues we look at is where some or all ports between servers are being blocked.  Typically, this is due to a misconfiguration, but being able to find the details of the issue is invaluable in tracking down the cause.

Basic Port Tests

When testing for blocked ports, it’s important to ensure that the server you’re connecting to is actually listening on the given port.  This can be confirmed with the “netstat -na” command, where you’ll see the port in a LISTENING state, or by testing from another device that is able to connect.  Once you’ve confirmed the server is listening, either with netstat or by testing that some clients are able to connect, you can proceed with troubleshooting blocked ports.

If you are having an issue and know the port number that’s failing, such as RDP/3389 or HTTP/80, it’s easy to test for connectivity to that specific port.

FOR SERVER 2012 R2+

The easiest method is the PowerShell command Test-NetConnection.  This will attempt a TCP connection to a specific hostname or IP on a specific port.  The success is noted in the “TcpTestSucceeded” property.  Here’s an example of a successful test using HTTP port 80 against google.com:

FOR OLDER OSES

On older operating systems, you’ll have to resort to using telnet to test port availability.  When telnet successfully connects, you’ll see the CMD screen reset.  You may still get disconnected immediately, for example if the server expects your client to immediately send some specific data.  However if the command prompt clears/resets, that indicates that the port is not blocked.

An example of a successful connection:

An example of a successful connection, where the server disconnects automatically:

An example of a failed connection that indicates the TCP port is not open:

Advanced Testing

There are some scenarios where Test-NetConnection and telnet aren’t totally adequate for testing port availability.

MULTIPLE PORTS

If there are multiple ports to test, it can be burdensome to use the above commands to test them repeatedly while troubleshooting.  This is where PortQryUI comes in handy.  Most commonly, we use PortQryUI to test for the availability of Active Directory ports.  There are a number of ports that need to be tested, and PortQryUI makes it easy to test all of them.  For example, here we test the default AD ports to the DC named MGLABDC5.  LISTENING indicates a successful connection.

A failed or closed port is usually indicated by a status of FILTERED instead of LISTENING.

UNKNOWN PORTS

If you aren’t sure what port is failing, you can use the netstat command on the client-end to look for incomplete connections.  For example, many Windows services listen on random high-numbered ports.  To find blocked ports in these cases, you can look for a connection in the SYN_SENT state on the client side.  SYN_SENT indicates that the client sent the first part of the TCP handshake but never received a response.  The most common cause of this is that the port is being blocked.  Here, we are trying to connect Event Viewer from one server to another, but the port is being blocked.  This is what the client-side netstat shows in that case:

SYN_SENT shows that this server is trying to initiate a connection with 10.250.0.12 on port 49665, but it never received an acknowledgement of the initial TCP packet.

PACKET CAPTURES

If you need to document in-depth evidence of a port issue, you can run concurrent packet captures on the client and server using Wireshark, netsh, etc.  After starting the packet capture on both sides, attempt to connect from the client to the server a few times.  After waiting a few seconds for the connections to be attempted, stop the packet captures.  When you compare them, you should see some packets leaving the client but never arriving on the destination server.  This would indicate they’re getting dropped or blocked somewhere along the way.

Troubleshooting Process

Generally speaking, troubleshooting blocked ports follows this process:

  1. Is the server listening on the expected port?  This can be confirmed with “netstat -na” on the server or through other evidence such as other clients successfully connecting.
  2. Can the server connect to itself on the expected port?  Test with telnet or Test-NetConnection using 127.0.0.1 or localhost.
  3. Can the client ping the server?  Note, that not all environments allow pings so this isn’t a conclusive test.
  4. If using the hostname of the server, can the client successfully resolve it?  Use “nslookup <servername>” to confirm.
  5. Is the Windows Firewall blocking the connection?  Temporarily disable on the server if needed to test.
  6. If you know the port number of the service that is failing, does Test-NetConnection or telnet show it is being blocked?
  7. If there are a number of ports required, does PortQryUI show them all as LISTENING?
  8. If you don’t know the port, do you see any broken connections in the SYN_SENT state on the client?
  9. If all else fails, run concurrent packet captures on client/server and compare them.

By running through all of the above, plenty of information can be gathered to show whether or not a network device is blocking the given port(s) between client and server.

General Tips

Here are a few general tips:

  • For most connections, the client uses a random high-numbered port for the outbound connection.  However, not all services do.  For example, Windows Time Service uses a client port of 123 for outgoing NTP connections, which is the same as the server port.
  • Although DNS typically goes over UDP port 53, it can also use TCP port 53 in certain circumstances. 
  • The troubleshooting information in this article only specifically applies to TCP connections.  Since UDP connections aren’t stateful, they can’t be tested using the above methods.  For example, telnet and Test-NetConnection can only be used to test for TCP ports.  Some examples of UDP services are NTP, DNS, SNMP, and SIP.  These services have to be tested through other methods.