Systematic four-layer debugging sequence — DNS, ICMP, TCP port, application auth — using purpose-built tools on both Linux and PowerShell. Covers the most common failure signatures and how to read them.
Systematic debugging walkthroughs — layer-by-layer sequences for both platforms: DNS → ICMP → TCP port → app auth → GCP firewall rule inspection; the “works from my machine” failure taxonomy
Operations and safety — use TCP port tests (nc -zv, Test-NetConnection -Port) as the primary diagnostic when ICMP is blocked on GCP; curl exits 0 on HTTP 4xx/5xx (use -f or -w '%{http_code}'); DNS caching persists until TTL expires after record changes; 3 warnings, 1 troubleshooting table (5 symptoms), 1 recommendations table (6 scenarios)
Glossary
ping
Command-line tool that sends ICMP Echo Request packets to a host and reports whether replies are received, along with round-trip timing.
Used to test basic IP-layer reachability and latency before moving to TCP-port or application-level diagnosis.
ICMP is frequently blocked
Many cloud environments and corporate firewalls drop ICMP by default. A host that does not respond to ping may still be fully reachable on TCP ports.
traceroute / tracert
Route-discovery tool that maps the path to a destination hop by hop by sending packets with increasing TTL values; each router that decrements TTL to zero returns an ICMP “time exceeded” response.
Used to identify where latency, routing failure, or packet loss begins along the network path.
Probe type differs by implementation
Linux traceroute commonly uses UDP probes by default, while Windows tracert uses ICMP Echo. Many firewalls treat those differently. On Linux, traceroute -T uses TCP SYN probes that more closely resemble real application traffic.
nc (netcat)
General-purpose TCP/UDP utility that can open connections, listen for connections, transfer raw data, or in scan mode (-z) test whether a remote port accepts a TCP connection.
Used to verify TCP port reachability before application-level testing, especially to distinguish “connection refused” from “timed out.”
Refused vs timed out are not equivalent
“Connection refused” means the host is reachable and actively rejected the connection — typically because nothing is listening on that port. “Connection timed out” usually means packets were dropped by a firewall or the host was unreachable.
/dev/tcp
Bash redirection feature that treats paths of the form /dev/tcp/<host>/<port> as a request to open a TCP connection, even though no real file exists on disk.
Used for minimal TCP port tests in environments where nc or telnet is not installed.
Bash-only feature
/dev/tcp is a Bash extension; it is not available in POSIX sh, dash, and is not enabled by default in most other shells.
dig
DNS query tool that can request specific record types and display answer, authority, and additional sections, along with TTL and resolver details.
Used for scriptable DNS validation because its output is stable and can be narrowed to exactly the records needed.
+short can return aliases as well as final answers
dig +short hostname may return a CNAME and then the resolved address records. Query a specific type such as dig +short hostname A or dig +short hostname AAAA when you need only final IP answers of one family.
mtr
Network diagnostic tool that combines repeated reachability probes with hop-by-hop path discovery, producing per-hop statistics such as packet loss and average latency over time.
Used to detect sustained path instability or intermittent packet loss that a single traceroute run may miss.
Install on Debian/Ubuntu
mtr is not installed by default. Install with apt install mtr. Use -r -c 10 for a non-interactive report suitable for logs or tickets.
ss
Linux utility that reads socket state from kernel networking interfaces, typically via netlink, and reports listening sockets, established sessions, queues, and owning processes.
Used to confirm whether a service is actually bound to the expected local port before attempting remote connectivity tests.
Replaces netstat
netstat is legacy on modern Linux systems. ss is generally faster and more complete for current socket inspection workflows.
Test-NetConnection
PowerShell diagnostic cmdlet that can test basic reachability, TCP port connectivity, and optionally trace the route to a destination, returning structured output.
Used on Windows as the primary interactive equivalent of combining ping, TCP port testing, and route checks in one command.
Slow for bulk port sweeps
Each Test-NetConnection call waits for built-in timeout behavior. Sweeping many unreachable ports can therefore be slow; for scripted sweeps, a direct .NET socket approach is usually faster.
Resolve-DnsName
PowerShell cmdlet that queries DNS and returns structured records containing properties such as name, type, TTL, and resolved data.
Used as the PowerShell equivalent of dig when scripts need typed DNS results instead of parsing raw command output.
Structured output vs text parsing
Unlike nslookup, Resolve-DnsName returns objects that can be filtered, sorted, and piped directly in PowerShell without fragile text parsing.
Get-NetTCPConnection
PowerShell cmdlet that reads TCP connection and listener state from Windows, returning objects with local and remote addresses, ports, state, and owning process ID.
Used on the server side to confirm that a service is listening before testing it remotely.
Requires joining with Get-Process
Get-NetTCPConnection returns OwningProcess as a PID, not a process name. Join it with Get-Process when a human-readable process identity is needed.
ICMP (Internet Control Message Protocol)
Network-layer control protocol used for diagnostic and error-reporting messages such as Echo Request, Echo Reply, Destination Unreachable, and Time Exceeded.
Used by tools like ping and traceroute to test reachability and reveal path behavior independently of any specific TCP or UDP application.
ICMP block does not mean host down
A host can ignore or block ICMP and still serve HTTP, SSH, or database traffic normally. A ping timeout alone does not prove the host is unavailable.
TTL (Time To Live)
Context-dependent field name used in two different places: in IP packets it is a hop limit decremented by each router; in DNS records it is a cache lifetime in seconds.
Used in IP networking by traceroute to reveal each hop, and in DNS to control how long resolvers keep an answer before querying again.
Two unrelated concepts share the same abbreviation
IP TTL and DNS TTL are separate mechanisms. One limits packet lifetime in transit; the other controls DNS cache duration.
GCP firewall rules
Google Cloud VPC firewall rules are stateful network rules that allow or deny ingress and egress traffic based on direction, protocol, ports, source, destination, and target scope before packets reach the VM guest operating system.
Used to control which traffic is allowed into or out of VM instances at the VPC level, and therefore a primary check when remote connectivity times out in GCP.
Enforced before guest OS firewalls
VPC firewall rules are evaluated in Google Cloud’s virtual network path before traffic reaches the VM. A packet blocked there never reaches iptables, nftables, or Windows Firewall inside the instance, so both layers may need to be checked separately.
The first question in any network debugging session is: “Can my client reach the server at all?” This seems simple, but there are multiple layers that can fail: DNS resolution, TCP routing, firewall rules, and the service itself. Working through the layers systematically turns a 2-hour debugging session into a 5-minute one.
flowchart TD
A([Start: can I reach the server?]) --> B[Step 1: DNS<br>dig +short hostname]
B --> B1{Resolves?}
B1 -- No --> B2[Check /etc/resolv.conf<br>VPC DNS settings]
B1 -- Yes --> C[Step 2: ICMP<br>ping -c 3 IP]
C --> C1{Responds?}
C1 -- Timeout --> C2[Possibly blocked by firewall<br>Skip to Step 3 on GCP]
C1 -- Yes --> D[Step 3: TCP port<br>nc -zv -w 5 IP port]
C2 --> D
D --> D1{Port state?}
D1 -- Refused --> D2[Host alive, service not running<br>or wrong port]
D1 -- Timed out --> D3[Firewall blocking port<br>or wrong IP]
D1 -- Succeeded --> E[Step 4: App auth<br>sqlcmd / psql / redis-cli]
E --> E1{Auth OK?}
E1 -- Login failed --> E2[Wrong credentials<br>or insufficient privileges]
E1 -- Cannot open DB --> E3[Database does not exist]
E1 -- Timeout --> E4[Connection pool exhausted<br>or server overloaded]
E1 -- Connected --> F([Connectivity confirmed])
Linux connectivity testing tools
This section covers the core Linux tools for diagnosing network connectivity at each layer: nc and /dev/tcp for TCP port reachability, dig for DNS resolution, traceroute and mtr for path tracing, and ss for inspecting local listening ports.
Linux | nc | port reachability and sweeping
nc (netcat) is the primary tool for testing whether a TCP port is open on a remote host. It can also sweep multiple ports in a loop, making it useful as a first diagnostic step after firewall changes or when connecting to a new VM.
Test whether a single port is open
-z runs nc in scan mode — it opens and immediately closes the connection without sending any data, which is the correct way to probe port availability. -v enables verbose output so the result is printed to stderr. -w 5 sets a 5-second timeout to avoid hanging on unreachable hosts.
nc -zv -w 5 hostname 1433
Connection to hostname (10.132.0.2) 1433 port [tcp/ms-sql-s] succeeded!
Refused vs timed out diagnosis
Refused = the host is reachable but nothing is listening on that port (service down, wrong port number).
Timed out = packets are being dropped (firewall rule, host unreachable, wrong IP).
“Refused” is good news — the host is alive. “Timed out” means a network-layer problem.
Test a port without netcat installed
Bash exposes a built-in TCP pseudo-device at /dev/tcp/<host>/<port>. Opening this path attempts a TCP connection — no external tools required. This works on minimal containers and Docker images where netcat is not installed. timeout 5 prevents the shell from hanging if the host is unreachable.
When connecting to a new environment or diagnosing after firewall changes, sweeping all relevant data engineering ports at once is faster than testing them individually. The loop below checks the most common ports and filters the verbose output to only show the result line.
for port in 1433 5432 6379 8080; do nc -zv -w 3 hostname $port 2>&1 | grep -E "succeeded|refused|timed out"done
Connection to hostname 1433 port [tcp/ms-sql-s] succeeded!Connection to hostname 5432 port [tcp/postgresql] succeeded!nc: connect to hostname port 6379 (tcp) timed out: Operation now in progressConnection to hostname 8080 port [tcp/http-alt] refused
Flag
Syntax
Description
-z
nc -z host port
Scan mode — connect and close immediately without sending data
-v
nc -v host port
Verbose — print result to stderr
-w <n>
nc -w 5 host port
Timeout after n seconds; prevents hanging on unreachable hosts
-u
nc -u host port
Use UDP instead of TCP
-l
nc -l -p port
Listen mode — start a server on the given port
-p
nc -p 4444 host port
Specify the local source port
-n
nc -n host port
Numeric only — skip DNS resolution
-k
nc -k -l -p port
Keep listening after the first connection closes (multi-client)
Linux | dig | DNS resolution
dig (Domain Information Groper) is the standard tool for querying DNS records. It supports all record types (A, AAAA, CNAME, MX, TXT, NS, SOA) and lets you target specific resolvers.
Look up the IP address of a hostname
+short strips all metadata from the output and returns only the answer section — the resolved IP address or CNAME target.
dig +short hostname
10.132.0.2
Query a specific record type
Appending a record type (A, AAAA, MX, TXT, NS, CNAME) constrains the query to that type. This avoids ambiguity when +short returns a CNAME instead of an IP.
dig hostname A
; <<>> DiG 9.18.12 <<>> hostname A;; ANSWER SECTION:hostname. 300 IN A 10.132.0.2
Query a specific DNS server
@8.8.8.8 overrides the system resolver and queries Google’s public DNS directly. Use this when you suspect the local resolver has stale cache entries or is misconfigured.
dig @8.8.8.8 hostname
; <<>> DiG 9.18.12 <<>> @8.8.8.8 hostname; (1 server found);; ANSWER SECTION:hostname. 299 IN A 203.0.113.10
dig +short may return a CNAME, not an IP
If the hostname is an alias, dig +short hostname returns the CNAME target rather than the final IP address. The returned CNAME must itself be resolved in a second query.
Force A-record resolution directly
Use dig +short hostname A to force resolution to the A record, skipping intermediate CNAME output. This returns the IP even if the hostname is a CNAME alias.
Flag
Syntax
Description
+short
dig +short hostname
Return only the answer — no metadata
+noall +answer
dig +noall +answer hostname
Show only the answer section with full TTL and record data
@<server>
dig @8.8.8.8 hostname
Query a specific DNS server instead of the system resolver
-x
dig -x 10.132.0.2
Reverse DNS lookup (IP to hostname)
+trace
dig +trace hostname
Trace the full delegation path from root servers
+dnssec
dig +dnssec hostname
Request DNSSEC signatures in the response
+tcp
dig +tcp hostname
Force query over TCP instead of UDP
Linux | traceroute | network path tracing
traceroute shows each hop between the local machine and the destination, reporting the round-trip time for each router. If the trace stops at a specific hop, that is where a firewall or routing failure is occurring. Hops displaying *** are blocking ICMP or dropping probe packets.
Trace the path to a host using TCP
-T switches traceroute from its default UDP probes to TCP SYN packets. TCP probes are more likely to pass through firewalls than UDP or ICMP, especially on GCP where ICMP is blocked between VPCs by default.
traceroute -T hostname
traceroute to hostname (10.132.0.2), 30 hops max, 60 byte packets 1 10.142.0.1 0.421 ms 0.398 ms 0.387 ms 2 172.16.0.1 1.234 ms 1.218 ms 1.201 ms 3 10.132.0.2 2.105 ms 2.089 ms 2.074 ms
traceroute uses UDP probes by default
The default UDP mode is frequently blocked by corporate firewalls and GCP VPC rules. Hops that show *** may actually be reachable — they are just silently dropping UDP probes.
Use TCP mode for reliable results on GCP
Run traceroute -T hostname to send TCP SYN packets. These follow the same path as real application traffic and are not filtered by GCP’s default ICMP rules.
Flag
Syntax
Description
-T
traceroute -T host
Use TCP SYN packets instead of UDP datagrams
-I
traceroute -I host
Use ICMP ECHO probes instead of UDP
-p <port>
traceroute -T -p 1433 host
Specify destination port (TCP/UDP mode)
-m <n>
traceroute -m 20 host
Set the maximum TTL / number of hops (default 30)
-n
traceroute -n host
Do not resolve hostnames — show IP addresses only
-w <n>
traceroute -w 2 host
Wait n seconds for a response per probe
-q <n>
traceroute -q 1 host
Send n probes per hop (default 3); use 1 for faster output
Linux | mtr | real-time path analysis
mtr combines ping and traceroute into a continuously updating display. It probes each hop repeatedly and accumulates statistics on loss percentage, average latency, and jitter. A sudden latency spike at a specific hop indicates a bottleneck at that router. Packet loss at a hop indicates congestion or active packet drops. Install with apt install mtr.
Run a fixed-count path analysis
-c 10 sends 10 probes per hop and then exits. Without -c, mtr runs interactively until interrupted. Use -c in scripts or when you need a one-shot report.
-r (report mode) combined with -c runs mtr non-interactively and prints the summary table to stdout. Use this in shell scripts or when piping output elsewhere.
Report mode — non-interactive, prints summary to stdout
-n
mtr -n host
Do not resolve hostnames — show IPs only
-T
mtr -T host
Use TCP SYN probes instead of ICMP
-P <port>
mtr -T -P 1433 host
Destination port for TCP/UDP probes
-b
mtr -b host
Show both hostnames and IP addresses
-i <n>
mtr -i 0.5 host
Probe interval in seconds (default 1)
-u
mtr -u host
Use UDP datagrams instead of ICMP
Linux | ss | local listening ports
ss is the modern replacement for netstat. It reads socket state directly from the kernel and is significantly faster than netstat on hosts with many connections. Use it to verify that a service is actually bound to the expected port before attempting remote connectivity tests.
List all listening TCP sockets with process names
-t filters to TCP sockets, -l to listening state only, -n disables hostname/port resolution (shows raw IPs and port numbers), and -p shows the owning process name and PID. For deeper connection state analysis, see socket-inspection.
Appending sport = :<port> limits output to sockets bound to that port. This is faster than piping through grep on hosts with hundreds of connections.
ss -tlnp sport = :1433
State Recv-Q Send-Q Local Address:Port Peer Address:Port ProcessLISTEN 0 128 0.0.0.0:1433 0.0.0.0:* users:(("sqlservr",pid=1234,fd=3))
Flag
Syntax
Description
-t
ss -t
Show TCP sockets only
-u
ss -u
Show UDP sockets only
-l
ss -l
Show only listening sockets
-n
ss -n
Do not resolve hostnames or port names
-p
ss -p
Show process name and PID for each socket
-a
ss -a
Show all sockets (listening and established)
-s
ss -s
Print a summary of socket counts by state
sport = :<n>
ss -tlnp sport = :1433
Filter by local port number
dport = :<n>
ss -tnp dport = :443
Filter by remote (destination) port number
Linux | systematic debugging walkthrough
Connectivity failures rarely announce their cause. The most efficient approach is to work up the network stack layer by layer, confirming each one before moving to the next. Each step below narrows the problem space.
Step 1 — resolve the hostname
DNS failure produces an empty result. An empty response means /etc/resolv.conf is misconfigured, the VPC DNS settings are wrong, or the hostname has no record in the relevant zone.
dig +short data-pipeline-sql
10.132.0.2
Step 2 — confirm ICMP reachability
ping verifies basic IP routing. On GCP, ICMP is blocked by default — a timeout here does not confirm the host is down. Always proceed to Step 3 regardless.
ping -c 3 10.132.0.2
PING 10.132.0.2 (10.132.0.2) 56(84) bytes of data.64 bytes from 10.132.0.2: icmp_seq=1 ttl=64 time=0.421 ms64 bytes from 10.132.0.2: icmp_seq=2 ttl=64 time=0.398 ms64 bytes from 10.132.0.2: icmp_seq=3 ttl=64 time=0.387 ms--- 10.132.0.2 ping statistics ---3 packets transmitted, 3 received, 0% packet loss
ping uses ICMP, which GCP blocks by default
A ping timeout does not mean the host is unreachable. GCP’s default firewall rules block ICMP between VPCs. A timeout at this step is inconclusive.
Skip to TCP port test on GCP
Go directly to Step 3 (nc -zv) when working in GCP environments. TCP port tests use the same path as real application traffic and are not blocked by the default ICMP deny rule.
Step 3 — test the TCP port
A refused response means the host is alive but the service is not listening on that port. A timeout means a firewall rule is dropping packets between client and server.
nc -zv -w 5 10.132.0.2 1433
Connection to 10.132.0.2 1433 port [tcp/ms-sql-s] succeeded!
Step 4 — authenticate at the application layer
If the TCP connection succeeds but the application rejects the connection, the problem is in credentials, database existence, or server capacity. -l 10 sets a 10-second login timeout. See sql-server-authentication for login types and troubleshooting.
If Step 3 times out, there is no application-level fix — the firewall must allow the port first. The command below lists all ingress rules and their allowed ports.
gcloud compute firewall-rules list --filter="direction=INGRESS" --format="table(name,network,direction,allowed,sourceRanges)"
NAME NETWORK DIRECTION ALLOW RANGESallow-sql-internal default INGRESS tcp:1433 10.0.0.0/8allow-ssh default INGRESS tcp:22 0.0.0.0/0default-allow-internal default INGRESS tcp,udp,icmp 10.128.0.0/9
Linux | the “works from my machine” problem
If a pipeline fails to connect but a manual test from the same VM succeeds, the failure is not network-level — it is environmental. Four causes account for the vast majority of these cases.
"Works from my machine" — four hidden differences
User context: The manual test runs as your user; the pipeline runs as a service account or container user. Different users may have different network namespaces (Docker), proxy settings (http_proxy env var), or firewall policies.
DNS: Your /etc/hosts may resolve the hostname to a local entry that the pipeline container does not have.
Connection pool exhaustion: The pipeline may have consumed all available connections. Check with SELECT COUNT(*) FROM sys.dm_exec_sessions WHERE is_user_process = 1.
TCP keepalive: Idle connections through a load balancer or NAT gateway are silently dropped after a timeout (often 5 minutes on GCP). The pipeline holds a stale connection handle that the network has already closed. Fix: set the connection pool’s idle timeout lower than the NAT timeout.
Isolate the environmental difference
Run the connection test as the same user and inside the same network namespace as the pipeline. For Docker-based pipelines: docker exec -it <container> nc -zv <host> <port>. Compare the results to your manual test.
PowerShell connectivity testing tools
This section covers the PowerShell-native equivalents: Test-NetConnection for TCP port testing and route tracing, Resolve-DnsName for DNS queries, and Get-NetTCPConnection for inspecting local listening sockets. These cmdlets are available on all supported Windows versions and in PowerShell 7+ on Linux.
PowerShell | Test-NetConnection | port reachability
Test-NetConnection is the primary PowerShell cmdlet for testing TCP connectivity. It returns a structured object with TcpTestSucceeded, SourceAddress, RemoteAddress, and PingSucceeded properties, making it suitable for both interactive debugging and scripted checks.
Test whether a port is open
The cmdlet prints a formatted summary to the console and returns the result object to the pipeline. TcpTestSucceeded: True confirms that the three-way TCP handshake completed.
Suppressing warnings with -WarningAction SilentlyContinue prevents the cmdlet from printing “WARNING: TCP connect to (hostname : 1433) failed” on a refused connection, which would pollute script output. The parentheses force evaluation before accessing the property.
Each call has a built-in connection timeout. Sweeping ten ports with Test-NetConnection can take 30+ seconds if any are unreachable.
Use TcpClient for faster programmatic checks
The [System.Net.Sockets.TcpClient] class connects immediately and throws on failure with a configurable timeout, making it the correct choice for scripted sweeps:
Resolve-DnsName is the PowerShell equivalent of dig. It returns structured objects with Name, Type, IPAddress, and TTL properties, which can be filtered and piped without parsing text output.
Look up the IP address of a hostname
Without specifying a record type, the cmdlet returns all records found, including CNAME chains resolved to their final A or AAAA records.
Resolve-DnsName hostname
Name Type TTL Section IPAddress---- ---- --- ------- ---------hostname A 300 Answer 10.132.0.2
Query a specific record type
Passing -Type constrains the query to a single record type, matching dig hostname A behavior.
Resolve-DnsName hostname -Type A
Name Type TTL Section IPAddress---- ---- --- ------- ---------hostname A 300 Answer 10.132.0.2
Query a specific DNS server
-Server overrides the system resolver and directs the query to a specific nameserver — the equivalent of dig @8.8.8.8 hostname.
Resolve-DnsName hostname -Server 8.8.8.8
Name Type TTL Section IPAddress---- ---- --- ------- ---------hostname A 299 Answer 203.0.113.10
Parameter
Syntax
Description
-Name
-Name hostname
Hostname or IP to resolve
-Type
-Type A
DNS record type: A, AAAA, CNAME, MX, TXT, NS, SOA
-Server
-Server 8.8.8.8
Use a specific DNS server instead of the system resolver
Test-NetConnection with -TraceRoute performs a hop-by-hop trace to the destination. The result object contains a TraceRoute property with the IP address of each hop.
PowerShell | Get-NetTCPConnection | local listening ports
Get-NetTCPConnection is the PowerShell equivalent of ss -tlnp. It reads TCP socket state from the system and can be joined with Get-Process to map each socket to a process name.
List all listening sockets with process names
The @{N='Process';E=...} calculated property looks up the process name by PID, producing a combined view equivalent to ss -tlnp.
For a broader systematic diagnosis approach that goes beyond network connectivity into application and query-level troubleshooting, see troubleshooting-flowcharts.
Warnings
Generic probes can mislead
Three common false signals appear during connectivity testing:
ICMP reachability is not TCP reachability. Many cloud VMs and corporate firewalls block ICMP. A host that does not respond to ping may still be fully reachable on HTTP, SSH, or database ports.
curl exit code 0 is not HTTP success.curl https://api.example.com/data exits successfully even when the server returns 404 or 500, unless you opt into failure-on-HTTP-error behavior.
DNS answers may be cached.dig can show the current authoritative record while your application or workstation still uses an older cached IP until TTL expiry.
A green or red result from the wrong probe is not enough to prove the path is healthy or broken.
Match the probe to the protocol
Use the check that proves the exact layer you care about:
TCP reachability. Test the real port with nc -zv host port or Test-NetConnection -Port.
HTTP success. Use curl -sf -o /dev/null -w '%{http_code}' https://endpoint/health so transport success and HTTP success are not conflated.
DNS changes. Compare dig output with the local resolver view, and flush caches with systemd-resolve --flush-caches on Linux or Clear-DnsClientCache on PowerShell when validating a recent change.
The recommendations table below applies the same rule to the common day-to-day diagnostics.
Recommendations
Scenario
Recommendation
Basic reachability
ping -c 4 <host> (Linux) or Test-NetConnection <host> (PowerShell).