CS Fundamentals Trainer · Guides
TCP in practice: connections, latency, timeouts and what breaks in production
What TCP does and does not promise, what a connection costs (handshakes, TLS, slow start, windows), the 40 ms stall of Nagle's algorithm with delayed ACKs, TIME_WAIT and port exhaustion, the accept backlog, half-open connections, keepalive and timeouts, with measurements from loopback experiments and a symptom table for diagnosing production problems.
30 min read
A request that normally takes 2 milliseconds sometimes takes 44. A service that handled load fine on Monday starts failing on Tuesday with "cannot assign requested address". A health check passes while users see timeouts. Each of these has the same kind of cause: the application treated the network as a function call, and TCP, the protocol underneath almost everything, behaves in ways that a function call does not. TCP is reliable and ordered, but it is also a byte stream with a startup cost, a memory, timers and limits, and those are the things that show up in production.
This guide explains TCP from the point of view of someone who writes and operates services: what a connection costs, what decides how fast data moves, the classic 40-millisecond stall (Nagle's algorithm meeting delayed acknowledgements), what the TIME_WAIT state is and when it runs a service out of ports, what happens when a server cannot keep up with new connections, and how to notice that the other side is gone. Several of these are measured on a real machine, and the code is in the appendix.
A note on measurements. The numbers marked "measured" come from Go 1.24 programs on a small cloud VM (2 vCPUs of an Intel Xeon at 2.1 GHz, 8 GB of memory, Linux 6 with default TCP settings except where stated), talking over the loopback interface. Loopback has no distance and no packet loss, so it shows the cost of the protocol and of the kernel, not the cost of a real network. That is the point of using it: it isolates behaviour (a stall, a limit, a timer) from the noise of a network. Where distance matters, the guide uses arithmetic that you can check, and says so.
1. What TCP promises, and what it does not
TCP gives an application a reliable, ordered stream of bytes between two endpoints. If bytes arrive, they arrive once and in order; lost packets are retransmitted; a sender that is too fast is slowed down. Four things that follow from this are often forgotten:
- A stream has no message boundaries. Two
writecalls of 4 and 100 bytes may arrive as onereadof 104 bytes, or as three reads of 40, 50 and 14. Messages are something the application builds on top (section 6). - "Sent" does not mean "received", and "received" does not mean "processed". A successful
writemeans the kernel accepted the bytes into a buffer. An acknowledgement means the other kernel has them, not that the other program has read them, and certainly not that it acted on them. Only an application-level answer says that. - TCP cannot tell you that the peer is gone, unless it tries to talk to it. A machine that loses power, or a cable that is cut, sends nothing; the other side sees silence (section 9).
- Reliability is paid for with delay. A lost packet makes everything behind it wait for the retransmission (head-of-line blocking), and connections start slow (section 4).
2. The life of a connection
Diagram: A TCP connection from open to close
The states that matter in practice:
| State | Meaning | If you see many of them |
|---|---|---|
| LISTEN | the server is waiting for connections | normal |
| SYN_SENT / SYN_RECV | a handshake is in progress | a flood of new connections, or an overloaded server (section 8) |
| ESTABLISHED | open, data may flow | normal; count them to size a connection tier |
| FIN_WAIT, CLOSING | this side has closed and waits for the peer | the peer is slow to close, or unreachable |
| TIME_WAIT | this side closed first and the connection is being kept for 60 s | many short connections (section 7) |
| CLOSE_WAIT | the peer closed, and the local program has not | a bug: the application is not closing its sockets |
CLOSE_WAIT deserves special mention. It means the other side finished and this program never called close. The sockets stay forever, and file descriptors slowly run out. It is never a network problem, always a leak in the program, typically a missing close or defer on an error path.
3. What a new connection costs
Opening a connection costs one round trip before the first byte of the request can be sent, and, with TLS, one or more round trips more. Distance multiplies it. The table is arithmetic: a response arrives after the number of round trips shown (the request itself takes one).
| Round-trip time | Reused connection | New TCP connection | New TCP connection + full TLS 1.3 handshake |
|---|---|---|---|
| 1 ms (same data centre) | 1 ms | 2 ms | 3 ms |
| 10 ms (same region) | 10 ms | 20 ms | 30 ms |
| 50 ms (across a continent) | 50 ms | 100 ms | 150 ms |
| 100 ms | 100 ms | 200 ms | 300 ms |
| 200 ms (intercontinental, mobile) | 200 ms | 400 ms | 600 ms |
There is also a CPU and kernel cost that does not depend on distance. Measured on loopback, one request of one byte took 9.5 microseconds on a reused connection and 68.5 microseconds when a new connection was opened and closed for each request: 7.2 times slower, with no network at all. (TLS adds certificate verification and key exchange, which cost much more.)
The consequence is the most important operational rule about TCP: do not open a connection per request. Use connection pools for databases and for HTTP clients, HTTP keep-alive (the default in HTTP/1.1) or HTTP/2 multiplexing, and TLS session resumption. A service that talks to ten other services should hold long-lived connections to each. An HTTP client created and thrown away for every call is a very common performance bug, and also the usual route to the port exhaustion of section 7.
4. How fast data moves: windows, slow start and the bandwidth-delay product
TCP limits how much data may be in flight, sent but not yet acknowledged. Two limits apply: the receive window (how much the receiver has room for, announced in every acknowledgement) and the congestion window (how much the sender believes the network can carry). The smaller of the two decides the speed.
This gives a hard ceiling that has nothing to do with the link's capacity: throughput ≤ window ÷ round-trip time. With a window of 64 KB and a round trip of 50 ms, at most 64 KB can be sent every 50 ms, which is 1.3 MB/s, or about 10 Mbit/s, even on a 10 Gbit/s link. To fill a link, the window must be as large as the bandwidth-delay product, bandwidth × round-trip time: a 1 Gbit/s path with 50 ms round trip holds 6.25 MB in flight. Modern systems scale the window automatically (the window scaling option and buffer auto-tuning), but the limit shows up with a small configured buffer, with a receiver that reads slowly, and with long-distance transfers.
New connections also start with a small congestion window and grow it with each round trip, a phase called slow start. The common starting size is 10 segments (about 14.6 KB), and it roughly doubles each round trip while nothing is lost. Counting the round trips needed for a response from a cold connection:
| Response size | Round trips (initial window of 10 segments, doubling, no loss) |
|---|---|
| 15 KB | 2 |
| 100 KB | 3 |
| 1 MB | 7 |
| 10 MB | 10 |
At 100 ms of round-trip time, a 1 MB response over a brand-new connection spends 700 ms just ramping up, before bandwidth matters at all. This is a second reason, after the handshake, to reuse connections: a warm connection has already grown its window. (Linux can reset the window after an idle period, which is why a pooled connection that has been idle for a while may be slow again; check the tcp_slow_start_after_idle setting on your hosts.)
Congestion control, the algorithm that decides how the window grows and shrinks on loss or delay, is a kernel setting. Linux hosts commonly use CUBIC, and this measurement VM was configured with BBR. You rarely tune it, but it explains why the same transfer behaves differently on different hosts.
5. The 40-millisecond stall: Nagle's algorithm and delayed acknowledgements
Two sensible optimisations, one on each side, combine into a famous pathology.
- Nagle's algorithm (sender side) saves the network from tiny packets: if there is unacknowledged data in flight, a small write is held back until an acknowledgement arrives, so that several small writes go out in one packet.
- Delayed acknowledgement (receiver side) saves the network from tiny acknowledgements: the receiver waits a little (up to 40 ms on Linux) hoping to attach the acknowledgement to a response it is about to send.
Now take a client that sends a request as two small writes, a 4-byte header and then a 100-byte body, and waits for the answer:
Diagram: Nagle plus delayed ACK: the stall
The client is waiting for an acknowledgement before sending the body, and the server is waiting for the body (or a timer) before acknowledging. Nothing is lost, and nothing is fast: the request spends about 40 ms in a deadlock that the delay timer breaks.
Measured on loopback, 300 requests on one connection, each sent as write(4 bytes), write(100 bytes), then wait for a one-byte answer:
| Configuration | Average per request | 99th percentile |
|---|---|---|
| Nagle on (the default socket), two writes | 44.02 ms | 47.32 ms |
TCP_NODELAY, two writes | 0.02 ms | 0.05 ms |
| Nagle on, one write of 104 bytes | 0.01 ms | 0.04 ms |
Same bytes, same machine, and a factor of two thousand between the first line and the others. A service with this pattern shows latency that is suspiciously stuck at a round 40 ms (or a multiple) and a CPU that is idle.
There are two fixes, and the second is better:
- Set
TCP_NODELAYon the socket, which turns Nagle off. Go does this by default for TCP connections; many other runtimes and libraries do not, so check the one you use. The cost is more small packets if the program writes in many tiny pieces. - Do not make several small writes for one message. Build the whole message in a buffer (a buffered writer, or gather the pieces into one write) and send it at once. This is faster in every configuration: fewer system calls, fewer packets, and no interaction with the timers at all.
The pattern to look out for has a name, write-write-read: two writes in a row, then a wait for the answer. It is common in code that sends a header and a body separately, in protocol implementations written straight from a specification, and in clients that flush a buffered writer in the middle of a message.
6. A stream has no messages: framing
Because TCP is a stream, the application must mark where one message ends. Real protocols do this in three ways: a length prefix (the message starts with its size, as in most binary protocols and in HTTP's Content-Length), a delimiter (a newline or a special byte sequence, as in line-based protocols, with escaping if the delimiter can occur in the data), or a fixed size. A length prefix is the simplest and the safest. The code below shows the three habits that matter: read exactly the number of bytes you need (io.ReadFull, because a single Read may return less), refuse absurd lengths before allocating memory, and write the length and payload with one write.
const maxMessage = 1 << 20
// readMessage reads one message that is sent as a 4-byte length followed by that many bytes.
func readMessage(r io.Reader) ([]byte, error) {
var n uint32
if err := binary.Read(r, binary.BigEndian, &n); err != nil {
return nil, err
}
if n > maxMessage { // never trust a length that came from the network
return nil, errors.New("message too large")
}
buf := make([]byte, n)
_, err := io.ReadFull(r, buf) // Read may return fewer bytes than asked; ReadFull loops until it has all of them
return buf, err
}
// writeMessage sends the length and the payload with one write, so Nagle's algorithm never holds back the second half.
func writeMessage(w io.Writer, payload []byte) error {
buf := make([]byte, 4+len(payload))
binary.BigEndian.PutUint32(buf, uint32(len(payload)))
copy(buf[4:], payload)
_, err := w.Write(buf)
return err
}
func call(addr string, payload []byte) ([]byte, error) {
d := net.Dialer{
Timeout: 3 * time.Second, // the handshake
KeepAlive: 30 * time.Second, // probes on an idle connection, well below a load balancer's idle timeout
}
c, err := d.Dial("tcp", addr)
if err != nil {
return nil, err
}
defer c.Close()
c.SetDeadline(time.Now().Add(5 * time.Second)) // one limit for the whole exchange
if err := writeMessage(c, payload); err != nil {
return nil, err
}
return readMessage(bufio.NewReader(c))
}
(The listing compiles; it is an illustration, not a complete protocol.) The same example shows the right way to set limits on a connection, which is the subject of the next sections: a dial timeout, a keep-alive period, and a deadline for the whole exchange.
7. Closing connections: TIME_WAIT and running out of ports
When a connection closes in an orderly way, the side that closes first does not forget it at once. It keeps the connection in the TIME_WAIT state for twice the maximum segment lifetime, which is 60 seconds on Linux. The reason is sound: a late duplicate packet from the old connection must not be taken for part of a new connection with the same addresses and ports, and the final acknowledgement may have been lost and need resending.
A TIME_WAIT socket costs little memory, but it occupies its local port towards that destination for the full 60 seconds. A client that connects to one server address has about 28,000 ephemeral ports (on this machine the range is 32768 to 60999, 28,232 ports). If 28,232 ports are each held for 60 seconds, the ceiling is about 470 new connections per second to one destination. Beyond that, connect fails with "cannot assign requested address".
Measured on loopback, a client opened 40,000 short connections to one address, closing each first. With the setting that lets the kernel reuse TIME_WAIT ports switched off:
| Setting | Connections attempted | Failed | Sockets in TIME_WAIT afterwards |
|---|---|---|---|
| Reuse of TIME_WAIT ports switched off | 40,000 | 8,091 ("cannot assign requested address") | 16,774 |
| Linux default on this host (reuse only for loopback) | 40,000 | 0 | 20,405 |
The default setting hides the problem on loopback, which is why a test on one machine passes and production fails: the kernel applies the reuse rule to loopback only, so the same code talking to a real remote server hits the port limit. In both runs, tens of thousands of sockets were sitting in TIME_WAIT at the end. (The failure count is below the 11,768 connections beyond the port range, because TIME_WAIT sockets from the beginning of the run expired while it was still running.)
What to do about it, in order of preference:
- Reuse connections (section 3). A pool turns thousands of connections per second into a few dozen.
- Spread across more destinations or source addresses if you truly need a very high connection rate: the limit applies per combination of addresses and destination port.
- Let the other side close first when you control the protocol. The TIME_WAIT is held by the side that closes first; for a protocol where the server can send "I am done" and close, the client is spared. (For servers, TIME_WAIT sockets do not use up ports in the same way, because every client has a different address.)
- Widen the ephemeral port range, a stopgap that moves the ceiling without removing the cause.
- Do not "fix" it by shortening TIME_WAIT or by dangerous options that ignore it (
tcp_tw_recyclewas removed from Linux because it broke connections behind NAT). The state exists for a reason.
A related and harmless setting: servers that restart and cannot bind their port because of old connections need SO_REUSEADDR on the listening socket, which most server libraries set for you.
8. When the server cannot keep up: the accept backlog
A server that listens keeps two queues for each listening socket: connections that are still shaking hands, and connections that have completed the handshake but not yet been picked up by the program's accept. The second queue has a limited length, the backlog. If the program is too busy to call accept fast enough (a blocked event loop, an exhausted thread pool), the queue fills, and further handshakes are ignored. The client does not get an error. Its kernel simply retransmits the first packet after a timer, 1 second, then 2 more, then 4 more, and so on.
Measured: a server that listens with a backlog of 1 and only starts accepting 2.5 seconds later. Four clients connect one after another.
| Client | Asked at | Connected after |
|---|---|---|
| 1 | 0.00 s | 0.000 s (queue had room) |
| 2 | 0.00 s | 0.000 s (queue had room) |
| 3 | 0.00 s | 3.061 s |
| 4 | 3.06 s | 0.000 s |
Connection 3 found the queue full at second 0. The server began accepting at 2.5 seconds, but the client was waiting for its retransmission timer, which fired at 1 second (queue still full) and at 3 seconds (success). A server that recovers still leaves some clients waiting for a retransmit timer, so overload shows up as latency steps of 1 s, 3 s, 7 s and as timeouts, while the server's own graphs show low load and fast responses for the requests it did process. The requests that never got in are invisible to its metrics.
What to look for and do:
- Latencies clustered at 1, 3 or 7 seconds point to dropped handshakes.
- Size the backlog sensibly. The actual queue length is the smaller of the program's value and the kernel limit (
net.core.somaxconn; on this host 4096), and the other queue for half-open handshakes has its own limit (tcp_max_syn_backlog, 512 here, with SYN cookies as the fallback). Many languages' standard servers already pick the kernel limit. - Do not hide overload with a deep queue. A long queue only turns "refused" into "slow". Apply load shedding and timeouts instead: reject quickly, and let the caller retry elsewhere.
- Make sure the accept loop never blocks on application work.
9. Knowing the other side is gone: keepalive, timeouts and half-open connections
If a peer crashes cleanly, the operating system closes its sockets and sends a FIN or RST, and you find out. If a machine loses power, is partitioned, or a NAT or firewall silently forgets the connection, nothing is sent, and your socket remains ESTABLISHED. This is a half-open connection. How long you wait depends on what you do:
- If you are sending data, the kernel retransmits and finally gives up after
tcp_retries2attempts, which with the default of 15 takes about 15 minutes. A write to a dead peer can seem to succeed for a long time. - If you are only reading, you wait forever, unless something probes the connection.
- The kernel's own keepalive probes an idle connection, but its defaults are very lazy: 7,200 seconds of idle time, then a probe every 75 seconds, up to 9 probes, which is more than two hours to notice a dead peer (these are the values on this host). Also, keepalive is off unless the program turns it on for the socket.
The conclusion is that you must set your own timeouts and not rely on the defaults:
- A connect timeout (seconds, not minutes).
- A deadline per request, covering the whole exchange (as in the code above), or at least a read timeout. Without it a single dead peer can hold a thread, a connection and a user's request for 15 minutes.
- Keepalive with a short period on long-lived idle connections (the example uses 30 seconds), or application-level heartbeats (a ping message every few seconds, which also tests that the other program, not just its kernel, is alive).
- Retries with limits and backoff, on top of idempotent operations (see the guide on idempotency).
A related trap sits in the middle of the network: load balancers, NAT gateways and firewalls drop idle connections silently after a time that is often between one and a few minutes. The next packet on such a connection is answered with a reset, or with nothing. Set the keepalive or heartbeat period below the shortest idle timeout on the path, and let connection pools discard connections that have been idle longer than that.
10. Reading the symptoms
| Symptom | Likely cause | Where to look |
|---|---|---|
| Latency stuck at about 40 ms, CPU idle | write-write-read with Nagle and delayed ACK | two writes per message; set TCP_NODELAY or merge the writes |
| Occasional latency of 1, 3 or 7 s; refused or timed-out connects | accept queue full, handshakes dropped | listen overflow counters in nstat or netstat -s; backlog; blocked accept loop |
| "Cannot assign requested address", thousands of TIME_WAIT | too many short connections from one source to one destination | ss -s; connection reuse and pooling |
| Growing number of sockets in CLOSE_WAIT | program does not close sockets | missing close on an error path; file descriptor count |
| Hangs of many minutes after a peer or network failure | no timeouts; half-open connection | deadlines, keepalive, heartbeats |
| Resets or hangs on connections idle for a minute or so | middlebox idle timeout | keepalive period below the idle timeout; pool expiry |
| Fast on the same machine, slow across regions | window and round-trip limits; new connection per request | bandwidth-delay product; reuse; larger buffers |
The tools to use: ss -tan (states and queue lengths per socket), ss -s (a summary of states), nstat or netstat -s (counters for retransmissions and listen overflows), tcpdump or Wireshark (the actual packets: the 40 ms gap and the retransmissions are visible at once), and, for HTTP, curl -w with timing variables to see the time of each phase (name lookup, connect, TLS, first byte).
11. Common mistakes
- Opening a connection for every request, and then being surprised by latency, CPU use and TIME_WAIT.
- Splitting a message into several small writes without a buffer; the stall of section 5.
- Assuming a
readreturns a whole message, or that awritewrites everything. Use framing and loop on partial reads and writes (io.ReadFull,io.Copy, a buffered writer). - Relying on default timeouts. There is almost no default that is right for a service: connect, read, write and idle all need explicit values.
- Treating a successful
writeor an ACK as "the other side did it." Wait for an application-level answer, and make the operation idempotent so that a retry is safe. - Leaking sockets, shown as CLOSE_WAIT and a rising file descriptor count.
- Testing only on loopback or in one data centre. Distance, loss and middleboxes create the failures; test the slow path on purpose.
- Turning dangerous kernel settings for a symptom (shortening TIME_WAIT, recycling) without understanding the cause.
- Making the accept queue deeper to fix overload, instead of rejecting early.
12. Things to mention in an interview
- TCP is a reliable, ordered byte stream: no message boundaries, no guarantee that the peer processed anything, and no way to detect a silent peer without timers.
- A new connection costs at least one round trip, plus TLS, plus slow start, so keep connections open and pool them. Give the numbers: at 100 ms round trip, a new connection with TLS 1.3 adds 200 ms before the request even leaves.
- Throughput is limited by window ÷ round-trip time; to fill a long fat pipe, the window must reach the bandwidth-delay product.
- The Nagle and delayed-ACK interaction (a 40 ms stall in a write-write-read pattern), and the two fixes:
TCP_NODELAYand one write per message. - TIME_WAIT: what it is for, who holds it, and how a client with many short connections runs out of ephemeral ports (about 470 new connections per second per destination with a 60 s hold and 28,000 ports). The cure is reuse.
- CLOSE_WAIT means a bug in your program.
- Describe the accept backlog and why overload appears as retransmission delays that the server's metrics do not show.
- Describe half-open connections and why every call needs a deadline, and how keepalive and heartbeats differ.
- Use measurements to make the point: "I measured a 44 ms stall that dropped to 0.02 ms."
Quick recap
- TCP gives a reliable ordered byte stream, not messages, and not proof that the peer acted. Add framing and application-level answers.
- A new connection costs round trips (1 for TCP, 1 more for TLS 1.3) plus slow start; measured 7 times the CPU cost of a reused connection even on loopback. Pool and reuse.
- Speed is bounded by window ÷ round-trip time; long distances need a window as big as the bandwidth-delay product.
- Nagle plus delayed ACK turns write-write-read into a 40 ms stall (measured 44 ms against 0.02 ms). Send each message in one write, and set
TCP_NODELAY. - TIME_WAIT lasts 60 s on the side that closes first; many short connections exhaust about 28,000 ports (measured: 8,091 of 40,000 connects failed when reuse was off).
- A full accept queue drops handshakes and shows up as 1, 3, 7 second delays (measured 3.06 s). Reject early instead of queueing deeply.
- Dead peers are silent. Set connect and request deadlines, short keepalive or heartbeats, and keep them below every idle timeout on the path.
Appendix: the experiment code
main.go below (called tcpx in the measurements) holds all four experiments, one per command: go run . nagle, go run . setup, go run . backlog and go run . timewait 40000. They use only the Go standard library and need Linux (the program reads /proc/net/tcp to count sockets by state). The port-exhaustion run was done twice: once with the default setting and once after writing 0 to /proc/sys/net/ipv4/tcp_tw_reuse (this needs root, and the setting should be restored to its previous value afterwards).
main.go
// tcpx: small experiments on TCP behaviour over the loopback interface (Linux). Usage: go run . nagle|setup|timewait|backlog
package main
import (
"bufio"
"errors"
"fmt"
"io"
"net"
"os"
"sort"
"strings"
"syscall"
"time"
)
func must(err error) {
if err != nil {
panic(err)
}
}
func pct(d []time.Duration, p float64) time.Duration {
s := append([]time.Duration(nil), d...)
sort.Slice(s, func(i, j int) bool { return s[i] < s[j] })
return s[int(float64(len(s)-1)*p)]
}
// ---------- 1. Nagle's algorithm and delayed acknowledgements ----------
// A server that reads a 4-byte header and a 100-byte body, then answers with one byte.
func requestServer(ln net.Listener, noDelay bool) {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func(c net.Conn) {
defer c.Close()
c.(*net.TCPConn).SetNoDelay(noDelay)
buf := make([]byte, 104)
for {
if _, err := io.ReadFull(c, buf); err != nil {
return
}
c.Write([]byte{1})
}
}(c)
}
}
func nagleRun(noDelay, oneWrite bool, n int) (avg, p99 time.Duration) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
must(err)
defer ln.Close()
go requestServer(ln, noDelay)
c, err := net.Dial("tcp", ln.Addr().String())
must(err)
defer c.Close()
c.(*net.TCPConn).SetNoDelay(noDelay)
hdr, body, both, one := make([]byte, 4), make([]byte, 100), make([]byte, 104), make([]byte, 1)
var lat []time.Duration
for i := 0; i < n; i++ {
t := time.Now()
if oneWrite {
c.Write(both)
} else {
c.Write(hdr) // two small writes, then wait for the answer: the classic write-write-read
c.Write(body)
}
io.ReadFull(c, one)
lat = append(lat, time.Since(t))
}
var sum time.Duration
for _, d := range lat {
sum += d
}
return sum / time.Duration(len(lat)), pct(lat, 0.99)
}
func nagle() {
fmt.Println("write(4 bytes), write(100 bytes), then wait for a 1-byte answer; 300 requests on one connection")
for _, tc := range []struct {
name string
noDelay, single bool
}{
{"Nagle on (default socket), two writes", false, false},
{"TCP_NODELAY, two writes", true, false},
{"Nagle on (default socket), one write ", false, true},
} {
avg, p99 := nagleRun(tc.noDelay, tc.single, 300)
fmt.Printf(" %-42s average %8.2f ms 99th percentile %8.2f ms\n", tc.name, float64(avg.Microseconds())/1000, float64(p99.Microseconds())/1000)
}
}
// ---------- 2. What a new connection costs ----------
func echoServer(ln net.Listener) {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func(c net.Conn) {
defer c.Close()
io.Copy(c, c)
}(c)
}
}
func setup() {
ln, err := net.Listen("tcp", "127.0.0.1:0")
must(err)
defer ln.Close()
go echoServer(ln)
addr := ln.Addr().String()
one := []byte{1}
rt := func(c net.Conn) { c.Write(one); io.ReadFull(c, make([]byte, 1)) }
const n = 5000
c, err := net.Dial("tcp", addr)
must(err)
t := time.Now()
for i := 0; i < n; i++ {
rt(c)
}
reuse := time.Since(t) / n
c.Close()
t = time.Now()
for i := 0; i < n; i++ {
c, err := net.Dial("tcp", addr)
must(err)
rt(c)
c.Close()
}
fresh := time.Since(t) / n
fmt.Printf("loopback, one request of 1 byte: reused connection %.1f us, new connection each time %.1f us (%.1fx)\n",
float64(reuse.Nanoseconds())/1000, float64(fresh.Nanoseconds())/1000, float64(fresh)/float64(reuse))
fmt.Println("(loopback has no distance, so this is the CPU and kernel cost only. On a real network every new connection also costs the round trips below.)")
fmt.Println()
fmt.Println("round trips before the first byte of a response, by round-trip time (RTT):")
fmt.Printf(" %-8s %-26s %-30s %-30s\n", "RTT", "reused connection", "new TCP connection", "new TCP + TLS 1.3 (full)")
for _, rtt := range []int{1, 10, 50, 100, 200} {
fmt.Printf(" %-8s %-26s %-30s %-30s\n", fmt.Sprintf("%d ms", rtt), fmt.Sprintf("%d ms", rtt), fmt.Sprintf("%d ms", 2*rtt), fmt.Sprintf("%d ms", 3*rtt))
}
// slow start: how many round trips to send 100 KB, 1 MB, 10 MB from a cold connection with an initial window of 10 segments
fmt.Println()
fmt.Println("slow start: round trips to send a response over a new connection (initial window 10 segments of 1,460 bytes, doubling every round trip, no loss)")
for _, size := range []int{15_000, 100_000, 1_000_000, 10_000_000} {
w, sent, rounds := 10*1460, 0, 0
for sent < size {
sent += w
w *= 2
rounds++
}
fmt.Printf(" %10s bytes: %d round trips\n", commas(size), rounds)
}
}
func commas(n int) string {
s := fmt.Sprint(n)
var out []string
for len(s) > 3 {
out = append([]string{s[len(s)-3:]}, out...)
s = s[:len(s)-3]
}
return strings.Join(append([]string{s}, out...), ",")
}
// ---------- 3. TIME_WAIT ----------
func countStates() map[string]int {
names := map[string]string{"01": "ESTABLISHED", "06": "TIME_WAIT", "08": "CLOSE_WAIT", "0A": "LISTEN"}
out := map[string]int{}
for _, f := range []string{"/proc/net/tcp", "/proc/net/tcp6"} {
fh, err := os.Open(f)
if err != nil {
continue
}
sc := bufio.NewScanner(fh)
sc.Scan()
for sc.Scan() {
fs := strings.Fields(sc.Text())
if len(fs) > 3 {
if n, ok := names[fs[3]]; ok {
out[n]++
}
}
}
fh.Close()
}
return out
}
func timewait() {
ln, err := net.Listen("tcp", "127.0.0.1:0")
must(err)
defer ln.Close()
go func() { // server: read to EOF, then close (so the CLIENT closes first and keeps the TIME_WAIT)
for {
c, err := ln.Accept()
if err != nil {
return
}
go func(c net.Conn) { io.Copy(io.Discard, c); c.Close() }(c)
}
}()
addr := ln.Addr().String()
before := countStates()["TIME_WAIT"]
n := 20000
if len(os.Args) > 2 {
fmt.Sscan(os.Args[2], &n)
}
t := time.Now()
failures := 0
var firstErr error
for i := 0; i < n; i++ {
c, err := net.DialTimeout("tcp", addr, 2*time.Second)
if err != nil {
failures++
if firstErr == nil {
firstErr = err
}
continue
}
c.Close()
}
el := time.Since(t)
st := countStates()
fmt.Printf("%d short connections to one address, the client closing first, in %.1f s (%.0f per second)\n", n, el.Seconds(), float64(n)/el.Seconds())
fmt.Printf(" sockets in TIME_WAIT: %d before, %d after (failed connects: %d)\n", before, st["TIME_WAIT"], failures)
if firstErr != nil {
fmt.Println(" first error:", firstErr)
}
lo, hi := portRange()
fmt.Printf(" ephemeral port range %d-%d = %d ports; a port towards one destination is blocked for 60 s after a close, so the ceiling is about %d new connections per second per destination\n",
lo, hi, hi-lo+1, (hi-lo+1)/60)
}
func portRange() (int, int) {
b, _ := os.ReadFile("/proc/sys/net/ipv4/ip_local_port_range")
var lo, hi int
fmt.Sscan(string(b), &lo, &hi)
return lo, hi
}
// ---------- 4. The accept backlog ----------
func backlog() {
fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, 0)
must(err)
defer syscall.Close(fd)
must(syscall.Bind(fd, &syscall.SockaddrInet4{Port: 0, Addr: [4]byte{127, 0, 0, 1}}))
must(syscall.Listen(fd, 1)) // backlog of 1, and the program never calls accept
sa, _ := syscall.Getsockname(fd)
addr := fmt.Sprintf("127.0.0.1:%d", sa.(*syscall.SockaddrInet4).Port)
fmt.Println("a server that listens with a backlog of 1 and does not call accept at first; clients connect one after the other; accept starts after 2.5 s")
// The server starts accepting after 2.5 seconds, which empties the queue.
go func() {
time.Sleep(2500 * time.Millisecond)
for {
nfd, _, err := syscall.Accept(fd)
if err != nil {
return
}
syscall.Close(nfd)
}
}()
var keep []net.Conn
start := time.Now()
for i := 1; i <= 4; i++ {
t := time.Now()
c, err := net.DialTimeout("tcp", addr, 10*time.Second)
d := time.Since(t)
if err != nil {
var ne net.Error
kind := "error"
if errors.As(err, &ne) && ne.Timeout() {
kind = "timed out"
}
fmt.Printf(" connection %d: %s after %.2f s\n", i, kind, d.Seconds())
continue
}
keep = append(keep, c)
fmt.Printf(" connection %d: asked at %.2f s, connected after %.3f s\n", i, t.Sub(start).Seconds(), d.Seconds())
}
for _, c := range keep {
c.Close()
}
}
func main() {
if len(os.Args) < 2 {
fmt.Println("usage: tcpx nagle|setup|timewait|backlog")
return
}
switch os.Args[1] {
case "nagle":
nagle()
case "setup":
setup()
case "timewait":
timewait()
case "backlog":
backlog()
}
}
Sources
- Wesley M. Eddy (ed.), RFC 9293: Transmission Control Protocol (TCP), 2022, the current specification: states, TIME_WAIT, retransmission.
- John Nagle, RFC 896: Congestion Control in IP/TCP Internetworks, 1984 (the Nagle algorithm), and RFC 1122: Requirements for Internet Hosts, 1989 (delayed acknowledgements, keepalive).
- RFC 5681: TCP Congestion Control (slow start) and RFC 6928: Increasing TCP's Initial Window (initial window of 10 segments).
- RFC 7323: TCP Extensions for High Performance (window scaling) and RFC 1337: TIME-WAIT Assassination Hazards in TCP.
- The Linux manual page tcp(7): the settings
tcp_tw_reuse,tcp_retries2,tcp_keepalive_*,tcp_max_syn_backlogandTCP_NODELAY. - Neal Cardwell et al., BBR: Congestion-Based Congestion Control, ACM Queue, 2016.
- W. Richard Stevens, TCP/IP Illustrated, Volume 1: The Protocols (Addison-Wesley), the chapters on connection establishment and termination, Nagle and delayed acknowledgements.
- The Go documentation for the net package (
TCPConn.SetNoDelay,Dialer.KeepAlive, deadlines).
Practise this topic: Networking & APIs