TCP Abnormal Disconnection

Actually there are two ways to release a TCP connection. The normal way to terminate a TCP connection is for one side to send a FIN segment, and it follows the normal releasing procedure. It’s also ok to send a segment which is called RST segment to release the connection immedietly. The difference between these two ways is that the normal release operation guarantee that the data transferred is reliable, the RST segment cut off the connection imedietly which mostly cause the data loss. Sometimes RST segment is involved into the communication software architecture on purpose, but most of the time when RST is caught in a network, it indicates that something goes wrong in the network.

Experiment Network Topology

Host IP addresses TCP port number
Server 10.22.5.3 55555
Client 10.16.56.2 1982

Case 1: Connection not establishment, server process not start

In this case , we presume that TCP/IP protocal stack is ok, the server process is not running, client sends the SYN to start connection request, SYN segment arrives at the TCP statck of the server. Case there’s no process listening on the specific port refering to the port in the SYN segment,the TCP layer can do nothing but reject the connection request. At this time, the server side TCP layer use the RST segment to response the SYN segment. After the client receive the RST, the TCP layer of the client will report to the application layer with the message “connect error: connection refused”. In the RST segment, besides the RST flag bit is set, the sequence number is set to 0, acknowledge number is equal “SYN Sequence number + 1”.

Case 2: Send a RST segment with socket API

As other segments, RST segment can be sent mannully with the socket API,

1
2
3
4
5
6
7
8
Connect(sockfd, (SA *) &servaddr, sizeof(servaddr));

struct linger ling;
ling.l_onoff = 1;
ling.l_linger = 0;
Setsockopt(sockfd, SOL_SOCKET, SO_LINGER, &ling, sizeof(ling));

Close(sockfd);

Here is a wireshark packets capture, after the three handshake, the client send a RST segment.

Case 3: Connection not establishment, server host not start

It’s different from the server process not running, the server host is power off. At this moment, the server TCP stack is not ready at all. After the client sends the SYN request. SYN will mostly lost in the network, the packet will keep routing in the network. SYN will not acknowledged by the remote side, client tcp stack will resend the SYN for several times, and keep waiting some time between these resending operations. The time is twice as the last time waiting. e.g. client 10.22.5.3 start a connection to server 10.16.56.2, after seven failures, the connect() api report an error, “connect error: Connection timed out”.

The time waiting sequence are 1s, 2s, 4s, 8s, 16s, 32s and the last 64s with an error to the application layer. And the total time of retrying is 6. the retries setting is differ according to the os kernels. The ubuntu is as this,

1
2
toto@guru:~$ sysctl net.ipv4.tcp_syn_retries 
net.ipv4.tcp_syn_retries = 6

Case 4: Client process crashes

If the client process crashes at different moments, it will result different consequences. For instance, the client process crashes at SYN_SENT status, the client sends a SYN and quits, the server process responses with SYN+ACK, waits for the will-never-received-ACK, the server tcp stack will start the retry procedure. If the client side does this deliberately, it is actually just another kind of SYN flood attacking. If the client process crash at ESTABLISHED status, the socket file descriptor of the client process will release by the client os kernel, and a FIN segment will be sent by tcp stack.

Case 5: Connection established, server process crashes

The server process crashes, all the file descriptors will be released by the server os kernel. And server will send a FIN to all the clients, client will answer an ACK, the unidirection pipe from the server to client is shut down. The server tcp status is FIN_WAIT2. The server stack is waiting for the FIN from the client. The client tcp status is CLOSE_WAIT, the client stack is waiting for the application layer to notify that there’s no more application data. The pipe from client to server is still keep opened. This is the TCP half-close situation.

The client TCP is waiting for the EOF from application. if the client application layer doesn’t send an EOF to TCP layer, the client tcp stack will stuck in the CLOSE_WAIT status. At the same time, the server will stay in the FIN_WAIT2 for a few moment, and after timeout, the satus transit to CLOSED.

1
2
3
4
5
6
7
//Server
toto@guru:~$ netstat -ant | grep 1982
tcp 0 0 10.22.5.3:1982 10.16.56.2:33383 FIN_WAIT2

//Client
toto@client:~$ netstat -ant | grep 1982
tcp 0 0 10.16.56.2:33383 10.22.5.3:1982 CLOSE_WAIT

When the server side is in FIN_WAIT2, if the client application call the function Write,which make the the message data arrive at the server tcp statck, but the server is expecting to get a FIN, so the server return a RST segment to the client to end the connection. The timer for FIN_WAIT2 could be manually set.

1
2
toto@guru:~$ sysctl net.ipv4.tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 60

After the FIN_WAIT2 timer time out, check the status of both endpoints.

1
2
3
4
5
6
//Server
toto@guru:~$ netstat -ant | grep 1982

//Client
toto@client:~$ netstat -ant | grep 1982
tcp 0 0 10.16.56.2:33383 10.22.5.3:1982 CLOSE_WAIT

Case 6: Connection not established, router crashes

This case only discussed according to my topology, other situation should be considered according to the contexts. If the APR cache is still there, TCP SYN request will retry for several time, and at the end, an error message “connect error: Connection timed out” print out. If the ARP cache is cleared, the function Connect will return “connect error: No route to host”.

Case 7: Connection established, router crashes

In order to simulate this case. We run the server process, run client process in debug mode step by step to function Connect, after Connect return, the 3-way-handshake finishes. Checking both endpoints state are ESTABLISHED. At this moment, power off the router or disable the ports of the router to kill the network layer transportation. Check that both TCP endpoints state are still ESTABLISHED. The client calls function Write to send a request message. Write is not block, it will provide the application data to TCP stack and return immediatly, and the programme will block at Readline wating for the feedback from server. Now, the network layer is not working.TCP layer will retransmit the data. If the router is kept locking, the application message will never arrive at the server.

If one of the host is powered off in this situation, the opponent will keep the TCP connection for ever, the resource will never be released. This is a situation called TCP half open. If there are too many half opened TCP connection on a server, the resources will run out. So the keep-alive mechanism is brought in.