If we take snapshot of netstat (netstat -nP tcp) the common
states we see would be ESTABLISHED, TIME_WAIT, CLOSE_WAIT.
I will try explaining what they mean.
Before we go into TIME_WAIT and CLOSE_WAIT, lets take close look at sequence of steps for socket closing.
Socket connection is essentially between two peers (Browser to webserver, a java client to webserver, webserver to DB server, a webserver to another webserver etc )
Say there is a socket connection established between webserver1 and webserver2.
This would be the closing sequence, once the data transfer is done: (From TCP sequence diagram)
Here I am assuming webserver1 initiates the close of connection.
1) Socket on webserver1 sends a TCP segment with FIN bit (in TCP header)
and the socket goes into FIN_WAIT_1 state.
2) Socket on webserver2 receives the FIN and responds back with ACK to acknowledge the FIN and
the socket goes to CLOSE_WAIT state.
Now until the application calls the close() on this socket this is going to be in CLOSE_WAIT state.
3) Socket on webserver1 receives the ACK and changes to FIN_WAIT_2
4) Socket on webserver2 closes the connection(once the application calls close())
and sends back FIN to its peer to close the connection and changes its state to Last Ack
5) Socket on webserver1 receives the FIN and sends back ACK.
At this point the socket implementation on webserver1 would start a timer (TIME_WAIT) to handle
the scenario where last ACK has been lost and server resends FIN.
Now the socket would wait for 2* MSL (Maximum segment lifetime- default is 4mins for solaris & windows)
6) Socket on webserver2 receives the ACK and it moves the connection to closed state
7) After TIME_WAIT is elapsed socket/connection will be closed on webserver1.
These multiple levels of acknowledgments & retransmits are needed
since TCP is a reliable protocol unlike basic UDP
Here is what the three states mean:
ESTABLISHED:
This is pretty explanatory which basically means the two ends are in a state where
data transfer can occur or occurring in both directions. (tcp socket is full duplex, i.e data can be received and responded to on same channel)
CLOSE_WAIT:
This is a state where socket is waiting for the application to execute close()
CLOSE_WAIT is not something that can be configured where as TIME_WAIT can be set through tcp_time_wait_interval (The attribute tcp_close_wait_interval has nothing to do with close_wait state and
this was renamed to tcp_time_wait_interval starting from Solaris 7)
A socket can be in CLOSE_WAIT state indefinitely until the application closes it.
Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to
pile up of close_wait sockets. (At java level, this manifests as "Too many open files" error)
TIME_WAIT:
This is just a time based wait on socket before closing down the connection permanently.
Under most circumstances, sockets in TIME_WAIT is nothing to worry about.
References:
http://www.port80software.com/200ok/archive/2004/12/07/205.aspx
Subscribe to:
Post Comments (Atom)
12 comments:
nicely explained.
Thanks, perfectly addressed my issue
Very good information; presented in in a very easy to understand way.
Good work.
- RJoshi
in theory that is all correct, however there are also cases where sockets stay in CLOSE_WAIT state although the application called close() on the socket.
not sure if those cases are due to JDK bugs. It seems that the CLOSE_WAIT state is usually left when the application called close() AND the garbage collector freed the resources.
I have seen the same application behaving perfectly correct on one system but piling up CLOSE_WAITs on another
Hi Micha,
Yes, there could be a case where sockets stay in CLOSE_WAIT state though an application called close() on the socket. As explained in TCP sequence flow, Socket on webserver2 closes the connection(once the application calls close()). So there could be scenario where application called close() but web server not being execute close() on socket or filedescriptor being leaked.
Please correct if my understanding is wrong.
-Ankur Agarwal
very helpful... thanks a lot for the information
Hi,
How do we resolve this issue?
How can I resolve the close_wait issue and prevent this from happening again?
Kind regards
Maphefo
how to stop close_wait in the scenario Micha has explained. thats even if application has closed but close_wait don't stops?
we are facing very frequently close_wait issue in MDW platform.As per temporary fix every time we are restarting servers..
Request you kindly feedback us your valuable suggestions to fix close waits permanently..
Hello you should try to capture comunication via wireshark (https://www.wireshark.org/) and inspect what is going on.
Please note that the above descriptions are somehow simplified. TCP connection is normally full-duplex, but either side can close it's part of connection which leads to so-called half-closed connection.
In this scenarion you can have connection in CLOSE_WAIT for wery long time and it CAN be legal.
Normaly side A sends FIN, side B responds with ACK+FIN and finally side A sends ACK.
The side B however may acknowledge that side A wishes to close it's end of connection, but may not want to close it's side of connection. Then it responds to FIN request from A with packet that has only ACK flag set. Side A then has read-only connection and side B has write-only connection. If that happens then netstat on side A will show FIN_WAIT_2 state - as side A still waits for the 2nd FIN required to fully close the connection. Netstat on side B will show CLOSE_WAIT since the connection si no longer ESTABLISHED, but side B did not closed his side of connection.
From application point of view you send FIN when you call close() on the socket. If you do that on ESTABLISHED connection then the call will block until the other side closes the connection as well. Yet while the thread calling close() is blocked you can still receive data from the connection - usually delivered to some kind of listener or as result to calling read() from on the socket - but this will always happen on other thread then the one which called close(). This is what happens when you are at side A - eg. you are closing the connection.
If you are at side B you will receive -1 from read() call - if you are using synchromous API or get some closed() notification if you provided listener. It is common mistake to assume that this means that the connection is closed. What it really means is that the othed side (side A) closed the connection. You still have to call close() on your side of connection to free all resources.
So if you are seeing CLOSE_WAIT connection it might mean that there is poorly written application (on the side where you see it) ot it might be pretty legal.
---
You might ask, as many others on the Internet, if the half-closed connections are any good. Well - there are prefectly reasonable situations for this. And they usually does no go with networking issues, but rather programming issues. Eg. you close your side of connection early so you do not have to keep reference to the socket. This is common if you use asynchronous API with listeners. It might be hard to pair writing and reading threads together or the properly synchronize them.
But asking such question is actually useless. TCP allows it and since you usually do not have control over your peers, and since they can use half-closed connections you have to handle the situation correctly. Problem is what you can do if you want to close the connection, but the other side keeps its part open. Then you end up in FIN_WAIT_2.
Have a look what Apache does: http://httpd.apache.org/docs/2.0/misc/fin_wait_2.html
Have you been thinking about the power sources and the tiles whom use blocks I wanted to thank you for this great read!! I definitely enjoyed every little bit of it and I have you bookmarked to check out the new stuff you post
Selenium training in Chennai
Selenium training in Bangalore
I like that your blog. Is very nice content.
Python Training in Chennai
Python Training in Bangalore
Python Training in Hyderabad
Python Training in Coimbatore
Python Training
python online training
python flask training
python flask online training
Post a Comment