Problem with: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

Question

I am trying to write a Python program using a client-server model that supports uploading a file from client to server by fragmenting the file and for each fragment, create a thread to send it.

I try to fragment the file into 32 KB chunks.

    def __init__(self, host, port, threads=5):
        self.host = host
        self.port = port
        self.fragment_size = 1024 * 32
        self.threads = threads

    def upload_file(self):
        file_name = filedialog.askopenfilename(title="Select file to upload")
        if file_name:
            with ThreadPoolExecutor(max_workers=self.threads) as executor:
                executor.submit(self.send_file, file_name)

    def send_file(self, filename):
        try:
            file_size = os.path.getsize(filename)
            fragment_count = (file_size // self.fragment_size) + (1 if file_size % self.fragment_size != 0 else 0)

            if not self.send_metadata(filename, file_size, fragment_count):
                raise Exception("Failed to send metadata.")

            with open(filename, 'rb') as file:
                with ThreadPoolExecutor(max_workers=self.threads) as executor:
                    futures = []
                    for fragment_number in range(fragment_count):
                        fragment = file.read(self.fragment_size)
                        futures.append(executor.submit(self.send_fragment, fragment, fragment_number, fragment_count))

                    for future in futures:
                        future.result()

        except Exception as e:
            messagebox.showerror("Error", f"Failed to upload file: {e}")

And for each send_fragment thread, I create a new socket:

client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client_socket.connect((self.host, self.port))

The code seems to work correctly with small files, but when it comes to large files (larger than 500MB), this error occurs:

[WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted.

The requirement of this program is to fragment the file and create a thread to send each fragment.

"And for each send_fragment thread, I create a new socket" - why? You should send the entire file using a single socket. Trying to fragment it over multiple connections is going to cause headaches on both sides. In any case, you do not need to use SO_REUSEADDR on a TCP client socket, especially when you are not explicitly bind()'ing it. That socket option is more useful for listening server sockets instead — Remy Lebeau
– Remy Lebeau, Commented Aug 9, 2024 at 15:24
Thank you for your response. Actually, my teacher required the file to be fragmented and sent these fragments concurrently — Rachal
– Rachal, Commented Aug 9, 2024 at 15:30
So is there any solution to it while maintaining the main task: fragmenting the file and create a thread to send each fragment — Rachal
– Rachal, Commented Aug 9, 2024 at 15:36

Remy Lebeau · Accepted Answer · 2024-08-09 17:01:37Z

0

Every time you call connect() on a socket, by default it will bind to a random local ephemeral port that is available. When the socket is closed, it may take the OS a few moments to release the port. If you run netstat on the command-line, you will likely see a lot of ports in TIME_WAIT state. That happens while the OS is holding the ports open to flush out any pending packets on the network.

Having send_fragment() use a new socket means that sending a 500+ MB file fragmented into 32K chunks will use 16000+ TCP connections. Creating and closing so many connections in a short amount of time will easily exhaust the range of available ephemeral ports.

To address this, you could try throttling the threads to introduce some delay after each connection is closed, so the local ports have more time to be released.

But a better design is to instead reuse your TCP connections. Send multiple fragments per connection (you will have to delimit your fragments so the server can identify where one fragment ends and the next begins). Put the sockets into a pool, and have each thread poll a socket from that pool when needed. When a fragment is done sending, don't close its socket, put it back in the pool. Don't create and connect a new socket unless the pool is empty.

Since you are running only 5 threads at a time, you should be able to send all 16000+ fragments using just 5 TCP connections rather than 16000+ connections.

If you don't want to write your own socket pool, there are 3rd party socket pools available for Python, for instance this pool and this pool.

edited Aug 9, 2024 at 17:01

answered Aug 9, 2024 at 15:39

Remy Lebeau

609k36 gold badges516 silver badges875 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rachal Over a year ago

"Thank you so much. I've tried to implement a socket pool just like you suggested, with a thread pool of 5 threads. The code now works even faster but uses fewer ports than before."

Collectives™ on Stack Overflow

Problem with: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related