What makes python stall when running http.server

Question

I was running some load tests with the tool wrk and noticed some strange numbers after running multiple tests. I created the following test to show the problem I am seeing.

# file to serve
echo '{"message":"Yeah,it works"}' > test.txt

# run webserver
python3 -m http.server 8000 --bind 127.0.0.1

In another session I run the wrk tool

# run a test with 1 thread and 1 connection for 15 seconds
# brew install wrk ( is how I installed the tool ) 
#
wrk -t1 -c1 -d15s "http://127.0.0.1:8000/test.txt"

When I run wrk the first time I see about 16360 requests in the 15seconds. However, if I run the wrk command again a few times I will see the performance drop to ~2000 requests in 15seconds. I have tried this on two computers running difference versions of Python (3.8, 3.9, 3.10), all with the same results. If I keep running the wrk test I will see the numbers go back up the results were never consistent.

What is happening and why would I see the performance drop like this?

I attempted to reproduce your use-case and had a similar experience. Although the number of requests bounced around, it wasn't just fast the first time and then slow all subsequent runs. However, when I switched to an http server that actually 'did something', my results got more consistent. Not sure why your example has so much flux. — synthesizerpatel
– synthesizerpatel, Commented Nov 11, 2021 at 20:37
So I think this is an issue with my laptop(s) as I spun up a VM in azure and ran the test and got consistent numbers. So I guess maybe it has to do with what is running more on my laptop than something with python. for x in {1..10}; do ( wrk -t1 -c1 -d15s "127.0.0.1:8000/test.txt" | grep "requests in"); done 28679 requests in 15.10s, 5.85MB read 28432 requests in 15.00s, 5.80MB read 28570 requests in 15.10s, 5.83MB read 28709 requests in 15.10s, 5.86MB read 28590 requests in 15.10s, 5.84MB read 28562 requests in 15.10s, 5.83MB read ... — ScottW
– ScottW, Commented Nov 11, 2021 at 21:28
Did you notice your fan turn on or get louder? It may throttle due to heat. — tdelaney
– tdelaney, Commented Nov 13, 2021 at 5:50

ScottW · Accepted Answer · 2021-11-11 21:38:04Z

So I think this has to do with what is running on my laptop more than what is happening with python. I tried to renice my python process to -20 and that didn't help either. Finally I gave up and spun up a vm in Azure. I ran the test again on a ubuntu server and got the following.

for x in {1..10}; do ( wrk -t1 -c1 -d15s "http://127.0.0.1:8000/test.txt" | grep "requests in"); done
  28679 requests in 15.10s, 5.85MB read
  28432 requests in 15.00s, 5.80MB read
  28570 requests in 15.10s, 5.83MB read
  28709 requests in 15.10s, 5.86MB read
  28590 requests in 15.10s, 5.84MB read
  28562 requests in 15.10s, 5.83MB read
  28578 requests in 15.10s, 5.83MB read
  28614 requests in 15.10s, 5.84MB read
  28702 requests in 15.10s, 5.86MB read
  28685 requests in 15.10s, 5.85MB read

which I would say it pretty consistent values. So I guess the lesson here is don't do performance testing on your laptop.

Collectives™ on Stack Overflow

What makes python stall when running http.server

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related