python socket GET

Question

From the other posts on stack overflow this should be working

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)                 

s.connect(("www.cnn.com" , 80))
s.sendall("GET / HTTP/1.1\r\n")
print s.recv(4096)
s.close

but for some reason it just hangs (at recv) and never prints. I know that a request to www.cnn.com will chunk it's data but I should at least read something from recv, right?

p.s. I know this isn't the best way to do it and that there are library like httplib and urllib2 out there, but I can't use those for this project (it's for school). I have to use the socket library

Takis · Accepted Answer · 2015-12-10 01:29:08Z

26

You forgot to send a blank line after your request line:

s.sendall("GET / HTTP/1.1\r\n\r\n")

Furthermore, HTTP 1.1 specifies you should add the Host header field as documented in the Host section in the HTTP 1.1 RFC.

s.sendall("GET / HTTP/1.1\r\nHost: www.cnn.com\r\n\r\n")

edited Dec 10, 2015 at 1:29

answered Dec 10, 2015 at 1:20

Takis

7265 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jay Over a year ago

Pardon my noobness, why is a Host header required if the socket connected to it?

Takis Over a year ago

To allow name based virtual hosting: When you connect to the hostname, the hostname is resolved (using DNS) to an IP-address, and the actual TCP-connection is to the IP-address. So, if you'd host several websites on the same server, using only one IP-address, the client would not know which website you were trying to connect to. By passing the "Host"-header the webserver knows which website you are trying to access. Virtual Hosting

mhawke · Accepted Answer · 2015-12-10 01:40:19Z

8

Your code is almost right, but you need to send 2 \r\n sequences to satisfy the HTTP protocol.

A valid GET request will look like this (note 2 lines):

GET / HTTP/1.1

So your code should be:

s.sendall('GET / HTTP/1.1\r\n\r\n')

Further to that, there are additional headers required for valid HTTP 1.1 requests, such as Host:. You need to add them to your request, something like this:

s.sendall('''GET / HTTP/1.1
Host: cnn.com

''')

edited Dec 10, 2015 at 1:40

answered Dec 10, 2015 at 1:21

mhawke

87.5k10 gold badges122 silver badges142 bronze badges

2 Comments

james smith Over a year ago

This does answer my questions and was first so I guess I'll make this correct. For others, see my own answer as well

mhawke Over a year ago

@jamessmith: you should choose the best answer, not the first answer. Anyway, I think that Takis answered first :)

Community · Accepted Answer · 2017-05-23 10:28:13Z

6

Sorry to waste everyone's time. I just found this solution here on Stack Overflow (just took some rewording in my Google search to find)

import socket
request = b"GET / HTTP/1.1\nHost: www.cnn.com\n\n"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("cnn.com", 80))
s.send(request)
result = s.recv(10000)
while (len(result) > 0):
    print(result)
    result = s.recv(10000)

And all of the answers were right as well about the ending \r\n\r\n however those returned 301 statuses. This solution seems to follow the redirect somehow? Anyways, this solutions worked for me

edited May 23, 2017 at 10:28

CommunityBot

11 silver badge

answered Dec 10, 2015 at 1:24

james smith

2271 gold badge2 silver badges7 bronze badges

1 Comment

mhawke Over a year ago

That code gives a 302 response. It does not follow the redirect. Do you need to handle redirects for your school project?

Jan Bodnar · Accepted Answer · 2019-09-06 09:00:42Z

3

I am cleaning up the examples for Python 3. We need bytes/string conversion and we can also use automatic closing of the connection using with:

#!/usr/bin/env python3

import socket

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:

    s.connect(("example.com" , 80))
    s.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\nAccept: text/html\r\n\r\n")
    print(str(s.recv(4096), 'utf-8'))

answered Sep 6, 2019 at 9:00

Jan Bodnar

11.8k6 gold badges73 silver badges84 bronze badges

Comments

sibi · Accepted Answer · 2019-12-04 13:27:24Z

2

@james: you did a SlowLoris attack there without aware of it. I can't explain better than here, https://www.youtube.com/watch?v=XiFkyR35v2Y I assumed that you found the solution from all the above answers but I just answered to bring this to your knowledge. :)

answered Dec 4, 2019 at 13:27

sibi

1742 silver badges9 bronze badges

Comments

Remi Guan · Accepted Answer · 2015-12-10 01:21:16Z

1

Try replace this line:

s.sendall("GET / HTTP/1.1\r\n")

with:

s.sendall("GET / HTTP/1.1\r\n\r\n")
                             ^^^^

Also, I think you need replace s.close with s.close() since it's a function.

answered Dec 10, 2015 at 1:21

Remi Guan

22.5k17 gold badges68 silver badges90 bronze badges

Collectives™ on Stack Overflow

python socket GET

6 Answers 6

2 Comments

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

2 Comments

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related