How can I add a string to a dictionary in Python?

Question

I'm trying to take some http proxies and append them to a list and then test them individually by opening them with urllib but I get the following type error. I have tried wrapping 'proxy' with str() in the test function but that returns another error.

proxies = []

with open('working_proxies.txt', 'rb') as working_proxies:
    for proxy in working_proxies:
        proxy.rstrip()
        proxies.append(proxy)

def test(proxy):
    try:
        urllib.urlopen(
            "http://google.com",
            proxies={'http': proxy}
        )
    except IOError:
        print "Connection error! (Check proxy)"
    else:
        working_proxy = True

working_proxy = False
while working_proxy == False:
    myProxy = proxies.pop()
    test(myProxy)

My error:

Connection error! (Check proxy)
Traceback (most recent call last):
  File "proxy_hand.py", line 26, in <module>
    test(proxy)
  File "proxy_hand.py", line 16, in test
    proxies={'http': proxy}
  File "/usr/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
  File "/usr/lib/python2.7/urllib.py", line 193, in open
    urltype, proxyhost = splittype(proxy)
  File "/usr/lib/python2.7/urllib.py", line 1074, in splittype
    match = _typeprog.match(url)
TypeError: expected string or buffer

Martijn Pieters · Accepted Answer · 2016-01-21 12:56:00Z

1

You opened the file with proxies as binary here:

with open('working_proxies.txt', 'rb') as working_proxies:

The b in the 'rb' mode string means you'll be reading binary, e.g. bytes objects.

Either open the file in text mode (and perhaps specify a codec other than your system default) or decode your bytes objects to str using an explicit bytes.decode() call:

proxies.append(proxy.decode('ascii'))

I'd expect ASCII to be sufficient to decode hostnames suitable to be used as proxies.

Note that your working_proxy flag won't work; it is not marked as global in test. Perhaps you want to catch the IOError exception outside of test instead, or move the loop into that function. You'll also need to figure out what you'll do when you run out of proxies (so when none of them work).

answered Jan 21, 2016 at 12:56

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I add a string to a dictionary in Python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related