Rotate server IPs using selenium (Python)

Question

I'm working on a project where i need to scrap some data using selenium.. For scrapping, i want to rotate my server IPs (my server has multiple IPs attached to single machine). For each request, i want to use a different IP so i don't get blocked. Not sure how this could be done. I'm using Firefox on ubuntu 16.04.

So far, by googling for many hours, i have found this but its not working.

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", 'my-ip')
profile.set_preference("network.proxy.http_port", 80)
profile.set_preference("network.proxy.https", 'my-ip')
profile.set_preference("network.proxy.https_port", 80)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://api.ipify.org/')
print driver.page_source

I'm testing with https://api.ipify.org/ just to find which IP is used for the generated request but it always shows me the primary IP of my server even though i have set different IP using the above code.

Any help/guidance or workaround would be highly appreciated.

Francois Mockers · Accepted Answer · 2017-05-12 09:25:17Z

1

What you would need is to bind Firefox to a different network interface, which is not possible.

A workaround would be to set up a proxy like squid on your server, configure it to route outgoing traffic through each interface based on a parameter you can control from selenium configuration for Firefox, for example the proxy port (it should be possible with squid acl).

answered May 12, 2017 at 9:25

Francois Mockers

7334 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Waqas Malik Over a year ago

Thanks, I'll try using squid

Collectives™ on Stack Overflow

Rotate server IPs using selenium (Python)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related