Selenium Webdriver using Proxy IP

Question

I was using selenium webdriver to scrap data from a website.due to heavy traffic from My IP, now i am not able to access website (may be my ip is blocked for the website).

Is there any way to setup the Proxy IP, so that it would be treated as a new IP every-time i run the webdriver..?

true it'll be treated as the new IP depending on the proxy IP pool you gonna use. — Igor Savinkin
– Igor Savinkin, Commented Oct 15, 2015 at 7:42
@IgorSavinkin but my concern is how to setup the Proxy ip with webdriver. — Pradeep Maurya
– Pradeep Maurya, Commented Oct 15, 2015 at 7:44
I know it. So do you use Selenium with python? Could you expose what you've already done? — Igor Savinkin
– Igor Savinkin, Commented Oct 15, 2015 at 7:47
@IgorSavinkin i am using it with java. as well as just creating a FF driver instance using following code driver=new FirefoxDriver(); driver.get("whatsmyip.net/"); — Pradeep Maurya
– Pradeep Maurya, Commented Oct 15, 2015 at 7:51

Helping Hands · Accepted Answer · 2015-10-15 08:11:40Z

1

You can set proxy IP with selenium webdriver as per following way :

FirefoxProfile profile = new FirefoxProfile();
profile.addAdditionalPreference("network.proxy.http", "localhost");
profile.addAdditionalPreference("network.proxy.http_port", "8080");
WebDriver driver = new FirefoxDriver(profile);

You should change your desired IP address and port in above code.

For more detail , Please have a look at : Selenium Webdriver with proxy

answered Oct 15, 2015 at 8:11

Helping Hands

5,41410 gold badges70 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Pradeep Maurya Over a year ago

I have used above code & its working. Do you have any website where we can get the proxy IPs which will work 100% ,as i visited several sites (i.e proxylist.hidemyass.com/search-1291967#listable)they provide the ip & ports but by using them we are getting 50% success rate.

Helping Hands Over a year ago

I will look for same and will update you once get site.

Igor Savinkin · Accepted Answer · 2015-10-15 08:30:25Z

Since you scrape a lot you'll probably need a reliable proxy provider. So most of them provide their own api for auth and using their proxy pool. I've got this piece of code (java) from Luminati.io

package example;

import org.apache.http.HttpHost;
import org.apache.http.client.fluent.*;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpHost proxy = new HttpHost("zproxy.luminati.io", 22225);
        String res = Executor.newInstance()
            .auth(proxy, "lum-customer-CUSTOMER-zone-YOURZONE", "YOURPASS")
            .execute(Request.Get("http://www.telize.com/geoip").viaProxy(proxy))
            .returnContent().asString();
        System.out.println(res);
    }
}

There are more complex example codes there.

Non-professional scrape

If you just want to plug in any deliberate proxy IP to test, you might use this:

FirefoxProfile profile = new FirefoxProfile();
host='149.215.113.110' 
port='9150'
profile.SetPreference("network.proxy.type", 1);
profile.SetPreference("network.proxy.http", host);
profile.SetPreference("network.proxy.http_port", int(port));
driver = new FirefoxDriver(profile);

Collectives™ on Stack Overflow

Selenium Webdriver using Proxy IP

2 Answers 2

2 Comments

Non-professional scrape

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Non-professional scrape

Comments

Your Answer

Sign up or log in

Post as a guest

Related