2

Is there a way to scrape more than 64 results from google with python without getting my IP address instantly blocked?

3
  • How do you do this? Please include your code. Commented Jun 6, 2013 at 6:41
  • Using tor could be a solution Commented Jun 6, 2013 at 6:42
  • I found this. incolumitas.com/2013/01/06/… But the author claims it is not ported to 2.7 yet. Commented Jun 6, 2013 at 6:59

2 Answers 2

1

I use tsocks and ssh-tunnels to machines with other ip addresses to achieve this.

Sign up to request clarification or add additional context in comments.

Comments

1

64 result is the limit? Sounds weird to me!

Even with browser, I can navigate till the 100th page with no problem.

I'm very curious about how did you reach this limit.

Anyway: classical possible solutions are:

  • proxying ( IE: tor )
  • delaying requests
  • randomly switch user agent

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.