Timeout problem with ASP.NET web scraping

Question

I'm trying to implement a web scraping crawler as a part of my ASP.NET MVC project. It works with large data gathered from different URLS using Html Agility Pack. The problem is when I want to actually run the function I get "The connection was reset" from remote server after a minute. I'm getting better result when I run it locally. I have access to remote IIS. Any suggestion to solve this problem and/or any alternatives?

Are you trying to retrieve all of the URLs in one request to your page? — Oded
– Oded, Commented Jan 9, 2011 at 9:17
Yes I guess. there is a loop which generates the URLs and try to capture them. but with this time limit it can just get few of the URLs. — Ehsan Tavakoli
– Ehsan Tavakoli, Commented Jan 9, 2011 at 9:20

Community · Accepted Answer · 2017-05-23 11:55:28Z

1

If you have a long running process in ASP.NET, it is best to let it run on a different thread.

See this and this - related questions and this MSDN article.

edited May 23, 2017 at 11:55

CommunityBot

11 silver badge

answered Jan 9, 2011 at 9:22

Oded

501k102 gold badges900 silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Darin Dimitrov · Accepted Answer · 2011-01-09 09:22:58Z

0

Connection and network problems could result in such problems. To avoid blocking scraping of other urls you could parallelize the work into separate threads.

answered Jan 9, 2011 at 9:22

Darin Dimitrov

1.0m276 gold badges3.3k silver badges3k bronze badges

Collectives™ on Stack Overflow

Timeout problem with ASP.NET web scraping

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related