C# Pull a webpage with HTTPWebRequest and execute the javascript from the site

Question

is there a way to fetch a webpage with HTTPWebRequest and then execute all the javascriptcode that was on the page along with ajax calls and save the result to a string?

I want to parse a website where some images are generated via js code and the images aren't on the webpage before the js script has run.

Now I searched the web for a solution to this but all I found were answers like this isn't possible, use the webbrowser instead which is not an option for me.

Greetings

Can you elaborate on why using the WebBrowser class is not an option? — alexn
– alexn, Commented Nov 16, 2012 at 19:35
notions.okuda.ca/2009/06/11/… elaboration on what alexn is suggesting. I can't tell from your Q if you can't use the WebBrowser object if you can't use a web browser. — Eli Gassert
– Eli Gassert, Commented Nov 16, 2012 at 19:37
WebBrowser is slow and sometimes buggy, speed matters to me. — maddo7
– maddo7, Commented Nov 16, 2012 at 19:49

gbjbaanb · Accepted Answer · 2012-11-16 20:10:45Z

3

You could do this with the C++ http system which gives you a lot of control over the pieces of data that are receievd, but I guess that's not really an answer so...

Why not try putting webkit into your app, and running off the events from it. It has several events that can notify you when downloads are started and finished.

answered Nov 16, 2012 at 20:10

gbjbaanb

52.8k12 gold badges111 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

C.M. Over a year ago

I think the quick and easy way to do this is the WebBrowser control, the best one for him is using that .NET WebKit wrapper.

AlbertVo · Accepted Answer · 2012-11-16 20:16:49Z

1

I did a little research just because I thought it was interesting. I found something called Phantom.js which looks like it does what you need. Check it out: http://phantomjs.org/

answered Nov 16, 2012 at 20:16

AlbertVo

7724 silver badges9 bronze badges

1 Comment

C.M. Over a year ago

Interesting project, but it appears to be more freestanding and not for use in C#.

Vinod Vishwanath · Accepted Answer · 2012-11-16 19:42:31Z

0

This indeed isn't possible, as I found out while trying to meet a much-required demand. The Javascript will execute, but the Ajax calls won't be made.

The solution I settled for was to get all the data the Ajax calls would have synchronously, through the code-behind file(C# code), and to assign code-behind variables to Javascript variables, using script tags, and let Javascript do its work(building dynamic elements etc.)

answered Nov 16, 2012 at 19:42

Vinod Vishwanath

5,9212 gold badges29 silver badges41 bronze badges

2 Comments

maddo7 Over a year ago

and what did you use to execute the javascript without the ajax calls?

Vinod Vishwanath Over a year ago

Sync Javascript executes anyway in any web request. It's only with async code(ajax calls, callbacks) that it fails. To answer the question: I used the WebRequest class in ASP.NET.

Collectives™ on Stack Overflow

C# Pull a webpage with HTTPWebRequest and execute the javascript from the site

3 Answers 3

1 Comment

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related