Is it possible to write a web crawler in javascript?

Question

I want the following functions and I know about the client side domain restriction in javascript however I don't know if this restrictions applies to what I want the crawler to do.

Javascript to load the text content of a given website's url address to a div or assign it to a var.
the tags should then be parsed from the text.

3 the body of text should be searched for a specific word. If the word is found it should take the neighbouring sentence including the word and display it on the alert message.

I am writing a firefox application so everthing must be done on client side.

Its firefox extensions which retrieves link address when the user hovers over links on any given page — user375065
– user375065, Commented Mar 2, 2011 at 14:52

SLaks · Accepted Answer · 2011-03-02 14:37:32Z

3

As you just said, you cannot use Javascript to retrieve arbitrary content from another domain.

However, you could write a server-side proxy in your own domain which forwards requests to arbitrary URLs and passes along the responses.

answered Mar 2, 2011 at 14:37

SLaks

891k182 gold badges1.9k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dynamic · Accepted Answer · 2011-03-02 14:59:12Z

1

Best and easiest thing you can do is:

make a dynamic page on your server that accepts a param: example page.php?url=
your javascript will AJAX that page with the url it needs so it can retrive the HTML (thru your php script) and then parse it in js/client

edited Mar 2, 2011 at 14:59

answered Mar 2, 2011 at 14:37

dynamic

48.2k56 gold badges163 silver badges236 bronze badges

4 Comments

user375065 Over a year ago

I ve got php server installed, I get the url address from my firefox extensions. Is it possible to make a call from the extension to a php script and the php script responds by sending a body of text to the js

dynamic Over a year ago

This is completly another question (Is it possible to make a call from the extension to a php script). I gave you the base on how to preceed, if you need specific help on firefox exstension or PHP code open another question :) (select this as answer before XD)

user375065 Over a year ago

I need urgent help man. I used yboss to search for keywords but it doesn't return good results. I am doing my final year project deadline is in 27th march need to think of another way to implement the behaviour I want so if you know there is a way a php script can be called from your firefox extensions then please do share your knowledge thanks

dynamic Over a year ago

sorry man never made a firefox ext neither used yboss. Open another question

Collectives™ on Stack Overflow

Is it possible to write a web crawler in javascript?

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related