Check if element exists in fetched URL [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened.

Closed 10 years ago.

Improve this question

I have a page with, say, 30 URLS, I need to click on each and check if an element exists. Currently, this means:

$('area').each(function(){
    $(this).attr('target','_blank');
    var _href = $(this).attr("href"); 
    var appID = (window.location.href).split('?')[1];
    $(this).attr("href", _href + '?' + appID);
    $(this).trigger('click');
});

Which opens 30 new tabs, and I manually go through them.

(all URLS are in the same domain)

It would be really nice have a crawler with the following logic:

$('area').each(function(){

 1) get the HREF
 2) follow it
 3) on that new page:
    if($('.element')){
     push the $('area') into array1 
    } else {
     push the $('area') into array2
        }
    });


   4) Display array1 in green
      Display array2 in red

Basically, Id like to generate a report that says:

X crawled pages had element Y

Z crawled pages did not have element Y

Im obviously stuck at making Javascript/jQuery work in a newly opened tab.

I've found this , this and this , but Im not entirely sure if this is viable.

Can this be done with Javascript/jQuery?

I'm only asking for the right direction, I'll do the steps myself.

Many thanks

Sergey · Accepted Answer · 2015-06-17 20:41:09Z

1

I can suggest you use iframe for loading pages.

For example:

$.each($your-links, function(index, link) {
    var href = $(link).attr("href");
    // your link preprocess logic ...

    var $iframe = $("<iframe />").appendTo($("body"));
    $iframe.attr("src", href).on("load", function() {
        var $bodyContent = $iframe.contents().find("body");
        // check iframe content and remove iframe
        $iframe.remove();
    }
}

But, I should say, if your crawler and checked pages have a different domains there will be the CORS problems.

I've created a simple project that shows how to implement this approach. You can download it here and run on some local web-server (apache, iis etc.)

answered Jun 17, 2015 at 20:41

Sergey

5,4864 gold badges28 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Check if element exists in fetched URL [closed]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related