How to scrape page requiring cookies and javascript in PHP

Question

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

The current response shows:

<body><noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.</noscript></body>

I put this in the code and it made no difference:

$strCookie = 'PHPSESSID=' . $_COOKIE['PHPSESSID'] . '; path=/';
curl_setopt( $ch, CURLOPT_COOKIE, $strCookie );

Is there a specific reason why you would want to do it at the server end rather than the client end? — francisco.preller
– francisco.preller, Commented Mar 9, 2012 at 8:11
The script is parsing to create an rss feed, to then become a twitter feed and the underlying site is happy with the arrangement too. — Adrian33
– Adrian33, Commented Mar 9, 2012 at 10:33

Community · Accepted Answer · 2017-05-23 12:23:37Z

2

HTML inside the <noscript> </noscript> will only get displayed if Javascript is disabled. So this is the test for whether Javascript is enabled.

There is jQuery-cookie-plugin for reading, writing and deleting cookies. But it will not work if Javascript is disabled.

For checking if cookie is enabled with PHP code, you need to set the cookie and read the cookie with another request. This post have some guides on it.

edited May 23, 2017 at 12:23

CommunityBot

11 silver badge

answered Mar 9, 2012 at 10:32

tusar

3,4306 gold badges41 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Your Common Sense · Accepted Answer · 2012-03-09 07:55:47Z

0

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

No.

answered Mar 9, 2012 at 7:55

Your Common Sense

158k42 gold badges226 silver badges374 bronze badges

Collectives™ on Stack Overflow

How to scrape page requiring cookies and javascript in PHP

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related