get sourcecode after javascript execution with curl

Question

Is it possible to get the html source code of a webpage with curl and the run a javascript interpreter over it, so i get the generated content?

The Page i need to get uses some encoded and genereated content in there so i want to first run the javascript to get the escaped and generated content... or do i need to regex the javascript and "compile" the javascript on my own? Like

curl <myurl> | perl -ne 'm/unescape\((.*)\)/; print "$1"' | <now to something with that>

i known there is no javascript engine in curl but can i just call another script / programm to do the job?

Try this: stackoverflow.com/a/28506533/3063226

Heitor
– Heitor

2017-02-10 19:51:17 +00:00
Commented Feb 10, 2017 at 19:51 — Heitor
– Heitor, Commented Feb 10, 2017 at 19:51

dcow · Accepted Answer · 2022-06-13 17:31:13Z

10

You can do it, but it's more involved than I think you realize. Neither curl nor wget have Javascript engines, so you'll need something that has one.

I would start by looking at PhantomJS and/or Chromium Headless.

edited Jun 13, 2022 at 17:31

answered May 9, 2012 at 11:09

dcow

8,0953 gold badges47 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

reox Over a year ago

nice, i didnt knew that there where such things :) I will also update my screenshot script which currently goes over some edges then using phantomjs!

cazort Over a year ago

PhantomJS has been suspended as of March 2018. I'd love to know of a more up-to-date option. Depending on the use, there might be security issues with using an unmaintained package.

Collectives™ on Stack Overflow

get sourcecode after javascript execution with curl

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related