How can you get the regular (!) source code of a page using selenium in java?

Question

Okay, so here's the thing: All of you are probably thinking the same thing: you can use

driver.getPageSource();

And this is partially true. The only issue is that the source code gets compiled in a rather strange way where all through the code

\&quot;

starts showing up. I tried removing this manually but that still doesnt fix the problem completely.

One example of what I mean:

normal source code:

\"query_title\":null}",encoded_title:"WyJoZW5rIl0",ref:"unknown",logger_source:"www_main",typeahead_sid:"",tl_log:false,impression_id:"bbdb1882",filter_ids:

Selenium output:

\\\&quot;query_title\\\&quot;:null}\&quot;,\&quot;encoded_title\&quot;:\&quot;WyJoZW5rIl0\&quot;,\&quot;ref\&quot;:\&quot;br_tf\&quot;,\&quot;logger_source\&quot;:\&quot;www_main\&quot;,\&quot;typeahead_sid\&quot;:\&quot;0.6583900225217523\&quot;,\&quot;tl_log\&quot;:false,\&quot;impression_id\&quot;:\&quot;e00060b4\&quot;,\&quot;filter_ids\&quot;

It seems to be the same type of thing as where you have to put something in front of certain symbols in quotes, to stop java from seeing it as one of those symbols, but I don't fully understand this behaviour, and have no idea how to fix it... hope you can help :)

edit: replacing doesn't work because of the way this got compiled. An example of why it won't work is actually in the example I included earlier:

original:

}",encoded_title:

compiled version:

}\&quot;,\&quot;encoded_title\&quot;:

Replacing \" with " would change it in to:

}","encoded_title":

which differs from the original...

And if I were to replace \" with nothing, I would get:

},encoded_title:

which, sadly, still differs from the original. The way this is compiled I just don't think replacing is a viable option...

Sers · Accepted Answer · 2018-08-19 19:05:47Z

1

You can use javascript to get html using outerHTML or innerHTML (How do I get the HTML source from the page?):

((JavascriptExecutor) driver).executeScript("return document.documentElement.outerHTML;")
((JavascriptExecutor) driver).executeScript("return document.documentElement.outerHTML;")
((JavascriptExecutor) driver).executeScript("return document.all[0].outerHTML")
((JavascriptExecutor) driver).executeScript("return new XMLSerializer().serializeToString(document);")

answered Aug 19, 2018 at 19:05

Sers

12.3k2 gold badges14 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

user4174054 Over a year ago

I'm sorry for misspeaking earlier, I experimented a bit more, and this actually doesn't work... It still gives me the sam shit in between...

Sers Over a year ago

Try @Amit replace solution

user4174054 Over a year ago

I edited the question, to explain why that won't work

Sers Over a year ago

Did you try driver.getPageSource().replaceAll("\\"", ""); ?

Sers Over a year ago

The part you shared is json? Can you share full html?

|

Amit Jain · Accepted Answer · 2018-08-19 22:06:09Z

0

You can use Java String Class replaceAll method to replace unwanted characters with the character you want.

OLD solution -

 driver.getPageSource().replaceAll("\\&quot;", "\"").replaceAll("\\\\", ""));

New approx solution - As page source can contain anything in HTML

public class CheckString {


    static String str = "\\\\\\&quot;query_title\\\\\\&quot;:null}\\&quot;,\\&quot;encoded_title\\&quot;:\\&quot;WyJoZW5rIl0\\&quot;,\\&quot;ref\\&quot;:\\&quot;br_tf\\&quot;,\\&quot;logger_source\\&quot;:\\&quot;www_main\\&quot;,\\&quot;typeahead_sid\\&quot;:\\&quot;0.6583900225217523\\&quot;,\\&quot;tl_log\\&quot;:false,\\&quot;impression_id\\&quot;:\\&quot;e00060b4\\&quot;,\\&quot;filter_ids\\&quot;";

    public static void main(String[] args) {

    System.out.println(str.replaceAll("\\\\&quot;,","\",")
                          .replaceAll(":\\\\&quot;", ":\"")
                          .replaceAll("\\\\&quot;","")
                          .replaceAll("\\\\\\\\", "\\\\\""));

    }

}

OutPut -

\"query_title\":null}",encoded_title:"WyJoZW5rIl0",ref:"br_tf",logger_source:"www_main",typeahead_sid:"0.6583900225217523",tl_log:false,impression_id:"e00060b4",filter_ids

Note - In earlier approach I forgot to escape & character which is used by replaceAll function to separate multiple condition in regex

edited Aug 19, 2018 at 22:06

answered Aug 19, 2018 at 19:28

Amit Jain

4,6152 gold badges21 silver badges21 bronze badges

5 Comments

user4174054 Over a year ago

Thanks for your contribution, I will try this option now!

user4174054 Over a year ago

I edited the question, to explain why your answer won't work

Amit Jain Over a year ago

Solution is not recommended for full page source but this works for the string you have given. Thanks.

user4174054 Over a year ago

Sadly, that will not suffice. I need loads of different parts of the source code, so I require a fix to rectify the entire source code.

Amit Jain Over a year ago

I dont think so that there will be fix in source code because in client side html, the html reserved tags if used in page are replaced in the way it is appearing in getPageSource method output ... e.g < ,> are replaced with &lt,&gt Please share what ever solution you find for this. Thanks

Collectives™ on Stack Overflow

How can you get the regular (!) source code of a page using selenium in java?

2 Answers 2

10 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related