Print web page source code in python

Question

I want to print a web page source code but python print command just prints empty space and I think it's because of its large size. Is there any way to print page source code in shell or at list in a file? I've tried printing in a file but this error occurred:

UnicodeEncodeError: 'charmap' codec can't encode character '\u06cc' in position 11826: character maps to <undefined>

How can I fix it?

import urllib.request
response = urllib.request.urlopen('http://www.farsnews.com')
html = response.read()

print(html)#prints empty space! 

hf=open('test.txt','w')
a=str(html,'utf-8')
hf.write(a)
hf.close()

Python easily prints a[0:1000] but for a[0:len(a)] as I said empty space!

Works for me (the webpage source code is displayed); I'm using Python 3.3.1 running on Ubuntu 13.04. — Ramchandra Apte
– Ramchandra Apte, Commented Nov 30, 2013 at 14:55
@Ramchandra Apte I'm using python 3.3.2 running windows 7 and I faced unicode error. — hamidfzm
– hamidfzm, Commented Nov 30, 2013 at 19:34
I tested it and for me it worked fine. I'm running on Debian jessie with python 3.3.2+. I would guess your terminal can't display some characters in the source code or the charmaps of your OS don't support them. — Kritzefitz
– Kritzefitz, Commented Dec 1, 2013 at 9:47

ustinov · Accepted Answer · 2013-12-03 15:05:01Z

I've just tried the same on Win7 using python 3.2.5 and here's what I got:

Python 3.2.5 (default, May 15 2013, 23:07:10) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import request
>>> r = request.urlopen("http://www.farsnews.com")
>>> bytecode = r.read()
>>> htmlstr = bytecode.decode()
>>> print(bytecode)

Printing bytecode works well as it prints the encoded representations for unicode chars but printing the htmlstr raises the UnicodeDecodeError on windows because some chars cannot be printed using current locale's default encoding (windows' cmd.exe is not unicode)

In my case the encoding that has been used was 'cp866' as I saw it in traceback.

By default py3k uses the 'utf-8' encoding to store string data and if you want to override it you should explicitly specify the encoding to use for decoding

So here's the possibble workaround:

>>> safe_str = bytecode.decode(encoding='cp866', errors='ignore')
>>> print(safe_str)

Actually, it's equivalent to

>>> safe_str = str(bytecode, encoding='cp866', errors='ignore')
>>> print(safe_str)

The second parameter errors tells whether the error should be rose when the encoding you're trying to use cannot decode the particular character

gariepy · Accepted Answer · 2016-06-02 15:36:24Z

0

I simply did

import requests
page = requests.get(url)
print (page.text.encode('utf8'))

If you're scraping websites with python then is this an awesome starting point. I also recommend that you look into BeautifulSoup (another method of parsing html).

edited Jun 2, 2016 at 15:36

gariepy

3,6846 gold badges24 silver badges34 bronze badges

answered Jun 2, 2016 at 15:03

Quinten

4375 silver badges13 bronze badges

Collectives™ on Stack Overflow

Print web page source code in python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related