Python Web Crawlers and "getting" html source code

An Example with python3 and the requests library as mentioned by @leoluk:

pip install requests

Script req.py:

import requests

url='http://localhost'

# in case you need a session
cd = { 'sessionid': '123..'}

r = requests.get(url, cookies=cd)
# or without a session: r = requests.get(url)
r.content

Now,execute it and you will get the html source of localhost!

python3 req.py

If you are using Python > 3.x you don't need to install any libraries, this is directly built in the python framework. The old urllib2 package has been renamed to urllib:

from urllib import request

response = request.urlopen("https://www.google.com")
# set the correct charset below
page_source = response.read().decode('utf-8')
print(page_source)

The first thing you need to do is read the HTTP spec which will explain what you can expect to receive over the wire. The data returned inside the content will be the "rendered" web page, not the source. The source could be a JSP, a servlet, a CGI script, in short, just about anything, and you have no access to that. You only get the HTML that the server sent you. In the case of a static HTML page, then yes, you will be seeing the "source". But for anything else you see the generated HTML, not the source.

When you say modify the page and return the modified page what do you mean?

All the above will fail on an https request behind Cloudflare. You can try this to fetch both http and https html:

import requests
url = 'https://your.link.here'   
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',}
response = requests.get(url, headers=headers)

if response.status_code == 200:
    print(response.text)
else:
    print(f'Request failed with status code: {response.status_code}')

Here you have a code to this task:

import requests
from requests.exceptions import RequestException
from datetime import datetime
import urllib.parse

def fetch_url(url, retries=3):

headers = {
    "User-Agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
}

for attempt in range(retries):
    try:
        response = requests.get(url, headers=headers, timeout=10, allow_redirects=True)
        if response.status_code == 200:
            response.encoding = response.apparent_encoding
            return response.text
        else:
            print(f"Error: {response.status_code}")
    except RequestException as e:
        print(f"Attempt {attempt + 1} failed: {e}")

return None

def get_filename_from_url(url):

parsed_url = urllib.parse.urlparse(url)
domain = parsed_url.netloc.replace("www.", "")
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
filename = f"{domain}_{timestamp}.html"
return filename

url = input("Introduce la URL: ")

source_code = fetch_url(url)

if source_code:
    filename = get_filename_from_url(url)
    with open(filename, "w", encoding="utf-8") as file:
        file.write(source_code)
    print(f"El código fuente se ha guardado en {filename}")
else:
    print("Failed to retrieve the webpage after multiple attempts.")