Web scrape links with Python, then turn them into a string [duplicate]

Question

With Python I'm having issues turning web scrapped links into strings so I can save them as either a txt or csv file. I would really like them as a txt file. This is what I have at the moment.

import requests
from bs4 import BeautifulSoup

url = "https://www.google.com/"
reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'html.parser')

urls = []
for link in soup.find_all('a'):
    print(link.get('href'))
type(link)

print(link, file=open('example.txt','w'))

I've tried all sort of things with no luck. I'm pretty much at a lose.

In Jupyter Notebook it outputs, but I cannot seem to get it to save as a text file (or csv) showing each output line. — D B
– D B, Commented Nov 21, 2022 at 15:42
I can only get the last "link" to print to a text file. I'm stuck trying to get all the "links" to be listed in the text document. — D B
– D B, Commented Nov 21, 2022 at 15:55
Show us your code regarding writing to the file, we can't see what you're doing wrong without the actual code — 0stone0
– 0stone0, Commented Nov 21, 2022 at 15:57
Everything is at the top of the page. The last line printing to the example.txt only shows the last output line found when scrapping the URL google.com. So the output is a text doc with <a href="/intl/en/policies/terms/">Terms</a> — D B
– D B, Commented Nov 21, 2022 at 16:10

0stone0 · Accepted Answer · 2022-11-21 16:12:30Z

0

print(link, file=open('example.txt','w'))

Will write the link variable, but that's only the last one.

To write them all, use:

import requests
from bs4 import BeautifulSoup

url = "https://www.google.com/"
reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'html.parser')

with open("example.txt", "w") as file:
    for link in soup.find_all('a'):
        file.write(link.get('href') + '\n')

Which uses a context manager to open the file, then write each href with a newline.

answered Nov 21, 2022 at 16:12

0stone0

45.5k6 gold badges53 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Web scrape links with Python, then turn them into a string [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related