5
\$\begingroup\$

I wrote a short program which should allow a user to specify a starting page in Discogs Wiki Style Guide, scrape the other styles listed on the page, and then output a graph (represented here as a dictionary of sets) of the relationship between subgenres.

I'm looking for guidance/critique on: (1) How to clean up the request_page function, I think there is a more elegant way both getting href attrs and filtering to only those with "/style/". (2) The general structure of the program. Self-taught and relative beginner so it's highly appreciated if anyone could point out general irregularities.

import re
import requests 
from bs4 import BeautifulSoup 

def get_related_styles(start):

    def request_page(start):

        response = requests.get('{0}{1}'.format(base_style_url, start))
        soup = BeautifulSoup(response.content,'lxml')

        ## these lines feel inelegant. considered solutions with
        ## soup.findAll('a', attrs = {'href': pattern.match})

        urls = [anchor.get('href') for anchor in soup.findAll('a')]
        pattern = re.compile('/style/[a-zA-Z0-9\-]*[^/]') # can use lookback regex w/ escape chars?
        style_urls = {pattern.match(url).group().replace('/style/','') for url in urls if pattern.match(url)}

        return style_urls

    def connect_styles(start , style_2):

        ## Nodes should not connect to self
        ## Note that styles are directed - e.g. (A ==> B) =/=> (B ==> A)

        if start != style_2:
            if start not in all_styles.keys():
                all_styles[start] = {style_2}

            else:
                all_styles[start].add(style_2)

        if style_2 not in do_not_visit:
            do_not_visit.add(style_2)
            get_related_styles(style_2)

    style_urls = request_page(start)

    for new_style in style_urls:
        connect_styles(start,new_style)

Example Use:

start = 'Avant-garde-Jazz'
base_style_url = 'https://reference.discogslabs.com/style/'

all_styles = {}
do_not_visit = {start}

get_related_styles(start)

print(all_styles)
{'Free-Jazz': {'Free-Improvisation', 'Free-Funk'}, 'Free-Improvisation': {'Free-Jazz', 'Avant-garde-Jazz'}, 'Avant-garde-Jazz': {'Free-Jazz'}, 'Free-Funk': {'Free-Jazz'}}
\$\endgroup\$

1 Answer 1

Reset to default
1
\$\begingroup\$

There is a simpler way to filter out the "style" links - using a CSS selector with a partial match on the href attribute:

style_urls = {anchor['href'].replace('/style/', '') 
              for anchor in soup.select('a[href^="/style/"]')]

where ^= means "starts with".

Here we, of course, lose the check we had on the style name part of the href. If this check is really needed, we can also use a regular expression to match the desired style links directly:

pattern = re.compile('/style/([a-zA-Z0-9\-]*)[^/]')
style_urls = {pattern.search(anchor['href']).group(1)
              for anchor in soup('a', href=pattern)

soup() here is a short way of doing soup.find_all().

\$\endgroup\$

You must log in to answer this question.