4
\$\begingroup\$

In this SO question the OP is unable to scrape a table from a dynamically loaded website. In monitoring the web traffic, via Chrome dev tools, I found that there is an API request made that returns a JSON string with the required info.

The following is the answer I wrote to extract the columns of interest from the API response.

Guide to columns of interest:

  1. Course Title = title
  2. Trainer = name (within trainers)
  3. Rating = rating
  4. Vendor = name (within vendors)
  5. IT Path = path_label (within paths)
  6. Skill Level = display (within difficulty)
  7. Course URL = concatenation of base with seoslug

The Vendors field has missing items hence my use of an if statement in the assignment to vendors. I am not sure what the usual placeholder value is for missing string values in Python.

I use repeated list comprehensions in loops over the JSON object data; where data = response.json()

I couldn't think of a way to remove these repeated loops and still have legible code.

I generate a dataframe by joining the lists in a dictionary and then converting with pandas.

I welcome any and all feedback please.

JSON response:

Example JSON dictionary within response. The response has a collection of such dictionaries.

Python 3

import requests
import pandas as pd


def main():
    base = 'https://www.cbtnuggets.com/it-training/'
    response  = requests.get('https://api.cbtnuggets.com/site-gateway/v1/all/courses/for/search?archive=false')
    
    data = response.json()
    
    titles = [item['title'] for item in data]
    trainers = [item['trainers'][0]['name'] for item in data]
    ratings = [item['rating'] for item in data]
    vendors = [item['vendors'][0]['display'] if len(item['vendors']) != 0 else 'N/A' for item in data]
    paths = [item['paths'][0]['path_label'] for item in data]
    skillLevel = [item['difficulty']['display'] for item in data]
    links = [base + item['seoslug'] for item in data]

    df=  pd.DataFrame(
      {'Course Title': titles,
       'Trainer': trainers,
       'Rating': ratings,
       'Vendor': vendors,
       'IT Path': paths,
       'Skill Level': skillLevel,
       'Course URL': links
      })

    #print(df)
    df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8',index = False )

if __name__ == "__main__":
   
    main()
\$\endgroup\$
1
  • 2
    \$\begingroup\$ The only thing I'll say is that, if each publication on Code Review had a presentation as clear, complete and pleasant as yours, the overall quality of this site would be improved. Pretty nice code BTW. \$\endgroup\$ Commented Nov 19, 2018 at 22:07

3 Answers 3

Reset to default
2
\$\begingroup\$

I couldn't think of a way to remove these repeated loops and still have legible code.

There is a way:

titles, trainers, ratings, vendors, paths, skillLevel, links = zip(*((
    item['title'],
    item['trainers'][0]['name'],
    item['rating'],
    item['vendors'][0]['display'],
    item['paths'][0]['path_label'],
    item['difficulty']['display'],
    base + item['seoslug']
) for item in data))

I haven't tested this, so you should.

\$\endgroup\$
3
  • \$\begingroup\$ Sorry for delay. If I replace with item['vendors'][0]['display'] if len(item['vendors']) != 0 else 'N/A' , this works beautifully. This is basically where I was trying to get to. Thank you. \$\endgroup\$ Commented Nov 21, 2018 at 9:27
  • \$\begingroup\$ What is the purpose of the * in the above please? Is it unpacking? \$\endgroup\$ Commented Nov 21, 2018 at 9:29
  • \$\begingroup\$ @QHarr Yes, it's the unpacking operator. \$\endgroup\$ Commented Nov 21, 2018 at 13:58
3
\$\begingroup\$

Instead of splitting the data into a separate variables by column, you could convert each JSON object into a flat dictionary using a function similar to this

def course_dict(item):
    return {'Course Title': item['title'],
            'Vendor': item['vendors'][0]['display'] if item['vendors'] else None,
            # and so on
            }

and construct the dataframe using

data = response.json()
df = pd.DataFrame([course_dict(item) for item in data])

Keeping related data together makes the code easier to follow. Also, since your final output is a csv file, you could skip the dataframe and use csv.DictWriter instead.

I am not sure what the usual placeholder value is for missing string values in Python.

None is the usual placeholder for missing values of any type.

\$\endgroup\$
2
  • \$\begingroup\$ Thanks. How would I expand this for the other items? Create a series of functions? \$\endgroup\$ Commented Nov 21, 2018 at 9:07
  • \$\begingroup\$ @QHarr Just expand the function, adding the other items next to the two \$\endgroup\$ Commented Nov 21, 2018 at 9:52
2
\$\begingroup\$

The only major thing I would change is that: 

if len(item['vendors']) != 0

Is the same as:

if item['vendors']

Because an empty list will return back as False. If you want to try it out: 

a = []
bool(a)      # False
b = [1,2,3]
bool(b)      # True

I would also be careful with what you have because those dictionaries that you are converting might have more than one value, in which case you would miss them. This is the line that I am referring to:

    paths = [item['paths'][0]['path_label'] for item in data]
\$\endgroup\$
1
  • \$\begingroup\$ Thank you for the feedback. Great point about vendors. And yes, I made an assumption, that the first value would suffice, for the other line you pointed out about dictionaries. + \$\endgroup\$ Commented Nov 19, 2018 at 19:19

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.