Web scraping lazy list (lazy loading) using python request (without selenium/scarpy)

Question

I have written a simple script for myself as practice to find who had bought same tracks as me on bandcamp to ideally find accounts with similar tastes and so more same music on their accounts.

The problem is that fan list on a album/track page is lazy loading. Using python's requests and bs4 I am only getting 60 results out of potential 700.

I am trying to figure out how to send request i.e. here https://pitp.bandcamp.com/album/fragments-distancing to open more of the list. After finding what request is send when I click in finder, I used that json request to send it using requests, although without any result

res = requests.get(track_link)
    open_more = {"tralbum_type":"a","tralbum_id":3542956135,"token":"1:1609185066:1714678:0:1:0","count":100}
    for i in range(0,3):
        requests.post(track_link, json=open_more)

Will appreciate any help!

folen gateis · Accepted Answer · 2021-11-26 00:59:19Z

1

i think that just typing a ridiculous number for count will do. i did some automation on your script too if you want to get data on other albums

from urllib.parse import urlsplit
import json

import requests
from bs4 import BeautifulSoup

# build the post link
get_link="https://pitp.bandcamp.com/album/fragments-distancing"
link=urlsplit(get_link)
base_link=f'{link.scheme}://{link.netloc}'
post_link=f"{base_link}/api/tralbumcollectors/2/thumbs"

with requests.session() as s:
    res = s.get(get_link)
    soup = BeautifulSoup(res.text, 'lxml')

    # the data for tralbum_type and tralbum_id
    # are stored in a script attribute
    key="data-band-follow-info"
    data=soup.select_one(f'script[{key}]')[key]
    data=json.loads(data)
    open_more = {
        "tralbum_type":data["tralbum_type"],
        "tralbum_id":data["tralbum_id"],
        "count":1000}
        
    r=s.post(post_link, json=open_more).json()
    print(r['more_available']) # if not false put a bigger count

edited Nov 26, 2021 at 0:59

answered Nov 24, 2021 at 23:46

folen gateis

2,0003 gold badges16 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

euh Over a year ago

Thanks, it does help! For some reason the first response needs to have count = 80 and only later ones can be higher number (either last print returns False) but it can be walked around easily)

folen gateis Over a year ago

last print should be always false. it's just to be sure that there's no more people available. if it's not false put a bigger count. i thought it was clear on the context, sorry

Collectives™ on Stack Overflow

Web scraping lazy list (lazy loading) using python request (without selenium/scarpy)

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related