2

For a project work on text analytics, I am trying to scrape some reviews. I am using python and beautiful soup to do the job. I am not getting any errors but not getting any data also. I am sure I am making mistake in specifying the div tags. Can someone help? The following is the code which I used:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.zomato.com/brewbot")
soup = BeautifulSoup(r.content)
links = soup.find.all("div")
k_data = soup.find_all({"class":"rev-text"})

for item in k_data:
    print item.text

I have changed "class":"rev-text" to "tabindex='0'", "class"-"rev.text", included the "itemprop"="description", and other combinations...nothing seem to work. Can someone help?

Improve this question

1 Answer 1

Reset to default
2

Reviews are dynamically loaded from a response to a POST request to the social_load_more.php endpoint. Simulate that in your code, get the HTML with reviews from the JSON response and parse it with BeautifulSoup. Complete working code:

import requests
from bs4 import BeautifulSoup


with requests.Session() as session:
    session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"}
    r = session.get("https://www.zomato.com/brewbot")
    soup = BeautifulSoup(r.content, "html.parser")
    itemid = soup.body["itemid"]

    # get reviews
    r = session.post("https://www.zomato.com/php/social_load_more.php", data={
        "entity_id": itemid,
        "profile_action": "reviews-top",
        "page": "0",
        "limit": "5"
    })
    reviews = r.json()["html"]

    soup = BeautifulSoup(reviews, "html.parser")
    k_data = soup.select("div.rev-text")

    for item in k_data:
        print(item.get_text())
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.