0

guys I'm trying to scrape amazon reviews using selenium but don't know how to handle next page URL and I want to scrape using dynamic conditions, not by self counting the pages and apply static method

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

#Using chrome browser

driver=webdriver.Chrome(executable_path='./chromedriver.exe')
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')

title_of_product = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "productTitle"))
    )
print(title_of_product.text)

Reviews=WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//span[@class='a-size-base review-text review-text-content']/span")))

next_button =WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME,"a-last"))).click()


time.sleep(10)
driver.close()
Improve this question
1
  • look the code is working fine , (handle next page URL ) means I want to navigate to this link by clicking next which is working fine now I want to know how can I handle them using for loop or something for multiple pages and get reviews amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/… Commented Aug 1, 2021 at 5:09

2 Answers 2

Reset to default
2

If I understand your question in the right way, the following should do it:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')

product_title = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "productTitle"))).text

print(product_title)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[@data-hook='see-all-reviews-link-foot']"))).click()

while True:
    for item in WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "[data-hook='review']"))):
        reviewer = item.find_element_by_css_selector("span.a-profile-name").text
        review = ' '.join([i.text.strip() for i in item.find_elements_by_xpath(".//span[@data-hook='review-body']")])
        print(reviewer,review)

    try:
        WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[@data-hook='pagination-bar']//a[contains(@href,'/product-reviews/') and contains(text(),'Next page')]"))).click()
        WebDriverWait(driver, 10).until(EC.staleness_of(item))
    except Exception as e:
        break

driver.quit()
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

but I when I printing reviews it is not working fine why is it so
I've only defined the selector to scrape reviewer name, not reviews.
yeah i know it is running that fast, even tags are not loading so that I could retrieve data like reviews please tell me where to add wait or edit code by fetching reviews
my XPath expression is right because the script prints only 1st or last review of the page,please help me,I so confused about what is happening {review = item.find_element_by_xpath("//span[@class='a-size-base review-text review-text-content']/span")} this is i have tried
Check out the edit @krishan. If you have any further question, please create a new post. Btw, your xpath is wrong. You need to add a dot sign at the very beginning like this .//span to make it correct.
|
1

I don't know whether you will like the way of pagination but simple, neat and clean, few code can complete everything the simplest way where as I scrape the site using Scrapy CrawlSpider.

CODE:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class AmazonReviewsSpider(CrawlSpider):

    name = 'reviews'

    allowed_domains = ['www.amazon.in']

    start_urls = ['https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews']

    rules = (
        Rule(LinkExtractor(restrict_xpaths='//a[@data-hook="review-title"]'), callback='parse_item', follow=False),
        Rule(LinkExtractor(restrict_xpaths='//*[@id="cm_cr-pagination_bar"]/ul/li/a'),follow=True),
        )
        
    def parse_item(self, response):
        yield{
            'Reviewer':response.xpath('//*[@class="a-profile-name"]/text()').get()
        }

OUTPUT:

{'Reviewer': 'Shaheen Khan'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1PL6L4U9NYL58/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2OBKWAHDBDKDA/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1PL6L4U9NYL58/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Sidhesh Mardolkar'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R21ZXYGSCCPE5V/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2OBKWAHDBDKDA/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Atowar Rahman'}
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R32Y55ISEX5B6P/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R19PEVFYAI50FE/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:18 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R21ZXYGSCCPE5V/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'leo'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R32Y55ISEX5B6P/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Asim ahmed'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R19PEVFYAI50FE/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Mahavir singh'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RYD458HW5E42N/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RYD458HW5E42N/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Ashish Modanwal'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RUHTIOZJGQ7YX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RUHTIOZJGQ7YX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'muskan'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2X2WTTAWX9V2J/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2X2WTTAWX9V2J/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Susheel Kumar'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R15DM4BMSG84D8/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2TU8L168L3NFO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R15DM4BMSG84D8/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shashi'}
2021-08-01 12:33:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2TU8L168L3NFO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Akhil'}
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1QZDPG9S17TDN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews)
2021-08-01 12:33:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1Z8DTMO2OF7PT/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1QZDPG9S17TDN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Siddhartha'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_1?ie=UTF8&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1Z8DTMO2OF7PT/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shravya kale'}
2021-08-01 12:33:20 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.amazon.in/gp/customer-reviews/R1XAMZ9LKHPV8A/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RKG3L6Y5ZDMGI/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)       
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R396ADCZXCRGSB/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RKG3L6Y5ZDMGI/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'S singh'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R396ADCZXCRGSB/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Shourya satyam'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R2USNFWP35AWMO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R12T25TWUEVJ80/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3L7T6GAL2W2GC/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R2USNFWP35AWMO/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'santhakumar'}
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RA9C9WOIMZOQR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)       
2021-08-01 12:33:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3NN436HHQPS5G/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_2?ie=UTF8&pageNumber=2&reviewerType=all_reviews)      
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R12T25TWUEVJ80/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Harsh Gupta'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3L7T6GAL2W2GC/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'vishal thakare'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RA9C9WOIMZOQR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Ayushi'}
2021-08-01 12:33:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3NN436HHQPS5G/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'NIKET'}
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/RFMWLES7SUYSR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)       
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R3LZD41TT5MPRN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)      
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/RFMWLES7SUYSR/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Best prodAmazon Customer'}
2021-08-01 12:33:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.in/gp/customer-reviews/R1RZQAQO5T2OAX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD> (referer: https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/B08Z1HHHTD/ref=cm_cr_arp_d_paging_btm_3?ie=UTF8&pageNumber=3&reviewerType=all_reviews)      
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R3LZD41TT5MPRN/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'rama krishna . y'}
2021-08-01 12:33:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.in/gp/customer-reviews/R1RZQAQO5T2OAX/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B08Z1HHHTD>
{'Reviewer': 'Amit Biswas'}
Improve this answer

3 Comments

can you do this in selenium
Yes, selenium with scrapy. Thanks
buddy i need the solution only in selenium without using scrapy and bs4 ,thanks

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.