Scrapy css selector returns an empty price list

Question

I'm scraping an interactive website with Scrapy but I cannot seem to get the price CSS-selector right. Here is the screenshot of the HTML:

Few selectors I've already tried:

price = response.css(".bui-price-display__value[aria-hidden='true']").css("::text").extract()

price = response.css(".prco-inline-block-maker-helper .bui-price-display__value").css("::text").extract()

price = response.css(".bui-price-display__value.prco-inline-block-maker-helper").css("::text").extract()

Any ideas on what it could work?

Website link: https://www.booking.com/searchresults.html?label=gen173nr-1DCAEoggI46AdIM1gEaJgCiAEBmAExuAEHyAEM2AED6AEB-AECiAIBqAIDuALD6_n6BcACAdICJGU2YTFmOTExLTJmZmMtNDZjOS1iYjk1LWY4OTM5OTFiZDA5ZdgCBOACAQ&sid=c3b17be33020b4a83d961a9fc14cf31d&sb=1&sb_lp=1&src=index&src_elem=sb&error_url=https%3A%2F%2Fwww.booking.com%2Findex.html%3Flabel%3Dgen173nr-1DCAEoggI46AdIM1gEaJgCiAEBmAExuAEHyAEM2AED6AEB-AECiAIBqAIDuALD6_n6BcACAdICJGU2YTFmOTExLTJmZmMtNDZjOS1iYjk1LWY4OTM5OTFiZDA5ZdgCBOACAQ%3Bsid%3Dc3b17be33020b4a83d961a9fc14cf31d%3Bsb_price_type%3Dtotal%26%3B&ss=Maribor&is_ski_area=0&ssne=Maribor&ssne_untouched=Maribor&dest_id=-88556&dest_type=city&checkin_year=2020&checkin_month=10&checkin_monthday=30&checkout_year=2020&checkout_month=11&checkout_monthday=4&group_adults=2&group_children=0&no_rooms=1&b_h4u_keep_filters=&from_sf=1

Hamza Rana · Accepted Answer · 2020-09-13 21:51:18Z

Ofcourse it returns an empty list. And you need to access it with the help of using the same session cookies with the user agents. For scrapy shell, provide the cookies and set user-agent like this:

>> scrapy shell
>> from scrapy import Request
>> req = Request('https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaJgCiAEBmAExuAEHyAEM2AEB6AEB-AECiAIBqAIDuALD6_n6BcACAdICJGU2YTFmOTExLTJmZmMtNDZjOS1iYjk1LWY4OTM5OTFiZDA5ZdgCBeACAQ&sid=88ee1f1b53ea99d93e04dd0a9bd2e49f&tmpl=searchresults&checkin_month=10&checkin_monthday=30&checkin_year=2020&checkout_month=11&checkout_monthday=4&checkout_year=2020&class_interval=1&dest_id=-88556&dest_type=city&dtdisc=0&from_sf=1&group_adults=2&group_children=0&inac=0&index_postcard=0&label_click=undef&no_rooms=1&offset=0&postcard=0&raw_dest_type=city&room1=A%2CA&sb_price_type=total&shw_aparth=1&slp_r_match=0&src=index&src_elem=sb&srpvid=e78f974223200104&ss=Maribor&ss_all=0&ssb=empty&sshis=0&ssne=Maribor&ssne_untouched=Maribor&top_ufis=1&selected_currency=USD&changed_currency=1&top_currency=1&nflt=', headers={'upgrade-insecure-requests': 1,'cookie': '_pxhd=07c11db292e542c424e639bc65a4c6405c9dff060cd2bc061e31fc54e4f0c3df%3A61810c81-f608-11ea-b101-d53d6a14d275', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36'})
>> fetch(req)
>> response.xpath('//*[@id="hotellist_inner"]/div[1]/div[2]/div[3]/div/div/div/div/div[2]/div[1]/div[2]/div/div[3]').get()

This will return:

'<div class="bui-price-display__value prco-inline-block-maker-helper" aria-hidden="true" data-et-mouseenter="\ncustomGoal:AdeKbCcBUfQUaSHbZFVXOJUNQKFcFXZYCaJFSSZRe:2\n">\nUS
$1,225\n</div>'

Use the same cookies you get from Network tab, and use the User-Agents in your code, and you'll be able to scrape. Good luck.

Sorry, I am more of a newbie. I know how to get to the network tab and already looked to get the answer from tutorials. Any way you could tell me which cookies to look for and how to use them in code to get the result from the spider?

Collectives™ on Stack Overflow

Scrapy css selector returns an empty price list

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related