0

This is the url https://www.lowes.com/store/AK-Anchorage/2955 when we reach this url there is a button name "Shop this store" if we click the button the request made by the clicking the button and using the link are the same but still after clicking the button one gets a different page then directly using the link. I need to make the same request as the button is making.

I need to make request to "https://www.lowes.com/store/AK-Anchorage/2955" then i need to make the same request as made my clicking the button.

I have tried making the requests two consecutive times to get the desired page but no luck.

url='https://www.lowes.com/store/AK-Anchorage/2955'
ua = UserAgent()
header = {'User-Agent':str(ua.chrome)}
response = requests.get(url, headers=header)
response = requests.get(url, headers=header)
Improve this question
5
  • Hmm, this one's a bit tricky. It makes a POST request (or 2 if you include the one to akstat.io) before the GET request to the actual page you want. You'll probably want to use a requests.Session. You need to replicate the POST payload by going through the stacktrace and reading the JavaScript. Alternatively, this might be a job for Selenium. Commented Oct 7, 2019 at 5:24
  • Please can you explain a bit i am using the session as well can you please look in the link, are the both requests same as i mentioned, i do not want to use selenium as it will slow down the process. Commented Oct 7, 2019 at 7:32
  • i want to make same request as the button makes using the python request module after of course making request to the store link first. Commented Oct 7, 2019 at 7:34
  • @GordonAitchJay. Commented Oct 7, 2019 at 8:15
  • I made mistake when I said it pages 2 POST requests. It does that when I click another blue "Shop this store" button further down the page. Check my answer. Clicking the blue button in the centre of the page takes you to the same page. What is the difference between the 2 pages? And, what do you actually want to get off the page? Commented Oct 7, 2019 at 9:38

2 Answers 2

Reset to default
2

So, this seems to work. I get a 200 OK response both times, and the content isn't the same length.

For what it's worth, in Firefox, when I click the blue "Shop this store" button, it takes me to what appears to be the exact same page, but without the blue button I just clicked. In Chrome (Beta), when I click the blue button, I get a 403 Access denied page. Their server isn't playing nice. You might struggle to achieve what you want to achieve.

If I call session.get without my headers, I never get a response at all. So they're obviously checking the user-agent, possibly cookies, etc.

import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0",
           "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
           "Accept-Language": "en-US,en;q=0.5",
           "Accept-Encoding": "gzip, deflate, br",
           "Upgrade-Insecure-Requests": "1",}

session = requests.Session()

url = "https://www.lowes.com/store/AK-Anchorage/2955"

response1 = session.get(url, headers=headers)
print(response1, len(response1.content))

response2 = session.get(url, headers=headers)
print(response2, len(response2.content))

Output:

<Response [200]> 56282
<Response [200]> 56323

I've done some more testing. The server times out if you don't change the user-agent from the default Python Requests one. Even changing it to "" seems to be enough for the server to give you a response.

You can get product information, including description, specifications, and price, without selecting a specific store. Take a look at this GET request, with no cookies, and no session:

import requests, json

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}

url = "https://www.lowes.com/pd/Google-Nest-Learning-Thermostat-3rd-Gen-Thermostat-and-Room-Sensor-with-with-Wi-Fi-Compatibility/1001080012"

r = requests.get(url, headers=headers, timeout=5)
print("return code:", r)
print("content length:", len(r.content))

for line in r.text.splitlines():
    if "window.digitalData.products = [" in line:
        print("This line includes the 'sellingPrice' and the 'retailPrice'. After some splicing, we can treat it as JSON.")
        left = line.find(" = ") + 3
        right = line.rfind(";")
        print(json.dumps(json.loads(line[left:right]), indent=True))
        break

Output:

return code: <Response [200]>
content length: 107134
This line includes the 'sellingPrice' and the 'retailPrice'. After some splicing, we can treat it as JSON.
[
 {
  "productId": [
   "1001080012"
  ],
  "productName": "Nest_Learning_Thermostat_3rd_Gen_Thermostat_and_Room_Sensor_with_with_Wi-Fi_Compatibility",
  "ivm": "753160-83910-T3007ES",
  "itemNumber": "753160",
  "vendorNumber": "83910",
  "modelId": "T3007ES",
  "type": "ANY",
  "brandName": "Google",
  "superCategory": "Heating & Cooling",
  "quantity": 1,
  "sellingPrice": 249,
  "retailPrice": 249
 }
]

The product description and specification can be found in this element:

<section class="pd-information met-product-information grid-100 grid-parent v-spacing-jumbo">

(It's ~300 lines, so I'm just going to copy the parent tag.)

There's an API that takes a product id and store number, and returns the pricing information:

import requests, json

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}

url = "https://www.lowes.com/PricingServices/price/balance?productId=1001080012&storeNumber=1955"

r = requests.get(url, headers=headers, timeout=5)
print("return code:", r)
print("content length:", len(r.content))
print(json.dumps(json.loads(r.text), indent=True))

Output:

return code: <Response [200]>
content length: 768
[
 {
  "productId": 1001080012,
  "storeNumber": 1955,
  "isSosVendorDirect": true,
  "price": {
   "selling": "249.00",
   "retail": "249.00",
   "typeCode": 1,
   "typeIndicator": "Regular Price"
  },
  "availability": [
   {
    "availabilityStatus": "Available",
    "productStockType": "STK",
    "availabileQuantity": 822,
    "deliveryMethodId": 1,
    "deliveryMethodName": "Parcel Shipping",
    "storeNumber": 907
   },
   {
    "availabilityStatus": "Available",
    "productStockType": "STK",
    "availabileQuantity": 8,
    "leadTime": 1570529161540,
    "deliveryMethodId": 2,
    "deliveryMethodName": "Store Pickup",
    "storeNumber": 1955
   },
   {
    "availabilityStatus": "Available",
    "productStockType": "STK",
    "availabileQuantity": 1,
    "leadTime": 1570529161540,
    "deliveryMethodId": 3,
    "deliveryMethodName": "Truck Delivery",
    "storeNumber": 1955
   }
  ],
  "@type": "item"
 }
]

It can take multiple product numbers. For example: https://www.lowes.com/PricingServices/price/balance?productId=1001080046%2C1001135076%2C1001091656%2C1001086418%2C1001143824%2C1001094006%2C1000170557%2C1000920864%2C1000338547%2C1000265699%2C1000561915%2C1000745998&storeNumber=1564

You can get information on every store by using this API which returns a 1.6MB json file. maxResults is normally set to 30, and query is your longitude and latitude. I would suggest saving this to disk. I doubt it changes much.

https://www.lowes.com/wcs/resources/store/10151/storelocation/v1_0?maxResults=2000&query=0%2C0

Keep in mind the PricingServices/price/balance endpoint can take multiple values for storeNumber separated by %2C (a comma), so you won't need 1763 separate GET requests. I still made multiple requests using a requests.Session (so it reuses the underlying connection).

Improve this answer
Sign up to request clarification or add additional context in comments.

13 Comments

Thanks for the response, actually after directly reaching the store using the link then clicking the search button and after that if i put a direct link to the product like "lowes.com/pd/…" i get product listing on that specific store for which after reaching the store using the link and pressing the shop this button. If i do not press the "shop this" store and search the product directly using the link i dont get the product listing of that specific store.
i need to scrape product information, when i click the"shop this" button after reaching the store i can only get product info from that store if i directly request product link, there are multiple stores, but links to the product listing are same, when we click the shop this store button and search the product link we get product info specific to that store.
I browsed their website with the network monitor open, logging all the requests. I then looked through it for anything interesting. I've found a way to get every store id (1763 total). Making GET requests to the PricingServices/price/balance endpoint, I found that 5 of the 12 products (that I linked to in my answer) are cheaper at certain stores. Is this specifically what information you were after? Do you care about any other information? What else do you need? (I've updated my answer again with the API to get every store id)
Yes, this endpoint returns information on every store, including the store id. As I explained in my answer, use (this other endpoint)[lowes.com/PricingServices/price/… to get the price for a specific product at a specific store. Both productId and storeNumber take multiple values, separated by %2C. If you put in 20 store numbers each time, you only need ~90 requests, instead of 1763.
I can't vouch for any specific learning resources, but you need to be comfortable with these things: manipulating/iterating/accessing lists and dicts, and creating a dict using either the json module or the json() method of a requests response object.
|
0

It depends on what do you want to do with the data. In the URL you already have shop ID.

When clicking on the button it issues the request to https://www.lowes.com/store/api/2955 to get shop information. Is it what you're looking for?

If so, you don't need 2 requests, but rather just one to get needed shop information.

Improve this answer

3 Comments

Thanks for the response, actually after directly reaching the store using the link then clicking the search button and after that if i put a direct link to the product like "lowes.com/pd/…" i get product listing on that specific store for which after reaching the store using the link and pressing the shop this button. If i do not press the "shop this" store and search the product directly using the link i dont get the product listing of that specific store.
i need to scrape product information, when i click the"shop this" button after reaching the store i can only get product info from that store if i directly request product link, there are multiple stores, but links to the product listing are same, when we click the shop this store button and search the product link we get product info specific to that store.
i need to replicate this browser behavior with requests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.