Failed to obtain response using Selenium python to automate visiting a website

Question

Basically what I want to do is to automate visiting this website. If you visit using Chrome, manually picking report type and date, a download link will appear. I tried to automate this process using python + selenium by following the suggestions in the linked question. But selenium clicking the Get Data button didn't work. I don't know why. Please help.

Please see this link https://stackoverflow.com/a/68598528/12469120 for more context about how to use DatePicker.

Her is my code.

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys


import time
driver = webdriver.Chrome()
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source":
        "const newProto = navigator.__proto__;"
        "delete newProto.webdriver;"
        "navigator.__proto__ = newProto;"
    })
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")

time.sleep(5)
datepicker = driver.find_element_by_id("date")
datepicker.click()

selectMonth = driver.find_element_by_xpath('//select[@class="ui-datepicker-month"]')
for option in selectMonth.find_elements_by_tag_name('option'):
    if option.text == 'Mar':
        option.click() 
        break

selectYear = driver.find_element_by_xpath('//select[@class="ui-datepicker-year"]')
for option in selectYear.find_elements_by_tag_name('option'):
    if option.text == '2018':
        option.click() 
        break 

days = driver.find_elements_by_xpath('//a[@class="ui-state-default"]')
days[4].click()
time.sleep(2)
select = Select(driver.find_element_by_id('h_filetype'))
select.select_by_visible_text('Market Activity Report')

wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()
time.sleep(20)

Update

The site can detect Selenium driven browsers. A workaround is added in the code. The download link can be obtained.

cruisepandey · Accepted Answer · 2021-07-31 17:04:03Z

1

The issue is that, there is no file available for the specified date.

No file found for specified date. Try another date.

Though, there were some tweak needed, I have made some changes, see below :-

driver = webdriver.Chrome()
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.nseindia.com/products/content/derivatives/equities/archieve_fo.htm")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[id='date']"))).click()

select_month = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[@class='ui-datepicker-month']"))))
select_month.select_by_visible_text('Mar')

select_year = Select(wait.until(EC.visibility_of_element_located((By.XPATH, "//select[@class='ui-datepicker-year']"))))
select_year.select_by_visible_text('2017')

days = driver.find_elements_by_xpath("//a[@class='ui-state-default']")
days[3].click()
time.sleep(2)

select = Select(wait.until(EC.element_to_be_clickable((By.ID, "h_filetype"))))
select.select_by_value('fomkt')

element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()

Update 1:

look, this css selector represent the 'file not available' message

div#spanDisplayBox td

it's actually working manually, but hangs with automation (chrome browser). I have faced this issue earlier as well with the same site. However, I have added a method that could help us in following way :

element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'getdata-button')))
element.click()

try:
    if(len(driver.find_elements(By.CSS_SELECTOR, "div#spanDisplayBox td"))) > 0:
        print("Response is present")
        print(wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#spanDisplayBox td"))).text)
    else:
        print("There is no response")
except:
    print("something went wrong")
    pass

However, when I ran this I got this output :-

There is no response

I would recommend you to check this with Firefox, (cross browser, cross platform and see if that helps).

edited Jul 31, 2021 at 17:04

answered Jul 31, 2021 at 16:33

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

bssrdf Over a year ago

thanks for looking into it. Yes, the date I picked has no files. But the point is my code and yours showed no response (either the download link if available or 'file not available' message) from server.

cruisepandey Over a year ago

@bssrdf : okay looks like issue with automation, you can get some idea if you look above into updated 1 section.

bssrdf Over a year ago

Thanks for your updates. You are right, that site can detect whether the browser is being automated by Selenium or not. I tried one trick from here and it worked (the download link showed up). I have updated my code with the trick.

Collectives™ on Stack Overflow

Failed to obtain response using Selenium python to automate visiting a website

Update

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Update

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related