Unable to save json data using python and selenium

Question

I am trying to save json data from https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY. The data.json file is empty. Here is the code

import time, json

from selenium import webdriver
 
driver = webdriver.Firefox()
 
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
 
driver.get(url)
 
time.sleep(5)

button = driver.find_element_by_id("rawdata-tab")
 
button.click()


data = driver.find_element_by_class_name("data").text

d = json.loads(data)

with open('data.json', 'w') as f:
    json.dumps(d, default=lambda o: '<not serializable>')


time.sleep(10)

driver.close()

Can't find where I am going wrong, is there a better way to achieve this?

Ok, what's wrong here is that your URL, i.e. "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY" results in a json directly as it is an API call. Now, button = driver.find_element_by_id("rawdata-tab") this is a selenium code that finds an element in html code with id="rawdata-tab" and clicks it but there is not html or webpage here. So, basically there is no point of using selenium here. — avats
– avats, Commented Nov 11, 2021 at 17:17
json.dump is for writing a file, json.dumps just returns the string. — Peter
– Peter, Commented Nov 11, 2021 at 17:24
@diggusbickus i took your suggestion and modified the code, able to save data in text file with open('data.txt', 'w') as f: f.write(data) time.sleep(5) driver.close() — Evelyn Smith
– Evelyn Smith, Commented Nov 11, 2021 at 17:54

avats · Accepted Answer · 2021-11-11 17:21:48Z

1

As, I have mentioned in my comment, selenium is used for web scraping or more generally mimicking human actions on webpages. But your URL doesn't results in a webpage. Now, from what I understand, try this instead:

import json, requests

url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
 
response = requests.get(url)

d = response.json()

with open('data.json', 'w') as f:
    json.dumps(d, default=lambda o: '<not serializable>')

As your url already results in json, here I am just using requests to get it and it results in a valid json which you can then write however you want to a file.

PS: Try to understand whatever you're writing in code as this will help you debug, otherwise you are always going to get stuck somewhere.

answered Nov 11, 2021 at 17:21

avats

4452 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Evelyn Smith Over a year ago

thank you for your input. I understood the above code and tried it. But it does nothing no errors nothing

folen gateis · Accepted Answer · 2021-11-11 18:52:19Z

0

just add a user-agent to your request headers

import requests
headers={'User-Agent':'Mozilla/5.0 (Android 4.2.1; Mobile; rv:32.0) Gecko/32.0 Firefox/32.0'}
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
data = requests.get(url, headers=headers)
with open('data.json', 'w') as f:
    f.write(data.text)

answered Nov 11, 2021 at 18:52

folen gateis

2,0003 gold badges16 silver badges30 bronze badges

Collectives™ on Stack Overflow

Unable to save json data using python and selenium

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related