How to replace None values in JSON response?

Question

I'm fetching financial information from an api endpoint and when I get a 200 response through

r = requests.get(url)
data = r.json()

It'll return None for all null values. How do I convert all null/None values to 0? Since it's financial data, the JSON is usually quite massive (300k-400k lines, some with deep nested nulls) so I can't do a try/except block on each TypeError.

An extract of the json response looks something like this:

{'0': 
'Highlights': {'QuarterlyRevenueGrowthYOY': 0.671, 'GrossProfitTTM': 3750684, 'DilutedEpsTTM': 0.2, 'QuarterlyEarningsGrowthYOY': 0.95
5}, 'Valuation': {'TrailingPE': 60.75, 'ForwardPE': 0, 'PriceSalesTTM': 2.0817, 'PriceBookMRQ': 4.207, 'EnterpriseValueRevenue': 1.
806, 'EnterpriseValueEbitda': 0.0952}, 'Technicals': {'Beta': None, '52WeekHigh': 12.35, '52WeekLow': 7.84, '50DayMA': 11.0197, '20
0DayMA': 10.2209, 'SharesShort': 0, 'SharesShortPriorMonth': 0, 'ShortRatio': 0, 'ShortPercent': 0}, 'SplitsDividends': {'ForwardAn
nualDividendRate': 0.18, 'ForwardAnnualDividendYield': 0.0151, 'PayoutRatio': 0.9, 'DividendDate': '0000-00-00', 'ExDividendDate':
'2020-06-11', 'LastSplitFactor': '', 'LastSplitDate': '0000-00-00'}, 'Earnings': {'Last_0': {'date': '2020-06-30', 'epsActual': 0.1
9, 'epsEstimate': None, 'epsDifference': None, 'surprisePercent': None}, 'Last_1': {'date': '2019-12-31', 'epsActual': 1.86, 'epsEs
timate': None, 'epsDifference': None, 'surprisePercent': None}, 'Last_2': {'date': '2019-06-30', 'epsActual': -0.82, 'epsEstimate':
 None, 'epsDifference': None, 'surprisePercent': None}, 'Last_3': {'date': '0000-00-00', 'epsActual': 0, 'epsEstimate': 0, 'epsDiff
erence': 0, 'surprisePercent': 0}}, 'Financials': {'Balance_Sheet': {'currency_symbol': 'EUR', 'quarterly_last_0': {'date': '2020-0
6-30', 'filing_date': None, 'totalAssets': '12810000.00', 'intangibleAssets': '281000.00', 'otherCurrentAssets': '60000.00', 'total
Liab': '4225000.00', 'totalStockholderEquity': '8585000.00', 'deferredLongTermLiab': '74000.00', 'otherCurrentLiab': '1274000.00',
'commonStock': '80000.00', 'retainedEarnings': '311000.00', 'otherLiab': '200000.00', 'goodWill': '3381000.00', 'otherAssets': '730
00.00', 'cash': '4983000.00', 'totalCurrentLiabilities': '4025000.00', 'shortLongTermDebt': None,
... 
}

Yeah you get the point.. a ton of None all over the place. Any quick fixes for this?

What do you want to replace None with? Do you just want to remove the items in the dictionary? — user15002651
– user15002651, Commented Feb 7, 2021 at 17:29
Write a function that goes through all values in the dictionary, and would check for a None variable. If that is true, then grab the value within the keys of the dictionary and replace the dictionary’s value with 0. — user15002651
– user15002651, Commented Feb 7, 2021 at 17:31
Since json size is your concern, consider using a streaming parser for json, which won't load a object model for the entire JSON at once. — Bhaskar
– Bhaskar, Commented Feb 7, 2021 at 17:44

Charles Duffy · Accepted Answer · 2021-02-07 18:05:22Z

1

def recursive_replace(obj, findVal, replaceVal):
  for k, v in obj.items():
    if v == findVal:
      obj[k] = replaceVal
    elif isinstance(v, dict):
      obj[k] = recursive_replace(obj[k], findVal, replaceVal)
  return obj

result = recursive_replace(json.loads(yourdata), None, 0)

answered Feb 7, 2021 at 18:05

Charles Duffy

299k43 gold badges441 silver badges496 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

erikvm · Accepted Answer · 2021-02-08 07:43:35Z

0

Found a way to do it, @Charles Duffy, thanks for the inspiration - borrowed some but couldn't get it quite to work. The final code looks like this if anyone would need it in the future

from collections.abc import Mapping, Iterable


def replace_none_values(noneVal, replaceVal='0.00'): # not sure if this is bad practice 
    if noneVal is None:
        return replaceVal
    if isinstance(noneVal, Mapping):
        return {k: replace_none_values(v, replaceVal) for k, v in noneVal.items()}
    elif not isinstance(noneVal, str) and isinstance(noneVal, Iterable):
        return [replace_none_values(v, replaceVal) for v in noneVal]
    return noneVal

answered Feb 8, 2021 at 7:43

erikvm

9581 gold badge12 silver badges37 bronze badges

1 Comment

Charles Duffy Over a year ago

FWIW, the original data doesn't have any lists, which is why I didn't bother to support them -- part of the importance of having a representative sample included in the question is to allow concrete acceptance tests for answers. That said, I'm glad you've got something that works for you.

Collectives™ on Stack Overflow

How to replace None values in JSON response?

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related