Parsing a json string with floats as strings in python

Question

I am writing a class object that uses python requests to read json from a URL. The json string is strange as the floats, dates, and integers are strings.

{
"financialStatementList" : [ {
"symbol" : "HUBS",
"financials" : [ {
  "date" : "2018-12-31",
  "Revenue" : "512980000.0",
  "Revenue Growth" : "0.3657",
  "Cost of Revenue" : "100357000.0",
  "Gross Profit" : "412623000.0",
  "R&D Expenses" : "117603000.0",
  "SG&A Expense" : "343278000.0",
  "Operating Expenses" : "460881000.0",
  "Operating Income" : "-48258000.0",
  "Interest Expense" : "21386000.0",
  "Earnings before Tax" : "-61960000.0",
  "Income Tax Expense" : "1868000.0",
  "Net Income - Non-Controlling int" : "0.0",
  "Net Income - Discontinued ops" : "0.0",
  "Net Income" : "-63828000.0",
  "Preferred Dividends" : "0.0",
  "Net Income Com" : "-63828000.0",
  "EPS" : "-1.66",
  "EPS Diluted" : "-1.66",
  "Weighted Average Shs Out" : "39232269.0",
  "Weighted Average Shs Out (Dil)" : "38529000.0",
  "Dividend per Share" : "0.0",
  "Gross Margin" : "0.8044",
  "EBITDA Margin" : "-0.033",
  "EBIT Margin" : "-0.0791",
  "Profit Margin" : "-0.124",
  "Free Cash Flow margin" : "0.1002",
  "EBITDA" : "-17146000.0",
  "EBIT" : "-40574000.0",
  "Consolidated Income" : "-63828000.0",
  "Earnings Before Tax Margin" : "-0.1208",
  "Net Profit Margin" : "-0.1244"
}

An example of the json at the api endpoint is here: financialmodelingprep.com

My problem is that when I decode this, I end up with objects/strings rather than floats or integers.

I've tried:

r = requests.get(url, params)
jd = json.loads(r.text)

as well as:

r = requests.get(url, params)
jd - r.json()

And, also variations using kwargs such as parse_float = float or parse_float=Decimal

My end goal is to get this into format with floats, int, and dates.

The response is literally giving you strings - so requests can't do any magic there and turn them into numbers for you. You have to do that yourself. — rdas
– rdas, Commented Oct 5, 2019 at 18:43
I would assume the way to do this is to use object_pairs_hooks argument in the json.loads and point it to a custom function? Hard to find examples of what that function needs to look like. — leeprevost
– leeprevost, Commented Oct 5, 2019 at 20:28
if i were you, i would not waste my time with re stuff. first try int -- wrap it in the try / catch block, if it works -- you have int. next, float, again in try/catch block. last one -- date conversion. — lenik
– lenik, Commented Oct 7, 2019 at 2:19

leeprevost · Accepted Answer · 2019-10-06 13:19:09Z

1

I ended up needing to write a custom object hook for my json decoder.

I also decided to add a camelizer to shorten the keys.

import requests
import re
import json
from datetime import datetime

quarterdateformat = '%Y-%m-%d'


def camelize(string):
    return "".join(string.split(" "))

def convert_types(d):
    for k, v in d.items():
        #print(k, type(v))
        new_v = v
        if type(v) is str:
            #match for float
            if re.match('[-+]?[0-9]*\.[0-9]+', v):  
                new_v = float(v)

            #match for date
            if re.match('([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))', v):  
                new_v = datetime.strptime(v, quarterdateformat).date()


        d[k] = new_v
    d = {camelize(k): v for k, v in d.items()}
    return d

url = "https://financialmodelingprep.com/api/v3/financials/income-statement/CRM,HUBS"
params = {'datatyupe' : 'json'}
r = requests.get(url, params)
jd= json.loads(r.text, object_hook=convert_types)

convert_types is the object-hook function which uses regex to look for floats and dates and converts them. The camelizer is used at the end of the object hook to convert all keys to CamelCase.

answered Oct 6, 2019 at 13:19

leeprevost

4541 gold badge6 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

leeprevost Over a year ago

Yes, I was lazy. I dont' have any exp notation in my strings.

Collectives™ on Stack Overflow

Parsing a json string with floats as strings in python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related