Difficulty converting string to JSON object in order to urlencode it

Question

I'm having some trouble converting a string that I built up over my script into a JSON object (I'm trying to build up post data to make a post request to the API endpoint).

I'm getting the following error:

bos-mpqpu:config_parse rabdelaz$ python3 lexparse.py
{"__class__":"HttpTestClient","description":"CP Codes","url":"://","serverip":"","method":"GET","enabled":true,"headers":[],"tests":[],"purge":false,"procs":[]}
Traceback (most recent call last):
  File "lexparse.py", line 448, in <module>
print(ast.literal_eval(test_case_ACT_template.substitute({'protocol': "",'host': "", 'path' : "", 'query' : "", 'serverip' : "", 'tests' : ""})))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 85, in literal_eval
return _convert(node_or_string)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 66, in _convert
    in zip(node.keys, node.values))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 65, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 84, in _convert
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x103c50c18>

Here is the code that generates the above:

test_case_ACT_template = Template('{"__class__":"HttpTestClient","description":"CP Codes","url":"$protocol://$host$path$query","serverip":"$serverip","method":"GET","enabled":true,"headers":[],"tests":[$tests],"purge":false,"procs":[]}')


print(test_case_ACT_template.substitute({'protocol': "",'host': "", 'path' : "", 'query' : "", 'serverip' : "", 'tests' : ""}))

print(ast.literal_eval(test_case_ACT_template.substitute({'protocol': "",'host': "", 'path' : "", 'query' : "", 'serverip' : "", 'tests' : ""})))

notice that the first print statement results in the output line that starts with {"__class__"

My overall objective is to urlencode my json string but some searching on SO lead me to believe I should first do the ast.literal_eval before urlencoding it.

the end goal looks like this:

form_data = {'name' : 'CP Code Demo', 'configfilename' : '1-DR4E2_kona-ion.aws-eu-ssl.template_pm-2', 'type' : 'json', 'script' : urlencode(json.load(post_data))}

Where post_data is a series of Template substitutions.

The original error that lead me to start looking into ast.literal_eval was as follows:

   Traceback (most recent call last):
   File "lexparse.py", line 521, in <module>
form_data = {'name' : 'CP Code Demo', 'configfilename' : '1-DR4E2_kona-ion.aws-eu-ssl.template_pm-2', 'type' : 'json', 'script' : urlencode(post_data)}
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/parse.py", line 862, in urlencode
"or mapping object").with_traceback(tb)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/parse.py", line 854, in urlencode
  raise TypeError
TypeError: not a valid non-string sequence or mapping object

Why in the world are you using ast.literal_eval() for this? What are you actually trying to achieve here? — zwer
– zwer, Commented Jul 25, 2017 at 19:00
Updated my question a bit. My end goal is to urlencode the data I'm collecting in the templates shown in the question. — Ramy
– Ramy, Commented Jul 25, 2017 at 19:02
What library are you using to send that form_data? Typically you don't want to do your own escaping/encoding. — zwer
– zwer, Commented Jul 25, 2017 at 19:04
Well, because the site i'm trying to post to requires a client certificate and between python and my Mac OS I'm not able to get the post working with the requests library. So now I'm trying to build a curl command to run on the command line - which is what I would do in Ruby. Hence the self-encoding here. So my goal at the moment (unless there's a better way) is to run subprocess.run(). — Ramy
– Ramy, Commented Jul 25, 2017 at 19:07

zwer · Accepted Answer · 2017-07-25 19:27:13Z

1

Just to be in line with the question, your Template seemingly produces a valid JSON so just parse it into a Python dict and then give it to the urllib.parse.urlencode() to turn it into valid URL params, something like:

import json
from urllib.parse import urlencode
from string import Template

your_template = Template('{"__class__":"HttpTestClient","description":"CP Codes",'
                         '"url":"$protocol://$host$path$query","serverip":"$serverip",'
                         '"method":"GET","enabled":true,"headers":[],"tests":[$tests],'
                         '"purge":false,"procs":[]}')

post_data = your_template.substitute({'protocol': "",
                                      'host': "",
                                      'path': "",
                                      'query': "",
                                      'serverip': "",
                                      'tests': ""})

form_data = {'name': 'CP Code Demo',
             'configfilename': '1-DR4E2_kona-ion.aws-eu-ssl.template_pm-2',
             'type': 'json',
             'script': urlencode(json.loads(post_data))}

print(form_data)

Then you can feed it to whatever you want. I'd still strongly advise on solving the problem you have with your certificates than just offsetting this to cURL - the issues you might experience when trying to pass large amount of data via CLI can easily overcome the issues you've been experiencing with certificates.

answered Jul 25, 2017 at 19:27

zwer

25.9k3 gold badges53 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ramy Over a year ago

this was actually really helpful. I know it's a shit way of doing this but baby steps. this is all just to get management on board with the project.

Ramy Over a year ago

wait I see another issue here. urlencode actually turns my json object into key value pairs. That's actually not what I want. What I want is to escape the entire json object and post it as is. Going to google that now but if you know how to escape rather than encode let me know.

zwer Over a year ago

If you want to URL escape it, don't parse it as JSON at all - just feed it to urllib.parse.quote, e.g. 'script': urllib.parse.quote(post_data) . It all depends on what do you want to send script value. Technically, you could just send it with no quoting at all - subprocess.Popen() ought to escape the special characters so you can pass it to cURL, and cURL should be able to deal with the rest all on its own.

Ramy Over a year ago

yes indeed! urllib.parse.quote is exactly what i needed. Thank you so much for the help!

Ken Kinder · Accepted Answer · 2017-07-25 19:30:33Z

The reason it's failing is that you aren't capitalizing True and False.

literal_eval evaluates an expression as a Python object, not a JSON one, but with constraints. Per the docs:

ast.literal_eval(node_or_string) Safely evaluate an expression node or a Unicode or Latin-1 encoded string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None.

This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.

Notice where it says "booleans and None". In Python, booleans are True and False, and must be capitalized. Your string literal has 'true' and 'false'.

You can take your literal expression and see directly why it doesn't evaluate:

>>> x = {"__class__":"HttpTestClient","description":"CP Codes","url":"://","serverip":"","method":"GET","enabled":true,"headers":[],"tests":[],"purge":false,"procs":[]}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'true' is not defined

If you're trying to evaluate JSON data, instead of using the AST module, you might consider the JSON module.

Collectives™ on Stack Overflow

Difficulty converting string to JSON object in order to urlencode it

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related