Execute curl command within a Python script

Question

I am trying to execute a curl command within a python script.

If I do it in the terminal, it looks like this:

curl -X POST -d  '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://localhost:8080/firewall/rules/0000000000000001

I've seen recommendations to use pycurl, but I couldn't figure out how to apply it to mine.

I tried using:

subprocess.call([
    'curl',
    '-X',
    'POST',
    '-d',
    flow_x,
    'http://localhost:8080/firewall/rules/0000000000000001'
])

and it works, but is there a better way?

You don't have to use cURL to POST something to a server. requests can do so quite easily (as can urllib, with a bit more effort) — roippi
– roippi, Commented Sep 23, 2014 at 16:50
Check this to know more about executing shell cmds in python stackoverflow.com/questions/89228/… — Sainath Motlakunta
– Sainath Motlakunta, Commented Sep 23, 2014 at 16:51

Community · Accepted Answer · 2020-06-20 09:12:55Z

285

Don't!

I know, that's the "answer" nobody wants. But if something's worth doing, it's worth doing right, right?

This seeming like a good idea probably stems from a fairly wide misconception that shell commands such as curl are anything other than programs themselves.

So what you're asking is "how do I run this other program, from within my program, just to make a measly little web request?". That's crazy, there's got to be a better way right?

Uxio's answer works, sure. But it hardly looks very Pythonic, does it? That's a lot of work just for one little request. Python's supposed to be about flying! Anyone writing that is probably wishing they just call'd curl!

it works, but is there a better way?

Yes, there is a better way!

Requests: HTTP for Humans

Things shouldn’t be this way. Not in Python.

Let's GET this page:

import requests
res = requests.get('https://stackoverflow.com/questions/26000336')

That's it, really! You then have the raw res.text, or res.json() output, the res.headers, etc.

You can see the docs (linked above) for details of setting all the options, since I imagine OP has moved on by now, and you - the reader now - likely need different ones.

But, for example, it's as simple as:

url     = 'http://example.tld'
payload = { 'key' : 'val' }
headers = {}
res = requests.post(url, data=payload, headers=headers)

You can even use a nice Python dict to supply the query string in a GET request with params={}.

Simple and elegant. Keep calm, and fly on.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Aug 1, 2015 at 17:11

OJFord

11.2k10 gold badges71 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

20 Comments

Gary Over a year ago

I am using python 2.4.3. can't use requests. ImportError: No module named requests.

OJFord Over a year ago

@Gary pip install requests

OJFord Over a year ago

@marciokoko Most definitely! :) requests is "just" HTTP under the hood. Use it as you would curl, but fly faster with Python.

idungotnosn Over a year ago

+1. To the best of my knowledge, anything that can be done with cURL can also be done via python requests. Might as well use that one instead.

galois Over a year ago

this worked perfectly for what I came looking for. Thanks @OJFord

|

Foysal Khandakar Joy · Accepted Answer · 2023-10-23 08:21:47Z

70

Use this tool (hosted here for free) to convert your curl command to equivalent Python requests code:

Example: This,

curl 'https://www.example.com/' -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Origin: https://www.example.com' -H 'Accept-Encoding: gzip, deflate, br' -H 'Cookie: SESSID=ABCDEF' --data-binary 'Pathfinder' --compressed

Gets converted neatly to:

import requests

cookies = {
    'SESSID': 'ABCDEF',
}

headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'Origin': 'https://www.example.com',
    'Accept-Encoding': 'gzip, deflate, br',
}

data = 'Pathfinder'

response = requests.post('https://www.example.com/', headers=headers, cookies=cookies, data=data)

edited Oct 23, 2023 at 8:21

Foysal Khandakar Joy

4266 silver badges17 bronze badges

answered Jul 31, 2018 at 6:30

Nitin Nain

5,5231 gold badge40 silver badges51 bronze badges

1 Comment

mdabdullah Over a year ago

While @OJFord has enlightened us why not to use curl within python, Nitin has depicted the simplest way to implement the same using "requests". I highly recommend this answer.

Uxío · Accepted Answer · 2014-09-23 17:04:04Z

46

You could use urllib as @roippi said:

import urllib2
data = '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}'
url = 'http://localhost:8080/firewall/rules/0000000000000001'
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
for x in f:
    print(x)
f.close()

answered Sep 23, 2014 at 17:04

Uxío

1,39111 silver badges12 bronze badges

3 Comments

Kiran Vemuri Over a year ago

is urllib2 more time efficient compared to subprocess?

Uxío Over a year ago

It depends of the subprocess, but spawing subprocesses calling commands when the language has core libraries to do so it's not definitely the right way to do it

foobar Over a year ago

TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

hyades · Accepted Answer · 2018-08-31 18:38:54Z

40

If you are not tweaking the curl command too much you can also go and call the curl command directly

import shlex
cmd = '''curl -X POST -d  '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://localhost:8080/firewall/rules/0000000000000001'''
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()

edited Aug 31, 2018 at 18:38

answered Sep 23, 2014 at 16:49

hyades

3,2101 gold badge22 silver badges36 bronze badges

3 Comments

Kiran Vemuri Over a year ago

Thanks, I implemented it using subprocess.call()

user3722096 Over a year ago

I was very well helped with the suggestion of @Ollie Ford: I have installed Requests under W10. From command line, started up Python and then composed my requested URL. From now I'll have to figure out how to set up (and view the contents of) a stream in a .py file. Suggestions welcome!

Anton Duzenko Over a year ago

"subprocess" is not defined

Simas Joneliunas · Accepted Answer · 2020-02-04 11:12:26Z

14

Try with subprocess

CurlUrl="curl 'https://www.example.com/' -H 'Connection: keep-alive' -H 'Cache- 
          Control: max-age=0' -H 'Origin: https://www.example.com' -H 'Accept-Encoding: 
          gzip, deflate, br' -H 'Cookie: SESSID=ABCDEF' --data-binary 'Pathfinder' -- 
          compressed"

Use getstatusoutput to store the results

status, output = subprocess.getstatusoutput(CurlUrl)

edited Feb 4, 2020 at 11:12

Simas Joneliunas

3,15620 gold badges32 silver badges39 bronze badges

answered Feb 4, 2020 at 10:35

Ramesh Ponnusamy

1,81716 silver badges24 bronze badges

Comments

Prashant Kanak · Accepted Answer · 2021-02-17 13:10:59Z

7

You can use below code snippet

import shlex
import subprocess
import json

def call_curl(curl):
    args = shlex.split(curl)
    process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    return json.loads(stdout.decode('utf-8'))


if __name__ == '__main__':
    curl = '''curl - X
    POST - d
    '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}'
    http: // localhost: 8080 / firewall / rules / 0000000000000001 '''
    output = call_curl(curl)
    print(output)

answered Feb 17, 2021 at 13:10

Prashant Kanak

1071 silver badge4 bronze badges

Comments

Ganesh prasad · Accepted Answer · 2018-06-20 09:53:03Z

3

Rephrasing one of the answers in this post, instead of using cmd.split(). Try to use:

import shlex

args = shlex.split(cmd)

Then feed args to subprocess.Popen.

Check this doc for more info: https://docs.python.org/2/library/subprocess.html#popen-constructor

answered Jun 20, 2018 at 9:53

Ganesh prasad

1661 silver badge7 bronze badges

Comments

Aditya · Accepted Answer · 2022-06-03 08:25:43Z

1

Inside the subprocess module, there is one more option called run use it

from subprocess import run
run(curl -X POST -d  '{"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"}' http://localhost:8080/firewall/rules/0000000000000001)

answered Jun 3, 2022 at 8:25

Aditya

4663 silver badges12 bronze badges

Comments

mysteryegg · Accepted Answer · 2023-07-19 01:59:10Z

0

With Python 3, the built-in HTTP protocol client is a viable alternative to cURL. Using the example provided:

>>> import http.client, urllib.parse
>>> params = urllib.parse.urlencode({"nw_src": "10.0.0.1/32", "nw_dst": "10.0.0.2/32", "nw_proto": "ICMP", "actions": "ALLOW", "priority": "10"})
>>> headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
>>> conn = http.client.HTTPConnection("localhost:8080")
>>> conn.request("POST", "/firewall/rules/0000000000000001", params, headers)
>>> response = conn.getresponse()
>>> print(response.status, response.reason)
302 Found
>>> data = response.read()
>>> conn.close()

edited Jul 19, 2023 at 1:59

answered Jul 18, 2023 at 23:57

mysteryegg

5415 silver badges13 bronze badges

Collectives™ on Stack Overflow

Execute curl command within a Python script

9 Answers 9

Don't!

Yes, there is a better way!

Requests: HTTP for Humans

20 Comments

1 Comment

3 Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

Don't!

Yes, there is a better way!

20 Comments

1 Comment

3 Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related