how to set proxy value and run linux command in python

Question

In linux to set a proxy value you do the following.

proxy=http://$user:[email protected]:${NUM}
http_proxy="$proxy" https_proxy="$proxy" ${COMMAND}

For security reasons, if you run this in a subshell, you are not explicitly letting your password in the open, or in the logs. Problem with this approach is, I have to set user name and password everytime I want to run a command.

Therefore decided to write a python code for it. I have a working version in C. Just wanted to learn more in Python. I have found nice ways to encode and decode my password, and after most of the hooplahs, I pass it to this function to test proxy connection.

def test_connection(creds,proxy_url):
    import pycurl
    import cStringIO

    buf = cStringIO.StringIO()
    test_url="http://www.google.com"

    c = pycurl.Curl()
    c.setopt(c.URL, test_url)
    c.setopt(c.WRITEFUNCTION, buf.write)
    c.setopt(c.PROXY, proxy_url)
    c.setopt(c.PROXYPORT, 8080)
    c.setopt(c.PROXYTYPE, c.PROXYTYPE_HTTP) 
    c.setopt(c.PROXYAUTH, c.HTTPAUTH_NTLM) 
    c.setopt(c.PROXYUSERPWD, creds) 
    c.perform()
    buf.close()
    return c.getinfo(c.RESPONSE_CODE)

Where I'm having problems is using suprocess, I do understand that subprocess does not allow you to use export, since it is not really a command. Subprocess module errors with 'export' in python on linux?

this is my implementation

finalCommand = ["/bin/sh", "-c"]
finalCommand.append(http_proxy)
finalCommand.append(https_proxy)
for x in bashCommand:
    finalCommand.append(x)

print subprocess.call(finalCommand)
process = subprocess.Popen(finalCommand,stdout=subprocess.PIPE)


out, err = process.communicate()

print "Output ... \n %s" % (out)

if err == None: 
    print "No errors"
else: 
    print "Errors ... \n %s" %(err)

Unfortunately, after several tests, my program always return no output and no error. I have printed the output of the curl, so i know the decode, encode, or proxy isn't the issue. Any suggestions?

POST-ANSWER EDIT: Interaction between Python script and linux shell

env did solve my problem, but I also had to refer to the thread above. Some of the commands I ran were interactive one, and as it explains it well in the thread, PIPE doesn't work properly with interactive programs.

Are you just looking for the env argument to Popen? Or are you actually trying to launch a shell and run scripts directly within it instead of in subshells, or something else funky? — abarnert
– abarnert, Commented Jul 23, 2014 at 20:15
Meanwhile, please give us a MCVE—in particular, notice the "complete" part. Without knowing what's in http_proxy, https_proxy, and bashCommand, it's hard to debug what might be going wrong. (Also, is there's a reason you're first doing call on the command, and then creating a Popen with the exact same command and calling communicate on it?) — abarnert
– abarnert, Commented Jul 23, 2014 at 20:22
It's not so much that "PIPE doesn't work properly with interactive programs" (although that is true before Python 3.2, you're probably not running into that, and if you are, install the subprocess32 backport to fix that), it's that some interactive programs require you to change the buffering, and others aren't talking to stdin at all (e.g., they're reading a password without echoing it and need a real tty, or they just ask libpam to do it for them, etc.) so piping to stdin doesn't do any good. And also that writing code to talk interactively over a pipe without blocking is hard. — abarnert
– abarnert, Commented Jul 24, 2014 at 17:40

abarnert · Accepted Answer · 2014-07-23 20:25:14Z

5

It's hard to be sure without knowing exactly what commands you're trying to run, but I'm pretty sure what you want to do here is just set up the environment for your subprocess, using the env argument to Popen:

env = dict(os.environ)
env['http_proxy'] = proxy
env['https_proxy'] = proxy
for command in commands:
    out = subprocess.check_output(command, env=env)

If you want to modify your own environment, rather than just the subprocesses' environments, just modify os.environ in place. (See the documentation for platform-specific issues, and how to deal with them.)

Meanwhile, the reason you're getting no errors is simple:

process = subprocess.Popen(finalCommand,stdout=subprocess.PIPE)
out, err = process.communicate()

If you don't pass stderr=subprocess.PIPE to the Popen constructor, it doesn't capture stderr, so err ends up as None.

As a side note, you almost never want to check == None. Often, just if not err: is sufficient. When it's not, if err is not None: is almost always required. The set of cases where == is necessary but not insufficient is vanishingly small. See the Programming Recommendations in PEP 8 for (slightly) more details.

And one more side note: You can just write finalCommand.extend(x). The list.extend method does the same thing as looping over an iterable and appending each element one by one, except that it's more readable, harder to get anything wrong, more concise, and faster.

edited Jul 23, 2014 at 20:25

answered Jul 23, 2014 at 20:18

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

kmassada Over a year ago

thanks for the tips.. I'll have to dig this out myself, but setting an env variable, wouldn't that show up inside the env variables of the command line..

abarnert Over a year ago

@kmassada: I'm not sure what your question means. If a child process (like your Python script) modifies its environment, or launches a child of its own with a different environment, that won't affect the parent shell, much less any other shells you have running. But if you launch a shell with subprocess.call(['/bin/sh'], env=env) or something, then yes, the keys in env will all show up as the variables of the shell you launched.

caram Over a year ago

If you're using nodejs puppeteer, don't forget to add env[no_proxy]=somehost if necessary.

Collectives™ on Stack Overflow

how to set proxy value and run linux command in python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related