1

I am trying to optimize the interaction between two scripts I have. Two things I thought of are the c++ program not terminating unless you manually kill it, or generating all info in python before feeding it to c++.

Explanation of the problem:

What the scripts do: C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple. Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.

In theory, this works. However, as it is right now, it opens and closes the c++ program for each call. For one array that is no problem, but I'm trying to upscale to 25k arrays, and in the future to 6+ million arrays. Obviously it is then no longer feasible to open/close it each time, especially since the c++ program first has to load a 130mb VCD file to function.

Two options I thought of myself were to generate all arrays first in python, then feed them to the c++ program and then analyze all results. However, I wouldn't know how to do this with 6M arrays. It is not important however that the results I get back are in the same order as the arrays I feed in.

Second option I thought of was to make the c++ program not quit after each call. I can't program in c++ though so I don't know if this is possible, keeping it 'alive' so you can just feed arrays into it at times and get an answer.

(Note: I cannot program in anything else than python, and want to do this project in python. The c++ program cannot be translated to python for speed reasons.)

Thanks in advance, Max.

Improve this question
6
  • 3
    I felt a great disturbance in the Force, as if millions of C++ programs suddenly cried out in terror and were suddenly silenced. And then I wondered "Where is the code?", for without the code there can be no real answer. Commented Jan 28, 2011 at 10:14
  • What way do you feed the C++ program (pipe on stdin/stdout) or socket ?. Commented Jan 28, 2011 at 10:15
  • As of now I don't yet have a set way to feed it to c++, I was brainstorming about how to do this and then came up with this question. I basically still have to write the entire 'wrapper' around the c++ program, so far I only have that and the python array-generator. Commented Jan 28, 2011 at 10:18
  • Why would you need a python script and a C++ program to pass each other number ? Can you be more specific in describing your problem and asking the question ? Commented Jan 28, 2011 at 10:20
  • Do you have the source of the C++ program, or do you only have the compiled program? Commented Jan 28, 2011 at 10:50

5 Answers 5

Reset to default
1

Firstly, just to be pedantic, there are no C++ scripts in normal use. C++ compiles, ultimately, to machine code, and the C++ program is properly referred to as a "program" and not a "script".

But to answer your question, you could indeed set up the C++ program to stay in memory, where it listens for connections and sends responses to your Python script. You'd want to study Unix IPC, particularly sockets.

Another way to approach it would be to incorporate what the C++ program does into your Python script, and forget about C++ altogether.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Ok I'm sorry, I'll refer to it as program from now on and no longer as script. Incorporating what the c++ program does in python is not feasible for speed reasons, it would take way too long for the intended purpose. But it's good to hear I can make the c++ program live in memory. I'll read up on Unix IPC and sockets, thanks.
nothing to be sorry about! just that you'll get answers faster if you use the right terminology. sometimes, anyway!
1

Without the source code or the exact specifications of the Python script and the C++ program, it's difficult to provide more information, but you could modify the C++ code to repeatedly read the array from the standard input and then write the results to standard output.

Then you could use the Python subprocess module to launch the C++ program from your Python script and communicate with it.

Note that simply wrapping a loop around the main() function of the C++ program will not be very helpful, because apparently the main issue is the time the program needs in order to read its data (the VCD that you mentioned).

The loop needs to be strictly around the code that computes the result - which means that you may have to factor everything else out in a way that allows the result computation to be done repeatedly without each run contaminating the next ones.

Improve this answer

2 Comments

The program itself is actually extremely simple. It is just a giant lookup table (the VCD) with a simple function that takes the array and returns the value from the VCD. So I would only need to 'loop' the one function that gives me the return value? That seems easy enough.
@Max: as long as you read from stdin, write to stdout in each loop iteration and nothing outside the loop is modified, it should be fine.
1

Okay, your best course of action is probably to write a C/C++ extension to Python that is able to call the C++ code that does the calculation you want. This is not terribly difficult, it will only require a minimal amount of C/C++ coding to make it work. A good explanation of extending Python can be found on the Python page at http://docs.python.org/extending/extending.html

What you in effect do is change your C++ program to be a dynamic library that the Python process can link in and call from the Python script.

If you need a bit of help getting it to work I'm sure we can help you out.

Improve this answer

Comments

0

I think the best way is to build C++ extension module for python.
There are lot of ways to do it.
If you have c++ sources you can try SWIG After that you can use c++ functions/object directly inside python - and manage them by python modules (here processing). It is really simple.

Improve this answer

2 Comments

Say my c++ program does only two things, load the VCD file and have some function in it. Using SWIG if I understand correctly lets me do 'import c++ program' in python, and then lets me call the function in python, right? That seems like the easiest way. Two questions though, does that retain the speed of the c++ function, so it doesn't get slowed down to 'python levels'? And secondly, at what moment does the VCD file get loaded? When I import the SWIG-made module, or each time I call the function?
The speed will be comparable (lets say 99% of the original program). So you shouldn't care about this. Do you have this source available? SWIG just makes c++ functions and classes available in python. So if you have a class X, you can reach it as a X, if you have main method you can reach it as a main method (the modules are compiled and not linked - so they are not callable like a program)
0

I think you're doing it wrong

What the scripts do: C++ program (not made by me, and I can't program in c++ very well): takes a 7 number array and returns a single number, simple. Python script (mine, and I can program a bit in python): generates those 7 number arrays, feeds them to the c++ program, waits for an answer and adds it to a list. It then makes the next array.

You have this?

python generate_arrays.py | someC++app | python gather_array.py

This allows you to run the three parts in parallel, using every Core of every CPU on the box. The OS makes sure that all three run concurrently.

If you're still not getting 100% CPU Load, you'll have to do something like this.

( python generate_arrays.py --even | someC++app >oneFile ) & ( python generate_arrays.py --odd | someC++app > anotherFile )
python gather_array.py oneFile anotherFile

That will run two copies of python generate_arrays.py and two copies of your magical C++ program.

You'll have to rewrite your generate_arrays.py program so that it takes a command-line option. When the option is --even, you generate 3 million arrays. When the options is --odd you generate the other 3 million arrays.

This (python | c++) & (python | c++) should get to 100% cpu use.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.