Python Numpy array2string performance

Question

I am working on a python script to set up an input file for a solid mechanics simulation software. The part of the script I'm struggling with is where I format nodal data (node numbers and the corresponding 3D coordinates) from a numpy array to string format, with one node's data per line. I've been working on improving the run time of the script, and this is by far the slowest portion of the whole thing. I originally used np.array2string, but found that it gets pretty slow above about 100,000 nodes.

The numpy array with nodal data is called 'nodes', and is an Nx4 array, where N is the number of nodes in the model and can vary from run to run. There is some additional formatting of the data in 'nodeString' that takes place later in the code to remove extraneous brackets, parentheses and commas, but that is relatively quick and pretty much the same between all methods below.

I've tried a couple different settings for the parameters of array2string:

np.set_printoptions(threshold=np.inf)
nodeString = np.array2string(nodes, precision=4, suppress_small=True, separator=',') # original syntax
nodeString = np.array2string(nodes, suppress_small=True, separator=',')
nodeString = np.array2string(nodes, precision=4, suppress_small=False, separator=',')

I've tried array_str instead:

np.set_printoptions(threshold=np.inf)
nodeString = np.array_str(nodes, precision=4, suppress_small=True)

I've also tried just writing the numpy array to a text file and opening it back up:

np.set_printoptions(threshold=np.inf)
fmt = '%s', '%.4f', '%.4f', '%.4f'
np.savetxt('temp.txt', nodes, delimiter=',', fmt=fmt)
with open('temp.txt', 'r') as file:
   nodeString = file.read()

Comparison of processing time vs number of nodes for different numpy array to string techniques

(The run time reported in the figure above is in seconds.) By far, the fastest technique I've found is to save the data and then read it back in. I'm really surprised by this, and I wonder if I'm doing something wrong with the native numpy functions like array2string that are negatively impacting their performance. I'm a Mechanical Engineer, and I've been told we code by brute force rather than by elegance, so if someone has a better way of doing what I'm trying to do, or an explanation why it's faster to write and read than just to reformat, I'd appreciated any insight. Thanks!

Taking a big step back; why do you want coordinates as strings in the first place? — roganjosh
– roganjosh, Commented Dec 21, 2022 at 19:46
Have you tried to set fixed value (close to your average array size) to treashold in set_printoptions ? — RomanPerekhrest
– RomanPerekhrest, Commented Dec 21, 2022 at 19:58
savetxt just iterates on the 'rows' of nodes, and for each does a Python string format, e.g. '%s, %.4f, %.4f, %.4f' % tuple(row)`, and writes that string to the file. — hpaulj
– hpaulj, Commented Dec 21, 2022 at 20:02
I haven't examined the array2string code, but I imagine it makes heavy use of python formatting. numpy does not implement any of its own compiled string code. It has good performance when working with numbers, not with strings. — hpaulj
– hpaulj, Commented Dec 21, 2022 at 20:42
@roganjosh The file that gets submitted to the simulation software is ultimately a text file. It contains the nodal information, but it contains other things too that are in a different format, like element and material definitions. I want to write the nodal data in the numpy array into the file with everything else eventually, so I have to get rid of the brackets, etc associated with numpy arrays. — J May
– J May, Commented Dec 22, 2022 at 19:31

David Kaftan · Accepted Answer · 2022-12-21 19:53:49Z

1

Instead of writing and reading from a file, read and write to a StringIO object:

from io import StringIO
sb = StringIO()
np.savetxt(sb, nodes, delimiter=',', fmt=fmt)
nodeString = sb.getvalue()

I believe this will save you time by avoiding reading and writing from the harddrive. Rather it keeps everything in memory.

answered Dec 21, 2022 at 19:53

David Kaftan

2,2241 gold badge14 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Numpy array2string performance

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related