Improving save/load performance with large arrays of structures in MATLAB

Question

I have a very large array of structs (more than 100k structs) that have to be saved to a file. Later, these have to be loaded and processed one at a time. The current approach is to save it using just save. This takes ~8s to save and ~100s to load.

I've tried a couple of ways to speed this up:

Using the -v6 flags with save. This sped things up, but not significantly.
Serializing and deserializing using getByteStreamFromArray() and getArrayFromByteStream() respectively. This had no effect. Specifically, serializing and deserializing took just as long as simply saving and loading.
(still working on this) Serializing the array, saving it, loading it, then only deserializing each structure as it is processed (rather than the whole array)

Does anyone have any recommendations to improve performance in this situation? It seems like it would be a common problem.

It may not be an option, but you could consider playing with how you store data in the structs. For example, arrays of structs with many fields will be less efficient than a single struct with arrays for fields. In general, anything that groups the data into contiguous blocks in memory should be faster. Also, if you only need to load/update a subset of your variables, just load/save what you need instead of everything. — user20160
– user20160, Commented Jun 11, 2016 at 22:11
@user20160 That last option is something I'm looking into. The entire array needs to be saved, and loading part of an array the recommended way (matfile) is taking just as long (just to call matfile). That's why I'm looking into serializing it, then deserializing only the needed parts (loading in the entire array as a vector of uint8s is almost instantaneous) — Voriki
– Voriki, Commented Jun 11, 2016 at 22:17
If this is an option for your case, you may want to convert your arrays of struct into simple arrays and back before the load/save operations. — P-Gn
– P-Gn, Commented Jun 12, 2016 at 16:55

Yair Altman · Accepted Answer · 2016-06-12 18:15:23Z

1

I believe that getByteStreamFromArray() and getArrayFromByteStream() are used by save() and load() under the hood, so your results are not very surprising to me. You might get better performance using hand-crafted serialization functions that crawl down your structs and only save what's really needed. Additional saving can possibly be achieved by compressing the saved data. You can read some implementation ideas here: http://undocumentedmatlab.com/blog/improving-save-performance

Note #1 - YMMV based on Matlab release, data, and platform

Note #2 - for readers who are not aware of this, getByteStreamFromArray() and getArrayFromByteStream() are both undocumented Matlab functions. The only [unofficial] explanation of their behavior, AFAIK, is provided here: http://undocumentedmatlab.com/blog/serializing-deserializing-matlab-data

answered Jun 12, 2016 at 18:15

Yair Altman

5,72233 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Improving save/load performance with large arrays of structures in MATLAB

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related