Efficient storing of tabular user generated data in Godot

Question

Over the course of the game, the user generates a lot of very simple data. And array of strings (single letters), timestamps (int) and a boolean array, all equal in size. The length of this "save file" could be up to 1M each array. Though median size might be a lot lower, say a tenth of that.

What mechanism should I use to write it to disk? This is only going to be written/read into memory on manual save/load. I am limited to gdscript solutions.

I considered using a Resource, as they can be saved as binary files which sounds like it might be fast (though I haven't tested). But I am seeing some people concerned about the security of this approach, as someone might sneak in malicious code into a resource and then share it online.

EDIT: This is intended to run on a low end PC, and my goal is not to optimize disk space usage. Since it is only used on save/load, which do not happen very often, it is not crucial that it takes much less than a second to read/write the thing.

Your question is a little bit broad. You should provide more information about your efficiency goals (space storage, time retrieval, data complexity, hardware limitations, intended target devices...). Also, security is a whole different topic that would benefit from being asked as a dedicated, different question. — liggiorgio
– liggiorgio, Commented Apr 21, 2024 at 9:48
Hey, thanks for the comment. I have added some info. Regarding data complexity, I think this is already covered by the first paragraph, unless I misunderstand what you meant by that. — shiroko
– shiroko, Commented Apr 21, 2024 at 11:00

Theraot · Accepted Answer · 2024-04-21 11:22:45Z

You want something you can convert from and to PackedByteArray.

Array of Characters:

You can store them in a PackedStringArray.

However, do not convert it directly to PackedByteArray (the conversion does not let you specify the encoding, and there isn't a guaranteed/cross-platform way to revert the conversion).

Intead concatenate your characters into a single String using String.join.

Then you can convert the String to PackedByteArray with either:
- to_ascii_buffer if you know you only have ascii.
- to_utf8_buffer if it is mostly ascii and you want to optimize for storage size.
- to_utf32_buffer if it is not mostly ascii or you want to optimize for time.
To revert the operation, PackedByteArray has get_string_from_ascii, get_string_from_utf8, and get_string_from_utf32.

Addendum: I think you might be able to store it as a PackedInt32Array storing the codes for the characters instead. I do not know if this would be more performant.
Array of Timestamps:

This one is easy, use either PackedInt32Array or PackedInt64Array depending on the size of your timestamps. Note: Godot int is 64 bits.

They both have a to_byte_array method to convert to PackedByteArray, and you can convert back with either to_int32_array or to_int64_array.
Array of Bools:

For your boolean array, use PackedByteArray directly. Then you can pick your trade off between storing one bool per byte, or using bit flags to better utilize the memory but less performance.

Once you have a PackedByteArray you can opt to compress it with compress, Godot gives you a few algorithms you can choose by passing the optional compression_mode paramter to compress including GZIP and DEFLATE, the default is FASTLZ. To decompress you use - you guessed - decompress.

Finally you can use the store_buffer of a FileAccess object to store the PackedByteArray, and get_buffer to read it.

Note: I remind you that first you need to use FileAccess.open to get a FileAccess object. It will close itself when there are no more references to it (if it is only referenced in a local variable, this is as soon as it goes out of scope), or you can use close.

If you want to store the three arrays on the same file, I recommend you to prepend the sizes, so you can tell get_buffer how much data to read to recover them. Plus you can handle the case where the file is smaller than the specified sizes.

Otherwise, if you have one file per array, you can assume the whole file is the array, and thus use the size of the file.

This approach will avoid going over each item in a loop to send them to a file, and thus will minimize GDScript overhead giving you a good performance.

And, of course, there would be no opportunity for code injection.

I have also mentioned how you can trade off some performance for lower storage. Since you will be loading and saving at specific moments instead of all the time, I believe using compression is acceptable.

If instead you wanted to be able to write to a file as new data became available, you would instead keep the files open, and use the store_* methods. The File API will also allow you to write a compressed file this way by using open_compressed, but that would have different performance, so test if that is acceptable.

If you want some confidence that the file was not tampered with, you might include a hash (Using HashingContext), or HMAC (Using HMACContext).

Just keep in mind that since the hashing / HMAC code is part of the game client, it would be easy to reverse engineer to forge false save files. It's not bad as a quick check of file integrity, but don't consider data loaded from the file to be "trusted". It's still user input, so take any cautions appropriate to your context. — DMGregory
– DMGregory ♦, Commented Apr 21, 2024 at 11:29
This is a phenomenal answer! In the meantime, I have stumbled upon the store_var() method. This seems to be able to store any Variant, and with full_objects = false, could also be resistant to code injection. Any thoughts on using that? — shiroko
– shiroko, Commented Apr 21, 2024 at 22:02
@Jakub Sure, that works. My guess is that being more generic it is less efficient, but the difference might not be significant. You can test if it is good. I'll point out to prevent code injection the important part is allow_objects=false in get_var (as a malicious file won't come from your code). — Theraot
– Theraot, Commented Apr 22, 2024 at 11:45

liggiorgio · Accepted Answer · 2024-04-24 07:32:14Z

To complement the answer provided by @theraot, here are some timings.

Test #0: Uses typed arrays and store_var() without any further optimization.
Test #0b: Identical to #0 but joins the string array into a single string.
Test #1: Uses packed arrays, but without conversion to packed byte array.
Test #2: Uses packed arrays, joined strings, conversion to packed byte array, and still relies on store_var().
Test #3: Same as #2 bur writes arrays separately via store_buffer().

Timings include all steps required (byte array conversion, string join etc.).

There is two-fold decrease in size between #0 and #1, and many-fold increase in speed between #2 VS #0 and #1. There is no big difference between #2 and #3.

Joining strings seems to have massive impact as well.

TEST #0 WRITE: ELAPSED MSEC: 220 FILE SIZE MB: 30.5176544189453
TEST #0 READ: ELAPSED MSEC: 232

TEST #0b WRITE: ELAPSED MSEC: 93 FILE SIZE MB: 20.0272369384766
TEST #0b READ: ELAPSED MSEC: 139

TEST #1 WRITE: ELAPSED MSEC: 172 FILE SIZE MB: 16.2125396728516
TEST #1 READ: ELAPSED MSEC: 101

TEST #2 WRITE: ELAPSED MSEC: 20 FILE SIZE MB: 12.3978424072266
TEST #2 READ: ELAPSED MSEC: 83

TEST #3 WRITE: ELAPSED MSEC: 18 FILE SIZES MB: [3.814697265625, 0.95367431640625, 7.62939453125]
TEST #3 READ: ELAPSED MSEC: 85

Here's a GDScript snippet to execute the above:

extends Control


var array_size := 1_000_000

var strings0: Array[String]
var bools0: Array[bool]
var ints0: Array[int]

var strings: PackedStringArray
var bools: PackedByteArray
var ints: PackedInt64Array

var start: int
var finish: int

var file: FileAccess
var MB := 1024*1024


func _ready():
    set_up()
    test_0()
    print("")
    test_0b()
    print("")
    test_1()
    print("")
    test_2()
    print("")
    test_3()


func set_up():
    for array in [strings, bools, ints, strings0, bools0, ints0]:
        array.resize(array_size)
    
    strings0.fill(".")
    strings.fill(".")
    
    ints0.fill(int(Time.get_unix_time_from_system() * 1000))
    ints.fill(int(Time.get_unix_time_from_system() * 1000))
    
    for i in array_size:
        bools0[i] = i % 2 == 0
        bools[i] = int(i % 2 == 0)


func test_0():
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test0", FileAccess.WRITE)
    file.store_var({
        "strings": strings0,
        "bools": bools0,
        "ints": ints0
    })
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #0 WRITE: ELAPSED MSEC:", finish - start, "FILE SIZE MB:", file.get_length() as float / MB)
    file.close()
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test0", FileAccess.READ)
    var data: Dictionary = file.get_var()
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #0 READ: ELAPSED MSEC:", finish - start)


func test_0b():
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test0b", FileAccess.WRITE)
    file.store_var({
        "strings": "".join(strings0),
        "bools": bools0,
        "ints": ints0
    })
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #0b WRITE: ELAPSED MSEC:", finish - start, "FILE SIZE MB:", file.get_length() as float / MB)
    file.close()
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test0b", FileAccess.READ)
    var data: Dictionary = file.get_var()
    data.strings = data.strings.split("")
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #0b READ: ELAPSED MSEC:", finish - start)





func test_1():
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test1", FileAccess.WRITE)
    file.store_var({
        "strings": strings,
        "bools": bools,
        "ints": ints
    })
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #1 WRITE: ELAPSED MSEC:", finish - start, "FILE SIZE MB:", file.get_length() as float / MB)
    file.close()
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test1", FileAccess.READ)
    var data: Dictionary = file.get_var()
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #1 READ: ELAPSED MSEC:", finish - start)


func test_2():
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test2", FileAccess.WRITE)
    file.store_var({
        "strings": "".join(strings).to_utf32_buffer(),
        "bools": bools,
        "ints": ints.to_byte_array()
    })
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #2 WRITE: ELAPSED MSEC:", finish - start, "FILE SIZE MB:", file.get_length() as float / MB)
    file.close()
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://test2", FileAccess.READ)
    var data: Dictionary = file.get_var()
    data.strings = data.strings.get_string_from_utf32().split("")
    data.ints = data.ints.to_int64_array()
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #2 READ: ELAPSED MSEC:", finish - start)


func test_3():
    var file_sizes: PackedFloat64Array
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://strings", FileAccess.WRITE)
    file.store_buffer("".join(strings).to_utf32_buffer())
    file_sizes.append(file.get_length() as float / MB)
    file.close()
    file = FileAccess.open("user://bools", FileAccess.WRITE)
    file.store_buffer(bools)
    file_sizes.append(file.get_length() as float / MB)
    file.close()
    file = FileAccess.open("user://ints", FileAccess.WRITE)
    file.store_buffer(ints.to_byte_array())
    file_sizes.append(file.get_length() as float / MB)
    file.close()
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #3 WRITE: ELAPSED MSEC:", finish - start, "FILE SIZES MB:", file_sizes)
    file.close()
    
    start = Time.get_ticks_msec()
    
    file = FileAccess.open("user://strings", FileAccess.READ)
    strings = file.get_buffer(file.get_length()).get_string_from_utf32().split("")
    file.close()
    file = FileAccess.open("user://bools", FileAccess.READ)
    bools = file.get_buffer(file.get_length())
    file.close()
    file = FileAccess.open("user://ints", FileAccess.READ)
    ints = file.get_buffer(file.get_length()).to_int64_array()
    file.close()
    
    finish = Time.get_ticks_msec()
    
    prints("TEST #3 READ: ELAPSED MSEC:", finish - start)

Stack Exchange Network

Efficient storing of tabular user generated data in Godot

2 Answers 2

You must log in to answer this question.

Hot Network Questions

Efficient storing of tabular user generated data in Godot

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions