How to write to a binary file using the fastest way possible for Linux?

Question

I'm working on a C project where we have to write some binary files in a Embedded Linux environment (2.6.37). Normally we are able to write the files in something like 200-300 ms, but eventually the file take up to 10 seconds to be written and we have no idea why - the occurrence is quite randomic with no special event happening in other parts of the system, such as in the UI app.

Either way we are revising or method to write to the file and doing some research on the web (here and here and here) we concluded that writing using native Linux code would be better then doing it pure C even though that may not end up helping much with our problem. For now we are writing in a way similar to this, that is, with these functions:

#include <stdio.h>
const unsigned long long size = 8ULL*1024ULL*1024ULL;
unsigned long long a[size];

int main()
{
    FILE* pFile;
    pFile = fopen("file.binary", "wb");
    for (unsigned long long j = 0; j < 1024; ++j){
        //Some calculations to fill a[]
        fwrite(a, 1, size*sizeof(unsigned long long), pFile);
    }
    fclose(pFile);
    return 0;
}

Well what I would like to know is which would be the native Linux way to do an equivalent operation (and in the fastest way possible)? The links mentioned only tell about copying files, not simply writing to them, so I suppose there might be more specific functions to be used.

Any help appreciated (as well as any tip regarding the original problem).

It's hard to say without details but I'd bet the problem is not the file writing method but something else as I can't imagine how that could take 10s. — olivecoder
– olivecoder, Commented Sep 24, 2015 at 16:49
Could it write in parallel? Is there a file locking? Could it have a deadlock? Could that calculation be more than just calculation? How long could that calculation take? What is the device? Is it really a local file system? — olivecoder
– olivecoder, Commented Sep 24, 2015 at 16:55
8 billion items at 8 bytes/items with 8 bits/byte = 512 gigabits. Writing 512 gigabits in 0.2 seconds means that you have a write bandwidth of 2.5 terabits/second. That's absurdly fast, especially for an embedded system. I don't believe you at all, unless you're building an embedded super-computer. — user3386109
– user3386109, Commented Sep 24, 2015 at 17:13
@TobySpeight I'd say it's worse than that. Assume that you're just writing data to DRAM, and the DRAM has a 64-bit bus. The DRAM would have to be running at 40GHz to sustain a rate of 2.5 terabits/second. There is no DRAM that runs at 40GHz and no processor that can write to DRAM at that speed. With current technology, you would need a DRAM bus that's 1024 bits wide to sustain a data rate of 2.5 terabits/second. — user3386109
– user3386109, Commented Sep 24, 2015 at 17:31

David Aubin · Accepted Answer · 2018-08-10 12:54:09Z

Sounds like you are writing to an SD Card or micro Sd Card. Not all card are created equally. Start with the fastest you can find. I recommend the SanDisk Extreme Pro. They claim to be able to store up to 90Mbits a second.

Two, your, "randomic" comment gives rise to the SD Card possibly doing a load balance. The card actually has a small cpu inside that will do storage reallocation based on sector hits. They do this to elongate the life of the sd card.

If you write a small amount to an sd card it goes to an internal buffer of ram on the sd card itself. Usually they are sized at 512 bytes per block. Sometimes they have multiple blocks of these, "caches" resulting in higher throughput. This gives excellent performance for small file sizes. Also, note that 512. If you can match your writes to be in chunks of 512 you are matching the fastest possible way to transfer to the underlying medium. I.E. never write less than that size ;-)

Collectives™ on Stack Overflow

How to write to a binary file using the fastest way possible for Linux?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related