Convert a large ArrayBuffer into hexadecimal string representation in JavaScript

Question

I have an ArrayBuffer with content that needs to be converted to a hex-string, that should be sent in a JSON message. The data can be up to 2 GB. For this I would like to have a string that works as a dynamic array, so I efficiently could push_back individual characters. In C++, we have std::string (or std::vector), I think Java has a StringBuilder. What is the way of doing it in JavaScript?

I tried += but that starts to take minutes after reaching 128 MB. I guess this is because it results in an O(n^2) algorithm.

Note: I also need to do the inverse operation.

You could build an array (each index being a single two-character string) and then .join() it, but 2 gigabytes (well, 8 once it's turned into a hex string, which really means 10 because the buffer is still there) is going to be pretty taxing on the machine. — Pointy
– Pointy, Commented Jul 9, 2023 at 13:24
Seems like a streaming solution would be a much better idea. — Pointy
– Pointy, Commented Jul 9, 2023 at 13:24
+= normally is optimised by JS engines into a constant average operation, but I guess this might break down when working with humongous strings. As Pointy suggested, do not build the message as a single string value but rather write it into a stream. — Bergi
– Bergi, Commented Jul 9, 2023 at 13:39
Related: JavaScript does not need a StringBuilder, Does JavaScript have a built-in stringbuilder class?, and How can I build/concatenate strings in JavaScript? — Peter Mortensen
– Peter Mortensen, Commented Jul 21, 2023 at 8:43
@PeterMortensen The answer to that question is probably. However to make a robust interface that does not expose problems with variable byte length may be difficult. But, having an efficient StringBuilder causes people to think about time complexity, which by itself is good for developing programming skills. — user877329
– user877329, Commented Jul 23, 2023 at 14:04

user877329 · Accepted Answer · 2023-07-10 18:46:13Z

0

Well, why not use a temporary Uint8Array, like so:

function to_hex_string(bytes)
{
    let tmp = new Uint8Array(2*bytes.length);
    for(let k = 0; k != bytes.length; ++k)
    {
        let value = bytes[k];
        let msb = value >> 4;
        let lsb = value & 0xf;
        tmp[2*k] = msb < 10 ? msb + 48 : (msb - 10) + 65;
        tmp[2*k + 1] = lsb < 10 ? lsb + 48 : (lsb - 10) + 65;
    }

    return new TextDecoder().decode(tmp);
}

This outperforms joining an array. Though it fails when trying to convert the array to a string when trying 512 MiB. I guess there is some hardcoded limit. Maybe a JS process may not use more than 4 GiB because of a 32-bit VM.

answered Jul 10, 2023 at 18:46

user877329

6,2968 gold badges55 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert a large ArrayBuffer into hexadecimal string representation in JavaScript

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related