How to convert binary data to utf8 string after dart2js

Question

I'm storing my data in the localStorage. Because I store a lot of data, I need to compress it. I do it with GZip. Everything works, but I found out that my convertion from List which the result of GZip to Utf8 is extremely slow. On 500k of data, it can take up to 5 min on a fast computer.

The problem appear mainly during the loading function.

Load:

import 'dart:convert';
import 'package:archive/archive.dart';
import 'package:crypto/crypto.dart';

var base64 = CryptoUtils.base64StringToBytes(window.localStorage[storageName]);
var decompress = new GZipDecoder().decodeBytes(base64);
storage = JSON.decode(new Utf8Decoder().convert(decompress));

Save:

var g = JSON.encode(storage);
List<int> bz2 = new GZipEncoder().encode(new Utf8Encoder().convert(g));
window.localStorage[storageName] = CryptoUtils.bytesToBase64(bz2);

Am I doing something wrong? Is there other way of converting a List to a String in Dart? Also, I do both save and load, so there is no need of utf8.

I tried Ascii, but it throws error for unsupported character. I got kind of working latin1, for which a much higher performance, but I still get some unsupported character.

I use GZipDecoder from the import 'package:archive/archive.dart'; which is fast, the problem is with the Utf8Decoder. — darkzangel
– darkzangel, Commented Feb 6, 2015 at 16:21
I filed dartbug.com/22330. I'm not sure when we will have the time to look into it, though. (Feel free to remind us from time to time on the bug if nothing seems to happen). — Florian Loitsch
– Florian Loitsch, Commented Feb 9, 2015 at 14:57

Pixel Elephant · Accepted Answer · 2015-02-06 16:37:32Z

4

If UTF-8 is really the bottleneck, and you don't need it, you can just use Dart's native String encoding which is UTF-16.

var compressed = new GZipEncoder().encode(str.codeUnits);
var decompressed = new GZipDecoder().decodeBytes(compressed);
var finalStr = new String.fromCharCodes(decompressed);

answered Feb 6, 2015 at 16:37

Pixel Elephant

21.5k9 gold badges70 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Günter Zöchbauer · Accepted Answer · 2015-02-06 16:47:35Z

This is just a hack and is not the best, but it works and it's giving me good performance for now:

String convertListIntToUtf8(List<int> codeUnits) {
  var endIndex = codeUnits.length;
  StringBuffer buffer = new StringBuffer();
  int i = 0;
  while (i < endIndex) {
    var c = codeUnits[i];
    if (c < 0x80) {
      buffer.write(new String.fromCharCode(c));
      i += 1;
    } else if (c < 0xE0) {
      buffer.write(new String.fromCharCode((c << 6) + codeUnits[i+1] - 0x3080));
      i += 2;
    } else if (c < 0xF0) {
      buffer.write(new String.fromCharCode((c << 12) + (codeUnits[i+1] << 6) + codeUnits[i+2] - 0xE2080));
      i += 3;
    } else {
      buffer.write(new String.fromCharCode((c << 18) + (codeUnits[i+1] << 12) + (codeUnits[i+2] << 6) + codeUnits[i+3] - 0x3C82080));
      i += 4;
    }
    if (i >= endIndex) break;
  }
  return buffer.toString();
}

Collectives™ on Stack Overflow

How to convert binary data to utf8 string after dart2js

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related