0

Objects in JavaScript contain key-value pairs. The cost of a typical pair (using DevTools Profiler) is a reference to the key name, 8 bytes, and the cost of the object: 4 bytes for small ints, 8 bytes for numbers, references, etc.

The cost of the keys add up, especially in arrays with millions of objects.

Is there a asm.js sort of way to use typed arrays for arrays of identical objects?

Yes, I know this seems like a pain, but for a particular project, this may be required.

The sort of approach I'm thinking of is using a template JS object who's keys describes the offset into a typed array for each key's value, along with its type. For arrays of these objects, there'd be multiple of these object spans.

Thus two questions:

1 - Are my assumptions correct .. and there is no optimization in chrome/modern browsers that optimize the key costs? Possibly with constraints used here: http://www.2ality.com/2013/08/protecting-objects.html

2 - If so, is there a library for handling typed arrays as objects? Or any articles or gists etc?

Improve this question
2
  • How many items are you going to have in your collection? And, if you don't use an object, how will you do key lookup? Commented Nov 13, 2013 at 19:02
  • Good point! Our goal is arrays in the 10s of millions with each item in the array an object. The simplest approach would be to have a single object with the keys we want, then have an array be the value for each key. So we may "invert" the array of objects to be an object with arrays. Another trick is to have the fixed object (the objects are all identical, instances of a class) described by a typed array buffer layout so that "subarrays" of the buffer map onto an object described by its template. Commented Nov 14, 2013 at 16:40

1 Answer 1

Reset to default
1

If you have millions of objects, all with the same set of known keys and you have so many of them that memory is an issue, then you probably don't want to store your data as javascript objects at all.

You probably want to think about this like a database problem. You want:

  1. A semi-compact storage format
  2. A way to find the right record in the storage format (this depends upon what the data is and how you need to access it.
  3. A way to read the semi-compact storage format and turn it into a live javascript object only when you want to actually use that object in your code.
  4. A way to write changes back to the semi-compact storage format.

For example, if you had seven keys (e.g. fields) and three were numbers and four were strings and one of the numbers was your lookup key, then you could do this:

  • Create three typed arrays (one for each numeric value)
  • Create one regular array. This will hold all the string values concatenated together with a unique separator character.
  • Create a master key lookup object

As you read your data in, presumably from multiple ajax calls, you do the following:

  • Note the length of one of your arrays (they are all the same length) as this will be your new record number.
  • Add the lookup key to the master key object and set the value of the key to be the record number.
  • Add each numeric value to each typed array (each will be the record number index in the array)
  • Concat all the string values together with a separator between them in order by key and then put the contactenated string value into the string array.

Now you have a semi-compact storage format and a key lookup means. When you want to lookup a value, you use the master key lookup object. The value for the key will be the record number which is an index into the other arrays. You can create two functions that will find a record and return a javascript object form of the record (all data in key/value pairs on the object) and another function that will write an object (that might have changed = but the master key can't change) back to your storage format.

This makes a few assumptions about your data that you have one master key that won't change that you use for lookup and that you can find a separator to bind all the string values together and then separate them apart later and that you can know when you go to store all this and that you know what the keys are and that the objects generally all have the same keys.

If any of those assumptions are not true, then the design would have to be adapted to deal with that, but hopefully you get the idea of using something other than a giant array of objects to store your data and then constituting a given object only when you need to work with that record's data.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

We've considered a similar thing model. It did bring to mind asm.js too! I wonder if there is an asm.js library of some sort that is typed array oriented but gives the illusion of objects.
Another question we've pondered is how Native Memory vs Heap Memory is allocated. My assumption is that they have a memory layout where Native starts at the top, and the heap from the bottom or something similar. This allows use to use as much of native as we'd like, but limiting heap in the process. Do you think this is the way it works?
I just posted a question on this .. didn't want this question to migrate into two questions! Here's the post: stackoverflow.com/questions/20007263/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.