Removing properties from array of objects, memory impact

Question

I am processing a heavy array of objects on a node app, and at some point I want to remove two properties from all objects in the array.

At the same time I am measuring the node memory impact on rss (Residente Set Size).

What I am finding is that just the act of deleting them consumes a lot of memory.

Example, it's actually a lot bigger and with lots of objects. File size of json is 200MB.

[
  {
   keep: 'foo',
   prop1: 'remove',
   prop2: 'remove'
  },...
]

This consumes the most from 500MB goes to 1000MB

const clean = original.map((obj) => {
  delete obj.prop1
  delete obj.prop2
  return obj
})

This also consumes a lot, also around 1000MB

original.forEach((obj) => {
  delete obj.prop1
  delete obj.prop2
})

This comsumes the least, goes to around 650MB

const clean = original.map(({ prop1, prop2, ...obj }) => obj)

But if I do not delete them at all then it does not consume anymore than the original 500MB. What is going on? Should not removing properties make the memory lighter?

because you are creating a second array. map creates a new array. developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/… if you want to loop use forEach — winner_joiner
– winner_joiner, Commented Nov 14, 2021 at 14:41
The foreach, which is not creating another array, also uses a lot of memory thou — Álvaro
– Álvaro, Commented Nov 14, 2021 at 14:42
The .map() method produces an array of the same size so that should be fairly straight forward. The .foreach() method passes the entire array as a parameter - hence doubles up for that reason. — Randy Casburn
– Randy Casburn, Commented Nov 14, 2021 at 14:42

Randy Casburn · Accepted Answer · 2021-11-14 15:32:28Z

3

You can use either of the .map() calls you provided (they are faster than .forEach()).

When you've completed the deletion, simply set the original variable to null. This will allow the JS engine to mark this variable for Garbage Collection and it will be removed from memory. You won't see the result immediately as we have no idea when GC will run and cannot control it. But the memory will be reclaimed very quickly.

Like this:

const clean = original.map((obj) => {
  delete obj.prop1
  delete obj.prop2
  return obj
})
original = null;

const clean = original.map(({ prop1, prop2, ...obj }) => obj);
original = null;

The issue here is that we as web devs don't have any control over memory consumption other than what I have proposed. Internally the V8 engine decides how things are put together in memory and all memory management issues are relegated to it.

edited Nov 14, 2021 at 15:32

answered Nov 14, 2021 at 15:06

Randy Casburn

14.2k1 gold badge19 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Álvaro Over a year ago

This is odd, If I set the original to null as you recommened, I get 724.6 MB of rss, If I don't I get 692.78 MB. Then I did it again just to double check and this time adding null frees 20MB

Randy Casburn Over a year ago

It sounds like you may be using references in your data structure. We don't know how your array of objects is constructed. If you've created the structure using references to objects rather than copies then it is possible those objects are still referenced in memory. As long as objects are referenced they will not be garbage collected and the memory will not be recovered - so you'll end up with some duplicates due to this referencing.

Álvaro Over a year ago

Thanks, what would be an example of 'references in your data structure' so I can look for that in my code? the array of objects comes from a JSON file I read

Randy Casburn Over a year ago

"the array of objects comes from a JSON file I read" - then this is not a factor.

Randy Casburn Over a year ago

Please see the update to my answer

|

skoll · Accepted Answer · 2024-03-22 07:58:12Z

0

Node uses the V8 engine, and V8 engine has two different internal representations of objects. A so called fast mode (default) and a slow mode (aka dictionary mode). When you delete a property from an object, the object gets converted to slow mode. What is significant in this case, is that the slow mode also uses more memory.

If possible, use obj.prop1 = null instead.

Read more: https://v8.dev/blog/fast-properties

answered Mar 22, 2024 at 7:58

skoll

2,4804 gold badges33 silver badges36 bronze badges

Collectives™ on Stack Overflow

Removing properties from array of objects, memory impact

2 Answers 2

10 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related