Javascript runs out of memory major browsers due to array size

Question

I am trying to create an array, that will be MAASSSIVVEE...i read somewhere that javascript can create an array of up to 4.xx Billion or so. The array i am trying to create will likely be in the quadrillions or higher. I don't even know where to go from here. I am going to assume that JS is not the proper solution for this, but i'd like to give it a try...it is for client side, and i would prefer not to bog down my server with this if there are multiple people using it at once. Also, not looking to learn a new language as i am just getting into JS, and code in general.

could i possibly use setTimeout(),0 breaks in the totalcombos function? Time is not really an issue, i wouldn't mind if it took a few minutes to calculate, but right now it just crashes.

i have tried this using a dedicated worker, but it still crashes the host. the worker code is what i am posting, as the host code is irrelevant to this question (it only compiles the original objects and posts them, then receives the messages back).

The code: (sorry for the mess...im coding noob and just an enthusiast)

onmessage = function(event){
//this has been tested on the very small sample size below, and still runs out of memory
//all the objects in these first arrays are formatted as follows.
// {"pid":"21939","name":"John Smith","position":"QB","salary":"9700","fppg":"23"}
// "PID" is unique to each object, everything else could appear in another object.
// There are no repeated objects.
var qbs = **group of 10 objects like above**
var rbs = **group of 10 objects like above**
var wrs = **group of 10 objects like above**
var tes = **group of 10 objects like above**
var ks = **group of 10 objects like above**
var ds = **group of 10 objects like above**

//This code works great and fast with small sets. ie (qbs, rbs, wrs)
function totalcombos() {
    var r = [], arg = arguments, max = arg.length-1;
    function helper(arr, i) {
        for (var j=0; j<arg[i].length; j++) {
            var a = arr.slice(0); // clone arr
            if(a.indexOf(arg[i][j]) != -1){
                j++;
            } else
                a.push(arg[i][j]);              
            if (i==max) {
                r.push(a);
            } else
                helper(a, i+1);
        }
    }
    helper([], 0);
    return r;
};
//WAY TOO BIG...commented out so as not to crash when run
//var tCom = totalcombos(qbs, rbs, wrs, tes, ks, ds);
//postMessage(tCom.length);
}

When the sets get to be larger like 50 objects in each, it just crashes as it is out of memory. I reduce the set with other code but it will still be very large. How would i fix it? I am trying to create all the possible combinations and then go through and reduce from there based on total salary of each group.

Keeping track of 1 quadrillion objects would require more than 1 terabyte of memory. Do you have 1 terabyte of memory? — JLRishe
– JLRishe, Commented Jan 3, 2015 at 14:00
Generally speaking, one should use typed arrays when dealing with a vast amount of homogenous array entries. — ComFreek
– ComFreek, Commented Jan 3, 2015 at 14:02
Im def out of my league on this...im trying to get all possible combos of these objects. Im sure someone has tried something similar before. Obviously i dont have a tb of memory...btw how did you calculate that? — Matthew Martini
– Matthew Martini, Commented Jan 3, 2015 at 14:03
I calculated it with 1 quadrillion * 1 byte = 1 quadrillion bytes ≈ 1 terabyte. An array will require at least 1 byte per element, and usually several times that. BTW, you say that your question is not a specific one, but that's precisely the problem here. Under What types of questions should I avoid asking? you will see "You should only ask practical, answerable questions based on actual problems that you face. Chatty, open-ended questions diminish the usefulness of our site and push other questions off the front page." This is a very open-ended question. — JLRishe
– JLRishe, Commented Jan 3, 2015 at 14:08
All the combinations of what? You've thrown 170 lines of code at us without a clear explanation of what you want to accomplish, and without any sample inputs. Simplify your problem to a sample of 10 input values, and we may be able to show you how to expand it to the number of inputs you need. This also has all the hallmarks of an XY problem (on multiple levels). You say you want all the combinations (of something). What do you want them for? That's the problem to solve. Not the intermediate problem of calculating combinations. — JLRishe
– JLRishe, Commented Jan 3, 2015 at 14:18

Samuel MacLachlan · Accepted Answer · 2015-01-03 14:04:02Z

1

When working with data, regardless of language or platform, its usually best practice to only load the data that's otherwise you encounter errors or bottlenecks etc. as you are finding.

If your data is being stored somewhere like a Database, a JSON file, or a Web Service or an API etc. (anything basically), you'd be better of searching that set of data to retrieve only that which you need, or to at least reduce the size of the Array data your're trying to traverse.

As an analogy, if you're trying to load the whole internet into memory on a PC with only 2GB of RAM, you're going to have a really bad time. :)

answered Jan 3, 2015 at 14:04

Samuel MacLachlan

1,75615 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Javascript runs out of memory major browsers due to array size

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related