Jquery crashes while parsing large csv file

Question

I have this jquery function which I found on a tutorial site that takes in a csv file input and tablifies it. I tried giving in a large csv file(10,000KB) and my browser crashes. I saw there's a parser library called papa to handle this but is there any other approach to keep my browser from crashing while doing this?

Here's the relevant code:-

 $("#upload").bind("click", function () {

        var regex = /^([a-zA-Z0-9\s_\\.\-:])+(.csv|.txt)$/;
        if (regex.test($("#fileUpload").val().toLowerCase())) {
                            var reader = new FileReader();
                reader.onload = function (e) {
                    var table = $("<table id='mytable' class='table table-striped table-bordered'/>");
                    var rows = e.target.result.split("\n");
                    text1=e.target.result;
                    var csvString = $.trim(text1);
                    var csvRows    = csvString.split(/\n/);
                    var csvHeaders = csvRows.shift().split(';');
                    var headerstr=String(csvHeaders);
                    var count = (headerstr.match(/,/g) || []).length+1;
                    for (var i = 0; i < rows.length; i++) {
                        var row = $("<tr />");
                        var cells = rows[i].split(",");
                        for (var j = 0; j < count; j++) {

                            if(cells[j]){
                            var cell = $("<td/>");

                            cell.html(cells[j]);
                            }
                            else{
                            var cell = $("<td class='info'/>");
                            cell.html("empty");}
                            row.append(cell);

                        }
                        table.append(row);

                    }


                    $("#dvCSV").html('');
                    $("#dvCSV").append(table);
                }

    });

How do I implement this functionality without crashing my browser? Thanks in advance.

Try using a profiler (Chrome's Dev Tools has one) and find out what eats up all the memory. I'm pretty sure those vars inside the for loop are particularly bad. — Sergiu Paraschiv
– Sergiu Paraschiv, Commented Mar 9, 2015 at 9:38
Hi @Sergiu.. Removing the vars inside the for loop didnt help. I am working with firefox.. :( I just need this functionality intact but without crashing my browser — Tania
– Tania, Commented Mar 9, 2015 at 9:45
I did not say removing the vars inside the loop will fix this. What I was saying is that doing that is particularly taxing on memory. You are doing a lot of other stuff that's not memory and CPU efficient. Repeatedly building the same HTML structure inside the loop is also bad. Try building it outside of the loop and only inject the data inside it, then append it to the DOM. Also try to add batches of table lines at a time, not one at a time. jQuery knows $(element).append([list of nodes]) and optimizes it using document fragments. — Sergiu Paraschiv
– Sergiu Paraschiv, Commented Mar 9, 2015 at 10:28
The point is event though PapaParse handles the data you are then responsible of displaying it. I'd first think about paging if I were you. If paging is not OK then you really really need to optimize your jQuery code. — Sergiu Paraschiv
– Sergiu Paraschiv, Commented Mar 9, 2015 at 10:32
Hi. I understand. I have to rewrite this code as to recreate the html outside the for loops. Can you please add your suggestions with a supporting piece of code as a solution? — Tania
– Tania, Commented Mar 9, 2015 at 11:04

Sergiu Paraschiv · Accepted Answer · 2015-03-09 13:43:03Z

4

Two big issues in tackling this problem:

1) The CSV parser. Papa Parse is great. It has support for workers and is a streaming parser - the only way to go with large files.

2) The way you display the data. Simply outputing each row in a table won't work. I crashed my PC twice trying to come up with a working solution. The only way of doing this and the one used by basically any system dealing with large files is to use virtualized lists. I ended up using this one. It's simple and the code is easy to understand.

Here is my JS:

$("#fUpload").bind("change", function(evt) {
    var bigFile = evt.target.files[0];
    var rows = [];
    Papa.parse(bigFile, {
        delimiter: ",",
        newline: "\n",
        header: false,
        dynamicTyping: false,
        worker: false,
        chunk: function(results) {
            rows.concat(rows, results.data);
        },
        complete: function() {
            var list = new VirtualList({
              h: 300,
              itemHeight: 30,
              totalRows: rows.length,
              generatorFn: function(row) {
                  var el = document.createElement("div");
                  el.innerHTML = "<p>ITEM " + row + ' -> ' + rows[row].join(' - ') + "</p>";
                  return el;
              }
            });
            document.body.appendChild(list.container)
        }
    });
});

HTML contains this input: <input type="file" id="fUpload" />

How I configured Papa:

delimiter and newline: if you allow it to try and detect them it will fail or take longer;
worker: this will spawn a worker process. It will be slower but will keep the UI responsive (the UI thread won't do any work). You'll probably want to set this to true in production. (This won't work on JSFiddle because of browser cross-domain security protocol!);
chunk: instead of a callback for each parsed row have one for a larger set of rows. This is faster;

The virtual list config is the default one.

You can run it here.

I tested with a 9.4 MB CSV file containing 1,Foo,100 repeated on each line.

Here's the same but using a table to output the data and added a -1 to VirtualList's totalRows to compensate for the actual length.

edited Mar 9, 2015 at 13:43

answered Mar 9, 2015 at 12:30

Sergiu Paraschiv

10.2k5 gold badges42 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Tania Over a year ago

Thanks for the working solution. Is it possible to tablify this output? And, how do I link papa parse to my code? Is there a CDN mirror to do it??

Sergiu Paraschiv Over a year ago

You can download Papa Parse from their website: papaparse.com It's hosted on GitHub.

Tania Over a year ago

I downloaded and added the papaparse folder inside my working folder. But it doesnt work.

Sergiu Paraschiv Over a year ago

You have to load it in your HTML: <script src="lib/papaparse.js"></script> and probably <script src="lib/vlist.js"></script> too... (where lib/whatever.js is the relative path to the js file). Also see my updated answer for a table version.

Tania Over a year ago

Thank you for the updated code I tried placing the js files alongside the html document which I am working with, also added it in a directory but papaparse.js and vlist.js arent recognized.. :( I can see it works well in the fiddle but i am not able to do it in my html page

|

ton · Accepted Answer · 2015-03-09 10:17:38Z

I recommend you to use console.log('...') to identify where it's locking/breaking, maybe in the first variables(rows, csvString) because the size. So, you can know where to attack.

If not in the before loop block, then i think it's too expensive use jquery in this context, so try a direct DOM approach(inside the loop,at least):

$("#upload").bind("click", function () {
    var regex = /^([a-zA-Z0-9\s_\\.\-:])+(.csv|.txt)$/;
    if (regex.test($("#fileUpload").val().toLowerCase())) {
                        var reader = new FileReader();
            reader.onload = function (e) {
                var table = $("<table id='mytable' class='table table-striped table-bordered'/>");
                var rows = e.target.result.split("\n");
                text1=e.target.result;
                var csvString = $.trim(text1);
                var csvRows    = csvString.split(/\n/);
                var csvHeaders = csvRows.shift().split(';');
                var headerstr=String(csvHeaders);
                var count = (headerstr.match(/,/g) || []).length+1;
                for (var i = 0; i < rows.length; i++) {
                    var row = document.createElement("tr");
                    var cells = rows[i].split(",");
                    for (var j = 0; j < count; j++) {
                        var cell = document.createElement("td");
                        if(cells[j]){

                        cell.appendChild(document.createTextNode(cells[j]));
                        }
                        else{
                            cell.setAttribute('class','info');
                            cell.appendChild(document.createTextNode("empty"));
                        }
                        row.appendChild(cell);

                    }
                    table.append(row);

                }


                $("#dvCSV").html('');
                $("#dvCSV").append(table);
            }

});

I don't tested this code.

Collectives™ on Stack Overflow

Jquery crashes while parsing large csv file

2 Answers 2

9 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related