How to load 30 csv files in parallel using queuejs with d3js?

Question

Two main questions here:

We want to load files in parallel
Because these files are loaded in parallel, we don't know the exact order they are loaded. I want to parse a key from each file name, with the array of file content parsed by d3.csv as their corresponding values

That is said, the returned results by

   q.awaitAll(function(error, results) { console.log("all done!"); });

should be

results={
  "key1":array_read_from_myData_key1.csv,
  "key2":array_read_from_myData_key2.csv,
  ...
  "key30":array_read_from_myData_key30.csv
}

How to add each file name as a key to each array (results[i]) below?

var q = queue(), // create the queue
    dataSources = [ // the data sources
        'myData_key1.csv',
        'myData_key2.csv',
        'myData_key3.csv',
        ...,
        'myData_key30.csv'
    ];


// Go through each data source and add it to the queue:
dataSources.forEach(function (source) {
    q.defer(function (callback) {
        d3.csv(source, callback);
    });
});

// Wait for all requests to be completed:
q.awaitAll(function (error, results) {
    console.log(results);
})

Alex_B · Accepted Answer · 2015-07-15 02:53:33Z

1

You could expand your call back inside here:

dataSources.forEach(function (source, index) {
  q.defer(function (callback) {
    d3.csv(source, function(data){
       var fileData = {
           name: dataSources[index],
           data: data
       };
       callback(fileData);
   });
  });
});

Based on your comment you could try doing the following but not 100% sure that it will work :

 var finalData = {};

 dataSources.forEach(function (source, index) {
  q.defer(function (callback) {
    d3.csv(source, function(data){
       if (data) {
         //regex the key from file name
         var matcher = /^(myData)_([a-z][a-z][a-z][0-9])\.csv$/;
         var key = matcher.exec(dataSources[index])[1];
         //assign value to external object
         finalData[key] = data;
         //callback success
         callback(true);
       }
       else {
        //log error if required
        callback(false);
       }
   });
  });
});

// Wait for all requests to be completed:
q.awaitAll(function (error, results) {
   console.log(finalData);
})

Hope this solves your issue.

edited Jul 15, 2015 at 2:53

answered Jul 14, 2015 at 20:52

Alex_B

8487 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Osiris Over a year ago

Thanks @Alex_B. It is very close to what I want. Is it possible to return the results as a dict instead of an array of dict? i.e. how to obtain results={ "key1":array_read_from_myData_key1.csv, "key2":array_read_from_myData_key2.csv, ... "key30":array_read_from_myData_key30.csv }

Alex_B Over a year ago

The regex needs to be improved based on actual file names.

Osiris Over a year ago

This doesn't work. q.awaitAll executed after loading the first csv file, not wait after all csv files loaded. And the first csv array now loaded into "error" variable of q.awaitAll. The "results" is undefined. Any further input?

Osiris Over a year ago

Sorry but both versions don't work. I cann't edit the comment above,

Alex_B Over a year ago

change callback(true); to callback(null,finalData) and see what the result of finalData is.

|

Collectives™ on Stack Overflow

How to load 30 csv files in parallel using queuejs with d3js?

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related