Merging arrays of objects from different CSVs

Question

I am using D3JS to visualize data collected by Jenkins. The only problem is that I want to visualize multiple lines at the same time and the Jenkins scripts creating the CSV files for these are creating multiple separate files. For example:

File1:

ID;ValueA;ValueB
ID01;10;10
ID02;30;40

File2:

ID;ValueC;ValueD
ID01;20;30
ID03;50;60

This way the final result should be something like this after merging:

ID;ValueA;ValueB;ValueC;ValueD
ID01;10;10;20;30
ID02;30;40;;
ID03;;;50;60

My problem was that concat simply does the following:

ID01;10;10
ID02;30;40
ID01;20;30
ID03;50;60

Your question is not exactly clear: do you want to merge the CSVs in a new CSV or do you want to merge the arrays of objects created after parsing the two CSVs? — Gerardo Furtado
– Gerardo Furtado, Commented Jan 9, 2019 at 23:10
Sorry, you are right. I would like to merge the arrays parsed by d3.csv — CAJ69I
– CAJ69I, Commented Jan 10, 2019 at 8:37

Gerardo Furtado · Accepted Answer · 2019-01-11 00:16:22Z

There are several ways to merge those arrays of objects, which can favour speed, or brevity, or readability etc...

This is one way to do it, out of several. First, let's get our parsed CSVs. Since I cannot use a real CSV in the StackOverflow snippet I'll use a template literal; also, since you have semicolons in your CSV, I'll use a DSV parser:

const file1 = `ID;ValueA;ValueB
ID01;10;10
ID02;30;40`;

const file2 = `ID;ValueC;ValueD
ID01;20;30
ID03;50;60`;

const csv1 = d3.dsvFormat(";").parse(file1);
const csv2 = d3.dsvFormat(";").parse(file2);

Now we get the headers, based on the CSVs:

const headers = [...new Set(csv1.columns.concat(csv2.columns))].filter(function(d) {
  return d !== "ID"
});

Finally comes the function:

const mergedData = [];

csv1.concat(csv2).forEach(function(row) {
  const foundObject = mergedData.find(function(d) {
    return d.ID === row.ID
  });
  if (foundObject) {
    headers.forEach(function(d) {
      if (row[d]) foundObject[d] = row[d];
    });
  } else {
    headers.forEach(function(d) {
      if (!row[d]) row[d] = "";
    });
    mergedData.push(row)
  };
});

We first create an empty array for the merged data. Then, we search in that array an object that matches the ID for each object in the two CSVs: if it's found, the extra columns are created; if it's not found the original row is pushed, with "" as the value for the columns that it doesn't have, which is exactly what d3.csv() does.

Here is the demo:

const file1 = `ID;ValueA;ValueB
ID01;10;10
ID02;30;40`;

const file2 = `ID;ValueC;ValueD
ID01;20;30
ID03;50;60`;

const csv1 = d3.dsvFormat(";").parse(file1);
const csv2 = d3.dsvFormat(";").parse(file2);

const headers = [...new Set(csv1.columns.concat(csv2.columns))].filter(function(d) {
  return d !== "ID"
});

const mergedData = [];

csv1.concat(csv2).forEach(function(row) {
  const foundObject = mergedData.find(function(d) {
    return d.ID === row.ID
  });
  if (foundObject) {
    headers.forEach(function(d) {
      if (row[d]) foundObject[d] = row[d];
    });
  } else {
    headers.forEach(function(d) {
      if (!row[d]) row[d] = "";
    });
    mergedData.push(row)
  };
});

console.log(mergedData);

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>

Collectives™ on Stack Overflow

Merging arrays of objects from different CSVs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related