create stream from another stream and join both streams in node.js

Question

I have a problem with dealing multiple streams. Consider this case, I have data.csv file whose content is

id,val
1,100
2,75

I want to produce an output like this, half should be 50% of val and we need to append same rows but multiplied by 10.

id,val,half
1,100,50
2,75,37.5
10,100,500
20,75,37.5

I am successful to read csv from file, transform it and process it. I am only able to produce 1 stream. thats is, In the beginning there are only 2 row, so my output is also containing 2 rows. I am able to produce result like this

id,val,half
1,100,50
2,75,37.5

but unable to add extra part. that is whole content to be multiplied by 10.

One solution is very simple, read the content in the file, have it in javascript Array/Object format, process it and write to disk. this do not deal with pipes. I want to write whole things in terms of pipes.

I already have code something like this.

// Initialize file streams where the output will be written to
const forecastcsv = fs.createWriteStream(forecastfile);
fs.createReadStream(argv.file).pipe(parser).pipe(transformer).pipe(stringifier).pipe(forecastcsv);

Unable to think how I can create two streams (one is doing half, other is doing multiply to 10) the join them to achieve double rows in final csv file.

Thanks

Did you look at the question stackoverflow.com/questions/16431163/… ? — kyrisu
– kyrisu, Commented Jul 6, 2015 at 4:46
@kyrisu yes, this will solve my problem upto some extent, but I am unable to to create two streams. — Narendra Sisodiya
– Narendra Sisodiya, Commented Jul 6, 2015 at 6:01
Do you want a format where ALL the data that is 2x is after the original data? — Chris Anderson
– Chris Anderson, Commented Jul 6, 2015 at 6:55
If you do this you'll get a, 10*a, b, 10*b etc. Order isn't guaranteed. If that's fine, just have your transform function emit two records one 1x and one 10x. Nothing forces you to emit 1:1. :) — Chris Anderson
– Chris Anderson, Commented Jul 6, 2015 at 7:42

Narendra Sisodiya · Accepted Answer · 2015-07-06 08:27:45Z

1

Thanks @Chris Anderson-MSFT I was having misunderstanding about 1:1, we can emit more data, So here is my transformer which is working. no need to multiple streams.

var transformer = csv.transform(function (record, callback) {
    record.half = record.value/2;
    var extra = JSON.parse(JSON.stringify(record));
    extra.id = extra.id * 10;
    extra.value = extra.value * 10;
    extra.half = extra.half * 10;
    callback.call(this, null, record, extra); 
});

answered Jul 6, 2015 at 8:27

Narendra Sisodiya

3812 gold badges5 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

create stream from another stream and join both streams in node.js

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related