Creating a Node.js Readable stream from a JavaScript object — The simplest possible way

Question

Can anyone explain why I'm getting an error after running the following code with node v0.10.21?

I'm aware that I could JSON.stringify() that object in order to achieve more or less the same result, the point here is to make sense of stream.Readable when objectMode is set true.

The error:

net.js:612
    throw new TypeError('invalid data');
          ^
TypeError: invalid data
    at WriteStream.Socket.write (net.js:612:11)
    at write (_stream_readable.js:583:24)
    at flow (_stream_readable.js:592:7)
    at _stream_readable.js:560:7
    at process._tickCallback (node.js:415:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:901:3

The code:

var stream = require('stream');
var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });

rs.push(edad);
rs.push(null);
rs.pipe(process.stdout);

for further readers: wrote an answer in another question regarding similar context. Link: (stackoverflow.com/a/35886734/1453339) — Abu Shumon
– Abu Shumon, Commented Mar 9, 2016 at 9:13

loganfsmyth · Accepted Answer · 2014-01-15 01:01:31Z

29

rs is an objectMode stream, but process.stdout is not, so it is expecting to have Buffer instances written into it. Since it is getting the wrong data type, it is throwing an error.

If you wanted to be able to pipe the objects like this, you would need to have an in-between stream that supports writing as objects and reading as Buffers.

Something like this:

var stream = require('stream');
var util = require('util');

function StringifyStream(){
    stream.Transform.call(this);

    this._readableState.objectMode = false;
    this._writableState.objectMode = true;
}
util.inherits(StringifyStream, stream.Transform);

StringifyStream.prototype._transform = function(obj, encoding, cb){
    this.push(JSON.stringify(obj));
    cb();
};


var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });
rs.push(edad);
rs.push(null);

rs.pipe(new StringifyStream()).pipe(process.stdout);

answered Jan 15, 2014 at 1:01

loganfsmyth

162k31 gold badges349 silver badges259 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

n370 Over a year ago

Anyway I'm still stringifing that object right? Is there any way to just dump the object into the destination leaving with this guy the job of making it a string?

loganfsmyth Over a year ago

@n370 Not sure I follow. That's what you'd get if you just return rs. You just can't pipe rs directly to a stream expecting a string. You could certainly leave it up to the destination stream to accept an object though.

n370 Over a year ago

Oh I got it! so in our case here, can we make process.stdout expect an object changing it into a string before output to stdout? can we make it with pure Node and JS? I really appreciate your attention @loganfsmyth

loganfsmyth Over a year ago

Unfortunately no, that's why you need an intermediate like StringifyStream. You could potentially write your stream to support both, but the reality is that is really depends on your usecase. Someone could just as easily want to encode as XML instead, so it doesn't make sense to have stdout know anything about that.

slang · Accepted Answer · 2015-07-21 15:37:35Z

4

As loganfsmyth noted, rs is in objectMode, while process.stdout isn't. So it is expecting to have Buffers/Strings written to it, and is throwing a TypeError when it gets an Object.

We need to convert the stream of Objects into text, which is what JSONStream is made to do:

JSONStream = require('JSONStream');
rs.pipe(JSONStream.stringify()).pipe(process.stdout);

edited Jul 21, 2015 at 15:37

answered Mar 27, 2015 at 23:15

slang

6427 silver badges26 bronze badges

Comments

Petr Cibulka · Accepted Answer · 2016-01-16 22:22:01Z

3

I am a person who does not know much about Node (I just need to have my Gulp setup working). Because of that, the fact that I need 20+ lines of code and weird calls like stream.push(null) to work with variable in the stream seems a little crazy.

This worked for me:

var data = { "prop1": "val1", "prop2": "val2" };
var dest = 'path/to/dest/config.json';
var s = require('stream');

var stream = new s.Readable();
stream.push(JSON.stringify(data));    // stream apparently does not accept objects
stream.push(null);                    // no idea why this is needed, but it is

return stream
    .pipe()                           // whatever you need
    .pipe($.fs.createWriteStream(dest));

edited Jan 16, 2016 at 22:22

answered Jan 16, 2016 at 22:03

Petr Cibulka

2,5335 gold badges31 silver badges46 bronze badges

1 Comment

Alexander Santos Over a year ago

I tried editting the answer to improve it, but it seems that the queue is full. The stream.push(null) actually represents the end of the stream (Like \0). Also, Readable stream apparently is an array of strings that is converted to stream, so all elements must be strings.

Collectives™ on Stack Overflow

Creating a Node.js Readable stream from a JavaScript object — The simplest possible way

3 Answers 3

4 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related