Node.js: Processing a stream without running out of memory

Question

I'm trying to read a giant logfile (250,000 lines), parsing each line into a JSON object, and insert each JSON object to CouchDB for analytics.

I'm trying to do this by creating a buffered stream that will process each chunk seperately, but I always run out of memory after about 300 lines. It seems like using buffered streams and util.pump should avoid this, but apparently not.

(Perhaps there are better tools for this than node.js and CouchDB, but I'm interested in learning how to do this kind of file processing in node.js and think it should be possible.)

CoffeeScript below, JavaScript here: https://gist.github.com/5a89d3590f0a9ca62a23

fs = require 'fs'
util = require('util')
BufferStream = require('bufferstream')

files = [
  "logfile1",
]

files.forEach (file)->
  stream = new BufferStream({encoding:'utf8', size:'flexible'})
  stream.split("\n")
  stream.on("split", (chunk, token)->
    line = chunk.toString()
    # parse line into JSON and insert in database
  )
  util.pump(fs.createReadStream(file, {encoding: 'utf8'}), stream)

You should be able to use a stream to feed the file to you. On 'data' events you can pause the stream then split each chunk on "\n". Keep the last item in the split for the next chunk if it doesn't end in a "\n". Make sure to process the remainder when the "end" event is fired. Using bufferstream will run you out of memory, your essentially moving the file in to an array of buffers in memory. Also stream.pipe() should be used instead of util.pump(). — Ryan Olds
– Ryan Olds, Commented Jul 6, 2012 at 21:28
I would recommend using fs.createReadStream() - nodejs.org/api/fs.html#fs_fs_createreadstream_path_options — Ryan Olds
– Ryan Olds, Commented Jul 6, 2012 at 21:33
Maybe this could help: https://github.com/nickewing/line-reader — inf3rno
– inf3rno, Commented Sep 29, 2012 at 22:09

Community · Accepted Answer · 2017-05-23 12:28:47Z

2

Maybe this helps: Memory leak when using streams in Node.js?

Try to use pipe() to solve it.

edited May 23, 2017 at 12:28

CommunityBot

11 silver badge

answered Nov 29, 2013 at 20:56

Leonardo

3682 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Node.js: Processing a stream without running out of memory

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related