2

I need to write chat logs and for a variety of reasons would like to save the data as JSON.

At any given time I may have many rooms open and am currently opening a writeStream when the first user enters the room then ending the stream when the last user exits.

This writeStream object is temporarily stored in an array along with other details about the room and the users that are currently online. With each room object available at any given time like this, it is my understanding that it should be very quick and efficient to continuously write to many chat log files as the users in the various rooms chat.

I am quite new to node.js so please feel free to correct me and offer a more efficient method that above.

But onto my actual question.

Each write is a single javascript object containing specific userdata, the body of the chat and of course the time it occured.

I am currently creating txt files, and using JSON.stringify() to append a stringified version of each chat object to the file as it comes in.

This of course will yield invalid JSON resulting in a textual representation of the following:

{"userid":"1","body":"hello world"}{"userid":"2","body":"hello world 2"}{"userid":"3","body":"hello world 3"}

When in reality it needs to read :

[{"userid":"1","body":"hello world"},{"userid":"2","body":"hello world 2"},{"userid":"3","body":"hello world 3"}]

Now an obvious solution to this would be to actually open the file in node.js, parse the JSON, push the object into the array and then write all the data back. But this feels very expensive to both IO and processing-wise to me.

Another alternative would be to add a special character such as | to the beginning of each object. Then when reading the log files client side simply split the file contents at that character and JSON.parse each object in the resulting array. But surely there is a better way?

Therefore what I am looking for is a more efficient method that could deal with the logging of many chatrooms at once. Taking advantage of writeStreams if possible.

Ideally I really would like to to simply write data in the following fashion :

',' + chatobject + ']'

and then for the next write operation simply trim the end of the file by removing the ']' character and writing the next one. Leaving me with a constantly valid JSON string in the txt file.

Alternatively, is there a better method by actually writing to a .json file itself and avoid parsing altogether.

Thanks in advance, I apologize for my naivety with regards to node.js.

Improve this question

2 Answers 2

Reset to default
2

Edit: I realize this doesn't exactly answer the question, but because you said that you're new to NodeJS I thought you may not know that this kind of thing is an option. So I'm giving this as an answer as a better way to accomplish what you're trying to accomplish.

Why don't you use MongoDB through MongooseJS. It's probably a million times more of each of these things:

  1. Fast
  2. Easy
  3. Scalable
  4. Better

I would definitely recommend doing it this way. And while you're at it, you may as well use the entire MEAN stack.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

My main reason for thinking the file system itself would be better suited to this task is simply that I have absolutely no need to ever query the data whatsoever, with each log file sharing its name with the ID of the room. Therefore by my understanding, using a database, even a NoSQL one would be unnecessary overhead would it not? Or again, am I misunderstanding MondoDB? Regardless, I appreciate the answer, and will definitely look deeper into MongoDB. Cheers.
@gordyr I would say that even if you never need to query the data using MongoDB would still be easier. It'll take a second to learn, but that's not too difficult, especially with a library like MongooseJS. It's really really simple and if you ever do want to take this app further you'll be happier that way :)
Having looked further into MongoDB, I've decided you are absolutely right. With the biggest plus being the ease of distribution as the app scales. Thanks for the point in the right direction, I will award you the answer. If anyone else has a direct answer to the question I would still be very interested to read it, for completeness sake. Anyway, Thanks again :)
@gordyr and I would definitely recommend using MongooseJS as well. I built an app without a library for MongoDB and it was a bit of a headache. After switching to MongooseJS my life got so much easier!
Thanks... will give be sure to give it a go :)
|
2

I think you are looking for something like bunyan. It lets you do stuff like this (from the docs):

var bunyan = require('bunyan');
var log = bunyan.createLogger({name: "myapp"});
log.info("hi");

Another option is to just write regular text logs separated by line instead of writing JSON. This way you can view or analyze the logs easily using regular unix-y text processing tools.

Improve this answer

1 Comment

This is great thanks. I've decided to go the MongoDB route as mentioned above. But this is fantastic info. Thanks mate. :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.