How to store a binary object in redis using node?

Question

I am trying to save a binary object in redis and then serve it back as an image.

Here is the code I am using to save the data:

var buff=new Buffer(data.data,'base64');
client.set(key,new Buffer(data.data,'base64'));

Here is the code to dump the data out:

client.get(key,function(err,reply){
        var data = reply;
        response.writeHead(200, {"Content-Type": "image/png"});
        response.end(data,'binary');

});

The first few byte of the data seem to be corrupted. The magic number is incorrect.

Did some experimenting:

when I do the following:

var buff=new Buffer(data.data,'base64');
console.log(buff.toString('binary'));

I get this:

0000000: c289 504e 470d 0a1a 0a00 0000 0d49 4844

when I do this

 var buff=new Buffer(data.data,'base64');
 console.log(buff);

I get the following:

Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00

I am not sure where the c2 is coming from

haven't worked with images this way, but isn't supposed to "decode" what was encoded in base64? — Gntem
– Gntem, Commented Dec 22, 2013 at 21:34
What do you get if you console.log(reply)? Can you also elaborate on what you are trying accomplish by putting the images in Redis? It is not the best way to store large amounts of binary data. — loganfsmyth
– loganfsmyth, Commented Dec 22, 2013 at 23:25
Follow up to this question: I'm trying to do basically the same thing, and when I read it out with get, the string length is different even when I'm writing the same thing. — Justin Thomas
– Justin Thomas, Commented Apr 2, 2015 at 13:52

mb. · Accepted Answer · 2014-05-31 01:21:30Z

28

The problem is that the Redis client for Node converts responses to JavaScript strings by default.

I solved this by setting the return_buffers option to true when creating the client.

var client = redis.createClient(7000, '127.0.0.1', {'return_buffers': true});

See here for more details.

answered May 31, 2014 at 1:21

mb.

1,3031 gold badge12 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

grahas · Accepted Answer · 2022-12-11 17:54:53Z

7

I ran in to a similar issue and in my case the corruption was with how I was getting data from Redis into Nodejs. In my case the value from Nodejs was stored correctly but when it came into node it was being converted to a ASCII string. I was able to force it to a buffer with the following line.

let file = await r.get(redis.commandOptions({ returnBuffers: true }), `zip/${req.params.taskId}`)

answered Dec 11, 2022 at 17:54

grahas

1051 silver badge5 bronze badges

Comments

majidarif · Accepted Answer · 2015-02-06 20:58:10Z

6

The problem with return_buffers is when you are not using pure buffer data's then you'll have to do something to convert other buffers to strings. While detect_buffers could be an option it is too unreliable.

If you don't mind the extra computing cycles. You could also try:

// convert your buffer to hex string
client.set(key, mybuffer.toString('hex'));
// create a new buffer from hex string
client.get(key, function (err, val) {
   var bin = new Buffer(val, 'hex');
});

answered Feb 6, 2015 at 20:58

majidarif

20.3k18 gold badges97 silver badges137 bronze badges

1 Comment

Gajus Over a year ago

In most cases, base64 is a better option since it will produce a smaller message than hex (at a cost of more expensive encoding).

Mauvis Ledford · Accepted Answer · 2021-06-26 07:39:17Z

3

This post is old but just an updated answer. As another answer stated all Node.js Redis libraries return data as utf8 strings because thats what people generally need. Node.js Redis libraries like ioRedis also provide alternatives to fetch data as a buffer. For example redis.getBuffer instead of redis.get.

answered Jun 26, 2021 at 7:39

Mauvis Ledford

42.6k17 gold badges83 silver badges86 bronze badges

Comments

SamFisher83 · Accepted Answer · 2013-12-23 16:40:50Z

2

I was unable to figure out how to get the binary string to store.

Here is my workaround:

Where data is the data in base64 string

client.set(count,data);

to serve the data:

 client.get(last,function(err,reply){
   var data = reply;
   response.writeHead(200, {"Content-Type": "image/png"});
   var buff=new Buffer(data,'base64');
   response.end(buff);
});

This isn't ideal since you have to do the conversion every time, but it seem to work.

answered Dec 23, 2013 at 16:40

SamFisher83

4,0059 gold badges42 silver badges54 bronze badges

1 Comment

Yury Kozlov Over a year ago

Please note, that new Buffer() is deprecated. Use: Buffer.from('some-data', 'base64');

Climax · Accepted Answer · 2017-04-03 05:11:13Z

0

What worked for me is to use data.toString('binary') if it is a Buffer. Also make sure not to reinterpret it as utf-8, but also as binary.

answered Apr 3, 2017 at 5:11

Climax

6737 silver badges17 bronze badges

Comments

Daniel Knowlton · Accepted Answer · 2019-12-18 17:15:32Z

0

I found this article on the topic which explains the implications of doing this, which I highly recommend reading before going down this path (regardless of programming language).

http://qnimate.com/storing-binary-data-in-redis/

In summary, since Redis is an in-memory key/value store, you would be wise to not store potentially large values such as images inside Redis as you will quickly use up available memory and degrade the performance of your Redis instance. It is better to store the location of the file in Redis rather than the file itself.

answered Dec 18, 2019 at 17:15

Daniel Knowlton

1714 bronze badges

Collectives™ on Stack Overflow

How to store a binary object in redis using node?

7 Answers 7

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related