Memcache in node.js is returning object with varying size

Question

Long-time reader, first-time poster.

I am using node v0.6.6 on OS X 10.7. I have not yet tried this in any other environment. I am using this client: https://github.com/elbart/node-memcache

When I use the following code, data randomly contains a few more bytes (as reported by console.log()), which leads to this image: https://i.sstatic.net/MzOsw.jpg (and many other JPG do this). favicon seems OK and HTML/CSS/javascript all work.

In other words: if I request the image, ~70% of the time the image is returned correctly; the other 30% - data reports a few more bytes and the image appears corrupt in the browser.

client.get(key, function(err, data) {
    if (err) throw err;
    if (data) {
        res.writeHead(200, {'Content-Type': type, 'Content-Length': data.length});
        console.log('Sending with length: ' + data.length);
        res.end(data, 'binary');
    }
});

I have been messing with this for several hours and I can honestly say I am stumped. I am hoping someone can show me the error in my ways. I tried searching if there was a way to properly store binary data with memcache but there's no relevant information.

Extra information: it happens with various JPG images; all images are around 100-300KB or less in filesize. For example, one image has reported the following sizes: 286442, 286443, 286441. This problem DOES NOT occur if I straight read data from disk and serve it with node.

Thanks in advance.

Edit I updated my node version and issue persists. Actual test source photo and corrupt photo can be found in my comment below (stackoverflow doesn't permit more links).

can you link to the uncorrupted version of the same picture for reference? Also, try upgrading your version of node, sounds like it could be a bug in how node handles buffers? — nak
– nak, Commented Mar 28, 2012 at 5:50
Thanks for your response. I updated my node version and issue persists. Actual test source photo: imgur.com/XUTf2 ; actual test corrupt photo: imgur.com/NuaK4 — joneszach
– joneszach, Commented Mar 28, 2012 at 6:09
I ran vbindiff on the 2 files and here's where the corruption begins: i.imgur.com/gqTKW.png ... and the way its corrupt rules out my only idea. Might be a bug in memcache. I've never used it myself, I can suggest trying Redis or updating your Memcache if there's a new version. Sorry I couldn't be of more help! — nak
– nak, Commented Mar 28, 2012 at 6:23
I think there is a bug in the github.com/elbart/node-memcache library you are using. I reviewed the code and although I could not spot the issue, I can say that it is working with node Buffers as if they are strings, which does not bode well for its ability to handle non-ascii data. — Steve Campbell
– Steve Campbell, Commented Apr 3, 2012 at 21:55

Blunt Jackson · Accepted Answer · 2012-04-16 16:23:59Z

2

Elbart's node-memcache does not handle binary values correctly for the reasons Steve Campbell suggests: node-memcache does not give the client direct access to the buffer. By stringifying the buffers, binary data is corrupted.

Use the 'mc' npm. ( npm install mc )

Caveat: I'm the author of the 'mc' npm. I wrote it specifically to handle binary values over memcache's text protocol.

edited Apr 16, 2012 at 16:23

answered Apr 16, 2012 at 3:48

Blunt Jackson

6645 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

joneszach Over a year ago

Thanks. As a temporary solution to my memcache issue, I used a var cache = {} as my cache. I started storing objects in a local variable in javascript. It works perfect so far when storing ~250MB of data and 4k objects. My question: what is wrong with doing this? I don't need to access the objects anywhere else; and I won't/shouldn't hit too much memory usage by caching what I need.

Collectives™ on Stack Overflow

Memcache in node.js is returning object with varying size

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related