NodeJS AWS Lambda ListObject returning empty array

Question

I am new to AWS. I am creating a Scheduled lambda function using NodeJS. There I need to get all the objects in a folder and iterate through each, to check the last modified date and I want to delete old objects. But I am getting empty array in ListObject's Response Contents.

var s3 = new aws.S3();
exports.handler = (event, context, callback) => {

var params = {
    Bucket: 'bucket_name',
    Delimiter: '/',
    Prefix: 'folder_name',
    MaxKeys:100000
};
s3.listObjects(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else
  {     
    var folderLength = data.Contents.length;
    data.Contents.forEach(function(metadata) {
      console.log('Key: ' + metadata.Key);
    });

  }
});
callback(null, 'Finished');
};

But in the response i am getting Contents: []. Can anyone tell me whats wrong?

Are you sure you are getting an empty array? To me, it seems you are terminating the lambda before any result comes back. The callback function call should be withing the listObject result function. — jens walter
– jens walter, Commented Mar 12, 2017 at 21:34
The pseudo-folders in S3 should have a hidden trailing slash. Please verify whether adding a trailing slash to the folder name fixes your issue, and I'll provide more explanation in an answer. Try Prefix: 'folder_name/',. — Michael - sqlbot
– Michael - sqlbot, Commented Mar 12, 2017 at 21:37
Thanks @Michael-sqlbot Prefix: 'folder_name/' was the issue. If u post it as answer i can accept it — Salini L
– Salini L, Commented Mar 13, 2017 at 4:56

Michael - sqlbot · Accepted Answer · 2017-03-13 07:07:21Z

When you call ListObjects() with Delimiter: '/', there are two arrays of interest inside data: Contents and CommonPrefixes.

Contents is the list of objects beginning with the Prefix value you specified up to but not including the next Delimiter, and CommonPrefixes is a list of the prefixes common to all the objects with the prefix of Prefix, up to and including the next occurrence of Delimiter (which is / by convention).

S3 doesn't actually have folders -- just objects that share common prefixes, delimited with /.

When you create a folder foo in the S3 console, an empty object with a key of foo/ is actually created. However, if you upload an object with key foo/bar.html using the API, the console will -- for convenience -- still display this as a file bar.html inside a folder named foo, even though in this latter case, no foo/ object was explicitly created. Significantly, in neither case is the object actually "in" a meaningful "folder" container. It's only presented that way. S3 has no actual folders -- only objects and buckets.

If you want all the objects "in a folder," what you're really saying is that you want the objects sharing a common key prefix.

But when S3 provides a list of all objects sharing the common key prefix "folder_name" up to the next / (as noted above) then there aren't any.

In your original code, Contents is empty, but you would have found that data.CommonPrefixes in the response contained a single value, folder_name/, because all of the objects whose keys begin with folder_name up to the next / actually share the common prefix folder_name/.

There's no point in making two requests with the first one being redundant and unnecessary (and, in fact, doing so will give you incorrect results if more than one folder has a name beginning with "folder_name") so when you want a list of objects "in a folder," the correct approach is to set the Prefix to folder_name + '/'.

Collectives™ on Stack Overflow

NodeJS AWS Lambda ListObject returning empty array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest