PHP: Syncing Data with Amazon S3

Question

While syncing directories with the s3client PHP SDK works perfectly

$s3client->uploadDirectory('/local/directory', 'my-bucket');

I still need the s3cmd --delete-removed feature to remove the unmatched files that exist in the bucket and not in the source directory.

ref:http://blogs.aws.amazon.com/php/post/Tx2W9JAA7RXVOXA/Syncing-Data-with-Amazon-S3

Why not first empty up your bucket & than sync the data ? In that way it will automatically maintain consistency as you require. — Rikesh
– Rikesh, Commented Apr 2, 2014 at 9:36
@Rikesh because there are users/bots connected to the bucket and deleting a directory will produce a lot of errors — Gadelkareem
– Gadelkareem, Commented Apr 2, 2014 at 13:08
Ok I got it I guess nothing such provided currently by PHP S3 SDK but I guess you need to do some work around. This blog might helps you. — Rikesh
– Rikesh, Commented Apr 2, 2014 at 14:04

Lance Rushing · Accepted Answer · 2016-03-09 06:03:28Z

make a list of existing files and their md5sum values
Get all of the s3 Objects
Compare md5s and transfer if different

remove any 'left over' (--delete-removed) files

function syncS3($bucket, $prefix, $targetDir)
{
    if (!is_dir($targetDir)) {
        mkdir($targetDir, 0755, true);
    }

    $localFileM5Sums = getMd5ForFilesInDir($targetDir);
    $s3Objects       = $this->s3Client->listObjects(['Bucket' => $bucket, 'Prefix' => $prefix]);

    foreach ($s3Objects['Contents'] as $contents) {
        $s3Key = $contents['Key'];
        if (substr($s3Key, -1, 1) == '/') { // is a 'directory'
            continue; // skip
        }

        $saveAs = $targetDir . DIRECTORY_SEPARATOR . $s3Key;
        if (isset($localFileM5Sums[$saveAs])) { // if we have local copy
            $localMd5 = $localFileM5Sums[$saveAs];
            unset($localFileM5Sums[$saveAs]); // remove so we don't try to delete it later
            if ($localMd5 === trim($contents['ETag'], '"')) { // and md5sumMatches
                continue; // don't transfer
            }
        }

        // transfer it
        if (!is_dir(dirname($saveAs))) {
            mkdir(dirname($saveAs), 0755, true);
        }
        $this->s3Client->getObject([
            'Bucket' => $bucket,
            'Key' => $s3Key,
            'SaveAs' => $saveAs
        ]);
    }

    // remove any missing files
    foreach ($localFileM5Sums as $fileName => $md5sum) {
        unlink($fileName);
    }
}


function getMd5ForFilesInDir($targetDir)
{
    $iterator = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($targetDir));
    $files    = [];
    foreach ($iterator as $file) {
        if (!$file->isDir())
            $files[$file->getPathname()] = md5_file($file->getPathname());
    }
    return $files;
}

Collectives™ on Stack Overflow

PHP: Syncing Data with Amazon S3

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related