Tuesday, July 16, 2013

AWS S3 Sample Using TransferUtility.DownloadDirectory...

I need to write a utility for a project that I'm working on, and the utility will need to download all of the files in an S3 directory. Luckily the AWSSDK provides an easy way to do this with the TransferUtility.DownloadDirectory method.

The following is a simple example usage of the DownloadDirectory method.  


public class S3Downloader
{

   public void DownloadS3Directory(string bucketName, string s3Directory, 
                                   string localDirectory)
   {
      var s3Config = new AmazonS3Config
      {
         ServiceURL = "s3-us-west-2.amazonaws.com",
         CommunicationProtocol = Protocol.HTTP
      };

      using (var s3Client = new AmazonS3Client(
                                   new EnvironmentAWSCredentials(), 
                                   s3Config))
      {
         using (var transferUtility = new TransferUtility(s3Client))
         {
            var ddr = new TransferUtilityDownloadDirectoryRequest
            {
               BucketName = bucketName,
               LocalDirectory = localDirectory,
               S3Directory = s3Directory
            };

            ddr.DownloadedDirectoryProgressEvent += DisplayProgress;
            transferUtility.DownloadDirectory(ddr);                
         }
      }
   }

   private void DisplayProgress(object sender, 
                                DownloadDirectoryProgressArgs args)
   {
      Console.WriteLine(args);
   }
}

public class Program
{
   public static void Main(string[] args)
   {
      string bucketName = "mybucket";
      string s3Directory = "/archived/files/2013-07";
      string localDirectory = @"C:\Temp\s3test";

      var s3Downloader = new S3Downloader();
      s3Downloader.DownloadS3Directory(bucketName, 
                                       s3Directory, 
                                       localDirectory);
   }
}

Here is an example of what is written to the console:

Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 8192
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 16384
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 24576
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 32768
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 40960
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 49152

2 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. I was trying to upload, but I am getting error:
    Data at root level is invalid.
    Please help!! I am stuck since 1 week

    ReplyDelete