Use Jupyter Lab with PySpark and S3

If you're having an issue accessing S3 data from JupyterLab, then read on! Perhaps the info here, or in the linked GitHub repo, might help you discover and resolve your issue.

Before I get into any of the details, feel free to use this GitHub repo as an example of how to configure your JupyterLab PySpark notebook (running in a docker container on your local machine) to access data in S3.

I was recently wanting to use JupyterLab to view some parquet data in S3. I chose the pyspark-notebook image from the Jupyter Docker Stacks repo as a base Docker image and added jar files that would allow Spark to connect and read/write data to S3. 

It seemed like it should be pretty easy and technically it was. However, I ended up getting 403 errors when the pyspark code would try to read data from S3. The reason for the 403 error was understandable when I realized what I had done, but I didn't find any posts or documentation that illustrated my exact problem, so I figured I would share it here.

I wasn't setting the credentials provider explicitly when configuring the Spark session. I had passed in temporary credentials as environment variables when running the a script to start the Jupyter Lab container. I retrieved the credentials by using the following command:

aws --profile <your aws profile name> --region us-west-2 sts get-session-token

The default credentials provider was being used because I didn't explicitly set a credential provider to use. I assume that the access key and session key had info tying them to a session so the default credentials provider received an error when attempting to connect since no session token would have been passed to AWS. 

Here is the documentation from Hadoop where it shows the property names and values to use when configuring Spark to access data from S3. It's what I read and realized that I needed to set the credentials provider explicitly. 

The Hadoop documentation also reminded me that you don't need to explicitly set the access key, secret key, and session token in the Spark session configuration if you use the standard AWS environment variable names. It makes sense - I just had been explicitly setting the credentials previously.

You can view a GitHub repo that includes a Dockerfile, run script to start the docker container that passes the AWS credentials as environment variables, and an example JupyterLab notebook that uses pyspark to connect to S3 and download data. The bucket referenced in the example is private, so you'll need to substitute the S3 URI to a URI that points to publicly available data or use a bucket that you have permission to access and read. 

If you're interested in the data I used then you can find it here: the CDC's COVID-19 dataset.

Good luck, and have fun Sparking!

Create an AWS Lambda using Java...

Here's a quick walk through for creating an AWS lambda using Java. I happen to use IntelliJ with maven, but you can use whatever IDE and package management you prefer to use. You can find a similar walk-through in the online AWS documentation or in the AWS Lambda In Action book.

1. Create an IAM role for the Lambda to use:

• Click the "Create new role" button.
• In the "Select role type" section, Click the "Select" button for "AWS Lambda" from the "AWS Service Role" section.
• Enter the policy name of "AmazonS3FullAccess", click the check box, and click the "Next step" button.
• Enter a name in the "Role name" text box (for this example, use "hello-lambda-role"), and enter a fitting description in the "Role description" text box. Click the "Create role" button.

2. Create an S3 bucket.

3. Create a Java project for your AWS Lambda code:

• Using IntelliJ, create a maven project using maven-archetype-quickstart.
• Add the aws lambda core dependency to the project's pom file:

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-lambda-java-core</artifactId>
  <version>1.1.0</version>
</dependency>

• Create a class called HelloWorldLambda that implements RequestHandler<String, String>:

public class HelloWorldLambda implements RequestHandler<String, String> {
@Override
public String handleRequest(String input, Context context) {
    String output = "Hello, " + input + "!";
    return output;
}
}

• Build the project so that the jar is created setting the output jar name to be HelloLambda.jar.

4. Create the lambda in the AWS console:

• Click on the "Get Started Now" button.
• Click on the "Blank Function" item.

• On the "Configure triggers" page, click in the grey dashed square and then select "S3".

• Select the bucket that you created in step 2.
• Select the event type "Object Created (All)".
• Click "Enable trigger".

• Click the "Next" button. 
• Enter a name for the lambda like "hello-lambda"
• Select "Java 8" for the Runtime

• Click on the "Upload" button and select your HelloLambda.jar.

• In the "Lambda function handler and role", enter the full package path to your HelloWorldLambda class.
• Select "Choose an existing role" for the Role section.
• Select the "hello-lambda-role" that you created in step 1.

• In the "Tags" section, enter the value "Name" for the key, and "hello-lambda" for the value.

• In the "Advanced settings", increase the memory to 512 MB. Leave the timeout at 15 seconds.

• Click the "Create function" button.

5. Test the lambda!

* Go to "Functions" section of the AWS console's Lambda page.

* Select the "hello-lambda" function by clicking the option button.

* Click on the "Actions" drop down, and click on "Test function". The "Input test event" dialg will appear.

* Enter the text "testing", and then click the "Save and test" button.

This will trigger the lambda function, and you'll see the output in the "Execution result" section.

6. Test the lambda with an S3 creation event:

Uploading a text file with a single line of text to your S3 bucket that you created in step 2 will trigger your lambda, and you can see that the lambda is invoked by using the following steps.

• Go to the AWS Lambda console page, and select the "Functions" section.
• Click on the "hello-lambda" function. This should take you to the details for your lambda.  
• Click on the "Monitoring" tab. 

You'll see that you have invocations for both the test run, and the S3 upload. My image shows invocations for multiple file uploads, and multiple tests.

Learn more about AWS Lambdas through AWS Lambda In Action.

AWS S3 - Example of searching files in S3 using regex and the ResponseStream

There have been times when I've needed to inspect contents of text files that were created as map reduce output and stored in S3. I had been downloading the files, but there were hundreds of files and they were all very big (around 360 MB each). It was a hassle since it would take a long time to download every file, and it wasted a lot of diskspace. I wanted a way to search for certain data, and then cancel my search so I could stop downloading so much data.  
The solution I chose to use was to use a Regex against the ResponseStream available when you do a GetObject call. That way I'm downloading data, but it isn't being stored on my computer.

Here is the main bit of code for searching the objects contents:

private void SearchObjectForString(AmazonS3 amazonS3, string bucketName, string key, string searchString)
{
    Cursor.Current = Cursors.WaitCursor;

    // Issue call
    var request = new GetObjectRequest();
    request.BucketName = bucketName;
    request.Key = key;

    using (var response = amazonS3.GetObject(request))
    {
        using (var reader = new StreamReader(response.ResponseStream))
        {
            string line;
            var rgx = new Regex(searchString, RegexOptions.IgnoreCase);

            while ((line = reader.ReadLine()) != null)
            {
                Application.DoEvents();
                if (cancelled)
                {
                    Cursor.Current = Cursors.Default;
                    return;
                }

                var matches = rgx.Matches(line);
                if (matches.Count > 0)
                {
                    lstResults.Items.Add(string.Format("{0}/{1}:{2}", request.BucketName, request.Key, line));
                }
            }
        }
    }

    Cursor.Current = Cursors.Default;
}

AWS S3 Sample Using TransferUtility.DownloadDirectory...

I need to write a utility for a project that I'm working on, and the utility will need to download all of the files in an S3 directory. Luckily the AWSSDK provides an easy way to do this with the TransferUtility.DownloadDirectory method.

The following is a simple example usage of the DownloadDirectory method.  

public class S3Downloader
{

   public void DownloadS3Directory(string bucketName, string s3Directory, 
                                   string localDirectory)
   {
      var s3Config = new AmazonS3Config
      {
         ServiceURL = "s3-us-west-2.amazonaws.com",
         CommunicationProtocol = Protocol.HTTP
      };

      using (var s3Client = new AmazonS3Client(
                                   new EnvironmentAWSCredentials(), 
                                   s3Config))
      {
         using (var transferUtility = new TransferUtility(s3Client))
         {
            var ddr = new TransferUtilityDownloadDirectoryRequest
            {
               BucketName = bucketName,
               LocalDirectory = localDirectory,
               S3Directory = s3Directory
            };

            ddr.DownloadedDirectoryProgressEvent += DisplayProgress;
            transferUtility.DownloadDirectory(ddr);                
         }
      }
   }

   private void DisplayProgress(object sender, 
                                DownloadDirectoryProgressArgs args)
   {
      Console.WriteLine(args);
   }
}

public class Program
{
   public static void Main(string[] args)
   {
      string bucketName = "mybucket";
      string s3Directory = "/archived/files/2013-07";
      string localDirectory = @"C:\Temp\s3test";

      var s3Downloader = new S3Downloader();
      s3Downloader.DownloadS3Directory(bucketName, 
                                       s3Directory, 
                                       localDirectory);
   }
}

Here is an example of what is written to the console:

Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 8192
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 16384
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 24576
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 32768
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 40960
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 49152

Using Amazon's AWS S3 via the AWS .Net SDK

Amazon's AWS S3 (Simple Storage Service) is incredibly easy to use via the AWS .Net SDK, but depending on your usage of S3 you might have to pay. S3 has a free usage tier option, but the amount of space allowed for use is pretty small by today's standards (5GB). The upside is that even if you end up going outside of the parameters for the free usage tier it is still cheap to use.

Here is some information from Amazon regarding the free usage tier limits for S3:

• 5 GB of Amazon S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests
• These free tiers are only available to existing AWS customers who have signed-up for Free Tier after October 20, 2010 and new AWS customers, and are available for 12 months following your AWS sign-up date. When your free usage expires or if your application use exceeds the free usage tiers, you simply pay standard, pay-as-you-go service rates (see each service page for full pricing details). Restrictions apply; see offer terms for more details.

Sign Up To Use AWS

You need to create an account in order to use the Amazon Web Services. Make sure you read the pricing for any service you use so you don't end up with surprise charges. In any case, go to http://aws.amazon.com/ to sign up for an account if you haven't done so already.

Install or Reference the AWS .Net SDK

To start using the AWS .Net SDK to access S3 you will want to either download the SDK from Amazon or use NuGet via Visual Studio. Start Visual Studio (this example is using Visual Studio 2010), and do the following to use NuGet to fetch the AWS SDK:

• Select the menu item "Tools | Library Package Manager | Manage NuGet Packages For Solution..."
• Type "AWS" in the "Search Online" search text box
• Select "AWS SDK for .Net" and click the "Install" button
• Click "OK" on the "Select Projects" dialog

Create a Project and Use the AWS S3 API

Create a project in Visual Studio, and add the following code:

string key = "theawskeythatyougetwhenyousignuptousetheapis";
string secretKey = "thesecretkeyyougetwhenyousignuptousetheapis";

// create an instance of the S3 TransferUtility using the API key, and the secret key
var tu = new TransferUtility(key, secretKey);

// try listing any buckets you might have
var response = tu.S3Client.ListBuckets();

foreach(var bucket in response.Buckets)
{
   Console.WriteLine("{0} - {1}", bucket.BucketName, bucket.CreationDate);

   // list any objects that might be in the buckets
   var objResponse = tu.S3Client.ListObjects(
      new ListObjectsRequest 
      {
         BucketName = response.Buckets[0].BucketName
      }
   );

   foreach (var s3obj in objResponse.S3Objects)
   {
      Console.WriteLine("\t{0} - {1} - {2} - {3}", s3obj.ETag, s3obj.Key, s3obj.Size, s3obj.StorageClass);
   }
}

// create a new bucket
string bucketName = Guid.NewGuid().ToString();
var bucketResponse = tu.S3Client.PutBucket(new PutBucketRequest
   {
      BucketName = bucketName
   }
);

// add something to the new bucket
tu.S3Client.PutObject(new PutObjectRequest
   {
      BucketName = bucketName,
      AutoCloseStream = true,
      Key = "codecog.png",
      FilePath = "C:\\Temp\\codecog.png"
   }
);

// now list what is in the new bucket (which should only have the one item)
var bucketObjResponse = tu.S3Client.ListObjects(
   new ListObjectsRequest
   {
      BucketName = bucketName
   }
);

foreach (var s3obj in bucketObjResponse.S3Objects)
{
   Console.WriteLine("{0} - {1} - {2} - {3}", s3obj.ETag, s3obj.Key, s3obj.Size, s3obj.StorageClass);
}