Sunday, August 6, 2017

Create an AWS Lambda using Java...

Here's a quick walk through for creating an AWS lambda using Java. I happen to use IntelliJ with maven, but you can use whatever IDE and package management you prefer to use. You can find a similar walk-through in the online AWS documentation or in the AWS Lambda In Action book.

1. Create an IAM role for the Lambda to use:
  • Click the "Create new role" button.
  • In the "Select role type" section, Click the "Select" button for "AWS Lambda" from the "AWS Service Role" section.
  • Enter the policy name of "AmazonS3FullAccess", click the check box, and click the "Next step" button.
  • Enter a name in the "Role name" text box (for this example, use "hello-lambda-role"), and enter a fitting description in the "Role description" text box. Click the "Create role" button.

2. Create an S3 bucket.

3. Create a Java project for your AWS Lambda code:
  • Using IntelliJ, create a maven project using maven-archetype-quickstart.
  • Add the aws lambda core dependency to the project's pom file:

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-lambda-java-core</artifactId>
  <version>1.1.0</version>
</dependency>
  • Create a class called HelloWorldLambda that implements RequestHandler<String, String>:

public class HelloWorldLambda implements RequestHandler<String, String> {
@Override
public String handleRequest(String input, Context context) {
    String output = "Hello, " + input + "!";
    return output;
}
}
  • Build the project so that the jar is created setting the output jar name to be HelloLambda.jar.


4. Create the lambda in the AWS console:
  • Click on the "Get Started Now" button.
  • Click on the "Blank Function" item.

  • On the "Configure triggers" page, click in the grey dashed square and then select "S3".
    • Select the bucket that you created in step 2.
    • Select the event type "Object Created (All)".
    • Click "Enable trigger".
  • Click the "Next" button. 
  • Enter a name for the lambda like "hello-lambda"
  • Select "Java 8" for the Runtime

  • Click on the "Upload" button and select your HelloLambda.jar.

  • In the "Lambda function handler and role", enter the full package path to your HelloWorldLambda class.
  • Select "Choose an existing role" for the Role section.
  • Select the "hello-lambda-role" that you created in step 1.

  • In the "Tags" section, enter the value "Name" for the key, and "hello-lambda" for the value.

  • In the "Advanced settings", increase the memory to 512 MB. Leave the timeout at 15 seconds.

  • Click the "Create function" button.



5. Test the lambda!

* Go to "Functions" section of the AWS console's Lambda page.
* Select the "hello-lambda" function by clicking the option button.
* Click on the "Actions" drop down, and click on "Test function". The "Input test event" dialg will appear.
* Enter the text "testing", and then click the "Save and test" button.

This will trigger the lambda function, and you'll see the output in the "Execution result" section.


6. Test the lambda with an S3 creation event:

Uploading a text file with a single line of text to your S3 bucket that you created in step 2 will trigger your lambda, and you can see that the lambda is invoked by using the following steps.

  • Go to the AWS Lambda console page, and select the "Functions" section.
  • Click on the "hello-lambda" function. This should take you to the details for your lambda.  
  • Click on the "Monitoring" tab. 


You'll see that you have invocations for both the test run, and the S3 upload. My image shows invocations for multiple file uploads, and multiple tests.


Learn more about AWS Lambdas through AWS Lambda In Action.





Wednesday, July 26, 2017

AWS IAM Users and MFA

AWS Identity and Access Management (IAM) Users and Multi-Factor Authentication (MFA)

Amazon Web Services are easy and incredibly fun to use. Need to spin up a web server and Redis cluster? No problem! But how do you protect the AWS account from unauthorized use? Well, IAM users and MFA of course!

The AWS Certified Solutions Architect exam guide covers IAM users and groups, as well as enabling MFA for your IAM user accounts, in Chapter 6.

The exercises at the end of the chapter have you create an IAM group, an IAM user, and then enable MFA for your newly created IAM user (in exercise 6.6). I've really enjoyed going through the exam guide specifically due to the chapter review quizzes (answers with explanations are in the back of the book) and the exercises. 

Here are the steps that I used for creating an IAM group and user (using exercises 6.1 and 6.3 as the motivator, and following along in the very easy to use AWS console interface).

Creating an IAM Group:
  • Go to the IAM service in the AWS console.
  • Click the "Groups" console item.
  • Click the "Create New Group" button to start the group creation wizard.
  • Enter your group name in the "Group Name' text box and then click "Next Step". I chose "Administrators" as the AWS exam guide suggested.
  • In the Attach Policy step, the exam book tells you to click the "IAMFullAccess" policy check box. The "IAMFullAccess" policy gives the group members full access to IAM via the AWS Management Console. The AWS online documentation for creating your first user and group has you select the "AdministratorAccess" policy - which will give you full access to AWS services and resources. I chose the "AdministratorAccess" policy.
  • The last step is to review your proposed settings. Click the "Create Group" button. You'll be returned to the "Groups" list view, and you'll see your new group.

Creating an IAM User:
  • Go to the IAM service in the AWS console.
  • Click the "Users" console item.
  • Click the "Add user" button to start the user creation wizard.
  • Enter a user name in the "User name" text box.
  • In the "Select AWS access type" section, click  the "AWS Management Console access" check box. This will cause the "Console password" options to appear.
  • Select the "Custom password" option, and enter a password. 
  • The "Require password reset" check box is checked by default. If you are creating a user for someone else to use, then it is a good idea to keep this option checked.
  • Click the "Next: Permissions" button.
  • On the "Permissions" step of the wizard, click the "Add user to group" image if it is not already highlighted (this is the default selection).
  • Check the checkbox for the group you created above.
  • Click the "Next: Review" button.
  • Click the "Create user" button. You'll be taken to "Success" page where you can see the user listed. It will contain a signin link that includes your AWS user ID as part of the url.  ie, https://123456789012.signin.aws.amazon.com/console.  You'll also be able to download the user credentials via a download button. The success page mentions that you can create new credentials at any time. The credentials file lists the user name and the signin link. 

Enable MFA for an IAM user:
  • Go to the IAM service in the AWS console.
  • Click the "Users" console item.
  • Click on the user name for the user you would like to enable MFA.
  • Click on the "Security credentials" tab.
  • Click on the edit icon for "Assigned MFA device".
  • Choose "A virtual MFA device" in the "Manage MFA Device" pop up dialog, and then click the "Next Step" button.
  • You're instructed to install an AWS MFA-compatible application on the device of your choice - PC, smartphone, etc. There is a link in the dialog that will take you to a list of MFA-compatible applications.  Install one of the compatible applications. I used the smart phone option, and installed the Google Authenticator application.
  • Click the "Next Step" button. 
  • A QR code is displayed in the AWS "Manage MFA Device" pop up dialog, and you are instructed to use your smart phone to scan the code.  
  • If you're using the Google Authenticator, then a 6 digit code is displayed on your device, and is refreshed every 30 seconds.
  • You're instructed to enter two sets of the 6 digit codes, and then told to click "Activate Virtual MFA"
At this point the user account is configured for MFA. The next time that user logs in they will be prompted to enter a 6 digit MFA code. Your MFA enabled user account is now a lot more secure than it was. 

I highly recommend the exam guide even though it is starting to get a bit dated. The book gives you a condensed and comprehensive look - and the exercises really help drive home the material. I found that some of the exercises were a bit sparse in information, and no longer match what the AWS console shows you, but it is close enough that you can figure things out without getting lost. 

The experience was very fun, and the end result is that I now have a much more secure admin account!



Thursday, May 18, 2017

Refactoring Java code using lambdas - Stream.filter

Refactoring Java Code Using Lambdas

I’ve been going through a book, and absolutely loving it - Java 8 in Action: Lambdas, Streams, and functional-style programming.

The thing that I’ve enjoyed the most so far are the notes on when to use certain methods. In the chapter on streams, chapter 3, there is one particular method that really stuck out for me - the filter method. 

The filter method takes a predicate, and returns a stream of all the elements that match the predicate.

The book points out the following:

“Any time you’re looping over some data and checking each element, you might want to think about using the new filter method on Stream.”

Here is an example - the following code will print out even numbers:

List<Integer> numbers = Arrays.asList(1,2,3,4,5,6,7,8);

for (int num : numbers) {
    if (num % 2 == 0) {
       System.out.println(num);
    }
 }

The code above is pretty straight forward, but it could be written like this instead:

List<Integer> numbers = Arrays.asList(1,2,3,4,5,6,7,8);

numbers.stream()
   .filter(n -> n % 2 == 0)
   .forEach(System.out::println);

This might not look like a huge advantage, because there isn’t a lot different between the two. The amount of code is basically the same as well.  However, the benefit of the lambda version is that it can be chained together with other Stream methods.

For example, imagine that you have some data containing user IDs, and some user IDs have a special prefix to indicate a special user type.  You might want to filter out the special users, and then return a list of users without the prefix and with the name in upper case letters.

List<String> userIds = Arrays.asList(“*alice”,”bob”,”*", "charlie","*dana","evelyn","*frank");

return userIds.stream()
   .filter(u -> u.startsWith(“*”) && u.length() > 1)
   .map(u -> u.substring(1).toUpperCase())
   .collect(Collectors.toList());

To do the same thing without using streams would look something like this:

List<String> userIds = Arrays.asList("*alice","bob","", "*", "charlie","*dana","evelyn","*frank");

List<String> specialUsers = new ArrayList<>();

for (String user : userIds) {
   if (user.startsWith("*") && user.length() > 1) {
      specialUsers.add(user.substring(1).toUpperCase());
   }
}

return specialUsers;

You can see that it would require that another variable would have to be declared to hold the special users. Kind of a waste. 


I’ll definitely be keeping my eyes open for code that is iterating over lists and inspecting each element!



Monday, March 7, 2016

Cleaning up after aws cli on Mac OSX...

I've installed the aws command line on my Mac. It's super handy. However, the aws s3 command creates $folder$ files for every "directory" when a recursive copy is performed. It's super annoying. 

For example, you could have a "directory" in S3 named "myfiles". When you download the objects with "myfiles" in the path you will end up with a file named "myfiles_$folder$".

Running aws --version returns this info:

    aws-cli/1.10.6 Python/2.7.10 Darwin/14.5.0 botocore/1.3.28


I haven't found anything that explains how I can prevent those files from being created, so I've been doing manual cleanup afterwards.  This is the command I run:

    > rm $(find . "*$folder$")



Tuesday, January 12, 2016

Debugging a local Spark job using IntelliJ

A coworker was working on a local Spark job and shared how he set up his environment for debugging the job (which is basically the same as debugging any other remote process). These are the instructions I followed:

1. Create a remote debug configurations.


Go to IntelliJ's "Run | Edit Configurations" screen
Click on the "+" to "Add New Configuration"
Select "Remote"


2. Copy the command line argument to use and modify it however you see fit.


I'm using Java 8, so I used the example command line arguments from the top edit box. The only change I made was to set "suspend=y" so the spark job would stop and wait for me to start my "Remote Debug" process.

This is what I used: 
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

3. Export the command line arg as SPARK_JAVA_OPTS (Spark uses this value when you submit a spark job).

I set the SPARK_JAVA_OPTS like this:

export SPARK_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

4. Start the spark job.

You should see your spark job start up, and then pause with the following line printed on the console:

Listening for transport dt_scoket at address: 5005

5. In IntelliJ, create whatever breakpoints you want to use and start the remote debug configuration. 


Thursday, August 27, 2015

Simple gremlin queries in Titan...

Titan is an open source graph database and, even though it isn't as easy to setup for use as Neo4J, it is easy enough to start using in just a few minutes. Here is an example of using Titan with HBase as the backend storage.

Setup

I'm using HBase 0.98.13. I downloaded hbase, untarred the files, changed to the hbase directory and ran "bin/start-hbase.sh".

I cloned Titan from the github repository and built it using the maven package command.

git clone https://github.com/thinkaurelius/titan.git
cd titan
mvn clean package

I started gremlin using "bin/gremlin.sh".

I followed the Titan Hbase instructions for initializing Titan for use with HBase within the gremlin console. You can also create a properties file that contains the same settings and load the settings from the gremlin console.

Gremlin

conf = new BaseConfiguration();
conf.setProperty("storage.backend","hbase");
conf.setProperty("storage.hbase.table", "test") 

g = TitanFactory.open(conf);

Now let's add some vertices.

alice = g.addVertexWithLabel('human')
alice.setProperty('name', 'alice')
alice.setProperty('age',25)
bob = g.addVertexWithLabel('human')
bob.setProperty('name', 'bob')
bob.setProperty('age',21)
clark = g.addVertexWithLabel('human')
clark.setProperty('name', 'clark')
clark.setProperty('age',93)
darwin = g.addVertexWithLabel('human')
darwin.setProperty('name', 'darwin')
darwin.setProperty('age',206)
ernie = g.addVertexWithLabel('android')
ernie.setProperty('name', 'ernie')


Let's list the vertices and their properties.

g.V().map()
==>{name=ernie}
==>{name=alice, age=25}
==>{name=darwin, age=206}
==>{name=clark, age=93}
==>{name=bob, age=21}

And now let's make these humans be friends with each other.

alice.addEdge('friend', bob)
alice.addEdge('friend', darwin)
bob.addEdge('friend', alice)
bob.addEdge('friend', darwin)
clark.addEdge('friend', darwin)
darwin.addEdge('friend',alice)
darwin.addEdge('friend', bob)
darwin.addEdge('friend', clark)



Now let's remove ernie from the graph.

g.V.has('name', 'ernie').remove()
==>null

Now we can see that ernie is gone

g.V.has('name', 'ernie').map()

(no results displayed, just the gremlin prompt)

Let's add ernie back, but this time he's a human.

ernie = g.addVertexWithLabel('human')
ernie.setProperty('name', 'ernie')



Let's try finding out who has friends

g.V().outE('friend').outV().name
==>darwin
==>darwin
==>darwin
==>alice
==>alice
==>bob
==>bob
==>clark


Wait - what happened? We see an entry for every friend edge, which is exactly what our gremlin query was asking for, but that doesn't look very nice.

Let's try the dedup method.

g.V().outE('friend').outV().dedup().name
==>darwin
==>alice
==>bob
==>clark


Ahh! That's more like it! But how else can we get that list?

g.V.filter{it.outE('friend').hasNext()}.toList()._().name
==>darwin
==>alice
==>bob
==>clark


Nice! We have two ways to get a distinct list.