I use AWS for work, and use terraform for creating the resources. My team uses a common directory structure for our terraform files, and it seems to work pretty well for separating resources between project groups, logical environments, and regions. However, creating new project directory structures can be a pain, so I decided to create a yeoman generator to automate the process. Please check out the generator I made, and let me know what you think!
I bought some Philips Hue Lights, and have really enjoyed them - but I enjoy them even more now that I have the IoT button integrated with the lights. Here is a video showing my AWS IoT button interacting with my Philips Hue Go lamp.
AWS IoT Button: I had seen the AWS IoT button on Amazon and, although I didn't have any ideas of what I would do with the button, I wanted to work on a project which would use one of the buttons. I found this fun project that also uses an AWS IoT button, and the Philips Hue API with the Go lamp. I had bought a Philips Hue Go light, as well as a number of other Philips Hue lights, so I decided to recreate the project from the youtube video above but using an AWS lambda instead of using a raspberry pi.
Something that was pointed out to me (embarrassingly) is that this method is not secure. Sending unencrypted information to the Hue bridge, which includes the auth, would allow an attacker to send their own API calls to the bridge. One of the API calls could have a security hole that could be used by an attacker / curious person.
A couple of ideas I've had for using the Philips Hue lights are flash lights with certain colors to indicate either a rise above, or drop below, stock or crypto currency price points, and flash lights when people are close to home (integrate with IFTTT). However, using the IoT button to control the lights looked fun and gave me an excuse to learn a little bit about AWS Lambdas. It's worth mentioning that Philips makes a switch that can be easily programmed to control your Philips Hue lights.
Set up IoT Button: I used the "Getting Started" guide to set up the IoT button. It walks you through registering your device, creating and activating a device certificate, creating and attaching an IoT policy to the device certificate, attach the certificate to a "Thing" (the button), and configuring your IoT button to know how to connect to your WiFi.
One of the last steps in the "Getting Started" guide is configuring and testing rules. The example has the IoT button pushes send an SNS message that gets sent as a text message to your phone. I decided to have the SNS message trigger a lambda, and use the lambda to send the REST calls to my Philips Hue bridge. AWS Lambda: Here is the AWS Lambda code that I used:
I have my router configured to use Dynamic DNS, and then I have a port forwarding rule to forward to the Philips Hue bridge. The lambda figures out if the button click was a single click, a double click, or a long click. The double clicks will turn the light on and off, the single click will increment the hue to set the light to, and a long click will set the light to use the color loop effect.
I hope you find this post useful! Please leave links to any projects you feel like sharing using AWS IoT buttons and/or Philips Hue lights in the comment section below.
Here's a quick walk through for creating an AWS lambda using Java. I happen to use IntelliJ with maven, but you can use whatever IDE and package management you prefer to use. You can find a similar walk-through in the online AWS documentation or in the AWS Lambda In Action book.
1. Create an IAM role for the Lambda to use:
Click the "Create new role" button.
In the "Select role type" section, Click the "Select" button for "AWS Lambda" from the "AWS Service Role" section.
Enter the policy name of "AmazonS3FullAccess", click the check box, and click the "Next step" button.
Enter a name in the "Role name" text box (for this example, use "hello-lambda-role"), and enter a fitting description in the "Role description" text box. Click the "Create role" button.
2. Create an S3 bucket.
3. Create a Java project for your AWS Lambda code:
Using IntelliJ, create a maven project using maven-archetype-quickstart.
Add the aws lambda core dependency to the project's pom file:
Create a class called HelloWorldLambda that implements RequestHandler<String, String>:
public class HelloWorldLambda implements RequestHandler<String, String> { @Override public String handleRequest(String input, Context context) { String output = "Hello, " + input + "!"; return output; } }
Build the project so that the jar is created setting the output jar name to be HelloLambda.jar.
4. Create the lambda in the AWS console:
Click on the "Get Started Now" button.
Click on the "Blank Function" item.
On the "Configure triggers" page, click in the grey dashed square and then select "S3".
Select the bucket that you created in step 2.
Select the event type "Object Created (All)".
Click "Enable trigger".
Click the "Next" button.
Enter a name for the lambda like "hello-lambda"
Select "Java 8" for the Runtime
Click on the "Upload" button and select your HelloLambda.jar.
In the "Lambda function handler and role", enter the full package path to your HelloWorldLambda class.
Select "Choose an existing role" for the Role section.
Select the "hello-lambda-role" that you created in step 1.
In the "Tags" section, enter the value "Name" for the key, and "hello-lambda" for the value.
In the "Advanced settings", increase the memory to 512 MB. Leave the timeout at 15 seconds.
Click the "Create function" button.
5. Test the lambda!
* Go to "Functions" section of the AWS console's Lambda page.
* Select the "hello-lambda" function by clicking the option button.
* Click on the "Actions" drop down, and click on "Test function". The "Input test event" dialg will appear.
* Enter the text "testing", and then click the "Save and test" button.
This will trigger the lambda function, and you'll see the output in the "Execution result" section.
6. Test the lambda with an S3 creation event:
Uploading a text file with a single line of text to your S3 bucket that you created in step 2 will trigger your lambda, and you can see that the lambda is invoked by using the following steps.
Go to the AWS Lambda console page, and select the "Functions" section.
Click on the "hello-lambda" function. This should take you to the details for your lambda.
Click on the "Monitoring" tab.
You'll see that you have invocations for both the test run, and the S3 upload. My image shows invocations for multiple file uploads, and multiple tests.
Learn more about AWS Lambdas through AWS Lambda In Action.
AWS Identity and Access Management (IAM) Users and Multi-Factor Authentication (MFA) Amazon Web Services are easy and incredibly fun to use. Need to spin up a web server and Redis cluster? No problem! But how do you protect the AWS account from unauthorized use? Well, IAM users and MFA of course! The AWS Certified Solutions Architect exam guide covers IAM users and groups, as well as enabling MFA for your IAM user accounts, in Chapter 6. The exercises at the end of the chapter have you create an IAM group, an IAM user, and then enable MFA for your newly created IAM user (in exercise 6.6). I've really enjoyed going through the exam guide specifically due to the chapter review quizzes (answers with explanations are in the back of the book) and the exercises. Here are the steps that I used for creating an IAM group and user (using exercises 6.1 and 6.3 as the motivator, and following along in the very easy to use AWS console interface). Creating an IAM Group:
Go to the IAM service in the AWS console.
Click the "Groups" console item.
Click the "Create New Group" button to start the group creation wizard.
Enter your group name in the "Group Name' text box and then click "Next Step". I chose "Administrators" as the AWS exam guide suggested.
In the Attach Policy step, the exam book tells you to click the "IAMFullAccess" policy check box. The "IAMFullAccess" policy gives the group members full access to IAM via the AWS Management Console. The AWS online documentation for creating your first user and group has you select the "AdministratorAccess" policy - which will give you full access to AWS services and resources. I chose the "AdministratorAccess" policy.
The last step is to review your proposed settings. Click the "Create Group" button. You'll be returned to the "Groups" list view, and you'll see your new group.
Creating an IAM User:
Go to the IAM service in the AWS console.
Click the "Users" console item.
Click the "Add user" button to start the user creation wizard.
Enter a user name in the "User name" text box.
In the "Select AWS access type" section, click the "AWS Management Console access" check box. This will cause the "Console password" options to appear.
Select the "Custom password" option, and enter a password.
The "Require password reset" check box is checked by default. If you are creating a user for someone else to use, then it is a good idea to keep this option checked.
Click the "Next: Permissions" button.
On the "Permissions" step of the wizard, click the "Add user to group" image if it is not already highlighted (this is the default selection).
Check the checkbox for the group you created above.
Click the "Next: Review" button.
Click the "Create user" button. You'll be taken to "Success" page where you can see the user listed. It will contain a signin link that includes your AWS user ID as part of the url. ie, https://123456789012.signin.aws.amazon.com/console. You'll also be able to download the user credentials via a download button. The success page mentions that you can create new credentials at any time. The credentials file lists the user name and the signin link.
Enable MFA for an IAM user:
Go to the IAM service in the AWS console.
Click the "Users" console item.
Click on the user name for the user you would like to enable MFA.
Click on the "Security credentials" tab.
Click on the edit icon for "Assigned MFA device".
Choose "A virtual MFA device" in the "Manage MFA Device" pop up dialog, and then click the "Next Step" button.
You're instructed to install an AWS MFA-compatible application on the device of your choice - PC, smartphone, etc. There is a link in the dialog that will take you to a list of MFA-compatible applications. Install one of the compatible applications. I used the smart phone option, and installed the Google Authenticator application.
Click the "Next Step" button.
A QR code is displayed in the AWS "Manage MFA Device" pop up dialog, and you are instructed to use your smart phone to scan the code.
If you're using the Google Authenticator, then a 6 digit code is displayed on your device, and is refreshed every 30 seconds.
You're instructed to enter two sets of the 6 digit codes, and then told to click "Activate Virtual MFA"
At this point the user account is configured for MFA. The next time that user logs in they will be prompted to enter a 6 digit MFA code. Your MFA enabled user account is now a lot more secure than it was. I highly recommend the exam guide even though it is starting to get a bit dated. The book gives you a condensed and comprehensive look - and the exercises really help drive home the material. I found that some of the exercises were a bit sparse in information, and no longer match what the AWS console shows you, but it is close enough that you can figure things out without getting lost. The experience was very fun, and the end result is that I now have a much more secure admin account!
I've installed the aws command line on my Mac. It's super handy. However, the aws s3 command creates $folder$ files for every "directory" when a recursive copy is performed. It's super annoying. For example, you could have a "directory" in S3 named "myfiles". When you download the objects with "myfiles" in the path you will end up with a file named "myfiles_$folder$". Running aws --version returns this info: aws-cli/1.10.6 Python/2.7.10 Darwin/14.5.0 botocore/1.3.28 I haven't found anything that explains how I can prevent those files from being created, so I've been doing manual cleanup afterwards. This is the command I run: > rm $(find . "*$folder$")
There have been times when I've needed to inspect contents of text files that were created as map reduce output and stored in S3. I had been downloading the files, but there were hundreds of files and they were all very big (around 360 MB each). It was a hassle since it would take a long time to download every file, and it wasted a lot of diskspace. I wanted a way to search for certain data, and then cancel my search so I could stop downloading so much data. The solution I chose to use was to use a Regex against the ResponseStream available when you do a GetObject call. That way I'm downloading data, but it isn't being stored on my computer. Here is the main bit of code for searching the objects contents:
private void SearchObjectForString(AmazonS3 amazonS3, string bucketName, string key, string searchString)
{
Cursor.Current = Cursors.WaitCursor;
// Issue call
var request = new GetObjectRequest();
request.BucketName = bucketName;
request.Key = key;
using (var response = amazonS3.GetObject(request))
{
using (var reader = new StreamReader(response.ResponseStream))
{
string line;
var rgx = new Regex(searchString, RegexOptions.IgnoreCase);
while ((line = reader.ReadLine()) != null)
{
Application.DoEvents();
if (cancelled)
{
Cursor.Current = Cursors.Default;
return;
}
var matches = rgx.Matches(line);
if (matches.Count > 0)
{
lstResults.Items.Add(string.Format("{0}/{1}:{2}", request.BucketName, request.Key, line));
}
}
}
}
I need to write a utility for a project that I'm working on, and the utility will need to download all of the files in an S3 directory. Luckily the AWSSDK provides an easy way to do this with the TransferUtility.DownloadDirectory method.
The following is a simple example usage of the DownloadDirectory method.
public class S3Downloader
{
public void DownloadS3Directory(string bucketName, string s3Directory,
string localDirectory)
{
var s3Config = new AmazonS3Config
{
ServiceURL = "s3-us-west-2.amazonaws.com",
CommunicationProtocol = Protocol.HTTP
};
using (var s3Client = new AmazonS3Client(
new EnvironmentAWSCredentials(),
s3Config))
{
using (var transferUtility = new TransferUtility(s3Client))
{
var ddr = new TransferUtilityDownloadDirectoryRequest
{
BucketName = bucketName,
LocalDirectory = localDirectory,
S3Directory = s3Directory
};
ddr.DownloadedDirectoryProgressEvent += DisplayProgress;
transferUtility.DownloadDirectory(ddr);
}
}
}
private void DisplayProgress(object sender,
DownloadDirectoryProgressArgs args)
{
Console.WriteLine(args);
}
}
public class Program
{
public static void Main(string[] args)
{
string bucketName = "mybucket";
string s3Directory = "/archived/files/2013-07";
string localDirectory = @"C:\Temp\s3test";
var s3Downloader = new S3Downloader();
s3Downloader.DownloadS3Directory(bucketName,
s3Directory,
localDirectory);
}
}
Here is an example of what is written to the console:
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 8192
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 16384
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 24576
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 32768
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 40960
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 49152
Updated 03-20-2017 Amazon's Simple Queue Service (SQS) provides an easy to use mechanism for sending and receiving messages between various applications/processes. Here are a few things that I learned while using the AWS Java SDK to use SQS.
SQS is notcan be FIFO
It used to be that AWS SQS didn't guarantee FIFO ordering. Now you can create a standard queue or a FIFO queue. However, there are some differences to be aware between standard and FIFO queues that are worth pointing out. The differences can be read about here. Here are some of the key differences: Standard Queues - available in all regions, nearly unlimited transactions per second, messages will be delivered at least once but might be delivered more than once, messages might be delivered out of order. FIFO Queues - available in US West (Oregon) and US East (Ohio), 300 transactions per second, messages are delivered exactly once, order of messages is preserved (as the queue type suggests). SQS Free Usage Tier
The SQS free usage tier is determined by the number of requests you make per month. You can make up to 1 million requests per month. The current fee is $.50 per million requests after the first million requests. The cost is pretty low, but it would be easy to start racking up millions of requests. Luckily, there are batch operations that can be done, and each batch operation is considered one request. Short Polling/Long Polling
You can set a time limit to wait when polling queues for messages. Short polling is when you make a request to receive messages without setting the ReceiveMessageWaitTimeSeconds property for the queue. Setting the ReceiveMessageWaitTimeSeconds property to up to 20 seconds (20 seconds is the maximum wait time) will cause your call to wait up to 20 seconds for a message to appear on the queue before returning. If there is a message on the queue, then the call will return immediately with the message. The advantage to using long polling is that you will make less requests without receiving messages. One thing to remember is that if you have only one thread being used to poll multiple queues, then you will have unnecessary wait times when only some of the queues have messages waiting. A solution to that problem is to use one thread for each queue being polled. Something that seemed a bit contradictory is that queues created through the web console have the ReceiveMessageWaitTimeSeconds set to 0 seconds (meaning it is going to use short polling). However, the FAQ mentions that the AWS SDK uses 20 second wait times by default. I created a queue using the AWS SDK, and the wait time was listed as 0 seconds in the web console. I shouldn't have to specifically set the wait time property to 20 seconds if the default wait time is 20 seconds. Perhaps the documentation just hasn't been updated yet. Message Size
The message size can be up to 256 KB in size. If you plan on using SQS as a way to manage a data process flow then you might want to consider how easy it is to reach the 256 KB limit. Avoid putting data into the queue messages. Instead, use the messages as notifications for work that needs to be done, and include information that identifies which data is ready to be processed. This is especially important to remember since the messages in the queue can be out of order, and you don't want to count on the data embedded in a message as being the latest version of the data. Message TTL On Queues
Messages have a default life span of 4 days on queues, but can be set to be kept for 1 minute to 2 weeks. Amazon May Delete Unused Queues
Amazon's FAQ mentions that queues may be deleted if no activity has occurred for 30 days. JARs Used By AWS Java SDK
There are certain jar files that you will need to reference when using the AWS Java SDK. They are located in the SDKs "third-party" folder. Here are the jar files I referenced while using the SQS APIs:
I had an issue the other day with AWS an Elastic Load Balancer (ELB) that said the instances I had assigned to the load balancer were "Out of Service". There was a link that was displayed as "(why?)", and would display the hint text of "Instance is in stopped state." This was particularly confusing, because the EC2 console displayed the instances as running.
It turns out that the problem was with the load balancer settings. Doing a search revealed that the error message "Instance is in stopped state." will be displayed when the health check fails. It turns out that the problem was that the health check ping target was pointing to the wrong location (a web page that didn't exist).
I wish that the AWS console would have listed a suggestion of "Please confirm that the health check ping target is correct." instead of just listing an invalid assumption that the instance was in a stopped state. Or, have the "(why?)" anchor display a page of possible troubleshooting steps. One of the suggested steps could still mention the possibility that the instance is stopped.
In the end it was resolved somewhat quickly, but it could have been a lot less stressful if the information provided was more accurate and more helpful.
Amazon's AWS S3 (Simple Storage Service) is incredibly easy to use via the AWS .Net SDK, but depending on your usage of S3 you might have to pay. S3 has a free usage tier option, but the amount of space allowed for use is pretty small by today's standards (5GB). The upside is that even if you end up going outside of the parameters for the free usage tier it is still cheap to use. Here is some information from Amazon regarding the free usage tier limits for S3:
5 GB of Amazon S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests
These free tiers are only available to existing AWS customers who have signed-up for Free Tier after October 20, 2010 and new AWS customers, and are available for 12 months following your AWS sign-up date. When your free usage expires or if your application use exceeds the free usage tiers, you simply pay standard, pay-as-you-go service rates (see each service page for full pricing details). Restrictions apply; see offer terms for more details.
Sign Up To Use AWS You need to create an account in order to use the Amazon Web Services. Make sure you read the pricing for any service you use so you don't end up with surprise charges. In any case, go to http://aws.amazon.com/ to sign up for an account if you haven't done so already. Install or Reference the AWS .Net SDK To start using the AWS .Net SDK to access S3 you will want to either download the SDK from Amazon or use NuGet via Visual Studio. Start Visual Studio (this example is using Visual Studio 2010), and do the following to use NuGet to fetch the AWS SDK:
Select the menu item "Tools | Library Package Manager | Manage NuGet Packages For Solution..."
Type "AWS" in the "Search Online" search text box
Select "AWS SDK for .Net" and click the "Install" button
Click "OK" on the "Select Projects" dialog
Create a Project and Use the AWS S3 API Create a project in Visual Studio, and add the following code:
string key = "theawskeythatyougetwhenyousignuptousetheapis";
string secretKey = "thesecretkeyyougetwhenyousignuptousetheapis";
// create an instance of the S3 TransferUtility using the API key, and the secret key
var tu = new TransferUtility(key, secretKey);
// try listing any buckets you might have
var response = tu.S3Client.ListBuckets();
foreach(var bucket in response.Buckets)
{
Console.WriteLine("{0} - {1}", bucket.BucketName, bucket.CreationDate);
// list any objects that might be in the buckets
var objResponse = tu.S3Client.ListObjects(
new ListObjectsRequest
{
BucketName = response.Buckets[0].BucketName
}
);
foreach (var s3obj in objResponse.S3Objects)
{
Console.WriteLine("\t{0} - {1} - {2} - {3}", s3obj.ETag, s3obj.Key, s3obj.Size, s3obj.StorageClass);
}
}
// create a new bucket
string bucketName = Guid.NewGuid().ToString();
var bucketResponse = tu.S3Client.PutBucket(new PutBucketRequest
{
BucketName = bucketName
}
);
// add something to the new bucket
tu.S3Client.PutObject(new PutObjectRequest
{
BucketName = bucketName,
AutoCloseStream = true,
Key = "codecog.png",
FilePath = "C:\\Temp\\codecog.png"
}
);
// now list what is in the new bucket (which should only have the one item)
var bucketObjResponse = tu.S3Client.ListObjects(
new ListObjectsRequest
{
BucketName = bucketName
}
);
foreach (var s3obj in bucketObjResponse.S3Objects)
{
Console.WriteLine("{0} - {1} - {2} - {3}", s3obj.ETag, s3obj.Key, s3obj.Size, s3obj.StorageClass);
}