Monday, May 6, 2013

Things I learned while using AWS SQS...

Updated 03-20-2017

Amazon's Simple Queue Service (SQS) provides an easy to use mechanism for sending and receiving messages between various applications/processes. Here are a few things that I learned while using the AWS Java SDK to use SQS.


SQS is not can be FIFO

It used to be that AWS SQS didn't guarantee FIFO ordering. Now you can create a standard queue or a FIFO queue. However, there are some differences to be aware between standard and FIFO queues that are worth pointing out. The differences can be read about here. Here are some of the key differences:


Standard Queues - available in all regions, nearly unlimited transactions per second, messages will be delivered at least once but might be delivered more than once, messages might be delivered out of order.

FIFO Queues - available in US West (Oregon) and US East (Ohio), 300 transactions per second, messages are delivered exactly once, order of messages is preserved (as the queue type suggests).

SQS Free Usage Tier

The SQS free usage tier is determined by the number of requests you make per month.  You can make up to 1 million requests per month.  The current fee is $.50 per million requests after the first million requests. The cost is pretty low, but it would be easy to start racking up millions of requests. Luckily, there are batch operations that can be done, and each batch operation is considered one request.


Short Polling/Long Polling

You can set a time limit to wait when polling queues for messages. Short polling is when you make a request to receive messages without setting the ReceiveMessageWaitTimeSeconds property for the queue. Setting the ReceiveMessageWaitTimeSeconds property to up to 20 seconds (20 seconds is the maximum wait time) will cause your call to wait up to 20 seconds for a message to appear on the queue before returning.  If there is a message on the queue, then the call will return immediately with the message.  The advantage to using long polling is that you will make less requests without receiving messages. 


One thing to remember is that if you have only one thread being used to poll multiple queues, then you will have unnecessary wait times when only some of the queues have messages waiting.  A solution to that problem is to use one thread for each queue being polled.


Something that seemed a bit contradictory is that queues created through the web console have the ReceiveMessageWaitTimeSeconds set to 0 seconds (meaning it is going to use short polling). However, the FAQ mentions that the AWS SDK uses 20 second wait times by default. I created a queue using the AWS SDK, and the wait time was listed as 0 seconds in the web console. I shouldn't have to specifically set the wait time property to 20 seconds if the default wait time is 20 seconds.  Perhaps the documentation just hasn't been updated yet.


Message Size


The message size can be up to 256 KB in size. If you plan on using SQS as a way to manage a data process flow then you might want to consider how easy it is to reach the 256 KB limit.  Avoid putting data into the queue messages.  Instead, use the messages as notifications for work that needs to be done, and include information that identifies which data is ready to be processed. This is especially important to remember since the messages in the queue can be out of order, and you don't want to count on the data embedded in a message as being the latest version of the data. 


Message TTL On Queues


Messages have a default life span of 4 days on queues, but can be set to be kept for 1 minute to 2 weeks. 


Amazon May Delete Unused Queues


Amazon's FAQ mentions that queues may be deleted if no activity has occurred for 30 days.


JARs Used By AWS Java SDK


There are certain jar files that you will need to reference when using the AWS Java SDK.  They are located in the SDKs "third-party" folder. Here are the jar files I referenced while using the SQS APIs:

  • third-party/commons-logging-1.1.1/commons-logging-1.1.1.jar
  • third-party/httpcomponents-client-4.1.1/httpclient-4.1.1.jar
  • third-party/httpcomponents-client-4.1.1/httpcore-4.1.jar

2 comments:

  1. I think the message size can be 256 KB

    ReplyDelete
    Replies
    1. Yes - that's correct. I originally wrote this entry in May of 2013, and in June of 2013 the size was increased from 64KB to 256 KB.

      https://aws.amazon.com/about-aws/whats-new/2013/06/18/amazon-sqs-announces-256KB-large-payloads/

      Delete