Ask the rubber duck...

Have you ever been moving along on a project and - BAM! - you hit a brick wall of a problem where you just can't see the solution no matter how many things you try?

I had that happen last night while I was working on a personal project. I had just read a post on codinghorror.com that mentioned a problem-solving technique that involves asking a rubber duck whatever question you might have about a problem before you go to a teammate or boss. The idea is that if you take the time to formulate the question in an understandable way and ask the question out loud, then you have a good chance of figuring out a solution to your problem by yourself.

I've had numerous times where I will start talking to a teammate about a problem I'm having and the solution will come to me while I'm explaining the issue.  While it is really helpful to be able to talk to a teammate to work through the problem, it is a lot less distracting to others, and much more rewarding, to work through the problem on my own.

I thought it might look a bit awkward if I started talking to inanimate objects at the house with my wife and kids there, so I decided to talk through the problem with my wife instead.  Of course, it worked great!  I spent 5 minutes (maybe less) talking about the problem, and the solution came to me.  Next time I think I will go to the garage and talk out the problem.

I've also had similar success when going for walks. I will specifically not think about the problem at first. About halfway through the walk, I'll have had a mental reset. The problem won't seem all-encompassing and will usually seem pretty simple.

Both approaches feel very similar. Talking to a person and trying to explain a problem will force you to think about the problem from a new perspective. Taking a break and letting your mind wander can help by giving you an opportunity to view the problem from a new perspective. Similar to a view from a window - if you move your position even just a tiny bit then you might end up seeing something that was previously out of view.

Rubber duck or no rubber duck - talking through problems is extremely helpful!  You might look a bit crazy, but at least you will be able to get your work done.

Great idea from Effective Programming book...

I was reading Jeff Atwood's book Effective Programming, which is based on his excellent Coding Horror blog, and came across a great suggestion. 

Git clone projects that you use.  Reason - if you encounter a problem while using some third party project, then you will better understand how to debug the problem.  You might be able to find and fix bugs in the third party project, but you might be just as likely to learn how you are using some other code incorrectly.

One step further - git clone interesting projects.  Reason - see how other people code and how it differs from what you do. Think about why you like some bit of code or why you don't like it.  The important thing is to have an opinion.  You can always change your mind later.

I've sort of been doing this, but I'm going to try to be more methodical about it now.  It seems like a great way to learn, and get faster at diagnosing root causes of bugs.

Using git with Google Code projects on Windows...

I decided to host a project on Google code, and chose git as the source repository since that is what I am familiar with. The Google code project page presents an auto generated password when you first create your project for use when pushing changes. You can regenerate this password if you need to.

One drawback of using an auto generated password is that they are purposely hard to remember, and therefore are not quite as easy to type in accurately on the command line. The projects source section suggests the following:

Add the following to your .netrc. 
machine code.google.com login <you google email address> password [generated googlecode.com password] 

I had no idea what a .netrc file was, so I looked around. At least on *nix machines, the .netrc file contains auto login information that can be used by ftp and rexec commands.

From what I read it sounded like I just needed to create the .netrc file in my user's home directory with the line as it is listed above.  I created the .netrc file, but I was still being prompted for my username and password when I would attempt to push changes to the repository. I searched a little more and I found a comment by named Dave Borowitz saying that the file should be named _netrc on Windows machines.

I changed my file name to _netrc and then I was able to push changes without entering my password.  

I didn't spend a great deal of time looking for the answer, but I feel like I was a bit lucky as well. I think there should be a mention about naming the file _netrc on the Google project source page.

Alias often...

It's often very frustrating to retype shell commands over and over.  One thing that I sometimes forget to do is to create aliases for common commands.  I just added an alias for something I do quite often - search the shell history for some command I typed in either the previous or current session.

Edit your config file of choice (I updated .profile) and add the following:

alias hgrep='history | grep'

Now I can just type hgrep along with the search phrase of choice.  ie,

somemachine:someuser:/opt/somedir/home/username: hgrep 'start'
  235  ../bin/startup.sh
  250  ../bin/startup.sh
  256  ../bin/startup.sh
 1023  hgrep 'startserv'
 1024  hgrep 'start'

Toyota Tacoma compass display fix...

The compass and temperature overhead display in my 2007 Toyota Tacoma had stopped working a couple of years ago.  I called the dealership and found out that they don't fix that issue - they just replace the unit. A new compass unit, and the replacement time, costs about $300 (at least it did a couple years ago). I didn't want to  pay that much, so I just left it alone.  

Recently I found a YouTube video that shows someone repairing the unit.  The issue that the repair video was addressing is that some of the resistors have weak solder connections, and re-soldering the resistors with poor contact will fix the issue. It turned out to be that easy for me, and I was happy to have a video as a guide.  The person who made the video warned against putting too much stress on some of the plastic tabs, and that warning kept me from breaking tabs.

I removed the overhead display, removed  the display from the housing, and inspected the resistors. There were a number of surface mount resistors, but only two resistors (the ones marked 510) looked like they might not be making good contact.  I used a multimeter to check all of the resistors on the board to make sure they were okay.  Perhaps that was overkill - I wouldn't expect that the resistors would be bad - but it gave me an excuse to use the multimeter and that was good enough for me. Next, I heated up the existing solder with my soldering iron. I figured that would  restore good contact between the board and the resistors.  Then I added a tiny amount of solder to ensure that the connection was good.  I rechecked the resistors with the multimeter, and then reinstalled the overhead display.  I started the truck, and nothing displayed. I was pretty disappointed for a second, but I pushed the button on the display and there it was - temperature and compass direction!  After two years or so of not having a functioning display I had forgotten that the button cycles between Celsius, Fahrenheit, and turning the display off.

Everything worked great, and I saved $300!

AWS S3 Sample Using TransferUtility.DownloadDirectory...

I need to write a utility for a project that I'm working on, and the utility will need to download all of the files in an S3 directory. Luckily the AWSSDK provides an easy way to do this with the TransferUtility.DownloadDirectory method.

The following is a simple example usage of the DownloadDirectory method.  

public class S3Downloader
{

   public void DownloadS3Directory(string bucketName, string s3Directory, 
                                   string localDirectory)
   {
      var s3Config = new AmazonS3Config
      {
         ServiceURL = "s3-us-west-2.amazonaws.com",
         CommunicationProtocol = Protocol.HTTP
      };

      using (var s3Client = new AmazonS3Client(
                                   new EnvironmentAWSCredentials(), 
                                   s3Config))
      {
         using (var transferUtility = new TransferUtility(s3Client))
         {
            var ddr = new TransferUtilityDownloadDirectoryRequest
            {
               BucketName = bucketName,
               LocalDirectory = localDirectory,
               S3Directory = s3Directory
            };

            ddr.DownloadedDirectoryProgressEvent += DisplayProgress;
            transferUtility.DownloadDirectory(ddr);                
         }
      }
   }

   private void DisplayProgress(object sender, 
                                DownloadDirectoryProgressArgs args)
   {
      Console.WriteLine(args);
   }
}

public class Program
{
   public static void Main(string[] args)
   {
      string bucketName = "mybucket";
      string s3Directory = "/archived/files/2013-07";
      string localDirectory = @"C:\Temp\s3test";

      var s3Downloader = new S3Downloader();
      s3Downloader.DownloadS3Directory(bucketName, 
                                       s3Directory, 
                                       localDirectory);
   }
}

Here is an example of what is written to the console:

Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 8192
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 16384
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 24576
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 32768
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 40960
Total Files: 14, Downloaded Files 0, Total Bytes: 57390654, Transferred Bytes: 49152

Line endings and views, although unrelated, can both cause confusion...

Line Endings

Boy how I wish there weren't a difference between Unix and Windows line endings. I recently needed to search through a bunch of map reduce output (tons of reduce files in text format), so I decided to bulk insert them into a SQL database table.

I wrote a bulk insert statement for each of the reduce files (well, some SQL code wrote the bulk insert statements based on a list of file names), and I quickly encountered errors when attempting to execute the statements. The bulk insert statements looked something like this:

BULK INSERT ReduceData_View  FROM 'path\part-r-00000' 
WITH (FIRSTROW = 1,FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n');

The reduce files are rows of data that are tab separated. The frustrating part is that I had failed to remember that the map reduce job was done on a Linux machine, and therefore each file contained Unix style line endings.  I had been viewing the files in Notepad++ (which will display different file formats without complaint), so it was confusing to me when I would see errors. I changed the ROWTERMINATOR value to '\r\n', and everything worked fine.  

Lesson - pay attention to the source system type when processing data files.  I didn't spend a long time on this, but it was still frustrating.

SQL Views

I had created a view to insert into, because I had an identity column on my target table.  It seemed like an easy way to avoid the problem of bulk insert trying to insert into the identity column.  The problem occurred when I ran the SQL to do the inserts, and some of the columns from the source files had values that were too wide for the table columns.  I changed the target table columns to use varchar(max), since some of the source data had values that were larger than 4000 chars.  I ran the bulk insert again and I still had errors due to the data being too large for one of the target columns.  What the heck?  What I hadn't realized is that the view I created had information for the columns in the target table from when the target table was first created.  The table had the column width changed to varchar(max), but the view thought the table column width was varchar(4000).  I ran the SQL to recreate the view and then the bulk insert worked just fine.