Thursday, October 23, 2014

Using SolrNet (0.4.0.2002) and AutoMapper...

If you're using Solr and you prefer to use .Net, then SolrNet is a great way to go. It's easy to use whether your indexing documents, querying for documents, or both.

Here is an example using Solr 4.10.1, SolrNet 0.4.0.2002, and AutoMapper

I installed Solr 4.10.1, and used the default example "collection1" core for this example. 

I created a POCO for the example and cleverly named it SolrDoc. I only bothered with a few of the values that are defined in the schema used in the collection1 core:

public class SolrDoc
{
    [SolrUniqueKey("id")]
    public string Id { get; set; }

    [SolrField("sku")]
    public string Sku { get; set; }

    [SolrField("name")]
    public string Name { get; set; }

    [SolrField("cat")]
    public List Categories { get; set; }

}

And created another POCO to act as a domain object (only to show how you can map from the Solr document), and named it SolrItem:


public class SolrItem
{
    public string Identifier { get; set; }
    public string Sku { get; set; }
    public string Name { get; set; }
    public List Cats { get; set; }
    public override string ToString()
    {
        var sb = new StringBuilder(
                 string.Format("Identifier: {0}, SKU: {1}, Name: {2}", 
                                Identifier, Sku, Name));
        if (Cats != null)
        {
            sb.Append("\nCategories:");
            foreach (var category in Cats)
            {
                sb.Append(string.Format("\n\t{0}", category));
            }
        }
        return sb.ToString();
    }
}

The following code isn't necessary for use with Solr, but it is necessary for creating the AutoMapper mapping between SolrDoc and SolrItem. Perhaps you won't need to map your Solr documents to some other contract object, but if you do have that need then AutoMapper is easy to use. 

One way that AutoMapper can be used is to map between public and private views of data. As an example, you might have an API that allows users to view either detailed or summary user info. Your private model of the user data can have all user information and you can have AutoMapper map the user info to summary or detailed user info objects that are returned to the user. I included the AutoMapper usage as a way to show that you can easily get the Solr documents and then map them to whatever form you might need.

Here is the code that configures AutoMapper's source to destination mapping.


public static void InitMappings()
{
    Mapper.CreateMap()
        .ForMember(dest => dest.Cats, opt => opt.MapFrom(src => src.Categories))
        .ForMember(dest => dest.Identifier, opt => opt.MapFrom(src => src.Id));

}

Here is the code that indexes a document into Solr, and then queries Solr for all documents. The foreach loop is where the SolrDoc is mapped to a SolrItem.


static void Main(string[] args)
{
    // Set up the automapper mapping
    InitMappings();
        
    // Point to the Solr server that was started using the Solr example
    // of "java -jar start.jar"
    Startup.Init<SolrDoc>("http://localhost:8983/solr/collection1");

    // Get an instance of the Solr service that will map Solr documents to 
    // a POCO of type SolrDoc
    var solr = ServiceLocator.Current.GetInstance<ISolrOperations<SolrDoc>>();

    // Create a SolrDoc to index into Solr
    var doc = new SolrDoc
    {
        Id = Guid.NewGuid().ToString(),
        Name = "Some SolrDoc",
        Sku = "Some_Sku",
        Categories = new List {"cat1", "cat2", "cat3"}
    };

    // Index and commit the doc
    solr.Add(doc);
    solr.Commit();

    // Query for all documents
    var results = solr.Query(new SolrQueryByField("id", "*"));

    // Loop through all results
    foreach (var result in results)
    {
        var solrItem = Mapper.Map<SolrItem>(result);
        Console.WriteLine(solrItem);
    }
}

Saturday, October 11, 2014

Testing XSLT changes can be slow if you don't have the right tools

There are probably 50 million tools out there for writing XSLT and viewing the output. I have been in the habit of using a command line app that would take a path to the XSLT file, the source xml file, and a target output file. It worked good enough since I don't usually have to do a lot of XSLT. I just started doing changes on a project that uses XSLT to transform data into XML that can be used to index into a Solr instance, and there are quite a few fields that I need to format in a variety of ways. I was using the simple command line app to view the output, but it was getting a bit tedious.  I was wishing I had something more like IntelliJ - something that would show me the output right away, and it dawned on me that all I needed would be three text boxes and a timer control to get what I wanted.  

Introducing the incredibly named XsltTester. It's pretty basic, and I'm sure that there are existing free tools that do the same thing. However, this took just 30 minutes or so and it was fun to put together.

Here is a link to the project on BitBucket: https://bitbucket.org/lee_wallen/xslttester

Here are a few views of the app:


New icons and buttons.


Example with XML and invalid XSL.

Example with XML and fixed XSL but wrong output.

Example with XML, fixed XSL and fixed output.



Monday, July 28, 2014

DIY Motion Detection Using a Raspberry Pi Camera

I received a Raspberry Pi camera for my birthday because my wife is awesome and she asked me what I wanted. I figured it would be fun to make a security camera that will take video when changes are detected.

There is existing software for doing motion detection, but the whole reason I wanted a raspberry pi and camera was to have fun writing code.

The approach I decided to take is to:

1. Take two photos in quick succession (using raspistill).
2. Write a Java app that will
2.1. Make a mosaic out of the photos, and use the average RGB value for each tile.
2.2. Compare each tile from the two mosaic images, and create a new image to show the changes.  The areas where no significant difference were found the code will write out a 0 RGB value (black).  Areas where a significant change is found will use the second mosaic image's RGB value for the tile being compared.
2.3. Check the output image to see if areas that have changed are large enough to seem interesting.  If any of the changed areas are large enough, then send a message to a message queue to request that a video be recorded.
3. If a message is read from the queue requesting that a video to be created, then:
3.1. Record a video.
3.2. Upload the video to a storage area.
3.3. Write a message to a message queue with the location of the video file.

I have the steps 1 through 2.2 done, and I'm pretty happy with the results so far. Here is an example of a couple of images that were taken and compared, and the output of the comparison was created.

Image 1 - notice the white truck towards the center of the image:


Image 2 - the white truck has moved:


Notable Differences:



My next project should be to incorporate something like this (joke):
Autonomous Backyard Defender

Wednesday, May 21, 2014

Raspberry Pi is really really fun...

Raspberry Pi is really really fun to use.  Really. I keep expanding my list of things to do.

1. Map the network and post results to RabbitMQ queue
2. Install NoSQL datastore (perhaps Redis or MongoDB) to store network info
3. Make an RaspBMC Server (following an XBMC DIY blog post)
4. Create a grid computing system based on nothing but Raspberry Pis, and use the computing power to take over the world!

Okay, I might be getting a little carried away, but it's crazy how many things you can do with the Raspberry Pi!

I continued with the "mapping the network" idea.

This is what I have done so far:

1. Installed RabbitMQ on the Raspberry Pi
2. Created a queue for holding nmap (network mapping program) info
3. Installed Java on the Raspberry Pi
4. Created a Java app that will parse nmap results and can send the results to the RabbitMQ nmap info queue
5. Created a cron job that will run the nmap command, and then execute the Java app
6. Created an Android app that will read from the queue and display the results

I plan to expand the Android app so that it will do something with the info.  One idea is to install OpenAP on a wireless AP hub, and then have the app block specific MAC addresses.  Or perhaps permanently assign specific IPs to certain MAC addresses.  Not sure yet.  Ideas?

Below is a screen shot of the Android app. It shows the OS type based on info that nmap was able to fetch from the machine. The grey WiFi symbol is for a machine that nmap wasn't able to determine the OS type.  Also, nmap wasn't able to recognize the Raspberry Pi, so I added code to check the vendor of the MAC address.  If the vendor name started with "edimax" (Edimax is the manufacturer of the WiFi dongle the Raspberry Pi is using), then I set the OS type to Raspberry Pi. I made a similar assumption for my Samsung phone - which is not shown below, but is displayed using an Android icon. The app is named NetHutch - which doesn't necessarily make sense except that I happen to be pulling network info from RabbitMQ.



I'll create a post with some of the Android code soon. Until then, I will update the app to list the date and times that network mappings were performed on the main screen, and the details of which hosts were found on the next screen.

Here is a wire frame view of what the update will look like (once I get the time to make the changes). I used Evolus Pencil to create the wire frame. The numbers to the right of the date on the first screen will be the number of hosts found.



Tuesday, May 13, 2014

Raspberry Pi fun...

I received a Raspberry Pi for Christmas and I hadn't found a good use for it yet - until now!  Well, sort of.  I at least found something kind of fun to do with it...as long as your definition of fun is very loose.

I've been using RabbitMQ at work, and I found someone's post about running RabbitMQ on their Raspberry Pi.  "A ha!", I thought. "I'll load interesting messages onto channels, and then I can write an Android app that will fetch the messages.  It will be so amazing!"  The problem is that I couldn't think of what type of data I would want to get.  I ended up deciding to capture all IP addresses on my home network using nmap.

I installed RabbitMQ on the Raspberry Pi by following someone's blog post (very well written!), and then added a user and virtual host. I then wrote a script and some Java code to log all of the IP addresses on my local network. We have a number of devices that connect and disconnect throughout the day, so there would be enough of a change as time goes on to make it interesting.

The script runs nmap to get the network info, and then it calls some Java code that publishes the info to a RabbitMQ channel. I decided to use Java because it was the easiest for me to write the code. I might redo the code in Python once I get some experience with Python.

The nmap command is pretty straight forward: 

sudo nmap -oX myscaninfo.xml 192.168.1.1/24

The -oX parameter tells nmap to output the results in xml format.  

The IP address with the CIDR (Classless Inter-Domain Routing) value at the end (192.168.1.1/24) specifies the IP addresses to scan.  In the example above, it would mean the following:

192.168.1.1 is the starting IP address. We would then mask the address with 24 1 bits => 255.255.255.0.  255 is what 11111111 equals (-1 due to 0 based index).  That means that we would want to view all IP address that are in the unmasked range - 192.168.1.1 to 192.168.1.255.

The Java code parses the XML and pulls out all the address info. It then pairs the IP address with a MAC address, and publishes the data to RabbitMQ.

The next step will be to create an Android app to read from the channel and display the data.

Wednesday, April 16, 2014

Windows service gotchas...

I'm working on a new Windows service, and it's been a while since I've had to create a Windows service from scratch (they are more work, but they taste the best when created from scratch - har har). Here are some of the gotchas I've run into, and have a vague recollection of dealing with these issues before:

 1. You go to install your service (installutil <path to your service executable>), and then start it (net start <your service name>) but get an "System error 5 has occurred. Access is denied." message. 

 The issue is that you need to make sure that the directory where the executable lives has the correct privileges for whatever user you are using to run the Windows service. In my case, I was trying to run the Windows service as "Local Service". That meant that I needed to set security permissions for the parent directory to include "LOCAL SERVICE" and let "LOCAL SERVICE" have full control.

You can see which user will be executing the service by right clicking on the service from withing the "Services" application, and selecting the "Properties" menu item.  Here is the view of my service properties:

Local Service has minimal privileges, and therefore is more secure to use than Network Service or Local System.  There is no password for Local Service despite what it looks like in the image to the left.



I granted access for Local Service to the bin\Debug folder for my Windows service while testing.  


















2. You go to debug your Windows service, but it errors and quits before you can get to Visual Studio and attach to your Windows service process. I ended up adding a Thread.Sleep(10000) in the Main() method right before the ServiceBase.Run() method is called. That seems to be enough time on my machine to open the Attach to Process dialog and then navigate to my Windows service.

Also, it is important that you are running your Visual Studio instance as administrator, or you won't have privileges to attach to your Windows service.

3. You uninstall your service and then attempt to re-install the service, but you get a message stating that the service has been marked for deletion. Once that happens, then you need to reboot your machine in order to install the service. I ran into that problem before, and wrote about it here, but the short version is that you should avoid having the "Services" application running while uninstalling/installing the service.

I'll add onto this post if I hit other generic issues while working on the Windows service.

Wednesday, January 22, 2014

SQL Server and XML DML

Arthur Olcot has a great blog that covers SQL Server and focuses on optimization and SQLXML. He has a post that lists examples of using XML DML that I found to be incredibly useful. Specifically, the conditional insert example is something that I was able to borrow from for a task at work.  

I needed to add some elements to XML in a bunch of rows of data, and I only wanted to add the elements if they didn't already exist. I was afraid that I might need to do a delete, and then an insert, to make sure that I would end up with duplicate values. Thankfully you can do conditional inserts, and his example made it very easy to understand how to use the conditional.

The following is an example of how to use the conditional insert.


DECLARE @xml XML

SET @xml = '
  Cat 1
  Cat 2
  Cat 3
'

SELECT 'before where Cat 3 already exists', @xml

SET @xml.modify('
 insert (
  if (//category[contains(., "Cat 3")]) then () 
  else element category{"Cat 3"} 
 ) as last into (categories[1])
')

SELECT 'after', @xml


SET @xml = '
  Cat 1
  Cat 2
'

SELECT 'before where Cat 3 does not exist', @xml

SET @xml.modify('
 insert (
  if (//category[contains(., "Cat 3")]) then ()
  else element category{"Cat 3"} 
 ) as last into (categories[1])
')

SELECT 'after', @xml

Thanks, Arthur!