tag:blogger.com,1999:blog-46959054096002621692024-03-05T00:25:15.914-08:00Code CognitionA blog about software development and whatever else I find interesting.Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.comBlogger73125tag:blogger.com,1999:blog-4695905409600262169.post-39797853734824030962023-01-07T17:21:00.000-08:002023-01-07T17:21:21.154-08:00Use the Philips Hue motion sensor to monitor temperature...<p><span style="font-family: inherit;">I was looking through the Philips Hue API to see how to interact with the Philips Hue motion sensor, and I was happy to find that the motion sensor includes a temperature sensor.</span></p><p><span style="font-family: inherit;">The temperature sensor will show up in the list of sensors once you add the motion sensor to your Philips Hue Hub.</span></p><p><span data-offset-key="5ijp5-3-0" style="background-color: white; color: #212121; white-space: pre-wrap;"><span data-text="true"><span style="font-family: inherit;">Here is an example of the response data for the temperature sensor (but only showing one sensor instead of all sensors):</span></span></span></p><p><span data-offset-key="5ijp5-3-0" style="background-color: white; color: #212121; font-family: Inter, system-ui, -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, "Fira Sans", "Droid Sans", Helvetica, Arial, sans-serif; white-space: pre-wrap;"><span data-text="true"></span></span></p><p><span style="font-family: courier;"><span class="resolvedVariable" data-testid="resolvedVariable" spellcheck="false" style="background-color: white; color: var(--base-color-brand); white-space: pre-wrap;"><span data-offset-key="5ijp5-0-0">curl http://</span></span></span><span style="font-family: courier;">$HUE_BRIDGE_IP</span><span style="font-family: courier;"><span data-offset-key="5ijp5-1-0" style="background-color: white; color: #212121; white-space: pre-wrap;"><span data-text="true">/api/</span></span></span><span style="font-family: courier;">$HUE_USERNAME</span><span style="font-family: courier;"><span data-offset-key="5ijp5-3-0" style="background-color: white; color: #212121; white-space: pre-wrap;"><span data-text="true">/sensors</span></span></span></p><p><span data-offset-key="5ijp5-3-0" style="background-color: white; color: #212121; font-family: Inter, system-ui, -apple-system, "system-ui", "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, "Fira Sans", "Droid Sans", Helvetica, Arial, sans-serif; font-size: 12px; white-space: pre-wrap;"><span data-text="true"></span></span></p><div style="background-color: #fffffe; line-height: 18px; white-space: pre;"><div><span style="font-family: courier;">{</span></div><div><span style="font-family: courier;"><span> "5": </span>{</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"state"</span>: {</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"temperature"</span>: <span style="color: #098658;">1929</span>,</span></div><div><span style="font-family: courier;"> <span> </span><span style="color: #a31515;">"lastupdated"</span>: <span style="color: #0451a5;">"2023-01-06T16:24:41"</span></span></div><div><span style="font-family: courier;"><span> </span> },</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"swupdate"</span>: {</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"state"</span>: <span style="color: #0451a5;">"noupdates"</span>,</span></div><div><span style="font-family: courier;"> <span> </span><span style="color: #a31515;">"lastinstall"</span>: <span style="color: #0451a5;">"2023-01-04T01:35:43"</span></span></div><div><span style="font-family: courier;"> <span> </span>},</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"config"</span>: {</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"on"</span>: <span style="color: #0451a5; font-weight: bold;">true</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"battery"</span>: <span style="color: #098658;">100</span>,</span></div><div><span style="font-family: courier;"> <span> </span><span style="color: #a31515;">"reachable"</span>: <span style="color: #0451a5; font-weight: bold;">true</span>,</span></div><div><span style="font-family: courier;"> <span> </span><span style="color: #a31515;">"alert"</span>: <span style="color: #0451a5;">"none"</span>,</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"ledindication"</span>: <span style="color: #0451a5; font-weight: bold;">false</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"usertest"</span>: <span style="color: #0451a5; font-weight: bold;">false</span>,</span></div><div><span style="font-family: courier;"> <span> </span><span style="color: #a31515;">"pending"</span>: []</span></div><div><span style="font-family: courier;"><span> </span> },</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"name"</span>: <span style="color: #0451a5;">"Hue temperature sensor"</span>,</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"type"</span>: <span style="color: #0451a5;">"ZLLTemperature"</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"modelid"</span>: <span style="color: #0451a5;">"SML003"</span>,</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"manufacturername"</span>: <span style="color: #0451a5;">"Signify Netherlands B.V."</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"productname"</span>: <span style="color: #0451a5;">"Hue temperature sensor"</span>,</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"swversion"</span>: <span style="color: #0451a5;">"2.53.6"</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"uniqueid"</span>: <span style="color: #0451a5;">"00:11:22:33:bb:00:ee:33-22-4444"</span>,</span></div><div><span style="font-family: courier;"><span> </span> <span style="color: #a31515;">"capabilities"</span>: {</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"certified"</span>: <span style="color: #0451a5; font-weight: bold;">true</span>,</span></div><div><span style="font-family: courier;"> <span> </span> <span style="color: #a31515;">"primary"</span>: <span style="color: #0451a5; font-weight: bold;">false</span></span></div><div><span style="font-family: courier;"> <span> </span>}</span></div><div><span style="font-family: courier;"><span> }</span><br /></span></div><div><span style="font-family: courier;">}</span></div></div><p><span data-offset-key="5ijp5-3-0"><span data-text="true"><span style="font-family: inherit;"><span style="background-color: white; white-space: pre-wrap;">The temperature value of "1929" converts to </span><span style="background-color: #fffffe; white-space: pre;">66.72 degrees Fahrenheit.</span></span></span></span></p><p><a href="https://github.com/leewallen/motion-sensor" target="_blank">Here is an example python app</a> that will watch and print out the temperature value returned from the temperature sensor in t<a href="https://amzn.to/3vJYGGQ" target="_blank">he Philips Hue motion sensor</a>.</p><p>The example app will monitor the temperature sensor, and print out the temperature for each sensor you ask to monitor via a comma separated string of sensor IDs you can pass in on the command line.</p><p>I have two motion sensors - one in the garage and one in a room in the house.</p><p>The following is the command line I used. Note that I used a comma separated string for the two temperature sensors, and set the monitoring interval to run every 10 seconds:</p><p><span style="font-family: courier;">python3 main.py -b $HUE_BRIDGE_IP -u $HUE_USERNAME -s 5,18 -i 10</span></p><p><br /></p><p>Here is an example of the output:</p><p><span style="font-family: courier;">Temp is unchanged. Temp: 55.42</span></p><p><span style="font-family: courier;">Temp is unchanged. Temp: 66.72</span></p><p><span style="font-family: courier;">Temp is unchanged. Temp: 55.42</span></p><p><span style="font-family: courier;">Temp changed! Temp: 67.17</span></p><p><br /></p><p><br /></p><p>Example app in Github: <a href="https://github.com/leewallen/motion-sensor">https://github.com/leewallen/motion-sensor</a></p>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-52578052187007050122022-12-30T08:51:00.000-08:002022-12-30T08:51:35.582-08:00Playing around with Philips Hue lights from the command line...<p style="text-align: left;">We own a bunch of <a href="https://amzn.to/3Z3QIFR" target="_blank">Philips Hue light bulbs</a> now, and we just bought some as a gift for one of our kids. You can use the Philips Hue app on your computer and/or mobile device, but it's also a lot of fun to play with the lights (and <a href="https://amzn.to/3G7knFG" target="_blank">other Philips Hue devices</a>) from a terminal or an app that you can write for yourself using the Philips Hue API. </p><h3 style="text-align: left;">Register to get access to the Developer's API documentation</h3><p style="text-align: left;">You can find the Philips Hue Developer documentation here: https://developers.meethue.com/</p><p style="text-align: left;">The Developer page has a login link. Click the <a href="https://developers.meethue.com/login/" target="_blank">Login link</a>, and then select the <span style="font-family: courier;">Register</span> option at the bottom of the Login form. Or, use <a href="https://developers.meethue.com/register/" target="_blank">this link to get to the Register</a> page.</p><p><br /></p><h3 style="text-align: left;">Find your Hue bridge IP address</h3><p style="text-align: left;">Follow the instructions listed <a href="https://developers.meethue.com/develop/get-started-2/#follow-3-easy-steps" target="_blank">here</a> to find the IP address of your Hue bridge.</p><p style="text-align: left;">The easiest method for finding your Hue bridge IP address is to make sure you're connected to the same network that the Hue bridge is connected to and then visit this address: <a href="https://discovery.meethue.com/">https://discovery.meethue.com/</a></p><p style="text-align: left;">The response will look something like this:</p><div style="text-align: left;"><span style="font-family: courier;">[<br /> {<br /> "id": "112233aabb44cc55",<br /> "internalipaddress": "10.0.0.1",<br /> "port": 443<br /> }<br />]</span></div><p><br /></p><h3 style="text-align: left;">Use curl to send HTTP requests</h3><p>You can use <span style="font-family: courier;">curl</span> to send requests to the Hue bridge.</p><p>If you're new to using <span style="font-family: courier;">curl</span>, then <a href="https://www.youtube.com/watch?v=7XUibDYw4mc" target="_blank">watch this short video about curl</a> for sending HTTP requests to a server. In our case, the server will be the Philips Hue Hub using its IP address. We'll pretend that the IP address of the Hue bridge is <span style="font-family: courier;">10.0.0.1</span> for the examples.</p><p><br /></p><h3 style="text-align: left;">Go to the Hue API docs</h3><p>After registering/logging in on the <a href="https://developers.meethue.com/" target="_blank">Philips Hue Developer page</a> select the <span style="font-family: courier;">Develop</span> menu and then select the <span style="font-family: courier;">Hue API</span> menu item. </p><p><br /></p><h3 style="text-align: left;">Create a username</h3><p>The <span style="font-family: courier;">Hue API</span> requires a username. The username is created and saved by your Philips Hue bridge, and the hub adds the username to a list of usernames that are allowed to send API calls to the Hue bridge.</p><p><a href="https://developers.meethue.com/develop/hue-api/7-configuration-api/#create-user" target="_blank">Follow the instructions here to create a username</a>.</p><p><br /></p><h4 style="text-align: left;">Use curl to get light info from the bridge</h4><p></p><p style="text-align: left;"></p><ol style="text-align: left;"><li>Open a terminal</li><li>Set a variable in your terminal that will hold the Hue Bridge IP address you retrieved above. For example, if your Hue bridge IP address is <span style="font-family: courier;">10.0.0.1</span>, then you would enter the following:<br /><br />HUE_BRIDGE=10.0.0.1<br /><br /></li><li>Set a variable in your terminal that will hold Hue username you retrieved above. For example, if your assigned username is <span style="font-family: courier;">nE1wnTaC00ki3</span>, then you would enter:<br /><br /><span style="font-family: courier;">HUE_USERNAME=nE1wnTaC00ki3<br /></span><br /></li><li>Send a curl command to your Hue bridge to get info about all of the lights you currently have associated with your bridge:<br /><br /><span style="font-family: courier;">curl -XGET http://$HUE_BRIDGE/api/$HUE_USERNAME/lights<br /></span><br /></li></ol><p></p><p>You should get a response that is a JSON string containing information about all of your Hue lights.</p><div><br /></div>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-59364808980552016742021-02-09T09:31:00.003-08:002021-02-10T09:10:10.943-08:00Use Jupyter Lab with PySpark and S3<p><span style="font-family: verdana;">If you're having an issue accessing S3 data from JupyterLab, then read on! Perhaps the info here, or in the linked GitHub repo, might help you discover and resolve your issue.</span></p><p><span style="font-family: verdana;">Before I get into any of the details, <a href="https://github.com/leewallen/jupyter-with-aws-s3" target="_blank">feel free to use this GitHub repo</a> as an example of how to configure your JupyterLab PySpark notebook (running in a docker container on your local machine) to access data in S3.</span></p><p><span style="font-family: verdana;">I was recently wanting to use JupyterLab to view some parquet data in S3. I chose the pyspark-notebook image from the Jupyter Docker Stacks repo as a base Docker image and added jar files that would allow Spark to connect and read/write data to S3. </span></p><p><span style="font-family: verdana;">It seemed like it should be pretty easy and technically it was. However, I ended up getting 403 errors when the pyspark code would try to read data from S3. The reason for the 403 error was understandable when I realized what I had done, but I didn't find any posts or documentation that illustrated my exact problem, so I figured I would share it here.</span></p><p><span style="font-family: verdana;">I wasn't setting the credentials provider explicitly when configuring the Spark session. I had passed in temporary credentials as environment variables when running the a script to start the Jupyter Lab container. I retrieved the credentials by using the following command:</span></p><p></p><blockquote><span style="font-family: verdana;">aws --profile <your aws profile name> --region us-west-2 sts get-session-token</span></blockquote><p></p><p><span style="font-family: verdana;">The default credentials provider was being used because I didn't explicitly set a credential provider to use. I assume that the access key and session key had info tying them to a session so the default credentials provider received an error when attempting to connect since no session token would have been passed to AWS. </span></p><p><span style="font-family: verdana;">Here is the </span><a href="https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Using_Session_Credentials_with_TemporaryAWSCredentialsProvider" rel="nofollow" style="font-family: verdana;" target="_blank">documentation from Hadoop where it shows the property names and values to use when configuring Spark to access data from S3</a><span style="font-family: verdana;">. It's what I read and realized that I needed to set the credentials provider explicitly. </span></p><p><span style="font-family: verdana;">The Hadoop documentation also reminded me that you don't need to explicitly set the access key, secret key, and session token in the Spark session configuration if you use the standard AWS environment variable names. It makes sense - I just had been explicitly setting the credentials previously.</span></p><p><span style="font-family: verdana;"><a href="https://github.com/leewallen/jupyter-with-aws-s3" target="_blank">You can view a GitHub repo</a> that includes a Dockerfile, run script to start the docker container that passes the AWS credentials as environment variables, and an example JupyterLab notebook that uses pyspark to connect to S3 and download data. The bucket referenced in the example is private, so you'll need to substitute the S3 URI to a URI that points to publicly available data or use a bucket that you have permission to access and read. </span></p><p><span style="font-family: verdana;">If you're interested in the data I used then you can find it here: </span><a href="https://covid.cdc.gov/covid-data-tracker/#datatracker-home" rel="nofollow" style="font-family: verdana;" target="_blank">the CDC's COVID-19 dataset.</a></p><p><br /></p><p><span style="font-family: verdana;">Good luck, and have fun Sparking!</span></p>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-19713358336033590012021-01-04T21:58:00.008-08:002021-01-12T07:41:32.194-08:00Notes from installing K3s on my Raspberry Pi cluster...<div style="text-align: left;"><span style="font-size: medium;"><span style="font-family: arial;">I put a Raspberry Pi cluster together - now what should I do?</span><br style="font-family: arial;" /></span></div><div style="text-align: left;"><span style="font-family: arial; font-size: medium;"><br /></span></div><div style="text-align: left;"><span style="font-size: medium;"><span style="font-family: arial;">I have a problem when it comes to Raspberry Pis. Actually, I have a problem with impulsivity, but I'll pretend that the issue is exclusive to Raspberry Pis. The great thing is that t</span><span style="font-family: arial;">he cost of the various Raspberry Pi models is low enough that it's relatively cheap to build a cluster. </span></span></div><p><span style="font-family: arial; font-size: medium;">I knew I wanted to do the following:<br /></span></p><ul style="text-align: left;"><li><span style="font-family: arial; font-size: medium;">Learn more about Kubernetes without using work resources.</span></li><li><span style="font-family: arial; font-size: medium;">Try some of the very neat open source projects that are mentioned on <a href="https://kubernetespodcast.com/" rel="nofollow" target="_blank">the Kubernetes podcast</a> (ie, <a href="https://www.openfaas.com/" rel="nofollow" target="_blank">OpenFaas</a>, <a href="https://github.com/minio/minio" rel="nofollow" target="_blank">MinIO</a>, etc).</span></li><li><span style="font-family: arial; font-size: medium;">Play around with some DIY home automation. I'm not completely sure what I want to do yet, but I have a bunch of Philips Hue lights that are begging to be controlled from the cluster.</span></li></ul><span style="font-family: arial; font-size: medium;"><div><span style="font-family: arial; font-size: medium;"><br /></span></div>I did a search for "Raspberry Pi cluster" and "Kubernetes", and found Jeff Geerling's <a href="https://www.youtube.com/watch?v=xNndbfxMCLo" target="_blank">Raspberry Pi Cluster Ep 2 - Setting up the Cluster YouTube video</a>. <br /><br />Jeff's Kubernetes and Raspberry Pi videos are packed with useful information and he communicates very clearly. Definitely check them out! <br /><br />I mostly followed <a href="https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-episode-2-setting-cluster" rel="nofollow" target="_blank">his related blog post</a> when I went to install K3s on my cluster. The K3s site has great information as well, and the install steps (all two of them) are very simple to follow. </span><div><span style="font-family: arial; font-size: medium;"><br /></span></div><div><span style="font-family: arial; font-size: medium;">However, I ran into an issue or two when I was first setting up the cluster, so I took some notes that I'm providing here. </span><p></p><h3 style="text-align: left;"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif" style="caret-color: rgb(0, 0, 0);"><span style="font-family: arial; font-size: medium;"><br /></span></span></h3><h3 style="text-align: left;"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif" style="caret-color: rgb(0, 0, 0);"><span style="font-family: arial; font-size: medium;">Initial OS Selection and Setup</span></span></h3><div><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif" style="caret-color: rgb(0, 0, 0);"><span style="font-family: arial; font-size: medium;"><br /></span></span></div><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">I used<span class="Apple-converted-space"> </span><a href="https://downloads.raspberrypi.org/raspios_lite_armhf_latest" style="color: #4895d9; margin: 0px; padding: 0px;">Raspberry Pi OS Lite</a><span class="Apple-converted-space"> </span>from the<span class="Apple-converted-space"> </span><a href="https://www.raspberrypi.org/downloads/raspberry-pi-os/" style="color: #4895d9; margin: 0px; padding: 0px;">Raspberry Pi OS Download</a><span class="Apple-converted-space"> </span>page.</span></p><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Jeff's blog post lists steps for how to copy the Raspberry Pi OS image onto the SD cards, so I'll leave those steps out. However, before I unmounted the SD card I copied the following text to the /boot/cmdline.txt file:<br /></span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-align: left; text-size-adjust: auto;"><code style="background-color: transparent; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-family: arial;"><span style="font-size: 15.3px;">cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory</span><span style="font-size: medium;">
</span></span></code></pre><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-align: left; text-size-adjust: auto;"></p><h3 style="text-align: left;"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif"><span style="font-family: arial; font-size: medium;"><br /></span></span></h3><h3 style="text-align: left;"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif"><span style="font-family: arial; font-size: medium;">On First Boot</span></span></h3><div><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif"><span style="font-family: arial; font-size: medium;"><br /></span></span></div><div><span style="caret-color: rgb(0, 0, 0); font-family: arial; font-size: medium;">I m</span><span style="caret-color: rgb(0, 0, 0); font-family: arial; font-size: medium;">ade the following changes on first boot of each image:</span></div><div><span style="caret-color: rgb(0, 0, 0); font-family: arial; font-size: medium;"><br /></span></div><ol style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em 2em; padding: 0px; text-size-adjust: auto;"><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Changed the password</span></li><li style="margin: 0px; padding: 0px;"><code style="box-sizing: border-box; color: #899199; margin: 0px 0px 1.7em; padding: 0px;"><span style="font-family: arial; font-size: medium;">sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get install -y vim</span></code></li><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Set the hostname. I named my Raspberry Pis so they matched my network cable colors, so I ran a command similar to this:<span class="Apple-converted-space"> </span><code style="box-sizing: border-box; color: #899199; margin: 0px 0px 1.7em; padding: 0px;">echo -e "green" | sudo tee /etc/hostname</code></span></li><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Set locale, timezone, and WLAN channel country via raspi-config</span></li><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Updated raspi-config</span></li><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Changed memory available to the GPU to 16 MB (A3 Memory Split)</span></li><li style="margin: 0px; padding: 0px;"><span style="font-family: arial; font-size: medium;">Expanded the file system to max size available</span></li></ol><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-size: medium;"><span style="font-family: arial;">There is (or was) an issue with Raspberry Pi OS Lite (Raspbian Buster) from a recent update, and the cgroup info that is added to /boot/cmdline.txt was being ignored. The</span><span class="Apple-converted-space" style="font-family: arial;"> </span><a href="https://blog.codybunch.com/2020/07/31/Fixing-cgroup-memory-on-Raspbian-Buster-for-Kernel-54x/" style="color: #4895d9; font-family: arial; margin: 0px; padding: 0px;">work-around is to run a raspberry pi update</a><span style="font-family: arial;">.</span></span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-size-adjust: auto;"><code style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: medium;">sudo PRUNE_MODULES=1 RPI_REBOOT=1 SKIP_WARNING=1 rpi-update</span></code></pre><h3 style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Before Installing K3s</span></h3><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">There is a section on rancher.com where it shows that if you are using Raspian Buster (which is Raspberry Pi OS Lite) then you should<span class="Apple-converted-space"> </span><a href="https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster" style="color: #4895d9; margin: 0px; padding: 0px;">enable legacy iptables</a>:</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-size-adjust: auto;"><code style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: medium;">sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot</span></code></pre><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">I ran the following on each of the Raspberry Pis:</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-align: left; text-size-adjust: auto;"><code style="background-color: transparent; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: medium;">echo -e "10.0.1.1\twhite" | sudo tee -a /etc/hosts
echo -e "10.0.1.2\tred" | sudo tee -a /etc/hosts
echo -e "10.0.1.3\tgreen" | sudo tee -a /etc/hosts
echo -e "10.0.1.4\tblue" | sudo tee -a /etc/hosts
echo -e "10.0.1.5\tblack" | sudo tee -a /etc/hosts</span></code></pre><h3 style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Installing K3s</span></h3><h5 style="caret-color: rgb(0, 0, 0); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 1em; line-height: 1.15; margin: 1.7em 0px 1em; padding: 0px; text-size-adjust: auto; text-transform: uppercase;">SERVER INSTALL</h5><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">This will tell the installer that it is a<span class="Apple-converted-space"> </span><strong style="margin: 0px; padding: 0px;">server</strong><span class="Apple-converted-space"> </span>because the<span class="Apple-converted-space"> </span><strong style="margin: 0px; padding: 0px;">K3S_URL</strong><span class="Apple-converted-space"> </span>wasn’t set:</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-size-adjust: auto;"><code class="language-shell" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; box-sizing: border-box; color: #616870; display: block; font-size: 15.3px; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;">curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -
</code></pre><h5 style="caret-color: rgb(0, 0, 0); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen-Sans, Ubuntu, Cantarell, "Helvetica Neue", sans-serif; font-size: 1em; line-height: 1.15; margin: 1.7em 0px 1em; padding: 0px; text-size-adjust: auto; text-transform: uppercase;">AGENT INSTALL</h5><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Get the token from the leader node and trigger the download and execution of the install script. The first Pi in the cluster has a white network cable, so I arbitrarily chose it as the leader node. I ran the following on each of the follower nodes.</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-align: left; text-size-adjust: auto;"><code style="background-color: transparent; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: 15.3px;">export TOKEN=`ssh -t pi@white sudo cat /var/lib/rancher/k3s/server/node-token`
curl -sfL https://get.k3s.io | K3S_URL=https://10.0.1.1:6443 K3S_TOKEN=$TOKEN sh -</span></code></pre><h3 style="caret-color: rgb(0, 0, 0); line-height: 1.15; margin: 1.7em 0px 1em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Test the Cluster</span></h3><p style="caret-color: rgb(0, 0, 0); line-height: 1.15; margin: 1.7em 0px 1em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">If you haven’t already installed<span class="Apple-converted-space"> </span><a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/" style="color: #4895d9; margin: 0px; padding: 0px;">kubectl</a>, then install it now.</span></p><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Copy the<span class="Apple-converted-space"> </span><code style="box-sizing: border-box; color: #899199; margin: 0px 0px 1.7em; padding: 0px;">/etc/rancher/k3s/k3s.yaml</code><span class="Apple-converted-space"> </span>file from the leader node of your Raspberry Pi cluster, to<span class="Apple-converted-space"> </span><code style="box-sizing: border-box; color: #899199; margin: 0px 0px 1.7em; padding: 0px;">~/.kube/config</code><span class="Apple-converted-space"> </span>on whatever computer you plan on using to access the cluster. The location might be different on a Windows machine, but that would most likely be the correct path if you’re using the bash shell.</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-size-adjust: auto;"><code style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: medium;">scp pi@white:/etc/rancher/k3s/k3s.yaml ~/.kube/config
</span></code></pre><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Now run<span class="Apple-converted-space"> </span><code style="box-sizing: border-box; color: #899199; margin: 0px 0px 1.7em; padding: 0px;">kubectl get pods —all-namespaces</code><span class="Apple-converted-space"> </span>and see that the cluster already has pods running on the nodes.</span></p><h3 style="caret-color: rgb(0, 0, 0); line-height: 1.15; margin: 1.7em 0px 1em; padding: 0px; text-align: left; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">Let's Run Something!</span></h3><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">I found Alex Ellis's blog post called <a href="https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/" rel="nofollow" target="_blank">"Will it cluster?"</a> and followed his instructions for installing OpenFaas. It seems funny now because I had no idea who <a href="https://www.openfaas.com/team/" rel="nofollow" target="_blank">Alex Ellis</a> was at the time. I only tried OpenFaas because of the "Will it cluster?" blog post. </span></p><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">I installed OpenFaas, copied the service and deployment yaml files Alex provided in the "Will it cluster?" post, and was able to use the Figlet application in just a few minutes. Very neat stuff!</span></p><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">I recommend going to the OpenFaas.com site to learn more about using OpenFaas. </span></p><p style="box-sizing: border-box; caret-color: rgb(0, 0, 0); margin: 0px 0px 1.7em; padding: 0px; text-size-adjust: auto;"><span style="font-family: arial; font-size: medium;">This example shows that sending text to one of the nodes in the cluster (it doesn't matter which node you pick) will run the figlet function - a function that converts your text data into an ASCII art version of the text.</span></p><pre style="background-color: #f6f7f8; border-radius: 3px; box-sizing: border-box; caret-color: rgb(0, 0, 0); margin-bottom: 1.7em; margin-top: 0px; padding: 1em; text-size-adjust: auto;"><code class="language-bash" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; box-sizing: border-box; color: #616870; display: block; margin: 0px; overflow-x: auto; padding: 0px; width: 714px;"><span style="font-size: medium;">> echo -n "Hello" | curl --data-binary @- http://red:31111</span><span style="font-size: 15.3px;">
_ _ _ _
| | | | ___| | | ___
| |_| |/ _ \ | |/ _ \
| _ | __/ | | (_) |
|_| |_|\___|_|_|\___/</span></code></pre></div>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-211535248263153312020-08-23T01:42:00.004-07:002021-01-04T19:14:49.074-08:00K3s Raspberry Pi 4 Cluster<p><span style="font-family: arial;">Do you have an interest in Raspberry Pis and cluster computing? Me too! </span></p><p><br /></p><div class="separator" style="clear: both; text-align: left;"><h3 style="text-align: left;"><span style="font-family: arial;">Parts</span></h3><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">One thing that I have enjoyed about building the Raspberry Pi cluster is that it's inexpensive to build up over time. </span></div></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;">I bought the Raspberry Pis and the PoE hats from Canakit, and everything else was bought from Amazon. Amazon had higher prices for the Raspberry Pi PoE hat, and didn't have a Raspberry Pi 4B 8GB board available when I first started buying the parts. <br /><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br />Here is a list of what I bought for the cluster. </span></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">
<table style="border-collapse: separate; border-spacing: 20px 5px;">
<thead>
<tr style="border-bottom: 1px solid black; padding: 5px;">
<th style="font-family: arial; text-align: left;">Part</th>
<th style="font-family: arial; text-align: left;">My Choice</th>
<th style="font-family: arial; text-align: left;">Qty</th>
<th style="font-family: arial; text-align: left;">Links</th>
</tr>
</thead>
<tbody>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">Case</span></td>
<td><span style="font-family: arial; font-weight: normal;">Cloudlet Cluster Case</span></td>
<td><span style="font-family: arial;">1</span></td>
<td><span style="font-family: arial;"><a href="https://amzn.to/2EljJsf">Amazon</a></span></td>
</tr>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">Raspberry Pis</span></td>
<td><span style="font-family: arial; font-weight: normal;">RPi 4B+ 8GB</span></td>
<td><span style="font-family: arial;">5</span></td>
<td><span style="font-family: arial;"><a href="https://www.canakit.com/raspberry-pi-4-8gb.html">Canakit</a><br /><a href="https://amzn.to/3hm8MVB">Amazon</a></span></td>
</tr>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">PoE Hats</span></td>
<td><span style="font-family: arial; font-weight: normal;">Raspberry Pi PoE HAT</span></td>
<td><span style="font-family: arial;">5</span></td>
<td><span style="font-family: arial;"><a href="https://www.canakit.com/raspberry-pi-poe-hat.html">Canakit</a><br /><a href="https://amzn.to/2Qj8Gly">Amazon</a></span></td>
</tr>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">SD Cards</span></td>
<td><span style="font-family: arial; font-weight: normal;">SanDisk 128GB microSDXC</span></td>
<td><span style="font-family: arial;">5</span></td>
<td><span style="font-family: arial;"><a href="https://amzn.to/3j4MDeJ">Amazon</a></span></td>
</tr>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">Network Switch </span></td>
<td><span style="font-family: arial; font-weight: normal;">TP-Link 8 Port PoE Switch </span></td>
<td><span style="font-family: arial;">1</span></td>
<td><span style="font-family: arial;"><a href="https://amzn.to/3hpEeCc">Amazon</a></span></td>
</tr>
<tr style="padding: 1px;">
<td><span style="font-family: arial;">Network Cables</span></td>
<td><span style="font-family: arial; font-weight: normal;">Cat7 1FT Multi-Color</span></td>
<td><span style="font-family: arial;">1 x 5pk </span></td>
<td><span style="font-family: arial;"><a href="https://amzn.to/3jiHJLB">Amazon</a></span></td>
</tr>
</tbody>
</table>
</div><div><br /></div><h3 style="clear: both; text-align: left;"><span style="font-family: arial;">Case</span></h3><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I used the Cloudlet Cluster Case by C4Labs.</span></div><div><div class="separator" style="clear: both; text-align: center;"><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCjNIeLPPA6oLzyYHGga62tYwFnuUuslPS9YHCSXSRg8h-VT9Mdq2R0zRF6dSDtOhSvE3Q4MoDtU8EeLItoORHU9ufKpZuIRtCG8ljFSWdf6ouzhVanf35IkuAxN62LBg9tCbXBYITyeM/s2048/k3s-on-rpis.jpg" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1536" data-original-width="2048" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCjNIeLPPA6oLzyYHGga62tYwFnuUuslPS9YHCSXSRg8h-VT9Mdq2R0zRF6dSDtOhSvE3Q4MoDtU8EeLItoORHU9ufKpZuIRtCG8ljFSWdf6ouzhVanf35IkuAxN62LBg9tCbXBYITyeM/w328-h246/k3s-on-rpis.jpg" width="328" /></a></td></tr><tr><td class="tr-caption"><span style="font-family: arial; font-size: small;"><i><b>Very tiny, but fulfilling, RPi cluster.</b></i></span></td></tr></tbody></table><p></p><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><p><span style="font-family: arial;">It practically hums like the </span><a href="https://www.youtube.com/watch?v=iRsycWRQrc8" style="font-family: arial;" target="_blank">WOPR</a><span style="font-family: arial;">!</span></p><div class="separator" style="clear: both;"><br /></div><span style="font-family: arial;">I found the case by searching on Amazon for "Raspberry Pi Cluster Case', and the Cloudlet Cluster Case was the first result I saw that I really liked. I like the look of the stackable cases, but the Cloudlet Cluster Case reminded me of a very tiny computer rack - it felt right.</span></div></div></div><div><span style="font-family: arial;"><br />The case has mounting boards and hardware for 8 Raspberry Pis. You mount the Raspberry Pi onto the acrylic board, and then the mounting board easily snaps into the case. </span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">This is great for allowing you to start very small and expand as you would like. The price might seem like a considerable jump from the stackable case options but I still picked the Cloudlet Cluster Case because I think it looks nice, it's very sturdy, and had enough room for the network switch. </span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">You can also see a blue square in the image above - that's the <a href="https://www.blinkstick.com/products/blinkstick-square" target="_blank">Blinkstick Square</a> with an enclosure. I plan to set up monitoring for the cluster, and for a variety of webhooks, and use the Blinkstick Square for showing status. I figured I would use white, red, green, blue, and purple to indicate which Pi/Node the status was for. </span></div><div><br /></div><h3 style="text-align: left;"> <span style="font-family: arial;">Raspberry Pis</span></h3><div><span style="font-family: arial;"><br /></span></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" style="clear: right; display: inline; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="340" data-original-width="583" height="187" src="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" width="320" /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/raspberry-pi-4-labelled-2857741801afdf1cabeaa58325e07b58.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a></div></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">Originally I was going to build a cluster from a couple old Raspberry Pis I have that I hadn't been using, but I bought a Raspberry Pi 4 bundle for my daughter and the performance is so good that I decided to get the newer Pis instead. </span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I would have liked to have bought the Raspberry Pis from Amazon, because I appreciate the customer service you get from Amazon. However, I have had great luck with smaller businesses that sell Raspberry Pi products, and usually they have better prices than what you would good from Amazon. Canakit had Raspberry Pi 4B 8GB boards available before Amazon, and the price is about $15 cheaper. Vilros and the PiShop also had Raspberry Pi 4B 8GB boards listed, but had the same price as Canakit.</span></div><div><span style="font-family: arial;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: arial;">PoE Hats</span></h3><div><div class="separator" style="clear: both; text-align: center;"><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: left; display: inline; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: left; display: inline; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: left; display: inline; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: left; display: inline; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="400" data-original-width="600" height="170" src="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" width="256" /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: right; display: inline; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: right; display: inline; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://www.raspberrypi.org/homepage-9df4b/static/c6243be5fb082dd225e2f4cdb64ce016/ae23f/b047e09cf9d833615760747414238e7fe5e99e4b_770a6339-1619x1080.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a></div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I found a few PoE hat options, but I went for the official </span><a href="https://www.raspberrypi.org/products/poe-hat/" style="font-family: arial;" target="_blank">Raspberry Pi PoE hat</a><span style="font-family: arial;">. </span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">The price was better than most options, and I assumed there would be more testing around the official option. Also, one PoE hat that I looked at seemed to have a nicer profile but the seller suggests buying a fan for it. </span><span style="font-family: arial;">The official Raspberry Pi PoE hats come with fans attached and there is no issues with clearance in the Cloudlet Case. </span></div><div><span style="font-family: arial;"><br /></span></div><div><br /></div><div><span style="font-family: arial;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: arial;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: arial;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: arial;">SD Cards</span></h3><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I bought 128 GB SD cards for each of the Pis. I didn't need SD cards that hold that much data because I can attach an external HDDs or SSDs to add storage. If I were to do this again, then I think I would buy smaller SD cards, and use the saved money to go towards external drives.</span></div><div><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br /></span></div><h3 style="clear: both; text-align: left;"><span style="font-family: arial;">Network Switch</span></h3><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;">I bought an 8 port PoE switch from TPLink mainly because it was the cheaper option between it and a Netgear 8 port PoE switch. The TPLink switch is $30 cheaper than the Netgear equivalent. I had no problems at all - it works great with the Raspberry Pi PoE hats. There were no special configurations for the Raspberry Pi, no jumper settings for the PoE hat, and nothing to configure for the switch. Just connect all the things.</span></div><div class="separator" style="clear: both; text-align: left;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://i.imgflip.com/4ceoyg.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="667" height="320" src="https://i.imgflip.com/4ceoyg.jpg" width="427" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><span style="font-family: arial;">There will be a post coming soon that will list the steps I took to set up the Raspberry Pis and get K3s installed. It was relatively simple, but not completely hassle-free. The first time I was able to see that all nodes were running and available to the cluster made it worth it. </span></div><div class="separator" style="clear: both; text-align: left;"><br /></div><p></p>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-69880215045342104122020-08-22T16:57:00.003-07:002020-08-22T16:59:26.694-07:00Welcome to 2020! Err...I mean, welcome to August 2020!<span style="font-family: arial;">I love home projects where the main purpose is to learn something new. The only projects I love more are the <b><i>next</i></b> learning projects.</span><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">Here are some areas I want to learn more about:</span></div><div><ul style="text-align: left;"><li><span style="font-family: arial;">Kubernetes</span></li><li><span style="font-family: arial;">Tracking useful metrics</span></li><li><span style="font-family: arial;">For the near future:</span></li><ul><li><span style="font-family: arial;">Serverless for Kubernetes</span></li><li><span style="font-family: arial;">Load testing at a small scale</span></li><li><span style="font-family: arial;">Data pipelines</span></li></ul></ul><span style="font-family: arial;"><br /></span><div><span style="font-family: arial;">Some of the items are very easy to explore at work but not all of them, so I plan to focus on projects that are not work specific.</span></div><div><span style="font-family: arial;"><br /></span></div><div><h3 style="text-align: left;"><b><span style="font-family: arial;">Kubernetes</span></b></h3></div><div><b><span style="font-family: arial;"><br /></span></b></div><div><span style="font-family: arial;">You might ask, "Who isn't using Kubernetes?"</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">Well, I wasn't until recently!</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">It's been really fun. There are so many open-source projects that I want to use that will easily run on Kubernetes that it's almost hard to force myself to start at the beginning and learn how to set up and manage a cluster. Or even a tiny cluster - but that's what I'll do first!</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I had wanted to make a Raspberry Pi cluster for a while and this provided the excuse. I bought some Raspberry Pis, a nice case, an unmanaged switch that had 8 PoE ports, PoE hats for the Raspberry Pis, SD cards, and network cables. I put the Pis in the case, installed <a href="https://k3s.io" target="_blank">k3s</a> (Lightweight Kubernetes), and now I just need some projects to help provide areas to start digging!</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I'll share the steps I followed, parts I used for putting the cluster together, and anything I might do differently if I were to start over in the near future.</span></div><div><span style="font-family: arial;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: arial;">Tracking Useful Metrics</span></h3><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">One of my first plans for the cluster will be to add metric tracking. I'm not sure what metric tracking options there are, so I searched to find out what other people are using. I found a number of references to <a href="https://github.com/carlosedp/cluster-monitoring" target="_blank">this cluster-monitoring repo</a>, and it looks like the setup for k3s is very simple. I forked and cloned that repo, followed the quick start info for k3s (</span><span style="font-family: arial;">updated some configs), and Prometheus, AlertManager, and Grafana were soon available and showing some useful metrics!</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">I'll create a post about what I learn with monitoring in the near future. </span></div><div><span style="font-family: arial;"><br /></span></div><h4 style="text-align: left;"><span style="font-family: arial;">Recommendations:</span></h4><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">Following the Quickstart for K3s from the carlosedp/cluster-monitoring repo was very straightforward, and it would be my suggested monitoring choice to anyone setting up a Raspberry Pi based Kubernetes cluster. I might change my mind, but for now it seems like an easy and quick way to go.</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">For now I suggest forking the repo, and then push any changes you make to vars.jsonnet to your cluster-monitoring repo. That way you can add/remove monitoring for your cluster quickly using a script that clones your repo first.</span></div><div><span style="font-family: arial;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: arial;">Near Future Projects</span></h3><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">There are a number of things that I want to explore at home:</span></div><div><ul style="text-align: left;"><li><span style="font-family: arial;">Serverless options for Kubernetes (for example, <a href="https://www.openfaas.com/" target="_blank">OpenFaaS</a>, <a href="https://knative.dev/" target="_blank">KNative</a>, and <a href="https://kubeless.io/" target="_blank">Kubeless</a>)</span></li><li><span style="font-family: arial;">Load testing at a small scale (for example, run load tests against a single service as part of a CI pipeline, or run a set of load tests against a smaller version of your production environment)</span></li><li><span style="font-family: arial;">Explore a variety of data pipelines</span></li></ul><div><span style="font-family: arial;"><br /></span></div></div><div><span style="font-family: arial;"><br /></span></div><div><br /></div><div><br /></div>
</div>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-43311324506170641082019-12-21T15:24:00.000-08:002019-12-21T15:27:44.838-08:00Taco Cat Goat Cheese Pizza - A great family game!<span style="font-family: "verdana" , sans-serif;"><b>Taco Cat Goat Cheese Pizza!</b></span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif; font-weight: normal;">Now say that 5 times really fast. It's hard to say. Now imagine having to say "Taco" "Cat" "Goat" "Cheese" "Pizza" in turn while laying down pictures of things that don't match the words. Then imagine having to slap your hand down on the card if the word you say matches the card. It's really difficult but fun!</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif; font-weight: normal;">Then add narwhals, groundhogs, and gorillas to the mix.</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif; font-weight: normal;">That's what you get with the game <a href="https://amzn.to/2sSDtx4" target="_blank">Taco Cat Goat Cheese Pizza</a>. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<br />
<div style="text-align: center;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.amazon.com/Taco-Cat-Goat-Cheese-Pizza/dp/B077Z1R28P/ref=as_li_ss_il?crid=XRYWY224F35&keywords=taco+cat+goat+cheese+pizza&qid=1576969474&sprefix=taco,aps,231&sr=8-2&linkCode=li2&tag=codecogniti0b-20&linkId=04c7e5158bf5792fb50b502d97f59a04&language=en_US" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" target="_blank"><img border="0" height="200" src="//ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=B077Z1R28P&Format=_SL160_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=codecogniti0b-20&language=en_US" width="155" /></a></div>
<div style="text-align: left;">
<span style="font-family: "verdana" , sans-serif;"><br /></span></div>
</div>
<span style="font-family: "verdana" , sans-serif;"><span style="font-family: "verdana" , sans-serif; font-size: small;"><span style="font-weight: normal;">Fun and sore hands. And lots of laughter!</span></span>
</span><br />
<div>
<span style="font-family: "verdana" , sans-serif; font-size: small;"><span style="font-family: "verdana" , sans-serif; font-weight: normal;">I bought this game a while ago but we only recently played it. It was surprising how much fun we had within the first round of playing it. We can hardly wait to play it again!<br /><br />The suggested age is 8 years and up, but if you have a child that can read fairly well, then it is probably fine for younger ages. It was definitely no problem for our 7-year-old daughter. She did much better than I did!</span></span></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-46606742913272596992019-06-26T07:03:00.002-07:002021-02-13T13:40:33.790-08:00Install Oh-my-zsh and powerline fonts on Ubuntu 18.04<span face=""verdana" , sans-serif">I recently installed Ubuntu 18.04 on my X1 Carbon (1st Gen that sat under the bed for years), and I'm</span><span face=""verdana" , sans-serif"> actually enjoying using this notebook again!</span><br />
<span face=""verdana" , sans-serif"><br /></span>
<span face=""verdana" , sans-serif">The first thing I did was install a few basics that included oh-my-zsh. I love the information that the prompt displays for your git repos. Shown below is oh-my-zsh using the</span><span face=""verdana" , sans-serif"> </span><span style="color: #073763; font-family: "courier new" , "courier" , monospace;">agnoster</span><span face=""verdana" , sans-serif"> theme. </span><br />
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKgyJWbE_gGhWj1SJr_Ny10C9MAMD4t4n7WOeefuSuZAoYEknn-oqb7xjqEA4s1q18gwU-J1yZnrxkj7f7-_veq2vqulvkfz4N7hTr978II7F4FM6jXtKdkjZqbseKt8rLGeJ_BxSOSTU/s1600/Screenshot+from+2019-06-26+06-33-44.png" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" data-original-height="470" data-original-width="887" height="211" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKgyJWbE_gGhWj1SJr_Ny10C9MAMD4t4n7WOeefuSuZAoYEknn-oqb7xjqEA4s1q18gwU-J1yZnrxkj7f7-_veq2vqulvkfz4N7hTr978II7F4FM6jXtKdkjZqbseKt8rLGeJ_BxSOSTU/s400/Screenshot+from+2019-06-26+06-33-44.png" title="Oh-My-Zsh with Git Repo Status" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span face=""verdana" , sans-serif"><i><b>Oh-My-Zsh with Git Repo Status</b></i></span></td></tr>
</tbody></table>
<br /></div>
<div>
<span face=""verdana" , sans-serif">However, if the powerline fonts aren't installed, then it doesn't look so great. The icons show up as boxes with X's in them. </span></div>
<div>
<br />
<span face=""verdana" , sans-serif">I didn't have the powerline fonts installed, so I searched for the correct way to install the fonts on Ubuntu and found that a bunch of people were having difficulties.</span><br />
<span face=""verdana" , sans-serif"><br /></span>
<span face=""verdana" , sans-serif">I ended up following the directions on the powerline font github repo's README, and it worked without too much effort, so I figured I would post all the steps I followed to get oh-my-zsh installed and configured the way I like it.</span><br />
<br />
<span face=""verdana" , sans-serif"><b><u>Oh-My-Zsh and Powerline Font Install</u></b></span><br />
<span face=""verdana" , sans-serif"><br /></span>
<span face=""verdana" , sans-serif">First, install oh-my-zsh.</span><br />
<blockquote class="tr_bq">
<span style="font-size: normal;"><span style="font-family: "courier new" , "courier" , monospace;">sh -c </span><span class="pl-s" style="box-sizing: border-box; color: #032f62; font-family: "courier new" , "courier" , monospace;"><span class="pl-pds" style="box-sizing: border-box;">"</span><span class="pl-s" style="box-sizing: border-box;"><span class="pl-pds" style="box-sizing: border-box;">$(</span>curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh<span class="pl-pds" style="box-sizing: border-box;">)</span></span><span class="pl-pds" style="box-sizing: border-box;">"</span></span></span></blockquote>
</div>
<span face=""verdana" , sans-serif"><br /></span>
<span face=""verdana" , sans-serif">After the install, you end up with a .zshrc file in your home directory. I updated the .zshrc file to use the agnoster theme instead of the default theme of robbyrussell. Just update the ZSH_THEME value.</span><br />
<span face=""verdana" , sans-serif"></span>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">ZSH_THEME="agnoster"</span></blockquote>
<br />
<span face=""verdana" , sans-serif">Second, install the powerline fonts so you can see the nice status icons for the current directory of your git repos. You can install the fonts this way:</span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt-get install fonts-powerline</span></blockquote>
<br />
<span face=""verdana" , sans-serif">Or by cloning the git repo and running their install script:</span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"></span>git clone https://github.com/powerline/fonts.git --depth=1</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">cd fonts</span></span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">./install.sh</span></span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">cd ..</span></span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">rm -rf fonts</span></span></blockquote>
<br />
<span face=""verdana" , sans-serif">If after running those commands (which probably only needed to consist of the apt-get install), the prompt for zsh has not started showing the nice status icons and colorized branches, then you can update the fontconfig information by creating a file in this directory (create the directory if it doesn't exist):</span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">~/.config/fontconfig/conf.d</span></blockquote>
<span face=""verdana" , sans-serif"><br /></span>
<span face=""verdana" , sans-serif">Then copy <a href="https://github.com/powerline/fonts/blob/master/fontconfig/50-enable-terminess-powerline.conf" target="_blank">this file</a> to </span><span face=""verdana" , sans-serif">~/.config/fontconfig/conf.d.</span><br />
<br />
<span face=""verdana" , sans-serif">Followed by running the font config cache command, which will force the font config cache to be update (-f) and display status information (-v).</span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">fc-cache -vf</span> </blockquote>
<br />
<span face=""verdana" , sans-serif">It was after I ran the fc-cache command that I noticed the terminal show the git repo status information with the branch and status icons. I used both the apt-get install fonts-powerline method, and the </span><br />
<br />Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-36304781614346161872019-01-27T23:50:00.002-08:002019-01-27T23:50:53.355-08:00Yeoman generator for creating a terraform directory structure for AWS providers...<span style="font-family: Arial, Helvetica, sans-serif;">I use AWS for work, and use terraform for creating the resources. My team uses a common directory structure for our terraform files, and it seems to work pretty well for separating resources between project groups, logical environments, and regions. However, c</span><span style="font-family: Arial, Helvetica, sans-serif;">reating new project directory structures can be a pain, so I decided to create a yeoman generator to automate the process. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Please check out </span><a href="https://github.com/leewallen/generator-tf-proj" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">the generator I made</a><span style="font-family: Arial, Helvetica, sans-serif;">, and let me know what you think!</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Clone from git:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">git clone https://github.com/leewallen/generator-tf-proj.git</span></blockquote>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">Install using npm:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">npm install --global generator-tf-proj</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Generate a terraform project structure using yeoman:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">yo tf-proj</span></blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-70314839104170074602018-01-20T16:47:00.002-08:002018-04-04T22:03:38.496-07:00AWS IoT Button and Philips Hue API...<span style="font-family: "verdana" , sans-serif;">I bought some Philips Hue Lights, and have really enjoyed them - but I enjoy them even more now that I have the IoT button integrated with the lights. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><a href="https://youtu.be/8k9T2Q8ara8" target="_blank">Here is a video</a> showing my AWS IoT button interacting with my Philips Hue Go lamp.</span><br />
<br />
<iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/8k9T2Q8ara8" width="560"></iframe><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Philips Hue API:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">The <a href="https://www.developers.meethue.com/philips-hue-api" target="_blank">Philips Hue REST API</a> is really easy to use to retrieve information about the lights connected to the Hue bridge, and for controlling the lights. You can <a href="https://www.developers.meethue.com/documentation/getting-started" target="_blank">follow the instructions on this page to help you get up and running with the API</a>.</span><br />
<br />
<span style="font-family: "verdana" , sans-serif;">AWS IoT Button:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">I had seen the AWS IoT button on Amazon and, although I didn't have any ideas of what I would do with the button, I wanted to work on a project which would use one of the buttons. I found <a href="https://youtu.be/rb7S0_T2kvM" target="_blank">this fun project</a> that also uses an AWS IoT button, and the Philips Hue API with the Go lamp. </span><span style="font-family: "verdana" , sans-serif;">I had bought a Philips Hue Go light, as well as a number of other Philips Hue lights, so I decided to recreate the project from the youtube video above but using an AWS lambda instead of using a <a href="http://amzn.to/2DYMaaZ" target="_blank">raspberry pi</a>. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<blockquote class="tr_bq">
<span style="font-family: "verdana" , sans-serif;">Something that was pointed out to me (embarrassingly) is that this method is not secure. Sending unencrypted information to the Hue bridge, which includes the auth, would allow an attacker to send their own API calls to the bridge. One of the API calls could have a security hole that could be used by an attacker / curious person. </span></blockquote>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">A couple of ideas I've had for using the Philips Hue lights are flash lights with certain colors to indicate either a rise above, or drop below, stock or crypto currency price points, and flash lights when people are close to home (integrate with IFTTT).</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">However, using the IoT button to control the lights looked fun and gave me an excuse to learn a little bit about AWS Lambdas. It's worth mentioning that Philips <a href="http://amzn.to/2DmPBqN" target="_blank">makes a switch</a> that can be easily programmed to control your Philips Hue lights.</span><br />
<br />
<span style="font-family: "verdana" , sans-serif;">Parts list :</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">- <a href="http://amzn.to/2Djbrv5" target="_blank">Philips Hue Go light</a>, but it works with all of the Philips Hue color lights</span><br />
<span style="font-family: "verdana" , sans-serif;">- <a href="http://amzn.to/2DwZeq5" target="_blank">AWS IoT Button</a></span><br />
<br />
<span style="font-family: "verdana" , sans-serif;">Set up IoT Button:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">I used the <a href="https://docs.aws.amazon.com/iot/latest/developerguide/iot-gs.html" target="_blank">"Getting Started" guide</a> to set up the IoT button. It walks you through registering your device, creating and activating a device certificate, creating and attaching an IoT policy to the device certificate, attach the certificate to a "Thing" (the button), and configuring your IoT button to know how to connect to your WiFi.</span><br />
<div id="amzn-assoc-ad-bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d">
</div>
<script async="" src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d"></script>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">One of the last steps in the "Getting Started" guide is configuring and testing rules. The example has the IoT button pushes send an SNS message that gets sent as a text message to your phone. I decided to have the SNS message trigger a lambda, and use the lambda to send the REST calls to my Philips Hue bridge.</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">AWS Lambda:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Here is the AWS Lambda code that I used: <br /><script src="https://gist.github.com/leewallen/9ffa66fe313e938310c6c7f6ac4f2472.js"></script></span><br />
<br />
<span style="font-family: "verdana" , sans-serif;">I have my router configured to use Dynamic DNS, and then I have a port forwarding rule to forward to the Philips Hue bridge. The lambda figures out if the button click was a single click, a double click, or a long click. The double clicks will turn the light on and off, the single click will increment the hue to set the light to, and a long click will set the light to use the color loop effect.</span><br />
<br />
<span style="font-family: "verdana" , sans-serif;">I hope you find this post useful! Please leave links to any projects you feel like sharing using AWS IoT buttons and/or Philips Hue lights in the comment section below.</span><br />
<br />
<span style="font-family: "verdana" , sans-serif;"><iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon&region=US&placement=B01CD5VC92&asins=B01CD5VC92&linkId=66013cd08689e91f34c83ad7b3b6bd68&show_border=true&link_opens_in_new_window=true" style="height: 240px; width: 120px;"></iframe></span>
<span style="font-family: "verdana" , sans-serif;">
<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon&region=US&placement=B00UVHAC1O&asins=B00UVHAC1O&linkId=72c264e16bf01b423553d952e72da40b&show_border=true&link_opens_in_new_window=true" style="height: 240px; width: 120px;"></iframe>
<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon&region=US&placement=B01KW6YCIM&asins=B01KW6YCIM&linkId=77d3c2201a8cda5193cc72223f9c93ed&show_border=true&link_opens_in_new_window=true" style="height: 240px; width: 120px;"></iframe>
<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon&region=US&placement=B00LY42URM&asins=B00LY42URM&linkId=e363fc55c1c57ae837adf229daf1e902&show_border=true&link_opens_in_new_window=true" style="height: 240px; width: 120px;"></iframe>
</span>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-82925883204683506352017-08-06T18:51:00.001-07:002018-01-21T12:49:40.281-08:00Create an AWS Lambda using Java...<div class="p1">
Here's a quick walk through for creating an AWS lambda using Java. I happen to use IntelliJ with maven, but you can use whatever IDE and package management you prefer to use. You can find a similar walk-through in the <a href="http://docs.aws.amazon.com/toolkit-for-eclipse/v1/user-guide/lambda-tutorial.html" target="_blank">online AWS documentation</a> or in the <a href="http://amzn.to/2hBKIno" target="_blank">AWS Lambda In Action</a> book.</div>
<div class="p1">
<br /></div>
<div id="amzn-assoc-ad-bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d"></script>
<div class="p1">
1. Create an IAM role for the Lambda to use:</div>
<div class="p1">
</div>
<ul>
<li>Click the "Create new role" button.</li>
<li>In the "Select role type" section, Click the "Select" button for "AWS Lambda" from the "AWS Service Role" section.</li>
<li>Enter the policy name of "<b>AmazonS3FullAccess"</b>, click the check box, and click the "Next step" button.</li>
<li>Enter a name in the "Role name" text box (for this example, use "<b>hello-lambda-role</b>"), and enter a fitting description in the "Role description" text box. Click the "Create role" button.</li>
</ul>
<br />
<div class="p1">
2. Create an S3 bucket.</div>
<div class="p2">
<br /></div>
<div class="p1">
3. Create a Java project for your AWS Lambda code:</div>
<div class="p1">
</div>
<ul>
<li>Using IntelliJ, create a maven project using <b>maven-archetype-quickstart</b>.</li>
<li>Add the aws lambda core dependency to the project's pom file:</li>
</ul>
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><<b>dependency</b>><br /><span class="Apple-converted-space"> </span><<b>groupId</b>>com.amazonaws</<b>groupId</b>><br /><span class="Apple-converted-space"> </span><<b>artifactId</b>>aws-lambda-java-core</<b>artifactId</b>><br /><span class="Apple-converted-space"> </span><<b>version</b>>1.1.0</<b>version</b>><br /></<b>dependency</b>></span></blockquote>
<div class="p1">
</div>
<ul>
<li>Create a class called <b>HelloWorldLambda</b> that implements <b>RequestHandler<String, String></b>:</li>
</ul>
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">public class HelloWorldLambda implements RequestHandler<String, String> {<br /><span class="Apple-tab-span"> </span>@Override<br /><span class="Apple-tab-span"> </span><b>public </b>String handleRequest(String input, Context context) {<br /><span class="Apple-tab-span"> </span><span class="Apple-converted-space"> </span>String output = <b>"Hello, " </b>+ input + <b>"!"</b>;<br /><span class="Apple-tab-span"> </span><span class="Apple-converted-space"> </span><b>return </b>output;<br /><span class="Apple-tab-span"> </span>}<br />}</span></blockquote>
<div class="p1">
</div>
<ul>
<li>Build the project so that the jar is created setting the output jar name to be <b>HelloLambda.jar</b>.</li>
</ul>
<br />
<div class="p2">
<br /></div>
<div class="p1">
4. Create the lambda in the AWS console:</div>
<div class="p1">
</div>
<ul>
<li>Click on the "Get Started Now" button.</li>
<li>Click on the "Blank Function" item.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGwNibMGLiFvFxgrGE_857Uwm7Eh3JvzBL-tHZrbmXSHd6qfGwnZkR3ZrrSJOqo-zAjZdbq0zagCawmRRcIsmElPurvm1Moc8NTUoyZYANT6SLWfmM7nPX4ikpYR3oS4Q0IruCeJbsDeg/s1600/Lambda-BlankFunction.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="702" data-original-width="642" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGwNibMGLiFvFxgrGE_857Uwm7Eh3JvzBL-tHZrbmXSHd6qfGwnZkR3ZrrSJOqo-zAjZdbq0zagCawmRRcIsmElPurvm1Moc8NTUoyZYANT6SLWfmM7nPX4ikpYR3oS4Q0IruCeJbsDeg/s320/Lambda-BlankFunction.png" width="291" /></a></div>
<ul>
<li>On the "Configure triggers" page, click in the grey dashed square and then select "S3".</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9WtHE6mBqjB9YrvporN2t_5nsbvEq0Yd-v-EA7M0_zGZEYufGCzBzq0heDXOIICGh0M4HQPm5XP6ZtN5YbEaaRYBMmhQayUijFDwd_ncIS19211YxM6U9tjLDLp3T8FtBJ7d1gTArJy8/s1600/Lambda-SelectS3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="886" data-original-width="700" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9WtHE6mBqjB9YrvporN2t_5nsbvEq0Yd-v-EA7M0_zGZEYufGCzBzq0heDXOIICGh0M4HQPm5XP6ZtN5YbEaaRYBMmhQayUijFDwd_ncIS19211YxM6U9tjLDLp3T8FtBJ7d1gTArJy8/s320/Lambda-SelectS3.png" width="252" /></a></div>
<ul><ul>
<li>Select the bucket that you created in step 2.</li>
<li>Select the event type "Object Created (All)".</li>
<li>Click "Enable trigger".</li>
</ul>
</ul>
<ul>
<li>Click the "Next" button.<span class="Apple-converted-space"> </span></li>
<li>Enter a name for the lambda like "<b>hello-lambda</b>"</li>
<li>Select "Java 8" for the Runtime</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3IMleDMsqtjtAJNVnQnVjsslMt4JyFoiwx9eaWqVjKlXYpQ9AGhbYOLd9srpgHYoMKtSuiuBmBfaHXXphS9Ka6B4aAsm772WW7XBbdu4gMyX-rvJcEoVsx_BBf3l0Zth9aLFyCNo6mv0/s1600/Lambda-ConfigureFunction.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="457" data-original-width="1600" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3IMleDMsqtjtAJNVnQnVjsslMt4JyFoiwx9eaWqVjKlXYpQ9AGhbYOLd9srpgHYoMKtSuiuBmBfaHXXphS9Ka6B4aAsm772WW7XBbdu4gMyX-rvJcEoVsx_BBf3l0Zth9aLFyCNo6mv0/s640/Lambda-ConfigureFunction.png" width="640" /></a></div>
<div>
<br /></div>
<ul>
<li>Click on the "Upload" button and select your <b>HelloLambda.jar</b>.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgF5cpAkMbvYhMycz9B6_lgNxJpRg7LO6PplLhOUEusxKw1-nzv3WWiSVrSRcGpsRTXL133i_Gi2HaZjuJb7Fw1XXtTXnyrgiV51CqsJKIo1565pwA4R2UivUAgNdCrNL7Ht0MSeIjdWZc/s1600/Lambda-UploadJar.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="386" data-original-width="1600" height="153" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgF5cpAkMbvYhMycz9B6_lgNxJpRg7LO6PplLhOUEusxKw1-nzv3WWiSVrSRcGpsRTXL133i_Gi2HaZjuJb7Fw1XXtTXnyrgiV51CqsJKIo1565pwA4R2UivUAgNdCrNL7Ht0MSeIjdWZc/s640/Lambda-UploadJar.png" width="640" /></a></div>
<div>
<br /></div>
<ul>
<li>In the "Lambda function handler and role", enter the full package path to your HelloWorldLambda class.</li>
<li>Select "Choose an existing role" for the Role section.</li>
<li>Select the "<b>hello-lambda-role</b>" that you created in step 1.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrCx_pjVH437nq6-TDcjVGHx_ym3UQa0Ngcgxn0C2IZFEdWNaALywxeE4XDq6qQijOrIBqXIU2R0uD6N78Fxtlrshf6unui0Xe2DmkR5Bbh-DyMsQwqEVZ6RtpkMcpmo6gNgJm5QCSdtY/s1600/Lambda-FunctionHandler.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="411" data-original-width="1600" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrCx_pjVH437nq6-TDcjVGHx_ym3UQa0Ngcgxn0C2IZFEdWNaALywxeE4XDq6qQijOrIBqXIU2R0uD6N78Fxtlrshf6unui0Xe2DmkR5Bbh-DyMsQwqEVZ6RtpkMcpmo6gNgJm5QCSdtY/s640/Lambda-FunctionHandler.png" width="640" /></a></div>
<div>
<br /></div>
<ul>
<li>In the "Tags" section, enter the value "Name" for the key, and "<b>hello-lambda</b>" for the value.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBhsc3tgEmQDwFGfn3A1rVfDs5BCutHOGW9PNztX9T036hQwLiECWg__yoyfUJjB0svjpaF6p_xT-BEtMJe3DwEH8PdI7rzSlgFpjkRheO7v_chJQX26XEmn0ojOOu5YeAvJ3MVUMRyjQ/s1600/Lambda-Tags.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="301" data-original-width="1600" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBhsc3tgEmQDwFGfn3A1rVfDs5BCutHOGW9PNztX9T036hQwLiECWg__yoyfUJjB0svjpaF6p_xT-BEtMJe3DwEH8PdI7rzSlgFpjkRheO7v_chJQX26XEmn0ojOOu5YeAvJ3MVUMRyjQ/s640/Lambda-Tags.png" width="640" /></a></div>
<div>
<br /></div>
<ul>
<li>In the "Advanced settings", increase the memory to 512 MB. Leave the timeout at 15 seconds.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaM_eyem85ANRkZ3-Ha1AazH9q0SrCew86RzwTp5lj0AumRSaLUTdRYUI66KpcbHs3q0piMSCPD49x_J5a0MNMVZdU5UWJehhJhnGbFc5jonUDAoAJMSOCMBwSfaOirkCwZCK-osyN7MY/s1600/Lambda-AdvancedSettings.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="412" data-original-width="1600" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaM_eyem85ANRkZ3-Ha1AazH9q0SrCew86RzwTp5lj0AumRSaLUTdRYUI66KpcbHs3q0piMSCPD49x_J5a0MNMVZdU5UWJehhJhnGbFc5jonUDAoAJMSOCMBwSfaOirkCwZCK-osyN7MY/s640/Lambda-AdvancedSettings.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<ul>
<li>Click the "Create function" button.</li>
</ul>
<br />
<div class="p2">
<br /></div>
<div class="p2">
<br /></div>
<div class="p1">
5. Test the lambda!</div>
<div class="p2">
<br /></div>
<div class="p1">
* Go to "Functions" section of the AWS console's Lambda page.</div>
<div class="p1">
* Select the "<b>hello-lambda</b>" function by clicking the option button.</div>
<div class="p1">
* Click on the "Actions" drop down, and click on "Test function". The "Input test event" dialg will appear.</div>
<div class="p1">
* Enter the text "testing", and then click the "Save and test" button.</div>
<div class="p2">
<br /></div>
<div class="p1">
This will trigger the lambda function, and you'll see the output in the "Execution result" section.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq0pHBUIsbTRkJG2wFZ1CWRG7ttEiiRxlAjvSqMX3WcFOvf4ouSsrf005gDtQhBBYBOQ_U2nOyLIWhpCr-cbtW4ulM3oUE2WlX32md-1lq63QQjzga6iafO66TEWAg4aHbqoqBkWmGjiI/s1600/Lambda-Test-Results.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="273" data-original-width="1600" height="108" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq0pHBUIsbTRkJG2wFZ1CWRG7ttEiiRxlAjvSqMX3WcFOvf4ouSsrf005gDtQhBBYBOQ_U2nOyLIWhpCr-cbtW4ulM3oUE2WlX32md-1lq63QQjzga6iafO66TEWAg4aHbqoqBkWmGjiI/s640/Lambda-Test-Results.png" width="640" /></a></div>
<div class="p1">
<br /></div>
<div class="p2">
<br /></div>
<div class="p1">
6. Test the lambda with an S3 creation event:</div>
<div class="p2">
<br /></div>
<div class="p1">
Uploading a text file with a single line of text to your S3 bucket that you created in step 2 will trigger your lambda, and you can see that the lambda is invoked by using the following steps.</div>
<div class="p2">
<br /></div>
<div class="p1">
</div>
<ul>
<li>Go to the AWS Lambda console page, and select the "Functions" section.</li>
<li>Click on the "<b>hello-lambda</b>" function. This should take you to the details for your lambda. <span class="Apple-converted-space"> </span></li>
<li>Click on the "Monitoring" tab.<span class="Apple-converted-space"> </span></li>
</ul>
<br />
<div class="p2">
<br /></div>
<div class="p1">
You'll see that you have invocations for both the test run, and the S3 upload. My image shows invocations for multiple file uploads, and multiple tests.</div>
<div class="p1">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinLjAa7BLdW_UyrgUeQHcNLzf_oea8x4wwOQPclRKyMptdE05t1N_ddQKZ8u210OxU-XANvV8ULlx8aI8qFOsfVNSXW5I6p5_CtFfumjinAs7jo7QajqMMSEssRGPwfZja5GFCn67ZJk8/s1600/CloudWatch-metrics.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1021" data-original-width="1040" height="391" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinLjAa7BLdW_UyrgUeQHcNLzf_oea8x4wwOQPclRKyMptdE05t1N_ddQKZ8u210OxU-XANvV8ULlx8aI8qFOsfVNSXW5I6p5_CtFfumjinAs7jo7QajqMMSEssRGPwfZja5GFCn67ZJk8/s400/CloudWatch-metrics.png" width="400" /></a></div>
<div class="p1">
<br /></div>
<div class="p1">
Learn more about AWS Lambdas through AWS Lambda In Action.</div>
<div class="p1">
<br /></div>
<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon&region=US&placement=1617293717&asins=1617293717&linkId=fd6afe74711374989a3d84e573b7829f&show_border=true&link_opens_in_new_window=true" style="height: 240px; width: 120px;"></iframe>
<div class="p1">
<br />
<br /></div>
<div class="p2">
<br /></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Courier New'}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Courier New'; min-height: 14.0px}
span.Apple-tab-span {white-space:pre}
</style>
<div id="amzn-assoc-ad-a46a4d06-d248-48de-a5ce-d67b7584e2a5"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=a46a4d06-d248-48de-a5ce-d67b7584e2a5"></script>
<br />
<div class="p2">
<br /></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-91934619381941437082017-07-26T23:28:00.000-07:002018-01-21T12:49:14.109-08:00AWS IAM Users and MFA<span style="font-family: "verdana" , sans-serif;">AWS Identity and Access Management (IAM) Users and Multi-Factor Authentication (MFA)</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Amazon Web Services are easy and incredibly fun to use. Need to spin up a web server and Redis cluster? No problem! </span><span style="font-family: verdana, sans-serif;">But how do you protect the AWS account from unauthorized use? Well, IAM users and MFA of course!</span><br />
<div id="amzn-assoc-ad-bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=bdf00f7f-ffb1-4466-bdc3-a348a6b9cc0d"></script>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">The <a href="http://amzn.to/2ePi3tV" target="_blank">AWS Certified Solutions Architect exam guide</a> covers IAM users and groups, as well as enabling MFA for your IAM user accounts, in Chapter 6.</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">The exercises at the end of the chapter have you create an IAM group, an IAM user, and then enable MFA for your newly created IAM user (in exercise 6.6). I've really enjoyed going through the exam guide specifically due to the chapter review quizzes (answers with explanations are in the back of the book) and the exercises. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Here are the steps that I used for creating an IAM group and user (using exercises 6.1 and 6.3 as the motivator, and following along in the very easy to use AWS console interface).</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Creating an IAM Group:</span><br />
<ul>
<li><span style="font-family: "verdana" , sans-serif;">Go to the IAM service in the AWS console.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Groups" console item.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Create New Group" button to start the group creation wizard.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Enter your group name in the "Group Name' text box and then click "Next Step". I chose "Administrators" as the AWS exam guide suggested.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">In the Attach Policy step, the exam book tells you to click the "IAMFullAccess" policy check box. The "IAMFullAccess" policy gives the group members full access to IAM via the AWS Management Console. The AWS online documentation for creating your first user and group has you select the "AdministratorAccess" policy - which will give you full access to AWS services and resources. I chose the "AdministratorAccess" policy.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">The last step is to review your proposed settings. Click the "Create Group" button. You'll be returned to the "Groups" list view, and you'll see your new group.</span></li>
</ul>
<br />
<span style="font-family: "verdana" , sans-serif;">Creating an IAM User:</span><br />
<ul>
<li><span style="font-family: "verdana" , sans-serif;">Go to the IAM service in the AWS console.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Users" console item.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Add user" button to start the user creation wizard.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Enter a user name in the "User name" text box.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">In the "Select AWS access type" section, click the "AWS Management Console access" check box. This will cause the "Console password" options to appear.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Select the "Custom password" option, and enter a password. </span></li>
<li><span style="font-family: "verdana" , sans-serif;">The "Require password reset" check box is checked by default. If you are creating a user for someone else to use, then it is a good idea to keep this option checked.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Next: Permissions" button.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">On the "Permissions" step of the wizard, click the "Add user to group" image if it is not already highlighted (this is the default selection).</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Check the checkbox for the group you created above.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Next: Review" button.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Create user" button. You'll be taken to "Success" page where you can see the user listed. It will contain a signin link that includes your AWS user ID as part of the url. ie, https://123456789012.signin.aws.amazon.com/console. You'll also be able to download the user credentials via a download button. The success page mentions that you can create new credentials at any time. The credentials file lists the user name and the signin link. </span></li>
</ul>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Enable MFA for an IAM user:</span><br />
<ul>
<li><span style="font-family: "verdana" , sans-serif;">Go to the IAM service in the AWS console.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Users" console item.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click on the user name for the user you would like to enable MFA.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click on the "Security credentials" tab.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click on the edit icon for "Assigned MFA device".</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Choose "A virtual MFA device" in the "Manage MFA Device" pop up dialog, and then click the "Next Step" button.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">You're instructed to install an AWS MFA-compatible application on the device of your choice - PC, smartphone, etc. There is a link in the dialog that will take you to a list of MFA-compatible applications. Install one of the compatible applications. I used the smart phone option, and installed the Google Authenticator application.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">Click the "Next Step" button. </span></li>
<li><span style="font-family: "verdana" , sans-serif;">A QR code is displayed in the AWS "Manage MFA Device" pop up dialog, and you are instructed to use your smart phone to scan the code. </span></li>
<li><span style="font-family: "verdana" , sans-serif;">If you're using the Google Authenticator, then a 6 digit code is displayed on your device, and is refreshed every 30 seconds.</span></li>
<li><span style="font-family: "verdana" , sans-serif;">You're instructed to enter two sets of the 6 digit codes, and then told to click "Activate Virtual MFA"</span></li>
</ul>
<span style="font-family: "verdana" , sans-serif;">At this point the user account is configured for MFA. The next time that user logs in they will be prompted to enter a 6 digit MFA code. Your MFA enabled user account is now a lot more secure than it was. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">I highly recommend the exam guide even though it is starting to get a bit dated. The book gives you a condensed and comprehensive look - and the exercises really help drive home the material. I found that some of the exercises were a bit sparse in information, and no longer match what the AWS console shows you, but it is close enough that you can figure things out without getting lost. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">The experience was very fun, and the end result is that I now have a much more secure admin account!</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<br />
<a href="https://www.amazon.com/Certified-Solutions-Architect-Official-Study/dp/1119138558/ref=as_li_ss_il?ie=UTF8&qid=1501185607&sr=8-1&keywords=aws+certified+solutions+architect&linkCode=li2&tag=codecogniti0b-20&linkId=8b3052b5f0e52690d2089c0b804e556c" target="_blank"><img border="0" src="//ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=1119138558&Format=_SL160_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=codecogniti0b-20" ></a><img src="https://ir-na.amazon-adsystem.com/e/ir?t=codecogniti0b-20&l=li2&o=1&a=1119138558" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-61160795576567866162017-05-25T15:12:00.001-07:002017-06-27T08:52:52.569-07:00Some books that I've really enjoyed...<span style="font-family: "verdana" , sans-serif;">A list of books that I've enjoyed:</span><br />
<br />
<br />
<ul>
<li><span style="font-family: "verdana" , sans-serif;"><a href="http://amzn.to/2qkXnuD" target="_blank">Java 8 in Action : Lambdas, Streams, and Functional-style Programming</a></span><span style="font-family: "verdana" , sans-serif;"> </span></li>
<li><span style="font-family: "verdana" , sans-serif;"><a href="http://amzn.to/2qldTe8" target="_blank">AWS Certified Solutions Architect Official Study Guide: Associate Exam 1st Edition</a></span></li>
<li><a href="http://amzn.to/2r17wxT" style="font-family: Verdana, sans-serif;" target="_blank">Head First Object Oriented Design and Analysis</a><span style="font-family: "verdana" , sans-serif;"> </span></li>
<li><a href="http://amzn.to/2rVHYly" style="font-family: Verdana, sans-serif;" target="_blank">Algorithms (4th Edition)</a> - <span style="font-family: "verdana" , sans-serif;">This book is used in conjunction with Coursera's <a href="https://www.coursera.org/learn/algorithms-part1">Algorithms 1 course</a></span></li>
<li><a href="http://amzn.to/2r2jVUf" style="font-family: Verdana, sans-serif;" target="_blank">Cracking the Coding Interview: 189 Programming Questions and Solutions 6th Edition</a></li>
<li><span style="font-family: "verdana" , sans-serif;"><a href="http://amzn.to/2rVxoLt" target="_blank">Solr in Action 1st Edition</a></span></li>
</ul>
<br />
<br />Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-56860943076326240292017-05-18T22:18:00.002-07:002017-05-18T22:31:24.164-07:00Refactoring Java code using lambdas - Stream.filter<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">Refactoring Java Code Using Lambdas</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">I’ve been going through a book, and absolutely loving it - <a href="http://amzn.to/2pZeToa" target="_blank">Java 8 in Action: Lambdas, Streams, and functional-style programming</a>.</span></div>
<br/>
<div class="p1">
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">The thing that I’ve enjoyed the most so far are the notes on when to use certain methods. In the chapter on streams, chapter 3, there is one particular method that really stuck out for me - the filter method. </span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">The filter method takes a predicate, and returns a stream of all the elements that match the predicate.</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">The book points out the following:</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">“Any time you’re looping over some data and checking each element, you might want to think about using the new filter method on Stream.”</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">Here is an example - the following code will print out even numbers:</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">List<Integer> numbers = Arrays.asList(1,2,3,4,5,6,7,8);</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">for (int num : numbers) {</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> if (num % 2 == 0) {</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> System.out.println(num);</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> }</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> }</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">The code above is pretty straight forward, but it could be written like this instead:</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">List<Integer> numbers = Arrays.asList(1,2,3,4,5,6,7,8);</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">numbers.stream()</span><br />
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> .filter(n -> n % 2 == 0)</span><br />
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> .forEach(System.out::println);</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">This might not look like a huge advantage, because there isn’t a lot different between the two. The amount of code is basically the same as well. However, the benefit of the lambda version is that it can be chained together with other Stream methods.</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">For example, imagine that you have some data containing user IDs, and some user IDs have a special prefix to indicate a special user type. You might want to filter out the special users, and then return a list of users without the prefix and with the name in upper case letters.</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">List<String> userIds = Arrays.asList(“*alice”,”bob”,”*", "charlie","*dana","evelyn","*frank");</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">return userIds.stream()</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span"></span> .filter(u -> u.startsWith(“*”) && u.length() > 1)</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span"></span> .map(u -> u.substring(1).toUpperCase())</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span"></span> .collect(Collectors.toList());</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">To do the same thing without using streams would look something like this:</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">List<String> userIds = Arrays.asList("*alice","bob","", "*", "charlie","*dana","evelyn","*frank");</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">List<String> specialUsers = new ArrayList<>();</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">for (String user : userIds) {</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> if (user.startsWith("*") && user.length() > 1) {</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> specialUsers.add(user.substring(1).toUpperCase());</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;"> }</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">}</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace;">return specialUsers;</span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">You can see that it would require that another variable would have to be declared to hold the special users. Kind of a waste. </span></div>
<div class="p2">
<span style="font-family: "verdana" , sans-serif;"><span class="s1"></span><br /></span></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Helvetica; color: #000000; -webkit-text-stroke: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Helvetica; color: #000000; -webkit-text-stroke: #000000; min-height: 13.0px}
span.s1 {font-kerning: none}
span.Apple-tab-span {white-space:pre}
</style>
<br />
<div class="p1">
<span class="s1" style="font-family: "verdana" , sans-serif;">I’ll definitely be keeping my eyes open for code that is iterating over lists and inspecting each element!</span></div>
</div>
<br/>
<iframe style="width:120px;height:240px;" marginwidth="0" marginheight="0" scrolling="no" frameborder="0" src="//ws-na.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=US&source=ss&ref=as_ss_li_til&ad_type=product_link&tracking_id=codecogniti0b-20&marketplace=amazon®ion=US&placement=1617291994&asins=1617291994&linkId=cd3b4c3e8140a3b496567580cbfdd2cb&show_border=false&link_opens_in_new_window=true"></iframe>
<br/>
<br/>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-37722735162810576332016-03-07T14:23:00.002-08:002016-03-09T10:19:36.324-08:00Cleaning up after aws cli on Mac OSX...<span style="font-family: "verdana" , sans-serif;">I've installed the aws command line on my Mac. It's super handy. However, the aws s3 command creates $folder$ files for every "directory" when a recursive copy is performed. It's super annoying. </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">For example, you could have a "directory" in S3 named "myfiles". When you download the objects with "myfiles" in the path y</span><span style="font-family: verdana, sans-serif;">ou will end up with a file named "myfiles_$folder$".</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Running aws --version returns this info:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"> aws-cli/1.10.6 Python/2.7.10 Darwin/14.5.0 botocore/1.3.28</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">I haven't found anything that explains how I can prevent those files from being created, so I've been doing manual cleanup afterwards. This is the command I run:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"> > rm $(find . "*$folder$")</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-18060506501511425042016-01-12T13:34:00.001-08:002016-01-12T13:34:39.243-08:00Debugging a local Spark job using IntelliJ<span style="font-family: Verdana, sans-serif;">A coworker was working on a local Spark job and shared how he set up his environment for debugging the job (which is basically the same as debugging any other remote process). </span><span style="font-family: Verdana, sans-serif;">These are the instructions I followed:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">1. Create a remote debug configurations.</span><br />
<br />
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Go to IntelliJ's "Run | Edit Configurations" screen</span><br />
<span style="font-family: Verdana, sans-serif;">Click on the "+" to "Add New Configuration"</span><br />
<span style="font-family: Verdana, sans-serif;">Select "Remote"</span></div>
<br />
<br />
<span style="font-family: Verdana, sans-serif;">2. Copy the command line argument to use and modify it however you see fit.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<br />
<div>
<span style="font-family: Verdana, sans-serif;">I'm using Java 8, so I used the example command line arguments from the top edit box. The only change I made was to set "suspend=y" so the spark job would stop and wait for me to start my "Remote Debug" process.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">This is what I used: </span></div>
<div>
<span style="font-family: Verdana, sans-serif;">-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">3. Export the command line arg as SPARK_JAVA_OPTS (Spark uses this value when you submit a spark job).</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">I set the SPARK_JAVA_OPTS like this:</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">export SPARK_JAVA_OPTS=</span><span style="font-family: Verdana, sans-serif;">-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">4. Start the spark job.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">You should see your spark job start up, and then pause with the following line printed on the console:</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">Listening for transport dt_scoket at address: 5005</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;">5. In IntelliJ, create whatever breakpoints you want to use and start the remote debug configuration. </span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-31591905603884934492015-08-27T23:02:00.001-07:002015-08-28T12:33:25.517-07:00Simple gremlin queries in Titan...<span style="font-family: Verdana, sans-serif;">Titan is an open source graph database and, even though it isn't as easy to setup for use as Neo4J, it is easy enough to start using in just a few minutes. Here is an example of using Titan with HBase as the backend storage. </span><br />
<br />
<u style="font-family: Verdana, sans-serif;">Setup</u><br />
<br />
<span style="font-family: Verdana, sans-serif;">I'm using HBase 0.98.13. I </span><a href="http://mirrors.sonic.net/apache/hbase/0.98.13/hbase-0.98.13-hadoop2-bin.tar.gz" style="font-family: Verdana, sans-serif;" target="_blank">downloaded hbase</a><span style="font-family: Verdana, sans-serif;">, untarred the files, changed to the hbase directory and ran "bin/start-hbase.sh". </span><br />
<br />
<span style="font-family: Verdana, sans-serif;">I cloned Titan from the </span><a href="https://github.com/thinkaurelius/titan.git" style="font-family: Verdana, sans-serif;">github repository</a><span style="font-family: Verdana, sans-serif;"> and built it using the maven package command. </span><br />
<br />
<span style="font-family: Verdana, sans-serif;">git clone https://github.com/thinkaurelius/titan.git </span><br />
<span style="font-family: Verdana, sans-serif;">cd titan </span><br />
<span style="font-family: Verdana, sans-serif;">mvn clean package</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">I started gremlin using "bin/gremlin.sh".</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">I followed the </span><a href="https://github.com/thinkaurelius/titan/wiki/Using-HBase" style="font-family: Verdana, sans-serif;">Titan Hbase instructions</a><span style="font-family: Verdana, sans-serif;"> for initializing Titan for use with HBase within the gremlin console. You can also create a properties file that contains the same settings and load the settings from the gremlin console.</span><br />
<br />
<u style="font-family: Verdana, sans-serif;">Gremlin</u><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">conf = new BaseConfiguration(); <br />conf.setProperty("storage.backend","hbase"); <br />conf.setProperty("storage.hbase.table", "test") </span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;">g = TitanFactory.open(conf);</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Now let's add some vertices.</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">alice = g.addVertexWithLabel('human')<br />alice.setProperty('name', 'alice')<br />alice.setProperty('age',25)<br />bob = g.addVertexWithLabel('human')<br />bob.setProperty('name', 'bob')<br />bob.setProperty('age',21)<br />clark = g.addVertexWithLabel('human')<br />clark.setProperty('name', 'clark')<br />clark.setProperty('age',93)<br />darwin = g.addVertexWithLabel('human')<br />darwin.setProperty('name', 'darwin') <br />darwin.setProperty('age',206)<br />ernie = g.addVertexWithLabel('android') <br />ernie.setProperty('name', 'ernie')</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Let's list the vertices and their properties.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">g.V().map()</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>{name=ernie}</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>{name=alice, age=25}</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>{name=darwin, age=206}</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>{name=clark, age=93}</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>{name=bob, age=21}</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">And now let's make these humans be friends with each other.</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">alice.addEdge('friend', bob)<br />alice.addEdge('friend', darwin)<br />bob.addEdge('friend', alice)<br />bob.addEdge('friend', darwin)<br />clark.addEdge('friend', darwin)<br />darwin.addEdge('friend',alice)<br />darwin.addEdge('friend', bob)<br />darwin.addEdge('friend', clark)</span><br />
<br />
<br /></div>
<div>
<span style="font-family: Verdana, sans-serif;">Now let's remove ernie from the graph.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">g.V.has('name', 'ernie').remove()</span><br />
<span style="font-family: Courier New, Courier, monospace;">==>null</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Now we can see that ernie is gone</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">g.V.has('name', 'ernie').map()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">(no results displayed, just the gremlin prompt)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Let's add ernie back, but this time he's a human.</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">ernie = g.addVertexWithLabel('human')<br />ernie.setProperty('name', 'ernie')</span><br />
<br />
<br />
<span style="font-family: Verdana, sans-serif;">Let's try finding out who has friends</span></div>
<div>
<span style="font-family: Verdana, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">g.V().outE('friend').outV().name</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">==>darwin<br />==>darwin<br />==>darwin<br />==>alice<br />==>alice<br />==>bob<br />==>bob<br />==>clark</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Wait - what happened? We see an entry for every friend edge, which is exactly what our gremlin query was asking for, but that doesn't look very nice. </span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Let's try the dedup method.</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">g.V().outE('friend').outV().dedup().name</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">==>darwin<br />==>alice<br />==>bob<br />==>clark</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Ahh! That's more like it! But how else can we get that list?</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">g.V.filter{it.outE('friend').hasNext()}.toList()._().name </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">==>darwin<br />==>alice<br />==>bob<br />==>clark</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Nice! We have two ways to get a distinct list.</span></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-28627810338327895852015-08-14T00:01:00.003-07:002018-01-21T12:42:47.445-08:00Add and remove fields from Solr schema using Schema API...<span style="font-family: "verdana" , sans-serif;">Okay - I know what you're thinking. "How can I quickly update my Solr schema without having to go into the schema config file - perhaps using a rest API?" You can use the Solr Schema API, that's how! </span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">I recently noticed that there is a <a href="https://cwiki.apache.org/confluence/display/solr/Schema+API" target="_blank">Schema API in Solr 5.X</a> that can be used to update the Solr schema. You need to have the schemaFactory set to use the "ManagedIndexSchemaFactory", and have the mutable property set to true. If you want to stop allowing the schema from being updated via the API, then you can change the mutable property to false.</span><br />
<div id="amzn-assoc-ad-18527490-f393-41b1-bade-cfe0cc1c39a3"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=18527490-f393-41b1-bade-cfe0cc1c39a3"></script>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Here are a few of the things that you can do with the schema API:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">View the schema for a collection:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<a href="http://localhost:8983/solr/yourcollectionname/schema"><span style="font-family: "courier new" , "courier" , monospace;">http://</span><span style="font-family: "courier new" , "courier" , monospace;">localhost</span><span style="font-family: "courier new" , "courier" , monospace;">:8983/solr/yourcollectionname/schema</span></a><br />
<br />
<span style="font-family: "verdana" , sans-serif;">View all of the fields in the schema:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<a href="http://localhost:8983/solr/yourcollectionname/schema/fields"><span style="font-family: "courier new" , "courier" , monospace;">http://</span><span style="font-family: "courier new" , "courier" , monospace;">localhost</span><span style="font-family: "courier new" , "courier" , monospace;">:8983/solr/yourcollectionname/schema/fields</span></a><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Example output:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "responseHeader":{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "status":0,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "QTime":101</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "fields":[</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"_text_",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"text_general",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "multiValued":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":false},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"_version_",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"long",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":true},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"id",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"string",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "multiValued":false,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "required":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "uniqueKey":true},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"somefieldname",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"lowercase",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":true},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"title",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"strings"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">View a specific field in the schema:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<a href="http://localhost:8983/solr/yourcollectionname/schema/fields/somefieldname"><span style="font-family: "courier new" , "courier" , monospace;">http://</span><span style="font-family: "courier new" , "courier" , monospace;">localhost</span><span style="font-family: "courier new" , "courier" , monospace;">:8983/solr/yourcollectionname/schema/fields/somefieldname</span></a><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Example output:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "responseHeader":{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "status":0,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "QTime":1},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "field":{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"somefieldname",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"lowercase",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":false,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Now add a new field called "anotherfield" that is of type "text_en", stored, and indexed:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":{"name":"anotherfield","type":"text_en","stored":true,"indexed":true }}' http://localhost:8983/solr/yourcollectionname/schema</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Now let's see that the field exists:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<a href="http://localhost:8983/solr/yourcollectionname/schema/fields/anotherfield"><span style="font-family: "courier new" , "courier" , monospace;">http://</span><span style="font-family: "courier new" , "courier" , monospace;">localhost</span><span style="font-family: "courier new" , "courier" , monospace;">:8983/solr/yourcollectionname/schema/fields/anotherfield</span></a><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "responseHeader":</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "status":0,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "QTime":1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "field":</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "name":"anotherfield",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "type":"text_en",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "indexed":true,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "stored":true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">Now let's delete the field:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">curl -X POST -H 'Content-type:application/json' --data-binary '{"delete-field" : { "name":"anotherfield" }}' http://localhost:8983/solr/yourcollectionname/schema</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">And check to see that it is deleted:</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<a href="http://localhost:8983/solr/yourcollectionname/schema/fields/anotherfield"><span style="font-family: "courier new" , "courier" , monospace;">http://</span><span style="font-family: "courier new" , "courier" , monospace;">localhost</span><span style="font-family: "courier new" , "courier" , monospace;">:8983/solr/yourcollectionname/schema/fields/anotherfield</span></a><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "responseHeader":</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "status":404,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "QTime":2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "error":</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "msg":"Field 'anotherfield' not found.",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> "code":404</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span><br />
<span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">There are other actions that you can do using the Schema API. Here are a few of the things that you can do using the Schema API:</span><br />
<span style="font-family: "verdana" , sans-serif;">- replace a field</span><br />
<span style="font-family: "verdana" , sans-serif;">- add and remove dynamic field patterns</span><br />
<span style="font-family: "verdana" , sans-serif;">- view dynamic fields</span><br />
<span style="font-family: "verdana" , sans-serif;">- and and remove field types</span><br />
<span style="font-family: "verdana" , sans-serif;">- view field types </span><br />
<div>
<br /></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-26285372359325963542015-08-12T16:07:00.004-07:002015-08-12T16:07:54.029-07:00Kill process using a specific port...<span style="font-family: Verdana, sans-serif;">I've been debugging an app that will sometimes get into an unresponsive state. The app is using the play framework, and I've been starting activator with the -jvm-debug option (and using port 9999). Sometimes when I try to terminate the activator terminal the CTRL+C will be ignored. I used lsof -i :9999 to find the pid to kill so that I can make changes and restart activator.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">A quick search led me to <a href="http://askubuntu.com/a/346457" target="_blank">this posting on askubuntu.com</a>, and it shows that you can kill a list of pids returned by lsof by using this form:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">kill -9 $(lsof -ti :9999)</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The "t" tells lsof to print terse info, and the info that is printed is just the pid.</span><br />
<div>
<br /></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-52473071910196070642015-08-11T13:08:00.001-07:002015-08-11T13:30:59.409-07:00Check for processes across multiple machines...<span style="font-family: Verdana, sans-serif;">My group at work needed to know if certain processes were running on a set of machines, and we didn't want to have to manually discover that the processes were running or not. The processes are running on Linux machines, and we use Splunk for capturing environment data, so I created a simple script to do the check and write it to a text file that Splunk consumes.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The minor issues I had were:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">1. I needed to run the script as a specific user due to using ssh as that user to other machines.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Solution: <a href="http://stackoverflow.com/a/8476992" target="_blank">Create a crontab entry for that user</a>. ie, crontab -u myuser -e</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Then add the crontab entry:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">*/15 * * * * /opt/myscripts/checkForProcesses.sh</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">2. I wanted to check for multiple processes but not all in one command string. I created separate "check" strings for each unique process. The issue I had was that the check string was interpreted as mutliple variables. "ps x | grep stuff | grep -v grep" was treated as "ps", "x", etc.</span><br />
<br />
<span style="font-family: Verdana, sans-serif;">Solution: I passed the check string in as the last variable to the method. If the check string was the 3rd value being passed in, <a href="http://stackoverflow.com/a/12938535" target="_blank">then I used the value like this</a>: ${@:3}</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The @ mean get all values, and the 3 says to start at value 3.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Here is a short version of the script with all details stripped out:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">#!/bin/sh</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">SCRIPT_DIR=$(dirname $0)</span><br />
<span style="font-family: Courier New, Courier, monospace;">LOG_DIR=/opt/logs</span><br />
<span style="font-family: Courier New, Courier, monospace;">STATUS_FILE=/opt/logs/status/process_status.txt</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">PROC1_CHECK="ps x | grep myproc1 | grep -v grep | grep -v less | grep java"</span><br />
<span style="font-family: Courier New, Courier, monospace;">PROC2_CHECK="ps x | grep myproc2 | grep -v grep | grep -v less | grep java"</span><br />
<span style="font-family: Courier New, Courier, monospace;">WORKER_LIST=$(cat $SCRIPT_DIR/worker.list)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># get the date/time for Splunk</span><br />
<span style="font-family: Courier New, Courier, monospace;">DATE_VAL=`date`</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">rm -f $STATUS_FILE</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">checkStatus()</span><br />
<span style="font-family: Courier New, Courier, monospace;">{</span><br />
<span style="font-family: Courier New, Courier, monospace;"> echo "Checking $1 for $2."</span><br />
<span style="font-family: Courier New, Courier, monospace;"> STATUS=`ssh myuser@$1 ${@:3}`</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if [ "" == "$STATUS" ];</span><br />
<span style="font-family: Courier New, Courier, monospace;"> then</span><br />
<span style="font-family: Courier New, Courier, monospace;"> echo "$DATE_VAL : ERROR : $2 not running on $1." >> $STATUS_FILE</span><br />
<span style="font-family: Courier New, Courier, monospace;"> else</span><br />
<span style="font-family: Courier New, Courier, monospace;"> echo "$DATE_VAL : STATUS : $2 running on $1." >> $STATUS_FILE</span><br />
<span style="font-family: Courier New, Courier, monospace;"> fi</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">#worker.list is a set of machine names to check for certain processes</span><br />
<span style="font-family: Courier New, Courier, monospace;">for worker in $WORKER_LIST; do</span><br />
<span style="font-family: Courier New, Courier, monospace;"> echo "Worker being checked is $worker"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> checkStatus $worker "MyProc1" $PROC1_CHECK</span><br />
<span style="font-family: Courier New, Courier, monospace;"> checkStatus $worker "MyProc2" $PROC2_CHECK</span><br />
<span style="font-family: Courier New, Courier, monospace;">done</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-25992517233080766852015-08-05T22:58:00.000-07:002015-08-05T23:12:31.565-07:00Shell script fun - pull, compile, and display errors...<span style="font-family: Verdana, sans-serif;">I am working on a project at work that has multiple project dependencies, and people from other time zones constantly updating those projects, so it is helpful to stay in sync so I can avoid merge issues.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">I wrote a shell script to pull and build the projects. We are using maven for our projects, so a nice side effect is that the maven build outputs "SUCCESS" or "FAILURE" for the project build. I just grep for "FAILURE" (or "error"), pipe the result to a text file, and then check each text file to see if it is 0 bytes or not. If the file has data, then I print a build error message, cat the file, and then delete it. I output the failed file in red if the build failed.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Here is a simple version of the script:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
#!/bin/sh<br />
source ~/.bash_profile<br />
<br />
export WORK_HOME=~/dev/source<br />
<br />
red=`tput setaf 1`<br />
green=`tput setaf 2`<br />
reset=`tput sgr0`<br />
#echo "${red}red text ${green}green text${reset}"<br />
<br />
grep_file()<br />
{<br />
if [ -s "$WORK_HOME/$2" ]<br />
then<br />
echo "${red}******** $1 had errors!${reset}"<br />
echo ""<br />
echo "${red}"<br />
cat $WORK_HOME/$2<br />
echo "${reset}"<br />
else<br />
rm -f $WORK_HOME/$2<br />
fi<br />
}<br />
<br />
do_build()<br />
{<br />
echo ""<br />
echo "${green}Updating $2.${reset}"<br />
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="<br />
cd $WORK_HOME/$2<br />
git pull<br />
if [ "$1" == "mci" ]<br />
then<br />
mvn clean install 2>&1 | grep FAILURE | grep -iv failures | grep -v grep > $WORK_HOME/$3<br />
else<br />
mvn clean package 2>&1 | grep FAILURE | grep -iv failures | grep -v grep > $WORK_HOME/$3<br />
fi<br />
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="<br />
}<br />
<br />
do_activator_build()<br />
{<br />
echo ""<br />
echo "${green}Running activator build for $1.${reset}"<br />
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="<br />
cd $WORK_HOME/$1<br />
git pull<br />
activator update<br />
activator compile | grep -i error | grep -v grep > $WORK_HOME/$2<br />
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-="<br />
}<br />
<br />
# project 1<br />
do_build "mci" "project1" "proj1_failed.txt"<br />
<br />
# project 2<br />
do_activator_build "project2" "proj2_failed.txt"<br />
<br />
echo ""<br />
echo "Checking for failures..."<br />
echo ""<br />
<br />
grep_file "project1" "proj1_failed.txt"<br />
grep_file "project2" "proj2_failed.txt"<br />
echo "Done."<br />
<br />
<br />
<span style="font-family: Verdana, sans-serif;">Here is the output:</span><br />
<br />
<br />
<span style="color: #38761d;">Updating project1.</span><br />
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<br />
Already up-to-date.<br />
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<br />
<br />
<span style="color: #38761d;">Running activator build for project2.</span><br />
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<br />
Already up-to-date.<br />
[info] Loading project definition from /Users/user/dev/source/project2<br />
[info] Updating {file:/Users/user/dev/source/project2/}project2...<br />
[info] Resolving org.fusesource.jansi#jansi;1.4 ...<br />
[info] Done updating.<br />
[<span style="color: #38761d;">success</span>] Total time: 16 s, completed Aug 5, 2015 11:06:25 PM<br />
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<br />
<br />
Checking for failures...<br />
<br />
Done.Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-6440347072811180062015-07-23T14:20:00.002-07:002015-07-23T14:20:34.341-07:00One line shell command to kill processes that match a search string...<span style="font-family: Verdana, sans-serif;">I had a bunch of java processes running on a server, and I needed to kill all of them. It's easy enough to write a shell script to kill them, but I wanted to do it all in one command. It is common enough to do, so <a href="http://stackoverflow.com/a/21470413" target="_blank">I did a search on stackoverflow and found what I needed</a>:</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">for pid in `ps x | grep java | grep -v grep | awk '{print $1}'` ; do kill -9 $pid ; done</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The key is the command in the back ticks. The shell will execute that first, and use the results in the for loop - reading each result into the variable $pid.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-92056759428187335202015-06-14T16:22:00.000-07:002015-06-14T16:26:01.312-07:00Simple Neo4J example...<div style="font-family: 'Courier New'; font-size: 12px;">
I've been using a graph database at work named Titan (<a href="https://github.com/thinkaurelius/titan">https://github.com/thinkaurelius/titan</a>). It's open source, and has some pretty nice features such as the ability to choose between a number of different databases for storage (Cassandra DB, HBase, Berkeley DB, etc), and use Solr or Elastic Search for external indexing of data. However, I wanted to try out Neo4J for home project. Here are a few of the things that I learned.</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
1. Neo4J is incredibly easy to add to your project using Maven. Just add a dependency like this:</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span><<b>dependency</b>><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> <<b>groupId</b>>org.neo4j</<b>groupId</b>><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> <<b>artifactId</b>>neo4j</<b>artifactId</b>><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> <<b>version</b>>2.2.2</<b>version</b>><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span></<b>dependency</b>></div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
2. I used Neo4J as an embedded database like this:</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>GraphDatabaseService graphDb;<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>graphDb = <b>new </b>GraphDatabaseFactory().newEmbeddedDatabase( <b><i>DB_PATH </i></b>);</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
3. The Neo4J documentation recommended that you use try with resource:</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span><b>try </b>(Transaction tx = graphDb.beginTx()) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> <i>populateDb</i>(graphDb, someDataToAdd);<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> tx.success();<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>} <b>catch </b>(Exception e) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> e.printStackTrace();<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
4. Adding nodes and relationships is very straightforward:</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>// Add nodes using createNode, then </div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>// use setProperty for each property</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>someDataToAdd.forEach(d -> {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> Node node = graphDb.createNode();<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> node.setProperty(<b>"name"</b>, d.getName());<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> node.setProperty(<b>"email"</b>, d.getEmail());<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> node.setProperty(<b>"id"</b>, d.getId());<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span> node.addLabel(DynamicLabel.<i>label</i>("Person"));<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>});</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>// Add relationships by calling createRelationshipTo </div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>// on the source node.</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>// Also, search for nodes by calling findNodes</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>Node friend = graphDb.findNodes(</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>DynamicLabel.<i>label</i>(<b>"Person"</b>), <b>"id"</b>, id).next();</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>node.createRelationshipTo(friend, RelTypes.<b><i>FRIEND</i></b>);</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
5. Get all relationships and nodes by using the GlobalGraphOperations:</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>GlobalGraphOperations.<i>at</i>(graphDb)</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>.getAllRelationships().forEach(n -> n.delete());</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span>GlobalGraphOperations.<i>at</i>(graphDb)</div>
<div style="font-family: 'Courier New'; font-size: 12px;">
<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>.getAllNodes().forEach(n -> n.delete());</div>
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: 'Courier New'; font-size: 12px;">
It's possible to query the graph using cypher - I just found it to be easier and more intuitive to use the Java API.</div>
<br />
<div style="font-family: 'Courier New'; font-size: 12px; min-height: 14px;">
<br /></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-34363518344530301392015-04-20T21:57:00.002-07:002015-04-20T22:04:45.819-07:00Vagrant + VirtualBox = Awesome<span style="font-family: Arial, Helvetica, sans-serif;">Have you ever wanted to create a test environment for some some code, but you didn't have the time or patience to create VMWare or VirtualBox instances for your target OS? Let <a href="http://docs.vagrantup.com/v2/getting-started/index.html" target="_blank">Vagrant</a> do the tedious work for you!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I first used vagrant when going through <a href="https://www.udacity.com/" target="_blank">Udacity's</a> <a href="https://www.udacity.com/course/ud381" target="_blank">Apache Storm</a> course. The course has you install git bash, vagrant, and virtual box. It then has you clone a repo that includes a vagrantfile that points to their vagrant box for the course. Running "vagrant up" in the directory with the vagrantfile will cause vagrant to download the vagrant box (or image). You can run "vagrant up" again after the box is downloaded, and it will start the box using Virtual Box. You can "connect" to the box using "vagrant ssh".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I used to use AWS to spin up instances for testing code, but that can be kind of expensive. Now I can get similar virtual machines in about the same amount of time, but <a href="https://youtu.be/c0y9DkJcxH4?t=5s" target="_blank">for freeeeeeee!!!!</a> </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The Vagrant website's </span><span style="font-family: Arial, Helvetica, sans-serif;">"Getting Started" guide has you downloading and running boxes right away. The guide also points you to the site where you can discover other vagrant boxes that are free to download and use. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I've seen references to vagrant numerous times, but never bothered to check it out before going through the Udacity course. I can't wait to see what I can hit with my vagrant hammer!</span>Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0tag:blogger.com,1999:blog-4695905409600262169.post-20854748090976382322015-03-21T15:52:00.001-07:002018-01-21T12:43:03.083-08:00Using CursorMark for deep paging in Solr<span style="font-family: Verdana, sans-serif;">Have you ever had to search through 100 million XML blobs to identify which XML blobs had some specific data (or, in my case, was missing some data)? It takes forever, and it's not very fun. I had a situation at work where it looked like I was going to have to do just that. Fortunately, we have the same data (more or less) in Solr cores. I just needed to do a NOT query on a specific field.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The <a href="https://wiki.apache.org/solr/CommonQueryParameters#Deep_paging_with_cursorMark" target="_blank">Solr documentation</a> suggests that you use a CursorMark parameter in your query if you need to page through more than a thousand records. </span><br />
<div id="amzn-assoc-ad-18527490-f393-41b1-bade-cfe0cc1c39a3"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=18527490-f393-41b1-bade-cfe0cc1c39a3"></script>
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">The following is an example of using CursorMark using SolrJ. The query will return all documents that do not have data for targetField. It then loops through the matches using the cursorMark parameter as a way to tell solr where to retrieve the next set of matches. One of the conditions of using cursorMark to page through results is that you need to sort on a unique field. The schema we are using has a unique field named "uid". </span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Something to note is that I had to explicitly set "timeAllowed" to 0 to say that I didn't want any time restrictions on the query.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Verdana, sans-serif;">Yonik Seeley made an <a href="http://yonik.com/solr/paging-and-deep-paging/" target="_blank">interesting point</a> in his <a href="http://yonik.com/" target="_blank">Solr 'n Stuff blog</a> about using cursorMark. You can change the returned fields, the facet fields, and number of rows returned, for a specific cursorMark since the cursorMark contains the state of the current search - the state isn't stored server-side. This makes it very easy for a client to allow a user to vary their experience while they page through results. This feels very similar to how you page through results using <a href="http://dev.mysql.com/doc/refman/5.0/en/select.html" target="_blank">MySQL</a> and <a href="https://technet.microsoft.com/en-us/library/gg699618.aspx" target="_blank">other databases</a>.</span><br />
<span style="font-family: Verdana, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.client.solrj.SolrQuery;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.client.solrj.SolrServer;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.client.solrj.SolrServerException;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.client.solrj.impl.HttpSolrServer;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.client.solrj.response.QueryResponse;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.common.SolrDocument;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.common.SolrDocumentList;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import org.apache.solr.common.params.CursorMarkParams;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import java.io.*;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import java.util.zip.ZipEntry;</span><br />
<span style="font-family: Courier New, Courier, monospace;">import java.util.zip.ZipOutputStream;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">public class FindMissingData {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> private static final int MAX_ALLOWED_ROWS=1000;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private static final String URL_PREFIX="http://somesolrserver:8983/solr/thecore";</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private static final String ZIP_ENTRY_NAME="TheData.txt";</span><br />
<span style="font-family: Courier New, Courier, monospace;"> private static final String UNIQUE_ID_FIELD="uid";</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> private static final String SOLR_QUERY="NOT <b><span style="color: blue;">targetField</span></b>:[* TO *]";</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public static void main(String[] args) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> if (args.length != 2) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> System.out.println("java FindMissingData <rows per batch - max is 1000> <output zip file>");</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> int maxRows = Integer.parseInt(args[0]);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String outputFile = args[1];</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> // Delete zip file if it already exists - going to recreate it anyway</span><br />
<span style="font-family: Courier New, Courier, monospace;"> File f = new File(outputFile);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if(f.exists() && !f.isDirectory()) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> f.delete();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">FindMissingData</span><span style="font-family: Courier New, Courier, monospace;">.writeIdsForMissingData(outputFile, maxRows);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> public static void writeIdsForMissingData(String outputFile, int maxRowCount) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> if (maxRowCount > MAX_ALLOWED_ROWS) </span><br />
<span style="font-family: Courier New, Courier, monospace;"> maxRowCount = MAX_ALLOWED_ROWS;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> FileOutputStream fos = null;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ZipOutputStream zos = null;</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> try {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> fos = new FileOutputStream(outputFile, true);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos = new ZipOutputStream(fos);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos.setLevel(9);</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> queryForMissingData(maxRowCount, zos);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (Exception e) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> e.printStackTrace();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } finally {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (zos != null) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> try {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos.flush();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos.close();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (Exception e) {}</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (fos != null) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> try {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> fos.flush();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> fos.close();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (Exception e) {}</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> private static void queryForMissingData(int maxRowCount, ZipOutputStream zos) throws IOException {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> ZipEntry zipEntry = new ZipEntry(ZIP_ENTRY_NAME);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos.putNextEntry(zipEntry);</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<br />
<span style="font-family: Courier New, Courier, monospace;"> SolrServer server = new HttpSolrServer(URL_PREFIX);</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;"> SolrQuery q = new SolrQuery(SOLR_QUERY);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> q.setFields(UNIQUE_ID_FIELD);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> q.setRows(maxRowCount);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> <span style="color: blue;"><b>q.setSort(SolrQuery.SortClause.desc(UNIQUE_ID_FIELD));</b></span></span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> // You can't use "TimeAllowed" with "CursorMark"</span><br />
<span style="font-family: Courier New, Courier, monospace;"> // The documentation says "Values <= 0 mean </span><br />
<span style="font-family: Courier New, Courier, monospace;"> // no time restriction", so setting to 0.</span><br />
<span style="font-family: Courier New, Courier, monospace;"> q.<b><span style="color: blue;">setTimeAllowed</span></b>(0);</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> String <b><span style="color: blue;">cursorMark</span></b> = CursorMarkParams.CURSOR_MARK_START;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> boolean done = false;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> QueryResponse rsp = null;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span><br />
<span style="font-family: Courier New, Courier, monospace;"> while (!done) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> q.set(CursorMarkParams.CURSOR_MARK_PARAM, <b><span style="color: blue;">cursorMark</span></b>);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> try {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> rsp = server.query(q);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } catch (SolrServerException e) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> e.printStackTrace();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> writeOutUniqueIds(rsp, zos);</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> String <span style="color: blue;"><b>nextCursorMark </b></span>= rsp.<b><span style="color: blue;">getNextCursorMark</span></b>();</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (<span style="color: blue;"><b>cursorMark</b></span>.equals(<span style="color: blue;"><b>nextCursorMark</b></span>)) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> done = true;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } else {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> <b><span style="color: blue;">cursorMark </span></b>= <b><span style="color: blue;">nextCursorMark</span></b>;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> private static void writeOutUniqueIds(QueryResponse rsp, ZipOutputStream zos) throws IOException {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> SolrDocumentList docs = rsp.getResults();</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> for(SolrDocument doc : docs) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> zos.write(</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String.format("%s%n",</span><br />
<span style="font-family: Courier New, Courier, monospace;"> doc.get("uid").toString()).getBytes()</span><br />
<span style="font-family: Courier New, Courier, monospace;"> );</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
<div>
<br /></div>
Leehttp://www.blogger.com/profile/14314686654330054910noreply@blogger.com0