RPi CPU load and temperature logging to Cosm

Here is a 15-minute recipe to get your Raspberry Pi logging data to Cosm.com, who provide a RESTful API to query the data and produce customized charts.

I use two scripts that cron runs every 5 minutes, one to log data to a CSV file, and another to upload the data to cosm, then delete the CSV file. That way, if my internet connection goes down for a while, the logged data is not lost. I have a copy of these scripts in a separate directory for each variable (‘datastream’) that I monitor, which makes it easier to manage.

CPU load

First, create an account on Cosm.com (it’s free & quick).

Now set up a new feed (a feed is a collection of datastreams; each datastream is a series of timestamped data points, aka a ‘time series’)

  • “+ Device/Feed”
  • What type of device/feed do you want to add?: Choose ‘Something Else’
  • Step 1 – Data: Choose ‘No, I will push data to Cosm’
  • Step 2 – Title: think of a relevant name for your feed, e.g. “MyDesktopPi”
  • Step 3 – Tags: give it any relevant tags that might help you find it in future (this is only really useful for public feeds)
  • Step 4 – Create. Make a note of the feed ID, we’ll use this later.

Once you’ve created a feed, you can add datastreams to it. A datastream represents a value that your Pi will monitor over time, like temperature or CPU load (or internet connectivity, washing machine activity, presence of your mobile phone on your LAN, etc..)

  • “+ Datastream”
  • ID: This doesn’t need to be numeric – you can enter something like ‘Pi_CPU_Load’. Make a note of this datastream ID too.
  • Tags: e.g. ‘pi cpu load’
  • Units: ‘Capacity’ – this is free text
  • Symbol: leave blank

At this point, your feed is public, i.e. anyone can view the current data. This may be fine, but if you want to change it, scroll down to the ‘Feed Status’ section at the bottom of the page.

Now, at the bottom of the screen click the green ‘Save Changes’ button. (this is an area of the cosm UI that needs work, IMHO, as you expect to find this button near the data that you’re editing..)

You can get back to the edit feed / add datastream page at any time by clicking the little cog icon on the right hand side of the feed name and choosing ‘edit’.

Now that we’ve defined our feed and datastream, we need to give our script permission to upload data to our Cosm datastream. This is done by generating an API key.

  • In the top-right of the cosm web page, click the ‘Keys’ icon.
  • Click the ‘+ Key’ icon, give the key a label (ID), e.g. ‘MyDesktopPi_UploadScriptKey’, and choose feed restrictions:’Use any feed (including my private feeds)’ and access privileges:’all’, then click ‘create’.
  • Make a note of the long alphanumeric API key string, as we’ll use that in a moment.

The last thing to do is to create the scripts on the Pi that will upload data to cosm.

Log in as ‘pi’.

cd ~
mkdir cosm
cd cosm
mkdir load
cd load

Install CURL
sudo apt-get install curl

Using your favourite text editor, create the file ‘log.sh’:
#!/bin/bash
####################################################
# Please customize these values appropriately:
LOCATION=/home/pi/cosm/load
#VALUE=$( uptime | awk -v FS="[, ]" '{print $18}' )
# alternatively:
VALUE=$( cat /proc/loadavg | awk {'print $2'} )
####################################################
TIME=`/bin/date -u +%FT%XZ`
if [[ "$VALUE" == "" ]]
then
VALUE=0
fi
echo "$TIME,$VALUE" >> $LOCATION/cosm.csv
exit 0

.. and save it. The line cat /proc/loadavg | awk {'print $2'} simply takes the second number from the /proc/loadavg file, which represents the 5-minute-average of the CPU load.

Create a file called ‘upload-cosm.sh’:
#!/bin/bash
####################################################
# Please customize these values appropriately:
LOCATION=/home/pi/cosm/load
API_KEY='your-long-alphanumeric-api-key-here'
FEED_ID='your-feed-id-here'
DATASTREAM_ID='your-datastream-id-here'
####################################################
COSM_URL=http://api.cosm.com/v2/feeds/$FEED_ID/datastreams/$DATASTREAM_ID/datapoints.csv
sleep 2 # gives any data logging scheduled at the same time a chance to run
echo $COSM_URL
curl -v --request POST --header "X-ApiKey: $API_KEY" --data-binary @$LOCATION/cosm.csv $COSM_URL
if [ $? -eq 0 ]
then
rm $LOCATION/cosm.csv
#echo "Would delete file now."
fi
#echo "Done"

.. and save that too. Exit the text editor, and make all the .sh scripts executeable with:
chmod u+x *.sh

(I’ll assume that you’ve changed LOCATION, API_KEY, FEED_ID and DATASTREAM_ID appropriately for your system)

The log.sh script will append to a file called cosm.csv every time it is run. You can try it now if you like:
./log.sh

To schedule these scripts to run, we edit ‘crontab’, a file that tells cron which scripts to run, and when. My favourite editor is called ‘joe’, yours might be ‘vi’, ‘emacs’ or another. The first line makes sure that crontab will use your editor to edit the crontab file:

EDITOR=joe
crontab -e

Append these lines to the end of your crontab file (leaving a blank line at the end):

*/5 * * * * /home/pi/cosm/load/log.sh
*/5 * * * * /home/pi/cosm/load/upload-cosm.sh

*/5 means ‘run every 5 minutes’. See this reference for more details.

Save the crontab file and exit your editor. Now you can either wait for five minutes, or simply run the upload script with:
./upload-cosm.sh

With luck, you should now be able to reload your Cosm.com page and see the data uploaded to your datastream as a chart.

CPU temperature

Simply copy the ~/cosm/load directory:
cp -R ~/cosm/load ~/cosm/temperature
cd ~/cosm/temperature

Edit the log.sh script, and replace:

LOCATION=/home/pi/cosm/temperature
# VALUE is in degrees celcius
VALUE=$( cat /sys/class/thermal/thermal_zone0/temp | awk -v FS=" " '{print $1/1000""}' )

This takes the 1st value from /sys/class/thermal/thermal_zone0/temp (in fact there is only one number), and divides it by 1000 to get degrees C (the raw value is in thousandths of a degree).

Edit the upload-cosm.sh script with your temperature datastream ID (update the LOCATION too), and add to crontab:

* * * * * /home/pi/cosm/temperature/log.sh
*/5 * * * * /home/pi/cosm/temperature/upload-cosm.sh

(I’ve chosen to record the temperature every minute, but upload the values every 5 minutes)

Note the daily variation, even though this is monitoring CPU temperature! I guess the Pi is in a south-facing room which warms up during the day, but I didn’t expect to see this so clearly. The dropouts/spikes that you see in the data are caused by occasional erroneous values returned by the temperature sensor.

If you find this useful, please post a comment indicating the type of data that you’re monitoring (and maybe the line of script that captures the data?).

Thanks for reading!

Ubuntu 12.04 on my Eee 900

I’ve just installed Ubuntu 12.04 (desktop version) on my aging Asus Eee 900 laptop, and although it’s taking me a while to adjust to the new Unity interface, my greatest concerns after googling a bit (wifi and sdcard drivers not working, or sdcard only accepting up to 1GB) have been unfounded.

I downloaded the 1GB DVD ISO of Ubuntu to my Windows desktop machine, then used the free unetbootin tool to write the ISO to a SD card. For some reason Ubuntu wouldn’t install from my 16GB SD card on the first try. I just saw the five animated dots; pressing F11 showed a screen with lots of errors like

udevd[100]: timeout: killing '/sbin/blkid -o udev -p /dev/sdc' [252]

Strangely, I rebooted and on the second try everything worked perfectly.

A word of warning, don’t install Ubuntu onto the 4GB drive partition on the Eee, as Ubuntu will rapidly run out of space (doh!). I reinstalled Ubuntu, choosing the 16GB drive partition and all has been well since. (After a few days, the installation plus updates appears to take around 4.5GB.) At one point I had to change the boot order in the Eee’s bios (F2 on startup) to pick the 16GB drive, otherwise I was left with a black screen with flashing white cursor.

After a day or two, I have to admit Unity looks nice and with a few extra packages installed really works well. It even seems faster than the old Eeebuntu (based on Ubuntu 9.04). Java, IntelliJ IDEA and the Arduino IDE need very little help to get working, and it’s great having all the latest package updates available again. Well worth the evening it took me to upgrade.

Golf Bravo Uniform Indigo Kilo

Yesterday I finally used the Flight Experience voucher that my wife bought me for my birthday. However as it turned out there were a few hurdles to cross first.. after nearly two weeks of sunshine and perfect blue skies, the forecast for Tuesday was for heavy rain. I couldn’t believe my luck. Thankfully the Met office showed the rain coming in the afternoon & my flight was booked for 11am so we might just be ok. I got all technical and checked the METAR (aviation forecast) for Luton which looked something like this:

EGGW 030720Z 29008KT 9999 FEW031 07/05 Q1003

Thankfully there was also a translation:

wind 290 degrees 08 knots, visibility 10km or more, a few clouds at 3100ft, temperature 8'c, dewpoint 5'c and air pressure 1003mb.

.. which basically meant that we were still on for flying. Phew! – first hurdle.

Just after setting off for the airfield (Panshangar, a few miles from Hertford), with plenty of time to spare, we got a call to say our 18 month old daughter had been sick at nursery, please could we come and pick her up? Aaargh. Second hurdle. So we had to drive back – she was sans breakfast but looking a lot brighter. We figured perhaps it was just a one off – nursery couldn’t keep her in case it was contagious, but maybe she’d be able to sit and watch a few planes?

We set off for Panshangar again… .. and then she threw up all over her car seat. Noooo! Clearly this wasn’t going to be the day that we’d planned – I had to drop my wife and daughter off at home but made it Panshangar on the dot of 11. (Third hurdle crossed – finally made it, phew!)

Thankfully the atmosphere at Panshangar was very relaxed & I had ten minutes to sit with a coffee before being introduced to Sergio, my instructor for the flight. Before this point, my expectation was that a real pilot would fly the plane, I would get to look out the window and enjoy the view, take a few photos and perhaps get a chance to have a go at the controls for a bit. Ho ho ho. As Sergio led me out to the aircraft, his words were “you’re the pilot today so you’ll be doing the whole flight, including the takeoff and all of the landing if you like”. Wow, gulp. Hopefully all those hours spent messing around with flight simulators on my computer would help. I knew what most of the instruments did, but this would be the real thing – and I’d been told that flight simulators are only about 10% of the real experience, so I was nervous.

Unfortunately my wife wasn’t there to be able to take any photos, but I did manage to get a few. This is G-BUIK (Golf Bravo Uniform Indigo Kilo) – a Piper Warrior II, just a couple of years younger than me:

(Sergio is doing the necessary pre-flight engine checks, so I got a chance to look under the covers. There was surprisingly little in there!)

After a walk around the aircraft talking about physics and wobbling various wobbly bits, it was time to hop in.

Right then, what does all this do again? Headsets on, turn the key to start the engine and then we’re off taxiing across the airfield, and I’m learning to drive a plane across the bumpy grass by steering with the pedals (or pushing with my toes to brake) and desperately trying not to embarrass myself!

After following another plane to the end of the runway, we waited & checked the engines one last time (it’s worth being sure, right?). Then I taxiied onto the runway, and with the throttle full open bounced along the grass pointed it at the sky and all of a sudden we’re up in the air. Wow. (You know that feeling of heavy-headedness and disorientation you get when your plane first leaves the runway? Yep.) But it was so smooth – the aircraft was wobbling around a bit, but not as much as I was expecting. The feeling was fantastic! We turned a few times and climbed up to 2000ft, where the view was just incredible – you could see the airports at Stansted and Luton, and even make out Canary Wharf in the distance.

(Sorry, no photos from this bit – I was having too much fun)

I had a go at navigating, practising turns using the rudder pedals as well as the yoke, controlling ascent and descent with the throttle, and even a stall (which I thought would mean we’d fall out of the sky rather dramatically, but it turns out that with this type of plane it just beeps at you and the tail flutters around a bit until you sort it out). The half hour seemed to last a long time as we flew around the skies of north Hertfordshire (obviously there were limits on where we could fly due to the nearby airports), but eventually, just a few drops of rain started to streak along the windshield, it was time to head back to Panshangar – but not before we’d passed over my house, where Sergio took the controls for a bit while I got the camera out:

.. and checked up on the progress with our garden. (see the patio and snaking stone path?)

Then came the hardest part – landing. It was fairly easy to steer the aircraft so that it was lined up with the runway, but Sergio had to help with the throttle a bit so that we didn’t come in too fast. Then we were over the trees and with the throttle closed, pulling hard back on the yoke to land – gently! – on the runway.

(approach to Panshangar – this pic from Google Earth – I was a bit busy at the time)

In fact as landings go, I might be biased, but I reckon it was smoother than some of the commercial flights I’ve been on. :-) Then I taxiied off the runway, back round to the parking area and we stopped pretty much bang on the marks. As the engine stopped and the generators whined down, I thanked Sergio for what had been a brilliant flight and tremendous fun. I might just have to do this again sometime.

(My hair was already like this before the flight, just for the record.)

Over and out.

Zephyr #1

Fighting the Gumblar worm

I’ve been having trouble recently with a javascript worm (‘Gumblar’) which infected my blog & tried to make browsers download some malware. I’ve finally tracked down the last infected script file (a wordpress blog theme header file), upgraded WordPress, changed passwords and updated my blog them. We’re there. Finally.

Theme change

Changed the theme of my blog again as I decided it was looking a bit too techy. Not that most of the stuff I post is at all tech-related. No.

Cloning Windows 2000 to a VirtualBox VM

VirtualBox is really quite impressive, especially as you find out more about what it does behind the scenes. It just seems so easy to use compared with VMWare (though I’m sure VMWare does more).

I finally got my ancient Windows 2000 system cloned last night. I didn’t want to have to go through the pain of completely reinstalling it, but felt the need to move to a (slightly) more up to date OS (Windows XP). I was already dual-booting into XP on a separate partition and having played with VirtualBox before, turning the Windows 2000 system into a Virtual Machine (VM) that I could run on Windows XP seemed the right thing to do.

For the record, the steps (after much experimenting) were:

  1. I slimmed down my windows drive by moving all my data files onto another partition
  2. I installed DesktopOK and saved my desktop icon positions
  3. I followed the section on “Hard Disk Support” from the VirtualBox guide (this is vital)
  4. I downloaded VMWare Converter, a free tool that makes a virtual machine out of a hard disk partition.
  5. VMWare Converter made a VMWare virtual machine for VMWare Player 2.5* out of my 20GB C: drive (onto another drive)
  6. I started up VirtualBox, created a new Windows 2000 virtual machine image, and specified the VMWare-created disk image as the hard drive image (which VirtualBox handily understands).
  7. I fired up the new VM.
  8. In the vm, Windows complained loudly about different hardware. I made it install the VGA components but ignored all else.
  9. I rebooted the VM, installed the VirtualBox guest addons, changed the screen resolution, restored DesktopOK’s icon positions and bingo!

I now have all my old applications and services running in the virtual machine, but with all the advantages that a VM gives. Making a backup of my Windows 2000 machine is now just a matter of copying a single 20GB file. I can pause and resume it. I can save its state, switch off my computer entirely, then come back and continue from exactly where I left off.

I’ve installed the basics (browser, email, etc) on my XP installation, but because that’s a minimal usable system (for me) it boots up much more quickly than the Windows 2000 system did, and if I need the Windows 2000 VM, I can restore it in just a few seconds.

The only real issue is that none of the graphics intensive games I had installed will run particularly well from inside a VM, but then most of them are outdated and modern games mostly require XP these days anyway. Besides, Chuckie Egg runs just fine inside the VM and who needs more anyway? ;-)

Unison

I’ve recently started using Unison to keep all my photos in sync between my Windows desktop machine, a Windows laptop and my Mac mini server.

It works really well, however the other day I started seeing the following error:

Fatal error: Warning: inconsistent state.
The archive file is missing on some hosts.
For safety, the remaining copies should be deleted.
Archive ar604df3 on host HOLLY is MISSING
Archive ar4fcd50c1d8c24915dbcbc359c32578fe on host Hal.local should be DELETED
Please delete archive files as appropriate and try again.

I couldn’t locate the archive file on Hal (the Mac mini) to delete it, until eventually I found it in:

/Users/my-username/Library/Application Support/Unison

Once deleted, Unison continued happily on its way.

XPDisplay for X-Plane

I’ve been enjoying X-Plane for several months now and figured that since I’m a programmer, it’s time to give something back. I’ve been messing around and have come up with an information display – useful if you have a second computer sitting around (maybe a laptop) that isn’t powerful enough to run X-Plane. It basically displays information useful for a fly-in:

* Time in GMT
* Altitude (relative to ground), speed, heading, etc.
* Moving map display (so you can get an idea of what you’re flying over)
* Basic radar (showing other aircraft)

It’s by no means perfect, but I’ve used it for the last few fly-ins and found it surprisingly helpful.

www.duncanjauncey.com/xplane/xpdisplay

JDarkRoom 14

The beginning of this month saw the release of version 14 of JDarkRoom, the full-screen text editor that has developed a loyal following since its first release over two years ago.

The new version is the first for almost a year and brings some much-needed bug fixes, new features and a complete rewrite of the settings screen. The main new feature is the ability to export your file via a converter – for example the supplied ‘Markdown‘ converter will generate a snippet of HTML suitable for posting into a web page, using the Markdown syntax rules for formatting. Suggestions for other converters are welcome.

Future development plans include a plugin system, so that other friendly Java developers can get in on the act to add features they’d like to see in JDarkRoom.

Finally, a big thank you to everyone who requested features, submitted bug reports and gave feedback on JDarkRoom – your input is always very much appreciated.

Java Swing UIManager Look and Feel Defaults

While working with Java Swing UIs, I’ve occasionally found it difficult to locate a complete list of all the tweakable colours, borders, fonts etc that form part of a Swing Look and Feel. Here’s my solution to that problem: www.duncanjauncey.com/java/ui

New Applications for CodeAlchemists

The last few months have been especially busy with the arrival of my beautiful daughter, but I have still managed to find some time to write a couple of new applications for CodeAlchemists.

BabyPatterns is a simple program that displays stimulating high-contrast patterns of the type which occasionally entertain young babies.

Contact Sheets generates a ‘Contact Sheet’ for each folder, showing all the photos in that folder. It’s vaguely intelligent about it, regenerating sheets for folders if they’ve changed, skipping folders that haven’t, that sort of thing. Useful when you’ve built up a large archive of photos, as I have. :-)

Next on the development list is a new version of JDarkRoom.

X-plane

I’ve been looking for a decent flight simulator for years & finally have found one that fits the bill: X-plane. I’ve taken some screenshots of a few Sunday online fly-ins for your viewing pleasure.

Blog migration & a new ‘retro’ theme

I’ve finally upgraded the version of WordPress which runs this blog. In doing so, sadly I managed to trash all the comments that people had contributed. One of the reasons for the upgrade was that I’d had to disable comments due to the sheer deluge of comment-spam (1000+ messages a day). Comments are now enabled again.

I’ve chosen this retro theme because I like it and it’s clear and easy to read – but it still needs a few tweaks.

NextDesk

Nearly three years after I first started the project, the first public beta of NextDesk is now online.

Put simply, NextDesk lets you easily drag and drop files from your computer to another computer. If you paste text to it from the clipboard, the pasted text will appear on the clipboard of the other computer.

Have fun!

Dansette