Moved from github to gitlab

I've moved my camcorder-dvd-extractor project from github to gitlab: https://gitlab.com/DrRob/camcorder-dvd-extractor

Update on extracting clips from Canon DC50 camcorder DVDs

In a previous post I described the problem I was trying to solve. 

Long story short, I've solved the issue of the missing first menu page of clips by altering the subtitleripper tool, and wrapped the solution up into a docker container (because containers are the dog's pyjamas). 

It will be interesting to see if anyone finds it and finds it useful.

I emailed the author of the subtitleripper package about the issue I had, but have not received any response.

Extracting timestamped video clips from Canon camcorder DVDs

We have a Canon DC50 camcorder that records onto mini DVDs.  We used it to record various family activities over a period of five or six years, and have a stack of recorded DVDs.

The content doesn't get viewed very often because it's not easy to see what there is without watching it all.  Most of the recordings consist of short clips.  It would be more accessible if the clips could be extracted from the DVDs with timestamps, and mixed in with all our photos (which are also timestamped).

So I set about working out how to extract the clips with timestamps.  The Camcorder supports two different disk formats for rewritable disks - VIDEO mode and VR mode, while recordable disks only support VIDEO mode.  It turns out that extracting timestamped video clips is easier when using VR mode, so I'll explain that one first.

VR mode

I stumbled across a tool on the web that extracts clips from DVD-VR disks: http://www.pixelbeat.org/programs/dvd-vr/

It's a command-line program that must first be compiled from the supplied source code.  Run the executable, passing the path to the .IFO and .VRO files on the DVD-VR disk, for example:

# ./dvd-vr/dvd-vr-0.9.7/dvd-vr M-leavers/VR_MANGR.IFO M-leavers/VR_MOVIE.VRO 

format: DVD-VR V1.0

tv_system   : PAL

resolution  : 720x576

aspect_ratio: 16:9

video_format: MPEG2

audio_channs: 2

audio_coding: Dolby AC-3

Number of programs: 27

num  : 1

date : 2012-08-08 12:11:05

size : 23511040       

num  : 2

date : 2012-08-08 12:13:30

size : 204558336      

num  : 3

date : 2012-08-08 12:16:36

size : 53760000       

num  : 4

date : 2012-08-08 12:17:31

size : 32878592       

num  : 5

date : 2012-08-09 10:13:58

size : 103884800      

num  : 6

date : 2012-08-09 10:15:30

size : 2369536        

num  : 7

date : 2012-08-11 10:04:54

size : 9584640        

num  : 8

date : 2012-08-11 10:12:03

size : 13516800       

num  : 9

date : 2012-08-11 10:12:57

size : 19445760       

num  : 10

date : 2012-08-11 10:25:03

size : 15503360       

num  : 11

date : 2012-08-11 10:29:44

size : 13492224       

num  : 12

date : 2012-08-11 10:51:46

size : 18182144       

num  : 13

date : 2012-08-11 10:52:11

size : 68724736       

num  : 14

date : 2012-08-11 11:00:29

size : 30570496       

num  : 15

date : 2012-08-11 11:00:59

size : 6033408        

num  : 16

date : 2012-08-11 11:01:18

size : 25321472       

num  : 17

date : 2012-08-11 14:14:48

size : 96055296       

num  : 18

date : 2012-08-15 09:18:56

size : 100737024      

num  : 19

date : 2012-08-15 09:20:37

size : 25937920       

num  : 20

date : 2012-08-15 09:21:05

size : 60123136       

num  : 21

date : 2012-08-15 09:24:37

size : 72810496       

num  : 22

date : 2012-08-23 09:29:27

size : 109633536      

num  : 23

date : 2012-08-25 11:09:01

size : 27938816       

num  : 24

date : 2012-08-25 11:09:43

size : 12443648       

num  : 25

date : 2012-08-25 11:11:41

size : 54155264       

num  : 26

date : 2012-08-25 11:13:04

size : 16809984       

num  : 27

date : 2012-08-29 14:21:29

size : 40706048       

This results in a set of .vob files being created, such as "20120808_121105.vob" which contain MPEG-2 video.  The filename is the datestamp of the start of the recording (i.e. in YYYYMMDD_HHMMSS format), and the file itself is datestamped the same.

VIDEO mode

This is more tricky.  The camcorder creates a set of DVD menus, spread across as many "pages" as required, with an entry for each clip recorded, and the datestamps are displayed in the menu highlights.  For example, here you can see the datestamp of the top-right clip in this menu screen is 12. APR. 2011, 11:07 AM:

I've found that the tcextract tool from the Ubuntu transcode package can extract the menu highlights from the DVD image as a set of bitmap images, which the subtitle2pgm tool from the subtitleripper package can convert to pgm format:

tcextract -i VIDEO_TS.VOB -x ps1 -t vob -a 0x20 | subtitle2pgm -o menu

The pamcut tool from the netpbm package can cut out specified regions of these images,

and after inverting each image with pnminvert (also from netpbm) the Optical Character Recognition (OCR) tool gocr will convert the datestamp text bitmaps into strings.

However, for some reason the tcextract command omits the first page of menu entries and starts with the second page, so the first six datestamps are missed.  This is very annoying.  Any ideas how I can resolve that?

Here's a script that extracts and prints the datestamps:

#!/bin/bash

# apt update && apt install transcode subtitleripper

tcextract -i VIDEO_TS.VOB -x ps1 -t vob -a 0x20 | subtitle2pgm -o menu

getdt () {

  pamcut  $1 $2 $3 $4 $5 | 

  pnminvert | 

  gocr - 2> /dev/null | 

  { 

    read dmy; 

    read hmt; 

    if [ "${dmy}$hmt" ]; then 

      y=$(echo $dmy | sed 's,\([0-9]*\)\.\([A-Z]*\)\.\([0-9]*\),\3,')

      m=$(echo $dmy | sed -e 's,\([0-9]*\)\.\([A-Z]*\)\.\([0-9]*\),\2,;s,JAN,01,;s,FEB,02,;s,MAR,03,;s,APR,04,;s,MAY,05,;s,JUN,06,;s,JUL,07,;s,AUG,08,;s,SEP,09,;s,OCT,10,;s,NOV,11,;s,DEC,12,')

      d=$(echo $dmy | sed 's,\([0-9]*\)\.\([A-Z]*\)\.\([0-9]*\),\1,')

      chmt=$(echo $hmt | tr 'O' '0')

      date '+%Y%m%d_%H%M' -d "$y-$m-$d $chmt"

    else

      echo "Missing"

    fi; 

  }

}

i=1

for m in menu[0-9]*.pgm; do

  echo $m > /dev/stderr

  n=$(echo $m | sed 's,menu,,')

  getdt  58 172 150 34 $m

  getdt 242 172 150 34 $m

  getdt 426 172 150 34 $m

  getdt  58 346 150 34 $m

  getdt 242 346 150 34 $m

  getdt 426 346 150 34 $m

done | awk '

BEGIN {

  n=1

}

{

  if ($1 == last) {

    n=n+1

  } else {

    n=1

  }

  if (n>1) {

    printf "%s-%d\n", $1, n

  } else {

    print $1

  }

  last=$1

}

'

rm menu[0-9]*.pgm menu.srtx

I'll add the steps to extract the video clips and name them with these datestamps later.

Bananas: 10p each or two for 25p

http://www.three.co.uk/Support/pay_as_you_go/Phone_tariffs

Add-on	Data	Minutes	Texts	Price
All in One 15	All-you-can-eat	300	3000	£15
All in One 25	All-you-can-eat	500	3000	£25

Given that the standard price for calls is 3p/min, what's the benefit of the "All in One 25" add-on if the "All in One 15" (£15) + 200 minutes of calls (200*3p=£6) would cost £21?

I tried asking Three, but they just replied, "Hi Rob, you've not missed anything, that's the correct price of our Pay As You Go Add-ons."

Update:

So, it seems that Three's response to my question was to cancel the "All in One 15" add-on and replace it with an "All in One 20" add-on, which has the exact same minutes, text and data, but costs £5 more.  #facepalm

TSL2561 light sensor ranges

I've been playing with ESP8266 microcontrollers with built-in wifi.  For want of a better excuse, I've used one to create a wifi connected temperature, humidity and light sensor for our greenhouse:

The recycled coffee jar contains a 3.3V regulator, the ESP8266 on an ESP-12E board (which contains flash memory and a PCB antenna, among other things), a TSL2561 light sensor, and on top of the coffee jar lid is an AM2302 temperature and humidity sensor.

The ESP8266 has been flashed with Nodemcu (http://www.nodemcu.com/index_en.html) which means it can be programmed with Lua scripts.  Lua, it turns out, is horrible, but that's another story.

Every five minutes, the ESP wakes up, collects temperature, humidity and light level values from the sensors, connects to our guest wifi network, and uploads the values to a ThingSpeak channel (https://thingspeak.com/channels/40838).  You can see some charts and dials that are linked to the recorded values here: http://www.noblesque.org.uk/greenhouse.html

The TSL2561 contains two sensors - one that measures visible plus infrared light, and another that measures infrared light only.  The two levels are supposed to be combined using a calculation that approximates brightness seen by the human eye, but I don't care about that for the greenhouse, I just want to see the two values separately.

The TSL2561 measure the light levels and provides output as integer values over a serial interface.  It has two parameters that affect the range - a 16x gain that can be switched on or off, and a variable integration time, with three preset durations of 402ms, 101ms or 13.7ms.  (Integration time can also be controlled manually.)

Initially I used the default integration time (402ms) with the 16x gain turned on, but it quickly became apparent that the values were hitting the maximum end of the range:

So I turned off the 16x gain, but it's still hitting the ceiling sometimes:

The good news is that it's bright in the greenhouse (good for the plants), but I need to use a shorter integration time to extend the range of the sensor further.

Sampling by integration

Sampling by integration is a bit like collecting rain in a glass and measuring the level.  If you put the glass out in the rain for one hour and measure the water level, then empty the glass and put it back in the rain for 10 hours, if it's raining at the same rate throughout that time, the water level will be 10 times higher.

If the glass has straight parallel sides, you can work out the rainfall in mm per hour by dividing the water level measured by the number of hours the glass was outside, and that will tell you the average rate of rainfall during that time.

It's hard to measure a mm level of water precisely, but errors in the measurement will be less significant if the glass is fuller.  In other words, if it's not raining very much, then leave the glass outside for a longer period of time to get a more accurate measurement of the average rainfall rate.

If you leave the glass outside for 10 hours and it rains a lot, the glass will overflow.  In this case, the sampling period needs to be reduced, so that the measured value comes back within the available range.

Enough talk about rain, back to the TSL2561

The integration time of the TSL2561 can be adjusted, much like adjusting the time that the glass is out in the rain, and like the glass, the TSL2561 has a maximum value it can measure, which is 65535.

There is an additional complication - if the integration time is reduced, the maximum value that can be measured also reduces.  The analogy with the glass breaks down a bit here - it's as if the glass is being constructed at the same time as the water is being collected, so if you only measure rain for one hour then you get a shorter glass to measure it in.  If you measure for ten hours, you get a full-size glass.  In fact, the glass is finished to its full height even before the ten hours is up.

With the TSL2561, if you select an integration time of 101ms, the maximum value that can be measured is 37177, and if you select 13.7ms, then it is 5047.

So which integration time option should be chosen, and should the 16x gain be switched on or off?

Switching off the gain and selecting the shortest integration time should provide the largest range of measurement values.  However, because maximum value is limited by the shorter integration time, the range at 13.7ms is about the same as at 101ms.

It's hard to visualise so I made a chart:

(which gets cropped in this layout - click it to see the full image).

Input light level is on the X-axis, with a range from 0 to 1 (where 1 is the brightest that the TSL2561 can measure without overflow).  The output values are on the Y-axis, and you can see that the 101ms and 13.7ms integration time options have a lower range of possible values than the 402ms option.

It's clear to see in the chart that the 13.7ms options provide about the same input range as the 101ms corresponding options, but with a lower range of output values.  The only benefit of choosing 13.7ms is that you get the measurements quicker.  This isn't useful for my greenhouse, so I won't use the 13.7ms setting.

The 101ms option with the 16x gain turned off provides the greatest input range.  If I had to use just one combination, this would be it.  However, at lower light levels, the granularity of output values is poor.

Therefore, at lower light levels, I can switch to one of the other options to get better granularity of output values.  

The output values will need to be scaled according to the chosen settings, so that the values don't alter significantly when changing the integration time or switching on/off the gain.

So, given that it doesn't matter too much how long the measurement takes, I think what my greenhouse monitor needs to do is try the remaining four options (ignoring the two 13.7ms ones) in order of smallest range, smallest granularity first, and return the first result that doesn't overflow, scaled to normalise the output values across the set of possible settings.

Next steps 

Turn the above into code.

Edit:

Hmm.  For reasons I cannot fathom, when I set the integration time to 101ms and the light is bright, I cannot get a "full scale" reading of 37177 on both channels unless I wait about 250ms after powering up.  I can't see any clues in the datasheet as to why that might be - it's supposed to begin integrating as soon as it powers up, then transfer the values to the data registers.

For now I've set it to wait 500ms in all cases before reading the values, to be sure.

Displayless interfaces

Thanks to the tube strike the other week, I was forced to get some exercise. I walked the few miles from Liverpool St to the office where I work, and because I wasn’t familiar with the route and didn’t want to walk head-first into a lamp-post while looking at the map on my phone screen, I put an earphone in one ear and enabled voice-guidance on Google Navigation, leaving the phone in my pocket, so that I could concentrate on avoiding lamp-posts and not getting run over.

Disappointingly, the voice directions proved to be less than adequate. For some reason, the spoken instructions are simpler when Walk is selected instead of Drive. At all but the simplest of junctions, the terse “turn left” or “continue straight” message in my ear was ambiguous, and if I turned around to consider the options then any sense of direction was quickly lost.

What I needed was more context. “Turn left into Aldgate St,” instead of just, “Turn left”, would have been a good start. Better still, if I could ask questions such as, “Which road should I take?”, “Describe this junction,” or the kids’ favourite - “Are we nearly there yet?”

Google Glass could have put the map into my view, but who wants to look like part of the Borg Collective? And for that matter, I don’t much want to be seen talking to myself either, so give me an earphone for one ear and some buttons I can press without looking at the them, and I’ll be happy.

I think Google may have missed a trick - Google Glass is expensive, but Google Ear could be a free downloadable app.

There’s a similar problem when using SatNav in my car. If I want to detour to get fuel, then using the touchscreen to zoom out and look around the local area is tricky, not mention dangerous and illegal. There’s trend among car manufacturers now to replace dashboard controls with a large touchscreen, expanding this problem to even more tasks, from turning up the fan to changing radio station. (At least someone is thinking about this, but the communication is not rich enough in that interface for what I want.)

If I was using Google Ear to navigate while driving, and the set of standard buttons I can operate without looking at are mounted on the steering column, I could reroute without even looking at the screen, let alone touching it.

The key requirements are these:

1. A set of buttons that can be operated without looking at them. Not too few, and not too many, and I’ll need a set I can use in my car, and a set I can use while walking, and perhaps a set at my desk.
2. The set of buttons needs to be standardised, so that multiple manufacturers can supply them in various forms - a set on a bluetooth-connected key-fob, a set for the steering column, a set on the side of a bluetooth earpiece (killing two birds with one stone), ...
3. An intuitive set of conventions for the use of said buttons, which remain the same in all contexts. Think of the buttons on a games console controller - the left and right buttons always mean the same thing, so their use quickly becomes second nature.
4. Audible communication from the device. It wouldn’t all have to be spoken - short sounds can indicate status, progress, etc
5. Some conventions for the structure of the voice output, which fit in with the button input conventions, and aim for efficient interaction between device and user.  For example, if I search for something, the spoken response could start with a simple, "28 results," and then I can decide if I want it to start listing the results or if I will refine the search first.
6. Voice input commands for situations where button input would be too complex - for example, trigger voice mode with the buttons and say, “Find nearby petrol stations.”
7. It lives in my phone, so I have it with me at all times.

What do you think? Would you use it?

Fixing a John Lewis clock that goes black in a dark room (and changing its colour too)

Despite their faults, we like the style of these John Lewis Indigo clocks.  We have two, and one of them recently started to flicker at low light levels, and then went completely dark whenever the room was dark.  They're supposed to dim in a dark room, but not go out completely.  I took it apart to look for the source of the problem.

WARNING: if you're going to do similar, be aware that there are mains level voltages inside this clock.  Disconnect from the mains and pull out the plug before you open the clock, and don't plug it in again until it's reassembled.  And don't come running to me if you kill yourself.

Inside the clock there are four blue 3mm LEDs that poke into holes in a plastic light-spreader behind the LCD display.  A photodiode, a couple of transistors and some resistors control the LEDs' brightness, however one of the LEDs wasn't lighting up at all.  (Ok, I admit it, I turned on the power while I had the thing in pieces in order to discover this.)  I can't say I'd noticed that the backlight was uneven - the light-spreader must do a good job.

I didn't have any spare 3mm blue LEDs in stock, but I did have white ones, so I desoldered and removed all four, and replaced them with white ones, taking care to insert them with the correct polarity.  I also had some red LEDs in stock, but I chose the white because white LEDs have very similar properties to blue ones (in particular, the forward voltage), whereas red ones are quite different (a much lower forward voltage).

For some reason, the faulty LED was also causing the other three to extinguish at low light levels - with all four replaced it is working perfectly again now.

The white backlight looks really good, and matches the room better too:

The quality of the soldering inside the clock is very poor.  I should have taken a photo, but forgot.  You'll have to take my word for it.  Poor, very poor.

Processing the gas meter monitor readings

I left the gas meter monitor running for a Saturday afternoon and captured the output to a file, then transferred the data into a spreadsheet to experiment.  If you read my previous post, you'll know that the meter monitor records three types of event, all timestamped to a one second resolution: 1) when the gas starts flowing and the meter wheels begin rotating, 2) when the shiny zero passes by on the least-significant digit, and 3) when the gas stops flowing and the meter wheels stop rotating.

The data is just a sequence of numbers - you can't "see" anything in it.  To see some patterns in the gas usage, it needs to be displayed in graph form.  The first graph I plotted was of time (horziontal) against accumulated count of zeros passed:

The points on this graph are the zero-passing events only - this is the data I would get if I'd used a simple reed switch to detect the magnet on the wheel.  

The graph is far straighter than I was expecting - there's hardly any variation visible, despite the sun coming out and going in again during the afternoon (which affects the temperature in the living room a lot), and the front door being opened a few times.

The next graph plots a point each time the wheel stops turning, showing the number of zeros passed during that on-cycle:

You may be able to see that the cycles occur at a fairly regular rate - around one every 315 seconds.  When the boiler is on, it uses gas at a constant rate.  What varies is the amount of time that the boiler remains on during each cycle.

Now at least you can see a bit of variation in the graph.  At the beginning of the recording, the heating hadn't been on for long and it was still bringing the house up to temperature, so the cycles were longer.  Later on, the front door was open for a while a couple of times, dropping the temperature in the hall.

It's obvious looking at the graph that the count of zeros is a whole number by the way the line snaps to equally spaced levels.  There are probably details being lost in the gaps between those levels.

The following graph plots a point for each stop event, showing the percentage of time since the last stop that the gas flowed for:

This graph doesn't directly take into account the amount of gas used - only the amount of time that the gas was flowing.  However, because our central heating boiler is not the "modulating" type, it uses gas at a fairly constant rate, so the shape of this graph is fairly close to the shape of a gas-used graph.

However I won't give up there, as I'd like the gas meter monitor processing to be suitable for other types of boiler, and for households where gas is also used for cooking, so here's one last graph which I think is as close as I can get to real-time gas usage:

This one was the most complicated to calculate - for each on-cycle, the average time between successive zero-crossings is calculated, then the periods between the cycle starting and the first zero, and the last zero and the cycle stopping, are compared to the average period between zeros to calculate a fractional zero count to add to the whole number of zeros passed.  The shape is very similar to the previous graph, but in this case the area under the graph should approximate to the total gas used for the period, as the y-axis is measured in "zeros passed".

If gas were being used at varying flow rates then this graph would be less accurate.  I think the only way I could improve on this would be to try to determine the speed at which the meter wheel is turning, by learning the pattern of varying light levels as the digits 0 to 9 pass by the sensor, and then matching against that as the wheel turns to determine the rate of turning.  Maybe for a future revision...

Resuming work on the gas meter monitor

It's been over a year since any progress has been made with the gas meter monitor, partly because my elder son became interested in microcontrollers and borrowed my mbed for a while.  We built a home-made LED matrix to connect to it, and he used it to tackle some challenges that I set:

Anyway, he's moved onto other things now so I've got the mbed back, and have progressed with the gas meter monitor.

Firstly, I've splashed out on a workshop development board to make it a bit neater and more robust.  Here you can see the board with the mbed attached, fixed to the back wall of the under-stair cupboard where the meter is:

Secondly, I've made some further progress with the firmware.  The mbed is now detecting when the meter wheel starts and stops turning, and when the zero passes by.  At the moment, these events are being multicast onto the home network, but the plan is eventually to save them onto a memory card (there's a micro-SD card slot under the mbed on the development board), and get the NAS backup drive to collect updates each day when it switches on for the daily backups.

Here's a sample of the multicast events it's sending out now:

nas:~# emcast 239.192.1.100:50000 | grep '^[01Z] '
1 1330030611 1 0
Z 1330030624 1 1
Z 1330030636 1 2
Z 1330030648 1 3
Z 1330030660 1 4
Z 1330030672 1 5
0 1330030676 2 5 5
1 1330030927 1 5
Z 1330030938 1 6
Z 1330030949 1 7
Z 1330030961 1 8
Z 1330030973 1 9
Z 1330030985 1 10
0 1330030987 2 10 5

The first character of each line indicates the type of event: '1' indicates that the meter wheel has started moving, 'Z' indicates that the zero passed by, and '0' indicates that the meter wheel stopped turning.  The second field of each line is the time of the event, measured in seconds since the epoch.  The third field is a count of the number of following fields, which varies according to message type.  The fourth field is the total count of zeros passed since the last reset.  And finally, in the case of '0' events, the fifth field is the count of zeros since the meter wheel last started turning.

When the heating is on, and the house and the hot water tank are up to temperature, the boiler cycles on and off with a fairly regular pattern.  It will switch on and use gas at a fairly constant rate for five or six zeros, and then switch off again.  It repeats after several minutes.  I expect that this gap is going to vary inversely with the temperature outside, but don't have enough data to say more precisely yet.  (That reminds me - I need to add temperature sensors so I can record inside and outside temperate with the meter logs...)

Our central heating does not have a thermostat - it only has thermostatic radiator valves.  It will be interesting to see how its behaviour does vary with outside temperature.  Does it cycle on the heating at a regular rate (e.g. once every five minutes) and stay on for longer when it's colder outside, or does it wait less time before switching back on when it's colder outside?  I believe that it can only judge by the temperature of the water being circulated, which will drop quicker when the radiators are losing heat quicker.  We shall see...

Ikea wardrobe light hacking

Passing through the Ikea Marketplace the other day I was tempted into buying a pack of two battery-powered, motion-activated LED lights, intended for installation inside a wardrobe. £4.99 for the pair. I thought they'd make a good night-light for nocturnal trips to the loo.

What I hadn't realised is that they activate regardless of whether it is light or dark, so if I put one of these in the bathroom it'll be coming on during the daytime as well, which will run the batteries down a lot quicker.

I opened one up to see what was inside. I found a datasheet for the BISS0001 chip that powers it, and this revealed that it has a "Trigger disable input" pin, which was tied high with a 1M resistor.

One light-dependent resistor salvaged from an old night-light, a bit of dremelling and soldering, and the light now only activates when it's dark. The LDR pokes through a small new hole on the front:

and is connected between pin 9 of the IC and ground:

Now the only problem is that it's much too bright.

Highlights from 2011

The truth is that 2011 wasn't lost entirely. Aside from the usual things like eating, sleeping and working to pay the bills, I also found the time to make a set of digital dimmers for a pair of Ikea LED spotlights, and built another desk with my elder son. Oh, and fitted about 6 metres of under-cabinet LEDs in our newly-refitted kitchen. (Did I mention like I like LEDs?)

LED spotlight dimmers

The dimmers were built partly out of need (the 3W spotlights are far too bright for my delicate eyes), but mostly as an excuse to learn about driving mosfets with a microcontroller and using rotary encoders.

The result is a little box that's fixed to the back of our bed headboard, with two controls on top that push to toggle the lights on/off, and rotate with a nice click feel to set the light level to one of 16 levels of brightness.

The LEDs give out a surprising amount of light at the lowest brightness level with only ~1.5ma passing through them.

Desk #2

My elder son built himself a PC early last year, and squeezed it onto an old computer trolley that I used as a teenager, but with a graphics tablet and various other peripherals added it was getting too crowded.

He liked the idea of building a custom-made desk, so we bought him the materials for his birthday, and we built it together, using the same techniques I'd used for my desk but with a different design to meet his requirements.

Pretty good eh?

What happened to 2011?

I think I must have overslept. My last blog post was in Sept 2010, and now it's 2012...

Nevermind - 2012 is a fresh new year, and I'm off to a great start as I've just begun ordering parts to build a Shapeoko. Making my own CNC machine has been something I've been thinking about and researching online for quite some time now, and meanwhile I've been saving up, and finally I've found a design that's within reach.

I hope to be able to make all sorts of things with it when it's done, including PCBs, which will hopefully unblock another project that has stalled due to needing a cheap way to make PCBs.

Watch this space...

Gas readings over the network!

Two more small steps today:

1) A couple of variable resistors added to the circuit - one to vary the current through the IR LED and one to adjust the strength of pull-up applied to the sensor, and with a bit of tweaking I've now got readings over a useful range.

2) Readings are now being multicast onto the home network, and being received with the Linux 'emcast' tool. Saved into a file, loaded into a spreadsheet, and converted to a chart, I now have:


Next step is to take the hundred samples a second and turn it into a lower data rate feed of readings.

mbed now a "connected device"

Today's progress: my mbed is now connected to the LAN (using an RJ45 snipped out of a broken router). It obtains an IP address via DHCP, sets the time via NTP, and then just displays the time on an LCD display.


Next step I think I will try multicasting the sensor readings from the gas meter onto the LAN. I've already prepared network and power sockets in the understair cupboard near the meter.

Spikes removed

I've reworked the circuit to put a low-pass RC filter between the phototransistor and the mbed analog input:



and the spikes are now gone:


There's a bit of noise on the flat section at the end where the meter had stopped, but that may only be visible now because the range of values is significantly reduced. Previously, ignoring the spikes, the values ranged from about 4000 to 40000, but now I'm only getting 3100 to 4250.

I'm not sure how to adjust the resistor and capacitor values to increase the range, and it doesn't help that I don't know what's inside the mbed on that analog pin.

Gas meter monitor

Since installing an electricity usage monitor we've been a lot more concious of the amount of electricity we're using. We'd like to do the same for gas usage, to answer questions such as how much gas is required to heat the water for one five minute shower, whether it's better to turn off hot water heating during the day, etc.

I've been recording the meter readings for a number of years, but the frequency of reading (roughly the rate at which the power company asks for readings) isn't anywhere near enough to tell what effect individual actions have on gas usage. Here's a plot of the last eight years of usage against week number:



It looks as though usage is up this year - could that be due to the new shower that we fitted recently? This is the kind of question that we want to be able to answer.

So I'm making a monitor that reads the gas meter for me, continuously.

Our gas meter is a common type with numbered wheels that show the reading. The zero of the least-significant digit is a shiny silver circle - obviously intended for detection by an optical sensor of some kind.

It's possible to buy a sensor unit designed specifically for the meter we have, but I didn't find that out until I'd already obtained a generic optical sensor. And besides, I'm a cheap-skate.

I also later discovered (with a cub-scout compass) that there's a magnet attached to the least-significant digit as well, so sensing the movement of that is another option. I don't have any reed switches "in stock" so I'm sticking with the optical sensor for now. Also, I'm hoping that the optical sensor will be able to detect rotation of the wheel as well as the zero-passing point, for a more accurate instant reading.

Here's a photo of the meter with a reflective IR sensor attached. The sensor is glued to a plastic rectangle, with two holes drilled to fit two small plastic protrusions on the meter. It simply pushes into place, so is easily removed. I hope it's obvious that it's not permanently or electrically attached to the meter, as I don't want the official meter reader getting upset when (s)he sees it.



Leaving a PC running 24/7/365 to record the data would sort of defeat the purpose of the exercise wouldn't it? On the other hand, I want to make the data as instantly available as possible, so it needs to be connected to the network (at least inside the house, if not to the Internet at large). I considered an Arduino with an Ethernet shield, but it turned out to be cheaper to use an mbed instead, which has Ethernet and various other nice things already built-in.

So far I have the IR sensor and a two-line LCD (so I can see what's going on) attached to the mbed. During the prototyping stages I'm powering the mbed from a 6V lead-acid battery that I had spare.

The mbed has a built-in power regulator and outputs 3.3V for other parts of the circuit to use. Unfortunately the LCD requires 5V and won't operate at 3.3V (grrr!) so I've had to add a low-drop 5V regulator to power that.

The IR LED is powered from the 5V supply via a 220 ohm resistor (1.7V forward voltage, so 5-1.7=3.3/220=15ma - well below the nominal 40ma but seems to work fine). The IR phototransistor is connected between and an analog input on the mbed, with a 100K pull-up resistor, and a capacitor to ground for smoothing.

Other websites documenting the use of a QRB1134 suggest using a 0.1uF capacitor. I didn't have one of those in my garage stock so used a 100pF instead. The resulting readings contained a lot of seemingly random spikes, so I swapped it for a 4.7uF, but it still has a lot of spikes. Not sure what to do about that.

Here's the circuit:


For now the mbed code is very simple:

#include "mbed.h"
#include "TextLCD.h"

TextLCD lcd(p24, p25, p26, p27, p28, p29, p30);
// rs, rw, e, d0, d1, d2, d3

LocalFileSystem local("local");
BusOut leds( LED1, LED2, LED3, LED4 );

AnalogIn ain(p20);
char filename[25];

int main() {
sprintf( filename, "/local/%d.txt", time(NULL) );
lcd.printf( "%s.txt\nBegin in 5s...", filename );
wait( 5 );
lcd.cls();
FILE *fp = fopen( filename, "w" );
for ( int i=0; i<4000;>
    unsigned short val = ain.read_u16();
    leds = val >> 12;
    fprintf( fp, "%u\n", val );
    lcd.cls();     
    lcd.printf( "Sample %u\n%u", i, val );
    wait( 0.01 );
}
lcd.cls();
lcd.printf( "Finished." );
fclose( fp );
}

I should really have added a button to start a capture, as getting the captured file onto the PC without it being overwritten is a bit tricky. I just leave the power on when the capture finishes, disconnect the sensor, carry the mbed to the PC, plug into USB and copy off the capture file.

I've been getting a few blue-screens-of-death on my PC when attaching the mbed to USB or resetting it while the USB is attached, which is a royal pain. No idea what's going on there.

Here the data after importing to a spreadsheet to plot it on a graph:



The spikes can be clearly seen, but ignoring those it's also clear to see the three places where the shiny zero passes, the short periods where the meter wasn't turning at the start and end of the capture, and even the "shape" of each of the individual digits 1-9 as they pass the sensor.

I have a way to go with this yet, but the results so far are encouraging.