Perl and anything else

Finally starting to get somewhere with the weather station stats

This post is really just a record of what I've done to get to the point of graphing and displaying the weather info from my weather station.

The weather stations reciever and display panel is hooked up via USB to the mediacentre, my MythTV box that never gets turned off.

I've finally written a working script captures the data and writes to the RRD database, however all of this sits on the mediacentre.  My web stuff sits on my linux server in a cupboard.

To get access to the database for the web page to come, I've had to think about the implimentation somewhat and have come up with the following:

On the mediacentre I have created an rrd directory under /var/local

Under that I'm going to create a directory structure based on the RRD databases I may have, so for the weather station, I now have weather, below that logs, which is where all the text based logs will be stored for later processing should I need some historical data, or just want to do something different.

I have exported /var/local/rrd and mounted it via nfs on the Linux server.

All I need to do now is write the code to generate the web page.

backuppc to nas over NFS

Hazzah!!! It works.

So the key thing here is in this post:

http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/backuppc-21/nfs-mount-for-backup-storage-73200/

And here is the key paragrah:

===

Hard links only work within the same filesystem because they are just
additional names pointing to the same inode. The cpool and pc
directories must live under the same mount point.

===

So the penny started to sink or the light bulb started to glow, but I got it wrong.

I gave up then for the night and made dinner.

Upon returning to the task at hand, I read that paragraph again:

The cpool and pc
directories must live under the same mount point.

Oooooh.... OK then.

So I moved back to mounting my nas under /mnt/nas/backuppc and once I did that, I did a simple:

<pre>

cp -aRp /var/lib/backuppc/* /mnt/nas/backuppc/

</pre>

This copied across all the directories with the same permissions.

I just kicked off a backup of the localhost ( the server itself ) and the log file shows:

<pre>

2011-04-04 21:19:36 full backup started for directory /etc

</pre>

So now to explore what it has to offer.

 

 

backuppc

I hurt my back lifting cement mix for the garden wall into the trailer, so I've taken the day, seen the oseteopath and bought "Linux Journal" for a bit of dead tree reading on the couch.

While flicking through it I came across mention of backuppc http://backuppc.sourceforge.net/index.html as a good backup application.

Given that I have got much of the data I want backed up already going to the Thecus N4200 via locally run rsync scripts, I figure it's time to get things sorted out a little better and where I can centralised.

Backuppc certainly seems to be the go.

I've installed backuppc on my debian server with a simple apt-get install backuppc.

The only thing I need to change is the password for the backuppc web interface user:

htpasswd /etc/backuppc/htpasswd backuppc

Now the plan is to nfs mount a mount point on the N4200 on the server and have all backups located here where we have RAID 5.  The server itself is just a desktop PC with only 1 hard drive, a recipe for disaster.

Creating the mount point on the server is simple, just the usual mkdir command(s).

Creating the Folder on the Thecus requires access to the web interface, then ShareFolder.

I created backuppc on the root. I added in the NFS permissions.

I then created the relevant entry in my fstab and mounted the filesystem.

All going fairly well at this point.

I change the TopDir path to point to my the nfs mounted directory and restart the server.

I'm just accepting the inital defaults to see if I can kick off a backup of local host.

<pre>

2011-04-04 12:21:55 Can't create a test hardlink between a file in /mnt/nas/backuppc/pc/localhost and /mnt/nas/backuppc/cpool.  Either these are different file systems, or this file system doesn't support hardlinks, or these directories don't exist, or there is a permissions problem, or the file system is out of inodes or full.  Use df, df -i, and ls -ld to check each of these possibilities. Quitting...

</pre>

Well, that hasn't worked so well.

Looking at the filesystem on the nas via the nfs mount on the server I can see the owner of the files is 117, the user created when backuppc installed itself, this doesn't map to any user on the N4200.

I figure that I should have a userid on the N4200 that matches the userid of the backuppc account on the server.

You can create a new user on the N4200 via the admin web interface under User and Group  Authentication.

So I create a backuppc user try to set its userid to 117, this fails, I have to user a userid > 1000, so I create it with 1117.

Now I have to set the backuppc userid on the server to 1117, I do this editing the passwd file and run the find command to chmod the files to backuppc.backuppc.

I used this link: http://it.toolbox.com/blogs/locutus/how-to-change-a-users-uid-and-gid-26368 as the basis for my userid changing requirements.

<pre>

find / -user 117 -exec chown backuppc.backuppc {} \;

</pre>

Now that I have userid the same on both, I now add the backuppc user on the N4200 to the backuppc folder I created ealier,  under Storage, click on ShareFolder, click on backuppc folder and then the ACL button at the top, add backuppc as writable, Apply.

Now to see if this works.

FAIL.

Seems that my command broke all the permissions to the directories and files such that backuppc web interface no longer works.

The correct way to chown was to just chown the owner of the file like so:

<pre>

find / -user 117 -exec chown backuppc {} \;

</pre>

So rather than fight this process, I'm going to uninstall and then reinstall backuppc as little has been done with the setup.

<pre>

apt-get --purge remove backuppc

apt-get install backuppc

</pre>

once done, I have a working web site again and the user is mapped to backuppc which is the right userid I need for the mapping between the server and the N4200.    

I edit the /etc/backuppc/config.pl file and change the TopDir to point to the server's nfs mount point for the NAS.

Save the change, restart the service and refresh the web admin page.

With nothing to loose, click "full backup".  The NAS ( yes I can hear it ) makes a bit of noise as the intitial part of the backup kicks off, a quick check to see if we have any errors....    

<pre>

2011-04-04 14:00:01 localhost: test hardlink between /mnt/nas/backuppc/pc/localhost and /mnt/nas/backuppc/cpool failed

</pre>

That would be a fail!

Back to googling.

 

Backing up to the N4200

OK, so the plan is rsync.  The execution, unknown.

I get the feeling when I start these things I really need to get it right first time, however, given I am still trying to understand how the Thecus N4200 does things, I guess I should just get over that issue and just go for it.

If nothing else, getting a copy of all the data to the NAS is better than ruminating over the hows and whys to lose the hard drive with NO copy at all.

The N4200 comes with an rsync module, this sets up the N4200 as an rsync target.

By default you seem to get an rsync_backup module, which is fine if you could lay out the file system to multiple source systems.

Given that I have a workstation and two laptops to backup, I'd like to have something like:

laptops

    laptop1

    laptop2

workstations

    wkstn1

as directories and stuff everything into their relevant folder.

I couldn't get this to work under the rsync_backup module, so instead I've created a "Backup" share  and made it public.

Then connected to the share and created the folder stucture I wanted.

Not really ideal, but I'm starting to learn many things I'd like to do end up being less than ideal with the N4200.

So with the folders now in place it's time to back up the windows box with rsync.

I've written a simple batchfile to start with.

<code>

@echo off
set RSYNC="c:\Program Files\ICW\bin\rsync.exe"

echo %RSYNC%

%RSYNC% -av /cygdrive/c/Users rsync@N4200::Backups/workstations/ripper < .\rsync.pass  > backup.log

</code>

 

basic for now, but it should get me going.

Some notes on this.

First of all you will need to make sure if you backing up something like Users, you will need to have admin rights, the simplest way to do this is create a short cut to the command file, or create a shortcut to cmd.exe and then right click, click on properties, then Advanced and then "Run as Adminsitrator".

Anything run under this cmd exe should be run with the admin rights.

The first backup to the NAS is running as I type this.

More things to distract me, Thecus N4200!

OK, so if I can't get anything else done I now have a NAS for home.

For way too long family photos have resided on a single spindle, and yes, while some of the photos have been burnt to CD or DVD, many are, in fact, I'd say that many dating back the last couple of years aren't. 

Worse, we have a single computer that everyone uses, that's 6 people, all who have profiles and photos and music etc.

So when the Thecus N4200 was advertised, I bit the bullet and bought it.  After a quick read around of course, I found that it performed pretty well and included its own battery to give it time to shutdown cleanly.. one hopes.

With that, I bought 4 2 Tb drives from the computer markets, so for a total outlay of just under $1000, I have a RAID5 solution to backup the all important data that for too long has been sitting unprotected.

Given that we are all entitled to an opinion on everything I'll gripe about my whole experience so far in another post.  To be honest, I'm not sure I get the paradigm, or I'm expecting too much.

Now to explore backup options.

Steps to installing all requirements on a debian based distro

The perl script I'm going to use to read the weather stations data logger ( reciever ), uses the Device::USB module.

To get to the point of installing this module on a "clean" system somethings need to be installed.

To build the required binaries, we'll need to install the build-essential package, we'll also need the libusb-dev package:

<code>

apt-get install build-essential

apt-get install libusb-dev

</code>

To build the Device::USB module there are dependancies that need to be met, once local::lib has been installed it's just a matter of using the cpan client to install:

Inline::MakeMaker

and

Device::USB

That should be it.

Just a note with the udev rules.  The example I followed was using deprecated terms for the device event and attributes.

My udev rule for the weatherstion is:

<code>

SUBSYSTEMS=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021", GROUP="weather-station", MODE="0660"

</code>

One other thing, when setting these things up using "screen", exit all the way out and re-login.  This actually fixed the problem of the script not being able to connect to the usb device.

That's it for now, the laptop is about to run out of battery and I can run the script to read the data back. Next up we'll look at what is coming out of the updated script.

First steps to capturing weather data from the reciever.

Having finally got my weather station home, I ended up googling around to find out what, if anything was out there that would allow me to hook up the receiver via USB to a linux computer to start capturing all the data.

It was good to see that it wasn't going to be impossible, I found solutions in C and Python and much to my surprise, Perl!

Of course the perl option is going to be the one I'll go with first.  I've grabbed the latest version of the data logger from here.

I had found this page some time around october/november 2010 and just as I was starting to get back into getting the data capturing sorted out again the web site went offline.  Thankfully it's back for this attempt at getting the data.

To capture the data 24/7 I need a computer that obviously doesn't get turned off.  Given the cost of electricity these days it's a hard argument to make to turn on a dedicated computer just to capture the data.

However, in this house, there is one computer that meets this requirement, the MythTV backend.

To do what I need I'll need to install a number of perl modules not likely to be included in the mythbuntu distribution ( which runs the MythTV box ).

To facilitate this, I'm using the excellent local::lib.  I'll boot strap the environmet following the instructions in the pod.

After my last attempt at capturing the data, I found that you need a udev rule to allow permission to the USB device with out running as root.  Again googling brought me tothis page. 

So a new udev rule is in order.

That's it for now.  A quick update iwth what I'm doing with some links.

 

Lets get started with the weatherstation

As I said in my blog reboot entry, this is where I'm going to cover things I'm doing personally rather than dedicated to Padre, the Perl IDE.

I'm still a perlwannabe at heart, so the blog names still fits.  :)

The story of the weather station starts many, many years ago.  As with all things I like to have a reason to do something so that I actually get around to do it.  I think this is normal so personally never hold myself accountable for not getting much done.. no reason, no reason to get it done.  :)

Anyway, at the time I figured that learning C would be a good thing, but knowing that once you get past the "hello world" example, you generally end up with the classic "CD database" to push you to the next level.

Neither of these are compelling enough to learn C, however, making your <insert device> work with linux is certainly one reason to learn a programming language, especially one that is as low level as C.

I think there's an inherant desire in most males to know things about their environment, and weather is certainly a big part of it.  I mentioned this one time while on a project that I thought it would be cool to have a weather station, interface the rs232 (serial) port to a linux server and dump the sensor data into a database.  Of course that would mean a web page with charts and all sorts of cool reasons to hack things.

Way back then, I wasn't earning alot and household income, with two young kids, wasn't something that would allow such an extravagence.

So I shelved the idea, with it every now and then popping up on "slow" days.

Over the years, as with all things technologywise, hardware gets cheaper, interfaces change and if you're lucky you have a little more cash to spend on yourself.

Given that one of the guys I was working with at the time I first mentioned my desire to have weather station paid out over $1000 for a rather nice Davis weather station ( I'm pretty sure it was ), I flicked through the jaycar catalogue and found their $150 "digitec" weather station was on special, for something like $99.

OMG!  $99 for a weather station.. and it was wireless!!! and even better USB!!!!

So I figured for $99 I'd just go with what I got and google my way around finding something that allowed me to hook it up to a linux computer.  Even more enticing was that during a long break between contracts, I discovered the Arduino and somehow discovered that one of the projects in a book just on the market covered creating a board that listened into the transmissions from weatherstations and allowed you to grab the data. 

It was like a perfect storm of reasoning that I felt that the $99 was worth the risk.  So I got myself my first ever weather station.

It turns out the digitech weather station is a rebranded Fine Offset WH-1080.  Once I knew this, googling for this to find code etc was simple.

But that's for another day.

The weather station sat on the floor for ages as I agonised over where to put.  Should I put it on the TV antenna mast high above the house (making it interesting come time to change the batteries) or someplace more accessable, while accepting that wind direction and speed may be effected.

In the end I just stuck it to the swing chair out the back of the yard just to put it outside and see if the base station ( the LCD screen ) could pick up the signal.

Turns out that it could.  So it stayed on the swing chair for months. 

I never got around to doing anything more than that, until recently.

We had a big tidy up of the back yard and due to the age and effects of the sun, the swing chair was dismantled and taken to the tip.  This of course meant that the weather station had to move.

While we were at the tip, we stopped at the recycling depot to search of a few things to do with building a chicken run (still not done.. sigh) and I caught a few "J" masts poking out of a big bin.  Awesome.. just what I needed to put the weather station on.

So I grabbed it.  $10. 

With my brand new drill in hand I screwed the new J mast to the side of the shed.

I figured that the wind speed and direction sensors would be high enough up to avoid too much turbulance and the idea was to keep the tempreture sensor low hopefully keeping it shade for as much of the day as possible.

And here it is!

 

 

You can see one of the poles for the chook yard in the shot too.

 

Another shot, showing the tempreture sensor in the shade.

 

 

And another photo.

I think the tempreture sensor is going to be effected by the reflected heat from the roof of the shed.  I'll have to get myself another thermometer to test the temp in the shade to see how much it's being effected.

Anyway, that's the story of the weather station so far.  The fact that it's attached to something that I screwed to an inanimate object myself really is pretty awesome in my world.  For some stupid reason I always figure it's too hard and I'd never get it right and so never start on anything.

I'm quite chuffed so far.

 

 

 

Starting this blog again...

I orginally created a blog account on vox to support the role of Release Manager, which I have still, with Padre, the Perl Progamming IDE.

I chose vox for the simple reason that it was driven by Perl.  After the closure of Vox though I had to find an outlet to post release announcements, which brought about the change to making the Padre release annoucements on blogs.perl.org, a somewhat more fitting location.

There are other things I like to do, or at least procastinate about doing.  Gardening is one of those things.. I don't do it very well.  I watch TV shows and follow their advice, that I can remember, and give a few things a go.

My favourite thing though is my vegie patch.  I've had a couple of reasonable seasons, and I have always thought that I'd mix my desire to have a reason to write code at home with my budding interest in gardening in that I'd write a blogging engine in Perl, specifically in Catalyst, mixing all the things that one could do with programming, namely, web, databases and coding.

However, my ability to actually make progress in achieving my desires, I'm still no where near anything codingwise that would allow me to write stuff and post pictures as a means of keeping a diary.

So for now, I'll try to write stuff about things here.  If for no other reason than at least there may be some sort of record of what I've done and plan to do.

One of the first things I'll cover will be my "adventures" with the home weather station.

Gardeningwise, well, we'll have to see if I can actually keep something going.  :)

 

Padre 0.69 Released - Still Unstable.

Padre 0.69 has been released, it's still marked as unstable while Adam Kennedy works on the various parts of the Task subsystem.  We hope that once the Australian Election is over Adam will be somewhat less distracted and focus on the 'bits that are missing' to remove the unstable tag.

Until then though, 0.69 has been released, it includes the beginnings of the Sync server work that will see the option to store your personalised settings on a server allowing you to share your settings between computers.

As always there's a few bug fixes, like the fix to the annoying bug in the Dialog about changed files that only gave you the option to Close the file or Close the dialog.

After a question through the padre-dev mailing list about cursors blinking and how to turn it off, this release sees a new preference setting that allow those who like to control  blinking cursor to turn it off, by setting the time to 0 or speed it up and slow it down.  If you wish to adjust the blink rate or turn it off, you can find it in Tools/Preferences under Behavior about 3/4's of the way down.

With little more to add, the list of changes for 0.69:

    - Landed ConfigSync branch (MATTP)
    - Task 2.0 restores support for having tasks send unlimited messages
      back to the main application while they are running (ADAMK)
    - Added Padre::Task::Run for background process execution with STDOUT
      lines streaming as events back to the main window. (ADAMK)
    - Fixed test failure in t/93-padre-filename-win.t under win32 (SEWI)
    - Devel plugin now has the option to dump the Task Manager (GARU)
    - Refactored, reskinned and polished ConfigSync functionality (ADAMK)
    - Added ->status to ::Main to allow rapid transient messages to be
      presented in the status bar, as fast as 100/sec (ADAMK)
    - Fixed the file-changed-on-disk dialog: Show "Reload" button instead
      of "Close" (SEWI)
    - Adding a ton of additional $BACKCOMPATIBLE variables so that every
      class consumed by the current family of plugins has them (ADAMK)
    - Nudging up the default background thread count now that we will
      start to see long-running threads looking after background
      processes (ADAMK)
    - Allow the opening of files exceeding the editor_file_size_limit of
      500_000. The file is opened if the user answers Yes to the
      dialog (MULANDER)
    - The Task Manager now records per-worker statistics on which tasks they
      have run in the past. This is needed to support various worker
      optimisation strategies to be implemented in the future (ADAMK)
    - Added a simple initial Task Manager optimisation strategy to favour
      workers which have run a task at least once before (ADAMK)
    - "Find method declaration" will not find forward-declaration (CHORNY)
    - Task manager now has separate maximum and minimum thread counts (ADAMK)
    - Minimum thread count set to zero. Padre starts up 600ms faster, at the
      cost of the directory tree appearing 200ms slower if you use it (ADAMK)
    - Command line switch to select locale (CHORNY)
    - Added configuration option to modify the cursor blink rate in Padre
      as requested via the padre-dev mailing list closes ticket number
      983 (PLAVEN)
    - Added Padre::Task::Daemon for bidirectional communication support in
      tasks. When the task is launched, messages to the chlid can be sent
      down the worker thread's message queue, and they will be tunneled
      through to the task, which can retrieve them Erlang-style via a
      dequeue method (ADAMK)

As you can see there are some interesting things to come into production in Padre.  Feel free to drop on into the #padre channel on irc.perl.org.