Of course, there are plenty of readily available tools that might suit your needs. Writing your own does have a few advantages though: first, it keeps it simple, highly customized and second, when things go wrong, you would probably have a better idea of why the damn thing isn't working!

A look at tape's workings

If this is your first time dealing with tapes(as it was for me), there's a few prerequisites.

Tools

Tape operations are carried out via the mt tool. Data is written with tar. Both of these are probably already installed on your system.

Writing data to tapes

Do you remember the good old days of cassette players ? Tapes are similar. A magnetic head reads and writes data from a magnetic ribbon spooled in an enclosure. With that in mind, there are a few operations that you would do frequently:

1. rewind: rewinds(of course!) the tape and points the tape head to the beginning of the magnetic ribbon. Example command: mt -f /dev/nst0 rewind

2. forward: Move forward count files. Every time data is written, a marker is set at the end. Let's say, you write some data to the beginning of the tape: tar cvf /dev/nst0 backupdir Rewind the tape as above. Now, you want to move forward one marker: mt -f /dev/nst0 fsf 1

3. erase: That's a really slow process and could take hours (if not days) for larger tapes. However, you can also do a short erase: mt -f /dev/nst0 erase 0.

Tar and incremental backups

tar has a handy feature that lets you do incremental backups and the workings are really simple. Let's look at an example:

1. tar -C /home —listed-incremental=diff.snar -clpMvf /dev/nst0 data This is what we call a Level 0 backup. diff.snar is special – it contains a log of all the files that were added to the archive.

2. Next, lets's say you add file.txt to folder data and run the above command again. The only file that would be added to the archive is file.txt. Moreover, diff.snar would also be overwritten with the only one entry that was just added to the archive. This would be a Level 1 archive.

Obviously, if you would want to have a record of all the backups, you wouldn't want to overwrite diff.snar but have rather something like this:

• diff0.snar: level0 backup
• diff1.snar: level1 backup and so on...

Backup Strategy

With all this quick preliminary information, we can try a incremental backup strategy as follows:

1. Maintain two sets of full backup tapes and two sets of incremental backup tapes.
2. Create a full backup the start of every cycle: could be a month, bimonthly, quarterly or whatever you prefer.
3. Until the beginning of next cycle, perform incremental backups.
4. At any point of time, you should always have a backup set that has a full backup of the last cycle as well as incremental backup tape(s) of the last cycle.

Tape utility script illustrates the idea. To perform a full backup, you would run something like:

tapeutility.sh -d /dev/nst0 -F -p /etc/tapeutility/folders.txt where “-F” does a full backup of folders listed in “folders.txt”.

For the next run, to create an incremental backup, you would run: tapeutility.sh -d /dev/nst0 -I -p /etc/tapeutility/folders.txt where “-I” does an incremental backup.

Please take a look at the script for how the metadata file is determined for incremental backup and other features available for basic tape maintenance.

Wrap-up

I presented a simple way to use tapes for backups. Using a combination of full and incremental backups, and maintaining two sets of tapes, we have reliable backup of data that you could combine with a RAID style setup for long term reliable data storage.

#backups #bash #diy

Discuss...

Underwhelming hardware

I used a RPi 3 B board that has a 1.2 Ghz quadcore ARM processor. For processing a video stream and running the motion detection daemon, it's not really very capable and you would end up with stuck/unusable frames on your stream. One of the things that makes a huge difference is the incoming stream frame rate and resolution. I got the best results with sliding down the incoming frame rate to as low as 10 on the camera that I am using.

B vs B+

The B+'s advantage is more on the I/O side and it really doesn't make much of a difference with processing power when it comes to the video stream. On the other hand, the B is more battery friendly which was a major requirement in my setup owing to the frequent power-cuts associated with Indian summers. Overclocking, too, isn't worth it if you consider the battery drain (as high as 20% faster) compared to any noticeable performance gain.

Backup power

As mentioned above, this was an important requirement. I used a 20000 mAH battery that has passthrough. On the downside, when passthrough triggers, there's a momentary disconnect in power which restarts the camera and the RPi which is undesirable but the small downtime is acceptable.

Network

One of the requirements was failover to a backup network but jumping back to the main network once it's back up. A reverse tunnel to a public IP takes care of ssh and http access and could be easily scripted as well. HTTPS is achieved by setting up a nginx reverse proxy on the public facing system and integrating with letsencrypt.

Motion detection

False positives is a major challenge and I could get a good compromise with a mix of a few things: – Setting up a manual mask. This is easy to do with the motioneye http interface. – Using a despeckle filter. Take a look at this article for a nice write up by the author. After experimenting with several combinations, EedDl gave the best results (which also happens to be the recommended starting poin). – Experimenting with thresholds. I used the threshold_maximum parameter to minimize the maximum pixel change. A script changes the threshold value based on input from a LDR similar to this setup.

Usability

The system is easy to use/configure with the Motioneye http interface but to make it a little bit more interesting, I used some NFC tags to enable/disable motion detection. This can be easily done with Tasker along with the NFC plugin for it. This script takes care of syncing up the config file with the current state of motion detection.

#thoughts #tech #diy #rpi #bash

Discuss...