Finding a storage service provider

First off I needed to find someplace where I could upload my files. The most obvious choices were Dropbox, SpiderOak or one of the alternatives found here: http://alternativeto.net/desktop/dropbox/. Even though some of the clients were possible to setup without a GUI and provided Linux clients, they did so for the x86 or x86_64 architecture. The problem is that the Bubba|Two runs Debian on a PPC architecture which I couldn’t find any single client for. Needing a proprietary client was kind of annoying anyway so I started looking for alternative solutions such as backing up over Rsync, Scp, Ftp or WebDav. I finally found two providers allowing me to connect to my storage area using WebDav: Humyo and BingoDisk. During my, not that scientific tests, I had better luck with performance on Humyo and as they were the (minimally) cheaper alternative I opened a trial account there.

Setting up the storage access

In order to access the storage using webdav we need a webdav client. Preferably the client should allow for easy mirroring of an entire directory of files. The natural choice would be the webdav FUSE filesystem davfs2. FUSE stands for file system in user space and allows developers to implement file systems that can be accessed from userspace with FUSE rather than being implemented in the Linux kernel. There are today several great FUSE filesystems such as davfs2, sshfs, ftpfs, encfs and many many more. Once the file system is mounted it appears as any other file system. FUSE also allows for file system chaining so that encfs can run on top of eg. davfs2 to provide file encryption on top of a remote WebDav file system.

The davfs2 package exists in both Debian and in Ubuntu but unfortunately with old versions which I had some trouble getting to work with Humyo. The problem I was experiencing was that all uploaded files would be of zero size and empty. This was solved as soon as I installed a more current version (1.4.1). As the upcoming Ubuntu Karmic release ships with this version I used the package source to build a new package for my Bubba|Two which worked out great. This is what I did:

Preparing the build environment

First off I needed a compiler and build environment to build from source, I installed it by issuing the following command:

sudo apt-get install build-essential make gcc

Then the davfs2 package requires libneon to build. In order to install this I did:

sudo apt-get install libneo26-gnutls-dev

Downloading the source code

Once these prerequisites were in place I downloaded the source code, decompressed it and patched it using the following:

mkdir $HOME/src
cd $HOME/src
wget http://archive.ubuntu.com/ubuntu/pool/universe/d/davfs2/davfs2_1.4.1.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/universe/d/davfs2/davfs2_1.4.1-1.diff.gz
tar xvzf davfs2_1.4.1.orig.tar.gz
zcat davfs2_1.4.1-1.diff.gz | patch -p0

Building the package from source

Now we can start to build the package by issuing:

cd $HOME/src/davfs2-1.4.1
chmod +x debian/rules
sudo debian/rules binary

This should give us a complete deb package in the parent directory ($HOME/src). The name of this package depends on the architecture which it was built for, in my case the name was: davfs2_1.4.1-1_powerpc.deb

Installing the package

The package can be installed by issuing the following command:

sudo dpkg -i $HOME/src davfs2_1.4.1-1_powerpc.deb

Setting up davfs2 to access Humyo

With the davfs2 package installed we now have to do some minor changes to a couple of files in order to mount the file system without the need for root access and prompting for credentials. Also as Humyo doesn’t seem to support file locking, we need to disable this to in order to avoid a warning each time we mount a file system.

Adding your Humyo username and password to davfs2

In order to add our credentials to davfs2 we need to add the following file to the /etc/davfs2/secrets file:

https://dav.humyo.com        ” ”

So if your username is foo@bar.com and your password is foobar the line should look like this:

https://dav.humyo.com    foo@bar.com    ”foobar”

Don’t forget that the server should be accessed over SSL, hence the https://dav.humyo.com

Disabling file locking

As Humyo doesn’t support file locking we need to disable this in order to remove the annoying warning each time a file system is mounted. This can be done either on a per-system, per-user level or in a custom configuration file. As I’m not running any other WebDav file system on the server I changed this in the global /etc/davfs2/davfs2.conf file. Consult the mount.davfs manual page (man mount.davfs) for information how this can be changed on a per-user or in a custom configuration file. To change it on a per-system level set the use_locks option to 0 (zero) in the /etc/davfs2/davfs2.conf file.

Importing the SSL certificate

We’re now almost all set to mount the remote WebDav file system. However, if you try to mount the file system now, you would most likely get the following message:

/sbin/mount.davfs: the server certificate is not trusted
issuer:      US, New Jersey, Jersey City, OptimumSSL CA, OptimumSSL CA
subject:     OptimumSSL Wildcard, Domain Control Validated, OptimumSSL Wildcard, Domain Control Validated, *.humyo.com
identity:    *.humyo.com
fingerprint: bf:28:74:8e:63:96:bf:07:9d:99:ed:f7:bf:44:cc:ad:f8:7b:c6:a7
You only should accept this certificate, if you can
verify the fingerprint! The server might be faked
or there might be a man-in-the-middle-attack.
Accept certificate for this session? [y,N]

The error occurs due to OpenSSL not being able to verify the SSL certificate chain. In order to get rid of this message we need to provide davfs2 with the root certificate. We do this by issuing the following command:

wget -O - "https://support.comodo.com/index.php?_m=downloads&_a=downloadfile&downloaditemid=10" | \
sudo tee /etc/davfs2/certs/addtrust_ab.pem

Then we need to instruct davfs2 to read this pem file when mounting our file system by adding the following line to /etc/davfs2/davfs2.conf:

servercert /etc/davfs2/certs/addtrust_ab.pem

So, now we should be all set!

Creating a mount point and adding it to fstab

We’re now ready to create a mount point. I chose to mount my Humyo storage to /mnt/humyo. In order to be able to do this I created the directory by issuing:

sudo mkdir /mnt/humyo

Once we have the mount point in place we can try to mount our file system by issuing the following command:

sudo mount -t davfs https://dav.humyo.com /mnt/humyo/

If all went well we should not have gotten any error messages and we can test whether it works by copying any file to /mnt/humyo. You can check whether the file actually made it to Humyo by checking their web interface. If it works your good to go, if not well, you need to start checking what went wrong.

Adding the mount point to fstab

The current setup only allows root to mount the file system and requires you to issue the above command to do so. In order to allow users to mount the file system you need to add a line into /etc/fstab.

https://dav.humyo.com    /mnt/humyo    davfs    user,noauto,rw    0    0

If you want the file system to be mounted at boot change the noauto to auto instead.

However, in the configuration file all looked fine and dandy: create mode, directory mode and inherit permissions were all still set. To make really sure I added some of the force options as well. Still, all folders were created with 0755 (drwxr-xr-x) permissions (I had it setup and expected 0750). I thought I was going nuts until I found some other articles outlining the exact same problems and pointing their finger at Mac OS X.

http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg657971.html
http://lists.samba.org/archive/samba/2009-June/148518.html

Apparently the servers incorrect setting of ACLs on the directories was due to the umask value on Mac OS X. Going through the article, I found that the following suggested settings fixed the problem for me.

unix extensions = no
force directory security mode = 0770

Once setting unix extensions to no, I had to disconnect and re-connect to the server in order to access my files and from then all directories were created with the correct permissions.

I also purchased an additional 1TB eSata drive, also from Excito, which looks almost the same as the server. Knowing from their specs that the device did not yet support raid-1 I was nevertheless hoping to get this running with the additional disk I bought. So far I have synced 39.2% of the new raid array I created and things are looking very promising. For anyone intrested, these are the steps I took:

First of all I setup a new account through which I could access the server through SSH. I then connected to the system over SSH to look at the partition table. Fortunately the layout was quite good and required only little work to adapt it for mirroring. The bubba two server I bought came with a 1TB disk partitioned like this:

/dev/sda1          0+   1216    1217-   9775521   83  Linux
/dev/sda2       1217  121456  120240  965827800   8E  Linux LVM
/dev/sda3     121457  121600     144    1156680   82  Linux swap / Solaris

My concern was to mirror the “storage” partition (sda2) where I will be keeping my data. This partition was currently mounted as /home which was in use as it contained my home directory. So in order to start building the RAID array I had to do some small preparations. All of the commands should be executed as root.

To be able to unmount the partition I first created a new “home” directory

mkdir /home2

I then copied all of the contents from /home to /home2

cp -a /home/* /home2/

With the home contents in the new directory I opened the passwd file and changed my home-directory from home to home2

 nano -w /etc/passwd

In order to minimize the chance of screw-up I opened a new terminal window after saving the passwd file to verify that I could still SSH to the box. Once this was done I unmounted the old /home partition, which should no longer be busy.

umount /home

Now we’re ready to start removing volume-groups, logical volumes and other disk mappings I did this by executing the following commands:

lvremove bubba/storage
vgremove bubba
pvremove /dev/sda2

Once this was done I needed to change the partition type for the second partition from LVM to RAID. I did this by starting cfdisk, selecting the second partition and then chose types. The type needs to be changed from 8E to FD. After this I simply copied the partition table from /dev/sda to /dev/sdb using sfdisk using:

sfdisk -d /dev/sda | sfdisk /dev/sdb

With the partitions in place I created the raid-1 mirror array by issuing:

mdadm --create /dev/md0 --level=raid1 --raid-devices=2 /dev/sda2 /dev/sdb2

In order to minimize the changes needed in the bubba configuration I simply created the same volume group and logical volume as before.

pvcreate /dev/md0
vgcreate bubba /dev/md0

In order to determine the size of the logical volume group I did

vgdisplay | grep 'Total PE'

In my case this value was 235797, so this gives us:

lvcreate -l 235797 -n storage bubba

I then created an ext3 file system by issuing the following command:

mkfs.ext3 /dev/mapper/bubba-storage

When this finnished, I was ready to mount the home partition back, but this time as a raid-1 mirror:

mount /home

Now I copied all file from /home2 back to /home

cp -a /home2/* /home/

I started nano again to edit /etc/passwd to change home2 back to home

nano -w /etc/passwd

Again, tried to ssh from a new terminal window in order to verify that it worked. Once I verified that it did, I issued a reboot just to see that it was all working. Which it did once the server was back on line. To check the state of the re-sync of the array I issued the following command:

cat /proc/mdstat

Now, it’s at 58.6% and will hopefully finish soon….