CrashPlan Online Backup on QNAP TS-210 NAS
Installing CrashPlan on my QNAP TS-210 NAS in headless mode means that I have an instant online backup of all of my archive data stored on my NAS drive. CrashPlan keeps revision history of all my files, allowing me to recover my NAS to a state at any date should I lose data or overwrite something.
I have for a while now been looking for a solution for backing up my important files and photos online. After much research and testing various backup providers I found that CrashPlan offered the best value vs quality of service.
Crashplan provides both a Windows and a Linux service that can be used to backup files, in particular interest is that the service can be run in a 'headless' mode whereby the service can be run on a linux box and administered from a windows desktop client.
I currently use a QNAP TS-210 NAS box to backup all of my data and my ideal setup is to keep this in place and automatically backup everything on it to the internet.
On a standard Linux box this setup is simple and straight-forward, unfortunately things on the QNAP are not so straight forward because it uses a Kirkwood Marvel Atom processor that the java jre that crashplan uses does not directly support.
I have used a number of resources in order to achieve my working setup, these acknowledgements are given at the bottom of this post.
The problem
The main problem in running the CrashPlan engine on the QNAP is that the java runtime that it installs fails to start the service correctly, reporting an error in the engine_error.log file. To try and rectify this I removed and reinstalled CrashPlan without installing the bundled java environment. I chose to manually install the java sdk using ipkg. This enabled the service to start but it immediately stopped with an invalid jtux error in the engine_output.log file. Google searching revealed that this meant that the java jtux library needs to be recompiled on the QNAP itself in order to generate a working jtux library file. After multiple attempts I found that I could not re-build this library on the QNAP firmware as the required packages could not be installed (or I simply couldn't find the correct combination of packages to install).
Debian Squeeze on the QNAP
I came across an interesting article that detailed how to install Debian on the QNAP and decided I would give this route a try in order to achieve what I wanted. This proved the correct choice. I followed this article in order to install Debain which went quite smoothly. The main steps for this were:
First create backups of the QNAP flash blocks:
cat /dev/mtdblock0 > mtd0
cat /dev/mtdblock1 > mtd1
cat /dev/mtdblock2 > mtd2
cat /dev/mtdblock3 > mtd3
cat /dev/mtdblock4 > mtd4
cat /dev/mtdblock5 > mtd5
Download the flash installer:
cd /tmp
busybox wget ftp://ftp.debian.org/debian/dists/testing/main/installer-armel/current/images/kirkwood/network-console/qnap/ts-219/initrd.gz
busybox wget ftp://ftp.debian.org/debian/dists/testing/main/installer-armel/current/images/kirkwood/network-console/qnap/ts-219/kernel
busybox wget ftp://ftp.debian.org/debian/dists/testing/main/installer-armel/current/images/kirkwood/network-console/qnap/ts-219/flash-debian
busybox wget ftp://ftp.debian.org/debian/dists/testing/main/installer-armel/current/images/kirkwood/network-console/qnap/ts-219/model
Run the flash installer, this will load the installer into flash so that upon the next reboot it will load the Debian installer:
sh flash-debian
This will display something similar to:
Updating MAC address...
Your MAC address is XX:XX:XX:XX:XX:XX
Writing debian-installer to flash... done.
Please reboot your QNAP device.
Reboot your NAS box. The NAS will start to boot and the status led will flash red/green for a short while before turning solid green and then it will give a short beep to indicate it is ready.
Debian Installer Setup
As the NAS doesn't have a display you need to SSH into the installer. If the NAS was setup before with a static IP it is likely this IP will still be used, if not it may have booted from DHCP. Login to SSH with the username: installer and the password: install.
Follow the installer GUI to setup the NAS box as required. The Installer Guide may help with the options available. I setup RAID0 to maximise my available disk space and otherwise used quite standard setup options. If I re-install the box again I will further detail these setup pages. For disk partitioning I used the 'Guided Partitioning' option and chose to use the entire disk (without LVM) and to basic partition split. After this I setup RAID0 choosing the two large partitions to RAID.
Setup Debian
Setup your Debian installation as required, I setup Apache, MySQL, and Samba to start with. Format your RAID partition using mke2fs and mount it by adding it to /etc/fstab.
Install Packages
Installed required packages in order to get crashplan working.
apt-get install openjdk-6-jdk
apt-get install make
apt-get install gcc
apt-get install libjna-java
Install CrashPlan
cd /usr/
wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.0.1_Linux.tgz
tar xfz CrashPlan_3.0.1_Linux.tgz
cd CrashPlan-install
./install.sh
Choose default paths for installation but choose NOT to install the embedded java runtime.
Patch jtux
Download the jtux patch file from this discussion post.
cd /usr/
wget http://basepath.com/aup/jtux/jtux.tar.gz
tar zxfv jtux.tar.gz
Copy jtux.PS3-YDL6.1.patch.txt into jtux path
patch <jtux.PS3-YDL6.1.patch.txt
nano Makefile
Set JAVA_INCLUDE = /usr/lib/jvm/java-6-openjdk/include
make
I have now attached my compiled version of the jtux library: libjtux.so
Backup existing crashplan jtux library:
cp /usr/local/crashplan/libjtux.so /usr/local/crashplan/libjtux.so.bak
Copy new file in:
cp libjtux.so /usr/local/crashplan/
Patch CrashPlanEngine:
cd /usr/local/crashplan/bin/
nano CrashPlanEngine
Replace line starting with FULL_CP:
FULL_CP="/usr/share/java/jna.jar:$TARGETDIR/lib/com.backup42.desktop.jar:$TARGETDIR/lang"
Start CrashPlan Engine:
./CrashPlanEngine start
At this point your CrashPlan engine should be starting, check its status with ./CrashPlanEngine status
Be patient though, it may take a few minutes for the service to come up fully. Check whether it is listening with: netstat -na | grep 4243 and see if any ports are open, once they are you should be able to connect using the CrashPlan desktop client.
CrashPlan Client
Configure your desktop client as per the guide on the CrashPlan support site: Configuring a Headless Client. I used Putty to SSH into the box, creating a tunnel and then connected using the client software.
You should now be able to login and setup backups as normal.
I selected my raid disk directory for backup and left it to upload my entire backup storage to the cloud!
If I have time I want to perform some further tests in order to see if I could have re-compiled the jtux library on the QNAP firmware without installing Debian and also I would like to provide some more detail (for my own reference as much as anything) on how the RAID partitions were setup.
Although not tested, I believe this method will work for any Marvel cpu device, including the QNAP TS-110, QNAP TS-119, QNAP TS-119P+, QNAP TS-210, QNAP TS-219, QNAP TS-219P QNAP TS-219P+, Sheevaplug and ReadyNAS equivalents. Please let me know if you have installed on any of these or any other devices.
Good luck!
November 22nd, 2011 - 22:17
Philipp,
I ran into the same stack trace:
java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
…
while installing crashplan on my Iomega ix4-200d NAS.
The problem was linked to jna-3.2.5.jar and libffi.so.5
I did put together a tutorial to set up crashplan on my NAS. I am sure you can take ideas from there http://vincesoft.blogspot.com/2011/11/how-to-install-crashplan-on-iomega.html
October 8th, 2011 - 16:14
Hello ! I’m not using CrashPlan at all, but found your comment here while googling for ‘qcontrol’… i found your tip very useful and spread the word in here : http://www.logicite.fr/2011/10/un-serveur-economique-pour-petits-groupes-de-travail-2/
With a reference to your website, i hope you don’t mind.
Laurent
May 27th, 2011 - 02:36
What did you do about the jniwrapper?
April 2nd, 2011 - 19:28
I own a TS-419P+ and was out there fighting JTux, too. But I succeeded and made a build of jtux on the original QNAP system using the header files of a normal JDK for x86 Linux and the current ejre-1_6_0_21-fcs-b09-linux-arm-sflt-eabi-headless. I got CrashPlan to start and work but suddenly it broke down with an error message like this:
Exception in thread “W8949978_ScanWrkr” java.lang.NoClassDefFoundError:
Could not initialize class com.code42.jna.inotify.InotifyManager
at
com.code42.jna.inotify.JNAInotifyFileWatcherDriver.(JNAInotifyFileWatcherDriver.java:21)
at
com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:376)
at
com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:314)
at
com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:64)
at
com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1007)
at com.code42.utils.AWorker.run(AWorker.java:158)
at java.lang.Thread.run(Thread.java:619)
Until today I was unable to solve this and ever since that the service wont even start and shows no error message. Any thoughts on the JNIWrapper Library and its compat with ARM?
March 18th, 2011 - 07:48
Yeah the 3.0.3 upgrade helped significantly with the upload speed. Now I get my full bandwidth on the TS-110. I had around 80-100 kbps on TS-110 on 3.0.2, changing to a PC based system increased the uploads to roundabout 400 kbps before updating. After the update I had 800 kbps on the PC, so I gave it a try and changed back to the TS-110. Same speed !
Only the 100% CPU utilization on TS-110 is still there.
Strange, but I can surely live with that
If I change my server system as mentioned below I will post back. I’m still planning on that due to the option to utilize more HDDs and have more CPU power. But right now I really like this little TS-110.
Thanks again Paul for your help and this blog!
March 17th, 2011 - 08:36
I’m happy to report 3.0.2 has been released and upload speeds are back up! Now uploading at about 1.2Mbps!
March 6th, 2011 - 18:57
Hans,
)
Yeh it is pretty slow still since 3.0.2, I’m holding out for 3.0.3 to come out in the next week or so (hopefully) and see if they have fixed the higher CPU requirements and speed the upload up again. In terms of building another system it is possible that with more modern CPUs that you could get a better speed to power usage ratio, of course if you dont mind using more power to run the system then I’d go for a slightly higher spec board. I’m now running quite a few things on the TS-210 without much issue, including streaming music, file sharing, crashplan, apache, mysql, etc, etc and it is coping very well. I think the main problem is just crashplan have introduced an in-efficient or compression greedy algorithm in their code in the last release that is too much for these low power systems. I’ll report back when the new release finally comes out and update on my new faster speed (i hope).
Please post back if you try some new hardware and what your results are.
March 6th, 2011 - 15:38
Paul,
I probably return to my first idea to change my NAS. I can live with the actual rsync backups, I managed the faults I had on the Qnap software, fine so far.
But this tiny machine is soo weak, there are no problems with the CrashPlan upload speed issue on other computers around here. Btw … switching the ‘watch file system’ and ‘back up open files’ to off in the advanced settings helped a bit. Not too much, but I got it a bit faster (my subjective feeling).
In a couple of weeks I plan to change to a ITX (Atom or AMD Ontario 9 Watts if available) board with Ubuntu Server or Amahi6. Any ideas/experiences on that?
February 22nd, 2011 - 19:30
In my implementation I didn’t specifically set anything up to start the service on boot, it automatically setup to start when I installed it. I know that it sometimes takes a few minutes for the service to fully start as when I first installed it I thought the same. I believe that you can add an entry in rc0 to do this yourself but havnt myself, if I get chance I’ll take a look and post my results!
Paul
February 21st, 2011 - 15:34
Paul,
since I’m not into Linux too much, I have some concerns about the automatic startup of the crashplan engine. Happy that samba starts automatically and the ssh thing …
Okay, I can start the engine manually. Today I looked in the rc*.d directories but I’m unsure to change things there and may kill the whole system
How do you start the engine? Also I’m not sure if it should be stopped during shutdown.
Hans
February 18th, 2011 - 18:01
This seems to be an issue in version 3.0.2 of Crashplan, it looks like they are working on a new release that will fix the bug. See here: https://crashplan.zendesk.com/entries/440487.
A few people with either large amounts of data or slow systems have experienced this bug by the looks of things and has slowed things down for me since the release of 3.0.2. Hopefully they will get the updated version released soon and also hope that fixes the speed issue!
I’ll post here if I come up with any other ideas or hear anything about that release!
February 18th, 2011 - 15:25
Crashplan software works fine and continously for about a week now.
But I’m still not happy with the upload rate which is 80-100 kbps, ~ 1/10 of what should be possible with my ISP. This leads roundabout to 1 GB per day. It will take years for all my QNAP data.
BTW … the Java process takes up to 100% of kirkwoods cpu time. What the hell is this software doing all the time?
Some thoughts or findings about this issue?
February 12th, 2011 - 18:18
Paul,
SSL tunneling works fine now … although i changed nothing. I guess yesterday I had a couple of active rsnyc processes on the usb ports simultaneously, taking that much cpu that the tunnel connection timed out. Now that everything is back on the QNAP disk it connects promptly. CrashPlan needs really a lot of top cpu time.
I changed the de-duplication setting according to your hint and I recognized that big compressed files go pretty slow through the connection but crashplan rushes over a couple of small files with high speed. I will continue to watch this behaviour and report any news here.
And I will add your fanspeed.sh in the next days, good thing – thank you – but right now the box is placed in a pretty cool room so I have no hurry.
February 12th, 2011 - 11:59
Because the Debian install does not automatically control the fan speed on the QNAP I have now added a script that checks the temperature and sets the fan speed accordingly. You may want to change the temperatures to suit but here it is:
Create a fanspeed script:
nano /usr/fanspeed.sh
#!/bin/sh
TEMPERATURE=$(/usr/sbin/hddtemp -n /dev/sda)
QCONTROL=/usr/sbin/qcontrol
echo $TEMPERATURE;
$QCONTROL -d &
sleep 5
if [ $TEMPERATURE -le 38 ]; then
$QCONTROL fanspeed stop
elif [ $TEMPERATURE -gt 37 ] && [ $TEMPERATURE -le 43 ]; then
$QCONTROL fanspeed silence
elif [ $TEMPERATURE -gt 42 ] && [ $TEMPERATURE -le 48 ]; then
$QCONTROL fanspeed low
elif [ $TEMPERATURE -gt 47 ] && [ $TEMPERATURE -le 53 ]; then
$QCONTROL fanspeed medium
elif [ $TEMPERATURE -gt 52 ] && [ $TEMPERATURE -le 58 ]; then
$QCONTROL fanspeed high
else
$QCONTROL fanspeed full
fi
sleep 5
/usr/bin/killall qcontrol
sleep 2
rm /var/run/qcontrol.sock
Then schedule the fan speed to be checked and updated every minute:
nano /etc/crontab
Add:
*/1 * * * * root cd /usr/ && ./fanspeed.sh
February 12th, 2011 - 08:46
Hans,
I’m glad this was helpful!
It’s a bit odd with the SSL tunnelling not working, I know I set it up once, saved the connection in Putty and then it has always worked first time. Sometimes it does take the Crashplan GUI a minute to connect. Also I have noticed after a power up or restart of the NAS box it takes a few minutes for Crashplan engine to fully start. You can check this by doing a:
netstat -na | grep 4243If ports are listed then it has started, if not then it might still be booting.
The slow upload I think will be caused by the data de-duplication setting in the Crashplan GUI settings, backup, advanced tab. I have turned this to minimum and get around 800kbps->1Mbps which is my line maximum, as opposed to around 200kbps with this set to auto.
February 11th, 2011 - 19:17
Hello Paul,
success on TS-110!
Thank you very much for this post … I found it just before changing my NAS due to faulty behaviour during USB backups. Nothing to loose I thought and started installing debian. For the little TS-11x family there is a special download, see this link
http://www.cyrius.com/debian/kirkwood/qnap/ts-119/
All the rest done as described here. Works.
My USB backups go safely with rsync now, no faults.
I just installed this crashplan thing and had a few problems with the tunneling, sometimes it works sometimes not. Maybe I will find out what’s the reason for this but I surely can live with SSLling to debian twice or so until it works. The upload rate to the crashplan center is a little low … I guess this is due to the slow kirkwood.
Thanks! Hans