Wednesday, January 16, 2013

Howto: Raspberry Pi Root NFS share - boot your System over NFS share and definitively deal with Flash data corruption



*** Updated March 14, 2013  ***


The Goal: 


If you have a Raspberry Pi and a Linux Server (or a NAS), you shall really be interested by this post! :-)

I had a lot of issue with my main Rpi when overclocked generating File System data corruption...
And finally, the real best solution, solid and efficient has been to convert my root installation into booting rootfs over NFS.

I recommend the most easy solution to first have a running installation of your system into your Flash card and simply migrate it to root fs over NFS.

Since i've done this, never had any system freeze, corruption or event kernel panic with my Rpi overclocked to turbo mode :-)

Also, you should ensure before beginning that your system is up to date (sudo apt-get update && sudo apt-get dist-upgrade -f) and your have the last firmware version (sudo rpi-update)

Major source: 

Summary of steps: 

Step 1: Set your NFS share
Step 2: Copy your root fs into your NFS share
Step 3: Modify your Raspberry Pi boot configuration
Step 4: Adapt your Rpi fstab
Step 5: Boot your Rpi!
Step 6: Correct your swap configuration by migrating to a loop device

Memorandum


Step 1: Set your NFS share

If you have a Linux Home Server or NAS, then you probably already share data using NFS.

To set a NFS share dedicated for your Rpi Root fs, add your share into "/etc/exports":
Adapt <raspberrypi_ip> with your Rpi LAN IP or LAN subnet if you prefer
# Raspberry Root FS                                     
/data/rpi_rootfs <raspberrypi_ip>(rw,sync,no_root_squash,no_subtree_check)

Under Debian / Ubuntu, reload your NFS server config:
sudo /etc/init.d/nfs-kernel-server reload


Step 2: Copy your Rpi root fs into your NFS share

Then simply copy all of your Rpi root fs into your new nfs share, you can do it directly under the Rpi or by plugging your Flash card into a client computer:

Example with a client computer having the flash card and NFS share mounted:
cp -rav /media/mmcblk0p2/* /data/rpi_rootfs/ 


Step 3: Modify your Raspberry Pi boot configuration

The only partition you will need to keep in your Rpi Flash card will be the boot partition (first partition), containing main boot configuration files and the Rpi firmware.

To boot over NFS, we need to modify the file "/boot/cmdline.txt" (contained into the first fat partition of your Flash card) to add/correct some sections:
  • root= --> Will be pointing to "/dev/nfs"
  • nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3 ip=dhcp (replace<nfs_server_ip> with your NFS server IP)
  • rootfstype=nfs
  • smsc95xx.turbo_mode=N --> is a workaround to prevent kernel panic under high network load (i recommend this)

Note:

In this example, we use DHCP to set the Rpi Lan IP at boot time, this is in my opinion the easiest way to do as you preset a fix address in your DHCP server for your Rpi.
Still you can also manually a fix IP at boot time.

Also note we will be using NFS V3 running under UDP for better performances. (see Memorandum for performances fine tuning)

"cmdline.txt" example with DHCP (the file must contain only one line):

Replace:
  • <nfs_server_ip> with the NFS server IP
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=dhcp rootfstype=nfs smsc95xx.turbo_mode=N 

"cmdline.txt" example with Fix IP(the file must contain only one line):

Replace:
  • <raspberrypi_ip> with the Lan IP of your Rpi
  • <nfs_server_ip> with the NFS server IP
  • <default_gateway> with the IP of your local gateway
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1
root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfs,udp,vers=3
ip=<raspberrypi_ip>:<nfs_server_ip>:<default_gateway>:<mask>:rpi:eth0:off rootfstype=nfs smsc95xx.turbo_mode=N


Step 4: Adapt your Rpi fstab

Now edit the Rpi "/etc/fstab" file before trying to boot, do under your NFS server or client computer.

/data/rpi_rootfs/etc/fstab:
  • Delete the original line corresponding to your root fs and pointing to the second partition of your flash card  (/dev/mmcblk0p2), we don't need anymore as it will automatically be mounted by the firmware at boot time

Step 5: Boot your Rpi!

Ok, let's go, time to boot :-)

If you follow all steps carefully, you system should boot with no major issue.

Therefore, you will not have anymore swap available, by default Raspbian uses dphys-swapfile os use a local file as swap.

We will correct this now.


Step 6: Correct your swap configuration by migrating to a loop device

By default, Raspbian uses dphys-swapfile to generate a local file being used as swap, this won't work anymore when booting under NFS.

I don't recommend to use your Flash card as a swap partition, this may generates system freeze or kernel panics if you have data corruption:

The better way is to set a local file as a loop device that will be used a swap device, here is how.

Clean current non working swap file and uninstall dphys-swapfile:
sudo apt-get remove --purge dphys-swapfile
sudo rm /var/swap
sudo rm /etc/init.d/dphys-swapfile
sudo update-rc.d dphys-swapfile remove

Create a new swap file, create the loop swap device and activate swap (exemple with 1GB swap):
sudo dd if=/dev/zero of=/var/swapfile bs=1M count=1024
sudo losetup /dev/loop0 /var/swapfile
sudo mkswap /dev/loop0
sudo swapon /dev/loop0

Check your current swap availability:
$ free

Output example:
             total       used       free     shared    buffers     cached
Mem:        237656     213092      24564          0         24      93192
-/+ buffers/cache:     119876     117780
Swap:      1048572       1556    1047016

Make it permanent, edit "/etc/rc.local" and add this section before "exit 0":
echo "Setting up loopy/var/swapfile.."
sleep 2
losetup /dev/loop0 /var/swapfile
mkswap /dev/loop0
swapon /dev/loop0


Step 7: Other tunings


There is also some other little thing to tune:

Edit "/etc/default/rcS" and:
  • add:ASYNCMOUNTNFS=no

Edit "/etc/sysctl.conf" and:
  • add or set: vm.min_free_kbytes = 12288 
This will ensure the system will always have 12Mb or RAM free to prevent kernel panic risk, your may try lower value if your prefer.


Memorandum

  • NFS version and fine tuning
You may want to try different settings to get the better performance possible.

First, if you test your write speed, using dd will be very easy:

Create a 10 Mb file test:
$ dd if=/dev/zero of=/tmp/test.file bs=1M count=10

Output sample:
10+0 enregistrements lus                                                                                                      
10+0 enregistrements écrits                                                                                                   
10485760 octets (10 MB) copiés, 1,52525 s, 6,9 MB/s

You may want to try with different NFS version, change in cmdline.txt:
  • Change the section nfsroot "vers=2/3"
You may want to try TCP versus UDP
  • Change the section nfsroot  "udp" or "tcp"
You may want to try different values of "rsize" and "wsize", example with NFS V3 and TCP:

Example:
  • root=/dev/nfs nfsroot=<nfs_server_ip>:/data/rpi_rootfsrsize=32768,wsize=32768,tcp,vers=3

In my case, it did not really change anything, so i kept kernel default values for wsize and rsize, udp with NFS V3.

  • Netfilter Iptables
When modifying your Iptables configuration, keep in mind that NFS traffic with your NFS server will result in system halt.

Applying default outbound policy to DROP (usually "iptables -P OUTPUT DROP") will in system crash.
You should apply instead "iptables -P OUTPUT ACCEPT" which will permit any outbound traffic from your Rpi (not a big deal, usually you trust your own machine)

Also you can ensure to accept NFS traffic with your NFS server before applying any other rules.


















39 comments:

  1. Hi there, thanks for the nice article ! I will try this.

    Still, i got a little question : if I boot on nfs, will it prevent the NAS disks to go to sleep ? Maybe it depends on the NAS model. Have you checked that ? Or did you boot only on a nfs linux server and not a NAS ? Thanks for your answer.

    ReplyDelete
    Replies
    1. Hi, thanks :-)

      In deed, it will prevent your NAS from going to spleep, you will always have I/O operations with a running system, moreover with swap over NFS...
      And your NAS will only go to sleep if there no I/O requests.

      The only solution would be to prevent NFS I/O by keeping one part of your system on your Flash card (the /var mount point) and swap on flash.
      But if you have data corruption, you will still loose your system.

      To answer your question, my NFS server is a Linux server

      Cheers

      Delete
    2. I'm gonna try booting from my Linkstation NAS, it's running Debian Squeeze.
      I have installed Laptop Mode Tools to enable the NAS hard drive to spin down when idle.
      The Laptop Mode Tools package does more than just spin down the drive though - it caches I/O requests and then flushes that cache to the drive either:
      1) When the cache reaches a user configurable % of system memory.
      2) When a user configurable time period has elapsed.

      So my drive spins down and unless there's any file serving going on it will remain spun down until 60% of memory is used by the cache or 6 hours have passed.

      Maybe Joe can get Laptop Mode Tools running and see if it help?

      Delete
    3. Hi,

      Thanks for the update, good to know and songs great, let us know if it works as expected!

      Delete
    4. Right - first the good news.
      Following your instructions i have managed to boot my Pi from my Linkstation NAS.
      It works well and has been running all day.

      Bad news.
      The Linkstation drive has not spun down once all day.
      Not sure if it's the ongoing logging that's keping it spun up or whether some system policy prevents a spin down as long as a client is connected to the NFS share.

      Martin.

      Delete
    5. Hi Martin,

      Good news, at least for the first :-)

      For testing purposes, you could create a local partition on the flash and mount /var/log under it, i think most of I/O should come there and swap (if you swap over NFS)

      Let me know ^^

      Delete
    6. Well i created and mounted a swap partition on my SD card, then removed all traces of the NFS swap.
      Then mounted var/log to my SD card.

      Unfortunately the NAS harddrive still fails to spin down.
      It's not a showstopper for me so i'll not spend any more time on it.

      Delete
    7. Youresuchageek: Howto: Raspberry Pi Root Nfs Share - Boot Your System Over Nfs Share And Definitively Deal With Flash Data Corruption >>>>> Download Now

      >>>>> Download Full

      Youresuchageek: Howto: Raspberry Pi Root Nfs Share - Boot Your System Over Nfs Share And Definitively Deal With Flash Data Corruption >>>>> Download LINK

      >>>>> Download Now

      Youresuchageek: Howto: Raspberry Pi Root Nfs Share - Boot Your System Over Nfs Share And Definitively Deal With Flash Data Corruption >>>>> Download Full

      >>>>> Download LINK eF

      Delete
  2. Hello,

    Thanks for your answer; well, i think that i will boot on a linux server too; i got one running here. Can't wait to receive my rasberry now :)

    ReplyDelete
  3. Hi,

    I can't find RAMRUN or RAMLOCK in /etc/default/rcS. Is this outdated or can I add the lines and it'll work?

    Thanks

    ReplyDelete
    Replies
    1. Hi, you're right, these 2 settings have disappeared, i guess in some recent updates.

      Works perfectly without, so don't add.

      I'll update.

      Cheers

      Delete
    2. Those settings are in /etc/defaults/tmpfs now

      Delete
    3. Hi, thanks for the information, will check and update

      Delete

Please feel free to comment ^^