Many years ago I built myself a home ‘server’ – a glorified NAS. It was mainly a media PC, and I used it to store a bunch of my data. Mainly big stuff like videography and desktop publishing files.
When I set it up, I used an onboard RAID controller. It worked well … right up until the point when it didn’t anymore, and I was wasting so much RAID storage on files which I didn’t need protected, like movies and TV shows. I started looking for a better solution, and found Greyhole.
My two favourite things about greyhole are:
- Simplicity of access – files are stored on a standard ext4 file system, and in the event of a system failure, can be plugged into a USB reader and accessed instantly.
- No wasted space – redundancy is managed at a folder level, meaning you can have one folder with 5 copies of redundancy, and one folder with only one. Both folders co-existing on the same JBOD and presented seamlessly.
I’ve been using greyhole for about 8 years and it’s been great. Every now and then I have to replace a drive, and because I rarely use linux, I have to re-learn every time. Here’s a very rough guide on how to do it:
Figuring out what drives you have in your system
I built this system years ago, so I don’t remember how it’s set up! Here’s how to find out:
sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
This will give you an easy to read display of your disks, their partitions, and their mount points. Here’s mine:
doble@doblesrv:~$ sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL NAME FSTYPE SIZE MOUNTPOINT LABEL sda 1.8T ├─sda1 ext4 1.8T / ├─sda2 1K └─sda5 swap 7.6G [SWAP] sdb 1.8T └─sdb1 ext4 1.8T /mnt/sdb1 sdc 1.8T └─sdc1 ext4 1.8T /mnt/sdd1 sdd 1.4T └─sdd1 ext4 1.4T /mnt/sde1 sde 2.7T ├─sde1 128M └─sde2 ntfs 2.7T
In my case, my drive /dev/sdc failed so I have replaced it. Note how this changed the disk labels – but the mounting points are still correct. (/dev/sde is the replacement drive)
Figuring out which drive is failing
Hopefully you’ve set up Greyhole to email you when a disk isn’t accessible anymore, but in some cases a disk might be partially working, or may just suffer degraded performance.
If your disk is still accessible, you can use SMART to find out if a disk is on the way out, or has failed.
If the disk isn’t accessible anymore, use the command above to list your current disks, and compare it to what’s listed in /etc/fstab
doble@doblesrv:~$ sudo pico /etc/fstab # /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc nodev,noexec,nosuid 0 0 # / was on /dev/sda1 during installation UUID=dbde690d-9e4c-4b0d-9339-a04de6612260 / ext4 errors=remount-ro 0 1 # swap was on /dev/sda5 during installation UUID=79732b91-7117-4170-80d8-20c1d48deff0 none swap sw 0 0 UUID=c8566de8-c8b8-44be-bef8-f26f38a97cd6 /mnt/sdb1 ext4 noatime,data=writeback,nofail 0 0 #UUID=fc5078a4-0de6-45d1-972b-b7f378ecd2c2 /mnt/sdc1 ext4 noatime,data=writeback,nofail 0 0 #UUID=fbce0da6-34cf-464d-a879-393c5ecb068c /mnt/sdc1 ext4 noatime,data=writeback,nofail 0 0 UUID=f188642f-cf98-4fd0-8747-5ea2f1bcfd3b /mnt/sdd1 ext4 noatime,data=writeback,nofail 0 0 UUID=eedadf3e-19d8-4f87-963e-be86c3ad0171 /mnt/sde1 ext4 noatime,data=writeback,nofail 0
You should see a mount point missing from your lsblk command. While you’ve got the fstab open, it’s probably a good time to comment out the failed drive so your system doesn’t keep trying to mount it.
You can see I’ve already had two drives fail in the past, both were sdc – just lucky I guess!
Figuring out which drive is which
When you have a disk fail, or start failing, the OS reports it as something like ‘/dev/sdc’ – which isn’t very helpful for identifying which actual drive you need to unplug!
Thankfully, disks have a sticker with a serial ID, and you can get the OS to report it by running this command (replace the /dev/sda with the disk you want to query)
udevadm info --query=all --name=/dev/sda | grep ID_SERIAL
Note that the disk order can change, so if you unplug /dev/sdb and reboot, /dev/sdc might become /dev/sdb ! For this reason, ensure that you are querying the right disk by checking the mount points with:
sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABELHowever, if your disk has failed completely, you might not be able to query it – so instead, query every other disk and work out which one is missing.
Once you’ve found the right one, shut down your machine and replace it.
Formatting the replacement drive
Here’s a guide from the Ubuntu wiki on how to do that.
If you want to use a disk larger than 2TB, you’ll need to use Parted, here’s a different guide.
Mounting the replacement drive
We want the partition to mount every time the machine boots, so we need to add it to fstab. First, you’ll need the UUID of the partition. Copy it to your clipboard.
ls -l /dev/disk/by-uuid
Then, open your fstab in a text editor. I use pico because I’m a noob.
sudo pico /etc/fstab
I usually copy the line which I commented out from the broken drive, and replace the UUID, this way the new drive gets mounted to the same location.
At this point, I’d suggest rebooting to test that your disk mounts up correctly, and should also get the drive letters matching their mounting points again.
doble@doblesrv:~$ sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL sda 1.8T ├─sda1 ext4 1.8T / ├─sda2 1K └─sda5 swap 7.6G [SWAP] sdb 1.8T └─sdb1 ext4 1.8T /mnt/sdb1 sdc 2.7T └─sdc1 ext4 2.7T /mnt/sdc1 sdd 1.8T └─sdd1 ext4 1.8T /mnt/sdd1 sde 1.4T └─sde1 ext4 1.4T /mnt/sde1
Looking good!
Telling Greyhole that we’ve replaced the drive
Lets check to see if Greyhole knows what is up with that failed/replaced drive
doble@doblesrv:~$ greyhole -s Greyhole Statistics =================== Storage Pool Total - Used = Free + Trash = Possible /gh: 1826G - 1687G = 46G + 0G = 46G /mnt/sdb1/gh: 1834G - 1745G = 0G + 0G = 0G /mnt/sdc1/gh: Offline /mnt/sdd1/gh: 1834G - 1746G = 0G + 0G = 0G /mnt/sde1/gh: 1375G - 1297G = 9G + 0G = 9G ========================================== Total: 6869G - 6474G = 55G + 0G = 55G
Yep – it’s gone alright! Next we need to make the ‘gh’ folder for Greyhole (change your drive letter to suit)
doble@doblesrv:/$ sudo mkdir /mnt/sdc1/gh
Finally, we tell Greyhole that the drive has been replaced (change your drive letter to suit)
doble@doblesrv:/$ sudo greyhole --replace=/mnt/sdc1/gh Storage pool drive /mnt/sdc1/gh has been marked replaced. The Greyhole daemon will now be restarted to allow it to use this new drive.
Greyhole will now start using the disk, however note that it will only copy new files, or shares which aren’t currently meeting their minimum number of copies as set in your greyhole config.
Here’s what mine looks like after a few hours:
doble@doblesrv:/mnt/sdc1/gh/Photography$ greyhole -s Greyhole Statistics =================== Storage Pool Total - Used = Free + Trash = Possible /gh: 1826G - 1687G = 47G + 0G = 47G /mnt/sdb1/gh: 1834G - 1745G = 0G + 0G = 0G /mnt/sdc1/gh: 2751G - 141G = 2470G + 0G = 2470G /mnt/sdd1/gh: 1834G - 1746G = 0G + 0G = 0G /mnt/sde1/gh: 1375G - 1297G = 9G + 0G = 9G ========================================== Total: 9619G - 6615G = 2525G + 0G = 2525G
Balancing disk usage
As you can see my other drives are all full, mainly because I’ve been running with a broken drive for a few months. To balance the disk usage, use the command:
doble@doblesrv:/$ sudo greyhole --balance A balance has been scheduled. It will start after all currently pending tasks have been completed. This operation will try to even the available space on all drives included in your storage pool.
You can use the ‘status’, ‘log’ and ‘iostat’ commands to view what greyhole is doing
Status: greyhole -S
Logs: greyhole -L
Disk IO status: greyhole -i
doble@doblesrv:/$ greyhole -S Currently working on task ID 501548: balance / Recent log entries: Feb 01 13:24:38 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5675.CR2 Feb 01 13:24:38 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5675.CR2 Feb 01 13:24:38 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5675.CR2 Feb 01 13:24:38 7 balance: Balancing storage pool drive: /gh (-53.2GB available, target: 445GB) Feb 01 13:24:38 7 balance: Working on file: Photography/Wedding/IMG_5580.CR2 (23.2MB) Feb 01 13:24:38 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:24:38 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:24:38 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:24:38 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:24:38 7 balance: Moving file copy... Last logged action: balance on 2018-02-01 13:24:38 (11s ago)
doble@doblesrv:/$ greyhole -L Feb 01 13:25:37 7 balance: Balancing storage pool drive: /gh (-50.7GB available, target: 445GB) Feb 01 13:25:37 7 balance: Working on file: Photography/Wedding/IMG_5453.CR2 (23.0MB) Feb 01 13:25:37 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:37 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:37 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:37 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:37 7 balance: Moving file copy... Feb 01 13:25:37 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5453.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5453.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5453.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5453.CR2 Feb 01 13:25:37 7 balance: Balancing storage pool drive: /gh (-50.6GB available, target: 445GB) Feb 01 13:25:37 7 balance: Working on file: Photography/Wedding/IMG_5808.CR2 (23.8MB) Feb 01 13:25:37 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:37 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:37 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:37 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:37 7 balance: Moving file copy... Feb 01 13:25:37 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5808.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5808.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5808.CR2 Feb 01 13:25:37 7 balance: Saving metadata in /mnt/sde1/gh/.gh_metastore/Photography/Wedding/IMG_5808.CR2 Feb 01 13:25:37 7 balance: Balancing storage pool drive: /gh (-50.6GB available, target: 445GB) Feb 01 13:25:38 7 balance: Working on file: Photography/Wedding/IMG_5679.CR2 (21.7MB) Feb 01 13:25:38 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:38 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:38 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:38 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:38 7 balance: Moving file copy... Feb 01 13:25:38 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5679.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5679.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5679.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5679.CR2 Feb 01 13:25:38 7 balance: Balancing storage pool drive: /gh (-50.6GB available, target: 445GB) Feb 01 13:25:38 7 balance: Working on file: Photography/Wedding/IMG_5432.CR2 (22.9MB) Feb 01 13:25:38 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:38 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:38 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:38 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:38 7 balance: Moving file copy... Feb 01 13:25:38 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5432.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5432.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5432.CR2 Feb 01 13:25:38 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5432.CR2 Feb 01 13:25:38 7 balance: Balancing storage pool drive: /gh (-50.6GB available, target: 445GB) Feb 01 13:25:38 7 balance: Working on file: Photography/Wedding/IMG_5500.CR2 (22.6MB) Feb 01 13:25:38 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:38 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:38 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:38 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:38 7 balance: Moving file copy... Feb 01 13:25:39 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5500.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5500.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5500.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5500.CR2 Feb 01 13:25:39 7 balance: Balancing storage pool drive: /gh (-50.6GB available, target: 445GB) Feb 01 13:25:39 7 balance: Working on file: Photography/Wedding/IMG_5529.CR2 (23.1MB) Feb 01 13:25:39 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:39 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:39 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:39 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:39 7 balance: Moving file copy... Feb 01 13:25:39 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5529.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5529.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5529.CR2 Feb 01 13:25:39 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5529.CR2 Feb 01 13:25:39 7 balance: Balancing storage pool drive: /gh (-50.5GB available, target: 445GB) Feb 01 13:25:39 7 balance: Working on file: Photography/Wedding/IMG_5832.CR2 (22.4MB) Feb 01 13:25:39 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:39 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:39 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:39 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:39 7 balance: Moving file copy... Feb 01 13:25:40 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5832.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5832.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5832.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sde1/gh/.gh_metastore/Photography/Wedding/IMG_5832.CR2 Feb 01 13:25:40 7 balance: Balancing storage pool drive: /gh (-50.5GB available, target: 445GB) ^XFeb 01 13:25:40 7 balance: Working on file: Photography/Wedding/IMG_5502.CR2 (23.8MB) Feb 01 13:25:40 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:40 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:40 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:40 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:40 7 balance: Moving file copy... Feb 01 13:25:40 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5502.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5502.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5502.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5502.CR2 Feb 01 13:25:40 7 balance: Balancing storage pool drive: /gh (-50.5GB available, target: 445GB) Feb 01 13:25:40 7 balance: Working on file: Photography/Wedding/IMG_5798.CR2 (22.6MB) Feb 01 13:25:40 7 balance: Drives with available space: /mnt/sdc1/gh (2.36TB avail) Feb 01 13:25:40 7 balance: Drives with enough free space, but no available space: /mnt/sde1/gh (-41.3GB avail) - /gh (-53.4GB avail) Feb 01 13:25:40 7 balance: Drives full: /mnt/sdb1/gh (0.00B free) - /mnt/sdd1/gh (0.00B free) Feb 01 13:25:40 7 balance: Target drive: /mnt/sdc1/gh (2.36TB available) Feb 01 13:25:40 7 balance: Moving file copy... Feb 01 13:25:40 7 balance: Saving 3 metadata files for Photography/Wedding/IMG_5798.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdb1/gh/.gh_metastore/Photography/Wedding/IMG_5798.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdc1/gh/.gh_metastore/Photography/Wedding/IMG_5798.CR2 Feb 01 13:25:40 7 balance: Saving metadata in /mnt/sdd1/gh/.gh_metastore/Photography/Wedding/IMG_5798.CR2 Feb 01 13:25:40 7 balance: Balancing storage pool drive: /gh (-50.5GB available, target: 445GB) ^C
doble@doblesrv:/$ greyhole -i /gh: 91992 kBps /mnt/sdb1/gh: 14 kBps /mnt/sdc1/gh: 88632 kBps /mnt/sdd1/gh: 5 kBps /mnt/sde1/gh: 0 kBps --- /gh: 59350 kBps /mnt/sdb1/gh: 27 kBps /mnt/sdc1/gh: 60083 kBps /mnt/sdd1/gh: 36 kBps /mnt/sde1/gh: 8 kBps --- /gh: 56573 kBps /mnt/sdb1/gh: 33 kBps /mnt/sdc1/gh: 54761 kBps /mnt/sdd1/gh: 35 kBps /mnt/sde1/gh: 9 kBps --- ^C
That’s it – depending on how much data you have to move, Greyhole should eventually re-balance your storage drives.
Hope this helped you!