HifiZine
The enthusiast's audio webzine

PiDisk – syncing the music library

This is the third part of a short series on PiDisk – an inexpensive music server based on the Raspberry Pi.

• Part 1: An inexpensive music server – introduces the concept of the PiDisk and sets it up as a basic file server.
• Part 2: Adding the music services – adds MinimServer and BubbleUPnP Server to the PiDisk so that it is a functioning music server (including streaming TIDAL and/or Qobuz).
• Part 3: Syncing the music library (this article) – explains how to automatically synchronize your music library to the PiDisk over the network, and concludes with a few additional tips.

This article continues right where Part 2 left off. This time, I am definitely assuming that you are using a Mac as your desktop computer. It’s just a pragmatic decision, but Linux users should not have much trouble adapting. For Windows users, I’m not sure what to suggest, although the Linux shell that Windows 10 is about to acquire might be worth a look.

Sections in Part 3:

• 12. Set up RSA authentication
• 13. Load music
• 14. Set up automatic updates
• 15. Additional tips
• Wrapping up

12. Set up RSA authentication

It’s time to set up RSA authentication to the PiDisk. This will not only save you from having to type your password every time you connect to it with a shell, but it’s necessary for the script that will automatically update the music library.

On your Mac, you will need to have a public/private key pair. You may already have these, so check first, as you do not want to overwrite any that you already have:

mymac:~ myname$ ls ~/.ssh
id_dsa id_dsa.pub id_rsa id_rsa.pub known_hosts

(A reminder: most of the commands in this article can simply be copied and pasted from the web browser into your Terminal window. You don’t have to re-type them.)

If the response as above shows the files id_rsa and id_rsa.pub, do not do the next step. If those files are not there, or if you get a “file or directory not found” error, execute the following command. When asked for a passphrase, just press the Return key without entering anything:

mymac:~ myname$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/pi/.ssh/id_rsa): 
Created directory '/home/pi/.ssh'.
Enter passphrase (empty for no passphrase):  <Return>
Enter same passphrase again:  <Return>
Your identification has been saved in /home/pi/.ssh/id_rsa.
Your public key has been saved in /home/pi/.ssh/id_rsa.pub.

Now copy your public key to the Raspberry Pi:

mymac:~ myname$ cat ~/.ssh/id_rsa.pub | ssh pi@pidisk.local "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

At this point, your public key should have been copied correctly to the PiDisk and you will be able to just log into it without typing a password. The first time, you may need to enter a “yes” confirmation, but thereafter you should be able to just log in and execute a command remotely. For example, try this:

mymac:~ myname$ ssh pi@pidisk.local ls /media/pidisk
Music
mymac:~ myname$

13. Load music

So far, you’ve been able to copy files onto the PiDisk with the Finder. For large numbers of files, though, this is not a very robust method. Better to use a program that is designed for copying large numbers of files over the network.

Log into the Pi and install rsync:

pi@pidisk:~ $ sudo apt-get install rsync

Back on your Mac, use TextEdit to create a file (“script”) named pisync.sh. I suggest that you create it in the folder Documents/Scripts, and will assume that in the following.

Paste the following text into the text editor and save the file:

#!/bin/sh
/usr/bin/rsync -a --delete ~/Music/iTunes/iTunes\ Media/Music/ pi@pidisk.local:/media/pidisk/Music
ssh pi@pidisk.local "echo rescan | ~/minimserver/bin/mscript"

This is what each line of this script means:

• The first line is the shebang.
• The second lines uses rsync to copy your whole iTunes music folder to the Music folder on the pidisk.
• The third line causes MinimServer to rescan the library so that new files show up in your controller app.

If your music is not in iTunes an iTunes library, then you will need to modify this script. For example, this version copies a music library located on an external hard drive named “DataHD” to the PiDisk:

#!/bin/sh
/usr/bin/rsync -a --delete /Volumes/DataHD/Music/ pi@pidisk.local:/media/pidisk/Music
ssh pi@pidisk.local "echo rescan | ~/minimserver/bin/mscript"

You can also use multiple rsync commands to copy files from multiple locations. If you do this, you will need each rsync command to write to a separate sub-directory of the PiDisk music folder. Here is an example:

#!/bin/sh
/usr/bin/rsync -a --delete ~/Music/iTunes/iTunes\ Media/Music/ pi@pidisk.local:/media/pidisk/Music/iTunes
/usr/bin/rsync -a --delete ~/Music/Samples/ pi@pidisk.local:/media/pidisk/Music/Samples
/usr/bin/rsync -a --delete /Volumes/DataHD/Music/ pi@pidisk.local:/media/pidisk/Music/Main
ssh pi@pidisk.local "echo rescan | ~/minimserver/bin/mscript"

If you need help customizing your rsync script, please ask in a comment below.

As shown above, the rsync command won’t print very much out. When you run the script the first time, you will be able to see it is working by taking a peek at the PiDisk file system in the Finder. If you really want to see everything that rsync is doing, add “‐‐progress” after “‐‐delete” e.g.

/usr/bin/rsync -a --delete --progress ...

Note that the “‐‐delete” means that any files that you delete from your library will also be deleted from the PiDisk. This also means that if you copy any files into the Music folder using the Finder, they will be deleted the next time you run this script. This is most likely what you want, but if not, remove “‐‐delete” from the commands.

To run your file transfer script, open Terminal and type:

mymac:~ myname$ sh ~/Documents/Scripts/pisync.sh

rsync will scan the library on your Mac’s hard drive and start copying files over. The first time, it might take a while…. I get a transfer speed of about 10 MB/sec, consistent with the Raspberry Pi’s 100 MB Ethernet port. That means 36 GB per hour or somewhat over a full day to transfer a Terabyte. Fortunately, you only have to do this once – once initialized, the script will only copy new files each time.

(If this is all too slow for you, you could try initializing the drive directly on your Mac with software that enables it to access an ext4 drive, like ExtFS.)

14. Set up automatic updates

To set up automatic nightly synchronization from a Mac, we will use launchd. (On Linux, you will probably use cron. On Windows, sorry but I have no idea.) You can do this by creating a “plist” file:

mymac:~ myname$ sudo nano /Library/LaunchAgents/com.hifizine.pisync.plist

Paste the following text into the editor. Edit it to change “myname” (two locations) to your username:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
 <dict>
  <key>Label</key>
  <string>com.hifizine.pisync</string>
  <key>Program</key>
  <string>/Users/myname/Documents/Scripts/pisync.sh</string>
  <key>StartCalendarInterval</key>
  <dict>
   <key>Hour</key>
   <integer>05</integer>
   <key>Minute</key>
   <integer>00</integer>
  </dict>
  <key>UserName</key>
  <string>myname</string>
  <key>GroupName</key>
  <string>staff</string>
 </dict>
</plist>

Press Ctrl-O to write the file, hit return at the confirmation prompt, and then press Ctrl-X to exit the editor.

Now tell launchd to use it. (The “sudo” in the following command is important: without it, the command will appear to succeed, but your script will run only if you are logged in.)

mymac:~ myname$ sudo launchctl load /Library/LaunchAgents/com.hifizine.pisync.plist

This is all that should be needed. Every morning at 5AM, launchdaemon will run your file transfer script. So if you add some files to your music library and wait until tomorrow, you should see that the files appear on the PiDisk and in your control app.

Here are some additional commands that you may find handy. To see if the command is loaded:

mymac:~ myname$ sudo launchctl list | grep hifizine
-	0	com.hifizine.pisync

To run the command immediately:

mymac:~ myname$ sudo launchctl start com.hifizine.pisync

If it’s already running, you can stop it with:

mymac:~ myname$ sudo launchctl stop com.hifizine.pisync

To unload it so it doesn’t run at all:

mymac:~ myname$ sudo launchctl unload /Library/LaunchAgents/com.hifizine.pisync.plist

If you want to see what the script has done, add the following to the plist file, just before the last “</dict>” (replace “myname” with you user name):

  <key>StandardOutPath</key>
  <string>/Users/myname/Documents/Scripts/pisync.out.txt</string> 
  <key>StandardErrorPath</key> 
  <string>/Users/myname/Documents/Scripts/pisync.log.txt</string>

Then you can open those files and see what the script has been doing.

15. Additional tips

Any time you need to reboot the PiDisk from your Mac, open Terminal and enter:

mymac:~ myname$ ssh pi@pidisk.local sudo reboot

If you need to shut down the PiDisk (recommended before removing power), open Terminal and enter:

mymac:~ myname$ ssh pi@pidisk.local sudo shutdown now

Then unplug the micro-USB power connector.

Wrapping up

And… that’s a wrap! I’ve been using the PiDisk for a few months now and it has barely skipped a beat (see below). If you’ve followed along, you now have a complete networked music system based on two Raspberry Pis. Total cost is in the 200’s range.

For those who are scratching their heads about what exactly you DO with a PiDisk, please refer to the earlier article “How to STREAM Tidal to the Raspberry Pi.” That article shows you how to set up a control app – on Android, iOS, and Mac OS X. The music library (your files) are served by the PiDisk, while TIDAL is streaming in from the Internet.

How does it sound? Way better than you would think for the cost. But here’s the thing: the sound is determined by the player, not the server. And, you don’t have to use the Raspberry Pi streamer that I described in earlier articles. You can use anything you like, as long as it’s aUPnP renderer. I’ve now added my BDP-103 to BubbleUPnP Server, so if I am sitting near the home theater (which also happens to be where the fireplace is located, and it’s been quite a cold winter in Australia), I can send audio to the BDP-103 and play it through the HT system. Or, if you have a high-end player, you can send audio from the PiDisk to it, and the sound will be just as good when using a PiDisk as when using any other server.

The only time things have gone really wrong with the PiDisk was when I tried to have it transcode from FLAC to WAV – a requirement for JPlay Streamer. The PiDisk actually failed and I had to swap another board in. I  haven’t had time to investigate further. The moral of the story is: don’t push it. The Pi is a neat device that can do a surprising amount of useful stuff, but don’t expect it to do too much. If you do what is in these articles: it works.

All in all, I’m very pleased with this new playback “ecosystem.” I can add new streamers in any room of the house, as well as stream to the existing networked playback devices I already have. Networked playback is an exciting development in hifi and it’s becoming more and more widespread. And, as I hope I’ve shown in these articles, you don’t have to spend much to get started, as long you have a bit of time and determination to spare.

An example of the type of compact playback system you can put anywhere once you have a PiDisk set up!