Simultaneous SatNOGS Clients
Introduction & Acknowledgements
This page documents an approach to running two simultaneous SatNOGS clients on a single RaspberryPi4 connected to two RTL-SDR v3 dongles. Before proceeding please note that this is not an officially supported configuration, however in practice, with some tweaking and a bit of a nasty hack, it appears to work reliably. As at mid February 2021, stations 1825 and 1891 have been running this way for about two months without any problems attributed to simultaneous operation.
The approach documented is just the way @hughhalf accomplished the goal, and I drew heavily on advice from @Acinonyx, @fredy and @cgbsat in this forum thread. @vk5qi also provided valuable assistance when I was first getting started with SatNOGS and @ansi with load testing/scheduling the two station setup. Any errors in this document are, however, of my own making :)
Prerequisites and Basics of the setup
It is assumed you have set up your first station as documented in the SatNOGS wiki here. Please have your first station working the way you want it before proceeding.
Once you’ve got the first station going, you may want to set up your 2nd set of hardware (SDR dongle, preamps etc. etc.) and test it through the first station instance before you proceed much further - just make temporary changes to the config as required.
Your second station will largely be a copy of the first, with just enough changes/duplicated files to allow the required alternate configuration values to be set as well as duplicate rigctl and rotctl daemons. You do not need a new API token from the SatNOGS backend, but you will need to create a new STATION_ID.
At the time of writing there does appear to be a quirk in the way the GNU Radio library writes some of it’s temporary files that necessitates some hackery to have a chroot environment for some bits of the 2nd instance. This is elaborated on in the original forum thread and below.
Before proceeding, ensure you have set unique serial numbers for your SDR dongles - I used
rtl_eeprom broadly following the method in the first steps of http://www.cloud-sdr.com/tutorial-sharing-two-rtlsdr-receivers/ to set mine to 27001 and 27003. The choice of number is arbitrary as long as they are unique.
Creating a second satnogs service
I created a duplicate service, called (imaginatively) satnogs-client-2. I did this by making a copy of /etc/systemd/system/satnogs-client.service called /etc/systemd/system/satnogs-client-2.service For now this file differs only from the original in having the EnvironmentFile entry pointing to a 2nd environment file which contains the different settings for the 2nd instance. We uncomment the RootDirectory directive later to make use of systemd’s ability to start a service in a chroot environment once basic functionality of the 2nd station is confirmed.
For now your file should look something like;
[Unit] Description=SatNOGS client two [Service] #RootDirectory=/home/pi/testroot EnvironmentFile=/etc/default/satnogs-client2 ExecStart=/var/lib/satnogs/bin/satnogs-client Restart=on-failure User=satnogs Group=satnogs [Install] WantedBy=multi-user.target
The choice of directory for the second config file is arbitrary as long as it can be ready by the satnogs user - I followed the convention used for the first client and put it in /etc/default/satnogs-client2 - the contents then being;
SATNOGS_API_TOKEN="feedb0b0xxxxxxxxx" SATNOGS_STATION_ID="1891" SATNOGS_STATION_LAT="-38.XX" SATNOGS_STATION_LON="142.XX" SATNOGS_STATION_ELEV="51" SATNOGS_SOAPY_RX_DEVICE="driver=rtlsdr,serial=271103" SATNOGS_RX_SAMP_RATE="2.048e6" SATNOGS_RF_GAIN="16" SATNOGS_ANTENNA="RX" SATNOGS_APP_PATH="/tmp/.satnogs2" SATNOGS_OUTPUT_PATH="/tmp/.satnogs2/data" SATNOGS_INCOMPLETE_OUTPUT_PATH="/tmp/.satnogs2/data/incomplete" SATNOGS_LOG_LEVEL="INFO" SATNOGS_RIG_PORT="4542"
For this second file note;
- You need to use your existing API_TOKEN
- Use the 2nd station ID you created through the SatNOGS Dashboard
- Create new temporary directories for the 2nd instance, checking they have the same permissions as the original ones
- Adjust the SOAPY_RX_DEVICE settings to suit your 2nd SDR dongle. As noted above, I used rtl_eeprom to re-write the serial number of my SDR dongles.
You’ll need to create a second rigctl service as well - even if you’re not controlling an external radio or rotator, the satnogs client uses the rigctl service to adjust the SDR receive freq uency in real time to account for doppler shift. To do this make copies of the existing hamlib-utils and rigctl.service files, rename and edit thus;
Defaults for the second instance put in /dev/default/hamlib-utils2
ROT_OPTS="" RIG_OPTS="-t 4542 -T 127.0.0.1 -m 1"
The alternate port number is the main thing here.
And a duplicate service in /etc/systemd/system/rigctl2.service
[Unit] Description=rigctld server two [Service] EnvironmentFile=-/etc/default/hamlib-utils2 ExecStart=/usr/bin/rigctld $RIG_OPTS Restart=on-failure User=hamlib-utils Group=hamlib-utils [Install] WantedBy=multi-user.target
All going well you should be able to do an initial fire up of your 2nd service.
systemctl daemon-reloadto reload config files
- Start your 2nd rigctl service with
sudo service rigctld2start
- Optionally check it's running with
sudo service rigctld2 status
- Optionally check it's running with
- Start your 2nd satnogs client with
sudo service rigctld2 status
- Check it’s happy with
sudo service satnogs-client2 status
- Or, better:
sudo journalctl -ef -u satnogs-client2
- Check it’s happy with
If all is well, you should see both stations as showing currently active in the SatNOGS dashboard.
You may wish to try scheduling a few passes to make sure both stations seem happy.
The following looks like line noise but you can use to watch the temporary directories during capture;
watch -n 1 ‘ls -l /tmp/.satnogs/data; echo""; ls -l /tmp/.satnogs2/data; echo""; ls -l /tmp/.satnogs; echo""; ls -l /tmp/.satnogs2'
If the passes you schedule do overlap you will likely see some weirdness in the captured data, more on which below. You can see example waterfalls of this behaviour in these two partially overlapping observations https://network.satnogs.org/observations/3289133/ and https://network.satnogs.org/observations/3289134/
There does seem to be an ongoing issue where if the two setups are running passes simultaneously on different bands they interact with each other. As I described it in a post to community.libre.space I suspect this is due to some GNU Radio temporary files being /var/lib/satnogs rather than under the tmp directory specified.
I lacked the build system fu to rebuild GNU Radio libraries with an attempted proper fix, so went the hacky way and ran the 2nd instance in a partial chroot to ensure the destination directory for the shared files /var/lib/satnogs was unique for each satnogs client instance. I’ll again reiterate what follows is very much in the “worked for me” category - better approaches welcomed :)
Creating a chroot environment - a very unreliable guide
Make sure satnogs-client-2 and rigctld2 are stopped before proceeding
Create and move to chroot directory
mkdir /home/pi/testroot cd /home/pi/testroot
Make a copy of the minimal fileset needed - this will consume of the order of 5G. I'm quite sure there are smarter ways to do this
sudo rsync -Pav /opt /etc /var /home /usr .
Create and set permissions on other misc directories
mkdir tmp dev sys proc sudo chown root.root tmp dev sys proc
And bind mount system directories
sudo mount --bind /dev dev/ sudo mount --bind /dev/shm dev/shm sudo mount --bind /sys sys/ sudo mount --bind /proc proc/ sudo mount --bind /tmp tmp
Get a bash shell in your chroot environment and look around. lsusb is a good starting point, you should still see USB devices for example
sudo chroot /home/pi/testroot /bin/bash
Exit back to actual shell with
Now to give 2nd station a proper test - edit the
/etc/systemd/system/satnogs-client-2.service file and uncomment
#RootDirectory=/home/pi/testroot this will tell systemd to use chroot when starting the service. Restart the two services - rigctld should be started first
sudo service rigctld2 start sudo service satnogs client-2 start
This rather long command will let you watch the various files being created during passes - success is if you see separate .gr_fftw_wisdom* files being created
watch -n 1 'cat /var/lib/satnogs/.gr_fftw_wisdom; echo""; cat /home/pi/testroot/var/lib/satnogs/.gr_fftw_wisdom; echo""; ls -l /var/lib/satnogs/.gr_fftw_wisdom*; echo""; ls -l /home/pi/testroot/var/lib/satnogs/.gr_fftw_wisdom*; echo""; ls -l /tmp/.satnogs/data; echo""; ls -l /tmp/.satnogs2/data; echo""; ls -l /tmp/.satnogs; echo""; ls -l /tmp/.satnogs2'
This should get you up and running. If you hit problems please do remember this isn't an officially supported configuration so please don't bug the core team about fixes :) I am happy to assist where I can, just reach out through the forums.