Simultaneous SatNOGS Clients

Introduction & Acknowledgements

This page documents an approach to running two simultaneous SatNOGS clients on a single RaspberryPi4 connected to two RTL-SDR v3 dongles. Before proceeding please note that this is not an officially supported configuration, however in practice, with some tweaking and a bit of a nasty hack, it appears to work reliably. As at mid February 2021, stations 1825 and 1891 have been running this way for about two months without any problems attributed to simultaneous operation.

The approach documented is just the way @hughhalf accomplished the goal, and I drew heavily on advice from @Acinonyx, @fredy and @cgbsat in this forum thread. @vk5qi also provided valuable assistance when I was first getting started with SatNOGS and @ansi with load testing/scheduling the two station setup. Any errors in this document are, however, of my own making :)

Prerequisites and Basics of the setup

It is assumed you have set up your first station as documented in the SatNOGS wiki here. Please have your first station working the way you want it before proceeding.

Once you’ve got the first station going, you may want to set up your 2nd set of hardware (SDR dongle, preamps etc. etc.) and test it through the first station instance before you proceed much further - just make temporary changes to the config as required.

Your second station will largely be a copy of the first, with just enough changes/duplicated files to allow the required alternate configuration values to be set as well as duplicate rigctl and rotctl daemons. You do not need a new API token from the SatNOGS backend, but you will need to create a new STATION_ID.

At the time of writing there does appear to be a quirk in the way the GNU Radio library writes some of it’s temporary files that necessitates some hackery to have a chroot environment for some bits of the 2nd instance. This is elaborated on in the original forum thread and below.

Before proceeding, ensure you have set unique serial numbers for your SDR dongles - I used rtl_eeprom broadly following the method in the first steps of http://www.cloud-sdr.com/tutorial-sharing-two-rtlsdr-receivers/ to set mine to 27001 and 27003. The choice of number is arbitrary as long as they are unique.

Creating a second satnogs service

I created a duplicate service, called (imaginatively) satnogs-client-2. I did this by making a copy of /etc/systemd/system/satnogs-client.service called /etc/systemd/system/satnogs-client-2.service For now this file differs only from the original in having the EnvironmentFile entry pointing to a 2nd environment file which contains the different settings for the 2nd instance. We uncomment the RootDirectory directive later to make use of systemd’s ability to start a service in a chroot environment once basic functionality of the 2nd station is confirmed.

For now your file should look something like;

    [Unit]
    Description=SatNOGS client two
    
    [Service]
    #RootDirectory=/home/pi/testroot
    EnvironmentFile=/etc/default/satnogs-client2
    ExecStart=/var/lib/satnogs/bin/satnogs-client
    Restart=on-failure
    User=satnogs
    Group=satnogs
    
    [Install]
    WantedBy=multi-user.target

The choice of directory for the second config file is arbitrary as long as it can be ready by the satnogs user - I followed the convention used for the first client and put it in /etc/default/satnogs-client2 - the contents then being;

    SATNOGS_API_TOKEN="feedb0b0xxxxxxxxx"
    SATNOGS_STATION_ID="1891"
    SATNOGS_STATION_LAT="-38.XX"
    SATNOGS_STATION_LON="142.XX"
    SATNOGS_STATION_ELEV="51"
    SATNOGS_SOAPY_RX_DEVICE="driver=rtlsdr,serial=271103"
    SATNOGS_RX_SAMP_RATE="2.048e6"
    SATNOGS_RF_GAIN="16"
    SATNOGS_ANTENNA="RX"
    SATNOGS_APP_PATH="/tmp/.satnogs2"
    SATNOGS_OUTPUT_PATH="/tmp/.satnogs2/data"
    SATNOGS_INCOMPLETE_OUTPUT_PATH="/tmp/.satnogs2/data/incomplete"
    SATNOGS_LOG_LEVEL="INFO"
    SATNOGS_RIG_PORT="4542"

For this second file note;

• You need to use your existing API_TOKEN
• Use the 2nd station ID you created through the SatNOGS Dashboard
• Create new temporary directories for the 2nd instance, checking they have the same permissions as the original ones
• Adjust the SOAPY_RX_DEVICE settings to suit your 2nd SDR dongle. As noted above, I used rtl_eeprom to re-write the serial number of my SDR dongles.

You’ll need to create a second rigctl service as well - even if you’re not controlling an external radio or rotator, the satnogs client uses the rigctl service to adjust the SDR receive freq uency in real time to account for doppler shift. To do this make copies of the existing hamlib-utils and rigctl.service files, rename and edit thus;

Defaults for the second instance put in /dev/default/hamlib-utils2

    ROT_OPTS=""
    RIG_OPTS="-t 4542 -T 127.0.0.1 -m 1"

The alternate port number is the main thing here.

And a duplicate service in /etc/systemd/system/rigctl2.service

 [Unit]
 Description=rigctld server two
 
 [Service]
 EnvironmentFile=-/etc/default/hamlib-utils2
 ExecStart=/usr/bin/rigctld $RIG_OPTS
 Restart=on-failure
 User=hamlib-utils
 Group=hamlib-utils
 
 [Install]
 WantedBy=multi-user.target 

All going well you should be able to do an initial fire up of your 2nd service.

• Run systemctl daemon-reload to reload config files
• Start your 2nd rigctl service with sudo service rigctld2 start
  • Optionally check it's running with sudo service rigctld2 status
• Start your 2nd satnogs client with sudo service rigctld2 status
  • Check it’s happy with sudo service satnogs-client2 status
  • Or, better: sudo journalctl -ef -u satnogs-client2

If all is well, you should see both stations as showing currently active in the SatNOGS dashboard.

You may wish to try scheduling a few passes to make sure both stations seem happy.

The following looks like line noise but you can use to watch the temporary directories during capture;

watch -n 1 ‘ls -l /tmp/.satnogs/data; echo""; ls -l /tmp/.satnogs2/data; echo""; ls -l /tmp/.satnogs; echo""; ls -l /tmp/.satnogs2' 

If the passes you schedule do overlap you will likely see some weirdness in the captured data, more on which below. You can see example waterfalls of this behaviour in these two partially overlapping observations https://network.satnogs.org/observations/3289133/ and https://network.satnogs.org/observations/3289134/

Known Issues

There does seem to be an ongoing issue where if the two setups are running passes simultaneously on different bands they interact with each other. As I described it in a post to community.libre.space I suspect this is due to some GNU Radio temporary files being /var/lib/satnogs rather than under the tmp directory specified.

I lacked the build system fu to rebuild GNU Radio libraries with an attempted proper fix, so went the hacky way and ran the 2nd instance in a partial chroot to ensure the destination directory for the shared files /var/lib/satnogs was unique for each satnogs client instance. I’ll again reiterate what follows is very much in the “worked for me” category - better approaches welcomed :)

Creating a chroot environment - a very unreliable guide

Make sure satnogs-client-2 and rigctld2 are stopped before proceeding

Create and move to chroot directory

mkdir /home/pi/testroot
cd /home/pi/testroot

Make a copy of the minimal fileset needed - this will consume of the order of 5G. I'm quite sure there are smarter ways to do this

sudo rsync -Pav /opt /etc /var /home /usr .  

Create and set permissions on other misc directories

mkdir tmp dev sys proc
    sudo chown root.root tmp dev sys proc

And bind mount system directories

    sudo mount --bind /dev dev/
    sudo mount --bind /dev/shm dev/shm
    sudo mount --bind /sys sys/
    sudo mount --bind /proc proc/
    sudo mount --bind /tmp tmp 

Get a bash shell in your chroot environment and look around. lsusb is a good starting point, you should still see USB devices for example

sudo chroot /home/pi/testroot /bin/bash 

Exit back to actual shell with exit

Now to give 2nd station a proper test - edit the /etc/systemd/system/satnogs-client-2.service file and uncomment #RootDirectory=/home/pi/testroot this will tell systemd to use chroot when starting the service. Restart the two services - rigctld should be started first

    sudo service rigctld2 start
    sudo service satnogs client-2 start

This rather long command will let you watch the various files being created during passes - success is if you see separate .gr_fftw_wisdom* files being created

    watch -n 1 'cat /var/lib/satnogs/.gr_fftw_wisdom; echo""; cat /home/pi/testroot/var/lib/satnogs/.gr_fftw_wisdom; echo""; ls -l /var/lib/satnogs/.gr_fftw_wisdom*; echo""; ls -l /home/pi/testroot/var/lib/satnogs/.gr_fftw_wisdom*; echo""; ls -l /tmp/.satnogs/data; echo""; ls -l /tmp/.satnogs2/data; echo""; ls -l /tmp/.satnogs; echo""; ls -l /tmp/.satnogs2' 

Conclusion

This should get you up and running. If you hit problems please do remember this isn't an officially supported configuration so please don't bug the core team about fixes :) I am happy to assist where I can, just reach out through the forums.