Difference between revisions of "Troubleshooting"

From SatNOGS Wiki
(Add journalctl command for logs; tidy up troubleshooting steps a bit; add link to issfanclub.com for ISS digipeater status)
(Add "Observer job lock acquiring timed out" section)
 
(33 intermediate revisions by 15 users not shown)
Line 1: Line 1:
  
−
== Client troubleshooting ==
+
==Client troubleshooting==
  
−
=== Client not showing up on the network? ===
+
{{Message| Before starting check the logs for an error:
  
−
* Check that you have ticked the "Is it operational?" checkbox in on the groundstation page.
 
−
* Check your settings and ensure that the API token and station ID are correct.  You can get these from your profile page on the SatNOGS network site.  If you have accounts in both dev and prod, make sure you're using the token and station ID from the right environment.
 
−
* Check your SATNOGS_NETWORK_API_URL.  It should point to https://network.satnogs.org/api/ (prod) or https://network-dev.satnogs.org/api/ (dev). 
 
−
* Check your network connectivity.  Can you ping network.satnogs.org or network-dev.satnogs.org?  Try running <code>curl https://network.satnogs.org</code> or <code>curl https://network-dev.satnogs.org</code>.
 
−
Check your network connectivity.
 
−
* Check the logs for an error (<code>journalctl -f -u satnogs-client.service</code> or <code>less /var/log/supervisor/satnogs-error.log</code>) and post to our forums at https://community.libre.space
 
  
−
== Signal troubleshooting ==
 
  
−
=== Not receiving anything? ===
+
* Check the whole error log with <code>journalctl -u satnogs-client.service</code>
 +
* Check live the error log <code>journalctl -f -u satnogs-client.service</code>}}
  
−
Make sure the satellite you are testing observations against is active and recently received by others on [https://network.satnogs.org our production network site].  If you click on a satellite name, a popup will appear and give you the option to click on "Past Observations". If everything in the past shows red, then the problem is likely with that satellite.
+
{{Message|If the hints for common issues below don't help solve your issue, you are welcome to post in the forum.  When doing so, please include any error from the logs and the output of the support report generated by the menu item <code>Advanced -> Support</code> from <code>sudo satnogs-setup</code>. This helps give context to the logging messages.}}
−
* SO-50 is a good satellite to use for testing as it is a strong FM voice signal assuming you have UHF capabilities. Schedule using "PE0SAT - Mode V/U FM Voice - 436.794 MHz".  Here is an example to compare against: https://network.satnogs.org/observations/3334/
+
===Raise or Set the debug level===
−
* ISS is a good test for VHF as the APRS digipeater is alive again (as of this writing; check [https://www.issfanclub.com/ issfanclub.com] for up-to-date information). When you schedule it, be sure to select the APRS downlink.
 
  
−
=== Observations seem off-frequency? ===
+
*Run <code>sudo satnogs-setup</code>
 +
*Go to '''Advanced Configuration''' and  set the '''Debug level''' to one of the Values below
 +
*'''Apply''' the change
  
 +
{| class="wikitable"
 +
!'''Setting'''
 +
!Values
 +
|-
 +
|'''DEBUG'''
 +
|
 +
{| class="wikitable"
 +
|<code>CRITICAL</code>
 +
|-
 +
|<code>ERROR</code>
 +
|-
 +
|<code>WARNING</code>
 +
|-
 +
|<code>INFO</code>
 +
|-
 +
|<code>DEBUG</code>
 +
|-
 +
|<code>NOTSET</code>
 +
|}
 +
|}
 +
 +
When done debugging, don't forget to reset the log level, or the files will fill up your disk space.
 +
 +
===Client not showing up on the network?===
 +
 +
*Check that you have ticked the "Is it operational?" checkbox on the groundstation page.
 +
*Check your settings and ensure that the API token and station ID are correct.  You can get these from your profile page on the SatNOGS network site.  If you have accounts in both dev and prod, make sure you're using the token and station ID from the right environment.
 +
*Check your SATNOGS_NETWORK_API_URL.  It should point to https://network.satnogs.org/api/ (prod) or https://network-dev.satnogs.org/api/ (dev).
 +
*Check your network connectivity.  Can you ping network.satnogs.org or network-dev.satnogs.org?  Try running <code>curl https://network.satnogs.org</code> or <code>curl https://network-dev.satnogs.org</code>.
 +
 +
===RuntimeError: RTL-SDR device not found.===
 +
Seems that the device is not there. Try rebooting and check if Soapy can identify it using <code>SoapySDRUtil --find</code>.
 +
 +
==="satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533"===
 +
 +
The client is trying to connect to rotctld but is unable to.
 +
 +
*If you have a no-rotator setup, ensure that satnogs-client is configured as such; see the [[Raspberry Pi]] page for info on how to do this.
 +
**These errors may continue to be reported until [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/issues/340 issue #340] is resolved.
 +
 +
*If you do have a rotator, ensure that rotctld is running.
 +
 +
<br />
 +
 +
===Some uploads missing & Doppler-correction not working properly===
 +
The station/client might have a clock offset. This causes a Doppler shift on the waterfall and some observations don't start because of a conflict between the time on the PC of the station and the time of the SatNOGS Network.
 +
 +
===Uploads (waterfall or audio) are missing for a past observation===
 +
When using <code>satnogs-setup</code> for editing/updating the configuration of the station,  the satnogs-client gets restarted. When this happens during an observation the observation will be aborted and no waterfall/audio/data files will be uploaded. Such observations should be voted as ''failed''.
 +
 +
The partial data from such observations is stored in <code>/tmp/.satnogs/data/</code>, look for files like <code>receiving_{satnogs|waterfall}_123456_2019-01-01T12-23-42.{out|dat}</code>. They can be manually removed.
 +
 +
===Invalid Signature for download.opensuse.org===
 +
Example error during satnogs-setup update or apt-get update:
 +
 +
<pre>
 +
  The following signatures were invalid: EXPKEYSIG 8BD3901736A40B6C home:librespace OBS Project <home:librespace@build.opensuse.org>
 +
</pre>
 +
 +
Fix:
 +
 +
<pre>
 +
wget -qO - http://download.opensuse.org/repositories/home:/librespace:/satnogs-unstable/Raspbian_10/Release.key > k
 +
gpg --no-default-keyring --keyring ./temp-keyring.gpg --import k
 +
gpg --no-default-keyring --keyring ./temp-keyring.gpg --export --output k.gpg
 +
mkdir /etc/apt/keyrings
 +
mv k.gpg /etc/apt/keyrings/satnogs2.gpg
 +
vi /etc/apt/sources.list.d/satnogs.list
 +
</pre>
 +
Insert a signed-by statement, as:
 +
<pre>
 +
deb [signed-by=/etc/apt/keyrings/satnogs2.gpg] /repositories/home:/librespace:/satnogs/Raspbian_11 - openSUSE Download ./
 +
</pre>
 +
 +
More info available at https://community.libre.space/t/sudo-apt-update-signature-invalid/7643/7 and https://community.libre.space/t/update-to-the-invalid-signature-problem-in-the-troubleshooting-document/11386
 +
 +
===apt-get update: connection failed===
 +
When using the sudo apt-get update command, we get the following connection error:
 +
[[File:UpdateFailed.png]]
 +
 +
However, a ping to the Ip address works.
 +
 +
If you use a firewall (Ex: stormshield SN210), it can include a default rule that blocks connection urls containing a dot-slash ("./"). (This rule can be named as follows: "path with self-reference")
 +
 +
This rule must be modified to allow the raspberry to access the repository
 +
 +
===  ERROR - Observer job lock acquiring timed out. ===
 +
 +
<pre>
 +
Mar 03 20:05:53 satnogs satnogs-client[257]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.
 +
</pre>
 +
 +
This message is shown when a previous Observation failed (due to a hardware issue or some bug) and didn't properly terminate. To fix this issue, first check the logs for the observation that failed to understand the root cause. Then, restart satnogs-client.
 +
 +
==Signal troubleshooting==
 +
 +
===Blank or solid purple waterfall?===
 +
{{Warning|Some information in this section seems outdated or wrong. Please take it with care.
 +
Once reviewed this banner can be removed.}}
 +
At the first sign of trouble, put your station into testing mode.
 +
 +
*Make sure the RTLSDR gain is set correctly. Follow the guide at [[Omnidirectional_Station_How_To#Setting_the_gain]] to find a proper gain setting.<br />Check your gain value is valid. Wrong values can result in blank waterfalls. For example '7.7.' will result in errors (and a blank waterfall - there is an extra '.' at the end).  <br /><code>journalctl -u satnogs-client.service</code> It might be a big file, but work through it and look for errors.  <br /><code>df -h</code> Ensure there is sufficient hard drive space. If temp files can not be created, the waterfall might be blank. <br />Run <code>rtl_test</code> for about 30 seconds and make sure you can connect with the dongle and that there are no errors.
 +
 +
===Not receiving anything?===
 +
 +
*Make sure the satellite you are testing observations against is active and recently received by others on [https://network.satnogs.org our production network site].  If you click on a satellite name, a popup will appear and give you the option to click on "Past Observations". If everything in the past shows red, then the problem is likely with that satellite.
 +
 +
*SO-50 is a good satellite to use for testing as it is a strong FM voice signal assuming you have UHF capabilities. Schedule using "PE0SAT - Mode V/U FM Voice - 436.794 MHz".  Here is an example to compare against: https://network.satnogs.org/observations/3334/
 +
 +
*ISS is a good test for VHF as the APRS digipeater is alive again (as of this writing; check [https://www.issfanclub.com/ issfanclub.com] for up-to-date information). When you schedule it, be sure to select the APRS downlink.
 +
 +
*If you're using an rtlsdr, check that it can be seen and is operating correctly by running rtl_test.  Let it run for 30 seconds or so, then hit Ctrl-c to kill it:
 +
 +
<pre>
 +
pi@raspberrypi:~ $ rtl_test
 +
Found 1 device(s):
 +
  0:  Realtek, RTL2838UHIDIR, SN: 00000001
 +
 +
Using device 0: Generic RTL2832U OEM
 +
Found Rafael Micro R820T tuner
 +
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6
 +
[R82XX] PLL not locked!
 +
Sampling at 2048000 S/s.
 +
 +
Info: This tool will continuously read from the device, and report if
 +
samples get lost. If you observe no further output, everything is fine.
 +
 +
Reading samples in async mode...
 +
^CSignal caught, exiting!
 +
 +
User cancel, exiting...
 +
Samples per million lost (minimum): 0
 +
</pre>
 +
 +
*You can also try a manual run of satnogs_afsk1200_ax25.py to make sure that works:
 +
 +
<pre>
 +
$ cd /tmp
 +
$ satnogs_afsk1200_ax25.py --soapy-rx-device="driver=rtlsdr" --antenna=RX --samp-rate-rx=2.048e6 --rx-freq=444000000 --file-path=./audio-out.ogg --waterfall-file-path=./waterfall.dat
 +
</pre>
 +
 +
Let it run for a minute or so.  If everything is working, this should create an .ogg file and a .dat file of non-zero size (probably a few MB each).
 +
 +
===Observations seem off-frequency?===
 
[[File:Waterfall_3519_2017-04-24T04-48-48_resized.png|frame|Check your location!]]
 
[[File:Waterfall_3519_2017-04-24T04-48-48_resized.png|frame|Check your location!]]
−
* '''PPM drift''' While newer SDR devices are very good and stable, there still may be some PPM drift to compensate for if you notice that signals are consistently off center. The SATNOGS_PPM_ERROR setting in /etc/supervisord.d/satnogs.ini can be used to correct for this.
+
 
−
* '''Clock sync''' Make sure your clock is synced. Ensure ntp is configured and running (especially with the Raspberry Pi which lacks a real time clock)
+
*'''PPM drift''' While newer SDR devices are very good and stable, there still may be some PPM drift to compensate for if you notice that signals are consistently off center. The SATNOGS_PPM_ERROR setting in /etc/supervisord.d/satnogs.ini can be used to correct for this.
−
* '''Wrong location''' If your signal seems to be on but drifts at the apex like in this image, check to make sure your Latitude, Longitude, and Elevation coordinates are set properly and in the right format.
+
*'''Clock sync''' Make sure your clock is synced. Ensure ntp is configured and running (especially with the Raspberry Pi which lacks a real time clock)
 +
*'''Wrong location''' If your signal seems to be on but drifts at the apex like in this image, check to make sure your Latitude, Longitude, and Elevation coordinates are set properly and in the right format.
 +
 
 +
 
 +
===USB===
 +
You can reset USB without having to reboot the system by running these commands:
 +
 
 +
    echo “usb1” > /sys/bus/usb/drivers/usb/unbind
 +
    echo “usb1” > /sys/bus/usb/drivers/usb/bind
 +
 
 +
This can be placed in the satnogs_post_observation_script for automation.
 +
 
 +
===RF Noise===
 +
If you notice a noise in the waterfall every time motors are spinning, you will need to:
 +
 
 +
*Twist each pair or the motor wire
 +
*Add proper grounding
 +
*Add capacitor to the DC input of drivers
 +
*Wrap the motors wire with adhesive aluminum and then connect it to GND on driver side
 +
*Add ferrites to motors wires
 +
 
 +
[[File:Rf noise.png|alt=RF Noise|left|thumb|400x400px|Too much RF Noise]]
 +
 
 +
[[Category:Operate]]
 +
[[Category:Hardware]]
 +
[[Category:Software]]

Latest revision as of 09:51, 4 March 2024

Client troubleshooting

Idea.png
Before starting check the logs for an error:


  • Check the whole error log with journalctl -u satnogs-client.service
  • Check live the error log journalctl -f -u satnogs-client.service
Idea.png
If the hints for common issues below don't help solve your issue, you are welcome to post in the forum. When doing so, please include any error from the logs and the output of the support report generated by the menu item Advanced -> Support from sudo satnogs-setup. This helps give context to the logging messages.

Raise or Set the debug level

  • Run sudo satnogs-setup
  • Go to Advanced Configuration and set the Debug level to one of the Values below
  • Apply the change
Setting Values
DEBUG
CRITICAL
ERROR
WARNING
INFO
DEBUG
NOTSET

When done debugging, don't forget to reset the log level, or the files will fill up your disk space.

Client not showing up on the network?

  • Check that you have ticked the "Is it operational?" checkbox on the groundstation page.
  • Check your settings and ensure that the API token and station ID are correct. You can get these from your profile page on the SatNOGS network site. If you have accounts in both dev and prod, make sure you're using the token and station ID from the right environment.
  • Check your SATNOGS_NETWORK_API_URL. It should point to https://network.satnogs.org/api/ (prod) or https://network-dev.satnogs.org/api/ (dev).
  • Check your network connectivity. Can you ping network.satnogs.org or network-dev.satnogs.org? Try running curl https://network.satnogs.org or curl https://network-dev.satnogs.org.

RuntimeError: RTL-SDR device not found.

Seems that the device is not there. Try rebooting and check if Soapy can identify it using SoapySDRUtil --find.

"satnogsclient - ERROR - Cannot connect to socket 127.0.0.1:4533"

The client is trying to connect to rotctld but is unable to.

  • If you have a no-rotator setup, ensure that satnogs-client is configured as such; see the Raspberry Pi page for info on how to do this.
    • These errors may continue to be reported until issue #340 is resolved.
  • If you do have a rotator, ensure that rotctld is running.


Some uploads missing & Doppler-correction not working properly

The station/client might have a clock offset. This causes a Doppler shift on the waterfall and some observations don't start because of a conflict between the time on the PC of the station and the time of the SatNOGS Network.

Uploads (waterfall or audio) are missing for a past observation

When using satnogs-setup for editing/updating the configuration of the station, the satnogs-client gets restarted. When this happens during an observation the observation will be aborted and no waterfall/audio/data files will be uploaded. Such observations should be voted as failed.

The partial data from such observations is stored in /tmp/.satnogs/data/, look for files like receiving_{satnogs|waterfall}_123456_2019-01-01T12-23-42.{out|dat}. They can be manually removed.

Invalid Signature for download.opensuse.org

Example error during satnogs-setup update or apt-get update: 

  The following signatures were invalid: EXPKEYSIG 8BD3901736A40B6C home:librespace OBS Project <home:librespace@build.opensuse.org>

Fix: 

wget -qO - http://download.opensuse.org/repositories/home:/librespace:/satnogs-unstable/Raspbian_10/Release.key > k
gpg --no-default-keyring --keyring ./temp-keyring.gpg --import k
gpg --no-default-keyring --keyring ./temp-keyring.gpg --export --output k.gpg
mkdir /etc/apt/keyrings
mv k.gpg /etc/apt/keyrings/satnogs2.gpg
vi /etc/apt/sources.list.d/satnogs.list

Insert a signed-by statement, as: 

deb [signed-by=/etc/apt/keyrings/satnogs2.gpg] /repositories/home:/librespace:/satnogs/Raspbian_11 - openSUSE Download ./

More info available at https://community.libre.space/t/sudo-apt-update-signature-invalid/7643/7 and https://community.libre.space/t/update-to-the-invalid-signature-problem-in-the-troubleshooting-document/11386

apt-get update: connection failed

When using the sudo apt-get update command, we get the following connection error: UpdateFailed.png

However, a ping to the Ip address works.

If you use a firewall (Ex: stormshield SN210), it can include a default rule that blocks connection urls containing a dot-slash ("./"). (This rule can be named as follows: "path with self-reference")

This rule must be modified to allow the raspberry to access the repository

ERROR - Observer job lock acquiring timed out.

 Mar 03 20:05:53 satnogs satnogs-client[257]: satnogsclient.scheduler.tasks - ERROR - Observer job lock acquiring timed out.

This message is shown when a previous Observation failed (due to a hardware issue or some bug) and didn't properly terminate. To fix this issue, first check the logs for the observation that failed to understand the root cause. Then, restart satnogs-client.

Signal troubleshooting

Blank or solid purple waterfall?

Important.png
Some information in this section seems outdated or wrong. Please take it with care. Once reviewed this banner can be removed.

At the first sign of trouble, put your station into testing mode.

  • Make sure the RTLSDR gain is set correctly. Follow the guide at Omnidirectional_Station_How_To#Setting_the_gain to find a proper gain setting.
    Check your gain value is valid. Wrong values can result in blank waterfalls. For example '7.7.' will result in errors (and a blank waterfall - there is an extra '.' at the end).
    journalctl -u satnogs-client.service It might be a big file, but work through it and look for errors.
    df -h Ensure there is sufficient hard drive space. If temp files can not be created, the waterfall might be blank.
    Run rtl_test for about 30 seconds and make sure you can connect with the dongle and that there are no errors.

Not receiving anything?

  • Make sure the satellite you are testing observations against is active and recently received by others on our production network site. If you click on a satellite name, a popup will appear and give you the option to click on "Past Observations". If everything in the past shows red, then the problem is likely with that satellite.
  • SO-50 is a good satellite to use for testing as it is a strong FM voice signal assuming you have UHF capabilities. Schedule using "PE0SAT - Mode V/U FM Voice - 436.794 MHz". Here is an example to compare against: https://network.satnogs.org/observations/3334/
  • ISS is a good test for VHF as the APRS digipeater is alive again (as of this writing; check issfanclub.com for up-to-date information). When you schedule it, be sure to select the APRS downlink.
  • If you're using an rtlsdr, check that it can be seen and is operating correctly by running rtl_test. Let it run for 30 seconds or so, then hit Ctrl-c to kill it:
pi@raspberrypi:~ $ rtl_test 
Found 1 device(s):
  0:  Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Found Rafael Micro R820T tuner
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6 
[R82XX] PLL not locked!
Sampling at 2048000 S/s.

Info: This tool will continuously read from the device, and report if
samples get lost. If you observe no further output, everything is fine.

Reading samples in async mode...
^CSignal caught, exiting!

User cancel, exiting...
Samples per million lost (minimum): 0
  • You can also try a manual run of satnogs_afsk1200_ax25.py to make sure that works:
$ cd /tmp
$ satnogs_afsk1200_ax25.py --soapy-rx-device="driver=rtlsdr" --antenna=RX --samp-rate-rx=2.048e6 --rx-freq=444000000 --file-path=./audio-out.ogg --waterfall-file-path=./waterfall.dat

Let it run for a minute or so. If everything is working, this should create an .ogg file and a .dat file of non-zero size (probably a few MB each).

Observations seem off-frequency?

Check your location!
  • PPM drift While newer SDR devices are very good and stable, there still may be some PPM drift to compensate for if you notice that signals are consistently off center. The SATNOGS_PPM_ERROR setting in /etc/supervisord.d/satnogs.ini can be used to correct for this.
  • Clock sync Make sure your clock is synced. Ensure ntp is configured and running (especially with the Raspberry Pi which lacks a real time clock)
  • Wrong location If your signal seems to be on but drifts at the apex like in this image, check to make sure your Latitude, Longitude, and Elevation coordinates are set properly and in the right format.


USB

You can reset USB without having to reboot the system by running these commands: 

   echo “usb1” > /sys/bus/usb/drivers/usb/unbind
   echo “usb1” > /sys/bus/usb/drivers/usb/bind

This can be placed in the satnogs_post_observation_script for automation.

RF Noise

If you notice a noise in the waterfall every time motors are spinning, you will need to:

  • Twist each pair or the motor wire
  • Add proper grounding
  • Add capacitor to the DC input of drivers
  • Wrap the motors wire with adhesive aluminum and then connect it to GND on driver side
  • Add ferrites to motors wires
RF Noise
Too much RF Noise
Retrieved from "https://wiki.satnogs.org/index.php?title=Troubleshooting&oldid=3897"
  • Creative Commons Attribution-ShareAlike
  • Powered by MediaWiki