Understanding satnogs-flowgraphs - Revision history

Pierros: Pierros moved page Understanding 'satnogs-flowgraphs' to Understanding satnogs-flowgraphs

2020-08-06T08:31:52Z

Pierros moved page Understanding 'satnogs-flowgraphs' to Understanding satnogs-flowgraphs

Pablo.pita: update

2020-05-07T17:31:12Z

update

Pablo.pita: /* Setting yourself up */

2020-05-07T16:43:28Z

‎Setting yourself up

Pablo.pita: Category

2020-05-07T16:12:18Z

Pablo.pita: /* Sample rate */

2020-05-06T21:01:44Z

‎Sample rate

Pablo.pita: /* So where did the IQ come from? */

2020-05-06T20:42:40Z

‎So where did the IQ come from?

Pablo.pita: Provide a recipe for installing satnogs-flowgraphs and gr-satnogs in Ubuntu

2020-05-06T20:37:33Z

Provide a recipe for installing satnogs-flowgraphs and gr-satnogs in Ubuntu

Matburnham: Sample rate calculation workings - raw, probably needs a tidy-up

2020-05-06T16:19:36Z

Sample rate calculation workings - raw, probably needs a tidy-up

Matburnham at 13:51, 30 April 2020

2020-04-30T13:51:18Z

Matburnham: Created page with "I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https..."

2020-04-30T13:25:31Z

Created page with "I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https..."

New page

I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] thread but as I worked through after picking things up after initially having a problem I realised everything I'd described now magically worked! Typical. Worth documenting.

== Setting yourself up ==

See elsewhere, but get GNU Radio up and running and a copy of satnogs-flowgraphs from GitLab.

I'm using a virtual Ubuntu 19.10 machine grabbed from osboxes.org, but you can use whatever you like. I imagine you could even do it on the Raspberry Pi itself. I did end up setting this VM up as a 'development station' but you should only need to install GNU Radio, and any pre-requisites needed by satnogs-flowgraphs - so probably gr-satnogs, I can't quite remember. As you find out please edit this :-)

== Storing an IQ recording ==

There are options in satnogs-setup under Advanced > Radio to store IQ files:

* ENABLE_IQ_DUMP: Enable IQ dump
* IQ_DUMP_FILENAME: Define IQ dump filename [/tmp/.satnogs/iq.raw]

Beware, IQ files are huge (220MB for my APT pass example below). They're stored in /tmp so therefore in RAM and are overwritten with each pass. You can do [https://community.libre.space/t/uploading-compressed-iq-files-to-dropbox-or-any-other-cloud-storage-provider/5395 something clever] to upload them elsewhere, but for playing just grab a copy immediately after a pass.

Don't worry about storing much else (which also vanishes quickly). You can easily grab waterfalls and OGG recording from the database via [https://github.com/deckbsd/glouton-satnogs-data-downloader Glouton].

== GNU Radio Companion ==

Here I'm working with [https://network.satnogs.org/observations/2073316/ Observation #2073316] as an example. I've saved a copy of the IQ recording my station briefly stored at `/tmp/.satnogs/iq.raw`. This is about 220 MB. I even remembered to turn the IQ collection off afterwards or it would be storing stuff on top of each other forever.

=== So where did the IQ come from? ===

In my case, I'm playing with an APT pass. It is therefore processed by [https://gitlab.com/librespacefoundation/satnogs/satnogs-flowgraphs/-/blob/master/satellites/noaa_apt_decoder.grc satnogs-flowgraphs/satellites/noaa_apt_decoder.grc]. Other signals use other flowgraphs. The default where nothing else is suitable appears to be [https://gitlab.com/librespacefoundation/satnogs/satnogs-flowgraphs/-/blob/master/generic/fsk.grc satnogs-flowgraphs/generic/fsk.grc].

[[File:flowgraph_save.png|600px]]

That is, it's post Doppler correction, which includes an LO offset; and the values are scaled to shorts, although 16768 seems an weird number as I'd have expected a power of two - 16384? I don't imagine it makes a lot of difference as it'll still fit in the short just lose a little bit of dynamic range.

=== Feeding your IQ in ===

Loading the flowgraph in requires some GNU Radio modules chaining together. Starting with [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] you'll end up with something like this:

[[File:flowgraph_open.png|800px]]

It took some time for me to work out the appropriate samples rates but I can't remember exactly how now. Scheduling a pass will generate them and then you can work it from there. Maybe I'll come back and update this later.

The rest just reverses throttles the output to real-time, and converts the short values to complex real including the scaling division.

=== Results ===

Initially, the waterfall may look a bit busy compared to the one from the [https://network.satnogs.org/observations/2073316/ observation]. There's obviously some gain setting within the waterfall I've not yet mastered, or the Satnogs ones does something slightly cleverer.

[[File:Waterfall.png|200px]]

However, if you peer closer you can just about make out the feint telltales of an APT signal.

Following the flowgraph through to output shows an identical output to that on the observation. Result!

[[File:visual.png|300px]]

=== OGG ===

For completeness, you can do the same thing using the OGG file as a source. Again, you need to reverse any resampling or other changes that were done before saving to file before feeding back into the flowgraph at an appropriate point. The throttle is optional, and in the screenshot bypassed:

[[File:flowgraph_all.png|800px]]

@@ Line 15: / Line 15: @@
 dch -b -M -v "${PACKAGE_VERSION}-1" "Bump to version '${PACKAGE_VERSION}-1'"
 dch -r -m ""
-sed -i '/0001-remove-git-maint-version.patch/ d' debian/patches/series
+sed -i '/0001-remove-git-maint-version.patch/ d' debian/patches/series  # removes that line
 debuild --unsigned-source --unsigned-changes --build=binary
 </pre>
-Proceed similarly for satnogs-flowgraphs.
+If all goes well, two debian packages, libgnuradio-satnogs and gr-satnogs, are produced in the parent directory.
 ==Storing an IQ recording==

@@ Line 8: / Line 8: @@
 <pre>
 sudo apt-get -qy install gnupg libcurl4 devscripts git-buildpackage
 git clone https://gitlab.com/librespacefoundation/satnogs/gr-satnogs.git
 cd gr-satnogs
 PACKAGE_VERSION="${PACKAGE_VERSION:-$(git describe --abbrev=8 2>/dev/null | tr '-' '+' || true)}"
 echo $PACKAGE_VERSION  # Or set it directly if not set OK
@@ Line 165: / Line 165: @@
 [[File:flowgraph_all.png|800px]]
 [[Category:Software]]
 [[Category:Develop]]

← Older revision		Revision as of 16:12, 7 May 2020
Line 165:		Line 165:

	[[File:flowgraph_all.png\|800px]]		[[File:flowgraph_all.png\|800px]]
		+
		+
		+	[[Category:Software]]
		+	[[Category:Develop]]

@@ Line 64: / Line 64: @@
 '''Where do these parameters come from?'''
-All jobs come via [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/scheduler/tasks.py#L47 scheduler/tasks.py]. This grabs scheduled jobs from the Satnogs network. In JSON format, this looks something like this:
+All observation jobs come via [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/scheduler/tasks.py#L47 scheduler/tasks.py]. This grabs scheduled jobs from the Satnogs network. In JSON format, the jobs look something like this:
 <pre>
+[
@@ Line 98: / Line 98: @@
 </pre>
-Clearly, there's no sample rate in that blob, but there is a mode and a baudrate.
+Clearly, there is no sample rate in those, but there is a mode and a baudrate. So that information brings many parameters. Others come from the satnogs configuration done via sudo satnogs-setup.
 '''How do these parameters get to the flowgraph?'''
-<code>exec_gnuradio</code> in [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/upsat/gnuradio_handler.py#L60 satnogsclient/upsat/gnuradio_handler.py#L60] takes all the parameters collected and passes them as command line arguments to the relevant flowgraph. It looks like the samp-rate-rx comes from SATNOGS_RX_SAMP_RATE in settings.py.
+The function <code>exec_gnuradio</code> in [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/upsat/gnuradio_handler.py#L60 satnogsclient/upsat/gnuradio_handler.py#L60] takes all the parameters collected and passes them as command line arguments to the relevant flowgraph. It looks like the samp-rate-rx comes from SATNOGS_RX_SAMP_RATE in settings.py.

@@ Line 42: / Line 42: @@
 [[File:flowgraph_save.png|600px]]
-That is, it's post Doppler correction, which includes an LO offset; and the values are scaled to shorts, although 16768 seems an weird number as I'd have expected a power of two - 16384? I don't imagine it makes a lot of difference as it'll still fit in the short just lose a little bit of dynamic range.
+That is, it's post Doppler correction, which includes an LO (Local Oscillator) offset; and the values are scaled to shorts, although 16768 seems an weird number as I'd have expected a power of two - 16384? I don't imagine it makes a lot of difference as it'll still fit in the short just lose a little bit of dynamic range.
 ===Feeding your IQ in===
@@ Line 50: / Line 50: @@
 [[File:flowgraph_open.png|800px]]
-=== Sample rate ===
+===Sample rate===
 It took some time for me to work out the appropriate sample rate. Here's my methodology:
@@ Line 57: / Line 57: @@
 From <code>fsk.grc</code>:
-# Soapy Source collects data at <code>samp_rate_rx</code>.
+#Soapy Source collects data at <code>samp_rate_rx</code>.
-# Doppler Compensation takes <code>samp_rate_rx</code> and aims for a Target Sampling Rate of <code>baudrate*decimation</code> (in <code>fsk.grc</code>) or <code>audio_samp_rate</code> (in <code>fm.grc</code>). One assumes these should normally be identical, but perhaps someone brighter can explain the difference and how Doppler Compensation converts - decimation?
+#Doppler Compensation takes <code>samp_rate_rx</code> and aims for a Target Sampling Rate of <code>baudrate*decimation</code> (in <code>fsk.grc</code>) or <code>audio_samp_rate</code> (in <code>fm.grc</code>). One assumes these should normally be identical, but perhaps someone brighter can explain the difference and how Doppler Compensation converts - decimation?
-# IQ sink stores the data to <code>iq_file_path</code>, with a scaling factor of <code>16768</code> applied.
+#IQ sink stores the data to <code>iq_file_path</code>, with a scaling factor of <code>16768</code> applied.

← Older revision	Revision as of 08:31, 6 August 2020
(No difference)

@@ Line 3: / Line 3: @@
 ==Setting yourself up==
-See elsewhere, but get GNU Radio up and running and a copy of satnogs-flowgraphs from GitLab.
+You need GNU Radio and satnogs-flowgraphs together with its dependencies like gr-satnogs and gr-soapy.
-I'm using a virtual Ubuntu 19.10 machine grabbed from osboxes.org, but you can use whatever you like. I imagine you could even do it on the Raspberry Pi itself. I did end up setting this VM up as a 'development station' but you should only need to install GNU Radio, and any pre-requisites needed by satnogs-flowgraphs - so probably gr-satnogs, I can't quite remember. As you find out please edit this :-)
+The easiest would be to have a supported distribution and do the normal installation of the binaries from repositories like the one from satnogs. In my case, I am using a virtual Ubuntu 19.10 machine grabbed from osboxes.org. Overthere, a recent GNU Radio is available from the official repository. The gr-soapy package is available from [https://github.com/pothosware/PothosCore/wiki/Ubuntu#add-ppas-to-your-system PothosCore PPA external repository]. As there are no ubuntu packages for satnogs-flowgraphs and gr-satnogs, those need to be built from the gitlab source. The [https://gitlab.com/librespacefoundation/satnogs/gr-satnogs/-/blob/master/.gitlab-ci.yml recipe for building the Debian packages] can be adapted for Ubuntu. For gr-satnogs, that means something like this:
 ==Storing an IQ recording==
@@ Line 51: / Line 65: @@
 All jobs come via [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/scheduler/tasks.py#L47 scheduler/tasks.py]. This grabs scheduled jobs from the Satnogs network. In JSON format, this looks something like this:
-<code>[</code>
+    },
-<code>    {</code>
+        "id": 2164887,
+        "start": "2020-05-06T15:51:13Z",
-<code>        "id": 2164959,</code>
+        "end": "2020-05-06T16:06:30Z",
+        "ground_station": 1450,
-<code>        "start": "2020-05-06T18:53:17Z",</code>
+        "tle0": "0 NOAA 19",
+        "tle1": "1 33591U 09005A   20126.94623650 +.00000063 +00000-0 +59678-4 0  9999",
-<code>        "end": "2020-05-06T19:00:30Z",</code>
+        "tle2": "2 33591 099.1965 131.1873 0013769 190.9197 169.1677 14.12406430579135",
+        "frequency": 137100000,
-<code>        "ground_station": 1450,</code>
+        "mode": "APT",
+        "transmitter": "kE4VaYKpnFmzEquEjKKi8D",
-<code>        "tle0": "0 METEOR M2",</code>
+        "baud": null
-−
+    }
-<code>        "tle1": "1 40069U 14037A   20126.65202808 -.00000042  00000-0  13670-6 0  9996",</code>
+</pre>
 Clearly, there's no sample rate in that blob, but there is a mode and a baudrate.
@@ Line 139: / Line 129: @@
 Using gr-satnogs/python/utils.py, this is easy to calculate:
+<pre>
-<code>>>> import utils as satnogs</code>
+>>>> import utils as satnogs
 >>>> baudrate = 80000
-<code>>>> baudrate = 80000</code>
+>>>> audio_samp_rate = 48000
 >>>> max(4,satnogs.find_decimation(baudrate, 2, audio_samp_rate))
-<code>>>> audio_samp_rate = 48000</code>
+</pre>
 So, by my calculation, my IQ should be at 4 * 80,000 = 320,000. Hopefully.

@@ Line 1: / Line 1: @@
-I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] thread but as I worked through after picking things up after initially having a problem I realised everything I'd described now magically worked! Typical. Worth documenting.
+''I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] thread but as I worked through after picking things up after initially having a problem I realised everything I'd described now magically worked! Typical. Worth documenting.''
 ==Setting yourself up==
@@ Line 32: / Line 32: @@
 ===Feeding your IQ in===
-Loading the flowgraph in requires some GNU Radio modules chaining together. Starting with [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] you'll end up with something like this:
+Loading the IQ into beginning of the flowgraph in requires some GNU Radio modules chaining together. Starting with [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] you'll end up with something beginning like the below. It takes the file source, throttles the output to real-time, then converts the short values to complex real including reversing the scaling division. This feeds then into the existing flowgraph exactly where it left off.
 [[File:flowgraph_open.png|800px]]
-It took some time for me to work out the appropriate samples rates but I can't remember exactly how now. Scheduling a pass will generate them and then you can work it from there. Maybe I'll come back and update this later.
+=== Sample rate ===
-The rest just reverses throttles the output to real-time, and converts the short values to complex real including the scaling division.
+The IQ recording is generated by the flowgraph in use. If there's no particular decoder in use then that's <code>GNURADIO_DEFAULT_SCRIPT_FILENAME</code> (see [https://gitlab.com/librespacefoundation/satnogs/satnogs-client/-/blob/master/satnogsclient/settings.py#L66 <code>settings.py</code>] as it keeps changing in HEAD - <code>satnogs_fsk_ax25.py</code>, then <code>fsk.grc</code> and now <code>fm.grc</code>). The flowgraph can vary, but it's probably fairly similar for each on the front end.
 ===Results===

@@ Line 1: / Line 1: @@
 I'm not entirely sure where to put this, but perhaps someone more experienced can move and link this from a sensible location. This started as a question continuing the [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] thread but as I worked through after picking things up after initially having a problem I realised everything I'd described now magically worked! Typical. Worth documenting.
-== Setting yourself up ==
+==Setting yourself up==
 See elsewhere, but get GNU Radio up and running and a copy of satnogs-flowgraphs from GitLab.
@@ Line 7: / Line 7: @@
 I'm using a virtual Ubuntu 19.10 machine grabbed from osboxes.org, but you can use whatever you like. I imagine you could even do it on the Raspberry Pi itself. I did end up setting this VM up as a 'development station' but you should only need to install GNU Radio, and any pre-requisites needed by satnogs-flowgraphs - so probably gr-satnogs, I can't quite remember. As you find out please edit this :-)
-== Storing an IQ recording ==
+==Storing an IQ recording==
 There are options in satnogs-setup under Advanced > Radio to store IQ files:
-* ENABLE_IQ_DUMP: Enable IQ dump
+*ENABLE_IQ_DUMP: Enable IQ dump
-* IQ_DUMP_FILENAME: Define IQ dump filename [/tmp/.satnogs/iq.raw]
+*IQ_DUMP_FILENAME: Define IQ dump filename [/tmp/.satnogs/iq.raw]
 Beware, IQ files are huge (220MB for my APT pass example below). They're stored in /tmp so therefore in RAM and are overwritten with each pass. You can do [https://community.libre.space/t/uploading-compressed-iq-files-to-dropbox-or-any-other-cloud-storage-provider/5395 something clever] to upload them elsewhere, but for playing just grab a copy immediately after a pass.
@@ Line 18: / Line 18: @@
 Don't worry about storing much else (which also vanishes quickly). You can easily grab waterfalls and OGG recording from the database via [https://github.com/deckbsd/glouton-satnogs-data-downloader Glouton].
-== GNU Radio Companion ==
+==GNU Radio Companion==
 Here I'm working with [https://network.satnogs.org/observations/2073316/ Observation #2073316] as an example. I've saved a copy of the IQ recording my station briefly stored at `/tmp/.satnogs/iq.raw`. This is about 220 MB. I even remembered to turn the IQ collection off afterwards or it would be storing stuff on top of each other forever.
-=== So where did the IQ come from? ===
+===So where did the IQ come from?===
 In my case, I'm playing with an APT pass. It is therefore processed by [https://gitlab.com/librespacefoundation/satnogs/satnogs-flowgraphs/-/blob/master/satellites/noaa_apt_decoder.grc satnogs-flowgraphs/satellites/noaa_apt_decoder.grc]. Other signals use other flowgraphs. The default where nothing else is suitable appears to be [https://gitlab.com/librespacefoundation/satnogs/satnogs-flowgraphs/-/blob/master/generic/fsk.grc satnogs-flowgraphs/generic/fsk.grc].
@@ Line 30: / Line 30: @@
 That is, it's post Doppler correction, which includes an LO offset; and the values are scaled to shorts, although 16768 seems an weird number as I'd have expected a power of two - 16384? I don't imagine it makes a lot of difference as it'll still fit in the short just lose a little bit of dynamic range.
-=== Feeding your IQ in ===
+===Feeding your IQ in===
 Loading the flowgraph in requires some GNU Radio modules chaining together. Starting with [https://community.libre.space/t/how-to-open-satnogs-iq-dat-files/3793 How to open satnogs iq.dat files] you'll end up with something like this:
@@ Line 40: / Line 40: @@
 The rest just reverses throttles the output to real-time, and converts the short values to complex real including the scaling division.
-=== Results ===
+===Results===
 Initially, the waterfall may look a bit busy compared to the one from the [https://network.satnogs.org/observations/2073316/ observation]. There's obviously some gain setting within the waterfall I've not yet mastered, or the Satnogs ones does something slightly cleverer.
 [[File:Waterfall.png|200px]]
 However, if you peer closer you can just about make out the feint telltales of an APT signal.
@@ Line 52: / Line 56: @@
 [[File:visual.png|300px]]
-=== OGG ===
+===OGG===
 For completeness, you can do the same thing using the OGG file as a source. Again, you need to reverse any resampling or other changes that were done before saving to file before feeding back into the flowgraph at an appropriate point. The throttle is optional, and in the screenshot bypassed:
 [[File:flowgraph_all.png|800px]]