Packaging gr-gsm

[Piotr Krysik]

Hi all,

I'm considering packaging gr-gsm (starting with Ubuntu 14.10 and above
and Debian) but I have no experience with this. I'm looking for advice
what is the easy way to create package in cmake and automake based projects.

Also - maybe there is someone on the list who would like to create
package for gr-gsm and libosmocore (as this library is now required by
gr-gsm).

--
Best Regards,
Piotr Krysik

[Tomcsányi Domonkos]

Forwarding my mail to the mailing list (forgot to hit reply to all instead of just reply), sorry about that.

Hi Piotr,

I’m really happy to see this project making such a good progress, thank you for doing it!

I’m sadly not experienced with creating packages, but it doesn’t look very complicated based on this tutorial:

http://packaging.ubuntu.com/html/packaging-new-software.html

There is also a separate chapter on libraries (for libmosmocore):

http://packaging.ubuntu.com/html/libraries.html

If you think it would require too much effort from your side I think I might be able to do this (learning my way as doing it, I always had some interest for creating packages), I just need to set up gr-gsm and this packaging system for myself (haven’t had the time for it sadly), so let me know.

Regards,

Domi

--
Otrzymujesz tę wiadomość, ponieważ subskrybujesz grupę dyskusyjną Google o nazwie „gr-gsm”.

Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com.
Aby zamieszczać posty w tej grupie, wyślij e-mail na adres gr-...@googlegroups.com.
Aby wyświetlić tę dyskusję w internecie, odwiedź stronę https://groups.google.com/d/msgid/gr-gsm/55C63967.8040502%40gmail.com.
Więcej opcji znajdziesz na https://groups.google.com/d/optout

[Piotr Krysik]

Hi Domonkos,

I've tied to follow the tutorial from the first link but I stopped at:
$ bzr dh-make hello 2.7 hello-2.7.tar.gz

with this result:

bzr: ERROR: Either run the command from an existing branch of upstream,
or move hello aside and a new branch will be created there.

It might be that the tutorial is outdated in this part.

If you want to take this task it would be great. I'm currently
implementing uplink reception/decoding, so probably I won't start doing
packaging before finishing that.

If you have Ubuntu/Debian - setting up gr-gsm should be relatively easy.
You can use pybombs to install it - here is the description:
https://github.com/ptrkrysik/gr-gsm/#installation

After that you will have gr-gsm installed with all dependencies. Then
you can clone and build gr-gsm yourself with:
git clone https://github.com/ptrkrysik/gr-gsm.git
mkdir build
cd build
cmake .. && make && make install

Best Regards,
Piotr

W dniu 09.08.2015 o 16:32, Tomcsányi Domonkos pisze:

>> Hi Piotr,
>>
>> I’m really happy to see this project making such a good progress,
>> thank you for doing it!
>>
>> I’m sadly not experienced with creating packages, but it doesn’t look
>> very complicated based on this tutorial:
>> http://packaging.ubuntu.com/html/packaging-new-software.html
>>
>> There is also a separate chapter on libraries (for libmosmocore):
>> http://packaging.ubuntu.com/html/libraries.html
>>
>> If you think it would require too much effort from your side I think I
>> might be able to do this (learning my way as doing it, I always had
>> some interest for creating packages), I just need to set up gr-gsm and
>> this packaging system for myself (haven’t had the time for it sadly),
>> so let me know.
>>
>> Regards,
>> Domi
>>
>>> 2015. aug. 8. dátummal, 19:16 időpontban Piotr Krysik

>>> <ptrk...@gmail.com <mailto:ptrk...@gmail.com>> írta:

>>>
>>> Hi all,
>>>
>>> I'm considering packaging gr-gsm (starting with Ubuntu 14.10 and above
>>> and Debian) but I have no experience with this. I'm looking for advice
>>> what is the easy way to create package in cmake and automake based
>>> projects.
>>>
>>> Also - maybe there is someone on the list who would like to create
>>> package for gr-gsm and libosmocore (as this library is now required by
>>> gr-gsm).
>>>
>>> --
>>> Best Regards,
>>> Piotr Krysik
>>>
>>> --
>>> Otrzymujesz tę wiadomość, ponieważ subskrybujesz grupę dyskusyjną
>>> Google o nazwie „gr-gsm”.
>>>
>>> Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej
>>> wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com

>>> <mailto:gr-gsm+un...@googlegroups.com>.

>>> Aby zamieszczać posty w tej grupie, wyślij e-mail na adres

>>> gr-...@googlegroups.com <mailto:gr-...@googlegroups.com>.

>>> Aby wyświetlić tę dyskusję w internecie, odwiedź stronę
>>> https://groups.google.com/d/msgid/gr-gsm/55C63967.8040502%40gmail.com.
>>> Więcej opcji znajdziesz na https://groups.google.com/d/optout
>>
>

> --
> Otrzymujesz tę wiadomość, bo subskrybujesz grupę „gr-gsm” w Grupach
> dyskusyjnych Google.

> Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej
> wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com

> <mailto:gr-gsm+un...@googlegroups.com>.
> Aby opublikować wpis w tej grupie, wyślij e-maila na
> gr-...@googlegroups.com <mailto:gr-...@googlegroups.com>.
> Aby wyświetlić tę dyskusję w internecie, otwórz
> https://groups.google.com/d/msgid/gr-gsm/3EE27141-1A6C-496D-8104-B41B44281F47%40tomcsanyi.net
> <https://groups.google.com/d/msgid/gr-gsm/3EE27141-1A6C-496D-8104-B41B44281F47%40tomcsanyi.net?utm_medium=email&utm_source=footer>.
> Więcej opcji znajdziesz na https://groups.google.com/d/optout.

[Tomcsányi, Domonkos]

Hi Piotr,

I have a full gnuradio setup, and a device as well, so I was trying to figure out yesterday what the capabioities of gr-gsm are (I compiled from source). I could see that it has some great ideas in it but it lacks a lot of documentation.
I would like to also work on that a little bit, because it is no good in my opinion to have for example hopping support, but nobody can use it :).

Also I tried decoding the example file from Srlabs, but I ended up only with noise in the output voice file. I used the right voice vodec (GSM FR) in my opinion. The terminal output showed some valid gsm frames, so I think the decimation/sample rate setup was right as well.

Also I noticed the large amount of ICMP messages while parsing a file. Airprobe and osmocomBB solved this problem as far as I remember so it might be worth looking into as well.

Uplink support sounds fantastic, it would be great for sure!

Regards,
Domi

>>>> > írta:
>>>>
>>>> Hi all,
>>>>
>>>> I'm considering packaging gr-gsm (starting with Ubuntu 14.10 and above
>>>> and Debian) but I have no experience with this. I'm looking for advice
>>>> what is the easy way to create package in cmake and automake based
>>>> projects.
>>>>
>>>> Also - maybe there is someone on the list who would like to create
>>>> package for gr-gsm and libosmocore (as this library is now required by
>>>> gr-gsm).
>>>>
>>>> --
>>>> Best Regards,
>>>> Piotr Krysik
>>>>
>>>> --
>>>> Otrzymujesz tę wiadomość, ponieważ subskrybujesz grupę dyskusyjną
>>>> Google o nazwie „gr-gsm”.
>>>>
>>>> Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej
>>>> wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com

>>>> .
>>>> Aby zamieszczać posty w tej grupie, wyślij e-mail na adres

>>>> gr-...@googlegroups.com .

>>>> Aby wyświetlić tę dyskusję w internecie, odwiedź stronę
>>>> https://groups.google.com/d/msgid/gr-gsm/55C63967.8040502%40gmail.com.
>>>> Więcej opcji znajdziesz na https://groups.google.com/d/optout
>>
>> --
>> Otrzymujesz tę wiadomość, bo subskrybujesz grupę „gr-gsm” w Grupach
>> dyskusyjnych Google.
>> Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej
>> wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com

>> .
>> Aby opublikować wpis w tej grupie, wyślij e-maila na

>> gr-...@googlegroups.com .

>> Aby wyświetlić tę dyskusję w internecie, otwórz
>> https://groups.google.com/d/msgid/gr-gsm/3EE27141-1A6C-496D-8104-B41B44281F47%40tomcsanyi.net

>> .
>> Więcej opcji znajdziesz na https://groups.google.com/d/optout.
>

> --
> Otrzymujesz tę wiadomość, ponieważ subskrybujesz grupę dyskusyjną Google o nazwie „gr-gsm”.
>

> Aby anulować subskrypcję tej grupy i przestać otrzymywać od niej wiadomości, wyślij e-maila na gr-gsm+un...@googlegroups.com.
> Aby zamieszczać posty w tej grupie, wyślij e-mail na adres gr-...@googlegroups.com.
> Aby wyświetlić tę dyskusję w internecie, odwiedź stronę https://groups.google.com/d/msgid/gr-gsm/55C83D0A.6050905%40gmail.com.

[Piotr Krysik]

Hi Domonkos,

Lack of documentation is definitely weak point of gr-gsm currently that
requires fixing. If you like to work on this I can explain the ideas and
show some examples. As infrastructure we can use doxygen and/or github
pages.

Regarding the decoding of full rate voice transmission - it is probably
regression in decoding support after extensive development. This
highlights another shortcoming which is low coverage by automated tests.
Roman added infrastructure for tests and started adding first automated
tests for some of his blocks.

As for ICMP messages - I solved this problem on the level of wireshark.
Users are adviced to use:
sudo wireshark -k -f udp -Y gsmtap -i lo

where "-f udp" is responsible for excluding anything else than udp.
As far as I remember in osmocomBB they don't have this problem because
they send packages multicast addresses (and not localhost like in
gr-gsm). Drawback of their approach might be that it requires to have a
rule in routing table for such addresses, which usually is added when
you activate physical interface.

Best Regards,
Piotr Krysik

W dniu 10.08.2015 o 09:30, Tomcsányi, Domonkos pisze:
> Hi Piotr,
>

[roman khassraf]

On Monday, 10 August 2015 13:46:59 UTC+2, Piotr Krysik wrote:

Regarding the decoding of full rate voice transmission - it is probably
regression in decoding support after extensive development.

Maybe it's just a missing magic bytes sequence at the beginning.
Another possible reason is that toast expects a 4 bit header before a every frame, could be just the wrong bit sequence used for padding.

I never tested it using toast, I just opened the resulting speech file with VLC player.
That is still working for me with the current branch.

 

As for ICMP messages - I solved this problem on the level of wireshark.
Users are adviced to use:
sudo wireshark -k -f udp -Y gsmtap -i lo

 
Wireshark also offers custom capture filters in the GUI, that may be more convenient.
Those filters can be set via "edit interface settings" (double click on the list of interfaces)

[Piotr Krysik]

W dniu 10.08.2015 o 16:55, roman khassraf pisze:

> On Monday, 10 August 2015 13:46:59 UTC+2, Piotr Krysik wrote:
>
>
> Regarding the decoding of full rate voice transmission - it is probably
> regression in decoding support after extensive development.
>
>
> Maybe it's just a missing magic bytes sequence at the beginning.
> Another possible reason is that toast expects a 4 bit header before a
> every frame, could be just the wrong bit sequence used for padding.
>
> I never tested it using toast, I just opened the resulting speech file
> with VLC player.
> That is still working for me with the current branch.

Hi Roman,

Indeed it works directly with normal players like vlc or mplayer - with
a bit of noisy hiccup at the beginning. But it works without having to
do conversion with toast, which is great! If it would be possible to get
rid of the noises at the beginning it would be just ideal.

I've attached versions of the audio file:
-obtained with older gr-gsm (form June 13) and toast,
-obtained with use of current gr-gsm and without toast.

[roman khassraf]

Indeed it works directly with normal players like vlc or mplayer - with
a bit of noisy hiccup at the beginning. But it works without having to
do conversion with toast, which is great! If it would be possible to get
rid of the noises at the beginning it would be just ideal.

Hmm, probably switching the Viterbi to the OpenBTS code causes the difference.
I will try to find the difference the next days, maybe it isn't too hard to fix for compatibility reasons.

The noise is on my radar, it's pretty annoying.
But I believe the header issue won't fix that, I think in your file it's toast's decoding which is optimizing the noise out.
I think the first 3-4 seconds shouldn't be decoded at all, probably there is some way to detect the corresponding bursts.

[Tomcsányi Domonkos]

Hi all,

Thanks for jumping on my issue so fast, I’ll answer to this mail, despite it’s not the latest, but it still contains the most info I’d like to react to.

2015. aug. 10. dátummal, 16:55 időpontban roman khassraf <rkha...@gmail.com> írta:

On Monday, 10 August 2015 13:46:59 UTC+2, Piotr Krysik wrote:

Regarding the decoding of full rate voice transmission - it is probably
regression in decoding support after extensive development.

Maybe it's just a missing magic bytes sequence at the beginning.
Another possible reason is that toast expects a 4 bit header before a every frame, could be just the wrong bit sequence used for padding.

I never tested it using toast, I just opened the resulting speech file with VLC player.
That is still working for me with the current branch.

I tried without toast, using VLC as well, the most I got was a continuous flow of “Missing GSM magic” messages or something like that.

I don’t know where I’m messing this up.

I use the airprobe_tchf.grc from the examples directory. I set the input file to the cfile from Srlabs. I set the FC to 1.8478G (since the call was recorded on ARFCN 725). Sample rate is 100e6/174 (174 decimation and the USRP2’s frequency is 100 MHz). I didn’t change the default TS5 since that’s the right one. I set the key [0x1E, 0xF0, 0x0B etc.]. I left the TCH/F decoder on GSM-FR, since it is not EFR (EFR would be the TE option of airprobe if I understand correctly).

The resulting file is just noise.

 

As for ICMP messages - I solved this problem on the level of wireshark.
Users are adviced to use:
sudo wireshark -k -f udp -Y gsmtap -i lo

 
Wireshark also offers custom capture filters in the GUI, that may be more convenient.
Those filters can be set via "edit interface settings" (double click on the list of interfaces)

I know about this solution for sure, but I think it is not the right one. Instead of filtering out the unwanted messages we should avoid creating them in the first place I think. I think the multicast solution is a good one, I’m pretty sure no routing is needed for that.

About the noise in the beginning of the voice file: I have always had that on all the calls I decoded with airprobe.

I’ll look into this stuff during the coming weeks and try to figure things out :). I’d really love to have this project running stable and fine, because I think after my Hacktivity talk 2 years ago I wasn’t able to deliver the needed stable tools for the community I was planning to. So I’m all up and ready for pushing this forward, that’s why I offered my help in the first place…sadly as you can see I’m not that great of a developer, but I’ll try to get into that as well :).

Regards,

Domi

[roman khassraf]

I tried without toast, using VLC as well, the most I got was a continuous flow of “Missing GSM magic” messages or something like that.

I don’t know where I’m messing this up.

I use the airprobe_tchf.grc from the examples directory. I set the input file to the cfile from Srlabs. I set the FC to 1.8478G (since the call was recorded on ARFCN 725). Sample rate is 100e6/174 (174 decimation and the USRP2’s frequency is 100 MHz). I didn’t change the default TS5 since that’s the right one. I set the key [0x1E, 0xF0, 0x0B etc.]. I left the TCH/F decoder on GSM-FR, since it is not EFR (EFR would be the TE option of airprobe if I understand correctly).

The resulting file is just noise.

I attached a flow graph that you can test, it worked for me. 

I know about this solution for sure, but I think it is not the right one. Instead of filtering out the unwanted messages we should avoid creating them in the first place I think. I think the multicast solution is a good one, I’m pretty sure no routing is needed for that.

From my perspective, there is nothing wrong with gr-gsm here. It just packs the gsm messages in gsmtap datagram, and sends it to udp port 4729, as expected.
The icmp messages are generated by the OS because the destination port is closed. Opening the port is beyond of the scope of gr-gsm's task.
So, if you really want to avoid the creation of those icmp responses, the correct solution from my view would simply be opening that port, using for example netcat. On the other hand, you probably still want to set a wireshark capture filter to capture only packets to udp port 4729, otherwise you may also have other packets in your capture, like dns queries and other connections on your localhost interface.

 

[Piotr Krysik]

Hi Domonkos,

W dniu 11.08.2015 o 00:22, Tomcsányi Domonkos pisze:

> Hi all,
>
> Thanks for jumping on my issue so fast, I’ll answer to this mail,
> despite it’s not the latest, but it still contains the most info I’d
> like to react to.
>
>> 2015. aug. 10. dátummal, 16:55 időpontban roman khassraf

>> <rkha...@gmail.com <mailto:rkha...@gmail.com>> írta:

>> On Monday, 10 August 2015 13:46:59 UTC+2, Piotr Krysik wrote:
>> Regarding the decoding of full rate voice transmission - it is
>> probably
>> regression in decoding support after extensive development.
>> Maybe it's just a missing magic bytes sequence at the beginning.
>> Another possible reason is that toast expects a 4 bit header before a
>> every frame, could be just the wrong bit sequence used for padding.
>>
>> I never tested it using toast, I just opened the resulting speech file
>> with VLC player.
>> That is still working for me with the current branch.
>>
>
> I tried without toast, using VLC as well, the most I got was a
> continuous flow of “Missing GSM magic” messages or something like that.
> I don’t know where I’m messing this up.
> I use the airprobe_tchf.grc from the examples directory. I set the input
> file to the cfile from Srlabs. I set the FC to 1.8478G (since the call
> was recorded on ARFCN 725). Sample rate is 100e6/174 (174 decimation and
> the USRP2’s frequency is 100 MHz). I didn’t change the default TS5 since
> that’s the right one. I set the key [0x1E, 0xF0, 0x0B etc.]. I left the
> TCH/F decoder on GSM-FR, since it is not EFR (EFR would be the TE option
> of airprobe if I understand correctly).
> The resulting file is just noise.
>

The example should work after changing only the input file path. The
rest can stay as it is - even though the carrier frequency is incorrect
as you observed. Anyway with changing carrier or not the result is the
same - a lot of noise at the beginning and then short quiet speech "this
is the test call six". Do you get different output file than what I
attached to my previous message (the version not requiring toast)?

>>
>> As for ICMP messages - I solved this problem on the level of
>> wireshark.
>> Users are adviced to use:
>> sudo wireshark -k -f udp -Y gsmtap -i lo
>>
>>
>> Wireshark also offers custom capture filters in the GUI, that may be
>> more convenient.
>> Those filters can be set via "edit interface settings" (double click
>> on the list of interfaces)
>>
>
> I know about this solution for sure, but I think it is not the right
> one. Instead of filtering out the unwanted messages we should avoid
> creating them in the first place I think. I think the multicast solution
> is a good one, I’m pretty sure no routing is needed for that.
>

Here my experience is different - if I don't have default route set the
for any network interface multicast packages are going nowhere. If I get
up an interface that has the default route - multicast packages start to
appear on it.
The next thing is that I want to keep configuration as simple an unified
for all users. By starting:

sudo wireshark -k -f udp -Y gsmtap -i lo

user doesn't have to do any additional configuration of wireshark or
network interfaces. The network interface also always stay the same -
lo. With multicast it could be any of plenty names like ethX, wlanX.

> About the noise in the beginning of the voice file: I have always had
> that on all the calls I decoded with airprobe.
>
> I’ll look into this stuff during the coming weeks and try to figure
> things out :). I’d really love to have this project running stable and
> fine, because I think after my Hacktivity talk 2 years ago I wasn’t able
> to deliver the needed stable tools for the community I was planning to.
> So I’m all up and ready for pushing this forward, that’s why I offered
> my help in the first place…sadly as you can see I’m not that great of a
> developer, but I’ll try to get into that as well :).
>

In gr-gsm I want to finish some of the TODO's that I left in the old
Airprobe, so my motivation to work on this topic is similar. I hope that
you wont be discouraged by initial difficulties of gr-gsm. We need
someone who will look at this software from user perspective and make it
into something more attractive for end user (by for example proposing
what could be added/done-differently in order to achieve that goal,
doing tests, writing tutorials, etc.). You have shown already with your
past work that you are able to do a lot in this area.

Best Regards,
Piotr

[Tomcsányi Domonkos]

Hi all,

This is what I remembered instead of the multicast addresses for GSMtap:

https://github.com/ttsou/airprobe/commit/99de0b0ce9ac711eaca1b3d8951e3bfc606306d8

So libosmocore provides a way to do that.

No pressure from me, just wanted to put it out there so you know about it.

Cheers,

Domi

[roman khassraf]

Hi Domi,

they are opening the destination port.
You can just add another Socket PDU to your flow graph: type udp server, host 127.0.0.1, port 4729.
Has the same effect.

Roman

 

[Piotr Krysik]

W dniu 12.08.2015 o 00:07, Tomcsányi Domonkos pisze:

Hi all,

The solution in that commit is to listen on UDP port to which the
program sends gsmtap data.
For us - this a matter of adding in gnuradio-companion another Socket
PDU block that is configured to be "UDP Server".

I can do this.

--
Best Regards,
Piotr

[Ash Wilson]

Hey Piotr,

  It's been a while, but I could figure out packaging again.  In a past life I've built packages for Debian, Solaris, and RHEL.  We should look at using tags and/or branches to reflect versions of the tool, though, before we begin.  That'll make it much easier to maintain going forward, and let us roll back easier if things get weird.

-Ash

[Piotr Krysik]

Hi Ash,

Your proposition is great news for me. I don't have experience with
packaging so I might need your guidance regarding required
infrastructure (like how to use tags or branches in order to assist
packaging).

Regarding distributions for which it would be good to have packages:
support for distributions mentioned by you (Debian, Ubuntu, Fedora)
would expose the project to very wide audience. The gr-gsm project
depends on libosmocore from osmocom project - decryption block uses
osmocom's implementation of GSM ciphers. As libosmocore is currently
packaged for Debian, it will probably be easiest to start with that
distribution. GSM ciphers are separated from the rest in the libosmogsm5
package (https://packages.debian.org/sid/libosmogsm5).

What do you need to start working on packaging of gr-gsm? (I will be
quite busy in coming week so please don't get distracted if it will take
some time for me to reply).

Best Regards,
Piotr


W dniu 24.01.2016 o 07:36, Ash Wilson pisze:

> --

[Ashley Wilson]

Hi Piotr,

So sorry to keep you waiting on a response!   I'm slammed this week but I hope to dig into it this weekend or early next week, and I'll get some actionable suggestions for you ASAP so we can get this easily deployable.

>
> --

--
Otrzymujesz tę wiadomość, ponieważ subskrybujesz temat w grupie „gr-gsm” w Grupach dyskusyjnych Google.
Aby anulować subskrypcję tego tematu, otwórz https://groups.google.com/d/topic/gr-gsm/HQsvTQkIp6M/unsubscribe.
Aby anulować subskrypcję tej grupy i wszystkich jej tematów, wyślij e-maila na gr-gsm+un...@googlegroups.com.

Aby zamieszczać posty w tej grupie, wyślij e-mail na adres gr-...@googlegroups.com.

Aby wyświetlić tę dyskusję w internecie, odwiedź stronę https://groups.google.com/d/msgid/gr-gsm/56A52E66.4040504%40gmail.com.