DBNBeatTracker on Stream Input

[Ian Charnas]

Hello, I am really enjoying working with the madmom library! 

I want to perform real-time beat detection on an input stream. Here is my code:

#!/usr/bin/env python3

from madmom.models import BEATS_LSTM

from madmom.features.beats import DBNBeatTrackingProcessor, RNNBeatProcessor

from madmom.audio.signal import Stream

stream = Stream(

    sample_rate = 44100,

    num_channels = 1,

    frame_size = 2048,

    fps = 100

)

rnn = RNNBeatProcessor(

    online = True,

    nn_files = [BEATS_LSTM[0]]

)

dbn = DBNBeatTrackingProcessor(

    min_bpm = 100,

    max_bpm = 180,

    fps = 100,

    online = True

)

dbn.reset()

for frame in stream:

    activations = rnn.process(frame)

    beats = dbn.process(activations, reset=False)

    if len(beats) > 0:

        print(beats)

I am not sure if I am doing "frame by frame" processing correctly, because there are two problems.

First, when it prints beats, I see that the time (in seconds) is much larger than it should be. After just 10 seconds of actual audio processing, the beat tracker is printing times of nearly 50 seconds.

Secondly, the code is printing around 2 times the number of actual beats. 

Am I setting the frame_size or fps incorrectly? Or any ideas what could be causing these problems?

many thanks in advance for any help and advice,

Ian

[Sebastian Böck]

Allow me to shorten the story a bit: simply use `DBNBeatTracker online` and you should be set :)

It's a bit tricky to get everything right, but the example script should help you to get everything right. But probably there's no need to do so because everything is working out of the box already.

HTH

[Ian Charnas]

Dear Sebastian, 

Thank you for your help! You live in Austria, right? I visited Linz several times and really loved it.

Your suggestion was great to help me figure this out. I followed that code and then I understood what was happening. For anyone else who wants to incorporate real-time beat tracking into their project, here is the finished and working code (below).

One issue - there is a noticeable delay (maybe 100 - 200ms?) between when I hear the beat of the music and when I see the beat printed to my screen. For my project that is OK, because I am sending these beats to a digital phase locked loop (DPLL) and I can subtract that delay from each of the beat times before sending it off to the DPLL process. 

However, Sebastian is there anything else I can do to optimize this or reduce that delay? Normally I would try lowering the sample rate, and changing the fft window size and hop size accordingly, however I believe the neural nets were trained on a 44100 sample rate, so I am thinking I should not change it?

from madmom.features.beats import DBNBeatTrackingProcessor, RNNBeatProcessor

from madmom.models import BEATS_LSTM

from madmom.processors import IOProcessor, process_online

kwargs = dict(

    fps = 100,

    correct = True,

    infile = None,

    outfile = None,

    max_bpm = 170,

    min_bpm = 100,

    nn_files = [BEATS_LSTM[0]],

    num_frames = 1,

    online = True,

    #verbose = 1

)

def beat_callback(beats, output=None):

    if len(beats) > 0:

        # Do something with the beat (for now, just print the array to stdout)

        print(beats)

in_processor = RNNBeatProcessor(**kwargs)

beat_processor = DBNBeatTrackingProcessor(**kwargs)

out_processor = [beat_processor, beat_callback]

processor = IOProcessor(in_processor, out_processor)

process_online(processor, **kwargs)

[Sebastian Böck]

Hi,

On Sunday, 1 September 2019 16:16:25 UTC+2, Ian Charnas wrote:

One issue - there is a noticeable delay (maybe 100 - 200ms?) between when I hear the beat of the music and when I see the beat printed to my screen. For my project that is OK, because I am sending these beats to a digital phase locked loop (DPLL) and I can subtract that delay from each of the beat times before sending it off to the DPLL process.

That surprises me a bit, since the whole system should be able to detect the beats within 1 (or maybe 2) frame(s). This should be negligible but more or less unavoidable. I am pretty sure about this, but you can try it for yourself by calling your Processor frame by frame with an offline signal you have annotations for. Of course this depends on the type of music you're processing, but for percussive music this is generally true.

However, Sebastian is there anything else I can do to optimize this or reduce that delay? Normally I would try lowering the sample rate, and changing the fft window size and hop size accordingly, however I believe the neural nets were trained on a 44100 sample rate, so I am thinking I should not change it?

Changing it slightly might work. However (as indicated above), the delay does not need to stem from the neural network. There's also a good chance that portaudio (or anything else in your audio setup) causes this delay.

If you are able to figure this out, please post your findings here.

HTH

[Ian Charnas]

On Sunday, 1 September 2019 16:16:25 UTC+2, Ian Charnas wrote:

One issue - there is a noticeable delay (maybe 100 - 200ms?) between when I hear the beat of the music and when I see the beat printed to my screen. For my project that is OK, because I am sending these beats to a digital phase locked loop (DPLL) and I can subtract that delay from each of the beat times before sending it off to the DPLL process.

That surprises me a bit, since the whole system should be able to detect the beats within 1 (or maybe 2) frame(s). This should be negligible but more or less unavoidable. I am pretty sure about this, but you can try it for yourself by calling your Processor frame by frame with an offline signal you have annotations for. Of course this depends on the type of music you're processing, but for percussive music this is generally true.

Sebastian, you are correct! I tracked down the source of the delay, and it was a combination of network delay (I was using ssh to login remotely), stdout buffering, and maybe a tiny bit of terminal slowness too. So there was not really any noticeable delay caused by madmom. The beat detection algorithm is VERY good! Much better than the other ones I tried before I found madmom. Well done!

One small suggestion - in signal.py where you open the pyaudio stream, it would be nice if I could specify which input device to use. When you call pa.open, could you pass in a "input_device_index" parameter, and allow me to pass that in as a kwarg when I call process_online? The parameter can default to None, since that is the default for the pyaudio function.

Excellent software! Thank you for your work.

Ian

[Sebastian Böck]

Hi Ian,

On Sunday, 1 September 2019 21:59:43 UTC+2, Ian Charnas wrote:

The beat detection algorithm is VERY good! Much better than the other ones I tried before I found madmom. Well done!

And it is even more robust if you use not a single network (as in your code example above) but multiple ones. Even using only a second one (no need to use all 8 networks) makes the tracker even better :)

One small suggestion - in signal.py where you open the pyaudio stream, it would be nice if I could specify which input device to use. When you call pa.open, could you pass in a "input_device_index" parameter, and allow me to pass that in as a kwarg when I call process_online? The parameter can default to None, since that is the default for the pyaudio function.

Could you please open an issue on Github (https://github.com/CPJKU/madmom/issues) so this suggestion won't be lost. Or — since it's a very minor change that should be easy to implement — you could also write a patch and submit a PR. As madmom's argument name I'd suggest `stream_input_device` and map it to pyaudio's `input_device_index`, but I don't have a very strong opinion on that.

Excellent software! Thank you for your work.

Thanks! I'm glad people find it useful.

[Ian Charnas]

Hi Sebastian,

I ran some experiments over the weekend and tried all combinations of neural networks on the music files I'm using in my project. 

The following was the most accurate for my music: 
nn_files = [BEATS_LSTM[0], BEATS_LSTM[1], BEATS_LSTM[2], BEATS_LSTM[3], BEATS_LSTM[4]]

I am running this on a Raspberry Pi 3, and beat tracking takes up about 80% of one core, but no problem - I have four cores!

Also, I did a fresh "git clone https://github.com/CPJKU/madmom.git" and made the changes we talked about for setting the input device index. I then did a "git diff" to produce a patch, and I'm attaching that to this email. I don't know how to do a pull request - do I need to have write permission to the github repo for that?

all the best,

Ian 

--
You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/c30a13ff-4ce1-4645-a19f-f69e97fe6bad%40googlegroups.com.

--

Ian Charnas (he/him) | Director of Innovation and Technology | Sears think[box]

Case Western Reserve University

+1 (216) 368-1492

Subscribe to think[box] Announcements

Email Policy

I respond to all emails within one business day.

For urgent issues, please call.

[Sebastian Böck]

Hi Ian,

thanks for sending this patch!

On Tuesday, 3 September 2019 16:50:55 UTC+2, Ian Charnas wrote:

Also, I did a fresh "git clone https://github.com/CPJKU/madmom.git" and made the changes we talked about for setting the input device index. I then did a "git diff" to produce a patch, and I'm attaching that to this email. I don't know how to do a pull request - do I need to have write permission to the github repo for that?

To open a PR, you

- need a GitHub account

- fork the project

- edit the files

  - either online on Github

  - or offline by creating a branch and pushing to it

- open a PR in order to merge your changes into the main madmom repo

But I can also simply use your supplied patch and do this for you. Btw, an option to `process_online` and the parser should be added as well so you can set the device from the command line.

[Sebastian Böck]

Hi,

I used your patch to create a new PR (https://github.com/CPJKU/madmom/pull/449). I also added the `--device` option to be able to select the device from command line.

However, I don't have a setup with multiple audio interfaces, so I can't really test it. Ian, could you be so kind to test this branch?

[Sebastian Böck]

Sorry, outdated link, new PR is here: https://github.com/CPJKU/madmom/pull/451

[Ian Charnas]

Thank you Sebastian!

I am happy to help test this, however I don't understand how to download the branch with the changes in it. 

I tried "git clone https://github.com/CPJKU/madmom/pull/451" but that didn't seem to work.

Sorry, I am very new to this!

On Wed, Sep 4, 2019 at 4:19 AM Sebastian Böck <sebastian.bo...@gmail.com> wrote:

Sorry, outdated link, new PR is here: https://github.com/CPJKU/madmom/pull/451

--

You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/4a23cfa9-7eee-40ac-8af3-f541d2f85139%40googlegroups.com.

[Sebastian Böck]

Hi Ian,

On Thursday, 5 September 2019 17:14:41 UTC+2, Ian Charnas wrote:

I am happy to help test this, however I don't understand how to download the branch with the changes in it. 

I tried "git clone https://github.com/CPJKU/madmom/pull/451" but that didn't seem to work.

After cloning the repo, you have to checkout the `select_stream_input` branch, i.e.

git clone https://github.com/CPJKU/madmom

git checkout select_stream_input

And then run the setup, e.g. with:

python setup.py develop --user

Then you should be able to simply run `DBNBeatTracker online --device 1` or the like.

HTH

[Ian Charnas]

Thanks Sebastian, I gave that a try but I got an error running DBNBeatTracker:

pi@host:/tmp/madmom/bin $ python3 ./DBNBeatTracker
Traceback (most recent call last):
  File "./DBNBeatTracker", line 12, in <module>
    from madmom.audio import SignalProcessor
  File "/home/pi/.local/lib/python3.7/site-packages/madmom/__init__.py", line 24, in <module>
    from . import audio, evaluation, features, io, ml, models, processors, utils
ImportError: cannot import name 'models' from 'madmom' (/home/pi/.local/lib/python3.7/site-packages/madmom/__init__.py)

It looks like it can't find the models module, and when I do an "ls /home/pi/.local/lib/python3.7/site-packages/madmom/" I get the following, which does not include models.

audio

conftest.py

evaluation

features

__init__.py

io

ml

processors.py

__pycache__

utils

Did I do something wrong?

--

You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/9d1d0514-15d4-4b83-96e1-7168dca339d2%40googlegroups.com.

[Sebastian Böck]

Hi Ian,

On Saturday, 7 September 2019 03:36:11 UTC+2, Ian Charnas wrote:

Thanks Sebastian, I gave that a try but I got an error running DBNBeatTracker:

[...]

It looks like it can't find the models module, and when I do an "ls /home/pi/.local/lib/python3.7/site-packages/madmom/" I get the following, which does not include models.

sorry for my incomplete instructions, here are the complete ones:

https://madmom.readthedocs.io/en/latest/installation.html#install-from-source

The `git submodule update --init --remote` step, which initialises the models submodule, was missing.

Best,

Sebastian

[Ian Charnas]

Hi Sebastian, sorry for the late reply. I was finishing up a big project and I lost track of this thread.

Now that I had some time to look into this again, here are my results.

I installed the new branch as you suggested and tried to use it:

git clone https://github.com/CPJKU/madmom.git

cd madmom

git checkout select_stream_input

python setup.py develop --user

DBNBeatTracker online --device 0

But I got this error:

Traceback (most recent call last):

  File "bin/DBNBeatTracker", line 111, in <module>
    main()
  File "bin/DBNBeatTracker", line 92, in main
    in_processor = RNNBeatProcessor(**vars(args))
  File "/Users/iancharnas/Downloads/madmom/madmom/features/beats.py", line 79, in __init__
    from ..models import BEATS_LSTM, BEATS_BLSTM
ImportError: cannot import name 'BEATS_LSTM' from 'madmom.models' (unknown location)

So then I checked to see what was in the models directory:

ls madmom/models

But there were no files in there at all. 

Did I do something wrong?

thanks for all of your help,

Ian

--

You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/65acc3f3-d72e-47be-8720-b7cbc1b12b0a%40googlegroups.com.

--

Ian Charnas | Creative Engineer | iancharnas.com

[Sebastian Böck]

Hi Ian,

On Tuesday, 1 October 2019 18:49:49 UTC+2, Ian Charnas wrote:

But I got this error:

[...]

ImportError: cannot import name 'BEATS_LSTM' from 'madmom.models' (unknown location)

So then I checked to see what was in the models directory:

ls madmom/models

But there were no files in there at all. 

Did I do something wrong?

You missed the `git submodule update --init --remote` step (to be called from within the directory where you cloned madmom into). This command populates the models directory with the models.

HTH

[Ian Charnas]

Of course! Sorry about that, your instructions were clear, my apologies.

Yes I just tried this and it worked perfectly! I had a USB audio device that pyaudio recognized at index 2, and "DBNBeatTracker online --device 2" worked without any issues. I am very happy!

If anyone is reading this forum in the future and wants to know how they can figure out the device index, try running this python code

import pyaudio

p = pyaudio.PyAudio()
num_devices = p.get_device_count()

for i in range(num_devices): print(p.get_device_info_by_index(i))

hope this helps someone, and three cheers to Sebastian and all of the people who have put time into developing this great piece of software! 

Ian

--

You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/9d34ce65-8771-4339-9676-3ba96e19300f%40googlegroups.com.

[Sebastian Böck]

Hi,

On Tuesday, 1 October 2019 19:15:26 UTC+2, Ian Charnas wrote:

If anyone is reading this forum in the future and wants to know how they can figure out the device index, try running this python code

import pyaudio

p = pyaudio.PyAudio()
num_devices = p.get_device_count()

for i in range(num_devices): print(p.get_device_info_by_index(i))

thanks for the suggestion! I added a `--list` option to online mode which shows a list of available PyAudio devices.

[Ian Charnas]

My hero!!!

--

You received this message because you are subscribed to the Google Groups "madmom-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to madmom-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/madmom-users/3f4ae332-d518-4bc1-8837-f89c01793e7e%40googlegroups.com.