Release 4.0.beta

[Klaas Reineke]

just pushed a new release live with a fix for soft replay gain. It should now work as expected.

Regards Klaas

[Olav Sunde]

Hi Klaas

I have observed a bug in 4.0beta that I believe comes from how brutefirwrapper handles 16bit files.

I've created separeat filters for 44.1, 48, 88.2, 96, 176.4 and 192 in Audiolense and, per your instructions, prepared these for use with BrutefirDRC.

I simply love this functionality as it enables me to have Audiolense create filters for all samplerates with it's internal (and superior) routine. Very nice work!

With BrutefirDrc set to FLAC all file types and sample rates are handled correctly. With PCM however, a 16bit flac file played through BrutefirDRC is just full level white noise, while the exact same file converted to 24bit plays normally.


I have found two ways to fix this for my self: If I let brutefirwrapper do it's resample routine (i.e. no *44100.txt filter present so brutefirwrapper engages resampling when it detects a 16bit stream), a 16/44.1 file plays as normal. If I keep the *44100.txt, but manually replace S24_LE with S16_LE, all 16/44.1 material play normal.

So the new automatic filter select routine in brutefirwrapper seems to ignore the '--copy-sample-format' command for PCM skipping "Patching brutefir input format from S24_LE to S16_LE" and "Patching brutefir output format from S24_LE to S16_LE" processing 16bit files.

Note: this happens with all 16bit files. A 16/88.2 or 16/48 file will have the same problem.

Olav

[Mervin Beng]

Hi Olav, Klaas,

I've not been playing around with the wrapper for a long time, but I recall that when I was streaming pcm to by SBTouch I realised that all the flac files would need to be converted to 24 bits. Fortunately with flac 16-bit audio stored as 24-bit takes up the same space, IIRC.

From my recollection it's because the server sees that your source format is xx/16, and after the audio is piped and filtered through the wrapper and then sent out by the server, it sends it as xx/16 pcm. Likewise if the source is /24, the server tells the player the pcm is /24.

The issue is that the wrapper *always* gets 24 bit audio (from the sox -b 24), and has no way that I know of to know if the source on the server was 16 converted to 24, or originally 24. Therefore the wrapper always returns 24 bit wav. When the server sends flav or wav to the player, that's no problem, as it always sends 24 bit, and flac includes the word depth. When sending pcm, the server apparently just strips the header and sends the rest (which is pcm). The problem is that the pcm is 24 bit, but the player is told it's 16 bit if the source material was 16 bit.

Please don't take my explanation to be 100% correct. It may be outdated or incorrect.

Cheers,
Mervin

[Olav Sunde]

Hi Mervin, good to hear from you!

Toby made some changes to brutefirwrapper for handling PCM as 16 or 24bit:

  --copy-sample-format  Patch the brutefir output sample format (16 or 24 bit)
                        to match the input wav stream. This matches the
                        behaviour expected by squeeezeboxserver for pcm output
                        streams.
After convolving in 24bit, this option should change output from the wrapper back to the file's on-disk bit depth so that LMS receive what it expects.

The behaviour I report may have existed before. I just did not notice as I have always used a correction file from Audiolense at 48kHz and  brutefirwrapper has done resampling.

Olav

[Klaas Reineke]

Hi Olav,

I did switch back to play all audio as flac quite some time ago, so this problem did go unnoticed. My update to brutefirwrapper is just a real simple search for a filter with the correct name. I guess the only option is to change all bitrates in the brutefir config file as you describe it. I guess I will not implement a fix for pcm in the near future, because of my limited knowledge of python and brutefirwrapper. I would recommend to use flac streaming but I guess you would like to keep pcm streaming :)

Your patch will not work for 24bit flac files I guess?

Regards Klaas

[Olav Sunde]

Hello Klaas

you are correct, I'd like to keep pcm streaming as it saves a couple of pcm-flac conversions in the chain. Today I went back to brutefirwrapper v10 to check behaviour. In all cases the config file for brutefir is changed to match the file bit depth, i.e the --copy-sample-format command for pcm is honored. I am certain it is possible to keep your file name search and the pcm feature, but I am not at all capable of progamming so it will have to wait... In the mean time I'll make sure all my 16bit files are 44.1 kHz only. Then I'll use the workaround described earlier

Olav

[Klaas Reineke]

Hi Olav

okay I have a look into the --copy-sample-format option, maybe I can get it to work, or provide a workaround :)

Regards Klaas

[Klaas Reineke]

Hi Olav

please try this patch. It should work as expected.

Regards Klaas

[Olav Sunde]

brutefirwrapper complains at line 169
    if filter_for_samplerate:
                            ^
IndentationError: unindent does not match any outer indentation level

Olav

[Mervin Beng]

The indentation on line and some lines close by is using spaces. On the rest of the Python file it's in <tab>. If you replace the leading spaces with tabs, and make sure the tabs align to the logical blocks, the wrapper should run with syntax error.

Mervin

--
You received this message because you are subscribed to the Google Groups "BrutefirDRC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brutefirdrc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Olav Sunde]

Thanks Mervin

that was indeed the issue. I tidied up and brutefirwrapper runs as expected.

I may also be on the track of an old bug with pcm playback: white noise mixed with the sound at track transitions on 16bit files. Here is part of the log output from brutefirwrapper when switching tracks:

Starting to write tail-mixed output (0.462s)
Finished writing tail-mixed output. Proper output follows (2.339s)
Readahead complete (2.341s)

If the tail is 24bit, but the actual file stream is 16bit the 24bit mixed in would only produce noise?

Olav

[Mervin Beng]

Hi Olav,

On Mon, 2013-05-27 at 00:15 -0700, Olav Sunde wrote:

If the tail is 24bit, but the actual file stream is 16bit the 24bit mixed in would only produce noise?

Yes, I would expect it to happen, but it's not guaranteed. The samples are 24-bit, and definitely produce noise. If the rest of the audio happens to be word aligned, it will be music, otherwise it will continue to be noise.

Mervin

[Mervin Beng]

Great work.

That's a tricky one. We certainly cannot mix 24 and 16-bit audio directly, so my suggestion is to throw away the tail in such circumstances, which for me tend to be rare (switching albums).

My impression is that the tail mix is relevant only when there's gapless playback, which most often is in classical music when movements are continuous (e.g., a set of variations) or very long and the tracks are there for convenient access. In this case the two tracks will always be of the same wordlength. For two consecutive unrelated tracks the tail mix is less important, IMO, and I would discard the tail (skip the tail mix).

Mervin

On Mon, 2013-05-27 at 00:15 -0700, Olav Sunde wrote:

[Olav Sunde]

Toby actually commented on the issue inside brutefirwrapper where logic for handling tail is coded:

# The tail was recorded from the output of brutefir running at a different
# sample rate. Ideally we should resample it to the new sample rate,
# however that is difficult right now. brutefir's output is 24 bit,
# and sox - our resampling toolkit - does not support 24 bit.
# The best we can do for now is drop the tail. This means you do not
# get gapless playback between changes in sample rate.

I believe sox can handle 24 bit today so maybe this part of the code can be revised.
I've mentioned this to Klaas in a PM. Hopefully he can do something here.

Olav

[Klaas Reineke]

I checked on Toby's code. He discards the tail as Mervin suggests if it has the wrong bitspersample or format:

        tail_format_name = info_file.readline().strip()

        if tail_format_name!=format.brutefir_name:

            sys.stderr.write('ignoring tail; it has the wrong sample format (%s not %s)\n'%(tail_format_name,format.brutefir_name))

            return None

Maybe there is a bug in the code, so that the different format is not discarded but this source at least tries to discard if the format is different. So my guess is, that the problem is somewhere else. Or is my understanding of this issue wrong?

I will fix the other issue and test it and make a release hopefully later this week.

Klaas

[Olav Sunde]

Yes, tail is discarded if you skip tracks within an album or change to a different album (or bit depth). It is when playing through several tracks of a 16 bit album using pcm you'll hear the problem. It sounds like a part of the sound is noise while I can also hear the complete track content. The length of this, 1 - 3 seconds, indicate that the tail is involved. If the complete tail logic could be disabled for testing the problem would disappear if my thinking is correct. If not, the problem would still be there and we'd have to look elsewhere.

This does not happen when flac is selected as format, only pcm. Is the tail different between flac and pcm? Is sox behaving diffrently? Where in the chain does merging of tail and start-of-track take place? Is the track back to 16 bit while tail is still at 24? Many questions, I know...

Gapless playback is very important so we need to keep it.

Olav

[Mervin Beng]

I'm guessing again, but....

There's a check for sample rate a few lines above:

        tail_samplerate = int(info_file.readline())
        if tail_samplerate==samplerate:
            body = open(tail_filename,'rb')
        else:
            .....

The subsequent check is for the format name for the tail, and format name of the info file for the tail, I think. If so, it will always be true.

I don't think that there is no checking for different sample rate in the tail and in the current track. It will need an extra argument (bitspersample) in read_tail(), and the test will be:

      if (tail_samplerate==samplerate) & (tail_bitspersample == bitspersampe):

To get tail_bitspersample, you need to read tail_format_name and extract the numeric portion, so the readline() has to move up.

Cheers,
Mervin

[Klaas Reineke]

I did guess the same and was about to change the file as you said but I found this:

        info_file = open(info_filename+'.new','w')

        info_file.write(str(samplerate)+'\n')

        info_file.write(format.brutefir_name+'\n')

        info_file.close()

so the Info file contains the format name from the format_xxle classes, i.e.

    brutefir_name = 'S16_LE'

or

    brutefir_name = 'S24_LE'

the first readline in the method read_tail gets the samplerate and compares it.

the second readline with strip gets the brutefire_name as stripped as string for string comparision from the info file that is written whenever a tail is written to disc. So the comparision should be okay, at least if the samplerate comparision is fine as well.

We could refactor this a to be more readable:

        info_file = open(info_filename,'rb')

        tail_samplerate = int(info_file.readline())

        if tail_samplerate!=samplerate:

            # The tail was recorded from the output of brutefir running at a different

            # sample rate. Ideally we should resample it to the new sample rate,

            # however that is difficult right now. brutefir's output is 24 bit,

            # and sox - our resampling toolkit - does not support 24 bit.

            # The best we can do for now is drop the tail. This means you do not

            # get gapless playback between changes in sample rate.

            sys.stderr.write('ignoring tail; it has the wrong sample rate (%d not %d)\n'%(tail_samplerate,samplerate))

            return None

        

tail_format_name = info_file.readline().strip()

        if tail_format_name!=format.brutefir_name:

            sys.stderr.write('ignoring tail; it has the wrong sample format (%s not %s)\n'%(tail_format_name,format.brutefir_name))

            return None

        

        body = open(tail_filename,'rb')

        body = body.read()

But this does not change what the script does. It should work as expected. Maybe I missed something?!

Regards Klaas

[Mervin Beng]

Hi Klaas,

The comments above this section seem to tell me that info_file is for the tail:

    #
    # At this point brutefir has completed its output, and we have in memory
    # several seconds of readahead followed by the tail. First we dump the tail
    # to disk, so it is ready for the next invocation of this process.
    tail_bytes = int(samplerate*TAIL_SECONDS)*CHANNELS*format.width
    if readahead.total_bytes>=tail_bytes:

        info_file = open(info_filename+'.new','w')
        info_file.write(str(samplerate)+'\n')
        info_file.write(format.brutefir_name+'\n')
        info_file.close()

If I understand this correctly, the format info is for the tail file, not the current stream which will be "tail merged".

Since the wrapper reads the tail of the previous stream, the read_tail() occurs before the section above:

    tail_filename = os.path.join(TMPDIR,'tail-%s.pcm'%(client_id,))
    info_filename = os.path.join(TMPDIR,'tail-%s.txt'%(client_id,))
    saved_tail = read_tail(tail_filename,info_filename,client_id,format,samplerate)

In calling read_tail(), info_filename is info for the tail file, while samplerate is for the current stream. I think the call may need to be be changed to:

    tail_filename = os.path.join(TMPDIR,'tail-%s.pcm'%(client_id,))
    info_filename = os.path.join(TMPDIR,'tail-%s.txt'%(client_id,))
    saved_tail = read_tail(tail_filename,info_filename,client_id,format,samplerate, format.bitspersample)

and read_tail checks that the tail samplerate *and( bitspersample are the same as for the current stream.

Cheers,
Mervin

[Klaas Reineke]

Hi Mervin,

no I understand what you mean :) I am 99% sure that there is no problem:

        tail_format_name = info_file.readline().strip()

        if tail_format_name!=format.brutefir_name:

            sys.stderr.write('ignoring tail; it has the wrong sample format (%s not %s)\n'%(tail_format_name,format.brutefir_name))

            return None

this reads the format information from the tail (info_file.readline()) and compares it to the format of this stream.

Klaas

[Olav Sunde]

Hi Klaas and Mervin,

...and still there is something wrong. When I edit the original brutefir config (*44100.txt) to S16_LE the tail is written as 16 bit and the info_file says S16_LE.

When the original brutefir file is set to (the normal) S24_LE brutefirwrapper does the patching from 24_LE to 16_LE when I play 16 bit material, BUT the tail is written as 24 bit and the info_file says S24_LE.
The error seems to be that brutefirwrapper does not relate to the patched file for sample_format, but picks it from the original. The tail and info_file is 24 while the track is 16 which produce the noise when mixed.

Olav

[Klaas Reineke]

I guess you nailed the issue. I check the Code and try to provide afix

(von unterwegs gesendet)

--

[Klaas Reineke]

Hi Olav

I tried to fix it on my travel to work. I could not test it, for some reason I did not get the commandline right, know I have to work a bit. I have to test it later at home probably not today. For this reason I upload the heavyly patched wrapper for you to try. I moved quite some code around so the chance to have a working version is quite slim, still it would be nice to get a feedback. Maybe we are lucky :)

Regards Klaas

[Olav Sunde]

Thanks Klaas, I'll see what I can do during the weekend. The next days are scheduled for "other" activities so I can not promise anything though.

Also thanks to Mervin. When you discussed the tail and info file a few posts up I knew where to look.

I hope this does not break anything else, like flac playback for example

Olav

[Olav Sunde]

Not so lucky I'm afraid. Brutefirwrapper runs with no complaints from python, but I get this error in brutefir.log when playing
:
======== brutefirwrapper rev 11 starting at 1370008663 2013-05-31 13:57:43
Found filter "/etc/squeezeboxserver/BrutefirDrc/filters/A15.05.13_44100.txt" for sample_rate 44100

Patching brutefir input format from S24_LE to S16_LE

Patching brutefir output format from S24_LE to S16_LE

Traceback (most recent call last):
  File "/c/.squeezeboxserver/Plugins/BrutefirDrc/Bin/brutefirwrapper", line 466, in read_tail
    age = time.time()-os.stat(tail_filename).st_mtime
OSError: [Errno 2] No such file or directory: '/tmp/.BrutefirDrc-0/tail-ec_a8_6b_f3_71_f7.pcm'
Output follows (0.062s)

Olav

[Olav Sunde]

That error is not very helpful. It is the same log output I see when I skip a track with a working version of brutefirwrapper..

Olav

[Klaas Reineke]

I try to fix it on my way bayck home. Thank you for trying

(von unterwegs gesendet)

--

[Klaas Reineke]

It plays music, I cannot say if it fixes the noise bug with pcm. I hope so. I did not upload the release on sourceforge, because I only want to push it there if the fix works.

It has version 12 of the wrapper.

Regards Klaas

[Olav Sunde]

I've played for more than an hour with no problems showing up. I think you have fixed it! Have you checked other formats too? I have only played flac as pcm. This bug has bothered me for a long time. It is nice to play my CD rips without noise between tracks.

Olav

[Olav Sunde]

That was with multi filters. I'll enable a single 48kHz filter and check that resampling works as expected. That will be later in the weekend.

Olav

[Klaas Reineke]

Hi Olav

this sounds great. I will test a bit and push a release if do not find any more problems. I always use flac for playback and for flac it is no difference, because the copy-sample-format option is not used with flac. All output is converted to 24_LE so that the bug could not happen here or for MP3 :)

Why are you using PCM can you hear it or is it a precaution against a few more conversions to flac and back?

Klaas

[Olav Sunde]

Single filter config with pcm also works correctly as far as I've tested. You seem to be the master of Python too now!

======== brutefirwrapper rev 12 starting at 1370077387 2013-06-01 09:03:07
Resampling brutefir filter from 48000 to 44100

Patching brutefir input format from S24_LE to S16_LE
Patching brutefir output format from S24_LE to S16_LE

tail is 14s old
----------------------------------------------------------------
the tail and info_file are both 16_LE

Also, the code that deals with levels of the various formats when resampled is working:
coeff "drc_l" {
filename:"/tmp/.BrutefirDrc-0/null-65k48-l.pcm-44100";
format: "FLOAT_LE"; # file format
attenuation: -0.736052958; # attenuation in dB

All very good!

I use PCM streaming because I can hear an improvement over flac when using a normal SB Touch. Today I use squeezelite on my Touch and the difference between pcm and flac is smaller. My main server is a ReadyNas with dual 1.8GHz Atom. Streaming pcm is economical as there is only one flac-pcm conversion in the chain. Otherwise it would be flac-pcm-flac on the server and again flac-pcm on the player.
I have not done much comparisons of this lately, but I prefer to stream pcm so I'll stick to that.

Best regards
Olav

[Klaas Reineke]

I don't get PCM streaming to work. I have some SACD and some 96kHz LP rips, that I cannot play via PCM. 44.1 plays fine, but all else does not really work for me. Do you have stuff with higher sampling rates? Maybe it is a problem with my synchronization of a SB3 with a USB-soundcard that supports higher sampling rates. With flac I don't have any trouble.

I thought about switching back to PCM but I guess it is not worth the hassle to fix this playback issues.

Regards Klaas

2013/6/1 Olav Sunde <ogs...@gmail.com>

--

[Olav Sunde]

I have material in all resolutions up to 24/192. All play fine with pcm streaming. 192kHz taxes my ReadyNas a bit, but I rarely see more than 20% per cpu for the brutefir processes. I think you may be on to something with syncing, but I've never tried this so I would not know. Are you using squeezelite with the USB card? Any relevant log output from brutefirwrapper?

Olav