help with decode_from_array

[Christopher Pross]

Hi!

i coded here a solution for load a sound from a encrypted zip file.

My code is working, but very unstable and the errors are random. Maybe
someone can help me?

The code works, when I use one thread, but that isn't what I want.

I have to use threads, because that's are many sounds to load.


I send you my python file, to encrypt a zip archive, create a folder
"sounds" and put some wav files in it.

Then, zip that and run the python file, done.

This works perfectly, now you go in the interactive, in the same folder
where the encrypted archive and the .py-file are stored.

Now here are different sessions from my:


<sessions>

without try-block:

*****

Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sound
>>> s = sound.sound("sounds.dat","abc")
sounds/ambiance/Cave.wav
Unhandled exception in thread started by <bound method
sound.get_files_from_archive of <sound.sound object at 0x02F1CF70
>>
Traceback (most recent call last):
File "sound.py", line 122, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\site-packages\libaudioverse\__init__.py", line
687, in decode_from_array
_lav.buffer_decode_from_array(self, array, len(array))
File "C:\Python27\lib\site-packages\libaudioverse\_lav.py", line 488,
in buffer_decode_from_array
raise make_error_from_code(err)
libaudioverse.MemoryError: Memory allocation error: bad allocation
(C:\projects\libaudioverse\src\libaudioverse\buffer.c
pp:161)
****


or

***

Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sound
>>> s = sound.sound("sounds.dat","abc")
sounds/ambiance/Cave.wav
Unhandled exception in thread started by <bound method
sound.get_files_from_archive of <sound.sound object at 0x02FACF70
>>
Traceback (most recent call last):
File "sound.py", line 122, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\site-packages\libaudioverse\__init__.py", line
687, in decode_from_array
_lav.buffer_decode_from_array(self, array, len(array))
File "C:\Python27\lib\site-packages\libaudioverse\_lav.py", line 488,
in buffer_decode_from_array
raise make_error_from_code(err)
libaudioverse.MemoryError: Could not call
speex_resampler_cpp::staticResampler.
(C:\projects\libaudioverse\src\libaudiov
erse\kernels\resamplers.cpp:19)

****


with try-block:


****

2) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sound
>>> s = sound.sound("sounds.dat","abc")
fehler sounds/ambiance/Cave.wav
Traceback (most recent call last):
File "sound.py", line 123, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\site-packages\libaudioverse\__init__.py", line
687, in decode_from_array
_lav.buffer_decode_from_array(self, array, len(array))
File "C:\Python27\lib\site-packages\libaudioverse\_lav.py", line 488,
in buffer_decode_from_array
raise make_error_from_code(err)
MemoryError: Could not call speex_resampler_cpp::staticResampler.
(C:\projects\libaudioverse\src\libaudioverse\kernels\resamplers.cpp:19)

D:\Programmierung\python\The Game of Dungeons\main\sound>python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sound
>>> s = sound.sound("sounds.dat","abc")
fehler sounds/ambiance/Cave.wav
Traceback (most recent call last):
File "sound.py", line 123, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\site-packages\libaudioverse\__init__.py", line
687, in decode_from_array
_lav.buffer_decode_from_array(self, array, len(array))
File "C:\Python27\lib\site-packages\libaudioverse\_lav.py", line 488,
in buffer_decode_from_array
raise make_error_from_code(err)
MemoryError: Memory allocation error: bad allocation
(C:\projects\libaudioverse\src\libaudioverse\buffer.cpp:161)
fehler sounds/ambiance/jungle.wav
Traceback (most recent call last):
File "sound.py", line 123, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\site-packages\libaudioverse\__init__.py", line
687, in decode_from_array
_lav.buffer_decode_from_array(self, array, len(array))
File "C:\Python27\lib\site-packages\libaudioverse\_lav.py", line 488,
in buffer_decode_from_array
raise make_error_from_code(err)
MemoryError: Could not call speex_resampler_cpp::staticResampler.
(C:\projects\libaudioverse\src\libaudioverse\kernels\resamplers.cpp:19)
fehler sounds/ambiance/light_forest.wav
Traceback (most recent call last):
File "sound.py", line 123, in get_files_from_archive
self.sounddict[i].decode_from_array(zipfile.read(i))
File "C:\Python27\lib\zipfile.py", line 935, in read
return self.open(name, "r", pwd).read()
File "C:\Python27\lib\zipfile.py", line 630, in read
data = self.read1(n)
File "C:\Python27\lib\zipfile.py", line 684, in read1
max(n - len_readbuffer, self.MIN_READ_SIZE)
MemoryError

*****


</sissions>


Well, the code is not the best readable code, but I hope you can help me...

I don't know enugh about libaudioverse, for fixing this allone.

My code is not clean, because I tried many thinks to get this code working.

When you have questions for the code, ask me. My docstrings are in
german, so hopefully you can understand the code.



Best regards,

Christopher Proß

[Austin Hicks]

The error you are receiving means that Libaudioverse is out of memory.
The decoding process takes a great deal of ram, so this is not
surprising to see in multithreaded configurations.

Leave the server's threads alone. This already does what you want.
Turning it up and down is just causing added inefficiency. All the
server's thread limit determines is how many threads will be used for
rendering audio, it doesn't affect your calls.


before we go on, I am having trouble following the logic of this
script. If I get anything wrong, this is the reason.

You appear to spawn a thread for each sound, and never wait for any of
them to finish before spawning the next one. Since Libaudioverse needs
2x the size of the sound file as a lower bound, this means Libaudioverse
is using a ton of memory. In addition, you are reading the entire file
into Python; this takes yet more memory. I suggest writing a function
to decode a single sound, finding a thread pool library, then using the
thread pool library to efficiently call your sound extracting function
on each item in the list.

Keep in mind that you are loading the file into Python strings. These
strings will stay around for an indefinite period of time and will be
quite, quite large.

The quickest fix to this script is probably to split the list of files
into a number of equally sized chunks depending on the number of cores
in the system; the multiprocessing module provides a function
multiprocessing.cpu_count() that will tell you how many there are.

Your logic with the locks appears to be holding it for the duration of
the decoding. This means that you are not actually getting much benefit
from using threads. I suspect this script isn't much faster with them
than without.

Your waiting logic is also going to completely eat a core: if you insist
on trying to do this without using a thread pool library, I suggest
looking at threading.Semaphore.

Once you do get this not eating all the ram and fix the lock so it
actually provides a benefit, I suspect you are going to also find a
bunch of thread safety bugs. Please, please find and use a library, or
at least the Queue module.

You don't want to start and abandon threads after each file: this has a
huge amount of overhead. As an idea of how much, multithreaded
Libaudioverse runs orders of magnitude faster, but multithreaded
Libaudioverse with an internal bug that accidentally stops them after
every block of audio was worse than just not bothering.

Even if you do get all this working, keep in mind that the bottleneck
here is likely to be the hard drive. What I mean is that threads will
only really benefit you on a system with an SSD. Zip files do not
compress audio by very much. Count on needing at least 20mb/s hard disk
I/O bandwidth per thread for decoding wave files. Given that the OS and
other parts of your game are also probably taking some, this will
probably not be available on all systems anyway. Libaudioverse decodes
even large wave files practically instantly as-is. The decoding
overhead comes in for other formats such as ogg; these formats are much,
much smaller, so the CPU has a chance to be a bottleneck.

Code like this script is why I encourage people to not use threads until
they have absolutely no other choice. If you learn nothing else from
this, please walk away with the understanding that threads are one of
the hardest things to get right as a programmer, not one of the
easiest. Your script is going to load sounds exactly once. Going
through the trouble to have threads so that this can take maybe a few
seconds less is not worth it.

In so far as I can tell and pending further information, this doesn't
look like a Libaudioverse bug. If you still think it is, I'm going to
need your test sounds and information on the computer you're testing
this on. There is a slight possibility that something is wrong, but I
really think it's the script.

[Christopher Pross]

Hi,

thank you for your answer, I found a solution for my problem!


best regards

Christopher