Sound versus Player component

[p....@gmail.com]

Hello,

I have a question concerning anyone's experience with using the Sound component versus Player component (as found in MIT AI2).     

Specifically, using Sound to play short .wav files repeatedly on Android seems to eventually have timing issues where it plays, then won't play, etc. such as when you setup a button as the kickoff for it.  

And this is even after minizing the minimum timeout between playing from 500 down to 50, then 20 and even 5 milliseconds.   

I therefore took a chance trying the Player component in place of the Sound component, and it seems more on target, without the timing out issue (where as I described it works at first, then intermittently stops working).

If you were to develop a piano app, for example, what would be the best choice?

Thank you.   

[p...@gmail.com]

Addendum to my previous post:    why might the use of Sound multimedia component (in MIT AI2 written app) when repeatedly playing (and stopping)  sound wav files, cause an app to intermittenly not play the sounds, when the buttons are pushed (and the program is still running ok based on other on-screen events occuring) ?   Something with Android ? 

[pgb...@gmail.com]

As there are yet no replies to this, I continued to look an issue I have found. 

I tried a test program watered down to just buttons which play tones (wav files) with the Sound element.   When 53 tones are available to play, and even when the program is altered down to 28, after a few minutes of pressing buttons and the tones playing, eventually one presses buttons and nothing plays intermittently or every other press only.   Finally I adjusted it own to 16 tones only available, and then the issue stopped happening no matter how long I tested pressing the buttons and playing the tones.   

It seems to me that even with use of the Sound element and .Play and .Stop in conjuction with the buttons being pushed, something occurs with the program or use of memory or similar, begins to be a problem with more than 16 notes.    

I would most appreciate someone on the MIT AI2 team looking at this, and getting back to me here?

Thank you.  

[Hal Abelson]

We can look into this, but it will may take us a while.   Could you attach the source (aia) of your test program with instructions about how

to elicit the problem?   Also, I'm guessing this happens with Sound and not with Play -- is that correct?

It's possible there is a memory leak in the buffer that Sound uses.   But we'd need to check.

Thanks for reporting this.

== Hal

[p....@gmail.com]

Hal,

I have a Sound component test program .aia file ready for you. I would prefer to send it to you directly, than to post it here.  Please give me an email I can send it to, and I will send instructions for re-creating the problem. 

Regarding Sound versus Player, it turned out that Sound was much quicker in response, while Player didn't sound good for creating musical notes fast - kind of crackly and too slow to respond.  

Thank you.  

[pgb...@gmail.com]

Hello?   I would like to get you the .aia and .apk for your testing, but need an email address to send it to.  I assume this is a public forum, and don't wish to attach this program here.  

Thank you.  

[pgb...@gmail.com]

PS:  My apologies, please ignore the last post, as I just realized Hal send me an email to my other address which I had not checked.  Hal, the program is coming your way.   :)  

[Hal Abelson]

This is now filed as an issue on our internal list.  #765

But it might be complicated to fix, and might require significant redesign of Sound component.   The issue here is that each App Inventor sound is implemented to use an individual Android SoundPool, and Android has a hard limit on the number of Sound Pools that can be open at once in the entire device (I think 52 per device).   So it won't work to have a lot of Sound Components in any single app.  If you want to have a lot of sounds, you'll have to implement your app so that there are a small number of Sound components -- maybe only one Sound component -- and switch the Source property to play different notes.

That might be too slow for any applications, but it's the best I can think of at the moment.   An other possibility would be for the App Inventor Sound component to give the app developer finer control over how sound pool resources are assigned and released, but it's not clear how that would work, and it would be rather complex for our target user population.  It might be better just  to limit the number of Sounds components that could be in any app.

[pgb...@gmail.com]

Hi Hal, thanks for adding this issue to the list.  I think as AI2 becomes more and more popular for us to develop andoid apps, this will definitely need looking at.  AI2 is such a wonderful way to program, so hitting issues/limitations like this can be dissappointing.  

Meanwhile I shall take your advise, when I get back, to attempt implementation with a smaller number of sound components, and changing them via source property.  I'll report on results.   Probably be in 2 weeks.  

Thank you. 

[pgb...@gmail.com]

PS:  I think this is the way one changes a sound source ?   Through path to local file, or web ?  

http://appinventor.mit.edu/explore/ai2/concepts/images-and-sounds.html

file:///sdcard/Music/Blondie/The%20Best%20of%20Blondie/Heart%20of%20Glass.mp3

http://www.whereever.com/whatever/filename.wav  

Looking or a AI2 visual example?  

[Hal Abelson]

Use the path to the local file.  If there's enough room, you can include those in the assets of the app and just use the file name as the source.   If not, you can store them on sdcard and use file:///sdcard.  Getting them from the web each time will bee too slow.

[pgb...@gmail.com]

So the code would be something like this?

[set (SoundName)  (Source) to ] [{text} ("path.....")]          /* where path = file:///where/does/AI2/store/sound/files  ????????????  (I don't know)  */  

               or

[set (SoundName)  (Source) to ] [get {global} (VariableName) ]       /* which contains the path */    

PS:  I don't think there's enough beefed up tutorial information about doing it this way.   It needs a couple valid examples.   ;) 

Thank you.  

[Hal Abelson]

See

http://ai2.appinventor.mit.edu/reference/other/media.html

for info on how to use pathnames fro media files

[pgb...@gmail.com]

As promised, the altered program and aia has been sent to you.   

Thank you. 

[Hal Abelson]

Glad to see that this is working.   I played only a tiny bit and it seemed to me that the sound sometimes did not play until I hit the key twice, but I'm not sure.

You can run the app and look in the Android system log to see if there are still any of those memory or "unable to allocate ID errors".

As far as the 703 error, that means the sound is in a format that the Android player cannot play.   If you look at the lost of Android supported media formats, you'll see that it cannot play
every WAV file -- there are restrictions on sampling rate.

When I ran into this problem, I ha to convert the WAV file I wanted to play, to an MP3.

Good luck

= Hal

[pgb...@gmail.com]

Hal,

The specifics of what I found with this latest test program, is this:

(1) the sound buttons which have initial sound components and wav file setting in the Designer section play no problem.

(2) the additional sounds must be touched a second time each to overcome a 703 error, but the interesting thing is *after that* they all play just fine.   

     In other words, after you play each of the sets of sounds, the error stops occuring, and they all play fine.  

Regarding the possibility these wav files are unsupported, the Andoid supported media formats says it supports PCM Wav, and these are indeed PCM Wav files.   

Here is an example output from mplayer tool on Linux:

==========================================================================

Opening audio decoder: [pcm] Uncompressed PCM audio decoder

AUDIO: 22050 Hz, 1 ch, u8, 176.4 kbit/100.00% (ratio: 22050->22050)

Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)

==========================================================================

AO: [pulse] 22050Hz 1ch u8 (1 bytes per sample)

My question is then, the Android support site says it supports 8, 16 and 44 KHz but says nothing about 22 KHz, so why would this not be supported?   

Again, after you hit the buttons for the sounds at least once, the 703 errors go away, and you can play them all without error after that.

I think the above represents something of interest?

Please advise,

Thank you.  

PS:  I can of course convert the 22KHz to 16 for example, or even to MP3, but I do find it interesting what I said above about the 703 error going away, and playing ok.  

[pgb...@gmail.com]

Hal,

Addendum to my previous reply.  The Android supported media site says MP3 of type  Mono/Stereo 8-320Kbps constant (CBR) or variable bit-rate (VBR) are supported. 

I converted my wav files with Linux lame,  to mp3 VBR as follows: 

`lame -v input.wav output.mp3`

LAME 3.99.5 64bits (http://lame.sf.net)

polyphase lowpass filter disabled

Encoding C12.wav to C12.mp3

Encoding as 22.05 kHz single-ch MPEG-2 Layer III VBR(q=2.44706)

And mplayer plays it nicely:

==========================================================================

Requested audio codec family [mpg123] (afm=mpg123) not available.

Enable it at compilation.

Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders

libavcodec version 54.35.0 (external)

AUDIO: 22050 Hz, 1 ch, floatle, 54.3 kbit/7.69% (ratio: 6784->88200)

Selected audio codec: [ffmp3float] afm: ffmpeg (FFmpeg MPEG layer-3 audio)

==========================================================================

AO: [pulse] 22050Hz 1ch floatle (4 bytes per sample)

But unfortunately, the program still results in error 703 the first time one pushes the buttons (except for the pre-defined mp3 sounds for button compinents seen in Designer section.)  

But after you hit the buttons at least once, for all the sounds, thereafter it will play them without the 703 error showing up.  

So, same result for mp3.  Sending you the program.  

Help?

[Hal Abelson]

I took a look at running your app with the android log.   Would you be willing to post the aia and apk to this forum so other people can expriment?   I can file the issue but I can't work on it right away.

From my look a the android system log, it looks like there are two different problems:

One is that a couple of the files can't be played.  As I said before, not all wav and mp3 can be played.  You'll have to experiment more to check the problematic ones.

The mode important one if the with your program running as it is, some of the sounds are being played before the sound sample is ready.  The android log is full of "sample not ready" errors.

That's why the sound doesn't play the first time (sample not ready) but plays the second time (sample ready)

Take a look at 

http://stackoverflow.com/questions/5202510/soundpool-sample-not-ready

To see what the issues is.  When I get time, I'll add the requisite delay to App Inventor's sound component.  I hope that fixes the problem.  (But i'm pretty busy so I won't get to that soon).

In the meantime, you can experiment with adding a small delay between the sound load and the actual play.

[pgb...@gmail.com]

Hal, sent you email on the first part. 

As for the rest, thanks for looking at the log info (I had installed aLogCat tool on my phone but could not get it to show me anything useful, so I have to keep looking into this).    

Since you are saying from your log entries that "a couple of the files can't be played", is strange since they are all created the same, and checked them all before importing to the program, and I checked the Android sound support chart, and made the mp3 files in accordance with the little information they give.   Seems they need more specifics for developers to avoid hazy areas on sound formats.   But more importantly, since your logs are showing some kind of delay is needed for the sounds to be ready (following the "set sound ..." structure - which explains why the primary sounds work being already set in Designer) I think I will go back to my original method of setting up all the sounds with components in Designer and my new mp3 files to see if this does better. I don't think that was playing more then a few sounds at a time anyway (should not reach the 10 limit). If that does not work, I'll check into adding a delay (but the worry is the delay makes the program too slow to respond to button pushes).    

I'll post an update when I get to this asap.   I'll also look at the link you provided on soundpool sample not ready.  

Thank you again.  

[Scott Ferguson]

Specifically, using Sound to play short .wav files repeatedly on Android seems to eventually have timing issues where it plays, then won't play, etc. such as when you setup a button as the kickoff for it.  

You must always set all sources that you will use for the Sound component first before attempting to play a sound.

This is normally done in the Screen Initialize block. Setting the source right before playing a sound does not always work as you have seen.

If you were to develop a piano app, for example, what would be the best choice?

Even though a piano note will normally be less than the 5 seconds maximum recommended for the Sound component, I always use the Player component as you don't need to initialize the Sources ahead of time as with the Sound component, it plays any short or long sound, and it has a volume control block. By the way, did you know you can play midi files for a piano app as well? The file sizes are way smaller!

---

Scott

[pgb...@gmail.com]

Regarding use of Sound component, and either setting them up in the AI2 Designer section with a default sound for each Sound component, versus setting just a few Sound components in Designer with the sound source set in AI2 Blocks code section, I have done it both ways, and both ways have resulted in varying issues, which I have discussed with Hal.    Not all application (such as a piano) will work with setting up all sources first in the Block section, since one cannot know which will be played, when there are a small number of Buttons configured in Designer, which need their sound source altered as the program is used.     

Regarding use of the Player component instead of the Sound component, I have tried it both ways, and found Sound to be much more responsive, and Player to be less responsive and also given me somewhat crackled/garbled sound when attempting to play sounds quickly or short sounds.    I therefore went back to Sound rather than Player.   As I understand it Player is really for playing pre-recorded songs, etc.   

So, thank you - but I have tried those methods.     

[pgb...@gmail.com]

Addendum to my previous reply to Scott....

I continued to think about what you said, regarding setting up the sounds before playing.   I realized in my own test program, there was a way to set them up (as in setting up the playing field per say) before playing notes on buttons.  I removed setting the sound sources into their own procedure, away from the button play procedures, and this seems to have done the trick.   No timing mechanism, just moving the sound button sources into their own procedure so they are not coupled with the .play functions.   

Hal, sending you the latest aia and apk on this.

Thank you.

[Scott Ferguson]

Great! -- I take it that you are doing this for the Sound component. It should not be necessary for the Player component.

Also, you might try refreshing your sound drivers. That may be the cause of the 'clicking'.

---

Scott

[pgb...@gmail.com]

Yes, for the Sound component.  Not using the Player component (though I tried it, was not happy with its result).   

[Hal Abelson]

Thanks for playing with this, and I'm glad you found a solution.   I'll add that check to the Sound component anyway.

[pgb...@gmail.com]

Thank you Hal, for all your patience and looking at this with me.    Anything which enhances Sound component will be worth it.   :)

Thanks Scott for pointing out the setting up the Sound component "field" first, which overcomes the delay issue. 

[pgb...@gmail.com]

PS:  Using mp3 instead of wav makes the apk package smaller too, which helps on Android, and probably with using AI2 online, through I have also been using offline version on Linux.  

[pgb...@gmail.com]

Hal and Scott,

One issue I have run into changing the logic where the screen of buttons (I call the 'playing field') is coupled with the sound sources, is when one wants extend a tone rather than stop it, becomes problematic.   

With the sound sources coupled to the .Play commands, I was able to (beforehand) simply bypass the .Stop to allow the tone to keep playing to finish.    But now with the sound sources coupled to the playing field of buttons, when the sound source changes, it immediately stops the sound one played via a button even if the .Stop was bypassed (like pulling the 'rug' out from under it).     

While I am hunting for a logical solution, this may be a situation in which coupling the sound sources with the .Play commands is useful, rather than with the 'button playing field'.

Hal, this aspect I am discussing here, is not in the test program.  But you can imagine where the button touchup usually .Stops the sound, unless some other variable causes it to bypass the .Stop to allow it to play to finish.  Except that now with the sound source part moved over to the 'button playing field' setup, changing the sound source stops the sound inadvertently.    

It might be useful for the Sound component to be aware when a particular sound is playing, to not stomp on it (stops it), just because the sound source is changed.     I am pretty certain this is what is occuring.      

Thank you.   

[pgb...@gmail.com]

On a different note, I've now found suddenly that using Chrome (which was working perfectly), my projects in both the offline 7 June version and many of those in the online 15 June version, have stopped allowing me to add Media and other components to the Designer desk.   I can drag such a component over, but then it does not stick and get added in place.     I tried Firefox, and that then worked ok.   

Wondering what changed ?   Or what could be causing this?   

[Hal Abelson]

I'm going to lock this thread, because what you are describing is almost certainly a different issue.

It's also strange, because there have not been any changes of the sort that could cause a problem like this.

Try some debugging on your own, and see if you can produce  very simple app that illustrates the problem.

The file a new issue, including post the aia file.  Please post the file
publicly, so that several of the forum modertor can have a look.

Thanks.

== Hal