Sound in CodeWorld

[pe...@vdbrand.nl]

Hi,

The past week I worked on a proof of concept for adding sound to CodeWorld. You can find a (very simple) demo here: http://vdbrand.nl/cw-sound/. You can find the code at https://bitbucket.org/pvdbrand/cw-sound, though the only interesting file is src/Sound.hs. It is by no means completed, but it gives a reasonable idea of my approach.

The API is based on the ideas of Paul Hudak, which are implemented in Euterpea and described in the Haskell School of Music book. Interestingly, the same ideas apply to animation, and are very similar to the API for composing animations suggested by Fernando Alegre (https://github.com/google/codeworld/issues/437).

Basically, Notes (and Samples) can be composed sequentially or in parallel into Sounds. There are a couple of modifiers for instruments, tempo, volume, etc. This is what a student will be working with mostly. There is a large set of instruments available already, and I also used a nice set of old-school game sound samples, for explosions, guns, etc.

The API I have in mind for CodeWorld is to modify the step and event functions so they have types step :: (state, Number, Sound) -> (state, Sound) and event :: (state, Event, Sound) -> (state, Sound). They would receive the "current" Sound, which will be replaced by the returned Sound. So if I just want to play what is already being played, then I just return the given sound. If I want to add a sound effect of an explosion, then I'd return (sound :=: Explosion) for example. If the music should change completely after the game is over for example, I can just return a completely different Sound at the game over transition. The CodeWorld system will make sure that the correct "current" Sound is computed and passed into the next step function, using the takeS and dropS functions.

My design choices are mostly in line with the ideas in https://github.com/google/codeworld/issues/47. 

I chose to use the Web Audio API. Browser support is quite good nowadays, and it is really very powerful and a perfect match for what I want to achieve. I don't think there is a better alternative. The Web Audio API also makes it possible to synthesize your own sounds with sine waves and filters etc. I am in doubt whether it would be a good idea to add support for this. Pedagogically, it is interesting. Practically, it is very difficult to make nice sound effects or music with it. It also makes the API bigger.

I decided to go with samples of musical instruments. It allows the student to focus more on music composition and music theory. I just used all instruments that were present in a pre-rendered sound font. This list can be improved by removing a lot of uncommon instruments, and perhaps add missing common instruments. The current implementation can already be used to compose almost any kind of music.

I also decided to add samples for sound effects. I personally feel that the retro sound effects match the kind of games kids typically create with CodeWorld, but I haven't asked any kids if they agree...

A nice implementation aspect is that only the required samples are loaded from the server, and that CodeWorld can prefetch them. The definition of a Sound can be infinite or depend on a state or state change, and still allow everything to work. 

I'm aware of https://github.com/google/codeworld/issues/492. If I'm going to add sound to CodeWorld, now would indeed be a good time to work on that issue as well. And then there is the animation API proposal, which shares a lot of similarities in the handling of time.

I'd be happy to get any feedback about the general direction I went with this implementation, the implementation details, and how it might fit well in the bigger CodeWorld picture.

Regards,

Peter

Some notes:

• I made a small stand-alone project so I wouldn't have to worry about existing CodeWorld code. Plus it's my first project using GHCJS. Of course, my goal would be to add it to CodeWorld and to make it conform to the project structure and conventions of CodeWorld.
• For now you just use the data constructors directly, but the plan is to add functions that create and compose these things.
• The CodeWorld step / event functions are not implemented yet, this version only plays a single Sound by scheduling all notes and samples right away. The delay of 1 second is a dirty hack to make sure the samples have been loaded, but this will be changed.
• I chose the :+: and :=: operators because those are used in Euterpea, but I think they're pretty awful.
• The API is quite big. It should probably be in its own package, like CodeWorld.Sound. Autocomplete is very helpful in the editor, but it won't be that useful with 500+ entries for the sound effect samples. 

[Chris Smith]

This is great!  This is a nice kick in the butt for me to start working on #492.  While I'm still pondering the best approach for codeworld-base, I can throw something into a separate module in codeworld-api pretty quickly.  Once we're happy with the result in codeworld-api, we can migrate to codeworld-base from there?

--
You received this message because you are subscribed to the Google Groups "codeworld-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codeworld-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to codeworld-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codeworld-discuss/7de02c20-a91f-404f-8068-85452ae435c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pe...@vdbrand.nl]

That makes sense. I will work a little more on my own separate project until I'm happy with it. I can then port it to codeworld-api and make a pull request on github so you can review it.

Op donderdag 20 juli 2017 17:57:22 UTC+2 schreef Chris Smith:

To unsubscribe from this group and stop receiving emails from it, send an email to codeworld-disc...@googlegroups.com.
To post to this group, send email to codeworl...@googlegroups.com.