[Haskell-cafe] Haskell audio libraries & audio formats
Eric Kidd
I'm working on a toy program for drum synthesis. This has lead me to
explore the various sound-related libraries in Hackage. Along the way,
I've learned several things:
1. There's a lot of Haskell sound libraries, but no agreement on how
to represent buffers of audio data.
2. Some of the most useful sound libraries aren't listed in Hackage's
"Sound" section, including HCodecs, SDL-mixer and hogg.
3. The Haskell OpenAL bindings are great, but ALUT has been removed
from MacOS 10.5. Of course, a pure-Haskell version of ALUT would
be pretty easy to write, and it could be used as a fallback.
4. '[[a]]' is _not_ a good representation for multi-channel audio
data. Manipulating even 10 seconds of audio is noticeably slow.
5. HCodecs is probably the best library for reading WAVE files. It
uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'.
It also supports MIDI and SoundFont, which is quite handy.
6. YampaSynth looks really cool, and it might be possible to turn it
into a reusable MIDI synthesizer library for use with HCodecs.
This would be extremely useful: it would provide a basis for all
kinds of crazy MIDI-based programming in Haskell.
At the end of this e-mail, you can find a list of Haskell sound
libraries and their supported audio formats.
What would the ideal Haskell sound API look like? Personally, I would
love to see:
a. Something like HCodecs's Data.Audio as the basic sound format.
Data.Audio is an array-based API, and it supports converting
between a wide range of common sample formats. I don't know how
fast this is, but it already exists, and it's used by YampaSynth.
b. OpenAL for sound playback, with a portable version of ALUT. This
may require writing a pure-Haskell version of ALUT.
c. Utility functions to convert Data.Audio to OpenAL.AL.Buffer.
d. An easy-to-use API for audio processing. It might be possible to
start with something like Jack's 'mainMono' and 'mainStereo'
functions, which map a function over an audio stream.
Other nice-to-have features might include:
e. A standard MIDI format, based on either the HCodecs package or the
midi package. (HCodecs is used by YampaSynth, and the midi package
is used by alsa-midi.)
f. A modular version of YampaSynth which can convert MIDI data
structures into Data.Audio values.
It looks like Haskell could be a really sweet audio programming
environment with just a bit of integration work. What do folks think?
Are there other libraries I should look at more closely? Other features
that should be included in an ideal audio API?
Thank you for your advice and feedback!
Cheers,
Eric
Haskell audio libraries
Reading and writing sound files:
HCodecs: (Audible a) => DiffUArray Int a
hsndfile: MArray with Float and Double
HSoundFile: [[Double]]
ALUT: OpenAL.AL.Buffer
WAVE: [[Int32]]
Playing sounds:
OpenAL: Ptr UInt8, Ptr Int16, 1-N channels
SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct
Sound processing libraries:
dsp: Array a, [a]
jack: Operates via mapping functions
mainMono :: (CFloat -> IO CFloat) -> IO ()
mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()
MIDI-based:
HCodecs: Reads and writes MIDI files
midi: Reads and writes MIDI files
alsa-midi: Uses midi library
YampaSynth: Stand-alone program
Special-purpose APIs only (FFIs, etc.):
hCsound: CsoundPtr
hsc3: UGen
No public sound-buffer API:
hbeat: The relevant source files are missing!
hogg: Very low-level API for Ogg internals only
libmpd: No sound buffer API
sonic-visualizer: No sound buffer API
truelevel: Stand-alone program (uses WAVE)
wavconvert: Stand-alone program
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
John Van Enk
I'm hoping to publish my library in the next week here:
darcs get http://code.haskell.org/portaudio/
It implements the Haskell bindings to the PortAudio library. This is the
library behind Audacious. I *have not* implemented the callback model yet,
but I plan to do that.
Perhaps this will be something which is useful to you?
--
/jve
Eric Kidd
> It implements the Haskell bindings to the PortAudio library. This is the
> library behind Audacious. I *have not* implemented the callback model yet,
> but I plan to do that.
>
> Perhaps this will be something which is useful to you?
Looks like a great binding! We actually use portaudio in callback mode
at work, with reasonable success (from C++).
One question: I notice that your writeStream function represents audio
using '[[a]]'.
writeStream :: (Storable a) => PaStream a -- ^ The output stream
-> [[a]] -- ^ The samples to be played
-> Int -- ^ Number of frames
-> IO (Either String ErrorCode) -- ^ The return status
of the write
In my experiments, I've been somewhat unsatisfied with the performance
of '[[a]]' as an audio format. Would you be interested in also
supporting an array-based format for audio data?
I'm currently converting my program to use HCodec's Data.Audio
representation, which looks pretty promising. I don't know how it
performs yet, but I'll let you know.
Thank you for the pointer to your library! It will make an excellent
addition to the available Haskell sound libraries.
Cheers,
Eric
Don Stewart
> At the end of this e-mail, you can find a list of Haskell sound
> libraries and their supported audio formats.
Could you add that detail to the Haskell wiki Audio page?
-- Don
John Van Enk
I was hoping to use a packed format like ByteString eventually. Right now, I
want to get everything working nicely. As it stands, I end up marshaling a
lot of information into/out of arrays which I'd much rather keep as a block
of memory.
I'm guessing that some sort of unboxed array would be close to what I want.
If you have suggestions, I'm more than open to them. (This is my first
attempt at writing a library.)
/jve
--
/jve
Eric Kidd
> I'm guessing that some sort of unboxed array would be close to what I want.
> If you have suggestions, I'm more than open to them. (This is my first
> attempt at writing a library.)
I'm still trying to answer this question myself. :-) I'll let you know
how HCodec's AudioData format actually works in practice once I my
code is converted.
Honestly, an ideal format should allow iteration over one or more
StorableArray chunks. This would make it much easier to interact with
the outside world. Unfortunately, HCodec's DiffUArray format seems to
require a conversion step to get a StorableArray, so I'm going to have
to benchmark it.
Good luck with your first Haskell library!
Henning Thielemann
On Sun, 24 Aug 2008, Eric Kidd wrote:
> Greetings, Haskell folks!
>
> I'm working on a toy program for drum synthesis. This has lead me to
> explore the various sound-related libraries in Hackage. Along the way,
> I've learned several things:
>
> 1. There's a lot of Haskell sound libraries, but no agreement on how
> to represent buffers of audio data.
In my sound synthesis library
http://darcs.haskell.org/synthesizer/
I use mainly two representations:
1. Chunky sequences based on StorableVector. This is for efficient storage
and functions that need to process data in a non-causal way. They are not
good for fusion.
http://darcs.haskell.org/synthesizer/src/Synthesizer/Storable/Signal.hs
2. A Stream like sequences without wrappers. In fact, they are no data
storages but generator functions equipped with a seed. I use them for
causal signal processes. I think that most functions needed for drum
synthesis are causal. By inlining them, many subsequent causal signal
processes are fused to one.
http://darcs.haskell.org/synthesizer/src/Synthesizer/State/Signal.hs
> 2. Some of the most useful sound libraries aren't listed in Hackage's
> "Sound" section, including HCodecs, SDL-mixer and hogg.
my library is also not at Hackage, it's too much experimental currently
> 3. The Haskell OpenAL bindings are great, but ALUT has been removed
> from MacOS 10.5. Of course, a pure-Haskell version of ALUT would
> be pretty easy to write, and it could be used as a fallback.
> 4. '[[a]]' is _not_ a good representation for multi-channel audio
> data. Manipulating even 10 seconds of audio is noticeably slow.
Using the signal representations I mentioned above, I can synthesize some
songs in realtime. For stereo signals I use Signal (a,a). However, a
StereoFrame data type with Storable instance might be even better.
> 5. HCodecs is probably the best library for reading WAVE files. It
> uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'.
> It also supports MIDI and SoundFont, which is quite handy.
> 6. YampaSynth looks really cool, and it might be possible to turn it
> into a reusable MIDI synthesizer library for use with HCodecs.
> This would be extremely useful: it would provide a basis for all
> kinds of crazy MIDI-based programming in Haskell.
If you are after realtime synthesis you might also want to look into
SuperCollider. I posted some code to control SuperCollider via MIDI to
haskell-art. I can send it to you, if you like.
> What would the ideal Haskell sound API look like? Personally, I would
> love to see:
>
> a. Something like HCodecs's Data.Audio as the basic sound format.
> Data.Audio is an array-based API, and it supports converting
> between a wide range of common sample formats. I don't know how
> fast this is, but it already exists, and it's used by YampaSynth.
> b. OpenAL for sound playback, with a portable version of ALUT. This
> may require writing a pure-Haskell version of ALUT.
So far I used Sox' play command fed by a pipe with runInteractiveCommand.
> Other nice-to-have features might include:
>
> e. A standard MIDI format, based on either the HCodecs package or the
> midi package. (HCodecs is used by YampaSynth, and the midi package
> is used by alsa-midi.)
midi package is mainly used by Haskore
> f. A modular version of YampaSynth which can convert MIDI data
> structures into Data.Audio values.
>
> It looks like Haskell could be a really sweet audio programming
> environment with just a bit of integration work. What do folks think?
> Are there other libraries I should look at more closely? Other features
> that should be included in an ideal audio API?
What do you mean by 'API'? I think there should not be one library
handling all kind of music tasks. For interoperability common data
structures are nice. However, what signal representation concerns,
I think we need more than one representation, because different
applications have different needs.
> Haskell audio libraries
I have advertised my libraries for years in Haskell-Wiki and HCAR, but it
seems that people do not look there. :-(
> Reading and writing sound files:
> HCodecs: (Audible a) => DiffUArray Int a
> hsndfile: MArray with Float and Double
> HSoundFile: [[Double]]
> ALUT: OpenAL.AL.Buffer
> WAVE: [[Int32]]
>
> Playing sounds:
> OpenAL: Ptr UInt8, Ptr Int16, 1-N channels
> SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct
Sox via pipe, using any data structure which can be written to a file
> Sound processing libraries:
> dsp: Array a, [a]
> jack: Operates via mapping functions
> mainMono :: (CFloat -> IO CFloat) -> IO ()
> mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()
synthesizer: StorableVector, State, Generic (older: list, fusable list)
> MIDI-based:
> HCodecs: Reads and writes MIDI files
> midi: Reads and writes MIDI files
> alsa-midi: Uses midi library
> YampaSynth: Stand-alone program
Haskore uses MIDI and other back-ends like Haskell Synthesizer, CSound,
SuperCollider
> Special-purpose APIs only (FFIs, etc.):
> hCsound: CsoundPtr
> hsc3: UGen
HasSound
Henning Thielemann
On Sun, 24 Aug 2008, John Van Enk wrote:
> Eric,
>
> I was hoping to use a packed format like ByteString eventually. Right now, I
> want to get everything working nicely. As it stands, I end up marshaling a
> lot of information into/out of arrays which I'd much rather keep as a block
> of memory.
>
> I'm guessing that some sort of unboxed array would be close to what I want.
> If you have suggestions, I'm more than open to them. (This is my first
> attempt at writing a library.)
In recent versions of StorableVector (not at hackage yet, only in darcs
repository) there is a chunky lazy variant. However there are also the
uvector and vector packages, and no-one seems to know, on which one we
should focus.
Let me elaborate on my point, that there is not one data structre which
fits all needs: The great thing about Haskell's laziness is that you can
implement feedback easily. But this does not work with chunky lazy
sequences (StorableVector.Lazy) and causes quadratic computation time with
my State.Signal representation. It works only with element-wise lazy
structures, aka lists. I have an idea for a more efficient data structure
for that problem, but I'm uncertain if it is really more efficient in the
end.
Don Stewart
>
> On Sun, 24 Aug 2008, John Van Enk wrote:
>
> >Eric,
> >
> >I was hoping to use a packed format like ByteString eventually. Right
> >now, I
> >want to get everything working nicely. As it stands, I end up marshaling a
> >lot of information into/out of arrays which I'd much rather keep as a
> >block
> >of memory.
> >
> >I'm guessing that some sort of unboxed array would be close to what I
> >want.
> >If you have suggestions, I'm more than open to them. (This is my first
> >attempt at writing a library.)
>
> In recent versions of StorableVector (not at hackage yet, only in darcs
> repository) there is a chunky lazy variant. However there are also the
> uvector and vector packages, and no-one seems to know, on which one we
> should focus.
There's no immediate plans to merge them, but uvector will likely gain
a lazy variant soonish. As 'vector's announcement said, uvector is
the stable, more conservative variant that works with 6.8.2, and has
a richer API.
-- Don
John Van Enk
storing audio data? One of the major issues I'm still working over is that I
want to maintain something similar to a [[a]] format (since the underlying
PortAudio library and hardware could support hundreds of interleaved
channels) but I would like to be able to build in some typechecking to the
functions to make sure the number of channels matches the nubmer expected in
the functions. I've considered doing the following:
> data SingleChannel
> data DualChannel
> data TriChannel
> type Frames = [[a]]
> data AudioData a = AudioData Frames
> adata = [[x] | x <- [...]] :: AudioData SingleChannel
I'd like to have a better way of:
1) packing the audio data in something other than a list
2) representing more information about the data parameters (such as sample
rate and number of channels) without relying as heavily upon runtime
checks/errors (which HCodecs Data.Audio format seems to rely on).
The ideal format would be [(a,a,..a)], but this seems cumbersome (since each
tuple would have to be defined individually),
I'm pretty sure that a runtime check on both sample rate, and number of
channels will be necessary, but I just want to ask a little more before
continuing down that path.
I also noticed that Data.Audio uses a DiffUArray. In this circumstance,
would a StorableVector be better, or a DiffUArray?
On Sun, Aug 24, 2008 at 1:06 PM, Eric Kidd <has...@randomhacks.net> wrote:
--
/jve
Chris Waterson
> Greetings, Haskell folks!
>
> I'm working on a toy program for drum synthesis. This has lead me to
> explore the various sound-related libraries in Hackage. Along the
> way,
> I've learned several things:
Hey Erik, I've also created (minimal, for fun) bindings to libmad and
OS/X CoreAudio.
The original post is here:
http://www.haskell.org/pipermail/haskell-cafe/2008-March/040796.html
Here is a direct link to the source:
http://maubi.net/~waterson/REPO/hmad
http://maubi.net/~waterson/REPO/CoreAudio
I represented the samples using a lazily generated list of numbers,
and it worked pretty well.
chris
Henning Thielemann
On Mon, 25 Aug 2008, John Van Enk wrote:
> How well would the storablevector package (Data.StorableVector) work for
> storing audio data? One of the major issues I'm still working over is that I
> want to maintain something similar to a [[a]] format (since the underlying
> PortAudio library and hardware could support hundreds of interleaved
> channels) but I would like to be able to build in some typechecking to the
> functions to make sure the number of channels matches the nubmer expected in
> the functions.
With
data Stereo a = Stereo !a !a
you could also use
Stereo (Stereo a)
for quadrophony and so on. Would this be convenient enough?
StorableVector stores everything of fixed length for which a Storable
instance is defined.
John Van Enk
general solution... I'm almost positive this will require runtime checks.
Your solution is what I was thinking for functions requiring exactly N
channels (I'm not sure if there are many functions like that).
On Tue, Aug 26, 2008 at 2:11 PM, Henning Thielemann <
lem...@henning-thielemann.de> wrote:
>
> On Mon, 25 Aug 2008, John Van Enk wrote:
>
> How well would the storablevector package (Data.StorableVector) work for
>> storing audio data? One of the major issues I'm still working over is that
>> I
>> want to maintain something similar to a [[a]] format (since the underlying
>> PortAudio library and hardware could support hundreds of interleaved
>> channels) but I would like to be able to build in some typechecking to the
>> functions to make sure the number of channels matches the nubmer expected
>> in
>> the functions.
>>
>
> With
> data Stereo a = Stereo !a !a
>
> you could also use
> Stereo (Stereo a)
> for quadrophony and so on. Would this be convenient enough?
>
> StorableVector stores everything of fixed length for which a Storable
> instance is defined.
>
--
/jve
Henning Thielemann
On Tue, 26 Aug 2008, John Van Enk wrote:
> I think the problem I'll run into is the 128 channel case. I'm hoping for a
> general solution... I'm almost positive this will require runtime checks.
> Your solution is what I was thinking for functions requiring exactly N
> channels (I'm not sure if there are many functions like that).
If the number of channels is variable it might be better to use a list of
StorableVectors instead. I think it is more common to process the channels
separately instead of simultaneously.
Gwern Branwen
> Greetings, Haskell folks!
>
> I'm working on a toy program for drum synthesis. This has lead me to
> explore the various sound-related libraries in Hackage. Along the way,
> I've learned several things:
>
> 1. There's a lot of Haskell sound libraries, but no agreement on how
> to represent buffers of audio data.
> 2. Some of the most useful sound libraries aren't listed in Hackage's
> "Sound" section, including HCodecs, SDL-mixer and hogg.
Have you contacted the authors to ask they add the Sound categorization? I already sent a hogg patch, but couldn't find any darcs repositories for SDL-mixer or HCodecs.
..
> No public sound-buffer API:
> hbeat: The relevant source files are missing!
The cabal file was missing the Other-modules field. I've sent the author a patch.
--
gwern
Surveillance Defcon S511 anarchy REP NSIRL grenades dort UT/RUS W