All about sound projects (audio)

[Simon Jackson]

New thread to continue latency etc. from feature requests.

[Simon Jackson]

The following is a rough outline for a sound application. It has a few things needing fixing. The while(true) s need replacing by while(process-running-boolean-variable) so as to end the background rendering threads. The code maybe needs checking over to see about the API version used.

https://groups.google.com/group/ioio-users/attach/e5d825e18aaf2405/M24-general-structure.tar.bz2?part=4&authuser=0&view=1

An IOIOLib is built into the program and maybe Control should be edited to provide some input!

There are many complaints against Android on latency issues. They involve the minimum buffer size for an AudioTrack stream. This is currently not avoidable. For monophonic (not polyphonic) instruments, maybe this is just a matter of flush() write(sound) write(silence), and then only the flush latency and a bit is involved. This may be too limiting for some.

Getting audio input involves a latency too, so making guitar effects pedals may be too difficult unless the effect involves delay, and some pass through is mixed in.

Of course you attach a DAC to the IOIO, and an SPI type DAC has been suggested.

Let's make a good resource here for people interested in making IOIO based audio devices with their phones.

Cheers Jacko

[Simon Jackson]

Some improvements to the synthesizer template. No IOIO customization yet. For those just starting in Android development, feel free to use the "uk.co.peopleandroid.thirdparty.<x>" to name any packages until you get a domain to place things onto market. When I set up for market, I will happily publish any synths you write using this template, as long as the source is supplied, no Trojans are included, it sounds good, and you wish to release them for free. The particular IOIO arrangement is not too important, and any described Control class of use will be allowed. This Android bad, iPhone good has to be stopped. Live long and boogie.

Cheers Jacko

[dain]

Great stuff Simon, really appreciate that you share this code!

Would you mind uploading it to Github or any other open source code
repository provider with good collaboration support?


Have a nice day,
dain

>  M24-interface-done-just-sound-todo.tar.bz2
> 285KViewDownload

[Simon Jackson]

When I've finished some more maybe. I'm currently working on an idea to detect what type of IOIO config is plugged in so as not to output any volts to a different setup. At the moment the idea is to detect pin 2 looping back to pin 1 (detect M24 panel). This allows a pin 2 to pin 3 loopback on a differing IOIO unit, and also a third and fourth mode with pin 1 or pin 3 to ground, as they would be weak pull up otherwise. This then only relies on pin 2 being made an output to detect a synth box. 

[Simon Jackson]

Also bare in mind RenderThread will not be filled out any further in a public release, save bug fixes. This is due to the fact I may wish to commercialize the particular M24 sound as a prebuilt unit, and may not wish the full algorithm details published or easy to duplicate. The general framework being public is in the interests of a fair game where sound variety is more important than interface individuality. To this end the idea of just changing the package name via 'refactor...', the application name via the manifest, the icon via adding a mini icon superimposed over the bottom right of the main supplied icon, editing the code of RenderThread, using any pre-designed synth IOIO box (or requesting a pin 2 loop to other pin number to be built in), and releasing on the market is a goal to music.

Any ideas are most welcome, as getting things to limited or plain wrong at this stage can lead to much a lack of sound later.

Cheers Jacko

[achim....@gmail.com]

On Tuesday, I got my IOIO (actually 2 of them :) ) and in the meantime I made some experiments with a VS1053b (i.e. the sparkfun breakout board). I think, I got the wiring right (at least, I hear music). However, I see no possibility to get the streaming / handshakes right for high quality audio.

Some background: The VS1053 has an 8KB audio buffer which is filled via SPI. If this buffer is full, it signals this via pin "dreq". If dreq gets high, there's room for at least 32 bytes. So basically, whats needed is:

while(true) {
    byte[] buffer = nextChunck(); // 32 bytes, currently, I just take this from a precomputed list of buffers

    dreq.waitForValue(true);

    spi.writeReadAsync(1, buffer, buffer.length, buffer.length, null, 0);
}

However, if I understand the ioio lib correctly, the spi.writeRead() is put into a queue and executed possibly some time later. I.e. the dreq signaling does not necessarily correspond with the spi writes. So I changed the spi write to request a 1byte answer:

    spi.writeRead(1, buffer, buffer.length, buffer.length, dummy_buffer, 1);

This way, I know the data has been written when the call to writeRead returns. Then querying dreq is safe. However, I can only stream mp3 data with the spec "MPEG ADTS, layer III, v2,  32 kbps, 22.05 kHz, Monaural".

MPEG ADTS, layer III, v2,  32 kbps, 22.05 kHz, JntStereo seems to be the limit. It already has contains slight noise which probably comes because the VS1053 buffer runs empty. Higher rates cause stutter.

BTW: I run SPI with a rate of 8Mhz, i.e. there's no room any more on the ioio side (if ioio would support higher rates and different rates for different spi clients on the same bus, the vs could hande slightly more than 15 MHz for audio data and 8.77 Mhz for commands).

I think, it would be best, if it would be possible to just push the data to the ioio (btw: it would be great if a spi write could send more than 64 bytes) and do the dreq data flow handling on the ioio.

It may also be ok if it would be possible to push 32 byte data packets and tell the ioio to send them as soon as some pin (i.e. dreq) gets high (or to be more general high or low as specified in the request). I don't know if this would allow for the needed throughput though. The goal would be to stream pcm 16bit stereo 44.1 or 48kHz ...

Is there a way, to do this?

[Simon Jackson]

That's approaching 1 Mb/s data rate, which is possible in the ADB and USB standards, I'm not sure of the core speed on the IOIO, and there is no synchronous parallel output of 32 bits in the IOIO as is. Maybe custom application firmware on the IOIO may be able to achieve this, but I'm not sure what the maximum USB rate and clock is. Maybe some preprocessing on the droid for 4 times oversampled delta sigma could quarter your bit rate, and reduce output changes to 2 pins (with analog alias filters), but I would not know as I have not investigated custom firmware.

Just as a side point, I wonder if any object creation, gc overheads could be removed from the IOIOLib by recycling objects? Haven't checked this out yet.

Maybe delaying the dreq by a number of clocks can allow a buffer greater than 32.

Cheers Jacko

[Ytai Ben-Tsvi]

Just use the async version of writeRead. You don't really need to block if I understand correctly. Btw, the blocking writeRead will block you even if you don't pass a response buffer.
Using async can also help you bypass the 64B limit by sending multiple requests immediately following without waiting for the round trip.

--
You received this message because you are subscribed to the Google Groups "ioio-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/ioio-users/-/dcpng7pUiycJ.
To post to this group, send email to ioio-...@googlegroups.com.
To unsubscribe from this group, send email to ioio-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ioio-users?hl=en.

[achim....@gmail.com]

We are talking about two different approaches:

Your approach seems to be: Synchronous (i.e. parallel) output of 32 bit (2x16bit samples?) to a DAC. Nice idea, but there seems to be no way for synchronous output of 32 bit. DACs with a serial interface seem to be a solution. Which leads me to my approach: use a VS1053b (it's not the cheapest chip, but it's little brother VS1003 should be cheap enough for a larger user base). This chip contains a "18-bit over- sampling, multi-bit, sigma-delta DAC" among other nice things (2xline in, decoding of various audio formats, headphone amplifier, ear speaker spatial processing, bass and treble controls...) and it uses a serial protocol for the sound data (SPI).

That's basically working. What's necessary though is to time the SPI transfers according to the dreq pin of the vs chip.Currently, this handshake handling (waiting for dreq, then triggering the spi transmission) has to be done via android which is to slow. Also, the dreq signal is generated by the vs chip. I see no way to change it or the 32 Byte size. So, I too think that a custom firmware is needed. I will look into that.

[achim....@gmail.com]

@Ytai: Fundamentally, the issue is not about blocking. The point is after the 8KB audio buffer of the vs chip has been filled, any bytes send to it via SPI are lost until there's space again in that buffer. The chip signals this via the dreq pin. If this pin goes high, there's space for at least 32 bytes. I.e. the SPI transfers have to be timed according to the state of that pin. Just firing (async) SPI transfers would lead to data loses since the audio buffer would overflow. Thus, it's necessary to wait for dreq to be high, then 32 bytes can be send. If the buffer becomes full due to that transfer, dreq will become low. So, before the next transfer is started, we have to wait until the previous transfer happened and assure that dreq is high.

If we use the async spi call, a simple check of dreq would not be sufficient, since dreq might still be high because the transfer did not happen yet (or because the state change of dreq to low has not yet been reported). This was the reason why I requested an answer byte. BTW: if I understand the sources correctly, writeRead without a response buffer of at least 1 byte effectively does _not_ block: writeRead calls writeReadAsync which returns a result in state "ready" if the readSize is 0. WriteRead then "waits" until the result is ready, which it already is. In effect writeRead does not block in that scenario.

[Ytai Ben-Tsvi]

Don't have the source in front of me, you may be right. But if you figure out the buffer size, why not just:
1. Async send enough bytes to fill it.
2. Every time you get the rising edge on the pin, async send additional 32B.
Either way, a trivial firmware mod can fix that.

On Feb 2, 2012 9:30 AM, "achim....@googlemail.com" <achim....@gmail.com> wrote:

@Ytai: Fundamentally, the issue is not about blocking. The point is after the 8KB audio buffer of the vs chip has been filled, any bytes send to it via SPI are lost until there's space again in that buffer. The chip signals this via the dreq pin. If this pin goes high, there's space for at least 32 bytes. I.e. the SPI transfers have to be timed according to the state of that pin. Just firing (async) SPI transfers would lead to data loses since the audio buffer would overflow. Thus, it's necessary to wait for dreq to be high, then 32 bytes can be send. If the buffer becomes full due to that transfer, dreq will become low. So, before the next transfer is started, we have to wait until the previous transfer happened and assure that dreq is high.

If we use the async spi call, a simple check of dreq would not be sufficient, since dreq might still be high because the transfer did not happen yet (or because the state change of dreq to low has not yet been reported). This was the reason why I requested an answer byte. BTW: if I understand the sources correctly, writeRead without a response buffer of at least 1 byte effectively does _not_ block: writeRead calls writeReadAsync which returns a result in state "ready" if the readSize is 0. WriteRead then "waits" until the result is ready, which it already is. In effect writeRead does not block in that scenario.

--

You received this message because you are subscribed to the Google Groups "ioio-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/ioio-users/-/r3W5dA_xgCUJ.

[achim....@gmail.com]

That's exactly what I tried. And yes, it works - for low quality audio streams (i.e. low transfer rate / buffer empties slowly). For high quality audio, it's to slow, because currently, I have to do this on android. Yes, I will look into the firmware and try to do it there (as soon as I have time). I just hoped there would be a way without a firmware change.

[Ytai Ben-Tsvi]

Got it.
Maybe open accessory mode will help, as it has lower latency.
Otherwise, custom firmware is the way to go.

On Feb 2, 2012 11:26 AM, "achim....@googlemail.com" <achim....@gmail.com> wrote:

That's exactly what I tried. And yes, it works - for low quality audio streams (i.e. low transfer rate / buffer empties slowly). For high quality audio, it's to slow, because currently, I have to do this on android. Yes, I will look into the firmware and try to do it there (as soon as I have time). I just hoped there would be a way without a firmware change.

--

You received this message because you are subscribed to the Google Groups "ioio-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/ioio-users/-/oOxXXvnMeB8J.

[Simon Jackson]

Some more progress. I've defined the sound RenderThread and I am debugging, hence the ToastThread class. The RenderThread class is copyrighted so read the head comments. The other classes are copyright but freely usable, within the scope of Java package naming conventions.

Happy sounds

[Simon Jackson]

A few minor bugs to sort out, and then some control inputs to build. I can see the need for using additive accumulators for indexing the sound and lfo, as after  while I think any change of pitch or tempo would lead to a squawk as the sound ran through notes catching upto where the playing would be at the new tempo or pitch. This is essential to work out as potentiometer ADC inputs can be noisy.

[achim....@googlemail.com]

Today, I worked on the soundcard. I tweaked the firmware (basically
SPI in blocks of 255 bytes & waiting for dreq of the VS-chip getting
high) and could play a 160kbit stereo mp3. My goal however, would be
PCM/16bit/48kHz/stereo, i.e. 16bitPerChannel*2channels*48000Hz=
1'536'000 bit/sec. I.e. I still need a factor of almost 10. It seems
to me like the communication via USB is the bottleneck.

Ytai, you said in the wishlist thread "We're talking about 1.5Mbit/sec
or so, which should be doable." Can you give me some pointers how you
would approach this? Did you think about a modified ioio firmware or a
"new" firmware? How would you transfer the data via USB? Does it make
sense to modify the ioio firmware to include some new protocol for
this case?

[Ytai Ben-Tsvi]

I'd start by simply running a a simple firmware on the IOIO which simply reads USB packets and acknowledges them without further processing, as well as a matching Android app that just sends data as fast as possible over the IOIOConnection, just to measure the theoretically possible bandwidth. You make want to look into the latency_tester firmware and Android app, which I think are a bit broken at the moment, but could serve as a starting point.

If you see that the bandwidth is high enough, we can start figuring out where the bottleneck is. What SPI rate are you using BTW?

--
You received this message because you are subscribed to the Google Groups "ioio-users" group.

[Simon Jackson]

It's more structured using final int, and now has no sensitivity to squawk. It also has the modulation turned down, as I missed the fixed point multiplication shift right. Doow!

Almost ready to start construction with ioio.

[Simon Jackson]

Started implementing controls in class Control. You may be able to see how the 32 dials will be handled. I'm also thinking about buttons and a pattern memory. You may freely use the 32 dials logic and code in your own projects.

[achim....@googlemail.com]

An update regarding the "Android soundcard".

In the meantime, I modified the latency test to only send data to the
ioio (instead of round-trips) and to use larger packet sizes. I have
the results here, but they were large enough, I think, above 300 (the
unit are 1024 Bytes per second right?). I now also modified the
firmware to only send via SPI when a certain pin (i.e. the dreq signal
from the vs chip) is high and to use larger packet sizes (up to 511
Bytes). There's still some bug I have not found yet for larger
packets. However, 511 Bytes should be enough for the moment. As
result, I can send up to 50kByte/sec to the VS chip (instead of ca.
20kByte/sec for 64Byte blocks). The goal is around 180kByte/sec.

Interestingly, I get around 200kByte/sec if I clear the ByteQueue at
the end of SPITransfer, effectively short-cutting the SPI transfers. I
guess, I will have to look more closely into interrupts and SPI
transfers on PIC processors. Ideas how to improve this are welcome ;)

On 12 Feb., 02:45, Ytai Ben-Tsvi <yta...@gmail.com> wrote:
> I'd start by simply running a a simple firmware on the IOIO which simply
> reads USB packets and acknowledges them without further processing, as well
> as a matching Android app that just sends data as fast as possible over the
> IOIOConnection, just to measure the theoretically possible bandwidth. You
> make want to look into the latency_tester firmware and Android app, which I
> think are a bit broken at the moment, but could serve as a starting point.
> If you see that the bandwidth is high enough, we can start figuring out
> where the bottleneck is. What SPI rate are you using BTW?
>

> On Sat, Feb 11, 2012 at 12:32 PM, achim.leub...@googlemail.com <

[Ytai Ben-Tsvi]

I think you must have some flow control mechanism, similar to the one available today in SPI, UART and I2C. Otherwise, you are guaranteed to run into an overflow or underflow.
Your results are encouraging though. They mean that your goal is feasible.
It would be nice if your new functionality can fit in the existing design rather than require an independent firmware and library. Do you know how much throughput can be achieved on SPI simply by increasing the SPI tx buffers and sending data as fast ad possible?
This could be measured on the Android side using the existing API, as flow control is already built into the protocol and implementation.

[achim....@googlemail.com]

So far, I used your flow control mechanism. It took some time for me
to understand/see it, but I now think that I need a different flow
control mechanism. Your approach (to signal the empty space in the
ioio buffer and to only send an spi packet when the buffer has enough
space for it) roughly corresponds to the default throughput scenario
of the latency test (i.e. a full round-trip). With this approach, I
can only reach around 150-160 kB/s in the latency test (1024 Byte
packets, using ByteQueuePushBuffer instead of ByteQueuePushByte in the
AppCallback, ...). Larger packets did not increase the throughput
anymore. So you were spot on mentioning the flow control.

I think, I will try to send packets as often as I can and to adjust
the packet size based on feedback from the ioio. I'm not sure about
the details yet though...

I would be happy if this eventually becomes a part of your firmware.
However, it probably would be a new "unidirectional(?) stream mode"
for SPI?

On Feb 16, 5:50 am, Ytai Ben-Tsvi <yta...@gmail.com> wrote:
> I think you must have some flow control mechanism, similar to the one
> available today in SPI, UART and I2C. Otherwise, you are guaranteed to run
> into an overflow or underflow.
> Your results are encouraging though. They mean that your goal is feasible.
> It would be nice if your new functionality can fit in the existing design
> rather than require an independent firmware and library. Do you know how
> much throughput can be achieved on SPI simply by increasing the SPI tx
> buffers and sending data as fast ad possible?
> This could be measured on the Android side using the existing API, as flow
> control is already built into the protocol and implementation.

> On Feb 15, 2012 11:54 AM, "achim.leub...@googlemail.com" <

[Ytai Ben-Tsvi]

The flow control on the Android side should be fairly easy to change, and the IOIO-side doesn't need changing I think (only increase the buffers).

But I'm not sure you actually need to change that at all. As long as you have large enough buffers on the IOIO and you're sending small enough packets on the Android, the difference between waiting until a certain amount of buffer space is available on the IOIO or adjusting the packet size to match however free space is available should be negligible.

[Simon Jackson]

The Controls class so far

package uk.co.peopleandroid.m24;

import ioio.lib.api.*;
/* The current pin assignments are all analog ins
* Pin 47 evens select output
* Pin 48 odds select output
* Pin 1 output to loopdetect via
* Pin 2 loop input to prevent activation
* of any other outs except pin 1
* unless loopback is detected.
*/
public class Control extends ToastThread {

public char[] dials = new char[32];//main dials
char[] adcIn = new char[32];
AnalogInput[] ai;
DigitalOutput c0, c1;

public boolean running = true;
final int shift = 16;

IOIO io;
Connect c;

class Connect extends ToastThread {

Connect() {
this.start();
}

void runCatch() {
while(running) {
io = IOIOFactory.create();
try {
io.waitForConnect();
check();
ai = new AnalogInput[16];
for(int i = 0; i < 16; i++) {
ai[i] = io.openAnalogInput(i+31);
}
c0 = io.openDigitalOutput(47,false);
c1 = io.openDigitalOutput(48,false);
} catch(Exception e) {
io.disconnect();
}
try {
io.waitForDisconnect();
} catch (Exception e) {

}
}
}

void check() throws Exception {
//tests pin 1 to pin 2 loopback
c0 = io.openDigitalOutput(1);
DigitalInput d1 = io.openDigitalInput(2);
for(int i = 0; i < 256; i++) {
c0.write((i & 1) == 0);
try {
Thread.sleep(10);
} catch (Exception e) { }
if(d1.read() != ((i & 1) == 0)) throw new Exception();
}
}
}

void sample(char odd) {
//sample adcs and then set control ready for next sampling
try {
for(int i = 0; i < 32; i += 2) {
//i+odd
adcIn[i+odd] = (char) (ai[i >> 1].read() * 65535);
}
//control signal out
if(odd == 0) {
c0.write(true);
c1.write(false);
} else {
c0.write(false);
c1.write(true);
}
} catch (Exception e) {

}
}

void set() {
// an iterative for root finding dials from
// adc inputs stored as odd even pairs based on
// the control major.
int est;
for(int i = 0; i < 32; i += 2) {
est = estimate(dials[i], dials[i + 1]);
dials[i] += (adcIn[i] - est) >>> 2;
limit(i);
est = estimate(dials[i + 1], dials[i]);
dials[i + 1] += (adcIn[i + 1] - est) >>> 2;
limit(i + 1);
}
}

void limit(int i) {
if(dials[i] < 0) dials[i] = 0;
if(dials[i] > ((1 << shift) - 1)) dials[i] = ((1 << shift) - 1);
}

int estimate(int major, int minor) {
/* estimates scaled voltage input from two pots
* such that the pots are common ground, each
* have the other wiper end to individual control
* signals output antiphase, and have the wiper centres
* are joined via resistors 10 times the pot value, to
* make a ADC input. The major pot has control +ve
* and the minor pot has control grounded.
*/
int rMajor = (major * (1 - major)) >> shift;
int rMinor = (minor * (1 - minor)) >> shift;
return ((major * (10 + rMinor)) / ((20 + rMinor) + rMajor)) >> shift;
}

public Control() {
for(int i = 0; i < dials.length; i++) {
dials[i] = (char) (Math.random() * 65535);//16 bit limit
}
c = new Connect();
this.start();
}

void runCatch() {
char b = 0;//even
while(running) {
sample(b);
b++;
b &= 1;
set();
try {
Thread.sleep(10);
} catch (Exception e) { }
}
}
}

[Simon Jackson]

Made some edits for the little disconnect, reconnect race condition maybe which would make the dial values wobble a little. But no major changes. Investigating 1/f noise for music generation, and deciding on a case for the parts which have now turned up. Decided to use pins 23,24 for loopback device protect detection as they are off the DIP lines.

Cheers