Hayden from Sonoma Wire Works
Hayden
I appreciate the enthusiasm for doing this. I'm curious as to what you
hope to achieve that's different from Sonoma's AudioCopy/Paste
implementation. Are you talking about this being for MIDI
compatibility only rather than audio being transferred back and forth?
It looks like the conversation started with switching between apps.
That's something that AudioCopy/Paste does for every app that's on the
list already, as you noted since you're talking about using our plist
file as a template.
If you're eager to make something for MIDI, I can understand that and
I'm all for it. A lot of the midi apps are already using AudioCopy/
Paste. I'd be happy to assist you in anyway and provide any input from
our experience with AudioCopy/Paste.
I think doing the MidiCopy/Paste would be awesome, but none of our
apps use MIDI currently, so as you can imagine, we have other things
that have been taken precedence. But if this group wants to steer that
train, Sonoma would gladly assist, host any files, or make
contributions where possible.
For starters, let me explain briefly why the Sonoma SDK isn't open. It
really comes down to the use of the private pasteboard. This may have
changed as iOS has progressed, but it was possible at one time to
essentially sabotage a private pasteboard if you knew the pasteboard's
name. This is why we have an agreement attached to the AudioCopy/Paste
SDK. Also, we did some things that added really awesome features that
wouldn't be possible with the general pasteboard using the Apple audio
identifier. We wanted to make a more refined standard so that apps
could share audio files with certainty, knowing that they would be
compatible with each other. This is why we also limit audio to
44.1kHz-16bit wave files.
So that's why 100% open didn't work for us. We wanted to make sure
that apps that made it onto the compatible list understood these
things and intended to follow along.
Without central control of a list of compatible apps, I don't think
the compatibility for MIDI would be as valuable. That's my input. If
you want to go another way, that's fine with us. But we would love to
be included in the discussion.
Looking forward to seeing what you come up with.
Thanks,
Hayden
nlogmusic (TempoRubato - NLogSynth)
thanks for commenting!
Just for some clarity:
We are not talking here about MIDI pasteboards, we are talking in
general about
realtime communication.
This MIDI realtime communication allows App A (like a MIDI Controller
app eg. SoundPrism,
Polychord etc.) to send to App B (like NLog, Sunrizer, etc.) MIDI
event in realtime for
playback much like you do with real MIDI cables.
All this is based on two Apple technologies: Core MIDI & Background
Audio.
We have a clear understanding that the Sonoma SDK in general is NOT
related to this.
However, there is one little aspect which both workflows have in
common:
There is a need for convenient switching between apps.
In discussing this the Sonoma Compatible App List came into my mind.
So, the only question is, wether you request any intellectual
propriety of having
a central stored plist with app metadata or if we may use this idea
also for our
purposes to switch between apps.
Again, we are not talking about you GUI implementation, just the
design pattern.
Feel free to answer in open, or in private by email to myself.
Best
Rolf
Hayden Bursk
Gabriel Gatzsche (Audanika - SoundPrism)
Rob Fielding
but I have wondered for a long time. :-) What exactly *is* going on
to address actual background recording; and similarly, sending audio
into background effects processors (ie: a stand-alone stomp-box
distortion that runs as its own process in between a sound generating
app and an app that records).
Background MIDI comes very close to addressing this issue; with the
remaining problems mostly being some of the limitations of
representing certain things in MIDI or if the controller happens to
have a really great sound engine that the recording background synth
just can't match.
As I understand it, copy/paste is limited to non-real-time transfers
of audio already recorded; so I have always immediately dismissed it
as a solution to the problem of properly doing audio tracks from an
on-board controller. (Ie: music decidedly not oriented around
recording short loops, but multi-track bouncing very long
improvisations, etc.)
Is iOS5 bringing something new? Something to do with Mach ports?
Something that cannot be discussed here? Or do I have this wrong?
nlogmusic (TempoRubato - NLogSynth)
"Background Audio" is Apple's definition is just one of the few
"justifications"
an app can choose why it needs to continue to run and get CPU
resources
it is put into background i.e. another app gets into foreground.
When Apple devised that they thought primarily about music players
like radio apps etc. So, the background audio app delivers its audio
stream directly to an output device.
There is to my knowledge currently no Apple API to exchange audio
streams between two or more apps in realtime. There is in contrast
such a thing for MIDI data using the virtual port features of Core
MIDI.
Exchanging audio in realtime between apps as you described is however
a feature very much requested from us. So, if Apple does not provide
anything
directly, the question is, if there is anything we can implement which
at least
does not break any Apple rules i.e. get through their review process.
One thing discussed here and even in a former Google group was the
idea
to use socket connections to communicate audio between apps in
realtime.
I think Mach ports are the underlying abstraction Apple uses for
implementing
interprocess communication tools like sockets. There are some things
to
be taken into consideration here: First, it is quite some work to
implement
this reliable and it is not clear, if that would go through Apple's
review process.
An alternative approach which recently came into my mind is this: You
can
view virtual Core MIDI as a structured way of inter-app
communications. It
adresses tasks like finding out about other apps, making connections,
sending
and receiving data etc.
With SysEx you can use Core MIDI for all kind of even non-MIDI
communication
between apps without caring about low level things like sockets, Mach
ports etc.
It is quite obvious, that SysEx'ing via virtual MIDI could be used for
all sort of
semantical info, requests, calls etc. between apps including but not
restricted to
audio pasteboard data.
So, how fast virtual Core MIDI actually is? I do not know, but seems
to be fast
enough for event based realtime communication.
WOULD IT BE FAST ENOUGH FOR REALTIME AUDIO AS WELL?
Don't know, but worth a try. Here you virtual Core MIDI would 'piggy
back' audio
snippets like of output buffer size or smaller.
Why I am optimistic? Well, a few arguments:
- Virtual Core MIDI has proven quite good for MIDI clock even when BPM
goes to
400-600 and potentially more. Each quarter it sends 24 'clocks' that
would give a
clock frequency of 160-240 Hz for the mentioned BPM range. Typical
audio buffer
sizes like 256 or 1024 would have a frequency of 43 rep. 172 Hz. So,
quite similar.
-> Ok, yes, data packet size is much bigger in the audio buffer piggy
back than MIDI
clock.
- If you were the engineer in Apple's team to implement virtual Core
MIDI, what would
you use for transport protocol? Well, probably sockets rep Mach ports
etc. If we are
lucky these guys were clever enough to do it in a way which scales to
the requirements
of sending audio buffers at MIDI clock rate.
-> Ok, just hope, but there is a chance.
- Virtual Core MIDI messages have host time stamps, which are really
useful for sending
audio buffers. It is in fact the same data structure we get in the
audio callback.
In a nutshell: It is worth a try! I would give it a 50% chance to work
well.
Still, it remains the question if piggy backing audio in Virtual Core
MIDI would make
Apple's reviewer suspicious? Could be, yes! But on the other hand,
formally we are not
breaking any rule, since we actually can put into SysEx what we want.
This is probably
less conspicuous than creating own socket connections and fiddling
around with low level
system calls.
Any opinions?
Rob Fielding
what I think it is; it can't be made to to stream audio callback data
in real-time, or write bytes into a never-ending stream, no? I know
many users will complain if I don't support it, but I find
non-real-time audio copy/paste to be a non-solution to the problem of
an ipad controller's audio being recorded into a DAW; and I am not yet
convinced that doing this correctly is impossible.
I bet everyone in this group thinks about abusing MIDI for an
arbitrary byte stream; and I have as well. :-) But then I come to my
senses and think to myself...what a mess! But yes, midi can be abused
into being a subliminal channel. (ie: an alternate way to transport
information, another example is streaming mpeg over dns so that you
can watch internet videos over a hotspot at a cafe even though it
doesn't let you actually connect to its internet; etc, etc.)
Mach ports comes to mind, and I read enough about it existing on iOS
that I have my doubts that it is strictly forbidden. It appears to be
an arbitrary byte stream with unknown latency characteristics,
correct? The ideal thing would be to push renderbuffers into it at
the place where the audio callback normally is. Maybe it's doable but
not standardized? I think the right approach is to keep using MIDI
until it just gets so wierd that it's no longer compatible enough to
justify the complexity of transporting over it; at which point we
investigate OSC. (OSC seems like such a good idea, but you can't do
anything with music over Wifi as that seems to be the current
option... the latency guarantees are just nowhere near where they need
to be for starters.)
1) Can you do Grand Central Dispatch between processes?
2) If so, are there obvious disadvantages to it, like callback
scheduling introducing latency and creating unreasonable minimum
buffer sizes?
3) Is CoreMIDI basically just some trickery with mach ports?
nlogmusic (TempoRubato - NLogSynth)
within
an app to abstract from multi-threading and give iOS and potentially
the dev a better way to optimize it on multi-core CPUs. But there was
some sentences in the doc which said, that for realtime streaming
and other heavy duty use you still better use threading.
Apat from this, GCD to my knowledge isn't a way for inter-app
communication, but 'just' king of concurrency work balancing within an
app
to overcome threading.
re 3) Well, yes, it has to: iOS is based like OS X on the Mach kernel.
And to my knowledge ALL communication tools, APIs and layers are
eventually implemented with Mach ports. But is there any advantage
to go down to Mach ports instead of using BSD sockets? And anyhow
the chances to be accepted by Apple are in my feeling better when
'just' BSD sockets are used, instead of going down to Mach ports. But
even BSD sockets may make them reject an app?
I agree that piggybacking audio via MIDI is a kludge, but it may be
well
received by reviewer since we are not directly tinkering with low
level APIs.
And look, everything including app finding, 'port', connections.
timestamps
etc. are already implemented by the nice Apple engineers. Why not
check if
we can just re-use it?
Probably, better than theoretically discussing it, would be to make a
short
prototype if it works at all ;-)
>
> read more »
nlogmusic (TempoRubato - NLogSynth)
to over come the stupid WiFi latency rates.
On Oct 20, 3:53 am, "nlogmusic (TempoRubato - NLogSynth)"
>
> read more »
Rob Fielding
over MIDI because we want a byte stream is wildly idiotic but it would
work for sure
!
:-)
Support (One Red Dog)
In the kernel inter-process communication is carried out with mach ports. The user-land API for this is CFMessaegPort. So BSD sockets etc will simply be a stack on top of that. At the basic level a task can hold a reference to a port. The port can either send or receive a queue of messages. A message is a data structure.
On OS X CFMessagePort works inter-process as well as inter-thread. The CFMessagePort is attached to a run-loop and you get a callback on receive. I'm now using CFMessagePort to do inter-thread communication in Arctic rather than the higher level NSNotification and performSelectorOnMainThread. I've yet to submit this to the App Store but it passes Xcode's Validation step.
In theory all you'll need is some way to advertise what processes are available and a memory buffer. Then you can copy the audio data directly between processes. Actually what happens is you create a CFDataRef and send it to the port. You then get a receive callback which copies the data into a new buffer.
Check the sample code for BackgroundExporter. If you can do this in iOS then you can open remote ports and talk between processes. It's very likely that this is how CoreMIDI actually implements the virtual ports. They simply have a dictionary that everyone registers against, each port has a UID and from there at the lower levels it knows what (mach) port to send to. Though this is likely implemented in kernel space in the CoreAudio driver because of tight timing, the IPC is still likely implemented with mach ports.
nlogmusic (TempoRubato - NLogSynth)
I think a Socket/MachPort based approach would give without question
the required throughput. There are a couple of tasks to be
implemented:
- some init stuff to setup up the basic infrastructure & ports like
MIDIClientCreate, MIDIInputPortCreate, MIDIOutputPortCreate,
MIDIDestinationCreate, MIDISourceCreate
- a way to make running & compatible apps acquaintance with, i.e.
something like MIDIGetNumberOfSources/MIDIGetSource &
MIDIGetNumberOfDestinations/MIDIGetDestination
- a way to initiate & break a connection like MIDIPortConnectSource,
MIDIPortDisconnectSource
- a way to send data like MIDISend (BTW I wouldn't trust MIDISendSysex
here because it sends asynchronously, needs to be researches, but no
problem to send SysEx with MIDISend itself)
- a way to register a callback to receive data like in
MIDIDestinationCreate & MIDIInputPortCreate
Ok, you see where I'm heading too: It's all already in Core MIDI done
for us. All we need to do is to find out if it is reusable or not
scaling up for audio.
Apart from Core MIDI throughput there is also the question of quality
in error handling and timing. In best case most stuff of Core MIDI
runs at kernel
level where we are never allowed to get to i.e. we won't beat Core
MIDI anyhow. Or if they are just using user callable APIs we can do
also.
If somebody has time, we could do an abstraction and then different
implementations: One with sockets, one directly with MachPorts rsp.
via CFMessagePort,
one with Core MIDI piggy backing.
If any Apple engineer is reading this, please comment. Maybe privately
by email ;-)
On Oct 20, 4:14 am, "Support (One Red Dog)" <supp...@onereddog.com.au>
wrote:
>
> read more »
Michael (A Tasty Pixel - Loopy)
I'm very interested in the idea of (technical limitations
notwithstanding) developing a scheme to transport audio in real time
between apps (an idea I thought was excitingly original, till I found
you guys had beaten me to the punch ;-)). I love the idea of using
apps as effects filters, for example, or, say, recording the output
from one of the guitar amp apps into a live-looper app.
It seems CoreMIDI is quite well placed to transport PCM audio via
SysEx messages, given that the registration and transport systems are
already all there. Its use might be particularly beneficial when the
apps in question are already communicating via Core MIDI (for example,
an app could be sending audio with a delay filter, in time with the
app receiving the audio, via clock messages).
I'm thinking about putting together a couple of test apps to see how
it works, but I thought I'd check first to see if it's been done
already.
On Oct 20, 7:46 am, "nlogmusic (TempoRubato - NLogSynth)"
> > On OS X CFMessagePort works inter-process as well as inter-thread. The CFMessagePort is attached to a run-loop and you get a callback on receive. I'm now using CFMessagePort to do inter-thread communication in Arctic rather than the higher level NSNotification and performSelectorOnMainThread. I've yet to submit this to the App Store but it passes Xcode's Validation step.
>
> > In theory all you'll need is some way to advertise what processes are available and a memory buffer. Then you can copy the audio data directly between processes. Actually what happens is you create a CFDataRef and send it to the port. You then get a receive callback which copies the data into a new buffer.
>
> > Check the sample code for BackgroundExporter. If you can do this in iOS then you can open remote ports and talk between processes. It's very likely that this is how CoreMIDI actually implements the virtual ports. They simply have a dictionary that everyone registers against, each port has a UID and from there at the lower levels it knows what (mach) port to send to. Though this is likely implemented in kernel space in the CoreAudio driver because of tight timing, the IPC is still likely implemented withmachports.
>
> > On 20/10/2011, at 12:53 PM, nlogmusic (TempoRubato - NLogSynth) wrote:
>
> > > re 1) I believe Grand Central Dispatch is a way to schedule tasks
> > > within
> > > an app to abstract from multi-threading and give iOS and potentially
> > > the dev a better way to optimize it on multi-core CPUs. But there was
> > > some sentences in the doc which said, that for realtime streaming
> > > and other heavy duty use you still better use threading.
> > > Apat from this, GCD to my knowledge isn't a way for inter-app
> > > communication, but 'just' king of concurrency work balancing within an
> > > app
> > > to overcome threading.
>
> > > re 3) Well, yes, it has to: iOS is based like OS X on theMachkernel.
> > > And to my knowledge ALL communication tools, APIs and layers are
> > > to go down toMachports instead of using BSD sockets? And anyhow
>
> read more »
Rob Fielding
I cant hide my disdain for in-app-record and audio-copy-paste, and it would be exciting news to see a proof of concept with this.
IMHO, this would be the only way to actually meet the real requirement to be able to bounce audio tracks correctly. It fits with background midi conceptually as well. My instrument is geared towards long improvisations and, and recording little snippets misses the point completely.
I am currently balking at requests for ACP because that is a lot of work to get only part of what i really want, especially because the ios memory model is to crash you when you are overbooked for memory. I also think it ridiculous that controllers not only embed a synth, but a mini track recorder as well; rather than communicating with apps that do this as their core competence.
Sent from my iPhone, which is why everything is misspelled. http://rfieldin.appspot.com
http://rrr00bb.blogspot.com
Michael Tyson
I'm on it!
--
Michael Tyson | atastypixel.com
A Tasty Pixel: Artisan apps
aim: mikerusselltyson
twitter: MichaelTyson
nlogmusic (TempoRubato - NLogSynth)
there are two use cases I think we discussed:
1. Controller, DAW or Sequencer app sends Synth or Drum app MIDI notes
plus start/stop record. Synth app
does recording and pushed wave data to pasteboard.
For this use case I have a beta which already gave two other apps. If
anyone interested, just email me. I really
like to see this going and it could be released quickly.
2. Realtime audio streaming packages transported via Core MIDI SysEx
I started some prototyping. If anyone else does same or similar thing,
I would be more than happy to
share thoughts, first experiences etc.
Best
Rolf
On Nov 10, 12:21 pm, "Michael (A Tasty Pixel - Loopy)"
>
> read more »
Michael Tyson
I'd definitely be interested to hear about your impressions thus far, regarding the realtime SysEx transport - how far have you gotten to date? Does it look like it might be viable?
If it does, I'd love to put together a library and invite app developers to start supporting it - I think it could be quite a significant move!
Cheers,
Michael
--
Michael Tyson | atastypixel.com
A Tasty Pixel: Artisan apps
aim: mikerusselltyson
twitter: MichaelTyson
Michael Tyson
- (id)initWithDestinations:(NSArray*)destinations;
- (void)addDestination:(PGMidiDestination*)destination;
- (void)removeDestination:(PGMidiDestination*)destination;
- (BOOL)sendAudio:(const SInt16*)audio length:(NSUInteger)lengthInSamples numberOfChannels:(NSUInteger)channels forTime:(SInt64)hosttime;
@property (nonatomic, readonly) NSArray *destinations;
@end
- (id)initWithSource:(PGMidiSource*)source;
- (void)receiveAudio:(SInt16*)audio length:(NSUInteger*)ioLengthInSamples timestamp:(uint64_t*)timestamp;
- (void)receiveAudio:(SInt16*)audio length:(NSUInteger*)ioLengthInSamples timestamp:(uint64_t*)timestamp isLive:(BOOL)liveAudio;
@property (nonatomic, retain) PGMidiSource* source;
@property (nonatomic, readonly) NSUInteger numberOfIncomingChannels;
@end
@protocol APAudioReceiverDelegate <NSObject>
- (void)audioReceiver:(APAudioReceiver*)receiver didReceiveAudio:(SInt16*)audio length:(NSUInteger)lengthInSamples atTime:(SInt64)hosttime;
@end
--
Support (One Red Dog)
>
> That's the good news (and as far as I'm concerned, that's *seriously* good news!)!
>
Very cool!
> The bad news (but I'm nowhere near done exploring yet) is that, in "live mode" (where old samples are dropped to keep latency up), there's a lot of stuttering when doing almost anything on the device, even innocuous things like scrolling in a table view. I have no idea how backgrounding is implemented in iOS, but I suspect that background apps are given a low execution priority, presumably except for the high-priority core audio thread. This isn't a problem when not playing the audio at the end of the pipe live (for example, when recording in one app a performance from another instrument app, where the audio timestamp is sufficient to record the audio at the right time - I'm very optimistic about this scenario), but for live audio, a solution will need to be found before it's viable.
>
IMHO virtual MIDI ports are implemented on top of mach_ports (I use these to pass messages between the audio thread and UI main thread) and the memory buffers are malloc'ed. I think you'll have to buffer the audio at the receiving thread (something like CARingBuffer) and then pass it off to the audio render proc. If you're copying the audio from the MIDI receive callback straight into the audio render proc, I think you'll have dropped audio. I'm also not sure what would happen if you overly delay the MIDIReadProc. Of course if you're already doing this then please disregard my thread hijack.
AUHAL and aggregate devices would be ideal
cheers
peter
Michael Tyson
Actually, I already am using a ring buffer to store the incoming audio, which is then drained by the render thread. It occurs to me that my prior theory is totally wrong, though - the holdup isn't at the sender's end, it's on the receiver, as it has to skip buffers to keep the latency low. Turning live mode off to avoid skipping buffers prevents the glitching, but causes big latency issues.
I'm just tweaking the receiver now, trying to figure out where the bottleneck is; I was using a GCD queue to do the processing of the incoming MIDI packets, which might be lagging behind when things get busy. I'm getting rid of that double-handling and moving the processing straight into the buffer drain routine, which may help.
Michael Tyson
Support (One Red Dog)
Lovely - that's totally fixed it! Live audio now coming through smoothly and with nice low latency.I'm just tweaking the receiver now, trying to figure out where the bottleneck is; I was using a GCD queue to do the processing of the incoming MIDI packets, which might be lagging behind when things get busy. I'm getting rid of that double-handling and moving the processing straight into the buffer drain routine, which may help.
nlogmusic (TempoRubato - NLogSynth)
If there is anything we can test, just let us know!
Also, it would be interesting to check with Apple review process if
they are getting against it. So, before
doing too much functional development on it, it might be wise to put
something early in review to check out.
You can even hold the app store release back.
Anyway, cheers to this great stuff!
Christopher Randall
Chris Randall
Audio Damage, Inc.
http://www.audiodamage.com
![Ben [Camel Audio]'s profile photo](http://lh3.googleusercontent.com/a-/ACB-R5QdLCoRN0-5EQFEN6gRj9EPRGtp-Sl32Ov4SZB2=s40-c)
Ben [Camel Audio]
of you involved with developing Virtual MIDI (which we support in
Alchemy Mobile). Thanks and congrats to Michael for this work on
getting live audio streaming proof of concept working! If Apple are
OK with doing live audio streaming in this way (and I agree with Rolf
and Chris that submitting a trial App to the App store is a good way
to test), we'd love to support this in a future Alchemy update.
Cheers
Ben
Michael Tyson
Does anyone know if there's a way to actually directly ask the review team questions (like, "is this okay"), or do we truly have to go through the entire app submission pantomime to test acceptability? If so, it seems rather inefficient =)
--
Michael Tyson | atastypixel.com
A Tasty Pixel: Artisan apps
aim: mikerusselltyson
twitter: MichaelTyson
Christopher Randall
Chris Randall
Audio Damage, Inc.
http://www.audiodamage.com
Jesse Chappell
gray area that some of their technical staff would probably say is
against the rules. I'm guessing that even our use of virtual midi for
midi would throw up some flags, if they really paid attention to it.
Better to preemptively win by community acceptance than to ask
permission.
It would be great if you put the code up into a git repository somewhere!
jlc
Sebastian Dittmann
Is the way this is going to be accomplished going to involve any closed source third party libraries? How's the behavior of apps running in the background going to be changed? These are the questions I'd probably have to address.
My skype nick is sebastian.dittmann - if Michael (or anyone else) wants to contact me about this.
Best,
Sebastian