Developing RTSP server
Stas Oskin
Is it possible to develop a Java-based RTSP server, using Xuggler / FFMPEG RTP streaming?
Regards.
Art Clarke
help answers questions you have about Xuggler if you decide to
implement it.
- Art
http://www.xuggle.com/
xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and video.
Use Xuggle to get the power of FFMPEG in Java.
Stas Oskin
Does Xuggler already supports RTP output?
Regards.
P.S., since hearing the name Xuggler, I constantly associate it with Googler. Any relation? :)
Art Clarke
as in "juggle".
Xuggler should support RTP output already since FFMPEG will write RTP.
That said, we don't have any networking built in -- you've have to
build the server.
- Art
Mondain
Stas Oskin
Thanks for the links, looks interesting.
I actually planned to try and adapt the live555 RTSP library via JNA, due to performance considerations of C++ Java. The library is located here:
http://www.live555.com/liveMedia/
What do you think, is the performance difference worth taking this path, or it's quite insignificant, as it basically a read/write to port while the most processing goes to encoding?
Regards.
Art Clarke
time spent encoding -- I wouldn't worry about it.
The reason we use JNI is so that we can do lots of things to stop you
from accidentally leaking memory in C++; that's really the advantage
that xuggler provides over every other FFMPEG wrapper out there. Do
this in Xuggler:
while(1)
{
IPacket pkt = IPacket.make();
container.readNextPacket(pkt);
}
and nothing bad happens (i.e. the garbage collector frees memory for
you). Do the equivalent in another wrapper, and you'll run out of
native heap quickly. They all provide some sort of "free" method you
must know how to call.
The live555 source code is C++ and so you'd have to worry about the
memory management of it in Java. Just be careful and you'll be fine,
or consider reusing the approach we use in our com.xuggle.ferry
package and namespace to do the same memory management we do (i.e.
make wrapper objects that derive from RefCounted, and use Swig to
wrap).
Lots of data there, but the answer to your first question is JNA is
only slower than JNI for the first method call.
- Art
Stas Oskin
JNA is slightly slower at startup than JNI, but it pails before the
time spent encoding -- I wouldn't worry about it.
The reason we use JNI is so that we can do lots of things to stop you
from accidentally leaking memory in C++; that's really the advantage
that xuggler provides over every other FFMPEG wrapper out there. Do
this in Xuggler:
while(1)
{
IPacket pkt = IPacket.make();
container.readNextPacket(pkt);
}
and nothing bad happens (i.e. the garbage collector frees memory for
you). Do the equivalent in another wrapper, and you'll run out of
native heap quickly. They all provide some sort of "free" method you
must know how to call.
I know what you mean - been in this situation :).
The live555 source code is C++ and so you'd have to worry about the
memory management of it in Java. Just be careful and you'll be fine,
or consider reusing the approach we use in our com.xuggle.ferry
package and namespace to do the same memory management we do (i.e.
make wrapper objects that derive from RefCounted, and use Swig to
wrap).
Lots of data there, but the answer to your first question is JNA is
only slower than JNI for the first method call.
What I really want to know, is whether it worth it to integrate live555 using JNA/JNI from performance consideration, or it's insignificant (as it's reading/writing over wire after all) I just should go a pure Java route.
Thanks for your opinion.
Art Clarke
nothing about RTSP.
What I do know is that encoding video and audio into payloads to send
over RTSP is (a) computationally significant but (b) more importantly,
silly to rewirte in pure Java given how great and active FFMPEG is.
Really it's because of how active and feature-full that FFMPEG is
already that I don't think you should replicate any FFMPEG code in
pure Java. But re other stuff, my general rule is implement something
in pure Java first, and then figure out where your performance
problems are, unless you have a good reason not to.
- Art
Stas Oskin
What I do know is that encoding video and audio into payloads to send
over RTSP is (a) computationally significant but (b) more importantly,
silly to rewirte in pure Java given how great and active FFMPEG is.
Well, that's my intent as well - use FFMPEG for packets creation. The question is what language to use to develop the transport "pipe".
Really it's because of how active and feature-full that FFMPEG is
already that I don't think you should replicate any FFMPEG code in
pure Java. But re other stuff, my general rule is implement something
in pure Java first, and then figure out where your performance
problems are, unless you have a good reason not to.
So you advice to go with Java first. Actually I'm drawn to this direction as well, as it should be much faster to develop a working version with this approach. My only concern is that the RTSP implemented in Java would be in magnitudes slower then a C++ counterpart.
Actually, after all the CPU intensive work done by FFMPEG already, I hope that sending and getting the packets would have minimal difference in performance between C++ and Java. Your opinion on the matter is appreciated of course.
Art Clarke
> So you advice to go with Java first. Actually I'm drawn to this direction as
> well, as it should be much faster to develop a working version with this
> approach. My only concern is that the RTSP implemented in Java would be in
> magnitudes slower then a C++ counterpart.
>
My experience is that dealing with pure Java implementations you can
achieve near C++ speeds with recent JVMs. I really think you're
taking the right approach in going C++ first. To hedge your bets,
make sure you implement via Interfaces -- that way later you can
replace with a "native implementation" like we do with Xuggler if
performance requires it.
> Actually, after all the CPU intensive work done by FFMPEG already, I hope
> that sending and getting the packets would have minimal difference in
> performance between C++ and Java. Your opinion on the matter is appreciated
> of course.
I mostly agree, although you'll probably quickly find that your
performance bottleneck is not going to be FFMPEG encoding time, but in
fact doing whatever you can to minimize network latency -- and that's
using Socket IO tricks, not whether it's Java or C++ (for example,
setting the right buffer size).
Regardless, there is almost 0% chance that your bottleneck will be
sending data between C++ and Java. In fact with Xuggler's IBuffer
object and a little care on your part, you can have Java code directly
access C++ bytes without copying between arrays (see
IBuffer#getByteBuffer). If you do this please READ THE DOCs as there
is a safety issue you need to follow (make sure the IBuffer lives
longer in the JVM than the returned ByteBuffer does), but it's hella
fast.
- Art
Stas Oskin
My experience is that dealing with pure Java implementations you can
achieve near C++ speeds with recent JVMs. I really think you're
taking the right approach in going C++ first. To hedge your bets,
make sure you implement via Interfaces -- that way later you can
replace with a "native implementation" like we do with Xuggler if
performance requires it.
Just to verify - you meant "going Java first" actually? (because AFAIK there are no Interfaces in C++ :) )
Regardless, there is almost 0% chance that your bottleneck will be
sending data between C++ and Java. In fact with Xuggler's IBuffer
object and a little care on your part, you can have Java code directly
access C++ bytes without copying between arrays (see
IBuffer#getByteBuffer). If you do this please READ THE DOCs as there
is a safety issue you need to follow (make sure the IBuffer lives
longer in the JVM than the returned ByteBuffer does), but it's hella
fast.
You probably mean that I need to keep IBuffer as otherwise it would de-allocate the memory and make the returned handle invalid.
Regards.
Art Clarke
>> My experience is that dealing with pure Java implementations you can
>> achieve near C++ speeds with recent JVMs. I really think you're
>> taking the right approach in going C++ first. To hedge your bets,
>> make sure you implement via Interfaces -- that way later you can
>> replace with a "native implementation" like we do with Xuggler if
>> performance requires it.
>
> Just to verify - you meant "going Java first" actually? (because AFAIK there
> are no Interfaces in C++ :) )
I mean write with an "interface-like" approach. Java supports
Interfaces naturally but C++ does not (well, C++ has "pure virtual"
methods). My recommendation is that regardless of language, implement
interfaces separately from underlying objects, and then you can swap
out implementations later as performance needs dictate.
For example in our other (non-open) code we have an object that merges
video streams on the fly. We implemented that with an
interface-approach, found that the original algorithm for detecting
picture-overlap was too slow, and completely replaced the
implementation without any users having to know for a 100x performance
improvement.
If you dig into Xuggler you'll see that our C++ implementation (which
is then just thinly wrapped in Java) does exactly this. For example,
com::xuggle::xuggler::IVideoResampler is actually implemented by
com::xuggle::xuggler::VideoResampler.
>
>>
>> Regardless, there is almost 0% chance that your bottleneck will be
>> sending data between C++ and Java. In fact with Xuggler's IBuffer
>> object and a little care on your part, you can have Java code directly
>> access C++ bytes without copying between arrays (see
>> IBuffer#getByteBuffer). If you do this please READ THE DOCs as there
>> is a safety issue you need to follow (make sure the IBuffer lives
>> longer in the JVM than the returned ByteBuffer does), but it's hella
>> fast.
>
> You probably mean that I need to keep IBuffer as otherwise it would
> de-allocate the memory and make the returned handle invalid.
Yes, that's exactly what I mean. Unfortunately though there is no way
for us to protect you from this given that Java doesn't let you
override the implementation of java.nio.ByteBuffer, and if you then
try to use the ByteBuffer that IBuffer#getByteBuffer returned after
all references to your original IBuffer have been cleared, you are
likely to crash the JVM. We try hard to make this impossible, but
figured that for some folks they'd rather have the dangerous access in
order to avoid the byte copy.
- Art
Stas Oskin
I mean write with an "interface-like" approach. Java supports
Interfaces naturally but C++ does not (well, C++ has "pure virtual"
methods). My recommendation is that regardless of language, implement
interfaces separately from underlying objects, and then you can swap
out implementations later as performance needs dictate.
For example in our other (non-open) code we have an object that merges
video streams on the fly. We implemented that with an
interface-approach, found that the original algorithm for detecting
picture-overlap was too slow, and completely replaced the
implementation without any users having to know for a 100x performance
improvement.
Sounds as a good approach, I'll stick to it.
Yes, that's exactly what I mean. Unfortunately though there is no way
for us to protect you from this given that Java doesn't let you
override the implementation of java.nio.ByteBuffer, and if you then
try to use the ByteBuffer that IBuffer#getByteBuffer returned after
all references to your original IBuffer have been cleared, you are
likely to crash the JVM. We try hard to make this impossible, but
figured that for some folks they'd rather have the dangerous access in
order to avoid the byte copy.
Well, it what one needs to pay for getting performance. JNA has same approach of letting to access the memory directly via getByteArray().
By the way, how the buffer freed then - by releasing the IBuffer object?
Regards.
Art Clarke
> Sounds as a good approach, I'll stick to it.
>
> approach of letting to access the memory directly via getByteArray().
>
> By the way, how the buffer freed then - by releasing the IBuffer object?
It gets freed when the IBuffer is no longer reachable by the JVM. No
explicit release is required.
Although if you want to force a release, you can call the delete()
method on the IBuffer. But then you must make sure that no one else
in Java still has that IBuffer.