How convert motion JPEG ( M-JPEG ) video frame to a still JPG file?
Ed of the Mountain
rather than a tool that does it for me.
I have a M-JPEG encoded video file. I want to learn how to extract a
frame and save as a JPP file. My current process has either too much
or too little data as JPG viewers report it as a damaged. Here's my
current process:
1 - Copy data from M-JPEG video file starting at SOI, DQT { ff d8 ff
db } to the next SOI, DQT and save as a JPG file
Thanks in advance for any tips or direction,
-Ed
Roman Ryl...
First of all two M-JPEG terms are often confused: server push stream
typically over HTTP and an encoding similar to JPEG.
Briefly, the first one is typically a HTTP stream of MIME type
multipart/x-mixed-replace with text boundaries that separate
individual frames. Textual parsing lets you split the stream into
frames.
The second is format very much similar to JPEG with a few sections
hardcoded so that to decode you need to assume you don't have these
section in the payload data and use hardcoded table instead. Or, in
order to convert from this type of Motion JPEG to JPEG you need to
merge MJPEG data with those tables to build up a valid JPEG frame.
If you have a stream of JPEGs without explicit separators, such as for
example a feed from StarDot camera on port 8001, you can crack the
stream into frames by looking for SOI markers FF D8 as you mentioned
above.
Roman
Alessandro Angeli
> I have a M-JPEG encoded video file. I want to learn how
[...]
As Roman said, M-JPEG is a bit vagues, since there are too
many formats that are called M-JPEG. Your technique only
works with streams of JPEG pictures serialized back to back
and will produce raw JPEG files, not valid JFIFs (.JPG - I
have no idea what a JPP file is supposed to be).
If your files use some kind of framing, they will end up
with extra data at the end (which should not be a problem).
If they use hardcoded data (e.g. AVI2.0/OpenDML MJPEG), they
will be missing possibly important parts (like the DHT or
the JFIF header).
Take a look at Q19 in my FAQ for sample code to extract
valid JFIFs from an OpenDML MJPEG AVI file using DirectShow.
If you have some other kind of MJPEG files, then you should
discover what layout they use.
--
// Alessandro Angeli
// MVP :: DirectShow / MediaFoundation
// mvpnews at riseoftheants dot com
// http://www.riseoftheants.com/mmx/faq.htm
Ed of the Mountain
Thank you for your reply.
> The second is format very much similar to JPEG with a few sections
> hardcoded so that to decode you need to assume you don't have these
> section in the payload data and use hardcoded table instead. Or, in
> order to convert from this type of Motion JPEG to JPEG you need to
> merge MJPEG data with those tables to build up a valid JPEG frame.
I have an example M-JPEG encoded video from a security camera. I
understand motion JPEG is basically a series of JPEG compressed stills
with no inter-frame compression.
> If you have a stream of JPEGs without explicit separators, such as for
> example a feed from StarDot camera on port 8001, you can crack the
> stream into frames by looking for SOI markers FF D8 as you mentioned
> above.
I must be missing something still.
1 - I save the data starting from the beginning of a SOI { FF, D8 }
until just before the start of the next SOI as a JPG file.
2 - Then starting at the end of the file, I search backwards for FF D9
and delete everything after it. I assume these 23 bytes at the end
were some sort of file container or frame header stuff.
See pastebin link below where I pasted the first 4 lines and last 4
lines from my hex editor after I followed the above two steps. Does
anything stand out?
Thanks again. Any suggestions on what I could be missing are much
appreciated,
-Ed
Roman Ryl...
> See pastebin link below where I pasted the first 4 lines and last 4
> lines from my hex editor after I followed the above two steps. Does
> anything stand out?
Not much, looks like JPEG, but who knows if it is a valid JPEG.
Perhaps if you tell the model of the security camera, it might
explain. The bytes before EOI marker don't look like JPEG friendly
data, but it's OK for security cameras, there is a lot of garbage in
the market.
Also just save the bytes between FF D8 ... FF D9 (including those) and
try to open as a JPEG file.
Roman
Alessandro Angeli
> I have an example M-JPEG encoded video from a security
> camera. I understand motion JPEG is basically a series
> of JPEG compressed stills with no inter-frame compression.
Basically, but not necessarily just that.
> 1 - I save the data starting from the beginning of a SOI
> { FF, D8 } until just before the start of the next SOI as
> a JPG file. 2 - Then starting at the end of the file, I
> search backwards for FF D9 and delete everything after
> it. I assume these 23 bytes at the end were some sort of
> file container or frame header stuff.
>
> See pastebin link below where I pasted the first 4 lines
> and last 4 lines from my hex editor after I followed the
> above two steps. Does anything stand out?
There's clearly an 'MJPG' FOURCC in the footer and the FF
sequence doesn't look like valid JPEG data. Notice that,
since the JPEG data is framed, there is no reason to think
that the framing syntax does not allow for start code
emulation, hence looking for SOI and EOI markers from both
ends of the file is not guaranteed not to find bogus codes
that are instead part of the non-JPEG framing data.
Ed of the Mountain
> many formats that are called M-JPEG. Your technique only
> works with streams of JPEG pictures serialized back to back
> and will produce raw JPEG files, not valid JFIFs (.JPG - I
> have no idea what a JPP file is supposed to be).
Sorry, "JPP" was a typo. I meant "JPG"
> If your files use some kind of framing, they will end up
> with extra data at the end (which should not be a problem).
Apparently, it does indeed use framing as there are an extra 23 bytes
between the SOI-to-EOI data blocks.
> If they use hardcoded data (e.g. AVI2.0/OpenDML MJPEG), they
> will be missing possibly important parts (like the DHT or
> the JFIF header).
Could a missing JFIF { FF, E0 } header be why "Windows Photo Gallery"
thinks the image file is corrupted?
> Take a look at Q19 in my FAQ for sample code to extract
> valid JFIFs from an OpenDML MJPEG AVI file using DirectShow.
Thank you for the nice FAQ. Sorry, where can I find a copy of
"BaseClasses? The VS solution says it is missing?
> If you have some other kind of MJPEG files, then you should
> discover what layout they use.
The M-JPEG encoded AVI file I found seems to UDP stream properly from
a VLC server to a VLC client which I use to simulate the hardware I
will eventually have. Locating sample M-JPEG encoded video files has
been challenging. I'll keep Googling. If you know of any sources you
can share I am interested.
< Cut from your more recent Post >
>There's clearly an 'MJPG' FOURCC in the footer and the FF
>sequence doesn't look like valid JPEG data. Notice that,
>since the JPEG data is framed, there is no reason to think
that the framing syntax does not allow for start code
emulation, hence looking for SOI and EOI markers from both
ends of the file is not guaranteed not to find bogus codes
that are instead part of the non-JPEG framing data.
So you are saying that since there is clearly 'MJPG' FOURCC stuff in
the SOI-to-EOI data I cut&paste then there could be other MJPG
associated data that my JPG image viewer interprets as a corrupted
file?
Thank very much for your time and detailed explanations!
-Ed
Roman Ryl...
> Could a missing JFIF { FF, E0 } header be why "Windows Photo Gallery"
> thinks the image file is corrupted?
If it was an analog security camera and video was compressed by some
codec then MJPG is more likely a version of JPEG with missing parts.
However if it was a network/IP camera, then most likely is is a JPEG
data. Far not every IP camera produces image with a JFIF header but it
does not make the image invalid an unrecognizable by hardware. On the
other hand a great deal of IP cameras produce M-JPEG feeds to be
recognized by their own client side thin clients and they may
mercilessly insert custom data that actually violate JPEG framing and
markers, but on the client side they quite happily recompose data into
valid JPEG and decode. I'd say it is very likely that you have this
scenario.
Roman
Roman Ryl...
hardware" should be "and unrecognizable by software".
Alessandro Angeli
> Could a missing JFIF { FF, E0 } header be why "Windows
> Photo Gallery" thinks the image file is corrupted?
It could, but I don't think WPG is that dumb. As Roman said,
it is more likely you are missing some other part, like the
DHT. Since below you wrote you are using an MJPEG AVI file,
that is most likely the case, since standard AVI MJPEG uses
a fixed DHT which is not present in the MJPEG stream itself
(my sample code shows how to deal with this case).
> Thank you for the nice FAQ. Sorry, where can I find a
> copy of "BaseClasses? The VS solution says it is missing?
You need an old DirectX SDK or (better) the latest Windows
SDK v6.1 for Win2008. It's in
"<SDK>\Samples\Multimedia\DirectShow\BaseClasses".
> The M-JPEG encoded AVI file I found seems to UDP stream
> properly from a VLC server to a VLC client which I use to
> simulate the hardware I will eventually have.
That is *not* a good simulation unless you know for sure
that your hardware uses the same MJPEG variant and the same
framing VLC uses.
> Locating
> sample M-JPEG encoded video files has been challenging.
> I'll keep Googling. If you know of any sources you can
> share I am interested.
The text of Q19 gives you a technique to easily generate
MJPEG AVI files.
> So you are saying that since there is clearly 'MJPG'
> FOURCC stuff in the SOI-to-EOI data I cut&paste then
> there could be other MJPG associated data that my JPG
> image viewer interprets as a corrupted file?
Yes.
Ed of the Mountain
> It could, but I don't think WPG is that dumb. As Roman said,
> it is more likely you are missing some other part, like the
> DHT. Since below you wrote you are using an MJPEG AVI file,
> that is most likely the case, since standard AVI MJPEG uses
> a fixed DHT which is not present in the MJPEG stream itself
> (my sample code shows how to deal with this case).
There are about 300 bytes extra at the top of the AVI file that could
be a fixed DHT. The byte values are 90% all FF's and I also see a
"MJPG" string. Both WMP and open source VLC ( using ffmpeg ) played
back the video so I assume it is not a proprietary security video
format. But then I expect players must be pretty smart to be able to
play almost any video format known to man.
> That is *not* a good simulation unless you know for sure
> that your hardware uses the same MJPEG variant and the same
> framing VLC uses.
My application is real-time encoding, decoding, and playback of video.
Basically replace the analog cables with Ethernet like surveillance IP
cameras. The encoding is being done in hardware. The decoding/
playback will be done in software. Eventually I need to support
motion JPEG2000 with either the OpenJPEG library or a third party SDK.
For my simulation purposes all I really need is a video that uses
frame-based encoding with no key frames or inter-frame compression. I
want to be able to identify the incoming frames being received over
UDP so I can pass that chunk off to a decoder routine for display.
The exact format is not important to me right now - only that I can
identify the frame and pass it off to any generic JPEG decoder. While
awaiting hardware, I want to finish writing my UDP receive thread that
identifies the frame starts and frame ends and begin writing the
decoder/display threads. Trying to manually save what I thought was a
frame as JPG file was my first step. I am no there yet.
Perhaps I would be better of to generate my own M-JPEG video using
your FAQ Q19. That way I would better understanding the encoding
format so I could identify frames. I already have installed DirectX
SDK (November 2008) and Windows SDK for Windows Server 2008 so I guess
I have no excuse. :-)
Thank you very much for all the support you and Roman have generously
shared! It has been very informative!
-Ed
Alessandro Angeli
> There are about 300 bytes extra at the top of the AVI
> file that could be a fixed DHT.
The DHT is *not* in the AVI file at all. Have you read the
text of Q19, including following the link to the Usenet
thread and the comments in the source code?
> The byte values are 90%
> all FF's and I also see a "MJPG" string. Both WMP and
> open source VLC ( using ffmpeg ) played back the video so
> I assume it is not a proprietary security video format.
WMP is not smart at all. If it can play the file, then the
file is a standard AVI file.
> inter-frame compression. I want to be able to identify
> the incoming frames being received over UDP so I can pass
> that chunk off to a decoder routine for display. The
Looking for the JPEG SOI/EOI markers is not a reliable way
to extract the images. You *must* rely on the framing used
by the specific protocol/format you use. Your file uses the
AVI syntax, so you need to parse the AVI following the AVI
rules. When you will be implementing the actual network
protocol, you will have to change the code to support *that*
protocol (and it will be fairly different, including the
fact that network protocol are asynchrous push sources
instead or synchronous pull-mode ones as files are).
> Perhaps I would be better of to generate my own M-JPEG
> video using your FAQ Q19. That way I would better
> understanding the encoding format so I could identify
> frames.
My sample supports the AVI MJPEG format with a few
variations. As Roman told you from the beginning, your real
hardware may produce very different and incompatible MJPEG
data.
Unless you know for a fact that your hardware will produce
MJPEG data that is compatible with the AVI MJPEG format (and
it is not likely), you are wasting your time and you should
instead investigate the exact protocol and format used by
actual hardware. You can not do this in a generic way
because there is no MJPEG standard format or MJPEG standard
transport.