Iphone Serial Decoder

[Skye Severy]

TheiPad/iOS has video streaming support for e.g. H.264 using MPMoviePlayerController etc., but i receive H.264 data through a custom, proprietary, stream and need to decode it in a soft real-time scenario.

Can the iPads/iOS' video decoder be accessed in any way to decode this data?

With iOS 8, you can use video toolbox ( ) to decode H264 to raw frames. VT APIs are hardware accelerated and will provide you much better performance when compared with libavcodec. If you want to play the frames or generate a preview, you can use eagl based renderer to play. I have written a sample app to encode the frames from raw to h.264 ( -h264Hw-Toolbox). h.264 to raw shouldn't be that difficult !

Have you tried writing the H.264 stream that you receive from your protocol to a temporary file which you continually append to, and then once you have written enough bytes to avoid buffering playback, passing the url of your temp file to MPMoviePlayerController?

Just fitted my TT William Whitelaw and the app (android) is just not picking it up in the scan. I know it got power because I can hear the motor buzzing a bit. App is working with other engines. Any tips? Or shall I try fitting it again - maybe a loose connection or something?

I've tried to get the app to detect 2 different HM7000 sound decoders in a class 50 and a class 56, both Hornby, and the app won't detect the decoder, but if I look at the Bluetooth scan for my phone, a Samsung A52S, it shows them as being functional.

Run the troubleshooter as suggested. Scan from within the app will pick up any decoders not allocated to another control device. The scan will say if a decoder needs a reset or can be linked to (not paired as they join a mesh of many). Select each decoder thus found and follow it through.

I have followed the troubleshooting steps, and even set a test track up next to my wifi router and on my Samsung A52S the app wouldn't detect the decoder, however on my Uncle's iPhone it detected the decoders instantly, so I have at least been able to get the sounds set up on the decoders. As an aside I found that to be very simple, great work by the designers.

What I meant by my phone detecting the decoders was, if you go on the Bluetooth connections option on the Settings page within the phone, not the app, which I did to see if my phone was detectable to other devices, the decoders show in the list of bluetooth devices that it has detected in the area.

As a step by step to what happens, on the app, if I press Link Device, then start scan, in the scan results box, there are no decoders, just the headings Ready to Link and Resettable Devices. It's like my phone isn't "seeing" the decoders for want of a better phrase.

Your A52 will never be able to see your decoders whilst they are still linked to your uncle's IOS Phone. You will have to unlink them from the IOS Phone first before trying to link them to your Android. When a decoder is linked to a DCC APP (i.e your uncle's IOS) it is blocked from being seen by any other DCC APP (i.e your Android).

I suggest you go back into your Android and dump any apparent pairing it has set up, then after following the iOS escape advice, follow the standard instructions in the several how to guides on the Hornby site. They will work.

Having got the decoder to pair so easily to the iPhone, I suspect that the A52S is the issue. I notice the previous model, the A51, is shown as being incompatible on Hornby's latest list dated 21/06/23.

An acquaintance of mine having had the issues with the app not detecting the decoder finally found that it is mandatory to enable the Android smartphone's location services at least with the Hornby app. This solved the problem for him.

VideoToolbox has convenient functions for creating descriptions of AVC and HEVC streams (CMVideoFormatDescriptionCreateFromH264ParameterSets and CMVideoFormatDescriptionCreateFromHEVCParameterSets), but not for AV1.

I then got caught out with a decode error because I didn't realise that I also had to pass in the Sequence Header OBU with the first frame data I attempted to decode. It wasn't enough that I had already given the same Sequence Header OBU when creating the video format description (via the extensions).

Decoding itself is slightly simpler than with HEVC, in that you don't need to parse the OBUs, you just pass the data straight to the decoder. With HEVC, you had to parse the NALUs and only pass in slice segments, while also doing some minor conversion of the way the NALU's length is presented to the decoder.

It would be helpful, Apple, if you could consider writing something like CMVideoFormatDescriptionCreateFromAV1SequenceHeaderOBU similar to the existing CMVideoFormatDescriptionCreateFromH264ParameterSets and CMVideoFormatDescriptionCreateFromHEVCParameterSets.

In my scenario, I am in control of the encoder (NVENC) so I get the sequence header OBU when I initialise the encoder, and send this across as my "extra data" before I send any encoded frame data. On the decode side, I then use this in 1 and 2 above. My encoder then doesn't send any further sequence header OBUs at all as in my case the format doesn't change.

In a more general scenario, there might be a sequence header with every IDR frame, so you'd just need to make sure you wait until you get the first sequence header so you can initialise the decoder as per 1. above, and then since the sequence header is already part of the encoded packet data 2. would be taken care of anyway.

Does your decoder callback always receive -12911 or is that an occasional error? I've had issues with occasional and unexplained errors, and the only solution was to wait for (or in my case request) a key frame to get it back on track. It seemed to relate to low bitrate AV1 streams.

Netflix also published a report that showed over the course of one month in early 2022, 21% of their streamed content benefited from the most recent improvements in codec efficiency, like Per-Title optimized AV1 and HEVC. They estimated that without those improvements, total Netflix traffic globally would have been around 24% higher, proving that you can see massive bandwidth and overall cost savings by encoding just a portion of your most popular content with AV1.

A month later in October 2023, Apple announced their newest generation of desktop processors would include AV1 hardware decoders. This includes the M3, M3 Pro and M3 Max chips, meaning all new models of Macbooks, iMacs and desktop computers with an M3 processor will also support AV1 video playback.

Earlier in 2023, while everyone was waiting for Apple to officially support AV1, Meta took matters into their own hands, sharing how they brought AV1 to their Reels videos for Facebook and Instagram, including on iOS devices. This became possible through ongoing open source software decoding efficiency improvements, in particular with the dav1d decoder, developed by VideoLAN. Meta also said they believe for their video products, AV1 is the most viable codec for the coming years. The image below shows how they significantly improved visual quality with AV1 over VP9 and H.264, while keeping the bitrate constant.

While Firefox was the first major browser to support AV1 playback, a long-standing bug (or lack of implementation) prevented DRM-protected AV1 from playing. When Apple added support to Safari for HLS + FairPlay streaming, it meant Firefox was the only major browser that still did not support premium, secure content. That changed in April 2024, when Firefox 125 added AV1 support in encrypted media extensions, meaning Widewine-protected AV1 is now supported.

In early May 2024, Apple continued their march toward full AV1 support with the announcement of their new M4 chip, which will power the new iPad Pro. The Media Engine of M4 is the most advanced to come to iPad, supporting several popular video codecs, like H.264, HEVC, and ProRes, in addition to AV1.

Generally speaking, the Chrome browser and Android ecosystem handle AV1 well across phones, tablets, smart TVs and set-top boxes/streaming sticks. Unfortunately, the same cannot be said for Safari and iOS where support had been lacking until the iPhone 15 Pro announcement.

As mentioned, Android handles AV1 quite nicely, which also applies to the Smart TVs running Android TV and Google TV operating systems. These include Sony Google TV models from 2021 on and many Amazon Fire TV models as far back as 2020. (FireOS is based on Android)

Even with gaps in support on some platforms, there is plenty of opportunity to see tangible bandwidth savings and quality improvements from AV1 right now and thankfully, the future looks even brighter. Intel, AMD, Samsung and Qualcomm have all announced additional AV1 support coming at the chip level.

I am new to the KMM world, coming from Android background.

While I could easily modulize my code and use the kmm for my business logic in a common module, there are still some small things that do not work as nicely as expected.

The easiest thing which popped into my mind was to use the expect and actual. Then it worked fine for Android, but it was not possible to make the actual method for iOS to decode a simple base64 string.

@dalewking:

The Ktor Base64 decoder seems to be an internal Ktor API which I could not use in my KMM project.

It is also marked as deprecated(?) in the docs: -base64.html

It would be amazing if I could find a workaround to use it.

I'm using Vegas Pro 20, trying to edit MOV video files from an iPhone, filmed in HEVC 4K 60fps with HDR enabled (I can post MediaInfo if needed, but I guess iPhones are popular enough that he video file properties are known). The Vegas project settings match the video files, and HDR mode is set to HLG.

Just dropping one of those video files into the timeline, with no editing, no video FX, no transitions, etc., and the preview window set to "Draft (Auto)", I'm getting less than 1 fps playback speed. Disabling Legacy HEVC Decoder slightly improves things and I get about 2 fps, but that's still unusable.

3a8082e126