Latest kaffeine ver 2.0.9 needs a higher version of Linux Mint than 17.x (was DVB-T2 adapter recommendation)

[Johnny B Good]

Following on from my earlier thread "Looking for advice on upgrading to
dual DVB-T2 (Mint 17.1 KDE 64)", I've since discovered that even after a
successful kernel update *and* copying the required fw files into the /
lib/firmware folder, my problem is proving to be the older 1.2.2 version
of Kaffeine not being able to use the HD (T2) muxes. It seems I have no
choice but to upgrade to LM 18.1 (Ubuntu 16.04) in order to install the
latest Kaffeine, version 2.09, in order to be able to watch and record
from the T2 muxes. :-(

It's rather late (or extremely early depending on how you view it - it's
just gone 4:30 in the am as I type this brief post) so I'll follow this
one up after I've had a chance to take stock and experiment with a fresh
install of LM18.1 on a test system.

In case anyone else was thinking of upgrading their Kaffeine based PVR
to DVB-T2 from an existing pre-version 2.09 installation on an Ubuntu
14.04 based distro, it isn't as straight forward as updating the kernel
to 3.19.xx and adding the fw files to the system. That just lets you
watch and record T1 muxes as before, only more expensively. :-(

--
Johnny B Good

[Pinnerite]

Kaffeine 2.x does not support UK HD-T2 channels. see quote dated Feb 2017:

"With regards to UK HG EPG handling, we'll need either some formal
authorization from copyright owners of their Huffman-based encoding or some
way to allow plugin an external parser for it that could be shipped outside
Kaffeine, in order to avoid possible legal issues for distributions."

I have seen no announcements since. :(

Alan

--
Mageia 5.1 for x86_64, Kernel:4.4.65-desktop-1.mga5
KDE version 4.14.5 on an AMD Phenom II X4 Black edition.

[Johnny B Good]

Wow! Thank you very much for that timely warning, Alan!

I've blown away the original LM17.1 on my Intel 180GB SSD that I had
cloned onto the 250GB Samsung EVO SSD about a year ago by way of an
upgrade (now running in my desktop machine) so I could install LM 18.1
onto the existing partitions (keeping the /home partition intact just for
laughs) using my 'Al Fresco' test setup (the innards removed from my
current desktop PC just over two years ago).

I've only gotten as far as installing Kaffeine 2.0.9 but without any DVB-
T adapters plugged into it. The only available PCIe slot is a single lane
job which rather unfortunately is blocked by an obtrusive chipset
heatsink as far as all but short single lane PCIe adapters are concerned.

The Asrock Alive Xfire-eSATA2 MoBo's remaining PCIe slots (both 16 lane)
have been implemented to support Xfire enabled PCIe video adapters
requiring a special MoBo supplied card to be dropped into the other (Xfire
only) PCIe slot whilst using only a single PCIe video adapter.

I was planning on temporarily fitting the LM18.1 afflicted Intel SSD
into my desktop PC for further testing before deciding whether it was
going to be worth the hassle of an otherwise undesired upgrade to the
later LM version cursed with a seemingly worse desktop environment (or at
least one that's going to involve a lot of customisation to make it more
palatable than the rather dark and moody default that's been so unwisely
chosen by the distributors).

BTW, it took a bit of googling to track down that quote (or possibly a
quote of that quote) to a bugzilla forum about an earlier version (2.0.4)
where the problem doesn't appear to be so much as the receiving and
recording of T2 TV channel streams so much as the need to manually
provide filenames as a workaround to not being able to extract such from
the epg.

After an abortive attempt to track down some more information on the
latest stable 2.0.9 version of Kaffeine, I've decided to use my old
single tuner PCI T1 adapter on the test rig to do some actual testing to
get some initial comparative results before swapping the SSD into the
desktop machine to try testing with the T2 tuner. Your timely information
means there's less of a rush to test the T2 adapter so I can spend a lot
more time with the test rig without experiencing the feeling that I'm
*merely* procrastinating before facing a new world of 'Upgrading Pain'.

It looks like I may have been a little premature in making the jump to
DVB-T2 (at least as far as continuing to use Kaffeine is concerned).
Although Kaffeine's development had suffered hiatus over the previous 3
or 4 years, it does look like it's now getting the well deserved
attention required to bring it up to date with the UK's use of T2 to
deliver HD content via the terrestrial UHF TV transmitter network (as
well as SD content from the lesser and 'cheaper' providers).

I'd prefer to avoid "alternatives" to Kaffeine if I can (other than
maybe for a dedicated PVR box) but this might prove to be the only way to
effectively utilise my brand spanking new DVB-T2 adapter in a timely
fashion. I wonder if you have any 'alternatives' in mind that you'd care
to suggest?

--
Johnny B Good

[Johnny B Good]

Well, for anyone interested in this saga with Kaffeine 2.0.9 and the
UK's DVB-T2 HD broadcast system, I did 'borrow' my desktop to do some
testing under LM18.1 with Kaffeine 2.0.9 installed on Sunday.

After copying the firmware files into the /lib/firmware folder and
making up an extra co-ax patch lead so as to connect both aerial sockets
on the adapter, I finally managed to get it working on the T2 muxes. Mind
you, whilst testing with only a single aerial feed connection, Kaffeine
kept dropping out to a blank screen within a second or two of tuning into
an HD mux. Eventually, I got it to work with one of the SD channels long
enough to allow Kaffeine to update its epg list and fill in the massive
gaps in the HD channel epg listings.

It was only after I made up the extra patch lead needed to connect both
aerial sockets to the aerial feed system that I was able to watch any and
all of the HD channels for long enough to fully populate the EPG list
(this version seems to share the same requirement as its predecessor to
allow it play a channel in live view mode for a few minutes in order to
update its epg listings).

The HD listings show the start and run times ok, which is an absolute
minimum to creating a recording schedule. However, the titles and
descriptions are in gobble-dee-gook due to the proprietary huffman
compression algorithm being used by the broadcasters.

As someone else pointed out, this can be gotten round by copying and
pasting from the same web TV schedules listings I had to consult to
choose each programme by its channel and time signature to replace the
filename with something a little more meaningful than the TV Channel name/
date/time coding that I had chosen to test with.

I captured a 48 minute segment of a sports event on BBC2 HD which
produced a 2.4GB m2t file which equates to a surprisingly generous 3GB
per hour transfer rate. This is some 3 times the rate for the iPlayer
sourced 1280 by 720 semi HD downloads! However, using the latest version
of Handbrake (free of the time editing bug that had afflicted the
previous version) to both top and tail and convert a couple of half hour
shows I'd previously downloaded from the iPlayer server as 1280 by 720
mp4s produced MPEG-4 m4v files only 10 to 15 percent bigger.

Unfortunately, whilst this does allow me to trim away the padding and
compress these off-air HD recordings, it's a rather cumbersome system in
that I need to take note, using pencil and paper, of the in and out
points using VLC so I can enter the start and end points in Handbrake
manually prior to adding each recording to the queue for overnight
processing at a slower than real-time pace (just over 15fps with last
night's Horizon programme recorded from BBC4 HD). I've only got to wait a
couple more minutes before I can confirm that I can batch process a bunch
of these files with the same success as I had doing the two previous
files on an individual basis.

Well, what was a 2.4GiB recording (1 /14 hours in total with the
padding) has now become a topped and tailed 1.5GiB 59:08 m4v file. I seem
to have over-run by a couple of seconds but, compared to some of iPlayer's
"Phone Ins", that's actually quite tightly trimmed (not as good, mind
you, as I get when editing Kaffeine's SD m2t files using MpegStreamClip
in the winXP VM to convert them to mpegs before submitting them to
Handbrake for transcoding into .h264 MKV files).

The next file in the list is now 18% the way through (avg 14.25fps), so
it looks like I have some sort of solution to processing the HD
recordings if and when I do finally get Kaffeine up and running on a work-
a-day desktop setup I can live with. I've swapped the SSD's back and
pulled the DVB-T2 card out for the time being in order to run more tests
on that Asrock Alive Xfire-eSATA2 R3.0 MoBo[1] I mentioned earlier.

To summarise: Yes, Kaffeine 2.x.x will work with DVB-T2 and the UK HD
broadcast standard as far as watching 'live' is concerned but you need to
make sure that both aerial sockets are supplied with a signal as well as
making sure to copy the firmware files into the /lib/firmware folder (not
as easy as you might think) and that not only is the kernel at version
3.19 or later but also that the distro version is the latest one if
you're going to rely on a ready made package rather than attempt to
compile Kaffeine 2.x.x from source to run under an older distro version.

Once you've gone through the necessary hoops, you'll discover that you
*can* use the epg to schedule recordings. However, you'll need to look at
the broadcasters' on-line TV schedules to pick the appropriate epg entry
by its time signature, using the same source to copy and paste a
meaningful filename after the recording has completed. If you choose
channel/date/time as the default naming format, this should avoid losing
track of what recording is what when you come to rename the files later
on.

It's actually pretty well what I had to do when I was using DTVR in win2k
up until just over two years ago. Back in those dark days, when none of
the supplied software which claimed to populate an epg listing over the
air could do so reliably, I had to make do with date/time named files
(any rare conflicts being resolved with -1, -2 being appended to the
name). I didn't have the luxury of a TV channel name to help. It was just
a date/time coded name on each file.

Furthermore, I had to manually add each recording to the schedule by
arbitrary assigned channel number and start time and programme length,
compensating for the start and end padding by calculating the appropriate
duration and actual start time. I could usually do this as an exercise in
mental arithmetic but it could become rather fraught if I was scheduling
several back to back programmes (only on any given channel) since I had
to calculate the total run time to record them as one single large file
for splitting up later (padding conflicts simply weren't allowed with any
of these recording programs).

As much of a dog's dinner as Kaffeine has now become in the matter of
dealing with the scheduling of recordings from the T2 HD muxes, it's
still a vast improvement over what the windows users had to (possibly
still do) face when trying to schedule Freeview recordings.

Even so, I'm beginning to wonder what exactly I have let myself in for.
On the face of it, upgrading to a T2 tuner seemed to be a fairly straight
forward exercise but I'm beginning to have doubts about the whole
business, especially in view of the need to upgrade the OS and the extra
system workload of processing the recordings into nicely topped and
tailed compressed video files.

This upgrading exercise doesn't seem to be going quite the way I was
expecting. Never mind, "Onward and downward" seems to be the way I'm
headed for the moment. Perhaps I'll discover a happy resolution in the
end.


[1] You'll remember that I had a PCIe slot issue with this MoBo.
According to the user guide supplied with the board, the supplied "PCIe
Switchcard" *had* to be inserted into the the PCIe2 slot when using only
a single PCIe VGA adapter plugged into PCIe1, leaving only the short
single lane PCIe3 slot available for short single lane adapters.

Since I couldn't use use either of the full length slots to plug the DVB-
T2 adapter into, according the requirements laid out for using a single
PCIe VGA adapter, I decided to use an old NVidia 32MB PCI adapter so I
could free up both PCIe slots to allow me to use the DVB-T2 adapter with
that board to carry on with some more testing.

I had some rather odd results booting with the recently installed LM18.1
on the 180GB Intel SSD. The desktop didn't materialise but the Kaffeine
window appeared ok as a result of being configured to auto-run at each
boot up. The mouse pointer worked so I was able to start the Live TV ok
but nothing else would work after closing Kaffeine.

I decided to blow away the misconfigured LM18.1 install and start over
again (retaining my home partition as before). I got it working again but
the response to resizing windows was so sluggish with the PCI adapter, I
decided to try and track down some more info on exactly what that mystery
"PCIe SwitchCard" was doing or was meant to do.

After searching on various phrases, I finally discovered what damn well
ought to have been fully explained in the user guide. It seems that all
that happens if you *don't* install the switchcard, is that slots 1 and 2
behave as 8 lane slots. The switchcard somehow makes slot 1 behave as a
16 lane slot when placed in slot2. That's it! Nothing worse than a drop
in performance if a 12 or 16 lane PCIe VGA adapter is used on its
lonesome in slot 1 (or 2).

I rather doubt the old Nvidia PCIe adapter I was using to begin with
even has a full 16 lane interface capability. Anyway, I've pulled the
switchcard out and refitted the PCIe Nvidia adapter with the DVB-T2
adapter sitting alongside where the switchcard used to be so I am ready
to do some more testing just as soon as I've made up yet another pair of
co-ax patchleads to let me feed the twin tuner inputs from yet another
one of my many 'spare' ex-NTL 3 way splitters. I just don't need the
extra grief that results from feeding only one of the aerial sockets with
a signal so until I've got that sorted out, there's no point in going off
half cocked with my subsequent testing.

--
Johnny B Good

[Pinnerite]

Wow! What a comprehensive report. I am still using version 1.3-git but I
also have Mythtv on a server. On balance I think I'll stay put for the
foreseeable.

Regards, Alan

[Andy Furniss]

Johnny B Good wrote:

> The HD listings show the start and run times ok, which is an
> absolute minimum to creating a recording schedule. However, the
> titles and descriptions are in gobble-dee-gook due to the proprietary
> huffman compression algorithm being used by the broadcasters.

It's strange the kaffine is "scared" over this.
I use tvheadend.org and the EPG for HD channels is handled OK.
Some years ago I read that "decrypting" then was easy as it was the
status quo for dvb-s channels already. I don't know how/if kaffine
handles those.

> I captured a 48 minute segment of a sports event on BBC2 HD which
> produced a 2.4GB m2t file which equates to a surprisingly generous
> 3GB per hour transfer rate.

7.2 mbit/s for 1080i25 (sometimes called 1080i50) sport in not what I
would call generous! Though being variable, it will peak a bit higher
when needed.

> This is some 3 times the rate for the iPlayer

I think the BBC do/did 5 mbit/s 720p50 for sport, possibly CBR rather
than VBR though.

> sourced 1280 by 720 semi HD downloads!

If made from a 1080p50 master 720p has more vertical res than 1080i25.
UK HD does also do 1080p25 for low action stuff - so that should be
better (but all UK HD looks soft to me).

> However, using the latest version of Handbrake (free of the time
> editing bug that had afflicted the previous version) to both top and
> tail and convert a couple of half hour shows I'd previously
> downloaded from the iPlayer server as 1280 by 720 mp4s produced
> MPEG-4 m4v files only 10 to 15 percent bigger.
>
> Unfortunately, whilst this does allow me to trim away the padding
> and compress these off-air HD recordings,

Compress! They've already been murdered, you want to kill them more :-)

> it's a rather cumbersome system in that I need to take note, using
> pencil and paper, of the in and out points using VLC so I can enter
> the start and end points in Handbrake manually prior to adding each
> recording to the queue for overnight processing at a slower than
> real-time pace (just over 15fps with last night's Horizon programme
> recorded from BBC4 HD).

I guess from the framerate that it's x264 not x265 (re-compressing using
something better could at least be justified!), though you'd probably be
looking at 0.5 fps or less. Saying that what framerate you get is very
controllable anyway lower being "better" usually - you are just getting
whatever handbrake chose as libx264 params for you.

I don't know how clever handbrake is as a front end for ffmpeg/libx264,
but there are several pitfalls waiting to lower quality vs efficiency.

Interlaced content being the first - it may be that handbrake doesn't
handle this well = assuming progressive, which will trash quality and
make de-interlacing harder. It may also see interlaced and de-interlace
- which may throw away temporal res (it doesn't have to). Of course a
lot of stuff is progressive anyway (apart from the credits). Some DVB-T2
HD is mixed and flagged as such, so it's not always easy for processing
software to adjust on the fly. Sport may start progressive for a the
studio intro and then switch to interlaced for the action, then back.

Gotcha 2 = sound, on DVB-T2 aac in latm is used and the channels may
flip between 2ch and 5.1, this confuses vlc and may also confuse encodes
to containers. There is also some meta data mainly on 5.1 movie content
to do with dynamic range that could be lost if the sound gets re-coded
along with the vid. Currently you can't handle it anyway with ffmpeg
based players - but in future it may be possible.

Your topping and tailing + ad cutting, if tight and before re-coding may
somewhat save you from the sound channel layout issues, but it's
something to look out for if you start recording movies/proms etc that
are 5.1.

[Johnny B Good]

On Tue, 20 Jun 2017 13:45:11 +0100, Pinnerite wrote:

> Johnny B Good wrote:
>
>> On Thu, 15 Jun 2017 16:20:53 +0000, Johnny B Good wrote:
>>
>>> On Thu, 15 Jun 2017 11:48:54 +0100, Pinnerite wrote:
>>>
>>>> Johnny B Good wrote:
>>>>

====huge snip of stuff wot I wrote====

>>
>> To summarise: Yes, Kaffeine 2.x.x will work with DVB-T2 and the UK HD
>> broadcast standard as far as watching 'live' is concerned but you need
>> to make sure that both aerial sockets are supplied with a signal as
>> well as making sure to copy the firmware files into the /lib/firmware
>> folder (not as easy as you might think) and that not only is the kernel
>> at version 3.19 or later but also that the distro version is the latest
>> one if you're going to rely on a ready made package rather than attempt
>> to compile Kaffeine 2.x.x from source to run under an older distro
>> version.
>>
>> Once you've gone through the necessary hoops, you'll discover that you
>> *can* use the epg to schedule recordings. However, you'll need to look
>> at the broadcasters' on-line TV schedules to pick the appropriate epg
>> entry by its time signature, using the same source to copy and paste a
>> meaningful filename after the recording has completed. If you choose
>> channel/date/time as the default naming format, this should avoid
>> losing track of what recording is what when you come to rename the
>> files later on.
>>

====more snippage====

>>
>> As much of a dog's dinner as Kaffeine has now become in the matter of
>> dealing with the scheduling of recordings from the T2 HD muxes, it's
>> still a vast improvement over what the windows users had to (possibly
>> still do) face when trying to schedule Freeview recordings.
>>
>> Even so, I'm beginning to wonder what exactly I have let myself in
>> for.
>> On the face of it, upgrading to a T2 tuner seemed to be a fairly
>> straight forward exercise but I'm beginning to have doubts about the
>> whole business, especially in view of the need to upgrade the OS and
>> the extra system workload of processing the recordings into nicely
>> topped and tailed compressed video files.
>>
>> This upgrading exercise doesn't seem to be going quite the way I was
>> expecting. Never mind, "Onward and downward" seems to be the way I'm
>> headed for the moment. Perhaps I'll discover a happy resolution in the
>> end.
>>

=====yet more snippage====

>>
> Wow! What a comprehensive report. I am still using version 1.3-git but I
> also have Mythtv on a server. On balance I think I'll stay put for the
> foreseeable.
>

I'm sorry about having to paint such a bleak picture with regard to the
latest Kaffeine and UK T2 broadcast standards. I can understand your
reaction, especially in the complete absence of any feedback or input
from other more experienced users whom I had hoped could have "put me
straight on my misconceptions".

The only consolation in all of this is that if I, a mere mortal, could
manage to get this far (setup a system where I can schedule HD recordings
via the epg using the latest version of Kaffeine), then almost anyone
else capable of installing a Unix based distro onto a desktop PC should
be able to do likewise if they're prepared to make the effort, imho. :-)

Since I appear to be the only "expert" on the subject (for the time
being at any rate) all I can do is to try and soldier on in the hope that
I can fathom out a workable solution to the task of recording and
archiving Freeview HD programmes which may well lead me to ditch Kaffeine
if I can track down something better. However, a quick google rather
suggests I'll be better off sticking with the "Devil I Know" and hope a
UK T2 EPG plugin becomes available in the not too distant future.

I've now got the testing hardware configured to let me run more
experiments with which should keep me busy for a while. I'll post
another, hopefully more succinct, update on my progress and experiences
when I feel I've got something worthwhile to report.

--
Johnny B Good

[Johnny B Good]

On Tue, 20 Jun 2017 19:50:43 +0100, Andy Furniss wrote:

> Johnny B Good wrote:
>
>> The HD listings show the start and run times ok, which is an absolute
>> minimum to creating a recording schedule. However, the titles and
>> descriptions are in gobble-dee-gook due to the proprietary huffman
>> compression algorithm being used by the broadcasters.
>

> It's strange that kaffine is "scared" over this.

I have to say I'm also disappointed at this lack of 'bravery'. Still, I
guess there must be a 'legal issue' over this so I suppose it must be a
case of "Discretion being the better part of valour." principle at work
here. If the hinted at 'plugin' solution materialises as an end user
option, that should to resolve the conflict.

> I use tvheadend.org and the EPG for HD channels is handled OK.
> Some years ago I read that "decrypting" then was easy as it was the
> status quo for dvb-s channels already. I don't know how/if kaffine
> handles those.

Nor do I. I've only ever used Kaffeine with DVB-T adapters until now.

>
>> I captured a 48 minute segment of a sports event on BBC2 HD which
>> produced a 2.4GB m2t file which equates to a surprisingly generous 3GB
>> per hour transfer rate.
>

> 7.2 mbit/s for 1080i25 (sometimes called 1080i50) sport is not what I

> would call generous! Though being variable, it will peak a bit higher
> when needed.
>
>> This is some 3 times the rate for the iPlayer
>
> I think the BBC do/did 5 mbit/s 720p50 for sport, possibly CBR rather
> than VBR though.
>
>> sourced 1280 by 720 semi HD downloads!
>
> If made from a 1080p50 master 720p has more vertical res than 1080i25.
> UK HD does also do 1080p25 for low action stuff - so that should be
> better (but all UK HD looks soft to me).
>
>> However, using the latest version of Handbrake (free of the time
>> editing bug that had afflicted the previous version) to both top and
>> tail and convert a couple of half hour shows I'd previously downloaded
>> from the iPlayer server as 1280 by 720 mp4s produced MPEG-4 m4v files
>> only 10 to 15 percent bigger.
>>
>> Unfortunately, whilst this does allow me to trim away the padding and
>> compress these off-air HD recordings,
>
> Compress! They've already been murdered, you want to kill them more :-)

Quite honestly? No, not really. All I really want to do is top and tail
the surplus paddings out and convert from the TS format to the
appropriate PS format just as I was doing with the SD m2t transport
stream files when 'converting' them to MPG programme stream files with
MpegStreamClip in my winXP VM (which I was then able to submit to
Handbrake as ready to transcode MPG files with no need to specify in and
out points).

>
>> it's a rather cumbersome system in that I need to take note, using
>> pencil and paper, of the in and out points using VLC so I can enter the
>> start and end points in Handbrake manually prior to adding each
>> recording to the queue for overnight processing at a slower than
>> real-time pace (just over 15fps with last night's Horizon programme
>> recorded from BBC4 HD).
>
> I guess from the framerate that it's x264 not x265 (re-compressing using
> something better could at least be justified!), though you'd probably be
> looking at 0.5 fps or less. Saying that what framerate you get is very
> controllable anyway lower being "better" usually - you are just getting
> whatever handbrake chose as libx264 params for you.

I'm no expert on what is a very complex subject, video format standards
and codecs, so tend to leave the presets well alone (other than perhaps
to select a de-noise filtering option and/or manually crop older 4:3 and
extra-wide-screen aspect ratio material when the BBC4's DOG has confused
the shit out of Handbrake's auto-cropping feature).

>
> I don't know how clever handbrake is as a front end for ffmpeg/libx264,
> but there are several pitfalls waiting to lower quality vs efficiency.
>
> Interlaced content being the first - it may be that handbrake doesn't
> handle this well = assuming progressive, which will trash quality and
> make de-interlacing harder. It may also see interlaced and de-interlace
> - which may throw away temporal res (it doesn't have to). Of course a
> lot of stuff is progressive anyway (apart from the credits). Some DVB-T2
> HD is mixed and flagged as such, so it's not always easy for processing
> software to adjust on the fly. Sport may start progressive for a the
> studio intro and then switch to interlaced for the action, then back.
>
> Gotcha 2 = sound, on DVB-T2 aac in latm is used and the channels may
> flip between 2ch and 5.1, this confuses vlc and may also confuse encodes
> to containers. There is also some meta data mainly on 5.1 movie content
> to do with dynamic range that could be lost if the sound gets re-coded
> along with the vid. Currently you can't handle it anyway with ffmpeg
> based players - but in future it may be possible.
>
> Your topping and tailing + ad cutting, if tight and before re-coding may
> somewhat save you from the sound channel layout issues, but it's
> something to look out for if you start recording movies/proms etc that
> are 5.1.

Ok Andy, thanks for that advice. It seems I need a simple .h264
transport to program stream converter/editor along the lines of
MpegStreamClip or maybe something even cruder such as MpegCut2 to simply
trim off the surplus end and start paddings. You wouldn't happen to have
any particular video editing software you'd care to recommend by any
chance?

--
Johnny B Good

[Andy Furniss]

Johnny B Good wrote:
> On Tue, 20 Jun 2017 19:50:43 +0100, Andy Furniss wrote:

>> Compress! They've already been murdered, you want to kill them more
>> :-)
>
> Quite honestly? No, not really. All I really want to do is top and
> tail the surplus paddings out and convert from the TS format to the
> appropriate PS format just as I was doing with the SD m2t transport
> stream files when 'converting' them to MPG programme stream files
> with MpegStreamClip in my winXP VM (which I was then able to submit
> to Handbrake as ready to transcode MPG files with no need to specify
> in and out points).

OK, with SD mpeg2 + post processing to h264 isn't totally pointless,
though the interlaced issue still exists.

With >= h264 post processing isn't needed as it's built in, so there
isn't really any gain in encoding again to h264.

h265 maybe, but then it doesn't really handle interlaced "properly" like
h264 does (or can assuming the right params are used).

> I'm no expert on what is a very complex subject, video format
> standards and codecs, so tend to leave the presets well alone (other
> than perhaps to select a de-noise filtering option and/or manually
> crop older 4:3 and extra-wide-screen aspect ratio material when the
> BBC4's DOG has confused the shit out of Handbrake's auto-cropping
> feature).

Yea - it's not simple at all, you really need to take account of the
source and ffmpeg/libx264 with or without a frontend are complex.

> Ok Andy, thanks for that advice. It seems I need a simple .h264
> transport to program stream converter/editor along the lines of
> MpegStreamClip or maybe something even cruder such as MpegCut2 to
> simply trim off the surplus end and start paddings. You wouldn't
> happen to have any particular video editing software you'd care to
> recommend by any chance?

Unfortunately as I don't usually edit anything I record, I can't help -
I just leave as is and skip forward with the player (not vlc for HD).

With transport streams you can just cut the files with dd using a
blocksize of 188 - not exactly user friendly!

Another way so you could at least use times is getting ffmpeg to cut but
not encode again using -c copy. This only works if you want another
transport stream (mkv should work but currently doesn't) other
containers may/will not like the sound stream, as UK DVB is a bit
peculiar in using aac in latm.

If you don't care about audio description or subs you could seek and
specify a time like

ffmpeg -i infile -c copy -ss 2:35 -t 1:05:10 outfile.ts

would seek to 2 min 35 and output the next 1 hour 5 minutes and 10
seconds. It would remux to a new transport stream. If you wanted to keep
the ad and subs more would be needed.

[Johnny B Good]

On Fri, 23 Jun 2017 00:27:43 +0100, Andy Furniss wrote:

> Johnny B Good wrote:
>> On Tue, 20 Jun 2017 19:50:43 +0100, Andy Furniss wrote:
>
>>> Compress! They've already been murdered, you want to kill them more
>>> :-)
>>
>> Quite honestly? No, not really. All I really want to do is top and tail
>> the surplus paddings out and convert from the TS format to the
>> appropriate PS format just as I was doing with the SD m2t transport
>> stream files when 'converting' them to MPG programme stream files with
>> MpegStreamClip in my winXP VM (which I was then able to submit to
>> Handbrake as ready to transcode MPG files with no need to specify in
>> and out points).
>
> OK, with SD mpeg2 + post processing to h264 isn't totally pointless,
> though the interlaced issue still exists.

Be that as it may, using Handbrake to transcode my mpeg2 recordings into
more compact .h264 mkv files with no perceptible drop in quality has been
a Ghodsend in saving valuable disk space on my 17TB fileserver (a
homebrewed NAS4Free box)

>
> With >= h264 post processing isn't needed as it's built in, so there
> isn't really any gain in encoding again to h264.

I had hoped that Handbrake had been written to intelligently avoid
unnecessary transcoding just as MpegStreamClip does when cleaning up the
so-called MPG output files generated by DTVR's "record as mpeg2" option
back in the days of recording Freeview broadcasts in windows 2000.

>
> h265 maybe, but then it doesn't really handle interlaced "properly" like
> h264 does (or can assuming the right params are used).

I don't really want to push encoding any further than the extreme
of .h264. A simple topping and tailing .h264 editor will suit me just
fine, thank you very much. :-)

>
>> I'm no expert on what is a very complex subject, video format standards
>> and codecs, so tend to leave the presets well alone (other than perhaps
>> to select a de-noise filtering option and/or manually crop older 4:3
>> and extra-wide-screen aspect ratio material when the BBC4's DOG has
>> confused the shit out of Handbrake's auto-cropping feature).
>
> Yea - it's not simple at all, you really need to take account of the
> source and ffmpeg/libx264 with or without a frontend are complex.
>
>> Ok Andy, thanks for that advice. It seems I need a simple .h264
>> transport to program stream converter/editor along the lines of
>> MpegStreamClip or maybe something even cruder such as MpegCut2 to
>> simply trim off the surplus end and start paddings. You wouldn't happen
>> to have any particular video editing software you'd care to recommend
>> by any chance?
>
> Unfortunately as I don't usually edit anything I record, I can't help -
> I just leave as is and skip forward with the player (not vlc for HD).
>
> With transport streams you can just cut the files with dd using a
> blocksize of 188 - not exactly user friendly!

It might be a better option that'll allow me to revert back to the
earlier version of Handbrake which I've just realised had been processing
the "Shaun the Sheep" episodes twice as fast as the latest version is
doing. :-( I'm not happy with the changes in the GUIs of *every* new
version of Linux related 'upgrades' (apps and desktop environments alike).

>
> Another way so you could at least use times is getting ffmpeg to cut but
> not encode again using -c copy. This only works if you want another
> transport stream (mkv should work but currently doesn't) other
> containers may/will not like the sound stream, as UK DVB is a bit
> peculiar in using aac in latm.
>
> If you don't care about audio description or subs you could seek and
> specify a time like
>
> ffmpeg -i infile -c copy -ss 2:35 -t 1:05:10 outfile.ts
>
> would seek to 2 min 35 and output the next 1 hour 5 minutes and 10
> seconds. It would remux to a new transport stream. If you wanted to keep
> the ad and subs more would be needed.

This need to examine each file with VLC or another media player with 'to
the second' seek accuracy just to manually take note of the in and out
points so the same can be used with ffmpeg's CLI or, as I've tried,
setting these points in the hours/minutes selection boxes in Handbrake is
what made use of a windows XP virtual machine to run MpegStreamClip such
an attractive option [1].

Unfortunately, I can't employ the programme start flags used by the
broadcasters to achieve an "Exact Record" function using Kaffeine. As far
as I know, the only option I have is Kaffeine's excellent conflict free
padding mechanism to overcome the broadcasters' "Scheduling Twattery"(tm)
[2].

Up until I started experimenting with DVB-T2 HD broadcasts, I've been
able to take full advantage of this conflict free padding using
MpegStreamClip to process Kaffeine's m2t mpeg transport stream files into
mpeg programme stream files (or even re-edit mpeg PS files into
differently sized mpeg PS files with no further processing loss other
than for the "on the fly converted to PS" mpg recordings made by DTVR
which proved to be inflated by 0.14% of, presumably FEC leftovers which
effected the earlier versions of windows media player's seek bar
functionality.

In this case, if I wasn't sure whether I'd already 'converted' such mpg
recordings, I simply ran them through MpegStreamClip to see whether the
output remained the same size or landed up shrunk by that signature
0.14%. If they didn't shrink, I knew they'd already been processed so
could choose to keep either file otherwise the output file was "The
Keeper" and I could discard the original. It was that simple. I could
safely re-process an mpg file as much as I liked without MpegStreamClip
"Fixing it some more" (in the American sense of 'breaking it some more'
derived from the principle of "If it ain't broke, don't fix it!").

MpegStreamClip was downloaded primarily to let me convert the Toppy's
*.rec files (mpg TS recordings) into proper mpg PS media files. Not only
did it prove itself as an efficient 'converter' of TS to PS files, it
also proved to be a better editor than the very basic "Mpg2Cut2" 'slice
and dice' editor I'd previously used for topping and tailing my padded
recordings eventually revealing the reason as to why DTVR's "MPG" files
could be so troublesome to navigate by completely curing the problem.
Unfortunately, its developer never created a Linux port so it remains
purely a win32 compatible application, hence the use of VirtualBox and a
winXP VM to run it in.

I've installed the DVB-T2 adapter into the Al Fresco test rig[3] but,
in spite of a second fresh install of LM18.1 and Kaffeine 2.0.9 I can't
quite emulate the initial success I had testing on the current desktop
hardware using the LM18.1 installed to the 180GB Intel series 300 SSD
whilst it had been connected to the Al Fresco test rig.

The 'deal breaker' is that the whole machine locks up (not just the
Kaffeine app alone) after viewing SD or HD material 'Live' for more than
half a minute to a maximum, so far, of 5 minutes. I've tried all the
usual hardware checks (I know better than to assume that *all* desktop
computer problems will be purely software issues alone) to no avail.

There's a problem which didn't show up when I was testing on the current
desktop hardware suggesting that I need to commit to a distro upgrade on
this desktop machine before I can make any further progress. I was rather
hoping to do a lot more testing before taking this 'final step'. :-(

[1] With SD recordings, every so often (2 or 3 times a day), I'd move
Kaffeine's *.m2t files from the root of my home folder (where I could
readily keep an eye on them) into the "TV Recordings (Kaffeine)" folder
which was read/write shareable in the winXP VM.

I could then maximise the VM window and move said files into one of the
two 100GB virtual disks (one per physical HDD on the system) set up to
provide much higher read/write performance than the VBox shares could
ever offer so that the following MpegStreamClip processing could run some
20 to 25 times quicker (8 seconds per 29 minute SD file versus 3 1/2
minutes when I'd been using the VBox shares directly).

Yes, there was the extra "Faff" and the 5 to 15 minutes waiting for
files to be moved from the VBox share into the working folder on the
virtual disk but this investment of 'time and effort' was more than amply
rewarded by the 20/25 fold boost in speed during the "Hunched over the
keyboard" actual processing by MpegStreamClip periods.

The bulk moving of files between VBox shares and virtual disks was a
"Fire and forget" operation that left me free to do other more important
things than merely sit at a console awaiting the moment I could continue
with the next stage of the operation.

[2] Back in the early days of DTVR under win2k, I used padding rules
tuned to each BBC channel (DTVR didn't provide a "Global Padding" option
- I had to calculate them on a case by case basis). The magnitude order
in which such scheduling twattery was applied to each channel went BBC1,
BBC2, BBC4/BBC3. That is to say I needed 3+7 minutes for a BBC1
programme, 2+6 minutes for BBC2 and 1+4 minutes for the "Digital Only"
BBC channels. Apart from "unexpected sports" and news events, it was rare
to lose the start or end of a programme due to 'scheduling slippage',
nevertheless, it did happen on occasion.

When I started using Kaffeine which allowed me to pick directly from the
epg rather than manually program each channel/time/duration into its
recording schedule, I made good use of the global padding feature,
initially going for a 3+7 minute padding rule until BBC1 manage to break
the 3 minute start padding (or possibly the 7, in practice 8, minute end
padding) when I reset it to the 5+10 minutes of padding I'm using today.

In fact this is so much padding that the 3 back to back recordings of
"Shaun the Sheep", "Shaun the Sheep" and "Strange Hill High" being
recorded earlier this afternoon on CBBC, I happened to notice were all
being streamed to the disk simultaneously. Not too surprising a result
when the 15 minutes of padding happens to be 3 times the 'notional' 5
minute scheduling slot time interval (7 minutes in reality) of the "Shaun
the Sheep" episodes.

Provided there's a simple means of topping and tailing to the nearest GoP
(1/2 second in the case of BBC SD material) when archiving Freeview aired
programmes, even outrageous paddings (say 10 + 30 minutes) can be used
with impunity thanks to Kaffein'e padding agnostic character. Given
enough disk storage space, you can use a "Shoot First, Ask Questions
Later." recording strategy (it's easy enough to delete, what upon later
inspection proves to be, a rubbish media file yet so impossible to undo
"neglecting to schedule a potentially interesting programme" that has
passed you by).

Unfortunately, topping and tailing such paddings on the T2 HD programme
material isn't going to be so pleasant a task unless I can find an HD
version of MpegStreamClip (whether a win32 or a Linux compatible app it
matters very little - but preferably a Linux utility if its UI isn't too
terribly broken).

[3] The major problem with Al Fresco testing is the lack of backplate
support for any adapter cards in the expansion slots. Not a major problem
with the old ISA boards (now non-existent on any MoBo less than a couple
of decades old). Even PCI and full length PCIe adapters are reasonably
stable with regard to edge connector contact reliability but when dealing
with a single lane PCIe DVB-T2 adapter with a couple of CT100 cables
hanging off its rear panel Belling Lee aerial sockets, there's an even
greater risk of causing a system crash if any of the connections are
disturbed. However, I don't think it was this that was causing the lockups
in this case.

I've just fired the test rig up again. Live view of BBC2 SD crashed
after 100 seconds. Rebooted and applied the AMD cpu micro-code update and
rebooted ok. I clicked the TV icon in Kaffeine (it's run each time at
boot up from the startup script) and then stopped the live play and
picked the current "Horrible Histories" on CBBC HD to add to the schedule
and monitored the growth of the m2t file until it stopped after just 5
minutes with the whole machine locked up (which result does offer the
consolation that it wasn't something missed in recording HD content over
a 7 hour period using the desktop hardware just because I didn't happen
to try watching Kaffeine's Live TV window for more than a minute or so).

Further power cycled reboots later revealed that Kaffeine would resume
the interrupted scheduled recording ok. The subsequent resumption crashed
after only about half a minute whilst on the second resumption with only
just on a minute left, it managed to finish the scheduled recording
without crashing.

The problem with the test rig does rather suggest a driver issue. I'd
like to try transferring the adapter to the single lane PCIe slot but the
Chipset heat sink is currently fouling all but short single lane
adapters. I'm rather tempted to trim 6mm off the fouling end of that
heatsink so as to complete a "No Stone Unturned" hardware test before
being forced to update to LM18.1 on my desktop machine.

However, there *is* one more thing left for me to try and that is the
kernel update to ver 4.4.0-81.104 (a level 5 update in Update Manager
which remains currently unselected). I notice there are three other
updates which I'm now applying. If one *is* going to chance a kernel
update, this is the best time to try it, on a system that's only being
used for testing rather than a 'production' system that needs to carry on
working without missing a beat.

Well, I've just updated the kernel and Kaffeine locked the system after
only about half a minute of watching BBC2 SD so that didn't help any. I
guess the only remaining option is the hacksaw job on that pesky heatsink!

--
Johnny B Good

[Johnny B Good]

On Sat, 24 Jun 2017 10:20:06 +0100, Chronos wrote:

> On Fri, 23 Jun 2017 19:50:51 GMT Johnny B Good
> <johnny...@invalid.ntlworld.com> wrote:
>
>> A simple topping and tailing .h264 editor will suit me just fine, thank
>> you very much. :-)
>

> I generally deal with this problem by noting down the start and ending
> time of the wanted content from VLC and using ffmpeg in copy mode with a
> fixed skip and length to get rid of the leader and trailer. It has the
> advantage of not pulling in heaps of [qt|gtk] rubbish that I don't need
> for anything else and is *fast*. Most other non-linear editors seem to
> want to re-encode the entire stream. Even with VCE or QuickSync, this is
> a ballache.

That's a much more attractive alternative to my using Handbrake to trim
the leading/trailing paddings after using VLC to determine the in and out
points. My only question is, "Does this just leave me with a 'nicely
trimmed' m2t TS file or can I persuade ffmpeg to 'convert it' to
something like an mp4 or, as I landed up doing with Handbrake, an m4v?".

Looking at get_iplayer's activity when it finishes a download and
invokes ffmpeg to convert the resulting *.m2t TS file into an mp4, I'm
guessing the answer would be yes, depending on the command line switches
used.

Whilst I'm responding to this follow up, I think it's worth mentioning
that I found a less 'destructive of chipset heatsink' temporary solution
to testing with the adapter in the short single lane PCIe slot (the
problem described in [note 3] which you may, understandably, not have
bothered reading in full the first time round). My apologies for such
long and rambling posts but I'm just trying to pre-empt as many questions
about my skills and experience so that any follow ups can be as
mercifully brief and to the point as yours with judicious use of
quotage. :-)

Anyhow, I found a suitable 'drop in' low profile replacement heatsink in
my spares box which I was able to substitute for the original "AMD"
labelled heatsink. Initial tests looked promising but I hit the same
problem 5 minutes in rather than 60 to 90 seconds in. I doubt it was
anything to do with increased chipset temperature due to the less
efficient heatsink even though I wouldn't care to rely on this as a
permanent solution (If I wanted a permanent solution to freeing up that
PCIe slot for normal length adapters, I'd simply modify the original AMD
one with a hacksaw as originally threatened).

It looks like I've gone as far as I can with my Al Fresco test rig and
HD Freeview recording tests. However, that doesn't stop me from tweaking
the system settings to better match my existing preferences to reduce
"System Shock" when I finally do upgrade the distro on my desktop PC to
match Kaffeine 2.0.9's requirements.

Incidentally, is it a good idea to apply an outstanding kernel update on
a freshly installed Linux Mint 18.1 when doing the post installation
updates? My own thought on this, despite the warning of possible dire
consequences[01], is that getting this out of the way before investing
any more time and effort to get the system set up to my specific
requirements would be the better option.

I'm thinking that if it all goes "Pete Tong" at that stage, I'd only be
faced, at worst, with repeating another 'clean install' (just "a mere 20
minutes or so of my life that I won't be getting back" rather a day or
three spent trying to get everything back to some semblance of its former
glory).

[01] I suspect I may be misinterpreting the circumstances under which
such 'dire consequences' can arise *after* completing a kernel upgrade. I
may be wrong but it seems to be more to do with reverting applications
back to earlier versions which may not be compatible with the latest
kernel upgrade. If this is what the warning is all about, then it makes
sense to apply a kernel upgrade straight away before installing any
applications which might fall into this trap.

--
Johnny B Good

[Andy Furniss]

Johnny B Good wrote:

> Looking at get_iplayer's activity when it finishes a download and
> invokes ffmpeg to convert the resulting *.m2t TS file into an mp4,
> I'm guessing the answer would be yes, depending on the command line
> switches used.

Probably not using just copy as DVB-T HD channel AAC is in LATM and
ffmpeg is lacking in LATM handling and AFAIK MP4 container won't like
native LATM. It (fmpeg) will "handle" LATM if you re-code.

IIRC a standalone LATM demuxer was on ffmpegs GSOC list - so maybe it
will/has changed.

mkv should work - but ffmpeg is currently buggy copying h264 from ts
into that, testing avconv from the libav fork it seems that this now works.

[Johnny B Good]

That doesn't sound at all encouraging. :-(

Low-overhead MPEG-4 Audio Transport Multiplex is what I found when
googling "LATM". Also, I came across a 5 year old thread by 4candles on a
problem trying to stream BBC HD material (a 30 second 1440 by 1080i
example DOGged with "BBC ONE HD" - presumably a satellite broadcast - All
the freeview BBC HD broadcasts I recorded a week ago had been full HD
with no DOGs on the BBC ONE and BBC TWO broadcasts).

I read through most of the thread before it occurred to me that, at 5
years old, it was probably a total waste of my time to review what in all
probability was by now an already solved issue. It seems I'll have
enough, more contemporary issues, to worry about without being distracted
by (what I sincerely hope is by now) 'Ancient History'.

I do have a couple of options when it comes to cleaning up the raw off-
air recordings, the already mentioned Handbrake and Kdenlive but I'm not
a happy bunny when it comes to using either of them. However, in
retrospect, I think Kdenlive might have the edge (just!) over Handbrake
when it comes to the basic task of topping and tailing the lead in and
out paddings. I'll have to perform additional tests to decide this. I'm
hoping Kdenlive wins out since I can revert back to the earlier Handbrake
which was easier to use and able to transcode twice as fast.

In the meantime, it looks like I'm going to have to commit to upgrading
my current LM17.1 KDE to LM18.1 KDE in order to start making productive
use of that DVB-T2 adapter that I spent all that money on nearly three
weeks ago. I'm thinking of using a compromise strategy whereby most of
the material will consist of iplayer half HD downloads (1280 by 720)
supplemented by off-air Full HD (1920 by 1080) recordings when iplayer
either totally fails to supply the goods or else 'the goods' consist of
low quality 'Phone Ins' that even a shit broadcaster like QVC would be
ashamed of.

First things first... Now, where's that bootable disk cloning usb pen
drive gotten to?

--
Johnny B Good