[RFC] vtunerc: virtual DVB device - is it ok to NACK driver because of worrying about possible misusage?

[HoP]

Hi folks.

I need to warn you that my mail is a bit little longer then I would like
to be.But I'm not able to ask my question without some
background information.

On June 19th, I was sending the driver to the Linux-media
mailing list. Original announcement is there:

http://www.spinics.net/lists/linux-media/msg34240.html

One would say that the code describes very well what it does = adds
virtual DVB device. To be more clear on it I have even done some
small picture:

http://www.nessiedvb.org/wiki/doku.php?id=vtuner_bigpicture

I was hoping to get any feedback regarding code implementation.
It was my first code for the kernel and I felt very well that some
part can be done better or even simpler.

What really surprised me badly was that when I read all 54 responses
I have counted only two real technical answers!!! All rest were about
POLITICAL issues - code was NACKed w/o any technical discussion.
Because of fear of possible abusing of driver.

I didn't know that there existed very big movement against such
code in dvb-core subsystem before.

I have one big problem with it. I can even imagine that some "bad guys"
could abuse virtual driver to use it for distribution close-source drivers
in the binary blobs. But is it that - worrying about bad boys abusing -
the sufficient reason for such aggressive NACK which I did? Then would
be better to remove loadable module API fully from kernel. Is it the right way?

Please confirm me that worrying about abusive act is enough to NACK
particular driver. Then I may be definitely understand I'm doing something
wrong and will stay (with such enemy driver) out of tree.

I can't understand that because I see very similar drivers in kernel for ages
(nbd, or even more similar is usbip) and seems they don't hamper to anybody.

I would like to note that I don't want to start any flame-war, so very short
answer would be enough for me.

Regards

Honza

PS: Please be so kind and CC the answer/comment to me, I'm
only on linux-media ML, not on linux-kernel ML. Thanks.

BTW, if accidentally, somebody find it interesting and would like to
help me doing code review, there is the code hosted now:
http://code.google.com/p/vtuner/source/browse?repo=linux-driver
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Michael Krufky]

Honza,

I, for one, would love to see your virtual DVB device driver hosted in
a repository for the purposes of experimentation and additional
development. I can think of many, many good uses for such a virtual
device driver. Unfortunately, however, all the device vendors also
have uses for it. It a guarunteed fact that if a driver like that got
merged into the kernel, any software company that previously sponsored
open-source kernel development would opt instead for closed source
userspace drivers that depend on a virtual DVB device.

Please don't let that discourage you -- I think you should continue
your work on this virtual DVB device driver, and I'd love to play with
it myself, and possibly even contribute to it. ...but I will never
support the merging of this into the kernel.

I do not nack the existence of the driver -- I love the idea, and I
encourage more development. I only nack it's merging into any
open-source linux kernel.

Please accept my answer with the greatest intentions for furthering
the development of the open-source community. My opinion is only for
the best intentions of continued contributions from companies such as
Hauppauge and any others that have contributed thus far to v4l/dvb.

Please, keep up the work. I repeat -- I would love to play with your work.

Best regards,

Mike Krufky

[Andreas Oberritter]

> answer would be enough for me.,

Hello Honza,

I still support the inclusion of your virtual DVB device driver, once
the technical issues[1] are solved (design clean interface based on
DVBv5 etc.). Mauro promised to consider it for inclusion then[2]. A
quick view at your code indicates that this clean-up hasn't happened
yet, e.g. there are hacks to support DVB-S2 over DVBv3 which aren't
necessary anymore with v5.

Regarding the kernellabs.com people[3] lobbying against your
contribution: Don't give up! If all attempts of merging your work
through the media subsystem are failing, try convincing some major
distributions to include your work. This would make their arguments
meaningless. On the long run, good code is likely to win over politics.

Regards,
Andreas

[1] http://www.spinics.net/lists/linux-media/msg34349.html
[2] http://www.spinics.net/lists/linux-media/msg34352.html
[3] http://www.kernellabs.com/blog/?page_id=6
[4]
http://code.google.com/p/vtuner/source/browse/vtunerc_proxyfe.c?repo=linux-driver#177

[Mauro Carvalho Chehab]

What I said on that time is that a virtual driver to run at the OS on a
VM machine (kvm or xen) that would export the DVB devices that are available
at the OS at the local host or on a remote host running the spice client
could be an interesting contribution, and would fit into kernelspace.

I'm not sure about such need nowadays, as very recent patches added on kvm
are now allowing to use USB 2.0 video devices on it (I tested and used it
a few days ago - it seems to be working at least with the devices I tested),
and PCI passthrough is already there also. Yet, a DVB-optimized passthrough
module there would likely perform better than just exporting the physical
device to the VM.

The driver, as proposed, is not really a driver, as it doesn't support any
hardware. The kernel driver would be used to just copy data from one userspace
application to the other. The same result could be obtained in userspace,
by implementing a library. Such library could even use LD_PRELOAD to support
binary only applications, like what libv4l does. In terms of performance,
such library would probably perform better than a kernel driver, as there's
no need to do context switching for each call, and no need to talk with a
device (where kernel outperforms userspace). Also, depending on how such library
is implemented, double buffering might be avoided.

So, from architectural POV, this code should be written as an userspace library.
BTW, alsa also came with the same results, years ago, as audio remote
streaming is supported via userspace tools, like pulseaudio.

> A quick view at your code indicates that this clean-up hasn't happened
> yet, e.g. there are hacks to support DVB-S2 over DVBv3 which aren't
> necessary anymore with v5.
>
> Regarding the kernellabs.com people[3] lobbying against your
> contribution: Don't give up! If all attempts of merging your work
> through the media subsystem are failing, try convincing some major
> distributions to include your work. This would make their arguments
> meaningless. On the long run, good code is likely to win over politics.
>
> Regards,
> Andreas
>
> [1] http://www.spinics.net/lists/linux-media/msg34349.html
> [2] http://www.spinics.net/lists/linux-media/msg34352.html

> [3] http://www.kernellabs.com/blog/?page_id=6
> [4]
> http://code.google.com/p/vtuner/source/browse/vtunerc_proxyfe.c?repo=linux-driver#177
> --

> To unsubscribe from this list: send the line "unsubscribe linux-media" in

[HoP]

Hi,

let me ask you some details of your interesting idea (how to
achieve the same functionality as with vtunerc driver):

[...]

> The driver, as proposed, is not really a driver, as it doesn't support any
> hardware. The kernel driver would be used to just copy data from one
> userspace

Please stop learning me what can be called driver and what nope.
Your definition is nonsense and I don't want to follow you on it.

> application to the other. The same result could be obtained in userspace,
> by implementing a library. Such library could even use LD_PRELOAD to support
> binary only applications, like what libv4l does. In terms of performance,
> such library would probably perform better than a kernel driver, as there's
> no need to do context switching for each call, and no need to talk with a
> device (where kernel outperforms userspace). Also, depending on how such
> library
> is implemented, double buffering might be avoided.
>
> So, from architectural POV, this code should be written as an userspace
> library.
> BTW, alsa also came with the same results, years ago, as audio remote
> streaming is supported via userspace tools, like pulseaudio.

Can you show me, how then can be reused most important part
of dvb-core subsystem like tuning and demuxing? Or do you want me
to invent wheels and to recode everything in the library? Of course
I can be wrong, I'm no big kernel hacker. So please show me the
way for it. BTW, even if you can find the way, then data copying
from userspace to the kernel and back is also necessery. I really
don't see any advantage of you solution.

Honza

[Mauro Carvalho Chehab]

On 01-12-2011 12:58, HoP wrote:
> Hi,
>
> let me ask you some details of your interesting idea (how to
> achieve the same functionality as with vtunerc driver):
>
> [...]
>
>> The driver, as proposed, is not really a driver, as it doesn't support any
>> hardware. The kernel driver would be used to just copy data from one
>> userspace
>
> Please stop learning me what can be called driver and what nope.
> Your definition is nonsense and I don't want to follow you on it.
>
>> application to the other. The same result could be obtained in userspace,
>> by implementing a library. Such library could even use LD_PRELOAD to support
>> binary only applications, like what libv4l does. In terms of performance,
>> such library would probably perform better than a kernel driver, as there's
>> no need to do context switching for each call, and no need to talk with a
>> device (where kernel outperforms userspace). Also, depending on how such
>> library
>> is implemented, double buffering might be avoided.
>>
>> So, from architectural POV, this code should be written as an userspace
>> library.
>> BTW, alsa also came with the same results, years ago, as audio remote
>> streaming is supported via userspace tools, like pulseaudio.
>
> Can you show me, how then can be reused most important part
> of dvb-core subsystem like tuning and demuxing?

Why do you need to implement it? Just forward the requests to the machine
with the real driver.

> Or do you want me
> to invent wheels and to recode everything in the library?

Kernel code is GPLv2. You can use its code on a GPLv2 licensed library.

> Of course
> I can be wrong, I'm no big kernel hacker. So please show me the
> way for it. BTW, even if you can find the way, then data copying
> from userspace to the kernel and back is also necessery.

See libv4l, at v4l2-utils.git (at linuxtv.org).

> I really
> don't see any advantage of you solution.

And I can't see any advantage on yours ;) Putting something that belongs
to userspace into kernelspace just because it is easier to re-use the
existing code inside the kernel is not a good argument.

Don't get me wrong but if you want to submit a code to be merged
on any existing software (being open source or not), you should be
prepared to defend your code and justify the need for it to the
other developers.

Regards,
Mauro

[HoP]

2011/12/1 Mauro Carvalho Chehab <mch...@redhat.com>:

I see. So if you think it is nice to get dvb-core, make a wrapper around
to get it usable in userspace and maintain totally same functionality
by myself then I say it is no go. If it looks for you like good idea
I must disagree. Code duplication? Two maintaners? That is crazy idea man.

>
>> Of course
>> I can be wrong, I'm no big kernel hacker. So please show me the
>> way for it. BTW, even if you can find the way, then data copying
>> from userspace to the kernel and back is also necessery.
>
> See libv4l, at v4l2-utils.git (at linuxtv.org).
>
>> I really
>> don't see any advantage of you solution.
>
> And I can't see any advantage on yours ;) Putting something that belongs
> to userspace into kernelspace just because it is easier to re-use the
> existing code inside the kernel is not a good argument.

It is only your POV that it should be in userspace.

Creating additional code which not only enlarge code size by 2
but I think by 10 is really not good idea. And it get no advantage
only disadvantages.

>
> Don't get me wrong but if you want to submit a code to be merged
> on any existing software (being open source or not), you should be
> prepared to defend your code and justify the need for it to the
> other developers.

Sure. I was prepared for technical disscussion, but was fully suprised
that it was not happend (ok, to be correct, few guys are exception, like
Andreas and few others. I really appreciate it).

So, my question was still not answered: "Can be driver NACKed only
because of worrying about possible misuse?"

Honza

[Mauro Carvalho Chehab]

I fail to see where do you need to duplicate dvb-core. An userspace
LD_PRELOAD handler that would do:

int socket;

int dvb_ioctl(int fd, unsigned long int request, ...)
{
void *arg;
va_list ap;

va_start(ap, request);
arg = va_arg(ap, void *);
va_end(ap);

send_net_ioctl_packet(socket, request, arg);
}

Is probably all you need to send _any_ ioctl's to a remote machine
(plus client's machine that would decode the ioctl packet and send
the ioctl to the actual driver).

Of course, you'll need hooks for all syscalls used (likely open, close,
ioctl, read, poll).

So, there's not much duplication, even if, for whatever reason, you
might need to hook some specific ioctls in order to optimize the
network performance.

>>> Of course
>>> I can be wrong, I'm no big kernel hacker. So please show me the
>>> way for it. BTW, even if you can find the way, then data copying
>>> from userspace to the kernel and back is also necessery.
>>
>> See libv4l, at v4l2-utils.git (at linuxtv.org).
>>
>>> I really
>>> don't see any advantage of you solution.
>>
>> And I can't see any advantage on yours ;) Putting something that belongs
>> to userspace into kernelspace just because it is easier to re-use the
>> existing code inside the kernel is not a good argument.
>
> It is only your POV that it should be in userspace.
>
> Creating additional code which not only enlarge code size by 2
> but I think by 10 is really not good idea. And it get no advantage
> only disadvantages.
>
>>
>> Don't get me wrong but if you want to submit a code to be merged
>> on any existing software (being open source or not), you should be
>> prepared to defend your code and justify the need for it to the
>> other developers.
>
> Sure. I was prepared for technical disscussion, but was fully suprised
> that it was not happend (ok, to be correct, few guys are exception, like
> Andreas and few others. I really appreciate it).
>
> So, my question was still not answered: "Can be driver NACKed only
> because of worrying about possible misuse?"

To answer your question: your driver were nacked because of several reasons:
it is not a driver for an unsupported hardware, you failed to convince people
why this can't be implemented on userspace, the driver adds hooks at kernelspace
that would open internal API's that several developers don't agree on exposing
at userspace, as would allow non GPL license compatible drivers to re-use
their work in a way they are against.

You're free to politely argue with regards to your solution at the ML.

From my side, due to already explained technical reasons, I nack the
current approach. I'm free to not comment it further, while there's
no new fact that would convince me otherwise.

Regards,
Mauro

[Andreas Oberritter]

On 01.12.2011 21:38, Mauro Carvalho Chehab wrote:
> I fail to see where do you need to duplicate dvb-core. An userspace
> LD_PRELOAD handler that would do:
>
> int socket;
>
> int dvb_ioctl(int fd, unsigned long int request, ...)
> {
> void *arg;
> va_list ap;
>
> va_start(ap, request);
> arg = va_arg(ap, void *);
> va_end(ap);
>
> send_net_ioctl_packet(socket, request, arg);
> }
>
> Is probably all you need to send _any_ ioctl's to a remote machine
> (plus client's machine that would decode the ioctl packet and send
> the ioctl to the actual driver).
>
> Of course, you'll need hooks for all syscalls used (likely open, close,
> ioctl, read, poll).
>
> So, there's not much duplication, even if, for whatever reason, you
> might need to hook some specific ioctls in order to optimize the
> network performance.

Mauro, we've already had that discussion last time. In order to
intercept ioctls of a device, the device needs to exist to begin with,
right? That's where vtuner comes in: It creates the virtual device.

For that reason your suggested approach using LD_PRELOAD won't work.

Besides that, suggesting LD_PRELOAD for something other than a hack
can't be taken seriously.

I think you didn't even understand what vtuner does, after all the
discussion that took place.

It's not a driver for supported hardware either. You named it before:
It's not a driver in your definition at all. It's a way to remotely
access digital TV tuners over a network.

> you failed to convince
> people
> why this can't be implemented on userspace,

Wrong. You failed to convince people why this must be implemented in
userspace. Even Michael Krufky, who's "only" against merging it, likes
the idea, because it's useful.

Just because something can be implemented in userspace doesn't mean that
it's technically superior.

> the driver adds hooks at
> kernelspace
> that would open internal API's that several developers don't agree on
> exposing
> at userspace, as would allow non GPL license compatible drivers to re-use
> their work in a way they are against.

What's left is your unreasonable GPL blah blah. So the answer to Honza's
question is: Yes, Mauro is nacking the driver because he's worrying
about possible misuse.

Regards,
Andreas

[Mauro Carvalho Chehab]

On 01-12-2011 20:55, Andreas Oberritter wrote:
> On 01.12.2011 21:38, Mauro Carvalho Chehab wrote:
>> I fail to see where do you need to duplicate dvb-core. An userspace
>> LD_PRELOAD handler that would do:
>>
>> int socket;
>>
>> int dvb_ioctl(int fd, unsigned long int request, ...)
>> {
>> void *arg;
>> va_list ap;
>>
>> va_start(ap, request);
>> arg = va_arg(ap, void *);
>> va_end(ap);
>>
>> send_net_ioctl_packet(socket, request, arg);
>> }
>>
>> Is probably all you need to send _any_ ioctl's to a remote machine
>> (plus client's machine that would decode the ioctl packet and send
>> the ioctl to the actual driver).
>>
>> Of course, you'll need hooks for all syscalls used (likely open, close,
>> ioctl, read, poll).
>>
>> So, there's not much duplication, even if, for whatever reason, you
>> might need to hook some specific ioctls in order to optimize the
>> network performance.
>
> Mauro, we've already had that discussion last time. In order to
> intercept ioctls of a device, the device needs to exist to begin with,
> right? That's where vtuner comes in: It creates the virtual device.

Yes.

> For that reason your suggested approach using LD_PRELOAD won't work.

If you're referring to the device name under /dev, a daemon emulating
a physical device could create Unix sockets under /dev/dvb.

Or (with is the right solution) bind such library into the applications
that will be used.

> Besides that, suggesting LD_PRELOAD for something other than a hack
> can't be taken seriously.

A Kernel pigback plugin is also a hack.

Yes, this is not a driver. It is just a hack to avoid adding network
support at the userspace applications.

>> you failed to convince
>> people
>> why this can't be implemented on userspace,
>
> Wrong. You failed to convince people why this must be implemented in
> userspace. Even Michael Krufky, who's "only" against merging it, likes
> the idea, because it's useful.

Sometimes, when I'm debugging a driver, I use to add several hacks inside
the kernelspace, in order to do things that are useful on my development
(debug printk's, dirty hacks, etc). I even have my own set of patches that
I apply on kvm, in order to sniff PCI traffic. This doesn't mean that
I should send all those crap upstream.

> Just because something can be implemented in userspace doesn't mean that
> it's technically superior.

True, but I didn't see anything at the submitted code or at the discussions
showing that implementing it in kernelspace is technically superior.

What I'm seeing is what is coded there:

http://code.google.com/p/vtuner/

The kernelspace part is just a piggyback driver, that just copies data from/to
the dvb calls into another device, that sends the request back to userspace.

A separate userspace daemon will get such results and send to the network stack:
http://code.google.com/p/vtuner/source/browse/vtuner-network.c?repo=apps

This is technically inferior of letting the application just talk to vtuner
directly via some library call.

Btw, applications like vdr, vlc, kaffeine and others already implement their
own ways to remotelly access the DVB devices without requiring any
kernelspace piggyback driver.

[Andreas Oberritter]

This won't work either, as it would conflict with kernel device
management. Furthermore, programs will detect that a socket is not a
character device.

> Or (with is the right solution) bind such library into the applications
> that will be used.

You mean "such" libraries broken by design?

>> Besides that, suggesting LD_PRELOAD for something other than a hack
>> can't be taken seriously.
>
> A Kernel pigback plugin is also a hack.

Inventing random terms doen't help. Even if you consider it a hack,
you're one of very few people to do so. So it's a better hack than using
LD_PRELOAD. Btw., since when are kernel modules called plugins?

One could argue about the term hack, but yes, exactly, that's what it
is. And this is a very good feature.

>>> you failed to convince
>>> people
>>> why this can't be implemented on userspace,
>>
>> Wrong. You failed to convince people why this must be implemented in
>> userspace. Even Michael Krufky, who's "only" against merging it, likes
>> the idea, because it's useful.
>
> Sometimes, when I'm debugging a driver, I use to add several hacks inside
> the kernelspace, in order to do things that are useful on my development
> (debug printk's, dirty hacks, etc). I even have my own set of patches that
> I apply on kvm, in order to sniff PCI traffic. This doesn't mean that
> I should send all those crap upstream.

That's a nice story, but it's a completely different topic.

>> Just because something can be implemented in userspace doesn't mean that
>> it's technically superior.
>
> True, but I didn't see anything at the submitted code or at the discussions
> showing that implementing it in kernelspace is technically superior.
>
> What I'm seeing is what is coded there:
>
> http://code.google.com/p/vtuner/
>
> The kernelspace part is just a piggyback driver, that just copies data
> from/to
> the dvb calls into another device, that sends the request back to
> userspace.

Is it a driver now or not? Why piggyback? Because it uses dvb-core? Are
all drivers using dvb-core piggyback drivers?

> A separate userspace daemon will get such results and send to the
> network stack:
> http://code.google.com/p/vtuner/source/browse/vtuner-network.c?repo=apps
>
>
> This is technically inferior of letting the application just talk to vtuner
> directly via some library call.

The goal is to *transparently* add remote tuners to existing
applications. Not so hard to understand.

> Btw, applications like vdr, vlc, kaffeine and others already implement
> their
> own ways to remotelly access the DVB devices without requiring any
> kernelspace piggyback driver.

Can vdr, vlc, kaffeine use remote tuners on hosts not running vdr, vlc
or kaffeine? Should we implement networking protocols in scan, w_scan,
czap, tzap, szap, mplayer, dvbsnoop, dvbstream, etc. instead?

[HoP]

[...]

>>> you failed to convince
>>> people
>>> why this can't be implemented on userspace,
>>
>> Wrong. You failed to convince people why this must be implemented in
>> userspace. Even Michael Krufky, who's "only" against merging it, likes
>> the idea, because it's useful.
>
> Sometimes, when I'm debugging a driver, I use to add several hacks inside
> the kernelspace, in order to do things that are useful on my development
> (debug printk's, dirty hacks, etc). I even have my own set of patches that
> I apply on kvm, in order to sniff PCI traffic. This doesn't mean that
> I should send all those crap upstream.

If you want to disscuss, please better use wordings. Even my Subject was
not about endless discussion that your userspace aproach is better
(me, Andreas and few others doesn't agree with you, it is your own POV,
nobody follows you, if you not noted) don't be so offensive regarding my code.
I don't thing it is so crap.

>
>> Just because something can be implemented in userspace doesn't mean that
>> it's technically superior.
>
> True, but I didn't see anything at the submitted code or at the discussions
> showing that implementing it in kernelspace is technically superior.
>
> What I'm seeing is what is coded there:
>
>        http://code.google.com/p/vtuner/
>
> The kernelspace part is just a piggyback driver, that just copies data
> from/to
> the dvb calls into another device, that sends the request back to userspace.
>
> A separate userspace daemon will get such results and send to the network
> stack:
>
>  http://code.google.com/p/vtuner/source/browse/vtuner-network.c?repo=apps
>
> This is technically inferior of letting the application just talk to vtuner
> directly via some library call.
>
> Btw, applications like vdr, vlc, kaffeine and others already implement their
> own ways to remotelly access the DVB devices without requiring any
> kernelspace piggyback driver.

This in not only about well-known "fatty" applications like vdr, kaffeine etc.
Using vtunerc allows to use any dvb api compatible tools. Like tools
from dvb-tools package (szap/dvbsnoop/...)

There is no more general way of having dvb adapter virtualized.

>
>>> the driver adds hooks at
>>> kernelspace
>>> that would open internal API's that several developers don't agree on
>>> exposing
>>> at userspace, as would allow non GPL license compatible drivers to re-use
>>> their work in a way they are against.
>>
>> What's left is your unreasonable GPL blah blah. So the answer to Honza's
>> question is: Yes, Mauro is nacking the driver because he's worrying
>> about possible misuse.

Well, yes. I feel the same what wrote Andreas. The only pity is that
nobody from outside linuxm-media need to comment that.

Honza

[Mauro Carvalho Chehab]

On 02-12-2011 09:57, HoP wrote:

> If you want to disscuss,

No, I don't want. There are architectural issues on your solution. As I said,
from the Kernel POV, just the network drivers is enough to run *any* client-server
solution on any OS that uses the TCP/IP stack. All streaming applications (DVB
or not) have their solution without requiring any virtual driver, using the TCP/IP
stack. You still think that your solution is technically better than theirs.
So, we agree do disagree on that matter.

From my side, I won't merge it due to the already explained reasons.
Mauro.

[Rémi Denis-Courmont]

Le jeudi 1 décembre 2011 21:59:56 HoP, vous avez écrit :
> > Kernel code is GPLv2. You can use its code on a GPLv2 licensed library.
>
> I see. So if you think it is nice to get dvb-core, make a wrapper around
> to get it usable in userspace and maintain totally same functionality
> by myself then I say it is no go. If it looks for you like good idea
> I must disagree. Code duplication?

Sure, some core code would be duplicated. That is not a big deal.

This proposal however has three big advantages:
- Proprietary drivers are not enabled as the library would be GPL.
- The virtual DVB device runs in the same process as the DVB application,
which saves context switching and memory copying.
- It would be your project. You do not need to agree with Mauro ;-)

> Two maintaners? That is crazy idea man.

Someone would have to maintain the device driver anyway. I don't see much of a
difference on maintainance side.

> > And I can't see any advantage on yours ;) Putting something that belongs
> > to userspace into kernelspace just because it is easier to re-use the
> > existing code inside the kernel is not a good argument.
>
> It is only your POV that it should be in userspace.

Except for backward compatiblity, this would actually belong in userspace. It
would be more efficient and easier to maintain as a userspace library than as
a kernel driver.

If you need backward compatibility, I am still inclined to believe that you
could write a CUSE frontend, so it does involve some extra work and looses the
performance benefit.

--
Rémi Denis-Courmont
http://www.remlab.net/
http://fi.linkedin.com/in/remidenis

[Andreas Oberritter]

On 02.12.2011 18:49, Rémi Denis-Courmont wrote:
> Le jeudi 1 décembre 2011 21:59:56 HoP, vous avez écrit :
>>> Kernel code is GPLv2. You can use its code on a GPLv2 licensed library.
>>
>> I see. So if you think it is nice to get dvb-core, make a wrapper around
>> to get it usable in userspace and maintain totally same functionality
>> by myself then I say it is no go. If it looks for you like good idea
>> I must disagree. Code duplication?
>
> Sure, some core code would be duplicated. That is not a big deal.
>
> This proposal however has three big advantages:
> - Proprietary drivers are not enabled as the library would be GPL.
> - The virtual DVB device runs in the same process as the DVB application,
> which saves context switching and memory copying.
> - It would be your project. You do not need to agree with Mauro ;-)
>
>> Two maintaners? That is crazy idea man.
>
> Someone would have to maintain the device driver anyway. I don't see much of a
> difference on maintainance side.
>
>>> And I can't see any advantage on yours ;) Putting something that belongs
>>> to userspace into kernelspace just because it is easier to re-use the
>>> existing code inside the kernel is not a good argument.
>>

>> It is only your POV that it should be in userspace.Also, LGPL drivers

>
> Except for backward compatiblity, this would actually belong in userspace. It
> would be more efficient and easier to maintain as a userspace library than as
> a kernel driver.

Maintaining the kernel module would be rather easy, because new
properties added to dvb_frontend would be handled transparently. The
implementation is quite simple. In contrast, implementing and then
maintaining all the users of a newly written userspace library would be
a nightmare in comparison.

> If you need backward compatibility, I am still inclined to believe that you
> could write a CUSE frontend, so it does involve some extra work and looses the
> performance benefit.

How would all this allow to use e.g. dvbsnoop or w_scan on a remote
tuner? Do you propose to add a dependency to this proposed library to
every application?

Furthermore, a GPLv2 library would artificially restrict its users, e.g.
you wouldn't be allowed to use it with gstreamer or just with anything
that isn't GPLv2, not even v3.

Regards,
Andreas

[Andreas Oberritter]

One more note: CUSE would conflict with dvb-core the same way Mauro's
sockets would do.

[VDR User]

> What really surprised me badly was that when I read all 54 responses
> I have counted only two real technical answers!!! All rest were about
> POLITICAL issues - code was NACKed w/o any technical discussion.
> Because of fear of possible abusing of driver.

To answer the original question -- absolutely not! Linux is not about
and should not try policing what it's users do, or passing moral
judgment. If there are technical reasons something shouldn't be merged
as is, no problem. There are times when a persons personal preference
should be left out of the equation, and this is an example of one.

[Alan Cox]

On Thu, 1 Dec 2011 15:58:41 +0100
HoP <jpet...@gmail.com> wrote:

> Hi,
>
> let me ask you some details of your interesting idea (how to
> achieve the same functionality as with vtunerc driver):
>
> [...]
>
> > The driver, as proposed, is not really a driver, as it doesn't support any
> > hardware. The kernel driver would be used to just copy data from one
> > userspace
>
> Please stop learning me what can be called driver and what nope.
> Your definition is nonsense and I don't want to follow you on it.

You can stick your fingers in your ears and shout all you like but given
Mauro is the maintainer I'd suggest you work with him rather than making
it painful. One of the failures we routinely exclude code from the kernel
for is best described as "user interface of contributor"

It's a loopback that adds a performance hit. The right way to do this is
in userspace with the userspace infrastructure. At that point you can
handle all the corner cases properly, integrate things like service
discovery into your model and so on - stuff you'll never get to work that
well with kernel loopback hackery.

> Can you show me, how then can be reused most important part
> of dvb-core subsystem like tuning and demuxing? Or do you want me
> to invent wheels and to recode everything in the library? Of course

You could certainly build a library from the same code. That might well
be a good thing for all kinds of 'soft' DV applications. At that point
the discussion to have is the best way to make that code sharable between
a userspace library and the kernel and buildable for both.

> I can be wrong, I'm no big kernel hacker. So please show me the
> way for it. BTW, even if you can find the way, then data copying
> from userspace to the kernel and back is also necessery. I really
> don't see any advantage of you solution.

In a properly built media subsystem you shouldn't need any copies beyond
those that naturally occur as part of a processing pass and are therefore
free.

Alan

[HoP]

Hi Alan.

2011/12/3 Alan Cox <al...@lxorguk.ukuu.org.uk>:

> On Thu, 1 Dec 2011 15:58:41 +0100
> HoP <jpet...@gmail.com> wrote:
>
>> Hi,
>>
>> let me ask you some details of your interesting idea (how to
>> achieve the same functionality as with vtunerc driver):
>>
>> [...]
>>
>> > The driver, as proposed, is not really a driver, as it doesn't support any
>> > hardware. The kernel driver would be used to just copy data from one
>> > userspace
>>
>> Please stop learning me what can be called driver and what nope.
>> Your definition is nonsense and I don't want to follow you on it.
>
> You can stick your fingers in your ears and shout all you like but given
> Mauro is the maintainer I'd suggest you work with him rather than making
> it painful. One of the failures we routinely exclude code from the kernel
> for is best described as "user interface of contributor"

You may be not read all my mails but I really tried to be very positive in them.
I wanted to focus on my Subject, but Mauro has, sometimes, the demand
to focus on insignificant things (like if the code is driver or not). At least
it is my feeling from all those disscussions with him.

>
> It's a loopback that adds a performance hit. The right way to do this is
> in userspace with the userspace infrastructure. At that point you can
> handle all the corner cases properly, integrate things like service
> discovery into your model and so on - stuff you'll never get to work that
> well with kernel loopback hackery.
>
>> Can you show me, how then can be reused most important part
>> of dvb-core subsystem like tuning and demuxing? Or do you want me
>> to invent wheels and to recode everything in the library? Of course
>
> You could certainly build a library from the same code. That might well
> be a good thing for all kinds of 'soft' DV applications. At that point
> the discussion to have is the best way to make that code sharable between
> a userspace library and the kernel and buildable for both.
>
>> I can be wrong, I'm no big kernel hacker. So please show me the
>> way for it. BTW, even if you can find the way, then data copying
>> from userspace to the kernel and back is also necessery. I really
>> don't see any advantage of you solution.
>
> In a properly built media subsystem you shouldn't need any copies beyond
> those that naturally occur as part of a processing pass and are therefore
> free.

I may describe project goal, in few sentences: We have small box, running
embedded linux with 2 satellite tuners on input and ethernet. Nothing more.
We have designed the box for live sat TV/Radio reception, distributing them
down to the network. One of the mode of working is "vtuner", what allows
reuse those tuners remotely on linux desktop. The kernel part is very simple
code exposing kernel's dvb-core to the userspace. Userspace client/server
tools do all resource discovery and connection management. It works
nicely and guys with vdr who is using it are rather satisfied with it.
So, the main
goal of vtuner code is to fully virtualize remote DVB adapter. To any
linux dvb api
compatible applications and tools. The vtuner kernel code seems to be
the simplest and straightforward way to achieve it.

I still think the code is very similar to NBD (Network block device) what sits
in the kernel and is using silently. I guess NBD also do data copying
from/to user space. Is there something what I overlooked?

Can you show me the way (hint please) I can initiate TCP connection
from within kernel space? If I can do it, then the big disadvantage
of data passing to and from kernel can be removed.

I must say that the box is primary focused to the DLNA/UpnP world, so
vtuner feature is something like interesting addon only. But I was myself
very nice surprised how good it behaves on real installations and that
was reason I decided to try to get it included in kernel. I see that
in present there is no willingness for code acceptation, so I will continue
out of the kernel tree.

Anyway, if I can find the way how to start TCP connection from the kernel
part, I understand it can boost throughput very nicely up.

Honza

[Andreas Oberritter]

Hello Alan,

On 03.12.2011 00:19, Alan Cox wrote:
> On Thu, 1 Dec 2011 15:58:41 +0100
> HoP <jpet...@gmail.com> wrote:
>
>> Hi,
>>
>> let me ask you some details of your interesting idea (how to
>> achieve the same functionality as with vtunerc driver):
>>
>> [...]
>>
>>> The driver, as proposed, is not really a driver, as it doesn't support any
>>> hardware. The kernel driver would be used to just copy data from one
>>> userspace
>>
>> Please stop learning me what can be called driver and what nope.
>> Your definition is nonsense and I don't want to follow you on it.
>
> You can stick your fingers in your ears and shout all you like but given
> Mauro is the maintainer I'd suggest you work with him rather than making
> it painful. One of the failures we routinely exclude code from the kernel
> for is best described as "user interface of contributor"
>
> It's a loopback that adds a performance hit. The right way to do this is
> in userspace with the userspace infrastructure. At that point you can
> handle all the corner cases properly, integrate things like service
> discovery into your model and so on - stuff you'll never get to work that
> well with kernel loopback hackery.

FWIW, the virtual DVB device we're talking about doesn't have any
networking capabilities by itself. It only allows to create virtual DVB
adapters and to relay DVB API ioctls to userspace in a
transport-agnostic way. Networking already takes place in a separate
userspace process.

>> Can you show me, how then can be reused most important part
>> of dvb-core subsystem like tuning and demuxing? Or do you want me
>> to invent wheels and to recode everything in the library? Of course
>
> You could certainly build a library from the same code. That might well
> be a good thing for all kinds of 'soft' DV applications. At that point
> the discussion to have is the best way to make that code sharable between
> a userspace library and the kernel and buildable for both.

You could certainly build a library to reach a different goal. The goal
of vtuner is to access remote tuners with any existing program
implementing the DVB API.

One of my primary use cases is to develop DVB API software on my
notebook, while receiving the data from an embedded device, so I don't
need an antenna cable connected to my notebook (or don't need to have an
antenna at my location at all, if an internet connection is available).

Another use case is to receive data streams over the internet from
satellite, cable and terrestrial networks that are impossible to receive
in my area to verify that my DVB API software works with those streams.

Furthermore, I'd like to use standard tools implementing theDid you even
unders DVB API, e.g. to scan for channels, to measure signal quality or
to decode streams on remote locations.

With vtuner, and an accompanying open source networking daemon, all of
the above works out of the box. If I'd like to change the way data gets
relayed between hosts (e.g. add IPv6 or SSL), I'd just need to update
the userspace client and server.

With a library, I'd need to implement networking in every little DVB API
tool I want to use, multiplying the code size of every tool, restricting
each tool to some arbitrarily chosen network protocol. If I'd like to
change the way data gets relayed between hosts, I'd need to update every
application using it.

>> I can be wrong, I'm no big kernel hacker. So please show me the
>> way for it. BTW, even if you can find the way, then data copying
>> from userspace to the kernel and back is also necessery. I really
>> don't see any advantage of you solution.
>
> In a properly built media subsystem you shouldn't need any copies beyond
> those that naturally occur as part of a processing pass and are therefore
> free.

The original version of the vtuner interface makes use of demux hardware
and also feeds the relevant streams to a/v decoders, in which case you
cannot avoid copying the MPEG data to kernel space.

That's also the way to playback local DVB recordings works today without
vtuners.

I'm fine with not merging this code, if there are technical arguments
against it. And personally I don't need it to be merged to be able to
use it.

However, Mauro invents half-baked arguments against it, because he fears
that the code may be "abused" by closed-source userspace drivers (he
already declared the latter publicly on the linux-media list).

Unfortunately, I must assume that your own arguments are based on
Mauro's emails instead of on the source code, because you clearly didn't
understand the architecture of vtuner, regarding what's supposed to
happen in kernel space and what not.

Regards,
Andreas

[Alan Cox]

> FWIW, the virtual DVB device we're talking about doesn't have any
> networking capabilities by itself. It only allows to create virtual DVB
> adapters and to relay DVB API ioctls to userspace in a
> transport-agnostic way.

Which you can do working from CUSE already, as has been pointed out or
with LD_PRELOAD. This btw makes the proprietary thing seem a rather odd
objection - they could do it too.

> The original version of the vtuner interface makes use of demux hardware
> and also feeds the relevant streams to a/v decoders, in which case you
> cannot avoid copying the MPEG data to kernel space.

Right and that aspect of it actually makes a lot more sense to me,
because receive and decode/playback are different functional units.

But jumping up and down arguing about how you want to name the code isn't
going to produce a productive result

To answer the other question posed by "HoP" - yes you can do TCP from
kernel space. We need this for things like NFS, CIFS etc. That might well
be valuable especially for low end hardware. The best code to read is
probably the net/sunrpc code to get an idea of how it ends up looking.

Alan

[VDR User]

On Sat, Dec 3, 2011 at 8:13 AM, Andreas Oberritter <o...@linuxtv.org> wrote:
> You could certainly build a library to reach a different goal. The goal
> of vtuner is to access remote tuners with any existing program
> implementing the DVB API.

So you could finally use VDR as a server/client setup using vtuner,
right? With full OSD, timer, etc? Yes, I'm aware that streamdev
exists. It was horrible when I tried it last (a long time ago) and I
understand it's gotten better. But it's not a suitable replacement for
a real server/client setup. It sounds like using vtuner, this would
finally be possible and since Klaus has no intention of ever
modernizing VDR into server/client (that I'm aware of), it's also the
only suitable option as well.

Or am I wrong about anything? If not, I know several users who would
like to use this, myself included.

[Andreas Oberritter]

On 03.12.2011 17:42, Alan Cox wrote:
>> FWIW, the virtual DVB device we're talking about doesn't have any
>> networking capabilities by itself. It only allows to create virtual DVB
>> adapters and to relay DVB API ioctls to userspace in a
>> transport-agnostic way.
>
> Which you can do working from CUSE already, as has been pointed out or
> with LD_PRELOAD. This btw makes the proprietary thing seem a rather odd
> objection - they could do it too.

Yes, both CUSE and LD_PRELOADED have already been suggested. But I
already explained in previous mails why both options are unsuitable:

- For LD_PRELOAD to intercept any calls to open, ioctl, read, write, the
character device to be intercepted must exist first. But it doesn't
exist, unless it was created by vtuner first. Of course you could also
intercept stat(), readdir() and all other possible functions to list or
access devices and fake entries, but I hope you're not suggesting this
seriously. Well, proprietary drivers doing that already exist today
(e.g. by sundtek), but implementing somthing like that if an easier
solution is possible would be crazy at best.

Additionaly, preloaded libraries conflict with other preloaded libraries
overwriting the same functions. Preloaded libraries don't work with
statically linked programs. Preloaded libraries can't be used
transparently, because you need to edit init scripts to use them, unless
you want them to be used globally (in /etc/ld.so.preload).

- CUSE would conflict with dvb-core. Once CUSE created a DVB adapter,
registering another non-virtual DVB adapter (by plugging in a USB device
for example) will try to assign the same adapter number.

The following only applies to the original version of the interface, but
is likely to apply to future development of the proposed interface: In
order to add virtual tuners to existing DVB adapters, to be able to use
their hardware demux or MPEG decoders with CUSE, the whole device driver
of this adapter has to be implemented in userspace.

Regards,
Andreas

[Alan Cox]

On Sat, 3 Dec 2011 09:21:23 -0800
VDR User <user...@gmail.com> wrote:

> On Sat, Dec 3, 2011 at 8:13 AM, Andreas Oberritter <o...@linuxtv.org> wrote:
> > You could certainly build a library to reach a different goal. The goal
> > of vtuner is to access remote tuners with any existing program
> > implementing the DVB API.
>
> So you could finally use VDR as a server/client setup using vtuner,
> right? With full OSD, timer, etc? Yes, I'm aware that streamdev
> exists. It was horrible when I tried it last (a long time ago) and I
> understand it's gotten better. But it's not a suitable replacement for
> a real server/client setup. It sounds like using vtuner, this would
> finally be possible and since Klaus has no intention of ever
> modernizing VDR into server/client (that I'm aware of), it's also the
> only suitable option as well.

I would expect it to still suck. One of the problems you have with trying
to pretend things are not networked is that you fake asynchronous events
synchronously, you can't properly cover error cases and as a result you
get things like ioctls that hang for two minutes or fail in bogus and
bizarre ways. If you loop via userspace you've also got to deal with
deadlocks and all sorts of horrible cornercases like the user space
daemon dying.

There is a reason properly working client/server code looks different -
it's not a trivial transformation and faking it kernel side won't be any
better than faking it in user space - it may well even be a worse fake.

Alan

[Devin Heitmueller]

This whole notion of creating fake kernel devices to represent
networked tuners feels like a hack. If applications want to access
networked tuners, adding support for RTP/RTSP or incorporating
libhdhomerun (LGPL) is a fairly straightforward exercise. In fact,
many applications already have incorporated support for one of these
two approaches. The fact that app maintainers have been
unwilling/uninterested to do such doesn't feel like it should be an
excuse for hacking this functionality into the kernel.

Devin

--
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

[Andreas Oberritter]

On 03.12.2011 18:42, Alan Cox wrote:
> On Sat, 3 Dec 2011 09:21:23 -0800
> VDR User <user...@gmail.com> wrote:
>
>> On Sat, Dec 3, 2011 at 8:13 AM, Andreas Oberritter <o...@linuxtv.org> wrote:
>>> You could certainly build a library to reach a different goal. The goal
>>> of vtuner is to access remote tuners with any existing program
>>> implementing the DVB API.
>>
>> So you could finally use VDR as a server/client setup using vtuner,
>> right?

Yes.

>> With full OSD, timer, etc? Yes, I'm aware that streamdev
>> exists. It was horrible when I tried it last (a long time ago) and I
>> understand it's gotten better. But it's not a suitable replacement for
>> a real server/client setup. It sounds like using vtuner, this would
>> finally be possible and since Klaus has no intention of ever
>> modernizing VDR into server/client (that I'm aware of), it's also the
>> only suitable option as well.
>
> I would expect it to still suck. One of the problems you have with trying
> to pretend things are not networked is that you fake asynchronous events
> synchronously, you can't properly cover error cases and as a result you
> get things like ioctls that hang for two minutes or fail in bogus and
> bizarre ways. If you loop via userspace you've also got to deal with
> deadlocks and all sorts of horrible cornercases like the user space
> daemon dying.

USB tuners may be removed anytime during any ioctl, too. Handling such
error cases is therefore already a requirement, at least for
hotplug-capable software.

> There is a reason properly working client/server code looks different -
> it's not a trivial transformation and faking it kernel side won't be any
> better than faking it in user space - it may well even be a worse fake.

It's certainly not suitable for every possible use case in the world.
For many, however, I think it's the optimal solution.

Regards,
Andreas

[Hans Petter Selasky]

Hi,

Some input from the sideline reading this discussion. As a FreeBSD'er I would
very much like to see two things happen:

- vtunerc goes into userspace like a client/server daemon pair using CUSE and
can support _any_ /dev/dvb/adapter, also those created by CUSE itself. That
means I could potentially use vtunerc in FreeBSD with drivers like cx88:

http://corona.homeunix.net/cx88wiki

- DVB-X solution in Linux gets mmap support to avoid endless copying of data
between kernel and userspace.

--HPS

[VDR User]

On Sat, Dec 3, 2011 at 3:30 PM, Walter Van Eetvelt
<wal...@van.eetvelt.be> wrote:
>> So you could finally use VDR as a server/client setup using vtuner,
>> right? With full OSD, timer, etc? Yes, I'm aware that streamdev
>> exists. It was horrible when I tried it last (a long time ago) and I
>> understand it's gotten better. But it's not a suitable replacement for
>> a real server/client setup. It sounds like using vtuner, this would
>> finally be possible and since Klaus has no intention of ever
>> modernizing VDR into server/client (that I'm aware of), it's also the
>> only suitable option as well.
>>
>> Or am I wrong about anything?  If not, I know several users who would
>> like to use this, myself included.
>

> This is the question on how you provide (live)tv content to multiple users
> and devices.
> And I think this goes beyond the simple "second PC in the house" where you
> want to watch TV or recordings.  How can you watch streams from your DVB
> device on your small screen Mobile phone/tablet?  Or stream over the
> internet and watch your last night recording when commuting in the train?
> Some commercial solutions are already available for some scenario's.  Also
> free software solutions support for some scenario's, but I think still too
> limited.
>
> With that in mind, for me the future is for a client server combination
> that can do proper live TV and recording streaming.
> And technically, a virtual driver is not bringing better user experience
> in the long term I think.  On the contrary, some users would use the
> vtunerc to fill in some of their needs.  This could bring developers to
> work less on client/server application supporting a broader range of use
> cases.

While I agree with your more broad view of the issue, I specifically
talked about VDR. AFAIK Klaus has no intention of adding true
server/client support to VDR, so for VDR users, this sounds like it
could be a working solution without the strict limitations of
streamdev.

[Walter Van Eetvelt]

On Sat, 3 Dec 2011 09:21:23 -0800, VDR User <user...@gmail.com> wrote:
...

> So you could finally use VDR as a server/client setup using vtuner,
> right? With full OSD, timer, etc? Yes, I'm aware that streamdev
> exists. It was horrible when I tried it last (a long time ago) and I
> understand it's gotten better. But it's not a suitable replacement for
> a real server/client setup. It sounds like using vtuner, this would
> finally be possible and since Klaus has no intention of ever
> modernizing VDR into server/client (that I'm aware of), it's also the
> only suitable option as well.
>
> Or am I wrong about anything? If not, I know several users who would
> like to use this, myself included.

This is the question on how you provide (live)tv content to multiple users
and devices.
And I think this goes beyond the simple "second PC in the house" where you
want to watch TV or recordings. How can you watch streams from your DVB
device on your small screen Mobile phone/tablet? Or stream over the
internet and watch your last night recording when commuting in the train?
Some commercial solutions are already available for some scenario's. Also
free software solutions support for some scenario's, but I think still too
limited.

With that in mind, for me the future is for a client server combination
that can do proper live TV and recording streaming.
And technically, a virtual driver is not bringing better user experience
in the long term I think. On the contrary, some users would use the
vtunerc to fill in some of their needs. This could bring developers to
work less on client/server application supporting a broader range of use
cases.

Walter

[Alan Cox]

> While I agree with your more broad view of the issue, I specifically
> talked about VDR. AFAIK Klaus has no intention of adding true
> server/client support to VDR, so for VDR users, this sounds like it
> could be a working solution without the strict limitations of
> streamdev.

So fix Klaus rather than mess up the kernel.

If you are trying to solve a VDR political problem then kernel hacks are
not the way to go. Someone who cares about it needs to fix VDR, with or
without its current maintainer.

[HoP]

Hi.

2011/12/3 VDR User <user...@gmail.com>:

> On Sat, Dec 3, 2011 at 8:13 AM, Andreas Oberritter <o...@linuxtv.org> wrote:
>> You could certainly build a library to reach a different goal. The goal
>> of vtuner is to access remote tuners with any existing program
>> implementing the DVB API.
>
> So you could finally use VDR as a server/client setup using vtuner,
> right? With full OSD, timer, etc? Yes, I'm aware that streamdev
> exists. It was horrible when I tried it last (a long time ago) and I
> understand it's gotten better. But it's not a suitable replacement for
> a real server/client setup. It sounds like using vtuner, this would
> finally be possible and since Klaus has no intention of ever
> modernizing VDR into server/client (that I'm aware of), it's also the
> only suitable option as well.
>
> Or am I wrong about anything?  If not, I know several users who would
> like to use this, myself included.

Well, initial report was made on vdr-portal because of our hardware announce,
but you can be sure the same is true if server is build on any linux hardware.
Here is some note:
http://www.vdr-portal.de/board18-vdr-hardware/board84-allgemein/106610-das-neue-netzwerk-client-der-f%C3%BCr-vdr-verwenden-k%C3%B6nnen/?highlight=vtuner

Additional info you can find (or ask) on our forum:
http://forum.nessiedvb.org/forum/viewforum.php?f=11

Please note, that compilation of vtunerc kernel driver (or loopback, or pigback
or whatever name the code should be used) is simple - no need for any
kernel real patching is required. Code can be compiled outside of the
kernel tree
(of course kernel headers are still needed).

Some useful hints regarding userland application daemons you
can find in our wiki:
http://wiki.nessiedvb.org/wiki/doku.php?id=vtuner_mode

When you get vtunerc and vtunerd applications connected, try
simple command line tuning (szap/tzap or czap) to check
if it works correctly. Only if you get zapping working switch
to vdr.

Honza

[HoP]

Devin,

I perfectly remember your opinion regarding vtuner.

2011/12/3 Devin Heitmueller <dheitm...@kernellabs.com>:

Still the same nonsense - why I should add 10x or even 100 times more
code to achieve not the same but may be 80-90% same result?

The idea is hell simple = allow to use those remote tuners by
100% of dvb api compliant applications. Not 80%, but 100%.

Honza

[HoP]

Hi,

> Some input from the sideline reading this discussion. As a FreeBSD'er I would
> very much like to see two things happen:
>
> - vtunerc goes into userspace like a client/server daemon pair using CUSE and
> can support _any_ /dev/dvb/adapter, also those created by CUSE itself. That
> means I could potentially use vtunerc in FreeBSD with drivers like cx88:

Vtuner already has userland client/server pair. The server application
is connecting to _any_ /dev/dvb/adapter device. Please look at small
picture here:
http://code.google.com/p/vtuner/wiki/BigPicture

That means the server part is using totally "clean" linux. Nothing more.

>
> http://corona.homeunix.net/cx88wiki
>
> - DVB-X solution in Linux gets mmap support to avoid endless copying of data
> between kernel and userspace.

Adding nmap support should be straightforward. I have it on my TODO.

Honza

[VDR User]

On Sun, Dec 4, 2011 at 3:22 PM, HoP <jpet...@gmail.com> wrote:
> Well, initial report was made on vdr-portal because of our hardware announce,
> but you can be sure the same is true if server is build on any linux hardware.
> Here is some note:
> http://www.vdr-portal.de/board18-vdr-hardware/board84-allgemein/106610-das-neue-netzwerk-client-der-f%C3%BCr-vdr-verwenden-k%C3%B6nnen/?highlight=vtuner
>
> Additional info you can find (or ask) on our forum:
> http://forum.nessiedvb.org/forum/viewforum.php?f=11
>
> Please note, that compilation of vtunerc kernel driver (or loopback, or pigback
> or whatever name the code should be used) is simple - no need for any
> kernel real patching is required. Code can be compiled outside of the
> kernel tree
> (of course kernel headers are still needed).
>
> Some useful hints regarding userland application daemons you
> can find in our wiki:
> http://wiki.nessiedvb.org/wiki/doku.php?id=vtuner_mode
>
> When you get vtunerc and vtunerd applications connected, try
> simple command line tuning (szap/tzap or czap) to check
> if it works correctly. Only if you get zapping working switch
> to vdr.

Thanks for the info and links. I do know many guys who would be
interested in this if it can provide good server/client ability with
VDR. However, a large number of us only speak english so places like
vdr-portal aren't much use a lot of the time. If you have english
forums somewhere, that link would be far more useful I think.

Thanks

[HoP]

>> Well, initial report was made on vdr-portal because of our hardware announce,
>> but you can be sure the same is true if server is build on any linux hardware.
>> Here is some note:
>> http://www.vdr-portal.de/board18-vdr-hardware/board84-allgemein/106610-das-neue-netzwerk-client-der-f%C3%BCr-vdr-verwenden-k%C3%B6nnen/?highlight=vtuner
>>
>> Additional info you can find (or ask) on our forum:
>> http://forum.nessiedvb.org/forum/viewforum.php?f=11
>>
>> Please note, that compilation of vtunerc kernel driver (or loopback, or pigback
>> or whatever name the code should be used) is simple - no need for any
>> kernel real patching is required. Code can be compiled outside of the
>> kernel tree
>> (of course kernel headers are still needed).
>>
>> Some useful hints regarding userland application daemons you
>> can find in our wiki:
>> http://wiki.nessiedvb.org/wiki/doku.php?id=vtuner_mode
>>
>> When you get vtunerc and vtunerd applications connected, try
>> simple command line tuning (szap/tzap or czap) to check
>> if it works correctly. Only if you get zapping working switch
>> to vdr.
>
> Thanks for the info and links. I do know many guys who would be
> interested in this if it can provide good server/client ability with
> VDR. However, a large number of us only speak english so places like
> vdr-portal aren't much use a lot of the time. If you have english
> forums somewhere, that link would be far more useful I think.

No problem. Simply ignore first URL. All others are english :-)

I would recommend start from how-it-works-diagram:
http://code.google.com/p/vtuner/wiki/BigPicture

and then have a look at wiki. where you can some
step-by-step info.

Anyway, don't hesitate to ask any question in our forum
or write me directly.

Honza

[Florian Fainelli]

Hello,

The company I work for also has something like this. We can attach a DVB
tuner to either the Gateway or the STB we provide and use it
indifferently , except that we have the following architecture:

- a DVB daemon controlling the physical DVB tuner and exposing methods
for tuning/scanning/zapping
- a web server and web services for accessing the DVB daemon methods
- a RTSP streamer with associated methods for controlling streaming

The software running on both devices is the same (one compiled for ARM,
the other for x86).

I do not see any problem with this solution, people wanting to get the
stream can still get the RTSP stream directly joining the multicast
group, which is fortunately OS agnostic at the same time.

>
> I still think the code is very similar to NBD (Network block device) what sits
> in the kernel and is using silently. I guess NBD also do data copying
> from/to user space. Is there something what I overlooked?
>
> Can you show me the way (hint please) I can initiate TCP connection
> from within kernel space? If I can do it, then the big disadvantage
> of data passing to and from kernel can be removed.

Don't do this in kernel-space (remember the mechanism/policy split).

>
> I must say that the box is primary focused to the DLNA/UpnP world, so
> vtuner feature is something like interesting addon only. But I was myself
> very nice surprised how good it behaves on real installations and that
> was reason I decided to try to get it included in kernel. I see that
> in present there is no willingness for code acceptation, so I will continue
> out of the kernel tree.
>
> Anyway, if I can find the way how to start TCP connection from the kernel
> part, I understand it can boost throughput very nicely up.

And here is a new hack.
--
Florian

[HoP]

Hi

2011/12/5 Florian Fainelli <f.fai...@gmail.com>:

Nice. Are your solution open-sourced? If so, can you give me URL?

>> I still think the code is very similar to NBD (Network block device) what
>> sits
>> in the kernel and is using silently. I guess NBD also do data copying
>> from/to user space. Is there something what I overlooked?
>>
>> Can you show me the way (hint please) I can initiate TCP connection
>> from within kernel space? If I can do it, then the big disadvantage
>> of data passing to and from kernel can be removed.
>
>
> Don't do this in kernel-space (remember the mechanism/policy split).

Yes, that can be the issue, I understand that.

You know - I'm a bit confused. Somebody are pointing on double
data copying (userspace networked daemon -> kernel -> application)
issue and another one warn me to not start network connection
from the kernel. But if it works for NFS or CIFS then it should not
be so weaky, isn't it?

>> I must say that the box is primary focused to the DLNA/UpnP world, so
>> vtuner feature is something like interesting addon only. But I was myself
>> very nice surprised how good it behaves on real installations and that
>> was reason I decided to try to get it included in kernel. I see that
>> in present there is no willingness for code acceptation, so I will
>> continue
>> out of the kernel tree.
>>
>> Anyway, if I can find the way how to start TCP connection from the kernel
>> part, I understand it can boost throughput very nicely up.
>
>
> And here is a new hack.

I'm really tired from all those "hack, crap, pigback ..." wordings.

What exactly vtuner aproach does so hackish (other then exposing
DVB internals, what is every time made if virtualization support is developing)?

The code itself no need to patch any line of vanilla kernel source, it even
doesn't change any processing of the rest of kernel, it is very simple
driver/code/whatever.

I can understand that some developers don't like to get dvb-core opened,
I don't agree with them, but I don't really need to fight with them.
Its theirs opinion.
But even I try to see what is so hackish in vtuner implementation, I don't catch
anything. Simplicity? I thought that simplicity is a big advanatge (simple code,
easy to analyze). What else?

Honza

[Alan Cox]

> You know - I'm a bit confused. Somebody are pointing on double
> data copying (userspace networked daemon -> kernel -> application)
> issue and another one warn me to not start network connection
> from the kernel. But if it works for NFS or CIFS then it should not
> be so weaky, isn't it?

And then you want to add multicast, or SSL or some other transport layer,
and so on. You can do it - but in the case of DVB it's really probably
not the overall way to go.

> What exactly vtuner aproach does so hackish (other then exposing
> DVB internals, what is every time made if virtualization support is developing)?

Exposing DVB internals is also not good - it creates an API which locks
out future trivial changes to that API because it might break your
stuff. Also you are trying to fake distributed network in the wrong place
so faking basically synchronous interfaces with undefined network
behaviour and time lines.

> The code itself no need to patch any line of vanilla kernel source, it even
> doesn't change any processing of the rest of kernel, it is very simple
> driver/code/whatever.

That's not a measure of whether something is a good idea, more of the
cleanness of the core code.

Fix the userspace bits that need fixing, or use ones that don't - in the
longer term that will be a bigger win by far.

Alan

[Michael Krufky]

Honza,

I just want to reiterate my *positive* opinion. (you already heard my
negative, I don't need to repeat it) ... I am very excited to hear
that you're working on this virtual tuner driver, and I don't care
what policy says what -- I, personally, want to play with this for my
own uses.

Let's stop arguing, and just write code! That's what makes us happy
- lets be happy :-)

I haven't had a chance yet to look at your full sources, but I
reiterate how excited I am that this virtual dvb adapter exists now.
Regardless of whether the community (or I) agrees as to its kernel
inclusion, nothing stops you from continuing your work, and publishing
tools that use this virtual adapter.

If, in the end, the community wants this merged into the kernel, then
the community will have spoken and things will go the way that they
should.

Meanwhile, you don't need to be included in the vanilla kernel if you
want to be used. People have gcc toolchains and they can build the
module themselves if they want it.

Let's not worry about politics or kernel inclusion -- let's just work
on our projects. The work speaks for itself best.

I hope you take these words as positive encouragement, regardless of
what I think, personally, about merging to the kernel -- that is an
entirely separate issue.

Best regards,

Michael Krufky

[Mauro Carvalho Chehab]

On 05-12-2011 12:28, HoP wrote:

>> And here is a new hack.
>
> I'm really tired from all those "hack, crap, pigback ..." wordings.
>
> What exactly vtuner aproach does so hackish (other then exposing
> DVB internals, what is every time made if virtualization support is developing)?
>
> The code itself no need to patch any line of vanilla kernel source, it even
> doesn't change any processing of the rest of kernel, it is very simple
> driver/code/whatever.
>
> I can understand that some developers don't like to get dvb-core opened,
> I don't agree with them, but I don't really need to fight with them.
> Its theirs opinion.
> But even I try to see what is so hackish in vtuner implementation, I don't catch
> anything. Simplicity? I thought that simplicity is a big advanatge (simple code,
> easy to analyze). What else?

You solution is called a hack becaus it is used to fool applications to think that a
remote device is actually a local device. Being it at kernelspace or at userspace
(with CUSE, LD_PRELOADER of whatever other solution) doesn't change this fact.

As applications know nothing about it, they'll still use timeouts and other coding logic
that fits well for a locally-accessible device, but that won't fit well for a remote
connection. You should remind that, by remote, this could mean a cross-ethernet
connection dedicated to it or a 20,000 Km far away machine, using satellite links.

When you put someone via the network, issues like latency, package drops, IP
congestion, QoS issues, cryptography, tunneling, etc should be taken into account
by the application, in order to properly address the network issues.

Those network-specific issues affect applications badly, if they're not prepared to
handle with that, and if the used protocol is not adequate. It is not for a reason that
there are lots of different protocols designed for streaming broadcast, and a lot of
money is spent on improving it.

Can you warrant that 100% of DVB applications you claimed to support will work well
if a high latency network with a few satellite links is used, even if it has enough
bandwidth)? If you think so, a typical satellite latency is 240 ms per link, in
the best case [1]. A round trip takes twice this time.

So, a 10 GB connection with just one satellite link would mean about 480 ms of round trip.
I doubt that scan or w_scan would support it. Even if it supports, that would mean that,
for each ioctl that would be sent to the remote server, the error code would take 480 ms
to return. Try to calculate how many time w_scan would work with that. The calculus is easy:
see how many ioctl's are called by each frequency and multiply by the number of frequencies
that it would be seek. You should then add the delay introduced over streaming the data
from the demux, using the same calculus. This is the additional time over a local w_scan.

A grouch calculus with scandvb: to tune into a single DVB-C frequency, it used 45 ioctls.
Each taking 480 ms round trip would mean an extra delay of 21.6 seconds. There are 155
possible frequencies here. So, imagining that scan could deal with 21.6 seconds of delay
for each channel (with it doesn't), the extra delay added by it is 1 hour (45 * 0.48 * 155).

On the other hand, a solution like the one described by Florian would introduce a delay of
480 ms for the entire scan to happen, as only one data packet would be needed to send a
scan request, and one one stream of packets traveling at 10GB/s would bring the answer
back.

Note: even links without satellite may suffer high delays. The delay introduced by an USB
modem for a 3G data connection is probably high enough to cause applications like w_scan
to fail.

Your approach for sure works on your network scenario, but it is a very sensitive to network
issues, as applications have no idea about the network connection, and would do the wrong
thing when a network error occurs.

What I'm saying you is that the proper way to address it is to create a library that would
abstract the DVB operations (scan, tuning, filtering, ...) into a way that it could
be called either locally or remotely. Then, work with userspace developers to support it.

As I said before, all network-based audio/video stream solutions work this way. They don't
try to hide themselves by emulating a local device, as this don't work in a general case.

If you succeed to create a clean, properly written DVB library that would work fine for
remote devices, I'm sure you will be able to convince most application maintainers to use it.

Regards,
Mauro

[1] http://www.satsig.net/latency.htm

[Andreas Oberritter]

See (*) below.

> You should remind that, by remote, this could mean a
> cross-ethernet
> connection dedicated to it or a 20,000 Km far away machine, using
> satellite links.
>
> When you put someone via the network, issues like latency, package
> drops, IP
> congestion, QoS issues, cryptography, tunneling, etc should be taken
> into account
> by the application, in order to properly address the network issues.

Are you serious? Lower networking layers should be transparent to the
upper layers. You don't implement VPN or say TCP in all of your
applications, do you? These are just some more made-up arguments which
don't have anything to do with the use cases I explained earlier.

> Those network-specific issues affect applications badly, if they're not
> prepared to
> handle with that, and if the used protocol is not adequate. It is not
> for a reason that
> there are lots of different protocols designed for streaming broadcast,
> and a lot of
> money is spent on improving it.

(*) DVB isn't local either. It's using broadcast media after all. It's
designed to cope with packet loss and includes relative decoding and
presentation time stamps in its media payloads. Although DVB specifies
some minimal repetition rates for repeatedly sent data, real life
applications need to handle much lower repetition rates anyway (i.e.
higher timeouts), in order to cope with individual settings of
broadcasters trying to save some bandwidth in their multiplexes. Real
life applications also need to handle high timeouts due to possible
packet loss, especially in satellite and terrestrial networks.

However, the TS transport between the client and server components may
very well use one of the protocols you're referring to, yes.

> Can you warrant that 100% of DVB applications you claimed to support
> will work well
> if a high latency network with a few satellite links is used, even if it
> has enough
> bandwidth)? If you think so, a typical satellite latency is 240 ms per
> link, in
> the best case [1]. A round trip takes twice this time.
>
> So, a 10 GB connection with just one satellite link would mean about 480
> ms of round trip.

Again, are you serious? Virtually all users I expect will use their
local network and you're using a satellite link as an example to explain
why you think it's going to suck for everybody? I don't know anybody
who's voluntarily using a satellite link for his internet connection.
Roundtrip time in my LAN is about 0.25 ms, which is 1/1920th of your
"typical" example.

I also don't understand how your 10 GB (sic!) of bandwidth come into
play, if only few MBit/s are required to transfer a DVB video service
(or few KBit/s for audio or data). You're suggesting high bandwidth
usage to uninformed readers, where maybe only few bytes are to be
transferred.

You're suggesting that this project is still a theory, but it's already
there and it's already in use by many people, who are happy with it. So
latency can't be such a big problem as you're trying to point out.

Regarding increased latency for ioctls: The Linux DVB API doesn't
specifiy any time frames within which ioctls must return. Most ioctls
are known to block, the few others handle non-blocking mode. Adding some
ms delay for the network transfer really is no problem at all.

Regarding the asynchronous nature of networking: USB transfers are
asynchronous too, but USB tuners work well using in-kernel drivers. The
usbip even combines both asynchronous worlds. Oh wait, usbip already
resides in staging. How come?

[stripped off-topic latency calculations for satellite IP networks]

> On the other hand, a solution like the one described by Florian would
> introduce a delay of
> 480 ms for the entire scan to happen, as only one data packet would be
> needed to send a
> scan request, and one one stream of packets traveling at 10GB/s would
> bring the answer
> back.

What you didn't take into account: Whatever you'll try to do
interactively over a satellite link (or even over most 3G connections),
it sucks. Florian's solution won't be an exemption, no matter how it's
implemented.

Have you ever tried SSH over a satellite link? It sucks. Does SSH suck
because of that? Rather not.

> Note: even links without satellite may suffer high delays. The delay
> introduced by an USB
> modem for a 3G data connection is probably high enough to cause
> applications like w_scan
> to fail.
>
> Your approach for sure works on your network scenario, but it is a very
> sensitive to network
> issues, as applications have no idea about the network connection, and
> would do the wrong
> thing when a network error occurs.

There's no difference between network errors in this case and USB
errors/disconnects in the case of USB tuners.

> What I'm saying you is that the proper way to address it is to create a
> library that would
> abstract the DVB operations (scan, tuning, filtering, ...) into a way
> that it could
> be called either locally or remotely. Then, work with userspace
> developers to support it.
>
> As I said before, all network-based audio/video stream solutions work
> this way. They don't
> try to hide themselves by emulating a local device, as this don't work
> in a general case.
>
> If you succeed to create a clean, properly written DVB library that
> would work fine for
> remote devices, I'm sure you will be able to convince most application
> maintainers to use it.

Unfortunately, you still wouldn't have solved the issue vtunerc is
trying to solve, if you built such a library and included it in szap et al.

Regards,
Andreas

[Alan Cox]

The USB case is quite different because your latency is very tightly
bounded, your dead device state is rigidly defined, and your loss of
device is accurately and immediately signalled.

Quite different.

Alan

[Andreas Oberritter]

On 05.12.2011 21:55, Alan Cox wrote:
> The USB case is quite different because your latency is very tightly
> bounded, your dead device state is rigidly defined, and your loss of
> device is accurately and immediately signalled.
>
> Quite different.

How can usbip work if networking and usb are so different and what's so
different between vtunerc and usbip, that made it possible to put usbip
into drivers/staging?

Regards,
Andreas

[Alan Cox]

> How can usbip work if networking and usb are so different and what's so
> different between vtunerc and usbip, that made it possible to put usbip
> into drivers/staging?

Where usbip seems to have remained for a long time without actually being
made useful or correct enough to progress. Meanwhile most remote USB
device access is at a higher level and works beautifully (eg remote USB
printing).

Case proven I think rather than the reverse

Alan

[HoP]

Andreas was excited by your imaginations and calculations, but not me.
Now you again manifested you are not treating me as partner for discussion.
Otherwise you should try to understand how-that-ugly-hack works.
But you surelly didn't try to do it at all.

How do you find those 45 ioctls for DVB-C tune? I still see only one ioctl
FE_SET_FRONTEND or v5+ variant FE_SET_PROPERTY.

Sorry, but, for example, szap tunes very close to local variant:

# time szap -c channels.conf -x Ocko
reading channels from file 'channels.conf'
zapping to 15 'ocko':
sat 0, frequency = 12168 MHz V, symbolrate 27500000, vpid = 0x00a0,
apid = 0x0050 sid = 0x1451
using '/dev/dvb/adapter0/frontend0' and '/dev/dvb/adapter0/demux0'
status 1f | signal d410 | snr d380 | ber 00000000 | unc 00000000 | FE_HAS_LOCK
0.00user 0.00system 0:01.33elapsed 0%CPU (0avgtext+0avgdata 2000maxresident)k
0inputs+0outputs (0major+171minor)pagefaults 0swaps

[HoP]

Hi Michael

2011/12/5 Michael Krufky <mkr...@linuxtv.org>:

I really appreciate your email.

Believe me that Mauro's NACK is not reason for me to stop work on vtuner.
Fortunatelly the code is very compact and not need for any kernel patching
so doing out-of-tree development is possible and works very well.

I hope for one day when the code gets accepted somehow. In the meantime
I'm going to place some notes about development progress time to time here.
Next time w/o any word regarding code inclusion :-)

Regards

Honza

[Mark Brown]

On Mon, Dec 05, 2011 at 10:20:03PM +0100, Andreas Oberritter wrote:
> On 05.12.2011 21:55, Alan Cox wrote:
> > The USB case is quite different because your latency is very tightly
> > bounded, your dead device state is rigidly defined, and your loss of
> > device is accurately and immediately signalled.

> > Quite different.

> How can usbip work if networking and usb are so different and what's so
> different between vtunerc and usbip, that made it possible to put usbip
> into drivers/staging?

USB-IP is a hack that will only work well on a tightly bounded set of
networks - if you run it over a lightly loaded local network it can
work adequately. This starts to break down as you vary the network
configuration.

[Mark Brown]

On Mon, Dec 05, 2011 at 09:41:38PM +0100, Andreas Oberritter wrote:
> On 05.12.2011 18:39, Mauro Carvalho Chehab wrote:

> > When you put someone via the network, issues like latency, package
> > drops, IP
> > congestion, QoS issues, cryptography, tunneling, etc should be taken
> > into account
> > by the application, in order to properly address the network issues.

> Are you serious? Lower networking layers should be transparent to the
> upper layers. You don't implement VPN or say TCP in all of your
> applications, do you? These are just some more made-up arguments which
> don't have anything to do with the use cases I explained earlier.

For real time applications it does make a big difference - decisions
taken at the application level can greatly impact end application
performance. For example with VoIP on a LAN you can get great audio
quality by using very little compression at the expense of high
bandwidth and you can probably use a very small jitter buffer. Try
doing that over a longer distance or more congested network which drops
packets and it becomes useful to use a more commpressed encoding for
your data which may have better features for handling packet loss, or to
increase your jitter buffer to cope with the less reliable transmit
times.

[Andreas Oberritter]

On 06.12.2011 12:21, Mark Brown wrote:
> On Mon, Dec 05, 2011 at 09:41:38PM +0100, Andreas Oberritter wrote:
>> On 05.12.2011 18:39, Mauro Carvalho Chehab wrote:
>
>>> When you put someone via the network, issues like latency, package
>>> drops, IP
>>> congestion, QoS issues, cryptography, tunneling, etc should be taken
>>> into account
>>> by the application, in order to properly address the network issues.
>
>> Are you serious? Lower networking layers should be transparent to the
>> upper layers. You don't implement VPN or say TCP in all of your
>> applications, do you? These are just some more made-up arguments which
>> don't have anything to do with the use cases I explained earlier.
>
> For real time applications it does make a big difference - decisions
> taken at the application level can greatly impact end application
> performance. For example with VoIP on a LAN you can get great audio
> quality by using very little compression at the expense of high
> bandwidth and you can probably use a very small jitter buffer. Try
> doing that over a longer distance or more congested network which drops
> packets and it becomes useful to use a more commpressed encoding for
> your data which may have better features for handling packet loss, or to
> increase your jitter buffer to cope with the less reliable transmit
> times.

Can you please explain how this relates to the topic we're discussing?

[Andreas Oberritter]

On 06.12.2011 12:18, Mark Brown wrote:
> On Mon, Dec 05, 2011 at 10:20:03PM +0100, Andreas Oberritter wrote:
>> On 05.12.2011 21:55, Alan Cox wrote:
>>> The USB case is quite different because your latency is very tightly
>>> bounded, your dead device state is rigidly defined, and your loss of
>>> device is accurately and immediately signalled.
>
>>> Quite different.
>
>> How can usbip work if networking and usb are so different and what's so
>> different between vtunerc and usbip, that made it possible to put usbip
>> into drivers/staging?
>
> USB-IP is a hack that will only work well on a tightly bounded set of
> networks - if you run it over a lightly loaded local network it can
> work adequately. This starts to break down as you vary the network
> configuration.

I see. So it has problems that vtunerc doesn't have.

[Mauro Carvalho Chehab]

On 06-12-2011 10:01, Andreas Oberritter wrote:
> On 06.12.2011 12:18, Mark Brown wrote:
>> On Mon, Dec 05, 2011 at 10:20:03PM +0100, Andreas Oberritter wrote:
>>> On 05.12.2011 21:55, Alan Cox wrote:
>>>> The USB case is quite different because your latency is very tightly
>>>> bounded, your dead device state is rigidly defined, and your loss of
>>>> device is accurately and immediately signalled.
>>
>>>> Quite different.
>>
>>> How can usbip work if networking and usb are so different and what's so
>>> different between vtunerc and usbip, that made it possible to put usbip
>>> into drivers/staging?
>>
>> USB-IP is a hack that will only work well on a tightly bounded set of
>> networks - if you run it over a lightly loaded local network it can
>> work adequately. This starts to break down as you vary the network
>> configuration.
>
> I see. So it has problems that vtunerc doesn't have.

The vtunerc has the same issues. High latency (due to high loads, high
latency links or whatever) affects it badly, and may cause application
breakages if if the device is opened are using O_NONBLOCK mode [1].

Regards,
Mauro.

[1] Btw, if some DVB ioctl currently waits in O_NONBLOCK, this is a POSIX
violation that needs to be fixed.

[Mauro Carvalho Chehab]

With strace. See how many ioctl's are called for each tune. Ok, perhaps scandvb
is badly written, but if your idea is to support 100% of the applications, you
should be prepared for badly written applications.

$strace -e ioctl scandvb dvbc-teste
scanning dvbc-teste
using '/dev/dvb/adapter0/frontend0' and '/dev/dvb/adapter0/demux0'
ioctl(3, FE_GET_INFO, 0x60a640) = 0
initial transponder 573000000 5217000 0 5
>>> tune to: 573000000:INVERSION_AUTO:5217000:FEC_NONE:QAM_256
ioctl(3, FE_SET_FRONTEND, 0x7fff5f7f2cd0) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(4, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(5, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(6, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(7, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(8, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(9, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(10, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(11, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(12, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(13, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(14, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(15, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(16, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(17, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(18, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(19, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(20, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(21, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(22, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(23, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(24, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(4, DMX_STOP, 0x1) = 0
ioctl(15, DMX_STOP, 0x1) = 0
ioctl(11, DMX_STOP, 0x1) = 0
ioctl(22, DMX_STOP, 0x1) = 0
ioctl(17, DMX_STOP, 0x1) = 0
ioctl(16, DMX_STOP, 0x1) = 0
0x0000 0x0004: pmt_pid 0x0108 (null) -- SBT (running)
0x0000 0x0005: pmt_pid 0x0250 (null) -- Globo (running)
0x0000 0x0007: pmt_pid 0x0128 (null) -- Record (running)
0x0000 0x000d: pmt_pid 0x0118 (null) -- Band (running)
0x0000 0x002e: pmt_pid 0x0148 (null) -- Cartoon Network (running, scrambled)
0x0000 0x0030: pmt_pid 0x0158 (null) -- TNT (running, scrambled)
0x0000 0x0039: pmt_pid 0x0168 (null) -- Boomerang (running, scrambled)
0x0000 0x0090: pmt_pid 0x0178 (null) -- DW-TV (running, scrambled)
0x0000 0x0098: pmt_pid 0x0188 (null) -- BBC World News (running, scrambled)
0x0000 0x00cb: pmt_pid 0x01a8 (null) -- NET Games (running)
0x0000 0x012c: pmt_pid 0x0198 (null) -- NET M�sica (running, scrambled)
0x0000 0x0133: pmt_pid 0x0022 (null) -- Pagode (running)
0x0000 0x0135: pmt_pid 0x0020 (null) -- Ax� (running)
0x0000 0x0146: pmt_pid 0x0023 (null) -- Festa (running)
0x0000 0x0156: pmt_pid 0x0024 (null) -- Trilhas Sonoras (running)
0x0000 0x015b: pmt_pid 0x0021 (null) -- Radio Multishow (running)
0x0000 0x0320: pmt_pid 0x02c4 (null) -- 01070136 (running)
0x0000 0x0321: pmt_pid 0x02c5 (null) -- 01070236 (running)
ioctl(5, DMX_STOP, 0x1) = 0
ioctl(24, DMX_STOP, 0x1) = 0
ioctl(23, DMX_STOP, 0x1) = 0
ioctl(7, DMX_STOP, 0x1) = 0
ioctl(19, DMX_STOP, 0x1) = 0
ioctl(18, DMX_STOP, 0x1) = 0
ioctl(21, DMX_STOP, 0x1) = 0
ioctl(12, DMX_STOP, 0x1) = 0
ioctl(13, DMX_STOP, 0x1) = 0
ioctl(20, DMX_STOP, 0x1) = 0
ioctl(10, DMX_STOP, 0x1) = 0
ioctl(9, DMX_STOP, 0x1) = 0
ioctl(8, DMX_STOP, 0x1) = 0
ioctl(14, DMX_STOP, 0x1) = 0
Network Name 'NET SJC'
ioctl(6, DMX_STOP, 0x1) = 0
>>> tune to: 591000000:INVERSION_AUTO:5217000:FEC_3_4:QAM_256
ioctl(3, FE_SET_FRONTEND, 0x7fff5f7f2cd0) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(4, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(5, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(6, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(7, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(8, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(9, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(10, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(11, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(12, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(13, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(14, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(15, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(16, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(17, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(18, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(19, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(4, DMX_STOP, 0x1) = 0
ioctl(9, DMX_STOP, 0x1) = 0
ioctl(10, DMX_STOP, 0x1) = 0
ioctl(12, DMX_STOP, 0x1) = 0
ioctl(11, DMX_STOP, 0x1) = 0
ioctl(14, DMX_STOP, 0x1) = 0
ioctl(8, DMX_STOP, 0x1) = 0
ioctl(7, DMX_STOP, 0x1) = 0
ioctl(13, DMX_STOP, 0x1) = 0
ioctl(15, DMX_STOP, 0x1) = 0
0x0090 0x0029: pmt_pid 0x0468 (null) -- GNT (running, scrambled)
0x0090 0x002a: pmt_pid 0x0458 (null) -- Multishow (running, scrambled)
0x0090 0x002f: pmt_pid 0x0077 (null) -- Warner Channel (running, scrambled)
0x0090 0x0042: pmt_pid 0x0478 (null) -- Canal Brasil (running, scrambled)
0x0090 0x0046: pmt_pid 0x0438 (null) -- ESPN Brasil (running, scrambled)
0x0090 0x0049: pmt_pid 0x0428 (null) -- HBO2 (running, scrambled)
0x0090 0x007c: pmt_pid 0x0488 (null) -- Premiere FC (running, scrambled)
0x0090 0x008c: pmt_pid 0x0108 (null) -- RAI (running, scrambled)
0x0090 0x0099: pmt_pid 0x0408 (null) -- CNN International (running, scrambled)
0x0090 0x013e: pmt_pid 0x0021 (null) -- Anos 80 (running)
0x0090 0x014d: pmt_pid 0x0022 (null) -- Blues (running)
0x0090 0x014e: pmt_pid 0x0023 (null) -- Rhythm & Blues (running)
0x0090 0x014f: pmt_pid 0x0020 (null) -- Standards (running)
ioctl(5, DMX_STOP, 0x1) = 0
ioctl(17, DMX_STOP, 0x1) = 0
ioctl(18, DMX_STOP, 0x1) = 0
ioctl(16, DMX_STOP, 0x1) = 0
ioctl(19, DMX_STOP, 0x1) = 0
Network Name 'NET SJC'
ioctl(6, DMX_STOP, 0x1) = 0
>>> tune to: 597000000:INVERSION_AUTO:5217000:FEC_3_4:QAM_256
ioctl(3, FE_SET_FRONTEND, 0x7fff5f7f2cd0) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(4, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(5, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(6, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(7, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(8, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(9, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(10, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(11, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(12, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(13, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(14, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(15, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(16, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(17, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(18, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(19, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(4, DMX_STOP, 0x1) = 0
ioctl(9, DMX_STOP, 0x1) = 0
ioctl(19, DMX_STOP, 0x1) = 0
ioctl(18, DMX_STOP, 0x1) = 0
ioctl(7, DMX_STOP, 0x1) = 0
0x0091 0x0010: pmt_pid 0x0e08 (null) -- TV Senado (running)
0x0091 0x0013: pmt_pid 0x0518 (null) -- Rede Vida (running)
0x0091 0x0023: pmt_pid 0x0508 (null) -- Canal Rural (running, scrambled)
0x0091 0x0038: pmt_pid 0x0578 (null) -- Disney XD (running, scrambled)
0x0091 0x003d: pmt_pid 0x0558 (null) -- Telecine Premium (running, scrambled)
0x0091 0x0041: pmt_pid 0x0568 (null) -- Telecine Cult (running, scrambled)
0x0091 0x0043: pmt_pid 0x0588 (null) -- Disney Channel (running, scrambled)
0x0091 0x005d: pmt_pid 0x0528 (null) -- Record News (running, scrambled)
0x0091 0x008d: pmt_pid 0x138f (null) -- TV5 (running, scrambled)
0x0091 0x0134: pmt_pid 0x0020 (null) -- Samba de Raiz (running)
0x0091 0x0149: pmt_pid 0x0021 (null) -- New Age (running)
0x0091 0x0151: pmt_pid 0x0022 (null) -- Jazz Contemporaneo (running)
0x0091 0x0152: pmt_pid 0x0023 (null) -- M�sica Cl�ssica (running)
ioctl(5, DMX_STOP, 0x1) = 0
ioctl(16, DMX_STOP, 0x1) = 0
ioctl(14, DMX_STOP, 0x1) = 0
ioctl(12, DMX_STOP, 0x1) = 0
ioctl(10, DMX_STOP, 0x1) = 0
ioctl(8, DMX_STOP, 0x1) = 0
ioctl(17, DMX_STOP, 0x1) = 0
ioctl(11, DMX_STOP, 0x1) = 0
ioctl(15, DMX_STOP, 0x1) = 0
ioctl(13, DMX_STOP, 0x1) = 0
Network Name 'NET SJC'
ioctl(6, DMX_STOP, 0x1) = 0
>>> tune to: 603000000:INVERSION_AUTO:5217000:FEC_3_4:QAM_256
ioctl(3, FE_SET_FRONTEND, 0x7fff5f7f2cd0) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(4, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(5, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(6, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(7, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(8, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(9, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(10, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(11, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(12, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(13, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(14, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(15, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(16, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(17, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(18, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(19, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(4, DMX_STOP, 0x1) = 0
ioctl(8, DMX_STOP, 0x1) = 0
ioctl(14, DMX_STOP, 0x1) = 0
ioctl(7, DMX_STOP, 0x1) = 0
ioctl(9, DMX_STOP, 0x1) = 0
ioctl(10, DMX_STOP, 0x1) = 0
ioctl(17, DMX_STOP, 0x1) = 0
ioctl(18, DMX_STOP, 0x1) = 0
ioctl(12, DMX_STOP, 0x1) = 0
ioctl(13, DMX_STOP, 0x1) = 0
0x0092 0x0028: pmt_pid 0x0668 (null) -- Globo News (running, scrambled)
0x0092 0x002b: pmt_pid 0x0658 (null) -- Universal Channel (running, scrambled)
0x0092 0x002c: pmt_pid 0x0628 (null) -- Nickelodeon (running, scrambled)
0x0092 0x003e: pmt_pid 0x0608 (null) -- Telecine Action (running, scrambled)
0x0092 0x003f: pmt_pid 0x0618 (null) -- Telecine Touch (running, scrambled)
0x0092 0x0058: pmt_pid 0x0688 (null) -- VH1 Mega Hits (running, scrambled)
0x0092 0x0059: pmt_pid 0x0648 (null) -- VH1 (running, scrambled)
0x0092 0x007e: pmt_pid 0x139a (null) -- Aquecimento BBB12 (running, scrambled)
0x0092 0x011d: pmt_pid 0x0678 (null) -- Sexy Hot (running, scrambled)
0x0092 0x0131: pmt_pid 0x0024 (null) -- R�dio Kids (running)
0x0092 0x0137: pmt_pid 0x0020 (null) -- Forr� (running)
0x0092 0x0148: pmt_pid 0x0023 (null) -- Lounge (running)
0x0092 0x0153: pmt_pid 0x0022 (null) -- M�sica Orquestrada (running)
ioctl(5, DMX_STOP, 0x1) = 0
ioctl(11, DMX_STOP, 0x1) = 0
ioctl(19, DMX_STOP, 0x1) = 0
ioctl(15, DMX_STOP, 0x1) = 0
Network Name 'NET SJC'
ioctl(16, DMX_STOP, 0x1) = 0
ioctl(6, DMX_STOP, 0x1) = 0
>>> tune to: 609000000:INVERSION_AUTO:5217000:FEC_3_4:QAM_256
ioctl(3, FE_SET_FRONTEND, 0x7fff5f7f2cd0) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(3, FE_READ_STATUS, 0x7fff5f7f2cfc) = 0
ioctl(4, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(5, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(6, DMX_SET_FILTER, 0x7fff5f7f1ad0) = 0
ioctl(7, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(8, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(9, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(10, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(11, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(12, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(13, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(14, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(15, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(16, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(17, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(18, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(19, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(20, DMX_SET_FILTER, 0x7fff5f7f1910) = 0
ioctl(4, DMX_STOP, 0x1) = 0
ioctl(9, DMX_STOP, 0x1) = 0
ioctl(17, DMX_STOP, 0x1) = 0
ioctl(20, DMX_STOP, 0x1) = 0
ioctl(10, DMX_STOP, 0x1) = 0
ioctl(19, DMX_STOP, 0x1) = 0
ioctl(13, DMX_STOP, 0x1) = 0
ioctl(14, DMX_STOP, 0x1) = 0
ioctl(16, DMX_STOP, 0x1) = 0
ioctl(15, DMX_STOP, 0x1) = 0
ioctl(7, DMX_STOP, 0x1) = 0
ioctl(18, DMX_STOP, 0x1) = 0
ioctl(11, DMX_STOP, 0x1) = 0
ioctl(8, DMX_STOP, 0x1) = 0
ioctl(12, DMX_STOP, 0x1) = 0
0x0093 0x000a: pmt_pid 0x0708 (null) -- NET Cidade (running)
0x0093 0x0012: pmt_pid 0x0728 (null) -- Rede Mundial (running)
0x0093 0x0014: pmt_pid 0x0738 (null) -- Rede 21 (running, scrambled)
0x0093 0x0015: pmt_pid 0x011f (null) -- TV Justi�a (running)
0x0093 0x0026: pmt_pid 0x0758 (null) -- SporTV2 (running, scrambled)
0x0093 0x0027: pmt_pid 0x0748 (null) -- SporTV (running, scrambled)
0x0093 0x0036: pmt_pid 0x0768 (null) -- FX (running, scrambled)
0x0093 0x0052: pmt_pid 0x0788 (null) -- The History Channel (running, scrambled)
0x0093 0x0060: pmt_pid 0x0778 (null) -- Fox Life (running, scrambled)
0x0093 0x0136: pmt_pid 0x0020 (null) -- Sertanejo (running)
0x0093 0x0140: pmt_pid 0x0024 (null) -- Rock Cl�ssico (running)
0x0093 0x0143: pmt_pid 0x0022 (null) -- Reggae (running)
0x0093 0x0147: pmt_pid 0x0023 (null) -- Eletr�nica (running)
0x0093 0x015d: pmt_pid 0x0021 (null) -- CBN (running)
ioctl(5, DMX_STOP, 0x1) = 0
Network Name 'NET SJC'
ioctl(6, DMX_STOP, 0x1) = 0
...

[Andreas Oberritter]

On 06.12.2011 14:10, Mauro Carvalho Chehab wrote:
> On 06-12-2011 10:01, Andreas Oberritter wrote:
>> On 06.12.2011 12:18, Mark Brown wrote:
>>> On Mon, Dec 05, 2011 at 10:20:03PM +0100, Andreas Oberritter wrote:
>>>> On 05.12.2011 21:55, Alan Cox wrote:
>>>>> The USB case is quite different because your latency is very tightly
>>>>> bounded, your dead device state is rigidly defined, and your loss of
>>>>> device is accurately and immediately signalled.
>>>
>>>>> Quite different.
>>>
>>>> How can usbip work if networking and usb are so different and what's so
>>>> different between vtunerc and usbip, that made it possible to put usbip
>>>> into drivers/staging?
>>>
>>> USB-IP is a hack that will only work well on a tightly bounded set of
>>> networks - if you run it over a lightly loaded local network it can
>>> work adequately. This starts to break down as you vary the network
>>> configuration.
>>
>> I see. So it has problems that vtunerc doesn't have.
>
> The vtunerc has the same issues. High latency (due to high loads, high
> latency links or whatever) affects it badly, and may cause application
> breakages if if the device is opened are using O_NONBLOCK mode [1].

O_NONBLOCK doesn't mean that an ioctl must consume zero time. It just
means that it should return instead of waiting for (more) data to become
available or writeable.

Mauro, if the network is broken, any application using the network will
break. No specially designed protocol will fix that.

If you want to enforce strict maximum latencies, you can do that in the
userspace daemon using the vtunerc interface. It has all imaginable
possibilities to control data flow over the network and to return errors
to vtunerc. For a DVB API application it doesn't matter whether a tuning
request fails with EIO because a USB device has been removed, a PCI
device encountered an I2C error or because the vtuner userspace daemon
returned an error.

> [1] Btw, if some DVB ioctl currently waits in O_NONBLOCK, this is a POSIX
> violation that needs to be fixed.

To the best of my knowledge, this doesn't happen.

I think we all realized some days ago that the code is not going to be
merged upstream anytime in the foreseeable future. You can stop using
such pointless arguments.

Regards,
Andreas

[Andreas Oberritter]

You don't need to wait for write-only operations. Basically all demux
ioctls are write-only. Since vtunerc is using dvb-core's software demux
*locally*, errors for invalid arguments etc. will be returned as usual.

What's left is one call to FE_SET_FRONTEND for each frequency to tune
to, and one FE_READ_STATUS for each time the lock status is queried.
Note that one may use FE_GET_EVENT instead of FE_READ_STATUS to get
notified of status changes asynchronously if desired.

Btw.: FE_SET_FRONTEND doesn't block either, because the driver callback
is called from a dvb_frontend's *local* kernel thread.

Regards,
Andreas

[Mauro Carvalho Chehab]

On 06-12-2011 11:35, Andreas Oberritter wrote:
> On 06.12.2011 14:10, Mauro Carvalho Chehab wrote:
>> On 06-12-2011 10:01, Andreas Oberritter wrote:
>>> On 06.12.2011 12:18, Mark Brown wrote:
>>>> On Mon, Dec 05, 2011 at 10:20:03PM +0100, Andreas Oberritter wrote:
>>>>> On 05.12.2011 21:55, Alan Cox wrote:
>>>>>> The USB case is quite different because your latency is very tightly
>>>>>> bounded, your dead device state is rigidly defined, and your loss of
>>>>>> device is accurately and immediately signalled.
>>>>
>>>>>> Quite different.
>>>>
>>>>> How can usbip work if networking and usb are so different and what's so
>>>>> different between vtunerc and usbip, that made it possible to put usbip
>>>>> into drivers/staging?
>>>>
>>>> USB-IP is a hack that will only work well on a tightly bounded set of
>>>> networks - if you run it over a lightly loaded local network it can
>>>> work adequately. This starts to break down as you vary the network
>>>> configuration.
>>>
>>> I see. So it has problems that vtunerc doesn't have.
>>
>> The vtunerc has the same issues. High latency (due to high loads, high
>> latency links or whatever) affects it badly, and may cause application
>> breakages if if the device is opened are using O_NONBLOCK mode [1].
>
> O_NONBLOCK doesn't mean that an ioctl must consume zero time. It just
> means that it should return instead of waiting for (more) data to become
> available or writeable.

O_NONBLOCK means (http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html#tag_16_412):


O_NONBLOCK
When opening a FIFO with O_RDONLY or O_WRONLY set:

If O_NONBLOCK is set, an open() for reading-only shall return without delay. An open() for writing-only shall return an error if no process currently has the file open for reading.

If O_NONBLOCK is clear, an open() for reading-only shall block the calling thread until a thread opens the file for writing. An open() for writing-only shall block the calling thread until a thread opens the file for reading.

When opening a block special or character special file that supports non-blocking opens:

If O_NONBLOCK is set, the open() function shall return without blocking for the device to be ready or available. Subsequent behavior of the device is device-specific.

If O_NONBLOCK is clear, the open() function shall block the calling thread until the device is ready or available before returning.

Otherwise, the behavior of O_NONBLOCK is unspecified.

Basically, syscall should not block waiting for some data to be read (or written).
The ioctl definition defines [EAGAIN] error code, if, for any reason, an
ioctl would block.

Btw, the vtunerc doesn't handle O_NONBLOCK flag. For each DVB ioctl, for example
read_snr[1], it calls wait_event_interruptible()[2], even if the application opens
it with O_NONBLOCK flag. So, it is likely that non-blocking-mode applications
will break.

[1] http://code.google.com/p/vtuner/source/browse/vtunerc_proxyfe.c?repo=linux-driver#75
[2] http://code.google.com/p/vtuner/source/browse/vtunerc_ctrldev.c?repo=linux-driver#420

> Mauro, if the network is broken, any application using the network will
> break. No specially designed protocol will fix that.

A high delay network (even a congested one) is not broken, if it can
still provide the throughput required by the application, and a latency/QoS
that would fit.

> If you want to enforce strict maximum latencies, you can do that in the
> userspace daemon using the vtunerc interface. It has all imaginable
> possibilities to control data flow over the network and to return errors
> to vtunerc.

Yes, you can do anything you want at the userspace daemon, but the
non-userspace daemon aware applications will know nothing about it, and
this is the flaw on this design: Applications can't negotiate what network
parameters are ok or not for its usecase.

> For a DVB API application it doesn't matter whether a tuning
> request fails with EIO because a USB device has been removed, a PCI
> device encountered an I2C error or because the vtuner userspace daemon
> returned an error.

When you go to network, there are several errors that are transitory. For example,
a dropped link may cause the routing protocol (RIP, BGP or whatever) to re-direct
several routes (or, on a LAN, a spanning-tree re-negotiation), causing a temporary
failure to deliver a few packets. All network-based application are written
to consider temporary failures.

This is fundamentally different than an application designed to talk directly with
the hardware, where an error is generally fatal.

>> [1] Btw, if some DVB ioctl currently waits in O_NONBLOCK, this is a POSIX
>> violation that needs to be fixed.
>
> To the best of my knowledge, this doesn't happen.
>
> I think we all realized some days ago that the code is not going to be
> merged upstream anytime in the foreseeable future. You can stop using
> such pointless arguments.

Agreed.

Regards,
Mauro

[Rémi Denis-Courmont]

Le mardi 6 décembre 2011 15:49:11 Andreas Oberritter, vous avez écrit :
> You don't need to wait for write-only operations. Basically all demux
> ioctls are write-only. Since vtunerc is using dvb-core's software demux
> *locally*, errors for invalid arguments etc. will be returned as usual.

That's a limitation, not a feature.
You should not transmit unwanted programs over the network, nor copy them to
kernel space.

--
Rémi Denis-Courmont
http://www.remlab.net/
http://fi.linkedin.com/in/remidenis

[Mark Brown]

On Tue, Dec 06, 2011 at 01:01:43PM +0100, Andreas Oberritter wrote:
> On 06.12.2011 12:21, Mark Brown wrote:
> > On Mon, Dec 05, 2011 at 09:41:38PM +0100, Andreas Oberritter wrote:

> >> Are you serious? Lower networking layers should be transparent to the
> >> upper layers. You don't implement VPN or say TCP in all of your
> >> applications, do you? These are just some more made-up arguments which
> >> don't have anything to do with the use cases I explained earlier.

> > For real time applications it does make a big difference - decisions
> > taken at the application level can greatly impact end application
> > performance. For example with VoIP on a LAN you can get great audio

> Can you please explain how this relates to the topic we're discussing?

Your assertatation that applications should ignore the underlying
transport (which seems to be a big part of what you're saying) isn't
entirely in line with reality.

[Mauro Carvalho Chehab]

Still, vtunerc waits for write operations:

http://code.google.com/p/vtuner/source/browse/vtunerc_proxyfe.c?repo=linux-driver#285

No matter if they are read or write, all of them call this function:

http://code.google.com/p/vtuner/source/browse/vtunerc_ctrldev.c?repo=linux-driver#390

That has a wait_event inside that function, as everything is directed to
the userspace.

This is probably the way Florian found to return the errors returned by
the ioctls. This driver is synchronous, with simplifies it, at the lack of
performance.

Ok, the driver could be smarter than that, and some heuristics could be
added into it, in order to foresee the likely error code, returning it
in advance, and then implementing some asynchronous mechanism that would
handle the error later, but that would be complex and may still introduce
some bad behaviors.

Regards,
Mauro.

[Andreas Oberritter]

On 06.12.2011 15:13, Mauro Carvalho Chehab wrote:
> O_NONBLOCK
> When opening a FIFO with O_RDONLY or O_WRONLY set:

^^^^ This does not apply.

[...]

> When opening a block special or character special file that supports
> non-blocking opens:
>
> If O_NONBLOCK is set, the open() function shall return without
> blocking for the device to be ready or available. Subsequent behavior of
> the device is device-specific.

This is the important part:
- It specifies the behaviour of open(), not ioctl(). I don't see a
reason why open should block with vtunerc.
- Read again: "Subsequent behavior of the device is device-specific."

> If O_NONBLOCK is clear, the open() function shall block the
> calling thread until the device is ready or available before returning.
>
> Otherwise, the behavior of O_NONBLOCK is unspecified.
>
> Basically, syscall should not block waiting for some data to be read (or
> written).

That's because open() does not read or write.

> The ioctl definition defines [EAGAIN] error code, if, for any reason, an
> ioctl would block.

Fine.

> Btw, the vtunerc doesn't handle O_NONBLOCK flag. For each DVB ioctl, for
> example
> read_snr[1], it calls wait_event_interruptible()[2], even if the
> application opens
> it with O_NONBLOCK flag. So, it is likely that non-blocking-mode
> applications
> will break.

Of course, read operations must wait until the value read is available
or an error (e.g. timeout, i/o error) occurs. Whether it's an i2c
transfer, an usb transfer or a network transfer doesn't make a
difference. Every transfer takes a nonzero amount of time.

As Honza already demonstrated, in a typical LAN setup, this takes only
few milliseconds, which with fast devices may even be faster than some
slow local devices using many delays in their driver code.

If an application breaks because of that, then it's a bug in the
application which may as well be triggered by a local driver and thus
needs to be fixed anyway.

>> Mauro, if the network is broken, any application using the network will
>> break. No specially designed protocol will fix that.
>
> A high delay network (even a congested one) is not broken, if it can
> still provide the throughput required by the application, and a latency/QoS
> that would fit.

Then neither vtunerc nor any other application will break. Fine.

>> If you want to enforce strict maximum latencies, you can do that in the
>> userspace daemon using the vtunerc interface. It has all imaginable
>> possibilities to control data flow over the network and to return errors
>> to vtunerc.
>
> Yes, you can do anything you want at the userspace daemon, but the
> non-userspace daemon aware applications will know nothing about it, and
> this is the flaw on this design: Applications can't negotiate what network
> parameters are ok or not for its usecase.

How do you negotiate network parameters with your ISP and all involved
parties on the internet on the way from your DSL line to some other
peer? Let me answer it: You don't.

>> For a DVB API application it doesn't matter whether a tuning
>> request fails with EIO because a USB device has been removed, a PCI
>> device encountered an I2C error or because the vtuner userspace daemon
>> returned an error.
>
> When you go to network, there are several errors that are transitory.
> For example,
> a dropped link may cause the routing protocol (RIP, BGP or whatever) to
> re-direct
> several routes (or, on a LAN, a spanning-tree re-negotiation), causing a
> temporary
> failure to deliver a few packets. All network-based application are written
> to consider temporary failures.

I seriously doubt that, unless "consider" means "print an error and
exit" or "all" means "some".

Anyway, such temporary failures can be handled by the userspace daemon.

> This is fundamentally different than an application designed to talk
> directly with
> the hardware, where an error is generally fatal.

Fatal or not, if you return a temporary error code like EAGAIN, for
example, that's not the case.

Do you recommend applications to just die if an ioctl fails?

Btw.: How do you handle firmware uploads via I2C in non-blocking mode?
Should applications always fail if a firmware upload that takes longer
than some ms, e.g. when tuning to a different delivery system or when
the firmware is yet to be loaded before the first ioctl may run?

Regards,
Andreas

[Andreas Oberritter]

On 06.12.2011 15:19, Mark Brown wrote:
> On Tue, Dec 06, 2011 at 01:01:43PM +0100, Andreas Oberritter wrote:
>> On 06.12.2011 12:21, Mark Brown wrote:
>>> On Mon, Dec 05, 2011 at 09:41:38PM +0100, Andreas Oberritter wrote:
>
>>>> Are you serious? Lower networking layers should be transparent to the
>>>> upper layers. You don't implement VPN or say TCP in all of your
>>>> applications, do you? These are just some more made-up arguments which
>>>> don't have anything to do with the use cases I explained earlier.
>
>>> For real time applications it does make a big difference - decisions
>>> taken at the application level can greatly impact end application
>>> performance. For example with VoIP on a LAN you can get great audio
>
>> Can you please explain how this relates to the topic we're discussing?
>
> Your assertatation that applications should ignore the underlying
> transport (which seems to be a big part of what you're saying) isn't
> entirely in line with reality.

Did you notice that we're talking about a very particular application?

VoIP really is totally off-topic. The B in DVB stands for broadcast.
There's only one direction in which MPEG payload is to be sent (using
RTP for example). You can't just re-encode the data on the fly without
loss of information.

[Mauro Carvalho Chehab]

Yes, posix is not 100% clear about what "non block" means for ioctl's, but
waiting for an event is clearly a block condition. This is different than
doing something like mdelay() (or even mleep()) in order to wait for an
specific amount of time for an operation to complete.

A vtunerc => daemon => network transfer =>daemon => vtunerc is a block condition,
as the network may return in a few ms or may not return and a long
timeout at the daemon would give an error. Also, as the daemon may be swapped
to disk (as the daemon runs on userspace), this may even involve other
blocking operations at the block layer.

> As Honza already demonstrated, in a typical LAN setup, this takes only
> few milliseconds, which with fast devices may even be faster than some
> slow local devices using many delays in their driver code.
>
> If an application breaks because of that, then it's a bug in the
> application which may as well be triggered by a local driver and thus
> needs to be fixed anyway.

It is not a bug in the application. It requested a non-block mode. The driver
is working in block mode instead. It is a driver's bug.

>>> Mauro, if the network is broken, any application using the network will
>>> break. No specially designed protocol will fix that.
>>
>> A high delay network (even a congested one) is not broken, if it can
>> still provide the throughput required by the application, and a latency/QoS
>> that would fit.
>
> Then neither vtunerc nor any other application will break. Fine.
>
>>> If you want to enforce strict maximum latencies, you can do that in the
>>> userspace daemon using the vtunerc interface. It has all imaginable
>>> possibilities to control data flow over the network and to return errors
>>> to vtunerc.
>>
>> Yes, you can do anything you want at the userspace daemon, but the
>> non-userspace daemon aware applications will know nothing about it, and
>> this is the flaw on this design: Applications can't negotiate what network
>> parameters are ok or not for its usecase.
>
> How do you negotiate network parameters with your ISP and all involved
> parties on the internet on the way from your DSL line to some other
> peer? Let me answer it: You don't.

TCP flow control mechanisms, RSVP, MPLS, IP QoS flags, ICMP messages, etc.

>>> For a DVB API application it doesn't matter whether a tuning
>>> request fails with EIO because a USB device has been removed, a PCI
>>> device encountered an I2C error or because the vtuner userspace daemon
>>> returned an error.
>>
>> When you go to network, there are several errors that are transitory.
>> For example,
>> a dropped link may cause the routing protocol (RIP, BGP or whatever) to
>> re-direct
>> several routes (or, on a LAN, a spanning-tree re-negotiation), causing a
>> temporary
>> failure to deliver a few packets. All network-based application are written
>> to consider temporary failures.
>
> I seriously doubt that, unless "consider" means "print an error and
> exit" or "all" means "some".
> Anyway, such temporary failures can be handled by the userspace daemon.
>
>> This is fundamentally different than an application designed to talk
>> directly with
>> the hardware, where an error is generally fatal.
>
> Fatal or not, if you return a temporary error code like EAGAIN, for
> example, that's not the case.
>
> Do you recommend applications to just die if an ioctl fails?

Network applications (Youtube, Skype, mplayer, you name it) have
mechanisms to handle high delays, congestion, etc. Cheating with a non-network
application to work via the network is not the same as a network-based application.
You'll never achieve the same result. Only if the network is close to a
"pure wire" (e. g., no congestion, no packet loss, etc), such solution
works.

> Btw.: How do you handle firmware uploads via I2C in non-blocking mode?
> Should applications always fail if a firmware upload that takes longer
> than some ms, e.g. when tuning to a different delivery system or when
> the firmware is yet to be loaded before the first ioctl may run?

The right thing to do is to return -EAGAIN for all syscalls while the firmware
is not loaded. Unfortunately, several driversdon't do the right thing.

Regards,
Mauro.

[Andreas Oberritter]

Please, stop writing such bullshit! Just before the call to wait_event
there's code to return from the function if waiting has not been requested.

> This is probably the way Florian found to return the errors returned by
> the ioctls. This driver is synchronous, with simplifies it, at the lack of
> performance.

The fix is easy: set the third parameter to 0. A DVB application doesn't
need to know whether SET_VOLTAGE etc. succeeded or not, because it won't
get any feedback from the switch. If tuning fails, it has to retry
anyway, even if all ioctls returned 0.

> Ok, the driver could be smarter than that, and some heuristics could be
> added into it, in order to foresee the likely error code, returning it
> in advance, and then implementing some asynchronous mechanism that would
> handle the error later, but that would be complex and may still introduce
> some bad behaviors.

There's no need to handle errors that don't occur.

Nobody said that the implementation of vtunerc was perfect. I've already
listed many things that need to be changed in order to even consider it
for merging.

[Andreas Oberritter]

On 06.12.2011 15:19, Rémi Denis-Courmont wrote:
> Le mardi 6 décembre 2011 15:49:11 Andreas Oberritter, vous avez écrit :
>> You don't need to wait for write-only operations. Basically all demux
>> ioctls are write-only. Since vtunerc is using dvb-core's software demux
>> *locally*, errors for invalid arguments etc. will be returned as usual.
>
> That's a limitation, not a feature.

You misunderstood.

> You should not transmit unwanted programs over the network, nor copy them to
> kernel space.

I agree. And nobody said that this would happen. The software demux
receives pre-filtered data. Still the ioctls are local. Just a trigger
to start or stop filters needs to be sent to the remote demux, just like
a register setting would be needed for a local hardware demux.

[Manu Abraham]

You would need a Data Link Protocol, which would be PPP of some form

http://en.wikipedia.org/wiki/Point-to-Point_Protocol

I don't think, you have much to negotiate there.

Regards,
Manu

[Manu Abraham]

On Tue, Dec 6, 2011 at 7:49 PM, Mark Brown
<bro...@opensource.wolfsonmicro.com> wrote:
> On Tue, Dec 06, 2011 at 01:01:43PM +0100, Andreas Oberritter wrote:
>> On 06.12.2011 12:21, Mark Brown wrote:
>> > On Mon, Dec 05, 2011 at 09:41:38PM +0100, Andreas Oberritter wrote:
>
>> >> Are you serious? Lower networking layers should be transparent to the
>> >> upper layers. You don't implement VPN or say TCP in all of your
>> >> applications, do you? These are just some more made-up arguments which
>> >> don't have anything to do with the use cases I explained earlier.
>
>> > For real time applications it does make a big difference - decisions
>> > taken at the application level can greatly impact end application
>> > performance.  For example with VoIP on a LAN you can get great audio
>
>> Can you please explain how this relates to the topic we're discussing?
>
> Your assertatation that applications should ignore the underlying
> transport (which seems to be a big part of what you're saying) isn't
> entirely in line with reality.

A point to be noted:

DVB itself is a network protocol; If you have a satellite network,
most likely it is a DVB network where the whole network packets
are encapsulated within DVB (MPEG TS) packets.

[HoP]

Hi Andreas

[...]

Exactly that was an intention for my first RFC - to get driver reviewed by
kernel hackers and enhance/fixe it to prepare it for merging.

I'm going to address all your findings during Xmass, so hopefully I will
spam here again after New Year :-)

I very appreciate your POV which helped me stay remembering
it is not way to the wall.

Thanks

Honza

[Mark Brown]

On Tue, Dec 06, 2011 at 03:48:27PM +0100, Andreas Oberritter wrote:
> On 06.12.2011 15:19, Mark Brown wrote:

> > Your assertatation that applications should ignore the underlying
> > transport (which seems to be a big part of what you're saying) isn't
> > entirely in line with reality.

> Did you notice that we're talking about a very particular application?

*sigh*

> VoIP really is totally off-topic. The B in DVB stands for broadcast.
> There's only one direction in which MPEG payload is to be sent (using
> RTP for example). You can't just re-encode the data on the fly without
> loss of information.

This is pretty much exactly the case for VoIP some of the time (though
obviously bidirectional use cases are rather common there's things like
conferencing). I would really expect similar considerations to apply
for video content as they certainly do in videoconferencing VoIP
applications - if the application knows about the network it can tailor
what it's doing to that network.

For example, if it is using a network with a guaranteed bandwidth it can
assume that bandwidth. If it knows something about the structure of the
network it may be able to arrange to work around choke points.
Depending on the situation even something lossy may be the answer - if
it's the difference between working at all and not working then the cost
may be worth it.

[Andreas Oberritter]

On 07.12.2011 14:49, Mark Brown wrote:
> On Tue, Dec 06, 2011 at 03:48:27PM +0100, Andreas Oberritter wrote:
>> On 06.12.2011 15:19, Mark Brown wrote:
>
>>> Your assertatation that applications should ignore the underlying
>>> transport (which seems to be a big part of what you're saying) isn't
>>> entirely in line with reality.
>
>> Did you notice that we're talking about a very particular application?
>
> *sigh*
>
>> VoIP really is totally off-topic. The B in DVB stands for broadcast.
>> There's only one direction in which MPEG payload is to be sent (using
>> RTP for example). You can't just re-encode the data on the fly without
>> loss of information.
>
> This is pretty much exactly the case for VoIP some of the time (though
> obviously bidirectional use cases are rather common there's things like
> conferencing). I would really expect similar considerations to apply
> for video content as they certainly do in videoconferencing VoIP
> applications - if the application knows about the network it can tailor
> what it's doing to that network.
>
> For example, if it is using a network with a guaranteed bandwidth it can
> assume that bandwidth. If it knows something about the structure of the
> network it may be able to arrange to work around choke points.
> Depending on the situation even something lossy may be the answer - if
> it's the difference between working at all and not working then the cost
> may be worth it.

Once and for all: We have *not* discussed a generic video streaming
application. It's only, I repeat, only about accessing a remote DVB API
tuner *as if it was local*. No data received from a satellite, cable or
terrestrial DVB network shall be modified by this application!

Virtually *every* user of it will use it in a LAN.

It can't be so hard to understand.

[Mark Brown]

On Wed, Dec 07, 2011 at 03:01:18PM +0100, Andreas Oberritter wrote:

> Once and for all: We have *not* discussed a generic video streaming
> application. It's only, I repeat, only about accessing a remote DVB API
> tuner *as if it was local*. No data received from a satellite, cable or
> terrestrial DVB network shall be modified by this application!

> Virtually *every* user of it will use it in a LAN.

> It can't be so hard to understand.

You're talking about a purely software defined thing that goes in the
kernel - it pretty much has to be able to scale to other applications
even if some of the implementation is left for later. Once things like
this get included in the kernel they become part of the ABI and having
multiple specific things ends up being a recipie for confusion as users
have to work out which of the options is most appropriate for their
application.

Really this feels like the pattern we've got with audio where we
restrict the drivers to driving hardware and there's a userspace which
wraps that and can also dispatch to a userspace implementation without
applications worrying about it. Perhaps given the current entirely in
kernel implementation a simple loopback in the style of FUSE which
bounces the kernel APIs up to userspace for virtual drivers would make
sense.

[Andreas Oberritter]

On 07.12.2011 17:10, Mark Brown wrote:
> You're talking about a purely software defined thing that goes in the
> kernel - it pretty much has to be able to scale to other applications
> even if some of the implementation is left for later. Once things like
> this get included in the kernel they become part of the ABI and having
> multiple specific things ends up being a recipie for confusion as users
> have to work out which of the options is most appropriate for their
> application.

And to repeat myself once more: No networking is to be perfomed by the
kernel in vtunerc.

[Andreas Oberritter]

On 07.12.2011 17:10, Mark Brown wrote:

> a simple loopback in the style of FUSE which
> bounces the kernel APIs up to userspace for virtual drivers would make
> sense.

That's exactly what vtunerc is.

[Patrick Dickey]

> To unsubscribe from this list: send the line "unsubscribe linux-media" in

> the body of a message to majo...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

I tend to stay out of these discussions, since like a couple of others,
I'm not a kernel developer (or hacker). However, I wanted to chime in
with my two cents here.

1. I agree that it's not acceptable to "NACK" purely for philosophical
reasons (except when it's a clear violation of a license--be that open
source or closed source (since we don't want to open ourselves up to
lawsuits).

2. In this case, there have been technical reasons provided. Granted
the developers (and people who are pro-inclusion) don't feel those are
justified, but they have been cited.

3. You'll catch more flies with honey than vinegar (in other words,
fighting with the person(s) who maintain the project will most
definitely *not* get your code included).

4 (and the reason I decided to chime in here). This email sums
everything up. Mark is pointing out that someone may want to use this in
a non LAN setting, and they may/will have problems due to the Internet
(and their specific way of accessing it). Andreas is arguing that it's
not the case.

I have to side with Mark on this one, solely because if I knew that it
would work, I'd use it to watch television when I'm traveling (as some
places don't carry the same channels that I have at home). So, I would
prove Mark's point.

Andreas, you said that "virtually EVERY (emphasis mine) user of it will
use it on a LAN". "Virtually" implies almost all-- NOT ALL. So, unless
there's some restriction in the application, which prevents it from
being used over the Internet, you can't guarantee that Mark's issues
aren't valid.

If as HoP pointed out in another reply on this thread, there's no kernel
patching required, then I suggest that you keep on developing it as a
userspace application. There's no law/rule/anything that says you can't
install your own driver in the kernel. It just won't be supported
upstream. That just means more work for you, if you want the
application to continue working in the future. Truthfully, that has it's
upsides also. If you find out about a way to improve the transmission,
you don't have to wait (and hope) that it gets included in the kernel.
You can include it in your driver.

Sorry for butting into this. You're free to flame or ignore me, as you
choose.

Have a great day:)
Patrick.

[Honza Petrouš]

2011/12/7 Patrick Dickey <pdick...@gmail.com>:

Yes, that I think we all know. But some problem is that the arguments
against it are very weak. Believe me I would prefer to work on all
hints which kernel hackers gave me after code reviewing and not
to be member of flamewar.

> 4 (and the reason I decided to chime in here).  This email sums
> everything up. Mark is pointing out that someone may want to use this in
> a non LAN setting, and they may/will have problems due to the Internet
> (and their specific way of accessing it). Andreas is arguing that it's
> not the case.
>
> I have to side with Mark on this one, solely because if I knew that it
> would work, I'd use it to watch television when I'm traveling (as some
> places don't carry the same channels that I have at home). So, I would
> prove Mark's point.

Some features are designed for LAN use. I think nobody wants to use
SMBFS over Internet. But in LAN it works perfectly stable.

> Andreas, you said that "virtually EVERY (emphasis mine) user of it will
> use it on a LAN". "Virtually" implies almost all-- NOT ALL. So, unless
> there's some restriction in the application, which prevents it from
> being used over the Internet, you can't guarantee that Mark's issues
> aren't valid.
>
> If as HoP pointed out in another reply on this thread, there's no kernel
> patching required, then I suggest that you keep on developing it as a
> userspace application.

I guess you mean by "developing like userspave app" variant
of development kernel driver out of tree.

> There's no law/rule/anything that says you can't
> install your own driver in the kernel. It just won't be supported
> upstream.  That just means more work for you, if you want the
> application to continue working in the future. Truthfully, that has it's
> upsides also. If you find out about a way to improve the transmission,
> you don't have to wait (and hope) that it gets included in the kernel.
> You can include it in your driver.

As you stated already - maintaining kernel-space code out of kernel
tree is much difficult. If anybody did any change in internal API, then
you have to catch it yourself, find the way to change your code
accordingly. If it would be in kernel, then such job is required to be
done by patch contributor.

To repeat my conclusion - I'm doing so already and seems I will
do it futher. No problem. At least as for now.

Honza

[Peter Kolta]

Honza Petrouš <jpetrous <at> gmail.com> writes:

>
> 2011/12/7 Patrick Dickey <pdickeybeta <at> gmail.com>:

> > On 12/07/2011 08:01 AM, Andreas Oberritter wrote:
> >> On 07.12.2011 14:49, Mark Brown wrote:
> >>> On Tue, Dec 06, 2011 at 03:48:27PM +0100, Andreas Oberritter wrote:
> >>>> On 06.12.2011 15:19, Mark Brown wrote:
> >>>
> > I tend to stay out of these discussions, since like a couple of others,
> > I'm not a kernel developer (or hacker). However, I wanted to chime in
> > with my two cents here.
> >
> > 1.  I agree that it's not acceptable to "NACK" purely for philosophical
> > reasons (except when it's a clear violation of a license--be that open
> > source or closed source (since we don't want to open ourselves up to
> > lawsuits).
> >
> > 2.  In this case, there have been technical reasons provided. Granted
> > the developers (and people who are pro-inclusion) don't feel those are
> > justified, but they have been cited.
> >
> > 3.  You'll catch more flies with honey than vinegar (in other words,
> > fighting with the person(s) who maintain the project will most
> > definitely *not* get your code included).
>
> Yes, that I think we all know. But some problem is that the arguments
> against it are very weak. Believe me I would prefer to work on all
> hints which kernel hackers gave me after code reviewing and not
> to be member of flamewar.
>

I took the time and went through this thread.
I like the idea of supporting the device via network, but I do not like the idea
of having a virtual driver and server separate for this.
Before adding this hackish approach you can do a better design at early stages
right now.

Why can't you just patch the necessary functions of the applications which you
pointed out to access a library? This would totally eliminate the need of the
kernel module. It would greatly enhance the usability of this feature.

> Some features are designed for LAN use. I think nobody wants to use
> SMBFS over Internet. But in LAN it works perfectly stable.
>

SMBFS does not need to deliver data at a constant bitrate, several companies are
doing that via VPNs.

>
> As you stated already - maintaining kernel-space code out of kernel
> tree is much difficult. If anybody did any change in internal API, then
> you have to catch it yourself, find the way to change your code
> accordingly. If it would be in kernel, then such job is required to be
> done by patch contributor.
>

I would not support this attempt, since right now you can do it a better way.
You even named it, streamdev for vdr why don't you improve it?
If you add some transcoding functionality to your idea you can do it all on
application level and even stream it to the internet.
The virtual device seems to be a dead end for me which makes all those things
more complicated in the end.

Peter

[Andreas Oberritter]

On 07.12.2011 22:48, Patrick Dickey wrote:
> 4 (and the reason I decided to chime in here). This email sums
> everything up. Mark is pointing out that someone may want to use this in
> a non LAN setting, and they may/will have problems due to the Internet
> (and their specific way of accessing it). Andreas is arguing that it's
> not the case.

I'm sorry if I was unclear, but I'm not doing that. Contrary, I'm sure
that people using dreamtuner (which uses vtunerc from userspace) over
the Internet will run into problems if they can't provide the necessary
bandwidth.

What I'm trying to point out is that dreamtuner is not trying to solve
these problems, because it specifically has been designed for a
different purpose.

> I have to side with Mark on this one, solely because if I knew that it
> would work, I'd use it to watch television when I'm traveling (as some
> places don't carry the same channels that I have at home).

Yes, if you knew. But you wouldn't, because when travelling, it's
unlikely that you could guarantee the necessary bandwidth all the time.
I'd highly recommend you and Mark to use a different solution than
dreamtuner for your use cases.

> So, I would
> prove Mark's point.

I wonder how.

> Andreas, you said that "virtually EVERY (emphasis mine) user of it will
> use it on a LAN". "Virtually" implies almost all-- NOT ALL. So, unless
> there's some restriction in the application, which prevents it from
> being used over the Internet, you can't guarantee that Mark's issues
> aren't valid.

It may or may not work for some people. There's no need to artificially
restrict dreamtuner to hosts on a LAN (which would be impossible anyway).

> If as HoP pointed out in another reply on this thread, there's no kernel
> patching required, then I suggest that you keep on developing it as a
> userspace application. There's no law/rule/anything that says you can't
> install your own driver in the kernel. It just won't be supported
> upstream. That just means more work for you, if you want the
> application to continue working in the future. Truthfully, that has it's
> upsides also. If you find out about a way to improve the transmission,
> you don't have to wait (and hope) that it gets included in the kernel.
> You can include it in your driver.

FWIW, It's HoP's code. I'm not developing it.

Although you seem to have noticed that all networking happens in
userspace, you're still discussing networking issues, which may or may
not be issues of dreamtuner, but provably not of vtunerc, which just
relays DVB driver calls to userspace.

Since the topic is about the inclusion of vtunerc, not dreamtuner, such
stuff really is totally off-topic, but unfortunately got brought up
again and again.

You don't drive a formula one car or a truck if you want to have a
picnic with your family, do you? I guess you'd rather choose the right
tool for this task. So should you do if you want to stream television
over the net. People complaining that they can't transport their family
in a formula one car don't help anybody. Still both formula one cars and
trucks may be useful for other purposes.

You're free to replace dreamtuner with your superior tool solving all of
Mark's issues, even without the need to change vtunerc.

So far many people jumped into this discussion, but virtually(!) nobody
took the time to understand what vtunerc actually does by looking at the
code or at the various links HoP provided.

Regards,
Andreas

[Peter martin]

One thing I have not seen addressed here is security. I've had a brief
scan of the code, but there does not appear to be any mechanism to
authenticate/authorise remote access to the vtunerc devices. In a home
entertainment environment, this is not too serious, but In a non-closed
network, for example a commercial installation in a hotel, what is to
stop a rogue client from accessing a tuner and potentially causing
mischief by changing settings, More seriously pulling a full MPTS of
decrypted content from a CI card, which would make content providers
most unhappy!

I don't know enough about kernel modules doing networking, but do the
packets go through the iptables route, and do things like DSCP still get
taken note of?

vtunerc sounds like a neat idea, but I am worried by the back doors it
opens up. Sorry if I've missed something obvious here!

Pete Martin

> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message tomaj...@vger.kernel.org
> More majordomo info athttp://vger.kernel.org/majordomo-info.html

[Alan Cox]

> I don't know enough about kernel modules doing networking, but do the
> packets go through the iptables route, and do things like DSCP still get
> taken note of?

If you go via the kernel tcp/ip stack then you'll get kernel iptables
rules, but you'd probably still want to add more security in many
environments, which means SSL which means user space and you are back to
not needing vtuner again.

Alan