Proposal for compile-time package overrides support
162 views
Skip to first unread message
Yuriy Yarosh
Jul 20, 2020, 10:52:30 AM7/20/20
to golang-nuts
Basically I've got some different tcp/ip stack implementations based on DPDK and want to be able to replace existing types and methods of the stock net package, which would allow to add DPDK support for existing apps without any amends as a complete plug'n'play.
Same goes for JSON serialization and similar non-optimized implementations.
Stdlib shouldn't be perfect, but developers should be able to use optimized implementations as a drop-in replacement when necessary.
What do you think guys ?
Same goes for JSON serialization and similar non-optimized implementations.
Stdlib shouldn't be perfect, but developers should be able to use optimized implementations as a drop-in replacement when necessary.
What do you think guys ?
Ian Lance Taylor
Jul 20, 2020, 2:38:26 PM7/20/20
to Yuriy Yarosh, golang-nuts
intend to support alternate protocols via calls like syscall.Socket
followed by os.NewFile and net.FileConn.
Ian
Yuriy Yarosh
Jul 20, 2020, 3:22:41 PM7/20/20
to golang-nuts
Basically I'm replacing net package partially rn with custom DPDK-enabled impl right now i.e. everything syscall.Socket related.
It's not about the protocols, it's about replacing whole BSD sock API and epoll/kqueue with a custom impl enterily.
I've managed to impl a PoC for this, might create a golang PR soon(-ish) after getting some opinions.
Dubbed at slack #dartarts.
It's not about the protocols, it's about replacing whole BSD sock API and epoll/kqueue with a custom impl enterily.
I've managed to impl a PoC for this, might create a golang PR soon(-ish) after getting some opinions.
Dubbed at slack #dartarts.
Ian Lance Taylor
Jul 20, 2020, 3:30:20 PM7/20/20
to Yuriy Yarosh, golang-nuts
On Mon, Jul 20, 2020 at 12:23 PM Yuriy Yarosh <void....@gmail.com> wrote:
>
> Basically I'm replacing net package partially rn with custom DPDK-enabled impl right now i.e. everything syscall.Socket related.
> It's not about the protocols, it's about replacing whole BSD sock API and epoll/kqueue with a custom impl enterily.
> I've managed to impl a PoC for this, might create a golang PR soon(-ish) after getting some opinions.
> Dubbed at slack #dartarts.
I see. I don't think it's going to be feasible to make it possible to
>
> Basically I'm replacing net package partially rn with custom DPDK-enabled impl right now i.e. everything syscall.Socket related.
> It's not about the protocols, it's about replacing whole BSD sock API and epoll/kqueue with a custom impl enterily.
> I've managed to impl a PoC for this, might create a golang PR soon(-ish) after getting some opinions.
> Dubbed at slack #dartarts.
replace the standard library net package. That's a lot of work and a
massive increase in required testing configurations for relatively
minimal gain.
Since you mentioned JSON I'll note that the encoding/json package does
not depend on the net package.
Ian
> On Monday, July 20, 2020 at 9:38:26 PM UTC+3 Ian Lance Taylor wrote:
>>
>> On Mon, Jul 20, 2020 at 7:53 AM Yuriy Yarosh <void....@gmail.com> wrote:
>> >
>> > Basically I've got some different tcp/ip stack implementations based on DPDK and want to be able to replace existing types and methods of the stock net package, which would allow to add DPDK support for existing apps without any amends as a complete plug'n'play.
>> > Same goes for JSON serialization and similar non-optimized implementations.
>> >
>> > Stdlib shouldn't be perfect, but developers should be able to use optimized implementations as a drop-in replacement when necessary.
>> >
>> > What do you think guys ?
>>
>> I'm not sure quite what you are suggesting. But the net package does
>> intend to support alternate protocols via calls like syscall.Socket
>> followed by os.NewFile and net.FileConn.
>>
>> Ian
>
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c37ffcc3-d787-446d-9812-bdfc25a3ac8en%40googlegroups.com.
Yuriy Yarosh
Jul 20, 2020, 3:47:49 PM7/20/20
to golang-nuts
> don't think it's going to be feasible to make it possible to
replace the standard library net package.
Well, I'm not proposing to replace entirely, just to add some annotations for the compiler, like use this instead of net package over here...replace the standard library net package.
> That's a lot of work and a
massive increase in required testing configurations
Reducing clusters from 32 machines to 4 is not a minimal gain, ofc it should be completely optional.
Ian Lance Taylor
Jul 20, 2020, 5:09:07 PM7/20/20
to Yuriy Yarosh, golang-nuts
On Mon, Jul 20, 2020 at 12:48 PM Yuriy Yarosh <void....@gmail.com> wrote:
>
> > don't think it's going to be feasible to make it possible to
> replace the standard library net package.
>
> Well, I'm not proposing to replace entirely, just to add some annotations for the compiler, like use this instead of net package over here...
Yes, that is also what I meant. Replace in the sense of "replace at
>
> > don't think it's going to be feasible to make it possible to
> replace the standard library net package.
>
> Well, I'm not proposing to replace entirely, just to add some annotations for the compiler, like use this instead of net package over here...
compile time."
> > That's a lot of work and a
> massive increase in required testing configurations
>
> Well, not really... it's application specific and could provide real highload capabilities for golang i.e. 300K-1M RPS per node because of Kernel bypass.
> Reducing clusters from 32 machines to 4 is not a minimal gain, ofc it should be completely optional.
See https://golang.org/issue/35283.
Yuriy Yarosh
Jul 20, 2020, 7:54:03 PM7/20/20
to golang-nuts
Reply all
Reply to author
Forward
0 new messages