Intel Storage Performance Dev Kit

[Vitaly Davidovich]

https://github.com/spdk/spdk

Avi, any interest for scylla? :)

sent from my phone

[Avi Kivity]

You bet!

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Burton]

Nice.. this is interesting.  I have an Intel NVMe device on my workstations so might play with this.

Would be interesting to write a facade on top of this so that you can swap in old school IO subystems. 

Looking at this...

https://spdk.github.io/spdk/doc/nvme/nvme__ns__cmd_8c.html#a4c8cb5fb5d0f458260ddb4af7fc27a15

I think you could use direct maps to use legacy flash files and then swap it out with NVMe in the future.

In 24 months we're all going to be on NVMe.  Screw SATA :)

[Dan Eloff]

Hey Avi, If you end up implementing support for the SPDK, and it's at all possible, please implement it in the Seastar framework like you did for the DPDK (probably that's the level that makes the most sense anyway.) Then we can all benefit from the work.

I'm using Seastar in my current project and I really love it.

Cheers,

Dan

[Avi Kivity]

The support will definitely be in seastar, not Scylla.

Can you share what you're using seastar for?

[Dan Eloff]

A web framework of all things. I'm gluing Go on top of Seastar and trying to get them to play nice together. Seastar is much faster, and scales much better than Go's IO stack and channels. It's a hobby project of mine.

[Avi Kivity]

Cool.  Are you modifying the go runtime?

I'd love to get go to seamlessly use the native networking stack, but that requires either hacking on the go runtime, or elf magic so we can redirect go libc calls to seastar.

[Dan Eloff]

It's possible with assembly methods in Go to call the runtime internal methods, which lets one hook the scheduler in a completely hacky, fragile, and evil way. There doesn't seem to be a lot of options though, because Go does syscalls directly, it doesn't go through libc (with a couple exceptions, such as name resolution in net package.)

I'm still working out what's the least evil way of getting a performant solution. Go doesn't expose any knobs for it's scheduler really, just spawning goroutines and yielding them. 

Anyway it will be open source, so whatever I figure out will be available for others to try out and improve on.

[Francis Stephens]

Daniel, do you have this on github? I had a look and couldn't find any obvious repos for this project. Thanks.

You bet!

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Dan Eloff]

Francis, It's not on github yet, but I'll announce it on this list when it's up there. Do you have a specific use case in mind for a Seastar + Go integration?

You bet!

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Francis Stephens]

I am building a matching engine in Go (I'm rewriting the reb-black tree at the moment). So I would be very interested to see if this integration could be useful for that project.

You bet!

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Avi Kivity]

Is there no red-black tree library in go?

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Francis Stephens]

There are several on github. They all generate a lot of garbage that is hard to mitigate.

The red black tree implementation is invasive and is combination red-black tree and fifo queue, both invasive. This allows the data structure to efficiently prioritise on price/time and any push/pop operations which touch the fifo queue and don't require any rebalancing at all.

I'm trying to clean it up at the moment, but time is always an issue :)

You bet!

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Avi Kivity]

Go seems an attractive language but the lack of generics is a turn-off.

In C++ you'd just use boost::intrusive for both the tree and the fifo with no problems.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Francis Stephens]

Go is a very attractive language, but the lack of generics makes this much harder than it should be. I could generalise the intrusive data-structures fairly easily, but without good generics I would need to make references to the actual stored data an interface{} (java's Object). Casting from interface{} has big impact when the rest of your system is quite fast. So everything is very bespoke - this could change, but Go is unlikely to get generics in a hurry.

I have another problem using some (ported from Martin's) in-memory queues. Without generics I can't easily pass non-pointer values and need to heap allocate elements passed through the queue. I may opt for code generation or just pass plain bytes across. Something along the lines of Simple Binary Encoding could be valuable here.

...

[Vitaly Davidovich]

Can someone explain what's so attractive about Go? I buy the fast compile times, static linking/easy deployment, and goroutines/channels, but lack of generics is a headscratcher.  Error codes is another puzzler, although less so than lack of generics.

sent from my phone

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

...

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Avi Kivity]

Goroutines and the queues connecting them, plus a nice library.

Goroutines allow programming without locks, using a single-threaded programming.  A more modern erlang.

[Vitaly Davidovich]

How's ownership handled of messages/objects passed down a channel? Does the compiler enforce anything here or you can share mutable state across multiple threads by accident? I like Rust's take on this.

Coroutines and channels can and are provided as libs in other languages.  Having first class support for it in the language is nice, but doesn't strike me as compelling unless it also then enforces safe usage via compiler.

sent from my phone

[Avi Kivity]

I'm no go expert (never written a gogram), but I understand that it's up to the gogrammer.

Go does have some special support for goroutines -- growable, disjoint stacks (without these, the amount of goroutines would be limited), and the library translates blocking call in a goroutine to non-blocking calls.

I agree it's not revolutionary.  I guess people like it because it is simple, yet offers reasonable performance.  Something in between C++ and Java.

[Francis Stephens]

I can tell you why I like Go.

I come from an enterprise Java background, with a distaste for spring-framework universal inversion of control and various things that are very popular in Java. I used them daily, but didn't enjoy them.

Go is a very good step from Java if you want to have a garbage collector but you also want to control of memory layout with structs and arrays of values etc.

Go dispenses with some object oriented mechanisms which I have found to be complex in practice. For instance you can't do this in Go

public abstract class SubclassableHelloWorld {

   public abstrct ImplementMe() String;

   

   public void run() {

       System.out.println(ImplementMe());

   }

}

You can't write that in Go (without serious contortions), and I personally am happy about that.

http://blog.golang.org/go-slices-usage-and-internals

Slices are really excellent. you can pass around sub-blocks in a byte buffer without passing offset/length as separate fields. This can make a difference for some kinds of programming.

Errors, this one if funny. I much prefer errors, I feel a lot more comfortable writing code that fails a lot, like networking code, with error returns. Exceptions are much harder for me to reason about and I always have a nagging feeling I have missed some control flow or some exception. This blog post explains how I feel about exceptions/errors better than I can

http://blogs.msdn.com/b/oldnewthing/archive/2004/04/22/118161.aspx

Go has goroutines and channels, and they are very useful, but they aren't my favourite parts of Go.

----------

Some useful things are really easy in Go. I recently added a hot polling loop with PAUSE inside. I could add assembly into the code base without too much friction (although using an idiosyncratic assembly language). It is a very good place to play with a lot of memory layout low level concurrency games, without a too-steep learning curve.

Having said that Go has a very very simplistic memory model, so it's not all roses on that front.

Having written all of this I worry that this isn't the mailing list for debating the merits of Go. I am happy to talk about it, but I don't want to fill up the list with unwanted language spam.

F

...

[Francis Stephens]

It is entirely up the programmer. You can easily send a message and then start messing with it's state and that will have the usual undefined behaviour.

Go comes with a very easy to use, and I believe quite effective, race checker. So you don't get Rust's provable racefree guarantees, but there is a very accessible tool that will probably get you most of the way there (YMMV). On this front, Rust always feels like a tradeoff, concurrency is hard but it's also usually a small portion of the lines of code in any software I've ever written. I worry about the cost of writing extensive business logic inside such a fussy language. Although I am also excited about Rust. (maybe we should all just be programming in Erlang :)

Dmitry Vyukov was the author of that tool as well as a large number of refinements to channels and the scheduler. He is, or was, arguing for a much more detailed memory model to allow for lock-free programming etc. In time we may see Go become quite proficient at these kinds of things (maybe :)

F

...

[Greg Young]

"Errors, this one if funny. I much prefer errors, I feel a lot more
comfortable writing code that fails a lot, like networking code, with
error returns. Exceptions are much harder for me to reason about and I
always have a nagging feeling I have missed some control flow or some
exception. This blog post explains how I feel about exceptions/errors
better than I can"

I much prefer the erlang way of doing things. Explicitly state what
you handle anything else kills you.

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-symp...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> ...
>

> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-symp...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

Studying for the Turing test

[Avi Kivity]

Erlang provides strong isolation between (internal) processes; I don't
think any other language provides that, except for database stored
procedures. "Anything else kills you" requires this strong isolation.

[Francis Stephens]

I also like Erlang a lot. But it's quite hard to reproduce much of Erlang without Erlang's (really wise) process heap separation. It's harder to crash by default if you don't have a supervisor tree to save you.

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mechanical-sympathy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to mechanical-sympathy+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mechanical-sympathy+unsub...@googlegroups.com.

[Rajiv Kurian]

On Monday, November 16, 2015 at 8:59:46 PM UTC+5:30, Avi Kivity wrote:

I'm no go expert (never written a gogram), but I understand that it's up to the gogrammer.

Go does have some special support for goroutines -- growable, disjoint stacks (without these, the amount of goroutines would be limited), and the library translates blocking call in a goroutine to non-blocking calls.

 

Agreed that these are all great features built into the language/runtime. It makes Go a great fit for web-servers that end up doing a scatter-gather style request processing. The growable stacks and translation to non-blocking calls is a great productivity boon for programmers who can afford to just start a new go-routine to handle IO. You get a lot of the performance and efficiency advantages of an epoll loop but with straight line code. In C++ most of the async protocol parsing code is done with hand-written or sometimes machine generated state machines. The introduction of resumable functions might confer similar advantages to C++17.

I agree it's not revolutionary.  I guess people like it because it is simple, yet offers reasonable performance.  Something in between C++ and Java.

 

Disclaimer - I am not a Go programmer and have never written anything beyond trivial Go programs. My opinions are mostly based on working with programmers at work, where one of our components is built in Go. TLDR: I started off really disliking Go but I think it offers a great step up from Java in particular.

Just having structs is such a boon. The typical cache-trashing Java program is fixed with flyweights/unsafe code when performance is an issue. Structs, arrays of structs, slices and real pointers make this a non-issue just like it does in C++. I regularly see Go programs doing the typical scatter gather web-server thing under 100 MB while still handling tens of thousands of requests per second. This is an absolute killer middle ground that comes out of the box. Just imagine doing the equivalent in Java without serious ByteBuffer/Unsafe gymnastics. Programmers also like the low (< 20ms) pause times. GC folks like to point out how a non-compacting collector makes low GC pause times much easier. Again I'd say that in practice this is not the biggest problem for the kinds of programs that are written in Go (citation missing). Our own web servers are typically under 100 MB and run for weeks without showing any signs of memory usage creeping up from fragmentation or otherwise. Of course the lack of generics is kind of a stinker. I personally really dislike this "simplification" but Go programmers that I know seem to be fine with it. The inbuilt data structures that most programmers use (maps, channels, arrays, slices) etc are generic. So in practice (again citation missing) this is more of an annoyance and less of a crippling problem. Further there are code generators that can bridge the gap even further. Don't expect the SFINAE kind of semantics obviously. Another under-appreciated advantage is the small statically compiled binaries.  I've seen Go binaries for production servers be under 5 MB. Compare this to shipping a JVM and a bloated JAR. Great compile times and small binaries make for a great developer experience during the write/deploy/test cycle. Actually I'd like to hear from Java developers as to what big advantages they think Java has over Go. A real world advantage I can think of is the JIT's ability to optimize interfaces. Interfaces seem to be used generously in typical Go code, but AFAIK they just result in dynamic dispatch. I am unaware of how good CHA is for the Plan 9 toolchain.

...