I know you cannot kill a goroutine, but ...

[rei...@gmail.com]

Background:

UniVerse is a Pick model, multivalue database, https://groups.google.com/forum/#!forum/mvdbms for some general information.

Multivalue databases permit more than one value in a column for a row. Since you do not know the size of the column with hundreds or thousands of values in the column-row, you do not know the size of the record. Access dictates linked lists to handle variability in record size. So, every row gets a unique primary key and stored in hash tables with an appropriately large number of hash buckets for the data - key / value pairs. With good hashing, I can often access a record in a single disk access. Group (hash table bucket) locking happens within the environment, while users request row locking. Each user in UniVerse is a separate O/S process. Locking activity occurs in shared memory, using assembly test-and-set kinds of instructions or host-specific semaphores.

After retiring, I decided it would be a fun experiment to build a clean-room implementation of UniVerse. Go does not do interprocess communication at the rate that would match the shared memory semaphore activity in UniVerse. A natural match implements each user as a goroutine. UniVerse provides an extended BASIC language for accessing and manipulating table data that runs close to the engine. In a language you can do silly things like create an infinite loop. In a production environment of 1000s of users, you cannot simply bounce the environment because one user is eating a CPU. An admin function to dismiss or kill a goroutine would be ideal. Not possible in the current Go world.

For an infinite loop to exist, you need to branch "backwards" or call a routine that calls you back with some possible indirection. (An equivalent in Go is "for {}" with no break. Here, you would not get back to the traditional mechanism of telling a goroutine to shut down where one of the channels for select is the shutdown indicator.)  There may be other examples I have yet to think of. When I "compile" BASIC, I can put checks into those two functions, call, and branch (to a lower address), without inducing too much overhead. an unimportant detail is that the BASIC compiles to a p-code run on a run machine, comparable to a JVM. I might even be able to find the PC, Program Counter or IA, Instruction Address, and insert some kind of trap instruction opcode, that would cause it to try the select statement and see the shutdown channel. But in the general case, this may be insufficient. I think a "context" timeout would interrupt a properly long-running program in a way that might be hard to restart if shutdown was not requested.

As a database, there is also the possibility of deadly embrace. Killing one of the two (or more) goroutines in a deadly embrace would be useful. This normally occurs on poorly managed acquisition of exclusive locks on database table rows. I could forcibly release a lock, but then would need extra work so that a user goroutine that thinks it exclusively owns a lock, finds that it does not, and would need to gracefully exit or handle the situation.

As a goroutine, each user does not have a STDIN, and STDOUT to communicate with its user. Currently, the user communicates with the goroutine with a socket connection. I can probably redesign, adding a new goroutine accepting user requests/commands on a channel, and monitoring the shutdown channel with a select in a loop. Otherwise, I could not get a goroutine to shut down if it is waiting on user input from a socket, or a dropped connection, although that begs the question of how to get the socket-reading goroutine to now shut down.

The current Go implementation smells of cooperative multitasking. Not a bad thing, per se, but makes it hard to stop in certain degenerate cases. Have I missed a way to deal with some of the discussed issues?

The project has certainly been a good way to become familiar with a number of the Go idioms. (BTW, I happen to *like* the current way of handling errors!)

[Robert Engels]

The standard method in nearly all languages including go is to close the connection from another routine. Alternatively you can use a timeout/deadline on every operation. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/622c48dd-5909-43a9-923a-ac74a7a8f0b2%40googlegroups.com.

[Mark Baldridge]

I think I can handle the deadly embrace problem. The locks distribute into "slots" based upon inode%slotCount. Just information irrelevant to the observation. At the moment, I use RWMutex to protect each lock table slot since it was more like previous thoughts on this subject. However, I plan to convert them to goroutines, and pass a structure with user information (return channel included, as well as the lock request information). Requesting a lock requires populating appropriate fields in the struct that describe the request. Presently, the mutex will wait if someone has my lock. The channel can queue the request in that case. The return result will indicate success of failure. For a deadly embrace, a new request would say, "return that user's lock request with a failure". Once back, it could go through the channel select and see a "clean up and go away" quit signal, if I really want that user session to exit. Or it could simply return and discover the request failed, and the program could retry or handle its error as it desires.

I think the infinite loop situation will need to save some information about each backward branch or call so the object code can receive a break point. If 1000 users are all using the same code, that might not be desirable if it is just some special data case that a particular user has encountered. It would be ugly to have 1000 users hit modified object code if the special case did not apply to them. It is comparable to having them all log out, at the same time, when they are geographically distributed around the world, so you can bounce the database, which I have seen occasionally in production. Not pretty. That will require some more thought. Perhaps a way to automatically migrate things to another instance of the database.

[Jim Idle]

Good luck on your implementation. I suggest though that you will find too many issues like this to use go routines as the base model for a user. I looked in to threads as a model for jBASE and even with that model, it was going to be too fraught with difficulty. Unless you are considering a production grade implementation with 1000's of users, then I would just implement it as separate go processes. You don't have to use shared memory unless you are trying to copy Universe, but even if you did, you can still use such things in go (though mmap might a better way). 

Your bigger problem, if you are trying to provide a complete copy, is reproducing the query language and conversion processor. with all their quirks and strangeness. The compiler is easy compared to that :)

Jim

[Jesper Louis Andersen]

On Fri, Aug 9, 2019 at 8:34 PM <rei...@gmail.com> wrote:

The current Go implementation smells of cooperative multitasking. Not a bad thing, per se, but makes it hard to stop in certain degenerate cases. Have I missed a way to deal with some of the discussed issues?

My spider sense (intuition) says you might enjoy reading up on the "context" package in the standard library. The core idea is to create a "network" of poison channels on which you send messages if things needs to stop. These channels are used solely for the purpose of managing lifetime of goroutines, not for normal communication[0] As long as a goroutine checks for the poison pill eventually, you can send an event to it in order to stop it. It will require some adaptation to your needs, but I think there are salvageable ideas in there for you to pursue.

[0] Essentially, you have a bigraph: the message channels forms the hypergraph part of the bigraph, and the context tree forms one of the tree sets of the bigraph. The context tree is mapping the "location" of goroutines, i.e., who belongs together. This allows the conjuration of your inner Alexandre Dumas: "Un pour tuos, tuos pour un"

--

J.

[Jason E. Aten]

I could not get a goroutine to shut down if it is waiting on user input from a socket, or a dropped connection, although that begs the question of how to get the socket-reading goroutine to now shut down.

My usual approach is to set 100ms read deadlines and then check for the shutdown request inside the read loop.

[Robert Engels]

You have to be careful with this approach and high volume (longer latency) tcp. Often there can be TCP "stalls" and if you have a messaging type protocol, you need to make sure you can handle partial reads and re-combine, because the deadline may fire during the read (typically a problem with text based line protocols).

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ab103eb3-a3d4-40c6-90f6-75dba0fd904a%40googlegroups.com.