How to run a go method on multiple cores ?

[Nitish Saboo]

Hi,

I want a go method to run on multiple cores.The go method is returning an object. Can we achieve this using goroutine ?

How can I achieve this ?

Thanks

[Jan Mercl]

In the first approximation, to run a a function/method f() in a
goroutine on N cores/threads, provided N is equal GOMAXPROCS, execute
`go f()` N times.

Why do you care about "stick to core"? If the goroutines have work to
do, they will get spread by the scheduler between threads/cores
anyway.

[Nitish Saboo]

Thanks Jan.

type Y struct {

M string

N string

}

func initalize() Y{

// I have a func that return an object ob type Y.

}

var obj1 Y  = go  initalize()

var obj2 Y  = go  initalize()

Let me know how to call a go routine in this case, because when I am doing it in this way I am getting compilation error.

Thanks.

[Jan Mercl]

On Mon, May 6, 2019 at 1:39 PM Nitish Saboo <nitish....@gmail.com> wrote:

> type Y struct {
>
> M string
> N string
>
> }
>
> func initalize() Y{
> // I have a func that return an object ob type Y.
> }
> var obj1 Y = go initalize()

No need to write Y above, the type will be inferred. However, see
https://golang.org/ref/spec#Go_statements The go statement is not an
expression while the `var foo = expr` requires an expression.

[Michel Levieux]

Hello,

Is:

type Y struct {

M string

N string

}

func initalize() Y{

// I have a func that return an object ob type Y.

}

var obj1 Y

go func() {

    obj1 = initalize()

}()

var obj2 Y

go func() {

    obj2 = initalize()

}()

What you are looking for?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lgo...@gmail.com]

If   func f (n int) []int {             is some fcn that requires CPU time T (large) to calculate its return value  

And I write :

func main() {

var s [ ] int; 

s1  := f(1)  ;  s1  := f(2) 

...

}

I get  results after CPU time  2 T

?? To obtain results in < 2T time,  can I write  s1  :=  go f(1)  ;  s1  := go f(2)

[Nitish Saboo]

Hi Michel,

Yes, this is what I was looking for.Thankyou..:)

Can we also find the processor on which this go routine is running?

Thanks

[Michel Levieux]

I don't know of any primitives that would return the current processor the goroutine is currently running on. Maybe if you explain more precisely what you are trying to do, we can find a better solution?

@lgodio2 no, as Jan said, the go statement is not an expression, so you can't assign with it, you need to do either:

func main() {

    var s1 TYPE

    go func () {

        s1 = f1()

    }

    s2 := f2()

}

OR

func main() {

    var s1 TYPE

    var s2 TYPE

    go func () {

        s1 = f1()

    }()

    go func() {

        s2 = f2()

    }()

}

However in the second solution, if you execute that in the main goroutine or if you need those two assignments to be done before doing something else, you might want to use synchronisation primitives such as waitgroups.

[Nitish Saboo]

Hi Michel,

I want to initialise parsing engines on two different processors.This I am achieving in the following manner:

var obj1 parser  = initalize() 

var obj2 parser

go func() {

obj2 = initalize()

}()

Post this I want to reload the engine of obj2 parser, which I am trying to achieve using the following code but I am not sure on which processor it is actually happening ?

Will it happen on the same processor where obj2 parser engine got initialised ?

go func() {

obj2.ReloadPattern(opts)

}()

Thanks

[Jan Mercl]

On Tue, May 7, 2019 at 11:27 AM Nitish Saboo <nitish....@gmail.com> wrote:

> I want to initialise parsing engines on two different processors.

Are goroutines meant where we read 'processors'? If not, than this
maybe is an XY problem[0] and please explain why you need distinct
CPUs to execute the code.

[0]: https://en.wikipedia.org/wiki/XY_problem

[Nitish Saboo]

Hi Jan,

I need two separate parsing engines to be running and I feel that is possible when it is running parallel on two different cores.

If operation on one of the parsing engine fails, I will reload the other parsing engine and do the operation.This is what I want to achieve.

I thought goroutines will help me initialising two separate parsing engines.

Thanks

[Michel Levieux]

If the only thing you want to achieve is:

main:

run engineA

run engineB

engineA:

do something

if engineB is down -> run engineB

engineB:

do something

if engineA is down -> run engineA

You don't really need the two processes to run on different cores, nor do you need them to be constantly running in parallel. You can just run both engines in separate goroutines and trust the scheduler for their management. Think of a goroutine as an abstraction. The goroutine is NOT a thread, it is NOT bound to a CPU physical core at runtime (meaning it can execute on any core, even on multiple ones in a single execution), the only thing a goroutine does is decoupling the execution of a portion of code from the other goroutines. I don't think you need to go deeper into the way goroutines work to do what you're trying to achieve :)

[lgo...@gmail.com]

Can anyone recommend a good on-line doc for goroutines ? The best one I've found so far uses a very SLOW approach to present essential info

https://gobyexample.com/goroutines

 

On Monday, May 6, 2019 at 7:11:00 AM UTC-4, Nitish Saboo wrote:

[Michel Levieux]

https://golang.org/ref/spec#Go_statements

Small but clear IMO.

That said, for the everyday-use of goroutines, there's not much to know about them. If you really want / need to go deep into the way goroutines work, there are articles written by people on the web (https://medium.com/rungo/anatomy-of-goroutines-in-go-concurrency-in-go-a4cb9272ff88, https://golangbot.com/goroutines/ and many others) and if this is still insufficient, you can read the code of the Go language. But I personally think that it is very rare to need to know more than what's given in the official documentation. :)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/52ca0b92-e335-4681-adbd-886cb69daede%40googlegroups.com.

[Michael Jones]

Perhaps you will find this blog post (and the related ones before and after) and the videos linked at the bottom to be faster.  

https://blog.golang.org/pipelines

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/52ca0b92-e335-4681-adbd-886cb69daede%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Nitish Saboo]

Thanks Michel for your detailed response.

I am initialising the syslog engine.

var obj1 parser  = initalize() 

var obj2 parser

go func() {

obj2 = initalize()

}()

if obj1 engine fails:

                go func() {

obj2.ReloadPattern(opts)

}()

My question is, will obj2.ReloadPattern reload the pattern on the second syslog-engine , how can I verify that?Because I am not maintaining the engine info. I am just having the parser object.

Will parser object obj2 go and reload the respective syslog engine that was initialised by it? I guess yes, but still need clarification on this .

Since I was not able to identify the exact syslog engine, I was trying to identify it using the processor.

Can we execute any linux command within goroutine to get some sort of info on this ?

Thanks

[Marvin Renich]

* Nitish Saboo <nitish....@gmail.com> [190507 14:07]:

> Thanks Michel for your detailed response.
> I am initialising the syslog engine.

Do you mean "log/syslog" from the standard library? What does
initialize do?

> var obj1 parser = initalize()
> var obj2 parser
> go func() {
> obj2 = initalize()
> }()

You have initialized obj1, then you create a go routine that initializes
obj2. Assuming the initialization step takes a relatively short amount
of time and terminates, and does not itself create another go routine,
obj2 is not "on another go routine" (or thread or CPU or anything else).
It is simply another variable whose value was assigned by a different go
routine, but that go routine terminated immediately after obj2 was given
its value.

Note that obj2 is a variable, not a running function; it is not
associated with a specific go routine, at least not in the sense that
you appear to be thinking. If obj2 is declared at the package level,
than any function in the package can reference it regardless of the go
routine in which that function is running.

> if obj1 engine fails:
> go func() {
> obj2.ReloadPattern(opts)
> }()
>
> My question is, will obj2.ReloadPattern reload the pattern on the second
> syslog-engine , how can I verify that?Because I am not maintaining the
> engine info. I am just having the parser object.
> Will parser object obj2 go and reload the respective syslog engine that was
> initialised by it? I guess yes, but still need clarification on this .

A new go routine will be created (possibly unnecessarily, but you don't
give enough information), and the ReloadPattern method of obj2 will be
invoked in that go routine. When ReloadPattern terminates, the go
routine will also terminate. obj2 is still a package-level variable (or
associated with whatever scope in which it was declared).

> Since I was not able to identify the exact syslog engine, I was trying to
> identify it using the processor.

Since you don't give enough information, I'm going to take some guesses,
which could be completely wrong. I am guessing that you are using the
standard library "log/syslog" package, and you have encapsulated the
result of either NewLogger, New, or Dial from that package in your own
structure which is returned by initialize. Your structure has
additional fields and methods, and some of the methods call appropriate
methods on the value obtained from the syslog package.

I don't know what your structure has or what initialize does, but the
result of one of those functions from syslog is not an "engine" in the
sense of code that runs in a loop waiting for requests to do some work.
The result of those functions from syslog is a variable containing
state. When you invoke a method on one of those results, the method
runs in the current go routine, not the go routine where NewLogger, New,
or Dial was run.

> Can we execute any linux command within goroutine to get some sort of info
> on this ?

You seem to have some misconceptions about Go as a whole. In Java,
which is an object oriented language, you can create a Thread object,
which lets you manipulate a new OS thread from within another, already
existing, thread.

First, Go is not an object oriented language, though you can (and it is
often convenient to) use object oriented paradigms, just like you can
use object oriented paradigms from C.

Second, go routines are not OS threads. Different go routines are not
guaranteed to run on different OS threads, though the Go scheduler is
allowed to use multiple OS threads to run multiple go routines at the
same time. Also, a specific go routine is not guaranteed to remain on a
specific OS thread, unless you use runtime.LockOSThread. However, I
would not recommend that you do so until your understanding of go
routines is much more solid that it appears to be; you are unlikely to
obtain the results you expect.

Third, Go does not natively provide any means to identify specific go
routines. This was an intentional language design. Such identification
is almost never necessary, and in other languages is a frequently abused
feature. Two of the common means in Go for one go routine to provide
information about its own state to another go routine are channels and
sync.WaitGroup, but one go routine cannot "inspect" another without
prior arrangement between the two.

It is unclear what you are trying to accomplish by having two "syslog
engines". I get the vague impression that you want something like "high
reliability" so that at least one "engine" is always running and can
restart the other if the other fails. However, there is no indication
of what might cause one to fail while the rest of your Go program is
still running.

If you are connecting to multiple syslog providers on distinct hosts
using syslog.Dial, and trying to recover from an outage on one of the
remote hosts, you are very likely better off having one log variable
(obj1 from above?) that manages multiple *syslog.Writer values, perhaps
with a select statement. No go routines would necessarily be involved.

Again, this is all wild conjecture based on a dearth of information.

...Marvin

[Nitish Saboo]

Hi Marvin,

Thanks for your response.

"Do you mean "log/syslog" from the standard library?  What does
initialize do?"

>>I have installed syslog-ng parser on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data.

so initialise() method in my Go code calls  C code that is wrapper around syslog-ng C header files.So ,I am making use of cgo

Following is my Go code:

func (obj Syslogparser) initialise{

X := C.CString(obj.X)

defer C.free(unsafe.Pointer(X))

Y := C.CString(obj.Y)

defer C.free(unsafe.Pointer(Y))

C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); 

}

And following is my C method 'initialize_engine':

int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function

{

// does some operation for initialising the engine of syslog-ng parser

  return 0;

}

1)I want two instances of syslog-ng engine running on two different processors.How can I achieve this?

2)I made the following change :

func main {

var obj1 Syslogparser  = initalize() // This initialises syslog-ng engine

var obj2 Syslogparser    // Not a package-level variable

obj1.ParseMessage(program,msg)// this will parse the data and get the result

if result["Z"] == "unknown"{ // On the basis of this result I want to initialise a new engine

go func() {

         opts := Syslog{}

obj2 = initalize()

obj2.ReloadPatternDB(opts)

obj2.ParseMessage(program, msg)

}()

}

}

Will this initialise two different syslog-ng engines for me ?

3)I understand goroutines are not OS threads and they just run on OS threads.Can we extract the process id of the running syslog-ng engines within the go routine where it spawned the syslog-ng engine ?

Thanks

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20190507213138.deg3xrvoj4osdprh%40basil.wdw.

[Nitish Saboo]

Hi Marvin,

Thanks for your response.

"Do you mean "log/syslog" from the standard library?  What does
initialize do?"

>>I have installed syslog-ng parser on my Linux box and I am planning you use syslog-ng parser and wanted to initialise it's engine for parsing the data.

so initialise() method in my Go code calls  C code that is wrapper around syslog-ng C header files.So ,I am making use of cgo

Following is my Go code:

func (obj Syslogparser) initialise{

X := C.CString(obj.X)

defer C.free(unsafe.Pointer(X))

Y := C.CString(obj.Y)

defer C.free(unsafe.Pointer(Y))

C.initialize_engine(X, Y, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo))); 

}

And following is my C method 'initialize_engine':

int initialize_engine(const gchar* X, const gchar* Y, key_value_cb cb) // cb is callback function

{

// does some operation for initialising the engine of syslog-ng parser

  return 0;

}

1)I want two instances of syslog-ng engine running on two different processors.How can I achieve this?

2)I made the following change :

func main {

var obj1 Syslogparser  = initalize() // This initialises syslog-ng engine.call to a C library via cgo

var obj2 Syslogparser    // Not a package-level variable

obj1.ParseMessage(program,msg)// this will parse the data and get the result

if result["Z"] == "unknown"{ // On the basis of this result I want to initialise a new engine

go func() {

         opts := Syslog{}

obj2 = initalize() //call to a C library via cgo

obj2.ReloadPatternDB(opts)  //call to a C library via cgo

obj2.ParseMessage(program, msg) //call to a C library via cgo

}()

}

}

Will this initialise two different syslog-ng engines for me ?

3)I understand goroutines are not OS threads and they just run on OS threads.Can we extract the process id of the running syslog-ng engines within the go routine where it spawned the syslog-ng engine ?

Thanks

On Wed, May 8, 2019 at 3:01 AM Marvin Renich <mr...@renich.org> wrote:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20190507213138.deg3xrvoj4osdprh%40basil.wdw.

[Michael Jones]

Can you share a little more of your motivation for saying: “I want two instances of syslog-ng engine running on two different processors.”

Do you mean (minimum) that you want two independent instances, or (maximum) that you want to prove that there are two instances, two physical processors, and that at no time does processor “a” execute code on behalf of instance “b” and likewise for processor “b” and instance “a”, (or something else completely).

This “proof” idea seems very unusual but also seems to be what you want. Maybe it is not. That’s why it might be good to share more of your thinking and not just the conclusion. 

The natural thing in most every situation with modern CPUs, OSs, and non hard real-time applications, is just having enough total CPU resources for the mix of work and enough independence (concurrency opportunity) in the application so that the load can be magically balanced. 

If this is not enough for you then you must have special circumstances. 

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALjMrq7SbZ_pYWjhy1Xu%3DCC5U4RRonDqufFv%3D9pJ%3D4JC8tZ_CQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Nitish Saboo]

Hi Michael,

Yes, I want (exactly) two instances of syslog-ng engines running since I initialised the engine twice.And I guess it is possible only when the syslog-ng engines are running on two different processors.I might be wrong here, please correct me if my understanding is not right.

My goal right now is how to verify if two independent syslog-ng engines are getting initialised or not ? 

Once this verification is done, Can we hook instance 'a' to syslog-ng engine on processor 'a' and instance 'b' to syslog-ng engine on processor 'b'.

I hope I am able to clarify my thinking here.

Thanks

[Kurtis Rader]

On Wed, May 8, 2019 at 10:17 PM Nitish Saboo <nitish....@gmail.com> wrote:

Yes, I want (exactly) two instances of syslog-ng engines running since I initialised the engine twice.And I guess it is possible only when the syslog-ng engines are running on two different processors.I might be wrong here, please correct me if my understanding is not right.

My goal right now is how to verify if two independent syslog-ng engines are getting initialised or not ? 

Once this verification is done, Can we hook instance 'a' to syslog-ng engine on processor 'a' and instance 'b' to syslog-ng engine on processor 'b'.

There may be a human language barrier here in as much as my native language is English and I suspect your native language is something else. Whether or not you instantiate two distinct syslog-ng engines is unrelated to which CPUs (processors) each runs on. There is no way, in general, for the Go scheduler to know which CPU a goroutine is running on. Nor is there any way to bind a goroutine to a specific CPU as far as I know. You could (in theory) create two distinct instances of the syslog-ng engine and they could still execute on the same CPU. If or no other reason than that your system only has a single CPU core available for your process.

> I hope I am able to clarify my thinking here.

No, sorry, you have not. If anything every reply you make increases the confusion about what you are trying to do. If your `initialize_engine()` method always instantiates, and returns, a new instance of the "syslog-ng" engine then the two Go objects initialized by that call will be two independent instances of that engine. Assuming, of course, that your syslog-ng engine does not share state between distinct `initialize_engine()` calls.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Nitish Saboo]

Hi Kurtis,

Thanks for the clarification .initialize_engine() is a C code and is being called from go method 'initialize()'.

If you see my reply 2 mails before, 'initialize_engine()` method starts a syslog-ng engine but does not return the instance of it.

And therefore there was a confusion if calling 'initialize_engine' twice is actually instantiating two independent syslog-ng instances or not.

Thanks

[Robert Engels]

I believe you can use lockOSThread to pin a processor to the current thread, then use native code to pin the thread to a particular processor, change its priority, etc. I haven’t tried it but I see no reason it wouldn’t work - similar to how it’s not supported in Java but you can still do it - watch out for priority inversion and scheduling deadlocks !

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALjMrq7jyZmDEwTxbQ-m-WUubP-A2_QWw-8UZhUbi%2BcLd1wpOQ%40mail.gmail.com.

[Marvin Renich]

* Nitish Saboo <nitish....@gmail.com> [190508 05:48]:

Please remove me from the CC on this thread; I am subscribed to the
list. I explicitly set Reply-To, which was not respected, and now I am
getting duplicates of every mail, with one copy in my personal mail and
one copy in my list mail folder. In general, it is a very good
assumption that someone posting to the list, unless he/she explicitly says
otherwise, is subscribed to the list and does not need an explicit CC.
In some cases, like for me, not only is this not needed, but it is an
unwanted distraction.

I intend to, but have not yet had time, reply to Nitish Saboo with some
questions that will help him provide the right information so that we
may give him some more help. It may be several days before I can do so.

Thank you...Marvin

[Nitish Saboo]

Apologies Marvin.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20190509132302.wybkhuf3rfrgedeb%40basil.wdw.

[Marvin Renich]

Sorry for the much delayed response. I have not had enough time lately!

* Nitish Saboo <nitish....@gmail.com> [190508 05:48]:

> "Do you mean "log/syslog" from the standard library? What does
> initialize do?"
>
> >>I have installed syslog-ng parser on my Linux box and I am planning you
> use syslog-ng parser and wanted to initialise it's engine for parsing the
> data.

There are two general aspects to your problem that are unclear. First,
what do you mean by "syslog-ng engine" or "syslog-ng parser" and how
does it run from within the C code? Second, what underlying problem are
you trying to solve that you believe running two of these "engines" on
different CPUs will solve?

In previous messages, you had just said "syslog engine" and gave no
indication that non-Go code was involved. Clarifying in your last
message that you are using C code with cgo is a good first step, but we
need more information.

I was thinking of three different things you might mean by "syslog
engine". The first is something that accepted messages from Go code in
this program and sent them to an external syslog daemon to be logged.
The second is something running in this program that acted as a
replacement for a standard syslog daemon. The third is something that
takes the output of a syslog daemon (the log files) and parses them to
allow analysis. A fourth possibility comes to mind, now that you have
specified that you are using syslog-ng through C code: a syslog-ng
plugin that is somehow called by an external syslog-ng daemon during its
handling of messages to be logged.

This fourth possibility now seems to be the most likely, based on your
last message, so I will focus my questions in this direction. Does the
initialize_engine function return after the initialization completes,
before any actual processing occurs, or does it keep running until that
"engine" has completed all its processing? If the former, how does the
"engine" ask your program to perform some processing? Is the
communication with the syslog-ng daemon hidden in a library linked to
your program, and the library invokes callbacks on a C function that you
provided, or does the library give you an open socket that you read and
write to handle processing requests from the daemon? Is the library
multi-thread aware? Does it create its own threads?

Regardless of which of the four possibilities (or others I haven't
thought of) is appropriate, the important question is how does the code
in initialize_engine arrange for processing of requests? Does it spawn
a new process? Does it create a new thread? Does it use C callback
functions? Does it use file descriptors, such as Unix sockets?

The second aspect, _why_ are you trying to run each engine on its own
CPU, is one that others in this thread have asked about, but your answer
has been "Yes, I want to run each engine on a separate CPU", which does
not give the type of information that helps us give you a good answer to
your question. You might be trying to provide fail-over, or load
balancing, or something else. Do you really need to lock each "engine"
on its own CPU, or is having two concurrently running "engines", letting
the Go scheduler take care of the details of OS threads and CPUs,
sufficient?

Keep in mind that when the function invoked as the go routine
terminates, Go no longer "knows" about the go routine. Any stack
allocated by Go for that go routine will be freed, and no further
scheduling of that sequence of instructions will occur. If the go
routine only calls Go code, this is simple enough, but when the go
routine calls C code which creates its own threads, you need to have a
deeper understanding of both cgo and the Go scheduler.

So, the LockOSThread call in the following might be completely
superfluous:

go func() {
runtime.LockOSThread()
C.initialize_engine(some, arguments)
}()

It will lock the go routine onto a single OS thread for the duration of
initialize_engine, but it may not have any effect on whatever "engine"
was started by initialize_engine after initialize_engine returns.

In fact, if initialize_engine, as part of creating the "engine", creates
a new OS thread, then the go routines themselves may be completely
superfluous, and something like this sequential code:

C.initialize_engine(some, arguments) // start first engine
C.initialize_engine(other, different_arguments) // start second engine

may be perfectly suitable.

In summary, for us to help you, you need to answer these two questions:

1. What problem are you trying to solve by locking each "engine" to its
own CPU (or thread or go routine)? (Hint: "running two instances of
engine" is not a sufficient answer.)

2. How does the code in initialize_engine change the execution state
(e.g. threads and processes) and how does the Go program communicate
with the "engine" after it is started?

...Marvin