q re idiom for data sharing
jimr
regular basis. The intent is that this server will collect statistics
extracted from that page over a long period of time, and will offer up
a summary of that information in its own little web server.
Is it the correct Go idiom to capture poll/store logic something along
the lines of:
type Collector struct {
... long term data, e.g., a map[string]int64 ...
}
func (c *Collector) Poll() (chan *Collector, chan os.Error) {
resCh := make(chan *Collector)
errCh := make(chan os.Error)
go func() {
...
if status,err := fetchStatus(); err != nil {
errCh <- err
else {
... add status data to long term data for
c ...
resCh <- c
}
}()
return resCh, errCh
}
and then have another goroutine creating the Collector:
c := NewCreator(url)
res,err := c.Poll()
and sitting in a loop running
select {
case v := <-res:
... read long term data from Collector ...
case v := <-err:
... handle error ...
}
If this is the proper idiom? If so, if I want to then provide some
sort of overall summary from a number of Collectors, do I need to copy
out the data from the Collector within the select statement (making
the copy to ensure that the maps within the Collector aren't read from
while a poll event is updating the map)?
Jim
Andrew Gerrand
idiomatic way of doing this.
You have your Site, which is the data for a particular site:
type Site struct {
url string
data map[int64]string // for example
}
func NewSite(url string) *Site {
// set up and return a pointer to a new Site
}
Then you might think about what kind of work you want to be doing, and
define that work as a struct.
type Work struct {
site *Site
err os.Error
}
Then you create a Poll function (of which you can run many in seprate
goroutines) that read form a channel of Work, and send the updated
Work to an output channel:
func Poll(in <-chan Work, out chan<- Work) {
for w := range in {
// do some work, update w
out <- w
}
}
Then in your main function, you set up those channels, launch a bunch
of Poll workers, create some Sites, and handle the distribution of
work:
var urls = []string {
"http://example.com/",
"http://example.net/",
"http://example.org/",
}
const numWorkers = 10
func main() {
// create channels
pending, complete := make(chan Work), make(chan Work)
// launch workers
for i := 0; i < numWorkers; i++ {
go Poll(pending, complete)
}
// create some initial work
go func() {
for _, url := range urls {
s := NewSite(url)
pending <- Work{s, nil}
}
}()
// process completed work
for w := range complete {
if w.err != nil {
// handle error
}
// process completed work somehow
// now put the site back into the pending queue for re-processing
// (or don't, if you don't want to process it again)
go func(s *Site) {
// we do this in a goroutine
// because the send will block until a worker is ready
// we could also put a time.Sleep here
// if you want to wait before re-processing the site
pending <- Work{s, nil}
}(w.site)
}
}
Now I haven't run or tested this code at all, and there may be
refinements or further simplifications to the process that can be
made, but I hope this gives you the general idea of what we mean by
"share memory by communicating".
Please let me know if you have any questions! :-)
Andrew
Charles
Thanks for starting this thread. This helps me understand how to
properly share/synchronize memory among goroutines.
In this example, do the Poll goroutines terminate or return if there
are no items in the pending channels? Or do they behave like a run
loop, waiting patiently until the next item shows up in the pending
channel?
If the later is the case, how do you terminate a go routine when there
is no more work to do, i.e. protect against process leaks?
Thanks,
Charles
Andrew Gerrand
when the 'in' channel is closed. (this is the behavior of range on a
channel) In this example, to end all the Poll goroutines you'd just
call:
close(pending)
Andrew