why is wait group necessary here?

[Alexander Mills]

I tried writing this http post request which streams the request body line-by-line as JSON.

(This is for an ElasticSearch bulk api request, newline separate JSON in the request body). I didn't think the waitgroup was necessary, but apparently the waitgroup is necessary. Does anyone know why? I assume the need for the waitgroup might represent a bug in my code.

Here is the same code in a github gist: https://gist.github.com/ORESoftware/06341057ca39e1b2a47dc8a8e4e4b55f

func doBulkUpsertWithResponse(baseUrl string, z []UpsertDoc) (*map[string]interface{}, error) {

var m *map[string]interface{}

if len(z) < 1 {
return m, nil
}

rd, wr := io.Pipe() // we stream json lines to post request

wg := sync.WaitGroup{}

wg.Add(1)
go func() {

defer func() {
    // close the stream after writing is finished

    wg.Done()
    wr.Close();
}()

for _, v := range z {

var doc = ElasticBulkUpsertable{
&v.Doc, true,
}

jsonStr, err := json.Marshal(&doc)

if err != nil {
continue
}

updateLine := fmt.Sprintf(`{"update":{"_id":"%s"}}`, v.Id) + "\n"
if _, err := wr.Write([]byte(updateLine)); err != nil {
  continue
}

docLine := string(jsonStr) + "\n"
if _, err := wr.Write([]byte(docLine)); err != nil {
    continue
}
}

}()

fullUrl := fmt.Sprintf("%s/%s", baseUrl, "_bulk")
req, err := http.NewRequest("POST", fullUrl, rd)

if err != nil {
log.Println(fmt.Errorf("cm:3ec045d3-4790-4d7a-aab8-65949933d263: '%v'", err))
return nil, err
}

req.Header.Set("Content-Type", "application/json")

resp, err := client.Do(req)

wg.Wait()

if err != nil {
return m, err
}

defer func() {
    resp.Body.Close()
}()

if resp.StatusCode != 200 {
  return m, errors.New("request failed with status code: " + resp.Status)
}

if err := json.NewDecoder(resp.Body).Decode(&m); err != nil {
    return nil, err
}

return m, nil

}

[burak serdar]

On Thu, Sep 10, 2020 at 11:46 AM Alexander Mills
<alexande...@gmail.com> wrote:
>
> I tried writing this http post request which streams the request body line-by-line as JSON.
> (This is for an ElasticSearch bulk api request, newline separate JSON in the request body). I didn't think the waitgroup was necessary, but apparently the waitgroup is necessary. Does anyone know why? I assume the need for the waitgroup might represent a bug in my code.

What makes you think waitgroup is necessary?

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4d6b60cd-4b9e-4bce-aa6f-2d2bfbefbd98n%40googlegroups.com.

[Alexander Mills]

It was behaving strangely before adding the waitgroup, my script was
getting stuck

--
Alexander D. Mills
New cell phone # (415)730-1805
linkedin.com/in/alexanderdmills

[burak serdar]

On Thu, Sep 10, 2020 at 4:43 PM Alexander Mills
<alexande...@gmail.com> wrote:
>
> It was behaving strangely before adding the waitgroup, my script was
> getting stuck

Waitgroup is unnecessary. There's a few problems: when you detect an
error you should stop writing, but the program "continue"s. You set
the content-type to json, but this is technically not valid json.

If it was getting stuck, then it may help to find out where it is getting stuck.