runtime/cgo: pthread_create failed: Resource temporarily unavailable

[jnade...@gmail.com]

Hi everyone, I posted this question on StackOverfow here with a 200 point bounty, but per the git issue I opened here, someone suggested I post it here as well.  I have included at the end of the SO question a gist with the entire program.  I would greatly appreciate any help

[Robert Engels]

When you reassign the file descriptors you need to close the old ones - this does not happen automatically. You also need to close the ffmpeg fds. 

By not closing the descriptors the pipe structures are remaining in the kernel. 

That’s my first guess anyway. 

On Feb 4, 2020, at 7:21 AM, jnade...@gmail.com wrote:



Hi everyone, I posted this question on StackOverfow here with a 200 point bounty, but per the git issue I opened here, someone suggested I post it here as well.  I have included at the end of the SO question a gist with the entire program.  I would greatly appreciate any help

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/457ad5bc-d650-4a44-97e4-3af3ee7eb0a1%40googlegroups.com.

[jnade...@gmail.com]

After spending some time in the gophers slack chat, I ended up changing the fuction to this

func pipeThruFfmpegToMp4(vi *VideoInfo, rw web.ResponseWriter) error {
    var ffmpeg *exec.Cmd
    ffmpeg = exec.Command(
        "ffmpeg",
        "-i", "-",
        "-i", "pipe:3",
        "-c:v", "copy", "-c:a", "copy",
        "-preset", "veryfast",
        "-metadata", fmt.Sprintf(`title=%s`, vi.GetTitle()),
        "-movflags", "frag_keyframe+empty_moov",
        "-f", "mp4",
        "-")


    youtubevideo := exec.Command(YoutubeDLPath, "-c", "-f", fmt.Sprintf("%s/bestvideo[ext=mp4]/bestvideo/best", vi.GetFormat()), "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))
    fmt.Println(youtubevideo)

    youtube := exec.Command(YoutubeDLPath, "-c", "-f", "bestaudio[ext=m4a]/bestaudio/best", "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))
    fmt.Println(youtube)

    var ytvbuf, ytbuf, ffbuf bytes.Buffer
    youtubevideo.Stderr = &ytvbuf
    youtube.Stderr = &ytbuf
    ffmpeg.Stderr = &ffbuf

    video, err := youtubevideo.StdoutPipe()
    if err != nil {
        log.Printf("pipeThruFfmpegToMp4: %v\n", err)
        return err
    }

    pipe3, err := youtube.StdoutPipe()
    if err != nil {
        log.Printf("pipeThruFfmpegToMp4: %v\n", err)
        return err
    }

    ffmpeg.Stdin = video
    ffmpeg.ExtraFiles = []*os.File{pipe3.(*os.File)}
    ffmpeg.Stdout = rw

    // Headers sent, no turning back now
    rw.Header().Set("Content-Type", "video/mp4")
    rw.Header().Set("Content-Disposition", fmt.Sprintf("attachment;filename=\"%s.mp4\"", vi.GetSlug()))
    rw.Flush()

    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    ffmpeg.Wait()
    youtubevideo.Wait()
    youtube.Wait()

    // check ytvbuf, ytbuf, ffbuf for stderr errors

    if ffbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ffbuf.String()})
        log.Printf("pipeThruFfmpegToMp4: %v\n", ffbuf.String())
    }
    if ytvbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ytvbuf.String()})
        log.Printf("pipeThruYouTubevDLToMp4: %v\n", ytvbuf.String())
    }
    if ytbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ytbuf.String()})
        log.Printf("pipeThruYouTubeDLToMp4: %v\n", ytbuf.String())
    }

    return nil
}

But it still ends up crashing.

On Tuesday, February 4, 2020 at 6:50:32 AM UTC-7, Robert Engels wrote:

When you reassign the file descriptors you need to close the old ones - this does not happen automatically. You also need to close the ffmpeg fds. 

By not closing the descriptors the pipe structures are remaining in the kernel. 

That’s my first guess anyway. 

On Feb 4, 2020, at 7:21 AM, jnade...@gmail.com wrote:



Hi everyone, I posted this question on StackOverfow here with a 200 point bounty, but per the git issue I opened here, someone suggested I post it here as well.  I have included at the end of the SO question a gist with the entire program.  I would greatly appreciate any help

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[jnade...@gmail.com]

I also thought the Wait() took care of closing the file descriptors? Are you saying I should add a pipe3.Close()? Or a youtube.Close()?

[Robert Engels]

I will take a more in-depth look this evening. 

On Feb 4, 2020, at 2:19 PM, jnade...@gmail.com wrote:



I also thought the Wait() took care of closing the file descriptors? Are you saying I should add a pipe3.Close()? Or a youtube.Close()?

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com.

[Robert Engels]

Are you certain you are not just starting too many processes? Ie use a “worker pool” so you have at most N conversions happening at the same time. 

On Feb 4, 2020, at 2:34 PM, Robert Engels <ren...@ix.netcom.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/A56FB7F5-95D2-4B0E-B90C-B335B8714009%40ix.netcom.com.

[jnade...@gmail.com]

I don't think that is the issue.  I have tried it on a few different servers.  Most recent one with 100 gig's of ram and 50 cores.  The load average never goes above 9, but the ram slowly but surely on htop starts to go up.  The go binary ends up climbing slowly in it's ram use over time, then after a hour or so, it reaches around 30 gigs of ram and then crashes, and restarts.

I have it under supervisor.

On Tuesday, February 4, 2020 at 2:00:55 PM UTC-7, Robert Engels wrote:

Are you certain you are not just starting too many processes? Ie use a “worker pool” so you have at most N conversions happening at the same time. 

On Feb 4, 2020, at 2:34 PM, Robert Engels <ren...@ix.netcom.com> wrote:



I will take a more in-depth look this evening. 

On Feb 4, 2020, at 2:19 PM, jnade...@gmail.com wrote:



I also thought the Wait() took care of closing the file descriptors? Are you saying I should add a pipe3.Close()? Or a youtube.Close()?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Ian Lance Taylor]

On Tue, Feb 4, 2020 at 11:22 PM <jnade...@gmail.com> wrote:
>
> I don't think that is the issue. I have tried it on a few different servers. Most recent one with 100 gig's of ram and 50 cores. The load average never goes above 9, but the ram slowly but surely on htop starts to go up. The go binary ends up climbing slowly in it's ram use over time, then after a hour or so, it reaches around 30 gigs of ram and then crashes, and restarts.
>
> I have it under supervisor.

That is not inconsistent with Robert's suggestion. If you are
starting C threads that don't do any work but never exit, that is
exactly what you would see.

It's not the only possible cause of this. There could also be a space
leak, either in C code with memory that is malloced but never freed,
or in Go code with memory that something keeps a permanent reference
to.

Ian

> On Tuesday, February 4, 2020 at 2:00:55 PM UTC-7, Robert Engels wrote:
>>
>> Are you certain you are not just starting too many processes? Ie use a “worker pool” so you have at most N conversions happening at the same time.
>>
>> On Feb 4, 2020, at 2:34 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>>
>> 
>> I will take a more in-depth look this evening.
>>
>> On Feb 4, 2020, at 2:19 PM, jnade...@gmail.com wrote:
>>
>> 
>> I also thought the Wait() took care of closing the file descriptors? Are you saying I should add a pipe3.Close()? Or a youtube.Close()?
>>
>> --
>> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/A56FB7F5-95D2-4B0E-B90C-B335B8714009%40ix.netcom.com.
>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0e7e67a6-6ad4-4187-9f15-0d9f278b55a2%40googlegroups.com.

[Robert Engels]

I think your problem may be

"Depending on the HTTP protocol version and the client, calling
// Write or WriteHeader may prevent future reads on the
// Request.Body. For HTTP/1.x requests, handlers should read any
// needed request body data before writing the response. Once the
// headers have been flushed (due to either an explicit Flusher.Flush
// call or writing enough data to trigger a flush), the request body
// may be unavailable. For HTTP/2 requests, the Go HTTP server permits
// handlers to continue to read the request body while concurrently
// writing the response. However, such behavior may not be supported
// by all HTTP/2 clients. Handlers should read before writing if
// possible to maximize compatibility."

You may need to write the ResponseHeader as a final stage and append the output - if you write the header you may be hanging the input stages. If the input stage hangs (you tube download hangs, etc.), the whole process is going to hang.

Did you debug the number of threads and go routines the process has while running? I am betting these are continually increasing. (Another check would be that all Waits() complete).

Finally, I would use a CommandContext with a Deadline to ensure stragglers are cleaned-up.

>To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVeSsLHe92m8eQue4Lfyk3uf%2B%3D94Qy7ZapHdjsgA1RjhA%40mail.gmail.com.

[jnade...@gmail.com]

How can I debug the number of threads and go routines running and checking if the Wait()'s finish? Because I believe that may be problem, that they hang.

And when you say append the output, are you saying make a go routine to write to the headers?  If you have an example I would appreciate it

>> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0e7e67a6-6ad4-4187-9f15-0d9f278b55a2%40googlegroups.com.
>
>--
>You received this message because you are subscribed to the Google Groups "golang-nuts" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

>To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVeSsLHe92m8eQue4Lfyk3uf%2B%3D94Qy7ZapHdjsgA1RjhA%40mail.gmail.com.

[Robert Engels]

For threads just use the os command top or ps. 

For go routines I would use the net based pprof

On Feb 5, 2020, at 3:20 PM, jnade...@gmail.com wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d5b5800c-5170-4b1f-a760-7ab9def549ab%40googlegroups.com.

[jnade...@gmail.com]

So someone answered on the SO question I had posted, explaining how to check the number of processes, they suggest I add a timeout, since it seems youtube-dl, after succesffully running, sometimes hangs, so it seems golang isn't closing it sometimes.

Should I do a timeout, orinstead the CommandContext?

[Robert Engels]

I am not familiar with YouTube-dl but if you can pass a timeout on the command line that would be safest. 

Using CommandContext will forcibly kill the process. 

On Feb 5, 2020, at 4:13 PM, jnade...@gmail.com wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d4e3fdc3-4373-43a0-b902-4115e1656c64%40googlegroups.com.

[Robert Engels]

And I’ll repeat, you probably need a pool to limit the number of concurrent processes/connections. 

If you had that you would probably of detected the problem sooner. 

On Feb 5, 2020, at 6:32 PM, Robert Engels <ren...@ix.netcom.com> wrote: