Handling multiple things happening at the same time

799 views
Skip to first unread message

Matt Davies

unread,
Sep 7, 2016, 11:43:31 AM9/7/16
to golang-nuts
Evening all

We're writing a new app and we'd like it to do a number of things at the same time.

We're planning on using gocron https://github.com/jasonlvhit/gocron to schedule grabbing some files from another source, then reformatting those files and placing them into a folder in the app.

That's all working fine.

What we now need to do is serve those file up from that folder all of the time using http.ListenAndServe.

Am I on the right track here?


Or maybe gocron can schedule the http.ListenAndServe to run all the time?

Any tips or advice greatly appreciated.






Asit Dhal

unread,
Sep 7, 2016, 11:56:27 AM9/7/16
to Matt Davies, golang-nuts

Hi Matt,

You should run the cronjob in a separate process and the FileServer in another process.

    mux := http.NewServeMux()
    fs := http.FileServer(http.Dir("fileserver"))
    mux.Handle("/", fs)
    http.ListenAndServe(":8080", mux)


The FileServer package provided by the standard library will be good(and small). In this way, you just separate the responsibility between two processes.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Warm Regards,
Asit Dhal
http://asit-dhal.github.io/

Caleb Doxsey

unread,
Sep 9, 2016, 5:52:18 AM9/9/16
to golang-nuts
It looks like gocron.Start() runs the jobs in another goroutine, so you can just do both things in your main function:

    func main() {
        s := gocron.NewScheduler()
        s.Every(3).Seconds().Do(task)
        stopper := s.Start()
        defer close(stopper)
        
        err := http.ListenAndServe(":5000", nil)
        if err != nil {
            log.Fatalln(err)

Simon Ritchie

unread,
Sep 10, 2016, 6:47:08 AM9/10/16
to golang-nuts
I agree with Asit, but I would go further.  You have one thing that is fetching files and putting them in a directory.  You have another thing that is exposing the files in a directory via a web interface.  Except that they happen to be working on the same directory, there is no connection between the two things.  They could run on different machines.  My suggestion would be to have two completely separate programs.  Then, assuming that the machine(s) they run on stay up, if the file fetcher falls over, the web server will still display the files that are there, and vice-versa.

I did similar things during a project I worked on a couple of years ago, but not using Go.  That solution uses a Java web server to display data which is fetched using an off-the shelf FTP server driven by cron scripts running TCL programs.  We authenticated the FTP connection using passwords and I recall that one of the challenges was to protect the password from exposure.  You could probably cobble together a prototype file fetcher more quickly using a solution like that, but it would be easier to make a Go solution secure. 

For maximum security, use certificate authentication rather than passwords, but that involves passing certificates around, which can delay getting the thing up and running, especially if there are lots of organisations providing the files.

Simon Ritchie

unread,
Sep 10, 2016, 7:06:03 AM9/10/16
to golang-nuts
> maybe gocron can schedule the http.ListenAndServe to run all the time?

The classic solution (which is how the Apache web server does it) is to have a shell script that loops forever, starting the web server, waiting for it to finish and starting it again.  So, if the server crashes, the parent script starts it again.  You set the starter script up so that it is kicked off when the machine starts up.

It sounds like this just moves the problem to the parent script, but actually that doesn't do very much, so it's much easier to make it (almost) never crash.

The problem then is to make the server stop when you need it to.  That's achieved by making the parent script a signal catcher.  One signal makes it kill its child and then stop.  Another makes the child reread its configuration files and reconfigure itself.  You have another script that sends the signals - the stop script.  You include a call to this in the stuff that is run when the machine shuts down, so the web server is shut down tidily.

To get all that to hang together, the first parent script records its PID in a file when it starts.  The stop script reads the PID from the file.

You could do worse than downloading Apache and stealing the startup and shutdown scripts.  They've been in use for twenty years or more, so they are now fairly robust.






Caleb Doxsey

unread,
Sep 10, 2016, 9:18:01 AM9/10/16
to golang-nuts
Why would you do this manually?

You should use systemd to create a service. It will handle starting on load, logging, restarting when it crashes, etc.

Asit Dhal

unread,
Sep 10, 2016, 10:43:16 AM9/10/16
to Caleb Doxsey, golang-nuts

Hi,

Don't you think it will be over engineering.

Like Simon said, Apache can be used as a file transfer app(it has logging, loading, restarting).

Probably, his small app is an extension of some bigger go app.

His small app can use two different processes which runs under two different user(you got system level isolation and North Korean hackers are no more allowed to read unformulated files).

File server app usually never fails. Cron App usually fails, but logs the error and works for the next cron job. Gocron doesn't seem to have elegant error handling mechanism. The developer needs to handle them somehow. Gocron will proudly fail if one tasks fails to do error handling.

1. CronApp runs under "cron_user". On success, file permission changes to allow read to "Ftp_user".

2. Ftp_User simply sees a new file and displays them(delivers). For a better file server, this one https://github.com/codeskyblue/gohttpserver

As both of these processes have clearly different responsibility, they don't need to be aware of each other.  

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matt Davies

unread,
Sep 12, 2016, 2:48:08 AM9/12/16
to golang-nuts
Thanks for everyones help.

I'm going to go with the subroutine for now.

This is all the app does, download a file, manipulate it, then serve it up for pick up.

It does the download and manipulate action once a day, and the upload happens once a day.

We've got raven emailing us if anything goes wrong.

If the download, file manipulation or http server fails, then we'll know about it.  If any of the first two jobs stop for any reason, and then stop the file server, that's not a bad thing in this instance, as we wouldn't want the other side picking up the file.

I don't see the solution being in place as a permanent one, moving data around in files like this these days seems a bit archaic.  We'll be looking into putting some resource into using the some of the API's form the 3rd parties involved pretty soon.

Thanks again to everyone who's replied, I learnt some stuff for sure!
Reply all
Reply to author
Forward
0 new messages