mutiple exe in same dir (feature request)

[rneer...@gmail.com]

Hi Dave,

I tried the 1 exe per dir setup that  we closed the previous thread with : https://groups.google.com/forum/#!topic/golang-nuts/57Cwx7i5g7U
However, I find it inefficient for me at 3 levels
a) firstly, when coding, I need to cd to a dir when I expect all my file at top level
b) secondly, at invocation time, I have to type the path with an extra dir in it on each invocation
c) when using it from any script, extra dir is typed in.
even logically thinking about set of files that monitor network, is easier when they can be found in 1 dir called netmon.
As opposed to a set of dir under under netmon ( netmon/goping, netmon/procdead, netmon/golatency) etc.

each of these is not a big bottleneck that I can't live with.
However, I am wondering what is the motivation for this restriction and what breaks in , say adding an option to go install, to work like go build and produce an exe output specified by -o ?

thanks
Neeraj

[Dave Cheney]

I think you are fighting against the tooling.

go install github.com/$YOU/$YOURPROJECT/...

will build and install any packages _and_ commands that are out of
date, then, assuming $GOPATH/bin is in your $PATH, you can just type
their names as expected.

On Wed, Nov 5, 2014 at 2:14 PM, <rneer...@gmail.com> wrote:
> Hi Dave,
>
> I tried the 1 exe per dir setup that we closed the previous thread with :
> https://groups.google.com/forum/#!topic/golang-nuts/57Cwx7i5g7U
> However, I find it inefficient for me at 3 levels
> a) firstly, when coding, I need to cd to a dir when I expect all my file at
> top level

If you are asking "why do I have to put files from different packages
in different directories" (I'm not sure if this is what you are
asking), then the answer is, because this is how the Go tool dictates
that you work

> b) secondly, at invocation time, I have to type the path with an extra dir
> in it on each invocation

See my first response at the top

> c) when using it from any script, extra dir is typed in.
> even logically thinking about set of files that monitor network, is easier
> when they can be found in 1 dir called netmon.
> As opposed to a set of dir under under netmon ( netmon/goping,
> netmon/procdead, netmon/golatency) etc.

I think this also already covered

[Nate Finch]

Just a further clarification, if you just cd into the top of your directory:

> cd $GOPATH/src/github.com/you/yourproject

then all you have to do is 

> go install ./...

and it'll install any commands in that directory or directories underneath it to $GOPATH/bin, add that directory to your $PATH, and things just work.

[rneer...@gmail.com]

Thanks for the response Dave and Nate.
Let  me clarify that I am not fighting anything, just trying to understand things and make my life easy.
I just learnt about incremental build last week and you have already enhanced my vocabulary with "tooling" - new word for me.

I don't have any projects on github, but I think what you guys said will work on local projects as well and I'll stick to that scenario.
The install at top level is useful, but as far as I understand it won't work if I have multiple go files in my dir.
  netmon/
      goping.go
     procdead.go
     golatency.go
install complains about multiple main packages.

Can you guys elaborate what does tooling mean, and why its not a good idea to have above structure.
What would break if install takes another option -o <binexe>. This is specially intriguing to me as I have been using go build with -o for some time now.
If this is not the right forum for the question, please point me in the right direction.

thx
Neeraj

[Carlos Castillo]

Packages are not meant to be limited just a single file, if you write a program of any significant size, you will have to specify "go build -o myapp foo.go bar.go baz.go" to build it, which means that as a tool, the "go" program has essentially failed.

The "go" program is not a compiler, it is a build tool, like make. The way it does it's job is by understanding the entities you have created (packages), and performing the task you want done (build/install/test/clean). The way it figures this out is by directory contents. The contents of a directory is considered the source of the package, and it is expected to compile all the files together into a single result. The big difference, and benefit over make is that no config files are needed, only the code. The downside is that only one package can exist in a given directory, otherwise it can't figure it out automatically. I consider that a suitable trade-off.

The go tool is meant to make developing code easier, setting up a package is simply making a directory, and putting related go files in it. Compiling/Installing is just "go install <path>", or "go build" (eg: in the current directory). It is a much simpler process then a separate configuration file, or specifying long command lines to the compiler.

Also, aside from the few situations where specifying individual source files or the output file might be useful, those options in general should not be used. Your confusion on this fact is likely more an argument to have those options removed, thus forcing you to use the tool properly, than anything else.

[Dave Cheney]

As far I can recall the only compelling reason go build supports -o, is to make building, then discarding _commands_ easier. There is no requirement for this when building packages as go build does this automatically.

This is the reason that the go tool will not grow more -o usage, it is a special case, can be confusing and leads people into incorrect usage of the tool, and can be easily emulated with the work around provided in the issue you raised.

[rneer...@gmail.com]

Hi Dave and Carlos,

thanks for taking time to explain this.
I came here looking for a users' perspective but it sounds like you guys are developers for the tool - got more than I bargained for :-)
That was the good news. The bad news (for me ) is that you have taken a systematic decision to not support stand alone exe outside its dir.
That's fine - I have never gone to gcc developer's to ask for features and won't be partial to "go install" developers.
You can skip the rest of  post

However, just for my memory sake, as to why I ended up doing things this way, I'll note down some more points here.
It seems like there is an assumption that multiple files will go into an exe.
While that is true for 10 of my tools, I  also have 20 more tools around it which are 20-50 lines and single file.
They do some dedicated work like checking the output of the my main packages etc. As such, I put them with the main package as the code is tightly coupled.

I was  building with "go build" using -o. This was turning out inefficient. So I asked and found a cool solution - "go install".
Now it seems like "go install" is not going to suit my needs as I would have to move a lot of code around and I run the risk of confusing myself and fellow developers.
But I respect your decision and thanks again for taking time and effort to explain the motivation behind this to me.

[Dave Cheney]

It sounds like you have N commands, and some shared logic which is
currently implemented by placing all the commands in the same package
and selectively building various combinations with go build x.go y.go
x.go -o something

As you've discovered, this isn't considered idiomatic, so this mode of
working is not well supported by the go tool.

What is recommend is to move this shared logic into a package, and
then import that wherever it is needed by commands. This means your
command packages contain the minimum amount of code, which is also
good, because writing tests for commands is much harder than writing
tests for packages.

I'm sorry this doesn't fit the mode of working you have decided on,
but that doesn't imply that the go tool, which is as opinionated as
the rest of the language, is going to grow options for your specific
use case.

I'm sure the rest of the commenters to this thread will join with me
to encourage you to restructure the way your code is written to be
more idiomatic, not just because we say so, but so that you can
interoperate with the rest of the Go universe.

Dave

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/50wO63bZ7rc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Robert Melton]

On Thu, Nov 6, 2014 at 10:04 PM, <rneer...@gmail.com> wrote:
> Now it seems like "go install" is not going to suit my needs as I would have
> to move a lot of code around and I run the risk of confusing myself and
> fellow developers.

You did it wrong from the start, you were confused, doubling down on
doing it wrong isn't the "right answer". Accept that you did it
incorrectly and fix it. You are going to create endless headaches for
yourself by fighting the Go idiomatic way. You are putting yourself
on an island where no one will be able help you. Docs online won't be
applicable, tooling won't work, people experienced with Go will have
no idea why your directory structure is so entirely broken.

Stop right now, and make your code work with "go install". It will
take a day to get used to and save you huge amounts of time in the
future.

--
Robert Melton | http://robertmelton.com

[rneer...@gmail.com]

Hi Carlos,
  "no config files are needed, only the code." - Yes that is indeed a plus.

  " The downside is that only one package can exist in a given directory, otherwise it can't figure it out automatically. I consider that a suitable trade-off"

I don't see the need for trade off. The "go install" is already reading the package name from the top of file (and complaining if I have 2 files with main in same dir)
Can it be made to write the output as filename instead of package name for such cases based on an option ?
Or some similar scheme can be devised if needed. For the "if needed" part, I need to reply to Dave.

Hi Dave,

I am not asking for any specific scenario to be supported. What I am doing is something I have seen at 10+ jobs sites I have been at.
But may be I am not explaining myself correctly or using outdated jargon.
I see that you listed a scenario of code structure that I am familiar with. So I'll make 1 more attempt to explain what my setup is.

I do have N cmds and shared code, but there's more.
a) I have a structure go/src and lots of shared packages under it (say sp1, sp2...) This is library code
b) I have N programs that use this library (packages). I call them programs as opposed to cmds to highlight a difference.
     pr1 (built from pra1.go, prb1.go, prc1.go)
     pr2 (built from pra2.go, prb2.go, prc2.go)
   So far it works. N programs can live in a dir and be complied into a exe with dir name.
   This is the crux of my app and in various places I have worked at, this has been done in C or Java etc.
c) I have supporting cmds (or scripts). sc1, sc2.
        Lots of them . Small programs. Stand alone go lang code. Single file.
    For these, it seems overkill to create a dir per script. There will be 100s of them and it will be very inefficient to create 100s of dir with single .go file in them.
   I have seen such scripts usually done in perl/python/curl
here we are using golang for the program and scripting.

If you say c) is not supported by golang, then I can evaluate writing them in different language.
If you say I can write small scripts, but they each need a dir, I am back to "different language " option.
I find it hard to believe that no one is writing hundreds of stand alone small programs in golang?

To put this discussion in perspective - You guys have made a great language available open source. And it is great the you are interacting with the users.
    If for any reason, or no reason, you decide not to support it, that is perfectly understandable.

[Dave Cheney]

We still seem to talking past each other. The structure mandated by the go tool is _not_ the Go language, if you want to use a different layout then you will need to use something else to build your Go code.

Honestly it sounds like your code is 90% compliant, but your just being stubborn. 

[Robert Melton]

On Fri, Nov 7, 2014 at 7:57 PM, <rneer...@gmail.com> wrote:
> I don't see the need for trade off. The "go install" is already reading the
> package name from the top of file (and complaining if I have 2 files with
> main in same dir)

Go is opinionated, if you continue to fight it, you are going to have
a rather awful experience.

> Can it be made to write the output as filename instead of package name for
> such cases based on an option ?

Option soup is the opposite of being opinionated. That would be like
adding options to gofmt for various formatting flavors.

> I am not asking for any specific scenario to be supported. What I am doing
> is something I have seen at 10+ jobs sites I have been at.

Unless those 10 job sites where all pure Go, I think it is rather
irrelevant. Stop trying to make Go work like <other thing>. If you
like <other thing> just USE IT.

> This is the crux of my app and in various places I have worked at, this
> has been done in C or Java etc.

Again, irrelevant. I know it feels relevant, but it isn't. You
aren't using C or Java. When you travel, you follow the laws of the
country you are in -- you are in Go, do things the Go way.

> c) I have supporting cmds (or scripts). sc1, sc2.
> Lots of them . Small programs. Stand alone go lang code. Single
> file.
> For these, it seems overkill to create a dir per script. There will be
> 100s of them and it will be very inefficient to create 100s of dir with
> single .go file in them.

First of all, is directory creation really such a chore... seems like
a mountain of a molehill? Secondly, I can't fathom the situation you
are in that you want to create hundreds of binaries... heck, just
naming them would get confusing. clientapp99, clientapp114. I would
say the creation of directories is the LEAST of your problems if your
workflow requires the creation of hundreds of trivial scripts or
binaries. In my mind either: [1] you have hundreds of small, unique,
unrelated, specific use cases that are different enough to be
impossible to generalize or [2] you have an easily generalizable
problem and you just haven't solved it properly.

If the case is [1] then each one deserves its own directory, it is
really its own little application, with its own problem domain,
maintenance and solution. In the case of [2] (IMHO, far more likely)
you actually need to solve the problem in a more sane way then
generating (or writing by hand?!) hundreds of scripts.

> I have seen such scripts usually done in perl/python/curl
> here we are using golang for the program and scripting.

Again, I am unclear of what these "scripts" do, their complexity,
interdependency... but if you are writing hundreds of them, my gut
instinct is something has gone horribly wrong.

> If you say c) is not supported by golang, then I can evaluate writing them
> in different language.
> If you say I can write small scripts, but they each need a dir, I am back to
> "different language " option.

I think looking at different languages would be wise at this point. I
think you have a workflow you value more than the language (or your
sanity) you use, so it makes sense to switch your language rather than
you workflow.

> I find it hard to believe that no one is writing hundreds of stand alone
> small programs in golang?

I find it mind boggling that anyone would even consider that "normal"
-- or anything short of insane. Hundreds (plural)... my gosh, I have
been doing Go fulltime for over 18 months and have generated a handful
of binaries (maybe 35) and a lot of them are trivial implementations
of things like parallel downloaders, parallel console concurrency
(simpler than GNU parallel), log parsers, etc. I can't imagine how I
would end up with hundreds... again, I am very curious what is in
these programs that is entirely non-generalizable.

[rneer...@gmail.com]

Hi Dave,

I'll take your suggestion and explore Makefile to build it. As you rightly pointed out , there is nothing in the golang that prevents me from writing 100s of scripts.
Its just the "tooling" that is not suitable for it.

Hi Robert,

Your post finally made it clear why we are talking past each other. We are talking different time spans.
The project I am working on has been around for 7+ years with multiple programmers.
And  this is nothing unique as I have worked at other places in different industry which had similar situations.
And you are right - golang wasn't used - it didn't exist back then.

I think we can close this thread as we have an agreement that golang build install is not supposed to be used for case c) above.

[Lars Seipel]

On Fri, Nov 07, 2014 at 04:57:49PM -0800, rneer...@gmail.com wrote:
> I find it hard to believe that no one is writing hundreds of stand alone
> small programs in golang?

Oh, people do that. Maybe not literally hundreds, but plenty of them.
There's no problem with that.

To make use of the go tool all you need to do is change to the directory
in your GOPATH that contains all those source files and tell your shell
to put them into nice and fresh directories. Something like:

for f in *.go; do
d=${f%.go}
mkdir $d
mv $f $d/
done

Now, to build all your programs just do 'go install ./...'. The binaries
will all sit happily in your GOBIN.

Really, it's that easy. Why make it any harder?

Lars

[Konstantin Khomoutov]

On Sat, 8 Nov 2014 06:02:25 -0800 (PST)
rneer...@gmail.com wrote:

> I'll take your suggestion and explore Makefile to build it. As you
> rightly pointed out , there is nothing in the golang that prevents me
> from writing 100s of scripts.
> Its just the "tooling" that is not suitable for it.

[...]

I'm not sure I've read this thread in its entirety so forgive me if
I'll repeat someone, but should you adopt Makefile instead of the `go`
tool, make sure you're not using go tool in that Makefile--but rather
appropriate compiler and linker directly. Basically, that's what `go`
tool does: it takes your directory structure and packages for what
`make` takes Makefiles. So let me recap that `go` is not a compiler
but rather a wrapper around the compiler and the linker.

You can run `go build` etc with the "-x" command line option to make it
be explicit about what tools it calls, with which options and in what
sequence.

P.S.
You might be curious to know that the `go` tool did not exist until
some relatively recent release of Go (like 1.0 or something like this),
and indeed using Makefiles was a way do go back then.

[rneer...@gmail.com]

Hi Lars,

thanks that's an easy one to try.

Hi Konstantin,

thanks for the inputs- even without reading full thread, you have provided valuable input.
I would like to explore the complers with makefile. I am fairly new to go and know little about its build process.
I looked at https://golang.org/cmd/ - it seems 8g is the newest compiler.
Would you recommend that ?
it seems go has a package for generating depedencies which could be fed to make. Is that recommended?

I just tried my local go build -x and it seem 6g and 6l are used by default. The following is the output.
>6g -o $WORK/command-line-arguments/_obj/_go_.6 -p command-line-arguments -complete -D ........
>pack grcP $WORK $WORK/command-line-arguments.a $WORK/command-line-arguments/_obj/_go_.6
>cd .
>6l -o /tmp/gotest/src/prog/exe1 -L $WORK $WORK/command-line-arguments.a

[Carlos Castillo]

6g/6l are for AMD64, 8g/8l are for X86 (32-bit). It's not about version.

Really though, you are in for a world of hurt down this path... learning to use the tools properly before you start is still preferable in my mind.

[Nick Patavalis]

On Saturday, November 8, 2014 2:57:49 AM UTC+2, rneer...@gmail.com wrote:

c) I have supporting cmds (or scripts). sc1, sc2.
        Lots of them . Small programs. Stand alone go lang code. Single file.
    For these, it seems overkill to create a dir per script. There will be 100s of them and it will be very inefficient to create 100s of dir with single .go file in them.
   I have seen such scripts usually done in perl/python/curl
here we are using golang for the program and scripting.

If you say c) is not supported by golang, then I can evaluate writing them in different language.
If you say I can write small scripts, but they each need a dir, I am back to "different language " option.
I find it hard to believe that no one is writing hundreds of stand alone small programs in golang?

As others have said, you 're doing it wrong.

I assume that these scripts have some kind of similarity. That is, they are used for similar purposes or are part of a "suite" of commands.

Then, instead of having a gazillion of binaries (exeutables) which is also inefficient in terms of both disc, and memory usage (remember these are statically linked binaries) you can arrange to have single binary which dispaches to different commands.

So instead of running 

  sc1 <arguments>

  sc2 <arguments>

  

You instead do:

  script sc1 <arguments>

  script sc2 <arguments>

Now you have only 1 executable, and each script becomes just a function (entry point). The source code looks pretty much the same: you still have sc1.go, sc2.go. sc3.go, etc. But instead of them being built each into its own binary (executable) they are all build into a single binary with a driver function as its main() that dispatches to them accordingly.

You can easily devise scheme where each such scX.go registers itself to the "driver" so that the driver knows what script entry-points are there.

Now  all you have is single binary (script.exe) which dispatches to the individual commands based on argv[1] ... All of the sc1.go, sc2.go, ... scX.go code is linked into this single binary.

This is similar to what many tools do, eg:

  git clone <args>

  git commit <args>

  ...

/npat

[Nick Patavalis]

On Saturday, November 8, 2014 9:35:40 PM UTC+2, Nick Patavalis wrote:

You can easily devise scheme where each such scX.go registers itself to the "driver" so that the driver knows what script entry-points are there.

Now  all you have is single binary (script.exe) which dispatches to the individual commands based on argv[1] ... All of the sc1.go, sc2.go, ... scX.go code is linked into this single binary.

Here's how it could look like:

  $GOPATH/src/my/project/cmd

      main.go  <-- like this: http://play.golang.org/p/VyVawib1Wm

      cmd1.go  <-- like this: http://play.golang.org/p/J2XPKE99eI

      cmd2.go  <-- like this: http://play.golang.org/p/Dus91LqEfj        

      ...  etc ...

/npat

[HaWe]

There's a lot of good advice given in this thread. Here's even more of it:
http://golang.org/doc/code.html

[rneer...@gmail.com]

thanks Carlos all info is appreciated.

Between the answers  you all provided and the 5 line shell script from Lars, I have the solution.

The compiler /linker info is useful even if I don't use it in this case at present.

We can close this thread.

[qcf...@gmail.com]

The limitation is that all files indicating "package main" would be combined into a single executable.

Go tooling uses "build constraints" (see http://golang.org/pkg/go/build/) to convey additional information yet remaining makefile-clean.

No less idiomatic would be "main constraints" (see https://groups.google.com/forum/#!topic/golang-nuts/50wO63bZ7rc").

Demo at https://github.com/qcf3001/golang-multiple-exe-in-one-dir

- qcf

[Waitman Gobble]

Two situations that I do not yet understand how to properly use the Go tooling to handle, 

1) combining ("linking") a shared 'common' exe with individual exe's in a project

2) deploying exe's in one project in various locations. 

i'm using a 'workaround' Makefile, which perhaps serves example to my dilemma of misunderstanding.

all:

        go build $(GO_FLAGS) -o build/poster.cgi src/poster.go src/arccommon.go

        go build $(GO_FLAGS) -o build/arcauth.cgi src/arcauth.go src/arccommon.go

        go build $(GO_FLAGS) -o build/arcuser src/arcuser.go src/arccommon.go

install:

        $(INSTALL) -s -m 0755 -o root -g root build/poster.cgi $(CGI-BIN)

        $(INSTALL) -s -m 0755 -o root -g root build/arcauth.cgi $(CGI-BIN)

        $(INSTALL) -s -m 0755 -o root -g root build/arcuser $(PREFIX)/bin

        $(INSTALL) -b -m 0644 -o root -g root etc/archiver.photo.json $(ETC)

clean:

        rm -f build/poster.cgi

        rm -f build/arcuser

        rm -f build/arcauth.cgi

deinstall:

        rm -f $(PREFIX)/bin/arcuser

        rm -f $(CGI-BIN)/poster.cgi

        rm -f $(CGI-BIN)/arcauth.cgi

I'm new to Go but I'm catching a drift that perhaps using a Makefile is taboo. It would be good to do it the correct way. Comments/pointers appreciated.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Waitman Gobble
San Jose California USA
510-830-7975

[Dave Cheney]

Hi Waitman,

The go tool works with packages, not files [1]. The answer to how to
structure your code to work with the go tool is to place the code in
separate packages. I suggest this form.

$GOPATH/src/github.com/waitman/yourproject
yourproject.go // this is the arccommon.go file mentioned above
/cmd
/poster
main.go // this is the main package for
the poster cgi command
/arcauth
main.go // this is the main package for
the arcauth cgi command
/authuser
main.go // this is the main package for
the authuser command

Any common code that you want to call from the three
cmd/{poster,arcauth,authuser} should be public (start with a capital
letter), so it can be used like this

// sample cmd/poster
package main

import "github.com/waitman/yourproject"

func main() {
yourproject.SomeFunction()
}

then building, testing and installing these programs to $GOPATH/bin is

go test github.com/waitman/yourproject/...
go install github.com/waitman/yourproject/...

That's it.

If you want to use make, go for it, I don't subscribe to the theory
that just because the go tool does 80% of the work, it invalidates the
other 20%.

Dave

1. There are obviously small exceptions to this rule, but I hold that
they serve to confuse new users of the tool, not aide them.

> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/50wO63bZ7rc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Dave Cheney]

The google.com/x/tools repo is an excellent example of how to combine
shared and unique packages along with commands that produce a variety
of programs.

https://code.google.com/p/go/source/browse?repo=tools

This is how a complex repository should be laid out -- you don't have
to like it, but if you want to use the go tool, this is the
recommended form.

[Waitman Gobble]

Dave,

Awesome, I'll check it out. This is a learning project, probably shouldn't do 'nails down chalkboard' stuff right off the bat. I'll see about rearranging the code structure.  I appreciate you taking the time for the pointers.

So, I'm kinda old-school. I guess man pages are out.

--

Waitman Gobble
Los Altos California USA
510-830-7975