How long does it take to import packages?

[ag9920]

Hi, recently I've been trying to make my unit test faster. It seems too much time were spent on initialization. After removing all init() function in relevant packages. The speed was still very slow.It even takes dozens of seconds before entering my real unit test function.

So I take a look at all the possible factors that might slow down the testing. The only possible reason I can think of is the time cost on import. Golang needs to import all packages recursively. And in my scenario, that's roughly dozens of packages.Maybe initializing const, var takes too much time. 

Is there any solution that could help me figured out the reason? I didn't find any tools that could tell me the time cost on import several packages.

And if that's the case,  import a package does take much time, is it still possible for me to speed up my unit test? 

 

[Axel Wagner]

On Wed, Aug 31, 2022 at 9:01 AM ag9920 <jingon...@gmail.com> wrote:

Hi, recently I've been trying to make my unit test faster. It seems too much time were spent on initialization. After removing all init() function in relevant packages. The speed was still very slow.It even takes dozens of seconds before entering my real unit test function.

So I take a look at all the possible factors that might slow down the testing. The only possible reason I can think of is the time cost on import. Golang needs to import all packages recursively. And in my scenario, that's roughly dozens of packages.Maybe initializing const, var takes too much time. 

"dozens of packages" is not a lot. It's very few, actually. It seems extremely unlikely to me, that if you observe dozens of seconds of startup time it has anything to do with importing those packages by themselves. IMO that is only really explainable by something waiting on I/O or sleeping or something like that in the initialization path.

 

Is there any solution that could help me figured out the reason? I didn't find any tools that could tell me the time cost on import several packages.

You could probably just do a CPU profile. If you exit immediately from your test functions, then anything appearing in the profile would necessarily be where the initialization is spent.

You can also try out tracing your test program. Personally, I found that a bit harder to set up and interpret, but it should give you an exact answer.  

And if that's the case,  import a package does take much time, is it still possible for me to speed up my unit test? 

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cd58955e-0c4b-4d56-afd7-1153d7be06dcn%40googlegroups.com.

[Jan Mercl]

On Wed, Aug 31, 2022 at 9:00 AM ag9920 <jingon...@gmail.com> wrote:

> Hi, recently I've been trying to make my unit test faster. It seems too much time were spent on initialization.

The "seems" part is a guess or a measurement? How much is "too much time"?

> After removing all init() function in relevant packages. The speed was still very slow.It even takes dozens of seconds before entering my real unit test function.

What operating system and what command line arguments are you using to
run the package tests? Latest Go version or not? Can you share the
go.mod file of the package? Do you use any variant of the 'go clean'
command, or something similar, before running the tests?

[Brian Candler]

"import" and const calculations are compile-time activities, whereas init() and global var assignment are run-time activities.

How does the time to compile your code to an executable, compare to running the tests? If compiling the code is fast but running the tests is slow, that rules out any problems with "import".

If it takes "dozens of seconds" to compile your code, that doesn't sound right unless either your codebase is huge, or you're running in a very slow environment (e.g. virtualization without hardware support like VT-x).

If it takes "dozens of seconds" to start your test suite, that doesn't sound right unless you're accidentally compiling and testing all your dependent libraries as well as your own code.  Or it could be that your test *is* starting quickly, but the first test runs very slowly - maybe it's opening network connections or doing DNS resolutions which time out, or something like that.

One thing you can try is to use strace to see system calls:

    strace -f -s128 go test ...etc...

This might give you some clues, especially if you see a long pause in the strace output (what system call did it do just before the pause?)

Another idea is to do a binary chop.  First, make a copy of your project and remove all the tests except for one "hello world" test; see if it's fast.  If it is, that shows that it's either building or running your own test code which is slow, not the importing of third-party libraries.  Then do the same but cut out only the first half of the tests, then repeat cutting out only the second half of the tests.  When you've found which half is slow, cut that in half again - and so on.

[Michael Pratt]

Setting GODEBUG=inittrace=1 will log a trace of init function execution, which you can use to determine if init is slow, and if so which packages.

e.g., `GODEBUG=inittrace=1 go version` ends with "init main @9.5 ms", indicating that it look 9.5ms to run all init functions.

The format is described at https://pkg.go.dev/runtime#hdr-Environment_Variables.

--

[Wojciech S. Czarnecki]

Dnia 2022-08-31, o godz. 00:00:41
ag9920 <jingon...@gmail.com> napisał(a):

If you are in windows ecosystem exclude your development areas, including
go installation and gocache from antivirus "heuristics". While at last most of AV
vendors "discovered" that Go exists and it has its own linker, some dutifully sends
out home binary artifacts of anything you compile "for heuristic analysis".
(Second culprit for "dozens of seconds") can be a misconfigured corporate
(or state-wide) proxy.

> Hi, recently I've been trying to make my unit test faster. It seems too
> much time were spent on initialization. After removing all init() function
> in relevant packages. The speed was still very slow.It even takes dozens of
> seconds before entering my real unit test function.

hope this helps,

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE