Mimosa Require - verify taking a long time

[James]

I've been using node inspector to debug through register.js and for this particular project a lot of the processing time (~20 mins) is spent in _buildTree and especially in _addDepsToTree.  For the latter function the result of the built tree seems to be a flat array with all the unique file dependencies of its children for the given require file. The time consuming part seems to be recursively going through the dependencies of the child files even though they have already been checked before in previous files. 

For this project there are three root require files which, once mimosa has completed building the tree, has ~250,~150, ~150 unique file locations respectively.  

So my questions are:

• Is the long verification time expected given the number of files?
• Are there any possible ways to decrease the verification time?

Thanks

[David Bashford]

We've got 3 bundles being built between around 200-250 javascript files.  I'm not sure how long verification specifically is taking, but the entire build, from A) deleting all compiled assets to Z) installing the production ready app on the server, essentially everything we have, takes about a minute.

For it to take that long is a bit crazy. Any chance its a resource constrained machine?  Might you have some circular dependencies that Mimosa isn't capturing properly?

Probably impossible to create, but it would be great if I had some sort of demo project to play with.  I can't fathom things taking that long so I'd love to know what input set is causing that issue..

--
You received this message because you are subscribed to the Google Groups "mimosajs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mimosajs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James]

I wouldn't say it's a resource constrained machine. As for the circular dependencies I've ran madge on the project and there was one file which was referencing itself. I removed that file as a test but the verify stage still takes the same time to complete.

There's essentially three projects which can all be run separately using mimosa:

P1 is the base project.

P2 is a project which adds P1 as a dependency and references those files.

P3 is a project which adds both P2 and P1 as a dependency and references those files.

P1 and P2 verify stages aren't noticeably long but P3 is the project which takes a long time to complete verification. 

As I mentioned before P3 has a root require file (say it's called 'rootRequire.js') which will reference both projects as dependencies. Going back to the code I imagine the _addDepsToTree will go through each of the files in all 3 projects and add any unique files to the variable this.tree['rootRequire.js']. At the end of the verification this.tree['rootRequire.js'] has an array of all the unique file path dependencies. I'm not sure what the usage of the tree is for (not even sure it looks like a tree) but this is taking a very long time to complete. The steps beforehand (_verifyShims, verifyFileDeps, verifyConfigForFile) seem to blitz through and They're the functions which seems to log mimosa errors if something is wrong.  

I can definitely picture the addDepsToTree call being very time consuming considering there's ~200 unique files referencing each other and the verify has to read each file's dependencies and then read the dependencies of those dependencies etc. This would have a lot of repetition since if for example one file (file A) had two dependencies (file B and file C), both of which had a dependency which were referencing the same file (file D), that file (file D) would be verified twice when file B and C are verified.

I'm just writing out my thought process of the mimosa-require verification steps but I could be completely wrong. Unfortunately I can't provide a demo project but hopefully the information I've provided makes sense.

Thanks

[dbashford]

I'm going to spend the next short bit of time doing some work in mimosa-require.  It's next on my list of modules to update (the r.js dependency is a good deal behind).   While I'm in there I'll take a look at this.

It's absolutely the case that if as its going through the require dependency tree is it processing the same files over and over there are likely some perforrmance improvements to make.

[dbashford]

Do me a favor.

mimosa mod:install mimosa-...@2.2.0

Then re-run your build.  Let me know the time difference?  I made a very minor change, and its not the perfect update, but locally instead of checking 1500 files (may being duplicates) on our project, it now just checks 600.  A much larger effort could likely get that down to ~250.  But I'm curious what effect the change I just made has for you.

[dbashford]

I've also released 2.2.0 with mimosa 2.2.20.  So you could just bump mimosa.

[James]

Early signs, the verification is completed almost instantaneously. I've had a quick debug through the verify again and it seems like it does reproduce the same tree in the 'this.tree' variable as before. From what you've said I'm surprised it's gone from 20 minutes to a few seconds for the verify process. Would it be possible to explain what you changed in the module at a high level?

Thanks 

[dbashford]

This commit: https://github.com/dbashford/mimosa-require/commit/b4b9ccb195430e99eb1c84fda3013a2e238be0f2 is what did it.  Now, 95% of that commit is commented out debugging.  Stuff I left in there to dig into this more later.

The actual code modification is this line (470): https://github.com/dbashford/mimosa-require/commit/b4b9ccb195430e99eb1c84fda3013a2e238be0f2#diff-b4dee062d515082ce84f89751beceecbR470

I'm just checking to see if the tree already has the dependency.  If it does, then that dependencies own "tree" is already present in data.  No need to do anything with it.

My guys are seeing performance improvements more like this:

Before:

Real: 0m15.301s

User: 0m14.252s

Sys: 0m1.110s

After:

Real: 0m11.666s

User: 0m11.390s

Sys: 0m1.005s

But I can imagine this blowing out on a much larger scale if you have a truly enormous app with a but of a hairball of a dependency graph.

What mimosa-require is doing is, once it has discovered each of your "base" require files, it takes each one and recurses through your dependency graph building up that structure.  What this code change doesn't do is take into account if a specific sub-tree has been encountered in the processing of another base file.  It only cares about the current one.  So if you have 10 main files and in all of those you have a common set of modules with a root level dependency and a tree of 50 more modules behind it, you'd end up going through those 10 times.  I have what I thought was a fix for that as well, but in early testing the tree diff was off very slightly.  I haven't dug into why yet.

Super happy to hear this has essentially resolved your issue.  Also very happy you brought this to light.  This should be a big performance boost for all folks using mimosa.  Thanks!!!