Terraform modules, mega-repos, and the TF Registry

[ja...@fpcomplete.com]

Hello,

I have a single git repo with a bunch of TF modules in it. The modules work well together in small groups, and together create the various types of stacks we deploy. Recently, I have been working to improve the release process for the repo, primarily to address a few UX issues for devs and users of the modules. While I'd prefer to distribute the modules through the official public Terraform Registry, the Registry imposes the repo naming scheme and one-repo-per-module requirements our mega-repo does not live up to. I like the workflow of the one repo, and do not want to break-up the modules into a bunch of individual git repos (it's worked well for us so far). In reviewing other ways of distributing and versioning modules, the git/repo-based methods are not ideal in that TF will do a single repo clone/checkout for each module, which in our case means 100 copies of the mega-repo if you have 100 instances of the modules in use in the project (some do).

This leads me to a few questions about the registry:

1. Is there an official Hashicorp position _against_ mega-repos for modules like this?
2. Does the Terraform Registry plan to avoid/block support for mega-repos, or to allow for their use on the Terraform platform?
   
3. Would the Registry be open to providing an API for us to register/publish a new module that is just a URL to the actual archive file? This concept of redirecting to the real URL of the archive seems to already be deeply embedded in the Registry and its spec/implementation, and so adding this API would allow (for example) a user's CI system to store modules on S3 but publish them through the TF Registry.

This also leads me to questions that are more about module dependencies and discovery:

So far, for the life of this mega-repo, we've simply placed the git repo down next to our client project (and more recently inside the project), and then referenced/sourced the modules by file-system path. We used git subtrees to manage the module repo per-project, and to get project-wide versioning for the upstream modules. In this case, when one of the modules used other modules, they would simply reference the dependencies with `source = "../module"` and such. This worked rather well when using subtrees, but still had a few annoyances. Either way, TF would discover all the modules and their dependents just fine. This also appeared to work when I tested sourcing modules from git/hub directly, Terraform would find the dependent modules and source them correctly. Though as previously mentioned, sourcing modules from a mega-repo over git means that Terraform will checkout many copies of the repo (which makes sense). So this leads me to a few questions:

1. Does Terraform have any plan or interest in supporting sourcing modules from mega repos, or otherwise improving the UX here? for example the case where modules are sourced from git?
2. How is the module discovery expected to work if not using the filesystem URLs? EG, if I have modules on an HTTP server, or S3, or github, how will Terraform attempt to find and retrieve the dependent modules? How are versions resolved?
3. For example, if I have a module that I've published on some S3 URL, there are version numbers included in URL, but Terraform probably doesn't understand them, and so what would I put in the `source` when a module has a dependency on one of the other modules in the mega repo?

Thanks for reading along, your feedback is welcome!

[Steve Wagner]

Thought I'd add my two cents in as a maintainer of an internal mega-repo (TF Registry is, sadly, not an option at this time).

Some years ago, I was in the business of maintaining a mega-repo of Chef cookbooks for a large number of teams.  Releases were often eventful.  I split the repos out as soon as the Berkshelf dependency manager became officially supported and I figured out the right combination of scripts and `git filter-branch` commands that would do it.

In a certain sense I am back where I started with my Terraform module mega-repo.  I test and release all of the modules together as a unit, which means a bug fix to any of them means testing all of them.  Downstream, I'm at least able to pin Submodule 1 to Tagged Release X and Submodule 2 to Tagged Release Y.  And since CI is the only thing that's actually _running_ Terraform in my world, having some extra Git checkouts isn't horrible.

The testing experience right now on this mega-repo is not great ("Do these 'examples' converge?  Eh, things are probably all right.").  I miss ChefSpec.  If I had more tests I'd probably have broken out the modules into their own repos by now because the test suite runs are already rather long for CI.

But really what it comes down to is, I want most of these modules to be individually composable.  I should probably be tracking, testing and releasing them that way.

[ja...@fpcomplete.com]

At least for what I experience in this repo, I'm ok with the mega repo setup and workflow that we have.

I'm mostly interested in what Terraform has in mind for module developers using a mego-repo to organize their modules in one or a few git repos, rather than one repo per module. The purpose of my post has been to elicit that feedback and discuss the issue in general (of mega repos and the limitations TF imposes or has currently).

[pba...@hashicorp.com]

Great question, thanks for taking the time to write it up with good background info.

I'll give you my thoughts on some of those questions. For background I'm the main engineer working on the module registry currently however this is just my take rather than an official HashiCorp position that has been set in stone!

> Is there an official Hashicorp position against mega-repos for modules like this?

I'd say HashiCorp is pretty pragmatic; I can see good reasons to arrange your TF modules as you described and I'd hesitate to say it was wrong and should never be done.

On the other hand, we need to impose a bunch of rules and structure on modules in order to make the registry a useful tool - both to make it tractable for machines to generate documentation and validate the code etc. but also for consumers to get a consistent experience with how they consume modules, manage versions and dependencies etc.

I would say it's unlikely we will encourage "mono repo" style modules in the public registry at least in the sense you describe, just because for all the potential benefits within a single org, it works against making modules that are discoverable and composable by consumers, and makes managing transitive dependencies much harder beyond the simple case you describe where _all_ of your dependencies are in one mono repo. Think of the public registry like any other programming language library/module repository, you really want well-defined modules that do one thing well and can be easily discovered and then composed with others rather than a whole bunch of kitchen-sink frameworks that are only really designed to be used on their own.

Longer term we'd hope to see more re-use of common modules, often maintained by vendors themselves, in the registry and the composition of several core modules from different vendors seems to loose a lot of the benefits of monolithic single-org repos.

Concretely, some of the challenges for mono-repos are around versioning since one git tag applies to all the nested modules, and that many organisations need control over permissions that can be easily granted at repo level with different teams owning different modules etc. There are also issues of release cadence - we chose to split even separate providers for the same logical module into separate repos in the registry because our experience with other parts of terraform is that combined releases become a night mare when different providers release features at different rates etc. That is only compounded if you have orthogonal concerns and even teams trying to work on modules in a single repo.

> Does the Terraform Registry plan to avoid/block support for mega-repos, or to allow for their use on the Terraform platform?

I don't think we'd consciously try to "block" that use case, however as things are it's not obvious that the choices we make to optimize for reusable modules will necessarily be the right choices for people publishing mono repos.

It's not something we've actively tried to guard against though and will certainly be interested to see how organisations are interested in using the registry.

> Would the Registry be open to providing an API for us to register/publish a new module that is just a URL to the actual archive file?

The public registry is unlikely to support this in the near future. The main reason for that is that we fairly intentionally choose to integrate it reasonably tightly with GitHub repositories to make it easier to work with and simpler to operate.

All that said I'll point out two things:

 1. The public registry _does_ support "nested" submodules - as long as there is at least one `.tf` file in the root module and preferably a README, you can put all your code into submodules which can actually be imported individually if required with `src = "org/module-name/provider//sub-module-name` we've not exposed that widely in docs/UI etc. yet but we do plan to make that a much more first-class experience soon. This doesn't solve for issues like the repo name convention but some of what you want can be achieved this way.

 2. If you are only concerned with publishing modules that you want to re-use internally, it is possible to implement your own "private" registry according to our documented spec that serves your needs and by design could be a very thin layer on top of S3. Terraform CLI will be able to consume from it as long as you follow the relatively simple API and discovery specs.

We have plans for private registry support built into Terraform Enterprise that will be more flexible in that area in order to integrate with a wider variety of in-house build chains and VCS/artefact stores.

Hope this is a useful response!

Paul

[ja...@fpcomplete.com]

Hi Paul,

On Monday, January 8, 2018 at 4:44:01 PM UTC-5, pba...@hashicorp.com wrote:

Great question, thanks for taking the time to write it up with good background info.

I'll give you my thoughts on some of those questions. For background I'm the main engineer working on the module registry currently however this is just my take rather than an official HashiCorp position that has been set in stone!

Hi Paul,

Thanks for taking the time to write, and I'm sorry for the terrible delay in getting back to you with my follow-up questions here.

 

I would say it's unlikely we will encourage "mono repo" style modules in the public registry at least in the sense you describe

right, I don't think they need to be _encouraged_, though I'm hoping it would be possible to use the registry with a mega-repo the way some of us choose to do (as you said, sometimes it makes a lot of sense).

... you really want well-defined modules that do one thing well and can be easily discovered and then composed with others rather than a whole bunch of kitchen-sink frameworks that are only really designed to be used on their own.

 I don't disagree, and I'd like to publish our modules as separate consumable modules, but I'd like to continue managing them in git via one mega-repo, and that does not seem possible ATM.

Longer term we'd hope to see more re-use of common modules, often maintained by vendors themselves, in the registry and the composition of several core modules from different vendors seems to loose a lot of the benefits of monolithic single-org repos.

I think that will be great! Unfortunately, at least in my experience with Terraform and other CM tools, there are times when common/upstream modules will work, but most of the time I'm going to be managing some repo of modules to accomplish what's being asked of me.

 

Concretely, some of the challenges for mono-repos are around versioning since one git tag applies to all the nested modules, and that many organisations need control over permissions that can be easily granted at repo level with different teams owning different modules etc.

I'd be fine with all modules in a mega-repo getting the same version.
 
At the same time, I think I can work out those issues without pushing them up to the registry, and I think this is easily accomplished with an API in the TF registry that allows us to create/release a module with a URL we control, EG, if I can upload a .zip to S3 and tell the registry about it and give it a version, we'd be fine, no?

 

> Does the Terraform Registry plan to avoid/block support for mega-repos, or to allow for their use on the Terraform platform?

I don't think we'd consciously try to "block" that use case, however as things are it's not obvious that the choices we make to optimize for reusable modules will necessarily be the right choices for people publishing mono repos.

It's not something we've actively tried to guard against though and will certainly be interested to see how organisations are interested in using the registry.

We would like to publish our modules on the registry, and use the registry as the standard way to distribute and use our modules, though we would like to manage some of these modules in mega-repos. Hosting the modules on S3 also seems to make sense for this situation.

 

> Would the Registry be open to providing an API for us to register/publish a new module that is just a URL to the actual archive file?

The public registry is unlikely to support this in the near future. The main reason for that is that we fairly intentionally choose to integrate it reasonably tightly with GitHub repositories to make it easier to work with and simpler to operate.

Integrating with Github is a great idea, but I am not sure I see how forcing or requiring it is. I would love to publish modules on the Terraform Registry. An API to publish the module as a zip built by my CI system would allow me to do so. Is this completely untenable?

 

All that said I'll point out two things:

 1. The public registry _does_ support "nested" submodules - as long as there is at least one `.tf` file in the root module and preferably a README, you can put all your code into submodules which can actually be imported individually if required with `src = "org/module-name/provider//sub-module-name` we've not exposed that widely in docs/UI etc. yet but we do plan to make that a much more first-class experience soon. This doesn't solve for issues like the repo name convention but some of what you want can be achieved this way.

I'll happily rename a repo to make it work. But I am wondering, in our case.. we have all modules in `tf-modules`, and a bunch of test/example env in `tests`. Those tests use a bunch of the modules in `tf-modules` and it all works very well. You have noted the registry will improve this experience soon, would our repo as it is laid out with the modules in `tf-modules` and TF env in `tests`, work out ok with the nested submodule idea you have presented? or would we need to shuffle around those module/test locations?

any recommendations are appreciated :)

Thanks!

[ja...@fpcomplete.com]

One other question...

On Monday, January 8, 2018 at 4:44:01 PM UTC-5, pba...@hashicorp.com wrote:

 1. The public registry _does_ support "nested" submodules - as long as there is at least one `.tf` file in the root module and preferably a README, you can put all your code into submodules which can actually be imported individually if required with `src = "org/module-name/provider//sub-module-name` we've not exposed that widely in docs/UI etc. yet but we do plan to make that a much more first-class experience soon. This doesn't solve for issues like the repo name convention but some of what you want can be achieved this way.

If the sub-modules reference other modules, this would work just fine, correct?

In my tests publishing modules to S3, this failed when the modules were published separately. I guess that would work if I published the mega-repo as a single module w/ submodules. Does the registry (today) support distributing submodules like this?

If I use 10 modules in an env, and all of those modules are distributed as submodules in one module, does TF download 1 or 10 copies of the module?

Thanks!

[Mitchell Hashimoto]

Hey,

Inline…

On Jan 25, 2018, 2:22 PM -0800, ja...@fpcomplete.com, wrote:

Best,

Mitchell

Hi Paul,

On Monday, January 8, 2018 at 4:44:01 PM UTC-5, pba...@hashicorp.com wrote:

Great question, thanks for taking the time to write it up with good background info.

I'll give you my thoughts on some of those questions. For background I'm the main engineer working on the module registry currently however this is just my take rather than an official HashiCorp position that has been set in stone!

Hi Paul,

Thanks for taking the time to write, and I'm sorry for the terrible delay in getting back to you with my follow-up questions here.

 

I would say it's unlikely we will encourage "mono repo" style modules in the public registry at least in the sense you describe

right, I don't think they need to be _encouraged_, though I'm hoping it would be possible to use the registry with a mega-repo the way some of us choose to do (as you said, sometimes it makes a lot of sense).

... you really want well-defined modules that do one thing well and can be easily discovered and then composed with others rather than a whole bunch of kitchen-sink frameworks that are only really designed to be used on their own.

 I don't disagree, and I'd like to publish our modules as separate consumable modules, but I'd like to continue managing them in git via one mega-repo, and that does not seem possible ATM.

Longer term we'd hope to see more re-use of common modules, often maintained by vendors themselves, in the registry and the composition of several core modules from different vendors seems to loose a lot of the benefits of monolithic single-org repos.

I think that will be great! Unfortunately, at least in my experience with Terraform and other CM tools, there are times when common/upstream modules will work, but most of the time I'm going to be managing some repo of modules to accomplish what's being asked of me.

 

Concretely, some of the challenges for mono-repos are around versioning since one git tag applies to all the nested modules, and that many organisations need control over permissions that can be easily granted at repo level with different teams owning different modules etc.

I'd be fine with all modules in a mega-repo getting the same version.
 
At the same time, I think I can work out those issues without pushing them up to the registry, and I think this is easily accomplished with an API in the TF registry that allows us to create/release a module with a URL we control, EG, if I can upload a .zip to S3 and tell the registry about it and give it a version, we'd be fine, no?

 

> Does the Terraform Registry plan to avoid/block support for mega-repos, or to allow for their use on the Terraform platform?

I don't think we'd consciously try to "block" that use case, however as things are it's not obvious that the choices we make to optimize for reusable modules will necessarily be the right choices for people publishing mono repos.

It's not something we've actively tried to guard against though and will certainly be interested to see how organisations are interested in using the registry.

We would like to publish our modules on the registry, and use the registry as the standard way to distribute and use our modules, though we would like to manage some of these modules in mega-repos. Hosting the modules on S3 also seems to make sense for this situation.

 

> Would the Registry be open to providing an API for us to register/publish a new module that is just a URL to the actual archive file?

The public registry is unlikely to support this in the near future. The main reason for that is that we fairly intentionally choose to integrate it reasonably tightly with GitHub repositories to make it easier to work with and simpler to operate.

Integrating with Github is a great idea, but I am not sure I see how forcing or requiring it is. I would love to publish modules on the Terraform Registry. An API to publish the module as a zip built by my CI system would allow me to do so. Is this completely untenable? 

For the public registry, being highly opinionated with GitHub allows us to hold a lot of useful assumptions (repo description for description, open source in the backend, etc.). We want to encourage open source, publicly usable modules in the public registry and GitHub aligns well with that use case.

The private registry in TFE does have an API that allows arbitrary modules to be pushed into the registry. I actually think we don’t expose this currently, though (and still require a VCS), but the functionality is written and there. This was never meant to be a specific selling point for TFE so I’m most certainly not trying to sell you anything, but it is common for “enterprises” to have fairly complex CI/CD workflows that don’t fit well with the “git push & release” mindset of the public registry & VCS flow.

It is unlikely we’ll uncouple ourselves from GitHub for the public registry, it was a design decision we made early on to simplify quite a lot for us.

 

All that said I'll point out two things:

 1. The public registry _does_ support "nested" submodules - as long as there is at least one `.tf` file in the root module and preferably a README, you can put all your code into submodules which can actually be imported individually if required with `src = "org/module-name/provider//sub-module-name` we've not exposed that widely in docs/UI etc. yet but we do plan to make that a much more first-class experience soon. This doesn't solve for issues like the repo name convention but some of what you want can be achieved this way.

I'll happily rename a repo to make it work. But I am wondering, in our case.. we have all modules in `tf-modules`, and a bunch of test/example env in `tests`. Those tests use a bunch of the modules in `tf-modules` and it all works very well. You have noted the registry will improve this experience soon, would our repo as it is laid out with the modules in `tf-modules` and TF env in `tests`, work out ok with the nested submodule idea you have presented? or would we need to shuffle around those module/test locations?

any recommendations are appreciated :) 

As long as there is a root module (a “.tf” file in the root), you can reference any subdirectory using the advanced double-slash syntax:

source = “myuser/mymodule/myprovider//subdir”

This is a feature that is documented but only meant for advanced usage. The double-slash “//“ tells our downloader what separates the download URL from a directory change once downloaded. In this case, it’d download your module, then set the root directory to be “subdir” relative to that module.

What you’ll be sacrificing by using a non-standard format of course is documentation generation and so on. We’re also about to ship automatic example parsing/showcasing (via the examples directory as documented currently), showing submodules in the UI, and some more. And by using a non-standard structure you unfortunately won’t have any of this. But, you could still use the module.

I hope that helps, I’ve tried to lay out the options and tradeoffs. 

Best,

Mitchell

Thanks!

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/3cb22192-7382-4c8c-ae91-daaaebd6bff4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mitchell Hashimoto]

Just saw your other message, responding below:

Best,

Mitchell

This should work, if I’m understanding correctly.

If they’re published separately, you have to reference them using their full names `user/name/provider` and perhaps a version constraint.

If they’re published as “nested modules” (in the “modules/“ directory) you can reference them using relative paths.

If they’re published in a mega-repo, you can use relative paths.

Best,

Mitchell

Thanks!

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/8f954277-bb2f-4263-bea7-120445ee8dc1%40googlegroups.com.

[ja...@fpcomplete.com]

Thanks for the super fast reply Mitchell!

On Thursday, January 25, 2018 at 5:31:26 PM UTC-5, Mitchell Hashimoto wrote:

Hey,

Inline…

On Jan 25, 2018, 2:22 PM -0800, ja...@fpcomplete.com, wrote:

Best,

Mitchell

It is unlikely we’ll uncouple ourselves from GitHub for the public registry, it was a design decision we made early on to simplify quite a lot for us.

OK, I appreciate the open and clear communication on this, and FTR, I completely understand and am in support overall (just trying to find a reasonable way to make this work for us).

 

All that said I'll point out two things:

 1. The public registry _does_ support "nested" submodules - as long as there is at least one `.tf` file in the root module and preferably a README, you can put all your code into submodules which can actually be imported individually if required with `src = "org/module-name/provider//sub-module-name` we've not exposed that widely in docs/UI etc. yet but we do plan to make that a much more first-class experience soon. This doesn't solve for issues like the repo name convention but some of what you want can be achieved this way.

I'll happily rename a repo to make it work. But I am wondering, in our case.. we have all modules in `tf-modules`, and a bunch of test/example env in `tests`. Those tests use a bunch of the modules in `tf-modules` and it all works very well. You have noted the registry will improve this experience soon, would our repo as it is laid out with the modules in `tf-modules` and TF env in `tests`, work out ok with the nested submodule idea you have presented? or would we need to shuffle around those module/test locations?

any recommendations are appreciated :) 

As long as there is a root module (a “.tf” file in the root), you can reference any subdirectory using the advanced double-slash syntax:

source = “myuser/mymodule/myprovider//subdir”

This is a feature that is documented but only meant for advanced usage. The double-slash “//“ tells our downloader what separates the download URL from a directory change once downloaded. In this case, it’d download your module, then set the root directory to be “subdir” relative to that module.

OK, that makes sense.

 

What you’ll be sacrificing by using a non-standard format of course is documentation generation and so on. We’re also about to ship automatic example parsing/showcasing (via the examples directory as documented currently), showing submodules in the UI, and some more. And by using a non-standard structure you unfortunately won’t have any of this. But, you could still use the module.

I'm a little confused, is the submodule  / advanced use noted above non-standard? Or what is the non-standard part here? 

I hope that helps, I’ve tried to lay out the options and tradeoffs. 

Yes, definitely, thank you!

[Mitchell Hashimoto]

Inline below:

I may be confused as well. :) 

So the standard submodule (“nested module”) structure would be a directory “./modules/<name>” for each nested module. This is a flat directory — a nested module can’t itself have nested modules — but nested modules can reference each other via relative paths. If you do follow this format, you can have many modules in your single module and the registry frontend UI will eventually (very soon) expose these, including their README.

This is the structure that is currently documented here: https://www.terraform.io/docs/modules/create.html#standard-module-structure

My “non-standard” comment was just if you put the modules in any other structure. You could still reference them using the double-slash notation but the registry (and any future tooling) wouldn’t find or treat them special.

Best,

Mitchell

I hope that helps, I’ve tried to lay out the options and tradeoffs. 

Yes, definitely, thank you!

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/2e91742b-1f04-42fc-9a66-7c5d658170b2%40googlegroups.com.

[ja...@fpcomplete.com]

I'm a little confused, is the submodule  / advanced use noted above non-standard? Or what is the non-standard part here?  

I may be confused as well. :) 

So the standard submodule (“nested module”) structure would be a directory “./modules/<name>” for each nested module. This is a flat directory — a nested module can’t itself have nested modules — but nested modules can reference each other via relative paths. If you do follow this format, you can have many modules in your single module and the registry frontend UI will eventually (very soon) expose these, including their README.

GREAT!

 

My “non-standard” comment was just if you put the modules in any other structure. You could still reference them using the double-slash notation but the registry (and any future tooling) wouldn’t find or treat them special.

 

OK, that definitely makes sense!

TY

 

[pba...@hashicorp.com]

Hey Jason,

I think Mitchell covered pretty much everything there so I won't repeat but I just wanted to add one response to:

> I'd be fine with all modules in a mega-repo getting the same version.
>
> At the same time, I think I can work out those issues without pushing them up to the registry, and I think this is easily accomplished with an API in the TF registry that allows us to create/release a module with a URL we control, EG, if I can upload a .zip to S3 and tell the registry about it and give it a version, we'd be fine, no?

You probably know this, but you can already import modules directly from GitHub [1] or S3 [2] (or many other places)  just as easily as the registry. You don't get version constraint support but as discussed that's of minimal value with a mono-repo anyway - you can still bind to a specific commit or pet a version string in the S3 object name etc.

I'm not trying to turn you away from the registry at all. We think it's going to be the best and most universal experience for publishing TF modules and as Mitchell has said you may well be able to use //paths and/or the standard structure to work with it as it is just fine. 

But to be pragmatic, if you don't need any of it's primary benefits of versioning and automated documentation and the ergonomics aren't working for you then there are other solutions too! The registry for now will likely remain optimised for people discovering smaller, composable modules since that is a use-case we think will become increasingly important and is not well served but the existing tooling (due to discovery, manual version constraint solving etc.).

Thanks!

Paul

 [1] https://www.terraform.io/docs/modules/sources.html#github

 [2] https://www.terraform.io/docs/modules/sources.html#s3-bucket

[ja...@fpcomplete.com]

Hi Paul,

Thanks for the follow-up.

On Friday, January 26, 2018 at 7:52:24 AM UTC-5, pba...@hashicorp.com wrote:

Hey Jason,

I think Mitchell covered pretty much everything there so I won't repeat but I just wanted to add one response to:

> I'd be fine with all modules in a mega-repo getting the same version.
>
> At the same time, I think I can work out those issues without pushing them up to the registry, and I think this is easily accomplished with an API in the TF registry that allows us to create/release a module with a URL we control, EG, if I can upload a .zip to S3 and tell the registry about it and give it a version, we'd be fine, no?

You probably know this, but you can already import modules directly from GitHub [1] or S3 [2] (or many other places)  just as easily as the registry. You don't get version constraint support but as discussed that's of minimal value with a mono-repo anyway - you can still bind to a specific commit or pet a version string in the S3 object name etc.

Yes, and we have run various tests with different methods for publishing modules. We ran into issues with most of them, and/or have always preferred the registry and didn't want to choose a path until we fully explored the registry options with you (TF devs/etc). That said.. this whole bit about submodules seems to address those issues. I think that's the missing link, or at least a hole I can squeeze though (rename the repo, shuffle some paths, add a top-level `main.tf` that does some no-op or default, and wait for the registry UI updates). I'm going to try it at least :)

 

But to be pragmatic, if you don't need any of it's primary benefits of versioning and automated documentation and the ergonomics aren't working for you then there are other solutions too! The registry for now will likely remain optimised for people discovering smaller, composable modules since that is a use-case we think will become increasingly important and is not well served but the existing tooling (due to discovery, manual version constraint solving etc.).

We've got a lot of experience with ecosystems of composable modules like this (Haskell), and we're completely in support of that, even if we package the modules as a group for a while longer :)

Thank you for the discussion and guidance Mitchell and Paul!

Best,

J