The problem of versioning a large project.

[Alex Sonar]

The problem of versioning a large project.

Hi guys, I really got stuck with the challenge, with the design of versioning for a large project.

Now we have organized versioning, where our project is divided into separate GIT repositories which are created for each application.

The new necessity point is to split some of them for front-end and back-and developers.

The task looks like bad practice, where we have to create a repository within another one.

Or redesign the file structures of the application itself, to meet this requirement.

If someone has a similar challenge or practice and helps me make the right decision, I will be very grateful.

[Aharon Leibman]

Could you give more information, please?
the project structure provided to your GIT repository
It is not entirely clear how you perform separation by application

среда, 29 марта 2023 г. в 23:17:41 UTC+3, Alex Sonar:

[cha...@gmail.com]

Like Aharon, I'm not entirely certain what you mean but this line:

> The task looks like bad practice, where we have to create a repository within another one.

reminded me of git submodules, which I don't think are considered bad practice. Is that what you mean?

[Jason Johns]

>Now we have organized versioning, where our project is divided into separate GIT repositories which are created for each application.

why? what's the reasoning for this? django already has apps, so why do you need to make multiple projects for single components?

[Alex Sonar]

It is about one project that includes many applications.

We would like to separate the repository specifically for front-end guest groups.

With access to parts like: views_groups, templates, dummy_data, forms_group, helpers_group, static_blob and media_blob as an example.

Now our application tier and the file structure of the project looks something like in the picture below.

   . 

    ├── apps

     .     ├── .DS_Store

           ├── app_alpha

           │   ├── .git              <<<<<<<<<<<<<<<<<<<<<<<<<   git alpha

           │   └── alpha

           │         ├── __init__.py

           │         ├── admin_alpha.py

           │         ├── api_connectors

           │         ├── apps_alpha.py

           │         ├── conf_parser_alpha.py

           │         ├── controllers_groups

           │         ├── dummy_data_alpha

           │         ├── db_router_alpha.py

           │         ├── forms_group

           │         ├── helpers_group

           │         ├── media_blob_alpha

           │         ├── metadata

           │         ├── migrations

           │         ├── models_groups

           │         ├── signals_alpha.py

           │         ├── static_blob__alpha

           │         ├── templates

           │         ├── urls_alpha.py

           │         └── views_groups

           ├── app_beta

           │   ├── .git              <<<<<<<<<<<<<<<<<<<<<<<<<    git beta

           │   └──  beta

           │         ├── __init__.py

           │          …….

           ├── app_gamma

           │   ├── .git              <<<<<<<<<<<<<<<<<<<<<<<<<    git gamma

           │   └──  gamma

           │         ├── __init__.py

           │          …….

           ├── app_delta

           │   ├── .git              <<<<<<<<<<<<<<<<<<<<<<<<<    git delta

           │   └──  delta

           │         ├── __init__.py

           …..      …….

[John Whitlock]

I'm not sure your issue is Django-specific, so it may be off-topic for this group. However, your question is a bit vague to be appropriate for Stack Overflow or other forums either.

"Versioning" could mean a few things. At first, I thought you meant something like date-based or semantic versioning. It sounds like you are talking about code repository organization, and how to integrate the work of several apps or teams. In my experience, this is a people and process problem first, and a code issue second or third.

My first advice would be to abandon a git repo per app and combine everything into a single repo, where all the code is in one repository. I find this much easier to work with and to collaborate with other developers, especially on a web project where deployment is frequent and mostly automated, and especially using a tool like GitHub or GitLab. A separate branch and a pull request is used to bring in new features and bug fixes. Reviewers are assigned based on the areas touched, and GitHub has features like code owners to automate review assignment. The HEAD is the latest version, and the git tag feature can be used to identify key versions like releases. I prefer a repo per "project", which may have several deployed "containers", such as a web frontend, an API, a background processor, and periodic tasks. We often use Docker containers to collect the project into an executable package, and use different entrypoints in the same image for the different containers.

Some companies have taken the single repo to the extreme of a "monorepo", a single codebase for all the company's projects. This makes code sharing possible across projects, but also requires a lot of tools and process to manage the repo and changes to it. Google's monorepo is a famous example. Mozilla's repo for Firefox / Gecko has similar qualities, including a custom search website, a pull request review system, and their own continuous testing architecture. I don't recommend this level of monorepo except for huge projects with hundreds of committers and the budget for dedicated build staff.

If you continue with multiple repos, the "git submodule" feature specifically supports a git repo inside another git repo. Updating a submodule to a different version is a commit in the "parent" project. I find submodules awkward to use, but useful for truly parallel work. At Mozilla, we often use a git submodule for translated strings. The translation team makes several daily commits to add new translations. A github action updates the submodule to the latest version overnight. To add new strings, we open a pull request on the translation repo.

If you decide to continue with separate repos per app, and further split to a frontend repo, the first question is "how many?" A single frontend repo could have the templates for all your projects. This would require good documentation and standardization for your view code, so that the templates could be coded without necessarily peeking at the view code. Many complex Django apps, such as django-allauth, ship simple base templates with instructions for overriding them to match the feel of your website.

Another method would be for your app to expose an API, and build the frontend on another technology like React. The frontend would call a RESTful API implemented in django-rest-framework or similar. You could use nginx to serve the frontend files, or a Django app like whitenoise. This would allow the frontend developers to work independently from the backend Django devs. I've found that even in this case it is useful for a frontend dev to have access to backend code, to coordinate a change or understand an API, and sometimes it is easier for the backend dev to fix something in the frontend. I still recommend a single repo, even with a frontend / API separation.

I suspect you have constraints you have not mentioned, such as using external contractors and not wanting them to have access to some of the code. If this is the case, separate repos may be the best way to split the code, and give access to some but not others. This is still doing things the hard way! If possible, work with reliable people with contracts to remove those barriers. If not, make the code work for your business constraints, such as an API to give a clean separation between insiders and outsiders.

I hope this helps
John

[Piyush Prasad]

Why not use a mono repo? 

---

From: django-d...@googlegroups.com <django-d...@googlegroups.com> on behalf of Alex Sonar <alex....@gmail.com>
Sent: Sunday, April 2, 2023 10:21:33 PM
To: Django developers (Contributions to Django itself) <django-d...@googlegroups.com>
Subject: Re: The problem of versioning a large project.

 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1c2df2fc-0158-4da1-a686-5f2a69435f43n%40googlegroups.com.

[Christian González]

I think that's not possible with Django without some quirks. You need to add apps to your settings.py anyway, so, if these apps are not present in your repo, it's not working, as there are imports that lead into nowhere.

Django provides really good support for creating reusable "apps" that can be used in different Django applications, if correctly done. But you have to include them hard-coded, it's not possible to create more apps that "plug" into ONE Django application and extend its functionality dynamically. And it seems you need that in your case.

So you need either a mono repo, or git submodules that need to be fetched. I solved a similar problem by building a library that adds such a plugin support to Django: https://gdaps.readthedocs.io

You can use a Django main application, and install app "plugins" that add funcationality to your app without touching the settings.py. Just install the Django/GDAPS app per pip, restart the Django server / manage migrate, and it should work.

It's not perfect ans still lacks some features, but should solve your problem.

Yours, Christian

Am 03.04.23 um 16:04 schrieb Piyush Prasad:

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/PH0PR15MB5640DD26A7890FC754671486FC929%40PH0PR15MB5640.namprd15.prod.outlook.com.

-- 
Dr. Christian González
https://nerdocs.at