How to deal with generated files in repo

32 views
Skip to first unread message

Brendan Barnwell

unread,
Nov 21, 2016, 1:27:46 PM11/21/16
to brython
I see that the github repo includes a number of files like brython.js and brython_dist that are generated from the main source files.  But there is no generated equivalent of the Brython3.2.9-xxxx.zip files that contain the ordinary Brython distribution.  This is causing some awkwardness for me in git.  Whenever I pull from the repo, I need to run make_full_dist.py to generate the zip file so I can deploy it.  But then it regenerates all the files.  Then the next time I pull, I have a bunch of changes to the generated files that conflict with whatever changes were made in the github repo.  It also means that nearly every commit includes a bunch of changes to those huge generated files, although these changes are redundant with the changes to the "real" source used to generate them.  There is also a static_doc folder which is not part of the repo, but is also generated by make_full_dist.  Since it is not ignored, everything in there shows up as "new" whenever make_full_dist is run.

How do other people handle this if they are making use of the github repo?  Or is there some procedure for incorporating changes into a locally deployed Brython other than make_full_dist?  Right now I have added the static_doc folder to .gitignore, and then every time I sync my repo with github I have to discard whatever local changes were made to the generated files.  Is this how everyone does it?  It seems a bit strange to have the generated files as part of the repo, because of the way they cause changes to be "amplified" through many files.

Kiko

unread,
Nov 23, 2016, 2:42:34 AM11/23/16
to bry...@googlegroups.com
2016-11-21 19:27 GMT+01:00 Brendan Barnwell <bren...@gmail.com>:
I see that the github repo includes a number of files like brython.js and brython_dist that are generated from the main source files.  But there is no generated equivalent of the Brython3.2.9-xxxx.zip files that contain the ordinary Brython distribution.  This is causing some awkwardness for me in git.  Whenever I pull from the repo, I need to run make_full_dist.py to generate the zip file so I can deploy it.  But then it regenerates all the files.  Then the next time I pull, I have a bunch of changes to the generated files that conflict with whatever changes were made in the github repo.  It also means that nearly every commit includes a bunch of changes to those huge generated files, although these changes are redundant with the changes to the "real" source used to generate them.  There is also a static_doc folder which is not part of the repo, but is also generated by make_full_dist.  Since it is not ignored, everything in there shows up as "new" whenever make_full_dist is run.

There isn't a 3.2.9 version.
To build brython.js, py_VFS.js and brython_dist.js you should use make_dist.py (https://github.com/brython-dev/brython/blob/master/scripts/make_dist.py). make_full_dist.py creates the full distribution in a zip file (among other stuff).
 

How do other people handle this if they are making use of the github repo?  Or is there some procedure for incorporating changes into a locally deployed Brython other than make_full_dist?  Right now I have added the static_doc folder to .gitignore, and then every time I sync my repo with github I have to discard whatever local changes were made to the generated files.  Is this how everyone does it?  It seems a bit strange to have the generated files as part of the repo, because of the way they cause changes to be "amplified" through many files.

I think the best way would be to create your own repo of your page that includes just the brython files used in the page and merge the changes there and not in a complete brython fork.
 

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+unsubscribe@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/dfda8cc8-72ec-4ee6-a03f-4585c8d737be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Brendan Barnwell

unread,
Dec 2, 2016, 2:23:03 AM12/2/16
to brython
On Tuesday, November 22, 2016 at 11:42:34 PM UTC-8, kiko (on pybonacci) wrote:


2016-11-21 19:27 GMT+01:00 Brendan Barnwell <bren...@gmail.com>:
I see that the github repo includes a number of files like brython.js and brython_dist that are generated from the main source files.  But there is no generated equivalent of the Brython3.2.9-xxxx.zip files that contain the ordinary Brython distribution.  This is causing some awkwardness for me in git.  Whenever I pull from the repo, I need to run make_full_dist.py to generate the zip file so I can deploy it.  But then it regenerates all the files.  Then the next time I pull, I have a bunch of changes to the generated files that conflict with whatever changes were made in the github repo.  It also means that nearly every commit includes a bunch of changes to those huge generated files, although these changes are redundant with the changes to the "real" source used to generate them.  There is also a static_doc folder which is not part of the repo, but is also generated by make_full_dist.  Since it is not ignored, everything in there shows up as "new" whenever make_full_dist is run.

There isn't a 3.2.9 version.

I'm just talking about the file that is generated when I run make_full_dist.
 
To build brython.js, py_VFS.js and brython_dist.js you should use make_dist.py (https://github.com/brython-dev/brython/blob/master/scripts/make_dist.py). make_full_dist.py creates the full distribution in a zip file (among other stuff).

Yes, I know.  What I am looking for is a way to generate that full distribution, instead of (or in addition to, brython_dist.js) so I can, if I want, deploy that instead of using brython_dist.js.
 

How do other people handle this if they are making use of the github repo?  Or is there some procedure for incorporating changes into a locally deployed Brython other than make_full_dist?  Right now I have added the static_doc folder to .gitignore, and then every time I sync my repo with github I have to discard whatever local changes were made to the generated files.  Is this how everyone does it?  It seems a bit strange to have the generated files as part of the repo, because of the way they cause changes to be "amplified" through many files.

I think the best way would be to create your own repo of your page that includes just the brython files used in the page and merge the changes there and not in a complete brython fork.

I could do that, but that seems rather awkward.  Then I would have to somehow manually integrate the changes from the real Brython repo (e.g., by copying the files over), rather than just pulling them.  Also, it doesn't make sense to do that if I'm working on Brython itself.  Right now, if I'm working on Brython, I can't even rebase without having to discard all the changes in the generated files.  I can do it, but it's annoying extra step any time I want to integrate upstream changes.

I guess my underlying question is why the Brython repo includes the generated files at all, rather than simply .gitignore-ing the entire /dist/ tree.  I think in general including generated files in the repo is a questionable practice, because it leads to redundancy and the possibility of generated files being out of sync with the files used to generate them.  (This can be seen in the current repo with occasional commits saying "updated generated files".)

Kiko

unread,
Dec 2, 2016, 3:21:22 AM12/2/16
to bry...@googlegroups.com
The make_dist.py or the make_full_dist.py files are not necessary to modify the brython source to commit changes to the repo. You do not have to modify directly brython.js, py_VFS.js or brython_dist.js directly.

In general, what I do is:
In general, the brython.js, py_VFS.js or brython_dist.js are usually generated by Pierre after a PR or before a new release except if you want to check something specific in your own or if you want to create a version for your site with your specific unofficial changes.

I don't know it this answer your question/doubts.

Best.

Brendan Barnwell

unread,
Dec 8, 2016, 2:49:13 AM12/8/16
to brython
On Friday, December 2, 2016 at 12:21:22 AM UTC-8, kiko (on pybonacci) wrote:
The make_dist.py or the make_full_dist.py files are not necessary to modify the brython source to commit changes to the repo. You do not have to modify directly brython.js, py_VFS.js or brython_dist.js directly.


I guess I'm not making myself clear.  Here is what I do when I make a change to Brython:

1. I make a modification to some in the Brython www/src directory
2. I run make_dist to make brython_dist
3. I load up some web page I have written to test the new functionality

If at any point during this I decide to pull new changes from the Github repository, I will have conflicting changes to my repo, because I had to regenerate the generated files.  I know that I don't have to make direct modifications to the files, but I can't avoid modifying them because I can't use Brython without them.
 
In general, what I do is:
In general, the brython.js, py_VFS.js or brython_dist.js are usually generated by Pierre after a PR or before a new release except if you want to check something specific in your own or if you want to create a version for your site with your specific unofficial changes.

I don't know it this answer your question/doubts.

I see.  But what if you want to test some other web page?  What I would like is to be able to make a web page that uses Brython exactly as it would in normal use --- that is, by referencing brython_dist or brython_js, not the individual files --- and be able to test Brython modifications on that.  Right now that is very cumbersome, because the generated files are stored in the repo.  I see from what you say that it is possible to test modifications if you make your test page reference all the files individually, but that in itself seems a cumbersome process.
Reply all
Reply to author
Forward
0 new messages