Repository
Yuv
Recommended reading (for everybody, not just developers) <http://
hginit.com/>
For years Hugin lived in Subversion (after being migrated there from
CVS, but that's pre-history). SVN was the state of the art. It has
worked well for the project as we learned to use its features and deal
with its limitations.
In the meantime a new generation of distributed version control
systems (DVCS) has matured and is now the state of the art. There are
at least three contenders:
* bazaar (bzr), used by Pablo recently for the patent-free control
point detector
* git, used amongst others for Linux kernel development. On the Hugin
project I know Gerry Patterson and a few others interface git
automatically to Hugin's SVN.
* mercurial (hg), Andrew Mihal migrated Enblend-Enfuse from CVS to hg
a year or so ago.
The one limitation of SVN that DVCS improve dramatically on is merging
of (development) branches.
Conceptually the current generation of DVCS are very similar and share
the major features set. I personally favor mercurial because I found
GUI integration (such as TortoiseHg, similar to TortoiseSVN) to be
more mature.
Recommended Hg tutorial (a well spent 30 minutes reading of nicely
illustrated text by Joel Spolsky): <http://hginit.com/index.html>
*PROPOSAL*
I've been toying with Hg, and I herewith propose a (partial) switch to
Hg, specifically for the *development* branches. This would make the
management of multiple codelines easier.
Before migrating, the following criteria must be met IMO (feel free to
discuss / add):
1. there has to be an easy two-way interface (from SVN to Hg and from
Hg to SVN), at least for the transition period. During such a
transition period (which could extend indefinitely if we find that the
hybrid SVN/Hg model works best for the project), developers would
start their branches on Hg (see instructions below) and from time to
time push back changes into SVN. The release process would still run
like today, on SVN. SVN would be reduced to "trunk" and "release".
2. the tools to access the new DVCS must be at least equally
comfortable on all currently used platforms. This is the advantage of
Hg, even if the other tools have caught up quickly.
3. all the revision history must be preserved in case of a complete
migration leaving SVN behind. This is the point at which I am
currently stuck.
There is at least one manipulation in the Hugin SVN repository at
revision 2494 that breaks the import of trunk. This is when Ippei's
development branch (from GSoC 2007) became the "new" trunk. Maybe
there will be a way around it by trying to migrate the whole
repository and not just Hugin's trunk. I'll look into that when I have
time. The process of importing a full history is very slow (and still
going while I write this).
Here is what I did in Kubuntu 9.10:
1. INSTALL THE TOOLS
$ sudo apt-get install python-setuptools
$ sudo easy_install hgsvn
2. IMPORT FROM SVN
$ hgimportsvn https://hugin.svn.sourceforge.net/svnroot/hugin/hugin/trunk/
wait a loooong time for the import to complete
3. FETCH THE HISTORY
$ cd trunk
$ hgpullsvn
wait an even longer time while the whole revision log slowly scrolls
under your eyes.
In *theory* I should be able to make changes to the code using Hg and
then push them back into SVN with the following command:
$ hgpushsvn
I have not tried it yet because I have nothing to commit at this time.
hgsvn is documented at <http://pypi.python.org/pypi/hgsvn>
The way I understand it now (not completely yet - more investigations
required) this interface is enough for current and future changes as
long as we keep SVN as the master repository. The Hg repository can be
used locally (faster, more atomic commits, less fear of breaking
things) and maybe even pushed to Hg on SourceForge. Developers (e.g.
GSoC students) could maintain a backup of their work on an Hg
repository on SourceForge, leaving SVN uncluttered for integration and
distribution work.
if the problem of hgpullsvn starting at revision 2495 only is solved
(I think it will be if I pull the whole repository rather than trunk
only, and after pulling clean up the folder structure), then we could
potentially migrate Hugin completely to Hg if there is a wish to do
so.
For now I'd suggest to stay with the best of both world, Hg for
development, SVN for release and distribution.
There are a few different projects all living in the same Hugin SVN
repository:
* KimageFuser
* autopano-sift-C
* hugin (Hugin itself)
* htdocs (the Hugin website)
* libpanorama
* panoglview
It would make sense IMHO to have a separate repository for each and
every one of them. If there is consensus, I can start migrating the
smaller ones and thus gain experience toward the migration of Hugin
itself.
What do you think?
Yuv
Yuv
Michael Witten
> The one limitation of SVN that DVCS improve dramatically on is merging
> of (development) branches.
I would add that there are essentially 2 related reasons why people contribute
to any project:
(1) To improve the project.
(2) To receive recognition and glory.
and in my experience, (1) is not nearly so sweet without (2).
One of the most important features provided by DVCSs is recognition;
you don't need commit privileges to have your name properly credited
for your contributions, and no particular repository is *inherently*
more important than another.
Also, patch series can be sculpted locally until they are beautiful
works of art, and patch transmission or repository communication is
easy.
Michael Witten
Harry van der Wolf
Recommended Hg tutorial (a well spent 30 minutes reading of nicely
illustrated text by Joel Spolsky): <http://hginit.com/index.html>
*PROPOSAL*
I've been toying with Hg, and I herewith propose a (partial) switch to
Hg, specifically for the *development* branches. This would make the
management of multiple codelines easier.
<snip>
For now I'd suggest to stay with the best of both world, Hg for
development, SVN for release and distribution.
There are a few different projects all living in the same Hugin SVN
repository:
* KimageFuser
* autopano-sift-C
* hugin (Hugin itself)
* htdocs (the Hugin website)
* libpanorama
* panoglview
It would make sense IMHO to have a separate repository for each and
every one of them. If there is consensus, I can start migrating the
smaller ones and thus gain experience toward the migration of Hugin
itself.
What do you think?
Yuv
T. Modes
> 1. there has to be an easy two-way interface (from SVN to Hg and from
> Hg to SVN), at least for the transition period. During such a
> transition period (which could extend indefinitely if we find that the
> hybrid SVN/Hg model works best for the project), developers would
> start their branches on Hg (see instructions below) and from time to
> time push back changes into SVN. The release process would still run
> like today, on SVN. SVN would be reduced to "trunk" and "release".
>
I don't think that this will work well. Keep two different
repositories at sync - bug fixing the one and new features in the
others - will make much trouble. E.g. a bug was reported, one
developer fixed it in svn; in the mean time other developer is working
on an own branch, found the same bug but fixed it other way. How do
you want to solve this conflict automatic. Therefor all development
should happen in only one repository and one system.
Other point: hugin in svn is using some svn features like
svn:keywords. I don't think that this can be automaticly converted
between different revision systems.
> 2. the tools to access the new DVCS must be at least equally
> comfortable on all currently used platforms. This is the advantage of
> Hg, even if the other tools have caught up quickly.
For my workflow TortoiseSVN is more comfortable. Import and export
patches (which I need for the normal development more than merging)
works good, IMHO it is more complicate solved in mercurial or bzr (at
least on windows). Also when working on bug fixes and new features at
the same time it works for me better with TortoiseSVN than with
mercurial/TortoiseHG. E.g in mercurial I find it complicated to
separate between the changes which should go into the public
repository and the changes which are only for the private repository.
So I would leave hugin leave in svn.
Thomas
Lukáš Jirkovský
Hi Yuv,
I'd love to see the mercurial as the main VCS. I'm using it internally
for hugin for some time via hgsvn. But there's one thing about hgsvn
everybody should be aware of – you can't mix usage of subversion and
mercurial. It happened to me once that I committed the change using
svn and later when I committed another change using hgpushsvn the
previous one was reverted because it wasn't in mercurial repository.
I think that first run of hgpullsvn makes tremendous load on sf.net
mirrors and I think that it's not good. I mean they allows us to use
their servers for free and doing this is not very nice. On the other
hand you no longer need to contact server when you want to try older
revision!
I'd like too add another nice guide to mercurial – http://hgbook.red-bean.com/
And for everyone who haven't used mercurial yet – try it, it's one of
the best VCSs I ever tried. It combines simplicity of subversion with
the great power of git.
Lukas
Yuval Levy
opinions about Hg. In the meantime, a few notes, trying not to advocate the
change, just addressing points raised and acknowledging that more discussion
and experimentation are needed on both sides of the argument.
On March 17, 2010 01:09:32 am Michael Witten wrote:
> One of the most important features provided by DVCSs is recognition;
> you don't need commit privileges to have your name properly credited
> for your contributions, and no particular repository is *inherently*
> more important than another.
The Hugin project has a very liberal SVN access privileges policy, so that
part of the argument is moot.
That said, the rest of the argument has enough weight to stand on its own. You
are right and it is worth repeating: *No particular repo is inherently more
important than another*. In the short term people may still look at the
"official" SF repo as being *the one* authoritative voice. As they understand
and embrace Hg (or other DVCs) they will see the superior power of
decentralization.
Interesting that Version *Control* Systems are becoming the ultimate tool to
do the opposite - undermine the control of a central authority to bring
freedom of the code to a higher level. One could argue that DVCS are the
ultimate forking tool - forks are made redundant because every cloned repo is
technically a fork, with equal weight / importance as any other repo, and with
equal opportunity to thrive, develop, and become the next authoritative expert
on whatever the code is meant to do.
DVCS are to source code management what Free licenses are to software
distribution. More power to the user.
On March 17, 2010 03:44:50 pm T. Modes wrote:
> I don't think that this will work well. Keep two different
> repositories at sync - bug fixing the one and new features in the
> others - will make much trouble. E.g. a bug was reported, one
> developer fixed it in svn; in the mean time other developer is working
> on an own branch, found the same bug but fixed it other way. How do
> you want to solve this conflict automatic.
Such conflicts already happen today, just not in a clear, formalized setup.
E.g when two developers decide to fix the same bug. One commits and make the
other's work redundant.
The described scenario would not happen because access to SVN is managed. It's
easy to lock SVN making it read-only and allowing commits only from an
hgsvnpush cron job.
The FreeBSD project still uses CVS for distribution but development has moved
on. They have Perforce; and they also support Hg.
<http://wiki.freebsd.org/LocalMercurial>
> Other point: hugin in svn is using some svn features like
> svn:keywords. I don't think that this can be automaticly converted
> between different revision systems.
<http://mercurial.selenic.com/wiki/KeywordExtension> is part of the default Hg
distro.
> For my workflow TortoiseSVN is more comfortable.
I have not been on Windows in a while. AFAIK TortoiseHg mirrors closely all
TortoiseSVN comfort?
> in mercurial I find it complicated to
> separate between the changes which should go into the public
> repository and the changes which are only for the private repository.
<http://www.joelonsoftware.com/items/2010/03/17.html>
<Quote>
And here is the most important point, indeed, the most important thing that
we’ve learned about developer productivity in a decade. It’s so important that
it merits a place as the very last opinion piece that I write, so if you only
remember one thing, remember this:
*When you manage changes instead of managing versions, merging works better,
and therefore, you can branch any time your organizational goals require it,
because merging back will be a piece of cake.*
</Quote>
Why do I have the impression that you are applying an SVN mindset to Hg?
Try to forget SVN and do it the Hg way. After you did it get back to compare
results. Did you read the tutorial, specifically <http://hginit.com/05.html> ?
it is written by somebody who was in your same situation.
The Hg solution to this problem is two (or more) local repositories: one
"public". You pull/push to/from it from/to the public SF Hg; the other(s) are
"private". You pull/push to/from them from/to your "public" repo. You can
atomically control which changeset is pushed/pulled to/from every repository.
If you want you can even push/pull between your local, "private" repos, and
revert individual changesets.
But maybe I am missing something?
On March 17, 2010 08:56:35 am Harry van der Wolf wrote:
> (For now) being the only developer for KImageFuser I kindly request you
> to create a central KImageFuser mercurial repository and give me access.
OK, I took note of this. Will do and get back to you here.
> And I do have a good backup mechanism in place ;-)
yes, that's one nice side benefits of DVCS - everybody gets to download the
whole repository (which his highly compacted) and so there are many backups in
place.
On March 18, 2010 04:04:03 am Lukáš Jirkovský wrote:
> you can't mix usage of subversion and
> mercurial. It happened to me once that I committed the change using
> svn and later when I committed another change using hgpushsvn the
> previous one was reverted because it wasn't in mercurial repository.
Did you try to hgpullsvn again before doing hgpushsvn? There have been a few
commits to SVN since I did my initial import. running hgpullsvn got the
changesets in correctly. But I have not made any change in my Hg repo, so I am
not sure if it will raise a merge conflict or simply revert changes.
> I think that first run of hgpullsvn makes tremendous load on sf.net
> mirrors
*IMPORTANT: NOBODY SHOULD REPEAT WHAT I DOCUMENTED AT THE BEGINNING OF THIS
THREAD.*
I just realized that I have caused a spike in the SF stats. When we normally
have around 150 read transactions per day, pulling all of the repo is roughly
5000 (one read transaction per revision). Please do not follow the commands I
posted at the beginning of this thread.
I'll post the new hg repo later (in a few days / weeks), so people who want to
play with it can start from that hg repo and only pull the latest changes from
SVN. Instructions will follow.
My previous experiments with SVN <-> Hg were on own server and I was not aware
of the TCP traffic involved. I had to interrupt them when I decommissioned the
old system drive and re-installed the FreeBSD 8.0 server from scratch on a new
drive. No SVN (yet) on the new drive, so I took the shortcut of doing the next
step directly on the Hugin SF repo. I understood after you mentioned it that I
basically pulled each and every single SVN revision since inception of the
project and this is how hgsvn builds an hg repo.
So again. please save SF's bandwidth and don't do what I did. I will make the
results available to everybody.
> I'd like too add another nice guide to mercurial –
> http://hgbook.red-bean.com/
yes, very nice and definitely more complete.
please, more!
Yuv
Lukáš Jirkovský
> Did you try to hgpullsvn again before doing hgpushsvn? There have been a few
> commits to SVN since I did my initial import. running hgpullsvn got the
> changesets in correctly. But I have not made any change in my Hg repo, so I am
> not sure if it will raise a merge conflict or simply revert changes.
I don't remember really well, it was about a month ago but it's possible.
Lukas
Simon Oosthoek
> Hi all. Thank you for the feedback. *Keep it coming*. I want to hear more
> opinions about Hg. In the meantime, a few notes, trying not to advocate the
> change, just addressing points raised and acknowledging that more discussion
> and experimentation are needed on both sides of the argument.
>
>
>
I have more experience with git, but hg is probably just as good,
perhaps easier to use if I understand correctly.
I think it will be great for hugin to use a distributed versioning
system, centralised systems are much less suitable for open source
projects I think. (And if you want centralised, it can be done using
git/Hg as well).
for git you can read a pretty good book online http://progit.org
One thing that git can do much cheaper than subversion is to make
branches. This does affect the workflow of the project and this is
something you need to define before getting too deep into it. In the
progit.org book (you can buy it on dead trees as well), a chapter is
dedicated to the workflow possibilities, this is a good read to get
started (doesn't matter if you choose Hg or git)
Cheers
Simon
Harry van der Wolf
On March 17, 2010 08:56:35 am Harry van der Wolf wrote:OK, I took note of this. Will do and get back to you here.
> (For now) being the only developer for KImageFuser I kindly request you
> to create a central KImageFuser mercurial repository and give me access.
> And I do have a good backup mechanism in place ;-)yes, that's one nice side benefits of DVCS - everybody gets to download the
whole repository (which his highly compacted) and so there are many backups in
place.
*IMPORTANT: NOBODY SHOULD REPEAT WHAT I DOCUMENTED AT THE BEGINNING OF THIS
THREAD.*
I just realized that I have caused a spike in the SF stats. When we normally
have around 150 read transactions per day, pulling all of the repo is roughly
5000 (one read transaction per revision). Please do not follow the commands I
posted at the beginning of this thread.
please, more!
Yuv
Bruno Postle
> Hi all. Thank you for the feedback. *Keep it coming*. I want to hear more
> opinions about Hg.
Just adding that I'm in favour of a move to mercurial so long as it
can be done reasonably painlessly.
Now is also the time to do it, with the 2010.0.0 branch about to
finish and before we start creating new branches for the Sumer of
Code.
--
Bruno
T. Modes
> The described scenario would not happen because access to SVN is managed. It's
> easy to lock SVN making it read-only and allowing commits only from an
> hgsvnpush cron job.
>
But then the svn repository is only a copy of the mercurial. What is
then the purpose of the svn repo?
This will make only problems. E.g. take the revision number or
changeset. First you will need two part in the cmake system to extract
the different numbers. And when the svn is read only, you have to test
the svn code in a hg repo? Okay, this needs only done once. But its a
additional complicitation.
But it can be more complicated: one user is using svn and reporting a
bug with a revision number, the other user/developer is using hg and
has a changeset (so you will need a translation between both numbering
schemes.
So I would recommend only mercurial or only subversion, but not both
at the same time.
> > Other point: hugin in svn is using some svn features like
> > svn:keywords. I don't think that this can be automaticly converted
> > between different revision systems.
>
> <http://mercurial.selenic.com/wiki/KeywordExtension> is part of the default Hg
> distro.
But why have I to activate it separately? But the point is, what
happens with the keywords when converting between svn and hg?
>
> > For my workflow TortoiseSVN is more comfortable.
>
> I have not been on Windows in a while. AFAIK TortoiseHg mirrors closely all
> TortoiseSVN comfort?
>
TortoiseHg has improved with the last version, which was released last
week. TortoiseHg was behind TortoiseSVN from my point of view. With
the new version TortoiseHg is a step further. But I have not tested it
much. (I updated today.)
> Why do I have the impression that you are applying an SVN mindset to Hg?
>
> Try to forget SVN and do it the Hg way. After you did it get back to compare
> results. Did you read the tutorial, specifically <http://hginit.com/05.html> ?
> it is written by somebody who was in your same situation.
>
> The Hg solution to this problem is two (or more) local repositories: one
> "public". You pull/push to/from it from/to the public SF Hg; the other(s) are
> "private". You pull/push to/from them from/to your "public" repo. You can
> atomically control which changeset is pushed/pulled to/from every repository.
> If you want you can even push/pull between your local, "private" repos, and
> revert individual changesets.
>
> But maybe I am missing something?
>
Okay, maybe I have to change my workflow with Hg. But the approach
with two or more repository has a major drawback (at least for me):
You will need for each repository an own build directory. This takes
much hard disk space. The normal build directory (only Release) has
already 1 GB. When I compile a Debug variant it becomes over 3 GB (or
even more). So this approach will take much space. (And don't argue
with hard disc space is no matter. That's not a solution for me.)
Currently I have already 2 repos (one is the 2010.0 branch, the other
is the trunk). When I will need an additional repo to work on new
features I will need even more space. Do you have also a solution for
this problem?
Thomas
Harry van der Wolf
Okay, maybe I have to change my workflow with Hg. But the approach
with two or more repository has a major drawback (at least for me):
You will need for each repository an own build directory. This takes
much hard disk space. The normal build directory (only Release) has
already 1 GB. When I compile a Debug variant it becomes over 3 GB (or
even more). So this approach will take much space. (And don't argue
with hard disc space is no matter. That's not a solution for me.)
Currently I have already 2 repos (one is the 2010.0 branch, the other
is the trunk). When I will need an additional repo to work on new
features I will need even more space. Do you have also a solution for
this problem?
Thomas
When branching of to a new repo in Hg you clone it. Hg uses hard links to do this. It will only keep the changes in the new cloned repo.
So comparing with your 2 svn branches you would have "hardly anything" in the new branch. "Hardly anything" means up to 50%.
This size difference works out better for large projects. In small projects the size reduction is relatively small due to the "administrative overhead". In large projects this is overhead is relatively smaller.
SVN has two complete copies which will diverse. Hg has one complete version and a strongly reduced copy (clone).
Harry
T. Modes
> When branching of to a new repo in Hg you clone it. Hg uses hard links to do
> this. It will only keep the changes in the new cloned repo.
> So comparing with your 2 svn branches you would have "hardly anything" in
> the new branch. "Hardly anything" means up to 50%.
> This size difference works out better for large projects. In small projects
> the size reduction is relatively small due to the "administrative overhead".
> In large projects this is overhead is relatively smaller.
>
> SVN has two complete copies which will diverse. Hg has one complete version
> and a strongly reduced copy (clone).
>
You mixed something: I was talking about the build directory - the
directory where the compiler and cmake creates the intermediate and
final files - and not about the repository directory.
Take the size of the following directories:
svn repository hugin trunk: 60 MB
build directory (release only): 1.02 GB (when compiling debug variant
it is 2.56 GB)
files to install: 110 MB
So you see, that the directory where you build the files is by far the
biggest one. All other are significant smaller. And I was only talking
of the build directory.
Furthermore I tested it again: hg clone creates on windows a "normal"
copy of the repository and no hard links on windows. Maybe this is
windows limitation. So also 2 cloned repositories requires (at least
with my current configuration - maybe I missed something) the doubled
space. This is nothing I worry. Which makes a big difference is the
doubled space for the build directory
Thomas
Harry van der Wolf
Hi Harry,
You mixed something: I was talking about the build directory - the
> When branching of to a new repo in Hg you clone it. Hg uses hard links to do
> this. It will only keep the changes in the new cloned repo.
> So comparing with your 2 svn branches you would have "hardly anything" in
> the new branch. "Hardly anything" means up to 50%.
> This size difference works out better for large projects. In small projects
> the size reduction is relatively small due to the "administrative overhead".
> In large projects this is overhead is relatively smaller.
>
> SVN has two complete copies which will diverse. Hg has one complete version
> and a strongly reduced copy (clone).
>
directory where the compiler and cmake creates the intermediate and
final files - and not about the repository directory.
Take the size of the following directories:
svn repository hugin trunk: 60 MB
build directory (release only): 1.02 GB (when compiling debug variant
it is 2.56 GB)
files to install: 110 MB
So you see, that the directory where you build the files is by far the
biggest one. All other are significant smaller. And I was only talking
of the build directory.
That's the same on OSX (maybe not the figures, but the configuration).
Furthermore I tested it again: hg clone creates on windows a "normal"
copy of the repository and no hard links on windows. Maybe this is
windows limitation. So also 2 cloned repositories requires (at least
with my current configuration - maybe I missed something) the doubled
space. This is nothing I worry. Which makes a big difference is the
doubled space for the build directory
Thomas
Understood !
Harry
Yuval Levy
let me first reassure you that I am not trying to proselytize you. I respect
your opinion; your criteria for evaluating the tools; and I don't want to move
away from Subversion if the hardship vs. benefits trade-off is not
overwhelmingly positive and if we have not tried, together, to solve all
annoyances, bugs, and other things.
For now, this is an exploration, and I appreciate that you are exploring the
potential of using Hg with us.
On March 19, 2010 01:55:36 pm T. Modes wrote:
> > It's easy to lock SVN making it read-only and allowing commits only from
> > an hgsvnpush cron job.
>
> But then the svn repository is only a copy of the mercurial. What is
> then the purpose of the svn repo?
distribution. There are plenty of "read-only" application. A user can download
a tarball, or fetch from SVN. This is particularly handy for source based
distributions such as Gentoo or FreeBSD, where it is technically possible to
fetch code from a repository rather than from a downloaded tarball. I also
don't know how packagers for distros such as Fedora, Debian, Ubuntu and the
dozens or hundreds of other do get their source code. Simply interrupting SVN
"read-only" service may interrupt the flow.
> This will make only problems. E.g. take the revision number or
> changeset. First you will need two part in the cmake system to extract
> the different numbers. And when the svn is read only, you have to test
> the svn code in a hg repo? Okay, this needs only done once. But its a
> additional complicitation.
There will indeed be changes. And one of the changes is not to think in SVN
terms. Revision number no longer identifies the version. We have it today in
SVN anyway, where trunk/@5071 is different than releases/2010.0/@5071
And there are many alternatives that can be discussed.
We can continue to rely on SVN for distribution. This means that the
developers use Hg to push around changesets. When they are ready, they push
them to SVN. This is conceptually not different from what we had now with the
development branches in SVN. There is just one added point when the code
transitions into SVN.
> But it can be more complicated: one user is using svn and reporting a
> bug with a revision number, the other user/developer is using hg and
> has a changeset (so you will need a translation between both numbering
> schemes.
It's even more complicated than this: I can pull changesets selectively. You
need to abandon the sequential thinking in terms of revisions that is at the
ground to SVN. The result is that we will need to manage a version number,
something that has been done conveniently for us by SVN. SVN has not only
disadvantages. Version numbers are needed on testing and distribution version.
Not necessarly on developers stuff. So one solution would be to keep SVN in
place, with a bridge from Hg. Development is in Hg (like it is currently in
branches/) and it is pushed from Hg to SVN with the same criteria as we have
today of merging an SVN branch into trunk. All building, testing, distribution
works from SVN. We take bug reports against SVN. Of course we can (and should)
not prevent power users to build from Hg, but with Hg based version people are
as much on their own as with self-brewed / modified versions of a tarball.
> So I would recommend only mercurial or only subversion, but not both
> at the same time.
only Hg for development, and only SVN for distribution (although we can not
prevent people distributing from Hg if they want to).
> > <http://mercurial.selenic.com/wiki/KeywordExtension> is part of the
> > default Hg distro.
>
> But why have I to activate it separately? But the point is, what
> happens with the keywords when converting between svn and hg?
The repository administrator has to configure keywords. It's the same for SVN
and for Hg. The difference is that with SVN there is a central repository and
thus one administrator for everybody, while with Hg and its decentralization
comes the responsibility of self-administration.
I believe Hg's keywords functionality was built to mirror SVN's. Making sure
that the details of keywords are respected is part of the nitty gritty detail
that a proper conversion entail. No conversion / transition to Hg without
properly working keyword. I think even the most fervent Hg supporter will
agree on that.
> Okay, maybe I have to change my workflow with Hg. But the approach
> with two or more repository has a major drawback (at least for me):
> You will need for each repository an own build directory.
Can't you have multiple repositories inside the same Hugin SDK? IIRC, when I
was building on Windows I had the SDK (which was a heavy beast) and then
checked different versions of the source code from SVN to different folders
inside the SDK.
> this approach will take much space.
how is it different from having multiple SVN branches build?
> And don't argue with hard disc space is no matter.
> That's not a solution for me.
neither for me. there are plenty of reasons not to accept HDD bloat and I am
all with you on this one.
> Currently I have already 2 repos (one is the 2010.0 branch, the other
> is the trunk). When I will need an additional repo to work on new
> features I will need even more space. Do you have also a solution for
> this problem?
First I have a question for you. 2010.0 branch? Bruno told me he needs a
report that it builds and works correctly on Windows to finalize release. Can
you please give him that feedback? thanks.
In terms of solution, let me understand this. We are talking 1-3 GB for a
single Hugin build. I recall something similar from my time in Windows. That
is the whole SDK, right? the dependecies, wxWidgets, libboost, etc.?
Inside that SDK folder for each branch of Hugin you are following there are
two folders, a source folder and a build folder. Is this correct?
With Hg the source folders should be clones, and theoretically they should be
cheap because of hard links. However hard links on Windows have some
requirements:
<http://mercurial.selenic.com/wiki/WindowsInstall#Hardlinks_on_Windows>
theoretically, the saving is the .hg folder at the root of the checkout.
While much of this issue is building related (I mean: also with SVN you need
the build folders?), I am all in favor to find ways of making it better - with
Hg, with SVN, or with whatever tools we will end up working with.
I want you on board, and I want you happy. I do not want to cause you HDD
bloat or otherwise waste your resources.
Thank you for working together on solving these issues.
Have a good weekend
Yuv
Pablo d'Angelo
I think that moving to mercurial would good, and now is the right time
to do so.
Yuval Levy schrieb:
> On March 19, 2010 01:55:36 pm T. Modes wrote:
>>> It's easy to lock SVN making it read-only and allowing commits only from
>>> an hgsvnpush cron job.
>> But then the svn repository is only a copy of the mercurial. What is
>> then the purpose of the svn repo?
>
> distribution. There are plenty of "read-only" application. A user can download
> a tarball, or fetch from SVN. This is particularly handy for source based
> distributions such as Gentoo or FreeBSD, where it is technically possible to
> fetch code from a repository rather than from a downloaded tarball. I also
> don't know how packagers for distros such as Fedora, Debian, Ubuntu and the
> dozens or hundreds of other do get their source code. Simply interrupting SVN
> "read-only" service may interrupt the flow.
I can see the value in keeping a subversion repository, but I dislike
keeping the SVN repo as the central repository for our branches. (Ie, no
developer will have/use an SVN checkout in any way).
If there is a real need for a SVN repo for distribution, I would much
more like a svn mirror of the "trunk" hg branch. However, as many
projects have moved from SVN to DVCS, I think that the major builder
(Gentoo, Freebsd) could handle a switch to hg relatively easily.
When I switched from CVS to SVN, I didn't keep a CVS mirror. It might
not be completely comparable to the current situation, but I don't
remember any complaints.
Another question is about the quality of service of the hg repos at sf
net. I haven't tried them yet, but in the past, I was not really
satisfied with the sf.net service.
Btw. when playing with conversions, it is best to download the entire
svn repository (was called svndump or similar 2 years ago, IIRC) from sf
and work with that.
It would be a good idea to try hg with panoglview source, this is not
critical and also relatively small.
ciao
Pablo
T. Modes
> distribution. There are plenty of "read-only" application. A user can download
> a tarball, or fetch from SVN. This is particularly handy for source based
> distributions such as Gentoo or FreeBSD, where it is technically possible to
> fetch code from a repository rather than from a downloaded tarball. I also
> don't know how packagers for distros such as Fedora, Debian, Ubuntu and the
> dozens or hundreds of other do get their source code. Simply interrupting SVN
> "read-only" service may interrupt the flow.
>
Thanks for explanation.
But I agree on this point with Pablos opinion. Either switch to
mercurial or stay at svn. But not a twitter approach.
End of last year enblend switch also from csv to mercurial and there
were also no backup or something similiar. So all builder made the
switch. This should not be the problem.
> > > It's easy to lock SVN making it read-only and allowing commits only from
> > > an hgsvnpush cron job.
> We can continue to rely on SVN for distribution. This means that the
> developers use Hg to push around changesets. When they are ready, they push
> them to SVN. This is conceptually not different from what we had now with the
> development branches in SVN. There is just one added point when the code
> transitions into SVN.
Now I'm confused. Are we discussing two different approaches?
> It's even more complicated than this: I can pull changesets selectively. You
> need to abandon the sequential thinking in terms of revisions that is at the
> ground to SVN. The result is that we will need to manage a version number,
> something that has been done conveniently for us by SVN. SVN has not only
> disadvantages. Version numbers are needed on testing and distribution version.
> Not necessarly on developers stuff. So one solution would be to keep SVN in
> place, with a bridge from Hg. Development is in Hg (like it is currently in
> branches/) and it is pushed from Hg to SVN with the same criteria as we have
> today of merging an SVN branch into trunk. All building, testing, distribution
> works from SVN. We take bug reports against SVN. Of course we can (and should)
> not prevent power users to build from Hg, but with Hg based version people are
> as much on their own as with self-brewed / modified versions of a tarball.
>
I don't think sequential. When I have a state from which I compile
hugin I'm at a defined point. Also users and beta testers are at a
defined point. This point needs a label. That can be a branch and rev.
number in subversion or a changeset in mercurial. It does not make a
difference on which way I'm come to this point (this is how the
systems differ)
Maybe when we switch to mercurial we could use the changeset as label/
identification. Or does this make problems because it is not an
increasing number?
> Can't you have multiple repositories inside the same Hugin SDK? IIRC, when I
> was building on Windows I had the SDK (which was a heavy beast) and then
> checked different versions of the source code from SVN to different folders
> inside the SDK.
>
That's the way I work it. I have only one SDK, but can have different
branches.
> > this approach will take much space.
>
> how is it different from having multiple SVN branches build?
>
This is not different, because this depends on the build system. But
now you are thinking in mercurial and not in subversion ;-)
In SVN I can work in only one repo and don't need several repos. So we
need to consider the whole workflow.
>
> First I have a question for you. 2010.0 branch? Bruno told me he needs a
> report that it builds and works correctly on Windows to finalize release. Can
> you please give him that feedback? thanks.
>
Done.
> In terms of solution, let me understand this. We are talking 1-3 GB for a
> single Hugin build. I recall something similar from my time in Windows. That
> is the whole SDK, right? the dependecies, wxWidgets, libboost, etc.?
>
No. This is the size of the build folder for hugin only. The SDK/
dependecies takes additional more than 1 GB.
> Inside that SDK folder for each branch of Hugin you are following there are
> two folders, a source folder and a build folder. Is this correct?
>
Yes.
> With Hg the source folders should be clones, and theoretically they should be
> cheap because of hard links. However hard links on Windows have some
> requirements:
>
> <http://mercurial.selenic.com/wiki/WindowsInstall#Hardlinks_on_Windows>
>
When I understand this site correctly, I have to compile mercurial for
myself. Than I will also need an other compiler, because Visual C++ is
not support by mercurial, but for hugin I use it. This is much effort.
It is also barrier for new user for hugin (at least when they are on
windows). To get work you need to compile first the tool to get the
source code.
But these hard links are not the main problem.
When we switch to mercurial, we should do it now, before gsoc starts.
For me some points are clearer now.
Thomas
Bruno Postle
>
> I also don't know how packagers for distros such as Fedora,
> Debian, Ubuntu and the dozens or hundreds of other do get their
> source code. Simply interrupting SVN "read-only" service may
> interrupt the flow.
They should be using the released tarballs, we don't need to worry
about supporting a distribution that doesn't.
--
Bruno
Yuval Levy
On March 20, 2010 02:41:16 am Pablo d'Angelo wrote:
> I think that moving to mercurial would good, and now is the right time
> to do so.
OK, let's try to fix what still needs fixing.
> I can see the value in keeping a subversion repository, but I dislike
> keeping the SVN repo as the central repository for our branches. (Ie, no
> developer will have/use an SVN checkout in any way).
I think we should keep it at least for historical / documentation purposes.
Can stay unmaintained, i.e. revoke write access to everybody and put a note
that the code has migrated to Hg.
> When I switched from CVS to SVN, I didn't keep a CVS mirror. It might
> not be completely comparable to the current situation, but I don't
> remember any complaints.
most of the CVS history seems to be available in SVN, while in the migration
from SVN to Hg so far I have three options:
a) hugin/trunk only, with all revisions from about 24.. (when Ippei's GSoC
2007 branch became officially trunk): 162 MB
b) hugin the whole story, that includes autopano-sift-c, panoglview,
kimagefuser, etc.: 5.2 GB
c) import trunk without history (top-skimming) and start from scratch: 1.1MB
I suspect that hgsvn does not deal well with branches. Probably digging a bit
further things could be improved, but what for? Given the choice I am inclined
to leave pre-Hg history in SVN (properly set to read-only) and start from a
clean import of trunk. What do others think?
> Another question is about the quality of service of the hg repos at sf
> net. I haven't tried them yet, but in the past, I was not really
> satisfied with the sf.net service.
yes, SF is constantly under extreme load. The good thing is that with Hg the
centralized repo becomes less and less important. It is easy, once we have
Hugin on Hg, to move it to another Hg service provider. Unless you already
know that you want to move to a specific service provider and that it does not
offer Hg. E.g. Launchpad AFAIK is bzr only.
I'm inclined to stay with SF for now, and do the migration. Better the devil
you know...
But if you want to review the choice of project hosting, then we need to do it
now, and to consider the version control they offer. GoogleCode offers Hg. Not
that I am a GoogleCode fan. BerliOS offers Hg, and I tend to prefer it over
GoogleCode. There are other choices as well, but again, unless there is
something that speaks against Hg, I would rather migrate to Hg on SF first.
> Btw. when playing with conversions, it is best to download the entire
> svn repository (was called svndump or similar 2 years ago, IIRC) from sf
> and work with that.
Yes. My mistake. I had it all set up on my old server with svnsync. Then I
decommissioned the old system drive (was my last PATA HDD and had become a
serious risk) and decided to restage FreeBSD from scratch. Had not restaged
SVN. Then I saw the timetable and realized that this is urgent because the
window of opportunity is *now*, before starting new GSoC branches.
May 23 is the beginning of GSoC2010 coding. We need to solve all outstanding
issues and implement the migration before that date. Do we have consensus on
this? Once we have collected all requirements / outstanding issues we can give
it a shot. I think that to succeed before May 23 it is best to stick with SF.
I think it is best to "top-skim" from trunk and keep history in SVN and in the
last SVN ChangeLog properly stored in Hg. The major issue to be solved is
Windows usability of the repo.
Anything else I have missed?
Yuv
Yuv
Yuval Levy
> But I agree on this point with Pablos opinion. Either switch to
> mercurial or stay at svn. But not a twitter approach.
OK, let's work on a full switch to Mercurial? what would be the criteria for
you to say: OK let's switch?
> When I have a state from which I compile
> hugin I'm at a defined point. Also users and beta testers are at a
> defined point. This point needs a label.
agree. that defined point is a tag in Hg.
> That can be a branch and rev.
> number in subversion or a changeset in mercurial. It does not make a
> difference on which way I'm come to this point (this is how the
> systems differ)
No. It can't be a changeset in Hg. The system don't differ only in the
sequence. In Hg, you can skip on a changeset.
Example:
my repo: 1,2,3,5,4,6
your repo: 1,3,2,4,5,6
both repos have the same. But now, let's say that changeset 4 (I am using
numbers instead of hashes to make the example more readable) adds nuts to the
recipe and Harry is allergic to nuts, so Harry's repos will look like this:
1,3,2,5,6
we can't use the changeset 6 as the label.
> Maybe when we switch to mercurial we could use the changeset as label/
> identification. Or does this make problems because it is not an
> increasing number?
it makes problem, also because it is not an increasing number. I would have
simply used SVN between Hg and the tarball, but it seems that consensus is to
get rid of SVN all together, so we need to come up with a modification of the
SVN-specific things that have made defining the point so easy in the CMake
build.
> > > this approach will take much space.
> >
> > how is it different from having multiple SVN branches build?
>
> This is not different, because this depends on the build system. But
> now you are thinking in mercurial and not in subversion ;-)
> In SVN I can work in only one repo and don't need several repos. So we
> need to consider the whole workflow.
Oh, yes, I see. In SVN you work with a single checkout, and you can make local
changes all over the place and keep it out-of-sync.
You probably can do the same with a single Hg repo while maintaining multiple
heads.
> No. This is the size of the build folder for hugin only. The SDK/
> dependecies takes additional more than 1 GB.
If you maintain multiple heads in a single Hg repo, you can probably use the
same build folder against it. But I have not tried it.
> > <http://mercurial.selenic.com/wiki/WindowsInstall#Hardlinks_on_Windows>
>
> When I understand this site correctly, I have to compile mercurial for
> myself.
I am not sure. Those pages tend to become easily outdated. You'd have to try
with the latest downloads (1.5.) from http://mercurial.selenic.com/downloads/
> This is much effort.
> It is also barrier for new user for hugin
I agree that if Hg has not reached a mature enough level on Windows for these
things to be available as a simple download, we should wait with the migration
until it reaches that stage. Should not take long. The features are already
there and too many projects that also support Windows are switching to Hg.
> When we switch to mercurial, we should do it now, before gsoc starts.
yes, that would be nice. but I don't want to rush, and I don't want to leave
important contributors behind.
So the question to you is: what criteria needs to be fulfilled for Hg to be an
acceptable replacement for SVN from the perspective of a Windows developer?
Yuv
Yili Zhao
> Another question is about the quality of service of the hg repos at sf
> net. I haven't tried them yet, but in the past, I was not really
> satisfied with the sf.net service.
The most popular distributed version control systems (DVCS) are
Git, Bazaar and Mercurial.
If the quality of service of the hg repos at sf.net is not good,
maybe we can consider to migrate to other open source hosting sites.
Pablo has used Ubuntu's LaunchPad for the panomatic development,
and LaunchPad uses Bazaar as the version control system.
Another candicate is Google's code site (http://code.google.com), and it
use Subversion and Mercurial as the version control system.
If we want to use git, then we can choose Gitorious (http://gitorious.org)
or GitHub (http://github.com)
--
Yili Zhao
Simon Oosthoek
> most of the CVS history seems to be available in SVN, while in the migration
> from SVN to Hg so far I have three options:
> a) hugin/trunk only, with all revisions from about 24.. (when Ippei's GSoC
> 2007 branch became officially trunk): 162 MB
> b) hugin the whole story, that includes autopano-sift-c, panoglview,
> kimagefuser, etc.: 5.2 GB
> c) import trunk without history (top-skimming) and start from scratch: 1.1MB
>
> I suspect that hgsvn does not deal well with branches. Probably digging a bit
> further things could be improved, but what for? Given the choice I am inclined
> to leave pre-Hg history in SVN (properly set to read-only) and start from a
> clean import of trunk. What do others think?
>
>
On the one hand I think that the project is moving mostly at the tips
(or HEADs in git-speak) so top-skimming may not be that bad. However
from a historical perspective it might be good to have everything in one
repository. Is a mercurial repository of "the whole story" really
5.2GiB? Or is that the entire storage for subversion's repo?
If top-skimming is chosen, at least all the branches that are "under
maintenance" should be pulled from svn.
/Simon
Simon Oosthoek
repository provider is most suitable. An important part of a move would
involve moving all the issues from the tracker to a new service
provider, this is at least as important as the source code, I think.
And the DVCS should be acceptable to use on all platforms that hugin
developers use.
/Simon
T. Modes
> agree. that defined point is a tag in Hg.
>
> > That can be a branch and rev.
> > number in subversion or a changeset in mercurial. It does not make a
> > difference on which way I'm come to this point (this is how the
> > systems differ)
>
> No. It can't be a changeset in Hg. The system don't differ only in the
> sequence. In Hg, you can skip on a changeset.
>
> Example:
>
> my repo: 1,2,3,5,4,6
> your repo: 1,3,2,4,5,6
>
> both repos have the same. But now, let's say that changeset 4 (I am using
> numbers instead of hashes to make the example more readable) adds nuts to the
> recipe and Harry is allergic to nuts, so Harry's repos will look like this:
>
> 1,3,2,5,6
>
> we can't use the changeset 6 as the label.
>
I read again the mentioned tutorial. So I understand it:
Each changeset has a decimal number and a hexadecimal number: the
decimal number can change in every copy of the repository, but the
hexadecimal number is unique (in each repository the same hex-number
stands for the same position in the repository). So this hex number
can be used to identify a state (Enblend is using this approach).
Each changeset has also one parent (when using the repository explorer
from TortoiseHG it shows the parents directly). (except when merging,
the merging changeset have 2 parents). So take the following example
(I make it simpler):
in my repo: 1-2-3 (2 has parent 1, 3 has parent 2),
When you want to skip changeset 2, you have to apply changeset 3 to
changeset 1. But changeset 3 can only be applied to parent 2. So you
have to make a branch or a head to apply changeset 3 to changeset 1.
In this case you will create a changeset 4 and in your repository is
now 1-4 and your state with skipped changeset 2 gets a new hex-number.
> > Maybe when we switch to mercurial we could use the changeset as label/
> > identification. Or does this make problems because it is not an
> > increasing number?
>
> it makes problem, also because it is not an increasing number. I would have
> simply used SVN between Hg and the tarball, but it seems that consensus is to
> get rid of SVN all together, so we need to come up with a modification of the
> SVN-specific things that have made defining the point so easy in the CMake
> build.
>
Maybe we could use the file rev.txt, which exists already. We would
need a procedure to update this file: Is it possible to implemented a
script in the hg repos, which can do this with each commit? Or can
this done by a chron job: e.g. update this rev number once a day/once
a week or so?
> If you maintain multiple heads in a single Hg repo, you can probably use the
> same build folder against it. But I have not tried it.
>
> > > <http://mercurial.selenic.com/wiki/WindowsInstall#Hardlinks_on_Windows>
>
> > When I understand this site correctly, I have to compile mercurial for
> > myself.
>
> I am not sure. Those pages tend to become easily outdated. You'd have to try
> with the latest downloads (1.5.) fromhttp://mercurial.selenic.com/downloads/
>
I tested with latest release. And there is the hard link not working
out of the box. But this is nothing I matter.
With the switch to a dynamic build, I have also reduced the size of
the build directory and install directory (in comparision to static
build it's less than the half size; e.g. the install dir is decreased
from 102 to 44 MB)
> So the question to you is: what criteria needs to be fulfilled for Hg to be an
> acceptable replacement for SVN from the perspective of a Windows developer?
I think, that the current TortoiseHG (vers. 1.0/Mercurial 1.5) is an
acceptable replacement for TortoiseSVN from my perspective. The main
points are working. And I think it will evolve with the next releases.
> most of the CVS history seems to be available in SVN, while in the migration
> from SVN to Hg so far I have three options:
> a) hugin/trunk only, with all revisions from about 24.. (when Ippei's GSoC
> 2007 branch became officially trunk): 162 MB
> b) hugin the whole story, that includes autopano-sift-c, panoglview,
> kimagefuser, etc.: 5.2 GB
> c) import trunk without history (top-skimming) and start from scratch: 1.1MB
>
> I suspect that hgsvn does not deal well with branches. Probably digging a bit
> further things could be improved, but what for? Given the choice I am inclined
> to leave pre-Hg history in SVN (properly set to read-only) and start from a
> clean import of trunk. What do others think?
I think variant b is a no-go. That's too heavy. When all developer and
builder have to download above 5 GB (at least to get a initial clone,
I know that the next pulls are significant smaller) this will create a
lot of traffic. On the other side the take too much space on my hard
disc.
Variant a has also a relative big size - compare the current checked
out svn branch is about 60 MB - you have given only the size of
the .hg folder, right? This size will add to it.
So at this stage I would prefer variant c. But it would be nice the
have the full history locally (variant a).
But maybe variant a is so big because hgsvn does the convertion
ineffectivly. In mercurial there is also the ConvertExtension (
http://mercurial.selenic.com/wiki/ConvertExtension ). Maybe this
extension can handle the import better; but I have not tried yet.
Thomas
Simon Oosthoek
branches contain a copy of the entire development tree at the moment of
branching and further changes on that. DVCSs on the other hand, have a
much more efficient way to branch (making them essentially "free"). So
after importing the entire history of hugin into mercurial or git, I
suspect the whole trunk+branches with history will likely be much
smaller in the DVCS and therefore manageable as a whole.
I found some instructions for git and svn repositories
http://www.contextualdevelopment.com/articles/2008/sva-to-git (follow
the links there)
it links to a script: which can probably be adapted to hugin's svn
repository:
http://www.contextualdevelopment.com/svn/oss/sc/trunk/script/sva2git.sh
there's also a description of this process (again for git, that's what I
know best) in the progit.org book:
http://progit.org/book/ch8-2.html
This should probably be approximately the same using mercurial or bzr
(but I'm not familiar with those tools, so I'm guessing)
/Simon
Bruno Postle
>
> May 23 is the beginning of GSoC2010 coding. We need to solve all
> outstanding issues and implement the migration before that date.
> Do we have consensus on this? Once we have collected all
> requirements / outstanding issues we can give it a shot. I think
> that to succeed before May 23 it is best to stick with SF. I
> think it is best to "top-skim" from trunk and keep history in SVN
> and in the last SVN ChangeLog properly stored in Hg.
If the entire trunk commit history really is such a big download
then it is ok archiving the existing ChangeLog file and starting a
clean mercurial tree - The SVN history doesn't contain anything that
happened in the various branches anyway (due to the unfortunate way
it handles merging) so we don't have a full commit history whatever
we do.
--
Bruno
Yuval Levy
> I think the choice of DVCS should be dependant on which open source
> repository provider is most suitable.
I think it should be the other way around. The choice of DVCS is more critical
than the choice of code hosting provider; and it is also more critical than
the choice of bug tracker and other tools which can also be hosted at
different providers. Hosting is an interchangeable commodity.
That said, the devil lies in the detail and I don't think we have the capacity
at this time to acquaint ourself in detail with other code hosting than
SourceForge. Hence my proposal is to implement Hg on SF. Of course everybody
is free to implement another DVCS at any other hosting provider any time, to
publish their results here, and to see that everybody else learn to use that
alternate hosting provider.
> An important part of a move would
> involve moving all the issues from the tracker to a new service
> provider, this is at least as important as the source code, I think.
This is out of the scope of the current migration project. It's massive: you'd
have to migrate 195 bug reports; 167 feature request reports; 13 patch
reports. And these are only the open ones. Of course there are plenty of
outdated things in there that need a good spring clean up. Massive effort.
Clean up is a must before even thinking of it. Then you'd need to analyze the
alternatives. Compare the existing bug tracker with the alternative. Show the
added value with respect to the massive effort required.
The value added of a DVCS over a traditional VCS is so obvious that it does
not really matter whether we choose Hg, Git, Bzr - all of them will unleash a
potential that is worth the effort. The benefits outweigh the effort.
I don't want to discourage you, and you're welcome to try and report back. I
don't particularly like Hugin's bug tracker, but it does its job and frankly
if we are to migrate to a new DVCS before May 23 I don't see any move of bug
tracker or move of hosting provider happen at the same time.
Once we're on a DVCS code hosting migration is easier than now - in fact,
everybody who pulls the Hg repo is a potential host, and you can push
changesets privately as you'd be emailing patches.
> And the DVCS should be acceptable to use on all platforms that hugin
> developers use.
Yes, which is one of the main reason to choose Hg. Windows support for Hg is
still not perfect, but last time I checked it was more mature than Bzr or Git.
> If top-skimming is chosen, at least all the branches that are "under
> maintenance" should be pulled from svn.
Yes. We're currently at a point where there are no branches "under
maintenance".
For older releases (including the current 2010.0.0), the tarballs are good
enough. In the unlikely event that a serious security issue requires a patch,
it can be done in SVN itself, or even just in the tarball. Let history rest in
peace. No need to pull them.
In the development branches the only piece of useful code that has not yet
been integrated to trunk may be gsoc2008_masking. It has complementary
functionality (brushes) to Thomas Modes' masking tool. I've pinged Fahim about
integration.
Thanks also for all the links about repository conversion. The more I think of
it, the more I feel that it is better to leave the past enshrined in SVN
(read-only) and move on to a better future. I see no reasonable use case to
make the effort and add the whole luggage of historic commits to a Hg
repository. But maybe somebody has a case to justify the effort?
Yuv
Yuval Levy
On March 29, 2010 12:40:11 pm T. Modes wrote:
> I read again the mentioned tutorial. So I understand it:
> Each changeset has a decimal number and a hexadecimal number: the
> decimal number can change in every copy of the repository, but the
> hexadecimal number is unique (in each repository the same hex-number
> stands for the same position in the repository). So this hex number
> can be used to identify a state (Enblend is using this approach).
I understand it slightly different. The hex-number is a hash of the changeset
that has nothing to do with what came before and what came after. It is unique
and consistent across repositories, while...
> Each changeset has also one parent (when using the repository explorer
> from TortoiseHG it shows the parents directly).
My understanding is that the revision number is not consistent across
repositories. I could miss or intentionally not apply one changeset.
> But changeset 3 can only be applied to parent 2.
I don't think so. But I may be wrong. My understanding is that a changeset is
like a patch. To get where Hugin is now, you'd have 5000+ patches aligned in a
sequence. Like a patch, sometimes it is applicable and sometimes not. When it
is not, that's when Hg throw up an exception and manual merging is required.
> Maybe we could use the file rev.txt, which exists already. We would
> need a procedure to update this file: Is it possible to implemented a
> script in the hg repos, which can do this with each commit? Or can
> this done by a chron job: e.g. update this rev number once a day/once
> a week or so?
yes, we should use the existing rev.txt, although currently it is generated by
CMake at build time. I am afraid that a hook on commit won't work. Because it
is commit *to which repo*? everybody would have to modify their own local hgrc
file. We're in a distributed environment! If all builds were done from the
same repo it would work but it would defeat the purpose of the distributed
repos.
I think that we'll have as of now, manual version number (major/minor/patch).
For everything that is built from a repo (as opposed than built from a
tarball) there will be some sort of reference that identifies which repo was
used, and either a date or the revision number associated with that repo, or
both.
This is one point where SVN still has an advantage over DVCS IMHO. A certain
level of control/centralization is needed to identify uniquely a build. Builds
for one own's consumption are not critical, but builds for public distribution
are. This is why I would have liked to implement an Hg->SVN bridge and have
the distributed binaries still built from the central SVN repo. With a little
bit of self-discipline amongst builder we can do it also from the distributed
Hg repos, but it will be more error prone.
> I think, that the current TortoiseHG (vers. 1.0/Mercurial 1.5) is an
> acceptable replacement for TortoiseSVN from my perspective. The main
> points are working. And I think it will evolve with the next releases.
Good, thanks. So do I understand correctly that you're OK with moving to Hg in
the coming weeks?
> So at this stage I would prefer variant c. But it would be nice the
> have the full history locally (variant a).
What is the use-case for variant a (which btw is not full history, it is only
the last 2500 commits, or so)? I can have a look and try to improve the
migration process if the effort is justified.
> But maybe variant a is so big because hgsvn does the convertion
> ineffectivly. In mercurial there is also the ConvertExtension (
> http://mercurial.selenic.com/wiki/ConvertExtension ). Maybe this
> extension can handle the import better; but I have not tried yet.
I suspect that hgsvn's inefficiency is part of the cause of that bloat. I have
not had time to investigate in detail. This is done in spare-time and I don't
have much time to spare these days :(
So I pragmatically suggest to start on a clean slate. The process would be:
1. lock SVN write access to everybody but whoever does the migration.
2. update the SVN generated ChangeLog one last time.
3. Rename it ChangeLog.SVN_legacy and commit it.
4. pull a fresh trunk from SVN.
5. commit it into a fresh Hg repo.
6. make sure keyword substitution and email notifications work in that Hg
repo.
7. push the fresh Hg repo to SF and set it free.
Am I missing something?
Yuv
Yuval Levy
> If the entire trunk commit history really is such a big download
> then it is ok archiving the existing ChangeLog file and starting a
> clean mercurial tree - The SVN history doesn't contain anything that
> happened in the various branches anyway (due to the unfortunate way
> it handles merging) so we don't have a full commit history whatever
> we do.
I will have a look again to see if I missed some inefficiency in the svn->hg
process that generated such a big download. What happened in the SVN branches
is enshrined in the SVN repo, so we shall not delete it, just make it read-
only.
Yuv
Pablo d'Angelo
Yuval Levy schrieb:
> Hi Pablo,
>
> On March 20, 2010 02:41:16 am Pablo d'Angelo wrote:
>> I think that moving to mercurial would good, and now is the right time
>> to do so.
>
> OK, let's try to fix what still needs fixing.
>
>
>> I can see the value in keeping a subversion repository, but I dislike
>> keeping the SVN repo as the central repository for our branches. (Ie, no
>> developer will have/use an SVN checkout in any way).
>
> I think we should keep it at least for historical / documentation purposes.
> Can stay unmaintained, i.e. revoke write access to everybody and put a note
> that the code has migrated to Hg.
Yes, I would leave the svn that way (and maybe commit some lines to the
CMakeLists.txt file in the svn repo that prints a warning message and
breaks compilation unless it is removed).
>> When I switched from CVS to SVN, I didn't keep a CVS mirror. It might
>> not be completely comparable to the current situation, but I don't
>> remember any complaints.
>
> most of the CVS history seems to be available in SVN, while in the migration
> from SVN to Hg so far I have three options:
> a) hugin/trunk only, with all revisions from about 24.. (when Ippei's GSoC
> 2007 branch became officially trunk): 162 MB
> b) hugin the whole story, that includes autopano-sift-c, panoglview,
> kimagefuser, etc.: 5.2 GB
> c) import trunk without history (top-skimming) and start from scratch: 1.1MB
I tried the following procedure:
1) rsync full svn repo at file level
rsync -Lav hugin.svn.sourceforge.net:svn/hugin hugin-svnrepo
2) convert full repo to hg -> 997 MB
$ hg convert hugin-svnrepo/hugin hugin-hgrepo
I used no special options (there seems to be a config file for branches
etc. Its usage might improve things, as our repo is not structured as
the usual svn repo, and we made some strange use of svn)
3) use hg convert to extract only the trunk from the full hg repo -> 51 MB
$ cat > hgtrimfilemap
include hugin/trunk
rename hugin/trunk .
$ hg convert --filemap hgtrimfilemap hugin-hgrepo hugin-hgrepo-trunk-only
So it is 51 MB vs 1.1 MB.
> I suspect that hgsvn does not deal well with branches. Probably digging a bit
> further things could be improved, but what for? Given the choice I am inclined
> to leave pre-Hg history in SVN (properly set to read-only) and start from a
> clean import of trunk. What do others think?
While there are probably usecases for having the full history, it might
be overkill. If possible, I'd like to keep at least the recent history,
though, that might be a compromise.
I just did some hunting why the full history is so much bigger, and the
main culprit seems to be the videos and pdf files that Ippei added to
the SVN repo during his GSOC 2007.
Also my svn mv to swap trunk and ippeis branch was not a good idea in
hindsight.
It would be great if these could be removed either from the SVN repo
(local copy), or we could take just the revision after the point where
the video has been removed from the trunk again.
> yes, SF is constantly under extreme load. The good thing is that with Hg the
> centralized repo becomes less and less important. It is easy, once we have
> Hugin on Hg, to move it to another Hg service provider. Unless you already
> know that you want to move to a specific service provider and that it does not
> offer Hg. E.g. Launchpad AFAIK is bzr only.
>
> I'm inclined to stay with SF for now, and do the migration. Better the devil
> you know...
Ack.
Some other points with the repo conversion:
- renaming of the authors (there is a name map feature in hg convert. I
wonder if we should move from the sf id to real names (and maybe email,
for people that agree with that?)
- Should we use named branches in hg (for the release branches etc), or
not? If we choose to do so, it might be nice to convert them (if its not
too hard).
ciao
Pablo
Robert
> extension can handle the import better; but I have not tried yet.
The 'preferred' way to handle semantically separate entities in
Mercurial is through different repositories, anyway. So one would make
one repo for Hugin, one for autopano-sift-c etc.; putting the whole
tree into one repository is not how Mercurial is designed, so it would
be very inefficient. Or I am misunderstanding something about the 'b'
option. This also means that one can switch over to Mercurial on a per-
project basis.
I do not think the 'a' option is that big of a problem, considering
that SVN is not very bandwidth-friendly anyway. So doing a 60MB SVN
checkout will result in quite a bit more traffic that just the 60MB,
and Mercurial's 160MB is actually rather small considering this
includes the entire tree history (a SVN repo containing only Hugin
would still be _much_ bigger than just 160MB, quite possibly several
hundred).
The checkout size will be locally added to that size, that's correct,
at least that's how I would interpret the numbers as well. So an
initial checkout would transfer 160MB and take 220MB on the disk
afterwards. However, if you e.g. branch on your harddisk (i.e., clone
the repo to a second directory), this will usually just result in a
lot of filesystem links (on systems that support it, i.e. at least
Unix and Windows) instead of a 'real' copy, so branching locally is a
very cheap operation.
Local disk space is usually no big problem nowadays (considering
current computers are routinely sold with disks in the TB range), so I
would not see 160MB overhead as a show stopper per se. However, it can
still be a viable option to leave the SVN repo as it is for reference
purposes and just start a bunch of new repos for the Hugin and PT
applications. Making a clean cut also allows for corrections in the
project structure, should there be reason for them.
Regards,
Robert
T. Modes
> I understand it slightly different. The hex-number is a hash of the changeset
> that has nothing to do with what came before and what came after. It is unique
> and consistent across repositories, while...
I think the hex number can be used to identify a changeset. It is
unique above all repositories.
The repository itself keeps track which changeset depends on which.
> I don't think so. But I may be wrong. My understanding is that a changeset is
> like a patch. To get where Hugin is now, you'd have 5000+ patches aligned in a
> sequence. Like a patch, sometimes it is applicable and sometimes not. When it
> is not, that's when Hg throw up an exception and manual merging is required.
My understanding is that you will always need a sequence or order of
patches. E.g. when in changeset x you introduce some code, and in
changeset x+1 you fixes a bug into the new code. So changeset x+1 has
always be applied after changeset x. When a changeset is in the
repository it has to apply smoothly, otherwise an other user can not
apply the patch/changeset. This will only work, if you have an order/
sequence.
You can take a look on the enblend repository - look at changeset
7ccee7cc30ef and 8675bd6f1f0b till 37c52e12b88c and use the graph
drawing.
> Good, thanks. So do I understand correctly that you're OK with moving to Hg in
> the coming weeks?
You are right, I'm ok with moving to Hg.
Thomas
Simon Oosthoek
Yuval Levy wrote:
> I think it should be the other way around. The choice of DVCS is more critical
> than the choice of code hosting provider; and it is also more critical than
> the choice of bug tracker and other tools which can also be hosted at
> different providers. Hosting is an interchangeable commodity.
>
> That said, the devil lies in the detail and I don't think we have the capacity
> at this time to acquaint ourself in detail with other code hosting than
> SourceForge. Hence my proposal is to implement Hg on SF. Of course everybody
> is free to implement another DVCS at any other hosting provider any time, to
> publish their results here, and to see that everybody else learn to use that
> alternate hosting provider.
>
I suppose you have a point, at least do one thing at a time, also below
I have some comments which may indicate that once the conversion is done
to a DVCS, switching to another system can be much easier if that turns
out to be necessary.
Also, regarding the tracker, that may cause a bit of lock-in to sf.net
for now ;-)
>> And the DVCS should be acceptable to use on all platforms that hugin
>> developers use.
>>
>
> Yes, which is one of the main reason to choose Hg. Windows support for Hg is
> still not perfect, but last time I checked it was more mature than Bzr or Git.
>
I think this may not be much of an issue if these DVCSes can easily
interoperate, see below.
>
> Thanks also for all the links about repository conversion. The more I think of
> it, the more I feel that it is better to leave the past enshrined in SVN
> (read-only) and move on to a better future. I see no reasonable use case to
> make the effort and add the whole luggage of historic commits to a Hg
> repository. But maybe somebody has a case to justify the effort?
>
Since Pablo had some success converting subversion history to mercurial,
I agree with him that it would be worthwhile to retain at least some
(recent) history. But perhaps more is not too hard and with some
specific dropping of parts of the history not even very large. I'm
keeping my eye open for similar conversions, of which there are many!
KDE is one huge project which has had quite a bit of experience with
conversion from subversion to git, so there may be some way to harness
that experience...
http://blog.hartwork.org/?p=763
http://gitorious.org/svn2git/svn2git
Also conversion/cooperation between mercurial and git is easy to do and
lossless, so using tools made for git (I think this may be more mature
than for mercurial, but I'm not in a position to judge that) can be used
as intermediary step towards a mercurial repository and if people would
prefer git for their personal repository, that's possible even if others
use mercurial.
http://mercurial.selenic.com/wiki/HgGit
If I can find some time for this, I will see what I can come up with,
but it may take a while for me to find time ;-)
/Simon
PS, sorry if you get this twice, I didn't see this arrive on the list,
so I'm sending again...
Simon Oosthoek
Yuval Levy wrote:
> I think it should be the other way around. The choice of DVCS is more critical
> than the choice of code hosting provider; and it is also more critical than
> the choice of bug tracker and other tools which can also be hosted at
> different providers. Hosting is an interchangeable commodity.
>
> That said, the devil lies in the detail and I don't think we have the capacity
> at this time to acquaint ourself in detail with other code hosting than
> SourceForge. Hence my proposal is to implement Hg on SF. Of course everybody
> is free to implement another DVCS at any other hosting provider any time, to
> publish their results here, and to see that everybody else learn to use that
> alternate hosting provider.
>
I suppose you have a point, at least do one thing at a time, also below
I have some comments which may indicate that once the conversion is done
to a DVCS, switching to another system can be much easier if that turns
out to be necessary.
Also, regarding the tracker, that may cause a bit of lock-in to sf.net
for now ;-)
>> And the DVCS should be acceptable to use on all platforms that hugin
>> developers use.
>>
>
> Yes, which is one of the main reason to choose Hg. Windows support for Hg is
> still not perfect, but last time I checked it was more mature than Bzr or Git.
>
I think this may not be much of an issue if these DVCSes can easily
interoperate, see below.
>
> Thanks also for all the links about repository conversion. The more I think of
> it, the more I feel that it is better to leave the past enshrined in SVN
> (read-only) and move on to a better future. I see no reasonable use case to
> make the effort and add the whole luggage of historic commits to a Hg
> repository. But maybe somebody has a case to justify the effort?
>
Yuval Levy
On March 30, 2010 02:36:00 pm T. Modes wrote:
> Hi Yuv,
>
> > I understand it slightly different. The hex-number is a hash of the
> > changeset that has nothing to do with what came before and what came
> > after. It is unique and consistent across repositories, while...
>
> I think the hex number can be used to identify a changeset. It is
> unique above all repositories.
> The repository itself keeps track which changeset depends on which.
I think you are right. we'll have time to fix this as we go along since this
is relevant to the rev.txt file that currently holds the revision number used
by CMake to set the version number.
> You are right, I'm ok with moving to Hg.
Thank you! my plan is to have a working Hg repo with keyword substitution and
something temporary (or maybe even something good for the longer term) for the
revision number in the version. I set myself early May (will aim for May 10)
as a deadline. If anybody gets there first, don't hesitate to publish results
here.
The repo will be Hugin only. As Robert stated, "the 'preferred' way to handle
semantically separate entities is through different repositories", and this is
what will happen.
I'll give it another try to get at least some of the past history into the
Hugin repo, now that my local subversion server is up and running again. I
have a local SVN mirror of Hugin. Below are the instructions of how I set it
up (it should not be as bandwidth consuming as my previous experiment) set up
a local SVN mirror of Hugin. You will need configured Apache and Subversion
servers on your box (I use a FreeBSD box).
On the server, do:
1. Create the new repo (your path to the repos might vary)
cd /var/repos
svnadmin create hugin
2. make sure the commit hooks are configured to allow svnsync user do anything
it wants. this is a shortcut for a safe, protected environment. there is a
more robust script described in [0]
cd hugin
echo '#\!/bin/sh' > hooks/pre-revprop-change
chmod 755 hooks/pre-revprop-change
On a client that is connected to both the internet and your svn server, do
(replace the IP address with your server's IP address):
export TOREPO=http://192.168.1.1/hugin/
export FROMREPO=https://hugin.svn.sourceforge.net/svnroot/hugin/hugin/
svnsync init ${TOREPO} ${FROMREPO}
svnsync --non-interactive sync ${TOREPO}
If the sync is aborted, you will need to reset the lock state and restart:
svn propdelete svn:sync-lock --revprop -r 0 ${TOREPO}
svnsync --non-interactive sync ${TOREPO}
to keep it updated, set a cron job
this is an abridged version of [1]
with the local repo it is easier / faster to play the Hg migration by trial
and error.
my next step is to "optimize" the repository, i.e. remove permanently the
movie and PDF files that have been checked in by mistake, using svnadmin dump
| svndumpfilter | svnadmin load as described in [2]
then convert the svn repo with all history to hg. aiming at getting the whole
history into hg while keeping the repo size reasonable. else I'll start an hg
repo from scratch.
The SVN repo is 227MB and I suspect around 150MB are the movie and pdf. Will
keep you updated.
Thanks to everybody who has contributed to the repository thread.
Yuv
[0] http://svn.apache.org/repos/asf/subversion/trunk/notes/svnsync.txt
[1] http://journal.paul.querna.org/articles/2006/09/14/using-svnsync/
[2] http://subversion.apache.org/faq.html#removal
Michael Witten
> Hi Yuv,
>
>> I understand it slightly different. The hex-number
>> is a hash of the changeset that has nothing to do
>> with what came before and what came after. It is
>> unique and consistent across repositories, while...
>
> I think the hex number can be used to identify a
> changeset. It is unique above all repositories.
> The repository itself keeps track which changeset
> depends on which.
Git keeps track of content, and content is named by a hash of that
content (of course, you can tag with human-friendly names and you
don't have to type out the full hash either in most cases).
With regard to the discussion quoted above, git keeps track of
*commits*, the 'contents' of which include what commits come before as
parents.
Hence, if anything about the commit is changed (lineage, contents of
files, whatever), then so does the commit hash that is used as the
commit's name.
Here's what's nice about this: If and only if one repository has a
commit object with the same hash as a commit object in another
repository, it is (almost) certain that those 2 commits provide the
same working-tree content and were drived from the exact same history
all the way back to the root commit. That's awesome.
Sincerely,
Michael Witten