SPIM Viewer in Fiji

[Johannes Schindelin]

Hi Tobias,

please find the new 'spimviewer' branch here:

https://github.com/fiji/fiji/compare/spimviewer

It integrates your SPIM Viewer into Fiji, in preparation for the upload to
the Fiji update site (the merge commit contains detailed documentation how
I managed to pull in a completely separate history without merge
conflicts). Could you please test?

For everybody else: in case you do not know about Tobias' cool project: it
is a viewer for large, multi-angle data. What's really impressive about it
is its performance, it is super-fast. You can zoom through the stack(s),
rotating the slice that is shown, switching on and off the different
angle's stacks, etc.

Ciao,
Dscho

[Johannes Schindelin]

Hi Tobias (and other beginning Git mavens),

On Tue, 14 May 2013, Johannes Schindelin wrote:

> please find the new 'spimviewer' branch here:
>
> https://github.com/fiji/fiji/compare/spimviewer
>
> It integrates your SPIM Viewer into Fiji, in preparation for the upload to
> the Fiji update site (the merge commit contains detailed documentation how
> I managed to pull in a completely separate history without merge
> conflicts).

So maybe that commit message was a bit terse? ;-)

The idea is to help Git by not performing a "real" merge:

git merge -s ours {commit}

makes a merge commit, yes, but it actually does not merge any code but
leaves the worktree completely unchanged. This is of course only the
start; we will make a few changes and the *amend* that merge.

The most important step is now:

git read-tree --prefix={prefix} HEAD^2

So first let's keep in mind that read-tree reads the information of a
given commit (or even tree or tag) and puts it into the staging area (as
if you had called "git add"), and the --prefix option makes sure that 1)
it does not replace the contents of the staging area, and 2) all the files
from the referenced object are put into a subdirectory in the staging
area.

The object given is HEAD^2 which is the second parent of the merge commit
that is HEAD.

Since read-tree only works on the staging area and does not touch the
working directory, I call

git stash -k

The -k means: keep everything that is in the staging area, but throw away
the changes in the working tree relative to the staging area. Now, we
never had those changes in the working tree, but git stash does not know.
So essentially, it brings the changes we performed on the staging area
also to the files in the working directory.

The last step is to call

git commit --amend -s

where --amend means that Git should rewrite the latest commit to include
the changes in the staging area (you want to avoid that if you already
pushed said latest commit).

The -s option asks Git to add a Signed-off-by: line which is kind of a
signed statement that yes, even if I am doing this during my day job (and
hence my employer has all rights to my work), I am entitled to release it
as Open Source.

Summary: I used "git merge" to start the merge, "git read-tree" to
introduce the files of the to-be-merged commit into a subdirectory, "git
stash" to reflect that in the working directory, and "git commit" to amend
the merge commit.

Ciao,
Dscho

[Johannes Schindelin]

Hi Tobias,

On Tue, 14 May 2013, Johannes Schindelin wrote:

> On Tue, 14 May 2013, Johannes Schindelin wrote:
>
> > please find the new 'spimviewer' branch here:
> >
> > https://github.com/fiji/fiji/compare/spimviewer
> >
> > It integrates your SPIM Viewer into Fiji, in preparation for the upload to
> > the Fiji update site (the merge commit contains detailed documentation how
> > I managed to pull in a completely separate history without merge
> > conflicts).
>
> So maybe that commit message was a bit terse? ;-)
>
> The idea is to help Git by not performing a "real" merge:
>

> [... detailed explanation ...]

Addressing the obvious follow-up question: Subsequent merges do *not* need
to jump through those hoops.

Another, slightly less obvious follow-up question: it builds here, even
with MiniMaven, but only due to me having built our project with Maven
before, so that the jhdf5 library got installed into the local repository
in $HOME/.m2/. To remedy that, I deployed it to ImageJ's Maven repository.

Ciao,
Johannes

[Tobias Pietzsch]

Hi Johannes,

Thank you very much for your work and explanations!

Unfortunately I'm still running into problems. I just tried to

git merge tobias-spimviewer/master

(after setting up the tobias-spimviewer remote)
That works only partially:

- Some existing files are updated by the merge in the src-plugins/spim_viewer as expected.

- For some files, git gets confused. For instance it tries to merge fiji's pom.xml with the pom.xml of the spimviewer, apparently not recognizing
that the modified pom.xml in tobias-spimviewer/master is the src-plugins/spim_viewer/pom.xml

- For deleted files in tobias-spimviewer/master I get a conflict:
CONFLICT (rename/delete): src-plugins/spim_viewer/jhdf5/cisd/jhdf5/maven-metadata-local.xml deleted in tobias-spimviewer/master and renamed in HEAD. Version HEAD of src-plugins/spim_viewer/jhdf5/cisd/jhdf5/maven-metadata-local.xml left in tree.

Can you suggest a way to solve that?

best regards,
Tobias

> --
> --
> Please avoid top-posting, and please make sure to reply-to-all!
>
> Mailing list web interface: http://groups.google.com/group/fiji-devel
>
> ---
> You received this message because you are subscribed to the Google Groups "Fiji-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Johannes Schindelin]

Hi Tobias,

On Fri, 17 May 2013, Tobias Pietzsch wrote:

> Unfortunately I'm still running into problems. I just tried to
>
> git merge tobias-spimviewer/master
>
> (after setting up the tobias-spimviewer remote)
> That works only partially:
>
> - Some existing files are updated by the merge in the
> src-plugins/spim_viewer as expected.
>
> - For some files, git gets confused. For instance it tries to merge
> fiji's pom.xml with the pom.xml of the spimviewer, apparently not
> recognizing that the modified pom.xml in tobias-spimviewer/master is the
> src-plugins/spim_viewer/pom.xml

Right, the rename detection fails for that case because both branches have
a file "pom.xml" in the same place.

> - For deleted files in tobias-spimviewer/master I get a conflict:
> CONFLICT (rename/delete): src-plugins/spim_viewer/jhdf5/cisd/jhdf5/maven-metadata-local.xml deleted in tobias-spimviewer/master and renamed in HEAD. Version HEAD of src-plugins/spim_viewer/jhdf5/cisd/jhdf5/maven-metadata-local.xml left in tree.

That is also a problem with Git's rename detection. There has been a
proposed patch to be able to fix Git's stupidity by manually overriding
it, but it went nowhere:

http://git.661346.n2.nabble.com/WIP-PATCH-Manual-rename-correction-td7564180.html

> Can you suggest a way to solve that?

I can. The best way would be to write a helper working around Git's
limitations (from the thread I linked above, it should become quite
obvious that Git will not be fixed).

Ciao,
Dscho

[Curtis Rueden]

Hi Dscho & Tobias,

> The idea is to help Git by not performing a "real" merge

This seems very complicated. Spimviewer is being merged into fiji.git, right? If so, why does Tobias need to merge more stuff again after Dscho's initial merge? Rather, why not just perform future branches, commits and merges from fiji.git directly? Is the issue that existing topic branches need to be merged, too?

Furthermore, I personally wonder why spimviewer must live in fiji.git at all. In my discussions with Dscho, my understanding is that the rule of thumb is: if it's distributed via the primary Fiji update site, it must live in fiji.git, unless it is an explicit dependency of something else that lives in fiji.git (e.g., TrakEM2, Bio-Formats).

Since nothing in fiji.git depends on spimviewer, wouldn't it be easier for spimviewer to simply stay in its own Git repository, and have its own update site (e.g., sites.imagej.net/Pietzsch)? Or alternately, if it is really preferred for it to live in fiji.git now, isn't that a one-time merge? Why all the continuing git-fu here?

Regards,

Curtis

[Johannes Schindelin]

Hi Curtis,

On Fri, 17 May 2013, Curtis Rueden wrote:

> > The idea is to help Git by not performing a "real" merge

I now have a script that does that:

https://github.com/fiji/fiji/blob/spimviewer/bin/renaming-merge.sh

> This seems very complicated.

It is, but should not be. Git's rename detection is just not quite where
it should be.

> Spimviewer is being merged into fiji.git, right? If so, why does Tobias
> need to merge more stuff again after Dscho's initial merge? Rather, why
> not just perform future branches, commits and merges from fiji.git
> directly? Is the issue that existing topic branches need to be merged,
> too?

Tobias wanted to continue developing in his little repository, outside of
Fiji. But he also wanted it to be on the public Fiji update site, which
requires the source to live in fiji.git (or at least referenced as an Open
Source dependency in fiji.git), as you point out here:

> Furthermore, I personally wonder why spimviewer must live in fiji.git at
> all. In my discussions with Dscho, my understanding is that the rule of
> thumb is: if it's distributed via the primary Fiji update site, it must
> live in fiji.git, unless it is an explicit dependency of something else
> that lives in fiji.git (e.g., TrakEM2, Bio-Formats).
>
> Since nothing in fiji.git depends on spimviewer, wouldn't it be easier for
> spimviewer to simply stay in its own Git repository, and have its own
> update site (e.g., sites.imagej.net/Pietzsch)? Or alternately, if it is
> really preferred for it to live in fiji.git now, isn't that a one-time
> merge? Why all the continuing git-fu here?

There is now a personal update site for Tobias (the URL you mentioned is
correct), however, there have been unspecified problems (all mentioned
privately).

I guess that one of the problems is that I try too hard to accomodate all
kinds of weird development setups.

As Tobias is away for two weeks (also mentioned privately), I guess we
should leave it at that for now. Speaking for myself, there has been quite
a bunch of tasks I left just to get the above-linked script to work,
before I realized that it does not matter for at least fourteen days. So
maybe I should take care of quite important challenges in ImageJ2 again...

Ciao,
Dscho

[Tobias Pietzsch]

Hi Johannes,

I'm not fully in vacation mode yet, so forgive me for answering prematurely... :-)

What you say is certainly true. You are very very convenient to have around when it comes to
questions on the more obscure git magic, let alone dealing with the Fiji updater. No use denying,
that it is often too tempting and easy to take advantage of that. I hope that I can repay some of
your time and effort, for example by work in imglib2 that is helpful for you guys.

Again my apologies for being pushy and thank you so much for making this work!

best regards,
Tobias

On May 17, 2013, at 10:43 PM, Johannes Schindelin <Johannes....@gmx.de> wrote:

> Hi Tobias,
>

> On Fri, 17 May 2013, Johannes Schindelin wrote:
>
>> On Fri, 17 May 2013, Curtis Rueden wrote:
>>
>>>> The idea is to help Git by not performing a "real" merge
>>
>> I now have a script that does that:
>>
>> https://github.com/fiji/fiji/blob/spimviewer/bin/renaming-merge.sh
>

> Hopefully you had a nice and relaxing vacation (and/or workshop in
> Athens).
>
> I'd like to point out something that you might have missed in the whole
> conversation. This is not meant as criticism, but it is something that you
> might want to be aware of.
>
> I do understand that it is convenient to just ask me about Git things and
> Fiji things. I do understand that it is convenient to do so in a Skype
> session. I do understand that it is convenient to ask me all these things,
> maybe even every single time any advanced Git question comes up because I
> am prone to know a lot about Git.
>
> However, it does leave the feeling on my side of being treated as a
> commodity.
>
> I do understand that it is easy to be dismissive of my ideas about design,
> despite relying on my competence when it comes to other things such as
> update sites. It might even be a little of a pleasure to show me my
> boundaries when it comes to a project I am not in charge of.
>
> However, it does leave a foul feeling on my side of being helpful when it
> saves other people time, but my needs not being met.
>
> I do understand that it is a comfortable position to know that there is
> always somebody who wants to keep things working and who goes out of his
> way to fix bugs and issues, someone to ask for help and being certain that
> the other guy just works until the job gets done because that's what he
> always does.
>
> However, it does leave me with a lot of my time spent on your problems. My
> problems, of course, wait patiently for me, and me alone (or would you
> help me?), to tackle them later. Or even later. And nobody sees why those
> things get delayed, because all of the requests were private. In the best
> case, I look like doing *something* else. In the worst case, I look lazy.
>
> Now, I hope to be able to be useful to the project(s) in the future as
> much as I seemed to be in the past. I also hope that I will be able to
> continue to help in a timely manner. I hope that my attempts to inspire
> knowledge will bear fruit, despite my failed attempts to do so (and
> consequentially either requiring me to take care of similar problems all
> the time, or in the worse case, having to deal with the outcomes of
> unfortunate decisions I seemed to be unable to influence with arguments).
> And finally, I hope that in the future, I might get my stuff done in
> addition to all that, and maybe every once in a while receive a little
> help for my problems in return.
>
> Again, this is not meant as criticism. But it might make it easier to
> understand my reactions in the future, and maybe even excuse them.
>
> Ciao,
> Dscho