Re: [v8-dev] Re: [chromium-dev] Getting gclient to checkout a modifiable subrepo (e.g., v8)

[Paweł Hajdan, Jr.]

+infra-dev,machenbach

On Fri, Dec 16, 2016 at 8:18 PM, Jeremy Roman <jbr...@chromium.org> wrote:

My workflow has been to have two repositories (a standalone one and a Chromium one), but with the standalone one added as a remote of the chromium one.

Mine are /src/chromium/src and /src/v8/v8

$ cd /src/chromium/src/v8

$ git remote add standalone /src/v8/v8

Then I can fetch revisions from the standalone repo, check them out, push/pull, etc., and do the final upload from the standalone supported repo.

Works well enough for me.

On Fri, Dec 16, 2016 at 12:36 PM, Jakob Kummerow <jkum...@chromium.org> wrote:

Working on V8 from within a Chromium checkout's src/v8/ directory is not a supported workflow. There are various ways to hack it, but The Official Way is to get a separate V8 checkout with "fetch v8" (which should be fairly quick, so the only cost is a bit of disk space).

For testing local changes, I have a secondary workdir of my standalone V8 checkout in my Chromium checkout (which I created with git-new-workdir, apparently "git worktree" is how you'd do it now), which I rename as needed (because symlinking used to confuse ninja, not sure if it still does):

# setup

$ cd /work/v8/

$ git worktree add /work/chromium/src/v8-custom

# implement new feature

$ git checkout -b my-feature

$ ...

$ git commit -am 'yay new feature'

# test feature

$ cd /work/chromium/src

$ git pull && gclient sync # let "gclient sync" see the official "v8" directory

$ mv v8 v8-official

$ mv v8-custom v8

$ (cd v8; git checkout -f my-feature)

$ ninja ...

# cleanup when done

$ mv v8 v8-custom

$ mv v8-official v8

# land it

$ cd /work/v8/

$ git cl upload

You can iterate on the patch with:

$ cd /work/v8/

$ edit...

$ git commit -a --amend

$ cd /work/chromium/src

$ (cd v8; git checkout -f my-feature)

$ ninja ...

On Fri, Dec 16, 2016 at 1:10 AM, Daniel Cheng <dch...@chromium.org> wrote:

​I can't say I recommend this, but for limited development, I just upload with --bypass-hooks and let the v8_presubmit bot make sure I didn't do anything bad.

​It's probably still worth pulling down an actual v8 checkout; I found running the v8 tests to be a bit painful without an actual v8 checkout.​

Daniel

On Thu, Dec 15, 2016 at 8:56 PM Trent Apted <tap...@chromium.org> wrote:

I encountered something similar with WebRTC. It has a helpful message when you try to `git cl upload` (basically: you *must* create a separate checkout):

Creating CLs from this location is not supported! Please make sure the current

working directory is the parent directory of this directory.

If you're working with a Chromium checkout, you'll have to create a full WebRTC

checkout and upload a CL from that. See

https://webrtc.org/native-code/development/ for instructions.

V8 has instructions here - https://github.com/v8/v8/wiki/Using%20Git - that seem up-to-date.  (tl;dr: `fetch v8` should work)

You might be able to save some electrons with something like

 $ mkdir v8 # at the level above your chromium/.gclient

 $ gclient-new-workdir.py chromium/src/v8 v8/v8

 $ cd v8/v8

 $ git checkout master

 $ cd ..

 $ cat -> .gclient

solutions = [

  {

    "url": "https://chromium.googlesource.com/v8/v8.git",

    "managed": False,

    "name": "v8",

    "deps_file": "DEPS",

    "custom_deps": {},

  },

]

^D

That will symlink all the git objects for v8 so they're not re-downloaded, and share disk blocks. But you'll still get dupes of v8's DEPS.

Getting much more creative probably risks wedging your chromium checkout into a weird state :/

On 16 December 2016 at 15:18, Matt Giuca <mgi...@chromium.org> wrote:

Hi,

TL;DR: How do you work on V8 (build, test and upload CLs) from within the v8 subrepo of a Chromium checkout (without having to make a separate v8 checkout)?

I was helping Nick get started on V8 development within Chromium and we couldn't find any way to run the presubmit from within the v8 directory that's checked out inside the chromium repo.

That is, we would like to be able to do:

~/chrome/src$ cd v8/src

~/chrome/src/v8/src$ git cl presubmit

However, if you do that, you get:

Traceback (most recent call last):

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/git_cl.py", line 5654, in <module>

    sys.exit(main(sys.argv[1:]))

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/git_cl.py", line 5636, in main

    return dispatcher.execute(OptionParser(), argv)

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/subcommand.py", line 252, in execute

    return command(parser, args[1:])

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/git_cl.py", line 4222, in CMDpresubmit

    change=cl.GetChange(base_branch, None))

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/git_cl.py", line 1685, in RunHook

    gerrit_obj=self._codereview_impl.GetGerritObjForPresubmit())

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/presubmit_support.py", line 1247, in DoPresubmitChecks

    results += executer.ExecPresubmitScript(presubmit_script, filename)

  File "/usr/local/google/home/mgiuca/chrome/depot_tools/presubmit_support.py", line 1157, in ExecPresubmitScript

    result = eval(function_name + '(*__args)', context)

  File "<string>", line 1, in <module>

  File "<string>", line 287, in CheckChangeOnCommit

  File "<string>", line 260, in _CommonChecks

  File "<string>", line 101, in _CheckUnwantedDependencies

ImportError: No module named checkdeps

Turns out this is because the v8 directory is missing the buildtools subdirectory which is checked out by its DEPS. Basically, when gclient checks out all the subrepos of chrome (including v8) it doesn't recursively check out the DEPS' DEPS. So we can't submit v8 from there and maybe some other things are broken too.

If you make a separate checkout of v8 (outside of chrome), and run gclient sync, it's fine. But then you can't test your changes in Chrome, resulting in tedious copying of git patch data back and forth between the two checkouts of v8. It would be much better if you could just do the full v8 edit/build/test/upload workflow from within the v8 checkout inside chrome.

I don't want gclient to be recursive by default. I just want to be able to set it up so it fetches v8.

We found this old (looks obsolete) documentation describing exactly how to do this:

http://www.chromium.org/developers/how-tos/chromium-modularization#Advanced_Usage

• It gives the example of v8 but then says "The V8 example is somewhat simple since V8 does not itself have other dependencies." That's no longer true since V8 *does* have other dependencies. This example doesn't get what we want.
• Then it goes on to say how to do webkit (no longer applicable) and get webkit's dependencies. But we tried setting up .gclient with v8 similar to how webkit is set up there, and it didn't work. I won't go into the details of how it didn't work (can discuss on request).

How do you do this? I'd imagine most v8 engineers have a workflow that solves this problem. There must be a way to have gclient check out the DEPS of v8 without needing to make a separate v8 checkout.

Thanks

Matt & Nick

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Matt Giuca]

Thanks for the detailed replies.

I have to say neither of these workflows seem particularly nice for extended development. I prefer Jeremy's (that's the one I set up for Nick) because it doesn't involve moving directories around. (Also use of git-worktree seems to just be a disk saving exercise, not actually helping simplify the workflow.) Neither fetch time nor disk usage are the issue; the issue is developer time when doing an edit/build/test/upload loop.

The problem is both Jakob and Jeremy's workflow involves a "final upload" -- what about iterative development? I like to upload early and often, and the upload is the hard part of this workflow since you need to:

1. Make sure your main repo does *not* have the branch checked out (since that will block push).

2. From the chrome repo: git push.

3. From the main repo: git checkout <branch>.

4. From the main repo: git cl upload.

Not to mention how hard it is to manage multiple stacked branches in this workflow (for example, for git push to work from the chrome repo, the branch upstream needs to be the remote branch, so you can't have a local branch hierarchy).

I really can't understand why doing all your work from inside a sub-repo of the chrome repo isn't a supported workflow. Literally the only thing missing is gclient being able to check out DEPS of the v8 repo. That's the only difference between the v8 inside chrome and the "main" standalone v8 repo, and it would surely make development much easier?

+infra-dev,machenbach

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Aaron Gable]

That's not actually it -- gclient is able to check out DEPS of the v8 subrepo. The problem is that, when embedded in src.git, v8 doesn't want to check out its DEPS, because many of them are shared with the rest of chrome, so it uses the versions supplied there. If chromium were restructured so that v8 was supposed to supply all of its own dependencies, then this would be much simpler. The src.git/DEPS file would simply specify "recursedeps = ['v8']" and all of v8's deps would be checked out.

If you want to change the way this works, talk to the OWNERS of src.git/DEPS :)

You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAHqYdcbG_NbSusovoNBX4S1gT3C_akWkF-RMFAJU%3D8QPw-CkgA%40mail.gmail.com.

[Jakob Kummerow]

The main point of using git-new-workdir is actually easy synchronization between the two workdirs, which also addresses the "upload often" concern. I have my IDE pointed at my main V8 checkout and do all my edits there. From a shell that's cd'ed to that same checkout I can upload as early and as often as I want. In order to test the current state of things, I "git commit -a --amend" in the main repo, switch to the shell that's cd'ed to the Chromium checkout, and "git checkout -f <branch>" there to pull in the recently-committed changes, then compile and test. So the overhead compared  to a standard one-repo-one-workspace scenario is minimal, just the "git commit"/"git checkout" pair of commands.

It just so happened that I got a chance to try out the builtin git-worktree after writing my previous mail. Turns out it has one difference to git-new-workdir in that it doesn't allow the same branch to be checked out in two workdirs (which makes sense for safety reasons). The workaround is simple: have a branch for Chromium's V8 checkout, and instead of "git checkout -f <branch>" do "git reset --hard <branch>". To clarify: commits in the main checkout go to branch B1, the secondary checkout is at branch B2, and B2 keeps getting hard-reset to B1.

It's probably possible to skip the directory moving and just keep the custom V8 directory in the Chromium checkout, but I haven't tried this. "gclient sync" should check out a given revision, so all it takes to get back to testing your own stuff is "git checkout B2" (+"git reset --hard B1" as needed).

+infra-dev,machenbach

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

[Primiano Tucci]

As long as your chromium's DEPS and the version of the full v8 checkout are consistent I think you can mount --bind ~/v8/src ~/chrome/v8/src

I never tried this in v8 but was using for a similar case (back in the days when I was building webview from the android tree but uploading cls from the chromium tree).

To be clear this will break for sure the ability to gclient sync from chromium (because the mounted v8/src will have no .git folder, as in a standalone v8 checkout src is just a folder of the top-level v8 project), so you'll have to remember to unmount and remount before gclient syncing.

However should give you the ability to use git cl from the v8 folder and all the standalone v8 checkout, but still build and test your changes from both the chrome and v8 tree without having to do commit, push and pull sequences.

All this unless the DEPS of standalone v8 changes the layout of folders within v8/src, in which case all this might not work. In other words, just try, you might be just lucky if this hack works :)

+infra-dev,machenbach

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAHqYdcbG_NbSusovoNBX4S1gT3C_akWkF-RMFAJU%3D8QPw-CkgA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.

To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAKSzg3TQB%3D9jvCJ%3D7BPxmHm2Lx-0DKu5PzQCqDB9QP8JuWmi6A%40mail.gmail.com.

[Nicholas Verne]

I appreciate the effort going in to describing multi-repro workflows, however they still sound alarmingly complex and error-prone to me. 

The bug that triggered this thread was in fact the failure of the v8 presubmit to find its buildtools/ dir. This could be remedied with a searchpath for buildtools: try local, then go up, right? Then we could support both independent v8 development and chromium/v8 development.

Nick Verne

+infra-dev,machenbach

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAHqYdcbG_NbSusovoNBX4S1gT3C_akWkF-RMFAJU%3D8QPw-CkgA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.

To post to this group, send email to infr...@chromium.org.

[Matt Giuca]

Jakob's approach sounds a bit better on explanation. Thanks for clarifying.

The advantage here (over manually pushing and pulling) is that the two workdirs share the same commit store, so if you commit from workdir 1 the commit object will automatically be available from workdir 2.

> The workaround is simple: have a branch for Chromium's V8 checkout, and instead of "git checkout -f <branch>" do "git reset --hard <branch>".

That sounds a bit scary (incorporating git reset --hard into your workflow) but reasonable. You're suggesting having two separate branches that are essentially kept in sync to the same point: one for standalone and one for Chromium. Another approach would be to always be in a detached head state on the Chromium side. Whenever you want to update the Chromium side, do "git checkout --detach <branch>".

[Matt Giuca]

(Re-sending from my chromium.org account.)

Jakob's approach sounds a bit better on explanation. Thanks for clarifying.

The advantage here (over manually pushing and pulling) is that the two workdirs share the same commit store, so if you commit from workdir 1 the commit object will automatically be available from workdir 2.

> The workaround is simple: have a branch for Chromium's V8 checkout, and instead of "git checkout -f <branch>" do "git reset --hard <branch>".

That sounds a bit scary (incorporating git reset --hard into your workflow) but reasonable. You're suggesting having two separate branches that are essentially kept in sync to the same point: one for standalone and one for Chromium. Another approach would be to always be in a detached head state on the Chromium side. Whenever you want to update the Chromium side, do "git checkout --detach <branch>".

On Tue, 20 Dec 2016 at 10:02 Nicholas Verne <nve...@chromium.org> wrote: