Need more people working on codesearch [was: What's people's workflow with gitiles?]

[Jeffrey Yasskin]

[re-subject'ing since this isn't about how to use gitiles.]

Really, this should be +1 for adding headcount in chrome-infra, or for
someone to volunteer to fill that headcount if they've already got it.
The person who used to be in charge of codesearch moved on, and that
job got split among 3 already-overworked people. They already know
there's a pile of improvements to make; they just need extra folks to
do the work.

Any volunteers?

Jeffrey

On Fri, Feb 5, 2016 at 10:51 AM, Tommy Nyquist <nyq...@chromium.org> wrote:
> Yeah, +1 to both having it in the main UI and keeping scroll position.
>
>
> On Fri, Feb 5, 2016 at 10:44 AM Scott Graham <sco...@chromium.org> wrote:
>>
>> Thanks, that's really useful. I wonder if we could get that into the UI
>> with a bit of smartness to help maintain file location in larger files.
>>
>> On Fri, Feb 5, 2016 at 10:39 AM, Tommy Nyquist <nyq...@chromium.org>
>> wrote:
>>>
>>> You can get to "Blame Parent Commit" from the blame view in gitiles, by
>>> adding either ~ or ^ suffix to the revision in the URL.
>>>
>>> For example from this SHA-1:
>>>
>>> .../chromium/src/+blame/060e0077890e52868120c4b56a91dad859de2038/base/android/jni_android.h
>>> --> append ~ after the SHA-1:
>>>
>>> ../chromium/src/+blame/060e0077890e52868120c4b56a91dad859de2038~/base/android/jni_android.h
>>> --> goes to this:
>>>
>>> ../chromium/src/+blame/1bb9c6ca7f71926b7b3cc09da957b0a457242fa1/base/android/jni_android.h
>>>
>>> It even works for the branch name if you happen to want to ignore the top
>>> commit:
>>> ../chromium/src/+blame/master/base/android/jni_android.h
>>> --> add ~ after 'master':
>>> ../chromium/src/+blame/master~/base/android/jni_android.h
>>> --> goes to this:
>>>
>>> ../chromium/src/+blame/429446698446a103171801d827c5dab5f98baa1c/base/android/jni_android.h
>>>
>>> On Fri, Feb 5, 2016 at 10:03 AM Scott Graham <sco...@chromium.org>
>>> wrote:
>>>>
>>>> +1 one to all that.
>>>>
>>>> When the toplevel CL in gittiles (via cs "View in") isn't what I'm
>>>> looking for, I go out to `git gui blame` to do Right click -> "Blame Parent
>>>> Commit" without pulling my hair out.
>>>>
>>>>
>>>> On Fri, Feb 5, 2016 at 9:34 AM, Elliott Sprehn <esp...@chromium.org>
>>>> wrote:
>>>>>
>>>>> I don't use a Chrome search engine (but that's clever... If only it
>>>>> worked on mobile), I go to cs.chromium.org and search for what I'm looking
>>>>> for, once I find it there's a dropdown "View In" that has Git, Blame and Log
>>>>> links.
>>>>>
>>>>> I often don't know what part of the code I need to see the history of
>>>>> to start, code search lets me find all callers, implementors, and context.
>>>>>
>>>>> Fwiw internal code search supports log and blame right inside the cs UI
>>>>> as layers, I wish our instance did too. Having to go out into gitiles all
>>>>> the time is unfortunate since you can't search or cross reference from over
>>>>> there.
>>>>>
>>>>> On Feb 5, 2016 9:22 AM, "Jeffrey Yasskin" <jyas...@chromium.org>
>>>>> wrote:
>>>>>>
>>>>>> I basically never use it, but several senior folks on the project use
>>>>>> it exclusively, and I'd like to figure out how you do it.
>>>>>>
>>>>>> My best guess is that you have a Chrome search engine set up so you
>>>>>> can type "crgit some/path" in your omnibox and it'll expand to
>>>>>> https://chromium.googlesource.com/chromium/src/+/master/some/path?
>>>>>>
>>>>>> When going from codesearch to gitiles, do you just copy&paste the
>>>>>> path?
>>>>>>
>>>>>> Are there any other good tips for people who haven't used gitiles
>>>>>> before?
>>>>>>
>>>>>> Thanks,
>>>>>> Jeffrey
>>>>>
>>>>> --
>>>>> --
>>>>> Chromium Developers mailing list: chromi...@chromium.org
>>>>> View archives, change email options, or unsubscribe:
>>>>> http://groups.google.com/a/chromium.org/group/chromium-dev
>>>>
>>>>
>>>> --
>>>> --
>>>> Chromium Developers mailing list: chromi...@chromium.org
>>>> View archives, change email options, or unsubscribe:
>>>> http://groups.google.com/a/chromium.org/group/chromium-dev
>>
>>
>

[Timo Reimann]

Are you looking for internal project members only to work on codesearch in a rather dedicated fashion; or is this something where external contributors might be able to help out as well?

Something else I've been curious about: Is Chromium simply using vanilla https://github.com/google/codesearch or possibly some customized version?

Thanks

Timo

[Thiago Farina]

On Saturday, February 6, 2016, Timo Reimann <ttr...@googlemail.com> wrote:

Are you looking for internal project members only to work on codesearch in a rather dedicated fashion; or is this something where external contributors might be able to help out as well?

Something else I've been curious about: Is Chromium simply using vanilla https://github.com/google/codesearch or possibly some customized version?

Is it? This seems to be command lines tools rather the web UI. Isn't cs.chromium.org a web interface for Grok?

I don't think the code for cs.chromium.org is open yet (though kythe seems to be very similar) but I could be very wrong.

--

Thiago Farina

[Timo Reimann]

I only had a closer look at the Github project now, and it looks like you're right: It seems to be a trimmed down, simplified version of a Google service that was discontinued in 2012.

Sorry for the confusion. You seem to be right about the closed-source nature of CS. Unfortunately, I cannot be of any help in this case.

[Aaron Gable]

At the moment, chromium codesearch is using a closed-source solution (to which Kythe is very similar). So unfortunately external contributions aren't possible at the moment, though I certainly would love for them to be.

More headcount isn't necessarily the solution here. Prioritizing is. We have a lot of projects going on (including monorail) which are about trying to directly improve the chromium developer experience. If people believe that improving codesearch will do more to improve developers' lives than reducing cq time or putting recipes in project repos, then please make that clear to us. We base our priorities on what we think Chromium developers want, but we're honestly always a bit out-of-touch. Tell us what you need most, and we'll try to listen (and also try to not only listen to the people who are willing to speak up the most :)).

Thanks,

Aaron

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/7b9eed27-5242-45ac-bd1d-ff03dc89de70%40chromium.org.

[Peter Kasting]

On Mon, Feb 8, 2016 at 9:16 AM, Aaron Gable <aga...@chromium.org> wrote:

If people believe that improving codesearch will do more to improve developers' lives than reducing cq time or putting recipes in project repos, then please make that clear to us.

Ignorance: I've seen "recipes" mentioned multiple times over the years but I don't know what they are.  My random guess: is this basically "if I git cl try a change in this repo it will pick a good set of bots"?

As to the other items you mention, CQ time improvements are always welcome, but CQ flakiness (which is thankfully far rarer now) is probably more annoying than CQ delays.  Improving codesearch is, to me, more useful than CQ improvements right at the moment, but that's primarily because I've seen the CQ get much better over the last few years, while codesearch and (more importantly) the code browser (gitiles) has mostly stayed the same except when we went from svn to git, which marked a definite degradation in tool quality (but not nearly as big of one as it could have been, thanks to some heroic work on the gitiles front to fix a variety of issues).  It's still significantly harder and slower for me to trace the blame of a particular line deeply back through version history than it was with the svn tools, so I'd love to see both speed and UX improvements here.

PK

[Aaron Gable]

Don't read too much into my choice of "recipes" and "cq latency" as straw men; even I don't have perfect visibility into the priorities of other infra subteams. I just picked examples which were likely to have meaning to my audience.

Recipes are the control system for the steps which get run on both trybots and the continuous integration waterfall. It used to be that the buildbot master configuration declared a static list of steps which must be executed over the course of the build. Now, that job is delegated to recipes, which run directly on the slave. This gives us much more flexibility (recipes can make decisions about what step to run next based on previous results, which is a prerequisite for bisecting) and more reproducibility (you can run recipes locally on your own machine) among other advantages. Any time you see the "steps" step, or almost any time you see "@@@" annotations in the logs, that's recipes.

CQ flakiness is an aspect of CQ latency: if your CL fails CQ multiple times when it should have passed, that means the CQ took too long on your CL.

Anyway, as I said, don't read too much into those particular examples. My point was just that throwing money and people at the problem is usually not the only solution, let alone the best solution. Instead, let's work on making sure the channels of communication between chromium and infra are wiiiiiide open, and accurately reflect the needs and abilities of people on both sides.

[Peter Kasting]

On Mon, Feb 8, 2016 at 4:22 PM, Aaron Gable <aga...@chromium.org> wrote:

Recipes are the control system for the steps which get run on both trybots and the continuous integration waterfall. It used to be that the buildbot master configuration declared a static list of steps which must be executed over the course of the build. Now, that job is delegated to recipes, which run directly on the slave. This gives us much more flexibility (recipes can make decisions about what step to run next based on previous results, which is a prerequisite for bisecting) and more reproducibility (you can run recipes locally on your own machine) among other advantages. Any time you see the "steps" step, or almost any time you see "@@@" annotations in the logs, that's recipes.

Ah.  To me, personally, this has zero obvious utility (it may have significant non-obvious utility, I'm ignorant, but I've never run a recipe locally), but I am reluctant to speak to its global value since I'm sure it matters to other people doing more directly-related work.

Anyway, as I said, don't read too much into those particular examples. My point was just that throwing money and people at the problem is usually not the only solution, let alone the best solution. Instead, let's work on making sure the channels of communication between chromium and infra are wiiiiiide open, and accurately reflect the needs and abilities of people on both sides.

The thing I like to say is "don't tell me what should be a higher priority, since people always want to make everything a higher priority -- tell me what I'm currently doing that I _shouldn't_ be spending time on."

I don't know what to tell you guys not to spend time on :(

PK 

[Aaron Gable]

That's... actually a really good philosophy. Yeah, we get asked to prioritize a lot of things :)

Aaron

P.S. For those following along from home, recipes provides (or will soon provide, in the cases noted below) the following direct benefits to chromium developers:

* A testing framework to make sure you're executing the steps you think you are

* The ability to run locally exactly what ran on the bot

* The ability to run much more complex builds, such as bisections, and things that don't seem like normal "builds", like gnumbd.

* The ability to run the exact same things on the tryserver and the main waterfall (we still often don't do this, for various efficiency reasons, but we could)

* (in limited availability) The ability to run jobs directly on swarming, bypassing buildbot entirely

* (coming soon) The ability to specify jobs and steps inside chromium/src, rather than having to file CLs against infra

You can also read more about recipes. 

[Julie Parent]

+chrishall, dsansome, jem

[ Replying to this thread too, since I just saw it, and my reply makes much more sense in this context. Sorry for the double post. ]

I've heard rumors that perhaps someone from our SRE team might want to take on CodeSearch as a way of helping contribute directly to developer productivity.  cc'ed a few of those folks here for context on the sort of requests we are hearing from chromium devs.

I'd also add, that if anyone from the larger Chrome team would like to contribute to infra, we welcome you with welcome arms.  agable@ might not have time to talk through all this on a mailing list, but if you were volunteering to make some of these changes yourself, I rather suspect he'd sing a different tune :)

Re: priorities - We try to listen to the grumbling around us to manage priorities, but please don't ever hesitate to reach out and share your thoughts on our priorities if we are missing the mark.  We might not agree with you, or might have a more complex set of priorities we are juggling than you realize, but we always welcome the feedback, and factor it into our decisions on where to focus our limited efforts.  (We are also hiring, by the way!)

--

You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To post to this group, send email to infr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/CAH58R2e7LWzkStnVU4OhtBG24qTpM6FtFA1U-U7-ryQuZQoFsA%40mail.gmail.com.

[Thiago Farina]

On Tuesday, February 9, 2016, Julie Parent <jpa...@chromium.org> wrote:

Re: priorities - We try to listen to the grumbling around us to manage priorities, but please don't ever hesitate to reach out and share your thoughts on our priorities if we are missing the mark.  We might not agree with you, or might have a more complex set of priorities we are juggling than you realize, but we always welcome the feedback, and factor it into our decisions on where to focus our limited efforts.  (We are also hiring, by the way!)

Is these prioritites visible somewhere or is this just talking about the Pri-1, Pri-2 bugs filed in crbug.com with Infra label? Or is it even something totaly different and just internal to the Infra team?

--

Thiago Farina

[Julie Parent]

Labels in crbug are a decent reflection of our priorities, although the various infra subteams use priorities differently.  Monorail team for example has been using Milestones at the higher level, and priorities within that.  I'm not sure exactly how other subteams handle tracking priorities.

The other more general sense of "priorities" is just where you see us spending effort as a larger team - a project with zero people working on it full time is lower priority than a project with say 3 people full time on it, and one with 1-2 people working on a side project is somewhere in the middle.