RCS and system testing

7 views
Skip to first unread message

Shai Barack

unread,
Aug 4, 2021, 3:21:55 PM8/4/21
to component-framework-dev, Gary Bressler, Fady Samuel, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson
Hey folks,

This question came up in the review of the out-of-tree (OOT) system testing roadmap doc:

For more context I'm assuming familiarity with the RCS protocol:
sdk/fidl/fuchsia.developer.remotecontrol/remote-control.fidl

The problem at hand is with exposing component monikers to OOT developers, as a detail that's not part of the Fuchsia System Interface (FSI) and may hinder our updatability.

To elaborate -
One of the challenges that we've met when changing platform implementation details, particularly when migrating components from CFv1 to CFv2 (which in turn changes their moniker), is that OOT tests that reach into these components via monikers (such as via component selectors) break. It's unfortunate but not surprising that these tests break - monikers aren't part of the FSI and are subject to change over time. We would like to make it possible for OOT product developers to write quality tests that automate the entire system to exercise their product Critical User Journeys (CUJs) without compromising on the platform team's ability to update the platform.
So far I've been leaning hard on RCS when advocating for what we should do differently in the next-gen system testing solutions. Particularly I find that allowing automation using Overnet, ffx, RCS, and in terms defined as FIDL would benefit updatability. But now it occurs to me (thanks Gary for pointing this out!) that RCS clients operate the target device by selecting which protocols to drive over Overnet via diagnostics selectors, which include stable parts like the protocol name, but also unstable parts like the moniker of the component that exposes the protocol capability. Furthermore RCS also exposes the full hub - understandable to power tools that do full system diagnostics such as `ffx component`, but maybe overpowered for test authors.

My questions please -
  1. RCS today has unscoped system access via hub, and exposes the host to monikers. Is this intended? Is this expected to change? I think the answers are yes and no respectively, but I just wanted to check.
  2. For system testing we would like to offer test authors a curated selection of protocols to receive access to so they could drive the system, and furthermore to expose these protocols in ways that we can manage via the SDK and FIDL and update over time. Perhaps a dedicated ffx proxy for a dedicated ffx plugin would be a better way to go, than have system test authors write system tests directly with RCS?
  3. Is there something else I'm missing? :)
Thank you!
Please feel free to add more people to this thread.

Fady Samuel

unread,
Aug 11, 2021, 8:18:19 PM8/11/21
to Shai Barack, component-framework-dev, Gary Bressler, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson
Hi Shai,

For 1:

RCS is not intended to use the Hub as it stands today. The Hub's directory structure is an implicit contract. I think we need a Debug protocol for the component manager instead of the Hub. Structured Hub is one possible solution. I think we should consider investing in this solution, and deprecate the Hub v2 as it stands today. 

For 2: 

If we're talking about protocols to drive the system I wonder if we should label them as such in component manager or in cmls. "Automation/Test APIs" should be automatically exposed to RCS somehow, maybe via an expose to framework idiom, which then offers them to RCS?

For 3:

I'm sure there'll be more the more we dig into the problem :-)

Thanks,

Fady


Shai Barack

unread,
Aug 12, 2021, 12:33:47 AM8/12/21
to Fady Samuel, component-framework-dev, Gary Bressler, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson
Agree that hub in general is a bit too much of a /dev or /proc. The vfs approach to examining components really hasn't proven itself either (sorry Rob Pike), and the diagnostics team knows this especially well.
I'm starting to think that RCS is too overpowered and under-opinionated for system automation. Rather, perhaps there ought to be an ffx proxy that specializes in system automation, expresses opinions in terms of which controller protocols for underlying system services it proxies to automation clients, and centralizes the knowledge of where to get those capabilities from - as this is a platform implementation detail, and should be encapsulated in a platform component.

Fady Samuel

unread,
Aug 12, 2021, 7:51:42 AM8/12/21
to Shai Barack, component-framework-dev, Gary Bressler, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson
Are you suggesting, Shai, that ffx shouldn't expose the Hub directly via ssh, but instead expose opinionated APIs for use by ffx plugins? If so, then I wholeheartedly agree! In fact, one of the root causes of slowness in `ffx component` was that it did a lot of Hub crawling, opening directories and files. If we had more direct Show and List APIs, then we'd be even faster than we are today I suspect.

Fady

Suraj Malhotra

unread,
Aug 12, 2021, 11:48:45 AM8/12/21
to Fady Samuel, Shai Barack, component-framework-dev, Gary Bressler, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson
For the driver framework, I roughly implemented a protocol specifically for development/debugging purposes, exposing similar info to the hub, only for use by ffx. It's not actually routed out driver_manager, because RCS can still use it. I am still a bit mixed on it because I would really love to run many of the same commands against snapshots/crash dumps, but can't. A vfs exporting debug info wouldn't help there, but using inspect possibly could. Another property I hope to enjoy at some point is the ability to run the same commands against isolated devmgr instances of the DF (e.g. tests), which should be possible.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "component-framework-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to component-framewo...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CAMZzjTmDS%2Bde98Kay4SyBWnKYJYG5PQC_NFTOCpG%2BKZ0ZYmqaA%40mail.gmail.com.

Gary Bressler

unread,
Aug 12, 2021, 2:32:31 PM8/12/21
to Suraj Malhotra, Fady Samuel, Shai Barack, component-framework-dev, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson, Christopher Johnson
There was a brainstorm this week that touched on this topic: https://docs.google.com/document/d/1n8wSoQ0gPXMsk9BoUW83U8EN5S4qNQLgNI2Slv6OH-0/edit

We had a consensus that the status quo of selecting protocols by moniker is not sustainable for out of tree, and had some ideas for how to present the choice of service in a more stable way. I see the hub as a superset of this problem: it presents a moniker-based view of components (through its nested structure), and it presents too much internal structure.

One saving grace is that at least all ffx plugins, at least for the moment, are developed in tree. This means that it is feasible (but maybe not convenient) to change a moniker atomically when a component is migrated or moved. However, this still leads to a potential compatibility issue if an old SDK is acting on a new Fuchsia image, or vice versa.

Shai Barack

unread,
Aug 12, 2021, 3:14:44 PM8/12/21
to Gary Bressler, Suraj Malhotra, Fady Samuel, component-framework-dev, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson, Christopher Johnson
Yes.
It's handy and also entirely intentional that we encapsulate platform implementation details inside ffx plugins and their proxies on the device, and then version and distribute ffx prebuilts together with platform images. 

Gary Bressler

unread,
Aug 13, 2021, 6:26:32 PM8/13/21
to Shai Barack, Suraj Malhotra, Fady Samuel, component-framework-dev, Xyan Bhatnagar, James Tucker, Jordon Wing, Scott Byer, Tom Robinson, Andrew Wilson, Christopher Johnson, Gabe Schine
Reply all
Reply to author
Forward
0 new messages