SL4F for CFv2

[Justin Mattson]

Are there any active workstreams on what users of SL4F should do if they want to move their components to v2 and still run their e2e tests?

Cheers,

Justin

[Shai Barack]

I'm assuming that the component that's being migrated is automated by sl4f using some sys capability that it serves. As long as that component continues to offer that capability to the sys realm (where sl4f.cmx is launched) then sl4f won't know the difference and automation should continue to work.

If sl4f is interacting with the migrating component by other means, such as injecting files into "known" global minfs paths or traversing v1 hub paths or some other approach that doesn't meet platform evolution best practices, then the migration will be more difficult and the solution might be case-by-case.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "component-framework-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to component-framewo...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX8u-MgZ%2Bry9iKu1JqOqZ2fGK9qv6dj0dXPVBD3%3DVNqNTg%40mail.gmail.com.

[Justin Mattson]

In this particular case a v1 test component is created which access a capability routed to the sys realm. This means we aren't precluded from testing the v2 component via this mechanism, but it does mean we still have a v1 test component.

I should have been clearer, do we have a path or planned path for components with e2e tests to get rid of all their v1 manifests.

[Shai Barack]

As in, sl4f launches the component? Then no, we'll have to keep the old v1 component around just for the test. This is the same as for instance where we keep a v1 component around because there are workflows that invoke it via `fx shell run ...` until there's a replacement (for instance an ffx component that uses RCS to perform the interaction).

The issue with sl4f dominating a lot of legacy/stale v1 components will be with us for some time I'm afraid. EngProd Testing team is committed to developing a v2 e2e automation solution, e2e test owners want to migrate, and the staffing gaps are being taken care to, but it'll be a while still.

[Shai Barack]

Even then, the migration is hugely beneficial. For starters it's one more unblocker for deprecating sl4f! 

[Justin Mattson]

Why not change it so SL4F could launch things in the test realm or the ffx-laboratory realm?

[Gabe Schine]

Is there a list somewhere of the e2e tests that directly launch components?

To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX_tyogFa2AxEugh09P7Ovu1aM4tD9M3wFbihOzC59DYHw%40mail.gmail.com.

--

Gabe Schine

Software Engineer / Manager

🔵 Blue Dot Ally

go/thatguy-readme

go/where-is-thatguy

[Shai Barack]

That's interesting actually, maybe it's worth asking ourselves if there is an incremental evolution path for sl4f that accelerates v2 migrations.

Finding all the sl4f tests that launch components is difficult because using an sl4f facade (such as the facade that launches a component) is not modeled as a distinct capability. Instead you have to do a code search for some known signature of what the sl4f client code would look like.

Also there is an sl4f ssh facade (which can launch v1 components in the sys realm, or anywhere really) which actually doesn't speak to an sl4f facade, it just sends characters to sshd. 

[Justin Mattson]

Assembling such a list I think would require manual inspection of all SL4F facades and possibly tracking back to all the tests that use them if there are any facades that are used by multiple tests. I don't have much experience with SL4F so I don't know what the common use pattern is.

[Shai Barack]

This is an area where we could ask Yuan or Robert Ma for help. If there's not that many usages, maybe we could do something about them and then remove this facade.

Another option is to remove the fuchsia.sys.Launcher capability from sl4f.cmx and see what tests fail. We'd have to run on both internal CQ and on OOT presubmits with the LSC tool, but it all comes down to a few commands and waiting a couple of hours for a report.

[Shai Barack]

https://fuchsia-review.googlesource.com/c/fuchsia/+/592154/

Added internal tryjobs for partners. You can see them if you're signed in.

[Fady Samuel]

+yuanzhi +robertma Thanks Shai! I looped them in. 

Fady 

To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CAK0PkCHitUw%3De08mJtUZnLKo1oSPLH0K%2B803mqMWQVRPO-GbVw%40mail.gmail.com.

[Shai Barack]

Here's one in-tree test that launches a component and fails with my change:

src/tests/end_to_end/harvester/test/harvester_test.dart

It just launches system monitor harvester and asks it to run once, expecting that it prints "Success". This doesn't need to be an e2e test at all, this could be a component test.

IIRC we're not using Harvester anymore since we stopped developing FDT, so I think it's also safe to delete this test.

There is this test:

src/tests/end_to_end/encode_camera/test/encode_camera_test.dart

It should fail, but it doesn't because it's disabled. According to the comment it's flaky.

That said, looking at the change I don't see why it needs to be an sl4f test at all. It launches a sort of camera demo component (which is very non-hermetic, but I'm sure this can be fixed - probably should, seeing as the test is said to be flaky) and then verifies that the component encoded some number of camera frames in the span of 5 seconds by looking at a trace captured during the runtime (why not Inspect?).

Maybe I'm missing something but I think the team could have written a better test (hermetic, less flaky, simpler, not in Dart - which is a fine language, but foreign to the camera team I'm sure) if we had given them better guidance and clarified which tools are best to use here. +tq-testing-everywhere 

Similar story here: src/tests/end_to_end/bt-a2dp/test/bt_a2dp_test.dart

Anyway, THE GOOD NEWS is that all the tests that use the sl4f facade to launch a component are in-tree, at least according to our LSC tools and to my Code Search abilities. And there's very few of them.

I think we can remove the ability to launch a component from sl4f. To de-risk the v2 migration, and to discourage the introduction of more e2e tests that could and should be component tests, I would advise that we proceed with this course of action.

What do folks think?

[Gabe Schine]

Anyway, THE GOOD NEWS is that all the tests that use the sl4f facade to launch a component are in-tree, at least according to our LSC tools and to my Code Search abilities. And there's very few of them.

I think we can remove the ability to launch a component from sl4f. To de-risk the v2 migration, and to discourage the introduction of more e2e tests that could and should be component tests, I would advise that we proceed with this course of action.

What do folks think?

+1 from me. The investment return we get from removing this feature altogether and the changes to the tests to facilitate that will have better returns in the long term.

[Shai Barack]

Thinking about this some more.

There is a long term goal of promoting good testing and creating good testing practices and tools. The harvester, bt, and camera tests that I linked to don't demonstrate testing best practices in my opinion, and we'd like to not have more tests like these. They are lacking good synchronization mechanisms, are difficult to debug and troubleshoot, there is no particular reason to control them from the host, they use e2e tools for things that can be achieved with component testing, among other things. Replacing these tests with new better tests, as well as perhaps preventing new such tests from being created, sounds like a good idea to me.

As a next step I would like to speak to the authors of these tests to get more context from them. I'm interested in how they feel about these tests, what value they derive from them, what problems these tests cause them from their perspective, and any other info or sentiments.

That said, because there's relatively few of these tests, and because sl4f can't launch v2 components, then from the perspective of unblocking v2 migrations we just need to implement the same test coverage / added value by other (better) means and move on.

This is important because the last thing I want for the v2 migration is to be blocked by sl4f features / deprecations. 

[Justin Mattson]

What I'm hearing is that we actually don't have many tests that launch v1 components.

A more general question is, what are SL4F tests doing in general and can these use cases be supported with things we have in v2 testing? Is our official recommendation to owners of SL4F tests to change their test architecture and stop using SL4F? So far I've not suggested people make these changes because I'm not aware of an explicit stance on SL4F.

As a next step I would like to speak to the authors...

Does this mean literally that you plan on doing this or is there an implicit request that someone do this?

As coincidence would have it I'm familiar with the codec factory encoding test. The reason it was structured as an e2e test is because the test wants to start a trace and then analyze the trace data host side. I'm not sure if the host side analysis is required.

I also know a little about the BT test. The main points here are the test wants to make sure to start a trace before the real part of the test begins. Another thing is that they wanted to be able to provide a parameter to the test (currently via command line) for how long the test should run.

Cheers,

Justin

[Justin Mattson]

This recently came up with DRM, so I asked that RJ create a bug to start tracking what he needs, which he did.

[Shai Barack]

Literally speak to the authors.

See this thread:

https://groups.google.com/a/google.com/g/tq-testing-everywhere/c/zmP4r4gkraQ

I think we lack clear guidance today on when it's recommended or not to use sl4f. Test authors are mostly left to figure things out on their own. There is some supporting evidence in the thread. 

The test authors get it right more often than not (we have smart people on this team after all). However due to lacking guidance and multiple competing solutions we inevitably see inconsistent results. Also I am concerned that we are accumulating lower-quality sl4f tests as a result.