test listener

39 views
Skip to first unread message

Manav Brar

unread,
May 26, 2015, 3:25:35 PM5/26/15
to jumi-tes...@googlegroups.com
need to replace testng with jumi simpletest runner but a few prerequisite required.

I need to replace testng listener I currently use in my current site of tests, what would I use in jumi?

/**

 * The underlying {@link Set<String>} instance which contains the

 * {@link WebDriver} session id. This listener only expects

 * {@link AbstractSessionIds} implemented in all the tests.

 * 

 * @author mbrar

 */

public class SauceMobileSiteListener extends TestListenerAdapter {

private static final Logger LOGGER = LoggerFactory.getLogger(SauceMobileSiteListener.class);

private static final String SELENIUM_IS_LOCAL = "SELENIUM_IS_LOCAL";


private CounterMobileSite count;


/**

* The instance of the Sauce OnDemand Java REST API client.

*/

private SauceREST sauceREST;


/**

* Treat this test as a local test or run in SauceLabs.

*/

private boolean isLocal = false;


/**

* @param testContext

*/

@Override

public void onStart(final ITestContext testContext) {

super.onStart(testContext);

final String local = System.getenv(SELENIUM_IS_LOCAL);

if (local != null && local.isEmpty()) {

isLocal = true;

}

}


/**

* @param result

*/

@Override

public void onTestStart(final ITestResult result) {

super.onTestStart(result);


if (isLocal) {

return;

}


this.sauceREST = SelectorsInputs.identity(SauceREST.class, "sauceRest");

}


private void initlize(final ITestResult result) {

if (result.getInstance() instanceof AbstractSessionIds) {

final AbstractSessionIds sessionIds = (AbstractSessionIds) result.getInstance();

if (count == null) {

this.count = new CounterMobileSite(7, null, 0);

}


final AbstractMobileSite local = new MobileImmutable(null, "", null);

final Set<String> localSet = sessionIds.getImmutableList();

final Boolean validateSize = count.getCount().compareTo(count.getTop()) <= 0 && !localSet.isEmpty();

if (validateSize) {

count = count.withImmutable(local.withSessionId(localSet.iterator().next()));

if (!localSet.isEmpty()) {

localSet.remove(count.getImmutable().getSessionId());

count.increment();

}

}

LOGGER.info("SauceOnDemandSessionID={} job-name={}", count.getImmutable().getSessionId(), result

.getMethod().getMethodName());

}

}


/**

* @param testResult

*/

@Override

public void onTestFailure(final ITestResult testResult) {

super.onTestFailure(testResult);

if (isLocal) {

return;

}

validateNextSuit();

initlize(testResult);

if (this.sauceREST != null

&& count.getImmutable().getSessionId() != null) {

markJobAsFailed();

printPublicJobLink();

}

}


private void validateNextSuit() {

if (count != null && count.score()) {

count.resetCount();

}

}


private void markJobAsFailed() {

final Map<String, Object> updates = new HashMap<String, Object>();

updates.put("passed", false);

Utils.addBuildNumberToUpdate(updates);

sauceREST.updateJobInfo(count.getImmutable().getSessionId(), updates);

}


private void printPublicJobLink() {

final String authLink = this.sauceREST.getPublicJobLink(count

.getImmutable().getSessionId());

LOGGER.info("Test failure public job link: {}", authLink);

}


/**

* @param testResult

*/

@Override

public void onTestSuccess(final ITestResult testResult) {

super.onTestSuccess(testResult);

if (isLocal) {

return;

}

validateNextSuit();

initlize(testResult);

markJobAsPassed();

}


private void markJobAsPassed() {

if (this.sauceREST != null && count.getImmutable().getSessionId() != null) {

final Map<String, Object> updates = new HashMap<String, Object>();

updates.put("passed", true);

Utils.addBuildNumberToUpdate(updates);

sauceREST.updateJobInfo(count.getImmutable().getSessionId(), updates);

}

}

}



Esko Luontola

unread,
May 27, 2015, 12:47:15 PM5/27/15
to jumi-tes...@googlegroups.com
Manav Brar wrote on 26.5.2015 22:25:
> need to replace testng with jumi simpletest runner but a few
> prerequisite required.
>
> I need to replace testng listener I currently use in my current site of
> tests, what would I use in jumi?

The equivalent of TestNG's TestListenerAdapter (or more accurately the
interfaces it implements) is Jumi's SuiteListener interface [1].
Currently Jumi exposes those events only to the JVM process which
invokes Jumi (i.e. a build tool or IDE) through the
JumiLauncher.getEventStream() method [2].

I understand that those events might also be useful to test code running
inside the test runner JVM process. Would you please tell more about
your use case, so that we could think of a good solution to it? Jumi is
still early access software and it is missing many features which more
mature test runners have.

Also, I'm curious to hear that why you are replacing TestNG with Jumi's
SimpleUnit, which is meant to be just a minimal example for testing
framework developers instead of a full testing framework. A more general
purpose solution would be to create an adapter for TestNG so that you
can run TestNG tests inside Jumi [3].


[1]
https://github.com/orfjackal/jumi/blob/v0.5.437/jumi-core/src/main/java/fi/jumi/core/api/SuiteListener.java
[2]
https://github.com/orfjackal/jumi/blob/v0.5.437/jumi-launcher/src/main/java/fi/jumi/launcher/JumiBootstrap.java#L71

https://github.com/orfjackal/jumi/blob/v0.5.437/jumi-launcher/src/main/java/fi/jumi/launcher/JumiLauncher.java#L31
[3] Jumi already has such an adapter for JUnit. It converts events from
JUnit's RunListener to Jumi's SuiteNotifier, making it possible to run
JUnit tests inside Jumi. See
https://github.com/orfjackal/jumi/blob/v0.5.437/jumi-core/src/main/java/fi/jumi/core/junit/JUnitRunListenerAdapter.java

--
Esko Luontola
www.orfjackal.net

Manav Brar

unread,
May 28, 2015, 3:30:36 PM5/28/15
to jumi-tes...@googlegroups.com
All of tests run on a VM for our CI environment, where I open a sauce labs tunnel to run them remotely.
Average runtime for sauce labs to bootup a simulator is 1 min 25 sec, so each test runs on average for 2:40 to 3 min total time 
All UI tests run on remote sauce simulators so things are slow, we recently gained the ability to run 46 test in parallel on sauce. 
At work run all the tests in parallel which was 16 limit when I started writing them, testNg and JUnit try to use native threads they get very heavy when things have to run for a long time

Like the fact that jumi is 
  1. light weight in terms on it's reliance on actors. 
  2. tests running in parallel still show results for each individual separately
  3. If each test is running in a light weight jumi actor then maybe will have an ability to restart on failure an UI test based on Webdriver
  4. Q: ON adapter for JUnit. Does the use of this adapter would let me run the tests in parallel across multiple classes and not use JUnit native threads, (regular java thread is basically a OS thread on a VM have limited resources).

Esko Luontola

unread,
May 29, 2015, 5:37:54 PM5/29/15
to jumi-tes...@googlegroups.com
Manav Brar wrote on 28.5.2015 22:30:
> At work run all the tests in parallel which was 16 limit when I started
> writing them, testNg and JUnit try to use native threads they get very
> heavy when things have to run for a long time
>
> Like the fact that jumi is
>
> 1. light weight in terms on it's reliance on actors.

There is nothing special about Jumi's threads that would magically make
them faster. It's a normal thread pool, the same kind as for example
TestNG uses.

Modern operating systems are good at scheduling threads, so you would
typically need to have thousands of threads to have noticable overhead
due to thread switching or the thread's stack size (typically 512KB).

What matters is the work that the threads do. If the tests do lots of IO
or use lots of memory, then that may drain those resources of the
computer, causing it to slow down or run out of memory. If the tests
just use the CPU, then it won't really matter much even if there are ten
times more threads than CPU cores.

> 2. tests running in parallel still show results for each individual
> separately

Yes, that's something that Jumi does well.

> 3. If each test is running in a light weight jumi actor then maybe will
> have an ability to restart on failure an UI test based on Webdriver

The Jumi Actors library doesn't have supervisors, so there is no
restarting. Also the Jumi test runner doesn't have any built-in support
for restarting failing tests.

But individual testing frameworks may support it regardless. It can even
be done with a simple JUnit rule: https://gist.github.com/abyx/897229

> 4. Q: ON adapter for JUnit. Does the use of this adapter would let me
> run the tests in parallel across multiple classes and not use JUnit
> native threads, (regular java thread is basically a OS thread on a
> VM have limited resources).

Jumi runs JUnit test classes in parallel at the class level. It uses a
thread pool for running them, which defaults to as many threads are
there are CPU cores (it's configurable with
`JumiBootstrap.daemon.setTestThreadsCount(int)`). There are only one
kinds of threads in Java and there's nothing special about Jumi's use of
them.

--
Esko Luontola
www.orfjackal.net

Manav Brar

unread,
Jun 5, 2015, 11:06:26 AM6/5/15
to jumi-tes...@googlegroups.com
sorry by light weight I meant scaling up to hundreds of message passing actors on a small thread pool of 10 or 20 threads.

Let say I need to run 46(regular threads) test concurrently, each test in turn spawn 40 to 50 actors that would (poll the DOM)[this is already encapsulated] for WebElements, test need to fail if WebElement not found.

as for restart https://gist.github.com/abyx/897229 might be fine, thank you
Reply all
Reply to author
Forward
0 new messages