Run dynamically created tests

[Will Coleda]

I am trying to add some tests to a project where the list of tests isn't known until runtime (it's based on the list of files checked into a project). I want to run a series of assertions against each one of these files.

I can do all this from inside a single method (or a small set of methods), but if I have multiple assertions per test (one per file), or if I use the files as a data driven test provider, the first failure stops processing.

I don't want this. I instead want *ALL* the assertions to fire, so I can get a report at the end of what passed and what failed.

I could run all the assertions as simple checks, and manually keep track of the list of failures, and debug() the output, but that seems like the wrong direction to me.

Any suggestions? (Dynamically create functions for each file? How? Dynamically build the test file?)

--
Will "Coke" Coleda

[Mark Mandel]

You should be able to do this with a customer Decorator, overwriting getRunnableMethods()

https://github.com/mxunit/mxunit/blob/master/framework/TestDecorator.cfc#L61

Then the underlying test that is being decorated could implement onMissingMethod to run those returns methods dynamically.

In theory it should work.

Mark

--
You received this message because you are subscribed to the Google Groups "mxunit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mxunit+un...@googlegroups.com.
To post to this group, send email to mxu...@googlegroups.com.
Visit this group at http://groups.google.com/group/mxunit?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast

http://www.2ddu.com/

[Jamie Jackson]

Here's a test with dynamic functions. The mandatory test case functions for this decorator type are getDynamicTestFunctionNames() and onMissingMethod():

https://gist.github.com/jamiejackson/38e06abf6e3d8bc20d04#file-dynamicfunctionstest-cfc

Here's the decorator: https://gist.github.com/jamiejackson/38e06abf6e3d8bc20d04#file-dynamicfunctionstestdecorator-cfc

I'm satisfied with the tidiness of the getRunnableMethods() override, but the getAnnotation() override seems like a hack. I seemed to need to ignore the throw to get it to work (without throwing an exception*):

https://gist.github.com/jamiejackson/38e06abf6e3d8bc20d04#file-dynamicfunctionstestdecorator-cfc-L32-L37

Is there a better way to handle that annotation issue?

Thanks,

Jamie

*

Lucee 4.5.1.000 Error (mxunit.exception.methodNotFound)
Message	An annotation of expectedException was requested for the barTest_1 method, which does not exist.
Detail	Check the name of the method.
Stacktrace	The Error Occurred in /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestCase.cfc: line 460  458: <cfthrow type="mxunit.exception.methodNotFound" 459: message="An annotation of #arguments.annotationName# was requested for the #arguments.methodName# method, which does not exist." 460: detail="Check the name of the method." /> 461: <cfelse> 462: <cfset metadataTarget = this[arguments.methodName] /> called from /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestDecorator.cfc: line 91  called from /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc: line 89  called from /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc: line 55  called from /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuite.cfc: line 131  called from /var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/test/runner.cfm: line 97
Java Stacktrace	An annotation of expectedException was requested for the barTest_1 method, which does not exist.   at mxunit.framework.testcase_cfc$cf.udfCall5(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestCase.cfc:460):460   at mxunit.framework.testcase_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestCase.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.callWithNamedValues(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl.callWithNamedValues(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(Unknown Source):-1   at mxunit.framework.testdecorator_cfc$cf.udfCall2(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestDecorator.cfc:91):91   at mxunit.framework.testdecorator_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestDecorator.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.callWithNamedValues(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl.callWithNamedValues(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(Unknown Source):-1   at mxunit.framework.testdecorator_cfc$cf.udfCall2(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestDecorator.cfc:91):91   at mxunit.framework.testdecorator_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestDecorator.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl.call(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunction(Unknown Source):-1   at mxunit.framework.testsuiterunner_cfc$cf.udfCall1(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc:89):89   at mxunit.framework.testsuiterunner_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.call(Unknown Source):-1   at lucee.runtime.type.scope.UndefinedImpl.call(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunction(Unknown Source):-1   at mxunit.framework.testsuiterunner_cfc$cf.udfCall1(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc:55):55   at mxunit.framework.testsuiterunner_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuiteRunner.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl.call(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunction(Unknown Source):-1   at mxunit.framework.testsuite_cfc$cf.udfCall1(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuite.cfc:131):131   at mxunit.framework.testsuite_cfc$cf.udfCall(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/mxunit/framework/TestSuite.cfc):-1   at lucee.runtime.type.UDFImpl.implementation(Unknown Source):-1   at lucee.runtime.type.UDFImpl._call(Unknown Source):-1   at lucee.runtime.type.UDFImpl.call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl._call(Unknown Source):-1   at lucee.runtime.ComponentImpl.call(Unknown Source):-1   at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(Unknown Source):-1   at lucee.runtime.PageContextImpl.getFunction(Unknown Source):-1   at test.runner_cfm$cf.call(/var/www/vmhost/apps/hudexchange/cfml/deployment_root/wwwroot/test/runner.cfm:97):97   at lucee.runtime.PageContextImpl.doInclude(Unknown Source):-1   at lucee.runtime.PageContextImpl.doInclude(Unknown Source):-1   at lucee.runtime.listener.ModernAppListener._onRequest(Unknown Source):-1   at lucee.runtime.listener.MixedAppListener.onRequest(Unknown Source):-1   at lucee.runtime.PageContextImpl.execute(Unknown Source):-1   at lucee.runtime.PageContextImpl.execute(Unknown Source):-1   at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(Unknown Source):-1   at lucee.loader.servlet.CFMLServlet.service(Unknown Source):-1   at javax.servlet.http.HttpServlet.service(HttpServlet.java:727):727   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303):303   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52):52   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241):241   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208   at org.apache.catalina.filters.RemoteIpFilter.doFilter(RemoteIpFilter.java:834):834   at org.apache.catalina.filters.RemoteIpFilter.doFilter(RemoteIpFilter.java:870):870   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241):241   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220):220   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122):122   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504):504   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170):170   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103):103   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116):116   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421):421   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074):1074   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611):611   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314):314   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145):1145   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615):615   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61):61   at java.lang.Thread.run(Thread.java:745):745
Timestamp	5/1/15 10:20:07 AM CDT

[Jamie Jackson]

Unfortunately, this also seems to need a core code change: DataProviderDecorator.cfc:

### Eclipse Workspace Patch 1.0

#P hudexchange

Index: cfml/deployment_root/lib/mxunit/framework/decorators/DataProviderDecorator.cfc

===================================================================

--- cfml/deployment_root/lib/mxunit/framework/decorators/DataProviderDecorator.cfc (revision 65029)

+++ cfml/deployment_root/lib/mxunit/framework/decorators/DataProviderDecorator.cfc (working copy)

@@ -14,8 +14,12 @@

  <cffunction name="invokeTestMethod" access="public" returntype="string" output="false">

  <cfargument name="methodName" hint="the name of the method to invoke" type="string" required="Yes"/>

  <cfargument name="args" hint="Optional set of arguments" type="struct" required="No"/>

-

- <cfset var dpName = getAnnotation(methodName, "dataprovider")/>

+

+ <cfset dpName = "" >

+ <cftry>

+ <cfset var dpName = getAnnotation(methodName, "dataprovider")/>

+ <cfcatch></cfcatch>

+ </cftry>

  <cfset var outputOfTest = "">

  <cfif len(dpName)>

  <!---<cflog text="inside dataproviderdecorator....  running dataprovider named #dpName#">--->

I'm just hacking, and I don't know what problems this might introduce, though.

Thanks,

Jamie

Visit this group at http://groups.google.com/group/mxunit.
For more options, visit https://groups.google.com/d/optout.