Applying BDD / Cucumber to large batch execution scenarios

[CS Wong]

[This question was previously asked in StackOverflow but since this is the official Cucumber community group, I thought I'd stand a better chance here]

I'm trying to apply BDD practices to my organization using Cucumber. I work in a bank where the nightly batch job is a huge orchestrated multi-system flow of batch jobs running and passing data between one another. 

During our tests, interactive online tests probably make up only 40-50% of test scenarios while the rest are embedded inside the batch job. As an example, the test scenario may be:

 1. Given that my savings account has a balance of $100 as of 10PM

 2. When the nightly batch is run at 11PM

 3. Then at 3AM after the batch run is finished, I should come back and see that I have an additional accrued interest of $0.001.

 4. And the general ledger of the bank should have an additional entry for accrued interest of $0.001.

So as you can see, this is an extremely asynchronous scenario. If I were to use Cucumber to trigger it, I can probably create a step definition to insert the $100 balance into the account by 10PM, but it will not be realistic to use Cucumber to trigger the batch to be run at 11PM as batch jobs are usually executed by operators using their own scheduling tools such as Control-M. And having Cucumber then wait and listen a few hours before verifying the accrued interest, I'm not sure if I'll run into a timeout or not.

This is just one scenario. Batch runs are very expensive for the bank and we always tack on as many scenarios as possible to ride on a single batch run. We also have aging scenarios where we need to run 6 months of batch just to check whether the final interest at the end of a fixed deposit term is correct or not (I definitely cannot make Cucumber wait and listen for that long, can I?)

My question is, is there any example where BDD practices were applied to large batch scenarios such as these? How would one approach this?

Please note that isolated batch scenarios in a controlled environment are quite possible to be implemented using Cucumber and I do have it implemented in some of my proof of concept work. But eventually, we need to hit a test level that has an entire end-to-end environment, typically in SIT. In this environment, it is a criteria for multiple test scenarios to be run in parallel, none of which have complete control over the environment. Depending on the scope of the project, this environment may run up to 200 applications. So customer channels such as Internet Banking will run transactional scenarios, whiles at the core banking system, scenarios such as interest calculation, automatic transfers etc will be executed. There will also be accounting scenarios where a general ledger system consolidates and balances all the accounts in the environment. To do manual testing in this environment frequently requires at least 30-50 personnel executing transactions and checking on results.

What I am trying to do is to find a way to leverage on a BDD framework to automate test execution and capture the results so that we do not have to manually track them all in the environment.

Thanks!

Wong

[Aslak Hellesøy]

On Thursday, May 12, 2016 at 2:02:16 PM UTC+1, CS Wong wrote:

[This question was previously asked in StackOverflow but since this is the official Cucumber community group, I thought I'd stand a better chance here]

I'm trying to apply BDD practices to my organization using Cucumber. I work in a bank where the nightly batch job is a huge orchestrated multi-system flow of batch jobs running and passing data between one another. 

I recently coached a team at a large company in the finance sector. They have batch jobs written in Cobol running on mainframes.

We figured out an elegant pattern to test them from Cucumber. I intend to write a longer blog post, but the process is basically:

* Given: FTP upload a file to the mainframe. The batch job will read this file.

* When: trigger the job by uploading a "trigger" file. Another batch job will poll for the presence of this file and trigger your batch job.

* Then: FTP download the file generated by your batch job. Compare its contents to expected results.

HTH,

Aslak

[CS Wong]

On Thursday, May 12, 2016 at 8:15:33 PM UTC+7, Aslak Hellesøy wrote:

On Thursday, May 12, 2016 at 2:02:16 PM UTC+1, CS Wong wrote:

[This question was previously asked in StackOverflow but since this is the official Cucumber community group, I thought I'd stand a better chance here]

I'm trying to apply BDD practices to my organization using Cucumber. I work in a bank where the nightly batch job is a huge orchestrated multi-system flow of batch jobs running and passing data between one another. 

I recently coached a team at a large company in the finance sector. They have batch jobs written in Cobol running on mainframes.

We figured out an elegant pattern to test them from Cucumber. I intend to write a longer blog post, but the process is basically:

* Given: FTP upload a file to the mainframe. The batch job will read this file.

* When: trigger the job by uploading a "trigger" file. Another batch job will poll for the presence of this file and trigger your batch job.

* Then: FTP download the file generated by your batch job. Compare its contents to expected results.

HTH,

Aslak

Hi Aslak, thanks for your answer. The scenario you described works very well for isolated testing of specific MF jobs. In my case, instead of uploading a trigger file, I would probably write a simple program to trigger the job to run via the Control-M job scheduler on MF. But for E2E scenarios, it's not as simple as that as part of what we are testing is an entire flow of different jobs wired to be triggered between each other. For me, it doesn't really make sense to write a technically-oriented scenario saying "We will run an E2E batch job" because there are hundreds of real business scenarios that are embedded into various jobs in the flow. If I write scenarios according to the business examples, I will end up with very extreme asynchronous wait times for the results to come back (example given in earlier post below) which is not very practical. 

I was thinking that perhaps a way to do this is to implement way to:

1. Use cucumber to start a scenario.
2. Run through "Given" step definitions to set up the scenario (e.g. make sure that accounts have correct balances in place etc)
3. Pause the scenario execution at the "When" stage of the scenario and set the status of the scenario as "Pending".
   
4. Exit cucumber with a certain execution ID.
5. Implement a callback mechanism where when the batch job that executed the job completes, I will find some way to invoke cucumber to recall this particular scenario execution using its execution ID.
6. Cucumber then runs the "Then" steps to check on results.

Would that be possible?

Thanks

Wong

[Andrew Premdas]

On 18 May 2016 at 07:23, CS Wong <lil...@gmail.com> wrote:

On Thursday, May 12, 2016 at 8:15:33 PM UTC+7, Aslak Hellesøy wrote:

On Thursday, May 12, 2016 at 2:02:16 PM UTC+1, CS Wong wrote:

[This question was previously asked in StackOverflow but since this is the official Cucumber community group, I thought I'd stand a better chance here]

I'm trying to apply BDD practices to my organization using Cucumber. I work in a bank where the nightly batch job is a huge orchestrated multi-system flow of batch jobs running and passing data between one another. 

I recently coached a team at a large company in the finance sector. They have batch jobs written in Cobol running on mainframes.

We figured out an elegant pattern to test them from Cucumber. I intend to write a longer blog post, but the process is basically:

* Given: FTP upload a file to the mainframe. The batch job will read this file.

* When: trigger the job by uploading a "trigger" file. Another batch job will poll for the presence of this file and trigger your batch job.

* Then: FTP download the file generated by your batch job. Compare its contents to expected results.

HTH,

Aslak

Hi Aslak, thanks for your answer. The scenario you described works very well for isolated testing of specific MF jobs. In my case, instead of uploading a trigger file, I would probably write a simple program to trigger the job to run via the Control-M job scheduler on MF. But for E2E scenarios, it's not as simple as that as part of what we are testing is an entire flow of different jobs wired to be triggered between each other. For me, it doesn't really make sense to write a technically-oriented scenario saying "We will run an E2E batch job" because there are hundreds of real business scenarios that are embedded into various jobs in the flow. If I write scenarios according to the business examples, I will end up with very extreme asynchronous wait times for the results to come back (example given in earlier post below) which is not very practical. 

I was thinking that perhaps a way to do this is to implement way to:

1. Use cucumber to start a scenario.
2. Run through "Given" step definitions to set up the scenario (e.g. make sure that accounts have correct balances in place etc)
3. Pause the scenario execution at the "When" stage of the scenario and set the status of the scenario as "Pending".
   
4. Exit cucumber with a certain execution ID.
5. Implement a callback mechanism where when the batch job that executed the job completes, I will find some way to invoke cucumber to recall this particular scenario execution using its execution ID.
6. Cucumber then runs the "Then" steps to check on results.

Would that be possible?

Thanks

Wong

Generally in test environments the test runner has control and the environment is isolated. So in the case of a batch job foo that runs once a day, you would write a step like

  Given foo has run

or

  When I run foo

and implement these in such a way that the batch job runs straight away.

If you don't have that sort of control then you are using Cucumber way out of its comfort zone, and you have to take responsibility for running things in such a way to meet your own particular circumstances..

One way to do this might be to split your test into different scenarios and then write a script to run each scenario e.g.

   script

     run cucumber upto_foo

     wait_for foo

     run cucumber after_foo

   end

This is unwieldy and clunky, but that's because of your environment, not cucumber

A better way would be be to set up a better environment, but you seem to be indicating that your environment is far to complex for that.

If you are going to test a live or near live aynchornous environment with scenarios that can run over a period of hours/days you have to script to your particular circumstances i.e. create your own tools, if you want to automate. As this is likely to outrageously expensive and time consuming, you might want to consider other alternatives like

1. Manual testing

2. Testing across shorter boundaries

All best

Andrew

 

 

During our tests, interactive online tests probably make up only 40-50% of test scenarios while the rest are embedded inside the batch job. As an example, the test scenario may be:

 1. Given that my savings account has a balance of $100 as of 10PM

 2. When the nightly batch is run at 11PM

 3. Then at 3AM after the batch run is finished, I should come back and see that I have an additional accrued interest of $0.001.

 4. And the general ledger of the bank should have an additional entry for accrued interest of $0.001.

So as you can see, this is an extremely asynchronous scenario. If I were to use Cucumber to trigger it, I can probably create a step definition to insert the $100 balance into the account by 10PM, but it will not be realistic to use Cucumber to trigger the batch to be run at 11PM as batch jobs are usually executed by operators using their own scheduling tools such as Control-M. And having Cucumber then wait and listen a few hours before verifying the accrued interest, I'm not sure if I'll run into a timeout or not.

This is just one scenario. Batch runs are very expensive for the bank and we always tack on as many scenarios as possible to ride on a single batch run. We also have aging scenarios where we need to run 6 months of batch just to check whether the final interest at the end of a fixed deposit term is correct or not (I definitely cannot make Cucumber wait and listen for that long, can I?)

My question is, is there any example where BDD practices were applied to large batch scenarios such as these? How would one approach this?

Please note that isolated batch scenarios in a controlled environment are quite possible to be implemented using Cucumber and I do have it implemented in some of my proof of concept work. But eventually, we need to hit a test level that has an entire end-to-end environment, typically in SIT. In this environment, it is a criteria for multiple test scenarios to be run in parallel, none of which have complete control over the environment. Depending on the scope of the project, this environment may run up to 200 applications. So customer channels such as Internet Banking will run transactional scenarios, whiles at the core banking system, scenarios such as interest calculation, automatic transfers etc will be executed. There will also be accounting scenarios where a general ledger system consolidates and balances all the accounts in the environment. To do manual testing in this environment frequently requires at least 30-50 personnel executing transactions and checking on results.

What I am trying to do is to find a way to leverage on a BDD framework to automate test execution and capture the results so that we do not have to manually track them all in the environment.

Thanks!

Wong

--

Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[Eric Kessler]

On Wednesday, May 18, 2016 at 5:12:09 AM UTC-4, apremdas wrote:

On 18 May 2016 at 07:23, CS Wong <lil...@gmail.com> wrote:

On Thursday, May 12, 2016 at 8:15:33 PM UTC+7, Aslak Hellesøy wrote:

On Thursday, May 12, 2016 at 2:02:16 PM UTC+1, CS Wong wrote:

[This question was previously asked in StackOverflow but since this is the official Cucumber community group, I thought I'd stand a better chance here]

I'm trying to apply BDD practices to my organization using Cucumber. I work in a bank where the nightly batch job is a huge orchestrated multi-system flow of batch jobs running and passing data between one another. 

I recently coached a team at a large company in the finance sector. They have batch jobs written in Cobol running on mainframes.

We figured out an elegant pattern to test them from Cucumber. I intend to write a longer blog post, but the process is basically:

* Given: FTP upload a file to the mainframe. The batch job will read this file.

* When: trigger the job by uploading a "trigger" file. Another batch job will poll for the presence of this file and trigger your batch job.

* Then: FTP download the file generated by your batch job. Compare its contents to expected results.

HTH,

Aslak

Hi Aslak, thanks for your answer. The scenario you described works very well for isolated testing of specific MF jobs. In my case, instead of uploading a trigger file, I would probably write a simple program to trigger the job to run via the Control-M job scheduler on MF. But for E2E scenarios, it's not as simple as that as part of what we are testing is an entire flow of different jobs wired to be triggered between each other. For me, it doesn't really make sense to write a technically-oriented scenario saying "We will run an E2E batch job" because there are hundreds of real business scenarios that are embedded into various jobs in the flow. If I write scenarios according to the business examples, I will end up with very extreme asynchronous wait times for the results to come back (example given in earlier post below) which is not very practical. 

I was thinking that perhaps a way to do this is to implement way to:

1. Use cucumber to start a scenario.
2. Run through "Given" step definitions to set up the scenario (e.g. make sure that accounts have correct balances in place etc)
3. Pause the scenario execution at the "When" stage of the scenario and set the status of the scenario as "Pending".
   
4. Exit cucumber with a certain execution ID.
5. Implement a callback mechanism where when the batch job that executed the job completes, I will find some way to invoke cucumber to recall this particular scenario execution using its execution ID.
6. Cucumber then runs the "Then" steps to check on results.

Would that be possible?

Thanks

Wong

Yes, it is possible. In fact...voila!

https://gist.github.com/enkessler/beb0bc2c7707ab8ae7ee23f30524df7a

Steps 1-3 are fine. Steps 4 and 5 are the ones that may need tweaking based on your particular set of circumstances, but special exit codes for individual tests seemed a bit narrow, so I went with a tagging system so that all of the asynchronous tests could be run at the same time (once before the batch cycle and once afterward). Step 6 happens normally during the second run. While the knowledge of whether they are being run for the first time or the  second time is built into each test, the additional tooling necessary in order to know when to actually kick off execution of the second run is left as an exercise for the reader. ;)

 

Generally in test environments the test runner has control and the environment is isolated. So in the case of a batch job foo that runs once a day, you would write a step like

  Given foo has run

or

  When I run foo

and implement these in such a way that the batch job runs straight away.

If you don't have that sort of control then you are using Cucumber way out of its comfort zone, and you have to take responsibility for running things in such a way to meet your own particular circumstances..

One way to do this might be to split your test into different scenarios and then write a script to run each scenario e.g.

   script

     run cucumber upto_foo

     wait_for foo

     run cucumber after_foo

   end

This is unwieldy and clunky, but that's because of your environment, not cucumber

 

Yep. All the usual 'you are using Cucumber wrong' arguments apply here. However, I think that splitting the test into multiple scenarios would further hurt its documentation/communication ability. As a single scenario, it can remain clear that it is one cohesive (if rather temporally spread out) behavior.

[CS Wong]

On Sunday, May 22, 2016 at 11:56:48 PM UTC+7, Eric Kessler wrote:

On Wednesday, May 18, 2016 at 5:12:09 AM UTC-4, apremdas wrote:

On 18 May 2016 at 07:23, CS Wong <lil...@gmail.com> wrote:

On Thursday, May 12, 2016 at 8:15:33 PM UTC+7, Aslak Hellesøy wrote:

On Thursday, May 12, 2016 at 2:02:16 PM UTC+1, CS Wong wrote:

I was thinking that perhaps a way to do this is to implement way to:

1. Use cucumber to start a scenario.
2. Run through "Given" step definitions to set up the scenario (e.g. make sure that accounts have correct balances in place etc)
3. Pause the scenario execution at the "When" stage of the scenario and set the status of the scenario as "Pending".
   
4. Exit cucumber with a certain execution ID.
5. Implement a callback mechanism where when the batch job that executed the job completes, I will find some way to invoke cucumber to recall this particular scenario execution using its execution ID.
6. Cucumber then runs the "Then" steps to check on results.

Would that be possible?

Thanks

Wong

Yes, it is possible. In fact...voila!

https://gist.github.com/enkessler/beb0bc2c7707ab8ae7ee23f30524df7a

Steps 1-3 are fine. Steps 4 and 5 are the ones that may need tweaking based on your particular set of circumstances, but special exit codes for individual tests seemed a bit narrow, so I went with a tagging system so that all of the asynchronous tests could be run at the same time (once before the batch cycle and once afterward). Step 6 happens normally during the second run. While the knowledge of whether they are being run for the first time or the  second time is built into each test, the additional tooling necessary in order to know when to actually kick off execution of the second run is left as an exercise for the reader. ;)

 

Thank you very much. I'll be trying this out over the next couple of days.

 

Generally in test environments the test runner has control and the environment is isolated. So in the case of a batch job foo that runs once a day, you would write a step like

  Given foo has run

or

  When I run foo

and implement these in such a way that the batch job runs straight away.

If you don't have that sort of control then you are using Cucumber way out of its comfort zone, and you have to take responsibility for running things in such a way to meet your own particular circumstances..

One way to do this might be to split your test into different scenarios and then write a script to run each scenario e.g.

   script

     run cucumber upto_foo

     wait_for foo

     run cucumber after_foo

   end

This is unwieldy and clunky, but that's because of your environment, not cucumber

 

Yep. All the usual 'you are using Cucumber wrong' arguments apply here. However, I think that splitting the test into multiple scenarios would further hurt its documentation/communication ability. As a single scenario, it can remain clear that it is one cohesive (if rather temporally spread out) behavior.

Totally agree with you on this. Ultimately, the only thing that I consider to be sacred is that the business scenario must remain as how the business analysts / product owners would have written it in their context. 

Thanks

Wong

[Justin Radcliffe]

We have something similar to this as well.

We would like to setup 50 accounts in different statuses, then run batch jobs that could have a few steps that could take 10-40 minutes per step.

Given an <active> account
 When I set the status to <closed>
 Then the status state should be set to <deactivated>


Given an <closed> account
 When I set the status to <active>
 Then the status state should be set to <activated>

etc.

etc.

So I can have scenarios/outlines to get all my data setup. I was thinking we could go store all this data in an 'automation team' database table for the account number, before status, tobe status etc. so we can validate in the Then for the batch job here:

Given my account cancellation job is ready to run                  #checks the DB to make sure we have the 50 account generated and ready to process
 When the account cancellation job is completed                     # this is the piece that could have 4 steps that could take 10-40 minutes
 Then the accounts are set to the new status                        # validates our automation table described above to the table where the batch job updates.

Does this seem reasonable since we have to get so many different preconditions setup? That in itself could take and hour or so to process, so I didn't want to put that all in the precondition of the batch scenario itself.

Thoughts?