Hi
I'm looking for some advice on whether Cucumber and SBE is a good fit for the data transformation logic in ETL applications.
I work on an integration platform where the core functionality is routing and transforming messages between source and target sytems. The various interfaces include csv, json and xml formats.
A typical epic for our dev teams might be:
'A cash trade (in json format) from System A needs to be transformed (to an industry standard xml format) and routed to accounting System B.'
The development process for such a story begins with an analyst (often in isolation) documenting the mapping rule for each piece of data which must be transformed between source and target systems. A piece of data is represented by a value and a location xpath/jsonpath/csv header depending on the format. There could be in the region of a few hundred pieces of data on a trade and each one has a transformation rule.
The bulk of transformations are what we call simple 1 to 1 mappings: the data value doesn't change but the location is obviously different between the source and target formats. More complicated transformation rules may have conditional logic based on several values .
Here is an example rule as written by the analyst for a json to xml transformation for a bond trade. The target settlement amount value is based the source trade principal value:
Target location xpath:
StockTransaction/paymentDetails/settlementAmount/amount
Target value:
If SM_CURRENCY = TRD_CURRENCY
TRD_PRINCIPAL
else
TRD_PRINCIPAL*SETTLE_EXCH_RATE
So I had an attempt at writing this rule as a Gherkin feature with a view to replacing the mapping spec with a series of features which could then be automated using either Cucumber or Spock.
Feature: Settlement amount
Target location xpath:
StockTransaction/paymentDetails/settlementAmount/amount
Target value:
If SM_CURRENCY = TRD_CURRENCY
TRD_PRINCIPAL
else
TRD_PRINCIPAL*SETTLE_EXCH_RATE
Scenario: Settlement currency is the same as trade currency
Given SM_CURRENCY is 'USD'
And TRD_CURRENCY is 'USD'
And TRD_PRINCIPAL is 1000000
Then amount is 1000000
Scenario: Settlement currency and trade currency are different
Given SM_CURRENCY is 'USD'
And TRD_CURRENCY is 'GBP'
And TRD_PRINCIPAL is 1000000
And SETTLE_EXCH_RATE is 1.5
Then amount is 1500000
To me that certainly
illustrates the business rule for that particular piece of data and looks like a specification the devs could
code from. The feature could also easily be automated using cucumber or
spock.
What do you guys think about this as an approach, especially when it needs to
scale for the many more pieces of data which make up the trade? Is it too low
level?
The analysts are currently are very comfortable using excel to document the business rules but rarely give example scenarios. As you can imagine this excel specification frequently gets out of sync with the code, and the tests we automate from it are often just decided by devs and testers without any input from the analysts. SBE seems like a good way to remove these discrepancies and have a single testable artefact.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Thanks for your reply Andrew.
So i f we ignore the cucumber automation part for a minute, are you saying that we would not benefit from Specification By Example and capturing the work the analyst does as Gherkin features?
Our main priority is to do away with the spreadsheets which the analysts produce and replace it with some form of living documentation. The spreadsheet is frequently getting out of sync with the code.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
If SM_CURRENCY = TRD_CURRENCY
settlement amount = TRD_PRINCIPAL
else
settlement amount = TRD_PRINCIPAL*SETTLE_EXCH_RATE
Scenario: Settlement currency and trade currency are different
Given settlement currency is 'USD'
And TRD_CURRENCY is 'GBP'
And TRD_PRINCIPAL is £1000000
And SETTLE_EXCH_RATE is 1.5
Scenario: Settlement currency and trade currency are different
Given settlement currency is 'USD'
And trade currency is 'GBP'
And trade principal is £1000000
And settlement exchange rate is 1.5
Then settlement amount is $1500000
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
Scenario: Cancellation message is valid
Given: A trade execution message with trade id 1234 has been processed
When: A trade cancellation message with trade id 1234 is received
Then The cancellation message is valid and should be processed
Scenario: Cancellation message is invalid
Given: A trade execution message with trade id 1234 has NOT been processed
When: A trade cancellation message with trade id 1234 is received
Then The cancellation message is invalid and should be discardedTo unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
Mark Levison | 1 (877) 248-8277 | Twitter | LinkedIn | Facebook Certified ScrumMaster Training: Vancouver | Edmonton | Ottawa | Montreal | Toronto Certified Product Owner & Private Training also available ~ Our Training Schedule Agile Pain Relief Consulting | Notes from a Tool User Proud Sponsor of Agile Tour Gatineau Ottawa and Agile Coach Camp Canada |
Brilliant, that makes sense. sa = tcr/scr * tpa is the most efficient way of describing the rule for calculating settlement amounts. Everything else is just fluff.
So how about following business rule. Is it benefiting from the GWT scenarios below or am i just restating the rule?
- A trade cancellation message must be rejected if the trade it is cancelling does not exist
Scenario: Cancellation message is valid
Given: A trade execution message with trade id 1234 has been processed
When: A trade cancellation message with trade id 1234 is received
Then The cancellation message is valid and should be processed
Scenario: Cancellation message is invalid
Given: A trade execution message with trade id 1234 has NOT been processed
When: A trade cancellation message with trade id 1234 is received
Then The cancellation message is invalid and should be discarded
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.