A few weeks ago someone discovered that their bank would not allow them to make a bank transfer of a very specific value: R$17.99 (about $3.50 USD). What happened? The bank automatically converted the transferred value to R$17.98.
The most curious thing is that transfers of other values worked perfectly. Well, almost all. People started discovering other specific cases where the problem occurred: R$32.23 or R$155.17.
The idea of this post is not to debate the cause of the problem, but to show a technique that can help teams detect similar problems.
The text includes some code in Elixir, but also includes illustrations to explain each concept. The goal is to make this content useful for both technical and non-technical people.
Would unit tests help? Well… not exactly.
Don’t get me wrong: I’m a fan of unit tests and I (almost) always practice TDD.
But a conventional test for a money transfer function would probably be something similar to the example below:
Isn’t it too specific?
Notice that the test where the transfer is possible focuses on only one case: a transfer of R$25.00. And in this case, the tests are a success:
But we can write a test for the R$17.99 case, right?
Sure! After knowing about the existence of the error we could include an additional test to ensure that bank transfers of R$17.99 can also be made:
Here we can put the famous TDD cycle (Test Driven Development) into action: Red, Green, Refactor.
This test fails (RED), and then we can fix the problem until the test passes (GREEN). And now, having the test as a safety net we can alter the code to make it more readable without fear of breaking anything (REFACTOR).
The test fails because the remaining balance was R$2.02 and not R$2.01
Done! Now when we make this test pass we’ll ensure that transfers of R$17.99 will work perfectly. But some questions remain:
-
Will the transfer of R$32.23 work?
-
What if there are other values that generate the same error?
-
What if there are values that generate a different error?
-
How can we prevent problems like this from reaching our customers?
And that’s where PBT comes in — Property Based Testing.
PBT: cleaning where conventional tests can’t reach
It’s quite simple to write the R$17.99 test after we know about the existence of the error.
What happens is that we write tests with cases we already have in mind, in a very linear and specific manner. But the test is so specific that it tests only one case. What if the problem continues to happen with transfers of other values?
In this case, the problem appeared only when trying to transfer R$6.05
But… how could we identify problematic values before even knowing about the problem? That’s where property-based tests come in.
Working with properties means not thinking about specific cases, but rather the characteristics of each variable involved in the test. In this case, we can say that:
-
The initial balance of whoever is going to transfer the money can be any value greater than zero.
-
The transfer value is greater than zero and less than or equal to the available balance.
To know if the transfer was made with the correct values, we just need to “check our work”, as we did in elementary school. :)
Transferred value + Remaining balance = Balance before transfer
When we define the rules this way, automated testing tools are able to validate several different cases. And many times these “different cases” end up including situations we never even imagined, and therefore would never write a conventional test about them.
Now, when running the tests we discover that there are other values that also generate the error:
Oops! If I have R$0.98 and try to transfer R$0.71 the problem also happens!
Ah! Notice that the testing library says: Counter example stored.
PBT libraries usually store the values that generated errors and use them again until the problem is fixed. That is, when the test passes it’s because the problem was solved, and not because the library selected only values for which the problem doesn’t occur.
Okay, but what does this have to do with agility?
Well, agility is the ability to change and generate results quickly.
It’s like being in a race car on a winding track: to take curves quickly and without losing much speed you’ll need a solid and well-built vehicle. Without it, you’re limited to two options:
-
Take the curve veeeery slowly to ensure the vehicle doesn’t break
-
Take the curve quickly and dismantle your vehicle
Example:
Imagine the bank decided to allow Bitcoin accounts. They would now have to work with a currency that has way more than two decimal places.
Now imagine the bank can have an application that:
a) Has no automated tests
b) Has some automated tests
c) Has conventional automated tests and some using PBT
In which scenario would the bank be able to have a functional product in production more quickly?
In summary: There is no agility without technical excellence.
Source Code and final considerations
I wrote the code to generate the error on purpose and thus illustrate how a certain technique could help solve a problem.
If you’re curious, the code used in this post is available at: https://github.com/mariomelo/post_pbt
