Rethink the way you approach testing in SAP CAP projects
Have you ever started out a project with a quickly growing code base and soon run into quality issues? In this article, we will take a look at our learnings from various SAP CAP projects. We investigated challenges for backend developers concerning testing in SAP CAP projects and in the following Lennart Seiffer, Software Engineer at sovanta, shares some of the solutions we defined.
So what may go wrong with the initial test implementation?
The awareness and priority for testing within a project team is often insufficient. But there are also technical constraints. So let me lay out three common challenges:
- Test files closely coupled to the files that made up the code
These files have a size of a couple of thousand lines of code. - Large size of the tests due to redundancy
Test cases are often very similar, but change only small parts of the input to increase the code coverage. Only REST calls to the backend are used for testing, so in a strict sense, there are no unit tests, but rather integration tests. In general, there is often no overall plan for how to cover test cases. - Relatively high line coverage at first, but the tests rarely actually catch any bugs
This is because the assertions do not sufficiently check the results.
Test suite design focused on readability and adaptability
How to combine various test suites to achieve high code and semantic coverage? For this, I would like to use an image: imagine you have a lake (your code base) and you want to catch every single fish in that lake (this might be lines of code or bugs). An effective approach is to use nets to catch large amounts at first, but the fewer fishes are left, the less effective the nets will become. So after you have caught about 90% of the fishes, you may continue fishing with rods or spears to catch the remaining ones. In our scenario, integration tests are the nets, and unit/robustness tests are the rods or spears. Trying to only use one method would result in much higher effort. In the following sections, I will explain the purpose of the test suites in more detail.
Integration tests
The primary goal of the integration tests is to verify the correct behavior of API endpoints from the domain’s perspective. They emphasize the semantic aspects of the system and consist of one or more API calls, and at the end, a series of assertions check the outcome. Note that the assertions concentrate on the aspects of the application that matter most to the caller; we avoid checking any technical details that are irrelevant in practice. The tests do not import any code or interact with it in any direct way; they only use API calls to trigger code. That has the following advantages:
- In one test, we usually check that multiple modules are working together correctly, by only focusing on the final outcome. With this, we can cover large portions of the code with just a few test cases.
- It makes designing test cases a lot easier, as we can imagine how the API is typically used, and thus we can cover cases that are closer to the real world, making it more likely that we catch bugs that are caused by semantic misunderstandings or integration issues.
- The test implementation is (apart from the interface of endpoints) not coupled to the implementation; therefore, changes to the code usually do not require a refactoring of the tests.
When you start to plan the integration test suite in the first place, you do not need to look at the code coverage at all, but rather think about how you could reasonably cover the most relevant functions of your application, leaving out unnecessary details. With this, you will already achieve a coverage of about 80–90%, while also having a very good test quality (meaning it semantically resembles the specification of the system and actually catches previously undetected bugs).
Unit tests
Having a set of tests that cover most of the system and having a code coverage report now that shows what is still open, giving you good insights about where you can continue with unit tests. You need to concentrate on functions that are close to the lowest level of the calling hierarchy of the system. That in turn means that these functions call no or few other functions, and you can test them well in isolation, without requiring mocking. These functions typically have many possible inputs and outputs, which will make integration tests very ineffective to fully cover them.
For unit tests, you should directly import the code into the test module, which couples them closely to the code. Therefore, you want to keep the amount of unit tests relatively low so that refactorings or code changes do not require major changes to the unit test suite. Typically you organize the tests into one file per module in the code and test many different input/output combinations. The main goal here is to verify for highly important low-level functions so that they also work in corner cases.
Robustness tests
Together with the unit tests, you now achieve a coverage of about 90–95%, but there is still room for improvement. When identifying the uncovered lines, you will recognize a common pattern. Much of the untested code is dealing with corner cases or invalid input. But in the end, that is still a relevant part of the system, especially in regard to stability and security, yet this is seldom intentionally tested. You therefore need to introduce a new suite: Robustness tests.
The main goal here is to intentionally try to break the system. This is primarily done by passing invalid or dangerous inputs to the API and seeing whether they are rejected correctly. You will be surprised by how many issues you will actually find here in the existing code, which leads to also improve the error handling and validation logic of the application while writing the tests.
Reviewing the code coverage
At this point you should have reached a coverage of about 95% across most of the code. This is enough to satisfy project owners. So do you want to stop here? While being fine with this coverage is tempting, there are good reasons to continue. The question is: why are some lines still not covered? These are some of the reasons we identified in different projects:
- Some inputs or special cases were not covered
- Unreachable or deprecated code
- Unevaluated branches
- Corner case error handling code
Now, there will always be cases that are not covered, and the goal should not be to reach 100% just to have satisfying numbers on your screen. Still, many of the points I listed above indicate quality issues. The uncovered lines actually help you to identify code that is never executed or even bugs, where some branches were not run, because of logic errors in the evaluation. What will follow is a big cleanup action, which significantly improves the quality of the code.
It should also be mentioned that in some cases, code is not executed in a test case, where it is supposed to run, but still the test does not fail. Here, we identified that sometimes the assertions in an integration test may not be checking a specific output. Once you add that, the test will fail, and you can now verify whether that branch gets executed or not. So this can also be a good opportunity to review the quality of your test.
Any questions?
The testing strategy needs to be tailored for the specific needs of the project, especially the architecture of the application and the data model. With a mix of integration, unit and robustness tests, you will achieve a high code coverage and significantly better semantic checks compared to any other test suite. If you would like to know more about testing in CAP projects, please contact us.
