Testing is a part of the development process that is often left until an app is almost complete. I’ve found that setting up for testing at the start of a project has a number of benefits:
- It’s easier to start with a project that is set up this way than retrospectively adding tests and automation.
- It avoids excuses for not doing testing.
- You get the benefits right from the start.
Here are fourteen things I’ll set up in an app project, before any of the specific functionality is added, so that I can be confident of being able to easily develop and test the app that’s built. For each of the points that follow, what you use is less important than that you use something and so I’ve avoided listing specific tools.
- Version Control
It’s crucial to use some form of version control to store the code and assets used by the app. You need to be able to say where the latest version of code is so you know what to build and test.
- Continuous Integration (CI)
Once code has been written and committed or checked-in to the repository it should be automatically built. It’s not acceptable to rely on a person creating builds or needing a specific machine to be able to build the app in the correct way. By automatically building the app each time something is checked-in any issues that are not evident on the machine of the developer who made the change (such as not committing all parts of a change) are detected as quickly as possible.
- Create identifiable builds
When the CI server creates a build it should assign a unique identifier or version number to that build. Having unique identifiers is especially useful during testing when bugs and issues are being found. When a bug is reported it should include details of the version of the app it was found in. This helps in knowing if a bug still exists in different versions, has already been fixed, or is a regression. It also helps testers easily identify if they have the latest version. Having unique identifiers is a start, but it’s better if they are sequential, incorporate the date and time the version was built, and show the machine it was built on.
It is not enough to rely on someone manually assigning a version number to each build as this is easily automatable and easy for a person to forget or get wrong.
- CI run Unit tests
Unit tests should be run as part of the CI process. It’s not enough to know that code compiles before it can be shipped. Coded unit tests should be automatically run after each build to verify the internal logic and behavior of the code. Creating unit tests at this stage ensures that the code is architected in a way that makes it possible to create unit tests. It’s easy to create code dependencies that are hard to test but if a codebase is first architected in a way that ensures unit tests can be written and run it avoids the need to do any complex re-architecting in the future or anyone arguing that they’re not writing tests because the structure of the project doesn’t support it.
- CI to generate code coverage reports of testing
Knowing the code coverage of tests is useful in three circumstances.1) If it’s very low that’s probably a sign that there aren’t enough tests. 2) If it’s 100% it’s probably a sign that unnecessarily trivial parts of the code are being tested, which is probably a waste of time and effort. 3) Changes over time, particularly if the level of coverage is going down, can indicate if sufficient tests are being written.
Any CI server capable of running unit tests should also be able to provide code coverage information and for the sake of enabling this setting, there’s no reason not to do this.
- CI run static analysis of code
These tests involve just looking at the source code and not running it. These types of tests can find potential issues with the code and they can also identify inconsistencies within the code. Having a consistent style for arranging, structuring, and naming within the code can be the subject of much debate between developers but having consistency within the code makes it easier to support over the lifetime of the code and by enforcing this from the start makes it easier to ensure that standards are followed.
- CI run Integration tests
While unit tests test pieces of code in isolation, integration tests verify how different pieces of code work together. You should have both and they should all be run as part of every automated build. At this stage a simple, single test is enough as the aim is to ensure that a test can and will be automatically run and the results reported appropriately.
- CI initiated Automated (UI) tests
It’s probable that there will be some tests that can only be run on actual devices. This will include tests verifying how things are displayed on the screen or which rely on actual hardware. While such tests cannot be run on the CI server it should be possible to have the CI server connect to remote, third party services which can offer such testing, and then gather the results once available. During the setup phase this will contain a single test to launch the app and take a screenshot but that’s enough to ensure the test is run each time.
- CI initiated Performance tests
The speed at which at an app can be used is normally very important but it might seem odd to start to try and test this before there is any real functionality in the app. Unfortunately, it is far too common for performance considerations to be left until the app is thought to be nearly finished. When the app is nearly finished it can be hard to identify what is contributing to performance issues. I’ve found it better to track this right from the start of a project. At this stage it’s only possible to measure the time it takes to start the app and have it usable but this is often the most important metric. With changes in the startup time being monitored right from the beginning of a project it becomes clear when something is added to the project that impacts this and it helps keep performance front-of-mind and not become something that can automatically be put off.
- Automated distribution of builds
Having the CI server build and test the code is great but you cannot rely solely on automated testing for your app. You’ll need to have people test them too but that requires making sure they have the latest versions to test. Your CI process should output them to a known location where people can access them. Better still is integrating them with a tool/service that will automatically distribute and install new beta versions when available.
- Have the app prompt when a new version is available
Manual testers can be an expensive resource and so it’s important to ensure they are making the best use of their time. Having them test older versions of an app when newer versions are available is normally something to be avoided. Having the app indicate when new versions are available helps avoid this. It’s also a useful feature to have when the app is generally available too.
By having the automated distribution of builds to known locations should make this easy to add yourself.
- Add automated 3rd party analytics and exception tracking
Particularly during testing it’s important to know how the app is being used and that you get lots of information about any detected problems. Any exceptions should be automatically recorded and sent to a central location where you can access them. Use of the different parts of the app is also important to track. You need to be confident that all parts of the app are tested and monitor what beta testers do with the app. There are many companies who offer services like this. They also offer other functionality that may be useful or appropriate to your app but in the early stages basic usage and exception tracking should be enough and is usually available for free.
- Add own analytics, event, and exception tracking
In addition to the use of third party services I will also add the ability for the app to capture any exceptions and present the use with the option to email them to me. Doing this provides an opportunity to communicate with the person doing the testing and ask them for more information, about what they were doing when the error occurred as this isn’t always obvious from, or available in fully automated exception logs.
Add a button to the UI that will throw an unhandled exception so you have an easy way of testing that exception tracking works.
- Add manual feedback option into the app
The final element to add to an app from the start is a way for the person using it to send feedback. The simplest way to do this is to open a new email addressed to you and with details about the version of the app in use and the device it is on already added to the body of the email. I add this right from the start so that there is always a way for feedback to be gathered. It’s often an afterthought but, especially early on, I don’t want to miss out on potentially valuable ideas, insights, and suggestions. At a later stage the feedback option will probably be changed to something other than just email but that’s sufficient in the early stages.
On a good day the above can all be set up in a couple of hours. If you're doing it for the first time it might be worth allowing a day (or two) depending on your confidence and familiarity with each part. Retrofitting the above into an existing project and code base can involve considerable time and effort depending on what already exists. That’s why it’s best to do this at the start of a project. If you have a project that isn’t automated and easily testable this is technical debt that you’ll have to pay back eventually. It may be a daunting prospect but the longer you leave it the greater the cost.
By Matt Lacey,
Matt is a Software developer with 13+ year experience and specialising in Mobile and .NET technologies.