How-To Guide to Crowdtesting

Written by Sahil Deva

Thursday, 11 August 2016

Crowdsourced testing is an innovative approach to testing that utilises a crowd that represents a digital product’s target audience to perform testing. It can be applied to both waterfall and agile testing methods but differs from traditional methods. This guide from Testbirds discusses how integrate it into the development cycle with ten key points.

As the world of technology continues to integrate with every aspect of day to day life, it's imperative that comprehensive and extensive testing of software and devices takes place. However, due to resource and time restrictions, organisational blindness, platform fragmentation among many other issues, testing often becomes an afterthought to the development process.

crowdtestingsq

Crowdsourced testing (crowdtesting), an original approach to testing that utilises a crowd that represents a digital product’s target audience to perform testing is able to address many of these issues has become an established solution as a complementary service to traditional testing methods in virtually all industries.

Its perks vary from being able to produce quick results and around the clock availability of testers to access to virtually every device and operating system imaginable. Compared to laboratory testing, crowdtesting tends to be more cost efficient and thanks to a diverse, global crowd, websites, apps and all manner of technology can function and be relevant on an international basis.

However, to benefit from its many advantages, the characteristics of the method need to be taken into consideration. In many ways, crowdtesting differs from traditional methods when it comes to checking functionality and usability of digital products. The 10 factors presented are a must to successfully perform crowdtesting and integrate it into the development process. They come from Testbirds, a company that specialises in the testing of software such as apps, websites and Internet of Things applications by using innovative technologies and methods.

testbirdsbanner

1. Employer Side Responsibility

Testing is an essential part of the development lifecycle and crowdtesting can be applied to both waterfall and agile testing methods. Therefore, it’s a must that the method is embedded into the overall strategy in a way that works on an individual basis. When working with a crowd, it’s vital that tasks are clearly outlined and crowdsourced workers have a strong grasp on the responsibilities that are expected of them. This is especially true when it comes to crowdtesting due to the fact that the crowd is often located internationally and communicated with remotely. Similarly, it’s important that the entire process is well organised and thought through in detail before testing begins. Who oversees the testers and who creates the test design? Is it the service provider or the company itself? Who implements the results of the testing process and responds to questions in case the software doesn’t work after fixes are applied?

Solution: Including crowdtesting into early planning allows it to become a fixed part of the development process. If responsibilities are clearly defined early on and the processes are well-established, service providers, clients and the crowd will all be able to have an efficient and successful testing experience.

2. Realistic Planning and the Right Testing Method

When developing technology today, time pressures are extremely high as companies attempt to deliver their digital products to the market as quickly as possible. Due to delays or a tough schedule, there’s often not enough time for testing. Even though the crowd works quickly and are extremely flexible, it’s still important that the testing phase is planned realistically and the right testing method is chosen based on the individual necessities of the digital product in question. It’s equally important that time is set aside for fixing issues found during testing. It doesn’t make sense to find problems if there is no time to fix them.

Solution: Depending on the digital product in question, it could make sense to test single components or functions before the product is ready. For example, using the crowd to test prototypes or mock-ups allows problems to be fixed before they arise. This in turn allows the development process to stick to tough time schedules despite the inclusion of extensive testing.

3. Careful Preparation

Planning is an absolute must no matter what testing method is used. With crowdsourced testing it’s especially important as testers work remotely, meaning that they are unable to receive feedback immediately. Before testing it’s important that the goal is kept in mind, questionnaires with detailed instructions on what to do are designed and the steps that will guide testers through the process are well defined. It’s important that the people designing the test try to visualise the problems that might arise or aspects that might be unclear to testers when setting them up. Asking the right questions means getting the right results.

Solution: Testing in a laboratory environment and remotely are two different things, therefore the requirements of those conducting the tests are also different. Due to the specific needs of remote testing, working with people who know how to set up clear instructions and have experience in how the testers will act and carry them out is a must.

4. Best Possible Conditions

In order to complete their tasks quickly and efficiently, it is a must that a high functioning and userfriendly online platform and infrastructure is available for testing. It’s equally important to keep in mind the interests of all stakeholders involved. Besides flexibility, when it comes to different testing methods, the platform should also feature helpful aspects such as bug exporting tools, tracking tools, and direct communication channels between testers and the managers of the test. It’s important that these factors are taken into consideration when choosing a service provider.

Solution: Continuous servicing and development are essential aspects of a productive and successful infrastructure. Regularly gathering feedback from stakeholders grants a better understanding of what is needed or missing. The rule of thumb here is to never stop improving.

5. Reliable Testing Environment

Testers need easy and reliable access to test, no matter the kind of digital product that is being tested (beta apps, live online shops, connected devices, etc.) Applications have to be stable enough to be tested. It’s difficult to test when updates are running as this affects the process and leads to unreliable results. Furthermore, it’s important to document which version has been tested and on which one bugs have occurred. For unreleased software, security issues have to be considered due to confidentiality, therefore a safe method of access needs to be provided.

Solution: A VPN for secure and easy access is a must during the testing or staging environment. For fast beta app distribution, online cloud based tools are often a great solution.

6. Entry Tests and Training

In the lab, test managers can help testers who encounter problems. Remotely, however, testers have to partly take over the tasks of the managers themselves. In essence, they have more responsibility. For this reason, it’s vital to train people appropriately and make them aware of requirements. It’s equally important to show them how best to use the platform. Entry tests can ensure that testers know how to handle their devices and the testing platform, which in turn can yield to results of a higher quality.

Solution:Tutorials and videos that show how the platform works and how to report bugs can ensure that testers are well prepared to handle the responsibilities expected of them.

7. Choose the Right Testers

The crowd is a reflection of society. It can represent a number of age groups, jobs, education levels, family members among many more demographics. These are often normal people without extensive testing knowledge or expertise. In addition, what is required of them differs from project to project. To select the right people you need as much information about them as possible. For usability testing, people that represent the target group of the digital product that is to be tested is a must as in the end, they will be the ones to use and review it. For functionality testing, this isn’t as big of an issue, instead it’s important that testers are experienced and have the necessary devices.

Solution: Inexperienced testers search for bugs and usability issues in a different manner than testing experts. When hunting for bugs or figuring out shortcomings in user experience, it’s important to have a group of testers with varying degrees of experience.

8. How to Handle Further Enquiries

Test managers are unable to provide direct feedback when remote testing. Therefore, it’s even more important that the feedback they do provide is perfect. It’s important to try to solve possible technical issues that testers may encounter with strong and clear instructions. When there are issues it’s vital that test managers react quickly to get high quality results. Checking the feedback from testers carefully and changing instructions based on misunderstandings is also a must.

Solution: Always provide testers with a personal contact and have their information readily available. Communicate the availability of this person in case testers need to ask something. A forum for communication between the testers themselves can also be helpful and allows them to fix issues on their own.

9. Detailed Documentation and Quality Assurance

One of the critical aspects of successful remote testing is comprehensive documentation. Testers have to report the steps they take in a detailed and precise manner. Every step needs to be documented and supported with evidence such as screenshots. Reports have to be reviewed in regards to how complete they are, if they are traceable, of a high quality and whether they are reproducible or not. Have the testers done what they were supposed to do? Did they test on the right devices and software? How important or critical is the bug in question?

Solution: Testers learn and develop with time and projects. Give feedback on what they should be aware of for the next project, what can be done better and clearly explain situations that may arise such as why a bug was not accepted.

10. Fair Payment and Incentives

When it comes to remote testing, testers are the key to success. Happy testers are good testers. It is therefore a must to provide adequate incentives and motivating tasks. Other than fair payment, a reward system can also be used to motivate people, based on, for example the quality of the bugs or gamification of the testing process. Additional payment for bugs found when performing functionality testing can lead to an increase in creativity and engagement when hunting for bugs.

Solution: Experience points for the participation of tests helps to qualify testers and can also serve as motivation for the testers themselves. Rankings and awards for extraordinary achievements can also be used to further motivate the crowd.

crowdtestingsq

Sahil Deva is part of Testbirds UK's team that, under the slogan “Testing 4.0 – The Next Generation of Quality”, is pioneering innovative testing solutions. With over 150,000 testers in 193 countries, Testbirds is one of the world’s leading crowdtesting providers. In addition, the company offers cloud based technologies for the optimization of digital products.