Testing of a Smartphone
By Antonio Montañez, Project Manager – Senior
In the last few months we have been working on one of the most peculiar and unusual projects I have ever seen since I became a translator. It is one of those jobs that you see as a challenge because it does not resemble anything you have done so far and, therefore, you do not have a clear idea on how to face it.
WHAT WAS THE TASK ABOUT?
Through one of our customers we received a request to perform the testing for the always-on voice feature of one of the latest mobile phones marketed in Spain by a well-known major manufacturer.
This feature benefits from the dedicated chipset built in the phone in order to be listening at all times, no matter if the device is locked or on standby. So, if we utter the trigger and then the corresponding command we will be able to call a contact, search Google and receive a spoken answer or just ask the device to read any notifications for us. All of this, even if the phone is somewhere else in the room.
This feature has also the added value that, once trained, it will only respond to the user’s voice and no one else’s.
The testing we were going to do was divided into three well-established phases. Of these, phase 3 was the most complex and also the one that required 90 % of the work.
Phase 1: Recording of sounds from daily life
First of all, four people (2 males/2 females) had to carry the device for approx. 52 hours while recording at all times. This was done in order to collect background noise from the different environments the phone would be exposed to on daily life (except for the sleep hours): lectures, workplace, supermarket, playgrounds and so on. Provided that recording was not forbidden in those places, of course. Once gathered, these sounds would be sent to the developer team.
Phase 2: Recording of commands
Then, other four people (2 males/2 females, again) had to read and record approx. 300 voice commands each and send them for analysis.
Phase 3: Live environment testing of the commands
At least we reach phase 3, where 20 people (10 males/10 females) had to test the voice recognition feature of the phone in several environments:
● a quiet environment (such as an office),
● a living room (with TV or radio on),
● a café (busy with background noise),
● and a car (with the engine both idle and running and placing the phone in several locations: phone holder and cup holder).
This cycle of tests would repeat itself several times in the months to come.
As mentioned before, in order to use this always-on voice feature, each person had to train the device to listen to his voice only. This was done repeating the same sentence three times in a very quiet environment. This usually would have to be done only once by a normal user, but while testing it had to be done before every participant would take part in the test. Once trained, and with the phone on standby, it was necessary to utter the trigger and then command, in the following manner
ok Google (trigger) + what’s new? (command)
ok Google + how tall is the Eiffel Tower?
ok Google + call Mike Fox
At first, only 4 commands were tested, in upcoming cycles it would be 6 of them. Every participant had to utter the same statement at least 10 times in a row in every environment while we logged on an Excel spreadsheet two types of data for every case:
● if the device had woken up to the trigger every time,
● if the device gave the appropriate answer to the command (or if it understood something else and reacted in an unexpected manner).
As soon as a participant finished testing in an environment, a bug report was collected by an application and sent over to the developers. This report contained the sound recorded as well as other relevant parameters.
WHY WAS IT A CHALLENGE?
Organising a testing with these requirements, with at least 20 people taking part, a tight deadline of 3/4 days and where any technical issue would cause data loss or waste of time (time is always of the essence, as you know) is no easy task.
Preparing the materials and setting up the environments
Once the devices had the latest software and application versions installed and we had learnt how to use the sound meters and other technical questions, it was time to think about the environments and how to recreate them.
For the quiet environment the company meeting room could be used but… what about the living room? The same room with a TV, a low table and a sofa should be enough. As for the cafeteria, it was important to use one located nearby to save time and petrol when moving between environments and also busy enough to have the right background chat or noise. Oh! And also Wi-Fi to save on data usage, since it was going to be rather intensive. Almost every command required googling something.
Searching for participants and setting up appointments
If we consider that our office is located in a haven of peace and tranquillity, far away from any big town, it was not easy to find participants that would come over for a reasonable price. On the other hand, for those staff members that took part in the testing it was difficult as well to find time and leave their computer monitors unattended for the hour and a half that the test lasted.
Once the appointments were set up depending on the availability of the participants, we established a planning for the week with 5 participants per day.
Solving unexpected problems
The materials were ready, the appointments set up, the car itinerary established and the cafeteria picked up. So it was time to start the testing. However, it is a well-known fact that things are never as easy as they seem. I am not saying that everything that could go wrong, went wrong (as Murphy would say), but we did face several unexpected issues on the go.
● Echo: When arranging the meeting room as a living room we had important issues because the device did not recognize the voice commands when the TV was on. This sort of recognition issues were expected in this environment, but the error rate was close to 100 %. The problem was that there were no curtains, carpets and other pieces of furniture that you would normally find in any living room and do muffle the sound. As the echo was louder than expected we had to adapt the room accordingly.
● Application crashes: With some of the builds we tested, the application would return an unknown error and stop working. Or suddenly the device did not recognize the SIM card. This forced us to reset the phone and restart the ongoing part of the testing.
● Data usage: Sometimes the application could not access Google at that moment, even with full signal strength. It took us a while to realise that we had used up all of our data plan. The reports we were sending over to the developers turned out to be very heavy (30 to 60Mb each), and sending 4 of them per participant incremented the data usage dramatically. The solution? We had to save every report as draft and send it over later on via Wi-Fi.
● Noise level and training the device: Due to unexpected circumstances, sometimes there was the need to train the device when in the cafeteria. The background noise for this device training should be almost non-existent and sometimes not even the toilets offered proper isolation. It was impossible to train the device in the cafeteria. Luckily enough, cars are isolated in a much better way and offered the silence required to train the phone and continue with the testing.
As mentioned above, taking on a project with these requirements can be a real headache at the beginning, but at the end of the day is a very rewarding experience. It always is when you are working in something totally new. And getting to know first-hand how this sort of applications are tested is so interesting if you are into new technologies as I am. Besides, it gives you the opportunity to leave the company of that computer monitor you spend so many hours in front of for a while. Even those unexpected issues are like small challenges themselves that give you a sense of accomplishing once solved.
And the feeling is even better when eventually you receive congratulations from the costumer on the good work done.
By the way, a new cycle of testing is due to start in a couple of weeks… Volunteers are welcome!