Behavioral Prototype for a Voice-Operated Extension
Testing usability and desirability of a voice operated extension
"Wizard of Oz" Technique
November 7 2023 – November 18 2023
Team Project for HCDE 451
By Andrew Christensen, Andrew Kwon, Chris Yamasaki, and Lushan Wang
The content of this process log was collaboratively created by the team.

Context
"Plants vs Zombies" is a gridlock game where players either use finger on tablets or mouse/trackpad on computers to operate the game. Our team wondered if voice control "Plants vs Zombies" is a usable and/or desirable way of playing the game, so we decided to use the "Wizard of Oz" prototyping technique to address the following questions for a voice operated extension that allows users to voice operate games like "Plants vs Zombies".
1. Is a voice-operate extension usable for playing "Plants vs Zombie"?
2. Would users want to use a voice operated extension to play "Plants vs Zombies? Why?
3. What commands would users use to voice control "Plants and Zombies"?
Plan
Design Concept
Voice operated extension that allows vocal control of gridlock games such as Plants vs Zombies
Task
Complete level 4 of Plants vs Zombies
Wizard Strategy
We will present the user with a screen displaying a game and given instructions to control it with their voice, the wizard will be somewhere in the room as a “note taker” and will be screencasting the game to the screen from their personal device following the participants instructions to simulate control.
User Testing Plan
Participants will be in a room with a screen and a vocal assistant
Recorder will be filming
Scribe will write down specific notable behaviors and interactions
Facilitator will introduce the concept and prompt them to begin playing the game (maybe give some guidance for the first command)
As user calls out commands the Wizard will follow them
Facilitator will conduct the post test interview
Facilitator: Andrew Christensen
Recorder: Andrew Kwon
Scribe: Lushan Wang
Wizard: Chris Yamasaki
Success Criteria
Usability: Users command words/phrases fell into our list of command phrases
Desirability: Participants want to use/enjoyed using the system
Data and Documentation
We will collect data on the words and phrases participants use when trying to control the game, as well as qualitative data through a post-test interview. Command words will be compared to a list of predicted command words to determine usability. Interviews will be coded and analyzed to determine user opinions of the system and feasibility.
Prototype
Setup
For this prototype, we decided to test the wizard of oz technique on the game interface, Plants vs. Zombies. The prototype was actually the most simple part of our project. Each participant was given the task of completing a given level in the game. We wanted to test and see how the participants would use voice commands to interact with the game “AI” (The AI was Chris controlling the interface behind the scenes) and from this we could better understand what the logic behind the voice analysis for the game interaction was.
The game
For this iteration, we chose level 4 as the standard game mode that we wanted our participants to complete. In this level, there are 5 rows and 9 columns. The participants have the ability to select from 4 different kinds of game pieces: sunflower, pea shooter, walnut, and cherry bomb.
Each piece has a different functionality:
Sunflower: produces sun which is the game currency
Pea shooter: shoots objects which kills the oncoming zombies
Walnut: is meant to be used as a wall barrier taking zombie a long time to destroy the object
Cherry bomb: implodes within 2 seconds of placing on a tile and does a massive area of effect damage
Objective
With these game pieces, the objective of the game is to kill all the oncoming zombies before the zombies can reach the house and the game is considered lost if the zombies are able to reach the house and the game is considered won if all zombies are killed by the end of the allotted time.
Participants
We recruited 3 participants and told them that the team has been developing a voice operated extension and were finding people to test out the product
The room setup and each person's responsibility
Participants were invited to a room with a screen and a vocal assistant
Facilitator: Introduced the concept, prompted participants to begin playing the game, and conducted the post-test interview
Recorder: Filming
Scribe: Wrote down specific notable behaviors and interactions


Screenshot from Plants vs Zombies
We had the participants played Level 4 of the game.
Screenshot from Plants vs Zombies level 4
Plants in level 4:
Sunflower: produces sun which is the game currency
Pea shooter: shoots objects which kills the oncoming zombies
Walnut: is meant to be used as a wall barrier taking zombie a long time to destroy the object
Cherry bomb: implodes within 2 seconds of placing on a tile and does a massive area of effect damage

Video Demo From One testing Session
Analysis
Feedback from Critique
What worked well?
Usability of voice control: All of our participants were able to use the voice-operated commands to defeat some or all of the zombies. Two out of three participants were able to successfully complete level four of Plants vs Zombies.
Video editing: The video clearly communicates the features we were testing — we fast-forwarded parts of the video and only kept the key interactions to its original speed.
What could be improved?
More structured start: In our testing sessions, we didn't give participants instructions on the specific commands of how to get started or play the game because we wanted to gather data about the intuitive commands words they would use to voice control the game. However, we received some feedback about giving a more structured start by giving a little bit more information or instruction on voice operate the game.
Labeling grids: One thing that jumped out was that participants were slowed down in the game because they spent a decent amount of time counting the grids. One improvement that we might consider is to label the grids such as A3, C6 to decrease the mental load for participants.
Game choice: Since plants vs. Zombies is a time limited game, and the time was tight for participants to figure out where to place the plants and how to call out the commands. If we were to test voice-operating a game in the future, we might want to choose a different game with less time pressure.
Data Analysis
The audio recordings of the three sessions were transcribed to interpret the data
Predicted vs. Actual Controls
Across our three participants we had slight variations in voice control and very consistent interview results. We only had two predicted voice controls for placing plants:
Place [plant] at [location] (Ex. Second from the top, two over)
[Plant] x row, y column
Across all three participants there were six command formats used:
[Plant] at [y] [x]
[Plant] behind [plant]
[Plant] at zombie
[Plant] [y] row [x] column
[Plant] [y] row from the bottom [x] column
One reason for this variety is that P1 seemed to be trying multiple command structures just to see if the system would work with them. P2 on the other hand picked one command structure ([Plant] [y] row (no x given)) and used it almost exclusively. P3 primarily leaned on “[Plant] [y] row from the bottom [x] column” with occasional variation. Because most commands followed one of our two predicted commands we believe we partially met the usability success criteria.
Post-task Interview
Interview results were almost the exact same across all three participants with three major sentiments coming up. First, participants expressed that they thought the concept and execution of our product was neat implying that they believed the system was real, “Definitely really interesting, nothing I've used before”(P2). With the exception of P3 who specifically stated they thought someone was controlling the device, but thought it was cool anyway. Second, participants found voice control inconvenient, specifically mentioning the effort required to think of a move, figure out how to explain it either through rough terms or using a grid, and then say their command. Participants and peers mentioned that this process would be significantly easier with some sort of grid system, “Something I would recommend is if it's going to be a vocalized game, you need to have a grid that says what each of the spaces are so that someone can input quickly.” We had considered this when designing our prototype but decided against it as we thought it would limit the variety of commands participants would use. Third, the concept was not desirable. All participants indicated that they would not use a voice control system for Plants vs Zombies citing the system's inconvenience. Some participants proposed potential use cases, perhaps if someone were unable to use their hands to control the game. But our success criteria was broader desirability, which we failed to achieve.
Result
Despite not meeting our success criteria we were able to get valuable feedback, both from our experiment and from peers. The biggest takeaway is that for voice control interfaces people need some sort of reference to be able to quickly give specific commands and reduce cognitive load.
Reflection
By planning, prototyping, and executing a behavioral prototype using Wizard of Oz technique, the team explored the usability and desirability of voice-controlling Plants and Zombies. The voice operated extension was usable because participants were able to actually use voice commands to place plants and defeat some or all of the zombies. However, the interaction was not necessarily desirable because voice control is not only slower than just using fingers or mouse, but also adds mental load for the participants to count the grid. Therefore, we concluded that using voice command might not be the best choice for playing Plants vs Zombies. Although the conclusion was less ideal, our prototyping experience is still valuable because we reached the conclusion that a voice operated extension is not desirable without actually spending money building the whole system.
© 2024 Lushan Wang Portfolio. All rights reserved.

