Behavioral Prototype for a Voice-Operated Extension

Testing usability and desirability of a voice operated extension

"Wizard of Oz" Technique


November 7 2023 – November 18 2023


Team Project for HCDE 451


By Andrew Christensen, Andrew Kwon, Chris Yamasaki, and Lushan Wang


The content of this process log was collaboratively created by the team.

Context

"Plants vs Zombies" is a gridlock game where players either use finger on tablets or mouse/trackpad on computers to operate the game. Our team wondered if voice control "Plants vs Zombies" is a usable and/or desirable way of playing the game, so we decided to use the "Wizard of Oz" prototyping technique to address the following questions for a voice operated extension that allows users to voice operate games like "Plants vs Zombies".

1. Is a voice-operate extension usable for playing "Plants vs Zombie"?
2. Would users want to use a voice operated extension to play "Plants vs Zombies? Why?
3. What commands would users use to voice control "Plants and Zombies"?

Plan

Design Concept

Voice operated extension that allows vocal control of gridlock games such as Plants vs Zombies

Task

Complete level 4 of Plants vs Zombies

Wizard Strategy

We will present the user with a screen displaying a game and given instructions to control it with their voice, the wizard will be somewhere in the room as a “note taker” and will be screencasting the game to the screen from their personal device following the participants instructions to simulate control.

User Testing Plan

Participants will be in a room with a screen and a vocal assistant

Recorder will be filming 

Scribe will write down specific notable behaviors and interactions 

Facilitator will introduce the concept and prompt them to begin playing the game (maybe give some guidance for the first command) 

As user calls out commands the Wizard will follow them 

Facilitator will conduct the post test interview

Facilitator: Andrew Christensen
Recorder: Andrew Kwon

Scribe: Lushan Wang

Wizard: Chris Yamasaki

Success Criteria

Usability: Users command words/phrases fell into our list of command phrases

Desirability: Participants want to use/enjoyed using the system

Data and Documentation

We will collect data on the words and phrases participants use when trying to control the game, as well as qualitative data through a post-test interview. Command words will be compared to a list of predicted command words to determine usability. Interviews will be coded and analyzed to determine user opinions of the system and feasibility.

Prototype

Setup

For this prototype, we decided to test the wizard of oz technique on the game interface, Plants vs. Zombies. The prototype was actually the most simple part of our project. Each participant was given the task of completing a given level in the game. We wanted to test and see how the participants would use voice commands to interact with the game “AI” (The AI was Chris controlling the interface behind the scenes) and from this we could better understand what the logic behind the voice analysis for the game interaction was.

The game

For this iteration, we chose level 4 as the standard game mode that we wanted our participants to complete. In this level, there are 5 rows and 9 columns. The participants have the ability to select from 4 different kinds of game pieces: sunflower, pea shooter, walnut, and cherry bomb.

Each piece has a different functionality:
Sunflower: produces sun which is the game currency
Pea shooter: shoots objects which kills the oncoming zombies
Walnut: is meant to be used as a wall barrier taking zombie a long time to destroy the object
Cherry bomb: implodes within 2 seconds of placing on a tile and does a massive area of effect damage

Objective

With these game pieces, the objective of the game is to kill all the oncoming zombies before the zombies can reach the house and the game is considered lost if the zombies are able to reach the house and the game is considered won if all zombies are killed by the end of the allotted time.

Participants

We recruited 3 participants and told them that the team has been developing a voice operated extension and were finding people to test out the product

The room setup and each person's responsibility


Participants were invited to a room with a screen and a vocal assistant


Facilitator: Introduced the concept, prompted participants to begin playing the game, and conducted the post-test interview
Recorder: Filming 

Scribe: Wrote down specific notable behaviors and interactions

Screenshot from Plants vs Zombies


We had the participants played Level 4 of the game.

Screenshot from Plants vs Zombies level 4


Plants in level 4:

Sunflower: produces sun which is the game currency

Pea shooter: shoots objects which kills the oncoming zombies

Walnut: is meant to be used as a wall barrier taking zombie a long time to destroy the object

Cherry bomb: implodes within 2 seconds of placing on a tile and does a massive area of effect damage

Video Demo From One testing Session

One of our testing sessions was recorded.

Some of the key user interactions are showcased in the video

Analysis

Feedback from Critique

What worked well?

  1. Usability of voice control: All of our participants were able to use the voice-operated commands to defeat some or all of the zombies. Two out of three participants were able to successfully complete level four of Plants vs Zombies.

  2. Video editing: The video clearly communicates the features we were testing — we fast-forwarded parts of the video and only kept the key interactions to its original speed.

 

What could be improved?

  1. More structured start: In our testing sessions, we didn't give participants instructions on the specific commands of how to get started or play the game because we wanted to gather data about the intuitive commands words they would use to voice control the game. However, we received some feedback about giving a more structured start by giving a little bit more information or instruction on voice operate the game.

  2. Labeling grids: One thing that jumped out was that participants were slowed down in the game because they spent a decent amount of time counting the grids. One improvement that we might consider is to label the grids such as A3, C6 to decrease the mental load for participants.

  3. Game choice: Since plants vs. Zombies is a time limited game, and the time was tight for participants to figure out where to place the plants and how to call out the commands. If we were to test voice-operating a game in the future, we might want to choose a different game with less time pressure.

Data Analysis

The audio recordings of the three sessions were transcribed to interpret the data

Predicted vs. Actual Controls

Across our three participants we had slight variations in voice control and very consistent interview results. We only had two predicted voice controls for placing plants:

  • Place [plant] at [location] (Ex. Second from the top, two over) 

  • [Plant] x row,  y column 


Across all three participants there were six command formats used:

  • [Plant] at [y] [x]

  • [Plant] behind [plant] 

  • [Plant] at zombie

  • [Plant] [y] row [x] column

  • [Plant] [y] row from the bottom [x] column


One reason for this variety is that P1 seemed to be trying multiple command structures just to see if the system would work with them. P2 on the other hand picked one command structure ([Plant] [y] row (no x given)) and used it almost exclusively. P3 primarily leaned on “[Plant] [y] row from the bottom [x] column” with occasional variation. Because most commands followed one of our two predicted commands we believe we partially met the usability success criteria.

Post-task Interview

Interview results were almost the exact same across all three participants with three major sentiments coming up. First, participants expressed that they thought the concept and execution of our product was neat implying that they believed the system was real, “Definitely really interesting, nothing I've used before”(P2). With the exception of P3 who specifically stated they thought someone was controlling the device, but thought it was cool anyway. Second, participants found voice control inconvenient, specifically mentioning the effort required to think of a move, figure out how to explain it either through rough terms or using a grid, and then say their command. Participants and peers mentioned that this process would be significantly easier with some sort of grid system, “Something I would recommend is if it's going to be a vocalized game, you need to have a grid that says what each of the spaces are so that someone can input quickly.” We had considered this when designing our prototype but decided against it as we thought it would limit the variety of commands participants would use. Third, the concept was not desirable. All participants indicated that they would not use a voice control system for Plants vs Zombies citing the system's inconvenience. Some participants proposed potential use cases, perhaps if someone were unable to use their hands to control the game. But our success criteria was broader desirability, which we failed to achieve.

Result

Despite not meeting our success criteria we were able to get valuable feedback, both from our experiment and from peers. The biggest takeaway is that for voice control interfaces people need some sort of reference to be able to quickly give specific commands and reduce cognitive load.

Reflection

By planning, prototyping, and executing a behavioral prototype using Wizard of Oz technique, the team explored the usability and desirability of voice-controlling Plants and Zombies. The voice operated extension was usable because participants were able to actually use voice commands to place plants and defeat some or all of the zombies. However, the interaction was not necessarily desirable because voice control is not only slower than just using fingers or mouse, but also adds mental load for the participants to count the grid. Therefore, we concluded that using voice command might not be the best choice for playing Plants vs Zombies. Although the conclusion was less ideal, our prototyping experience is still valuable because we reached the conclusion that a voice operated extension is not desirable without actually spending money building the whole system.

© 2024 Lushan Wang Portfolio. All rights reserved.

Create a free website with Framer, the website builder loved by startups, designers and agencies.