Sega Pocket Kingdom Focus Group
I had a chance to go down last Tuesday and try Sega's Pocket Kingdom for the N-Gage, the massively mobile multiplayer adventure game. I was asked not to say anything about the game, so I won't besides the fact that it works - there were a bunch of us in a room all playing in the same game via GPRS. Very cool to be a part of.
What I will talk about is the process itself. I emailed the company doing the focus group after I saw a note about the testing on All About Symbian and got a response back asking my age, and my experience with the N-Gage (which games I owned) and my experience with RPGs like EverQuest or FinalFantasy. I replied honestly though I figured I was too old, but I got a response back saying to come down to Burlingame (a suburb south of the Airport) at 6:30 p.m. And better yet, that they were going to pay me $50 for my troubles. Niice.
I showed up to a non-discreet building a bit late, and walked in to find a waiting area of a normal office filled with all variations of geeky males. Young, old, game-geeks, computer-geeks, geek-geeks, etc. It turns out I was right on time, and a group of us was herded into a board room where there were a bunch of N-Gage QDs ready for us to play. We were told that we were being recorded, and in fact, one wall was made of glass and we could hear shuffling behind it as the observers (who we came to find out are probably some of the developers or testers of the game) watched us play.
We were in there for almost two hours, though it didn't feel like it to those of us with games in hand, the people who were running the thing were obviously sick of us before we even started. They were uninformative to the extreme and pretty short with us as well. There was no "selling" of the application or company (which may have been intentional), no tutorial for the game and only meager instructions on a xeroxed page. It was pretty easy to tell that there was a level of frustration with us clueless geeks that was really high. And though it was obvious that none of us were here in the room for the $50, there were several cracks by the testers insinuating just that. Yes, we're not only cattle, but greedy cattle. Thanks.
After quite a while, we were finally asked to stop playing and asked to fill out a dead-simple form about our thoughts on the graphics, gameplay, etc. Then we were interviewed as a group about our thoughts in general. This is where I learned the most. Though our group was the middle of three (game players, n-gage players, non-gamers) we were still a pretty disperse group of different races, ages and backgrounds. All of us had incredible insight into the game: What made it work, what the problems were, how we could improve it at such a late stage in development, etc. Whether these guys paid any attention to use will be seen. I had some specific criticisms that I gave concerning the menu system, and will be very interested to see if they actually do something about it.
Jacob Nielson has a great article about testing, and how it only takes around 5 people or so to find the flaws in any web app (because they mostly repeat the same things, so after 5 people you've heard 90% of what you need to know). This is the first time I've been on this end of a focus group and I have to say it was enlightening. I did write an application for Coca-Cola back in the 1990s tracking focus group results (they do a *lot* of focus group testing) so I know a bit about the subject from the tester's angle, but actually sitting in a room being watched and giving commentary was very interesting. As long as you've got thick-enough skin, I'm sure you can learn a ton about making your product better.
Anyways, from now on I'm going to be really pushing focus groups for the products I work on.
-Russ