Unlike Research Methods in Human-Computer Interaction recommends, I wanted anyone to be able to respond to this survey. This report to is to evaluate the ease in which anyone, not just an experienced game player, would find using the new control scheme. A non-hueristic wider sample group than my immediate peers was needed, so twenty-two subjects were chosen from the staff and customers of a local restaurant. Representing an age range of 18 - 50, from part time bar staff, to office staff and management. A good representation of different game playing experience was found, from zero, to console, pc, and mobile, with which I hope to draw some meaningful conclusions about the validity of the new control system, in effectiveness, and ease of use.
Given the two-player nature of the game, subjects were paired off, observed playing, and interviewed afterwards about their previous game playing experience, opinions of the controls, and preference of crosshair or the new Touch's Actual Position (T.A.P.) firing method. In addition, given the easily distracting 'after work' testing environment, the perceived interest in the test was recorded as well.
Whilst mirroring the target playing scenario and players, alcohol and tiredness after potentially a long shift in a busy restaurant introduces an element of unpredictability and distraction. Attempting to force a player to engage whilst their concentration is on a different conversation topic, or an unfinished game of pool nearby, could potentially result in inaccurate data. To account for this, I have attempted to quantitatively judge each subject's engagement with the play test. If needed, this can then be factored into indeterminate results to perhaps provide some clarity. This is especially important in a mobile game where attention to the screen is paramount, and as with any game, players will only engage if they are in the mood to play.
This social situ is also highly prone to positivity. Interviewing any colleague or friend as opposed to a blinded trial may introduce a politeness bias. With the two-player aspect often acting as a joint interview, the subjects may also be prone to agree with each other skewing their own results. By triangulating the results from my expert assessment (as the developer) of their ease of use of the controls, to their own assessment or preferences, then the correct conclusion should be supported.
Alcohol, even in a minute amount, will potentially tar any results gained as inaccurate, so I have chosen to limit my testing to subjects who have consumed less than three units. If a subject can be legally trusted to operate a motor vehicle, then I will choose to trust their ability to play this game.
The tests began by introducing them to the control instructions screen within the options menu, and talking them through it to ensure that they understood. After all, this is not an assessment of the quality of the instructions screen. They were then quizzed on their approval of the concept. This surprisingly returned unanimous positivity. I pessamistically interprete this as confirmation of a politeness bias. Afterwards they were free to navigate through the game options and play as they wished. Whether at this point they chose to compare different gun stats or maps, was a very good indication of their engagement with the test. More uninterested participants tended to rush through this, with a focus I assume to consume more than three units and return to their unfinished pool game. Whilst playing, I assessed from an expert's perspective both subject's grasp of the controls and ease of use. After playing, they were requested to play again with the alternative centre crosshair firing only, subsequently revealing their preference, and most importantly, the details of their experience of playing video games. Dissecting into console or P.C. (to determine twin stick joystick movement experience), first person shooter (FPS) or not, and lastly whether they play games on a touch screen mobile device, and if so, which?
Strong correlation as described by Jacob Cohen - A Power Primer is apparent when the correlation coefficient r > 0.50. This rating will form the basis of any conclusions made.
As predicted, the more a subject had experience playing video games, strongly positively correlated with the ease with which they could apparently control their character in the game with a correlation coefficient of 0.722. heteroskedasticity in the lower quartile of experience could be due to a naive willingness to accept the first FPS controls that they have ever seen, or a pre existing comfort with touch screen device navigation from their own phones or tablets. Perhaps the quantitative video game experiance question should related to the subject's experience with touch screen interfaces. There is always the possibility that they are under estimating their previous experience - perhaps they simply haven't played recently, thereby discounting some level of skill.
As my assumptions continue, the correlation begins to become less significant. Participants who had large amounts of experience playing FPS games only performed marginally better than those who did not. With only extreme values yielding any outlying results and a correlation coefficient of 0.474 the results are inconclusive. Although interestingly, the same larger variance towards the lower end of the scale indicates a threat to validity, or that a lack of previous experience may leave the player more open to the idea of a new control scheme.
Considering only those with experience of playing video games, of those only who had played FPS games to some extent, the proposal breaks down. There was no significant variance between those experienced in moving around an FPS game with two joysticks on a console, and those with a keyboard and mouse. This is both surprising and relieving. Surprising, because when designing the game, we based the controls heavily upon the twin joysticks of a console controller, and relieving, because we no longer fear that in by doing so we have cornered our market to a small niche.
Reflecting on this, the argument could be made that the sample size after this culling is too small to make any meaningful conclusions. Having this survey be applicable to participants of all gaming standards has resulted in a drastic breakdown of sample size as certain parameters are focussed upon. Broadly, the initial proposal is still valid, and the prerequisite of a wide audience stands. But were a more particular proposal be made, for example experienced FPS mobile gamers will accept these new controls with more difficulty than non-FPS mobile gamers, then those particular subsets of mobile users must be specifically surveyed. In this data set only four out of twenty-two played games on a mobile device, none of which were FPS.
Given the unconventional setting, a great variance in interest was shown. Whilst not apparently strongly correlated with the ease of use of the controls (correlation coefficient of 0.307), by factoring in the previous experience of the player, a much stronger trend can be observed. This indicates that even lesser experienced gamers can effectively use the controls were they properly engaged.
Although the data appears to be more tightly grouped, the standard deviation is 30.58 over a range of 0 - 100 factoring in apparent engagement, compared to a standard deviation of 2.75 over a range of 0 - 10 purely considering the ease of use. This may indicate a failing in my interpretation of either the subjects grasp of the controls, a failing in my interpretation of their interest in the play test, or both.
As the examiner, I found it very hard at times to equally and fairly observe both players at the same time. Especially given the inverted split screen. When observing two players from two viewpoints, two examiners should have been employed. Of course, both expert examiners should be of the same disposition, skill, and opinion as each other, so I would propose future testing be conducted with the aid of another gameplay programmer on the development team. Alternatively, with a single examiner, only one subject be observed at a time.
All players were observed to struggle with the control's sensitivity. The option to set it per player is within the main menu options, and given the observed and qualitatively described problems; over sensitive on a phone, and under sensitive on a tablet, as the development team we must prioritise creating an in-game way to adjust this. Most likely through a mid-game pause screen. Perhaps, this indicates a larger problem with the turning and moving mechanics? A redesign of the algorithms may be necessary, or less efficiently, a platform specific profile per machine. Movement speeds are an extremely personal detail per player, with console players investing in expensive modified controllers, often costing more than the game itself, and modern PC gamers measuring their turning circle to specific dots per inch. When the second player in Battle Screens is often challenged on a whim, using the first player's device to play upon, the settings to alter the controls must be easily and quickly accessible.
Alan Bryman wrote 'Triangulation refers to the traditional view that quantitative and qualitative research might be combined to triangulate findings in order that they may be mutually corroborated.'
However, I choose to use it as a method to discredit the politeness bias that was anticipated. Of the twenty two subjects surveyed, all twenty two gave positive approval of the concept, but only eighteen subsequently preferred the T.A.P. method of firing over the crosshair. Still remarkably high, but indicative of some underlying opinion against the high concept. With a mean perceived interest of 7.55, with a standard deviation of 2.32, situational distractions did not prove as much of a hazard as I had predicted, and had no apparent effect observing individual cases relating to qualitative assessments of the user interface (U.I.) minimalism, with an interest range of 3 - 10 in both U.I. complete approval, and with those recommending a better joystick system.
Nearly everyone who complained about oversensitive turning wanted a better virtual joystick option. But, I believe those to be linked, and a subsequent improvement on the sensitivity algorithms and options to change, will result in an overall improvement in the qualitative player assessments. This of course will need to be tested.
Whilst the data strongly supports the initial proposals, more data regarding the subset is needed to support more particular prepositions. Previous experience playing games helps the ease of controlling the player in the game, but not necessarily experience of FPS games. I propose that with a touch screen game, any previous touch screen interface experience will help a new player adapt to the new control system, indicated by the greater range of results in the lower experienced portion of the sample group. This appears to be more significant than any previous experience with a twin joystick method of moving, shown by the relative similarity between console and P.C. gamer participants.
Politeness bias is a large factor in qualitative and quantitative survey data when the test is not blinded. Only by triangulating results to discover the biased outliers can some results be omitted, or in this case, entire sections such as high concept approval, be ignored.
Whilst gathering the data for this report in a social setting helped to see the game being played as it would be by the target audience, distracting factors were unavoidable, and attempting to account for this is debatable in its effectiveness. Mainly by further overloading the examiner with another quantitative assessment to make, whilst simultaneously trying to assess two separate players. One screen of which is inverted.
As the developer, it can be very hard to see the controls from the view of a first time player. This usability report has been invaluable in not only investigating the proposals set out in the introduction, but also to identify the user issues with sensitivity. A key priority for us now is to create an in game pause screen with the option to alter each player's control sensitivity, and also over a larger range than previously possible.
After working together as a student team for a group project at Falmouth University we continued to develop together as a summer project. The limitations of mobile platforms, and the unique control options have always fascinated us, and we believe the final project to be a great representation of what is now possible using Unreal engine for mobile devices.