Last week, on June 11th, the new & improved version of Fireside made first contact with its players. This was an exciting moment for us as developers since it was the first time showing our game to any audience. Not everything went smoothly, as usual. Especially one of the core elements of gameplay (the trading system) didn't work well from a game design perspective. But overall we got positive feedback on many important parts of the game.

However, this is expected in game development and players did have some engaging moments when trading their goods. So, we're happy to say that we won't have to scrap anything completely based on Fireside's first (real) test. In this devlog, we want to share the way we organized our playtest and what results we got from testing.

Goals

First we needed to define some goals for our test. This may sound obvious, but we've made the mistake of not doing this before, and it makes a test way less efficient. One important decision is whether you want to do a broad or a narrow test. A broad test means testing many things at once to get a better idea about an overarching idea, while a narrow test means testing one very specific thing extensively. Testing the game's aesthetic or combat system might be considered a broad test, while deciding what color to make the health bar might be considered a narrow test.

As you might have guessed, the first test for Fireside was a broad test focusing on the game's tone and gameplay. This means we were looking for general statements about the engagement the players were experiencing through the game's mechanics (gameplay) as well as statements about the game's graphics, sound, writing, and coherence of said assets (tone).

In testing gameplay, we were mostly interested in the systemic aspects of the game. This means we weren't looking for small details like whether a particular part of an animation felt juicy, or some specific item was over- or underpowered. We simply wanted to see if the game's systems provided engagement for the players and the core game loop worked as intended, supporting the wholesome atmosphere we’re striving to create with Fireside. So we were closely watching players interact with the game to see if and when they were engaged.

Reading player engagement is a difficult skill to practice and based largely on intuition. It involves evaluating signals such as the player hesitating to interact, chuckling, sighing, the tone of their voice when criticizing or praising elements of the game, and more. Facial expression and body posture are a big part of signaling engagement, which gets lost when conducting remote tests.

Of course, feedback on any part of the game is still valuable! It's just not the prime type of feedback we were looking for at this stage.

The Setup

To manage the test, we decided to have two team members manage one tester. One person would talk to the tester and help them understand the game. A second person would be taking notes. This way, we not only were able to focus on the two most important tasks while testing (assisting & taking notes) but we were also able to gather impressions from two distinct perspectives in each test. Quietly observing how someone is playing is a very different perspective from engaging with a player during the test.

If possible, we tried to get the testers to play the game physically in our office (check out our TikTok) to make sure we could watch them as closely as possible. Of course, COVID and the locations of our testers made it impossible in many cases. So, for most tests we simply used our Discord server as a platform.

Each test lasted about 30 minutes and we made sure no team member was assigned to more than three consecutive tests, since it can be very hard to stay focused for longer. With this setup, we conducted twelve tests in total. After finishing, we evaluated the test protocols and had a team meeting where we discussed our impressions from the tests.

The Results

To get results from our test we first worked through the test protocols and counted occurrences of certain remarks. In total we gathered 115 unique compliments, suggestions, pains and observations our playtesters generated. That's a lot of data for us to improve our game, so thanks to everyone who contributed! Here you can see the most common remarks (mostly in German). The most frequent points were compliments about our music by the amazing Matthias Meeh, a concrete complaint on how gathering clickable objects works, and the suggestion to add tooltips to our items.

As you can see, none of these points are exactly what we were looking for, but they're still very valuable feedback. We will (of course) improve the way items are collected and add tooltips for items. However, we got the important insight we wanted mostly from observing players and by finding patterns in their remarks.

For example, five testers explicitly praised our asset's coherence and the game's tone, but it's also relevant that people who talked to NPCs, seemed engaged in the conversation, and added firewood to the campfire without hesitation while talking to NPCs. This dynamic is exactly what we're going for with the atmosphere of Fireside and seeing testers enjoy the gameplay gives us confidence that we're hitting the right tone with the game's aesthetics and methods of interaction. You can read more about Fireside's design in our previous devlog.

However, it didn't go as smoothly for the trading system, which was to be expected since this part of the game had not been tested at all previously. A positive takeaway was that nobody explicitly told us that our gameplay sucks and doesn't work at all. But people usually glanced over the trading menu and frequently made remarks that they “need this” to gather soul energy, indicating that they were perceiving it more as an obstacle rather than an intrinsically engaging element of the game.

After finishing the evaluation of the trading system, it was unclear to us if the system itself simply didn’t work or if it was more specific usability issues preventing players from finding engagement. But since we’re still convinced that a trading-focused method of interaction is best for Fireside, we decided to fix some obvious shortcomings in usability first to see if these could push the trading system more towards a state where it provides engagement. These issues were especially:

• Removing the hard quantitative display of Soul Energy, since displaying a number invited players to min-max the resource.
• Better representation of the internal state of NPCs (their needs & desires) so the player had some way to tell what to trade with NPCs (and thus an objective).

We scheduled another quick test for a week later and indeed noticed significant improvements in the players' interaction with the trading system. Still, there’s much to be done to optimize this rather unconventional gameplay system.

The Outlook

The next test for Fireside will happen towards the end of July. Till then, we’re going to conduct weekly small tests and play the game ourselves extensively. We’ve reached the phase in the project where the core systems are operating as intended and we’ve got a solid asset-base to support gameplay. Now it’s all about finding the right set screws to influence gameplay in difficulty, atmosphere, and pacing. This means building an iterated version of the game’s map, trying to find good variables for balancing, figuring out what parts of the system to show explicitly and what parts to hide, and many more details in game design.

If you want to join us for Fireside’s next test, please join our Discord server to get notified. Otherwise, you can follow us on Twitter, TikTok or Twitch for regular Fireside content! We’ll go back to developing now and are excited for what is to come!