Dev Notes: From the Trenches
MMC v.0.1.5 (Google Play), Phase 3 App Status 5/25/2021
Server Crashes: The server crashes over the last few weeks have slowed new feature development down considerably. We have instrumented servers right in play and when one crashes the other should take over and carry the load. There are debugging tools that we turn on to watch every move on those servers but it drags down performance and players experience a lot of lag (and the flood of lag emails prove this). We tried our best to be done over the weekends so players could have a better game experience, but that means we're not gathering any data on crashes over the weekend. We tanked at least one weekend working on it full time. The problems were not easy to pin down and we needed the systems under load to reproduce them. Alpha man... Let me tell ya... Well I don't have to, most of you were there.
We discovered two significant causes behind the server crashes and believe we have some fixes in place to resolve them.
One root cause was from hacks manipulating their clients to change things like damage values and messages. To put it simply, the game is still in Alpha and our code is not yet hardened enough to deal with those variances and their hacked mods crashed the servers. This is a sore spot with us because we know where the opportunities to hack the game are at, and we have plans to deal with that down the road, but we hoped that we could focus on building the heart and fun of the game first. Everyone keeps asking for new guns, skins, game modes and maps, but we can't even get through building the infrastructure that will let us add those new items without having to stop everything and fix the crashing servers caused by hacked mods. Making hacking mods does not help us. Playing destructive and cheat hacking mods certainly does not help us. We lost several days (week+) worth of development time dealing with this.
Turns out there was an issue with our leaderboards trying to retain some null entries that were crashing the game server. Again, the hacker induced crashes created so much noise we couldn't isolate the issue for well over a week. For legitimate issues with the Leaderboards, we're still looking for people who might be missing entries or other anomalies. That work continues, but is pretty stable at the moment.
We thank everyone for the reports. When you submit complete information we can build up a case against players. The more damning evidence the quicker we conclude. And we started banning players. We're not at the point of retroactively removing their kills, but it is not off the table either. For now we will just let them QQ in the ranks. We're serious about policing this space and keeping it as legit as we can (and we haven't turned on the anomaly detection net either, it will become more automatic. You can't lie when faced with server truth, good code willing.)
As already mentioned with the server crashes, we wasted a lot of time dealing with people hacking and causing problems. We are basically rebuilding and redesigning DA2 from the ground up and each new development piece has been arranged into the different phases of our roadmap. A big part of the Beta phase (at least that was part of our original plan) was to reduce the footprint of the hackable content. We really didn't think people would take such joy in ruining an Alpha game still under development. The people creating these hacking mods are playing their own game, thinking that they are making MMC "better" by showing us what is wrong with the game. But instead of just telling us any details, they wrap their exploits into a mod and share it with the masses and disrupt the game. There is no shared information, we have to figure it out by getting angry hacker reports from users or hunting down crash reports from our servers, and reverse engineer what is causing the problem. All new forward development stops and no new game features can be worked on.
This is another great example of why we can't give out exact dates for releases. We had to stop our development for nearly two weeks to put in some TEMPORARY solutions to deal with the most extreme cases (that is time spent NOT working on the final product), manage all the hacker report emails, and gather evidence to start banning players. It's hard not to get angry about the time lost here. We have new designs that we want to use but they aren't built yet; however, the cheater problem got so bad we had to bring old systems online just so we could get by for now. All of this will have to be redone in the future.
There isn't a lot we can do about the people creating the mods to cheat. All we can do is ban the people that choose to implement them and use them on our servers. As it stands right now, with a Global ID and good evidence submitted to us (video is preferred) we will ban the player. For BANNED players:
- Their scores will no longer be counted in the scoreboard once banned
- Will not be given access to play on the servers
- Game servers will deny them the ability to play
- Will lose access to friends and messaging
- Will no longer have access to any other server-based service
- (future) Friends/Clans will see the player is banned in their profile
Reminder: How do I report anomalies/cheaters/glitches/etc.?
- Clear Information: Describe what you saw with as much detail as you can.
- What were you expecting the item to be like? How was it different?
- What did the hack let them do?
- What was the game mode?
- Does the hack relate to a known game bug being exploited? (like flying through walls)
- Provide Evidence: It's not always possible, but if you can provide screenshots or video that is ideal. Please don't just send the file though. Include notes about what is happening with a timestamp of the issue shown in the video and a description of the issue. When reviewing a video, knowing where to look (the time stamp) saves us considerable time in analyzing all the bug reports that are sent to us.
- Custom games (including CTF for now) NEED to be managed by the host. It IS their job. Quickplay is the only place the DA2ID console command works atm.
- Report Global ID: For cheater reports you can now send us their Appsomniacs Global ID (shown in [brackets] next to the player name). To access the Global ID type DA2ID into the chat of a running game (not in the lobby). Take a screenshot of the scoreboard showing the players' GIDs.
- YOU HAVE TO INPUT DA2ID IN A RUNNING GAME; NOT WHILE IN THE LOBBY. Also remember, the DA2ID is not available in Custom/CTF. You will need to use the game host to manage the room and kick out anyone cheating. If the host is cheating, leave that room. It's a community managed space for the time being. We have more to do there to firm up that space, but it will be a while.
WHAT IS NEXT?
The new Profiles replaces the old Avatar screen. We've shifted the image around a bit to make room for the scores. With the new Leaderboard system, there is just a lot more information now available for us to use. At first there will be just a few lines of data like your totals and average per game for kills/death/damage/best kill streak/kill death ratio (KDR), etc. The data will grow over time, be tweaked, and we can add things like time played and other information players want to see on their profile cards. Because the data is being managed on our servers we won't need to make a client update to add more to the expandable profile stats section. Reminder: scoring is only on Quick Play right now. Until scores are collected from other game modes, Profiles will only show data from the Quick Play games. Scoring for solo, offline, CTF, and custom will likely be separated into their own sections. We got a lot of flexibility in this system for future growth so we will revisit it to make it useful. Looking at stats of other players on the leaderboards will have to wait, but it builds off this same system, so it should be easier to develop. Storing all these stats is gonna cost a pretty pile of rupee over time, so expect us to monetize this part (mildly) but we gotta start thinking about how we are going to pay for all this.
- iOS Release: You may have noticed the checkmark on the roadmap instead of a percentage. A release was made and Apple rejected it (again). Chad's working on it right now. We also found another way to distribute the iOS version that may bypass Apple but it requires some more research and we don't really have the time at the moment. If we can explore that path we might start bypassing Apple which will make dropping iOS releases much quicker. We haven't given up on iOS but we need an IPA that passes Apple before we distribute. Now because you read every word so far... Here is your award:
We got an Apple build passed for the first time for an internal testing. It was deployable, but it was broken because of a change we made after the release... The other catch. You have to have ios v13 or higher to run TestFlight. We're looking for another way to open this up to our older devices. SO... we will make some changes and push that out asap, but there is a lot happening this week. If you want to get the invite to test the iOS version please send an email to testing...@appsomniacs.com and title it "iOS Alpha Test Signup Request" or something like this. We will be getting the invites out as soon as we have a fixed version that allows you to play on Quick Play.
- Interface Input: Lots of requests for changes to the look of the interface. Feel free to send in ideas. Once MMC launches, I see us changing the interface look and feel at some point. For now and the immediate future, we're sticking with our 2018 v4 blue buttons and chunky font (although the more I see it, I see issues with the font. D&O look a lot alike as do R&A... and now that you know you will see that everywhere. It can't be unseen!).
- Roadmap Update: Minor updates to the Phase 3 section of the roadmap. Percentages were updated slightly, the list was rearranged based on current priority (top --> down), and a new server refactor line item was added as a response to issues found on the server.
- Roadmap Files: Attached are updated files.
- Here is the shared folder link: https://drive.google.com/drive/folders/1oOdaFTQF18rN-_gzysJljA9OCAZMRcPl?usp=sharing
- Files located in the Roadmap folder and latest screenshots are found in the DevScreenshots folder (profiles and leaderboards).
- We've begun creating a group for content creators and event coordinators for when we can finally gather again. Its pretty free form at the moment, but we had 4 folks sign up, so I am gonna call out their channels for being first. We're hoping more folks join it. We have a communications officer, Martin, now. It's part time, and unpaid, and we're just grateful for the help. They will be helping us organize more. It's not something I planned on turning on until Beta starts, but no harm in gathering now, it will be a place to have Q&A managed by the Comms Officer. Let us know if you want to be among those channels. Just reply or email us at testing...@appsomniacs.com if you want to be part of that group as it grows. Make sure to include your channel or event info and your formal names, etc. Here are the first, you probably know at least one of them. Shout outs for signing up and helping us plan how we can better get the word out.
- The link for our hi res versions of files included in this mail can always be found on our public share:
Building a scalable stat management system is hard. But we're getting there! Testing this will be a new kind of nightmare for us to deal with. ;)