Blog

Take that, CPU Usage and Bandwidth!

We’ve been optimising our arses off for the past month or so leading up to release, and the good news is, it’s been successful!

We’re currently on the fence with regards to the JIT compiler, we got it working but it has two side effects: code ran slightly differently, meaning subtle, hard to find bugs were introduced to all scripts; and it C++11 support was required, meaning anyone on old debian or centos would be out of luck running the new game without patching their libstdc++ manually. We’re currently running without it but keeping it as an option in the future if we really need to squeeze more performance out – for the moment it’s looking like we wont need to though!

With all the fancy new stuff KAG beta does, it starts at ~8% CPU usage (mid range dedi box) for 1 player (0.1% for zero players because the game gets completely paused with a new serverside option).

Now, that might sound like bad news; fortunately, it scales really, really well with the number of players (performance is contingent more on the number of complicated objects in the game now, rather than number of players), with our server pushing ~10% CPU for 4v4 (about 0.25% increase per player).

This is largely due to a massive netcode rewrite that’s still partially underway. We’ve learned a lot about networking this kind of game over the past few years, and the way we were doing it initially (based on the quake 3 model) isn’t optimal for a game with a large number of objects – it’s fine for classic where the majority of objects are players, but in the beta the number of game objects is easily into the hundreds, with trees, bushes, inventory items… As such it has been almost completely gutted out.

Long story short, bandwidth up and down per player at 1kBs up, 3kBs down (client end, for 8 players), compared to before the rewrite 2kBs up and 70 kBs down. This means KAG beta outperforms KAG classic in bandwidth utilisation, even though there’s a lot more going on!

We’ve still got to sort out some desync and disconnection issues, as is to be expected with a large network rewrite, but it’s gone very smoothly so far.

We’re going to be crunching all weekend to see whether we can get something together for a semi-public release (hint: watch the forums) because all the major technical hurdles have been stomped, there’s just gameplay issues to iron out now 🙂

Max/Geti

Guards vs The World ON NOW

The Third Guards vs The World event is about to start, join the server in the next few hours for some high class fun!

Status update: services failure

Everything should work now as expected. There is still some work that’s need to be done to recover some of our internal services but it’s nothing which will affect you (our customers) directly.

Thanks for your patience.

Furai

Our services being down temporarily

All of our services are down at the moment due to hardware failure of a critical network device in our datacenter. FLAB is working hard on fixing the issue. As soon as we’ll deal with the problem we’ll let you know.
We are terribly sorry for any inconvenience. We should have it resolved in the following hour.

Furai

Snags

I mentioned last post that I’d have a talk about some snags we’ve hit.

While I understand any halt to progress hurts for you guys, I figure that telling you about what’s going on keeps you in the loop with our expectations, and is definitely for the better.

Our major snag at the moment is performance. While we were expecting worse performance than with classic, we certainly weren’t expecting a hit this hard.
The client is not a problem – it runs at 30fps even on a shitty laptop with a 1.8Ghz cpu and a 64mb graphics card, so – great!
The server however, is a problem. It runs fine for a TDM game with a small number of players, however even for 8v8 in war mode, the CPU usage is unacceptable. With 12 players, the CPU of our box started maxing out. This is something we need to address before releasing anything, because we feel that it’s certainly not fair to server owners to plunder their boxes, and not fair to you guys to release something that cant have a stable server ecosystem.

The client runs quickly because it doesn’t have to do nearly enough logic – it doesn’t have to think about what’s happening for every object in the game, just the ones near to the player. The server, on the other hand, has to simulate everything.

To address this we’re considering a few things:

1. JIT compiling the scripts – this basically means the scripts get turned into something that can run on the CPU instead of on a separate VM that runs on the CPU. By taking out that layer of abstraction, we’d get a speed up of script execution. This requires working with the authors of an existing JIT compiler for AS (as the existing one doesn’t work with KAG as it stands, we’re looking into why), or commissioning a custom one (as we’re not in a position to write our own, time and experience wise). The existing JIT compiler doesn’t have an existing mac compatible version, but since it’s only the server that has performance issues, and not many people host OSX servers, it may still be an option.
2. Cheating, and building some of the script logic into the optimised, precompiled KAG binary. This really does feel like a last resort bandaid, as it doesn’t fix the issue of scripts being too slow when used in massive numbers (there’s about 30 scripts per player object, compared to 1-4, which is why the performance is tied to player count). It would also reduce our ability to iterate on the player logic at a basic level, because we’d lose the runtime reloading and no compile/distribute cycle waits that we get from scripts. We would like to avoid this if we can, but we might have to bite the bullet if we want to ship before we run out of money.
3. Continuing to madly optimise the scripts as we have been and hope we can make it fast enough without a JIT compiler or other native compilation options. We’ll likely do this alongside the other options anyway, but if we push it too far, the code will start to become very hard to understand and debug as more and more clever optimisation hacks make their way into the codebase. One one hand, this is kind of good – modders will get a look at what code looks like in the real world when you need it to run fast and need to ship your game, but on the other hand, it’s very easy to mistake carefully optimised code for code written without rhyme or reason, and we don’t want to encourage people to write bad code.

The other snags are related to bandwidth use (it’s high, not extreme but not nearly as minimal as classic, especially during action packed siege fights in war) and modding issues – modding works fine inside of base, but making mods using the intended system has quite a few large issues. Progress with single player content hasn’t been nearly as quick as we’d like either.

We’re weighing up our options and we’ll keep you posted on how things are going. We wanted to get a release out over the past week (actually, last weekend), but we’ve just uncovered more work to do.

As I said, I understand that more posts about the game coming out can be very frustrating, especially as you’ve been ready for it for months and months, but I hope you can support us while we drive forwards on making sure the game itself is ready for you.

Once it’s playable at a larger scale, we’ll be giving access to the beta via the forums. We’re very likely to omit fixing modding issues, optimising bandwidth use and waiting for tutorial content to be finished for that release so it can be out as soon as physically possible. Those points will definitely be addressed in time (and likely in that order as well), but we’re on the same page as you guys – we want you to be playing it above all else.

On a lighter note, there’s probably a video of some WAR gameplay coming (including boat to boat ballista combat, and possibly accidental keg hilarity) once MM gets time to put it together.

Max/Geti

More photos of GDC Soon – It’s over!

We’ve just had a full-on 2 weeks, and are heading home. Bye bye San Francisco!

Since I’ve got some messages here and there that people weren’t sure what was going on, here’s the low down:

Last week, MM and I went to Chicago to work with FliesLikeABrick at his place, working on a user reputation system, discussing how some of the modding stuff should work, and doing a lot of testing and long days.
 jrgp came up for a few days as well to grind out some work on the site, was fun to meet him.

MM and I then went to San Francisco for the Game Developers Conference (GDC). We met up with Rayne for some of that time, and while a lot of our time was taken showing people the game, listening to people talk at the conference, shoving our faces full of food and meeting up with friends we don’t get to see many times per year, we also got a lot of work done on KAG, and hit a few snags.

We’ll drop some more photos once we’re all home and slept and showered, figured you guys could use an update on what we’ve been up to though.

I’ll talk about those snags in the next post, I need to grab a drink and find a power outlet before I continue 🙂

Max/Geti