Over the course of the last few months, as I was reading forums and talking to people on IRC, I couldn't help but notice the staggering amount of suggestions and fairly simple requests that we've been offered. While this would be by default a fantastic thing, it triggers a returning problem that most players are blissfully unaware of: the complexity of our architecture, and our dedication to keep the architecture as robust as possible. What this essentially means is that sometimes the most trivial minuscule feature has to go through a rather arduous chain of procedures to be finally implemented on the client side. My attempt in this post today is to present you with a case study of how this happens in case of something most people would consider a single line of code and 20 seconds of work. But first, let's talk about concepts.

The principle of our system

The idea of the Perpetuum infrastructure has been engineered to be robust from pretty much Day 1, although admittedly we kept expanding on several things over time. The concept was the following: Since we're a very very small group of developers, we cannot afford turnaround time and taking time away from each other when it comes to fixing things, so the optimal solution is to allow anyone to fix anything they can. Of course, this doesn't mean that people with no interest in coding can happily add mov ax, 1 into random places into our source tree, but it's important to realize that in a project like Perpetuum (which is a rather strongly data-driven game), we have several times more the data than we have actual code. This means that as long as the artists and content folks can change the data, programmers can work independently of them, so essentially what it boils down to is exposing as much data as we can. (If you started giggling, do a few push-ups. It helps.)

The way we do it is manifold: For artists, who mostly work with standard format data (bitmaps, meshes, sound files), we have a file server where they can upload new versions of resources, and they get downloaded the next time someone starts the game. Of course this can result in some problems when we change some internal file formats, but as long as the client is able to do an update, this shouldn't be an issue. (We also have revision control, so reverting to an earlier version is easy.) On the other side, we have the content guys who work on managing the actual game logic data (i.e. item data, balancing, that sort of stuff) - this isn't as trivial because this data needs to run through both the server and the client: the server needs this data for calculating the game logic, the client needs this for displaying data and doing client-side prediction. This also needs to be both persistent and easily alterable, so for this, we need a strictly database-driven environment.

So here's how it works.

A simple example

The most trivial example I already noted when we were working on it was related to sparks: The spark GUI was pretty much finished codewise, when Zoom noted that the order of the sparks doesn't make much sense because the syndicate ones are on the bottom while the ones that are harder to access are on the top. The reason for that, obviously, was that Alf simply added them in the order he felt like at the time, assuming that we can just sort it on the GUI-level later. He was obviously right about that, except we didn't really think of a method.

The idea looks simple on the surface, but there are several solutions to fix it and all of them are either laborious or asinine:

• We could leave them unsorted. That doesn't fly because it's annoying.
• We could sort them on a GUI level, by hardcoding weight values. This is just a big no-no considering that any change would need to go through the client coder. (=me)
• We could create a resource file in the resource repo that describes an order, and thus any non-programmer could edit it. Sounds good on paper, but sparks can be added or removed at any time, which means that if the resources are out of sync, the whole thing becomes a mess, and you're still working with two things at the same time. So no, this is just shooting ourselves in the foot. (Update - I might have forgotten to mention the most important part: non-programmers and markup languages rarely mix well.)
• We could sort them on the server, by hardcoding etc etc see above yeah no.
• We could extend the database records for sparks with a sort order value. Shiny clean solution, but with the most work to be done. Why? See below.

Implementation

So here's the entire process of how a single integer value traverses through our infrastructure. As you may see these aren't particularly complicated steps by themselves, but the amount of steps and their versatility eventually amounts up into a surprising amount of time.

First, we take the database table for sparks, and add a field. This is done on the database server, usually by Crm.

Second, we extend the administrative web-based back-end (what we lovingly christened "the MCP") with the field so that we can have data to work with. This can get tricky if the field isn't just a standard number (but e.g. a new item type or an extension ID), but in this case this is just a simple text field. This is done by me.

Let's remind ourselves of something at this point: Because of our rather rigorous security-principles, no game-related data will be kept on client side, everything comes through the server. Now because sparks are very much a core gameplay element, everything related to them will also come through the server as well. So our next step is to add server code that loads the new value from the database, and hands it to the client. This is done by Crm again, and usually involves a server restart, much to the chagrin of whoever is working on the development server at the time.

So now our value is en route to the client, all is left for me is to add fields to the client cache and use it to actually sort the sparks in the correct order, i.e. the aforementioned "single line of code". Of course, the actual sorting algorithm code is already there, I just have to build a new comparison function, test it, deploy it as the newest client update, and ultimately the feature is done easily.

Simple enough, but if you look back, the ricochet performed by that one measly integer value is pretty staggering.

What you have to understand here is that this is the best solution, as strenuous as it seems:

• It's robust (no hardcoding involved, extending doesn't involve breaking anything previous)
• It's completely "user"-driven, anyone can use the web-interface to change the values without having to worry about breaking something in the code level (mostly); database-constraints and form-validation can take care of values we don't want in there.
• Most of the code-changes needed are just passing around values without thinking of what it really is.
• Once the code works, it'll always work as long as the data itself is solid.

But of course, as mentioned, the downside is that it needs four different levels of changes (database, back-end, server code, client code), which is only done by two people because of the involuntary competence we acquired over the years - in a "larger" project, this could possibly take a lot of cross-department management, while here it's usually just strong language and the occasional "Eh, I'll just do it myself". However, once we're done, we can wash our hands and never think of this pipeline again, because the content will go in on one end and the code will take care of it in the other. (On a tangential note, this applies to blogposts too: This post will probably be / was proofread and reviewed by several of us.)

Exercise for the reader

Now, to make sure that you understand the gravity of this stuff, here's a little thought-experiment for you: When you right-click an item and select "Information", there's a lot of parametric data that ends up on the screen, each with their specific measurement unit and number format. Now remind yourself that some of these values may look completely different internally: percentage values that show as 5% are obviously stored as 0.05, bonuses that show up as -25% may be stored as 0.75, some values need two decimal digits while others need five, and so on.

So here's the challenge: knowing the above principle (i.e. be data-driven), what kind of system do you build to make sure that each of those values end up being displayed correctly for the player? (No prize for the correct answer, although we do offer occasional trauma counselling.)

In a later post, I will go into a deeper insight about how an actual feature (case in point the whole spark-doohickey) goes from an idea to design to concept to task to implementation, down to the level of "BUG #4170 - please move that text one pixel to the left".

This blog post is intended to give all of you my take on the last few days. Since it's been such an exciting time for us here, I'd like to give my own perspective on it. This is a pretty tech-heavy post, so be warned.

Last Thursday I travelled to Paris to give a couple of interviews to the mainstream French gaming media about Perpetuum. All went well, and I'd like to seize this opportunity to send a big shout to all the participants for their kindness and professionalism.

Saturday morning, while roaming the streets of Paris, I started receiving messages and phone calls from the dev team at home about a nice amount of new registrations. As time went on the number just went up and up. Having no idea what's going on, we were simply happy about it and tried to craft theories around it. Later that day I met Guillaume - who is one of our players - and he explained everything about the current situation on the MMO landscape. (Thanks Guillaume for the hospitality, I owe you one!) In the meantime, I was being spammed with messages that something weird was going on with the relay server, causing insane lags. You can imagine the number of calls, as even the battery ran out in my mobile!

Sunday night I got back to Budapest, went home, took a shower and immediately headed to the office to check on the situation and fix the server problem. In the office I found out Calvin came to pick me up at the airport, but since my phone died on me earlier, we managed to miss each other. From this point on, I completely lost track of days and hours with almost no sleep only focusing on the server issue.

We checked everything: optimized the SQL, implemented new caches on the server, tried many other things, but they were minor problems compared to the evil seed that caused the problem. The main difficulty is that it's very hard to generate load on the dev server similar to the live one. For the tech guys out there, the transaction coordinator (MSDTC) was NOT the problem, it causes an insignificant load, so we are fine with that.

We had to put in two nights in a row. We simply had to rest, but it was made hard by the stress and the guilt of leaving the gimped server alone, not knowing what would await us when we wake up. Things were looking grim.

When we realized the seriousness of the problem, I contacted one of the most knowledgeable people I know in this field and asked him to help out as a fresh mind always has a better chance finding hard bugs. Our deepest respect Soci! (Shameless promotion: http://soci.hu/). Luckily, he had time to check out our architecture and together we were able to start an investigation session on Wednesday night. He gave us several suggestions, pointed out and helped optimize several things in the database layer. Then we moved on to inspect the relay server's code. Since the source is huge, the quickest and most realistic method was to attach an analyzer to the live server application. I must admit this was the last thing I wanted to do on my own, but at this point I was willing to sacrifice anything to find the root of the problem.

He instantly figured out that one innocent-looking little function, namely the one which returns who is online (in chat channels, for example) has an emergent behavior, resulting in an exponential load and suffocating the server. (Insert random sarcasm about glaring oversight and tech madness here.) We then ran Visual Studio performance monitor on the live server to dig down to the heart of it. Soci said it might cause some load, so we messaged the server with a warning about what we were doing. And then we accidentally the whole server. :) 0245 server time, bye-bye field containers!

Thursday we closed the session with Soci and went back to implement what we'd learned. Quick tests on the dev server showed pretty amazing results so it made us rather confident. With shaky hands we patched and let the first 50 players in. This was the moment of success! So we immediately let 100 more players in. During this period we constantly checked the load which became insignificant (2-3%) so we raised finally the cap to JUST OVER 9000!!!!! >:)

The time to chill out has finally come, so we popped open one of Gargaj’s precious Norwegian treasures that had been sitting on our shelves for some years, a bottle of blackcurrant wine. I don’t know if it was the quality of the booze or the grace of that moment, but it was one of the sweetest sips we ever had.

Well, that’s the end of that, hopefully I managed to shed some light on the problem and our day-to-day march until we reached victory. The code belongs to us, but the world of Nia is yours. Enjoy!

So what are we up to when we are not out deleting NIC from your accounts and playing Insurance Fraud Online? Well, for one, we make plans. And since we got a lot of complaints from you that we don’t publish too much information about the upcoming features in the game (not unjustly, I might add), it’s high time to sit down and share.

The features below are listed more or less in the order we currently plan to introduce them, with the first ones being only a few weeks away and the last ones planned more towards the second half of the year.

Artifacts and discovery

When we recently hinted that we are working on new PvE content, most of you probably thought “Ohnoez, more Buttkicker Observers and boring missions...”. But nah, we are perfectly aware that it would be just more of the same. Instances probably crossed your mind too, but we don’t want to separate PvE and PvP and we definitely want to keep the game world open and persistent.

So we came up with something that is easy to get into, is available to everyone, from solo carebears on Alpha islands to the proud cowboys of Beta islands, and is something that can reward you with shiny new stuffz.

Artifact scanning (as we call it) is basically an extension to your current geoscanner modules. Every island will have an ever-changing collection of hidden spots that players will be able to search for with their scanners and reap their rewards when they find them. There will be several types of artifacts ranging from simple salvage containers, through hidden stashes guarded by angry NPCs, science vessel wreckages holding calibration templates of new Mk2 robots or even alien data storages with precious intel that will get you on special missions. Since this is a very easily expandable system, you can expect more artifact types every now and then.

The energy must flow

Those who are familiar with the backstory of Perpetuum are probably well aware that we are here to exploit the energy of the planet and send it back to hungry Earth via that same wormhole, which we used to get our tiny “sparks” to Nia. You were probably also wondering “where in the game do we do that?”. Well, currently nowhere since we are still at the early stage of “setting foot”, but this is soon going to change.

We’ll be intruducing a new energy-credit system which will work a bit like a second currency. These energy-credits (let’s call it EC for the time being) will be awarded to those who contribute to the energy-packages that will be periodically sent back to Earth by the Syndicate.

You’ll be able to achieve this via two ways, one PvE and one PvP option:

• Using fragments of a rare kind of crystalline material, which can be charged with energy and can be found through artifact scanning mentioned above, you have to go out to shiny energy fields, deploy the crystal and wait until it’s fully charged, then return it to a terminal. Of course indigenous Nians also like to be around these energy-rich areas, so you won’t have an easy time doing this.
• Own an outpost with a corporation which is producing the energy for you.

So what will EC be good for? New modules, rare components used in Mk2 robot production, CTs available only here... you know, stuff™. The details are still being worked on but we’re confident that this system coupled with artifact scanning will give a nice boost to what you can do in the game.

Assignment improvements

Yes, were aware that the current assignment system rather follows the bad concept of “quantity over quality”. We have plans to make current assignments more interesting, include new types of objectives or combine current ones, revisit their rewards and introduce payout bonuses for those who complete them within a certain time.

Corporation recruitment

There are basically two games in the world of Perpetuum. One is the initial “run around, complete missions, collect stuff” side of it, and we’ll be the first to admit that it lacks content to be truly enjoyable. And then there is the second layer of the game, the player corporations.

We fear that some players do not get to the point where they actually join a player-run corporation, and leave the game without ever realizing what is going on behind the scenes in this little world of ours.

Therefore we would like to implement some features that will guide players towards finding and joining a corporation that is suitable for them. The tutorial will also have a chapter on this and we’ll introduce a “yellow pages” type of directory, where players will be able to filter corporations based on their of size, language, timezone and profile, which CEOs and officers will be able to set in the extended corporation information panels.

Artillery and area-of-effect explosions

Continuing our efforts to “fight the blob”, we’ll be introducing area-of-effect explosions, which will hurt everyone within a certain radius. On one hand this damage will be done by the explosions of robots when they are destroyed, but we’ll also bring in new weapons of mass destruction: artillery modules. You’ll see both standard Earth-technology built by the Syndicate and also different types of modules for all 4 factions (yes, even industrial ones) and they will certainly stir up the crowd.

Even more types of weapons

We’re not stopping there, new factional weapon types have been already in the works for a while. These new modules will be of course rather high-end, with high extension requirements and energy usage and while they probably won’t have the highest DPS around, all of them will have secondary effects: Ion cannons will jam enemy sensors much like ECM modules, energy torpedoes neutralize a hefty amount of energy on hit and plasma guns will reduce the armor-resistances of the target.

Hybrid robots

I suppose some of you have already encountered the funny misfits certain DEVs like to run around in. The system where we can freely build robots by combining any head, chassis and leg components has been in the game from the very beginnings. The only problem: it’s nearly impossible to balance it properly. We haven’t given up yet though, so you’ll most likely see a system like this in one form or another.

New lands to explore

This doesn’t need much explanation - having such a limited area available makes territory control too easy for large corps, makes it impossible for smaller ventures to hide and build, and generally limits exploration and exploitation of game mechanics. Even with our procedural terrain generation system, creating whole new islands which actually have some content as well is one of the most time-consuming tasks, but it is also a part of the game that can be expanded indefinitely (in theory). So this will be an ever-ongoing venture, but the plan is to have six new fully featured zones in our first package.

Terraforming and player-built structures

New lands hold new possibilities, and terraforming is an old promise that we intend to keep. While it’s not feasible to allow players to dig up the soil under dumbfounded newbies sitting on the alpha islands, certain newly introduced zones will provide this strategically very engaging option.

Terraforming can be interesting in itself as well, but in the end it’ll be mostly a tool to make room for structures built by the players. This will be our biggest expansion of the game (of what we have solid plans for anyway) and as you might guess this is the feature that I can provide the least details about currently. I think it’s suffice to say that we want to make it very modular, expandable and complex, so you can create something that you call “home”.

Closing words

Well, I think that’s about it for now. These are only the bigger incoming features, but we’re of course continuously working on fixing those bugs and re-thinking current features like the waypoint and the geoscan result system or the event messaging system.

As you can see we are determined to make the game better and more interesting and hopefully this post provides new hope for those who fear for the future of the game. Fear is the path to the dark side and we don’t have pathfinding yet.

I took the liberty to hijack the blog from BoyC to start a little skunk-work series of tech-articles about the stuff that goes on behind the curtains in Perpetuum. There's some fairly unusual ideas and solutions we came up with over the years and I hope they're as interesting to you as they were for us to invent / implement. (Or well, potentially more interesting, and without the malnutrition / hangover / sleep deprivation.)

So, maps.

The main principle

What is the task at hand, what are the conditions we can work under?

• We need to have a map in the game where players can see the main navigation points. The goal is the map to look as good as possible within the given circumstances, since players will be staring at it for prolonged periods, and we don't want them to stuff their own eyes into the recycling facility 5 minutes into the first transport mission.
• We do not have expendable artists who can spend time on painting maps manually, but more importantly, unlike other games, the map here needs to be as accurate as possible, for strategic reasons.
• The terrain is a height-map - we can safely assume the client has access to this data, but this data has the possibility to change fairly often. (Think terraforming, either by players or by level design.)
• Not really a mission-statement as such, but resolution-independence is a plus.

Summed up, we need a fairly non-linear data flow coming from a 2D dataset, and ending up in a good looking texture. In other words, procedural textures. Luckily, we're fairly comfortable with that. (*wink* *wink* 64k *cough*)

The steps

Let's look at our starting dataset first:

This is the actual terrain height-map, a 2048x2048 2D dataset of 16-bit height values. The reason it looks so dark is because this is a linear mapping of the 0..65535 range to the 32 bit colors available on your screen. Given that we don't really have to care about the peaks so much, since robots can rarely pass those, we can re-map the range of interest between two completely arbitrary values following "This Looks Good Enough"-principle, ideally somewhere around the water level for the low value, and the highest walkable point on the map for the high value.

This already looks a lot more usable. Sure, the mountain peaks become a big white spot, but we don't have to concern ourselves with those because they're rare, usually uninteresting for the player, not to mention the strange yodeling sound they emit.

It's time to add color. Cartographers generally follow a principle where they assign certain colors to certain heights, which works perfect for us, because we can use a look-up table internally to do so. This look-up table, in our case (and ideally), is a simple 256 pixel wide bitmap, painted by a graphics artist:

So if we use this to color up our altitude levels, we get this:

This almost looks good enough and no-one would blame us if we'd stop here (no, you wouldn't, shut up), but we knew we could do something more interesting given the data at hand a little maths. The image already has a nice faux-natural quality to it, but it needs contrast, the same way maps often have a little lighting / shading on them.

Adding contrast

At this point, the idea was to take the altitudes, more specifically the altitude-differences (or slopes), and turn them into shading. For this, what we've done is an implementation of what generally bitmap-manipulation software call as "Emboss", and basically means taking surrounding pixels, and calculating a normal vector out of them. Many of you might wonder if it means we're calculating a normal map, and in one sense yes, but the trick here is that the direction of the light can be fixed, because why not, it's Good Enough, and that allows us to speed up the calculations simply by assuming that the light always comes straight from the left.

For less technical minded: This practically could mean that we start walking (figuratively speaking) from left to right on every row of pixels, and if the next pixel is a point higher, we get a bright point, if it's lower, we get a dark point. (If you're indeed a code-savvy person, please refrain from gnawing out your own cerebrum at this stage because of this rather sloppy explanation, I was trying to make a point here.)

If we take this normal value and map it (with a bit of magic number wizardry) to the grayscale color range of 0..255 (0 meaning the next pixel was lower, 255 meaning higher, 128 meaning it was the same level), we get this:

Not the prettiest of pictures, but it's a vital component towards what we want: contrast.

So we have a colorful picture with no contrast, and we have essentially a bump map with no colors - how do we blend the two without calculating lighting on every pixel?

Simple: Prepared lookup-tables. We know we have 256 levels of color, and we know we have 256 levels of contrast, so we can pre-calculate a 256x256 pixel lookup table with all the possible combinations and utilize that to render the final image:

This lookup-table is ostensibly the above gradient, with the top half fading to black and the bottom half fading to white. Between that, in the very center, is our original gradient.

All it remains now is to run the following little lookup, matching our color and contrast:

for every pixel {

  tx = get_remapped_altitude() // 0..255

  ty = get_contrast() // also 0..255

  final_color = shaded_gradient_lut(tx,ty)

}

And here's our final image:

The difference is subtle, but definitely visible when the maps are zoomed up close - which is what will happen in most cases.

The advantages of this method are manifold: it's fairly fast (it's entirely fixed point arithmetics), it's resolution-independent, it looks Good Enough, and most importantly, it's responsive to the level of immediacy, i.e. every change done on the map can be rapidly re-rendered on the map as well. (This also goes for the passable terrain calculations, but that's a whole new story.)

That concludes our first visit to the forsaken wretched nadirs of the Perpetuum codebase. I'm not sure what to write about next time, but I suppose a quick look at the "number of changes per line of code" graph will quickly indicate where the "interesting" bits in our code are.