703 players online!

Today, we reached another milestone: over 700 players online. Counting the bots, we had over 750 connections.
The server load was about 10%, which means we can perhaps have 5K players online. The outgoing bandwidth was at a confortable 1.1 Mbps (peak), which is excellent.
Since the beginning of the year, we got perhaps 25% more players, and there are still two months left. It is usually in the winter when most of the people are playing games (too cold to go out, the day is short, what better thing to do than play a MMO? :D ).

Bots and spiders

Last night, my host asked me if I know what causes some abnormal spikes of bandwidth on our webserver. Learner took a look, and noticed that one IP was spidering the online players page like crazy, a few times a second.

For example, here is some small sample:
64.72.119.194 - - [24/Oct/2006:19:52:03 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:03 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:04 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:04 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:05 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:05 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:06 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:06 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:07 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:07 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
As you can see, the server sends a 403 response. This is because I banned that IP, and a few others from our webserver. The bot is still trying to spider, but is not able to anymore. And it's been doing this for 8 days now. Too bad that the person who is doing this is not playing the game from the same IP, or he would be permanently banned.

Our policy on bots and spiders that access the online players page is simple: not more than once a minute. 120 times a minute is really a little bit too much.

Open Source Software

Now that my laptop fixed itself, for the lack of a better term, or any clue as to what was wrong, I have an extra machine (the laptop) to play with. This is because my new desktop is much faster and better.

So I installed PCBSD, which didn't recognize my wifi card.. Great, let's try Ubuntu. Afterall, everyone says that it has a great hardware detection. Same thing..
So I went back to Mepis, version 6.0. Last year, Mepis version 3.2 recognized the wifi card, and pretty much all the devices on the laptop.
Well, it turns out that now it won't anymore. I googled around, and found out that other folks had similar problems, with their network card being recognized in the past versions, but not anymore.

Yes, I know, ndiswrapper. But to be honest, I don't really have time to do all those tricks, especially that I don't have the windows drivers for it (I'd need to reinstall windows from the rescue disk to get them). And if in the past I enjoyed hacking my computer and modifying autoexec.bat and config.sys for each game I played, now I am too old and busy for this shit.

Is it too much to expect from an OSS to actually keep their supported devices from one version to the next?
Speaking of OSS, I decided to get the latest version of Gimpshop. 2.2.8_fix1
I had a previous, older Gimpshop version, and thought that a newer version would be better, right? Turns out that this new version has a bug, it tries to read some cardreader drive, and will give a "no disk in drive" popup error every time you do something remotely useful, such as opening images, etc.

Now, some would say that I have no right to complain, because we also introduced bugs in our new versions. This is true, but we do our very best to fix them as soon as possible, so long as we know what causes them. And the bugs that make it to the final client are usually relatively minor, which is why they evade prior testing, where people can download and test the client prior to the official update.

One other other argument is that you get what you pay for. You have no right to complain if a free product doesn't match your expectations.
I usually donate money to OSS that I use. In the past I donated money to Mepis, OpenBSD, FreeBSD, FireFox, SDL, and some other software I don't even remember the name of.
But most of the people do not donate money, so their developers can't afford to work full time at their software, which is reflected in the quality of their products.

We are among the few, lucky developers that actually make some money. Most of our money doesn't come from donations, but from people purchasing items or services.
Which is why I think other OSS developers should do the same thing. For example, they can ask people to pay a certain amount of money in order for them to implement a new feature, or perhaps fix a minor bug, or port it to another platform.
In return, those who pay can get the newer version faster than those who do not pay.
The guys from Cedega, for example, even charge a monthly subscription fee for their product, and from what I understand, they are doing pretty well, being able to hire developers that add new features, optimize stuff, fix bugs, provide some customer support..
IMO, that's pretty much the only way to be able to thrive in the everchanging world of computer software.

My laptop is unusable.

This evening, while 'playing' EL, looking for some info for an upcoming contest, the EL window became garbled. It might have seemed to be an EL bug, but the problem was that both my screens got all kind of lines on them, covering other windows as well.
As soon as I closed the El window, those lines disappeared.
I thought maybe the video card fan (if any) died, so EL overheats it, which causes this problem, or, maybe, ZP is acting up.
Restarting the computer didn't help, so I said, hmm, maybe I should update my video drivers, who knows?

After installing the latest Omega drivers, I restarted the computer and had the unpleasant surprise to notice that the POST screen also has similar visual artifacts. Shit!
Now there are lines allover the screen, and 1 in 4 pixels are missing :/ Which makes doing anything with the computer a pain. EL doesn't even work anymore, so my only choice is to get a new computer.
Now, I planned to get a new computer sometime next year, so it's not really a tragedy that my current computer is unusable. I tend to see the good thing: tomorrow I will have a faster computer, and, if I am lucky, maybe I will manage to fix this laptop as well, so I will have a brand new desktop and an usable laptop.
Tomorrow most of the day will be taken by buying the new computer, installing it, configuring stuff, transferring my data and settings, open the laptop and see if there is anything I can do with it..

Sometimes life sucks, but you have to take the minor annoyances that happen to you as a lesson, and try to find the good part. And maybe this is God's way of telling me to update my computer :)

Finally a good update

Or at least this is what most of the players are thinking.
I took some time off from programming and made quite a few ballance changes in the game.

I looked at the monster spawns, and eliminated some unused ones, added some 'high demand' monsters in various previously rarely used map, changed some monsters strengths, added a cheap source for vials in a rarely used map, added a new spell, adjusted a previous spell to make it level dependent..
The players were very happy with those changes, which is pretty rare this days.

Now I am working at some other minor changes, as well as some new additions such as a few new summonable monsters, a new special day, and possibly some new spells.
After I am done with those things, I will most likely work at the summonable stones, then I have to implement some really complex things, such as finally making dynamical maps. That will take months, and requires a LOT of work and bug testing.
I will describe how it will work in a future post.

Plans for the future

With the schools now in game, where you can create items without obtaining the item itself, but twice the experience, most of my current plans for EL are finished.

One idea was to start working at a few new items, such as the summoning stones and various other combat related items, but considering the large amount of bitching I am getting regarding pretty much anything I implement, I think it's time to take a small break from adding new items, and focus on some internal things that were on my to do list for quite a while. So for a few months there will be no new items.

We also decided to actually enforce our forum rules, namely the no bitching rule.
From now on, everyone not following the rules they implicitly agree to when they use our forums, will be suspended for 1 year. Depending on the gravity of the offense, all their posts might be deleted as well, and they can even lose their game account.
This is pretty much standard practice in the big, commercial MMOs. I didn't really want to do that, but it's getting to a point where it becomes too frustrating for us, and it is not worth it.

Other people also complain about how some people buy ingame items for cash. Nevermind that most of the people in the top 50 paid very little if any cash to us.
Nevermind that we wouldn't be able to spend so much time developing this game if not for some cash.
Of course, they do not really care about such insignificant details, to them complaining and whining is more rewarding than actually playing the game.
So as I told them on our forums, if I keep getting complaints like that, we will remove the option to buy items for dollars, use our existing cash reserves to develop the game, and in the not so distant future sell out to some company interested in our technology.

The next thing I will start working at soon is the bandwidth statistics. We want to know which packets take the most traffic, both up and down, and try to focus on optimizing them. This will be very helpful for when we have over 1500 players playing the game at the same time.
Next, me and Learner will work at dynamic maps, which will keep us busy for a few months.

Device Polling on FreeBSD

As mentioned in my previous post, the Device Polling makes a HUGE difference.

Here is the "before" graph


And here is the "after"


To be fair, the "before" one was taken in the weekend, where we had about 100 more connections. However, notice how now the usage is at 1/4 of that (with about 570 peak connections). The actual server process took about 6.5% of the CPU with 600 connections.
And as a clarification, the spikes you see are not caused by the game server but by various stuff such as recompiling the test server, stress testing, etc.

I am so happy with FreeBSD that I donated 100 USD to the FreeBSD Foundation. I also donated 50 USD to the OpenBSD guys a few months ago, for their effort.

Problems solved

It turned out that the problem wasn't really that range calculation function. While it is the most expensive function in the whole server, the OS kernel also plays an important role. It has to deliver the data to and from the network cards, and that can take a lot of system resources.
So most of the delay was associated with the networking. Our host figured out that enabling Device polling, as it was on the old server, might help. And he was 100% right, now with about 150 connections, the server uses ~2% of the CPU, rather than 5%.
Tomorrow I will be able to see how much CPU it will take with around 500 connections, and if we are lucky, it should be under 10%

Unrelated to this, I implemented a few new things lately, and I think the players will have a nice surprise once they will be merged with the new server. One of the new things is a special cape, and the other thing is a secret.