I was playing Project Explore and decided to launch a new world with the size set to "Extra Large". I began my new world and discovered to my surprise that I would have to wait a whole 90 seconds before the game finished loading. What was even more concerning was the 1.2 gigabytes of memory the game now required. This resource intensive system I have in place disgusts me and leaves me feeling like I have a challenge to overcome.
The current system keeps the entire world in memory via a simple system of lists and grids (1D and 2D arrays). As you can imagine 45,000 objects and all of their information takes up quite a bit of memory. Well now we have an issue. If we don't have the world cached in memory then we may experience hiccups during the gameplay while chunks of the world are read from a file.
If each chunk costs about 0.06 MB of memory, then you can see that in an "Extra Large" world that is 120x120 chunks, you are using a total of at least 864 MB just to keep the information cached. That is ridiculous in my opinion. So here is my new system.
Of course that picture alone doesn't explain the whole system by itself so I'll divulge a bit. We still keep all of our chunk data in memory but we keep it in in file storage instead of RAM. What this means is that the data is not as easily manipulated or accessed, however the gain being that we can have much larger worlds while using much less memory. The size of the world will only affect file storage and not our RAM usage.
I expect that the multiplayer servers will benefit greatly from this change. The down side of it might be that a queue system will likely be used for file access since you can't write to hundreds of files at once. Still I don't think this will be a problem and if needed the server will be able to use a lot more memory than a client since it doesn't have to worry about graphics, and all that other jazz.
When Will The New Chunk System Be Finished?
I have most of it already completed but I am working out a few more bugs as well as making some last minute optimizations. It will definitely be present in the next stable release.