Saturday, September 27, 2008

Me, Myself, and AI

(I just made myself cringe writing that title. I'm sorry. The following is reposted from my introduction at the AiGameDev forums.)

Hi, I'm David. I've been programming games for fun for as long as I can remember and professionally for the last two years.

AI became a passion of mine largely due to its mystifyingly transparent nature. Even before I understood the math behind it, I could look at a textured 3D model in a game and understand that it was made of triangles and texels and projected onto the screen. AI, by contrast, was just this magical black box, responding in sometimes wildly unpredictable ways to the player. Playing Thief was a landmark moment for me; although its AI was generally predictable and could be gamed with relative ease, it sold its characters as believable human beings with a combination of interesting behaviors and a near constant stream of barks that relayed everything the player needed to know about the AI's state. Years later, when I had begun to study and implement AI behaviors, I revisited Thief and began to see how AI was not simply a programming endeavor but a cross-section of all disciplines of game development, a massive collaborative effort of engineering, animation, design, and audio.

I attended the outstanding Guildhall at SMU program as a way to catch up on professional-level game programming and eventually break into the industry. Notably, my application to the school and both of my individual projects were focused on AI. For my application, I wrote a little grid-based "mini-Thief" game. It was my first time implementing (and even learning about) the A* algorithm. Eye-opening stuff at the time. Halfway through my education there, I revisited the Thief concept in 3D and with more complex group behaviors. As I scoured the internet for interesting AI behavior topics for my Master's thesis, I came across Jeff Orkin's publications on goal-oriented action planning. I was intrigued and ended up writing an UnrealScript-based implementation of GOAP and building a hierarchical squad behavior system on it. (Jeff even agreed to advise on the project--thanks!)

Then I entered the real world of professional game development. Of course, I didn't expect to jump straight into an AI dream job, and in fact, I barely touched gameplay code let alone AI code at my first job. I was disappointed at first, but it gave me a great opportunity to explore other aspects of game development that I had largely overlooked before (especially audio, but that's another story). In the meantime, I took on two personal ventures: first, a hobby engine that would eventual require some custom AI, and second, writing an article for the fourth installment of the AI Wisdom series. For the latter, I pared down my Master's thesis, focusing on the original aspect of hierarchical squad behaviors using GOAP. (Again, thanks to Jeff for encouraging me to do it, and thanks to Kevin Dill for his assistance on our section of the book.)

I've continued to work part-time on my hobby project, and finally, more than a year later, I'm beginning to implement some AI. I've just got pathfinding over navigation meshes working and am evaluating whether I want to use GOAP again or whether Alex-style behavior trees might be better suited for my needs. In the meantime, I've found an awesome new job where I'm able to stretch my engineering muscles and work on a huge amount of exciting tasks including some fun AI behaviors. With this recent rekindling of my interest in game AI, I decided it might be a good time to check out the community here and get involved.

Tuesday, September 9, 2008

Content Pipelines and Build Processes (Part One?)

Perhaps the most important thing I've learned since graduating from the Guildhall and working in a couple of professional studios is the effectiveness of a good content pipeline and build process. These are things that were mentioned in passing at Guildhall lectures but that didn't register with me at the time. Since then, two things have happened to elucidate to me why content pipelines matter. The first was working with a bad pipeline and worse build process at my former company. The second was when I needed to roll my own pipeline for my hobbyist engine.

To be clear, what I mean by content pipelines and build processes are the answers to the questions "How do you get data into your game?" and "How do you produce code and data builds?" respectively. The two problems are closely related, but build processes are sometimes limited in scope to only building code, with data being processed at a different time (especially when baking content for consoles).

Given the popularity of the Unreal Engine, it's worth looking at how their content pipeline works. The decision was made long, long ago for the original Unreal that the game's raw and baked files should be the same (at least for levels and asset packages). There's a lot of interesting implications to that, and it's probably worth arguing whether the original rationale for this decision is still valid today, but that's the way it has stayed for over ten years of development of the engine. What it means with regard to the pipeline and build process is that there is no process.

In Unreal, content creators import raw files (textures, meshes, etc.) and produce package files--binary bundles that blur the line between raw and baked because they can be edited in UnrealEd but are also ready for use at runtime. Likewise, level files never need to be exported or baked into any other form; the raw level files that designers work on are the same files that the player loads in the finished game. Of course, Unreal does have baking of a sort, but it happens in-editor, when the designer clicks the Build button, instead of as an automated process, and the product is saved into the original file. For PC-based Unreal games, that's more or less all there is. Iteration time is fast, because all the data is already in its final form. For console games, the data will be baked for the target platform, which means fixing the endianness, building seek-free packages for fast loading, etc. It's a bit slower but not terrible, and the baked data can be cached and only rebaked when it changes, so iteration time is still relatively light.

In stark contrast to the Unreal model is the pipeline and build process used at my former place of employment. We used an in-house-developed engine, and there seemed to have been no clear vision as it grew, resulting in a monstrous pile of conflicting tools and methodologies. That doesn't mean that the general design of the pipeline was bad, it was just burdened with years of baggage from hastily-made, myopic decisions.

For most assets in this engine, the raw files were standard filetypes (TGAs for textures, AIFF for samples, INIs for configurations, etc.). The rest were custom text formats for worlds, particle definitions, and other non-standard asset types. (Unrelated note: As far as I recall, there were no custom binary formats; however, the text files that the world editor produced were indeterminate, with the contents being ordered apparently randomly each time the file was saved.) Because the raw files and baked files were different formats, every file had to be baked before it could be viewed in the game. That's pretty common. Making it more difficult was a complex dependency chain that I have to assume was a too-early step to produce seek-free packages. If a texture was changed, it was not merely enough to rebake the texture; every world, mesh, or UI element that referenced that texture needed to be rebaked. And there was no automated dependency checking. And the dependency graph was so convoluted that no programmer on the team understood it, not to mention the content creators. The implications? Almost every time a content creator changed anything, they rebuilt everything. That was an iteration time of almost two hours, sometimes just to verify something as simple as changing a texture.

Ironically, I decided after that experience to implement a very similar pipeline in my hobby engine. It's a fairly obvious paradigm: develop raw files, bake them, run the game with the baked files. Having noted that the problems I observed on the aforementioned project were due to other issues than this core design, I figured that I could implement a simple system this way and use my experience to avoid the pitfalls. In practice, it has worked surprisingly well. Given that my only target platform is the PC, I don't have to worry about seek-free packages, so there are no dependency issues in the build process. When I change a texture, I only need to rebake the texture file. Worlds and meshes simply reference the texture by name instead of storing the texture inline. (As an aside, it seems that it would be far simpler to do this seek-free stuff as a late post-bake step, the way Unreal does it. I haven't thought it through completely, but I can't see why it would be done any earlier.) I use timestamp checking to determine what needs to be rebaked, so builds are minimal. Iteration time is only an issue when building world files, because of the amount of heavy calculations they need for lighting. Most other changes take less than a second to build. After baking, I can either use the loose baked files to run the game or zip everything up into compressed package files.

There's more I could say on this subject, so I'll probably be following this up with a second part eventually. I'd like to write about the horror of the automated build process on my last project and delve a bit further into my own system. I'll also try to revisit the loose ramblings from this article and tie it together into a more concise evaluation of how each engine answers the questions I posed at the start.

For more information on content pipelines (including some refreshing examples of pipeline design with some forethought), check out these slides and audio from Gamefest 2008: Next-Gen Content Pipelines: A Study of 1st Party Titles