Pyrel dev log

[Sirridan]

I feel bad that I never finished my python port

If you want any help though, I'll gladly throw in my two or three cents. Especially now that my life is much less chaotic.

[Derakon]

Persistent maps can be readily handled by only comparatively minor tweaks to the game:

1) Create a new class, call it something like StashedMap.
2) Set up a mapping of (staircase XY position + dungeon level) to StashedMaps.
3) When taking a staircase, first generate a new StashedMap and stick everything into it (except for the entities that always persist across levels, of course). Then, if we've been to the new level before, use a level generator that pulls info from its StashedMap; otherwise, create a new level.

This gets a bit more complicated if you want non-player things to be able to move from a StashedMap to the current map but should still be doable.

Fair point on the parser being comparatively simple to implement.

For what it's worth, staircases are Terrain instances whose interactions include "go up" or "go down" -- which invoke GameMap.makeLevel(current level +- 1) as appropriate.

[AnonymousHero]

I don't really like the idea of implementing my own parser into Pyrel, mostly because parsers are solved problems -- writing my own doesn't really add anything interesting beyond new bug opportunities. This ought to be a situation in which there's a readily-available library that does what I need (once I substitute in the appropriate values for the strings we care about, of course).

Certainly, however your point about dependencies is a real worry, especially if you don't pick something which has wide support on distros (and Windows + Mac OS X).

My point was simply that a recursive descent parser is about the same complexity as actually specifying the grammar. Unless you're going for efficiency (or a different class of languages that recursive descent can't handle) it's not actually a big deal to just implement this yourself. (Contrary to other types of parsers.)

When we want to generate a new level, we call GameMap.makeLevel(level). "level" here is a numeric argument indicating the depth. Obviously in variants this could be extended to specify which dungeon you're in, or to differentiate between different "level 0" maps instead of implicitly assuming the town. makeLevel() then invokes a function to decide on what map generator to use.

Implementing a highly linked structure, having every map location contain links to all the different exits from the location could a simpler and make it easier for variants to implement highly branching level structures (including persistent levels).

So you'd have something like this:

{ name: "town"
, level: 0,
, exits: {
      "down" : { name: "dlvl 1"
                  , level: 1,
                  , exits: {
                         "down" : ....
                         "up" : (reference back to town level },
                         "..." : ...
                 }
      }
}

The exit names are free-form and are referred to by terrain in the map levels, e.g. ">" in "dlvl 1" refers to "down".

You might want to permit exits which are actually functions/objects which can dynamically compute the map which the exit leads to. This lets you avoid computing the full map structure and do things like persistent levels.

This kind of structure should make it really easy to implement various kinds of map interconnectedness structures -- even things like infinite dungeons that are more interesting than simply going from level 1.. infinity.

[Derakon]

I don't really like the idea of implementing my own parser into Pyrel, mostly because parsers are solved problems -- writing my own doesn't really add anything interesting beyond new bug opportunities. This ought to be a situation in which there's a readily-available library that does what I need (once I substitute in the appropriate values for the strings we care about, of course).

Anyway, it's not critical that I tackle that problem right now, so I'm moving on to map generation. The current map generator is completely hardcoded, so obviously it has to go. Here's how I plan the new sequence to work:

We have the GameMap class instance. This holds all of the entities in the game. We create a single instance of this class at the start, and it remains for the duration of the game. It persists across levels largely because there are other entities that also persist across levels, and the easiest way to handle this is to have a "PERSISTENT" container in the GameMap instance that I can add those entities to. Examples of persistent entities would be the player and everything they're carrying; shops; hypothetical monsters that are allowed to chase the player across levels; etc. Note that the GameMap can contain entities that are not "physically present" on the current level.

When we want to generate a new level, we call GameMap.makeLevel(level). "level" here is a numeric argument indicating the depth. Obviously in variants this could be extended to specify which dungeon you're in, or to differentiate between different "level 0" maps instead of implicitly assuming the town. makeLevel() then invokes a function to decide on what map generator to use.

As of v4 we have four options: town, standard dungeon, labyrinth, and cavern. However, given that we've talked about mix&matching these generators (having levels that are largely "vanilla" but mix in a bit of natural caverns, for example), it seems likely that instead each "generator" will consist of calling a selection of functions that form the actual map. For example, everyone will share the same magma/quartz seam generator code; likewise, everyone needs the same rules for deciding if down staircases are generable (e.g. because the player's at level 99 and Sauron still lives). Still, a high-level function will be needed to decide how those different sub-functions are organized and invoked. And of course we need different rules for placing monsters and items as well.

We'll need an Allocator class that can handle picking monsters and items -- not just for level generation, but also for item drops and summons. Monsters and items use the same basic "rarity" mechanic, so the Allocator can iterate through the entirety of the monster/item records, find everything that's in-depth, sum up their rarities, and use that to decide how common various things are with respect to each other. I believe Magnate said this was called something like a "table selector" in Vanilla's code. Actually what will probably be done is that every monster/item in the game is added to the table, but they get multipliers to their rarity based on how in-depth they are. For example, every additional level of out-of-depthness makes a monster 25% more rare and an item 50% more rare. Or whatever.

Actual placement of the monsters/items in the level is up to the generator, of course; the allocator merely selects what should be placed.

As far as items are concerned, here's how I think generation to be done:

* Compile the frequency table for all items.
* Append the frequency table for all artifacts to this table.
* Select from the table. If it is a non-artifact equipment item, hand it off to the affix/promotion system, otherwise just use the item straight.

I'm not entirely clear on how exactly artifacts are allocated in current Vanilla, but this system I'm proposing would let us directly compare the frequency of a given artifact to the frequencies of normal items, which I don't believe is currently possible.

(Of course, I'll be postponing artifacts for the forseeable future, as well as the affix system. They're in that big bucket of things that I consider to be "fluff", which also includes stuff like proper combat calculations, actual creature AI, the look command, etc.)

Any thoughts?

[AnonymousHero]

A recursive descent expression parser/evaluator should be pretty easy to do. If you have a grammar without left recursion you can translate it pretty directly into simple python code. The same wiki page has an example grammar of which a subset is expressions and the necessary C code to implement expression parsing. (Though I wouldn't recommend a direct translation of that code -- it looks pretty old-school what with global state, etc.)

Tokenization is easily handled with regexes. (A simple dictionary of regex->token type should work ok for a simple expression language.)

[Derakon]

Let's say you have "<actor.attack> + 1d20". Do all the string substitution and randomizing on the string. So, you get for example "3 + 12". Once you're down to only numbers and mathematical operators, run it through eval. This won't require any further parsing, since you only really manipulate single tokens, which doesn't require any context, while the final calculating is done by Python. And I don't think any harm can done by mathematical expressions, so it's safe despite using eval (if done right, of course).

The problem is recognizing e.g. "import os; os.rmdir('~')" in the string you're going to eval.

There may well be a mathematical-expression evaluator that would ban anything else, but it's probably not a built-in and thus would introduce another dependency for developers to have to install before they could work on Pyrel.

[Narvius]

Let's say you have "<actor.attack> + 1d20". Do all the string substitution and randomizing on the string. So, you get for example "3 + 12". Once you're down to only numbers and mathematical operators, run it through eval. This won't require any further parsing, since you only really manipulate single tokens, which doesn't require any context, while the final calculating is done by Python. And I don't think any harm can done by mathematical expressions, so it's safe despite using eval (if done right, of course).

[Derakon]

I assume your use of "actor" means that this generic skill check won't be limited to the player, which is good. This lack of access is exactly the problem I had with my effects theorycrafting, so I've been waiting eagerly to see how you solve this.

It's a tricky problem. The best idea I've had so far is to do the following:

1) Assume the actor is a Creature. This seems fairly safe.
2) Add two skill mappings for the Creature class -- the intrinsic skills and the skills modified by equipment and temporary effects (much like the way item properties are currently handled, and in fact I would probably use the same code).
3) Promote the "boosted die" equation system into a full-on equation evaluator with smart string substitution -- the code would search for key strings like "<actor.magicDevice>" or "<target.savingThrow>" and insert the appropriate skill values. The annoying thing about this is that actually solving the equation would require either eval(), which is a security hole (it amounts to "run this string as Python code"), or writing my own math parser, which seems like overkill.

But I'll keep thinking about it.

My other problem was the control of the factors affecting difficulty: presumably the correct place to resolve them all is in the proc itself?

Right. The data files have proc declarations, which may include variable parameters. When the proc is created, it generates a mapping of parameter names to parameter values, which it will need when it is triggered -- but at creation, it may decide to evaluate some or all of its parameters. For example, the difficulty of activating a wand is a parameter that is decided at proc creation, because it doesn't change from one attempt to another. But the amount of damage that wand deals would be a parameter that is decided when the proc is triggered.

[Magnate]

Eventually I'd like to implement some kind of generic "skill check" that would replace the specific "tunnel" proc currently used for the granite wall definition, as well as future procs for picking locks, aiming wands, etc. that all basically boil down to "the player has a chance of succeeding based on some formula." The implementation difficulty would be in how to handle the skill lookup / task difficulty, since the proc definition in terrain.txt doesn't have direct access to the object that represents the actor.

I assume your use of "actor" means that this generic skill check won't be limited to the player, which is good. This lack of access is exactly the problem I had with my effects theorycrafting, so I've been waiting eagerly to see how you solve this.

My other problem was the control of the factors affecting difficulty: presumably the correct place to resolve them all is in the proc itself?

[Derakon]

I played the heck out of Diablo 2, but as soon as I heard about D3 not being moddable (even before we heard that it would be online-only), I lost interest. Unmodded D2 is a pretty straight-up clickfest, but the mods created so much more depth in terms of skill balance, item crafting, mob difficulty, and so on, that pretty much as soon as I tried them I lost all interest in the unmodded game. I fully expected D3 to have a similar problem, and now it's unfixable. Oh well...

Anyway, I went ahead and implemented the generic "replace terrain with other terrain" proc, as well as a first attempt at a message proc that does some basic string substitution (the "<2>" in the string means "replace this with the string representation of the second argument to the proc", which is really rather horribly hackish).

Eventually I'd like to implement some kind of generic "skill check" that would replace the specific "tunnel" proc currently used for the granite wall definition, as well as future procs for picking locks, aiming wands, etc. that all basically boil down to "the player has a chance of succeeding based on some formula." The implementation difficulty would be in how to handle the skill lookup / task difficulty, since the proc definition in terrain.txt doesn't have direct access to the object that represents the actor.

Here's terrain.txt. It's a bit wordy but I think it gets the job done pretty well.

[
    {
        "name": "granite wall",
        "display": {"ascii": {"symbol": "#", "color": [180, 180, 180]}},
        "flags": ["OBSTRUCT"],
        "interactions": [
            {
                "action": "tunnel",
                "procs": [{"name": "tunnel", "difficulty": 2}]
            }
        ]
    },
    {
        "name": "door",
        "display": {"ascii": {"symbol": "+", "color": [128, 64, 0]}},
        "flags": ["OBSTRUCT"],
        "interactions": [
            {
                "action": "open",
                "procs": [{"name": "print smart message", 
                           "messageString": "<2> opened the door."},
                          {"name": "replace terrain", "newTerrain": "open door"}]
            }
        ]
    },
    {
        "name": "open door",
        "display": {"ascii": {"symbol": "'", "color": [128, 64, 0]}},
        "interactions": [
            {
                "action": "close",
                "procs": [{"name": "print smart message",
                           "messageString": "<2> closed the door."},
                          {"name": "replace terrain", "newTerrain": "door"}]
            }
        ]
    },
    {
        "name": "decorative floor tile",
        "display": {"ascii": {"symbol": ".", "color": [128, 128, 255]}}
    }
]

[Magnate]

The way variable stuff is handled is by using the same "boosted die" mechanic that is used to decide item powers (the "A + XdY MZ" format. Both the terrain flags and the procs can thus have random variance as well as a level-dependent component.

Neat! I hadn't thought of that.

It's a good job you don't play Diablo III ...

[Derakon]

I ended up going with the system I described with the giant codeblock example a few posts up (the one with the "preProcs", "procs", and "postProcs" lists). It seemed the least objectionable. I've implemented tunneling and opening of doors; the appropriate terrain transformations occur successfully. At this point I think I'm going to want to create a more generic "transform terrain into other terrain" proc, and clean up how we handle determining if a given terrain is eligible for a given interaction -- but the basics are there, so that's progress.

The way variable stuff is handled is by using the same "boosted die" mechanic that is used to decide item powers (the "A + XdY MZ" format. Both the terrain flags and the procs can thus have random variance as well as a level-dependent component.

[Nick]

We could possibly stick multiple terrain objects into a single tile, representing a locked door as a lock + a door. Then the lock would block all attempts at opening until it is picked (at which point it is destroyed), and the door would block all attempts at moving through the tile. Of course, to do this properly we need to handle multiple actionable items in a single tile -- when you say you want to open what is on that tile, the game needs to first go to the lock, and not the door. This would also make the act of 'l'ooking around more complicated -- how do we know to represent the lock + door pairing as a "locked door"?

Are you regarding traps as terrain? Because if not, you could treat the lock as a trap which needs to be disarmed before the door can open.

[fizzix]

Okay, another post about terrain, with still no actual code written yet. That's okay! Far better to spend X time now thinking of a good system than to spend 5X time down the road rewriting a bad one (and then still having to spend the X to think of a good system anyway!).

I saw Magnate ask on IRC for some information about how terrain fits into the object model, hence the post.

Pyrel's tiles are general-purpose containers that can hold just about anything, with no requirements on quantity or type. So you could theoretically stuff any number of monsters into a single tile if you wanted to -- though behaviorally this is blocked because each monster is an obstacle to movement by any other monster or the player. Similarly, you could have multiple traps, multiple terrain objects, or of course multiple items.

Magnate, you suggested basically having arbitrary characteristics that could be associated with a given terrain object -- for example lock difficulty, or tunnel difficulty, or degree of LOS obstruction, etc. That is absolutely doable; the main question then becomes how we cleanly integrate those kinds of features into the main codebase, which is what I've been discussing earlier.

We could possibly stick multiple terrain objects into a single tile, representing a locked door as a lock + a door. Then the lock would block all attempts at opening until it is picked (at which point it is destroyed), and the door would block all attempts at moving through the tile. Of course, to do this properly we need to handle multiple actionable items in a single tile -- when you say you want to open what is on that tile, the game needs to first go to the lock, and not the door. This would also make the act of 'l'ooking around more complicated -- how do we know to represent the lock + door pairing as a "locked door"?

It seems like you have two options. The first is to do what current angband does and have every possible terrain have a unique entry along with its descriptions. You could imagine this would not be so good because you have silliness like 10 different doors for varying difficulties of lockedness or stuckness.

The second approach, where you allow multiple terrains to be on the same square seems like the better way to go. Especially as this would allow additional effects like lingering poison or fog clouds.

When you attempt to interact with a terrain tile you interact with all the terrains on it. So the door + lock works as you said. To determine what to see when looking you probably need a priority value so you know which to display. In this case door would probably have priority so you'd see just a plain door, or closed door on looking. You won't know it's locked until you try to open it. You could go further and say that the lock is hidden until you attempt to open it, and then it's not hidden. Just like a trap is hidden until you discover it.

If you really want a player to be able to tell that the door is locked, you can allow for a "verbose" command on look that tells you all the known properties of the tile. In the locked door case it might be something like:

• floor
• closed door
• door lock

So you can imagine some of the characteristics that go into a general terrain feature as something like

• name
• look priority (door > door lock > floor)
• action priority (door lock > door > floor)
• strength (i.e. difficulty of tunneling, picking, or damage from poison cloud)
• duration
• movement cost
• sound impedance (we *might* want this someday)
• sight impedance (right now we only have 0, or infinite. But you can imagine something like fog which shortens MAX_SIGHT when looking through the fog)
• projectile impedance (again, right now we're either 0 or infinite)

[Derakon]

Okay, another post about terrain, with still no actual code written yet. That's okay! Far better to spend X time now thinking of a good system than to spend 5X time down the road rewriting a bad one (and then still having to spend the X to think of a good system anyway!).

I saw Magnate ask on IRC for some information about how terrain fits into the object model, hence the post.

Pyrel's tiles are general-purpose containers that can hold just about anything, with no requirements on quantity or type. So you could theoretically stuff any number of monsters into a single tile if you wanted to -- though behaviorally this is blocked because each monster is an obstacle to movement by any other monster or the player. Similarly, you could have multiple traps, multiple terrain objects, or of course multiple items.

Magnate, you suggested basically having arbitrary characteristics that could be associated with a given terrain object -- for example lock difficulty, or tunnel difficulty, or degree of LOS obstruction, etc. That is absolutely doable; the main question then becomes how we cleanly integrate those kinds of features into the main codebase, which is what I've been discussing earlier.

We could possibly stick multiple terrain objects into a single tile, representing a locked door as a lock + a door. Then the lock would block all attempts at opening until it is picked (at which point it is destroyed), and the door would block all attempts at moving through the tile. Of course, to do this properly we need to handle multiple actionable items in a single tile -- when you say you want to open what is on that tile, the game needs to first go to the lock, and not the door. This would also make the act of 'l'ooking around more complicated -- how do we know to represent the lock + door pairing as a "locked door"?