Stats collection

[Magnate]

Hi all. myshkin and fizzix each independently wrote some code for collecting stats in V, and we've been discussing how to harmonise it into a single set of functions that will be useful to everyone. So this thread is to gather views from variant writers and others who are interested in stats. We're talking about monte carlo stats, i.e. gathered from large numbers of simulated levels/monsters, not derived algorithmically. In summary it should work a bit like this:

1. We generate each dungeon level from 1-100

2. On each level we collect a percentage of the floor and chest loot, and a percentage of the monster drops, according to how fast we're diving. So 100% is full level-clearing, while (say) 10% is extremely fast diving. This isn't quite how it works yet - when it gets more sophisticated we can use monster power to determine which drops we might collect, and compute path distances to determine which floor loot, but for now we represent diving by just clearing every Nth level and skipping the others completely. (N.B. We don't generate any summoned monsters or their drops - yet. We don't read acquirement scrolls either.)

3. We unkill all the uniques, un-make all the artifacts, and start again. After tens of thousands of runs, you have vaguely useful stats. After a few million, they become quite reliable (subject to the limitations of the RNG).

Now, a single run of full-clearing generates about 5000 items and 10000 monsters, which is about 500k of raw data. So we need to do some aggregation of the data to avoid ending up with gigabytes of raw data from a single sim. I was thinking that a data structure could contain all the data for a specific level - so we'd need a hundred of them, which get added to each run:

struct level[100] {
    int kinds[z_info->k_max][ORIG_MAX]; 
    int egos[z_info->k_max][z_info->e_max][ORIG_MAX]; 
    int arts[z_info->a_max][ORIG_MAX];
    int flags[z_info->k_max][OF_MAX][PVAL_MAX];
    int races[z_info->r_max]; 
    int vaults[z_info->v_max];
};

That is:
kinds - the number of each base object generated on that level, indexed by origin (floor, vault floor, monster, vault monster, pit monster, unique) EDIT: I forgot that pits now have floor objects too.
egos - the number of each ego type per base item - slight overkill but it's important to be able to distinguish between daggers and blades of chaos for a given ego type
arts - the number of times each artifact is found on that level
flags - the number of times each object flag is found on each item type (as with egos, the item type matters), indexed by pval if it's a pval flag
races - number of each monster type, or number of times unique is seen on that level
EDIT: vaults - the number of times each vault is generated on that level (could easily include pit/nest types too)

I think this structure would enable us to work out most of what we would want to calculate - what's the earliest I'm likely to see resist chaos and on what sort of item, what's the most common depth for a holy avenger, is the pval on speed rings noticeably depth-dependent, etc.

Comments, thoughts?

[Derakon]

Seems reasonable, though your code block is causing a lot of horizontal scrolling.

I think it's also worth generating stats on the number of times each vault or pit is generated.

I'd also like to see stats for just generating randart sets -- how many times do randarts get immunities? Off-weapon combat bonuses? Protection from confusion? Pvals over 3? 4? 5? And so on.

[Magnate]

Seems reasonable, though your code block is causing a lot of horizontal scrolling.

I'll tidy it up.

I think it's also worth generating stats on the number of times each vault or pit is generated.

Nice thought, thanks - easy to add with a structure element int vault[z_info->v_max];.

I'd also like to see stats for just generating randart sets -- how many times do randarts get immunities? Off-weapon combat bonuses? Protection from confusion? Pvals over 3? 4? 5? And so on.

I have thought of this ;-)

Basically the save_object_data() function needs a mode flag, and if we're in randart analysis mode, it simply doesn't store the data of any non-artifact. So the same code gives you exactly the same output, but we're looking only at randarts rather than the whole dungeon full of stuff. That's basically why I added the flags[][][] array to the structure (though it is also useful for normal objects too).

[fizzix]

On level clearing:
-----------------
A sophisticated approach, pathfinding from origin to the stairs and grabbing loot and killing easy monster that are nearby is possible but may not be worth the effort. Especially since calculation time may be important.

The blanket kill 10% of all monsters isn't that good, mainly because a shrewd player is much more likely to seek out certain monsters (read: dragons) than others. Alternatively, certain monsters (Zs and Cs) are much more likely to seek out the player (although they won't contribute to loot).

Of course the biggest factor that will skew all this will be pits. Pits are a prime source of loot since they represent a high reward/time spent. The code should certainly overestimate how often pits are cleared, and should include pits as a possible source for ORIG_MAX. Anyone that's clearing 50% of items should clear 100% of orc/troll/giant/dragon pits. At least that's my guess.

Alternatively nests and graveyards are much less likely to be cleared because of "Q"s and "s"s. Demon pits are also likely to be decimated by horned reapers, so assuming a full clear of a demon pit is probably a bad idea in any situation.

[bulian]

I think this is a pretty cool tool. A few thoughts:

1. Accounting for summoners will be difficult if possible at all. These can have substantial impact on loot, even when exploiting them is not the intent. In particular, I've gotten either several levels or several good artifacts off of unintentional mid game demon explosions.

2. I don't understand why the dungeon generation and randart programs are being coupled - they seem independent to me. Why not run and analyze the randart generation algorithm a large number of times?

3. How fast is the borg code? Can you disable the graphics and run it in tandem to compare the distributions you get? Even 100 borg runs should give some interesting results. This may require a special version where all combat actions (save summons) are replaced by farmer maggot text.

[camlost]

What about borg-derived stats? Obviously, borg-play isn't the same as human-play, but it might produce useful information.

[Magnate]

I think this is a pretty cool tool. A few thoughts:

1. Accounting for summoners will be difficult if possible at all. These can have substantial impact on loot, even when exploiting them is not the intent. In particular, I've gotten either several levels or several good artifacts off of unintentional mid game demon explosions.

Agreed. For my personal agenda, I don't think the extra loot from summons is terribly important, because I'm interested in what happens at a given depth, and summons aren't generally massively OOD. But if someone is keen to add the handling of summons, that would be great.

2. I don't understand why the dungeon generation and randart programs are being coupled - they seem independent to me. Why not run and analyze the randart generation algorithm a large number of times?

The obvious answer is not having to write two separate pieces of generation and analysis code. The more subtle answer is that I'm less interested in what the randart generator puts in the spoiler, and more interested in what actually drops. So it seems sensible to me to use the same stats code, but ignore all other items.

3. How fast is the borg code? Can you disable the graphics and run it in tandem to compare the distributions you get? Even 100 borg runs should give some interesting results. This may require a special version where all combat actions (save summons) are replaced by farmer maggot text.

I thought of this too, but I didn't like the fact that the borg reaches different depths each time he plays. But yes, adding stats collection to the borg would be an alternative source of info. It would take much longer to get the same volume of data, but it would be a closer approximation to real (albeit slow) play.

[fizzix]

The obvious answer is not having to write two separate pieces of generation and analysis code. The more subtle answer is that I'm less interested in what the randart generator puts in the spoiler, and more interested in what actually drops. So it seems sensible to me to use the same stats code, but ignore all other items.

You can end run around this by making the (perhaps faulty) assumption that drop frequency is completely dependent on level and rarity. And then you can generate a table that provides drop frequencies based on levels and rarities only.

After that you can generate N spoilers and get what actually drops based on the randart tables.

edit: however, all this is kind of useless because IIRC, the randart code takes 4-5x more time than descending through 100 levels.

[camlost]

I thought of this too, but I didn't like the fact that the borg reaches different depths each time he plays. But yes, adding stats collection to the borg would be an alternative source of info. It would take much longer to get the same volume of data, but it would be a closer approximation to real (albeit slow) play.

What about adding cheat_death to the borg? That shouldn't be *too* hard, right?

[Derakon]

I honestly don't think that the borg is going to provide relevant statistics here. Yes, the gear he gets "actually drops", but his playstyle is so far removed from normal play that it's not worth basing design decisions on. He's more paranoid that Neo!

[Magnate]

You can end run around this by making the (perhaps faulty) assumption that drop frequency is completely dependent on level and rarity. And then you can generate a table that provides drop frequencies based on levels and rarities only.

If that was safe, I don't think we'd be bothering with monte carlo stats at all. The whole point, IIUC, is to try and gauge the impact of OOD drops and vault items on the overall game.

After that you can generate N spoilers and get what actually drops based on the randart tables.

edit: however, all this is kind of useless because IIRC, the randart code takes 4-5x more time than descending through 100 levels.

Yes, this is a crucial point. The randart generator will be faster when I've rewritten it, but still much slower than the stats generation. Generating N randart sets *will* require a separate stats module, as that's about the occurrence of stuff in the spoiler, not the occurrence of stuff in normal play. But as I said earlier, I'm more interested in the latter first. I don't need many randart sets put through the stats module to tell me whether it's a decent approximation to the standarts in terms of drops.

[bulian]

If that was safe, I don't think we'd be bothering with monte carlo stats at all. The whole point, IIUC, is to try and gauge the impact of OOD drops and vault items on the overall game.

Why do you need to apply the randart generation code then? The frequency and distribution for OOD drops shouldn't matter if the standart or randart set is used, especially if the artifact generation algorithm is changed to make the creation of all artifacts special - is that happening? If its not, then I could see how the use of different base objects from the standart set would affect the distribution. Still, an artifact with base item PDSM should drop as (frequency of PDSM)*(frequency of passing artifact check), so you could just multiply your distributions together to get your predicted randart PDSM generation. Making artifact generation "special" eliminates this whole problem.

It seems you are more interested in the distribution of randart power to depth obtained. After all, the player doesn't know their uber randart MoD with 5 slays had a native depth of 99. So the important factor is understanding how object power affects native depth and frequency in the randart generation code. For that I might suggest altering the depth for a well defined set of items, such as the standart set, and comparing the outcome. How frequent is bladeturner (very rare) if its native depth is 90 compared to 100? Conversely, how does Thorin (very common) change if its native depth is 30 or 50? It seems you can apply those results to the randart generation code to get the type of distribution you want.

Sorry if this is overly pedantic.

[Derakon]

Part of the goal of the stats generator is to verify that we're getting the results we expect to. Just assuming that because the code is written a certain way, the randarts and standarts have similar distributions, is not necessarily accurate -- in fact, there were complaints awhile back that randarts were far too common...and then after a patch, far too rare.

In short, we want to check that "should work this way" and "does work this way" are in alignment.

[bulian]

in fact, there were complaints awhile back that randarts were far too common...and then after a patch, far too rare.

This is a different topic than what I was addressing but it is an example of another use of the tool.

My main point is that the frequency of OOD objects is only very loosely related to whether standart or randarts are used and may be completely decoupled if the artifact generation algorithm is changed.

[Magnate]

This is a different topic than what I was addressing but it is an example of another use of the tool.

My main point is that the frequency of OOD objects is only very loosely related to whether standart or randarts are used and may be completely decoupled if the artifact generation algorithm is changed.

But my interest in the impact of OOD objects on game balance is not limited to standarts vs randarts. When I made the weak standarts easier to find (Forasgil, Elvagil, 'thancs), *how much* easier was that? Did it really mean that one of them would turn up before dl20 every game, for instance, or was it really only every ten or twenty games.

You are right that changing artifact generation to the "all-special" approach would remove all the differences arising from base items - we would then only need to check that randart rarities were approximating to standart rarities. But I would still want to run the monte carlo stats on randart sets.