list of bugs and wishes

[Pete Mack]

Right. My bad.

[PowerWyrm]

Note: I've already failed a spell with 1% failure rate five times in a row

[Sky]

Ran some tests: created a human mage, advanced him with debug commands to lvl21 and enough int to get 4% failure rate, and cast detect monsters 335 times. Got 14 failures. p-value of 0.47. Verdict: the RNG seems to work OK. Complaint "refuted by an appeal to advanced mathematical statistics".

All you've proven is that under controlled test circumstances the rng doesnt fail.
I hope it's evident why that isnt enough.

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results.

Empyrical observation does have its merits, y'know.

Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.

[fph]

All you've proven is that under controlled test circumstances the rng doesnt fail.
I hope it's evident why that isnt enough.

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt. And god knows what it is. What is the rng based on anyway, system clock? That has to go through the OS which could unwittingly manipulate the results.

Empyrical observation does have its merits, y'know.

Oh btw congrats PowerWyrm on a 1:10,000,000,000 chance.

As far as I understand (I didn't write the code) the RNG does one call to time() in Rand_init() when the game is started, then it's all state-based. If I read the branches correctly it uses this RNG.

[Nick]

Everyone has reported unusual rng results. At some point you gotta accept that it's not an epidemic of bias and something in the code is doing something it shouldnt.

Actually, no. Humans are built to recognise patterns; if there is no actual pattern, we will still find one. The correct thing to do is to have a mathematically sound source of randomness, check the implementation from time to time, and otherwise accept that people will continue to tell you that the RNG is broken

[Pete Mack]

Nick nailed it. A 1 in 10,000,000 chance for an event that is not specified in advance is not actually 1 in 10,000,000 at all. And it is happening in a game with roughly 100k moves, which reduces the per-game chance by a lot. And it is happening to one player on a forum, rather than one in isolation. We have to *expect* to see these events fairly regularly, be it a 1% fail repeated 5 times, or a ring of speed +24...or Dwarven PDSM.

[Sky]

Civ 2 lied to everyone for years and it turned out that the computer *was* in fact cheating. The same did BloodBowl.

Anyway, the "people find patterns" applies to randomly tested unaware people, and i must point out that .. we are none of those. I put it to you that the average angband player is far more likely to have confidence with maths and abstract concepts than your typical God Of War player.

Im sorry pal but when yet again i fail 7 times a 5% spell, those things dont go unnoticed.


If you guys Really want to do some testing... play a game. A whole game, from town to morgoth. And write down everything.
You've done your lab test, now it's time for field testing.

[Pete Mack]

We do play the game. And we recognize both confirmation bias and reporting bias. If want to test the RNG, put it through an autocorrelator with N = 100. The bottom line is the Mersenne twister has been tested by people who know a lot more statistics than you and I.

[Philip]

As Pete Mack notes, there are actually two cognitive biases occurring here. The first is the one that convinces people that failing a 5% spell 7 times in a row during gameplay is a statistically reasonable trial from which to derive that the RNG is failing. The second is, of course, "everyone has reported unusual rng results". I have not. I imagine many other people on this forum have not. However, those that do are more remarkable, and so they seem more common than they actually are. In order to perhaps correct that perception, I will now report RNG results.

I just started a new mage run today, inspired by this thread. That character is now clvl 26 and with 18/70 INT, and has 4% fail on detect monsters. I made a keymap that casts detect monsters 5 times. I then pressed it many many times, counting each time that it failed. It failed 13 times within the first 335 casts (I cast it a bunch of times after as well because I was running through my whole mana and it didn't fail after, but those don't count because I decided to run 335 casts). I'm not actually any good at stats, but I'm pretty sure 13 failures is what you'd expect from a fair dice roll. There were no apparent anomalies during the test, though since I wasn't testing for those, it doesn't really matter.

[Pete Mack]

Philip--
I suspect you have posted AARs that support reporting bias (assuming you post AARs at all.) Every crazy speed ring, and every YASD that occurs because of repeated failure. By and large people post AARs when something unlikely happens, not something ho-hum. I do not mean to imply that everyone thinks the RNG is broken and is actively busy generating reporting bias.

[Philip]

Oh, for sure in that sense. I was more talking about the bias that causes Sky to suggest that reports of (perceived) problems with the RNG are very common, even though they're really not, though they are rather memorable, especially if you're already inclined to believe the RNG is bad.

[fph]

If you guys Really want to do some testing... play a game. A whole game, from town to morgoth. And write down everything.
You've done your lab test, now it's time for field testing.

Looks like a reasonable test to me. Do you volunteer to run it, since we are fairly sure ourselves that the RNG is right? It would be a valuable contribution. I guess this could even be automated with a couple of fprintf() in the right place.

I suggest a couple of rules to ensure fairness:

* write a forum post before starting, detailing which events you are going to track (for instance, spell failure rates). Only "pre-registered" characters count.
* report the results after your game no matter what they are and no matter what happens to the character (e.g., inglorious death at 500' by the hands of Wormtongue).

[Werbaer]

Im sorry pal but when yet again i fail 7 times a 5% spell, those things dont go unnoticed.

The last time i noticed i failed 3 times in a row at a 5% fail, i then realized i was stunned.

anecdotical evidence:
When i fight Morgoth as a mage, i always use Mana Storm exclusively. Depending on my level and the fail rate, i needed 66 - 78 Mana Storms.
I usually note the real and expected fail rate in my final dump. Fail rates:
32% (expected 35%)
34% (expected 26%)
37% (expected 32%)
23% (expected 23%)
17% (expected 20%)
So it looks like the RNG works correct.


Additional info regarding the RNG:
If you fail 7 times in a row at 5% fail rate, this doesn't even mean that the RNG did a low roll 7 tims in a row. Unless that happened in an empty town, there were probably 10 - 100 rolls for monster actions between your castings.

[Derakon]

Even in the town, the RNG deciding whether and where to spawn more townsfolk.

[Vivit]

Crashing during standard roller chargen

If I repeatedly alternate between pressing 'p' for "previous roll" and 'r' for "new roll" (to keep an old roll and compare my new rolls to it) during standard roller character generation, the game eventually crashes, sometime after only a couple rolls. No error message is shown by default, but when I traced the process with gdb, gdb tells me that the program exited with SIGABRT because of corrupted memory or a double free.

Can anybody else reproduce this? This is in 4.1.3.