Functions from data files

[Nick]

I'm currently making an effort to get as much data as possible into text files in lib/gamedata (here's an example I've just done).

Some of this involves getting information at run time to put into functions that have to be defined at compile time. We have two ways of dealing with this:

1. Preprocessor shenanigans, like:
   

   PROJ_ENV(LIGHT_WEAK,	COLOUR_ORANGE,	"light")

   ,
   

   static const project_feature_handler_f feature_handlers[] = {
   	#define PROJ_ENV(a, col, desc) project_feature_handler_##a,
   	#include "list-project-environs.h"
   	#undef PROJ_ENV
   	NULL
   };

   and
   

   /* Light up the grid */
   static void project_feature_handler_LIGHT_WEAK(project_feature_handler_context_t *context)
   {
   	const int x = context->x;
   	const int y = context->y;
   
   	/* Turn on the light */
   	sqinfo_on(cave->squares[y][x].info, SQUARE_GLOW);
   
   	/* Grid is in line of sight */
   	if (square_isview(cave, y, x)) {
   		if (!player->timed[TMD_BLIND]) {
   			/* Observe */
   			context->obvious = true;
   		}
   
   		/* Fully update the visuals */
   		player->upkeep->update |= (PU_UPDATE_VIEW | PU_MONSTERS);
   	}
   }

2. Explicit coupling, like:
   

   name:BA_ACID
   hit:100
   effect:BALL:ACID:2
   dice:15+1d$S
   expr:S:MONSTER_LEVEL:* 3

   ,
   

   expression_base_value_f spell_value_base_by_name(const char *name)
   {
   	static const struct value_base_s {
   		const char *name;
   		expression_base_value_f function;
   	} value_bases[] = {
   		{ "MONSTER_LEVEL", spell_value_base_monster_level },
   		{ "PLAYER_LEVEL", spell_value_base_player_level },
   		{ "DUNGEON_LEVEL", spell_value_base_dungeon_level },
   		{ "MAX_SIGHT", spell_value_base_max_sight },
   		{ "FOOD_FAINT", spell_value_base_food_faint },
   		{ "FOOD_STARVE", spell_value_base_food_starve },
   		{ "WEAPON_DAMAGE", spell_value_base_weapon_damage },
   		{ NULL, NULL },
   	};
   	const struct value_base_s *current = value_bases;
   
   	while (current->name != NULL && current->function != NULL) {
   		if (my_stricmp(name, current->name) == 0)
   			return current->function;
   
   		current++;
   	}
   
   	return NULL;
   }

   and
   

   static int spell_value_base_monster_level(void)
   {
   	int level = 0;
   
   	/* Check the reference race first */
   	if (ref_race)
   		level = ref_race->level;
   	/* Otherwise the current monster if there is one */
   	else if (cave->mon_current > 0)
   		level = cave_monster(cave, cave->mon_current)->race->level;
   
   	return level;
   }

Note that the first one the data is actually not in a data file but in a .h template file, but there are cases where it is and the .h file for the preprocessor is just a link to get the name of the function compiled in.

I was going to ask which of these is better, but on looking at them I think the second one clearly wins - which is a shame, since there's only one of those and lots of the other.

Am I right? Have I missed something? Is there another method (not involving lua, python or whatever)?

[Derakon]

If I understand the question correctly, you're trying to figure out how to identify a) what function needs to be invoked, and b) what parameters to pass to the function, at runtime? If so, then yes, absolutely, the explicitly coupled version is superior to using preprocessor hacks, if only for readability -- the problem I have with the preprocessor directives is that functions get defined/invoked "magically" -- there's not an obvious path in the code to get from point A to point B because the code as written isn't complete.

The value_bases array in the second example is much more like what I'd aim for. It's effectively a manually-created dispatch table, where you use a string to look up the function you want to invoke. Then someone reading the code can say "ah, this function is going to invoke one of these other functions depending on the value of this string"; nice and straightforward to comprehend.

[Pete Mack]

Neither is better. When you want an enum, you must use a .h file, at which point it's better to put all associated info in the same place. Same holds for generated code.
When you want pure data, of course a data file is better

[fizzix]

Neither is better. When you want an enum, you must use a .h file, at which point it's better to put all associated info in the same place. Same holds for generated code.
When you want pure data, of course a data file is better

Why can't you get an enum out of the data file? It would be something like this:

Read through file once and catalog the number of functions of different types
Make the enumerated list
Read through again and get the data

[Pete Mack]

Enums are language constants. There is no way to initialize them at run time

[Nick]

Thanks for the answers to my badly expressed question. I think I had actually answered it for myself by writing it anyway, but this has given me more to think about.

[AnonymousHero]

I think a variation of what fizzix suggested would work:

Write a little code generator which pre-processes the data files into something usable for the C compiler, be it struct definitions or enums or whatever. An added bonus is that the generated definitions/code are easily greppable contrary to the "X macro" pattern.

(Unless I'm misunderstanding the goals here. As long as the values in the data files actually don't change at run-time, what I'm suggesting should work pretty nicely.)

[Nick]

The goal is to make the game as player modifiable - without compiling - as possible. Obviously there are limits, but I want to get close to them. And there is quite a variety of situations to deal with, so one solution won't fit all of them.

[AnonymousHero]

The goal is to make the game as player modifiable - without compiling - as possible. Obviously there are limits, but I want to get close to them. And there is quite a variety of situations to deal with, so one solution won't fit all of them.

Oh, right. Forgot the "without compiling" bit...

In that case you're going to eventually run into Greenspun's tenth rule. The only real way "out" of that is to bite the bullet and embed an interpreter/compiler rather than ending up implementing one's own (inevitably awful) variant of a LISP. However, personally, I don't think doing so is a good idea unless it's a language with LISP-level support for DSLs and meta-programs. Not mention that it's already been tried to limited success in both Angband and ToME2 -- though pretty half-heartedly AFAIUI in both cases[1]. If it were up to me, I'd rather focus on just making it easier to make changes to the C code and making it as easy as possible to compile.

[1] In the ToME2 case the interface between the C bits and the Lua bits were an awful mix of high-level and low-level -- all the C structs were basically just exported completely as-is to the Lua bits. It was incredibly error prone and the dynamic type checking of Lua exacerbated that whenever one had to interface with the C data/code; so we got basically none of the benefits of dynamic type checking, but all of the drawbacks. Not a happy place.

[Derakon]

Note that the solution I came up with for Pyrel for this problem did not involve embedding code into the data files, even though I could have trivially done that as Python has an exec() function that allows it to execute strings as programs*. Instead, if I wanted e.g. a wand that fired three elemental balls in succession, then the wand's object file entry would look something like:

"name": "Wand~ of Examples",
"alloc_depth": 50,
"alloc_rarity": 10,
...
"use_effect": [
  {
    "proc_type": "ball",
    "radius": 2,
    "target": "manual",
    "element": "darkness",
    "damage": "$plvl * (5 + 2d5)"
  },
  {
    "proc_type": "ball",
    "radius": "$plvl / 10",
    "target": "repeat",
    "element": "nether",
    "damage": "100"
  },
  {
    "proc_type": "ball",
    "radius": "100",
    "target": "repeat",
    "element": "hard light",
    "damage": "10000"
  }
]

The code that parsed these files would hand off the "use_effect" section to a separate parser, which would examine the "proc_type" entry to determine which function to call, and then hand that function the rest of the entry (with the radius, target info, element, and damage) which it would use as parameters to decide what to actually do. So somewhere in the code was a mapping of strings to functions that looked like this:

PROC_TYPES = {
  "ball": proc_ball_func,
  "beam": proc_beam_func,
  "all_los": proc_hit_everything_func,
  ...
}

All of those functions had the same signature: they took a "record" parameter, which is one of the mappings loaded from the data file. They were then responsible for extracting data from the record, providing default values for anything that wasn't present, etc.

Really the only fancy part of this is the equation evaluator.

* You should never, ever call this function as it is a security nightmare, but it does exist.

[Pete Mack]

In Python (like other languages with first class types) the compiler is part of the language. The same holds for C#, Java, Haskell. This is a natural way to do these things, unlike the hideous days of Lua, which really was the worst of both worlds.

In the mean time, adding to .h files is much, much less of a chore than actually writing code. As far as I know, the only remaining hard-coded stuff is for elemental resistance, where side effects, vulnerability, and immunity, make things more complicated than the .h file encodes for.

[Nick]

The code that parsed these files would hand off the "use_effect" section to a separate parser, which would examine the "proc_type" entry to determine which function to call, and then hand that function the rest of the entry (with the radius, target info, element, and damage) which it would use as parameters to decide what to actually do. So somewhere in the code was a mapping of strings to functions that looked like this:

PROC_TYPES = {
  "ball": proc_ball_func,
  "beam": proc_beam_func,
  "all_los": proc_hit_everything_func,
  ...
}

All of those functions had the same signature: they took a "record" parameter, which is one of the mappings loaded from the data file. They were then responsible for extracting data from the record, providing default values for anything that wasn't present, etc.

Right, and this is very similar to how the effect code currently behaves, except that there's an entry like this in the wand template in object.txt:

effect:BALL:COLD:2
dice:100

, a list of effect templates like this in list-effects.h:

EFFECT(BALL, true, "dam",	3, EFINFO_BALL, "fires a ball of %s with radius %d, dealing %s damage at the centre")

and then the array of effect handlers is set up like this in effects.c:

static const struct effect_kind effects[] =
{
	{ EF_NONE, false, NULL, NULL, NULL },
	#define F(x) effect_handler_##x
	#define EFFECT(x, a, b, c, d, e)	{ EF_##x, a, b, F(x), e },
	#include "list-effects.h"
	#undef EFFECT
	#undef F
	{ EF_MAX, false, NULL, NULL, NULL }
};

In the case of effects, this probably has the info pushed out almost far as possible, with anything the user will really want to edit in object.txt (arguably the string in list-effects.h should be in a text file too - in particular if we're looking at ease of translating the game - but that wasn't really part of the question).

The question I was asking (and which you answered in your first reply) was whether to use #defines and stringification, or direct association like your PROC_TYPES array, or some third thing.

[AnonymousHero]

Note that the solution I came up with for Pyrel for this problem did not involve embedding code into the data files, even though I could have trivially done that as Python has an exec() function that allows it to execute strings as programs*. Instead, if I wanted e.g. a wand that fired three elemental balls in succession, then the wand's object file entry would look something like:

Right, the tricky part in this scheme is coming up with a suitable "algebra" for combining spells/effects/etc without accidentally ending up with a TC language. (And spells are one thing, effects of wearing items another.)

Even still, this only gets you so far: You don't really get much in the way of being able to mod entirely new flags, say, without an embedded language. (But as I say, I'm not sure it's worth the effort. The ToME2/old Angband way definitely wasn't.)

[AnonymousHero]

In Python (like other languages with first class types) the compiler is part of the language. The same holds for C#, Java, Haskell. This is a natural way to do these things, unlike the hideous days of Lua, which really was the worst of both worlds.

Nope, not for (at least) Java nor Haskell. There's no guarantee that a running problem will have access to the compiler in either of those languages. (After all, the JDK is separate from the JRE. You only need the JRE to run Java bytecode. However, it is possible to generate bytecode at runtime and to use that. Can't recall if something similar might be possible with the GHC Haskell interpreter, but definitely not in general Haskell code. It's possible to load (compiled) code at runtime, though.)

In the mean time, adding to .h files is much, much less of a chore than actually writing code. As far as I know, the only remaining hard-coded stuff is for elemental resistance, where side effects, vulnerability, and immunity, make things more complicated than the .h file encodes for.

I think the point here is that if you use e.g. a LISP (i.e. good for DSLs), it would be entirely possible to keep the data files just as succinct as they are now while also allowing things like changing how e.g. BLINK i monster.txt functions. (Spell effects are mostly hardcoded; see e.g. CAUSE_1, CAUSE_2, etc.)

[Derakon]

Right, the tricky part in this scheme is coming up with a suitable "algebra" for combining spells/effects/etc without accidentally ending up with a TC language. (And spells are one thing, effects of wearing items another.)

Even still, this only gets you so far: You don't really get much in the way of being able to mod entirely new flags, say, without an embedded language. (But as I say, I'm not sure it's worth the effort. The ToME2/old Angband way definitely wasn't.)

Given my experience with Pyrel and trying to make a fully-general equipment/stats/spells system, it's not worth the complexity of implementation. I would definitely stick to trying to support the current use case only, at least as far as data files are concerned. We want to make it easy to add new behaviors, but really mostly only in that we have a big glowing sign in the source code that says "ADD NEW BEHAVIORS HERE". Ideally the dev creates a function, declares a string for it, adds the string/function pair to a mapping somewhere, and then starts using the new behavior in the data files.