CityRP Documentation Repository

How integration test cases stay independent

Last updated

Status: Accepted · 2026-09-15 · charted by #1372

An integration leg is one server process, one Lua state, one tmysql4 connection and one throwaway MariaDB database, and every suite in the leg runs against all four. Each leg already gets its own database (#1368), so this ADR is about what happens inside a leg. The bar is the one #1372 set: any single suite, run on its own, gives the same result as it does in the full run.

The decision

Cases own their data, and the seed is read-only

  • The seeded rows are a fixture every suite reads. No case changes them.
  • A case that needs data creates its own: bots from CityRPTestBots.Add, its own clan, property or item rows.
  • Reads are scoped to the identities the case created. Counting or scanning a whole table picks up whatever the last suite left behind.
  • A case removes what it wrote. Cleanup is required, not best-effort (see the rejected alternatives for why "only write rows nobody else reads" is not enough).
  • Data a connected player owns is written through the gamemode, not raw SQL. The 60-second saveDataAll upserts every connected player's in-memory data over their row, so a direct UPDATE lasts until the next save.
  • Data that is not keyed on a player (clans, properties, anything loaded into memory once) is the suite's to tear down, rows and in-memory state together. Whether it does that with a gamemode reload or its own teardown is its call.

The guarantee holds between groups

GLuaTest runs a group's cases in file order, so dependence between cases in one file is deterministic and cannot make a suite behave differently alone. A group may share a beforeAll fixture across its cases and remove it in afterAll. Cases should still clean up after themselves, but the guarantee the harness enforces is at group level.

The harness cleans up, drains and checks

Integration runs only; the unit run is unchanged. A scope is a case, or a group for whatever its beforeAll created. A group a suite builds and runs itself while a scope is open, such as a probe group testing the harness, belongs to that scope and gets no scope of its own. When a scope closes, the harness:

  1. Runs the suite's own cleanup and afterEach (or afterAll), with errors caught.
  2. Kicks the scope's bots and waits until they have left.
  3. Removes the entities created during the scope, minus an allowlist of classes the gamemode spawns on its own schedule.
  4. Drains: waits for the queries and timer.Simple calls started during the scope, and whatever they start, capped at 10 seconds. It reuses the counting TrackLoads already does at boot.
  5. Purges the departed bots' identities: every row keyed on their SteamID(), SteamID64 or UniqueID.
  6. Drains again, because the purge's own queries count.
  7. At group end, runs the leak check below.
  8. Releases the runner to start the next case.

The order is load-bearing. The kick has to come before the purge, because the 60-second save re-inserts the row of any bot still connected. The drain has to come before the next case, because GLuaTest starts it synchronously from done() or the timeout, and a late callback would otherwise land inside an unrelated case. If the drain hits its cap, the case that owns the outstanding work fails, not the one after it.

The purge finds its tables from information_schema, by column name (steamid, steam_id, _SteamID, steamid64, _UniqueID and the like), against an explicit exclusion list. A column that looks like a SteamID but is on neither list fails the run before any suite starts, so a new table cannot quietly escape it. The purge refuses any identity that appears in the seed. It sends one probe asking which of those columns hold a departed identity, then deletes from only those, one at a time to avoid deadlocks: every query waits a server tick for its callback, so a delete per column cost about a second on every case that used a bot.

Hooks and timers a case body registers are GLuaTest's to remove: each case runs with its own hook and timer tables, and GLuaTest cleans them up. Those registered from beforeAll or a file-level helper are not covered, so the leak check reports them instead. The shared helpers in CityRPTestHelpers run outside the case too, so the harness's own tracker waits for the timers they start instead.

A leak fails the group that caused it

Before a group's beforeAll, the harness snapshots, and after its afterAll it compares:

  • connected players and entities
  • hooks and timers whose function comes from a file under tests/
  • the row count of every table except logs, which grows in the background
  • CHECKSUM TABLE for the seeded tables (players, inventory, lite_characters, clans, clan_ranks, clan_members, bans), which catches a changed seed row that a count cannot

A difference fails that group with an added case naming what leaked ("1 bot, 3 rows in inventory"). The harness force-cleans the bots and entities it can and the run continues. It cannot force-clean rows without knowing which ones, so a later group can still be affected; the shuffle seed below is what makes that reproducible. A table joins the logs exemption only after a false positive is actually seen, with a comment naming what writes it.

Order dependence is surfaced rather than hoped away

  • GLUATEST_SUITES runs the named suites only, as project-relative paths without .lua, so any suite can be run alone locally. An entry that matches nothing fails the run. GLUATEST_SKIP_SUITES drops suites in the same syntax, which is how CI's remainder shard runs everything the other shards of a leg do not.
  • CI shuffles group order with its own seeded generator (never math.randomseed, which would change gamemode randomness), prints the seed, and replays it from GLUATEST_SHUFFLE_SEED. Local runs shuffle the same way unless the seed is off.
  • The harness checks, the suites covering the harness itself, always run first, in order, and are never filtered out. database.lua asserts the pristine seed, and a broken harness makes every later failure noise. CityRPTestSuites.HarnessChecks holds the list.

The harness leans on GLuaTest internals, so the version is pinned

GLuaTest 0.35.2 has no hook for the end of a case or group, and starts the next case synchronously. The harness therefore wraps the loaded group tables (beforeAll, afterAll, each case's cleanup) in GLuaTest_TestGroupLoaded, and wraps GLuaTest.TestCaseRunner and GLuaTest.TestGroupRunner in GLuaTest_StartedTestRun only to hold their callbacks until cleanup finishes. The group runner is wrapped too because the next group starts from the same call that runs afterAll, so nothing else can wait after it. None of GLuaTest's code is copied.

That depends on field names, global names and cleanup order, so GLUATEST_IMAGE is pinned to 0.35.2 and the harness fails the run on any other build. The image reports its version as unknown, so the check compares checksums of the GLuaTest files the harness wraps. Upgrading is a deliberate change that re-checks the wrapping.

GLuaTest runs beforeAll, afterAll, beforeEach, cleanup and afterEach without error protection, and an error in any of them stops the run from ever finishing: it ends as a 20-minute job timeout rather than a named failure. The wrapping catches those errors and fails the group instead, and every piece of the harness's own cleanup is protected the same way.

Rejected alternatives

  • A transaction per case. The gamemode sends every query through one shared, asynchronous connection, and it keeps starting queries of its own (the 60-second save, the bans refresh) that would land inside whichever case's transaction happened to be open.
  • Truncate and reseed between suites. Clans, properties, server rules and contraband tracking load into memory once, on DBConnected, with no working reload, and meska.achievements.load appends duplicates when run again. Resetting the tables would leave memory and database disagreeing, which is a worse starting point than the leak it was meant to fix.
  • A fresh server and database per suite. Every leg already boots twice (it migrates, then changes level), and doing that per suite puts the job's 20-minute limit in reach as the suite count grows, for isolation the rules below get without a reboot.
  • Scoped identities with no cleanup. Loaders that read whole tables break it. The bans refresh pulls every row into _allbans each minute, so another case's ban is visible to everyone.
  • Pausing the background writers while suites run. Deterministic, but it hides the save races #1365 set out to catch.
  • Removing hooks and timers by call-stack attribution. GLuaTest already removes those a case body registers, and the remainder is small enough for the leak check to report.
  • Replacing GLuaTest's case and group runners. Full control, at the cost of carrying their internals in this repo.

Consequences

  • The harness lands in three tickets: scopes, cleanup, drain and purge (#1434), leak detection (#1435), and the suite filter and shuffle (#1436). Until each lands, the rules it enforces are kept by hand, and CONTRIBUTING describes only what is true at the time.
  • Background writers keep running during suites, so a case can still race the 60-second save. That is accepted as realistic; a suite about those timers calls them directly rather than waiting for one.
  • Nothing documents whether a new bot reuses a departed bot's SteamID64 or UniqueID. The purge makes either behaviour harmless as long as it is complete, and bots.lua pins whichever one a run shows, so a wrong assumption becomes a red case rather than a shared record.
  • Two defects surfaced here make in-memory teardown harder than it should be: !updateclans empties the clan list and reloads nothing (#1437), and meska.achievements.load appends duplicates when run twice (#1438). A suite can use either once fixed; until then it builds its own teardown.