APWorldTester/README.md

# Archipelago World Generation Tester
_Disclaimer: AI was used in this repository_

This repository contains a python package that can check whether Archipelago worlds (APworlds) created by the community can generate a valid seed without bringing down the multi-game world.  

The [community worlds spreadsheet](https://docs.google.com/spreadsheets/d/1iuzDTOAvdoNe8Ne8i461qGNucg5OuEoF-Ikqs8aUQZw/edit?gid=58422002#gid=58422002) is a great resource to showcase the games that are in development, but the stability label may not reflect its current state. I wanted to play games that were labeled `unstable` but were at least be stable enough to allow reliable world generation. This tester uses the sheet to collect the latest APworlds and apply a series of tests to assess their generation stability (see `Testing overview`). Only APworlds that are available via direct links or Git repositories are tested.

To see the results of this live in action, see [worlds.archipelago.skillysue.nl](https://worlds.archipelago.skillysue.nl) for an overview of all listed games and their test results.

## Contents
- [Requirements](#requirements)
- [Installation](#installation)
  - [System packages](#system-packages)
  - [Install the package](#install-the-package)
  - [Configuration](#configuration)
  - [GitHub](#github)
- [Testing overview](#testing-overview)
- [Test worlds](#test-worlds)
- [Test all worlds](#test-all-worlds)
- [Test a local `.apworld`](#test-a-local-apworld)
- [Related Projects](#related-projects)

## Requirements

- **Python 3.10 or newer**
- **Git** — used to clone Archipelago and check out the version being tested
- **Docker** — must be usable by your user without `sudo`
- **A GitHub token** — required to access the GitHub API. It does not need any scopes
- **Base ROMs** — optional; some worlds cannot generate without them

## Installation

### System packages

On Debian-based distros:

```sh
# update and install the required packages
sudo apt update
sudo apt install -y git python3 python3-venv
```

Install Docker:

```sh
# Download docker
curl -fsSL https://get.docker.com | sudo sh

# Create docker usergroup
sudo usermod -aG docker "$USER"
newgrp docker

# Test if it can run without sudo
docker run --rm hello-world
```

### Install the package
Install the latest version with pip:
```sh
pip install --no-cache-dir https://git.skillysue.nl/Skilly/APWorldTester/archive/latest.tar.gz
```

Or clone the repo and install it:
```sh
# Clone repo
git clone git.skillysue.nl/APWorldTester && cd APWorldTester

# Create environment
python3 -m venv .venv
source .venv/bin/activate

# Install APWorldTester
pip install -e .
```

### Configuration

Write a starter config and edit the paths in it:

```bash
apworld-tester init
```

That writes the template config to `~/.config/apworld-tester/config.yaml`.
Two settings that require attention:

- `general.output_directory` The output directory for running the tester, including cloning the Archipelago repo and downloaded apworlds. Must be changed, otherwise an error is thrown.
- `general.roms_directory` Directory where base ROMs are located for apworlds that need one. Optional, but increases the number of testable worlds.

### GitHub
The tester checks and downloads hundreds of apworlds from GitHub repositories. It is highly recommended to register a GitHub token and store it in an .env file as follows:

```text
GITHUB_TOKEN=ghp_<your_token_here>
```

## Testing overview

Each world is tested five ways:

- by itself
- with two players of the same game
- with 2–5 of Archipelago's built-in worlds
- by itself with its options randomized
- with other worlds and its options randomized

Each test is repeated with different seeds. The results are combined into one of six stability values:

| Value | Meaning |
| --- | --- |
| Stable | Every attempt succeeded |
| Minor issues | Solo and multiworld tests worked, but a randomized test failed |
| Flaky | Some attempts succeeded and others failed |
| Solo only | Works by itself, but fails with other worlds |
| Broken | Could not generate at all |
| Unknown | No usable result, for example because the `.apworld` was missing or a ROM was required |

A successful test means the world loaded, generated a seed, and computed the spoiler log at the configured level. With the default `playthrough` level, this also checks that the seed is completable.
 

## Test worlds
The `TestRun` class can be used to test one or more worlds:

```python
from apworld_tester import TestRun

test = TestRun("Baba Is You")
test.run()
```

To test multiple worlds, supply the names as a list:

```python
test = TestRun(["Baba Is You", "Anodyne"])
test.run()
```

To test all worlds that have already been downloaded, add `cached=True`:

```python
test = TestRun(cached=True)
test.run()
```

The first run clones Archipelago and builds the Docker image, which can take several minutes.

## Test all worlds
The real benefit of the tester is to parse the community worlds spreadsheet and re-test all changed apworlds. The `UpdateRun` class does exactly this. It updates Archipelago if needed, refreshes the community worlds spreadsheet, downloads and tests changed worlds:

```python
from apworld_tester import UpdateRun

updater = UpdateRun()
updater.run()
```

## Test a local `.apworld`
In case you want to test a world for which you have an apworld locally, you can supply a directory instead. First, put the `.apworld` in its own directory:

```text
~/my-apworlds/
└── My Game/
    └── my_game.apworld
```

Then run the following:

```python
from apworld_tester import TestRun

test = TestRun(
    game="My Game",
    root_directory="~/my-apworlds",
)
test.run()
```

## Related Projects
- [Community Spreadsheet](https://docs.google.com/spreadsheets/d/1iuzDTOAvdoNe8Ne8i461qGNucg5OuEoF-Ikqs8aUQZw/edit?gid=58422002#gid=58422002): A curated list of games and resources. Without the spreadsheet, this work would be impossible.
- [Archipelago Games Library](https://mk-404.github.io/Archipelago-Games-Library/): A list of available games and tools made by the community. Very player friendly.
- [Archipelago fuzzer](https://github.com/Eijebong/Archipelago-fuzzer): A set of scripts to generate multiworlds to record failures.