Semi-automated Judging Assistant

[James Bowery]

In alpha test:

https://github.com/jabowery/HutterPrizeJudgingAssistant

README.md

Hutter Prize Judging Assistance

This repository minimizes the manual work required to evaluate a Hutter Prize submission. judging_assistance.sh rebuilds and runs the submitted software, enforces the technical resource limits, verifies the result, and records the evidence and proposed score for human review.

Entrant-provided processes run within Docker containers to minimize the risk of an adversarial entry. The security model is conditioned on a Linux host whose kernel is hardened against container escape.

The authoritative rules remain the Hutter Prize detailed rules. Entrants should follow ENTRANT_INSTRUCTIONS.md.

Run

From the repository root:

./judging_assistance.sh \ --work-root /mnt/large-disk/HutterPrizeJudging \ Entries/NAME ./enwik9

The work filesystem must have at least the configured 100 GB allowance. By default, submitted-archive qualification and rebuilt compression may overlap (--jobs 2). Use --serial when a disputed CPU timing must be repeated without concurrent work. --geekbench-score N reuses a separately verified Geekbench 5 single-core score.

The current Docker worker executes Linux x86/x86-64 entries. The manifest also defines Windows x86/x86-64 names so the same orchestration contract can be used by a native Windows worker; this Linux worker rejects a Windows manifest rather than running it under an unscored compatibility layer.

Automatic host initialization

Invoke judging_assistance.sh as an ordinary user. When necessary, it installs Git LFS through the separate install-host-dependencies.sh helper, materializes the required repository objects, and re-executes the trusted host orchestrator through sudo to access Docker. The customary password prompt is the only required interaction. Results created by the elevated process are returned to the invoking user's ownership.

The Docker socket must not be made world-writable; access to it is root-equivalent. Contestant executables do not receive that access and run as UID 65532 in their execution containers.

Before any entrant-provided code is unpacked, built, or executed, the orchestrator runs a host-security preflight. It rejects a non-Linux daemon, a nonlocal Docker endpoint, a daemon using a kernel other than the host kernel, inactive seccomp filtering, failure to apply no-new-privileges, or the absence of a verifiably enforcing AppArmor or SELinux container profile. The checks follow Docker's documented capability and kernel-isolation model and verify the resulting test container rather than relying only on daemon configuration.

The preflight records whether rootless Docker or user-namespace remapping maps container UID 0 away from host UID 0. Their absence currently produces a warning: entrant executables still run as UID 65532, but that is not a separate user-namespace boundary. The preflight also warns that local inspection cannot prove the absence of an unpatched host-kernel or Docker Engine vulnerability. Its complete findings are retained as host-security.env in the results tree.

[James Bowery]

Some questions have arisen about this automated judging assistant.  The reasons are manifold, but to get a start on what will likely be an associated FAQ:

Q:  Is this a new rule?

A: No.  This is an alpha test of a tool.

Q: Will it become a new rule?

A: Probably not in its present form if at all.

Q: Will my entry be given priority over entries that do not comply with its instructions?

A: Not in terms of ultimate judgement but perhaps in terms of the timeliness of the judgement.

Q: Then, what is the purpose?

A: In no particular order (without attempting pretend the reasons are independent) to:

* better communicate the rules.  It is one thing to express a rule in natural language.  It is another to express it in an algorithm.

* reduce the time spent by judges in reaching a decision.

* reduce the time spent by judges explaining the rules.

* reduce the risk of adversarial attacks on those testing a submission -- judges and 30 day commentariate alike.

* deal with the 30x increase in rate of entries.

* avoid a rule that requires contestants to rent cloud services to achieve the foregoing.

The last of these is particularly important given that the Hutter Prize has always been about remaining open to contestants who are skill-rich but cash-poor.