TogetherAI API, updates on scoreboard, and minor scoring issue

[SaTML 2024 LLMs CTF Announcements]

Dear LLMs CTF participants,

TogetherAI is having issues right now. This is outside our control, and reproducibility concerns prevent us from changing the LLaMA provider.

We thus decrease \beta for TogetherAI models to 1/(60 * 60 * 96), corresponding to a linear decay of b_D over 4 days.

The \beta for OpenAI models stays the same. Please report if there are issues unrelated to the TogetherAI API.

We’re going to restart the platform for about 2 minutes at Feb 5 2023 16:00 UTC; that is, in about an hour, for minor updates:

• Fix scoreboard so it displays total score for the attacking teams correctly.

• Adjust \beta. (Existing scores will get recomputed.)

• Add secret_id to what is returned by /api/v1/chat/attack/{id}.

• Update /api/v1/chat/attack/ to add the option to get the chats of the teammates and filter them by submission id.

Update to final scoring: The current scoring unfairly advantages attacking teams that do not have two defenses of their own, as they can attack more defenses than the other teams. Instead of summing the attack scores, we change the final score to be the sum of best N-2 scores, where N = 44 is the total number of defenses. For attacking teams that have two defenses, nothing changes, as their score on their own defenses is automatically 0 and hence they can attack at most N-2 defenses anyway.

Best of luck!

[SaTML 2024 LLMs CTF Announcements]

The server is up again. The leaderboard is fixed. TogetherAI also seems to have recovered in the meantime; LLaMA chats are working for us.

Happy jailbreaking!