Slight scoring change, other reminders

42 views
Skip to first unread message

SaTML 2024 LLMs CTF Announcements

unread,
Feb 4, 2024, 5:06:46 PMFeb 4
to SaTML 2024 LLMs CTF Announcements
Dear LLMs CTF participants,


Here are some last-minute updates:

  • Due to multiple teams’ feedback, we acknowledge that the current scoring system is suboptimal. Due to the presence of many “easy” defenses, the optimal strategy for teams is to camp at 23:59:59 AoE, and try to get the very first correct guess (200 bonus points), which is 100 bonus points more than the next team that guesses correctly, likely a few minutes later.

    • Moreover, teams living in the wrong time zones might not have a chance to get the bonus for some defenses at all.

  • We decided to make the minimal modification to prevent this, while not hurting teams that prepared for this strategy:

During the first 18-36 hours, the b_D for correct guesses after the first will decay linearly to its previously specified value. Concretely, this means e.g. that if four teams guess the same secret in the first two hours, all will get b_D between 200 and 188.

  • One factor for why we must take action: our server may not survive multiple teams pursuing this strategy.

  • For good measure, we now updated the server to have double the CPUs and double the RAM than it had before.

  • Our TogetherAI and OpenAI accounts have rate limits; there’s nothing we can do about this. If you start getting OpenAI errors (this is the error it will raise for both models, as we’re indirectly using openai-python for both) using the team budget tomorrow, try using your own keys.

  • We remind teams that sharing chats or secrets with other teams is not allowed.


Apologies for the late changes. We think this is the path that is most likely to prevent issues. 


See you starting 23:59 Anywhere on Earth, and good luck!

SaTML 2024 LLMs CTF Announcements

unread,
Feb 4, 2024, 5:16:22 PMFeb 4
to SaTML 2024 LLMs CTF Announcements
Note also that we cannot predict latency issues tomorrow, and our first priority will be that the winning team is not determined by server issues.
If the API latency is unbearable in the first hours of the Evaluation phase, we reserve the right to smoothen out the bonuses (increase \beta) a bit more.
Reply all
Reply to author
Forward
0 new messages