[CLAS 2024 Jailbreaking Attack Track] Submission doesn't support JSONL format

[Y Wu]

Dear organizers,

I want to point out a conflict between the submission page https://www.llmagentsafetycomp24.com/upload-files/ and the starter kit.

In starter kit of jailbreaking track, it claims:

The submission is supposed to be a .jsonl file with 100 lines. 

However, the submission page only allows json files to be uploaded.  

I want to bring this matter to your attention as it would be confusing for every participant.

Thank you for your effort in holding this competition.

Best,

Yu

[Chaos]

I would like to add that, in fact, uploading a .json file also leads to an error. It seems the online evaluation system might not be set up yet.

Additionally, regarding the constraint mentioned in track I, there seems to be a discrepancy. The guidelines note a hard constraint of 100 tokens for each jailbreak prompt, whereas the utils.py checks for the number of added tokens exceeding 100 (if num_added_token > 100). Could the organizers please clarify which guideline we should follow?

Appreciate your efforts and looking forward to your guidance.

Best,
Chao

[Zhen Xiang]

Dear Participants,

Please rename your submission file for Track I as `jailbreak.jsonl.json` before submission. The starter kit has been changed accordingly. Thank you for pointing this out

Best,

Organizers

[Zhen Xiang]

Dear Participants,

The constraint of 100 tokens is applied to the total number of injected tokens. We have made the changes to the website. Thank you for pointing this out.

Best,

Organizers