Submit on JMMMU Benchmark

Introduction

We do not recommend including results obtained from extensive prompt engineering since it is important to prevent performance hacking and better reflect real-world use cases. For more details, please refer to the lmms-eval code base and the upcoming paper (coming soon).

1. Obtain Result JSON File from lmms-eval code base.
2. If you want to update existing model performance by uploading new results, please ensure 'Model Name Revision' is the same as what's shown in the leaderboard. For example, if you want to modify LLaVA-OV 7B's performance, you need to fill in 'LLaVA-OV 7B' in 'Revision Model Name'.
3. Please provide the correct link of your model's repository for each submission.
4. After clicking 'Submit Eval', you can click 'Refresh' to obtain the latest result in the leaderboard.

Note: The example of the submitted JSON file is this url: result.json.

Submit Example

If you want to upload LLaVA-OV 7B's result in the leaderboard, you need to:

1. Select LMM in 'Model Type'.
2. Fill in 'LLaVA-OV 7B' in 'Model Name' if it is your first time to submit your result (You can leave 'Revision Model Name' blank).
3. Fill in 'LLaVA-OV 7B' in 'Revision Model Name' if you want to update your result (You can leave 'Model Name' blank).
4. Fill in 'https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov' in 'Model Link'.
5. Fill in '7B' in 'Model size'.
6. Upload results.json.
7. Click the 'Submit Eval' button.
8. Click 'Refresh' to obtain the uploaded leaderboard.

To check whether the submission is successful, you can click the 'Logs' button. If the message 'Success! Your submission has been added!' appears, the submission is successful.

If you have any questions or deletion requests, please contact miyai@cvm.t.u-tokyo.ac.jp.

⚠️ Please do not submit any malicious file (e.g, files you manually edited).

✉️✨ Submit your model evaluation json file here!