Downloading the Data
Option 1: Direct download from the tasks page, in JSON lines format.
Option 2: Using the datasets library.
The code used to run our experiments is available on GitHub.
Making a Leaderboard Submission
Create a comma-separated values (CSV) file with the headers (Task, ID, Prediction), where each row represents one output.
We recommend using our conversion script to produce the CSV file from JSON prediction files to avoid discrepancies.
As inputs, it expects a predictions JSON file with a mapping from an ID to a textual prediction, for every task, e.g:
Login to the website (using your Google account is recommended).
Upload your CSV file via the submission page.
Within a few minutes, check your email for a confirmation message that your submission has been received.
Results will be sent by email within 24 hours. Valid public submissions will immediately appear on the leaderboard.
Each user is limited to 5 submissions per week and a total of 10 submissions per month.
If you need any help, please reach out to email@example.com
qasper,8941956c4b67e2436bbaf372a120f358f50c377b,"English, German, French"
summ_screen_fd,fd_Gilmore_Girls_01x13,"Rory's charity rummage sale is a disaste..."
from datasets import load_dataset
z_scrolls_datasets = ["gov_report", "summ_screen_fd", "qmsum","squality","qasper", "narrative_qa", "quality","musique","space_digest", "book_sum_sort"]
data = [load_dataset("tau/zero_scrolls", dataset) for dataset in z_scrolls_datasets]