The 2000-year-old scroll discusses music, food, and how to enjoy life’s pleasures.

February 5th, 2024

We’re announcing the winners of the Vesuvius Challenge 2023 Grand Prize. We’ll look at how they did it, what the scrolls say, and what comes next.

Join us for a celebration at the Getty Villa Museum in Los Angeles on March 16th, 4pm. More information here.

Victory​Two thousand years ago, a volcanic eruption buried an ancient library of papyrus scrolls now known as the Herculaneum Papyri.

The scrolls were carbonized by the eruption of Mount Vesuvius in 79 AD.In the 18th century the scrolls were discovered. More than 800 of them are now stored in a library in Naples, Italy; these lumps of carbonized ash cannot be opened without severely damaging them. But how can we read them if they remain rolled up?

The scroll read by the winners.

Result of an attempt to physically unroll a scroll.On March 15th, 2023, Nat Friedman, Daniel Gross, and Brent Seales launched the Vesuvius Challenge to answer this question. Scrolls from the Institut de France were imaged at the Diamond Light Source particle accelerator near Oxford. We released these high-resolution CT scans of the scrolls, and we offered more than $1M in prizes, put forward by many generous donors.

Artistic visualization of constructing a 3D volume.

A global community of competitors and collaborators assembled to crack the problem with computer vision, machine learning, and hard work.

Less than a year later, in December 2023, they succeeded. Finally, after 275 years, we can begin to read the scrolls:

Text from PHerc.Paris. 4 (Institut de France), unseen for 2,000 years. Roughly 95% of the scroll remains to be read.The thoughts of our ancestors, locked in mud and ash for 2000 years, hidden in darkness — now, with the light of a worldwide effort shining upon them, finally seen again.

Grand Prize​We received many excellent submissions for the Vesuvius Challenge Grand Prize, several in the final minutes before the midnight deadline on January 1st.

We presented these submissions to the review team, and they were met with widespread amazement. We spent the month of January carefully reviewing all submissions. Our team of eminent papyrologists worked day and night to review 15 columns of text in anonymized submissions, while the technical team audited and reproduced the submitted code and methods.

There was one submission that stood out clearly from the rest. Working independently, each member of our team of papyrologists recovered more text from this submission than any other. Remarkably, the entry achieved the criteria we set when announcing the Vesuvius Challenge in March: 4 passages of 140 characters each, with at least 85% of characters recoverable. This was not a given: most of us on the organizing team assigned a less than 30% probability of success when we announced these criteria! And in addition, the submission includes another 11 (!) columns of text — more than 2000 characters total.

The results of this review were clear and unanimous: the Vesuvius Challenge Grand Prize of $700,000 is awarded to a team of three for their excellent submission. Congratulations to Youssef Nader, Luke Farritor, and Julian Schilliger!

Youssef Nader

Luke Farritor

Julian SchilligerAll three winning team members have been strong community contributors since the very beginning of the Vesuvius Challenge. You may remember Youssef. He is the Egyptian PhD student in Berlin who was able to read a few columns of text back in October, winning the second-place First Letters Prize. His results back then were particularly clear and readable, which made him the natural lead for the team that formed.

You might remember Luke as well: he is the 21-year-old college student and SpaceX intern from Nebraska, who was the first person in history to read an entire word from the inside of a Herculaneum scroll (ΠΟΡΦΥΡΑϹ, “purple”). This won him the first-place First Letters Prize, a few weeks before Youssef’s results.

And finally, you might remember Julian. He is the Swiss robotics student at ETH Zürich, who won three Segmentation Tooling prizes for his incredible work on Volume Cartographer. This enabled the 3d-mapping of the papyrus areas you see before you.

For the Grand Prize, they assembled into a superteam, crushing it by creating what was unanimously deemed the most readable submission.

The submission contains results from three different model architectures, each supporting the findings of the others, with the strongest images often coming from a TimeSformer-based model. Multiple measures prevent overfitting and hallucination, including results from multiple architectures, a study across input/output window sizes, label smoothing, and varying validation folds. Like with all our prizes, this ink detection code has been made public as open source (on GitHub), leveling up everyone in the community.

The winners’ main submission image (TimeSformer 64×64).In addition to unparalleled ink detection, the winning submission contained the strongest auto-segmentation approach we have seen to date (more about the process of “segmentation” below). ThaumatoAnakalyptor (roughly: Miracle Uncoverer) by Julian generates massive papyrus segments from multiple scrolls. Re-segmentations of well known areas validate previous ink findings, and entirely new segmentations reveal writing elsewhere, such as the outermost wrap of the scroll!

Outputs from auto-segmentation. The top row overlaps with the submission image, the bottom row has new segments. Much work remains to improve this promising tool.Congratulations to Youssef, Luke, and Julian. You are the well-deserved winners of the 2023 Vesuvius Challenge Grand Prize!

Runners up​Of the remaining submissions, the scores from our team of papyrologists identify a three-way tie for runner up. These entries show remarkably similar readability to each other, but still stand out from the rest by being significantly more readable. Congratulations to the following teams, each taking home $50,000!

Elian Rafael Dal Prá, Sean Johnson, Leonardo Scabini, Raí Fernando Dal Prá, João Vitor Brentigani Torezan, Daniel Baldin Franceschini, Bruno Pereira Kellm, Marcelo Soccol Gris, and Odemir Martinez Bruno. GitHub

Louis Schlessinger and Arefeh Sherafati. GitHubThese teams each brought to the table new approaches to the subtleties of ink labeling and sampling. Be sure to check out their methods at the links above. Other teams may also now choose to share their approaches, so be sure to follow our Discord community for updates. Joining our community also provides access to the CT data and more images under our data agreement, as well as a front-row seat to daily discovery and collaboration!

To date, our efforts have managed to unroll and read about 5% of the first scroll. Our eminent team of papyrologists has been hard at work and has achieved a preliminary transcription of all the revealed columns. We now know that this scroll is not a duplicate of an existing work; it contains never-before-seen text from antiquity. The papyrology team are preparing to deliver a comprehensive study as soon as they can. You all gave them a lot of work to do! Initial readings already provide glimpses into this philosophical text. From our scholars:

The general subject of the text is pleasure, which, properly understood, is the highest good in Epicurean philosophy. In these two snippets from two consecutive columns of the scroll, the author is concerned with whether and how the availability of goods, such as food, can affect the pleasure which they provide.

Do things that are available in lesser quantities afford more pleasure than those available in abundance? Our author thinks not: “as too in the case of food, we do not right away believe things that are scarce to be absolutely more pleasant than those which are abundant.” However, is it easier for us naturally to do without things that are plentiful? “Such questions will be considered frequently.”

Since this is the end of a scroll, this phrasing may suggest that more is coming in subsequent books of the same work. At the beginning of the first text, a certain Xenophantos is mentioned, perhaps the same man — presumably a musician — also mentioned by Philodemus in his work On Music.

Philodemus, of the Epicurean school, is thought to have been the philosopher-in-residence of the villa, working in the small library in which the scrolls were found.

Initial, rough draft transcriptions:

col. -8, ll. 2-14:

2 …]ι̣μ̣εν τοὺϲ̣ [πα]ρ̣[ὰ Ξ]ε̣-

νοφάντωι το̣ιούτου[ϲ,

ὃ καὶ ὑπ’ ἄ̣λλων δοκεῖ

5 γείνεϲθαι, παραπλη-

ϲίωϲ δ̣’ ο̣ὐδὲ παρ̣’ ἑτέρωι

ἴδι̣ον το̣ῦ δ̣οκοῦ̣ντοϲ̣

εἶναι καὶ παρὰ πλε̣ί-

οϲ̣ι̣ν̣ ἥδιο̣ν, ἀλλ’ ὡ̣ϲ̣ καὶ

10 ἐ̣π̣ὶ τῶν βρω̣μ̣άτ̣ων

ο̣ὐ̣κ ἤδ̣η τὰ ϲπάνια

πάντωϲ̣ καὶ ἡδ̣ίω

τῶν δ̣αψιλῶν̣ ε̣ἶναι̣

14 νομίζ̣ο̣με̣ν· οὐ γ̣ὰρ̣

col. -7, ll. 4-10:

λ̣ει παρὰ τὰ δαψιλῆ.

5 θεωρηθήϲεται δὲ τὰ

τοιαῦθ’ οὕτω{ι} πολ̣λά-

κιϲ πότερον ὅ̣ταν πα-

ρῇ τὸ δαψιλέϲτερον

ἡ φύϲιϲ ἥδιον ἀπαλλάτ-

10 τει το̣ύ̣τ̣ο̣υ̣ καὶ πάλ̣ι̣ν̣ ̣ ̣

Later in the scroll:

In the closing section of the text our author takes a parting shot at his adversaries, who “have nothing to say about pleasure, either in general or in particular, when it is a question of definition.”

col. -2, ll. 2-8:

2 ἑ̣κάϲτηϲ κριτηρίων

θεωροῦνται. πρὸϲ δὲ

οὔτε καθόλου περὶ

5 ἡδονῆϲ ἐχόντων τι

λέγειν οὔτε περὶ τῆϲ

κατὰ μ̣έ̣ρο̣ϲ̣, ὅ̣τε ὡ-

8 ριϲμένον τι, ἀλλ’ οὖν

Finally the scroll concludes:

“… for we do [not] refrain from questioning some things, but understanding/remembering others. And may it be evident to us to say true things, as they might have often appeared evident!”

col. -1, ll. 1-6:

1 ὰρ ἀπ̣εχόμ̣ε̣θ̣α̣ τὰ

μὲν κρίνειν, τὰ δὲ

κατέχειν καὶ ἐμφαί

νoιθ’ ἡμῖν ἀληθῆ λέ-

5 γειν ὥϲπερ πολλά̣κιϲ

ἂν ἐ̣μφανε̣ίη̣{ι}.

Richard Janko writes:

“Is the author Epicurus’ follower, the philosopher and poet Philodemus, the teacher of Vergil? It seems very likely.

Is he writing about the effect of music on the hearer, and comparing it to other pleasures like those of food and drink? Quite probably.

Does this text come from his four-part treatise on music, of which we know Book 4? Quite possibly: the title should soon become available to read.

Is the Xenophantus who is mentioned the celebrated flute-player, or the man famous in antiquity for being unable to control his laughter, or someone else entirely? So many questions! But improvements to the identification of the ink, which can be expected, will soon answer most of them. I can hardly wait.”

Federica Nicolardi told us:

“Epicureanism says hi, with a text full of music, food, senses, and pleasure!”

From Bob Fowler:

“Like other Epicureans, he valued pleasure above all – but pleasure rightly understood, not mere indulgence. Living in ancient Rome – a society not known for abstinence – Philodemus could expect to meet with scepticism from his readers.”

Scholars might call it a philosophical treatise. But it seems familiar to us, and we can’t escape the feeling that the first text we’ve uncovered is a 2000-year-old blog post about how to enjoy life. Is Philodemus throwing shade at the stoics in his closing paragraph, asserting that stoicism is an incomplete philosophy because it has “nothing to say about pleasure?” The questions he seems to discuss — life’s pleasures and what makes life worth living — are still on our minds today.

We can expect many more works from Philodemus in the current collection, once we’re able to scale up this technique. But there could be other text as well — an Aristotle dialog, a lost history of Livy, a lost Homeric epic work, a poem from Sappho — who knows what treasures are hidden in these lumps of ash.

And there is the hope of a much bigger library still in the ground, since two levels of the villa remain unexcavated. More about this below!

How accurate are these pictures?​Machine learning models are infamous for “hallucinating”: making up text or pictures that look similar to their training data. Similarly, there might be ways for contestants to cheat by making up images themselves, e.g. by embedding those in the model weights. How do we know that that’s not happening here? There are a couple of answers:

Technical reproduction. The Vesuvius Challenge Technical Review Team reproduced the winning submissions manually. We made sure to clearly understand every part of the code, and that when we run it independently we get similar output images. Since all code and training data is now open source, you can do the same!Multiple submissions of the same area. You might have noticed that all submission images above show the same area of the scroll. This is because we released 3d-mapped papyrus sheets within the CT-scan (“segments”) created by our segmentation team, which were then used by all contestants. The resulting output images — created by different ML models and training labels — have produced extremely similar results. This holds not just for the winners and runner ups, but also for the other submissions that we received.Small input/output windows. The ink detection models are not based on Greek letters, optical character recognition (OCR), or language models. Instead, they independently detect tiny spots of ink in the CT scan, the writing appearing later when these are aggregated. As a result, the text appearing in the images is not the imagined output of a machine learning model, but is instead directly tied to the underlying data in the CT scan.The models use small input/output windows. In some cases, the output is even only binary (“ink” vs “no ink”), as shown in this animation. This makes it extremely unlikely for the model to hallucinate shapes that look like letters.How does the unrolling work?​Roughly, virtual unwrapping works in three steps:

Scanning: creating a 3D scan of a scroll or fragment using X-ray tomography.Segmentation: tracing the crumpled layers of the rolled papyrus in the 3D scan and then unrolling, or flattening, them.Ink Detection: identifying the inked regions in the flattened segments using a machine learning model.These scrolls were scanned at Diamond Light Source, a particle accelerator near Oxford, England. The facility produces a parallel beam of X-rays at high flux, allowing for fast, accurate, and high-resolution imaging. The X-ray photos are turned into a 3D volume of voxels using tomographic reconstruction algorithms, resulting in a stack of slice images.

Scrubbing through the slice images of the scroll.The next step is to identify individual sheets of papyrus in 3D space. For this we primarily use a tool called Volume Cartographer, created by Seth Parker and others in Brent Seales’ lab, and augmented by our contestants, primarily Julian Schilliger (Grand Prize winner) and Philip Allgaier.

Volume Cartographer is operated by our team of full-time segmenters: Ben Kyles, David Josey, and Konrad Rosenberg. They use a combination of automatic algorithms and manual adjustments to map out large areas of papyrus. This is still a painstaking process, with lots of room for improvement if we’re going to segment all the scrolls.

Animation showing manual and automatic segmentation in Volume Cartographer.Finally, ink detection. Stephen Parsons at Brent’s lab had shown that Herculaneum ink could theoretically be detected in CT scans, but so far only using smaller fragments — detecting ink in the larger scans of complete scrolls had yet to be achieved. For months this part proved elusive, until progress was made on two separate tracks:

Crackle pattern. Last summer, Casey Handmer discovered a strange pattern of “crackle” by looking at raw flattened surface volumes. This pattern appeared to form letters. Casey won the First Ink Prize for this monumental discovery and shared it with the community, and a flurry of activity followed.Luke Farritor (Grand Prize winner), immediately started hunting for more crackle in flattened surface volumes produced by the segmentation team. He then trained a machine learning model on the shapes he found, which led directly to him winning the First Letters Prize in October.

Kaggle competition. Separately, hundreds of teams tried building the best machine learning model for detecting ink in open fragments — pieces that had broken off during the physical unrolling process of scrolls, hundreds of years ago. Instead of labeling crackle (which wasn’t known yet), they had the benefit of ground truth data directly from photos of these fragments.

Photo of Fragment 1

Aligned infrared

Aligned binary ink labelsThis resulted in excellent models, but they did not seem to work on the flattened segments which the segmentation team produced. That was, until Youssef Nader (Grand Prize winner) used domain adaptation techniques on them, the start of a technique that ultimately won him the second place First Letters Prize.

After the success of the First Letters Prize, the Grand Prize seemed within reach. Youssef, Luke, and Julian teamed up, with several other teams putting in strong submissions as well.

What did it take?​With the Vesuvius Challenge, we hope not only to solve the problem of reading the Herculaneum Papyri, but also to inspire similar projects. For that, it’s helpful to know what has contributed to our success in 2023. Here are some things we believe were important:

An inspiring goal and a clear target. There are many worthy causes in the world, so it helps that our goal is unusual for a computing competition. It drew more press and donations early on, it attracted an intrinsically motivated community, and it increased our probability of success to begin with (emerging research area => a higher
Read More