Tiny Model, Massive PDF Corpus: URL Embeddings for 8.3 Million PDFs

2024 March 20

Here is a visualization of roughly 8.3 million unique PDF urls. I was bored like two months ago and wanted to classify pdfs by their metadata. I ended up generating embeddings for pdf-corpus (the biggest pdf corpus in the world).

I used a tiny model, I used bge-small-en-v1.5 because I ran this on a Kaggle notebook (sucks to be GPU poor).

I decided to see if you could classify by URL this information because at the end of the day, the original corpus is 4TB uncompressed and the metadata is only 1.4GB of text.

More information can be found here and if you want to download the dataset or the metadata it is here.

I might later do a bigger blog post on this. For now, that's it.