Common Crawl
commoncrawl.org
What Jamie wrote
A compelling non-profit that I am honestly surprised I wasn't aware of. They are crawling the web, like the search engine giants do, but as a non-profit and making the data available for others to use. This reminds me a bit of what Internet Archive does but for a very different purpose. Having a huge set of data like this should lower the barrier to Internet scale projects like new search engines. You can donate here.