https://commoncrawl.org/

Frequently Asked Questions

https://commoncrawl.org/big-picture/frequently-asked-questions/

General Questions

What is Common Crawl?

Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis.

What can you do with a copy of the web?

The possibilities are endless, but people have used the data to improve language translation software, predict trends, track the disease propagation and much more.

Can’t Google or Microsoft just do that?

Our goal is to democratize the data so everyone, not just big companies, can do high quality research and analysis.

What terms is the data released under?

As strong believers in open data, we apply as few restrictions as possible to the dataset. The terms we add, primarily in an effort to prevent abusive or illegal usage, are fully described on our terms of use page.

Actual Open-Orca Dataset from openorca team

A member of the orca team is taking the orca data set and forking the project. This is billed as the "uncensored" data set. The orca team claims it is an earlier set with less refinement.

paper: https://lemmy.intai.tech/post/41821

cross-posted from: https://lemmy.intai.tech/post/41747

• paper
• dataset

On the Coverage of Cognitive mmWave Networks with Directional Sensing and Communication

Authors: Shuchi Tripathi, Abhishek K. Gupta, SaiDhiraj Amuru

Word Count: 5400

Average Reading Time: ~30 minutes

Highlights:

• The authors propose an analytical framework to evaluate the performance of a cognitive mmWave network consisting of a primary link and multiple secondary links using stochastic geometry.

• They consider directional channel sensing and communication in contrast to omnidirectional sensing, which allows secondary transmitters to transmit based on their orientation instead of being outside a certain distance. This provides better spatial reuse for secondary transmitters.

• They analyze the medium access probability, activity factor, and coverage probability of the pri

An open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2.

This distribution was created by Aaron Gokaslan and Vanya Cohen of Brown University.

Initial Data Collection and Normalization The authors started by extracting all Reddit post urls from the Reddit submissions dataset. These links were deduplicated, filtered to exclude non-html content, and then shuffled randomly. The links were then distributed to several machines in parallel for download, and all web pages were extracted using the newspaper python package. Using Facebook FastText, non-English web pages were filtered out.

Subsequently, near-duplicate documents were identified using local-sensitivity hashing (LSH). Documents were hashed into sets of 5-grams and all documents that had a similarity threshold of greater than 0.5 were removed. The the remaining documents were tokenized, and documents with fewer than 128 tokens were removed. This left 38GB of text data (40GB using SI units) from

Introducing HomoScriptor - A human-written, community-driven dataset for fine-tuning large language models.

Greetings, AI Community!

I am thrilled to announce the launch of HomoScriptor, a collaborative project that aims to revolutionize language models and drive innovation in natural language processing. And I want YOU to join me on this incredible journey!

What is HomoScriptor?

HomoScriptor is a vibrant and collaborative initiative where language model enthusiasts like myself can come together to create a remarkable human-written dataset for fine-tuning language models. I have curated a diverse collection of meticulously organized JSON files, specifically designed to enhance the training of large language models (LLMs).

Key Features:

📁 Categorized JSON Files: The dataset in HomoScriptor is thoughtfully organized into various categories, each with its own JSON file. This structured approach makes it effortless for us to explore spec