Paste research text, lecture notes, logs, or short documents and watch the full preprocessing pipeline unfold step by step: normalization, tokenization, stop words, stemming, lemmatization, n-grams, entities, and frequency analysis.
Current text
2
Sentences
26
Raw tokens
23
Useful tokens
Everything runs in your browser. Text is not uploaded to MuhammadLab servers.
Input
Best for short teaching examples, abstracts, case notes, CSV snippets, log excerpts, and classroom demos.
204 / 12,000 characters.
204
Characters
2
Sentences
26
Raw tokens
23
Useful tokens
25
Unique raw terms
22
Unique useful terms
0
Entities
22
Bigrams
The raw text before any transformation. In real NLP work, this might come from notes, reports, logs, abstracts, or exported records.
After
Natural Language Processing helps computers understand human language. Students often clean text by lowercasing, removing stop words, generating n-grams, and building features for machine learning models.
Split text into sentence-level units so downstream models can analyze smaller chunks.
2 sentences detected with compromise.
Lowercasing reduces duplicate vocabulary such as Model/model/MODEL. Whitespace cleanup prevents accidental empty tokens.
Before
Natural Language Processing helps computers understand human language. Students often clean text by lowercasing, removing stop words, generating n-grams, and building features for machine learning models.
After
natural language processing helps computers understand human language. students often clean text by lowercasing, removing stop words, generating n-grams, and building features for machine learning models.
Remove punctuation for bag-of-words features while keeping word characters and simple apostrophes/hyphens.
Before
natural language processing helps computers understand human language. students often clean text by lowercasing, removing stop words, generating n-grams, and building features for machine learning models.
After
natural language processing helps computers understand human language students often clean text by lowercasing removing stop words generating n-grams and building features for machine learning models
Break the cleaned text into tokens. Tokens are the building blocks for counts, TF-IDF, classifiers, and many classic NLP models.
26 raw tokens.
Remove very common words so content-bearing terms become easier to see.
3 stop words removed.
Approximate words to rough roots using suffix rules. This is fast and explainable, but it can create non-dictionary roots.
Before
natural language processing helps computers understand human language students often clean text lowercasing removing stop words generating n-grams building features machine learning models
After
natural language process help computer understand human language student often clean text lowercas remov stop word generat n-gram build feature machine learn model
Convert words toward dictionary forms. This tool uses a small teaching dictionary plus compromise noun/verb normalization.
Before
natural language processing helps computers understand human language students often clean text lowercasing removing stop words generating n-grams building features machine learning models
After
natural language process help computer understand human language student often clean text lowercase remove stop word generate n-gram build feature machine learn model
Create adjacent word sequences. Bigrams and trigrams capture short context that single tokens miss.
22 bigrams and 21 trigrams generated.
Teaching flow
This page is a good first stop before the TF-IDF tool because students can see exactly how text becomes tokens and features before scoring, ranking, or model training.
How to use in class
1. Start with the messy social text to show why preprocessing matters.
2. Switch to a research abstract and compare raw tokens with useful tokens.
3. Explain stemming versus lemmatization using the before/after panels.
4. Move into TF-IDF and show how cleaned tokens become ranking features.