Loading...
Loading...
Loading Curriculum...
Raw text collected from the web, user input, or documents is messy by default. It contains HTML markup, inconsistent casing, Unicode anomalies, boilerplate noise, and domain-specific artifacts. Feeding such text directly into a model inflates vocabulary size, confuses embeddings, and burns through context tokens unnecessarily.
Text preprocessing is the systematic cleaning and normalization layer that sits between data collection and model training or inference. The right pipeline depends on the task — what helps a TF-IDF search engine can actively harm a fine-tuned LLM.
Typical NLP preprocessing pipeline
Each technique serves a specific purpose. Applying them indiscriminately is a common mistake — understand what each one does and when to use it.
Remove HTML tags, XML elements, and decode HTML entities. Essential for any web-scraped corpus.
Collapse tabs, newlines, and multiple spaces into a single space. Trim leading/trailing whitespace.
Convert to lowercase so 'Python' and 'python' map to the same token. Skip for NER or sentiment.
Apply NFC or NFKC normalization to unify equivalent character sequences and remove zero-width chars.
Strip non-alphanumeric characters. Useful for bag-of-words; harmful for LLMs that need syntax.
Remove high-frequency words with low semantic signal. Improves TF-IDF; usually hurts LLMs.
Crude suffix-stripping to root forms. Fast but produces non-words ('running' → 'run' → 'run').
Dictionary-based root reduction. 'better' → 'good'. More accurate than stemming, but slower.
Walk through each preprocessing step interactively. Edit the input text to see how your own content transforms. Toggle Show diff to highlight exactly what each step removes or changes.
The unprocessed text exactly as received — HTML tags, extra whitespace, inconsistent casing, and all.
A composable, type-safe pipeline using pure functions — no dependencies required.
Use this matrix to decide which steps belong in your pipeline. "Importance" refers to how much impact the technique typically has on downstream model performance.
| Technique | Best For | Skip When | Importance | Speed Cost | Complexity |
|---|---|---|---|---|---|
| Lowercasing | All tasks | NER, Sentiment, QA | High | None | Simple |
| Stop Word Removal | TF-IDF, Search | LLMs, Translation | Medium | Low | Easy |
| Stemming | Search, IR | LLMs, QA | Medium | Low | Easy |
| Lemmatization | NLP, Chatbots | Speed-critical apps | High | Medium | Moderate |
| HTML Stripping | Web corpora | Clean text sources | High | None | Easy |
| Unicode Norm. | Multilingual NLP | ASCII-only pipelines | High | Low | Easy |
| Sentence Split | Summarization | Sub-sentence tasks | High | Medium | Moderate |
The Node.js/TypeScript ecosystem has mature libraries for every preprocessing need.
Comprehensive NLP toolkit: tokenizers, stemmers (Porter, Lancaster), TF-IDF, BM25, sentence boundary detection, and more.
Lightweight, zero-dependency NLP for the browser and Node. Lemmatization, POS tagging, entity recognition, and sentence splitting.
Curated stop word lists for 60+ languages. Drop-in removeStopwords() function — much more comprehensive than rolling your own.
Robust HTML entity encoder/decoder. Handles all named and numeric character references correctly — use instead of ad-hoc regex.
Configurable HTML stripper with allowlist support. Safer than raw regex for stripping markup from untrusted input.
Detect the language of a text snippet. Useful for routing multilingual content to the right locale-specific preprocessing pipeline.
HTML strip → Unicode norm → whitespace → lowercase → dedupHTML strip → Unicode norm → whitespace onlyAll steps including stop words + stemmingHTML strip → whitespace → keep case & punctGolden rule: the less you touch the text, the better for large language models. Preprocess aggressively only when working with classical ML pipelines or curating training data at scale.