Skillware skills · data_engineering/novelty_extractor
Novelty Extractor
Filters text datasets by semantic novelty using local fastembed embeddings — keeps only chunks with new information.
Capabilities
- Local fastembed
- Novelty threshold
- Multi-turn baseline
Problems this skill solves
People search for tasks — not always "skillware":
- deduplicate dataset embeddings local
- filter redundant text for training
- semantic novelty filter ai
- remove duplicate chunks rag dataset
Install
pip install skillware then SkillLoader.load_skill("data_engineering/novelty_extractor")
FAQ
- How to filter redundant text from a dataset locally?
- Skillware data_engineering/novelty_extractor uses fastembed (no API key) to keep chunks above a novelty threshold.