I’m looking for a duplicate/similarity checker against a custom set of documents. This is possibly like a plagiarism checker, but with a custom reference (instead of everything that exists).
But I could not find a solution that can be selfhosted, and have some simple UI and capabilities like Turnitin. Any suggestions?
Thanks’
Maybe look into self hosted llm. I’ve used two recently to help analyze a large volume of books, by ingesting them into the data set, then chat with the bot for specifics. It worked pretty well but there are some limitations, such as token length and general hallucinations. But they both use citations of the data they used, so it helps to check their work.
PrivateGPT - https://github.com/imartinez/privateGPT
Llamaindex - https://github.com/run-llama/llama_index
Both have simple selfhosted webui or actual applications. So, in theory you should be able to ingest data and then see if it then matches any submissions you submit later. But I have not really tried it for this though, so it might not work.
This is very specific and niche without a good reason to exist, so I doubt this exists.
Grep. If you have the sources digitally and local, you can just run a search for a string.