Readme update.
This commit is contained in:
parent
2eba340523
commit
32ba623b96
31
readme.md
31
readme.md
|
@ -11,6 +11,8 @@ So far:
|
||||||
of concept/springboard for more advanced narrative memory.
|
of concept/springboard for more advanced narrative memory.
|
||||||
- **GPU Scaling Filter:** Reduce number of GPU layers in use if Ollama
|
- **GPU Scaling Filter:** Reduce number of GPU layers in use if Ollama
|
||||||
crashes due to running out of VRAM.
|
crashes due to running out of VRAM.
|
||||||
|
- **Output Sanitization Filter:** Remove words, phrases, or
|
||||||
|
characters from the start of model replies.
|
||||||
|
|
||||||
## Memory Filter
|
## Memory Filter
|
||||||
|
|
||||||
|
@ -95,7 +97,7 @@ Other things to do or improve:
|
||||||
events, instead of dumping it all into the vector DB.
|
events, instead of dumping it all into the vector DB.
|
||||||
- Improve multi-user handling. Should technically sort of work due to
|
- Improve multi-user handling. Should technically sort of work due to
|
||||||
messages having UUIDs, but is a bit messy. Only one collection is
|
messages having UUIDs, but is a bit messy. Only one collection is
|
||||||
used, so multiple users = concurrency issues.
|
used, so multiple users = concurrency issues?
|
||||||
- Block user input while updating the knowledgebase.
|
- Block user input while updating the knowledgebase.
|
||||||
|
|
||||||
## GPU Scaling Filter
|
## GPU Scaling Filter
|
||||||
|
@ -108,6 +110,7 @@ account the number of layers in models or dynamically monitor VRAM
|
||||||
use.
|
use.
|
||||||
|
|
||||||
There are three settings:
|
There are three settings:
|
||||||
|
|
||||||
- **Initial Reduction:** Number of layers to immediately set when an
|
- **Initial Reduction:** Number of layers to immediately set when an
|
||||||
Ollama crash is detected. Defaults to 20.
|
Ollama crash is detected. Defaults to 20.
|
||||||
- **Scaling Step:** Number of layers to reduce by on subsequent crashes
|
- **Scaling Step:** Number of layers to reduce by on subsequent crashes
|
||||||
|
@ -115,6 +118,30 @@ There are three settings:
|
||||||
- **Show Status:** Whether or not to inform the user that the
|
- **Show Status:** Whether or not to inform the user that the
|
||||||
conversation is running slower due to GPU layer downscaling.
|
conversation is running slower due to GPU layer downscaling.
|
||||||
|
|
||||||
|
## Output Sanitization Filter
|
||||||
|
|
||||||
|
This filter is intended for models that often output unwanted
|
||||||
|
characters or terms at the beginning of replies. I have noticed this
|
||||||
|
especially with Beyonder V3 and related models. They sometimes output
|
||||||
|
a `":"` or `"Name:"` in front of replies. For example, if system prompt is
|
||||||
|
`"You are Quinn, a helpful assistant."` the model will often reply with
|
||||||
|
`"Quinn:"` as its first word.
|
||||||
|
|
||||||
|
There is one setting:
|
||||||
|
|
||||||
|
- **Terms:** List of terms or characters to remove. This is a list,
|
||||||
|
and in the UI, each item should be separated by a comma.
|
||||||
|
|
||||||
|
For the above example, the setting textbox should have `:,Quinn:` in
|
||||||
|
it, to remove a single colon from the start of replies, and `Quinn:`
|
||||||
|
from the start of replies.
|
||||||
|
|
||||||
|
### Other Notes
|
||||||
|
|
||||||
|
Terms are removed in the order defined by the setting. The filter
|
||||||
|
loops through each term and attempts to remove it from the start of
|
||||||
|
the LLM's reply.
|
||||||
|
|
||||||
# License
|
# License
|
||||||
|
|
||||||
AGPL v3.0+.
|
All filters are licensed under AGPL v3.0+.
|
||||||
|
|
Loading…
Reference in New Issue