greybeard

Little eMudhra was asked to write down his a secret and hand it in to the teacher. He accidently gave the teacher the wrong paper.

I recently gave Strata a shot for my local AI. I’m very impressed with the performance they have eeked out of my hardware. I’ve got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

Strata acts like an OpenAI compatible API server, I then have Hermes as a “harness” to make an “agent”, which I interact with on “buzz”, which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

One thing I didn’t mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn’t a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn’t have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.

I think most of the comments misunderstood you, or I did.

Keeping a dedicated admin separate from your daily use user is best practice. In a business setting it is a critical security line, and also seriously helps with auditing and logging. It makes sure that if my computer gets compromised, the active cookies and session tokens are all a standard user. It’s not fool proof, but it really reduces risk.

That said, in my home lab, I don’t bother with that separation. It’s far more work than the effort justifies, when the worst result of your systems being compromised is your media library not being available. What you will want, in any case, is a “break glass” account into every system you setup to use SSO. A local administrator that isn’t tied to SSO available in case your IDP is down, certificates expire, or you accidentally paint yourself into a corner (and you will).

I recommend, if you don’t already, using a password manager like BitWarden and having every single local admin be a long, randomly generated password.

Qwen is what I’ve found to work the best for me so far. I haven’t done an exhaustive search or anything, but the mixture of experts 35b models seem to work pretty well.

I’ve got a little more vRAM to play with, 20GB. It still struggles with giving it enough context to be useful for agentic stuff.