#nobridge

  • 0 Posts
  • 20 Comments
Joined 2 years ago
Cake day: March 14th, 2025
  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•Home server on old android phones?English
    4·
    1 month ago

    Completely depends on where you are too, my NAS uses ~30W, which means ~22kWh in 30 days, which translated into local electricity costs are 53SEK or $5.39.
    It has enough CPU (Intel Celeron J4125) to run my production loads, RAID1 16TB drives and 18GB RAM so it successfully hosts DNS, DHCP, Jellyfin, Vikunja and network shares. And I get the knowledge that it is all local and under my control.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•[AIT] Local LLM for Server Setup Guidance?English
    1·
    3 months ago

    Depending on the use and the size of your graphics card you might be better of trying Gemma 4 12B too.
    On my Nvidia 5060 Ti 16GB I can use either Gemma 4 A4B IQ3_M (larger model but highly compressed) or Gemma 4 12B Q8_0 (smaller model but much less compression) and after a night of testing with the Gemma 4 12B Q8_0 I’d say it’s a better helper when it comes to coding but a worse gm for my dnd inspired fantasy world roleplaying adventure.

    I imagine your “setup a selfhosted environment” is closer to the first.

  • anamethatisnt@sopuli.xyztoSelfhosted@lemmy.world•[AIT] Local LLM for Server Setup Guidance?English
    2·
    3 months ago

    Easy setup would be to use koboldcpp + SillyTavern + Gemma 4 26B A4B GGUF of the largest quant your graphics card can fit together with your context.
    Remember to setup SillyTavern to allow network connections and create a user and password, default installation is localhost only.
    Set the temperature to 0.3 if using IQ3_M, higher quants allow higher temperature, and make sure all the formatting templates inside Advanced Formatting are set to Gemma 4.

    Then use either Gemma 4 itself or the free google.com AI to create some “W++ Denze with Horizontal Lines summaries for Gemma 4” of the relevant documentation of the latest version of the home server apps your friends are gonna install.
    You should ensure the resulting lorebook entries are no more than 1k tokens each to leave some context for your friends chats, add it to a character card in SillyTavern and set some keywords to allow them to load dynamically and not stay in context memory all the time.

    Regarding the character card you can ask Gemma 4 to write that for you too, I find “Write a character card in W++ Denze with Horizontal Lines style for Gemma 4 with this name, personality, attitude and skillset” works well for that.
    Then ask it to write a “First message prompt that starts with X, continues with Y and ends with Z for that character card” and you get a first draft to rewrite and paste into the “First Message” of the character card. The first message works as a template that Gemma 4 will imitate when you chat with it. Then simply try the chatbot out before letting others use it.

    I find my own Gemma 4 26B A4B IQ3_M works well for practicing hiragana and katakana, discussing programming or troubleshoot existing code or writing a small function but it can’t be expected to write a correct DatabaseService.cs from scratch and stuff like that.

    Oh and forget about finding good cards and lorebooks for SillyTavern use online, most users use it for NSFW Roleplaying chats. It is a very easy UI to use to create harnesses for your local LLM though.