• 1 Post
  • 30 Comments
Joined 3 years ago
cake
Cake day: January 16th, 2024

help-circle

  • JasonDJ@lemmy.ziptoSelfhosted@lemmy.world•Is anyone self-hosting their own AI?
    link
    fedilink
    English
    arrow-up
    25
    arrow-down
    2
    ·
    16 days ago

    What a timely comment.

    I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB

    Combined with a 5600x3d and 32gb of system memory.

    I just started messing with local LLMs yesterday. “Getting started” is rather easy. pacman -S lm-studio (I use arch, btw), launch it, download a model that your hardware supports, and start talking.

    Right now I’m working on getting it to talk to vscode.


  • Router is a bunch of old PCs I frankensteined and put opnsense on.

    Ended up i5-7500, 8gb mem, 500gb SSD.

    Put in a dual-port SFP+ NIC and a dual port 2.5GbaseT nic.

    One of the 2.5gb ports goes to my cable modem.

    One of the 10gb ports takes Twinax to my dumb LAN switch, the other to my dumb server switch, which right now is really just my storage network…the link to the firewall isn’t doing anytuhing.

    The dumb switches I have here, though, they’ve got some sort of weird MAC learning disability. It cannot handle macflaps, which I found out about because I had a hard time roaming between APs.

    So I got a Netgear GS108T and put OpenWRT on it, and that switches between my APs.

    My APs are Linksys MX4200v1’s. Also openwrt. Got four of them…one upstairs, two in the ground floor, one in the basement. Can get very close to 1Gbps anywhere in the house on wifi. Not the best platform for openwrt (ipq807x), as I’ve found out…but it was a hella cheap deal for tri-band AX APs.





  • It’s okay for small scale, dev, etc…on a fast and flat/local network.

    But databases really don’t like NFS, SMB, or most file sharing protocols. There’s too much transaction overhead.

    The effect gets exponentially worse as you add in latency. Database may as well be on the moon if you’ve got more than 10ms between it and the app (hyperbole but not really).

    Use fast local storage, ideally. Or block if you have it. Async replicas if you are chasing 9s.

    When databases fail, they fail spectacularly and cause ripples. Remember that you’ll be the one supporting it when they do.










  • This is one of the right answers, and probably the easiest.

    But…who is your domain registered with? A lot of the big registrars (I.e. CF) can easily integrate into several opnsense plugins…namely ACME/HAProxy, and Dynamic DNS.

    Dynamic DNS would make sure that “pubip.mydomain.com”, from the outside, always pointed to your Internet connection, even if your IP changes.

    Then just register CNAMEs for your external services to point to your pubip.mydomain.com.

    HAProxy reads the CNAME and directs it to the right service.

    You can either use split DNS and have HAProxy listen on both the internal and external, and map DNS around that way…or you can just listen on your external interface and connect that way and only update one end.

    Imo this is the more comprehensive solution, because this also easily solves certificates (via ACME). The last leg (from HAProxy to backend) and be unencrypted or just use long-life self-signed certs if you really care…but I wouldn’t worry about encryptipn on the inside of a homelab.