• 0 Posts
  • 3 Comments
Joined 3 years ago
cake
Cake day: June 16th, 2023

help-circle
  • I’ve been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It’s a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya

    My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

    I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling