I just found https://www.arliai.com/ who offer LLM inference for quite cheap. Without rate-limits and unlimited token generation. No-logging policy and they have an OpenAI compatible API.

I’ve been using runpod.io previously but that’s a whole different service as they sell compute and the customers have to build their own Docker images and run them in their cloud, by the hour/second.

Should I switch to ArliAI? Does anyone have some experience with them? Or can recommend another nice inference service? I still refuse to pay $1.000 for a GPU and then also pay for electricity when I can use some $5/month cloud service and it’d last me 16 years before I reach the price of buying a decent GPU…

Edit: Saw their $5 tier only includes models up to 12B parameters, so I’m not sure anymore. For larger models I’d need to pay close to what other inference services cost.

Edit2: I discarded the idea. 7B parameter models and one 12B one is a bit small to pay for. I can do that at home thanks to llama.cpp

  • hendrik@palaver.p3x.deOPEnglish
    0·
    2 months ago

    Hmmh, I lately use mistral-nemo which is 12B parameters. Since I’m more a programmer than a gamer, I didn’t put a graphics card into my PC, and I believe it’s too old to accomodate any recent one. (older PCIe generation, only x8 ports) I’d have to replace everything. And then I might as well go for a Radeon RX 7900 XTS or something. That’s $1.000(?) but has 24GB of VRAM. I don’t think buying an entire PC and then going for an old GPU will make me happy. And thanks to llama.cpp I get about 2 tokens per second just on the CPU. It’d have to be a considerable step up to be worth it. And last time I checked even a P40 was like $300+ and it’s super old and unclear if it’ll continue to be supported in the major frameworks. I’m not sure. I still lean towards paying for cloud GPU compute.

    Thanks for the numbers on your setup. That certainly helps weighing my options. Maybe some of my friends have some upgrades planned and want to give me their older 8GB NVidia cards…

    • tpWinthropeIII@lemmy.worldEnglish
      0·
      2 months ago

      I tried Mistral Nemo 12B instruct this morning. It’s actually quite good. I’d say it’s close to dolphin mistral 8x7B which is a monster in size and very smart, about 45 or 50GB. So I’d say Arli is a good deal Mistral Nemo 12B for 4 or $5 per month and privacy so they claim.

      If you don’t mind logging for some questions, you can get access to very good or if not the best models at lmsys.org without monetary cost. Just go to the “Arena”. This is where you contribute with your blind evaluation by voting which of two is better. I often get models like 4o and sonnet 3.5 by Anthropic, google’s best, etc., and at other times many good 70B models. You see two answers at once and vote your favorite between the two. In return, you get “free” access.

      Be careful with AMD GPUs as they are not as well supported for local AI. However, support is gaining ground. Some people are doing it but it takes effort and hassle, from what I’ve read.