Setup your own private chatGPT \ stacker news ~tech

2604 sats \ 11 comments \ @nullama 15 Aug 2023 tech

I like that you can use chatGPT as a tool, like a spellchecker on steroids, but I'm not happy to send all my private information to a server outside of my control.

There are a few projects out there that try to replicate chatGPT with a local application, that do not send any of your information to servers outside of your device. Everything is done locally.

I tried a few of them, such as privateGPT and gpt4all and each have their pros and cons. Here I want to tell you how to setup one of them that I found the best for most people.

Step 1: Download the installer for your OS here

Step 2: Open the application, it will ask you to download a model. Select GPT4All Falcon.

Step 3: Use it.

https://imgprxy.stacker.news/q6cbwi0_YsB31pVje7V53HSUkCbbPIhjAiMe2sMZ48A/rs:fit:600:500:0/g:no/aHR0cHM6Ly9pLmltZ3VyLmNvbS9ZcjlsRXNKLnBuZw

It only takes a couple of seconds to generate a response.

You can also add your own documents under Settings->Plugins->LocalDocs Plugin. Once you add a folder, click on the Databse looking icon and select which folders you want to include in your query.

It works surprisingly well for a local app that runs on a standard computer in real time.

Of course, it's not perfect, but I see it as a nice tool to have.

view all related items

30 sats \ 0 replies \ @tomlaies 15 Aug 2023

There was a post about exactly that in combination with Obsidian here recently

10 sats \ 5 replies \ @bataroot 15 Aug 2023

Has anyone here experimented and documented the specs of a home AI standalone machine setup?

Following what Guy Swann is doing here, but that would be great to have recommendations: https://snort.social/e/nevent1qqswtzhnzm995h69t2mwnmte9ntflknrsdc4shcrje266ec5s7026xcpzpmhxue69uhkummnw3ezuamfdejsyg9euaj5dwsxg4hdxqweu54uf8ay3ec2d0ezs2l85xh899rkzgprmspsgqqqqqqs3rs8np

15 sats \ 4 replies \ @nullama OP 15 Aug 2023

It really depends on what you want to achieve.

A reasonable laptop can run this GPT4ALL and StableDiffusion quite well.

Probably the best deal would be a mac mini with at least an M1 chip, those are extremely powerful machines, and their are silent. They do run both GPT and SD without any issues.

10 sats \ 3 replies \ @m00ninite 16 Aug 2023

I've been considering an M1 Mac mini as an AI workhorse. The minis in particular are unusually cost-efficient for an apple machine

0 sats \ 2 replies \ @nullama OP 16 Aug 2023

Apple silicon is a game changer.

0 sats \ 1 reply \ @m00ninite 16 Aug 2023

How quick are stable diffusion and gpt with M1, in your experience? I tried running a gpt-like on some regular x86 desktop hardware and it was horrible, basically unusable.

20 sats \ 0 replies \ @nullama OP 16 Aug 2023

SD, particularly from here takes a few seconds, maybe a minute or two depending on other load (I'm usually doing other stuff there) to generate a full image on an M1 mac mini. GPT responses following this particular setup should generate a response in a few seconds.

There's a lot of tweaking that can be made, in particular, if you have a GPU, you can configure a llama model to be run in parallel on the GPU.

12 sats \ 0 replies \ @aljaz 15 Aug 2023

there is also https://www.premai.io/

2 sats \ 0 replies \ @2bithits 15 Aug 2023

Thanks for sharing!

0 sats \ 1 reply \ @Zepasta 15 Aug 2023

I tried it out. It's extremely slow. I sent a simple "Hello" prompt, and he took like 10 seconds to write me response back. My PC is somewhat decent (i7 and RTX 2000 with 16 GB RAM).

0 sats \ 0 replies \ @nullama OP 16 Aug 2023

Interesting.

I think there's no GPU acceleration at all by default at least on this setup, so basically the entire speed is based on the CPU and the instructions it supports. The more modern the CPU, the better.

I've read that you can configure llama to use the GPU to get much better results