Local AI will be harvested - if not today, then as soon as tomorrow. I recommend not trusting any system like this with any sensitive information… Or, honestly, with most non-sensitive information.
When people say Local AI, they mean things like the Free / Open Source Ollama (https://github.com/ollama/ollama/), which you can read the source code for and check it doesn’t have anything to phone home, and you can completely control when and if you upgrade it. If you don’t like something in the code base, you can also fork it and start your own version. The actual models (e.g. Mistral is a popular one) used with Ollama are commonly represented in GGML format, which doesn’t even carry executable code - only massive multi-dimensional arrays of numbers (tensors) that represent the parameters of the LLM.
Now not trusting that the output is correct is reasonable. But in terms of trusting the software not to spy on you when it is FOSS, it would be no different to whether you trust other FOSS software not to spy on you (e.g. the Linux kernel, etc…). Now that is a risk to an extent if there is an xz style attack on a code base, but I don’t think the risks are materially different for ‘AI’ compared to any other software.
If you connect it to the Internet then sure it can be easily harvested by large companies. Pretty sure you can host an offline AI in a device you have made sure the hardware isn’t gonna be phoning home and it’ll probably be fairly safe if you aren’t an idiot like me and actually know what you’re doing.
Scenario time: A loved one has recently passed away, and I want to find all the photos I have of them. I would love to be able to have a local AI perform facial recognition to help me find these photos. The classification and tagging info doesn’t get fed into surveillance capitalist garbage, and I’m still able to benefit.
Mate, something like Immich or digikam (if you want local) will do a good job at this. Not perfect but perfection is utopia. I fed 40k images to Immich and it did a reasonable job in not too many hrs.
scenario time: you haven’t taken 40 thousand pictures over the last three years because you aren’t cripplingly addicted to technology so you can sort through them manually in about 10 hours or so.
There’s no reason to judge someone for taking many photos. If you’re not willing to help, you don’t have to. There’s no need to write sarcastic comments.
likely GPU ram, there is some tech that can offload ram, but generally it’s all hosted in VRAM, this requirement will likely fade as NPUs start becoming a thing though.
Don’t trust the cloud. You can setup AI locally.
Local AI will be harvested - if not today, then as soon as tomorrow. I recommend not trusting any system like this with any sensitive information… Or, honestly, with most non-sensitive information.
This guy is clueless
if you use windows, sure.
Don’t use windows.
When people say Local AI, they mean things like the Free / Open Source Ollama (https://github.com/ollama/ollama/), which you can read the source code for and check it doesn’t have anything to phone home, and you can completely control when and if you upgrade it. If you don’t like something in the code base, you can also fork it and start your own version. The actual models (e.g. Mistral is a popular one) used with Ollama are commonly represented in GGML format, which doesn’t even carry executable code - only massive multi-dimensional arrays of numbers (tensors) that represent the parameters of the LLM.
Now not trusting that the output is correct is reasonable. But in terms of trusting the software not to spy on you when it is FOSS, it would be no different to whether you trust other FOSS software not to spy on you (e.g. the Linux kernel, etc…). Now that is a risk to an extent if there is an xz style attack on a code base, but I don’t think the risks are materially different for ‘AI’ compared to any other software.
How? It is running locally in a VM. I could even air gap the VM if I wanted
If you connect it to the Internet then sure it can be easily harvested by large companies. Pretty sure you can host an offline AI in a device you have made sure the hardware isn’t gonna be phoning home and it’ll probably be fairly safe if you aren’t an idiot like me and actually know what you’re doing.
If you install it locally, it will be as secure as any other thing you do on your computer.
But I run the cloud myself
Mate, please be kind and help a fellow brother with that…
For example I need a tool that will “automagically” sort all my documents, photos and videos on premises.
Thank you!
I use Ollama plus openwebUI. It won’t sort your documents but you can upload a doc for AI tasks.
Sort them… by what?
Just fackin’ sort it ooouuttt!
Edit, but seriously sorting in most scenarios is just by date created or name, which most file explorers can do.
That Isn’t generally an AI task.
If OP wants the AI to read the file and sort by colour for example, then this is maybe an AI task but sounds more like a software task.
Scenario time: A loved one has recently passed away, and I want to find all the photos I have of them. I would love to be able to have a local AI perform facial recognition to help me find these photos. The classification and tagging info doesn’t get fed into surveillance capitalist garbage, and I’m still able to benefit.
Mate, something like Immich or digikam (if you want local) will do a good job at this. Not perfect but perfection is utopia. I fed 40k images to Immich and it did a reasonable job in not too many hrs.
scenario time: you haven’t taken 40 thousand pictures over the last three years because you aren’t cripplingly addicted to technology so you can sort through them manually in about 10 hours or so.
You’re proposing to waste 10 hours sorting photos when the right tool could probably do it in less than 2 minutes? What?
And how does taking pictures translate to being addicted to tech?? We’ve had photography for over 100 years
There’s no reason to judge someone for taking many photos. If you’re not willing to help, you don’t have to. There’s no need to write sarcastic comments.
Even OpenAI can’t do that lol
An LLM like Ollama won’t help with that. Something like Photoprism could, it uses ML to automatically tag media and recognize people.
Is there a magazine or site that breaks this down for the less tech savvy? And is the quality of the AI on par?
Your best bet is YouTubing ollama.
On par? No. Good enough? Definitely. Ollama baby
Ollama with Lava and Mistral
Check my notes https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence but as others suggested a good way to start is probably https://github.com/ollama/ollama/ and if you need a GUI https://gpt4all.io
I’m not the person who asked, but still thanks for the information. I might give this a try soon.
Ditto, thanks to everyone’s for their suggestions
Is this gpu ram or cpu ram?
likely GPU ram, there is some tech that can offload ram, but generally it’s all hosted in VRAM, this requirement will likely fade as NPUs start becoming a thing though.
ollama is your friend
*Mistral