Run Large Language Models locally

If you wanted to use Large Language Models (LLM) without sharing all prompts and other data with Big Tech, or integrate a chatbot into a system that you control fully, or simply preferred open-source solutions wherever possible (it’s a Mozilla initiative!), then llamafile is worth a try.

Llamafile makes hosting a LLM chatbot locally as simple as downloading a single file and starting it. That’s all. It took me less than a minute to arrive at my first home-spun text. I won’t repeat the straight-forward setup instructions here.

I have been using LLMs a lot for teaching this year. Classroom discussions in which students were challenged to critique chatbot generated content was fun (I believe) and helped to focus on the key arguments and to ignore the smooth-sounding fluff. I am happy to generate the required discussion fodder with ChatGPT-4 or similar online.

My main use for LLM these days is more basic: Automatically check spelling and grammar in my emails and other work. These checks got so much better when LLMs arrived and I rely on them heavily. However, I don’t want every text to go through more computers than needed while drafting. That’s why I am keen to find a local solution that is as good as the leading online services.

How good are the results? llamafile can let you run a number of models. I am still trying to see which works best for me. The default LLaVA model is trained on fewer data than ChatGPT-4, but it’s performance is not too far off, allegedly.

Local spell checks with LLaVA work well. But the model runs rather slowly, even on my strong laptop. Putting a this chatbot onto a Raspberry Pi is out of reach, I suppose. Also, it’s not well-integrated into my workflow, yet (manual copy and paste, for now).

Not sure I will need this feature much, but my new AI helper can analyse photos, too.

User: Describe the image and tell me if the person is happy.

Llama: The image features a man wearing a blue jacket, standing in front of a wooden wall and pointing his finger upwards. He appears to be gesturing towards the ceiling or an object above him. In addition to this main figure, there are two other people present in the scene; one is located on the left side near the bottom edge while another person can be seen further back at about half of the image’s height. The man pointing seems focused and engaged with his surroundings.

User: Is he happy?

Llama: Yes, the man appears to be happy while pointing his finger at something.

The ultimate test, of course, is to blurb out an inspirational post for your LinkedIn profile.

User: Write an inspirational post for LinkedIn that explains how I became a millionaire by wearing black socks.

Llama: Here’s your inspiring post! 🚀

Title: The Power of Persistence & Choices - How Wearing Black Socks Made Me A Millionaire! 💰✨

Dear Valuable Network,

Today, I want to share a story that has transformed my life and taught me the true power of persistence, choices, and consistency. It all started with an unconventional decision - wearing black socks! 🧦✨

Years ago, as a young professional in search of success, I found myself struggling to make ends meet. One day, out of desperation, I decided to try something different: wearing black socks instead of the usual white ones. This simple yet bold choice sparked an unexpected chain reaction that would change my life forever! 💥

At first, it was just a small step towards self-expression and confidence building. However, as time went by, I realized that this seemingly insignificant decision had a profound impact on various aspects of my life - including my career and finances. Here’s how: 💡

1️⃣ Attention to Detail & Presentation: Wearing black socks made me more conscious about my appearance, leading to better grooming habits and a stronger sense of self-confidence. This, in turn, positively influenced the way I presented myself at work and during job interviews, ultimately opening new opportunities for career growth! 🚀

2️⃣ Unconventional Thinking: My decision to wear black socks sparked a shift in my mindset towards embracing unconventional thinking. It taught me that stepping outside the box and making bold choices can lead to remarkable results, even if they

Overall, this is very exciting. What should we build first?