# Everything I've learned so far about running local LLMs

<https://nullprogram.com/blog/2024/11/10/>

- Host: [nullprogram.com](https://links.thingelstad.xyz/hosts/nullprogram.com/)
- Saved: November 11, 2024 (2024-11-12T02:20:46Z)
- Record: https://links.thingelstad.xyz/links/8e792ca985e5/
- JSON: https://links.thingelstad.xyz/links/8e792ca985e5.json

## What Jamie wrote

Wellon's shares expansive feedback about his experience with many different LLM models that he has personally worked with. The whole read is interesting. I liked his conclusion that is based on what he sees LLMs best at right now.

> So then, what can I do with LLMs? A list is apt because LLMs love lists:
> 
> - Proofreading has been most useful for me. I give it a document such as an email or this article (~8,000 tokens), tell it to look over grammar, call out passive voice, and so on, and suggest changes. I accept or reject its suggestions and move on. Most suggestions will be poor, and this very article was long enough that even ~70B models suggested changes to hallucinated sentences. Regardless, there's signal in the noise, and it fits within the limitations outlined above. I'm still trying to apply this technique ("find bugs, please") to code review, but so far success is elusive.
> - Writing short fiction. Hallucinations are not a problem; they're a feature! Context lengths are the limiting factor, though perhaps you can stretch it by supplying chapter summaries, also written by LLM. I'm still exploring this. If you're feeling lazy, tell it to offer you three possible story branches at each turn, and you pick the most interesting. Or even tell it to combine two of them! LLMs are clever and will figure it out. Some genres work better than others, and concrete works better than abstract. (I wonder if professional writers judge its writing as poor as I judge its programming.)
> - Generative fun. Have an argument with Benjamin Franklin (note: this probably violates the [Acceptable Use Policy](https://ai.meta.com/llama/use-policy/) of some models), hang out with a character from your favorite book, or generate a new scene of [Falstaff's blustering antics](https://nullprogram.com/blog/2023/06/22/#76-henry-iv). Talking to historical figures has been educational: The character says something unexpected, I look it up the old-fashioned way to see what it's about, then learn something new.
> - Language translation. I've been browsing foreign language subreddits through Gemma-2-2B translation, and it's been insightful. (I had no idea German speakers were so distrustful of artificial sweeteners.)
> 
> Despite the short list of useful applications, this is the most excited I've been about a new technology in years!

That last sentence — even with the limitations he highlights — most excited in years. I think we are still at the very early beginnings of all this. [via Simon Willison](https://simonwillison.net/2024/Nov/10/running-llms/).

## Facts

- Kind: [Page](https://links.thingelstad.xyz/kinds/page/)

## Appears in

- [WT301: Weekly Thing 301 / TinyTroupe, Monarch, Leaving](https://weekly.thingelstad.com/archive/301/), Notable

## Saved the same week

- [Clear ice | Seth's Blog](https://links.thingelstad.xyz/links/c4dbf2ea8820/) (seths.blog, November 11, 2024)
- [Your Hacker News](https://links.thingelstad.xyz/links/3579f249f7c6/) (yourhackernews.com, November 11, 2024)
- [How to Tame Sequoia’s Window Tiling - TidBITS](https://links.thingelstad.xyz/links/c874c3416cff/) (tidbits.com, November 11, 2024)
- [Draw.Audio](https://links.thingelstad.xyz/links/c8ad78bd7e89/) (draw.audio, November 11, 2024)
- [Monarch - Spotlight Search with Superpowers](https://links.thingelstad.xyz/links/bc3b9a57025b/) (monarchlauncher.com, November 11, 2024)

## Elsewhere

- Wayback Machine: https://web.archive.org/web/*/https://nullprogram.com/blog/2024/11/10/