I think I pinpointed a misgiving I have about general collaborative LLM use: that the LLM will never stop and say "I don't know" or "I don't have this mental model, can you help me understand it". That's the fundamental limitation of being trained to plausibly (not correctly) generate text.

I guess it gets fuzzier with the reinforcement learning layered on top, right? But can you really reinforcement train it to say "I don't know"? You can train it to say "I don't know" based on patterns of when it's training data shows people saying "I don't know". But it's against its grain, right?

If I had a collaborator who never said "I don't know" that would be a bad collaborator - I'd feel like I never had a steady foundation with them. Sometimes this does kind of happen because people feel embarrassed not to know things - they may commit code they don't understand and not want to admit it. But I feel like a key part of me getting to know someone and working with them is getting past that to where we can talk about what doesn't make sense - because a solid, shared mental model benefits us both.

I definitely use LLMs to speed up my development, but mostly in the form of autosuggest, which to me feels like the most 'honest' or 'with the grain' format for it. I'll also have it generate more generic components or bash scripts or database logic. But even then I feel like I really want to keep thinking of it as a form of 'codegen', of 'predictive text', and not a collaborator.

I don't know the consequences of all this - maybe accuracy crosses a threshold where the hallucination tradeoff is negligible. Maybe grounding in documentation helps a lot. And possibility you can do some chaining of an evaluator model... Ideally I think you'd get a confidence or 'novelty' score from the model itself, that you'd learn to factor into your intuition about how to treat what it's generated, but I don't see a lot of progress or talk about that lately.

Maybe this is just an argument that there's some metaphor beyond chat is called for. Something that signals its going to 'run with idea' or 'play it out', not 'answer your question definitively'.

One of my favorite pieces of writing is The Web's Grain by Frank Chimero https://frankchimero.com/blog/2015/the-webs-grain/. Lately I've been trying to think about what the grain of LLM-related development could be.

There's the interface for using the LLM, where Cursor has done a lot of work. But I want to think about what kind of websites and software you can build with LLM assistance.

There's using LLMs as a piece of the pipeline - unstructured data into a JSON schema, for example. There are possibilities there, I think. The dilemma is probably giving the user a mental model of what they can do - LLMs can unlock a new level of fuzzy input, but will it be clear to the user where and how that input breaks down? Will they run to the edges and then be frustrated there?

Then there's codegen and vibe-coding. This is such a knotty feedback-loop. It's safe to say LLMs are good at boilerplate, particularly the most popular project boilerplate. So one approach to going with the grain could be picking a stable tech stack and coding according to convention.

A really interesting piece of this is how libraries fit in. I think a lot of people's favorite vibe-coded examples use libraries like three.js or matter.js, to simulate worlds or physics. I think in general there's an imprecise linking of a model's 'intelligence' and physics sims, where we imagine a model has an understanding of the world, but what it actually has is ability to hook into a carefully designed library intended to simulate the world.

But I do love physics sims, and I think there is some energy there where we've always wanted physics sims in our software, but the complexity and the clash between digital logic and physical world logic has been to much to contain. Maybe LLMs can be the glue that helps us meld those two together.

Another place LLMs have been useful for me is approaching FFMPEG syntax commands. Another powerful library case where it allows me to dip into the powerful capabilities there and dip out without having to puzzle over syntax.

There's definitely a flow where you mainly use LLMs for boilerplate and for tech adjacent to your main focus. I've done that on this blog with most of the database logic being codegen. I can see a near-term future where my focus is experimental UIs, stuff that can't be easily generated, and I lean on LLMs to generate the surrounding tech.

But I don't think that's exactly approaching LLMs as Chimero approaches websites in the article - it's not looking at what 'LLMs' want to make. For that I don't have an answer currently beyond popular web stacks and this interesting relationship to libraries. Going to keep thinking, though.

I've been working on building rough shelves out of 2x4s in our basement. Partly for storage, partly to get more of a feel for how to frame things with 2x4s. The big improvement on the second set was using 3" deck screws. The process made me think about learning and how it feels different to have solved a problem versus avoiding having a problem.

On the first set I had just got some long phillips head screws from the hardware store, but they kept stripping out. I had some idea that construction screws used other drive heads - and it became clear that this was probably the reason.

I switched to star head screws and those drove better, if still a little rough. Then I watched this https://www.youtube.com/watch?v=SG6FJCWYeRU video and followed the advice to get 3" deck screws.

For the second set of shelves I used the deck screws, they drove easier, and the space at the top of the screw meant the two 2x4s pulled tight.

Thinking about learning: I could have got all this information at the start, but it feels so much more real to have gone through the process of encountering the problem - really felt that problem - and then the difference of the solution. I love that process.

I think that sort of feeling is a key part of building intuition.

Another interesting puzzle for me is why intuition is hard to pass on - having experienced this issue could I now tell someone else that the secret to building 2x4 shelves is to use deck screws? I would probably mention it. But I think the tricky part is there's lots of other parts to building shelves, which the person I'm talking to might know more or less about. It's a huge set of information. I (and I think a lot of people) need to be in the process and solving problems step-by-step versus being given a giant checklist at the start. At least that's how I enjoy doing it.

I read (listened) to Nick Harkaway's Karla's Choice recently and then went back to the BBC audio versions of a bunch of the George Smiley novels. It's fascinating Harkaway can write Smiley so well when the tone of his own novels (which I also like) are so different.

I'm not sure why I like the Smiley novels so much. They're always weirder than I expected -- they are kind of plotty in a detective way -- but really they're most often weirder and sadder. I think I always like an emotionally repressed person who occasionally says something that reveals what all is going on unsaid. I guess that's English-associated. I always think of Remains of the Day as the big example though it's been a long time since I read and watched that.

The le Carre book I think about the most is A Perfect Spy, which I think I got because someone recommended it as his most literary work. It was tough to get through at times -- just because of the bad things piling up -- the indications things were going to a terrible end.

The thing it was best at was demonstrating how weird it is to want to be a spy. How broken of a person you have to be. Interesting to think about in relationship to how much media we have about spies. In this case the character's broken by his relationship to his con-man father. By needing to take care of him. By learning from him even where he didn't want to. There's an unsparing-ness to it that I always find impressive -- like he's willing to see things as they are. (Though that's a fine line -- some art veers towards "if it's the rougher view it's the true one" which is not.)

scrawl is back. using waku RSC based on the feed rebuild.

Shower idea: spreadsheet style image editor. (Also very much inspired by the way canvas works.)

There would be a source image and a destination image (maybe call the destination image canvas?). Break the image up into 16x16 cells and stretch image to fit. Then you've got a selector, that can expand to different cell sizes, and can copy-paste paint from the source image to the canvas.

So for that you'd need to be able to controld the selector, resize it, and change the source cells (while the destination cells stay the same).

You might also need layers to keep things non-destructive.

Actually you might want two modes of movement on the canvas, one where the source moves with it, and one were the source stays static. All of these should be instituted like live formulas from excel. Although you wouldn't be able to do a mutating feedback loop exactly, because your original is kept in original condition and you're always linking from there -- unless you also have the option to link from your destination. THat might be fun.

You have two different color outline to indicate source and destination. That's actually quite a few combinations of modes - might take a while to get controls that feel good.

I thought you would want to be able to change resolution (up and down). But actually it might be better to be able to zoom by factors of 2, like xzoom does. Depends which project this is a part of.

It would definitely be fun to show the source and destination info at the bottom in text as an excel type formula.

Something I've remembered for a long time, is the description, Chris Langan, who supposedly had the highest IQ in the world, of the kind of thinking he does for pleasure (as told in a First Person episode). He talks about imagining the universe, and all the forces at play within it. I recognized the pleasure of some of that. One of my main pleasures from design/programming is holding a system in my head, and imagining how different adjustments would ripple out through the system. Often it is problem solving: 'how can I better balance this system', but not always.

I have been thinking about that pleasure, and what is at the core of it in relation to mental models. It seems connected to so many other descriptions but I still feel like I have a fuzzy picture of it. It seems connected to the pleasure of physicists, coming up with an elegant explanation for the physical world (and I have read some discussion of what elegant actually means: to explain the most with the least, although there are tons of judgements within that). It also seems connected to the pleasures of worldbuiling, of telling a believable (recognizable) fictional story. Because, I think, being able to do that implies you have the model of the real world in your head, or of a system of relationships between people.

Yesterday I was reading Color and Light, about how painters learned to paint the effects of light. From that, I got a picture of painting physical scenes as your ability to hold a model of the affects of light in your head, and then to apply those to the fictional characters and objects you put in the scene. Interesting to compare that to present-day videogame rendering, where people need to build a model of the world into the game. In both cases I wonder how big of a piece the understanding of the systems in the world is, and how much of it is emulating those systems in your chosen medium (paint or computer graphics rendering), that's something I want to write more about.

How about flow state? Is holding things in your mind pretty much flow state? I don't think so, and they seem close enough that the effort to distinguish between them could be useful in understanding both. Flow state I think also implies that you have a model in your head (although it is implied it is more inutitive, less explicit?), that's how you're able to maintain flow state, I think. You can operate in a flow state because you have enough of a model that you can move confidently, without getting stuck in errors and having to look stuff up. A flow state is full engagement with the system. I don't think it means you have a complete understanding. Even now, just thinking about it I can conjure up this feeling of full engagement, of attention, almost like I'm balancing a system of physical objects.

The idea of a model you're trying to approach seems easy to relate to programming. What about art? What about just trying to create something beautiful? What is it there? In Color and Light, Gurney talks about painters trying to ocnvey human emotion through a landscape. Why use a landscape? Why not communicate more directly? Why use attention to the physical world as an intermediary? After that he showed some symbolist paintings, and looking at those I thought, well, maybe that is one reason, if you try to convey them directly (anger as a demon) it looks trite. That explains some of it I think. But it still doesn't explain to me why the landscape is a successful way of conveying emoltion.

I had another answer, which I think I can write well but still doesn't seem enough. hHat painting a landscape with the real attention to detail is an act of attention. Is it that attention is everything? Real engagement is everything? And that because we know something of how a painting is made, the labor involved, the successful painting is just a message 'this is worth paying attention to'. And then it could be anything? It seems to follow that it could be anything. And beautiful paintings have been made of anything. But that still seems like a dissatisfying answer in terms of how to make art, how to live. Maybe it is 'pay attention to things that resonate for you'. Then the question is, is the success because it also resonates for another person? Or because you successfully communcated your experience of the resonating?

Was this what I wanted to write about? Mostly. I wanted to think about holding a system in your head in relation to using tools. Because do I want to make tools you can hold in your head? What am I making tools for? How does ego come into all of this?

I've been really interesting in Dan Abramov's Just JavaScript course. The course (so far) takes the approach of treating the rules of JavaScript as natural laws, and a lot of the explanatory effort so far has been about aligning your mental model to those laws (there is a long running idea of the connection between variables and values as 'wires', for example). As it's been introductory so far, it hasn't exposed me to a lot of new thinking, but it has strengthened my understanding of how object properties work. This was not something that tripped me up a lot, but I realized after going through the JJ explanation that it had been weighing on me as I wrote code -- I'd been working around the gaps in my knowledge to an extent and I'd also been actively defending against bugs related to those gaps. So if I wrote code I was unsure about I would save and test it in the middle of writing, versus feeling confident enough to finish out the function I was trying to make. I was surprised how much of a benefit that 'in-world' knowledge of javascript provided. I felt on much surer footing while writing code (even though the new knowledge had very few direct effects on the code I wrote). It was a feeling of comfort, I keep wanting to call the feeling 'being on sure footing'. It wasn't that I could access new places exactly, but I felt a lot more grounded and confident writing the usual stuff, which made writing code a more pleasant experience. It was also like having less drag -- especially as I built a larger app, where accumulated uncertainty kind of weighs you down as you go (some expectation that something is going to break and you're not going to know the reason).

So I"ve been enjoying that experience and also thinking a lot about what it means on the meta-level. Because along with that course, most of my thinking about code lately has been focused on learning more about how a computer itself workers. This really got kickstarted with reading Code: A Hidden History. The idea behind all that, I think, was that these languages are leaky abstractions, and that the only way to feel on truly solid ground would be to really understand it from the ground up. I'm not done with that approach, but it Just JavaScript is causing me to reassess it a bit. Maybe there is no tension between the two projects, it just depends on what you're doing, but it definitely feels like there's something to be productively explored there.

The question is maybe just at what level of abstrac tion do you want to explore things. That seems like the central question for any tool. Closely related to it is what level of primitives do you want to expose to your user (and there is lots of nuance in that, maybe you expose one level of primitives, but there is an escape hatch to let people go even lower-level than that). That is something I want to think about. I want to think about why ithat decision is important, too (what are you ultimately trying to enable)? People to create things (yes)? People to create things where they feel in control (on solid ground) (yes, I think so)? How does this compare to the physical processes people use to create things (like pottery, or paining, where the material has a feel and a resistance to it)? It seems like making a tool is a process of finding the right constraints. The constraints of physical things are so varied (what makes learning painting something people waht to do? learning to play an instrument? Think about the end product versus the feel of the process).

That's something I possibly hadn't articulated before: end product versus the process. Painting and playing an instrument have a long learning curve -- I would imagine motivated by the end product (think of what I could make). What do I want to do with a tool? The end product is the carrot, but I want the process to feel good. Are video games the opposite end of the pole (not quite, they're pretty goal-oriented as well).

Slowing down, but I will throw in that also, always with this stuff, I am thinking of lego and minecraft. About their phenomenal success, their enduring attraction, and why more things aren't in that realm (why not more games that play to those impulses? Is it that people only have room for a few of those things? There's also the Sim- series of games, I suppose, but still it seems underexplored).

What else? Another inspirational part of Just Javascript is that there is still room, giving all the approaches to teaching code, for an approach that (seems) new to me and is effective. It is affirming for my faith in the importance of mental models, and as I think about making my own stuff, and have worries that I'm operating at too basic a level (that all the opportunities for this sort of thing are exhausted) this makes me feel like, no, there's still room, start from the beginning again, don't take things for granted, if something doesn't make sense dig into it more. I should write more about primitives more, I think that level of abstraction you enter into is something I need to think more about.

Rgb is the newest experiment on Constraint Systems, it lets you select different rgb values using key color combinations -- simulating the experience of mixing colors.

Rgb fell out of another project, where I was returning to the grid-based editing of Hex trying to put it together with the sprite-based font editing from Face. With outlining and text, I'd be close to the grid-based paint program useful for creating diagrams. I decided the missing piece was color, and then I got caught up trying to decide the best interface for inputing color. I originally thought I'd use a color palette with a few (8, 16?) colors to be selected, but I wanted to experiment to see if there were more intuitive interaction set-ups -- something more similar to the hex experiment.

I decided to try a method where you use key combos to set the rgb values, one column for red, one column for green, one column for blue. Each column has options for 0, 122 (half), and 255 (full). This lets you do the basic combos, and also get mixes for things like yellow, cyan, and orange.

I used rgb because it corresponds to the colors pixels are made up of in the computer display. In some ways, if I want an intuitive, painterly interface, it would make sense to use a color space that fits better with human perception like HSL. But something I'm narrowing in on with these experiments is what exactly I want to do. I think what I want to do is bridge the gap between the computer in the user in an interesting way. I don't want to build an interface where the user doesn't have to think about what is going on in the computer at all (an approach I associate with Apple). I want to "go with the grain" of what's there. By providing an interface where people can productively collaborate with the computer, while also getting used to some of the lower-level abstractions, I can help them build knowledge that will carry over to the larger computer system and will make them feel more empowered to do other things.

That is the more philosophical motivation. It's also true I think that creative systems just thrive on constraints -- so why not use the constraints that are built into the computer. This makes for a collaborative interface on the part of the user, and it's also an interesting collaborative process for me to develop the interface, choosing which parts to augment of make more forgiving and which parts of the representation to keep laid bare.

The interface also provides a playground. One of the interesting things related to AI and its current limitations, is the question of whether it can really learn without an embodied experience of cause and effect. One of the experiments of constraint systems is how quickly we can adapt to a new system if we have quick and consistent feedback. It's something I've been exploring myself in the computer programs I use -- switching to Linux and using lots of keyboard shortcuts, using a smaller keyboard. I've been surprised at how possible the switch is, though it eased a lot by a consistent set of expectations, and a consistent mental model of how the shortcuts and the system works. I'm interested in creating those experiments in miniature. I'm also interested in the overlap between these ideas and how you learn a videogame or an instrument. I think about the system behind Minecraft a lot. Where you have this set of rules that you can reason about, there's also a lot of eccentricities, but I think the block rule set keeps it grounded. I think about how popular Minecraft is (the most popular game of all time!) how it fits into a human desire to create.

I released Face on Monday. It got more Twitter attention then any of the other Constraint System releases so far. Which feels good! Although I also do notice that there wasn't a lot of direct feedback -- or (what I'd really like to see) people sharing things they made with it -- just likes. I did get some feedback I've been thinking about: that the 'hjkl' direction navigation (borrowed from Vim) is a high barrier to people jumping in.

Why do I use the navigation? Well, first because it is what I use daily, in Vim, in i3wm, and while browsing the web thanks to the Vim plugin. Also since I've switched to the Atreus keyboard, my actual arrow keys are a couple layers deep, making 'hjkl' even more convenient.

Ok, that makes sense, but why not give people not used to it the arrow option? That speaks to a broader design question. I started doing the Constraint Systems stuff directly inspired by 100 Rabbits, specifically Noodle, which is a pretty uncompromising drawing tool (for one thing it starts with a completely blank screen). It drove home to me that you can just do that -- make exactly the tool you want exactly how you want it, and, in 100 Rabbits case, you can even build a community of like-minded people around that tool. This is especially in contrast to when I'm designing something at work and I feel like I have a responsibility: to the team, to the project, to make things accessible to a broader audience.

Closely connected to that is feedback/iteration speed. If I'm building the tool exactly how I want it, then when adding or modifying a feature I simply have to determine if it feels good and makes sense to me. If I start extending the configuration, so that it can be used in ways that I don't, I add overhead, where I have to check new features against both my use and the alternate flow I've added in. (Another option is to make things configurable, which I'd like to explore sometime soon, but I also can't expect configurability to solve all my problems, the vast majority of folks will stick with the defaults.)

Also tied to that, is that I view the constraint systems project as my journey to 'find my own way', something I always have trouble with when giving feedback is if my feedback makes sense for my mental model of the project, but not the person creating it. Sometimes they'll try to satisfy my feedback, but if they aren't on the same page about how it fits in, chances are they'll do something down the line that will clash with my suggestions. Usually I think that I would rather see the fully- formed, fully-developed thing from someone else. I'd rather see a coherent mental model I don't totally align with than a mish-mash.

In fact, I think what is interesting in a lot of the 100 Rabbits projects is to me it often feels like they're coming from an alien logic (Orca especially is like this). That's a big part of the thrill of it. It's like my favorite trope in sci-fi books of how an alien language can unlock new modes of thought.

But what comes with this is a big learning curve. And I think that's what I need to think about more seriously with the experiments. Because if I want people to drop in and create and share, that unfamiliar confrontation is going to cause many to bounce off. I think in the back of my head I'm still imagining Constraint Systems building in to some sort of app that combines writing, drawing, diagramming and maybe image editing, and if you're going to go through all that you may be more willing to perservere through an unfamiliar control scheme for a payoff down the line. But if you just see a GIF on twitter and click through... you're less likely to do that.

So I need to think about what I want to do. I also feel like I am doing a slippery slope thing, where I want to make sure I don't chase after every piece of feedback I hear (I've worked on projects I felt were doing that and found it frustrating), but I also shouldn't get too strongly attached to some perceived purity. Because on a basic level what I'm talking about is enabling arrow keys for navigation along with hjkl. Something even Vim, which is not exactly beginner friendly, does. And I think I can go ahead and do that without feeling like I'm really compromising the vision of anything -- especially if I think of it as making it more possible for people to make things, which is my top goal.

It doesn't come without a cost (some part of my brain shouts) though. Even though enabling them is trivial code-wise, I will need to think through how I expose it in the instructions, instead of saying 'j' for down, I'll need to say 'j' or '↓' for down. Which does add some overhead. Probably worth it, though? Trade-offs!