feat: i built a transparent magic 8-ball (missed the point entirely)

My phone buzzes on a Tuesday night, and it's a friend of mine. And the message is a screenshot. Not a screenshot of a conversation, a screenshot of a Magic 8 Ball. The little triangle floating in the blue window, and underneath it, in her own words, says, "Okay, said yes three out of five times, so that's basically a yes, right?" And I asked the obvious question, which was, "Yes to what?" And she says, "Texting him back." I asked my actually obvious question, "What were the other two rolls?" She says, and I'm quoting this directly, "One said 'ask again later,' and one said 'my sources say no.' But that one doesn't count because I just spilled my drink right before I asked." That is a real argument made with complete sincerity by a grown professional adult more than once about more than one topic over the many years I've known her. She's not unintelligent. I need to say that up front because we're about to spend an episode gently making fun of a decision-making process, and I don't want it to sound like I'm making fun of her. She has a graduate degree. She manages teams of people, and in every other observable way, she is a rigorous thinker. She also, that Tuesday, rolled a random number generator five times until it agreed with what she already decided to do, and then used that as evidence. And so, I did what any reasonable friend with a software engineering background does in this exact situation. I built her a better one.

Welcome back to Chaotic Commits. I'm Joanne Skiles, and I spent about, maybe less than 30 minutes building my friend a Magic 8 Ball. Not because she needed one, because I wanted to show her something. I wanted her to see, plainly, sitting right in front of her, that the thing she's been treating like an oracle was 20 lines of code and an array. And I wanted to open the black box. She loved it. That was not the lesson I was trying to teach. Today we're talking about Magic 8 Balls, the CI pipelines, and why you can hand someone the entire source code for the thing they trust and it will not touch, even slightly, why they trust it. This is an episode about transparency and what it's actually for, and the uncomfortable thing I learned about my own research when the test subject was somebody I'd known for 15 years and the whole experiment ran for free on her couch. Let's get into it.

The Magic 8 Ball, the actual physical toy, was invented in the 1950s by Alabe Crafts before Mattel picked up the license. Inside the ball is a plastic icosahedron, a 20-sided die, floating in dark blue alcohol-based fluid. You shake it, the die tumbles, one face presses against the little window and you read the answer through the murk. There are 20 possible answers on that die. Not 200, not as many as the situation requires, 20. 10 affirmative, five negative, five noncommittal, the ask again later and all its cousins. That is the entire epistemic range of the device that my friend, and probably several of yours, have used to make actual decisions about actual lives. 20 faces. No memory of the last question, no awareness of what the question even was. The die does not know you're asking about a man named whoever he is. It doesn't know you exist. It is a fluid dynamics problem wearing a personality.

The online version, the one people actually use now, isn't even physical. It's an array. 10 to 20 strings of text, a random number generator, pick an index, print the string. If you have ever written a guess the number tutorial in your first programming class in whatever your first language was, you have built something functionally identical to a Magic 8 Ball. You just didn't put a plastic sphere around it.

None of this is a knock on the toy, by the way. The toy is honest about being a toy. It's the size of a grapefruit, shaped like a novelty item. Nobody hands you a Magic 8 Ball in the hospital, unless it's a joke. The part that isn't honest, or more accurately the part that isn't understood, is what happens when you ask it more than once. If you ask a Magic 8 Ball a question and it says my sources say no, and you don't like that, you ask again. You have not gathered a second data point. You have resampled the exact same random distribution. The die does not remember it already told you no. It has no state. Every roll is an independent draw from the same 20 options at the same odds, no matter what came before it.

It's not a metaphor for how a die works. That is mechanically, literally, what's happening inside the ball or inside the array. Which means asking five times and getting yes three times isn't a trend. It isn't the balance tipping in your favor. With 10 affirmative answers out of 20 on a die, you're more likely than not to land on a majority yes across five rolls by pure structural accident, before we even got into the part where you're the kind of person who stops scrolling the second she hears a yes she likes.

Engineers do this, too. I'm not writing this from a place of superiority, because I do this. You got a flaky test in your CI pipeline, fails sometimes, passes sometimes, nobody has the sprint capacity to actually fix the root cause. So, what do you do? You hit rerun. Fails again. Rerun again. Third time green, you ship it. That test did not become more true on the third run. Nothing about the underlying code changed between run one and run three. You resampled a noisy process until it produced the answer you wanted, and then used the answer you wanted as evidence that everything was fine. That's the Magic 8 Ball. We just built it in our deployment pipelines and gave it a green check mark instead of a blue plastic window.

So, here's what happened after the drink spill exoneration text. I told her, mostly joking, "I'm going to build you a Magic 8 Ball that doesn't lie to you." And she said, "Great, please. Mine has an ad for a mobile game and it's ruining the vibe." Hold on to that for a second.

So, I spent at most 30 minutes. It's not sophisticated software to write a small web page. Actually, most of the time was because I was plugging it into my existing website. So, it was a text box, a button, an array of 20 responses that could be changed if you wanted to, but I wanted to be faithful to the original, so it's the original 20 responses. It's a random index. There's no ads, because not trying to fund anything. I just want her to stop getting interrupted by a farming simulator every time she asks the universe a question.

And then I did the part I actually cared about. I sat down next to her, pulled up the code, walked her through it out loud on purpose. She had to sit through Dr. Skiles's little lecture here. And I said, "Here's an array. Here are all the 20 things it can possibly say. Here's the random function. It just picks a number between 0 and 19. That's the whole program. It doesn't know your question. It isn't reading your question. Look." I said, "I'll prove it." And I deleted the text box and I ran it and it still worked because it was never using the input for anything in the first place.

And she watched the whole demonstration, nodded at the right parts, said, "That's so cool." in a tone at that time I read as engagement with the material. And she's used it a lot since. I know because she texts me results. Roughly a third of those texts are some version of asked three times, got yes twice, so. So, just so. Like the sentence completes itself. Like I hadn't sat next to her and shown her the exact 20-item array that yes twice was coming from. What she actually took away, the code walk-through, the deleted text box demonstration, was no ads. That was the whole lesson. She loves it because there's no ads.

And I want to be precise here. She made zero updates to how she uses the tool. She only updated her opinion of the tool's user experience. I built her a completely transparent system, opened the box all the way, showed her the wiring, and she used transparency to conclude the box was nicer to look at.

So, here's where I have to be honest about my own research because this is the exact question I study and my friend's living room just ran an experiment on it that I did not design and probably wouldn't have gotten past the ethics board. My research thesis compressed is that AI systems should be legible to the people using them, not just functional, legible. You should be able to see what's happening well enough to calibrate how much to trust it. Transparency as infrastructure, not as a nice to have, not a compliance checkbox. The assumption underneath that thesis, the one I hadn't examined closely enough until my friend's living room, is that legibility produces calibration. That if you show someone the mechanism, they will update their behavior to match what the mechanism can actually do. That assumption is doing a lot of unearned work.

I showed her an array. She saw an array. She understood an array in a sense that she could repeat it back to me correctly. It's just picking a random one. It doesn't know anything. She could have passed a quiz on how the tool worked. And that same week she texts me asked four times, three said yes. Seeing the mechanism and updating your relationship to the mechanism are two different events and the first one does not automatically cause the second one. A lot of transparency work, mine included, quietly assumes it does. Show the source code, ship the model card, publish the system prompt, and the assumption is that an informed user becomes a calibrated user.

Now, watch this pattern show up somewhere with actual stakes. You know people who regenerate AI responses. Everybody listening has done it. You ask a language model something, a real question, sometimes a real decision, and you don't love the first answer. So, you hit the little regenerate. You get a second answer. Maybe it's close to the first, too. Maybe it agrees. And sometimes in your head you go, "Okay, two out of two, that's probably right."

So, here's the part almost everyone listening already knows intellectually, the same way my friend could recite it's just picking a random one. A language model's output is a sample from a probability distribution. Temperature, sampling, whatever term you want to reach for, the mechanism is closer to my friend's array than most of us are comfortable admitting. A much, much bigger array, informed by an unfathomable amount of training data, genuinely more sophisticated in what it's drawing from. But, regenerating and getting a similar answer twice is not independent confirmation of a fact. It's asking the same distribution to sample itself again.

You already knew that. I already knew that. We hit regenerate anyway, and when the two answers rhyme, something in us relaxes, like we just got a second opinion from a different doctor. We did not get a second opinion. We got the same doctor saying it again because we asked.

And back to that CI pipeline, since I promised I would come back to it, the engineer rerunning a flaky test until it got green knows, if you talked to them and asked directly that the test passing on attempt three doesn't mean the code is correct, they'll tell you that. They probably feel a little defensive about the question, and they'll still ship it on green. Knowing the mechanism and acting on the mechanism are different muscles. My friend has the first one. I have the first one most days. Neither of us reliably have the second one.

So, if showing someone the mechanism doesn't reliably change their behavior, the natural next question is why not? And I think the honest answer is that I was solving the wrong problem. Because I assumed she was using the 8 ball for information. She wasn't. Think back to the actual night. She'd already decided to text him. You can hear it in the message. She wasn't asking the 8 ball whether to text him. She was asking it to agree with her. If it had said no the first roll, and I think she'd tell you this herself, she was not going to sit with the no. She was going to roll it again. Rolling wasn't a search for the true answer. It was a search for the roll that let her stop deciding by herself. That's not a math error. You cannot fix it with a better array. You cannot fix it with showing someone the random function because the random function was never what she was actually consulting. She was consulting the fact that it wasn't her. If it goes badly, it's not I decided to text him and it went badly. It's, well, the 8 ball said yes. The tool isn't providing information. It's absorbing the accountability she didn't want to hold by herself for a decision she'd already made.

Once I saw it that way, a lot of this stopped being annoying and started being a little touching. Honestly, she wasn't being irrational. She was doing something extremely human. Wanting company for a decision she felt too big to make alone on a random Tuesday. The regenerate button does the same job in a slightly different outfit. If the first answer a model gives you says something you didn't want to hear about your code, about your relationship, about your five-year plan, regenerating isn't always a search for more information. Sometimes it's a search for permission. And unlike your actual friends, an AI system never gets tired of rolling again for you. There's no limit to how many times it will restate a version of the answer until one of the versions happens to be that vibe you're looking for. Nobody designed it to be a permission machine. But it functions like one for exactly the questions where you'd most benefit from a friend saying, "No. I already answered that. The answer is still no."

So, where does that leave the research thesis? I still believe systems should be legible. I'm not walking back on that. An opaque system takes the choice away from you entirely. You can't even ask the right question of it because you don't know what it is. That's still worse. Legibility is necessary. I just don't think it's sufficient anymore. And I didn't fully believe that until I watched someone I love pass a pop quiz on exactly how a system worked and then use that system exactly the same way. Transparency tells you what the box can and can't do. It doesn't touch what you're actually asking the box for. And if what you're asking for is permission, not information, you can hand someone the entire source code and they will thank you for the interface and go right on rolling until it agrees with them. Douglas Adams already told this joke by the way. Deep Thought spent seven and a half million years computing the answer to life, the universe, and everything and hands back 42. And it's completely correct and completely useless because nobody actually agreed on the question first. That's not a joke about computers being dumb. It's a joke about us skipping the one step that mattered. We'll spend years on the answer machine and about four seconds on figuring out what we were actually asking it. I don't think the fix is to stop being transparent. I think the fix is to actually ask before you build the see-through version of anything. What the person in front of you is really asking the black box to do for them. Sometimes it's tell me something true. Sometimes it's agree with me so I don't have to carry this by myself. Those need completely different responses and only one of them gets fixed by better documentation. She still texts me the results by the way. No ads. She's very happy about that part. That's this episode of Chaotic Commits. I'm Joanne Skiles, someone who's definitely rerolled a decision in my head until it landed on an answer I already wanted. No 8 ball required. If that sounded familiar, do me a favor and hit follow or subscribe wherever you're listening. That's how the next one finds you. New episodes wherever you get podcasts. No highlight reel, just the commit history.