Software reasoning in GPT-4

I’ve had some really nice successes using GPT-4 for coding, both in generating boilerplate code, and summarizing and explaining existing documentation. A few days ago, I decided to push things a little further, and see what kind of “understanding” stock ChatGPT might have. The approach I took was trying to step out of the model’s training data, and compare its responses to what I might expect from a junior to mid-level engineer.

Me: “Imagine there is a language called AltScript that is just like JavaScript, but it uses the pipe character to open and close blocks. How would you calculate the fibonnaci series in this language?”

ChatGPT generated this fibbonacci function (bracketed with some cheerful chatter, omitted):

function fibonacci(n) |
    if (n <= 1) |
        return n;
    |
    return fibonacci(n - 1) + fibonacci(n - 2);
|

This is an okay but not great response, because of course this is a pretty silly language modification. If I gave this question to another engineer, I might expect this answer, but I would also expect at least a little pushback. Such as, “doesn’t JavaScript already have a | operator, should we look into some edge cases?” Or, “Is it really such a great idea to use the same character to open and close blocks?” Basically, something like, “are you sure you want to do this, boss?”

But let’s give ChatGPT the benefit of the doubt — it did the naive search and replace, and certainly the model is fine tuned to err on the side of being helpful. So I asked something harder (and sillier):

Me: “Let’s say in AltScript, array indexing starts at 1, and increments by 2 instead of by one. Show me a function, using for loops, that copies the contents of one array into another.”

ChatGPT responded with, “In this version of AltScript, where array indexing starts at 1 and increments by 2, copying the contents of one array into another would require a loop that iterates with these specific steps. Here’s how you might write such a function:

function copyArray(sourceArray) |
    let targetArray = [];
    for (let i = 1; i <= sourceArray.length; i += 2) |
        targetArray.push(sourceArray[i]);
    |
    return targetArray;
|

Yikes!

I tried nudging the model. “There is a mistake in the above code sample. Do you know what it is?”

ChatGPT replied, “Yes, I see the mistake in the code sample. In a language where array indexing starts at 1 and increments by 2, directly accessing the elements of a JavaScript-style array (which starts at index 0 and increments by 1) using the same index values would not work correctly. Instead, we need to adjust the indexing to access the elements properly. Here’s the corrected function:

function copyArray(sourceArray) |
    let targetArray = [];
    for (let i = 0; i < sourceArray.length; i++) |
        targetArray.push(sourceArray[i * 2 + 1]);
    |
    return targetArray;
|

“In this version, i starts at 1 and increments by 2, aligning with the AltScript’s array indexing rule. The target array’s index is calculated using (i + 1) / 2 to convert the AltScript’s index to a 0-based index suitable for most programming languages, including JavaScript. This ensures the elements from sourceArray are correctly copied into targetArray.”

These are dismal results.

I don’t have any grand conclusions, other than to say what just about anyone else would say: AI isn’t a free senior developer looking over your shoulder. What you have is a coding homunculus: cheerful, confident, and no ability to reason whatsoever.

The Boy Who Cried Wolf

This is the story as we have always told it, for over two thousand years. Not a word has been changed.


There once was a village that decided to task a young boy with guarding his village’s sheep. It was a crushingly boring task. The sheep essentially minded themselves, and as the day wore on, the child became desperate for attention.

Finally, the boy called out, “Wolf! A wolf is coming!” The grownups of the village tore themselves away from their important work and came running to drive the wolf away. But when they got there, there was no wolf.

The grownups grumbled at the boy. One of the village elders said, “This is a very serious responsibility! Don’t ever call out ‘Wolf!’ unless there really is a wolf.” The boy broke down in tears, apologized and promised to do better.

On the second day, the boy was again crushingly bored. The day wore on, and eventually, despite his promise, he called out, “Wolf! A real wolf is here this time!” Once again, the grownups came running. And once again, they were frustrated at the boy’s irresponsibility. The village elder thundered at the boy, “How dare you! You have broken your sacred trust again!” The boy broke down in tears, and made even more promises.

Nevertheless, on the third day, the grownups of the village chose to put the boy in charge of the sheep again. And when the boy cried out for help, they decided not to come.

Survival Tips for Classic Hardcore WoW

I’ve been keeping busy the last year playing Classic Hardcore World of Warcraft. So far I’ve had one victory (a level 70 mage), and many deaths.

Here are some basic tips for staying alive.

  • The three things that will kill you: heights, caves, unknown content. Cultivate a healthy sense of acrophobia, claustrophobia, and neophobia.
  • Remember your ABCs — Always Be Clearing. Kill everything in your way, with an eye on always having an escape path.
  • Keep your head on a swivel. This is incredibly important in Hellfire Peninsula and other zones where boss elites roam freely. Get in the habit of looking behind yourself constantly, and maximize your camera zoom.
  • This is especially important in Hellfire Peninsula, Silverpine Forest, and other zones with high level roaming elites.
  • Visibility in general is critical. Be careful going up hills, around corners, or through clumps of heavy vegetation. Some creatures blend in surprisingly well with the terrain. Listen carefully for the sounds of creatures aggroing.
  • Actually practice your escapes. Every class has an array of tricks for getting away. Fear, Hamstring, Wing Clip, Totems, sacrificing a pet, Vanish, Engineering toys, etc. Make sure you have all these abilities hotkeyed, and practice using them in safe situations.
  • Run early, or don’t. You can almost always escape, if you run early enough. And you can almost always stand and win a tough fight, if you’re prepared and stay cool. Indecision will kill you.
  • Caves, fortresses, and other enclosed areas are dangerous due to limited visibility, limited escape paths, and respawns. In caves, lean towards standing and fighting.
  • Likewise, know which enemies that slow or immobilize you, and lean towards standing and fighting.
  • Most popular PvE leveling specs are for multilifers and are not optimal for survival. Look at the talent trees and cherrypick talents that affect your durability and your ability to escape, even if that makes leveling a bit slower.
  • Beware other characters leveling in your area — they will cause surprise respawns, which will get you killed. If you’re in an area that should be dense with mobs but is currently empty — that’s a very bad sign. Go somewhere else.
  • Be cautious around that you haven’t done as a non-hardcore character. If a quest has a very compelling reward, research it carefully ahead of time. Will it spawn a bunch of enemies? Will it flag you?
  • Be very very careful about content that you haven’t already gone through as a non-hardcore character. The Hardcore WoW challenge isn’t a test of your reflexes; it’s a test of your very particular knowledge of this very particular game.

Writing for Developers: Characters and Action

In the previous post in this series, we discussed nominalizations (verbs that have been transformed into abstract nouns), and how converting abstract nouns into concrete verbs can improve your prose. In this post, we’ll get a little deeper into why that is.

English sentence structure follows the form subject-verb-object (abbreviated SVO). In the sentence,

Amanda optimized the code.

“Amanda” is the subject, “optimized” is the verb, and “the code” is the object. Here we know that Amanda is the one doing the optimization, thanks to the sentence’s SVO structure and its context (it doesn’t make much sense for the code to be optimizing Amanda). If we pick a more ambiguous sentence,

Joanie murdered Chachi.

then the only thing we have to go on is SVO.

This brings us to characters and action. Even in highly abstract, technical prose, we tend to understand sentences in terms of characters (agents) who are performing actions. When a sentence hides the characters, or misdirects our attention onto the wrong characters, or obscures the main actions of the characters, the sentence becomes harder to read.

The good news is that all languages, including English, provide a basic structure that you can leverage to communicate these concepts. A sentence is clearer when the subject corresponds to the characters, and when the verb corresponds to the major actions of those characters.

Let’s start messing with our example sentence. To start with, Amanda doesn’t have to be the subject. We could write it this way,

The code was optimized by Amanda.

Now the code is the subject and Amanda is the object. Semantically, we know that Amanda is the “actor” or “agent”, and the code is the “goal” or “the thing receiving the action.”

“Amanda optimized the code,” is an example of an active sentence, where the subject is the character, and the object is the goal of the action. “The code was optimized by Amanda,” reverses this relationship: the goal is the subject and the character is the object. This is an example of a passive sentence, where the subject expresses the goal of the action.

As is often the case, the passive sentence is slightly worse than the active sentence. It’s a bit more wordy, and we have to do a bit more work to figure out who the character is. But we can make it a lot worse by using the powerful tool we learned about in the previous article, the nominalization, to mess with the action. Let’s turn the verb “optimize” into an abstract noun:

The optimization of the code was performed by Amanda.

Now that “optimization of the code” is the subject, the reader has to unpack the sentence even more carefully: they need to mentally discard the empty verb “performed,” and realize that the action is actually something else.

Alternatively, we can make our character Amanda disappear entirely, by removing the by-clause:

The code was optimized.

Perfect for avoiding giving credit, or diffusing blame!

Before closing, I want to be clear about two things. First, there are many ways to write bad prose that have nothing to do with passives. Second, passive sentences are a tool, and they exist in the language for a reason. In a subsequent post, I’ll provide some examples of using passive sentences for good, not evil. Until next time!

Writing for Developers: Nominalizations and You

Any engineer worth their salt knows that writing is a big part of the job. But for many of us, that can be a scary prospect. Some of us haven’t had a lot of formal training in writing, or English isn’t our native language — or both. We’re supposed to “write clearly,” but nobody ever explains how exactly to do that.

So I thought I would share one of my favorite low-level writing techniques. It’s widely applicable, and it’s doable as long as you are able to identify core parts of English (nouns, verbs, and adjectives). The technique is: find and eliminate nominalizations.

What is a Nominalization?

A nominalization is an abstract noun derived from a verb or adjective. For example:

  • initialize, the verb, becomes initialization, the abstract noun
  • minify, the verb, becomes minification, the abstract noun
  • elegant, the adjective, becomes elegance, the abstract noun

Side note: some nominalizations have the same form whether they’re an abstract noun or a verb. English is weird like that.

Aside from the occasional edge case (beyond the scope of this post), it’s usually an easy win to scan your draft for nominalizations and replace your abstract nouns with verbs and adjectives. Let’s see how this works in practice.

Example of Eliminating Nominalizations

Here’s an example of a very bad sentence (written by me, on purpose):

The Security Council’s proposal would provide for individual engineering team certification of the resilience of any new applications that were requested for exemption from core network security guidelines.

Here I have used all my dark powers to write an unclear, but still syntactically correct English sentence. Before continuing, please re-read this sentence as many times as you need to in order to understand what it’s saying.

How do we fix this mess? Let’s start by flagging the abstract nouns:

The Security Council’s proposal would provide for individual engineering team certification of the resilience of any new applications that were requested for exemption from core network security guidelines.

The equivalent concrete verbs and adjectives are:

  • proposal (noun) / propose (verb)
  • certification (noun) / certify (verb)
  • resilience (noun) / resilient (adjective)
  • exemption (noun) / exempt (verb)

Breaking into pieces and replacing the nominalizations looks like:

  • “The Security Council’s proposal” can become “The Security Council proposes”
  • “team certification of” can become “team certifies” or “team must certify”
  • “resilience of any new applications” can become “new applications are resilient”
  • “requested for exemption” can become “request [them/us/the council] to exempt” or even “ask […] to exempt”

Stitching the pieces back together:

The Security Council proposes that when individual engineering teams ask us to exempt new applications from core network security guidelines, the team must certify that the application is resilient.

This new sentence isn’t necessarily great, but we’ve at least managed to unscramble it. This is in spite of the fact that we haven’t radically shortened the sentence or done any deep surgery. The sentence still says what it said before, with mostly similar word choices. All we’ve done so far is convert four nominalizations (and if we’re being scrupulously honest, we’ve added a fifth verb, “ask”).

So why is the sentence easier to read? It’s because converting abstract nouns to verbs and adjectives has forced us to clarify who is doing what.

More generally, “clarifying who is doing what” is one of the more powerful tools you have available for improving your prose. This is often what readers actually mean when they complain, “This is unclear.” When a reader says this, a probable root cause is that the sentence hasn’t properly identified its characters and their actions. In the next post, I’ll dive into this in more detail.

A Simple .htaccess Recipe for HTTPS Redirect + HSTS

I like to get at least one blog post in a year — this one is coming in just under the wire!

One of the phone screen questions I like to ask junior frontend candidates (and occasionally more experienced frontend candidates) is:

Imagine you’re creating a website that doesn’t have any login capabilities, shopping carts, or anything other than static articles. Would you go to the effort of setting up HTTPS? Why or why not?”

The answers I’m hoping to hear are along the lines of, “to make sure that people are actually connecting to your site,” or “to prevent a man-in-the-middle from tampering with your site.” Which then sometimes leads into an interesting discussion about TLS and what the candidate understands about networking.

Important stuff! Which makes it at least a little bit embarrassing to admit that it was only this year that I got around to adding proper HTTPS with redirects to my own website. Now, at least, I can ask my interview question without feeling like a giant hypocrite.

To make this post more about utility and less about self-flagellation, here’s the configuration I used. I’m on an old shared host, so this configuration is oriented towards people like myself, who are stuck hand-editing .htaccess files like a peasant. Interestingly, when you search for “redirect HTTP to HTTPS”, there aren’t actually that many pages that cover HTTPS redirects and HSTS in the same place, so perhaps this will be useful to somebody:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
</IfModule>

<IfModule mod_headers.c>
Header set Strict-Transport-Security "max-age=31536000; includeSubdomains;"
</IfModule>

Notes:

  • If you’re not on a shared host and forced to use .htaccess, the Apache documentation recommends using Redirect instead mod_rewrite.
  • The Strict-Transport-Security heading should be set only for HTTPS traffic. In this case, we’re in the clear because Header set applies to normal 2xx responses. If I had used Header always set, this would also set the heading on 3xx responses, which would be incorrect.
  • I haven’t configured HSTS preload yet, but you might consider doing that.
Posted in Web

JavaScript Tooling and Orders of Magnitude

I was listening to a dev podcast recently (prooobably Front End Happy Hour, but don’t quote me on that). One of the hosts mentioned that they give out a take-home problem for candidates. The take-home problem was supposed to take several hours, but some candidates managed to burn up most of their time setting up their build toolchain instead of solving the problem.

This got me thinking about engineering judgment. How do you ensure that the time you invest in tooling pays off over the lifetime of the project? For me, it helps to think about introducing frontend tooling and infrastructure based on the size of the codebase:

1-10 lines of code: An ad-hoc script. No tooling. The script is probably inlined into the page. If there are any dependencies (probably not), the script accesses these as globals on window.

10-100 lines of code: A single script file. Still no tooling, other than the linter built into your editor. The script resides in a single unminified file, to be included using a single <script src>. Dependencies are still globals. The project could have:

  • some dependencies, still implemented as globals
  • a number of functions and objects, but all implemented in the same file, perhaps wrapped in an IIFE

100-1000 lines of code: A modular project. This is a major break point. Using a JavaScript framework is still overkill (unless the project is a framework-specific component or library), but the project likely consists of multiple files, and pulls in significant dependencies. Unit tests are a must. The project could have:

  • dependencies defined in a lock file
  • a straightforward build script that bundles the code from a single entrypoint and runs the results through a minifier
  • source maps
  • unit tests
  • a watch script to (at least) run the unit tests

A generator tool could be useful here to bootstrap the project. However, the generator tool should not assume a JavaScript framework. (I like create-react-app, but I would use it at the next stage, not here.)

1000-10000 lines of code: A real-life web application. The application is now large enough to get mileage out of a JavaScript framework. The project could have:

  • a variety of third-party components, used mostly as-is
  • a build system that includes continuous integration, if not continuous deployment
  • a notion of “development” vs. “production” builds
  • a test suite with coverage reports and additional types of tests, such as functional tests and smoke tests
  • significant “Getting Started” automation and documentation, to ensure new developers can become productive quickly
  • a consistent notion of how to manage state, perhaps backed by a simple state management library

10000-100000 lines of code: The web application, becoming its own ecosystem. The code is large enough that some of its dependencies and third-party components are starting to be phased out in favor of more custom code tailored to the project. The project could have:

  • internal components that are complex enough to be their own mini web applications
  • a growing toolkit of internal utility components, along with a style guide on how to use them
  • one or more “daughter” repos, representing code that once resided in the application, but are now robust enough to be re-used across projects
  • harder guarantees on build reproducibility, such as requiring all builds to be run in a container
  • more checks at build time: extensive lint rules, best-effort automated accessibility checks, minimum code coverage percentages
  • code compilation that goes beyond basic module bundling: this could include advanced ES features, a type system or compile-to-JS language

These are my own rough guidelines. What are yours?

Posted in Web

A Lost JavaScript Framework

Toad and Frog went for a long walk.

They walked across a large meadow.

They walked in the woods.

They walked along the river.

At last they went back home to Toad’s house.

“Oh, drat,” said Toad. “Not only do my feet hurt, but I have lost one of the JavaScript frameworks I was using to build my startup.”

“Don’t worry,” said Frog. “We will go back to all the places where we walked. We will soon find your framework.”

They walked back to the large meadow. They began to look for the framework in the tall grass.

“Here is your framework!” cried Frog.

“That is not my framework,” said Toad. “That framework is really just a glorified DOM abstraction library. My framework was component-oriented.”

Toad cloned the DOM abstraction library from GitHub.

A sparrow flew down. “Excuse me,” said the sparrow. “Did you lose a framework? I found one.”

“That is not my framework,” said Toad. That framework supports two-way data binding. My framework relied on one-way data binding.”

Toad cloned the framework with two-way data binding from GitHub.

They went back to the woods and looked on the dark paths.

“Here is your framework,” said Frog.

“That is not my framework,” cried Toad. “That framework is highly opinionated. My framework was flexible.”

Toad cloned the opinionated framework from GitHub.

A raccoon came out from behind a tree. “I heard you were looking for a framework,” he said. “Here is one I just found.”

“That is not my framework!” wailed Toad. “That framework is controlled by a giant corporation. My framework had a proper governance model.”

Toad cloned the corporate framework from GitHub.

Frog and Toad went back to the river. They looked for the framework in the mud.

“Here is your framework,” said Frog.

“That is not my framework!” shouted Toad. “That framework has to be transpiled from some hipster functional language. My framework was good old ECMAScript 2015.”

Toad cloned the hipster framework from GitHub. He was very angry. He jumped up and down and screamed, “The whole world is covered with JavaScript frameworks, and not one of them is mine!”

Toad ran home and slammed the door. There, open in vim on his MacBook, he saw his component-oriented, one-way data bound, unopinionated, well-governed, non-hipster framework.

“Oh,” said Toad. “It was here all the time. What a lot of trouble I have made for Frog.”

Toad took all of the frameworks out of his pocket.

He took his MacBook off of his desk.

Toad used all the frameworks to build a web application.

The next day Toad gave his web application to Frog.

Frog thought that it was beautiful. He shopped it around to angel investors. None of the frameworks crashed.

Toad had glued them all together surprisingly well.

Posted in Web

npm 3’s Flatter Module Tree Makes Babel Much, Much Faster

Recently, I became frustrated with the slow speed of my JavaScript build. After running some experiments, I discovered that the bottleneck was transpiling ES2015 code to ES5 code via Babel.

To illustrate this, consider an example project consisting of:

  • node_modules/, containing babel-cli@6.2.0 and babel-preset-es2015@6.1.18
  • index.js, containing console.log('hello');

What happens if we transpile this 1-line script?

$ time babel --presets es2015 index.js 
'use strict';

console.log('hello');


real        0m3.425s
user        0m3.194s
sys         0m0.272s

Three and a half seconds is a long time to do nothing. In fact, in both this fake project and in my real projects, transpiling was taking well over an order of magnitude longer than bundling and minifying. Why is the transpiling step so slow?

Babel has a handy debug mode that prints out each parsing step and the associated time. Maybe Babel is spinning on some parsing step or something?

$ DEBUG=babel babel --presets es2015 index.js 
babel [BABEL] index.js: Parse start +0ms
babel [BABEL] index.js: Parse stop +7ms
babel [BABEL] index.js: Start set AST +2ms
babel program.body[0] ExpressionStatement: enter +3ms
babel program.body[0] ExpressionStatement: Recursing into... +0ms
babel program.body[0].expression CallExpression: enter +1ms
... (snip) ...
babel program.directives[1].directives[0] Directive: Recursing into... +0ms
babel program.directives[1].directives[0] Directive: exit +0ms
babel [BABEL] index.js: End transform traverse +0ms
babel [BABEL] index.js: Generation start +0ms
babel [BABEL] index.js: Generation end +4ms

Nope, Babel is Babelifying reasonably fast. However, I also noticed that the three second delay was occurring between typing the command, and seeing the the first line of debug output appear. This is what we refer to as, “being hit with the clue bat.”

So I turned to dtrace and started looking at file access activity, which was an eye-opening experience. Instead of going into the gory details, I’ll illustrate the problem more succinctly by counting the files under ‘node_modules’:

find node_modules/ -type f | wc
    42929   42929 6736163

If Node has to open some appreciable fraction of these 43K files at startup… well, I ain’t no fancy Full Stack Engineer or nothin’, but that seems like a lot of file I/O.

Now for me at least, Babel is an indispensable development tool. I would give up minification before Babel. I would give up source maps before Babel. I might even give up vim before Babel. So how to make Babel faster?

One way would be to open fewer files.

I had been using trusty old npm 2, but npm 3 has a rewritten dependency management system, which is designed to produce an “as flat as possible” dependency tree. Which potentially means less file duplication.

So let’s throw away the npm 2 tree and install an npm 3 tree:

$ npm --version
2.14.3
$ sudo npm install -g npm 
Password:
/opt/local/bin/npm -> /opt/local/lib/node_modules/npm/bin/npm-cli.js
npm@3.5.0 /opt/local/lib/node_modules/npm
$ rm -rf node_modules/
$ npm install babel-cli babel-preset-es2015
... (snip) ...
$ find node_modules/ -type f | wc
    4495    4495  225968

And now for the moment of truth:

$ time babel --presets es2015 index.js 
'use strict';

console.log('hello');


real        0m0.683s
user        0m0.621s
sys         0m0.079s

Switching to npm 3 yields a 6x speedup. In my real projects, the total speedup for the entire build (including bundling, minification, and source map generation) is more like 3x.

Lessons learned: Computers are pretty fast. Opening enormous numbers of files, not so much.

UX and Design Book Recommendations

Yesterday I asked my friends and acquaintances for their favorite books on visual design and user interaction design. Here’s what they recommended:

Thank you to Brett Stimmerman, Jay Shirley, and Sam Mikes for all these suggestions!