Sunday, May 14, 2023 :: Tagged under: engineering essay. ⏰ 14 minutes.

🎵 The song for this post is CHEMICAL LOVE, by “Kaleb James and Chey” for the game Bust-a-Groove. 🎵

I’m talking with friends and coworkers about programming languages (surprise),
and I’m landing on a rough shape that these conversations take. I’ll share it
here and hope you find it useful, especially if we ever talk about them. Then
I’ll use that framework to make the case that the way we talk about them in
company settings strikes me as fear-based and bogus (🔥s ahead).

When you compare languages, what are you actually comparing?

While visiting a friend, he noticed I had a Golang shirt, he told me he loved
Golang, I told him I like the shirt but the language less. He got excited at the
idea of “getting into it” later, and after dinner we plopped down on couches and
said “alright! Let’s get into it!” I wanted to talk about its garbage collector
(1, 2), how goroutines/channels are a delightful abstraction but I
prefer BEAM’s abstractions
because it allows for
Supervisors, Golang’s very loose approach to correctness… and he didn’t
want to talk about any of that. He, on the other hand, emphatically talked about
how much he loved that “the Go developers knew that all you need is a for
loop. Someone brought Scala into my company and I hate the mental shift.”

This was not a fruitful conversation, I think we both felt like we weren’t
valuing what the other cared about. When people talk about languages they like
or dislike, I group the things people talk about into three broad categories,
which I’ll call soil, surface, and atmosphere:

Soil is the properties of running code in that language. Most of it is
when the code is actually running (basic performance characteristics, “is it
a binary or interpreted by a VM,” scheduler and/or relationship to
multicore/parallelism, garbage collector) but a more broad generalization is
“everything that isn’t code editing that doesn’t directly involve “community,””
so I’ll involve things like build times and some properties of its tooling. So
the stuff I brought up in the Golang discussion are “soil,” but so is:

Surface is what people usually think about when comparing languages: the
features! Source code! It’s whether it has your favorite looping construct (or
doesn’t, per the “only need a for loop”). Syntax, FFI, which regexes it
supports, the semantics of its specific object system. Other examples are:

  • Less/Sass and CoffeeScript were pure “surface” plays: not about the core
    capabilities, but seemingly expanding them with pure “fixes” to suboptimal
    surfaces.

  • “Ruby is so much better than Java! Look at how you open a file in Java (shows
    you 30 lines) vs. Ruby! (2 lines)”

  • Much of the appeal of MERN or MEAN stacks is that “it’s all
    JavaScript/JSON,” meaning you can reduce the amount of “surface” to learn.

  • Strong reactions on both sides of CSS-in-JS and Tailwind feel “surface”-ey,
    though they can have “soil”-ey impacts.

Finally atmosphere: these are things that aren’t the language or its code,
but the broader community. Hiring, Stack Overflow answers, number of stars on
its popular GitHub projects, number of packages in their repositories.
I’ll go a little further and include downstream effects of that community, e.g.
“does the language have a VSCode plugin and language server” seems to be more a
function of atmosphere than the others:

  • Most “you can hire React developers easily” is completely divorced from any
    technical discussion of React as a framework. Ditto Java, Python, Ruby, JS…

  • Most of a language’s culture: Elm and Clojure both have something like “if
    Evan/Rich didn’t think you should have that thing you wanted, have you
    considered that maybe you’re wrong? 🙂”. Golang telling you you’re dumb and
    wrong for wanting generics for a decade before shipping them. Scala having all
    that silly drama.

A light note for exceptions: as Hillel says, “all taxonomies are broken,
full stop.”
Here, “tooling” fits all three: usually, the existence of
tooling (like the language server) is a community thing, but the properties of
that tooling is very “soil,” based on decisions the core developers made and
with big downstream effects on the working with the code day-to-day (e.g.
compile times) or even after it deploys (e.g. “can you do postmortem
debugging?” — kind of a runtime thing too).

The other is “static types”… I feel like this is surface, but you can argue it’s
soil.

Anyway, with this in mind, here are some observations!

A collection of 18 logos for various tech projects; the ones I recognize are Remix, Fly, SQLite, Prisma, Docker, Tailwind, TypeScript, GitHub.

These 18 logos are the “starter project” components of “It’s time to Just Ship” of the “Epic Stack.” But sure, keep saying it’s more simple to “just use TypeScript.”

Discussions get less concrete as you go upwards

I feel like when discussing language properties, “soil” is often pretty concrete
(e.g. general performance, memory safety of Rust vs. C++ programs), “surface” is
more feelings-ey but still grounded somewhere in reality (BEAM languages can
have supervisors because of OTP and immutability, Go programs can’t; but what
this means for the safety of your program and ultimate business impacts are hard
to measure). Whereas “atmosphere” questions are extremely based in feelings:
“you can’t hire FP developers!” (have you actually tried? almost every FP shop I
know has never had trouble finding enthusiasts, and per the Python
paradox
, they’re often quite good…) “There isn’t library support!” (most
repos have an awesome repo collection, what specifically do you need?).

I’m not saying atmosphere questions don’t matter, just that I think they’re the
hardest to qualify the impacts on. You can quantify, for example, that npm has
n packages and Cargo has m, but what do those numbers actualy mean for the
purposes of a long-lived, business-effective codebase? You can estimate from
sources like HackerRank or Leetcode or Greenhouse or GitHub Jobs how many Java
developers there are vs. Scala ones, but how does that number translate to
finding exceptional people for your specific company? Are you hiring… 30,000
people this year? Unless it’s a raw “bodies” problem (and most companies, the
harder part is finding the right bodies) how is that super relevant?

This frustrates me because I feel like…

The current thinking is “atmosphere at all costs!”

My computing career came of age in the late aughts. It was a very different
time: people played with technology choices a lot more! Twitter was suffering
with Ruby on Rails, so they tried a new language with promise: Scala. Heroku
invested in Erlang, and so did a little messaging app called WhatsApp. In 2006,
ITA Software received $100m in funding even though they were 10 years old
and written in Common Lisp (they sold to Google in 2010 for $700m in cash. This
was impressive back then; Instagram didn’t have its industry-breaking $1bn
acquisition until 2012).

Today, we’re afraid to do anything that’s not JavaScript, Java, Python, or Ruby.
I have a feeling 10 years from now it’ll just be JS. We all read that excellent
McKinley article about “innovation tokens” and we decided to be
floor-avoiders instead of ceiling-breakers. Engineering leaders lead from a
place of fear and risk-aversion instead of optimism and believing in their team.
Where have all the c̶o̶w̶b̶o̶y̶s̶ hackers gone?

Yes, using an exotic technology may spend an innovation token, but like a lot of
things in life, innovation tokens are completely made up, it’s like talking
about the finite number of “love tokens” you can give your spouse in a given
year. I’d like to offer a different framing: maybe the biggest way to reduce
your innovation tokens is to have a small imagination. Imagine receiving a bag
of tokens after you pick your a tech stack. Picking a great, appropriate
technology that’s not one of the Big Four, you have to reach into a bag to spend
a token, but then you see your bag has 6 left; when you pick a Boring one, you
don’t spend any immediately, but there are only 3 in the bag.

If your browser doesn’t support HTML5 video, here’s a link to the video instead.

Seen here: happy engineers feasting on innovation tokens they’ve been gifted by a CTO who trusted them. From the dinner scene in Hook.

Is it hard to learn a language?…

People often talk about great engineering teams (and the teams they’d
like to build and work on) in pretty high-minded ways. Raise if your hand if you
feel this way:

  • I prefer to work with an excellent software engineer, who doesn’t tie their
    identity to a specific language or technology (e.g. would prefer a great
    hacker than someone who identifies as a “Ruby developer” or “JS developer.”)

  • My preferred teammates are people who understand the concepts underneath the
    technologies, and don’t equate the technology to the entire stack (e.g.
    “databases” to this person doesn’t just mean “Oracle” or “Mongo”, someone who
    understands BOM and DOM as distinct from “Svelte”).

  • The kind of person I want on a team has had exposure to multiple technologies,
    can articulate tradeoffs between them, and is capable of understanding and
    working with new ones.

Like with equal housing rights, in theory, almost everyone polled agrees on
a set of neutrally-stated positive ideals. And also like equal housing rights,
somehow, when it comes time to practice or vote on those ideals, the outcomes
don’t match. It turns out, many engineering teams are perfectly fine hiring
“Java developers” who sweat if you showed them some Python, people who can’t
effectively articulate tradeoffs between various tech stacks, and people who, if
asked to spend a week getting up to speed on something new, will instead
complain and argue for using something that will be suboptimal for years and
years to follow.

Here’s an interesting question: as with housing rights, how come people feel one
way in their head but forget those ideals when actually voting with their
actions? I touch on this more later (for the tech case). But take a moment to
reflect on your ideal engineering team, then reflect on one you’ve built or are
building, think of where they may be different, and why. “You go to war with the
army you have, not the army you want.” Yeah, but why?! You built, and are
building this team!

For all my advocacy in this post, it may surprise you to hear that I believe it
takes years to be excellent at a language. It’s not just syntax, it’s soil and
atmosphere: common bug flows and how to spot them, footguns, tooling, library
ecosystem, culture. I have a bunch of tips for learning a new language,
especially weird ones,
and you’ll even note that even that post starts with
please please please be careful before introducing one of these to your
companies.

So why am I making all these arguments here for Going For It? Because I think in
the presence of experienced mentors, it only takes a few days, maybe a week or
two, for a developer to get up to speed on a new language to an adequate level,
especially with mandatory code review. This may sound preposterous, but in my
observation…

…most people don’t know much about Boring Stacks everyone chooses. Also, they don’t generalize!

The thing about most Python Developers™ I’ve worked with is that they don’t even
know that much about Python. They couldn’t articulate when and how to use
abc vs. just inheriting from object, or why you might prefer namedtuple
over dataclasses. They won’t know that import x.y.z will import
x/__init__.py and x/y/__init__.py and x/y/z.py (or
x/y/z/__init__.py). They couldn’t tell you who GIL is or why you shouldn’t
def send_emails(list_of_recipients, bccs=[]):. This game is especially fun when
talking about JavaScript 😉

Most developers have major gaps in their understandings of the tools they use.
This is fine.
But the thing about “pure atmosphere” arguments like “it’s easier
to hire people with [X] experience” is, in my observation, that their experience
isn’t necessarily that deep, and that people who think they’re hiring skilled
users of these languages are lying to themselves. Many people across the 7
companies I’ve worked for are best described as “adequate.”

Training isn’t free, but it’s a fixed cost per developer, and especially with
skilled mentors, probably smaller than you think. I find it extremely unlikely
that the fixed cost of 2-4 days of training is less than the cost of superlinear
build times + managing 1000s of Sidekiq or Celery queues when you could have
just picked Go.


Another thing: these techs don’t actually standardize. I’ve worked at 4
companies using Flask; none had comparable app structure or used the same
plugins (yay microframeworks!). “Just use React,” but for a while, it was “to
state manager or not state manager (Redux),” then it was “to hooks or not to
hooks,” then it was “to CSS-in-JS or Something Else (and/or BEM, or now,
Tailwind).” Every company I’ve worked for builds their own Design System and/or
component library that has its own weird calling conventions. No two React
companies I’ve worked at tested the same way, or had transferable knowledge on
their asset pipeline or JS transpilation options.

Why are we at “atmosphere at all costs”?

These can probably be separate blog posts, but this is long enough so I’ll just
bullet out some flamebait and maybe follow-up later:

A decade of 0% and dumb money made tech stupid; “VC tech company” became a
playbook
.
Startups used to be the Wild West, and VC was expected to go
to 0 because the bets were so dumb and big. Wild-eyed creatives were more
welcome there. Over time “venture capital” kept the name and tried to keep the
cowboy branding of Big Bets, but in practice became more akin to regular
investing, and VC-backed tech developed a playbook that expected regular
returns. Punks in garage bands who might have “the juice” got replaced with
clean-cut studio musicians who got Music degrees. This led to risk aversion and
a fetishization of Boring.

Infusion of traditional Business and Capital culture, which traditionally
serves smoother-brains, the boring, and the unimaginative.
Look at airport
business books: they’re often quite braindead. Like what happened to The
Rude Press
, edgy media outlets that were “merely” profitable, and
watching blue checks on Twitter licking their lips thinking that we can replace
TV writers with AI (or that they’re a whole artist by typing “hot cyberpunk
girl” in a Midjourney textfield); as tech got more successful, we got more of
the class of person in the business game who gets uncomfortable with creatives
and their expression, and wants to believe you can create something amazing and
innovative with Known, Controlled, Riskless Process.

Engineering Management and Leadership culture becoming A Thing; people became
herd animals around “Boring.”
The last decade we saw a rise in the Tech
Managerial class and its thought leaders. Managers from other domains without
tech knowledge didn’t do well, but neither did engineers promoted to management.
What was needed was a hidden, Third Thing. A lot of influencing is speaking
authoritatively even if narratives aren’t settled; a favorite snowball fight
in this community was Manager READMEs (for, against). Despite
having lots of Assured Professional Voice, I’ve observed Engineering Leadership
to be a winding road in practice (excellent short Brandur read here). I
think “Boring” is a fine strategy some of the time, but in the spirit of being
an “adult in the room,” I think it caught on as a Universal Good, and the class
of managers herd animal-ed their way into it.

The newest, easiest way to Fear the Unknown. The funny thing about
“atmosphere costs/benefits are the most important” when they’re the hardest to
make concrete: I feel like the late aughts that I’m lionizing used to be about
“surface,” and were similarly, maddeningly fuzzy. Functional programmers
insisted without evidence that their programs were More Correct. Dynamic types
people insisted without evidence that their programs weren’t less
correct. I think we were playing the same game of appealing to feelings, but
before a decade of increased CS enrollment, and boot camps, and Stack Overflow,
and GitHub, and advances in text editing (Language Servers, the monoculture of
VSCode), we couldn’t talk about atmosphere, so we went as high up as we could.

Meaning: it is, as it always was, a cope for the terror of leading an
engineering team. You’re at sea with the livelihoods of many in your decisions,
and while you’re pretty sure you know how to read the stars, it’s the second
week of cloudy skies in a row; you start to get superstitious. You’re more
inclined to believe the Authoritative Voice of a Calm Professional. You’ll take
fewer risks in the things you can control because you’re so afraid of the
things you can’t.

(sidenote: more of y’all need to work in the arts for a bit 😉)


I have a lot more thoughts on those points (there are a million ways to fuck
up bringing exotic tech and with all this said, most people shouldn’t do it;
also, “Boring” has saved a ton of companies. But I promised
flamebait!). Takeaways:

  • Consider, when talking with someone else about about technologies: is it
    a soil, surface, or atmosphere conversation? Decide together
    which one you’d like to have, and the limitations of the layer you picked,
    because it’s easy to talk past each other if you don’t.

  • Re-evaluate what you consider “boundaries” between technologies. Is React
    really just React everywhere? If your team uses Retool, for example,
    isn’t this also a programmable interface with various bolted-on technologies
    with access to your production datastores? Why is adding Retool easier than a
    Java service? And do consider the “training time” of learning Retool onerous?

    Another fun thought example: do you consider it so dangerous to build native
    phone apps instead of React Native, Flutter, or LiveView
    Native
    ? Why is that different?

  • Tech choice won’t be ultimately responsibe for the success of your company.
    But it will definitely shape what your path looks like, and it’s a mistake to
    say “tech doesn’t matter.” Bleacher Report went from 150 Ruby servers to 5
    (“probably overprovisioned”) Elixir servers
    after a port. Languages with
    proper concurrency like JVM, BEAM, or Go don’t require Sidekiq or Celery
    queues and additional workers. BEAM’s preemptive scheduler means you don’t get
    a noisy neighbor issue like Lyft dealt with in Python, or which will
    probably bite you in Node’s “event loop but on a single core.” Training isn’t
    free (let me say that again: it’s not free!) but some tech was purposely
    built to be efficient and others scale horribly.

  • Reconsider “sacred cows” in all areas of tech, generally. Microservices aren’t
    inevitable (consider FB’s Blue App, Dropbox, YouTube, Instagram: all
    monoliths). PHP was the laughingstock of the last decade and yet it powered
    Slack, Lyft for many years, and Hack still runs Facebook. “Cloud is better,”
    except Stack Overflow has much more data and traffic than you do and runs on
    9 machines.
    WhatsApp got acquired for $19b while doing manual, bare-metal
    deploys.

    And: if the conditions are right and your team has the fire for it, you can
    run in a tech stack that isn’t JS, Ruby, Python, Java.

Additionally:

  • Take a moment to ask yourself: what don’t I know about my favorite
    technologies? What would be unlocked if I did?

  • What training or exercises could you and/or your team undertake to level up on
    the tech stacks you do use?


If you liked these, you might like:

Edit (5/18/2023): The original version of this article claimed Golang had a
cooperative scheduler (and linked to this 2018 article), but thanks to
some friends at HN, Go has had a preemptive scheduler since 2020.

Thanks for
the read!
Disagreed? Violent agreement!? Feel free
to join my mailing list, drop me a line at , or leave a comment below! I’d love to
hear from you 😄

Read More