Being “rockstars”: when software was a talents/creatives industry

Tuesday, June 13, 2023 :: Tagged under: engineering essay. ⏰ 15 minutes.

🎵 The song for this post is Torn, by Natalie Imbruglia. 🎵

What was behind the flames of Where have all the hackers gone?? The
biggest one I’m conscious of is Kellan’s Software and its Discontents
series. It’s an excellent set of articles that try to
provide answers to the question “why is everyone so miserable about software
these days?”

I really loved it, but I felt it didn’t articulate the overarching narrative I
observed behind all the glum feelings: application software is no longer a game
of “creatives” and “talents,” it’s now a “commodified sludge” version with a
playbook on how to produce it. This playbook describes a game that’s a) not fun
or motivating for most of the people involved, b) based on risk management
(fear) rather than outcome maximization (hope), and c)… still doesn’t seem to
work, actually? But everyone yells at you and calls you immature or Not Serious
if you deviate?

The labor/hiring market was very slow to catch up to this, so for a while,
remuneration and narrative were about creatives/hackers, but the work and
outcomes were conservative and managerial. Software was Taylor Swift-priced but
“uncredited studio musician”-outcomed.

For many people working in software, this gets incredibly demotivating, but
also, of particular interest to me: when we overreact and play to a playbook
too much, we prevent any Taylor Swifts from ascending, since we decide to work
with mediocre tools and “lowest common denominator” teams. Getting back to “that
magic feeling” comes from reaching for the ceiling and maximizing your people,
rather than flattening them, as scary as that may be.

This post describes what I mean by a “creative”/”talents” industry, argues that
moving away from this model made software more miserable for everybody, and that
maybe we need to reconsider the playbook that represents that move, at least in
part.

“Do they have “the juice””?!

Something you see a lot in these industries: scouts and producers looking for
talent that delivers, and using their “gut” to find it, because it’s not obvious
on inspection what’s going to work. These industries are… colorful. An example
is entertainment: if you were a music executive who wanted to cash in on grunge
in the 90’s, you could try to create a new grunge band with professional
musicians, or you could have talent scouts finding teenagers in garages.

One of these is strictly the better option: there’s no formula to create
successful grunge; some bands have “the juice” that resonated with disaffected
White teens who bought CDs en masse, and others can technically play the
instruments and say the words, but it’s missing that magic, and the teens reject
it. So like shrimping boats, most successful grunge bands (like most other
“youth culture whisperers” in entertainment) weren’t farmed and harvested, they
were wild-caught.

Here’s a clip from one of the most famous Reply-All episodes, where a
musician gets a meeting with a major record executive who heard his demo. I
recommend giving it a listen but I’ll put the transcript in the summary tab if
you’re just skimming. Consider the lifestyles and values of the various people
involved, what “work” looks like to them, and how funny it is that these two men
even met and could be collaborators:

Transcript of audio file

PJ: so i was very surprised to learn that Evan actually made this song
completely alone. And once he was done he sent it to this company that would
print CDs for you. And when he would play shows locally, he would just give
the CDs out for free.

ALEX: Ok.

PJ: The thing that happened next is the part that just feels like the
embarrassing dream that every musician has that they don’t tell anybody. Which
is, out of the blue, he gets a phone call from this guy who says, “Hey, I work
for Universal Music [Alex: Oh wow], the largest music label in the world”
[Alex: Uh huh]. They send a Lincoln town car to his house.

Evan: First of all, I live in Greensboro North Carolina, it’s like a medium
sized town, it’s not a big town, but they send this town car, this really nice
town car with this driver. Really cool guy, just telling me stories and talkin
up a storm and so I get to the, I get to the plane and I’m in first class, and
then I get off the plane and you know how people hold up signs with your name
on ‘em you know?

PJ: No one has ever done that for me.

EVAN: There’s a guy with a sign with my name on it and he takes me to the
hotel, and I get into this big huge suite and you know, how do you, how do you
absorb that kind of experience I dunno.

PJ: Evan says he kept thinking, “This song is really quirky, are they sure
it’s gonna be a hit?” But it didn’t matter, the whole thing was like a fever
dream. He gets whisked into this meeting with Doug Morris, legendary label
exec at the time, he ran Universal Music.

EVAN: I walked in there and he says you like ice cream? And I said yea I love
ice cream and then he said something into this monitor on his desk [PJ: What?]
and then this beautiful like 6 foot tall … I guess it was his secretary I
dunno, but she comes out with this tray of ice cream, no check this out, like
coconut amazing like coconut flavored ice cream that’s like organic coconut
flavored ice cream in martini glasses. And I, and I sit there and I eat ice
cream with Doug Morris and there’s a rapper called Juvenile?

PJ: Yeah I know Juvenile.

EVAN: And he had a song called ‘Back That Azz Up’?

PJ: Yeah I know ‘Back That Azz Up’.

EVAN: First time I was in Doug Morris’s office, he played me that song.

[JUVENILE SONG PLAYS]

EVAN: I thought, wow this makes my song look really stupid. It was just funny
cause Doug Morris was at that time was probably in his 60s and he was really
digging this song.

EVAN: And next thing I know I was signing a record contract– it all kinda
happened so fast. It was really unbelievable.

The point is: software was more like that. Getting flown as an undergrad
to SF in 2009 for an interview at Facebook, put in a hotel, served free food,
all because you could code? You felt cool, especially if you’d spent the
last two years in libraries hacking up side projects, but also: you were dazed
and confused by whatever machine was spending that much money just to see if you
had the magic they needed.

Here are the ingredients for a talents/creatives industry:

• It needs to be very valuable, like professional athletes, musicians, movie
  stars. Winning big needs to mean winning big.

• It must be hard or impossible to replicate success “synthetically.” We
  even have a show meant to farm pop stars (American Idol) and while many
  contestants had some kind of career in music, it’s really only produced one
  major pop star (Kelly Clarkson, the very first winner). An industry like this
  must have some degree of mystery of who’s got “the juice.”

• “The juice”… has actually got to matter. Here’s where I think tech went
  wrong: there needs to be an actual force multiplier to having that ineffable
  talent. Kendrick Lamar being Kendrick Lamar needs to actually produce a
  Kendrick Lamar-sized result. I think tech really convinced ourselves that if
  we wrote Lisp like Paul Graham we could Smart Person our way into riches, but
  it turns out it’s much easier to convince yourself you’re a tech genius who
  innovates than actually be a tech genius who innovates. Very, very few
  companies actually made profits investing in tech with genius hackers: for
  most, the path of less resistance was to adopt the brand and sheen of “tech
  company,” make Engineering a cost center, and play finance instead.

Okay, but why bring this up?

Because while there’s nothing wrong with the analysis in Software and its
Discontents, I think much of its diagnoses are mitigated by teams that embrace
the narrative (and build a reality of) “being in an industry of creatives.” That
said, he and I will likely never see eye-to-eye on this because he more-or-less
invented Choose Boring Technology, and most of what I advocate is to
break playbook, embrace narrative, and be interesting.¹

He brings up four major bullet points in his essays:

• an explosion in the complexity of software development
• tech talent becoming significantly more expensive
• success becoming more elusive than ever, with startups having “lost that magic feeling”
• conflicts over changing expectations of the work environment

Let’s tackle these. If you get the “creative industry” mindset, you can operate
out of the worst of them.

tech talent becoming significantly more expensive

First: hire fewer people. If you aren’t feeling desperately, desperately
constrained, don’t hire. At around the 5-7 year mark engineers start
smugly saying “it’s best to write no code at all! Every line is expensive and
must be maintained,” and while that’s absolutely true I want you to imagine a
new hire as at least as operationally expensive to your Eng org as a 200k line
service.

When WhatsApp was acquired by Facebook, they had 35 engineers. They were
able to do this in part because they didn’t “follow playbook,” and while you
probably can’t go that extreme, per the creative industries point: do you
think SNL would be improved if they had 400 more writers? If HBO added 15
showrunners to Game of Thrones?

Teams are an immutable data structure: when you change them, you get a new one.
Adding too many people, too quickly, adds links to the game of Telephone that is
every conversation your company already has. If people are introduced to a
company without enough seasoned mentors, and without its culture and vision
established, they add process and produce documents because it feels like
productive work. Your customer doesn’t care about any of those things.

The example I give is the ecosystem of a pond: imagine it’s got turtles and fish
and bacteria and algae. That’s your company’s culture. If you add 5% of its
volume in water from a different pond (with different bacteria, fish, algae),
the resulting pond over a few weeks is pretty much the first pond but with more
water. But if you ever add 40% of its volume in water, the new pond’s fauna may
win out, and the resulting pond may be unrecognizable.

Hire carefully, but moreover: hire as slowly as you can manage. Bonus, this
also works well with being extremely selective about who you hire.

(I recognize that one of the reasons we all hired so much was investor pressure.
If you’re leadership at a company, your daily dashboard includes cash-on-hand,
and the investors don’t want that cash back: they want “cash go down, stock
price go up,” and you needed to make that happen by the next board meeting. In
the 0% era, one way you made that happen was hiring expensive engineers.)

Other “pretend we’re a creative industry” things you can do is break playbook:

• Consider colocating servers instead of cloud. Squarespace did it, Twitter did
  it, Basecamp is doing it, Stack Exchange does it. It’s not only possible but
  often way, way cheaper.

• Hell, even consider a “pets, not cattle” model while you get started. Don’t
  get trapped into creating a shitty version of Heroku and your product.

• Consider stateful deploys, at least as another layer on top of stateless. If
  you’ve got a Docker-based flow that git clones from a fresh base image,
  consider hosting a persistent build server that git pulls. Fred Hebert talks
  about how companies he’s worked for lost functionality and flexibility that
  they never recovered (ditto) in the name of removing stateful deploys and
  moving from BEAM to Go + Docker (“following playbook, Choosing Boring”).

Each of these can be their own blog post, with lots of caveats and supporting
evidence. But I’m feeling this get pretty long already, so I’ll jet through the
rest.

“an explosion in the complexity of software development”

I hate the word complexity. As an industry we don’t share a definition, it
usually means “shit that I don’t like.” Luckily Kellan offers some precision:

In both systems thinking and software the term “complex” is a technical one.
It refers to the number of distinct parts in a system, and the connections
between them. Complex systems are characterized by nonlinearity, randomness,
emergence, and surprise. Complexity is why communication and coordination
dominate all other costs when it comes to building software.

Did you notice that the examples from the previous point also removed links
between things? People, an entire infra stack, and the idea that when you want a
cake you should build and maintain a full-on bakery. All gone!

But it doesn’t have to end there. Breaking playbook, be a little creative:

• Does every company need a full team of FTEs to support a React client, that’s
  SSR-ed with extra infra for “hydration” + more engineering built into its
  build system and node_modules folder than the entire Flash Runtime had?
  Consider htmx, or server-side templates, or LiveView.

• Do you really need microservices? Kubernetes? Terraform? Pretend it’s 2011
  again and deploy JARs or binaries, keep and maintain a monolith. Dropbox,
  YouTube, Instagram, Facebook Blue App, Shopify… all (mostly) monoliths, all
  (mostly) pre-date the Misery Era.

• Can you use technology that has better scaling properties than Ruby, Python,
  or Node? I can write more about these, but talk to any eng who’s worked on
  giant codebases in these stacks, bad things happen that don’t happen in, say,
  the JVM. I said in another blog post: “you’ve created an environment
  that’s optimized for your founding engineers to build something quickly in the
  first 7 months, but the price is a set of recurring obstacles which your
  engineers will pay down over the next 7 years.”

“success becoming more elusive than ever, with startups having “lost that magic feeling””

All my suggestions above radically reduce costs of developing software, which
hopefully leads to some increased version of success.

Additionally, the talent you’ll acquire to paint outside the lines
here will both be more inspired, and inspire others on the team. A favorite
article of Ramp CEO Eric Glyman’s is Amp it Up!, which contains a lot of
ideas, but a core one: keep the energy high! Someone on a team of 50 really
inspiring engineers who own the whole product and stack, WhatsApp-style, are
going to perform better (and be a lot happier) than someone on a team of 50
React devs who owns 3 pages.

Tech won’t be the reason your business wins, but it can help create that
high-energy, high-motivation environment, which will strongly up your chances.

---

I’ll end with a question I don’t have the answer to: is the application software
industry “irreducibly creative?” By that I mean: can it ever produce a set of
“established solutions” that are cost-effective?

I’ll say some industries cannot reliably be made into a cost-effective sludge
factory: you must catch wild, and you can’t produce “the goods” reliably and
synthetically. I gave the example of grunge music, but a more favorite example
is the content wars of HBO, Disney, and Netflix.

See, HBO and Disney have been kings of content for decades. They have
pipelines for producing quality content people love. They know that you gotta
pay a bunch of desperate weirdos in LA and take strange bets based on gutfeel.
In this story about HBO, look how close they were to losing the biggest show in
television for a decade, shepherded by a person they were firing:

Carolyn Strauss is among them. Despite coming up with hit after hit for HBO,
in 2008 she was shown the door. In her telling, it was because she passed on
one show, which was Mad Men. And as a result, months later, she was fired. And
on the way out, she’s like, ‘There’s one show I’m developing that I want to
keep my hands on. It’s the dragon show.’ And they’re like, ‘You want that?
Sure. I’m sure we’ll make that.’ And they did. And Carolyn Strauss is a top
producer of HBO’s most successful drama of all time, Game of Thrones.

These are very human processes. Netflix was a newcomer, and tech-first, so
when they got into original programming, they tried to data their way out of it.
“Data says new subscribers fall off after two seasons, shows will only have
two seasons” (bonus, it’s easier to exploit LA entertainment workers like
this).

And they’re losing the bet. Turns out people want Marvel movies, Andor, and
The Rehearsal more than they want a $200m movie with hit stars nobody
watched or a clumsy Will Smith racism allegory.

(Anecdotally, I have a friend who worked in Disney Animation during Frozen,
and she said offhandedly that many of them were convinced it was going to be a
dud. You can’t predict success in this game)

So some industries don’t “sludge.” There comes a point where the humans gotta
human. I wonder to what degree software is? Probably more than entertainment,
since it was easier for entertainers to sludge into a streaming service than it
was for a streaming service to sludge into entertainment. But I don’t think it’s
as deterministic as our “playbook” is sold as.

---

^
In his (and my) defense, the tenets of Choose Boring Technology (when you
read the articles) have the disclaimers and asterisks that make it
pretty unimpeachable. It should be understood as an argument against a certain
type of “novelty-driven development.” It never explicitly says “popularity is
worth suspending your Engineering sensibilities,” and in Kellan’s Discontents
post, he writes “As an engineering leader raising the quality of technical
decision making is arguably your most important job after building the team
itself.” So I don’t think he wants people saying “Choose Boring Technology!”
to mean “we’ll use Kubernetes and microservices because everyone else is using
it” for a pre-launch B2B.

Similarly, please read my posts with the same understanding. I add a million
similar disclaimers on the bottom of my “Against Boring” post for similar
reasons. Choose Boring was a reaction to a real problem.

I keep saying the point of these posts is to emphasize that the tech has to
matter. To use the framework of the last post, we should consider soil in
addition to atmosphere, and based on his quote on technical decisions, I think
he’d agree! Unfortunately, I don’t think that’s how most tech leaders use
“Boring.”

Thanks for
the read! Disagreed? Violent agreement!? Feel free
to join my mailing list, drop me a line at , or leave a comment below! I’d love to
hear from you 😄