Anything can be a message queue if you use it wrongly enough

You may think that the world is in a state of relative peace. Things
look like they are somewhat stable, but reality couldn’t be farther
from the truth. There is an enemy out there that transcends time,
space, logic, reason, and lemon-scented moist towelettes. That enemy
is a scourge of cloud costs that is likely the single reason why
startups die from their cloud bills when they are so young.

The enemy is Managed NAT
Gateway.
It is a service that lets you egress traffic from a VPC to the public
internet at $0.07 per gigabyte. This is something that is probably
literally free for them to run but ends up getting a huge chunk of
their customer’s cloud spend. Customers don’t even look too deep into
this because they just shrug it off as the cost of doing business.

This one service has allowed companies like the duckbill
group to make millions by showing
companies how to not spend as much on the cloud.

However, I think I can do one better. What if there was a better way
for your own services? What if there was a way you could reduce that
cost for your own services by up to 700%? What if you could bypass
those pesky network egress costs yet still contact your machines over
normal IP packets?

Base facts

Before I go into more detail about how this genius creation works,
here’s some things to consider:

When AWS launched originally, it had three services:

• S3 – Object storage for
  cloud-native applications
• SQS – A
  message queue
• EC2 –
  A way to run Linux virtual machines somewhere

Of those foundational services, I’m going to focus the most on S3: the
Simple Storage Service. In essence, S3 is malloc() for the cloud.

The C programming language

When using the C programming language, you normally are working with
memory in the stack. This memory is almost always semi-ephemeral and
all of the contents of the stack are no longer reachable (and
presumably overwritten) when you exit the current function. You can do
many things with this, but it turns out that this isn’t very useful in
practice. To work around this (and reliably pass mutable values
between functions), you need to use the
malloc()
function. malloc() takes in the number of bytes you want to allocate
and returns a pointer to the region of memory that was allocated.

When you get a pointer back from malloc(), you can store anything in
there as long as it’s the same length as you passed or less.

Oh, also anything stored in that pointer to memory you got back from
malloc() is stored in an area of ram called “the heap”, which is
moderately slower to access than it is to access the stack.

S3 in a nutshell

Much in the same way, S3 lets you allocate space for and submit
arbitrary bytes to the cloud, then fetch them back with an address.
It’s a lot like the malloc() function for the cloud. You can put
bytes there and then refer to them between cloud functions.

And these arbitrary bytes can be anything. S3 is usually used for
hosting static assets (like all of the conversation snippet avatars
that a certain website with an orange background hates), but nothing
is stopping you from using it to host literally anything you want.
Logging things into S3 is so common it’s literally a core product
offering from Amazon. Your billing history goes into S3. If you
download your tax returns from WealthSimple, it’s probably downloading
the PDF files from S3. VRChat avatar uploads and downloads are done
via S3.

IPv6

You know what else is bytes? IPv6
packets. When you send an
IPv6 packet to a destination on the internet, the kernel will prepare
and pack a bunch of bytes together to let the destination and
intermediate hops (such as network routers) know where the packet
comes from and where it is destined to go.

Normally, IPv6 packets are handled by the kernel and submitted to a
queue for a hardware device to send out over some link to the
Internet. This works for the majority of networks because they deal
with hardware dedicated for slinging bytes around, or in some cases
shouting them through the air (such as when you use Wi-Fi or a mobile
phone’s networking card).

The core Unix philosophy: everything is a file

There is a way to bypass this and have software control how network
links work, and for that we need to think about Unix conceptually for
a second. In the hardcore Unix philosophical view: everything is a
file. Hard drives and storage devices are files. Process information
is viewable as files. Serial devices are files. This core philosophy
is rooted at the heart of just about everything in Unix and Linux
systems, which makes it a lot easier for applications to be
programmed. The same API can be used for writing to files, tape
drives, serial ports, and network sockets. This makes everything a lot
conceptually simpler and reusing software for new purposes trivial.

In Linux, you can create a
TUN/TAP device to let
applications control how network or datagram links work. In essence,
it lets you create a file descriptor that you can read packets from
and write packets to. As long as you get the packets to their intended
destination somehow and get any other packets that come back to the
same file descriptor, the implementation isn’t relevant. This is how
OpenVPN, ZeroTier, FreeLAN, Tinc, Hamachi, WireGuard and Tailscale
work: they read packets from the kernel, encrypt them, send them to
the destination, decrypt incoming packets, and then write them back
into the kernel.

In essence

So, putting this all together:

• S3 is malloc() for the cloud, allowing you to share arbitrary
  sequences of bytes between consumers.
• IPv6 packets are just bytes like anything else.
• TUN devices let you have arbitrary application code control how
  packets get to network destinations.

In theory, all you’d need to do to save money on your network bills
would be to read packets from the kernel, write them to S3, and then
have another loop read packets from S3 and write those packets back
into the kernel. All you’d need to do is wire things up in the right
way.

So I did just that.

Here’s some of my friends’ reactions to that list of facts:

• I feel like you’ve just told me how to build a bomb. I can’t belive
  this actually works but also I don’t see how it wouldn’t. This is
  evil.
• It’s like using a warehouse like a container ship. You’ve put a
  warehouse on wheels.
• I don’t know what you even mean by that. That’s a storage method.
  Are you using an extremely generous definition of “tunnel”?
• sto psto pstop stopstops
• We play with hypervisors and net traffic often enough that we know
  that this is something someone wouldn’t have thought of.
• Wait are you planning to actually implement and use ipv6 over
  s3?
• We’re paying good money for these shitposts 🙂
• Is routinely coming up with cursed ideas a requirement for working
  at tailscale?
• That is horrifying. Please stop torturing the packets. This is a
  violation of the Geneva Convention.
• Please seek professional help.

Hoshino

Hoshino is a system for putting outgoing IPv6 packets into S3 and then
reading incoming IPv6 packets out of S3 in order to avoid the absolute
dreaded scourge of Managed NAT Gateway. It is a travesty of a tool
that does work, if only barely.

The name is a reference to the main character of the anime Oshi no
Ko, Hoshino Ai. Hoshino is
an absolute genius that works as a pop idol for the group B-Komachi.

Hoshino is a shockingly simple program. It creates a TUN device,
configures the OS networking stack so that programs can use it, and
then starts up two threads to handle reading packets from the kernel
and writing packets into the kernel.

When it starts up, it creates a new TUN device named either hoshino0
or an administrator-defined name with a command line flag. This
interface is only intended to forward IPv6 traffic.

Each node derives its IPv6 address from the
machine-id
of the system it’s running on. This means that you can somewhat
reliably guarantee that every node on the network has a unique address
that you can easily guess (the provided ULA /64 and then the first
half of the machine-id in hex). Future improvements may include
publishing these addresses into DNS via Route 53.

When it configures the OS networking stack with that address, it uses
a netlink socket to do this.
Netlink is a Linux-specific socket family type that allows userspace
applications to configure the network stack, communicate to the
kernel, and communicate between processes. Netlink sockets cannot
leave the current host they are connected to, but unlike Unix sockets
which are addressed by filesystem paths, Netlink sockets are addressed
by process ID numbers.

In order to configure the hoshino0 device with Netlink, Hoshino does
the following things:

• Adds the node’s IPv6 address to the hoshino0 interface
• Enables the hoshino0 interface to be used by the kernel
• Adds a route to the IPv6 subnet via the hoshino0 interface

Then it configures the AWS API client and kicks off both of the main
loops that handle reading packets from and writing packets to the
kernel.

When uploading packets to S3, the key for each packet is derived from
the destination IPv6 address (parsed from outgoing packets using the
handy library
gopacket) and the
packet’s unique ID (a
ULID to ensure that
packets are lexicographically sortable, which will be important to
ensure in-order delivery in the other loop).

When packets are processed, they are added to a
bundle for
later processing by the kernel. This is relatively boring code and
understanding it is mostly an exercise for the reader. bundler is
based on the Google package
bundler,
but modified to use generic types because the original
implementation of bundler predates them.

cardio

However, the last major part of understanding the genius at play here
is by the use of cardio.
Cardio is a utility in Go that lets you have a “heartbeat” for events
that should happen every so often, but also be able to influence the
rate based on need. This lets you speed up the rate if there is more
work to be done (such as when packets are found in S3), and reduce the
rate if there is no more work to be done (such as when no packets are
found in S3).

When using cardio, you create the heartbeat channel and signals like
this:

heartbeat, slower, faster := cardio.Heartbeat(ctx, time.Minute, time.Millisecond)

The first argument to cardio.Heartbeat is a
context that lets you cancel the
heartbeat loop. Additionally, if your application uses
ln‘s
opname facility, an
expvar gauge will be created and named
after that operation name.

The next two arguments are the minimum and maximum heart rate. In this
example, the heartbeat would range between once per minute and once
per millisecond.

When you signal the heart rate to speed up, it will double the rate.
When you trigger the heart rate to slow down, it will halve the rate.
This will enable applications to spike up and gradually slow down as
demand changes, much like how the human heart will speed up with
exercise and gradually slow down as you stop exercising.

When the heart rate is too high for the amount of work needed to be
done (such as when the heartbeat is too fast, much like tachycardia in
the human heart), it will automatically back off and signal the heart
rate to slow down (much like I wish would happen to me sometimes).

This is a package that I always wanted to have exist, but never found
the need to write for myself until now.

Terraform

Like any good recovering SRE, I used
Terraform to automate creating
IAM users and security policies for
each of the nodes on the Hoshino network. This also was used to create
the S3 bucket. Most of the configuration is fairly boring, but I did
run into an issue while creating the policy documents that I feel is
worth pointing out here.

I made the “create a user account and policies for that account” logic
into a Terraform module because that’s how you get functions in
Terraform. It looked like this:

data "aws_iam_policy_document" "policy" {
  statement {
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:ListBucket",
    ]
    effect = "Allow"
    resources = [
      var.bucket_arn,
    ]
  }

  statement {
    actions   = ["s3:ListAllMyBuckets"]
    effect    = "Allow"
    resources = ["*"]
  }
}

When I tried to use it, things didn’t work. I had given it the
permission to write to and read from the bucket, but I was being told
that I don’t have permission to do either operation. The reason this
happened is because my statement allowed me to put objects to the
bucket, but not to any path INSIDE the bucket. In order to fix this, I
needed to make my policy statement look like this:

statement {
  actions = [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket",
  ]
  effect = "Allow"
  resources = [
    var.bucket_arn,
    "${var.bucket_arn}/*", # allow every file in the bucket
  ]
}

This does let you do a few cool things though, you can use this to
create per-node credentials in IAM that can only write logs to their
part of the bucket in particular. I can easily see how this can be
used to allow you to have infinite flexibility in what you want to do,
but good lord was it inconvenient to find this out the hard way.

Terraform also configured the lifecycle policy for objects in the
bucket to delete them after a day.

resource "aws_s3_bucket_lifecycle_configuration" "hoshino" {
  bucket = aws_s3_bucket.hoshino.id

  rule {
    id = "auto-expire"

    filter {}

    expiration {
      days = 1
    }

    status = "Enabled"
  }
}

The horrifying realization that it works

Once everything was implemented and I fixed the last bugs related to
the efforts to make Tailscale faster than kernel
wireguard, I tried to
ping something. I set up two virtual machines with
waifud and installed Hoshino.
I configured their AWS credentials and then started it up. Both
machines got IPv6 addresses and they started their loops. Nervously, I
ran a ping command:

xe@river-woods:~$ ping fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f
PING fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f(fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f) 56 data bytes
64 bytes from fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f: icmp_seq=1 ttl=64 time=2640 ms
64 bytes from fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f: icmp_seq=2 ttl=64 time=3630 ms
64 bytes from fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f: icmp_seq=3 ttl=64 time=2606 ms

It worked. I successfully managed to send ping packets over Amazon S3.
At the time, I was in an airport dealing with the aftermath of Air
Canada’s IT system falling the heck
over
and the sheer feeling of relief I felt was better than drugs.

Then I tested TCP. Logically holding, if ping packets work, then TCP
should too. It would be slow, but nothing in theory would stop it. I
decided to test my luck and tried to open the other node’s metrics
page:

$ curl http://[fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f]:8081
# skipping expvar "cmdline" (Go type expvar.Func returning []string) with undeclared Prometheus type
go_version{version="go1.20.4"} 1
# TYPE goroutines gauge
goroutines 208
# TYPE heartbeat_hoshino.s3QueueLoop gauge
heartbeat_hoshino.s3QueueLoop 500000000
# TYPE hoshino_bytes_egressed gauge
hoshino_bytes_egressed 3648
# TYPE hoshino_bytes_ingressed gauge
hoshino_bytes_ingressed 3894
# TYPE hoshino_dropped_packets gauge
hoshino_dropped_packets 0
# TYPE hoshino_ignored_packets gauge
hoshino_ignored_packets 98
# TYPE hoshino_packets_egressed gauge
hoshino_packets_egressed 36
# TYPE hoshino_packets_ingressed gauge
hoshino_packets_ingressed 38
# TYPE hoshino_s3_read_operations gauge
hoshino_s3_read_operations 46
# TYPE hoshino_s3_write_operations gauge
hoshino_s3_write_operations 36
# HELP memstats_heap_alloc current bytes of allocated heap objects (up/down smoothly)
# TYPE memstats_heap_alloc gauge
memstats_heap_alloc 14916320
# HELP memstats_total_alloc cumulative bytes allocated for heap objects
# TYPE memstats_total_alloc counter
memstats_total_alloc 216747096
# HELP memstats_sys total bytes of memory obtained from the OS
# TYPE memstats_sys gauge
memstats_sys 57625662
# HELP memstats_mallocs cumulative count of heap objects allocated
# TYPE memstats_mallocs counter
memstats_mallocs 207903
# HELP memstats_frees cumulative count of heap objects freed
# TYPE memstats_frees counter
memstats_frees 176183
# HELP memstats_num_gc number of completed GC cycles
# TYPE memstats_num_gc counter
memstats_num_gc 12
process_start_unix_time 1685807899
# TYPE uptime_sec counter
uptime_sec 27
version{version="1.42.0-dev20230603-t367c29559-dirty"} 1

I was floored. It works. The packets were sitting there in S3, and I
was able to pluck out the TCP
response
and I opened it with xxd and was able to confirm the source and
destination address:

00000000: 6007 0404 0711 0640
00000008: fd5e 59b8 f71d 9a3e
00000010: c05f 7f48 de53 428f
00000018: fd5e 59b8 f71d 9a3e
00000020: 59e5 5085 744d 4a66

It was fd5e:59b8:f71d:9a3e:59e5:5085:744d:4a66 trying to reach
fd5e:59b8:f71d:9a3e:c05f:7f48:de53:428f.

Cost analysis

When you count only network traffic costs, the architecture has many
obvious advantages. Access to S3 is zero-rated in many cases with S3,
however the real advantage comes when you are using this cross-region.
This lets you have a worker in us-east-1 communicate with another
worker in us-west-1 without having to incur the high bandwidth cost
per gigabyte when using Managed NAT Gateway.

However, when you count all of the S3 operations (up to one every
millisecond), Hoshino is hilariously more expensive because of simple
math you can do on your own napkin at home.

For the sake of argument, consider the case where an idle node is
sitting there and polling S3 for packets. This will happen at the
minimum poll rate of once every 500 milliseconds. There are 24 hours
in a day. There are 60 minutes in an hour. There are 60 seconds in a
minute. There are 1000 milliseconds in a second. This means that each
node will be making 172,800 calls to S3 per day, at a cost of $0.86
per node per day. And that’s what happens with no traffic. When
traffic happens that’s at least one additional PUT–GET call pair
per-packet.

Depending on how big your packets are, this can cause you to easily
triple that number, making you end up with 518,400 calls to S3 per day
($2.59 per node per day). Not to mention
TCP overhead from the three-way handshake and acknowledgement packets.

This is hilariously unviable and makes the effective cost of
transmitting a gigabyte of data over HTTP through such a contraption
vastly more than $0.07 per gigabyte.

Other notes

This architecture does have a strange advantage to it though: assuming
a perfectly spherical cow, adequate network latency, and sheer luck
this does make UDP a bit more reliable than it should be otherwise.

With appropriate timeouts and retries at the application level, it may
end up being more reliable than IP transit over the public internet.

I guess you could optimize this by replacing the S3 read loop with
some kind of AWS lambda handler that remotely wakes the target
machine, but at that point it may actually be better to have that
lambda POST the contents of the packet to the remote machine. This
would let you bypass the S3 polling costs, but you’d still have to pay
for the egress traffic from lambda and the posting to S3 bit.

Shitposting so hard you create an IP conflict

Something amusing about this is that it is something that technically
steps into the realm of things that my employer does. This creates a
unique kind of conflict where I can’t easily retain the intellectial
property (IP) for this without getting it approved from my employer.
It is a bit of the worst of both worlds where I’m doing it on my own
time with my own equipment to create something that will be ultimately
owned by my employer. This was a bit of a sour grape at first and I
almost didn’t implement this until the whole Air Canada debacle
happened and I was very bored.

However, I am choosing to think about it this way: I have successfully
shitposted so hard that it’s a legal consideration and that I am going
to be absolved of the networking sins I have committed by instead
outsourcing those sins to my employer.

I was told that under these circumstances I could release the source
code and binaries for this atrocity (provided that I release them with
the correct license, which I have rigged to be included in both the
source code and the binary of Hoshino), but I am going to elect to not
let this code see the light of day outside of my homelab. Maybe I’ll
change my mind in the future, but honestly this entire situation is so
cursed that I think it’s better for me to not for the safety of
humankind’s minds and wallets.

I may try to use the basic technique of Hoshino as a replacement for
DERP, but that sounds like a lot of effort after I have proven that
this is so hilariously unviable. It would work though!

---