Fly

Flyctl Builtins - The Fly Changelog for August

2020-08-26T01:00:00+01:00

Super-simple builtin builders and smart certificates creation - it's all in the latest flyctl (v0.0.139) available now for your command line. Find out more about it in the Changelog.

For the latest version of flyctl, we've focussed on making your life fast and simple. From getting your first deployment up and running to setting up a host's certificate.

Builtins

First up, we've got the new “builtin” builders which you can select when you init your App. We have builtin builders for Node, Ruby, Deno, Go and there's also a static web server available. No need for a Dockerfile, Just init and deploy.

You can find out more at the command line with flyctl builtins list which will give you information on all of the builtins, and flyctl builtins show <builtin-name> which will give you all the details, including the virtual Dockerfile that the builtins use.

With builtins all you need to do is:

Write your code
Run flyctl init
Select a builtin for the code's language, hit return for defaults otherwise
Run flyctl deploy
Finally, run flyctl open to view your app in a browser.

You can see full examples for Go, Node, Ruby, Deno and a static web server in the documentation.

Images

In previous versions of flyctl, we have been able to deploy an image from the command line. In this version, you can now use the Image builder (or --image flag on flyctl init) to select a public Docker image to be published to Fly. This persists the selected image in a fly.toml file.

By using the fly.toml file, it also lets you configure the ports and health checks for the image. Best of all, it lets you save that information into your git repository so you can perform repeatable builds with images.

Backup Regions

When, for whatever reason, an App instance is unable to deploy in a particular region, Fly will look to deploying it in a backup region. This is a longstanding behavior of Fly, but flyctl has been not been good in communicating that that is what is happening, leading to confusion over why an App is deploying in a region not in the region list.

Not anymore! flyctl regions list will now show the regions and the associated backup regions that an app instance may appear in. Also, flyctl status will display (b) net to any region which is not in the region pool and therefore a backup region. This should give everyone a better view of where their app is running.

Guided Certs

In the past, when creating a certificate for a hostname, you had to refer to the documentation or UI for the various steps you needed to take. In this version of flyctl, we've embedded some smart-documentation into the process. When you add a certificate now, flyctl will give you detailed instructions on what you need to set with your DNS provider to direct traffic to your App through your domain name and verify your ownership (allowing a certificate to be generated). It's all part of making flyctl your preferred way to work with Fly.

Orgs and DNS

Also implemented in 0.0.138, are orgs and dns commands. Orgs allows you to add and remove organizations - which you can also do from the Web UI. What you currently can't do is invite and remove users from the organization; you'll have to use the Web UI for that. This, and the dns command, are works in progress and could change in a later version. We thought it was better to let you see what was in the longer pipeline.

Other changes

flyctl init now has an --import option which will use the settings of the imported fly.toml file while creating a new app name and slot.
JSON status output now has timestamps.
flyctl deploy now has a --local-only flag to force builds to happen locally or not at all.

This is the Fly Changelog where we list all significant changes to the Fly platform, tooling and web sites. You can also use the RSS feed of just changelog posts available on fly.io/changelog.xml or consult our dedicated ChangeLog page with all the recent updates.

25th August

flyctl: Version 0.0.139 released.

fix issue with builtin init

Fly Platform/Web

Updated documentation in sync with new flyctl.

25th August

flyctl: Version 0.0.138 released

Added flyctl builtins and flyctl builtins list.
Added flyctl builtins show.
Added images as an option to builder selection.
Added --image as flag to flyctl init
Added prompting to certs process to give directions to user on next steps after adding or checking a certificate.
Changed certs create/delete to add/remove (aliased for back compatibilty)
Added prompting for internal port to flyctl init.
Added display of backup regions.
Added --import to flyctl.
Added --local-only option to flyctl deploy.
Added orgs and dns commands.
Displays app URL before deployment.
JSON status output now has timestamp.
Caught errors in suspend (thanks @alrs).
Clarified monitoring error messages.
Only display auth URL when an error has occurred.
Setting secret to same value no longer causes a "no deployment available" error.
Improved internal API use.
Immediate duplicates in logs supressed in client.

Sandboxing and Workload Isolation

2020-07-29T01:00:00+01:00

Workload isolation makes it harder for a vulnerability in one service to compromise every other part of the platform. It has a long history going back to 1990s qmail, and we generally agree that it’s a good, useful thing.

Despite a plethora of isolation options, in the time I spent consulting for technology companies I learned that the most common isolation mechanism is “nothing”. And that makes some sense! Most services are the single tenant of their deployment environment, or at least so central to the logical architecture that there’s nothing to meaningfully isolate them from. Since isolation can be expensive, and security is under-resourced generally, elaborate containment schemes are often not high up on the list of priorities.

That logic goes out the window when you’re hosting other people’s stuff. Fly.io is a content delivery network for Docker containers. We make applications fast by parking them close to their users; we do that by running bare metal servers in a bunch of data centers around the world, and knitting them together with a global WireGuard mesh. Fly.io is extremely easy to play with — single-digit minutes to get your head around, and rather than talk about it, I’ll just suggest you grab a free account and try it.

Meanwhile, I’m going to rattle off a bunch of different isolation techniques. I’ll spoil the list for you now: we use Firecracker, the virtualization engine behind Amazon’s Lambda and Fargate services. But the solution space we chose Firecracker from is interesting, and so you’re going to hear about it.

chroot

People like to say “chroot isn’t a security boundary”, but of course that isn’t really true, it’s just not very strong by itself. Chroot is the original sandboxing technique.

The funniest problem with chroot is how it’s implemented: in the kernel process table, every struct proc (I was raised on BSD) has a pointer to its current working directory and to its root directory. The root directory is “enforced” when you try to cd to “..”; if your current working directory is already the root, the kernel won’t let “..” go below it. But when you call chroot(2), you don’t necessarily change directories; if you’re “above” your new root, the kernel will never see that new root in a path traversal.

The real problem, of course, is the kernel attack surface. We don’t need to get cute yet; by itself, considering no other countermeasures, chroot gives you ptrace, procfs, device nodes, and, of course, the network.

You shake a lot of these problems off by not running anything as “root”, but not everything. A quick-but-important aside: in real-world attacks, the most important capability you can concede to an attacker is access to your internal network. It’s for the same reason that SSRF vulnerabilities (“unexpected HTTP proxies”) are almost always game-over, even though at first blush they might not seem much scarier than an unchecked redirect: there will be something you can aim an internal HTTP request to that will give an attacker code execution. Network access in a chroot jail is like that, but far more flexible.

This problem will loom over almost everything I write about here; just keep it in mind.

chroot is a popular component in modern sandboxes, but none of them really rely on it exclusively.

Privilege Separation

It’s 1998 and the only serious language you have available to build in is C. You want to receive mail for a group of users, or authenticate and kick off a new SSH session. But those are complicated, multi-step operations, and nobody knows how to write secure C code; it’ll be 30 years before anyone figures that out. You assume you’re going to screw up a parse somewhere and cough up RCE. But you need privileges to get your job done.

One solution: break the service up into smaller services. Give the services different user IDs. Connect services with group IDs. Mush the code around so that the gnarliest stuff winds up in the low-privileged services with the fewest connections to other services. Keep the stuff that needs to be privileged, like mailbox delivery or setting the login user, as tiny as you can.

Call this approach “privsep”.

Despite what its author said about his design, this approach works well. It’s not foolproof, but it has in fact a pretty good track record. The major downside is that it takes a lot of effort to implement; your application needs to be aware that you’re doing it.

Aside: Pledge

If you can change your applications to fit the sandbox, you can take privsep pretty far. OpenBSD got this right with “pledge” and “unveil”, which allow programs to gradually ratchet down the access they get the kernel. It’s a better, more flexible idiom than seccomp, about which more later. But you’re not running OpenBSD, so, moving on.

Prelapsarian Containers

People like to say “Docker isn’t a security boundary”, but that’s not so true anymore, though it once was.

The core idea behind containers is kernel namespacing, which is chroot extended to other kernel identifiers — process IDs, user IDs, network interfaces. Configured carefully, these features give the appearance of a program running on its own machine, even as it shares a running kernel with other programs outside its container.

But even with its own PID space, its own users and groups, and its own network interfaces, we still can’t have processes writing handler paths to /sys, rebooting the system, loading kernel modules, and making new device nodes, and while many of these concerns can be avoided simply by not running as root, not all of them can.

Systems security people spent almost a decade dunking on Docker because of all the gaps in this simplified container model. But nobody really runs containers like this anymore.

Incarceration

Enter mandatory access control, system call filtering, and capabilities.

Mandatory access control frameworks (AppArmor is the one you’ll see) offer system- (or container-) wide access control lists. You can read a version of Docker’s default AppArmor template to see what problems this fixes; it’s a nice concise description of the weaknesses of namespaces on their own.

System call filters let us turn off kernel features; in 2020, if you’re filtering system calls, you’re probably doing it with seccomp-bpf.

Capabilities split “root” into a whole mess of sub-privileges, ensuring that there’s rarely a need to give any program superuser access.

There are lots of implementations of this idea.

Modern Docker, for instance, takes advantage of all these features. Though imperfect, the solution Docker security people arrived at is, I think, a success story. Developers don’t harden their application environments consciously, and yet, for the most part, they also don’t run containers privileged, or give them extra capabilities, or disable the MAC policies and system call filters Docker enforces by default.

It may be even easier to jail a process outside of Docker; Googlers built minijail and nsjail, Cloudflare has “sandbox”, there’s “firejail”, which is somewhat tuned for things like browsers, and systemd will do some of this work for you. Which tool is a matter of taste; nsjail has nice BPF UX; firejail interoperates with AppArmor. Some of them can be preloaded into uncooperative processes.

With namespaced jails, we’ve arrived at the most popular current endpoint for workload isolation. You can do better, but the attacks you’ll be dealing with start to get subtle.

Language Runtimes

A limitation of jailed application environments is that they tend to be applied container- or at least process-wide. At high-volumes, allocating a process for every job might be expensive.

If you relax the requirement to run ordinary Unix programs, you can get some of the benefits of jails without fine-grained per-process security models. Just compile everything to Javascript and run them in v8 isolates. The v8 language runtime makes promises, which you might or might not trust, about what cotenant jobs can access. Or you could use Fastly’s Lucet serverside WASM framework.

From a security perspective, assuming you trust the language runtimes (I guess I do) these approaches are attractive when you can expose a limited system interface, which is what everyone does with them, and less attractive as a general design if you need all of POSIX.

Emulation

Here’s a problem we haven’t addressed yet: you can design an intricate, minimal whitelist of system calls, drop all privileges, and cut most of the filesystem off. But then a Linux kernel developer restructures the memory access checks the kernel uses when deref’ing pointers passed to system calls, and someone forgets to tell the person who maintains waitid(2), and now userland programs can pass kernel addresses to waitid and whack random kernel memory. waitid(2) is innocuous, you weren’t going to filter it out, and yet there you were, boned.

Or, how about this: every time a process faults an address, the kernel has to look up the backing storage to resolve the address. Since this is relatively slow, the kernel caches. But it has to keep those caches synchronized between all the threads in a process, so the per-thread caches get counters tied to the containing process. Except: the counters are 32 bits wide, and the invalidation logic is screwed up, so that if you roll the counter, then immediately spawn a thread, then have that thread roll the counter again, you can desynchronize a thread’s cache and get the kernel to follow stale pointers.

Bugs like this happen. They’re called kernel LPEs. A lot of them, you can mitigate by tightening system call and device filters, and compiling a minimal kernel (you weren’t really using IPv6 DCCP anways). But some of them, like Jann Horn’s cache invalidation bug, you can’t fix that way. How concerned you are about them depends on your workloads. If you’re just running your own applications, you might not care much: the attacker exploiting this flaw already has RCE on your systems and thus some access to your internal network. If you’re running someone else’s applications, you should probably care a lot, because this is your primary security barrier.

If namespaces and filters constitute a “jail”, gVisor is The Village from The Prisoner. Instead of just filtering system calls, what if we just reimplement most of Linux? We run ordinary Unix programs, but intercept all the system calls, and, for the most part, instead of passing them to the kernel, we satisfy them ourselves. The Linux kernel has almost 400 system calls. How many of them do we need to efficiently emulate the rest? gVisor needs less than 20.

With those, gVisor implements basically all of Linux in userland. Processes. Devices. Tasks. Address spaces and page tables. Filesystems. TCP/IP; the entire IP network stack, all reimplemented, in Go, backended by native Linux userland.

The pitch here is straightforward: you’re unlikely to have routine exploitable memory corruption flaws in Go code. You are sort of likely to have them in the C-language Linux kernel. Go is fast enough to credibly emulate Linux in userland. Why expose C code if you don’t have to?

As batshit as this plan is, it works surprisingly well; you can build gVisor and runsc, its container runtime, relatively easily. Once you have runsc installed, it will run Docker containers for you. After reading the code, I sort of couldn’t believe it was working as well as it did, or, if it was, that it was actually using the code I had read. But I scattered a bunch of panic calls across the codebase and, yup, that all that stuff is actually happening. It’s pretty amazing.

You are probably strictly better off with gVisor than you are with a tuned Docker configuration, and I like it a lot. The big downside is performance; you’ll be looking at a low-double-digits percentage hit, degrading with I/O load. Google runs this stuff at scale in GCE; you can probably get away with it too. If you’re running gVisor, you should brag about it, because, again, gVisor is pretty bananas.

Lightweight Virtualization

If you’re worried about kernel attack surface but don’t want to reimplement the entire kernel in userland, there’s an easier approach: just virtualize. Let Linux be Linux, and boot it in a virtual machine.

You almost certainly already trust virtualization; if hypervisors are comprehensively broken, so is all of AWS, GCE, and Azure. And Linux makes hypervising pretty simple!

The challenge here is primarily about performance. A big part of the point of containers is that they’re lightweight. In a sense, the grail of serverside isolation is virtualization that’s light enough to run container workloads.

It turns out, this is a reasonable ask. A major part of what makes virtual machines so expensive is hardware emulation, with enough fidelity to run multiple operating systems. But we don’t care about diverse operating systems; it’s usually fine to constrain our workloads to Linux. How lightweight can we a virtual machine if it’s only going to boot a simple Linux kernel, with simple devices?

Turns out: pretty lightweight! So we’ve got Kata Containers, which is the big-company supported serverside lightweight virtualization project that came out of Intel’s Clear Containers (mission statement: “come up with a container scheme that is locked in to VT-x”). Using QEMU-Lite, Kata gets rid of BIOS boot overhead, replaces real devices with their virtio equivalents, and aggressively caches, and manages to get boot time down by like 75%. kvmtool, an alternative KVM runtime, gets even lighter.

There’s two catches.

The first, and really the big problem for the whole virtualization approach, is that you need bare metal servers to efficiently do lightweight virtualization; you want KVM but without nested virtualization. You’re probably not going to shell out for EC2 metal instances just to get some extra isolation.

The second, more philosophical problem is that QEMU and kvmtool are relatively complicated C codebases, and we’d like to minimize our dependence on these. You could reasonably take the argument either way between gVisor, which emulates Linux in a memory-safe language, or Kata/kvmtool, which runs virtualized Linux with a small memory-unsafe hypervisor. They’re both probably better than locked-down runc Docker, though.

Firecracker

Lightweight virtualization is how AWS runs Lambda, its function-as-a-service platform, and Fargate, its serverless container platform. But rather than trusting (and painstaking tuning) QEMU, AWS reimplemented it, in Rust. The result is Firecracker.

Firecracker is a VMM optimized for security. It’s really kind of difficult to oversell how clean Firecracker is; the Firecracker paper boasts that they’ve implemented their block device in around 1400 lines of Rust, but it looks to me like they’re counting a lot of test code; you only need to get your head around a couple hundred lines of Rust code to grok it. The network driver, which adapts a Linux tap device to a virtio device a guest Linux kernel can talk to, is about 700 lines before you hit tests — and that’s rust, so something like 1/3 of those lines are use-statements! It’s really great.

The reason Firecracker (and, if you overlook the C code, kvmtool) can be this simple is that they’re pushing the system complexity down a layer. It’s still there; you’re booting an actual, make-menuconfig’d kernel, in all of it’s memory-unsafe glory. But you’re doing it inside a hypervisor where, in the Firecracker case, really you’re only worried about the integrity of the kvm subsystem itself.

We aren’t yet significant contributors to Firecracker, but it still feels weird talking the project up because it’s such a core part of our offering. That said: the team at AWS really did this thing the Western District Way:

The Firecracker VMM is tiny, easily readable, and deliberately implements the minimal number of concepts required to run a Linux server workload.
The VMM is written in Rust.
The VMM seccomp-bpf’s itself down to something like 40 system calls], several, including basic things like fcntl, socket, and ioctl, with tight argument filters.
Runs itself under an external jailer that chroots, namespaces, and drops privileges.

General Thoughts

Keep in mind, I think, that no matter how intricate your Linux system isolation is, the most important attack surface you need to reduce is exposure to your network. If you can spend time segmenting an unsegmented single-VPC network or further tightening the default Docker seccomp-bpf policy, your time is probably better spent on the network.

Remember also that when security tools designers think about isolation and attack surface reduction, they’re generally assuming that you need ordinary tools to run, and ordinary tools want Internet access; your isolation tools aren’t going to do the network isolation out of the box, the way they might, for instance, shield you from Video4Linux bugs.

It seems to me like, for new designs, the basic menu of mainstream options today is:

Jailing otherwise-unmanaged Unix programs with nsjail or something like it.
Running unprivileged Docker containers, perhaps with a tighter seccomp profile than the default.
Going full gVisor.
Running Firecracker, either directly or, in a K8s environment, with something like Kata.

These are all valid options! I’ll say this: for ROI purposes, if time and effort is a factor, and if I wasn’t hosting hostile code, I would probably tune an nsjail configuration before I bought into a containerization strategy.

Serve small with Fly.io and GoStatic

2020-07-27T01:00:00+01:00

Static websites are great for carrying unchanging content, be it assets, images, fonts or even, as in this case, an entire site. Well, I say entire site, but if you saw my last article, you'll know I recently rebranded a Maker organization and needed to deploy a "signpost" page pointing people to the new site. I want this signpost to have a tiny footprint so it will never cost anything to deploy.

Now, the thing with Docker images is that you don't really notice the layers of OS and applications that pile up in the background. Something as notionally simple as say running Apache HTTPD will still need an OS layer under it, no matter how minimal, you put the two parts together and the image size soon builds up. And you still have to add the content.

This is where something like GoStatic comes in. It's a small, self-contained web page server which can run in a bare Docker image - no OS, just the binary. As the author points out, the official Golang images can weigh in with as much as half a gigabyte of image. For GoStatic, the image is an unchunky 6MB.

Go GoStatic

So, how do you make use of GoStatic on Fly? Let's step though it now.

One thing you need to know is that by default GoStatic uses port 8043. So add -p 8043 to your fly init command when you create your project. That'll route traffic to port 80 and 443 to port 8043 on the application.

❯ mkdir examplegostatic
❯ cd examplegostatic
❯ fly init examplegostatic -p 8043
Selected App Name: examplegostatic

? Select organization: Dj (dj)

? Select builder: Dockerfile
(Create an example Dockerfile)

New app created
Name = examplegostatic
Owner = dj
Version = 0
Status =
Hostname = <empty>

Wrote config file fly.toml

We already have an index.html we want to serve, so our next stop is the Dockerfile. Delete the example contents and replace it with just two lines.

FROM pierrezemb/gostatic

COPY index.html /srv/http/index.html

And we are ready to deploy! Just run flyctl deploy:

Deploying examplegostatic
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 8043

Deploy source directory '/Users/dj/examplegostatic'
Docker daemon available, performing local build...
==> Building with Dockerfile
Using Dockerfile: /Users/dj/examplegostatic/Dockerfile
Step 1/2 : FROM pierrezemb/gostatic
 ---> 4569615e9ed0
Step 2/2 : COPY index.html /srv/http/index.html
 ---> b0b723d0cb24
Successfully built b0b723d0cb24
Successfully tagged registry.fly.io/examplegostatic:deployment-1595848012
--> Building with Dockerfile done
Image: registry.fly.io/examplegostatic:deployment-1595848012
Image size: 7.7 MB
==> Pushing Image
The push refers to repository [registry.fly.io/examplegostatic]
77bf40a52322: Pushed
3530b7ebed24: Pushed
deployment-1595848012: digest: sha256:d60567799841a4480e410acef113d33c1156eb960b94ae591931801089f61b1a size: 735
--> Done Pushing Image
==> Optimizing Image
--> Done Optimizing Image
==> Creating Release
Release v0 created
Deploying to : examplegostatic.fly.dev

Monitoring Deployment
You can detach the terminal anytime without stopping the deployment

1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v0 deployed successfully

Once deployed, all you need to do then is flyctl open and a browser will open and navigate to the site.

You'll also notice that this has been upgraded to an https connection. All that is left is to attach a custom domain to it and we're done.

Behind the scenes

So, what magic is going on here? Well, the Dockerfile in GoStatic explains a lot of it.

It uses a Docker multistage build to build our server binary in the first stage. Then it starts a new stage from scratch, literally using the command FROM SCRATCH. This says that there is no base image, just start building on top of nothing, an empty image. The rest of the GoStatic Dockerfile creates a passwd file so there are some usernames to work with and copies over the GoStatic binary.

And then it all hands over to our own Dockerfile. GoStatic serves files out of /srv/http so we copy over our index.html to that directory. And that's it. Everything else is managed by Fly, the build, the deployment and the upgrading to an https connection. There used to be a version of GoStatic which would handle HTTPS connections and certificates, but that functionality has been retired now servers like Caddy exist. On Fly, the lack of HTTPS support means it's simple to just let Fly take on the HTTPS work for you.

A Small Squeeze

One last tip. Fly deploys new applications with 512MB of RAM and about a quarter of a virtual CPU. It's called a micro-2x firecracker VM. But, we're doing so little here, we could scale the VM size down. Let's look at the scale settings:

~/examplegostatic
❯ flyctl scale show
     Scale Mode: Standard
      Min Count: 0
      Max Count: 10
        VM Size: micro-2x

And now we can set the vm size:

~/examplegostatic
❯ flyctl scale vm micro-1x
Scaled VM size to micro-1x
      CPU Cores: 0.12
         Memory: 128 MB
  Price (Month): $2.670000
 Price (Second): $0.000001

If you want to find out about the other VM sizes, run flyctl platform vm-sizes.

Wrapping up

We've got ourselves a tiny static web server and deployed it to Fly.io, and as a bonus, shrunk the VM's footprint to match it and save running costs. Enjoy!

Hugo's There - Flying with Hugo and Caddy

2020-07-21T01:00:00+01:00

There I was wondering what to do about a website for a new community venture I was running where I thought, yes, let's generate the site with Hugo, serve it with Caddy and run it all on Fly. Why Hugo and Caddy? Well, they both have good reputations as Go-based tooling thats compact and powerful, so let's go make and host a site...

Fly First

I'll need to be able to reference what the app name, and consequently site name, will be. Because of that, I'm going to start by initializing my Fly deployment:

$ flyctl init makeronicc --dockerfile -p 80

Selected App Name: makeronicc

? Select organization: Dj (dj)

New app created
  Name     = makeronicc
  Owner    = dj
  Version  = 0
  Status   =
  Hostname = <empty>

Wrote config file fly.toml

$

Select your own app name or let the system generate one for you; it won't matter as we're going to front this set-up to a custom domain. We're going to be using a Dockerfile (hence --dockerfile) and our server will operate on port 80 (-p 80).

The Hugo Configuration

The site doesn't have much content and a stroll through the Hugo Quickstart will get us a front page and a blog article built in no time at all. It boils down to installing hugo, then installing a hugo theme and finally creating a config.toml file. Here's ours:

baseURL = "https://makeronicc.fly.dev"
languageCode = "en-us"
title = "Makeroni"
theme = "ananke"
[params]
site_logo = "/images/makeroni-logo-small.png"

That's enough for our basic site to build running hugo which generates all the static files. Running hugo server -D lets us browse it locally on localhost:1313. The one thing to note? We'll be serving this site up on the makeronicc.fly.dev domain for now.

Docker And Hugo

Next we want to get Docker to do the build work. Time to make a Dockerfile.

There are various Hugo docker images out there but the one I like is klakegg/hugo on Docker Hub. As well as having Docker images that can be used for running Hugo as a Docker container, there's an ONBUILD image. This is designed to be a stage in a multi-stage Docker build. It's run as part of the pipeline, but then results can be copied from its image to a new, cleaner image, less all the build tools.

So our Dockerfile starts like this:

FROM klakegg/hugo:0.74.0-onbuild AS hugo

That, when built with Docker, will load up the image and the current working directory contents, run hugo over it and deposit the results in /target in the hugo image. Hugo build, done. Now, let's talk about serving it up.

Caddy Hack

Now we come to Caddy, which is a great web server "built for now" - It has integrated handling of obtaining and managing Let's Encrypt certificates so running an HTTPS site becomes super-simple. There's only one issue - Fly already does all that certificate management for us, so although we want Caddy because it's compact and easy to work with, we're going to want to turn off Caddy's own certificate system.

Caddy is configured in a number of ways, JSON, API or the Caddyfile. I use the Caddyfile for this as its more human-readable. But now a public service announcement:

When you search for Caddy and, well, anything at all, when you get to a result, scroll to the top of the page to make sure you aren't on the Caddy 1 documentation. Caddy 2 is the current version but the google-juice for Caddy 1 documentation is still super high and the two are so similar yet different, it can be terribly frustrating to keep landing on the wrong docs.

Right, back to creating our Caddyfile. Most of what I just talked about can be summed up in one opening block.

{
    auto_https off
}

That turns off all the certificate management. Now we can tell Caddy to serve files for our makeroni.cc domain.

http://makeronicc.fly.dev {
    root * /usr/share/caddy
    file_server
}

Notice that this is just for the http protocol connections. That's because, once the TLS connection has passed through the Fly edge, it travels on the encrypted Fly network as a normal HTTP request.

That's the Caddyfile created. Now to pull the two parts together in the Dockerfile.

Adding Caddy

At the moment, our Dockerfile simply brings in and runs the Hugo static generator at build time. We need to take the results of that and put it into a Caddy docker image. There are official Caddy images, so I'll use one of them:

FROM klakegg/hugo:0.74.0-onbuild AS hugo

FROM caddy:2.1.1

Docker will now start with this Caddy image. Our Caddyfile says it will serve files out of /usr/share/caddy so we'll want to copy the files from our Hugo build over to there by adding:

COPY --from=hugo /target/ /usr/share/caddy/

The --from points to the named image we created at the start with AS hugo. Now all we need is to put the Caddyfile in place.

COPY ./Caddyfile /etc/caddy/Caddyfile

Ready To Fly

We're ready to publish the site. Run flyctl deploy and watch as the Hugo site is built, copied into a Caddy image, that image is then flattened and despatched to a Fly firecracker node. You don't have to worry about that though, just run flyctl open and your browser will open on your application.

One thing worth noticing is that, although we only configured http://makeroni.fly.dev, it's being automatically upgraded to https: to secure the connection.

But we aren't done yet. Remember we wanted the site to be provisioned on makeroni.cc.

Fly Domain

The first step is to get the DNS system to point makeroni.cc to the IP address of makeroni.fly.dev. I can get the IP Address by running flyctl ips list.

$ flyctl ips list

TYPE ADDRESS                              CREATED AT
v4   77.83.141.28                         2020-07-07T21:26:36Z
v6   2a09:8280:1:9f04:7aa6:a706:a3d7:ccba 2020-07-07T21:26:36Z

We want the V4 address. Now, I need to go to the registrar of the DNS entry, in my case NameCheap, and get to the DNS management pages, specifically the Advanced Management page of that.

It's there I can add an A record. A @ 77.83.141.28 (The @ goes in the host column on Namecheap).

Save that and let it propogate and then go to http://makeroni.cc and you should see your site. If you follow any links though, you'll notice you are back on https://makeronicc.fly.dev. That's because Hugo generated all the links with that address.

That's easy enough to fix (we'll get back to it in a moment), but there's another more important thing to look at.

If I try to go to https://makeroni.cc and I get an error saying the connection is not secure. I haven't created a TLS certificate for the domain.

The quickest way to do this is to add an AAAA record in the same way I added an A record. The AAAA record in a DNS record should point to the IP V6 address for the host; it's up in the flyctl ips list output too. So I add AAAA @ 2a09:8280:1:9f04:7aa6:a706:a3d7:ccba to the DNS. With that in place and propagated I can now go and request a certificate.

$ flyctl certs create makeroni.cc
Hostname = makeroni.cc
Configured = true
Issued =
Certificate Authority = lets_encrypt
DNS Provider = enom
DNS Validation Instructions = CNAME _acme-challenge.makeroni.cc => makeroni.cc.2yz0.flydns.net.
DNS Validation Hostname = _acme-challenge.makeroni.cc
DNS Validation Target = makeroni.cc.2yz0.flydns.net
Source = fly
Created At = just now
Status = Ready

And the traffic could flow securely.... Except there's one last change we need to make to the Caddyfile.

Caddy Changes

Remember I set up the Caddyfile with

http://makeronicc.fly.dev {
    root * /usr/share/caddy
    file_server
}

Well, I'm not serving the files on that URL now so I'll need to change that to match our custom domain:

http://makeroni.cc/ {
    root * /usr/share/caddy
    file_server
}

Now it'll respond to requests for files from the makeroni.cc domain... but what if I want to make sure that people who accidentally access the old makeronicc.fly.dev site end up in the right place. For that, another rule in the Caddyfile is needed:

http://makeronicc.fly.dev {
    redir https://makeroni.cc/
}

Now those old requests will head to our new server. Redeploy the image to Fly and everything is ready to roll.

Wrapping Up

We've gone through configuring a multistage image build which generated a static Hugo site, then loaded it into an image with Caddy. We've configured Caddy for development deployments on Fly and then we've got ourselves a custom domain set up, and made that work with TLS certificates from Let's Encrypt and a small modification to Caddy's setup.

Flyctl Evolved - Fly Changelog

2020-07-10T01:00:00+01:00

There's a new flyctl (v0.0.137) available for your command line, with cleaner commands and extra helpers. Find out more about it in the Changelog.

This flyctl release brings in some big changes in the command structure as we move to an app-centric command style. What does that mean? Well, the apps subcommand is being deprecated; we've kept it in place for this release but now all its commands have top level commands of their own:

Was	Now
apps create	init
apps destroy	destroy
apps list	list apps
apps move	move
apps restart	restart
apps resume	resume
apps suspend	suspend

The move, restart, resume, suspend commands also now take an appname as their last argument. The status command has also followed suit in this change, so you can now type flyctl status appname rather than flyctl status -a appname - The -a option will remain supported.

The list command at the top level has been around for a while and has advantages over the older list command: you can match appnames with fragments of text and filter on status or organization. Talking about organizations, flyctl list orgs will list the organizations your account has access to.

We've also made some small usability changes in how you initialize an application. The init command offers you a selection of builders or the chance to use a Dockerfile. If you don't have one, init will create one for you with a simple hello world deployment to get you going. If you don't want the example generated, use --dockerfile when running init. And, yes, you can still specify a builder with --builder, that's not going away.

Other changes

The move command will now tell you what organization your app is in so you know where you are moving it from.
Setting and unsetting secrets on suspended deployments is now blocked.
Setting and unsetting secrets will start a deployment monitor. Add the --detach flag to return before starting the deployment monitor.
If you are setting a secret on an undeployed app, then we don't start the deployment monitor, so no need to --detach.
The info command now supports --host which will display just the host name, along with (-n / --name ) which just displays the appname. These flags are designed to make it easier to script with flyctl.
The version command now outputs just the bare version number - add --full to get the detailed version/commit/date information if you need it.

Platform changes

It's not all been flyctl changes. There's a fix for a problem with cookie headers and HTTP/2 which is now in place. Also, if your application parses headers, you'll find that a Via header has been added to enable applications to trace their route through the Fly edge.

9th July

flyctl: Version 0.0.137 released

New top level commands
Apps subcommand deprecated
--host added to info command
version displays bare version number, --full displays full details
secrets setting and unsetting will follow deployment where appropriate
secrets now supports --detach
move command now prompts with current organization
init command now prompts with list of builders or option to create/use Dockerfile
init command supports --dockerfile flag to completely skip builder query

Fly Platform/Web

Fixed a bug related to incoming HTTP/2 request cookies sent as multiple headers. Incoming HTTP/2 connections can present multiple cookie headers. The headers were sent on as is when the Fly edge downcast the connection to HTTP/1 which is used within the Fly network. Some servers could not handle the multiple headers though. Now, the downcasting process concatenates the multiple cookie headers into a single cookie header.
Added the Via header to both http requests and responses
Boosted the performance of the Optimizing Image phase of deployment by making better use of existing identical images.

Run Apollo Server Close to Your Users

2020-07-01T01:00:00+01:00

Fly.io can run API servers close to users. It's kind of like a CDN for your GraphQL server. Here's a guide to building an edge GraphQL Server with Apollo and Redis.

I'm a newly minted GraphQL convert. We built Fly on top of GraphQL and the experience turned me into a shameless cheerleader. An API format with static typing? That's my jam.

(If you don't care for JAMStack puns you can just go read our guide on building an Edge GraphQL service with Apollo)

Speaking of jam, you've probably used application stacks that push content close to users. Hosting JavaScript and markup on a CDN can help make an app snappy.

You can also apply CDN like infrastructure to a GraphQL API to get a nice speed boost. All you need is a way to run API servers and an application cache close to users.

Which is why we built a platform to run API servers close to users and paired it with a global Redis cache service. It's a great place to run Apollo Server, for example, with its cache capabilities and first class Redis support.

We built a demo GraphQL API based on the Open Library REST API. Read the guide or check out the source code.