Run Docker and Nomad on Android - truly serverless mobile compute

Karim Jedda -

I run a lot of servers and many workloads around the world on my own mini-cloud. It's distributed amongst Hetzner, Azure and AWS. Some of these workloads are better located on the edge, especially when there are sensitive credentials or cost concerns or whatever. Remember, the cloud is just someone else's computer.  

I have been contacted by a friend who's running a pretty cool use case and I suggested to use Docker on many mobile phones to do what he wants to do. The whole mobile farm is running and is managed by Nomad just like a regular cluster, only that it's really mobile. More on that probably in another article.

I really believe that mobile compute like this is the future. Just imagine how much compute power can be accumulated easily and quickly: everyone has a mobile phone right? But first things first, you certainly came here for a guide and not my technical ruminations.

Note: Raspberry pi's could also be an option. But they involve too many cables, they don't come with a screen, storage or battery and aren't compact or optimized like a mobile. Besides, a Raspberry pi looks sketchy to people who don't know what it is, really. Mobile terminals is where it's at.

Get an Android phone

I'm not particularly a fan of any brand of mobile phones, they're just machines, the condition for something like this to work though is to get something easy and forgiving to tinker with, hence Android.

It should be possible to do something like this on an iPhone as well, I'm sure, although with a bit more effort and I don't want to be the one that publicly documents something like that. The ultimate goal is to get a configurable compute running on the phone by avoiding the "app" way, which 10 times out of 10 sent me yak shaving for months without end (Kotlin? Flutter? Java? React Native?). No, I'm a man of simple tastes.

There is a very handy utility to debug any android phone you have physical access to. It's called adb and it will be a great help, I paid for the device and I am in Europe/Germany, you better believe I'll be making it do what I want it to do (without Java too, thank you)

To get setup, install adb, fastboot and throw in scrcpy in the mix too. I'm doing all these operations from a vanilla Ubuntu.

Once setup, enable developer mode, USB debugging and OEM Unlock on your Android phone. There are enough guides for this online.

Note: Enable OEM Unlock only for Test 1. It isn't necessary for Test 2.

Test 1 - Difficult : Root the Android phone

In it's pristine state, the phone is like a computer you'd get at your #day-job, it doesn't allow you to install anything outside of a specific catalog of apps, you don't get sudo rights (aka footgun) and if you try to do anything sketchy, you're at the risk of breaking (bricking) the device or locking yourself out from the app stores. This is of course to prevent hackers from getting access to your phone and syphoning your private data. That's an activity reserved for other, more honest entities.

Bad things tend to happen though when you're expected to program or do anything on a device that was intended for content consumption in the first place: little documentation, a lot of trial and error, wrong documentation... Solitude.

One common way to get those sudo rights on Android is to root the device, meaning removing any safeguards and locks.

Rooting the device (aka jailbreaking it), means installing some "stuff" on it to remove the shackles keeping us from being 10x mobile developers and doing fun things like installing a custom kernel or removing pre-installed applications.  This is a risky operation and depends on the device, if you don't know what you're doing you can completely destroy your phone. Only do it on spare, expendable phones and always think before running a command or clicking a shiny download button.

I suggest you check other articles by people smarter than me on how to root your specific phone. I had success with a tool called Magisk. The high level steps are to get some file, copy it on the phone, put your phone in recovery mode and then flash the file using Magisk/Odin.

One huge warning: Take your own images and run any software you download on a separate computer. You've been warned.

I got a Samsung A22 which was discounted for 120€, your mileage may vary. The only grief I have with this device is that the battery cannot be removed easily. But thankfully, the EU has my back for future devices. Removable batteries will definitely extend the lifespan of the devices.

Once that is done, you're officially in the Wild West with a rooted phone, and the real fun can start. (still no Java in sight btw) Basically just install stuff on it, you're good to go!

It is utterly confusing though and there are no recipes, just mashed up text and guides, and most of the guides are not correct. They just replace the phone model with the model you're searching for (an attempt at SEO) and call it a day. Also, it is weird that you get redirected to sketchy download sites where you have to download 4 GB firmware files to only use a 32MB Boot.img file. Ultimately, I got it to work, but it felt dirty so I reset the phone to its initial state.

There really has to be a better way.  

Useful links:

Trial 2 - Easy: Running a VM on Android

At first this got me thinking. Why would I want to run a VM on Android? Well because I can get root within that VM and essentially run what I want on it directly, keeping the host clean.

There are three stars for this trick: Termux, QEMU and Alpine Linux.

Termux is a terminal emulator and Linux environment, we can use to run different commands and tools. It's fantastic software  and I recommend installing it from F-Droid. By the way, it has nothing to do with tmux!

QEMU is an emulator. It's fantastic technology, read a bit about it, a great place to start is their main website.

Alpine Linux is a version of Linux that comes from a secret penguin hideout in the Alps. Obviously, what I said is not true, Alpine is a lightweight Linux distribution, just like Ubuntu, sort of.

Start of by installing Termux. Download it from F-Droid and then install it on your mobile phone using adb. Then start it and play around with the interface a bit. Look up how to install things and run things. I find it's a fantastic piece of software. It'll get a bit clunky to type on the phone with your fingers, so I recommend using a tool to simplify the next steps.

If you have many devices or don't like typing long commands on the phone, I'd recommend going with https://github.com/Genymobile/scrcpy to display the phone screens on your computer and send keystrokes to multiple devices at the same time.

scrcpy --max-size 1024 --legacy-paste
Mirror device to desktop PC

The legacy paste flag allows you to copy paste from your clipboard directly into the phone. Neat? Neat!

Control your android phone from the comfort of your desktop PC

Installing the VM

Now you can start installing the real stuff.

You can pipe all these commands through adb, with ssh or scrcpy but you need to run them within Termux. 

pkg install qemu-utils qemu-common qemu-system-x86_64-headless
Installing QEMU and dependencies
mkdir alpine && cd alpine
wget https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/x86_64/alpine-virt-3.15.4-x86_64.iso
Fetch alpine iso
qemu-img create -f qcow2 alpine.img 4G
Create a disk to hold the VM data
qemu-system-x86_64 -machine q35 -m 1024 -smp cpus=2 -cpu qemu64 \
  -drive if=pflash,format=raw,read-only,file=$PREFIX/share/qemu/edk2-x86_64-code.fd \
  -netdev user,id=n1,hostfwd=tcp::2222-:22 -device virtio-net,netdev=n1 \
  -cdrom alpine-virt-3.15.4-x86_64.iso \
  -nographic alpine.img
Start the VM

You can login with the user root and no password, we will set it shortly. 

# just press enter till done
localhost:~# setup-interfaces 
localhost:~# ifup eth0
Setup the network configuration
localhost:~# wget https://gist.githubusercontent.com/KarimJedda/c63978db5ea2b3d213962d842de373e7/raw/3bef5e82c8b1906833eda8e92e3df464f1d7d741/answerfile
Configuration file for QEMU, I recommend making your own
localhost:~# sed -i -E 's/(local kernel_opts)=.*/\1="console=ttyS0"/' /sbin/setup-disk
I am still figuring out what this does
localhost:~# setup-alpine -f answerfile
Apply your answerfile, it's like a default values for a cloud-init if you want, I guess

This will ask you to setup your root password and format the disk we created earlier.

The setup should be complete when you see "Installation finished. No error reported". There might be another thing that will display a loading bar with no indication of what is going on, I love those, totally help with my anxiety. All in all this process might take from 10 to 30 minutes depending on the capabilities of the device.

The real last message is "Installation is complete. Please reboot." Läuft! Thank you, next.

You can now shutdown the VM using the poweroff command, don't reboot yet, what's QEMU gonna do anyway?

Installing and running docker

Alright after all that yak shaving, let's get to work.

qemu-system-x86_64 -machine q35 -m 1024 -smp cpus=2 -cpu qemu64 \
  -drive if=pflash,format=raw,read-only,file=$PREFIX/share/qemu/edk2-x86_64-code.fd \
  -netdev user,id=n1,hostfwd=tcp::2222-:22 -device virtio-net,netdev=n1 \
  -nographic alpine.img
Run your newly configured VM (notice, we don't need the iso anymore)

Now witness the glory of scrolling text and wait for the VM to boot up and login with root and the password you provided (What do you mean you forgot it?)

You can also put the above snippet in a bash file and adapt it to your needs (more memory, other port forwards, whatever) and now we can install docker. 

alpine:~# apk update && apk add docker
alpine:~# service docker start
alpine:~# rc-update add docker
Install docker 

From here on out, it's open bar. You did it! 🎉

Docker running on android

But why stop here? You can try installing podman, nomad, LXC, heck even QEMU again. Yes, Kubernetes too. Plug a Sim card in for the lulz and see what happens! (I only tried nomad and docker so far!)

The wonders of the world belong to the explorers and builders. Try stuff out and please write about it.

What's left now? Getting access to the camera, microphone and other hardware stuff within the VM and do funny things. What is that you say? Not possible?

Resources used:

Closing thoughts

Once all of this is understood, you can automate it with ansible through a hub for an extra fun challenge. I've had loads of fun with these pocket servers frankly.

Is that a Kubernetes cluster in your pocket or are you just happy to see me?

However, we need a way to run Linux on the devices directly. I don't need the apps to be honest.  

It's quite inefficient now and I admit it, since what we're doing is Android > Termux > VM but it's okay I guess, it works. I'm pretty sure someone can fine tune this and run a cloud provider completely on top of mobile phones. That would be absolutely hilarious! How would we call this? Serverless-less!

There is an estimate of 15 billion mobile devices in the world (3 billion of them run Java). By pooling their resources we could surely do some fun mind bending stuff. I certainly do with 30.

Without Java!

PS: Did you know this works on some car infotainment systems too? ;-)

Random stuff

# Copy file to device storage
adb push Magisk-24.2.zip '/data/local/

# Manually install apk
adb install Magisk-v24.2.apk

# Start device in download mode
adb reboot download
adb reboot recovery
adb reboot bootloader
fastboot flash boot boot.img
flashboot flash recovery recovery.img

# Termux can run rsync and crontab

# Install apk from edge
apk add nomad --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community

# Mix I listened to while doing this 
https://www.youtube.com/watch?v=G9UwNdv5oMA
Commands reference