Increase Root Volume Size for AWS EC2 Linux Instances

It turns out it is extremely simple to increase volume size for AWS EC2 Linux instances. In this article we use Ubuntu 20.04 as an example to show how it works in simple 3 steps, without restarting the instance.

  1. Increase volume size – Go to AWS console; find the volume used for the instance as the root device; Choose “Modify volume” action item; increase the size to the desired number
  2. Log into the AWS machine, type “lsblk” to verify the root device size has been increase. Also confirm that the partition size remains the same as before.
  3. Expand partition size to fill up the drive. For example, if root device is /dev/xvda and root partition is the first partition, you would run “sudo growpart /dev/xvda 1”. Run “lsblk” again to verify partition size.
  4. Resize filesystem to use the new space. For the previous example, one would run “sudo resize2fs /dev/xvda1”. Run “df -h /” to verify.

Viola! You are done!

Install Full Ubuntu on Portable USB Drive

Background

My goal is to install fully updateable Ubuntu 20.04 onto a USB stick, so that I can boot it up with any Intel-based PC’s or laptops. However, due to what I considered a bug in Ubuntu, this is actually harder than it should be. So I wrote down this blog in the hope it might help others, as well as my future self, in the similar shoes.

Note that the objective is different from so-called LiveUSB ubuntu with persistency, where Ubuntu OS itself will remain as a static ISO image and updates are added on-top in a separate persistent partition. My goal is to install a standard Ubuntu OS on a USB disk, which can be updated and upgraded just like normal PC case, except that a) it is on a portable USB drive or disk and b) it is portable across different PC’s. I suppose this setup gives longer life span of the installation, which potentially allows you to even upgrade your OS later.

In the following steps, I will also show an optional feature which creates an encrypted home directory.

Assumptions and Prerequisites

  • You need Intel x86_64 PC
    • We assume it support UEFI and GPT partitions which are standards for all recent ones
  • A USB drive that holds Ubuntu ISO image for installation, a.k.a. the installation media drive. This needs to have 4GB minimum size.
  • A second USB drive or disk that will hold installed Ubuntu OS, a.k.a. the installation target drive. This one needs 16GB minimum size

Step 1 – flash Ubuntu ISO image to the installation media drive

I will not repeat the process here. Please refer to many pages below.

Step 2 – Prepare the partition table on the target drive

  • Insert installation media drive into PC.
  • Interrupt normal booting sequence and choose the media USB drive as the boot device
    • Different PC have different process to do this. On Lenovo PC, one has to press ENTER on bootup, and then press F12 to select boot device
  • Select “Try Ubuntu” when presented the option
  • Insert target USB drive
    • Identify which drive is target USB drive by examining the output “lsblk”
    • In most case if you follow the instructions exactly, it would be “/dev/sdb”
  • Once Ubuntu is up and running, start a terminal and type “sudo gparted /dev/sdb” (replace “/dev/sdb” with the right usb device you have for the target USB drive)
    • create GPT partition table
      • click “Device”/”Create Partition Table …”
      • select “gpt” as partition table type
      • See Pic #1 below
    • create 100MB fat32 partition as ESP partition
      • Click “Partition”/”New”;
      • Enter “100MB” as size and select “FAT32” as file type
      • See Pic #2 below
    • set “esp”, “boot” attributes to the new ESP partition
      • Apply changes to actually crate the partition
      • Select the ESP partition and then select “Partition”/”Manage Flags”
      • In the pop-up window, select “esp” and “boot” flag
      • See pic #3 below
    • create an ext4 partition that takes the rest of space for root partition
      • See pic #4 below
    • (optional) if you like to have encrypted home partition, create an ext4 root partition with size of 10GB or more, and leave the rest free space open for encrypted home petition later.
      • Pic #5 shows the partition table at this step.
Pic #5 – after create ESP and root partitions

Step 3 – Install Ubuntu

Once we finish the above step, quit gparted and we are ready to install Ubuntu into the target USB drive.

  • click “install ubuntu” icon on the desk to to start installation
    • select “something else” in partition page. See Pic #6 below.
    • select the ESP partition on the target USB drive as the “ESP” partition. See Pic #7 below.
    • select root partition and mount as “/”. See Pic #8 below.
    • (optional) Create encrypted physical volume
      • Select the free space left during creating partitions
      • Click “+” to create a new partition/volume
      • select “encrypted physical partition”.
      • See Pic #9 below.
    • wait for a while, select “/dev/mapper/sdb3_crypt” as “/home”. See Pic #10 below
    • Finish installation.

Step 4 – Fix EFI on target USB drive

At this point you might get an illusion everything is working, because if you reboot the PC you will be able to select either Ubuntu or Windows to boot up, and they all work. However, there are 2 very serious problems

  • If you boot into BIOS and select the USB disk as boot device, it won’t work.
  • Even worse, your PC is likely not able to boot up Windows either if you remove the target USB disk.

The reason for these problems is that, despite we told Ubuntu installer to use the ESP partition on target USB disk, it still uses the ESP partition on PC built-in disk, and thus screws everything up.

So the first thing we need to do is to install Ubuntu loader into USB ESP partition and install grub into USB disk

  • Reboot into BIOS firmware and select “ubuntu” as the boot target
  • switch EFI mount
    • Type “lsblk” to verify that ESP partition on PC built-in drive is mounted
    • umount it, “sudo umount /boot/efi”
    • mount the right one, ” sudo mount /dev/sda1 /boot/efi”
  • install grub boot on USB EFI partition
    • “sudo modprobe efivars”
    • “grub-install -d /usr/lib/grub/x86_64-efi –efi-directory=/boot/efi/ –removable /dev/sda”
    • Note “–removable” flag is important in the above command as it allows USB to boot on any Intel-based PCs
  • Reboot into BIOS firmware and select target USB as boot device. It should work now.
    • You can try the USB disk on other PCs, and it should work as well.

Step 5 – Restore PC boot loader

Now let us fix PC does not boot up problem (if you encounter this)

  • Boot up the PC into BIOS and select the newly made Ubuntu USB drive as boot device. This will boot into Ubuntu.
  • Open a terminal
  • Remove Ubuntu from PC EFI partition, and PC will boot up windows again
    • Mount host EFI partition as /mnt, “sudo mount /dev/nvme01n1p1 /mnt”
    • “cd /mnt/EFI/”
    • “sudo rm -rf ubuntu”
  • NOTE: if your PC was already installed with grub (e.g., your PC is a linux system), you should fix the problem by removing target USB Ubuntu entry in /boot/grub/grub.cfg file instead.

Clone a Website Behind Login with WinHTTrack

I have an very old Fedora server that stopped active running almost 10 years ago. I need to backup its data for archive and physically dispose it.

It used to run wiki site based on moinmoin. If I just back its static files plus database files, chances are I will never be able to re-active those wiki pages and see them again. So I thought a better idea is to clone the web pages and turn them into static local web site, which I can still access easily.

It turns the journey is more complicated than expected. It took me more than a couple of hours to finally nail it down. So I figure it is worth a post here.

The Software

A quick search shows HTTrack seems to be best software for this job. It has a GUI version for Linux, called WebHTTrack. It also has a windows version called WinHTTrack. During my fiddling around, I ended up using WinHTTrack. In retrospect, the method used here should also work for WebHTTrack.

The Challenge

WinHTTrack is generally user friendly. The biggest problem is moin wiki requires a user login to view the content.

The cookie capture method (“Add URL” followed with “Capture URL”) does not work for moin wiki. Only first page works. Later pages still returns “You are not allowed to view this page”

The Solution

Step 1 – prepare cookies.txt file

  • Install and launch firefox browser
  • Install cookies.txt extension
  • Log into the web site and start browsing
  • Click on “cookies.txt” extension and export cookies.txt for the current site.
  • Also note the firefox user agend
    • Clock on top-right “settings” icon
    • Then click on “Help”/”Troubleshooting Information”
    • Note the “User Agent” string. We will use it later.

Step 2 – run WinHTTrack once

  • Start WinHTTrack and set up the project normally without worrying about login
    • Specifically, you don’t need to use “Add URL” button to do anything special. Just type in the URL in the text input area.
  • Click on “Finish” to start downloading
    • It will warn the site seems empty. That is expected.

Step 3 – copy over the cookies.txt

  • Copy the exported cookies.txt file to the top level directory of the local clone.
    • For example, if HTTrack working directory is “%USER%\Documents\httrack-websites” and your project name is “erick-wiki”, then the destination directory is “%USER%\Documents\httrack-websites\erick-wiki”

Step 4 – run it again with proper options

A few options had to be set properly

  • Click on “Set options …” and a new pop-up window shows up
  • On “Spider” tab, set “Spider” to “no robots.txt rules”
  • On “Browser ID” tab, set “Browser Identity” to the “User Agent” of Firefox browser noted in step 1
  • On “Scan Rules” tab, click on “Exclude links”, another pop-up windows shows up
    • Here you can exclude the files you like to clone
    • Most importantly, you need to exclude the “logout” URL. Otherwise cookies.txt will be updated/deleted and you will note be able to continue cloning
      • In my case, the URL will end with “?action=logout”. So I set “criterion” to “Links containing:”, and set “String” to “action=logout”
  • (Optional) On “Experts Only” tab, change “Travel mode” to “can go both up and down”
  • (Optional) On “Flow control”, set “number of connections” to higher number for faster cloning. Note: your website may have surge limit as Moinmoin does, in which case you will need to turn off this feature on the web server in order to increase bandwidth.
  • Once all set, click on “Next” and “Finish” to start cloning.

That is it!

Build, Flash and Un-Flash AOSP Image on Pixel Phones

In this post we will go through a full cycle of building a custom AOSP image, flashing it on a Pixel phone, and then restoring to original stock image. We will use Pixel 4 (codenamed as “flame”) as the device example. We will walk through in a succinct manner without detailed explanations.

Prepare the Device

  • Note the Android build number
    • Click “Settings”/”About Phone”.
    • In my case it is “QQ3A.200805.001”
  • Enable developer options
    • click on “Build Number” row 7 times to enable developer options
  • Enable “OEM unlocking”
    • Toggle “Settings”/”System”/”Advanced”/”Developer Options”/”OEM unlocking”

Build AOSP Image

Prepare the host

My host is Ubuntu 20.04. Generally we are following this guide from Google. Below is a quick gist.

sudo apt-get install git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig
sudo apt install python2    #repo needs python
sudo usermod -aG plugdev $LOGNAME
apt-get install android-sdk-platform-tools-common   # install udev rules and adb

Download AOSP source

Generally we are following this guide from Google. Below are some specific steps.

  • Find exact branch/tag that matches device current build.
    • Go to the Android build tag page
    • Search for the build number noted at the previous step
    • Find corresponding AOSP tag. In my case, it is “android-10.0.0_r41”
  • Check out this branch
mkdir android-10.0.0_r41-shallow
cd android-10.0.0_r41-shallow
repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-10.0.0_r41
repo sync  --force-sync --no-clone-bundle --no-tags -j$(nproc)

Fetch Proprietary Binaries

  • Go to Google proprietary binary page
  • Find the section corresponding to your device (Pixel 4 in my case) and the build number (“QQ3A.200805.001”)
  • Download tarballs from Google and Qualcomm
  • Untar them under the root directory of AOSP source tree
cd ~/work/android/android-10.0.0_r41-shallow
tar xzf ~/Downloads/google_devices-flame-qq3a.200805.001-a7a5499d.tgz
tar xzf ~/Downloads/qcom-flame-rp1a.200720.009-1157dde5.tgz
./extract-google_devices-flame.sh
./extract-qcom-flame.sh

Build images

source build/envsetup.sh
lunch aosp_flame-userdebug
m

Note “flame” is the codename for Pixel 4. If you have a different device, you should use a different target. See Pixel factory image page for device codenames.

Command ‘m’ starts the build process, which can take very long. My laptop has AMD Ryzen™ 7 Pro 4750U Processor (1.70 GHz, up to 4.10 GHz Max Boost, 8 Cores, 16 Threads, 8 MB Cache) and has a fast NVME SSD disk. It takes about 2.5 hours to finish the build.

Flash AOSP Built Image

  • On the phone, enable USB debugging
    • Toggle “Settings”/”System”/”Advanced”/”Developer options”/”USB debugging”
  • On the PC in the same terminal where you just build the AOSP image
adb devices             # it should show the phone device attached over ADB
adb reboot bootloader   # reboot to fastboot
fastboot devices        # it should show the phone device attached over ADB
fastboot flashing unlock
fastboot flashall -w

During “fastboot flashing unlock” step, you need to following instruction on the screen, press up/down volume button to select “unlock bootloader”, and press power button to commit.

If things go smoothly, phone will reboot a couple of times and eventually boots into the freshly built AOSP image with root access.

Restore to Stock ROM Image

After you played enough with the AOSP image, you may want to return to the original retail state. Google has made it easy recently.

  • Open Chrome browser and go to “https://flash.android.com”
  • Allow browser to connect to the device
  • Go to Pixel 4 factory image page and find the desired build number to flash
    • Just for fun, I’m selecting Android 11, build number RQ1A.210205.004.
  • Select “Wipe device” and “Lock bootloader”
  • Click “Confirm” to start the process.

What If Things Go Wrong?

Things can go wrong when phone does not boot up anymore. It can be due to build AOSP build, or something due to mismatch between AOSP image and baseban/firmware version (BTW, this is the reason why this guide asks you to download the same version of AOSP that matches existing build).

unzip flame-qq3a.200805.001-factory-d93c74e6.zip
cd flame-qq3a.200805.001/
fastboot devices   # make sure phone is connected
./flash-all.sh 

(Bonus) How to Add a System Privileged App

Before Android 10, you simply re-mount /system as rw and push apk into /system/priv-app/

adb root
adb shell remount -orw,remount /system
adb push my-app.apk /system/priv-app
adb reboot    # to complete the app installation

Since Android 10, /system is mounted as root and cannot be re-mounted as rw. Instead you will need to use “adb remount” to enable overlay FS for any modification.

adb root
adb remount     # if this is first time, "adb reboot" to disable verity first
adb push my-app.apk /system/priv-app
adb reboot      # to complete the app installation

Often times you need to assign PivilegedOrSystem level permissions to system apps (which is why you want to install system app at the first place). Starting from Android 8.1, you will need to explicitly whitelist these permissions in system xml files.

For example, to give READ_PRIVILEGED_PHONE_STATE permission to a testing app, net.junsun.idattestation, you need to create the following file on phone, /etc/permissions/priv-app/privapp-permissions-idattestation.xml

<?xml version="1.0" encoding="utf-8"?>
<!--
This XML file declares which signature|privileged permissions should be granted to privileged
applications that come with the platform
-->
<permissions>
    <privapp-permissions package="net.junsun.idattestation">
        <permission name="android.permission.READ_PRIVILEGED_PHONE_STATE"/>
    </privapp-permissions>
</permissions>

Setup Bitcoin/ElectrumX/Lightning Network on Raspberry Pi 4

In the previous post I described the process of setting up a headless Raspberry Pi 4 (8GB) with remote VNC desktop access. It includes 1GB SDD drive, which is needed to run bitcoin full node with full archive of blockchain transactions.

In this post I will describe the process I used to set up a powerful multi-purpose Bitcoin node for the cryptocurrency community, that includes the following elements.

  • Bitcoin core – the default and standard bitcoin full node software that started the whole thing
  • ElectrumX server – de-centralized cryptocurrency wallet server. Wallet software needs it to make payments/transfers etc.
  • Lighnting Network Daemon (LND) – Level 2 network built on top of bitcoin to facilitate fast and cheap small amount payments.

Install and Run Bitcoin Core

wget https://bitcoin.org/bin/bitcoin-core-0.21.0/bitcoin-0.21.0-aarch64-linux-gnu.tar.gz
tar xzf bitcoin-0.21.0-aarch64-linux-gnu.tar.gz
sudo cp -a bitcoin-0.21.0/* /usr/local/
  • Run bitcoin-qt -txindex and start populating the blockchain data. It will take 2+ days and will consume >350GB in disk space!
    • DON’T SELECT “prune” MODE! Otherwise Electrum server won’t work.
  • Configure bitcoin for hardening and interactions with ElectrumX/LND
    • Create ~/.bitcoin/bitcoin.conf file. See sample setup below.
    • In this setup, we don’t listen to incoming peers over IPv4/IPv6 for enhanced privacy. New transactions will be broadcasted to Tor network only.
    • I also disabled wallet feature for better privacy again.
    • zmq (ZeroMQ) is needed for LND connection.
# index all transactions; LND/ElectrumX needs it
txindex=1

# bitcoin-qt accept RPC or not; needed for EPS/ElectrumX
server=1

# EPS needs wallet feature; electrumx does not
disablewallet=1

# don't broadcast transactions from our own wallet; will submit through Tor
# only useful when disablewallet==0
walletbroadcast=0

# # Maximum number of inbound+outbound connections. default 125
maxconnections=32

# enable electrumx server and LND
rpcuser=YOUR_RPC_USER
rpcpassword=YOUR_RPC_PASSWORD

# Listening for peers, enabled by default except when 'connect' is being used
# we don't export bitcoin ports or accept any incoming connections,
# because we would expose our own transactions to peers associated with our IP
# We only connect outgoing through tor
listen=0

# outgoing traffic use onion/tor only (default any)
proxy=127.0.0.1:9050
onlynet=onion

# debug logging - values: 0, 1, net, ....
debug=0

# needed by LND; I don't understand
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

Install ElectrumX Server

  • Install dependencies
sudo apt install -y python3-pip libleveldb-dev
sudo pip3 install aiohttp aiorpc
sudo pip3 install pylru
sudo pip3 install plyvel
  • Clone the the repo and install into /usr/local.
    • Note there are 2 primary electrumx repos on github.com. Make sure choose “spesmilo” one. The other one, albeit older, stopped supporting bitcoin due to recent segwit address introduction.
git clone https://github.com/spesmilo/electrumx.git
cd ~/electrumx
sudo python3 setup.py install
  • Add user to ssl-cert group to access the private SSL key
sudo usermod -a -G ssl-cert $USER
  • If bitcoin core has finished initial data sync, you can start ElectrumX server with the following script. Make sure you forward port 50001 and 50002 to the Raspberry Pi 4.
#!/bin/bash

# REQUIRED env variables
export DB_DIRECTORY=$HOME/.electrumx/db
export COIN=Bitcoin
DAEMON_URL="http://YOUR_RPC_USER:YOUR_RPC_PASSWORD@localhost:8332"

# Optional env variables
export PUBLIC_IP=$(curl -s http://whatismyip.akamai.com/)
export SERVICES=rpc://localhost:8000,tcp://:50001,ssl://:50002
export REPORT_SERVICES=ssl://$PUBLIC_IP:50002
export SSL_CERTFILE=/etc/ssl/certs/ssl-cert-snakeoil.pem
export SSL_KEYFILE=/etc/ssl/private/ssl-cert-snakeoil.key

# replace with your wallet address!
export DONATION_ADDRESS="bc1q52we5s8qyddhvrfscjn4qyn5nvq8hf92k9rt2w"

# log all sent transactions
electrumx_server

Install LND

Lightning Network in some sense is more complex than bitcoin. You are strongly suggested to review some basic concepts before setting up the software. There are multiple LN server implementations. LND is a popular one.

  • Install the pre-built binary image for ARM64
wget https://github.com/lightningnetwork/lnd/releases/download/v0.12.0-beta.rc6/lnd-linux-arm64-v0.12.0-beta.rc6.tar.gz
tar xzf lnd-linux-arm64-v0.12.0-beta.rc6.tar.gz
sudo cp lnd-linux-arm64-v0.12.0-beta.rc6/* /usr/local/bin
  • Set up config file
mkdir ~/.lnd
wget -O ~/.lnd/lnd.conf https://raw.githubusercontent.com/lightningnetwork/lnd/master/sample-lnd.conf
  • Below is a sample setup based on my own node
listen=0.0.0.0:9735
nolisten=false
rpclisten=0.0.0.0:10009
externalip=xxx.xxx.xxx.xxx
alias=YourFavoriateNameForThisNode
[Bitcoin]
bitcoin.active=true
bitcoin.mainnet=true
bitcoin.node=bitcoind
bitcoin.basefee=1
bitcoin.feerate=1
[Bitcoind]
bitcoind.dir=~/.bitcoin
bitcoind.rpcuser=YOUR_RPC_USER
bitcoind.rpcpass=YOUR_RPC_PASSWORD
bitcoind.zmqpubrawblock=tcp://127.0.0.1:28332
bitcoind.zmqpubrawtx=tcp://127.0.0.1:28333
bitcoind.estimatemode=ECONOMICAL
  • Run LND with “lnd” command
    • First time run, you will need to create a wallet, lncli create
    • For later running, you will need to unlock wallet, lncli unlock
      • If you like to automate LND startup, you will likely need lncli unlock --stdin option to pass in the wallet password via a script.
  • (Optional) if you like to set up a public LND node so that others can reach you, enable port forward on port 9735. You need to be a public node to participate in LN routing.

(Optional) Install RTL

RTL stands for “Ride The Lightning”, which provides a web interface to visually interact with LND. Otherwise you can just the commandline interface, which can be a little boring (and hard)

  • Install package and dependencies
git clone https://github.com/Ride-The-Lightning/RTL.git
cd RTL/
sudo apt install nodejs
sudo apt install npm
npm install --only=prod
cp sample-RTL-Config.json RTL-Config.json
vi RTL-Config.json
node rtl.js 
  • In order to run “node rtl.js” successfully, you need to configure RTL-config.json properly. Below is the sample setup derived from my own node.
{
  "multiPass": "YOUR_RTL_PASSWORD",
  "port": "3000",
  "defaultNodeIndex": 1,
  "SSO": {
    "rtlSSO": 0,
    "rtlCookiePath": "",
    "logoutRedirectLink": ""
  },
  "nodes": [
    {
      "index": 1,
      "lnNode": "Node 1",
      "lnImplementation": "LND",
      "Authentication": {
        "macaroonPath": "/home/ubuntu/.lnd/data/chain/bitcoin/mainnet/",
        "configPath": "/home/ubuntu/.lnd/lnd.conf",
        "swapMacaroonPath": ""
      },
      "Settings": {
        "userPersona": "MERCHANT",
        "themeMode": "DAY",
        "themeColor": "PURPLE",
        "channelBackupPath": "/home/ubuntu/.lnd/backup",
        "enableLogging": false,
        "lnServerUrl": "https://localhost:8080",
        "swapServerUrl": "https://localhost:8081",
        "fiatConversion": false
      }
    }
  ],
}
  • Once RTL starts running, you can open a browser and point at Raspberry Pi 4 IP address at port 3000. You see stats and perform actions with LND.

How You May Benefit

  • Use your own ElectrumX server for increased trust and privacy.
    • In this guide, we disable bitcoin core listening from public IP addresses. All new transactions are broadcasted from Tor. That increases privacy and anonymity.
  • You set up donation wallet address in ElectrumX. It seems people do donate, although not that common.
  • Once you set up LND properly, you can collect routing fees.

Show Your Support

This post is only complete if I can show you how I set up the donation and LN connection. 🙂 And your reading is only complete when you perform one of the following actions. 😛

For donation, Bitcoin wallet address is “bc1qyc48kmpweyx2kpqhq8n6r0ckr2fqsfvz2qxpx4”

For Lightning Network, my node public key is “0362a4372375bd24c5b8f8bb2ea85ae2ccf783808477e68cfb06121694b34d1927”

Appendix A – Reference Links on Lightning Network

Appendix B – How to create inbound liquidity for lightning network

Android AOSP Repo Checkout Method and Size Comparison

Android open source AOSP size has grown to be very huge now, often pushing over 100GB in disk size.  In this post, we look at 3 different ways to check out AOSP and compare their sizes.

1. Full clone – regular checkout with complete history, branch, tags and all files

repo init -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4
repo sync -j$(nproc)

2. Partial clone – only clone needed files with smaller-sized files initially, where the left-out files can be fetched later over the network when needed.

repo init -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4 --partial-clone --clone-filter=blob:limit=10M
repo sync -j$(nproc)

3. Shallow clone – don’t clone history or previous revisions or other branches/tags

repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4
repo sync  -f --force-sync --no-clone-bundle --no-tags -j$(nproc)

Below is the list of disk spaces consumed by each method. Clearly the winner size-wise is shallow clone. The disadvantage is that you don’t have any revision/branch/tag information. If that is OK with you, it is the way to go. Partial clone might work for you if you have fast network, but it does not seem to save much space.

MethodSize
Full clone105GB
Partial clone98GB
Shallow clone77GB

(Added later on 2/10, 2021) Additionally one may wish to set up a local AOSP mirror to reduce check out time. The total size of a full AOSP mirror is around 540GB.

mkdir aosp-mirror
cd aosp-mirror
repo init -u https://android.googlesource.com/mirror/manifest --mirror
repo sync

Set Up Ubuntu/XFCE4 on Headless Raspberry Pi 4 via VNC

Background

The new Raspberry Pi 4 with 8GB RAM is a little attractive  beast that is suitable for  home server.  You can set it up basically fan-less.  I was intrigued and I wanted to set it up headless, running Bitcoin full node, ElectrumX server plus Lightning Network (see my next post).

However, while there are many tutorials around Internet, I just can’t find the ideal one that suit me:

  • It must be 64bit.  Come on!  We are in 2021 already.
    • That rules out 32bit Raspberry Pi OS.
    • The 64bit Raspberry Pi OS (beta) runs fine on the board, but it lacks some basic Linux/UNIX features (e.g., middle mouse button pasting).  The desktop settings keep getting lost.  It is just frustrating.
  • That leaves 64bit Ubuntu, which should be a decent base OS.
  • I want to boot from USB without slow SD card.
  • I want to run headless with remote graphic UI 

The hardware

Below are the components I purchased.  The total cost is $270.  Not exactly dirty cheap, but that is pretty meaty piece. 

Putting it together is simper than setting up a toaster machine. See a couple of pictures below as well.

Step 1 – Prepare SSD drive and USB boot

  • Download and install rpi-imager on your host PC (Linux, Windows, etc)
  • Flush raspberry pi OS into SD card (I used 64bit RPIOS beta, but I think 32bit one should work). This will be a throw-away stepping stone.
  • Insert the SD card to RPI4 and boot from SD card.
    • Insert USB keyboard/mouse into black USB slot (USB2)
    • Connect a HDMI cable to a monitor
    • Use USB type-C power supply, ideally > 2.4A
  • Install (if not already installed) and run rpi-imager on RPI4 booted from SD card.
  • (IMPORTANT) Choose “Ubuntu 20.10 64bit Server” OS image and flush onto the 1TB Samsung SSD drive.
    • I found Ubuntu 20.04LTS don’t work!
  • Follow this link to update EEPROM firmware
  • Follow this link to boot from USB (i.e., SSD driver) first instead of SD card.
  • Remove SD card, and now you should be able to boot from SSD drive.

Step 2 – Fix up the board

  • Connect an Ethernet cable to enable Internet
  • Fix sluggish USB mouse
    • Add “usbhid.mousepoll=0” to /boot/firmware/cmdline.txt; Reboot
  • Install argon 1 fan control
    • curl https://download.argon40.com/argon1.sh | bash

Step 3 – Install XFCE desktop

Note choose “lightdm” when install xfce4

sudo apt update
sudo apt dist-upgrade
apt install xfce4 xfce4-goodies
apt install tigervnc-standalone-server novnc firefox

Create /etc/lightdm/lightdm.conf file:

[Seat:*]
allow-guest=false
autologin-user=ubuntu
user-session=xfce
#greeter-session=unity-greeter
xserver-command=/etc/X11/xinit/xserverrc

After this you should be able to log into graphic XFCE desktop.

Step 4 – Set up VNC and headless

  • Configure wifi assuming that is what you want to run RPI4
  • Set up VNC password
    • sudo vncpasswd /etc/vncpasswd
  • Modify /etc/X11/xinit/xserverrc file
#!/bin/sh

ARG=`echo $@ | sed -e "s#vt. -novtswitch##"`
GEOMETRY="1600x1000"
DEPTH="24"
# single client mode; VNC password auth; TLS encryption optional;
exec Xvnc -desktop ubuntu -SecurityTypes "VncAuth,TLSVnc" -NeverShared -DisconnectClients -geometry $GEOMETRY -depth $DEPTH -rfbauth /etc/vncpasswd $ARG
  • Make sure WiFi is connected before user logs in so that you can connect via VNC
    • To do this, launch NetworkManager, select “Applications”/”Settings”/”Advanced Network Configuration”
    • Then select the wifi network you are connecting to, and then click on button at the bottom to edit the connection
    • Under “General” tab, check “All users may connect to this network”

After this setup, if you reboot RPI4, you will get text console on display.

Meanwhile you should be able to access RPI4 over VNC remotely

  • On your PC, install VNC viewer (e.g., tigerVNC viewer)
  • Launch the viewer, enter the IP address and port 5900
  • Enter the VNC password
  • You should see XFCE desktop

Step 5 – Optional noVNC to enable web browser access

  • Install noVNC package
    • sudo apt install novnc net-tools
  • Run /usr/share/novnc/utils/launch.sh
  • On PC, open a browser and point at “http://<RPI4 IP address>:6080/vnc_auto.html”
  • Enter VNC password and you will be connected to the desktop

You can automate this process by adding this to /etc/rc.local or create a small systemd service file.

AWS ARM64 vs. X86_64 – Bitcoin Performance Comparison

I’m running some experiments with bitcoin core software on AWS. I’m intrigued by 2 questions:

  • Which performs better, ARM64 vs X86_64?
  • What is performance loss of 32bit ARM vs 64bit ARM?

The Experiment

  • I fired up a t3.medium instance with ubuntu desktop 20.04(x86_64) image
  • I fired a second instance of t4g.medium type with ubuntu desktop 20.04 (arm64) image
  • To run 32bit ARM program, I followed the chroot approach documented in my previous post and set up an armhf Ubuntu 20.04 (focal) chroot environment
  • Then I download all 3 versions of bitcoin core software from its download site : 32bit ARM, 64bit ARM and 64bit Intel.
  • Run bitcoin-qt from scratch (delete ~/.bitcoin directory if any) and count the time duration for the verifying first 200,000 blocks.
    • Each configuration runs 2 times. We then take the average.
    • Other system variables are monitored to ensure similar running environment. Specifically networking or disk don’t seem to be a factor.

The Results

See results listed below. It seems for bitcoin related workload 64bit ARM version performs 20% faster than 32bit ARM version and 64bit Intel instance.

category64bit ARM 32bit ARM64bit Intel
Instance typet4g.mediumt4g.mediumt3.medium
CPU Graviton2Graviton2Intel Xeon Platinum 8000 series
# CPU cores222
RAM (GB)444
run duration #18’44” (524″)10’24” (624″)11’16″(676″)
run duration #27’30” (450″)9’58″(598″)9’04″(544″)
run duration average487″611″610″
relative performance to 64bit ARM100%80%80%
Pricing (us-west-2/Orgon) ($/hr)0.03360.03360.0416

Geekbench Performance on AWS Graviton 2

There is much touting about the new AWS Graviton 2 (ARM64) offering as a game changer. Let us run some benchmark to test it out.

Settings

We pick 3 EC2 instance types to compare:

  • a1 – First generation of ARM64 AWS Graviton CPU
  • m6g – Second generation of ARM64 Graviton 2 CPU
  • m5 – Intel Xeon Platinum 8259CL CPU

We run Geekbench 4 on all xlarge instances of these EC2 types. We mostly focus on 64bit performance, but we will also touch 32bit performance as well.

Overview

Instance typea1.xlargem6g.xlargem5.xlarge
vCPU444
Memory(GB)81616
Hourly price(us-ea-1,Linux)0.1020.1540.192
64bit single-core score189936093647
64bit multi-core score5227111428017

From the above table, several observations are obvious:

  • Graviton 2 has doubled the performance of Graviton 1.
  • For single core performance Graviton 2 is similar to Intel Xeon CPU
  • For multi-core performance, Graviton 2 scales up much better, likely because Intel uses hyper-threading technology, where vCPU count is only 1/2 of true CPU core count. By contrast, vCPU count in Graviton CPU is true CPU core count.

I also listed the pricing. It looks like Graviton 2 is a good deal!

Look into the Details

This link gives detailed scores for each test suite and each instance type. A few highlighted cells indicate interesting contrast between Intel Xeon and Graviton 2:

  • Intel Xeon is 20 times faster than Graviton 2 in AES test! This is *very* likely due to non-optimized implementation for ARM64, i.e., it is not using NEON instructions. Otherwise the performance should be more comparable.
  • Intel Xeon are 2x better in SGEMM and 50% better in SFFT, both heavily relying Intel AVS/SSE instructions while ARM64 using NEON instructions.
  • Graviton 2 shines in memory area, 2x better in Memory Copy and 4x better in Memory bandwidth.

32bit Performance

While 32bit performance is probably not interesting on those servers, it is still interesting to take a look. Below is the overview comparison table and this link gives detail scores.

Instance typea1.xlargem6g.xlargem5.xlarge
32bit single-core score170729753053
32bit multi-core score469290166886

Overall we see similar patterns in 64bit case:

  • Graviton 2 is about 2x faster than Graviton 1
  • Intel Xeon performs relatively same as Graviton 2 in single core and wanes in multi-core performance.
  • Detail scores also reflect similar pattern as 64bit case.

Develop 32bit Applications on 64bit Linux Machines

Overview

Nowadays almost all PC and servers are 64bit. Once in a while one may need to develop and run 32bit apps. In this post we will go through several methods for both x86_64 and ARM64 machines.

I can think of 4 ways of doing this:

  • Run a virtual machine, which in turn runs a 32bit OS. Not covered in this post.
  • Set up 32bit chroot environment. (Recommended)
  • Set up 32bit architecture with Debian multi-arch support (Recommended)
  • Use gcc multi-lib feature to build 32bit binaries.

X86_64/Ubuntu 20.04

Here we go through some detailed steps for those approaches on X86_64/Ubuntu 20.04. Instructions should apply to other Debian-based distributions and different versions as well.

GCC Multi-Lib Approach

Although this is the simplest approach, I don’t recommend it because the GCC support varies for different CPU architectures and the command line options also varies, making it hard to generalize and port existing software. I believe mult-arch is the future direction.

  • Install GCC multi-lib package
sudo apt install gcc-multilib
  • Compile with -m32 option
gcc -m32 hello.c -o hello
file hello
./hello

Multi-Arch Approach

  • Add 32bit as secondary architecture
sudo dpkg --add-architecture i386
  • Install dev tools and 32bit specific dev tools and libraries
sudo apt update
sudo apt install -y build-essential
sudo apt install -y crossbuild-essential-i386
sudo apt install -y libc6:i386
sudo apt install -y libstdc++6:i386
  • Develop 32bit apps with i686-linux-gnu- cross-compile prefix
i686-linux-gnu-gcc hello.c -o hello
file hello
./hello
  • Note for Ubuntu 18.04, crossbuild-essential-i386 meta-package does not exist. One needs to install below packages explicityly
sudo apt install -y dpkg-cross g++-i686-linux-gnu  gcc-i686-linux-gnu

Chroot Approach

For more detailed descriptions, please refer to http://logan.tw/posts/2018/02/24/manage-chroot-environments-with-schroot/

  • Install packages
sudo apt-get install schroot debootstrap
  • Install 32bit environment (e.g., Ubuntu 20.04) (note the last URL arg can be ommited)
sudo debootstrap --arch=i386 focal /var/chroot/focal32 http://archive.ubuntu.com/ubuntu
  • Edit /etc/schroot/schroot.conf file
[focal32]
description=Ubuntu 20.04 Focal Fossil 32bit
directory=/var/chroot/focal32
root-users=ubuntu,jsun
users=ubuntu,jsun
type=directory
personality=linux32
  • Create and enter 32bit chroot session
schroot -c focal32          ; enter as normal user
schroot -c focal32 -u root  ; enter as root user
  • From inside the session, one can install build-essential meta-package and develop 32bit apps as if inside a 32bit machine.
  • (Optional) One can even run 32bit graphic apps from inside the chroot environment.
export DISPLAY=:0.0

ARM64/Ubuntu18.04

ARM64 is also called aarch64. “ARM64” is rooted from Linux community and “aarch64” is official name from ARM. We will use those 2 terms inter-changeably.

Multi-Lib approach

GCC for aarch64 does not support multi-lib. So this option is not available.

Multi-Arch Approach

It is similar to i386 case, except that architecture is “armhf” and 32bit cross-compile prefix is “arm-linux-gunueabihf-“.

sudo dpkg --add-architecture armhf
sudo apt update
sudo apt install -y build-essential
sudo apt install -y crossbuild-essential-armhf
sudo apt install -y libc6:armhf
sudo apt install -y libstdc++6:armhf

arm-linux-gnueabihf-gcc hello.c -o hello
file hello
./hello

Chroot Approach

For chroot approach, it is similar to i386 case, with 2 exceptions:

  • The architecture name is “armhf” instead of “i386”
  • The repo URL is “http://ports.ubuntu.com/ubuntu-ports”, instead of “http://archive.ubuntu.com/ubuntu”

Note we can also set up the root filesystem from debian repos. Below is an example to set up debian sid for armhf 32bit architecture:

  • Install packages and bootstrap root filesystem
sudo apt-get install schroot debootstrap
sudo debootstrap --arch=armhf sid /var/chroot/sid-armhf http://ftp.debian.org/debian/
  • Modify and add to /etc/schroot/schroot.conf file
[sid32]
description=Debian Sid 32bit
directory=/var/chroot/sid-armhf
root-users=ubuntu
users=ubuntu
type=directory
personality=linux32
  • Create and enter sid32 session, “schroot -c sid32”

About MIPS64

  • Multilib works for MIPS64.
    • sudo apt install gcc-multilib g++-multilib
    • compile with -mabi=32 flags, i.e., “gcc -mabi=32 hello.c”
  • Multi-Arch does not work for MIPS64, at least from 32bit app development perspective
  • I have not tried chroot approach on MIPS64 yet. Let me know if you have had any success.

Last Words

  • In Ubuntu chroot environment, the initial apt repo source is limited to main. You may find many packages are missing. To fix this, edit /etc/apt/source.list
deb http://archive.ubuntu.com/ubuntu xenial main universe multiverse