How I Rooted OnePlus 12 with Magisk

There are many conflicting sources on the Internet. Specifically I tried this one and did not work. Below is a short recap what has worked for me.

Short Recap

  • Follow Magisk official installation guide
    • OnePlus 12 has ramdisk and uses init_boot.img
    • get oneplus 12 image zip file from this site
    • use payload dumper to extract init_boot.img from here
    • patch init_boot.img and flash the patched version according to the guide
  • By now, Magisk should be installed and you should have root access
  • Install Magisk Module Manager to install modules
    • for unknown reasons, I could install modules with Magisk app itself, nor through the manual method
  • (Bonus) I like to create your own Magisk module. I used the template
    • Specifically if you like to modify a file under /system_ext, please use the path /system/system_ext.
    • For example, if you like to add a file /system_ext/foo, use /system/system_ext/foo instead.

That is it!

Build, Flash and Un-Flash AOSP Image on Pixel Phones

In this post we will go through a full cycle of building a custom AOSP image, flashing it on a Pixel phone, and then restoring to original stock image. We will use Pixel 4 (codenamed as “flame”) as the device example. We will walk through in a succinct manner without detailed explanations.

Prepare the Device

  • Note the Android build number
    • Click “Settings”/”About Phone”.
    • In my case it is “QQ3A.200805.001”
  • Enable developer options
    • click on “Build Number” row 7 times to enable developer options
  • Enable “OEM unlocking”
    • Toggle “Settings”/”System”/”Advanced”/”Developer Options”/”OEM unlocking”

Build AOSP Image

Prepare the host

My host is Ubuntu 20.04. Generally we are following this guide from Google. Below is a quick gist.

sudo apt-get install git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig
sudo apt install python2    #repo needs python
sudo usermod -aG plugdev $LOGNAME
apt-get install android-sdk-platform-tools-common   # install udev rules and adb

Download AOSP source

Generally we are following this guide from Google. Below are some specific steps.

  • Find exact branch/tag that matches device current build.
    • Go to the Android build tag page
    • Search for the build number noted at the previous step
    • Find corresponding AOSP tag. In my case, it is “android-10.0.0_r41”
  • Check out this branch
mkdir android-10.0.0_r41-shallow
cd android-10.0.0_r41-shallow
repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-10.0.0_r41
repo sync  --force-sync --no-clone-bundle --no-tags -j$(nproc)

Fetch Proprietary Binaries

  • Go to Google proprietary binary page
  • Find the section corresponding to your device (Pixel 4 in my case) and the build number (“QQ3A.200805.001”)
  • Download tarballs from Google and Qualcomm
  • Untar them under the root directory of AOSP source tree
cd ~/work/android/android-10.0.0_r41-shallow
tar xzf ~/Downloads/google_devices-flame-qq3a.200805.001-a7a5499d.tgz
tar xzf ~/Downloads/qcom-flame-rp1a.200720.009-1157dde5.tgz
./extract-google_devices-flame.sh
./extract-qcom-flame.sh

Build images

source build/envsetup.sh
lunch aosp_flame-userdebug
m

Note “flame” is the codename for Pixel 4. If you have a different device, you should use a different target. See Pixel factory image page for device codenames.

Command ‘m’ starts the build process, which can take very long. My laptop has AMD Ryzen™ 7 Pro 4750U Processor (1.70 GHz, up to 4.10 GHz Max Boost, 8 Cores, 16 Threads, 8 MB Cache) and has a fast NVME SSD disk. It takes about 2.5 hours to finish the build.

Flash AOSP Built Image

  • On the phone, enable OEM unlocking and USB debugging
    • Toggle “Settings”/”System”/”Advanced”/”Developer options”/”USB debugging”
    • Toggle “Settings”/”System”/”Advanced”/”Developer options”/”OEM unlocking”
  • On the PC in the same terminal where you just build the AOSP image
adb devices             # it should show the phone device attached over ADB
adb reboot bootloader   # reboot to fastboot
fastboot devices        # it should show the phone device attached over ADB
fastboot flashing unlock
fastboot flashall -w

During “fastboot flashing unlock” step, you need to following instruction on the screen, press up/down volume button to select “unlock bootloader”, and press power button to commit.

If things go smoothly, phone will reboot a couple of times and eventually boots into the freshly built AOSP image with root access.

Restore to Stock ROM Image

After you played enough with the AOSP image, you may want to return to the original retail state. Google has made it easy recently.

  • Open Chrome browser and go to “https://flash.android.com”
  • Allow browser to connect to the device
  • Go to Pixel 4 factory image page and find the desired build number to flash
    • Just for fun, I’m selecting Android 11, build number RQ1A.210205.004.
  • Select “Wipe device” and “Lock bootloader”
  • Click “Confirm” to start the process.

What If Things Go Wrong?

Things can go wrong when phone does not boot up anymore. It can be due to build AOSP build, or something due to mismatch between AOSP image and baseban/firmware version (BTW, this is the reason why this guide asks you to download the same version of AOSP that matches existing build).

unzip flame-qq3a.200805.001-factory-d93c74e6.zip
cd flame-qq3a.200805.001/
fastboot devices   # make sure phone is connected
./flash-all.sh 

(Bonus) How to Add a System Privileged App

Before Android 10, you simply re-mount /system as rw and push apk into /system/priv-app/

adb root
adb shell remount -orw,remount /system
adb push my-app.apk /system/priv-app
adb reboot    # to complete the app installation

Since Android 10, /system is mounted as root and cannot be re-mounted as rw. Instead you will need to use “adb remount” to enable overlay FS for any modification.

adb root
adb remount     # if this is first time, "adb reboot" to disable verity first
adb push my-app.apk /system/priv-app
adb reboot      # to complete the app installation

Often times you need to assign PivilegedOrSystem level permissions to system apps (which is why you want to install system app at the first place). Starting from Android 8.1, you will need to explicitly whitelist these permissions in system xml files.

For example, to give READ_PRIVILEGED_PHONE_STATE permission to a testing app, net.junsun.idattestation, you need to create the following file on phone, /etc/permissions/priv-app/privapp-permissions-idattestation.xml

<?xml version="1.0" encoding="utf-8"?>
<!--
This XML file declares which signature|privileged permissions should be granted to privileged
applications that come with the platform
-->
<permissions>
    <privapp-permissions package="net.junsun.idattestation">
        <permission name="android.permission.READ_PRIVILEGED_PHONE_STATE"/>
    </privapp-permissions>
</permissions>

Android AOSP Repo Checkout Method and Size Comparison

Android open source AOSP size has grown to be very huge now, often pushing over 100GB in disk size.  In this post, we look at 3 different ways to check out AOSP and compare their sizes.

1. Full clone – regular checkout with complete history, branch, tags and all files

repo init -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4
repo sync -j$(nproc)

2. Partial clone – only clone needed files with smaller-sized files initially, where the left-out files can be fetched later over the network when needed.

repo init -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4 --partial-clone --clone-filter=blob:limit=10M
repo sync -j$(nproc)

3. Shallow clone – don’t clone history or previous revisions or other branches/tags

repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android-11.0.0_r4
repo sync  -f --force-sync --no-clone-bundle --no-tags -j$(nproc)

Below is the list of disk spaces consumed by each method. Clearly the winner size-wise is shallow clone. The disadvantage is that you don’t have any revision/branch/tag information. If that is OK with you, it is the way to go. Partial clone might work for you if you have fast network, but it does not seem to save much space.

MethodSize
Full clone105GB
Partial clone98GB
Shallow clone77GB

(Added later on 2/10, 2021) Additionally one may wish to set up a local AOSP mirror to reduce check out time. The total size of a full AOSP mirror is around 540GB.

mkdir aosp-mirror
cd aosp-mirror
repo init -u https://android.googlesource.com/mirror/manifest --mirror
repo sync

How to build/run Android Cuttlefish emulator on AWS

Background

Cuttlefish is new virtual-machine based Android emulator.  Earlier I have written  an article on how to build/run it on PC and ARM64 machines.  Towards the end of 2019, AWS has introduced a1.metal instance which allows KVM to run on their ARM64 machines.  It opens the possibility of running cuttlefish on AWS (which itself opens a lot of possibilities!)

The steps are similar to those I mentioned before.  This article summarizes them here specifically for AWS a1.metal instance, running Ubuntu 19.10.

Build AOSP cuttlefish images on x86_64

This step is done on PC, while all the rest steps are done on AWS a1.metal instance.

  • Refer to https://source.android.com/setup/build/building to set up your host machine
  • Check out the source from master and build distribution packages, which will be transferred to a1.metal instance later.  This step takes very loooong time.
mkdir cuttlefish
repo init -u https://android.googlesource.com/platform/manifest
repo sync -j8
source build/envsetup.sh
lunch aosp_cf_arm64_phone-userdebug
make dist

If you like to download from a branch and like to download as little as possible, the following command do so with android10-gsi branch without git history.

repo init --depth=1 -u https://android.googlesource.com/platform/manifest -b android10-gsi
repo sync -f --force-sync --no-clone-bundle --no-tags -j$(nproc)

Setup AWS a1.metal

Start an Ubuntu 19.10 instance on a1.metal.  First,  we install GUI for better debugging and viewing. 

sudo apt-get update -y 
sudo apt-get install lxde xrdp -y
sudo passwd ubuntu

Setup to run x86_64 binaries

Many tools (e.g., cros_vm) in cuttlefish are still built as x86_64 binaries, not as arm64.  Current solution is to use qemu-user to run those binaries (*ouch!*).  As such, we set up x86_64(amd64) as the secondary architecture on a1.metal.

  • Override /etc/apt/sources.list file with the following content
deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports eoan main restricted universe multiverse
deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports eoan-updates main restricted universe multiverse
deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports eoan-backports main restricted universe multiverse
deb [arch=arm64,armhf] http://ports.ubuntu.com/ubuntu-ports eoan-security main restricted universe multiverse
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ eoan main restricted universe multiverse
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ eoan-updates main restricted universe multiverse
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ eoan-backports main restricted universe multiverse
deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ eoan-security main restricted universe multiverse
  • Install qemu-user-static and add amd64 secondary architecture
sudo apt install qemu-user-static
sudo dpkg --add-architecture amd64
sudo apt install libc6:amd64
  • After this you should be able to run simple x86_64 binaries such as “cat”.
    • scp over “cat” program from your x86_64 linux machine
    • it should run!

Build and install cuttlefish-common package

  • install packages needed for build
sudo apt install dpkg-dev
sudo apt install cdbs config-package-dev debhelper
  • Download and build
mkdir cuttlefish-common
cd cuttlefish-common
git clone https://github.com/google/android-cuttlefish.git
cd android-cuttlefish
dpkg-buildpackage --no-sign
  • Install
sudo apt install bridge-utils dnsmasq-base f2fs-tools libarchive-tools libfdt1 libwayland-client0 net-tools python2
sudo dpkg -i ../cuttlefish-common_0.9.13_arm64.deb

Setup cuttlefish for running

  • copy (scp) the following  the files from x86_64 PC host:
out/dist/aosp_cf_arm64_phone-img-eng.jsun.zip
out/dist/cvd-host_package.tar.gz
  • untar cvd-host_package.tar.gz to ~/cuttlefish/host directory
  • unzip the zip file to ~/cuttlefish/image directory
  • Add user to proper groups before running; power off machine; then power on again. 
sudo usermod -aG kvm $USER
sudo usermod -aG cvdnetwork $USER

Run Cuttlefish

You have 2 choices to view the screen of emulated Android device:

  1. Use remote desktop and view screen via local VNC connection to Cuttlefish emulator
  2. Use a web browser and view the screen via remote WebRTC connection

View via Local VNC

  • Start a remote desktop viewer and connect to your EC2 a1.metal instance (Note: you need to open port 3389)
  • Below steps are running on remote EC2 a1.metal instance via remote desktop
  • Start a LXTerminal and run the following commands to start cuttlefish
export ANDROID_PRODUCT_OUT=~/cuttlefish/image/
export ANDROID_HOST_OUT=~/cuttlefish/host/
export PATH=$PATH:$ANDROID_HOST_OUT/bin
launch_cvd -decompress_kernel=true
  • Start browser and download VNC viewer
    • download tightvnc viewer (jar file): https://www.tightvnc.com/download.php
    • install java if not done yet: sudo apt install openjdk-11-jre
  • Start a second LXTerminal and run this command : java -jar tightvnc-jviewer.jar
    • Please use 127.0.0.1 as IP address and 6444 as port number
  • To stop cvd, run stop_cvd

View via WebRTC

  • Note you need to enable port 8443 on EC2 a1.metal instance
  • On EC2 instance run
launch_cvd -start_webrtc -webrtc_public_ip=`curl http://169.254.169.254/latest/meta-data/public-ipv4` -decompress_kernel=true
  • On local PC, start a browser on local machine, and connect via “https://<public IP>:8443”

As of writing today (4/5, 2020), the WebRTC method is not working yet.  It shows a black screen.

How to Build/Run Android Cuttlefish Emulator on PC/ARM64

Cuttlefish is new virtual-machine based Android emulator. It uses virtio devices instead of emulated devices as in original Android emulator.   As such, it needs lighter VM support (to the extent it can run on ARM64 host), unlike Android Emulator which requires heavily modified QEMU to emulate various devices.  The virtio architecture can potentially offer better performance as well.

Refer to  a slide deck on cuttlefish.  Or find a local copy of it at here.

Many thanks to Alistair Delva from Google, who provided many technical guidance in going through this exercise.

How to Build/Run Cuttlefish on PC (X86_64)

My host Ubuntu 18.04.  Refer to https://source.android.com/setup/build/building

Build and install cuttlefish-common package

git clone https://github.com/google/android-cuttlefish.git 
cd android-cuttlefish
dpkg-buildpackage --no-sign
  • Install :
dpkg -i  ../cuttlefish-common_0.9.9_amd64.deb
  • it requires dnsmasq-base and a few other packages; install them as requested
  • Check status  with  /etc/init.d/cuttlefish-common status

Build cuttlefish

  • Checkout AOSP pie-gsi branch.
repo init -u https://android.googlesource.com/platform/manifest -b pie-gsi repo sync -j 8
source build/envsetup.sh
lunchaosp_cf_x86_64_phone-userdebug
make

Run cuttlefish

  • Add user to proper groups before running; power off machine; then power on again.  (Strange, rebooting did not seem to work somehow).  You may need to run “sudo apt install qemu-kvm” if you get error, “group kvm doesn’t exist”.
sudo usermod -aG kvm $USER
sudo usermod -aG cvdnetwork $USER
  • Run: launch_cvd
  • Install tightvnc to view phone
    • download tightvnc viewer (jar file): https://www.tightvnc.com/download.php
    • install java if not done yet:  sudo apt install openjdk-11-jre
    • java -jar tightvnc-jviewer.jar  #use 127.0.0.1:6444
  • Run stop_cvd to kill the cvd

How to Build/Run Cuttlefish on ARM64

My ARM64 board is rockpro64, running ubuntu 18.04.

[on X86_64] Cross-build cuttlefish for ARM64

  • Similar as above, except build with a different target and with distribution packages
lunch aosp_cf_arm64_phone-userdebug
make dist

Note the following output files, which need to be copied to ARM64 host

out/dist/aosp_cf_arm64_phone-img-eng.jsun.zip
out/dist/cvd-host_package.tar.gz

[on X86_64] Configure and build arm64 kernel

You will need CONFIG_BINFMT_MISC.  Otherwise below step will fail.  Check /proc/sys/fs/binfmt_misc to be sure.

In addition, you will need a few other kernel configs, which according to Alistair are only supported in kernel after 4.9.  Here is the set of configs I added to rockpro64 default v5.2 kernel.

CONFIG_BINFMT_MISC=y
CONFIG_EVENTFD=y
CONFIG_VSOCKETS=y
CONFIG_VHOST_NET=m
CONFIG_VHOST_SCSI=m
CONFIG_VHOST_VSOCK=m
CONFIG_VHOST=m
CONFIG_VIRTIO_BLK_SCSI=m
CONFIG_VIRTIO_INPUT=m
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_VIRTIO_VSOCKETS_COMMON=m
CONFIG_VIRTIO_VSOCKETS=m

In specific, here are the exact commands I used to build my rockpro64 kernel on my PC ubuntu:

git clone https://github.com/ayufan-rock64/linux-mainline-kernel.git
cd linux-mainline-kernel/
git checkout -b 5.2.0-1116-ayufan-js 5.2.0-1116-ayufan
vi arch/arm64/configs/rockchip_linux_defconfig    # add the above configs to the end
vi dev.mk   # BUG? change HOSTCC=aarch64-linux-gnu-gcc to HOSTCC=gcc
./dev-make kernel-image-and-modules
./dev-make kernel-package

Copy over the .deb package file to arm64 host and install with “dpkg -i <pkg file>” command.  Reboot afterwards.

[on ARM64] Setup to run x86_64 binaries

Do following as root user:

apt install qemu-user-static
dpkg --add-architecture amd64
[You may need to correct /etc/apt/source.list file here.  See an example at this link]
apt install libc6:amd64

After this you should be able to run simple x86_64 binaries such as “cat”.  Give it a try.

[on ARM64] Build and install cuttlefish-common package

This is similar to x86_64 case, except that you do this step on ARM64 host.

The following packages are needed before you can install cuttlefish-common:

apt install bridge-utils libarchive-tools libfdt1 python iptables

[on ARM64] Setup and run cuttlefish

  • copy the following  the files from x86_64 PC host:
out/dist/aosp_cf_arm64_phone-img-eng.jsun.zip
out/dist/cvd-host_package.tar.gz
  • untar and unzip them into 2 directories, say /home/jsun/work/cuttlefish/host and /home/jsun/work/cuttlefish/image
  • Add user to proper groups before running; power off machine; then power on again.  You may need to create kvm group first and make sure  /dev/kvm is read/writable by kvm group
sudo usermod -aG kvm $USER
sudo usermod -aG cvdnetwork $USER
  • Run the following commands to start cuttlefish
export ANDROID_PRODUCT_OUT=/home/jsun/work/cuttlefish/image/
export ANDROID_HOST_OUT=/home/jsun/work/cuttlefish/host/
export PATH=$PATH:$ANDROID_HOST_OUT/bin
launch_cvd -decompress_kernel=true
  • The rest are similar to x86_64 case

Appendix – Install Ubuntu Desktop on ARM64

Many ARM64 ubuntu distros are minimal or server, which means no desktop included.  It is easy to install one.  However, without a few key steps (see first a few commands below),  you can easily get some headaches.

Below are the commands I used to install Xubuntu on rockpro64, starting from their minimal Ubuntu 18.04 distro.

  1. Download Ubuntu 18.04 minimal img for SD card (refer to this page),
  2. Copy to SD card
dd if=./bionic-minimal-rockpro64-0.8.3-1141-arm64.img of=/dev/sdb bs=4M
  1. Use parted or gparted to expand /dev/sdb7 to take over whole SD space
  2. Boot up rockpro64:
sudo su
locale-gen
localectl set-locale LANG="en_US.UTF-8"
apt update
apt install -y xubuntu-desktop