Posted on

Install Full Ubuntu on Portable USB Drive

Background

My goal is to install fully updateable Ubuntu 20.04 onto a USB stick, so that I can boot it up with any Intel-based PC’s or laptops. However, due to what I considered a bug in Ubuntu, this is actually harder than it should be. So I wrote down this blog in the hope it might help others, as well as my future self, in the similar shoes.

Note that the objective is different from so-called LiveUSB ubuntu with persistency, where Ubuntu OS itself will remain as a static ISO image and updates are added on-top in a separate persistent partition. My goal is to install a standard Ubuntu OS on a USB disk, which can be updated and upgraded just like normal PC case, except that a) it is on a portable USB drive or disk and b) it is portable across different PC’s. I suppose this setup gives longer life span of the installation, which potentially allows you to even upgrade your OS later.

In the following steps, I will also show an optional feature which creates an encrypted home directory.

Assumptions and Prerequisites

  • You need Intel x86_64 PC
    • We assume it support UEFI and GPT partitions which are standards for all recent ones
  • A USB drive that holds Ubuntu ISO image for installation, a.k.a. the installation media drive. This needs to have 4GB minimum size.
  • A second USB drive or disk that will hold installed Ubuntu OS, a.k.a. the installation target drive. This one needs 16GB minimum size

Step 1 – flash Ubuntu ISO image to the installation media drive

I will not repeat the process here. Please refer to many pages below.

Step 2 – Prepare the partition table on the target drive

  • Insert installation media drive into PC.
  • Interrupt normal booting sequence and choose the media USB drive as the boot device
    • Different PC have different process to do this. On Lenovo PC, one has to press ENTER on bootup, and then press F12 to select boot device
  • Select “Try Ubuntu” when presented the option
  • Insert target USB drive
    • Identify which drive is target USB drive by examining the output “lsblk”
    • In most case if you follow the instructions exactly, it would be “/dev/sdb”
  • Once Ubuntu is up and running, start a terminal and type “sudo gparted /dev/sdb” (replace “/dev/sdb” with the right usb device you have for the target USB drive)
    • create GPT partition table
      • click “Device”/”Create Partition Table …”
      • select “gpt” as partition table type
      • See Pic #1 below
    • create 100MB fat32 partition as ESP partition
      • Click “Partition”/”New”;
      • Enter “100MB” as size and select “FAT32” as file type
      • See Pic #2 below
    • set “esp”, “boot” attributes to the new ESP partition
      • Apply changes to actually crate the partition
      • Select the ESP partition and then select “Partition”/”Manage Flags”
      • In the pop-up window, select “esp” and “boot” flag
      • See pic #3 below
    • create an ext4 partition that takes the rest of space for root partition
      • See pic #4 below
    • (optional) if you like to have encrypted home partition, create an ext4 root partition with size of 10GB or more, and leave the rest free space open for encrypted home petition later.
      • Pic #5 shows the partition table at this step.
  • Pic #1 – Create GPT Partition Table
  • Pic #2 – Create ESP Partition
  • Pic #3 – set esp/boot attributes
  • Pic #4 – create root partition
Pic #5 – after create ESP and root partitions

Step 3 – Install Ubuntu

Once we finish the above step, quit gparted and we are ready to install Ubuntu into the target USB drive.

  • click “install ubuntu” icon on the desk to to start installation
    • select “something else” in partition page. See Pic #6 below.
    • select the ESP partition on the target USB drive as the “ESP” partition. See Pic #7 below.
    • select root partition and mount as “/”. See Pic #8 below.
    • (optional) Create encrypted physical volume
      • Select the free space left during creating partitions
      • Click “+” to create a new partition/volume
      • select “encrypted physical partition”.
      • See Pic #9 below.
    • wait for a while, select “/dev/mapper/sdb3_crypt” as “/home”. See Pic #10 below
    • Finish installation.
  • Pic #6 – Select “Other” installation type
  • Pic #7 – Set ESP partition
  • Pic #8 – set root partition
  • Pic #9 – create encrypted physical volume
  • Pic #10 – Create/set home partition

Step 4 – Fix EFI on target USB drive

At this point you might get an illusion everything is working, because if you reboot the PC you will be able to select either Ubuntu or Windows to boot up, and they all work. However, there are 2 very serious problems

  • If you boot into BIOS and select the USB disk as boot device, it won’t work.
  • Even worse, your PC is likely not able to boot up Windows either if you remove the target USB disk.

The reason for these problems is that, despite we told Ubuntu installer to install Ubuntu on the USB disk, which implies it should use the ESP partition on target USB disk, it still uses the ESP partition on PC built-in disk, and thus screws up EFI partition on PC and leave an empty EFI on target USB disk. See more details at this very old bug report.

So the first thing we need to do is to install Ubuntu loader into USB ESP partition and install grub into USB disk

  • Reboot into BIOS firmware and select “ubuntu” as the boot target
  • switch EFI mount
    • Type “lsblk” to verify that ESP partition on PC built-in drive is mounted
    • umount it, “sudo umount /boot/efi”
    • mount the right one, ” sudo mount /dev/sda1 /boot/efi”
  • install grub boot on USB EFI partition
    • “sudo modprobe efivars”
    • “grub-install -d /usr/lib/grub/x86_64-efi –efi-directory=/boot/efi/ –removable /dev/sda”
    • Note “–removable” flag is important in the above command as it allows USB to boot on any Intel-based PCs
  • Reboot into BIOS firmware and select target USB as boot device. It should work now.
    • You can try the USB disk on other PCs, and it should work as well.

At this point, /etc/fstab is probably mounting PC EFI partition to /boot/efi, which is wrong and will break when you boot the target USB from another PC. You can either delete the /boot/efi line in /etc/fstab, or replace the UUID with the one for EFI partition on target USB. You can find out the UUID with “blkid” command (e.g., blkid /dev/sda1)

Step 5 – Restore PC boot loader

Now let us fix PC does not boot up problem (if you encounter this)

  • Boot up the PC into BIOS and select the newly made Ubuntu USB drive as boot device. This will boot into Ubuntu.
  • Open a terminal
  • Remove Ubuntu from PC EFI partition, and PC will boot up windows again
    • Mount host EFI partition as /mnt, “sudo mount /dev/nvme01n1p1 /mnt”
    • “cd /mnt/EFI/”
    • “sudo rm -rf ubuntu”
  • NOTE 1: If your PC is already installed with another Ubuntu system, the bootloader entry “ubuntu” will collide with each other and the previous Ubuntu will not be able to boot. You can follow this guide to restore booting the previous Ubuntu system.
  • NOTE 2: Annoyingly, if your PC is already installed with another Ubuntu system, each time you boot up with the USB target disk Ubuntu, it will modify the bootloader entry and cause previously installed Ubuntu unable to boot. It is possible to rename previous Ubuntu bootloader in some tricky way so that both can live peacefully. That probably warrant another blog.

Posted on

Set Up Ubuntu/XFCE4 on Headless Raspberry Pi 4 via VNC

Background

The new Raspberry Pi 4 with 8GB RAM is a little attractive  beast that is suitable for  home server.  You can set it up basically fan-less.  I was intrigued and I wanted to set it up headless, running Bitcoin full node, ElectrumX server plus Lightning Network (see my next post).

However, while there are many tutorials around Internet, I just can’t find the ideal one that suit me:

  • It must be 64bit.  Come on!  We are in 2021 already.
    • That rules out 32bit Raspberry Pi OS.
    • The 64bit Raspberry Pi OS (beta) runs fine on the board, but it lacks some basic Linux/UNIX features (e.g., middle mouse button pasting).  The desktop settings keep getting lost.  It is just frustrating.
  • That leaves 64bit Ubuntu, which should be a decent base OS.
  • I want to boot from USB without slow SD card.
  • I want to run headless with remote graphic UI 

The hardware

Below are the components I purchased.  The total cost is $270.  Not exactly dirty cheap, but that is pretty meaty piece. 

Putting it together is simper than setting up a toaster machine. See a couple of pictures below as well.

Step 1 – Prepare SSD drive and USB boot

  • Download and install rpi-imager on your host PC (Linux, Windows, etc)
  • Flush raspberry pi OS into SD card (I used 64bit RPIOS beta, but I think 32bit one should work). This will be a throw-away stepping stone.
  • Insert the SD card to RPI4 and boot from SD card.
    • Insert USB keyboard/mouse into black USB slot (USB2)
    • Connect a HDMI cable to a monitor
    • Use USB type-C power supply, ideally > 2.4A
  • Install (if not already installed) and run rpi-imager on RPI4 booted from SD card.
  • (IMPORTANT) Choose “Ubuntu 20.10 64bit Server” OS image and flush onto the 1TB Samsung SSD drive.
    • I found Ubuntu 20.04LTS don’t work!
  • Follow this link to update EEPROM firmware
  • Follow this link to boot from USB (i.e., SSD driver) first instead of SD card.
  • Remove SD card, and now you should be able to boot from SSD drive.

Step 2 – Fix up the board

  • Connect an Ethernet cable to enable Internet
  • Fix sluggish USB mouse
    • Add “usbhid.mousepoll=0” to /boot/firmware/cmdline.txt; Reboot
  • Install argon 1 fan control
    • curl https://download.argon40.com/argon1.sh | bash

Step 3 – Install XFCE desktop

Note choose “lightdm” when install xfce4

sudo apt update
sudo apt dist-upgrade
apt install xfce4 xfce4-goodies
apt install tigervnc-standalone-server novnc firefox

Create /etc/lightdm/lightdm.conf file:

[Seat:*]
allow-guest=false
autologin-user=ubuntu
user-session=xfce
#greeter-session=unity-greeter
xserver-command=/etc/X11/xinit/xserverrc

After this you should be able to log into graphic XFCE desktop.

Step 4 – Set up VNC and headless

  • Configure wifi assuming that is what you want to run RPI4
  • Set up VNC password
    • sudo vncpasswd /etc/vncpasswd
  • Modify /etc/X11/xinit/xserverrc file
#!/bin/sh

ARG=`echo $@ | sed -e "s#vt. -novtswitch##"`
GEOMETRY="1600x1000"
DEPTH="24"
# single client mode; VNC password auth; TLS encryption optional;
exec Xvnc -desktop ubuntu -SecurityTypes "VncAuth,TLSVnc" -NeverShared -DisconnectClients -geometry $GEOMETRY -depth $DEPTH -rfbauth /etc/vncpasswd $ARG
  • Make sure WiFi is connected before user logs in so that you can connect via VNC
    • To do this, launch NetworkManager, select “Applications”/”Settings”/”Advanced Network Configuration”
    • Then select the wifi network you are connecting to, and then click on button at the bottom to edit the connection
    • Under “General” tab, check “All users may connect to this network”

After this setup, if you reboot RPI4, you will get text console on display.

Meanwhile you should be able to access RPI4 over VNC remotely

  • On your PC, install VNC viewer (e.g., tigerVNC viewer)
  • Launch the viewer, enter the IP address and port 5900
  • Enter the VNC password
  • You should see XFCE desktop

Step 5 – Optional noVNC to enable web browser access

  • Install noVNC package
    • sudo apt install novnc net-tools
  • Run /usr/share/novnc/utils/launch.sh
  • On PC, open a browser and point at “http://<RPI4 IP address>:6080/vnc_auto.html”
  • Enter VNC password and you will be connected to the desktop

You can automate this process by adding this to /etc/rc.local or create a small systemd service file.

Posted on

Install WordPress on Ubuntu 18.04

As I said in my Home Page, I finally spent some time and migrated my ancient homepage to WordPress, the most popular web hosting platform (>60% of web sites).

However, quite contrarily to my expectations, the process is rather complicated. I went through quite a few hiccups, taking more than a couple of days, to reach today’s state. So I decide to write my experience down to hopefully save someone’s else pain.

For the Inpatients

  • Don’t follow the official instructions. It looks easy and simple, but WordPress is too old and does not seem to be easily updateable.
  • Don’t use WordPress docker container. A lot of trouble to get MySQL setup and connected. You still have a broken environment that many plug-in’s don’t work. In my case, I need to send emails (who don’t?), and that is VERY HARD to do with container.
  • I suppose Bitnami WordPress on AWS might be a fine choice. But I find more flexibility in just doing it myself, and it is actually not that hard, with a good instruction.
  • I ended up installing WordPress directory on Ubuntu 18.04
    • I install WordPress as a subdrectory of WWW root (wordpress/). This way I can migrate my original web site (in classic HTML) to WWW directory and co-exist with new shiny WordPress pages.
    • I chose Apache2 and MySQL. You can use Niginx and Mariabdb as well.

Installation

  • install required packages
apt update
apt install apache2 mysql php php-mysql
apt install php-curl php-gd php-mbstring php-xml php-xmlrpc php-soap php-intl php-zip php-gd
  • configure mysql
create database wordpress;
grant SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER on wordpress.* to 'wp_admin'@'localhost' identified by 'somepasswordofyourown';
  • install wordpress
cd /tmp && wget https://wordpress.org/latest.tar.gz
tar xzf latest.tar.gz -C /var/www/html/
mkdir /var/www/html/wordpress/uploads
chown -R www-data.www-data /var/www/html/wordpress/
  • create /etc/apache2/sites-available/wordpress.conf file:
<Directory /var/www/html/wordpress>
    Options FollowSymLinks
    AllowOverride Limit Options FileInfo
    DirectoryIndex index.php
    Order allow,deny
    Allow from all
</Directory>
<Directory /var/www/html/wordpress/wp-content>
    Options FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>
  • enable WordPress in apache2
sudo apache2ctl configtest      # test syntax
sudo a2ensite wordpress
sudo a2enmod rewrite
sudo systemctl restart apache2
  • Initialize WordPress
    • Open a browser and go to “http://<server ip/name>/wordpress”
    • Enter site name, database name (“wordpress”), database user (“wp_admin”) and password.
    • You are done! Well, sort of. The rest is about WordPress itself, which is another topic.

References

Posted on

How to Trio-Boot Linux/Windows

Background

As a typical programmer, my working environment is Linux (compiling, building and debugging).  Meanwhile my digital social life also depends on Windows (Skype, games, etc).  When you buy a new laptop, like I just did with Lenovo T490S, a big question is how to make both ends happy.

Common solution such as dual booting does not quite cut it because the context switching overhead is too big.  VM solution like VMware player or virtual box is another common solution, which gets pretty close.  You loose about 15% to 25% performance in Linux VM but you can access both of them at the same time.

I like to push the envelop a little bit further and implemented something what I call as Trio-boot:

    • You can dual-boot into either Linux or Windows host
    • Once you are in windows host OS, you can also use VMware Player to boot the *same* Linux as guest OS

By doing this it has the convenience of VM solution, while also preserves the performance and compatibility of native Linux host if needed.  Note whether you boot natively or start from VMware player, it is the same Linux environment, which is very important.

Steps At-a-Glance

  • Prepare Windows to leave disk space for Linux partition
  • Install Linux (e.g., Ubuntu 18.04) to the spare space and do dual booting
  • Boot into Windows and install VMWare Player 15
  • Create a 64bit Linux VM that includes a virtual disk (/dev/sda) and native Linux partition on the physical disk

Prepare Windows

  • Start “Create and format disk partitions” from “Control Panel”
  • Shrink the main windows partition to leave sufficient space for Linux
    • If the PC has been used for a while, you may find some non-movable files that prevent you from shrinking sufficient amount.  Google around for solutions.

Install Ubuntu 18.04

  • Download 64bit Ubuntu ISO image (https://ubuntu.com/download/desktop )
  • Use rufus to burn a Ubuntu 18.04 USB disk (16+GB) (https://rufus.ie/ )
  • Boot PC with Ubuntu 18.04 USB disk
  • Choose “Install Ubuntu” option when prompted
  • At “Install Type” page, choose “Something else” (See https://i.stack.imgur.com/KURnS.png)
  • Create a new partition to have ext4 fs type and mounted as root “/”
    • In my case it create a new partition called “nvme0n1p5”
  • Continue installation as usual

At the end you should be able to dual-boot Windows and Ubuntu

Start Ubuntu in VMware Player

The key of this step is to add 2 disks to the VM: 1 virtual disk which will install MBR and serves as the boot disk, and 1 physical partition that holds Linux which is already installed.  A theoretically simpler solution is to add both EFI partition and Linux to the VM and don’t use a virtual disk.  However, this approach does not always work.

  • Boot into Windows and install VMware Player
  • Create 64bit Linux VM with 0.1GB virtual disk
    • add a second hard disk, which is physical disk and select only Linux partition
    • I chose NVMe disk type to match my physical disk type, but that probably does not matter
  • Boot VM with Ubuntu CD image with “Try Ubuntu” option
  • Change root to native Linux partition
    • mount /dev/nvme0n1p5 /mnt
    • mount -B /sys /mnt/sys
    • mount -B /proc /mnt/proc
    • mount -B /dev /mnt/dev
    • chroot /mnt
    • fdsik -l   # it should show /sda/ as well as the nvme0n1 disk partitions
  • remove auto mount of EFI partition (since it is not available in VM enviroment of Linux)
    • vi /etc/fstab
    • add “#” to the line that contains “/boot/efi”
  • prepare /dev/sda
    • fdisk /dev/sda
    • create a new primary partition with default values, i.e., a linux partition that takes over the whole disk
      • Note by default it will initialize the disk partition table as a DOS-type disk with MBR boot.  Keep this setting.
  • Install grub on /dev/sda, grub-install /dev/sda
  • power off VM; disconnect CD-ROM; restart VM, and choose Ubuntu as startup OS
  • (optional) install vm tools to enable copy’n’paste and resizing desktop, etc
    • sudo apt install open-vm-tools open-vm-tools-desktop
  • (optional) Mount hgfs automatically on every boot
    • create /etc/rc.local file with the following code
#!/bin/bash
vmhgfs-fuse .host:/ /mnt/hgfs/ -o allow_other -o uid=1000
    • add executable attribute, “sudo chmod +x /etc/rc.local”
    • enable /etc/rc.local, “sudo systemctl enable rc-local”
  • restart VM

TIPS

  • always shut down VM completely before rebooting into Linux natively.  Otherwise you will be left with some inconsistent harddisk state.

Additional notes on 9/22, 2019

In my original settings, computer will boot from grub2, where I can choose to start windows or ubuntu.  Quite a few times, for some unknown reasons, the computer boot Ubuntu straight, which can cause some disasters if your windows’ vmplayer is playing the same ubuntu image and windows went hibernation.

To overcome this, I finally settled on the following:

  • Enter BIOS and set windows boot manager as the default startup OS.
    • I need to press ENTER + F12 on my Dell machine in order to boot Ubuntu natively
  • in Windows vmplayer, I edit grub.cfg and remove windows so that vmplayer would always boot up Ubuntu