Set Up Dynamic DNS for AWS EC2 Instance with Lambda Service

Background

It’s pretty stupid and annoying (maybe intentional?) that AWS provides DNS service (Route 53) but does not provide dynamic DNS to its own instances!!! This blog describes a method to achieve just that with various AWS features, namely EC2 instance events, Lambda functions and Route 53.

Specifically,

  • You must already host some domains with Route 53
  • You like to launch EC2 instances with public IP addresses
  • You like to assign some cool domain names to those EC2 instances, automatically
  • When those instances are stopped, you like those domain names are removed automatically

I have done this for over 5 or 6 years now. The blog is an attempt to capture what I did and keep my memory fresh! The approach was based on some early articles, most likely an early version of this one. However, I spent time to develop my own version which has diverged significantly now. For example, my version supports multiple domains and doesn’t use database. Also my blog will focus on using AWS console operations instead of using CLI.

Usage

Suppose you own a domain called mydomain.com and you are hosting it with Route 53. When you launch an EC2 instance, you can set the name tag as “ddns-fun.mydomain.com” during launch or startup time. See pictures below. After the instance starts running, you will automatically have a A-type DNS record “fun.mydomain.com” pointing to the instance’s IP address.

When you stop or terminate the instance, the DNS record will be removed automatically.

Overview of the process

It is relatively complicated. Below is an overview.

  • The central piece is a lambda function written in Python 3.x called ddns_lambda. This function will receive events when EC2 instances are started or stopped. It will examine the name tag or DNS records to determine whether it should add some DNS records or remove them.
  • In order for the ddns_lambda to run with right permissions and access the recourses, you will create an IAM policy, called ddns-lambda-policy and an IAM role, called ddns-lambda-role.
  • Lastly you will create an event triggering rule that monitors EC2 instance start/stop. When such event happens, ddns_lambda function will be called.

Let us dive in!

Create a policy for DDNS lambda role

  • Go to AWS/Services/IAM/Policies
  • Click on “Create policy” on the top-right
  • On “Specify permissions” page, choose JSON option and enters following code. The policy allows access to ec2 instance query, write logs, and full access to route53.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:Describe*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
  • Name the policy as “ddns-lambda-policy”
  • Click on “Create policy”. Done!

Create a role for DDNS lambda function

  • Go to AWS/Services/IAM/Roles
  • Click on “Create role” button on the top-right
  • Select “AWS services” (default) and pick “Lambda” as the target service
  • Next page, search and select “ddns-lambda-policy”
  • Next page, name role as “ddns-lambda-role”
  • Click on “Create role”. Done!

Create EC2 instance event trigger

  • Go to AWS/Services/Amanzon EventBridge/Rules
  • Click on “Create rule”
  • Name it as “ec2_lambda_ddns_rule”. Click on “Next”
  • Leaving most as default and scroll down to “Event pattern” section
    • “AWS service” – select “EC2”
    • “Event type” – select “EC2 Instance State-change Notification”
    • “Event Type Specification 1” – select “Specific state(s)”
    • Select “Running”, “terminated”, “stopped” 3 states
    • At this point, you should have event pattern shown as following. Then Click “Next”
{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["running", "terminated", "stopped"]
  }
}
  • For “Target 1”, select “AWS service”, “Lambda function”, “ddns_lambda”
  • Then just click “Next”, “Next”, “Create rule”. Done!

Create and test DDNS lambda function

Create function template

  • Go to AWS/Services/Lambda/Functions
  • Click on “Create function” on the top-right
    • Choose “Author from scratch”
    • Function name – enter “ddns_lambda” for example
    • Runtime – Choose any python3.x
    • Architecture – Choose “arm64” architecture for tiny bit of money savings
    • Execution role – choose “Use an existing role” and select “ddns-lambda-role”
  • Click on “Create function” at the bottom.

Upload the code

  • First, download the code and save this zip file
  • Go to AWS/Lambda/Functions/ddns_lambda, the function template you just created.
  • Click and select “Upload from”/”.zip file” menu item to import the code into source code console
  • IMPORTANT!!! Update zone_names variable with your own domain names. It supports multiple domain names as an array.
  • Scroll down. Update “Runtime settings”/”handler” to be “union-py3.lambda_handler”

Configure the function

  • Go to AWS/Lambda/Functions/ddns_lambda
  • Click on “Configuration” tab in “Code source” panel
    • Click on “General configuration”, change timeout to 100 seconds.
    • Clock on “Triggers”
      • Click on “add triggers”
      • Select “EventBridge (CouldWatch Events)”
      • Choose “Existing rules” and select “ec2_lambda_ddns_rule” created above
      • Click “Add”

Deploy and testing

  • Go to AWS/Lambda/Functions/ddns_lambda
  • in “Code source” panel, click on “Deploy” button. Done!

Now you can start to launch an instance, stop it or terminate and see if corresponding DNS records are added or removed. A few tips about debugging:

  • Watch lambda function logs
    • Go to AWS/Lambda/Functions/ddns_lambda
    • Click on “Monitor” tab
    • Click on “View CloudWatch logs”
    • I usually do “Live Tail” monitoring

That is it! Let me know what you think.

Set Up Your Own NordVPN Meshnet VPN Server

China great firewall is having a dreaded effect on my decision whether I should go visit. With a typical hacker fashion I decided to roll my own sleeves and take the matter under my own control – set up my VPN servers.

A long story short, two solutions emerge, OpenVPN and NordVPN meshnet. Both have some commercial backing. So it is not exactly under my own control in some sense, but the solutions are all free. This article talks about NordVPN meshnet.

Set up VPN Server on AWS Ubuntu 22.04

I mostly follow this page that describes well. Below are the the commands I used.

<register nordvpn account>  # max 10 devices are allowed

sh <(curl -sSf https://downloads.nordcdn.com/apps/linux/install.sh)

nordvpn login --token
nordvpn set technology nordlynx
nordvpn set meshnet on
nordvpn mesh peer list

nordvpn mesh peer inv send <email>  # invite others

nordvpn mesh peer routing allow <peer node>  # allow others to connect and route

# to stop/disable nordvpn
sudo systemctl disable --now nordvpnd

# to uninstall nordvpn
sudo apt-get --purge remove 'nordvpn*'

Set up on Ubuntu Client

The primary source information is at this page. Below are commands I used


<register nordvpn account>

sh <(curl -sSf https://downloads.nordcdn.com/apps/linux/install.sh)

nordvpn login --token
nordvpn set technology nordlynx
nordvpn set meshnet on

# accept invitation, if using other's server
nordvpn mesh inv list
nordvpn mesh inv accept <email of Server user, if needed>

# connect and route internet traffic via meshnet VPN server
nordvpn mesh peer list
nordvpn mesh peer connect <server node>

# status and disconnect
nordvpn status
nordvpn disconnect

Set up Android Client and Other Clients

  • install NordVPN Android app
  • Sign up and/or log into NordVPN account
  • (optional) If using others’ VPN server, accept their invitation first
  • Then follow this page to start VPN

Other platforms are most likely similar, but I have not tried them. See iOS page and macOS page. Note if you use someone else’s VPN servers, you will need to accept their invitation first so that you can see their servers on your peer list.

Also note NordVPN is mixing meshnet features with their own paid VPN services, which make the UI very confusing. Just follow this guide and steer clear from the paid subscriptions.

Build Spec Tools for CPU2006 Benchmark

CPU2006 is an old obsolete benchmark. But in modern days we may still need to build and run it. The biggest problem is usually in building the tools needed by benchmark itself, called spec tools.

Below are the steps I used to build spec tools for AArch64 (64bit ARM) and RISC-V 64 on Ubuntu OS (22.04 and 23.04).

  • Install Fortran compiler, sudo apt install gfortran
  • Obtain CPU2006 install ISO image (need license)
  • Untar and install the source tree (install_archives/cpu2006.tar.xz)
  • cd cpu2006/tools/src
  • obtain and update all the config.guess and config.sub files with the latest ones
wget -O config.guess 'https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD'
wget -O config.sub 'https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD'

Once build is done, copy the attached config files below to under config/ directory and run with them. For example,

runspec --config ubuntu_riscv64 --action=setup int fp
runspec --config ubuntu_aarch64 --noreportable --size=test --iterations=1 mcf gcc
runspec --config ubuntu_aarch64 --size=ref --iterations=1 int fp

Attached files:

Experiments on Ethereum Staking Upload Bandwidth

I have set up rocket pool Ethereum staking node for about 1 month now. I have to say the experience is relatively smooth and the support is great. In fact it is so smooth, setting up the node itself is not worth a blog. 🙂 So far it has already produced its first block.

The only issue is the upload bandwidth concern. It was using almost 5mbps, about half of my ISP service allowance. While technically this is fine, I feel I would be comfortable if I have more headroom. Plus, I plan to add minipools. So I looked around and found that I could reduce number of peers to reduce bandwidth. However, there is very little information on how much bandwidth can be reduced when you reduce number of peers. So I set out to do an experiment.

My ETH1 client is Besu (Java) and my ETH2 client is Lighthouse (Rust). BTW, I’m choosing clients purely based on the programming language.

By default Besu has 25 peers and Lighthouse has 80 peers. In week 1 of the experiment, I used the default peer numbers. In week 2, I reduced Eth1 peers to half, 13 peers. That did not yield too much bandwidth saving. In week 3, I reduced Eth2 peers to half, 40 peers.

See results below. The numbers are taken from router. Note I only focus on upload bandwidth, not only because that is the issue of concern, but also I have highly variable download bandwidth and the numbers are not reliable.

Rocketpool pool uses about 75% of total bandwidth. So the true percentage change is amplified by 33%. From the table it seems we can save about 25% upload bandwidth of Ethereum staking node when we slash peers to half for both ETH1 and ETH2 clients.

weekweek 1week 2week 3
ETH1 (Besu) peers251314
ETH2 (Lighthouse) peers808040
total upload (GB)327.4312.1266.4
daily upload (GB)46.844.638.1
mpbs4.334.123.52
% against week 1100%95%81%
staking node %100%93.5%75%

PS – A few weeks later, I added a second minipool. I was expecting the bandwidth increase. However, the daily upload bandwidth actually dropped to about 27.5GB (2.55mpbs). This is puzzling. It could be due to the drop of actual peers connected (which is around 33 now). Or it could be due to “maturing” nodes or connections? In any case I’m happy upload bandwidth does not appear to be an issue anymore. And most likely I will create 2 more minipool after the Ethereum Shanghai upgrade and Rocketpool LEB8 introduction.

Unboxing Ray-Ban Stories

Ray-Ban Stories is a smart sunglass product that is co-developed by Ray-Ban and Facebook (now called Meta). Yesterday I got one pair of them and tried it on for about one hour. This post summarizes my experience of setting it up, wearing it, using it and interacting with the app (“Facebook View”).

Unboxing

I was very impressed by the little details for unboxing. The shipping box can be used for return shipping, with returning address label included. The plastic wrap has a tab for tearing it apart. I have never seen that before! Similarly there is a tab for tearing a sealing label on the box. No more hassle looking for a knife or a pair of scissors!

The packaging felt premium. Not bad for a device sold for $299. Mine is actually $379 due to the transition lens. It came with a USB type-C cable and a charging case. That is pretty much it.

Tuning On

The tutorial guide was very illustrative. The first step is to turn on the device with a switch at the left corner of the glass (see pic below)

The initial BT pairing proved to be a disaster. The glasses refused to enter blinking blue LED mode. Several tries later it entered a blinking white LED mode, where BT pairing obviously cannot succeed. From there on, nothing seemed to work. I had to search online to do a factory reset. Skipping some details here, I believe I actually did a reset. BT pairing eventually worked as expected.

The app then wanted to do a firmware upgrade before anything else, which is kind of expected. However it complained about not enough battery. So I had to charge the glasses for another 20 minutes, a big laydown for an enthusiastic user.

Wearing the glasses is pretty comfortable. It does not feel like a burdensome gadget. The arms do feel a little rigid and thicker than normal. (ADDED on Dec 6th : the glasses tend to slip down. I ordered a nose pad set which hopefully can stop that.)

Using the Glasses

Glasses can be controlled by either voice (“hey, facebook. take a photo”) or pressing a button + tapping the arm. I found they are relatively intuitive.

I will cut it short and go straight to the likes and dislikes. Below is a list of likes:

  • App has a good tutorial guide
  • App has an easy montage feature to combine several video clips, even with music
  • App also has a flashback feature for animating a picture. I probably need a few more practice to master it.
  • It has a verbal warning when battery drops to 10%
  • Generally good image/video quality, except when taking shots indoors, perhaps due to low light
  • Phone can still communicate and control the device while it is being charged. Good.

Here are a few dislikes:

  • Is the initial BT pairing failure due to glasses already paired somehow? If so, would need a better cue to lead users. Factory resetting a device which is just out of box seems really harsh for an end user.
  • Not enough initial battery to do the initial firmware update. Must wait for 20 minutes before playing with it.
  • Why can’t we update the firmware while charging it? that should give sufficient power.
  • App has a “Facebook View is active” notification that is always on. What is the purpose? Tapping on it does not even bring up the app.
    • Instead, I might need a notification to remind me about the battery level of the glasses and turning it off if necessary.
    • ADDED on Dec 6th: After 2 days, I found this notification is REALLY annoying. It is there. It cannot be dismissed. And it is completely useless!
  • Using voice to stop the video recording is a little strange. There is no audio prompt after saying “Hey, Facebook”.
  • I myself don’t know whether a video recording is on or not. This is not a big problem because you can only record a 30-second clip and typically you will master the recording after a few trials.
  • Sometimes there is a audio sound played shortly after stopping a video. I still don’t understand what that is. Maybe telling me processing is done and I can start recording again? That sound does not always come.

Below are some pictures taken during that session. (Somehow I could not share the video due to WP limitations. *sigh*)

Last Words about Battery

I played with it intensively for about 1 hour and used up all the 34% of battery. During that period, I took 19 video clips (380 seconds total) and 17 pictures. That seems to be consistent with Ray-Ban website claiming about 6 hours of moderate usage and 3 hours continuous usage.

Save Bash Output to A File – Automatically

I often have a need to save the output a bash file to another file, e.g., log file. I know I could use redirection “>” or “tee”. But I would have to type it from command line. This post talk about doing it from within the script itself.

Google search does not yield much meaningful results. To save the hassle for me and potentially others. Here is the straight no-BS code.

#!/bin/bash

readonly file="output"
exec 1> >(tee $file)
exec 2>&1

Experience of Donation With Cardano

I just made a donation of 15 ADA to Cardano Forest project. I like share my experience and some thoughts.

The process is relative simple:

  • View the project page on web browser,
  • Click the donate button and copy the receiver address
  • Then switch to Yoroi wallet app on the phone
  • Click “Send” button, paste the receiver address, enter 15 ADA and click “continue” finish the sending.

First, this is really just a payment use case, which in theory it should not be too much different from, say, a PayPal-enabled transaction. However, there are 3 important differences.

  • No intermediary party needed. It is just one address paying another address. No platform company like PayPal or banks needed to facilitate the transaction.
  • No personal information exchange. None of email, name, or phone number get exchanged.
  • NFT token to ensure tracking of your donation to the true beneficiary. You will get a NFT token for every tree planted. Each ADA will plant 1 tree. People can still make flaws here, but the whole process is definitely more transparent and more trackable.

You can find out more at their web site. It is a good cause, and I encourage everyone to donate, not only for the cool and new experience, but also for its cause.

A few wishes that would make the experience even better.

  • I wish there is a “pay” button from the donation web page which triggers Yoroi mobile app directly. This is more of Android/iOS issue.
  • I wish the receive address and related qr code can embed the ADA amount, and perhaps even a short memo.
  • I wish Yoroi mobile app can scan a QR code from a picture, which is useful when you have the mobile phone for both web viewing and QR code scanning.

Please consider staking with us @ MYADA pool

Increase Root Volume Size for AWS EC2 Linux Instances

It turns out it is extremely simple to increase volume size for AWS EC2 Linux instances. In this article we use Ubuntu 20.04 as an example to show how it works in simple 3 steps, without restarting the instance.

  1. Increase volume size – Go to AWS console; find the volume used for the instance as the root device; Choose “Modify volume” action item; increase the size to the desired number
  2. Log into the AWS machine, type “lsblk” to verify the root device size has been increase. Also confirm that the partition size remains the same as before.
  3. Expand partition size to fill up the drive. For example, if root device is /dev/xvda and root partition is the first partition, you would run “sudo growpart /dev/xvda 1”. Run “lsblk” again to verify partition size.
  4. Resize filesystem to use the new space. For the previous example, one would run “sudo resize2fs /dev/xvda1”. Run “df -h /” to verify.

Viola! You are done!

Install Full Ubuntu on Portable USB Drive

Background

My goal is to install fully updateable Ubuntu 20.04 onto a USB stick, so that I can boot it up with any Intel-based PC’s or laptops. However, due to what I considered a bug in Ubuntu, this is actually harder than it should be. So I wrote down this blog in the hope it might help others, as well as my future self, in the similar shoes.

Note that the objective is different from so-called LiveUSB ubuntu with persistency, where Ubuntu OS itself will remain as a static ISO image and updates are added on-top in a separate persistent partition. My goal is to install a standard Ubuntu OS on a USB disk, which can be updated and upgraded just like normal PC case, except that a) it is on a portable USB drive or disk and b) it is portable across different PC’s. I suppose this setup gives longer life span of the installation, which potentially allows you to even upgrade your OS later.

In the following steps, I will also show an optional feature which creates an encrypted home directory.

Assumptions and Prerequisites

  • You need Intel x86_64 PC
    • We assume it support UEFI and GPT partitions which are standards for all recent ones
  • A USB drive that holds Ubuntu ISO image for installation, a.k.a. the installation media drive. This needs to have 4GB minimum size.
  • A second USB drive or disk that will hold installed Ubuntu OS, a.k.a. the installation target drive. This one needs 16GB minimum size

Step 1 – flash Ubuntu ISO image to the installation media drive

I will not repeat the process here. Please refer to many pages below.

Step 2 – Prepare the partition table on the target drive

  • Insert installation media drive into PC.
  • Interrupt normal booting sequence and choose the media USB drive as the boot device
    • Different PC have different process to do this. On Lenovo PC, one has to press ENTER on bootup, and then press F12 to select boot device
  • Select “Try Ubuntu” when presented the option
  • Insert target USB drive
    • Identify which drive is target USB drive by examining the output “lsblk”
    • In most case if you follow the instructions exactly, it would be “/dev/sdb”
  • Once Ubuntu is up and running, start a terminal and type “sudo gparted /dev/sdb” (replace “/dev/sdb” with the right usb device you have for the target USB drive)
    • create GPT partition table
      • click “Device”/”Create Partition Table …”
      • select “gpt” as partition table type
      • See Pic #1 below
    • create 100MB fat32 partition as ESP partition
      • Click “Partition”/”New”;
      • Enter “100MB” as size and select “FAT32” as file type
      • See Pic #2 below
    • set “esp”, “boot” attributes to the new ESP partition
      • Apply changes to actually crate the partition
      • Select the ESP partition and then select “Partition”/”Manage Flags”
      • In the pop-up window, select “esp” and “boot” flag
      • See pic #3 below
    • create an ext4 partition that takes the rest of space for root partition
      • See pic #4 below
    • (optional) if you like to have encrypted home partition, create an ext4 root partition with size of 10GB or more, and leave the rest free space open for encrypted home petition later.
      • Pic #5 shows the partition table at this step.
Pic #5 – after create ESP and root partitions

Step 3 – Install Ubuntu

Once we finish the above step, quit gparted and we are ready to install Ubuntu into the target USB drive.

  • click “install ubuntu” icon on the desk to to start installation
    • select “something else” in partition page. See Pic #6 below.
    • select the ESP partition on the target USB drive as the “ESP” partition. See Pic #7 below.
    • select root partition and mount as “/”. See Pic #8 below.
    • (optional) Create encrypted physical volume
      • Select the free space left during creating partitions
      • Click “+” to create a new partition/volume
      • select “encrypted physical partition”.
      • See Pic #9 below.
    • wait for a while, select “/dev/mapper/sdb3_crypt” as “/home”. See Pic #10 below
    • Finish installation.

Step 4 – Fix EFI on target USB drive

At this point you might get an illusion everything is working, because if you reboot the PC you will be able to select either Ubuntu or Windows to boot up, and they all work. However, there are 2 very serious problems

  • If you boot into BIOS and select the USB disk as boot device, it won’t work.
  • Even worse, your PC is likely not able to boot up Windows either if you remove the target USB disk.

The reason for these problems is that, despite we told Ubuntu installer to install Ubuntu on the USB disk, which implies it should use the ESP partition on target USB disk, it still uses the ESP partition on PC built-in disk, and thus screws up EFI partition on PC and leave an empty EFI on target USB disk. See more details at this very old bug report.

So the first thing we need to do is to install Ubuntu loader into USB ESP partition and install grub into USB disk

  • Reboot into BIOS firmware and select “ubuntu” as the boot target
  • switch EFI mount
    • Type “lsblk” to verify that ESP partition on PC built-in drive is mounted
    • umount it, “sudo umount /boot/efi”
    • mount the right one, ” sudo mount /dev/sda1 /boot/efi”
  • install grub boot on USB EFI partition
    • “sudo modprobe efivars”
    • “grub-install -d /usr/lib/grub/x86_64-efi –efi-directory=/boot/efi/ –removable /dev/sda”
    • Note “–removable” flag is important in the above command as it allows USB to boot on any Intel-based PCs
  • Reboot into BIOS firmware and select target USB as boot device. It should work now.
    • You can try the USB disk on other PCs, and it should work as well.

At this point, /etc/fstab is probably mounting PC EFI partition to /boot/efi, which is wrong and will break when you boot the target USB from another PC. You can either delete the /boot/efi line in /etc/fstab, or replace the UUID with the one for EFI partition on target USB. You can find out the UUID with “blkid” command (e.g., blkid /dev/sda1)

Step 5 – Restore PC boot loader

Now let us fix PC does not boot up problem (if you encounter this)

  • Boot up the PC into BIOS and select the newly made Ubuntu USB drive as boot device. This will boot into Ubuntu.
  • Open a terminal
  • Remove Ubuntu from PC EFI partition, and PC will boot up windows again
    • Mount host EFI partition as /mnt, “sudo mount /dev/nvme01n1p1 /mnt”
    • “cd /mnt/EFI/”
    • “sudo rm -rf ubuntu”
  • NOTE 1: If your PC is already installed with another Ubuntu system, the bootloader entry “ubuntu” will collide with each other and the previous Ubuntu will not be able to boot. You can follow this guide to restore booting the previous Ubuntu system.
  • NOTE 2: Annoyingly, if your PC is already installed with another Ubuntu system, each time you boot up with the USB target disk Ubuntu, it will modify the bootloader entry and cause previously installed Ubuntu unable to boot. It is possible to rename previous Ubuntu bootloader in some tricky way so that both can live peacefully. That probably warrant another blog.

Clone a Website Behind Login with WinHTTrack

I have an very old Fedora server that stopped active running almost 10 years ago. I need to backup its data for archive and physically dispose it.

It used to run wiki site based on moinmoin. If I just back its static files plus database files, chances are I will never be able to re-active those wiki pages and see them again. So I thought a better idea is to clone the web pages and turn them into static local web site, which I can still access easily.

It turns the journey is more complicated than expected. It took me more than a couple of hours to finally nail it down. So I figure it is worth a post here.

The Software

A quick search shows HTTrack seems to be best software for this job. It has a GUI version for Linux, called WebHTTrack. It also has a windows version called WinHTTrack. During my fiddling around, I ended up using WinHTTrack. In retrospect, the method used here should also work for WebHTTrack.

The Challenge

WinHTTrack is generally user friendly. The biggest problem is moin wiki requires a user login to view the content.

The cookie capture method (“Add URL” followed with “Capture URL”) does not work for moin wiki. Only first page works. Later pages still returns “You are not allowed to view this page”

The Solution

Step 1 – prepare cookies.txt file

  • Install and launch firefox browser
  • Install cookies.txt extension
  • Log into the web site and start browsing
  • Click on “cookies.txt” extension and export cookies.txt for the current site.
  • Also note the firefox user agend
    • Clock on top-right “settings” icon
    • Then click on “Help”/”Troubleshooting Information”
    • Note the “User Agent” string. We will use it later.

Step 2 – run WinHTTrack once

  • Start WinHTTrack and set up the project normally without worrying about login
    • Specifically, you don’t need to use “Add URL” button to do anything special. Just type in the URL in the text input area.
  • Click on “Finish” to start downloading
    • It will warn the site seems empty. That is expected.

Step 3 – copy over the cookies.txt

  • Copy the exported cookies.txt file to the top level directory of the local clone.
    • For example, if HTTrack working directory is “%USER%\Documents\httrack-websites” and your project name is “erick-wiki”, then the destination directory is “%USER%\Documents\httrack-websites\erick-wiki”

Step 4 – run it again with proper options

A few options had to be set properly

  • Click on “Set options …” and a new pop-up window shows up
  • On “Spider” tab, set “Spider” to “no robots.txt rules”
  • On “Browser ID” tab, set “Browser Identity” to the “User Agent” of Firefox browser noted in step 1
  • On “Scan Rules” tab, click on “Exclude links”, another pop-up windows shows up
    • Here you can exclude the files you like to clone
    • Most importantly, you need to exclude the “logout” URL. Otherwise cookies.txt will be updated/deleted and you will note be able to continue cloning
      • In my case, the URL will end with “?action=logout”. So I set “criterion” to “Links containing:”, and set “String” to “action=logout”
  • (Optional) On “Experts Only” tab, change “Travel mode” to “can go both up and down”
  • (Optional) On “Flow control”, set “number of connections” to higher number for faster cloning. Note: your website may have surge limit as Moinmoin does, in which case you will need to turn off this feature on the web server in order to increase bandwidth.
  • Once all set, click on “Next” and “Finish” to start cloning.

That is it!