Raspberry Pi: Image Distribution

I was tasked to setup all 17 Raspberry Pi’s in the robotics lab and change their hostnames for future SSH. All the Pi has Noobs installed on the SD card came with the package. The problem is, to install the package from NOOBS, I have to connect the Pi to a monitor, watch the thing go through a 20 minutes progress bar, and connect it to the WiFi. It is very not practical to do it for all devices.

The solution I came up with was pretty easy to think about but slightly hard to do –– that is, to flash a modified Raspbian image to every single Pi with things I need. It sounds hard, but in essence the Raspbian installation image is just a bunch of files, because in Linux everything is a file. I made the following modifications:

WiFi Configurations

Modifications to /etc/wpa_supplicant/wpa_supplicant.conf:

  • Added robotics-5 WiFi credential

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

network={
        ssid="robotics-5"
        psk="redacted"
        key_mgmt=WPA-PSK
}

Keyboard Configurations

Modifications to /etc/default/keyboard:

  • changed default keyboard layout from UK to US

# KEYBOARD CONFIGURATION FILE

# Consult the keyboard(5) manual page.

XKBMODEL="pc105"
XKBLAYOUT="us"
XKBVARIANT=""
XKBOPTIONS=""

BACKSPACE="guess"

Hostname Configurations

Modifications to /etc/hostnames:

  • Changed the hostname to pattern choate-robotics-rpi-xx, where xx is the device index.

choate-robotics-rpi-xx

Modifications to /etc/hosts:

  • Changed the local loopback hostname to match the contents in /etc/hosts

127.0.0.1   localhost
::1         localhost ip6-localhost ip6-loopback
ff02::1             ip6-allnodes
ff02::2             ip6-allrouters

127.0.1.1   choate-robotics-rpi-xx

Boot Configurations

Modifications to /boot/ssh/:

  • created file.

Modifications to /boot/config.txt:

  • Enable I2C

  • Enable SPI

# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
#disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=on
#dtparam=i2s=on
dtparam=spi=on

# Uncomment this to enable the lirc-rpi module
#dtoverlay=lirc-rpi

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

Device Setup

A pre-compiled Python3.7 zip file is placed under /home/pi.

A setup script (setup.sh) is placed under /home/pi/. It is meant to be run only once when the device starts up the first time. It performs the following tasks:

  • Install build-essential

  • Install vim

  • Install i2c-tools

  • Install smbus

  • Install Python3.7 (pre-compiled)

  • Install Python package RPi.GPIO

  • Install Python package gpiozero

  • Install adafruit circuit-python

  • Setup Python3.7 to be the default Python3 interpreter

  • Change log in password

  • Destroy all evidences of its existence

#!/bin/bash

cd /home/pi
sudo unzip python.zip
cd Python-3.7.3
sudo apt update -y
sudo apt-get install build-essential vim i2c-tools python-smbus -y
sudo make altinstall
sudo rm /usr/bin/python3
sudo rm /usr/bin/pip3
sudo ln -s /usr/local/bin/python3.7 /usr/local/bin/python3
sudo ln -s /usr/local/bin/pip3.7 /usr/local/bin/pip3
cd ..
sudo rm -r Python-3.7.3
sudo rm python.zip
sudo pip3 install gpiozero RPi.GPIO smbus adafruit-blinka
hostname | python3 -c 'h=input();print("pi:raspberry"+h.split("-")[-1])'| sudo chpasswd
echo "Setup Completed!"
rm "$0"

Raspberry Pi: Flashing Script

Again, it’s very laborious to flash so many Raspberry Pi’s by hand even with a custom installation image, because I still have to flash the disk image using dd, change the hostname of every single device (remember the placeholder xx at the end of the hostname configuration file?), change its password, run the setup script etc. So I made an installation script that can do everything above for me. All it takes is a single input, the device input. Because the installation script is meant to be ran on other computers, I did some careful disk checking. After all, nobody wants their entire hard-drive to be replaced by a stupid Raspian disk image. In addition, I used a hex editor to find the absolute location of the xx in the hostname in the disk image, so the flashing script can dynamically change the hostname of the device before it flashes the image by changing the particular 4 bytes at the target locations, which is the equivalent of mounting the disk image and edit both /etc/hostname and /etc/hosts. Upon finishing the flashing process, the script automatically ssh into the target Raspberry Pi and run the setup script (of course, someone has to plug in the SD card and power the Pi on). And everything is done. It takes about 10 minutes on my computer to run the whole thing, and about 30 minutes on an older computer. The speed limiting factor here is the IO speed of the SD card. Also, because dd by default produces no output. The script sends SIGINFO to the dd process periodically and parses the output to provide a clear indication of the progress.

Anyway, here is the installation script:

#!/usr/local/bin/python3.7
# -*- coding:utf-8 -*-
# Author: Jerry Wang

HOSTNAME_LOC = 148619264
HOSTS_LOC = 185983095
DISK_IMAGE_NAME = 'raspbian.img'

import platform, os, sys, subprocess, multiprocessing, signal, re, time, plistlib, fcntl

os.system('clear')
print(
"""
This is the installation script for deploying a modified raspbian image. It may not be tested thoroughly and use under your own risk.
""")

if platform.system() != 'Darwin':
print('This script should only be ran on macOS.', file=sys.stderr)
sys.exit(1)

if os.getuid() != 0:
print('Root privilege required. Please re-run this script with sudo', file=sys.stderr)
sys.exit(1)

os.chdir(os.path.dirname(os.path.abspath(__file__)))

image = open(DISK_IMAGE_NAME, 'rb+')
image.seek(HOSTNAME_LOC)
hostname_prefix = image.read(20)
assert hostname_prefix == b'choate-robotics-rpi-'
index = int(input('Please enter the index for the machine. It should be a single integer: '))
assert 0 <= index < 100
index = str(index).zfill(2).encode('ascii')
print('The generated hostname is %s.' % (hostname_prefix + index).decode('ascii'))
hostname = hostname_prefix + index
image.write(index)
image.seek(HOSTS_LOC)
hosts_prefix = image.read(20)
assert hosts_prefix == b'choate-robotics-rpi-'
image.write(index)
image.seek(HOSTNAME_LOC)
assert image.read(22) == hostname
image.seek(HOSTS_LOC)
assert image.read(22) == hostname
image.close()

print('-------------DISK IMAGE MODIFICATION COMPLETED-------------')

print('Waiting for the SD card.')
print('If the SD card is connected, please disconnect and reconnect it.')

drives = set(filter(lambda s: s.startswith('rdisk'), os.listdir('/dev/')))

while True:
new_drives = set(filter(lambda s: s.startswith('rdisk'), os.listdir('/dev/')))
if new_drives - drives != set():
    break
time.sleep(0.1)
drives = new_drives

for d in new_drives - drives:
res = re.match(r"^rdisk\d+$", d)
if res:
    partition_name = d
    break
else:
print('ERROR: Cannot match the name of the new drive')
sys.exit(1)

print('Target partition %s detected.' % partition_name)

commands = ['dd', 'if=%s' % DISK_IMAGE_NAME, 'of=/dev/%s' % partition_name, 'bs=1m', 'conv=sync']

proc = subprocess.run(['diskutil', 'list', '-plist', '/dev/' + partition_name], check=True, capture_output=True)
plist = plistlib.loads(proc.stdout)
disks=plist['AllDisksAndPartitions'][0]['Partitions']
try:
assert disks[0]['VolumeName'] == 'boot'
assert disks[1]['Content'] == 'Linux'
except AssertionError:

if disks[0]['Content']=='Windows_FAT_32' and disks[0]['Size']==32006733824 and disks[0]['VolumeName']=='CANAKIT':
    pass
else:
    print('The new device does not appear to be an Raspberry Pi installation card. Please re-run this script.')
    sys.exit(1)

proc = subprocess.run(['diskutil', 'list', '/dev/' + partition_name], check=True, capture_output=True, text=True)
print('-----------------------------------------------------------')
print(proc.stdout)
print('-----------------------------------------------------------')
print('Please confirm the target disk information. ')
print('ALL CONTENTS ON THE TARGET DISK WILL BE ERASED AND OVERWRITTEN!')
print('COMMAND TO EXECUTE: ', ' '.join(commands))
input('Press Ctrl-C to quit, press Enter to continue.')
print('-----------------------------------------------------------')

try:
print('Unmounting disk...')
os.system('diskutil unmountDisk /dev/%s' % partition_name)
pipe_r, pipe_w = os.pipe()
os.set_blocking(pipe_r, False)
proc = subprocess.Popen(['dd', 'if=%s' % DISK_IMAGE_NAME, 'of=/dev/%s' % partition_name, 'bs=1m', 'conv=sync'], stdout=pipe_w, stderr=pipe_w)
fize_size = os.path.getsize(DISK_IMAGE_NAME)
while proc.poll() is None:
    try:
        os.kill(proc.pid, signal.SIGINFO)
        time.sleep(0.5)
        out = os.read(pipe_r, 200).decode('ascii')
        match = re.search(r'(\d+) bytes transferred .* \((\d+) bytes/sec\)', out)
        if match:
            percentage = int(match.group(1)) / fize_size * 100
            speed = int(match.group(2)) / 1024 / 1024
            print('\r%.2f%% completed. Speed: %.2f MiB/s' % (percentage, speed), end='')
        time.sleep(0.5)
    except BlockingIOError:
        time.sleep(0.5)
print()
except:
proc.terminate()
raise
print('Ejecting the SD card...')
os.system('diskutil eject /dev/%s' % partition_name)

print('------------------SD CARD IMAGING COMPLETED------------------\x07')
proc = subprocess.run(['/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport', '-I'], capture_output=True,
                  encoding='ascii')
out = proc.stdout
res = re.search(r'\n\s*SSID: (.+)\n', out)
print('Checking connection...')
if res:
if res.group(1) != 'robotics-5':
    print('Current WiFi: %s' % res.group(1))
    print('Switching to robotics-5 ... ', end='')
    sys.stdout.flush()
    if os.system('networksetup -setairportnetwork en0 robotics-5 password-redacted') == 0:
        print('Success')
    else:
        print('Failed')
        print('Please manually connect to robotics-5 WiFi.')
print()
print('Please disconnect the SD card and put it into the Raspberry Pi,')
print('and power it up. Attempting to connect to the Pi...')

hostname=hostname.decode('ascii')
index=index.decode('ascii')
while True:
proc = subprocess.run(['ping', hostname + '.local', '-c', '4', '-t', '1'],capture_output=True)
if proc.returncode != 0:
    time.sleep(0.5)
else:
    break

command = r"""expect -c 'spawn ssh -o StrictHostKeyChecking=no -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null [email protected]%s.local "bash setup.sh"; expect "assword:"; send "raspberry\r"; interact'"""
if os.system(command % hostname) == 0:
os.system('clear')
print('Setup completed. The password for your Pi is raspberry%s\nEnjoy!' % index)
os.execv('/usr/bin/ssh', ['/usr/bin/ssh', '[email protected]%s.local'%index])