stv0g's weblog | Blog

A New Home for My Open Source Projects: Embracing Codeberg

Thu, 18 Dec 2025 00:00:00 GMT

More than 14 years ago, I migrated my 26 open source code project to GitHub. Back then GitHub was a fresh and modern platform pushing the boundaries of collaborative software development. Over the years, however, my perspective on GitHub has changed significantly.

I have recently migrated my by now 79 repositories from GitHub to Codeberg, a decision driven by several significant factors.

Concerns Regarding Generative AI

The pervasive trend of generative AI is impacting many corporations, including Microsoft and by extension, GitHub. There are widespread concerns about the disregard for existing copyrights held by creators, artists, and scientists, as well as the substantial environmental footprint of these technologies. While generative AI may offer some legitimate applications, it is frequently being employed for less constructive purposes: degrading existing services, exacerbating data harvesting, and increasing surveillance. I view the current trajectory of “AI” development with significant apprehension. GitHub’s own offering, CoPilot, exemplifies these concerns. By incorporating code from numerous developers without regard for licensing terms, it generates output that is often suboptimal and may infringe on original authorship and licensing. This has led to the emergence of “AI Policies” within projects to manage the influx of automated, low-quality contributions. I am unwilling to have my code used in this manner, particularly without proper attribution or respect for open source principles like the GPL. This is fundamentally at odds with the spirit of open source collaboration.

Prioritizing Data Privacy

GitHub’s ownership by Microsoft places it within the sphere of potential US government influence. This means that data held by Microsoft could be subject to disclosure orders from the US government, regardless of the legality or ethical implications.

A Stance Against Authoritarianism

My aim is to minimize my reliance on US-based products and services as a personal protest against what I perceive as authoritarian tendencies. While I can generally accept services from various nations, the US stands out due to perceived hostility towards European values, freedoms, rights, and ways of life. Furthermore, major tech corporations, including Amazon, Microsoft, and Google, appear to align themselves with and financially support the current political climate, which I view as problematic. Consequently, I am hesitant to utilize services associated with such entities.

Supporting Open Source Alternatives

GitHub’s proprietary, closed-source infrastructure undermines data sovereignty by locking users into systems they cannot audit, modify, or self-host. Users have no control over how their data is processed, stored, or potentially exploited for profit-driven features, creating dependency on a single corporate entity’s decisions and practices. Codeberg, on the other hand, is built on Forgejo, an open source code forge that emphasizes user privacy and data sovereignty.

The Transition to Codeberg

As a result of these concerns, I have moved my projects to Codeberg: codeberg.org/stv0g. For already two years, I am supporting Codeberg e.V. as a regular member. By now migrating my projects Codeberg, I am aligning my open source work with a platform that shares my values regarding privacy, data sovereignty, and respect for open source principles.

Codeberg’s provides a reliable and privacy-respecting source code hosting for open source projects. While in most cases still accepted, this excludes private repositories, or projects not licensed under an open-source license, such as this blog’s source code. For this reason I also operate a self-hosted Forgejo instance at git.0l.de for my private and non-open source projects.

Freeing a Xiaomi Humidifier from the Cloud

Fri, 28 Nov 2025 00:00:00 GMT

I recently moved into a new apartment which I used as an opportunity to make our home a little smarter. As a big open source supporter I built my smart home platform with Home Assistant of course.

Unfortunately, there are still far too few products that are directly compatible with Home Assistant. Especially in the area of humidifiers where I only found products that rely on a proprietary app or cloud from the manufacturer. Something that I would like to avoid at all costs. For one thing, such dependence is a certain form of planned obsolescence, as the product becomes useless as soon as the app loses its compatibility with new smartphone operating system versions or the manufacturer’s cloud is no longer operated.

Therefore, it was important for me to find a smart humidifier that integrates directly with my Home Assistant setup. To achieve this goal, I identified two options:

Add sensors / actuators to a classic humidifier to make it smart.
Replace the firmware of a smart humidifier with my own source code.

I decided to use the second approach, because it required less effort, since I would have had to implement my own firmware anyway.

Next, I was faced with the task of finding a suitable humidifier whose firmware I could easily replace. I specifically looked for devices that contained an ESP8266 or ESP32 microcontroller from Espressif, because for these I could easily create a new firmware with ESPHome.

ESPHome is a system that allows you to control your ESP8266/ESP32 through simple but powerful configuration files and remotely control it through home automation systems.

Xiaomi Mi Smart Antibacterial Humidifier.

Thanks to Sören Beye /Hypfer , I quickly became aware of the Xiaomi Mi Smart Antibacterial Humidifier, as Sören himself wrote his own firmware for this humidifier.

Unfortunately, his original version of the customized firmware ( /Hypfer/esp8266-deerma-humidifier ) is no longer compatible with the current version of the producr, as Xiaomi has modified the internal communication protocol.

Therefore, I decided to re-implement the firmware based on ESPHome as an “external component”.

You can find the code for this component here: /stv0g/esphome-config/components/xiaomi_deerma_humidifier

How to Hack your own Humidifier

This section is going to walk you through the process of modifying your own Xiaomi Humidifier.

Find the correct model

The internal Mi Model ID of the supported device is deerma.humidifier.jsq. The Model on the packaging being ZNJSQ01DEM.
Disassembly

There are 4 Philips-head screws hidden under the rubber foot ring which can be easily pealed of. Its usually find to remove the rubber only in areas where the screws are located. That way you can easily reattach it later.

Internals of the Humidifier.

Bottom view of Humidifier. Screws are behind rubber foot ring.

In the inside, you will find a small Wifi module attached to the back of the housing. Remove the module and solder on the wires as shown in the picture below.
Wire UART

Wifi board featuring an ESP-WROOM-02 module.

In the last picture, the colors of the wires correspond to the following:
- Orange: GND
- Grey: VCC (3.3 Volt!)
- Yellow: GPIO0
- Brown: RX
- Red: TX
In order to flash the module, you will need to tie GPIO0 to GND and attach a 3.3V serial adapter to the RX, TX pins. I recommend to also to disconnect the module from the humidifier and power it via your Serial to USB adapter via the VCC pin.

To flash the ESP8266 chip you can either use ESPHome’s build-in web flasher or esptool.py.

Create backup of original firmware

But before we flash the new firmware, I recommend to first make a backup of the original Xiaomi firmware by running:

esptool.py \
  --chip esp8266 \
  --baud 230400 \
  --port /dev/tty.usbserial-31310 \
  read_flash 0x0 0x200000 xiaomi-deerma-humidfier-original-2mb.bin

Flash new firmware

Afterwards, we can flash the new firmware:

esptool.py \
  --chip esp8266 \
  --baud 230400 \
  --port /dev/tty.usbserial-31310 \
  write_flash 0x0 deerma.bin

This blog has joined the Fediverse

Wed, 18 Oct 2023 00:00:00 GMT

ActivityPub Logo.

TL;DR

My blog noteblok.net has joined the Fediverse. You can follow my posts via this new handle: stv0g@noteblok.net.

This has been made possible by the Wordpress ActivityPub Plugin. With the ActivityPub plugin installed, the WordPress blog functions as a federated profile, along with profiles for each author. For example, my blog-wide profile can be found at @blog@noteblok.net. Authors like myself, on the other hand, would have their individual profiles at @stv0g@noteblok.net.

The integration allows following the blog from your own Fediverse platform and account like Mastodon. I return you can also react and comment to my blog posts via simply replying with your existing Fediverse account.

Fritz!DNS - An authoritative DNS server for AVM FRITZ!Box routers

Sun, 08 Jan 2023 00:00:00 GMT

Fritz!Box.

In my home network, I am using an AVM FRITZ!Box Cable 6690. It handles DHCP, DNS, Wifi and recently also interfaces my home network via WireGuard to my servers.

Just like the venerable Dnsmasq AVM’s FRITZ!OS uses hostnames learned from its DHCP leases and makes them resolvable via its internal DNS server.

Unfortunately, this feature in FRITZ!OS has some limitations:

The name of the DNS Zone is hard coded to fritz.box and can not be adjusted. Hence, the resolvable names have the following schema: myhostname.fritz.box
The internal DNS server only supports recursive DNS looks. It does not act as an authoritative DNS server. Hence the local zone can not be delegated.
AXFR zone transfers are not supported.

My solution to these shortcomings is Fritz-DNS which:

Is a small tool written in the Go programming language.
Is a small authoritative DNS server which serves A / AAAA resource records for local hosts connected to an AVM Fritz Box home WiFi router.
Can be used in a hidden master configuration as it supports AXFR zone transfers.
Uses the custom extension (X_AVM-DE_GetHostListPath) of the TR-064 Hosts SOAP-API as documented here to retrieve a list of local hosts.
Supports the generation of AAAA (IPv6) resource records based on the hosts MAC addresses using 64-Bit Extended Unique Identifier (EUI-64) and a configured unique local address (ULA) prefix.
Does not yet support PTR resource records (to be implemented…)
Is licensed under the Apache 2.0 license

You can find Fritz-DNS at Codeberg: /stv0g/fritz-dns .

Here is a small figure illustrating the interaction of Fritz-DNS with the Fritz!Box and other DNS servers / clients:

Fritz!DNS Architecture.

CLI Usage

$ fritz-dns
Usage of fritz-dns
  -ipv6-ula-prefix string
      Fritz Box IPv6 ULA Prefix (default "fd00::/64")
  -pass string
      FritzBox password
  -port int
      Listen port (default 53)
  -soa-expire duration
      SOA expire value (default 744h0m0s)
  -soa-mbox string
      SOA mailbox value
  -soa-minttl duration
      SOA minimum TTL value (default 1h0m0s)
  -soa-ns string
      Authorative DNS server for the zone
  -soa-refresh duration
      SOA refresh value (default 2h0m0s)
  -soa-retry duration
      SOA retry value (default 1h0m0s)
  -ttl duration
      default TTL values for records (default 5m0s)
  -url string
      FritzBox URL (default "http://fritz.box/")
  -user string
      FritzBox username (default "admin")
  -zone string
      DNS Zone (default "fritz.box.")

Aachen wird Transparent!

Tue, 02 Aug 2022 00:00:00 GMT

Ich möchte Stadtpolitik in Aachen für alle verständlich machen. Mein aktuellstes Projekt aachen-transparent.de ermöglicht es, die öffentlichen Informationen aus dem städtischen Ratsinformationssystem modern und benutzerfreundlich aufzubereiten. Dazu habe ich das bereits existieren Open-Source Projekt Meine-Stadt-Transparent erweitert und für die Bedürfnisse in Aachen angepasst.

Screenshot von aachen-transparent.de.

Aachen Transparent ist ein Projekt, dass ich ehrenamtlich im Rahmen des Open Data Labs Aachen ins Leben gerufen habe. Es versucht einige der Unzulänglichkeiten des Ratsinformationssystems der Stadt Aachen zu umgehen. Dazu nutzt es dessen öffentliche OParl Schnittstelle um die dort hinterlegten Informationen über eine moderne Oberfläche zugänglich zu machen.

Dies ist besonders deshalb von Interesse, da das Aachener Ratsinformationssystem leider keine Indexierung von Suchmaschinen wie Google zulässt und Bürger daher bisher nur die recht beschränkte Suchfunktion der AllRis Software nutzen konnten. Aachen Transparent unterstützt und fördert explizit die Indizierung aller Anträge, Dokumente, Tagesordnungspunkte und Beschlüsse durch gängige Suchmaschinen und stellt auch eine eigene Volltextsuche zu Verfügung.

Funktionalität

Durch Meine Stadt Transparent erfahren Bürgerinnen und Bürger, wer für sie im Stadtrat sitzt, wann der Stadtrat tagt und welche Themen besprochen werden. Da alle Dokumente räumlich auf einer Karte verortet werden, sind leicht jene Themen erkennbar, die sich auf das unmittelbare Lebensumfeld beziehen.

Großen Wert habe ich auf eine intuitive und flexible Suche gelegt: Mit einer zentralen Suche lässt sich die gesamte Seite - Sitzungsvorlagen, Tagesordnungen, und vieles mehr - im Volltext durchsuchen, genauso wie gewählte Stadtratsmitglieder oder Ausschüsse. Um auch dauerhaft auf dem Laufenden zu bleiben, bietet „Aachen Transparent“ verschiedene Anschlusspunkte: Nutzer können sich E-Mail-Benachrichtigungen anlegen, die über neue Dokumente zu abonnierten Suchbegriffen informiere. Termine und Terminreihen lassen sich per iCal in eigene Kalendersysteme einbetten, die aktuellen Dokumente lassen sich auch im eigenen RSS-Reader lesen.

Aachen Transparent ist kein isoliertes System, sondern greift auf das existierende Ratsinformationssysteme der Stadt Aachen zurück, um von dort Daten automatisch zu beziehen und aufzubereiten. Die Verwaltung von Dokumenten erfolgt weiterhin über ein klassisches Ratsinformationssystem, zu dem Aachen Transparent eine moderne Oberfläche bietet.

Im Überblick

Volltextsuche durch alle Dokumente
Texterkennung in gescannten Dokumenten
Geo Referenzierung von Dokumenten
iCal Kalender für Integration in Outlook, Google und andere Kalender
RSS Feeds für Ausschüsse und Personen
Abonnierbare Suchen mit E-Mail Benachrichtigungen und RSS Feeds

Geo-referenzierte Dokumente

Interessant ist auch die Verortung von Dokumenten und Anträgen. Aachen Transparent durchsucht alle indizierten Text auf Straßennamen im Aachener Stadtgebiet und nutzt diese zur Georeferenzierung. Wie im folgenden Bild zu sehen kann dadurch eine Karte generiert werden, welche einen Überblick über die aktuellen Themen in der Stadtpolitik gibt. Diese Georeferenzierung ist zudem auch über die Suchfunktion nutzbar, sodass zum Beispiel eine Suche auf einen Stadtteil oder Straße eingeschränkt werden kann.

Karte mit verorteten Anträgen.

Informationen zu Rats- und Ausschussmitgliedern

Des Weiteren stellt Aachen Transparent eine Übersicht aller aktuellen Rats und Ausschussmitglieder bereit, die deren Mitgliedschaft in Ausschüssen zusammengefasst.

Personen Übersichtsseite.

Offene Schnittstellen

Aachen Transparent unterstützt auch bereits existierende offene Schnittstellen um Daten aus dem Ratsinformationssystem auch über andere Kanäle zu verarbeiten.

So unterstützen wir z.B einen iCal Export des es ermöglicht Sitzungskalender in andere Kalendersoftware (z.B Outlook oder Android Kalender) zu integrieren und diese kontinuierlich zu synchronisieren.

Neue Anträge und Beschlusssachen können zudem über RSS Feeds abonniert werden. Interessant ist dabei auch die Möglichkeit individualisierte Feeds basierend auf Suchanfragen zu erstellen.

Häufige Fragen

Wer steckt hinter Aachen Transparent?

Aachen Transparent ist Angebot das auf ehrenamtlicher Basis von mir im Umfeld des Open Data Labs Aachen betrieben wird. Es ist daher kein offizielles Angebot der Stadt Aachen.

Kann ich Aachen Transparent auch in meiner Stadt/Gemeinde einsetzen?

Ja! Aachen Transparent basiert auf der Open Source Software Meine Stadt Transparent, die gemeinschaftlich von mehreren Open Data Labs und interessierten Privatpersonen entwickelt wird. Daher ist es prinzipiell auch möglich eine separate Instanz der Software für ihre Gemeinde aufzusetzen.

Aktuelle Statistiken aus Aachen

Dateien: 11.628
Sitzungen: 3.439
Gremien: 165
Vorlagen: 18.367
Personen: 1.370

(Stand: 2. August 2022)

A highly available WireGuard VPN setup

Thu, 28 Jul 2022 00:00:00 GMT

WireGuard is a communication protocol and free and open-source software that implements encrypted virtual private networks (VPNs), and was designed with the goals of ease of use, high speed performance, and low attack surface.

My Journey to WireGuard

Tinc Logo.

I’ve been using it in my home lab setup since about 2020. When, in the end of 2021, it was finally merged into the Linux mainline with release 5.9, I started to replace my former Tinc-VPN setup with it. Tinc-VPN is another great open source VPN solution. Unfortunately, its development has stalled over the last years which motivated me to look for alternatives. In contrast to WireGuard, Tinc runs as a user-space daemon and uses tun / tap devices which adds a significant processing overhead. Like WireGuard, it is also using UDP for tunneling data, but falls back to TCP in situations where direct datagram connectivity is not feasible. Another big advantage of Tinc is its ability to form a mesh of nodes and to route traffic within it when direct P2P connections are not possible due to firewall restrictions. At the same time, this mesh is also used for facilitating direct connections by signaling endpoint addresses of NATed hosts.

Tinc's mesh capability.

This mesh functionality made Tinc quite robust against the failure of single nodes as usually we could route traffic via other paths.

Highly Available WireGuard server setup

To counteract this shortcoming, this blog post will present a highly available WireGuard setup using the Virtual Router Redundancy Protocol (VRRP) as implemented by the keepalived daemon.

That said, it is worth noting that this setup does will not bring back some of the beloved features of Tinc. Both meshing, the peer and and endpoint discovery features of Tinc are currently and will never be supported by WireGuard. Jason A. Donenfeld the author of WireGuard focused the design of WireGuard on simplicity, performance and auditability. Hence advanced features like the ones mentioned will only be available to WireGuard by additional agents / daemons which control and configure WireGuard for you. Examples for such are Tailscale, Netmaker and Netbird.

The setup presented in this post is a so called active / standby configuration consisting of two almost equal configured Linux servers running both WireGuard and the keepalived daemon. As the name suggest only one of those two servers will by actively handling WireGuard tunneling traffic while the other one stands by for the event of a failure or maintenance of the active node.

VRRP Wireguard Setup.

Requirements

Before get started some requirements for the setup:

2 Servers running Linux 5.9 or newer.
A working Wireguard configuration.
A local L2 network segment two which both servers are connected.
Upstream connectivity without NATing via gateway connected to the network segment (usually provided by your internet or hosting provider).
An unused address to be used as Virtual IP (VIP) which roamed between the two servers by VRRP.

An important point is here the assumption that we are running both servers in the same switched network segment as this is a requirement for VRRP. We are also assuming that the upstream gateway performs no NATing. This guide covers only IPv6 addressing. However all steps can be also adapted or repeated for a dual stack or IPv4-only setup.

Detailed steps

Here are some of the specifics for my setup which need to be adapted by you:

Server Key (same use by both servers)
- Private: YIEDx+A2ONo5+uv3mrk/p7ileL3T5QQ8hleQK0IYEEI=
- Public: XGubrkGtuECdvoykKeUiNMigk2onfLCPfEo9Im+vmx8=
Peer Key (In this example we only have a single peer)
- Private: OIbpWVIVVBOtWfwkmXkFRN7Q/jBdfYtsGt7j97aHx1Q=
- Public: 3NGl6gTOGs6ai0RE91VmVFgF+N4gw1EBG11KOeiKJAg=
Public Server Subnet: 2001:DB8:1::/64
- Gateway: 2001:DB8:1::1
- Virtual IP: 2001:DB8:1::2
- Server A: 2001:DB8:1:::3
- Server B: 2001:DB8:1::4
WireGuard Tunnel Subnet: 2001:DB8:2::1/64
- Server: 2001:DB8:2::1 (same used by both servers)
- Peer: 2001:DB8:2::2
Interface names
- Wireguard: wg1
- Upstream: eno1

Prepare servers

We start of preparing the two servers by installing WireGuard and keepalived:

sudo apt install keepalived wireguard-tools iproute2

Next we configure a WireGuard interface on both servers using exactly the same configuration file at /etc/wireguard/wg1.conf:

[Interface]
Address = 2001:DB8:2::1/64
PrivateKey = YIEDx+A2ONo5+uv3mrk/p7ileL3T5QQ8hleQK0IYEEI=
ListenPort = 51800

[Peer]
PublicKey = 3NGl6gTOGs6ai0RE91VmVFgF+N4gw1EBG11KOeiKJAg=
AllowedIPs = 2001:DB8:2::2/128
PersistentKeepalive = 25

Similarly, a reciprocal configuration file is needed on the client side which skip here for brevity. Before proceeding, we activate the interface on both servers:

systemctl enable --now wg-quick@wg1

wg show wg1 # Check if interface is up

Configuring Keepalived

Create a configuration file for keepalived at /etc/keepalived/keepalived.conf

global_defs {
    enable_script_security
    script_user root
}

# Check if the server the WireGuard interface configured
vrrp_script check_wg {
    script "/usr/bin/wg show wg1"
    user root
}

vrrp_instance wg_v6 {
    interface eno1
    virtual_router_id 52
    notify /usr/local/bin/keepalived-wg.sh

    state BACKUP # use BACKUP for Server B
    priority 99 # use 100 for Server B

    virtual_ipaddress {
    2001:DB8:1::1/64
    }

    track_script {
        check_wg
    }
}

Create a notification script for keepalived at /usr/local/bin/keepalived-wg.sh

#!/usr/bin/env bash

TYPE=$1
NAME=$2
STATE=$3
PRIO=$4

WGIF=wg1

case ${STATE} in
    MASTER)
        ip link set up dev ${WGIF}
        ;;

    BACKUP|FAULT|STOP|DELETED)
        ip link set down dev ${WGIF}
        ;;

    *)
        echo "unknown state"
        exit 1
esac

Now start the keepalived daemon:

chmod +x /usr/local/bin/keepalived-wg.sh
systemctl enable --now keepalived

Testing the fail over

In our configuration, Server A has a higher VRRP priority and as such will be preferred if both servers are healthy. To test our setup, we simply bring down the WireGuard interface on Server A and observe how the VIP gets moved to Server B. From the WireGuard peers perspective not much changes. In fact no connections will be dropped during the fail-over. Internally, the clients WireGuard interface renegotiate the handshake. However, that step is actually not observable by the user.

Run the following commands on Server A while alongside test the connectivity from the client side through the tunnel via ping -i0.2 2001:DB8:2::1:
```
# Check that keepalived has moved the VIP to interface eno1
ip addr show dev eno1

# Bring down the Wireguard interface
wg-quick down wg1

# Keepalived should now have moved the VIP to Server B
ip addr show dev eno1
```

Going further

In my personal network, I operate a Interior Gateway Protocol (IGP) to dynamically route traffic within and also towards other networks. Common IGPs are OSPF, ISIS or BGP. In my specific case, both Servers A & B run the Bird2 routing daemon with interior and exterior BGP sessions.

So how does the WireGuard HA setup interoperates with my interior routing? Quite well actually. As my notify script (keepalive-wg.sh) will automatically bring up / down the interface, the routes attached to the interface will be picked up by Bird’s direct protocol.

I am also planning to extend my WireGuard agent cunīcu ( /cunicu/cunicu ) to support the synchronization of WireGuard interface configurations between multiple servers.

Background

Surprisingly, the setup works by using Keepalived and does not require any iptables or nftables magic to rewrite source IP addresses. I’ve seen some people mentioning that SNAT / DNAT would be required to convince WireGuard to use the virtual IP instead of the server addresses. However, in my experience this was not necessary.

Another concern has been that the backup Wireguard interface still might attempt to establish a handshake with its peers. This would quite certainly interfere with the handshakes originated by the current master server. However, also this has not been proven to be the case. I assume the fact that our notify script brings down the WireGuard interface on the backup server causes them to cease all communication with its peers.

SSH Access for Netgear's Nighthawk M5 Mobile LTE/Router

Sun, 03 Jul 2022 00:00:00 GMT

In my previous post, I demonstrated how to gain root access by enabling a Telnet daemon via the routers AT-over-TCP interface. In this post I will close this gasping security hole by replacing the Telnet with a Secure Shell (SSH) daemon. Netgear’s firmware does not ship with a SSH daemon itself. So we first build a statically linked Dropbear instead of the rather heavy OpenSSH daemon.

Building Dropbear SSH

I’ve build a statically linked version of Dropbear using a Debian-based Docker image as it allows us to use the packaged cross-compiler toolchains by Debian:

Dockerfile

FROM debian:bullseye

RUN apt-get update && \
    apt-get -y install \
        wget tar bzip2 build-essential \
    gcc-arm-linux-gnueabihf \
    binutils-arm-linux-gnueabihf

RUN wget https://matt.ucc.asn.au/dropbear/releases/dropbear-2022.82.tar.bz2
RUN tar xvf dropbear-2022.82.tar.bz2

WORKDIR /dropbear-2022.82

ENV CC=arm-linux-gnueabihf-gcc
ENV CFLAGS="-DDROPBEAR_SVR_PASSWORD_AUTH=0"

RUN ./configure --host=arm-linux-gnueabhf \
    --disable-zlib \
    --disable-shadow \
    --disable-syslog \
    --disable-lastlog \
    --enable-static
RUN make PROGRAMS="dropbear scp" MULTI=1

RUN arm-linux-gnueabihf-strip dropbearmulti

With this Dockerfile we can build the image, create a temporary container and copy the resulting binary from the image to your local folder:

docker build -t dropbear .

id=$(docker create dropbear)
docker cp ${id}:/dropbear-2022.82/dropbearmulti ./
docker rm ${id}

Installing Dropbear

Now that we have a statically linked version of the SSH daemon, we will need to copy it to our target. I accomplished this by using netcat (nc):

On the target

mkdir -p /data/mod/bin
pushd /data/mod/bin

nc -l -p 1234 > dropbearmulti
chmod +x dropbearmulti

ln -s dropbearmulti dropbear
ln -s dropbearmulti scp

On the machine which builds Dropbear

nc <ip-of-target> 1234 < dropbearmulti

This is followed by installing a SystemD service which start the SSH daemon on system boot:

cat > /etc/systemd/system/dropbear.service <<EOF
[Unit]
Description=Dropbear SSH server
After=network.target

[Service]
Type=forking
ExecStart=/data/mod/bin/dropbear -R
PIDFile=/var/run/dropbear.pid

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now dropbear.service

Before you will be able to connect to the target, you will need to install an authorized_keys file. Password login is not supported.

mkdir -p /home/root/.ssh
cat > /home/root/.ssh/authorized_keys <<EOF
ssh-rsa AAAAB3NzaC1...GaoxPrQ== # replace by your SSH key
EOF

All that remains is a quick test:

ssh root@<ip-of-target>

Disable SSH daemon

In order to restore the security of the device we must also disable the Telnet daemon. There are in principle two options to achieve this:

Reverse the steps from my first blog post via the AT-over-TCP interface.
Use the iptables firewall to block access to the Telnet port

I’ve decided to go for the second option:

cat > /etc/systemd/system/block-telnet.service <<EOF
[Unit]
Description=Block Telnet access
After=network.target

[Service]
Type=simple
ExecStart=/usr/sbin/iptables -I INPUT -p tcp --dport telnet -j DROP
ExecStop=/usr/sbin/iptables -D INPUT -p tcp --dport telnet -j DROP

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now block-telnet.service

Having a detailed look at the Netgear Nighthawk M5 Mobile LTE/Router

Wed, 29 Jun 2022 00:00:00 GMT

After gaining root access to the device in the first post of this series, we will have a closer look at the device and its firmware.

This post is documenting some internals of the device which is not the most exciting stuff to read. I mainly collected it here for documentation purposes.

All information in this post has been collected from a device running firmware version NTGX55_12.04.12.00.

Software

Netgear’s firmware is Linux-based and uses quite a lot of common open-source tools. They provide all modifications to GPL licensed code via their support area: NETGEAR Open Source Software for Programmers.

From what I can tell only their user interface and configuration management is developed by Netgear themself apart from a bunch of binary blobs provided by Qualcomm which contains the modem firmware which gets loaded to the baseband processor.

One curiosity caught my eye: there is a running X server on the device. It is used by the front-panel display of the device. A custom application developed by Netgear uses Webkit’s engine to render the touch screen interface which just like the web UI is based on HTML and Javascript.

Here is an almost complete list of open source software components which I found on the device:

atk (v2.28)
Avahi (v0.7)
bash (v4.4.23)
base-files (v3.0.14)
BusyBox (v1.29.3)
conntrack-tools (v1.0.1)
D-Bus (v1.12.10)
ddclient (v3.8.1)
dhcpcd (v5.2.10)
DiG (v9.11.5-P4)
Dnsmasq (v2.85)
ethtool (v4.19)
font-config (v2.12.6)
freetype (v2.9.1)
glib (v2.58.0)
hostapd (v2.8-devel)
iproute2 (iproute2-ss140804)
iptables (v1.6.2)
iw (v4.14)
libcap (v2.25)
libnfnetlink (v1.0.0)
Linux Kernel (v4.14.117)
miniupnpd
mtd-utils (v2.0.2)
nettle (v3.4)
OpenSSL (v1.1.1b)
pimd (v2.1.8)
pppd (v2.4.7)
strace (v4.24)
SystemD (v239)
tinyproxy (v1.8.3)
util-linux (v2.32.1)
wireless-tools (v30)
wpa_supplicant (v2.9)
Xorg (v1.20.1)
xz (v5.2.4)

Basic facts

Lets first have a look at the Kernel version:

$ uname -a
Linux sdxprairie 4.14.117 #1 PREEMPT Thu Aug 19 23:42:26 UTC 2021 armv7l GNU/Linux

/ # cat /proc/version
Linux version 4.14.117 (oe-user@oe-host) (clang version 6.0.9 for Android NDK) #1 PREEMPT Thu Aug 19 23:42:26 UTC 2021

Apparently the firmware has been built by Open Embedded as indicated by the kernel notice “oe-user”.

There is also a /target file lying around. I assume that “sdxprairie” is Qualcomm’s name for the SDK / BSP which is used by Netgear.

$ cat /target
sdxprairie

The application processor of the Snapdragon X55 is a fairly low powered single-core ARM v7:

$ cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : Qualcomm Technologies, Inc SDXPRAIRIE
Revision        : 0000
Serial          : 0000000000000000

With around 780 MB of RAM:

$ free -m
             total       used       free     shared    buffers     cached
Mem:           781        387        393          0          0        142
-/+ buffers/cache:        245        535
Swap:          109          0        109

SoC details

Within the SysFS we can find some details about the SoC. More details about the meaning can be found in the Kernel documentation:

SysFS Entry	Value
`/sys/devices/soc0/accessory_chip`	0
`/sys/devices/soc0/chip_family`	0x5e
`/sys/devices/soc0/chip_name`	SDX55
`/sys/devices/soc0/family`	Snapdragon
`/sys/devices/soc0/foundry_id`	1
`/sys/devices/soc0/hw_platform`	MTP
`/sys/devices/soc0/image_crm_version`	:ntgrbc-fwbuild6
`/sys/devices/soc0/image_variant`	MAATANAZA
`/sys/devices/soc0/image_version`	00:BOOT.SBL.4.1-00231
`/sys/devices/soc0/machine`	SDXPRAIRIE
`/sys/devices/soc0/ncluster_array_offset`	0xb0
`/sys/devices/soc0/ndefective_parts_array_offset`	0xb4
`/sys/devices/soc0/nmodem_supported`	0xffffffff
`/sys/devices/soc0/nproduct_id`	0x410
`/sys/devices/soc0/num_clusters`	0x1
`/sys/devices/soc0/num_defective_parts`	0xd
`/sys/devices/soc0/platform_subtype`	Invalid
`/sys/devices/soc0/platform_subtype_id`	5
`/sys/devices/soc0/platform_version`	65536
`/sys/devices/soc0/pmic_die_revision`	131072
`/sys/devices/soc0/pmic_model`	65568
`/sys/devices/soc0/raw_device_family`	0x6
`/sys/devices/soc0/raw_device_number`	0xb
`/sys/devices/soc0/raw_id`	207
`/sys/devices/soc0/raw_version`	2
`/sys/devices/soc0/revision`	2.0
`/sys/devices/soc0/select_image`	0
`/sys/devices/soc0/serial_number`	27453XXXX
`/sys/devices/soc0/soc_id`	357
`/sys/devices/soc0/vendor`	Qualcomm

$ cat /sys/devices/soc0/images
0:
        CRM:            00:BOOT.SBL.4.1-00231
        Variant:        MAATANAZA
        Version:        :ntgrbc-fwbuild6
1:
        CRM:            01:TZ.FU.5.9-00147
        Variant:        EATAANBAA
        Version:        :CRM
11:
        CRM:            11:MPSS.HI.2.0.c3.5-00010-SDX55_CPEALL_PACK-1.403198.3
        Variant:        sdx55.gennatch.prod
        Version:        :ntgrbc-fwbuild6

Kernel command line

$ cat /proc/cmdline
noinitrd rw rootwait console=ttyMSM0,115200,n8 androidboot.hardware=qcom msm_rtb.filter=0x237 androidboot.console=ttyMSM0 lpm_levels.sleep_disabled=1 firmware_class.path=/lib/firmware/updates service_locator.enable=1 net.ifnames=0 atlantic_fwd.rx_ring_size=1024 pci=pcie_bus_perf rootfstype=ubifs rootflags=bulk_read root=ubi0:rootfs ubi.mtd=24 androidboot.serialno=105d0dc7 androidboot.baseband=msm

Kernel log

Unfortunately, I was not able to capture early kernel log messages. I assume those are only printed via a serial port and lost as the circular buffer for the kernel log has not been set up.

Full log: /stv0g/netgear-nighthawk-mobile/reveng/dmesg.txt

More details

Network interfaces: /stv0g/netgear-nighthawk-mobile/reveng/interfaces.txt (ip address show)
Devices: /stv0g/netgear-nighthawk-mobile/reveng/devices.txt (ls -l /dev)
Kernel modules: /stv0g/netgear-nighthawk-mobile/reveng/modules.txt (lsmod)
Running processes: /stv0g/netgear-nighthawk-mobile/reveng/processes.txt (ps aux)
Mounted filesystems: /stv0g/netgear-nighthawk-mobile/reveng/mounts.txt (mount)
Flash partitions: /stv0g/netgear-nighthawk-mobile/reveng/mtd.txt (cat /proc/mtd)
Open ports: /stv0g/netgear-nighthawk-mobile/reveng/ports.txt (netstat -plnt)
Ubi devices & volumes: /stv0g/netgear-nighthawk-mobile/reveng/ubinfo.txt (ubinfo -a)
Kernel config: /stv0g/netgear-nighthawk-mobile/reveng/config.txt (zcat /proc/config.gz)
Devicetree source: /stv0g/netgear-nighthawk-mobile/reveng/device-tree.dts (dtc -I fs -O dts /proc/device-tree)

Feel free to contact me if I missed any particular detail which is interesting to you.

A 3D-printed wall mount for Netgear Nighthawk mobile 5G/LTE routers

Sat, 25 Jun 2022 00:00:00 GMT

I have recently designed and printed a wall mount for Netgear’s Nighthawk mobile 5G/LTE routers. More specifically a Nighthawk M5 (MR5200).

I have been inspired by an existing commercial wall mount for the Nighthawk M-series routers by FTS Hennig GmbH:

My inspiration: the wall-mount and antenna adapter from FTS Hennig.

Unfortunately, the mount is with a price tag of around 50 € rather expensive. So I decided to use our new lab 3D-printer and try do design it myself usings AutoDesk’s Fusion 360 software.

My own 3D printed holder is released under a creative commons license at Codeberg: /stv0g/3d-printing/netgear-m5-wall-mount

The mount contains two three mounting holes which can be used for screwing it against a wall as well as some cutouts at the bottom for the accessibility of the TS9 antenna, USB-C and Ethernet ports.

My model rendered by AutoDesk Fusion 360.

For the TS9 antenna ports, I am using the following TS-9 to SMA adapters which can be screwed into the respective holes of the mount. This allows a permanent installation of an external 5G/LTA antenna while the router can be easily removed as the adapters align right with the connectors of the router.

Screw-in TS-9 to SMA Adapters.

Final print.

Gaining Root Access on Netgear Nighthawk Mobile 5G/LTE Routers

Sat, 25 Jun 2022 00:00:00 GMT

This blog posts covers the required steps to gain root access via Telnet on Netgear Nighthawk Mobile 5G/LTE Routers. Its the first post in a small series covering my experiences playing around with this device.

Last month I obtained one of Netgear’s latest mobile 5G routers, the Netgear Nighthawk M5 (model MR5200-100EUS) . Being one of the most expensive consumer 5G routers, I was lucky to get a fairly good second hand deal from eBay.

Netgear Nighthawk M5 mobile 5G/LTE Router.

The router is powered by Qualcomm’s® Snapdragon™ X55 5G Modem-RF system. Looking closer at the internals of the device by checking the FCC filing for the closely related American model MR5100, we can see that the system consists of a Qualcomm SDX55 chipset which combines both the mobile baseband and application processors.

Gaining Root Access

Gaining root access to the device is actually fairly simple in comparison to rooting modern Android-based devices. The router exposes an open TCP port providing an AT command interface. However, this port is only accessible via a tethered USB connection, not via Wifi.

Using this AT command interface, we can interact with the modem, unlock an extended command set which allows us enable a Telnet daemon.

Detailed Steps

Install the Sierra Wireless debug tools from bkerler ( /bkerler/edl )
```
sudo apt install python3 git
git clone https://github.com/bkerler/edl.git
cd edl
sudo python setup.py install
```
(More detailed installation instructions are covered in the README file of the repo.)
Connect your machine via USB-C to the Netgear router
Make sure to disconnect from the Netgear Wifi
Open a terminal an connect to the AT command interface via netcat (nc)

(Make sure not to miss the -c option as it will the enable nc to use the proper CRLF line-endings which are required for the AT interface).
```
nc -c 192.168.1.1 5510
```
Once connected to the AT command interface, you need to request a unlock challenge code by sending
```
AT!OPENLOCK?
```
The previous command will return a challenge code which we use to generate a corresponding response code via the previously installed sierrakeygen.py tool:
```
sierrakeygen.py -l <replace_with_challenge_code> -d SDX55
```
The previous command will print out another AT+OPENLOCK command which you need to copy verbatim back to your AT command session.
Run the following AT commands to enable the Telnet daemon
```
AT!TELEN=1
AT!CUSTOM="RDENABLE", 1
AT!CUSTOM="TELNETENABLE", 1
```
You can now close the AT command session by pressing Ctrl+C.
Power-cycle the Netgear Router to start the Telnet daemon

Voila, you can now telnet into the device via both the tethered USB-C cable or Wifi.
```
nc -c 172.23.156.129 23
��������
mdm 1623 sdxprairie
/ # uname -a
uname -a
Linux sdxprairie 4.14.117 #1 PREEMPT Thu Aug 19 23:42:26 UTC 2021 armv7l GNU/Linux
```
Disclaimer: Please be aware that the device security is now breached as all devices connected to the Wifi or USB can gain root access to the device. The root Telnet login requires no password.

Next steps

Before proceeding we should make sure that we can bring the device back to a secure state by replacing the Telnet by an Secure Shell (SSH) daemon. In one of the next posts of this series, I will be building a statically linked version of the Dropbear SSH server to replace Telnet.

Before continuing my reverse engineering efforts on the device, I would like to ensure that I will not brick the router while doing so by dumping the firmware and extract all the details from it. This will allow us to hopefully restore the device by flashing the original firmware. Maybe we will be able to run OpenWRT on it.

I have also designed a wall mount for the router which allows me to mount it permanently into by van.

Open-Data Lab Aachen

Wed, 22 Jun 2022 00:00:00 GMT

Aachener Rathaus.

Logo des Open Data Lab Aachen.

Im September letzten Jahres hat sich in Aachen das Open Data Lab mit einer virtuellen Kick-off Veranstaltung gegründet.

Im Open Data Lab wollen wir ehrenamtlich Projekte rund um Offene Daten in Aachen voranbringen. Wir suchen dazu Personen, die daran generell interessiert sind, ob Entwickler*innen, Designer*innen, Datenjournalist*innen aus Verwaltung, Politik und Gesellschaft.

Wir wollen Daten und Ideen zusammenbringen und daraus Projekte generieren.

Was ist Open Data?

Als Open Data werden Daten bezeichnet, die von jedermann zu jedem Zweck genutzt, weiterverbreitet und weiterverwendet werden dürfen. Einschränkungen der Nutzung sind nur erlaubt, um Ursprung und Offenheit des Wissens zu sichern, beispielsweise durch Nennung des Urhebers. Ausgenommen sind personenbezogene Daten sowie Daten, die anderweitig schutzwürdig sind.

Wer kann Mitmachen?

Im Prinzip ist jeder bei uns willkommen. Egal ob du bereits Vorkenntnisse hast oder nicht 😊.

Bürgerinnen
(Kommunal) Politikerinnen
Mitarbeiterinnen der Verwaltung
Wissenschaftlerinnen
Unternehmerinnen
Open Knowledge Labs & Digitales Ehrenamt
Datenjournalisteninnen
Studierende, SchülerInnen und LehrerInnen

Aktuelle Projekte

Die folgende Liste ist eine Übersicht aktueller Projekte, die in Aachen im Umfeld des Open Data Labs mit offenen Daten entwickelt werden:

https://oecher.info (@mrtopf)
https://baustellen.offenes-aachen.de (@mrtopf)
https://unserac.de (ALLRIS-Import, @mrtopf + @Unser_AC)
https://lokalpetition.de
/stv0g/cfac
/tapwork/AachenMobDashboard
https://regionaachenrettet.de
https://owi19.adfc-ac.de
https://verkehr.aachen.de
https://aachen-transparent.de
https://www.aachen-app.de

Mach mit

Interessierst du dich für offene Daten? Macht mit und lerne uns beim nächste Treffen kennen. Dieses findet bereits nächste Woche statt:

Datum: 12. April 2022
Uhrzeit: 19:00
Ort: Virtuell via Zoom

Weiterhin kannst du uns auch online über folgende Kanäle erreichen:

GoSƐ - A terascale file-uploader

Sun, 03 Apr 2022 00:00:00 GMT

GoSƐ Logo.

GoSƐ is a modern and scalable file-uploader focusing on scalability and simplicity.

It is a little hobby project I’ve been working on over the last weekends.

The only requirement for GoSƐ is a S3 storage backend which allows to it to scale horizontally without the need for additional databases or caches. Uploaded files a divided into equally sized chunks which are hashed with a MD5 digest in the browser for upload. This allows GoSƐ to skip chunks which already exist. Seamless resumption of interrupted uploads and storage savings are the consequence.

And either way both upload and downloads are always directed directly at the S3 server so GoSƐ only sees a few small HTTP requests instead of the bulk of the data. Behind the scenes, GoSƐ uses many of the more advanced S3 features like Multi-part Uploads and Pre-signed Requests to make this happen.

Users have a few options to select between multiple pre-configured S3 buckets or enable browser & mail notifications about completed uploads. A customizable retention / expiration time for each upload is also selectable by the user and implemented by S3 life-cycle policies. Optionally, users can also opt-in to use an external service to shorten the URL of the uploaded file.

Currently a single concurrent upload of a single file is supported. Users can observe the progress via a table of details statistics, a progress-bar and a chart showing the current transfer speed.

GoSƐ aims at keeping its deployment simple and by bundling both front- & backend components in a single binary or Docker image. GoSƐ has been tested with AWS S3, Ceph’s RadosGW and Minio. Pre-built binaries and Docker images of GoSƐ are available for all major operating systems and architectures at the release page: /stv0g/gose (Releases) .

GoSƐ is open-source software licensed under the Apache 2.0 license.

Live Demo

Screencast

GoSƐ Demo.

Features

De-duplication of uploaded files based on their content-hash
- Uploads of existing files will complete in no-time without re-upload
S3 Multi-part uploads
- Resumption of interrupted uploads
Drag & Drop of files
Browser notifications about failed & completed uploads
User-provided object expiration / retention time
Copy URL of uploaded file to clip-board
Detailed transfer statistics and progress-bar / chart
Installation via single binary or container
- JS/HTML/CSS Frontend is bundled into binary
Scalable to multiple replicas
- All state is kept in the S3 storage backend
- No other database or cache is required
Direct up & download to Amazon S3 via presigned-URLs
- Gose deployment does not see an significant traffic
UTF-8 filenames
Multiple user-selectable buckets / servers
Optional link shortening via an external service
Optional notification about new uploads via shoutrrr
- Mail notifications to user-provided recipient
Cross-platform support:
- Operating systems: Windows, macOS, Linux, BSD
- Architectures: arm64, amd64, armv7, i386

Roadmap

I consider the current state of GoSƐ to be production ready. Its basic functionality is complete. However, there are still some ideas which I would like to work on in the future:

BEP-17 and BEP-19 Torrent Web-seeding
Uploading to Interplanetary File System
Support End-to-End or Server-Side-Encryption
Better access control
- Limit by number of up/downloads
- Limit by token
- Limit by IP ranges / Geographical origin
Add End-to-End testing with Selenium Web-driver

Also checkout the Codeberg Issue Tracker /stv0g/gose (Issues) for a detailed overview.

Running a Xilinx hw_server as Docker Container

Wed, 23 Feb 2022 00:00:00 GMT

Dockerized Xilinx hw_server Setup.

This article describes the necessary steps to run a Xilinx hw_server as a Docker container.

Xilinx’s hw_server is a command line utility which handles JTAG communication between a Xilinx FPGA board and usually the Vivado IDE. It can be used to configure the FPGA bitstream, connect to the embedded logic analyzer cores (ILA) or perform debugging of processor cores via GDB and the Xilinx System Debugger (XSDB). The hw_server is usually used when those tasks shall performed remotely as the connection between Vivado or XSDB is established via TCP connection and allows us to run it on a remote system.

Running the hw_server as a Docker container has the benefit that its installation is simplified to starting a Docker container by running:

docker run \
  --restart unless-stopped \
  --privileged \
  --volume /dev/bus/usb:/dev/bus/usb \
  --publish 3121:3121 \
  --detach \
  ghcr.io/stv0g/hw_server:v2021.2

It also allows us to run the hw_server on architectures which are not natively supported by Xilinx such as the commonly used Aarch / ARM64 and ARMv7 architectures found in Raspberry Pis.

This is enabled by Dockers support for running container images for non-native architectures. I am using the aptman/qus Docker image ( /dbhi/qus ) to setup this user-mode emulation. The qemu-user-static (qus) image is a compilation of utilities, examples and references to build and execute OCI images (aka docker images) for foreign architectures using QEMU’s user-mode emulation.

Run the following commands to run the hw_server on a embedded device:

# Install docker
sudo apt-get update && sudo apt-get upgrade
curl -sSL https://get.docker.com | sh

# Start Docker
sudo systemctl enable --now docker

# Enable qemu-user emulation support for running amd64 Docker images
# *Note:* only required if your system arch is not amd64!
docker run --rm --privileged aptman/qus -s -- -p x86_64

# Run the hw_server
docker run --restart unless-stopped --privileged --volume /dev/bus/usb:/dev/bus/usb --publish 3121:3121 --detach ghcr.io/stv0g/hw_server:v2021.2

This setup has been tested with a Raspberry Pi 4 running the new 64-bit Debian Bullseye Raspberry Pi OS.

The pre-built Docker image for the hw_server of Vivado 2021.2 is available via:

/stv0g/xilinx-hw-server-docker (Packages)

Detailed instructions can be found at Codeberg: /stv0g/xilinx-hw-server-docker .

Encrypted credentials for Amazon AWS command line client

Wed, 09 Sep 2015 00:00:00 GMT

In this quick post I will show you how to use the password manager “password-store”¹ to securely store your credentials used by the Amazon Webservices command line client.

AWS CLI Logo.

The installation for Mac and Linux system is fairly easy:

pip install awscli

The credentials are stored as key-value pairs inside a PGP-encrypted file. Every time you call the AWS CLI tool, your keys will be decrypted and directly passed to the aws tool.

Use pass to add your keys in the store:

pass edit providers/aws

An editor opens. Use the following format:

User: stv0g
Access-Key: AKB3ASJGBS3GOMXK6KPSQ
Secret-Key: vAAABn/PMAksd235gAs/FSshhr42dg2D4EY3

Add the following snippet to your ~/.bashrc:

function aws {
  local PASS=$(pass providers/aws)
  local AWS=$(which aws)

  # Start original aws executable with short-lived keys
  AWS_ACCESS_KEY_ID=$(sed -En 's/^Access-Key: (.*)/\1/p' <<< "$PASS") \
  AWS_SECRET_ACCESS_KEY=$(sed -En 's/^Secret-Key: (.*)/\1/p' <<< "$PASS") $AWS $@
}

Then use the cli tool aws as usual:

aws iam list-access-keys { "AccessKeyMetadata": [ { "UserName": "stv0g", ...`

I covered password-store already a few times earlier: Use YubiKey and Password-store for Ansible credentials, Workshop: Security Token. ↩

Use Yubikey and Password-store for Ansible credentials

Wed, 24 Jun 2015 00:00:00 GMT

I spent some time over the last months to improve the security of servers and passwords. In doing so, I started to orchestrate my servers using a configuration management tool called Ansible. This allows me to spin-up fresh servers in a few seconds and to get rid of year-old, polluted and insecure system images.

Ansible loves Yubico.

My ‘single password for everything’ has been replaced by a new password policy which enforces individual passwords for every single service. This was easier than I previously expected:

To unlock the ‘paranoid’ level, I additionally purchased a Yubikey Neo token to handle the decryption of my login credentials in tamper-proof hardware. ‘pass’ is just a small shell script to glue several existing Unix tools together: Bash, pwgen, Git, xclip & GnuPG (obeying the Unix philosophy). The passwords are stored in simple text files which are encrypted by PGP and stored in a directory structure which is managed in a Git repository.

Yubikey Neo und Neo-n.

There are already a tons of tutorials which present the tools I describes above. I do not want to repeat all of it. So, this post is dedicated to solve some smaller issues I encountered.

Use One-Time passwords across multiple servers

The Yubikey Neo can do much more than decrypting static passwords via GnuPG:

Generate passwords:
- fixed string (insecure!)
- with Yubico OTP algorithm
- with OATH-HOTP algorithm
Do challenge response authentication
- via FIDO’s U2F standard
- with HMAC-SHA1 algorithm
- with Yubico OTP algorithm

Some third-party services already support FIDO U2F standard or traditional OATH-{H,T}OTP TFA, like used by the Google authenticator app. I suggest to have a look at: https://twofactorauth.org.

For private servers there are several PAM modules available to integrate OTP’s or Challenge Response (CR) methods. Unfortunately, support for CR is not widespread across different SSH- and mail clients.

So, you want to use OTP’s which leds to another problem: OTP’s rely on a synchronized counter between the hardware token and the server. Once you use multiple servers, those must be synchronized as well. I’m using a central Radius server to facilitate this.

Integrate `pass` into your Ansible workflow

Ansible uses SSH and Python scripts to manage several remote machines in parallel. You must use key-based SSH authentication, because you do not want to type every password manually! Additionally you need to get super user privileges for most of your administrative tasks on the remote machine.

The SSH authentication is handled by gpg-agents --enable-ssh-support option and a PGP key on your token.

To get super user privileges, I use the following variable declaration my Ansible group_vars/all file:

---
ansible_sudo_pass: "{{ lookup('pipe', 'pass servers/' + inventory_hostname) }}"

There is a separate root password for every server (e.g. pass servers/lian.0l.de). I wrote some Ansible roles to easily and periodically rotate these passwords.

Integrate ‘pass’ into OS X

There are already several plugins and extensions to integrate the pass password store into other Programs like Firefox and Android.

A prompt for the password you want.

I added support for OS X by writing a small AppleScript which can be found here: /zx2c4/password-store/contrib/pass.applescript .

A notification with countdown.

Workshop: Security Token

Mon, 15 Jun 2015 00:00:00 GMT

Der Open Source Arbeitskreis (OSAK) der Fachschaft FSMPI, veranstaltet nun zum zweiten Mal eine Crypto Party auf der sich Interessierte über Verschlüsselung und verwandte Themen informieren können.

Ich möchte hier die Gelegenheit nutzen um etwas Werbung für diese Veranstaltung zu machen. Genaue Infos findet Ihr unten im Flyer.

Dieses Mal wird es auch einen kleinen Workshop von mir geben:

Hardware Crypto Tokens

„I know none of my passwords“

Ich werde in circa 20 Min eine kurze Übersicht über Security Tokens wie beispielsweise den Yubikey oder die OpenPGP Smartcard geben. Dabei wird der Fokus auf verschiedenen Anwendungsszenarien wie One-Time-Passwords, Logins, sowie E-Mail Verschlüsselung liegen.

Update: Hier sind die Vortragsfolien und das Handout:

Download:Präsentation Download:Handout

Cryptoparty im Sommersemester 2015.

Casting between Qt and OpenCV primitives

Sat, 13 Jun 2015 00:00:00 GMT

OpenCV & QT.

As a follow-up to the previous post, I’d like to present some code which I think might be helpful for other Qt / OpenCV projects as well.

This code was written for Pastie. Pastie is a piece of software I wrote as part my image processing seminar. It makes use of the well known libraries:

Qt for the graphical user interface
OpenCV for image processing and computer vision

I wrote a C++ header file to facilitate the co-operation of those two libraries. This file enables the conversion / casting of OpenCV and Qt types e.g.:

#include <QImage>
#include <cv/core.hpp>

QImage qimg("filename.png");

cv::Mat cvimg = toCv(qimg);

The source code is available at Codeberg: /stv0g/snippets/c/qcv_cast.h .

The following conversions are supported:

QImage	⇆	cv::Mat
QTransform	⇆	cv::Mat
QPoint	⇆	cv::Point2i
QPointF	⇆	cv::Point2f
QRect	⇆	cv::Rect2i
QRectF	⇆	cv::Rect2f
QSize	⇆	cv::Size

You can find some examples in the real code here: /stv0g/pastie/filters/pattern.cpp and here /stv0g/pastie/cast.h .

Seminar: Camera-based PCB Analysis for Solder Paste Dispensing

Tue, 26 May 2015 00:00:00 GMT

Mini Kossel 3D Printer.

The lectures during my last semester were largely focused on digital image processing. Combining this with the inspiration for 3D printing, I gathered through my trip though South Korea, resulted in the following seminar paper. Seminars are a compulsory part of our curriculum which I like due the self-contained work and the ability to pick an individual topic.

Over the past year, I’ve built my own Kossel 3D printer. The Mini Kossel is based on a novel parallel delta kinematic which was developed by Johann C. Rocholl, a Google engineer from Germany.

This paper is targeting the automation of solder paste dispensing onto printed circuit boards by using computer vision and RepRap robots.

Download:Presentation Slides Download:Full Paper Source Code

Seminar: Image Processing and Content Analysis

Camera-based PCB Analysis for Solder Paste Dispensing

Author: Steffen Vogel (steffen.vogel@rwth-aachen.de)
Academic Advisor: Wei Li (wei.li@lfb.rwth-aachen.de)
Institute of Imaging & Computer Vision (LfB)
Rheinisch-Westfälische Technische Hochschule (RWTH), 52056 Aachen

1 Abstract

Two of the main challenges for PCB prototyping are the time-consuming setup of involved machines and their economic feasibility for small laboratories and hobbyists. This paper tries to offer solutions for both of these issues:

The complex setup process of industrial machines can be accelerated by computer vision. It is preferable to automate this process as far as possible to enable the operation by untrained personnel and hobbyists. The workflow can be further simplified by not relying on external CAD data. This includes: detection of components, pads and footprints; mapping between available components and footprints and planning of shortest tool paths.
The adaption of proven 3D printers allows to lower the costs for such machines. The lightweight and fast kinematics of parallel 3D-delta robots like the RepRap Mini Kossel are perfectly suited for the assembly of PCBs. Only the print head has to be exchanged between the individual steps of the process.

This work presents a workflow to control DIY 3D printers for the purpose of PCB assembly. By using cheap and easy-obtainable parts like proven RepRap 3D printers, this technique is viable for small laboratories, FabLabs and hobbyists. During the seminar, a analysis and control software for RepRap printers was written. Hence, we focus on the overall workflow and tools and less on algorithms and theory. Here, the task of solder paste dispensing was chosen to be explored in detail. This work establishes the groundwork for more complex task like the pick and placing of electronic components.

2 Motivation

The ongoing miniaturization of electronic products like smartphones and Ultra Books has led to a new form factor for electronic components. Surface-mounted devices (SMD) are already widespread in electronic design and production. As a result, previously used through-hole components are gradually phased out. This miniaturization of SMD components is an ongoing trend and raises the barrier for hobbyists to produce PCBs themselves. Soldering and placement of 0401-sized resistors or BGA packages is not possible by hand any longer.

This work is motivated by the vision to build an all-in-one machine for the complete process of prototype PCB assembly (PCBA). To accelerate the development process and to reduce the costs, all of these tasks can be handled by a single workbench 3D printer / CNC mill. The PCB production process roughly can consists of the following steps:

Isolation milling or pen plotting of PCB traces
Drilling of holes and contours
Solder paste dispensing for SMD pads with a syringe
Pick-and-place of SMT components with vacuum
Soldering with hot air, a hot plate or by a laser

For the scope of this paper, the process of solder paste dispensing was chosen. This task offers the biggest margin to profit from computer vision. Industrial mass production uses stencils to apply solder paste onto the PCB. For small prototype assemblies the fabrication of stencils is not worthwhile. Therefore, solder paste is applied manually with a pressurized syringe, which is hold by hand. The dispensing of solder paste requires the knowledge exact solder pad positions and dimensions. Traditionally, this information is exported by CAD design tools and is required to produce the stencils. But sometimes the CAD data is not available or stored in an inaccessible proprietary format. This paper presents techniques to gather the pad locations and dimensions by means of computer vision.

Figure 1: Solder paste dispensing techniques.

Figure 2: 0805-sized resistor.

3 Workflow

The task of solder paste dispensing can be further separated in several sub stages, which are explained in the following sections.

Workflow steps.

3.1 Imaging

In the original proposal, a webcam was planed be used to capture images for further analysis. But using other image sources like scanners or digital cameras has proven to provide superior image quality (see Table 1). The resolution of a webcam is insufficient to capture the complete PCB in a single image. Even with a high resolution webcam¹ satisfying results were only achievable by stitching several detailed shots together. This issue can easily be resolved by using a professional digital camera (see Figure 3d). However, this prevents the direct attachment of the camera to the print head of the printer, because the weight of a camera would increase the inertia and thereby limit the maximum speed of the printer. The workflow does not require the image acquisition to be done on top of the working area of the printer. By using cameras, all images are subject to distortion caused by lenses and perspective projection. This issue can be solved by calibrating the camera and by applying the inverse perspective transformation. These disadvantages can be avoided by using an imaging method which does not impose these distortions like a flatbed scanner. First tests showed biased results (see Figure 3c and 3a). The previous methods do not rely on the existence of CAD layout data. In the end, best results can be achieved by directly using available CAD data (see Figure 3b). The industry uses a widespread format called Gerber for exchanging PCB layouts used for production. Almost every CAD tool can export PCB layouts as a Gerber file. This Gerber format stores every part of the design (silkscreen, pads, board contour, milling, coper) in a separate layer. Basically it is a vector graphic format, whose layers are built out of basic geometric shapes. This allows them to be rendered to raster images of arbitrary resolutions².

Figure 3: Imaging methods for PCBs.

Table 1: Image sources for PCB analysis.

3.2 Undistortion

Camera-based imaging methods are subject to distortions caused by optical lenses. These non-linear distortions must be compensated by camera calibration. During the camera calibration, a set of distortion parameters and a camera matrix is determined. Usually, a set of different views of a known pattern is used to measure the distortion coefficients. In this seminar, both chessboard and circle patterns were used. But it has become apparent, that non-linear distortions are mostly negligible when using recent DSLRs because most of them already compensate them by their internal image processing pipeline. The workflow involves several different coordinate systems (C, U, R, S, P). This stage transforms the source image from the camera coordinate system C to the system of undistorted images U.

Figure 6: Coordinates systems and transformations.

3.3 Perspective Correction

Camera-based imaging methods are also subject to perspective projection. The following analysis demands a top view onto the PCB. Thus, the perspective projection is turned into a two-dimensional euclidean plane. To simplify the correction, we assume that the PCB has a rectangular shape with known dimensions (w × h). This can be realized by enclosing the PCB with a set of four markers which are arranged in a rectangle as shown in Figure 9a. The camera projection warps this rectangle into a quadrilateral shape. Mathematically, this is described by a perspective transformation between two coordinate systems. The undistorted image with coordinate system U has to be mapped to the reference system R. The required transformation matrix is obtained by solving equation 1 with four points x of the image plane and their matching counterparts x′ of the object plane. The points of the object plane are given by the known dimensions and spacings of the markers or PCB³. In other words, respective markers are used to describe basis vectors for the reference coordinate system. Figure 9a shows the original image with the detected markers. Figure 9b shows the same image after the correction. Note that the markers are still partially visible in the corners of the warped image.

3.4 Preprocessing

The previous steps are mandatory for camera based imaging methods. Scans or CAD imported Gerber files already provide a top view onto the PCB, and therefore, allow skipping the previous stages. Before starting with the segmentation of PCB solder joints, several preprocessing steps have to be executed. They improve the results of the following steps or allow saving computational time.

Blurring: The segmentation relies on clustering and thresholding to detect solder joints. A prior smoothing of the image helps to cope with noise in the source image.
Scaling: To reduce the required computational time, the source image is restricted to a sufficient resolution. In rare cases the resolution of the source image is too low for further analysis. In this case, the image is scaled up. This way the following algorithms can work with sub-pixel accuracy.
Alignment: Subsequent steps are using morphological image operations to improve the results. These operations are using square-shaped kernels. By aligning the orientation of the PCB and its pads to the shape of the kernel, details and sharp edges of PCB pads are preserved. Furthermore, certain algorithms like the Harris edge detector can profit from known orientations of the vertices. The alignment can be identified by evaluation of Hough lines or the polar magnitude spectrum of and FFT (see [BreierLfB]).

3.5 Segmentation

To separate the solder pads from the remaining PCB and the background, a color-based segmentation method is used. The K-Means clustering algorithm reduces the color space to k clusters. Usually, these classes correspondent to four portions of the PCB. These portions are ordered according to their occurrences in the image.

Image background
PCB base material (solder mask)
Solder pads
Silkscreen⁴

Hence, it makes sense to use k = 4 for the algorithm. Figure 10a shows the result of the K-Means algorithm. The correct selection of the solder pad class is crucial for valid results. Therefore, it is advisable to make use of more criteria for its selection. The current implementation only consults the frequency of each class. Additionally, the amount of connected objects per class might be used. The background and base material classes often only consist of a few connected objects. The solder pad class in contrast often has hundreds of individual objects. Once a class has been selected (potentially by hand), a binary image is created. The resulting binary image is shown in Figure 10b.

3.6 Pad Detection & Filtering

For solder paste dispensing, solder pad coordinates must be exactly located onto the PCB. In this paper, the detection is limited to pads with a rectangular shape as shown in Figure 2b. This limitation is negligible for most SMD packages, as they generally are using rectangular pads like shown in Figure 4. The previous segmentation step produces in a binary image. This image is used to find contours, which are then used to analyze the shape of the individual objects. These contours are filtered and classified by the following criteria:

Figure 4: Footprints of various SMD packages.

Figure 5: Pad contour with minimum bounding rectangle.

In a first attempt, polygons were used to approximate the contour. The Ramer-Douglas-Peucker algorithm finds a polygon with as little as possible vertices. At the same time, the maximum deviation to the original contour is limited by a threshold. To qualify the contour as a solder pad, the approximated polygon must be a quadrangle, convex and the angles must be close to 90 degrees. This approach only delivered unsatisfying results because of the low resolution and random starting points of the contours, which do not coincide with the corners of the rectangle. Better results can be achieved by comparing the enclosed area of a contour with the area of its minimal bounding rectangle (see Figure 5). A ratio above 0.8 usually yields a usable hit rate. Figure 10c shows the found rectangles (magenta frames). Subsequently, the rectangles are filtered by their absolute area and aspect ratio. Overlapping rectangles are not allowed and are filtered out as well. Figure 10d shows the pads after these plausibility checks.

3.7 Tool Path Planning

To speed-up the execution, the pads have to be arrange in an order which results in a path with the minimum length. To find an optimal tool path, a Hamiltonian path with minimum length must be determined. A Hamiltonian path connects all vertices of a graph. We can think of the set of detected pads as a fully connected graph. Each edge is weighted by the euclidean distance between the connected edges. This problem description is a little different to other well known problems:

Traveling Salesman Problem: The TSP looks for the shortest Hamiltonian cycle (equal start end end nodes). For apply the solder paste, this constraint is dispensable. Apart from that, the TSP describes the problem quiet well. Most TSP solvers are designed for huge problem sizes (> 1000). This does not fit to our problem where usually 100-400 pads must be visited.
Shortest Path Problem: There is a large amount of algorithms which find the shortest path between two vertices of a graph. This problem usually is faced for navigation. The Dijkstra algorithm is a well known representative in this category. The shortest path problem only describes the shortest path between two vertices of a graph and is therefore not suitable for the described case.

For this task, a simple nearest neighbor heuristic (NNR) was implemented. This heuristic starts with a random vertex and successively travels along the edge with the lowest weight, which corresponds to the nearest neighbor. However, this procedure will not result in an optimal path. The results can be easily improved by repeating the heuristic with different start vertices. Finally, the one which results in the minimal length is selected. This is known as the repetitive nearest neighbor (RNNR). Figure 10e shows the final result with the found path as an overlay.

3.8 Robot Control

Equation 1 shows the transformation between the undistorted camera images and reference coordinates. Similarly, another transformation between the reference system and the robot coordinate system is necessary.

Both the reference system R and the printer system P are two dimensional euclidean planes. We can map points (pads) from the reference system by scaling, rotating and translating them. Mathematically, this can be modeled by an affine transformation⁵:

The coordinates of the reference markers xi are given by the corners of the processed image. The robot calibration points x′i have to be determined manually by aligning the printhead with markers of on the PCB, which is positioned on its working area. Once the print head is positioned over the marker (or corner of the PCB), the related printer coordinates are requested from the printer controller.

4 System Description

This section shortly describes the overall system design and used hardware. The system consists of a modified 3D printer which is capable to extrude solder paste⁶ and a common digital camera. Additionally, a laptop, PC or embedded system is used to carry out the computation.

4.1 Camera: Olympus OM-D E-M10

As shown in Table 1, digital cameras offer the best image quality and flexibility. Good lighting is crucial for satisfactory results. Shadows, uneven exposure and reflexions are the main issues during the analysis. The results in this paper we produced with a dedicated flash to provide indirect and diffuse illumination. For enhanced usability, the camera can be connected via WiFi to the PC. This allows remote control and image retrieval from the camera. A dedicated library called libqt-omd⁷ is currently developed by the author. Integration in the analysis software is planned.

4.2 Software: Pastie

The previously described workflow has been implemented in a graphical application named ”Pastie”⁸. For adequate performance, the tool was developed by using C++11 and specialized libraries. The graphical user interface (GUI) is based on Qt 5.4. Computer vision and image processing is handled by OpenCV 3.0.

Pastie Screenshot 1.

Pastie Screenshot 2.

Pastie Screenshot 3.

Thanks to Qt and OpenCV as the only dependencies, the tool is portable and can run on many architectures. The implementation was tested on Windows, Linux and Mac OS X. Theoretically it should be capable to run on Android and iOS as well. The software offers a front-end to most common OpenCV algorithms. It allows creating a filter pipeline and interactively adjusting parameters of the involved algorithms. Processing an image with the presented pipeline takes about 5-10 seconds depending on the resolution.

4.3 Printer: RepRap Mini Kossel

The RepRap project is a community effort to develop cheap and self-replicating 3D printers using commonly available parts and tools. In this work a relatively new style of printer called Mini Kossel was used. This printer consist of three towers which are arranged in an equilateral triangle. This delta kinematic allows movements of the tool center point (TCP) in all three directions with an accuracy of 100 steps/mm. This is perfectly sufficient for placing even tiny 0603 components. The RepRap project develops printer hardware as well as electronics and controller firmware. For the purpose of this work only the toolhead must exchanged.

RepRap Kossel Mini 3D printer.

Similar to the camera, recent printer electronics allow a wireless connection over WiFi to schedule prints, adjusting settings or for visual inspection of the process with a webcam⁹.

The printer is controlled with G-Code, a human readable language to control CNC mills, lathes, 3D printers and laser cutters. Listing 8 shows a typical excerpt of a G-Code program. Every line correspondents to one instruction and consists of at least one tuple. The first tuple of every line describes the operation (move, stop, set settings). Following tuples are optional parameters whose meaning is depending on the operation. Every class of machine has their of dialect of G-Code. The RepRep community agreed on a dialect for 3D printers¹⁰.

M204  S1000 Z100 E500 ; set acceleration
G28                   ; home all axes
G29                   ; calibrate bed
GO X10 Y-5 F100       ¡ move to x=10, y=-5
M112                  ; emergency stop

Figure 8: Excerpt of a G-Code program.

5 Results

The following images are showing the intermediate steps of the processing pipeline which is described in Section 3. The results have been produces with the Pastie tool. It comes with a variety of more PCB images, scans and Gerber files which have been collected during this seminar.

Figure 9a: Source image with detected pattern.

Figure 9b: Perspective correction.

Figure 10a: K-Means color clustering (ROI, k=4).

Figure 10b: Color/class filter (ROI).

Figure 10c: Pad/rectangle detection (ROI).

Figure 10d: Filtered pads (ROI).

Figure 10e: Planned tool path (ROI).

6 Conclusion & Outlook

This work was focused on the development of a software tool for solder pad detection and printer control. Apart from that, the developed software can act as an universal front-end for OpenCV. This allows fast evaluation of new algorithms by interactively changing parameters by hand. In the future, this tool shall be extended to handle more complex tasks of the PCB assembly process. The software can work with a variety of different data sources as photographies, scans and Gerber files. Even screenshots of the CAD tool or datasheets are usable. This is an advantage to existing methods, where layout data is required. The quality of results is also depending on the surface finish of the PCB. Hot Air Solder Leveling (HASL) coats the solder pads with a thin film of tin. Unfortunately, this results in an uneven surface finish which highly reflective. To cope with bright spots and reflexions a diffuse illumination is essential. Electroless Nickel Immersion Gold (ENIG) or Immersion Tin (ISn) coating in contrast, will result in a even and soft surface with a yellow / golden color. Another influencing factor is the color of the silkscreen and solder resist. The solder resist is a thin mask which is applied to the bare PCB to protect the traces from oxid. Common colors for solder resists are green, black, white and blue. Best results were achieved with black to gain a high contrast. There is still a lot space for further usability improvements. At the current state, pictures and G-Code instructions are transferred with memory cards between the camera, printer and PC. This can be simplified by using the existing WiFi capabilities of those devices. The import of Gerber files is currently only possible by using a conversion tool called gerbv. However, it provides a software library called libgerbv to facilitate the import of Gerber files as part of the implementation. Within the scope of this seminar, a detailed evaluation of the results and testing was not possible due to the limited amount of time. It is eligible to compare the results the of camera based analysis with a Gerber based one of the same PCB. In doing so the deviation between the original CAD data and their reconstruction can be evaluated. For real application, a dispenser tool head for the printer is missing and has to be designed. It’s desirable to use the existing 3D printing technology to do this. The results in this paper are showing, that existing standard algorithms are sufficient for the detection of solder pads. However there are situations in which they fail. In these cases a segmentation based on the Watershed algorithm or using the Harris corner detection may be used to improve the results.

7 Literature and Materials

[BreierLfB] Rotation Estimation for Printed Circuit Board Recycling by Matthias Breier et. al. (LfB, 2014)
[MVGeom] Multiple View Geometry in Computer Vision by Richard Hartley, Andrew Zisserman (Cambridge University Press, March 2004)
[LearnOCv] Learning OpenCV by Gary Bradski und Adrian Kaehler (O’Reilly & Associates, October 2013)
[OCvDoc] OpenCV Documentation and referenced papers: https://www.opencv.org
[QtDoc] Qt Project Documentation: https://www.qt-project.org
[RRap] RepRap Project Wiki: https://reprap.org/wiki

7.1 Links and Software

The Pastie implementation: /stv0g/pastie .
GerbV Gerber File Viewer and Library: https://gerbv.github.io/.
libqt-omd Communication library for Olympus OM-D & Pen cameras: /stv0g/libqt-omd .

Logitech HD Pro C920: 2304x1536 pixel with autofocus. ↩
libgerbv is a free software library for rendering Gerber files. ↩
See OpenCV function getPerspectiveTransform(). ↩
The silkscreen contains printed markers and text on top of the PCB. ↩
See OpenCV function getAffineTransform(). ↩
The solder paste extruder itself is not part of this work. ↩
libqt-omd is available as free software at: /stv0g/libqt-omd . ↩
Pastie is available as free software at /stv0g/pastie . ↩
This functionality is provided by a software called Octoprint: https://www.octoprint.org. ↩
The dialect is documented on the RepRap website: https://reprap.org/wiki/G-code. ↩

Bachelor Thesis: Extended Abstract

Thu, 22 Jan 2015 00:00:00 GMT

Almost fourteen months ago, I started working on my bachelor thesis. Although I finished it half a year ago, it’s still part of my work as a student research assistant.

During my initial work, most of the code was written for an internal research kernel. I’m now happy that we were able to port it to an open source kernel called eduOS: /RWTH-OS/eduOS ). This minimal operating system is used for practical demo’s and assignments during the OS course at my university. There’s much more I could write about. So this will probably be another separate blog post.

The motive for this article is an abstract I wrote for the student research competition of the ASPLOS conference which is held this year in Istanbul, Turkey. Unfortunately my submission got rejected. But as a nice side-effect, I’ve now the chance to present my work to an English audience as well:

Download:Extended Abstract (PDF)

Self-referencing Page Tables for the x86-Architecture

A simple Paging Implementation for a minimalistic Operating System

Steffen Vogel

Academic advisor: Dr. rer nat. Stefan Lankes Institute for Automation of Complex Power Systems E.ON Energy Research Center, RWTH Aachen University Mathieustr. 10, 52074 Aachen, Germany

This was a submission for ASPLOS Student Research Competition ’15 Istanbul, Turkey¹.

Abstract

The adoption of 64 bit architectures went along with an extension of the virtual address space (VAS). To cope with this growth, the memory management unit (MMU) had to be extended as well. For paging-based systems like Intel’s x86-architecture this was realized by adding more levels of indirection to the page table walk.

This walk translates virtual pages to physical page frames (PF) by performing look-ups in a radix / prefix tree in which every node represents a page table (Figure 1a). Since the tables are part of the translation process, they must be referenced by physical page frame numbers (PFN, blue line). As the operating system is only eligible to access the VAS, it cannot follow the path of a walk. In order to allow the manipulation of page tables, it must provide:

Figure 1a: Page table walk in the x86 64 longmode: Traditional, without self-reference.

Access to the table entries, by mapping the tables themselves to the VAS.
A mapping between physical references to corresponding locations in the VAS.

Additionally, every level of the page table walk increases the complexity of managing these mappings. They also increase the memory consumption by occupying physical page frames. It is possible to avoid both drawbacks by the technique described in the following.

In my bachelor thesis, I presented an approach, which is compatible with both the 32 bit and 64 bit version of Intel’s x86-architecture. This allows for a replacement of two code bases, one for each architecture, by one supporting both. Thus, results in a shorter, easier comprehensible, and maintainable code. As foundation for this implementation our teaching OS called “eduOS” was used². “eduOS” supports only the 32 bit protected mode whereas the 64 bit longmode is only implemented for an internal research kernel.

Thanks to the sophisticated design of Intel’s x86 MMU, it is possible to avoid most of the complexity and space requirements by using a little trick. Adding a self-reference in the root table (PML4 resp. PGD) automatically enables access to all page tables from the VAS without the need for manual mappings as described above (Figure 1b). The operating system does not need to manually follow the path of a page table walk, as this task is executed by the MMU for accessing individual tables instead of page frames.

Figure 1b: Page table walk in the x86 64 longmode: With self-reference.

An access to the VAS region covered by a self-reference causes the MMU to look up the root table twice (red line). Effectively, this shifts the whole page table walk by one level. Therefore, it stops with the PFN of page tables instead of page frames that are usually translated by the MMU. Here, both the PML4 and PDPT indexes are used to choose an entry out of the PML4 table. Therefore, it must be guaranteed that PML4 entries can be interpreted as PDPT entries, too. This demands for the following requirements:

Homogenous coding of paging flags across all paging levels.
Equal table sizes across all paging levels.

Fortunately, the x86-architecture complies with this prerequisites as shown in Figure 2. Green colored flags are coded consistently across all paging levels. Only PAT, size and global flags have a slightly different meaning for entries in the PGT. My bachelor thesis shows that these deviations still allow maintaining full control caching and memory protection properties of self-mapped tables. This includes for common system calls like fork() and kill().

Figure 2: Similar flags across all paging levels.

By repeatedly addressing the self-reference, it is also possible to access tables of the upper levels (PGD to PML4). Table 1 shows the resulting virtual addresses of all page tables when using the last (512th) entry of the PML4 table for the self-reference³. This grants access to all possible page tables, including those which might not yet exist and may be allocated in the future. Hence, the self-reference reserves a fixed fraction of the VAS for the page tables. The size of this region is equal to 256 TiB / 512 = 512 GiB for 64 bit (resp. 4 GiB / 1024 = 4 MiB for 32 bit), which is negligible in comparison to the huge VAS of 248 byte.

Table 1: Virtual addresses of self-mapped tables.

For the manipulation of page table entries two approaches are feasible:

Top-down Use known tree traversals, starting at the root node, which corresponds to the PML4 respectively PGD.
Bottom-up Use the page fault handler to create new tables on-the-fly, when they are not yet present.

But there are also other architectures which satisfy the prerequisites described above. One of these is the Alpha⁴ architecture, which suggests a similar approach in the reference manual. Intel and AMD do not mention the technique in their x86 manuals. In the field of operating systems, support is far more limited. There is only a single reference⁵ dated to 2010 indicating that Microsoft might use a similar approach for its NT kernel. Linux cannot profit because its paging implementation must support a broad selection of virtual memory architectures of which not all fulfill the requirements mentioned above.

A full version of the thesis and slides are available in this post. ↩
Description and source code at: www.os.rwth-aachen.de. ↩
This is an arbitrary choice. All other entries are feasible, too. ↩
Compaq Computer Corporation: Alpha Architecture Reference Manual. January 2002. ↩
Dave Probert, Microsoft: Windows Kernel Architecture Internals. April 2010. ↩

noteblok.{de,net,org,dn42}

Thu, 19 Jun 2014 00:00:00 GMT

Dies ist das neue Logo und Name meines Blogs.

Bisher gab es hier nur wenige persönliche Beiträge. Da ich das auch so beibehalten möchte, habe ich mich entschlossen meinen Namen aus dem Titel zu streichen. Vielleicht findet so auch mal der ein oder andere Gastbeitrag seinen Weg hierher.

Domains

Mit dem neuen Namen hat sich auch die Domain geändert. Der Blog ist nun erreichbar unter noteblok.{de,net,org,dn42}. Über meine persönliche Domain gelangt man nun direkt zu ein paar Infos über mich.

Neben den neuen Domains sind nun auch alle Webseiten/Blogs über IPv6 erreichbar 😊.

dn42

Zudem ist der Blog auf über des dn42 Darknet erreichbar. Das dn42 ist ein dezentrales und dynamisches VPN Netzwerk. Es besteht aus einem Verbund von Freiwilligen Admins, die jeweils Peer-to-Peer Verbindungen über VPNs herstellen. Es baut damit als Overlay Netzwerk auf dem bestehenden Internet auf. Zudem nutzt das dn42 mit BGP, DNS, Whois die gleichen Protokolle wie das reguläre Internet.

Als Teil von (/dev/nulll) betreibe ich das Autonome System AS4242422428 und unterhalte Peerings mit anderen Knoten des dn42 Netzwerkes.

Screenshot

Hier ist noch ein Screenshot meines Blog aus dem Jahr 2014:

Screenshot noteblok.net vom 2014-12-22.