Development

6 posts with the tag “Development”

GoSƐ - A terascale file-uploader

Apr 3, 2022

Steffen Vogel

code poet, bit juggler & logic wizard

GoSƐ Logo.

GoSƐ is a modern and scalable file-uploader focusing on scalability and simplicity.

It is a little hobby project I’ve been working on over the last weekends.

The only requirement for GoSƐ is a S3 storage backend which allows to it to scale horizontally without the need for additional databases or caches. Uploaded files a divided into equally sized chunks which are hashed with a MD5 digest in the browser for upload. This allows GoSƐ to skip chunks which already exist. Seamless resumption of interrupted uploads and storage savings are the consequence.

And either way both upload and downloads are always directed directly at the S3 server so GoSƐ only sees a few small HTTP requests instead of the bulk of the data. Behind the scenes, GoSƐ uses many of the more advanced S3 features like Multi-part Uploads and Pre-signed Requests to make this happen.

Users have a few options to select between multiple pre-configured S3 buckets or enable browser & mail notifications about completed uploads. A customizable retention / expiration time for each upload is also selectable by the user and implemented by S3 life-cycle policies. Optionally, users can also opt-in to use an external service to shorten the URL of the uploaded file.

Currently a single concurrent upload of a single file is supported. Users can observe the progress via a table of details statistics, a progress-bar and a chart showing the current transfer speed.

GoSƐ aims at keeping its deployment simple and by bundling both front- & backend components in a single binary or Docker image. GoSƐ has been tested with AWS S3, Ceph’s RadosGW and Minio. Pre-built binaries and Docker images of GoSƐ are available for all major operating systems and architectures at the release page: /stv0g/gose (Releases) .

GoSƐ is open-source software licensed under the Apache 2.0 license.

Live Demo

Screencast

Features

De-duplication of uploaded files based on their content-hash
- Uploads of existing files will complete in no-time without re-upload
S3 Multi-part uploads
- Resumption of interrupted uploads
Drag & Drop of files
Browser notifications about failed & completed uploads
User-provided object expiration / retention time
Copy URL of uploaded file to clip-board
Detailed transfer statistics and progress-bar / chart
Installation via single binary or container
- JS/HTML/CSS Frontend is bundled into binary
Scalable to multiple replicas
- All state is kept in the S3 storage backend
- No other database or cache is required
Direct up & download to Amazon S3 via presigned-URLs
- Gose deployment does not see an significant traffic
UTF-8 filenames
Multiple user-selectable buckets / servers
Optional link shortening via an external service
Optional notification about new uploads via shoutrrr
- Mail notifications to user-provided recipient
Cross-platform support:
- Operating systems: Windows, macOS, Linux, BSD
- Architectures: arm64, amd64, armv7, i386

Roadmap

I consider the current state of GoSƐ to be production ready. Its basic functionality is complete. However, there are still some ideas which I would like to work on in the future:

BEP-17 and BEP-19 Torrent Web-seeding
Uploading to Interplanetary File System
Support End-to-End or Server-Side-Encryption
Better access control
- Limit by number of up/downloads
- Limit by token
- Limit by IP ranges / Geographical origin
Add End-to-End testing with Selenium Web-driver

Also checkout the Codeberg Issue Tracker /stv0g/gose (Issues) for a detailed overview.

Running a Xilinx hw_server as Docker Container

Feb 23, 2022

Steffen Vogel

code poet, bit juggler & logic wizard

Featured

This article describes the necessary steps to run a Xilinx hw_server as a Docker container.

Xilinx’s hw_server is a command line utility which handles JTAG communication between a Xilinx FPGA board and usually the Vivado IDE. It can be used to configure the FPGA bitstream, connect to the embedded logic analyzer cores (ILA) or perform debugging of processor cores via GDB and the Xilinx System Debugger (XSDB). The hw_server is usually used when those tasks shall performed remotely as the connection between Vivado or XSDB is established via TCP connection and allows us to run it on a remote system.

Running the hw_server as a Docker container has the benefit that its installation is simplified to starting a Docker container by running:

docker run \
  --restart unless-stopped \
  --privileged \
  --volume /dev/bus/usb:/dev/bus/usb \
  --publish 3121:3121 \
  --detach \
  ghcr.io/stv0g/hw_server:v2021.2

It also allows us to run the hw_server on architectures which are not natively supported by Xilinx such as the commonly used Aarch / ARM64 and ARMv7 architectures found in Raspberry Pis.

This is enabled by Dockers support for running container images for non-native architectures. I am using the aptman/qus Docker image ( /dbhi/qus ) to setup this user-mode emulation. The qemu-user-static (qus) image is a compilation of utilities, examples and references to build and execute OCI images (aka docker images) for foreign architectures using QEMU’s user-mode emulation.

Run the following commands to run the hw_server on a embedded device:

# Install docker
sudo apt-get update && sudo apt-get upgrade
curl -sSL https://get.docker.com | sh

# Start Docker
sudo systemctl enable --now docker

# Enable qemu-user emulation support for running amd64 Docker images
# *Note:* only required if your system arch is not amd64!
docker run --rm --privileged aptman/qus -s -- -p x86_64

# Run the hw_server
docker run --restart unless-stopped --privileged --volume /dev/bus/usb:/dev/bus/usb --publish 3121:3121 --detach ghcr.io/stv0g/hw_server:v2021.2

This setup has been tested with a Raspberry Pi 4 running the new 64-bit Debian Bullseye Raspberry Pi OS.

The pre-built Docker image for the hw_server of Vivado 2021.2 is available via:

/stv0g/xilinx-hw-server-docker (Packages)

Detailed instructions can be found at Codeberg: /stv0g/xilinx-hw-server-docker .

Casting between Qt and OpenCV primitives

Jun 13, 2015

Steffen Vogel

code poet, bit juggler & logic wizard

As a follow-up to the previous post, I’d like to present some code which I think might be helpful for other Qt / OpenCV projects as well.

This code was written for Pastie. Pastie is a piece of software I wrote as part my image processing seminar. It makes use of the well known libraries:

Qt for the graphical user interface
OpenCV for image processing and computer vision

I wrote a C++ header file to facilitate the co-operation of those two libraries. This file enables the conversion / casting of OpenCV and Qt types e.g.:

#include <QImage>
#include <cv/core.hpp>

QImage qimg("filename.png");

cv::Mat cvimg = toCv(qimg);

The source code is available at Codeberg: /stv0g/snippets/c/qcv_cast.h .

The following conversions are supported:

QImage	⇆	cv::Mat
QTransform	⇆	cv::Mat
QPoint	⇆	cv::Point2i
QPointF	⇆	cv::Point2f
QRect	⇆	cv::Rect2i
QRectF	⇆	cv::Rect2f
QSize	⇆	cv::Size

You can find some examples in the real code here: /stv0g/pastie/filters/pattern.cpp and here /stv0g/pastie/cast.h .

Bachelor Thesis

Jan 20, 2014

Steffen Vogel

code poet, bit juggler & logic wizard

Seit fast zwei Monaten schreibe ich jetzt am Lehrstuhl für Betriebssysteme meine Bachelorarbeit:

Eine generische Speicherverwaltung mit Hilfe von Seitentabellen für ein minimalistisches Betriebssystem

oder auf Englisch:

A generic memory management with paging for a minimalistic operating system

calcelestial

May 5, 2013

Steffen Vogel

code poet, bit juggler & logic wizard

/stv0g/calcelestial ist ein kleines Linux-Tool zum Berechnen von Auf- und Untergangszeiten sowie der Position sämtlicher Planeten unseres Sonnensystems.

Es ist der Weiterentwicklung von /stv0g/sun , das ursprünglich als kleines Bash-Skript für meinen Router startete. Mittlerweile ist das Tool zu einem weit umfangreicherem Werkzeug gewachsen, welches nicht mehr nur die Auf- und Untergangszeit der Sonne berechnen kann:

Es sind mit dem Mond, Mars, Neptun, Jupiter, Merkur, Uranus, Saturn, Venus und Pluto eine Menge neuer Planeten dazugekommen. Auch kann nun die Position dieser Himmelskörper zu jedem beliebigen Zeitpunkt oder dem Auf- und Untergang berechnet werden.

Nun bin ich selber kein kleiner Hobby-Astronom, sodass ich diese ganzen Berechnungen aus dem Ärmel schütteln könnte. Stattdessen nutze ich die Bibliothek libnova. libnova benutzt die sehr genauen Algorithmen “Variations Séculaires des Orbites Planétaires” (kurz VSOP-87), die Pierre Pratagnon 1987 entwickelte.

$ calcelestial
Usage:
  calcelestial [options]

Options:
  -p, --object    calc for celestial object: sun, moon, mars, neptune,
       jupiter, mercury, uranus, saturn, venus or pluto
  -H, --horizon    calc rise / set time with twilight: nautic, civil or astronomical
  -t, --time    calc at given time: YYYY-MM-DD [HH:MM:SS]
  -m, --moment    calc position at moment of: rise, set, transit
  -n, --next    use rise, set, transit time of tomorrow
  -f, --format    output format: see strftime (3) for more details
  -a, --lat    geographical latitude of observer: -90° to 90°
  -o, --lon    geographical longitude of oberserver: -180° to 180°
  -q, --query    query geonames.org for geographical coordinates
  -z, --timezone  override system timezone
  -u, --universal  use universial time for parsing and formatting
  -h, --help    show this help
  -v, --version    show version

A combination of --lat & --lon or --query is required.
Please report bugs to: post@steffenvogel.de

Beispiele

Die einfachste Variante nutzt das Unix Tool at:

echo ~/bin/enable-lightning | at $(calcelestial -p sun -m set -q Frankfurt -H civil)

Mit folgenden Cronjobs, lässt sich dieses Prinzip auch leicht auf andere Anwendungen übertragen:

0 0 * * * echo 'fnctl stop && fnctl fade -c 000000' | at $(calcelestial -m rise -p sun -q Aachen)
0 0 * * * echo 'fnctl start' | at $(calcelestial -m set -p sun -q Frankfurt)

Mit dem Tool nvram-wakeup, lässt sich so beispielsweise der Rechner jeden Tag 10 Minuten for Sonnenaufgang automatisch starten:

nvram-wakeup -s $(date -d "-10 min $(calcelestial -m rise -p sun -q Berlin)" +%s)

Oder möchtest du deinen Rechner nach Sonnenuntergang automatisch herunterfahren?

shutdown $(date -d +10 min $(calcelestial -m rise -p sun --lat=50.55 --lon=-6.2) +%H:%M)

Die aktuelle Position des Mondes kann beispielsweise so bestimmt werden:

calcelestial -p moon -q Aachen -f "az: §a alt: §h"

Detailiertere Dokumentation findet ihr in der Manpage calcelestial(1).

Download

calcelestial ist wie immer auf Codeberg verfügbar /stv0g/calcelestial .

Conway's Game of Life

Dec 1, 2010

Steffen Vogel

code poet, bit juggler & logic wizard

Als Übung für meine Informatik Vorlesung an der RWTH Aachen habe ich diese C Implementation von Conways Game of Life geschrieben.

Dieses simple “zero player” Game wird komplett im Terminal ausgeführt. Mit Mausunterstützung und Farben wurden mit libncurses realisiert.

Den Quelltext findet ihr inklusive eines Makefiles auf Codeberg: /stv0g/rwth-info1/src/conway.c .

Video

Hints & Shortcuts

“p” pausiert das Spiel
“q” beendet das Spiel
“c” leert das Spielfeld
Leertaste setzt eine neue Zelle in das aktuelle Feld
Pfeiltasten bewegen den Cursor
Maus kann zum Bewegen des Cursors genutzt werden
“0” fügt ein chaotisches Anfangsmuster ein
“1” fügt einen Glider in das Spielfeld ein
“2” fügt einen Segler in das Spielfeld ein
“3” fügt einen Buffer in das Spielfeld ein
”+” erhöht die Anzahl der Generationen pro Sekunde (frames per second)
”-” erniedrigt die Anzahl der Generationen pro Sekunde (frames per second)