david alfonso

Book review: Performance Testing

2024-05-29T00:00:00+02:00

Subtitle: An ISTQB Certified Tester Foundation Level Specialist Certification Review

Author: Keith Yorkston

Review

The book is oriented towards obtaining the ISTQB Certified Performance Tester certification and uses its syllabus as a guide to organize the contents.

The author does a good job of explaining the certification syllabus, expanding it with information from other resources, adding his own experiences, and including summary boxes to favour studying. It does not feel like other certification books, where understanding and learning are disregarded in favour of an extreme focus on passing the exam.

On the other hand, it only scratches the technical surface of systems performance, sometimes feels repetitive, and assumes you have a testing background (which seems reasonable if you're trying to get the certificate).

It's a good read, even if you're not planning on taking the exam (like me), but I'd recommend skimming over the chapters related to formal processes and to testing tools.

Key Insights

The problem with performance is where the cut-off between “good” and “bad” performance exists.
Performance is a component of a user’s “good experience” and forms part of an acceptable quality level.
Quality attributes of "Performance Efficiency" as per ISO 25010 (which superseded ISO 9126): Time behavior, Resource utilisation, and Capacity.
Performance testing is any kind of testing that focuses on the performance (responsiveness) of a system.
Performance testing can be performed in all phases of the software development lifecycle.
Components of a typical load test: Test controller, Load generators, Scripts (virtual users), SUT, and Monitoring.
There are multiple causes for bad performance. For example:
- Resource saturation (cpu, memory, disk i/o, network i/o)
- Differences between testing and production (e.g. missing a load balancer)
- Graceless error handling (e.g. resource pools, queues, timeouts)
- Database design
- Networking issues
- Unexpected background loads
Performance test goals are defined and achieved by the result metrics gathered during the test. Metrics are defined with measurements.
Metrics are aggregated or derived data that provide insights and summarize the performance (e.g. peak CPU usage, p95 response time).
Measurements are raw, quantitative data points collected directly during performance tests (e.g. individual response times, CPU usage readings).
Metrics and measurements should be defined to be meaningful in the context of the requirements/user stories.
Measurements should be accurate and captured with a level of precision defined by the context.
- Accuracy refers to how close the measurement is to the real value.
- Precision refers to how exact the measurement is (i.e. how close are the measurements to each other).
Typical measurements and metrics are similar to those seen in systems performance analysis.
Consider the Goal-Question-Metric (GQM) approach to select what metrics to use.
A deductive approach conducts multiple test iterations with changes to single items.
A diagnostic approach prefers to capture more information earlier and analyze the results data if there is a problem.
Avoid the probe effect by considering the precision required and the measurement load on the system.
Metrics can be categorized depending on the context where they are needed: Technical, Business, and Operational.
Key sources of metrics: Testing tools, Monitoring tools, and Log analysis tools.
Consider adding a tolerance to time-related requirements to avoid white or black results.
Failing to plan is planning to fail... Have a test plan.
Test planning involves business and technical stakeholders, reviews requirements and user stories, performs a volumetric analysis, and defines the testing environment.
Performance test cases are created in modular form to be used as the building blocks of larger performance tests.
Categories of test data: Master data (base data), User-defined data (test input data), Transactional data (created when executing the test).
Different types of system architectures have different types of performance risks.
Transactions describe a set of activities from initiation to process completion, to be measured as part of the performance test to identify issues.
Think time is used to simulate a real user performing actions. The response time plus the think time is the elapsed time.
Operational profiles describe a user’s interaction with the system. Multiple operational profiles are combined to form a load profile (scenario) to fulfill a test objective.
A load profile combines operational profiles with groups of virtual users ramping up, running for a duration, and ramping down to replicate production load.
Consider throughput and concurrency when modeling operational and load profiles.
System throughput is the amount of transactions processed in a given time. It defines the load on the system. Concurrency represents parallel sessions consuming resources.
Performance scripts should simulate the real system user, in the same order and at the same speed.
Data typically analyzed after a performance test: Status of (virtual) users, Transaction response time, Transactions per second, Transaction failures, Hits (request) per second, Network throughput (amount of data received by users), HTTP responses.
The goal of analyzing the test results is twofold:
- Develop cause-effect relationships chains to uncover root causes of performance issues.
- Prove the SUT has achieved the stated performance requirements.
In performance testing, minimum outliers are better removed as they were probably gathered when the load was still low.
Time, relative or absolute, is the easiest way to correlate metrics.

Certbot webroot auth with Ansible

2024-03-26T00:00:00+01:00

Certbot is the recommended software to generate Let's Encrypt certificates and prove you control the domain. It does so by implementing the ACME protocol.

The webroot authenticator plugin allows to renew certificates without stopping the web server. It works by writing a challenge file to the domain's document root path that will be requested by the Let's Encrypt validation service. This file will live in a hidden folder named .well-known/acme-challenge/ and will be accessed through the standard HTTP port.

Apache configuration

Let's start from the end by making sure that Apache will serve files in the expected path:

<VirtualHost *:80>
  # ...

  # Avoid redirecting to HTTPS if it's an acme challenge request
  RewriteEngine On
  RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
  RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>

We now face a chicken-egg problem where we need a web server in the authentication process but Apache will fail to start if a referenced file does not exist.

A solution is to add a conditional clause to the HTTPS virtual host so that it's only active when the certificate exists:

<IfFile /etc/letsencrypt/live/your-domain.com/fullchain.pem>
  <VirtualHost *:443>
    # ...
    SSLCertificateFile /etc/letsencrypt/live/your-domain.com/fullchain.pem
    # ...
  </VirtualHost>
</IfFile>

This allows to start Apache serving only on the HTTP port, but we'll need to reload the configuration after the certificate is generated.

Ansible orchestration

Let's assume we're managing multiple domains and have these variables defined:

apache_cert_path: /etc/letsencrypt/live
apache_sites:
  - hostname: domain1.com
    # other info for domain1
  - hostname: domain2.com
    # other info for domain2

We need to first install and start Apache with the appropriate configuration. We could use the geerlingguy apache role for that:

- name: Use geerlingguy apache role
  ansible.builtin.include_role:
    name: geerlingguy.apache
  vars:
    # our config...

If we're using a firewall like UFW, we have to open the ports, but note that it won't restart and apply the new config until the end of the play:

- name: UFW - Allow HTTP/HTTPS connections
  community.general.ufw:
    rule: allow
    name: Apache Full
  notify: Restart ufw

Now, before setting up certbot, let's check if there are any certificates missing and, if that's the case, force a UFW restart (reload is not supported) to make sure the HTTP port will be opened for the authentication process.

- name: Certbot - Check if certificates exist
  ansible.builtin.stat:
    path: "{{ apache_cert_path }}/{{ item.hostname }}/fullchain.pem"
  loop: "{{ apache_sites }}"
  register: certbot_certs_stat

- name: Certbot - Force UFW restart
  ansible.builtin.service:
    name: ufw
    state: restarted
  when: certbot_certs_stat.results | rejectattr('stat', 'defined') != []

Finally, using geerlingguy certbot role, create the certificates and reload the Apache config, if necessary.

- name: Certbot - Include certbot role
  ansible.builtin.include_role:
    name: geerlingguy.certbot
  vars:
    certbot_create_method: webroot
    certbot_create_if_missing: true
    # other settings...

- name: Certbot - Force Apache config reload after creating new certificates
  ansible.builtin.service:
    name: apache2
    state: reloaded
  when: certbot_certs_stat.results | rejectattr('stat', 'defined') != []

Discourse dev setup instructions

2024-02-04T00:00:00+01:00

The installation steps for developing Discourse aren't updated and don't match the minimal requirements. Most are figured out easily, but some require some lookup.

Redis

For example, Redis uses Chris Lea's ppa, which doesn't contain packages for Ubuntu 22.04 (jammy). If you had already added it, remove it like this:

sudo add-apt-repository --remove ppa:chris-lea/redis-server

and add the official Redis Apt repo:

curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update

This repository contains the most recent stable versions of Redis, which can be listed with:

apt-cache policy redis

And installed with:

# This is the latest 7.2.x available, which is supposedly
# compatible with Discourse.
sudo apt-get install redis=6:7.2.4-1rl1~jammy1

# Enable it
sudo systemctl daemon-reload
sudo systemctl enable redis-server
sudo systemctl start redis-server

Miscelanea

There's also oxipng, which can be easily installed using Rust's cargo:

cargo install oxipng@8.0.0

Once inside the discourse source folder, there are some more things to do beyond those described in the instructions:

Install the PostgreSQL client libraries: sudo apt install libpq-dev
Use a Node.js version that's compatible, e.g. 18.19: echo "18.19" > .nvmrc && nvm use
Enable corepack, which is the preferred way to install yarn: corepack enable
Install the js dependencies: yarn install

Now we can run all those bundle commands without errors.

Mailpit

Finally, because MailHog seems to be abandoned, I went for Mailpit and installed it with Docker, using some specific options to avoid dealing with authentication in a local environment:

docker run -d \
    --restart unless-stopped \
    --name=mailpit \
    -v /var/mailpit:/data \
    -e MP_DATA_FILE=/data/mailpit.db \
    -e MP_SMTP_AUTH_ACCEPT_ANY=1 \
    -e MP_SMTP_AUTH_ALLOW_INSECURE=1 \
    -p 8025:8025 \
    -p 1025:1025 \
    axllent/mailpit

Removing all my tweets from X/Twitter

2023-11-06T00:00:00+01:00

I want to remove all my tweets. It's been too many years since I've used Twitter, either as a reader or as a writer, and it's about time to leave for good ¹ ². This post explains the steps I've done to achieve it.

The API is useless, long live the API

The free Twitter API tier has been super restrictive since a few months ago. Some developers (who don't make money with it) have started dropping support.

OK, but what about my use case? It turns out I can still delete (and post) tweets! But wait, there are some caveats:

Rate-limit: 50 api requests/day, which is a bit too scarce.
Unknown tweet IDs: No free endpoint available to list your own tweets and get their ID, which is needed to delete them.

The first issue is a matter of abiding by the limits and deleting just 50 tweets per day. I can live with that, but if you have tweets in the order of thousands, you will need some patience. For example, for 10.000 tweets, you'd need 200 days to delete them all :O Maybe you'd be better off deleting your account.

To bypass the second issue, I downloaded a complete account backup file that contains all my tweets as a JavaScript dictionary, including their IDs.

Create a new app in the Twitter Developer Portal

To be able to authenticate yourself in the Twitter API, you need to have a project app created in the Developer Portal.

The first time I filled out the request, I never got a response (was it denied? I don't know). The second time, I tried to explain the intent of the app with more detail. Somehow, it quickly got accepted.

Once I had the app, I needed to generate the following keys and tokens required for OAuth 1 authentication:

Consumer keys: These identify your app so that the API knows what app is doing the request.
Access token and secret: These are user-specific credentials, which is perfect because we want to delete our tweets.

Delete 50 tweets per day

I implemented a Python script that issues a DELETE method request to the /2/tweets API endpoint up to 50 times, using the backup file as the source of Tweet IDs.

The script has the following features:

It receives one command-line argument: the path to the Twitter backup (unzipped).
It reads credentials from environment variables.
Tweets are deleted sequentially in the order they appear in data/tweets.js inside the backup.
The number of deleted tweets is stored in ~/.twitter-delete-all-start-position to continue where it left it.
Use the position file as a marker filer to wait at least 24 hours after its last modification time.

Running the script daily

Given the features and requirements of the script, I want it to run whenever I turn on my computer. Because the script won't call the API unless at least 24 hours have passed since the last time, we can run it multiple times each day without hitting the rate limit.

I'm using Ubuntu 22.04, so I'll go with Systemd as the mechanism to run the script.

First, create the user service file ~/.config/systemd/user/twitter-delete.service:

[Service]
Type=oneshot
RemainAfterExit=true
StandardOutput=journal

# Define DISPLAY for desktop notifications and remember to provide
# all the tokens (you might want to use a more secure alternative
# if you have the need for it)
Environment=DISPLAY=':0' CONSUMER_KEY= CONSUMER_SECRET= ACCESS_TOKEN= ACCESS_TOKEN_SECRET=

# I use pyenv to manage my virtual envs and have one named `scripts`
# just for my scripts.
ExecStart=%h/.pyenv/versions/scripts/bin/python3 %h/scripts/twitter-delete-all.py %h/backups/twitter-2023-10-18.zip

[Install]
WantedBy=default.target

Finally, enable the service and restart the computer to make sure it's working:

$ systemctl --user enable twitter-delete.service

Now, every time my computer is on, the script will run in the background, sleep if necessary, and delete up to 50 tweets of my account.

Well, I'm not going to completely delete my account (for now) as it would be quickly taken by someone that could impersonate me or just be confusing to people that knows me. ↩
It is known that xAI uses all tweets in Twitter to train its new chatbot and when a tweet is deleted it's actually soft deleted, so it remains in Twitter servers. In fact, they are part of the backup archive. In any case, could deleting my Twitter account stop xAI from using my tweets? Most certainly no. ↩

E-mail privacy in GitHub

2023-10-17T00:00:00+02:00

Future commits

Check both Keep e-mail private and Block command-line pushes options in your GitHub e-mail settings.
Set your GitHub private e-mail address globally of per-project:

# current project
$ git config user.email "username@users.noreply.github.com"

# global
$ git config --global user.email "username@users.noreply.github.com"

Past commits

WARNING: These steps rewrite your repository history. Read about collaboration and rewriting history.

Last commit

$ git commit --amend --reset-author

Bulk edition

Install git-filter-repo which is the official git filter-branch replacement.
Clone repo (bare or not)

$ git clone --bare https://hostname/user/repo.git
$ cd repo.git

Update e-mail and/or name from all commits matching your old@email.com note the usage of bytes)

# update e-mail only
$ git-filter-repo --email-callback 'return email if email != b"old@email.com" else b"username@users.noreply.github.com"'

# update both e-mail and name
$ git-filter-repo --name-callback 'return b"Your Name"' --email-callback ...

Re-add remote repository (git-filter-repo deletes remotes as a safety measure to avoid unintended history rewrites!)

$ git remote add origin <remote-url>

Force push changes (!)

$ git push origin -f --all

References

Updating my Neovim config from upstream

2023-09-27T00:00:00+02:00

When I picked a reference Neovim config, one of the main criteria was that it should be updated regularly. This note explains how to bring updates from the upstream repository (i.e. LunarVim/Launch.nvim) to my own config repo.

Bringing changes from upstream to main branch

Creating the repo from a GitHub template only makes it reference the original repo but it's not a real fork and commits are completely unrelated. This means that I can't use any GitHub feature to deal with forked repositories and the process will be manual.

Let's see it step by step.

Add upstream as a remote to the local clone, if not done already:

$ git remote add upstream https://github.com/LunarVim/Launch.nvim

Fetch changes from upstream

$ git fetch --all

Merge remote master branch with local master branch

$ git checkout master

# The first time you merge from upstream the unrelated histories flag is
# required
$ git merge upstream/master --allow-unrelated-histories

# After the first time, we can just merge
$ git merge upstream/master

Rebase my mine branch (the one which contains my changes) with master
- Note: This could be a merge but I don't want to preserve the commits, as I'm the sole user of this repo.

$ git checkout mine

$ git rebase master

Push changes to remote origin repo

# Force should not be required
$ git push origin master

# We have rebased mine, so let's force push
$ git push -f origin mine

Updating the submodule commit in my dotfiles repo

Now that my nvim-config repo is updated, I can do the same with my dotfiles repo which references the former with a submodule.

Cd home, because my dotfiles are my home directory.

$ cd

Pull changes from the just updated remote nvim-config into the local submodule clone.
- Note: The commit referenced in the mine branch is the one we want our submodule to point to.

$ git fetch --recurse-submodules origin
Fetching submodule .config/nvim
From https://github.com/davidag/nvim-config
   ea52394..2d88040  master     -> origin/master
 + f9f407d...e51cad0 mine       -> origin/mine  (forced update)

Make the submodule HEAD point to the new commit

# -C: Cd to the submodule local path before running the Git command
$ git -C .config/nvim reset origin/mine

Push submodule change

# Only a file is modified when updating a submodule
# Verify that the new commit is the one shown in the step above
# (!) If the hash ends in -dirty, you need to enter ~/.config/nvim and clean the repo
$ git diff
diff --git a/.config/nvim b/.config/nvim
index f9f407d..e51cad0 160000
--- a/.config/nvim
+++ b/.config/nvim
@@ -1 +1 @@
-Subproject commit f9f407df596755ad6e03a63e83d6d8ca5c58dcc2
+Subproject commit e51cad0f5dffef66a4a401a1558712f67884c33d

# This force is required because my `.gitignore` contains `*` (this is not
# related to the process described here)
$ git add -f .config/nvim

# Commit the change
$ git commit

# Update origin
$ git push origin

Conclusions

This setup has several advantages:

Upstream changes are brought into the nvim-config master branch without conflicts
My modifications to the original config live in a different branch, mine, and are easily viewable
I can work on my nvim-config directly in the submodule and quickly try new config options
Multiple commits in my neovim config can be translated into one commit in the dotfiles repo, leading to different Git history granularities

Using a Lua-based Neovim config

2023-08-14T00:00:00+02:00

When I migrated to Neovim some time ago, I took the easy path of just using my existing vim configuration (in VimL). It was fine and it worked. But if you want to experiment all the benefits of Neovim, you have to go Lua.

Choosing a base configuration

These are the Lua-based configurations that I considered:

LunarVim/Launch.nvim
- 1k stars, uses lazy.nvim instead of packer, that's the newest trend, updated
- requires installing multiple tools to work
tokiory/neovim-boilerplate
- simpler than lunarvim's, uses less plugins
- both use lazy.nvim to manage plugins
frans-johansson/lazy-nvim-starter
- single-file but with examples to move to multiple files
- documented, modular, small

I decided to use LunarVim's Launch.nvim as it seemed the most powerful, clean and updated.

Learning some Lua

Lua is well known for its simplicity and power. It's also designed to be embedded in applications which make it the perfect option to support user plugins and customizations.

Besides the official docs, just read the learnxinyminutes quick tutorial and have the Neovim references at hand:

Creating my config from the template

I decided to keep my Neovim config in a separate repo from my dotfiles, as I want to keep it in sync with upstream and have more freedom to branch and test new things.

GitHub uses the concept of templates to allow a "soft" clone where you don't inherit all the Git history. Instead, a new repo with the latest state of the template contents is created ¹.

The new repo still points to the original stating:

generated from LunarVim/Launch.nvim

My config repo has a master branch that mirrors upstream and a new mine branch to keep my customizations.

$ git checkout -b mine

$ git push -u origin mine
To github.com:davidag/nvim-config.git
 * [new branch]      mine -> mine
Branch 'mine' set up to track remote branch 'mine' from 'origin'.

$ git branch -vv
  master ea52394 [origin/master] Initial commit
* mine   ea52394 [origin/mine] Initial commit

Referencing Neovim's config from my dotfiles repository

Since my dotfiles repo no longer contains my Neovim's config, there must be some mechanism in place to download it in the proper directory. This is a good use case for Git submodules.

# remove previous contents
~ $ git rm -rf .config/nvim

# add the submodule
~ $ git submodule add -f -b mine \
    https://github.com/davidag/nvim-config .config/nvim

We don't want absolute paths in .gitmodules:

[submodule "nvim"]
  path = .config/nvim
  url = https://github.com/davidag/nvim-config
  branch = mine

Continue setting up the local repo (unnecessary if you clone my dotfiles with --recurse-submodules):

~ $ git submodule init
Submodule 'nvim' (https://github.com/davidag/nvim-config) registered for path '.config/nvim'

When there are changes in my Neovim's config repo, I can update to these changes (since Git 1.8) with:

# This will update the commit referenced in `~/.config/nvim`
~ $ git submodule update --remote --merge

# Commit the new commit reference
~ $ git commit

The commit that removes my old vim config and adds a submodule with the new Lua-based Neovim config can be viewed here.

GitHub docs: Creating a repository from a template ↩

Signing PDF documents

2023-07-29T00:00:00+02:00

I have to "sign" a pdf document with my signature, which it's in a png file.

Let's use pdfstamp, which depends on ImageMagick and the PDF Toolkit (pdftk).

# install dependencies if needed
$ sudo apt install imagemagick pdftk

pdfstamp is available as a node package publised to the npm registry.

Assuming you have a npm/npx working environment (I use nvm to manage node versions):

## install globally with --global/-g
$ npm -g install pdfstamp

Check installation:

$ pdfstamp doctor
running $ which convert
running $ which pdftk

Sign the pdf:

$ pdfstamp stamp \
    --input ./file.pdf \
    --signature ./signature.png \
    --output ./signed.pdf \
    --page 1 \
    --zoom 15 \
    --left 95 \
    --bottom 40

If you get an error that states: "attempt to perform an operation not allowed by the security policy", it's because of a security vulnerability in Ghostscript, just comment the following line:

$ sudo sed -i 's/\(<policy domain="coder" rights="none" pattern="PDF" \/>\)/<!-- \1 -->/' /etc/ImageMagick-6/policy.xml

Debugging PHP in a Docker container

2022-09-03T00:00:00+02:00

I'm migrating a local development setup from Vagrant to Docker and I've had to setup debugging to deal with some problems.

I've used xdebug, PHP's debugger, which supports remote debugging. This is needed because VS Code runs in the host machine and PHP in a Docker container.

Build Docker container with xdebug support

Note: I'm assuming a repository with a docker/ subfolder where all the Docker-related files reside.

Install and enable xdebug in the docker/Dockerfile:

# Assuming the official php docker image
RUN apt-get update \
# ...
&& pecl install xdebug-3.1.5 \
&& docker-php-ext-enable xdebug \
# ...

Configure xdebug in a file docker/conf.d/xdebug.ini that will be mapped inside the container

zend_extension=xdebug

[xdebug]
xdebug.mode=develop,debug
xdebug.client_host=host.docker.internal
xdebug.start_with_request=yes

Add alias for docker host in docker-compose.yaml because I'm on Linux.

services:
  web:
    extra_hosts:
      - host.docker.internal:host-gateway

Configure VS Code

Install the official PHP Debug extension.

Create a launch.json with the default PHP config and include a path mapping from server to host paths:

 "configurations": [
  {
   "name": "Listen for Xdebug",
   "type": "php",
   "request": "launch",
   "pathMappings": {
    "/var/www/html": "/home/username/project/html"
   },
   "port": 9003
  },

Start debugging with "Listen for Xdebug" and set some breakpoint.
Open the container url where Apache is listening and wait for the breakpoint to hit.

Default resource values in the Python CDK

2021-06-02T00:00:00+02:00

Imagine that we want find out what is the default subnet configuration created when instantiating a VPC like this:

from aws_lib import aws_ec2 as ec2, Stack

class MyStack(Stack):
    # ...
    def create_vpc(self):
        # there is an optional `subnet_configuration` parameter
        self.vpc = ec2.Vpc(self, "MyVpc", max_azs=2)

Method 1: Synthetizing

There is one straightforward way to know what resources will be created with a stack: synthetizing it. This means creating the CloudFormation template that would be provisioned if this stack was ever deployed.

You can do this by executing cdk synth and greping for MyVpc.

$ cdk synth --json | jq '.Resources | map_values(.Type)' | grep MyVpc | sort
  "MyVpcF9F0CA6F": "AWS::EC2::VPC",
  "MyVpcIGW5C4A4F63": "AWS::EC2::InternetGateway",
  "MyVpcPrivateSubnet1DefaultRouteA8CDE2FA": "AWS::EC2::Route",
  "MyVpcPrivateSubnet1RouteTable8819E6E2": "AWS::EC2::RouteTable",
  "MyVpcPrivateSubnet1RouteTableAssociation56D38C7E": "AWS::EC2::SubnetRouteTableAssociation",
  "MyVpcPrivateSubnet1Subnet5057CF7E": "AWS::EC2::Subnet",
  "MyVpcPrivateSubnet2DefaultRoute9CE96294": "AWS::EC2::Route",
  "MyVpcPrivateSubnet2RouteTableAssociation86A610DA": "AWS::EC2::SubnetRouteTableAssociation",
  "MyVpcPrivateSubnet2RouteTableCEDCEECE": "AWS::EC2::RouteTable",
  "MyVpcPrivateSubnet2Subnet0040C983": "AWS::EC2::Subnet",
  "MyVpcPublicSubnet1DefaultRoute95FDF9EB": "AWS::EC2::Route",
  "MyVpcPublicSubnet1EIP096967CB": "AWS::EC2::EIP",
  "MyVpcPublicSubnet1NATGatewayAD3400C1": "AWS::EC2::NatGateway",
  "MyVpcPublicSubnet1RouteTableAssociation2ECEE1CB": "AWS::EC2::SubnetRouteTableAssociation",
  "MyVpcPublicSubnet1RouteTableC46AB2F4": "AWS::EC2::RouteTable",
  "MyVpcPublicSubnet1SubnetF6608456": "AWS::EC2::Subnet",
  "MyVpcPublicSubnet2DefaultRoute052936F6": "AWS::EC2::Route",
  "MyVpcPublicSubnet2EIP8CCBA239": "AWS::EC2::EIP",
  "MyVpcPublicSubnet2NATGateway91BFBEC9": "AWS::EC2::NatGateway",
  "MyVpcPublicSubnet2RouteTable1DF17386": "AWS::EC2::RouteTable",
  "MyVpcPublicSubnet2RouteTableAssociation227DE78D": "AWS::EC2::SubnetRouteTableAssociation",
  "MyVpcPublicSubnet2Subnet492B6BFB": "AWS::EC2::Subnet",
  "MyVpcVPCGW488ACE0D": "AWS::EC2::VPCGatewayAttachment",

In this case, all associated resources begin with the same name, but in more complex cases it might be better to grep in the raw synthethized output and find out dependent resources.

Method 2: Looking at the CDK library resource

By synthetizing our stack, we've learnt that two private and two public subnets, along a couple of NAT gateways, would be created. But where is this defined?

The key is that all supported CDK libraries end up calling the TypeScript CDK library. This is done through the jsii library:

jsii allows code in any language to naturally interact with JavaScript classes. It is the technology that enables the AWS Cloud Development Kit to deliver polyglot libraries from a single codebase!

Looking at the Python definition of aws_ec2.Vpc we can see the actual name to look for in the CDK library:

@jsii.implements(IVpc)
class Vpc(
    _Resource_45bc6135,
    metaclass=jsii.JSIIMeta,
    jsii_type="aws-cdk-lib.aws_ec2.Vpc",  # This is the class in the aws-cdk-lib
):

We just have to open the appropriate file. In this case, packages/aws-cdk/aws-ec2/lib/vpc.ts (link) and search for class Vpc. These are the relevant contents:

export class Vpc extends VpcBase {
  // ...

  /**
   * The default subnet configuration
   *
   * 1 Public and 1 Private subnet per AZ evenly split
   */
  public static readonly DEFAULT_SUBNETS: SubnetConfiguration[] = [
    {
      subnetType: SubnetType.PUBLIC,
      name: defaultSubnetName(SubnetType.PUBLIC),
    },
    {
      subnetType: SubnetType.PRIVATE_WITH_NAT,
      name: defaultSubnetName(SubnetType.PRIVATE_WITH_NAT),
    },
  ];

  // ...

  /**
   * Vpc creates a VPC that spans a whole region.
   * It will automatically divide the provided VPC CIDR range, and create public and private subnets per Availability Zone.
   * Network routing for the public subnets will be configured to allow outbound access directly via an Internet Gateway.
   * Network routing for the private subnets will be configured to allow outbound access via a set of resilient NAT Gateways (one per AZ).
   */
  constructor(scope: Construct, id: string, props: VpcProps = {}) {
    // ...
    const defaultSubnet = props.natGateways === 0 ? Vpc.DEFAULT_SUBNETS_NO_NAT : Vpc.DEFAULT_SUBNETS;
    this.subnetConfiguration = ifUndefined(props.subnetConfiguration, defaultSubnet);
    // ...

And there it is, the explanation of why two public and two private subnets along two NAT gateways were being defined.

Forking Watson: License

2020-06-05T00:00:00+02:00

DISCLAIMER: I'm not an expert in software licensing by any means, so please don't take this post as legal advice.

As described in a previous post, I'm doing a fork of Watson, a time tracking command-line tool. You can find in there my motivations for doing a fork instead of other alternatives. The subject of this post is describing why and how I've switched the forked project license from MIT/Expat to GNU GPL¹, what's the REUSE project and how to apply it to help in the migration process.

Why would anyone ever want to modify the license?

Both MIT and GPL are OSI approved licenses, meaning both are accepted by the Open Source Initiative as complying with their definition of Open Source Software.

The GNU General Public License (GNU GPL) is a license created by the Free Software Foundation, which defines free software using a four-point definition. You can find arguments in favor and against the OSI and the FSF definitions in their respective sites and all over the Internet.

The GPL, in its third revision (GPLv3), adds some interesting features, though not very relevant for my project. What matters most, is that the GPL is a copyleft license:

Copyleft is a general method for making a program (or other work) free (in the sense of freedom, not “zero price”), and requiring all modified and extended versions of the program to be free as well.

This is not the case of the MIT/Expat license, which allows you to distribute the software (and any modifications) under a non-free license, ie. not complying with any or all of the four free software points. This is why it is said that the MIT license is more permissive than the GPL.

Choosing one license or another is a matter of circumstances, strategy, and convictions. In a small project like this, not distributed as a library, it's a conviction that takes most of the weight.

Switching from MIT to GPL

First of all, it's important to note that I'm not changing the license of the original project. Watson is a different piece of software and is still MIT-licensed. It is the forked software (xtimetracker) license that is being changed. Relicensing a software project with many contributors can be quite challenging and is not the subject of this post.

Second, we can take this upgrade path because the MIT license is GPL-compatible, meaning we can combine code under the MIT and GPL licenses in one larger program and distribute it as GPL.

Finally, the original project only stated the license in a LICENSE file in the project root. However, according to the GPL FAQ, license notices should be included in all relevant files, especially when mixing permissive and GPL-licensed files and, in this case, also for honesty's sake. This means all files should be modified to state copyright and license information.

Enter REUSE

Applying multiple licenses to the same files is not an easy task, legally speaking; that's the reason I found the REUSE initiative so appealing to the problem at hand.

REUSE defines a set of recommendations to apply copyright and licensing information to software projects. It was created by the Free Software Foundation Europe (FSFE) to make licensing clear both to humans and computers. That's the reason there is a tool to automate many tasks involved in applying the REUSE recommendations, including compliance validation according to the latest REUSE specification.

Reality is that other tools are not yet compatible with the REUSE recommendations and that the reuse tool itself is under continuous development to handle the large number of special cases that can arise in software projects. This means you'll probably have to manually handle and verify most of your files (which is recommended anyway).

How to declare multiple licenses

The current situation is such that most project files have already been modified by me. This means that both MIT and GPL licenses should apply to them. Specifically, the parts that I've coded are licensed under the GPL license, and the parts from the original project fall under the MIT license. However, this distinction is difficult to state explicitly.

When multi-licenses apply, each file should list what licenses apply in separate SPDX-License-Identifier tags (wait, what's an SPDX identifier?).

In the project properties (e.g. your setup() function in a Python package), the AND operator should be used because the project is governed by two different licenses:

# setup.py
license="GPL-3.0-or-later AND MIT"

You would use the OR operator when the user can choose between them.

Using the `reuse` tool to include licensing information

Let's run reuse to change all files with supported comment styles in bulk. Note that we could apply multiple copyrights and licenses at once, but, because the years are different, we have to run two commands:

# OLD COPYRIGHT
$ git ls-files | xargs reuse addheader --skip-unrecognised \
    --copyright="Tailordev" --license=MIT --year=2015-2019

# NEW COPYRIGHT
$ git ls-files | xargs reuse addheader --skip-unrecognised \
    --copyright="David Alfonso" --license=GPL-3.0-or-later --year=2020

Now we can list the files missing licensing information and deal with each of them on a case-by-case basis:

$ reuse lint
### MISSING COPYRIGHT AND LICENSING INFORMATION

The following files have no copyright and licensing information:
* .flake8
* MANIFEST.in
* requirements-dev.txt
* tests/resources/autocompletion/frames
* tests/resources/sample_data/frames
* tt.completion
* tt.zsh-completion

The following files have no licensing information:
* README.rst

If we know the comment style of any of these files, we can force the reuse tool to use it. For example:

$ reuse addheader --copyright="David Alfonso" \
    --license=GLP-3.0-or-later --year=2020 \
    --style=python tt.completion

You can find the style shorthands inspecting the reuse code.

Licensing non-code files

According to the REUSE tutorial, files which can be considered insignificant, like configuration files, might be licensed differently. One alternative is the CC0 license, which is very similar to releasing the file to the public domain. However, because I'm already using the MIT license, and to avoid blocking contributors, I prefer to use the MIT license for this kind of files.

Conclusions

This post has tried to explain the findings and tools I've found along the way of switching the license of a forked FOSS project.

Of special importance was the REUSE website which has proved to be a great resource for solving licensing doubts, together with the GPL FAQ.

Finally, the reuse tool has turned out very effective in automating the process of adding copyright and license information to all files.

When I write GPL, please assume GNU GPLv3 or later. Being forward-compatible is relevant. ↩

POSIX Shell Scripting

2020-05-07T00:00:00+02:00

The Portable Operating System Interface (POSIX) family of standards are specified by the IEEE to maintain OS interoperability¹.

The standardized user command line and scripting interface were based on the UNIX System V shell. But not only these are standardized, also are common utility programs.

POSIX.1-2008 combined most parts of previous POSIX standards into one. Latest revision is POSIX.1-2017

Resources

POSIX.1-2017 Specification with search box and frames

POSIX.1-2017 Rationale TOC (e.g. to be strictly POSIX compliant there must be NO #! at the beginning of the file)

The Shell Command Language Documentation

POSIX Utility Conventions: Naming of utilities, specification of options, option-arguments, and operands (The getopt() function assists in this matter).

Shell Portability Guidelines: If scripts need to be portable, some of the BASH-specific syntax elements should be avoided. Others should be avoided for all scripts, e.g. if there is a corresponding POSIX®-compatible syntax.

Pure sh Bible: a collection of pure POSIX alternatives to external processes.

Shells

Evolution of Shells in Linux

Comparison of command shells

Hyperpolyglot: Unix Shells (bash, fish, ksh, tcsh, zsh)

Dash as default /bin/sh shell in Ubuntu: Dash is more efficient than Bash. For system scripts this matters. It was changed in Ubuntu 6.10 (long time ago). Includes a list of "bashisms"². It was Ubuntu who migrated first to Dash, and then Debian decided to adopt it.

ash, the original Kenneth Almquist shell. It's a clone of the Bourne Shell to avoid licensing issues. It's the default shell in BusyBox.

dash: The Debian Almquist Shell is a modern POSIX-compliant implementation of /bin/sh. It has a low memory footprint compared to Bash. It's used in Debian as the default non-interactive shell.

mrsh: a minimal POSIX shell.

Tooling

shellcheck: a static analysis tool for shell scripts. Use shellcheck -s sh (or if script starts with #!/bin/sh) to check for POSIX and warn on portability issues.

ts(1) -- test script: ts provides functions for writing tests in shell. The test scripts can be run individually or in a batch format using ts as a command. ts makes a test directory available on a per-test basis so it's easy to sandbox tests that write or manipulate files. ts tries to use POSIX exclusively and so should (hopefully) work with any POSIX-compliant shell.

checkbashisms

Examples

A shell script to demonstrate various POSIX features related to shell variables

KISS package manager: Tiny and straightforward package manager for KISS (a Linux distribution) written in 600 lines of POSIX sh. Related software. Script related guidestones:

All shell code must be written in a safe way, pass the shellcheck linter and match the style of any existing code.
All distribution tooling and shell code must be written in a portable way. Otherwise, the user will be locked into a single coreutils and shell.
One exception is made for sed -i as it is too useful to let go of. The -i flag has rather good support across implementations regardless.

Static site generator (180LoC), written in shell.

RSS feed generator (148LoC), written in shell.

git-issue: Git-based decentralized issue management. Interesting shellchecks to support, e.g. local variables, which are not standard but supported by most modern shells.

asdf-vm: Extendable version manager. Supports many shells.

bashmarks: Save and jump to commonly used directories. Has tab completion embedded in the same script (zsh and bash).

https://en.wikipedia.org/wiki/POSIX ↩
bashism: "a shell feature that is not required to be supported by POSIX". ↩

Forking Watson: Motivations

2020-05-05T00:00:00+02:00

This story began about a year ago when I started contributing to the Watson project. It was a good experience and I learnt a lot, especially about collaborating in a FOSS project. Nevertheless, along the way, I noticed some patterns and signals that hinted me that this project was not for me. But let's start from the beginning.

What is a fork?

There are (at least) two types of forks:

A development fork is just a copy of the original project where a contributor makes some changes and submits those back to the main project. This meaning of fork is controversial as it's mostly associated to the GitHub concept of Fork, which is far in intention and meaning from the Bazaar model where contributors just clone the original repository, make some changes and send them back (usually in patch form using e-mail).
A hard fork is when some people (or even just one person) decides to part ways with an existing project (the forked project) and start its own independent project (the fork) based on the former. This can be done for a myriad of reasons, in multiple ways and with infinite purposes.

This post refers to the second meaning and I will use the term "fork" meaning "hard fork" through the rest of the text.

Why forking?

The general advice is to fork a project only as a last resort measure, as it effectively splits the efforts weakening both the fork and the forked projects. Still and all, there are multiple reasons to fork a FOSS project and one or more can apply at the same time. Let's review some of them in no particular order:

The project leadership does not align with your personal values

As a contributor, you should evaluate the value system of the leadership and make a personal determination as to whether or not it aligns with your own. If it does, participate. If it does not, find an alternative or fork the project. Drew DeVault - Effective project governance
You have a different vision of the direction the project should follow

Some forks are due to amicable but irreconcilable disagreements about the direction of the project; perhaps more are due to both technical disagreements and interpersonal conflicts. Of course, it's not always possible to tell the difference between the two, as technical arguments may involve personal elements as well. What all forks have in common is that one group of developers (or sometimes even just one developer) has decided that the costs of working with some or all of the others now outweigh the benefits. Producing OSS - Forks
The project leadership has left or lost the original motivation, ie. the project is abandoned/unmaintained

In my opinion, there are two good ways to abandon a project: the fork it option and the hand-off option. The former is faster and easier, and you can pick this if you want to wash your hands of the project ASAP, but has a larger effect on the community. The latter is not always possible, requires more work on your part, and takes longer, but it has a minimal impact on the community. Drew DeVault - How to abandon a FLOSS project

You will come up with more if you try, but these seem to me like the most common reasons to fork a project.

The case of Watson

Watson is a small tool to track your time from the command line. It was built by the French company TailorDev. As part of their open core values, they developed a variety of very interesting projects that had quite an impact on the global software development audience. They were an inspiration to many people (among which I include myself).

Nevertheless, their activity started to slow down since 2018 and their opensource projects switched to maintenance mode, not because they were complete, but because it was no longer a priority for the people involved. Despite this, they still care and answer questions from the community, always in a kind and friendly manner.

For me, the problem is that these projects (and Watson specifically) have many traits of what I've called a FOSS zombie project. It sounds worse than the maintainers deserve, but I couldn't find a better metaphor to convey the situation as I see it: a project that is alive, but dead.

I'd also like to say that after speaking privately with the maintainers, there is a disposition to improve the current situation to some extent; though they are OK with the current state of things.

To sum it all up, as I haven't seen a clear path to keep participating in Watson, I've prefered to take the situation as a challenge to learn about FOSS maintenance and simplify and customise the software to fit my needs (and hopefully the ones of others too).

Khal: Codebase review

2020-04-05T00:00:00+02:00

Project information

Code repository: https://github.com/pimutils/khal
License: Expat/MIT License (license and copyright notice)
Type of program: Python cli and terminal application
Supported Python versions: 3.4+ (see setup.py or tox.ini)
Version reviewed: v0.10.1

Documentation

High-level analysis

This analysis is based on setup.py contents and on a first look at the code.

Concepts

iCalendar: A standard specification for Internet Calendaring and Scheduling. It defines a MIME type and files that comply with this specification usually have an extension of .icf.
vdir: A storage format for storing calendars and contacts in the filesystem with the goal of being easy to implement. It uses the iCalendar and vCard formats, where each file contains one contact or event.
etag and ctag: The concept of ETag is used in HTTP as a mechanism to provide web cache validation. In Khal, both etag and ctag are used for synchronizing the SQLite database cache with the actual events and calendars.

Dependencies

Runtime dependencies:

click: composable command line interface toolkit
click_log: logging integration for Click
icalendar: iCalendar parser/generator
urwid: a full-featured console (xterm et al.) user interface library
pyxdg: implementations of freedesktop.org standards in Python
pytz: world timezone definitions, modern and historical
python-dateutil: extensions to the standard Python datetime module
configobj: config file reading, writing and validation
atomicwrites: atomic file writes
tzlocal: tzinfo object for the local timezone

Testing dependencies:

freezegun: let tests travel through time by mocking the datetime module
vdirsyncer: synchronize calendars and contacts

Extra dependencies:

proctitle: context manager to set/reset the current process name

Package organization

khal: The main package. It contains all CLI-related code based in Click that handles the creation of the CalendarCollection according to the configuration and arguments, and delegates to the multiple controller functions operations over the collection depending on the requested user action. There are modules for iCalendar utils, converting strings to datetime or events objects, handling the Terminal (colors, columns), and displaying one or more months along its corresponding events in the console.
khal.khalendar: Exports a CalendarCollection class which allows access to various calendars stored in vdirs and cached in an SQLiteDB for performance. This collection uses an Event model that represents recurring or non-recurring instances of events. This package also contains a module with tools to read/write from/to vdirs.
khal.settings: Handles Khal configuration, file paths, and vdir config.
khal.ui: Contains multiple widgets for urwid (a console UI library) and all the code that glues together the khal.khalendar package with the UI. In this case, the controlling is performed by urwid which handles events to the UI classes defined in this module.

Entry points

Two console scripts:

khal = khal.cli:main_khal, which provides a command-result terminal experience.
ikhal = khal.cli:main_ikhal, which provides a TUI (terminal-user interface) handled by code in khal.ui.

Configuration and data files

Configuration

The configuration file is looked up into the list of paths defined by the XDG Base Directory Standard using the pyxdg package. The preferred name of the file is config.
The configuration file contents are handle using the configobj library, and its contents specified by the file khal.spec.

Calendars (data)

The path of calendars is defined in the configuration file.
The used data format is the Vdir Storage Format, defined in the vdirsyncer project (also under the pimutils organization umbrella).

Architecture

The following diagram tries to represent the components and relations of the khal (cli) application. This is not complete by any means, many pieces are missing and some might be wrong.

Object-oriented design

The khalendar package contains code which is more object-oriented than in other khal's modules. Let's analyze its classes:

The CalendarCollection class can be seen as this package interface. It uses composition to make use of the SQLiteDb class implementation and also is composed of one Vdir object per configured calendar. It also allows CRUD of Events saved in the SQLite cache and in vdirs using multiple criteria.

Events are represented as Event class instances, which is a base class for different derived classes:

LocalizedEvent, which handles start and end datetime timezones.
FloatingEvent, which are not all day events.
AllDayEvent, which are all day events.

The vdir module makes use of many OO design concepts:

The class Vdir, which is the one used from the outside, is just a mixture of behavior and interfaces from three other classes it inherits from:
1. VdirBase which implements operations in a vdir folder, including handling metadata information or writing, reading and deleting data.
2. DisplayNameMixin which provides methods for accesing the displayname metadata file.
3. ColorMixin which provides methods for accessing the color metadata file.
Multiple exception classes inheriting from VdirError(IOError).
An Item class, used from the controllers to import items in a calendar.
A Color class only used in the ColorMixin class.

Besides the khalendar package, the other interesting package is ui, which contains multiple classes following urwid's object oriented development design.

Python techniques

The logging module is used to show the user warnings in multiple parts of the code. There is a khal logger:
```
logger = logging.getLogger('khal')
```
Use __init__.py files in subpackages to auto-import "exported" functions/classes.
- In khal/settings/__init__.py one function and one class are made available through settings.get_config and settings.InvalidSettingsError respectively:
```
from .settings import get_config  # noqa
from .exceptions import InvalidSettingsError  # noqa
```
- Then, in khal/cli.py is enough to do:
```
from .settings import InvalidSettingsError, get_config
```
Type hints are used in most method arguments of backend.SQLiteDb (see code), while they're not used at all in other classes.

Avoid class instantiation to simulate a pure abstract base class (e.g Event):

def __init__(self, vevents, ref=None, **kwargs):
   if self.__class__.__name__ == 'Event':
        raise ValueError('do not initialize this class directly')

Factory methods using @classmethod vs @staticmethod in Event base class (see code):
- A simple factory method is provided by Event._create_calendar() using the @staticmethod decorator, because the method is logically contained in Event but does not use any of its methods nor it creates a Event object.
- The @classmethod decorator is used to define a couple of more complex factory methods. They are based on VEvents (Event.fromVEvents()) or on a string (Event.fromString()), detecting the correct class to instantiate based on the attributes of the provided elements.
- In the same module, event.py, there is a factory method for creating icalendar vtimezones from pytz.tzinfo objects. This method is outside any class definition but is there probably because of its relation with events and is, in fact, used only from a method in the Event class (see code).
Comparable events by overloading __lt__() in the Event class. This class handles the different types of date/time objects that can be used in an Event-derived object.

Context managers, for example SQLiteDb.at_once(), allows to perform multiple operations on the database atomically, by delaying the SQLite transaction commit until the scope is exited without exceptions being raised (see code).

    @contextlib.contextmanager
    def at_once(self):
        assert not self._at_once
        self._at_once = True
        try:
            yield self
        except:  # noqa
            raise
        else:
            self.conn.commit()
        finally:
            self._at_once = False

Example usage in khalendar.py:update() (see code) where an event is updated both in a vdir and in the cache database.

    with self._backend.at_once():
        event.etag = self._storages[event.calendar].update(event.href, event, event.etag)
        self._backend.update(event.raw, event.href, event.etag, calendar=event.calendar)
        self._backend.set_ctag(self._local_ctag(event.calendar), calendar=event.calendar)

A class-based decorator named cached_property in the khalendar.vdir module which transforms an instance method into a descriptor object attribute:

# Implements the descriptor protocol making this object a descriptor,
# specifically a non-data descriptor because it implements only the
# __get__ method.
class cached_property:
    '''A read-only @property that is only evaluated once. Only usable
    on class instances' methods.
    '''
    def __init__(self, fget, doc=None):
        self.__name__ = fget.__name__
        self.__module__ = fget.__module__
        self.__doc__ = doc or fget.__doc__
        self.fget = fget

    # This method is called once because an attribute is added to the
    # object via the self.__dict__ object and, in the case of non-data
    # descriptors, the instance's dictionary entry takes precedence.
    def __get__(self, obj, cls):
        if obj is None:  # pragma: no cover
            return self
        obj.__dict__[self.__name__] = result = self.fget(obj)
        return result

We can use it like this:

class A:
    # When this method is processed by the interpreter, an instance
    # of the class cached_property is created with this function as
    # argument 'fget'.
    @cached_property
    def prop_name(self):
        return 1

Usage example

>>> a = A()
>>> vars(a)
{}
>>> print(a.prop_name)
1
>>> vars(a)
{'prop_name': 1}

There is much more in khal and I recommend you to take a look at the code and find out by yourself ;-)

PHP 7.2 Docker image analysis

2019-11-15T00:00:00+01:00

Base Docker images are full of sysadmin best practices and interesting tips to learn from. Besides that, knowing what's going on behind the curtains of a FROM is mandatory if you plan to trust a production system with it.

TL;DR image features:

It uses Apache2 from Debian repositories.
It installs PHP following the official installation recommendations.
Apache can be run as an arbitrary user: /var/www/html has 777 permissions.
PHP configuration resides in: /usr/local/etc/php/.

Let's review all sections of the php:7.2-apache Docker base image mentioning some shell, Debian and Dockerfile best practices along the way.

Base image

FROM debian:buster-slim

The image is based on Debian buster, aka Debian 10. Specifically, a slimmed down version of it. But, hey, we're not going down this rabbit hole.

RUN instructions

The RUN instruction, in its shell form, will execute the given command in the /bin/sh shell using the -c option.

In Debian, /bin/sh corresponds to the Debian Almquist Shell (dash) which strives to be a POSIX-compliant and slim shell. This means no "bashisms" in RUN instructions (unless you run /bin/bash -c, of course).

As we will see, the first subcommand is always this:

set -eux

This will enable the following shell options:

errexit: exit immediately if any untested command fails.
nounset: exit immediately if attempting to expand an unset variable.
xtrace: write each command to stderr preceded by a + before being executed.

Block official PHP Debian packages

RUN set -eux; \
    { \
        echo 'Package: php*'; \
        echo 'Pin: release *'; \
        echo 'Pin-Priority: -1'; \
    } > /etc/apt/preferences.d/no-debian-php

This image compiles and installs its own PHP version and extensions directly without using the corresponding Debian packages. This is because:

They strive to use the recommended PHP installation method.
They want to be able to keep up with upstream releases since Debian will only update PHP in stable releases if there are security vulnerabilities.

This is implemented using an apt preferences file, by preventing installation (Pin-Priority: -1) of all packages whose name starts with php (using a glob expression: Package: php*) belonging to all distribution releases (Pin: release *).

Installing PHP compilation dependencies

ENV PHPIZE_DEPS \
    # ...
RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        $PHPIZE_DEPS \
        ca-certificates \
        curl \
        xz-utils \
    ; \
    rm -rf /var/lib/apt/lists/*

This is a typical Dockerfile pattern for installing package dependencies:

Update repositories.
Install packages, assuming yes to all answers and not installing recommended packages.
Cleaning the storage area for state information for each package resource in the available repositories. This will save hundreds of MB in the resulting image.

Following Dockerfile best practices, it's important to execute them all in the same RUN command (ie. in the same Docker layer).

Separating phpize dependencies in an environment variable allows having them both organized and sorted alphabetically, at the same time that there is only one apt-get install call (ie. one layer).

It's worth noting that the package cache in /var/cache/apt/archives/ is not being explicitly removed. This is because the official Debian base image is already executing apt-get clean after every install (see code).

Setup PHP / Apache directories

ENV PHP_INI_DIR /usr/local/etc/php
RUN set -eux; \
    mkdir -p "$PHP_INI_DIR/conf.d"; \
# allow running as an arbitrary user
    [ ! -d /var/www/html ]; \
    mkdir -p /var/www/html; \
    chown www-data:www-data /var/www/html; \
    chmod 777 /var/www/html

Sets PHP_INI_DIR environment variable, which is later used to specify the php configuration path /usr/local/etc/php, and create Apache's html root directory with all permissions granted to all users.

Shell tip: Note that /var/www/html shouldn't exist before this RUN instruction; otherwise, this will halt in [ ! -d /var/www/html ] and stop the image building process.

Install Apache 2

Another RUN instruction is executed with all the following steps inside (ie. again, all in one layer to reduce size):

Apache2 from Debian repositories is installed following the aforementioned technique (ie. apt-get install).
Allow Apache environment variables to be overriden at runtime from the Docker cli (e.g. -e APACHE_RUN_USER=...). This is accomplished by modifying /etc/apache2/envars using the following sed invocation:
```
sed -ri 's/^export ([^=]+)=(.*)$/: ${\1:=\2}\nexport \1/' "$APACHE_ENVVARS";
```
- Command options: -r enables extended regexps and -i edits files in place.
- Matches lines like: export SOME_VAR=any value
- Transforms these lines to the form:
```
: ${SOME_VAR:=any value}
export SOME_VAR
```
- The form ${parameter:=word} assigns word to parameter if it was not set.
- The colon character (:) before the string assignment is the builtin null command that returns a 0 (true) exit value. This allows executing the string assignment without running the resulting parameter value (and getting an error when doing it).
Using another builtin command (.), reads and executes the just modified Apache /etc/apache2/envars file in the current shell.

Grant all permissions to all users and changes ownership of Apache's lock, log, and run folders, creating them from scratch.

for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR"; do
    rm -rvf "$dir";
    mkdir -p "$dir";
    chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir";
    chmod 777 "$dir";
done

Delete the index.html created by Apache.
```
rm -rvf /var/www/html/*
```
Redirect all Apache log contents to stdout / stderr by creating symbolic links to the corresponding device files
```
ln -sfT /dev/stderr "$APACHE_LOG_DIR/error.log"
ln -sfT /dev/stdout "$APACHE_LOG_DIR/access.log"
ln -sfT /dev/stdout "$APACHE_LOG_DIR/other_vhosts_access.log"
```
- -s: Create symbolic link (not hard link).
- -f: Remove existing destination if exists.
- -T: Treat the destination name (the log file) as a normal file (not a directory).
Finally, change the owner of all log files recursively (-R):
```
chown -R --no-dereference "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$APACHE_LOG_DIR"
```
- --no-dereference changes the symbolic link owner (vs the link target).

Enable preforking MPM

RUN a2dismod mpm_event && a2enmod mpm_prefork

Multi Processing Modules (MPMs) are responsible for binding to network ports, accepting requests, and dispatching children to serve them. Only one mpm module can be enabled at the same time.
According to the PHP manual, a threaded MPM is not recommended with Apache2; that's why the prefork module is enabled.

Make PHP handle PHP file requests

RUN { \
        echo '<FilesMatch \.php$>'; \
        echo '\tSetHandler application/x-httpd-php'; \
        echo '</FilesMatch>'; \
        echo; \
        echo 'DirectoryIndex disabled'; \
        echo 'DirectoryIndex index.php index.html'; \
        echo; \
        echo '<Directory /var/www/>'; \
        echo '\tOptions -Indexes'; \
        echo '\tAllowOverride All'; \
        echo '</Directory>'; \
    } | tee "$APACHE_CONFDIR/conf-available/docker-php.conf" \
    && a2enconf docker-php

A specific configuration file is created, which is later enabled with the a2enconf command.
It's interesting how it writes the contents of a file by using a grouping command piped into the tee utility, which outputs all it reads from stdin to stdout and also to any file given as a parameter.

From the dash man page: "Note that “}” must follow a control operator (here, “;”) so that it is recognized as a reserved word and not as another command argument."

Establish PHP compilation options and dependencies

ENV PHP_EXTRA_BUILD_DEPS apache2-dev
ENV PHP_EXTRA_CONFIGURE_ARGS --with-apxs2 --disable-cgi

We'll require the apache2-dev Debian package which contains development headers and the apxs2 binary, apart from some debhelper scripts.
apxs2 is the APache eXtenSion tool which is used for building and installing extension modules: Dynamic Shared Objects (DSOs). These extensions are loaded into Apache using the mod_so module.
--disable-cgi avoids building the CGI version of PHP and implicitly enables FastCGI.

ENV PHP_CFLAGS="-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"
ENV PHP_CPPFLAGS="$PHP_CFLAGS"
ENV PHP_LDFLAGS="-Wl,-O1 -Wl,--hash-style=both -pie"

GCC compiler flags related to code generation conventions:
- -fpic: generate position-independent code (PIC) for use in shared libraries.
- -fpie: similar to -fpic but the PIC generated can only be linked into executables. It requires to link using -pie.
GCC compiler flags related to program instrumentation for error/attack detection
- -fstack-protector-strong: check for buffer overflows on functions with vulnerable objects (ie. functions that call alloca, have buffers larger than 8 bytes, have local array definitions or have references to local frame addresses).
GCC compiler flags related to control optimization
- -O2: Optimize for reduced code and execution time, without involving space-speed tradeoff optimizations, but taking more time to compile than -O1.
Large File Support (LFS) is enabled as per the PHP documentation.
GNU linker (ld) flags:
- -Wl,-O1: optimize shared library linkage.
  
  From Dockerfile: "this sorts the hash buckets to improve cache locality, and is non-default"
- -Wl,--hash-style=both: set the linker's hash table type for both the classic ELF ".hash" and the new style GNU ".gnu.hash".
  
  From Dockerfile: "GNU hash is used if present, and is much faster than sysv hash; in this configuration, sysv hash is also generated".
- -pie: required to use PIC generated objects.

Establish PHP version and validation mechanisms

ENV GPG_KEYS 1729F83938DA44E27BA0F4D3DBDB397470D12172 B1B44D8F021E4E2D6021E995DC9FF8D3EE5AF27F

Set an environment variable with a couple GPG key IDs required to verify PHP source packages.
Key IDs are V4 fingerprints (160-bits).

ENV PHP_VERSION 7.2.25
ENV PHP_URL="https://www.php.net/get/php-7.2.25.tar.xz/from/this/mirror" \
    PHP_ASC_URL="https://www.php.net/get/php-7.2.25.tar.xz.asc/from/this/mirror"
ENV PHP_SHA256="746efeedc38e6ff7b1ec1432440f5fa801537adf6cd21e4afb3f040e5b0760a9" \
    PHP_MD5=""

Set the PHP version, source URL, signature URL and SHA256 hash.
The .asc file extension implies the signature is in plain-text.

Download PHP source code and verify its integrity

This corresponds to a long RUN instruction composed of multiple steps:

RUN set -eux; \

Save the list of manually installed packages using apt-mark showmanual.

From the man pages: When you request that a package is installed, and as a result other packages are installed to satisfy its dependencies, the dependencies are marked as being automatically installed, while the package you installed explicitly is marked as manually installed.
Install the gnupg and dirmngr packages using apt as seen previously.

gnupg

GNU privacy guard - a free PGP replacement

dirmngr

GNU privacy guard - network certificate management service
Create /usr/src directory and enter on it.
Download the PHP sources:
```
curl -fsSL -o php.tar.xz "$PHP_URL"; \
```
- -f: fail silently on server errors (no output at all).
- -s: be silent, don't show any progress or messages.
- -S: show error messages even if -s is provided.
- -L: follow redirects (Location: header and a 3xx response code).
- -o php.tar.xz: write output to php.tar.xz instead of stdout.
Check source code integrity using SHA256 and/or MD5. In this case, only the more secure SHA256 hash was previously defined.
```
if [ -n "$PHP_SHA256" ]; then \
    echo "$PHP_SHA256 *php.tar.xz" | sha256sum -c -; \
fi; \
```
- -: read from standard input instead of a file.
- -c: read sums from stdin and check them.
From the sha256sum man: "When checking, the input should be a former output of this program. The default mode is to print a line with checksum, a space, a character indicating input mode ('' for binary, ' ' for text or where binary is insignificant), and name for each FILE*."
Download and verify signature:
```
if [ -n "$PHP_ASC_URL" ]; then \
    curl -fsSL -o php.tar.xz.asc "$PHP_ASC_URL"; \
    export GNUPGHOME="$(mktemp -d)"; \
    for key in $GPG_KEYS; do \
        gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; \
    done; \
    gpg --batch --verify php.tar.xz.asc php.tar.xz; \
    gpgconf --kill all; \
    rm -rf "$GNUPGHOME"; \
fi; \
```
1. Download .asc signature file.
2. Create a temporary directory and use it as GNUPGHOME (default is ~/.gnupg otherwise).
```
export GNUPGHOME="$(mktemp -d)"; \
```
  - mktemp prints to stdout the file or directory created and we use $() to substitute its output in place of the command name itself, ie. the temp folder name. See the Dash Command Substituion man section.
  - Mark for automatic export to the environment the GNUPGHOME variable to subsequently executed programs using export.
3. For each provided keyID download the key itself from the ha.pool.sks-keyservers.net key server. This works because we have installed the dirmngr package.
```
gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys "$key"; \
```
  - --batch: use batch mode, ie. don't allow interactive messages.
  - --keyserver: set keyserver to use (this option is deprecated).
  - --recv-keys "$key": import the keys with the given keyIDs from a keyserver.
  ha.pool.sks-keyservers.net
  
  This is a high-availability subset of the pool that require all servers to be identified as a clustered setup (marked with a blue indicator for reverse proxy in the status pages)
  
  What is an SKS keyserver?
  
  SKS is an OpenPGP keyserver whose goal is to provide easy to deploy, decentralized, and highly reliable synchronization. That means that a key submitted to one SKS server will quickly be distributed to all key servers, and even wildly out-of-date servers, or servers that experience spotty connectivity, can fully synchronize with the rest of the system.
4. Verify the signature in the detached .asc signature file without user interaction:
```
gpg --batch --verify php.tar.xz.asc php.tar.xz; \
```
5. Kill all gpg components running as daemons and remove the GPG temporary key home.
```
gpgconf --kill all; \
rm -rf "$GNUPGHOME"; \
```
Purge all packages installed in this step (ie. remove and delete configuration files) along with all its dependencies.
- Mark all installed packages as automatically installed discarding any output:
```
apt-mark auto '.*' > /dev/null; \
```
- Mark as manually installed the same packages that were marked as manual at the beginning of this step (ie. not including gnupg, dirmngr and its installed dependencies) discarding any output:
```
apt-mark manual $savedAptMark > /dev/null; \
```
- Remove automatically installed packges that are no longer needed.
```
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
```
  - -y: automatic yes to prompts.
  - --auto-remove: removes unused dependency packages.
  - -o APT::AutoRemove::RecommendsImportant=false: sets this configuration option to false, effectively forcing the removal of recommended packages whose dependent packages are also being removed. See explanation.

Copy `docker-php-source` script

COPY docker-php-source /usr/local/bin/

Make available the docker-php-source script built for managing the php source tarball lifecycle inside the container.

Install compilation dependencies, configure build process and compile PHP

This is the last big RUN command and is responsible for building the PHP sources following the recommended procedure. These long RUN commands are common in Docker to minimize the number of layers. Let's review all the steps involved:

RUN set -eux; \

First of all, save the list of manually installed packages to clean everything else later:
```
savedAptMark="$(apt-mark showmanual)"; \
```
Install compilation dependencies using Apt. Packages previously defined in PHP_EXTRA_BUILD_DEPS are also installed. The typical apt-get update, apt-get install, rm cycle is used.

Export the previously defined PHP_* variables for the GCC compiler suite.

export \
    CFLAGS="$PHP_CFLAGS" \
    CPPFLAGS="$PHP_CPPFLAGS" \
    LDFLAGS="$PHP_LDFLAGS" \
; \

Extract PHP source tarball in /usr/src/php (default dir):
```
docker-php-source extract; \
```
Obtain package building architecture information using dpkg-architecture.
```
gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; \
debMultiarch="$(dpkg-architecture --query DEB_BUILD_MULTIARCH)"; \
```
From the Debian wiki:

Multiarch is the term being used to refer to the capability of a system to install and run applications of multiple different binary targets on the same system.

Multiarch also simplifies cross-building, where foreign-architecture libraries and headers are needed on a system during building.

From the Debian wiki:

BUILD is the machine we are building on

This is use to avoid the disparity between GNU_TYPE and MULTIARCH

DEB_BUILD_GNU_TYPE=i586-linux-gnu DEB_BUILD_MULTIARCH=i386-linux-gnu

Avoid a PHP compilation bug where curl development headers are not found.

if [ ! -d /usr/include/curl ]; then \
    ln -sT "/usr/include/$debMultiarch/curl" /usr/local/include/curl; \
fi; \

Execute ./configure to setup compilation using Autotools.
- define the architecture we want to build PHP for (ie. this system).
```
--build="$gnuArch"
```
- define PHP configuration paths:
```
--with-config-file-path="$PHP_INI_DIR" \
--with-config-file-scan-dir="$PHP_INI_DIR/conf.d" \
```
- enable some extensions which can't be easily (or at all) compiled after having built PHP: mhash, ftp, mbstring, mysqlnd, password-argon2, sodium, curl, libedit, openssl and zlib.
- use system SQLite:
```
--with-pdo-sqlite=/usr \
--with-sqlite3=/usr \
```
- don't use pcre-jit for s390x architecture:
```
$(test "$gnuArch" = 's390x-linux-gnu' && echo '--without-pcre-jit') \
```
- define lib directory:
```
--with-libdir="lib/$debMultiarch" \
```
- add previously defined configure args (apxs2 and disable cgi):
```
${PHP_EXTRA_CONFIGURE_ARGS:-} \
```
Build PHP using all system cores in parallel:
```
make -j "$(nproc)"; \
```
Delete all generated static libraries:
```
find -type f -name '*.a' -delete; \
```
Install generated files in default system location:
```
make install; \
```
Strip symbols from all generated binaries:
```
find /usr/local/bin /usr/local/sbin -type f -executable -exec strip --strip-all '{}' + || true; \
```
- The -exec command {} + variant builds the command line to run by appending the selected file names at the end. This is similar to xargs and reduces considerably the number of executions. Quoting {} is recommended when running inside a shell to avoid interpretation.
- Adding || true at the end stops the command from failing. This effectively shadows any error running strip because of binaries without symbols (not really an error).
Clean compilation intermediate files:
```
make clean; \
```
Copy the sample php.ini to the default PHP configuration folder:
```
cp -v php.ini-* "$PHP_INI_DIR/"; \
```
Delete extracted PHP source folder (/usr/src/php):
```
cd /; \
docker-php-source delete; \
```
Purge all build dependencies:
- Mark all installed packages as automatically installed discarding any output:
```
apt-mark auto '.*' > /dev/null; \
```
- Mark as manually installed the same packages that were marked as manual at the beginning of this step (ie. not including all compilation dependencies) discarding any output:
```
[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \
```
- Mark as manually installed the binary dependencies required for PHP runtime execution (ie. not for compilation). This will stop them from being uninstalled.
```
find /usr/local -type f -executable -exec ldd '{}' ';' \
    | awk '/=>/ { print $(NF-1) }' \
    | sort -u \
    | xargs -r dpkg-query --search \
    | cut -d: -f1 \
    | sort -u \
    | xargs -r apt-mark manual \
; \
```
  - Run ldd against all executable files in /usr/local to obtain their entire shared library dependency tree. Dependencies are printed in the format libname.so.X => /full/path/libname.so.X (hex number).
  - Obtain the full path of every library dependency using awk matching all lines containing => with /=>/ and printing the second last field (awk automatically splits each record in fields separated by whitespace).
  - For each library dependency path, obtain the package that installed it searching the dpkg database: xargs -r dpkg-query --search
  - Using -r in xargs avoids running the command if the input does not contain any nonblanks.
  - As previously seen, mark each package as manually installed to avoid purging it later: xargs -r apt-mark manual.
- Remove automatically installed packages that are no longer needed (see command explanation previously in this article):
```
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
```
Update pecl definitions:
```
pecl update-channels; \
rm -rf /tmp/pear ~/.pearrc; \
```
- PECL is "a repository for PHP Extensions, providing a directory of all known extensions and hosting facilities for downloading and development of PHP extensions."
- Remove any PEAR (PHP Extension and Application Repository) local configuration.
Test that php runs OK or fail image building otherwise:
```
php --version
```

Copy all other custom shell scripts inside the image

COPY docker-php-ext-* docker-php-entrypoint /usr/local/bin/

These scripts are located next to the Dockerfile in the git repository. Some of them are meant to be used by the image users, while others are internal.

Enable sodium PHP extension

RUN docker-php-ext-enable sodium

This is an example usage of the docker-php-ext-enable which is a very interesting shell script that handles enabling one or more PHP extensions.

freetype-config bug workaround

RUN { echo '#!/bin/sh'; echo 'exec pkg-config "$@" freetype2'; } > /usr/local/bin/freetype-config && chmod +x /usr/local/bin/freetype-config

As we have already seen, using {} to group several echo commands to write in a file is a common Dockerfile / sysadmin pattern.

Set image entry point

ENTRYPOINT ["docker-php-entrypoint"]

According to the documentation: "An ENTRYPOINT allows you to configure a container that will run as an executable".

The exec form is being used, which is the preferred one (versus the shell form: ENTRYPOINT command param1 param2), which makes the run process PID 1, effectively receiving Unix signals (e.g. a SIGTERM sent by a docker stop command).

The very simple script allows the user to run the apache2-foreground script (see next steps) with user-defined parameters or to just run any command passed to it (like php or any other executable in the image).

Define stop signal

STOPSIGNAL SIGWINCH

SIGWINCH is the recommended signal to use for gracefully stopping Apache.

Copy apache2-foreground script

COPY apache2-foreground /usr/local/bin/

This script handles the setup required to launch Apache in the foreground. All params received will be passed to the main apache2 process in the last line of the script:

exec apache2 -DFOREGROUND "$@"

Using exec avoids undesirable resident shell processes, as would be the case if using Apache's apache2ctl script.

Set the working directory

WORKDIR /var/www/html

This affects only Dockerfile instructions following this line, in this case, it will affect the CMD final instruction.

Expose port 80

EXPOSE 80

Inform Docker that this container will listen to port 80 at runtime (by default, a TCP port). It's important to note that this instruction does not publish the port, it's seen as a kind of documentation between the image creators and the person running the image.

Port publishing is done on docker run using the -p or -P flags.

Specify the command to use in an executing container

CMD ["apache2-foreground"]

Because both the CMD and ENTRYPOINT instructions are specified with the exec form, this instruction is defining default parameters to ENTRYPOINT, which fits nicely with the docker-php-entrypoint script.

FOSS zombies

2019-10-30T00:00:00+01:00

First and foremost, FOSS projects are usually maintained by volunteers that donate their own time to the community. We can only be thankful for that. Nevertheless, it's our right to find ways to devote our time where it will make more impact (whatever that entails for each of us).

Now imagine you've been contributing for some months to a FOSS project. You've learned its internals to a large extent and your change proposals have been quite consistently merged.

Someday, you realize that your patches take more and more time to receive attention; what used to take days, now takes weeks. You start feeling a bit frustrated. You have so many ideas to improve the software, but at this pace, your code will hit master within months if you're lucky.

At this very time, you start noticing things in the project that you didn't pay much attention in the beginning, but which can explain crystal clear why you've got involved in what I'm calling a "FOSS zombie".

Let's review some signals and tips to avoid falling in the trap.

Zombie signs

Inactive or dismantled organization

The organization behind the project is not active anymore.

What to look for?

Is the organization owning the project still alive and kicking?
Do some research on the main contributors. Do they used to work for/be part of the organization owning the project, but they are not anymore?

External contributors are the only contributors

Most releases are minor/patch versions proposed by external contributors and new features are rare.

What to look for?

Review the changelog and latest commits; are most of them from external authors? What type of changes contain?
Do maintainers write new code? Do they only merge other people's code?
Try to find plans for new features. Roadmap anywhere?

Maintainers accumulate all responsibilities

Maintainers still have all responsibilities even if they haven't made any significant contribution for months/years.

Nevertheless, they have not vanished; they still provide feedback, merge contributions and release versions.

What to look for?

Find out what level of involvement do the maintainers have in the project?
Are there external developers allowed to accept contributions in the project?
In general, is there any kind of responsibility delegation?

Initial boost

The project received much attention at its creation, but in the following years/months it was pretty much inactive.

What to look for?

Use your preferred tool to show git repo statistics and look for tendencies.
Watch for messages of the project being completed. Many projects will match this pattern but don't seek further development and this is OK.

Open issues and contributions

There are many open issues and contributions, which means ideas and developments that didn't go well for whatever reason.

What to look for?

Check the contribution open/close ratio. Compare it with other succesful projects.

Useless work and code

There are unmerged contributions which have received a lot of feedback and work, but that didn't work out well and now all that code is in the limbo.

What to look for?

Look for open contributions with a number of reviews/comments strangely above normal.
Look for open contributions opened months/years ago. Find root causes. Try to be critic and impartial.

Conclusions

It's important to distinguish between a "FOSS zombie" and a project which is simply done. If that's the case, you'll find out easily by reading the documentation.

Now that we can identify a "FOSS zombie" and tell the difference from an abandoned/unmaintained one, let's realize that we are not obliged to contribute to it; there are always alternatives:

Find a similar project with a more active community.
Talk to the maintainers in case they are willing to support a fork or hand-off.
Create a fork and start improving and using it.

In my opinion, "FOSS zombies" lead to confusion to potential contributors. If the maintainer(s) don't want to work anymore in the project, which is perfectly fine, they'd be better off letting go of it. Otherwise, contributors might get quickly burned out and the project will miss opportunities to grow and improve.

Book review: How to Become an Expert Software Engineer

2019-08-12T00:00:00+02:00

Title: How to Become an Expert Software Engineer

Subtitle: A programmer's guide to the secret art of free and open source software development.

Author: Marcus Tomlinson

Review

The book is oriented towards developers just starting their careers or who are not familiar with open source and the typical tools. The author proposes open source as the main way to obtain experience for your résumé. It is divided into four parts: Define, Design, Develop, and Deliver.

The first part (Define) seeks to inspire the reader to work for the job they want, and also describes a systematic process to find what technologies and skills you should learn and practice.

The second part (Design) dives right into how to create an open source project that matches the technologies and subdomains that you identified in the first part. The author seems especially interested in creating a popular open-source project.

The third part (Develop) deals with solo project management, productivity, and motivation and explains some API design best practices. It's like a summary of multiple disciplines, but it only manages to point out the most obvious perils along the way.

The last part (Deliver) explains how to handle software releases and market your project. It is the one I found more irrelevant, as the initial goal (finding your career path) fades in the distance of the first chapters.

Summing up, the book is not long and can be read in a few hours. It touches many subjects, although very lightly. Nevertheless, it can be a good reference for new developers who are not aware of the multiple elements that are involved in software engineer projects. Some chapters might be motivating to a more experienced developer, even if you don't pick up many new things.

Key Insights

Proving your proficiency in the job you’re asking for is by far the most convincing argument you can make.
Hard work spent doing something worthless will always be worthless no matter how hard you work.
The trick is to find a job that will be a perfect fit for you, then get to work on making yourself perfectly fit for it.
Contribute good free software that is well-written and helpful to ourselves and to others.
Expertise is usually measured in the time you’ve spent deliberately practicing:
- 20 hours to learn almost anything and reach a decent level of proficiency.
- 10,000 hours to become an expert in almost anything.
Create a prioritized list of job requirements with technologies and subdomains in order to decide what to focus on.
Manage your project like you would manage a work project: do research, split tasks, define good APIs, implement quality tests, estimate tasks, etc.
Always consider research time when estimating tasks and rate it according to your relevant experience.
Plan your project to keep your motivation high and start with the minimum set of must-do features.
Try to focus exclusively on solving just the immediate problem at hand, be mindful of how long you’re spending on it, and postpone any divergent issues to separate tasks for later.
Put a few hobbies on hold for a while, work whenever you can find the time, and try to avoid taking breaks that are longer than a week, at least until you reach your project’s first release.
Motivation is key: All you need to create good software is a personal interest in seeing the problem solved.
Design and test your interfaces before implementing them.

Watson: Codebase review

2019-05-27T00:00:00+02:00

Project information

Python version support

At first, I had installed the Debian 10 packages (python3-watson and watson), but I wanted to review and use the latest version, preferably in a virtualenv, so I decided to get the latest code from the GitHub repository.

Just by looking at the Debian package names you can notice that Watson supports Python 3, but does it also support Python 2?

Let's take a look at the tox.ini file which is used for running tests in multiple python environments:

[tox]
envlist = flake8,py27,py34,py35,py36,py37

We can see that Watson effectively supports python 2.7 which is pretty typical these days.

Dependencies

Runtime

Being packaged in Debian, this means we can just look up for its runtime dependencies in a Debian system:

$ apt-cache depends watson python3-watson | awk '/Depends: [^<]/ {print $2}' | xargs dpkg -l | grep ii
ii  python3-arrow    0.12.1-2     all   Python3 library to manipulate dates, times, and timestamps
ii  python3-click    7.0-1        all   Wrapper around optparse for command line utilities - Python 3.x
ii  python3-requests 2.21.0-1     all   elegant and simple HTTP library for Python3, built for human beings
ii  python3-watson   1.6.0-6      all   Library for Watson (Python 3)

Development

Besides this high-level info, there are two requirements files in the root folder, but why? Let's grep around and see if we can get more info:

# some results are not shown as seemed irrelevant to me
$ grep -r requirements *
Makefile:   $(PIP) install -r requirements-dev.txt
MANIFEST.in:include requirements-dev.txt
MANIFEST.in:include requirements.txt
setup.py:    install_requires=parse_requirements('requirements.txt'),
setup.py:    tests_require=parse_requirements('requirements-dev.txt'),
docs/contributing/hack.md:        $ pip install -r requirements-dev.txt

As for the requirements-dev.txt, we can see tools for running the tests, generating the documentation, and more:

flake8, which wraps pep8, pyflakes, mccabe and more, to check the quality and style of python code.
mkdocs, to build the docs
mock, for creating mocks for testing
py, a development support library, which is now deprecated.
pytest, a (simpler than unittest) testing framework, along with some extensions (datafiles, mock and runner)
tox, which aims to standardize testing in python
twine, a utility for publishing python packages on PyPI.

Folder organization

The file structure is as follows:

docs/: markdown documentation; some of it is automatically generated with make docs.
scripts/: auxiliary scripts.
tests/: tests, mainly unit tests.
watson/: source code.
Root dir:
- travis ci
- testing with tox
- setup config for package generation
- Makefile for automation
- shell completion scripts
- python package requirements
- license
- other docs

Code organization

$ cloc --by-file watson
--------------------------------------------------------------------------------
File                              blank        comment           code
--------------------------------------------------------------------------------
watson/cli.py                       237            445            848
watson/watson.py                    110             50            395
watson/fullmoon.py                    7              6            220
watson/utils.py                      53             63            168
watson/frames.py                     38              0            117
watson/config.py                     33             47             32
watson/__main__.py                    1              0              5
watson/__init__.py                    1              0              3
watson/version.py                     1              1              1
--------------------------------------------------------------------------------
SUM:                                481            612           1789
--------------------------------------------------------------------------------

$ cloc --by-file tests
------------------------------------------------------------------------------------
File                                  blank        comment           code
------------------------------------------------------------------------------------
tests/test_watson.py                    251             44            570
tests/test_utils.py                      56              4            170
tests/test_config.py                     24             47             82
tests/__init__.py                        15              1             28
tests/test_fullmoon.py                    6              1             19
tests/conftest.py                         6              1              8
------------------------------------------------------------------------------------
SUM:                                    358             98            877
------------------------------------------------------------------------------------

Testing

You can run all tests in all available Python environments in your system like this:

$ tox

Watson uses pytest along with unittest.mock to do unit testing.

Let's understand some testing constructs used in Watson:

mock.patch.object(): patch the named member (attribute) on an object (target) with a mock object.

# In tests/__init__.py
# path.object(target, attribute, new) with new being the new object to replace the target
return mock.patch.object(dt_module, 'datetime', MockedDateTime)

@pytest.mark.parametrize(): enables parametrization of arguments for a test function documentation.
Pytest fixtures: some functions receive pre-filled arguments, like watson, mock, or config_dir. These are fixtures used as function arguments.

$ pytest --fixtures
--------------------------------------- fixtures defined from pytest_datafiles ----------------------------------------
datafiles
    pytest fixture to define a 'tmpdir' containing files or directories
    specified with a 'datafiles' mark.

------------------------------------------ fixtures defined from pytest_mock ------------------------------------------
mocker
    return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.
mock
    Same as "mocker", but kept only for backward compatibility.

---------------------------------------- fixtures defined from tests.conftest -----------------------------------------
watson
    tests/conftest.py:14: no docstring available
config_dir
    tests/conftest.py:9: no docstring available

--------------------------------------- fixtures defined from tests.test_watson ---------------------------------------
json_mock
    tests/test_watson.py:31: no docstring available

Contributing

Migrating from TiddlyWiki to Markdown

2018-07-04T00:00:00+02:00

After many years of using TiddlyWiki to keep my notes and journal, both personal and work-related, I have decided to migrate to a more simple solution: markdown files.

What is TiddlyWiki?

TiddlyWiki 5 (aka, modern TW) was a significant step in its development. The same software supported two platforms: node.js and the browser. A single-file HTML TW could be generated using node.js allowing a myriad of customization options. The client-side part (i.e. HTML/JS) supports multiple savers, including the single-file itself or TW running as a backend in node.js. No doubt, TiddlyWiki is an unusual piece of code.

Solution design

My focus was on leveraging free and opensource tools to build a combination which led to a simple solution to convert tiddlers to markdown files. After reading the development documentation and evaluating several alternatives, including creating a new TW module, I decided to take advantage of TiddlyWiki’s available commands.

The following block diagram shows the steps, programs and intermediate files that make up the process.

Solution implementation

These are the steps involved in the whole migration process:

Setup empty TiddlyWiki5
Load tiddlers
Render to HTML
Rename tiddler files
Convert to Markdown

Setup empty TiddlyWiki5

We initialize an empty edition TiddlyWiki and add support for markdown and TiddlyWiki’s old format.

$(NODEJS) $(TIDDLYWIKI_JS) $(WIKI_NAME) --init empty
$(NODEJS) $(ADD_PLUGIN_JS) $(TIDDLYWIKI_INFO) tiddlywiki/tw2parser
$(NODEJS) $(ADD_PLUGIN_JS) $(TIDDLYWIKI_INFO) tiddlywiki/markdown

Load tiddlers

We can use the load command to load the tiddlers from several sources into a TiddlyWiki. One of these sources is a single-file html file (classic or modern):

--load wiki.html

Render to HTML

TW5 has a powerful render command that allows to render individual tiddlers, filter its contents and save the results in the filesystem. In our case, we render tiddler content and metadata in separate files to facilitate the conversion process.

--render [!is[system]] [encodeuricomponent[]addsuffix[.html]] \
--render [!is[system]] [encodeuricomponent[]addsuffix[.meta]] \
    text/plain $$:/core/templates/tiddler-metadata

[!is[system]] avoids rendering system tiddlers (99% of TW is built using its own tiddler concept).
encodeuricomponent[] encodes the tiddler name to avoid filesystem name issues.

Important: Loading and rendering has to be done in the same command-line execution to avoid having to persist tiddlers to disk.

Rename tiddler files

I want my markdown file names to be as much compatible as possible with any filesystem. This is implemented in this simple Node script.

// Remove accents/diacritics
.normalize('NFD').replace(/[\u0300-\u036f]/g, "")
// Make lowercase
.toLowerCase()
// Convert separators to low line
.replace(/\s+/g, '_')
// Remove any non-safe character
.replace(/[^0-9a-zA-Z-._]/g, "");

Convert to Markdown

Metadata is prefixed at beginning of the Markdown files using a format equivalent to YFM (YAML Front Matter). Tiddlers in HTML are converted to CommonMark using pandoc (the universal document converter). This variant of Markdown was the one that produced more simple results in my tests.

It’s interesting to see how the GNU Make utility is used to convert all files using one rule:

$(MARKDOWN_DIR)/%.md : $(TW_OUTPUT_DIR)/%.html
    @echo "Generating markdown file '$(@F)'..."
    @echo "---" > "$@"
    @cat "$(^:html=meta)" >> "$@"
    @echo "---" >> "$@"
    @$(PANDOC) -f html-native_divs-native_spans -t commonmark \
        --wrap=preserve -o - "$^" >> "$@"

As html files don’t exist until this final step, make execution must be done in two separate commands:

$ make # or 'make export-html'
$ make convert

Conclusions

I am quite happy with the final result. This little project took me three full days, including evaluating other design options and learning a bit about the internals of TiddlyWiki (worth a look!). It’s amazing how using free and opensource software allows you to put together powerful applications with a few lines of code.

david alfonso

Book review: Performance Testing

Review

Key Insights

Certbot webroot auth with Ansible

Apache configuration

Ansible orchestration

Discourse dev setup instructions

Redis

Miscelanea

Mailpit

Removing all my tweets from X/Twitter

The API is useless, long live the API

Create a new app in the Twitter Developer Portal

Delete 50 tweets per day

Running the script daily

E-mail privacy in GitHub

Future commits

Past commits

Last commit

Bulk edition

References

Updating my Neovim config from upstream

Bringing changes from upstream to main branch

Updating the submodule commit in my dotfiles repo

Conclusions

Using a Lua-based Neovim config

Choosing a base configuration

Learning some Lua

Creating my config from the template

Referencing Neovim's config from my dotfiles repository

Signing PDF documents

Debugging PHP in a Docker container

Build Docker container with xdebug support

Configure VS Code

Default resource values in the Python CDK

Method 1: Synthetizing

Method 2: Looking at the CDK library resource

Forking Watson: License

Why would anyone ever want to modify the license?

Switching from MIT to GPL

Enter REUSE

How to declare multiple licenses

Using the reuse tool to include licensing information

Licensing non-code files

Conclusions

POSIX Shell Scripting

Resources

Shells

Tooling

Examples

Forking Watson: Motivations

What is a fork?

Why forking?

The case of Watson

Khal: Codebase review

Project information

Documentation

High-level analysis

Concepts

Dependencies

Package organization

Entry points

Configuration and data files

Architecture

Object-oriented design

Python techniques

PHP 7.2 Docker image analysis

Base image

RUN instructions

Block official PHP Debian packages

Installing PHP compilation dependencies

Setup PHP / Apache directories

Install Apache 2

Enable preforking MPM

Make PHP handle PHP file requests

Establish PHP compilation options and dependencies

Establish PHP version and validation mechanisms

Download PHP source code and verify its integrity

Copy docker-php-source script

Using the `reuse` tool to include licensing information

Copy `docker-php-source` script