Development

Architecting the Skpr Configuration System

By Nick Schuch24 February 2021

Skpr provides development teams with a robust system for configuring their applications.

This blog post covers the considerations that went into the design and development of that system.

What Is Configuration?

Configuration can be a loaded term for infrastructure, we define configuration as the values used to modify the behaviour of an application.

These behaviours are typical:

  • Connecting to services eg. database, search, external api etc
  • Flags to enable/disable features
  • Provide context to an application for where it is running. eg. dev/stg/prod

Existing Patterns

Before building our configuration system we evaluated the cloud hosting ecosystem as a whole to find existing patterns.

In this section, we cover two common patterns we've observed.

1) Include Files

The hosting platform provides a file that is language/application type sensitive that can be included in the application's codebase eg. Acquia provides a settings.php which can be included by a Drupal application.

This include file only configures the bare minimum to avoid conflicts with the application. This includes:

  • Database connection
  • Filesystem locations

If you have deployed an application on Acquia or Pantheon than you have most likely had to use this pattern before.

# Inheriting the platform's configuration.
require '/var/www/my-application/settings.php';

# Setting my own custom configuration.
$settings['googleanalytics_account'] = 'UA-XXXXXXXX-X';

The example above demonstrates how this pattern can lead to a simple configuration solution. The downside is that you need to manage any custom configuration yourself.

The typical workaround we have seen is for teams to hardcode these configurations in the application and commit them to the code base.

This example from Pantheon's documentation demonstrates how a development team might set a custom configuration.

if (PANTHEON_ENVIRONMENT == 'dev') {
  // Google Analytics.
  $conf['googleanalytics_account'] = 'UA-XXXXXXXX-X';
}
else if (PANTHEON_ENVIRONMENT == 'test') {
  // Google Analytics.
  $conf['googleanalytics_account'] = 'UA-XXXXXXXX-Y';
}
else if (PANTHEON_ENVIRONMENT == 'live') {
  // Google Analytics.
  $conf['googleanalytics_account'] = 'UA-XXXXXXXX-Z';
}

This leads to security and development workflow issues as:

  • Credentials are now committed to the codebase and accessible on developer workstations, code hosting services, CI/CD pipelines, etc.
  • Configuration changes need to be deployed eg. an API Token update

In summary, this approach works great until custom configuration is required. Development teams then need to make compromises on security and workflow to have a custom configuration set.

2) Environment Variables

Environment variables are a popular approach for configuring applications and are supported by a wide range of hosting providers. They are also very easy to integrate into an application's codebase.

For example:

$conf['googleanalytics_account'] = getenv('GOOGLEANALYTICS_ACCOUNT');

Acquia, Lagoon and Pantheon all have support for environment variables. However, they all have varying workflows to support them:

  • Acquia provides a web interface
  • Lagoon and Pantheon provide this functionality via lower level tooling eg. command-line interface, API interface and codebase "commit" options

While environment variables are convenient, they do come with the downside that the application must be restarted or redeployed for an update to take effect. While this should not lead to downtime, it could under certain conditions.

A potential condition would be setting the configuration 10 times requiring 10 restarts or redeployments on the platform. While this shouldn't be a problem with rolling updates and robust traffic routing configurations, we were still concerned that this could happen.

One method to combat the restart/redeploy downside is to set the configuration (via the web, command line or API interface) and wait for the next deployment for the configuration update to take effect. While this will solve the issue, it creates a new one - predictability.

We found that coupling configuration updates to deployments led to confusion eg.

  • Is the configuration already applied?
  • Are we still waiting for a deployment?
  • Are there other configuration values which have been set that are waiting for a deployment?

With all of the above in mind, we ultimately chose not to use environment variables for our configuration system.

Developer Experience Matters

While all the cloud providers mentioned in this blog post provide a solution, we felt they lacked a great developer experience.

Developers deserve a simple set of tools to interact with, and certainty when setting their configuration.

Skpr's Solution

Taking the above patterns into consideration, we have built a system with a focus on:

  • Avoiding application restarts/redeployments to apply configuration
  • Improved Developer Experience

Developer Experience

Skpr provides a simple command-line interface for developers to set, list and delete the configuration.

Setting Configuration

Configuration can be created or updated using the set command:

# Setting a non-secret config.
$ skpr config set prod api.username nickschuch

# Setting a secret.
$ skpr config set prod --secret api.password xxxyyyzzz

Listing Configuration

Developers can view which configuration is set for each environment, this example demonstrates which config is set for production.

$ skpr config list prod
 ───────────────────────── ───────────────────
  KEY (26)                  VALUE
 ───────────────────────── ───────────────────
  api.username              nickschuch
  api.password              [secret]

Delete Configuration

Configuration can also be deleted using the delete command:

$ skpr config delete prod api.username
$ skpr config delete prod api.password 

How Do We Avoid Restarts?

Skpr provides applications with a JSON file containing all the configuration provided by both the platform and the developer.

We chose a JSON file because:

  • It's language agnostic
  • Can be automatically updated without restarts/redeployments

Below is an example of the JSON file which can be loaded by an application.

$ cat /etc/skpr/data/config.json | jq
{
  "api.username": "nickschuch",
  "api.password": "xxxyyyzzz"
}

The file is typically much larger given it contains, by default, connection details for the following services:

  • MySQL
  • Solr/Elasticsearch
  • ClamAV
  • SMTP

The Skpr platform team maintain a set of packages (PHP, Node, Ruby and Go) to smooth out the integration between the Skpr configuration system and the application.

Integrating It Into Your Site

Below is an example using the PHP package with a Drupal application.

use Skpr\SkprConfig;

$skpr = SkprConfig::create()->load();

# Simple example.
$conf['api_username'] = $skpr->get('api.username');
$conf['api_password'] = $skpr->get('api.password');

# Advanced example (with fallback).
$databases['default']['default'] = [
  'driver'   => 'mysql',
  'database' => $skpr->get('mysql.default.database') ?: 'local',
  'username' => $skpr->get('mysql.default.username') ?: 'local',
  'password' => $skpr->get('mysql.default.password') ?: 'local',
  'host'     => $skpr->get('mysql.default.hostname') ?: '127.0.0.1',
  'port'     => $skpr->get('mysql.default.port') ?: '3306',
];

Extending This Solution

This solution isn't just for setting API keys etc. It can also be used to provide additional context about the environment to which the application is deployed.

Hostnames

Skpr also exposes a list of hostnames as a JSON file.

$ cat /etc/skpr/data/hostnames.json | jq
[
  "dev.example.cluster.skpr.dev"
]

This allows for applications like Drupal 8+ to automatically configure its trusted_host_patterns setting based on the environment.

foreach ($skpr->hostNames() as $hostname) {
  $settings['trusted_host_patterns'][] = '^' . preg_quote($hostname) . '$';
}

IP Ranges

Environments also have access to a list of internal IP ranges which can be useful when determining external addresses eg. the client IP address.

The following example demonstrates how easy it is to integrate with Drupal.

$settings['reverse_proxy_addresses'] = $skpr->ipRanges();

Environment Variables

The PHP library also transforms Skpr configuration into environment variables for each request using putenv.

This allows for a frictionless experience when migrating applications with environment variables configuration to the Skpr platform.

# Using the library.
$conf['api_username'] = $skpr->get('api.username');
$conf['api_password'] = $skpr->get('api.password');

# Using environment variables.
$conf['api_username'] = getenv('API_USERNAME');
$conf['api_password'] = getenv('API_PASSWORD');

Resources

The following resources were used during the writing of this blog post.

Tags

config
developer experience

Getting Started

Interested in a demo?

🎉 Awesome!

Please check your inbox for a confirmation email. It might take a minute or so.

🤔 Whoops!

Something went wrong. Check that you have entered a valid email and try submitting the form again.

We'll be in touch shortly.