Table of Contents

Configuration

Every server application needs settings that are not in the code: the URL it listens on, the databases it connects to, a password. Unit Bcl.Configuration reads them from JSON files and environment variables, merges them into one structure, and gives them back as typed values - or fills an object with them. A value that is missing or wrong is reported with a message that says which key it is and where it came from.

  BaseUrl := TConfiguration.Current.RequireString('BaseUrl');
  Port := TConfiguration.Current.GetInteger('Catalog.Port', 5432);

That is all an application named appserver needs in order to read the following file, placed beside its executable, with any of its values overridden by an environment variable. The examples in this chapter use it:

{
  "BaseUrl": "http://localhost:2001",
  "AllowedOrigins": ["https://app.example.com"],
  "Catalog": {
    "Hostname": "db.internal",
    "Port": 5432,
    "Database": "catalog",
    "User": "app",
    "Password": "...",
    "PoolSize": 3
  },
  "Shards": [
    {
      "Name": "shard-0",
      "Connection": { "Hostname": "shard0.internal", "Database": "shard_0", "User": "app", "Password": "..." }
    }
  ]
}

What It Is For

  • One executable for every environment. The file holds the settings that are common to all of them, and each deployment overrides only what differs - a host name, a port - with environment variables. Nothing is rebuilt and no file is edited to go from development to test to production.
  • Containers and cloud platforms. Docker, Kubernetes, systemd and most hosting platforms supply settings and secrets as environment variables. An application that reads its configuration this way runs there with no extra code.
  • Secrets out of the file. A password does not have to be in a file that ends up in version control or in a backup. It comes from an environment variable, or from a vault or parameter store that the application reads at startup.
  • No configuration code to write. There is no INI or JSON reading code to maintain, and no global variables to carry the values around: any unit reads the setting it needs, or the application loads all of them into a class with defaults and validation rules.
  • Mistakes found at startup. A setting that is missing, misspelled or out of range stops the application from starting, with a message that names the setting and the file or variable it came from, instead of causing a failure hours later. The message never contains the value, so passwords stay out of the logs.
  • Knowing where a value came from. When a server behaves differently from what its file says, one call lists every key and its source, which shows the environment variable that overrode it.

The Application Configuration

TConfiguration.​Current is the configuration of the application: one instance, shared by the whole process and reachable from anywhere - a form, a data module, a service implementation, a background thread.

It needs no setup. Nothing exists until it is first read. That first read loads the conventional sources of the application, named after its executable. For appserver:

  1. The file appserver.json beside the executable, if it is there - or the file named by the environment variable APPSERVER_CONFIG, when that is set.
  2. The environment variables prefixed with APPSERVER_, which override the file.

Every later read returns the same configuration, and reading it is safe from any thread.

An application with no file and no variables simply gets an empty configuration. What raises is a configuration that is there and is broken: a malformed file, or an APPSERVER_CONFIG that names a file that does not exist. The exception comes from the read that tried to load it, so read the configuration once while the application starts: such a failure then stops it from starting, instead of showing up later.

Using other sources

An application that wants different sources builds its own configuration and assigns it, at startup, before anything reads it. The conventional sources are then never loaded:

  Config := TConfiguration.Create;
  Config.AddJsonFile('settings.json');
  Config.AddEnvironmentVariables('MYAPP_');
  TConfiguration.Current := Config;

A configuration of your own

Nothing requires the shared one. A configuration is created with TConfiguration and held in an IConfigurationRoot variable. It is reference counted, so there is nothing to free. Code that only reads it receives an IConfiguration, which has no way of changing it:

procedure QuickStart;
var
  Config: IConfigurationRoot;
  BaseUrl: string;
  Port: Integer;
begin
  Config := TConfiguration.Create;
  Config.AddJsonFile('appserver.json');
  Config.AddEnvironmentVariables('APPSERVER_');

  BaseUrl := Config.RequireString('BaseUrl');
  Port := Config.GetInteger('Catalog.Port', 5432);
  StartServer(BaseUrl, Port);
end;

Everything in the rest of this chapter applies to both: TConfiguration.​Current is an IConfiguration like any other.

Sources

A configuration created with TConfiguration starts empty. Nothing is searched for or loaded on its own: it holds what the application adds to it, in the order it is added.

JSON files

IConfiguration​Root.​Add​Json​File adds the content of a file:

  // Beside the executable, and it must be there.
  Config.AddJsonFile('appserver.json');

  // Skipped when it does not exist.
  Config.AddJsonFile('appserver.local.json', True);

  // A full path is used as it is.
  Config.AddJsonFile('/etc/appserver/shards.json');
  • A relative name is resolved against the directory of the executable, not the current directory, which is not where the application is when it runs as a Windows service or under systemd.
  • A file that does not exist is an error, unless the second parameter says it is optional. A file that exists and is malformed is always an error.
  • The file must be encoded in UTF-8 and hold a JSON object.

Environment variables

IConfiguration​Root.​Add​Environment​Variables adds the variables whose name starts with the given prefix:

  Config.AddJsonFile('appserver.json');
  Config.AddEnvironmentVariables('APPSERVER_');

The prefix is removed from the name, and two underscores separate levels:

Environment variable Key
APPSERVER_BASEURL BaseUrl
APPSERVER_CATALOG__PORT Catalog.Port
APPSERVER_CATALOG__POOL_SIZE Catalog.Pool_Size

A single underscore is part of the key, and letter case does not matter. The value of a variable is a string until it is read as something else: APPSERVER_CATALOG__PORT=6543 reads as the integer 6543 and as the string 6543.

A list is given as a JSON array, which replaces the list that came from the file:

APPSERVER_ALLOWEDORIGINS=["https://app.example.com", "https://admin.example.com"]

Two variables that would set the same key - names that differ only in letter case, or APPSERVER_CATALOG together with APPSERVER_CATALOG__PORT - are rejected, instead of letting the outcome depend on the order the system lists them in.

Values set by the application

IConfiguration​Root.​Set​Value sets one value. It is how a secret that is kept somewhere else - a vault, a parameter store - gets into the configuration:

  Config.AddJsonFile('appserver.json');

  Config.SetValue('Catalog.Password', ReadSecret('catalog'), 'the secret store');

  // What was loaded so far can be read to decide what to set.
  for Shard in Config.GetSection('Shards').GetChildren do
    Config.SetValue(Shard.Path + '.Connection.Password',
      ReadSecret(Shard.RequireString('Name')), 'the secret store');

Levels that do not exist are created. An index, as in Shards[0].Connection.Password, addresses an item of an array that is already there. The optional third parameter says how error messages refer to the origin of the value.

JSON text

IConfiguration​Root.​Add​Json adds JSON that the application already holds in memory. Added first, it provides defaults that every other source can override:

  Config.AddJson('{ "Catalog": { "Port": 5432, "PoolSize": 3 } }', 'built-in defaults');
  Config.AddJsonFile('appserver.json');

The conventional sources

Most applications want the same two sources: a JSON file whose location can be changed at deployment, then environment variables on top of it. IConfiguration​Root.​Add​Defaults adds both:

  Config := TConfiguration.Create;
  Config.AddDefaults('appserver');

For an application named appserver:

  1. If the environment variable APPSERVER_CONFIG is set, the file it names is loaded, and it must exist. Otherwise appserver.json beside the executable is loaded if it is there.
  2. The environment variables prefixed with APPSERVER_ are added.

Called with no name, it uses the name of the executable, which is what TConfiguration.​Current does when it loads itself. TConfiguration.​Default​File​Name returns the file that would be loaded, for an application that wants that convention with sources of its own.

How Sources Are Merged

A source added later overrides an earlier one, key by key:

  • Objects are merged member by member. Setting Catalog.Hostname in a second source leaves Catalog.Port as the first source defined it.
  • Arrays are replaced whole. A list in a later source takes the place of the earlier list; it is never appended to it or merged item by item. An empty array clears the list.
  • Anything else replaces what was there. A string or an array that takes the place of an object discards everything the object held.

This is what lets a deployment split its settings over several files, or keep a full file and override two values of it in the environment. Each value remembers the source it came from.

Loading a source is all or nothing: if it fails, the configuration stays as it was.

Reading Values

There is a pair of methods for each type. The one starting with Get takes a default, returned when the key is absent. The one starting with Require raises EConfiguration​Exception when the key is absent:

  // Must be there: an absent key raises.
  BaseUrl := Config.RequireString('BaseUrl');

  // May be absent: the default is used then, and only then.
  Port := Config.GetInteger('Catalog.Port', 5432);
  UseTls := Config.GetBoolean('Catalog.UseTls', False);

The default is only for a key that is not there. A key that is present with a value of the wrong type is always an error, never a silent fallback to the default.

Methods Accepts
IConfiguration.​Get​String, IConfiguration.​Require​String A JSON string. It is returned as it is, never trimmed. RequireString also rejects an empty string.
IConfiguration.​Get​Integer, IConfiguration.​Require​Integer, IConfiguration.​Get​Int64, IConfiguration.​Require​Int64 A JSON number without a fractional part, or a string holding one. A value out of range is an error.
IConfiguration.​Get​Boolean, IConfiguration.​Require​Boolean A JSON boolean, or the strings true and false in any letter case.
IConfiguration.​Get​Strings, IConfiguration.​Require​Strings A JSON array of strings. GetStrings returns an empty list for an absent key; RequireStrings accepts a list that is present and empty.

Two rules hold for all of them:

  • A JSON null is an error. It does not mean that the key is absent, and it does not remove a value set by an earlier source.
  • A number or a boolean is not converted to a string: "Password": 123456 has to be written with quotes.

Reading a list:

  for Origin in Config.GetStrings('AllowedOrigins') do
    AllowOrigin(Origin);

Paths

A path uses dots to separate levels, as in Catalog.Port, and brackets to address an item of an array, as in Shards[0].Connection.Hostname. Key lookup is not case-sensitive. For that reason a JSON file with two properties that differ only in letter case is rejected, and so is a property name that contains a dot, a bracket or two consecutive underscores.

Sections

IConfiguration.​Get​Section returns a part of the configuration as an IConfiguration of its own, with paths relative to it. IConfiguration.​Get​Children returns the members of an object, in the order they were defined, or the items of an array:

  Catalog := Config.GetSection('Catalog');
  CreatePool('catalog',
    Catalog.RequireString('Hostname'),
    Catalog.GetInteger('Port', 5432),
    Catalog.GetInteger('PoolSize', 3));

  for Shard in Config.RequireSection('Shards').GetChildren do
  begin
    Connection := Shard.RequireSection('Connection');
    CreatePool(Shard.RequireString('Name'),
      Connection.RequireString('Hostname'),
      Connection.GetInteger('Port', 5432),
      Connection.GetInteger('PoolSize', 3));
  end;

GetSection never returns nil. For a key that is not present, IConfiguration.​Exists of the result is False, it has no children, and every Get method returns its default. IConfiguration.​Require​Section raises instead, and also when the key holds a single value where an object or an array was expected. IConfiguration.​IsObject and IConfiguration.​IsArray tell the two apart.

A section keeps the configuration it belongs to alive: it remains valid for as long as it is referenced, even after the variable holding the root is gone.

Binding to Objects

An application with more than a handful of settings is better served by a class than by dozens of reads. Declare the settings as fields:

  TConnectionOptions = class
  private
    [Required]
    FHostname: string;
    [Range(1, 65535)]
    FPort: Integer;
    [Required]
    FDatabase: string;
    FUser: string;
    FPassword: string;
    FPoolSize: Integer;
  public
    constructor Create;
    property Hostname: string read FHostname;
    property Port: Integer read FPort;
    property Database: string read FDatabase;
    property User: string read FUser;
    property Password: string read FPassword;
    property PoolSize: Integer read FPoolSize;
  end;

  TShardOptions = class
  private
    [Required]
    FName: string;
    [Required]
    FConnection: TConnectionOptions;
  public
    destructor Destroy; override;
    property Name: string read FName;
    property Connection: TConnectionOptions read FConnection;
  end;

  TServerOptions = class
  private
    [Required]
    FBaseUrl: string;
    [Required]
    FCatalog: TConnectionOptions;
    [MinItemCount(1)]
    FShards: TObjectList<TShardOptions>;
  public
    constructor Create;
    destructor Destroy; override;
    property BaseUrl: string read FBaseUrl;
    property Catalog: TConnectionOptions read FCatalog;
    property Shards: TObjectList<TShardOptions> read FShards;
  end;

and fill an object with TConfigurationBinder, in unit Bcl.​Configuration.​Binder:

  Options := TConfigurationBinder.Bind<TServerOptions>(Config);
  try
    UseOptions(Options);
  finally
    Options.Free;
  end;

IConfiguration.Bind does the same for an object that already exists, and any section can be bound:

  Catalog := TConnectionOptions.Create;
  try
    Config.GetSection('Catalog').Bind(Catalog);
    CreatePool('catalog', Catalog.Hostname, Catalog.Port, Catalog.PoolSize);
  finally
    Catalog.Free;
  end;

Which members are bound

The members are found the way the JSON framework finds them, so a class that serializes to JSON binds unchanged: every field is bound, under its name without the leading F, and so are the properties marked with [JsonProperty]. [JsonProperty('pool-size')] gives a member a different key, and [JsonIgnore] leaves a field out. Keys are matched without regard to letter case.

A member can be:

  • A string, an integer or a boolean, converted by the rules of the typed reads. A value that came from an environment variable is a string, and still fills an Integer or a Boolean field.
  • Any other type the JSON framework converts: an enumeration (by name), a floating point number, a date, a TGuid, a Nullable.
  • An object, bound the same way.
  • A TList<T> or TObjectList<T> of any of the above.
  • A dynamic array of anything but objects. Use TObjectList<T> for a list of objects, so that something owns them.
Note

Declare the class of the items of a list in the interface section of a unit. It is located by its name at run time, and a class declared in an implementation section, or in the program file, cannot be found that way.

Defaults and required members

A member whose key is absent keeps the value it has. The constructor of the class is therefore where defaults go:

constructor TConnectionOptions.Create;
begin
  inherited Create;
  FPort := 5432;
  FPoolSize := 3;
end;

A member marked [Required] must have its key present, and a required string must not be empty.

Ownership

An object member that the instance already holds is filled in place. One that is nil is created, and the class that declares it is responsible for freeing it:

constructor TServerOptions.Create;
begin
  inherited Create;
  FShards := TObjectList<TShardOptions>.Create(True);
end;

destructor TServerOptions.Destroy;
begin
  FShards.Free;
  FCatalog.Free;
  inherited;
end;

destructor TShardOptions.Destroy;
begin
  FConnection.Free;
  inherited;
end;

A list is cleared before it is filled: like every array, it is replaced whole.

Validation

The attributes of unit Bcl.Validation.Attributes - [Range], [MinLength], [MaxLength], [RegularExpression], [MinItemCount] and the others - are checked against the values the object ends up with, defaults included. A method marked [OnValidate] checks what no single member can, such as one value against another.

Unknown keys

A key that matches no member is ignored. Pass TUnknownMember​Handling.Error to turn it into an error instead, which catches a misspelled key:

  Options := TConfigurationBinder.Bind<TServerOptions>(Config,
    TUnknownMemberHandling.Error);

Errors

Every problem raises EConfiguration​Exception. The message states the key, the type that was expected, and the source of the value:

Configuration key "Catalog.Port" from environment variable "APPSERVER_CATALOG__PORT" must be an integer.

It never contains the value, which may be a password. For the same reason nothing in this unit writes values to a log.

Reading by path stops at the first problem. Binding does not: it goes through the whole object and reports everything it found in a single exception, so that a deployment with three mistakes is fixed in one round rather than three:

The configuration has 3 errors:
  Configuration key "Catalog.Port" from environment variable "APPSERVER_CATALOG__PORT" must be an integer.
  Configuration key "Shards[0].Connection.Hostname" is required but was not found.
  Field "Shards[1].Connection.Port" from file "/etc/appserver/appserver.json" must be between 1 and 65535

Load and bind the configuration before the application starts doing its work, so that such a failure stops it from starting:

  try
    Config := TConfiguration.Create;
    Config.AddDefaults('appserver');
    Options := TConfigurationBinder.Bind<TServerOptions>(Config);
  except
    on E: EConfigurationException do
    begin
      Writeln(ErrOutput, E.Message);
      Halt(1);
    end;
  end;

Listing Keys and Their Sources

IConfiguration.​Describe returns one line for each key, with the source its value came from and without the value. Written to the log at startup, it answers the question of why a setting has the value it has:

  for Line in Config.Describe do
    Writeln(Line);

BaseUrl: file "/etc/appserver/appserver.json"
Catalog.Hostname: file "/etc/appserver/appserver.json"
Catalog.Port: environment variable "APPSERVER_CATALOG__PORT"
Catalog.Password: the secret store

Lifetime and Threads

A configuration is a snapshot. Files are not watched and the environment is not read again: what was loaded is what is there until the process ends.

Add every source first, then hand the configuration to the code that reads it. Reading is safe from any number of threads at once. Adding sources is not, and must be finished before the configuration is shared. Reading and assigning TConfiguration.​Current are thread safe, the first read included.

Configuration in TMS Sparkle

A server hosted by TMS Sparkle reads TConfiguration.​Current like any other application. What the host adds is the right moment to build one from sources of your own, so that a failure is reported the way every other startup failure is. See Configuration in the Hosting chapter.