Table of Contents

Multi-Tenancy

A multi-tenant application serves several customers (tenants) from the same code base while keeping the data of each tenant separated from the others. TMS Aurelius provides two complementary building blocks for that:

  • Global filters isolate the rows of each tenant inside a database that is shared by several tenants.
  • The tenant catalog (units Aurelius.Tenancy.*) maps each tenant to the database that hosts its data, pools connections per database, caches the routing information, enforces tenant status and provisions new tenants.

Both are independent from HTTP. TMS XData integrates them into the request pipeline through design-time components: see the Multi-Tenancy chapter in the TMS XData documentation. This chapter documents the Aurelius layer, which you can use directly in any application, including background workers and provisioning tools.

Topologies

There are three common ways of organizing tenant data:

Topology Description Aurelius features involved
Single database All tenants share one database. Every row carries a tenant id column. Global filters. No catalog needed.
Database per tenant Each tenant has its own database. Tenant catalog (one tenant per shard). Filters are optional.
Sharded (hybrid) Tenants are distributed over a few databases (shards). Several tenants share each shard. Tenant catalog (tenant to shard routing) plus global filters inside the shard.

The tenant catalog treats the sharded topology as the general case: a shard is a database that hosts one or more tenants, and the catalog maps every tenant to exactly one shard. Database-per-tenant is simply a configuration where every shard hosts a single tenant, and single-database is a configuration with one shard and no catalog at all.

Core concepts

  • Tenant: a customer of the application, identified by a string id (the TTenantInfo.TenantId). The id is the value used everywhere: in the catalog, in HTTP headers and claims, and in the global filter parameter.
  • Shard: a database hosting tenant data, identified by a logical string id (TShardInfo.ShardId). The catalog stores only this logical id. Connection settings and credentials for the shard stay in the application configuration and are never written to the catalog.
  • Catalog: the store of tenants, shards and identifiers (ITenantCatalog).
  • Status: every tenant has a TTenantStatus (Pending, Active, Suspended, Disabled) and every shard has a TShardStatus (Active, Draining, Disabled). Only active tenants on non-disabled shards are served. Draining shards keep serving their tenants but do not receive new ones.
  • Identifier: an optional external key that maps to a tenant, such as an API client id, a custom domain or a user id. Identifiers allow resolving the tenant from data that is not the tenant id itself.

The records TTenantInfo and TShardInfo are the snapshots exchanged with the catalog. Lookups return an empty record (check TTenantInfo.IsEmpty) when the item is unknown.

Isolating tenants inside a shard

When several tenants share a shard, the rows of each tenant are isolated by an Aurelius global filter. Declare the filter once in your model and apply it to every entity that holds tenant data. The convention used throughout the tenancy features is a filter named Multitenant with a parameter named tenant_id:

uses
  TypInfo, Aurelius.Mapping.Attributes;

type
  [Entity, Automapping]
  [FilterDef('Multitenant', '{TenantId} = :tenant_id')]
  [FilterDefParam('Multitenant', 'tenant_id', TypeInfo(string))]
  [Filter('Multitenant')]
  TProduct = class
  strict private
    FId: Integer;
    FName: string;
    FTenantId: string;
  public
    property Id: Integer read FId write FId;
    property Name: string read FName write FName;
    property TenantId: string read FTenantId write FTenantId;
  end;

Activate a TFilterEnforcer for the model so writes are validated against the filter and new rows are stamped with the current tenant automatically:

uses
  Aurelius.Mapping.Explorer, Aurelius.Mapping.FilterEnforcer;

var
  Enforcer: TFilterEnforcer;

initialization
  Enforcer := TFilterEnforcer.Create('Multitenant', 'tenant_id', 'FTenantId');
  Enforcer.AutoComplyOnInsert := True;
  Enforcer.AutoComplyOnUpdate := True;
  Enforcer.Activate(TMappingExplorer.Default);
finalization
  Enforcer.Free;

Every TObjectManager that touches tenant data must then enable the filter with the current tenant id:

Manager.EnableFilter('Multitenant').SetParam('tenant_id', TenantId);

See the Global Filters chapter for the details of filter definitions and the filter enforcer. In a database-per-tenant topology the filter is not needed, but it does no harm either.

The tenant catalog

The ITenantCatalog interface is the store of routing information. It has a read path (FindTenant, FindTenantByIdentifier, FindShard, GetShards) and a write path (SaveTenant, SaveShard, SaveIdentifier, RemoveIdentifier). Save methods are idempotent upserts.

Database-backed catalog

TDbTenantCatalog is the default implementation. It stores the catalog in a database, using the Aurelius entities declared in Aurelius.Tenancy.Catalog.Entities: tables tc_tenants, tc_shards and tc_identifiers, mapped in their own model named Biz.TenantCatalog (constant cTenantCatalogModelName). Because the entities belong to a separate model, they never mix with your application entities, and the catalog tables can live in a dedicated catalog database or in any existing database.

Create the catalog over an IDBConnectionPool that reaches the catalog database, and call TDbTenantCatalog.​Update​Schema once (typically at startup) to create or update the tables:

uses
  Aurelius.Drivers.Interfaces,
  Aurelius.Tenancy.Catalog,
  Aurelius.Tenancy.Catalog.Db,
  Aurelius.Tenancy.Types;

var
  DbCatalog: TDbTenantCatalog;
  Catalog: ITenantCatalog;
  Shard: TShardInfo;
  Tenant: TTenantInfo;
begin
  // CatalogPool is an IDBConnectionPool reaching the catalog database
  DbCatalog := TDbTenantCatalog.Create(CatalogPool);
  Catalog := DbCatalog;
  DbCatalog.UpdateSchema;

  // Register a shard
  Shard := TShardInfo.Empty;
  Shard.ShardId := 'shard-a';
  Shard.Status := TShardStatus.Active;
  Shard.Weight := 1;
  Catalog.SaveShard(Shard);

  // Register a tenant hosted by that shard
  Tenant := TTenantInfo.Empty;
  Tenant.TenantId := 'acme';
  Tenant.DisplayName := 'ACME Corp';
  Tenant.ShardId := 'shard-a';
  Tenant.Status := TTenantStatus.Active;
  Catalog.SaveTenant(Tenant);
end;

TDbTenantCatalog is reference counted; keep an ITenantCatalog reference to control its lifetime. Every catalog call takes a connection from the pool and uses its own object manager, so one catalog instance can be shared by all threads of the application.

Registering shards is optional: a tenant may point to a shard id that has no record in the catalog, in which case the shard is treated as active. Register shards when you want to control their status, use weighted allocation, or enumerate them (for example, to update the schema of every shard).

Custom catalogs

Any store can act as a catalog: a REST service, a configuration file, an in-memory list for tests. Descend from TCustomTenantCatalog and override at least TCustomTenant​Catalog.​Find​Tenant. The other lookup methods return "unknown" by default, and the write methods raise ENotSupportedException, which is the right behavior for a read-only client of a remote catalog:

uses
  Aurelius.Tenancy.Catalog,
  Aurelius.Tenancy.Types;

type
  TRemoteTenantCatalog = class(TCustomTenantCatalog)
  public
    function FindTenant(const TenantId: string): TTenantInfo; override;
  end;

function TRemoteTenantCatalog.FindTenant(const TenantId: string): TTenantInfo;
begin
  // Query your catalog service here and fill the record.
  // Return TTenantInfo.Empty when the tenant does not exist.
  Result := TTenantInfo.Empty;
end;

The routing cache described next makes remote lookups affordable: the catalog is only queried when an entry is missing or expired.

Routing cache

TTenantRoutingCache sits between the application and the catalog so the catalog is not hit on every request:

uses
  Aurelius.Tenancy.Cache;

var
  Cache: TTenantRoutingCache;
  Tenant: TTenantInfo;
begin
  Cache := TTenantRoutingCache.Create(Catalog);
  Cache.Ttl := 60000;         // cache entries for 60 seconds (default 30000)
  Cache.NegativeTtl := 5000;  // remember unknown tenants for 5 seconds (default)

  Tenant := Cache.GetTenant('acme');   // from cache, or from the catalog
  Cache.InvalidateTenant('acme');      // next lookup re-reads the catalog

  // Identifier lookups are cached too
  Tenant := Cache.FindTenantByIdentifier('user_id', '42');
end;

The cache is thread-safe, and catalog reads happen outside its locks, so a slow catalog never blocks lookups that hit the cache.

Note

In a deployment with several server processes, a change made by one process reaches the others when their cached entries expire. Keep the TTL short enough for your needs, or call the invalidation methods from each process (for example, through an administrative endpoint).

Shard pools and tenant routing

Connecting a tenant to its database is a chain of three classes, all in unit Aurelius.​Tenancy.​Provider:

  1. IShardPoolFactory creates the connection pool of a shard. This is the only piece the application must provide: given a TShardInfo, return an IDBConnectionPool to that database, reading connection settings and credentials from your configuration. TCallbackShard​Pool​Factory adapts an anonymous function to the interface.
  2. TShardPoolProvider keeps one pool per shard, created on first use through the factory. Pools can be removed at runtime with TShardPoolProvider.​Remove​Pool, for example after rotating credentials: connections already in use keep the old pool alive until they are released, and new requests get a fresh pool.
  3. TTenantPoolProvider maps a tenant id to the pool of its shard. On every call to TTenantPoolProvider.​Get​Pool it reads the tenant from the routing cache, checks that the tenant is active and that its shard is not disabled, and returns the shard pool. Because the status is checked on every call, a suspended tenant is blocked as soon as its cache entry is refreshed or invalidated, even when its connections are already pooled.
uses
  Aurelius.Drivers.Base,
  Aurelius.Drivers.Interfaces,
  Aurelius.Drivers.SQLite,
  Aurelius.Engine.ObjectManager,
  Aurelius.Tenancy.Provider,
  Aurelius.Tenancy.Types;

var
  Provider: TTenantPoolProvider;
  Manager: TObjectManager;
begin
  Provider := TTenantPoolProvider.Create(Catalog,
    function(const Shard: TShardInfo): IDBConnectionPool
    begin
      // Build the pool for the shard. Connection settings usually come
      // from your configuration, keyed by Shard.ShardId
      Result := TDBConnectionFactory.Create(
        function: IDBConnection
        begin
          Result := TSQLiteNativeConnectionAdapter.Create(Shard.ShardId + '.db');
        end);
    end);

  // Code that already knows the tenant asks for its pool explicitly
  Manager := TObjectManager.Create(Provider.GetPool('acme').GetConnection);
  try
    Manager.EnableFilter('Multitenant').SetParam('tenant_id', 'acme');
    // ...
  finally
    Manager.Free;
  end;
end;

The example uses TDBConnectionFactory, which creates a new connection on every request. Use a real pool in production, such as the one provided by the TAureliusConnection component or the TDBConnectionPool class shipped with TMS XData.

TTenantPoolProvider implements ITenantDBPool​Provider and is reference counted. It also exposes TTenantPoolProvider.​Invalidate​Tenant, TTenantPoolProvider.​Invalidate​Shard (drops the cached shard information and its pool) and TTenantPoolProvider.​InvalidateAll.

A connection pool that routes by tenant

Most code does not want to ask for a tenant pool explicitly; it wants a plain IDBConnectionPool that does the right thing. TTenantRouting​Connection​Pool is that facade: on every TTenantRouting​Connection​Pool.​Get​Connection call it asks an ITenantResolver for the current tenant id and returns a connection from that tenant's shard pool. Hand it to anything that expects a connection pool, such as an XData server module.

uses
  Aurelius.Tenancy.Context,
  Aurelius.Tenancy.Provider;

var
  Pool: IDBConnectionPool;
begin
  Pool := TTenantRoutingConnectionPool.Create(TAmbientTenantResolver.Create, Provider);
end;

The resolver decides where the current tenant comes from. TAmbientTenant​Resolver reads the ambient tenant scope described next; the XData integration provides a resolver that also reads the HTTP request context.

Ambient tenant scope

Code that runs outside an HTTP request, such as background jobs, scheduled tasks and provisioning routines, still needs a current tenant so that routed pools and filters know which tenant to use. TTenantScope establishes an ambient, thread-local tenant:

uses
  Aurelius.Tenancy.Context;

var
  Scope: ITenantScope;
  Manager: TObjectManager;
begin
  Scope := TTenantScope.Use('acme');
  try
    // Pool is a TTenantRoutingConnectionPool: it now routes to acme's shard
    Manager := TObjectManager.Create(Pool.GetConnection);
    try
      Manager.EnableFilter('Multitenant').SetParam('tenant_id', TTenantScope.CurrentTenantId);
      // ...
    finally
      Manager.Free;
    end;
  finally
    Scope := nil;
  end;
end;

Scopes nest: releasing the ITenantScope restores the previously active tenant, if any. TTenantScope.​Current​TenantId and TTenantScope.​Has​Tenant inspect the current scope. Release a scope on the same thread that created it.

Provisioning tenants

TTenantProvisioner implements the tenant lifecycle over a catalog: creating tenants with automatic shard allocation, changing their status and moving them between shards. Every write invalidates the routing cache passed to the constructor, so the change is effective immediately.

uses
  Aurelius.Tenancy.Provisioning;

var
  Provisioner: TTenantProvisioner;
  Tenant: TTenantInfo;
begin
  Provisioner := TTenantProvisioner.Create(Catalog, TRoundRobinShardAllocator.Create, Cache);

  Provisioner.OnProvisionTenant :=
    procedure(const Tenant: TTenantInfo)
    var
      Manager: TObjectManager;
    begin
      // Runs inside TTenantScope.Use(Tenant.TenantId): routed pools already
      // point to the tenant's shard. Seed the tenant's initial data here
      Manager := TObjectManager.Create(Pool.GetConnection);
      try
        Manager.EnableFilter('Multitenant').SetParam('tenant_id', Tenant.TenantId);
        // Manager.Save(...)
      finally
        Manager.Free;
      end;
    end;

  Tenant := Provisioner.AddTenant('acme', 'ACME Corp');   // allocator picks the shard
  Tenant := Provisioner.AddTenant('globex', 'Globex', 'shard-b');   // explicit shard

  Provisioner.SuspendTenant('acme');
  Provisioner.ActivateTenant('acme');
  Provisioner.MoveTenantToShard('acme', 'shard-b');

  // External identifiers that resolve to the tenant
  Provisioner.SaveIdentifier('user_id', '42', 'acme');
  Provisioner.SaveIdentifier('domain', 'app.acme.com', 'acme');
  Provisioner.RemoveIdentifier('user_id', '42');
end;

TTenantProvisioner.​Add​Tenant is idempotent and safe to retry:

  1. If the tenant already exists and is not pending, it is returned as is.
  2. Otherwise the tenant is created with status Pending, hosted by the given shard or by the shard chosen by the allocator. EShardUnavailable is raised when no shard is available.
  3. If the shard has no catalog record, TTenantProvisioner.​OnProvision​Shard is invoked so the application can create the database and register the shard.
  4. TTenantProvisioner.​OnProvision​Tenant is invoked inside a tenant scope for the new tenant.
  5. On success the tenant becomes Active, unless TTenantProvisioner.​Activate​OnProvision is False. If a callback raises an exception, the tenant stays Pending and a later call to AddTenant for the same id resumes the provisioning. Both callbacks must therefore be idempotent.

TTenantProvisioner.​Move​Tenant​ToShard changes only the routing. Copying the tenant's data from one shard database to the other is the application's job; do it before pointing the tenant to the new shard.

TTenantProvisioner.​Save​Identifier and TTenantProvisioner.​Remove​Identifier maintain the external identifiers of a tenant: the keys that applications use to find the tenant when the request does not carry the tenant id, such as the id of a user, of an API client or a custom domain. SaveIdentifier requires the tenant to exist, and both methods invalidate the cached mapping of the identifier.

Shard allocators

An IShardAllocator chooses the shard of a new tenant among the shards registered in the catalog. Only shards with status Active are considered; Draining shards keep serving their tenants but receive no new ones.

Allocator Strategy
TWeightedHash​Shard​Allocator (default) Hashes the tenant id over the active shards expanded by their TShardInfo.Weight. Deterministic: the same tenant id always maps to the same shard for a given shard configuration, and a shard with weight 2 receives about twice as many tenants as a shard with weight 1.
TRoundRobinShard​Allocator Cycles through the active shards in shard id order.
TFixedShardAllocator Always returns the shard given in its constructor.
TShardPerTenant​Allocator Returns the tenant id as the shard id, giving every tenant a dedicated database.
TCallbackShard​Allocator Delegates the choice to an anonymous function.

Database per tenant

Combine TShardPerTenant​Allocator with TTenantProvisioner.​OnProvision​Shard to create a database for every new tenant. The shard id equals the tenant id, and the callback runs once per new shard, before the tenant is provisioned:

Provisioner.Allocator := TShardPerTenantAllocator.Create;
Provisioner.OnProvisionShard :=
  procedure(const ShardId: string)
  var
    Shard: TShardInfo;
  begin
    CreateTenantDatabase(ShardId);   // application code: create the physical database

    Shard := TShardInfo.Empty;
    Shard.ShardId := ShardId;
    Shard.Status := TShardStatus.Active;
    Shard.Weight := 1;
    Catalog.SaveShard(Shard);
  end;

The shard pool factory then connects to the database named after the shard id, as in the earlier examples.

The TAureliusTenantCatalog component

TAureliusTenant​Catalog packages the catalog, the routing cache and the provisioner into a non-visual component, so they can be configured at design time and shared by other components (TMS XData's tenant connection pool component uses it):

The lifecycle methods are available directly on the component:

TenantCatalog.UpdateCatalogSchema;
TenantCatalog.AddTenant('acme', 'ACME Corp');
TenantCatalog.SuspendTenant('acme');
TenantCatalog.ActivateTenant('acme');
TenantCatalog.MoveTenantToShard('acme', 'shard-b');
TenantCatalog.SaveIdentifier('user_id', '42', 'acme');
TenantCatalog.RemoveIdentifier('user_id', '42');
TenantCatalog.InvalidateTenant('acme');

Exceptions

All tenancy exceptions descend from ETenantException, declared in Aurelius.​Tenancy.​Exceptions:

Exception Raised when
ETenantNotResolved An operation needs the current tenant but none could be determined (no scope, no request context).
ETenantNotFound The tenant id is not registered in the catalog.
ETenantNotActive The tenant exists but is pending, suspended or disabled.
ETenantMismatch Two resolution sources provided different tenant ids.
EShardUnavailable The tenant's shard is disabled, no pool could be created for it, or no shard was available for allocation.
ETenantCatalog​Unavailable The catalog could not be reached. Distinct from not found: the tenant may exist.

The TMS XData integration maps these exceptions to HTTP status codes automatically.