Table of Contents

Multi-Tenancy

A multi-tenant XData server serves several customers (tenants) from one API while keeping the data of each tenant separated. XData assembles three pieces for that:

  1. Tenant resolution: the TMS Sparkle TTenantMiddleware finds the tenant id in the incoming request (header, query parameter, URL, subdomain, user claim or domain map) and stores it in the request context.
  2. Tenant catalog and routing: the TAureliusTenantCatalog component maps each tenant to the database (shard) hosting its data, and the TXDataTenant​Connection​Pool component routes every request to the connection pool of the right shard, enforcing tenant status and translating tenancy errors into HTTP responses.
  3. Row-level isolation: an Aurelius global filter keeps the rows of each tenant apart inside a shard shared by several tenants. The pool component enables the filter automatically on every request.

The catalog, cache, routing and provisioning machinery lives in TMS Aurelius and works outside HTTP as well. This chapter covers the XData side; read the Multi-Tenancy chapter in the TMS Aurelius documentation for the underlying classes, and the Global Filters chapter for filters.

Choosing a topology

Topology Setup
Single database, all tenants share it TTenantMiddleware for resolution and the global filter for isolation. No catalog: use a regular TXDataConnectionPool and enable the filter in the OnManagerCreate event, as shown in Server-Side Events.
Sharded: a few databases, several tenants per database TTenantMiddleware, TAureliusTenantCatalog, TXDataTenant​Connection​Pool with EnableFilter set to True. This is the setup described in this chapter.
Database per tenant Same as sharded, with one tenant per shard. Use TShardPerTenantAllocator to provision; the filter is optional.
Multiple databases without a catalog TXDataMultiDBConnectionPool, where the tenant id is the database id. See Multiple databases without a catalog.

Quick start

The steps below build a sharded server with design-time components. The MultiTenantCatalog project in the MultiTenancy demo folder is a complete example.

  1. Drop a TXDataServer and its dispatcher as usual.
  2. Add a TSparkleTenantMiddleware to the server's middleware list. Set HeaderName to the header clients use to choose the tenant (for example, tenant-id) and set RequireTenant to True, so requests without a tenant are rejected with a 400 response before reaching your code.
  3. Drop a TAureliusConnection (CatalogConnection) configured to reach the catalog database: the database that stores which tenant lives in which shard.
  4. Drop a TAureliusTenantCatalog (TenantCatalog) and set its Connection property to CatalogConnection.
  5. Drop a TXDataTenant​Connection​Pool (TenantPool), set its Catalog property to TenantCatalog, set EnableFilter to True, and implement its OnCreateShard​Connection event.
  6. Set the Pool property of the TXDataServer to TenantPool.

The OnCreateShardConnection event is the only application-specific code needed for routing. Given a shard, return a connection to its database. Connection settings and credentials typically come from your configuration file, keyed by the shard id; they are never stored in the catalog:

uses
  Aurelius.Drivers.Interfaces, Aurelius.Drivers.SQLite, Aurelius.Tenancy.Types;

procedure TServerContainer.TenantPoolCreateShardConnection(Sender: TObject;
  const Shard: TShardInfo; var Connection: IDBConnection);
begin
  Connection := TSQLiteNativeConnectionAdapter.Create(Shard.ShardId + '.db');
end;

At startup, create the catalog tables, register the shards, update their schema and register the tenants. All these operations are idempotent, so they can run on every start:

procedure TServerContainer.DataModuleCreate(Sender: TObject);
var
  Shard: TShardInfo;
begin
  // Create/update the catalog tables (tc_tenants, tc_shards, tc_identifiers)
  TenantCatalog.UpdateCatalogSchema;

  // Register the shard databases
  Shard := TShardInfo.Empty;
  Shard.ShardId := 'shard-a';
  Shard.Status := TShardStatus.Active;
  Shard.Weight := 1;
  TenantCatalog.GetCatalog.SaveShard(Shard);

  // Create/update the application schema in every registered shard
  TenantPool.UpdateShardSchemas;

  // Register tenants. Omit the shard to let the allocator choose one
  TenantCatalog.AddTenant('acme', 'ACME Corp', 'shard-a');
  TenantCatalog.AddTenant('globex', 'Globex');
end;

Declare the global filter in your entities and activate the filter enforcer, exactly as described in the Aurelius Global Filters chapter. The filter must be named Multitenant with a parameter named tenant_id, or change the FilterName and FilterParamName properties of the pool component to match your names:

uses
  TypInfo, Aurelius.Mapping.Attributes, XData.Model.Attributes;

type
  [Entity, Automapping]
  [FilterDef('Multitenant', '{TenantId} = :tenant_id')]
  [FilterDefParam('Multitenant', 'tenant_id', TypeInfo(string))]
  [Filter('Multitenant')]
  TProduct = class
  strict private
    FId: Integer;
    FName: string;
    [XDataExcludeProperty]
    FTenantId: string;
  public
    property Id: Integer read FId write FId;
    property Name: string read FName write FName;
    property TenantId: string read FTenantId write FTenantId;
  end;

The XDataExcludeProperty attribute keeps the tenant id out of the JSON payloads: clients never see nor set it, the filter enforcer stamps it on every insert.

Clients choose the tenant by sending the header on every request:

GET /tms/api/Product HTTP/1.1
tenant-id: acme

With TXDataClient, add the header in the OnSendingRequest event of its HTTP client:

Client.HttpClient.OnSendingRequest :=
  procedure(Req: THttpRequest)
  begin
    Req.Headers.SetValue('tenant-id', 'acme');
  end;

How a request is processed

  1. TTenantMiddleware extracts the tenant id from the request and stores an ITenant item in the request context. If RequireTenant is True and no tenant is found, it answers 400 immediately.
  2. When XData needs a database connection for the request, TXDataTenant​Connection​Pool resolves the tenant id (see Resolving the tenant), reads the tenant from the routing cache (or from the catalog on a cache miss), checks that the tenant is active and its shard is not disabled, and takes a connection from the pool of that shard, creating the pool on first use through OnCreateShard​Connection.
  3. If EnableFilter is True, the TObjectManager created for the request gets the Multitenant filter enabled with the tenant id, so every query and every write is scoped to the tenant.
  4. Tenancy errors (unknown tenant, suspended tenant, unavailable shard) are mapped to HTTP responses with proper status codes. See Errors and status codes.

Service operations get the same behavior: TXDataOperationContext.Current.GetManager returns a manager connected to the current tenant's shard, with the filter enabled.

Resolving the tenant

TTenantMiddleware (component TSparkleTenantMiddleware) supports several sources. Set the corresponding property to enable each one; leave it empty to disable it.

Property Source
HeaderName HTTP request header, for example x-tenant-id.
QueryParam Query string parameter, for example ?tenantId=acme.
BasePath First path segment after the base path: with BasePath set to /api, the URL /api/acme/Product resolves acme and is rewritten to /api/Product.
BaseDomain Subdomain: with BaseDomain set to example.com, a request to acme.example.com resolves acme.
UserClaim A claim of the authenticated user (requires an authentication middleware, such as JWT, before the tenant middleware). Recommended for authenticated APIs: the tenant comes from the token and cannot be forged by the client.
DomainMap Explicit map of host names to tenant ids, one domain=tenantId per line.

If several sources are configured and provide different tenant ids, the middleware rejects the request with 400. Set RequireTenant to True to reject requests without a tenant.

TXDataTenant​Connection​Pool resolves the tenant id in this order:

  1. The ambient tenant scope (TTenantScope), if the current thread is running inside one. This is how background jobs and provisioning code select a tenant; see Background jobs.
  2. The ITenant item stored in the request context by the tenant middleware.
  3. The TenantLookups, which resolve the tenant from the identifiers registered in the catalog (see below).
  4. The OnResolveTenant event, which receives the id resolved so far (possibly empty) and may change it.

Looking up the tenant from identifiers

Often the request carries no tenant id, only something that belongs to a tenant: the user id in the sub claim of an access token, an API key header, a custom domain. The catalog stores identifiers for that purpose: external keys registered for a tenant with TAureliusTenantCatalog.SaveIdentifier and resolved with FindTenantByIdentifier. The TenantLookups collection makes the pool component resolve them automatically. Each TTenantLookup says where to read the value from (Source: a user claim, a header, a query parameter or the host name), the Name of that claim, header or parameter, and the catalog Identifier to match the value against (when empty, Name is used).

Configure the lookups at design time in the collection editor, or in code:

uses
  XData.Comp.TenantConnectionPool;

// The tenant of the authenticated user: token claim 'sub' -> identifier 'user_id'
TenantPool.TenantLookups.Add(TTenantLookupSource.UserClaim, 'sub', 'user_id');
// The tenant of an API client: header 'x-api-key' -> identifier 'api_key'
TenantPool.TenantLookups.Add(TTenantLookupSource.Header, 'x-api-key', 'api_key');
// The tenant owning a custom domain: request host name -> identifier 'domain'
TenantPool.TenantLookups.Add(TTenantLookupSource.Host, '', 'domain');

Register the identifiers when the tenant is provisioned, or whenever a user, an API client or a domain is added to a tenant:

TenantCatalog.SaveIdentifier('user_id', UserId, 'acme');
TenantCatalog.SaveIdentifier('domain', 'app.acme.com', 'acme');

Lookups run after the tenant middleware, with these rules:

  • A lookup whose value is absent from the request, or is not registered in the catalog, contributes nothing. If no source resolves a tenant, the request fails with 400 (TenantRequired).
  • If a tenant was already resolved (by the middleware or by a previous lookup) and a lookup resolves a different one, the request fails with 400 (TenantMismatch). A user of one tenant cannot reach another tenant by sending its id in a header.
  • Results are cached by the routing cache of the catalog component, including unknown identifiers (negative cache), so lookups do not hit the catalog on every request. SaveIdentifier and RemoveIdentifier invalidate the affected entry.
  • Lookups are skipped when the tenant comes from an ambient TTenantScope.

UserClaim lookups require an authentication middleware (such as JWT) before the XData module, so the request user is populated.

Custom resolution

Use OnResolveTenant when the tenant must be derived in a way the lookups do not cover. The handler receives the id resolved so far (possibly empty) and can change it. TAureliusTenantCatalog.FindTenantByIdentifier provides cached identifier lookups from code. For example, when API keys have the form <key id>.<secret> and only the key id is registered as an identifier:

uses
  Sparkle.HttpServer.Context;

procedure TServerContainer.TenantPoolResolveTenant(Sender: TObject; var TenantId: string);
var
  ApiKey: string;
  P: Integer;
begin
  if TenantId <> '' then Exit;
  ApiKey := THttpServerContext.Current.Request.Headers.Get('x-api-key');
  P := Pos('.', ApiKey);
  if P > 1 then
    TenantId := TenantCatalog.FindTenantByIdentifier('api_key', Copy(ApiKey, 1, P - 1)).TenantId;
end;

Tokens with a tenant claim

When TMS Sphinx issues the tokens, add the tenant id as a claim at issue time, in the OnConfigureToken event of TSphinxConfig, and let the tenant middleware read it through UserClaim. If the Sphinx server has access to the catalog, its identifiers provide the user-to-tenant mapping:

uses
  Sphinx.Consts, Aurelius.Tenancy.Types;

procedure TServerContainer.SphinxConfigConfigureToken(Sender: TObject;
  Args: TConfigureTokenArgs);
var
  Tenant: TTenantInfo;
begin
  if (Args.Token.TokenType <> TokenTypes.AccessToken) or (Args.User = nil) then Exit;
  Tenant := TenantCatalog.FindTenantByIdentifier('user_id', Args.User.Id);
  if not Tenant.IsEmpty then
    Args.Token.Claims.AddOrSet('tenant_id', Tenant.TenantId);
end;

For machine clients (client credentials flow) Args.User is nil; look the tenant up from Args.Client.ClientId instead, for example through an identifier of type client_id.

In the XData server, place the JWT middleware before the tenant middleware and set UserClaim to tenant_id. The tenant then comes from the signed token: clients cannot switch tenants by changing a header, and no catalog lookup is needed to resolve it.

Connecting to shards

TXDataTenant​Connection​Pool keeps one connection pool per shard, created on first use:

  • OnCreateShard​Connection returns a single connection to the shard. The component wraps it in a standard pool of PoolSize connections (default 20) per shard, destroying idle connections after CleanupTimeout milliseconds when that property is not zero. OnDBConnection​Release fires when a connection returns to the pool.
  • OnCreateShardPool returns a complete IDBConnectionPool for the shard. Use it when you need full control over pooling; when it provides a pool, OnCreateShardConnection is not fired for that shard.

Schema migration is an explicit step: UpdateShardSchemas runs the Aurelius database manager on every registered shard that is not disabled, either for the default model or for the models given as argument. Call it at startup, after registering the shards. It is never run as a side effect of serving a request.

Errors and status codes

The tenancy exceptions raised while serving a request are mapped to HTTP responses using the standard XData JSON error format:

Condition Exception Status Error code
No tenant could be determined ETenantNotResolved 400 TenantRequired
Tenant not in the catalog ETenantNotFound 404 TenantNotFound
Tenant pending, suspended or disabled ETenantNotActive 403 TenantNotActive
Two sources disagree on the tenant ETenantMismatch 400 TenantMismatch
Catalog unreachable ETenantCatalogUnavailable 503 TenantCatalogUnavailable
Shard disabled or pool not available EShardUnavailable 503 ShardUnavailable

The 503 responses carry a generic message, so infrastructure details from the underlying error are never sent to clients.

Set AntiEnumeration to True to answer inactive tenants with the same 404 response as unknown tenants. Clients then cannot probe which tenant ids exist.

Set HandleTenant​Exceptions to False to disable the mapping. The exceptions then follow the regular XData error handling (a 500 response by default), and you can map them yourself in the OnModuleException event, as described in Server-Side Events.

Tenant status is enforced on every request from the cached catalog information, so suspending a tenant blocks it within the cache TTL, or immediately when the change is made through the catalog component in the same process.

Cache and invalidation

TAureliusTenantCatalog caches the routing information it reads from the catalog. CacheTTL (default 30 seconds) is the lifetime of cached tenants and shards; NegativeCacheTTL (default 5 seconds) is how long an unknown tenant id is remembered, so repeated requests for nonexistent tenants do not hit the catalog.

Changes made through the catalog component (AddTenant, SuspendTenant, ActivateTenant, MoveTenantToShard) invalidate the affected entries automatically. When the catalog is modified by other means, or when shard connection settings change, invalidate explicitly on the pool component:

  • InvalidateTenant drops the cached information of a tenant. The next request re-reads the catalog, so a status change or a move to another shard takes effect at once.
  • InvalidateShard drops the cached information of a shard and its connection pool. Use it after rotating the shard credentials: requests in progress finish with their current connections, new requests get a pool built with the new settings. No restart needed.
  • InvalidateAll drops everything.

In a deployment with several server processes, a change reaches the other processes when their cache entries expire, unless each one is told to invalidate (for example, through an administrative service operation).

Provisioning tenants

TAureliusTenantCatalog.AddTenant registers a tenant, allocates its shard and runs your provisioning steps. It is idempotent: an existing active tenant is returned as is, and a tenant left pending by a failed provisioning is provisioned again on the next call.

  • Allocator selects the shard when AddTenant receives no shard id. The default is a weighted hash of the tenant id over the active shards. Other strategies (round robin, fixed shard, shard per tenant, callback) are described in the Aurelius chapter. OnAllocateShard lets you review or override the choice.
  • OnProvisionTenant runs after the tenant is registered, inside a tenant scope, so TenantPool already routes to the new tenant's shard. Seed the tenant's initial data here.
  • OnProvisionShard runs when the tenant's shard has no catalog record yet, before the tenant is provisioned. Create the shard database and register the shard here.

Provisioning is typically exposed as a service operation of an administrative API:

procedure TAdminService.CreateTenant(const TenantId, DisplayName: string);
begin
  ServerContainer.TenantCatalog.AddTenant(TenantId, DisplayName);
end;

procedure TAdminService.SuspendTenant(const TenantId: string);
begin
  ServerContainer.TenantCatalog.SuspendTenant(TenantId);
end;

Database per tenant

To give every tenant its own database, assign a TShardPerTenantAllocator and create the database in OnProvisionShard:

uses
  Aurelius.Tenancy.Provisioning, Aurelius.Tenancy.Types;

procedure TServerContainer.DataModuleCreate(Sender: TObject);
begin
  TenantCatalog.Allocator := TShardPerTenantAllocator.Create;
end;

procedure TServerContainer.TenantCatalogProvisionShard(Sender: TObject; const ShardId: string);
var
  Shard: TShardInfo;
begin
  CreateTenantDatabase(ShardId);   // application code: create the physical database

  Shard := TShardInfo.Empty;
  Shard.ShardId := ShardId;
  Shard.Status := TShardStatus.Active;
  Shard.Weight := 1;
  TenantCatalog.GetCatalog.SaveShard(Shard);
end;

The shard id equals the tenant id, so OnCreateShardConnection connects to the database named after the tenant. In this topology EnableFilter can be left False.

Background jobs

Code that runs outside a request (scheduled jobs, message consumers, provisioning) has no request context. Establish the tenant with TTenantScope and use the pool component as usual:

uses
  Aurelius.Engine.ObjectManager, Aurelius.Tenancy.Context;

procedure TReportJob.Run(const TenantId: string);
var
  Scope: ITenantScope;
  Manager: TObjectManager;
begin
  Scope := TTenantScope.Use(TenantId);
  try
    Manager := TObjectManager.Create(ServerContainer.TenantPool.GetPoolInterface.GetConnection);
    try
      Manager.EnableFilter('Multitenant').SetParam('tenant_id', TenantId);
      // ...
    finally
      Manager.Free;
    end;
  finally
    Scope := nil;
  end;
end;

The scope is thread-local and takes precedence over the request context, so the same code works whether it is called from a job or from inside a service operation that needs to act on another tenant. Alternatively, GetPoolProvider returns an ITenantDBPoolProvider that resolves a tenant id explicitly: TenantPool.GetPoolProvider.GetPool(TenantId).GetConnection.

Filters are not enabled automatically outside requests; enable them on the managers you create, as in the example.

Code-first setup

Everything the components do is available from code, for servers built without the design-time components. The tenant resolver below reads the ambient scope first and then the request context, like the component does:

uses
  Aurelius.Drivers.Interfaces, Aurelius.Drivers.SQLite,
  Aurelius.Tenancy.Catalog, Aurelius.Tenancy.Catalog.Db, Aurelius.Tenancy.Context,
  Aurelius.Tenancy.Interfaces, Aurelius.Tenancy.Provider, Aurelius.Tenancy.Types,
  Sparkle.HttpServer.Context, Sparkle.Middleware.Tenant,
  XData.Aurelius.ConnectionPool, XData.Server.Module;

type
  THttpTenantResolver = class(TInterfacedObject, ITenantResolver)
  public
    function GetTenantId: string;
  end;

function THttpTenantResolver.GetTenantId: string;
var
  Tenant: ITenant;
begin
  Result := TTenantScope.CurrentTenantId;
  if (Result = '') and (THttpServerContext.Current <> nil) then
  begin
    Tenant := THttpServerContext.Current.Item<ITenant>;
    if Tenant <> nil then
      Result := Tenant.TenantId;
  end;
end;

function CreateModule(CatalogPool: IDBConnectionPool): TXDataServerModule;
var
  Catalog: ITenantCatalog;
  Provider: TTenantPoolProvider;
  Pool: IDBConnectionPool;
  Middleware: TTenantMiddleware;
begin
  Catalog := TDbTenantCatalog.Create(CatalogPool);
  Provider := TTenantPoolProvider.Create(Catalog,
    function(const Shard: TShardInfo): IDBConnectionPool
    begin
      Result := TDBConnectionPool.Create(20,
        function: IDBConnection
        begin
          Result := TSQLiteNativeConnectionAdapter.Create(Shard.ShardId + '.db');
        end);
    end);
  Pool := TTenantRoutingConnectionPool.Create(THttpTenantResolver.Create, Provider);

  Result := TXDataServerModule.Create('http://+:2001/tms/api', Pool);

  Middleware := TTenantMiddleware.Create;
  Middleware.HeaderName := 'tenant-id';
  Middleware.RequireTenant := True;
  Result.AddMiddleware(Middleware);
end;

Add the filter activation in the module's OnManagerCreate event and the exception mapping in OnModuleException (see Server-Side Events) to reproduce the EnableFilter and HandleTenant​Exceptions behavior of the component.

When the server is built with a TXDataServer component but the pool must be created in code, return the TTenantRoutingConnectionPool from the server's OnGetPoolInterface event instead.

Multiple databases without a catalog

TXDataMultiDBConnectionPool is a simpler alternative for servers that do not need a catalog: the tenant id resolved by TTenantMiddleware is the database id. The component asks the OnCreateDatabasePool event for the pool of a database the first time it is needed and caches it forever. OnGetDatabaseId can translate the tenant id into a different database id.

procedure TServerContainer.MultiDBPoolCreateDatabasePool(Sender: TObject;
  const DatabaseId: string; var Pool: IDBConnectionPool);
begin
  Pool := TDBConnectionPool.Create(10,
    function: IDBConnection
    begin
      Result := TSQLiteNativeConnectionAdapter.Create(DatabaseId + '.db');
    end);
end;

Choose it when the set of databases is fixed and known to the application (for example, configured in a file), and no tenant status, provisioning or cache invalidation is required. There is no validation of the tenant id: an unknown id reaches OnCreateDatabasePool, and the error raised there is returned to the client as a server error. The MultiTenantByHeader project in the MultiTenancy demo folder shows this setup, side by side with the catalog-based MultiTenantCatalog project.