API:REST API/Extensions/Modules

MediaWiki version:

≥ 1.46

This page describes how REST modules work, how to create one for a set of routes, and how to migrate from an old-style set of flat routes to a module.

Purpose of modules

REST modules are a useful way of organizing, exposing, and documenting a set of endpoints that form a coherent RESTful API. Typically, an API centers around some kind of theme, such as the content and history of pages, the content and metadata of revisions, the editing activity of users, or some domain of data managed by an extension. A module defines the programmatic actions that can be taken upon each of a set of related resources. Each action is identified by an HTTP verb and resource path pattern.

As a simple example, suppose we have a domain of user-generated content objects with automatically assigned IDs and human-readable names. Some typical programmatic CRUD actions that might be useful for core JavaScript, user gadgets, bots, and OAuth applications might include:

create a my_item object via POST /my_module/v1/my_item/
read a my_item object via GET /my_module/v1/my_item/{id}
modify a my_item object via PATCH /my_module/v1/my_item/{id}
delete a my_item object via DELETE /my_module/v1/my_item/{id}

Each of these actions needs an endpoint for clients to send their requests and get responses. REST modules provide a standard and convenient way to specify a single modern, RESTful, versioned, API, providing all of the necessary endpoints. The module approach builds off the rest.php/Handler system to further improve logging, traffic analysis, rate limiting, endpoint discovery, routing logic performance, maintainability, endpoint ownership, and the end-user experience with tools like SwaggerUI.

Background concepts

Terminology

Resource path pattern: a rest.php URI path containing any number of bracket-encased named parameters called path parameters. Each path parameter acts as generic placeholder for many possible specific values. A resource path pattern can be thought of as a matching rule for a class of similar resource paths, each one referring to a logical location at which a resource can exist. All the resources at these locations, e.g. the relevant resources, are expected to have a certain common type.
Resource path: a particular rest.php URI path with all of the path parameters specified. Such a path is interpreted to convey both the logical location at which a resource can exist, and, the expected type of any such resource.
Handler: a class, constructed in a particular fashion, designed to processing certain types of client requests delegated via the REST framework and yield a response. A handler expects the requests to be for resources limited to a certain type. Normally, a handler defines the processing a certain operation (HTTP verb) upon a resource (as specified by the path parameters). The handler defines the expected structure of requests and responses, along with the semantics of the operation (e.g. what it does). Some legacy handlers might define how to process several different operations upon a resource (e.g GET and PUT), though this is discouraged due various complications, such as inaccurate generation of request/response specifications.
Route: a combination of (resource path pattern, HTTP verb) that corresponds to some operation that can be performed upon a resource by clients through the REST API.
Route specification: a combination of (resource path pattern, HTTP verb, MediaWikiServices-aware ObjectFactory specification for constructing a Handler subclass instance in a particular fashion). A route specification defines exactly what operation is performed when clients make requests via the given route in the REST API. Since only one Handler can be specified for any given route, the route can be used to identify routes.
Module: a group of related routes that share a resource path prefix. Part of this prefix is a version component. In module definition files, this prefix is specified relative to "rest.php/", (e.g. my_module/v1/).

Note: The version component may soon be able to contain an audience designation suffix per phab:T366567.

Subroute: a route that belongs to a module. Its resource path will have the module's resource path prefix. When a subroute is defined in a module definition file, its resource path is specified relative to the module's resource path prefix.
Endpoint: a collection of rest.php URLs corresponding to a given route, under some particular domain, that are intended to be reachable over HTTP (to the relevant clients), and that are intended to have the request/effect/response behavior of the handler assigned to that route. While the URLs may totally cease to function due to route removal, exceptions, blocking, or outages, they should never function with incompatible semantics. This entails that server errors like "path .../rest.php not found" are reliably distinguishable from rest.php errors like "rest-unknown-module" and rest.php handler errors like "page not found", even if all of them have an HTTP 404 status. Handlers should provide structured error response bodies so clients can reliably make these distinctions.
API: a collection of endpoints, often those of a module. When displayed on the REST sandbox, each API will have its own entry in the dropdown selector.

Resource paths in config

For clarity, this document treats resources path patterns and resource paths as absolute URI paths. Note that in module definition files, resource path patterns (under the paths key) are specified relative to the module's base path. The module's base path is formed by appending its moduleId to the to the rest.php path. For example, a /entities/items/{item_id} key under paths, for a module with moduleId wikibase/v1, refers to the resource path pattern rest.php/wikibase/v1/entities/items/{item_id}. Also, note that in the legacy RestRoutes setting, resource path patterns are specified relative to rest.php.

Resource paths as locations

In normal cases, there should be no need to account for things like non-path parameters or client session when determining the logical location to which a resource path refers. For example, a resource path pattern like /my_module/v1/user-status/{id} is preferable over one like /my_module/v1/current-user-status. The second pattern can often make it harder to reason about API usage and can preclude any possibility of effective caching. However, in some cases, in might be impractical to avoid such resource path patterns (e.g. due to standards like OAuth or the nature of the problem being solved).

Note that some resource path patterns might include parameters like {wikiId} or {project}/{lang}, which situate the resource relative to the entire wiki farm; in such cases, the URI domain is either a general purpose domain for the wiki-farm or the domain of a wiki which belongs on the wiki farm.

Resource path components

There should be an an intuitive, or, at least minimally-surprising, correspondence among resource paths, the logical locations they refer to, and the types of resources that can exist at those locations. This is in addition to the use of response schemas, which can be defined in Handler classes, often by creating and referencing JSON schema files.

Furthermore, it is important that the resource path pattern makes it clear when the expected type of the relevant resources is a collection of objects (e.g. a set page revisions). This is typically done by using a pluralized rightmost component. For example, it would be reasonable for the resource path pattern wikibase/v1/entities/properties/{property_id}/statements to characterize a class of resource paths that each refer to a location where only a collection of "statements", belonging to a "property" entity, can exist. Such pluralization should be avoided when the expected type of the relevant resources is a singleton object (e.g. a single page revision). It should be intuitive when clients have to iterate through a list of things when processing API responses. If a singleton object is expected from a GET on a resource path, it would be surprising and cumbersome for the normal response case to always be a list of one or zero objects to iterate (no HTTP 404). In theory, such endpoint design might facilitate tools that implement batching/joining logic on the client level, and, guard against buggy clients checking for 404 statuses without checking for API error codes in the body (e.g. conflating "wrong endpoint URL" with "resource not found"). If such a response style is useful, then a properly pluralized resource path should be used with a filter query (e.g. my_module/v1/things?id={id}).

The components of resource path schemas are typically used to convey hierarchy, e.g. /my_module/v1/items/{id}/text can convey that it is referencing the "text" field of the singleton object in the "items" collection having the given item ID. More generally, these path components can convey graph-like relations among resources, e.g. /my_module/v1/items/{id}/dependee-item. Avoid using slashes simply as a word separator; instead, use dashes to avoid muddling the meaning of path components. Given the possible use of rightmost resource path component pluralization, it might be natural to use pluralization more broadly, even for various "higher" path components to the left. One can use pluralization in such cases to emphasize subset relations among collection as hierarchy (e.g. /my_module/pages/articles/{id}/tags). Alternatively, one could use component names to emphasize sub-component relations among systems/domains (e.g. /my_module/page/article/{id}/tags).

Generally speaking, filtering, sorting, and limiting parameters should not go in the resource path pattern but rather in query parameters since any URI with such parameters specifies a query of the same underlying collection. Resource path like /my_module/users/{group}/{year_registered}/{limit} are less readable (e.g. /my_module/users/sysop/2014/100), while /my_module/users/group-{group}/year_registered-{year}/limit-{limit} is a bit more readable but still "muddies the waters" regarding the underlying collection and the path component hierarchy. Technically, one can always justify these in a RESTful way by thinking of the path pattern as corresponding to a "collection of collection resources that happen to be different aggregations of an underlying collection of objects that exist in the database". Nevertheless, it might be preferable to move resource path parameters to request query parameters on the relevant route. While query parameters can be provided in URLs in any order, theoretically fragmenting the cache, Wikimedia sites have CDN-level mitigations for this sort of issue; see query string normalization.

Resource paths and existence

Clarity should be when making statements like "the resource does not exist" or "the resource at path X does not exist". Note that there is a distinction between a resource path merely being recognized by the system as a valid location and an actual resource object being located there. Since resources can be collections of resource objects and not just singleton objects, one must further distinguish an empty collection of objects being located at a recognized resource path from no object being located there. For example, suppose each "page" resource object might have a "tags" resource object, itself a collection. These might be accessible via the REST API via /my_module/v1/pages/{id} and /my_module/v1/pages/{id}/tags, respectively. If there is no "page" with ID 12, then there can be no resource object at /my_module/v1/pages/12/tags. Some clients may want to distinguish this situation from an empty collection for an existing "page" resource.

Some resources can be though of a "lazy", being generated on the fly, though possibly cached. For example, one can imagine a GET /my_module/v1/markdown-html/pages/{id} route that yields the markdown-to-HTML rendering of some 'page' resource object that exists in the database. One can think of these lazy generated resources as "existing" at their respective resource paths for all "page" resources. A similar example would be a GET /my_module/v1/user-group-member-counts route that yields breakdown of the number of users in each available user group.

Some resources will act more like content transformers. For example, one can imagine a POST /my_module/v1/markdown-html (or QUERY /my_module/v1/markdown-html) route that gives the HTML rendering of the markdown text provided in the request body. In this case, the transformer itself can be though of a the resource object, somewhat analogous to FIFO files in linux.

Module definition files

Module definition files are JSON files which describe aspects of a module, including:

the module route prefix
the module version
the modules title and description (both message keys and the raw fallback English)
the module subroutes and their corresponding handler construction object spec

The full schema for these files is defined by the latest "mw-api-x.x.json" file in includes/docs/rest/ (e.g. mw-api-1.1.json at the time of this writing).

Here is an simplified example module:

{
	"mwapi": "1.1.0",
	"moduleId": "growthexperiments/v0",
	"info": {
		"version": "0.1.0",
		"title": "Growth experiments API",
		"x-i18n-title": "rest-module-growthexperiments.v0-title",
		"x-i18n-description": "rest-module-growthexperiments.v0-desc"
	},
	"paths": {
		"/suggestions/info": {
			"get": {
				"x-i18n-description": "rest-endpoint-desc-get-growthexperiments-suggestions-info",
				"handler": {
					"class": "GrowthExperiments\\Rest\\Handler\\SuggestionsInfoHandler",
					"services": [
						"GrowthExperimentsSuggestionsInfo",
						"MainWANObjectCache",
						"GrowthExperimentsFeatureManager"
					]
				}
			}
		},
		"/user-impact/{user}": {
			"get": {
				"x-i18n-description": "rest-endpoint-desc-get-growthexperiments-user-impact",
				"handler": {
					"class": "GrowthExperiments\\Rest\\Handler\\UserImpactHandler",
					"services": [
						"GrowthExperimentsUserImpactStore",
						"GrowthExperimentsUserImpactLookup",
						"GrowthExperimentsUserImpactFormatter",
						"StatsFactory",
						"JobQueueGroup",
						"UserFactory"
					]
				}
			},
			"post": {
				"x-i18n-description": "rest-endpoint-desc-post-growthexperiments-user-impact",
				"handler": {
					"class": "GrowthExperiments\\Rest\\Handler\\UserImpactHandler",
					"services": [
						"GrowthExperimentsUserImpactStore",
						"GrowthExperimentsUserImpactLookup",
						"GrowthExperimentsUserImpactFormatter",
						"StatsFactory",
						"JobQueueGroup",
						"UserFactory"
					]
				}
			}
		}
	}
}

Development prerequisites

Make sure that the relevant extension is enabled on your local wiki.
Make sure you locally enable DevelopmentSettings.php so that the specs endpoints and the Special:RestSandbox special page are enabled.

Creating a module

Creating the module patch

These steps involve editing extension.json to reference a new module definition file and making that into a commit.

Determine the route path prefix of the module, in the form of "/<module_name>/v<major_version>", along with all of the subroutes. For example, "/growthexperiments/v0" might correspond to what should be a module having module ID "growthexperiments/v0". Each subroute belongs under a particular (subpath, HTTP verb) combination and contains the Handler object specification (e.g. how to construct the Handler instance from services).
Make a patch adding the module definition file named "<moduleId_without_version>.v<moduleId_major_version>.json". This is sometimes called the "module spec". This file must meets the newest mw-api schema under /docs/rest/ (e.g. "mwapi-1.1.json" at the time of this writing). See includes/Rest/growthexperiments.v0.json from GrowthExperiments as an example.
Update extension.json to include "RestModuleFiles": [ "<path_to_module_definition_file_relative_to_the_extension>" ] (e.g. use "includes/Rest/growthexperiments.v0.json").
Make sure the new specs appear at rest.php/specs/v0/discovery.
Make sure that the new specs appear at rest.php/specs/v0/module/<moduleId> (e.g. rest.php/specs/v0/module/growthexperiments/v0). This is the module's API spec (not to be confused with the "module spec" mentioned above). Create any messages that appear as placeholders (make the en.json and qqq.json files and update "MessagesDirs" in extension.json as needed). Make sure that the rest-module-<moduleId_without_version>.<moduleId_major_version>-title message, in English, says something like "Such-and-such API" (e.g. "Growth Experiments API"). It's probably a good idea to use a Rest/ subdirectory for such messages instead of mixing them with a huge grab-bag of messages for the extension.
Note: You might need to clear APCu or run rebuildLocalisationCache.php after adding new messages.
You can paste the pretty-printed output of rest.php/specs/v0/module/<moduleId> into https://apinotes.io/openapi-validator to lightly validate the output.
You can also paste this output into https://wmf-openapi-linter.toolforge.org/ to check for Wikimedia-specific rules.
Note: Many of the openapi-validator and openapi-linter issues likely already exist with the original flat module API specs, and they might be quite numerous. It is probably best to fix such issues in follow-up work.

Testing module in the RestSandbox

To test how the routes appear in Special:RestSandbox, you will need to modify $wgRestSandboxSpecs.

Note: This step will change in the future, once phab:T409517 is addressed. In the future, this step would be combined into the module patch and would either involve setting an entry under an extension.json key or setting an extra property in the module definition file. Until then, try to avoid moving the location of the module definition file.

1. Locally, to your config, add code that checks is_file() on the new module definition file to set $wgRestSandboxSpecs["<moduleId_without_version>.v<moduleId_major_version>.json"] to an array, having just one key, file, set to "$wgExtensionDirectory/<extension_name>/<path/to/module_definition_file>". Here is an example patch.

Note: For development/experimental endpoints within MediaWiki core, the REST Sandbox Special page activates based on the presence or absence of URLs in the $wgRestSandboxSpecs config variable. To test changes for those endpoints locally, add the module to your LocalSettings.php. For example:

$wgRestSandboxSpecs = [
    'mw' => [
       'url' => $wgScriptPath . '/rest.php/specs/v0/module/-',
       'name' => 'MediaWiki REST API',
    ],
    'specs.v0' => [
       'url' => $wgScriptPath . '/rest.php/specs/v0/module/specs/v0',
       'name' => 'Specs API',
    ],
];

2. Navigate to http://localhost:8080/w/index.php?title=Special%3ARestSandbox (the host/port may be slightly different depending on your local install details). Verify that the Special:RestSandbox page is rendering, and that it shows a working dropdown for the new module. The dropdown name, in English, should be of the form "<something> API".

Migrating flat routes to a new module

The section describes the steps for migrating flat REST routes of an extension to a REST module. This guide assumes that a set of rest routes already exists in extension.json via "RestRoutes", and that they share a module-like path prefix (e.g. "my_module/v1/" or "growthexperiments/v0").

Additional prerequisites

These steps involve testing the unmodified extension in your local development environment.

Verify that the old flat routes are included in the output of rest.php/specs/v0/module/-.
Verify that the old flat routes appear in the Special:RestSandbox page when the dropdown is set to the "(routes not in modules)" option.
Verify that the existing unit tests work.
Verify that the existing api-testing integration tests work locally as a reference point.

Note: This might be a good time to write a patch simply adding tests, if the existing ones are lacking.

Creating the module conversion patch

These steps involve editing extension.json to remove a set of inline route definitions (via "RestRoutes") and replace them with a single reference to a new module definition file (via "RestModuleFiles"). The routes do not change in any meaningful way, e.g. clients still work using the same URI paths and HTTP verbs, but they end up being defined as a proper module of routes rather than flat routes that happen to have the same URI path prefix.

Look at the extension's current flat routes (referenced from "RestRoutes" in extension.json). Find the path prefix, in the form of "/<module_name>/v<major_version>", that corresponds to what should be a module. For example, "/growthexperiments/v0" might correspond to what should be a module having module ID "growthexperiments/v0".
Make a patch adding the module definition file named "<moduleId_without_version>.v<moduleId_major_version>.json". This is sometimes called the "module spec". Update extension.json to include "RestModuleFiles": [ "<path_to_module_definition_file_relative_to_the_extension>" ] (e.g. use "includes/Rest/growthexperiments.v0.json").
Remove the old, redundant, flat route entries from "RestRoutes".
Make sure the file defines all the routes and i18n- message keys and looks like it meets the newest mw-api schema under includes/docs/rest/ (e.g. "mw-api-1.1.json" at the time of this writing). See includes/Rest/growthexperiments.v0.json from GrowthExperiments as an example.
Make sure that the old flat routes are no longer listed at rest.php/specs/v0/module/-.
Verify that the phpunit and api-testing tests pass, since they map to handlers doing the same thing. Note that core will structure test the new module definition files for validity in testModuleDefinitionFiles.
Push the patch for review with "[DNM]" at the start of the message, meaning "do not merge".

Creating the RestSandbox config patch

This step involves creating a config patch to expose the module API on production.

Note: This step will change in the future, once phab:T409517 is addressed. In the future, this step would be combined into the module patch and would either involve setting an entry under an extension.json key or setting an extra property in the module definition file. Until then, try to avoid moving the location of the module definition file.

Create a CommonSettings.php patch in the mediawiki-config repo that conditionally sets the wgRestSandboxSpecs key right after the relevant wfLoadExtension() call, similar to this example. Once deployed, this will add a dropdown for the module at the Special:RestSandbox page, on all relevant wikis, as soon as the module patch rolls out with the deployment train.
Push the patch for review with "[DNM]" at the start of the message, meaning "do not merge".

Merging and deploying

These steps involve deploying the config patch and merging the module patch in a way that will not result in endpoints temporarily going missing from Special:RestSandbox, even if the deployment train rolls back.

Wait for the config and module patch to get an appropriate "good to merge" review.
Remove the "DNM" tag from RestSandbox config patch and get someone to merge/deploy it.
Remove the "DNM" tag from the module patch and ask someone to merge it.
The module patch will go out with the next deployment train.

Maintaining modules

Modules with a version of the form 0.X.Y (e.g. having a path prefix like /my_module/v0/), according the their module definition file, are not considered to have a stable interface. Modules with a major version of 0 are sometimes used for modules that are only meant for "internal" use. For example, a MediaWiki extension might expose an API for by JavaScript components bundled with that very extension. Since the client and API code live in the same git repo, breaking changes can easily be made to support new functionality.

Modules with a major version of 1 or greater, that have been deployed or released into a branch, are expected to maintain a stable interface. Backwards-compatible changes can be made to the module, requiring only minor version bumps. Consumers of APIs, including bots and tools, should generally be careful to either access the expected fields of objects by explicit key, or, be prepared for new fields to be added. The stability requirement is meant to protect consumers from sudden changes to paths, parameters, semantics or losses to response information or structure.

If breaking changes to an endpoint are desired, then either a new endpoint must be defined under the current module version or a new version of the module must be defined with the changed endpoints. For example, a field from an endpoint in /my_module/V1 module might not be present in the corresponding endpoint of /my_module/V2. Module definition files allow for deprecation on both the module and the endpoint level via the "deprecationSettings" field (see mw-api-1.1.json). When a client uses a deprecated endpoint, which includes any endpoint of a deprecated module, a "Deprecation" header will be emitted to the client, mentioning the UNIX timestamp of the deprecation date. Deprecations should be included in any relevant release notes file, API:REST_API/Changelog should be updated, and notice should be sent to the mediawiki-api-announce/wikitech-l mailing lists. Major consumers should be contacted if they continue to use the deprecated endpoint. Before disabling the old API, another notice should be sent to these mailing lists months ahead of actual removal.

Additional information about deprecation can be found at API/Deprecation.