WikiLambda provides for hosting, and letting users call evaluation, of functions written, managed, and evaluated on a central wiki, Wikifunctions.
From whatever directory you wish to set up your development environment, run:
git clone "https://gerrit.wikimedia.org/r/mediawiki/core" mediawiki cd mediawiki git clone --recurse-submodules --remote-submodules https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLambda extensions/WikiLambda cd extensions/WikiLambda npm run local:setup
Note that all of the docker container locations mentioned here and in docker-compose.override.yml have the mediawiki- prefix. This is derived from the name of the directory to which you have cloned the mediawiki/core repository.
mediawiki/extensions/ sub-directory, clone the extension as follows:git clone --recurse-submodules --remote-submodules https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLambda
mediawiki/extensions/ sub-directory, also clone the WikimediaMessages and UniversalLanguageSelector extensions:git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/WikimediaMessages git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/UniversalLanguageSelector
git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/EventLogging git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/EventBus git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/TestKitchenFor configuration:
git clone <https://gerrit.wikimedia.org/r/mediawiki/extensions/CommunityConfiguration>When CommunityConfiguration is loaded, WikiLambda registers two providers (visible at
Special:CommunityConfiguration): WikifunctionsSuggestions for the recommended-Wikifunctions list shown in the VisualEditor {{#function:…}} dialog (on client-mode wikis), and AbstractWikiSuggestedWikifunctions for the HTML-returning functions offered in the Abstract Article "Add fragment" menu (on the abstract-mode wiki). Without CommunityConfiguration loaded, neither list is populated — install it on any wiki where you want administrators to be able to curate these lists. For Wikidata content use:git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseThis powers the Wikidata integration in client-mode and abstract-mode features (page-to-QID default values for
{{#function:…}} arguments, Wikidata usage tracking, abstract-mode entity label lookup, and the two PHPUnit integration test files which otherwise markTestSkipped). See the Optional integration: WikibaseClient section below for what changes when it's installed (and the LocalSettings.php snippet to enable client-only mode pointed at production Wikidata). Cloning Wikibase here is also what makes composer phan resolve the absolute \Wikibase\Client\WikibaseClient::* references in our integration call sites.composer.local.json file in your mediawiki/ directory:{
"extra": {
"merge-plugin": {
"include": [
"extensions/WikiLambda/composer.json"
]
}
}
}
docker compose exec mediawiki composer update or similar.LocalSettings.php file:wfLoadExtension( 'WikiLambda' ); $wgWikiLambdaEnableRepoMode = true; $wgWikiLambdaEnableClientMode = true; $wgWikiLambdaEnableAbstractMode = true; $wgWikiLambdaEnableAbstractClientMode = true; $wgWikiLambdaEnableAbstractClientModeIntegration = true; wfLoadExtensions( ['WikimediaMessages', 'UniversalLanguageSelector'] ); # Required wfLoadExtensions( ['EventLogging', 'EventBus', 'TestKitchen'] ); # Recommended, for metrics wfLoadExtensions( ['CommunityConfiguration'] ); # Recommended, for configuration
php maintenance/run.php createAndPromote --custom-groups functioneer,functionmaintainer --force Admin (or docker compose exec mediawiki php maintenance/run.php createAndPromote --custom-groups functioneer,functionmaintainer --force Admin if MediaWiki is setup through Docker) to give your Admin user the special rights for creating and editing ZObjects.php maintenance/run.php update (or docker compose exec mediawiki php maintenance/run.php update if MediaWiki is setup through Docker) to provision necessary schemas and initial content (this step could take around 20 minutes).Done! Navigate to the newly created Z1 page on your wiki to verify that the extension is successfully installed.
There are a number of maintenance scripts used for loading data into your WikiLambda environment. Some data objects are mandatory for a basic configuration and others are just useful for local development and testing purposes.
Here's a summary of your options depending on your needs:
For Wikifunctions Repo:
For Abstract Wikipedia Repo:
function-schemata/data/definitions contains all the JSON files for the canonical built-in ZObjects that a blank Wikifunctions environment must have in a blank installation. When following the general installation instructions, you will be asked to run the MediaWiki maintenance/run.php update script, which loads into the database all the built-in data if they are not there yet. However, the update script will not overwrite any existing data.
To make fresh loads of the built-in data, load a given Zid or range of Zids, merge with existing data, or other more complex loading operations, use the loadPreDefinedObject.php maintenance script.
The loadPreDefinedObject script uses the function-schemata data definition files so always make sure that the function-schemata is up to date. From the Wikilambda directory:
$ git submodule update --init --recursive
To run the script, from your MediaWiki installation directory, do:
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php <OPTIONS>
The script can be called with a number of options that allow you to set the Zid or Zids to insert, and how to handle merges with already existing data.
To configure what objects to load, use the following flags:
--all: Loads all built-in objects, from Z1 to Z999.--zid <Zid>: Loads only the specified Zid from the built-in object range.--from <Zid>: Sets the lower range of the set of sequential Zids to be loaded.--to <Zid>: Sets the upper range of the set of sequential Zids to be loaded.In the absence of any special flags, if an object already exists in the database, the script will skip it. To set a different update or merge strategy, use the following flags:
--force: Whenever an object already exists in the database, overwrite it with the version available in the built-in data directory.--merge: Whenever an object already exists in the database, attempt to merge the current version with the built-in version. The multilingual and other important changes will be kept as currently stored, while other changes will trigger a conflict. Every conflict will request input from the user to decide whether to keep the current version or restore to the builtin version. To skip the interactive mode, use the additional flags:--current: Whenever a conflict is found, this will automatically default to keeping the current value.--builtin: Whenever a conflict is found, this will automatically default to restoring the builtin value.Other configuration flags are:
--wait <ms>: Adds a sleep time (in milliseconds) between insertions.Whenever the script inserts a subset of the builtin objects, specified with --zid or --from and --to, it will track the dependencies and output a notification message at the end of the insertions. These dependencies are not inserted automatically, so you must make sure that they are already available:
Done! > 70 objects were created or updated successfully. Make sure the following dependencies are inserted and up to date: Z14, Z50, Z61, Z1002
To load all built-in objects (only new ones, skip those which are already loaded), use the --all option:
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --all
To load a given Zid, use the --zid option:
# Update Z14 $ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --zid Z14
To load all missing objects in a range of zids, use the --from and --to options:
# Update from Z6000 to Z7000 $ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --from Z6000 --to Z7000
To forcefully insert all objects, overwriting the ones that already exist, use the --force option:
# Force all $ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --all --force # Rewrite Z6005 with its builtin version $ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --zid Z6005 --force
To insert all non-existing objects, and merge the already existing ones, use the --merge option.
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --all --merge
If conflicts are found, the script will request input from the user:
> Conflict: | Zid: Z507 | Path: Z2K2.Z50K1.1.Z3K1 | Current value: "Z7" | Builtin value: "Z99" > Restore to builtin value? (y/n) [n] >
As explained in this warning message, the object Z507 could not be merged automatically. The value found down the key path is currently stored as Z7, but the value as per the function-schemata/data/definitions/Z507.json file is Z99. This requires the user to make an informed decision to either keep the version currently preset in the database (n) or to restore it to its builtin value (y).
To insert all non-existing objects, and merge the already existing ones by always keeping the current values whenever there are conflicts, use the --merge --current options.
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --all --merge --current
To insert all non-existing objects, and merge the already existing ones by always resetting the builtin values whenever there are conflicts, use the --merge --builtin options.
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadPreDefinedObject.php \ --all --merge --builtin
Sometimes it might be necessary to replicate locally the current state of the Wikifunctions.org production database. This can be useful for data debugging purposes, analytics, etc. For this, you can use the loadJsonDump.php maintenance script.
This script requires objects being downloaded via the wikifunctions-content-download script.
To use this script, clone the project and follow the usage guide in the README.md file:
# With ssh: git@gitlab.wikimedia.org:repos/abstract-wiki/wikifunctions-content-download.git # With https: git clone https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions-content-download.git # Install dependencies: cd wikifunctions-content-download npm ci # Run script so that the objects are stored in your Wikilambda directory: npm start -- --dir /home/user/mediawiki/core/extensions/WikiLambda/zobjectcache
This script will download all the production current objects into the zobjectcache folder in your WikiLambda extension. If this is the first time using the download script, it will take some time to complete the job. Otherwise it will just download the objects that have new versions available.
Make sure that your zobjectcache directory contains the following files:
Z0.json which contains a map with the latest revision ID for every ZObject Id.Zid.revision.json or Zid.revision.done.json.You are now ready to use the loadJsonDump.php maintenance script to load all the data into your local installation, using the argument --dir to specify the data directory. The script tracks files that have already been inserted and renames them with *.done.json, so on a default run the script will only insert those files that have not been inserted before:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadJsonDump.php --dir zobjectcache
To force the script to refresh all files, updating even those that have been loaded before, add the flag --refresh:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadJsonDump.php \ --dir zobjectcache --refresh
To insert a given zid, use the argument --zid:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadJsonDump.php --dir zobjectcache --zid Z6005
To load a Zid range, use --from and --to together:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadJsonDump.php --dir zobjectcache --from Z1 --to Z500
NOTE: You must use --zid and --from/to options very carefully, as they will insert the specified Zid or range of Zids without being concerned about their dependencies with other objects. While it's safe to push locally the whole production database, as we are replicating the full context in one go, pushing specified Zids should only be done if we are sure that the context matches the inserted object(s) perfectly.
For Abstract Wikipedia content, use the separate downloader and importer flow:
loadAbstractDump.php.After downloading pages into extensions/WikiLambda/abstractcache, load all pages with:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadAbstractDump.php --dir abstractcache
Load one page:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadAbstractDump.php --dir abstractcache --title Q1
Load a title range:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadAbstractDump.php --dir abstractcache --from Q1 --to Q500
The script tracks files that have already been inserted and renames them with *.done.json, so on a default run the script will only insert those files that have not been inserted before. To force the script to refresh all files, updating even those that have been loaded before, add the flag --refresh:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadAbstractDump.php --dir abstractcache --refresh
To locally test the WikibaseClient's EntittyLookup features that enrich Abstract Wikipedia, this maintenance script helps load a small collection of Wikidata fixtures locally to replicate a minimal setup of Wikibase Repository, so that the Wikibase Client extension can retrieve these entities in a non-pruduction environment.
Before running this script, make sure that you have correctly setup your Wikibase extension following these instructions.
This script loads one or all the items available in the Wikidata fixtures directory. The default fixture directory is in the WikiLambda base repo directory, under the fixtures/wikidata subdiretory. The script can also be called so that it gets the fixtures from another directory with --path.
To load all the files from /fixtures/wikidata/*.json do:
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadWikibaseFixtures.php
To load all the files from another directory (reachable from the docker volume):
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadWikibaseFixtures.php \ --path ./relative/path/to/fixtures
The fixtures directory must contain one JSON file per Wikidata Item object to insert (or update). Each file must have a serializable Item object, like:
{ "type": "item", "id": "Q319", "labels": { "en": { "language": "en", "value": "Jupiter" } }, "descriptions": { "en": { "language": "en", "value": "fifth planet in the Solar System and largest among all" } } }
By default, all files in the fixtures directory will be inserted in alphabetical order. To load only one, pass the name with the option --file.
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadWikibaseFixtures.php \ --file Q319_with_descriptions.json
WikiLambda uses two back-end services for running user-defined and built-in functions in a secure, scalable environment; a set of "evaluators" that run user-defined native code, and an "orchestrator" that receives execution requests and determines what to run.
On install, the extension will try to use the orchestrator and evaluator services, which you may not have running. This default configuration will let you do rudimentary tests of the system, but not proper integration or full running without the back-end services.
You can test your installation by running the PHPUnit test suite as described in the MediaWiki install instructions:
docker compose exec mediawiki composer phpunit:entrypoint -- extensions/WikiLambda/tests/phpunit/integration/ActionAPI/ApiFunctionCallTest.php
If the tests all pass, your installation has successfully called the mocked function orchestrator with the calls, executed them, and got the expected results back. Congratulations!
If you would like to use your own installation of the function orchestrator and evaluator services, please perform the following additional step:
services block in mediawiki/extensions/WikiLambda/docker-compose.sample.yml to the analogous services block in your mediawiki/docker-compose.override.yml.$wgWikiLambdaOrchestratorLocation configuration variable in your LocalSettings.php as explained in the Orchestrator location section below.You can evaluate an arbitrary function call by navigating to localhost:8080/wiki/Special:RunFunction, and selecting a function.
Your Mediawiki installation needs to know the location of your orchestrator service. This variable is set by default to the standard local name for the function orchestrator. You can see this in the extension.json file, in the config property:
"WikiLambdaOrchestratorLocation": {
"description": "Host and port of the function orchestrator.",
"value": "https://mediawiki-function-orchestrator-1:6254/1/v2/evaluate"
}
To run your local orchestrator, you may need to override this variable, by setting its new value in your LocalSettings.php file with the correct name and port of your orchestrator container.
For example, if your orchestrator is called core-function-orchestrator-1 and its port is 6254, set the configuration variable to:
$wgWikiLambdaOrchestratorLocation = "http://core-function-orchestrator-1:6254/1/v2/evaluate";
NOTE: Container names are automatically assigned by docker and often uses the name of the directory where your Mediawiki repository is checked out. To find out the correct details of your docker containers, run docker compose ps in your mediawiki directory. The output should be something similar to this:
NAME COMMAND SERVICE STATUS PORTS core-function-evaluator-javascript-1 "node server.js" function-evaluator-javascript running 0.0.0.0:6929->6927/tcp, :::6929->6927/tcp core-function-evaluator-python-1 "node server.js" function-evaluator-python running 0.0.0.0:6928->6927/tcp, :::6928->6927/tcp core-function-orchestrator-1 "node server.js" function-orchestrator running 0.0.0.0:6254->6254/tcp, :::6254->6254/tcp core-mediawiki-1 "/bin/bash /php_entr…" mediawiki running 9000/tcp core-mediawiki-jobrunner-1 "/bin/bash /entrypoi…" mediawiki-jobrunner running core-mediawiki-web-1 "/bin/bash /entrypoi…" mediawiki-web running 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
In your docker-compose.override.yml file, under the function-orchestrator service section, you'll need to set the following environment variables, which will allow you to enable and configure different features
WIKI_API_URL (mandatory): How to access the Mediawiki API from the function orchestrator. This URL uses the service name of the mediawiki-web container: http://<MEDIAWIKI WEB SERVICE NAME>:8080/w/api.php.http://mediawiki-web:8080/w/api.phpWIKI_VIRTUAL_HOST (optional): E.g. www.wikifunctions.orgORCHESTRATOR_CONFIG (mandatory): Internal configuration object for the orchestrator.evaluatorConfigs (mandatory): Configuration object for each of the available evaluators. Each of them will need to set the correct value for their evaluatorUri configuration to: http://<EVALUATOR CONTAINER NAME>:6927/1/v1/evaluate/evaluatorUri to http://core-function-evaluator-javascript-1:6927/1/v1/evaluate/evaluatorUri to http://core-function-evaluator-python-1/1/v1/evaluate/doValidate (deprecated)addNestedMetadata (optional): Feature flag to activate the return of nested metadata objects.useWikidata (optional): Feature flag to allow internal access to Wikidata.WIKIDATA_API_URL (mandatory): This is necessary to work with Wikidata items and should point at the public Wikidata base URL.https://www.wikidata.orgWIKIDATA_VIRTUAL_HOST (optional): E.g. www.wikidata.orgORCHESTRATOR_TIMEOUT_MS (optional): Sets the orchestrator timeout limit, in milliseconds. If not set, the default value is 20000 ms.localhost:8080/wiki and log in (login: Admin, password: dockerpass)orchestrate function in includes/OrchestratorRequest.php, setting $bypassCache = true:public function orchestrate( $query, $bypassCache = true ) : string {
If you would instead like to develop changes to the function orchestrator or evaluators, you will need to use a locally-built version of the services. To do this for the orchestrator:
git clone --recurse-submodules --remote-submodules https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator.docker build -f .pipeline/blubber.yaml --target development -t local-orchestrator .mediawiki/docker-compose.override.yml to comment out image: docker-registry... in the function-orchestrator service stanza and uncomment the image: local-orchestrator:latest line instead.To do this for the evaluator:
git clone --recurse-submodules --remote-submodules https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator.docker build -f .pipeline/blubber.yaml --target development-javascript-all-wasm -t local-evaluator-js .docker build -f .pipeline/blubber.yaml --target development-python3-all-wasm -t local-evaluator-py .mediawiki/docker-compose.override.yml to comment out image: docker-registry... in the function-evaluator javascript and python services and uncomment the image: local-evaluator-js:latest and image: local-evaluator-py:latest lines instead.Memcached is used in the backend services to store Objects needed for evaluation. The use of Memcached from the backend is completely separated from its use from MediaWiki, so to replicate this behavior in your local environment you must set up a memcached container that is accessible to the backend services.
The Objects are stored up to date by sending a post request to the orchestrator 1/v1/persist-to-cache endpoint during the Deferred Updates. To enable this behavior, set in your LocalSettings.php the following flag:
$wgWikiLambdaPersistBackendCache = true;
To set up a memcached container in your local environment, you can use the memcached Docker image. In docker-compose.override.yaml, enable a separate memcached service:
services:
# ...
memcached:
image: memcached
ports:
- 11211:11211
# ...
11211 is the default port used by memcached.
Then, ensure that the appropriate environment variables are set in the orchestrator config. A partial docker-compose.override.yaml with memcached support looks like this:
services:
# ...
memcached:
image: memcached
ports:
- 11211:11211
function-orchestrator:
image: <some_orchestrator_image>
ports:
- 6254:6254
environment:
WIKI_API_URL: "http://wikifunctions.org/w/api.php"
WIKIDATA_API_URL: "https://www.wikidata.org"
ORCHESTRATOR_TIMEOUT_MS: 40000
ORCHESTRATOR_CONFIG: |
{
"useWikidata": true,
"wikidataImport": {
"qualifiers": true,
"references": true
},
"memcachedUri": "<memcached-container>:11211",
"useMemcached": true,
"memcachedTTL": 60000,
"useWikidataMemcached": true,
"wikidataMemcachedTTL": 60000,
"useBatching": true,
# ...
useMemcached controls whether memcached should be used at all. If this is true, then ZObjects will be cached locally.
memcachedUri should be set to $CONTAINER:$PORT, where $PORT will be 11211 (if using the suggested memcached image) and $CONTAINER will be core-memcached-1 (or similar, depending on the name of the MW core checkout directory and the name assigned to the memcached service in docker-compose.override.yaml).
useWikidataMemcached will enable caching of Wikidata objects, but only if useMemcached is also true.
memcachedTTL and wikidataMemcachedTTL are TTLs for ZObjects and Wikidata objects, respectively.
As of this writing, the TTL parameters are in flux and may not work as expected. For this reason, recency suffers in local deployments, and it may be necessary to regularly take down the memcached service if testing ZObjects/Wikidata objects that have recently changed.
To run the extension PHPunit tests, from the MediaWiki installation directory, run:
docker compose exec mediawiki composer phpunit:entrypoint extensions/WikiLambda/tests/phpunit/
To run the Jest unit and integration tests, from the WikiLambda directory, do:
# Install npm dependencies npm ci # Run tests npm test # Run linter npm run lint:fix # Run unit tests npm run test:unit
From your extensions/WikiLambda directory run:
npm run local:selenium
A set of Selenium tests, used to run end-to-end tests of the application, is available within the project. The tests require an environment with specific versions of things to run, and so it is suggested you use "fresh-node" to run them locally without the need to modify your personal environment.
The tests need a specific set of environment variable to be available. Please see the following list on how to set this https://www.mediawiki.org/wiki/Selenium/How-to/Set_environment_variables
For information on how to run fresh-node and how to get started, see the following documentation: https://www.mediawiki.org/wiki/Selenium/Getting_Started/Run_tests_using_Fresh
After the environment variable and fresh node are both set locally, run the following commands:
// go to your WikiLambda extension cd path/to/WikiLambda # you can start in the extension, no need for core // Initialize Fresh node fresh-node -env -net # you can start fresh in the extension folder, no need for core // Set the variable inline (unless you are loading them from a .env file) export MW_SERVER=http://localhost:8080 export MW_SCRIPT_PATH=/w export MEDIAWIKI_USER=Admin export MEDIAWIKI_PASSWORD=dockerpass # for additional debugging, you can enable video recording # see https://www.mediawiki.org/wiki/Selenium/How-to/Record_videos_of_test_runs export DISPLAY=:1 Xvfb "$DISPLAY" -screen 0 1280x1024x24 & # run the browser tests npm run selenium-test NOTE: the tests will produce some snapshot after completition (both on failure and success). This can be found on "extensions/WikiLambda/tests/selenium/log"
WikiLambda uses PoolCounter to limit the number of concurrent function calls a user may have in flight at any given time. In order to set the concurrency limit, you need to add configuration for a WikiLambdaFunctionCall pool to $wgPoolCounterConf in LocalSettings.php.
The example below allows users to have at most two functions executing at a given time, placing any function calls that exceed the concurrency limit in a queue:
$wgPoolCounterConf = [
'WikiLambdaFunctionCall' => [
'class' => MediaWiki\PoolCounter\PoolCounterClient::class,
'timeout' => 1, // wait timeout in seconds
'workers' => 2, // maximum number of active threads in each pool
'maxqueue' => 5, // maximum number of total threads in each pool
]
];
WikiLambda provides the {{#function:…}} parser function, which lets you embed function calls inside wikitext documents which are then resolved at parse time.
It is currently very limited, allowing only a single direct function call to a function which both takes only strings as its inputs and emits a string as its output.
To use, simply edit any wikitext page. The target function is the zeroth parameter (named by ZID), e.g.:
If your function takes parameters, they can be passed in pipe-delimited, e.g.:
{{#function:Z12345|Hello|world|!}}
Much further functionality is to come.
Create a Function and connected Implementation that outputs an HTML fragment, such as one that wraps a given input string in "…". Note that all direct use of the Function will show you HTML fragment Objects in the interface with the source HTML displayed, and never try to render it as HTML.
Now try to use your new Function as an embedded call:
You should see that the HTML is rendered into the article, suitably sanitised. For example, bad user input of <script>…</script> should get escaped or silently dropped (depending on how it is passed into the system), Functions that try to output unpermitted HTML like <meta> tags will see them dropped, and <a> links will only be allowed to the local server (or, if in a SiteMatrix cluster, local servers) without any attributes beyond the href.
Create a Function and connected Implementation that uses a Wikidata item reference, such as one that displays the label of a given item in a given language. You should be able to use it directly as expected.
Now try to use your new Function as an embedded call:
Note that for embedded Function calls, we default blank language parameter values to the page content render language, and Wikidata item parameter values to the associated QID of the page on which the fragment is being rendered, if available. Consequently, on https://en.wikipedia.org/wiki/Douglas_Adams, the above could be written simply as {}. The page-to-QID lookup behind that default is a WikibaseClient feature; see the next section for what changes when WikibaseClient is (or isn't) installed locally.
WikiLambda has a soft optional dependency on the WikibaseClient extension. Every integration point gates on ExtensionRegistry::getInstance()->isLoaded( 'WikibaseClient' ) at runtime, so the extension loads, runs, and serves pages without it, but a few features degrade gracefully or are silently disabled.
These are the Wikibase-enabled features, the environments or setups in which they are applicable and the fallback behavior in case WikibasClient is not available.
Wikidata item default values for {{#function:…}} arguments.
Wikidata Item/Z6001 or Wikidata Item Reference/Z6091), if this argument is omitted, the parser function replaces the missing value with the QID linked to the current client page (if any). This resolution is done through WikibaseClient's SiteLinkLookup service.Wikidata usage tracking for QID arguments.
{{#function:…}} call passes a QID-shaped string (whether explicit or from the default above), WikiLambda records the page as a user of that entity through the WikibaseClient's UsageAccumulatorFactory service. This record makes the page purge and refresh when the linked Wikidata Item is edited.Wikidata-item labels in abstract-mode rendering.
EntityLookup service.Additionally, the absence of the WikibaseClient extension affects the following dev workflows:
Two PHPUnit integration test files. tests/phpunit/integration/HookHandler/PageRenderingHandlerAbstractModeTest.php and tests/phpunit/integration/ParserFunction/WikifunctionsPFragmentHandlerTest.php both call markTestSkipped( 'WikibaseClient not available' ) if the extension is absent. Without WikibaseClient these tests show as skipped (not failed); if you are debugging abstract-mode label rendering or PFragment usage tracking, install Wikibase first or you will be running with no coverage of those paths.
composer phan static analysis. .phan/config.php lists ../../extensions/Wikibase in directory_list (and excludes it from analysis) so that absolute references like \Wikibase\Client\WikibaseClient::getEntityIdParser() resolve. Without a Wikibase clone in extensions/, composer phan reports undefined classes at the integration call sites.
Other Wikidata features that do not depend on WikibaseClient services and will work in any environment, are:
resources/ext.wikilambda.app/components/types/wikidata/*), as they depend on remote requests to the Wikidata Action APIs.ext.wikilambda.search/wikidata.js) talks directly to the public Wikidata Action API over HTTP.Other things to know:
wgWikibaseItemId is read from the front-end (in resources/ext.wikilambda.app/components/visualeditor/FunctionInputPreview.vue and store/stores/visualeditor.js), it is populated by WikibaseClient itself on pages that have a sitelink. Without WikibaseClient the global is undefined and the code coerces it to '' — VE preview just won't pre-fill the QID.git submodule update --init --recursive in function-schemata/ — Wikidata-typed builtins (Z6001, Z6091, Z6010, Z6011, Z6062, Z6064) live there and a stale submodule is a common cause of confusing test failures around these types.In Wikimedia production, WikibaseClient is integrated with Wikidata as the central entity repository. Features such as sitelink resolution and EntityLookup rely on production-specific infrastructure shared between the client wikis and the repository, including site configuration and cross-service integration.
This setup is not available by default in local or non-production environments, even if WikibaseClient is enabled. As a result, lookup-dependent features cannot be tested locally against real Wikidata infrastructure. To use and test EntityLookup in non-production environments, you can set up a minimal WikibaseRepository and load a minimal set of Wikidata fixtures. To accomplish this, follow the steps:
1. Install Wikibase
Clone the Wikibase extension and its submodules, in mediawiki/extensions/, do:
git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase cd Wikibase git submodule update --init --recursive
Make sure that Wikibase is added in your MediaWiki composer.local.json file and the dependencies are up to date with:
docker compose exec mediawiki composer install
2. Enable Wikibase Repository and Client
Then add at the bottom of your LocalSettings.php:
# WikibaseRepository settings: # =========================== wfLoadExtension( 'WikibaseRepository', "$IP/extensions/Wikibase/extension-repo.json" ); require_once "$IP/extensions/Wikibase/repo/ExampleSettings.php"; # WikibaseClient settings: # ======================= wfLoadExtension( 'WikibaseClient', "$IP/extensions/Wikibase/extension-client.json" ); require_once "$IP/extensions/Wikibase/client/ExampleSettings.php"; $wgWBClientSettings['entitySources'] = [ 'local' => [ 'entityNamespaces' => [ 'item' => 120, 'property' => 122, ], 'repoDatabase' => $wgDBname, 'baseUri' => $wgServer . '/entity/', 'rdfNodeNamespacePrefix' => 'wd', 'rdfPredicateNamespacePrefix' => '', 'interwikiPrefix' => '', ], ]; $wgWBClientSettings['itemAndPropertySourceName'] = 'local'; $wgWBClientSettings['siteGlobalID'] = 'enwiki';
3. Run the maintenance scripts
Initialize the WikibaseRepository and WikibaseClient tables and site metadata:
docker compose exec mediawiki php maintenance/run.php update docker compose exec mediawiki php maintenance/run.php ./extensions/Wikibase/lib/maintenance/populateSitesTable.php docker compose exec mediawiki php maintenance/run.php ./extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php docker compose exec mediawiki php maintenance/run.php populateInterwiki
4. Verify that Item creation is possible
Not really a necessary step, but it doesn't hurt to check:
5. Insert Wikidata fixtures
Run the loadWikibaseFixtures script.
docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/loadWikibaseFixtures.php
Then test that the pages stored in the /fixtures/wikidata directory are correctly inserted by going to their view pages. For example, for testing Jupiter/Q319 go to http://localhost:8080/wiki/Item:Q319
6. You can now test your EntityLookup features
If you want, you can unset WikibasRepository by commenting the repository-related configuration lines and leaving only the WikibaseClient configuration ones.
For more configuration options, see Wikibase advanced installation guide
Remember to re-enable WikibasRepository whenever you want to run the loadWikibaseFixtures script.
To develop in abstract mode you need the Wikibase extension (WikibaseClient) and CommunityConfiguration installed alongside WikiLambda.
WikibaseClient is used to resolve Wikidata item labels server-side — for example, turning a QID like Q42 into "Douglas Adams" in page titles and link text. Without it the code falls back gracefully to displaying raw QIDs.
CommunityConfiguration provides:
See the recommended extensions section above for installation instructions.
An important distinction to make in Abstract Wikipedia related environments is the one between Repository and Client mode.
We call Abstract Wikipedia Repository or Abstract Repo a MediaWiki instance that is set up to host AbstractWikiContent, or Abstract Wikipedia multilingual article definitions. The Abstract Articles hosted in Abstract Repo can be rendered into Wikipedia Articles in any language. In production, the Abstract Repo can be accessed at https://abstract.wikipedia.org.
We call Abstract Wikipedia Client or Abstract Client a MediaWiki instance that is set up to use or display rendered Abstract Articles in a particular language. For example, any language Wikipedia which integrates articles generated from Abstract Repo source.
On a mediawiki-docker environment for local development, there are two options for developing Abstract Wikipedia:
$wgWikiLambdaClientTargetAPI = 'http://mediawiki-web:8080'; $wgWikiLambdaEnableRepoMode = true; $wgWikiLambdaEnableClientMode = true; $wgWikiLambdaEnableAbstractMode = true; $wgWikiLambdaEnableAbstractClientMode = false; $wgWikiLambdaEnableAbstractClientModeIntegration = false;
$wgWikiLambdaClientTargetAPI = 'https://www.wikifunctions.org'; $wgWikiLambdaEnableRepoMode = false; $wgWikiLambdaEnableClientMode = true; $wgWikiLambdaEnableAbstractMode = true; $wgWikiLambdaEnableAbstractClientMode = true; $wgWikiLambdaEnableAbstractClientModeIntegration = true;
The rendering of Abstract Articles in specific languages is done using the updateAbstractWikiArticleStore maintenance script.
This script can be called with:
--topics A list of topics (Wikidata Item qids) to render to the wanted languages. If no --topics is passed, then it uses the list of topics set for the config variable WikiLambdaAbstractWikiArticleStoreTopics--langs A list of language codes (Wikifunctions accepted BCP-47 identifiers). If no --langs is passed, then it uses the list of languages set for the config variable WikiLambdaAbstractWikiArticleStoreLangs--pending A boolean flag to determine whether we want to regenerate only the content marked as pending.This script does the following:
As an exception to this behavior, if a section is not fully rendered for a language -- it contains one or more pending fragments:
This behavior highlights that we prefer showing an outdated version of an article rather than showing an incomplete one. But we prefer showing an incomplete or pending article than showing nothing.
To run this script for the topics and languages set in the configuration variables, do:
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/updateAbstractWikiArticleStore.php
To run this script for a given set of topics and languages, do:
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/updateAbstractWikiArticleStore.php \ --topics Q42,Q319 --langs en,es,ru,zh
To only rewrite the sections flagged as pending, do:
$ docker compose exec mediawiki php maintenance/run.php \ ./extensions/WikiLambda/maintenance/updateAbstractWikiArticleStore.php \ --topics Q42,Q319 --langs en,es,ru,zh --pending