Page MenuHomePhabricator

Consider using var_export in StaticArrayWriter for l10n cache (and possibly other files that are not linted)
Open, Needs TriagePublic

Description

LCStoreStaticArray uses StaticArrayWriter to store l10n cache as a static PHP array on disk. For example, this is the default in development environments, and the plan is to use it in production as well T99740.

StaticArrayWriter has its own logic for pretty-printing PHP variables, and it ensures that the output respects our coding standards. Here's the catch though: we don't need to generate style-compliant l10n cache files. We never run PHPCS on these, and generally, people don't even read them in the first place.

PHP's var_export comes with the guarantee that the returned string is valid PHP code. So, we could either add method parameters or just new methods to let developers choose between the pretty-print implementation (for things that get linted or read) and the native one.

The obvious gain from this is write speed: locally, running rebuildLocalisationCache on the first 100 languages goes from ~9.6s to ~8.2s, roughly -15%. Unfortunately though, the format used by var_export smells like dinosaur: it uses long array syntax (array() instead of []), always outputs array keys even for lists, etc. This is not just a readability thing: it also takes more space. Again locally, for the first 100 languages, the l10n dir size went from 222M (pretty-printed) to 243M (var_export), roughly +9%. We need to make a tradeoff here, and production needs (T99740) should be a priority.

For completeness, I should note that there was an RFC a few years ago about introducing an alternative to var_export with prettier output. Obviously, it didn't pass. var_representation has been implemented as a PHP library, and it works really well in my experience. Sadly though, being userland code, it's even slower than the current StaticArrayWriter implementation.

Event Timeline

Change #1102731 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/core@master] [POC] Replace loops in StaticArrayWriter with var_export

https://gerrit.wikimedia.org/r/1102731

Change #1102731 abandoned by Thiemo Kreuz (WMDE):

[mediawiki/core@master] [POC] Replace loops in StaticArrayWriter with var_export

https://gerrit.wikimedia.org/r/1102731

@Daimona This makes sense to me and seems worth trying.

The raw textual size matters, but I think it matters less than any these points (in chronologicla order, not priority):

  • generation time (scap deploy time),
  • generation memory (scap / docker build memory),
  • container storage (docker image server),
  • data transfer size+time (rsync for bare metal / docker pull for container image layers),
  • opcache size (webserver memory).

Afaik both container storage and data transfer will use compression for text files. Both rsync (zlib stream) and docker image layers (.tar.gz). And opcache size is presumably unaffected by the coding style of the PHP array.

Please confirm this with RelEng/SRE, but I imagine if all else is equal or better, than the raw size of php files currently materialised on disk, would be fine to increase for localisation cache from e.g. 1GB to 1.1GB for the on-disk localisation cache.