Page MenuHomePhabricator
Create Task
Maniphest T221428

Scap should only sync built CDB files to production appserver hosts, not the build files as well
Closed, InvalidPublic0 Estimated Story Points
Actions

Description

Our i18n is currently (see T99740) held and used in l10n_cache-{lang}.cdb files, which are generated from l10n_cache-{lang}.cdb.json files. The former are needed on deployment hosts, the latter aren't. However, we're currently copying both the built and the "upstream" files to each production appserver:

jforrester@mwdebug1002:/srv/mediawiki/php-1.34.0-wmf.1/cache/l10n$ du -sh .
3.5G	.
jforrester@mwdebug1002:/srv/mediawiki/php-1.34.0-wmf.1/cache/l10n$ du -sh upstream/
1.9G	upstream/

That's 2GB to each host we could save each week (and whenever else we do a full scap)…

Related Objects

Event Timeline

Jdforrester-WMF created this task.Apr 18 2019, 11:24 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptApr 18 2019, 11:24 PM
• thcipriani closed this task as Invalid.Apr 19 2019, 12:25 PM
• thcipriani added subscribers: bd808, • thcipriani.

We don't actually sync the cdb files https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/tasks.py#58

We only sync the json files and use those files to regenerate cdb files on each appserver as a last step in a full scap.

This is because (parroting @bd808 who [AFAIK] figured this out initially) the generated cdb files are largely rewritten when any values change and (due to rsync internals) we end up syncing more data than just those values that have changed.

We're saving on time and CPU cycles at the expense of disk space.

moving to T99740 would mean we could stop syncing json and that'd be a nice space savings.

bd808 added a subscriber: aaron.Apr 19 2019, 7:27 PM

Pedantic historian note: the switch from syncing the CDB binary files to syncing the json data files was done all the way back in the DevOps Sprint 2013 by @aaron. This predated the port of the scap code to Python. The reason for the switch was rsync transport optimization as noted by @thcipriani in T221428#5125118.

Krinkle mentioned this in T297326: Stop trying to avoid rsyncing l10n CDB files.Dec 15 2021, 8:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits