Issues for projects hosted in Cloud-VPS (not: Toolforge which is toolforge.org).
Please check first if there is a sub-project, talkpage, or another dedicated place to file bug reports, so you will get the attention of their developers.
Note: Issues with the Cloud-VPS infrastructure itself should be filed under Cloud-VPS.
Remove wmflabs.org config from Apache conf
{{done}}
That worked! I went ahead and deleted the instance now, as well as the old "patchdemo3-wikis" storage volume that was attached to it.
I found this: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Migrate_from_a_*.wmflabs.org_proxy_to_a_*.wmcloud.org_proxy which says I can just delete the patchdemo.wmflabs.org proxy, and Cloud VPS will make that domain automatically redirect to patchdemo.wmcloud.org. I also found my old notes at https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/work_items/611#note_95789, where 2 years ago I wrote that we should do this "in a few weeks". I don't think there's a way to test it without actually deleting the proxy, so we'll have to take a leap of faith. I will do that now.
Can you advise on how to do that? I don't see an option to move it to another project in the Horizon interface. I presumably also can't just delete it and re-create it under catalyst, since there is no option to choose wmflabs.org as the domain for a newly created proxy.
Yes, you'll need to move the proxy from visualeditor project to catalyst and point it to an instance which will do the redirect. There will be some downtime while the move is in progress, though I think that's acceptable
Thanks. This is resolved
The dumps seems to be mounted and accessible, I'm tentatively resolving this @santhosh , feel free to re-open it if you encounter any issue:
@santhosh as mount_nfs: true is set at project puppet level, the dumps NFS will be mounted on all instances in the project.
Change #1303479 merged by Volans:
[operations/puppet@production] cloudnfs: add dumps to language project
Change #1303479 had a related patch set uploaded (by Volans; author: Volans):
[operations/puppet@production] cloudnfs: add dumps to language project
+1 from me
Still happening as of today (May 2026). The login loop messages continue in #wikimedia-external-links.
You're not a member of project-bastion though - https://ldap.toolforge.org/user/jony
Thinks @bd808! re @Anton.Kokh, FWIW “wikibase.world is on Cloud” is the only connection I saw, so feel free to untag yourself if you prefer.
Based on the 'Wikibase Cloud' tag assigned, is there some involvement expected from our team? I'm struggling to understand how we can support in all this. The only link to us is that wikibase.world is on Cloud.
All the rest seems to be WMF stuff.
@Lydia_Pintscher do you have some opinion?
Wikibase World is not Wikibase Registry at a new domain, but a different wiki that just happened to absorb the Registry's entire scope. This is the mapping between Wikibase Registry and Wikibase World: P49426 (the old redirect service implemented this mapping service).
The redir1 redirect service long stopped working. I do not know why it stopped working.
@bd808 - Would you have any input on?
(Task motivated by the cloud VPS 2025 purge, for the record.)
Thank you very much. This works now as expected. Here the logs from the puppet run for future reference.
sudo puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for qlever1.wikiqlever.eqiad1.wikimedia.cloud
Info: Applying configuration version '(f7f921a774) Muehlenhoff - Remove documentation about the legacy way to configure TLS for Kafka'
Notice: /Stage[main]/Labstore::Traffic_shaping/File[/usr/local/sbin/tc-setup]/ensure: defined content as '{sha256}a122c6d3c9865cca22ef5967f5374099668211de80a5e2c0cab43a9b16edcc7b'
Info: /Stage[main]/Labstore::Traffic_shaping/File[/usr/local/sbin/tc-setup]: Scheduling refresh of Exec[apply_tc_config]
Notice: /Stage[main]/Labstore::Traffic_shaping/Exec[apply_tc_config]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Labstore::Traffic_shaping/File[/etc/network/if-up.d/tc]/ensure: created
Notice: /Stage[main]/Labstore::Traffic_shaping/File_line[enable_ifb]/ensure: created
Notice: /Stage[main]/Labstore::Traffic_shaping/File_line[enable_act_mirred]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/File[/data]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/File[/mnt/nfs]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/File[/public]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/File[/usr/local/sbin/nfs-mount-manager]/ensure: defined content as '{sha256}423f0e73844dc254a2ea40e18df5da0422bc6402d574b7a66bf87777d4764bb9'
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/File[/etc/modprobe.d/nfs-no-idmap.conf]/ensure: defined content as '{sha256}d515adc28fa8c3a24b14ef1dc4255e81cc2567ce0d9b87d846152078ab473cfb'
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Package[nfs-common]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Mount[/mnt/nfs/secondary-scratch]/ensure: defined 'ensure' as 'defined'
Info: Computing checksum on file /etc/fstab
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Mount[/mnt/nfs/secondary-scratch]: Scheduling refresh of Mount[/mnt/nfs/secondary-scratch]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Mount[/mnt/nfs/secondary-scratch]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Mount[/mnt/nfs/secondary-scratch]: Scheduling refresh of Mount[/mnt/nfs/secondary-scratch]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Exec[create-/mnt/nfs/secondary-scratch]/returns: executed successfully
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Exec[ensure-nfs-scratch-on-secondary]/returns: mounting /mnt/nfs/secondary-scratch
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Exec[ensure-nfs-scratch-on-secondary]/returns: mount.nfs: trying text-based options 'bg,intr,sec=sys,proto=tcp,lookupcache=all,nofsc,soft,timeo=300,retrans=3,vers=4.2,addr=172.16.2.104,clientaddr=172.16.18.116'
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[scratch-on-secondary]/Exec[ensure-nfs-scratch-on-secondary]/returns: executed successfully
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/data/scratch]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]/ensure: defined 'ensure' as 'defined'
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]: Scheduling refresh of Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]: Scheduling refresh of Mount[/mnt/nfs/dumps-clouddumps1001.wikimedia.org]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Exec[create-/mnt/nfs/dumps-clouddumps1001.wikimedia.org]/returns: executed successfully
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Exec[ensure-nfs-clouddumps1001.wikimedia.org]/returns: mounting /mnt/nfs/dumps-clouddumps1001.wikimedia.org
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Exec[ensure-nfs-clouddumps1001.wikimedia.org]/returns: mount.nfs: trying text-based options 'bg,intr,sec=sys,proto=tcp,lookupcache=all,nofsc,soft,timeo=300,retrans=3,vers=4.2,addr=208.80.154.142,clientaddr=172.16.18.116'
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1001.wikimedia.org]/Exec[ensure-nfs-clouddumps1001.wikimedia.org]/returns: executed successfully
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/public]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/incr]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-all-sites]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-raw]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pageviews]/ensure: created
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]/ensure: defined 'ensure' as 'defined'
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]: Scheduling refresh of Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]: Scheduling refresh of Mount[/mnt/nfs/dumps-clouddumps1002.wikimedia.org]
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Exec[create-/mnt/nfs/dumps-clouddumps1002.wikimedia.org]/returns: executed successfully
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Exec[ensure-nfs-clouddumps1002.wikimedia.org]/returns: mounting /mnt/nfs/dumps-clouddumps1002.wikimedia.org
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Exec[ensure-nfs-clouddumps1002.wikimedia.org]/returns: mount.nfs: trying text-based options 'bg,intr,sec=sys,proto=tcp,lookupcache=all,nofsc,soft,timeo=300,retrans=3,vers=4.2,addr=208.80.154.71,clientaddr=172.16.18.116'
Notice: /Stage[main]/Profile::Wmcs::Nfsclient/Labstore::Nfs_mount[clouddumps1002.wikimedia.org]/Exec[ensure-nfs-clouddumps1002.wikimedia.org]/returns: executed successfully
Notice: /Stage[main]/Labstore::Traffic_shaping/Kmod::Options[ifb]/File[/etc/modprobe.d/options-ifb.conf]/ensure: defined content as '{sha256}38f0f0e304781d687a4c490dc8b719cbe0b6956a05a186e7962ed852c617cf0e'
Notice: Applied catalog in 20.35 secondsVerified that mounts are working on qlever1.wikiqlever.eqiad1.wikimedia.cloud:
Change #1229159 merged by FNegri:
[operations/puppet@production] cloudnfs: Add wikiqlever project to dumps mounts
Change #1229159 had a related patch set uploaded (by FNegri; author: FNegri):
[operations/puppet@production] cloudnfs: Add wikiqlever project to dumps mounts