Hi everyone,

In T429061#12024164, @APizzata-WMF wrote:

Hey @nshahquinn-wmf the two queries are counting different entities. In mediawiki_history the event_user_central_id is the actor performing the action, whereas the user_central_id is the actual created account. The latter column matches the user_central_id column in the mediawiki_user_history table.

FYI: T423952: Create mediawiki.user_change event stream. event.mediawiki_user_change_dev0 has data now.

maybe you have ideas about the difference between MWH and MWUH? Am I missing anything in my queries?

Hey @nshahquinn-wmf the two queries are counting different entities. In mediawiki_history the event_user_central_id is the actor performing the action, whereas the user_central_id is the actual created account. The latter column matches the user_central_id column in the mediawiki_user_history table.

Related: I've drafted a Dbt model for global registrations (as part of T422732). Anyone interested in this should definitely review it!

FYI, I don't consider this is a urgent issue. My main desire here is to document the oddity, figure out if others know or suspect the reason, and collect ideas for future work.

@JAllemandou maybe you have ideas about the difference between MWH and MWUH? Am I missing anything in my queries?

@Htriedman do you maintain this?

Is the list of countries that we should expect to be filtered out, public? For example, I'm noticing that the data for CZ is very odd - while the country stats for cswiki suggest CZ to be a top country, it does not seem to show up here. Sometimes it does though... and Claude claims (grain of salt, but sofar it seems to hold up) that in all those cases it's labeled as "Czech Republic CZ" . Another explanation (which would be in line with NL) is that MaxMind uses "Czechia" and this means that the country is therefore somewhere in the pipeline rejected.

In T426559#11932608, @Effeietsanders wrote:

(initially posted on the wrong ticket)
It looks like NL has silently reappeared starting with the 2026-03-01 file, confirmed via a timeline scan.

Claude says: This appears to be a side effect of canonical-data commit 556296aae (merged 2026-01-26, @nshahquinn-wmf ), which re-uploaded canonical_data.countries to the Hive Data Lake as part of a routine CTPL refresh. The underlying bug — canonical_data.countries being uploaded without data_risk_classification values for NL after MR !11 (2024-01-25) — was never explicitly fixed. Without a formal fix and test, any future CTPL refresh that doesn't re-upload the full table could reintroduce the same NULL exclusion for NL or other countries.

(initially posted on the wrong ticket)
It looks like NL has silently reappeared starting with the 2026-03-01 file, confirmed via a timeline scan.

I looked into this a bit. Two things I noticed:

Boldly declining this based on feedback above. please re-open as needed.

+1 to @Isaac . I re ran the query that @Krinkle ran using pageview_actor for a day in Oct and Sep 2025. Most of the requests are now tagged as 'automated'. I think this could be a result of the new heuristic we put into place in T395934.

1. the proportion of non https requests are much smaller than the ones tagged correctly as search_engine referer
2. of the ones tagged as external or unknown, majority are automated ie user pageviews arent impacted
3. also, looking at browser_family, the automated requests are mostly coming from 'Chrome' or 'Other' browser family. Firefox isnt dominating this anymore.

Just commenting here too as a duplicate of T406531#11303547: I personally would leave in place the expectation that referrers start with http or https. My read is that that behavior is largely coming from bots who are improperly mocking up a referrer. I don't see nearly the volume that Krinkle saw in January and the majority of it is being labeled as automated (query below). Given that it's also not acceptable behavior per the specs, I'd lean towards us enforcing the expectation of having legitimate referers as a further check against bot data.

Change #1198313 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery/source@master] Update referer classification patterns

https://gerrit.wikimedia.org/r/1198313

Patch has been deployed, last 2 days of backfill ongoing, calling this done :)

Fix unique_devices canonical domain

Mentioned in SAL (#wikimedia-analytics) [2025-10-15T16:05:56Z] <mforns> Finished deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains

Mentioned in SAL (#wikimedia-analytics) [2025-10-15T15:18:31Z] <mforns> Deploying Refinery at 94efa6e8221602a331c19c39ea909eeaa90d98b4 for T405533 unique devices domains

Change #1194885 merged by Joal:

[analytics/refinery@master] Fix unique-devices per domain domains

https://gerrit.wikimedia.org/r/1194885

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1743

Change #1194885 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Fix unique-devices per domain domains

https://gerrit.wikimedia.org/r/1194885

In T405533#11235980, @JAllemandou wrote:

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

In T405533#11235615, @nshahquinn-wmf wrote:

In T405533#11213571, @JAllemandou wrote:

The fix you're demanding

Just to be clear, I'm not demanding anything. I'm requesting a fix I think would be useful, and as I understood it, you were kindly taking the time to discuss it and explain your reasoning, even though you have the authority to make the final decision and are already very inclined to say no.

I'm very sorry for the misunderstanding here. I made a vocabulary mistake with the French false-friend "demander". I was really not meaning that you were forcing for the change, I'll be careful when using the word "demand" in future.

In T405533#11213571, @JAllemandou wrote:

We have changed the source-field used for unique-devices computation when adapting to the .m subdomain removal. We now use the project field used in pageviews, which doesn't have the www for the projects you mention.

Grooming: Joseph, we assigned this to you, feel free to resolve or decline as you see fit. Thank you!

This is all done, no more rows with 0 uniques_estimate on those tables :)