We may want to emit this change directly from EventBus, not as from a streaming enrichment job. We'd need notifications of the page link changes to do streaming enrichment. We could use the existent mediawiki.page-links-change stream (which does not represent the current state of pages, only links removed and added) as a notification event to enrich a new event, but it would probably be easier to just emit from EventBus.

@calbon This is probably a task that Event Platform will do, unless you want @AikoChou to do some PHP EventBus work :)

If you all do want to do this, we are happy to review, and it would probably help expedite getting it done.

When we do this, we should also consider all the other kinds of mediawiki 'links'. Ideally we can create a common schema for each of these, but emit them as separate streams, e.g. page_links_change, page_images_change, page_category_change, page_templates_change etc. (maybe we'd put links in all the stream names too? TBD I guess).

See T333497#8772933 and below

In https://gerrit.wikimedia.org/r/c/schemas/event/primary/+/914867, @pfischer is modeling a 'link target' in order to get information about page redirect targets for updating search indexes.

This is looking pretty relevant, and I think to do this right we should think together about how we want to model page links in our MW state event schemas. It might be that the redirect target of a page should be propagated to search indexes via the described new page_links_change stream. But, even if we do want to put redirect target info page_change stream, it'd be nice if the data model we use for a page link was the same.

Currently the link_target entity does map to MW's LinkTarget. It is related to the page more than to a revision as this is the information we get without further lookups in PageChangeHooks. There is only one use case for us (cirrus search update pipeline, see https://phabricator.wikimedia.org/T325315#8827182) where it's relevant at what revision the redirect existed/pointed to a certain target.

@pfischer pasting some stuff from Slack here so it doesn't get lost.

For the event data model we need to:

• determine if we can/should make this ‘mediawiki link’ data model able to represent all kinds of mediawiki links, including redirects? Including external links?

• if we decide that this data model shouldn’t represent a kind of link (e.g. external) then we should document that in this phab task, and explain why, and potentially explain how we might want represent those with a different data model.

• Document the possible different kinds of links in the event schema.

Quick recap on what LinkTarget does...

MediaWiki distinguishes between several different kinds of links:

• links to pages (possibly plus a section on that page, and possibly specified as a relative sub-page link; may link to a special page or non-existing page)
• relative section links (jumps on the same page, not recorded in the database)
• interwiki links (links to some external site using an interwiki prefix - the target site doesn't actually have to be a wiki; we can distinguish between "local" interwikis and external interwikis)
• language links (to sister wikis)
• external links (full URL)
• category links
• image (media) links
• template links
• ...maybe I missed something?

While LinkTarget can represent the targets of all kinds of links, it does not distingish between links with different semantics (e.g. [[Category:Foo]] is a categorization, while [[:Category:Foo]] is a regular link to a category). Thes would be recorded to different database tables, but the LinkTarget representation would look the same.

Aye, thanks Daniel! This is great. Similar to but with more detail to what Isaac wrote here.

Most likely what we want to do is define a common event data model that can represent as many of these concepts as we can. But we will probably want to put different kinds of links changes into different streams. E.g. mediawiki.page_wiki_links_change, mediawiki.page_external_links_change, mediawiki.page_image_links_change, who knows. Or maybe we'll want most of them in one stream. TBD. The important part for me right now is the data model(s).

I wrote a schema comparison based on the current database schemas linked in @Isaac's comment. Based on that (see DB Schema Transposed sheet), we would end up with the following bag of properties to fit all link schemas listed above:

• from (all)
• title (pl, iwl, ll)
• from_namespace (pl, il, tl)
• namespace (pl)
• prefix|lang (iwl, ll)
• target_id (tl)
• to (cl, il, el)
• sortkey (cl)
• sortkey_prefix (cl)
• timestamp (cl)
• collation (cl)
• type (cl)
• id (el)
• index (el)
• index_60 (el)
• to_domain_index (el)

@pfischer thanks for compiling all this!
Some fields like to can be of different meaning, a full url for external links and a page_title for images and categories I wonder if we should not rename some of them.
Some of these fields are also db optimizations that I'm not sure we need to expose (e.g. index, index_60, to_domain_index) because I think they'd be trivial to extract.

Would something like this work?

• link_type: enum (page, interwiki, language, external, category, media, template)
• page_title: optional[string] (for page, interwiki, language, category, media, template)
• page_namespace: optional[int] (for page, category, media, template)
• page_id: optional[int] (for page only?)
• is_redirect: optional[boolean] (for page, category?, template?)
• external_url: optional[string] (for external)
• iw_prefix: optional[string] (interwiki, language)
• fragment?: optional[string] (page, language, interwiki, external)

Examples: P48704
Problems I've seen:

• no way to distinguish local uploads from media uploaded to commons for media links
• fragment: is not stored in the db but could be extracted I hope?
• page_id: is not really stored in the db and will be the target page_id at the time the event is emitted, might help to identify redlinks
  • would this be needed for other types like template?
• might be hard to distinguish between external vs local interwiki links (should there be a another field to distinguish those?)
• interwiki prefixes are local to the source wiki, wikt might mean en.wiktionary.org on en.wikipedia.org but fr.wiktionary.org on fr.wikipedia.org, same for language links en might mean en.wiktionary or en.wikisource depending the project the event is emitted from
• there are many optional fields here, might be a code smell that indicates we mix too many different types in the same model?
• the link text itself is not there [[Page#Fragment|click here]] or [https://wikipedia.org click here], click here is not there
• what happens if the page has multiple links to the same target URL but with varying fragment? or varying link text? (they might be dedupped by MW no?)

Thanks!

no way to distinguish local uploads from media uploaded to commons for media links

Wha? How does MW do this then? I guess if the page doesn't exist locally...look for it in commons? Is that built into MW core? Seems weird!

might be hard to distinguish between external vs local interwiki links (should there be a another field to distinguish those?)

Why is this hard? Or do you mean when a wiki link is saved as an external link because the editor used [] instead of [[]]?

interwiki prefixes are local to the source wiki

Maybe we can add a project (or domain?) field that normalizes this?

what happens if the page has multiple links to the same target URL but with varying fragment? or varying link text?

I guess not much we can do about that?

there are many optional fields here, might be a code smell that indicates we mix too many different types in the same model?

The main one that I'm not sure about is external_url. I'd like to use the same model for all links, but this one is the only one is exclusive to (almost) all of the other fields. Perhaps external links should get their own model.

Also, categorylinks is just weird in general, as these don't really seem like 'links' to me, but just happen to be implemented that way.

Anyway, so for @pfischer's purpose in T325315: Add support for redirects in CirrusSearch, the basic model we need now is for redirects, which is the same basic model for normal (local) page links, I think?

• link_type
• page_namespace_id
• page_title
• page_id
• is_redirect

And, actually...this is almost the same as entity/page schema schema fragment (minus link_type, plus revision_count)

I wonder if we can just reuse page entity in the link fragment schema. If we exclude external links from this model, we could do:

title: fragment/mediawiki/state/entity/page_link
$id: /fragment/mediawiki/state/entity/page_link/1.0.0
type: object

allOf:
  - $ref: /fragment/mediawiki/state/entity/page/1.0.0

properties:
  link_type:
    type: string
  is_redirect:
    type: boolean
  
  # + fragment if we can?
  # + link_text if we can?

(We might want to move the page fragment schema revision_count field elsewhere.)

And actually, T325315: Add support for redirects in CirrusSearch is not trying to model in general links on a page, but info about a page's redirectness. So perhaps we don't need to use page_link model there at all, as long as redirect is always to a local page? I'll comment in that ticket now about this idea.

In T331399#8898566, @Ottomata wrote:

Thanks!

no way to distinguish local uploads from media uploaded to commons for media links

Wha? How does MW do this then? I guess if the page doesn't exist locally...look for it in commons? Is that built into MW core? Seems weird!

I mean MW certainly knows at some point but it does not seem that LinkTarget or anything in the imagelinks table has any info about that.

might be hard to distinguish between external vs local interwiki links (should there be a another field to distinguish those?)

Why is this hard? Or do you mean when a wiki link is saved as an external link because the editor used [] instead of [[]]?

I meant "hard" in the sense how to properly model this, MW uses the "interwiki" type for encoding both but we end-up for example encoding the uri_path of an external interwiki link as a page_title which sounds weird to me.

interwiki prefixes are local to the source wiki

Maybe we can add a project (or domain?) field that normalizes this?

I believe this would be helpful indeed.

what happens if the page has multiple links to the same target URL but with varying fragment? or varying link text?

I guess not much we can do about that?

For the CirrusSearch use cases we actually don't care, I just added this for other use-cases that might be more interested in knowing fine-grained details about the links themselves.

there are many optional fields here, might be a code smell that indicates we mix too many different types in the same model?

The main one that I'm not sure about is external_url. I'd like to use the same model for all links, but this one is the only one is exclusive to (almost) all of the other fields. Perhaps external links should get their own model.

Also, categorylinks is just weird in general, as these don't really seem like 'links' to me, but just happen to be implemented that way.

---

Anyway, so for @pfischer's purpose in T325315: Add support for redirects in CirrusSearch, the basic model we need now is for redirects, which is the same basic model for normal (local) page links, I think?

• link_type
• page_namespace_id
• page_title
• page_id
• is_redirect

And, actually...this is almost the same as entity/page schema schema fragment (minus link_type, plus revision_count)

I wonder if we can just reuse page entity in the link fragment schema. If we exclude external links from this model, we could do:

title: fragment/mediawiki/state/change/page_link
$id: /fragment/mediawiki/state/change/page_link/1.0.0
type: object

allOf:
  - $ref: /fragment/mediawiki/state/change/page/1.0.0

properties:
  link_type:
    type: string
  is_redirect:
    type: boolean
  
  # + fragment if we can?
  # + link_text if we can?

(We might want to move the page fragment schema revision_count field elsewhere.)

And actually, T325315: Add support for redirects in CirrusSearch is not trying to model in general links on a page, but info about a page's redirectness. So perhaps we don't need to use page_link model there at all, as long as redirect is always to a local page? I'll comment in that ticket now about this idea.

For CirrusSearch we want to know when the page is a redirect:

• what is the target page_id
• ignore it if the target is yet another redirect (double redirects)
• ignore if the target is an interwiki link

So re-using page_link sounds fine to me (we might actually don't need link_type if we don't model interwiki links yet).
If we want to model interwiki links as part of the page_link model and re-uses this for the page-state stream then I'm not sure we have a good model yet.
Not modeling interwiki links yet might mean that we don't set the link target field when we hit a redirect that points to an interwiki link, we'd just know that it's redirect.

I wonder if we can just reuse page entity in the link fragment schema. If we exclude external links from this model, we could do:

title: fragment/mediawiki/state/change/page_link
$id: /fragment/mediawiki/state/change/page_link/1.0.0
type: object

allOf:
  - $ref: /fragment/mediawiki/state/change/page/1.0.0

properties:
  link_type:
    type: string
  is_redirect:
    type: boolean
  
  # + fragment if we can?
  # + link_text if we can?

(We might want to move the page fragment schema revision_count field elsewhere.)

I think we can omit link_type. Instead, we could add iw_prefix|project|domain:

title: fragment/mediawiki/state/change/page_link
$id: /fragment/mediawiki/state/change/page_link/1.0.0
type: object

allOf:
  - $ref: /fragment/mediawiki/state/entity/page/1.1.0

properties:
  domain:
    type: string
  is_redirect:
    type: boolean
  
  # + fragment if we can?
  # + link_text if we can?

Why did you $ref: /fragment/mediawiki/state/change/page/1.0.0 instead /fragment/mediawiki/state/entity/page/1.1.0, @Ottomata?

Why did you $ref: /fragment/mediawiki/state/change/page/1.0.0 instead /fragment/mediawiki/state/entity/page/1.1.0, @

Oops mistake! I did indeed mean to use entity/page, good catch!

And

fragment/mediawiki/state/change/page_link

Should have been fragment/mediawiki/state/entity/page_link

too. Edited.

Alright, so if we reduce the scope from general link to page link -- that is a link to wiki page, either local or in another wiki -- you would be fine with the schema we sketched up above, @Ottomata? If so, I'd adapt my change request accordingly.

Based on all the info you've gathered, and comments in T325315, I think we can avoid committing a link_target entity schema for now, and just use page info fields in a redirect_target_page as described in T325315#8898609.

When we get around to working on this ticket, we'll use a schema similar to the one you and David proposed above.

So, I think, we can pause on this ticket, and focus on T325315 without making a new generic link target entity. If so, let's work on that in T325315.

Ya?

Change 914867 had a related patch set uploaded (by Ottomata; author: Peter Fischer):

[schemas/event/primary@master] Encode redirect targets in page change events.

https://gerrit.wikimedia.org/r/914867

Alright, in the latest patch for including redirect page link info (T325315), we have added a page_link entity model, that could be used as the following field example:

page_link_target:
  # Info about the page linked to:
  interwiki_prefix: mw
  is_redirect: false
  namespace_id: 0
  page_id: 123
  page_title: The_Page_Title

We can expand and add more fields later as we need to support more types of links than redirects.

My outstanding questions is if we will want/need to be able to represent external links with the same data model. I'm leaning towards not doing it. If we don't, this means there would be two different fields in a new mediawiki.page_links_change stream: page_links and external_links. @achou, reading the Model Card Data section it looks like your model will not use external links for T328899 outlink, right?

This data model is a fragment schema, so we can rename it later, or make another one, if we decide it should include more than links to MW pages.

If there are no objections, @pfischer and I will move forward with this for adding redirect_page_link field to mediawiki/page/change schema in T325315: Add support for redirects in CirrusSearch.

Change 914867 merged by jenkins-bot:

[schemas/event/primary@master] Encode redirect targets in page change events.

https://gerrit.wikimedia.org/r/914867

@achou, reading the Model Card Data section it looks like your model will not use external links for T328899 outlink, right?

Sorry for the delayed response. That's right, the outlink topic model does not use external links to predict article topics.

In my volunteer capacity, I would love to have a stream of external links added (e.g. links to bbc.com, etc.) That'll help me and many volunteers to fight spam much more efficiently which is quite a pain for us at the moment. For example, a tool that would have an "ignore list" of known sites, and then just listen and log anything added that doesn't belong to that list.

In my volunteer capacity, I would love to have a stream of external links added (e.g. links to bbc.com, etc.) That'll help me and many volunteers to fight spam much more efficiently which is quite a pain for us at the moment. For example, a tool that would have an "ignore list" of known sites, and then just listen and log anything added that doesn't belong to that list.

@Ladsgroup I haven't looked at it for years so can't vouch for its current state, but there at least used to be such a stream with all link changes (including external ones). Baha created this little snippet years ago that I saved for seeing what's in the stream: https://codepen.io/bmansurov/pen/BMqKpo

Thanks! That can be quite useful. I might try it out in the hackathon

I just reread this task and comments to try to remember our past intentions and how to make progress. I'll summarize!

What we did:

• We made a page_link data model that is used to represent redirect targets in the existent mediawiki.page_change.v1 stream.
• The page_link data model can currently only represent direct links to local or interwiki pages.

What we want to do.

1. create a new mediawiki.page_links_change.v1 stream that the links on a page before and after a revision is made.
2. Also model and emit other types of link changes; media links, template links, category links, external links, etc.

It is not yet clear how to do 2. Media, Templates and Categories are pages themselves, so we may be able to evolve and use the same page_link data model for these. External links is different enough that it probably needs its own data model (but should it be the new page links change stream? maybe?). We also have to decide what kind of links belong in which streams. E.g. do we want embedded images to show up in mediawiki.page_links_change.v1 stream? Based on comments here and T333497#8775767 and T373644, it seems these should likely be separate streams, even if they might use a similar data model.

So! Next steps:

• Prototype a new mediawiki/page/links_change schema and mediawiki.page_links_change.v1 stream that only has direct (hyper) links to actual MW pages (no external hyperlinks, categories, images or template usages, etc.)
• Try to include external hyperlinks in this model and stream.

After that, we can think about how we might model and emit the meaningful 'usage' links, e.g. categories, images, etc.

interwiki prefixes are local to the source wiki

Maybe we can add a project (or domain?) field that normalizes this?

I believe this would be helpful indeed.

Hm! It looks like we did not do this! We probably should add wiki_id to page_link model. Which makes me wonder why it isn't on the base page entity model! Hm!

Are a page's links a property of the page or a property of the revision?

Links are in raw (wikitext) revision content. We model content as part of the revision. I suppose it makes sense to model information parsed out of the raw content as part of the revision too? If/when we do T360794: Event stream with latest revision HTML & parent revision HTML diff, html content will almost certainly be modeled as part of the revision.

So, I'm leaning towards modeling this in a revision.links field rather than page.links field.

This is awkward (as usual) since page.is_redirect exists but redirectness is a property of a page revision too.

A revision belongs to a page so perhaps a more correct model would have always been to nest revision as page.revision and have fields like page.revision.is_redirect and page.revision.links.

Hm.

Relevant: https://www.mediawiki.org/wiki/Manual:Domain_events/Hierarchy#Page_Rendering_Events

In T331399#11513365, @Ottomata wrote:

Are a page's links a property of the page or a property of the revision?

They are properties of a (typically the current) rendering of page content (aka a ParserOutput object). Page links can change without the page record changing.

Links are in raw (wikitext) revision content.

Some links are. Links can also be generated by templates, or Lua, or parser functions. They may depends on information outside the page, and may change withtout the wikitext changing (and even without a template changing).

This is awkward (as usual) since page.is_redirect exists but redirectness is a property of a page revision too.

Not really, RevisionRecord doesn't know anything about redirects. Content objects have getRedirectTarget(), but that's realyl just a "good guess". The source of truth is the parser. A redirect could be constructed dynamically from templates and parser functions, the Content object would not be able to know that, and the redirect target would not be present in the wiki text. So ultimately, the redirect target is a property of the rendering of a page.

A revision belongs to a page so perhaps a more correct model would have always been to nest revision as page.revision and have fields like page.revision.is_redirect and page.revision.links.

I think it would be useful to explicitly model the current rendering of a page apart from the page record or revision history.

...in other words, we need to distinguish a page-you-can-visit-anf-look-at, the rendered page, from the PageRecord. One may change without affecting the other. Conflating them causes no end of confusion.

This is awkward (as usual) since page.is_redirect exists but redirectness is a property of a page revision too

[...]

So ultimately, the redirect target is a property of the rendering of a page.

Hm, very interesting! I suppose, when I say 'is this a property of a page or revision?' I think what I am meaning to say is 'is this piece of information derived from the page or from the revision'? I wanted to say that Redirectness is ultimately derived from the revision, even if the redirectness may change based on e.g. templates.

But, I suppose your point is, that Redirectness (and links) are not always derived from the page's revision, they are derived from the revisions' rendering which also is derived from many other revisions of other (template etc.) pages.

So while we may want to associate links with a revision, ideally we'd have some other stable versioning of the rendering of the page that we could use to ID that rendering as an entity. And links would be a 'property' of the rendering.

I suppose it gets crazier because there are possibly more than one rendering of a page? Apps might render a page differently? I guess the links we are talking about are a property of the Parser's main desktop html rendering of a page?

I think it would be useful to explicitly model the current rendering of a page apart from the page record or revision history.

I like this idea. I'm not really sure how to start though. Let's keep it simple and use the the redirect link example. Here's how we model this in mediawiki.page_change now.

If a page is currently a redirect page, here is what will show in page_change:

page:
  is_redirect: true
  namespace_id: 1
  page_id: 1
  page_title: example
  redirect_page_link:
    # This is just another page entity 
    is_redirect: false
    namespace_id: 1
    page_id: 2
    page_title: Redirect_Target_Page

(redirect_page_link is just another page entity model).

If we model redirectness (and links etc.) under the page's rendering, we'd want something like:

page:
  is_redirect: true
  namespace_id: 1
  page_id: 1
  page_title: example
page_rendering:
  redirect_page_link:
    is_redirect: false
    namespace_id: 1
    page_id: 2
    page_title: Redirect_Target_Page

And for a new page_links_change stream, we'd do something like:

page:
  is_redirect: true
  namespace_id: 1
  page_id: 1
  page_title: example
page_rendering:
  page_links:
    - is_redirect: false
      namespace_id: 1
      page_id: 3
      page_title: Linked_to_page
  external_links:
    - http://link1.org
    - http://link2.org

?

Also relevant: T360794: Event stream with latest revision HTML & parent revision HTML diff.

Years ago we intended the revision slot model to allow for us to emit html content in the event under revision in the main content slot but with a different content type, like:

yaml
revision:
  comment: changed a thing
  content_slots:
    main:
      content_format: text/hml
      content_model: wikitext
      content_sha1: 16619839a55cfb5c61bcf520bf9734e0c67f98cc
      content_size: 100
      origin_rev_id: 2
      slot_role: main
      content_body: <RENDERED HTML HERE<>

This conversation makes me wonder if this is way off. cc @JMonton-WMF

So while we may want to associate links with a revision, ideally we'd have some other stable versioning of the rendering of the page that we could use to ID that rendering as an entity. And links would be a 'property' of the rendering.

Yes, indeed. We introduced a "render ID" into mediawiki core a while ago, for this reason. But it'S not tracked in the databe at the moment, and there is no "render history".

I suppose it gets crazier because there are possibly more than one rendering of a page? Apps might render a page differently? I guess the links we are talking about are a property of the Parser's main desktop html rendering of a page?

Yes, indeed: the links that are tracked in the database are the ones from the canonical rendering. In theory, other renderings (mobile, but also language specific renderings) may have completely different links. We conveniently ignore this fact, since in practice, the links are usually the same, or almost the same.

I like this idea. I'm not really sure how to start though. Let's keep it simple and use the the redirect link example. Here's how we model this in mediawiki.page_change now.

One could argue that the page is a property of the rendering, or the other way around. The rendering is of a revision of a page. The page has a current (canonical) rendering oif the current revision. With respect to slots, the rendering is of all slots of a revision combined.

We introduced a "render ID" into mediawiki core a while ago, for this reason. But it'S not tracked in the databe at the moment, and there is no "render history".

Oh very interesting! And I see that I can do $linksUpdate->getParserOutput()->getRenderId()! That is great news!

I wonder, for T360794: Event stream with latest revision HTML & parent revision HTML diff, we'll be fetching the HTML content from the REST API. I suppose, since the render ID is not actually tracked in the database, there is no way to fetch the renderId from the revision API, is there? cc also @cscott ?

Just to add a point about the rendering being separate from the revision (I didn't know about render ID but excellent concept): there are certain templates on Wikipedia that dynamically calculate how long since e.g., the page was last edited. For example: https://en.wikipedia.org/wiki/Template:Under_construction which depends on https://en.wikipedia.org/wiki/Module:Time_ago. So alongside the challenges of keeping track of what transcluded content from templates or Wikidata has changed, the text of the page can change from moment to moment based upon when it's rendered too. I don't think these things should matter for these more structured changes you're capturing in links, but they matter a lot for computing HTML-based diffs between pages because you have to make sure that both revisions that you're compared were rendered at the same time.

QQ: are templatelinks part of the current canonical rendering of the current revision?

you have to make sure that both revisions that you're compared were rendered at the same time.

And yeah, this is going to get way more complicated with wikifunctions, eh? ;)

there is no way to fetch the renderId from the revision API ?

Answering my own question: The renderId is returned in the e-tag response header. So we can get render_id from the REST API.