During the 16 March 2026 Editing Team meeting, we estimated the engineering work to be relatively straightforward and the work involved with converging on the UX (including copy and policy/guideline links) to be complex.

Change #1254182 had a related patch set uploaded (by Esanders; author: Esanders):

[VisualEditor/VisualEditor@master] Add LLM paste source detectors

https://gerrit.wikimedia.org/r/1254182

the instability of this metadata will lead to false positives (read: the Check not appearing in cases when people paste text directly from LLMs)

You actually mean "false negatives": it does not appear (negative), and it is false.

Change #1254182 merged by jenkins-bot:

[VisualEditor/VisualEditor@master] Add LLM paste source detectors

https://gerrit.wikimedia.org/r/1254182

Change #1268700 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (2f5c8c924)

https://gerrit.wikimedia.org/r/1268700

Change #1268700 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (2f5c8c924)

https://gerrit.wikimedia.org/r/1268700

Paste check includes a survey if you click "keep" - does LLM check need a survey too and if so are the options the same?

Change #1285882 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/VisualEditor@master] [WIP] PasteCheck: Show different messages when AI source detected

https://gerrit.wikimedia.org/r/1285882

Test wiki created on Patch demo by ESanders (WMF) using patch(es) linked to this task:
https://564a50573d.catalyst.wmcloud.org/wiki/Regent's Park?veaction=edit&ecenable=1

In T420258#11909616, @Esanders wrote:

Paste check includes a survey if you click "keep" - does LLM check need a survey too and if so are the options the same?

@Esanders: great spot. Yes, a survey will be needed. Exact options TBD. I've updated the task description to hold us accountable to defining these options.

In parallel, I'd like to share the patch demo you helpfully created with volunteers for feedback. Before doing so, accurate for me to think testing instructions are as follows?

1. Visit https://564a50573d.catalyst.wmcloud.org/wiki/Regent's_Park?veaction=edit&ecenable=1 on desktop or mobile
2. Copy at least one full paragraph of text from the web interface of ChatGPT, Claude or Gemini
3. Paste the text you copied in "2." into the edit session you started in "1."
4. ✅ Notice the Potential AI-generated content Check appear

This looks amazing! Btw there's a list of each project's policies on AI https://meta.wikimedia.org/wiki/Artificial_intelligence/Policies_by_project

On enwiki a decent amount of LLM use is by people not fluent/confident enough to write English without assistance, I don't know whether a message saying something like "Some editors use AI because they are not fluent enough in [the language]. If this is the case, you may find it easier to contribute to a Wikipedia in your native language.", linking to https://meta.wikimedia.org/wiki/List_of_Wikipedias, would be helpful (or even appropriate). I guess we would test that by how often the link gets clicked?

Wonderful!!! One observation from testing: choosing "No, remove it" from the popup doesn't always remove all of the bullet points or blockquotes from my pasted-in LLM text. I notice that in such cases, the bullets/quotes are not highlighted yellow as "possible AI-generated text" in the first place, so it may have to do with how the detection is working. It may not be a problem pragmatically -- the text is easy to delete manually, and hopefully someone choosing "No, remove it" is already really rethinking their edit -- but it has a "clunky" feeling.

Reporting my positive results too-- it popped up as desired for multiple-paragraph text from Claude (manual copy-paste but not "copy button") and Gemini (both forms of copying).

Fantastic work!

I've been testing it on mobile:

• The standard way I paste on mobile is using the clipboard. It does not trigger either edit check.
• when I paste using the paste button, AI detection works for Gemini, and half for Claude and chatGPT (not with their copy button, but yes if I manually copy)
• when you paste substantial content, the check is off-screen at the top of the pasted content. So it's easy to miss (I assume this is true for all the checks, but most noticeable for paste check)

The standard way I paste on mobile is using the clipboard. It does not trigger either edit check.

I assume you mean the paste button that appears above the keyboard, e.g. on Android devices. That triggers a plain text paste, which means we have no HTML to inspect to work out what the source is. I have considered at least showing paste check for large "inserts" of this nature on mobile in T406841, but we won't be able to detect LLM pastes.

AI detection works for Gemini, and half for Claude and chatGPT (not with their copy button, but yes if I manually copy)

Yes - their copy button writes plainer HTML to the clipboard which doesn't have anything we can detect. I documented these in T379908#10322254.

when you paste substantial content, the check is off-screen at the top of the pasted content.

Probably an edge case - you'd need a very long paragraph for the check message to scroll off screen, but it would be nice if we kept the check message in view for very tall checks.

Probably an edge case - you'd need a very long paragraph for the check message to scroll off screen

I don’t see how this is an edge case? Anecdotally, I’d assume that pasting in an entire generated article is one of the most common uses of LLMs. Unless you mean this problem only appears when an individual paragraph is too long?

BTW, there are two similar TextMatch checks on ruwiki (and I believe enwiki too?) that might be useful (and at least would help with non-English text): LLM-100%-indicators and LLM-indicators.

I don't see this as an edge case either: it's fairly common for people to paste multiple LLM paragraphs in one go. Should I open a separate phab for this?

In T420258#12001569, @Femkemilene wrote:

I don't see this as an edge case either: it's fairly common for people to paste multiple LLM paragraphs in one go. Should I open a separate phab for this?

Feel free to - although even pasting multiple paragraphs it would have to be a lot of content cause a scroll (on desktop at least).

We have generic logging and tagging for paste check. As-implemented LLM is a special case of paste check. I would suggest we log it as such, i.e. regular pastes and LLM pastes continue to trigger regular paste tags and paste actvitiy, in addition to any LLM tags - otherwise there will a sudden dip in our paste data when LLM is deployed.

In T420258#11927776, @Femkemilene wrote:

• when you paste substantial content, the check is off-screen at the top of the pasted content. So it's easy to miss (I assume this is true for all the checks, but most noticeable for paste check)

@Femkemilene: I think T428798 will address this issue...

In T420258#12014408, @Esanders wrote:

We have generic logging and tagging for paste check. As-implemented LLM is a special case of paste check. I would suggest we log it as such, i.e. regular pastes and LLM pastes continue to trigger regular paste tags and paste actvitiy, in addition to any LLM tags - otherwise there will a sudden dip in our paste data when LLM is deployed.

Good spot, @Esanders. I've added this as an open question in the task description.

I've reviewed the copies proposed and I suggest some small changes to make the description shorter (now it's extremely long and difficult to scan) and align better with the rest of suggestions:

• Title: Potential AI-generated content
• Description: This text may include AI-generated content, which is generally prohibited by Wikipedia policy. Help keep Wikipedia trustworthy by rewriting it in your own words and make sure the references support the claims.
• Question: Did you verify this text?
• Buttons: Yes, keep it No, remove it

Questions about links:

• Do we really need the link to "Wikipedia policy"?
  • If no, I would try to remove links in this description.
  • If yes, let's include the link just in "Wikipedia policy" to be shorter.
• Could we use one single link in "references" instead of adding 2 adjacent links?

Paste includes any of the following pieces of metadata:
Target HTML matches...
Gemini data-path-to-node attribute OR <response-element OR BardVeMetadataKey
Claude font-claude-
ChatGPT ( data-start AND data-end attributes ) OR utm_source=chatgpt.com

Can you please also include Grok and DeepSeek detection if it could be supported? They are also quite popular, e.g. DeepSeek is popular in Russia. (also, maybe Microsoft Copilot)?

✖ Grok's HTML looks fairly generic:

<p dir="auto" style="white-space: pre-wrap;">The <strong>cat</strong> (''Felis catus''), also known as the <strong>domestic cat</strong>

It's possible that combination of dir="auto" and white-space: pre-wrap is somewhat unique to Grok, but I'm not sure how we'd verify that.

✔ DeepSeek has the ds-markdown-paragraph class when copying a whole paragraph. It also has a long font stack, but I wonder if that comes from a library rather than being unique to DS.

<p class="ds-markdown-paragraph" style="margin: 16px 0px; color: rgb(15, 17, 21); font-family: quote-cjk-patch, Inter, system-ui, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Oxygen, Ubuntu, Cantarell, &quot;Open Sans&quot;, &quot;Helvetica Neue&quot;, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);"><span class="">The </span><span style="font-weight: 600;">cat</span><span class=""> (</span><em>Felis catus</em><span class="">), also known as the </span><span style="font-weight: 600;">domestic cat</span><span class=""> or </span><span style="font-weight: 600;">house cat</span><span class=""> ...</span></p>

✔ Microsfot Copilot appears to have IDs for the form <hash>-content-<incrementingid> which looks fairly unique:

<div id="Q3DpZeHLT4Yr9nf4FTr7E-content-0"><p><span style="white-space: pre-wrap;">Cats are <strong>small, carnivorous mammals</strong> known </span></p></div>

Change #1305616 had a related patch set uploaded (by Esanders; author: Esanders):

[VisualEditor/VisualEditor@master] Add LLM detectors for DeepSeek and Copilot

https://gerrit.wikimedia.org/r/1305616

For reference

source: https://momenticmarketing.com/blog/top-ai-chatbots

Perplexity has some interesting class attributes, but I'd want to see how stable they are over time first. Given they have a much smaller market share I'm in no rush to add them:

<p class="my-2 [&amp;+p]:mt-4 [&amp;_strong:has(+br)]:inline-block [&amp;_strong:has(+br)]:align-top">The cat (Felis catus) is a small domesticated ....</p>

Change #1305616 merged by jenkins-bot:

[VisualEditor/VisualEditor@master] Add LLM detectors for DeepSeek and Copilot

https://gerrit.wikimedia.org/r/1305616

Change #1307818 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (eafa39fcf)

https://gerrit.wikimedia.org/r/1307818

Change #1307818 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (eafa39fcf)

https://gerrit.wikimedia.org/r/1307818

I've been thinking about the popup, and I'm concerned it's not firmly worded enough to avoid a bait-and-switch feeling from people who paste in LLM content and think they have done "enough" when they are still violating enwiki policies. On enwiki, if someone both rewrote and verified an LLM's output, they'd probably be in the clear, but not if they did only one of the two, and many newcomers are mistaken about how much rewriting & verification are sufficient.

I'd suggest saying "Help keep Wikipedia trustworthy by writing in your own words" (not "rewriting") and asking "Did you write this text?" instead of "Did you verify this text?" to steer more firmly toward the desired outcome.

Is it possible to include Grammarly as well? It's a common one we see, and people are usually unaware it's LLM-powered and that it often goes beyond copyediting the text into introducing hallucinations/changing the meaning

@LEvalyn Thank you! Yes, this is a very complicated piece of wording to get 'right' (in particular for the default version that goes to translatewiki), and we're planning (T429656) on asking a number of communities for feedback on ways to improve those strings of text before we finalize them, probably over the course of the next 2 weeks. Your feedback here (with the precise alternative-wording suggestions) is exactly the kind of thing we're hoping other editors will respond with.

In T420258#12092355, @Alexanderkowal wrote:

Is it possible to include Grammarly as well? It's a common one we see, and people are usually unaware it's LLM-powered and that it often goes beyond copyediting the text into introducing hallucinations/changing the meaning

Grammarly does include identifiable metadata, however I think the signal that the text used violates wiki policy is much weaker. It can generate lots of text, but it is also frequently used to make trivial grammar fixes. Ultimately this will be up to communities to decide, but at the moment I think including Grammarly would generate too many false positives.