I've backported @jhathaway's upstream fixes onto PHP 8.1.33 and the packages are ready to go (but not yet included in our apt repo). If there's interest, and folks are available to monitor the result, I can aim to deploy those this week.

Edit: So they exist outside of the build host, I've attached the resulting patches, corresponding to upstream commits c5e7490963a1c53ec12c09264d0dbe1654ec8cfc and 6a7561203de89031147c0de6fe09cad905f82984. These apply cleanly on top of the existing series for the Debian PHP team's debian/main/8.1 branch as of debian/8.1.33-1 and all related tests pass (after reenabling the test suite in debian/rules).

Thanks! We are definitely interested. By monitoring do you just mean checking if this error becomes less frequent / the error message becomes more useful (can do) or testing email sending after the deployment (can do as well if you ping me)?

Thanks, @Tgr!

In T383047#11062417, @Tgr wrote:

Thanks! We are definitely interested. By monitoring do you just mean checking if this error becomes less frequent / the error message becomes more useful (can do)

Yes, just broadly anything you might think of to confirm that the patches are working as expected. I don't have a great mental model of what the observable effect of the error-handling correctness fix might look like up in MediaWiki side of things.

or testing email sending after the deployment (can do as well if you ping me)?

Ah, that's a good idea as well. Would this just be via Special:EmailUser, or something else?

In any case, I'd propose we aim for the 17:00 UTC MediaWiki infrastructure window if that works (this requires an update to the base PHP 8.1 image, which is preferable to do during a dedicated window).

If it works as intended, I think the only change is Logstash error messages (link) getting more informative.

All email sending goes through the same code, so I think testing Special:UserEmail (or EmailAuth, signup etc) should be enough. For less frequent errors, I can look at EmailAuth stats after a day and see if verification success rates changed. Maybe do the same for email confirmation, not sure how we log that.

In any case, I'd propose we aim for the 17:00 UTC MediaWiki infrastructure window if that works

The Wednesday one, right? Sounds good.

That sounds good - thanks, @Tgr. Also yes, apologies, I should mentioned explicitly I meant Wednesday 2025-08-06.

Change #1175951 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/docker-images/production-images@master] php8.1: rebuild to pick up 8.1.33-1+wmf11u2

https://gerrit.wikimedia.org/r/1175951

Mentioned in SAL (#wikimedia-operations) [2025-08-06T17:03:54Z] <swfrench-wmf> reprepro include php8.1_8.1.33-1+wmf11u2 in component/php81 - T383047

Change #1175951 merged by Scott French:

[operations/docker-images/production-images@master] php8.1: rebuild to pick up 8.1.33-1+wmf11u2

https://gerrit.wikimedia.org/r/1175951

Mentioned in SAL (#wikimedia-operations) [2025-08-06T17:10:40Z] <swfrench-wmf> built and published php8.1 production image stack at 8.1.33-1-s3 - T383047

Mentioned in SAL (#wikimedia-operations) [2025-08-06T17:11:21Z] <swfrench@deploy1003> Started scap sync-world: Deployment to pick up new 8.1.33-1-s3 production images - T383047

Mentioned in SAL (#wikimedia-operations) [2025-08-06T17:32:52Z] <swfrench@deploy1003> swfrench: Deployment to pick up new 8.1.33-1-s3 production images - T383047 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-08-06T17:56:05Z] <swfrench@deploy1003> Finished scap sync-world: Deployment to pick up new 8.1.33-1-s3 production images - T383047 (duration: 45m 10s)

The error-handling improvements should be live as of a bit before 18:00 UTC today. Many thanks to @Tgr for hanging around to verify the behavior of email-related workflows.

Seems to be working:
Could not send confirmation email: Sendmail exited with non-zero exit code 74
Thanks a lot @jhathaway and @Scott_French for fixing the error reporting issue!

(74 is EX_IOERR, which typically means some sort of disk error. We'll see if all errors are caused by this, or other error codes show up as well.)

So far:
exit code 74 (EX_IOERR): 72
exit code 65 (EX_DATAERR): 7
exit code 64 (EX_USAGE): 1

Do we want to keep using this task for debugging those, or close it (since the issue that caused "Unknown error" to be reported was fixed) and file separate tasks?

We get the correct exit codes now, which is fabulous. However, what is less fabulous as @Tgr notes is that the majority of the error codes are 74, which is used by msmtp to indicate either a disk I/O error or a network I/O error, both of which are broad categories. So, we really need more debug info, how do we obtain it?

Ideas:

1. Modify msmtp to emit more specific exit codes
2. Configure msmtp to log to a file, then slurp those files up via rsyslog
3. Wrap the msmtp command with a script, which sends stderr and stdout to a logger process configured to ship logs to a remote rsyslog, something akin to https://gist.github.com/solidsnack/6090947

My preference is (3), which I think would be easier to implement than (2), while giving us the same data. One issue with (3), is avoiding logging users email addresses, perhaps we could redact them in the wrapper program?

I don't know enough about mailservers to have an opinion on the options (other than that (1) sounds very painful since all we'd get is an integer), but I wouldn't worry much about logging email addresses - they are logged in other parts of the call stack anyway.

In T383047#11087096, @Tgr wrote:

I wouldn't worry much about logging email addresses

In some unrelated conversation someone pointed out that some of the emails we send are quite sensitive - donors, partners etc. So let me rephrase that, I wouldn't worry about logging email addresses for emails sent by MediaWiki. Not sure to what extent those are separate from various other kinds of emails.

In T383047#11073192, @Tgr wrote:

So far:
exit code 74 (EX_IOERR): 72
exit code 65 (EX_DATAERR): 7
exit code 64 (EX_USAGE): 1

Now it's
exit code 74 (EX_IOERR): 549
exit code 65 (EX_DATAERR): 94
exit code 64 (EX_USAGE): 1

Just noting I just got Sendmail exited with non-zero exit code 74 twice in a row when attempting to email the Oversight user (!) on enwiki. (I then switched to sending via normal email.)

Edit: Actually, I was just notified both attempts /did/ get sent to the Oversight mailing list despite the error message.

Another user would have gotten the errors also. We have about 3-4 emails to the OS team per OS request because people think it's not getting through when it is.

Got the errors too today, three times in a row but the emails were sent.

Getting "Sendmail exited with non-zero exit code 74" using en-Wikimail to another established user.

Just adding that I'm getting this too, over the past three hours, using "email this user" from both meta and enwp, with mail going to two different users.

Update: got this error over a period of several hours on 10/19. When I tried again on 10/20, it had resolved.

I just observed this error at VRT wiki.

Would it make sense to retry once in MediaWiki on sendmail errors?

This is still happening, ~3.3k per in last 4 weeks:

What would be the next steps? I assume this error is something that can happen - sendmail can fail due to various reasons. @Tgr - do we do any kind of retries here?

Last 24 hours - 127 instances of message: Could not send confirmation email: Sendmail exited with non-zero exit code 74 - https://logstash.wikimedia.org/goto/1425bc3e85340cb5965e3757623deb8c

Saw the issue in the enwiki UI (non-reproducible):

It's level: WARNING but causes confirmation code to expire; a user need to start everything over.

In T383047#11469444, @pmiazga wrote:

What would be the next steps?

Either do one of the debugging options @jhathaway mentioned in T383047#11075860, or add retry functionality.

I just ran into the same issue when I tried to send an email to someone.

The volume of this error has doubled after Jan 26-28 (Logstash permalink)

• Server Admin Log
  • 2026-01-27 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.46.0-wmf.13
  • 2026-01-28 23:09 brennen@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.13
  • 2026-01-28 23:22 brennen@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.13
• MediaWiki 1.46/wmf.13
  • Nothing stands out in terms of MediaWiki changes.

Hi all, I just experienced this error as well as a few community members have reached out that they did not receive an email from me during the Community Insights distribution yesterday (however, they did receive an echo ping to check their email). When I tried to send a message directly via EmailUser on the project to one of those community members just now to troubleshoot, I got the Sendmail exited with non-zero exit code 74 message. I'm unsure of the scale of the unsent emails as I didn't get a failure for these with the EmailUser API script I was using, but it should be visible in yesterday's log (if such a log exists, I presume it does per @Krinkle's last message but I seem to not have access).

In T383047#11469444, @pmiazga wrote:

This is still happening, ~3.3k per in last 4 weeks:

Currently, ~5k in the last 4 weeks https://logstash.wikimedia.org/goto/ebe07e84765c3f2896ffc26e64e24166

What would be the next steps? I assume this error is something that can happen - sendmail can fail due to various reasons. @Tgr - do we do any kind of retries here?

I'll post a patch for an automatic retry.

Change #1275406 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1275406

Change #1275407 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] Mail: Retry transient mail() failures in UserMailer

https://gerrit.wikimedia.org/r/1275407

In T383047#11834654, @TAndic wrote:

Hi all, I just experienced this error as well as a few community members have reached out that they did not receive an email from me during the Community Insights distribution yesterday (however, they did receive an echo ping to check their email). When I tried to send a message directly via EmailUser on the project to one of those community members just now to troubleshoot, I got the Sendmail exited with non-zero exit code 74 message. I'm unsure of the scale of the unsent emails as I didn't get a failure for these with the EmailUser API script I was using, but it should be visible in yesterday's log (if such a log exists, I presume it does per @Krinkle's last message but I seem to not have access).

I was sending a single message and got that error too. Just noting that re-sending seemed to work - at least technically, can't say if the user actually got the message.

On April 29th, I also encountered the same "Sendmail exited with non-zero exit code 74" on the Estonian Wikipedia. The user to whom I sent the mail mentioned today that he got the letter, but not the notification on Wikipedia.

Change #1305272 had a related patch set uploaded (by Reedy; author: CDanis):

[mediawiki/core@REL1_43] UserMailer: hack: preserve multiple error callbacks

https://gerrit.wikimedia.org/r/1305272

Change #1305272 merged by jenkins-bot:

[mediawiki/core@REL1_43] UserMailer: hack: preserve multiple error callbacks

https://gerrit.wikimedia.org/r/1305272

Change #1305615 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305615

Change #1275406 merged by jenkins-bot:

[mediawiki/core@master] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1275406

Change #1305699 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_46] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305699

Change #1305713 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_45] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305713

Change #1305715 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_44] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305715

Change #1305699 merged by jenkins-bot:

[mediawiki/core@REL1_46] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305699

Change #1305716 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_43] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305716

Change #1305713 merged by jenkins-bot:

[mediawiki/core@REL1_45] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305713

Change #1305715 merged by jenkins-bot:

[mediawiki/core@REL1_44] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305715

Change #1305716 merged by jenkins-bot:

[mediawiki/core@REL1_43] Mail: Extract sendWithMailFunction() from UserMailer::send()

https://gerrit.wikimedia.org/r/1305716

Change #1305615 merged by jenkins-bot:

[mediawiki/core@master] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305615

Change #1305891 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_46] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305891

Change #1305892 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_45] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305892

Change #1305894 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_44] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305894

Change #1305895 had a related patch set uploaded (by Reedy; author: Kosta Harlan):

[mediawiki/core@REL1_43] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305895

Change #1305894 merged by jenkins-bot:

[mediawiki/core@REL1_44] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305894

Change #1305892 merged by jenkins-bot:

[mediawiki/core@REL1_45] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305892

Change #1305891 merged by jenkins-bot:

[mediawiki/core@REL1_46] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305891

Change #1305895 merged by jenkins-bot:

[mediawiki/core@REL1_43] Mail: Log PHP mail() send failures with recipient count

https://gerrit.wikimedia.org/r/1305895