Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix SMTP STARTTLS for Twisted >= 21.2.0 #5386

Open
TobiMayr opened this issue Feb 1, 2022 · 10 comments
Open

Fix SMTP STARTTLS for Twisted >= 21.2.0 #5386

TobiMayr opened this issue Feb 1, 2022 · 10 comments

Comments

@TobiMayr
Copy link

@TobiMayr TobiMayr commented Feb 1, 2022

Summary

The Mail settings don't have an option to choose a TLS version. Only to enforce upgrading connections to use SSL/TLS.
Mail servers like smtp.office365.com dropped support for TLS1.0 and TLS1.1 and now require TLS1.2: https://techcommunity.microsoft.com/t5/exchange-team-blog/new-opt-in-endpoint-available-for-smtp-auth-clients-still/ba-p/2659652

It seems that scrapy mail doesn't support TLS1.2. The error message (with MAIL_TLS = True):

[scrapy.mail] Unable to send mail: To=['user@gmail.com'] Cc=[] Subject="Test" Attachs=0- 421 b'4.7.66 TLS 1.0 and 1.1 are not supported. Please upgrade/update your client to support TLS 1.2. Visit https://aka.ms/smtp_auth_tls. [AM6P194CA0047.EURP194.PROD.OUTLOOK.COM]'

Motivation

Without TLS1.2 it's not possible anymore to send mails via smtp.office365.com. An option to use TLS1.2 would fix this issue

@wRAR
Copy link
Contributor

@wRAR wRAR commented Feb 1, 2022

Scrapy doesn't configure protocol versions explicitly when using ESMTPSenderFactory and neither does that class, so I would expect it to use global OpenSSL settings, including TLS 1.2 or 1.3 if supported. See https://twistedmatrix.com/documents/current/core/howto/ssl.html

Can you show the library versions printed by Scrapy?

@TobiMayr
Copy link
Author

@TobiMayr TobiMayr commented Feb 1, 2022

It depends on the version of scrapy and which library version it uses for OpenSSL?
I use scrapy 2.0.1.
Scrapy prints the following when starting a job:
[scrapy.utils.log] Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.2 (default, Feb 26 2020, 15:09:34) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Linux-4.15.0-66-generic-x86_64-with-glibc2.2.5
Thanks for your help!

@wRAR
Copy link
Contributor

@wRAR wRAR commented Feb 1, 2022

I suggest trying newer pyOpenSSL/cryptography/Twisted versions first.

@TobiMayr
Copy link
Author

@TobiMayr TobiMayr commented Feb 3, 2022

I upgraded the versions to the newest:
cryptography==36.0.1 pyOpenSSL==22.0.0 twisted==21.7.0

Now I get a new error:
502 Server does not support secure communication via TLS / SSL

So it definitely looks like the issue is my mail server now. I will close this feature request.
Thank you for your quick help

@TobiMayr TobiMayr closed this Feb 3, 2022
@TobiMayr
Copy link
Author

@TobiMayr TobiMayr commented Feb 4, 2022

Hi @wRAR I might still need your help.

The smtp server supports TLS1.2. I also tested this by creating a script with smtplib.
This script successfully sends my email using TLS1.2:

import smtplib, ssl

port = 587
smtp_server = 'smtp.office365.com'
sender_email = 'test1@test.com'
receiver_email = 'test2@test.com'
user = 'test3@test.de'
password = input('Type your password and press enter:')
message = 'test'

context = ssl.create_default_context()
context.minimum_version = ssl.TLSVersion.TLSv1_2
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(user, password)
    server.sendmail(sender_email, receiver_email, message)

I can't seem to get it to work with scrapy. I created a new scrapy project with a newer version(2.5.1) as well.
These are all my versions:
[scrapy.utils.log] INFO: Versions: lxml 4.7.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.9.7 (default, Sep 3 2021, 12:37:55) - [Clang 12.0.5 (clang-1205.0.22.9)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1m 14 Dec 2021), cryptography 36.0.1, Platform macOS-11.6.2-x86_64-i386-64bit

I still get the error:
[scrapy.mail] ERROR: Unable to send mail: To=['test@test.com'] Cc=[] Subject="tls-test" Attachs=0- 502 Server does not support secure communication via TLS / SSL

Do you have an idea what I could try next?

@wRAR
Copy link
Contributor

@wRAR wRAR commented Feb 4, 2022

Are you using MAIL_TLS and port 587?

@TobiMayr
Copy link
Author

@TobiMayr TobiMayr commented Feb 4, 2022

Both yes

@wRAR wRAR reopened this Feb 5, 2022
@wRAR wRAR changed the title Support TLS1.2 for SMPT Auth Support TLS1.2 for SMTP Auth Feb 5, 2022
@wRAR wRAR added bug and removed needs more info labels Feb 5, 2022
@wRAR
Copy link
Contributor

@wRAR wRAR commented Feb 5, 2022

I'll try to check this later

@TobiMayr
Copy link
Author

@TobiMayr TobiMayr commented Feb 9, 2022

Hi @wRAR, sorry to bother you.
Do you have some more ideas of what I could try?

@wRAR
Copy link
Contributor

@wRAR wRAR commented Feb 9, 2022

Actually "Server does not support secure communication via TLS / SSL" is a client error from Twisted and it's misleading because in fact it's the client that doesn't support STARTTLS.

Since Twisted 21.2.0 (twisted/twisted@abef121), ESMTPSenderFactory needs to be passed hostname to use STARTTLS. Unfortunately this isn't documented, but the whole of ESMTPSenderFactory isn't, the public interface seems to be sendmail() which we don't use.

The fix to this seems to be just passing hostname=self.smtphost to ESMTPSenderFactory() in MailSender._sendmail(), but only for Twisted 21.2.0 and above.

@wRAR wRAR changed the title Support TLS1.2 for SMTP Auth Fix SMTP STARTTLS for Twisted >= 21.2.0 Feb 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants