Mail::Client::Send question

Added by Mario Diethelm almost 2 years ago

Hello,

as far as I understand the Mail::Client::Send routine uses SMTPS (SMTP encapsulated over TLS) to communicate with a smtp server to deliver e-mail messages (after having set transport encryption to StartTLS or TLS and assuming that the smtp server that you are using supports TLS).

How does the smpt server answer to the Mail::Client::Send request? I assume that it should be some kind of TCP message to an specific port (SMTPS to 587, 465 o 25 port?).

I am using Mail::Message and Mail::Client to send SMTP email messages through a smtp service(Mailgun, smtp.mailgun.org) in a Wt-4.9.1 httpd connector application running under Windows. In my development machine it works ok but when I test it in my integration environment (AWS instance), the Wt application crashes after establishing the connection with the smtp server (no other messages logged after "Mail.Client: connecting to 'smtp.mailgun.org:587'" message). Based on a dump report from the above mentioned application the problem is: "The thread tried to read from or write to a virtual address for which it does not have the appropriate access", so I guess that some kind of inbound permission is lacking in my integration environment.

I would appreciate your comments about it. Thanks a lot.

Best regards.

Mario.

Replies (9)

RE: Mail::Client::Send question - Added by Roel Standaert almost 2 years ago

I looked at the Wt::Mail::Client implementation and found some things:

If the resolver fails, Client::connect throws an exception instead of returning false (as documented)
If the client is not connected, and Client::send is used, this will result in a null pointer dereference, this can be due to:
- the user forgetting to call connect
- similarly, send being called after disconnect
- connect being used with TransportEncryption set to anything other than None, but Wt is built without OpenSSL (Wt will log "TLS requested, but Wt built without OpenSSL"). This may be what's causing your crash.

I think we should fix the client so:

Client::connect logs an error and returns false if the resolver fails (matching the documentation)
Client::send logs an error and returns false if the client is not connected

RE: Mail::Client::Send question - Added by Mario Diethelm almost 2 years ago

Hello Roel, thank you for your feedback.

I built Wt with OpenSSL (actually I built OpenSSL from the 3.0.7 version source code), so I think that TransportEncryption is not the problem. I set some flags in the Client.C source code using LOG_INFO to establish what pice of code is causing the crash in my application. Mail::Client::connect works fine and it actually returns true after connecting to 'smtp.mailgun.org:587', using TransportEncryption set to StartTLS. The crash happens inside the Mail::Client::send function member, specifically when it tries to write data: message.write(data). Are those the attachment data or the e-mail data?

I did some tests changing the security access of the attachment files, granting "Full control" for the group Users, but it did not solve the problem. The application crashes when Mail::Client::send tries to execute message.write(data). As I commented in my previous post, the same application works ok in my development machine so I will continue reviewing security settings in my integration environment. My guess is that some security configuration is not allowing message.write(data) to work correctly.

Thanks again. Best regards

Mario.

RE: Mail::Client::Send question - Added by Mario Diethelm almost 2 years ago

Hello again,

I did several tweaks regarding security configuration but none of them did solve the problem. I returned to my development machine where as I commented in my previous posts the "Debug" version works ok and I give a try to the "Release" version, that is the version that I have installed in my integration environment in AWS. I found the same problem in my development machine with the "Release" version. Every time that I add an attachment, the Mail::Client::send crashes. I re built the "Release" version eliminating all code optimizations (Visual Studio C++ compiler flags set to /Od and /Ob0). After that all works ok. The "Release" version was originally built using Visual Studio C++ compiler flags related to optimization set to /O2 and /Ob2.

I use optimization flags in other Wt applications and I have never found this behavior. Any idea why optimization could impact Mail::Client::send and specifically message.write(data)?

Best regards,

Mario.

RE: Mail::Client::Send question - Added by Roel Standaert almost 2 years ago

First of all, make sure you're not mixing debug and release builds. That causes problems on Windows.

Apart from that, release builds have a tendency to reveal undefined behaviour that goes unnoticed when using a debug build. You could try to enable address sanitizer to see if that reveals anything: https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170

RE: Mail::Client::Send question - Added by Mark Travis over 1 year ago

I believe I am having this problem as well. Local dev machine works fine to connect to my SMTP server via 465 TLS and send the verify new account e-mail. On AWS, it doesn't work. I allow ALL outbound traffic. I have a firewall rule in place to allow inbound from 465 (as well as 80 and 443) to facilitate any SMTP handshaking that is necessary.

I tried re-compiling for release -O0 as well as debug but nothing seems to work on AWS.

Mario, did you ever get this working?

RE: Mail::Client::Send question - Added by Mario Diethelm over 1 year ago

I Mark.

Yes. What I did was to to put the code to send an email in a separated module. This module is compiled with all optimization flags disables (/Od and /Ob0 in Visual C++). Doing that I can send SMTP emails without problems from the application hosted at AWS (I am using Mailgun as SMTP sending service. At AWS Route 53 hosted zone level I have registered all the record names to allow the traffic from and to my email domain name, myEmaildomain.net).

I am attaching the smtp send email module in case i could be useful for you. GetFileMimeType is a function to get file mime type from its extension (for instance, for a file named xxxxx.pdf, it returns "application/pdf"). smtpDomain is the email domain (myEmaildomain.net), smtpHost is the host name of the smtp email service provider (in my case, smtp.mailgun.org) and port is the smtp port (tipically 587).

Best regards.

sendsmtpemail.cpp (2.04 KB) sendsmtpemail.cpp

RE: Mail::Client::Send question - Added by Mark Travis over 1 year ago

Thanks Mario! I'll try this out and report back.

I also heard rumors that most email providers blacklist AWS EC2 servers from remote logins. I am checking with my email provider (encirca) to see if that is the case.

RE: Mail::Client::Send question - Added by Mark Travis over 1 year ago

The EC2 instance is not blacklisted, so that is not the cause.

I just tried something to see what differences between local dev environment and AWS might be. Interestingly enough, It appears that the AWS EC2 server logs on fine. The SMTP codes are identical.

(I modified Client.C in Wt\Mail to log the SMTP return code.)

Local copy:
[2023-Aug-31 15:21:52.139] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: using 'smtp.us.appsuite.cloud:465' (from smtp-host and smtp-port properties) as SMTP host"
[2023-Aug-31 15:21:52.139] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: connecting to 'smtp.us.appsuite.cloud:465'"
[2023-Aug-31 15:21:55.558] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '220' "
[2023-Aug-31 15:22:03.848] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '250' "
[2023-Aug-31 15:22:10.695] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '334' "
[2023-Aug-31 15:22:15.034] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '334' "
[2023-Aug-31 15:22:15.949] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '235' "
[2023-Aug-31 15:22:22.847] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '250' "
[2023-Aug-31 15:22:22.998] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '250' "
[2023-Aug-31 15:22:23.474] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '354' "
[2023-Aug-31 15:22:23.657] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '250' "
[2023-Aug-31 15:22:23.901] 44853 [/ aK7mS1Ca5p8ExzwK] [info] "Mail.Client: SMTP return code: '221' "
127.0.0.1 - - [2023-Aug-31 15:22:23.911] "POST /?wtd=aK7mS1Ca5p8ExzwK HTTP/1.1" 200 48 [2023-Aug-31 15:22:23.911] 44853 - [info] "WebRequest: took 39508.1 ms"
127.0.0.1 - - [2023-Aug-31 15:23:14.391] "POST /?wtd=aK7mS1Ca5p8ExzwK HTTP/1.1" 200 0 [2023-Aug-31 15:23:14.391] 44853 - [info] "WebRequest: took 50471.4 ms"
127.0.0.1 - - [2023-Aug-31 15:24:05.389] "POST /?wtd=aK7mS1Ca5p8ExzwK HTTP/1.1" 200 0 [2023-Aug-31 15:24:05.389] 44853 - [info] "WebRequest: took 50997.8 ms"
127.0.0.1 - - [2023-Aug-31 15:24:56.393] "POST /?wtd=aK7mS1Ca5p8ExzwK HTTP/1.1" 200 0 [2023-Aug-31 15:24:56.393] 44853 - [info] "WebRequest: took 51004.7 ms"
127.0.0.1 - - [2023-Aug-31 15:25:47.394] "POST /?wtd=aK7mS1Ca5p8ExzwK HTTP/1.1" 200 0 [2023-Aug-31 15:25:47.394] 44853 - [info] "WebRequest: took 51002.4 ms"

AWS EC2:
[2023-Aug-31 19:26:50.711] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: using 'smtp.us.appsuite.cloud:465' (from smtp-host and smtp-port properties) as SMTP host"
[2023-Aug-31 19:26:50.711] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: connecting to 'smtp.us.appsuite.cloud:465'"
[2023-Aug-31 19:26:50.872] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '220'"
[2023-Aug-31 19:26:50.891] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '250'"
[2023-Aug-31 19:26:50.908] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '334'"
[2023-Aug-31 19:26:50.933] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '334'"
[2023-Aug-31 19:26:51.286] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '235'"
[2023-Aug-31 19:26:51.592] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '250'"
[2023-Aug-31 19:26:51.681] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '250'"
[2023-Aug-31 19:26:51.706] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '354'"
[2023-Aug-31 19:26:51.743] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '250'"
[2023-Aug-31 19:26:51.769] 23828 [/ RZTF85O35jyKUEQj] [info] "Mail.Client: SMTP return code: '221'"
172.31.8.89 - - [2023-Aug-31 19:26:51.770] "POST /?wtd=RZTF85O35jyKUEQj HTTP/1.1" 200 47
[2023-Aug-31 19:26:51.770] 23828 - [info] "WebRequest: took 1078.3 ms"
[2023-Aug-31 19:26:54.176] 23828 [/ ESZonsE1swIaBmYE] [info] "WebController: timeout: expiring"
[2023-Aug-31 19:26:54.176] 23828 [/ ESZonsE1swIaBmYE] [info] "Wt: session destroyed (#sessions = 2)"
[2023-Aug-31 19:26:54.460] 23828 - [info] "Wt: session created (#sessions = 3)"
[2023-Aug-31 19:26:54.460] 23828 [/ VQNnx5cKFuEPBAGA] [info] "WEnvironment: UserAgent: ELB-HealthChecker/2.0"
172.31.16.238 - - [2023-Aug-31 19:26:54.461] "GET / HTTP/1.1" 200 2073
[2023-Aug-31 19:26:54.461] 23828 - [info] "WebRequest: took 1.072 ms"
[2023-Aug-31 19:27:04.177] 23828 [/ VQNnx5cKFuEPBAGA] [info] "WebController: timeout: expiring"
[2023-Aug-31 19:27:04.177] 23828 [/ VQNnx5cKFuEPBAGA] [info] "Wt: session destroyed (#sessions = 2)"
[2023-Aug-31 19:27:05.950] 23828 - [info] "Wt: session created (#sessions = 3)"

RE: Mail::Client::Send question - Added by Mark Travis over 1 year ago

For the record, I ran address sanitizer in debug mode on the server (Ubuntu 22LTS) and nothing came of it for this problem. I built with OpenSSL 3 and am delivering (successfully) over https on the AWS server via 443.

I think the only explainable difference between what works and what doesn't is I'm running http on my dev machine and https on AWS.

I'm getting a hunch that there is something in the encoding portion. I'm going to look through the AuthService.verifyEmailAddress and the Wt/Mail code to see if the code does something similar to what Mario did in the following block:

    if (attachment.length()) {
        try {
            if (sourceFile.open(attachment.c_str(), ios::in | ios::binary)) {
                string mimeType(GetFileMimeType(db, attachment));
                mailMessage.addAttachment(mimeType, attachment, &istream(&sourceFile));
            }
            else
                okToGo = false;
        }
        catch (...) {
            okToGo = false;
        }
    }

I'm hypothesizing at this point that the optimized AWS code is not setting the mimeType or istream properties correctly, but I have no idea if and why that might be the case. And that doesn't include any hypothesis for http vs https.

(1-9/9)

Project

General

Profile

Wt