Message Boards Message Boards

1
|
13406 Views
|
11 Replies
|
10 Total Likes
View groups...
Share
Share this post:

Mailgun API - Problem with encoding in HTTPRequest

Rodrigo Murta, Looqbox
Posted 3 years ago

I'm working with a Mailgun API update for Mathematica, based this post. After config the API account, emails works so far so good, but I discovered that when I send an attached file, the diacritics get broked.

Here is the code:

HTTPRequest["https://api.mailgun.net/v3/mydomain.com/messages",
      <|
      Method->"POST"
      ,"User"->"api:MY_API_KY"
      ,"Body" -> {
          "from" -> "test@mydomain.com"
         ,"to" -> "me@me.com"
         ,"subject" -> "With Attachment"
         ,"text" -> "Diacritics test: àáâãäåçèéêëìíî"
         , "attachment" -> File["m.JPG"]
         }
        |>
     ]//URLRead[#, "Body"]&

Here are the result, commenting and uncommenting the line: , "attachment" -> File["m.JPG"]

Diacritics Problem

I tried to add the header below, without success too:

,"Headers"-> {"Content-type" -> "multipart/form-data; charset=utf-8"}

Any help is welcome

UPDATE 01

Using the old function URLFetch, I have problem with and without sending a file.

URLFetch["https://api.mailgun.net/v3/mydomain.com/messages",
      "Method"->"POST"
      ,"MultipartElements" -> {
         {"from","text/plain"} -> "test@mydomain.com"
         ,{"to","text/plain"} -> "me@me.com"
         ,{"subject","text/plain"} -> "Hello YOUR-NAME"
         ,{"text","text/plain"} -> "Diacritics test: àáâãäåçèéêëìíî"
          }
      ,"Username"->"api"
      ,"Password"->"MY_API_KY"
  ]

UPDATE 02

I created a better toy code below using https://httpbin.org, so anyone can execute the test.
POSTED BY: Rodrigo Murta
11 Replies
Sort By:

Hi Rodrigo,

Thank you for starting this discussion.

You might be interested to know that we are working on a framework to create connections from Wolfram Language to external APIs in an easy way and share them as resources.

Please let me know if you want to have early access to test it. Thanks!
POSTED BY: Christian Pasquel
Mike Besso
Mike Besso
Posted 3 years ago

@Christian:

I volunteer to help test that new API. I'm spending too much time trying to hack something together instead of working on the problem I'm trying to solve.

THANKS
POSTED BY: Mike Besso

@Mike,

Thanks Mike. Do you mind leaving your info here in this form?

https://www.wolframcloud.com/obj/christianp/tmp/SCFrameworkBetaTestingList

Thanks
POSTED BY: Christian Pasquel

Cool Jesse! Using ExportString worked! Both for HTTPRequest and the old URLFetch form!

ExportString["Diacritics test: àáâãäåçèéêëìíî", "Text", CharacterEncoding -> "UTF-8"]

Thanks for your investigation and time!
POSTED BY: Rodrigo Murta

Here is a better toy code using httpbin.org (cool service!).

HTTPRequest["https://httpbin.org/anything",
      <|
      Method->"POST"
      ,"Body" -> {
          "text" -> "Diacritics test: àáâãäåçèéêëìíî"
          , "attachment" -> File[Export["image.png", Graphics@Circle[], ImageSize-> 1]]
         }
        |>
     ]//URLRead[#, "BodyBytes"]&//FromCharacterCode

As you can see, "text" is broke:

text: "Diacritics test: \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd"

If you comment the attachment part, we get it right:

text: "Diacritics test: \u00e0\u00e1\u00e2\u00e3\u00e4\u00e5\u00e7\u00e8\u00e9\u00ea\u00eb\u00ec\u00ed\u00ee"
POSTED BY: Rodrigo Murta

I think the reality is actually a bit more subtle than what httpbin shows. The raw HTTP request without "attachment" looks like this:

POST /anything HTTP/1.1
Host: httpbin.org
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
User-Agent: Wolfram HTTPClient 12.2
Connection: close
Content-Length: 112

text=Diacritics%20test%3A%20%C3%A0%C3%A1%C3%A2%C3%A3%C3%A4%C3%A5%C3%A7%C3%A8%C3%A9%C3%AA%C3%AB%C3%AC%C3%AD%C3%AE
(...)

The "text" parameter is sent as a URL-encoded query string. If other string parameters are added to the request, they're appended to this single query string. There's no binary encoding issue because the extended ASCII characters are simply percent-encoded.

With "attachment", it looks like this:

POST /anything HTTP/1.1
Host: httpbin.org
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: Wolfram HTTPClient 12.2
Connection: close
Content-Length: 778
Content-Type: multipart/form-data; boundary=------------------------a073b7b6522cb4a2

--------------------------a073b7b6522cb4a2
Content-Disposition: form-data; name="text"
Content-Type: 

Diacritics test: ��������������
--------------------------a073b7b6522cb4a2
Content-Disposition: form-data; name="attachment"; filename="image.png"
Content-Type: image/png
(...)

Now it's a multipart form request. New string parameters are added as independent parts in the request body. Note that the special characters are sent as single ASCII bytes (0xE0, 0xE1, etc.) instead of as UTF-8-encoded chunks. It seems like a bug (although I'm not certain) that the string parameters are being given empty Content-Types, which the solution I gave addresses, but that's not helping with the underlying problem. Try this:

HTTPRequest[
   "https://httpbin.org/anything", <|
    Method -> "POST", 
    "Body" -> {"text" -> 
       ExportString["Diacritics test: àáâãäåçèéêëìíî", "Text", 
        CharacterEncoding -> "UTF-8"], 
      "attachment" -> 
       File[Export["image.png", Graphics@Circle[], 
         ImageSize -> 1]]}|>] // 
  URLRead[#, "BodyBytes"] & // FromCharacterCode

(The CharacterEncoding -> "UTF-8" setting isn't necessary, but it's good to be explicit.)
This returns:

(...)
text: "Diacritics test: \u00e0\u00e1\u00e2\u00e3\u00e4\u00e5\u00e7\u00e8\u00e9\u00ea\u00eb\u00ec\u00ed\u00ee"
(...)

And the raw request body looks like:

POST /anything HTTP/1.1
Host: httpbin.org
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: Wolfram HTTPClient 12.2
Connection: close
Content-Length: 787
Content-Type: multipart/form-data; boundary=------------------------3b920b8202986fa1

--------------------------3b920b8202986fa1
Content-Disposition: form-data; name="text"
Content-Type: 

Diacritics test: àáâãäåçèéêëìíî
--------------------------3b920b8202986fa1
Content-Disposition: form-data; name="attachment"; filename="image.png"
Content-Type: image/png
(...)

You may wish to specify an explicit Content-Type using an association as in my previous reply, although it may not matter to the API you're talking to.

If this doesn't work I can try getting my own Mailgun API key to test with.

P.S. https://webhook.site/ is useful for debugging issues like this (although I still had to look at the raw request data to see what was going on).
POSTED BY: Jesse Friedman

Like Arnoud, I can't test directly with the API you're using, but some local experimentation suggests this might work:

In[2]:= HTTPRequest[
"https://api.mailgun.net/v3/mydomain.com/messages", <|
Method -> "POST", "User" -> "api:MY_API_KY", 
"Body" -> {"from" -> "test@mydomain.com", "to" -> "me@me.com", 
"subject" -> "With Attachment", 
"text" -> <|
"Content" -> 
StringToByteArray["Diacritics test: àáâãäåçèéêëìíî"], 
"MIMEType" -> "text/plain;charset=utf-8"|>, 
"attachment" -> File[FindFile["ExampleData/mathematica.pdf"]]}|>,
CharacterEncoding -> "UTF8"] // URLRead[#, "Body"] &

When character encodings are an issue I generally find it better to manually encode the relevant portions of a request with StringToByteArray. The documentation for HTTPRequest suggests a ByteArray should work, but unfortunately it doesn't:

In[2]:= HTTPRequest[
"https://api.mailgun.net/v3/mydomain.com/messages", <|
Method -> "POST", "User" -> "api:MY_API_KY", 
"Body" -> {"from" -> "test@mydomain.com", "to" -> "me@me.com", 
"subject" -> "With Attachment", 
"text" -> <|
"Content" -> 
StringToByteArray["Diacritics test: àáâãäåçèéêëìíî"], 
"MIMEType" -> "text/plain;charset=utf-8"|>, 
"attachment" -> File[FindFile["ExampleData/mathematica.pdf"]]}|>,
CharacterEncoding -> "UTF8"] // URLRead[#, "Body"] &

During evaluation of In[2]:= General::erropts: The value{{from,Automatic} ->test@mydomain.com, {to,Automatic} ->me@me.com, {subject,Automatic} -> With Attachment, {text,text/plain;charset=utf-8,58b41feb-7b98-458f-afc8-8fd836c4461e} ->ByteArray[45 bytes], {attachment,Automatic}->File[/Applications/Mathematica 12.2.app/Contents/Documentation/English/System/ExampleData/mathematica.pdf]}specified for the option MultipartElements is invalid.

Out[2]= Failure["ConnectionFailure", 
Association[
"URL" -> "https://api.mailgun.net/v3/mydomain.com/messages",
"MessageTemplate" -> "Unable to perform the request.", 
"MessageParameters" -> Association[], 
"HTTPRequest" -> HTTPRequest[
"https://webhook.site/efe31524-3df5-4d3f-8d80-b5be37df83b0", 
Association[
Method -> "POST", "User" -> "api:MY_API_KY", 
"Body" -> {"from" -> "test@mydomain.com", "to" -> "me@me.com", 
"subject" -> "With Attachment", 
"text" -> Association[
"Content" -> ByteArray[{68, 105, 97, 99, 114, 105, 116, 105, 
99, 115, 32, 116, 101, 115, 116, 58, 32, 195, 160, 195, 
161, 195, 162, 195, 163, 195, 164, 195, 165, 195, 167, 195,
168, 195, 169, 195, 170, 195, 171, 195, 172, 195, 173, 
195, 174}], "MIMEType" -> "text/plain;charset=utf-8"], 
"attachment" -> File["/Applications/Mathematica12.2.app/Contents/Documentation/English/System/ExampleData/mathematica.pdf"]}], CharacterEncoding -> "UTF8"], "Counter" -> 1]]
POSTED BY: Jesse Friedman

I really appreciate your attention Jesse. Unfortunately, if I specify the type as you describe, I get de msg below:

"{
  \"message\": \"'text' parameter is not a string\"
}"

What is very strange if MIMEType is: "text/plain;charset=utf-8"

Below I did a better testable toy code.
POSTED BY: Rodrigo Murta

Hi Arnoud, thanks for your suggestion. But in this case the problem is not to Read the request, the problem is that it sends the text data wrong when URLRead is executed. It's like when I pass the File, some encode changes.
POSTED BY: Rodrigo Murta

Ok, I can't think of a solution (yet).
POSTED BY: Arnoud Buzing

I can't reproduce this (because of the API key required), but try doing it this way (I have has success with it in other cases):

request = HTTPRequest[...]
response = URLRead[request]
ImportByteArray[response["BodyByteArray"]]

(Or something equivalent based on working with the ByteArray and not text directly.)
POSTED BY: Arnoud Buzing
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract