Mark,
Your diagnosis is exactly right: the problem is related to URL encoding.
When you pass a URL (which itself may contain encoded characters like %20 for a space) as the value of a query parameter (?url=...), the external client (like a browser) often fails to correctly encode the entire URL string, or the Wolfram Cloud sometimes double-decodes the parameter.
The key difference is that your local Wolfram Language session is often more forgiving with Import, but the automated Cloud API environment requires the string to be a perfectly structured URL.
Solution
The most reliable way to fix this issue is to explicitly use the URLDecode function inside your APIFunction. This guarantees that whatever string the client sends as the value of the url parameter is decoded back into the original intended URL string before Import tries to fetch it.
urlScraperFixed = APIFunction[
{"url" -> "String"},
Import[URLDecode[#url], "Plaintext"] &
];
Explanation
When a client calls your endpoint with the PDF URL: .../urlScraper? url=https://.../Perfect%20Chocolate%20Chip%20Cookies%20Recipe%20-%20NYT%20Cooking.pdf
The Wolfram Cloud's APIFunction receives the value of the url parameter, which is the full URL string (including the %20).
We apply URLDecode to the input #url This step is crucial because it ensures that any extraneous encoding or malformed characters introduced by the client's API call are removed, leaving a clean, valid URL string for the next function.
The cleaned URL string is then passed to Import, which now successfully recognizes the resource and retrieves the plaintext content of the PDF.
Deployment
Redeploy the corrected function using the same deployment command:
CloudDeploy[urlScraperFixed, "urlScraper", Permissions -> "Public", CloudObjectNameFormat -> "CloudUserUUID"]
This fixed API function should now successfully handle both simple web URLs and complex URLs like your PDF link when called from outside the Wolfram Language.