Message Boards Message Boards

0
|
9136 Views
|
11 Replies
|
0 Total Likes
View groups...
Share
Share this post:

How to optimize package reading in the cloud?

Posted 9 years ago
11 Replies
POSTED BY: Chad Knutson

Yes the issue is when FormLayoutFunction is defined. I tried Delayed[CloudGet[CloudObject[]]] because this implies it will only get evaluated when it is “externally” requested. But this does not work. It is very possible that the CloudGet package loading needs to happen somewhere else in the form but I haven’t got a clue. Hope you can help.

Thanks again Chad,

This opened my eyes. I did not connect package loading to storing function calls “persistently” in the cloud. This means you can load once and use in many places. So for example direct after a user logs into my web-application a package is loaded and all functions in it are available to be used in all Webforms the application exists of.

Oh, and your random number in api4 is not actually defined by your api function (you can verify by evaluating CloudImport on api4). Notice that the values of 'a' are the same as previous evaluations of api3. Kernel memory is not cleared immediately! If you wait awhile, you'll find that a is no longer defined when you evaluate api4.

POSTED BY: Chad Knutson

Yep, that is correct behavior. When you deploy api4, the definition of qpackage`rFunc is written to the cloud. You can do this with any function that you have defined locally.

Another simple example:

In[26]:= func[] := $CloudEvaluation

In[32]:= co = CloudDeploy[APIFunction[{}, func[] &]];
URLExecute[co, {}]
Out[33]= True

You can look at the file contents for the deployed api by calling CloudImport on the cloud object:

In[35]:= CloudImport[co, "Text"]
Out[35]= "func[] := $CloudEvaluation

APIFunction[{}, func[] & ]"
POSTED BY: Chad Knutson

I Changed your package to +1 in {x+1,a} as below

BeginPackage["qpackage`"]
Unprotect[rFunc];

rFunc::usage="does it keep same random number?"

Begin["`Private`"]
a=RandomReal[];
rFunc[x_] := {x+1,a}

End[]
EndPackage[]

api3 with load of package and api4 without loading the package

api3 = CloudDeploy[APIFunction[{"x" -> "Number"},
   (Get["qpackage`"];
     qpackage`rFunc[#x])
    &],
  CloudObject["api3"],
  Permissions -> "Public"]


api4 = CloudDeploy[APIFunction[{"x" -> "Number"},
   (qpackage`rFunc[#x])
    &],
  CloudObject["api4"],
  Permissions -> "Public"]

api3 delivers 1 added to 4 and 8 unique numbers

Table[URLExecute[api3, {"x" -> 4}], {8}] // Tally

{{{5, 0.977889}, 1}, {{5, 0.672346}, 1}, {{5, 0.600011}, 
  1}, {{5, 0.949759}, 1}, {{5, 0.811612}, 1}, {{5, 0.625906}, 
  1}, {{5, 0.530472}, 1}, {{5, 0.658866}, 1}

}

api4 delivers 1 added to each i but no unique random numbers anymore. They repeat every 4 times. So the function is active and can be used if defined delayed?

Table[URLExecute[api4, {"x" -> i}], {i, 1, 8}] 

{{2, 0.530472}, {3, 0.658866}, {4, 0.811612}, {5, 0.625906}, {6, 
  0.530472}, {7, 0.658866}, {8, 0.811612}, {9, 0.625906}}

Right, in this case, you could use Needs or Get @"qpackage`". However, you need to be careful with paths in the cloud. If the package isn't in your "Home" directory, I don't think it will be found by the api function.

I find that wrapping files in CloudObject is useful for me (I know that I mean cloud files).

I'm not sure what precisely you're asking about package loading. The kernel that evaluates the api loads the packages that are stored in the cloud file system. The only communication between the cloud and browser is the sending of the request (along with parameters, headers, etc), and the returning of the result.

I would guess that big packages would be faster than splitting the same contents across multiple files, but am not positive about this. I'd be surprised if it made a huge difference either way.

POSTED BY: Chad Knutson

One other test to see what is happening: edit the package file that is saved in the cloud. I changed the definition of rFunc to return {#x,a,1}. Then I evaluated the api:

In[6]:= URLExecute[CloudObject["api"], {"x" -> 3}]
Out[6]= {3, 0.876432, 1}

So the package definitions are clearly not saved in the deployed function.

POSTED BY: Chad Knutson

Chad, thanks for the reply,

In my code the &-sign was postfix over only the mypackage`myFunc part. This is why the CloudGet[] was loaded in the local $ContextPath.

Okay so the package is loaded every call to the API. In your code you write:

CloudGet[CloudObject["qpackage.wl"]]

It seems also possible to write:

Get["qpackage`"]

or

Needs["qpackage`"]

Does this imply that loading of the packages happens between the "instant" kernel and the diskfilesystem locally in the wolfram cloud servers. So internet communication to the browser client is not involved? It's not slowing things down from a browser-user point of view? So for a WL developer there is no need to chop a big package into smaller pieces?

I think this shows that the package is loaded every evaluation in the cloud: define a random number in the package, 'a'.

My package:

BeginPackage["qpackage`"]
Unprotect[rFunc];

rFunc::usage="does it keep same random number?"

Begin["`Private`"]
a=RandomReal[];
rFunc[x_] := {x,a}

End[]
EndPackage[]

Now deploy the function and evaluate as you have done:

CopyFile["qpackage.wl", CloudObject["qpackage.wl"]];
api = CloudDeploy[
  APIFunction[{"x" -> "Number"}, (CloudGet[CloudObject["qpackage.wl"]];
     qpackage`rFunc[#x]) &], CloudObject["api"], Permissions -> "Public"];

And evaluate a bunch of times

In[22]:= Table[URLExecute[api, {"x" -> 3}], {4}]
Out[22]= {{3, 0.425269}, {3, 0.898631}, {3, 0.808372}, {3, 0.146791}}

Note that the value of 'a' changes with each evaluation. If I do the same with Mathematica on my desktop, the value of 'a' is fixed.

In[2]:= << qpackage`
In[3]:= rFunc /@ ConstantArray[3, 4]
Out[3]= {{3, 0.495344}, {3, 0.495344}, {3, 0.495344}, {3, 0.495344}}

I cannot reproduce the loading of the package into the ContextPath on your local machine. Perhaps you evaluated the APIFunction locally before deploying? If not, please let me know the $Version you are using, and I'll investigate further.

POSTED BY: Chad Knutson
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract