How to optimize package reading in the cloud?

pietervanbijnen · February 18, 2015, 2:53pm

I have several FormFunctions cloud deployed and all together they are forming an application. The function parts are getting very long and have shared parts with the other forms. So the logical thing to do is to put recycled parts in a package and load the package in each web form to enable using the functions in it.

I was searching for a mechanism to prevent loading the package each time the form is populated and submitted. It seems there is no efficiency because the $SessionID changes for each refresh. A new kernel results in a need to load the package from the start. So if a package is long this delays the loading of the page. It looks like 4 $SessionIDs are recycled for all my form actions.

In a desktop session I am able to keep one $SessionID and with Needs[package`] I can prevent a second load.

What options do I have if I have 8 forms and 4 of them have 3 shared functions and the other 4 have 5 other functions to share. Do I need to create 2 package files for more efficient loading.

One other option I was pondering is to ask the 8 webforms to submit their results to one and the same API. This is possible but every API request also has its own $SessionID cycling over 4 IDs.

Ive also tested if within 1 browser session it would keep the same $SessionID and therefore it would not load the package every time. But this does not seem to be so.

The question is what the mechanism is in loading (partial) packages. Can you tell me what the big picture behind my struggles is? I seem to have a few black holes in my thinking.

Below my testing code

BeginPackage[“mypackage`”]

Unprotect[myFunc];

myFunc::usage="provides session ID"

Begin["`Private`"]

myFunc[x_] := $SessionID

End[]

Protect[myFunc];

EndPackage[]

Copy it in the cloud

CopyFile["e:\\folder\\mypackage.wl", CloudObject["mypackage.wl"]]

API definition in desktop

api = CloudDeploy[
  APIFunction[{"x" -> "Number"},
   CloudGet["mypackage.wl"];
   mypackage`myFunc[#x] &]]

Search for matching session IDs

Table[URLExecute[api, {"x" -> 3}], {16}] // Tally

When I check the $ContextPath it seems the mypackage context is also loaded in my desktop kernel. CloudGet is defined as reading a cloud object and evaluating each expression. But this does not tell me how the API on the cloudserver loads the package. Always new or is there a Needs[] mechanism at work.

chadk · February 18, 2015, 5:50pm

I think this shows that the package is loaded every evaluation in the cloud: define a random number in the package, ‘a’.

My package:

BeginPackage["qpackage`"]
Unprotect[rFunc];

rFunc::usage="does it keep same random number?"

Begin["`Private`"]
a=RandomReal[];
rFunc[x_] := {x,a}

End[]
EndPackage[]

Now deploy the function and evaluate as you have done:

CopyFile["qpackage.wl", CloudObject["qpackage.wl"]];
api = CloudDeploy[
  APIFunction[{"x" -> "Number"}, (CloudGet[CloudObject["qpackage.wl"]];
     qpackage`rFunc[#x]) &], CloudObject["api"], Permissions -> "Public"];

And evaluate a bunch of times

In[22]:= Table[URLExecute[api, {"x" -> 3}], {4}]
Out[22]= {{3, 0.425269}, {3, 0.898631}, {3, 0.808372}, {3, 0.146791}}

Note that the value of ‘a’ changes with each evaluation. If I do the same with Mathematica on my desktop, the value of ‘a’ is fixed.

In[2]:= << qpackage`
In[3]:= rFunc /@ ConstantArray[3, 4]
Out[3]= {{3, 0.495344}, {3, 0.495344}, {3, 0.495344}, {3, 0.495344}}

I cannot reproduce the loading of the package into the ContextPath on your local machine. Perhaps you evaluated the APIFunction locally before deploying? If not, please let me know the $Version you are using, and I’ll investigate further.

chadk · February 18, 2015, 7:17pm

One other test to see what is happening: edit the package file that is saved in the cloud. I changed the definition of rFunc to return {#x,a,1}. Then I evaluated the api:

In[6]:= URLExecute[CloudObject["api"], {"x" -> 3}]
Out[6]= {3, 0.876432, 1}

So the package definitions are clearly not saved in the deployed function.

pietervanbijnen · February 18, 2015, 7:48pm

Chad, thanks for the reply,

In my code the &-sign was postfix over only the mypackage`myFunc part. This is why the CloudGet[] was loaded in the local $ContextPath.

Okay so the package is loaded every call to the API. In your code you write:

CloudGet[CloudObject[“qpackage.wl”]]

It seems also possible to write:

Get[“qpackage`”]

or

Needs[“qpackage`”]

Does this imply that loading of the packages happens between the “instant” kernel and the diskfilesystem locally in the wolfram cloud servers. So internet communication to the browser client is not involved? It’s not slowing things down from a browser-user point of view? So for a WL developer there is no need to chop a big package into smaller pieces?

chadk · February 18, 2015, 8:22pm

Right, in this case, you could use Needs or Get @“qpackage`”. However, you need to be careful with paths in the cloud. If the package isn’t in your “Home” directory, I don’t think it will be found by the api function.

I find that wrapping files in CloudObject is useful for me (I know that I mean cloud files).

I’m not sure what precisely you’re asking about package loading. The kernel that evaluates the api loads the packages that are stored in the cloud file system. The only communication between the cloud and browser is the sending of the request (along with parameters, headers, etc), and the returning of the result.

I would guess that big packages would be faster than splitting the same contents across multiple files, but am not positive about this. I’d be surprised if it made a huge difference either way.

pietervanbijnen · February 18, 2015, 9:26pm

I Changed your package to +1 in {x+1,a} as below

BeginPackage["qpackage`"]
Unprotect[rFunc];

rFunc::usage="does it keep same random number?"

Begin["`Private`"]
a=RandomReal[];
rFunc[x_] := {x+1,a}

End[]
EndPackage[]

api3 with load of package and api4 without loading the package

api3 = CloudDeploy[APIFunction[{"x" -> "Number"},
   (Get["qpackage`"];
     qpackage`rFunc[#x])
    &],
  CloudObject["api3"],
  Permissions -> "Public"]


api4 = CloudDeploy[APIFunction[{"x" -> "Number"},
   (qpackage`rFunc[#x])
    &],
  CloudObject["api4"],
  Permissions -> "Public"]

api3 delivers 1 added to 4 and 8 unique numbers

Table[URLExecute[api3, {"x" -> 4}], {8}] // Tally

{{{5, 0.977889}, 1}, {{5, 0.672346}, 1}, {{5, 0.600011}, 
  1}, {{5, 0.949759}, 1}, {{5, 0.811612}, 1}, {{5, 0.625906}, 
  1}, {{5, 0.530472}, 1}, {{5, 0.658866}, 1}

}

api4 delivers 1 added to each i but no unique random numbers anymore. They repeat every 4 times. So the function is active and can be used if defined delayed?

Table[URLExecute[api4, {"x" -> i}], {i, 1, 8}] 

{{2, 0.530472}, {3, 0.658866}, {4, 0.811612}, {5, 0.625906}, {6, 
  0.530472}, {7, 0.658866}, {8, 0.811612}, {9, 0.625906}}

chadk · February 18, 2015, 9:42pm

Yep, that is correct behavior. When you deploy api4, the definition of qpackage`rFunc is written to the cloud. You can do this with any function that you have defined locally.

Another simple example:

In[26]:= func[] := $CloudEvaluation

In[32]:= co = CloudDeploy[APIFunction[{}, func[] &]];
URLExecute[co, {}]
Out[33]= True

You can look at the file contents for the deployed api by calling CloudImport on the cloud object:

In[35]:= CloudImport[co, "Text"]
Out[35]= "func[] := $CloudEvaluation

APIFunction[{}, func[] & ]"

chadk · February 18, 2015, 9:46pm

Oh, and your random number in api4 is not actually defined by your api function (you can verify by evaluating CloudImport on api4). Notice that the values of ‘a’ are the same as previous evaluations of api3. Kernel memory is not cleared immediately! If you wait awhile, you’ll find that a is no longer defined when you evaluate api4.

pietervanbijnen · February 18, 2015, 10:39pm

Thanks again Chad,

This opened my eyes. I did not connect package loading to storing function calls persistently in the cloud. This means you can load once and use in many places. So for example direct after a user logs into my web-application a package is loaded and all functions in it are available to be used in all Webforms the application exists of.

pietervanbijnen · February 24, 2015, 3:48pm

Package loading (cloud kernel) and code embedding (local kernel) revisited:

An APIfunction loads a package in the cloud by including CloudGet[Cloudobject[package]] in the pure function argument. So a form should work in just the same way.
A form consists of a form, and when submitted a reply. We need a navigation bar on the form and on the reply page. I created a package that generates a simple navigation bar.

Because of the “form” and the “reply” page are in the cloud processed by different kernels we need to load the package 2 times.

It is my intention to link the code and not embed it into the cloud object. Therefore it is necessary to make sure the 2 CloudGet[CloudObject[]] commands are NOT evaluated by the local kernel.

If this would happen the package would load in the local kernel, the context is added to $Contextpath, resulting in a --second-- CloudDeploy evaluation with embedded code.

The CloudGet[CloudObject[“rkc.wl”]] in the FormFunction does not seem to be the problem. The contents of ()& is not evaluated by the local kernel.

The second CloudGet[CloudObject[“rkc.wl”]] in the rhs of the FormLayOutFunction is a problem because it is evaluated by the local kernel. Because the package gets loaded rkc is put on the

contextpath and the --second-- clouddeploy evaluation results in all code embedded in the cloudobject.

The question is “how to load a package by the cloudkernel in the form submit side.”
I tried a few things like RuleDelayed, a Delayed on the CloudGet, and changing Function[] in (( )&) but no luck.

Here is the form code

tst = CloudDeploy[
  FormFunction[ 
   FormObject[
    <| {"firstname", "First Name:"} -> "String",
     {"lastname", "Last Name:"} -> "String",
     {"wlexperience", "WL experience"} -> <|
       "Interpreter" -> Restricted["Number", {1, 10}], 
       "Hint" -> "min is 1 and max is 10"|>
     |>,
    AppearanceRules -> <|
      "SubmitLabel" -> "go to database"|> 
    ],
   (CloudGet[CloudObject["rkc.wl"]]; 
     (*URL database fetchremoved *)
     ExportString[
      StringJoin["<center>",
       ExportString[
        Grid[{
          {rkc`inav["a_adddata"]},
          {Style["Your record has been added", 22]}}],
        "HTMLFragment"
        ], "</center>"]
      , "HTMLFragment", "FullDocument" -> True]
     ) &,
   "HTML",
   FormLayoutFunction ->
    Function[ 
     CloudGet[CloudObject["rkc.wl"]];
     Grid[{
       {Style["Please enter data", 28], SpanFromLeft},
       {rkc`inav["adddata"] , SpanFromLeft},
       {#["firstname", "Label"], #["firstname", "Control"]},
       {#["lastname", "Label"], #["lastname", "Control"]},
       {#["wlexperience", "Label"], #["wlexperience", "Control"]}
       }, Alignment -> Center]
     ],
   FormTheme -> "Blue"],
  "/adddata", Permissions -> "Public"]

check the ContextPath to find rkc on it.

Main Problem : rkc is now visable! When you evaluate the CloudDeploy for a second time this results in unwanted embedding of code. (use CloudImport to check).

CloudImport[tst, "Text"]

Here is the package. The function inav[] is used in the CloudDeploy. rkc`inav[“adddata”] results in 2 buttons with Home and Query labels on the buttons plus a timestamp.

rkc`inav["adddata"] (*to test package *)

Put this in a package and copy it to the cloud with :

CopyFile[“e:\folder\rkc.wl”, CloudObject[“rkc.wl”]];

BeginPackage["rkc`"]
Unprotect[inav];

inav::usage = "inav[pagename] pure expression returned"


Begin["`Private`"]
inav[pagename_] := 
 Row[mkb[#] & /@ 
   Transpose[{nb[pagename], Lookup[lbl, #] & /@ nb[pagename]}]]
mkb[{lnk_String, lbl_String}] :=
 Hyperlink[
  Graphics[{EdgeForm[{Thin, Gray}],
    LightBlue,
    Rectangle[{0, 0}, {4, 1}], Black,
    Text[Style[lbl, 11, Black], {0.5, .5}, {Left, Center}]},
   PlotRangePadding -> None, ImagePadding -> 0, 
   ImageSize -> {120, 30}], lnk]
dd := DateString[{"Time"}]
nb := <|"splash" -> {"rkcform1", "rkcform2"}, 
  "adddata" -> {"splash1", "readdata1"}, 
  "a_adddata" -> {"splash1", "adddata1", "readdata1"}, 
  "readdata" -> {"splash", "adddata"}, 
  "a_readdata" -> {"splash", "adddata", "readdata"}|>
lbl := <|"login" -> "enter" <> dd, "splash1" -> "Home" <> dd, 
  "adddata1" -> "Add" <> dd, "a_adddata1" -> "Add result" <> dd, 
  "readdata1" -> "Query" <> dd, "a_readdata1" -> "Q result" <> dd|>
End[]
EndPackage[]

Note : when investigating a solution make sure your browser is not caching anything. This will trick you into thinking it worked but it served you old data. (in chrome ctrl - shft - I then rightclick the reload button)

finally the message above is not scrambled …

chadk · February 27, 2015, 4:30pm

If I understand your new question correctly, the issue is when the FormLayoutFunction is defined. I have assumed that it must be defined (evaluated) when the FormFunction is deployed to the cloud. But I don’t know for sure. I have asked the developer about this.

pietervanbijnen · March 2, 2015, 5:43pm

Yes the issue is when FormLayoutFunction is defined. I tried Delayed[CloudGet[CloudObject[]]] because this implies it will only get evaluated when it is externally requested. But this does not work. It is very possible that the CloudGet package loading needs to happen somewhere else in the form but I havent got a clue. Hope you can help.

Topic		Replies	Views
Clean package update for API/FormFunctions on Wolfram Cloud? Questions software-development , packages-and-paclets , wolfram-cloud	6	15770	October 21, 2021
CloudDeploy with a Package. ExternalBundle? Questions packages-and-paclets , wolfram-cloud	8	18360	September 3, 2016
Return the Wolfram CloudDeploy [ ] on the same page as the form Questions mathematica , wolfram-cloud , wolfram-engine	0	2676	August 8, 2014
Working with FormFunction Questions wolfram-cloud , wolfram-engine	8	17735	May 23, 2015
Instant API works in Notebook but fails when deployed. Questions wolfram-cloud	12	11237	October 30, 2014

How to optimize package reading in the cloud?

Related topics