Group Abstract Group Abstract

Message Boards Message Boards

3
|
12.1K Views
|
6 Replies
|
6 Total Likes
View groups...
Share
Share this post:

Clean package update for API/FormFunctions on Wolfram Cloud?

Posted 8 years ago

Cross posted on mathematica.stackexchange

TLDR

There is a pool of kernels/sessions available for each user, you don't have control over the pool, only over specific kernel you currently evaluate in. Re-upload of a package should be followed by quit on every kernel where previous version of a package was used. I failed to find tools for that.

Background

I'm developing a package which I want to deploy to my Wolfram Public Cloud's account and I want a set of APIFunctions/FormFunctions/etc to be able to use it.

  • PacletManager has limited functionality so I'm just uploading package's archive and extracting it to WPC's $UserBaseDirectory / Applications /.

  • APIFunctions and friends have form:

     APIFunction[{}, (Needs["TestPackage`"];Symbol["TestPackage`api2"][...]) &]
    

I use Symbol because otherwise TestPackage` definitions are uploaded and I want to avoid that. There should be one code source, the package in $UserBaseDirectory.

Problem

The problem is that kernels' management on WPC is a closed black box. If you call the api twice, each time it uses one of the kernels in the pool. It also applies to CloudEvaluate etc.

$UserBaseDirectory is shared but Get will find the current version ony for current kernel.

If in another kernel TestPackage` was loaded earlier, Needs will not load the current version there :-/ I don't wan't to Get in my APIFunctions, a proper way is to call Needs.

And there is no way to Quit every available kernel.

Example

is worth of 10^3 words:

First we will mimic a package upload, 10 times. So it was uploaded, something was fixed it was uploaded again etc:

packageTemplate = StringTemplate["
   BeginPackage[\"TestPackage<*\"`\"*>\"];
   myValue = \"``\"
   EndPackage[];
"];

Do[
   CloudExport[packageTemplate@RandomReal[], "Text", "TestPackage.m"]
 ; CloudEvaluate[
       CopyFile[
           "TestPackage.m"
         , StringRiffle[
               {$UserBaseDirectory, "Applications", "TestPackage.m"}
             , "/"
           ]
         , OverwriteTarget -> True
       ]
     ; Get @ "TestPackage`"
   ]
 , {10}
]

Now let's call code based on the package 25 times:

Table[
    CloudEvaluate[
        Needs["TestPackage`"]
      ; {$SessionID, Symbol["TestPackage`myValue"]}
    ]
 ,  {25}
] // CountBy[Last] // Normal // Column

enter image description here

7 different results, I expect only one variant, from the last deployment.

The question

How to upload a package and clean properly, as we can see Get after the upload only affects one particular kernel/sessopm. CloudEvaluate@Quit[] won't help either.

How to reset them all?

Requirements

 APIFunction[{}, (Get["TestPackage`"];Symbol["TestPackage`api2"][...]) &]

could solve it but if the package contains Protected/Locked symbols you will get a flood of errors. And because of the same reason you can't ClearAll symbols. And obviously you can't Quit in ApiFunction.

POSTED BY: Kuba Podkalicki
6 Replies

Kuba, do you have any update on this to share with us? This is a nice topic to WTC2021.

Reading the docs, maybe the IncludeDefinitions option can help. I’m studying CloudAPI deeply.

POSTED BY: Rodrigo Murta

Unfortunately I don't have good news. I went with automatic dependency collection for APIs and Initialization>Get[...] in cloud notebooks. I am fortunate that there isn't anything fancy in my code like working with the LibraryLink, external data/caching or whatever.

While I don't like it didn't cause any problems except I would gladly shave those few milliseconds associated with loading.

And afaik there is no solution. Even with the EPC you'd need to restart it which is hardly a convenient solution.

POSTED BY: Kuba Podkalicki

As you have observed, api kernels (for now) don't clear their symbols after every evaluation. So, yes, your symbols definitions remain as long as that kernel remains active (I don't know how many evaluations are done before a kernel is killed and a new one started). Note that a user that calls an api has no control over which kernel will run the code (again, as your test above demonstrates).

As far as the 'inherently unreliable' comment, consider this case: your nemesis uses the same package name and symbols as you. So when your api gets to the Needs["TestPackage`"], it is possible that your nemesis' package was already loaded, and her symbols will be used in the remainder of your api. I'm not sure that this means apis are 'inherently unreliable'. I think it means you should use Get in your api.

I am unaware of any documentation about the configuration of the cloud. I have learned much of what I know by doing experiments such as the one you showed ab

POSTED BY: Chad Knutson
POSTED BY: Kuba Podkalicki

I would advise you to use Get rather than Needs. This is the public cloud after all; the api kernels are running code for many different users.

So, simply avoid using the Locked attribute. Near the beginning of your package, you can Unprotect all symbols that will be defined and later Protected. If you don't want other users to be able to read DownValues, then ReadProtect them.

POSTED BY: Chad Knutson

So If my API Gets a package from my private $UserBaseDirectory then everyone whose API/Form is attached to the same kernel will have access to my package? Is it account type dependent or is using packages by API/Form on WPC inherently unreliable?

This is the public cloud after all; the api kernels are running code for many different users.

I don't see how is that obvious. Could you point me to the relevant part of the documentation?

All things considered I don't think Get/Needs or fiddling with attributes matter here as sooner or later interference will happen.

So I should probably setup deployment in such a way that Language` will take care about my definitions. And I should trust it will shield it well. Is this package and the flow documented anywhere?

POSTED BY: Kuba Podkalicki
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard