Message Boards Message Boards

GROUPS:

Clean package update for API/FormFunctions on Wolfram Cloud?

Posted 2 years ago
2405 Views
|
4 Replies
|
2 Total Likes
|

Cross posted on mathematica.stackexchange

TLDR

There is a pool of kernels/sessions available for each user, you don't have control over the pool, only over specific kernel you currently evaluate in. Re-upload of a package should be followed by quit on every kernel where previous version of a package was used. I failed to find tools for that.

Background

I'm developing a package which I want to deploy to my Wolfram Public Cloud's account and I want a set of APIFunctions/FormFunctions/etc to be able to use it.

  • PacletManager has limited functionality so I'm just uploading package's archive and extracting it to WPC's $UserBaseDirectory / Applications /.

  • APIFunctions and friends have form:

     APIFunction[{}, (Needs["TestPackage`"];Symbol["TestPackage`api2"][...]) &]
    

I use Symbol because otherwise TestPackage` definitions are uploaded and I want to avoid that. There should be one code source, the package in $UserBaseDirectory.

Problem

The problem is that kernels' management on WPC is a closed black box. If you call the api twice, each time it uses one of the kernels in the pool. It also applies to CloudEvaluate etc.

$UserBaseDirectory is shared but Get will find the current version ony for current kernel.

If in another kernel TestPackage` was loaded earlier, Needs will not load the current version there :-/ I don't wan't to Get in my APIFunctions, a proper way is to call Needs.

And there is no way to Quit every available kernel.

Example

is worth of 10^3 words:

First we will mimic a package upload, 10 times. So it was uploaded, something was fixed it was uploaded again etc:

packageTemplate = StringTemplate["
   BeginPackage[\"TestPackage<*\"`\"*>\"];
   myValue = \"``\"
   EndPackage[];
"];

Do[
   CloudExport[packageTemplate@RandomReal[], "Text", "TestPackage.m"]
 ; CloudEvaluate[
       CopyFile[
           "TestPackage.m"
         , StringRiffle[
               {$UserBaseDirectory, "Applications", "TestPackage.m"}
             , "/"
           ]
         , OverwriteTarget -> True
       ]
     ; Get @ "TestPackage`"
   ]
 , {10}
]

Now let's call code based on the package 25 times:

Table[
    CloudEvaluate[
        Needs["TestPackage`"]
      ; {$SessionID, Symbol["TestPackage`myValue"]}
    ]
 ,  {25}
] // CountBy[Last] // Normal // Column

enter image description here

7 different results, I expect only one variant, from the last deployment.

The question

How to upload a package and clean properly, as we can see Get after the upload only affects one particular kernel/sessopm. CloudEvaluate@Quit[] won't help either.

How to reset them all?

Requirements

 APIFunction[{}, (Get["TestPackage`"];Symbol["TestPackage`api2"][...]) &]

could solve it but if the package contains Protected/Locked symbols you will get a flood of errors. And because of the same reason you can't ClearAll symbols. And obviously you can't Quit in ApiFunction.

4 Replies

I would advise you to use Get rather than Needs. This is the public cloud after all; the api kernels are running code for many different users.

So, simply avoid using the Locked attribute. Near the beginning of your package, you can Unprotect all symbols that will be defined and later Protected. If you don't want other users to be able to read DownValues, then ReadProtect them.

So If my API Gets a package from my private $UserBaseDirectory then everyone whose API/Form is attached to the same kernel will have access to my package? Is it account type dependent or is using packages by API/Form on WPC inherently unreliable?

This is the public cloud after all; the api kernels are running code for many different users.

I don't see how is that obvious. Could you point me to the relevant part of the documentation?

All things considered I don't think Get/Needs or fiddling with attributes matter here as sooner or later interference will happen.

So I should probably setup deployment in such a way that Language` will take care about my definitions. And I should trust it will shield it well. Is this package and the flow documented anywhere?

As you have observed, api kernels (for now) don't clear their symbols after every evaluation. So, yes, your symbols definitions remain as long as that kernel remains active (I don't know how many evaluations are done before a kernel is killed and a new one started). Note that a user that calls an api has no control over which kernel will run the code (again, as your test above demonstrates).

As far as the 'inherently unreliable' comment, consider this case: your nemesis uses the same package name and symbols as you. So when your api gets to the Needs["TestPackage`"], it is possible that your nemesis' package was already loaded, and her symbols will be used in the remainder of your api. I'm not sure that this means apis are 'inherently unreliable'. I think it means you should use Get in your api.

I am unaware of any documentation about the configuration of the cloud. I have learned much of what I know by doing experiments such as the one you showed ab

Thanks for your input Chad. Yes this is what I meant by 'inherently unreliable'. And I still think so, here is why:

Get wouldn't be the core of the solution as old DownValues reordering/overwriting by new ones would be hard to predict, so ClearAll["TestPackage", "TestPackage*"]` is a must I think. Which requires forgetting about Locked+Protected, let's say I can live with this. But as you've noticed my nemesis may not be so flexible and could leave them.

One could say it is highly unlikely but we don't need to talk about custom packages, there are packages widely used by community and since Paclet's are not documented, in case of adjustments, people will just add custom changes to those packages by hand, without any versioning info. It will be not possible to manage this, especially on the public cloud.

The bottom line is, all this is unacceptable in development/deployment/testing/debugging cycle. Sooner or later it will cause hard to debug problems. This should be officially discouraged untill tools to manage API kernels appear or a documentation of how it works is written. I don't mean details but an overview which will help to predict what causes what.

Till then the official guideline should to trust the Language`FullDefinition, and to develop isolated packages for API. Which is still arguable as it is a black box solution but at least one can write reports to support based on CloudDeploy docs:

CloudDeploy[expr,[Ellipsis]] automatically deploys all definitions needed to evaluate expr, much like CloudSave.

p.s. people should not need to do test like this to figure out fundamental aspects of working with cloud.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract