Message Boards Message Boards

3
|
11060 Views
|
6 Replies
|
6 Total Likes
View groups...
Share
Share this post:

Clean package update for API/FormFunctions on Wolfram Cloud?

Kuba Podkalicki
Posted 8 years ago

Cross posted on mathematica.stackexchange

TLDR

There is a pool of kernels/sessions available for each user, you don't have control over the pool, only over specific kernel you currently evaluate in. Re-upload of a package should be followed by quit on every kernel where previous version of a package was used. I failed to find tools for that.

Background

I'm developing a package which I want to deploy to my Wolfram Public Cloud's account and I want a set of APIFunctions/FormFunctions/etc to be able to use it.

  • PacletManager has limited functionality so I'm just uploading package's archive and extracting it to WPC's $UserBaseDirectory / Applications /.

  • APIFunctions and friends have form:

     APIFunction[{}, (Needs["TestPackage`"];Symbol["TestPackage`api2"][...]) &]

I use Symbol because otherwise TestPackage` definitions are uploaded and I want to avoid that. There should be one code source, the package in $UserBaseDirectory.

Problem

The problem is that kernels' management on WPC is a closed black box. If you call the api twice, each time it uses one of the kernels in the pool. It also applies to CloudEvaluate etc.

$UserBaseDirectory is shared but Get will find the current version ony for current kernel.

If in another kernel TestPackage` was loaded earlier, Needs will not load the current version there :-/ I don't wan't to Get in my APIFunctions, a proper way is to call Needs.

And there is no way to Quit every available kernel.

Example

is worth of 10^3 words:

First we will mimic a package upload, 10 times. So it was uploaded, something was fixed it was uploaded again etc:

packageTemplate = StringTemplate["
   BeginPackage[\"TestPackage<*\"`\"*>\"];
   myValue = \"``\"
   EndPackage[];
"];

Do[
   CloudExport[packageTemplate@RandomReal[], "Text", "TestPackage.m"]
 ; CloudEvaluate[
       CopyFile[
           "TestPackage.m"
         , StringRiffle[
               {$UserBaseDirectory, "Applications", "TestPackage.m"}
             , "/"
           ]
         , OverwriteTarget -> True
       ]
     ; Get @ "TestPackage`"
   ]
 , {10}
]

Now let's call code based on the package 25 times:

Table[
    CloudEvaluate[
        Needs["TestPackage`"]
      ; {$SessionID, Symbol["TestPackage`myValue"]}
    ]
 ,  {25}
] // CountBy[Last] // Normal // Column

enter image description here

7 different results, I expect only one variant, from the last deployment.

The question

How to upload a package and clean properly, as we can see Get after the upload only affects one particular kernel/sessopm. CloudEvaluate@Quit[] won't help either.

How to reset them all?

Requirements

 APIFunction[{}, (Get["TestPackage`"];Symbol["TestPackage`api2"][...]) &]

could solve it but if the package contains Protected/Locked symbols you will get a flood of errors. And because of the same reason you can't ClearAll symbols. And obviously you can't Quit in ApiFunction.
POSTED BY: Kuba Podkalicki
6 Replies
Sort By:

I would advise you to use Get rather than Needs. This is the public cloud after all; the api kernels are running code for many different users.

So, simply avoid using the Locked attribute. Near the beginning of your package, you can Unprotect all symbols that will be defined and later Protected. If you don't want other users to be able to read DownValues, then ReadProtect them.
POSTED BY: Chad Knutson

So If my API Gets a package from my private $UserBaseDirectory then everyone whose API/Form is attached to the same kernel will have access to my package? Is it account type dependent or is using packages by API/Form on WPC inherently unreliable?

This is the public cloud after all; the api kernels are running code for many different users.

I don't see how is that obvious. Could you point me to the relevant part of the documentation?

All things considered I don't think Get/Needs or fiddling with attributes matter here as sooner or later interference will happen.

So I should probably setup deployment in such a way that Language` will take care about my definitions. And I should trust it will shield it well. Is this package and the flow documented anywhere?
POSTED BY: Kuba Podkalicki

As you have observed, api kernels (for now) don't clear their symbols after every evaluation. So, yes, your symbols definitions remain as long as that kernel remains active (I don't know how many evaluations are done before a kernel is killed and a new one started). Note that a user that calls an api has no control over which kernel will run the code (again, as your test above demonstrates).

As far as the 'inherently unreliable' comment, consider this case: your nemesis uses the same package name and symbols as you. So when your api gets to the Needs["TestPackage`"], it is possible that your nemesis' package was already loaded, and her symbols will be used in the remainder of your api. I'm not sure that this means apis are 'inherently unreliable'. I think it means you should use Get in your api.

I am unaware of any documentation about the configuration of the cloud. I have learned much of what I know by doing experiments such as the one you showed ab
POSTED BY: Chad Knutson

Thanks for your input Chad. Yes this is what I meant by 'inherently unreliable'. And I still think so, here is why:

Get wouldn't be the core of the solution as old DownValues reordering/overwriting by new ones would be hard to predict, so ClearAll["TestPackage", "TestPackage*"]` is a must I think. Which requires forgetting about Locked+Protected, let's say I can live with this. But as you've noticed my nemesis may not be so flexible and could leave them.

One could say it is highly unlikely but we don't need to talk about custom packages, there are packages widely used by community and since Paclet's are not documented, in case of adjustments, people will just add custom changes to those packages by hand, without any versioning info. It will be not possible to manage this, especially on the public cloud.

The bottom line is, all this is unacceptable in development/deployment/testing/debugging cycle. Sooner or later it will cause hard to debug problems. This should be officially discouraged untill tools to manage API kernels appear or a documentation of how it works is written. I don't mean details but an overview which will help to predict what causes what.

Till then the official guideline should to trust the Language`FullDefinition, and to develop isolated packages for API. Which is still arguable as it is a black box solution but at least one can write reports to support based on CloudDeploy docs:

CloudDeploy[expr,[Ellipsis]] automatically deploys all definitions needed to evaluate expr, much like CloudSave.

p.s. people should not need to do test like this to figure out fundamental aspects of working with cloud.
POSTED BY: Kuba Podkalicki

Kuba, do you have any update on this to share with us? This is a nice topic to WTC2021.

Reading the docs, maybe the IncludeDefinitions option can help. I’m studying CloudAPI deeply.
POSTED BY: Rodrigo Murta

Unfortunately I don't have good news. I went with automatic dependency collection for APIs and Initialization>Get[...] in cloud notebooks. I am fortunate that there isn't anything fancy in my code like working with the LibraryLink, external data/caching or whatever.

While I don't like it didn't cause any problems except I would gladly shave those few milliseconds associated with loading.

And afaik there is no solution. Even with the EPC you'd need to restart it which is hardly a convenient solution.
POSTED BY: Kuba Podkalicki
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract