Message Boards Message Boards

Where is my notebooks permanent storage?

GROUPS:

A pretty stupid question. But I cannot answer it.

I work with Windows 10 and the Wolfram Programming Lab desktop. I have some notebook, say "x.nb", which is persistently stored somewhere within the Wolfram cloud. This notebook has to make some of its data v persistent with "Save", "DumpSave" or the like. Of course I want to have that data nearby to x.nb. Say as "xv" within the same cloud directory as x.nb or as "v" in some subdirectory "xdata" of the location of my x.nb.

Pretty stupid, I know. (The same would apply if x.nb where on local storage). The question is: If I work with some notebook x.nb: How can I find a relative cloud or local object where I can save its objects nearby?

POSTED BY: Werner Geiger
Answer
3 months ago

I do not use either the Programming Lab or Wolfram Cloud (only desktop Mathematica), so I cannot answer your question.

But perhaps I can offer an alternative. There are many reasons why you might want to have some files/data "attached" (stored next to) a notebook. One is the application you describe.

When you simply want to store some data that is not too large (at most a few megabytes), you can conveniently save right inside a notebook using this trick:

It does work in Wolfram Cloud too.

How would you use this with DumpSave/Save without going through a temporary on-disk file? I do not know. But perhaps you'll find the technique useful (or you can use those temporaries for saving and ToExpression for loading).

POSTED BY: Szabolcs Horvát
Answer
3 months ago

Thanks Szabolcs, that sounds pretty convenient, although a bit tricky. Maybe I will try it, if I cannot find how to (Dump)Save/Get my data from an external file.

I have two concerns with that approach:

  1. I want to use that saved data v from other notebooks as well. Actually it is a large sparse array which holds some kind of tabular data of already calculated values by some very expensive algorithm. (Its a recursive algorithm for my own integerPartionsP[n,{k}]. Unfortunately Mathematica knows IntegerPartitionsP[n], but not that parameter k).

  2. Months ago I had huge problems with notebooks holding large data variables and/or large output when saved to the cloud. In some cases they were corrupted and could not be reloaded. (Since then I normally do ClearAll["Global`*"] and even delete large output before saving).

POSTED BY: Werner Geiger
Answer
3 months ago

Sorry. I wrote: "Its a recursive algorithm for my own integerPartionsP[n,{k}]. Unfortunately Mathematica knows IntegerPartitionsP[n], but not that parameter k."

This has to read: "Its a recursive algorithm for my own partionsP[n,{k}]. Unfortunately Mathematica knows PartitionsP[n], but not that parameter k."


Basically it is very simple. To get the number of integer partitions of n with exactly k parts just implement the well known formula:

P(n,k) = P(n-1,k-1) + P(n-k,k), where P(n,k)=0 for k<1||k>n, P(n,k)=1 for k=1||k=n.

partitionsPsimple[n_Integer, {k_Integer}] :=
 If[k < 1 || k > n, 0,
  If[k == 1 || k == n, 1,
   partitionsPsimple[n - 1, {k - 1}] + partitionsPsimple[n - k, {k}]
   ]
  ]

But for larger n and k this takes astronomical time and exceeds all reasonable recursion limits. That is why I want to store every intermediate result in a persistent sparse array and reuse it. Then it seems to be usable.

POSTED BY: Werner Geiger
Answer
3 months ago

Are you familiar with memoization in Mathematica?

Does this help?

Clear[pp]
pp[n_, 1] = pp[n_, n_] = 1;
pp[n_, k_] /; k < 1 || k > n = 0;
pp[n_, k_] := pp[n, k] = pp[n - 1, k - 1] + pp[n - k, k]
POSTED BY: Szabolcs Horvát
Answer
3 months ago

Yes of course I am familiar with that.

But I do not think that this is more readable than my simple if-then-else-coding.

What you claim is overloading or polymorphism in object-oriented-programming. Very useful, but not for replacing simple if-then-else-constructs. And without any doubt much more expensive at runtime. And much less readable.

POSTED BY: Werner Geiger
Answer
3 months ago

I claimed nothing about object oriented programming. I simply showed how to avoid "astronomical times and exceeding all reasonable recursion limits" by using memoization.

In[89]:= pp[1234, 321] // AbsoluteTiming
Out[89]= {1.06454, 713311038115479967299445475964}

How big $n$ and $k$ do you need?

As for efficiency, I doubt that the If construct will outperform pattern matching, but why don't you just try? Anyway, the vast majority of the performance improvement comes from memoization, which avoids the exponential complexity caused by the function calling itself twice. Note that memoization will recreate some of those patterns that I used to replace the Ifs, even if they are not used in the original definition.

POSTED BY: Szabolcs Horvát
Answer
3 months ago

Wow!

I am ashamed and deeply impressed and obviously had no idea of "memoization". I thought, you just talked about polymorphism, but it is far from that and much better. I tried your code and ist is far better than all I have written until now within WL.

I still have to learn a lot. Thank you!


I need that P(n,k) for around n=1600, k=100. My coding is almost lost with that. Your code does it at not time:

If called the first time after load notebook it leads:

In[459]:= AbsoluteTiming[pp[1600, 100]]

Out[459]= {0.914443, 397378545506689283221615392942159658087}

If called again within the same session it leads:

In[460]:= AbsoluteTiming[pp[1600, 100]]

Out[460]= {3.29889*10^-6, 3973785455066892832216153929421596
POSTED BY: Werner Geiger
Answer
3 months ago

Of course I read a lot and did some tests. The filename (without path) of my notebook ist "170819 100 Affen.nb" (sorry about that weird name). It is stored within some directory of the Wolfram cloud. I need that filename and an object for that directory and then I have to construct a CloudObject vor my v-data relative to that directory-object. Say something like directory/filename/v-data which would be one level below my notebook within a subdirectory with name filename.

It seems obvious to use NotebookDirectory[] for directory. But this gives an error:

In[420]:= NotebookDirectory[]

During evaluation of In[420]:= DirectoryName::string: String expected at position 1 in DirectoryName[CloudObject[https://www.wolframcloud.com/objects/94af901a-0f86-495c-8125-9779955dea0d/170819 100 Affen.nb]].

Out[420]= DirectoryName[
CloudObject[
 "https://www.wolframcloud.com/objects/94af901a-0f86-495c-8125-\
9779955dea0d/170819 100 Affen.nb"]]

To get the CloudObject of my notebook I can use:

In[421]:= myCO = "Uri" /. NotebookInformation[]

Out[421]= \
CloudObject["https://www.wolframcloud.com/objects/94af901a-0f86-495c-\
8125-9779955dea0d/170819 100 Affen.nb"]

or even simpler:

In[424]:= myCO = NotebookFileName[]

Out[424]= \
CloudObject["https://www.wolframcloud.com/objects/94af901a-0f86-495c-\
8125-9779955dea0d/170819 100 Affen.nb"]

This myCO seems to be all I need. But I cannot find how to split that CloudObject into objects for directory and filename and then reassemble those parts into a CloudObject for my v-data.

Pretty stupid, I know. Probably I have to get the myCO-URI as a string, spilt and recompose that as usual and create a new CloudObject from that new string. Why not? But we work with objects, not strings.

POSTED BY: Werner Geiger
Answer
3 months ago

In Mathematica I just do a

SetDirectory[NotebookDirectory[]]

before exporting data, but I'm not sure if this works with the "Wolfram Programming Lab desktop".

POSTED BY: Gustavo Delfino
Answer
3 months ago

This does not work. It issues errors:

In[444]:= SetDirectory[NotebookDirectory[]]

During evaluation of In[444]:= DirectoryName::string: String expected at position 1 in DirectoryName[CloudObject[https://www.wolframcloud.com/objects/94af901a-0f86-495c-8125-9779955dea0d/170819 100 Affen.nb]].

During evaluation of In[444]:= SetDirectory::fstr: File specification DirectoryName[CloudObject[https://www.wolframcloud.com/objects/94af901a-0f86-495c-8125-9779955dea0d/170819 100 Affen.nb]] is not a string of one or more characters.

Out[444]= SetDirectory[DirectoryName[
CloudObject[
  "https://www.wolframcloud.com/objects/94af901a-0f86-495c-8125-\
9779955dea0d/170819 100 Affen.nb"]]]

I'm afraid, I have some basic misunderstanding. Each and every operating system is organized in some kind of tree-alike hierarchical directories and files within those directories. Not only local file systems but cloud systems as well. The Wolfram cloud is organized like that as well. Hence "Path" and "File" are principal features of all kinds of persistent storage. I do not understand why those parts are not directly accessible.

I think the mess started ten years ago when Apple began to hide their file system on smartphones from users by creating nonsense-"Folders" or -"Hubs" like "Music" or "Fotos" or "Documents". Since then nobody knows anymore what ist stored where and how it is named.

Unfortunatly all the others have adopted this nonsense-thinking. And now you don't even know on your own PC what ist stored where. Let aside all those clouds.

POSTED BY: Werner Geiger
Answer
3 months ago

Test message — I think this is going to trigger a Wolfram Community bug ...

POSTED BY: Szabolcs Horvát
Answer
3 months ago

OK, no bug triggered, so I can write without fear of losing my message.

It's hard to write efficient programs in Mathematica, partly because there are so many ways to do anything, and it's often difficult to predict which way will be fast.

Leonid Shifrin's post is a very useful read:

Your original question still stands though:

How can we have "sidecar files" to notebooks and sync them between desktop and cloud?

POSTED BY: Szabolcs Horvát
Answer
3 months ago

Group Abstract Group Abstract