Message Boards Message Boards

Data Drop with images via API?

I have a Python data acquisition application that generates images. I'd like to be able to share those to a DD Databin for subsequent analysis.

It appears that the 'programmatic' interfaces are still 'coming soon' so I presume I would need to use the Web API. I can see how to use HTTP Requests to upload simpler key-value pairs, but images seem to require a bit more heavy lifting than I have experience with.

Does anyone happen to have a simple example of doing this?

POSTED BY: Flip Phillips
7 Replies

Wonderful, thanks Bob! I will give this a shot.

POSTED BY: Flip Phillips

Flip, Databins allows larger limits for entries with png or jpg images. You can take advantage of this by using the correct mime type in your post request. Or, from a WL interface, you can use DatabinAdd and it will handle all the request forming for you.

Here is a screen shot showing both an http request and DatabinAdd:

enter image description here

You can see here that the size overhead is more modest.

When you use an "Image" interpreter in a data drop webform it imports the file to Wolfram's Image[] format before storing it, which can add significant size depending on the content and original format of the image. That is something we hope to improve.

POSTED BY: Bob Sandheinrich

So, just to test things, I created a structured databin

foo = CreateDatabin[
  "Interpretation" -> {"Contact" -> "Image", "Object" -> "Image", 
    "Drape" -> "Image", "tContact" -> "Number", "tObject" -> "Number",
     "tDrape" -> "Number", "ID" -> "String", 
    "Experiment" -> "String"}]

I added an entry via its web form located here. Note that each image is c.a. 19Kb for a total of about 60Kb and change.

The resulting databin, with a single entry, is almost 4Mb - which seems like a totally crazy amount of overhead. Notice also that the creation-time is wrong for some reason.

wtf

I have gotten closer sending the data via a POST operation via Python, but, at present, the data is not interpreted correctly (e.g., as an Image ) but is, rather, stored as gigantic pool of binary crud. So... onward!

POSTED BY: Flip Phillips

Yes, it is. I assume the reasons for the limit are that Data Drop is focused on internet of things data (consisting of small and frequent observations), and that increasing the limit would increase the server load and impact the performance of Data Drop.

POSTED BY: Peter Heemeijer

Do you happen to know if this is a WRI-imposed limit?

POSTED BY: Flip Phillips

The 25Kb limit of Data Drop can be bypassed by

  1. Using Compress to generate a compressed string of an expression (in this case a figure)

  2. Using StringTake or related function to split the compressed string in substrings smaller than 25Kb

  3. Uploading the substrings to a databin

  4. Downloading the substrings, concatenating them, and using uncompress to reconstruct the original object.

POSTED BY: Peter Heemeijer

I'm getting closer- but

An interesting thing I just discovered- I can upload an image, properly, to the data drop from within Mathematica, but, if I try to use the Web API I receive the message:

"Your databin entry could not be added because it exceeds the 25 kb limit."

from the POST request.

So I'm guessing this is going to make things a little difficult for me, since my images are, generally, larger than 25kb.

POSTED BY: Flip Phillips
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract