Message Boards Message Boards

4
|
8570 Views
|
3 Replies
|
8 Total Likes
View groups...
Share
Share this post:

File size limits in Version 9

My group uses Mathematica for three-dimensional image processing.  Our light sheet-based fluorescence microscopes (LSFM, SPIM, DSLM) generate Terabytes of data from live specimens that are usually processed as stacks of images, which are 1-4 GBytes large.  We segment the data and store the results as Excel workbooks, which in turn are further processed by other programs.

Switching from Version 8 to 9 we suddenly realized that TIFF images larger than 2GByte cannot be imported anymore - see Mathematica SE question: Maximum size of TIF files in Mathematica .

We also realized that we have a problem with saving moderately large (more than ~20k rows) Excel workbooks.

From the various comments we have received so far I understand that Excel Import/Export as well as TIFF Import/Export are handled by Java code and that a new TIFF library is used since version 9.  The Excel Export issue can be addressed by increasing the memroy available to the Java Engine but I still do not understand that no error message and no warning is generated.  In any case we can only assume that the error will occur again once our Excel workbooks become even larger. The TIFF Import issue is still unresolved.

To make matters really bad, these errors occur silently without creating any error messages.  We only realize that the Jave Scripts fail because the data we intend to process does not exist.

I find this extremely worrying since this could hint at a fundamental problem.  We have already reported our findings as errors but we have not heard that they will be resolved with 9.0.1.  All our machines operate with two Xeons (8 or 12 cores), 96GByte of RAM, nVidia's D5000 and Windows 7-64bit.
3 Replies
Thanks for your suggestion Szabolcs. I posted a comment on the question on stack exchange...

As we dug even more into the problem, it turned out that the issue of the 2GB limit was not specifically due to the libtiff version. In fact the same library version 3.8.2 is being used in both M8 and M9. Moreover the support for BigTIFF in libtiff 4 allows handling TIFF files that are greater than 4GB. That's the main reason why the same old library worked fine in M8. There were other issues that were causing the failure in M9 and also M9.0.1.

As I mentioned in my other comment, the problem is already internally fixed and is waiting for another release.
POSTED BY: Shadi Ashnai
Hello Shadi,   Why don't you also post a note about this as a reply the question Prof. Stelzer has linked to?  I think people would appreciate to have this information, and that question is currently the top Google hit when searching for the problem.  If this were a MathGrou post, I'd just post the note myself and link back to the message, but that is not possible while this site is in beta.
POSTED BY: Szabolcs Horvát
We have started using libtiff in Mathematica 9 for importing and exporting TIFF files. Version 3.8.2 was shipped with Mathematica 9 which did not support BigTIFF. However, this change has brought in many new features and bug fixes that makes it worthwhile.

After we received your comment via email, we updated the library to a version that does support BigTIFF (the feature was released in version 4.0.0), however, it just missed the 9.0.1 release. Unfortunately I don't think there is any work around for it in the meantime.

Please let us know if there were any other issues in this regard. Best,

  -- Shadi
POSTED BY: Shadi Ashnai
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract