Functions needed in Image Processing for future release of Mathematica

Ideas
, , , , , , ,
1

Update:

Some more desirable functions:

  • Superpixels

  • Graph cut segmentation: the “GrowCutComponents” in Mathematica uses cellular automata. Is the performance better than graph cut?

  • faster execution of Watershed components for 3D images for a considerable number of components.

  • built in functions (based on neural nets) that can be trained on datasets to do binary segmentation or semantic segmentation. Wolfram Neural Net repository nets can be made into special functions perhaps?

  • Interpolate between two binary images in 2D and 3D

Original Post

I am thrilled by the image processing capabilities built into Mathematica and witness its
power on a daily basis. I never have to use any other program for image processing,
save FIJI. In my opinion Mathematica’s image processing core, albeit powerful and broad,
can adopt a thing or two from FIJI. As a biophysics student - who works with microscopy
images and other images in general - I firmly believe that the suggested functions mentioned
below can further add a lot of punch to the image processing core. Furthermore,
several of these functions have a role much broader than microscopy/fluorescence microscopy.
I sincerely hope that the image processing development team takes a note of the post.

Please let me know in the comments section if some function already exists. And feel free
to add functionalities to the list.

DESIRABLE FUNCTIONALITIES

7 Likes
2

i know that the first one is already there, albeit a bit hidden:

ClearAll[oFindImageCircles]
oFindImageCircles[img_/;ImageQ@img&&ImageChannels@img==1,
{sizemin_Integer,sizemax_Integer},(*range of radii for circles*)
mindist_Real,(*minimum distance in between circle centers*)
thrCanny_Real,(*minimum canny edge strength---the function computes edge detection on the input*)
thrStrength_Real,(*minimum strength of circle (aka votes)*)
resolutionfactor_,(*between 0 and 1,how big is the hough array compared to the image*)
nmax_  (*max number of circles*)
]:=Module[{res},Needs["OpenCVLink`"];
    res=OpenCVLink`Private`$HoughCircles[img,1/resolutionfactor,mindist,(255*thrCanny),thrStrength,sizemin,sizemax];(*sorted by strength already*)
    If[Head@res=!=List,Throw@$Failed];
        res={{#1,#2},#3}&@@@Partition[res,3];
        If[nmax===All,
            res,
            Take[res,UpTo[nmax]]
        ]
];
3 Likes
3

Thanks Ali for sharing your wish list. It really helps to hear user feedback. Most of these were already on our todo list, we will add the missing ones and reprioritize all of them.

Also, here are a few things from the list that are immediately possible:

  • Hough circles, as Sanders mentioned, we have the OpenCV version of it, it is waiting for a more complete/robust version that works in more cases
  • Strahler Analysis can be done with morphological operations, look at Pruning
  • morphological segmentation is the same as what we have under WatershedComponents
  • lookup table application to images is available through ImageApply. Typically, for integer images a LUT is created as the pixels are processed anyways. One can also provide a specific LUT.
4 Likes
4

Thanks Sander ! was really hidden :slight_smile:

5

Thanks Shadi for taking note of the post. I am really looking forward to the new functionalities that you and your team will incorporate in the next release(s). I will also remove those features from my post that have already been implemented (e.g. Strahler Analysis and Morphological Segmentation) or at least mention that they are already there :slight_smile:

1 Like
6

Can you clarify what you mean by 16-bit image display?

As for Fiji, I asked a question about interoperability here:

Mathematica can’t be expected to have everything Fiji has (especially special plugins!) but it would be great if I could use Mathematica as the centre of my workflow and still employ ImageJ for certain specific tasks. It seems that J/Link currently has some limitations that make it difficult to use ImageJ (and I now knowing much Java didn’t help …) I wrote to Wolfram Support about this and the feedback I got so far sounded encouraging (i.e. that they’d look into making things work). Let’s see how that works out!

I also have a small LTemplate-based framework to make it easier to call SimpleITK functions directly from Mathematica. Sometimes SimpleITK has useful stuff that Mathematica doesn’t (e.g. local histogram equalization, which I hope Mathematica will get in the future).

Occasionally I use MATLAB stuff too through MATLink.

Overall, what I am wishing for the most is better interoperability with other systems.

If some operation is built-in, that’s obviously the best. But Wolfram only has limited resources. A high-quality interface to other systems, or toolkits that make it easier to build such interfaces (like LTemplate), can often give the highest benefit for the least amount of work.

3 Likes
7

Reading it now, I was definitely not clear in my post. I have some images with fluorescent signals that are 16 bit. In a few of those images the maximum pixel intensity does not exceed say more than 128 - which is quite low as far as the range a 16 bit image spans. The problem is even if I use ImageAdjust I still see a dark image and it is only by artificially multiplying the image by some integer value that I begin to see the features. Whereas if I load the same image with FIJI I can see the signal without any manual manipulation. Here i was mentioning that Mathematica should adopt a similar strategy to display images wherein the pixels span the lower end of the entire intensity range.

While it is important to ensure interoperability, I believe it is quintessential for WRI to implement many of these functionalities within Mathematica since they have tried to cater to people working with computational microscopy. PIV, Kymographs, robust registration and single molecule/cell tracking, even though plugins in FIJI, are the bread and butter of many imaging labs and are essential tools of many physicists/engineers. Other features like the ability to perform local contrast and weighted local contrast enhancement - as you also suggested - will be extremely useful to many of us for processing 2D and 3D images.

I have three main reasons as to why WRI needs to add these functionalities to its core image processing capabilities.

A constant transfer of data between two systems may prove to be slow (just making a hypothesis here; I am no specialist by any means). I wonder how slow it will become if we consider a series of processing task on 3D images that need to be transferred back and forth.

Furthermore, FIJI while being good at manipulating 2D images seem to not work that well on 3D image stack. So far, the brilliant feature about Mathematica’s image processing is that functions are equally capable for 2-D and 3-D images. I hope they continue to develop and maintain functions which work seamlessly for both cases.

FIJI is opensource (which is great) but the downside is that the developers are not paid. Therefore, a developer of a special feature may choose to leave some features in the beta stage for personal reasons. Mathematica being the exact opposite is expected to perform.

But having said this, I agree with you that interoperability will only be beneficial.

2 Likes
8

Btw this version of the function written by someone (forgot the name) is slow but sometimes I find it useful:

HistogramTransformationWeightedLocalAdaptive[img_, radius_] := 
  Module[{funs, nodes, w, h, f, fun, fi},
   funs = Map[HistogramTransformInterpolation, ImagePartition[img, {2*radius + 1}], {2}];
   {w, h} = ImageDimensions@img;
   nodes = Map[Mean@Flatten[N@ImageData[#, Automatic], 1] &, 
     ImagePartition[Image[Array[{#1, #2} &, {h, w}], "Bit16"], {2*radius + 1}], {2}];
   nodes = ArrayPad[nodes, {{1, 1}, {1, 1}}, "Fixed"];
   nodes[[All, {1, -1}, 2]] = {1, w};
   nodes[[{1, -1}, All, 1]] = {1, h};
   funs = ArrayPad[funs, 1, "Fixed"];
   fi = Interpolation[Thread[{Round@Flatten[nodes, 1], 
       Table[f[#], {f, Flatten@funs}]}], InterpolationOrder -> 1];
   fun[v_, {r_, c_}] := Evaluate[fi[r, c]] &[v];
   Quiet@ImageApplyIndexed[fun, img]
   ];

contenhancement.jpg
contenhancement.jpg1157×402 135 KB

9

I am absolutely not arguing against adding more features to Mathematica. What I was trying to say is that Wolfram has zero chance of reaching parity with a tool like Fiji, which has a lot more resources and developers.

If Mathematica users can only use built-in stuff, then the Mathematica userbase will gradually get eroded to nothing in the next 10-20 years. However, if interfacing with other systems were embraced, if third-party package development were encouraged, then people could keep Mathematica the centre of their workflow while selectively using tools from other systems as needed.

FIJI is opensource (which is great) but the downside is that the developers are not paid.

At my institute, several research groups contribute to ImageJ. Some people are effectively paid to work on it. I am quite sure that ImageJ has vastly more resources available to it than Wolfram’s image processing group. A huge amount of research funding goes into it.

Finally, many of these groups are developing new methods that let you do things that were simply not possible before. Wolfram will either integrate existing code (see how they use OpenCV and ITK internal;y) or implement existing, well-tested methods. You’ll never get the state of the art from Wolfram (except perhaps in the form of third-party packages, but developing those isn’t particularly encouraged). If you stick to Mathematica, and Wolfram doesn’t start putting more effort into practical interoperability (i.e. not stuff like ExternalEvaluate in M11.3), then you’ll be missing out soon … I’m saying this as someone who would very much prefer to keep working in Mathematica, but is more and more often yanked out of the system because the tools I need are found elsewhere (including one instance with Fiji just today).

So what can one do in this situation? You can be an optimist like I was and hope that M will improve according to your dreams. You can even create your own interfaces to the systems you need, like I did. I put countless and countless hours into IGraph/M in the hope that the built-in Graph functionality, which the IGraph/M package relies on, will improve in the future. There are no signs of this happening, and I have almost lost all hope. Luckily, the image processing stuff is in a much better shape, but how can you be sure that it won’t meet the same fate as Graph? And when that happens, do you really want to be locked into the Wolfram system because that’s where your expertise and your existing tools (developed over many years) are, and there is no bridge to anything else (no interoperability)?

3 Likes
10

I’m veering off-topic though.

I would, of course, like to see the image processing functionality improve too. I would like to see common tools be included as built-in functions.

My point was simply that there will always be many more tools that Wolfram won’t have resources to include. Like you, I love to work in Mathematica, so I hope they’ll make it as easy as possible to use Mathematica as a glue language and call other systems directly from Mathematica as needed. E.g. J/Link is already very good, but alone it’s not enough.

11

I second you on this. For my current pipeline, I am doing registration using a plugin in ImageJ and then have to export all those images back to M. If there were an easier integration with other potentially powerful software then it will only make the pace of the work faster and seem a bit effortless.

The best approach then is thus two-pronged: (1) WRI should make it easier for M to interface with other programs (2) to extend the current capabilities of the core.

And I do hope that Charles Pooh is going to do what he said earlier that they are not abandoning Graph and associated functions. We are all aware how important graphs are - in their own right and as a useful data structure.

12

And btw if you deem so, would not it be nice to have a post where you can jote down all the Graph related functionality that you desire as an experienced user? I am interested to know what needs to be added to Graph. Maybe someone at WRI can take notes as to what the community desires.

13

Last off-topic message in this thread—if you have figured out a way to control ImageJ/Fiji from within Mathematica, even in a very rudimentary way, please do send me an email. We can continue the discussion there then.

14

I am still doing it very crudely and am myself trying to find a way to connect the two together. I will let you know if I get any success !

15

Hi Shadi,

I made an update of the post: included super pixel, GraphCut segmentation and a possible faster execution for Watershed opponents for 3D.

Regards,
Ali

16
  • Strahler Analysis can be done with morphological operations, look at Pruning

@Shadi, how would I select a certain branch, mark it as “the root”, and prevent it from getting pruned by Pruning?

One way would be to manually add a small cycle, e.g. this pixel pattern, at the end of the branch:

0 1 0
1 0 1
0 1 0

This seems troublesome to automate though.

Talking of cycle, Fiji has a feature to prune small cycles with various methods. One way I can think of to achieve this in Mathematica is to negate the image, then DeleteSmallComponents. If there is a more efficient way that could be made part of Pruning, that would be a welcome extension.

1 Like