OK final post here. Did some more code optimization.

Both functions generate the same image (with some color rounding errors since I only use a fixed amount of colors). But since the output is "identical", so is the frontend time to render the images. Therefore we only have to look at the kernel time to compare. I get a 7x improvement in kernel time which for my application is what I need.

And for people interested the final code.

(*some fixed settings*)
ncol = 512;
colfuncs = {"Normal", "Reverse", "Symmetric", "Reverse Symmetric"};
colorNames = {"GrayTones", "Rainbow", "ThermometerColors",
   "SunsetColors", "TemperatureMap", "LightTemperatureMap",
   "GrayYellowTones", "BlueGreenYellow", "AvocadoColors",
   "SouthwestColors"};
colorFunctions = # -> ColorData[#] & /@ colorNames;

(*convert a RBBColor to {r,g,b,alpha} with each 0-255 integers*)
Attributes[Col2List] = {Listable};
Col2List[RGBColor[c___]] := Round[255 PadRight[List[c], 4, 1.]]
(*needed to convert white, black an transparent*)
Col2List[c_] := Col2List[RGBColor[c]]

(*define the color function orders*)
ColSel[func_, cfunc_] := With[{fun = Switch[func,
     "Normal", #1 &,
     "Reverse", 1 - #1 &,
     "Symmetric", Abs[2 #1 - 1] &,
     "Reverse Symmetric", Abs[Abs[2 #1 - 1] - 1] &]
   , cfun = cfunc /. colorFunctions}, cfun[fun[#]] &]

(*generate all the color lookup tables*)
ColorLookup[___] = ConstantArray[Col2List[Darker[Red]], ncol];
With[{ran = Range[0, 1., 1./(ncol - 1)]}, 
  Table[ColorLookup[j, i] = Col2List[ColSel[j, i][#] & /@ ran], {j, 
    colfuncs}, {i, colorFunctions[[All, 1]]}]];

(*color lookup function, prosersing clipping and transparentcy. It \
converts color index to color values*)
LookUpTable[{lstyle_, color_}, {minclip_, maxclip_}] := With[{
   (*generate color lookup table with correct clipping colors for min \
and max,add opacity for Image function*)
   collist = 
    Join[{Col2List[minclip]}, (ColorLookup[lstyle, color]), {Col2List[
       maxclip]}]
   }, Function[{x}, 
   Which[NumberQ[x] || VectorQ[x], collist[[x]], MatrixQ[x], 
    collist[[#]] & /@ x]]
  ]

(*converts numbers to integers for color lookup, accounting for \
clipped values*)
ColorRound = With[{n = ncol}, Compile[{{x, _Real, 0}, {r, _Real, 1}},
    n + 2 - 
     Ramp[n + 1 - 
       Ramp[1 + Round[(n - 1) ((x - r[[1]])/(r[[2]] - r[[1]]))]]],
    RuntimeOptions -> "Speed", RuntimeAttributes -> {Listable}]];

(*the "fast" image function with the needed options*)
Options[FastImage] = {
   PlotRange -> Automatic,
   ClippingStyle -> {Black, White},
   ImageOpacity -> 1,
   ColorFunction -> "Rainbow",
   ColorFunctionOrder -> "Normal",
   ImageSize -> 300
   };

FastImage[data_, OptionsPattern[]] := 
 Block[{cfo, cf, cl1, cl2, tcl, plr, op, imAll, size},
  (*get the options*)
  cfo = OptionValue[ColorFunctionOrder];
  cf = OptionValue[ColorFunction];
  {cl1, cl2} = OptionValue[ClippingStyle];
  plr = OptionValue[PlotRange] /. Automatic -> MinMax[data];
  op = OptionValue[ImageOpacity];
  size = OptionValue[ImageSize];

  (*convert number to color index*)
  imAll = LookUpTable[{cfo, cf}, {cl1, cl2}][ColorRound[data, plr]];
  (*add correct opacity if needed*)
  If[op < 1, imAll[[All, All, 4]] = Round[op imAll[[All, All, 4]]],
   If[Col2List[cl1][[-1]] === 255 && Col2List[cl2][[-1]] === 255,
    imAll = imAll[[All, All, 1 ;; 3]]]];
  (*make image*)
  Image[NumericArray[imAll, "UnsignedInteger8"], "Byte", 
   ColorSpace -> "RGB", ImageSize -> size, Magnification -> Automatic]
  ]

Yes indeed that is my experience as well, but your input got me inspired. I use most of my visualization code within Manipulate. So from all the information here a short kernel time and a short frontend time will result in a "smoother experience".

Image has by far the best frontend time, so is it possible to reduce the kernel time?

I noticed when explicitly giving Image the RGB and alpha values and forcing ColorSpace->"RGB" makes Image much faster. So the challenge was how to efficiently convert an arbitrary range of data values to RGB values and feeding that to Image. Also, I needed some specific options for the code to work in the larger scope.

My solution now is to pre-generate some lookup tables with predefined gradients. Next, I convert my data to integers with the same range as the length of the color lookup table. Finally, it is just selecting the correct color values from the lookup table.

With that, i end up with something faster than both the Graphics option or the Colorize@Image option and with the flexibility that I needed.

and this allows me to use ImageCompose.

And most important it runs very smoothly within manipulate with the size of images I typically work with.

(*some fixed settings*)
ncol = 1024;
colfuncs = {1 -> "Normal", 2 -> "Reverse", 3 -> "Symmetric", 
   4 -> "Reverse Symmetric"};
colfuncsR = Reverse /@ colfuncs;
colorNames = {"GrayTones", "Rainbow", "ThermometerColors",
   "SunsetColors", "TemperatureMap", "LightTemperatureMap",
   "GrayYellowTones", "BlueGreenYellow", "AvocadoColors",
   "SouthwestColors"};

(*convert a RBBColor to {r,g,b,alpha}*)
ClearAll[Col2Num];
Col2Num[c_] := N@Join[List @@ c, {1}];

(*define the color function orders*)
ClearAll[ColSel]
ColSel[func_, cfunc_] := 
 With[{fun = {# &, 1 - # &, Abs[(2 # - 1)] &, 
      Abs[Abs[(2*#) - 1] - 1] &}[[func]]}, ColorData[cfunc][fun[#]] &]

(*generate all the color lookup tables*)
ClearAll[ColorLookup2]
ColorLookup2[___] = Darker[Red];
With[{ran = Range[0, 1., 1./(ncol - 1)]}, Table[
   ColorLookup2[j, i] = Col2Num[ColSel[j, i][#]] & /@ ran,
   {j, 1, 4}, {i, colorNames}]];

(*color lookup function, presersing clipping and transparentcy. It \
converts integers to color values*)
(*returs the color function*)
ClearAll[LookUpTable3]
LookUpTable3[{lstyle_, color_}, {trans_, minclip_, maxclip_}, {pmin_, 
   pmax_}] := Module[{collist, fun, minc, maxc},
  (*generate color lookup table with correct clipping colors for min \
and max,add opacity for Image function*)
  collist = If[trans,
    {{1., 1., 1., 0.}}~Join~(ColorLookup2[lstyle, color])~
     Join~{{1., 1., 1., 0.}},
    {Col2Num[minclip]}~Join~(ColorLookup2[lstyle, color])~
     Join~{Col2Num[maxclip]}
    ];
  (*define the color function fun*)
  fun[x_?VectorQ] := collist[[x]];
  fun[x_?MatrixQ] := collist[[#]] & /@ x;
  fun
  ]

(*converts numbers to integers for color lookup*)
ClearAll[ColorRound2]
ColorRound2 = With[{n = ncol},
   Compile[{{x, _Real, 0}, {pmin, _Real, 0}, {pmax, _Real, 0}},
    If[pmin <= x <= pmax, 
     Round[(n - 1) ((x - pmin)/(pmax - pmin))] + 2, 
     If[x < pmin, 1, n + 2]],
    RuntimeOptions -> "Speed", RuntimeAttributes -> {Listable}]
   ];

(*the "fast" image function with the needed options*)
Options[FastImage] = {
   PlotRange -> Automatic,
   TransparentClipping -> False,
   ClippingStyle -> {Black, White},
   ImageOpacity -> 1,
   ColorFunction -> "Rainbow",
   ColorFunctionOrder -> "Normal"
   };

FastImage[data_, OptionsPattern[]] := 
 Block[{cfo, cf, cl1, cl2, tcl, plr, op, imAll},
  cfo = OptionValue[ColorFunctionOrder] /. colfuncsR;
  cf = OptionValue[ColorFunction];
  {cl1, cl2} = OptionValue[ClippingStyle];
  tcl = OptionValue[TransparentClipping];
  plr = OptionValue[PlotRange] /. Automatic -> MinMax[data];
  op = OptionValue[ImageOpacity];

  imAll = 
   LookUpTable3[{cfo, cf}, {tcl, cl1, cl2}, plr][
    ColorRound2[data, plr[[1]], plr[[2]]]];
  SetAlphaChannel[
   Image[imAll[[All, All, ;; -2]], ColorSpace -> "RGB", 
    ImageSize -> 300], Image[op imAll[[All, All, {-1}]]]]
  ]

(*make the data and manipulate*)

im = 750;
data = ToPackedArray@N@Table[i, {i, 1, im}, {j, 1, im}];

{mi, ma} = MinMax[data];
ra = ma - mi;

Manipulate[
 FastImage[data,
  PlotRange -> {min, max},
  TransparentClipping -> trans,
  ClippingStyle -> {minCl, maxCl},
  ImageOpacity -> op,
  ColorFunction -> col,
  ColorFunctionOrder -> cfun]
 ,
 {{min, mi}, mi, Dynamic[max - 0.01 ra], ra/100},
 {{max, ma}, Dynamic[min + 0.01 ra], ma, ra/100},
 {col, colorNames},
 {{trans, False}, {True, False}},
 {{op, 1}, 0, 1},
 {minCl, Red},
 {maxCl, Blue},
 {cfun, colfuncs}]

Another possible approach for measuring evaluation&rendering time is by using the CellChangeTimes option of the printed cells.

My experiments with screen recording software show that the last value of the CellChangeTimes option of a printed cell reflects the time when this cell was fully prepared for rendering, not the time when its rendering was finished. Hence with this approach, one should output a dummy cell after the cell of interest, and take the last value of the CellChangeTimes option of this dummy cell as the time of finishing rendering of the previous cell. The complete evaluation&rendering time computed in this way agrees closely with what the screen recording software records and also with the content of WindowStatusArea after finishing the evaluation (when EvaluationCompletionAction->"ShowTiming" is set).

Then the following MMa.SE answers are likely to be of interest to you:

https://mathematica.stackexchange.com/a/184912/280 https://mathematica.stackexchange.com/a/135179/280 https://mathematica.stackexchange.com/a/14673/280

That's very interesting! Will definitely look further into this. It's a completely new approach to me evaluation computation and render times like this.

Thanks for the excellent ideas.

Indeed, but including the render time of the front end, it still makes a difference (40 or 4 frames per second).

Hi Alexey,

Thanks for the input! Yes, I noticed that rasterizing graphics removes the problem. (Image is basically Rasterize[Graphics[]] as far as I get it at least).

However, I would really like to keep using the Head Graphics, since I use many of the options later that Graphics has but Image does not.

Furthermore, all the plot functions including Image are slow compared to Raster and their performance does not really scale well with the image size. I'm using this code in a Manipulate for dynamic interaction where one really notices the difference in speed.

TableForm[
 Table[
  data = RandomReal[{-.1, 1.1}, {im, im}];

  graphics = 
   Graphics[Raster[data, ColorFunction -> "SunsetColors"]] // 
     RepeatedTiming // First;
  array = 
   ArrayPlot[data, ColorFunction -> "SunsetColors", 
      ColorFunctionScaling -> False, Frame -> False] // 
     RepeatedTiming // First;
  image1 = 
   Colorize[Image[data], ColorFunction -> "SunsetColors"] // 
     RepeatedTiming // First;
  image2 = 
   Image[Reverse@Map[ColorData["SunsetColors"][#] &, data, {2}]] // 
     RepeatedTiming // First;

  Round[{im, 1000 graphics, array, image1, image2}, .0001]
  , {im, 50, 500, 50}],
 TableHeadings -> {None, {"Image\nDimensions", "Graphics\n[ms]", 
    "Array\n[s]", "Image\nColorize [s]", "Image\nMap [s]"}}]

Diving into the issue further, it seems there is more wrong with the meshing of Rasters in ArrayPlot.

They are antialiasing artifacts from drawing the raster as a set of primitives. The simplest way to deal with this is to add ImageResolution->96 to each Raster.