According to the Documentation page for ImageCompose, it uses "standard Duff-Porter compositing operators" with the default being the "over" operator.

However my experiments show that in the recent versions of Mathematica (up to version 11.2.0) alpha blending in ImageCompose is implemented incorrectly: the developer has made the most typical and very popular mistake when implementing the Duff-Porter's "over" operator by forgetting to return from pre-multiplied color channel values to the straight color channel values (native for Mathematica's Image), a proof follows.

Theoretical background

As a reference for correct definition of the standard "A over B" operation we can use the original paper by Thomas Porter and Tom Duff "Compositing Digital Images" (1984) where they give on page 256 the following definition for the case when pre-multiplied (!) colors are used:

$c_O = c_A + c_B (1 - \alpha_A),$

where $\alpha_A$ is alpha channel value of image A, $c_A$ and $c_B$ are pre-multiplied (!) color channel values of images A and B correspondingly, and $c_O$ is the final pre-multiplied (!!!) color channel value.

The most common mistake when implementing the above definition is to forget that $c_O$ is pre-multiplied color which must be divided by the new alpha channel value $\alpha_O$ in order to obtain straight color:

$\alpha_O = \alpha_A + \alpha_B (1 - \alpha_A).$

Note that Porter and Duff [1984] don’t provide this last formula, they simply assume pre-multiplied colors. An elaborated and very enlightening discussion on this topic can be found in recent paper

• Glassner, Andrew (2015), "Interpreting Alpha", Journal of Computer Graphics Techniques 4 (2): 30–44.

where the last formula is provided as formula (2) on page 33 (thanks to Charles Poynton for the reference!).

Mathematica's implementation

Starting from version 11.1 we can implement the correct definition for the "A over B" operation as follows:

imageCompose[b_Image, a_Image] :=
 Module[{alphaA = AlphaChannel@a, alphaB = AlphaChannel@b, alphaO,
         cA = RemoveAlphaChannel@a, cB = RemoveAlphaChannel@b},
  alphaO = 1 - (1 - alphaA) (1 - alphaB);
  SetAlphaChannel[(alphaA*cA + (1 - alphaA) alphaB*cB)/alphaO, alphaO]]

And the incorrect definition (without division by $\alpha_O$) can be implemented as follows:

imageComposeWrong[b_Image, a_Image] :=
 Module[{alphaA = AlphaChannel@a, alphaB = AlphaChannel@b, alphaO,
         cA = RemoveAlphaChannel@a, cB = RemoveAlphaChannel@b},
  alphaO = 1 - (1 - alphaA) (1 - alphaB);
  SetAlphaChannel[alphaA*cA + (1 - alphaA) alphaB*cB, alphaO]]

It is easy to ensure that imageComposeWrong is equivalent to the current ImageCompose (with tiny differences due to rounding-off errors):

a = Image[{{{.1, .3, .7, .8}}}, ColorSpace -> "RGB"];
b = Image[{{{.7, .3, .1, .4}}}, ColorSpace -> "RGB"];
ImageData[ImageCompose[a, b]] == ImageData[imageComposeWrong[a, b]]

True

I hope that my report will help to improve Mathematica by fixing the above-described bug in ImageCompose. Note also that I've checked only the "over" operator, other compositing operators supported by ImageCompose can also be affected.

P.S. I discuss this issue in more details on Mathematica.SE. It is worth to mention that Overlay and Graphics currently use the correct implementation of the OVER compositing operator.

_{Reported to the support as [CASE:3967986].}