I really don't know if I understand what you want to do with the image...
I made an attempt below, see if it is useful in any way or if it has nothing to do with the goal (?):
im = ColorConvert[image, "RGB"];
a = Delete[Map[RGBColor[#] &, Tuples[{1, 0}, 3]], {{1}, {-1}}];
a1 = ColorQuantize[im, a];
swap = {a[[1]] -> a[[3]], a[[3]] -> a[[1]], a[[2]] -> a[[5]],
a[[5]] -> a[[2]], a[[4]] -> a[[6]], a[[6]] -> a[[4]]};
rule = {RGBColor[{1, 1, 0}] -> Black, RGBColor[{1, 0, 1}] -> Black,
RGBColor[{0, 1, 1}] -> Black};
left = ColorReplace[a1, rule];
right = ColorReplace[ColorReplace[a1, swap], rule];
{Grid[{{"", "left", ""}, {Black, Green, Black}, {Red, Black, Blue}},
Frame -> All],
Grid[{{"", "right", ""}, {Red, Black, Blue}, {Black, Green, Black}},
Frame -> All]}
{{"left", Show[left, ImageSize -> Medium]}, {"right",
Show[right, ImageSize -> Medium]}}
A small sample with the same pixel range for the two images:
interval = {484175 ;; 484185}[[1]];
{PixelValue[left, {All, All}][[interval]],
Map[RGBColor[#] &, PixelValue[left, {All, All}][[interval]]]}
{PixelValue[right, {All, All}][[interval]],
Map[RGBColor[#] &, PixelValue[right, {All, All}][[interval]]]}
Sorry if I couldn't help you (I think maybe I really couldn't understand)..