Message Boards Message Boards

FITS Import incompatible between 11.2 and earlier versions

GROUPS:

I was happy to note that the FITS Im/Export got some attention in the lates Update to Version 11.2. However, Import is now incompatible to its earlier incarnations. I am not sure wether I discovered all changes in its behavior and I am asking here for any experience others got with Import of FITS files.

This is what I found:

  • Header, i.e. Metadata, information is now imported as an Association with the first key "GENERAL INFORMATION" . This is far from any FITS standard. Compare the header import from 11.1.1 and 11.2:

11.1.1

{"SIMPLE" -> {True, ""}, "BITPIX" -> {-32, ""}, "NAXIS" -> {4, ""}, 
 "NAXIS1" -> {121, ""}, "NAXIS2" -> {121, ""}, "NAXIS3" -> {221, ""}, 
 "NAXIS4" -> {1, ""}, "DATAMIN" -> {0., ""}, "DATAMAX" -> {0., ""}, 
 "BUNIT" -> {"K (Tmb)", ""}, "CTYPE1" -> {"RA---GLS", ""}, 
 "CRVAL1" -> {306.861, ""}, "CDELT1" -> {-0.00166667, ""}, 
 "CRPIX1" -> {61.6741, ""}, "CROTA1" -> {0., ""}, 
 "CTYPE2" -> {"DEC--GLS", ""}, "CRVAL2" -> {37.38, ""}, 
 "CDELT2" -> {0.00166667, ""}, "CRPIX2" -> {61.5654, ""}, 
 "CROTA2" -> {0., ""}, "CTYPE3" -> {"VRAD", ""}, "CRVAL3" -> {0., ""},
  "CDELT3" -> {500., ""}, "CRPIX3" -> {101., ""}, 
 "CROTA3" -> {0., ""}, "CTYPE4" -> {"", ""}, "CRVAL4" -> {1., ""}, 
 "CDELT4" -> {1., ""}, "CRPIX4" -> {1., ""}, "CROTA4" -> {0., ""}, 
 "OBJECT" -> {"S106", ""}, "RA" -> {306.861, "Right Ascension"}, 
 "DEC" -> {37.38, "Declination"}, "EQUINOX" -> {2000., ""}, 
 "LINE" -> {"CII", ""}, "ALTRPIX" -> {101., ""}, 
 "ALTRVAL" -> {1.90054*10^12, ""}, "RESTFREQ" -> {1.90054*10^12, ""}, 
 "VELO-LSR" -> {0., ""}, "VELREF" -> {257, ""}, 
 "SPECSYS" -> {"LSRK", ""}, "BMAJ" -> {0.00416667, ""}, 
 "BMIN" -> {0.00416667, ""}, "BPA" -> {0., ""}, 
 "ORIGIN" -> {"GILDAS Consortium", ""}, 
 "DATE" -> {"2017-06-01T10:55:26.668", "Date written"}}

11.2

<|"GENERAL INFORMATION" -> <|"SIMPLE" -> True, "BITPIX" -> -32, 
   "NAXIS" -> 4, "NAXIS1" -> 121, "NAXIS2" -> 121, "NAXIS3" -> 221, 
   "NAXIS4" -> 1, "DATAMIN" -> 0., "DATAMAX" -> 0., 
   "BUNIT" -> "K (Tmb)", "CTYPE1" -> "RA---GLS", "CRVAL1" -> 306.861, 
   "CDELT1" -> -0.00166667, "CRPIX1" -> 61.6741, "CROTA1" -> 0., 
   "CTYPE2" -> "DEC--GLS", "CRVAL2" -> 37.38, "CDELT2" -> 0.00166667, 
   "CRPIX2" -> 61.5654, "CROTA2" -> 0., "CTYPE3" -> "VRAD", 
   "CRVAL3" -> 0., "CDELT3" -> 500., "CRPIX3" -> 101., "CROTA3" -> 0.,
    "CTYPE4" -> Missing["NotAvailable"], "CRVAL4" -> 1., 
   "CDELT4" -> 1., "CRPIX4" -> 1., "CROTA4" -> 0., "OBJECT" -> "S106",
    "RA" -> 306.861, "DEC" -> 37.38, "EQUINOX" -> 2000., 
   "LINE" -> "CII", "ALTRPIX" -> 101., "ALTRVAL" -> 1.90054*10^12, 
   "RESTFREQ" -> 1.90054*10^12, "VELO-LSR" -> 0., "VELREF" -> 257, 
   "SPECSYS" -> "LSRK", "BMAJ" -> 0.00416667, "BMIN" -> 0.00416667, 
   "BPA" -> 0., "ORIGIN" -> "GILDAS Consortium", 
   "DATE" -> "2017-06-01T10:55:26.668"|>|>

Note that the comment entry of each header is is now omitted (last entry "DATE" This violates the FITS standard as far as I understand? In any case it breaks my code that expects to import a List and not an Association - and yes I know that I can use Normal to transform it into a List again. I didn't check yet how the header is imported if data is stored in a more complex arrangement, such as Bintable or another Extension format.

  • Data is imported in reverse order now. Image and RawData Import now return the same row ordering. I don't know what the standard order should be, but it changed after 11.2. As an example I Import an array of spectra:

11.1.1

  In[9]:=Dimensions[raw]

  Out[9]:={1, 1, 221, 121, 121}

  In[10]:=raw[[1, 1, All, 50, 50]]

 Out[10]:= {0.125237, 2.99011, 0.571144, -0.834969, 1.56174, 0.110322, -1.4104, \
-2.62194, -1.9974, 1.81346, 1.41345, -0.346708, -0.284448, 1.31853, \
-1.36181, -3.43505, 2.01738, 3.3406, -0.353997, 3.28855, -0.4002, \
-0.415156, 2.57086, -0.826749, -0.493768, -1.06639, -0.795575, \
0.438714, 1.41906, -1.45034, -1.13682, -1.06962, 0.349537, -1.69687, \
-0.618968, -1.31552, -1.63609, -1.14806, -1.78705, -1.59089, \
-0.0172433, -1.34023, -1.68296, -0.169548, 0.525515, 2.61707, \
-0.10389, 1.23272, -1.37291, 2.35375, 1.26516, -0.9361, -2.47334, \
0.319303, 0.574864, -1.37799, 2.33915, 2.38415, -0.855974, 0.500247, \
1.93797, -0.510813, -1.4609, 1.37657, -1.31001, 1.82468, -0.0288664, \
0.788793, -0.0313442, 1.0192, 0.661234, -0.551474, 3.03574, \
-0.258694, 0.83997, -0.590292, -0.164056, -0.675589, 1.22346, \
1.87058, -0.421079, -0.670318, 0.758016, 1.17182, -1.83596, 0.130186, \
-1.47637, 0.0714399, -0.381047, -0.654595, 0.337103, 0.0541637, \
3.54652, 0.473411, -0.751777, 2.00137, 1.23871, 0.964236, 0.697787, \
0.745029, 1.64881, 3.73721, -1.57351, 0.0654506, 1.01008, -3.37733, \
0.536895, 1.05536, -2.57754, -0.439541, -1.92688, -1.43538, \
-0.580122, -2.33661, 1.18908, -0.270191, -1.23731, -1.61967, \
-2.18623, -1.54892, -2.25664, -1.60237, -1.10626, -3.27524, -2.8534, \
-1.30801, -2.94565, -1.62026, -1.18614, -1.68366, -3.44239, -4.29097, \
0.139907, 1.39606, 0.885509, 1.06719, -0.0184753, -1.556, 1.02081, \
-0.15279, 1.59421, -0.582037, 0.377889, -0.0525626, 1.52673, \
0.449664, 0.628198, 2.56478, 1.04696, -0.246007, -1.73447, 0.984217, \
1.40674, -1.03528, 1.74808, 0.661529, -1.55615, 2.12005, 0.0215735, \
-0.843181, -0.187671, 2.10022, 1.94109, 0.461266, -0.836785, \
-1.85344, 1.4057, -1.1845, -0.0455038, -2.07055, -1.30267, -0.957898, \
-0.813964, 0.0770546, 1.1605, -0.0788292, 0.352559, -2.4903, \
-0.0513624, 0.927214, -1.23293, -0.846625, -2.09944, -0.909728, \
-0.178735, -0.756856, -0.0954211, -2.28924, -0.616761, -0.295874, \
0.97779, -0.325473, 1.3985, 0.176696, -0.936313, 0.919589, 2.14291, \
-0.287233, 1.11923, 1.55756, 0.128433, 0.739541, -0.211153, 0.56651, \
2.30676, 0.714307, 1.81921, 0.0436757, 0.384643, 0.223706, 1.79563, \
-0.512, 1.53679, 1.84418, 0.788868, 2.69285, 0.688162, 1.14756, \
-0.599565, -1.12756, 0.227808}

11.2

In[11]:= raw[[1, 1, All, 50, 50]]

Out[11]= {0.741499, 2.69608, -0.835699, 0.260684, 0.344929, \
-0.862212, 0.103039, -0.406144, 1.25696, 2.21517, -3.61728, \
-0.390149, -0.20015, -0.221925, -0.133629, -0.127833, -0.537916, \
-1.6329, 0.529082, 2.89574, 2.174, 0.509376, -0.12798, -2.91033, \
-0.936878, 1.72893, -0.144214, -1.6564, -0.203739, 1.31158, 0.482649, \
-2.29575, 1.49968, -1.43943, -0.907162, 2.33743, -0.134048, -1.52299, \
0.700724, -1.32794, 0.0540263, -0.0130849, -1.3927, -0.358867, \
0.457883, 0.920364, 0.0981134, 0.398595, 2.10525, -1.26789, \
-0.120008, 1.87544, 1.49796, 1.14598, -0.445654, 0.0828433, -0.5025, \
0.48467, -0.903544, 0.721465, -1.19323, 1.8576, -0.913446, \
-0.0233543, -0.130562, 0.171933, -1.76813, 1.22534, -0.769345, \
0.596699, -1.53008, -0.806187, 0.376273, 0.274861, 0.879051, \
-0.285103, 0.652643, -2.05536, -1.18484, 1.23708, -0.524947, \
-1.10875, 0.632588, 0.362561, 0.521122, 1.09359, 2.34802, 0.219165, \
0.528764, 3.89038, 2.3092, 4.25087, 2.91008, 3.86056, 4.43583, \
5.34523, 3.33701, 5.04292, 6.16093, 3.11196, 4.95334, 3.44107, \
3.13834, 1.72268, 1.4871, 2.09813, -1.24582, 1.52399, 1.33164, \
-1.41426, -0.438202, 1.09399, 1.99072, -0.0511786, 1.6797, 1.81019, \
0.234106, 0.0070522, -0.191584, -1.71418, 0.300965, 1.56744, 2.86724, \
-0.710091, -0.042246, 0.0861327, 0.404054, 0.245493, 1.15148, \
-0.107975, -0.609298, -0.358749, -0.93885, -0.575624, 1.58488, \
0.605153, -0.148833, 1.31476, 0.886797, -0.500806, 0.690805, 1.55135, \
-2.82947, -1.44928, 0.0264999, 0.0159517, 0.381241, -0.710363, \
0.308564, -0.362335, 0.384097, 0.0797061, -0.522967, -2.15822, \
0.146816, 1.26085, 1.6938, 0.86811, 1.65988, -0.939591, -0.786369, \
-0.473823, 0.299186, 2.18688, -0.564252, -1.95808, -0.297, \
-0.00392977, 0.0266746, -0.849348, 1.03588, 0.392564, 0.383135, \
0.955122, -0.398982, 0.96028, 0.159508, -0.806305, -0.292513, \
0.168193, 1.60123, 0.0350197, -2.45768, -0.809964, -0.966695, \
0.389624, -0.296024, 1.07444, -0.276236, -0.0449804, 0.135233, \
0.994279, -0.915011, -0.196694, 0.568144, -0.0751389, 0.0356748, \
-1.49772, -2.33957, 2.12178, -0.122009, 1.63197, 1.28452, -1.27409, \
1.67394, -0.64527, -1.91461, 0.0593593, -0.0333713, -0.355916, \
0.164881, 1.38631, 2.90664, 1.8102, -0.422445, 0.256066, 0.393217, \
1.07279, 2.3621, -0.935344, 1.38919}

In[12]:=  raw[[1, 1, All, -50, 50]]

Out[12]:= {0.125237, 2.99011, 0.571144, -0.834969, 1.56174, 0.110322, -1.4104, \
-2.62194, -1.9974, 1.81346, 1.41345, -0.346708, -0.284448, 1.31853, \
-1.36181, -3.43505, 2.01738, 3.3406, -0.353997, 3.28855, -0.4002, \
-0.415156, 2.57086, -0.826749, -0.493768, -1.06639, -0.795575, \
0.438714, 1.41906, -1.45034, -1.13682, -1.06962, 0.349537, -1.69687, \
-0.618968, -1.31552, -1.63609, -1.14806, -1.78705, -1.59089, \
-0.0172433, -1.34023, -1.68296, -0.169548, 0.525515, 2.61707, \
-0.10389, 1.23272, -1.37291, 2.35375, 1.26516, -0.9361, -2.47334, \
0.319303, 0.574864, -1.37799, 2.33915, 2.38415, -0.855974, 0.500247, \
1.93797, -0.510813, -1.4609, 1.37657, -1.31001, 1.82468, -0.0288664, \
0.788793, -0.0313442, 1.0192, 0.661234, -0.551474, 3.03574, \
-0.258694, 0.83997, -0.590292, -0.164056, -0.675589, 1.22346, \
1.87058, -0.421079, -0.670318, 0.758016, 1.17182, -1.83596, 0.130186, \
-1.47637, 0.0714399, -0.381047, -0.654595, 0.337103, 0.0541637, \
3.54652, 0.473411, -0.751777, 2.00137, 1.23871, 0.964236, 0.697787, \
0.745029, 1.64881, 3.73721, -1.57351, 0.0654506, 1.01008, -3.37733, \
0.536895, 1.05536, -2.57754, -0.439541, -1.92688, -1.43538, \
-0.580122, -2.33661, 1.18908, -0.270191, -1.23731, -1.61967, \
-2.18623, -1.54892, -2.25664, -1.60237, -1.10626, -3.27524, -2.8534, \
-1.30801, -2.94565, -1.62026, -1.18614, -1.68366, -3.44239, -4.29097, \
0.139907, 1.39606, 0.885509, 1.06719, -0.0184753, -1.556, 1.02081, \
-0.15279, 1.59421, -0.582037, 0.377889, -0.0525626, 1.52673, \
0.449664, 0.628198, 2.56478, 1.04696, -0.246007, -1.73447, 0.984217, \
1.40674, -1.03528, 1.74808, 0.661529, -1.55615, 2.12005, 0.0215735, \
-0.843181, -0.187671, 2.10022, 1.94109, 0.461266, -0.836785, \
-1.85344, 1.4057, -1.1845, -0.0455038, -2.07055, -1.30267, -0.957898, \
-0.813964, 0.0770546, 1.1605, -0.0788292, 0.352559, -2.4903, \
-0.0513624, 0.927214, -1.23293, -0.846625, -2.09944, -0.909728, \
-0.178735, -0.756856, -0.0954211, -2.28924, -0.616761, -0.295874, \
0.97779, -0.325473, 1.3985, 0.176696, -0.936313, 0.919589, 2.14291, \
-0.287233, 1.11923, 1.55756, 0.128433, 0.739541, -0.211153, 0.56651, \
2.30676, 0.714307, 1.81921, 0.0436757, 0.384643, 0.223706, 1.79563, \
-0.512, 1.53679, 1.84418, 0.788868, 2.69285, 0.688162, 1.14756, \
-0.599565, -1.12756, 0.227808}

I am not asking how to correct this - I can do it. I am curious on why this was implemented and what else had been changed. I would also like to see a mentioning of such changes in the Documentation. Any other experiences so far?

POSTED BY: Markus Roellig
Answer
9 days ago

Hi Markus,

Thanks for the interest in FITS.

As you already have noticed we switched from list of rules to associations, since we believe it's easier to extract data from an association, but as you also mentioned, you could always use Normal over association to make your old code work. To get exactly the same output as in 11.1, you could just use the legacy element: Import[file, "MetadataLegacy"]. One benefit you will notice in upcoming versions, is an indexed association and not a list of associations as an output of metadata, which allows you to easily associate the data to the HDU it was extracted from.

Previously you could see the values in the metadata being lists of a form {value, comment}, which has been replaced by value in 11.2, but we are bringing back the original output design of values in upcoming versions with a slight modification; now you will be having meaningful Wolfram Language objects instead of just empty strings (e.g. " " will be shown as Missing["NotAvailable"]); "RawMetaInformation" will still be giving the unprocessed version of the output.

Introducing "GENERAL INFORMATION" as a general key to metadata was motivated by changes described above, and the fact that association can't hold duplicate keys. For files with richer metadata this makes a lot of sense; let's take a look at one example:

Import["w0bs0102t_c0h.fit", "Metadata"]//Keys
(* ==> {{"GENERAL INFORMATION", "WFPC DATA DESCRIPTOR KEYWORDS", "SCIENCE INSTRUMENT CONFIGURATION", "FILTER CONFIGURATION", "EXPOSURE TIME AND RELATED INFORMATION", "TARGET INFORMATION", "INSTRUMENT STATUS USED IN DATA PROCESSING", "IMAGE TYPE CHARACTERISTICS", "RSDP CONTROL KEYWORDS", "CALIBRATION REFERENCE FILES", "DEFAULT KEYWORDS SET BY STSCI", "PLANETARY SCIENCE KEYWORDS", "PODPS FILL VALUES", "MISC KEYWORDS", "DMF EXPOSURE DATA"}, {"GENERAL INFORMATION"}} *)

Which looks as follows in upcoming versions:

Import["w0bs0102t_c0h.fit",  {"MetaInformation",  1}]//Keys
(* ==> {"GENERAL INFORMATION", "WFPC DATA DESCRIPTOR KEYWORDS", "SCIENCE INSTRUMENT CONFIGURATION", "FILTER CONFIGURATION", "EXPOSURE TIME AND RELATED INFORMATION", "TARGET INFORMATION", "INSTRUMENT STATUS USED IN DATA PROCESSING", "IMAGE TYPE CHARACTERISTICS", "RSDP CONTROL KEYWORDS", "CALIBRATION REFERENCE FILES", "DEFAULT KEYWORDS SET BY STSCI", "PLANETARY SCIENCE KEYWORDS", "PODPS FILL VALUES", "MISC KEYWORDS", "DMF EXPOSURE DATA"} *)

You could see that the first HDU of the file contains metadata with multiple domains of key-value pairs; "GENERAL INFORMATION" is here, in a sense, for consistency, when all other keys above are present in the file. If we wouldn't have done this, some metadata would have been ignored:

DuplicateFreeQ[Flatten[Normal[Import["w0bs0102t_c0h.fit",  {"MetaInformation",  1}]//Values]]//Keys]
(* ==> False *)

This is just one example, you could in fact find more. Concerning the "dimensions"; to be short and to admit the truth, we were doing it incorrectly in 11.1, and just fixed it in 11.2 :) Upcoming versions will introduce a new option which is called DataReversed to control the dimensions of the import and/or export.

Upcoming versions will introduce a lot of interesting design decisions that will hopefully make FITS manipulations in WL easier and more intuitive; of course all of them will be nicely documented.

Last, but not least, upcoming versions will introduce full import and export support for all types of FITS compression, as well as all types of HDU.

Please let me know if you have any other questions and/or concerns, and I will do my best to guide you through.

Cheers, Mikayel

POSTED BY: Mikayel Egibyan
Answer
8 days ago

Hi Mikayel,

thank you for the reply. Looking forward to 11.3 then :)

Markus

POSTED BY: Markus Roellig
Answer
7 days ago

Group Abstract Group Abstract