Hi Marco,

I guess looking at the emails between myself and Wolfram Technical Support that it was StackExchange that accepted it as a bug. Wolfram Technical Support: "This particular issue has been filed as a report and our development group is working on solving it for a future release. The introduction of this behavior comes from the reworking of CSV importing in Mathematica 11.2 and, as some of your posts mention, the workaround is to not specify {"Data", All} when all of the data needs to be imported."

I replied back that this was a broader issue, asked that it be raised to a higher supervisor, and copied Steven Wolfram on the email chain:

"If you export/import CSV data in Mathematica 11.2, no truncation of rows happens and the TextDelimiters parameter is not needed. But all CSV files exported in earlier releases of Mathematica, like release 11.1, are subject to hidden row truncation. So all user CSV files storing critical data are at risk. You can't tell 100,000s of existing Mathematica users, to update every program that imports CSV files created from previous Mathematica releases. You have to fix this. The default behavior should be, all existing Mathematica import/export CSV works as it always has, and newer Mathematica CSV functionality can be utilized by adding additional parameters. Not the other way around.

The same is true with the long elapsed time issue.

Please raise this to a higher supervisor.

You have to fix this.

Just some additional comments on this issue:

1. You should halt all further downloads of Mathematica 11.2 until this issue is fixed in Mathematica 11.3

2. This issue may apply to other file types, not just CSV files. You are going to have to test that.

3. If you have a multimillion dollar financial or drug company that heavily relies on and uses Mathematica for simulations and financial forecasts, they should not rollout Mathematica 11.2 in their production and tests environments until this issue is fixed. You have no idea what hidden data truncation will have on simulations and financial data. And you can't take that chance, whether it's one file, one user, or hundreds of thousands.

4. If I export 10,000 rows of data, I better get back 10,000 rows of data on import. Whether a data field has missing or questionable values is not the issue. Most coding will filter or throw out bad data. But an export/import should return all rows. "

This issue is now recognized as a bug in Mathematica 11.2.

See the below link:

Bug Introduced In Mathematica 11.2

Hi Ivan, On your first reply, I've been using that version of the Import statement on CSV datasets for a while, across different versions of Mathematica with no issues, until Mathematica 11.2. There were two issues with the Import statement under that newer release. The first was that it truncated rows of data on an import without any notification, unless you were looking for it with a Length function. The"TextDelimiters"->"" fixed that problem in Mathematica 11.2, but it should of never occurred, and will impact other users, without them realizing it. The second issue is the 100 times increase in elapsed time. I've just opened a bug report with WolfRam.

Dear Joseph,

I only post this because it shows how I identified the problem. The next post gives a potential solution:

I can, to some extend reproduce what you describe. It would have been useful to have your code in a code box to avoid typing it again:

AbsoluteTiming[Length[fix4 = Import["~/Desktop/allStarData4.csv", {"Data", All}, "HeaderLines" -> 1] /. 
Evaluate[ToExpression /@ Table["c" <> ToString[i] <> "_", {i, 1, 22}] -> ToExpression /@ Table["c" <> ToString[i], {i, 1, 22}]]]]

Note that the two examples you show in your post are slightly different: the first one contains the

"TextDelimiters" -> ""

option which gives an error in MMA 11.1.1 on my machine. Here are my results:

and

On 11.2 it give an error message and takes about 30 times longer to load. It also has only half of the entries. On MMA11.2 this here

AbsoluteTiming[Length[fix4 = Import["~/Desktop/allStarData4.csv", {"Data", All}, "HeaderLines" -> 1]]]

gives the same results as the upper command, but in less than 28 seconds; and it also only leads to 5002 rows.

Your "fixed" MMA 11.2 code:

AbsoluteTiming[Length[fix4 = Import["~/Desktop/allStarData4.csv", {"Data", All}, "HeaderLines" -> 1, "TextDelimiters" -> ""] /. 
Evaluate[ToExpression /@ Table["c" <> ToString[i] <> "_", {i, 1, 22}] -> ToExpression /@ Table["c" <> ToString[i], {i, 1, 22}]]]]

gives an error message and runs for 336 seconds.

The Import probably takes long because it is gigantic:

fix4[[1]]

gives

and

fix4 // ByteCount

gives 77724720080.

By the way, does exporting the data work for you in MMA11.2 directly?

Export["~/Desktop/allStarData412.csv", fix4]

takes excessively long on my machine, probably because it is enormous. In fact, I interrupted it after about 20 minutes without success.Can anyone check whether this works on their machines?

I also noted that SemanticImport does success in importing the file and is relatively fast:

AbsoluteTiming[Length[fix5 = SemanticImport["~/Desktop/allStarData4.csv", "HeaderLines" -> 1]]]

gives

The HeaderLine option doesn't make any difference so you can delete it in this case. If you look at the output you can recover your data, but it misinterpreted the header.

Best wishes,

Marco