At the moment in Mathematica 13 the Dataset implementation is still less efficient than we'd like, but we are working on it.

Am I missing something, or is the test "compare" actually showing that the Association returned by AssociationThread is the memory-hungry culprit? I don't see Dataset taking up much more memory:

In[23]:= l1 = RandomInteger[1000000, {100000, 4}];
l2 = AssociationThread["Columns" <> ToString@# & /@ Range[4] -> #] & /@
    l1;
ds = Dataset[l1];

In[26]:= ByteCount /@ {l1, l2, ds}

Out[26]= {3200152, 67200080, 3200656}

Hi Stefan, the point is that there is no way to represent tabular data using Dataset otherwise, but Taliesin and his team are already aware of it. I imagine they are working on something that, when creating a Dataset for tabular data (as using SQL, CSV import or a regular nxm list with heads), the way to handler the data would be exactly as it works in the current designe, but the way it use memory would be as efficient as a regular packed array.

The AllowedHeads option in Transpose takes care of this:

Transpose[<|"col1" -> {1, 2, 3}, "col2" -> {x, y, z}|>, AllowedHeads -> All]

It also works with Dimensions:

In[38]:= Dimensions[<|a -> {1, 2}, b -> {3, 4}|>]

Out[38]= {2}

In[39]:= Dimensions[<|a -> {1, 2}, b -> {3, 4}|>, AllowedHeads -> All]

Out[39]= {2, 2}

I believe the design for it wasn't quite ready for M10, which is why it's currently undocumented, but it's there and it works (as far as I know). Please note that like any undocumented symbol, this might change in a future version.

For me, it's strange the column redundant implementation of Dataset, that do not allow a memory efficient representation of a simpler tabular data format (as SQL data).

As each column name is in each line, is't not memory efficient.

See this test:

compare[lines_,columns_]:=Module[{l1,l2},
    l1=RandomInteger[1000000,{lines,columns}];
    l2=AssociationThread["Columns"<>ToString@#&/@Range[columns]-> #]&/@l1;
    ByteCount[l2]/ByteCount@Dataset[l1]//N
]
compare[100000,4]

19.9959

I get approximately 20x more memory for it!..

I know that Dataset is a cool tool for Hierarchical data structure. But I believe that 95% of data problems are still rectangular (all SQL queries are), and the current format of Dataset is very memory inefficient for that.

Maybe Dataset could have a option TabularData-> True, so:

ds=Dataset[<|"col1" -> {1, 2, 3, 4}, "col2" -> {"a", "b", "c", "d"}|>, TabularData-> True]

Here we can check that this structure do not need more memory:

compare[lines_,columns_]:=Module[{l1,l2},
    l1=RandomInteger[1000000,{lines,columns}];
    l2=Thread["Columns"<>ToString@#&/@Range[columns]-> Transpose@l1];
    ByteCount[l2]/ByteCount@Dataset[l1]//N
]
compare[100000,4]

1.0007

There are some Wolfram plans better handler Tabular Data in Dataset?