Dear All,
I have the following binary file format:
- Every 'frame' starts with an "UnsignedInteger32" which gives the number of datapoints.
Then for each datapoint of that frame it is the following form:
- number of cameras as "UnsignedInteger8" (call it ncam)
- position x,y,z and distance d as "Real32"
- ncam times an "UnsignedInteger8" followed by "UnsignedInteger16" (e.g. ncam 2 => "UnsignedInteger8", "UnsignedInteger16","UnsignedInteger8", "UnsignedInteger16")
As you can see the datapoints have to be read one-by-one, as the first number of the datapoint determines the length of that datapoints (which varies between 2 and 3 in my case, but could be higher.
so each datapoint is the following sequence of data-types:
- ncam=2: "UnsignedInteger8", "Real32", "Real32", "Real32", "Real32", "UnsignedInteger8", "UnsignedInteger16", "UnsignedInteger8", "UnsignedInteger16"
- ncam=3: "UnsignedInteger8", "Real32", "Real32", "Real32", "Real32", "UnsignedInteger8", "UnsignedInteger16", "UnsignedInteger8", "UnsignedInteger16", "UnsignedInteger8", "UnsignedInteger16"
I devised the following code to read the data:
ClearAll[ncamString]
ncamString[n_Integer]:=ncamString[n]=Join[{"Real32","Real32","Real32","Real32"},Join@@ConstantArray[{"UnsignedInteger8","UnsignedInteger16"},n]]
str=OpenRead["trial.png",BinaryFormat->True];
n=BinaryRead[str,"UnsignedInteger32"];
Print["Reading frames\[Ellipsis]"];
j=0;
start=AbsoluteTime[];
Dynamic[j]
alldata=Reap[
While[n=!=EndOfFile,
j++;
data=Table[
cams=BinaryRead[str,"UnsignedInteger8"];
other=ncamString[cams];
other=BinaryRead[str,other];
Prepend[other,cams]
,
{n}
];
Sow[data];
n=BinaryRead[str,"UnsignedInteger32"] (* for next frame *)
]
][[2,1]];
Close[str];
elapsed=AbsoluteTime[]-start;
Print["Reading ",Length[alldata]," frames took: ",elapsed," sec. Or ",elapsed/Length[alldata]," sec/frame."]
This takes roughly 40 milliseconds per frame (or roughly 500KB/s which is very low for an SSD that can do 3 GB/s, yes capital b!). I have attached the test-file with just 5 frames with the extension .png only because the forum does not allow for .bin files.
If I run RuntimeTools`Profile on my code it shows that ~90% of the time is spent on the many many BinaryRead calls. Is there a faster way to 'read in' the entire data in to memory first an then to interpret from there? Something like StringToStream... Or an alternative to BinaryRead
Attachments: