Message Boards Message Boards

GROUPS:

Speed problem: Import and post-process 1000 images

Posted 11 months ago
1114 Views
|
6 Replies
|
2 Total Likes
|

Hi community :-)

I have 1024 csv-files in my folder. Each csv-file represents an 16-bit image with 1024 x 1024 (rows and columns).

My goal: Import the images, apply ImageTrim[] to every image to get rid of the unnecessary parts (the same for every image). Afterwards calculate the total power of each image. Finally, save this result. Repeat this for every image.

Problem: Import takes very long (20-30 minutes). Post-Processing takes another 20 minutes.

Question: How can I optimize my code?

Read file names and create temporary file to save results:

SetDirectory["C:\\Users\\images"];  (*path to the folder where all the images are stored*)
NameofFiles = FileNames[]; (*lists all files in the current working directory.*)
NumberofFiles = Length[FileNames["*"]]; (*number of files in the current working directory.*)
file = OpenWrite["file.dat", FormatType -> OutputForm](*Opens a file called file.dat which is used later to temporary save data*)

Loop in which I import every single csv-file and calculate and save intensity:

Do[
 ImagesCSV = Import[NameofFiles[[t]]] ;    (*import file*)
 Img = Image[ImagesCSV, "Bit16"];  (*transforms image to csv*)
 ImgTrim = ImageTrim[Img, {{150, 130}, {1000, 890}}];  (*trimms the image*)
ImageTrimmed = ImageMeasurements[ImgTrim, "Total"]; (*calculate total intensity of trimmed image*)
Write[file, ImageTrimmed];  (*writes result into temporary file*)
, {t, 1, NumberofFiles}]

Now it try to close my temporary file: file.dat (I think it doesn't really close because when I try to delete it with DeleteFile[] it doesn't really work)

The code works fine, but is very slow. Any ideas how to speed it up?

Peter

EDIT: For some reason the first step (read file names...) seems to be significantly faster today. Maybe I have to quit the kernel before I run the code. To loop takes now 15-20 minutes which I think is reasonable. So, seems that today is much faster then yesterday. Anyway, do you have ideas to improve the code in terms of speed? If not, I think I can live with 20 minutes processing time.

Attachments:
6 Replies

Please provide a sample file to benchmark on.

Posted 11 months ago

Hi

The complete code is already included in my first post. Additionally I have attached 2 (out of 1000) csv-files to it.

I was unable to get ReadList to handle commas as separators, but the following at least provides a ~2.3x speed boost to importing on my computer.

csvRead[file_] :=
 Module[{stream, data},
  stream = StringToStream@StringReplace[ReadString[file], "," -> "\t"];
  data = ReadList[stream, Real, RecordLists -> True];
  Close[stream];
  data
  ]

In[59]:= csvRead["~/Downloads/11.4449.csv"]; // AbsoluteTiming

Out[59]= {0.664697, Null}

In[60]:= Import["~/Downloads/11.4449.csv", "CSV"]; // AbsoluteTiming

Out[60]= {1.57064, Null}
Posted 11 months ago

Hi Szabolcs

Indeed, this technique is definitely faster. Thanks a lot. I think with this improvement I can now say, the speed is reasonably okay. Yesterday it was quite slow but with your code and "Quit Kernel" the speed is much higher.

Thanks a lot :-)

Peter

Actually, a speedup factor of 2 isn't very great. I would love to see a faster solution.

Posted 11 months ago

Hahaha, true :-) It is not very fast. I am sure it can be done much faster. But how much effort you have to put into this just to be a little bit faster....? Is it worth it? But I agree, would be interesting to see and I am very curious about it.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract