Message Boards

WOLFRAM COMMUNITY

9845 Views

9 Replies

0 Total Likes

View groups...

Follow this post

Share this post:

GROUPS:

Import and Export Wolfram Language

Merging Large Text Files Without Reading Entire File

Bob Stephens

Posted 10 years ago

I have some large data files that each contain 2 columns of data as shown below (strings in row are separated by spaces). I need to merge these files into one large file that contains 4 columns each separated by tabs. Is there an efficient built-in function that will let me do this using file streams without the need to read these large files in completely? thanks input1.txt 613 -112 1737 1685 1443 891 -39 273 1761 356 2007 -8 -198 -882 -1634 input2.txt 1044 249 -716 207 -841 989 1076 1446 899 969 -1574 1003 -1833 1192 241 -406 804 -1836 Desired output file 613 -112 1044 249 1737 392 -716 207 1685 -176 -841 989 1443 -813 1076 1446 891 -39 899 969 273 1761 -1574 1003 356 2007 -1833 1192 -8 -198 241 -406 -882 -1634 804 -1836

POSTED BY: Bob Stephens

9 Replies

Sort By:

Bob Stephens

Posted 10 years ago

sorry, missed the export to a file........thanks for your help

POSTED BY: Bob Stephens

Bob Stephens

Posted 10 years ago

Yes, here is another solution I came up with - last question, what is the best way to export this result to a file if we read everything in at once?

POSTED BY: Bob Stephens

Bob Stephens

Posted 10 years ago

Looks like the issue is that each row is currently a string and so the Join function is not getting the number of expected elements

POSTED BY: Bob Stephens

Henrik Schachner

Henrik Schachner, Radiation Therapy Center, Weilheim, Germany

Posted 10 years ago

A direct (but not very elegant) way could be: ClearAll["Global`*"] SetDirectory[NotebookDirectory[]]; input1 = Import["first.txt", "Data"]; input2 = Import["second.txt", "Data"]; result = Join[input1, input2, 2]; Export["result.txt", StringReplace[ ToString[result], {"}, {" -> "\n", ", " -> "\t", "{{" -> "", "}}" -> ""}]]; But I have to confess that this is not solving your primary question: This way the entire input files have to be red in first. Henrik

A direct (but not very elegant) way could be:

ClearAll["Global`*"]
SetDirectory[NotebookDirectory[]];
input1 = Import["first.txt", "Data"];
input2 = Import["second.txt", "Data"];
result = Join[input1, input2, 2];
Export["result.txt", 
  StringReplace[
   ToString[result], {"}, {" -> "\n", ", " -> "\t", "{{" -> "", 
    "}}" -> ""}]];

But I have to confess that this is not solving your primary question: This way the entire input files have to be red in first.

Henrik

POSTED BY: Henrik Schachner

Bob Stephens

Posted 10 years ago

Adding sample files to discussion. Attachments:

POSTED BY: Bob Stephens

Bob Stephens

Posted 10 years ago

So reading in the enclosed files and trying the join on the lists resulted in an error file1List = ReadList[ "first.txt", Number]; file2List = ReadList[ "second.txt", Number]; Join[file1List , file2List , 2]; Resulted in Expression {1836 -12,1012 -2661,-156 -2916,967 232,1224 1637,-1030 -754,-2002 -2651,-667 -1822,-267 -561,-[Ellipsis] 33,[Ellipsis] ,1292 1055,584 -469,584 -1311,1150 -974,-309 -767,-1949 -212,-719 1262,838 1754,-21 405} at position 1 is expected to have nonatomic subexpression at level 2. >>

POSTED BY: Bob Stephens

Henrik Schachner

Henrik Schachner, Radiation Therapy Center, Weilheim, Germany

Posted 10 years ago

Well, yes, according to you second posting I was assuming that your data are already in form of a matrix like so: {{613, -112}, {1737, 392}, {1685, -176}, {1443, -813}, {891, -39}, {273, 1761}, {356, 2007}, {-8, -198}, {-882, -1634}} // TableForm (output: 613 -112 1737 392 1685 -176 1443 -813 891 -39 273 1761 356 2007 -8 -198 -882 -1634) Obviously this is not the case in your example ...

Well, yes, according to you second posting I was assuming that your data are already in form of a matrix like so:

{{613, -112}, {1737, 
   392}, {1685, -176}, {1443, -813}, {891, -39}, {273, 1761}, {356, 
   2007}, {-8, -198}, {-882, -1634}} // TableForm

(*output: 613   -112
1737    392
1685    -176
1443    -813
891 -39
273 1761
356 2007
-8  -198
-882    -1634*)

Obviously this is not the case in your example ...

POSTED BY: Henrik Schachner