Message Boards Message Boards

GROUPS:

raspiSFV - Creating/Testing file hash codes

Posted 4 months ago
583 Views
|
4 Replies
|
4 Total Likes
|

Trott files downloaded from http://extras.springer.com

Introduction

This is my attempt of a first tiny app ("Hello World!"). The main motivation behind the coding was to make me apply hands-on some of the contents learned from the superficial Introduction to Control Objects. I wanted to experience if the language is really(?) suitable for building a small software, think of the thousands of FREE primitive yet handy utilities which one can download from the internet like unfreez or file renamer. My 'program' is to compute file hash codes like "CRC-32" in a user-friendly manner, i.e. with a click of a button. The real question is how an experienced WL coder would program such an app with similar functionality and user interface. I am a beginner, let me share how far i got. I am a beginner, let me share how far i got. Note that I adopted the code formatting convention introduced by Boris.

Check files through SFV

Simple File Verification (SFV) is a file format for storing CRC32 checksums of files to verify the integrity of files. The following code imports the *.SFV-file from the HDD as text string, listing the lines which the text is made of. From these text lines, only the relevant lines are selected, i.e. lines which contain the file name and its file checksum (also called file hash code or file hash value). SFV files created by other SFV utilities may contain comments and other irrelevant text. A crc32 code consists of 8 hexadecimal chars, a SHA-512 code consists of 128 hex chars. The hex chars can be in upper or lower case, without difference in meaning.

rellines := Select[
   Import[sfvpath, {"Text", "Lines"}]
   , StringMatchQ[#
     , WordCharacter ~~ RegularExpression[".+\\s[[:xdigit:]]{8,128}"] ~~ EndOfLine] &
  ];

The following code separates the relevant text lines into a list of file names and a list of their hash codes. One could use parallel assignment with Set[ ] instead of SetDelayed[ ] to write it more concisely but we really need the latter here.

pairs := Transpose[Flatten[
    Map[
     StringCases[#
       , fn : (StartOfString ~~ __) ~~ WhitespaceCharacter ~~ 
         hs : HexadecimalCharacter .. ~~ EndOfString :> {fn, ToLowerCase@hs}] &
     , rellines]
    , 1]
  ];
fnames := pairs[[1]];
fhashs := pairs[[2]];

With the help of the file names we are able to create their full paths on the computer, assuming that the SFV file is in the same directory (folder) as the files in question.

flist := Map[
   FileNameJoin[{
      FileNameDrop[sfvpath, -1]
      , #
     }] &
  , fnames];

The core functionality of the app is the computation of the file hash value. In Wolfram L we have the function Hash[ ] and FileHash[ ]. The latter is just what we need. By using MapIndexed[ ], we could track the mapping progress through the variable n.

hashes[files_List] :=
  MapIndexed[
    {FileHash[#, htype, All, "HexString"], n = First@#2} &
    , files
   ][[All, 1]];

Create your own SFV file

If the user wants to create an SFV file, he can choose to have the list of hash codes in lower case chars or capital letters/numbers. A checkbox will provide this setting in the graphical user interface (GUI). By default, the Mathematica hash functions output hash values in lower case chars.

yourhashed := If[
   ! lowcase
   , ToUpperCase[hashes[filespath]]
   , hashes[filespath]
  ];

Q: How do we extract the file names from a list of file paths? A: By mapping the function FileNameTake[ ] on the list, with -1 as the second argument.

yourfils := FileNameTake[#, -1] & /@ filespath;

We have the file names in a list, we have the file hashes in another list. With StringJoin[ ] we can join the two strings and do so for all the files. This recreates the official format of an SFV file, its relevant content, plus some comment/information for the end consumer.

out := Join[
    {";", "; NOTE: The hash codes in this SFV file are of type " <> htype, ";", ""}
    , #1 <> " " <> #2 & @@@ Transpose[{yourfils, yourhashed}]
    , {""}
   ];

We want the utility to handle not only CRC32 codes but also all other supported hash codes in Wolfram L. Currently twelve hash code types are supported. Many many more types do exist and are handled by third-party FREE software tools, but these twelve are among the more common ones.

panel = Panel[
   Grid[{
     {RadioButtonBar[Dynamic[htype]
       , {"CRC32" -> Tooltip[Style["CRC32", Bold], "32-bit cyclic redundancy check", TooltipDelay -> Automatic]
        , "Adler32" -> Tooltip["Adler32", "Adler 32-bit cyclic redundancy check", TooltipDelay -> Automatic]
       }, Appearance -> "Vertical"]
      , RadioButtonBar[Dynamic[htype]
       , {"MD5" -> Tooltip["MD5", "128-bit MD5 code", TooltipDelay -> Automatic]
        , "MD4" -> Tooltip["MD4", "128-bit MD4 code", TooltipDelay -> Automatic]
        , "MD2" -> Tooltip["MD2", "128-bit MD2 code", TooltipDelay -> Automatic]
       }, Appearance -> "Vertical"]
      , RadioButtonBar[Dynamic[htype]
       , {"RIPEMD160" -> Tooltip["RIPEMD-160", "160-bit RIPEMD code", TooltipDelay -> Automatic]
        , "RIPEMD160SHA256" -> Tooltip["RIPEMD-256", "RIPEMD-160 following SHA-256 (as used in Bitcoin)", TooltipDelay -> Automatic]
       }, Appearance -> "Vertical"]
      , RadioButtonBar[Dynamic[htype]
       , {"SHA" -> Tooltip["SHA-1", "160-bit SHA-1 code", TooltipDelay -> Automatic]
        , "SHA256" -> Tooltip["SHA-256", "256-bit SHA code", TooltipDelay -> Automatic]
        , "SHA256SHA256" -> Tooltip["SHA-256 Base64", "double SHA-256 code (as used in Bitcoin)", TooltipDelay -> Automatic]
        , "SHA384" -> Tooltip["SHA-384", "384-bit SHA code", TooltipDelay -> Automatic]
        , "SHA512" -> Tooltip["SHA-512", "512-bit SHA code", TooltipDelay -> Automatic]
       }, Appearance -> "Vertical"]
     }
    }, Alignment -> Top, Background -> Lighter[Yellow, 0.8]
   ]
  ];

With the above definitions we are ready to put the pieces together and bring life into the GUI. The GUI consists of one single TabView[ ] statement. It is basically a "one-liner", if you will. But … "Boy, That Escalated Quickly!"

CreateDialog[
 TabView[{
   "Check SFV" -> Column[{
      panel
      , Row[{
         FileNameSetter[Dynamic[sfvpath], "Open", {"SFV files" -> {"*.sfv"}}, WindowTitle -> "Find and open SFV file"]
         , Style[Dynamic[sfvpath], Small]
        }]
      , Button["Check SFV"
       , Column[{
          Style["The status of the SFV file is: ", 18, Green]
          , And @@ Equal @@@ Transpose[Sort /@ {fhashs, hashes[flist]}]
          }] // MessageDialog
       ; n = 0
       , Tooltip -> "Click to check your files against the loaded SFV file"
       , TooltipDelay -> Automatic
       , Method -> "Queued"]
      , Row[{
         ProgressIndicator[Dynamic[n], {0, Dynamic@Length[flist // Quiet]}]
        , StringForm[" processing\[Ellipsis]``/``", Dynamic[n], Dynamic@Length[flist // Quiet]]
       }]
     }]
   , "Create SFV" -> Column[{
      panel
      , Row[{
         Checkbox[Dynamic[lowcase]]
         , Style[" Use lowercase hash values", Small, FontFamily -> "Arial"]
        }]
      , Row[{
         FileNameSetter[Dynamic[filespath], "OpenList", {"All files" -> {"*"}}, WindowTitle -> "Select files to be included in SFV file"]
         , Style[Dynamic@TableForm[filespath], Tiny]
        }]
      , Row[{
         FileNameSetter[Dynamic[sfvpath], "Save", {"SFV files" -> {"*.sfv"}}, WindowTitle -> "Specify SFV file name and saving location"]
         , Style[Dynamic[sfvpath], Small]
        }]
      , Button["Export to SFV file"
       , Export[sfvpath, out, "Text"]
       ; Column[{
          Style["SFV file successfully created at: ", 18, Green]
          , sfvpath
         }] // MessageDialog
       ; n = 0
       , Tooltip -> "Click to create SFV file in your specified saving location"
       , TooltipDelay -> Automatic
       , Method -> "Queued"]
      , Row[{
         ProgressIndicator[Dynamic[n], {0, Dynamic@Length[filespath]}]
         , StringForm[" processing\[Ellipsis]``/``", Dynamic[n], Dynamic@Length[filespath]]
        }]
      }]
   , "About" -> Grid[{
       {}
       , {Style["raspiSFV v1.0", Italic, FontFamily -> "MV Boli"]}
       , {Panel@FlipView[{
            Import["ExampleData/spikey.tiff"]
            , Import["https://abload.de/img/avatar_moose150x150vys5e.png"]
           }] }
       , {Style["©2018 Wolfram L", Italic, FontFamily -> "MV Boli"]}
      }, ItemSize -> 30]
   }], WindowTitle -> "raspiSFV \[LongDash] \"Hello World!\""
]

This pretty much wraps it up. I made very basic use of Dynamic[ ], and all my attempts to wrap the whole set of code lines with DynamicModule[ ] or Module[ ] would give me new problems to deal with. The progress indicator will not work with the default preemptive method of the button; using method "Queued" was very crucial here.

Conclusion

As both experts and beginners should be able to see, this program is very primitive but already looking complex due to the necessary nested placement of interface elements. So much complexity (already), even though the whole program revolves around one single function, namely FileHash[ ], are you aware? With this in mind, a more experienced programmer could hopefully come up with a better GUI-implementation for an SFV software tool — just for the sake of showing/teaching how it's done better, in a more exemplary manner.

Anyone up for the challenge?

Attachments:
4 Replies

nice interface. but if you did that for all mathematica functions what would it be like? we learn by doing.

mm's interface is more advanced than you think. everything the front end can do is an option in "Option Inspector" - it's amazingly complete.

now as for point and click you can see for a scripted language with thousands of functions: just finding which to click could take a long time compared to typing (this would bad/obstruct people who have good memory for language).

but the the interface is slyly point and click. you can find any function in help and run altered example. More commonly you type part of a word and mm completes the word it (perhaps selecting completion with mouse or keyboard arrow). then as you type an option mm can fill in symbol names, then for options: mm pops up a list of available optioins.

so as far as using "text functions not being point and click" - well actually it already is. but mm does use dialogue style interfaces for some things but not others.

my advice is don't spend time doing small projects you cannot afford to pay for out of your own pocket, and if programming is something you really believe you can profit from: work ONLY on the goal that will make profit not "many things in general". don't get tied up in general things unless you (tried to avoid that and avoiding it didn't work)

nice interface. but if you did that for all mathematica functions what would it be like? we learn by doing.

mm's interface is more advanced than you think. everything the front end can do is an option in "Option Inspector" - it's amazingly complete.

now as for point and click you can see for a scripted language with thousands of functions: just finding which to click could take a long time compared to typing (this would bad/obstruct people who have good memory for language).

but the the interface is slyly point and click. you can find any function in help and run altered example. More commonly you type part of a word and mm completes the word it (perhaps selecting completion with mouse or keyboard arrow). then as you type an option mm can fill in symbol names, then for options: mm pops up a list of available optioins.

so as far as using "text functions not being point and click" - well actually it already is. but mm does use dialogue style interfaces for some things but not others.

my advice is don't spend time doing small projects you cannot afford to pay for out of your own pocket, and if programming is something you really believe you can profit from: work ONLY on the goal that will make profit not "many things in general". don't get tied up in general things unless you (tried to avoid that and avoiding it didn't work)


I tried it and found two problems

(1) the interface was too slow to use on an imac core 2 duo (2.4 Mhz) 2010 PC. i cannot wait 10 seconds to choose a file for any reason. this of course is not your fault in interface coding but the fault of a whole chain of bloat and software wars ending in stalemated interfaces.

(2.a) i evaluated the notebook, the dialogue popped up (2.b) i started with "create svf", choosing a 1k human readable text file chosen from dialogue (i chose 2 filenames then clicked create. i tried it 2x incase pathing was a problem). but i got this as the .svf file contents:

Join[{";", StringJoin["; NOTE: The hash codes in this SFV file are of type ", Gl

obalhtype], ";", ""}, Transpose[StringJoin["xxxxx ", If[ !Globallowcase, ToUpp erCase[hashes[filespath]], hashes[filespath]]]], {""}]

likely there is something on your personal PC which "just happens" to evaluate that you didn't include in the distributed .nb, thus it works on your side not mine

i did not "closely read" the .nb i just glanced through it. if there was any "installation required" i may have missed it.

out of interest, an "older version" of mathematica had a far less featured "button/palette" ability but ran quickly on a PC that was at least 5x slower, so at 5x slower it ran FAR FASTER - about immediately

Hello Mr Hendrickson, thanks for your comments. I can agree with all of the points you made, no problem. I tried to build this little thingy only for my own educational purposes, to learn what it's like to compose a simple custom user interface from scratch out of the blue, how much effort it takes to place the elements (their alignments) exactly where i want them to be. It was just an exercise for practicing new vocab items. In fact, i used so many functions which i never used before but just grabbed from search hits in the WLDC, which i find amazing. I started learning Wolfram L in April and i am not talented, ambitious, nor productive .. so it really amazes me what can be done even with little code/effort/practice.

I deliberately chose to program such a tiny thing, so that readers of this thread can grasp right away the principle of my implementation and what's awkward about it. I wished that i could present a more robust program with more bells and whistles, production-quality stuff, working on all OS-platforms, all versions, and also in the Cloud.

It is true that the code works without glitches on my Win7 system but produces error messages on Raspbian. Thanks for the detailed report. I certainly have an idea where/what to look for in case of debugging. Maybe i should not have called it "raspisfv" after all, haha.

Could at least any other Windows user confirm that the program works without glitches on their system (Windows)?

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract