Character frequency in your code

Ideas
, ,
1

I was asked earlier how many closing brackets in a row I’d written in Mathematica. The challenger presented a the end of a line with a hefty ten consecutive brackets, closing functions, array references, and lists.

I did not know if I’d beaten ten. I could spot fives and sixes in my open document, but not much higher…

I wrote some code to import the plaintext of my ‘archive’ folder, which contains a lot of notebooks, and answer the question for me. The results of naive character tallying across notebooks are quite interesting. for example I (somewhere) have six sets of unclosed square brackets, representing a 0.005% square bracket closing failure rate! I didn’t manage to beat or match 10 brackets, but I’ve used a string of nine brackets on twelve occasions.

Capture01.PNG
Capture01.PNG592×237 2.46 KB

Pictured above: The occurrence of strings of bracket characters in my notebooks.

Pictured below: The occurrence of different individual characters in my notebooks. (Open image separately for higher resolution)

Capture02.PNG
Capture02.PNG1656×235 4.85 KB

I’d be interested to hear these arbitrary measures for other peoples old notebooks. Can you differentiate peoples coding styles from such simple metrics I wonder. Notebook attached (you’ll need to specify a directory containing notebooks to run it on).

2 Likes
2

Bedford’s law relates to the distribution of leading digits in data. I was too lazy to reprogram David’s program to look for leading digits, but I still wanted to see how well the digits in my codes correspond to Benford’s law. I need to only look at digits in my code (instead of all characters, like David does) and then remove the zeros - as they are ignored as leading characters in Benford’s law. Here’s the result:

Show[DiscretePlot[
PDF[BenfordDistribution[10], x] // Evaluate, {x, 1, 9}, 
PlotMarkers -> Directive[Red, Automatic], 
PlotRange -> {{1, 9}, {0, 0.35}}], 
ListLinePlot[
ToExpression@
Sort@Select[
SortBy[{#[[1]], 
N@#[[2]]/
Total[Select[
SortBy[{#[[1, 1]], Total[#[[;; , 2]]]} & /@ 
SplitBy[
Sort[Flatten[#[[2]] & /@ # & /@ data, 
2]], #[[1]] &], #[[2]] &], 
MemberQ[CharacterRange["1", "9"], #[[1]]] &][[All, 
2]]]} & /@ 
Select[SortBy[{#[[1, 1]], Total[#[[;; , 2]]]} & /@ 
SplitBy[Sort[
Flatten[#[[2]] & /@ # & /@ data, 
2]], #[[1]] &], #[[2]] &], 
MemberQ[CharacterRange["1", "9"], #[[1]]] &], #[[2]] &], 
MemberQ[CharacterRange["1", "9"], #[[1]]] &], PlotRange -> All]]

ScreenShot2016-03-31at14.56.27.png
ScreenShot2016-03-31at14.56.27.png557×352 10.1 KB

Of course, this data set does not really fit the Benford distribution:

datasymbols = 
  ConstantArray[#[[1]], Floor[#[[2]]]] & /@ (ToExpression@
     Sort@Select[
       SortBy[{#[[1]], N@#[[2]]} & /@ 
         Select[SortBy[{#[[1, 1]], Total[#[[;; , 2]]]} & /@ 
            SplitBy[
             Sort[Flatten[#[[2]] & /@ # & /@ data, 
               2]], #[[1]] &], #[[2]] &], 
          MemberQ[CharacterRange["1", "9"], #[[1]]] &], #[[2]] &], 
       MemberQ[CharacterRange["1", "9"], #[[1]]] &]);
DistributionFitTest[RandomChoice[Flatten[datasymbols], 1000], 
 BenfordDistribution[10], "TestConclusion"]

which gives a rejection:

ScreenShot2016-03-31at15.10.35.png

Ok, that is not really doing proper statistics on these numbers, but just playing a bit.

Cheers,
Marco

2 Likes
3

Marco has sent me his character frequency plot:

PastedGraphic-2.png
PastedGraphic-2.png7273×1550 93.1 KB

We can quite easily take the frequency rank from the axis of this plot, and so compare the frequency of characters in our code.

Relativefrequency.PNG
Relativefrequency.PNG748×737 6.07 KB

Here my characters are bellow the diagonal, and Marco’s above, showing I use several non numeric characters in preference.

2 Likes