Abstract

Goal of the project

In order to verify if the set of guidelines of a notebook submission are respected, we create a test framework code for the computational essay. The main specification is to read in notebook and generate a feedback report. The screening of article is the first check in the editorial process. Typically this control verifies that all elements of the submission are well presented: author information, code length, sections composed of text,code text and code, references.

We want to implement a feedback report as the author upload the submission and a report for the editor on a set of notebooks. A preliminary test is made on a randomly selected notebook from a directory.

Summary of the results

• Identify the set of guidelines.
• Match between submission and required guidelines.
• Immediate warnings for author upload.

Read in Notebook Template and Generate Feedback Report

Get file and divide by cells

A preliminary test is made on a randomly selected notebook from a directory.

testFile = 
 RandomChoice@
  FileNames[___, 
   FileNameJoin[{NotebookDirectory[], "essay-submissions", 
     "ComputationalEssays", "ComputationalEssays"}]]

A second test is made on all the files of the directory, useful for the editor who has to check a list of notebook multiple submissions and generate a status report.

allFiles = 
FileNames[___, 
FileNameJoin[{NotebookDirectory[], "essay-submissions", 
"ComputationalEssays", "ComputationalEssays"}]];
cells = NotebookImport[testFile, _ -> "Cell"];

The notebook template has identifying cells, listed below:

cells[[All, 2]]

Test for functions

We implement three functions corresponding to the tests:

• The author contact information, the corresponding author, is present at the bottom of the notebook submission.
• The code cells shouldn't be too big and they should be divided in multiple code cells.
• Each code cell should be presenter with a single grey line of TextCode.

Author Information

The first check of the report is about the Author Section. If the author doesn't add his contact email or change the author contact information this will be reported as an error.

authorSection = 
Position[cells, Cell["Author contact information", "Section", ___]]
hasOnlyOneAuthorSectionQ[authorSection_] := Length@authorSection === 1
If[
hasOnlyOneAuthorSectionQ[authorSection],
1,
issueError["Cannot find just one AuthorInfo cell"]
]

Metric on Line Code

The second check is evaluating the length of the Code cells. Each code cell is of Input/Output type, once they are identified, they are converted as images. For each image of code it has been considered only the height and it has been compared to a reference height of 75 pixels. An error is given for each image of code cell has his size greater than the reference measure.

Select the cell with the code: input/output type.

NotebookImport[testFile, "Input" | "Output", "FlattenCellGroups" -> False]

Take all code cells and transform them as images.

cellObj = SelectionMove[EvaluationCell[], Previous, Cell];
cellExpr = NotebookRead@EvaluationNotebook[];
Rasterize@cellExpr
ImageDimensions@Rasterize@cellExpr

Then take their dimensions.

cellHeights = 
 Last@*ImageDimensions@*Rasterize /@ Cases[cells, Cell[___, "Input", ___]]

Count the number of cells of code for which the height of the image is bigger than 75.

numberOfBigPictures = Count[cellHeights/75., _?(# >= 1 &)]

Give a warning with the number of code cells exceeding the reference height.

If[numberOfBigPictures >= 1, 
 issueError[
  numberOfBigPictures " code cells are exceeding the suggested size"]]

Warning on Text Before Each Line of Code

Each code cell should be introduced with a single line of text code explaining each segment.

Cell codes are identifies as triplets code text input and output.

n = Length[
  SequenceCases[
   cells[[All, 2]], {Except["CodeText" | "Input" | "Output"], "Input"}]]

Every time there is no code text before input a warning is given.

If[n > 0, ReportPrint[
   "You have " <> ToString[n] <> 
    " cases of Input cell (your code) without descriptive grey text.", 
   "WARNING"]];

Notebook Report

A notebook is analysed and the checks are implemented, the resulting report is shown as a grid on the web submission page.

• quickParse is the notebook reader .

    quickParse[notebook_] := Module[
    {cells, testHasAuthor, codeCellTooBig, TextCodeNotFound},
    cells = NotebookImport[notebook, _ -> "Cell"]];

• testHasAuthor checks if the author contact information has been modified.

    testHasAuthor[notebook_] := Module[
    {cells, section},
    cells = NotebookImport[notebook, _ -> "Cell"];
    section = 
    Position[cells, Cell["Author contact information", "Section", ___]];
    Which[
    Length @ section === 0,
    "No author specified",
    Length @ section > 1,
    "Multiple authors specifieds",
    True,
    None
    ]
    ];

• codeCellTooBig transforms each input cell in an image and it checks whether it exceeds a reference height.

    codeCellTooBig[notebook_] :=
    Module[
    {cells, selection}, 
    cells = NotebookImport[notebook, "Code" | "Input" -> "Cell"];
    selection = Select[
    Map[ImageDimensions@Rasterize[#] &, cells], 
    Last[#] > 75 &
    ];
    If[
    Length[selection] > 0,
    TemplateApply["`` cells are too big", Length[selection]],
    None
    ]
    ];

• TextCodeNotFound checks if the text code is present before the input cell.

  TextCodeNotFound[notebook_] :=
  Module[ 
  {n, cells},
  cells = NotebookImport[notebook, _ -> "Cell"];
  n = Length[
  SequenceCases[
  cells[[All, 2]], {Except["CodeText" | "Input" | "Output"], "Input"}]];
  If[n > 0,
  TemplateApply[
  "You have " <> ToString[n] <> 
  " code cells without the single line of text code."],
  None
  ]
  ];
  

• AllTests implements the checks all together.

  AllTests = {
  <|"Test" -> testHasAuthor, "Title" -> "Author check"|>,
  <|"Test" ->  codeCellTooBig, "Title" -> "Code cell check"|>,
  <|"Test" ->  TextCodeNotFound, "Title" -> "TextCode cell check"|>
  };
  

• notebookReport results in a table with the check answers.

    notebookReport[notebook_List] := 
    Column[Map[notebookReport, notebook]]
    notebookReport[notebook_String] := 
    Module[
    {tests, titles, results},
    tests = Map[#Test[notebook] &, AllTests];
    titles = AllTests[[All, "Title"]];
    results = Join @@ Replace[
    Transpose[{titles, tests}], {
    {title_, result_String} :> {{title, result}},
    {title_, {result_String}} :> {{title, result}},
    {title_, {result_String, results__String}} :> 
    Join[{{title, result}}, Map[{SpanFromAbove, #} &, {results}]],
    {title_, None} :> {{title, Style["OK", Green]}}
    },
    {1}
    ];
    Labeled[
    If[
    Length[results] > 0,
    Grid[results,
    Alignment -> Left,
    Frame -> All
    ],
    Style["No error found :)", Blue]
    ],
    FileBaseName[notebook],
    Top
    ]
    ]

Deploy to the web

The test report is showed after the notebook submission.

CloudDeploy[
FormPage[{"submission" -> Restricted["CachedFile", "NB"]}, 
Rasterize@notebookReport@#submission &], "/publish", 
Permissions -> "Public"]

Code Repository

https://github.com/vivlet/Summer2018Starter

Future Directions

• Implement future specifications required by the author's guidelines.

• Automatic check of references as Crossed Check References http://www.ithenticate.com/.

• Automatic check of data repository, https://datarepository.wolframcloud.com/, https://resources.wolframcloud.com/NeuralNetRepository/

• Automatic check of software repository, es. GitHub.

• Integration of keywords and reviewers database.

References

http : // blog.stephenwolfram.com/2017/11/what - is - a - computational - essay/

Keywords

• Notebook Feedback Report

• Notebook Testing framework

Other information

• Publishing notebook

• Authoring notebook