Message Boards

GROUPS:

Staff Picks Data Science Mathematica Add-Ons Import and Export Wolfram Language Packages Computational Linguistics Computational Humanities

Faizon Zaman

Parsing Markdown Files

Faizon Zaman, Wolfram|Alpha LLC

Posted 2 months ago

1134 Views

5 Replies

10 Total Likes

Follow this post

Symbolic Markdown Expression Graph

A few years ago, at the Wolfram Summer School, I worked on Import/Export functions for Twine projects. One of the goals ended up being styling the passage’s markdown. As I began to look into the task, I realized it would be a great exercise to try parsing markdown into a symbolic form, formalizing the structure of the document before I work on styling. And so I did just that! With the parser in a decent enough state I thought I’d share it with the community!

Usage

After downloading the repo and installing the package:

Get the package

In[]:=

<<MarkdownParse`

Parse a Markdown file and display it in a Column

In[]:=

mdp=MarkdownParse

File[

]

;ColumnTake[mdp,10],

SequenceIcon



The parser returns a list, containing a mixture of MarkdownElements, Lists, and Strings. You’ll notice the general structure of MarkdownElement’s is:

MarkdownElement[tag,data]

Common markdown syntax is supported, and below is a list of supported symbolic forms.

Supported MarkdownElements

Note: In several places there’s a

"$n"

, this is because the displayed list contains the right hand sides of my replacement rules. In each replacement rule, the left hand side contains a regex with capture groups, whose index corresponds to the

after the dollar sign. This is how I slot the data into the MarkdownElement! The blue dollar-n symbols are blue because I apply ToExpression to them; in practice, they correspond to digit strings which become numbers after applying ToExpression.

The Parser Process

Here’s what the MarkdownParse function does:

Extract footnotes from the file

Isolate and parse tables

Repeatedly apply MarkdownParser via FixedPoint to the remaining strings.

To handle different cases in step 3, I’ve set up several DownValues to guide the behavior of the parser. Here’s what happens when the MarkdownParser comes into direct contact with a MarkdownElement.

Let’s look at the first definition:

If the MarkdownParser encounters a MarkdownElement whose tag is a String (that’s not

"TableHeader"

), and the data is a List, then keep the tag as is, and apply MarkdownParser to the list of data.

Testing

Throughout the process were moments of toiling over esoteric and complicated regular expressions wondering why, for example, my inline code regex was picking up pairs of back-ticks that were meant for a code-block. What helped me get a sort of tractable view of regex and what patterns I was writing, was Taliesin Beynon’s WFR entry ToRegularExpression. If you find writing String patterns with StringExpression’s much easier, or nicer, but encounter regex in the wild more often than you’d like, this function can help bridge the gap between what you know and what you see. I highly recommend playing around with it.

At any rate, there were some patterning issues I ran into time after time. It was at that point I decided to start writing parser tests. After a cumbersome attempt at DIYing a testing workflow, VerificationTest came to the rescue!

It’s a simple way to check if an expression evaluates how you expect it to. In my case, it’s set up like this:

For example, here’s how I can check my Italics parsing:

And, so! I made a separate package to hold these verification tests and a function to run TestReport on them. Writing these tests was instrumental in focusing my debugging time, allowing me to pinpoint breakages in the parser.

Heading VerificationTests

After installing the MarkdownParserTests package, here’s how to run the tests:

Get the MarkdownParserTests package

In[]:=

<<MarkdownParserTests`

Run the parser tests

In[]:=

MarkdownParserTestReport

Out[]=

TestReportObject

Title: Automatic
Success rate: 100%	Tests run: 28
Succeeded: 28
Failed: 0
Failed with wrong results: 0
Failed with messages: 0
Failed with errors: 0
Time elapsed: 0.0132 s



And they all work as expected! (for now at least ;) )

What can I do with this?

I think a potentially interesting application is analyzing document structure, for example, here’s a graph of the test file included in the repository (linked below)

In[]:=

ExpressionGraph[mdp,VertexLabelsPlaced[Automatic,Tooltip]]

Out[]=

Additionally, I’m interested in more comprehensive Import/Export for

'.md'

files; importing as a notebook, exporting notebooks and cells as markdown (there’s already a great resource function by Andrew Steinacher for converting tables of data to markdown table strings).

Where can I get the parser?

The parser and related documents are available here on my Github.

Thanks for reading!

Future

◼

The following are works in progress which I expect to have ready relatively soon.

◼

Implement Import/Export Converters for Markdown files

◼

Render the parse list as a list of styled Cells, as well as

◼

Generate a Notebook from it

◼

Wrap the parse list with a MarkdownObject head

◼

Of lesser priority, but still keen on

◼

Highlighting Subgraphs of Symbolic Markdown

◼

Adding HTML support

◼

Adding YAML front matter support

◼

Adding support for more flavors of markdown

POSTED BY: Faizon Zaman

Answer

5 Replies

Sort By:

Moderation Team

Moderation Team, WOLFRAM

Posted 2 months ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team

Answer

Anton Antonov

Anton Antonov, Accendo Data LLC

Posted 2 months ago

Interesting work! Can you post the link to `MarkdownParse` repository you mention? Is it this one: https://github.com/dishmint/Markdown2WL ?

POSTED BY: Anton Antonov

Answer

Faizon Zaman

Faizon Zaman, Wolfram|Alpha LLC

Posted 2 months ago

Thanks Anton! Yeah, that's the one.

POSTED BY: Faizon Zaman

Answer

Anton Antonov

Anton Antonov, Accendo Data LLC

Posted 2 months ago

I think you should update your notebook to include the line: Import["https://raw.githubusercontent.com/dishmint/Markdown2WL/master/Parser/MarkdownParse.wl"] Or something similar...