Message Boards Message Boards

[LiVE] "Functional Dataflow"? Wolfram Live Coding Session // Mar 19

Alan Calvitti, Ettain Group
Posted 6 years ago

For those interested in data science, please join us for a 1 hour Wolfram Live Coding Session Tuesday Mar 19 2019 at 4pm ET on Zoom (https://wolfram.zoom.us/j/4659236576) . Please see the Linkedin post:

https://www.linkedin.com/pulse/functional-dataflow-wolfram-live-coding-session-mar-19-calvitti-phd

Abstract

This live coding session will focus on functional methods and patterns to write compact but flexible pipes for data science using real-world examples. The Wolfram Language is ideal to data science due to its functional orientation, large-scale symbolic processing and pattern matching abilities. Many of these features have no direct counterpart in object-oriented languages like Python. Consequently WL workflow is more programmer efficient: line-count ratios of 5:1 and often much more, are common relative to equivalent Python (inclusive of its extensive libraries). Functional programming lets you do more with less code and is more convenient and safer to refactor. After a brief tour of the forthcoming book Functional Dataflow, the focus will be on data transformation, aka "data wrangling" - recognized to be a highly time-consuming aspect of workflow - rather than on math or statistical methods. Exposition will be adapted from Excursion chapters in FD. Key skills described include composing operator pipes, combining tabular, hierarchical, temporal and graphical datasets, exploring queries on data slices then scaling up to the whole dataset, creating insightful visualizations, and writing flexible utility (eg one-liner operator) libraries for reuse in future projects.

Overview Mathematica notebook containing links to data and dependencies: enter link description here
POSTED BY: Alan Calvitti
17 Replies
Sort By:

Vince,

That's a position-coded curry syntax that I invented and use extensively. It's found in Appendix A of my book but here it is.

[Bullet] /: f[pre, [Bullet] , post_] := With[{n = Length[List@pre], m = Length[<a href="mailto:List@post]">List@post]} , Curry[f, Join[Range[n], {n + m + 1}, Range[m] + n]][pre, post] ];

Put the solid dot, aka bullet (Option-8 on most keyboards), in the position of a function's argument list to curry that parameter. As per operator forms in general, It streamlines syntax in data pipes.

Thanks to Jose Martin-Garcia from WRI for help in refactoring from an earlier version.

POSTED BY: Alan Calvitti
Rohit Namjoshi
Rohit Namjoshi
Posted 6 years ago

Alan,

Looking forward to this. BTW the link above is broken, I think it should be this.

Rohit
POSTED BY: Rohit Namjoshi

I find some unfamiliar syntax in Example "In[215]…". What is the solid dot following "10^" and in the first argument of StringJoin? I thought I knew functional notation...

Thank you,

Vince
POSTED BY: Vincent Virgilio

Rohit,

Thanks. Was the original link malformed or misspelled?

Al
POSTED BY: Alan Calvitti

Slick.

Seems like pre and post are heads of BlankNullSequence's? (three underscores)
POSTED BY: Vincent Virgilio

That's right, so that the bullet can be placed anywhere for any function taking arbitrary number of parameters.

POSTED BY: Alan Calvitti
Rohit Namjoshi
Rohit Namjoshi
Posted 6 years ago

Wow, that is very cool. Looks like several characters were interpreted as markup and messed up the definition. I think this is the correct definition.

\[Bullet] /: f_[pre___, \[Bullet], post___] := 
  With[{n = Length[List@pre], m = Length[List@post]}, 
   Curry[f, Join[Range[n], {n + m + 1}, Range[m] + n]][pre, post]];
POSTED BY: Rohit Namjoshi
Updating Name
Updating Name
Posted 6 years ago

Since I'm on Windows and don't have a shortcut for 'bullet', I find \[NumberSign] a bit faster: esc # esc
POSTED BY: Updating Name

Thanks for the edit. And yes beyond cool, this bit of syntactic sugar simplifies readability. There's a good thread on mathematica.stackexchange about operator forms versus nested function/slot lookup (see especially Mr. Wizard's comments)
POSTED BY: Alan Calvitti

Use whatever you like - ideally this would be standardized in the language.

But I would advise against [NumberSign] in particular: (1) it looks too similar to Slot notation - especially if you share code, you can't control other users' fonts. (2) how is 3 keystrokes in esc # esc faster than 2 keystrokes in option 8 (alt 8 on Windows)?
POSTED BY: Alan Calvitti

Yes, agreed that the notation is too close to Slot.

Alt-8 on Windows wasn't producing a bullet for me...hum.
POSTED BY: Vincent Virgilio

Thanks! But I think you missed one _ character after the f, which makes a big difference:

ClearAll[\[Bullet]];
\[Bullet] /: f_[pre___, \[Bullet], post___] := With[
   {
    n = Length[List@pre],
    m = Length[List@post]
    },
   Curry[f, Join[Range[n], {n + m + 1}, Range[m] + n]][pre, post]
   ];

I have to say, this is perhaps the coolest thing I've stumbled across in a long time. As another Windows user, I couldn't find an easy key combo to enter the bullet point, so instead I found that hitting Escape bu Escape to print out \[Bullet] in a notebook works pretty well (and should be OS-independent). It's perhaps not as fast as Option + 8, but it should be good enough.

As somebody who really likes to use RightComposition, this would be a fantastic feature to build into the language, as it's quite a pain to have to use parentheses (and the &) when chaining many "pure functions" together. Until then (assuming it would be implemented) I think I'll be including this definition into more of my projects.
POSTED BY: Andrew Steinacher
Rohit Namjoshi
Rohit Namjoshi
Posted 6 years ago

Thanks Andrew. I updated my reply.
POSTED BY: Rohit Namjoshi

Hi Alan, I liked the functional dataflow stream! Would be great if you could share this stuff on the "functions" page somewhere. The now famous [Bullet] solution baffles me a bit so might deserve a new thread. I tried to work with it (following the above definition) but it seems to work only on self defined functions and not on build in. Anyway again look forward to follow a new stream.
POSTED BY: l van Veen

For the bullet • character in Windows, try with ALT-7 where the 7 is from your numeric key pad. That is easier than ALT-0149 which also works.
POSTED BY: Gustavo Delfino

Windows Alt-7 on numeric keypad indeed produces a bullet.

Thank you!
POSTED BY: Vincent Virgilio
Rohit Namjoshi
Rohit Namjoshi
Posted 6 years ago

l van Veen

but it seems to work only on self defined functions and not on build in

Make sure you are using the definition in the reply from Andrew Steinacher.

Here is a very simple example of using the "magic bullet". Pick a function that does not have an operator form (most likely because it is varargs). To use it in postfix form

Range[10] // Take[#, {3, 5}] &
(* {3, 4, 5} *)

Using the "magic bullet" is a lot cleaner

Range[10] // Take[\[Bullet], {3, 5}]
(* {3, 4, 5} *)

There are a lot of more complex examples in Alan's notebook.
POSTED BY: Rohit Namjoshi
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract