Message Boards Message Boards

GROUPS:

A short exploration of the featured contributors

Posted 2 years ago
4180 Views
|
15 Replies
|
40 Total Likes
|

enter image description here

Most of you have probably seen that some post that exemplify the power and breadth of the Wolfram language are selected as featured posts. I was wondering how many posts are there now, and some of the statistics behind it, so I wrote this small code that imports data from the web:

baseurl="http://community.wolfram.com";
featurelinkstring="/people?p_p_id=people_WAR_peopleportlet&p_p_lifecycle=0&p_p_state=normal&p_p_mode=view&p_p_col_id=column-1&p_p_col_pos=1&p_p_col_count=2&_people_WAR_peopleportlet_tabs1=All&_people_WAR_peopleportlet_orderVar=featured&_people_WAR_peopleportlet_delta=20&_people_WAR_peopleportlet_keywords=&_people_WAR_peopleportlet_advancedSearch=false&_people_WAR_peopleportlet_andOperator=true&_people_WAR_peopleportlet_resetCur=false&_people_WAR_peopleportlet_cur=`n`";
ClearAll[FeaturedLink,SimplifiedProfileLink,GetProfileDataFromPage,GetFeaturedPosts]
FeaturedLink[n_Integer]:=FeaturedLink[n]=StringTemplate[baseurl<>featurelinkstring][<|"n"->n|>]
SimplifiedProfileLink[url_String]:=FirstCase[URLFetch[url,"Headers"],{"Location",link_String}:>link,url,\[Infinity]]
SimplifiedProfileLink[x_]:=x
GetFeaturedPosts[profilelink_String]:=GetFeaturedPosts[profilelink]=Module[{list},
    list=FirstCase[Import[profilelink,"XMLObject"],XMLElement["div",{"class"->"content-wrapper staff-picks-content"},content__]:>content,Missing[],\[Infinity]];
    If[MissingQ[list],{},Cases[list,XMLElement["li",{},{XMLElement["a",{"shape"->"rect","href"->url_},{title_}]}]:>{title,baseurl<>url},\[Infinity]]]
]
GetProfileDataFromPage[url_String]:=GetProfileDataFromPage[url]=Module[{xml,out,names,avatarurl,profilelinks,featureposts},
    xml=Import[url,"XMLObject"];
    out=Cases[xml,XMLElement["tr",{"class"->"portlet-section-alternate results-row alt"|"portlet-section-body results-row"|"portlet-section-alternate results-row alt last"},___],\[Infinity]];
    names=FirstCase[#,XMLElement["span",{"class"->"user-name"},{name_}]:>StringTrim[name],Missing[],\[Infinity]]&/@out;
    avatarurl=FirstCase[#,XMLElement["img",{"class"->"avatar",___,"src"->u_},{}]:>baseurl<>u,Missing[],\[Infinity]]&/@out;
    profilelinks=FirstCase[#,XMLElement["a",{"shape"->"rect","href"->u_},{" ",XMLElement["span",{"class"->"user-profile-image"},{" "}]}]:>u,Missing[],\[Infinity]]&/@out;
    profilelinks=SimplifiedProfileLink/@profilelinks;
    featureposts=GetFeaturedPosts/@profilelinks;
    Association/@Transpose[Thread/@{"Name"->names,"AvatarURL"->avatarurl,"Avatar"->(Import/@avatarurl),"ProfileURL"->profilelinks,"NumberOfFeaturedPosts"->(Length/@featureposts),"FeaturedPosts"->featureposts}]
]

Then we can call these functions and get the information:

out = Join @@ (GetProfileDataFromPage[FeaturedLink[#]] & /@ Range[5]); (* currently 5 pages of featured contributors*)

This will return a big dataset, here a random sample:

Dataset[out[[3]]]

enter image description here

So we get a Association back with the various properties, including his (or her for other entries) featured posts:

enter image description here

We can now do some fun stuff with this dataset:

tmp=Values/@out[[All,{"Name","Avatar","FeaturedPosts"}]];
tmp[[All,3]]=Length/@tmp[[All,3]];
tmp=SortBy[tmp,Last];
imgs=ImageCrop[ImageResize[#,50],{50,50}]&/@tmp[[All,2]];
insets=MapThread[Inset[#1,{#3+1,#2},Center,{1.75,1.75}]&,{imgs,Range[Length[imgs]],tmp[[All,3]]}];
BarChart[tmp[[All,3]],
    BarOrigin->Left,
    ChartLabels->{tmp[[All,1]]},
    Frame->True,
    ImageSize->800,
    GridLines->Automatic,
    FrameStyle->Directive[GrayLevel[0.3],AbsoluteThickness[1]],
    PlotRangePadding->{{0,0},{-1,1}},
    PerformanceGoal->"Speed",
    PlotRange->{{0,50},Automatic},
    PlotLabel->Style["Number of featured posts per featured contributor",14,Black],
    ChartStyle->Directive[GrayLevel[0.7],EdgeForm[GrayLevel[0.4]]],
    BarSpacing->0.1,
    AspectRatio->4,
    Epilog->insets,
    FrameTicks->{{Automatic,Automatic},{Range[0,50,5],Range[0,50,5]}},
    FrameStyle->Black,
    FrameTicksStyle->Black
]

enter image description here

Clayton is the clear winner!

We can also look at the profile photos in more detail:

Magnify[out[[All, "Avatar"]], 0.5]

enter image description here

Let's find all the faces, and calculate the 'average' face:

ClearAll[GetFaces]
GetFaces[img_Image]:=With[{s=FindFaces[img]},ImageTrim[img,#]&/@s]
imgs=RemoveAlphaChannel/@ConformImages[Join@@(GetFaces/@out[[All,"Avatar"]]),{200,200}];
Image[Mean[ImageData/@imgs]]

enter image description here

Or a word-cloud of all the titles:

    WordCloud[StringRiffle[Join @@ out[[All, "FeaturedPosts", All, 1]], " "]]

enter image description here

Feel free to dig deeper!

15 Replies

@Vitaliy Kaurov You're the random sample, haha!

I'd swear I've seen that face before. I think it was in a post office...

@Daniel Lichtblau @Vitaliy Kaurov You mean this?

Overlay[{out[[3, "Avatar"]], Style["WANTED", 20, Red]}, Alignment -> {Center, Top}]

enter image description here

That's the one! Where do I get to claim the reward?

I kind of disagree that the mean face was made with simple averaging. That gives wrong impressions (and deja-vu-s to Danny). A weighted face averaging is much more appropriate.

What should the weights be? Number of featured posts? Sqrt of that? Log thereof?

I would say log or square root. Also, I guess, some image outliers (say, that are not faces) have to be removed...

I already use FindFaces already to align/crop them, and to get rid of avatar without a face (Szabolcs for example).

Right, I should have read your code more carefully. (Cool post, BTW, +1.)

Interestingly, your deja-vu might be right:

Nearest[out[[All, "Avatar"]] -> out[[All, "Name"]], meanface]

returns Daniel Lichtblau as the one with the most average face...

Awesome post, @Sander, thanks! This is very timely and is sort of celebratory as we just surpassed 13,000 members. Excellent example of web data parsing.

enter image description here - another post of yours has been selected for the Staff Picks group, congratulations ! We are happy to see you at the top of the "Featured Contributor" board. Thank you for your wonderful contributions, and please keep them coming!

How many Featured Posts are there about how many Featured Posts there are?

BarChart[{1}]

enter image description here

@Vitaliy Kaurov @Christopher Carlson @Sander Huisman Interesting to learn about the main authors who write amazing codes on the website (Wolfram Tweet-a-Program) https://twitter.com/wolframtap since 2014 until today ;)

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract