Message Boards

GROUPS:

Staff Picks Data Science Dynamic Interactivity Graphs and Networks Wolfram Language Wolfram Cloud Machine Learning Wolfram High School Summer Camp

Austin Geng

[WSC21] Exploring methods of parsing user agent strings

Austin Geng

Posted 1 year ago

1777 Views

1 Reply

4 Total Likes

Follow this post

Graph of User Agent Spoofing

Exploring methods of parsing User Agent strings
by Austin Geng

The objective of this project was to create a User Agent string parser. We begin by breaking up the string into a simpler, more abstract form. We then use the built-in Classify function with multiple forms of preprocessing. Lastly, we construct a graph of user agents based on their common components.

Introduction

User Agent strings are sent by a program communicating with the Web (such as a browser or web crawler bot) to provide information about itself to a server, such as its software, version, and operating system. This allows the server to send content appropriate to the limitations of the user agent.

However, it is up to the server to figure out what features each program supports, and so it may not understand a new or updated program. As a result, there has been a long history of browsers copying each other’s User Agent strings to “trick” servers into giving them the correct features. This has led to almost all modern browsers having convoluted User Agent strings; for example, Chrome pretends to be Safari, Mozilla, and Netscape and pretends to use the WebKit, KHTML, and Gecko layout engines.

Furthermore, although there are standards and conventions for the format of User Agent strings, these are sometimes ignored, and in these cases, anything beyond string-matching becomes tricky.

Although feature detection through the User Agent string is now discouraged, the information found within the string still has uses for operating system detection and web analytics.

Splitting up the string

An example (Chrome) user agent string:

In[]:=

userAgentExample="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

The typical user agent string is composed of a list of products (software names, often followed by the version after a slash) and comments (parenthesized groups of other terms delimited by semicolons). We can break up these strings with relative ease.

Split the string by spaces (excluding those inside a comment):

tokens=StringSplit[userAgentExample,RegularExpression[" (?![^(]*\\))"]]

Out[]=

{Mozilla/5.0,(X11; Linux x86_64),AppleWebKit/537.36,(KHTML, like Gecko),Chrome/51.0.2704.103,Safari/537.36}

Group by whether or not a token is a comment:

grouped=GroupBy[tokens,StringMatchQ@RegularExpression["\\(.*\\)"]]

Out[]=

False{Mozilla/5.0,AppleWebKit/537.36,Chrome/51.0.2704.103,Safari/537.36},True{(X11; Linux x86_64),(KHTML, like Gecko)}

Split each product's name and version and convert to an association:

products=Association[Rule@@PadRight[#,2,Missing[]]&/@(StringSplit[#,"/",All]&/@Replace[grouped[False],_Missing->{}])]

Out[]=

Mozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36

Split each comment by semicolons and remove parentheses:

comments=StringSplit[StringTake[#,{2,-2}],"; "]&/@Replace[grouped[True],_Missing->{}]

Out[]=

{{X11,Linux x86_64},{KHTML, like Gecko}}

Combine products and comments:

<|"Products"->products,"Comments"->comments|>

Out[]=

ProductsMozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36,Comments{{X11,Linux x86_64},{KHTML, like Gecko}}

All together:

splitUserAgent[userAgent_String]:=Module[{tokens,grouped,products,comments},tokens=StringSplit[userAgent,RegularExpression[" (?![^(]*\\))"]];grouped=GroupBy[tokens,StringMatchQ@RegularExpression["\\(.*\\)"]];products=Association[Rule@@PadRight[#,2,Missing[]]&/@(StringSplit[#,"/",All]&/@Replace[grouped[False],_Missing->{}])];comments=StringSplit[StringTake[#,{2,-2}],"; "]&/@Replace[grouped[True],_Missing->{}];<|"Products"->products,"Comments"->comments|>];splitUserAgent[userAgentExample]

Out[]=

ProductsMozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36,Comments{{X11,Linux x86_64},{KHTML, like Gecko}}

Interactive version with Manipulate:

Manipulate[DynamicModule[{split},split=splitUserAgent[userAgent];Row[Pane[#,BaselinePositionTop]&/@Prepend[Dataset/@split["Comments"],Dataset@split["Products"]],Spacer[10]]],{{userAgent,userAgentExample,"User Agent"},InputField[#,String,BaseStyle->12,FieldSize->{{60.,60.},{1.,Infinity}}]&},SaveDefinitionsTrue]

Out[]=

Data gathering

I was fortunate enough to find a sample database of already-analyzed User Agent strings from the website whatismybrowser.com.

Import data:

In[]:=

userAgentData=CloudImport@CloudObject@"https://www.wolframcloud.com/obj/yygengjunior/whatismybrowserSampleData";

Convert to Dataset and replace blank entries with Missing[]:

In[]:=

userAgentDataset=Map[If[#==="",Missing[],#]&,Dataset[AssociationThread[First@userAgentData->#]&/@Rest@userAgentData],2];

Random sample of entries:

RandomSample[userAgentDataset,5]

Out[]=

id	user_agent	times_seen	simple_software_string	simple_sub_description_string	simple_operating_platform_string	software	software_name	software_name_code	software_version
6376	Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.0.19) Gecko/2010031218 Firefox/3.0.19	105	Firefox 3 on Mac OS X (Leopard)	—	—	Firefox 3	Firefox	firefox	3
3678	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; ms-office)	28	Office on Windows 7	—	—	Office	Office	office	—
10270	Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20100101 Firefox/20.0	7775	Firefox 20 on Windows XP	—	—	Firefox 20	Firefox	firefox	20
6503	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)	992	Internet Explorer 8 on Windows XP	—	—	Internet Explorer 8	Internet Explorer	internet-explorer	8
1090	Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; GT-P5100 Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30	48	Android Browser 4 on Android (Jelly Bean)	—	Samsung GT-P5100	Android Browser 4	Android Browser	android-browser	4
columns 1–10 of 39

Classifying

Many properties in the database look suitable to be treated as classes for Classify to train on. However, we first do some preprocessing on the user-agent string to remove unnecessary components and make it simpler for Classify.

Preprocessing Functions

We use 3 different methods of preprocessing: get the list of comments, get the list of products, and get both.

Get lists of comments and flatten to a single list:

getAllComments[userAgent_String]:=Flatten@splitUserAgent[userAgent]["Comments"];getAllComments[userAgentExample]

Out[]=

{X11,Linux x86_64,KHTML, like Gecko}

Get product pairs and concatenate back to slash-separated form:

getAllProducts[userAgent_String]:=KeyValueMap[If[MissingQ@#2,#1,#1<>"/"<>#2]&,splitUserAgent[userAgent]["Products"]];getAllProducts[userAgentExample]

Out[]=

{Mozilla/5.0,AppleWebKit/537.36,Chrome/51.0.2704.103,Safari/537.36}

Get both comments and products:

getAllComponents[userAgent_String]:=getAllComments@userAgent⋃getAllProducts@userAgent;getAllComponents[userAgentExample]

Out[]=

{AppleWebKit/537.36,Chrome/51.0.2704.103,KHTML, like Gecko,Linux x86_64,Mozilla/5.0,Safari/537.36,X11}

Training Data

Through some experimental testing, we determine the best preprocessor for each property. The results roughly match up with where the information is found in the string: Software information is usually in a product, layout engine information can be either, and other information is usually in comments.

Note that some properties have been excluded, either due to being more complicated, being equivalent to another property, or because they were meant for human reading.

Association between preprocessors and the group of properties they work best with:

In[]:=

classifyGroups=<|getAllComments->{"operating_system_name","operating_system_version","operating_system_version_full","operating_system_flavour","operating_platform","operating_platform_vendor_name","software_type","hardware_type","hardware_sub_type"},getAllComponents->{"software_version","software_version_full","layout_engine_name","layout_engine_version"},getAllProducts->{"software_name","software_sub_type"}|>;

We now convert this information to train a classifier for each property.

Get unprocessed pairs of user agents and a given property:

getRawExamples[prop_String]:=Select[Rule@@@Normal@userAgentDataset[[All,{"user_agent",prop}]],!MissingQ@#[[2]]&];RandomSample[getRawExamples["operating_system_name"],5]

Out[]=

{Mozilla/5.0 (Linux; Android 4.4.2; A518 Build/MocorDroid) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36Android,Mozilla/5.0 (Linux; Android 5.0.2; MotoE2(4G-LTE) Build/LXI22.50-29.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.93 Mobile Safari/537.36Android,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 ( .NET CLR 3.5.30729)Windows,Mozilla/5.0 (Windows NT 6.3; rv:29.0) Gecko/20100101 Firefox/29.0Windows,Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)Windows}

Process each pair's user agent:

getProcessedExamples[prop_String,processor_]:=MapAt[processor,getRawExamples@prop,{All,1}];RandomSample[getProcessedExamples["operating_system_name",getAllComponents],5]

Out[]=

{{AppleWebKit/533.3,en-US,Firestorm-Releasex64,KHTML, like Gecko,Mozilla/5.0,Safari/533.3,SecondLife/4.7.3.47323,U,vintage skin,Windows,Windows NT 6.1}Windows,{compatible,GTB7.5,Mozilla/4.0,MSIE 8.0,.NET4.0C,.NET4.0E,.NET CLR 1.1.4322,.NET CLR 2.0.50727,.NET CLR 3.0.4506.2152,.NET CLR 3.5.30729,Trident/4.0,Windows NT 5.1}Windows,{compatible,Mozilla/5.0,MSIE 9.0,Trident/5.0,Windows NT 6.0,WOW64}Windows,{Android 5.1,AppleWebKit/537.36,Chrome/39.0.0.0,Google Nexus 7 - 5.1.0 - API 22 - 800x1280 Build/LMY47D,KHTML, like Gecko,Linux,Mozilla/5.0,Safari/537.36,Version/4.0}Android,{AppleWebKit/537.36,Chrome/44.0.2403.125,Dragon/44.5.7.268,KHTML, like Gecko,Mozilla/5.0,Safari/537.36,Windows NT 5.1}Windows}

Map each property to its processed training data:

In[]:=

classifyGroupsData=MapIndexed[<|"Property"->#1,"Data"->getProcessedExamples[#1,#2[[1]][[1]]]|>&,classifyGroups,{2}];

Distribute each preprocessor over its properties:

In[]:=

classifyGroupsList=Flatten@Values@MapIndexed[Prepend[#1,"Preprocessor"->#2[[1]][[1]]]&,classifyGroupsData,{2}];

Properties we want to know about each classifier:

In[]:=

classifyProps={"ClassifierFunction","Accuracy","AccuracyBaseline","EvaluationTime","BatchEvaluationTime"};

A function to train and test a classifier for a given set of training data:

In[]:=

classifyTest[data:{___Rule}]:=AssociationThread[classifyProps,ClassifierMeasurements[Classify@data,data,classifyProps]];

Results

Train and test a classifier on each entry:

classifyResults=Dataset@Association@ParallelMap[#Property->Join[<|"Examples"->Length@#Data,"Preprocessor"->#Preprocessor|>,classifyTest[#Data]]&,classifyGroupsList]

Out[]=

Note that since "not applicable" is treated as missing, some ClassifierFunctions output nonsense when they don't apply.

Example classification:

userAgentExample#ClassifierFunction[#Preprocessor[userAgentExample],{"Decision","TopProbabilities"}]&/@classifyResults

Out[]=

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36

Out[]=

operating_system_name	Linux	{Linux0.997889}
operating_system_version	10	{100.750936,90.187194}
operating_system_version_full	10.1	{10.10.618106}
operating_system_flavour	Ubuntu	{Ubuntu0.852983,Fedora0.103621}
operating_platform	iPad	{iPad0.955774}
operating_platform_vendor_name	Apple	{Apple0.997215}
software_type	browser	{browser0.999721}
hardware_type	computer	{computer0.989073}
hardware_sub_type	tablet	{tablet0.835627,phone0.149216}
software_version	43	{ … 12 }
software_version_full	45.0.2454.85	{45.0.2454.850.258279,45.0.2454.840.223283,44.0.2403.1570.149869,44.0.2403.1550.0951002,44.0.2403.1330.0670966,28.0.1500.940.036073,43.0.2357.930.0267238}
layout_engine_name	Blink	{Blink0.97819}
layout_engine_version	["537", "36"]	{["537", "36"]0.727072,[]0.272928}
software_name	Chrome	{Chrome0.736635,Opera0.0812056,Samsung Browser0.0752063}
software_sub_type	web-browser	{web-browser0.937175}

Browser Spoofing Graph

We wish to find an efficient way to search through all browsers and programs to find the one that matches a given User Agent string the closest. Since browsers typically copy a different browser's User Agent string and attach their own product, we can create a directed graph of the relationship between the product names in User Agent strings, with each path representing the products added between two User Agent strings.

Procedure

Since two programs may share the same products in their User Agent strings, we can have each vertex represent a set of products, and the label of each vertex lists the programs that use that set of products.

An example graph of the relationship between browsers:

exampleGraph=

example graph

Out[]=

A formatted version with more information displayed and tooltips from hovering:

formatGraph[g_Graph]:=Graph[g,ImageSize->Full,VertexLabels->(#[[1]]->Placed[Tooltip[Style[Framed[#[[2]][[1]],FrameMargins->0],Background->RGBColor[1,1,1,0.75]],Column[{#[[1]],Multicolumn@Sort@#[[2]]}]],Center]&/@AnnotationValue[g,VertexLabels]),EdgeLabels->{e:_List_List:>Style[Complement[e[[2]],e[[1]]],Background->RGBColor[1,1,1,0.75]]},EdgeShapeFunction->({Arrowheads[{{Max[0.01*Norm@Subtract[#[[2]],#[[1]]],0.01],0.7}}],Arrow@#}&)];exampleGraph//formatGraph

Out[]=

To add a new program to the graph, we find the closest potential parents (incoming vertices) to the program's product list. These are the vertices whose products are a subset of the new program's products, and whose children (outgoing vertices)'s products are not (since these would be closer parents).

A fake example browser:

In[]:=

exampleBrowser="Cool Browser"->{"Opera","Presto","Version","Mozilla","Gecko","Chrome"};

Get the children of a vertex:

In[]:=

outgoingList[g_Graph,v_]:=VertexOutComponent[g,{v},{1}]

Find the parent vertices:

getParentVertices[g_Graph,v:{___String}]:=Reap[DepthFirstScan[g,{},{"PrevisitVertex"->(If[Length@Complement[#,v]==0&&AllTrue[outgoingList[g,#],Length@Complement[#,v]!=0&],Sow[#]]&)}]][[2]][[1]];HighlightGraph[exampleGraph,getParentVertices[exampleGraph,exampleBrowser[[2]]]]//formatGraph