Message Boards

GROUPS:

Austin Geng

[WSC21] Exploring methods of parsing user agent strings

Austin Geng

Posted 6 months ago

735 Views

0 Replies

3 Total Likes

Follow this post

Graph of User Agent Spoofing

Exploring methods of parsing User Agent strings
by Austin Geng

The objective of this project was to create a User Agent string parser. We begin by breaking up the string into a simpler, more abstract form. We then use the built-in Classify function with multiple forms of preprocessing. Lastly, we construct a graph of user agents based on their common components.

Introduction

User Agent strings are sent by a program communicating with the Web (such as a browser or web crawler bot) to provide information about itself to a server, such as its software, version, and operating system. This allows the server to send content appropriate to the limitations of the user agent.

However, it is up to the server to figure out what features each program supports, and so it may not understand a new or updated program. As a result, there has been a long history of browsers copying each other’s User Agent strings to “trick” servers into giving them the correct features. This has led to almost all modern browsers having convoluted User Agent strings; for example, Chrome pretends to be Safari, Mozilla, and Netscape and pretends to use the WebKit, KHTML, and Gecko layout engines.

Furthermore, although there are standards and conventions for the format of User Agent strings, these are sometimes ignored, and in these cases, anything beyond string-matching becomes tricky.

Although feature detection through the User Agent string is now discouraged, the information found within the string still has uses for operating system detection and web analytics.

Splitting up the string

An example (Chrome) user agent string:

In[]:=

userAgentExample="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36";

The typical user agent string is composed of a list of products (software names, often followed by the version after a slash) and comments (parenthesized groups of other terms delimited by semicolons). We can break up these strings with relative ease.

Split the string by spaces (excluding those inside a comment):

tokens=StringSplit[userAgentExample,RegularExpression[" (?![^(]*\\))"]]

Out[]=

{Mozilla/5.0,(X11; Linux x86_64),AppleWebKit/537.36,(KHTML, like Gecko),Chrome/51.0.2704.103,Safari/537.36}

Group by whether or not a token is a comment:

grouped=GroupBy[tokens,StringMatchQ@RegularExpression["\\(.*\\)"]]

Out[]=

False{Mozilla/5.0,AppleWebKit/537.36,Chrome/51.0.2704.103,Safari/537.36},True{(X11; Linux x86_64),(KHTML, like Gecko)}

Split each product's name and version and convert to an association:

products=Association[Rule@@PadRight[#,2,Missing[]]&/@(StringSplit[#,"/",All]&/@Replace[grouped[False],_Missing->{}])]

Out[]=

Mozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36

Split each comment by semicolons and remove parentheses:

comments=StringSplit[StringTake[#,{2,-2}],"; "]&/@Replace[grouped[True],_Missing->{}]

Out[]=

{{X11,Linux x86_64},{KHTML, like Gecko}}

Combine products and comments:

<|"Products"->products,"Comments"->comments|>

Out[]=

ProductsMozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36,Comments{{X11,Linux x86_64},{KHTML, like Gecko}}

All together:

splitUserAgent[userAgent_String]:=Module[{tokens,grouped,products,comments},tokens=StringSplit[userAgent,RegularExpression[" (?![^(]*\\))"]];grouped=GroupBy[tokens,StringMatchQ@RegularExpression["\\(.*\\)"]];products=Association[Rule@@PadRight[#,2,Missing[]]&/@(StringSplit[#,"/",All]&/@Replace[grouped[False],_Missing->{}])];comments=StringSplit[StringTake[#,{2,-2}],"; "]&/@Replace[grouped[True],_Missing->{}];<|"Products"->products,"Comments"->comments|>];splitUserAgent[userAgentExample]

Out[]=

ProductsMozilla5.0,AppleWebKit537.36,Chrome51.0.2704.103,Safari537.36,Comments{{X11,Linux x86_64},{KHTML, like Gecko}}

Interactive version with Manipulate:

Manipulate[DynamicModule[{split},split=splitUserAgent[userAgent];Row[Pane[#,BaselinePositionTop]&/@Prepend[Dataset/@split["Comments"],Dataset@split["Products"]],Spacer[10]]],{{userAgent,userAgentExample,"User Agent"},InputField[#,String,BaseStyle->12,FieldSize->{{60.,60.},{1.,Infinity}}]&},SaveDefinitionsTrue]

Out[]=

User Agent

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36

Mozilla	5.0
AppleWebKit	537.36
Chrome	51.0.2704.103
Safari	537.36

X11

Linux x86_64

KHTML, like Gecko

Data gathering

I was fortunate enough to find a sample database of already-analyzed User Agent strings from the website whatismybrowser.com.

Import data:

In[]:=

userAgentData=CloudImport@CloudObject@"https://www.wolframcloud.com/obj/yygengjunior/whatismybrowserSampleData";

Convert to Dataset and replace blank entries with Missing[]:

In[]:=

userAgentDataset=Map[If[#==="",Missing[],#]&,Dataset[AssociationThread[First@userAgentData->#]&/@Rest@userAgentData],2];

Random sample of entries:

RandomSample[userAgentDataset,5]

Out[]=

id	user_agent	times_seen	simple_software_string	simple_sub_description_string	simple_operating_platform_string	software	software_name	software_name_code	software_version
6376	Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.0.19) Gecko/2010031218 Firefox/3.0.19	105	Firefox 3 on Mac OS X (Leopard)	—	—	Firefox 3	Firefox	firefox	3
3678	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; ms-office)	28	Office on Windows 7	—	—	Office	Office	office	—
10270	Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20100101 Firefox/20.0	7775	Firefox 20 on Windows XP	—	—	Firefox 20	Firefox	firefox	20
6503	Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)	992	Internet Explorer 8 on Windows XP	—	—	Internet Explorer 8	Internet Explorer	internet-explorer	8
1090	Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; GT-P5100 Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30	48	Android Browser 4 on Android (Jelly Bean)	—	Samsung GT-P5100	Android Browser 4	Android Browser	android-browser	4
columns 1–10 of 39

Classifying

Many properties in the database look suitable to be treated as classes for Classify to train on. However, we first do some preprocessing on the user-agent string to remove unnecessary components and make it simpler for Classify.

Preprocessing Functions

We use 3 different methods of preprocessing: get the list of comments, get the list of products, and get both.

Get lists of comments and flatten to a single list:

getAllComments[userAgent_String]:=Flatten@splitUserAgent[userAgent]["Comments"];getAllComments[userAgentExample]

Out[]=

{X11,Linux x86_64,KHTML, like Gecko}

Get product pairs and concatenate back to slash-separated form:

getAllProducts[userAgent_String]:=KeyValueMap[If[MissingQ@#2,#1,#1<>"/"<>#2]&,splitUserAgent[userAgent]["Products"]];getAllProducts[userAgentExample]

Out[]=

{Mozilla/5.0,AppleWebKit/537.36,Chrome/51.0.2704.103,Safari/537.36}

Get both comments and products:

getAllComponents[userAgent_String]:=getAllComments@userAgent⋃getAllProducts@userAgent;getAllComponents[userAgentExample]

Out[]=

{AppleWebKit/537.36,Chrome/51.0.2704.103,KHTML, like Gecko,Linux x86_64,Mozilla/5.0,Safari/537.36,X11}

Training Data

Through some experimental testing, we determine the best preprocessor for each property. The results roughly match up with where the information is found in the string: Software information is usually in a product, layout engine information can be either, and other information is usually in comments.

Note that some properties have been excluded, either due to being more complicated, being equivalent to another property, or because they were meant for human reading.

Association between preprocessors and the group of properties they work best with:

In[]:=

classifyGroups=<|getAllComments->{"operating_system_name","operating_system_version","operating_system_version_full","operating_system_flavour","operating_platform","operating_platform_vendor_name","software_type","hardware_type","hardware_sub_type"},getAllComponents->{"software_version","software_version_full","layout_engine_name","layout_engine_version"},getAllProducts->{"software_name","software_sub_type"}|>;

We now convert this information to train a classifier for each property.

Get unprocessed pairs of user agents and a given property:

getRawExamples[prop_String]:=Select[Rule@@@Normal@userAgentDataset[[All,{"user_agent",prop}]],!MissingQ@#[[2]]&];RandomSample[getRawExamples["operating_system_name"],5]

Out[]=

{Mozilla/5.0 (Linux; Android 4.4.2; A518 Build/MocorDroid) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36Android,Mozilla/5.0 (Linux; Android 5.0.2; MotoE2(4G-LTE) Build/LXI22.50-29.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.93 Mobile Safari/537.36Android,Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 ( .NET CLR 3.5.30729)Windows,Mozilla/5.0 (Windows NT 6.3; rv:29.0) Gecko/20100101 Firefox/29.0Windows,Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)Windows}

Process each pair's user agent:

getProcessedExamples[prop_String,processor_]:=MapAt[processor,getRawExamples@prop,{All,1}];RandomSample[getProcessedExamples["operating_system_name",getAllComponents],5]

Out[]=

{{AppleWebKit/533.3,en-US,Firestorm-Releasex64,KHTML, like Gecko,Mozilla/5.0,Safari/533.3,SecondLife/4.7.3.47323,U,vintage skin,Windows,Windows NT 6.1}Windows,{compatible,GTB7.5,Mozilla/4.0,MSIE 8.0,.NET4.0C,.NET4.0E,.NET CLR 1.1.4322,.NET CLR 2.0.50727,.NET CLR 3.0.4506.2152,.NET CLR 3.5.30729,Trident/4.0,Windows NT 5.1}Windows,{compatible,Mozilla/5.0,MSIE 9.0,Trident/5.0,Windows NT 6.0,WOW64}Windows,{Android 5.1,AppleWebKit/537.36,Chrome/39.0.0.0,Google Nexus 7 - 5.1.0 - API 22 - 800x1280 Build/LMY47D,KHTML, like Gecko,Linux,Mozilla/5.0,Safari/537.36,Version/4.0}Android,{AppleWebKit/537.36,Chrome/44.0.2403.125,Dragon/44.5.7.268,KHTML, like Gecko,Mozilla/5.0,Safari/537.36,Windows NT 5.1}Windows}

Map each property to its processed training data:

In[]:=

classifyGroupsData=MapIndexed[<|"Property"->#1,"Data"->getProcessedExamples[#1,#2[[1]][[1]]]|>&,classifyGroups,{2}];

Distribute each preprocessor over its properties:

In[]:=

classifyGroupsList=Flatten@Values@MapIndexed[Prepend[#1,"Preprocessor"->#2[[1]][[1]]]&,classifyGroupsData,{2}];

Properties we want to know about each classifier:

In[]:=

classifyProps={"ClassifierFunction","Accuracy","AccuracyBaseline","EvaluationTime","BatchEvaluationTime"};

A function to train and test a classifier for a given set of training data:

In[]:=

classifyTest[data:{___Rule}]:=AssociationThread[classifyProps,ClassifierMeasurements[Classify@data,data,classifyProps]];

Results

Train and test a classifier on each entry:

classifyResults=Dataset@Association@ParallelMap[#Property->Join[<|"Examples"->Length@#Data,"Preprocessor"->#Preprocessor|>,classifyTest[#Data]]&,classifyGroupsList]

Out[]=

	Examples	Preprocessor	ClassifierFunction	Accuracy	AccuracyBaseline	EvaluationTime	BatchEvaluationTime
operating_system_name	9537	getAllComments	…	0.977142	0.467547	0.00401566	0.0000778403
operating_system_version	8608	getAllComments	…	0.961083	0.230832	0.00492416	0.0000869314
operating_system_version_full	8608	getAllComments	…	0.946678	0.225604	0.0066348	0.0000988949
operating_system_flavour	129	getAllComments	…	0.953488	0.674419	0.00457104	0.000143267
operating_platform	2722	getAllComments	…	0.537472	0.0870683	0.0349697	0.000191846
operating_platform_vendor_name	3078	getAllComments	…	0.87037	0.387589	0.00391326	0.0000770428
software_type	9934	getAllComments	…	0.964465	0.976243	0.00420412	0.000082984
hardware_type	9932	getAllComments	…	0.95882	0.563834	0.0037111	0.0000750743
hardware_sub_type	2684	getAllComments	…	0.964605	0.637109	0.00402311	0.000094357
software_version	9477	getAllComponents	…	0.859344	0.125989	0.0056535	0.000117885