Welcome to Wolfram Community!
Please make sure you know the rules: https://wolfr.am/READ-1ST
Images don't help other members to answer your question. They need to copy the elements involved in your question such as code and data.
If you don't want to share your data file, you can create a sample file that contains part of your data and attach it, otherwise we won't be able to help you.

Please next time mention that this post is a continuation for another earlier question that you already had some answers on it.

Hello, Rohit, Thank you for your reply - i have uploaded a screenshot of the CSV file - For some reason the website is placing two of the same pictures.

Hi Rohit, I have now attached a sample csv. I have tried to use the map function, but I was not able to obtain the results. So above, i am using another method to retrieve the data, and taking your previous advice, I am partitioning my data gathering process into three parts.

1) the urls in the csv file above takes me to an index page. What I am trying to do above is to go into each of the URLs and scrape the part where it says "xxx.htm" and put it into a variable because I would like to use the "xxx.htm" part to get to the actual webpage that I would like to scrape.

2) the next process I would like to conduct (not part of this question) is to have Mathematica go into each of the links and download the texts of the .htms I get from step #1

3) then I will run a textual analysis based on the saved texts from #2

Sorry for the confusion I hope I'm making it clearer. I truly appreciate your time!

Hi Young,

Thanks for providing the data in a usable form.

In your example you extracted "proxy2010.htm". Is that the end result you want for each row in the CSV? Based on the rest of the question, seems like the answer is "No", because you want to scrape "proxy2010.htm", but that is not possible because it is not a full URL, domain and path are missing.

Maybe you want this

getProxyStatementURL[link_] := Import[link, "Hyperlinks"] // 
   Select[StringMatchQ[#, __ ~~ "Archives" ~~ __ ~~ ".htm"] &] // 
   First

Import["~/Downloads/def14asample.csv", "Dataset", HeaderLines -> 1]

dataWithProxyStatementURL = 
 data[All, <|#, "proxyStatementURL" -> getProxyStatementURL[#["path"]]|> &]

The result is a Dataset with a new column proxyStatementURL which has the full URL to the proxy statement, so rather than just "proxy2010.htm", it is "https://www.sec.gov/Archives/edgar/data/34782/000003478210000011/proxy2010.htm"

You can then Import the plaintext from those URL's and do whatever processing you need to do. If you need help with that part, please specify the details in your response.

Not sure why it did not work. Did you change the path to "def14asample.csv" to the right one on your system. Anyway, I have attached a notebook with the working solution. To make it simpler, just make sure "def14asample.csv" is in the same folder as the notebook.

However, may I also ask how i could do this analysis for 100 rows (100 proxy statements)?

The code in the notebook will process all of the rows from the imported CSV. You can see this in the attached notebook.

Attachments:

ProxyStatementURL.nb

Hi Young,

Looks like there are issues with the code in "Notebook-9-3-Conservative Range COPY.nb", several cells failed to evaluate (I did not evaluate any cells, the errors are present in the notebook). I have a hard time following the code in the notebook.

Some expressions have much simpler implementations

StringReplace[#, (StartOfString ~~ Whitespace) | (Whitespace ~~ EndOfString) :> ""] & /@ data0;

is the same as the following, assuming you want to remove all whitespaces from the beginning and end of the string.

StringTrim@data1

Some parts of the code look incorrect

y2019 = Length[StringPosition[data2, "2019"]];
y2018 = Length[StringPosition[data2, "2018"]];
y2017 = Length[StringPosition[data2, "2017"]];
y2016 = Length[StringPosition[data2, "2016"]];
y2015 = Length[StringPosition[data2, "2015"]];
y2014 = Length[StringPosition[data2, "2014"]];
y2013 = Length[StringPosition[data2, "2013"]];
y2012 = Length[StringPosition[data2, "2012"]];
y2011 = Length[StringPosition[data2, "2011"]];
y2010 = Length[StringPosition[data2, "2010"]];

Since data2 is a list of strings, StringPosition is going to return a list for every string (empty list for no match). The Length is going to be identical in all cases so this is not going to give the right fiscal year

fiscalyear = 
  Max[y2019, y2018, y2017, y2016, y2015, y2014, y2013, y2012, y2011, y2010];

Rather than enumerating every year, why not search for a pattern

"20"~~DigitCharacter~~DigitCharacter

Also not clear why the Max is used to determine fiscal year. StringPosition is going to return the position in each string, not the position in the entire document.

When the following is evaluated, only a and b are defined, so Max will not evaluate, so If will not evaluate.

truename = 
  If[a == Max[a, b, c, d, e, f, g, h, i, j, k], 
   "name of registrant as specified in its charter)", 
   If[b == Max[a, b, c, d, e, f, g, h, i, j, k], 
    "name of registrant as specified in its certificate)"]];

The rest of the code has even more complex logic that looks incorrect and there are no comments in the code.

Anyway, assuming you are able to get your code to work correctly, wrap it in a function that accepts the proxy url and returns the desired result. Here is an example that just returns an association of word and sentence counts.

processProxyURL[proxyURL_] := 
 Module[{plainText, allWords, nonStopWords, sentences},
  plainText = Import[proxyURL, "Plaintext"];
  allWords = TextWords@plainText;
  nonStopWords = DeleteStopwords@allWords;
  sentences = TextSentences@plainText;
  <|"Words" -> Length@allWords, 
   "Non Stopwords" -> Length@nonStopWords, 
   "Sentences" -> Length@sentences|>
  ]

To run it on every proxy URL in the dataset and add the result as a new column

dataWithProxyStatementURL[All, <|#, "Processed Data" -> processProxyURL[#["proxyStatementURL"]]|> &]

Hi Young,

Glad to hear that you are making progress. There are a few problems with the code.

allWords is a list of words, so a phrase like "based on discretion" will never match a word. The match should be done against plainText. The right function to use is StringCount, not Position. allWords contains newlines, so a phrase you are looking for can be split across lines and will not match exactly. The word/phrase in plainText may be in mixed case, but the words/phrases you are looking for are all lowercase. Much better to use allWords converted to lowercase and joined with spaces to form a single string of all the words.

keyWords = {"judgment-based", "based on discretion", "exercise judgment", "exercising judgment", 
   "exercise discretion", "exercising discretion", "subjectivity", "discretion", 
   "not based on any mathematical", "not quantifiable", "adjusted upward", "adjusted up", 
   "upward adjustment", "qualitative",  "non-financial", "nonfinancial"};

cleanPlainText = StringRiffle@ToLowerCase@allWords
keyWordCounts = AssociationMap[StringCount[cleanPlainText, #] &, keyWords]

(*
<|"judgment-based" -> 0, "based on discretion" -> 0, 
 "exercise judgment" -> 0, "exercising judgment" -> 0, 
 "exercise discretion" -> 0, "exercising discretion" -> 0, 
 "subjectivity" -> 0, "discretion" -> 2, 
 "not based on any mathematical" -> 0, "not quantifiable" -> 0, 
 "adjusted upward" -> 0, "adjusted up" -> 0, "upward adjustment" -> 0,
  "qualitative" -> 0, "non-financial" -> 0, "nonfinancial" -> 0|>
*)

Similarly for keywords2 (give it a better name) and trapWords. Add keyWordCounts and the counts for the other lists to the results association i.e. "keywordsfrequency" -> keyWordCounts. etc.

You should manually check that the counts match by searching for the text in the browser to make sure there are no other edge cases to be considered.