Recently, I had the need to parse html documents for special bits of data. Of course, the data resided on multiple sites, all of which had their own structure, classes, etc..  Building regular expressions for the parsing was my first thought, although I knew that using REGEX comes at a cost.  After doing a little research, I found a .Net library, the HTML Agility Pack hosted on CodePlex.

This library is probably one of the best I've used to date.  It's flexible enough to work around malformed HTML and it is very fast.  It takes longer to make and receive a response from the web server containing the page, then it does to load and parse through thousands of nodes from a given document.

An example on how to use the library is below:

WebClient webClient = new WebClient();
byte[] reqHTML;
StreamWriter sw = null;
try
   {
   reqHTML = webClient.DownloadData(URL_TO_FETCH);
   ASCIIEncoding objASC = new ASCIIEncoding();
   String rs = objASC.GetString(reqHTML);
   sw = new StreamWriter(SAVE_FILE_TO_DISK_PATH);
   sw.Write(rs); sw.Flush(); sw.Close(); sw.Dispose();
   }
catch (Exception ex) { errlog.LogError(---)}
finally { if (sw != null) { sw.Dispose(); }}

//Create a new doc object
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(SAVE_FILE_TO_DISK_PATH);
 //get a result set, use an XPath expression to navigate to the starting point of where you want to parse.
 var results = from node in doc.DocumentNode.SelectNodes(@"/html[1]/body[1]/div[2]/div[1]/div[1]/div[3]")
                                      select node;
 //Loop through the results, save/parse what you want. 
 foreach (HtmlNode node in results) { LOOP and EVALUATE }

I was thrilled to find this library, so I thought I would share my little snippet and shameless plug for all their hard work.