Using web browser macros, the need to get more control over what is happening during the automated use of a web page becomes increasingly obvious. After trying a number of possible solutions, C# and .NET provided the best option. Find out how.

There are lots of reasons why you might want to automate your interaction with a web page. In this project the example, and this is a real example, is of an ISP providing a monthly bandwidth allowance. The problem is keeping track of how much data has been used via the ISP's website. Checking it manually is a time- consuming chore. Automating the job is made more difficult by the need to log on, navigate to the correct page, and then extract the data. While this can be done using browser macro language, it doesn't provide an easy way to process the retrieved data. 

My first attempts were to try to create a JavaScript web app to do the job, but there are problems. JavaScript won't let you perform a cross domain Ajax call so you can't easily retrieve any arbitrary page on the web. There are ways around this, but they are all messy or require the cooperation of the owner of the website you are downloading. 

After a lot of consideration the best and quickest solution was to create a classic desktop app for Windows using C#. The complete program listing is provided the end of the article.

The key to the problem is the WebBrowser control/class. This is a managed wrapper around an ActiveX browser component. Back in the days when ActiveX was the way to do components, Microsoft implemented IE as an ActiveX module and this is still the case today. However, to make use of the WebBrowser you don't need to know how this works, you can simply use it.  What you do need to know is that the WebBrowser control is based on whatever version of IE is installed on the system and for some web pages this might make a difference. 

Creating The WebBrowser

Start a new C# Windows Forms project. To add a WebBrowser to your project you can simply select one from the Toolbox and drop the control on the form.  This has the advantage that you can size the control interactively and you can watch how the interaction is going because you can see the results of the page load. However in many cases, after you have debugged the application you don't want to see the web browser contents at all. Given that the web browser also uses a lot of resources you might want to load and unload it as required by the program. 

Putting all of this together suggests that creating the WebBrowser object in code is a good option. 

First we need a global variable to reference the WebBrowser:

private WebBrowser webBrowser1;

and then we can create an instance when ever it is needed.

Add a multiline Textbox to the form and a button - the Textbox is used to display the results and the button starts the interaction with the web page. 

Next we need a makeBrowser method:

private void makeBrowser()
{
 webBrowser1 = new System.Windows.
                         Forms.WebBrowser();
 webBrowser1.Name = "webBrowser1";
}

While we don't actually need a method to show the browser it is invaluable during debugging:

private void showBrowserInForm()
{
 webBrowser1.Location =
         new System.Drawing.Point(12, 300);
 webBrowser1.MinimumSize =
         new  System.Drawing.Size(20, 20);
 webBrowser1.Size =
         new System.Drawing.Size(783, 338);
 webBrowser1.TabIndex = 1;
 Controls.Add(webBrowser1);
}

This adds the control to the current form. We could have used a new form if you wanted to make the browser pop-up separately from the application.

Multipage

The first thing we have to figure out is how to navigate to a page and respond when the page is loaded. Navigating to a page is just a matter of setting the URL property to a valid URL - finding out when the page is fully loaded is a little more difficult. When the document is loaded it triggers a DocumentCompleted event. The problem is that this event is triggered for each frame loaded. This means that you could get multiple DocumentCompleted events per page load.  The solution to this problem is to test the ReadyState and make sure that it is set to Complete before processing the page. 

void webBrowser1_DocumentCompleted(
   object sender,    
   WebBrowserDocumentCompletedEventArgs e)
{
 if ( webBrowser1.ReadyState!=
     WebBrowserReadyState.Complete) return;

This however doesn't always work so be on your guard for attempts to process a page before it is complete. 

Next we need to find a way to keep track of where we are in the the overall process of using the web site. In most examples of using the WebBrowser control a single page is processed. In this case we have a log in page, a menu page and the results page to work with. 

How do we keep track of which page we are processing?

One solution is to change the event handler after each page is visited to an event handler that is going to process the next page. This is a possible approach but .NET doesn't make it easy to change the event handler - you have to remove the old one and then add the new one. 

An alternative approach is to write a single event handler that works for all the pages and works out which one using the current URL. This has the disadvantage that many websites change the URL you supply to something else or augment it with query parameters. Checking the URL can be made to work but it isn't robust against changes.

A simpler alternative is to use a "state machine" approach. 

All we do is to set a state indicator to the next page to be loaded and processed. In general the state indicator can be an object which holds multiple properties determining the current state of the system e.g. the target URL etc. 

In this case the state indicator can be just a simple variable:

private int page;

When the button is clicked we first need to navigate to the site's login page and set the state to 1 i.e. we are processing page one.

private void button1_Click(
             object sender,
             EventArgs e)
{
 page = 1;
 makeBrowser();
 showBrowserInForm();
 webBrowser1.DocumentCompleted += new
   WebBrowserDocumentCompletedEventHandler(
                DocumentCompleted);
 webBrowser1.Navigate(
             "url for login page");
}

Notice that it is up to you to provide the URL for the login page you want to work with. Also notice that the DocumentComplete event handler is also set up.