Using HTTPWebRequest & HTTPWebResponse to automate web browsing
One way is to automate this process. Here I demonstrate a simple script to fetch one page and read its content. Here is the sample code:
1> string URL = "http://www.google.com";
3> HttpWebRequest HWR = (HttpWebRequest)HttpWebRequest.Create(URL);
4> HWR.Method = "GET";
5> StreamReader SR = new StreamReader(HWR.GetResponse().GetResponseStream());
6> string Response = SR.ReadToEnd();
8> string Pattern = @"dd?d?.dd?d?.dd?d?.dd?d?";
9> Regex R = new Regex(Pattern, RegexOptions.Singleline | RegexOptions.IgnoreCase);
10> Match M = R.Match(Response);
11> string IP = M.ToString();
Line 1 sets up the web page to download. You can set this to a sub page as well.
Line 3 creates a new HTTPWebRequest object. This class in .NET allows you to make HTTPWebRequest and supports custom headers as well. In this example we have not used any headers.
Line 4 sets the request method. Most popular are GET and POST. POST is used when submitting forms. GET is the basic request type for fetching a page.
Line 5 HTTP protocol is a streaming protocol and hence we require a StreamReader to read the response
Line 6 captures the response as an in-memory string. This might be ok for smaller pages but for larger pages you might want to consider saving to disk or an external database
Line 8 demonstrates parsing page response to extract some data (in this case IP address). Any IP address in a valid format is extracted. Since Match class is used only the first match is returned which is stored in the string IP
RegEx itself is a popular class in .NET and worth learning at the earliest. It has been very useful for me. Will blog about it later.
- To RAID or not to RAID
- Getting your dynamic public IP in C# & .net
- Introduction to OpenCV
- Rotating or Resizing an Image in OpenCV
- Speed testing a website
- Color-based Blob Detection
- Mixing Assembly language with Visual Basic
- Tbot: The Self-Balancing Transformer Robot
- Do all versions of OpenCV run at the same speed?
- Buying vs renting