Using HTTPWebRequest & HTTPWebResponse to automate web browsing
One way is to automate this process. Here I demonstrate a simple script to fetch one page and read its content. Here is the sample code:
1> string URL = "http://www.google.com";
3> HttpWebRequest HWR = (HttpWebRequest)HttpWebRequest.Create(URL);
4> HWR.Method = "GET";
5> StreamReader SR = new StreamReader(HWR.GetResponse().GetResponseStream());
6> string Response = SR.ReadToEnd();
8> string Pattern = @"dd?d?.dd?d?.dd?d?.dd?d?";
9> Regex R = new Regex(Pattern, RegexOptions.Singleline | RegexOptions.IgnoreCase);
10> Match M = R.Match(Response);
11> string IP = M.ToString();
Line 1 sets up the web page to download. You can set this to a sub page as well.
Line 3 creates a new HTTPWebRequest object. This class in .NET allows you to make HTTPWebRequest and supports custom headers as well. In this example we have not used any headers.
Line 4 sets the request method. Most popular are GET and POST. POST is used when submitting forms. GET is the basic request type for fetching a page.
Line 5 HTTP protocol is a streaming protocol and hence we require a StreamReader to read the response
Line 6 captures the response as an in-memory string. This might be ok for smaller pages but for larger pages you might want to consider saving to disk or an external database
Line 8 demonstrates parsing page response to extract some data (in this case IP address). Any IP address in a valid format is extracted. Since Match class is used only the first match is returned which is stored in the string IP
RegEx itself is a popular class in .NET and worth learning at the earliest. It has been very useful for me. Will blog about it later.
- Color-based Blob Detection
- RAID 1 vs RAID 5
- Setting up IIS 7.5 and Apache on same server
- Buying vs renting
- Changing domain IP address without modifying DNS
- Motivation for hand-optimized Assembly code
- Fiddling with Fiddler
- ARM’s 64-bit mode (AArch64) ARMv8)
- Color Conversions
- Hardware Platforms for Embedded Computer Vision, Image Processing and Deep Learning