C# Use WebClient to log in to the website and crawl the logged-in web page information method

This article describes the implementation method of C# using WebClient to log in to a website and crawling login web page information. Share it for your reference, as follows:

C# login to the website is actually a simulation of the browser submitting a form and then recording the session cookie value returned by the browser response. When sending the request again, you can achieve the simulation login effect by bringing this session cookie value to the request.

The following CookieAwareWebClient implementations carry cookies when sending requests.

public class CookieAwareWebClient : WebClient
{
  private CookieContainer cookie = new CookieContainer();
  protected override WebRequest GetWebRequest(Uri address)
  {
    WebRequest request = (address);
    if (request is HttpWebRequest)
    {
      (request as HttpWebRequest).CookieContainer = cookie;
    }
    return request;
  }
}

The following is an example of using mock form submission login:

var client = new CookieAwareWebClient();
 = @"/any/base/url/";
var loginData = new NameValueCollection();
("login", "YourLogin");
("password", "YourPassword");
("", "POST", loginData);
//Now you are logged in and can request pages
string htmlSource = ("");

For more information about C#, please visit the special topic of this site:Summary of C# coding operation skills》、《Summary of XML file operation skills in C#》、《Tutorial on the usage of common C# controls》、《Summary of WinForm control usage》、《C# data structure and algorithm tutorial》、《Introduction to C# object-oriented programming tutorial"and"Summary of thread usage techniques for C# programming》

I hope this article will be helpful to everyone's C# programming.