When trying to gather information from a website:
DefaultHttpRequester req = new DefaultHttpRequester();
req.Headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0";
IConfiguration config = Configuration.Default.With(req).WithDefaultLoader().WithDefaultCookies();
var afdas = BrowsingContext.New(config).OpenAsync("https://www.wg-gesucht.de/api/profiles/contact-data/5520144?asset_type=0&asset_id=6839242").Result;
I got an ERROR-Message:
<html>
<head></head>
<body>{"type":"http:\/\/www.w3.org\/Protocols\/rfc2616\/rfc2616-sec10.html","title":"Not Acceptable","status":406,"detail":"Cannot honor Accept type specified"}</body>
</html>
Based on the ERROR-Message:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
406 Not Acceptable
The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.
Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.
Note: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.If the response could be unacceptable, a user agent SHOULD temporarily stop receipt of more data and query the user for a decision on further actions.
I have added all headers which could be seen in Firefox:
DefaultHttpRequester req = new DefaultHttpRequester();
req.Headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0";
req.Headers["Accept"] = "*/*"; // https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
req.Headers["Accept-Encoding"] = "gzip, deflate, br";
req.Headers["Accept-Language"] = "de,en-US;q=0.7,en;q=0.3";
req.Headers["Connection"] = "keep-alive";
req.Headers["Sec-Fetch-Dest"] = "document";
req.Headers["Sec-Fetch-Mode"] = "navigate";
req.Headers["Sec-Fetch-Site"] = "cross-site";
req.Headers["Upgrade-Insecure-Requests"] = "1";
IConfiguration config = Configuration.Default.With(req).WithDefaultLoader().WithDefaultCookies();
var afdas = BrowsingContext.New(config).OpenAsync("https://www.wg-gesucht.de/api/profiles/contact-data/5520144?asset_type=0&asset_id=6839242").Result;
But I still got the same error, so what did I do wrong?
This worked for me