When crawling, i notice on every page there is a cookie request. With the "IsSendingCookiesEnabled = true,", i thought this would disappear.
However, as the cookies are not confirmed ( this is normally done by the user in a screen version and not in the headless version ), how can it save the cookies at first ?
Can i access the site first with firefox, then save the cookies and then let the site be crawled so that the cookie question no longer appears ?
Is the user agent important for the recognition of the cookies ?