Fortiguard Servers Failed To Respond

[Dagny Westall]

Westarted getting this Web Filter error recently and it's blocking traffic to places like apple.com and microsoft.com. I don't know why Fortiguard servers would be failing to respond now. We had to remove Web filtering due to this error. Any ideas?

The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.

In FortiGate, webfilter does the real-time lookup for the Web Filter rating query.

If Fortigate cannot get the answer to what category the website belongs to, access to this website will be blocked by default.

In case for any reason Fortigate cannot reach Fortiguard servers rules where webfilter is called will start blocking the sites.

Below are a few points to check for the proper Fortiguard connectivity:-

>> Please check the Fortiguard license status

>> Confirm that on FGT DNS is getting resolved for update.fortinet.net, service.fortinet.net

>> From Fortigate service.fortinet.net should be pingable

>> Fortigate can use ports 53,8888,443 to talk to Fortiguard servers

>> Make sure that using the above ports firewall can reach the Fortiguard servers

The error "all FortiGuard servers failed to respond" indicates that your Fortinet device was unable to communicate with the FortiGuard servers. FortiGuard is Fortinet's threat intelligence and research division, and their servers provide real-time updates for security ratings, web filtering, antivirus definitions, and other services.

Most critical of them is Web Filter rating query - if your Fortigate cannot get answer what category the web site belongs to, access to this web site will be blocked by default. It means that if for any reason Fortigate cannot reach Fortiguard servers and it has security rules with Web Filtering by Category configured - those rules will BLOCK users access to ANY website, not just malicious ones.

First, check status of license/subscription and FortiGuard connection status in System -> FortiGuard - the Web Filtering status should be in green. This checks subscription license status, but not always detects connection to the FortiGuard status. If you see it red, it is most probably a license/subscription issue to be checked with Fortinet TAC, as subscription checks are done once in a while and are cached. To check actual connectivity to the FortiGuard servers - on the same page, under Filtering subsection, there is Test Connectivity button to push. It should return status as Up/green. Also pay attention to the widget on the same page in the right bottom corner FortiGuard Filter Rating Servers, it shows real time stats and IP addresses of the servers the Fortigate is trying to reach. If timings are unusually high and in red, there could be network connectivity problem, we will look at next.

Even better check is to run ping exe ping to all the hostnames above to see if the Fortigate can resolve AND can reach them. The most important of them being service.fortiguard.net.

Here:

Status - shows if Web Filtering as a service is enabled.

Protocol - via what protocol this Fortigate is trying to reach FortiGuard servers (more on this below).

Anycast - whether this Fortigate is trying to reach Anycast servers of FortiGuard (more on this below).

Server List - actual list of FortiGuard servers that this Fortigate was/is trying to reach. Here most important is status legend:

- F: failed, bad - Fortigate tried few times to reach this server to no avail. Note that it is bad only if ALL servers in the list have this status. It is OK if only few of the servers are unreachable.

- D: this server was successfully resolved from FQDN to its IP address, but it does not indicate its reachability yet.

- I: server to which Fortigate tries to initiate connection, most frequently goes with D,it does not indicate if a server is working or not yet.

- T: server was found, it answered, and is now being "timed", i.e. its answer time/RTT is being measured.

- TZ: Time Zone, while not a status indicator, Fortigate tries and prefers servers with the least time zone difference in hope of geographic proximity. Therefore, it is quite important to set correctly the time zone for your Fortigate.

Fortigate communicates for its functions with just one server at a time - the one on top of the list. The rest of the servers are being constantly monitored and their RTT, and packet loss measured. If the top-list server fails, it will be replaced with the next best one and so on. We do not have capability to influence this server list manually.

So if all servers in the list have F(ailed), what do we do next?. This may mean either all Fortiguard servers at the Fortinet side are down (less likely), or that this Fortigate has the problem of reaching them at the network level.

Fortigate can use several ports to talk to Fortiguard servers (or Fortiguard Distribution Network as they call it) - 53, 8888, 443, the default being 8888. The port 53 is a well known DNS protocol/port, only that Fortigate uses proprietary UDP/53 obfuscated/encrypted protocol to query the servers, and for this reason some IPS/anti-DDoS/etc protections on the way from Fortigate to FortiGuard may mark such traffic as malicious and drop it. You can check if it is the case by going to System -> FortiGuard -> Filtering and change (if set so) from port 53 to port 8888. On newer FortiOS versions (6.4 and up) they moved this to CLI only: config sys fortiguard then set port 538888443. So, as first debug measure it is recommended to try all possible ports and see if status of connection to the FortiGuard servers changes. Note about protocol I mentioned before - in 6.4 and newer they added option to force the communication to FortiGuard servers to be a valid HTTPS traffic, which is most likely to pass the Internet successfully. For this you have to enable it (in addition to setting port to 443) via CLI: config sys fortiguard, then set protocol https end.

Important note if you have VDOMs enabled - all communication to the Fortiguard network is initiated from management/root VDOM only! The frequent human error I've seen - someone by mistake changes management domain to the VDOM that has no/limited access to the Internet and as a consequence, it cannot reach FortiGuard network. Very common, indeed. To verify who is the management VDOM:

Anycast servers - starting with FortiOS 6.4 the default setting to reach FortiGuard is anycast. The intention was good - to improve reachability of FortiGuard servers, but unfortunately the implementation did not live up to the expectations. More often than not it actually creates a problem in reaching the Fortinet servers. It may improve in the future, but for now my advice is to disable anycast and switch back to unicast servers. You do so in CLI:

Hi We recently upgraded from 6.0.5 to 6.0.9 and then changed the Fortiguard protocol from UDP to HTTPS. Since then we are occasionally getting block messages 'An error occurred while trying to rate the website using the webfiltering service.

I've looked at the kb article =FD33528 but am slightly confused about the sentence "This will allow users to access the web sites when a rating error occurs, and will allow the FortiGate unit to use the FortiGuard Web Filtering database that it has stored on the unit to rate the web site." Can I just confirm that it means when it can't reach fortiguard it will rate the website using the local db and will allow/block access accordingly. Just looking at the option on the FG "Allow websites when a rating error occurs" suggests that it is going to allow it whatever the rating is. Thanks for your time

Keep in mind that the "local DB" cache is only a cache of the ratings for recently visited websites. It is a very small list compared to the full FortiGuard database. With this setting enabled there is a very good chance that traffic will get through to sites that would otherwise be blocked, so enable it at your own risk.

The reason that "a rating error occurs" happens more often with HTTPS vs. UDP is that Fortinet doesn't seem to have the same capacity to handle HTTPS web ratings lookups compared to UDP. If you run "diag debug rating" when in UDP mode vs. HTTPS mode you'll see that there are far more servers available to respond to UDP ratings lookups vs. HTTPS.

I can't say for sure, but I suspect this is a capacity issue with the FortiGuard rating servers getting overloaded occasionally and the HTTPS protocol is more sensitive to delays compared to a fast, connectionless protocol like UDP. Also, the default FortiGuard setting is to use anycast, and I've found that with anycast enabled, these errors seem to happen more often.

In my experience, if you're seeing a lot of "a rating error occurs" messages in the logs, use the UDP protocol and disable anycast for your FortiGuard settings. This is all covered by this Fortinet Tech Tip:

3a8082e126