That is status field from the “Alert message control” on System Dashboard. that status indicates the critical level from FortiGate device if it has entered conserve mode.
This problem happens when shared memory goes over 80%, to exit this conserve mode you have to wait (or kill some of the processes) until the memory goes under 70%. A FortiGate goes into the conserve mode state as a self protection measure when a memory shortage appears on the system. When entering conserve mode the FortiGate activates protection measures in order to recover memory space. When enough memory is recovered, the system is leaving/exiting the conserve mode state and releases the protection measures.
Antivirus fail-open is a safeguard feature that determines the behavior of the FortiGate AntiVirus system, when it becomes overloaded with high traffic.
to mitigate this you have more type of options:
# set av-failopen { off | on-shot | pass | idledrop}
Below we will describe what all of them do:
a. Off – if the FortiGate enters conserve mode, the FortiGate will stop accepting new AV sessions, but will continue to process currently active sessions
b. One-shot – if the FortiGate enters conserve mode, all new connections will bypass the AV system, but currently sessions will continue to be processed. This is the same as the “pass” options, but it will NOT turn off once the condition causing the av-failopen has stopped
c. Idle-drop – will drop connection based on the clients that has the most opened connection
d. Pass – this is the default option
Please keep in mind that with one-shot and pass option, NO content filtering of the traffic is done. The data stream could contain malicious content.
Below are some commands to troubleshoot when the system enters conserve mode:
a. Check if the system is in Conserve Mode:
# diagnose hardware sysinfo shm
SHM counter: 67
SHM allocated: 1556480
SHM total: 101220352
conservemode: 0 [conservemode 0 means not in conserve mode, 1 means on conserve mode, 2 means on kernel conserve mode]
shm last entered: n/a
system last entered: n/a
SHM FS total: 106827776
SHM FS free: 105205760
SHM FS avail: 105205760
SHM FS alloc: 1622016
b. Check if there any errors on the interfaces:
#diagnose hardware deviceinfo nic
So, If this problem occurs somehow we need to reduce shared memory usage on Fortigate, some optimization which I have try to improve performance on this box is:
Session timer optimizations
# config system global
set tcp-halfclose-timer 30 [ default 120 s ]
set tcp-halfopen-timer 30 [ default 60 s ]
set tcp-timewait-timer 0 [ default 120 s ]
set udp-idle-timer 60 [ default 120 s ]
end
# config system session-ttl
set default 300 [ default 300 ]
config port
edit 0
set protocol 17
set timeout 10
set end-port 53
set start-port 53
end
end
Reduce the FortiGuard services for the cache
# config system fortiguard
set webfilter-cache-ttl 500 [ default 3600 ]
set antispam-cache-ttl 500 [ default 1800 ]
end
DNS cache optimization
# config system dns
set dns-cache-limit 300 [ default: 5000 ]
end
Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering)
on FortiOS 5.0: System > Config > Features [ enable/disable ]
Turn off all non mandatory features such as Logging, archiving, data leak prevention, IPS
Display CPU/Memory usage:
# get system performance top <delay> <max_lines>
or
# diag sys top <delay> <max_lines>
And to kill process:
# diagnose sys kill 9
Restart any applications:
# diagnose test application <application> <option>
Restart IPS engine:
# diagnose test application ipsengine 99
Turn off DHCP-server services
# config sys dhcp server
delete (reference number for dhcp-server)
end
Reduce the maximum file size for antivirus scanning
On FortiOS 5.0: Go to Policy > Proxy Options > Common Options > Change: Amount (bytes)
On FortiOS 4.0: Go to Firewall > Policy > Protocol Options > reduce the file size threshold
This problem happens when shared memory goes over 80%, to exit this conserve mode you have to wait (or kill some of the processes) until the memory goes under 70%. A FortiGate goes into the conserve mode state as a self protection measure when a memory shortage appears on the system. When entering conserve mode the FortiGate activates protection measures in order to recover memory space. When enough memory is recovered, the system is leaving/exiting the conserve mode state and releases the protection measures.
Antivirus fail-open is a safeguard feature that determines the behavior of the FortiGate AntiVirus system, when it becomes overloaded with high traffic.
to mitigate this you have more type of options:
# set av-failopen { off | on-shot | pass | idledrop}
Below we will describe what all of them do:
a. Off – if the FortiGate enters conserve mode, the FortiGate will stop accepting new AV sessions, but will continue to process currently active sessions
b. One-shot – if the FortiGate enters conserve mode, all new connections will bypass the AV system, but currently sessions will continue to be processed. This is the same as the “pass” options, but it will NOT turn off once the condition causing the av-failopen has stopped
c. Idle-drop – will drop connection based on the clients that has the most opened connection
d. Pass – this is the default option
Please keep in mind that with one-shot and pass option, NO content filtering of the traffic is done. The data stream could contain malicious content.
Below are some commands to troubleshoot when the system enters conserve mode:
a. Check if the system is in Conserve Mode:
# diagnose hardware sysinfo shm
SHM counter: 67
SHM allocated: 1556480
SHM total: 101220352
conservemode: 0 [conservemode 0 means not in conserve mode, 1 means on conserve mode, 2 means on kernel conserve mode]
shm last entered: n/a
system last entered: n/a
SHM FS total: 106827776
SHM FS free: 105205760
SHM FS avail: 105205760
SHM FS alloc: 1622016
b. Check if there any errors on the interfaces:
#diagnose hardware deviceinfo nic
So, If this problem occurs somehow we need to reduce shared memory usage on Fortigate, some optimization which I have try to improve performance on this box is:
Session timer optimizations
# config system global
set tcp-halfclose-timer 30 [ default 120 s ]
set tcp-halfopen-timer 30 [ default 60 s ]
set tcp-timewait-timer 0 [ default 120 s ]
set udp-idle-timer 60 [ default 120 s ]
end
# config system session-ttl
set default 300 [ default 300 ]
config port
edit 0
set protocol 17
set timeout 10
set end-port 53
set start-port 53
end
end
Reduce the FortiGuard services for the cache
# config system fortiguard
set webfilter-cache-ttl 500 [ default 3600 ]
set antispam-cache-ttl 500 [ default 1800 ]
end
DNS cache optimization
# config system dns
set dns-cache-limit 300 [ default: 5000 ]
end
Reduce memory caching in some features (Explicit proxy, FortiGuard Antispam/Webfiltering)
on FortiOS 5.0: System > Config > Features [ enable/disable ]
Turn off all non mandatory features such as Logging, archiving, data leak prevention, IPS
Display CPU/Memory usage:
# get system performance top <delay> <max_lines>
or
# diag sys top <delay> <max_lines>
And to kill process:
# diagnose sys kill 9
Restart any applications:
# diagnose test application <application> <option>
Restart IPS engine:
# diagnose test application ipsengine 99
Turn off DHCP-server services
# config sys dhcp server
delete (reference number for dhcp-server)
end
Reduce the maximum file size for antivirus scanning
On FortiOS 5.0: Go to Policy > Proxy Options > Common Options > Change: Amount (bytes)
On FortiOS 4.0: Go to Firewall > Policy > Protocol Options > reduce the file size threshold
