Collecting Glaber diagnostics

Server

1. Server crashes

When a server defect leads to a crash, the server leaves detailed information in the logs about the code location where the crash occurred.

Usually the last 2000 lines contain everything needed. You can obtain such a log with:

tail -n 2000 /var/log/glaber/glaber_server.log

Attach this information, as well as register and pointer contents, to the issue / send it to the developers.

(On older installs the log path may still be /var/log/zabbix/zabbix_server.log.)

2. Metric collection logic, polling, and trigger issues

When metric processing results differ from what you expect, it is useful to collect logs for the processing steps of a single metric. Logging one metric is limited to that metric and can safely be done on the fly even on very heavily loaded configurations.

In the UI go to Administration → Debug and set the problematic metric ID in Debug Item, or the trigger ID in Debug Trigger. Use the object ID (you can copy it from the browser URL while editing the metric or trigger).

No server restart is required; the server immediately starts writing debug output for only that metric to the log file (typically /var/log/glaber/glaber_server.log).

Collect the journal for a period equal to 3–5 metric collection intervals. Trigger debug works the same way. Provide developers with the debug lines from the log:

grep debug /var/log/glaber/glaber_server.log

After a server restart, selective debug stops. If you need information from server start, enable item/trigger debug in glaber_server.conf (or zabbix_server.conf on older installs):

DebugItem=234525
DebugTrigger=3456453

Selective debug requires the Custom Debug module. See Extending.

3. Web interface and API

On errors, the failure point and call stack are in the web server error log. Attach that to the issue. You can also enable UI debug mode and use the Debug button to get an API activity log and SQL query statistics.

4. Additional sources of debug information

The server periodically dumps the value cache. The dump is stored in gzip-compressed JSONL format.

With zcat and grep by metric ID you can select the metric values present in the cache at dump time.

5. Diagnosing low server performance

Collect system-call wait time with strace (wall-clock time in syscalls):

timeout 60 strace -w -c -p <pid>

Collect CPU time spent in syscalls:

timeout 60 strace -c -p <pid>

Also useful:

top -c

For stage-level bottlenecks (history syncer, SQL, locks, value cache), see glb_prof profiling.