Troubleshooting
Increase Log Level
Set the log level to DEBUG to identify the activity that is occurring at the same time as the problem.
# set the logLevel to "debug" in /var/lib/ziti/config.json
sudo -u ziti ziti-edge-tunnel set_log_level --loglevel DEBUG
The tunneler obeys the value of logLevel in /var/lib/ziti/config.json. The initial value may be set with run
--verbose 4, but setting this option on subsequent runs has no effect on log level.
Capture the Current Process Log
journalctl _SYSTEMD_INVOCATION_ID=$(systemctl show -p InvocationID --value ziti-edge-tunnel.service) -l \
| tee /tmp/ziti-edge-tunnel.log
Systemd Service Won't Start or Keeps Restarting
Reload the systemd service unit definitions to rule out a stale definition.
sudo systemctl daemon-reload
Inspect the service unit.
sudo systemctl cat ziti-edge-tunnel.service
Check the service status for an error message.
sudo systemctl status ziti-edge-tunnel.service
Monitor the service logs.
sudo journalctl -u ziti-edge-tunnel.service
Intercepting or Hosting Not Working
Inspect the identity and router info for a running tunneler process. This creates a file named like {{identity name}}.ziti
for each loaded identity. Each file summarizes the available services and router connections for the identity.
sudo -u ziti ziti-edge-tunnel dump -p /tmp/ziti-dump-dir/
Find tunneler's nameserver IP.
$ resolvectl --interface=ziti0 dns
Link 19 (tun0): 100.64.0.2
Query the Ziti nameserver to find the intercept IP address for a service.
dig +noall +answer my.ziti.service.example.com @100.64.0.2
The tunneler provides end-to-end TCP handshake. Test the service's ability to accept connections even if it does not provide a greeting or banner as shown in the OpenSSH server example below.
# wait up to 3 seconds for a TCP handshake on port 443
$ ncat -vzw3 100.64.0.3 443
Ncat: Connected to 100.64.0.3:443.
Ncat: 0 bytes sent, 0 bytes received in 0.08 seconds.
# wait up to 3 seconds for an OpenSSH server greeting on port 22
$ ncat -vw3 100.64.0.3 22
SSH-2.0-OpenSSH_7.4
Process Keeps Crashing
A crash may be caused by a segmentation fault. If saving a Corefile is enabled, Linux will create a core dump file
according to this pattern file: /proc/sys/kernel/core_pattern. Ubuntu configures this to use
Apport. Read more about core dumps.
Please raise a GitHub issue if the tunneler crashes.
Operation Not Permitted
Delete the /tmp/.ziti directory and restart the tunneler to solve this issue.
The tunneler may log a warning about an operation not permitted and failure to start the socket server if the hard-coded path /tmp/.ziti is not owned by the run-as user. This can happen if you first run the tunneler as root and then as a non-root user.
WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1686 make_socket_path() failed to set ownership of /tmp/.ziti to 1003:1003: Operation not permitted (errno=1)
WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1730 run_tunneler_loop() One or more socket servers did not properly start.
Another symptom of the same issue is this error when attempting to send a message to the IPC socket server, which is not running.
failed to connect: -111/connection refused