General Troubleshooting

This article describes troubleshooting practices that are applicable to various problem scenarios.

Checking Microservice Status

You can verify the microservice statuses on the status page, available at (replace https://privx.example.com with the address of your PrivX server:

https://privx.example.com/privx/status

Restarting PrivX

Some issues may be fixed by restarting the PrivX services. You can do this by running the following command on your PrivX servers:

# systemctl restart privx

If microservices report 502 errors, check the database connectivity. If running local DB server, make sure the pg_hba.conf on the PostgreSQL server allows PrivX servers to connect to the PrivX database with md5 authentication (PostgreSQL 9.2 - 9.6, on CentOS 7) or scram-sha-256 authentication (from PostgreSQL 10 onwards).

More information about the error may be provided by the microservice logs.

Getting Debug Logs

You can enable debug logging per microservice, as described in PrivX Log Settings.

By default, microservices log to /var/log/privx/ as follows:

MICROSERVICE	LOG LOCATION
Authorizer	/var/log/privx/authorizer.log
Connection Manager	/var/log/privx/connectionmanager.log
Hoststore	/var/log/privx/hoststore.log
Keyvault	/var/log/privx/keyvault.log
Monitor	/var/log/privx/monitorservice.log
OAuth2	/var/log/privx/auth.log
RDP Bastion	/var/log/privx/rdpmitm.log
RDP Proxy	/var/log/privx/rdpproxy.log
RDP-Trail Encoder	/var/log/privx/redemption.log
Role Store	/var/log/privx/rolestore.log
SSH Bastion	/var/log/privx/sshmitm.log
SSH Proxy	/var/log/privx/sshproxy.log
Trail Indexer	/var/log/privx/trail-indexer.log
Userstore	/var/log/privx/userstore.log
Watchdog	/var/log/privx/watchdog.log
Workflow Engine	/var/log/privx/workflowengine.log

The logs are also visible on /var/log/messages.

Problems Related to Authentication

More information about authentication issues may be found from syslog messages.

Since PrivX uses short-term certificates for user authentication, deviations in system times (minutes or more) will easily result in failing authentications. It is extremely important to synchronize the clocks between all the involved systems:

PrivX servers
Hosts serving as connection endpoints.

We strongly recommend using authenticated NTP clock synchronization on all the involved systems.

Contacting Support

You can request technical support and let us know about PrivX issues on the SSH Communications Security support site.

When submitting a support ticket, please include following details:

Detailed description about the error:

Causes: User actions and steps that caused the error, screenshot of the error, and/or details about the error message.
Occurrence: Is the error reproducible? Has it worked before?
Impact: Does it happen to a specific entity (user, connection, hosts, etc), or to multiple entities?

Related configurations in PrivX admin UI. For example if the error is regarding a user cannot connect to a host, take some screenshots of the host configuration, screenshots of user's details and roles
Troubleshooting data

To generate troubleshooting data, run the following on your PrivX server:

# /opt/privx/scripts/troubleshoot.sh info

And attach the generated troubleshooting-data package to the support ticket. By default the package is generated to the current working directory, with the name:

Privx-troubleshoot-data-<host>-<timestamp>.tar.gz

If PrivX service is completed or partially down, please run the troubleshooting script and collect the data when the problem is happening.

🚧 Caution

Troubleshooting data may contain security critical data, and must be distributed with appropriate care.

By default, the troubleshooting script uses /tmp as the working folder. If /tmp is not accessible or has limited usable space, you can specify a different working folder using the --t or --tempdir option.

To reduce the size of the troubleshooting data, you can use the -s or --log-since option to specify the number of days of logs to collect.

For additional usage instructions related to the troublehsooting script, you may run:

# /opt/privx/scripts/troubleshoot.sh -h