General Troubleshooting
This article describes troubleshooting practices that are applicable to various problem scenarios.
Checking Microservice Status
You can verify the microservice statuses on the status page, available at (replace https://privx.example.com with the address of your PrivX server:
https://privx.example.com/privx/status
Restarting PrivX
Some issues may be fixed by restarting the PrivX services. You can do this by running the following command on your PrivX servers:
# systemctl restart privxIf microservices report 502 errors, check the database connectivity. If running local DB server, make sure the pg_hba.conf on the PostgreSQL server allows PrivX servers to connect to the PrivX database with md5 authentication (PostgreSQL 9.2 - 9.6, on CentOS 7) or scram-sha-256 authentication (from PostgreSQL 10 onwards).
More information about the error may be provided by the microservice logs.
Getting Debug Logs
You can enable debug logging per microservice, as described in PrivX Log Settings.
By default, microservices log to /var/log/privx/ as follows:
| MICROSERVICE | LOG LOCATION |
|---|---|
| Authorizer | /var/log/privx/authorizer.log |
| Connection Manager | /var/log/privx/connectionmanager.log |
| Hoststore | /var/log/privx/hoststore.log |
| Keyvault | /var/log/privx/keyvault.log |
| Monitor | /var/log/privx/monitorservice.log |
| OAuth2 | /var/log/privx/auth.log |
| RDP Bastion | /var/log/privx/rdpmitm.log |
| RDP Proxy | /var/log/privx/rdpproxy.log |
| RDP-Trail Encoder | /var/log/privx/redemption.log |
| Role Store | /var/log/privx/rolestore.log |
| SSH Bastion | /var/log/privx/sshmitm.log |
| SSH Proxy | /var/log/privx/sshproxy.log |
| Trail Indexer | /var/log/privx/trail-indexer.log |
| Userstore | /var/log/privx/userstore.log |
| Watchdog | /var/log/privx/watchdog.log |
| Workflow Engine | /var/log/privx/workflowengine.log |
The logs are also visible on /var/log/messages.
Problems Related to Authentication
More information about authentication issues may be found from syslog messages.
Since PrivX uses short-term certificates for user authentication, deviations in system times (minutes or more) will easily result in failing authentications. It is extremely important to synchronize the clocks between all the involved systems:
PrivX servers
Hosts serving as connection endpoints.
We strongly recommend using authenticated NTP clock synchronization on all the involved systems.
Contacting Support
You can request technical support and let us know about PrivX issues on the SSH Communications Security support site.
When submitting a support ticket, please include following details:
- Detailed description about the error:
- Causes: User actions and steps that caused the error, screenshot of the error, and/or details about the error message.
- Occurrence: Is the error reproducible? Has it worked before?
- Impact: Does it happen to a specific entity (user, connection, hosts, etc), or to multiple entities?
Related configurations in PrivX admin UI. For example if the error is regarding a user cannot connect to a host, take some screenshots of the host configuration, screenshots of user's details and roles
Troubleshooting data
To generate troubleshooting data, run the following on your PrivX server:
# /opt/privx/scripts/troubleshoot.sh infoAnd attach the generated troubleshooting-data package to the support ticket. By default the package is generated to the current working directory, with the name:
PrivX-troubleshoot-data-<host>-<timestamp>.tar.gz
If PrivX service is completed or partially down, please run the troubleshooting script and collect the data when the problem is happening.
🚧 Caution
Troubleshooting data may contain security critical data, and must be distributed with appropriate care.
By default, the troubleshooting script uses /tmp as the working folder. If /tmp is not accessible or has limited usable space, you can specify a different working folder using the --t or --tempdir option.
To reduce the size of the troubleshooting data, you can use the -s or --log-since option to specify the number of days of logs to collect.
For additional usage instructions related to the troublehsooting script, you may run:
# /opt/privx/scripts/troubleshoot.sh -h