HomeDocumentationAPI Reference
Log In
These docs are for v24. Click to read the latest docs for v33.

High-Availability Deployment

Suggest Edits

PrivX high availability deployment can be done for on-prem or cloud environments. For AWS based deployments, see CDK based installation scripts. at [https://github.com/SSHcom/privx-on-aws] (https://github.com/SSHcom/privx-on-aws)

Load Balancer

For high availability installations, we recommend a load balancer in front of PrivX server instances. The recommendation is to use HTTPS load balancer, which terminates the TLS certificate and forwards traffic to PrivX instances. This way only the load balancer requires an official TLS certificate, which simplifies the installation. PrivX web UI and additional PrivX components (Extender, Carrier, Web-Proxy) require using sticky sessions/affinity cookies.

Officially tested cloud provider load balancers include AWS, Azure and Google load balancers.
PrivX components have been preconfigured to work with AWS, Google, Azure and Nginx. Other load balancers might require additional settings.

For on-prem installations, most HTTPS load balancers (Nginx, HAProxy, Netscaler etc) will work, as long as they support sticky session cookies. For Extender, Carrier and Web Proxy components, Round-robin routing algorithm need to be used instead of least-connections. If you don't need PrivX components, any load balancing algorithm will be acceptable.

For native SSH Bastion and native RDP client traffic a TCP load balancer or DNS load balancing is required. It is also possible to use two load balancers for the same servers, HTTPS load balancer for web UI and PrivX components traffic and TLS network load balancer for native clients.
If using optional client certificate authentication for web UI, a TLS load balancer need to be used for that as well.

For health checks and autoscaling, it's recommended to use any microservice status endpoint to check the availability of your PrivX instance. For example, https://<privx_lb_endpoint>/auth/api/v1/status
Monitoring root web page monitors the status of Nginx server instead of PrivX, which is not a recommended solution.

Example Nginx load balancer configuration

Load balancing PrivX components

For load balancing PrivX components, both Extender and web components (Carrier+Web Proxy) can be grouped using Routing Prefix setting in PrivX UI. For example, setting the same Routing Prefix variable for two or more different Extenders will group them logically, allowing PrivX to load balance between the components. Load balancing algorithm (least connections or round-robin) can be changed in PrivX configuration.
Note that this feature still requires using separate configuration files for each component. Sharing config files or credentials between components is not currently supported.

Standard HA installation

1737

  1. A load balancer directs traffic to a specific PrivX application server. The load balancer keeps track of PrivX application server statuses and removes non-working ones from the pool. The load balancing can be based on either source address or sticky session. Sticky session/session affinity cookie configuration for load balancer is recommended. Sticky load balancer cookies with round-robin routing algorithm are required, if using PrivX-Extender, PrivX-Carrier and PrivX-Web-Proxy components and there's more than one PrivX node.

  2. A PrivX application server consists of an Nginx reverse proxy and a number of PrivX microservices. Nginx reverse proxy also serves the PrivX HTML5 UI static resources for the requesting clients. The PrivX microservices offer REST APIs over HTTPS. The PrivX application servers store all persistent data in a database - once a PrivX application server has been configured, it is just a matter of taking a snapshot of the server and deploying a new instance of the snapshot to add application nodes.

  3. (Optional) The PrivX microservices may optionally use Redis in-memory cache (instead of the PostgreSQL database) to sync state between themselves. We only recommend Redis in environments where PostgreSQL database performance is limited.

    If opting to use Redis, it should be configured in clustered mode. PrivX will require write access to Redis master node, so it should not be connected directly to replica nodes.

  4. The PrivX microservices persist data to PostgreSQL. PostgreSQL should also be highly available, configured in master<>slave mode or as a cluster.

  5. PrivX also supports various Hardware Security Modules (optional) for storing keys to separate HSM.

  6. The PostgreSQL database should be configured to clustered mode. If the database master dies, a
    DNS failover PrivX application servers can be configured to contact the replica via configuration change or a DNS change.

  7. The trail storage (by default /var/privx/audit) should be mounted on NFS or a secure NAS device and shared between PrivX instances. (Optional: It is also possible to use shared storage for /opt/privx -folder to before installing PrivX to make updates easier in the future).

PrivX-Server HA Setup

📘

Note

These high-availability (HA) installation instructions apply to non-cloud environment. For cloud HA deployment, please see Deploying PrivX to Amazon Web Services

The high-level steps for setting up PrivX in HA configuration involves:

  1. Setting up one initial PrivX node.
  2. Setting up additional PrivX servers by duplicating the initial PrivX node.

Initial PrivX Node

  1. Install PostgreSQL master and slave nodes and configure a replication model which best suits your environment (vendor instructions at https://www.postgresql.org/docs/11/different-replication-solutions.html)

  2. If using Redis: Install Redis and configure Redis cluster (vendor instructions at https://redis.io/topics/cluster-tutorial).

  3. Install PrivX from the PrivX repository:

    sudo yum install PrivX

    Or from RPM package:

    sudo yum install PrivX-*.x86_64.rpm

  4. Run post-install configuration with:

    sudo /opt/privx/scripts/postinstall.sh

    During postinstall, instruct PrivX to connect to the external database and Redis.

  5. Verify that the installation works by connecting to the initial PrivX node via browser.

The initial PrivX node is ready.

Additional PrivX Nodes

If using physical servers

Create a backup of the initial PrivX node, then use the backup to create additional nodes.

  1. On the initial PrivX node, create a backup by running:

    sudo /opt/privx/scripts/backup.sh

    This creates a backup to /var/backups/privx/hostname_yyyy-mm-dd-hhmm

  2. On all your your additional nodes, set up PrivX using the backup:

    1. Transfer the backup directory to the node.

    2. Install PrivX without running postinstall. If installing from repository:

      sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX

      If installing from RPM package:

      sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX-*.x86_64.rpm

    3. Duplicate PrivX setup from backup by running:

      sudo /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm`

    4. Finalize setup by running postinstall:

      sudo /opt/privx/scripts/postinstall.sh

    Repeat these steps on all additional nodes.

If using virtual machines

After installing PrivX to the first node, you can clone the node and use the cloned instance to launch additional PrivX nodes.

📘

Note

To ensure that all machines have unique machine IDs, you may need to regenerate it on cloned machines.

Extender HA Setup

Extenders are grouped into HA clusters by Routing prefix: Any Extenders with similar Routing prefixes belong in the same HA cluster. When an Extender is down, connections are established via other available Extenders from the HA cluster.

To set up Extenders in a HA configuration:

  • Set up two or more Extenders. All Extenders in the HA cluster must have the same Routing prefix. For more information about setting up Extenders, see PrivX Extender Setup.

Carrier and Web-Proxy HA Setup

PrivX Carriers and PrivX Web Proxies are grouped into HA clusters by Routing prefix. Any Carriers and Web Proxies pairs with similar Routing prefixes belong in the same HA cluster. When a Carrier/Web-Proxy pair is down, connections are established via other available Carrier/Web-Proxy pairs from the HA cluster.

To set up Carriers and Web Proxies in HA configuration:

  • Set up multiple Carrier/Web-Proxy pairs. All Carriers/Web Proxies in the HA cluster must have the same Routing prefix. see PrivX Carrier and Web Proxy Setup.

    📘

    Note

    Just like Extenders, each Carrier/Web-Proxy pair requires their own credentials and configuration file. You will need to manually ensure that all configurations for the same HA cluster have the same Routing prefix.

High-Availability Upgrade

📘

Note

These high-availability (HA) upgrade instructions apply to non-cloud environment, but can be in some cases used for upgrading cloud HA setup as well.

Ensure the PrivX database has enough free space before upgrade: Migrations during upgrade may temporarily triple the database size. If necessary, you may reduce the database size before upgrade with Data Retention settings.

We strongly recommend backing up the PrivX database before upgrade, to allow restoring the previous version in case of a failed upgrade.

  1. Choose a PrivX server for performing upgrade on. This can be any PrivX server in the deployment. In these instructions we call this the primary node. Other PrivX servers are called secondary nodes.

  2. On all secondary nodes, stop the PrivX service with:

    sudo systemctl stop privx

    This prevents secondary nodes from writing conflicting data to the database while the primary node is being upgraded.

  3. On the primary node, upgrade PrivX:

    1. Upgrade PrivX from the repository:

      sudo yum install PrivX

      Or from RPM package:

      sudo yum install PrivX-*.x86_64.rpm

      The command upgrades the PrivX database, and the PrivX software on the primary node. Allow the command some time to finish.

    2. Verify that the node works before proceeding.

    📘

    Note

    On upgrade, postinstall is run automatically. This also automatically restarts PrivX services on the node.

  4. Upgrade the PrivX software on all secondary nodes:

    • If your PrivX servers use individual installation directories (default), upgrade and configure the nodes with sudo yum install PrivX or sudo yum install PrivX-*.x86_64.rpm.

    • If your PrivX servers use a shared NFS-mounted installation directory, you only need to configure the nodes by running sudo /opt/privx/postinstall.sh on them.

    Secondary nodes can be upgraded in parallel

Backup and Restore

PrivX automatically creates daily full backups (certificates and configuration files) and stores them in /var/backups/privx. Ideally, this directory should be mounted on a secure NAS solution or periodically backed up to a secure location.

📘

Note

PrivX does not automatically backup Postgres databases on external database. Backing up the external Postgres database is outside of the scope of this document. You should always take a backup of your database data before upgrading PrivX version.

To restore a PrivX HA deployment from backup:

  1. On all PrivX nodes, stop PrivX services with:

    sudo systemctl stop privx

    This is to prevent database changes during restore.

  2. Restore the PrivX database.

  3. On each PrivX node, restore your PrivX backup:

    1. Copy the PrivX-backup directory to the node.

    2. Install PrivX without running postinstall. If installing from repository:

      sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX

      If installing from RPM package:

      sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX-*.x86_64.rpm

      📘

      Note

      PrivX version to be installed here must match the PrivX version used for the backups you're trying to restore.

    3. Restore PrivX setup from backup by running:

      sudo /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm_privx-version`

    4. Finalize setup by running postinstall:

      sudo /opt/privx/scripts/postinstall.sh

    Repeat on each PrivX node.

Updated over 2 years ago