High-Availability Deployment

PrivX high availability deployment can be done for on-prem or cloud environments. For AWS based deployments, see CDK based installation scripts. at https://github.com/SSHcom/privx-on-aws

Load Balancer

For high availability installations, we recommend a load balancer in front of PrivX server instances. The recommendation is to use HTTPS load balancer, which terminates the TLS certificate and forwards traffic to PrivX instances. This way only the load balancer requires an official TLS certificate, which simplifies the installation. Using PrivX components (Extender, Carrier, Web-Proxy) with PrivX HA setup require using HTTPS load balancer with sticky sessions/affinity cookies.

Officially tested cloud provider load balancers include AWS, Azure and Google load balancers.
PrivX components have been preconfigured to work with AWS, Google, Azure, Kubernetes ingress and Nginx. Other load balancers might require additional settings.

For on-premises installations, most HTTPS load balancers (Nginx Plus, HAProxy, Netscaler etc) will work, as long as they support sticky-session cookies and preferrably active health checks.

Example Nginx Load-Balancer Configuration
Example HAProxy Load-Balancer Configuration

For Extender, Carrier and Web-Proxy components, Round-Robin routing algorithm needs to be used instead of least-connections. If you don't need any of these PrivX components, any load balancing algorithm or network load balancer will be acceptable.

For native SSH Bastion and native RDP client traffic a TCP load balancer or DNS load balancing is required. It is also possible to use two load balancers for the same servers, HTTPS load balancer for web UI and PrivX components traffic and TLS network load balancer for native clients.
If using optional client certificate authentication for web UI, a TLS load balancer need to be used for that as well.

For health checks and autoscaling, we recommended using monitor-service instance status endpoint to check the availability of your PrivX instance. For example, https://<privx_lb_endpoint>/monitor-service/api/v1/instance/status. It is also possible to use other microservice status endpoints (https://<privx_lb_endpoint>/<servicename>/api/v1/status) for this purpose.

Load Balancing PrivX Components

For load balancing PrivX components, both Extender and web components (Carrier+Web Proxy) can be grouped using Routing Prefix setting in PrivX UI. For example, setting the same Routing Prefix variable for two or more different Extenders will group them logically, allowing PrivX to load balance between the components. Load balancing algorithm (least connections or round-robin) can be changed in PrivX configuration.
Note that this feature still requires using separate configuration files for each component. Sharing config files or credentials between components is not currently supported.

Standard HA installation

A load balancer directs traffic to a specific PrivX application server. The load balancer keeps track of PrivX application server statuses and removes non-working ones from the pool. The load balancing can be based on either source address or sticky session. Sticky session/session affinity cookie configuration for load balancer is recommended. Sticky load balancer cookies with round-robin routing algorithm are required, if using PrivX-Extender, PrivX-Carrier and PrivX-Web-Proxy components and there's more than one PrivX node.
A PrivX application server consists of an Nginx reverse proxy and a number of PrivX microservices. Nginx reverse proxy also serves the PrivX HTML5 UI static resources for the requesting clients. The PrivX microservices offer REST APIs over HTTPS. The PrivX application servers store all persistent data in a database - once a PrivX application server has been configured, it is just a matter of taking a snapshot of the server and deploying a new instance of the snapshot to add application nodes.
(Deprecated) The PrivX microservices may optionally use Redis in-memory cache (instead of the PostgreSQL database) to sync state between themselves. We only recommend Redis in environments where PostgreSQL database performance is limited.
If opting to use Redis, it should be configured in clustered mode. PrivX will require write access to Redis master node, so it should not be connected directly to replica nodes.
The PrivX microservices persist data to PostgreSQL.
PostgreSQL should also be highly available, configured in master<>slave mode or as a cluster.
PrivX also supports various Hardware Security Modules (optional) for storing keys to separate HSM.
The PostgreSQL database should be configured to clustered mode. If the database master dies, a
DNS failover PrivX application servers can be configured to contact the replica via configuration change or a DNS change.
The trail storage (by default /var/privx/audit) should be mounted on NFS or a secure NAS device and shared between PrivX instances. To avoid file permission issue, make sure to use the same privx user and group IDs in all PrivX application servers.

PrivX-Server HA Setup

These high-availability (HA) installation instructions apply to non-cloud environment. For cloud HA deployment, please see Deploying PrivX to Amazon Web Services

The high-level steps for setting up PrivX in HA configuration involves:

Setting up one initial PrivX node.
Setting up additional PrivX servers by duplicating the initial PrivX node.

Initial PrivX Node

Install PostgreSQL master and slave nodes and configure a replication model which best suits your environment (vendor instructions at https://www.postgresql.org/docs/11/different-replication-solutions.html)
If using Redis: Install Redis and configure Redis cluster (vendor instructions at https://redis.io/topics/cluster-tutorial).
Install PrivX from the PrivX repository:
```
sudo yum install PrivX
```
Or from RPM package:
```
sudo yum install PrivX-*.x86_64.rpm
```
Run post-install configuration with:
```
sudo /opt/privx/scripts/postinstall.sh
```
During postinstall, instruct PrivX to connect to the external database and Redis.
Verify that the installation works by connecting to the initial PrivX node via browser.

The initial PrivX node is ready.

Additional PrivX Nodes

If using physical servers

Create a backup of the initial PrivX node, then use the backup to create additional nodes.

On the initial PrivX node, create a backup by running:
```
sudo /opt/privx/scripts/backup.sh
```
This creates a backup to /var/backups/privx/hostname_yyyy-mm-dd-hhmm
On all your your additional nodes, set up PrivX using the backup:
1. Transfer the backup directory to the node.
2. Install PrivX without running postinstall. If installing from repository:
```
sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX
```
  If installing from RPM package:
```
sudo export SKIP_POSTINSTALL=1
sudo yum install PrivX-*.x86_64.rpm
```
3. Duplicate PrivX setup from backup by running:
```
sudo /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm` 
```
4. Finalize setup by running postinstall:
```
sudo /opt/privx/scripts/postinstall.sh
```
Repeat these steps on all additional nodes.

If using virtual machines

After installing PrivX to the first node, you can clone the node and use the cloned instance to launch additional PrivX nodes.

To ensure that all machines have unique machine IDs, you may need to regenerate it on cloned machines.

Extender HA Setup

Extenders are grouped into HA clusters by Routing prefix: Any Extenders with similar Routing prefixes belong in the same HA cluster. When an Extender is down, connections are established via other available Extenders from the HA cluster.

To set up Extenders in a HA configuration:

Set up two or more Extenders. All Extenders in the HA cluster must have the same Routing prefix. For more information about setting up Extenders, see PrivX Extender Setup.

Carrier and Web-Proxy HA Setup

PrivX Carriers and PrivX Web Proxies are grouped into HA clusters by Routing prefix. Any Carriers and Web Proxies pairs with similar Routing prefixes belong in the same HA cluster. When a Carrier/Web-Proxy pair is down, connections are established via other available Carrier/Web-Proxy pairs from the HA cluster.

To set up Carriers and Web Proxies in HA configuration:

Set up multiple Carrier/Web-Proxy pairs. All Carriers/Web Proxies in the HA cluster must have the same Routing prefix. see PrivX Carrier and Web Proxy Setup.

Just like Extenders, each Carrier/Web-Proxy pair requires their own credentials and configuration file. You will need to manually ensure that all configurations for the same HA cluster have the same Routing prefix.

High-Availability Upgrade

These high-availability (HA) upgrade instructions apply to non-cloud environment, but can be in some cases used for upgrading cloud HA setup as well.

Ensure the PrivX database has enough free space before upgrade: Migrations during upgrade may temporarily triple the database size. If necessary, you may reduce the database size before upgrade with Data Retention settings.

We strongly recommend backing up the PrivX database before upgrade, to allow restoring the previous version in case of a failed upgrade.

Choose a PrivX server for performing upgrade on. This can be any PrivX server in the deployment. In these instructions we call this the primary node. Other PrivX servers are called secondary nodes.
On all secondary nodes, stop the PrivX service with:
```
sudo systemctl stop privx
```
This prevents secondary nodes from writing conflicting data to the database while the primary node is being upgraded.
On the primary node, upgrade PrivX:
1. Upgrade PrivX from the repository:
```
sudo yum install PrivX
```
  Or from RPM package:
```
sudo yum install PrivX-*.x86_64.rpm
```
  The command upgrades the PrivX database, and the PrivX software on the primary node. Allow the command some time to finish.
2. Verify that the node works before proceeding.

On upgrade, postinstall is run automatically. This also automatically restarts PrivX services on the node.

Upgrade the PrivX software on all secondary nodes:
- If your PrivX servers use individual installation directories (default), upgrade and configure the nodes with sudo yum install PrivX or sudo yum install PrivX-*.x86_64.rpm.
- If your PrivX servers use a shared NFS-mounted installation directory, you only need to configure the nodes by running sudo /opt/privx/scripts/postinstall.sh on them.
Secondary nodes can be upgraded in parallel