Warm Spare HA

At the present time, the only way to achieve an automated fail over in the event of a random service failure is with Sangoma's Advanced Recovery module license. While the Advanced Recovery module will work with AWS FreePBX and we fully support its use, the $800/year ($400/server/year) licensing cost puts it out of budgetary reach for many small customers who will already be paying for two AWS FreePBX instances. The Advanced Recovery module is NOT AT ALL required to use Warm Spare HA strategies, as manually switching to the Backup instance is as simple as swapping the Elastic IP and, in some cases, enabling Trunks. This recludes the $800/yr Commercial Module for automated fail-over and sync-back. The choice is yours and you can switch to the Advanced Recovery module any time in the future.

This guide will show you how to setup Manual Warm Spare High Availability with AWS FreePBX.

• 2 AWS FreePBX Instances, Primary and Secondary
  • The Primary Instance can be a production instance already in use, or a newly launched deployment
  • The Secondary Instance should be launched clean from the same major AMI version as the Primary Instance

• 1 Elastic IP address
  • This should be Attached to the Primary Instance when you begin these steps

• 1 EC2 Security Group to be used on both Instances
  • Needs to include access to TCP ports 22 and 80 granted to either the VPC Private Subnet CIDR (172.x.x.x) -OR- the Security Group ID (self-referencing) -OR- the Primary and Secondary Private IPs (two SG entries), allowing the Primary Instance to connect to the Secondary Instance to copy files and execute restore operations

Throughout the steps below, we will be switching back and forth between the Primary and Secondary Instances in order to complete steps via both the GUI and SSH. To save time, have the GUIs of both Primary and Secondary Instances loaded/logged into your browser and establish a single SSH connection to the Secondary Instance.

• Navigate the Primary GUI to Admin > Backup and Restore
• Access the Global Settings tab and copy the entire ssh-rsa Public Key field

• Connect to the Backup Instance via SSH and edit /root/.ssh/authorized_keys

  sudo su
nano /root/.ssh/authorized_keys
  

• On a new empty line, paste the entire ssh-rsa key you coped in Step 1 and ensure an empty new line remains at the very end of the file, then save and close the file with Ctrl+o (letter O); Enter ; Ctrl+x

• Navigate the Primary GUI to Settings > FileStore, switch to the SSH tab, and click Add SSH Server
• Populate the fields to create a new SSH FileStore connection to the Secondary Instance and Submit
  • Server Name = Backup Node
  • Hostname = (Private IPv4 Address of Backup Instance)
  • Port = 22
  • Username = root
  • Key = /home/asterisk/.ssh/id_rsa
  • Path = /var/spool/asterisk

If you wish to keep an additional copy of your most recent backup file in an S3 Bucket, setup a second FileStore connecting to S3.

• If necessary, create a set of IAM credentials with Access Key ID and Secret Access Key. YOU MUST BE CERTAIN TO SAFEGUARD THIS INFORMATION AS IT GRANTS FULL ACCESS TO YOUR AWS S3 BUCKETS AND WOULD BE VERY DANGEROUS IN THE WRONG HANDS!!!
  • https://console.aws.amazon.com/iam/home?#/users
  • On this page, you will click Add User. Enter a Username (this is for your reference only) and choose the Programmatic Access option. Then select the Attach Existing Policies Directly tab and search for the AmazonS3FullAccess permission. Once you click the Create User button on the last page, you MUST save the Access Key ID and Secret Access Key (click the 'show' link) to setup the FileStore connection. You can also download this information in csv format for your records. If this information becomes compromised in the future, you can return to this page, delete the user, create a new one, and reprogram the FileStore with the new keys.
• If necessary, create a new S3 Bucket in the same region as your Production Instance to house your backup files
  • https://s3.console.aws.amazon.com/s3/bucket/create
• Navigate the Primary GUI to Settings > FileStore, switch to the S3 tab, and click Add S3 Bucket
• Populate the fields to create a new S3 FileStore connection to your desired Bucket and Submit
  • Local Display Name = FreePBX Backups
  • Bucket Name = (YourBucketName)
  • Description = (optional)
  • AWS Region = (Region you created the Bucket in)
  • AWS Access Key = (Access Key ID you created)
  • AWS Secret = (Secret Access Key you created)
  • Path = (optional; if you wish to store within a sub-directory within the Bucket)

• Navigate the Secondary GUI to Connectivity > API, then click Add Application > Machine-to-Machine App
• Fill in Your App Name and Description, supply gql:backup in the Allowed Scopes field, then click Add Application
• You must now make note of the Token URL, GraphQL URL, Client ID, and Client Secret for use in the next step.

  YOU MUST BE CERTAIN TO SAFEGUARD THIS INFORMATION AS IT GRANTS READ/WRITE ACCESS TO YOUR FREEPBX BACKUPS AND WOULD BE VERY DANGEROUS IN THE WRONG HANDS!!!

  

• Navigate the Primary GUI to Admin > Backup and Restore, then click Add Backup

• Please note the following caveats for certain deployments:
  • SIPSTATION Customers
    • You will need a separate Trunk Group for each Instance
    • You must set Exclude Trunks = Yes
    • You must set Modules > Core > Disable Trunks on Restore = Yes
  • Registration-based Trunk Providers
    • You may need separate Trunk connections/credentials for each Instance and set Exclude Trunks = Yes
    • You may need to set Modules > Core > Disable Trunks on Restore = Yes
  • IP-based Trunk providers
    • You should always send your incoming calls to only the Elastic IP, so only the active instance is receiving your calls

• Populate the fields and set the parameters of your Warm Spare backup job
  • Backup Name = Warm Spare
  • Click the Modules button and deselect the following modules: Backup & Restore, FileStore, SIPSTATION (see caveats above!)
  • Leave the Custom Files (ASTETCDIR) Directory entry UNTOUCHED!
  • Storage Location = check SSH > Backup Node (also check S3 > FreePBX Backups, if setup in 3a*)
  • Schedule: Enabled = Yes; Scheduling = Every Hour
  • Delete After Runs = 1
  • Warm Spare - Enable = Yes
  • Exclude CERTIFICATE settings = No
  • Exclude NAT settings = No
  • Exclude Bind Address = Yes
  • Exclude DNS = No
  • Apply Configs = Yes
  • Exclude Trunks = No (see caveats above!)
  • Connect Warm Spare Server Over = API
  • Warm Spare Server = Backup Node
  • Access Token URL = (from Step 4, substituting the private 172. IPv4 address)
  • Client ID = (from Step 4)
  • Client Secret = (from Step 4)
  • GraphQL URL = (from Step 4, substituting the private 172. IPv4 address)
  • Now click the Get Warm Spare Token button, which should populate the Client Access Token field if the connection is setup properly and the Backup Instance can be reached

If you acquire the Client Access Token successfully, you may now Save the backup job.

This step is necessary so, when a fail-over is performed, calls are routed over NAT correctly. First, navigate the Secondary Instance GUI to Settings > Asterisk SIP Settings

If the Local Networks field is empty, click the Detect Network Settings button to populate the fields. Then make note of the CIDR presented (usually 172.31.x.0 / 20)

Navigate the Primary Instance GUI to Settings > Asterisk SIP Settings and paste the CIDR you copied from the Backup Instance or your VPC Subnet information. If an extra blank field is not showing, click the Add Local Network Field button. You should now have two sets of Local Networks showing, one for each Instance.

Your Warm Spare Setup is now complete. Data will be cloned from the Production Instance to the Backup Instance once per hour to keep it up-to-date.

If your Production Instance fails for any reason or you need to keep call services active while you upgrade the Production Instance (24-hour offices, etc), switching to the Backup Instance is as simple as moving your Elastic IP to the Backup Instance via the EC2 Console or AWS APIs.

• If your Trunking setup necessitates disabling trunks on the Backup Instance, you'll also need to enable them once you've moved the Elastic IP to the Backup Instance
• Switching back to the Production Instance is completed in reverse: Disable Trunks on Backup (if necessary), then swap Elastic IP to Production Instance

• Create a Machine Image backup of the 🔷Production🔷 Instance via the EC2 Console before performing any upgrades.
  • This will allow you to quickly rollback the Production Instance if upgrades fail and break the instance. This is accomplished using the Replace Root Volume wizard on the EC2 Console
• Temporarily Disable the Schedule of the Warm Spare Backup Job on the Production Instance
  1. Navigate the GUI to Admin > Backup and Restore on the Primary Instance
  2. Edit the Warm Spare job, change Schedule Enabled to No, then Save
• Run SmartUpgrade on the Primary Instance
  • smartupgrade –auto

• Run SmartUpgrade on the Secondary Instance
  • smartupgrade –auto
• Re-Enable the Schedule of the Warm Spare Backup Job on the Primary Instance
  1. Navigate the GUI to Admin > Backup and Restore on the Primary Instance
  2. Edit the Warm Spare job, change Schedule Enabled to Yes, then Save

1. Ensure the old Primary Instance can't interfere with the Secondary Instance by removing the old credentials from the Secondary Instance
   1. Remove the “ssh-rsa…” line from /root/.ssh/authorized_keys:
      • nano /root/.ssh/authorized_keys
   2. Remove the API Machine-to-Machine Application Connection from Connectivity > API
2. Switch over to the Secondary Instance
   1. Swap Elastic IP to Secondary Instance via the EC2 Console or AWS API
   2. Enable Trunks on the Secondary Instance, if necessary

The Secondary Instance is now your Primary Instance. Follow this guide from the start to setup a new Secondary Instance and restore redundancy.