Geographic redundancy

You can implement geographic redundancy if your primary data center and standby data center are in geographically separate locations. If one data center experiences a catastrophic event, such as a hurricane or other natural disaster, you can make the standby data center active and operations can resume.

Geographic redundancy has the following specific requirements to be successful:

• The primary and standby environments must be duplicated. Each data center must have the same number of machines in the ArcGIS Enterprise deployment, and the URLs used to access the components must be the same.
• ArcGIS Server directories must have the same name. Paths to the directory can be different, but the folder name must be the same on the primary and standby environments.
• Folders registered with the ArcGIS Server sites in the primary and standby environments can have different paths but the folder names must be the same, and they must contain exact copies of the same source data.
• Geographic redundancy typically follows an active-passive approach; therefore, data and content must be replicated to the standby ArcGIS Enterprise deployment consistently.
• Geographic redundancy relies on third-party components to be successful. For example, a global site selector or global domain name system (DNS) server is important so that when a switch has to occur from the primary data center to the standby, there is no disruption to any ArcGIS Enterprise users.

To ensure the least amount of downtime in event of a failure or catastrophe, you could deploy a highly available, geographically redundant ArcGIS Enterprise. This is the most complex deployment to achieve, as it requires the most machines and the most maintenance. Configure two separate data centers, each with their own highly available ArcGIS Enterprise deployment. In each data center, all the machine names are configured identically and there are no single points of failure, which include the data, whether it resides in a highly available file server or highly available database, all web servers and load balancers, as well as the ArcGIS Enterprise components. Backups of the primary deployment are consistently created, and restoration to the standby deployment in the separate data center can occur immediately or when a failure in the primary deployment occurs.

Plan for a replicated deployment

First, determine how many machines you require. Next, plan for the following disaster recovery requirements for a replicated ArcGIS Enterprise deployment:

• Duplication: Ensure both data centers and ArcGIS Enterprise deployments contain the same architecture.
• Replication: Back up content and data from the primary data center and restore to the standby.
• Monitoring: Review logs to determine when a failure occurs and determine whether the severity of the failure requires you to fail over to the standby data center.
• Failover: Decide whether to fail over to a different component within ArcGIS Enterprise or fail over the entire ArcGIS Enterprise deployment to a different data center.

Also keep the following in mind when planning your replicated deployment:

• The webgisdr utility does not move map service cache tiles. If you include map service or hosted tile layer caches used by the GIS Server site in your deployment, make a backup of all directories where your cache tiles are stored (for example, the entire arcgiscache directory under C:\arcgisserver\directories\ or <ArcGIS Server installation directory>/arcgis/server/usr/directories). Manually place the copies in the corresponding arcgiscache directory on the standby deployment.
• Multiple ArcGIS Server clusters are not supported when using the webgisdr utility to replicate ArcGIS Enterprise to a disconnected standby deployment.
• All machines in both deployments must use the same operating system. For example, your primary deployment cannot be on Windows machines and your standby be on Linux.
• The webgisdr utility records the software versions of the ArcGIS Enterprise components when you create a backup file. The standby deployment to which you import the file must be at the same version as your primary deployment.

Determine machine requirements

The number of machines you need depends on how you configure ArcGIS Enterprise. At a minimum, you need two machines. If your ArcGIS Enterprise deployment does not store a lot of data and services, does not include a spatiotemporal big data store, does not include a graph store, and is not accessed by many people, you can configure a primary deployment composed of a single-machine GIS Server site, and install Portal for ArcGIS and ArcGIS Data Store on the same machine. You need a second machine to store the replicated standby deployment.

If your ArcGIS Enterprise deployment is more heavily used—for example, if a large number of people access it, your organization stores a large number of items, or your deployment is heavily edited—you may need a single or multimachine GIS Server site, and you should install Portal for ArcGIS and ArcGIS Data Store on machines separate from each other and separate from the GIS Server machines. If you publish multiple hosted scene layers, you may want to configure ArcGIS Data Store (tile cache data store) to store the scene cache databases on another machine. If you'll be using a graph store, you need an additional machine. If you'll be using a spatiotemporal big data store, you'll need at least one additional machine. In this case, calculate the number of machines required using the following formula:

(<number of GIS Server machines> + 1 Portal for ArcGIS machine + <number of machines in the data store>) X 2

Note that additional ArcGIS licenses are not required for the standby deployment because it is not actively accessed; you only make it the active deployment if the primary fails.

Required settings for duplicate deployments

To ensure effective disaster recovery for a replicated ArcGIS Enterprise deployment, the standby deployment must duplicate the range of system settings, security configurations, and storage locations found in the primary deployment. Regularly creating backups and ensuring consistency between replicated deployments is the best way to minimize downtime in the event of a failure. These considerations should be made across your deployment. Some examples are as follows:

• Map services rely on data in a shared folder or through a database connection.
• Machines in an ArcGIS Enterprise deployment communicate with each other through specific URLs.

  Tip:

  Use DNS entries or modify hosts files on the machines of your replicated deployment to achieve host name consistency. The recommended approach to doing this is to set up a separate machine to act as the public portal URL. You can install the ArcGIS Web Adaptor or reverse proxy server on this machine and modify the hosts files on the portal and server machines.

• Installation directory locations need to be identical between replicated deployments.
• The number of machines in your data centers should match to avoid performance issues in response to user load.

The following system and security settings should not be replicated across deployments, as they are specific to each deployment:

• Forward proxy information, including server names, user-facing URLs, and port numbers
• Security settings, including the list of addresses approved by the portal's proxy capability
• Identity store configuration properties for user and group stores, if applicable
• SAML and LDAP identity provider settings

Since 10.4, the list of items and settings that must be identical across your source and target deployments when running the webgisdr utility has been shortened. The following table summarizes these changes across recent versions of Portal for ArcGIS and ArcGIS Server:

Must this item or setting be identical across deployments when running the webgisdr utility?

Replicate ArcGIS Enterprise

The webgisdr utility allows you to export portal content, federated ArcGIS Server sites, and ArcGIS Data Store relational and tile cache data store content to a file that you can move to the standby machine to restore. The utility maintains Portal for ArcGIS, ArcGIS Server, and ArcGIS Data Store configured settings, and copies all content created in the portal as well as data that's copied to the hosting server and data store while publishing.

The utility does not copy data from databases or folders registered with the hosting server or federated ArcGIS Server sites. It's the administrator's responsibility to replicate that data to the standby ArcGIS Enterprise deployment and ensure that services on the standby machine can access the replicated data.

When you register data sources with ArcGIS Server sites, you provide specific information on how to access the data. That information must be the same for the standby deployment as for the primary. For example, if you copy file geodatabases used for source data to the standby deployment, directory paths to the file geodatabases must be the same as on the primary deployment. Also, the standby deployment must be able to access a database using the same connection information you provided when you registered the database with the ArcGIS Server site on the primary deployment.

You can run the webgisdr utility as a cron job in a Linux environment. Additionally, the utility can be moved to and run from a different machine than the portal installation, as long as communication is open between the machine where the utility is run and the ArcGIS Enterprise components.

You should restore the ArcGIS Enterprise backups to the standby deployment as soon as they're exported from the primary deployment. This avoids restoring incremental backups in the wrong order and means that minimal data loss or downtime will occur in the event that the primary deployment fails. If you do not immediately restore backups, there may be additional overhead in importing the backup and failing over to the standby deployment.

Also consider that if something is incorrect in the primary deployment when the backup is created and there are automated processes to import the backup to the standby, those incorrect settings will be imported to the standby deployment.

See Configure disaster recovery for instructions on replicating an ArcGIS Enterprise deployment.

Monitor ArcGIS Enterprise

Monitoring is important in both replicated and highly available environments. In a highly available environment, certain parts of the deployment fail over without human intervention. For example, if the primary portal in ArcGIS Enterprise fails, the software immediately fails over to the standby without any human intervention. Similarly, ArcGIS Server and ArcGIS Data Store components can fail, and the system can function as normal as there are no single points of failure. Considering there may be no visible disruptions in ArcGIS Enterprise, you should put mechanisms in place to notify administrators of failures on any particular component within the ArcGIS Enterprise deployment.

You can use ArcGIS Monitor to analyze the health of the Portal for ArcGIS, ArcGIS Server, and relational ArcGIS Data Store components of your deployment. You can also use the Portal Index Task to query the indexer status on the primary portal machine before you replicate your deployment. If you use a registered PostgreSQL, Oracle, or Microsoft SQL Server database with your deployment, you can use one of the Egdb tasks available in the ArcGIS Monitor gallery to monitor the statistics for those databases.

You'll need to use Python or the scripting language of your choice with the ArcGIS Server REST API to automate validation of connections to registered folders, big data file shares, raster data stores, tile caches, and spatiotemporal big data stores.

In a replicated environment, failover requires human intervention; therefore, you must monitor your deployment to determine when failures occur so you can decide if a failover is necessary.

If you automate the replication of your deployment from primary to standby, you also need to monitor these processes to be sure backups, moving of files, and restore operations complete successfully.

About failover

In ArcGIS Enterprise, Portal for ArcGIS, ArcGIS Server, and ArcGIS Data Store have their own internal mechanisms to fail over. In a highly available configuration, each component can fail over without significant disruption to the overall ArcGIS Enterprise.

Failover of a replicated deployment from the primary to the standby data center typically involves the organization's IT department and can be achieved through a global site selector (GSS) or global DNS. Members of an organization typically reach their ArcGIS Enterprise deployment through a few URLs, for example, https://myportalwa.organization.com/portal for the portal URL and https://myserverwa.organization.com/server for the ArcGIS Server services URL. The GSS or global GNS can assign an IP address to each host name. If you need to fail over to a different data center, the GSS or global DNS will reassign the myportalwa.organization.com and myserverwa.organization.com host names to the IP addresses associated with the standby data center. Clients and users will not be affected, but all requests are sent to the standby data center. Once the primary data center is back online, the IP address for the primary site hosts can be reassigned to IP addresses within the original data center. You would then need to reconcile data from the standby to the primary to ensure the primary data center contains all of the new content and data that was created while the standby was active.

If data in any of the hosting server or federated ArcGIS Server site's registered databases (enterprise geodatabase or database) was edited, use database replication tools to ensure the original primary ArcGIS Enterprise deployment contains that updated data. If data in file-based data sources, such as file geodatabases, registered with any of the ArcGIS Server sites in the ArcGIS Enterprise deployment has changed, copy the edited files to the original directory where it was stored. Finally, use the webgisdr utility to export an ArcGIS Enterprise backup from the standby and import it to the primary. The utility will replicate the content in the portal, including associated hosted feature and scene layer data and new services registered with the portal, to the original primary ArcGIS Enterprise deployment.