Troubleshoot Resource Connectors and Connector Groups

This guide provides information about troubleshooting and managing your deployed Cisco Secure Access Resource Connectors. You can run the diagnostic commands in the Connector Diagnostics command-line interface (CLI) or the troubleshooting tools for containers to get the status of a resource connector.

Table of Contents

General Troubleshooting

If you see issues with the resource connectors or resource connector groups that are deployed, try the following:

  • Check the Connectivity section on the Overview in Secure Access.
    The Secure Access Overview provides a high-level view of the status of your organization's configured connectors and connector groups. For more information, see Secure Access Overview Dashboard.
  • Check the Connector Groups page for issues.
    Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
  • Check Monitor > Activity Search for logged events related to connectors.
  • Determine whether the issue affects a single connector or all connectors in the group.
    If the issue affects only one connector, you can delete the connector and deploy a new one.
  • Check for a Secure Access service outage. For more information, see View Cloud Security Service Status.
  • Temporarily disable a connector, then enable the connector again as appropriate.
  • Run diagnostics on a resource connector that is deployed with virtual machines. For more information, see Connector Diagnostics (CLI).
  • Run diagnostics on a resource connector that is deployed in a container. For more information, see Connector Troubleshooting Tools.

Note: Some troubleshooting guidance outside of this document may refer to connectors as "agents", "app connectors", or "application connectors".

About Resource Connector Issues

Resource Connectors may be exchanging traffic successfully but not communicating otherwise with Secure Access.

This can cause problems with automatic software updates, monitoring connector health, gathering metrics, and deleting or revoking a connector, including automatically deleting inactive connectors.

These issues are typically temporary and self-repairing.

Throughput Capacity is Less Than Expected

Users Cannot Connect to Private Resources

  • Ensure that you have assigned the resource to the applicable connector group. For more information, see Assign Private Resources to a Connector Group.
  • Verify that all connectors in the group can reach the resource.
  • Check the status of the connector group that the resource is associated with:
    Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
  • For causes of user connectivity issues that are not specifically related to resource connectors, see Troubleshoot Private Access Rules.

Connector Software Auto-Upgrade Fails

For general information about automatic connector updates, see Maintain Resource Connectors.

  • The upgrade problem may be temporary and self-repairing. Secure Access automatically retries the upgrade periodically.
  • If the upgrade fails for a single connector, this will prevent upgrades for all connectors in the group that have not yet been upgraded.
  • To determine the problem connector, look for a yellow icon (Yellow icon: Connector version issue ) in the Version column on the connector group page.
  • Try deleting the connector instance and deploying a new connector. For more information, see Disable, Revoke, or Delete Resource Connectors and Groups.
  • Make sure the connector can connect to the Secure Access destinations. For more information, see Allow Resource Connector Traffic to Secure Access.
  • If the upgrade continues to fail:
  • The network requirements for traffic on resource connectors may change. You will have to update the organization's firewall or access control list periodically. For information about the network requirements for Resource Connectors, see Allow Resource Connector Traffic to Secure Access.

Connector Operating System (OS) Version has Security Vulnerabilities

Secure Access shows connector version issues in a list of connectors with a yellow icon (Yellow icon: Connector version issue ). Hover over the icon to identify the problem.

If you see a yellow icon for a connector version on the connector group's list of connectors, the platform operating system may have vulnerabilities. To determine whether this is the case, hover over the status icon. If OS vulnerabilities have been reported, you should replace the affected connectors.

To replace connectors, see Maintain and Monitor Resource Connectors and Connector Groups: Connector platform operating system (OS) updates.

Manage Access Control and Vulnerabilities for Containers

  • The organization's administrators are responsible for the management of vulnerabilities and access control on the host or virtual machine.

Connector is Expired

To ensure the integrity of your deployed connectors, Secure Access validates each connector during the connector's renewal period. The renewal period spans several weeks where Secure Access checks the status of a deployed connector. Toward the end of the renewal period if the connector's state is not valid, Secure Access disconnects the resource connector and the connector expires.

Check the Status of Your Connector

We recommend that you log into a deployed resource connector periodically and check the status of the connector.

  1. Log in to the virtual machine for the resource connector.
  2. In the connector's diagnostic output, locate sse_cloud, and then review the resource connector's diagnostic information.

If your resource connectors expire frequently, we recommend that you review the prerequisites for deploying resource connectors in your environment. For more information, see Requirements and Prerequisites for Resource Connectors and Connector Groups.

(VMware Only) View Connector Diagnostic Information

The resource connector console on VMware displays diagnostic information before the resource connector is shutdown.

For example:

systemd-shutdown[1]\: Failed to acquire terminal, using /dev/null stdin/stdout/stderr instead: Input/output error

Check for an Expired Connector in Secure Access

  1. Navigate to Connect > Network Connections > Connector Groups.
  2. Locate the Connector Group table and click on a Connector Group.
  3. Locate the connector and confirm that its Status is Expired.

Clean Up an Expired Connector

  1. Delete the expired connector instance from your virtual environment.
  2. Delete the expired connector from the connector group in Secure Access. For more information, see Disable, Revoke, or Delete Resource Connectors and Groups.
  3. If needed, deploy a new connector. For more information, see Add Connectors to a Connector Group.

Delete the Connector Container

For information about removing the connector container, see Delete the Container.

Stop a Connector

If you need to stop a Resource Connector for any reason, power it off from within the deployment environment (for example: AWS, VMware, Azure).

Stop the Connector Container

For information about stopping the connector container, see Deploy a Connector in Docker – Stop the Container.

Unable to Revoke or Delete a Connector

This includes the inability of Secure Access to automatically delete an inactive resource connector.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run diagnostics on the resource connector. For more information, see Connector Diagnostics (CLI).

Unable to Sync

An unable to sync error condition indicates that the resource connector can not connect to the Secure Access Controller component.

When a resource connector is unable to sync, Secure Access does not:

  • Display the metrics for the resource connector.
  • Update the configuration for the resource connector.
  • Perform over-the-air upgrades on the resource connector.

To troubleshoot the error condition, run the Command-line Interface (CLI) diagnostic tool to check the internet connectivity on the Controller. Also ensure that the firewall rules in the organization's environment allow the resource connector to make outbound connections to the Controller. For more information, see Allow Resource Connector Traffic to Secure Access.

If the error condition persists, contact Cisco Support at https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html.

Connector-Related Status Graphs are not Current

For example, the CPU Load chart does not have current data, or the graphs that you see when you click a connector ID in the list of connectors in a group are not up to date.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, we recommend that you get the diagnostic information about the resource connector. For more information, see Connector Diagnostics (CLI) and Connector Troubleshooting Tools.

(Container Only) Connector Troubleshooting Tools

To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run the Secure Access troubleshooting scripts on the container and connector instance.

To understand the results of the diagnostic command, see Diagnostic Codes.

Supported Linux Commands

Secure Access supports Linux commands that you can run on a resource connector instance. For more information, see Supported Standard Linux Troubleshooting Commands.

Run Diagnostic or Techsupport Scripts

For more information, see Run Diagnostic and Techsupport Scripts.

The diagnostic script runs a series of ping and DNS tests on the local IP address, gateway, Secure Access APIs, and headend IP addresses. The script also checks connectivity on HTTPS ports to various Secure Access endpoints, along with system artifacts created through the connector instance provisioning process.

The techsupport script displays the following information:

  • Version of the resource connector software.
  • VPN tunnel state and related information.
  • System monitoring metrics, such as disk and memory statistics.
  • Information snapshot of the connector host, from DNS to Network interface details, iptables, routes, sysctl system settings, and monitoring information.
  • Recent periodic resource connector software logs.

Troubleshoot Container Deployments

For more information, see Troubleshoot Container Deployments.

(VM Only) Connector Diagnostics (CLI)

To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run diagnostic commands on the connector instance.

Supported Commands

Secure Access supports commands that you can run on a resource connector instance. For more information, see Supported Standard Linux Troubleshooting Commands.

CommandDescription
diagnosticTry this command first when troubleshooting using the CLI.

Runs a series of ping and DNS tests on the local IP address, gateway, Secure Access APIs, and headend IP addresses. It also checks connectivity on HTTPS port to various Secure Access endpoints, along with system artifacts created through the connector instance provisioning process.
helpLists commands available to the acadmin user.
routeaddTo add routes:

Use "routeadd -n IP/MASK -g GW_IP"
where the network is (-n IP/MASK) and the gateway is (-g GW_IP).

Routes added with this command do not persist across reboots.
routedelTo delete routes added using the routeadd command:

Use "routedel -n IP/MASK -g GW_IP"
where network is (-n IP/MASK) and gateway is (-g GW_IP).

This command does not delete system created routes.
routeshowDisplay all routes in the system.
tcpdumpDisplay packet capture information and filter on IP address and port, with limited functionality.

You can run this command as:
tcpdump $IP [$Port] (Port is optional)
techsupportDisplays the following information:

- software version
- VPN tunnel state and related information
- System monitoring metrics, such as disk and memory statistics
- Information snapshot of the connector host, from DNS to Network interface details, iptables, routes, sysctl system settings, and monitoring information
- Recent periodic resource connector software logs
versionShow the connector software version and the platform OS version running on the connector instance.

Run the Diagnostic Command

To run the diagnostic command:

  1. Run the connector diagnostic tool from the connector's command line:
    1. SSH to the connector instance.
      Use the SSH key you configured while deploying the connector instance.
      Sign in with user name acadmin.
    2. At the $ prompt, enter diagnostic or another command of interest.
      To understand the results of the diagnostic command, see Diagnostic Codes.
  2. If the issue persists, contact Support.

Diagnostic Codes

When you run the diagnostic commands, you will see state codes. For any issues that are not described with a diagnostic code, contact Cisco Secure Access Support at https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html.

If an issue is not listed in the following tables, or an issue persists after you have taken the recommended action, contact Cisco TAC.

Diagnostic Codes for Connector Update Issues

Code for Upgrade StateDescriptionRecommended ActionIf Issues Persist
1–99Normal state or upgrade is in progress.No action needed.N/A
100,102Upgrade failed.Contact Support (Cisco TAC).
101Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC).
200Docker login authentication failure.
Check docker login credentials for container.
For VM, monitor to ensure successful retry.Contact Support (Cisco TAC) to provide guidance when setting up the Docker account.
201–204Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC).
204Issue may be self-repairing.Monitor to ensure successful upgrade.Contact Support (Cisco TAC).
206Upgrade failed.Check for full disk.Contact Support (Cisco TAC).
207,208Upgrade failed.To get more information, run the techsupport diagnostic command.Contact Support (Cisco TAC).
210Upgrade failed.Secure Access will automatically attempt the upgrade again.
Check for disk issue.
Monitor to ensure successful retry.
Contact Support (Cisco TAC).
211Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC).
217Docker hub download failed due to rate limiting.Authenticate with docker, using docker login.Contact Support (Cisco TAC) to provide guidance when setting up the Docker account.
219Host does not have sufficient disk space.Free up at least 2G of disk space in host to ensure this auto recovers.Contact Support (Cisco TAC).

Diagnostic Codes for Other Connector Issues

Code for
Daemon Init State
Recommended ActionIf Issues Persist
6None; there are no issues.N/A
101Deploy a new connector instance.--
102Deploy a new connector instance.

Ensure that you have correctly copied the provisioning key.
--
103Deploy a new connector instance.Contact Support (Cisco TAC)
104Deploy a new connector instance.Contact Support (Cisco TAC)
105Deploy a new connector instance.--
106Ensure that you have correctly copied the provisioning key.

Ensure that connector can connec to the Secure Access cloud. For more information, see Allow Resource Connector Traffic to Secure Access.
--
108Ensure that the connector disk is writeable.--
109Ensure that the connector disk is writeable.--
110Ensure that the connector disk is writeable.--
111Ensure that connector can connect to the Secure Access cloud. For more information, see Allow Resource Connector Traffic to Secure Access.--
112Ensure that the connector instance time is correct.

Ensure that connector can connect to the Secure Access cloud. For more information, see Allow Resource Connector Traffic to Secure Access.
--
113Ensure that the connector disk is writeable.--
114Deploy a new connector instance.

Ensure that connector can connect to the Secure Access cloud. For more information, see Allow Resource Connector Traffic to Secure Access.
--
115Unable to sync the resource connector with Secure Access. If the error condition persists, contact Cisco Support at https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html.

Ensure that the connector can connect to Secure Access. For more information, see Allow Resource Connector Traffic to Secure Access.
--
116Deploy a new connector instance.Contact Support (Cisco TAC)
117Deploy a new connector instance.Contact Support (Cisco TAC)
118, 119Deploy a new connector instance.--
All othersDeploy a new connector instance.Contact Support (Cisco TAC)

Supported Standard Linux Troubleshooting Commands

You can use these Linux commands to provide information about a resource connector instance:

  • ping
  • nslookup
  • traceroute
  • tcptraceroute
  • netstat
  • free
  • df
  • vmstat
  • iostat
  • mpstat
  • reboot
  • uptime
  • date
  • clear

Maintain and Monitor Resource Connectors and Connector Group < Troubleshoot Resource Connectors and Connector Groups > Secure Access Regions