Troubleshoot Resource Connectors and Connector Groups

Use the following information to troubleshoot connector issues:

General Troubleshooting

If you are seeing issues with connectors or connector groups, try the following:

  • Check the Connector Groups page for issues:
    Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
  • Check Monitor > Activity Search for logged events related to connectors.
  • Determine whether the issue affects a single connector or all connectors in the group.
    If the issue affects only one connector, it may be simplest to delete the connector and deploy a new one.
  • Check for a Secure Access service outage. See View Cloud Security Service Status.
  • Temporarily disable a connector while you test the situation, then re-enable it as appropriate.
  • Run Connector Diagnostics (CLI).

Note: Some troubleshooting materials outside this document may refer to connectors as "agents", "app connectors", or "application connectors."

Throughput capacity is less than expected

Ensure that connectors are using DTLS and not TLS:

Run Connector Diagnostics (CLI) on the connector and look for Protocol in the output.

If the protocol shows TLS, check the following:

If you make changes, reboot the instance.

Users cannot connect to private resources

  • Ensure that you have assigned the resource to the applicable connector group. See Assign Private Resources to a Connector Group.
  • Verify that all connectors in the group can reach the resource.
  • Check the status of the connector group that the resource is associated with:
    Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
  • For causes of user connectivity issues that are not specifically related to resource connectors, see Troubleshoot Private Access Rules.

Connector software auto-upgrade fails

For general information about automatic connector updates, see Maintain Resource Connectors.

  • The upgrade problem may be temporary and self-repairing. Secure Access automatically periodically retries the upgrade.
  • If upgrade fails for a single connector, this will prevent upgrades for all connectors in the group that have not yet been upgraded.
  • To determine the problem connector, look for a yellow icon in the Version column on the connector group page:Yellow icon: Connector version issue.
  • Try deleting the problem connector instance and deploying a new one. See Disable, Revoke, or Delete Resource Connectors and Groups.
  • Make sure the connector can connect to the destinations in Allow Resource Connector Traffic to Secure Access.
  • If the upgrade continues to fail:

Connector operating system (OS) version has security vulnerabilities

Connector version issues are indicated in a list of connectors with a yellow icon (Yellow icon: Connector version issue ) Hover over the icon to identify the problem.

If you see a yellow icon for a connector version on the connector group's list of connectors, the platform operating system may have vulnerabilies. To determine whether this is the case, hover over the status icon. If OS vulnerabilities have been reported, you should replace affected connectors.

To replace connectors, see the "Connector platform operating system (OS) updates" section in Maintain and Monitor Resource Connectors and Connector Groups.

Connector is expired

In order to ensure connector integrity, connector validation is periodically renewed. If a connector is disconnected at the time renewal must occur, it expires and can no longer be used.

Delete the expired connector instance from your virtual environment, delete the expired connector from the connector group in Secure Access, and deploy a new connector if needed.

Stop a connector

if you need to stop a connector for any reason, power it off from within the deployment environment (AWS, VMware, etc.)

Unable to revoke or delete a connector

This includes the inability of Secure Access to automatically delete an inactive connector.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run Connector Diagnostics (CLI).

Connector-related status graphs are not current

For example, the CPU Load chart does not have current data, or the graphs that you see when you click a connector ID in the list of connectors in a group are not up to date.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run Connector Diagnostics (CLI).

A note about connector issues

Connectors may be passing traffic successfully but not communicating otherwise with the Secure Access cloud.

This can cause problems with automatic software updates, monitoring connector health, gathering metrics, and deleting or revoking a connector, including automatically deleting inactive connectors.

These issues are typically temporary and self-repairing.

If they are not, first try restarting the connector instance.

If the issues continue, you can run connector diagnostics to troubleshoot these issues. See Connector Diagnostics (CLI).

Connector diagnostics (CLI)

To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run diagnostic commands on the connector instance.

Supported commands:

CommandDescription
diagnosticTry this command first when troubleshooting using the CLI.

Runs a series of ping and DNS tests on the local IP address, gateway, Secure Access APIs, and headend IP addresses. It also checks connectivity on https port to various Secure Access endpoints, along with system artifacts created through the connector instance provisioning process.
helpLists commands available to the acadmin.
routeaddTo add routes:

Use "routeadd -n IP/MASK -g GW_IP"
where the network is (-n IP/MASK) and the gateway is (-g GW_IP).

Routes added with this command do not persist across reboots.
routedelTo delete routes added using the routeadd command:

Use "routedel -n IP/MASK -g GW_IP"
where network is (-n IP/MASK) and gateway is (-g GW_IP).

This command does not delete system created routes.
routeshowDisplay all routes in the system.
tcpdumpDisplay packet capture information and filter on IP address and port, with limited functionality.

You can run this command as:
tcpdump $IP [$Port] (Port is optional)
techsupportDisplays the following information:

- software version
- VPN tunnel state and related information
- System monitoring metrics, such as disk and memory statistics
- Information snapshot of the connector host, from DNS to Network interface details, iptables, routes, sysctl system settings, and monitoring information
- Recent periodic resource connector software logs
versionShow the connector software version and the platform OS version running on the connector instance.
MoreSee Supported Standard Linux Troubleshooting Commands

Run the diagnostic command

To run the diagnostic command:

  1. Run the connector diagnostic tool from the connector's command line:
    1. SSH to the connector instance.
      Use the SSH key you configured while deploying the connector instance.
      Sign in with user name acadmin.
    2. At the $ prompt, enter diagnostic or another command of interest.
      To understand the results of the diagnostic command, see the Diagnostic Codes tables below.
  2. If the issue persists, contact Support.

Diagnostic Codes

When you run the diagnostic command as described above, you will see state codes as described in the following tables.

If an issue is not listed in the following tables, or an issue persists after you have taken the recommended action, contact Cisco TAC.

Diagnostic Codes for Connector Update Issues

Code for
Upgrade State
DescriptionRecommended ActionIf Issues Persist
1Upgrade started.No action needed.N/A
5Upgrade successful.No action needed.N/A
101Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC)
50, 51, 52Upgrade is in progress.No action needed.N/A
200, 201, 202, 203Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC)
204Issue may be self-repairing.Monitor to ensure successful upgrade.Contact Support (Cisco TAC)
206Upgrade failed.Check for full disk.Contact Support (Cisco TAC)
207Upgrade failed.Run the techsupport diagnostic command to get more information.Contact Support (Cisco TAC)
208Image signature is invalid.Run the techsupport diagnostic command to get more information.Contact Support (Cisco TAC)
210Upgrade failed.

Secure Access will automatically attempt the upgrade again.
Check for disk issue.

Monitor to ensure successful retry.
Contact Support (Cisco TAC)
211Secure Access will automatically attempt the upgrade again.Monitor to ensure successful retry.Contact Support (Cisco TAC)
All othersIssues that are not actionable by customers.Contact Support (Cisco TAC)N/A

Diagnostic Codes for Other Connector Issues

Code for
Daemon Init State
Recommended ActionIf Issues Persist
6None; there are no issues.N/A
101Deploy a new connector instance.--
102Deploy a new connector instance.

Ensure that you have correctly copied the provisioning key.
--
103Deploy a new connector instance.Contact Support (Cisco TAC)
104Deploy a new connector instance.Contact Support (Cisco TAC)
105Deploy a new connector instance.--
106Ensure that you have correctly copied the provisioning key.

Ensure that connector can connec to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access.
--
108Ensure that the connector disk is writeable.--
109Ensure that the connector disk is writeable.--
110Ensure that the connector disk is writeable.--
111Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access.--
112Ensure that the connector instance time is correct.

Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access.
--
113Ensure that the connector disk is writeable.--
114Deploy a new connector instance.

Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access.
--
115Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access.--
116Deploy a new connector instance.Contact Support (Cisco TAC)
117Deploy a new connector instance.Contact Support (Cisco TAC)
118, 119Deploy a new connector instance.--
All othersDeploy a new connector instance.Contact Support (Cisco TAC)

Supported Standard Linux Troubleshooting Commands

You can use the following commands on a connector instance:

ping, nslookup, traceroute, tcptraceroute, netstat, free, df, vmstat, iostat, mpstat, reboot, uptime, date, clear


Maintain and Monitor Resource Connectors and Connector Group < Troubleshoot Resource Connectors and Connector Groups > Secure Access Regions