Troubleshoot Resource Connectors and Connector Groups

This guide provides information about troubleshooting and managing your deployed Cisco Secure Access Resource Connectors. You can run the diagnostic commands in the Connector Diagnostics command-line interface (CLI) or the troubleshooting tools for containers to get the status of a resource connector.

Table of Contents

• General Troubleshooting
• About Resource Connector Issues
• Throughput Capacity is Less Than Expected
• Users Cannot Connect to Private Resources
• Connector Software Auto-Upgrade Fails
• Connector Operating System (OS) Version has Security Vulnerabilities
• Connector is Expired
• Stop a Connector
• Unable to Revoke or Delete a Connector
• Unable to Sync
• Connector-Related Status Graphs are not Current
• (Container Only) Connector Troubleshooting Tools
• (VM Only) Connector Diagnostics (CLI)
• Diagnostic Codes

General Troubleshooting

If you see issues with the resource connectors or resource connector groups that are deployed, try the following:

• Check the Connectivity section on the Overview in Secure Access.
  The Secure Access Overview provides a high-level view of the status of your organization's configured connectors and connector groups. For more information, see Secure Access Overview Dashboard.
• Check the Connector Groups page for issues.
  Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
• Check Monitor > Activity Search for logged events related to connectors.
• Determine whether the issue affects a single connector or all connectors in the group.
  If the issue affects only one connector, you can delete the connector and deploy a new one.
• Check for a Secure Access service outage. For more information, see View Cloud Security Service Status.
• Temporarily disable a connector, then enable the connector again as appropriate.
• Run diagnostics on a resource connector that is deployed with virtual machines. For more information, see Connector Diagnostics (CLI).
• Run diagnostics on a resource connector that is deployed in a container. For more information, see Connector Troubleshooting Tools.

Note: Some troubleshooting guidance outside of this document may refer to connectors as "agents", "app connectors", or "application connectors".

About Resource Connector Issues

Resource Connectors may be exchanging traffic successfully but not communicating otherwise with Secure Access.

This can cause problems with automatic software updates, monitoring connector health, gathering metrics, and deleting or revoking a connector, including automatically deleting inactive connectors.

These issues are typically temporary and self-repairing.

• If they are not, first try restarting the connector instance.
• If the issues continue, you can run diagnostics on the connector to troubleshoot these issues. For more information, see Connector Diagnostics (CLI) and Connector Troubleshooting Tools.

Throughput Capacity is Less Than Expected

• Ensure that connectors are using DTLS and not TLS.
• Run diagnostics on the connector and look for Protocol in the output. For more information, see Connector Diagnostics (CLI) and Connector Troubleshooting Tools.
• If the protocol shows TLS, check the following:
  • DTLS and TLS connections must originate from the same IP address.
  • The egress firewall must allow UDP connections on port 443.
  • The connector is the required instance type. For more information, see Requirements and Prerequisites for Resource Connectors and Connector Groups.
  • In the connector instance, Auto-Assign Public IP must be enabled.
• If you make changes, reboot the instance.

Users Cannot Connect to Private Resources

• Ensure that you have assigned the resource to the applicable connector group. For more information, see Assign Private Resources to a Connector Group.
• Verify that all connectors in the group can reach the resource.
• Check the status of the connector group that the resource is associated with:
  Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group.
• For causes of user connectivity issues that are not specifically related to resource connectors, see Troubleshoot Private Access Rules.

Connector Software Auto-Upgrade Fails

For general information about automatic connector updates, see Maintain Resource Connectors.

• The upgrade problem may be temporary and self-repairing. Secure Access automatically retries the upgrade periodically.
• If the upgrade fails for a single connector, this will prevent upgrades for all connectors in the group that have not yet been upgraded.
• To determine the problem connector, look for a yellow icon ( ) in the Version column on the connector group page.
• Try deleting the connector instance and deploying a new connector. For more information, see Disable, Revoke, or Delete Resource Connectors and Groups.
• Make sure the connector can connect to the Secure Access destinations. For more information, see Allow Resource Connector Traffic to Secure Access.
• If the upgrade continues to fail:
  • Check for problems with your network or the connector instances themselves (such as a full disk).
  • Run diagnostics on the connector. For more information, see Connector Diagnostics (CLI) and Connector Troubleshooting Tools.
• The network requirements for traffic on resource connectors may change. You will have to update the organization's firewall or access control list periodically. For information about the network requirements for Resource Connectors, see Allow Resource Connector Traffic to Secure Access.

Connector Operating System (OS) Version has Security Vulnerabilities

Secure Access shows connector version issues in a list of connectors with a yellow icon ( ). Hover over the icon to identify the problem.

If you see a yellow icon for a connector version on the connector group's list of connectors, the platform operating system may have vulnerabilities. To determine whether this is the case, hover over the status icon. If OS vulnerabilities have been reported, you should replace the affected connectors.

To replace connectors, see Maintain and Monitor Resource Connectors and Connector Groups: Connector platform operating system (OS) updates.

Manage Access Control and Vulnerabilities for Containers

• The organization's administrators are responsible for the management of vulnerabilities and access control on the host or virtual machine.

Connector is Expired

To ensure the integrity of your deployed connectors, Secure Access validates each connector during the connector's renewal period. The renewal period spans several weeks where Secure Access checks the status of a deployed connector. Toward the end of the renewal period if the connector's state is not valid, Secure Access disconnects the resource connector and the connector expires.

Check the Status of Your Connector

We recommend that you log into a deployed resource connector periodically and check the status of the connector.

1. Log in to the virtual machine for the resource connector.
2. In the connector's diagnostic output, locate sse_cloud, and then review the resource connector's diagnostic information.

If your resource connectors expire frequently, we recommend that you review the prerequisites for deploying resource connectors in your environment. For more information, see Requirements and Prerequisites for Resource Connectors and Connector Groups.

(VMware Only) View Connector Diagnostic Information

The resource connector console on VMware displays diagnostic information before the resource connector is shutdown.

For example:

systemd-shutdown[1]\: Failed to acquire terminal, using /dev/null stdin/stdout/stderr instead: Input/output error

Check for an Expired Connector in Secure Access

1. Navigate to Connect > Network Connections > Connector Groups.
2. Locate the Connector Group table and click on a Connector Group.
3. Locate the connector and confirm that its Status is Expired.

Clean Up an Expired Connector

1. Delete the expired connector instance from your virtual environment.
2. Delete the expired connector from the connector group in Secure Access. For more information, see Disable, Revoke, or Delete Resource Connectors and Groups.
3. If needed, deploy a new connector. For more information, see Add Connectors to a Connector Group.

Delete the Connector Container

For information about removing the connector container, see Delete the Container.

Stop a Connector

If you need to stop a Resource Connector for any reason, power it off from within the deployment environment (for example: AWS, VMware, Azure).

Stop the Connector Container

For information about stopping the connector container, see Deploy a Connector in Docker – Stop the Container.

Unable to Revoke or Delete a Connector

This includes the inability of Secure Access to automatically delete an inactive resource connector.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run diagnostics on the resource connector. For more information, see Connector Diagnostics (CLI).

Unable to Sync

An unable to sync error condition indicates that the resource connector can not connect to the Secure Access Controller component.

When a resource connector is unable to sync, Secure Access does not:

• Display the metrics for the resource connector.
• Update the configuration for the resource connector.
• Perform over-the-air upgrades on the resource connector.

To troubleshoot the error condition, run the Command-line Interface (CLI) diagnostic tool to check the internet connectivity on the Controller. Also ensure that the firewall rules in the organization's environment allow the resource connector to make outbound connections to the Controller. For more information, see Allow Resource Connector Traffic to Secure Access.

If the error condition persists, contact Cisco Support at https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html.

Connector-Related Status Graphs are not Current

For example, the CPU Load chart does not have current data, or the graphs that you see when you click a connector ID in the list of connectors in a group are not up to date.

This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, we recommend that you get the diagnostic information about the resource connector. For more information, see Connector Diagnostics (CLI) and Connector Troubleshooting Tools.

(Container Only) Connector Troubleshooting Tools

To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run the Secure Access troubleshooting scripts on the container and connector instance.

To understand the results of the diagnostic command, see Diagnostic Codes.

Supported Linux Commands

Secure Access supports Linux commands that you can run on a resource connector instance. For more information, see Supported Standard Linux Troubleshooting Commands.

Run Diagnostic or Techsupport Scripts

For more information, see Run Diagnostic and Techsupport Scripts.

The diagnostic script runs a series of ping and DNS tests on the local IP address, gateway, Secure Access APIs, and headend IP addresses. The script also checks connectivity on HTTPS ports to various Secure Access endpoints, along with system artifacts created through the connector instance provisioning process.

The techsupport script displays the following information:

• Version of the resource connector software.
• VPN tunnel state and related information.
• System monitoring metrics, such as disk and memory statistics.
• Information snapshot of the connector host, from DNS to Network interface details, iptables, routes, sysctl system settings, and monitoring information.
• Recent periodic resource connector software logs.

Troubleshoot Container Deployments

For more information, see Troubleshoot Container Deployments.

(VM Only) Connector Diagnostics (CLI)

To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run diagnostic commands on the connector instance.

Supported Commands

Secure Access supports commands that you can run on a resource connector instance. For more information, see Supported Standard Linux Troubleshooting Commands.

Run the Diagnostic Command

To run the diagnostic command:

1. Run the connector diagnostic tool from the connector's command line:
   1. SSH to the connector instance.
      Use the SSH key you configured while deploying the connector instance.
      Sign in with user name acadmin.
   2. At the $ prompt, enter diagnostic or another command of interest.
      To understand the results of the diagnostic command, see Diagnostic Codes.
2. If the issue persists, contact Support.

Diagnostic Codes

When you run the diagnostic commands, you will see state codes. For any issues that are not described with a diagnostic code, contact Cisco Secure Access Support at https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html.

If an issue is not listed in the following tables, or an issue persists after you have taken the recommended action, contact Cisco TAC.

Diagnostic Codes for Connector Update Issues

Diagnostic Codes for Other Connector Issues

Supported Standard Linux Troubleshooting Commands

You can use these Linux commands to provide information about a resource connector instance:

• ping
• nslookup
• traceroute
• tcptraceroute
• netstat
• free
• df
• vmstat
• iostat
• mpstat
• reboot
• uptime
• date
• clear

---

Maintain and Monitor Resource Connectors and Connector Group < Troubleshoot Resource Connectors and Connector Groups > Secure Access Regions