Troubleshoot Resource Connectors and Connector Groups
Use the following information to troubleshoot connector issues:
- General Troubleshooting
- Throughput Capacity is Less Than Expected
- Users Cannot Connect to Private Resources
- Connector software Auto-Upgrade Fails
- Connector Operating System (OS) Version has Security Vulnerabilities
- Connector is Expired
- Stop a Connector
- Unable to Revoke or Delete a Connector
- Connector-Related Status Graphs are not Current
- A Note about Connector Issues
- Connector Diagnostics (CLI)
General Troubleshooting
If you are seeing issues with connectors or connector groups, try the following:
- Check the Connectivity section on the Overview page.
This provides a high-level view of the status of your organization's configured connectors and connector groups. - Check the Connector Groups page for issues:
Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group. - Check Monitor > Activity Search for logged events related to connectors.
- Determine whether the issue affects a single connector or all connectors in the group.
If the issue affects only one connector, it may be simplest to delete the connector and deploy a new one. - Check for a Secure Access service outage. See View Cloud Security Service Status.
- Temporarily disable a connector while you test the situation, then re-enable it as appropriate.
- Run Connector Diagnostics (CLI).
Note: Some troubleshooting guidance outside of this document may refer to connectors as "agents", "app connectors", or "application connectors".
Throughput Capacity is Less Than Expected
Ensure that connectors are using DTLS and not TLS:
Run Connector Diagnostics (CLI) on the connector and look for Protocol in the output.
If the protocol shows TLS, check the following:
- DTLS and TLS connections must originate from the same IP address.
- The egress firewall must allow UDP connections on port 443.
- The connector is the required instance type.
See Requirements and Prerequisites for Resource Connectors and Connector Groups. - In the connector instance, "Auto-Assign Public IP" must be enabled.
If you make changes, reboot the instance.
Users Cannot Connect to Private Resources
- Ensure that you have assigned the resource to the applicable connector group. See Assign Private Resources to a Connector Group.
- Verify that all connectors in the group can reach the resource.
- Check the status of the connector group that the resource is associated with:
Navigate to Connect > Network Connections > Connector Groups and look at the Status column for the applicable connector group. - For causes of user connectivity issues that are not specifically related to resource connectors, see Troubleshoot Private Access Rules.
Connector Software Auto-Upgrade Fails
For general information about automatic connector updates, see Maintain Resource Connectors.
- The upgrade problem may be temporary and self-repairing. Secure Access automatically periodically retries the upgrade.
- If upgrade fails for a single connector, this will prevent upgrades for all connectors in the group that have not yet been upgraded.
- To determine the problem connector, look for a yellow icon in the Version column on the connector group page:.
- Try deleting the problem connector instance and deploying a new one. See Disable, Revoke, or Delete Resource Connectors and Groups.
- Make sure the connector can connect to the destinations in Allow Resource Connector Traffic to Secure Access.
- If the upgrade continues to fail:
- Check for problems with your network or the connector instances themselves (such as a full disk.)
- Run Connector Diagnostics (CLI).
Connector Operating System (OS) Version has Security Vulnerabilities
Connector version issues are indicated in a list of connectors with a yellow icon ( ) Hover over the icon to identify the problem.
If you see a yellow icon for a connector version on the connector group's list of connectors, the platform operating system may have vulnerabilies. To determine whether this is the case, hover over the status icon. If OS vulnerabilities have been reported, you should replace affected connectors.
To replace connectors, see the "Connector platform operating system (OS) updates" section in Maintain and Monitor Resource Connectors and Connector Groups.
Connector is Expired
To ensure the integrity of your deployed connectors, Secure Access validates each connector during the connector's renewal period. The renewal period spans several weeks where Secure Access checks the status of a deployed connector. Toward the end of the renewal period if the connector's state is not valid, Secure Access disconnects the resource connector and the connector expires.
Check the Status of Your Connector
We recommend that you log into a deployed resource connector periodically and check the status of the connector.
- Log in to the virtual machine for the resource connector.
- In the connector's diagnostic output, locate sse_cloud, and then review the resource connector's diagnostic information.
If your resource connectors expire frequently, we recommend that you review the prerequisites for deploying resource connectors in your environment. For more information, see Requirements and Prerequisites for Resource Connectors and Connector Groups.
(VMware Only) View Connector Diagnostic Information
The resource connector console on VMware displays diagnostic information before the resource connector is shutdown.
For example:
systemd-shutdown[1]\: Failed to acquire terminal, using /dev/null stdin/stdout/stderr instead: Input/output error
Check for an Expired Connector in Secure Access
- Navigate to Connect > Network Connections > Connector Groups.
- Locate the Connector Group table and click on a Connector Group.
- Locate the connector and confirm that its Status is
Expired
.
Clean Up an Expired Connector
- Delete the expired connector instance from your virtual environment.
- Delete the expired connector from the connector group in Secure Access. For more information, see Disable, Revoke, or Delete Resource Connectors and Groups.
- If needed, deploy a new connector. For more information, see Add Connectors to a Connector Group.
Stop a Connector
if you need to stop a connector for any reason, power it off from within the deployment environment (AWS, VMware, etc.)
Unable to Revoke or Delete a Connector
This includes the inability of Secure Access to automatically delete an inactive connector.
This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run Connector Diagnostics (CLI).
Connector-Related Status Graphs are not Current
For example, the CPU Load chart does not have current data, or the graphs that you see when you click a connector ID in the list of connectors in a group are not up to date.
This situation is likely to be temporary and self-repairing, and can occur even if the connector is still passing traffic to private resources. If the problem does not resolve itself, run Connector Diagnostics (CLI).
A Note About Connector Issues
Connectors may be passing traffic successfully but not communicating otherwise with the Secure Access cloud.
This can cause problems with automatic software updates, monitoring connector health, gathering metrics, and deleting or revoking a connector, including automatically deleting inactive connectors.
These issues are typically temporary and self-repairing.
If they are not, first try restarting the connector instance.
If the issues continue, you can run connector diagnostics to troubleshoot these issues. See Connector Diagnostics (CLI).
Connector Diagnostics (CLI)
To troubleshoot connector issues that cannot be resolved by restarting or redeploying the connector instance, run diagnostic commands on the connector instance.
Supported Commands:
Command | Description |
---|---|
diagnostic | Try this command first when troubleshooting using the CLI. Runs a series of ping and DNS tests on the local IP address, gateway, Secure Access APIs, and headend IP addresses. It also checks connectivity on https port to various Secure Access endpoints, along with system artifacts created through the connector instance provisioning process. |
help | Lists commands available to the acadmin. |
routeadd | To add routes: Use "routeadd -n IP/MASK -g GW_IP" where the network is (-n IP/MASK) and the gateway is (-g GW_IP). Routes added with this command do not persist across reboots. |
routedel | To delete routes added using the routeadd command: Use "routedel -n IP/MASK -g GW_IP" where network is (-n IP/MASK) and gateway is (-g GW_IP). This command does not delete system created routes. |
routeshow | Display all routes in the system. |
tcpdump | Display packet capture information and filter on IP address and port, with limited functionality. You can run this command as: tcpdump $IP [$Port] (Port is optional) |
techsupport | Displays the following information: - software version - VPN tunnel state and related information - System monitoring metrics, such as disk and memory statistics - Information snapshot of the connector host, from DNS to Network interface details, iptables, routes, sysctl system settings, and monitoring information - Recent periodic resource connector software logs |
version | Show the connector software version and the platform OS version running on the connector instance. |
More | See Supported Standard Linux Troubleshooting Commands |
Run the Diagnostic Command
To run the diagnostic command:
- Run the connector diagnostic tool from the connector's command line:
- SSH to the connector instance.
Use the SSH key you configured while deploying the connector instance.
Sign in with user name acadmin. - At the $ prompt, enter diagnostic or another command of interest.
To understand the results of the diagnostic command, see the Diagnostic Codes tables below.
- SSH to the connector instance.
- If the issue persists, contact Support.
Diagnostic Codes
When you run the diagnostic command as described above, you will see state codes as described in the following tables.
If an issue is not listed in the following tables, or an issue persists after you have taken the recommended action, contact Cisco TAC.
Diagnostic Codes for Connector Update Issues
Code for Upgrade State | Description | Recommended Action | If Issues Persist |
---|---|---|---|
1 | Upgrade started. | No action needed. | N/A |
5 | Upgrade successful. | No action needed. | N/A |
101 | Secure Access will automatically attempt the upgrade again. | Monitor to ensure successful retry. | Contact Support (Cisco TAC) |
50, 51, 52 | Upgrade is in progress. | No action needed. | N/A |
200, 201, 202, 203 | Secure Access will automatically attempt the upgrade again. | Monitor to ensure successful retry. | Contact Support (Cisco TAC) |
204 | Issue may be self-repairing. | Monitor to ensure successful upgrade. | Contact Support (Cisco TAC) |
206 | Upgrade failed. | Check for full disk. | Contact Support (Cisco TAC) |
207 | Upgrade failed. | Run the techsupport diagnostic command to get more information. | Contact Support (Cisco TAC) |
208 | Image signature is invalid. | Run the techsupport diagnostic command to get more information. | Contact Support (Cisco TAC) |
210 | Upgrade failed. Secure Access will automatically attempt the upgrade again. | Check for disk issue. Monitor to ensure successful retry. | Contact Support (Cisco TAC) |
211 | Secure Access will automatically attempt the upgrade again. | Monitor to ensure successful retry. | Contact Support (Cisco TAC) |
All others | Issues that are not actionable by customers. | Contact Support (Cisco TAC) | N/A |
Diagnostic Codes for Other Connector Issues
Code for Daemon Init State | Recommended Action | If Issues Persist |
---|---|---|
6 | None; there are no issues. | N/A |
101 | Deploy a new connector instance. | -- |
102 | Deploy a new connector instance. Ensure that you have correctly copied the provisioning key. | -- |
103 | Deploy a new connector instance. | Contact Support (Cisco TAC) |
104 | Deploy a new connector instance. | Contact Support (Cisco TAC) |
105 | Deploy a new connector instance. | -- |
106 | Ensure that you have correctly copied the provisioning key. Ensure that connector can connec to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access. | -- |
108 | Ensure that the connector disk is writeable. | -- |
109 | Ensure that the connector disk is writeable. | -- |
110 | Ensure that the connector disk is writeable. | -- |
111 | Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access. | -- |
112 | Ensure that the connector instance time is correct. Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access. | -- |
113 | Ensure that the connector disk is writeable. | -- |
114 | Deploy a new connector instance. Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access. | -- |
115 | Ensure that connector can connect to the Secure Access cloud. See Allow Resource Connector Traffic to Secure Access. | -- |
116 | Deploy a new connector instance. | Contact Support (Cisco TAC) |
117 | Deploy a new connector instance. | Contact Support (Cisco TAC) |
118, 119 | Deploy a new connector instance. | -- |
All others | Deploy a new connector instance. | Contact Support (Cisco TAC) |
Supported Standard Linux Troubleshooting Commands
You can use the following commands on a connector instance:
ping, nslookup, traceroute, tcptraceroute, netstat, free, df, vmstat, iostat, mpstat, reboot, uptime, date, clear
Maintain and Monitor Resource Connectors and Connector Group < Troubleshoot Resource Connectors and Connector Groups > Secure Access Regions
Updated 3 months ago