Habitat Stalled Agents Tool

Support forums for the Habitat Automate plugin
User avatar
Cubert
Posts: 1536
Joined: Tue Dec 29, 2015 7:57 pm
Contact:

Habitat Stalled Agents Tool

Post by Cubert »

Habitat has taken Stalled Agents to the next level by automating the management of the LTagent services with ScreenConnect. Habitat looks at offline agents and will attempt to restart the LT agent services on that agent if it sees that agent is up in Screenconnect. After the agent goes offline for more than 24 hours the attempts will stop.

Stalled Agent detector is just that, This tool looks into all agents to see if any agents show running excessive commands or polling the commands, to see if any agent shows offline in either Automate or Screenconnect or shows know active scripting bug issues and displays those agents in plugin. The plugin provides tools to manually clear and or repair a given agent and an automated service to look for offline agents in Automate that are active in Screenconnect and restarts the Automate Services through Screenconnect.

SAD-SC-Offline_agents.PNG
SAD-SC-Offline_agents.PNG (23.67 KiB) Viewed 11826 times

The tool uses 4 main tabs to display possible troubled agents. if you do not see any agents in a view then this is "Great"! that view shows you have no agents with issues.

Click into a view then right click an agent in view to pull up menu of actions to perform on agent.

STA-OfflinetAgents.PNG
STA-OfflinetAgents.PNG (38.96 KiB) Viewed 11826 times


Red dots = agents offline over 24 hours
Green dots = agents offline less than 24 hours.


Use AgentID for Automation Task

The automation time it takes to send and receive screenconnect request using the automation in the Automate server would cause issues and overruns as the timing repeats to fast to make easy use of the automate host for agent screenconnect repairs so we offload that to an agent that can has time to spare. This process is not a heavy or strenuous process for the agent and it resolves the timing issues created when multiple agents are offline and needing service restarts. Select an agent that is always on, can be a server or workstation that can see the Screenconnect URL. Only one agent is needed for all clients and not 1 agent per client.

AgentID-for-automation-tasks.PNG
AgentID-for-automation-tasks.PNG (1.38 KiB) Viewed 11826 times

Using ScreenConnect to restart offline agents.

For you to use this feature you will need to first install and configure the RMM+ tool for Screen Connect. This tool can be found in your SrceenConnect web console under "integrations" in the admin panel.


ScreenConnect-InstallRMMPlusTool.PNG
ScreenConnect-InstallRMMPlusTool.PNG (182.26 KiB) Viewed 11824 times

Once installed and enabled and you created your private key for it during the setup process, take that key and update the Habitat stalled Agents tool by selecting an agent and right clicking for menu.

SAD-SetAccessCode.PNG
SAD-SetAccessCode.PNG (43.71 KiB) Viewed 11824 times

This will open a window to place in your RMM+ security key code.

SAD-Save-code.PNG
SAD-Save-code.PNG (14.08 KiB) Viewed 11824 times

Now your able to use screenconnect to manage the LTAgent services on the local computers.

MarkHodges
Posts: 21
Joined: Tue Jun 04, 2019 6:07 pm

Re: Habitat Stalled Agents Tool

Post by MarkHodges »

Do we need to manually install and configure the RMM+ screenconnect plugin to get the non RPC restart working correctly?
Might be something to include documentation on as part of the habitat as some people might not know about RMM+ at all.

I believe the whole point of the plugin was to find machines where screenconnect is running, but the automate agent is stopped (ie, LT offline Agents). Every machine I try this on gives me a variation of this error.

Starting Screen Connect connections to PCNAME, Sending wakeup call
Lets give the wake up call a few seconds to do it's job
Agent Online time in second (if a negitive number then it is the number of seconds offline) -> -2385627

Agents ScreenConnect service is Offline, we are unable to use screenconnect to access agent.

I am guessing this actually means that something with the rmm+ isn't working correctly which according to the actual rmm+ posts, looks like I am not alone. I've confirmed that the RMM+ plugin in SC is up to date and configured with the access key.

I did just update to 2020.1 but I don't think thats likely to be the problem.

User avatar
Cubert
Posts: 1536
Joined: Tue Dec 29, 2015 7:57 pm
Contact:

Re: Habitat Stalled Agents Tool

Post by Cubert »

Mark,
Yes, RMM+ must be installed separately by you and configured with passcode.

Yea we are still in full beta which means the docs are yet to be written. The possibility that things change a lot between versions at this point prevents us from completing docs. With that said, as the plugin starts to solidify we will start releasing documentation.

As for your issue.

From the error, well actually not an error but from the response of the ScreenConnect Server said that your agent is offline in the "Connect" Server as well as in the Automate Server.

The stalled agent detector can use "Connect" to restart "Automate" agent or the "Automate" agent to restart the "Connect" services on agent but if both show offline we do not have any options to do anything but try a RPC call to agent from another agent. RPC as most know will work to restart services but only if all remote services have been turned on for windows OS so that one agent can send RPC calls to other agent and permissions and such allow agent to respond. This all must be done in advance of ever needing the tool to send RPC calls.

Now if you believe that the "Connect" offline agent for -xxxxxxxx seconds is not accurate and that the agent is actually available in Connect consoles then that's a different bean. Then the question would be why does the call to Connect say agents offline for -xxxxxx seconds when it is not. That might be a RMM+ issue as we only react to the data given us and that data says agent is offline in Connect.

MarkHodges
Posts: 21
Joined: Tue Jun 04, 2019 6:07 pm

Re: Habitat Stalled Agents Tool

Post by MarkHodges »

ok...so basically once we have habitat installed as a plugin, it will monitor the systems and if they go offline in automate it will auto try and restart them, until they are offline 24+ hours when it will stop trying.

I'd like to see the actual tabs to be agents that are offline in LT but online in SC (regardless of how long they have been on).
and the reverse for the other tab..online in Automate, but offline in SC.

if an agent if offline in both, I don't really call that stalled per se...broken maybe..or just offline completely...
Maybe a tab for offline <30 days instead....

User avatar
Cubert
Posts: 1536
Joined: Tue Dec 29, 2015 7:57 pm
Contact:

Re: Habitat Stalled Agents Tool

Post by Cubert »

The tabs do show that now, its only the automation trying to restart them that stops after 24 hours if I am understanding what you are expecting to see..


Green dots say less than 24 hours and red dots say over 24 hours

GV-OfflineLT.png
GV-OfflineLT.png (21.62 KiB) Viewed 13998 times

MarkHodges
Posts: 21
Joined: Tue Jun 04, 2019 6:07 pm

Re: Habitat Stalled Agents Tool

Post by MarkHodges »

When I look at the offline LT agents tab....I saw 248 agents offline, which is probably accurate for ALL of automate.
However, when I run gavsto's api scripts, there were only about 12 that were offline in automate AND were online in SC...
so, I had to manually scroll through the list of 248 agents and then I was able to restart them using SC successfully.

However that really wasn't all that friendly to do....

User avatar
Cubert
Posts: 1536
Joined: Tue Dec 29, 2015 7:57 pm
Contact:

Re: Habitat Stalled Agents Tool

Post by Cubert »

Ah, I got you. Let me poke around with that and see what we can do to get a better view out there.

MarkHodges
Posts: 21
Joined: Tue Jun 04, 2019 6:07 pm

Re: Habitat Stalled Agents Tool

Post by MarkHodges »

Any update on this?

User avatar
Cubert
Posts: 1536
Joined: Tue Dec 29, 2015 7:57 pm
Contact:

Re: Habitat Stalled Agents Tool

Post by Cubert »

I have it on my todo list so it will get traction as I get back around to that piece. Trying to get documentation out to the masses so they can control whats it the product now.

MarkHodges
Posts: 21
Joined: Tue Jun 04, 2019 6:07 pm

Re: Habitat Stalled Agents Tool

Post by MarkHodges »

what is this "documentation" you speak of?

:)

Post Reply