Support Forum

I am writing about the Habitat 1.0.0.73 build, specifically the 3rd party patching component. I think this product is great with some very useful features. However, I have a few questions and observations that affect its ability to be used in a production environment. I realize that this component utilizes the Chocolatey framework, and that may be where some of the limitations reside. I have a couple of suggestions or feature requests to help overcome those issues.

1. Habitat requires that the Chocolatey framework be installed before it can function properly. That's all good, but there are a few limitations and setbacks with that. First, the framework can only be downloaded in very small increments. We have found that if you are running the habitat script or the Chocolatey framework on more than five systems per minute, then you will be locked out from the Chocolatey site. Downloading the script and all components and launching it from your own Labtech server will fix that. Chocolatey does have instructions on how to set up your own server to provide the chocolatey framework installation, but that is a bit of an inconvenience and requires maintaining that PowerShell code and some prerequisites. If there were a way to build in a limiter of 5 per minute into the application, that would resolve that issue without having to jump through these hoops. Secondly, the scripted download of the framework that is built into the Habitat main script only works on certain types of Windows installs or only after security is partially disabled on those systems. It does download on some systems, mainly only on workstations with security lowered, but we have found that the PowerShell download and execution on line 16 of the Habitat App Manage Maintenance Script does not work on over 90% of our systems which are all built with default security left enabled. We have found line 16 does not work on any of our newer servers or virtual desktops. We have corrected that setback by using the following script on systems which is nothing more than downloading first and then launching the PowerShell differently. It's nothing fancy and could use some polishing, but it was a quick down and dirty method for us to move things along before we could even use Habitat. It still must only be deployed to no more than five systems per minute with a 5-minute break in between each, or just only do one system per minute. Also, if you hit the rate limit, the execution would stall, and you would have to use a method to delete the folder and start over, which we dealt with in line 14 below, or you will be stuck in a loop of the framework being partially installed. 2. The second setback we noticed is if you're trying to patch more than a half dozen systems with a half dozen apps to be patched, then they will all fail. This process works fine in a small testing environment, but deploying this to a production network results in immediate failure with only the first system working. This is because there is a rate limit of 20 packages that can be checked per network for upgrade per minute, and then you are locked out for an hour or longer depending on how many systems the Habitat script ran on since the lockout keeps getting extended. We like the Habitat product and feel it has tremendous potential, but for production networks, we were hoping that a rate limiter can be put in place to allow the software to work properly. Otherwise, your forced to disable the script and just run it manually on systems at once per minute to allow it to work. This will allow Habitat to abide by the Chocolately rate terms, so no more than 20 packages are ever called within one minute. Something such as only allowing one system to be patched per minute would allow a slow and steady progression of the patches allowing for all systems to be completed without hitting any rate limit by the Chocolatey servers. When doing initial patching with multiple apps, it will only take a few systems to cause a failure in the whole process, and a limiting feature would allow a slow and steady rollout. Is there a possibility for adding a button to the customer location in the 3rd party patching window to limit the habitat script to one system per minute as a rate limiter? Adding a limiting button would allow both the framework and the packages to fully run on any one system but not proceed to the next system until the next minute. This would allow any framework or packages to finish before moving on to the next, permitting a slow and smooth rollout without failure.

3. Also, is there a way to set the upgrades to only once per week or per month instead of every day? There is already a frequency selection by using the turtle or the rabbit to allow daily or multiple times a day. Could that be expanded to be quarterly per day, daily, weekly, and monthly. There are numerous packages on the chocolatey packages site that have version numbers that are not written correctly. This causes a reinstall of that application every time the script runs. This isn't a huge deal, but if it only happened once a month, then if there is a version error, that is not as bad if it's happening every day. Reinstalling a few applications every day can obviously be a problem, and we don't need every system checking for updates every single day, as once a month or week would be sufficient.

4. These same issues also exist in Chocolatey for Automate and AppGenie, which we also tried. We found Habitat to be more refined, though with more features and additional utilities to be more beneficial. We have also tried Chocolatey for Labtech, which has some of the same issues and more capabilities, but not exactly what we need.

5. In conclusion, we really like the Habitat product. We are impressed with how simple it is to do a search for an application, apply a package to that search, and just set it and walk away and know the applications will be managed and updated. It provides a very easy interface that any member of our support staff can use. Unfortunately, for now, we can only use it in very small testing environments until the above issues are resolved. We realize that we could switch over to Chocolatey for business subscription and host our own repository and our own mechanism and scripts, but that would defeat the purpose of using the Habitat product with an easy-to-use and a set it and forget mentality. We also realize that Automate is releasing its own 3rd party application update solution, but we like the expanded capabilities of Habitat. If the above issues can be considered, I'm sure that customers experiencing those same issues, such as ourselves, will greatly appreciate the implemented suggestions to use the product in production.

DBitters,

That was a very well written and thought out description of issues, Thanks for that.

The development of these plugins has been on mainly Dev environments with several small companies that agreed to be our Gennie pigs. So we sometime don't get to see the effects of a function in a large environment and have to rely on Larger MSPs to help in the guidance.

We can absolutely add and expand the timing inside the plugin. In fact that maybe no more than a 2 or 3 line edit in our functions. I will be happy to look at adding that to 1.0.0.75 as we just released 1.0.0.74 today so it will need to go into that build at this point.

What I will do is look to add some new logic to the script scheduler function that will group agents by location first then loop through each location scheduling the scripts incrementing the schedule by 1 minute per agent. This way the logic should look like the following:

Client A
-Location1
agent1 = 10:01am
agent2 = 10:02am
agent3 = 10:03am
-Location2
agent4 = 10:01am
agent5 = 10:02am
Client B
-Location3
agent6 = 10:01am
agent7 = 10:02am
agent8 = 10:03am
-Location4
agent9 = 10:01am
agent10 = 10:02am

Assuming all locations are separate networks and external IPs this should hopefully prevent limiting. We can add in a properties variable to allow you the ability to adjust the number of minutes between schedules if you find the default 1 minute stilling failing out.


That should resolve your first issues.

Now lets talk Choco installs.

Sounds like 3 issues:

#1 to many installs at once causes lockouts. This should be fixed with issues #1 new scheduler

# 2 Permission issues : You said you created a script that did it differently and was successful with your permissions sets.

What did you do differently that allowed for this to function for you? I will be happy to add in improved methods to installer script.


#3 Can we add to Rabbit and Turtle: Snail? I don't see why not. I can add a snail configuration that allows you to set either a day of the week or day of the month setting that forces all the scans, installs and updates all in that one day. so all activity needed by plugin happens once a week or once a month based on your settings. I can make this a per client setting to allow you to set different clients to different schedules. I could take it down to the location level for this setting but that might get excessive for a larger MSP to configure. (having to set 500 locations to different frequencies could be more hassle then its worth)

All of these are great ideas for addition, some of it can be done quickly and some will need a little time to work in all the details.

I will get started with the simple stuff for the next build.

Also Just FYI,

What's new in 1.0.0.74 with 3rd Party App Manager!


You now can exclude Apps and Agents from Updates. It will still scan then so you see what versions they have but will not do any installs or updates.

I knew it wouldn't take to long to fix these issues.

I am now testing out in build 1.0.0.75 a new scheduler and delay for App Manager.


At the Client Level you can configure the Snail, The Snail is a optional setting which will further reduce when the scheduled scripts for updates run. Options are once a week on a given day or once a month on a given day. During that day the system will cycle through all agents under client scheduling installs/updates


Default Rabbit or Turtle configuration

Client with Snail setting configured

Snail works with both Rabbit and Turtle. Rabbit and Turtle controls the number of times per day we test and check and Snail controls what days. So rabbit and Snail, Turtle and Snail or just Rabbit or Turtle. All four configuration types are valid.



Now for that pesky limiter chocolatey.org controls. We now added a delay in the scheduling of scripts to agents of a client. This control sets the number of minutes apart we will schedule scripts.


I would set this to (1 or 2 ) minutes first before extending delays beyond that. The delay will allow you to set up to 59 minute delays but if you have more than a few agents under any client then setting delay to 59 would only allow 24 agents in a day to execute (per client) . A client with 100 agents set to a 2 minute delay will require a 200 minute window total to schedule all agents within. That's a little over 3 hours of time for all agents to complete the updates.

We have also updated the script to attempt a second install if first fails. It will use the process of downloading the install.ps1 locally then executing using powershell directly.

This should resolve the installing issues.


I'll let this cook over the weekend and see how its doing on Monday. If all is good and no adverse effects we should be able to let you all have the new build (1.0.0.75) middle of next week.

Thank you for looking into this. Everything you mentioned sounds great. The issue with the choco framework install is that you are calling the execution from the same line that is downloading it. That trips some kind of security issue in server OS's and newer OS's and virtual platforms. If you check out the screenshot in my original post, I included a full working copy of what we use to deploy the framework and it works very well with just some minor logic. The general idea is that if you download it first and then call PowerShell and then point it to the pre-downloaded PS1 from a local location instead of a temporary location, that gets around any Microsoft default security gotchas. I just haven't gotten around to adding any limiting into the script, so we just have to launch it with a scheduler. I'm hoping if you could implement something similar into the master Habitat script then we won't need ours anymore.

Thanks for your time.

Yeap all done,


The script now tries twice to install, first the original way and if it fails then it tries your way before continuing or failing out.

The scheduling has also bee handled. You can now set a delay time in minutes where the script with be scheduled that many minutes apart from the agent scheduled before it (per client). You should now be able to control the flow speed in which agents get their updates and installs. Do not need to add anything into script to limit speed yet. We will look at that after we see how the schedule delay works. We might need to add a sleep function to script during updates if loop goes over 20 so that it doesn't cause 1 agent to lock updates out for next agent coming to do updates.

This is version 1 of this new feature so there maybe some need to tweak it as we go.

So I think we covered everything, now it time to see how well it works. We know by early next week if she is good for release.

So I had a chance to test on a larger client with 100 + agents and did see more failures in time outs. We were updating 20 + apps per agent which caused a trigger at times depending on agent. So the scheduling of agents would not have an effect in this case. We need to have small break between (X) number of calls to chocolatey.org

So I have added to the Automate script in 3 sections (Installs, Updates and Versioning) a "Loop Sleeper"

The Loop Sleeper places a break of 60 seconds in the execution loops for the Install, Update and Versioning functions of the script. Every 5 passes causes a 60 second sleep.

This should slow down and prevent the default timers at Chocolatey.org from firing.

We are still testing functions with the improved sleepers and should see a release very soon.

Is there any auditing in the Habitat toolset? (And if so, how could we see it).
Someone enabled the Workstation and Server checkboxes for 3rd party app management on one of our larger customers on May 21st, and it wreaked havoc with their environment.
Is there any audit log to show who clicked those "Enable Workstation" and "Enable Servers" buttons?
(There's also no save. Just clicking those checkboxes enables it, and it will take *global* settings and push them to a single Client.)
We need a lot more per-Client granularity for this to make this work successfully.

We are open to suggestions on how you would like this tool to operate. Currently we do not log the users activity inside the plugin but that could be a new addition to the plugin. We could create a audit tool that would allow you to enabled it afterwards it would log any activity we had it set to log. Then it would be going through each tool set and adding in these log points to capture when users executed a function like enabling App manager on a client. We will look into this as a new feature request.

As for the enable / disable and Clients starting with a global set of defaults, what type of granularity would you want to see?