3rd Party Application Management

Support forums for the Habitat Automate plugin
User avatar
Cubert
Posts: 2430
Joined: Tue Dec 29, 2015 7:57 pm
8
Contact:

Re: 3rd Party Application Management

Post by Cubert »

dbitters wrote: Fri Aug 20, 2021 6:03 pm Can you check to see if the script is published fully or possibly renamed? It auto deleted itself from our server unless it's no longer called "Habitat App Manage Maintenance" or something similar.

Thanks.
Thanks for pointing that out to us. We found the issues and have resolved it in 1.0.0.80 due out in auto update tonight.


This fix should resolve several script issues.

dbitters
Posts: 58
Joined: Tue May 18, 2021 8:11 pm
2

Re: 3rd Party Application Management

Post by dbitters »

Thanks, however, there is still an issue with randomly launching the chocolatey installer so all the installs don't launch and all fail at the same time. Each system still shows attempting the install at the same time and then shows a failure. I have attached a screenshot showing this.
Chocolatey_install_failure.PNG
Chocolatey_install_failure.PNG (61.58 KiB) Viewed 3580 times
Chocolatey_install_failure2.PNG
Chocolatey_install_failure2.PNG (7.73 KiB) Viewed 3580 times

Below is part of the output from the script log:

Installer returned -> [Exception calling "DownloadFile" with "2" argument(s): "The remote server
returned an error: (429) Too Many Requests."
At C:\Windows\LTSvc\Habitat\AppManage\install.ps1:257 char:5
+ (Get-Downloader $url @ProxyConfiguration).DownloadFile($url, $fil ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : WebException

Microsoft.PowerShell.Archive\Expand-Archive : The path
'C:\Windows\TEMP\chocolatey\chocoInstall\chocolatey.zip' either does not exist
or is not a valid file system path.
At C:\Windows\LTSvc\Habitat\AppManage\install.ps1:515 char:5
+ Microsoft.PowerShell.Archive\Expand-Archive -Path $file -Destinat ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (C:\Windows\TEMP...\chocolatey.
zip:String) [Expand-Archive], InvalidOperationException
+ FullyQualifiedErrorId : ArchiveCmdletPathNotFound,Expand-Archive

& : The term
'C:\Windows\TEMP\chocolatey\chocoInstall\tools\chocolateyInstall.ps1' is not
recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify
that the path is correct and try again.
At C:\Windows\LTSvc\Habitat\AppManage\install.ps1:526 char:3
+ & $chocoInstallPS1
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (C:\Windows\TEMP...ateyInstall.p
s1:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org)
Getting latest version of the Chocolatey package for download.
Not using proxy.
Getting Chocolatey from https://community.chocolatey.org/api/v2 ... ey/0.10.15.
Downloading https://community.chocolatey.org/api/v2 ... ey/0.10.15 to C:\Windows\TEMP\chocolatey\chocoInstall\chocolatey.zip
Not using proxy.
Extracting C:\Windows\TEMP\chocolatey\chocoInstall\chocolatey.zip to C:\Windows\TEMP\chocolatey\chocoInstall
Installing Chocolatey on the local machine
Ensuring Chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder]

dbitters
Posts: 58
Joined: Tue May 18, 2021 8:11 pm
2

Re: 3rd Party Application Management

Post by dbitters »

The issue with the applications being unable to upgrade due to too many requests is also still an issue. If I install Chocolatey manually instead of relying on the script, I can test the application installs. However, If I approve 10 applications and we have more than 10 systems, then nothing will update because of hitting the limit. Please see the output below.

Chocolatey v0.10.15
Upgrading the following packages:
gotomeeting
By upgrading you accept licenses for the packages.
gotomeeting is not installed. Installing...
gotomeeting not installed. An error occurred during installation:
The remote server returned an error: (429) Too Many Requests. Too Many Requests
gotomeeting package files upgrade completed. Performing other installation steps.
The upgrade of gotomeeting was NOT successful.
gotomeeting not installed. An error occurred during installation:
The remote server returned an error: (429) Too Many Requests. Too Many Requests

Chocolatey upgraded 0/1 packages. 1 packages failed.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
- gotomeeting (exited 1) - gotomeeting not installed. An error occurred during installation:
The remote server returned an error: (429) Too Many Requests. Too Many Requests

User avatar
Cubert
Posts: 2430
Joined: Tue Dec 29, 2015 7:57 pm
8
Contact:

Re: 3rd Party Application Management

Post by Cubert »

Sorry it took me a day to get back to you on this, I needed time to review current chocolatey standards to see if something has changed.

Unfortunately it may have. Statically you would see some level of deployment taking place as the blocks go applied and released. But it looks like now they are pushing for a more stringent policy.


Pulled from https://docs.chocolatey.org/en-us/commu ... disclaimer

Excessive Use
Please note that individuals (even organizations) using the community repository are unlikely to hit excessive use numbers under normal usage scenarios.

📝 NOTE: If you do find you have been blocked / rate-limited, having commercial licenses will not have any effect the policies with the community package repository. These policies were put into place to ensure stability and availability for the entire community, not to try to get folks to pay for licensing.

See rate limiting below if you are seeing 429 errors (too many requests).

Another aspect to keep in mind is that the community package repository is meant for the community. Perceived abuses of the community package repository that affect it in a detrimental way for the rest of the community will not be allowed. By abusive, it may mean more than 100 installs per hour on average over an internally determined amount of time (it could be more, could be less) - this is not queries, this is installs, upgrades where actual package downloads are occurring. Let's say that is 30 days - that would mean 72,000+ package downloads over 30 days. When that is seen, our community team will make attempts to warn folks if we have known contacts (keep in mind it's highly unlikely we will have your contact information), and implement a temporary block to ensure your usage does not affect the community in a detrimental way. Many times this is due to a misconfiguration and can be corrected quickly.

Blocks are meant to be temporary bans, but require you to act to remedy the situation. If you have been blocked, please see the next sections for corrective actions.

📝 NOTE If you or your organization feels you will need to go over this limit with good reason and need whitelisted, please reach out at https://chocolatey.org/contact, choose "Blocked IP Address". As we have limited information, please include your name, email address, phone number, and the IP addresses you believe are blocked so we can contact you and verify if there is a ban. Once you have resolved any issues on your side, we can lift the ban.



Previously this was an auto lifted temp band for one hour but now looks to be a hard band till resolution.

We also maybe hitting their attempts force the sale of Chocolatey licenses for business/MSP. Next option maybe licensing Chocolatey for business or MSP.

The above notice speaks of excessive as
72,000+ package downloads over 30 days
which I believe is on the high side for them but this should not be a concern with client locations with only a handful of agents. The problem will be with client locations with lots of agents all making requests and adding to the count. Compound that daily and you could see several thousand requests a week going out even to just to check version availability.

What this means to Habitat is that we will need to go back to the drawing board and redesign how we manage packages installs. Build some type of cache area for each location and force Chocolatey to look there to acquire package data. Do some fancy coding to create a limited stream of requests going out to the community repo. Just brain storming here but maybe a master agent at each location makes calls to community repo and holds that version information and choco installer nuspec/nupkg files so that it stores data to preform local installs for all other agents.

dbitters
Posts: 58
Joined: Tue May 18, 2021 8:11 pm
2

Re: 3rd Party Application Management

Post by dbitters »

Thanks for the consideration, I have a few thoughts to brainstorm on.

I still think part of the issue is that it may not be staggering the chocolatey framework installs or the package installs as intended. I could be wrong, but the timestamps all look the same unless that is just when the individual counters start. Like I said, if I manually install the framework in patches using a script, that still works fine, but when they kick off via the Habitat script, they all seem to happen at the same time causing the failure. The same issue is with the package installs, but that might be also caused by doing checks and then a delay that I'm not seeing.

On the dozen customers I am testing this on with 15-50 systems per customer, I only have it set to check on Sundays or Mondays and never daily. A small handful of around 6 systems or less per customer may appear to check in once a week, and perform a few upgrades, but it's rare, so I know there is no monthly block happening, as I would then see none. Also, each customer has completely different public IP's/locations, so there is no overlap of limits here. Ideally, we'd like this to work on any customer with a system count of 15 to 500 per location.

In addition to the staggering, I think one of the problems is packages such as adobe acrobat where the version numbers on the package are always misrepresented, so it tries to upgrade at every check. There are a few other software packages that are the same. It would be nice, if we had a way to "adjust" the version at the plugin level, for decimal places or filler zeros to be accommodated instead of thinking it's a different version. That may help cut down on the total number of requests, but I think if we ensured that the installs and upgrades are truly being staggered that we won't have this problem.

Maybe there would be a way to track at the company or site level, the maximum observed requests to the chocolatey servers per min, and the total requests per day and per month. If you did it at the site level, you could have a Habitat tab similar to your other plugins that allowed for an off or on function and then provide the tracking details or usage. That might help determine when an MSP is approaching the license limits and to help with troubleshooting.

So these are just some thoughts to help, but thanks for keeping at it, as MSP such as us really need this kind of solution for compliance reasons, so I really appreciate the work on this project.

dbitters
Posts: 58
Joined: Tue May 18, 2021 8:11 pm
2

Re: 3rd Party Application Management

Post by dbitters »

By the way, I do like the idea of a master Agent and cache area which would overall help keep all the connections to a minimum and reducing these types of problems going forward.

By the way, there is a minor bug within the snail button. I mean it's not a big deal, as it's just the display lookup, but when you switch the time frame to monthly, it says "set to monthly updates on Monday day of month". If you change the number to 2, then it says Tuesday, which I'm assuming it instead should be pulling the numerical value.

User avatar
Cubert
Posts: 2430
Joined: Tue Dec 29, 2015 7:57 pm
8
Contact:

Re: 3rd Party Application Management

Post by Cubert »

Thanks for the feedback, I will look at that control to see what's up.

I have started actively working this issue moving forward with a new caching service. Look for updates on it soon.

User avatar
Cubert
Posts: 2430
Joined: Tue Dec 29, 2015 7:57 pm
8
Contact:

Re: 3rd Party Application Management

Post by Cubert »

We are redesigning how the Application Manager (Chocolatey) works under the hood. Feel free to read our current post below on how its going to work moving forward. If you have any comments or suggestions this would be a good time to voice them here so we can consider them as we complete development.

Here's our problem:

It the original manager scheduling was a little dodgy and clients defaulted to testing updates several times a day causing hundreds if not thousands of HTTPS requests to Chocolatey.org each day of the month. This causes lockouts and blocks from Chocolatey,org and basically stops a location from updating or installing applications or chocolatey.


We can not predict the number of agents a MSP will have at any one client location, so we must assume enough to cause lock outs from the Chocolatey.org repos within the first day of operation. Well that's just not gonna cut it as we quickly find out. Even with tinkering with the schedules was not enough for larger client locations.

For us to scale we have to do things differently, a redesign is a must!


The Fix:

In comes the Reaper!
We will reap the cache from the Chocolatey.org repository for the applications used at a given location. Basically creating a local repository for each client location. Then the agents can go hog wild without triggering a lockout. Now just one agent ever talks to Chocolatey.org reducing traffic (X) fold.

Habitats Application Manager is going 2.0, If you do not want to go to 2.0 do not upgrade Habitat above 1.0.0.80.

It looks and operates pretty much the same but there are a whole new way of deploying and updating Applications at each location. We have also improved a few other items to make the manager easy to use.

So how do we manage installs and updates at the location level?

We now enforce the use of location drives along with valid usernames and passwords to allow each agent at a location the ability to access the location cache for all Chocolatey installs and updates. You assign one agent at each location (server or workstation) the role of caching agent inside the plugin. This agent will be the only agent communicating with chocolatey.org and its role will be to manage the location cache that all other agents will use to install and update applications. We will also provide a way for off network agents to get their updates and installs.

The caching agent(s) uses our Cache Reaper program to capture the files needed to deploy approved (Enabled) packages and then stores them to the Location drive. All other agents are instructed to use this resource to manage their local source for chocolatey installs and update. The caching agents are responsible for looking for package version changes so you can see when versioning has changed.

This is showing at our largest location over a 90% reduction in chocolatey.org traffic. We are running tests now and checking for bugs. Looking to release first edition in Habitat 1.0.0.81 sometime this month (September 2021).

Screen shots and how-to documentation to come shortly. I will make an announcement here..

User avatar
Cubert
Posts: 2430
Joined: Tue Dec 29, 2015 7:57 pm
8
Contact:

Re: 3rd Party Application Management

Post by Cubert »

We just pre released the new Habitat 1.0.1.1 build

viewtopic.php?f=62&t=5889&sid=157e70496 ... 03a106fe57

IT has our first full update to the 3rd party app manager, now just Application Manager. We still have a lot of work to do with documentation so please be patent as I update all the docs with the changes.

Application Manager now works completely different from before in the backend of things. You will need to setup each clients location with a cache location (drive) to support the Application deployments. Documentation will follow over the next several days.

Feel free to download and try the new Habitat build. Please report back on the builds forum post how its going for you and any thing you notice that may need attention.

Thanks for all the input!

dbitters
Posts: 58
Joined: Tue May 18, 2021 8:11 pm
2

Re: 3rd Party Application Management

Post by dbitters »

moved
Last edited by dbitters on Tue Oct 12, 2021 3:52 am, edited 1 time in total.

Post Reply

Return to “Habitat”