AeroLab has a service called agi-monitor
for deployments in GCP and AWS. The monitor listens to events coming from agi
instances and serves the following purposes:
spot-instance-termination
by the cloud provider due to capacity issues, the monitor will recreate the instance as an on-demand
automatically to ensure service continuityagi
to continue running; on AWS this is not an issue as EFS volumes are unlimited in sizeIn AWS, an IAM instance policy must be created first, in order to allow the instance to assume the required role to perform it’s actions. To do that, head to the AWS Web Console, navigate to the IAM
service, and set it up as seen here and here.
The following AWS-managed policies MUST be attached:
--disable-pricing-api
isn’t used at AGI-monitor creation)AeroLab will automatically request to assign the https://www.googleapis.com/auth/compute
scope to the instance. If --disable-pricing-api
is not specified, AeroLab will also assign https://www.googleapis.com/auth/cloud-billing.readonly
.
In order for the AGI monitor to be able to perform sizing or spot-to-ondemand rotation, the AGI instances must be started with extra volumes. On GCP that is --gcp-with-vol
and on AWS that is --aws-with-efs
parameter when creating AGI instances.
The easiest way to deploy, use and have running AGI monitor is by telling the agi create
command to automatically work out if a monitor exists, and, if not, create one.
Below example starts an AGI instance, processing local logs, using EFS and an AGI monitor. Add --aws-spot-instance
or --gcp-spot-instance
to request AGI spot instances.
aerolab agi create --source-local ./logs/ --with-monitor --aws-with-efs
The AGI monitor can be manually deployed, controlled and fine-tuned with aerolab agi monitor
commands. Simply run aerolab agi monitor help
for more information. Likewise, run aerolab agi monitor create help
, and so forth.
The monitor logs can be checked as aerolab client attach -n agimonitor -- journalctl -u agimonitor -f
. The monitor can be normally attached to, and interacted with, as it is simply an AeroLab client machine.
The monitor is deployed as a standard AeroLab client with a client type of agimonitor
. As such, it is visible from aerolab client list
and can be removed with aerolab client destroy -n agimonitor -f
command.