Deploy a Soda Agent in an Azure AKS cluster
Last updated
Was this helpful?
Last updated
Was this helpful?
You have an Azure account and the necessary permissions to enable you to create, or gain access to an existing AKS cluster in your region. Consult the for details.
You have installed the . This is the command-line tool you need to access your Azure account from the command-line. Run az --version to check the version of an existing install. Consult the for details.
You have logged in to your Azure account. Run az login to open a browser and log in to your account.
You have installed v1.22 or v1.23 of . This is the command-line tool you use to run commands against Kubernetes clusters. If you have already installed the Azure CLI tool, you can install kubectl using the following command: az aks install-cli.
Run kubectl version --output=yaml to check the version of an existing install.
You have installed . This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run helm version to check the version of an existing install.
Kubernetes cluster size and capacity: 2 CPU and 2GB of RAM. In general, this is sufficient to run up to six scans in parallel.
Scan performance may vary according to the workload, or the number of scans running in parallel. To improve performance for larger workloads, consider fine-tuning the cluster size using the resources parameter for the agent-orchestrator and soda.scanlauncher.resources for the scan-launcher. Adding more resources to the scan-launcher can improve scan times by as much as 30%. Be aware that allocating too many resources may be costly relative to the small benefit of improved scan times.
To specify resources, add the following parameters to your values.yml file during deployment. Refer to Kubernetes documentation for for information on values to supply for x.
For reference, a Soda-hosted agent specifies resources as follows:
The following table outlines the ways you can install the Helm chart to deploy a Soda Agent in your cluster.
Install the Helm chart via CLI by providing values directly in the install command.
Use this as a straight-forward way of deploying an agent on a cluster.
Install the Helm chart via CLI by providing values in a values YAML file.
Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart.
Use Helm to add the Soda Agent Helm chart repository.
Replace the value of soda.agent.name with a custom name for your agent, if you wish.
Specify the value for soda.cloud.endpoint according to your local region: https://cloud.us.soda.io for the United States, or https://cloud.soda.io for all else.
(Optional) Specify the format for log output: raw for plain text, or json for JSON format.
(Optional) Specify the level of log information you wish to see when deploying the agent: ERROR, WARN, INFO, DEBUG, or TRACE.
The command-line produces output like the following message:
(Optional) Validate the Soda Agent deployment by running the following command:
In your Soda Cloud account, navigate to your avatar > Agents. Refresh the page to verify that you see the agent you just created in the list of Agents. Be aware that this may take several minutes to appear in your list of Soda Agents.
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart.
Use Helm to add the Soda Agent Helm chart repository.
Using a code editor, create a new YAML file called values.yml.
To that file, copy+paste the content below, replacing the following values:
Replace the value of name with a custom name for your agent, if you wish.
Specify the value for endpoint according to your local region: https://cloud.us.soda.io for the United States, or https://cloud.soda.io for all else.
(Optional) Specify the format for log output: raw for plain text, or json for JSON format.
(Optional) Specify the level of log information you wish to see when deploying the agent: ERROR, WARN, INFO, DEBUG, or TRACE.
Save the file. Then, create a namespace for the agent.
In the same directory in which the values.yml file exists, use the following command to install the Soda Agent helm chart.
(Optional) Validate the Soda Agent deployment by running the following command:
In your Soda Cloud account, navigate to your avatar > Agents. Refresh the page to verify that you see the agent you just created in the list of Agents.
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
helm install commandhelm install
the action helm is to take
soda-agent (the first one)
a release named soda-agent on your cluster
soda-agent (the second one)
the name of the helm repo you installed
soda-agent (the third one)
the name of the helm chart that is the Soda Agent
The --set options either override or set some of the values defined in and used by the Helm chart. You can override these values with the --set files as this command does, or you can specify the override values using a values.yml file.
--set soda.agent.name
A unique name for your Soda Agent. Choose any name you wish, as long as it is unique in your Soda Cloud account.
--set soda.apikey.id
With the apikey.secret, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here.
--set soda.apikey.secret
With the apikey.id, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here.
--set soda.agent.logFormat
(Optional) Specify the format for log output: raw for plain text, or json for JSON format.
--set soda.agent.loglevel
(Optional) Specify the leve of log information you wish to see when deploying the agent: ERROR, WARN, INFO, DEBUG, or TRACE.
--namespace soda-agent
Use the namespace value to identify the namespace in which to deploy the agent.
Delete everything in the namespace which you created for the Soda Agent.
Delete the cluster. Be patient; this task may take some time to complete.
Problem: After setting up a cluster and deploying the agent, you are unable to see the agent running in Soda Cloud.
Solution: The value you specify for the soda-cloud-enpoint must correspond with the region you selected when you signed up for a Soda Cloud account:
Usehttps://cloud.us.soda.io for the United States
Use https://cloud.soda.io for all else
Problem: You need to define the outgoing port and IP address with which a self-hosted Soda Agent can communicate with Soda Cloud. Soda Agent does not require setting any inbound rules as it only polls Soda Cloud looking for instruction, which requires only outbound communication. When Soda Cloud must deliver instructions, the Soda Agent opens a bidirectional channel.
Solution: Use port 443 and passlist the fully-qualified domain names for Soda Cloud:
cloud.us.soda.io for Soda Cloud account created in the US region
OR
cloud.soda.io for Soda Cloud account created in the EU region
AND
collect.soda.io
Problem: When you attempt to create a cluster, you get an error that reads, An RSA key file or key value must be supplied to SSH Key Value. You can use --generate-ssh-keys to let CLI generate one for you.
Solution: Run the same command to create a cluster but include an extra line at the end to generate RSA keys.
Use this as a way of deploying an agent on a cluster while keeping sensitive values secure. - provide sensitive API key values in this local file or in an external secrets manager - store data source login credentials as environment variables in this local file; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See:
(Optional) You have familiarized yourself with .
Use the following command to install the Helm chart which deploys a Soda Agent in your cluster. (Learn more about the .)
Replace the values of soda.apikey.id and soda-apikey.secret with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud. By default, Soda uses as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets.
(Optional) You have familiarized yourself with .
id and secret with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account. By default, Soda uses as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets.
