# Introduction
This website contains documentation for [OPS](https://github.com/nanovms/ops), a compilation and orchestration tool for the Nanos unikernel.
Most Unikernels out there are specialized for a high-level language, but Nanos is capable of executing any valid ELF-binary. We provide pre-tested packages for common linux software, including support for interpreted languages to provide a similar Linux like experience.
OPS has packages for [PHP](/ops/examples), [Node](/ops/examples), Ruby, Lua, Perl and much more. Note that you are not required to use a package. OPS is explicitly built to be able to run standalone static binaries such as Go and C.
This is a project that is being actively developed and sponsored by [NanoVMs](https://www.nanovms.com).
#### What are Unikernels?
Unikernels are specialized single process operating systems.
Unikernels dramatically shrink the attack surface and resource footprint of cloud services while providing a much better isolation model. They are machine images that can be run on a hypervisor such as Xen or KVM. Since hypervisors power all public cloud computing infrastructure such as Amazon EC2 and Google Cloud, this lets your services run cheaper, more securely and with finer control than with a full general purpose operating system such as Linux.
Unlike containers that add layers to the stack unikernels go the other way and remove/compress layers of the stack:
**Improved Security**
Unikernels reduce the amount of code deployed, which reduces the attack surface, and results in improved security. They also do not allow you to SSH into them and most importantly they embrace the single process model.
**Note:** This does have implications for some software. See the [FAQ](/ops/other/faq) for more details.
**Small Footprints**
Unikernel images are often orders of magnitude smaller than traditional OS deployments. You can create and deploy sub-megabyte unikernels depending on what you want/need.
**Highly Optimized**
Unikernels can achieve greater performance from their single process nature and greater pairing with the kernel.
**Fast Boot**
Unikernels can boot extremely quickly, with boot times measured in milliseconds if you are running on servers you control.
### Architecture
Unikernels are typically deployed directly to the cloud as native disk images. For example on AWS when you do an 'ops image create' we create a new AWS AMI every single time. That AMI when you spin it up with 'ops instance create' spins up a new ec2 instance. This removes the need for a complex orchestration system that you might be used to using with something like kubernetes.
#### How do I Get Started?
To get started, go to the [Getting Started](/ops/getting_started) section.
# Getting Started
## Installing OPS
```sh
$ curl https://ops.city/get.sh -sSfL | sh
```
If that works you can try this simple hello world example. You don't need Node installed for this as ops will automatically download a Node package for you:
```sh
$ ops pkg load eyberg/node:20.5.0 -p 8083 -f -n -a hi.js -m 256
```
Note: We have a list of pre-made Node packages on the [repo](https://repo.ops.city/) if you are looking for a different version or need a different architecture (eg: arm64 for Macs).
Put this in your hi.js:
```sh
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(8083, "0.0.0.0");
console.log('Server running at http://127.0.0.1:8083/');
```
If the script complains about missing Homebrew on Mac, please install it from and re-run the command above.
On Linux and Mac ensure that you have QEMU version 2.5 or greater installed. The latest QEMU version is 10.2 so if you are using something older - it's really old.
**Debian / Ubuntu (apt-get)**
To install `QEMU`, run
```sh
$ sudo apt-get install qemu-kvm qemu-utils
```
**Fedora (dnf/yum)**
```sh
$ sudo dnf update
$ sudo dnf install qemu-kvm qemu-img
```
Or:
```sh
$ sudo yum update
$ sudo yum install qemu-kvm qemu-img
```
**macOS (brew)**
```sh
$ brew install qemu
```
## Verify the OPS Installation
```sh
$ ops version
```
**Note:** During the installation `$PATH` will be configured, and already opened shell windows may not be updated with these `$PATH` modifications until the console is restarted or the user has logged out and in again.
If running `ops version` in the console fails, try `source ~/.bash_profile` or open a new terminal.
# Basic Usage
## Build and Deploy Nanos Unikernel
For a more thorough set of examples for your preferred language or framework check out our [examples repo](https://github.com/nanovms/ops-examples).
### [Golang](https://golang.org) Example
Create a file called `main.go` with the content below:
```golang
package main
import "fmt"
func main() {
fmt.Println("Hello World!")
}
```
Build it:
```sh
$ GOOS=linux go build main.go
```
Package and deploy:
```sh
$ ops run main
```
For more examples using Golang, visit our [examples repository](https://github.com/nanovms/ops-examples/tree/master/golang).
### [PHP](https://www.php.net/) Example
Create a file called `ex.php` with the content below:
```php
```
Package and deploy:
```sh
$ ops pkg load eyberg/php:7.2.13 -a ex.php
```
### [NodeJS](https://nodejs.org/en/) Example
Create a file called `ex.js` with the content below:
```node
console.log("Hello World!");
```
Package and deploy:
```sh
$ ops pkg load eyberg/node:20.5.0 -a ex.js
```
You should see "Hello World!" printed on your screen and then the ops command exits. For more examples using NodeJS, visit our [examples repository](https://github.com/nanovms/ops-examples/tree/master/javascript).
### Run and Pass an Environment Variable
This can be done via the configuration file but if you want to dynamically inject without having to rely on the configuration file this is the way:
```go
package main
import (
"fmt"
"os"
"strings"
)
func main() {
fmt.Println("FOO:", os.Getenv("FOO"))
fmt.Println("BAR:", os.Getenv("BAR"))
fmt.Println()
for _, e := range os.Environ() {
pair := strings.SplitN(e, "=", 2)
fmt.Println(pair[0])
}
}
```
```sh
$ GOOS=linux go build
$ ops run -e FOO=1 tea
```
#### Load vs Run
While both of these commands are used to execute code, there is a big difference when you would choose to run one rather than the other. For `ops run`, you would use this command to run compiled code (executable machine code). So, for example, any golang code that you may run, you would use this command. Any code that is compiled at runtime, you would use `ops pkg load`. Example of languages where you would use this are NodeJS or PHP. Each supported language for `ops pkg load`, will have a corresponding package. To see a list of available packages, run `ops pkg list`.
### Nightly vs. Main
Right now we have two release channels. If you run ops as is you are tracking the main release channel. There is no update frequency associated with it just whenever we deem significant work has been done and we are not horribly breaking anything.
The other channel you can track is the nightly channel. This is populated every night via the build system with whatever is in master. So if you want bleeding edge you can utilize that.
The magic incantation for tracking this channel is simply to switch the `-n` or `--nightly` flag on.
For example:
```sh
$ ops -n run main
```
If there are already cached images you can use the `--force` or `-f` flag to replace them with the lastest images from a given channel. For example, the following command will both track the nightly channel and replace any cached images with the lastest from that channel:
```sh
$ ops -nf run main
```
You may also specify a recent git commit from the Nanos repo. For instance if you want to try a new feature from a PR you can set the short git commit like via '--nanos-version' like so:
```
ops pkg load eyberg/node:20.5.0 --nanos-version=3c1fb76
```
OPS itself also tracks nightly builds:
You can also check the md5 of any download by using the .md5 suffix:
## Supported Operating Systems
OPS can run on many different Unix based environments.
* [MacOS](https://github.com/nanovms/homebrew-ops)
* Debian
* Ubuntu
* [Arch](https://aur.archlinux.org/packages/ops/)
* Fedora
* Centos
* [Windows](/ops/other/windows)
## Libc
OPS supports whatever libc you wish to utilize. Well over 99% of our users, customers and packages will default to glibc as that is what the vast majority of applications are linked to today. However other alt-libcs such as MUSL are supported. They come with the same/similar caveats that one might find when utilizing them on Linux.
# Configuration
The configuration file used by `ops` (such as one passed as a parameter, e.g. `ops --config myconfig.json`) specifies various options and attributes of code execution, such as files to include, arguments, and environment variables. This file follows the standard json format.
For a complete sample of a configuration, see our [sample](#sample).
You can also enable config interpolation for things like secrets, hosts or other configuration that you would like to inject from your environment. This is helpful to reduce duplicate config:
For example:
```sh
secret=test ops_render_config=true ops run -c config.json
```
Will populate 'secret' with 'test':
```json
{
"Env": {
"secret": "$secret"
}
}
```
## Configuration Attributes
### OPS\_HOME
The OPS\_HOME environment variable points to the location of where ops will store build images and releases locally. By default it is set to \~/.ops . However there are some use-cases where you might want it set to something else like so:
```bash
OPS_HOME=/opt/ops
```
### Args
`Args` is an array of arguments passed to your program to execute when running the image. Most programs will consider the program name as arg\[0] but not all.
```json
{
"Args": ["--report-on-fatalerror", "ex.js"]
}
```
### DisableArgsCopy
`DisableArgsCopy`, when set to *true*, disables the auto copy of files from host to image when present in args.
```json
{
"Args": ["ex.js"],
"DisableArgsCopy": true
}
```
### BaseVolumeSz
The `BaseVolumeSz` is an optional parameter you can pass to specify the size of the base volume. By default the size is the end of blocks written by TFS.
To specify 100 megabytes:
```json
{
"BaseVolumeSz": "100m"
}
```
To specify 1 gigabyte:
```json
{
"BaseVolumeSz": "1g"
}
```
### Boot
`Boot` sets the path of bootloader file.
```json
{
"Boot": "nanos/output/platform/pc/boot/boot.img"
}
```
### Uefi
`Uefi` indicates whether image should support booting via UEFI. Default is `false`.
```json
{
"Uefi": true
}
```
### UefiBoot
`UefiBoot` sets the path of UEFI bootloader file.
```json
{
"UefiBoot": "nanos/output/platform/pc/boot/bootx64.efi"
}
```
### BuildDir
`BuildDir` path to the temporary directory used during image build.
```json
{
"BuildDir": "/tmp/my_app_build_dir"
}
```
### CloudConfig
The `CloudConfig` configures various attributes about the cloud provider, we want to use with ops.
#### BucketName
`BucketName` specifies the bucket to store the Ops built image artifacts.
```json
{
"CloudConfig": {
"BucketName": "my-bucket"
}
}
```
#### BucketNamespace
`BucketNamespace` is required on uploading files to cloud providers as oci.
```json
{
"CloudConfig": {
"BucketNamespace": "my-namespace"
}
}
```
#### DedicatedHostId
This option is supported on [**AWS**](https://aws.amazon.com/ec2/dedicated-hosts/pricing/#Dedicated_Hosts_Configuration_Table) only. Please keep in mind that, depending on the dedicated host type, only some instance flavors may be available.
`DedicatedHostId` is used to specify the dedicated host id placement on cloud provider.
```json
{
"CloudConfig" : {
"DedicatedHostId": "h-deadbeefdeadbeef1",
"Flavor": "c4.large",
"BucketName":"my-bucket"
}
}
```
#### DomainName
This option is supported on **AWS**, **GCP** and **Vultr**.
`DomainName` is used to update a A record DNS entry with the started instance IP.
```json
{
"CloudConfig": {
"DomainName": "ops.city"
}
}
```
#### StaticIP
This option is only supported on **AWS**, **GCP**.
`StaticIP` by cloud provider to assign a public static IP to a NIC.
```json
{
"CloudConfig": {
"StaticIP": "1.2.3.4"
}
}
```
#### EnableIPv6
This option is only supported on **AWS**, **GCP**.
If `EnableIPv6` is set to true and a VPC is created, the new VPC will have ipv6 support. Otherwise, it doesn't affect the selected VPC ipv6 support.
```json
{
"CloudConfig": {
"EnableIPv6": true
}
}
```
#### Flavor
`Flavor` specifies the machine type used to create an instance. Each cloud provider has different types descriptions.
```json
{
"CloudConfig": {
"Flavor": "t2.micro"
}
}
```
#### ImageType
`ImageType` allows the user to specify an image type (whose possible values are target platform-specific) when creating an image. Can be used to create images for Hyper-V generation 2 instances.
```json
{
"CloudConfig": {
"ImageType": "gen2"
}
}
```
#### ImageName
`ImageName` specifies the image name in the cloud provider.
It might be used by *ops* to set `"IMAGE_NAME"` and `"RADAR_IMAGE_NAME"` environment variables.
```json
{
"CloudConfig": {
"ImageName": "web-server"
}
}
```
#### InstanceProfile
`InstanceProfile` sets up an IAM role for an instance. Currently, this is only used for **AWS** but, in the future, it might be used to set roles for other cloud providers.
```json
{
"CloudConfig": {
"InstanceProfile": "my-iam-rolename"
}
}
```
#### KMS
`KMS` optionally encrypts AMIs if set. 'default' may be used for the default key or a KMS arn may be specified. This is only used for **AWS**.
```json
{
"CloudConfig": {
"KMS": "default"
}
}
```
#### Platform
`Platform` defines the cloud provider to use with the ops CLI.
Currently supported platforms:
* Default `onprem`
* Google Cloud Platform `gcp`
* AWS `aws`
* Vultr `vultr`
* Vmware `vsphere`
* Azure `azure`
* Openstack `openstack`
* Upcloud `upcloud`
* Hyper-v `hyper-v`
* DigitalOcean `do`
* OpenShift `openshift`
* Oracle Cloud Infrastructure `oci`
* Oracle VM VirtualBox `vbox`
* Proxmox VE `proxmox`
See further instructions about the cloud provider in dedicated documentation page.
```json
{
"CloudConfig": {
"Platform": "gcp"
}
}
```
#### ProjectID
`ProjectID` is used in some cloud providers to identify a workspace.
```json
{
"CloudConfig": {
"ProjectID": "proj-1000"
}
}
```
#### RootVolume
`RootVolume` is an optional set of options one can apply to the root/base volume of an instance on AWS. These settings include the name, typeof, iops, size, and throughput.
For more information see the [AWS](/ops/clouds/aws) page.
```json
{
"CloudConfig": {
"RootVolume": {
"Name": "my-root-volume",
"Typeof":"gp3",
"Size": 10
}
}
}
```
#### SecurityGroup
`SecurityGroup` allows an instance to use an existing security group in the cloud provider.
On **AWS**, both the name and the id of the security group may be used as value.
```json
{
"CloudConfig": {
"SecurityGroup": "sg-1000"
}
}
```
#### Subnet
`Subnet` allows an instance to use an existing subnet in the cloud provider.
On **AWS**, both the *name* and the *id* of the subnet may be used as value.
```json
{
"CloudConfig": {
"Subnet": "sb-1000"
}
}
```
#### Tags
`Tags` is a list of keys and values to provide more context about an instance or an image. There are a set of pre-defined tags to identify the resources created by `ops`.
The following example demonstrates the default format used when deploying on different cloud providers.
```json
{
"CloudConfig": {
"Tags": [
{
"key": "instance-owner",
"value": "joe-smith"
},
{
"key": "function",
"value": "web-server"
}
]
}
}
```
On **GCP** (*only*) there is also an extended configuration `attribute` that can be used to detail/control the purpose and destination of the tag
```json
{
"CloudConfig": {
"Tags": [
{
"key": "instance-owner",
"value": "joe-smith",
"attribute": {
"image_label": false,
"instance_label": true,
"instance_network": false,
"instance_metadata": false
}
},
{
"key": "function",
"value": "web-server",
"attribute": {
"image_label": true,
"instance_label": true,
"instance_network": false,
"instance_metadata": false
}
},
{
"key": "network-tag",
"value": "http",
"attribute": {
"image_label": false,
"instance_label": false,
"instance_network": true, // will use the tag value
"instance_metadata": false
}
}
]
}
}
```
### UserData
`UserData` allows you to populate data from the IMDS datastore at instance creation time. This allows to create a single image and deliver specific configuration data to each deployed instance.
**Note**: cloud providers have different endpoints for IMDS, even though the data is always served from the 169.254.169.254 address which is *only* reachable from the instance. Since nanos does not have any concept of users or other processes running on an instance this can be used to deliver configuration data safely to an instance.
There is also klib (`userdata_env`) that can set the environment variables on an instance, taking care of the specifics for most cloud providers such as the endpoint to use and whether to base64 decode the data.
```json
{
"CloudConfig": {
"UserData": "BOB=something\nTOM=something-else"
},
"Klibs": ["userdata_env", "tls"]
}
```
Example setting a json configuration fragment as `UserData`, the application must fetch it from the IMDS endpoint and parse it:
```json
{
"CloudConfig":{
"UserData":"{\"Port\":8080,\"Endpoint\":\"/my/api/endpoint\"}"
}
}
```
#### VPC
`VPC` allows instance to use an existing vpc in the cloud provider.
On AWS and DO, both the *name* and the *id* of the vpc may be used as value. On DO setting the *id* (uuid) avoids an extra call to the api at the cost of readability.
```json
{
"CloudConfig": {
"VPC": "vpc-1000"
}
}
```
#### Zone
`Zone` is used in some cloud providers to identify the location where cloud resources are stored.
```json
{
"CloudConfig": {
"Zone": "us-west1-b"
}
}
```
### Dirs
`Dirs` defines an array of directory locations to include into the image.
```json
{
"Dirs": ["myapp/static"]
}
```
*File layout on local host machine:*
```
-myapp
app
-static
-example.html
-stylesheet
-main.css
```
*File layout on VM:*
```
/myapp
app
/static
-example.html
/stylesheet
-main.css
```
### Env
`Env` defines a map of environment variables to specify for the image runtime.
```json
{
"Env": {
"Environment": "development",
"NODE_DEBUG": "*"
}
}
```
### Files
`Files` defines an array of file locations to include into the image.
```json
{
"Files": ["ex.js"]
}
```
### Kernel
`Kernel` sets the path of kernel image file.
```json
{
"Kernel": "nanos/output/platform/pc/bin/kernel.img"
}
```
### KlibDir
`KlibDir` sets the host directory where kernel libs are located.
```json
{
"KlibDir": "nanos/output/klib/bin"
}
```
#### Klibs
Defines a list of klibs to include. For example to run the NTP klib (eg: ntpd):
```json
{
"Klibs":["ntp"]
}
```
### MapDirs
`MapDirs` sets map of a local directory to a different path on the guest VM. For example the below adds all files under `/etc/ssl/certs` on host to `/usr/lib/ssl/certs` on VM.
```json
{
"MapDirs": {"/etc/ssl/certs/*": "/usr/lib/ssl/certs" },
}
```
### Mounts
`Mounts` is used to mount a volume in an instance.
```json
{
"Mounts": {
"vol": "/files"
}
}
```
In the case of `onprem` provider, the *volume* can be a **host directory** (as opposed to a *pre-made volume*) that the guest will mount as a **9P filesystem**.
```json
{
"Mounts": {
"/tmp/files": "/files"
}
}
```
See further instructions about volumes in dedicated [documentation](/ops/volumes) page.
### NameServers
`NameServers` is an array of DNS servers to use for DNS resolution. By default it is Google's `8.8.8.8`.
```json
{
"NameServers": ["10.8.0.1"]
}
```
### NanosVersion
`NanosVersion` sets nanos version to be used on image manifest.
```json
{
"NanosVersion": "nightly"
}
```
### NightlyBuild
`NightlyBuild` flag forces the use of latest dev builds.
```json
{
"NightlyBuild": true
}
```
### NoTrace
`NoTrace` is an array of syscalls to mute tracing for.
```json
{
"NoTrace": ["syscall-here"]
}
```
### Program
`Program` specifies the path of the program to refer to on attach/detach.
```json
{
"Program":"lua_5.2.4/lua"
}
```
### ProgramPath
`ProgramPath` specifies the original path of the program to refer to on attach/detach.
```json
{
"Program":"/ops_apps/lua_5.2.4/lua"
}
```
### RebootOnExit
`RebootOnExit` reboot your application immediately if it crashes (*exit code is not 0*). Is turned off by default, but you can enable it.
```json
{
"RebootOnExit": true
}
```
same as
```json
{
"Debugflags": ["reboot_on_exit"]
}
```
### LocalFilesParentDirectory
`LocalFilesParentDirectory` is the parent directory of the files/directories specified in Files and Dirs. The default value is the directory from where the ops command is running
```json
{
"LocalFilesParentDirectory": "."
}
```
### TargetRoot
`TargetRoot` *TODO*
```json
{
"TargetRoot": "unix"
}
```
### VolumesDir
`VolumesDir` is the directory used to store and fetch volumes.
```json
{
"VolumesDir": ""
}
```
### PackageBaseURL
`PackageBaseURL` provides the URL for downloading the packages.
```json
{
"PackageBaseURL": "https://repo.ops.city/v2/packages"
}
```
### PackageManifestURL
`PackageManifestURL` provides the URL to download the manifest file that stores info about all packages.
```json
{
"PackageManifestURL": "https://repo.ops.city/v2/manifest.json"
}
```
### Version
`Version` ops package version. Like docker if the user doesn't provide version of the image we consider "latest" as the version.
```json
{
"Version": "v18.9.0"
}
```
### Language
`Language` ops package language.
```json
{
"Language": "javascript"
}
```
### Runtime
`Runtime` ops package runtime.
```json
{
"Runtime": "node"
}
```
### Description
`Description` ops package description.
```json
{
"Description": "node.js runtime"
}
```
### TargetConfig
`TargetConfig` provides a limited support to adding cloud provider specific config. This is done as to not overload the other configuration settings.
Currently only the `proxmox` target has support here. For example:
```json
{
"TargetConfig": {
"isoStorageName": "local",
"Arch": "x86_64",
"Machine": "q35",
"Sockets": "1",
"Cores": "1",
"Numa": "0",
"Memory": "512M",
"StorageName": "local-lvm",
"BridgePrefix": "vmbr",
"Onboot": "0",
"Protection": "0"
}
}
```
### RunConfig
The `RunConfig` configures various attributes about the runtime of the ops instance, such as allocated memory and exposed ports.
#### Accel
Defines whether hardware acceleration should be enabled in qemu. This option is enabled by default, but will be disabled when [Debug](#runconfig.debug) is set to true.
```json
{
"RunConfig": {
"Accel": true
}
}
```
#### AtExit
Defines hooks to be ran after instance stops.
```json
{
"RunConfig": {
"AtExit": "echo `date` - START; sh shell_script.sh; echo `date` - DONE"
}
}
```
#### Bridged
Connects the unikernel network interface to a bridge with the name `br0`. The bridge name may be overriden with the property [BridgeName](#runconfig.bridge_name).
```json
{
"RunConfig": {
"Bridged": true
}
}
```
#### BridgeName
Connects the unikernel network interface to a bridge with the name specified. If the bridge does not exist in host machine it is created.
```json
{
"RunConfig": {
"BridgeName": "br1"
}
}
```
#### CanIPForward
Enable IP forwarding when creating an instance. Only **GCP**, **Azure** provider.
```json
{
"RunConfig": {
"CanIPForward": true
}
}
```
#### CPUs
Specifies the number of CPU cores the unikernel is allowed to use.
```json
{
"RunConfig": {
"CPUs": 2
}
}
```
#### GPUs
Specifies the number of GPUs the unikernel is allowed to use. Only **GCP** provider.
```json
{
"RunConfig": {
"GPUs": 1,
"GPUType": "nvidia-tesla-t4"
}
}
```
#### GPUType
Specifies the type of GPU available. Only **GCP** provider.
```json
{
"RunConfig": {
"GPUs": 1,
"GPUType": "nvidia-tesla-t4"
}
}
```
#### Debug
Opens a port in unikernel to allow a connection with the GDB debugger. See further instructions in [Debugging](/ops/other/debugging). If debug is set to true the hardware acceleration [Accel](#runconfig.accel) is disabled.
```json
{
"RunConfig": {
"Debug": true
}
}
```
#### Gateway
Defines the default gateway IP of the network interface.
```json
{
"RunConfig": {
"Gateway": "192.168.1.255"
}
}
```
#### GdbPort
Define the gdb debugger port. It only takes effect if debug is enabled. By default the `GdbPort` is `1234`.
```json
{
"RunConfig": {
"GdbPort": 1234,
"Debug": true
}
}
```
#### Imagename
Sets the name of the image file.
```json
{
"RunConfig": {
"Imagename": "web-server"
}
}
```
#### InstanceName
Sets the name of the instance.
```json
{
"RunConfig": {
"InstanceName": "web-server-instance"
}
}
```
#### InstanceGroup
Sets the group of the instance.
```json
{
"RunConfig": {
"InstanceGroup": "my-autoscale-service"
}
}
```
#### IPAddress
Defines the IP address of the network interface.
```json
{
"RunConfig": {
"IPAddress": "192.168.1.75"
}
}
```
The IP address has to be specified along with the [netmask](#runconfig.netmask) and the [gateway](#runconfig.gateway), so the unikernel can assign the IP address to the network interface.
```json
{
"RunConfig": {
"IPAddress": "192.168.1.75",
"NetMask": "255.255.255.0",
"Gateway": "192.168.1.1"
}
}
```
#### IPv6Address
Defines the static IPv6 address of the network interface.
```json
{
"RunConfig": {
"IPv6Address": "FE80::46F:65FF:FE9C:4861"
}
}
```
#### Memory
Configures the amount of memory to allocated to `qemu`. Default is 128 MiB. Optionally, a suffix of "M" or "G" can be used to signify a value in megabytes or gigabytes respectively.
```json
{
"RunConfig": {
"Memory": "2G"
}
}
```
### Vga
Defines whether to emulate a VGA output device in `qemu`.
```json
{
"RunConfig": {
"Vga": false
}
}
```
#### Mounts
Defines a list of directories paths in the host machine whose data will be copied to the unikernel.
```json
{
"RunConfig": {
"Mounts": ["./files","./assets"]
}
}
```
#### NetMask
Defines the netmask of the network interface.
```json
{
"RunConfig": {
"NetMask": "255.255.255.0"
}
}
```
### Nics
Is a list of pre-configured network cards.Meant to eventually deprecate the existing single-nic configuration. Currently only supported for **Proxmox**
```json
{
"Nics": [
{
"IPAddress": "192.168.1.75",
"NetMask": "255.255.255.0",
"Gateway": "192.168.1.1",
"BridgeName": "br1"
}
]
}
```
#### Background
Starts unikernels in background. You can stop the unikernel using the onprem instances stop command.
```json
{
"RunConfig": {
"Background": true
}
}
```
#### Ports
A list of ports to expose. Alternatively, you can also use `-p` in the command line.
```json
{
"RunConfig": {
"Ports": ["80", "8008"],
}
}
```
#### QMP
Enables the QMP monitor for onprem targets. This must be enabled for various operations such as stats, volumes and rebooting when running against the onprem target.
```json
{
"RunConfig": {
"QMP": true
}
}
```
#### ShowDebug
Enables printing more details about what ops is doing at the moment. Also, enables the printing of warnings and errors.
```json
{
"RunConfig": {
"ShowDebug": true,
}
}
```
#### ShowErrors
Enables printing errors with more details.
```json
{
"RunConfig": {
"ShowErrors": true,
}
}
```
#### ShowWarnings
Enables printing warnings details.
```json
{
"RunConfig": {
"ShowWarnings": true,
}
}
```
#### TapName
Connects the unikernel to a network interface with the name specified. If the `tap` does not exist in host machine it is created.
```json
{
"RunConfig": {
"TapName": "tap0"
}
}
```
#### UDPPorts
Opens ports that use UDP protocol.
```json
{
"RunConfig": {
"UDPPorts": ["60", "70-80", "6006"],
}
}
```
#### Verbose
Enables verbose logging for the runtime environment. As of now, it prints the command used to start `qemu`.
```json
{
"RunConfig": {
"Verbose": true
}
}
```
#### VolumeSizeInGb
This property is only used by cloud provider **openstack** and sets the instance volume size. Default size is 1 GB.
```json
{
"RunConfig": {
"VolumeSizeInGb": 2
}
}
```
#### AttachVolumeOnInstanceCreate
This property is only used by cloud provider **gcp** to attach an existing cloud disk/volume to the instance being created.
```json
{
"RunConfig": {
"AttachVolumeOnInstanceCreate": true
}
}
```
### ManifestPassthrough
There is the concept of the [manifest](https://nanos.org/thebook#manifest) in Nanos where there exists many other config options that don't boil up to ops configuration, however, sometimes you still wish to pass these settings down to the manifest. Certain klibs, in particular, have variables that need to be set. To set them do this:
```json
{
"ManifestPassthrough": {
"my_manifest_setting": "some_value"
}
}
```
#### Consoles
Nanos offers a 'consoles' feature that allows multiple methods for monitoring a running unikernel. By default, the 'serial' and 'vga' consoles are enabled for testing purposes. However, in a production environment, it is highly advised against using these consoles as they significantly slow down the unikernel, rendering it nearly non-functional.
To activate or deactivate consoles, you can use the '+' or '-' symbols. Here is an example configuration:
```json
"ManifestPassthrough": {
"consoles": [
"+net",
"-serial",
"-vga"
]
}
```
In the above example, `+net` enables the net console locally for debugging purposes. The net console is faster than 'serial' when it comes to testing performance.
The net console also provides two options: `netconsole_port` and `netconsole_ip`. These options allow you to send logs to a remote machine for debugging. However, it is advisable to use 'syslog' instead of net console for production workloads.
#### Exec Protection
Nanos has an 'exec protection' feature that prevents the kernel from executing any code outside the main executable and other 'trusted' files explicitly marked. The program is further limited from modifying the executable file and creating new ones. This flag may also be used on individual files within the children tuple. This prevents the application from exec-mapping anything that is not explicitly mapped as executable.
This is not on by default, however, as many JITs won't work with it turned on.
```json
{
"ManifestPassthrough": {
"exec_protection": "t"
}
}
```
#### CWD
Some applications expect to have a working directory in a different place than where they have been placed. You can adjust this via the manifest variable 'cwd':
```json
{
"ManifestPassthrough": {
"cwd": "/my_new/path"
}
}
```
#### Exec Wait For IPv4
This is an optional configuration setting that allows Nanos to wait for a valid ipv4 address to become available via DHCP using the timeout of 'exec\_wait\_for\_ip4\_secs'. If static ip is set than there is no effect. This configuration is not on by default.
```json
{
"ManifestPassthrough": {
"exec_wait_for_ip4_secs": "5"
}
}
```
This configuration can also be associated to a specific network interface, i.e:
```json
{
"ManifestPassthrough": {
"en2": {
"exec_wait_for_ip4_secs": "5"
}
}
}
```
#### Exec Wait For IPv6
This is an optional configuration setting that allows Nanos to wait for a valid ipv6 address to become available via DHCP using the timeout of 'exec\_wait\_for\_ip6\_secs'. If static ip is set than there is no effect. This configuration is not on by default.
```json
{
"ManifestPassthrough": {
"exec_wait_for_ip6_secs": "5"
}
}
```
This configuration can also be associated to a specific network interface, i.e:
```json
{
"ManifestPassthrough": {
"en2": {
"exec_wait_for_ip6_secs": "5"
}
}
}
```
#### expected\_exit\_code
This is an optional configuration setting that changes the program *exit code* to *0*, if a match is found on the configuration.
Some possible configurations:
```json
{
"ManifestPassthrough": {
"expected_exit_code": "1"
}
}
```
```json
{
"ManifestPassthrough": {
"expected_exit_code": "!6"
}
}
```
```json
{
"ManifestPassthrough": {
"expected_exit_code": ["1", "6"]
}
}
```
```json
{
"ManifestPassthrough": {
"expected_exit_code": "*"
}
}
```
#### Mmap Min Address
This is an optional configuration setting that defines the minimum virtual address that a process is allowed to mmap. If set to zero, allow zero-page mappings to occur.
```json
{
"ManifestPassthrough": {
"mmap_min_addr": "0"
}
}
```
#### Program Overwrite Protection
By default, the user application is not allowed to overwrite the **program** binary file (and **interpreter**, if present). This default behavior can be overridden by inserting a `program_overwrite` attribute in the root tuple of the manifest.
```json
{
"ManifestPassthrough": {
"program_overwrite": "t"
}
}
```
#### ltrace
This enables tracing calls made by the application binary to dynamic library functions. It works on both *pie* and *no-pie* programs, and also works with *aslr*.
```json
{
"ManifestPassthrough": {
"ltrace": {}
}
}
```
#### log\_level
Setting a different log level will print to the console if the level is equal to or lower than the user-provided setting.
The default log level is 'err'. You may adjust this to 'err', 'warn', or 'info'.
```json
{
"ManifestPassthrough": {
"log_level": "info"
}
}
```
#### mtu
This is an optional configuration setting that changes the *MTU* (Maximum Transmission Unit) size of the network interface(s). This **doesn't** affect *TUN* interface(s).
```json
{
"ManifestPassthrough": {
"mtu": "1420"
}
}
```
#### so\_rcvbuf
This is an optional configuration setting used to manage the size (in bytes) of the socket receive buffer. The default buffer size is *208 KB*, so to change the size to, say, *512 KB*, you could use the following config:
```json
{
"ManifestPassthrough": {
"so_rcvbuf": "524288"
}
}
```
#### static\_map\_program
This is an optional configuration setting that will disable demand paging. This is overriden when ltrace is in use.
```json
{
"ManifestPassthrough": {
"static_map_program": "t"
}
}
```
#### trace
This is an optional configuration setting that allows specifying trace flags (i.e. a comma-delimited set of trace message types) in the `trace` symbol of the root tuple. *A given trace message is output only if its message type is enabled in the trace flags.*
* `pf` - page-fault-related messages - sets `TRACE_PAGE_FAULT`
* `threadrun` - messages output when returning to user threads - sets `TRACE_THREAD_RUN`
* `all` - all tracing messages are enabled
* for backward compatibility, **any unknown trace flag** *enables all messages output via calls to thread\_log()*. - sets `TRACE_OTHER`
*Example*: a `"trace:pf,other"` value in the manifest enables page-fault-related messages and messages classified as "other".
```json
{
"ManifestPassthrough": {
"trace:pf,threadrun,all": {}
}
}
```
#### reboot\_on\_exit
This is an optional configuration setting that reboots your application immediately if it stops with an *exit code that matches the configuration*.
This is checked after `manifestpassthrough.expected_exit_code` execution (when both configured).
**Note**: `poweroff/stop` operations work normally, regardless of the presence and value of the `reboot_on_exit` option, and regardless of the application exit code.
Some possible configurations:
```json
{
"ManifestPassthrough": {
"reboot_on_exit": "!0"
}
}
```
```json
{
"ManifestPassthrough": {
"reboot_on_exit": "1"
}
}
```
```json
{
"ManifestPassthrough": {
"reboot_on_exit": ["0", "1", "6"]
}
}
```
```json
{
"ManifestPassthrough": {
"reboot_on_exit": "*"
}
}
```
#### idle\_on\_exit
This is an optional configuration setting that keeps a VM running after the user program exits with an *exit code that matches the configuration*. The VCPUs remain halted.
Some possible configurations:
```json
{
"ManifestPassthrough": {
"idle_on_exit": "0"
}
}
```
```json
{
"ManifestPassthrough": {
"idle_on_exit": "!0"
}
}
```
```json
{
"ManifestPassthrough": {
"idle_on_exit": ["0", "1", "6"]
}
}
```
```json
{
"ManifestPassthrough": {
"idle_on_exit": "*"
}
}
```
#### io-queues
By default the virtio-net driver utilizes as many tx/rx queues as supported by the attached device but limited to the number of cpus exposed to the instance. This setting may be changed via the following config for the guest:
```
"ManifestPassthrough": {
"en1": {
"io-queues": "2"
}
}
```
#### uname
This is an optional configuration that will make the kernel return your chosen strings in the sysname and release fields of the uname syscall. Several programs expect known names or versions, to for example, enable certain features.
The default is set to "Nanos" for the sysname field, and "5.0-\" for the release.
```json
"ManifestPassthrough": {
"uname": {
"sysname": "Linux",
"release": "5.0.5"
}
}
```
#### transparent\_hugepage
Transparent HugePage Support (THP) is *enabled* by default on nanos.
To *disable* THP support set `"transparent_hugepage"` to `"never"`. Any other value, will keep THP *enabled*.
```json
{
"ManifestPassthrough": {
"transparent_hugepage": "never"
}
}
```
### Debugflags
`Debugflags` adds additional debug flags to the runtime.
```json
{
"Debugflags": ["trace:pf,threadrun,all", "debugsyscalls"]
}
```
#### idle\_on\_exit
`idle_on_exit` keeps a VM running after the user program exits successfully (*exit code is 0*). The VCPUs remain halted.
```json
{
"Debugflags": ["idle_on_exit"]
}
```
#### reboot\_on\_exit
`reboot_on_exit` reboot your application immediately if it crashes (*exit code is not 0*). Is turned off by default, but you can enable it.
```json
{
"Debugflags": ["reboot_on_exit"]
}
```
### Force
*TODO*
# Examples
Check out this [list of examples here](#examples).
## Deploying a Static File Server
Lets start with a basic go static file server, copy-and-paste below to `server.go`:
```golang
package main
import (
"log"
"net/http"
)
func main() {
fs := http.FileServer(http.Dir("static"))
http.Handle("/", fs)
log.Println("Listening...on 8080")
http.ListenAndServe(":8080", nil)
}
```
Now build `server.go`:
```sh
$ GOOS=linux go build server.go
```
Prepare the HTML content:
```sh
$ mkdir static
$ cd static
```
Create `hello.html`:
```html
A static page
Hello from a static page
```
Create a config file named `config.json`:
```json
{
"Dirs" : ["static"],
"Files":["/lib/x86_64-linux-gnu/libnss_dns.so.2", "/etc/ssl/certs/ca-certificates.crt"]
}
```
If you are on linux you can use the above example to enable dns/tls, otherwise you can use this for local Mac examples.
**Note:** This is more than likely to change in the very near future.
```json
{
"Dirs" : ["static"]
}
```
The directory structure should look like below after these steps:
```bash
.
├── config.json
├── server
└── static
└── hello.html
```
Package and deploy:
```sh
$ ops run -p 8080 -c config.json server
```
Notice that we are using KVM user-mode networking and have to forward host port 8080 to VM.
Curl it:
```bash
curl http://localhost:8080/hello.html
```
To learn more about various config options visit [OPS GitHub repository](https://github.com/nanovms/ops).
### Examples
More examples can be found from the [`ops-examples` repository](https://github.com/nanovms/ops-examples).
* [ada](https://github.com/nanovms/ops-examples/tree/master/ada)
* [austral](https://github.com/nanovms/ops-examples/tree/master/austral)
* [bog](https://github.com/nanovms/ops-examples/tree/master/bog)
* [c](https://github.com/nanovms/ops-examples/tree/master/c)
* [c3](https://github.com/nanovms/ops-examples/tree/master/c3)
* [clojure](https://github.com/nanovms/ops-examples/tree/master/clojure)
* [cpp](https://github.com/nanovms/ops-examples/tree/master/cpp)
* [crystal](https://github.com/nanovms/ops-examples/tree/master/crystal)
* [cyber](https://github.com/nanovms/ops-examples/tree/master/cyber)
* [d](https://github.com/nanovms/ops-examples/tree/master/d)
* [dada](https://github.com/nanovms/ops-examples/tree/master/dada)
* [dotnet](https://github.com/nanovms/ops-examples/tree/master/dotnet)
* [elixir](https://github.com/nanovms/ops-examples/tree/master/elixir)
* [forth](https://github.com/nanovms/ops-examples/tree/master/forth)
* [golang](https://github.com/nanovms/ops-examples/tree/master/golang)
* [hare](https://github.com/nanovms/ops-examples/tree/master/hare)
* [haskell](https://github.com/nanovms/ops-examples/tree/master/haskell)
* [hvm](https://github.com/nanovms/ops-examples/tree/master/hvm)
* [Idris](https://github.com/nanovms/ops-examples/tree/master/Idris)
* [java](https://github.com/nanovms/ops-examples/tree/master/java)
* [javascript](https://github.com/nanovms/ops-examples/tree/master/javascript)
* [kotlin](https://github.com/nanovms/ops-examples/tree/master/kotlin)
* [lua](https://github.com/nanovms/ops-examples/tree/master/lua)
* [nature](https://github.com/nanovms/ops-examples/tree/master/nature)
* [nelua](https://github.com/nanovms/ops-examples/tree/master/nelua)
* [nginx](https://github.com/nanovms/ops-examples/tree/master/nginx)
* [nim](https://github.com/nanovms/ops-examples/tree/master/nim)
* [ocaml](https://github.com/nanovms/ops-examples/tree/master/ocaml)
* [odin](https://github.com/nanovms/ops-examples/tree/master/odin)
* [onyx](https://github.com/nanovms/ops-examples/tree/master/onyx)
* [pascal](https://github.com/nanovms/ops-examples/tree/master/pascal)
* [perl](https://github.com/nanovms/ops-examples/tree/master/perl)
* [php](https://github.com/nanovms/ops-examples/tree/master/php)
* [picat](https://github.com/nanovms/ops-examples/tree/master/picat)
* [polygon](https://github.com/nanovms/ops-examples/tree/master/polygon)
* [pony](https://github.com/nanovms/ops-examples/tree/master/pony)
* [prolog](https://github.com/nanovms/ops-examples/tree/master/prolog)
* [python](https://github.com/nanovms/ops-examples/tree/master/python)
* [raku](https://github.com/nanovms/ops-examples/tree/master/raku)
* [R](https://github.com/nanovms/ops-examples/tree/master/R)
* [ruby](https://github.com/nanovms/ops-examples/tree/master/ruby)
* [rune](https://github.com/nanovms/ops-examples/tree/master/rune)
* [rust](https://github.com/nanovms/ops-examples/tree/master/rust)
* [scala](https://github.com/nanovms/ops-examples/tree/master/scala)
* [scheme](https://github.com/nanovms/ops-examples/tree/master/scheme)
* [spark](https://github.com/nanovms/ops-examples/tree/master/spark)
* [swift](https://github.com/nanovms/ops-examples/tree/master/swift)
* [tarantool](https://github.com/nanovms/ops-examples/tree/master/tarantool)
* [typescript](https://github.com/nanovms/ops-examples/tree/master/typescript)
* [umka](https://github.com/nanovms/ops-examples/tree/master/umka)
* [wasm](https://github.com/nanovms/ops-examples/tree/master/wasm)
* [yaksha](https://github.com/nanovms/ops-examples/tree/master/yaksha)
* [zig](https://github.com/nanovms/ops-examples/tree/master/zig)
# Crons
Crons are schedules that can run unikernels according to a cron schedule or rate.
As of today crons are only supported on AWS.
To list all crons:
```
ops crons list
```
You need an IAM role that can assume for EventBridge Scheduler and have permissions for ec2. You might also wish to add SQS and Cloudwatch if you need to debug the schedule itself.
To create a new cron from an existing image:
```
export EXECUTIONARN="arn:aws:iam::123456789012:role/cronjobtest"
ops cron create ctest "rate(1 minutes)"
```
To delete a cron:
```
ops cron delete
```
To enable a cron:
```
ops cron enable
```
To disable a cron:
```
ops cron disable
```
# Images
Images (disk images) are the artifacts produced by OPS. These disk images can be ran on a variety of hypervisors including the top public cloud providers such as AWS and Google Cloud.
All local images are stored locally in `~/.ops/images/` and can be viewed like:
`ops image list -t onprem`
OPS has several commands that allow one to work on disk images when not running them. This can help provide debugging support.
### ls
`ops image ls myimage`
Provides a 'ls' style listing of files in a local (on-prem) image from **rootfs** partition.
`ops image ls myimage --bootfs`
Provides a 'ls' style listing of files in a local (on-prem) image from **bootfs** partition.
### cat
`ops image cat g /etc/resolv.conf`
Cats the file out of an image.
### cp
`ops image cp -r redis-server lib .`
Recursively copies the output of the folder '/lib' into the host from the image 'redis-server' **rootfs** partition.
`ops image cp redis-server kernel . --bootfs`
Copies the 'kernel' into the host from the image 'redis-server' **bootfs** partition.
`ops image cp -r redis-server klib .`
Recursively copies the output of the folder '/klib' into the host from the image 'redis-server' **bootfs** partition.
### tree
`ops image tree redis-server`
Displays the tree output of local (on-prem) image from **rootfs** partition.
`ops image tree redis-server --bootfs`
Displays the tree output of local (on-prem) image from **bootfs** partition.
### env
`ops image env redis-server`
Displays the list of environment variables found in a local (on-prem) image.
### mirror
`ops image mirror myimage `
Copies an image from one region to another. This command is for simple images or images that rely on ext. config stores.
# Instances
Instances are virtual machines that are created from disk images.
To list all instances:
```
ops instance list
```
To create a new instance from an existing image:
```
ops instance create -i
```
For instances run locally; as each instance writes to the same image file, each image can only have a single instance running. Attempting to create a second local instance from an image that already has a local instance will fail. Likewise multiple local instances attempting to use the same network port will also fail.
If one desires to use tap networking you can place an instance on a new local ip to have [multiple applications run on the same port](https://nanovms.gitbook.io/ops/networking).
Instances running on a cloud provider do not have this restriction.
To delete an instance:
```
ops instance delete
```
# Volumes
Volumes are one or more drives you can attach to an instance.
## Create Volume
A volume can be created using `ops volume create `.
You can specify the volume size with the flag `-s`. The values should respect the next examples format.
```
* 512 (512 bytes)
* 56k (56 KiloBytes)
* 500m (500 MegaBytes)
* 10g (10 GigaBytes)
```
If you have a directory in your filesystem which you want to access in the unikernel you can use the flag `-d`. Ops will create the volume and copy the directory files to the volume filesystem.
Combining the previous flags a command with the next format.
```sh
$ ops volume create -s -d
```
Someone who needs to create a volume to host 9.6 GigaBytes of assets can use the volume create command the next way.
```sh
$ ops volume create my-assets -s 10g -d ./my-assets
```
You can check the existing ops volumes with `ops volume list`.
```sh
$ ops volume list
+--------------------------------------+------+--------+-----------+---------------------------------------------------------------------+--------------------------------+----------+
| UUID | NAME | STATUS | SIZE (GB) | LOCATION | CREATED | ATTACHED |
+--------------------------------------+------+--------+-----------+---------------------------------------------------------------------+--------------------------------+----------+
| e1a9a40e-8d2b-19b7-61b7-57c43f362aaa | vol | | 1.1 GB | /home/xyz/.ops/volumes/vol:e1a9a40e-8d2b-19b7-61b7-57c43f362aaa.raw | 2021-03-06 09:03:07.21 +0000 | |
| | | | | | GMT | |
+--------------------------------------+------+--------+-----------+---------------------------------------------------------------------+--------------------------------+----------+
```
## Volume Content listing (on-prem)
To display the `tree` output of local (on-prem) volume, use `ops volume tree `
```sh
$ ops volume tree vol # ops volume tree e1a9a40e-8d2b-19b7-61b7-57c43f362aaa
/
| .secret
| db_data
| | IDENTITY
| | 000004.log
| | CURRENT
| | MANIFEST-000005
| | OPTIONS-000055
| | LOG
| | LOCK
| | OPTIONS-000057
```
To display a `ls` style listing of files in a local (on-prem) volume, use `ops volume ls []`
```sh
$ ops volume ls vol -l # ops volume ls e1a9a40e-8d2b-19b7-61b7-57c43f362aaa -l
212 Tue Dec 12 09:17:47 2023 .secret
Tue Dec 12 09:17:47 2023 db_data/
$ ops volume ls vol db_data -l # ops volume ls e1a9a40e-8d2b-19b7-61b7-57c43f362aaa db_data -l
0 Tue Dec 12 09:17:47 2023 LOCK
193 Tue Dec 12 09:17:48 2023 000004.log
1683 Tue Dec 12 09:17:49 2023 MANIFEST-000005
112810 Tue Dec 12 09:17:35 2023 OPTIONS-000055
167464 Tue Dec 12 09:17:47 2023 LOG
36 Tue Dec 12 09:17:47 2023 IDENTITY
16 Tue Dec 12 09:17:49 2023 CURRENT
112810 Tue Dec 12 09:17:35 2023 OPTIONS-000057
```
## Volume Content copy (on-prem)
To recursively copy the content of a local (on-prem) volume, use `ops volume cp ... `
```sh
$ ops volume cp vol / . -r -L # ops volume cp e1a9a40e-8d2b-19b7-61b7-57c43f362aaa / . -r -L
$ ops volume cp vol db_data . -r -L
```
## Volume info from raw files (on-prem)
To get some volume information about `label`/`uuid` from a raw volume file, use `ops volume info `
```sh
$ ops volume info /tmp/snapshot-vol-01.raw
vol:e1a9a40e-8d2b-19b7-61b7-57c43f362aaa
$ ops volume info /tmp/snapshot-vol-01.raw --json
{
"label": "vol",
"uuid": "e1a9a40e-8d2b-19b7-61b7-57c43f362aaa"
}
```
## Setup Mount Path
Mounting a volume in an instance requires an image to know beforehand which volumes it should expect and the directories where the volumes will be mounted. The unikernel recognizes the volume by its name/label, UUID, or virtual id. The virtual id may be used when you wish to have the same volume name or mount a different volume with a different volume name to the same mount point. The virtual id syntax is currently supported on GCP, Azure and AWS.
You can specify mounts details using the command flag or the configuration file.
Using configuration file, you have to set the `Mounts` property.
```json
{
"Mounts": {
"": ""
}
}
// Volume Name Example
{
"Mounts": {
"vol": "/files"
}
}
// Virtual ID Example
{
"Mounts": {
"%1": "/files"
}
}
// VirtFS VirtIO 9p Example
{
"Mounts": {
"/path-on-host-dir": "/files"
}
}
```
And, then you can use the configuration file on creating the image.
```
$ ops image create -c
```
If you want to test quickly you can also use the `mounts` command flag instead.
```sh
$ ops image create --mounts /
// Volume Name Example
$ ops image create webapp --mounts vol:/files
// Virtual ID Example
$ ops image create webapp --mounts %1:/files
// VirtFS VirtIO 9p Example
$ ops image create webapp --mounts /path-on-host-dir:/files
```
The mounts must be specified on creating an image. All commands that require to build an image accept the mounts details.
```sh
$ ops run <...>
$ ops build <...>
$ ops image create <...>
$ ops pkg load <...>
$ ops deploy <...>
```
## Attach
If your image was created with the mounts details specified and you would like to attach a volume to a running instance use next command.
```
$ ops volume attach
```
Locally you can attach volumes at instance boot using --mounts with `ops run`, `ops pkg load`, or `ops instance create`.
If you wish to dynamically attach and detach volumes locally you'll need to create the instance with QMP support enabled:
```
{
"RunConfig": {
"QMP": true
}
}
```
You should note that this opens a local mgmt port that uses the instance id's last 4 digits as the port. (This is only for local hotplug.)
## Detach
You can detach a volume from one instance using `ops detach `
## Read Only
You can set a volume as read-only in your manifest like this:
```
"Mounts": {
"%1": "/files1:ro",
}
```
or if you are passing via the cli flags:
```
--mounts "%1:/files1:ro"
```
If you wish to make the root/base volume readonly you can do so by passing the manifest config like so:
```
"ManifestPassthrough": {
"readonly_rootfs": "true"
}
```
Note: Read-only in this context applies to the actual volume mount not user filesystem access as Nanos doesn't have the notion of users.
## Cloud Providers
All mounting volumes requirements applies to the cloud providers.
Ops uses cloud providers APIs to execute volumes operations on their infrastructure.
The commands format is the same, we just need to add the target cloud provider flag.
```sh
$ ops volume create -t
$ ops image create --mounts : -t
$ ops instance create --instance-name -t
$ ops volume attach -t
```
You can create arbitrarily large empty volumes on providers such as AWS by specifying the requested size. AWS will re-size the volume to whatever you set it as.
Sometimes it might be helpful to know when a volume is attached and detached at run-time as AWS supports attaching/detaching volumes at run-time.
You can look at the pseudo-file of `/proc/mounts` to determine whether or not the volume is available. For instance:
Using this config file:
```
➜ g cat config.json
{
"Mounts": {
"bob": "/bob"
}
}
```
and this program:
```
➜ g cat main.go
package main
import (
"fmt"
"os"
)
func main() {
body, err := os.ReadFile("/proc/mounts")
if err != nil {
fmt.Println(err)
}
fmt.Print(string(body))
}
```
## Hot-Plugging Support
OPS supports hot-plugging volumes on GCP, AWS, Azure, and onprem allowing you to, at runtime, attach and detach volumes and mount/unmount filesystems.
Today there is no support for hot-plugging virtfs shares, however, since it is a share one can simply change the contents on the host.
## Accessing host filesystem from nanos guest using 9p
When running `onprem`, `Ops` contains logic to detect whether a mount directive is for a host directory (as opposed to a pre-made volume), and set up the configuration so that:
1. Qemu properly exports the directory as a virtfs share, and
2. the guest can mount that share as a 9P filesystem.
You can run multiple virt-fs shares locally.
* sample go http server
```go
package main
import (
"net/http"
)
func main() {
fs := http.FileServer(http.Dir("static/"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.ListenAndServe(":8080", nil)
}
```
* build `my_server` binary
```sh
$ go build
```
* create **host** folder to be shared with **guest**
```sh
$ mkdir /tmp/vfs9p
$ touch /tmp/vfs9p/index.html
```
* `config.json` ops config file
```json
{
"Mounts": {
"/tmp/vfs9p": "/static"
},
"RunConfig": {
"Ports": [
"8080"
]
}
}
```
* run `ops` with `config.json` file
```sh
$ ops run -c config.json my_server -f
```
* or run `ops` without using `config.json` file
```sh
$ ops run my_server --mounts /tmp/vfs9p:/static -p 8080 -f
```
* modify content on the host and check that guest has the updates available
```sh
$ echo "1." >> /tmp/vfs9p/index.html
$ curl -L localhost:8080/static/
1.
$ echo "2." >> /tmp/vfs9p/index.html
$ curl -L localhost:8080/static/
1.
2.
```
One can use virtual identifiers on virt-fs shares, to for instance build images on a different host and use them elsewhere:
```
ops image create -t onprem -c imgcreate.json my_server
```
imgcreate.json:
```
{
"Mounts": {
"%1": "/static"
},
"RunConfig": {
"Ports": [
"8080"
]
}
}
```
However, at instance creation time you still need to provide the host mount point using a different configuration:
```
ops instance create -t onprem -c instcreate.json my_server
```
```
➜ vfs cat instcreate.json
{
"Mounts": {
"/tmp/vfs9p": "/static"
}
}
```
* Read more in our [9p tutorial](https://nanovms.com/dev/tutorials/accessing-host-filesystem-from-nanos-guest-using-9p)
## Wrap-up
Mounting volumes require us to create an image with the mounts details, launch an instance using the image and attaching a volume. Locally, you are able to mount a volume with 2 commands.
```sh
$ ops volume create vol -s 1g -d ./files
$ ops run webapp --mounts vol:/assets
```
As mentioned above you can mount multiple volumes with the config.json or pass multiple mounts flags like so:
```
$ ops run g --mounts vol1:/vol1 --mounts vol2:/vol2
```
See an example in our [tutorial](https://nanovms.com/dev/tutorials/working-with-unikernel-volumes-in-nanos).
# Networking
Currently, `ops` supports two networking modes; usermode and bridged. In this document we will dive into the differences between these networking modes and when it makes sense to select one over the other.
## Native Networking
Note that by default you should probably just use the native networking provided by your cloud provider. These instructions only exist if you wish to build your own private cloud.
## Usermode Network
Usermode is the default and simplest mode of networking. It utilizes the ready available network connection and shares it with the ops instance.
It is worth noting that the simplicity of usermode does come at a performance cost. Therefore, while usermode is a good option for demos, testing, and local development; it may not be a good choice for production environments.
No action is required to configure usermode networking, it is used by default.
Usermode does nat on a port but sometimes you'll want to add dns. This is fairly straight-forward to do. Again - this is *only* for dev/test not production deploys:
server:
```go
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to my website!")
})
fs := http.FileServer(http.Dir("static/"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.ListenAndServe(":8080", nil)
}
```
build/run:
```bash
GO111MODULE=off GOOS=linux go build
ops run -p 8080 g
```
client:
```go
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
resp, err := http.Get("http://bob:8080")
if err != nil {
fmt.Println(err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(body))
}
```
You just need to include a /etc/hosts entry for your client to talk to 10.0.2.2:
Sample config:
```bash
➜ z tree
.
├── config.json
├── etc
│ └── hosts
├── main.go
└── z
➜ z cat etc/hosts
10.0.2.2 bob
ops run -c config.json z
```
## Bridged Network
Bridged networking connects a virtual machine to a network using the host computer's Ethernet adapter. For more information about bridged networking, see this [article](https://en.wikipedia.org/wiki/Bridging_%28networking%29).
We only recommend running bridge networking for local dev environments or environments where you are planning on having full control over orchestration. We do not recommend running this on the cloud as it'll be slow. You should instead use the native cloud deploy options (eg: `ops instance create -t gcp`) as they will be using bridges underneath.
### Via Instance Create
If you are on a Mac M1, M2 or M3 we recommend grabbing [OPS desktop](https://storage.googleapis.com/cli/darwin/ops.pkg) as it has vmnet-bridged support enabled by default and is pre-setup to run correctly. This way of working with bridges is much nicer too as you don't have to configure anything. Using this 'ops instance create' for the on-prem target will use bridge networking by default.
If you want to build yourself you need at least qemu 7.1 or qemu HEAD from brew. You currently need to be root, or in a proper group or set the suid bit (chmod +s) to the qemu binary.
On Mac creating a new instance with bridge support is easy as running:
```
ops instance create -p 8080 mywebserver
```
On Linux:
To create a new network and attach a dhcp server to it:
```
ops network create
```
Then you can create a new instance with this sample config:
```
{
"RunConfig": {
"Bridged": true,
"TapName": "tap1",
"BridgeName": "br1"
}
}
```
Then you can run it:
```
ops instance create -c myconfig.json mywebserver -p 8080
```
### Via Ops Run
Ops run currently only supports using a bridge on linux - not a mac. If you are on a mac you should use 'instance create' with the 'onprem' provider. This automatically manages bridges for you.
Here is a simple Go example demonstrating two applications sitting on their respective tap interfaces:
```go
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"time"
)
func main() {
clientIP := os.Args[1]
URL := fmt.Sprintf("http://%s/hello", clientIP)
client := http.Client{
Timeout: time.Duration(5 * time.Second),
}
request, err := http.NewRequest("GET", URL, nil)
if err != nil {
log.Fatal(err)
}
ticker := time.NewTicker(5 * time.Second)
quit := make(chan struct{})
for {
select {
case <-ticker.C:
resp, err := client.Do(request)
if err != nil {
fmt.Println(err)
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
} else {
fmt.Print(string(body))
}
}
case <-quit:
ticker.Stop()
return
}
}
}
```
```go
package main
import (
"fmt"
"net/http"
)
func main() {
hello := func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, "hello test 2\n")
}
headers := func(w http.ResponseWriter, req *http.Request) {
for name, headers := range req.Header {
for _, h := range headers {
fmt.Fprintf(w, "%v: %v\n", name, h)
}
}
}
http.HandleFunc("/hello", hello)
http.HandleFunc("/headers", headers)
http.ListenAndServe(":8081", nil)
}
```
You'll either need the appropriate permissions or you can just give yourself sudo then you can run:
```sh
ops run server -p 8081 -b -t tap0 --ip-address 192.168.42.19
ops run client -a 192.168.42.19:8081 -b -t tap1 --ip-address 192.168.42.20
```
You may also optionally set the bridgeip like so:
```
ops run server -p 8081 -b -t tap0 --ip-address 192.168.42.19 --bridgeipaddress 192.168.42.2
```
Note: On the mac for 'ops run', you won't be able to use this syntax as there is no actual ethernet card (for most laptops).
## IPv6
IPv6 support varies from cloud to cloud. Please consult the individual instructions found for each cloud:
[Google Cloud](/ops/clouds/google_cloud) [AWS](/ops/clouds/aws) [Azure](/ops/clouds/azure)
# Packages
Packages are officially supported images with pre-compiled languages (ie nodejs and php) or applications (ie Redis and Nginx). This makes it easier and more convenient to execute code without needing to compile your own interpreter(s).
You can also create and upload your own public and private packages at as well.
### Listing Packages
To get a list of currently supported packages, run the following command:
```sh
$ ops pkg list
```
The package list can also be searched by simply providing a regular expression to the `list` command's `--search` option (`-s` for short).
```sh
$ ops pkg list --search ^[lr]
+-------------+---------+----------+-------------+
| PACKAGENAME | VERSION | LANGUAGE | DESCRIPTION |
+-------------+---------+----------+-------------+
| ruby_2.3.1 | 2.3.1 | ruby | ruby |
+-------------+---------+----------+-------------+
| lua_5.2.4 | 5.2.4 | lua | lua |
+-------------+---------+----------+-------------+
```
Note: This command will probably go away in the future in favor of search.
### Getting package locally
Package can be downloaded to the local cache `~/.ops/packages` using `ops get` command.
```sh
$ ops pkg get
```
Not sure what the latest package version would be? You can just grab the latest by substituting the version with the ':latest' tag:
```
ops pkg load eyberg/redis:latest -p 6379
```
### Information of Package
Package description provides common assumptions and settings required for running your application with help of package. You can get package description using `ops pkg describe ` command.
```sh
$ ops pkg describe eyberg/nginx:1.15.6
Information for eyberg/nginx:1.15.6 package:
this is a test readme
```
### Contents of Package
If you want see contents of a package, you can use `ops pkg contents ` command.
```sh
$ ops pkg contents eyberg/nginx:1.15.6
File :/README.md
File :/nginx
File :/package.manifest
Dir :/sysroot
Dir :/sysroot/etc
File :/sysroot/etc/group
File :/sysroot/etc/passwd
Dir :/sysroot/lib
Dir :/sysroot/lib/x86_64-linux-gnu
File :/sysroot/lib/x86_64-linux-gnu/libc.so.6
File :/sysroot/lib/x86_64-linux-gnu/libcrypt.so.1
File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2
File :/sysroot/lib/x86_64-linux-gnu/libnss_files.so.2
File :/sysroot/lib/x86_64-linux-gnu/libpcre.so.3
File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0
File :/sysroot/lib/x86_64-linux-gnu/libz.so.1
Dir :/sysroot/lib64
File :/sysroot/lib64/ld-linux-x86-64.so.2
Dir :/sysroot/usr
Dir :/sysroot/usr/local
Dir :/sysroot/usr/local/nginx
Dir :/sysroot/usr/local/nginx/conf
File :/sysroot/usr/local/nginx/conf/nginx.conf
Dir :/sysroot/usr/local/nginx/logs
File :/sysroot/usr/local/nginx/logs/error.log
```
### Removing local packages
By default, downloaded packages are stored in `~/.ops/bin/.packages`. If you'd like to remove a package from your hard drive, whether its to conserve space, or to force a re-download of the package, you can do it here utilizing the `rm` cli tool.
### Creating a Custom or Local Dev packages
If you want to create a packaeg manually you can follow these instructions:
then copy over an un-tarred dir to \~/.ops/local\_packages
Finally you can run it via the local flag:
```
ops pkg load --local mypkg -a b.php -v
```
You can use the --missing-files flag on 'ops run' to hunt down any missing shared libraries that might not be getting loaded from the ldd output. For example with ruby:
```
$ ops run --missing-files /usr/local/bin/ruby
booting /home/eyberg/.ops/images/ruby.img ...
en1: assigned 10.0.2.15
`RubyGems' were not loaded.
`did_you_mean' was not loaded.
missing_files_begin
encdb.so
encdb.so.rb
encdb.so.so
ansi_x3_4_1968.so
ansi_x3_4_1968.so.rb
ansi_x3_4_1968.so.so
rubygems.rb
rubygems.so
did_you_mean.rb
did_you_mean.so
missing_files_end
```
You can also turn on '--trace' to find the locations it might be in. A common idiom would to be run:
```
$ ops run --trace /usr/local/bin/ruby &>/tmp/missing
```
then grep through the output to find loads that failed.
If you wish to create your own local package you can use this as a template with your program being 'test':
```
➜ test_0.0.1 tree
.
├── package.manifest
├── sysroot
└── test
1 directory, 2 files
```
At a minimum your package.manifest should have the following fields:
```
{
"Program":"test_0.0.1/test",
"Args" : ["test"],
"Version":"0.0.1"
}
```
It is important to note that the Program path should be valid. If you wish you may also add Language, Runtime, Description into the manifest so they show up in the search.
These have been traditionally populated from \~/.ops/packages/manifest.json . You may also wish to include a README.md inside the package.
Then we can directly add the package in question as we develop on it:
```
ops pkg add test_0.0.1 --name test_0.0.1
```
We can finally load the package:
```
ops pkg load -l test_0.0.1
```
### Create a Cloud Image from Local Package
Creating a cloud image from your local package is just like creating it from a regular package but you pass the '-l' flag in:
```sh
ops image create -c test.json -l --package c2_0.0.1 -t gcp -i mytest2
```
### Create a New Package from an existing Package
You can use an existing package as a base to create a new package. For instance if you are deploying a node application you might have something that looks like this:
```sh
ops pkg load eyberg/node:v18.9.0 -c config.json -p 8080
```
```
{
"Files": ["hi.js"],
"Args": ["hi.js"]
}
```
You can create a new package from this config like so:
```sh
ops pkg from-pkg eyberg/node:v18.9.0 -c config.json --name mypkg --version 0.0.1
```
and use it:
```
ops pkg load -l mypkg_0.0.1
```
Then you can upload it to :
```
ops pkg push mypkg_0.0.1
```
### Create a Package from Docker
You can create a local package from an existing docker container. If the container does not exist it will download it first and then convert it into a unikernel. Currently this just grabs the binary and libraries necessary to run it versus the entire filesystem that might be present.
```sh
$ ops pkg from-docker node:16.3.0 --file node
$ ops pkg load -l node-16.3.0 -a a.js
booting /Users/qtmsheep/.ops/images/program ...
en1: assigned 10.0.2.15
Hello
exit status 1
```
If you would like to create a package from a local image try this workflow:
Sample Dockerfile:
```
FROM debian
CMD ["/bin/ls"]
```
Note: Right now the build process is dependent upon a non-scratch environment, for instance using ldd and cut.
```
docker build -t bob .
```
```
ops pkg from-docker bob --name bob --file ls
ops pkg load --local bob
```
You might be using a 'distroless' image. In this case you can try something like:
```
ops pkg from-docker gcr.io/distroless/python3-debian12:debug --nodiscover --copy --file python3.11
ops pkg load --local gcr.io/distroless/python3-debian12_debug -a hi.py
```
The nodiscover flag instructs ops not to try and discover libraries via ldd. Instead we copy the entire filesystem over using '--copy', however we still want to know which binary from the container we are interested in running.
### Create a Package from 'ops run' style
You can create a local package in the same style you use `ops run`
```sh
$ ops run myprogram
```
by using `ops pkg from-run`
```sh
$ ops pkg from-run --name mypackage --version v0.1 myprogram
creating new pkg
$
$ ops pkg list --local
+-----------+-------------+---------+----------+---------+--------------------------------+
| NAMESPACE | PACKAGENAME | VERSION | LANGUAGE | RUNTIME | DESCRIPTION |
+-----------+-------------+---------+----------+---------+--------------------------------+
| | mypackage | v0.1 | | | |
+-----------+-------------+---------+----------+---------+--------------------------------+
$ ops pkg load --local mypackage_v0.1
...
```
### Login to the OPS Repo
To upload private and public packages or use your private packages:
You'll need to initially create an account at . For now it just relies on github auth:
```
ops pkg login
```
### Upload a Package
If you have an account from the above step you can login with your apikey:
```
ops pkg login
```
Then, for example, you could create a new node package from docker and then push it up:
```
ops pkg from-docker node:16.3.0 --file node
ops pkg push node-16.3.0
```
### Search
You can search for a pakage on like so:
```
ops pkg search gatsby
```
Then intention is that this will eventually deprecate 'ops pkg list'.
### Debugging
When working with scripting languages you might run into issues where they load libraries at run-time that you aren't aware you need. There are two ways to identify these:
You can turn on --trace with ops to look for it loading libraries:
```
ops run myprogram --trace &>/tmp/out
grep -B 1 "direct return: -" /tmp/out
```
You can use strace and run it normally and look for explicit dlopen calls:
```
strace myprogram 2>&1 | grep -E '^open(at)?\(.*\.so' > /tmp/dlopen
```
Once you find the missing library you can create the proper directory in your package and copy it in or put it into a local directory structure if just using 'ops run'.
Sometimes when building a package you'll need to set some env vars that are being set via a shell script or some nested scripts. Since Nanos doesn't run this you'll want to extract them out. Ltrace comes in handy here and can be ran like so:
```
ltrace -e getenv ./myprogram arg1
```
Another way to get some env vars is to look at proc like so:
```
xargs -n 1 -0 < /proc//environ
```
Sometimes you'll have a binary that was built but not installed on the local linux system and so is linked to some libraries but they're not being found.
For instance here we have a binary with some missing libraries, even though we have them:
```
eyberg@venus:~/kn/knot-dns$ ldd knotd
linux-vdso.so.1 (0x00007ffdc37d6000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x0000749ebf8ee000)
libknot.so.15 => not found
libdnssec.so.9 => not found
libzscanner.so.4 => not found
```
If you don't know where the libraries should be you can patch the binary using patchelf:
```
patchelf --replace-needed libknot.so.15 ./libknot.so.15 knotd
```
Now you'll see that it is finding it:
```
eyberg@venus:~/kn/knot-dns$ ldd knotd
linux-vdso.so.1 (0x00007ffe904a4000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007aae3bb17000)
./libknot.so.15 (0x00007aae3bad1000)
```
Sometimes scripts will change your working directory silently. To detect that on the host try running `pwdx ` and this will show you the CWD of the process - which you can then change in ops by setting .
### Manual Tips:
If you have a large number of libraries to copy you may try something like this:
```
ldd /usr/bin/php8.2 | awk {'print $3'} | xargs cp -t ~/.ops/packages/amd64/eyberg/wordpress_6.6.1/sysroot/usr/lib/x86_64-linux-gnu/
```
Another useful way of doing this is like so:
```
ldd nntpit | grep usr/lib/x86_64-linux-gnu | awk {'print $3'} | xargs -I {} cp {} sysroot/usr/lib/x86_64-linux-gnu/.
```
Note: You may wish to replace any symlinks with copies of those files using something like:
```
cp -L /etc/myfolder/symlinks .
```
# Klibs
Klibs are plugins which provide extra functionality to unikernels.
As of Nanos [9dc963b](https://github.com/nanovms/nanos/commit/9dc963bb2773b7a3dcb39aade3556350babae16b) there are 16 klibs in the kernel source tree:
* **cloud\_init** - used for Azure && env config init
* **cloud\_azure** - implements the Azure VM agent (not available in config, auto-included by **cloud\_init**)
* **acme** - allows automatic tls certificate renewal via ACME
* **azure** - memory metrics for azure
* **cloudwatch** - use to implement cloudwatch agent for AWS
* **aws** - not available in config (auto-included by **cloudwatch**)
* **digitalocean** - used to send memory metrics to Digital Ocean.
* **firewall** - use to implement network firewall
* **gcp** - logging and memory metrics for GCP
* **ntp** - used for clock syncing
* **radar** - telemetry/crash data report using the external [Radar](https://nanovms.com/radar) APM service
* **s3fs** - allows mounting of s3 buckets to the localy filesystem
* **sandbox** - provides OpenBSD-style [pledge](https://man.openbsd.org/pledge.2) and [unveil](https://man.openbsd.org/unveil.2) syscalls
* **shmem** provides shared memory features; specifically memfd\_create functionality; this relies on the tmpfs klib
* **special\_files** provides nanos-specific pseudo-files
* **syslog** - used to ship stdout/stderr to an external syslog - useful if you can't/won't modify code
* **tls** - used for radar/ntp and other klibs that require it
* **tmpfs** - use to provide functionality for mounting a memory backed filesystem at an arbitrary place in the root filesystem
* **tun** - supports tun devices (eg: vpn gateways)
* **umcg** - User-Managed Concurrency Groups (used for user-mode threading)
* **userdata\_env** - User Data
* **test** - a simple test/template klib
Not all of these are available to be included in your config (**cloud\_azure**, **aws**). Only the ones found in your `~/.ops/NANOS-VERSION/klibs` folder can be specified, where `NANOS-VERSION` is the version of nanos you are using, ie:
* **0.1.50** - `~/.ops/0.1.50/klibs`
* **nightly** - `~/.ops/nightly/klibs`
Some of these are auto-included as they provide support that is required by certain clouds/hypervisors. You should only include a klib if you think you need it. The ones that are required for certain functionality will be auto-included by ops. For instance if you are deploying to Azure we'll auto-include the cloud\_init and tls klibs.
## acme
The acme klib allows you to automate your TLS certificate renewals. It'll request a new certificate if there is no existing certificate or if the certificate is near the renewal date (less than 1/3rd life left).
The klib takes several attributes:
**attributes**
* server: the server of which to renew your certificates from
* web\_dir: the directory of which the running webserver can serve static
* files (challenge responses) from.
* email: optional email setting for the ACME server to send renewal reminders to
* reload: optional signal handler that can be invoked to reload the certificate (SIGHUP or SIGUSR1)
* certificates: an array of certificate pairs that should be managed by the acme client
* cert\_chain: public certificate file
* private\_key: the private key for the certificate
* domains: an array of FQDNs for the certificate
```
ops pkg load eyberg/nginx_1.28.3 -c config.json -n -p 8083 -p 4443
```
sample nginx config:
```
"Klibs": ["acme", "tls"],
"ManifestPassthrough": {
"acme": {
"server": "https://acme-v02.api.letsencrypt.org/directory",
"web_dir": "/",
"email": "ddd@ex.com",
"reload": "SIGHUP",
"certificates": [
{
"cert_chain": "/etc/ssl/certs/server.crt",
"private_key": "/etc/ssl/private/server.key",
"domains": ["exampl.com"]
}
]
},
"log_level": "info"
}
```
For live certs (for instance against letsencrypt) you will need to expose the domain publically to prove control.
## Azure
### Overview
The `azure` klib adds functionality for sending memory metrics via an Azure diagnostic extension.
### Key Features
1. **Azure VM Agent**:
* Responds to requests to enable and configure the diagnostic extension, applying settings from the manifest.
2. **Azure Klib**:
* Implements an Azure extension similar to the Linux Diagnostic extension.
* Supports sending **four** types of memory metrics: available and used memory (in bytes and as a percentage of total memory):
* availablememory
* usedmemory
* percentavailablememory
* percentusedmemory
### Configuration
The Azure klib is configured in the manifest options via an `azure` tuple. Diagnostic functionalities are enabled and configured by inserting a `diagnostics` tuple with the following attributes:
* **storage\_account**: The Azure storage account for storing metrics data. Must be in the same region as the Azure instance.
* **storage\_account\_sas**: Shared Access Signature token for accessing the storage account. Must have permissions to create Azure storage tables and add table entities.
* **metrics**: Tuple that enables sending memory metrics. Optional attributes include:
* **sample\_interval**: Interval in seconds for collecting metrics data (default: 15).
* **transfer\_interval**: Interval in seconds for aggregating and sending metrics data to the storage account (default: 60).
### Example Configuration
```json
{
"Klibs": ["azure", "cloud_init"],
"ManifestPassthrough": {
"azure": {
"diagnostics": {
"storage_account": "mystorageaccount",
"storage_account_sas": "sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-05-22T14:50:28Z&st=2024-05-12T06:50:28Z&spr=https&sig=xxyyzz",
"metrics": {
"sample_interval": "15",
"transfer_interval": "60"
}
}
}
}
}
```
For the sas token ensure you've allowed the proper resource types:
You've set a proper expiry:
and you are using the right SAS token:
### Metrics Data
Aggregated memory metrics data include the number of samples, minimum, maximum, last, and average values, and the sum of all values. These data are inserted into an Azure storage table named in the format "WADMetricsxxxxP10DV2Syyyymmdd", where `xxxx` is the transfer interval in ISO8601 format, and `yyyymmdd` represents the 10-day date interval for the metrics.
### Displaying Metrics in Azure Portal
By default, the Azure portal does not display these metrics in its charts. To make metrics available in the portal:
* Enable and configure the Linux Diagnostics Extension in a running instance via the "Diagnostic settings" section in the portal.
* Ensure the storage account and metric aggregation interval in the Azure diagnostic settings match those in the Nanos manifest options.
### Loading the Azure Klib
To load the Azure klib, include it in your Ops configuration file under the `Klibs` section as shown in the example configuration above.
### Notes
* The Azure VM agent in the `cloud_init` klib applies settings from the manifest, not from the extension requests.
## Cloudwatch
The `cloudwatch` klib implements two functionalities that can be used on AWS cloud:
1. `logging` - **console driver** that sends **console output** to **AWS CloudWatch**
2. `metrics` - emulates some functions of AWS CloudWatch agent to send **memory utilization metrics** to **AWS CloudWatch**
### Cloudwatch logging
The **cloudwatch** *klib* implements a console driver that sends log messages to AWS CloudWatch when Nanos runs on an AWS instance. This feature is enabled by loading the cloudwatch and tls klibs and adding a `"logging"` tuple to the `"cloudwatch"` tuple in the root tuple. The `"logging"` tuple may contain the following attributes:
* `"log_group"`: specifies the CloudWatch log group to which log messages should be sent; if not present, the log group is derived from the image name (taken from the environment variables), or from the name of the user program if no IMAGE\_NAME environment variable is present
* `"log_stream"`: specifies the CloudWatch log stream to which log messages should be sent; if not present, the log stream is derived from an instance identifier (e.g. 'ip-172-31-23-224.us-west-1.compute.internal')
The **log group** and the **log stream** are automatically created if not existing.
In order for the cloudwatch klib to retrieve the appropriate credentials needed to communicate with the CloudWatch Logs server, the AWS instance on which it runs must be associated to an IAM role with the `CloudWatchAgentServerPolicy`, which must grant permissions for the following actions, as described in :
* `logs:PutLogEvents`
* `logs:CreateLogGroup`
* `logs:CreateLogStream`
Example contents of Ops configuration file:
```json
{
"Klibs": ["cloudwatch", "tls"],
"ManifestPassthrough": {
"cloudwatch": {
"logging": {
"log_group": "my_log_group",
"log_stream": "my_log_stream"
}
}
}
}
```
**Note**: If `log_group` is not set, the `"IMAGE_NAME"` environment variable, if present, will be used to set the `log_group`, while the *program name* is used as a *fallback* setting for `log_group`.
### Cloudwatch metrics
The **cloudwatch** *klib* implements sending **memory utilization metrics** to **AWS CloudWatch**. Analogously to the implementation in the Linux CloudWatch agent, the metrics being sent are under the **"CWAgent"** namespace, and have an associated dimension whose name is "host" and whose value is an instance identifier formatted as in the following example: `"ip-111-222-111-222.us-west-1.compute.internal"`.
The list of supported metrics is:
* **mem\_used**
* **mem\_used\_percent**
* **mem\_available**
* **mem\_available\_percent**
* **mem\_total**
* **mem\_free**
* **mem\_cached**
A description for each of these metrics can be found at , in the section for the Linux CloudWatch agent. In order for the cloudwatch klib to retrieve the appropriate credentials needed to communicate with the CloudWatch server, the AWS instance on which it runs must be associated to an IAM role with the CloudWatchAgentServerPolicy, as described in . Memory metrics are defined as standard-resolution metrics, i.e. *they are stored with a resolution of 60 seconds*.
The **cloudwatch** *klib* is configured via a `cloudwatch` tuple in the image manifest. Sending of memory metrics is enabled by specifying a sending interval (expressed in seconds) via a `mem_metrics_interval` attribute inside the `cloudwatch` tuple. The `cloudwatch` klib depends on the `tls` klib for cryptographic operations (which are needed in order to sign the requests being sent to the CloudWatch server).
Example Ops configuration to send memory metrics with a `60-second` interval:
```json
{
"Klibs": ["cloudwatch", "tls"],
"ManifestPassthrough": {
"cloudwatch": {
"mem_metrics_interval": "60"
}
}
}
```
## DigitalOcean
The `digitalocean` klib provides a method for sending internal memory metrics to the Digital Ocean monitoring service. The default interval is set to 120 seconds but can be configured with the 'interval' setting.
As of right now cpu and disk i/o metrics are not displayed when this klib is enabled as DO disables the older metrics data when the agent is sending metrics. This can be addressed in future versions of this klib.
```json
"ManifestPassthrough": {
"digitalocean": {
"metrics": {
"interval":"180"
}
}
}
```
## GCP
The `gcp` klib implements two functionalities that can be used on GCP cloud:
1. `logging` - *console driver* that sends **console output** to **GCP logs**
2. `metrics` - emulates some functions of GCP ops agent to send **memory usage metrics** to the **GCP monitoring service**
**Note:** When executing the ops instance create command to create a GCP instance, if the `CloudConfig.InstanceProfile` configuration parameter is a non-empty string, the instance being created is associated to the service account identified by this string, with cloud-platform access scope. GCP service accounts are identified by an email address; the special string "default" indicates the default service account for a given project. The service account specified in the configuration must exist in the GCP project when an instance is being created, otherwise an error is returned. For more information about GCP service accounts, see .
### GCP logging
The **gcp** *klib* implements a *console driver* that sends *console output* to GCP logs. Instance-specific information that needs to be known in order to interface with the GCP logging API is retrieved from the instance metadata server, which is reachable at address `169.254.169.254`. GCP logging is enabled by loading the gcp and tls klibs and adding to the root tuple a `"gcp"` tuple that contains a `"logging"` tuple. The `"logging"` tuple may optionally contain a `"log_id"` attribute that specifies the **\[LOG\_ID]** string that is sent in the **"logName"** parameter (which is formatted as `"projects/[PROJECT_ID]/logs/[LOG_ID]"`) associated to GCP log entries; **if not present**, the *log ID is derived from the instance hostname as retrieved from the metadata server*.
In order for the GCP klib to retrieve the appropriate credentials needed to communicate with the GCP logging server, the instance on which it runs must be associated to a service account (see ).
Instances created by Ops are associated to a service account via the `CloudConfig.InstanceProfile` configuration parameter.
Example contents of Ops configuration file:
```json
{
"CloudConfig" :{
"Platform" :"gcp",
"ProjectID" :"prod-1000",
"Zone": "us-west1-a",
"BucketName":"my-s3-bucket",
"InstanceProfile":"default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"logging": {
"log_id": "my_log"
}
}
}
}
```
### GCP metrics
The **gcp** *klib* can be configured for sending `memory` and `disk` usage metrics to the **GCP monitoring service**, thus emulating the **GCP ops agent**. The `ID` of the running instance and the `zone` where it is running (which are necessary to be able to send API requests to the monitoring server) are retrieved from the instance *metadata server*.
* **memory** usage metrics being sent (see ), are the `bytes_used` and `bytes_percent` metric types; for each type, a value is sent for each of the **"cached"**, **"free"** and **"used"** states.
* **disk** usage metrics being sent (see ), are the `bytes_used` and `bytes_percent` metric types; for each type, a value is sent for each of the **"free"** and **"used"** states.
In order for the gcp klib to retrieve the appropriate credentials needed to communicate with the GCP monitoring server, the instance on which it runs must be associated to a service account (see ).
Instances created by Ops are associated to a service account via the `CloudConfig.InstanceProfile` configuration parameter.
The allowed configuration properties are:
* `metrics` - enable metrics. By default, only **memory** metrics are sent, **disk** metrics are disabled.
* `interval` - value expressed in **seconds** to modify the time interval at which metrics are sent. The default (and minimum allowed) value is `60 seconds`.
* `disk` - enable disk metrics. By default, only **read-write** mounted disk(s) metrics are sent, **read-only** disk metrics are disabled.
* `include_readonly` - enable also **read-only** disk metrics.
Example Ops configuration to enable sending **memory** only metrics every `2 minutes`:
```json
{
"CloudConfig": {
"Platform": "gcp",
"ProjectID": "prod-1000",
"Zone": "us-west1-a",
"BucketName": "my-s3-bucket",
"InstanceProfile": "default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"metrics": {
"interval":"120"
}
}
}
}
```
Example Ops configuration to enable sending **memory** metrics and **disk** metrics (*read-write* only) every `2 minutes`:
```json
{
"CloudConfig": {
"Platform": "gcp",
"ProjectID": "prod-1000",
"Zone": "us-west1-a",
"BucketName": "my-s3-bucket",
"InstanceProfile": "default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"metrics": {
"interval": "120",
"disk": {}
}
}
}
}
```
Example Ops configuration to enable sending **memory** metrics and **disk** metrics (*read-write* and *read-only*) every `2 minutes`:
```json
{
"CloudConfig": {
"Platform": "gcp",
"ProjectID": "prod-1000",
"Zone": "us-west1-a",
"BucketName": "my-s3-bucket",
"InstanceProfile": "default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"metrics": {
"interval": "120",
"disk": {
"include_readonly": "true"
}
}
}
}
}
```
## NTP
The `ntp` klib allows to set the configuration properties to synchronize the unikernel clock with a ntp server.
The allowed configuration properties are:
* `ntp_servers` - array of ntp servers, with each server specified using the format `[:` string can contain an IP address or a fully qualified domain name; if it contains a numeric IPv6 address, it must be enclosed in square brackets, as per RFC 3986 (example: `"ntp_servers": ["[2610:20:6f97:97::4]", "[2610:20:6f97:97::5]:1234"]`). The default value is **pool.ntp.org:123**.
* `ntp_poll_min` - the minimum poll time is expressed as a power of two. The default value is **4**, corresponding to 16 seconds (2^4 = 16). The minimum value is **4**, corresponding to 16 seconds (2^4 = 16).
* `ntp_poll_max` - the maximum poll time is expressed as a power of two. The default value is **10**, corresponding to 1024 seconds (2^10 = 1024). The maximum value is **17**, corresponding to 131072 seconds (2^17 = 131072 = \~36.4 hours).
* `ntp_reset_threshold` - This is a difference threshold expressed in **seconds** to use *step/jump* versus *smearing* on ntp - the default is set to **0** *meaning it will never step/jump*. If the difference is over this threshold then step/jump will be used allowing correction over much longer periods.
* `ntp_max_slew_ppm` - maximum slewing rate for clock offset error correction, expressed in PPM; default value: **83333**
* `ntp_max_freq_ppm` - maximum clock frequency error rate, expressed in PPM; default value: **25000**
When running on AWS, the Elastic Network Adapter (ENA) interface driver on Nanos, has support for retrieving the current time from a *PTP hardware clock*, when supported by the network interface.
`ntp` klib is able to retrieve time from the *PTP clock* **alternatively** to *using NTP*, by using the following configuration:
* `"chrony":{"refclock": "ptp"}`
* the ntp klib *attempts* to use the PTP clock for synchronizing the system time
* if a *PTP clock* is **not available**, the klib *falls back to NTP*
The `ntp` klib needs to collect some data samples from ntp server(s) before deciding the actions needed to update the clock (if any).
* It needs a minimum of **4** *samples* to analyze the needed changes - `MIN_SAMPLES 4`
* It keeps a maximum of **30** *samples* to analyze the needed changes - `MAX_SAMPLES 30`
It is important to understand that until `ntp` has collected at least 4 samples, no actions will be taken to update the clock. During that time the system will operate under the clock value provided by the hypervisor, which may or may not be correct. This means that, with default config setup and no ntp request failures, it will take about **53** *seconds* after boot for the klib to update the system clock if needed.
This can be observed from a debug build of the klib:
```
en1: assigned 10.0.2.15
[0.319779, 0, ntp] adding server 0.us.pool.ntp.org (port 123)
en1: assigned FE80::447A:E3FF:FECF:B36F
[5.697626, 0, ntp] selecting 0.us.pool.ntp.org as current server
[5.700948, 0, ntp] insert 0: 1618876805.261140, off=67692786.324026320, rtd=0.317320199, jit=67692786.324026320
[21.595184, 0, ntp] insert 1: 1618876821.208357, off=67692786.382279612, rtd=0.211364274, jit=0.058253291
[37.591244, 0, ntp] insert 2: 1618876837.206483, off=67692786.380658678, rtd=0.207233731, jit=-0.001620934
[53.585795, 0, ntp] insert 3: 1618876853.203833, off=67692786.377978558, rtd=0.201635557, jit=-0.002680119
[53.591257, 0, ntp] packet offset=67692786.377978558 est_offset(total)=67692786.376225217(67692786.376225217) est_freq(total)=0.000087361(0.000087361) offset_sd=0.008933535 skew=0.118788699
[69.587293, 0, ntp] insert 4: 1686569655.580903, off=0.000968804, rtd=0.202942327, jit=-0.000793821
[69.592481, 0, ntp] packet offset=0.000968804 est_offset(total)=0.003349950(0.003349950) est_freq(total)=-0.000005103(0.000082258) offset_sd=0.009849557 skew=0.040548864
[85.590427, 0, ntp] insert 5: 1686569671.587333, off=-0.001286386, rtd=0.205845955, jit=0.001095304
[85.595407, 0, ntp] packet offset=-0.001286386 est_offset(total)=-0.000979362(-0.000979362) est_freq(total)=-0.000023029(0.000059228) offset_sd=0.007714219 skew=0.012092993
...
[758.312460, 0, ntp] insert 28: 1686570344.311070, off=-0.011912567, rtd=0.269852664, jit=-0.002127423
[758.318662, 0, ntp] packet offset=-0.011912567 est_offset(total)=-0.000798835(-0.000798835) est_freq(total)=-0.000001312(0.000040926) offset_sd=0.003480576 skew=0.000028015
[774.309514, 0, ntp] insert 29: 1686570360.309574, off=-0.010015399, rtd=0.266696978, jit=0.001098517
[774.312045, 0, ntp] packet offset=-0.010015399 est_offset(total)=-0.000703166(-0.000703166) est_freq(total)=-0.000001130(0.000039796) offset_sd=0.003435532 skew=0.000027003
[790.313905, 0, ntp] insert 0: 1686570376.311817, off=-0.006775507, rtd=0.270922496, jit=0.002536878
[790.315987, 0, ntp] packet offset=-0.006775507 est_offset(total)=-0.001349425(-0.001349425) est_freq(total)=-0.000003542(0.000036254) offset_sd=0.002574216 skew=0.000019820
[806.313070, 0, ntp] insert 1: 1686570392.310801, off=-0.005923203, rtd=0.269834293, jit=-0.000496634
[806.316524, 0, ntp] packet offset=-0.005923203 est_offset(total)=-0.000628994(-0.000628994) est_freq(total)=-0.000001222(0.000035031) offset_sd=0.002575573 skew=0.000019651
...
```
Use the configuration file to enable the `ntp` klib and setup the settings.
```json
{
"Klibs": ["ntp"],
"ManifestPassthrough": {
"ntp_servers": ["127.0.0.1:1234"],
"ntp_poll_min": "5",
"ntp_poll_max": "10",
"ntp_reset_threshold": "0",
"ntp_max_slew_ppm": "83333",
"ntp_max_freq_ppm": "25000",
"chrony": {
"refclock": "ptp"
}
}
}
```
### Verifying your NTP is working:
To verify that NTP is working you can run this sample go program and then manually adjust the clock on qemu:
```json
➜ g cat config.json
{
"Klibs": ["ntp"],
"ManifestPassthrough": {
"ntp_servers": ["0.us.pool.ntp.org:123"],
"ntp_poll_min": "4",
"ntp_poll_max": "6"
}
}
```
```go
➜ g cat main.go
package main
import (
"fmt"
"time"
)
func main() {
for i := 0; i < 10; i++ {
fmt.Println("Current Time in String: ", time.Now().String())
time.Sleep(8 * time.Second)
}
}
```
Run it once to build the image:
```bash
GOOS=linux go build
ops run -c config.json g
```
Then you can grep the ps output of the 'ops run' command which should look something like this:
and tack on the clock modifier to the end of the command:
```bash
-rtc base="2021-04-20",clock=vm
```
```bash
➜ g qemu-system-x86_64 -machine q35 -device
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x3.0x2
-device virtio-scsi-pci,bus=pci.2,addr=0x0,id=scsi0 -device
scsi-hd,bus=scsi0.0,drive=hd0 -vga none -smp 1 -device isa-debug-exit -m
2G -accel hvf -cpu host,-rdtscp -no-reboot -cpu max,-rdtscp -drive
file=/Users/eyberg/.ops/images/g.img,format=raw,if=none,id=hd0 -device
virtio-net,bus=pci.3,addr=0x0,netdev=n0,mac=e6:44:9c:ca:fd:ca -netdev
user,id=n0,hostfwd=tcp::8080-:8080 -display none -serial stdio -rtc base="2021-04-20",clock=vm
en1: assigned 10.0.2.15
Current Time in String: 2021-04-20 00:00:00.093924867 +0000 UTC
m=+0.001076816
en1: assigned FE80::E444:9CFF:FECA:FDCA
Current Time in String: 2021-04-22 12:06:53.91767521 +0000 UTC
m=+8.008671695
```
## Radar
The `radar` klib allows to send telemetry/crash data to an external [Radar](https://nanovms.com/radar) APM service.
The actual **radar** *klib* is pre-configured to send data to `https://radar.relayered.net:443` and requires an api key `Radar-Key` that can be provided as an environment variable. The available configuration items that can be passed as envs are:
* `RADAR_KEY` - mandatory, used to set `Radar-Key` and enable `radar` klib fuctionality
* `RADAR_IMAGE_NAME` - optional, used to set an `"imageName"` that will be sent to the APM
The `radar` klib depends on the `tls` klib for cryptographic operations.
Sample config to activate the klib functionality:
```json
{
"Env": {
"RADAR_KEY": "RADAR_KEY_xyz",
"RADAR_IMAGE_NAME": "RADAR_IMAGE_NAME_xyz"
},
"Klibs": ["radar", "tls"]
}
```
### Radar boot report
Upon boot, the `radar` klib will send some initial data to the Radar server and expects to get back a numeric `"id"` that will be assigned to this boot event as `"bootID"`.
The information sent looks like this:
```
POST /api/v1/boots HTTP/1.1
Host: radar.relayered.net
Content-Length: 116
Content-Type: application/json
Radar-Key: RADAR_KEY_xyz
```
```json
{
"privateIP": "10.0.2.15",
"nanosVersion": "0.1.45",
"opsVersion": "master-dc83aee",
"imageName": "RADAR_IMAGE_NAME_xyz"
}
```
The radar server should return back an unique id:
```json
{"id":1234}
```
**Note**: If there is a **crash dump** to be sent, it will be sent **before** sending the *boot report*.
### Radar metrics report
The `radar` klib retrieves from the kernel ifromation about:
* **memory usage** - retrieved from the kernel *total memory usage in bytes* **every minute**, and sends retrieved data to the server every **5 minutes**. The **5 samples** are sent under `"memUsed"` attribute.
* **disk usage** - retrieved and sent every **5 minutes**. Data is sent under `"diskUsage"` attribute whose value is an array of JSON objects (one for each disk mounted by the instance). Each array element contains 3 attributes:
* `"volume"`: a string that identifies the volume. For the root volume, the string value is "root", for any additional volumes, the string value is the volume label if present, or the volume UUID if the label is not present
* `"used"`: the number of bytes used by the filesystem (file contents and meta-data) in the volume
* `"total"`: the total number of bytes in the storage space of the volume, i.e. the upper limit for the "used" attribute value
The information sent looks like this:
```
POST /api/v1/machine-stats HTTP/1.1
Host: radar.relayered.net
Content-Length: 138
Content-Type: application/json
Radar-Key: RADAR_KEY_xyz
```
```json
{
"bootID": 1234,
"memUsed": [
68329472,
68329472,
68329472,
68329472,
68329472
],
"diskUsage": [
{
"volume": "root",
"used": 5516288,
"total": 39779328
},
{
"volume": "2375a5a1-2d36-15cf-ea6b-2fa01df2ccde",
"used": 12345,
"total": 39779328
}
]
}
```
### Radar crash report
A **"crash"** is defined as either a user application fatal error (e.g. anything that causes the application to terminate with a non-zero exit code), or a kernel fatal error. When any of these happens, before shutting down the VM the kernel dumps on disk a trace of log messages (printed by both the user application and the kernel). At the next boot, the `radar` klib detects the log dump and sends it to the radar server as a crash report. The value of the **"id"** field of a crash report is the same as the **"boot id"** (sent right after each boot), so that a crash can be unequivocally associated to a boot.
**Note**: The log dump is saved in a section of the disk that is being carved between *stage2* and the *boot* filesystem.
The information sent looks like this:
```
POST /api/v1/crashes HTTP/1.1
Host: radar.relayered.net
Content-Length: 213
Content-Type: application/json
Radar-Key: RADAR_KEY_xyz
```
```json
{
"bootID":1234,
"nanosVersion":"0.1.45",
"opsVersion":"master-dc83aee",
"imageName":"RADAR_IMAGE_NAME_xyz",
"dump":"en1: assigned 10.0.2.15\nen1: assigned FE80::2885:1DFF:FE7E:9D71\nProcess crash message in here\n"
}
```
**Note**: In order for the submission to be considered successful, radar server needs to respond with a **"specific" confirmation message**, otherwise the crash dump submit will be attempted over and over.
## SpecialFiles
The 'special\_files' klib provides a set of pseudo-files that are Nanos specific.
In particular the config below will populate the path of `/sys/devices/disks` with the name of each disk attached with the volume name and UUID of the disk:
```json
{
"RunConfig": {
"QMP": true
},
"Klibs": ["special_files"],
"ManifestPassthrough": {
"special_files": {
"disks": {}
}
},
"Mounts": {
"bob": "/bob"
}
}
```
You can test this behavior with the below program:
```go
package main
import (
"fmt"
"os"
"time"
)
func main() {
for i := 0; i < 30; i++ {
body, err := os.ReadFile("/proc/mounts")
if err != nil {
fmt.Println(err)
}
fmt.Print(string(body))
body, err = os.ReadFile("/sys/devices/disks")
if err != nil {
fmt.Println(err)
}
fmt.Print(string(body))
time.Sleep(2 * time.Second)
}
}
```
This functionality provides similar lsblk like functionality you might find on linux, however, different cloud providers do not put the same uniquely identifiable information into the serial/id of the device so we implemented this instead.
## Syslog
If you can point your app at a syslog we encourage you to do that:
However, if you have no control over the application than you can direct Nanos to use the syslog klib and it will ship everything over.
Just pass in the desired syslog server along with the syslog klib in your config.
If the `"IMAGE_NAME"` environment variable is present, it is used to populate the `APP_NAME` field in syslog messages, while the `program` *name* is used as a *fallback*.
For example, if running locally via user-mode you can use 10.0.2.2:
```json
{
"Env": {
"IMAGE_NAME": "app-name-in-syslog-msg"
},
"ManifestPassthrough": {
"syslog": {
"server": "10.0.2.2",
"server_port": "514"
}
},
"Klibs": ["syslog"]
}
```
If you need the logs to be also stored in a file:
```json
{
"Env": {
"IMAGE_NAME": "app-name-in-syslog-msg"
},
"ManifestPassthrough": {
"syslog": {
"server": "10.0.2.2",
"server_port": "514",
"file": "/tmp/sys.log",
"file_max_size": "8M",
"file_rotate": "9"
}
},
"Klibs": ["syslog"]
}
```
If you are running on Linux you can use rsyslogd for this. By default rsyslogd will not listen on UDP 514 so you can un-comment the lines in /etc/rsyslog.conf:
```
module(load="imudp")
input(type="imudp" port="514")
```
and restart:
```
sudo service rsyslog restart
```
also you can disable the dupe filter or add a timetstamp to skirt around that.
## TUN
The `tun` klib is used to create TUN interfaces, namely *network TUNnel*, that simulate a network layer device operating in **layer 3** of OSI Model carrying IP packets.
Can be used when running vpn gateways (i.e userspace wireguard).
The allowed configuration properties for the tun interface(s) are:
* `interface base name` - i.e `wg`, `tun`
* `ipaddress` - ip address of the interface
* `netmask` - netmask of the interface
* `mtu` - configures the **mtu** value of the interface, default value is **32768** (32KiB)
* `up` - **true**/**false** manages the interface initial state
Sample config:
```json
{
"Klibs": ["tun"],
"ManifestPassthrough": {
"tun": {
"wg": {
"ipaddress": "172.16.0.1",
"netmask": "255.255.255.0",
"up": "true",
"mtu": "1420"
}
}
}
}
```
## Cloud Init - HTTP(s) to File/Env KLIB
Cloud Init has 3 functions.
1. For Azure machines it is auto-included to check in to the meta server to tell Azure that the machine has completed booting. This is necessary, otherwise Azure will think it failed.
2. One can include this on any platform to download one or more extra files to the instance for post-deploy config options. This is useful when security or ops teams are separate from dev or build teams and they might handle deploying tls certificates or secrets post-deploy. All files are downloaded before execution of the main program.
3. One can include this on any platform to populate the environment of the user process with variables whose name and value are retrieved from an HTTP(S) server during startup.
The cloud\_init klib supports a configuration option to overwrite previous files: it's called `overwrite`. If you specify this option for a given file, by inserting an `overwrite` JSON attribute with any string value, cloud\_init will re-download the file at every boot.
The cloud\_init klib also supports simple authentication mechanisms by using `auth` config option. It uses HTTP Authorization request header `Authorization: ` where `` and `` are configurable.
Certain caveats to be aware of:
* Only direct download links are supported today. (no redirects)
* When cloud\_init cannot download one or more files, the kernel does not start the user program. The rationale for this is that we want all files to be ready and accessible when the program starts.
* When used to populate the user environment, only string-valued attributes are converted to environment variables (**non-string-valued attributes are ignored**).
* The `cloud_init` klib (`download_env` functionality) is **not meant** to be used to set environment variables *whose value is used in other klibs or in the kernel code* (e.g. the `"IMAGE_NAME"` environment variable used by the `syslog` klib), because the code that uses an environment variable *can be executed before* `cloud_init` sets a value for the variable.
Also, be aware that you set an appropriate minimum image base size to accomodate your files.
Example Go program:
```go
package main
import (
"fmt"
"os"
)
func main() {
body, err := os.ReadFile("/nanos.md")
if err != nil {
fmt.Println(err)
}
fmt.Print(string(body))
}
```
Example config - *no overwrite* - existing destination file won't be changed/overwritten:
```json
{
"BaseVolumeSz": "20m",
"Klibs": ["cloud_init", "tls"],
"ManifestPassthrough": {
"cloud_init": {
"download": [
{
"src": "https://raw.githubusercontent.com/nanovms/ops-documentation/master/README.md",
"dest": "/nanos.md"
}
]
}
}
}
```
Example config - *overwrite* - existing destination file will be replaced/overwritten at every boot:
```json
{
"BaseVolumeSz": "20m",
"Klibs": ["cloud_init", "tls"],
"ManifestPassthrough": {
"cloud_init": {
"download": [
{
"src": "https://raw.githubusercontent.com/nanovms/ops-documentation/master/README.md",
"dest": "/nanos.md",
"overwrite": "t"
}
]
}
}
}
```
Example config, basic access authentication - *auth* - `Authorization` header will be added to the request:
```json
{
"BaseVolumeSz": "20m",
"Klibs": ["cloud_init", "tls"],
"ManifestPassthrough": {
"cloud_init": {
"download": [
{
"src": "https://httpbin.org/hidden-basic-auth/user/passwd",
"dest": "/basic_auth_test.txt",
"auth": "Basic dXNlcjpwYXNzd2Q="
}
]
}
}
}
```
Example config, with environment variables management - `download_env` config:
The configuration syntax in the manifest to use this functionality is as in the following example:
```json
"ManifestPassthrough": {
"cloud_init": {
"download_env": [
{
"src": "http://10.0.2.2:8200/v1/secret/data/hello",
"auth": "Bearer hvs.6v6yY5yZf32uZzaa7HhiX6AZ",
"path": "attr1/attr2"
}
]
}
},
```
The `download_env` attribute is an array where each element specifies
* `src` - download source URL
* `auth` - *optional* authentication header
* `path` - *optional* attribute path
For each element in the `download_env` attribute, the klib executes an HTTP request to the specified URL, and if the peer responds successfully, the response body is parsed as a JSON object, and from this object an "environment object" is retrieved.
If the attribute path (i.e. the path element in the manifest) is not present (or is empty), the environment object corresponds to the root JSON object of the response body, otherwise, for each element in the attribute path (`where the element separator is the / character`), a *nested* JSON object is retrieved from the response body by looking up an attribute named after the element (**the corresponding attribute value must be a JSON object**): *the environment object is the nested object corresponding to the last element of the attribute path*.
Once the environment object is identified, **all string-valued attributes** in this object are converted to environment variables (**non-string-valued attributes are ignored**).
Examples:
an empty attribute path can be used to retrieve the environment variables from the following response body:
```json
{
"VAR1": "value1",
"VAR2": "value2"
}
```
an attribute path set to "obj1/obj2" can be used to retrieve the environment variables from the following response body:
```json
{
"obj1": {
"obj2": {
"VAR1": "value1",
"VAR2": "value2"
}
}
}
```
`"VAR1"` and `VAR2`, being **non-string-valued attributes**, will be **ignored** from the following response body:
```json
{
"VAR1": 1,
"VAR2": true,
"VAR3": "value3"
}
```
Full example:
nanos config:
```json
{
"BaseVolumeSz": "20m",
"Klibs": [
"cloud_init",
"tls"
],
"ManifestPassthrough": {
"cloud_init": {
"download": [
{
"src": "https://httpbin.org/hidden-basic-auth/user/passwd",
"dest": "/cloud_init_test.json",
"overwrite": "t",
"auth": "Basic dXNlcjpwYXNzd2Q="
}
],
"download_env": [
{
"src": "https://httpbin.org/hidden-basic-auth/user/passwd",
"auth": "Basic dXNlcjpwYXNzd2Q=",
"path": ""
}
]
}
}
}
```
go program sample:
```go
package main
import (
"fmt"
"os"
)
const (
AUTH_RESULT_FILE = "cloud_init_test.json" // {"authenticated": true, "user": "user"}
ENV_AUTHENTICATED = "authenticated"
ENV_USER = "user"
)
func main() {
// ManifestPassthrough.cloud_init.download
authResult, err := os.ReadFile(AUTH_RESULT_FILE)
if err != nil {
fmt.Printf("error reading file - %s (%s)\n", AUTH_RESULT_FILE, err.Error())
} else {
fmt.Printf("%s - content:\n", AUTH_RESULT_FILE)
fmt.Printf("%s\n", string(authResult))
}
// ManifestPassthrough.cloud_init.download_env
authenticated, ok := os.LookupEnv(ENV_AUTHENTICATED)
if !ok {
fmt.Printf("ENV: %q - not found\n", ENV_AUTHENTICATED)
} else {
fmt.Printf("ENV: %q = %q\n", ENV_AUTHENTICATED, authenticated)
}
// ManifestPassthrough.cloud_init.download_env
user, ok := os.LookupEnv(ENV_USER)
if !ok {
fmt.Printf("ENV: %q - not found\n", ENV_USER)
} else {
fmt.Printf("ENV: %q = %q\n", ENV_USER, user)
}
}
```
from the result, as expected, `"authenticated"` - being **non-string-valued attribute**, is not available in the environment:
```sh
booting /home/ops/.ops/images/cloudinit_test ...
...
cloud_init_test.json - content:
{
"authenticated": true,
"user": "user"
}
ENV: "authenticated" - not found
ENV: "user" = "user"
...
```
## Firewall
This klib implements a **network firewall**. The firewall can drop IP packets received from any network interface, based on a set of rules defined in the manifest. Each rule is specified as a tuple located in the rules array of the firewall tuple in the manifest. Valid attributes for a firewall rule are the following:
* `ip`: matches IPv4 packets, and is a tuple that can have the following attributes:
* `src`: matches packets based on the **source IPv4 address**, which is specified with the standard *dotted notation* **aaa.bbb.ccc.ddd**, can be *prefixed* by an optional **!** character which makes the rule match packets with an address different from the provided value, and can be *suffixed* with an **optional netmask** (with format /, where can have a value from 1 to 32) which matches packets based on the first part of the provided address
* `ip6`: matches IPv6 packets, and is a tuple that can have the following attributes:
* `src`: matches packets based on the **source IPv6 address**, which is specified with the standard notation for IPv6 addresses, and similarly to its IPv4 counterpart can be *prefixed* with a **!** character and *suffixed* with a **netmask (with allowed values from 1 to 128)**
* `tcp`: matches TCP packets, and is a tuple that can have the following attributes:
* `dest`: matches packets based on the **TCP destination port** (whose value can be prefixed with a **!** character to negate the logical comparison with the port value in a packet)
* `udp`: matches UDP packets, and is a tuple that can have the following attributes:
* `dest`: matches packets based on the **UDP destination port** (whose value can be prefixed with a **!** character to negate the logical comparison with the port value in a packet)
* `action`: indicates which action should be performed by the firewall when a matching packet is received: *allowed* values are **"accept"** and **"drop"**; if this attribute is not present, **the default action for the rule is to drop matching packets**
Firewall rules are *evaluated for each received packet in the order they are defined*, until a matching rule (i.e. a rule where all the attributes match the packet contents) is found and the corresponding action is executed. *If a packet does not match any rule, it is* **accepted**.
Example contents of Ops configuration file:
* accept all TCP packets to port 8080, drop all other packets:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"tcp": {"dest": "8080"}, "action": "accept"},
{"action": "drop"}
]
}
}
}
```
* accept all packets coming from IP address 10.0.2.2, drop packets from other addresses unless they are to TCP port 8080:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip": {"src": "10.0.2.2"}, "action": "accept"},
{"tcp": {"dest": "8080"}, "action": "accept"},
{"action": "drop"}
]
}
}
}
```
* drop all packets coming from IP address 10.0.2.2:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip": {"src": "10.0.2.2"}, "action": "drop"}
]
}
}
}
```
* drop IPv4 packets coming from addresses other than 10.0.2.2:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip": {"src": "!10.0.2.2"}, "action": "drop"}
]
}
}
}
```
* drop all UDP packets coming from IP addresses in the range 10.0.2.0-10.0.2.255:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip": {"src": "10.0.2.0/24"}, "udp": {}, "action": "drop"}
]
}
}
}
```
* drop all packets coming from IPv6 address FE80::FF:7762:
```json
{
"Klibs": ["firewall"],
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip6": {"src": "FE80::FF:7762"}, "action": "drop"}
]
}
}
}
```
* drop all tcp fragments:
```
"ManifestPassthrough": {
"firewall": {
"rules": [
{"ip": {"fragment": "y"}, "action": "drop"}
]
}
}
```
### Dynamic Firewall Rules
In addition to static firewall rules defined in the configuration manifest, the firewall can retrieve dynamic rules from Radar (this requires a Radar API key). When this functionality is enabled, the firewall downloads from the Radar server a list of IP addresses that are known sources of spam or other abusive behavior, and automatically adds a set of rules that block all network traffic originating from those addresses.
Example contents of Ops configuration file to retrieve firewall rules from Radar:
```
"ManifestPassthrough": {
"firewall": {
"dynamic_rules":["radar"]
}
},
"Env": {
"RADAR_KEY": "my_radar_api_key"
}
```
## s3fs
The s3fs klib allows one to mount a s3 bucket to the local filesystem so that you can read and write to it simply without further additional client config. This can all be done at image build time allowing you to apply the configuration completely separate from the code.
For files over 4kb in size you'll need to use O\_DIRECT.
```
{
"Klibs": ["s3fs", "tls"],
"ManifestPassthrough": {
"s3fs": {
"region": "us-west-1",
"access_key": "MY_ACCESS_KEY",
"secret": "my_secret",
"buckets": [
{
"name": "my-bucket",
"mount": "/mnt"
}
]
}
}
}
```
## Sandbox
This klib implements a **sandboxing mechanism** by which the capabilities of the running process can be *restricted* by *overriding certain syscall handlers*.
The current implementation supports the following OpenBSD syscalls:
* `pledge` -
* `unveil` -
A `pledge` invocation forces the process into a restricted-service operating mode by limiting the set of syscalls (and in some cases the allowed values of syscall arguments) that can be invoked. Syscalls that are present in both Nanos and OpenBSD are handled by following the pledge implementation in OpenBSD, except for syscalls that are not applicable to unikernels (e.g. those related to inter-process communication). Syscalls that are present (or could be implemented in the future) in Nanos but not in OpenBSD are handled based in the same way as "similar" or related OpenBSD syscalls when such syscalls could be found, and are always forbidden when no related syscalls could be found in OpenBSD.
The `unveil` syscall allows adding and removing (or restricting) visibility of filesystem paths for the running process.
The `pledge` and `unveil` features are enabled in the **sandbox** *klib* if the **manifest** contains a `sandbox` tuple with a `pledge` and `unveil` tuple, respectively.
```json
{
"Klibs": ["sandbox"],
"ManifestPassthrough": {
"sandbox": {
"pledge": {},
"unveil": {}
}
}
}
```
### C
The `pledge` syscall can be invoked in a C program by calling the `pledge()` function from the following code:
```c
int pledge(const char *promises, const char *execpromises)
{
return syscall(335, promises, execpromises);
}
```
It can be invoked in a C program by calling the `unveil()` function from the following code:
```c
int unveil(const char *path, const char *permissions)
{
return syscall(336, path, permissions);
}
```
### Go
The following Go code provides the basis for a `nanos_sandbox` package:
```go
package nanos_sandbox
import (
"syscall"
"unsafe"
)
const (
// OpenBSD syscalls, mapped to unused syscall numbers in Linux
SYS_pledge = 335
SYS_unveil = 336
)
// Pledge implements the pledge syscall.
//
// For more information see pledge(2).
func Pledge(promises, execpromises string) error {
pptr, err := syscall.BytePtrFromString(promises)
if err != nil {
return err
}
// pass execpromises to the syscall.
exptr, err := syscall.BytePtrFromString(execpromises)
if err != nil {
return err
}
_, _, e := syscall.Syscall(SYS_pledge, uintptr(unsafe.Pointer(pptr)), uintptr(unsafe.Pointer(exptr)), 0)
if e != 0 {
return e
}
return nil
}
// PledgePromises implements the pledge syscall.
//
// This changes the promises and leaves the execpromises untouched.
//
// For more information see pledge(2).
func PledgePromises(promises string) error {
// This variable holds the execpromises and is always nil.
var exptr unsafe.Pointer
pptr, err := syscall.BytePtrFromString(promises)
if err != nil {
return err
}
_, _, e := syscall.Syscall(SYS_pledge, uintptr(unsafe.Pointer(pptr)), uintptr(exptr), 0)
if e != 0 {
return e
}
return nil
}
// Unveil implements the unveil syscall.
// For more information see unveil(2).
// Note that the special case of blocking further
// unveil calls is handled by UnveilBlock.
func Unveil(path string, flags string) error {
pathPtr, err := syscall.BytePtrFromString(path)
if err != nil {
return err
}
flagsPtr, err := syscall.BytePtrFromString(flags)
if err != nil {
return err
}
_, _, e := syscall.Syscall(SYS_unveil, uintptr(unsafe.Pointer(pathPtr)), uintptr(unsafe.Pointer(flagsPtr)), 0)
if e != 0 {
return e
}
return nil
}
// UnveilBlock blocks future unveil calls.
// For more information see unveil(2).
func UnveilBlock() error {
// Both pointers must be nil.
var pathUnsafe, flagsUnsafe unsafe.Pointer
_, _, e := syscall.Syscall(SYS_unveil, uintptr(pathUnsafe), uintptr(flagsUnsafe), 0)
if e != 0 {
return e
}
return nil
}
```
### JavaScript
There are 2 npm packages published and ready for use.
* `pledge` -
* `unveil` -
## shmem
```
#include
#include
#include
#include
#include
#include
int main() {
const char* shm_name = "/myshm";
const char* message = "Hello from tmpfs in Nanos!";
int shm_fd;
void* ptr;
shm_fd = shm_open(shm_name, O_CREAT | O_RDWR, 0666);
if (shm_fd == -1) {
perror("shm_open");
return 1;
}
if (ftruncate(shm_fd, 4096) == -1) {
perror("ftruncate");
return 1;
}
ptr = mmap(0, 4096, PROT_WRITE, MAP_SHARED, shm_fd, 0);
if (ptr == MAP_FAILED) {
perror("mmap");
return 1;
}
size_t len = strlen(message);
memcpy(ptr, message, len + 1);
printf("Written to tmpfs: %s\n", (char*)ptr);
munmap(ptr, 4096);
close(shm_fd);
shm_unlink(shm_name);
return 0;
}
```
create a stub /dev/shm:
```
mkdir -p dev/shm
```
```
{
"Dirs": ["dev"],
"Klibs": ["shmem", "tmpfs"]
}
```
## User Data
The User Data klib allows you to populate data from the IMDS datastore at instance creation time. Unlike the existing cloud\_init klib this doens't need an external datastore as you can provision using the meta data store found on many instances instead. You also do not need to include these env vars at image build time.
Separate each env var name pair with a carriage return && newline and the key/values with the equals operator.
You may use this simple config as an example:
```
{
"CloudConfig": {
"UserData": "BOB=something\r\nTOM=something-else"
},
"Klibs": ["userdata_env", "tls"]
}
```
## Out-of-tree klibs
In addition to the above klibs, whose source code is included in the Nanos kernel source tree, out-of-tree klibs can also be created to enhance Nanos with additional functionality. Provided that such klibs are compiled against the exact version of the kernel being used, they can be added to a given image and will be loaded by the kernel during initialization just like in-tree klibs.
### gpu\_nvidia
This klib, whose source code is available on [GitHub](https://github.com/nanovms/gpu-nvidia/), implements a driver for NVIDIA Graphical Processing Units. It is a port to the Nanos kernel of the open-source Linux driver released by NVIDIA for its GPU cards; at the time of this writing, the gpu\_nvidia klib is based on version 535.113.01 of the NVIDIA driver.
As with any out-of-tree klib, in order to build this klib the source code of the Nanos kernel version being used needs to be available in the development machine; then, the following command can be used to build the klib (replace `/path/to/nanos/kernel/source` with the actual filesystem path where the kernel source is located):
```bash
make NANOS_DIR=/path/to/nanos/kernel/source
```
The resulting klib binary file will be located at kernel-open/\_out/Nanos\_x86\_64/gpu\_nvidia in the klib source tree. In order for this file to be found by Ops, either the file has to be copied (or symlinked) in the folder containing the in-tree klibs for the kernel version being used (e.g. \~/.ops/0.1.47/klibs/), or the `KlibDir` configuration option has be to added to the Ops configuration file to point to the folder where the klib file is located; example using the latter approach:
```json
{
"KlibDir": "/usr/src/gpu-nvidia/kernel-open/_out/Nanos_x86_64",
"Klibs": ["gpu_nvidia"]
}
```
The NVIDIA driver requires a firmware file to be available in the image, so that this file can be transferred to the GPU card during initialization. A GPU firmware file is tied to a specific kernel version; in order to retrieve the correct firmware files, download the NVIDIA Linux driver package for Linux 64-bit from , then extract the package: firmware files will be located in the `firmware` subfolder of the extracted package contents. For driver version 535.113.01, there are two separate firmware files (gsp\_ga10x.bin and gsp\_tu10x.bin), each of which is used for a subset of the supported GPUs; which of these files is needed by the driver during initialization depends on the type of NVIDIA card attached to the instance: for example, GeForce 3090 and 4090 cards use the gsp\_ga10x.bin file. The firmware file has to be put in the unikernel image in the `/nvidia//` folder; for example, to use an NVIDIA GeForce 3090 GPU with driver version 535.113.01, the /nvidia/535.113.01/gsp\_ga10x.bin file has to be included in the image.
The Ops configuration options to deploy a Nanos image to a GPU-equipped instance are "GPUs" (which specifies the number of GPUs to be attached to the instance) and "GPUType" (which specifies the type of GPU, and can be used on certain cloud providers such as GCP which allow attaching different GPU types to the same instance type). Example configuration for GCP:
```json
{
"CloudConfig" :{
"ProjectID" :"my-proj",
"Zone": "us-west1-b",
"BucketName":"my-bucket",
"Flavor":"n1-standard-1"
},
"RunConfig": {
"GPUs": 1,
"GPUType": "nvidia-tesla-t4"
}
}
```
For on-prem instances, if the host machine is equipped with one or more GPUs, it is possible to attach (a subset of) these GPUs to a Nanos instance via PCI passthrough, by specifying the "GPUs" configuration option with the desired number of GPUs. For example:
```json
{
"RunConfig": {
"GPUs": 1
}
}
```
PCI passthrough is only available on x86\_64 Linux host machines, and requires I/O virtualization to be supported and enabled in the host: this may require enabling VT-d (for Intel CPUs) or AMD IOMMU (for AMD CPUs) in the BIOS settings, and adding the `intel_iommu=on iommu=pt` (for Intel CPUs) or `amd_iommu=on iommu=pt` (for AMD CPUs) options to the Linux kernel command line. In addition, the vfio-pci Linux kernel driver must be loaded and bound to the GPU device(s) to be attached to an instance: if the driver is built into the kernel binary, add `vfio-pci.ids=::` (where `` is the PCI vendor ID and `` is the PCI device ID of the host GPU) to the kernel command line, otherwise (if the driver is built as a kernel module) ensure that the driver is loaded with the "ids" option set to the PCI vendor and device ID of the GPU (for example, create a file named /etc/modprobe.d/vfio.conf which contains `options vfio-pci ids=10de:1eb8"`). In order for VFIO devices to be accessible by the hypervisor process, their file attributes must be properly configured, e.g. with the following udev rule: `SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm"`.
# GPU
Nanos is the only unikernel with native GPU support. This is sometimes necessary for blockchain or AI based workloads. The GPU support targets most common cloud and consumer Nvidia based GPUs.
On the cloud you can specify the number and GPU type via:
```
{
"RunConfig": {
"GPUs": 1,
"GPUType": "nvidia-tesla-t4"
}
}
```
You must also include the 'gpu\_nvidia' klib and include the Nvidia firwmware:
```
"Klibs": ["gpu_nvidia"],
"Dirs": ["nvidia"]
```
Nightly binary builds are built and uploaded to:
Note: For companies over 50 employees using for commercial purposes a paid subscription is required at for using binary builds NanoVMs builds. If you don't wish to sign up for that you can build the source yourself.
For more information checkout the tutorial at and .
## OnPrem Instructions
* VT-d (Intel) or IOMMU (AMD) must be enabled in bios
* Add the proper kernel command line argument:
Intel: `intel_iommu=on iommu=pt`
AMD: `amd_iommu=on iommu=pt`
# Hardware Acceleration
Ops can utilize your CPU's virtualization extension through the various supported hypervisors. Ops will only attempt to enable acceleration on systems that support it.
If you deploy to Google Cloud or AWS hardware acceleration will always be enabled however if you are developing locally you'll typically need to turn it on manually.
Also bear in mind that currently the VDSO is tied to pvclock and that requires hardware acceleration so you'll get another speed boost from that as well.
### Linux
To have Ops enable hardware acceleration when running an image, first check to see if your system's chip has the appropriate support. An easy way to do this on modern Linux based systems is to run the following command:
```sh
$ grep -woE 'svm|vmx' /proc/cpuinfo | uniq
```
If you have a supported Intel CPU you should see the output `vmx`, for supported AMD processors you should see `svm`. If your CPU does not support a virtualization extension you will see no output. If you are certain that your CPU provides virtualization support and yet you see no output, then please ensure that the extension is enabled in your BIOS/UEFI system.
If using KVM you need to ensure that your user is a member of the `kvm` group. Ops will not attempt to enable acceleration if your user is not a member of this group. First check if your user is apart of the kvm group:
```sh
$ groups
```
If your user is a member of the group you should see `kvm` in the list. If not already a member you can add yourself to the group with the following command:
```sh
$ usermod -aG kvm `whoami`
```
The change will take effect upon the next login.
You can then verify by issuing the groups command ensuring you are in the kvm group:
```sh
$ groups
```
Finally, you can check to see if Ops is using virtualization support by inspecting the command it uses to run an image. You can do this by enabling verbose output when using the `run` or `load` commands. So assuming I have hardware acceleration enabled in my runtime configuration:
```json
{
"RunConfig": {
"Accel": true
}
}
```
Then the following command should indicate that hardware acceleration is being used:
```sh
$ ops run -v hello -c config.json
...
qemu-system-x86_64 -drive file=${HOME}/.ops/images/hello.img,format=raw,if=virtio -device virtio-net,netdev=n0 -netdev user,id=n0 -enable-kvm -nodefaults -no-reboot -device isa-debug-exit -m 2G -display none -serial stdio
...
```
Notice that the Qemu command is generated with `-enable-kvm` option.
Accelaration support can also be enabled by providing the --accel(-x) flag to the ops `run` or `load` commands.
```sh
$ ops run --accel hello # or ops run -x hello
```
### macOS
For macOS users things are a bit simpler. To check for virtualization support you can use the sysctl(8) command with the argument `kern.hv_support`:
```sh
$ sysctl kern.hv_support
```
The output will be `kern.hv_support: 1` if support is enabled and `kern.hv_support: 0` otherwise. Nothing more is needed on macOS, if hardware support is enabled in your runtime configuration then Ops generated hypervisor commands should include the appropriate acceleration options.
# Azure
OPS can run on Microsoft Azure assuming your environment is setup.
## Pre-requisites
1. Create a new resource group.
2. Create a storage account - that is your bucket.
3. You might need to login first.
```sh
az login --scope https://graph.windows.net//.default
```
4. Create a quickstart auth:
```sh
az ad sp create-for-rbac --sdk-auth > quickstart.auth
```
5. Create a role using the clientID as the assignee inside quickstart.auth:
```sh
az role assignment create --assignee client-id-goes-here --role Contributor --scope /subscriptions/sub-id-goes-here/resourceGroups/resource-group-name
```
Most of the environment variables you need will be found in this json file.
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. You need to add [CloudConfig](/ops/configuration#cloudconfig) which mentions specific details like `ProjectID`, `BucketName`.
```json
{
"CloudConfig" :{
"Zone": "us-west2-a",
"BucketName":"nanostest"
}
}
```
Once, you have updated `config.json` you can create an image in Azure with the following command.
```sh
#!/bin/sh
# used for uploading blob
export AZURE_STORAGE_ACCOUNT="nanostest"
export AZURE_STORAGE_ACCESS_KEY=""
# used for create
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
# used for everything?
export AZURE_BASE_GROUP_NAME="bob"
GOOS=linux go build -o gtest2
ops image create -c config.json -t azure -a gtest2
```
The newer version of ops uses Azure Galleries to store images as Azure has been moving away from their older image storage now. In fact most new instance types require it.
If you wish to override the default gallery name of 'nanos\_gallery' you can export an env var like so:
```
export AZURE_GALLERY_NAME="my_new_gallery"
```
### List Images
You can list existing images on Azure with `ops image list`.
```sh
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
ops image list -t azure
```
### Delete Image
```sh
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
ops image delete -t azure bob2
```
## Instance Operations
### Create Instance
```sh
export AZURE_STORAGE_ACCOUNT=""
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
export AZURE_BASE_GROUP_NAME=""
ops instance create bob -z westus2 -t azure
```
#### VPC, Subnet and Security Group
By default, ops creates a VPC, a subnet and a security group per instance.
You can select a different VPC, subnet or security group using the configuration file. The keys to set are `CloudConfig.VPC`, `CloudConfig.Subnet` and `CloudConfig.SecurityGroup`.
```json
{
"CloudConfig":{
"VPC": "vpc-name",
"Subnet": "subnet-name",
"SecurityGroup": "sg-name"
}
}
```
#### IP Forwarding
By default, IP forwarding is `disabled` on Azure.
If you would like to enable IP forwarding when creating the instance you can use the following:
```json
{
"RunConfig":{
"CanIPForward": true
}
}
```
### List Instances
```sh
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
export AZURE_BASE_GROUP_NAME=""
ops instance list -t azure -z us-west-2
```
### Get Logs for Instance
You can get logs from serial console of a particular instance using `ops instance logs` command. The logs are stored in your cloud storage bucket.
```sh
export AZURE_STORAGE_ACCOUNT=""
export AZURE_STORAGE_ACCESS_KEY=""
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT=""
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
export AZURE_BASE_GROUP_NAME=""
ops instance logs -t azure gtest2-image -z us-west-2
```
### Delete Instance
## Volume Operations
### Create Volume
Create a small 1mb min. volume locally:
```
ops volume create avol
```
Upload and set the size to 50gb remotely:
```
#!/bin/sh
export AZURE_STORAGE_ACCOUNT=""
export AZURE_STORAGE_ACCESS_KEY=""
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT="westus2"
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
export AZURE_BASE_GROUP_NAME=""
ops volume create avol -s 50g -t azure
```
### List Volumes
```
#!/bin/sh
export AZURE_STORAGE_ACCOUNT=""
export AZURE_STORAGE_ACCESS_KEY=""
export AZURE_SUBSCRIPTION_ID=""
export AZURE_LOCATION_DEFAULT="westus2"
export AZURE_TENANT_ID=""
export AZURE_CLIENT_ID=""
export AZURE_CLIENT_SECRET=""
export AZURE_BASE_GROUP_NAME=""
ops volume list -t azure
```
## Networking Considerations
### IPV6 Networking
IPV6 support differs from cloud to cloud.
You can enable IPV6 on Azure with the following config:
```json
{
"CloudConfig" :{
"EnableIPv6": true,
}
}
```
# AWS
Ops can integrate with your existing Amazon Web Services (AWS) account. You can use the Ops CLI to create and upload an image in your AWS account. Once, you have uploaded the image, you can also create an instance with a particular image using CLI.
We currently automate installing a [vmimport role](https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-troubleshooting.html). It needs to be tied to the bucket you use and uses an unique name. If you are having trouble you might wish to see if you have an existing vmimport role defined already. You can also bypass this verification/creation step by turning on the SkipImportVerify flag as such:
```json
{
"CloudConfig": {
"SkipImportVerify": true
}
}
```
## Instance type Support
Nanos supports running on many of the AWS instance types including normal instance types like t2 and the newer Nitro based instances such as t3, c5, etc.
We recently introduced support for ARM based instances in the form of AWS Graviton instances 2 and 3. We also support SMP there.
## Pre-requisites
1. Ensure your \~/.aws/credentials file is setup correctly - you can use the 'aws ec2' cli tools to verify this.
2. Create a bucket in S3 storage for ami creation.
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. You need to add [CloudConfig](/ops/configuration#cloudconfig) which mentions specific details like `BucketName`.
```json
{
"CloudConfig" :{
"Zone": "us-west-1",
"BucketName":"my-s3-bucket"
}
}
```
Once, you have updated `config.json` you can create an image in AWS with the following command:
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -t aws -c config.json -p node_20.5.0 -a ex.js
```
### List Images
You can list existing images on AWS with `ops image list`.
```sh
$ ops image list -t aws -z us-west-1
+----------------+-----------+--------------------------+
| NAME | STATUS | CREATED |
+----------------+-----------+--------------------------+
| nanos-webg-ata | available | 2019-07-16T05:56:06.000Z |
+----------------+-----------+--------------------------+
| will-00 | available | 2019-08-07T20:21:19.000Z |
+----------------+-----------+--------------------------+
| nanos-webg | available | 2019-07-16T05:01:44.000Z |
+----------------+-----------+--------------------------+
| node-image | available | 2019-08-15T22:31:03.000Z |
+----------------+-----------+--------------------------+
```
### Delete Image
`ops image delete -i ` can be used to delete an image from AWS.
```sh
$ ops image delete -i nanos-main-image
```
## Instance Operations
### Create Instance
After the successful creation of an image in AWS, we can create an instance from an existing image.
```sh
$ ops instance create
```
Alternatively, you can pass project-id and zone with cli options.
```sh
$ ops instance create -p prod-1000 -z us-west-2
```
You can also pass config, if you have mentioned project-id and zone in project's config.json.
```sh
$ ops instance create -c config.json
```
You can provide list of ports to be exposed on aws instance via config and command line.
CLI example:
```sh
$ ops instance create -p prod-1000 -z us-west-2 --port 80 --port 443
```
Sample config
```json
{
"CloudConfig" :{
"Platform" :"aws",
"Zone": "us-west-1",
"BucketName":"my-s3-bucket"
},
"RunConfig": {
"Ports" : ["80", "443"]
}
}
```
If you need to specify a specific zone when creating the instance use the [Subnet config var](https://nanovms.gitbook.io/ops/configuration#cloudconfig.subnet).
This is necessary for certain operations such as when you want to mount a volume to an instance.
#### VPC and Security Group
By default, ops uses the first VPC found in aws and creates a security group per instance.
You can select a different VPC or use a existing security group using the configuration file. The keys to set are `CloudConfig.VPC` and `CloudConfig.SecurityGroup`.
```json
{
"CloudConfig":{
"VPC": "vpc-name",
"SecurityGroup": "sg-name"
}
}
```
#### Private Static IP
By default, ops uses will rely on DHCP.
If you would like to set a static private ip you can use the following:
```json
{
"RunConfig":{
"IPAddress": "172.31.33.7"
}
}
```
**Note**: You must choose an **available IP** that is *within your chosen/default VPC*.
### List Instances
You can list instance on AWS using `ops instance list` command.
```sh
$ ops instance list -p prod-1033 -t aws -z us-west-2
$ ops instance list
+------+----+--------+---------+-------------+------------+
| NAME | ID | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS |
+------+----+--------+---------+-------------+------------+
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance list -p prod-1000 -z us-west1-b
```
### Get Logs for Instance
You can get logs from serial console of a particular instance using `ops instance logs` command.
```sh
$ ops instance logs
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance logs -t aws -p prod-1033 -z us-west-2 i-08815dc4b29b44294
```
On Nitro based systems the serial console will only show output when you are connected to it.
For production use we recommend shipping your logs to [syslog](https://docs.ops.city/ops/klibs#syslog) or using cloudwatch.
To utilize cloudwatch you need to specify an IAM role (`CloudConfig.InstanceProfile`), include the `cloudwatch` and `tls` klibs and specify your log group and log stream like so:
```json
{
"Klibs": ["cloudwatch", "tls"],
"RunConfig": {
"Ports": ["8080"]
},
"CloudConfig" :{
"BucketName":"nanos-test",
"InstanceProfile": "CloudWatchAgentServerRole"
},
"ManifestPassthrough": {
"cloudwatch": {
"mem_metrics_interval": "5",
"logging": {
"log_group": "my-log-group",
"log_stream":"my_log_stream"
}
}
}
}
```
Then you can tail your logs in real-time:
```sh
aws logs tail my-log-group --follow
```
Furthermore, it should be stated that shipping a lot of output through the serial console is going to degrade performance. You can explicitly disable both serial and vga using the following config:
```json
{
"ManifestPassthrough": {
"consoles": {"-serial", "-vga"}
}
}
```
### Delete Instance
`ops instance delete` command can be used to delete instance on AWS.
```sh
$ ops instance delete my-instance-running
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance delete -p prod-1000 -z us-west1-b my-instance-running
```
### Create Instance with Instance Group
OPS has initial support for putting an instance into an AWS auto scaling group. This allows you to load balance a handful of instances and scale up/down on demand.
The instance group must already be created to use this feature. When deploying through 'instance create' OPS will create a new launch template, apply it to the AWS auto scaling group, and then attach it to the instance when creating.
```sh
$ ops instance create -t aws -z us-west1-a --instance-group my-instance-group
```
### Elastic IP:
If you have already provisioned an elastic ip you may use it by setting it in the Cloud Config:
```json
{
"CloudConfig" :{
"StaticIP": "1.2.3.4"
}
}
```
### IPV6 Networking
IPV6 support differs from cloud to cloud. On AWS DHCPV6 is used. You can have an ip auto-assigned or you can set one yourself but you must be aware that a subnet must have IPv6 enabled.
A sample config for assigning an ip:
```json
➜ g cat config.json
{
"CloudConfig" :{
"Zone": "us-west-1",
"BucketName":"my-bucket",
"EnableIPv6": true,
"VPC":"test-ipv6-test-2"
},
"RunConfig": {
"IPv6Address": "2600:1f1c:604:9a00:7a34:8c37:5b08:f104",
"Ports": [
"80",
"8080",
"443"
]
}
}
```
To test:
```sh
[ec2-user@ip-172-33-75-200 ~]$ ping6 2600:1f1c:604:9a00:7a34:8c37:5b08:f104
PING 2600:1f1c:604:9a00:7a34:8c37:5b08:f104(2600:1f1c:604:9a00:7a34:8c37:5b08:f104) 56 data bytes
64 bytes from 2600:1f1c:604:9a00:7a34:8c37:5b08:f104: icmp_seq=1 ttl=255 time=0.573 ms
64 bytes from 2600:1f1c:604:9a00:7a34:8c37:5b08:f104: icmp_seq=2 ttl=255 time=0.419 ms
^C
```
Be aware that you might not have IPV6 connectivity from the laptop/server you are testing from. You can verify within an instance on AWS to test connectivity.
### Volumes
When you mount a volume to an AWS instance you must specify the zone modifier for the region. So instead of using:
```json
"Zone": "us-west-1",
```
Use
```json
"Zone": "us-west-1c",
```
OPS will strip the zone 'c' from other operations that only require/desire region.
To set the volume type:
```
ops volume create myvolume --typeof gp3 -z us-west-1c -t aws -c config.json
```
To set iops:
(Note: This is required for io1 and io2 volumes. It is not supported for the default of gp2. There is also a 50:1 iops to volume size ratio that is required.)
```
ops volume create myvolume -s 100 --iops 5000 --typeof gp3 -z us-west-1c -t aws -c config.json
```
To set throughput:
(Note: This is only valid for gp3 volumes.)
```
ops volume create myvolume --throughput 500 --typeof gp3 -z us-west-1c -t aws -c config.json
```
# Digital Ocean
Ops can integrate with your existing Digital Ocean account.
## Pre-requisites
1. Create an account in [Digital Ocean](https://cloud.digitalocean.com/);
2. Access [Application & API page](https://cloud.digitalocean.com/account/api/tokens):
* Generate a token and export the value to the environment variable `DO_TOKEN`.
* Generate space access keys and export the key and the secret to the environment variables `SPACES_KEY` and `SPACES_SECRET`.
3. Access [Spaces page](https://cloud.digitalocean.com/spaces) and create a space;
4. Set your bucket name with the space name and set the region name where the space was created in the configuration file;
Configuration example.
```json
{
"CloudConfig": {
"BucketName": "ops",
"Zone": "sfo3"
}
}
```
## Image Operations
### Create Image
You can create an image in Digital Ocean with the following command.
```sh
$ ops image create -t do -c config.json
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t do -c config.json
```
### List Images
You can list existing images on Digital Ocean with `ops image list -t do -c config.json`.
```sh
$ ops image list -t do -c config.json
+--------------------------------------+------------------------+--------+---------+------------------+
| UUID | NAME | STATUS | SIZE | CREATEDAT |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01071a8d-eb59-49c1-aa69-028f84cc6d06 | nanos-main-image | online | 10.0 GB | 3 days ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01c5bd28-aa19-4883-a4c9-31f3ff9fd061 | nanos-node-image | online | 10.0 GB | 1 month ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01cd3190-df52-47e8-b5c3-b05f7107819e | nanos-server-image | online | 10.0 GB | 1 year ago |
+--------------------------------------+------------------------+--------+---------+------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from Digital Ocean.
```
$ ops delete image nanos-main-image -t do -c config.json
```
## Instance Operations
### Create Instance
After the successful creation of an image in Digital Ocean, we can create an instance from an existing image.
```
$ ops instance create -t do -c config.json -i
```
### List Instances
You can list instance on Digital Ocean using `ops instance list` command.
```sh
$ ops instance list -t do -c config.json
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| ID | NAME | STATUS | PRIVATE IPS | PUBLIC IPS |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| 00d954cc-9603-43a0-915b-5c2ae75772b6 | nanos-main-image-15566 | stopped | 10.8.3.63 | 209.151.144.166 |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
```
### Get Logs for Instance
Work in progress.
### Delete Instance
`ops instance delete` command can be used to delete instance on Digital Ocean.
```sh
$ ops instance delete my-instance-running -t do -c config.json
```
# Google Cloud Integration
Ops can integrate with your existing Google Cloud Platform (GCP) account. You can use Ops CLI to create and upload an image in GCP account. Once, you have uploaded image, you can also create an instance with a particular image using CLI.
By using the **gcp** *klib* it is possible to send `memory usage metrics` to the GCP monitoring service, thus emulating the **GCP ops agent**.
## Pre-requisites
1. Create a Service Account (SA) in your GCP account and download the Service Account key json file.
2. Please make sure your Service Account has access to the Google Compute Engine and Google Storage.
3. Get the name of your Google Cloud account project where you would be creating images and instances.
4. Create a bucket in Google Cloud storage for image artifacts storage.
5. Please make sure you export `GOOGLE_APPLICATION_CREDENTIALS` with the Service Account key json file path, before invoking below commands.
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS=~/service-key.json
```
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. You need to add [CloudConfig](/ops/configuration#cloudconfig) which mentions specific details like `ProjectID`, `BucketName`.
```json
{
"CloudConfig": {
"ProjectID": "prod-1000",
"Zone": "us-west1-b",
"BucketName":"my-deploy"
},
"RunConfig": {
"Memory": "2G"
}
}
```
Once, you have updated `config.json` you can create an image in Google Cloud with the following command.
```sh
$ ops image create -c config.json -i -t gcp
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t gcp
```
Nanos supports running ARM payloads on ARM instances but in order to do so you must build your image with an ARM instance type:
```json
{
"CloudConfig": {
"Flavor":"t2a-standard-1"
}
}
```
Also note that this instance type is not supported in every region. You can try us-central1-a.
### List Images
You can list existing images on Google cloud with `ops image list`.
```sh
$ ops image list
+--------------------+--------+-------------------------------+
| NAME | STATUS | CREATED |
+--------------------+--------+-------------------------------+
| nanos-main-image | READY | 2019-03-21T15:06:17.567-07:00 |
+--------------------+--------+-------------------------------+
| nanos-node-image | READY | 2019-04-16T23:16:03.145-07:00 |
+--------------------+--------+-------------------------------+
| nanos-server-image | READY | 2019-03-21T15:50:04.456-07:00 |
+--------------------+--------+-------------------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from Google Cloud.
```sh
$ ops delete image nanos-main-image
```
## Instance Operations
### Create Instance
After the successful creation of an image in Google Cloud, we can create an instance from an existing image.
You need to export `GOOGLE_APPLICATION_CREDENTIALS` and pass project-id and zone with cli options.
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS=
$ ops instance create -g prod-1000 -z us-west1-b -t gcp
```
Alternatively, you can pass config, if you have mentioned project-id and zone in project's config.json.
```sh
$ ops instance create -t gcp -c config.json
```
You can provide list of ports to be exposed on gcp instance via config and command line.
CLI example
```sh
$ ops instance create -t gcp -p prod-1000 -z us-west1-a --port 80 --port 443
```
Sample config
```json
{
"CloudConfig" :{
"Platform" :"gcp",
"ProjectID" :"prod-1000",
"Zone": "us-west1-a",
"BucketName":"my-s3-bucket",
"InstanceProfile":"default"
},
"RunConfig": {
"Ports" : ["80", "443"]
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"metrics": {"interval":"120"}
}
}
}
```
#### Spot Provisioning
You maybe enable spot provisioning using the following config:
```json
{
"CloudConfig": {
"Spot": true
}
}
```
#### Disable SMT
You can disable SMT if you so desire by flagging 'ThreadsPerCore' to 1. By default Nanos will have acess to all vCPUs available but this setting can force it to only use one thread per core. Some performance, security or licensing concerns might benefit from this setting. It is important to note that not all instances allow this setting and you will still be billed for all vCPUs provisioned.
```json
{
"RunConfig": {
"ThreadsPerCore": 1
}
}
```
#### AMD-SEV (Secure Encrypted Virtualization)
You can enable encryption in-use AMD-SEV on select flavors and regions in Google Cloud. This will generate an encryption key that is generated for each new vm.
```
{
"CloudConfig" : {
"ConfidentialVM": true,
"Flavor": "n2d-standard-2"
}
}
```
#### Private Static IP
By default, ops uses will rely on DHCP.
If you would like to set a static private ip you can use the following:
```json
{
"RunConfig":{
"IPAddress": "172.31.33.7"
}
}
```
Note: You must choose an available IP that is within your chosen/default VPC.
#### IP Forwarding
By default, IP forwarding is `disabled` on GCP.
If you would like to enable IP forwarding when creating the instance you can use the following:
```json
{
"RunConfig":{
"CanIPForward": true
}
}
```
#### GCP metrics - memory
The **gcp** *klib* emulates some functions of GCP ops agent to send **memory usage metrics** to the **GCP monitoring service**.
Example Ops configuration to enable sending memory metrics every `2 minutes`:
```json
{
"CloudConfig" :{
"Platform" :"gcp",
"ProjectID" :"prod-1000",
"Zone": "us-west1-a",
"BucketName":"my-s3-bucket",
"InstanceProfile":"default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"metrics": {
"interval":"120"
}
}
}
}
```
#### GCP logging - console
The **gcp** *klib* implements a *console driver* that sends *console output* to GCP logs.
```json
{
"CloudConfig" :{
"Platform" :"gcp",
"ProjectID" :"prod-1000",
"Zone": "us-west1-a",
"BucketName":"my-s3-bucket",
"InstanceProfile":"default"
},
"Klibs": ["gcp", "tls"],
"ManifestPassthrough": {
"gcp": {
"logging": {
"log_id": "my_log"
}
}
}
}
```
### List Instances
You can list instance on Google Cloud using `ops instance list` command.
You need to export `GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_ZONE` before firing command.
```sh
$ export GOOGLE_CLOUD_PROJECT=prod-1000
$ export GOOGLE_CLOUD_ZONE=us-west1-b
$ ops instance list
+-----------------------------+---------+-------------------------------+-------------+--------------+
| NAME | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS |
+-----------------------------+---------+-------------------------------+-------------+--------------+
| nanos-main-image-1556601450 | RUNNING | 2019-04-29T22:17:34.609-07:00 | 10.240.0.40 | 34.83.204.40 |
+-----------------------------+---------+-------------------------------+-------------+--------------+
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance list -g prod-1000 -z us-west1-b
```
### Get Logs for Instance
You can get logs from serial console of a particular instance using `ops instance logs` command.
You need to export `GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_ZONE` before firing command.
```sh
$ export GOOGLE_CLOUD_PROJECT=prod-1000
$ export GOOGLE_CLOUD_ZONE=us-west1-b
$ ops instance logs -t gcp
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance logs -g prod-1000 -z us-west1-b
```
You may also tail the serial console using:
```
$ ops instance logs --watch my-running-instance -g prod-1000 -t gcp -z us-west2-a
```
### Delete Instance
`ops instance delete` command can be used to delete instance on Google Cloud.
You need to export `GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_ZONE` before firing command.
```sh
$ export GOOGLE_CLOUD_PROJECT=prod-1000
$ export GOOGLE_CLOUD_ZONE=us-west1-b
$ ops instance delete my-instance-running
```
Alternatively you can pass project-id and zone with cli options.
```sh
$ ops instance delete -g prod-1000 -z us-west1-b my-instance-running
```
### Create Instance with Instance Group
OPS has initial support for putting an instance into an instance group. This allows you to load balance a handful of instances and scale up/down on demand.
The instance group must already be created to use this feature. When deploying through 'instance create' OPS will create a new instance template, apply it to the instance group, and then force re-create all the instances with the new instance template. The instance template will track any firewall rule changes (such as ports).
```sh
$ ops instance create -t gcp -p prod-1000 -z us-west1-a --port 80 --port 443 --instance-group my-instance-group
```
## Volume Operations
### Create Volume
You need to set the `BucketName`, `ProjectID` and `Zone` in the `CloudConfig` section of your configuration file and export `GOOGLE_APPLICATION_CREDENTIALS` before firing the command.
```json
{
"CloudConfig" :{
"ProjectID" :"prod-1000",
"Zone": "us-west1-b",
"BucketName":"my-deploy"
}
}
```
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS=
$ ops volume create -t gcp -c
```
For create a volume with existing files you can add the `-d` flag and the directory path.
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS=
$ ops volume create -t gcp -c -d
```
### List Volumes
You can list volumes on Google Cloud using `ops volume list -t gcp -c ` command.
You need to set the `ProjectID` and `Zone` in the `CloudConfig` section of your configuration file and export `GOOGLE_APPLICATION_CREDENTIALS` before firing the command.
```sh
$ ops instance list -t gcp -c
+-----------------------------+---------+-------------------------------+-------------+--------------+
| NAME | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS |
+-----------------------------+---------+-------------------------------+-------------+--------------+
| nanos-main-image-1556601450 | RUNNING | 2019-04-29T22:17:34.609-07:00 | 10.240.0.40 | 34.83.204.40 |
+-----------------------------+---------+-------------------------------+-------------+--------------+
```
### Delete Volume
`ops volume delete` command can be used to delete an instance on Google Cloud.
You need to set the `ProjectID` and `Zone` in the `CloudConfig` section of your configuration file and export `GOOGLE_APPLICATION_CREDENTIALS` before firing the command.
```sh
$ export GOOGLE_APPLICATION_CREDENTIALS=
$ ops volume delete -t gcp -c
```
### Attach Volume
For attaching a volume you need a running instance using a image configured with a mount point. This means you have to create a volume before running the instance. After the volume created you have to specify the volume label with the same name of the volume created. You can create the image running the next command.
```sh
$ ops image create -i -c config.json --mounts :
```
After having the instance running you can attach a volume using `ops volume attach -t gcp -c `.
**Note:** You need to stop and start the instance to see the changes applied.
### Detach Volume
You can detach a volume from a running instance using `ops volume detach -t gcp -c `.
## Networking Considerations
If you specify a port in your config you are stating you wish the public ip associated with the instance to be exposed with that port. If you don't specify the port by default the private ip allows any instance in the same vpc to talk to it.
### Elastic IP:
If you have already provisioned an elastic ip you may use it by setting it in the Cloud Config:
```json
{
"CloudConfig" :{
"StaticIP": "1.2.3.4"
}
}
```
### IPV6 Networking
IPV6 support differs from cloud to cloud.
To use IPv6 on Google Cloud you must create a VPC and a subnet with IPv6 enabled. You can not use the legacy network nor can you use an auto-created subnet.
After you create a new VPC and subnet you can adjust the subnet to be dual stack like so:
```sh
$ gcloud compute networks subnets update mysubnet \
--stack-type=IPV4_IPV6 --ipv6-access-type=EXTERNAL --region=us-west2
```
When you create it you won't see in the UI that it is IPv6 enabled but you can click the 'REST' button to see it.
A sample config:
```json
{
"CloudConfig" :{
"ProjectID": "my-project",
"Zone": "us-west2-a",
"BucketName":"nanos-test",
"EnableIPv6": true,
"VPC": "ipv6-test",
"Subnet": "ipv6-test"
},
"RunConfig": {
"Ports": [
"80",
"8080",
"443"
]
}
}
```
Be aware that you might not have IPV6 connectivity from the laptop/server you are testing from. You can verify within an instance on Google or some other IPv6 capable machine via telnet:
```sh
$ telnet 2600:1900:4120:1235:: 8080
```
or ping:
```sh
$ ping6 2600:1900:4120:1235::
```
Also, keep in mind that when you create a new VPC by default there are no firewall rules so things like ICMP (ping) won't work without adding them manually nor would ssh'ing into a test instance work without a corresponding rule on the new VPC for ssh (22).
# Hetzner
Ops can deploy unikernels to Hetzner by utilizing a build-server approach.
Most operations will require the following env vars to be set:
```
export HCLOUD_TOKEN=api-token
export OBJECT_STORAGE_DOMAIN=hel1.your-objectstorage.com
export OBJECT_STORAGE_KEY=storage-pub-key
export OBJECT_STORAGE_SECRET=storage-priv-key
```
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. Set uefi to true and set your bucket.
```json
{
"Uefi": true,
"CloudConfig": {
"BucketName": "nanostest"
}
}
```
Once, you have updated `config.json` you can create an image in Hetzner with the following command.
```sh
$ ops image create -t hetzner -c config.json
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t hetzner
```
### List Images
You can list existing images on Hetzner with:
`ops image list -t hetzner`.
### Delete Image
`ops image delete ` can be used to delete an image from Hetzner.
```sh
$ ops delete image nanos-main-image -t hetzner
```
## Instance Operations
### Create Instance
Alternatively, you can pass config, if you have mentioned project-id and zone in project's config.json.
```sh
$ ops instance create -t hetzner -c config.json myprogram -p 8080
```
### List Instances
```
ops instance list -t hetzner -c config.json
```
### Get Logs for Instance
This currently is not supported.
### Delete Instance
`ops instance delete` command can be used to delete instance on Hetzner.
# IBM
Ops can integrate with your existing IBM Cloud Platform (IBM) account. You can use Ops CLI to create and upload an image to your account. Once, you have uploaded image, you can also create an instance with a particular image using CLI.
## Pre-requisites
[Create an api key](https://cloud.ibm.com/iam/apikeys) and export it:
```sh
$ export TOKEN="somethinguniqueandrandom"
```
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. You need to add [CloudConfig](/ops/configuration#cloudconfig) which mentions specific details like `BucketName`.
```json
{
"RunConfig": {
"Ports": ["8080"]
},
"CloudConfig" :{
"Zone": "us-south-2",
"BucketName":"nanos-test"
}
}
```
Once, you have updated `config.json` you can create an image in IBM Cloud with the following command.
```sh
$ ops image create -t ibm -c config.json
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t ibm
```
### List Images
You can list existing images on IBM cloud with `ops image list`.
```sh
$ ops image list
+--------------------+--------+-------------------------------+
| NAME | STATUS | CREATED |
+--------------------+--------+-------------------------------+
| nanos-main-image | READY | 2019-03-21T15:06:17.567-07:00 |
+--------------------+--------+-------------------------------+
| nanos-node-image | READY | 2019-04-16T23:16:03.145-07:00 |
+--------------------+--------+-------------------------------+
| nanos-server-image | READY | 2019-03-21T15:50:04.456-07:00 |
+--------------------+--------+-------------------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from IBM Cloud.
```
$ ops delete image nanos-main-image
```
## Instance Operations
### Create Instance
After the successful creation of an image in IBM Cloud, we can create an instance from an existing image.
You need to export `TOKEN`.
```sh
$ export TOKEN=
$ ops instance create -t ibm -c config.json
```
### List Instances
You can list instance on IBM Cloud using `ops instance list` command.
You need to export `TOKEN`:
```sh
$ export TOKEN=token
$ ops instance list
+-----------------------------+---------+-------------------------------+-------------+--------------+
| NAME | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS |
+-----------------------------+---------+-------------------------------+-------------+--------------+
| nanos-main-image-1556601450 | RUNNING | 2019-04-29T22:17:34.609-07:00 | 10.240.0.40 | 34.83.204.40 |
+-----------------------------+---------+-------------------------------+-------------+--------------+
```
Alternatively you can pass zone with cli options.
```sh
$ ops instance list -t ibm
```
### Get Logs for Instance
### Delete Instance
`ops instance delete` command can be used to delete instance on IBM Cloud.
You need to export `TOKEN`.
```sh
$ export TOKEN=mytoken
$ ops instance delete my-instance-running
```
Alternatively you can pass zone with cli options.
```sh
$ ops instance delete -z us-south-2 my-instance-running
```
## Volume Operations
### Create Volume
### List Volumes
### Delete Volume
### Attach Volume
### Detach Volume
# Linode
OPS has support for creating unikernel images and deploying them as instances to Linode.
## Pre-requisites
Create a token and export it:
```sh
$ export TOKEN="somethinguniqueandrandom"
```
## Image Operations
### Create Image
```sh
$ ops image create -t linode -c config.json
```
### List Images
You can list existing images on Linode with `ops image list`.
```sh
$ ops image list -t linode
+--------------------+--------+-------------------------------+
| NAME | STATUS | CREATED |
+--------------------+--------+-------------------------------+
| nanos-main-image | READY | 2019-03-21T15:06:17.567-07:00 |
+--------------------+--------+-------------------------------+
| nanos-node-image | READY | 2019-04-16T23:16:03.145-07:00 |
+--------------------+--------+-------------------------------+
| nanos-server-image | READY | 2019-03-21T15:50:04.456-07:00 |
+--------------------+--------+-------------------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from Linode.
```
$ ops delete image -t linode nanos-main-image
```
## Instance Operations
### Create Instance
You need to export `TOKEN`.
```sh
$ export TOKEN=
$ ops instance create -t linode -c config.json
```
### List Instances
You can list instance on Linode using `ops instance list` command.
You need to export `TOKEN`:
```sh
$ export TOKEN=token
$ ops instance list -t linode
+-----------------------------+---------+-------------------------------+-------------+--------------+
| NAME | STATUS | CREATED | PRIVATE IPS | PUBLIC IPS |
+-----------------------------+---------+-------------------------------+-------------+--------------+
| nanos-main-image-1556601450 | RUNNING | 2019-04-29T22:17:34.609-07:00 | 10.240.0.40 | 34.83.204.40 |
+-----------------------------+---------+-------------------------------+-------------+--------------+
```
### Get Logs for Instance
### Delete Instance
## Volume Operations
### Create Volume
### List Volumes
### Delete Volume
### Attach Volume
### Detach Volume
# OCI
Ops can integrate with your existing Oracle Cloud Infrastructure account. You can use Ops CLI to create and upload an image in OCI account. Once, you have uploaded image, you can also create an instance with a particular image using CLI.
## Pre-requisites
1. Create an Oracle Cloud Infrastructure account ().
2. Set up a config file with the required credentials.
1. Go to `User Settings` page, whose link should be seen after clicking on profile avatar in top navigation bar.
2. Click on `API Keys` link.
3. Click on `Add API Key` button. A modal with further instructions will appear.
4. Download the private key.
5. Click on `Add` button. You should see a configuration preview in the modal.
6. Copy configuration preview content to `~/.oci/config` file in your machine. Create the directory `.oci` in your home folder if it does not exist yet.
7. Update the `key_file` in OCI configuration file with the path of the file you downloaded in step 4.
3. Setup a bucket in your OCI account to receive your images.
1. Access `Object Storage` page.
2. Click on `Create Bucket` button.
3. Access bucket details page and copy `Namespace` value to ops configuration file.
4. Copy your bucket details to ops configuration.
```
{
"CloudConfig": {
"BucketName": "",
"BucketNamespace": ""
}
}
```
5. You are now able to use ops commands to interact with your OCI Account
```
# OCI configuration file preview
[DEFAULT]
user=ocid1.user.oc1..
fingerprint=38:9d:75:00:71:6c:b7:e0:21:d4:08:82:e4:db:bc:a7
tenancy=ocid1.tenancy.oc1..
region=
key_file= # TODO
```
## Image Operations
### Create Image
You can create an image in OCI with the following command.
```sh
$ ops image create -t oci -c config.json
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t oci -c config.json
```
### List Images
You can list existing images on OCI with `ops image list -t oci -c config.json`.
```sh
$ ops image list -t oci -c config.json
+--------------------------------------+------------------------+--------+---------+------------------+
| UUID | NAME | STATUS | SIZE | CREATEDAT |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01071a8d-eb59-49c1-aa69-028f84cc6d06 | nanos-main-image | online | 10.0 GB | 3 days ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01c5bd28-aa19-4883-a4c9-31f3ff9fd061 | nanos-node-image | online | 10.0 GB | 1 month ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01cd3190-df52-47e8-b5c3-b05f7107819e | nanos-server-image | online | 10.0 GB | 1 year ago |
+--------------------------------------+------------------------+--------+---------+------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from OCI.
```
$ ops delete image nanos-main-image -t oci -c config.json
```
## Instance Operations
### Create Instance
After the successful creation of an image in OCI, we can create an instance from an existing image.
```
$ ops instance create -t oci -c config.json
```
### List Instances
You can list instance on OCI using `ops instance list` command.
```sh
$ ops instance list -t oci -c config.json
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| ID | NAME | STATUS | PRIVATE IPS | PUBLIC IPS |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| 00d954cc-9603-43a0-915b-5c2ae75772b6 | nanos-main-image-15566 | stopped | 10.8.3.63 | 209.151.144.166 |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
```
### Get Logs for Instance
Work in progress.
### Delete Instance
`ops instance delete` command can be used to delete instance on OCI.
```sh
$ ops instance delete my-instance-running -t oci -c config.json
```
# Scaleway
Ops can deploy unikernels to Scaleway.
Most operations will require the following env vars to be set:
```
export SCALEWAY_ACCESS_KEY_ID="access-key-id"
export SCALEWAY_SECRET_ACCESS_KEY="secret-access-key"
export SCALEWAY_ORGANIZATION_ID="my-org-id"
```
## Image Operations
### Create Image
If you have already created an Ops project, you can update your existing `config.json`. Set uefi to true and set your bucket.
```json
{
"CloudConfig": {
"BucketName": "nanos-test",
"Zone": "pl-waw-1"
}
}
```
Once, you have updated `config.json` you can create an image in Scaleway with the following command.
```sh
$ ops image create -t scaleway -c config.json
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t scaleway
```
### List Images
You can list existing images on Scaleway with:
`ops image list -t scaleway`.
### Delete Image
`ops image delete ` can be used to delete an image from Scaleway.
```sh
$ ops delete image nanos-main-image -t scaleway
```
## Instance Operations
### Create Instance
Alternatively, you can pass config, if you have mentioned project-id and zone in project's config.json.
```sh
$ ops instance create -t scaleway -c config.json myprogram -p 8080
```
### List Instances
```
ops instance list -t scaleway -c config.json
```
### Get Logs for Instance
This currently is not supported.
### Delete Instance
`ops instance delete` command can be used to delete instance on Scaleway.
# UpCloud
Ops can integrate with your existing UpCloud account. You can use Ops CLI to create and upload an image in UpCloud account. Once, you have uploaded image, you can also create an instance with a particular image using CLI.
## Pre-requisites
1. Create an UpCloud account ().
2. Set next environment variables.
```sh
$ export UPCLOUD_USER=
$ export UPCLOUD_PASSWORD=
$ export UPCLOUD_ZONE=
```
**Note:** Check the available zones in [Upcloud Documentation](https://developers.upcloud.com/1.2/5-zones/).
You also need to enable API access:
## Image Operations
Images for UpCloud are created by directly creating storage through the api not by uploading to a bucket.
### Create Image
You can create an image in UpCloud with the following command.
```sh
$ ops image create -i -t upcloud
```
Note: Creating an image can take a while in certain regions. There is a default timeout of 10min but can be adjusted via . You can see the status live via as well.
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `-p` option.
```sh
$ ops image create -c config.json -p node_v14.2.0 -a ex.js -i -t upcloud
```
### List Images
You can list existing images on Upcloud with `ops image list -t upcloud`.
```sh
$ ops image list -t upcloud
+--------------------------------------+------------------------+--------+---------+------------------+
| UUID | NAME | STATUS | SIZE | CREATEDAT |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01071a8d-eb59-49c1-aa69-028f84cc6d06 | nanos-main-image | online | 10.0 GB | 3 days ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01c5bd28-aa19-4883-a4c9-31f3ff9fd061 | nanos-node-image | online | 10.0 GB | 1 month ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01cd3190-df52-47e8-b5c3-b05f7107819e | nanos-server-image | online | 10.0 GB | 1 year ago |
+--------------------------------------+------------------------+--------+---------+------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image from UpCloud.
```
$ ops delete image nanos-main-image -t upcloud
```
## Instance Operations
### Create Instance
After the successful creation of an image in UpCloud, we can create an instance from an existing image.
```sh
$ ops instance create -t upcloud
```
### List Instances
You can list instance on UpCloud using `ops instance list` command.
```sh
$ ops instance list -t upcloud
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| ID | NAME | STATUS | PRIVATE IPS | PUBLIC IPS |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| 00d954cc-9603-43a0-915b-5c2ae75772b6 | nanos-main-image-15566 | stopped | 10.8.3.63 | 209.151.144.166 |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
```
### Get Logs for Instance
Work in progress.
### Delete Instance
`ops instance delete` command can be used to delete instance on UpCloud.
```sh
$ ops instance delete my-instance-running -t upcloud
```
# Vultr
OPS works perfectly fine with Vultr.
For most operations you'll want to use the following config and you'll need access to your API key and the access key and secret access key found in your portal:
```json
{
"CloudConfig": {
"Zone": "ewr1",
"BucketName": "nanos-test"
}
}
```
```sh
package main
import (
"fmt"
"net/http"
)
func main() {
fmt.Println("hello world!")
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to my website!")
})
fs := http.FileServer(http.Dir("static/"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.ListenAndServe("0.0.0.0:8080", nil)
}
```
## Image Operations
### Create Image
```sh
VULTR_TOKEN=my_api_token \
VULTR_ACCESS=my_access_token \
VULTR_SECRET=my_vultr_secret \
ops image create -i -c config.json -z ewr1 -t vultr
```
### List Images
```sh
VULTR_TOKEN=my_api_token \
VULTR_ACCESS=my_access_token \
VULTR_SECRET=my_vultr_secret \
ops image list -z ewr1 -t vultr
```
### Delete Image
## Instance Operations
### Create Instance
```sh
VULTR_TOKEN=my_api_token \
VULTR_ACCESS=my_access_token \
VULTR_SECRET=my_vultr_secret \
ops instance create 6f5e4eebdf761 -z ewr1 -t vultr
```
You may assign a static ("reserved ip") using the following config:
```sh
"CloudConfig": {
"StaticIP": "1.2.3.4"
}
```
### List Instances
```sh
VULTR_TOKEN=my_api_token \
VULTR_ACCESS=my_access_token \
VULTR_SECRET=my_vultr_secret \
ops instance list -z ewr1 -t vultr
```
### Get Logs for Instance
### Delete Instance
# Bhyve
Nanos works on Bhyve with some caveats:
* Nanos uses ELF files so this support is intended more to run Nanos images not necessarily build them since the default file format on FreeBSD is a.out.
Nanos has been test on 13.6.0 and 14.x.
Install bhyve firmware:
```
pkg install sysutils/bhyve-firmware
```
Create a tap/bridge:
```
# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm tap0
# ifconfig bridge0 up
```
Create/run a Nanos VM:
```
# bhyve -AHP -s 0:0,hostbridge -s 1:0,virtio-blk,./disk.raw \
-s 2:0,virtio-net,tap0 -s 3:0,virtio-rnd -s 31:0,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio nanos
```
To stop the VM:
```
killall bhyve
```
When building with OPS include the UEFI loader in your config.json:
```
{
"Uefi": true
}
```
# Compose
OPS has 'docker-compose' style functionality where you can orchestrate locally many unikernels. This allows you to bring up a set of services together and utilize service discovery.
You can either download the official installer from with the functionality turned on or you can enable it manually. We highly recommend using the installer if you can.
If you run into any issue please let us know via the ops github.
## Special Instructions for Source
If you aren't using the official installer you'll need to do the following:
First find out which qemu you are using:
```
which qemu-system-x86_64
```
Then using that path (if for instance it is in /usr/local/bin):
```
sudo chown -R root:admin /usr/local/bin/qemu
chmod u+s /usr/local/bin/qemu-system-x86_64
```
This is done in order to utilize bridge networking. There are other ways of accomplishing this as well.
Next you'll need to enable the OPSD build flag:
```
go build -ldflags "-X github.com/nanovms/ops/qemu.OPSD=true"
```
This is to ensure a seamless experience for those using or not using this feature.
## Basics
Now you create a compose.yaml file. For example we have 2 packages in our local \~/.ops/local\_packages/amd64:
```
packages:
- pkg: myserver
name: mynewserver_0.0.1
local: true
- pkg: myclient
name: myclient_0.0.1
local: true
```
If you've used 'ops run' in the past you can create a new pkg using the same workflow like so:
```
ops pkg from-run client --name myclient --version 0.0.1
```
The 'pkg' itself is what will be used for dns resolution while the name is pointing towards the package on your local system.
Here is an example where we have used two different packages from the repo (notice the user namespace) and one package from our local which is enabled via the 'local' flag.
As with other ops command by default the arch will default to whatever your underlying platform is but you can set it to what the package is by setting it different:
```
packages:
- pkg: graylog
name: eyberg/graylog:5.2.6
arch: amd64
- pkg: mongo
name: eyberg/mongodb:7.0.8
arch: amd64
- pkg: myserver
name: myserver_0.0.1
arch: amd64
local: true
```
Finally you need to have the ops-dns package installed which is a small dns server that provides service discovery:
```
ops pkg get eyberg/ops-dns:0.0.1
```
Now you can boot your unikernels via:
```
ops compose up
```
You can turn them off via:
```
ops compose down
```
OPS has had normal bridge support for linux for a while now but allows the end-user to define that.
On MacOS vmnet-bridged is used. On Linux a new bridge is created automatically and all instances are given taps to listen on that bridge. Upon 'compose down' the bridge group is taken down.
## Examples
While you can certainly find compose examples on we have [compose examples](https://github.com/nanovms/compose-examples) here.
# Docker
It is important to note that docker containers and unikernels are fairly differernt and are not at all the same thing. Most notably a docker container typically has a large userland that can run many different programs whereas a unikernel can only run one. Not only that but a unikernel is single process (however has support for multi-threading).
In practice this means that sometimes there are tweaks that need to happen for you to run something as a unikernel vs a container, these tend to come with substantial security and performance benefits.
Many applications can be imported from docker containers into unikernels quite easily as shown in:
On can set the arguments for the package that is created from the docker export as they sometimes will differ from a Dockerfile.
For example to export a nextjs docker container to a unikernel package:
```
ops pkg from-docker njs2 --copy --file /usr/local/bin/node -a app/server.js
ops pkg load --local njs --nanos-version=3c1fb76
```
# Firecracker
OPS works well with Firecracker from AWS. OPS doesn't currently interact with the firecracker api but can produce disk images for you to use with firecracker.
To run:
```
./firecracker --api-sock /tmp/firecracker.socket --config-file vm_config.json
```
You should replace the kernel.img file with the location of your kernel and the my\_img.img with the unikernel disk image of choice:
The vm\_config.json:
```
{
"boot-source": {
"kernel_image_path": "/home/eyberg/.ops/nightly/kernel.img",
"boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": "/home/eyberg/.ops/images/g",
"is_root_device": true,
"is_read_only": false
}
],
"network-interfaces": [
{
"iface_id": "eth0",
"guest_mac": "AA:FC:00:00:00:01",
"host_dev_name": "tap0"
}
],
"machine-config": {
"vcpu_count": 1,
"mem_size_mib": 1024
},
"logger": {
"log_path": "log.fifo",
"level": "Info",
"show_level": true,
"show_log_origin": true
}
}
```
You should have a dhcp listener. You can create a bridge and a dhcp server by running:
```
ops network create
```
If you are running via ops we'll create and attach the tap device for you but or firecracker you'll need to Create a tap device:
```
sudo ip tuntap add dev tap0 mode tap
sudo ip addr add 10.0.2.1/24 dev tap0
sudo ip link set tap0 up
```
and you should see the unikernel snag an ip:
```
Server started on port 8080
assigned: 10.0.2.10
assigned: 0.0.0.0
```
If you would like diagnostic logs you can try this before turning on the vm:
boot.sh:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X PUT 'http://localhost/boot-source' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"kernel_image_path": "/home/bob/.ops/0.1.26/kernel.img",
"boot_args": "console=ttyS0 reboot=k panic=1 pci=off" }'
```
drives.sh:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X PUT 'http://localhost/drives/rootfs' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"drive_id": "rootfs",
"path_on_host": "/home/bob/.ops/0.1.26/images/my_img.img",
"is_root_device": true,
"is_read_only": false
}'
```
machine.sh:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X PUT 'http://localhost/machine-config' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"vcpu_count": 1,
"mem_size_mib": 1024,
"ht_enabled": false
}'
```
start.sh:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X PUT 'http://localhost/actions' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"action_type": "InstanceStart"
}'
```
### Logs
logs.sh:
```
#!/bin/sh
mkfifo log.fifo
curl --unix-socket /tmp/firecracker.socket -i \
-X PUT 'http://localhost/logger' \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d '{ "log_path": "log.fifo", "level": "Info", "show_level": true, "show_log_origin": true }'
```
Finally read your logs:
read\_fifo.sh:
```
#!/bin/bash
while true
do
if read line <$1; then
if [[ "$line" == 'quit' ]]; then
break
fi
echo $line
fi
done
echo "Reader exiting"
```
```
./read_fifo.sh log.fifo
```
### Ballooning
For the most current documentation please see .
Nanos now has ballooning support for firecracker which utilizes mmio transport vs pci with qemu.
To use it with firecracker you'll need to add an appropriate section in your config:
```
"balloon": {
"amount_mib": 0,
"deflate_on_oom": false,
"stats_polling_interval_s": 1
},
```
Then you may poll using something like this:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X GET 'http://localhost/balloon' \
-H 'Accept: application/json'
```
You may inflate or deflate the balloon by sending a PATCH request:
```
#!/bin/sh
curl --unix-socket /tmp/firecracker.socket -i \
-X PATCH 'http://localhost/balloon' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d "{
\"amount_mib\": "512"
}"
```
### Configuration
You can now dynamically re-configure your unikernel via the firecracker kernel boot\_args line such as adding tracing support:
```
"boot-source": {
"kernel_image_path": "/Users/bob/.ops/nightly/kernel.img",
"boot_args": "trace=t debugsyscalls=t"
}
```
or you could set a static ipv4:
```
"boot-source": {
"kernel_image_path": "/Users/bob/.ops/nightly/kernel.img",
"boot_args": "en1.ipaddr=10.3.3.6 en1.netmask=255.255.0.0 en1.gateway=10.3.0.1"
}
```
or set an env var:
```
"boot-source": {
"kernel_image_path": "/Users/bob/.ops/nightly/kernel.img",
"boot_args": "environment.PROD=1"
}
```
or set arbitrary arguments:
```
"boot-source": {
"kernel_image_path": "/Users/bob/.ops/nightly/kernel.img",
"boot_args": "arguments.0=/python3 arguments.1=my_new_entrypoint.py"
}
```
### Known Limitations
Note the following limitations when running workloads under Firecracker. These are *not* Nanos limitations but limitations imposed via Firecracker because of its unique setup:
(A lot of this has to do with the lack of PCI but that is one of the reasons why it boots much faster.)
* No GPU support.
* No MQ (multi-queue) taps.
* No Volume hot-plugging.
Note: If you're interested in using something like firecracker but need GPU support check out
# Inception
Inception is a fully supported type 2 hypervisor for running virtualized workloads on normal AWS instances without the need for expensive metal isntances.
Inception is a great tool if you need full control over orchestration such as the ability to do instance pause/resume of vms or you wish to run firecracker or if you want to run kubevirt. (For instance you might need way to run unikernels inside your existing kubernetes installation).
You may also run normal linux vms inside Inception as well. Inception is available as the latest ubuntu release.
[NanoVMs Inception](https://nanovms.com/inception) is available on the AWS market. .
If you need more information or you've already started an Inception instance and need further guidance please get in touch with the NanoVMs team.
# OpenVMM
OPS works well with the OpenVMM VMM from Microsoft.
You'll need to enable uefi for your image:
config.json:
```
{
"Uefi": true
}
```
You'll need to download the firmware from - it's in the 'FV' folder.
If you don't want to build openvmm from source you can grab a binary from the ci system at .
To run:
```
./openvmm --uefi --uefi-firmware MSVM.fd --disk file:/home/myuser/.ops/images/myimage --hv
```
# Hyper-V
Nanos can run on hyper-v.
## Pre-requisites
Hyper-v is only supported on Windows 10/11 Enterprise, Pro and Education.
You need to enable the next Windows features:
* [WSL 2](https://docs.microsoft.com/en-us/windows/wsl/install-win10)
* [Hyper-v](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v).
After enabling the required windows features you may be able to manage virtual machines with ops commands from a wsl shell with administrator permissions.
If you see the error message
```
exec: "qemu-img": exeuctable file not found in $PATH
```
install qemu-utils in WSL2.
As of right now OPS on Windows requires to be ran in WSL2 with admin privileges. You can start WSL2 by right-clicking on it.
## Image Operations
### Create Image
```sh
$ ops image create -i -t hyper-v
```
### List Images
```sh
$ ops image list -t hyper-v
+------+--------------------------------------+---------+----------------+
| NAME | PATH | SIZE | CREATEDAT |
+------+--------------------------------------+---------+----------------+
| main | /home/xyz/.ops/vhdx-images/main.vhdx | 33.6 MB | 21 minutes ago |
+------+--------------------------------------+---------+----------------+
```
### Delete Image
```
$ ops delete image -t hyper-v
```
## Instance Operations
### Create Instance
```sh
$ ops instance create -t hyper-v
```
### List Instances
```sh
$ ops instance list -t hyper-v
+------+---------+----------------+--------------+------+
| NAME | STATUS | CREATED | PRIVATE IPS | PORT |
+------+---------+----------------+--------------+------+
| main | running | 21 minutes ago | 192.168.1.75 | 8080 |
+------+--------------------------------------+---------+
```
### Delete Instance
```sh
$ ops instance delete main -t hyper-v
```
# Onprem
The Onprem target is for users that wish to handle their own infrastructure. It is also the default target. While 'ops run' and 'ops pkg load' target the local machine as well the onprem target runs workloads in the background.
## Pre-requisites
## Image Operations
### Create Image
### List Images
### Delete Image
`ops image delete ` can be used to delete an image from IBM Cloud.
```
$ ops delete image nanos-main-image -t onprem
```
## Instance Operations
### Create Instance
### List Instances
### Get Logs for Instance
The onprem target is currentlty the only target that allows you to 'tail' serial logs:
```sh
$ ops instance logs --watch my-running-instance
```
### Delete Instance
### Stats
The onprem target is currently the only target that allows you to get 'stats' (for now in the form of guest memory utilization).
To get all the stats for all instances:
```sh
$ ops instance stats
```
To get the stats for just one instance:
```sh
$ ops instance stats -
```
## Volume Operations
### Create Volume
### List Volumes
### Delete Volume
### Attach Volume
To dynamically attach/detach volumes at run-time for the onprem target you must be using QMP.
```
{
"RunConfig": {
"QMP": true
}
}
```
### Detach Volume
To dynamically attach/detach volumes at run-time for the onprem target you must be using QMP.
```
{
"RunConfig": {
"QMP": true
}
}
```
Typically one invokes OPS for single commands and is scriptable and callable from other libraries/projects however a daemon does exist in daemon/ that can be built so that you can run OPS as a daemon that serves up JSON and GRPC.
To build the daemon you'll need to have a handful of protobufs dependencies installed:
```
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
go install github.com/bufbuild/buf/cmd/buf@latest
wget https://github.com/bufbuild/buf/releases/download/v1.13.1/buf-Linux-x86_64 && sudo mv buf-Linux-x86_64 /usr/bin/buf && sudo chmod +x /usr/bin/buf
wget https://github.com/grpc-ecosystem/grpc-gateway/releases/download/v2.15.0/protoc-gen-openapiv2-v2.15.0-linux-x86_64 -o ~/go/bin/protoc-gen-openapiv2 && chmod +x ~/go/bin/protoc-gen-openapiv2
wget https://github.com/grpc-ecosystem/grpc-gateway/releases/download/v2.15.0/protoc-gen-grpc-gateway-v2.15.0-linux-x86_64 -o ~/go/bin/protoc-gen-grpc-gateway && chmod +x ~/go/bin/protoc-gen-grpc-gateway
```
Then you can run 'make generate' which generates the protobufs.
# OpenStack
OPS works perfectly fine with OpenStack.
Some cloud providers using OpenStack that have been tested are [vexxhost](https://vexxhost.com/) and [ovh](https://www.ovh.com/).
In the cloud provider, download a script with the env variables required.
1. Create an account;
2. Login;
3. Click `API Access` in left menu;
4. Click on button with the description `Download OpenStack RC File`;
5. Select an option which contains the description `RC File`. 
6. Run `RC file` script in the terminal where you will use ops.
```
$ ./6050015361872676-openrc.sh`
```
Alternatively, you can set the env variables manually.
```
export OS_USERNAME=""
export OS_PASSWORD=""
export OS_DOMAIN_NAME="Default"
export OS_AUTH_URL="https://auth.vexxhost.net/"
export OS_REGION_NAME="sjc1"
export OS_PROJECT_NAME=""
export OS_PROJECT_ID=""
```
## Image Operations
### Create Image
```sh
GOOS=linux go build -o
ops image create -c config.json -t openstack -i
```
### List Images
```sh
ops image list -t openstack
```
### Delete Image
```sh
ops image delete -t openstack
```
## Instance Operations
### Create Instance
```sh
ops instance create -t openstack
```
While creating instance on openstack OPS needs flavour name. If you don't provide, OPS selects one for you. You can provide flavor name via CLI and config file.
CLI example
```sh
ops instance create -t openstack -f
```
Sample config file
```json
{
"CloudConfig" :{
"Platform" : "openstack",
"flavor" : ""
}
}
```
OPS provides configurable instance volume size. Add volume size in config file. Default size is 1 GB.
```json
{
"RunConfig": {
"VolumeSizeInGb" : 2
}
}
```
### List Instances
```sh
ops instance list -t openstack
```
### Get Logs for Instance
```sh
ops instance logs -t openstack
```
### Delete Instance
```sh
ops instance delete -t openstack
```
You can fetch instance name from **ops instance list** command.
## Volume Operations
### Create Volume
```sh
ops volume create -t openstack
```
### List Volumes
```sh
$ ops volume list -t openstack
+--------------------------------------+------+-----------+-----------+----------+-------------------------------+----------+
| UUID | NAME | STATUS | SIZE (GB) | LOCATION | CREATED | ATTACHED |
+--------------------------------------+------+-----------+-----------+----------+-------------------------------+----------+
| 2977268e-7c83-4191-a30d-f6fa25c3ddab | | available | 1 | | 2021-02-24 13:24:54 +0000 UTC | |
+--------------------------------------+------+-----------+-----------+----------+-------------------------------+----------+
```
### Delete Volume
```sh
ops volume delete -t openstack
```
### Attach Volume
```sh
ops volume attach -t openstack
```
### Detach Volume
```sh
ops volume detach -t openstack
```
# ProxMox
OPS works perfectly fine with Proxmox.
You'll want to create an API token on the datacenter in the proxmox UI.
You'll also want to ensure you have the correct permissions for that user's token on both the datacenter and storage for the operations you'd like to perform.
The default node name is set to 'pve' but you can reset this through the environment variable of 'NODE\_NAME':
```
export NODE_NAME='somethingelse'
```
```sh
package main
import (
"fmt"
"net/http"
)
func main() {
fmt.Println("hello world!")
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to my website!")
})
fs := http.FileServer(http.Dir("static/"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.ListenAndServe("0.0.0.0:8080", nil)
}
```
## Image Operations
### Create Image
```sh
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
export NODE_NAME="proxmox"
ops image create g -t proxmox -c config.json
```
### List Images
```sh
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
ops image list -t proxmox
```
### Delete Image
## Instance Operations
### Create Instance
First ensure that you have set local storage to allow disk images to be stored as by default it's not allowed.
When running proxmox 9.X and up you'll need to use ticket based auth for instance creation (at least for the time being).
```sh
export PROXMOX_USER="user@pam"
export PROXMOX_PASS="pass"
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
ops instance create g.img -t proxmox
```
### List Instances
```sh
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
ops instance list -t proxmox
```
### Start Instance
```sh
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
ops instance start 100 -t proxmox
```
### Stop Instance
```sh
export API_URL="https://172.16.41.133:8006"
export TOKEN_ID="myuser@pam!tokenid"
export SECRET="some-uuid-goes-here"
ops instance stop 100 -t proxmox
```
### Get Logs for Instance
### Delete Instance
## Networking
Proxmox has support for assigning multiple nics to the instance. By default DHCP is used but static ip addresses can be set as well. Proxmox is the only target that we support multiple nics for currently.
A config could look like so:
```
{
"RunConfig": {
"Nics": [
{
"BridgeName": "vmbr0",
"IPAddress": "1.2.3.4",
"IPv6Address": "",
"Netmask": "",
"Gateway": "",
},
{
"BridgeName": "vmbr0"
}
]
}
}
```
Notes:
If you are running through nested virtualization such as using VMWare Fusion there is a known bug where you need to disable KVM on your instances.
# Virtual Box
You can use Ops to run your local nanos in Virtual Box. You can use Ops CLI to create a `.vdi` image and launch a virtual machine with the image.
## Pre-requisites
1. Install [Virtual Box](https://www.virtualbox.org/wiki/Downloads) in your machine.
2. Ensure `VBoxManage` is in your shell path.
**Note:** If you are using WSL you may need to create a symlink in Windows filesystem with the name `VBoxManage` pointing to `VBoxManage.exe`.
## Image Operations
### Create Image
You can create an image with the following command.
```sh
$ ops image create -i -t vbox
```
For creating an image using a particular package, you need to provide the package name to `ops image create` command with `--package` option.
```sh
$ ops image create -c config.json --package eyberg/node:20.5.0 -a hi.js -i js -t vbox
```
The Ops CLI will store the image in the directory `vdi-images` inside Ops default directory.
### List Images
You can list existing images with `ops image list -t vbox`.
```sh
$ ops image list -t vbox
+--------------------------------------+------------------------+--------+---------+------------------+
| UUID | NAME | STATUS | SIZE | CREATEDAT |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01071a8d-eb59-49c1-aa69-028f84cc6d06 | nanos-main-image | online | 10.0 GB | 3 days ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01c5bd28-aa19-4883-a4c9-31f3ff9fd061 | nanos-node-image | online | 10.0 GB | 1 month ago |
+--------------------------------------+------------------------+--------+---------+------------------+
| 01cd3190-df52-47e8-b5c3-b05f7107819e | nanos-server-image | online | 10.0 GB | 1 year ago |
+--------------------------------------+------------------------+--------+---------+------------------+
```
### Delete Image
`ops image delete ` can be used to delete an image.
```
$ ops delete image nanos-main-image -t vbox
```
## Instance Operations
### Create Instance
After creating the image, we can launch a virtual machine from an existing image.
```sh
$ ops instance create -t vbox
```
### List Instances
You can list the virtual machines running on Virtual Box using `ops instance list` command.
```sh
$ ops instance list -t vbox
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| ID | NAME | STATUS | PRIVATE IPS | PUBLIC IPS |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
| 00d954cc-9603-43a0-915b-5c2ae75772b6 | nanos-main-image-15566 | running | 10.8.3.63 | 209.151.144.166 |
+--------------------------------------+------------------------+---------+-------------+-----------------------------------------+
```
### Get Logs for Instance
```sh
ops instance logs -t vbox js-1738285116
```
### Delete Instance
`ops instance delete` command can be used to delete a virtual machine.
```sh
$ ops instance delete my-instance-running -t vbox
```
# VSphere
OPS works perfectly fine with VSphere ESX 6.7 and 7.0.
It uses the pvsci driver for storage and vmnetx3 for networking.
For most operations you'll want to use the following config and you'll need access to your API key and the access key and secret access key found in your portal:
```json
{
"CloudConfig" :{
"BucketName":"nanos-test"
}
}
```
```sh
package main
import (
"fmt"
"net/http"
)
func main() {
fmt.Println("hello world!")
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to my website!")
})
fs := http.FileServer(http.Dir("static/"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
http.ListenAndServe("0.0.0.0:8080", nil)
}
```
## Image Operations
### Create Image
This creates a monolithicFlat image (2 files) and uploads it to vsphere's datastorage.
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
GOOS=linux go build -o gtest
ops image create gtest -c config.json -t vsphere
```
### List Images
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
ops image list -t vsphere
```
### Delete Image
## Instance Operations
### Create Instance
The domain part of the Resource Pool is in this example, 'localhost.localdomain'. You can find that in your portal.
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
export GOVC_RESOURCE_POOL="/ha-datacenter/host/localhost.localdomain/Resources"
ops instance create gtest -t vsphere
```
### Start an Instance
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
ops instance start mytest -t vsphere
```
### List Instances
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
ops instance list -t vsphere
```
Note: The first time you run this command you might need to have the Guest IP Hack set. OPS will try to set that automatically for you but all existing vms won't report their ips until it is set *and* will require a new ARP request to be sent. If you already know the ip in question you can simply connect to it or you can reboot the vm and that will generate traffic.
### Get Logs for Instance
You can obtain the logs via:
```
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
ops instance logs -t vsphere gtest
```
### Delete Instance
Right now the delete instance deletes both the instance and the image. We will probably change this in the future to do just the instance.
```sh
export GOVC_INSECURE=1
export GOVC_URL="login:pass@host:port"
export GOVC_RESOURCE_POOL="/ha-datacenter/host/localhost.localdomain/Resources"
ops instance delete -t vsphere named
```
# WorkStation
These are notes from a user about working with VMWare Workstation:
1. Create a raw disk image following the instructions in the Bare Metal guide for ops. I've tested a few options including using DHCP and a static IP and they seem to work fine. I also did my testing with VMWare Workstation 15, and ESXi 6.7
2. Convert the raw image created by ops using qemu-img convert. The command I used was qemu-img convert -f raw -O vmdk old.img new\.vmdk
3. In VMWare Workstation Create a new VM, by going to File-> New Virtual Machine
4. In the popup window Click "Custom (advanced)" before clicking "Next"
5. Select your hardware compatibility. If you plan on running it on Workstation, you can leave it set to the default. For me that was Workstation 15.X. If you plan on uploading it to ESXi select an ESXi version from the dropdown (this can be changed at a later point) and click "Next"
6. On the next page select "I will install the operating system later" and click "Next"
7. On the next screen it asks what the Guest operating system is. I selected Other, and Other. It's possible there are other combinations, like Linux that might work fine here.
8. Continue through the next several screens selecting the options that are most appropriate for your nanos image, name, cores, ram, etc.) until you get to the "Select a Disk" screen.
9. Select the option for "Use an existing disk", click next, and browse to find the vmdk you created using qemu-img convert. When you click "Next" you may get asked if you want "Convert existing virtual disk to newer format" In my testing so far, you have to select "Convert" in order for it to work properly, but I have not dug to deep to see if you could keep the format and still have it working.
10. After it converts click finish, before running the VM, open up the VMs ".vmx" file Look for a line starting with ethernet0.virtualDev and make sure it is set to = "vmxnet3". It's possible this line won't exist and needs to be added (as was the case in my testing)
11. Boot your nanos vm and it should work.
# Pulumi
Create a New Project:
```
pulumi new --dir bob
pulumi up
```
You might need to login:
```
az login --scope https://management.core.windows.net//.default
```
Need to Start From Scratch?
```
pulumi destroy
pulumi up
```
# Terraform
Terraform is a declarative way of setup your infrastructure, and may be used to deploy nanos instances with [terraform-ops-provider](https://registry.terraform.io/providers/nanovms/ops/latest) on supported cloud providers.
Download and install [terraform](https://www.terraform.io/downloads.html).
## GCP
See below an explanation of the resources needed to upload a nanos image and to launch an instance to google cloud provider. You can see the entire configuration file at the end of this section. All files required to run this example is in [ops terraform plugin repository](https://github.com/nanovms/terraform-provider-ops/tree/master/examples/gcp).
1. Start importing the ops terraform plugin and set your google cloud settings.
```
terraform {
required_providers {
ops = {
source = "nanovms/ops"
}
}
}
provider "google" {
project = "prod-1033"
region = "us-west2"
zone = "us-west2-a"
}
provider "ops" {
}
```
2. Create an image in your local machine with your application binary file. You have to specify your google cloud account settings in a configuration file and set the target cloud to `gcp`.
```
resource "ops_images" "walk_server_image" {
name = "walk-server"
elf = "./walk-server"
config = "./config.json"
targetcloud = "gcp"
}
```
3. Upload the nanos image to google storage. The resources below create a bucket in google storage and upload the nanos image.
```
resource "google_storage_bucket" "images_bucket" {
name = "terraform-images"
location = "us"
force_destroy = true
}
resource "google_storage_bucket_object" "walk_server_raw_disk" {
name = "walk-server.tar.gz"
source = ops_images.walk_server_image.path
bucket = google_storage_bucket.images_bucket.name
}
```
4. Use the path of the object uploaded to create the image.
```
resource "google_compute_image" "walk_server_image" {
name = "walk-server-img"
raw_disk {
source = google_storage_bucket_object.walk_server_raw_disk.self_link
}
labels = {
"createdby" = "ops"
}
}
```
5. Use the image as base to launch an instance. Make sure you add firewall rules to make your application publicly available.
```
resource "google_compute_instance" "walk_server_instance" {
name = "walk-server"
machine_type = "f1-micro"
boot_disk {
initialize_params {
image = google_compute_image.walk_server_image.self_link
}
}
labels = {
"createdby" = "ops"
}
tags = ["walk-server"]
network_interface {
# A default network is created for all GCP projects
network = "default"
access_config {
}
}
}
resource "google_compute_firewall" "walk_server_firewall" {
name = "walk-server-firewall"
network = "default"
allow {
protocol = "tcp"
ports = ["8080"]
}
target_tags = ["walk-server"]
}
```
After joining all pieces we get the next terraform configuration file.
```
terraform {
required_providers {
ops = {
source = "nanovms/ops"
}
}
}
provider "google" {
project = "prod-1033"
region = "us-west2"
zone = "us-west2-a"
}
provider "ops" {
}
resource "ops_images" "walk_server_image" {
name = "walk-server"
elf = "./walk-server"
config = "./config.json"
targetcloud = "gcp"
}
resource "google_storage_bucket" "images_bucket" {
name = "terraform-images"
location = "us"
force_destroy = true
}
resource "google_storage_bucket_object" "walk_server_raw_disk" {
name = "walk-server.tar.gz"
source = ops_images.walk_server_image.path
bucket = google_storage_bucket.images_bucket.name
}
resource "google_compute_image" "walk_server_image" {
name = "walk-server-img"
raw_disk {
source = google_storage_bucket_object.walk_server_raw_disk.self_link
}
labels = {
"createdby" = "ops"
}
}
resource "google_compute_instance" "walk_server_instance" {
name = "walk-server"
machine_type = "f1-micro"
boot_disk {
initialize_params {
image = google_compute_image.walk_server_image.self_link
}
}
labels = {
"createdby" = "ops"
}
tags = ["walk-server"]
network_interface {
# A default network is created for all GCP projects
network = "default"
access_config {
}
}
}
resource "google_compute_firewall" "walk_server_firewall" {
name = "walk-server-firewall"
network = "default"
allow {
protocol = "tcp"
ports = ["8080"]
}
target_tags = ["walk-server"]
}
output "image_path" {
value = ops_images.walk_server_image.path
}
output "instance_ip" {
value = google_compute_instance.walk_server_instance.network_interface[0].access_config[0].nat_ip
}
```
After running this configuration you will see the path where the image file was created locally and the ip address of the launched instance.
# Kubernetes
## Security Warning
Running unikernels under kubernetes diminishes some of their security benefits, however, a lot of organizations still have heavy kubernetes installations so these are some ways you can still run unikernels in k8s.
## Pre-requisites
You need access to hardware virtualization. If you are in the cloud the best way to get that is through [NanoVMs Inception](https://nanovms.com/inception) available on the AWS market. . This will allow you to run kube-virt on plain old normal ec2 instances like an ec2.small without having to resort to expensive metal instances.
If you don't want to do that you'll either need a real physical computer or use metal instances.
## Installing K8s / Initial Setup
Install KubeCtl:
```
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv kubectl /usr/local/bin/.
sudo mv kubectl /usr/local/bin/.
kubectl version --client
```
Install Minikube:
```
curl -Lo minikube
https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
&& chmod +x minikube
minikube start --vm-driver=kvm2
```
Install KVM tooling:
```
sudo apt-get install libvirt-daemon-system libvirt-clients bridge-utils
```
Ensure you are setup for KVM via libvirt and have associated permissions:
```
virt-host-validate
```
```
groups
```
Install KubeVirt:
```
export KUBEVIRT_VERSION=$(curl -s
https://api.github.com/repos/kubevirt/kubevirt/releases | grep tag_name
| grep -v -- - | sort -V | tail -1 | awk -F':' '{print $2}' | sed
's/,//' | xargs)
echo $KUBEVIRT_VERSION
kubectl create -f
https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-operator.yaml
```
Create a Resource:
```
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-cr.yaml
```
Install Virtctl:
```
curl -L -o virtctl \
https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/virtctl-${KUBEVIRT_VERSION}-linux-amd64
chmod +x virtctl
```
Import CDI:
```
wget https://raw.githubusercontent.com/kubevirt/kubevirt.github.io/master/labs/manifests/storage-setup.yml
kubectl create -f storage-setup.yml
export VERSION=$(curl -s https://github.com/kubevirt/containerized-data-importer/releases/latest | grep -o "v[0-9]\.[0-9]*\.[0-9]*")
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
kubectl get pods -n cdi
```
## Building / Deploying
If you have the base kubernetes installation up and running you can move on to the final part.
You need to compress the disk image in question to xz format.
```bash
cp .ops/images/goweb.img .
xz goweb.img
```
Now you need to upload that to a url for k8s to import.
Download a sample PVC template:
```
wget https://raw.githubusercontent.com/kubevirt/kubevirt.github.io/master/labs/manifests/pvc_fedora.yml
```
Edit the line to point to your xz'd image:
```
cdi.kubevirt.io/storage.import.endpoint: "https://storage.googleapis.com/totally-insecure/goweb.img.xz"
```
Import:
```
kubectl create -f pvc_fedora.yml
kubectl get pvc fedora -o yaml
```
Create the Actual VM:
```
wget https://raw.githubusercontent.com/kubevirt/kubevirt.github.io/master/labs/manifests/vm1_pvc.yml
kubectl create -f vm1_pvc.yml
```
If you `minikube ssh` you should now be able to hit up your instance.
# Bare Metal
Ops can run on bare metal (eg: linux on real hardware) just fine.
## Pre-requisites
## Image Operations
### Build Image
```sh
$ ops build -e PORT=8080 -c config.json mylilwebserver -i mynewimg
```
### List Images
You can list existing images on bare metal with `ops image list`.
```sh
$ ops image list -t onprem
+--------------------+--------+-------------------------------+
| NAME | STATUS | CREATED |
+--------------------+--------+-------------------------------+
| nanos-main-image | READY | 2019-03-21T15:06:17.567-07:00 |
+--------------------+--------+-------------------------------+
| nanos-node-image | READY | 2019-04-16T23:16:03.145-07:00 |
+--------------------+--------+-------------------------------+
| nanos-server-image | READY | 2019-03-21T15:50:04.456-07:00 |
+--------------------+--------+-------------------------------+
```
### Delete Image
## Instance Operations
### Create Instance
```sh
ops instance create -t onprem -z onprem -p 8080
```
### List Instances
```sh
ops instance list -t onprem -z onprem
```
## Get Logs for Instance
### Delete Instance
For the time being pids are used as instance ids. Perhaps this will change in the future if more people opt to use this and they need stronger prevention of data leakage.
```
ops instance delete 53502 -t onprem -z onprem
```
# Integrations
Ops can integrate with several editors and configuration management systems.
## Editors
[VSCode](https://marketplace.visualstudio.com/items?itemName=nanovms.ops) [IntelliJ](https://plugins.jetbrains.com/plugin/16899-nanovms-ops)
# Debugging
## Using the GNU Debugger
If you think you found a problem in OPS or Nanos first thing is to verify where the bug lies. Generally, if you turn on the `-d` flag and study the bottom you can tell if it is in user or kernel:
```
1 general protection fault in user mode, rip 0x13783afe7
```
Clearly in this example we see a GPF in user.
### Debugging the user program
If you wish to have symbol access within your user program the following conditions must be met:
* Ensure your program is statically linked.
* Ensure you have debugging symbols to begin with:
You can do this with c by doing the following:
**Example**
For this example will examine a segfault (that we purposely injected):
```
#include
#include
void mybad() {
int x = 1;
char *stuff = "asdf";
printf("about to die\n");
*(int*)0 = 0;
}
int main(void) {
mybad();
printf("should not get here\n");
return 0;
}
```
We compile with debugging symbols and link statically:
```
cc main.c -static -g -o main
```
If we run this example program with OPS (`ops run main`), we can see that it crashes soon after starting:
```
running local instance
booting /home/user/.ops/images/main ...
[0.087239] en1: assigned 10.0.2.15
about to die
*** signal 11 received by tid 2, errno 0, code 1
fault address 0x0
```
Next, we'll run with the debug flag turned on:
```
ops run -d main
```
This will pause waiting on a gdb to attach to it.
In another window, we'll launch gdb pointing it at the executable file:
```
gdb main
```
You can see the source via the `list` gdb command:
```
(gdb) list
1 #include
2 #include
3
4 void mybad() {
5 int x = 1;
6 char *stuff = "asdf";
7
8 printf("about to die\n");
9 *(int*)0 = 0;
```
We'll connect to the target VM by specifying the remote:
```
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x000000000000fff0 in ?? ()
```
Great - now we are connected.
Let's set a breakpoint for 'mybad':
```
(gdb) b mybad
Breakpoint 1 at 0x401d35: file main.c, line 4.
```
Now let's continue:
```
(gdb) c
Continuing.
Breakpoint 1, mybad () at main.c:4
4 void mybad() {
```
You should see in your other window (where you started OPS) that the program starts:
```
[0.225679] assigned: 10.0.2.15
```
Now if we single step through the program we can print out various locals:
```
4 void mybad() {
(gdb) step
5 int x = 1;
(gdb) step
6 char *stuff = "asdf";
(gdb) step
8 printf("about to die\n");
(gdb) p x
$1 = 1
(gdb) p stuff
$2 = 0x495004 "asdf"
```
This should get you further down the path when you find various bugs in your program. Hope this helps.
If your program isn't even starting there might be an issue with OPS.
However, if you open an issue in Nanos please provide the following:
Your OPS profile:
```
➜ ~ ops profile
Ops version: 0.1.9
Nanos version: 0.1.25
Qemu version: 4.1.90
Arch: darwin
```
Nightly vs Master?
Does this work on the nightly build? Running '-n' will run ops with whatever was in the master branch last night.
Reproducible steps
### Debugging the Nanos kernel
The Nanos kernel implements address space layout randomization not only for the user program, but also for the kernel itself (KASLR). To easily debug a running kernel, we need to disable KASLR, and to do so we have to re-compile the kernel from source: download the Nanos source code from , then in a terminal window go to the source code folder and build the kernel with the 'no-kaslr' configuration option:
```
make KCONFIG=no-kaslr
```
The newly built kernel file will be located (assuming we are using an Intel or AMD machine) in the output/platform/pc/bin directory. To run an image with the re-compiled kernel, we first move to the folder where the user program is located, then create a JSON configuration file specifying the path of the kernel file (adjust it to match the actual path on your machine):
```
{
"Kernel": "/usr/src/nanos/output/platform/pc/bin/kernel.img"
}
```
Then, we can build and run an image containing our debug-friendly kernel and the user program with the following OPS command:
```
ops run -c config.json -d main
```
In the above command line, `config.json` is the name of the JSON configuration file, `-d` is the debug flag, and `main` is the executable file of the user program. The VM is now paused waiting for gdb to attach to it. In another window, move to the folder containing the Nanos kernel source, then start gdb:
```
gdb -ix .gdbinit-x86-64
```
The gdb initialization file in the above command line automates the process of loading the kernel file symbols (with the appropriate offset) and connecting gdb to the VM. We can now debug the kernel (e.g. set breakpoints, single-step, display variables, etc.) using standard gdb commands.
Note: in the early boot phase, the kernel image is located in the guest VM memory at an address that does not correspond to the final runtime address; only after the boot code executes the `kaslr()` kernel function, the kernel is located at its runtime address; therefore, debugging with the default settings in the above gdb initialization file works only for code executed after the `kaslr()` function. If we want to debug early boot code, we must first reset the symbol offset with the following gdb command:
```
symbol-file output/platform/pc/bin/kernel.elf
```
## GUI Debugger in VSCode
This part of the debugging guide shows how to use the GNU Debugger (GDB) in combination with VSCode to get better visualization of the debugging process. It requires that the [Native Debug](https://marketplace.visualstudio.com/items?itemName=webfreak.debug) extension is installed.
### Prerequisite
Launch the application in debug mode with `ops`:
```
$ ops run -d main
booting ~/.ops/images/main.img ...
You have disabled hardware acceleration
Waiting for gdb connection. Connect to qemu through "(gdb) target remote localhost:1234"
See further instructions in https://nanovms.gitbook.io/ops/debugging
```
### Attach the Debugger
1. Click the `Run` icon on the left sidebar (alternatively use `Ctrl+Shift+D`) and then `create a launch.json file`.
2. Select `GDB` as the environment. This will create an autogenerated `launch.json` file.
3. Replace the contents of the `launch.json` file with following:
```
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Debug",
"type": "gdb",
"request": "attach",
"executable": "${workspaceFolder}/main",
"target": "localhost:1234",
"remote": true,
"cwd": "${workspaceRoot}",
"valuesFormatting": "parseText"
}
]
}
```
4. Set a `Breakpoint` in the source file (`main.c`) and start the debugging session from the `Run` on the left sidebar (alternatively use `Ctrl+Shift+D`) and click on the `> Debug` icon.
It is now possible to use the debugging palette to debug the application code.
## Dump
Ops provides a tool that allow you to inspect image crash logs and image manifests. The dump tool binary is inside the ops version directory (`~/.ops//dump`). Make sure you use the dump tool of the same ops version you used to build the image you want to analyze.
If the application crashes the unikernel writes the error stack to a log file before exiting. You are able to see the log content if you run the command `dump -l `.
```
$ dump -l ~/.ops/main.img
detected filesystem at 0x11600
klog offset: 0x10600
klog size: 4096 bytes
boot id: 0
exit code: 1
en1: assigned 10.0.2.15
2021/03/22 11:35:35 Failed
```
The image manifest has details about the image like:
* files and their paths in the filesystem;
* environment variable values, including nanos base image version used;
* program and arguments to run on initialization;
* static IP, gateway and netmask to configure the network interface;
* etc. You can look into your image manifest by using the command `dump `.
```
$ dump ~/.ops/main.img
detected filesystem at 0xc11600
Label:
UUID: ad63ee2d-58d1-336f-7484-9fc81f3bc835
metadata (
program:/main
arguments:(0:main)
environment:(PWD:/ NANOS_VERSION:0.1.32 USER:root)
children:(
.:
proc:(children:(
.:
..:
self:(mtime:6942441066663300181 atime:6942441066663300181 children:(
.:
..:
exe:(mtime:6942441066669261459 atime:6942441066669261459 linktarget:/main)
))
sys:(children:(
.:
..:
kernel:(children:(
.:
..:
hostname:(extents:(0:(allocated:1 length:1 offset:5668)) filelength:7)
))
))
))
etc:(children:(
ssl:(children:(
.:
..:
certs:(children:(
ca-certificates.crt:(extents:(0:(allocated:406 length:406 offset:1078)) filelength:207436)
.:
..:
))
))
passwd:(extents:(0:(allocated:1 length:1 offset:1485)) filelength:33)
.:
..:
resolv.conf:(extents:(0:(allocated:1 length:1 offset:1484)) filelength:18)
))
..:
lib:(children:(
.:
..:
x86_64-linux-gnu:(children:(
libnss_dns.so.2:(extents:(0:(allocated:53 length:53 offset:1025)) filelength:26936)
.:
..:
))
))
main:(extents:(0:(allocated:4182 length:4182 offset:1486)) filelength:2140732)
)
booted:)
```
## FUSE
Nanos has a FUSE driver which allows us to mount the TFS image used on the host filesystem. This makes it easy for rapid development, ease of debugging and other interesting tools.
If you'd like to read more check out .
To try it out on Linux:
```
sudo apt-get install libfuse-dev
```
To try it out on Mac:
```
brew install macfuse
```
Then you simply create a mount point and mount your desired image:
```
mkdir -p /tmp/mnt
~/.ops/0.1.34/tfs-fuse /tmp/mnt ~/.ops/images/myimg.img
```
## Debug Logging
You can use the 'net console' logging feature if you'd like to log without printing via serial/vga. Simply specify it via:
```
{
"ManifestPassthrough": {
"consoles": ["+net"]
}
}
```
Then you can capture it via netcat:
```
nc -l -u 4444
```
The default ip/port that Nanos will ship the logging to is 10.0.2.2 (assuming user-mode) and port of 4444.
You can adjust these via the following config though:
`netconsole_ip`
`netconsole_port`
## Syscall Execution Time
OPS/Nanos has the ability to trace the number of syscall executions and timing information much like `strace -c`.
Here is a short c example:
```
#include
int main() {
printf("hello!\n");
}
```
The output:
```
eyberg@box:~/z$ ops run --syscall-summary main
booting /home/eyberg/.ops/images/main.img ...
en1: assigned 10.0.2.15
hello!
% time seconds usecs/call calls errors syscall
------ ------------ ------------ ------------------------------------------
48.53 0.000644 215 3 brk
25.16 0.000334 334 1 read
16.05 0.000213 213 1 write
3.31 0.000044 6 7 mmap
1.73 0.000023 6 4 pread64
1.58 0.000021 4 5 4 openat
0.97 0.000013 13 1 close
0.90 0.000012 3 4 mprotect
0.67 0.000009 9 1 uname
0.37 0.000005 5 1 1 access
0.22 0.000003 2 2 fstat
0.22 0.000003 1 3 3 stat
0.22 0.000003 2 2 1 arch_prctl
------ ------------ ------------ ------------------------------------------
100.00 0.001327 35 9 total
exit status 1
```
## Core Dumps
Nanos supports core dumps. By default they are turned off and enabled if specifying a > 0 'coredumplimit' config variable. Ensure that the volume size is large enough to contain the core dump as well however.
Locally you can do something like this:
```
#include
#include
int main() {
printf("yo\n");
abort();
return 0;
}
```
with a config like:
```
{
"BaseVolumeSz": "200m",
"ManifestPassthrough": {
"coredumplimit": "150m"
}
}
```
Your size will vary with the size and scope of your application.
```
ops run -c config.json main
```
You can now see a core has been generated:
```
➜ ops image tree main
/
| proc
| | sys
| | | kernel
| | | | hostname
| | self
| | | exe -> /main
| lib
| | x86_64-linux-gnu
| | | libnss_dns.so.2
| etc
| | passwd
| | ssl
| | | certs
| | | | ca-certificates.crt
| | resolv.conf
| main
| coredumps
| | core
```
Now you can copy it out:
```
ops image cp main coredumps/core .
```
And quickly view a backtrace:
```
➜ gdb -ex bt -ex quit main core
GNU gdb (Ubuntu 11.1-0ubuntu2) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from main...
[New LWP 2]
Core was generated by `main'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=761004893632) at pthread_kill.c:44
44 pthread_kill.c: No such file or directory.
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=761004893632) at pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=761004893632) at pthread_kill.c:80
#2 __GI___pthread_kill (threadid=761004893632, signo=signo@entry=6) at pthread_kill.c:91
#3 0x0000007a1af59476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x0000007a1af3f7b7 in __GI_abort () at abort.c:79
#5 0x000000000056d185 in main () at main.c:6
```
## Tracing
If you are using '--trace' you can set an optional 'notrace' variable to exclude certain output such as:
```
"notrace": ["open", "close"]
```
You can also do the opposite such as:
```
"tracelist": ["open", "close"]
```
You can also trace groups of related syscalls:
```
"tracelist": ["%file"]
```
The supported groups are:
* %file
* %desc
* %network
* %process
* %signal
* %memory
# Monitoring
Nanos works well with many monitoring, observability and telemetry systems.
For kernel crashes on cross-cloud monitoring you can use Radar:
**AWS**
You can ship metrics on [AWS](https://docs.ops.city/ops/klibs#cloudwatch-metrics).
**GCP**
You can send metrics on [GCP](https://docs.ops.city/ops/clouds/google_cloud#gcp-metrics-memory).
**Azure**
You can send metrics to [Azure](https://docs.ops.city/ops/klibs#azure).
**Digital Ocean**
You can send metrics on [Digital Ocean](https://docs.ops.city/ops/klibs#digitalocean).
## Other
Lots of other APM tooling works well with Nanos. If you require a more native integration please get in touch with your NanoVMs support engineer.
# CrossBuilding
OPS has initial support for cross-building Nanos unikernels on a Mac. Nanos uses the ELF binary format just like Linux so when building with OPS on Linux using 'run' or 'instance create' locally libraries are resolved appropriately and loaded onto the disk image.
For many use-cases this works the same on Mac. For statically linked builds produced by Go or Rust you won't have any trouble and for simple projects using official Node, Ruby, Python packages you shouldn't have any trouble.
However, there are times when you might need a native Linux environment to build a c library and then load it in a package.
For instance if you wanted to use the redis-fast-driver in NPM, in the past you might'e spun up a docker container and built it there or spun up a Linux vm and built it there. This is where the OPS cross-build environment capability comes into play. It spins up a small linux vm, executes your build instructions and then allows you to spit out the artifacts to your host system (MacOS in this case). This allows you to build native add-ons without resorting to a full-blown Linux vm or docker (which runs inside of a linux vm regardless).
#### Example
To get started first you need to create an env. This downloads and instatitates a cross-build environment:
```
$ ops env install
```
Now we'll specify our build steps like so in a build.txt:
```
apt-get install -y nodejs make gcc g++
npm install redis-fast-driver --save
```
We'll also specify a config detailing what we'd like to extract from the build-environment. In this case the node\_modules directory and the node binary with it's linked libraries:
```
{
"Dirs": ["node_modules"],
"Program": "/usr/bin/node"
}
```
Now we build. This runs the commands found in build.txt in the vm and then copies the artifacts back out to the host:
```
$ ops env build build.txt -c config.json -r .
```
You'll notice you now have all the appropriate files:
```
$ redis-test ls
build.txt config.json config_1.json example.js lib
lib64 node node_modules usr
```
```
$ redis-test tree lib*
lib
└── x86_64-linux-gnu
├── libc.so.6
├── libdl.so.2
├── libgcc_s.so.1
├── libm.so.6
├── libpthread.so.0
└── libstdc++.so.6
lib64
└── ld-linux-x86-64.so.2
1 directory, 7 files
```
Now we can test with another json for running our custom node build on Mac:
```
{
"Dirs": ["node_modules"],
"Files": ["/lib/x86_64-linux-gnu/libstdc++.so.6", "example.js"],
"Program": "/usr/bin/node",
"Args": ["example.js"]
}
```
Now your node example is running as a Nanos unikernel on Mac without a linux vm involved.
```sh
$ redis-test ops run usr/bin/node -c config_1.json -r .
booting /Users/eyberg/.ops/images/node.img ...
en1: assigned 10.0.2.15
en1: assigned FE80::FC65:67FF:FEF2:F10D
====================================================
Start test: PING command 2500 times
```
# MacOS
OPS and Nanos work great on both x86 and arm based macs. This includes the older x86 models from circa 2014, 2015 up to and including the arm64 m1 and m2s.
We've also added support for [ops desktop](https://ops.city/downloads) for the m1 and m2 which is a graphical desktop interface.
We have set some (what we think) to be sane defaults.
If on x86 all operations such as 'image create', 'pkg load', 'ops run', etc default to using x86 Nanos and expect a x86 binary/package.
If on arm64 (m1 and m2) all operations default to using arm64 Nanos and expect an arm64 binary/package.
You may wish to create and/or run Nanos unikernels on a different architecture and that is 100% supported. For instance it is common now to do development on a Mac m2 which is arm64 based but deploy to an x86-64 server on AWS or GCP.
If we want to build and run an x86 instance on a m2 we would do the following (for example with go):
```sh
GOARCH=amd64 GOOS=linux go build myapp
ops run --arch=amd64 myapp
```
Or if we wish to crate an image:
```
ops image crate --arch=amd64
```
If you are using an x86 machine and you'd like to run an arm payload you'd do the following:
```
ops run --arch=arm64 myapp
```
If you wish to use the native architecture you don't need to specify the arch flag and ops will default to whatever the native arch is.
When enabling a different architecture than the one that is natively present the underlying qemu binary will also be switched along with many various flags.
It is important to note that you will get the best performance when using the native arch. So if you are on arm64 for performance reasons you'll want to use arm64 images/packages or vice-versa for x86.
We'd like to make this experience super seamless so if you have any issues don't be shy on opening an issue on the ops repo.
# Windows
OPS and Nanos work great on Windows. You'll just need to install it under WSL2.
# Deploy
`Ops` provides a command that allows to skip the normal deployment workflow that would involve multiple commands to create and destroy images and instances. It allows to quickly update an image and an instance in a cloud provider from an elf file or a package.
The command format is `ops deploy -t `.
Internally, the command executes the next actions.
1. First, deletes an image with the same name of the elf file or the image name specified in configuration;
2. Next, creates an image with the instructions to execute the elf file program or the package program in the cloud provider.
3. Stops and deletes the instances that were launched from an image with the same name.
4. Finally, launches an instance with the image created.
Likewise on instance creation and on image creation, all the properties set in a configuration file are applied in this command.
# FAQ
### Q: What hypervisors does `ops` currently support?
A: Currently, nanos can run on both qemu and xen [QEMU](https://www.qemu.org/). We are obviously open to adding more support for other various hypervisors but there are considerations to be had.
Also - Nanos runs on Google Cloud && AWS.
### Q: What version of qemu is best?
A: We track latest qemu in homebrew as we have found issues before. The latest qemu release is currently 6.0.X. If you don't have that release try to install from brew:
```
brew tap nanovms/homebrew-qemu
brew install nanovms/homebrew-qemu/qemu
```
### Q: Can I contribute?
A: Sure! We accept pull requests and if you have prior kernel experience please get in touch with NanoVMs - they might even want to pay you.
### Q: What artifact does `ops` produce?
A: `ops` builds a disk image artifact that you can find in \~/.ops/images. . You can get a list of all your images via:
```
ops image list -t onprem
```
This disk image is ran by `qemu` to execute your application.
### Q: Can I run `ops` inside a container?
A: Yes, it is possible to run `ops` within a container such as docker. Although, it is NOT recommended to do so, especially for production environments. You will likely run into performance related issues.
### Q: What if I need to 'shell out'?
A: We consider this an anti-pattern of software development and a scourge on the software ecosystem. It is advised to utilize your native language API and libraries to achieve the task instead.
### Q: What about the filesystem?
A: Nanos currently makes use of the TFS filesystem and can run databases and other things through the POSIX api you all know.
### Q: How do I run `ops` on a cloud provider with bridged networking?
A: We do not suggest running ops on a cloud provider on top of linux. We suggest using the native way of provisioning your unikernels so they run by themselves standalone and you won't have do any network configuration.
Please see the following cloud documentation for whichever cloud provider you wish to run on:
* [Google Cloud Integration](/ops/clouds/google_cloud)
* [AWS Integration](/ops/clouds/aws)
* [Digital Ocean](/ops/clouds/digital_ocean)
* [Vultr](/ops/clouds/vultr)
### Q: My ssh connection gets terminated while running `ops net setup`
A: This is only necessary if you wish to create your own bridges and attach tap interfaces to them. It's not a task we suggest most people do though.
### Q: Does this work with kubernetes?
A: This can work with kubernetes but kubernetes is a container orchestrator and is typically deployed on top of an existing virtual machine whereas the intention for these are to run *as* virtual machines. Further, kubernetes has very serious security issues that should be considered such as sharing a kernel amongst multiple tenants.
Please see [the documentation](/ops/hypervisors/k8s) for running Nanos under kubernetes.
### Q: Is there an API?
Yes, if you are coding in Go you can use the API found at .
If you are looking for a web-accessible API [Nanos C2](https://nanovms.com/nanos-c2) has you covered.
### Q: How can I tell if I'm running inside Nanos at runtime?
You can use the Uname syscall and extract the sysname from it:
```go
package main
import (
"fmt"
"syscall"
)
func int8ToStr(arr []int8) string {
b := make([]byte, 0, len(arr))
for _, v := range arr {
if v == 0x00 {
break
}
b = append(b, byte(v))
}
return string(b)
}
func main() {
utsname := syscall.Utsname{}
err := syscall.Uname(&utsname)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%+v\n", int8ToStr(utsname.Sysname[:]))
}
```
# Benchmarking
Benchmarking is a common task that users will want to perform as many workloads are more performant in various manners running on Nanos, however, to get accurate results benchmarking needs to be done as 1:1 as possible.
One of the first tasks you need to state before trying to benchmark a workload is to define what you want to benchmark.
Request/second Throughput Latency Storage vs Networking Writes/seconds or Reads/second Random Reads/Writes or Sequential Small vs Large Reads/Writes
There is almost an infinite amount of metrics you can measure and the more precise you can be about what you are measuring the more accurate you can state your results.
It is important to isolate your workloads from each other and to take into account any differeces. This can include but is not limited to:
* cost of virtualization (eg: measuring something from a host vs measuring something from a guest)
* networking configuration
* linux vs nanos variance
* app specific differences and app configuration (eg: is it multi-process on linux but single process on nanos?)
It is highly suggested when benchmarking to do so on a public cloud for multiple reasons:
1. It is in all liklihood where the workloads will run in production.
2. You can isolate both your linux workload and your nanos workload on identical instance types.
3. You can run a third (if involving perhaps networking benchmarks) instance as your test instance away from your measurable instances.
4. You are guaranteed not to have interference from host/guest problems or bridged/usermode issues.
Networking Specific Things to Consider:
* Always measure on a bridge/tap setup - never on usermode it is slow. (By dafult ops uses usermode for local/dev work but bridge/tap on prod.)
* Always measure on the same class c or /24 but not on the same instance you are trying to measure. You don't want to measure across the public internet but you also don't want to incorrectly measure from the same system you expect to measure.
* Linux guest tuning: If you are measuring on real hardware there are other configurations to be aware of. Measuring soething running on a host vs a guest will always be superior for instance. You should also take into account pot limits and the fact that most linux systems will be setup for multiple queues.
If measuring HTTP you may wish to consider wrk instead of ab for multi-threaded reasons.
These are just a handful of suggestions to help you get more accurate results and helps the NanoVMs team address performance issues in a quicker manner.
### Benchmarking with Apache Bench
You should note that without using keep-alive you might quickly run into issues with time wait.
ab uses a range of \~14k ports when opening new connections and when that is used up the ports start being reused, however you can't accept new connections from the port until the TIME\_WAIT period expires (default is 2 minutes).
```
ab -c 10 -n 100 -k http://host:port/
ab -c 10 -n 1000 -k http://host:port/
```
### Benchmarking with Wrk
Generally speaking wrk is a better tool to use here as it is multi-threaded. You are almost guaranteed to see better results for a variety of reasons but mainly cause of the threaded approach.
```
wrk http://host:port/
```
As with all benchmarking your results could vary quite a bit from workload to workload.
### Benchmarking with FIO
FIO allows you to test I/O.
In order to use fio you'll need to make two changes:
1. Disable shared memory with fio:
```sh
./configure --disable-shm
```
2. Stub get/setpriority syscalls:
```sh
+++ b/src/unix/syscall.c
@@ -2493,6 +2493,8 @@ void register_file_syscalls(struct syscall *map)
register_syscall(map, io_uring_enter, io_uring_enter);
register_syscall(map, io_uring_register, io_uring_register);
register_syscall(map, getcpu, getcpu);
+ register_syscall(map, getpriority, syscall_ignore);
+ register_syscall(map, setpriority, syscall_ignore);
}
```
A sample config might look like so:
```
:~/$ cat testnew.fio
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=4
time_based=1
runtime=10
filename=test
thread
[file1]
size=500m
ioengine=libaio
iodepth=16
```
Note: If using libaio you'll want direct=1 otherwise you might end up just testing the page cache. If you want to use direct=0 you'll want to test using other engines.
If you are testing remotely (eg: in the cloud) you may wish to turn on idle\_on\_exit:
```
"ManifestPassthrough": {
"idle_on_exit": "*"
}
```
Otherwise you can simply use a sample json like:
```
{
"BaseVolumeSz": "800m",
"Args": ["testnew.fio"],
"Files": ["testnew.fio"]
}
```
You may also wish to prevent fio from starting immediately:
```
diff --git a/fio.c b/fio.c
index 3d6ce597..72f1aea2 100644
--- a/fio.c
+++ b/fio.c
@@ -22,11 +22,18 @@
*
*/
#include "fio.h"
+#include
+#include
int main(int argc, char *argv[], char *envp[])
{
int ret = 1;
+ printf("waiting for 5\n");
+
+ sleep(5);
+ printf("done sleeping\n");
```
# Browsers
**Chrome**
You might find in recent chrome && mac releases the inability to connect to a local private ip such as 192.168.x.x. There is a known issue where MacOS is not recoginizing newer builds of chrome as the same application.
To get around this you can go to Applications. Right-click on chrome, show package contents. Then simply rename the binary in Contents/MacOS/Google Chrome to something like 'chrome 2'. Try to open it, let it fail, then rename it back. The next time you try to open a local ip you'll be prompted to allow access to the local network.
# Visual Studio Code Extension
You can use Ops extension to manage your local unikernels in Visual Studio Code. Access the [marketplace](https://marketplace.visualstudio.com/items?itemName=nanovms.ops) to download or search for ops in Visual Studio Code extension explorer.
Use the shorcut `ctrl+shift+P` or `cmd+shift+P` to open the commands pallete and select one of the next commands.
| Command | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| Ops: Run | Allows to select an elf or javascript file from the filesystem and launch the program on an unikernel. |
| Ops: Stop | Allows to select a running unikernel and stops the execution. |
| Ops: Run open file | Launches an unikernel with the file that is opened in the editor. Currently, this command only supports javascript files. |
| Ops: Build | Allows to select an elf file or javscript file and prepares a nanos image to run the program. |
# Env Setup
These instructions are only for those that wish to work on the OPS codebase itself otherwise you should consult [getting\_started](/ops/getting_started).
### Supported Operating Systems
Currently Ops support various forms of linux operating systems.
* MacOS
* Debian
* Ubuntu
* Fedora
* Centos
* Windows
## Install Prerequisites
If you plan on installing Ops using the install script vs source code, you only need to install QEMU. All other prerequisite installs can be skipped on this page.
### Install QEMU
Ensure that you install a version of QEMU that is equal to or greater than version 7. NanoVMs will support older versions of qemu for organizations with support contracts, however, most developers will want to use newer versions of qemu to take advantage of the latest features.
**MacOS (homebrew)**
If you're running on a Mac, you do need to install QEMU. The easiest way to install QEMU is via homebrew.
```sh
$ brew install qemu
```
For alternate ways of installing QEMU, see their [website](https://www.qemu.org/).
**Debian / Ubuntu (apt-get)**
1. To install `QEMU`, run the following command...
```sh
$ sudo apt-get install qemu-kvm qemu-utils
```
**Fedora (dnf/yum)**
1. To install `qemu`, run the following command...
```sh
$ sudo dnf update
$ sudo dnf install qemu-kvm qemu-img
```
Or...
```sh
$ sudo yum update
$ sudo yum install qemu-kvm qemu-img
```
### Install Make
You'll also need to install the `make` command line tool. Various Linux distributions have different methods for doing so.
#### MacOS
1. To install `make`, run the following command line command...
```sh
$ xcode-select --install
```
#### Debian / Ubuntu (apt-get)
1. To install `make`, run the following command...
```sh
$ sudo apt-get install build-essential
```
2. Check that `make` is properly installed by running the following command...
```sh
$ make --version
```
#### Fedora (dnf/yum)
1. To install `make`, run the following command...
```sh
$ sudo dnf update
$ sudo dnf groupinstall "Development Tools"
```
Or...
```sh
$ sudo yum update
$ sudo yum groupinstall "Development Tools"
```
2. Check that `make` is properly installed by running the following command...
```sh
$ make --version
```
### Install Git
First we need to make sure you have `git` installed. You can find the official `git` installation instructions [here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
#### MacOS (homebrew)
1. To install `git`, run the following command. Make sure you have [homebrew](https://brew.sh/) installed first.
```sh
$ brew install git
```
#### Debian / Ubuntu (apt-get)
1. To install `git` on a Debian or Ubuntu instance, run the following command...
```sh
$ sudo apt-get update
$ sudo apt-get install git
```
2. Next verify that `git` is installed by running the following command...
```sh
$ git --version
git version 2.18.0
```
3. Configure your `git` username and email address. These details will be associated with any commit that may be made.
```sh
$ git config --global user.name "Emma Paris"
$ git config --global user.email "eparis@atlassian.com"
```
#### Fedora (dnf/yum)
1. To install `git` on a Fedora instance, run the following command...
```sh
$ sudo dnf install git
```
Or...
```sh
$ sudo yum install git
```
2. Next verify that `git` is installed by running the following command...
```sh
$ git --version
git version 2.18.0
```
3. Configure your `git` username and email address. These details will be associated with any commit that may be made.
```sh
$ git config --global user.name "Emma Paris"
$ git config --global user.email "eparis@atlassian.com"
```
#### Build from Source
**Debian / Ubuntu**
1. There are a few dependencies that are needed to build `git` from source on Debian / Ubuntu. You can install them with the following command...
```sh
$ sudo apt-get update
$ sudo apt-get install libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev asciidoc xmlto docbook2x
```
2. To get the `git` source code, [download and extract it](https://www.kernel.org/pub/software/scm/git/).
3. Use the `make` command to build and install `git`.
```sh
$ make all doc info prefix=/usr
$ sudo make install install-doc install-html install-info install-man prefix=/usr
```
**Fedora**
1. There are a few dependencies that are needed to build `git` from source on Fedora. You can install them with the following command...
```sh
$ sudo dnf install curl-devel expat-devel gettext-devel openssl-devel perl-devel zlib-devel asciidoc xmlto docbook2X
```
For installing via `yum`, you need to install the Extra Packages for Enterprise Linux (EPEL) repository first:
```sh
$ sudo yum install epel-release
$ sudo yum install curl-devel expat-devel gettext-devel openssl-devel perl-devel zlib-devel asciidoc xmlto docbook2X
```
2. Symlink `docbook2X` to the filename that `git` build expects...
```sh
$ sudo ln -s /usr/bin/db2x_docbook2texi /usr/bin/docbook2x-texi
```
3. To get the `git` source code, [download and extract it](https://www.kernel.org/pub/software/scm/git/).
4. Use the `make` command to build and install `git`.
```sh
$ make all doc info prefix=/usr
$ sudo make install install-doc install-html install-man prefix=/usr
```
### Install Go
It is a requirement to have go installed on your system. To do so, follow the documentation below. `ops` requires that the minimum version of go that you install is v1.19.
#### Build from Source
1. First pull the source and extract it.
```sh
$ curl -O https://storage.googleapis.com/golang/go1.11.2.linux-amd64.tar.gz
$ tar -xvf go1.11.2.linux-amd64.tar.gz
$ sudo mv go /usr/local
```
2. Next, we need to setup your Go path. Open up the following file with your editor of choice.
```sh
$ nano ~/.profile
```
```
Add the following lines...
```
```sh
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
```
```
Once the file is edited, source it to load the new environment variables.
```
```sh
$ source ~/.profile
```
3. Now that go is installed, let's check and make sure it is working properly.
```sh
$ go version
```
It is important to note that building ops can require more than 0.5 gig of ram. OPS can currently run applications under this limit but *building* ops requires at least this much.
## Next Steps
Now that we have the prerequisites installed, the next step is to install Ops. Go to the [installation instructions](/ops/developer/source-installation).
# Install from Source
To install Ops, you will first need to [install prerequisites](https://github.com/nanovms/ops-documentation/tree/master/prerequisites.md#install-prerequisites).
## Install from Source
### Getting the Source Code
To get the source code, clone it from the official github repo, which can be found [here](https://github.com/nanovms/ops)
```sh
$ git clone git@github.com:nanovms/ops.git $GOPATH/src/github.com/nanovms/ops
```
Once you've clone the repo, `cd` into the `ops` directory.
```sh
$ cd $GOPATH/src/github.com/nanovms/ops
```
### Installing dependencies
Use the `make` command to install the various dependencies of Ops.
```sh
$ make deps
```
### Build the Executable
To compile the source code into an executable, use the `make` command.
First you need to generate some go code from Protocol Buffers included in ops source repository. If you don't have the needed tools, you can install them by issuing the following commands:
```sh
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
$ go install github.com/bufbuild/buf/cmd/buf@latest
```
Once ready you can generate the necessary code by using:
```sh
$ make generate
```
Finally you can build the `ops` binary by using:
```sh
$ make build
```
#### Build the Executable including specified providers only
By default, when running `make build`, all providers are included and available.
However you can select a subset of provider(s) to be included if required (except `onprem` that is always built).
* the following command builds only `onprem` and all other providers are disabled.
```sh
$ go build -tags onlyprovider
```
* the following command builds only `aws` and all other providers are disabled (except `onprem` that is always built).
```sh
$ go build -tags onlyprovider,aws
```
To select the providers to build you need to always use `onlyprovider` tag accompanied with desired provider tags:
```sh
$ go build -tags onlyprovider,aws,azure,do,gcp,ibm,linode,upcloud,vultr
```
available tags:
* **onlyprovider**
* **aws**
* **azure**
* **digitalocean** (or **do**)
* **gcp**
* **hyperv**
* **ibm**
* **linode**
* **oci**
* **openshift**
* **openstack**
* **proxmox**
* **upcloud**
* **vbox**
* **vsphere**
* **vultr**