Overview

MAYANodes service the MAYAChain network. Each MAYANode is comprised of several independent servers in a cluster. All MAYANodes communicate and operate in cooperation to create a cross-chain swapping network.

Running a node is a serious undertaking. While Node Operators are well compensated for running a node, there are also risks, skills required and costs.

See the Node Operator 101 Video to learn more before running a node.

To set up a node, you have three choices:

1. Set up manually (not recommended unless you are an expert)

2. Set up via Kubernetes (recommended)

3. Set up via Provider (coming soon).

MAYANode Stack

Each MAYANode is comprised of 5 major components.

1. mayanode - this is a daemon that runs the MAYAChain chain itself and a HTTP server, that gives a RESTful API to the chain.

2. bifrost - this daemon creates connections to remote chains (like Bitcoin, Ethereum, Binance, etc) to both observe activity on those chains (incoming/outgoing transactions), and also sign/broadcast outgoing transactions (moving funds on remote chains).

3. gateway: MAYANode gateway proxy to get a single IP address for multiple deployments

4. midgard - this daemotn is a layer 2 REST API that provides front-end consumers with semi real-time rolled up data and analytics of the MAYAChain network. Most requests to the network will come through Midgard. This daemon is here to keep the chain itself from fielding large quantities of requests. You can think of it as a “read-only slave” to the chain. This keeps the resources of the network focused on processing transactions.

5. Full nodes - for every chain that is supported by the network, each MAYANode operator will run their own full node of each chain (Bitcoin, Ethereum, Binance, etc).

Churning

As new nodes join/leave the network, this triggers a “churning event”. Which means the list of validators that can commit blocks to the chain changes, and also creates a new Asgard vault, while retiring an old one. All funds in this retiring vault are moved to the new Asgard vault.

Normally, a churning event happens every 3 days (50,000 blocks), although it is possible for it to happen more frequently (such as when a node optionally requests to leave the network using the LEAVE memo).

Churning Out

On every churn, the network selects one or more nodes to be churned out of the network (which can be typically churned back in later). In a given churning event, multiple nodes may be selected to be churned out, but never more than 1/3rd of the current validator set. The criterion the network will take into account is the following:

1. Requests to leave the network (self-removal)

2. Banned by other nodes (network-removal)

3. How long an active nodes has been committing blocks (oldest gets removed)

4. Bad behavior (accrued slash points for poor node operation)

Churning In

On every churn, the network may select one or more nodes to be churned into the network but never adds more than one to the total. Which nodes that are selected are purely by validator bond size. Larger bond nodes are selected over lower bond nodes.

Risk of Running a Node

Deciding to run a node should be carefully considered and thought through. While the payoffs/rewards can be significant, there can also be an equally significant cost.

Risks to Bond

To run a node, you must obtain a significant amount of LP, and minimums apply. This CACAO is sent into the network as a “bond” and held as leverage on each node to ensure they behave in the best interest of the network.

Running a malicious node or stealing from the network results in a slashing of this bond. Here are the ways in which a validator’s bond can get slashed.

• Double Sign (5% of minimum bond) - if it is observed that a single validator node is committing blocks on multiple chains. To avoid this, never run two nodes with the same node account at the same time.

• Unauthorized transaction (1.5x transaction value) - if a node sends funds without authorization, the bond is slashed 1.5x the value of the stolen funds. The slashed bond is dumped into the pool(s) where the funds were stolen and added to the reserve.

Bond slashing takes directly from the bond and does not affect rewards.

Risk to Income

When a node is active, it earns rewards from the network in CACAO. Sufficient rewards are required to be earned for a Validator to be profitable. Running an unreliable node results in rewards being slashed. Here are the ways in which a validator’s rewards can be slashed.

• Not Observing (2 slash pts) - if a node does not observe transactions for all chains while other nodes do, they get slash points added.

• Not signing a transaction (600 slash pts) - if a node does not sign an outbound transaction, as requested by the network, they will get slash points added.

• Fail to keygen (1 hr of revenue) - When the network attempts to churn and attempts to create a new Asgard pubkey for the network and fails to do so due to a specific node(s), they will lose 1 hr of revenue from their bond.

Slash points undo profits made on the network. For every 1 slash point a node receives, they lose 1 block of rewards. Rewards slashing reduces earned rewards and does not affect a validator’s bond.

Compensation

Bond Rewards

Node Operators receive rewards if they are bonded and active on the network and are paid out in Cacao. While revenue is generated every block (every 5 seconds) to each operator, those funds are not available to the operator until they churn out of the network. Over time, this incentive increases the median bonded amount, increases the network's security, and allows the network to grow. They're claimed whenever a node leaves the network. See Keeping Track of Rewards below for more details.

Rewards are affected by Incentive Curve. The Incentive Pendulum increases and decreases the amount of CACAO allocated to nodes according to the security and capital leverage of the network.

Dynamic Inflation adds up to rewards. When the system is healthy, there is no inflation; should LPs withdraw CACAO considerably, inflation starts to kick in to incentivize pooled liquidity.

Keeping Track

When a node joins the network, the current block height is recorded. The system creates one block unit for every active node for every active block and has a running total of the block units created. When a node leaves, it cashes in its block units for a portion of the bond rewards. The spent block units are destroyed.

For example, there are 10000 CACAO in bond rewards outstanding. Node A has been active for 30 blocks and has 33 block units but accrued 3 slash points. There are 1000 block units in total. Node A leaves the network and cash in its 30 block units (33 - 3). It receives 300 CACAO ((30/1000) * 10000), leaving 9700 CACAO in node rewards. Its 33 block units are destroyed, leaving 967 block units left.

Costs

Depending on how the node was set up, it will likely cost between $1000 and $2000 per month, potentially more as the blockchain scales. The main driver of costs is resource allocation to hosting each MAYANode service.

Skillsets

Running a MAYANode is no simple task. As an operator, you must run/maintain multiple Linux servers with extremely high uptime. It is encouraged that only professional systems engineers run nodes to maintain the highest quality, reliability, and security of the network. The simple question to know if you have the skillsets to run a MAYANode is:

Have you used pagerduty before?

If the answer is no, it’s probably best not to run a node and participate in the network in other ways. The following skill sets are required to be an effective node operator.

• Advanced knowledge of Linux server administration and security

• Advanced knowledge of Kubernetes

• Advanced experience running a host of nodes on a hosted platform such as AWS, Google Cloud, Digital Ocean, etc

• Knowledge of running full nodes for other chains such as Bitcoin, Ethereum, and Binance.

• Willingness to be “on call” at all times to respond to issues when/if your node becomes unavailable

MAYANode Details

Node Accounts

When you run a MAYANode, each MAYANode will have its own node account. An example node account looks like this:

{
  "node_address": "maya19h62vypuelrj0pv4jhl26wf79yr5zhxcmd5w85",
  "aztec_address": "",
  "status": "active",
  "pub_key_set": {
    "secp256k1": "mayapub1addwnpepq0ylvhrqepmsm3rqxr4ecuyx42l4y29g2d932zlse258432ccuy8spmddj5",
    "ed25519": "mayapub1addwnpepq0ylvhrqepmsm3rqxr4ecuyx42l4y29g2d932zlse258432ccuy8spmddj5"
  },
  "validator_cons_pub_key": "mayacpub1zcjduepqzknjn39xtkdzr6a2zuzry7f02rn3cnqvy8ar7p2admk80j8xageqj9slpw",
  "bond": "10240000000000",
  "active_block_height": "180",
  "bond_address": "maya1rqhrnvyex4p5zchhu0slgr76dc4cl5dnvzxx2h",
  "status_since": "123",
  "signer_membership": [
    "mayapub1addwnpepqwm3arc5xqgjf8yt70psygcjasfj3c776cux4yxcaacyd9vm859lckczll7"
  ],
  "requested_to_leave": false,
  "forced_to_leave": false,
  "leave_height": "0",
  "ip_address": "206.189.235.75",
  "version": "0.1.0",
  "slash_points": "17",
  "jail": {
    "node_address": "maya19h62vypuelrj0pv4jhl26wf79yr5zhxcmd5w85",
    "release_height": "2393",
    "reason": "failed to perform keysign"
  }
}

Most importantly, this will tell you how many slash points the node account has accrued, their status, and the size of their bond (which is in 1e8 notation, 1 Cacao == 100000000).

Node Statuses

Types of node status:

1. Unknown - this should never be possible for a valid node account

2. Whitelisted - node has been bonded, but hasn’t set their keys yet

3. Standby - waiting to have minimum requirements verified to become “ready” status. This check happens on each churn event (3 days on average).

4. Ready - node has met minimum requirements to be churned and is ready to do so. Could be selected to churn into the network. Cannot unbond while in this status.

5. Active - node is an active participant of the network, by securing funds and committing new blocks to the MAYAChain blockchain. Cannot unbond while in this status.

6. Disabled - node has been disabled by use of LEAVE, and cannot re-join.

To get node account information, make an HTTP call to your maya-api port which will look like the following:

http://<host>:1317/mayachain/nodeaccount/<node address>
http://<host>:1317/mayachain/nodeaccounts

Node Voting

MAYANodes have the ability to vote on Mimir settings.

Mimir settings have specific abilities. The process for voting from a Node is:

Make mimir
=> Enter MAYANode Mimir key: <key>
=> Enter MAYANode Mimir value: <value>

1. Mimir keys and values are listed in the Mimir endpoint.

2. A node can vote at any time on any key value.

3. A node's vote is valid as long as they are active (and removed if they are not).

4. 2/3rds of active nodes need to agree for the change to happen

5. If 2/3rds consensus is not reached, Mimir admin takes priority, or a constant if present.

6. A node can change its vote anytime.

7. A node can delete its vote by using -1 value

8. Voting costs one native transaction fee, which is deducted from their bond.

Deploy a Kubernetes cluster

In order to deploy all the different services and provide a high availability environment to operate your node, Kubernetes is the preferred scheduling platform. Any production-grade Kubernetes cluster can be used to run and deploy a MAYANode. You need your Kubernetes provider to offer external load balancers services type features. Azure, Digital Ocean, GCE, OpenStack are compatible with external load balancers.

This Terraform deployment will deploy a Kubernetes cluster using your VPS provider credentials and EKS service. The cluster will have autoscaling capabilities, which means you don’t have to deal with how many nodes you need to deploy to run your MAYANode services.

All the default configurations used in these instructions are for a production environment with enough resources to run your MAYANode in good conditions.

Steps

There are three important steps to getting your node set up, deployed and churned in.

1. Setting up Cluster

2. Deploying MAYANode Services

3. Joining ("Churning In")

Repository Management

Your repository should be organised as follows:

./mayanode-ops
  |./cluster-launcher
  |./node-launcher

All of your set up commands are run in cluster-launcher and all of your deploying/joining/managing/leaving commands are run from node-launcher

Running Two or More Nodes

./mayanode-ops
  |./cluster-launcher
  |./node-launcher
./mayanode-ops2
  |./cluster-launcher
  |./node-launcher

All of your commands can now be run separately.

Deploy a Kubernetes cluster in Linode using LKE service.

Requirements

1. a Linode account

2. linode-cli and linode credentials configured

3. kubectl

linode-cli

To install the linode-cli (Linode CLI), follow these instructions.

You need to have pip (python) on your system.

pip install linode-cli --upgrade

Create a Linode API token for your account with read and write access from your profile page. The token string is only displayed once, so save it in a safe place.

Use the API token to grant linode-cli access to your Linode account. Pass in the token string when prompted by linode-cli.

linode-cli

kubectl

To install the kubectl (Kubernetes CLI), follow these instructions or choose a package manager based on your operating system.

MacOS:

Use the package manager homebrew to install kubectl.

brew install kubernetes-cli

Windows:

Use the package manager Chocolatey to install kubectl.

choco install kubernetes-cli

wget

To install the wget, follow these instructions or choose a package manager based on your operating system.

MacOS:

Use the package manager homebrew to install wget.

brew install wget

Windows:

Use the package manager Chocolatey to install wget.

choco install wget

Deploy Kubernetes Cluster

Use the commands below to deploy a Kubernetes cluster.

You can run the make command that automates those command for you like this:

make linode

Or manually run each commands:

cd linode/
terraform init
terraform plan # to see the plan
terraform apply

Configure kubectl

Now that you've provisioned your Kubernetes cluster, you need to configure kubectl.

To configure authentication from the command line, use the following command, substituting the ID of your cluster.

# Store it - method #1
jq -r ".resources[].instances[].attributes.kubeconfig" linode/terraform.tfstate | base64 -D > ~/.kube/config-linode

# Store it - method #2
linode-cli lke kubeconfig-view <use_your_cluster_id> > ~/.kube/config-linode

# Merge it and set current context
KUBECONFIG=~/.kube/config:~/.kube/config-linode kubectl config view --flatten > ~/.kube/tmpcfg && mv -f ~/.kube/tmpcfg ~/.kube/config && kubectl config use-context $(kubectl config current-context --kubeconfig=~/.kube/config-linode)

# Or just view it - method #1
jq -r ".resources[].instances[].attributes.kubeconfig" linode/terraform.tfstate | base64 -D
# Or just view it - method #2
linode-cli lke kubeconfig-view <use_your_cluster_id>

Note: If the above linode-cli command is broken you can download the file from the web dashboar for the respective cluster.

This replaces the existing configuration at ~/.kube/config.

Once done, you can check your cluster is responding correctly by running the command:

kubectl version
kubectl get nodes

Clean up your workspace

To destroy and remove previously created resources, you can run the command below.

make destroy-linode

Or run the commands manually:

cd linode/
terraform destroy

Deploy a Kubernetes cluster in Azure using AKS service.

Requirements

1. Azure account

2. az and Azure credentials configured

3. kubectl

Steps

Firstly, clone and enter the cluster-launcher repository. All commands in this section are to be run inside this repo.

git clone https://gitlab.com/mayachain/devops/cluster-launcher
cd cluster-launcher

Then install the terraform CLI:

brew install terraform

Azure CLI

The Azure CLI allows you to manage your Azure services.

brew install azure-cli
az login

Kubernetes Control Tool

You must install and configure the Kubernetes CLI tool (kubectl). **To install kubectl** , follow these instructions, or choose a package manager based on your operating system.

brew install kubernetes-cli

wget && jq

You also need wget and jq, follow these instructions, or choose a package manager based on your operating system.

brew install wget
brew install jq

Deploy Kubernetes Cluster

Use the commands below to deploy an AKS cluster:

make azure

During the deploy, you will be asked to enter information about your cluster:

var.location
  The location where the Managed Kubernetes Cluster should be created

  Enter a value: eastus2

var.name
  The base name used for all resources

  Enter a value: ma-k8s

• Location -- az account list-locations -o table

• Name

• Confirm yes

CONFIGURE

Now that you've provisioned your AKS cluster, you need to configure kubectl. Customize the following command with your cluster name and resource group. It will get the access credentials for your cluster and automatically configure kubectl.

az aks get-credentials -a -g <resource_group> -n <cluster_name>

This replaces the existing configuration at ~/.kube/config.

Once done, you can check if your cluster is responding correctly by running the following commands.

kubectl version
kubectl get nodes

You are now ready to deploy a MAYANode.

Checkout the repository source to manage a cluster of dedicated servers on Hetzner.

The scripts in this repository will set up and maintain one or more kubernetes clusters consisting of dedicated Hetzner servers. Each cluster will also be provisioned to operate as a node in the MAYACHain network.

Executing the scripts in combination with some manual procedures will get you highly available, secure clusters with the following features on bare metal.

• Kubespray (based)

• Internal NVMe storage (Ceph/Rook)

• Virtual LAN (also over multiple locations) (Calico)

• Load Balancing (MetalLB)

Preparations

Servers

Acquire a couple of servers as the basis for a cluster (AX41-NVME's are working well, for instance). Visit the admin panel and name the servers appropriately.

ma-k8s-node1
ma-k8s-node2
ma-k8s-node3
...

ma-k8s-master1
ma-k8s-master2
ma-k8s-worker1
ma-k8s-worker2
ma-k8s-worker3
...

Refer to the reset procedure to initialize them properly.

vSwitch

Create a vSwitch and order an appropriate subnet (it may take a while to show up after the order). Give the vSwitch a name (i.e. ma-k8s-net) and assign this vSwitch to the servers.

Check out the docs for help.

Usage

Clone this repository cd into it and download kubespray.

git submodule init && git submodule update

Create a Python virtual environment or similar.

# Optional
virtualenv -p python3 venv

Install dependencies required by Python and Ansible Glaxy.

pip install -r requirements.python.txt
ansible-galaxy install -r requirements.ansible.yml

Note: Mitogen does not work with ansible collections and the strategy must be changed (i.e. strategy: linear).

Provisioning

Create a deployment environment inventory file for each cluster you want to manage.

cp hosts.example inventory/production.yml
cp hosts.example inventory/test.yml
cp hosts.example inventory/environment.yml
...

cp hosts.example inventory/production-01.yml
cp hosts.example inventory/production-02.yml
...

cp hosts.example inventory/production-helsinki.yml
cp hosts.example inventory/whatever.yml

Edit the inventory file with your server ip's and network information and customize everything to your needs.

# Manage a cluster
ansible-playbook cluster.init.yml -i inventory/environment.yml
ansible-playbook --become --become-user=root kubespray/cluster.yml -i inventory/environment.yml
ansible-playbook cluster.finish.yml -i inventory/environment.yml

# Run custom playbooks
ansible-playbook private-cluster.yml -i inventory/environment.yml
ansible-playbook private-test-cluster.yml -i inventory/environment.yml
ansible-playbook private-whatever-cluster.yml -i inventory/environment.yml

Check this out for more playbooks on cluster management.

MAYAChain

For the cluster to operate as a node in the MAYACHain network, deploy as instructed here. You can also refer to the node-launcher repository, if necessary, or the MAYAChain documentation as a whole.

Resetting the bare metal servers

This will install and use Ubuntu 20.04 on only one of the two internal NVMe drives. The unused ones will be used for persistent storage with ceph/rook. You can check the internal drive setup with lsblk. Change it accordingly in the command shown above when necessary.

Manually

Visit the console and put each server of the cluster into rescue mode. Then execute the following script.

installimage -a -r no -i images/Ubuntu-2004-focal-64-minimal.tar.gz -p /:ext4:all -d nvme0n1 -f yes -t yes -n hostname

Automatically

Create a pristine state by running the playbooks in sequence.

ansible-playbook server.rescue.yml -i inventory/environment.yml
ansible-playbook server.bootstrap.yml -i inventory/environment.yml

Instantiation

Instantiate the servers.

ansible-playbook server.instantiate.yml -i inventory/environment.yml

Deploy a Kubernetes cluster in GCP using GKE service.

Requirements

1. GCP account

2. gcloud and GCP credentials configured

3. kubectl

Steps

Firstly, clone and enter the cluster-launcher repository. All commands in this section are to be run inside this repo.

git clone https://gitlab.com/mayachain/devops/cluster-launcher
cd cluster-launcher

Then install the terraform CLI:

brew install terraform

gcloud CLI

The gcloud CLI allows you to manage your GCP services.

brew install google-cloud-sdk

After the installation, perform the steps outlined below. This will authorize the SDK to access GCP using your user account credentials and add the SDK to your PATH. It requires you to log in and select the project you want to work in. Then add your account to the Application Default Credentials (ADC). This will allow Terraform to access these credentials to provision resources on GCP. Finally, you need to enable the Compute Engine and Kubernetes Engine API services for your GCP project.

gcloud init
gcloud auth application-default login
gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com

Kubernetes Control Tool

You must install and configure the Kubernetes CLI tool (kubectl). **To install kubectl** , follow these instructions, or choose a package manager based on your operating system.

brew install kubernetes-cli

wget && jq

You also need wget and jq, follow these instructions, or choose a package manager based on your operating system.

brew install wget
brew install jq

Deploy Kubernetes Cluster

Use the commands below to deploy an GKE cluster:

make gcp

During the deployment, you will be asked to enter information about your cluster:

var.project_id
  Project ID

  Enter a value: ma-k8s-123456

var.zone
  GCP zone in region

  Enter a value: us-east1-d

• Project ID

• Zone -- gcloud compute zones list

• Confirm yes

CONFIGURE

Now that you've provisioned your GKE cluster, you need to configure kubectl. The following command will get the access credentials for your cluster and automatically configure kubectl.

(cd gcp && gcloud container clusters get-credentials $(terraform output cluster_name) --zone $(terraform output zone))

This replaces the existing configuration at ~/.kube/config.

Once done, you can check if your cluster is responding correctly by running the following commands.

kubectl version
kubectl get nodes

You are now ready to deploy a MAYANode.

Deploy an umnanaged Kubernetes cluster in hcloud

Requirements

1. hcloud account

2. hcloud and hcloud credentials configured

3. kubectl

4. ansible

Steps

Firstly, clone and enter the cluster-launcher repository. All commands in this section are to be run inside this repo.

git clone https://gitlab.com/mayachain/devops/cluster-launcher
cd cluster-launcher

Then install the terraform CLI:

brew install terraform

hcloud CLI

The hcloud CLI allows you to manage your hcloud services.

brew install hcloud
hcloud context create <project_name>

Kubernetes Control Tool

You must install and configure the Kubernetes CLI tool (kubectl). **To install kubectl** , follow these instructions, or choose a package manager based on your operating system.

brew install kubernetes-cli

wget && jq

You also need wget and jq, follow these instructions, or choose a package manager based on your operating system.

brew install wget
brew install jq

Environment

Initialize the git submodule.

git submodule update --init

Use direnv, venv or whatever you prefer to manage a python environment inside the hcloud folder.

# Optional
(cd hcloud && direnv allow)

# Optional
(cd hcloud && virtualenv -p python3 venv)

Install dependencies required by Python and Ansible Galaxy.

(cd hcloud && pip install -r ansible/requirements.txt)
(cd hcloud && ansible-galaxy install -r ansible/requirements.yml)

Deploy Kubernetes Cluster

Use the commands below to deploy an hcloud cluster:

make hcloud

During the deploy, you will be asked to enter information about your cluster:

var.name
  The base name used for all resources

  Enter a value: ma-k8s

var.token
  Hetzner Cloud API token

  Enter a value: <secret>

var.user_name
  The admin user name for the nodes

  Enter a value: admin

• Name

• Token

• Confirm yes

Quotas

If necessary, request a quota increase here.

CONFIGURE

Now that you've provisioned your hcloud cluster, you need to configure kubectl. Customize the following command with your cluster name and resource group. It will get the access credentials for your cluster and automatically configure kubectl.

(cd hcloud && scp $(terraform output -raw hcloud_config) ~/.kube/config-hcloud)

# Merge it and set current context
KUBECONFIG=~/.kube/config:~/.kube/config-hcloud kubectl config view --flatten > ~/.kube/tmpcfg && mv -f ~/.kube/tmpcfg ~/.kube/config && kubectl config use-context $(kubectl config current-context --kubeconfig=$HOME/.kube/config-hcloud)

kubectl version

You are now ready to deploy a MAYANode.

Deploy a Kubernetes cluster in DO using EKS service.

Requirements

1. DO account

2. doctl and DO credentials configured

3. kubectl

Steps

Firstly, clone and enter the cluster-launcher repository. All commands in this section are to be run inside this repo.

git clone https://gitlab.com/mayachain/devops/cluster-launcher
cd cluster-launcher

Then install the terraform CLI:

brew install terraform

DOCLI

The Digital Ocean Control tool allows you to manage your DO services.

brew install doctl
doctl auth init --context <NAME>
doctl auth switch --context <NAME>
doctl account get

Kubernetes Control Tool

You must install and configure the Kubernetes CLI tool (kubectl). **To install kubectl** , follow these instructions, or choose a package manager based on your operating system.

brew install kubernetes-cli

wget && jq

You also need wget and jq, follow these instructions, or choose a package manager based on your operating system.

brew install wget
brew install jq

Deploy Kubernetes Cluster

Use the commands below to deploy a DOKS cluster:

make do

During the deploy, you will be asked to enter information about your cluster:

• Name

• DO Region -- see valid List of Regions (use lower-case)

• Confirm yes

Final success message: Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

CONFIGURE

Now that you've provisioned your DOKS cluster, you need to configure kubectl. Customize the following command with your cluster name and region.

doctl kubernetes cluster kubeconfig save <use_your_cluster_name>
kubectl version

If successful, you will see:

Notice: Adding cluster credentials to kubeconfig file found in "/home/user/.kube/config"
Notice: Setting current-context to do-<region_name>-<cluster_name>

Test this configuration,

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T06:30:04Z", GoVersion:"go1.14.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

To verify, run this, and check the status is "Ready":

kubectl get nodes

NAME                          STATUS   ROLES    AGE     VERSION
<cluster_name>-pool-5xhc1     READY    <none>   6m      v1.18.6

You are now ready to deploy a MAYANode.

Deploy a Kubernetes cluster in AWS using EKS service.

Requirements

1. AWS account

2. CLI and AWS credentials configured

3. AWS IAM Authenticator

4. kubectl

5. wget (required for EKS module)

Steps

Firstly, clone and enter the cluster-launcher repository. All commands in this section are to be run inside this repo.

git clone https://gitlab.com/mayachain/devops/cluster-launcher
cd cluster-launcher

Then install the terraform CLI:

brew install terraform

AWS CLI

In order for Terraform to run operations on your behalf, you must install and configure the AWS CLI tool. ****To install the AWS CLI, follow these instructions, or choose a package manager based on your operating system.

brew install awscli
aws configure

AWS IAM Authenticator

You also must install and configure the AWS IAM Authenticator tool. ****To install, follow these instructions, or choose a package manager based on your operating system.

brew install aws-iam-authenticator

Kubernetes Control Tool

You must install and configure the Kubernetes CLI tool (kubectl). ****To install kubectl , follow these instructions, or choose a package manager based on your operating system.

brew install kubernetes-cli

wget && jq

You also need wget and jq, follow these instructions, or choose a package manager based on your operating system.

brew install wget 
brew install jq

Deploy Kubernetes Cluster

Use the commands below to deploy an AWS EKS cluster. You can run the make command that automates those command for you like this:

make aws

During the deploy, you will be asked to enter information about your cluster:

• Name

• AWS Region -- see valid List of Regions

• Confirm yes

Or manually:

cd aws/
terraform init
terraform plan # to see the plan
terraform apply

Final success message: Apply complete! Resources: 30 added, 0 changed, 0 destroyed.

CONFIGURE kubectl

This is done automatically during provisioning. To configure authentication from the command line, use the following command. It will get the access credentials for your cluster and automatically configure kubectl in case you need to to manually reconfigure kubectl.

make kubeconfig-aws

Or get your kubeconfig file manually:

(cd aws && aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name))

To verify, run this, and check the status is "Ready":

kubectl version
kubectl cluster-info
kubectl get nodes

You are now ready to deploy a MAYANode.

BACKUPS (OPTIONAL)

Once your node is running, use the following command to automatically backup the Persistent Volumes for your Kubernetes cluster. This may help in recovering your node in the event of a disaster.

Enable backups:

make aws-backups

Disable backups:

make aws-destroy-backups

Deploy MAYANode services

Now you have a Kubernetes cluster ready to use, you can install the MAYANode services.

Requirements

• Running Kubernetes cluster

• Kubectl configured, ready and connected to running cluster

Steps

Clone the node-launcher repo. All commands in this section are to be run inside of this repo.

git clone https://gitlab.com/mayachain/devops/node-launcher
cd node-launcher
git checkout master

Install Helm 3

Install Helm 3 if not already available on your current machine:

make helm
make helm-plugins

Tools

make tools

make destroy-tools

If you are successful, you will see the following message:

If there are any errors, they are typically fixed by running the command again.

Deploy MAYANode

Before deploy

Currently MAYAChain supports ARB, BTC, ETH, DASH, KUJI & THOR. For all of the chains that we support a fullnode is used in order to read transactions from them. Sync time varies from chain to chain, but there's a couple of things you can do to speed up some of them:

ETH

By default the fullnode of ETH will connect checkpoint node provided by Maya Protocol. If you want to customize the checkpoint node to use you can edit the checkpoint_url in ethereum-daemon/values.yaml attribute.

THOR

In order to sync THOR from a ninerealms snapshot you need to explicitly enable it by setting recover: true in thornode-daemon/values.yaml.

You have multiple commands available to deploy different configurations of MAYANode. You can deploy mainnet or stagenet (currently not churning in third-party nodes). The commands deploy the umbrella chart mayanode-stack in the background in the Kubernetes namespace mayanode (or mayanode-stagenet for stagenet) by default.

make install

After deploy

Once it's all deployed you'll need to sync from a snapshot.

$ make recover-maya
Maya only provides mainnet snapshots. Continue?
:: Are you sure? Confirm [y/n]: y

=> Select snapshot type
   pruned
   full
=> Select snapshot height
   1283017
   1359274
   1690642
   1837687
   1893933

We recommend choosing the defaults, but in case you have a specific need you can choose otherwise.

You are now ready to join the network:

Debugging

Use the following useful commands to view and debug accordingly. You should see everything running and active. Logs can be retrieved to find errors:

kubectl get pods -n mayanode
kubectl get pods --all-namespaces
kubectl logs -f <pod> -n mayanode

Kubernetes should automatically restart any service, but you can force a restart by running:

kubectl delete pod <pod> -n mayanode

kubectl logs -f binance-daemon-xxx -n mayanode

CHART SUMMARY

MAYANode full stack umbrella chart

• mayanode: Umbrella chart packaging all services needed to run a fullnode or validator MAYANode.

This should be the only chart used to run MAYANode stack unless you know what you are doing and want to run each chart separately (not recommended).

MAYANode services:

• mayanode: MAYANode daemon

• gateway: MAYANode gateway proxy to get a single IP address for multiple deployments

• bifrost: Bifrost service

• midgard: Midgard API service

External services:

• mayachain-daemon: Mayachain fullnode daemon

• arbitrum-daemon: Arbitrum fullnode daemon

• bitcoin-daemon: Bitcoin fullnode daemon

• dash-daemon: Dash fullnode daemon

• ethereum-daemon: Ethereum fullnode daemon

• kuji-daemon: Kujira fullnode daemon

• thornode-daemon: Thorchain fullnode daemon

• chain-daemon: as required for supported chains

Tools

• prometheus: Prometheus stack for metrics

• loki: Loki stack for logs

• kubernetes-dashboard: Kubernetes dashboard

Joining MAYAChain

Now that you have a MAYANode deployed in your Kubernetes cluster, you need to start operating your node to join the network.

There are a couple of steps to follow to do so.

1. Add Liquidity with Node Operator wallet

MAYAChain nodes are backed up by added liquidity, just as a normal liquidity provider to one of the supported bonding pools: ARB, BTC, DASH, ETH, KUJI or THOR at the moment.

2. Bond for the first time using the Node Operator wallet

For now, the only ways to send custom deposit memos are using custom memo deposit on El Dorado or with your node-launcher setup:

$ make shell # Select mayanode
=> Select MAYANode service
   mayanode
   bifrost
   midgard
   gateway
   thornode-daemon
   binance-daemon
   dogecoin-daemon
   gaia-daemon
   avalanche-daemon
   dash-daemon
   ethereum-daemon
   bitcoin-daemon
   litecoin-daemon
   bitcoin-cash-daemon
   midgard-timescaledb

Defaulted container "mayanode" out of: mayanode, init-external-ip (init)
/ # mayanode keys add [name for you key] --recover --keyring-backend file
> Enter your bip39 mnemonic
[paste your mnemonic]
Enter keyring passphrase:[Enter you password used for `node-launcher`]
/ # mayanode tx mayachain deposit 2000000 cacao bond:[pool]:[amount of LP units to bond]:[maya address] --from [name of your key] --keyring-backend file --yes --chain-id mayachain-mainnet-v1 --gas auto --node tcp://localhost:27147

3. Check your current node status

Next, verify your node is running correctly. _**_To check the current status of your node, you can run the command status from the node-launcher repository on your terminal:

make status

You will get an output along those lines, the example below is for a mainnet node:

   __  ________  _____   _  __        __
  /  |/  /   \ \/ /   | / |/ /__  ___/ /__
 / /|_/ / /| |\  / /| |/    / __\/ __ / -_)
/_/  /_/_/ |_|/_/_/ |_/_/|_/\___/\_,_/\__/

ADDRESS     maya1v7gqc98d7d2sugsw5p4pshv0mm24mfmzgmj64n
IP
VERSION
STATUS      Active
BOND        100
REWARDS     0.00
SLASH       0
PREFLIGHT   {
  "status": "Standby",
  "reason": "node account has invalid registered IP address",
  "code": 1
}

API         http://18.217.85.10:1317/mayachain/doc/
RPC         http://18.217.85.10:27147
MIDGARD     http://18.217.85.10:8080/v2/doc

CHAIN      SYNC       BEHIND         TIP
MAYA       100.000%   0              2,086,031
THOR       100.000%   0              11,866,225
ETH        100.000%   0              17,773,085
ETH (beacon slot)  100.000%   0              6,958,344
BTC        100.000%   0              800,261
DASH       100.000%   0              1,910,014

Your node is running but as you can see in the `Preflight` section, your node is not yet ready to be churned in and currently is in standby status, since your node has no IP address setup yet.

But to be able to set up the node IP address, you first need to get it registered in the chain by sending your BOND.

4. Setup Node IP Address

You must tell MAYAChain your IP-Address for its address book and seed-service to run properly:

make set-ip-address

If you run the status command again, you should now see a different message for the Preflight section saying you need to set your node keys.

5. Setup Node keys

Tell MAYAChain about your public keys for signing sessions:

make set-node-keys

If you run the make status command again, you should now see that your node is in status “ready” and is now ready to be churned in the next rotation.

6. Set Version

Make sure your node broadcasts its latest version, else you won't churn in since MAYAChain enforces a version requirement. This version will appear in your make status. If you are on 0.0.0 then you haven't set your version:

make set-version

Bonding The Right Amount

Although your node is ready to be churned in, it doesn’t mean it will be the next one to be selected, since someone else could have posted a higher bond than you. To maximize chances of a quick entry, monitor Midgard to see what everyone else is bonding and try to outbid them. Keep an eye on maximumStandbyBond and make sure you are bonding higher than that amount.

curl http://midgard.mayanode.mayachain.info/v2/network

resp:
 "bondMetrics" : {
      "minimumActiveBond" : "10001000000000",
      "medianStandbyBond" : "1010000000000",
      "medianActiveBond" : "15001000000000",
      "averageStandbyBond" : "1010000000000",
      "maximumActiveBond" : "15001000000000",
      "averageActiveBond" : "12006800000000",
      "maximumStandbyBond" : "1010000000000",
      "totalStandbyBond" : "1010000000000",
      "totalActiveBond" : "60034000000000",
      "minimumStandbyBond" : "1010000000000"
   }

The endpoint will show data on average, median, total, minimum and maximum bond amounts. For fastest entry, bond higher than the current maximum.

Bonding More

At any time during standby or active, you can bond more by adding more liquidity to any LP position of the whitelisted bond providers (including the bond_address)

MAYANode commands

The Makefile provide different commands to help you operate your MAYANode.

There are two types of make commands, READ and WRITE.

READ COMMANDS

Read commands simply read your node state and doesn't commit any transactions.

make status

make shell

make logs

make mnemonic

make password

make restart

make reset

WRITE COMMANDS

Write commands actually build and write transactions into the underlying statechain. They cost CACAO from your bond, currently 0.02, but you can check this on the /constants endpoint "CLICOSTINCACAO". This will post state in the chain which will be now updated globally. The CACAO fee is to prevent DDoS attacks.

make set-node-keys

make set-ip-address

kubectl set image deployment/thor-daemon thor-daemon=registry.gitlab.com/mayachain/mayanode:mainnet-0.2.0
kubectl set image deployment/thor-api thor-api=registry.gitlab.com/mayachain/mayanode:mainnet-0.2.0
kubectl set image deployment/bifrost bifrost=registry.gitlab.com/mayachain/mayanode:mainnet-0.2.0

kubectl set image deployment/midgard midgard=registry.gitlab.com/mayachain/midgard:mainnet-0.2.0

kubectl get deployment/thor-daemon

make set-version

Tools

Note, all of these should already be installed from make tools. However you can install them separately useing the DEPLOY tabs below.

To access the tools, navigate to the ACCESS tabs below.

All of these commands are to be run from node-launcher

LOGS MANAGEMENT (LOKI)

It is recommended to deploy a logs management ingestor stack within Kubernetes to redirect all logs within a database to keep history over time as Kubernetes automatically rotates logs after a while to avoid filling the disks. The default stack used within this repository is Loki, created by Grafana and open source. To access the logs you can then use the Grafana admin interface that was deployed through the Prometheus command.

make install-loki

ACCESS

Access Grafana

See previous section to access the Grafana admin interface through the command make grafana.

Browse Logs

Within the Grafana admin interface, to access the logs, find the Explore view from the left menu sidebar. Once in the Explore view, select Loki as the source, then select the service you want to show the logs by creating a query. The easiest way is to open the "Log browser" menu, then select the "job" label and then as value, select the service you want. For example you can select mayanode/bifrost to show the logs of the Bifrost service within the default mayanode namespace when deploying a mainnet validator MAYANode.

make destroy-loki

METRICS MANAGEMENT (Prometheus)

It is also recommended to deploy a Prometheus stack to monitor your cluster and your running services.

make install-metrics

ACCESS

We have created a make command to automate this task to access Grafana from your local workstation:

Copy

make grafana

Open http://localhost:3000 in your browser.

Login as the admin user. The default password should have been displayed in the previous command (make grafana).

Access Prometheus admin UI

We have created a make command to automate this task to access Prometheus from your local workstation:

Copy

make prometheus

Open http://localhost:9090 in your browser.

As part of the tools command deployment, you also have deployed a Prometheus stack in addition to the Elasticsearch in your Kubernetes cluster. All CPU, memory, disk space, and MAYANode / MAYAChain custom metrics are automatically being sent to the Prometheus database backend deployed in your cluster.

You should have available different dashboards to see the metrics across your cluster by nodes, deployments, etc, and also a specific MAYANode / MAYAChain dashboard to see the MAYAChain status, with current block height, how many validators are currently active and other chain related information.

For a more in-depth introduction of Grafana, please follow the documentation here.

Kubernetes Dashboard

You can also deploy the Kubernetes dashboard to monitor your cluster resources.

make install-dashboard

make dashboard

View your kubernetes dashboard by running the following:

make dashboard

Backing up a MAYANode

You should backup your MAYANode in case of failures. By default, if you are using the Kubernetes deployments solution, all the the deployments are automatically backed up by persistent volume disks. Depending on your provider, the volumes are usually available in the provider administration UI, for example in AWS, you can find those volumes in your regular console, in the region you chose to deploy your Kubernetes cluster.

Again by default, with Kubernetes, by using persistent volumes used in the default configuration, you are already protected again restart failures at container level, or node failures. As long as you don’t specifically use the destroy commands from the Makefile or manually delete your Kubernetes deployments, your volumes will NOT be deleted at any time.

It is still recommended, as any project, to have different backups on top of those volumes to make sure you can recover in admin error deleting those volumes or other Kubernetes resources that would imply deleting those volumes.

For AWS, you can easily setup in your console to have snapshots of your cluster volumes be taken every day. For other provider there can be different ways to achieve this as well either manually or automatically.

It is up to the node operator to setup those extra backups of the core volumes to be able to recover in any kind of failures or human errors.

Some volumes would be more critical than others, for example Midgard deployment are also by default backed up by persistent volumes, so it can recover in case of container restarts, failures or node failures and the deployment being automatically scheduled to a different node, but if you were to delete the Midgard volume, it would reconstruct its data from your MAYANode API and events from scratch. For that specific service having extra backups might not be critical, at the time of the writing of that document, Midgard implementation might change in the future.

At minimum you should also securely backup your node keys: node_key.json and priv_validator_key.json. Do this as follows:

kubectl get pods -n mayanode

Copy the mayanode pod name, e.g. mayanode-abcdefg-hijkl and replace with {mayanode pod name} below:

kubectl cp mayanode/{mayanode pod name}:root/.mayanode/config/node_key.json node_key.json

kubectl cp mayanode/{mayanode pod name}:root/.mayanode/config/priv_validator_key.json priv_validator_key.json

For full disaster recovery (complete loss of cluster), it is possible to issue LEAVE command from the original BOND wallet and manual return of funds from your Yggdrasil. In this case you need a secure backup of make mnemonic (Yggdrasil mnemonic) and a working wallet that did the original BOND. See Leaving.

Node Security

The following are attack vectors:

1. If anyone accesses your cloud credentials, they can log in and steal your funds

2. If anyone accesses the device you used to log into kubernetes, they can log in and steal your funds

3. If anyone accesses your hardware device used to bond, they can sign a LEAVE transaction and steal your bond once it is returned

4. If anyone has your Yggdrasil make mnemonic phrase, including in logs, they can steal your funds

5. If any GitLab repo is compromised and you git pull any nefarius code into your node and run make <any command>, you can lose all your funds.

Checking diffs

Prior to git pull or make pull updates, review node-launcher repo diffs:

git fetch
git diff multichain..origin/multichain

Regularly review patches in GitLab: https://gitlab.com/mayachain/devops/node-launcher/-/commits/multichain

When chain clients have updated tags (version number or sha256), inspect the GitLab diffs for the relevant image in https://gitlab.com/mayachain/devops and ensure the CI build checksum matches the expected. This ensures you are executing code on your node that you are satisfied is free from exploits. Some images such as Ethereum use the 'official' docker image, e.g. https://hub.docker.com/r/ethereum/client-go/tags.

Yggdrasil vaults

At the moment , there are five external chain get connected to MAYAChain, there are

• Binance Chain

• Bitcoin

• Bitcoin cash

• Litecoin

• Ethereum

Each node has a unique address on each supported chain. This is their Yggdrasil vault. The network will fund all nodes Yggdrasil vaults and instruct them to perform small transactions in order to lower the number of computationally expensive TSS signatures.

Finding Yggdrasil addresses

To find your Yggdrasil addresses, firstly navigate to https://viewblock.io/mayachain/vaults

1. Find your node address and click on the link.

Alternatively, visit any mayachain endpoint using your node address from make status:

http://mayanode.mayachain.info/mayachain/node/<Node Address>

Copy your secp256k1 public key and put it here:

http://mayanode.mayachain.info/mayachain/vault/<Public Key>

And look for addresses array at the bottom.

Finding Yggdrasil Private Key

1. Run make mnemonic and securely store this.

2. Visit https://iancoleman.io/bip39/ - or for more safety, clone the GitHub repo and open src/index.html offline.

3. Paste in your mnemonic and choose CACAO - MAYAChain from the drop-down list.

4. Your private key string is the first one: m/44'/931'/0'/0/0

Dealing with slash

When running a node, it is quite common to get slashed. The network relies on slash points to rate node quality. When your node is slashed, the first thing you need to do is run make status, and make sure all your chains are 100% in sync. If any of the external chains are not 100% in sync, then it will cause node to be slashed due to missing observations.

The best prevention is to have a cluster with lots of fast resources (cpu, memory, IO, network) and good backups/redundancy to prevent downtime.

Unfortunately even when your node is fully in-sync, it is still possible to be slashed due to external chain events. Here are some of the scenarios:

600 point slash (isolated)

When a node is slashed 600 points, it is typically because the yggdrasil vault failed to send an outbound transaction (more accurately: the transaction it was tasked to perform wasn't mined within a specified time limit). This most likely to happen on ETH chain. Here is what you need to check:

1. Find your Yggdrasil ETH address. Use the previous instructions.

2. Visit https://etherscan.io/ and paste in your Yggdrasil ETH address.

Potential problem 1: Transaction ran out of Gas (wrong estimate):

Cause: The network uses geth inbuilt eth_estimateGas function to estimate how much gas to set as limit for a transaction. On rare occasions this can return a number too low causing the transaction to fail. In this case there is nothing you can do - just wait it out. Note: your Yggdrasil ETH vault is now insolvent by a small amount of gas burned in the failed transaction that you will need to personally top-up prior to LEAVE. See section on LEAVE for more details.

Potential problem 2: Transaction didn't mine after 15mins

Cause: External unexpected Gas price hike. The network uses a 1.5x the previous N blocks as the gas rate to use. If there is a sudden increase in Gas price required due to unforseen external events, the transaction may not be mined. In order to make sure customer is paid in a reasonable time, there is a auto cancel transaction process build in bifrost. The network will keep monitoring the outbound transactions and if any of the outbound transaction signed out by yggdrasil vault didn't commit after 15 minutes, it will automatically cancel it and assign to another node to send.

You should be able to see a transaction like the following, which is sending 0 ETH back to itself which cancels anything pending:

600 point slash (repeated)

If your node is slashed 600 points continuously, it is likely your ETH vault is stuck or transactions sent to your local geth aren't propagated fully into mempools used by miners. This might happen if your local ethereum-daemon doesn't sync well with the network, even though it reports 100% in sync.

1. Run make logs and choose bifrost

2. Search your logs for cancel and look for transactions such as:

{"level":"info","service":"bifrost","module":"ethereum","time":"2021-05-28T14:43:58Z","message":"broadcast cancel transaction , tx hash: 0xec396286e54f9a95081e60424c73fcc0e580c47d2ffacb216ad9ef2d9c787082, nonce: 25 , new tx hash:0x5823abbee421f4c2ce230f5e7808b4dc6728ebeb5e21b62d95b812144d522672"}

1. Find the last cancel tx in the logs (with the highest nonce).

2. Search etherscan for the new tx hash transaction ID.

3. Your geth is stuck and out of sync if etherscan does NOT find the new tx hash cancelled transaction. If you do not find your tx in etherscan - proceed as follows:

4. Find the lowest nonce from Etherscan:

5. Go to https://etherscan.io and paste your yggdrasil ETH address in the search box

6. Find the last successful transaction send out from your yggdrasil ETH address. It is the top transaction in the list:

• Click the transaction. Note the nonce used in the last good transaction (e.g. 39), and then plus 1 (e.g. 40). This is the lowest stuck tx nonce.

1. Find the highest stuck nonce from your local geth:

2. make shell then choose ethereum-daemon

3. geth attach

4. eth.getTransactionCount('{YOUR YGGDRASIL ETH ADDRESS}','pending') -- this is the highest stuck tx nonce

5. exit and exit again.

6. If the highest nonce is larger than your lowest nonce, it means there are a few transactions sent and stuck in the mempool of your local ETH daemon. You need to unstuck these from highest to lowest.

7. Make a fork of https://replit.com/@mayachain/YggCancelETH

8. Update index.js lastPendingNonce and firstStuckNonce. Also put in your hex encoded private key to KEY variable. Remember to add the 0x prefix which bip39 calculator above will not have.

9. Update gasPrice to a very high gas price. The price should be higher than all the transactions stuck in the mempool. Recommend to spend more than 200 Gwei or double the highest from https://mayanode.mayachain.info/mayachain/inbound_addresses (which ever is higher).

10. Run the script using node index.js. Note: you may need to install some dependecies first with npm install ethers. The output should look like:

    Copy

    0x2aefdf705d1b28dfdf6b524ec697082326b23a2e62b7f25a60c1d2a1a9108243
    CANCELLING 39 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0x1cc81e968d68cfddcd8b605591b9ac7955ec7fdabf222678d3bcfaea4d7c4fd0
    CANCELLING 38 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xc0a15e2ec9c92300aae3924c26a26d64f964b2c2aba8d445fea9e54bf734e58a
    CANCELLING 37 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0x1d6311b0fb33717091ec4e86282f445216068a9e13d48748906a6abf375a8392
    CANCELLING 36 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xf1ed11a1172937ed37514ba9d4d315806e1c8f49a075dde7e0c0239eed30853e
    CANCELLING 35 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xfd73ab1c15a18edc13aa9a192d85c369b4ba02fa72aa04e836fd8e9d41b5159c
    CANCELLING 34 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0x45fd4c88168e9ce86716a45f1f0ea25b233a4affa90b866023b2ca7d25f57367
    CANCELLING 33 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xd2d283479386156b9c9441dd4630b25af42d5842b910c3aaa95e28e7bc499be2
    CANCELLING 32 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0x2d37f455c9066c2d8c8f5ae2f62a720123677707047f8a4e3ee61bb1c261015e
    CANCELLING 31 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xdce9df663b7c35eb7aefc52d5640b710e2af38979615ce7686729d37dd0da5a8
    CANCELLING 30 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0xccdd1b9d8e6ac0c7314fa39f78442c7702688db56091e8d379da75a3cfbce936
    CANCELLING 29 for 0x3eb68bF15A7A6769219A66C5c493fa7C40511E19
    0x43bad098782f7bac68e401390ef300dc97d9d1d1b322eb566de1ff06b2cf9b21

11. make restart and choose ethereum-daemon

Constantly accumulating slash points

Problem: Sometimes bifrost fails to forward observations to mayanode, due to an account number / sequence number mismatch. Here is what you need to check:

1. run make logs , and choose bifrost

2. Search your bifrost logs for {"level":"error","service":"bifrost","module":"observer","error":"fail to send the tx to mayachain: fail to broadcast to MAYAChain,code:32, log:account sequence mismatch, expected 26806, got 26807: incorrect account sequence","time":"2021-05-30T07:28:18Z","message":"fail to send to MAYAChain"} 3. Solution: make restart and choose bifrost

Summary

Skilled Node Operators who don't individually have enough Liquidity Units (Asset + CACAO) to run a node themselves can use the Bond Provider feature to collect bond from other Liquidity Providers.

Node Operators can define up to 8 Maya addresses as Bond Providers. These addresses will be able to bond and unbond to the node, earning rewards proportional to the amount of bond they contribute. Node Operators define an operator fee in basis points, which defaults to zero and must be set explicitly when bond providers are added. The Node Operator fee is taken from rewards and paid directly to the Node Operator address after each churn.

Rationale

The minimum Value of Liquidity Units needed to churn in as a MAYANode can be pricy (currently above $100k).

Not many people in the world have both the technical skills to run a validator AND at least $100k, which limits the supply of Node Operators who secure MAYAChain.

Pooled MAYANodes provide a way for a skilled Operator to enter a trusted agreement with Bond Providers to bootstrap enough capital for running a MAYANode. The Network's security increases, and Liquidity Providers can earn more yield.

Economic Security

At first glance it might seem Pooled Validators contradict the economic security model of MAYAChain (i.e. that Node Operators put up more in value in slash-able bond than the assets they secure). With Pooled Validators it is possible for the Node Operator to individually put up less bond than the value of the assets that the node secures. However this nuance only exists within the relationship between the Node Operator and the Bond Providers. The Network only considers the MAYANode as single entity thus the economic model is intact.

Managing a Pooled MAYANode

Node Operator

Adding a Bond Provider

Add a bond provider using a BOND transaction with a modified memo from a wallet you control (APP or mayacli):

BOND:<NodeAddress>:<BondProviderAddress>:<NodeOperatorFee>

Example: BOND:maya7djftrgu98z84hef6dt6ykhe7cmjf3f8dcpkfun:maya9diy4q8u50qzsznpvh02s8j483aga63cl02k6jt:2000

• NodeAddress - MAYANode address (prefixed by maya)

• BondProviderAddress - Bond Provider address to whitelist (prefixed by maya)

• NodeOperatorFee - fee in basis points to be taken from rewards and paid directly to Node Operator's address after each churn.

• CACAO TX Value - 2 minimum (anything over 1 is added to the Operator's Bond).

A Node Operator is the first on-chain bonding transaction to a new node. You cannot change the operator address after the fact.

The Operator is also added as a Bond Provider.

Removing a Bond Provider

While the node is churned out, A Node Operator can remove a Bond Provider using an UNBOND transaction with a modified memo:

UNBOND:<NodeAddress>:<Amount>:<BondProviderAddress>

• NodeAddress - MAYANode address (prefixed by maya)

• Amount - amount of Bond Provider's bond to refund

• BondProviderAddress - Bond Provider address to refund/remove (prefixed by maya)

• RUNE Tx Value - 1 minimum

This command will refund the Bond Provider their bond and remove them from the Bond Provider list only if Amount constitutes all of the bond the Bond Provider is owed.

Node Operator Fee

Node operators can set a fee that is paid to them from the earnings each churn.

To set a Node Operator fee, send a deposit transaction with the following memo:

BOND:<node address>:<bond wallet address>:<operator fee in basis pts>

Example: BOND:maya1agftrgu74z84hef6dt6ykhe7cmjf3f8dcpkfun:maya1tfm4q8u57qzsznpvh02s8j483aga63cl02k6jt:2000

To adjust the Fee, The no operators can send: BOND:<node address>::<operator fee in basis pts>

Example: BOND:maya3guru5gu74z79hef6dt6ykhe7cmjf3f8dcfudge::4000 to see the fee to 40%.

Fees can range from 100 to 9900 basis pts. Setting 10000 causes all rewards to be to the node operator each churn. Setting it to 0 causes rewards to accrue to the bond.

Bond Provider

Adding/Removing Bond

Once whitelisted, a Bond Provider can Bond and Unbond from the node as normal.

Adding Bond:

BOND:<NodeAddress>

• NodeAddress - MAYANode address (prefixed by maya)

• CACAO Tx Value - Amount of LP to bond

Removing Bond:

UNBOND:<NodeAddress>:<Amount>

• NodeAddress - MAYANode address (prefixed by maya)

• Amount - Amount of LP to unbond

• CACAO Tx Value - 1

Reward Shares

Operators and Providers all have a bond amount registered to the node. Operators can start at 0.00 bonded. This on-chain bond amount is summed to the total bond, and thus everyone has a fair share in the MAYANode's principle + Liquidity Pool & transaction fees.

The Operator Fee is distributed to the Node Operator address from all CACAO rewards earned by a node after each churn.

If an Operator LEAVES, all the bond is fairly distributed.

Community alert channels

To listen for update announcements, join the following channels:

• Telegram MAYANode Announcements Join here

• Discord MAYAChain Community Devs Join here especially #mayanode-mccn

Overview

Every 3 days the system will churn its nodes. The exact churn interval in blocks is ChurnInterval in the MAYAChain Constants.

Outgoing:

1. Nodes wishing to leave, and/or

2. The most unreliable node(s), and/or

3. The oldest node

4. But a maximum of 1/3rd the network

Incoming:

1. The node(s) with the highest bond (typically 4).

Churned out nodes will be put in standby, but their bond will not automatically be returned. They will be credited any earned rewards in their last session. If they do nothing but keep their cluster online, they will be eventually churned back in.

Alternatively, an "Active" node can leave the system voluntarily, in which case they are marked to churn out first. Leaving is considered permanent, and the node-address is permanently jailed. This prevents abuse of the LEAVE system since leaving at short notice is disruptive and expensive.

Unbonding

If a Node Operator wants to retrieve part of their bond & rewards (such as deciding to take profits), they can simply Unbond. This keeps their Node on standby, ready to be churned back in.

To unbond from the system, simply send an UNBOND transaction.

Example, this will draw out 10k in CACAO from the bond, as long as the remaining amount is higher than the minimum bond.

UNBOND:maya1ryr5eancepklax5am8mdpkx6mr0rg4xjnjx6zz:1000000000000

Issues with Unbonding/Leaving

If you can't UNBOND, it means your ygg-vault still has funds on it. This means your node spent more gas than it was supposed to during the cycle (various reasons) and is partially insolvent. To fix this you need to rectify your node's insolvency first (send it the missing funds directly) before doing anything.

Leaving

Leaving is considered permanent. There are two steps.

1. If you are Active, send a LEAVE transaction to be ear-marked to churn out. This will take several hours even after changing your status to 'Standby'.

2. If you are Standby, send a LEAVE transaction to get your bond back and be permanently jailed.

To leave the system, send the following transaction from your original bond address to the Vault Address: LEAVE:<ADDRESS> with at least 1 tor in funds. Or use ASGARDEX.

Example:

LEAVE:maya1ryr5eancepklax5am8mdpkx6mr0rg4xjnjx6zz

⏱_Wait a few hours, verify on the /nodeaccount endpoint that you are now **Disabled**👀_ Then send another LEAVE:

LEAVE:maya1ryr5eancepklax5am8mdpkx6mr0rg4xjnjx6zz

⏱_Wait a few minutes, verify you have received your bond back 👀_ - make status should show BOND 0.00 and your wallet should get the full Bond back.

🔥 Commence destroying your node 🔥

Contingency: Manual Leave Procedure

If your node is completely offline or destroyed, you will have to perform a manual return of Yggdrasil funds in order to prevent 1.5x bond fine. Ensure you have reviewed this procedure and have all tools ready to go in case you need to do it in anger. This is a time-critical event - you have a few hours to return all funds before the network assumes you have stolen them.

Requirement: You have your make mnemonic Yggdrasil mnemonic available. If you do not have this, you cannot manually return funds.

Options: 1. Coming Soon: Use ASGARDEX for Manual Return. 2. Coming Soon: Check Discord Dev channels for manual return cli tool. 3. Extract Private Key + Manual return each asset using wallets:

Your Yggdrasil make mnemonic phrase is used to generate the m/44'/931'/0'/0/0 private key which is used for all chains. Pasting the mnemonic into common wallets will not work as they will be looking under a different "standard" HD Path. Instead, go to https://iancoleman.io/bip39/ and paste in your mnemonic, select CACAO from the Dropdown list and in the bottom table, copy the m/44'/931'/0'/0/0 private key string. Use this to import into wallets.

The next step is to find the latest inbound addresses. Use https://mayanode.mayachain.info/mayachain/inbound_addresses

The memo required is YGGDRASIL-:<BlockHeight>. For example YGGDRASIL-:782412. The block height can be found from the status_since field here:

https://mayanode.mayachain.info/mayachain/node/<your node address>

Destroying

Confirming you have left

You should complete this checklist before you do the next step:

1. Have you sent a final LEAVE transaction and have you received your BOND back - ie 1,000,000 CACAO, and can your account for any slash points or rewards?

If yes, then proceed:

DESTROY

To destroy and remove previously created resources, you can run the command below.

1) Destroying the Node and Tools

First, destroy the node and tools, this will delete your node then your tooling 1-by-1. Do this from the node-launcher repo:

make destroy-tools

2) Destroy the cluster

Then destroy the cluster from the cluster-launcher repo:

make destroy-aws

make destroy-do

You will be asked to confirm:

MAYAChain is a distributed network. When the network is under attack or a funds-at-risk bug is discovered, Node Operators should react quickly to secure and defend.

Reporting a Bug

There is a formal Bounty Program in place for reporting bugs. If you have discovered a bug, you should immediately DM the team or any other admins and/or report via the bounty program. If the bug is deemed to be serious/criticial, you will be paid a bounty commensurate to the severity of the issue. Reports need to include:

1. Description of the bug

2. Steps to reproduce

3. If funds are at risk

Admin Procedures

Once the bug has been verified, admin should make a decision on the level of response, including any mimir over-rides and announcements:

Critical - Funds At Risk

• Determine which level of halt is required:

• Can a trader siphon out funds by swapping? If yes, haltTrading using mimir

• Can yggdrasil vaults siphon out funds? If yes, haltInternalTx using mimir

• Can funds be siphoned out from any vault? If yes, haltOutboundTx using mimir

• Determine what type of announcement is required:

• Does the network need coordination from nodes? If yes, announce, with directions.

• Can the team immediately apply a bug patch with no coordination? If yes, simply announce the halt.

• Generate a bug fix

• Is there KVStore migration required? If yes, will require migration testing

• Is there Bifrost/TSS changes? If yes, will require network testing

• Test on Mocknet

• Release for Chaosnet

Major - Funds Not At Risk, but Network At Risk (disruption)

• Determine what type of announcement is required:

• Does the network need coordination from nodes? If yes, announce, with directions.

• Can the team immediately apply a bug patch with no coordination? If yes, consider delaying the announcement until the bug fix is ready.

• Generate a bug fix

• Is there KVStore migration required? If yes, will require migration testing

• Is there Bifrost/TSS changes? If yes, will require network testing

• Test on Mocknet

• Release for Chaosnet

Minor - Funds Not At Risk, Network Not At Risk

• Generate a bug fix

• Is there KVStore migration required? If yes, will require migration testing

• Is there Bifrost/TSS changes? If yes, will require network testing

• Deploy to Testnet

• Release for Chaosnet

Network Upgrades

The network cannot upgrade until 100% of active nodes are on the updated version. This can be accelerated:

1. Naturally, by allowing the network to churn out old nodes

2. Assertive, by waiting until a super-majority has upgraded (demonstrating acceptance of the upgrade) than banning old nodes

3. Forced by hard-forking out old nodes.

During a natural upgrade cycle, it may take several days to churn out old nodes. If the upgrade is time-critical, the network may elect to ban old nodes. Banning a node will cycle them to be churned, kick them from TSS and eject them from the consensus set. That node will never be able to churn in again, they will need to fully leave, destroy their node, and set up a new one. Hard-forking out old nodes is also possible but comes with a significant risk of consensus failures.

Network Recovery

The network will not be able to recover until the upgrade is complete, any mimir over-rides are removed, and TSS is re-synced. Additionally, there may be a backlog of transactions that will need to be processed. This may take some time. If external chain daemons were stopped, re-syncing times may also be a factor.

All wallets and frontends should monitor for any of the halts and automatically go into maintenance mode when invoked.

Node Migration

Create node backup

• Copy mnemonic with the following command

kubectl get secret mayanode-mnemonic -n mayanode -o yaml > ~/mnemonic_secret_new.yaml

• Copy key files from MAYANode pod

/root/.mayanode/MAYAChain-ED25519 /root/.mayanode/keyring-file/ (directory) /root/.mayanode/config/node_key.json /root/.mayanode/config/priv_validator_key.json

#Replace $pod with the mayanode pod name
kubectl cp mayanode/\$pod:/root/.mayanode/MAYAChain-ED25519 ~/MAYAChain-ED25519
kubectl cp mayanode/\$pod:/root/.mayanode/keyring-file/ ~/keyring-file/
kubectl cp mayanode/\$pod:/root/.mayanode/config/node_key.json ~/config/node_key.json
kubectl cp mayanode/\$pod:/root/.mayanode/config/priv_validator_key.json ~/config/priv_validator_key.json

• Make Bifrost backup

#Replace $pod with the old bifrost pod name
kubectl exec deploy/mayanode -c bifrost -- sh -c "cd /root/.mayanode && tar cf \"bifrost.tar\" localstate-*.json"
kubectl cp mayanode/\$pod:/root/.mayanode/bifrost.tar ~/bifrost.tar

Restore Node backup

• Copy node backup files made in Node backup section to the new node's control station

• Create yaml config files for temporary pods mayanode-recovery.yaml

kind: Pod
apiVersion: v1
metadata:
  name: mayanode-recovery
spec:
  volumes:
    - name: volume-to-debug
      persistentVolumeClaim:
       claimName: mayanode
  containers:
    - name: debugger
      image: busybox
      command: ['sleep', '3600']
      volumeMounts:
        - mountPath: "/data"
          name: volume-to-debug

bifrost-recovery.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: bifrost-recovery
spec:
  volumes:
    - name: volume-to-debug
      persistentVolumeClaim:
       claimName: bifrost
  containers:
    - name: debugger
      image: busybox
      command: ['sleep', '3600']
      volumeMounts:
        - mountPath: "/data"
          name: volume-to-debug

• Scale mayanode and bifrost deployments to 0

kubectl scale deployment mayanode bifrost --replicas=0

• After mayanode and bifrost are stopped run temporary pods

kubectl create -f mayanode-recovery.yaml
kubectl create -f bifrost-recovery.yaml

• Copy key files from the backup to the temporary pods

kubectl cp node_key.json mayanode-recovery:/data/.mayanode/config/node_key.json
kubectl cp priv_validator_key.json mayanode-recovery:/data/.mayanode/config/priv_validator_key.json
kubectl cp MAYAChain-ED25519 mayanode-recovery:/data/.mayanode/MAYAChain-ED25519
kubectl cp keyring-file mayanode-recovery:/data/.mayanode/
kubectl cp MAYAChain-ED25519 bifrost-recovery:/data/mayanode/MAYAChain-ED25519
kubectl cp keyring-file/mayachain.info bifrost-recovery:/data/mayanode/keyring-file/mayachain.info

• Restore bifrost backup

kubectl cp bifrost.tar bifrost-recovery:/data/mayanode/bifrost.tar
kubectl exec deploy/mayanode -c bifrost-recovery -- sh -c "cd /root/.mayanode && tar xf bifrost.tar"

• Stop temporary pods

kubectl delete -f mayanode-recovery.yaml
kubectl delete -f bifrost-recovery.yaml

If you're maling live migration, then after stopping temporary pods on the NEW node stop mayanode and bifrost daemons on the OLD node

• Re-scale mayanode and bifrost deployments

kubectl scale deployment mayanode bifrost --replicas=1

• Remove mnemonic

kubectl delete secret mayanode-mnemonic

• Restore mnemonic from the backup

kubectl apply -f mnemonic_secret_new.yaml

• Set external IP address

cd ./node-launcher/ && make set-ip-address

• Test new MAYANode

• Check pods status

• Check chain sync status

• Check node name

• Check status on the dashboard (e.g. https://mayanode.network/)

• Check slash rate

Deploying

• I have set up a local mayanode repository with a cluster-launcher and node-launcher git cloned into the right sub-folders

• I have installed all the pre-requisites

• I have a AWS/DO account ready with credentials if using a service provider

• I have deployed a cluster and saved my node details (name of node, region)

• I have installed tools via make tools

• I have deployed a node by running make <provider>

• I have securely saved my MAYANode Password and MAYANode Mnemonic

Joining

• I have added liquidity to one of the supported pools

• I have bonded my liquidity to my corresponding node

• I have confirmed my node has been credited the small bond and I am the bond_address of the node

• (Optional) I have confirmed I can unbond a small amount

• (Optional) I have whitelisted any bond providers for my node and set the node operator fee

• I have sent the final bond, higher than the MinimumBond

• I have run make set-ip-address to set my Node's IP address

• I have run make set-node-keys to set my Node's public keys

• I have run make set-version to set my Node's version

• I have verfied in make status that my node is ready to churn in

Upgrading

• I have run make status to verify the current state of my node

• I have run make pods to verify all my node's services are running properly

• I have run make update

• I have run make set-version to set my new Node version

• I have run make status to verify the final state of my node

Unbonding

• I have waited until my node is churned out and is in standby

• I have sent an UNBOND transaction from the same wallet registered to my node

• I have verified that my liquidity is no longer bonded

Leaving Whilst Active

• I have confirmed my node is active and I wish to leave before waiting to churn naturally

• I have sent the first LEAVE transaction and verified my node has "requested to leave".

• I have verified that a churn has taken place and my node is in STANDBY

• I have sent the final LEAVE transaction and my liquidity is no longer bonded

Leaving Whilst Standby

• I have verified that a churn has taken place and my node is in STANDBY

• I have verified that my node has returned all hot funds by checking my node's yggdrasil vault on the explorer

• I have sent the final LEAVE transaction and my liquidity is no longer bonded

Destroying a Node

• I have verified that I have either UNBONDED my entire BOND or LEFT and received my BOND back.

• I have run make destroy destroy-tools to destroy my node from node-launcher

• I have run make destroy-<provider> to destroy my cluster from cluster-launcher

Some node operators may desire to run multiple validators within the same cluster, while sharing a single set of daemons among them to save resource cost. This can be performed via the following method. These instructions are relevant for mainnet at the time of writing, but please ensure that correct network and current set of daemons are used.

https://gitlab.com/mayachain/devops/node-launcher/-/blob/master/docs/Multi-Validator-Cluster.md

Install Go and other requirements

Install Go

We will use Go v1.22.0 as an example here. The code below also cleanly removes any previous Go installation.

sudo rm -rvf /usr/local/go/
wget https://golang.org/dl/go1.22.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.22.2.linux-amd64.tar.gz
rm go1.22.2.linux-amd64.tar.gz

Configure Go

Unless you want to configure in a non-standard way, then set these in the ~/.bash_profile for bash and ~/.zshrc for zsh.

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GO111MODULE=on
export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin

Install dependencies protobuf-compiler

apt-get update
apt-get install -y git \
     make \
     protobuf-compiler \
     curl \
     wget \
     jq \
     build-essential \
     musl-tools \
     linux-headers-generic

Install docker and docker compose plugin

https://docs.docker.com/engine/install/

Install Node

Install the current version of node binary.

*You can see tags in mayanode's repo tags

git clone https://gitlab.com/mayachain/mayanode
cd mayanode
git checkout <latest version tag>
make protob

Install binary

TAG=mainnet NET=mainnet make install

Run Node

MAYA_COSMOS_TELEMETRY_ENABLED=true CHAIN_ID=mayachain-mainnet-v1 NET=mainnet SEEDS=15.156.45.237,18.217.85.10 bash ./build/scripts/fullnode.sh

Create Service File

Create a mayanode.service file in the /etc/systemd/system folder with the following code snippet. Make sure to replace USER with your Linux username. You need sudo privilege to do this step.

[Unit]
Description="Mayanode"
After=network-online.target

[Service]
User=USER
ExecStartPre=/home/USER/.go/bin/mayanode render-config
ExecStart=/home/USER/.go/bin/mayanode start
Restart=always
RestartSec=3
LimitNOFILE=4096
Environment="MAYA_COSMOS_TELEMETRY_ENABLED=true"
Environment="CHAIN_ID=mayachain-mainnet-v1"
Environment="NET=mainnet"
Environment="SIGNER_NAME=mayachain"
Environment="SIGNER_PASSWD=password"

[Install]
WantedBy=multi-user.target

Download Snapshot

Please use a snapshot in order to avoid syncing from start. You will need to install aws-cli

mkdir -p ~/.mayanode/data
aws s3 cp --recursive s3://public-snapshots-mayanode/full/<snapshot height> ~/.mayanode/data

Start Node Service

# Enable service
sudo systemctl enable mayanode.service

# Start service
sudo systemctl start mayanode

# Check logs
sudo journalctl -feu mayanode

Other Considerations

This installation guide is the bare minimum to get a node started. You should consider the following as you become a more experienced node operator.

• Configure firewall to close most ports while only leaving the p2p port (typically 27146) open

• Use custom ports for each node so you can run multiple nodes on the same server

If you find a bug in this installation guide, please reach out to our Discord Server and let us know.