Running a Node in Production

This document outlines the process for running a full ZetaChain node. It is intended for those who are familiar with the Linux operating system and the command line interface.

It is expected that you are familiar with setup and state sync.

Set Limits on Open Files and Number of Processes

To better manage the resources of your nodes, we recommend setting some limits on the maximum number of open file descriptors (nofile) and maximum number of processes (nproc).

Edit /etc/security/limits.conf to include or modify the following parameters:

*       soft    nproc   262144
*       hard    nproc   262144
*       soft    nofile  262144
*       hard    nofile  262144

Edit /etc/sysctl.conf to include the following:

fs.file-max=262144

Installing Cosmovisor

From the cosmos docs (opens in a new tab):

cosmovisor is a process manager for Cosmos SDK application binaries that automates application binary switch at chain upgrades. It polls the upgrade-info.json file that is created by the x/upgrade module at upgrade height, and then can automatically download the new binary, stop the current binary, switch from the old binary to the new one, and finally restart the node with the new binary.

From Source

go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@v1.5.0
mv $(go env GOPATH)/bin/cosmovisor /usr/local/bin/cosmovisor

From Precompiled Binaries

Download the correct binary from the cosmovisor v1.5.0 release (opens in a new tab). For example:

wget https://github.com/cosmos/cosmos-sdk/releases/download/cosmovisor%2Fv1.5.0/cosmovisor-v1.5.0-linux-arm64.tar.gz
tar xf cosmovisor-v1.5.0-linux-arm64.tar.gz
mv cosmovisor /usr/local/bin

Setting Up Cosmovisor

Create cosmovisor Directories

mkdir -p ~/.zetacored/config
mkdir -p ~/.zetacored/cosmovisor/genesis/bin
mkdir -p ~/.zetacored/cosmovisor/upgrades

Install the Current zetacored Binary

Find the current zetacored binary as specific in setup. Place it in ~/.zetacored/cosmovisor/genesis/bin rather than $PATH.

Configure zetacored systemd service

Create "zetachain" User Account

We recommend running ZetaChain binary files with a user account rather than as root.

useradd -m -s /bin/bash zetachain

Create ZetaChain Directory Structure

This is needed to store ZetaChain binary and configuration files.

sudo su zetachain
mkdir /home/zetachain/.zetacored/config

Configure a Systemd Unit File

Create the file /etc/systemd/system/zetacored.service with the following contents:

[Unit]
Description=zetacored (running under cosmovisor)
After=multi-user.target
StartLimitIntervalSec=0
[Install]
WantedBy=multi-user.target
[Service]
WorkingDirectory=/home/zetachain/.zetacored/cosmovisor
Environment="DAEMON_HOME=/home/zetachain/.zetacored"
Environment="DAEMON_NAME=zetacored"
Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=true"
Environment="DAEMON_RESTART_AFTER_UPGRADE=true"
Environment="DAEMON_DATA_BACKUP_DIR=/home/zetachain/.zetacored"
Environment="CLIENT_DAEMON_NAME=zetaclientd"
Environment="CLIENT_SKIP_UPGRADE=false"
Environment="CLIENT_START_PROCESS=false"
Environment="UNSAFE_SKIP_BACKUP=true"
Type=simple
Restart=always
RestartSec=600
User=zetachain
LimitNOFILE=262144
ExecStart=cosmovisor run start --home /home/zetachain/.zetacored/ --log_format json  --moniker <YOUR_NODE_NAME_HERE>

Run systemctl daemon-reload to load the service.

Run systemctl enable --now zetacored to start the service and to automatically start it at boot.

Upgrades

Follow the #testnet-announcements (opens in a new tab) and/or #mainnet-announcements (opens in a new tab) channels on discord to receive upgrade proposal notifications.

Download the new binaries from the github release and install them into the cosmovisor upgrade directory. The name of the directory must match the upgrade name. The upgrade name is not the the release name on github. Reference the Governance Upgrade Proposals page to easily see the upgrade names. Here are some examples:

After installing the binary, you should:

• validate the checksum is correct
• run zetacored version to validate that you can actually run the downloaded binary

Monitoring

In a production environment we recommend monitoring the node resources (CPU load, Memory Usage, Disk usage and Disk IO) for any performance degradation.

ZetaChain Core generates a log that can be monitored for errors and used for troubleshooting. If you install zetacored as a Systemd service using the instructions above you can view this log with journalctl -f -u zetacored.

Prometheus can be enabled to serve metrics which can be consumed by Prometheus collector(s). Telemetry include Prometheus metrics can be enabled in the app.toml file. See the Cosmos SDK Telemetry Documentation (opens in a new tab) for more information.

See more about your Validator Monitoring here.