> For the complete documentation index, see [llms.txt](https://canopy-network.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://canopy-network.gitbook.io/docs/secure-canopy/node-runner/manage.md).

# Manage

{% hint style="danger" %}
**Disclaimer**: Canopy is alpha-stage software. While documentation and guides are provided to assist users, they are offered without warranties, guarantees, or assurances of reliability. Users are solely responsible for any issues that may arise, including but not limited to the loss of funds. Please proceed with caution.
{% endhint %}

## Network Interfaces

#### Endpoints

Familiarize yourself with the created endpoints from the [Quickstart](/docs/secure-canopy/node-runner/quickstart.md) guide

* **Grafana:** <mark style="color:blue;"><https://monitoring.\\>\<YOUR\_DOMAIN></mark>
* **Web wallet for CNPY:** <mark style="color:blue;"><https://node1.\\>\<YOUR\_DOMAIN>/wallet</mark>
* **Block Explorer for CNPY:** <mark style="color:blue;"><https://node1.\\>\<YOUR\_DOMAIN>/explorer</mark>
* **Web wallet for CNRY:** <mark style="color:blue;"><https://node2.\\>\<YOUR\_DOMAIN>/wallet</mark>
* **Block Explorer for CNRY:** <mark style="color:blue;"><https://node2.\\>\<YOUR\_DOMAIN>/explorer</mark>

***

#### Ports

Expose the below ports via firewall

**Canopy Node Ports**

* **9001**: TCP P2P communication for node1
* **9002**: TCP P2P communication for node2

**Load Balancer Ports**

* **80**: HTTP traffic (redirects to HTTPS in production)
* **443**: HTTPS traffic (SSL/TLS)

Familiarize yourself with the internal ports

**Monitoring Ports**&#x20;

* **3000**: Grafana web interface
* **9090**: Prometheus metrics endpoint
* **3100**: Loki log aggregation
* **8082**: Traefik metrics endpoint
* **9115**: Blackbox exporter metrics
* **8080**: cAdvisor container metrics
* **9100**: Node exporter host metrics

**Canopy service ports**

* **50000**: Wallet service for node1 (exposed via Traefik)
* **50001**: Explorer service for node1 (exposed via Traefik)
* **50002**: RPC service for node1 (exposed via Traefik)
* **50003**: Admin RPC service for node1 (exposed via Traefik)
* **40000**: Wallet service for node2 (exposed via Traefik)
* **40001**: Explorer service for node2 (exposed via Traefik)
* **40002**: RPC service for node2 (exposed via Traefik)
* **40003**: Admin RPC service for node2 (exposed via Traefik)

***

## Validator Transactions

When managing your Validator, there are a few key transaction types to be aware of

* **Stake:** Register a validator for operation
* **Edit-Stake:** Edit an existing validator
* **Pause:** Temporarily remove an existing validator from active operation
* **Unpause:** Re-enlist a paused validator for operation&#x20;
* **Unstake:** Permanently remove a validator from active service

These transactions may be executed over CLI, RPC, or the built-in Web Wallet.

***

#### STAKE TRANSACTION

<figure><img src="/files/nfcZcwAOVlmJvsE2bR89" alt="" width="563"><figcaption></figcaption></figure>

**Parameters:**

* Account: Operator address - where subsequent validator transactions like (edit-stake and unstake) should be sent from.
* Delegate: Whether or not the validator will be actively operating (signing and producing blocks)
* Committees: Chain IDs where restake is allocated
* Amount: currency to stake (6 decimals)
* Withdrawal: `false` to auto-compound rewards - `true` to withdrawal the rewards automatically (comes with penalty \<see [Governance Params](/docs/canopy-network/how-does-canopy-work/specifications/governance-params.md)>)
* Net-addr: `tcp://<YOUR_DOMAIN>` P2P TCP address
* Output: The rewards address
* Signer:  Account that signs the transaction
* Memo: `optional`
* Txn-fee: Transaction fee (default is pre-filled)
* Password: Password of the signer account

**Notes:**

* Signer account is where funds are deducted from. This should be the operator address
* Delegate status may not be changed once set

***

#### ➤ EDIT-STAKE TRANSACTION

<figure><img src="/files/idbI7Q2fBLJlnVneMoyE" alt="" width="563"><figcaption></figcaption></figure>

**Parameters:**

* Account: Operator address
* Committees: Chain IDs where restake is allocated
* Amount: currency to stake (6 decimals)
* Withdrawal: `false` to auto-compound rewards - `true` to withdraw the rewards automatically (comes with penalty \<see [Governance Params](/docs/canopy-network/how-does-canopy-work/specifications/governance-params.md)>)
* Net-addr: `tcp://<YOUR_DOMAIN>` P2P TCP address
* Output: The rewards address
* Signer:  Output or operator address
* Memo: `optional`
* Txn-fee: Transaction fee (default is pre-filled)
* Password: Password of the signer account

**Notes:**

* The operator address may not be edited
* Use the 'same-value' inputs that you don't want edited
* The output address is only able to be changed by the output address
* The stake amount cannot be lowered
* Fees are deducted from the signer's address

***

#### ➤ **UNSTAKE TRANSACTION**

<figure><img src="/files/ROyu9tJRLFQ4101Qr6op" alt="" width="563"><figcaption></figcaption></figure>

**Parameters:**

* Account: Operator address
* Signer: Output or operator address

**Notes:**

* Unstaking takes 7 days for a Validator and 3 days for a delegate
* Fees are deducted from the signer's address

***

#### **PAUSE TRANSACTION**

<figure><img src="/files/FK1vwBT4FuvNegjdmNR5" alt="" width="563"><figcaption></figcaption></figure>

**Parameters:**

* Account: Operator address
* Signer: Output or operator address

**Notes:**

* A validator may be paused for 7 days before it automatically begins unstaking
* Fees are deducted from the signer's address

***

#### ➤ **UNPAUSE TRANSACTION**

<figure><img src="/files/09E3xs2VRi0ZwjHGaCdg" alt="" width="563"><figcaption></figcaption></figure>

**Parameters:**

* Account: Operator address
* Signer: Output or operator address

**Notes:**

* Validators that are unpaused are expected to immediately begin consensus operation for all chains
* If your validator was auto-paused, be sure to fix the issue with the validator before unpausing. See [#debugging-and-support](#debugging-and-support "mention")
* Fees are deducted from the signer's address

## Slashing

⚠️ Why Did My Tokens Go Down? (Understanding Slashing) TL;DR: You got slashed because your validator went offline or misbehaved. That means fewer tokens and missed rewards. Want to avoid this? Keep your validator online at all times.

#### What Causes a Slash?

Canopy slashes Validators for two main reasons:

* Inactivity – If your validator misses 60 out of 100 blocks (each \~20s) within a moving window, you're considered offline and get slashed.
* Double Signing – Signing two blocks at the same height for the same chain breaks consensus rules and results in a slash.

⚠️ Subchains may define additional slashing rules, but these two are the main ones on mainnet.

#### How Bad Is the Slash?

* Slash amount for inactivity (non-sign) is 1% of staked tokens
* Slash amount for double signing is 10% of staked tokens

A governance parameter called `MaxSlashPerCommittee` limits how much a validator may be slashed per block. Canopy caps the slashes and starts ejecting validators from the committee to prevent an escalation.

You’ll lose tokens and be kicked out of the committee temporarily — meaning no rewards, and you’ll need to regain trust.

#### How to Stay Safe

It’s all about uptime. Here’s how to avoid slashing:

* Use auto-restart tools like systemd, supervisord, or Docker’s restart policies.
* Set up monitoring with alerts to catch issues early — we already provide a full monitoring stack (Grafana + Prometheus) in the step-by-step setup guide, so make sure it’s running and you’re checking it regularly.
* Keep keys secure to avoid double-signing across different instances.
* Test failover setups carefully — don’t accidentally run two validators at once.

## ➜  Grafana + Prometheus Configuration

Basic Scrape Configuration

```yaml
scrape_configs:
  - job_name: 'canopy'
    static_configs:
      - targets: ['localhost:9090']  # Default Canopy metrics endpoint
    scrape_interval: 15s
    scrape_timeout: 10s
```

You can find a working example at `deployments/monitoring-stack/monitoring/prometheus`\
`/prometheus.yml`

Recommended Recording Rules

```yaml
groups:
  - name: canopy
    rules:
      # Node Health
      - record: canopy:node_up
        expr: canopy_node_status == 1
      
      # Peer Health
      - record: canopy:peers_total
        expr: sum(canopy_peer_total{status="connected"})
      
      # Validator Status Summary
      - record: canopy:validators_by_status
        expr: sum by (status) (canopy_validator_status)
      
      # Transaction Rates
      - record: canopy:transaction_rate
        expr: rate(canopy_transaction_received[5m]) + rate(canopy_transaction_sent[5m])
```

### Metric Types and Usage

#### Gauges

Gauges represent current values that can go up or down:

* Node status
* Peer counts
* Block height
* Memory usage
* Validator status

#### Counters

Counters represent monotonically increasing values:

* Transaction counts
* Block processing counts

#### Histograms

Histograms track the distribution of values:

* Block processing time

### Recommended Alerts

```yaml
groups:
  - name: canopy
    rules:
      # Node Health
      - alert: CanopyNodeDown
        expr: canopy_node_status == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Canopy node is down"
          description: "Node has been down for more than 5 minutes"

      # Sync Status
      - alert: CanopyNodeNotSynced
        expr: canopy_node_syncing_status == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Canopy node is not synced"
          description: "Node has been out of sync for more than 15 minutes"

      # Peer Health
      - alert: CanopyLowPeerCount
        expr: sum(canopy_peer_total{status="connected"}) < 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low peer count"
          description: "Node has fewer than 3 connected peers"

      # Performance
      - alert: CanopyHighBlockProcessingTime
        expr: rate(canopy_block_processing_seconds_sum[5m]) / rate(canopy_block_processing_seconds_count[5m]) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High block processing time"
          description: "Average block processing time is above 1 second"
```

### Grafana Dashboard Recommendations

You can find the example of the default grafana dashboard implementation at `deployments/monitoring-stack/monitoring/grafana/dashboards/canopy`\
`/canopy_dashboard.json`

#### Key Panels to Include

1. Node Status
   * Node up/down status
   * Sync status
   * Uptime
2. Peer Network
   * Total peers
   * Inbound/outbound peers
   * Peer connection status
3. Validator Status
   * Validator count by status
   * Validator types
   * Staking status
4. Transaction Metrics
   * Transaction rate
   * Transaction volume
   * Transaction types
5. Performance
   * Block processing time
   * Memory usage
   * CPU usage

#### Example Queries

```promql
# Node Health
canopy_node_status

# Peer Network Health
sum(canopy_peer_total{status="connected"})

# Validator Status Distribution
sum by (status) (canopy_validator_status)

# Transaction Rate
rate(canopy_transaction_received[5m]) + rate(canopy_transaction_sent[5m])

# Block Processing Performance
rate(canopy_block_processing_seconds_sum[5m]) / rate(canopy_block_processing_seconds_count[5m])
```

## Additional Nested Chains

* Canopy supports running multiple chains (like CNPY and its first subchain, CNRY) on the same machine.&#x20;
  * To set this up, simply follow the step-by-step guide and run the setup.sh script.&#x20;
  * This script automatically:&#x20;
    * Configures both CNPY and CNRY&#x20;
    * Reuses the same validator\_key and keystore for both&#x20;
    * Sets up each chain with the correct ports and configuration so they can run side by side
* How The Script Works Behind the scenes, each node (node1, node2, etc.) is configured with:&#x20;
  * \- A unique `chainId` (`1` for CNPY, `2` for CNRY)&#x20;
  * \- A different `listenAddress` (e.g. 0.0.0.0:9001 for CNPY and 0.0.0.0:9002 for CNRY)&#x20;
  * No manual config needed, this is all handled automatically&#x20;
  * This ensures there are no port conflicts, and both chains can run simultaneously using a shared validator identity.

If you plan to add more nested chains in the future, follow the same pattern: - Assign a unique `chainId` to each chain - Ensure each node uses a different `listenAddress`

## Debugging and Support

* Most logs tagged <mark style="color:red;">**ERROR**</mark> should be reported, although some `P2P`  ERROR logs are expected as peers churn (especially upon startup).
* If you're encountering repeated alerts or facing issues while running your node, don't hesitate to reach out.

{% hint style="info" %}
Open an issue on our GitHub repository: <https://github.com/canopy-network/canopy/issues>
{% endhint %}

{% hint style="info" %}
For comms and support, join our official Discord: <https://discord.com/invite/WdeBG8jcuB>
{% endhint %}

## Community Snapshot and Public RPC Endpoint

This [link](https://faucet.nodefleet.org/) is a community-maintained endpoint providing **RPC** access and **blockchain snapshots** to support fast and easy bootstrapping of Canopy. It is offered as a public resource in the spirit of community goodwill.

## Validator Migration Guide

**Complete server migration with minimal downtime (5-10 minutes)**

### Prerequisites

* Access to both OLD and NEW servers
* Domain DNS management access (Cloudflare recommended)
* New server with Ubuntu 22.04+ and root access

***

### Phase 0: Pause Validator

*Be sure to pause your existing Validator (if applicable) before beginning the migration process to ensure no slashing risk due to downtime, peering issues, or duplicate identities.*

### Phase 1: Prepare New Server

#### 1. Install Dependencies on New Server

```bash
# Update system
apt update && apt upgrade -y

# Install required packages
apt install -y ca-certificates curl gnupg lsb-release apache2-utils make git tree

# Create Docker keyring directory
mkdir -p /etc/apt/keyrings

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) \
  signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine + Compose
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Install Loki Docker driver for log aggregation
docker plugin install grafana/loki-docker-driver --alias loki

# Configure firewall
ufw allow 22/tcp && ufw allow 9001/tcp && ufw allow 9002/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw --force enable
```

#### 2. Update DNS Records (CRITICAL - Do this first!)

**In your DNS provider (Cloudflare recommended):**

```
YOUR_DOMAIN.com          A record -> NEW_SERVER_IP
*.YOUR_DOMAIN.com        A record -> NEW_SERVER_IP
```

#### 3. Verify DNS Propagation

```bash
# Check DNS propagation from multiple servers
dig @8.8.8.8 YOUR_DOMAIN.com        # Google DNS
dig @1.1.1.1 YOUR_DOMAIN.com        # Cloudflare DNS
dig @208.67.222.222 YOUR_DOMAIN.com # OpenDNS

# All should return NEW_SERVER_IP
```

***

### Phase 2: Migration (5-10 minutes downtime)

#### 4. Stop Validator on Old Server

```bash
# Navigate to monitoring stack
cd ~/deployments/monitoring-stack

# Stop all services
sudo make down

# Verify everything is stopped
sudo make ps
```

#### 5. Create Backup on Old Server

```bash
# Navigate to home directory
cd ~/

# Create complete backup (excludes large blockchain data for speed)
tar --exclude='*/canopy/*.vlog' \
    --exclude='*/canopy/*.sst' \
    --exclude='*/logs/*' \
    --exclude='*/prometheus/data/*' \
    --exclude='*/grafana/data/grafana.db' \
    -czf complete-validator.tar.gz deployments/

# Check backup size (should be much smaller without logs/data)
du -sh complete-validator.tar.gz
```

#### 6. Transfer to New Server

```bash
# Transfer backup to new server
scp complete-validator.tar.gz root@NEW_SERVER_IP:~/

# Alternative: Use rsync for better progress tracking
# rsync -avz --progress complete-validator.tar.gz root@NEW_SERVER_IP:~/
```

#### 7. Deploy on New Server

```bash
# Extract backup
cd ~/
tar -xzf complete-validator.tar.gz

# Fix permissions (CRITICAL STEP)
chmod +x ~/deployments/docker_image/entrypoint.sh
find ~/deployments/ -name "*.sh" -exec chmod +x {} \;
find ~/deployments/ -name "cli" -exec chmod +x {} \;

# Navigate to monitoring stack
cd ~/deployments/monitoring-stack

# Start with fresh snapshots (downloads latest blockchain data)
sudo make start_with_snapshot
```

#### 8. Verify Deployment

```bash
# Check all containers are running
sudo make ps

# Check logs for any errors
sudo make logs | head -30

# Test web interfaces
curl -k https://YOUR_DOMAIN.com
curl -k https://monitoring.YOUR_DOMAIN.com

# Check validator status
docker logs canopy-validator-node1-1 | tail -20
docker logs canopy-validator-node2-1 | tail -20
```

***

### Phase 3: Post-Migration Verification

#### 9. Access Web Interfaces

Once services are running, verify these endpoints:

* **Grafana Dashboard**: `https://monitoring.YOUR_DOMAIN.com`
* **Node 1 Wallet**: `https://node1.YOUR_DOMAIN.com/wallet`
* **Node 1 Explorer**: `https://node1.YOUR_DOMAIN.com/explorer`
* **Node 2 Wallet**: `https://node2.YOUR_DOMAIN.com/wallet`
* **Node 2 Explorer**: `https://node2.YOUR_DOMAIN.com/explorer`

#### 10. Monitor Sync Status

```bash
# Watch sync progress in logs
sudo make logs -f

# Check validator status in web wallet
# Navigate to wallet interfaces and check "monitoring" tab
```

#### 11. Verify Staking Status

* Log into your web wallet interfaces
* Check that your validator status shows as "STAKED"
* Monitor for 10-15 minutes to ensure stability

***

### Troubleshooting

#### Common Issues and Solutions

**Permission Denied Errors:**

```bash
# Fix all executable permissions
find ~/deployments/ -type f -name "*.sh" -exec chmod +x {} \;
chmod +x ~/deployments/docker_image/entrypoint.sh
```

**DNS Not Resolving:**

```bash
# Check DNS propagation
nslookup YOUR_DOMAIN.com
# Wait for DNS to propagate (1 hour max)
```

**SSL Certificate Issues:**

```bash
# Check Traefik logs
docker logs traefik-1

# SSL certificates auto-renew via Let's Encrypt
# May take 2-3 minutes after DNS propagation
```

**Containers Not Starting:**

```bash
# Check detailed logs
sudo make logs SERVICE_NAME

# Common fixes:
sudo make down
sudo make start_with_snapshot
```

***

### Important Notes

#### Critical Points

* **Pause:** Always pause your validator before migration
* **DNS First**: Always update DNS records before migration
* **Permissions**: Always run chmod commands after extracting backup
* **Snapshots**: Use `start_with_snapshot` for fastest sync
* **Monitoring**: Watch logs for first 10-15 minutes after migration

#### Expected Timeline

* **DNS Propagation**: 1-2 minutes (Cloudflare)
* **Service Stop**: 30 seconds
* **File Transfer**: 2-4 minutes (depending on connection)
* **Service Start**: 2-3 minutes
* **SSL Certificate**: 1-2 minutes
* **Total Downtime**: 5-10 minutes

#### Post-Migration

* Monitor validator status for 24 hours
* Backup the new server setup
* Update any monitoring alerts with new IP
* Test all web interfaces thoroughly

***

### Recovery Plan

If migration fails, quickly restore old server:

```bash
# On OLD server
cd ~/deployments/monitoring-stack
sudo make start_with_snapshot

# Revert DNS records to OLD_SERVER_IP
```

Your validator will resume from where it left off with minimal impact.

***

**Migration Complete!** 🎉

Your Canopy validator is now running on the new server with fresh blockchain data and preserved validator identity.