> For the complete documentation index, see [llms.txt](https://canopy-network.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://canopy-network.gitbook.io/docs/secure-canopy/node-runner/manage.md).

# Manage

This section details the management of Canopy nodes.

## Network Interfaces

#### Endpoints

Familiarize yourself with the created endpoints from the [Quickstart](/docs/secure-canopy/node-runner/quickstart.md) guide

* **Grafana:** <mark style="color:blue;"><https://monitoring.\\>\<YOUR\_DOMAIN></mark>
* **Web wallet for CNPY:** <mark style="color:blue;"><https://wallet.\\>\<YOUR\_DOMAIN></mark>
* **Block Explorer for CNPY:** <mark style="color:blue;"><https://explorer.\\>\<YOUR\_DOMAIN></mark>

***

#### Ports

Expose the below ports via firewall

**Canopy Node Port**

* **9001**: TCP P2P communication for the node

**Load Balancer Ports:** These Only apply when the monitoring stack is used

* **80**: HTTP traffic (redirects to HTTPS in production)
* **443**: HTTPS traffic (SSL/TLS)

Familiarize yourself with the internal ports. These are not exposed outside of docker but could be useful for debugging and troubleshooting

**Monitoring Ports:** These only apply when the monitoring stack is used

* **3000**: Grafana web interface
* **9090**: Prometheus metrics endpoint
* **9093:** AlertManager endpoint
* **8080:** cAdvisor metrics
* **3100**: Loki log aggregation

**Canopy service ports**

* **50000**: Wallet service for the node (exposed via Caddy)
* **50001**: Explorer service for the node (exposed via Caddy)
* **50002**: RPC service for the node (exposed via Caddy)
* **50003**: Admin RPC service for the node (exposed via Caddy)

***

## Validator Transactions

When managing your Validator, there are a few key transaction types to be aware of

* **Stake:** Register a validator for operation
* **Edit-Stake:** Edit an existing validator
* **Pause:** Temporarily remove an existing validator from active operation
* **Unpause:** Re-enlist a paused validator for operation
* **Unstake:** Permanently remove a validator from active service

These transactions may be executed over CLI, RPC, or the built-in Web Wallet.

***

#### STAKE TRANSACTION

<figure><img src="https://2343293880-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLrZDX2RTu3ClIHkvyP7T%2Fuploads%2Fou1LBvIBKAqhlSIRsWwf%2Fimage.png?alt=media&amp;token=ac6e91e0-af36-4f7c-8451-78b6ba4bb800" alt=""><figcaption></figcaption></figure>

**Stake form fields:**

* **Staking (Operator) address**: where subsequent validator transactions like (`edit-stake` and `unstake`) should be sent from.
* **Rewards Address**: Where block rewards should go to, can be the same as operator address
* **Amount**: CNPY to stake
* **Stake Type (Validation or Delegation)**: Whether or not the validator will be actively operating (signing and producing blocks)
* **AutoCompound**: To Withdraw the rewards automatically (comes with penalty \<see [Broken mention](broken://pages/qofG7hAMu4L3E3nLtCye)>)
* **Committees**: Comma-separated chain ids where the stake is allocated
* **Validator Address**: `tcp://<YOUR_DOMAIN>` P2P TCP address for other validators to connect to
* **Transaction Fee**: The transaction fee (default is pre-filled)
* **Password**: Password of the operator account

**Notes:**

* Operator account is where funds are deducted from
* Delegate status may not be changed once set

***

#### ➤ EDIT-STAKE TRANSACTION

<figure><img src="https://2343293880-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLrZDX2RTu3ClIHkvyP7T%2Fuploads%2F9zCIfyh70DatpUFsacsO%2Fimage.png?alt=media&amp;token=435d617b-7933-4c2b-bcc6-3e39f0de1351" alt=""><figcaption></figcaption></figure>

**Parameters:**

* **Staking (Operator) address**: where subsequent validator transactions like (`edit-stake` and `unstake`) should be sent from.
* **Rewards Address**: Where block rewards should go to, can be the same as operator address
* **Amount**: CNPY to stake, must be greater than the current stake.
* **Stake Type (Validation or Delegation)**: Whether or not the validator will be actively operating (signing and producing blocks)
* **AutoCompound**: To Withdraw the rewards automatically (comes with penalty \<see [Broken mention](broken://pages/qofG7hAMu4L3E3nLtCye)>)
* **Committees**: Comma-separated chain ids where the stake is allocated
* **Validator Address**: `tcp://<YOUR_DOMAIN>` P2P TCP address for other validators to connect to
* **Transaction Fee**: The transaction fee (default is pre-filled)
* **Password**: Password of the operator account

**Notes:**

* The operator address may not be edited
* Use the 'same-value' inputs that you don't want edited
* The output address is only able to be changed by the output address
* The stake amount cannot be lowered
* Fees are deducted from the signer's address

***

#### ➤ **UNSTAKE TRANSACTION**

<figure><img src="https://2343293880-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLrZDX2RTu3ClIHkvyP7T%2Fuploads%2F2ZIgnsymr6eZtQ7c2Klz%2Fimage.png?alt=media&amp;token=371fd901-756e-4f99-97fc-10c65c498f17" alt=""><figcaption></figcaption></figure>

**Parameters:**

* **Validator (Operator) address**: Address of the validator currently staked.
* **Signer Address**: The address that signs the transaction, could be the same as the validator address.
* **Memo**: Optional message note for the transaction.

**Notes:**

* Unstaking takes 1 month for a Validator and 7 days for a delegate
* Fees are deducted from the signer's address

***

#### **PAUSE TRANSACTION**

<figure><img src="https://2343293880-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLrZDX2RTu3ClIHkvyP7T%2Fuploads%2FjT8jWU34hP4rCSJHs6yF%2Fimage.png?alt=media&amp;token=faadf898-68e2-4281-8028-80b1e6df6c64" alt=""><figcaption></figcaption></figure>

**Parameters:**

* **Validator (Operator) address**: Address of the validator currently staked.
* **Signer Address**: The address that signs the transaction, could be the same as the validator address.
* **Memo**: Optional message note for the transaction.

**Notes:**

* A validator / delegate may be paused for 1 month before it automatically begins unstaking
* Fees are deducted from the signer's address

***

#### ➤ **UNPAUSE TRANSACTION**

<figure><img src="https://2343293880-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLrZDX2RTu3ClIHkvyP7T%2Fuploads%2FPTZUzv6DGYIkZ1vyT1YW%2Fimage.png?alt=media&amp;token=2d591453-a93d-458e-93bd-2fb19017e15c" alt=""><figcaption></figcaption></figure>

**Parameters:**

* **Validator (Operator) address**: Address of the validator currently staked.
* **Signer Address**: The address that signs the transaction, could be the same as the validator address.
* **Memo**: Optional message note for the transaction.

**Notes:**

* Validators that are unpaused are expected to immediately begin consensus operation for all chains
* If your validator was auto-paused, be sure to fix the issue with the validator before unpausing. See [#debugging-and-support](#debugging-and-support "mention")
* Fees are deducted from the signer's address

## Slashing

⚠️ Why Did My Tokens Go Down? (Understanding Slashing) TL;DR: You got slashed because your validator went offline or misbehaved. That means fewer tokens and missed rewards. Want to avoid this? Keep your validator online at all times.

#### What Causes a Slash?

Canopy slashes Validators for two main reasons:

* Inactivity – If your validator misses 2,160 out of 4,320 blocks (each \~20s) within a moving window, you're considered offline and get slashed.
* Double Signing – Signing two blocks at the same height for the same chain breaks consensus rules and results in a slash.

⚠️ Subchains may define additional slashing rules, but these two are the main ones on mainnet.

#### How Bad Is the Slash?

* Slash amount for inactivity (non-sign) is 1% of staked tokens
* Slash amount for double signing is 10% of staked tokens

A governance parameter called `MaxSlashPerCommittee` limits how much a validator may be slashed per block. Canopy caps the slashes and starts ejecting validators from the committee to prevent an escalation.

You’ll lose tokens and be kicked out of the committee temporarily — meaning no rewards, and you’ll need to regain trust.

#### How to Stay Safe

It’s all about uptime. Here’s how to avoid slashing:

* Use auto-restart tools like systemd, supervisord, or Docker’s restart policies.
* Set up monitoring with alerts to catch issues early — we already provide a full monitoring stack (Grafana + Prometheus) in the step-by-step setup guide, so make sure it’s running and you’re checking it regularly.
* Keep keys secure to avoid double-signing across different instances.
* Test failover setups carefully — don’t accidentally run two validators at once.

## ➜ Grafana + Prometheus Configuration

Basic Scrape Configuration. The monitoring stack already comes with Grafana + Prometheus preconfigured

```yaml
scrape_configs:
  - job_name: 'canopy'
    static_configs:
      - targets: ['localhost:9090']  # Default Canopy metrics endpoint
    scrape_interval: 15s
    scrape_timeout: 10s
```

Recommended Recording Rules

```yaml
groups:
  - name: canopy
    rules:
      # Node Health
      - record: canopy:node_up
        expr: canopy_node_status == 1
      
      # Peer Health
      - record: canopy:peers_total
        expr: sum(canopy_peer_total{status="connected"})
      
      # Validator Status Summary
      - record: canopy:validators_by_status
        expr: sum by (status) (canopy_validator_status)
      
      # Transaction Rates
      - record: canopy:transaction_rate
        expr: rate(canopy_transaction_received[5m]) + rate(canopy_transaction_sent[5m])
```

### Metric Types and Usage

#### Gauges

Gauges represent current values that can go up or down:

* Node status
* Peer counts
* Block height
* Memory usage
* Validator status

#### Counters

Counters represent monotonically increasing values:

* Transaction counts
* Block processing counts

#### Histograms

Histograms track the distribution of values:

* Block processing time

### Recommended Alerts

The recommended alerts are already bundled on the monitoring stack and can be viewed on the [node repository](https://github.com/canopy-network/node/blob/main/monitoring/prometheus/alerts.yml).

### Grafana Dashboard Recommendations

You can find the default grafana dashboard implementation bundled on the [node repository](https://github.com/canopy-network/node/blob/main/monitoring/grafana/dashboards/node_monitoring.json).

#### Key Panels to Include

1. Node Status
   * Node active status
   * Sync status
   * Chain/Root Chain ID
2. Peer Network
   * Total peers
   * Inbound/outbound peers
   * Peer connection status
3. Validator Status
   * Validator count by status
   * Validator types
   * Staking status
4. Transaction Metrics
   * Transaction rate
   * Transaction volume
5. Performance
   * Block processing time
   * Memory usage
   * CPU usage
   * Network I/O

#### Example Queries

```promql
# Node Health
canopy_node_status

# Peer Network Health
sum(canopy_peer_total{status="connected"})

# Validator Status Distribution
sum by (status) (canopy_validator_status)

# Transaction Rate
rate(canopy_transaction_received[5m]) + rate(canopy_transaction_sent[5m])

# Block Processing Performance
rate(canopy_block_processing_seconds_sum[5m]) / rate(canopy_block_processing_seconds_count[5m])
```

## Debugging and Support

* Most logs tagged <mark style="color:red;">**ERROR**</mark> should be reported, although some `P2P` ERROR logs are expected as peers churn (especially upon startup).
* If you're encountering repeated alerts or facing issues while running your node, don't hesitate to reach out.

{% hint style="info" %}
Open an issue on our GitHub repository: <https://github.com/canopy-network/canopy/issues>
{% endhint %}

{% hint style="info" %}
For comms and support, join our official Discord: <https://discord.com/invite/WdeBG8jcuB>
{% endhint %}

## Community Snapshot and Public RPC Endpoint

This [link](https://faucet.nodefleet.org/) is a community-maintained endpoint providing **RPC** access and **blockchain snapshots** to support fast and easy bootstrapping of Canopy. It is offered as a public resource in the spirit of community goodwill.

## Validator Migration Guide

**Complete server migration with minimal downtime (5-10 minutes)**

### Prerequisites

* Access to both OLD and NEW servers
* Domain DNS management access (Cloudflare recommended)
* New server with Ubuntu 26.04+
* A server user account with access to `docker` commands

### Phase 0: Pause Validator

*Be sure to pause your existing Validator (if applicable) before beginning the migration process to ensure no slashing risk due to downtime, peering issues, or duplicate identities.*

### Phase 1: Prepare New Server

#### 1. Update DNS Records (CRITICAL - Do this first!)

**In your DNS provider (Cloudflare recommended):**

```
YOUR_DOMAIN.com          A record -> NEW_SERVER_IP
*.YOUR_DOMAIN.com        A record -> NEW_SERVER_IP
```

#### 2. Verify DNS Propagation

```bash
# Check DNS propagation from multiple servers
dig @8.8.8.8 YOUR_DOMAIN.com        # Google DNS
dig @1.1.1.1 YOUR_DOMAIN.com        # Cloudflare DNS
dig @208.67.222.222 YOUR_DOMAIN.com # OpenDNS

# All should return NEW_SERVER_IP
```

#### 3. Follow the quickstart guide to prepare the new node

Follow steps [1 through 10 of the QuickStart guide](/docs/secure-canopy/node-runner/quickstart.md) and the migration step to prepare the new node

### Phase 2: Migration (5-10 minutes downtime)

#### 4. Stop Validator on Old Server

```bash
# Navigate to the node
cd ~/node

# Stop all services
make down

# Verify everything is stopped
make status
```

#### 5. Create Backup on Old Server

```bash
# Navigate to node directory
cd ~/node

# Create backup of critical data
tar -czvf validator_config_backup.tar.gz config/keystore.json config/validator_key.json .env

# Check backup files
tar -tzvf validator_config_backup.tar.gz
```

#### 6. Transfer to New Server

```bash
# Transfer backup to new server
scp validator_config_backup.tar.gz root@NEW_SERVER_IP:~/node

# Alternative: Use rsync for better progress tracking
rsync -avz --progress validator_config_backup.tar.gz root@NEW_SERVER_IP:~/node
```

#### 7. Deploy on New Server

```bash
# Extract backup
cd ~/node
tar -xzf validator_config_backup.tar.gz

# Start with fresh snapshots (downloads latest blockchain data)
make snapshot-up
```

#### 8. Verify Deployment

```bash
# Check all containers are running
make ps

# Check logs for any errors
make logs | head -30

# Test web interfaces (Only if monitoring stack is setup)
curl -k https://wallet.YOUR_DOMAIN.com
curl -k https://monitoring.YOUR_DOMAIN.com

# (Validator Only) Check if node is reachable online
nc -vz YOUR_DOMAIN.com 9001
```

### Phase 3: Post-Migration Verification

#### 9. Access Web Interfaces

Once services are running, verify these endpoints:

* **Grafana Dashboard**: `https://monitoring.YOUR_DOMAIN.com`
* **Node 1 Wallet**: `https://wallet.YOUR_DOMAIN.com`
* **Node 1 Explorer**: `https://explorer.YOUR_DOMAIN.com`

#### 10. Monitor Sync Status

```bash
# Watch sync progress in logs
make logs

# Check validator status in web wallet
# Navigate to wallet interfaces and check "Monitoring" tab
```

#### 11. Verify Staking Status

* Log into your web wallet interfaces
* Check that your validator status shows as "STAKED"
* Monitor for 10-15 minutes to ensure stability

### Troubleshooting

#### Common Issues and Solutions

**DNS Not Resolving:**

```bash
# Check DNS propagation
nslookup YOUR_DOMAIN.com
# Wait for DNS to propagate (1 hour max)
```

**SSL Certificate Issues:**

```bash
# Check Caddy logs
docker compose -f docker-compose.monitoring.yml logs caddy

# SSL certificates auto-renew via Let's Encrypt
# May take 2-3 minutes after DNS propagation
```

**Containers Not Starting:**

```bash
# Check detailed logs
docker compose logs -f docker-compose.monitoring.yml  SERVICE_NAME

# Common fixes:
make down
make snapshot-up
```

### Important Notes

#### Critical Points

* **Pause:** Always pause your validator before migration
* **DNS First**: Always update DNS records before migration
* **Permissions**: Always run chmod commands after extracting backup
* **Snapshots**: Use `` `snapshot-up` `` for fastest sync
* **Monitoring**: Watch logs for first 10-15 minutes after migration

#### Expected Timeline

* **DNS Propagation**: 1-2 minutes (Cloudflare)
* **Service Stop**: 30 seconds
* **File Transfer**: 2-4 minutes (depending on connection)
* **Service Start**: 2-3 minutes
* **SSL Certificate**: 1-2 minutes
* **Total Downtime**: 5-10 minutes

#### Post-Migration

* Monitor validator status for 24 hours
* Backup the new server setup
* Update any monitoring alerts with new IP
* Test all web interfaces thoroughly

***

### Recovery Plan

If migration fails, quickly restore old server:

```bash
# On OLD server
cd ~/deployments/node
make snapshot-up

# Revert DNS records to OLD_SERVER_IP
```

Your validator will resume from where it left off with minimal impact.

***

**Migration Complete!** 🎉

Your Canopy validator is now running on the new server with fresh blockchain data and preserved validator identity.