Paloma Weekly Wings: November 29, 2023 - Upgrade Paloma to fix CometBFT mempool in Testnest , New Validator Cool-Off, and a Dynamic EventNonce in Constructor Input

Your new Paloma hatchlings this week?

Welcome to the weekly update on new software from the Paloma community of developers.

REMINDER WARNING: The Hatchling Paloma Network of “Messenger” (v1.10.1) and the Pigeon relay module is on v1.10.1. Paloma is still young, and its Cosmwasm and EVM contracts are still un-audited and undergoing continuous upgrades on protocol as well as on the smart contracts. The GRAIN token is launched but not yet trading on any exchanges. Although the flock is flying on the messenger mainnet, it is still subject to the continued development of Paloma’s cross-chain messaging system as well as to the security guarantees offered by the GRAIN token. We expect “Messenger” to experience weekly chain halts and chain restarts to upgrade. We also expect numerous undiscovered vulnerabilities. We also do not control nor influence the price of GRAINs. The community will strive to preserve mainnet state, which includes balances, but considers this mainnet high-risk. Proceed with caution.

1. Fix CometBFT Rejection of Mempool Transactions

Pigeon Operator Keys testing on Paloma testnet revealed a newly discovered issue. Transactions were being stuck in the Paloma mempool for some validators. If you recall from our announcement of Pigeon Operator Keys here: https://www.palomachain.com/blog/paloma-weekly-wings-november-15-2023-paloma-and-pigeon-upgrade-to-v1-10-0-and-launch-pigeon-operator-keys/ our goal with Pigeon Operator Keys are to allow validators to maximize message relay and attestation output by adding more signing keys to their pigeon module. While testing this new functionality, we noticed a number of transactions persisting in the CometBFT v0.37.2 mempool. We were unable to to determine why the transaction would not clear the mempool. We suspected that this was an artefact of the numerous validator keys signing, but determined it was a problem with CometBFT itself. According to ByteBandit:

After more investigation, it's become apparent that there are transactions which are being rejected in the CometBFT mempool. It is not apparent that these transactions ever reached the custom Paloma application mempool. The reason for the rejection of the transaction is the account sequence mismatch of the prior transaction.

Theoretically, this should be verified when submitting a transaction into the mempool. Looking at the code, there is a precondition in our ante handler for signature verification that's being skipped if we're running CheckTX - so it's possible for a transaction that would actually fail to still be included in the mempool, as it looks like the check is only run once the TX is actually executed.

Tested this change in on paloma-testnet-15 against a developer Paloma build and had promising results, meaning most TXs got rejected and evicted during the stage.

However, 6 transactions remain stuck in the mempool for our testnest validators. It’s still not clear why some transactions are still stuck. It's possible they're being re-broadcasted from the other nodes. This release will remove the checkTX and thus address the account sequence mismatch bug preventing validators from relaying more messages. Once all validators upgrade to the next version with this change in place, we will know if this fix was definitive.

Paloma will continue its Cosmos v0.50 upgrade path to CometBFT 0.38.2 so will test again when we are on the new release. Of course, we’re excited to bring greater validator distributed message relay power to Paloma pigeons.

2. Validator Cool-Off Feature

Validators that use unjail scripts on Paloma will now find that this release will keep those validators jailed for increasingly longer periods of time. One of the blockers to effective message relay on Paloma has been the use of unjail scripts.

These scripts immediately bring a validator back to validating and securing the blockchain, but those pigeons that don’t have correct pigeon module configuration will then receive messages they fail to deliver. Failure to deliver a relay message means that the network value decreases as each failed message contributes to a higher unreliability standard that reduces the value of the GRAIN token for all other validators that have correct and working configurations. Waiting for failed messages to clear the message queue is slow and Paloma wants to be the fastest distributed-message relay system.

Thus, Paloma v10.0.1 introduces increasing validator jail times for validators that fail to fix or address pigeon configuration issues. The jail time increases for each attempt at 30 minutes + 20% of the time of the last jailing. Thus, if a validator that runs the unjail command, the first time, and is jailed again for a pigeon-specific performance reason, then that validator will need to cool-off for 30 minutes + 6 minutes. Once the cool-off period is complete, the validator may unjail and rejoin the network via unjail script or manual restart. In Paloma today, the unjail script reduces network effectiveness and reliability. This hurts all the other pigeons in the network and GRAIN token holders. Thus, this new feature:

This change introduces a dynamic backoff that increases the given sentence to a jailed validator should they become jailed repeatedly within short periods of time.

This is done to incentivise validators to examine their setup and ensure they remain part of the active validator set, as well as reduce the amount of valset changes resulting from frequent jailing.

3. Dynamic EventNonce in Constructor Input for Compass-EVM

In order to fix failing batches on Paloma Gravity bridge, Compass-EVM was given a new nonce for tracking called the EventNonce. This nonce is distinct from the BatchNonce as it adds more security against replay attacks. The last Compass-EVM included a temporary hardcoded nonce of 100000 that replaced the constructor input. In this release, we have a new ABI that supports the event_id as constructor input. This now requires a redeployment of Compass-EVM light client on all chains and an upgrade to pigeon to use the dynamic nonce. Pigeons will expect another upgrade to Pigeon and Compass-EVM once Paloma v10.0.1 deploys after governance.

Stay up to date and follow the Paloma public repositories and the commits of ongoing upgrades to Paloma Cosmos-SDK blockchain and the Pigeon relay module, here:

https://github.com/palomachain/paloma/releases

https://github.com/palomachain/pigeon/releases

Pigeon of the Week: AI Pigeon - The Operator Pigeon

The Operator Pigeon is species of pigeon that is able to sign multiple transactions in a block due to its unique ability to host multiple signing keys in each of its feathers. Each signing key allows the Operator Pigeon to relay multiple messages within a block, up to the maximum block size. Pigeons such as these are trained to acquire the highest paying messages for relay and to deliver as quickly as possible, so that the pigeon reputation as operator will continue to reward it more GRAINs.

source: https://twitter.com/LewisTaariq/status/1729546240843526144

Conclusions:

We are excited to make validators a top-tier citizen on the Paloma network with these improvements. The Gravity Bridge and Pigeon Operator Keys highlighted a number of fixes Paloma needs to be able to execute more onchain-bot transactions. These fixes also allow Paloma network to continue to grow its volume of transactions as it brings its long-awaited Pigeon Feed features onto the Paloma-Testnest-15.

Special thanks to the Vitwit team that continues to upgrade Paloma for Cosmos v0.50.0. Looking forward to testing modules on a new v0.50.0 branch of Paloma. COO! COO!

To learn more about Paloma, please visit https://palomachain.com

To follow the project on Github, please star the project https://github.com/palomachain/paloma

To participate in the community, please join the Paloma Discord: https://discord.gg/HtUvgxvh5N