Volume AMA: The State of Paloma & A Peek at Racer
Overview
Taariq Lewis @ LewisTaariq, CEO of VolumeFi (volume.finance) offers a post-mortem on the recent network chain halt in this week’s AMA episode. Nearly three weeks after the Cosmos SDK 50 upgrade to v1.13.1, Paloma faced challenges from AppHash errors, UpgradeHandler, and consensus failure. A vote for a chain halt was passed last week and. the devs team at Paloma quickly buckled down to work and investigated issues causing the failure to restart the chain and get past pre-vote. After issues related to IAVL, commit hash, state sync, and IBC parameters were resolved, the team detected non-determinism due to version mismatch adversely impacting Paloma.
The new release, Paloma v1.13.2 ushers in a new genesis that will have fixes to the build process. Once validators upgrade to the new version and genesis, the network will begin building blocks quickly and pave the way for Racer to come online.
Squeeze, Paloma’s L2 EVM, to be launched on Arbitrum is coming! More bots are in the brew as Paloma Pigeons get ready to take flight. Sequencers and a quorum member of nodes for data availability designated from the Paloma validator set signal more opportunities for Pigeons to further monetize on L2.
Paloma continues to explore the AI space for the best fit. Plans to forge partnerships with AI tool companies are underway to open access for Paloma validators for message relays, AI tooling, and AI execution.
Definitions
AMA - Ask Me Anything
Validator - or Pigeon in Paloma-speak, a participant in a blockchain network responsible for validating new transactions and maintaining the security of the blockchain.
GRAINS - Paloma’s native token (unlisted).
L2 - or Layer 2 refers to a secondary framework or protocol built on top of an existing blockchain system. The main goal of these protocols is to scale transaction processing speed and overall throughput on the network.
Squeeze - Paloma’s L2 EVM to be launched on Arbitrum.
Intro
Taariq welcomes everyone to Volume’s Twitter Spaces. He explains that today’s episode will not be about Paloma’s Racer because the network is at a chain halt. He shares a brief background about Volume Finance.
“Volume is a software company assigned by the Paloma Foundation to build on Paloma. Our goal is decentralized trading bots, decentralized AI bots, and decentralized leverage bots managed by the Paloma validators.”
He adds that the focus is to set a path for Paloma validators to make massive profits and revenues.
Chain Halt Post-Mortem
It has been nearly three weeks since the upgrade to Cosmos SDK 50, and the team continues to face challenges as some validators are unable to get back on-chain. Initially, the Cosmos team thought it was an IAVL upgrade issue as they were seeing edge cases and UpgradeHandler errors. It was decided to do a chain halt and then proceed with the upgrade. However, efforts to restart the chain failed. Attempts to recover from state sync and snapshot also failed.
The team decided to export a new genesis where it was discovered that validators were using the wrong commit hash. After tracking validators to make sure that they are on the right commit hashes, the chain still could not restart. Golang was updated from v.1.21 to v.1.22 with validators now on the same commit hash, and yet, the chain was not producing blocks and was unable to get past pre-vote. Devs did a comparison of the exported state of validators and detected non-determinism due to a version mismatch.
“It appears that for some validators who were manually building their version number in their build is ‘1.13.1’, and not ‘v1.13.1’. So apparently the non-determinism and the different hash, are most likely coming from different versions of the build.”
The Paloma team is now going to create a v.1.13.2 that will build upgrades on the makefile so that the build version will always come with the preceding “v” even if it is built manually. The new Cosmos requires the build version number to be exact. IAVL will be updated to 10.12. There will be a new genesis for Racer that will be the required minimum value and will come with a checksum.
The New Genesis
“The core is that we're going to have a new genesis and that new genesis will have a bump. The new genesis also will have a fix.”
Taariq explains that IBC has a new allowed list called LocalHost. This allowed list was not properly migrated in the upgrade which might have contributed to the AppHash error during chain start and pre-vote.
“We're going to put back this LocalHost, which is why we need a new genesis because we're going to make sure that that is available for IBC, upgrade the LocalHost, and load list for IBC parameters.”
He adds that it appears Cosmos 50 is sensitive to these types of matches and once the adjustments are made, he is confident that Paloma will be able to make blocks again.
“We made blocks in our private testnet before. We did not have a problem with our private testnet. But in our private testnet, we did not test for a manual build environment versus a downloaded build environment.”
All Paloma private testnets are downloaded build environments. The team will now have the testnet system automatically deploy manual builds in a mixed-build environment.
“It turns out that the simplest, smallest solutions have the biggest impact, and that smallest solution might be the letter ‘v’ in the various builds.”
He thanks the validators for being patient through testing. They are active and ready to go.
“Once we have that release up on the new Genesis, we will give it to the active validators at chain halt so that Racer can come online.”
Taariq muses that while the network did not experience these challenges before the upgrade to Cosmos SDK 50, the present issues are clear-cut and workable.
“We're going to work on this. We're going to get Compass deployed. We want to get back to relaying really fast. Everybody's asking for Paloma relays. We want to get that up and running so we can get moving onward.”
Volume Update
Taariq announces that Squeeze, Paloma’s L2 EVM is coming and will be launched on Arbitrum. The team is working on more bots. Sequencers and a member of the quorum of nodes for data availability will be designated from the Paloma validator set. These opportunities, in addition to relaying messages, will allow validators to participate and further monetize on that layer.
“The sequencer will be a Paloma validator. The quorum of nodes for the data availability will be Paloma validators. Paloma validators acting as relayers will get paid. So big money opportunity here for Paloma validators.”
A new engineer will be coming on board to work on the EVM L2 to make sure it is up and running and launched quickly.
Paloma is actively exploring the best fit in the AI space. They will begin talking to AI tool companies to see where Paloma can help with automation so that validators can access relays for AI tooling and AI execution.
Closing
Taariq assures that the team will continue to prep for a genesis on Racer. States and balances have been preserved for all validators and all users of Paloma. They will ensure that all builds are consistent and that IBC features are fully allowed.
“We will be ready and cranking with blocks, if not tomorrow, by the end of the week!”
Stay tuned for the next AMA!
To find out more about Volume, check out Volume Finance (https://volume.finance/), join the Discord (https://discord.com/invite/Ebh6YjMShu), and follow us on Twitter (@Volumefi). Check out Paloma Chain on palomachain.com (https://www.palomachain.com/), follow them on Twitter (@palomachain), and join the flock on Discord (https://discord.com/invite/tNqkNHvVNc). Coo!
