Corrosion

Gossip-based service discovery (and more) for large distributed systems.

Why we built Corrosion

We built Corrosion specifically for service discovery across a large global network, replacing Consul’s central state database with eventually consistent state distributed across our hosts.

Our new tool needed to deliver the following:

Fast reads and writes

Getting state (data) from a central remote source can be incredibly expensive (at least 300ms for a round-trip to something on the other side of the world), but usually takes less than 1ms from a local source.

Fast, eventual, consistency

Many use cases can cope with eventual consistency, especially if a consistent state is attained sooner than later. Raft fell short for some use cases at Fly.io where round-trips to a centralized location are too expensive.

Flexibility

Global state for a distributed system isn't one-size-fits-all. Flexible schemas and queries are essential.

How Corrosion works

In a nutshell, Corrosion:

• Maintains a SQLite database on each node
• Gossips local changes throughout the cluster
• UsesCR-SQLitefor conflict resolution with CRDTs
• UsesFocato manage cluster membership using a SWIM protocol
• Periodically synchronizes with a subset of other cluster nodes, to ensure consistency

Features

• A flexible API to read from and write to Corrosion's store using SQL statements
• File-based schemas with on-the-fly updates
• HTTP streaming subscriptions based on SQL queries
• Live population of configuration files from Corrosion state with user-definedRhaitemplates
• Storage and propagation of state from locally registered Consul services, replacing the central database with Corrosion's distributed state
• Secure peer-to-peer communication with theQUICtransport protocol (usingQuinn)

Usage overview

Run the Corrosion agent on every node/host in the cluster. Other programs running on the node useCorrosion's HTTP APIto query the local Corrosion SQLite database, add and update data, and subscribe to change notifications.

TheCorrosion CLIprovides commands for administration and access to database and features.

Quick start

• Prepare the Corrosion configuration file
• Specify the initial database schema
• Start the Corrosion agent

See the WIPCorrosion documentationfor more details.

Building Corrosion

Clonehttps://github /superfly/corrosion.git.

From within the repo directory:

cargo build --release && mv target/release/corrosion./