Skip to content

docs: remove Clay's deleted Ford cache and correct the Arvo state #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sigilante wants to merge 1 commit into
base: master
Choose a base branch
from
+112 −77
Open

docs: remove Clay's deleted Ford cache and correct the Arvo state #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 70 additions & 23 deletions content/urbit-os/kernel/arvo/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ Which perspective is more fruitful depends on the problem being considered.

We consider Arvo to be deterministic at a high level. By that we mean that it is stacked on top of a frozen instruction set known as Nock. Frozen instruction sets are a new idea for an operating system, but not for computing in general. For instance, the CPU instruction sets such as [x86-64](https://en.wikipedia.org/wiki/X86-64) are frozen at the level of the chip. A given operating system may be adapted to run on more than one CPU instruction set, we merely freeze the instruction set at a higher level in order to enable deterministic computation.

Arvo handles nondeterminism in an interesting way. Deciding whether or not to halt a computation that could potentially last forever becomes a heuristic decision that is akin to dropping a packet. Thus it behooves one to think of Arvo as being a stateful packet transceiver rather than an ordinary computer: events are never guaranteed to complete, even if one can prove that the computation would eventually terminate. We elaborate on this in the [solid state interpreter](#solid-state-intrepeter) section.
Arvo handles nondeterminism in an interesting way. Deciding whether or not to halt a computation that could potentially last forever becomes a heuristic decision that is akin to dropping a packet. Thus it behooves one to think of Arvo as being a stateful packet transceiver rather than an ordinary computer: events are never guaranteed to complete, even if one can prove that the computation would eventually terminate. We elaborate on this in the [solid state interpreter](#solid-state-interpreter) section.

Because Arvo is run on a virtual machine, nondeterministic information such as the stack trace of an infinite loop that was entered into may be obtained. This is possible because while Arvo may be unable to obtain that information, the runtime may inject that information into the event log.

Being deterministic at a high level enables many things that are out of reach of any other operating system. For instance, we are able to do [over-the-air](#over-the-air-updates) (OTA) updates, which allows software updates to be implemented across the network without needing to worry whether it won't work on someone's ship. Since Arvo is an [interpreter](#solid-state-intrepeter), it can accept source code with which to update itself instead of requiring a pre-compiled binary. This essential property makes Urbit much simpler and more accessible than any comparable personal server setup.
Being deterministic at a high level enables many things that are out of reach of any other operating system. For instance, we are able to do [over-the-air](#over-the-air-updates) (OTA) updates, which allows software updates to be implemented across the network without needing to worry whether it won't work on someone's ship. Since Arvo is an [interpreter](#solid-state-interpreter), it can accept source code with which to update itself instead of requiring a pre-compiled binary. This essential property makes Urbit much simpler and more accessible than any comparable personal server setup.

#### Event log {#event-log}

Expand Down Expand Up @@ -332,45 +332,92 @@ As we follow functional programming paradigms, the state of Arvo is considered t
Thus besides the battery of the Arvo core, we have the payload which is as follows.

```hoon
:: persistent arvo state
:: cached reflexives
::
=/ pit=vase !>(..is) ::
=/ vil=vile (viol p.pit) :: cached reflexives
=| $: lac=_& :: laconic bit
eny=@ :: entropy
our=ship :: identity
bud=vase :: %zuse
vanes=(list [label=@tas =vane]) :: modules
== ::
=/ pit=vase !>(..part)
=/ vil=vile (viol p.pit)
::
:: arvo state, as a discriminable sample
::
=| [_arvo soul]
=* sol ->
```

Let's investigate the state piece by piece.

```hoon
=/ pit=vase !>(..is) ::
=/ pit=vase !>(..part)
```

This `$vase` is part of the state but does not get directly migrated when `+poke` is called. `!>(..is)` consists of the code in `arvo.hoon` written above this core contained in a `$vase`. Thus this part of the state changes only when that code changes in an update.
This `$vase` is part of the state but does not get directly migrated when `+poke` is called. `!>(..part)` consists of the code in `arvo.hoon` written above this core contained in a `$vase`. Thus this part of the state changes only when that code changes in an update.

```hoon
=/ vil=vile (viol p.pit) :: cached reflexives

=/ vil=vile (viol p.pit)
```

This is a cache of specific types that are of fundamental importance to Arvo - namely `$type`s, `$duct`s, `$path`s, and `$vase`s. This is kept because it is unnecessarily wasteful to recompile these fundamental types on a regular basis. Again, this part of the state is never updated directly by `+poke`.

The real state is a `$soul`:

```hoon
+$ soul
$: mien
$= fad
$: lac=?
==
$= zen
$: ver=vere
lag=_|
==
$= mod
$: fat=(axal (cask))
lul=vase
zus=vase
van=(map term vane)
==
==
```

where `+$ mien [our=ship now=@da eny=@uvJ]`.

- `mien` is identity, time and entropy: `our` is the ship, permanently frozen during the larval stage; `now` is the current event time; `eny` is entropy.
- `fad` is configuration. `lac` determines whether Arvo's output is verbose, which can be set using the `|verb` command in the dojo.
- `zen` is knowledge of the Outside: `ver` is the runtime version, and `lag` records whether an upgrade is blocked.
- `mod` is the internal modules: `fat` is the kernel filesystem, `lul` is `%lull`, `zus` is `%zuse`, and `van` is the vanes.

Note that `van` is a `(map term vane)` — vanes are keyed by name, not held in a list — and a `+$ vane` is `[=vase =worm]`.

As you can see, the state of Arvo itself is quite simple. Its primary role is that of a traffic cop, and most of the interesting part of the state lies in the vanes.

#### Larval state and upgrades {#larval-state-and-upgrades}

During the [larval stage](#larval-stage-core), before Arvo knows its identity, the state is instead a `$grub`, whose fields are units filled in as they are learned:

```hoon
=| $: lac=_& :: laconic bit
eny=@ :: entropy
our=ship :: identity
bud=vase :: %zuse
vanes=(list [label=@tas =vane]) :: modules
==
+$ grub
$: who=(unit ship)
eny=(unit @)
lac=?
ver=(unit vere)
fat=(unit (axal (cask)))
lul=(unit (trap vase))
zus=(unit (trap vase))
van=(map term (trap vase))
==
```

This is where the real state of the Arvo kernel is kept. `lac` detemines whether Arvo's output is verbose, which can be set using the `|verb` command in the dojo. `eny` is the current entropy. `our` is the ship, which is permanently frozen during the larval stage. `bud` is the standard library. Lastly, `vanes` is of course the list of vanes, which have their own internal states.
An upgrade arrives wrapped in a `$heir`, which is the envelope Arvo accepts when
loading a previous state:

```hoon
+$ heir
$% [%grub %234 =grub]
[?(%240 %239 %238 %237 %236 %235) =debt =soul:a235]
[%234 =debt =soul]
==
```

As you can see, the state of Arvo itself is quite simple. Its primary role is that of a traffic cop, and most of the interesting part of the state lies in `vanes`.
The current Arvo kelvin is `%234`.

### Vanes {#vanes}

Expand Down
71 changes: 34 additions & 37 deletions content/urbit-os/kernel/clay/data-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,6 @@ Formal state
$: rom=room :: domestic
hoy=(map ship rung) :: foreign
ran=rang :: hashes
fad=flow :: ford cache
mon=(map term beam) :: mount points
hez=(unit duct) :: sync duct
cez=(map @ta crew) :: permission groups
Expand All @@ -121,7 +120,6 @@ This is the state of the vane. Anything that must be remembered between calls to
- `rom`: the state for all local desks. It consists of a `$duct` to [Dill](../dill/README.md) and a collection of `$desk`s.
- `hoy`: the state for all foreign desks.
- `ran`: the global, hash-addressed object store. It has maps of commit hashes to commits and content hashes to content.
- `fad`: the global build cache. Each desk has its own fast-lookup index over this global cache.
- `mon`: a collection of Unix mount points. `$term` is the mount point (relative to th pier) and `$beam` is a domestic Clay directory.
- `hez`: the duct used to sync with Unix.
- `cez`: a collection of named aermission groups.
Expand Down Expand Up @@ -480,7 +478,6 @@ Desk data
tom=(map tako norm)
nor=norm
mim=(map path mime)
fod=flue
wic=(map weft yoki)
liv=zest
ren=rein
Expand All @@ -495,7 +492,6 @@ A `$dome` is the state of a `$desk` and associated data.
- `tom` contains the tombstoning policies for all files in the desk.
- `nor` is the default tombstoning policy.
- `mim` is a cache of the content in the directories that are mounted to Unix.
- `fod` is the Ford cache, which keeps a cache of the results of builds performed at this `$desk`'s current revision, including a full transitive closure of dependencies for each completed build.
- `wic` contains commits waiting for future kernel versions.
- `liv` says whether agents on the desk are running or suspended.
- `ren` records which agents have been forced on or off, differing from the desk's `desk.bill` manifest.
Expand Down Expand Up @@ -770,7 +766,13 @@ Ford build with content.
==
```

Like a [`$mist`](#mist) except the leaf nodes (files and directories) contain the [`$lobe`](#lobe) (content hash).
A Ford build key in which the leaf nodes (files and directories) carry the
[`$lobe`](#lobe) (content hash).

> **Legacy.** This type is no longer part of Clay's public interface. It survives
> only inside `clay.hoon`'s state-migration core, where it is used to type
> pre-existing state during an upgrade. The global Ford cache it belonged to was
> removed from vane state in 2026.

---

Expand Down Expand Up @@ -982,6 +984,11 @@ Ford result

The actual data in the Ford cache.

> **Legacy.** This type is no longer part of Clay's public interface. It survives
> only inside `clay.hoon`'s state-migration core, where it is used to type
> pre-existing state during an upgrade. The global Ford cache it belonged to was
> removed from vane state in 2026.

---

### `$soba` {#soba}
Expand Down Expand Up @@ -1061,6 +1068,19 @@ This is a parametrized type for list changes. For example, `(urge @t)` is a list

---

### `$weft` {#weft}

Kernel version

```hoon
+$ weft [lal=@tas num=@ud]
```

A single kernel version, such as `[%zuse 408]`. `lal` is the component name and
`num` its kelvin. Defined in `arvo.hoon`.

---

### `$waft` {#waft}

Kelvin range
Expand Down Expand Up @@ -1202,6 +1222,11 @@ Ford cache key

This includes all build inputs, including transitive dependencies, recursively.

> **Legacy.** This type is no longer part of Clay's public interface. It survives
> only inside `clay.hoon`'s state-migration core, where it is used to type
> pre-existing state during an upgrade. The global Ford cache it belonged to was
> removed from vane state in 2026.

---

### `$flow` {#flow}
Expand All @@ -1216,38 +1241,10 @@ Refcount includes references from other items in the cache, and from `spill`s in

This is optimized for minimizing the number of rebuilds, and given that, minimizing the amount of memory used. It is relatively slow to lookup, because generating a cache key can be fairly slow (for files, it requires parsing; for `$tube`s, it even requires building the marks).

---

### `$flue` {#flue}

Per-desk build cache

```hoon
+$ flue [spill=(set leak) sprig=(map mist [=leak =soak])]
```

- `spill` is the set of "roots" we have into the [global ford cache](#flow). We add a root for everything referenced directly or indirectly on a desk, then invalidate them on commit only if their dependencies change.
- `sprig` is a fast-lookup index over the global ford cache. The only goal is to make cache hits fast.

---

### `$mist` {#mist}

Ford build without content

```hoon
+$ mist
$% [%file =path]
[%nave =mark]
[%dais =mark]
[%cast =mars]
[%tube =mars]
[%vale =path]
[%arch =path]
==
```

This is used at the index of `sprig`s in [`$flue`](#flue)s.
> **Legacy.** This type is no longer part of Clay's public interface. It survives
> only inside `clay.hoon`'s state-migration core, where it is used to type
> pre-existing state during an upgrade. The global Ford cache it belonged to was
> removed from vane state in 2026.

---

Expand Down
25 changes: 8 additions & 17 deletions content/urbit-os/kernel/clay/scry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,6 @@ Each of the possible `[path]`s are described below.

***

### `/sweep` - Cache check <a href="#sweep---cache-check" id="sweep---cache-check"></a>

A buc scry with a path of `/sweep` will check the global ford cache for refcount errors. It returns a `(list [need=@ud have=@ud leak])`, where a [`$leak`](data-types.md#leak) is a Ford cache key used internally by Clay.

Example:

```
> .^((list [need=@ud have=@ud *]) %cx /=//=/sweep)
~
```

***

### `/rang` - Get `$rang` <a href="#rang---get-rang" id="rang---get-rang"></a>

A buc scry with a path of `/rang` will return the full [`$rang`](data-types.md#rang) from Clay's state.
Expand Down Expand Up @@ -101,15 +88,19 @@ Example:

***

### `/flow` - Build cache <a href="#flow---build-cache" id="flow---build-cache"></a>
### `/esse` - Desk essential? <a href="#esse---desk-essential" id="esse---desk-essential"></a>

A buc scry with a path of `/esse/[desk]` will return whether the given desk is
marked essential. An essential desk is not suspended when an incompatible kernel
update arrives. The type returned is a `?`.

A buc scry with a path of `/flow` will return the global build cache. The type returned is a [`$flow:clay`](./data-types.md#flow).
The desk is mandatory; a bare `/esse` fails.

Example:

```
> ~(wyt by .^(flow:clay %cx /=//=/flow))
960
> .^(? %cx /=//=/esse/base)
%.y
```

***
Expand Down