pithos.sia docs: storage for scripts, backups and AI agents
Covers giving each job its own S3 user, the exported JSON, using it from the AWS CLI, restic, rclone, boto3 and the Node SDK, where the endpoint is reachable from (local agents vs cloud agents), the narrower folder keys on Drives on Silent Mode, and why there is no SSH key: s3d speaks S3 only, with rclone serve sftp as the bridge for tools that need SFTP.
This commit is contained in:
Local Dev
parent
a539f2d90d
commit
3220a2461f
2 changed files with 142 additions and 1 deletions
|
|
@ -66,7 +66,7 @@
|
|||
<a class="md" href="pithos-docs.md" title="The same page as plain Markdown, for AI assistants">Markdown ↓</a>
|
||||
</nav>
|
||||
<div class="layout">
|
||||
<nav class="toc" aria-label="On this page"><div class="t">On this page</div><a href="#how-the-pieces-fit">How the pieces fit</a><a href="#getting-started-pithos-in-theseus">Getting started (Pithos in Theseus)</a><a href="#the-pithos-screen">The Pithos screen</a><a href="#drives-on-silent-mode">Drives on Silent Mode</a><a href="#other-ways-to-run-pithos">Other ways to run Pithos</a><a href="#s3d-on-the-command-line">s3d on the command line</a><a href="#using-drives-from-other-apps">Using drives from other apps</a><a href="#common-tasks">Common tasks</a><a href="#troubleshooting">Troubleshooting</a><a href="#limits">Limits</a><a href="#for-ai-assistants">For AI assistants</a></nav>
|
||||
<nav class="toc" aria-label="On this page"><div class="t">On this page</div><a href="#how-the-pieces-fit">How the pieces fit</a><a href="#getting-started-pithos-in-theseus">Getting started (Pithos in Theseus)</a><a href="#the-pithos-screen">The Pithos screen</a><a href="#drives-on-silent-mode">Drives on Silent Mode</a><a href="#other-ways-to-run-pithos">Other ways to run Pithos</a><a href="#s3d-on-the-command-line">s3d on the command line</a><a href="#using-drives-from-other-apps">Using drives from other apps</a><a href="#storage-for-scripts-backups-and-ai-agents">Storage for scripts, backups and AI agents</a><a href="#common-tasks">Common tasks</a><a href="#troubleshooting">Troubleshooting</a><a href="#limits">Limits</a><a href="#for-ai-assistants">For AI assistants</a></nav>
|
||||
<article>
|
||||
<h1>Pithos documentation</h1>
|
||||
<div class="agent"><strong>Using an AI assistant?</strong> Give it this page or the <a href="pithos-docs.md">Markdown version</a>, then describe what you want to do. Type recovery phrases and secret keys only into s3d or Pithos on your own computer, never into a chat.</div>
|
||||
|
|
@ -176,6 +176,54 @@ force_path_style = true</code></pre>
|
|||
<pre class="code" data-lang="bash"><button class="copy" type="button" aria-label="Copy">Copy</button><code>rclone ls s3d:my-drive
|
||||
rclone copy ./photos s3d:my-drive/photos --progress</code></pre>
|
||||
<p><strong>Environment variables</strong> for SDKs and tools: <code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code>, <code>AWS_DEFAULT_REGION=us-east-1</code>, <code>AWS_ENDPOINT_URL=http://127.0.0.1:8000</code>.</p>
|
||||
<h2 id="storage-for-scripts-backups-and-ai-agents"><a class="anchor" href="#storage-for-scripts-backups-and-ai-agents" aria-hidden="true">#</a>Storage for scripts, backups and AI agents</h2>
|
||||
<p>A script, a backup tool or an AI agent uses your storage like any S3 app: an endpoint and an access key. Give each one its own key, so you can see what it does and switch it off alone.</p>
|
||||
<h3 id="1-give-it-its-own-user-and-key"><a class="anchor" href="#1-give-it-its-own-user-and-key" aria-hidden="true">#</a>1. Give it its own user and key</h3>
|
||||
<ol><li><strong>Users & keys › New user</strong>, named after the job (<code>backup-agent</code>, <code>claude-notes</code>). Pithos creates its key and a drive of the same name.</li><li>Optional: <strong>Drives › New drive</strong> for that user if you want a differently named drive.</li><li>Open the key (<strong>Show secret & config</strong>) and press <strong>Export JSON…</strong>. Pick the drive and save the file.</li></ol>
|
||||
<p>The file looks like this:</p>
|
||||
<pre class="code" data-lang="json"><button class="copy" type="button" aria-label="Copy">Copy</button><code>{
|
||||
"endpoint": "http://127.0.0.1:8000",
|
||||
"accessKey": "…",
|
||||
"secretKey": "…",
|
||||
"bucket": "backup-agent"
|
||||
}</code></pre>
|
||||
<ul><li>The key reaches <strong>every drive of that user</strong>; <code>bucket</code> only says which one to use. A user's keys cannot see other users' drives, which is why one user per job is the boundary.</li><li>The file is <strong>not encrypted</strong>. Keep it out of git, chats and shared folders. <strong>Save credentials…</strong> makes an encrypted copy instead (password or Theseus vault) for keeping.</li><li>To stop the agent: <strong>Users & keys › Revoke</strong> on its key (or delete the user). Nothing else changes.</li></ul>
|
||||
<h3 id="2-use-it"><a class="anchor" href="#2-use-it" aria-hidden="true">#</a>2. Use it</h3>
|
||||
<p><strong>Environment variables</strong> (most tools read these):</p>
|
||||
<pre class="code" data-lang="bash"><button class="copy" type="button" aria-label="Copy">Copy</button><code>export AWS_ACCESS_KEY_ID=$(jq -r .accessKey pithos-backup-agent.json)
|
||||
export AWS_SECRET_ACCESS_KEY=$(jq -r .secretKey pithos-backup-agent.json)
|
||||
export AWS_REGION=us-east-1
|
||||
export AWS_ENDPOINT_URL=$(jq -r .endpoint pithos-backup-agent.json)
|
||||
BUCKET=$(jq -r .bucket pithos-backup-agent.json)
|
||||
aws s3 cp report.pdf "s3://$BUCKET/reports/report.pdf"
|
||||
aws s3 ls "s3://$BUCKET/reports/"</code></pre>
|
||||
<p><strong>Backups with restic</strong> (encrypted, deduplicated snapshots; restore any day):</p>
|
||||
<pre class="code" data-lang="bash"><button class="copy" type="button" aria-label="Copy">Copy</button><code># the four exports above, then:
|
||||
export RESTIC_REPOSITORY="s3:$AWS_ENDPOINT_URL/$BUCKET/restic"
|
||||
export RESTIC_PASSWORD_FILE=~/.config/restic/password # keep this password: without it the backup cannot be read
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 init
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 backup ~/Documents
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 snapshots</code></pre>
|
||||
<p><strong>rclone</strong> (sync, mount, serve): add a remote with the four values (<code>provider = Other</code>, <code>force_path_style = true</code>), as in "Using drives from other apps" above, then <code>rclone sync ~/Projects backup:backup-agent/projects</code>.</p>
|
||||
<p><strong>Python</strong> (boto3):</p>
|
||||
<pre class="code" data-lang="python"><button class="copy" type="button" aria-label="Copy">Copy</button><code>import json, boto3
|
||||
from botocore.config import Config
|
||||
c = json.load(open("pithos-backup-agent.json"))
|
||||
s3 = boto3.client("s3", endpoint_url=c["endpoint"], aws_access_key_id=c["accessKey"],
|
||||
aws_secret_access_key=c["secretKey"], region_name="us-east-1",
|
||||
config=Config(s3={"addressing_style": "path"}))
|
||||
s3.upload_file("notes.md", c["bucket"], "notes/notes.md")
|
||||
print([o["Key"] for o in s3.list_objects_v2(Bucket=c["bucket"]).get("Contents", [])])</code></pre>
|
||||
<p><strong>Node</strong> (AWS SDK v3): <code>new S3Client({ endpoint: c.endpoint, region: "us-east-1", forcePathStyle: true, credentials: { accessKeyId: c.accessKey, secretAccessKey: c.secretKey } })</code>.</p>
|
||||
<h3 id="3-where-the-agent-can-reach-it"><a class="anchor" href="#3-where-the-agent-can-reach-it" aria-hidden="true">#</a>3. Where the agent can reach it</h3>
|
||||
<ul><li><strong>On the same computer</strong> as s3d: the endpoint <code>http://127.0.0.1:8000</code> works as is. Local agents (Claude Code, a cron job, a backup app) need nothing else.</li><li><strong>From another machine or a cloud agent</strong>: <code>127.0.0.1</code> points to <em>that</em> machine, so it fails. Either move the drives to <strong>Drives on Silent Mode</strong> (the endpoint becomes <code>https://s3.silentmode.st</code> and keeps working with your computer off) or put s3d behind your own server with TLS and write that address into the JSON as <code>endpoint</code>.</li><li><strong>Narrower keys</strong>: on Drives on Silent Mode, ⋯ › Share… on a folder can make a key for one folder only: <em>read</em>, <em>add new files</em> (it can create files but never overwrite or delete one, good for an agent that drops reports or exports) or <em>add, change and delete</em>. restic needs the last one, because it removes its own lock files. A plain s3d only has whole-user keys.</li><li>Files a tool uploads wait on this computer until a ~40 MB block fills; automatic upload sends leftovers after 10 quiet minutes, or press <strong>Upload now</strong>. Until then they are not on Sia yet.</li></ul>
|
||||
<h3 id="ssh-keys-and-sftp"><a class="anchor" href="#ssh-keys-and-sftp" aria-hidden="true">#</a>SSH keys and SFTP</h3>
|
||||
<p>s3d speaks only S3: there is no SSH login or SSH key to add. If a tool insists on SFTP, run rclone in front of it:</p>
|
||||
<pre class="code" data-lang="bash"><button class="copy" type="button" aria-label="Copy">Copy</button><code>rclone serve sftp backup:backup-agent --addr 127.0.0.1:2022 --authorized-keys ~/.ssh/authorized_keys</code></pre>
|
||||
<p>That gives SSH-key logins to one drive, but underneath it is the same S3 key, and rclone has to keep running. For agents and scripts, an S3 key per job (above) is simpler and easier to revoke; use SFTP only for tools that cannot do S3. <code>rclone serve webdav</code> and <code>rclone mount</code> work the same way for apps that want a network drive.</p>
|
||||
<h3 id="telling-an-ai-agent"><a class="anchor" href="#telling-an-ai-agent" aria-hidden="true">#</a>Telling an AI agent</h3>
|
||||
<p>Give the agent the JSON file path, not the secret in the chat, and a short brief such as:</p>
|
||||
<p>> Use the S3 storage described in <code>pithos-backup-agent.json</code> (keys <code>endpoint</code>, <code>accessKey</code>, <code>secretKey</code>, <code>bucket</code>; region <code>us-east-1</code>; path-style addressing). Only read and write inside that bucket, under <code>backups/</code>. Never print the secret key. Do not delete objects unless I ask.</p>
|
||||
<h2 id="common-tasks"><a class="anchor" href="#common-tasks" aria-hidden="true">#</a>Common tasks</h2>
|
||||
<ul><li><strong>Give an app its own key.</strong> Users & keys › create a user for the app (or use yours) › Create key. Revoke it later without touching your other keys.</li><li><strong>Share one file or folder with anyone.</strong> ⋯ › Share… › Create public link. The link reads only that file from your Sia account; nothing else in the account is exposed.</li><li><strong>Make a whole drive public.</strong> Keep the files in a drive of their own and set Access › Public read. Files are then at <code>http://<s3d address>/<drive>/<path></code>. On a default setup that address is reachable only from this computer; for other people s3d needs a public address (for example behind your own server and a TLS proxy).</li><li><strong>Rename or move.</strong> ⋯ › Rename… on a file, folder or drive. Renaming a drive changes its name for S3 apps too.</li><li><strong>Back up.</strong> Keep the recovery phrase offline (or use the vault-derived phrase and back up the Theseus vault). Optionally <code>POST /snapshots</code> to keep a copy of s3d's database on Sia.</li><li><strong>Move to another computer.</strong> Install s3d, run <code>s3d login</code> with the same indexer login and the same phrase, then restore the database snapshot or recreate users and keys.</li></ul>
|
||||
<h2 id="troubleshooting"><a class="anchor" href="#troubleshooting" aria-hidden="true">#</a>Troubleshooting</h2>
|
||||
|
|
@ -185,6 +233,7 @@ rclone copy ./photos s3d:my-drive/photos --progress</code></pre>
|
|||
<h2 id="for-ai-assistants"><a class="anchor" href="#for-ai-assistants" aria-hidden="true">#</a>For AI assistants</h2>
|
||||
<p>You are helping someone run <strong>s3d</strong> (the Sia S3 gateway), optionally through <strong>Pithos</strong>. Facts to rely on:</p>
|
||||
<ul><li>s3d v0.2.0+, S3 API on <code>127.0.0.1:8000</code>, admin API on <code>127.0.0.1:8001</code> (Basic auth, empty user, <code>adminPassword</code>), region <code>us-east-1</code>, path-style addressing.</li><li>Setup order: <code>s3d config</code> → <code>s3d login</code> (indexer URL + 12-word phrase, approve the "S3d" app at the indexer) → run <code>s3d</code> → <code>s3d users create</code> → <code>s3d keys create</code> → use any S3 client.</li><li>Small files wait locally until a ~40 MB block fills; <code>s3d flush</code> or <code>POST /objects/flush</code> uploads them now.</li><li>Buckets ("drives") can be private, public read, or public read + list; nothing finer.</li><li>Config lookup: <code>S3D_CONFIG_FILE</code>, else the OS default path listed above.</li></ul>
|
||||
<p>For an agent that should store files: ask the user to create a separate S3 user in Pithos and export its key as JSON (Users & keys › Show secret & config › Export JSON…), then read the file instead of having secrets pasted. See "Storage for scripts, backups and AI agents".</p>
|
||||
<p>Rules: never ask the user to paste their recovery phrase or secret keys into the conversation; have them type secrets only into s3d or Pithos on their own machine. Prefer read-only checks (<code>s3d status</code>, <code>s3d users list</code>, <code>aws s3 ls</code>) before changing anything, and confirm before deleting users, keys, drives or files.</p>
|
||||
<footer>Pithos is open source: <a href="https://code.silentmode.st/silentmode/pithos">code.silentmode.st/silentmode/pithos</a>. s3d is by the Sia Foundation: <a href="https://github.com/SiaFoundation/s3d">github.com/SiaFoundation/s3d</a>.</footer>
|
||||
</article>
|
||||
|
|
|
|||
|
|
@ -253,6 +253,96 @@ rclone copy ./photos s3d:my-drive/photos --progress
|
|||
|
||||
**Environment variables** for SDKs and tools: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION=us-east-1`, `AWS_ENDPOINT_URL=http://127.0.0.1:8000`.
|
||||
|
||||
## Storage for scripts, backups and AI agents
|
||||
|
||||
A script, a backup tool or an AI agent uses your storage like any S3 app: an endpoint and an access key. Give each one its own key, so you can see what it does and switch it off alone.
|
||||
|
||||
### 1. Give it its own user and key
|
||||
|
||||
1. **Users & keys › New user**, named after the job (`backup-agent`, `claude-notes`). Pithos creates its key and a drive of the same name.
|
||||
2. Optional: **Drives › New drive** for that user if you want a differently named drive.
|
||||
3. Open the key (**Show secret & config**) and press **Export JSON…**. Pick the drive and save the file.
|
||||
|
||||
The file looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"endpoint": "http://127.0.0.1:8000",
|
||||
"accessKey": "…",
|
||||
"secretKey": "…",
|
||||
"bucket": "backup-agent"
|
||||
}
|
||||
```
|
||||
|
||||
- The key reaches **every drive of that user**; `bucket` only says which one to use. A user's keys cannot see other users' drives, which is why one user per job is the boundary.
|
||||
- The file is **not encrypted**. Keep it out of git, chats and shared folders. **Save credentials…** makes an encrypted copy instead (password or Theseus vault) for keeping.
|
||||
- To stop the agent: **Users & keys › Revoke** on its key (or delete the user). Nothing else changes.
|
||||
|
||||
### 2. Use it
|
||||
|
||||
**Environment variables** (most tools read these):
|
||||
|
||||
```bash
|
||||
export AWS_ACCESS_KEY_ID=$(jq -r .accessKey pithos-backup-agent.json)
|
||||
export AWS_SECRET_ACCESS_KEY=$(jq -r .secretKey pithos-backup-agent.json)
|
||||
export AWS_REGION=us-east-1
|
||||
export AWS_ENDPOINT_URL=$(jq -r .endpoint pithos-backup-agent.json)
|
||||
BUCKET=$(jq -r .bucket pithos-backup-agent.json)
|
||||
aws s3 cp report.pdf "s3://$BUCKET/reports/report.pdf"
|
||||
aws s3 ls "s3://$BUCKET/reports/"
|
||||
```
|
||||
|
||||
**Backups with restic** (encrypted, deduplicated snapshots; restore any day):
|
||||
|
||||
```bash
|
||||
# the four exports above, then:
|
||||
export RESTIC_REPOSITORY="s3:$AWS_ENDPOINT_URL/$BUCKET/restic"
|
||||
export RESTIC_PASSWORD_FILE=~/.config/restic/password # keep this password: without it the backup cannot be read
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 init
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 backup ~/Documents
|
||||
restic -o s3.bucket-lookup=path -o s3.region=us-east-1 snapshots
|
||||
```
|
||||
|
||||
**rclone** (sync, mount, serve): add a remote with the four values (`provider = Other`, `force_path_style = true`), as in "Using drives from other apps" above, then `rclone sync ~/Projects backup:backup-agent/projects`.
|
||||
|
||||
**Python** (boto3):
|
||||
|
||||
```python
|
||||
import json, boto3
|
||||
from botocore.config import Config
|
||||
c = json.load(open("pithos-backup-agent.json"))
|
||||
s3 = boto3.client("s3", endpoint_url=c["endpoint"], aws_access_key_id=c["accessKey"],
|
||||
aws_secret_access_key=c["secretKey"], region_name="us-east-1",
|
||||
config=Config(s3={"addressing_style": "path"}))
|
||||
s3.upload_file("notes.md", c["bucket"], "notes/notes.md")
|
||||
print([o["Key"] for o in s3.list_objects_v2(Bucket=c["bucket"]).get("Contents", [])])
|
||||
```
|
||||
|
||||
**Node** (AWS SDK v3): `new S3Client({ endpoint: c.endpoint, region: "us-east-1", forcePathStyle: true, credentials: { accessKeyId: c.accessKey, secretAccessKey: c.secretKey } })`.
|
||||
|
||||
### 3. Where the agent can reach it
|
||||
|
||||
- **On the same computer** as s3d: the endpoint `http://127.0.0.1:8000` works as is. Local agents (Claude Code, a cron job, a backup app) need nothing else.
|
||||
- **From another machine or a cloud agent**: `127.0.0.1` points to *that* machine, so it fails. Either move the drives to **Drives on Silent Mode** (the endpoint becomes `https://s3.silentmode.st` and keeps working with your computer off) or put s3d behind your own server with TLS and write that address into the JSON as `endpoint`.
|
||||
- **Narrower keys**: on Drives on Silent Mode, ⋯ › Share… on a folder can make a key for one folder only: *read*, *add new files* (it can create files but never overwrite or delete one, good for an agent that drops reports or exports) or *add, change and delete*. restic needs the last one, because it removes its own lock files. A plain s3d only has whole-user keys.
|
||||
- Files a tool uploads wait on this computer until a ~40 MB block fills; automatic upload sends leftovers after 10 quiet minutes, or press **Upload now**. Until then they are not on Sia yet.
|
||||
|
||||
### SSH keys and SFTP
|
||||
|
||||
s3d speaks only S3: there is no SSH login or SSH key to add. If a tool insists on SFTP, run rclone in front of it:
|
||||
|
||||
```bash
|
||||
rclone serve sftp backup:backup-agent --addr 127.0.0.1:2022 --authorized-keys ~/.ssh/authorized_keys
|
||||
```
|
||||
|
||||
That gives SSH-key logins to one drive, but underneath it is the same S3 key, and rclone has to keep running. For agents and scripts, an S3 key per job (above) is simpler and easier to revoke; use SFTP only for tools that cannot do S3. `rclone serve webdav` and `rclone mount` work the same way for apps that want a network drive.
|
||||
|
||||
### Telling an AI agent
|
||||
|
||||
Give the agent the JSON file path, not the secret in the chat, and a short brief such as:
|
||||
|
||||
> Use the S3 storage described in `pithos-backup-agent.json` (keys `endpoint`, `accessKey`, `secretKey`, `bucket`; region `us-east-1`; path-style addressing). Only read and write inside that bucket, under `backups/`. Never print the secret key. Do not delete objects unless I ask.
|
||||
|
||||
## Common tasks
|
||||
|
||||
- **Give an app its own key.** Users & keys › create a user for the app (or use yours) › Create key. Revoke it later without touching your other keys.
|
||||
|
|
@ -293,4 +383,6 @@ You are helping someone run **s3d** (the Sia S3 gateway), optionally through **P
|
|||
- Buckets ("drives") can be private, public read, or public read + list; nothing finer.
|
||||
- Config lookup: `S3D_CONFIG_FILE`, else the OS default path listed above.
|
||||
|
||||
For an agent that should store files: ask the user to create a separate S3 user in Pithos and export its key as JSON (Users & keys › Show secret & config › Export JSON…), then read the file instead of having secrets pasted. See "Storage for scripts, backups and AI agents".
|
||||
|
||||
Rules: never ask the user to paste their recovery phrase or secret keys into the conversation; have them type secrets only into s3d or Pithos on their own machine. Prefer read-only checks (`s3d status`, `s3d users list`, `aws s3 ls`) before changing anything, and confirm before deleting users, keys, drives or files.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue