d25ed36e14
Publishers no longer need external hosting at all. The service runs its own s3d (SiaFoundation gateway) as a Docker sidecar on the internal compose network; scan-clean uploads land there via PUT; readers fetch through /dl/<sha256> on prometheus.x which proxies back out from s3d. - deploy/docker-compose.yml: s3d service added. Image is ghcr.io/siafoundation/s3d:latest, data folder bind-mounted at ./s3d-data/ so the operator can seed it from a Pithos connection export. Only prometheus-internal network — never public. - deploy/setup-s3d.sh: one-time import of the Pithos connection tarball, writes a fresh s3d.yml with a random adminPassword, mints an S3 access key for the Prometheus service, appends PROMETHEUS_S3_* lines to .env. Idempotent on re-run. - service/storage.js: endpoint now defaults to the internal s3d (http://s3d:8000); publicBase defaults to https://prometheus.x so download URLs flow through the service's own /dl handler. Bucket is auto-created on first use so a fresh data folder bootstraps. Added getObjectStream() for the read side. - service/downloads.js: GET/HEAD /dl/<sha256> — forwards Range, content-type and content-length from s3d, drops s3d-specific headers (x-amz-*), sets immutable long-cache. Hash is validated against ^[0-9a-f]{64}$ before touching upstream. - service/server.js: route table wires /dl/* to the handler above.
113 lines
4 KiB
Bash
113 lines
4 KiB
Bash
#!/usr/bin/env bash
|
|
# One-time setup for the s3d sidecar.
|
|
#
|
|
# Pre-conditions
|
|
# --------------
|
|
# - The operator holds a Pithos connection export tarball
|
|
# (`pithos-connection.tar.gz`) created from the Pithos account that
|
|
# should own Prometheus's bucket. The export contains the connected
|
|
# s3d data folder — this script copies it into place.
|
|
# - This script runs from `~/prometheus/deploy/` on the VPS.
|
|
#
|
|
# What this script does
|
|
# ---------------------
|
|
# 1. Extracts the Pithos data folder into ./s3d-data/
|
|
# 2. Writes ./s3d-data/s3d.yml with a fresh random adminPassword
|
|
# 3. Starts the s3d container
|
|
# 4. Mints an S3 access key for the Prometheus service
|
|
# 5. Appends PROMETHEUS_S3_* lines to .env
|
|
# 6. Reports the keys that landed (operator reads them off the terminal)
|
|
#
|
|
# Safe to re-run: steps 2-5 are gated behind "file already there?" checks.
|
|
# If you need a clean import, `docker compose down s3d && rm -rf s3d-data`
|
|
# first.
|
|
|
|
set -euo pipefail
|
|
|
|
cd "$(dirname "$0")"
|
|
|
|
TARBALL="${1:-$HOME/pithos-connection.tar.gz}"
|
|
DATA_DIR="./s3d-data"
|
|
ENV_FILE="./.env"
|
|
|
|
if [ ! -d "$DATA_DIR" ] || [ ! -f "$DATA_DIR/s3d.db" ]; then
|
|
if [ ! -f "$TARBALL" ]; then
|
|
echo "error: no s3d data folder yet, and $TARBALL does not exist." >&2
|
|
echo " scp the pithos-connection.tar.gz up first, or pass its path as \$1." >&2
|
|
exit 1
|
|
fi
|
|
echo "→ importing Pithos data folder from $TARBALL …"
|
|
mkdir -p "$DATA_DIR"
|
|
tar -xzf "$TARBALL" -C /tmp
|
|
cp -a /tmp/data/. "$DATA_DIR/"
|
|
rm -rf /tmp/data /tmp/MANIFEST.json /tmp/RESTORE.md
|
|
chown -R 1000:1000 "$DATA_DIR" # s3d image runs as uid 1000
|
|
fi
|
|
|
|
if [ ! -f "$DATA_DIR/s3d.yml" ]; then
|
|
echo "→ writing fresh s3d.yml with a random adminPassword …"
|
|
# Random 32-byte base64 — plenty for an admin password that stays local.
|
|
ADMIN_PW="$(head -c 32 /dev/urandom | base64 | tr -d '/+=' | head -c 40)"
|
|
cat > "$DATA_DIR/s3d.yml" <<EOF
|
|
directory: /data
|
|
apiAddress: 0.0.0.0:8000
|
|
adminAddress: 127.0.0.1:8001
|
|
adminPassword: $ADMIN_PW
|
|
EOF
|
|
chmod 600 "$DATA_DIR/s3d.yml"
|
|
chown 1000:1000 "$DATA_DIR/s3d.yml"
|
|
fi
|
|
|
|
echo "→ bringing up s3d …"
|
|
docker compose up -d s3d
|
|
|
|
# Wait for the daemon to answer.
|
|
echo -n "→ waiting for s3d to accept connections "
|
|
for _ in $(seq 1 30); do
|
|
if docker compose exec -T s3d /usr/bin/s3d status >/dev/null 2>&1; then
|
|
echo " ok"; break
|
|
fi
|
|
echo -n "."; sleep 1
|
|
done
|
|
|
|
# Pick a user to mint the key under. The Pithos export lands with one user
|
|
# (the account the export was taken from); default to that.
|
|
USER="$(docker compose exec -T s3d /usr/bin/s3d users list 2>/dev/null | awk 'NR==1{print $1}' || true)"
|
|
if [ -z "$USER" ]; then
|
|
USER="prometheus"
|
|
echo "→ no user found, creating '$USER' …"
|
|
docker compose exec -T s3d /usr/bin/s3d users create "$USER"
|
|
fi
|
|
echo "→ using user: $USER"
|
|
|
|
# If PROMETHEUS_S3_ACCESS_KEY is already set in .env, assume this already
|
|
# ran and skip the key mint — avoids piling up unused keys on re-run.
|
|
if grep -q '^PROMETHEUS_S3_ACCESS_KEY=' "$ENV_FILE" 2>/dev/null; then
|
|
echo "→ .env already has PROMETHEUS_S3_ACCESS_KEY — leaving it alone."
|
|
else
|
|
echo "→ minting an access key for '$USER' …"
|
|
OUT="$(docker compose exec -T s3d /usr/bin/s3d keys create "$USER")"
|
|
AK="$(echo "$OUT" | sed -nE 's/.*Access Key:[[:space:]]+(.+)/\1/p' | tr -d '\r')"
|
|
SK="$(echo "$OUT" | sed -nE 's/.*Secret Key:[[:space:]]+(.+)/\1/p' | tr -d '\r')"
|
|
if [ -z "$AK" ] || [ -z "$SK" ]; then
|
|
echo "error: could not parse keys from s3d output:" >&2
|
|
echo "$OUT" >&2
|
|
exit 2
|
|
fi
|
|
touch "$ENV_FILE"
|
|
{
|
|
echo ""
|
|
echo "# s3d sidecar — created $(date -u +%Y-%m-%dT%H:%M:%SZ) by setup-s3d.sh"
|
|
echo "PROMETHEUS_S3_ENDPOINT=http://s3d:8000"
|
|
echo "PROMETHEUS_S3_ACCESS_KEY=$AK"
|
|
echo "PROMETHEUS_S3_SECRET=$SK"
|
|
echo "PROMETHEUS_S3_BUCKET=prometheus"
|
|
echo "PROMETHEUS_S3_PREFIX=apps/"
|
|
echo "PROMETHEUS_PUBLIC_BASE=https://prometheus.x"
|
|
} >> "$ENV_FILE"
|
|
echo "→ wrote PROMETHEUS_S3_* to $ENV_FILE"
|
|
fi
|
|
|
|
echo ""
|
|
echo "✓ s3d sidecar is up. Restart prometheus to pick up the new .env:"
|
|
echo " docker compose up -d --force-recreate prometheus"
|