tawasuyu/brahman
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(shipote): data plane + DAG fan-in/out + stats + lifecycle (fases F-I)

Browse Source 
Pipeline runtime:
- Fan-out 1→N (splitter task replica al N consumers) y fan-in N→1 (merger
  task con mpsc + reader-per-input). DAGs no lineales soportados.
- Flow channels: Unix socket + tokio broadcast con replay buffer
  configurable por pipeline (DiscernPolicy.replay_chunks). Subscribers
  externos vía `shipote flow tail <socket>`.
- Templating en specs con `${KEY}` (CLI `--var KEY=VALUE`). Walk
  recursivo sobre serde_json::Value, soporta todos los strings del schema.
- Pipelines guardados (`pipeline save/saved-list/drop/run-saved`)
  persisten con el snapshot.

Lifecycle de comandos:
- Log capture per-stream (stdout/stderr separados) via pipe O_CLOEXEC +
  AsyncFd. CLI `shipote logs <ws> <cmd> --stream {stdout,stderr,both}`.
- Stop graceful con tiempo configurable: SIGTERM → grace → SIGKILL.
  Tanto a nivel workspace como pipeline individual.
- TTL auto-stop ya existente (Fase C) sigue funcionando.

ente-incarnate:
- ChildStdio declarativo (Fase C) + ChildPreExec declarativo nuevo:
  NoNewPrivs, ParentDeathSig, Dumpable, NewSession, Chdir, Umask.
- Aplicación pre-execve async-signal-safe en ambos paths (plain via
  Command::pre_exec, namespaced via callback del clone(2)).

Observabilidad:
- WorkspaceStats: RSS + RSS peak (VmHWM o memory.peak cgroup) + CPU usec
  + uptime. Fuente per-proc o cgroup según delegation.
- shipote-shell con sparkline ASCII por workspace (history cap 24),
  card de flow channels activos, vista de comandos + saved pipelines.
- Tap → broker: cada edge enriquecido con TypeRef se anuncia como Card
  efímera vía SidecarPool (graceful si broker no corre).

Discern:
- Integrado en yahweh-provider-fs (mime_type en EntityNode).
- Integrado en nouser-core::cluster::pick_lens como fallback cuando la
  extensión cae a Lens::Grid.

79 tests pasan: ente-incarnate (16), nouser-core (27), shipote-card (8),
shipote-core (20), shipote-discern (5), yahweh-provider-fs (3).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
sergio
2026-05-11 00:29:46 +00:00
parent c22d2480b9
commit 36dac00c8d
13 changed files with 2187 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/modules/shipote/shipote-core/src/lib.rs
+318 -47
View File
@@ -11,9 +11,11 @@
// el módulo concreto.
#![deny(unsafe_op_in_unsafe_fn)]
pub mod flow_channel;
pub mod logbuf;
pub mod persist;
pub mod pipeline;
pub mod stats;
use brahman_card::{Card, Payload, Supervision};
use ente_incarnate::{Incarnator, IncarnatorConfig};
@@ -55,10 +57,22 @@ pub struct CommandState {
pub pid: Pid,
pub alive: bool,
pub exit_status: Option<i32>,
/// Ring buffer compartido con la tokio task que drena stdout+stderr
/// del comando. `None` para comandos que no capturan output (futuro:
/// comandos con stdout=inherit).
pub logs: Option<logbuf::LogBuf>,
/// Ring buffer del stdout. `None` para comandos sin captura.
pub stdout: Option<logbuf::LogBuf>,
/// Ring buffer del stderr. Separado de `stdout` para que el CLI
/// pueda filtrarlos. `None` para comandos sin captura.
pub stderr: Option<logbuf::LogBuf>,
/// Si el comando fue lanzado como parte de un Pipeline, su ULID.
pub pipeline_id: Option<Ulid>,
}
/// Stream a leer en `get_command_logs`. `Both` concatena stderr-después-stdout
/// para una vista combinada (orden temporal aproximado).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LogStream {
Stdout,
Stderr,
Both,
}
pub struct WorkspaceManager {
@@ -72,6 +86,11 @@ struct Inner {
/// que "pipelines vivos" — son specs guardados para reusar con
/// `run-saved`. Sobreviven restart vía snapshot.
saved_pipelines: HashMap<String, PipelineSpec>,
/// Flow channels vivos por pipeline. Se retienen hasta que el
/// pipeline termine — cuando todos los hijos del pipeline murieron,
/// el reaper los borra (futuro). v1: viven hasta `stop_pipeline_flows`
/// explícito o hasta shutdown.
pipeline_flows: HashMap<Ulid, Vec<crate::flow_channel::FlowChannel>>,
}
#[derive(Debug, Clone)]
@@ -156,11 +175,134 @@ impl WorkspaceManager {
inner: Arc::new(Mutex::new(Inner {
workspaces: HashMap::new(),
saved_pipelines: HashMap::new(),
pipeline_flows: HashMap::new(),
})),
incarnator: Arc::new(Incarnator::new(cfg)),
}
}
/// Registra los comandos lanzados por un pipeline en el workspace.
/// Esto permite `pipeline_stop` (matar selectivamente sólo los pids
/// de un pipeline). `pipeline_id` se setea en cada CommandState.
pub async fn register_pipeline_commands(
&self,
workspace: WorkspaceId,
pipeline_id: Ulid,
commands: Vec<(String, i32)>,
) {
let mut g = self.inner.lock().await;
let Some(ws) = g.workspaces.get_mut(&workspace) else { return };
for (label, pid) in commands {
let cmd_id = Ulid::new();
ws.commands.insert(
cmd_id,
CommandState {
id: cmd_id,
label,
pid: Pid::from_raw(pid),
alive: true,
exit_status: None,
stdout: None,
stderr: None,
pipeline_id: Some(pipeline_id),
},
);
}
}
/// Detiene selectivamente los comandos de un pipeline. SIGTERM →
/// `grace` → SIGKILL. Devuelve cantidad reapeada. Si no hay comandos
/// del pipeline en ningún workspace, retorna 0.
pub async fn stop_pipeline(
&self,
pipeline_id: Ulid,
grace: std::time::Duration,
) -> u32 {
// 1) Recolectamos pids de ese pipeline en todos los workspaces.
let mut targets: Vec<Pid> = Vec::new();
{
let g = self.inner.lock().await;
for ws in g.workspaces.values() {
for cmd in ws.commands.values() {
if cmd.alive && cmd.pipeline_id == Some(pipeline_id) {
targets.push(cmd.pid);
}
}
}
}
if targets.is_empty() {
return 0;
}
let initial = if grace.is_zero() { Signal::SIGKILL } else { Signal::SIGTERM };
for pid in &targets {
let _ = kill(*pid, initial);
}
let mut reaped = 0u32;
let mut still = targets.clone();
let deadline = std::time::Instant::now() + grace;
let poll = std::time::Duration::from_millis(20);
while !still.is_empty() && std::time::Instant::now() < deadline {
still.retain(|pid| match waitpid(*pid, Some(WaitPidFlag::WNOHANG)) {
Ok(WaitStatus::StillAlive) => true,
Ok(_) => {
reaped += 1;
false
}
Err(_) => false,
});
if !still.is_empty() {
tokio::time::sleep(poll).await;
}
}
for pid in &still {
let _ = kill(*pid, Signal::SIGKILL);
let _ = waitpid(*pid, None);
reaped += 1;
}
// Marcar como dead en estado in-memory.
let mut g = self.inner.lock().await;
for ws in g.workspaces.values_mut() {
for cmd in ws.commands.values_mut() {
if cmd.pipeline_id == Some(pipeline_id) && cmd.alive {
cmd.alive = false;
}
}
}
// Drop flows del pipeline.
g.pipeline_flows.remove(&pipeline_id);
info!(%pipeline_id, reaped, "pipeline stopped");
reaped
}
/// Retiene los FlowChannels de un pipeline para que sobrevivan al
/// fin del request. Drop = cierre del data plane.
pub async fn retain_pipeline_flows(
&self,
pipeline: Ulid,
flows: Vec<crate::flow_channel::FlowChannel>,
) {
self.inner.lock().await.pipeline_flows.insert(pipeline, flows);
}
/// Lista pipelines vivos con sus sockets activos.
pub async fn list_flow_pipelines(&self) -> Vec<(Ulid, Vec<std::path::PathBuf>)> {
let g = self.inner.lock().await;
g.pipeline_flows
.iter()
.map(|(id, flows)| {
(
*id,
flows.iter().map(|f| f.socket_path().to_path_buf()).collect(),
)
})
.collect()
}
/// Cierra el data plane de un pipeline (drop = remove_file de sockets).
pub async fn drop_pipeline_flows(&self, pipeline: Ulid) -> bool {
self.inner.lock().await.pipeline_flows.remove(&pipeline).is_some()
}
pub fn incarnator(&self) -> &Incarnator {
&self.incarnator
}
@@ -208,6 +350,35 @@ impl WorkspaceManager {
.map(|w| w.spec.label.clone())
}
/// Estadísticas de recursos del workspace: RSS + CPU agregado de sus
/// comandos vivos. Lee `/proc/<pid>/` directamente; si el spec declara
/// `soma.cgroup.path`, también intenta el cgroup (más preciso, incluye
/// descendants).
pub async fn workspace_stats(&self, id: WorkspaceId) -> Option<stats::WorkspaceStats> {
let g = self.inner.lock().await;
let ws = g.workspaces.get(&id)?;
let alive: Vec<i32> = ws
.commands
.values()
.filter(|c| c.alive)
.map(|c| c.pid.as_raw())
.collect();
let total = ws.commands.len() as u32;
let cgroup_path = if ws.spec.soma.cgroup.path.is_empty() {
None
} else {
// resolve_cgroup_path está en ente_incarnate, pero acá basta
// con el path absoluto bajo /sys/fs/cgroup. Resolución gruesa.
Some(std::path::PathBuf::from(format!(
"/sys/fs/cgroup{}",
ws.spec.soma.cgroup.path
)))
};
let mut s = stats::measure(&alive, cgroup_path.as_deref(), ws.started);
s.commands_total = total;
Some(s)
}
pub async fn create(
self: &Arc<Self>,
spec: WorkspaceSpec,
@@ -269,31 +440,66 @@ impl WorkspaceManager {
}
pub async fn stop(&self, id: WorkspaceId) -> Result<u32, CoreError> {
self.stop_with_grace(id, std::time::Duration::from_millis(1000)).await
}
/// Variante con tiempo de gracia configurable. SIGTERM → espera `grace`
/// → SIGKILL si quedan vivos. `grace=0` = SIGKILL inmediato.
pub async fn stop_with_grace(
&self,
id: WorkspaceId,
grace: std::time::Duration,
) -> Result<u32, CoreError> {
let mut g = self.inner.lock().await;
let ws = g.workspaces.remove(&id).ok_or(CoreError::WorkspaceNotFound(id))?;
// También limpiamos flow_channels del workspace si los hubiera —
// por workspace lo retenemos por pipeline, no por workspace.
drop(g);
// 1) SIGTERM (o SIGKILL si grace=0) a todos vivos.
let initial_signal = if grace.is_zero() { Signal::SIGKILL } else { Signal::SIGTERM };
let alive_pids: Vec<Pid> = ws
.commands
.values()
.filter(|c| c.alive)
.map(|c| c.pid)
.collect();
for pid in &alive_pids {
let _ = kill(*pid, initial_signal);
}
// 2) Esperar hasta `grace` haciendo polling WNOHANG.
let mut reaped = 0u32;
for (_cid, cmd) in ws.commands {
if cmd.alive {
let _ = kill(cmd.pid, Signal::SIGTERM);
// Cosecha sin bloquear infinito: WNOHANG en loop con un par de intentos.
for _ in 0..50 {
match waitpid(cmd.pid, Some(WaitPidFlag::WNOHANG)) {
Ok(WaitStatus::StillAlive) => {
std::thread::sleep(std::time::Duration::from_millis(20));
}
Ok(_) => {
reaped += 1;
break;
}
Err(_) => break,
}
let mut still_alive: Vec<Pid> = alive_pids.clone();
let deadline = std::time::Instant::now() + grace;
let poll_interval = std::time::Duration::from_millis(20);
while !still_alive.is_empty() && std::time::Instant::now() < deadline {
still_alive.retain(|pid| match waitpid(*pid, Some(WaitPidFlag::WNOHANG)) {
Ok(WaitStatus::StillAlive) => true,
Ok(_) => {
reaped += 1;
false
}
// Último recurso: SIGKILL.
let _ = kill(cmd.pid, Signal::SIGKILL);
let _ = waitpid(cmd.pid, None);
Err(_) => false,
});
if !still_alive.is_empty() {
tokio::time::sleep(poll_interval).await;
}
}
info!(%id, reaped, "workspace stopped");
// 3) SIGKILL forzoso a los que quedan, y wait blocking.
for pid in &still_alive {
let _ = kill(*pid, Signal::SIGKILL);
let _ = waitpid(*pid, None);
reaped += 1;
}
info!(
%id,
reaped,
grace_ms = grace.as_millis() as u64,
sigkilled = still_alive.len(),
"workspace stopped"
);
Ok(reaped)
}
@@ -321,31 +527,36 @@ impl WorkspaceManager {
};
let card = cmd_ref.to_card(0, &workspace_label)?;
// Pipe para capturar stdout. O_CLOEXEC para que hijos del hijo
// no hereden la copia. v1: stderr=inherit (simplicidad; tail útil
// para stdout solo). Futuro: stderr separado en el ring.
let (capture_r, capture_w) =
// Dos pipes O_CLOEXEC: uno para stdout, otro para stderr.
use std::os::fd::IntoRawFd;
let (sout_r, sout_w) =
nix::unistd::pipe2(nix::fcntl::OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
use std::os::fd::IntoRawFd;
let capture_r_fd = capture_r.into_raw_fd();
let capture_w_fd = capture_w.into_raw_fd();
let (serr_r, serr_w) =
nix::unistd::pipe2(nix::fcntl::OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
let sout_r_fd = sout_r.into_raw_fd();
let sout_w_fd = sout_w.into_raw_fd();
let serr_r_fd = serr_r.into_raw_fd();
let serr_w_fd = serr_w.into_raw_fd();
let logs = logbuf::LogBuf::new();
let stdout_buf = logbuf::LogBuf::new();
let stderr_buf = logbuf::LogBuf::new();
let stdio = ente_incarnate::ChildStdio {
stdin_fd: None,
stdout_fd: Some(capture_w_fd),
stderr_fd: None,
stdout_fd: Some(sout_w_fd),
stderr_fd: Some(serr_w_fd),
};
let out = self.incarnator.incarnate_with(&card, stdio)?;
let cmd_id = card.id;
let cmd_label = cmd_ref.label.clone();
let pid = out.pid;
// Drainer: tokio task que lee capture_r_fd y appendea al ring.
spawn_log_drainer(capture_r_fd, logs.clone());
spawn_log_drainer(sout_r_fd, stdout_buf.clone());
spawn_log_drainer(serr_r_fd, stderr_buf.clone());
let mut g = self.inner.lock().await;
if let Some(ws) = g.workspaces.get_mut(&id) {
@@ -357,7 +568,9 @@ impl WorkspaceManager {
pid,
alive: true,
exit_status: None,
logs: Some(logs),
stdout: Some(stdout_buf),
stderr: Some(stderr_buf),
pipeline_id: None,
},
);
}
@@ -372,16 +585,28 @@ impl WorkspaceManager {
}
/// Devuelve el tail del log capturado para `(workspace, command)`.
/// `stream` selecciona stdout/stderr/both.
pub async fn get_command_logs(
&self,
workspace: WorkspaceId,
command: Ulid,
tail_bytes: usize,
stream: LogStream,
) -> Option<Vec<u8>> {
let g = self.inner.lock().await;
let ws = g.workspaces.get(&workspace)?;
let cmd = ws.commands.get(&command)?;
cmd.logs.as_ref().map(|lb| lb.tail(tail_bytes))
match stream {
LogStream::Stdout => cmd.stdout.as_ref().map(|lb| lb.tail(tail_bytes)),
LogStream::Stderr => cmd.stderr.as_ref().map(|lb| lb.tail(tail_bytes)),
LogStream::Both => {
let so = cmd.stdout.as_ref().map(|lb| lb.tail(tail_bytes)).unwrap_or_default();
let se = cmd.stderr.as_ref().map(|lb| lb.tail(tail_bytes)).unwrap_or_default();
let mut out = so;
out.extend_from_slice(&se);
Some(out)
}
}
}
/// Lista comandos de un workspace.
@@ -397,7 +622,8 @@ impl WorkspaceManager {
pid: c.pid.as_raw(),
alive: c.alive,
exit_status: c.exit_status,
log_bytes: c.logs.as_ref().map(|l| l.written_total()).unwrap_or(0),
log_bytes: c.stdout.as_ref().map(|l| l.written_total()).unwrap_or(0)
+ c.stderr.as_ref().map(|l| l.written_total()).unwrap_or(0),
})
.collect();
// Orden estable por ULID (temporal).
@@ -435,7 +661,9 @@ impl WorkspaceManager {
pid: out.pid,
alive: true,
exit_status: None,
logs: None, // run_pipeline NO captura logs (los conecta por pipes).
stdout: None, // run_pipeline NO captura (conecta por pipes).
stderr: None,
pipeline_id: None,
},
);
}
@@ -538,19 +766,16 @@ mod tests {
};
let (id, _) = mgr.create(spec).await.unwrap();
let summary = mgr
.run(
id,
"/bin/echo".into(),
vec!["captured-output".into()],
vec![],
)
.run(id, "/bin/echo".into(), vec!["captured-output".into()], vec![])
.await
.unwrap();
// Esperamos a que el comando termine y el drainer drene.
for _ in 0..50 {
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
mgr.reap_dead().await;
let logs = mgr.get_command_logs(id, summary.id, 0).await.unwrap_or_default();
let logs = mgr
.get_command_logs(id, summary.id, 0, LogStream::Stdout)
.await
.unwrap_or_default();
if !logs.is_empty() {
let s = String::from_utf8_lossy(&logs);
assert!(s.contains("captured-output"), "got: {s:?}");
@@ -560,6 +785,52 @@ mod tests {
panic!("logs never captured");
}
#[tokio::test]
async fn run_captures_stderr_separately() {
let mgr = Arc::new(WorkspaceManager::new(IncarnatorConfig::default()));
let spec = WorkspaceSpec {
label: "stderr".into(),
soma: Default::default(),
permissions: Default::default(),
ttl: None,
flow_dirs: vec![],
on_exit: shipote_card::ExitPolicy::Reap,
};
let (id, _) = mgr.create(spec).await.unwrap();
// sh -c "echo OUT; echo ERR >&2"
let summary = mgr
.run(
id,
"/bin/sh".into(),
vec!["-c".into(), "echo OUT; echo ERR >&2".into()],
vec![],
)
.await
.unwrap();
for _ in 0..50 {
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
mgr.reap_dead().await;
let so = mgr
.get_command_logs(id, summary.id, 0, LogStream::Stdout)
.await
.unwrap_or_default();
let se = mgr
.get_command_logs(id, summary.id, 0, LogStream::Stderr)
.await
.unwrap_or_default();
if !so.is_empty() && !se.is_empty() {
let so_s = String::from_utf8_lossy(&so);
let se_s = String::from_utf8_lossy(&se);
assert!(so_s.contains("OUT"), "stdout: {so_s:?}");
assert!(se_s.contains("ERR"), "stderr: {se_s:?}");
assert!(!so_s.contains("ERR"), "stdout no debería tener ERR");
assert!(!se_s.contains("OUT"), "stderr no debería tener OUT");
return;
}
}
panic!("logs never captured on both streams");
}
#[tokio::test]
async fn run_true_in_workspace() {
let mgr = Arc::new(WorkspaceManager::new(IncarnatorConfig::default()));