feat(shipote): data plane + DAG fan-in/out + stats + lifecycle (fases F-I)

Pipeline runtime:
- Fan-out 1→N (splitter task replica al N consumers) y fan-in N→1 (merger
  task con mpsc + reader-per-input). DAGs no lineales soportados.
- Flow channels: Unix socket + tokio broadcast con replay buffer
  configurable por pipeline (DiscernPolicy.replay_chunks). Subscribers
  externos vía `shipote flow tail <socket>`.
- Templating en specs con `${KEY}` (CLI `--var KEY=VALUE`). Walk
  recursivo sobre serde_json::Value, soporta todos los strings del schema.
- Pipelines guardados (`pipeline save/saved-list/drop/run-saved`)
  persisten con el snapshot.

Lifecycle de comandos:
- Log capture per-stream (stdout/stderr separados) via pipe O_CLOEXEC +
  AsyncFd. CLI `shipote logs <ws> <cmd> --stream {stdout,stderr,both}`.
- Stop graceful con tiempo configurable: SIGTERM → grace → SIGKILL.
  Tanto a nivel workspace como pipeline individual.
- TTL auto-stop ya existente (Fase C) sigue funcionando.

ente-incarnate:
- ChildStdio declarativo (Fase C) + ChildPreExec declarativo nuevo:
  NoNewPrivs, ParentDeathSig, Dumpable, NewSession, Chdir, Umask.
- Aplicación pre-execve async-signal-safe en ambos paths (plain via
  Command::pre_exec, namespaced via callback del clone(2)).

Observabilidad:
- WorkspaceStats: RSS + RSS peak (VmHWM o memory.peak cgroup) + CPU usec
  + uptime. Fuente per-proc o cgroup según delegation.
- shipote-shell con sparkline ASCII por workspace (history cap 24),
  card de flow channels activos, vista de comandos + saved pipelines.
- Tap → broker: cada edge enriquecido con TypeRef se anuncia como Card
  efímera vía SidecarPool (graceful si broker no corre).

Discern:
- Integrado en yahweh-provider-fs (mime_type en EntityNode).
- Integrado en nouser-core::cluster::pick_lens como fallback cuando la
  extensión cae a Lens::Grid.

79 tests pasan: ente-incarnate (16), nouser-core (27), shipote-card (8),
shipote-core (20), shipote-discern (5), yahweh-provider-fs (3).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
sergio
2026-05-11 00:29:46 +00:00
parent c22d2480b9
commit 36dac00c8d
13 changed files with 2187 additions and 253 deletions
+129 -1
View File
@@ -203,7 +203,7 @@ pub struct FlowEdge {
pub to_input: String,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DiscernPolicy {
/// Bytes a samplear por flow para el discernidor. Default 4 KiB.
#[serde(default = "default_sample_bytes")]
@@ -211,6 +211,21 @@ pub struct DiscernPolicy {
/// Si `true`, enriquece la Card del producer con el TypeRef detectado.
#[serde(default = "default_true")]
pub enrich_producer: bool,
/// Chunks que el FlowChannel guarda en replay buffer para subscribers
/// tarde. Default 32. Subir si los productores escriben en ráfagas y
/// querés que los consumidores tardíos vean toda la salida.
#[serde(default = "default_replay_chunks")]
pub replay_chunks: usize,
}
impl Default for DiscernPolicy {
fn default() -> Self {
Self {
sample_bytes: default_sample_bytes(),
enrich_producer: default_true(),
replay_chunks: default_replay_chunks(),
}
}
}
fn default_sample_bytes() -> usize {
@@ -219,6 +234,9 @@ fn default_sample_bytes() -> usize {
fn default_true() -> bool {
true
}
fn default_replay_chunks() -> usize {
32
}
// =====================================================================
// Compilación a Card
@@ -358,6 +376,116 @@ pub fn load_pipeline_spec(path: &std::path::Path) -> Result<PipelineSpec, LoadEr
}
}
/// Sustituye `${KEY}` en todos los strings del spec por el valor de
/// `vars["KEY"]`. Variables sin match quedan intactas (no se borra el
/// placeholder — útil para detectar olvidos).
///
/// Walk recursivo sobre la representación JSON intermedia para cubrir
/// labels, argv, envp, paths y cualquier String del schema.
pub fn substitute_vars(
spec: &PipelineSpec,
vars: &std::collections::HashMap<String, String>,
) -> Result<PipelineSpec, serde_json::Error> {
if vars.is_empty() {
return Ok(spec.clone());
}
let mut v = serde_json::to_value(spec)?;
walk_subst(&mut v, vars);
serde_json::from_value(v)
}
fn walk_subst(v: &mut serde_json::Value, vars: &std::collections::HashMap<String, String>) {
match v {
serde_json::Value::String(s) => {
*s = subst_str(s, vars);
}
serde_json::Value::Array(arr) => {
for item in arr {
walk_subst(item, vars);
}
}
serde_json::Value::Object(obj) => {
for (_, val) in obj.iter_mut() {
walk_subst(val, vars);
}
}
_ => {}
}
}
fn subst_str(s: &str, vars: &std::collections::HashMap<String, String>) -> String {
let mut out = String::with_capacity(s.len());
let bytes = s.as_bytes();
let mut i = 0;
while i < bytes.len() {
if i + 1 < bytes.len() && bytes[i] == b'$' && bytes[i + 1] == b'{' {
// Buscar el cierre `}`.
if let Some(close) = bytes[i + 2..].iter().position(|&b| b == b'}') {
let key = std::str::from_utf8(&bytes[i + 2..i + 2 + close]).unwrap_or("");
if let Some(val) = vars.get(key) {
out.push_str(val);
i += 2 + close + 1;
continue;
}
}
}
out.push(bytes[i] as char);
i += 1;
}
out
}
#[cfg(test)]
mod subst_tests {
use super::*;
use std::collections::HashMap;
#[test]
fn substitute_in_argv_and_label() {
let mut vars = HashMap::new();
vars.insert("MSG".into(), "hola-mundo".into());
vars.insert("LABEL".into(), "renamed".into());
let spec = PipelineSpec {
label: "p-${LABEL}".into(),
workspace: WorkspaceId::new(),
nodes: vec![CommandRef {
label: "node-${LABEL}".into(),
payload: Payload::Native {
exec: "/bin/echo".into(),
argv: vec!["${MSG}".into()],
envp: vec![],
},
soma: Default::default(),
flows: Default::default(),
supervision: Supervision::OneShot,
}],
edges: vec![],
discern: DiscernPolicy::default(),
};
let out = substitute_vars(&spec, &vars).unwrap();
assert_eq!(out.label, "p-renamed");
assert_eq!(out.nodes[0].label, "node-renamed");
match &out.nodes[0].payload {
Payload::Native { argv, .. } => assert_eq!(argv[0], "hola-mundo"),
_ => panic!("wrong payload"),
}
}
#[test]
fn unknown_var_left_intact() {
let vars = HashMap::new();
let spec = PipelineSpec {
label: "p-${UNDEFINED}".into(),
workspace: WorkspaceId::new(),
nodes: vec![],
edges: vec![],
discern: DiscernPolicy::default(),
};
let out = substitute_vars(&spec, &vars).unwrap();
assert_eq!(out.label, "p-${UNDEFINED}");
}
}
#[cfg(test)]
mod tests {
use super::*;