feat(shipote): data plane + DAG fan-in/out + stats + lifecycle (fases F-I)

Pipeline runtime:
- Fan-out 1→N (splitter task replica al N consumers) y fan-in N→1 (merger
  task con mpsc + reader-per-input). DAGs no lineales soportados.
- Flow channels: Unix socket + tokio broadcast con replay buffer
  configurable por pipeline (DiscernPolicy.replay_chunks). Subscribers
  externos vía `shipote flow tail <socket>`.
- Templating en specs con `${KEY}` (CLI `--var KEY=VALUE`). Walk
  recursivo sobre serde_json::Value, soporta todos los strings del schema.
- Pipelines guardados (`pipeline save/saved-list/drop/run-saved`)
  persisten con el snapshot.

Lifecycle de comandos:
- Log capture per-stream (stdout/stderr separados) via pipe O_CLOEXEC +
  AsyncFd. CLI `shipote logs <ws> <cmd> --stream {stdout,stderr,both}`.
- Stop graceful con tiempo configurable: SIGTERM → grace → SIGKILL.
  Tanto a nivel workspace como pipeline individual.
- TTL auto-stop ya existente (Fase C) sigue funcionando.

ente-incarnate:
- ChildStdio declarativo (Fase C) + ChildPreExec declarativo nuevo:
  NoNewPrivs, ParentDeathSig, Dumpable, NewSession, Chdir, Umask.
- Aplicación pre-execve async-signal-safe en ambos paths (plain via
  Command::pre_exec, namespaced via callback del clone(2)).

Observabilidad:
- WorkspaceStats: RSS + RSS peak (VmHWM o memory.peak cgroup) + CPU usec
  + uptime. Fuente per-proc o cgroup según delegation.
- shipote-shell con sparkline ASCII por workspace (history cap 24),
  card de flow channels activos, vista de comandos + saved pipelines.
- Tap → broker: cada edge enriquecido con TypeRef se anuncia como Card
  efímera vía SidecarPool (graceful si broker no corre).

Discern:
- Integrado en yahweh-provider-fs (mime_type en EntityNode).
- Integrado en nouser-core::cluster::pick_lens como fallback cuando la
  extensión cae a Lens::Grid.

79 tests pasan: ente-incarnate (16), nouser-core (27), shipote-card (8),
shipote-core (20), shipote-discern (5), yahweh-provider-fs (3).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
sergio
2026-05-11 00:29:46 +00:00
parent c22d2480b9
commit 36dac00c8d
13 changed files with 2187 additions and 253 deletions
@@ -12,7 +12,7 @@ use brahman_card::Payload;
use ente_incarnate::{ChildStdio, Incarnator};
use nix::fcntl::OFlag;
use nix::unistd::pipe2;
use shipote_card::{FlowEdge, PipelineSpec};
use shipote_card::PipelineSpec;
use shipote_discern::{DiscernPipeline, Discernment, Hint};
use std::os::fd::{AsRawFd, IntoRawFd, RawFd};
use std::sync::Arc;
@@ -22,7 +22,7 @@ use tracing::{debug, info, warn};
use ulid::Ulid;
/// Resultado de lanzar un pipeline.
#[derive(Debug, Clone)]
#[derive(Debug)]
pub struct PipelineLaunch {
pub pipeline: Ulid,
pub command_pids: Vec<(String, i32)>,
@@ -37,19 +37,29 @@ pub struct EdgeDiscernment {
pub to_label: String,
pub to_input: String,
pub discernment: Option<Discernment>,
/// Path del Unix socket donde otros módulos pueden suscribirse al
/// stream replicado por este edge. `None` cuando tap=false (no hay
/// data plane porque no hay sampling).
pub flow_socket: Option<std::path::PathBuf>,
}
/// Lanza un pipeline conectando nodos por stdin/stdout. Cada nodo se
/// encarna via `Incarnator` (con o sin namespacing según su SomaSpec).
///
/// v1: pipeline lineal (un edge entrante por nodo). Múltiples edges
/// entrantes generan warning y sólo el primero se honra.
/// Soporta:
/// - Pipeline lineal (1 producer → 1 consumer).
/// - **Fan-out** (1 producer → N consumers): shipote interpone un
/// splitter que duplica bytes a cada destino. Cuando `tap=true`, el
/// splitter además samplea para discernir.
/// - Múltiples predecessors por nodo NO se soporta aún (fan-in): sólo se
/// honra el primer edge entrante.
pub async fn run_pipeline(
spec: &PipelineSpec,
workspace_label: &str,
tap: bool,
discerner: Arc<DiscernPipeline>,
incarnator: Arc<Incarnator>,
manager: Option<Arc<crate::WorkspaceManager>>,
) -> Result<PipelineLaunch, CoreError> {
spec.validate()?;
let n = spec.nodes.len();
@@ -60,30 +70,100 @@ pub async fn run_pipeline(
"launching pipeline (incarnated)"
);
// Predecessor: para cada nodo, su edge entrante (si tiene).
let mut predecessor: Vec<Option<&FlowEdge>> = vec![None; n];
for e in &spec.edges {
if predecessor[e.to].is_some() {
warn!(node = e.to, "v1 pipeline: nodo con múltiples predecessors — sólo se honra el primero");
continue;
}
predecessor[e.to] = Some(e);
// Pre-compute grafo:
// - `consumers[i]` = índices de edges salientes de `i`.
// - `predecessors[j]` = índices de edges entrantes a `j`.
let mut consumers: Vec<Vec<usize>> = vec![Vec::new(); n];
let mut predecessors: Vec<Vec<usize>> = vec![Vec::new(); n];
for (idx, e) in spec.edges.iter().enumerate() {
consumers[e.from].push(idx);
predecessors[e.to].push(idx);
}
let mut pids = Vec::with_capacity(n);
let mut taps: Vec<TapHandle> = Vec::new();
// Para cada nodo i que produce, guardamos el FD de read del pipe
// del productor → al armar el consumidor lo consume.
// Pero como puede haber tap intermedio, llevamos un esquema:
// - Sin tap: read FD del pipe productor → stdin del consumidor.
// - Con tap: read FD del pipe productor → tokio proxy → write FD
// del pipe consumidor → stdin del consumidor.
// Para simplicidad lineal, `pending_stdin_for_next` guarda el FD que
// el siguiente consumidor debe usar como stdin.
let mut pending_stdin_for_next: Option<RawFd> = None;
// Por cada edge: par (r_to_consumer, w_from_producer_side).
// El consumer recibe r_to_consumer; el producer escribe a w_from_producer_side
// (directa o vía splitter).
let mut edge_r: Vec<RawFd> = vec![-1; spec.edges.len()];
let mut edge_w: Vec<RawFd> = vec![-1; spec.edges.len()];
for i in 0..spec.edges.len() {
let (r, w) = pipe2(OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
edge_r[i] = r.into_raw_fd();
edge_w[i] = w.into_raw_fd();
}
let mut consumer_stdin_fd: Vec<Option<RawFd>> = vec![None; n];
let mut producer_stdout_fd: Vec<Option<RawFd>> = vec![None; n];
let mut splitter_specs: Vec<SplitterSpec> = Vec::new();
let mut merger_specs: Vec<MergerSpec> = Vec::new();
// Stdout del producer: directo a edge_w[único] si tiene 1 consumer y NO tap;
// sino, pipe propio que va al splitter task.
for i in 0..n {
if consumers[i].is_empty() {
continue;
}
if consumers[i].len() == 1 && !tap {
producer_stdout_fd[i] = Some(edge_w[consumers[i][0]]);
continue;
}
// Splitter: pipe propio para el productor → splitter lee y replica a edge_w[*].
let (prod_r, prod_w) = pipe2(OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
producer_stdout_fd[i] = Some(prod_w.into_raw_fd());
let prod_r_fd = prod_r.into_raw_fd();
let mut consumer_writes: Vec<RawFd> = Vec::with_capacity(consumers[i].len());
let mut edge_meta: Vec<EdgeMeta> = Vec::with_capacity(consumers[i].len());
for edge_idx in &consumers[i] {
let edge = &spec.edges[*edge_idx];
consumer_writes.push(edge_w[*edge_idx]);
edge_meta.push(EdgeMeta {
from_label: spec.nodes[edge.from].label.clone(),
from_output: edge.from_output.clone(),
to_label: spec.nodes[edge.to].label.clone(),
to_input: edge.to_input.clone(),
});
}
splitter_specs.push(SplitterSpec {
producer_r_fd: prod_r_fd,
consumer_w_fds: consumer_writes,
edges: edge_meta,
tap,
sample_bytes: spec.discern.sample_bytes,
});
}
// Stdin del consumer: edge_r[único] si tiene 1 predecessor; sino, merger.
for j in 0..n {
match predecessors[j].len() {
0 => {}
1 => {
consumer_stdin_fd[j] = Some(edge_r[predecessors[j][0]]);
}
_ => {
// Merger: lee de N edge_r y escribe a un nuevo pipe cuyo
// read end es el stdin del consumer.
let (cons_r, cons_w) = pipe2(OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
consumer_stdin_fd[j] = Some(cons_r.into_raw_fd());
let inputs: Vec<RawFd> = predecessors[j]
.iter()
.map(|eidx| edge_r[*eidx])
.collect();
merger_specs.push(MergerSpec {
producer_r_fds: inputs,
consumer_w_fd: cons_w.into_raw_fd(),
});
}
}
}
// Encarnamos cada nodo con su stdin/stdout fd asignado.
let mut pids = Vec::with_capacity(n);
for (i, node) in spec.nodes.iter().enumerate() {
// Validar payload ejecutable.
match &node.payload {
Payload::Native { .. } | Payload::Legacy { .. } => {}
_ => {
@@ -92,91 +172,98 @@ pub async fn run_pipeline(
))
}
}
// Compilamos a Card.
let card = node.to_card(i, workspace_label)?;
// ¿Soy productor? Necesito stdout_fd hacia un pipe nuevo.
let i_is_producer = spec.edges.iter().any(|e| e.from == i);
let stdin_fd: Option<RawFd> = pending_stdin_for_next.take();
let mut stdout_fd: Option<RawFd> = None;
let mut next_pending: Option<RawFd> = None;
// FDs que el PADRE debe cerrar tras spawn (son nuestra copia del
// extremo que pasamos al hijo).
let mut parent_closes: Vec<RawFd> = Vec::new();
if i_is_producer {
let (r, w) = pipe2(OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
let r_raw = r.into_raw_fd();
let w_raw = w.into_raw_fd();
stdout_fd = Some(w_raw);
parent_closes.push(w_raw);
if tap {
// Necesitamos un segundo pipe entre tap y consumidor.
let (r2, w2) = pipe2(OFlag::O_CLOEXEC).map_err(|e| {
CoreError::Incarnate(ente_incarnate::IncarnateError::Pipe(e))
})?;
let r2_raw = r2.into_raw_fd();
let w2_raw = w2.into_raw_fd();
next_pending = Some(r2_raw);
// El tap lee de r_raw y escribe a w2_raw.
let edge = predecessor
.iter()
.find_map(|p| *p)
.and_then(|e| if e.from == i { Some(e) } else { None })
// Edge donde i es from:
.or_else(|| spec.edges.iter().find(|e| e.from == i));
let from_label = node.label.clone();
let to_label = edge
.map(|e| spec.nodes[e.to].label.clone())
.unwrap_or_default();
let from_output = edge.map(|e| e.from_output.clone()).unwrap_or_default();
let to_input = edge.map(|e| e.to_input.clone()).unwrap_or_default();
let sample_bytes = spec.discern.sample_bytes;
let disc = discerner.clone();
let h = spawn_tap(
r_raw, w2_raw, sample_bytes, disc, from_label, from_output, to_label, to_input,
);
taps.push(h);
// r_raw y w2_raw pasaron a manos del tokio task. No los
// cerramos en el padre.
} else {
// Sin tap, el read del productor va directo al stdin del
// siguiente consumidor.
next_pending = Some(r_raw);
}
}
let stdio = ChildStdio {
stdin_fd,
stdout_fd,
stdin_fd: consumer_stdin_fd[i],
stdout_fd: producer_stdout_fd[i],
stderr_fd: None,
};
// Incarnator absorbe los fds de `stdio` — no los cerramos acá.
// `parent_closes` queda obsoleto.
let _ = parent_closes;
let outcome = incarnator
.incarnate_with(&card, stdio)
.map_err(CoreError::Incarnate)?;
let pid = outcome.pid;
pids.push((node.label.clone(), pid.as_raw()));
debug!(label = %node.label, pid = pid.as_raw(), "node incarnated");
pending_stdin_for_next = next_pending;
}
let pipeline_id = Ulid::new();
let pipeline_id_for_flows = Ulid::new();
// Si tap=true, creamos un FlowChannel por edge para el data plane.
// Cada splitter pushea al sender del channel correspondiente.
let pipeline_id = pipeline_id_for_flows;
let mut flow_channels: Vec<crate::flow_channel::FlowChannel> = Vec::new();
let mut splitter_channels: Vec<Vec<Option<crate::flow_channel::FlowSender>>> =
Vec::with_capacity(splitter_specs.len());
let mut edge_socket_for_splitter: Vec<Vec<Option<std::path::PathBuf>>> = Vec::new();
for s in &splitter_specs {
let mut senders_per_edge = Vec::with_capacity(s.edges.len());
let mut paths_per_edge = Vec::with_capacity(s.edges.len());
for (i, em) in s.edges.iter().enumerate() {
if !s.tap {
senders_per_edge.push(None);
paths_per_edge.push(None);
continue;
}
let id = format!(
"{}-{}-{}-{}",
short_ulid(&pipeline_id),
em.from_label,
em.from_output,
i
);
let socket = crate::flow_channel::default_flow_socket_path(&id);
match crate::flow_channel::FlowChannel::with_replay_cap(socket.clone(), spec.discern.replay_chunks) {
Ok(fc) => {
senders_per_edge.push(Some(fc.sender_handle()));
paths_per_edge.push(Some(socket));
flow_channels.push(fc);
}
Err(e) => {
warn!(?e, "flow channel new failed");
senders_per_edge.push(None);
paths_per_edge.push(None);
}
}
}
splitter_channels.push(senders_per_edge);
edge_socket_for_splitter.push(paths_per_edge);
}
let mut edge_discernments = Vec::with_capacity(taps.len());
for t in taps {
match t.handle.await {
Ok(d) => edge_discernments.push(d),
Err(e) => warn!(?e, "tap handle joined with error"),
// Registramos los flow_channels en el manager AHORA, antes de await
// las tasks. Esto permite que clientes externos hagan `flow list` y
// se suscriban mientras el pipeline aún produce data.
if let Some(mgr) = &manager {
if !flow_channels.is_empty() {
let drained: Vec<crate::flow_channel::FlowChannel> = flow_channels.drain(..).collect();
mgr.retain_pipeline_flows(pipeline_id, drained).await;
}
}
// Spawn mergers + splitters después del incarnate. Cada task posee
// sus fds y los cierra al terminar (via Drop de OwnedFd).
let mut merger_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
for m in merger_specs {
merger_handles.push(spawn_merger(m));
}
let mut tap_handles: Vec<SplitterHandle> = Vec::new();
for (s, senders) in splitter_specs.into_iter().zip(splitter_channels.into_iter()) {
tap_handles.push(spawn_splitter(s, discerner.clone(), senders));
}
let mut edge_discernments = Vec::new();
for (h, paths) in tap_handles.into_iter().zip(edge_socket_for_splitter.into_iter()) {
match h.handle.await {
Ok(eds) => {
for (mut ed, path) in eds.into_iter().zip(paths.into_iter()) {
ed.flow_socket = path;
edge_discernments.push(ed);
}
}
Err(e) => warn!(?e, "splitter handle joined with error"),
}
}
for h in merger_handles {
if let Err(e) = h.await {
warn!(?e, "merger handle joined with error");
}
}
@@ -187,57 +274,156 @@ pub async fn run_pipeline(
})
}
struct TapHandle {
handle: tokio::task::JoinHandle<EdgeDiscernment>,
fn short_ulid(u: &Ulid) -> String {
let s = u.to_string();
s[s.len() - 6..].to_string()
}
#[allow(clippy::too_many_arguments)]
fn spawn_tap(
producer_r_fd: RawFd,
consumer_w_fd: RawFd,
sample_bytes: usize,
discerner: Arc<DiscernPipeline>,
#[derive(Debug, Clone)]
struct EdgeMeta {
from_label: String,
from_output: String,
to_label: String,
to_input: String,
) -> TapHandle {
// Marcar non-blocking ANTES de envolverlos en AsyncFd. Sino tokio
// bloquea el reactor en operaciones lentas.
set_nonblocking(producer_r_fd);
set_nonblocking(consumer_w_fd);
}
struct SplitterSpec {
producer_r_fd: RawFd,
consumer_w_fds: Vec<RawFd>,
edges: Vec<EdgeMeta>,
tap: bool,
sample_bytes: usize,
}
struct SplitterHandle {
handle: tokio::task::JoinHandle<Vec<EdgeDiscernment>>,
}
struct MergerSpec {
producer_r_fds: Vec<RawFd>,
consumer_w_fd: RawFd,
}
fn spawn_merger(spec: MergerSpec) -> tokio::task::JoinHandle<()> {
for fd in &spec.producer_r_fds {
set_nonblocking(*fd);
}
set_nonblocking(spec.consumer_w_fd);
// Patrón: una task lectora por cada producer reenvía bytes a un mpsc.
// El merger principal consume del mpsc y escribe al consumer.
// Esto evita el "block en reader idle" del enfoque round-robin sobre
// AsyncFd::ready() (los readers idle nunca dejan turno).
tokio::spawn(async move {
let (tx, mut rx) = tokio::sync::mpsc::channel::<Vec<u8>>(32);
let nr = spec.producer_r_fds.len();
for fd in spec.producer_r_fds {
let tx = tx.clone();
tokio::spawn(async move {
// SAFETY: ownership transferida.
let owned = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(fd) };
let r = match AsyncFd::with_interest(owned, Interest::READABLE) {
Ok(a) => a,
Err(e) => {
warn!(?e, "merger reader AsyncFd");
return;
}
};
let mut buf = [0u8; 4096];
loop {
match async_read(&r, &mut buf).await {
Ok(0) => break,
Ok(n) => {
if tx.send(buf[..n].to_vec()).await.is_err() {
break;
}
}
Err(_) => break,
}
}
// Drop de tx → cuando todos los readers cerraron, el rx
// recibe None y el merger termina.
});
}
drop(tx); // sólo los reader tasks tienen sus clones ahora.
// SAFETY: ownership transferida al task.
let w_owned = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(spec.consumer_w_fd) };
let w = match AsyncFd::with_interest(w_owned, Interest::WRITABLE) {
Ok(a) => a,
Err(e) => {
warn!(?e, "merger AsyncFd w");
return;
}
};
let mut total: u64 = 0;
while let Some(chunk) = rx.recv().await {
if async_write_all(&w, &chunk).await.is_err() {
return;
}
total += chunk.len() as u64;
}
debug!(bytes = total, readers = nr, "merger finished");
})
}
fn spawn_splitter(
spec: SplitterSpec,
discerner: Arc<DiscernPipeline>,
edge_senders: Vec<Option<crate::flow_channel::FlowSender>>,
) -> SplitterHandle {
set_nonblocking(spec.producer_r_fd);
for fd in &spec.consumer_w_fds {
set_nonblocking(*fd);
}
let handle = tokio::spawn(async move {
// SAFETY: el caller transfiere ownership de los fds al task.
let r_std = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(producer_r_fd) };
let w_std = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(consumer_w_fd) };
let r = AsyncFd::with_interest(r_std, Interest::READABLE).expect("AsyncFd r");
let w = AsyncFd::with_interest(w_std, Interest::WRITABLE).expect("AsyncFd w");
// SAFETY: ownership transferida al task.
let r_owned = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(spec.producer_r_fd) };
let r = match AsyncFd::with_interest(r_owned, Interest::READABLE) {
Ok(a) => a,
Err(e) => {
warn!(?e, "splitter AsyncFd r");
return Vec::new();
}
};
let mut writers: Vec<AsyncFd<std::os::fd::OwnedFd>> = Vec::with_capacity(spec.consumer_w_fds.len());
for fd in spec.consumer_w_fds {
let owned = unsafe { std::os::fd::OwnedFd::from_raw_fd_compat(fd) };
match AsyncFd::with_interest(owned, Interest::WRITABLE) {
Ok(a) => writers.push(a),
Err(e) => warn!(?e, "splitter AsyncFd w"),
}
}
let mut sample: Vec<u8> = Vec::with_capacity(sample_bytes);
let mut sample: Vec<u8> = Vec::with_capacity(spec.sample_bytes);
let mut buf = [0u8; 4096];
let mut total: u64 = 0;
// Fase 1: sampling + pump.
let mut eof = false;
while !eof && sample.len() < sample_bytes {
// Fase 1: sampling (sólo si tap=true) + replicación.
while !eof && (spec.tap && sample.len() < spec.sample_bytes) {
let n = match async_read(&r, &mut buf).await {
Ok(0) => { eof = true; 0 }
Ok(n) => n,
Err(e) => { warn!(?e, "tap producer read failed"); break; }
Err(e) => { warn!(?e, "splitter read"); break; }
};
if n == 0 { break; }
let take = n.min(sample_bytes - sample.len());
sample.extend_from_slice(&buf[..take]);
if let Err(e) = async_write_all(&w, &buf[..n]).await {
warn!(?e, "tap consumer write failed");
break;
if spec.tap {
let take = n.min(spec.sample_bytes - sample.len());
sample.extend_from_slice(&buf[..take]);
}
broadcast_chunk(&writers, &edge_senders, &buf[..n]).await;
total += n as u64;
}
let d = discerner.discern(&sample, &Hint { path: None, size_total: None });
// Fase 2: pump-only hasta EOF.
let d = if spec.tap {
discerner.discern(&sample, &Hint { path: None, size_total: None })
} else {
None
};
// Fase 2: replicación pura.
while !eof {
let n = match async_read(&r, &mut buf).await {
Ok(0) => { eof = true; 0 }
@@ -245,19 +431,50 @@ fn spawn_tap(
Err(_) => break,
};
if n == 0 { break; }
if async_write_all(&w, &buf[..n]).await.is_err() { break; }
broadcast_chunk(&writers, &edge_senders, &buf[..n]).await;
total += n as u64;
}
debug!(bytes = total, "tap finished");
EdgeDiscernment {
from_label,
from_output,
to_label,
to_input,
discernment: d,
}
debug!(bytes = total, consumers = writers.len(), "splitter finished");
// Mismo discernment para todos los edges del splitter (es el mismo
// stream replicado). Devolvemos N entries (una por edge) para que
// la UI/CLI los liste todos. flow_socket lo rellena el caller.
spec.edges
.into_iter()
.map(|em| EdgeDiscernment {
from_label: em.from_label,
from_output: em.from_output,
to_label: em.to_label,
to_input: em.to_input,
discernment: d.clone(),
flow_socket: None,
})
.collect()
});
TapHandle { handle }
SplitterHandle { handle }
}
async fn broadcast_chunk(
writers: &[AsyncFd<std::os::fd::OwnedFd>],
edge_senders: &[Option<crate::flow_channel::FlowSender>],
data: &[u8],
) {
// Internal pipes a los consumers del pipeline.
for w in writers {
let _ = async_write_all(w, data).await;
}
// Externos: broadcast a subscribers vía FlowChannel.
// Cada edge tiene su propio sender (mismo data — el sample/discernment
// viaja por broadcast separados para que un subscriber por edge vea su
// stream específico).
if edge_senders.iter().any(|s| s.is_some()) {
let shared = std::sync::Arc::new(data.to_vec());
for s in edge_senders {
if let Some(s) = s {
let _ = s.send(shared.clone());
}
}
}
}
async fn async_read(
@@ -377,7 +594,7 @@ mod tests {
};
let disc = Arc::new(DiscernPipeline::default_pipeline());
let inc = Arc::new(Incarnator::new(IncarnatorConfig::default()));
let launch = run_pipeline(&spec, "ws", false, disc, inc).await.unwrap();
let launch = run_pipeline(&spec, "ws", false, disc, inc, None).await.unwrap();
assert_eq!(launch.command_pids.len(), 2);
// Cosecha.
for (_, pid) in &launch.command_pids {
@@ -385,6 +602,78 @@ mod tests {
}
}
#[tokio::test]
async fn pipeline_fanin_two_to_one() {
// 2 productores → 1 consumer (cat). El merger multiplexa.
let spec = PipelineSpec {
label: "fanin".into(),
workspace: WorkspaceId::new(),
nodes: vec![
cmd("p1", "/bin/echo", &["from-p1"]),
cmd("p2", "/bin/echo", &["from-p2"]),
cmd("c", "/bin/cat", &[]),
],
edges: vec![
FlowEdge {
from: 0,
from_output: "stdout".into(),
to: 2,
to_input: "stdin".into(),
},
FlowEdge {
from: 1,
from_output: "stdout".into(),
to: 2,
to_input: "stdin".into(),
},
],
discern: DiscernPolicy::default(),
};
let disc = Arc::new(DiscernPipeline::default_pipeline());
let inc = Arc::new(Incarnator::new(IncarnatorConfig::default()));
let launch = run_pipeline(&spec, "ws", false, disc, inc, None).await.unwrap();
assert_eq!(launch.command_pids.len(), 3);
for (_, pid) in &launch.command_pids {
let _ = nix::sys::wait::waitpid(nix::unistd::Pid::from_raw(*pid), None);
}
}
#[tokio::test]
async fn pipeline_fanout_one_to_two() {
// 1 productor (echo) → 2 consumers (wc -c). Splitter replica.
let spec = PipelineSpec {
label: "fanout".into(),
workspace: WorkspaceId::new(),
nodes: vec![
cmd("p", "/bin/echo", &["fanout-test"]),
cmd("c1", "/bin/cat", &[]),
cmd("c2", "/bin/cat", &[]),
],
edges: vec![
FlowEdge {
from: 0,
from_output: "stdout".into(),
to: 1,
to_input: "stdin".into(),
},
FlowEdge {
from: 0,
from_output: "stdout".into(),
to: 2,
to_input: "stdin".into(),
},
],
discern: DiscernPolicy::default(),
};
let disc = Arc::new(DiscernPipeline::default_pipeline());
let inc = Arc::new(Incarnator::new(IncarnatorConfig::default()));
let launch = run_pipeline(&spec, "ws", false, disc, inc, None).await.unwrap();
assert_eq!(launch.command_pids.len(), 3);
for (_, pid) in &launch.command_pids {
let _ = nix::sys::wait::waitpid(nix::unistd::Pid::from_raw(*pid), None);
}
}
#[tokio::test]
async fn pipeline_isolated_with_tap_captures_discernment() {
let spec = PipelineSpec {
@@ -403,11 +692,12 @@ mod tests {
discern: DiscernPolicy {
sample_bytes: 4096,
enrich_producer: true,
replay_chunks: 32,
},
};
let disc = Arc::new(DiscernPipeline::default_pipeline());
let inc = Arc::new(Incarnator::new(IncarnatorConfig::default()));
let launch = run_pipeline(&spec, "ws", true, disc, inc).await.unwrap();
let launch = run_pipeline(&spec, "ws", true, disc, inc, None).await.unwrap();
assert_eq!(launch.edge_discernments.len(), 1);
let d = &launch.edge_discernments[0];
let dis = d.discernment.as_ref().expect("discernment present");