tawasuyu/brahman
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(shipote): data plane + DAG fan-in/out + stats + lifecycle (fases F-I)

Browse Source 
Pipeline runtime:
- Fan-out 1→N (splitter task replica al N consumers) y fan-in N→1 (merger
  task con mpsc + reader-per-input). DAGs no lineales soportados.
- Flow channels: Unix socket + tokio broadcast con replay buffer
  configurable por pipeline (DiscernPolicy.replay_chunks). Subscribers
  externos vía `shipote flow tail <socket>`.
- Templating en specs con `${KEY}` (CLI `--var KEY=VALUE`). Walk
  recursivo sobre serde_json::Value, soporta todos los strings del schema.
- Pipelines guardados (`pipeline save/saved-list/drop/run-saved`)
  persisten con el snapshot.

Lifecycle de comandos:
- Log capture per-stream (stdout/stderr separados) via pipe O_CLOEXEC +
  AsyncFd. CLI `shipote logs <ws> <cmd> --stream {stdout,stderr,both}`.
- Stop graceful con tiempo configurable: SIGTERM → grace → SIGKILL.
  Tanto a nivel workspace como pipeline individual.
- TTL auto-stop ya existente (Fase C) sigue funcionando.

ente-incarnate:
- ChildStdio declarativo (Fase C) + ChildPreExec declarativo nuevo:
  NoNewPrivs, ParentDeathSig, Dumpable, NewSession, Chdir, Umask.
- Aplicación pre-execve async-signal-safe en ambos paths (plain via
  Command::pre_exec, namespaced via callback del clone(2)).

Observabilidad:
- WorkspaceStats: RSS + RSS peak (VmHWM o memory.peak cgroup) + CPU usec
  + uptime. Fuente per-proc o cgroup según delegation.
- shipote-shell con sparkline ASCII por workspace (history cap 24),
  card de flow channels activos, vista de comandos + saved pipelines.
- Tap → broker: cada edge enriquecido con TypeRef se anuncia como Card
  efímera vía SidecarPool (graceful si broker no corre).

Discern:
- Integrado en yahweh-provider-fs (mime_type en EntityNode).
- Integrado en nouser-core::cluster::pick_lens como fallback cuando la
  extensión cae a Lens::Grid.

79 tests pasan: ente-incarnate (16), nouser-core (27), shipote-card (8),
shipote-core (20), shipote-discern (5), yahweh-provider-fs (3).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
sergio
2026-05-11 00:29:46 +00:00
parent c22d2480b9
commit 36dac00c8d
13 changed files with 2187 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/apps/shipote-daemon/src/main.rs
+179 -52
View File
@@ -17,7 +17,7 @@ use shipote_core::WorkspaceManager;
use shipote_discern::{DiscernPipeline, Hint};
use shipote_protocol::{
default_socket_path, read_frame, write_frame, CommandInfo as ProtoCommandInfo,
EdgeDiscernmentInfo, Request, Response, WorkspaceSummary,
EdgeDiscernmentInfo, FlowInfo, Request, Response, WorkspaceStatsInfo, WorkspaceSummary,
};
use std::sync::Arc;
use tokio::net::{UnixListener, UnixStream};
@@ -38,10 +38,18 @@ async fn main() -> anyhow::Result<()> {
let listener = UnixListener::bind(&sock).with_context(|| format!("bind {}", sock.display()))?;
info!(socket = %sock.display(), "shipote-daemon listening");
// Sidecar al broker: shipote se anuncia como sesión. Si el Init no
// está corriendo, el sidecar loguea y termina; el daemon sigue
// standalone (UX de v1: ningún feature requiere broker).
brahman_sidecar::spawn(build_daemon_card(&sock));
// Sidecar pool: una sesión global del daemon + N sesiones efímeras
// por edge enriquecido tras cada pipeline tap.
let sidecar_pool = match brahman_sidecar::SidecarPool::new() {
Ok(p) => Some(Arc::new(p)),
Err(e) => {
warn!(?e, "SidecarPool falló — broker integration disabled");
None
}
};
if let Some(pool) = &sidecar_pool {
pool.spawn(build_daemon_card(&sock));
}
let mgr = Arc::new(WorkspaceManager::new(IncarnatorConfig {
// El daemon aún no se conecta al broker; cuando lo haga, este path
@@ -101,8 +109,9 @@ async fn main() -> anyhow::Result<()> {
Ok((stream, _)) => {
let mgr = mgr.clone();
let disc = discerner.clone();
let pool = sidecar_pool.clone();
tokio::spawn(async move {
if let Err(e) = handle_client(stream, mgr, disc).await {
if let Err(e) = handle_client(stream, mgr, disc, pool).await {
warn!(?e, "client handler error");
}
});
@@ -119,6 +128,7 @@ async fn handle_client(
mut stream: UnixStream,
mgr: Arc<WorkspaceManager>,
disc: Arc<DiscernPipeline>,
pool: Option<Arc<brahman_sidecar::SidecarPool>>,
) -> anyhow::Result<()> {
loop {
let req: Request = match read_frame(&mut stream).await {
@@ -126,12 +136,17 @@ async fn handle_client(
Err(shipote_protocol::ProtocolError::Closed) => return Ok(()),
Err(e) => return Err(e.into()),
};
let resp = dispatch(&mgr, &disc, req).await;
let resp = dispatch(&mgr, &disc, &pool, req).await;
write_frame(&mut stream, &resp).await?;
}
}
async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Request) -> Response {
async fn dispatch(
mgr: &Arc<WorkspaceManager>,
disc: &DiscernPipeline,
pool: &Option<Arc<brahman_sidecar::SidecarPool>>,
req: Request,
) -> Response {
match req {
Request::Ping => Response::Pong,
@@ -155,10 +170,15 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
Response::WorkspaceList { items }
}
Request::WorkspaceStop { id } => match mgr.stop(id).await {
Ok(reaped) => Response::WorkspaceStopped { id, reaped },
Err(e) => Response::Error { message: format!("{e}") },
},
Request::WorkspaceStop { id, grace_ms } => {
match mgr
.stop_with_grace(id, std::time::Duration::from_millis(grace_ms))
.await
{
Ok(reaped) => Response::WorkspaceStopped { id, reaped },
Err(e) => Response::Error { message: format!("{e}") },
}
}
Request::Run { workspace, exec, argv, envp } => {
match mgr.run(workspace, exec, argv, envp).await {
@@ -171,7 +191,12 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
}
}
Request::PipelineRun { spec, tap } => {
Request::PipelineRun { spec, tap, vars } => {
let vars_map: std::collections::HashMap<String, String> = vars.into_iter().collect();
let spec = match shipote_card::substitute_vars(&spec, &vars_map) {
Ok(s) => s,
Err(e) => return Response::Error { message: format!("template: {e}") },
};
let disc = DiscernPipeline::default_pipeline();
let inc = mgr.incarnator_handle();
let ws_label = mgr.workspace_label(spec.workspace).await.unwrap_or_default();
@@ -181,27 +206,22 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
tap,
std::sync::Arc::new(disc),
inc,
Some(mgr.clone()),
)
.await
{
Ok(launch) => Response::PipelineStarted {
pipeline: launch.pipeline,
command_pids: launch.command_pids,
edges: launch
.edge_discernments
.into_iter()
.map(|e| EdgeDiscernmentInfo {
from_label: e.from_label,
from_output: e.from_output,
to_label: e.to_label,
to_input: e.to_input,
ty: e.discernment.as_ref().map(|d| format!("{:?}", d.ty)),
mime: e.discernment.as_ref().and_then(|d| d.mime.clone()),
lens: e.discernment.as_ref().and_then(|d| d.lens.clone()),
confidence: e.discernment.as_ref().map(|d| d.confidence).unwrap_or(0.0),
})
.collect(),
},
Ok(launch) => {
let pipeline_id = launch.pipeline;
announce_edges_to_broker(pool.as_deref(), &pipeline_id, &launch.edge_discernments);
let cmds = launch.command_pids;
mgr.register_pipeline_commands(spec.workspace, pipeline_id, cmds.clone()).await;
let edges = launch.edge_discernments.into_iter().map(map_edge_to_info).collect();
Response::PipelineStarted {
pipeline: pipeline_id,
command_pids: cmds,
edges,
}
}
Err(e) => Response::Error { message: format!("{e}") },
}
}
@@ -240,8 +260,13 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
Response::CommandList { items }
}
Request::CommandLogs { workspace, command, tail_bytes } => {
match mgr.get_command_logs(workspace, command, tail_bytes).await {
Request::CommandLogs { workspace, command, tail_bytes, stream } => {
let s = match stream.as_str() {
"stdout" => shipote_core::LogStream::Stdout,
"stderr" => shipote_core::LogStream::Stderr,
_ => shipote_core::LogStream::Both,
};
match mgr.get_command_logs(workspace, command, tail_bytes, s).await {
Some(bytes) => Response::CommandLogs { bytes },
None => Response::Error {
message: format!("no logs for command {command} in workspace {workspace}"),
@@ -264,8 +289,13 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
Response::PipelineDropped { name, existed }
}
Request::PipelineRunSaved { name, tap } => match mgr.get_saved_pipeline(&name).await {
Request::PipelineRunSaved { name, tap, vars } => match mgr.get_saved_pipeline(&name).await {
Some(spec) => {
let vars_map: std::collections::HashMap<String, String> = vars.into_iter().collect();
let spec = match shipote_card::substitute_vars(&spec, &vars_map) {
Ok(s) => s,
Err(e) => return Response::Error { message: format!("template: {e}") },
};
let disc = DiscernPipeline::default_pipeline();
let inc = mgr.incarnator_handle();
let ws_label = mgr.workspace_label(spec.workspace).await.unwrap_or_default();
@@ -275,27 +305,22 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
tap,
std::sync::Arc::new(disc),
inc,
Some(mgr.clone()),
)
.await
{
Ok(launch) => Response::PipelineStarted {
pipeline: launch.pipeline,
command_pids: launch.command_pids,
edges: launch
.edge_discernments
.into_iter()
.map(|e| EdgeDiscernmentInfo {
from_label: e.from_label,
from_output: e.from_output,
to_label: e.to_label,
to_input: e.to_input,
ty: e.discernment.as_ref().map(|d| format!("{:?}", d.ty)),
mime: e.discernment.as_ref().and_then(|d| d.mime.clone()),
lens: e.discernment.as_ref().and_then(|d| d.lens.clone()),
confidence: e.discernment.as_ref().map(|d| d.confidence).unwrap_or(0.0),
})
.collect(),
},
Ok(launch) => {
let pipeline_id = launch.pipeline;
announce_edges_to_broker(pool.as_deref(), &pipeline_id, &launch.edge_discernments);
let cmds = launch.command_pids;
mgr.register_pipeline_commands(spec.workspace, pipeline_id, cmds.clone()).await;
let edges = launch.edge_discernments.into_iter().map(map_edge_to_info).collect();
Response::PipelineStarted {
pipeline: pipeline_id,
command_pids: cmds,
edges,
}
}
Err(e) => Response::Error { message: format!("{e}") },
}
}
@@ -304,6 +329,45 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
},
},
Request::PipelineStop { pipeline, grace_ms } => {
let reaped = mgr
.stop_pipeline(pipeline, std::time::Duration::from_millis(grace_ms))
.await;
Response::PipelineStopped { pipeline, reaped }
}
Request::WorkspaceStats { workspace } => match mgr.workspace_stats(workspace).await {
Some(s) => Response::WorkspaceStats {
info: WorkspaceStatsInfo {
commands_alive: s.commands_alive,
commands_total: s.commands_total,
rss_bytes: s.rss_bytes,
rss_peak_bytes: s.rss_peak_bytes,
cpu_usec: s.cpu_usec,
source: s.source,
uptime_ms: s.uptime_ms,
},
},
None => Response::Error {
message: format!("workspace {workspace} not found"),
},
},
Request::FlowList => {
let items = mgr
.list_flow_pipelines()
.await
.into_iter()
.map(|(pipeline, sockets)| FlowInfo { pipeline, sockets })
.collect();
Response::FlowList { items }
}
Request::FlowDrop { pipeline } => {
let existed = mgr.drop_pipeline_flows(pipeline).await;
Response::FlowDropped { pipeline, existed }
}
Request::Capabilities => {
let c = mgr.incarnator().capabilities();
Response::Capabilities {
@@ -317,6 +381,69 @@ async fn dispatch(mgr: &Arc<WorkspaceManager>, disc: &DiscernPipeline, req: Requ
}
}
fn map_edge_to_info(e: shipote_core::pipeline::EdgeDiscernment) -> EdgeDiscernmentInfo {
EdgeDiscernmentInfo {
from_label: e.from_label,
from_output: e.from_output,
to_label: e.to_label,
to_input: e.to_input,
ty: e.discernment.as_ref().map(|d| format!("{:?}", d.ty)),
mime: e.discernment.as_ref().and_then(|d| d.mime.clone()),
lens: e.discernment.as_ref().and_then(|d| d.lens.clone()),
confidence: e.discernment.as_ref().map(|d| d.confidence).unwrap_or(0.0),
flow_socket: e.flow_socket,
}
}
/// Por cada edge con TypeRef detectado, spawneamos una Card efímera en el
/// SidecarPool que se anuncia al broker como producer del TypeRef
/// enriquecido. Esto permite a otros explorers (broker-explorer, etc.)
/// ver que shipote vio JSON/text/wasm/etc. saliendo de un pipeline.
fn announce_edges_to_broker(
pool: Option<&brahman_sidecar::SidecarPool>,
pipeline: &ulid::Ulid,
edges: &[shipote_core::pipeline::EdgeDiscernment],
) {
let Some(pool) = pool else { return };
for e in edges {
let Some(d) = &e.discernment else { continue };
let label = format!(
"shipote.flow.{}.{}.{}.{}",
short_ulid(pipeline),
e.from_label,
e.from_output,
type_label(&d.ty)
);
let mut card = Card::new(label);
card.kind = CardKind::Data;
card.lifecycle = Lifecycle::Oneshot;
card.payload = Payload::Virtual;
card.supervision = Supervision::OneShot;
card.flow = Flows {
input: Vec::new(),
output: vec![Flow {
name: e.from_output.clone(),
ty: d.ty.clone(),
pin_to: None,
}],
};
pool.spawn(card);
info!(pipeline = %pipeline, from = %e.from_label, ty = ?d.ty, "edge announced to broker");
}
}
fn short_ulid(u: &ulid::Ulid) -> String {
let s = u.to_string();
s[s.len() - 6..].to_string()
}
fn type_label(t: &TypeRef) -> String {
match t {
TypeRef::Primitive { name } => name.clone(),
TypeRef::Wit { package, name, .. } => format!("{package}.{name}"),
}
}
/// Card del daemon. La presentamos al broker así otras sesiones pueden
/// descubrir que shipote está corriendo y, eventualmente, conectarse
/// como consumidoras del flow `workspaces` (futuro: que la GUI o el