Deep Compositor: Distributed Architecture Guide¶

This document outlines the strategy for moving from a local CLI renderer to a distributed, cloud-native system using Go and C++.

1. Why Protobuf & gRPC?¶

Protobuf (The Contract): A language-neutral, binary serialization format. It is significantly faster and smaller than JSON. It ensures that the Go Coordinator and C++ Worker always agree on the data structure.
gRPC (The Transport): Handles the "plumbing" of networking (connections, timeouts, concurrency). It allows the Go Coordinator to call RenderLayer() on a remote C++ node as if it were a local function.
Side-Channel Storage: We use URIs (gs://) to pass heavy image data. Protobuf only carries the "instructions," while Google Cloud Storage (GCS) carries the "payload" (Deep EXR files).

1. Protobuf Runtime (Required by the code generated by protoc-gen-go)¶

go get google.golang.org/protobuf

2. gRPC Runtime (Required to create the server/client connect`ions)¶

go get google.golang.org/grpc

3. Google Cloud Storage (For your "GCS Hookup" to store .keb files)¶

go get cloud.google.com/go/storage

2. Component Roles¶

Component	Language	Role	Location
CLI	C++	User interface & process manager. Spawns Coordinator.	Local Machine
Coordinator	Go	The "Brain." Manages jobs & cloud provisioning (BYOP).	Local Machine (Auto-run)
Render Worker	C++	The "Brawn." Performs heavy raytracing.	User's GCP Project
Compositor	C++	The "Librarian." Merges deep layers.	Local Machine

For a detailed visual mapping of how these components interact, see Control Flow Guide.

3. Development Roadmap (PR & Issue Strategy)¶

Phase 1: Local Foundation¶

Issue: Setup Go workspace and Proto generation.
PR: feat: infra-go-grpc-bindings (Include go.mod and generated .pb.go files).
PR: feat: coordinator-skeleton (Implement SubmitJob and RegisterWorker stubs).
PR: feat: auth-byop (Implement Service Account key handling for multi-tenant/multi-project provisioning).

Phase 2: C++ Integration (The Hybrid Step)¶

Issue: Integrate gRPC into C++ Build.
PR: feat: cpp-grpc-server (Update CMakeLists.txt to find gRPC and implement the RendererService base class).
PR: feat: local-e2e-loop (Go Coordinator calls local C++ Worker over localhost).

Phase 3: Containerization & Cloud¶

Issue: Dockerize components for GKE.
PR: feat: docker-worker-image (Create multi-stage Dockerfile for C++ worker).
PR: feat: gcs-integration (Update C++ worker to upload results to gs:// URIs).
PR: feat: k8s-manifests (Deployment and Service YAMLs for GKE).

4. C++ Implementation Pattern¶

To implement the "Contract," your C++ worker must inherit from the generated service:

class RendererServiceImpl final : public renderer::v1::RendererService::Service {
    grpc::Status RenderLayer(grpc::ServerContext* context, 
                            const RenderLayerRequest* request,
                            RenderLayerResponse* response) override {
        // 1. Map Proto -> C++ Options
        // 2. session.Render()
        // 3. Upload to request->output_target_uri()
        // 4. Return success status
        return grpc::Status::OK;
    }
};

5. Build Strategy¶

Local: Install gRPC via brew or apt to keep CMake runs fast.
Docker: Use a pre-built base image with gRPC installed to avoid 20-minute compile times on every change.