Technical Report · Draft for Academic Review

CodeFlock: A Formal Model for Structural Codebase Transformation

Abstract
Migrating a codebase from one architectural structure to another NEEDS a formal model that treats the structure of a codebase — its shape — as a first-class object of transformation. There is currently no CodeFlock proposes such a model. A Canon is a formal description of structural rules a codebase must satisfy, giving it a definite shape. A Retcon is a function mapping one Canon to another. A Plane is an intermediate representation that makes certain transformations tractable. These three abstractions admit a composition algebra: complex migrations decompose into chains of atomic building blocks, each verifiable before execution. Beyond the engineering model, we show that this formalism generates natural training data for machine learning: building blocks are labeled atomic transformations, intermediate Canons are supervision checkpoints, and the Plane lattice induces a multimodal representation of the same project. A model trained on this structure can learn to recognize the shape of a codebase and plan a migration path — a capability that current code models entirely lack.

1. Motivation and Problem Statement

Consider a team maintaining a Django monolith of 80,000 lines directed to migrate to a set of FastAPI microservices — one per business domain. The migration involves identifying domain boundaries, extracting modules, replacing in-process function calls with HTTP or message-queue calls, generating Dockerfiles, and updating routing configuration. Each of these steps is performed manually, inconsistently, and unreproducibly. There is no artifact that describes what the target codebase should look like structurally, and no way to verify that an intermediate state is valid.

The same pattern appears across many common tasks: renaming all exported symbols to follow a new convention, splitting a flat C project into a header/source pair per module, converting CommonJS to ES Modules, generating TypeScript type declarations from a SQL schema. Each is a structural transformation of a codebase — it changes how code is organized, not what it computes.

Tools like jscodeshift, ast-grep, and sed-based scripts address fragments of this problem through local rewrites. They do not address the global structural question — what shape should the entire codebase be in — nor do they let an engineer specify, compose, or verify whole-codebase structural migrations. The absence of a formal model for codebase shape has a direct practical consequence: migrations are underdetermined. There is no specification to check against, no way to detect that a step has left the codebase in a structurally invalid intermediate state, and no basis for automating or learning the transformation.

CodeFlock addresses this by treating codebase structure as a typed object. Once a codebase has a formal shape — a Canon — it becomes possible to define what migrations are valid, to compose them from reusable building blocks, and critically, to train a model to recognize shapes and plan migrations between them. The building block strategy is not just an engineering convenience: it produces a structured vocabulary of codebase shapes and the transitions between them that can drive both static verification and machine learning.

2. Core Abstractions

2.1 Canon

Definition 2.1 — Canon
A Canon is a finite set of structural rules that a project must satisfy. A project P conforms to Canon C, written P ⊨ C, if every rule in C holds for P.

A Canon is not a type system for the code inside files — it is a type system for the arrangement of files, modules, and components. Two projects can contain identical logic but conform to different Canons. Crucially, Canon conformance is checkable: given a project and a Canon, a validator can decide P ⊨ C without executing the project.

Example 2.1 — Two Canons for the same Python logic
Canon A (Django Monolith): One top-level apps/ directory. All business logic lives in models.py, views.py, serializers.py per app. Inter-app communication is direct Python import. No Dockerfile per module.

Canon B (FastAPI Microservice): One directory per service. Each service has main.py, routes/, schemas/, a Dockerfile, and communicates with other services only via HTTP or a message queue — no cross-service Python import.

A Canon is a predicate on a project, not a recipe for building one. The same predicate can be checked against any project to determine conformance, making it reusable across migrations.

2.2 Retcon

Definition 2.2 — Retcon
A Retcon R from Canon A to Canon B is a total function
R : {P | P ⊨ A} → {P' | P' ⊨ B}
that takes a project conforming to A and produces a project conforming to B.

Retcons compose: if R₁ : A → B and R₂ : B → C, then R₂ ∘ R₁ : A → C is a valid Retcon. Complex migrations are built from sequences of simpler ones.

2.3 Building Block

Definition 2.3 — Building Block
A Building Block is an atomic Retcon — one that performs a well-understood structural operation. It could be simple or complex in implementation but it must be sementically well defined.
Example 2.2 — Building blocks for the Django → FastAPI migration
B₁  cluster_by_call_graph   : Monolith     → Clustered
    (identify domain boundaries via call-graph analysis)

B₂  extract_clusters        : Clustered    → Extracted
    (move each cluster to its own directory)

B₃  replace_cross_imports   : Extracted    → Decoupled
    (replace in-process calls with HTTP stub calls)

B₄  add_service_entrypoint  : Decoupled    → ServiceReady
    (add main.py + FastAPI router per service)

B₅  generate_dockerfile     : ServiceReady → Microservice
    (emit Dockerfile + requirements.txt per service)

Retcon: MonolithToMicroservice = B₅ ∘ B₄ ∘ B₃ ∘ B₂ ∘ B₁
Each intermediate Canon — Clustered, Extracted, Decoupled, ServiceReady — is a formal predicate that can be checked after each step.

2.4 Plane

Definition 2.4 — Plane
A Plane is a representation of a project that makes a particular class of operations efficient or tractable. The same project can be represented in multiple planes simultaneously; a plane switch changes the representation, not the project.

The choice of plane is an algorithmic decision. The operation cluster by domain boundary cannot be performed efficiently on the File plane — it requires the Graph plane (a call graph of functions across files). The operation rename all exported symbols cannot be performed correctly on the Text plane without false positives — it requires the AST plane.

Plane Representation Operations enabled Operations impractical
File Paths and byte content Move, copy, delete, create files; repackage directory structure Rename a symbol; find callers; detect unused imports
Text Sequence of characters Regex search/replace; delimiter-split; string interpolation Structure-aware rename; cross-file reference tracking
Token Lexical token stream Token-type filtering; count identifiers; detect keywords Structural queries (scope, nesting, call targets)
AST Abstract syntax tree Rename with scope awareness; extract function; inline variable; detect unused exports Cross-file call graph; dynamic dispatch resolution
Graph Call graph (nodes = functions, edges = calls) Find strongly connected components; topological sort; cluster by reachability; detect cycles Inspect syntax of individual functions
Service Deployable units with endpoints Generate Dockerfiles; define API contracts; configure routing Inspect source code structure

2.5 Crossover and the Vertical Retcon

Definition 2.5 — Crossover
A Crossover is a Retcon X : (P, Plane_A) → (P, Plane_B) that changes the representation plane of a project without changing the Canon the project conforms to.
Definition 2.6 — Vertical Retcon
A Vertical Retcon is a Retcon implemented as:
R_vertical = X_up ∘ R_lateral ∘ X_down
where X_down is a Crossover to an intermediate plane where the lateral transformation is tractable, R_lateral is a Lateral Retcon (same-plane, possibly changing Canon), and X_up is a Crossover back to the original plane.

The decomposition captures a practical engineering principle: the difficulty of a transformation is determined by the plane, not the Canon. Moving to a richer plane makes hard operations easy; moving back makes the result portable.

Example 2.3 — Vertical Retcon: finding domain boundaries via the call graph
Building block B₁ (cluster_by_call_graph) is a Vertical Retcon:

X_down: Parse all Python files → construct call graph (File plane → Graph plane). Canon unchanged.

R_lateral: Run Louvain community detection on the call graph to identify clusters of high internal cohesion. Annotate each function node with its cluster ID. (Graph → Graph, now conforming to Clustered Canon.)

X_up: Emit cluster annotation as a configuration file for downstream building blocks. (Graph → File plane.)

2.6 Universe

Definition 2.7 — Universe
A Universe is a closed pair U = (𝒞, ℬ) where 𝒞 is a set of Canons and ℬ is a set of Building Blocks, such that for every B ∈ ℬ, both the source Canon and target Canon of B belong to 𝒞. A Retcon is well-formed in universe U if all intermediate Canons in its composition belong to 𝒞.

A Universe bounds the vocabulary of a migration. A team working on Python projects defines a Python Universe; a team working on frontend repositories defines a Frontend Universe. Universes can share Building Blocks where operations are language-agnostic.

3. Structural Diagrams

3.1 The Retcon as a typed function

Canon A rules: [R₁, R₂, …] validate(P): bool Retcon R apply(P: A) → P': B plane: Plane Canon B rules: [R₁, R₂, …] validate(P): bool input output invariant: output.validate() == true
Figure 1. A Retcon R as a typed function between two Canons. The critical invariant: every output project must conform to Canon B.

3.2 Vertical Retcon decomposition

VerticalRetcon boundary Plane A (e.g. File) Plane B (e.g. Graph) P Canon A P' Canon A→B P'' Canon B X_down R_lateral (same plane) X_up
Figure 2. Decomposition of a Vertical Retcon. The Canon changes only during R_lateral; the Crossovers preserve it.

3.3 Composition chain for the Django → FastAPI migration

Monolith Django, flat in-process calls B₁ cluster_by call_graph [Graph plane] Clustered domain IDs annotated B₂ extract_ clusters [File plane] Extracted one dir per domain B₃ replace_cross _imports [AST plane] Decoupled HTTP stubs no cross-import B₄ add_service_ entrypoint [File plane] ServiceReady main.py + router per service B₅ generate_ dockerfile [File plane] Microservice Dockerfile per svc no cross-import Building Block (Bᵢ) Intermediate Canon Final Canon (target)
Figure 3. Composition chain: MonolithToMicroservice Retcon as five Building Blocks. Each intermediate Canon is a verifiable checkpoint. The plane used by each block is noted below it.

3.4 The Plane lattice and Crossover directions

File paths + bytes Text character stream Token lexical stream AST syntax tree, scope-aware Graph call graph Service deployable units lossy Crossover (lossless) lossy = information dropped in target plane
Figure 4. The Plane lattice: arrows indicate valid Crossover directions. Downward Crossovers (File → AST → Graph) are generally lossy. Upward Crossovers complete the round-trip.

4. Extended Worked Example: SQL Schema → TypeScript Types

This example crosses language boundaries entirely — source is SQL DDL, target is TypeScript — making the role of Canons and Planes concrete.

Source Canon (SQL-DDL): One .sql file. Contains only CREATE TABLE statements. Each column has a SQL type. Foreign keys declared with REFERENCES.

Target Canon (TS-Types): One .ts file. One exported TypeScript interface per table. Each column maps to a TypeScript property. Foreign key columns reference the corresponding interface type. No any. File compiles without errors.

The Retcon SQLToTypeScript is a Vertical Retcon:

X_down:    Parse SQL DDL text → schema graph
           (Text plane → Graph plane)
           Nodes = tables, edges = foreign-key relations
           Canon unchanged: still SQL-DDL shape, now in graph form

R_lateral: Walk schema graph in topological order
           (Graph → Graph, now conforming to TS-Types-Graph Canon)
           Map SQL types: VARCHAR → string, INT → number,
           BOOLEAN → boolean, TIMESTAMP → Date, nullable → T | null
           Replace FK edges with interface reference edges

X_up:      Serialize graph nodes as TypeScript interface declarations
           (Graph → File plane, TS-Types Canon)
           Emit one interface per node, properties in column order
  

The intermediate Graph plane is essential: topological sort on the schema graph ensures that no interface references a type not yet declared. This ordering is impossible to determine correctly on the Text plane without re-implementing a parser.

5. Static Verification of Retcon Compositions

A composed Retcon can be verified before any code is transformed. Two properties are checked:

Canon-chain validity: For R = Bₙ ∘ … ∘ B₁, the output Canon of Bᵢ must equal the input Canon of Bᵢ₊₁ for all i.

Plane-continuity: When the output Plane of Bᵢ differs from the input Plane of Bᵢ₊₁, a Crossover must appear between them. A composition that switches planes without a Crossover is ill-formed.

These two checks give CodeFlock a type system over transformations — one that surfaces errors before the system touches any file on disk. This matters most in long migrations: a malformed step at position 7 of a 12-step chain otherwise produces a codebase that is in neither the source nor the target Canon, with no record of where it diverged.

Error type Example Detection phase
Canon mismatch B₂ expects Clustered Canon but B₁ outputs Monolith Canon Static (before execution)
Missing Crossover B₁ outputs Graph plane, B₂ expects File plane, no Crossover between them Static (before execution)
Canon violation B₃ output fails validate() for Decoupled Canon Runtime (after B₃ executes)
Crossover loss X_down is lossy; X_up cannot reconstruct needed information Runtime or design-time review

6. Learning Implications: CodeFlock as a Training Framework

The CodeFlock model was designed as an engineering framework, but its structure has an underexplored consequence: it generates well-formed training data for machine learning, and in doing so opens several distinct learning problems. This section formalizes what can be learned from a corpus of Canons and building block traces.

6.1 Building blocks as labeled atomic transformations

Every execution of a building block Bᵢ : Canon_A → Canon_B on a real project produces a triple:

  (P_in, label(Bᵢ), P_out)    where  P_in ⊨ Canon_A  and  P_out ⊨ Canon_B

This is a supervised training example by construction. The input is a project in a known structural state. The output is the same project in a different structural state. The label is discrete and unambiguous — the name of the building block applied. Collected across many real codebases, this produces a dataset of (before-state, transformation-type, after-state) triples at a granularity that no existing code dataset provides. Current code datasets contain files and commits; they do not label what structural transformation a commit performs.

Learning Problem 6.1 — Block Classification
Given a pair of project snapshots (P, P'), predict which building block was applied. This is a multi-class classification problem over the building block vocabulary of the relevant Universe. A model solving this problem has learned to perceive codebase shape change as a typed action.

6.2 Intermediate Canons as curriculum checkpoints

A migration chain B₁, B₂, …, Bₙ produces not one training signal but n intermediate project states, each conforming to a known Canon. This structure maps directly onto curriculum learning: rather than training a model on the full migration in one shot, it can be trained on each step in sequence, with Canon conformance serving as the reward signal at each checkpoint.

The intermediate Canons also define a natural difficulty ordering on the training data. Early steps (e.g. annotating domain boundaries) are local and recoverable. Late steps (e.g. generating Dockerfiles) depend on all prior steps being correct. A curriculum that sequences training examples by step index mirrors the dependency structure of the transformation.

Learning Problem 6.2 — Next Block Prediction
Given a project P conforming to some Canon C_k, predict the next building block to apply. This is a sequential decision problem: the model must identify the current Canon of the project and select the block whose source Canon matches. Over a full migration trace, this becomes a planning problem — predicting a sequence of building blocks that takes P from its current Canon to a target Canon.

6.3 Canon conformance as a structured verifier

Canon conformance — the predicate P ⊨ C — is a binary classifier with a formal definition. A model trained to predict conformance can serve as a learned verifier, replacing the hand-written rule-checker for Canons that are too complex or context-dependent to specify exhaustively. More importantly, a model that predicts conformance has implicitly learned an embedding of codebase structure: two projects that conform to the same Canon should be nearby in the learned representation space, regardless of language, size, or domain.

Learning Problem 6.3 — Canon Recognition
Given a project P, predict which Canon in Universe U it most closely conforms to. A model solving this problem has learned a structured embedding of codebase shape. The embedding space is not arbitrary — it is organized by the Canon graph of the Universe, where two Canons are adjacent if a single building block connects them.

6.4 The Plane lattice as a multimodal representation

The same project represented in multiple planes — File, AST, Graph — constitutes a multimodal view of the same object. This is directly analogous to multimodal learning in vision and language: the same content expressed in different modalities, with cross-modal alignment as a training objective. In the CodeFlock setting, a Crossover from File to AST is a deterministic translation between modalities. A model trained on both the File and AST representations of the same project, with the Crossover as alignment, learns to ground structural descriptions at two levels of abstraction simultaneously.

This representation also supports cross-plane retrieval: given a project in the Graph plane (a call graph), retrieve the most structurally similar project in the File plane (an actual directory layout). This is a non-trivial retrieval problem — it requires the model to understand that structural similarity in one plane implies structural similarity in another.

Learning Problem 6.4 — Cross-Plane Alignment
Given a project in Plane A and the same project in Plane B (connected by a Crossover), learn a shared embedding where the two representations are mapped to the same point. Evaluated by: given a project in Plane A, retrieve the correct project in Plane B from a corpus. This is analogous to image-text contrastive learning, with Planes as modalities and Crossovers as alignment pairs.

6.5 Universe closure as a constrained generation problem

A Universe is a closed set of Canons and building blocks. A model generating a migration plan is constrained to output sequences of building blocks that (a) belong to the Universe and (b) form a valid Canon chain. This is a constrained generation problem — structurally identical to grammar-constrained decoding in language models, where the output must conform to a formal grammar at each decoding step.

The Canon graph of a Universe — where nodes are Canons and edges are building blocks — defines the valid transition system. A migration planner is a model that performs reachability search over this graph, guided by a learned heuristic. The heuristic can be learned from observed migration traces: which paths between Canons are taken most often, which building blocks tend to appear together, which intermediate Canons are most commonly used as waypoints.

Learning Problem 6.5 — Migration Planning
Given a source Canon C_s, a target Canon C_t, and a Universe U, generate a building block sequence B₁, …, Bₙ such that Bₙ ∘ … ∘ B₁ : C_s → C_t is a valid Retcon in U. This is a structured prediction problem over the Canon graph. The model must simultaneously respect the type constraints of the Universe and optimize for migration quality — shortest path, minimum information loss, or maximum reversibility.

6.6 Summary of learning dimensions

Learning problem Input Output ML formulation
Block classification (P_before, P_after) pair Building block label Multi-class classification
Canon recognition Project P in any plane Canon C ∈ U Multi-class classification with structured label space
Next block prediction Project P, target Canon C_t Next building block Bᵢ Autoregressive sequence prediction
Migration planning C_source, C_target, Universe U Block sequence B₁…Bₙ Constrained structured prediction / graph search
Cross-plane alignment Project in Plane A Embedding shared with Plane B Contrastive learning
Canon conformance Project P, Canon C P ⊨ C (binary) Binary classification / learned verifier

These six problems are not independent. A model that solves Canon recognition has learned a representation useful for block classification. A model that solves cross-plane alignment has learned embeddings that support migration planning. The CodeFlock formalism ties them together: they are all facets of the same underlying structure, and progress on one informs the others.

This document was prepared as a technical overview to support a discussion with an academic supervisor. The definitions are presented informally but with enough precision to anchor a more rigorous formalization. The examples use Python/Django and SQL/TypeScript as concrete instances; the model is language-agnostic.