Start with six rows

Imagine six sites connected by a small network. You know two feature values at each site, but deliberately cover the row at B. What could a model use to reconstruct it? Nearby sites offer one source of context. The connections say which sites are nearby in the graph.

We use a power-grid-shaped synthetic graph. Its two feature channels, $u$ and $v$, are dimensionless invented numbers. They are not voltages, power injections, sensor readings, or simulated grid states. The shape supplies a memorable example; it supplies no evidence about electrical behavior.

A feature vector is an ordered list. At B it is $x_B=(1,2)$; swapping the entries changes its meaning. Stack the six vectors as rows to get $X\in\mathbb{R}^{6\times2}$. The graph $G=(V,E)$ adds a node set $V$ and an edge set $E$. An edge joins two rows; it is not another feature value.

CL-012 · All six feature rows and graph connections are teaching assumptions.

One message-passing step

B has three neighbors: A, C and D. A message-passing layer collects their vectors, combines them, and transforms the result using shared learned weights. “Shared” means the same rule is applied at different nodes; it does not mean every node receives the same result.

Let $\mathcal{N}(i)$ be the neighbors of $i$, $h_i^{(0)}=x_i$ its starting vector, and $W_{\rm self},W_{\rm nbr}$ learned matrices. One illustrative layer is:

The sum collects neighbors, division makes a mean, and $\sigma$ is a nonlinear transformation. With two input features and $q$ output features, both matrices have shape $q\times2$, treating each vector as a column in this equation.

Before the transformation, B's original neighbor mean is $((2,1)+(2,2)+(3,1))/3=(7/3,4/3)$. That is not the target $(1,2)$. Graph context offers information to learn from; adjacency alone does not specify a perfect reconstruction rule.

This mean layer is a refresher, not a reproduction of GraphMAE's architecture. Graph attention can instead learn neighbor-dependent weights. The authors' main GraphMAE recipe uses graph attention, while studying other architecture choices.

CL-002 · Gilmer et al., message-passing formulation, §2

CL-002 · Veličković et al., Graph Attention Networks, §2

Targets without manual labels

A training objective compares a prediction with a target. Self-supervision changes where that target comes from: here, the original feature row supplies it. We hide a row from the predictor while retaining its original values to score the prediction.

GraphMAE, introduced by Hou and colleagues in 2022, centers its original objective on masked node-feature reconstruction. Its graph supplies context, and its encoder learns a representation through this pretext task. This is a particular graph method, not a new name for every masked autoencoder.

CL-001 · GraphMAE, original paper, §3

The next question is more precise than “what is masked?” We need to follow the original row, the input token, and the hidden representation separately.

One row, two paths

Let $M$ be the selected nodes. In the clean example, $M=\{B\}$. The encoder receives a corrupted matrix $\widetilde X$, where B's row is replaced by a learned input mask token $m_{\rm in}\in\mathbb{R}^{2}$. The scoring path retains the original $x_B=(1,2)$.

The token is a trainable vector, not the target and not a command to delete B. In the reviewed author code it starts at zero, then can be updated during learning. B remains a node with its edges.

CL-003 · Author implementation, encoding_mask_noise and enc_mask_token

Why mask again?

The encoder computes $H=f_\theta(G,\widetilde X)$. Each row $h_i$ is now a learned representation rather than a raw feature vector. A learned, bias-free projection maps the encoder output into the decoder's input space. In the graph-decoder path, the implementation overwrites the projected rows at the same nodes in M with zero.

Call that re-masked matrix $\widetilde H$. The graph decoder predicts $\widehat X=g_\phi(G,\widetilde H)$, producing a feature row for each node. At B, it has a zero input row and can aggregate neighboring representations. This removes the direct route from B's own encoder row into B's decoder input.

That routing choice does not prove all information about B has vanished elsewhere. Message passing mixes information across connected nodes. Re-masking specifies which local decoder input is removed; it is not an information-theoretic guarantee against shortcuts.

CL-004 · MLP/linear alternatives do not follow this re-masking branch. This guide teaches the graph decoder path.

CL-004 · Author implementation, projection, re-mask and decoder, lines 246–260

The decoder emits more than the loss scores

All six reconstructed rows may exist, but this objective scores only the selected rows. With $M=\{B\}$, an error at A does not enter this step's loss. With $M=\{B,D\}$, the objective averages B's and D's terms. “Masked-only” describes which rows are scored, not which intermediate activations or weights can receive gradients.

The encoder, input token, projection and decoder participate in learning. Our interactive lab will do none of those updates: it evaluates the same loss selection on explicitly prescribed reconstructions.

CL-005 · Author implementation, recon[mask_nodes] and x[mask_nodes]

Hide one row. Then hide its neighbors.

First hide B. A, C and D still have their original rows. Now hide B, C and D together. B remains connected to all three neighbors, but only A still provides its original row directly. C and D provide mask-token inputs at the encoder's first layer.

The distinction is between a neighbor existing and that neighbor's original features being visible. With multiple message-passing layers, context can travel farther: C connects to F, and D to E. Repeated aggregation expands the paths through which information can influence B. It does not make missing features recoverable by definition.

CL-012 · The comparison counts visible original feature rows. It is not a measured masking-difficulty curve.

A mask ratio leaves out its location

Our masks select one, two, or three of six nodes: approximately 16.7%, 33.3%, and 50%. These are teaching states, not a recommended schedule. Two masks of the same size can preserve different neighborhoods and different signals.

Holding a prediction fixed while changing the scored mask asks: which errors are averaged? Retraining under different masking policies asks: what representations does that learning process produce? The second needs controlled runs, a downstream task, and uncertainty estimates. The first can be answered exactly in a browser.

Define the prediction-time information boundary before pretraining. If an evaluation claims new-graph generalization, using that test graph during pretraining changes the claim, even without labels. A same-graph transductive protocol can be valid; it needs to be named.

CL-010 · Author inductive entry point; evaluation advice here is a qualified inference

Using edges is not reconstructing edges

Our edges determine message-passing neighborhoods. The target remains a feature row such as $(1,2)$. We do not predict whether AB or BC exists. Edge reconstruction needs its own target, output, and objective.

The original GraphMAE paper and reviewed core graphmae/models/edcoder.py path focus on features. There is an optional edge-masking loss branch in the repository's separate chemistry pretraining script. That detail is a reason to name the exact path and configuration, rather than infer behavior from a project name.

CL-001 · GraphMAE, original objective

CL-016 · Separate chemistry script, optional mask_edge branch

We now have the target and scoring set. The remaining choice is what it means for a reconstructed row to match.

One row, by hand

Take B's target, $x_B=(1,2)$, and prescribe $\widehat x_B=(2,1)$. This is a chosen vector, not a trained decoder's output. Their dot product is $1\cdot2+2\cdot1=4$. Each vector's Euclidean norm, or length, is $\sqrt{5}$.

Cosine similarity $c_i$ measures alignment. For nonzero vectors it lies between −1 and 1: opposite directions at −1, a right angle at 0, the same direction at 1. Here $c_B=4/5=0.8$. The angular residual $1-c_B$ is 0.2.

GraphMAE's scaled cosine error raises the residual to an exponent $\gamma\ge1$, then averages over selected nodes:

With only B selected and $\gamma=2$, the score is $(1-0.8)^2=0.04$. The exponent acts before the mean. “Scaled” here refers to this power, not an extra fitted multiplier.

CL-006 · Author SCE implementation: normalize, dot, residual power, mean

Inspect the objective

Predict before changing the controls: if you double the prescribed prediction from $(2,1)$ to $(4,2)$, does SCE change? Does squared error change? Start with one variable; the mask and exponent can stay fixed.

For comparison, this lab uses per-feature mean squared error, then averages over selected nodes. The feature dimension is $d=2$:

At B the baseline is $((2-1)^2+(1-2)^2)/2=1$. Doubling gives $((4-1)^2+(2-2)^2)/2=4.5$. SCE remains 0.04: positive rescaling cancels between the cosine numerator and denominator.

Now select B, C and D. The average can fall even though more features are withheld. C's prescribed direction is exactly right and D's nearly right; their small terms lower the mean. The predictions were not learned under the new mask. A lower number here does not imply easier training or a better representation.

CL-013 · Deterministic derivations from CL-012's prescribed vectors. Browser, static table and runnable lab share one numerical module.

What the exponent changes

Increasing $\gamma$ from 1 to 2 changes B's term from 0.2 to 0.04. For a residual $r=1-c$ between zero and one, higher powers make its numerical contribution smaller. Larger residuals become more prominent relative to smaller ones.

That is not a promise that every individual loss decreases. If $c=-0.5$, then $r=1.5$: squaring raises the term from 1.5 to 2.25. Changing the loss scale changes optimization behavior; a smaller displayed value does not itself mean better training.

A perfect angle is not a perfect row

There is a stronger counterexample: let $\widehat x_B=2x_B=(2,4)$. The direction is exact, so SCE is zero for every positive exponent. Yet the values are wrong. Per-feature squared error is $((2-1)^2+(4-2)^2)/2=2.5$.

This is a property of the objective, not a defect discovered in a trained model. If absolute magnitude matters downstream, this distinction belongs in your experimental design. Evaluate magnitude recovery directly or compare a suitable magnitude-sensitive objective. Whether doing so improves performance remains an empirical question.

What leaves pretraining?

After pretraining, extract representations with the encoder. The reviewed implementation's embed(g,x) calls the encoder on the supplied graph and features, without the corruption-and-decoder path. A linear probe freezes these representations and trains a small linear predictor. Fine-tuning updates some or all encoder weights for the downstream target.

A probe asks what the representation already makes accessible to a simple predictor. Fine-tuning also asks how it adapts with supervision. Neither question is answered by our lab's reconstruction score.

CL-008 · Author inductive linear evaluation and pretraining entry point

CL-008 · Author embed method, encoder output only

Can it handle a different topology?

A message-passing encoder applies shared local rules, so it can be evaluated on compatible graphs with different numbers of nodes or different adjacency patterns. That is a statement about computation. Accuracy depends on the learned rule, features, graph construction, and training/test distributions.

For a new grid topology, compatible tensor dimensions are not enough. Changed connectivity, feature definitions, operating regimes, or measurement availability may change useful relationships. Our example establishes none of those real-system relationships; the original paper's public benchmarks do not certify a particular grid application.

A defensible evaluation plan makes these decisions before inspecting final test results:

1. Name the transfer unit. Hold out whole topologies for unseen-topology claims, or declare a same-graph protocol. For temporal transfer, state the cutoff and correlated-sample grouping.
2. Define the information boundary. Fit preprocessing on allowed training data. State whether test features or edges enter pretraining; unlabeled access is still access.
3. Compare matched baselines. Use the same downstream labels, splits and tuning budget for random initialization, a supervised baseline, and pretraining. Separate frozen-probe and fine-tuned results.
4. Report the relevant outcome. Choose metrics and failure slices in advance; include multiple seeds and uncertainty. Report topology-level variation separately from reconstruction diagnostics.

CL-010 · Methodological recommendations inferred from the mechanism and inductive/transductive distinction. This plan has not been executed.

Keep the method names precise

Generic masked autoencoding is a broader pattern. Vision MAE, for example, encodes visible image patches with an asymmetric design; its exact token routing should not be substituted for GraphMAE's graph path.

Original GraphMAE is the feature mechanism traced here. GraphMAE2 adds multi-view random re-masking and latent representation prediction. Those additions differ from original GraphMAE's same-node re-mask.

“GraphMAE+Edge” does not identify a method by itself. A user-specific variant needs its own code, objective, protocol and evidence. Neither the name nor the optional chemistry branch establishes what that variant implements or how it performs.

CL-015 · He et al., Masked Autoencoders, §3

CL-009 · Hou et al., GraphMAE2, §2.2 and Figure 2

CL-011 · User-specific variant remains unverified. No private research results are inferred.

Try it on a document graph

A new graph has text boxes as nodes and reading-order links as edges. Each node stores box area and ink mass in fixed units. Your prescribed reconstruction doubles both entries at every selected node. You get zero SCE. A downstream task needs absolute box area.

What has the reconstruction score established—and what has it missed? How would you evaluate a claim about unseen document layouts?

Return to B: we withheld its row, retained its target, routed context through a graph encoder and re-masked decoder, and scored selected reconstruction directions. Every later transfer claim must add evidence beyond that computation.