PyGraphistry Bug Report: UMAP in GFQL Chains Fails with Missing Tracking Column

Repository: graphistry/pygraphistry
Version: v0.43.0
Component: graphistry/compute/chain.py:191 in combine_steps()

Summary

UMAP operations fail when used in GFQL chains (even simple 1-operation chains) with error:

ValueError: Column 'index' not found in edges step DataFrame.
Step has id='index', available columns: ['_src_implicit', '_dst_implicit', '_weight'].

Minimal Reproduction

import pandas as pd
import graphistry

# Create simple graph
edges_df = pd.DataFrame({
    'src': ['A', 'B', 'C', 'D', 'E'],
    'dst': ['B', 'C', 'D', 'E', 'A']
})

nodes_df = pd.DataFrame({
    'id': ['A', 'B', 'C', 'D', 'E'],
    'town': ['Birchford', 'Cedarland', 'Maplewood', 'Oakville', 'Pineville'],
    'poi': ['store', 'theatre', 'pet_shop', 'square', 'park']
})

# Create plottable
g = graphistry.edges(edges_df, 'src', 'dst').nodes(nodes_df, 'id')

# Try UMAP in a GFQL chain - THIS FAILS
from graphistry.compute.ast import ASTCall

umap_op = ASTCall('umap', {
    'n_components': 2,
    'n_neighbors': 3,
    'umap_kwargs': {'random_state': 42}
})

# This will fail even with just one operation
result = g.gfql([umap_op], engine='pandas')

Error Output

ValueError: Column 'index' not found in edges step DataFrame.
Step has id='index', available columns: ['_src_implicit', '_dst_implicit', '_weight'].
Operation: <graphistry.compute.ast.ASTCall object at 0x...>

Traceback:
  File "graphistry/compute/gfql_unified.py", line 257, in gfql
    return chain_impl(self, converted_query, engine, policy=policy)
  File "graphistry/compute/chain.py", line 275, in chain
    return _chain_impl(self, ops, engine, validate_schema, policy)
  File "graphistry/compute/chain.py", line 512, in _chain_impl
    final_edges_df = combine_steps(g, 'edges', list(zip(ops, reversed(g_stack_reverse))), engine_concrete)
  File "graphistry/compute/chain.py", line 191, in combine_steps
    raise ValueError(f"Column '{id}' not found in {kind} step DataFrame. ...")

Root Cause

1. PyGraphistry's chain implementation adds tracking columns (like 'index') to DataFrames to track which operation produced each row
2. UMAP creates completely new edges with only ['_src_implicit', '_dst_implicit', '_weight'] columns
3. UMAP rebinds the graph with these new edges (which is correct behavior)
4. When combine_steps() tries to merge results, it looks for the tracking column 'index' but UMAP's new edges don't have it

Expected Behavior

UMAP should work in GFQL chains. Since UMAP creates an entirely new graph with new edges, the chain code should recognize this and not try to combine it with tracking columns from previous steps.

Actual Behavior

Chain fails because combine_steps() assumes all operations preserve DataFrame structure and tracking columns.

Suggested Fix

Option 1: Skip tracking for graph-creating operations

• Detect operations like UMAP that create new graphs (by checking if edges/nodes are completely replaced)
• Don't try to combine them using tracking columns

Option 2: Make tracking columns optional in combine_steps

• Handle missing tracking columns gracefully
• If tracking column is missing, assume the operation created a new graph

Option 3: UMAP preserves tracking columns

• Have UMAP copy tracking columns from input to output
• However, this doesn't make semantic sense since UMAP's edges are completely different from the input

Impact

• UMAP cannot be used in any GFQL chain (even single-operation chains that go through the chain code path)
• Blocks workflows that need UMAP + other operations (e.g., UMAP followed by filtering or degree calculations)

Workaround

None. UMAP can only be used via direct API calls like g.umap(), not through GFQL chains.

Environment

• PyGraphistry version: v0.43.0
• Python version: 3.10
• Engine tested: both 'pandas' and 'cudf'
• Context: Discovered during GFQL remote operations testing in Graphistry server v2.44.0-rc2

Additional Notes

The issue specifically occurs in combine_steps() which is called by the chain implementation. The error message shows it's looking for a column with id='index' which is a tracking column added by the chain machinery, not something from the user's data.

UMAP correctly creates new edges with new bindings - the bug is that chain doesn't handle this case properly.