Create draft plans for addressing each limitation

plans/06-streaming-results.md

@@ -0,0 +1,118 @@
+# Streaming results instead of fetching them fully
+
+## Problem
+
+`Session._sql` calls `fetchall()` (`boltage/backend.py:178`), `_run_cypher` parses
+every row with `agtype.parse`, and the whole list sits in `Result.rows` until the
+client has pulled it (`BoltConnection.pull`, `boltage/bolt.py:327`). A
+`MATCH (n) RETURN n` over a large graph holds the full result in boltage's memory
+three times over (PostgreSQL's wire data, the agtype text, the parsed objects), and
+the first record only reaches the client after the last one has been read from
+PostgreSQL. Bolt's `PULL {n}` already lets the client ask for batches; boltage just
+does not pass that on to PostgreSQL.
+
+## Approach: server-side cursors
+
+Declare a PostgreSQL cursor for the query and `FETCH` from it as the client pulls.
+
+### Why named cursors, not single-row mode
+
+psycopg offers two ways to stream:
+
+- **`cursor.stream()`** (libpq single-row / chunked mode) works without a
+  transaction, but occupies the connection until the result is fully read. Bolt
+  allows several open results in one explicit transaction (each with a `qid`), and a
+  client may `RUN` again before pulling the previous result. With one PostgreSQL
+  connection per Bolt connection, that would deadlock or force buffering anyway.
+- **Named (server-side) cursors** (`conn.cursor(name=...)`, `DECLARE ... CURSOR`)
+  can be open side by side within a transaction and fetched in any order. They need
+  a transaction: in autocommit mode only `WITH HOLD` cursors survive, and those
+  materialise the whole result at commit, which is what this plan wants to avoid.
+
+So: named cursors, inside a transaction.
+
+### Auto-commit queries get a transaction
+
+Neo4j commits an auto-commit query when its result has been fully consumed or
+discarded, which matches a transaction around the cursor:
+
+1. `RUN` outside an explicit transaction: `BEGIN`, `DECLARE boltage_c<qid> NO SCROLL
+   CURSOR FOR SELECT * FROM cypher(...)`, then `FETCH` the first batch to surface
+   errors and measure `t_first`, as Neo4j reports errors on `RUN` where it can.
+2. `PULL {n}`: `FETCH n` (or batches of e.g. 1000 rows for `n = -1`), parse, send the
+   records, reply `has_more: true` while rows remain.
+3. When the cursor is exhausted: `CLOSE`, `COMMIT`, then the final `SUCCESS` with
+   the bookmark (and, with plan 02, the write statistics).
+4. `DISCARD`: `MOVE FORWARD ALL` (so writes in the rest of the query still execute,
+   as in Neo4j), `CLOSE`, `COMMIT`.
+5. `RESET`, a failure, or the client disconnecting before the end: `ROLLBACK`,
+   like Neo4j rolls back an unconsumed auto-commit query.
+
+This is the per-query transaction from the
+[shared groundwork](README.md#shared-groundwork); plan 02 uses the same one.
+
+### Explicit transactions
+
+1. Each `RUN` declares its own cursor (`boltage_c<qid>`), and `PULL {qid}` fetches
+   from the matching cursor. `self.results` in `BoltConnection` holds cursors
+   instead of row lists.
+2. `COMMIT` with results still open: drain them (`MOVE FORWARD ALL`) before
+   committing, so their writes happen, then `COMMIT`. `ROLLBACK` just rolls back.
+3. A failure while fetching (PostgreSQL errors can surface mid-result, for example
+   a division by zero on row 5000) fails the transaction, as in Neo4j.
+
+### Writes inside cursors
+
+AGE's write clauses run as custom scan nodes that execute as rows are pulled through
+them. **Spike:** confirm that
+
+- `DECLARE CURSOR` accepts a `cypher()` call containing `CREATE`/`MERGE`/`SET`/`DELETE`
+  (it is syntactically a `SELECT`, but PostgreSQL rejects data-modifying statements
+  in cursors in some cases);
+- partially fetching and then `MOVE FORWARD ALL` gives the same end state as
+  `fetchall()`.
+
+If either fails, keep the current fully-buffered path for queries that write (known
+from the masked-text keyword check, see plan 02) and stream only reads. Write
+queries rarely return large results, so that is an acceptable fallback.
+
+### Smaller improvements in the same change
+
+- `pull()` sends each record with its own `write` + `drain`. Pack a batch of records
+  into one buffer and write it once per batch.
+- Parse agtype per fetched batch rather than up front.
+- Queries without a `RETURN` (no columns) have no rows to stream; skip the cursor
+  and execute them directly inside the transaction.
+- System and emulated procedure results (`_run_system`) are small and stay as lists;
+  give `Result` a common interface (`fetch(n)`) with a list-backed and a
+  cursor-backed implementation.
+
+## Memory and performance
+
+- boltage's memory per query becomes one batch instead of the whole result.
+- PostgreSQL holds the cursor's state, not the result, for plain plans; sorts and
+  `DISTINCT` still materialise on the PostgreSQL side, which is where they belong.
+- A slow client keeps a transaction open for as long as it takes to pull. Neo4j has
+  the same behaviour; document that long-open auto-commit results hold locks and
+  delay vacuum, and that `idle_in_transaction_session_timeout` in PostgreSQL bounds it.
+- Extra round trips for `BEGIN`/`DECLARE`/`COMMIT`: use psycopg's pipeline mode to
+  send `BEGIN` and `DECLARE` together.
+
+## Tests
+
+- The existing "large result over several PULLs" smoke test keeps passing.
+- A query returning more rows than one batch, pulled with `fetch_size` small, gives
+  all rows in order.
+- `DISCARD` of a write query's result still applies the write.
+- Two results open in one explicit transaction, pulled alternately.
+- A runtime error halfway through a result fails the `PULL`, and the session
+  recovers after `RESET`.
+- A disconnect mid-result rolls back an auto-commit write.
+- Optional: measure boltage's peak memory on `UNWIND range(1, 5000000) AS i RETURN i`
+  before and after.
+
+## README changes
+
+Remove the last Limitations bullet. Mention in "How it works" that results are
+streamed through a PostgreSQL cursor, and that an auto-commit query's transaction
+stays open until its result is consumed.