# Streaming results instead of fetching them fully

## Problem

`Session._sql` calls `fetchall()` (`boltage/backend.py:178`), `_run_cypher` parses
every row with `agtype.parse`, and the whole list sits in `Result.rows` until the
client has pulled it (`BoltConnection.pull`, `boltage/bolt.py:327`). A
`MATCH (n) RETURN n` over a large graph holds the full result in boltage's memory
three times over (PostgreSQL's wire data, the agtype text, the parsed objects), and
the first record only reaches the client after the last one has been read from
PostgreSQL. Bolt's `PULL {n}` already lets the client ask for batches; boltage just
does not pass that on to PostgreSQL.

## Approach: server-side cursors

Declare a PostgreSQL cursor for the query and `FETCH` from it as the client pulls.

### Why named cursors, not single-row mode

psycopg offers two ways to stream:

- **`cursor.stream()`** (libpq single-row / chunked mode) works without a
  transaction, but occupies the connection until the result is fully read. Bolt
  allows several open results in one explicit transaction (each with a `qid`), and a
  client may `RUN` again before pulling the previous result. With one PostgreSQL
  connection per Bolt connection, that would deadlock or force buffering anyway.
- **Named (server-side) cursors** (`conn.cursor(name=...)`, `DECLARE ... CURSOR`)
  can be open side by side within a transaction and fetched in any order. They need
  a transaction: in autocommit mode only `WITH HOLD` cursors survive, and those
  materialise the whole result at commit, which is what this plan wants to avoid.

So: named cursors, inside a transaction.

### Auto-commit queries get a transaction

Neo4j commits an auto-commit query when its result has been fully consumed or
discarded, which matches a transaction around the cursor:

1. `RUN` outside an explicit transaction: `BEGIN`, `DECLARE boltage_c<qid> NO SCROLL
   CURSOR FOR SELECT * FROM cypher(...)`, then `FETCH` the first batch to surface
   errors and measure `t_first`, as Neo4j reports errors on `RUN` where it can.
2. `PULL {n}`: `FETCH n` (or batches of e.g. 1000 rows for `n = -1`), parse, send the
   records, reply `has_more: true` while rows remain.
3. When the cursor is exhausted: `CLOSE`, `COMMIT`, then the final `SUCCESS` with
   the bookmark (and, with plan 02, the write statistics).
4. `DISCARD`: `MOVE FORWARD ALL` (so writes in the rest of the query still execute,
   as in Neo4j), `CLOSE`, `COMMIT`.
5. `RESET`, a failure, or the client disconnecting before the end: `ROLLBACK`,
   like Neo4j rolls back an unconsumed auto-commit query.

This is the per-query transaction from the
[shared groundwork](README.md#shared-groundwork); plan 02 uses the same one.

### Explicit transactions

1. Each `RUN` declares its own cursor (`boltage_c<qid>`), and `PULL {qid}` fetches
   from the matching cursor. `self.results` in `BoltConnection` holds cursors
   instead of row lists.
2. `COMMIT` with results still open: drain them (`MOVE FORWARD ALL`) before
   committing, so their writes happen, then `COMMIT`. `ROLLBACK` just rolls back.
3. A failure while fetching (PostgreSQL errors can surface mid-result, for example
   a division by zero on row 5000) fails the transaction, as in Neo4j.

### Writes inside cursors

AGE's write clauses run as custom scan nodes that execute as rows are pulled through
them. **Spike:** confirm that

- `DECLARE CURSOR` accepts a `cypher()` call containing `CREATE`/`MERGE`/`SET`/`DELETE`
  (it is syntactically a `SELECT`, but PostgreSQL rejects data-modifying statements
  in cursors in some cases);
- partially fetching and then `MOVE FORWARD ALL` gives the same end state as
  `fetchall()`.

If either fails, keep the current fully-buffered path for queries that write (known
from the masked-text keyword check, see plan 02) and stream only reads. Write
queries rarely return large results, so that is an acceptable fallback.

### Smaller improvements in the same change

- `pull()` sends each record with its own `write` + `drain`. Pack a batch of records
  into one buffer and write it once per batch.
- Parse agtype per fetched batch rather than up front.
- Queries without a `RETURN` (no columns) have no rows to stream; skip the cursor
  and execute them directly inside the transaction.
- System and emulated procedure results (`_run_system`) are small and stay as lists;
  give `Result` a common interface (`fetch(n)`) with a list-backed and a
  cursor-backed implementation.

## Memory and performance

- boltage's memory per query becomes one batch instead of the whole result.
- PostgreSQL holds the cursor's state, not the result, for plain plans; sorts and
  `DISTINCT` still materialise on the PostgreSQL side, which is where they belong.
- A slow client keeps a transaction open for as long as it takes to pull. Neo4j has
  the same behaviour; document that long-open auto-commit results hold locks and
  delay vacuum, and that `idle_in_transaction_session_timeout` in PostgreSQL bounds it.
- Extra round trips for `BEGIN`/`DECLARE`/`COMMIT`: use psycopg's pipeline mode to
  send `BEGIN` and `DECLARE` together.

## Tests

- The existing "large result over several PULLs" smoke test keeps passing.
- A query returning more rows than one batch, pulled with `fetch_size` small, gives
  all rows in order.
- `DISCARD` of a write query's result still applies the write.
- Two results open in one explicit transaction, pulled alternately.
- A runtime error halfway through a result fails the `PULL`, and the session
  recovers after `RESET`.
- A disconnect mid-result rolls back an auto-commit write.
- Optional: measure boltage's peak memory on `UNWIND range(1, 5000000) AS i RETURN i`
  before and after.

## README changes

Remove the last Limitations bullet. Mention in "How it works" that results are
streamed through a PostgreSQL cursor, and that an auto-commit query's transaction
stays open until its result is consumed.