Streaming results instead of fetching them fully

Status

Done for read queries: they run through a cursor, in a transaction of their own outside an explicit Bolt transaction. On UNWIND range(1, 3000000) AS i RETURN i, 'some text in every row' AS t, boltage's peak memory went from 1419 MB to 44 MB, the first record from 10.8 s to 0.5 s, and the whole query from 44 s to 33 s.

Queries that can write are deliberately still buffered, unlike what this plan proposes below. The spike found that AGE performs writes as rows are fetched: fetching 2 of 5 rows of UNWIND ... CREATE ... RETURN and committing creates 2 nodes. Writes in a cursor would therefore not be visible to the next query in the transaction, and their errors would move from RUN to PULL. Reads have none of this, and have no need for MOVE FORWARD ALL. Any query containing CREATE, MERGE, SET, DELETE, REMOVE or CALL is buffered, and runs in autocommit as before. The sections on draining below only apply if write results are ever streamed; that needs a way to keep write ordering, and write queries rarely return much.

Tests: tests/streaming_test.py, plus additions to tests/smoke_test.py.

Problem

Session._sql calls fetchall() (boltage/backend.py:178), _run_cypher parses every row with agtype.parse, and the whole list sits in Result.rows until the client has pulled it (BoltConnection.pull, boltage/bolt.py:327). A MATCH (n) RETURN n over a large graph holds the full result in boltage's memory three times over (PostgreSQL's wire data, the agtype text, the parsed objects), and the first record only reaches the client after the last one has been read from PostgreSQL. Bolt's PULL {n} already lets the client ask for batches; boltage just does not pass that on to PostgreSQL.

Approach: server-side cursors

Declare a PostgreSQL cursor for the query and FETCH from it as the client pulls.

Why named cursors, not single-row mode

psycopg offers two ways to stream:

cursor.stream() (libpq single-row / chunked mode) works without a transaction, but occupies the connection until the result is fully read. Bolt allows several open results in one explicit transaction (each with a qid), and a client may RUN again before pulling the previous result. With one PostgreSQL connection per Bolt connection, that would deadlock or force buffering anyway.
Named (server-side) cursors (conn.cursor(name=...), DECLARE ... CURSOR) can be open side by side within a transaction and fetched in any order. They need a transaction: in autocommit mode only WITH HOLD cursors survive, and those materialise the whole result at commit, which is what this plan wants to avoid.

So: named cursors, inside a transaction.

Auto-commit queries get a transaction

Neo4j commits an auto-commit query when its result has been fully consumed or discarded, which matches a transaction around the cursor:

RUN outside an explicit transaction: BEGIN, DECLARE boltage_c<qid> NO SCROLL CURSOR FOR SELECT * FROM cypher(...), then FETCH the first batch to surface errors and measure t_first, as Neo4j reports errors on RUN where it can.
PULL {n}: FETCH n (or batches of e.g. 1000 rows for n = -1), parse, send the records, reply has_more: true while rows remain.
When the cursor is exhausted: CLOSE, COMMIT, then the final SUCCESS with the bookmark (and, with plan 02, the write statistics).
DISCARD: MOVE FORWARD ALL (so writes in the rest of the query still execute, as in Neo4j), CLOSE, COMMIT.
RESET, a failure, or the client disconnecting before the end: ROLLBACK, like Neo4j rolls back an unconsumed auto-commit query.

This is the per-query transaction from the shared groundwork; plan 02 uses the same one.

Explicit transactions

Each RUN declares its own cursor (boltage_c<qid>), and PULL {qid} fetches from the matching cursor. self.results in BoltConnection holds cursors instead of row lists.
COMMIT with results still open: drain them (MOVE FORWARD ALL) before committing, so their writes happen, then COMMIT. ROLLBACK just rolls back.
A failure while fetching (PostgreSQL errors can surface mid-result, for example a division by zero on row 5000) fails the transaction, as in Neo4j.

Writes inside cursors

AGE's write clauses run as custom scan nodes that execute as rows are pulled through them. Spike: confirm that

DECLARE CURSOR accepts a cypher() call containing CREATE/MERGE/SET/DELETE (it is syntactically a SELECT, but PostgreSQL rejects data-modifying statements in cursors in some cases);
partially fetching and then MOVE FORWARD ALL gives the same end state as fetchall().

If either fails, keep the current fully-buffered path for queries that write (known from the masked-text keyword check, see plan 02) and stream only reads. Write queries rarely return large results, so that is an acceptable fallback.

Smaller improvements in the same change

pull() sends each record with its own write + drain. Pack a batch of records into one buffer and write it once per batch.
Parse agtype per fetched batch rather than up front.
Queries without a RETURN (no columns) have no rows to stream; skip the cursor and execute them directly inside the transaction.
System and emulated procedure results (_run_system) are small and stay as lists; give Result a common interface (fetch(n)) with a list-backed and a cursor-backed implementation.

Memory and performance

boltage's memory per query becomes one batch instead of the whole result.
PostgreSQL holds the cursor's state, not the result, for plain plans; sorts and DISTINCT still materialise on the PostgreSQL side, which is where they belong.
A slow client keeps a transaction open for as long as it takes to pull. Neo4j has the same behaviour; document that long-open auto-commit results hold locks and delay vacuum, and that idle_in_transaction_session_timeout in PostgreSQL bounds it.
Extra round trips for BEGIN/DECLARE/COMMIT: use psycopg's pipeline mode to send BEGIN and DECLARE together.

Tests

The existing "large result over several PULLs" smoke test keeps passing.
A query returning more rows than one batch, pulled with fetch_size small, gives all rows in order.
DISCARD of a write query's result still applies the write.
Two results open in one explicit transaction, pulled alternately.
A runtime error halfway through a result fails the PULL, and the session recovers after RESET.
A disconnect mid-result rolls back an auto-commit write.
Optional: measure boltage's peak memory on UNWIND range(1, 5000000) AS i RETURN i before and after.

README changes

Remove the last Limitations bullet. Mention in "How it works" that results are streamed through a PostgreSQL cursor, and that an auto-commit query's transaction stays open until its result is consumed.

6.9 KiB Raw Blame History