Stream results instead of fetching the entire set

plans/02-write-statistics.md

@@ -77,8 +77,8 @@ counters for the graph's schema before and after the query and take the differen
    `query_type` (`r`, `w` or `rw`, depending on whether there are columns and writes)
    instead of the regex.
 
-With streaming (plan 06) the "after" snapshot must be taken once the cursor is
-exhausted or drained, just before the final summary.
+Streaming (plan 06) does not affect this: only read queries go through a cursor, and
+queries that can write have run to completion when `RUN` returns.
 
 ## Alternatives considered
 

plans/06-streaming-results.md

@@ -1,5 +1,24 @@
 # Streaming results instead of fetching them fully
 
+## Status
+
+Done for read queries: they run through a cursor, in a transaction of their own
+outside an explicit Bolt transaction. On `UNWIND range(1, 3000000) AS i RETURN i,
+'some text in every row' AS t`, boltage's peak memory went from 1419 MB to 44 MB, the
+first record from 10.8 s to 0.5 s, and the whole query from 44 s to 33 s.
+
+Queries that can write are deliberately still buffered, unlike what this plan
+proposes below. The spike found that AGE performs writes as rows are fetched: fetching
+2 of 5 rows of `UNWIND ... CREATE ... RETURN` and committing creates 2 nodes. Writes in
+a cursor would therefore not be visible to the next query in the transaction, and
+their errors would move from `RUN` to `PULL`. Reads have none of this, and have no
+need for `MOVE FORWARD ALL`. Any query containing `CREATE`, `MERGE`, `SET`, `DELETE`,
+`REMOVE` or `CALL` is buffered, and runs in autocommit as before. The sections on
+draining below only apply if write results are ever streamed; that needs a way to
+keep write ordering, and write queries rarely return much.
+
+Tests: `tests/streaming_test.py`, plus additions to `tests/smoke_test.py`.
+
 ## Problem
 
 `Session._sql` calls `fetchall()` (`boltage/backend.py:178`), `_run_cypher` parses

plans/README.md

@@ -28,7 +28,9 @@ plans small:
    transaction-local counters, streaming (06) needs a transaction to keep a server-side
    cursor open, and `PROFILE` (01) needs a savepoint. All three want `Session.run` to
    wrap auto-commit queries in `BEGIN ... COMMIT` instead of relying on the
-   connection's autocommit mode. Do this once, in 06 or 02, whichever comes first.
+   connection's autocommit mode. Plan 06 only did this for read queries, which run
+   in a cursor's transaction; queries that can write still use autocommit, so 02 still
+   has to add it for them.
 3. **Unit tests.** The only test is the end-to-end `tests/smoke_test.py`. Most of
    this work is text rewriting in `cypher.py`, which is cheap to unit test without a
    database. Add `tests/test_cypher.py` (pytest) alongside the first plan that