5.4 KiB
Write statistics
Problem
Neo4j reports what a write query changed in the stats entry of the final PULL
summary (nodes-created, relationships-deleted, properties-set, ...).
neo4j-browser shows these as "Created 2 nodes, set 4 properties". boltage sends no
stats (BoltConnection.pull, boltage/bolt.py:327), so the browser says
"(no changes, no records)". Drivers' summary.counters.contains_updates is always
false, which also breaks code that checks it.
AGE's cypher() does not report counts, and AGE writes label tables without firing
triggers, so triggers cannot count either.
Approach: PostgreSQL's transaction-local table statistics
pg_stat_xact_user_tables has, per table, the number of rows inserted, updated
and deleted by the current transaction so far (n_tup_ins, n_tup_upd,
n_tup_del). Every AGE label is a table in the graph's schema, and
ag_catalog.ag_label says whether it holds vertices or edges. Snapshot those
counters for the graph's schema before and after the query and take the difference.
-
Spike first. Confirm on the supported PostgreSQL versions (the image uses 18):
- the counters include the changes of a statement that has just finished, within the same transaction;
- a non-superuser can read them for tables they wrote to;
- how AGE's
SETandREMOVEshow up (anUPDATEof the row, son_tup_upd), andDETACH DELETE(deletes on edge tables and the vertex table); MERGEthat matches counts nothing,MERGEthat creates counts inserts.
-
Transaction per auto-commit query. The counters reset at commit, and the connection runs in autocommit mode (
boltage/backend.py:126). Wrap each auto-commit Cypher query inBEGIN/COMMITinSession.run(see shared groundwork). Inside an explicit Bolt transaction nothing changes: the counters accumulate, and the diff per query still works. -
Snapshot query. One query, run before and after:
SELECT l.kind, s.relname <> ALL(%(defaults)s) AS labelled, sum(s.n_tup_ins), sum(s.n_tup_upd), sum(s.n_tup_del) FROM pg_stat_xact_user_tables s JOIN ag_catalog.ag_label l ON l.relation = s.relid JOIN ag_catalog.ag_graph g ON g.graphid = l.graph WHERE g.name = %(graph)s GROUP BY 1, 2To avoid two extra round trips per query, send the "before" snapshot in the same pipeline as
BEGIN(psycopg'sconn.pipeline()), and skip both snapshots for queries that cannot write (noCREATE/MERGE/SET/DELETE/REMOVEkeyword). The existing keyword check atboltage/backend.py:232runs on the raw text, so a string literal like'SET'counts as a write; run it onscan().maskedinstead. A false positive only costs the snapshots, a false negative loses the counts. -
Mapping to Neo4j counters:
Neo4j counter Source nodes-created/nodes-deletedinserts / deletes on vertex tables relationships-created/relationships-deletedinserts / deletes on edge tables labels-addedinserts on vertex tables other than _ag_label_vertex(AGE has one label per node)labels-removeddeletes on vertex tables other than _ag_label_vertexproperties-setsee below contains-updatestrue when any of the above is non-zero Only send non-zero counters, like Neo4j, plus
contains-updates. -
properties-sethas no exact source: an update rewrites a row's wholepropertiesmap. Start with an approximation and document it:- for inserts: not counted (Neo4j counts the properties in
CREATE; leaving this out under-reports, but never claims changes that did not happen); - for updates: the number of updated rows.
A more exact count would need comparing old and new
propertiesmaps, which AGE does not expose. Revisit if users ask for it.
- for inserts: not counted (Neo4j counts the properties in
-
System commands.
CREATE DATABASE/DROP DATABASE(and later user management, plan 04) reportsystem-updates: 1andcontains-system-updates: true. -
Plumbing. Add a
statsdict toResult, filled inSession._run_cypherand sent byBoltConnection.pullin the finalSUCCESS. Use the counts to setquery_type(r,worrw, depending on whether there are columns and writes) instead of the regex.
Streaming (plan 06) does not affect this: only read queries go through a cursor, and
queries that can write have run to completion when RUN returns.
Alternatives considered
- Counting rows before and after with
count(*)per label table: slow on large graphs, wrong under concurrent writers, and blind to updates. - Rewriting the query to count, for example appending a
RETURN count(...): changes the query's result and does not work for queries that already return. - Statement-level triggers with transition tables: AGE does not fire triggers on its label tables.
Tests
Smoke test, using the driver's summary.counters:
CREATE (a:Person)-[:KNOWS]->(b:Person): 2 nodes, 1 relationship, 2 labels,contains_updates.MATCH (n:Person) SET n.x = 1: properties set equals the number of persons.MATCH (n:Person) DETACH DELETE n: nodes and relationships deleted.MATCH (n) RETURN n: no updates.- Same queries in an explicit transaction, two queries in one transaction report their own counts.
CREATE DATABASE: system updates.
README changes
Remove the write statistics bullet; add a note that properties-set is approximate.