boltage/README.md

# boltage

[neo4j-browser](https://github.com/neo4j/neo4j-browser), cypher-shell and the Neo4j
drivers on top of PostgreSQL, using [Apache AGE](https://github.com/apache/age) as the
graph engine.

boltage is a small Python server that speaks Neo4j's Bolt protocol and translates Cypher
into AGE `cypher()` calls. It can also serve neo4j-browser. A [container
image](#container-images) bundles everything, together with
[pgAdmin](https://www.pgadmin.org/) for working with the same data in SQL.

```
 neo4j-browser (static files)      boltage (Python)                 PostgreSQL + AGE
 ┌──────────────────────┐  Bolt  ┌────────────────────────-──┐  SQL  ┌────────────────┐
 │ in your web browser  │───────▶│ Bolt server (TCP + WS)    │──────▶│ cypher('g',$$  │
 │ served on :7474      │ :7687  │ Cypher → AGE translation  │ :5432 │   ... $$)      │
 └──────────────────────┘        │ emulated Neo4j procedures │       └────────────────┘
                                 └─────────────────────────-─┘
```

## Installation

boltage needs Python 3.11 or later and a PostgreSQL server with the AGE extension installed
and loaded (`shared_preload_libraries=age`).

```sh
pip install 'boltage[binary]'
BOLTAGE_PG_HOST=localhost BOLTAGE_PG_DATABASE=postgres boltage
```

The `binary` extra pulls in psycopg's prebuilt wheels; leave it out to use a psycopg
built against your system's libpq. `python -m boltage` works the same as `boltage`.

Neo4j clients can now connect to `neo4j://localhost:7687` with a PostgreSQL user name
and password.

boltage does not include neo4j-browser itself. To serve it, install the `browser` extra
([Starlette](https://www.starlette.io/) and [uvicorn](https://www.uvicorn.org/)),
build neo4j-browser and point `BOLTAGE_BROWSER_DIR` at its `dist` directory; it is then
served on <http://localhost:7474/>. The [container images](#container-images) do all of
this for you.

```sh
pip install 'boltage[binary,browser]'
BOLTAGE_PG_HOST=localhost BOLTAGE_PG_DATABASE=postgres BOLTAGE_BROWSER_DIR=/path/to/neo4j-browser/dist boltage
```

## Configuration

boltage is configured through environment variables:

| Variable | Default | Purpose |
| --- | --- | --- |
| `BOLTAGE_PG_HOST` | `127.0.0.1` | PostgreSQL host. |
| `BOLTAGE_PG_PORT` | `5432` | PostgreSQL port. |
| `BOLTAGE_PG_DATABASE` | `$POSTGRES_DB`, else `$POSTGRES_USER`, else `postgres` | PostgreSQL database holding the graphs. |
| `BOLTAGE_DEFAULT_GRAPH` | `neo4j` | Graph used when a client does not pick a database. |
| `BOLTAGE_BOLT_PORT` | `7687` | Port the Bolt server listens on (TCP and WebSocket). |
| `BOLTAGE_HTTP_PORT` | `7474` | Port neo4j-browser is served on, when `BOLTAGE_BROWSER_DIR` is set. |
| `BOLTAGE_BROWSER_DIR` | (unset) | neo4j-browser `dist` directory to serve; needs the `browser` extra. |
| `BOLTAGE_ADVERTISED_BOLT_PORT` | `$BOLTAGE_BOLT_PORT` | Bolt port advertised to the browser; set it when clients reach the Bolt port on another port. |
| `BOLTAGE_ADVERTISED_HOST` | `localhost` | Host reported in `SHOW DATABASES` and to clients that do not send a routing address. |
| `BOLTAGE_LOG_LEVEL` | `INFO` | `DEBUG` logs every generated SQL statement. |

## How it works

- **Databases are AGE graphs.** Every AGE graph shows up as a Neo4j database, and
  `:use <name>` switches between them. Clients that do not pick a database get `BOLTAGE_DEFAULT_GRAPH`
  (`neo4j`); the full container image and the compose setup create that graph on first start. `CREATE DATABASE x` and `DROP DATABASE x` on the `system`
  database call `create_graph` / `drop_graph`; AGE requires names of at least 3 characters.
- **Queries** are sent to AGE as `SELECT * FROM cypher('<graph>', $$ <query> $$) AS (...)`.
  boltage works out the result columns from the final `RETURN` clause and inlines
  query parameters as Cypher literals.
- **Neo4j procedures** that neo4j-browser relies on (`dbms.components`, `SHOW DATABASES`,
  `db.labels`, `db.schema.visualization`, `SHOW PROCEDURES`, ...) are emulated from the
  AGE catalog.
- **Workarounds for AGE**: neo4j-browser's "expand node" query uses a pattern
  comprehension, and its relationship lookup uses `id(n) IN [...]`, which AGE rejects
  for lists of more than one element. boltage rewrites both into equivalent Cypher
  that AGE accepts.
- **Authentication** is passed straight through: every Bolt connection opens its own
  PostgreSQL connection with the credentials you log in with.

## Querying the graph from SQL

AGE stores each node label and relationship type as a table in the graph's schema,
with all properties in one `agtype` column, `properties`; `properties ->> 'key'::text`
returns a property as text. For typed columns, define a view on top of the label table:

```sql
CREATE VIEW neo4j.movies AS
    SELECT id,
        properties ->> 'title'::text AS title,
        (properties ->> 'released'::text)::integer AS released
    FROM neo4j."Movie";
```

A plain view converts the properties every time it is read. To index the typed
columns, make it a materialized view and refresh it when the graph has changed; a
unique index on `id` lets the refresh run without blocking readers:

```sql
CREATE MATERIALIZED VIEW neo4j.movies AS
    SELECT id,
        properties ->> 'title'::text AS title,
        (properties ->> 'released'::text)::integer AS released
    FROM neo4j."Movie";
CREATE UNIQUE INDEX ON neo4j.movies (id);
CREATE INDEX ON neo4j.movies (released);

REFRESH MATERIALIZED VIEW CONCURRENTLY neo4j.movies;
```

Prefer (materialized) views over adding columns to the label tables themselves:

- The graph stays schema-free, like a Neo4j graph. A property value that does not fit
  the view's column type, such as `CREATE (:Movie {released: 'soon'})`, is still
  written; the view fails when it is read, and a refresh fails and keeps the
  materialized view's previous contents.
- A view can be changed or dropped at any time, while AGE does not allow dropping
  columns from its label tables.
- A view can use any SQL expression. A generated column only accepts immutable
  expressions, which rules out some conversions, see [Dates and times](#dates-and-times).
- Creating a view does not rewrite or lock the label table.

### Dates and times

AGE has no date or time types, so store a point in time either as a Unix timestamp
(seconds since 1970-01-01 UTC) or as an ISO 8601 string with a UTC offset, and convert
it in the view:

```sql
CREATE VIEW neo4j.movie_dates AS
    SELECT id,
        to_timestamp((properties ->> 'updated'::text)::bigint) AT TIME ZONE 'UTC' AS updated,
        (properties ->> 'premiere'::text)::timestamptz AS premiere
    FROM neo4j."Movie";
```

Here `updated` is a Unix timestamp, such as `1729500000`, that becomes a `timestamp` in
UTC, and `premiere` is a string, such as `'1999-03-24T19:00:00-08:00'`, that becomes a
`timestamptz`. Casting a string without an offset to `timestamptz` reads it in the
`TimeZone` of the session that reads (or refreshes) the view, so always include one.

### Generated columns

If a typed column has to be in the label table itself, kept up to date on every Cypher
`CREATE`, `MERGE`, `SET` and `REMOVE`, add it as a stored generated column:

```sql
ALTER TABLE neo4j."Movie"
    ADD COLUMN released integer GENERATED ALWAYS AS ((properties ->> 'released'::text)::integer) STORED;
```

Unlike a view, this makes Cypher writes fail when a value does not fit the column's
type, the column cannot be removed again, and adding it rewrites and locks the table.

Generated columns also only accept immutable expressions. The Unix timestamp
conversion above is immutable, but casting text to `timestamptz` is not: its result
depends on the `TimeZone` setting for strings without an offset, on `DateStyle` for
formats like `'01/02/2024'`, and on the clock for strings like `'now'`. So
`GENERATED ALWAYS AS ((properties ->> 'premiere'::text)::timestamptz)` is rejected with
`generation expression is not immutable`. It takes a function that only accepts strings
for which the cast always gives the same result, ISO 8601 with a UTC offset, and
declares itself immutable:

```sql
CREATE FUNCTION public.iso8601_timestamptz(value text) RETURNS timestamptz
    LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
    SET TimeZone = 'UTC'
AS $$
BEGIN
    IF value !~ '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}(:?\d{2})?)$' THEN
        RAISE EXCEPTION 'not an ISO 8601 timestamp with a UTC offset: %', value;
    END IF;
    RETURN value::timestamptz;
END
$$;

ALTER TABLE neo4j."Movie"
    ADD COLUMN premiere timestamptz GENERATED ALWAYS AS
        (public.iso8601_timestamptz(properties ->> 'premiere'::text)) STORED;
```

PostgreSQL trusts the `IMMUTABLE` label without checking it, so the regular expression
is what makes it true. A view needs none of this.

## Exposing existing tables as a graph

AGE only queries its own label tables, so to query existing tables with Cypher, mirror
them into a graph with triggers. The graph then changes in the same transaction as the
tables. Take these tables:

```sql
CREATE TABLE public.customer (customer_id bigint PRIMARY KEY, name text, city text);
CREATE TABLE public.product (product_id bigint PRIMARY KEY, title text, price numeric);
CREATE TABLE public.purchase (
    customer_id bigint REFERENCES public.customer,
    product_id bigint REFERENCES public.product,
    bought_at timestamptz,
    quantity integer,
    PRIMARY KEY (customer_id, product_id)
);
```

Create a graph with a label for each table. AGE needs `ag_catalog` in the `search_path`
to create them:

```sql
SET search_path = ag_catalog, "$user", public;
SELECT ag_catalog.create_graph('shop');
SELECT ag_catalog.create_vlabel('shop', 'Customer');
SELECT ag_catalog.create_vlabel('shop', 'Product');
SELECT ag_catalog.create_elabel('shop', 'BOUGHT');
```

A vertex id is a label id and a 48-bit number, so `_graphid` turns a primary key into a
vertex id. Edges can then be built straight from the foreign keys. Every change to a
row replaces its vertex:

```sql
CREATE FUNCTION public.customer_to_graph() RETURNS trigger
    LANGUAGE plpgsql SECURITY DEFINER SET search_path = ag_catalog, pg_catalog
AS $$
BEGIN
    IF TG_OP IN ('UPDATE', 'DELETE') THEN
        DELETE FROM shop."Customer"
        WHERE id = _graphid(_label_id('shop', 'Customer'), OLD.customer_id);
    END IF;
    IF TG_OP IN ('INSERT', 'UPDATE') THEN
        INSERT INTO shop."Customer" (id, properties)
        VALUES (_graphid(_label_id('shop', 'Customer'), NEW.customer_id),
                agtype_build_map('customer_id', NEW.customer_id, 'name', NEW.name, 'city', NEW.city));
    END IF;
    RETURN NULL;
END
$$;

CREATE TRIGGER to_graph AFTER INSERT OR UPDATE OR DELETE ON public.customer
    FOR EACH ROW EXECUTE FUNCTION public.customer_to_graph();
```

`public.product` gets the same trigger for `Product`. Edges keep the ids AGE gives
them and are found by their endpoints:

```sql
CREATE FUNCTION public.purchase_to_graph() RETURNS trigger
    LANGUAGE plpgsql SECURITY DEFINER SET search_path = ag_catalog, pg_catalog
AS $$
BEGIN
    IF TG_OP IN ('UPDATE', 'DELETE') THEN
        DELETE FROM shop."BOUGHT"
        WHERE start_id = _graphid(_label_id('shop', 'Customer'), OLD.customer_id)
          AND end_id = _graphid(_label_id('shop', 'Product'), OLD.product_id);
    END IF;
    IF TG_OP IN ('INSERT', 'UPDATE') THEN
        INSERT INTO shop."BOUGHT" (start_id, end_id, properties)
        VALUES (_graphid(_label_id('shop', 'Customer'), NEW.customer_id),
                _graphid(_label_id('shop', 'Product'), NEW.product_id),
                agtype_build_map('bought_at', NEW.bought_at::text, 'quantity', NEW.quantity));
    END IF;
    RETURN NULL;
END
$$;

CREATE TRIGGER to_graph AFTER INSERT OR UPDATE OR DELETE ON public.purchase
    FOR EACH ROW EXECUTE FUNCTION public.purchase_to_graph();
```

To copy the rows that are already there, update each of them once, vertices first:

```sql
UPDATE public.customer SET customer_id = customer_id;
UPDATE public.product SET product_id = product_id;
UPDATE public.purchase SET customer_id = customer_id;
```

This rewrites every row; for large tables, an `INSERT INTO shop."Customer" ... SELECT`
with the same expressions is faster. After that, `MATCH (c:Customer)-[:BOUGHT]->(p:Product)`
works on the `shop` database, including unlabelled and variable-length patterns. Some
things to keep in mind:

- This only works in one direction. AGE writes label tables without firing triggers, so
  a Cypher `CREATE`, `SET` or `DELETE` on a mirrored label does not reach the tables,
  and the next change to the row overwrites it. AGE does check table privileges, so
  grant the users that log in through boltage only `SELECT` on the mirrored label tables.
  The triggers are `SECURITY DEFINER`, so relational writes still reach the graph.
  Superusers bypass privileges.
- Vertex ids need a `bigint` primary key below 2<sup>48</sup>. Other keys, such as
  composite keys or UUIDs, need a table that maps each key to a number.
- Updating a row in `public.purchase` gives its edge a new id; vertex ids never change.
- The data is stored twice, and every write to the tables also writes to the graph.

## Container images

The [`Containerfile`](Containerfile) builds two images:

- **full** (the default): PostgreSQL with AGE, boltage, neo4j-browser and pgAdmin in one
  container.
- **slim** (`--target slim`): only boltage and neo4j-browser, for a PostgreSQL server
  with AGE that runs elsewhere. [`compose.yaml`](compose.yaml) runs it next to
  separate PostgreSQL and pgAdmin containers.

Any OCI-compatible build tool works. Docker does not pick up a `Containerfile` by
itself, so pass it explicitly: `docker build -f Containerfile -t boltage .`. `docker`
takes the same options as `podman` in the commands below.

### Everything in one container

```sh
podman build -t boltage .
podman run -d --name boltage -e POSTGRES_PASSWORD=secret \
    -p 7474:7474 -p 7687:7687 -p 5050:5050 -p 5432:5432 \
    -v boltage-data:/var/lib/postgresql -v boltage-pgadmin:/var/lib/pgadmin boltage
```

| Port | Service | Log in with |
| --- | --- | --- |
| 7474 | neo4j-browser: <http://localhost:7474/> | PostgreSQL user `postgres` and `POSTGRES_PASSWORD` |
| 7687 | Bolt, for Neo4j drivers and cypher-shell (`neo4j://localhost:7687`) | same as above |
| 5050 | pgAdmin: <http://localhost:5050/> | `admin@example.com` and `POSTGRES_PASSWORD` |
| 5432 | PostgreSQL, for `psql` and other SQL clients | same as neo4j-browser |

pgAdmin comes with the bundled PostgreSQL server already registered as "boltage";
it asks for the PostgreSQL password the first time you open it. AGE stores each
graph as a schema of the same name, with one table per label.

Besides boltage's own variables above, the image uses these:

| Variable | Default | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | (required) | Password of the PostgreSQL superuser, see the [postgres image docs](https://hub.docker.com/_/postgres). `POSTGRES_USER` and `POSTGRES_DB` work too. |
| `BOLTAGE_DEFAULT_GRAPH` | `neo4j` | Also the graph created when the database is initialised. |
| `PGADMIN_DEFAULT_EMAIL` | `admin@example.com` | pgAdmin login, created on first start. |
| `PGADMIN_DEFAULT_PASSWORD` | `$POSTGRES_PASSWORD` | pgAdmin login password, set on first start. |
| `BOLTAGE_PGADMIN` | `true` | Set to `false` to not start pgAdmin. |
| `BOLTAGE_PGADMIN_PORT` | `5050` | Port pgAdmin listens on inside the container. |

pgAdmin keeps its users and settings in `/var/lib/pgadmin`; changing
`PGADMIN_DEFAULT_*` after the first start has no effect unless that directory is empty.

### boltage with a separate PostgreSQL server

The slim image runs boltage and serves neo4j-browser, and is configured with the
variables in [Configuration](#configuration). Point it at any PostgreSQL server
that has AGE installed and loaded (`shared_preload_libraries=age`):

```sh
podman build --target slim -t boltage:slim .
podman run -d --name boltage -p 7474:7474 -p 7687:7687 \
    -e BOLTAGE_PG_HOST=db.example.com -e BOLTAGE_PG_DATABASE=graphs boltage:slim
```

boltage creates no graphs of its own. If the database has none yet, run
`CREATE DATABASE neo4j` on the `system` database (`:use system` in neo4j-browser),
or `SELECT ag_catalog.create_graph('neo4j');` in SQL.

### Compose: boltage, PostgreSQL and pgAdmin as separate containers

[`compose.yaml`](compose.yaml) builds the slim image and runs it with
[`apache/age`](https://hub.docker.com/r/apache/age) for PostgreSQL and the official
[`dpage/pgadmin4`](https://hub.docker.com/r/dpage/pgadmin4) image:

```sh
echo POSTGRES_PASSWORD=secret > .env
podman compose up -d
```

It works with `podman compose` and `docker compose`, and publishes the same ports and
logins as the full image: neo4j-browser on 7474, Bolt on 7687, pgAdmin on 5050 and
PostgreSQL on 5432. Set `BOLTAGE_POSTGRES_PORT` to publish PostgreSQL on another host
port. The default graph (`BOLTAGE_DEFAULT_GRAPH`, `neo4j`) is created when the database
is initialised, and pgAdmin comes with the server already registered as "boltage".
`PGADMIN_DEFAULT_EMAIL`, `PGADMIN_DEFAULT_PASSWORD` and `BOLTAGE_LOG_LEVEL` can be set
in `.env` too. Data is kept in the `postgres-data` and `pgadmin-data` volumes;
`podman compose down -v` removes them.

### Build arguments

`AGE_IMAGE` (base of the full image, default `apache/age:release_PG18_1.8.0`),
`SLIM_BASE_IMAGE` (base of the slim image, default `debian:trixie-slim`),
`NEO4J_BROWSER_REF` (neo4j-browser commit to build) and `PGADMIN_VERSION` (pgAdmin in
the full image, default `9.18`).

## Limitations

This is a first version. Things that do not work (yet):

- `RETURN *`, `EXPLAIN` and `PROFILE`.
- Write statistics: write queries report "(no changes, no records)" even when they
  changed the graph.
- Anything AGE does not support, for example multiple labels per node, pattern
  comprehensions, temporal and spatial types, indexes and constraints through Cypher.
- User and role management, `SHOW TRANSACTIONS`, `:sysinfo`.
- TLS (`bolt+s://`, HTTPS).
- Each query's full result is fetched from PostgreSQL before it is streamed to the
  client, so very large results use memory in boltage.

## Development

boltage depends on `psycopg` 3 and `websockets`, plus Starlette and uvicorn for the
`browser` extra. To work on it, install it in editable mode; `python -m build` (or
`uv build`) builds an sdist and wheel:

```sh
pip install -e '.[binary,browser]'
```

[`tests/smoke_test.py`](tests/smoke_test.py) runs end-to-end checks through the official
Neo4j Python driver against a running instance:

```sh
pip install neo4j
python tests/smoke_test.py neo4j://localhost:7687 postgres secret
```

## License

boltage is free software, licensed under the GNU General Public License v3.0 or
later; see [LICENSE](LICENSE). The container images bundle third-party software
under their own licenses, including neo4j-browser (GPL-3.0), Apache AGE
(Apache-2.0), PostgreSQL and pgAdmin (PostgreSQL License). See
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for the full list and where
each license text and source archive is in the image.

Neo4j is a trademark of Neo4j, Inc.; boltage is not affiliated with or endorsed
by Neo4j.