boltage/README.md

# peo

[neo4j-browser](https://github.com/neo4j/neo4j-browser), cypher-shell and the Neo4j
drivers on top of PostgreSQL, using [Apache AGE](https://github.com/apache/age) as the
graph engine.

peo is a small Python server that speaks Neo4j's Bolt protocol and translates Cypher
into AGE `cypher()` calls. It can also serve neo4j-browser. A [container
image](#container-images) bundles everything, together with
[pgAdmin](https://www.pgadmin.org/) for working with the same data in SQL.

```
 neo4j-browser (static files)      peo (Python)                     PostgreSQL + AGE
 ┌──────────────────────┐  Bolt  ┌────────────────────────-──┐  SQL  ┌────────────────┐
 │ in your web browser  │───────▶│ Bolt server (TCP + WS)    │──────▶│ cypher('g',$$  │
 │ served on :7474      │ :7687  │ Cypher → AGE translation  │ :5432 │   ... $$)      │
 └──────────────────────┘        │ emulated Neo4j procedures │       └────────────────┘
                                 └─────────────────────────-─┘
```

## Installation

peo needs Python 3.11 or later and a PostgreSQL server with the AGE extension installed
and loaded (`shared_preload_libraries=age`).

```sh
pip install 'peo[binary]'
PEO_PG_HOST=localhost PEO_PG_DATABASE=postgres peo
```

The `binary` extra pulls in psycopg's prebuilt wheels; leave it out to use a psycopg
built against your system's libpq. `python -m peo` works the same as `peo`.

Neo4j clients can now connect to `neo4j://localhost:7687` with a PostgreSQL user name
and password.

peo does not include neo4j-browser itself. To serve it, install the `browser` extra
([Starlette](https://www.starlette.io/) and [uvicorn](https://www.uvicorn.org/)),
build neo4j-browser and point `PEO_BROWSER_DIR` at its `dist` directory; it is then
served on <http://localhost:7474/>. The [container images](#container-images) do all of
this for you.

```sh
pip install 'peo[binary,browser]'
PEO_PG_HOST=localhost PEO_PG_DATABASE=postgres PEO_BROWSER_DIR=/path/to/neo4j-browser/dist peo
```

## Configuration

peo is configured through environment variables:

| Variable | Default | Purpose |
| --- | --- | --- |
| `PEO_PG_HOST` | `127.0.0.1` | PostgreSQL host. |
| `PEO_PG_PORT` | `5432` | PostgreSQL port. |
| `PEO_PG_DATABASE` | `$POSTGRES_DB`, else `$POSTGRES_USER`, else `postgres` | PostgreSQL database holding the graphs. |
| `PEO_DEFAULT_GRAPH` | `neo4j` | Graph used when a client does not pick a database. |
| `PEO_BOLT_PORT` | `7687` | Port the Bolt server listens on (TCP and WebSocket). |
| `PEO_HTTP_PORT` | `7474` | Port neo4j-browser is served on, when `PEO_BROWSER_DIR` is set. |
| `PEO_BROWSER_DIR` | (unset) | neo4j-browser `dist` directory to serve; needs the `browser` extra. |
| `PEO_ADVERTISED_BOLT_PORT` | `$PEO_BOLT_PORT` | Bolt port advertised to the browser; set it when clients reach the Bolt port on another port. |
| `PEO_ADVERTISED_HOST` | `localhost` | Host reported in `SHOW DATABASES` and to clients that do not send a routing address. |
| `PEO_LOG_LEVEL` | `INFO` | `DEBUG` logs every generated SQL statement. |

## How it works

- **Databases are AGE graphs.** Every AGE graph shows up as a Neo4j database, and
  `:use <name>` switches between them. Clients that do not pick a database get `PEO_DEFAULT_GRAPH`
  (`neo4j`); the full container image and the compose setup create that graph on first start. `CREATE DATABASE x` and `DROP DATABASE x` on the `system`
  database call `create_graph` / `drop_graph`; AGE requires names of at least 3 characters.
- **Queries** are sent to AGE as `SELECT * FROM cypher('<graph>', $$ <query> $$) AS (...)`.
  peo works out the result columns from the final `RETURN` clause and inlines
  query parameters as Cypher literals.
- **Neo4j procedures** that neo4j-browser relies on (`dbms.components`, `SHOW DATABASES`,
  `db.labels`, `db.schema.visualization`, `SHOW PROCEDURES`, ...) are emulated from the
  AGE catalog.
- **Workarounds for AGE**: neo4j-browser's "expand node" query uses a pattern
  comprehension, and its relationship lookup uses `id(n) IN [...]`, which AGE rejects
  for lists of more than one element. peo rewrites both into equivalent Cypher
  that AGE accepts.
- **Authentication** is passed straight through: every Bolt connection opens its own
  PostgreSQL connection with the credentials you log in with.

## Querying the graph from SQL

AGE stores each node label and relationship type as a table in the graph's schema,
with all properties in one `agtype` column, `properties`; `properties ->> 'key'::text`
returns a property as text. For typed columns, define a view on top of the label table:

```sql
CREATE VIEW neo4j.movies AS
    SELECT id,
        properties ->> 'title'::text AS title,
        (properties ->> 'released'::text)::integer AS released
    FROM neo4j."Movie";
```

A plain view converts the properties every time it is read. To index the typed
columns, make it a materialized view and refresh it when the graph has changed; a
unique index on `id` lets the refresh run without blocking readers:

```sql
CREATE MATERIALIZED VIEW neo4j.movies AS
    SELECT id,
        properties ->> 'title'::text AS title,
        (properties ->> 'released'::text)::integer AS released
    FROM neo4j."Movie";
CREATE UNIQUE INDEX ON neo4j.movies (id);
CREATE INDEX ON neo4j.movies (released);

REFRESH MATERIALIZED VIEW CONCURRENTLY neo4j.movies;
```

Prefer (materialized) views over adding columns to the label tables themselves:

- The graph stays schema-free, like a Neo4j graph. A property value that does not fit
  the view's column type, such as `CREATE (:Movie {released: 'soon'})`, is still
  written; the view fails when it is read, and a refresh fails and keeps the
  materialized view's previous contents.
- A view can be changed or dropped at any time, while AGE does not allow dropping
  columns from its label tables.
- A view can use any SQL expression. A generated column only accepts immutable
  expressions, which rules out some conversions, see [Dates and times](#dates-and-times).
- Creating a view does not rewrite or lock the label table.

### Dates and times

AGE has no date or time types, so store a point in time either as a Unix timestamp
(seconds since 1970-01-01 UTC) or as an ISO 8601 string with a UTC offset, and convert
it in the view:

```sql
CREATE VIEW neo4j.movie_dates AS
    SELECT id,
        to_timestamp((properties ->> 'updated'::text)::bigint) AT TIME ZONE 'UTC' AS updated,
        (properties ->> 'premiere'::text)::timestamptz AS premiere
    FROM neo4j."Movie";
```

Here `updated` is a Unix timestamp, such as `1729500000`, that becomes a `timestamp` in
UTC, and `premiere` is a string, such as `'1999-03-24T19:00:00-08:00'`, that becomes a
`timestamptz`. Casting a string without an offset to `timestamptz` reads it in the
`TimeZone` of the session that reads (or refreshes) the view, so always include one.

### Generated columns

If a typed column has to be in the label table itself, kept up to date on every Cypher
`CREATE`, `MERGE`, `SET` and `REMOVE`, add it as a stored generated column:

```sql
ALTER TABLE neo4j."Movie"
    ADD COLUMN released integer GENERATED ALWAYS AS ((properties ->> 'released'::text)::integer) STORED;
```

Unlike a view, this makes Cypher writes fail when a value does not fit the column's
type, the column cannot be removed again, and adding it rewrites and locks the table.

Generated columns also only accept immutable expressions. The Unix timestamp
conversion above is immutable, but casting text to `timestamptz` is not: its result
depends on the `TimeZone` setting for strings without an offset, on `DateStyle` for
formats like `'01/02/2024'`, and on the clock for strings like `'now'`. So
`GENERATED ALWAYS AS ((properties ->> 'premiere'::text)::timestamptz)` is rejected with
`generation expression is not immutable`. It takes a function that only accepts strings
for which the cast always gives the same result, ISO 8601 with a UTC offset, and
declares itself immutable:

```sql
CREATE FUNCTION public.iso8601_timestamptz(value text) RETURNS timestamptz
    LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
    SET TimeZone = 'UTC'
AS $$
BEGIN
    IF value !~ '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}(:?\d{2})?)$' THEN
        RAISE EXCEPTION 'not an ISO 8601 timestamp with a UTC offset: %', value;
    END IF;
    RETURN value::timestamptz;
END
$$;

ALTER TABLE neo4j."Movie"
    ADD COLUMN premiere timestamptz GENERATED ALWAYS AS
        (public.iso8601_timestamptz(properties ->> 'premiere'::text)) STORED;
```

PostgreSQL trusts the `IMMUTABLE` label without checking it, so the regular expression
is what makes it true. A view needs none of this.

## Container images

The [`Containerfile`](Containerfile) builds two images:

- **full** (the default): PostgreSQL with AGE, peo, neo4j-browser and pgAdmin in one
  container.
- **slim** (`--target slim`): only peo and neo4j-browser, for a PostgreSQL server
  with AGE that runs elsewhere. [`compose.yaml`](compose.yaml) runs it next to
  separate PostgreSQL and pgAdmin containers.

Any OCI-compatible build tool works. Docker does not pick up a `Containerfile` by
itself, so pass it explicitly: `docker build -f Containerfile -t peo .`. `docker`
takes the same options as `podman` in the commands below.

### Everything in one container

```sh
podman build -t peo .
podman run -d --name peo -e POSTGRES_PASSWORD=secret \
    -p 7474:7474 -p 7687:7687 -p 5050:5050 -p 5432:5432 \
    -v peo-data:/var/lib/postgresql -v peo-pgadmin:/var/lib/pgadmin peo
```

| Port | Service | Log in with |
| --- | --- | --- |
| 7474 | neo4j-browser: <http://localhost:7474/> | PostgreSQL user `postgres` and `POSTGRES_PASSWORD` |
| 7687 | Bolt, for Neo4j drivers and cypher-shell (`neo4j://localhost:7687`) | same as above |
| 5050 | pgAdmin: <http://localhost:5050/> | `admin@example.com` and `POSTGRES_PASSWORD` |
| 5432 | PostgreSQL, for `psql` and other SQL clients | same as neo4j-browser |

pgAdmin comes with the bundled PostgreSQL server already registered as "peo";
it asks for the PostgreSQL password the first time you open it. AGE stores each
graph as a schema of the same name, with one table per label.

Besides peo's own variables above, the image uses these:

| Variable | Default | Purpose |
| --- | --- | --- |
| `POSTGRES_PASSWORD` | (required) | Password of the PostgreSQL superuser, see the [postgres image docs](https://hub.docker.com/_/postgres). `POSTGRES_USER` and `POSTGRES_DB` work too. |
| `PEO_DEFAULT_GRAPH` | `neo4j` | Also the graph created when the database is initialised. |
| `PGADMIN_DEFAULT_EMAIL` | `admin@example.com` | pgAdmin login, created on first start. |
| `PGADMIN_DEFAULT_PASSWORD` | `$POSTGRES_PASSWORD` | pgAdmin login password, set on first start. |
| `PEO_PGADMIN` | `true` | Set to `false` to not start pgAdmin. |
| `PEO_PGADMIN_PORT` | `5050` | Port pgAdmin listens on inside the container. |

pgAdmin keeps its users and settings in `/var/lib/pgadmin`; changing
`PGADMIN_DEFAULT_*` after the first start has no effect unless that directory is empty.

### peo with a separate PostgreSQL server

The slim image runs peo and serves neo4j-browser, and is configured with the
variables in [Configuration](#configuration). Point it at any PostgreSQL server
that has AGE installed and loaded (`shared_preload_libraries=age`):

```sh
podman build --target slim -t peo:slim .
podman run -d --name peo -p 7474:7474 -p 7687:7687 \
    -e PEO_PG_HOST=db.example.com -e PEO_PG_DATABASE=graphs peo:slim
```

peo creates no graphs of its own. If the database has none yet, run
`CREATE DATABASE neo4j` on the `system` database (`:use system` in neo4j-browser),
or `SELECT ag_catalog.create_graph('neo4j');` in SQL.

### Compose: peo, PostgreSQL and pgAdmin as separate containers

[`compose.yaml`](compose.yaml) builds the slim image and runs it with
[`apache/age`](https://hub.docker.com/r/apache/age) for PostgreSQL and the official
[`dpage/pgadmin4`](https://hub.docker.com/r/dpage/pgadmin4) image:

```sh
echo POSTGRES_PASSWORD=secret > .env
podman compose up -d
```

It works with `podman compose` and `docker compose`, and publishes the same ports and
logins as the full image: neo4j-browser on 7474, Bolt on 7687, pgAdmin on 5050 and
PostgreSQL on 5432. Set `PEO_POSTGRES_PORT` to publish PostgreSQL on another host
port. The default graph (`PEO_DEFAULT_GRAPH`, `neo4j`) is created when the database
is initialised, and pgAdmin comes with the server already registered as "peo".
`PGADMIN_DEFAULT_EMAIL`, `PGADMIN_DEFAULT_PASSWORD` and `PEO_LOG_LEVEL` can be set
in `.env` too. Data is kept in the `postgres-data` and `pgadmin-data` volumes;
`podman compose down -v` removes them.

### Build arguments

`AGE_IMAGE` (base of the full image, default `apache/age:release_PG18_1.8.0`),
`SLIM_BASE_IMAGE` (base of the slim image, default `debian:trixie-slim`),
`NEO4J_BROWSER_REF` (neo4j-browser commit to build) and `PGADMIN_VERSION` (pgAdmin in
the full image, default `9.18`).

## Limitations

This is a first version. Things that do not work (yet):

- `RETURN *`, `EXPLAIN` and `PROFILE`.
- Write statistics: write queries report "(no changes, no records)" even when they
  changed the graph.
- Anything AGE does not support, for example multiple labels per node, pattern
  comprehensions, temporal and spatial types, indexes and constraints through Cypher.
- User and role management, `SHOW TRANSACTIONS`, `:sysinfo`.
- TLS (`bolt+s://`, HTTPS).
- Each query's full result is fetched from PostgreSQL before it is streamed to the
  client, so very large results use memory in peo.

## Development

peo depends on `psycopg` 3 and `websockets`, plus Starlette and uvicorn for the
`browser` extra. To work on it, install it in editable mode; `python -m build` (or
`uv build`) builds an sdist and wheel:

```sh
pip install -e '.[binary,browser]'
```

[`tests/smoke_test.py`](tests/smoke_test.py) runs end-to-end checks through the official
Neo4j Python driver against a running instance:

```sh
pip install neo4j
python tests/smoke_test.py neo4j://localhost:7687 postgres secret
```

## License

peo is free software, licensed under the GNU General Public License v3.0 or
later; see [LICENSE](LICENSE). The container images bundle third-party software
under their own licenses, including neo4j-browser (GPL-3.0), Apache AGE
(Apache-2.0), PostgreSQL and pgAdmin (PostgreSQL License). See
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for the full list and where
each license text and source archive is in the image.

Neo4j is a trademark of Neo4j, Inc.; peo is not affiliated with or endorsed
by Neo4j.