Move peo package to a different spot to prepare it for PyPI
This commit is contained in:
Bart Broere committed
1 parent
9a25aa11c9
commit
3e8e498b56
16 files changed
+408
-59
No files matched your search
@@ -1,57 +1,73 @@
|
||||
# peo
|
||||
|
||||
[neo4j-browser](https://github.com/neo4j/neo4j-browser) running on PostgreSQL, using
|
||||
[Apache AGE](https://github.com/apache/age) as the graph engine.
|
||||
[neo4j-browser](https://github.com/neo4j/neo4j-browser), cypher-shell and the Neo4j
|
||||
drivers on top of PostgreSQL, using [Apache AGE](https://github.com/apache/age) as the
|
||||
graph engine.
|
||||
|
||||
Everything ships in one container image, together with
|
||||
[pgAdmin](https://www.pgadmin.org/) for working with the same data in SQL:
|
||||
peo is a small Python server that speaks Neo4j's Bolt protocol and translates Cypher
|
||||
into AGE `cypher()` calls. It can also serve neo4j-browser. A [container
|
||||
image](#container-image) bundles everything, together with
|
||||
[pgAdmin](https://www.pgadmin.org/) for working with the same data in SQL.
|
||||
|
||||
```
|
||||
neo4j-browser (static files) peo bridge (Python) PostgreSQL + AGE
|
||||
┌──────────────────────┐ Bolt ┌──────────────────────────┐ SQL ┌────────────────┐
|
||||
│ in your web browser │───────▶│ Bolt server (TCP + WS) │──────▶│ cypher('g',$$ │
|
||||
│ served on :7474 │ :7687 │ Cypher → AGE translation │ :5432 │ ... $$) │
|
||||
neo4j-browser (static files) peo (Python) PostgreSQL + AGE
|
||||
┌──────────────────────┐ Bolt ┌────────────────────────-──┐ SQL ┌────────────────┐
|
||||
│ in your web browser │───────▶│ Bolt server (TCP + WS) │──────▶│ cypher('g',$$ │
|
||||
│ served on :7474 │ :7687 │ Cypher → AGE translation │ :5432 │ ... $$) │
|
||||
└──────────────────────┘ │ emulated Neo4j procedures │ └────────────────┘
|
||||
└──────────────────────────┘
|
||||
└─────────────────────────-─┘
|
||||
```
|
||||
|
||||
## Quick start
|
||||
## Installation
|
||||
|
||||
peo needs Python 3.11 or later and a PostgreSQL server with the AGE extension installed
|
||||
and loaded (`shared_preload_libraries=age`).
|
||||
|
||||
```sh
|
||||
podman build -t peo .
|
||||
podman run -d --name peo -e POSTGRES_PASSWORD=secret \
|
||||
-p 7474:7474 -p 7687:7687 -p 5050:5050 -p 5432:5432 \
|
||||
-v peo-data:/var/lib/postgresql -v peo-pgadmin:/var/lib/pgadmin peo
|
||||
pip install 'peo[binary]'
|
||||
PEO_PG_HOST=localhost PEO_PG_DATABASE=postgres peo
|
||||
```
|
||||
|
||||
| Port | Service | Log in with |
|
||||
The `binary` extra pulls in psycopg's prebuilt wheels; leave it out to use a psycopg
|
||||
built against your system's libpq. `python -m peo` works the same as `peo`.
|
||||
|
||||
Neo4j clients can now connect to `neo4j://localhost:7687` with a PostgreSQL user name
|
||||
and password. peo does not include neo4j-browser itself: build it and point
|
||||
`PEO_BROWSER_DIR` at its `dist` directory to serve it on <http://localhost:7474/>, or
|
||||
use the container image.
|
||||
|
||||
## Configuration
|
||||
|
||||
peo is configured through environment variables:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| 7474 | neo4j-browser: <http://localhost:7474/> | PostgreSQL user `postgres` and `POSTGRES_PASSWORD` |
|
||||
| 7687 | Bolt, for Neo4j drivers and cypher-shell (`neo4j://localhost:7687`) | same as above |
|
||||
| 5050 | pgAdmin: <http://localhost:5050/> | `admin@example.com` and `POSTGRES_PASSWORD` |
|
||||
| 5432 | PostgreSQL, for `psql` and other SQL clients | same as neo4j-browser |
|
||||
|
||||
pgAdmin comes with the bundled PostgreSQL server already registered as "peo";
|
||||
it asks for the PostgreSQL password the first time you open it. AGE stores each
|
||||
graph as a schema of the same name, with one table per label.
|
||||
|
||||
`docker` works as a drop-in replacement for `podman` in the commands above.
|
||||
| `PEO_PG_HOST` | `127.0.0.1` | PostgreSQL host. |
|
||||
| `PEO_PG_PORT` | `5432` | PostgreSQL port. |
|
||||
| `PEO_PG_DATABASE` | `$POSTGRES_DB`, else `$POSTGRES_USER`, else `postgres` | PostgreSQL database holding the graphs. |
|
||||
| `PEO_DEFAULT_GRAPH` | `neo4j` | Graph used when a client does not pick a database. |
|
||||
| `PEO_BOLT_PORT` | `7687` | Port the Bolt server listens on (TCP and WebSocket). |
|
||||
| `PEO_HTTP_PORT` | `7474` | Port neo4j-browser is served on, when `PEO_BROWSER_DIR` is set. |
|
||||
| `PEO_BROWSER_DIR` | (unset) | neo4j-browser `dist` directory to serve. |
|
||||
| `PEO_ADVERTISED_BOLT_PORT` | `$PEO_BOLT_PORT` | Bolt port advertised to the browser; set it when clients reach the Bolt port on another port. |
|
||||
| `PEO_ADVERTISED_HOST` | `localhost` | Host reported in `SHOW DATABASES` and to clients that do not send a routing address. |
|
||||
| `PEO_LOG_LEVEL` | `INFO` | `DEBUG` logs every generated SQL statement. |
|
||||
|
||||
## How it works
|
||||
|
||||
- **Databases are AGE graphs.** Every AGE graph shows up as a Neo4j database, and
|
||||
`:use <name>` switches between them. A graph named `neo4j` is created on first start
|
||||
(`PEO_DEFAULT_GRAPH`). `CREATE DATABASE x` and `DROP DATABASE x` on the `system`
|
||||
`:use <name>` switches between them. Clients that do not pick a database get `PEO_DEFAULT_GRAPH`
|
||||
(`neo4j`); the container image creates that graph on first start. `CREATE DATABASE x` and `DROP DATABASE x` on the `system`
|
||||
database call `create_graph` / `drop_graph`; AGE requires names of at least 3 characters.
|
||||
- **Queries** are sent to AGE as `SELECT * FROM cypher('<graph>', $$ <query> $$) AS (...)`.
|
||||
The bridge works out the result columns from the final `RETURN` clause and inlines
|
||||
peo works out the result columns from the final `RETURN` clause and inlines
|
||||
query parameters as Cypher literals.
|
||||
- **Neo4j procedures** that neo4j-browser relies on (`dbms.components`, `SHOW DATABASES`,
|
||||
`db.labels`, `db.schema.visualization`, `SHOW PROCEDURES`, ...) are emulated from the
|
||||
AGE catalog.
|
||||
- **Workarounds for AGE**: neo4j-browser's "expand node" query uses a pattern
|
||||
comprehension, and its relationship lookup uses `id(n) IN [...]`, which AGE rejects
|
||||
for lists of more than one element. The bridge rewrites both into equivalent Cypher
|
||||
for lists of more than one element. peo rewrites both into equivalent Cypher
|
||||
that AGE accepts.
|
||||
- **Authentication** is passed straight through: every Bolt connection opens its own
|
||||
PostgreSQL connection with the credentials you log in with.
|
||||
@@ -84,16 +100,65 @@ and `REMOVE`, and they can be indexed. Some things to keep in mind:
|
||||
- AGE has no date or time types, so store those as ISO 8601 strings and cast them in
|
||||
SQL.
|
||||
|
||||
## Configuration
|
||||
## Querying the graph from SQL
|
||||
|
||||
AGE stores each node label and relationship type as a table in the graph's schema,
|
||||
with all properties in one `agtype` column, `properties`; `properties ->> 'key'::text`
|
||||
returns a property as text. For typed columns, add them yourself, either as a view or
|
||||
as a generated column:
|
||||
|
||||
```sql
|
||||
CREATE VIEW neo4j.people AS
|
||||
SELECT id, properties ->> 'name'::text AS name, (properties ->> 'age'::text)::bigint AS age
|
||||
FROM neo4j."Person";
|
||||
|
||||
ALTER TABLE neo4j."Person"
|
||||
ADD COLUMN age bigint GENERATED ALWAYS AS ((properties ->> 'age'::text)::bigint) STORED;
|
||||
```
|
||||
|
||||
PostgreSQL keeps generated columns up to date on every Cypher `CREATE`, `MERGE`, `SET`
|
||||
and `REMOVE`, and they can be indexed. Some things to keep in mind:
|
||||
|
||||
- A value that does not fit the column's type makes the Cypher write fail: with the
|
||||
column above, `CREATE (:Person {age: 'old'})` is rejected. A view only fails when it
|
||||
is read.
|
||||
- AGE does not allow dropping columns from its label tables, so a generated column
|
||||
cannot be removed again.
|
||||
- Adding a stored generated column rewrites the table and locks it while doing so.
|
||||
- AGE has no date or time types, so store those as ISO 8601 strings and cast them in
|
||||
SQL.
|
||||
|
||||
## Container image
|
||||
|
||||
The container image runs PostgreSQL with AGE, peo, neo4j-browser and pgAdmin in one
|
||||
container:
|
||||
|
||||
```sh
|
||||
podman build -t peo .
|
||||
podman run -d --name peo -e POSTGRES_PASSWORD=secret \
|
||||
-p 7474:7474 -p 7687:7687 -p 5050:5050 -p 5432:5432 \
|
||||
-v peo-data:/var/lib/postgresql -v peo-pgadmin:/var/lib/pgadmin peo
|
||||
```
|
||||
|
||||
| Port | Service | Log in with |
|
||||
| --- | --- | --- |
|
||||
| 7474 | neo4j-browser: <http://localhost:7474/> | PostgreSQL user `postgres` and `POSTGRES_PASSWORD` |
|
||||
| 7687 | Bolt, for Neo4j drivers and cypher-shell (`neo4j://localhost:7687`) | same as above |
|
||||
| 5050 | pgAdmin: <http://localhost:5050/> | `admin@example.com` and `POSTGRES_PASSWORD` |
|
||||
| 5432 | PostgreSQL, for `psql` and other SQL clients | same as neo4j-browser |
|
||||
|
||||
pgAdmin comes with the bundled PostgreSQL server already registered as "peo";
|
||||
it asks for the PostgreSQL password the first time you open it. AGE stores each
|
||||
graph as a schema of the same name, with one table per label.
|
||||
|
||||
`docker` works as a drop-in replacement for `podman` in the commands above.
|
||||
|
||||
Besides peo's own variables above, the image uses these:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `POSTGRES_PASSWORD` | (required) | Password of the PostgreSQL superuser, see the [postgres image docs](https://hub.docker.com/_/postgres). `POSTGRES_USER` and `POSTGRES_DB` work too. |
|
||||
| `PEO_DEFAULT_GRAPH` | `neo4j` | Graph created at initialisation and used when a client does not pick a database. |
|
||||
| `PEO_PG_DATABASE` | `$POSTGRES_DB` | PostgreSQL database holding the graphs. |
|
||||
| `PEO_ADVERTISED_BOLT_PORT` | `7687` | Bolt port advertised to the browser; set it when you publish 7687 on another host port. |
|
||||
| `PEO_ADVERTISED_HOST` | `localhost` | Host reported in `SHOW DATABASES` and to clients that do not send a routing address. |
|
||||
| `PEO_LOG_LEVEL` | `INFO` | `DEBUG` logs every generated SQL statement. |
|
||||
| `PEO_DEFAULT_GRAPH` | `neo4j` | Also the graph created when the database is initialised. |
|
||||
| `PGADMIN_DEFAULT_EMAIL` | `admin@example.com` | pgAdmin login, created on first start. |
|
||||
| `PGADMIN_DEFAULT_PASSWORD` | `$POSTGRES_PASSWORD` | pgAdmin login password, set on first start. |
|
||||
| `PEO_PGADMIN` | `true` | Set to `false` to not start pgAdmin. |
|
||||
@@ -117,19 +182,15 @@ This is a first version. Things that do not work (yet):
|
||||
- User and role management, `SHOW TRANSACTIONS`, `:sysinfo`.
|
||||
- TLS (`bolt+s://`, HTTPS).
|
||||
- Each query's full result is fetched from PostgreSQL before it is streamed to the
|
||||
client, so very large results use memory in the bridge.
|
||||
client, so very large results use memory in peo.
|
||||
|
||||
## Development
|
||||
|
||||
The bridge is a dependency-light Python package in [`bridge/peo`](bridge/peo)
|
||||
(only `psycopg` 3). To run it outside the container against any AGE-enabled
|
||||
PostgreSQL:
|
||||
peo's only dependency is `psycopg` 3. To work on it, install it in editable mode;
|
||||
`python -m build` (or `uv build`) builds an sdist and wheel:
|
||||
|
||||
```sh
|
||||
pip install 'psycopg[binary]'
|
||||
cd bridge
|
||||
PEO_PG_HOST=localhost PEO_PG_DATABASE=postgres PEO_BROWSER_DIR=/path/to/neo4j-browser/dist \
|
||||
python -m peo
|
||||
pip install -e '.[binary]'
|
||||
```
|
||||
|
||||
[`tests/smoke_test.py`](tests/smoke_test.py) runs end-to-end checks through the official
|
||||
|
||||
Reference in new issue
Block a user