# Jozie AI — Model Builder

A platform for building, training, and improving domain-specific AI models: document
ingestion, cleaning, instruction dataset generation, RAG, evaluation, a prompt library,
a conversation-testing playground, continuous learning, and a hardened single-machine
production deployment.

Every module below is real, not a stub — including fine-tuning, which runs real LoRA
training on whatever hardware is available: Apple's MLX on Apple Silicon, or Unsloth on
an NVIDIA GPU (CUDA) elsewhere.

## What works

- **Auth & RBAC** — JWT login, four roles (`owner` > `admin` > `editor` > `viewer`)
  enforced on every mutating endpoint. `owner`/`admin` manage users and API keys;
  `editor` creates/edits content; `viewer` is read-only.
- **Knowledge Base** (2) — upload PDF/DOCX/TXT/MD/HTML/CSV/JSON/XML, async extraction via
  Celery, magic-byte content verification against the declared extension.
- **Text Cleaning** (3) — strips page numbers, copyright notices, duplicate paragraphs,
  references sections, OCR hyphenation artifacts; Unicode-normalizes; preview/edit/confirm
  before use.
- **Knowledge Organization** (4) — tags, category (per-project taxonomy), source.
- **Instruction Dataset Generator** (5) — Ollama-generated instruction/response examples
  from confirmed documents, with a human review queue (approve/edit/reject/regenerate).
- **Fine-Tuning** (8) — real LoRA fine-tuning, with the training backend picked per
  machine at runtime (`app/services/finetune/detect.py`): [Apple's MLX](https://github.com/ml-explore/mlx)
  on Apple Silicon, or [Unsloth](https://github.com/unslothai/unsloth) on an NVIDIA GPU
  (CUDA) — everywhere else, the module reports it's unavailable rather than pretending
  to work. Either way: trains against a dataset's approved examples, fuses the adapter
  into the base model, converts it to GGUF via llama.cpp's conversion script, and
  imports it into Ollama as a new tag — it then shows up automatically in Model
  Registry, Evaluation, and Conversation Testing with zero changes to those modules.
  See "Fine-Tuning setup" below for the one-time setup this needs.
- **Synthetic Dataset Expansion** (6) — expands an approved example into beginner,
  advanced, edge-case, troubleshooting, and comparative-reasoning variants via Ollama,
  grounded in the original example (and source document, if available); variants land
  back in the same review queue as any other generated example, tagged "expanded
  variant" with lineage back to the parent example.
- **Dataset Manager** (7) — search/filter/edit/delete, export approved examples as
  JSONL/JSON/CSV/Parquet.
- **Model Registry & Ollama Integration** (9/10) — list/pull/delete installed Ollama
  models; create a custom model by baking a saved prompt into a base model via a
  generated Modelfile (no fine-tuned adapter required).
- **RAG Builder** (11) — pgvector-backed chunking + embedding (`embeddinggemma`),
  semantic and hybrid (vector + keyword, reciprocal rank fusion) search with citations.
- **Evaluation** (12) — runs a target model against approved examples (optionally
  RAG-augmented), scores each with an LLM judge (accuracy, groundedness, hallucination),
  captures latency/token counts from Ollama's own response metadata, exports a report.
- **Prompt Library** (13) — org-scoped, versioned system prompts; editing appends a new
  version rather than overwriting.
- **Conversation Testing** (14) — chat playground against any installed Ollama model,
  optional saved-prompt system message, optional RAG augmentation with shown sources,
  per-turn latency/tokens/a labeled confidence *heuristic*, thumbs up/down feedback.
- **Continuous Learning** (15) — promote any conversation turn straight into a dataset's
  review queue (same approve/reject flow as generated examples); a stats view shows
  how many examples originated this way.
- **Administration** (16) — user management (create with a one-time generated password,
  change role, deactivate — no email service, so this is explicit), API key issuance
  (`jz_live_...`, SHA-256 hashed, shown once, revocable, usable as an alternate
  `Authorization: Bearer` credential), and a real audit log written from every mutating
  endpoint.
- **Dashboard** (1) — knowledge/dataset status counts, Ollama/DB reachability, host
  CPU/memory.
- Live job progress over WebSockets (`/ws/jobs/{id}`), Redis pub/sub, published from the
  Celery workers running extraction, cleaning, dataset generation, RAG indexing,
  evaluation, and model pull/create.

## Production hardening

- **Config gating** — refuses to boot with the default JWT secret when
  `ENVIRONMENT=production`; docs (`/docs`) disabled in production by default.
- **Rate limiting** — `slowapi`, tight limits on `/auth/login` and `/auth/register`.
- **Security headers** — `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`,
  `Permissions-Policy` on every response.
- **Structured logging** — rotating files at `backend/logs/{backend,worker}.log`
  (`LOG_JSON=true` for JSON-formatted lines).
- **Process supervision** — launchd plists for the API server and Celery worker
  (`deploy/launchd/`, installed via `deploy/scripts/install-services.sh`).
- **Reverse proxy + local HTTPS** — Caddy (`deploy/Caddyfile`) serves the built frontend
  and proxies `/api/*` to the backend under a single `https://localhost` origin (no CORS
  needed for that path).
- **Backups** — `deploy/scripts/backup-db.sh` (nightly via launchd) and `restore-db.sh`.
- **Upload hardening** — magic-byte content verification, not just extension checks.
- **Tests** — `backend/tests/` (pytest): cleaning-pipeline regexes, RAG chunking
  (including the word-boundary edge cases), RBAC enforcement, export formats, auth flows.

## Prerequisites

- Python 3.11+ (developed against 3.14)
- Node 18+ (developed against 22)
- Homebrew (macOS) — Postgres, Redis, pgvector, Caddy
- [Ollama](https://ollama.com) running locally with at least one chat model
  (defaults to `llama3.1:8b`) and an embedding model (`embeddinggemma`) pulled
- For Fine-Tuning (Module 8) specifically: either an Apple Silicon Mac (uses MLX) or a
  machine with an NVIDIA GPU (uses Unsloth/CUDA). Every other module works fine
  without either — Intel Macs, AMD, or CPU-only machines included.

## One-time local setup

### 1. Postgres + Redis + pgvector

```bash
brew install postgresql@16 redis pgvector
LC_ALL="en_US.UTF-8" /opt/homebrew/opt/postgresql@16/bin/postgres -D /opt/homebrew/var/postgresql@16 &
/opt/homebrew/opt/redis/bin/redis-server /opt/homebrew/etc/redis.conf &

export PATH="/opt/homebrew/opt/postgresql@16/bin:$PATH"
createdb jozie_ai
psql -d jozie_ai -c "CREATE ROLE jozie_app WITH LOGIN PASSWORD 'jozie_dev_local' CREATEDB;"
psql -d jozie_ai -c "ALTER DATABASE jozie_ai OWNER TO jozie_app;"
psql -d jozie_ai -c "CREATE EXTENSION IF NOT EXISTS vector;"
```

Homebrew's `pgvector` bottle may only target newer Postgres majors than the one you
installed; if `CREATE EXTENSION vector` fails with a missing-file error, build it from
source against your `pg_config`:

```bash
git clone --branch v0.8.5 --depth 1 https://github.com/pgvector/pgvector.git /tmp/pgvector-src
cd /tmp/pgvector-src
PG_CONFIG=/opt/homebrew/opt/postgresql@16/bin/pg_config make -j4
PG_CONFIG=/opt/homebrew/opt/postgresql@16/bin/pg_config make install
```

(`brew services start postgresql@16` / `redis` also works if your environment supports
launchd background services — the commands above run them directly for environments
where launchd services don't take effect.)

### 2. Backend

```bash
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # defaults already match the DB/Redis setup above
mkdir -p data/uploads logs
alembic upgrade head
```

### 3. Frontend

```bash
cd frontend
npm install
cp .env.example .env   # dev: http://localhost:8000/api/v1
```

### 4a. Fine-Tuning setup — Apple Silicon (MLX)

`mlx`/`mlx-lm`/`huggingface_hub` live in `backend/requirements-macos.txt`, not the main
`requirements.txt` — MLX has no Linux/Windows wheels, so it's kept out of the core
install (and out of the Docker image) and layered on top only on Apple Silicon:

```bash
cd backend
pip install -r requirements-macos.txt
```

GGUF conversion additionally needs its own isolated venv — llama.cpp's converter pins
an older `numpy`/`transformers`/`torch` that would otherwise downgrade the main app's
dependencies:

```bash
cd backend
git clone --depth 1 https://github.com/ggml-org/llama.cpp vendor-llama-cpp-convert
python3 -m venv .venv-gguf
source .venv-gguf/bin/activate
pip install -r vendor-llama-cpp-convert/requirements/requirements-convert_hf_to_gguf.txt
deactivate
```

That's it — the fine-tuning pipeline (`backend/app/services/finetune/`) shells out to
`.venv-gguf/bin/python3` and `vendor-llama-cpp-convert/convert_hf_to_gguf.py` by path, so
no activation or PATH changes are needed at runtime. Base models are pulled from Hugging
Face on first use of each (a few hundred MB to a few GB depending on the model) and
cached under `~/.cache/huggingface`.

### 4b. Fine-Tuning setup — NVIDIA GPU (CUDA)

Runs inside Docker with GPU passthrough — see "Running it with Docker" below for the
base stack first. This adds a GPU-enabled variant of the `worker` service
(`backend/Dockerfile.cuda`, base image `pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime`)
via a Compose override file, so a GPU-less install is completely unaffected.

**Before touching this app**, confirm Docker's GPU passthrough itself works:

```bash
docker run --rm --gpus=all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```

This should print your GPU. On Windows, that means Docker Desktop with the WSL2 backend
(no separate NVIDIA Container Toolkit install needed — Docker Desktop bundles it). On
Linux, install [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
first. If this command fails, it's a driver/Docker setup issue to resolve before going
further — not something in this app.

Once that works:

```bash
cp .env.docker.example .env   # if you haven't already
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d --build
```

> **Honesty note**: this backend (`backend/app/services/finetune/cuda_backend.py`,
> `backend/app/services/finetune_scripts/cuda_train.py`/`cuda_fuse.py`,
> `backend/requirements-cuda.txt`) was built without access to any NVIDIA hardware to
> test against — unlike every other requirements file in this project, the CUDA one
> isn't pinned to exact versions verified by a real install. It follows Unsloth/TRL's
> documented APIs, but a version mismatch or API drift on first real run is plausible.
> If `docker compose logs worker` shows an error, that's expected first-run friction,
> not a sign something is fundamentally wrong — the fix is usually a quick pin/argument
> adjustment once the real error is visible.

## Running it locally (dev mode)

Three processes, each in its own terminal:

```bash
# 1. API server
cd backend && source .venv/bin/activate
uvicorn app.main:app --reload --port 8000

# 2. Background worker (extraction, cleaning, generation, RAG indexing, evaluation)
cd backend && source .venv/bin/activate
celery -A app.core.celery_app worker --loglevel=info

# 3. Frontend
cd frontend
npm run dev
```

Open http://localhost:5173, click **Create org** to register, then walk the pipeline:

1. **Knowledge Base** → upload a document → wait for `cleaned` → confirm the cleaned text.
2. **Datasets** → create a dataset → **Generate** from the confirmed document → approve
   examples in the review queue.
3. **RAG Builder** → index the confirmed document → semantic/hybrid search over it.
4. **Prompt Library** → save a system prompt.
5. **Conversation Testing** → start a conversation with a saved prompt + RAG on → chat →
   thumbs-up a good answer → **Add to training data** (this is Continuous Learning) →
   it lands back in the Dataset Manager queue.
6. **Evaluation** → run a target model against the approved examples, optionally judged
   by a different model, optionally RAG-augmented.
7. **Model Registry** → bake a saved prompt into a named custom Ollama model.
8. **Fine-Tuning** (needs the one-time setup above — 4a for Apple Silicon, 4b for
   NVIDIA) → once a dataset has 6+ approved examples, pick a base model and start a
   job — watch it train, fuse, convert to GGUF, and land as a new Ollama tag (a few
   minutes; the smallest base model in the list is the fastest to try first).
9. **Administration** → invite a teammate (role + one-time password), issue an API key,
   review the audit log.

API docs (OpenAPI/Swagger, local only by default): http://localhost:8000/docs

## Running it with Docker (Mac, Linux, or Windows)

The alternative to the native setup above — one `docker-compose.yml` runs the backend,
worker, frontend, PostgreSQL+pgvector, and Redis identically on Mac, Linux, or Windows
(via Docker Desktop). This is the easiest path if you don't need Fine-Tuning, and the
recommended path on Linux/Windows.

**What's still native, on every OS**: Ollama. It needs direct GPU access (Metal on Mac,
CUDA on Windows/Linux) that Docker can't cleanly provide, and native install is how
Ollama is distributed anyway — [install it](https://ollama.com) and pull your models as
usual; the containerized app reaches it automatically via `host.docker.internal`.

**Fine-Tuning in the Docker path**: on an NVIDIA GPU, it works — see step 4b above for
the GPU-enabled `worker` override. On Apple Silicon it stays native-macOS-only (MLX
needs direct Metal access, which containers can't provide on macOS) — use the native
setup instead (step 4a) if that's your machine.

```bash
cp .env.docker.example .env   # edit JWT_SECRET before any real use
docker compose up -d --build
```

Open http://localhost and click **Create org** — same walkthrough as above. The API is
also reachable directly at `http://localhost:8000` if you need it outside the proxy.

Useful commands:

```bash
docker compose logs -f backend worker   # tail logs
docker compose down                      # stop everything, keep data
docker compose down -v                   # stop everything, DELETE all data
```

Uploaded documents, fine-tuning artifacts (n/a in this path), and logs persist in the
`backend_data`/`backend_logs` named volumes; database data persists in `postgres_data`.

By default the frontend container serves plain HTTP on port 80 — fine for a local
trial, but put a real TLS terminator (your own reverse proxy/load balancer, or extend
`deploy/docker/Caddyfile` with a real domain for Caddy's automatic HTTPS) in front of it,
set a real `JWT_SECRET`, and set `ENVIRONMENT=production` before exposing this beyond
your own machine.

## Running it as a supervised local "production" deployment

```bash
cd frontend && npm run build          # builds against .env.production (relative /api/v1)
brew install caddy
caddy trust                            # installs Caddy's local CA (prompts for sudo once)
./deploy/scripts/install-services.sh   # launchd: backend + worker + nightly backup
caddy run --config deploy/Caddyfile    # or install Caddy as its own launchd service
```

Then visit `https://localhost`. Update `ENVIRONMENT=production` and generate a real
`JWT_SECRET` (`openssl rand -hex 32`) in `backend/.env` before doing this for real —
the app refuses to boot in production mode with the default secret.

## Project structure

```
backend/app/
  core/        config (incl. production gating), JWT + API-key security, Celery app,
               rate limiter, logging, security-headers middleware
  db/          SQLAlchemy session/engine
  models/      ORM models
  schemas/     Pydantic request/response models
  api/v1/      routers (one per module)
  services/    extraction, cleaning, Ollama client, RAG, evaluation, model registry,
               conversation, audit, upload content-verification
  services/finetune/          pluggable fine-tuning backend (see detect.py) —
                               shared.py (hardware-agnostic pipeline), mlx_backend.py,
                               cuda_backend.py
  services/finetune_scripts/  standalone CUDA training/merge scripts, run as
                               subprocesses by cuda_backend.py — never imported by the
                               app itself, so torch/unsloth never load into the main process
  workers/     Celery tasks
  ws/          Redis pub/sub → WebSocket bridge
backend/alembic/          migrations
backend/tests/             pytest suite
backend/vendor-llama-cpp-convert/   llama.cpp clone, GGUF conversion script only (Fine-Tuning setup)
backend/.venv-gguf/        isolated venv for GGUF conversion (Fine-Tuning setup)
backend/data/finetuning/   per-job working directories (data/adapters/fused weights/gguf)
backend/requirements.txt          core, cross-platform — used by the native setup and Docker
backend/requirements-macos.txt    Fine-Tuning extras (mlx, mlx-lm, huggingface_hub) — Apple Silicon only
backend/requirements-cuda.txt     Fine-Tuning extras (unsloth, peft, trl, bitsandbytes) — NVIDIA GPU only
backend/Dockerfile                 API server + worker image (same image, different command)
backend/Dockerfile.cuda            GPU-enabled worker image, used only by docker-compose.gpu.yml

frontend/src/
  App.tsx, main.tsx    routing + providers
  lib/                 api client, auth, theme, websocket hook, module nav metadata
  components/layout/   Sidebar, Topbar, AppLayout
  pages/                one directory per module
frontend/Dockerfile      multi-stage: node build -> Caddy serve

docker-compose.yml         orchestrates postgres/redis/backend/worker/frontend
docker-compose.gpu.yml     override: swaps `worker` to the CUDA image + GPU reservation
.env.docker.example        template for docker-compose.yml's env vars

deploy/
  launchd/      plists for the backend, worker, and nightly backup (native macOS path)
  scripts/      install/uninstall-services.sh, backup/restore-db.sh
  Caddyfile     reverse proxy + local HTTPS config (native macOS path)
  docker/       containerized Caddyfile + Postgres pgvector init script (Docker path)
```

## Notes

- `passlib` was dropped for user passwords in favor of calling `bcrypt` directly —
  `passlib` 1.7.4's backend self-test breaks under `bcrypt` 4.1+/5.x. API keys use a
  plain SHA-256 hash instead of bcrypt since they're already high-entropy random tokens,
  not human-chosen passwords.
- Ollama's `format: "json"` forces valid JSON but not a specific shape (a single-example
  request sometimes returns one object instead of an array); the dataset generator and
  RAG/evaluation code defensively unwrap whatever shape actually comes back.
- Conversation Testing's "confidence" score is an explicitly-labeled heuristic (response
  length + finish reason) — not a calibrated probability.
- The `ollama create` flow (Model Registry) needs Ollama's own model blob files to be
  owned by the user running the daemon; if you ever ran `sudo ollama pull`, some blobs
  may be root-owned and this will fail with a permissions error. Fix with
  `sudo chown -R $(whoami):staff ~/.ollama/models`.
- Fine-Tuning gotchas worth knowing if you're touching `app/services/finetune/`:
  - `mlx_lm.fuse --export-gguf` (MLX's own exporter) crashes on modern rope-scaling
    configs (e.g. Llama 3.2's `"rope_type": "llama3"`) — that's why the MLX backend
    dequantizes and fuses via `mlx_lm.fuse`, then converts to GGUF with llama.cpp's own
    `convert_hf_to_gguf.py` in the isolated `.venv-gguf` instead (shared by every
    backend — see `shared.py::run_gguf_convert`).
  - Fusing stamps `tokenizer_config.json` with whatever `transformers` version wrote it
    (e.g. `"tokenizer_class": "TokenizersBackend"` on transformers 5.x), which the
    isolated venv's older transformers doesn't recognize — `shared.py::patch_tokenizer_class`
    forces it to `"PreTrainedTokenizerFast"` before conversion. Applies to every
    backend's fuse step, not just MLX's.
  - The isolated venv's Python must be invoked via its `bin/python3` **symlink**, not a
    fully-resolved realpath — `Path.resolve()` follows the symlink chain down to the
    system Homebrew binary and loses the venv's site-packages entirely.
  - Ollama's HTTP `/api/create` only accepts a `FROM` referencing an already-pulled
    model or an uploaded blob digest, not a raw local file path — that upload dance is
    what the `ollama` CLI does internally, so the pipeline shells out to
    `ollama create -f Modelfile` rather than reimplementing it over HTTP.
  - `ensure_base_model_downloaded`'s `huggingface_hub` import is deliberately lazy
    (inside the function, not at module level) — it's a hardware-specific extra (macOS
    or CUDA), and a top-level import crashes the *entire app* at boot anywhere it isn't
    installed (found via the Docker image, which doesn't include it by default), not
    just this one feature. The same reasoning is why `cuda_backend.py` never imports
    torch/unsloth directly — those only ever get imported inside
    `finetune_scripts/cuda_train.py`/`cuda_fuse.py`, which run as subprocesses, not as
    part of the app's own import graph.
  - The CUDA backend (`cuda_backend.py`, `finetune_scripts/cuda_train.py`/`cuda_fuse.py`,
    `requirements-cuda.txt`) was built without NVIDIA hardware available to test
    against — see the honesty note in "Fine-Tuning setup" above. Treat a first-run
    error here as expected friction to iterate on, not evidence of a deeper problem.