Prompt Evaluation

Use promptfoo to test and evaluate prompts before deeper integration into Unujual.

ExperimentalSetup requiredManual validation firstNot integrated yet

Operator readiness

Not ready inside the app yet

Current state

promptfoo is still external and experimental.

What you can do today

Run manual validation outside the app.

Next milestone

Confirm the local setup works.

Success looks like

The sample eval runs, or only stops on missing provider keys.

What it is

promptfoo is an external evaluation lab for prompt testing, regressions, and red-team style checks. Inside Unujual, it is useful as an experimental measurement surface, not as part of the core runtime.

What to do now

Follow this sequence manually. This page is guidance only and does not run promptfoo.

Step 1

Open the vendored promptfoo folder

Work inside AI-tools_OS/promptfoo so setup happens in the correct repo.

Step 2

Match Node, then install + build

promptfoo requires Node ^20.20.0 || >=22.22.0 (see package.json engines). Upgrade if needed, then install and build.

Step 3

Run the sample eval command

Use the getting-started config to confirm the local flow works end to end.

Commands

Run these in your terminal, not inside the app.

Step 1

cd AI-tools_OS/promptfoo

Step 2 — check Node (required before npm ci)

Required by vendored promptfoo: ^20.20.0 || >=22.22.0. Example: Node v20.19.x is too low; use 20.20+ or 22.22+.

node -v

If npm ci fails with EBADENGINE

npm is refusing install because your Node version does not satisfy ^20.20.0 || >=22.22.0. Upgrade Node (e.g. nvm: install 20.20+ or 22.22+), then run npm ci again. Do not rely on ignoring engines for this repo unless you accept breakage risk.

Step 3

npm ci

Step 4

npm run build

Step 5

npm run local -- eval -c examples/getting-started/promptfooconfig.yaml --no-cache

Done means

• the sample eval command runs successfully
• or it only fails because provider API keys are missing

Why this matters

• evaluate prompts before operational use
• compare outputs more safely
• create a repeatable evaluation habit inside Unujual

Compact status summary

Fast read of where promptfoo stands today inside Unujual.

Tool type

CLI + evaluation lab

Hybrid CLI / library surface inside a vendored monorepo.

Runtime state

Requires local setup

Install, build, and provider preparation happen outside the app.

Integration state

Not connected to Unujual yet

No subprocess execution, proxy, or embedded viewer.

Risk note

Keep it isolated

Do not merge promptfoo into the root app dependency tree.

Repo path

AI-tools_OS/promptfoo

Example config

getting-started/promptfooconfig.yaml

Node requirement

^20.20.0 || >=22.22.0

npm ci will error with EBADENGINE if your Node is below this range.

Manual validation

Required before deeper integration

Workspace facts

Observed from the vendored tree in this workspace. Useful, but secondary to the manual validation flow.

package.json version

0.121.3

engines.node

^20.20.0 || >=22.22.0

node_modules/

Not present or not visible

Install may still be required in your environment.

dist/

Not present or not visible

Build may still be required in your environment.

In the audited snapshot, node_modules/ and dist/ were missing. Your local checkout may differ after setup work.

Technical references

Keep these nearby after you finish the basic manual validation flow.

Paths and notes

Reference only. Not a repo browser and not a live runtime check.

Vendored root

AI-tools_OS/promptfoo

Example config

AI-tools_OS/promptfoo/examples/getting-started/promptfooconfig.yaml

Ports note

promptfoo dev or viewer flows may use ports like 3000 or 5173. Unujual does not assume those are live and does not embed or proxy them here.

More technical notes

Secondary detail from the vendored repo.

Use a compatible Node version before install. Meaningful evals usually need provider API keys.

Successful manual validation should happen before any bounded wrapper, helper surface, or runtime embedding work.

Selected npm scripts (from package.json)

build

concurrently -g --kill-others-on-fail "tsc --noEmit" "NODE_OPTIONS='--max-old-space-size=8192' tsdown" "npm run build:app"

local
```
tsx src/main.ts
```
local:web
```
npm run dev --prefix src/app
```

dev

concurrently "npm run dev:server" "npm run dev:app"

dev:server
```
tsx --watch src/server/index.ts
```
dev:app
```
npm run dev --prefix src/app
```
test
```
vitest run
```

test:smoke

vitest run --config vitest.smoke.config.ts