#!/bin/sh # ============================================================================ # mediamatrix.sh - verify which input types each FREE OpenCode Zen model # actually accepts, and write a plain-ASCII HTML report. # # POSIX sh only: no arrays, no [[ ]], no here-strings, no $SECONDS, no # ${BASH_SOURCE}. Checked with `shellcheck -s sh`, executed under dash. # # WHY THIS DESIGN # --------------- # These are agentic coding models running inside opencode, and EVERY tool is # enabled by default (see https://opencode.ai/docs/tools/ - "By default, all # tools are enabled"). A model asked "what is in this image" will therefore # frequently answer CORRECTLY WITHOUT SEEING ANYTHING, by running bash+python # and decoding the file itself. Observed in practice: # # * space-bunny-free read a PNG, then ran PIL to crop/zoom it, saved # zoom.png and zoom2.png, re-read those, and reported the colour/digits. # * cat-jam-free ran ~20 bash commands (connected-component labelling, # per-column ink density, ASCII-art rendering) to reverse-engineer glyph # shapes pixel by pixel - and never once said it could not see the image. # # So a correct answer proves NOTHING unless the answer cannot be recovered by # any amount of computation. Two rules make that true here: # # 1. Each trial runs in its OWN mktemp dir containing EXACTLY ONE attachment, # so there are no sibling files to list, read or diff against. # 2. The answer is a RANDOM code generated fresh per modality, rendered only # as PIXELS (never as text/metadata), and never written into a filename. # A code that exists only as raster data cannot be extracted with strings, # grep, pdftotext, or ffmpeg metadata - only genuinely decoded. # # The script also records whether tools were used, so a "correct" answer can be # reported as CHEATED rather than as evidence of the capability being tested. # # WHICH MODELS GET TESTED # ----------------------- # The list is NOT hardcoded. It is rebuilt on every run from models.dev # (cost.input == 0, status != deprecated) intersected with the models opencode # actually serves (/zen/v1/models), so a model added or retired upstream is # picked up without editing this file. The table baked in on 2026-10-17 is only # an offline fallback, used when both requests fail. # # SCOPE / LIMITS (read before trusting a cell) # * ISOLATION IS BY WORKING DIRECTORY, NOT BY A SANDBOX. Every model runs as # the same user with bash enabled, so a model can always walk up to $HERE and # read this script, the transcripts, and the .meta files that hold the # EXPECTED CODE. Observed on 2026-10-18: beam-free listed .mediamatrix-logs # and catted this file. Such a trial is voided (tools=escaped), never scored. # Reading the key is the only path to a HIT that carries no tool marker, so # the detector has to be watched; it is checked, not assumed. # * IMAGE ONLY, on purpose. pdf and video used to be probed here and both # columns were worthless, for reasons that are about the opencode CLIENT # rather than about the models: # - pdf is forwarded as input_file only on /responses transports, and no # free model is on /responses any more (they all use the provider-wide # @ai-sdk/openai-compatible default). The models that did answer # correctly had run pdftoppm, so they proved nothing about pdf input. # - video is DROPPED before the request is built. Every model that # "read" a video code had run ffmpeg and read the extracted frames. # Neither can be settled by asking a model; it needs the outgoing request # body, which was captured separately on 2026-10-13/17 and is recorded in # the clients. So the probes are gone rather than kept as decoration. # # PDF SPOT CHECK (2026-10-18, glm-5.3-flash-free, manual - NOT a matrix column) # Kept as evidence because it is the one free model that declares pdf, and # because it was measured with a control rather than inferred from the # transport argument above. Same isolated dir, same pixel-only 6-char code, # one attachment per trial, no tool use in any transcript: # # pdf x3 probe.pdf (code B9BD2A) -> Error: Provider request failed with # HTTP 500 # png x1 probe.png (code B713B5) -> B713B5, correct, no tools # # The PNG control is what makes this conclusive rather than merely suggestive: # without it, three identical 500s are equally consistent with "the provider # is down" and "this attachment type is rejected". The control ran on the same # model in the same minute and answered cleanly, so the 500 tracks the pdf and # not the host. The code was verified absent from the PDF's raw bytes AND from # every decompressed stream, so strings/pdftotext/grep recover nothing. # # Two honest limits. First, the failure is an OPAQUE 500: the clients still # quote an older, explicit rejection ('content part type "file" is not # supported; only text and image_url are'), and that string did NOT reproduce # here - treat the quoted message as historical, not as the current symptom. # Second, a 500 cannot distinguish "type unsupported" from "temporarily # unavailable", so this confirms delivery FAILS, not why. Deciding between the # two needs the outgoing request body, which is why pdf stays out of the # matrix. The workaround (pdftoppm -png, then attach the PNG) is unaffected - # and is exactly what the control row demonstrates working. # * audio is NOT tested: no TTS engine is available on this host, so there is # no way to put an unknown code into an audio stream. # * one trial per round is evidence of CAPABILITY, not of reliability; use -r # for rounds and read the reliability section. # * jev-* is skipped: it is a /systemone typed-decision endpoint, not chat. # * results are per-machine and time-stamped; re-run to re-verify. # # USAGE # ./mediamatrix.sh # one image trial per free model, writes the HTML # -r 3 # 3 fresh image codes per model -> reliability # -a # add rounds after the highest one on disk # -l # free models + declared input media, run nothing # -n # rebuild the HTML from a previous raw log # -s # re-score the saved transcripts, call nothing # -m 'a,b' # restrict to these model ids # -t 300 # per-trial cap in seconds # -h # this text # # A trial lost to a transport fault (dead socket, upstream 5xx, expired cap) is # retried up to MEDIAMATRIX_RETRIES times (default 3) with the SAME code, and only # the last attempt is scored. Retry therefore costs wall-clock, never a data # point: 5 of 27 trials were lost this way on 2026-10-18. # # Requires: python3 (stdlib only for the model list and the report; Pillow for # the probe), an `opencode` on PATH. No GNU coreutils: the per-trial cap is a # pure-POSIX run_with_timeout, so stock macOS /bin/sh works unmodified. # ============================================================================ set -u HERE=$(cd "$(dirname "$0")" 2>/dev/null && pwd) || HERE=. RAW="$HERE/.mediamatrix-raw.tsv" META="$HERE/.mediamatrix-models.tsv" LOGDIR="$HERE/.mediamatrix-logs" REPORT="$HERE/mediamatrix-report.html" PROBE_DIR="$HERE/.mediamatrix-probe" TIMEOUT=${MEDIAMATRIX_TIMEOUT:-150} DO_RUN=1 DO_LIST=0 DO_RESCORE=0 DO_APPEND=0 REPEAT=1 ONLY_MODELS="" # Attempts per trial, including the first. A transport fault is retried in place # with the SAME code; see run_one for why the code must not change. RETRIES=${MEDIAMATRIX_RETRIES:-3} usage() { sed -n '/^# USAGE$/,/^# Requires:/{/^# Requires:/d;s/^# \{0,1\}//;p}' "$0" } # getopts is the POSIX parser and the only parser here: short options only, no # long spellings and no hand-rolled pre-pass to translate them. # The colons must sit immediately after the letters that take a value: "r:", # "t:" and "m:" here. A stray "qnraslt:m:h" reads perfectly well and still # leaves -r's value behind as an unexpected argument, because that colon # belongs to t and not to r. while getopts "qnasr:lt:m:h" opt; do case $opt in n) DO_RUN=0 ;; r) REPEAT=$OPTARG ;; a) DO_APPEND=1 ;; s) DO_RESCORE=1; DO_RUN=0 ;; l) DO_LIST=1 ;; t) TIMEOUT=$OPTARG ;; m) ONLY_MODELS=$OPTARG ;; h) usage; exit 0 ;; *) usage >&2; exit 2 ;; esac done shift $((OPTIND - 1)) if [ $# -gt 0 ]; then printf 'unexpected argument: %s\n\n' "$1" >&2 usage >&2 exit 2 fi case $REPEAT in ''|*[!0-9]*|0) printf -- '-r needs a positive integer, got: %s\n' "$REPEAT" >&2; exit 2 ;; esac # --------------------------------------------------------------------------- # The free models. Writes iddeclared-input-modalitiessource to $META # and echoes the ids, one per line. stdlib only, so -l works without Pillow. # --------------------------------------------------------------------------- fetch_models() { python3 - "$META" <<'PY' import gzip, json, sys, urllib.request meta_path = sys.argv[1] INDEX = "https://models.dev/api.json" ZEN = "https://opencode.ai/zen/v1/models" # Offline fallback, re-read from models.dev on 2026-10-20 (space-bunny-free # retired upstream that day; minimax-m3.1-flash-free added on 2026-10-19). # Used only when the two requests below both fail; it keeps the script testing # something rather than exiting silently. FALLBACK = [ ("beam-free", "text"), ("big-pickle", "text"), ("deepseek-v4.1-flash-free", "text,image"), ("glm-5.3-flash-free", "text,image,video,pdf"), ("longcat-2.5-preview-free", "text,image"), ("mimo-v2.6-flash-free", "text,image,audio,video"), ("minimax-m3.1-flash-free", "text,image,video"), ("nemotron-3.5-lightning-free", "text"), ("nemotron-3.5-super-vl-free", "text,image"), ] def get_json(url, timeout=25): req = urllib.request.Request( url, headers={"User-Agent": "mediamatrix", "Accept-Encoding": "gzip"}) with urllib.request.urlopen(req, timeout=timeout) as r: raw = r.read() if r.headers.get("Content-Encoding") == "gzip": raw = gzip.decompress(raw) return json.loads(raw.decode("utf-8", "replace")) rows = [] try: catalog = (get_json(INDEX) or {}).get("opencode") or {} models = catalog.get("models") or {} # /zen/v1/models is the authoritative served list, but it has no cost or # modality data, so it can only be used to REMOVE entries models.dev still # carries. If the call fails, trust models.dev alone. served = None try: served = {m.get("id") for m in (get_json(ZEN, 12) or {}).get("data", [])} except Exception: served = None for mid, entry in sorted(models.items()): if entry.get("status") == "deprecated": continue if (entry.get("cost") or {}).get("input") != 0: continue if served is not None and mid not in served: continue mods = (entry.get("modalities") or {}).get("input") or [] rows.append((mid, ",".join(mods) if mods else "unknown", "models.dev")) except Exception: rows = [(m, mods, "fallback 2026-10-20") for m, mods in FALLBACK] with open(meta_path, "w") as fh: for mid, mods, src in rows: fh.write("%s\t%s\t%s\n" % (mid, mods, src)) for mid, _mods, _src in rows: print(mid) PY } # One random PNG code per TRIAL, written into a per-model directory. Never # shared between the parallel workers: two models regenerating the same file # while a third copies it would hand someone the wrong code, and a wrong code # reads as a miss. make_image_probe() { mip_dir=$1 mkdir -p "$mip_dir" python3 - "$mip_dir" <<'PY' import secrets, sys from PIL import Image, ImageDraw out = sys.argv[1] code = secrets.token_hex(3).upper() # 6 hex chars, unknowable in advance img = Image.new("RGB", (560, 240), (250, 250, 252)) ImageDraw.Draw(img).text((70, 105), code, fill=(0, 0, 0)) img.save(f"{out}/probe.png") print(code) PY } # --------------------------------------------------------------------------- # POSIX run_with_timeout: cap a command at N seconds without GNU `timeout`. # GNU timeout is NOT on stock macOS (it is coreutils), and the trial cap is # load-bearing: an opencode run that never ends would otherwise hang its # parallel worker forever. Everything here is POSIX sh (dash, /bin/sh on macOS): # background the command, poll it with kill -0, TERM (then KILL) at the # deadline. Output goes through a temp file rather than a pipe, so the polls # cannot deadlock and the caller still sees both stdout and stderr. # Prints nothing itself: its stdout is the command output, captured verbatim. # --------------------------------------------------------------------------- run_with_timeout() { rwt_secs=$1 shift rwt_tmp=$(mktemp "${TMPDIR:-/tmp}/mediamatrix.timeout.XXXXXX") \ || { "$@"; return; } "$@" >"$rwt_tmp" 2>&1 & rwt_pid=$! rwt_deadline=$(( $(date +%s) + rwt_secs )) rwt_alive=1 while :; do if ! kill -0 "$rwt_pid" 2>/dev/null; then rwt_alive=0 break fi [ "$(date +%s)" -ge "$rwt_deadline" ] && break sleep 1 done if [ "$rwt_alive" = 1 ]; then kill "$rwt_pid" 2>/dev/null sleep 1 kill -0 "$rwt_pid" 2>/dev/null && kill -9 "$rwt_pid" 2>/dev/null fi wait "$rwt_pid" 2>/dev/null cat "$rwt_tmp" rm -f "$rwt_tmp" } # One trial: isolated dir, one attachment, TIMEOUT cap. Prints the transcript. # --------------------------------------------------------------------------- run_trial() { rt_model=$1 rt_src=$2 rt_question=$3 rt_work=$(mktemp -d "${TMPDIR:-/tmp}/mediamatrix.XXXXXX") || return 1 rt_base=${rt_src##*/} cp "$rt_src" "$rt_work/$rt_base" || { rm -rf "$rt_work"; return 1; } rt_out=$(cd "$rt_work" && run_with_timeout "$TIMEOUT" opencode run --standalone \ -m "opencode/$rt_model" -f "$rt_base" "$rt_question" | tr -d '\r') rm -rf "$rt_work" printf '%s' "$rt_out" } # The ESC character for ANSI stripping, needed by used_tools, escaped_harness and # extract_answer, so it is built once here rather than next to any of them. ESC=$(printf '\033') # Strip ANSI escapes and CR from stdin. Every matcher must run on CLEAN text: # the CLI paints the word that follows the literal "Error" marker, so a pattern # expecting a bare space after the colon sees the reset sequence instead and # misses. That is not theoretical - it is why the upstream-5xx case below was # recorded as a MISS (deepseek r2, 2026-10-18) on the first pass of this patch. strip_ansi() { tr -d '\r' | sed -e "s/${ESC}\[[0-9;?]*[a-zA-Z]//g" -e "s/${ESC}[()][A-Z0-9]//g" } # Transport faults, defined once because run_one and rescore_log must agree. # # Two subtleties, both learned from a wrong cell: # * Matched against strip_ansi output. The CLI wraps the word that follows # "Error:" in an ANSI colour reset, so a pattern expecting a bare space # there never matches and a server-side 500 gets recorded as a MISS - i.e. # as evidence that a model cannot see images (deepseek r2, 2026-10-18). # * The pattern does match this file's own source, and deliberately so. Making # it echo-proof is possible but fragile - it meant guarding every bare # alternative against the "|" that separates them, which broke as soon as one # alternative gained a "(" ahead of it. Instead the harm is removed at the # consumer: tally() ranks tools=yes (which includes every escaped trial) # above ERROR, so a model that cats the script to fake a transport fault # voids its own trial and gains nothing. rescore_log asserts that a REAL # transport message is still matched, which is the half that matters. # For the same reason the check is NOT skipped for escaped transcripts: an # escaped trial that genuinely hit a transport fault is still worth retrying. ERR_RE='Error:[[:space:]]+(Transport:|Upstream request failed[[:space:]]|Provider request failed[[:space:]])|socket connection was closed unexpectedly|ECONNRESET|fetch failed|ETIMEDOUT' # Did the model shell out? Any of these markers means the answer may have been # computed from the file rather than perceived from it. # # Four rules, each learned from a wrong cell in the report: # * Strip ANSI escapes FIRST. A command line arrives as ESC[0m$ ESC[0mls -la, # so an anchor that expects the line to start with whitespace then '$' never # matches anything. # * A line that starts with a shell prompt is a COMMAND LINE, so any decoder on # it counts - including one behind a chain. Matching only the first token # missed `mkdir -p frames && ffmpeg -i probe.mp4 frames/f_%03d.png`. # * The word-level markers stay case-SENSITIVE. An earlier case-insensitive # "Read " matched the words in "I can't read the image" and scored every # honest refusal as CHEATED. # * Both patterns must be OR-ed explicitly. Letting the function return the # status of the last grep silently ignored the first one, so transcripts # that only ever ran ffmpeg were logged as tools=no. used_tools() { printf '%s' "$1" | strip_ansi | grep -qE \ '^[[:space:]]*[$>][[:space:]]+.*(python3?|ffmpeg|pdftoppm|pdftotext|convert|magick|identify|tesseract|strings|xxd|base64)' \ && return 0 printf '%s' "$1" | strip_ansi | grep -qE \ 'python3? -c|Image\.open|from PIL|import PIL|numpy|tesseract|(^|[^A-Za-z])(Read|Write|Edit|MultiEdit|Bash|Glob|Grep|LS|List|Todowrite|Task)\(' \ && return 0 return 1 } # Did the model leave the isolated directory and read the harness itself? # # The isolation in rule 1 above is by WORKING DIRECTORY, not by a sandbox: every # model runs as the same user with bash enabled, so nothing stops it from # walking up to $HERE. On 2026-10-18 beam-free did exactly that - it listed # .mediamatrix-logs and catted this file, which is where the per-trial .meta # files holding the EXPECTED CODE live. A model that reads the answer key # cannot be scored either way, so such a trial is voided rather than trusted. # # Detection is deliberately the conservative direction: a false positive only # downgrades a trial to UNSCORABLE, while a missed escape would let an # answer-key read masquerade as a clean HIT. escaped_harness() { # Match the HARNESS paths, never the bare stem. run_trial's own work dir is # created as /tmp/mediamatrix.XXXXXX, so a model listing the directory it is # standing in would trip a loose "mediamatrix" match (space-bunny r3 and # deepseek r2, 2026-10-18) - and those trials were doing nothing wrong. The # tokens below are the ones that actually give away the harness: the log dir # holding the expected code, the shared probe dir, the answer-key .meta files, # and this script's own name. printf '%s' "$1" | strip_ansi | grep -qiE \ 'mediamatrix-logs|mediamatrix-probe|mediamatrix-models|probecode|\.meta|mediamatrix\.sh|MEDIAMATRIX_(TIMEOUT|RETRIES)' } # Reduce the transcript to the model's own answer, then test for the code. # Strips ANSI, banner lines ("build . model") and tool-call markers. Deliberately # keeps ANY line that merely mentions the code, because a model that answers # correctly in prose counts exactly as much as one replying with the bare code. extract_answer() { printf '%s' "$1" | strip_ansi \ | grep -vE '^[[:space:]]*$|^> build|✱|^→ |^> ' \ | tail -n 4 | tr '\n' ' ' } # One row in the matrix: run the trial, score it, append to the raw log, and # keep the transcript plus the expected code for later re-scoring. # # RETRY: a transport fault is retried in place, with the SAME expected code, and # only the final outcome is written. That is the whole point - the code is # unknowable before the trial and must not change on a retry, or round N and its # retry would be two different questions. A dead socket says nothing about the # model, so keeping it as a cell would spend a real data point on infrastructure # noise (5 of 27 trials were lost this way on 2026-10-18). run_one() { ro_model=$1 ro_mod=$2 ro_src=$3 ro_question=$4 ro_code=$5 ro_round=${6:-1} ro_start=$(date +%s) printf ' -> %-30s %-5s r%s starting\n' "$ro_model" "$ro_mod" "$ro_round" >&2 ro_try=1 while :; do ro_out=$(run_trial "$ro_model" "$ro_src" "$ro_question") ro_esc=no if escaped_harness "$ro_out"; then ro_esc=yes; fi if [ -z "$ro_out" ] || printf '%s' "$ro_out" \ | strip_ansi | grep -qiE "$ERR_RE"; then ro_hit=ERROR else ro_hit=MISS case $(extract_answer "$ro_out") in *"$ro_code"*) ro_hit=HIT ;; esac break fi if [ "$ro_try" -ge "$RETRIES" ]; then break; fi ro_try=$((ro_try + 1)) printf ' ~~ %-30s r%s transport fault, retry %s/%s (same code)\n' \ "$ro_model" "$ro_round" "$ro_try" "$RETRIES" >&2 sleep 3 done ro_ans=$(extract_answer "$ro_out") ro_tools=no if used_tools "$ro_out"; then ro_tools=yes; fi # Escaped the sandbox and read the harness: the answer may have come from the # key rather than from the pixels, so the trial is voided like a tool call. [ "$ro_esc" = no ] || ro_tools=yes [ -n "$ro_ans" ] || ro_ans="(no answer)" mkdir -p "$LOGDIR" printf '%s\n' "$ro_out" > "$LOGDIR/$ro_model.$ro_mod.$ro_round.txt" printf '%s\t%s\n' "$ro_round" "$ro_code" > "$LOGDIR/$ro_model.$ro_mod.$ro_round.meta" # Single printf, single line: concurrent appends from the worker subshells # must not interleave, or the TSV contract breaks. printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ "$ro_model" "$ro_mod" "$ro_round" "$ro_hit" "$ro_tools" "$ro_ans" >> "$RAW" printf ' OK %-30s %-5s r%s %-5s tools=%-3s %ss\n' "$ro_model" "$ro_mod" \ "$ro_round" "$ro_hit" "$ro_tools" "$(( $(date +%s) - ro_start ))" >&2 } # Rebuild the raw log from the saved transcripts, without calling any model. # The scoring rules change as the detector gets sharper; re-running 27 trials to # apply a stricter grep would be absurd when the transcripts are on disk. rescore_log() { [ -d "$LOGDIR" ] || { printf 'no transcripts in %s\n' "$LOGDIR" >&2; return 1; } # The error detector CAN match this script's own source - the pattern spells # its own alternatives out. That is harmless, and the reason is settled in the # REPORT instead of here: a trial that escaped the harness is already voided # (tools=yes), and tally() reports tools=yes as UNSCORABLE ahead of ERROR. So # a model that cats this script to fake a transport fault only ever gets its # own trial voided, which is the outcome it deserved anyway. # # What is NOT harmless is the reverse case, so it is asserted here: a genuine # transport message must still be recognised. Without the ANSI strip this # misses, because the CLI paints the word after "Error:" (deepseek r2, # 2026-10-18, recorded as a MISS = "cannot see images"). printf 'x Error: Transport: The socket connection was closed unexpectedly' \ | strip_ansi | grep -qE "$ERR_RE" \ || { printf 'FATAL: ERR_RE no longer matches a real transport error\n' >&2 return 1; } printf 'x Error: Provider request failed with HTTP 500' \ | strip_ansi | grep -qE "$ERR_RE" \ || { printf 'FATAL: ERR_RE misses the upstream 5xx shape\n' >&2 return 1; } printf 'model\tmodality\tround\thit\ttools\tanswer\n' > "$RAW" for rs_file in "$LOGDIR"/*.txt; do [ -f "$rs_file" ] || continue rs_stem=${rs_file##*/} rs_stem=${rs_stem%.txt} # Transcript names are ..txt for a single round and # ...txt when -r was used. Model ids contain # dots (nemotron-3.5-lightning-free), so the modality is always the segment # before a trailing round, and a trailing numeric segment means a round. rs_last=${rs_stem##*.} case $rs_last in ''|*[!0-9]*) rs_mod=$rs_last rs_round=1 rs_model=${rs_stem%.*} ;; *) rs_round=$rs_last rs_rest=${rs_stem%.*} rs_mod=${rs_rest##*.} rs_model=${rs_rest%.*} ;; esac # Image is the only modality probed any more. Older pdf/video transcripts # are left on disk as evidence of what was tried, but they must not come # back into the log: they measure the client's transport, not the model. [ "$rs_mod" = image ] || continue if [ -f "$LOGDIR/$rs_stem.meta" ]; then IFS=' ' read -r rs_round rs_code < "$LOGDIR/$rs_stem.meta" fi rs_out=$(cat "$rs_file") rs_ans=$(extract_answer "$rs_out") rs_hit=MISS [ -n "$rs_code" ] || rs_code="@@nocode@@" case $rs_ans in *"$rs_code"*) rs_hit=HIT ;; esac rs_tools=no if used_tools "$rs_out"; then rs_tools=yes; fi # Same voiding rule as run_one: a transcript that read the harness has seen # the answer key, so it can never be scored as perception. There is no retry # here - re-scoring must not call a model - so this only ever removes rows # from the PASS column, never invents one. if escaped_harness "$rs_out"; then rs_tools=yes; fi [ -n "$rs_ans" ] || rs_ans="(no answer)" if [ -z "$rs_out" ] || printf '%s' "$rs_out" \ | strip_ansi | grep -qiE "$ERR_RE"; then rs_hit=ERROR fi printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ "$rs_model" "$rs_mod" "$rs_round" "$rs_hit" "$rs_tools" "$rs_ans" >> "$RAW" done printf 'rescored %s trials from %s\n' \ "$(($(wc -l < "$RAW") - 1))" "$LOGDIR" } list_models() { fetch_models > /dev/null || return 1 while IFS=' ' read -r lm_id lm_mods _lm_src; do [ -n "$lm_id" ] || continue case ,$lm_mods, in *,image,*) lm_img="image" ;; *) lm_img="no-image" ;; esac printf '%-30s %-28s %s\n' "$lm_id" "$lm_mods" "$lm_img" done < "$META" } # --------------------------------------------------------------------------- # Main # --------------------------------------------------------------------------- MODELS=$(fetch_models) if [ -z "$MODELS" ]; then printf 'could not determine the free model list\n' >&2 exit 1 fi if [ -n "$ONLY_MODELS" ]; then keep="" for want in $(printf '%s' "$ONLY_MODELS" | tr ',;' ' '); do for have in $MODELS; do if [ "$have" = "$want" ]; then keep="$keep $have" break fi done done MODELS=${keep# } fi if [ "$DO_LIST" = 1 ]; then list_models exit 0 fi # Deliberate word splitting: $MODELS is a newline-separated id list, and the # positional parameters are the POSIX way to count words. # shellcheck disable=SC2086 # the split is the point here set -- $MODELS N_MODELS=$# TOTAL=$((N_MODELS * REPEAT)) if [ "$DO_RESCORE" = 1 ]; then rescore_log elif [ "$DO_RUN" = 1 ]; then rm -rf "$PROBE_DIR" mkdir -p "$PROBE_DIR" IMG_Q="Read the 6-character alphanumeric code shown in this image. Reply with exactly the code and nothing else." printf 'matrix: %s models, %s trials, %ss cap each\n' "$N_MODELS" "$TOTAL" "$TIMEOUT" [ "$REPEAT" -gt 1 ] && printf 'image gets %s fresh codes per model: a pass is capability, the ratio is reliability\n' "$REPEAT" printf 'all models run in parallel; results stream below as each finishes\n' printf -- '--------------------------------------------------------------\n' if [ "$DO_APPEND" = 1 ]; then printf 'appending rounds after the highest one already on disk\n' >&2 else printf 'model\tmodality\tround\thit\ttools\tanswer\n' > "$RAW" fi # Cross-process-safe progress: every worker appends to the raw log, and a # reporter polls that file so you see progress live instead of a blank wait. ( while :; do sleep 5 [ -f "$RAW" ] || continue n=$(($(wc -l < "$RAW") - 1)) [ "$n" -ge "$TOTAL" ] && break printf ' [progress] %d/%d trials complete\n' "$n" "$TOTAL" done ) & REPORTER=$! for model in $MODELS; do ( # -a continues after the highest round already stored for this # model, so a retry of a lost round cannot overwrite the round it replaces. first=1 if [ "$DO_APPEND" = 1 ]; then for prev in "$LOGDIR/$model.image."*.txt; do [ -f "$prev" ] || continue prev_base=${prev##*/} prev_base=${prev_base%.txt} prev_n=${prev_base##*.} case $prev_n in ''|*[!0-9]*) continue ;; esac if [ "$prev_n" -ge "$first" ]; then first=$((prev_n + 1)); fi done fi last=$((first + REPEAT - 1)) round=$first while [ "$round" -le "$last" ]; do img_code=$(make_image_probe "$PROBE_DIR/$model") || break run_one "$model" image "$PROBE_DIR/$model/probe.png" "$IMG_Q" "$img_code" "$round" round=$((round + 1)) done ) & done wait kill "$REPORTER" 2>/dev/null # With -a the log was never truncated, so rebuild it from the transcripts # and let one place decide what the rows are. if [ "$DO_APPEND" = 1 ]; then rescore_log fi printf -- '--------------------------------------------------------------\n' printf 'all trials finished\n' rm -rf "$PROBE_DIR" printf 'raw log: %s\n' "$RAW" printf 'transcripts: %s\n' "$LOGDIR" fi # --------------------------------------------------------------------------- # Render the plain-ASCII HTML report from the raw log. # --------------------------------------------------------------------------- python3 - "$RAW" "$META" "$REPORT" "$(date '+%Y-%m-%d %H:%M')" <<'PY' import sys, html, datetime, os raw, meta_path, out_path, run_ts = sys.argv[1:5] def ascii_only(s): # Model answers carry em dashes and smart quotes. The report is ASCII by # contract, and replacing them per answer keeps the whole-document check at # the end silent instead of blaming the model for the file. return s.encode("ascii", "replace").decode("ascii") rows = [] log_mtime = None if os.path.exists(raw): log_mtime = datetime.datetime.fromtimestamp(os.path.getmtime(raw)) with open(raw) as fh: next(fh, None) for line in fh: p = line.rstrip("\n").split("\t") # 6 columns: model, modality, round, hit, tools, answer. A 5-column # log from an older run has no round column; treat it as round 1. if len(p) >= 6: rows.append((p[0], p[1], p[2], p[3], p[4], ascii_only(" ".join(p[5:])[:160]))) elif len(p) == 5: rows.append((p[0], p[1], "1", p[2], p[3], ascii_only(" ".join(p[4:])[:160]))) # Image only. An older log may still carry pdf/video rows from before those # probes were dropped; they are not evidence about any model, so they go. rows = [r for r in rows if r[1] == "image"] # id -> declared input modalities (models.dev), plus where that came from. declared, source = {}, "none" if os.path.exists(meta_path): with open(meta_path) as fh: for line in fh: parts = line.rstrip("\n").split("\t") if len(parts) >= 2: declared[parts[0]] = parts[1] source = parts[2] if len(parts) > 2 else "models.dev" # Model order: the free catalogue order (sorted ids), then anything the log # knows about that models.dev did not list - a deprecated model can still be # served, and a result is worth showing even when the catalogue dropped it. MODELS = sorted(declared) seen = set(MODELS) for r in rows: if r[0] not in seen: seen.add(r[0]) MODELS.append(r[0]) # A trial only counts as evidence when the answer came from perception. Any tool # use means the model could have decoded the file itself (PIL, ffmpeg, pdftoppm, # strings), so the result is UNSCORABLE rather than a pass or a fail. A declared # refusal is its own class: it proves the model never got the pixels. REFUSAL = ("cannot", "can't", "can not", "unable", "no image", "not able", "without", "not support", "text-only", "text only", "as an ai", "not access", "do not have", "don't have", "no attachment", "not attached", "i cannot") def score(hit, tools, ans): low = (ans or "").lower() # A tool-using or escaped trial is UNSCORABLE before the hit is even read. # tools=yes covers an escaped transcript too (run_one sets it), and that # ordering is load-bearing: the error detector matches this script's own # source, so a model that cats the script to fake a transport fault must not # be able to earn an ERROR verdict for it. if tools == "yes": return "unscorable", "UNSCORABLE" if hit == "ERROR": return "error", "ERROR" if hit == "HIT": return "pass", "CONFIRMED" if any(p in low for p in REFUSAL): return "refused", "REFUSED" return "fail", "no" def esc(s): return html.escape(s, quote=True) def declares_image(m): mods = declared.get(m, "unknown") return "yes" if "image" in mods.split(",") else ("no" if mods != "unknown" else "?") # (model, modality) -> every round run for it. One trial per cell is the norm; # -r N gives N, which is what turns capability into a reliability ratio. grid = {} for model_id, modality, round_no, hit, tools, ans in rows: grid.setdefault((model_id, modality), []).append((round_no, hit, tools, ans)) # Timestamp the DATA, not the render: a -n re-render must not claim the # trials just happened. measured_ts = (log_mtime.strftime("%Y-%m-%d %H:%M") if log_mtime and rows else ("no trials in this log" if not rows else run_ts)) def tally(trials): """One cell's worth of rounds -> (css class, label).""" scored = [score(h, t, a) for (_r, h, t, a) in trials] hits = sum(1 for c, _l in scored if c == "pass") if hits == len(trials): return "pass", "CONFIRMED" # A round that used tools, or one lost to a dead socket, says nothing about # the model - so it must not drag the others down, and it must not be # averaged into a clean-looking ratio either. Report the hits over the total # rounds as long as nothing actually FAILED or was REFUSED. if any(c == "unscorable" for c, _l in scored): return "unscorable", "UNSCORABLE" if any(c == "error" for c, _l in scored): if hits and not any(c in ("fail", "refused") for c, _l in scored): return "pass", "%d/%d confirmed" % (hits, len(trials)) return "error", "ERROR" if hits: return "pass", "%d/%d confirmed" % (hits, len(trials)) if all(c == "refused" for c, _l in scored): return "refused", "REFUSED" return "fail", "no" def cell(m, d): trials = grid.get((m, d)) if not trials: return 'not run' cls, label = tally(trials) # Build the tag by concatenation: a nested-quote f-string breaks under # bash 5.2's stricter heredoc handling. tip = " | ".join("r%s: %s" % (r, ascii_only(a)[:120]) for r, _h, _t, a in trials) detail = ' title="' + esc(tip) + '"' if tip else "" return '" + label + "" # Multi-trial observations, kept apart from the single-trial probe above: # capability and reliability are different questions. RELIABILITY = { "space-bunny-free": "forwards images but reads them unreliably: 2/6 on colour naming, answering 'Green' and 'Red' for a solid blue PNG (2026-10-13). It was the alpha of minimax-m3.1-flash-free; deprecated and unserved since 2026-10-20 - kept for history only.", "muse-spark-1.3-contributor-free": "4/4 on solid-colour PNGs (2026-10-13). DEPRECATED since - kept for history only.", } # The previous run, for comparison. Every model in it that still matters is in # the table above; the rest have been retired upstream. HISTORY = """

Image probe, 2026-10-13, single trial, code known only as pixels:

space-bunny-free                   CONFIRMED
muse-spark-1.3-contributor-free    CONFIRMED  (deprecated since)
mimo-v2.6-flash-free               CONFIRMED
longcat-2.5-preview-free           CONFIRMED
step-5-preview-free                CONFIRMED  (deprecated since)
big-pickle                         no (declines)
cat-jam-free                       CHEATED (20 bash calls, pixel forensics; deprecated since)
ling-3.1-flash-free                no (declines; deprecated since)
nemotron-3.5-lightning-free        no (tried Read, then declined)

Four of those nine are now marked deprecated by models.dev and are no longer tested. beam-free, deepseek-v4.1-flash-free, glm-5.3-flash-free, nemotron-3-ultra-free and nemotron-3.5-super-vl-free did not exist in that catalogue; the model list is rebuilt from models.dev on every run, so they are covered by the table above instead of a hand-maintained list.

""" html_out = [] A = html_out.append A("") A('') A('') A("OpenCode Zen free models - input type verification") A("""""") A("

OpenCode Zen free models: verified input types

") A('
trials %s  |  model list from %s  |  ' 'source: real opencode CLI, one trial per cell
' % (esc(measured_ts), esc(source))) A("

Which free models take an image?

") summary_rows = [] for m in MODELS: trials = grid.get((m, "image")) summary_rows.append((m, declares_image(m), tally(trials) if trials else ("skip", "not tested"))) if summary_rows: A('' '') for m, dec, (vcls, verdict) in summary_rows: cls = "decl-yes" if dec == "yes" else ("decl-no" if dec == "no" else "decl") A('' '' % (esc(m), cls, dec, vcls, verdict)) A("
modelmodels.dev declares imagemeasured here
%s%s%s
") confirmed = [m for m, _d, (_c, _v) in summary_rows if _c == "pass"] declared_yes = [m for m, d, _v in summary_rows if d == "yes"] A("

Declared but not confirmed by this run: %s.

" % (", ".join("%s" % esc(m) for m in declared_yes if m not in confirmed) or "none")) A("

Why these numbers are trustworthy

") A("""

Every opencode tool is enabled by default, so a model asked about an attachment will often answer correctly without perceiving it - by running bash and decoding the file itself. Observed directly: space-bunny-free cropped and zoomed a PNG with PIL and re-read its own output files; cat-jam-free ran ~20 bash commands (connected-component labelling, ink density, ASCII-art glyph rendering) and never said it could not see the image.

""") A("

Two controls make a correct answer meaningful:

    " "
  • each trial runs in its own empty temp directory holding exactly one attachment, " "so there is nothing to list, diff or read alongside it;
    • " "
  • the answer is a random 6-character code generated per run and rendered " "only as pixels - never as text, metadata or filename. The PDF " "embeds the code as a raster image XObject rather than text, so strings " "and pdftotext cannot recover it.
") A('

A cell is CONFIRMED only when the code came back ' 'correctly and no tool call appears in the transcript. With ' '-r N the cell shows hits/N confirmed instead, ' 'which is the only way to separate capability from reliability. ' 'UNSCORABLE means the model used a tool, so the ' 'answer may have been computed from the file rather than perceived - it is neither ' 'a pass nor a fail. REFUSED is a model that said it ' 'cannot see the attachment: that is a real result, not a failure to answer. ' 'ERROR is a dead socket or an expired timeout - ' 'infrastructure, never evidence about the model.

') A("

Why pdf and video are not in this table

") A("""

They used to be probed here and both columns were removed, because neither can be settled by asking a model:

  • pdf is never delivered. The opencode client forwards an attachment as input_file only on /responses transports, and no free model is on /responses any more - they all use the provider-wide @ai-sdk/openai-compatible default. Any model that answered a pdf question correctly had run pdftoppm, which proves nothing about delivery.
  • video is dropped before the request is built. Every model that "read" a video code had run ffmpeg and read the extracted frames.
  • Both are machine-decodable by definition: any model with bash can rasterise or extract them, so a correct answer is not evidence of perception.
""") A("""

That question - does the client forward the attachment at all - is answered by capturing outgoing request bodies, not by prompting models. It was measured on 2026-10-13/17 and the conclusion is recorded in the clients next to this script.

""") # A pdf spot check run by hand, kept here because it covers the one free model # that declares pdf AND because it carried a control. See mediamatrix.sh for the # full write-up; the control is what makes it evidence rather than an anecdote. A("""

pdf spot check (2026-10-18, manual, not a column)

Glm-5.3-flash-free is the only free model that declares pdf, so it was tried by hand with the same controls as this matrix - isolated dir, one attachment, and a code existing only as pixels:

pdf x3  probe.pdf   -> Error: Provider request failed with HTTP 500
png x1  probe.png   -> read correctly, no tools used

The png control is the point: same model, same minute, and the image came back clean, so the 500 tracks the pdf rather than the host. Two limits stay attached to it. The failure is an opaque 500 - the explicit content part type "file" is not supported message quoted in the clients did not reproduce, so read that message as historical. And a 500 cannot tell "unsupported" from "temporarily unavailable", so this establishes that delivery fails, not why. Converting first (pdftoppm -png) is unaffected, and is what the control row demonstrates.

The remaining unknown, audio, is untested: no TTS engine on this host means no way to place an unknown code into an audio stream.

""") A("

Results

" "") for m in MODELS: dec = declares_image(m) cls = "decl-yes" if dec == "yes" else ("decl-no" if dec == "no" else "decl") A('' % (esc(m), cls, dec)) A(cell(m, "image")) note = RELIABILITY.get(m) or ("declared input: " + declared.get(m, "unknown")) A('' % esc(note)) A("
modeldeclares imageimagenote
%s%s%s
") A("

How to read this

    ") A("
  • One trial per cell. A pass is evidence of capability, not of " "reliability. Nothing here predicts day-to-day accuracy.
    • ") A("
  • Capability != reliability. space-bunny-free can see images " "and still answer wrongly; the reliability section keeps the two apart.
    • ") A("
  • Declared != delivered. models.dev lists what a model claims to accept. " "The opencode client forwards image, and pdf only on /responses transports; video " "and audio have previously been observed dropped entirely.
    • ") A("
  • audio is untested - no TTS engine on this host, so an unknown code " "cannot be placed into an audio stream.
    • ") A("
  • jev-* is excluded: it is a /systemone typed-decision " "endpoint, not a chat transport.
    • ") A("
  • The model list is live - models.dev free (cost 0, not deprecated) " "intersected with /zen/v1/models, so it changes without a code edit.
    • ") A("
") A("

Separate axis: reliability

") # Measured ratios from -r runs are not hardcoded: they are read out of the # log, so this section cannot drift away from the table above it. measured_repeat = {} for (m_id, modality), trials in grid.items(): if modality == "image" and len(trials) > 1: measured_repeat[m_id] = trials # keep the rounds, not the label A("") for m, trials in sorted(measured_repeat.items()): hits = sum(1 for (_r, h, t, a) in trials if score(h, t, a)[0] == "pass") unscorable = sum(1 for (_r, h, t, _a) in trials if t == "yes") errors = sum(1 for (_r, h, _t, _a) in trials if h == "ERROR") refusals = sum(1 for (_r, h, t, a) in trials if score(h, t, a)[0] == "refused") bits = ["%d/%d read the code with no tool use" % (hits, len(trials))] if refusals: bits.append("%d refused the attachment" % refusals) if unscorable: bits.append("%d went after the file with bash/PIL/ffmpeg instead" % unscorable) if errors: bits.append("%d lost to a transport error" % errors) A('' % (esc(m), len(trials), esc("; ".join(bits)))) for m, v in sorted(RELIABILITY.items()): A('' % (esc(m), esc(v))) A("
modelroundsobservation
%s%d%s
%s-%s
") if not measured_repeat: A('

No repeated image trials in this log. Re-run with ' '-r 3 to get a reliability ratio per model.

') A("

Previous run of this method

") A(HISTORY) A("

Reproduce

") A("""
./mediamatrix.sh            # one image trial per free model, writes the HTML
./mediamatrix.sh -r 3 -a     # 3 more rounds per model, appended to the archive
./mediamatrix.sh -l          # free models + declared input media, runs nothing
./mediamatrix.sh -s          # re-score saved transcripts, call no model
./mediamatrix.sh -n          # re-render HTML from .mediamatrix-raw.tsv
./mediamatrix.sh -m 'a,b' -t 300   # restrict models, per-trial seconds
""") A('

Per-cell hover shows the model\'s final answer. Raw log: ' ".mediamatrix-raw.tsv, transcripts in " ".mediamatrix-logs/.

") A("") text = "\n".join(html_out) + "\n" # Hard guarantee: the report is plain ASCII. non_ascii = sorted({c for c in text if ord(c) > 127}) if non_ascii: text = text.encode("ascii", "replace").decode("ascii") print("WARNING: replaced non-ASCII:", non_ascii[:10]) with open(out_path, "w", encoding="ascii", errors="replace") as fh: fh.write(text) print("wrote %s (%d bytes, ASCII-only)" % (out_path, len(text))) PY