Using AI in a Way You Can Trust

Using AI comes down to proving three things — whether the result can be trusted enough to put into real work, whether the gain stays with the organization and keeps running, and whether the tools are chosen critically. What each topic asks, and why, is restated at the head of its own section.

flowchart TB
  R(["🧭 How I use AI — three things to prove"]):::root
  R --> H1
  R --> H2
  R --> H3
  H1["① Problem solving · architecture  (§1)
Can the result be trusted in real work
reliability = risk management"]:::h1
  H2["② Work productivity  (§2)
Does the gain stay and keep running
personal skill → team capability"]:::h2
  H3["③ Tools · models  (§3)
Chosen knowing the limits
technical decision-making"]:::h3

  classDef root fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:2.4px;
  classDef h1 fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.5px;
  classDef h2 fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.5px;
  classDef h3 fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.5px;

0.Summary

1.Problem solving or architecture improvement

What this topic asks — can the result be trusted enough to put into real work.

Two projects are laid out in the same order — background → key prompts → verification. Read one, and the other puts the same things in the same places.

• Case 1 · 💧 Hydrological survey cost verification — a government project done at the Korea Management Analysis Institute. There is no ground-truth set to grade against. (1-1 · 1-2 · 1-3)
• Case 2 · ✈️ Aviation-law RAG — a learning project built in a public repository. A ground-truth set can be built. (1-4 · 1-5 · 1-6)

I chose these two because the presence or absence of a ground-truth set splits the entire verification design. The domains (river budgets / aviation law) have nothing to do with each other, and that is what separates what the design earned from what the domain gave.

flowchart TB
  subgraph GA["💧 Case 1 · hydrology · no ground truth"]
    direction LR
    A1["1-1 Background"]:::s1 --> A2["1-2 Key prompts"]:::s1 --> A3["1-3 Verification
traceable to the source"]:::s1
  end
  subgraph GB["✈️ Case 2 · aviation RAG · ground truth exists"]
    direction LR
    B1["1-4 Background"]:::s1 --> B2["1-5 Key prompts"]:::s1 --> B3["1-6 Verification
hundreds of runs, auto-graded"]:::s1
  end
  GA --> GB
  classDef s1 fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.3px;

💧 1-1. Case 1 Background — what the work was, and why verification was hard

1-1-1 · What a hydrological survey is

• Hydrological survey — a national program that repeatedly measures discharge at river sites.
• Discharge — how many m³ of water pass that site per second.
• Where it is used — flood forecasting · dam release decisions · water allocation · river planning.
• Scale — several hundred sites nationwide. Repeated every year. Funded every year.

1-1-2 · What cost verification is

• The budget is built on a per-site annual unit cost — "how much to survey one site for one year".
• The bidding contractor submits its own cost estimate.
• A third-party institute independently re-estimates the same cost — that is the role of the Korea Management Analysis Institute.
• The two values are compared, over-estimated items are identified, and the findings are presented to the government with evidence.

The point is to trace everything back to the standards the state has published, not to instinct or custom.

flowchart LR
  A["Allocated budget
per-site annual unit cost
money actually set aside"]:::a
  B["Back-calculated from the Manual
cost-plus-fixed-fee
money the standard implies"]:::b
  A --- Q{{"The two differ
at which assumption?"}}:::q --- B
  Q --> T["Trace each assumption back to the source
measurement count · staffing · adjustment factors …"]:::t
  classDef a fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef b fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.3px;
  classDef q fill:#EFEBFE,color:#3B2A82,stroke:#5B3FD9,stroke-width:2.4px;
  classDef t fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.3px;

1-1-3 · What the app did · did not do

Three things are traced — how many measurements were assumed, how many people and hours per measurement, and which adjustment factors were applied. Change an assumption and the screen shows how the numbers move.

1-1-4 · The previous method (10 hours → 1 minute)

• The current procedure is manual Excel work. No code, no AI.
• It takes more than 10 hours per case.
• Estimation and checking happen in the same hands, inside the same sheet. So even when an error creeps in, it has nowhere to surface.
• Above all, no one but the person who produced the result can readily check it.

That last row is the actual goal of the project: letting someone who knows nothing about hydrological surveys verify the work within an hour, just by following the evidence the app presents. It is about increasing the number of people who can verify.

1-1-5 · There is no ground-truth set

• There is someone who judges — an experienced verifier decides after looking at the evidence.
• But there are no labels to grade against in advance — no settled value called "the true cost".
• All that exists is the calculation procedure fixed by law — and even its application leaves room for interpretation.
• So accuracy cannot be measured automatically. A person has to look.

So I replaced accuracy with traceability. Instead of grading whether a value is right, the criterion became whether every number can be traced back to its source — because if it can be compared by eye, you can say it is wrong even without knowing the right answer.

💧 1-2. Case 1 Key prompts — demanding verification before code

1-2-1 · The first prompt, verbatim

Instead of a polished summary, the first prompt is kept raw. It came from audio dictation, so the sentences are rough and there are typos ("파악을 할 수 있느", "validtion") — which is exactly why it shows the flow of thought rather than a conclusion. Only the key parts are highlighted. (One internal data folder name is masked.)

강 별로 수문조사들을 한 것들이 다 저장되어 있는 데 너무 내용이 많고 그 예산이라고 할만한 부분도 카테고리가 너무 많다. 그래서 이 전체적으로 파악이 안되는데. 파악을 할 수 있느. 어떻게 보면 1분만에 이해할 수 있는 적절한 spread sheet와 인사이트를 보여주는 html을 만들어주고, 그 후 10분 정도 시간 들여서 이해할 수 있는 html도 만들어주고. 그 다음에 결국에는 궁극적으로 내가 원하는것은 1개 강의 수문을 조사시 몇명이 몇시간 얼마의 비용이 들어가는지 분석하는 것이다. 하지만 이게 바로 하게 되면 환각이 일어날수도있고, 이게 나름의 방정식일텐데 이해하기가 어려울 수도 있다. 마치 선형회귀로 서울 아파트 가격을 예측할때 다양한 feature들을 고려해서 더 정확하게 예측할 수 있겠다만, 그러면 이해가 안되니 heuristic하게 일단 size만 가지고 예측하는 것처럼. 처음에는 아주 간단한 마치 하나의 feature 고려하는 것처럼 정확도는 떨어지더라도 validtion이 될 수 있는 방향으로 해야겟다. html을 만들때 아무래도 streamlit으로 해야될 거 같으니. 아까 말한것과 다르게 html이 아니라 streamlit으로 만들어줘. 다시한번 말하는데 validation이 가장 중요하니, 가 문장의 근거는 실제 [내부 데이터]를 한번 가공해서 그 데이터를 매번 citation해줘.

Original prompt, in Korean — kept verbatim.

The line of thought inside that one paragraph

1. There is too much data to see the whole Many files, and the budget categories are chopped far too finely.
2. So I asked for the forest and the tree at once It will take a while anyway, so get both out of a single wait. The AI is good at the overall structure; I adjust the details.
3. Why not ask only for the detail? The narrower and deeper the result, the more easily it diverges from what I wanted. And if the AI invents plausible numbers, there is no way to notice.
4. So I asked first for a form I could check Even at some cost in accuracy, I must be able to follow where a value came from. Even if it fails, the overall context remains as material for the next turn.
5. In the end what matters most is verification Before building fast comes whether the result can be trusted.
6. So I made it attach a source to every number Which file, which cell a value came from is shown on screen alongside it.
7. I switched tools mid-request too Static HTML → Streamlit. Because I need to change values and see the effect.

Not a single line about code. Every requirement is about "how will I trust this", not "what shall we build".

1-2-2 · Why I asked for the forest and the tree at once

This is the most misread part. It does not mean "make me three screens", nor "I will cross-check three results".

• The wait happens only once — a large request takes a long time anyway. That single wait returns both the final target (the tree) and the overall context (the forest). With luck, the tree arrives right away.
• The more local it is, the more it misses — detailed calculations get built while the AI still does not know what I want. Narrow and deep means easy to diverge.
• The overall structure it usually gets right — pulling a skeleton out of scattered material is something it does well. So the forest is delegated and only the tree is adjusted.
• Failing costs nothing — even if the tree misses, the forest remains. That forest becomes material the AI refers to in the next turn, so the same background never has to be explained again.

1-2-3 · How the three screens actually came out

The conversation that followed is carried below in 1-2-4~1-2-8 as raw (masked) text, and the outcome is organized in 1-3 — the process of turning back an AI that had mistaken the purpose, refusing coordinate citations, and distrusting the extraction channel itself.

How the conversation developed — five corrections

In every one, the AI was wrong first and I found it and changed direction.

1-2-4 · Misreading the purpose itself

아이고 니가 목적을 완전히 헷갈렸는데 … 우리가 원가 검증을 하는 프로젝트를 하고 있거든. 이 예산이 실제 그 법령하고 맞는지를 항상 보는 …

Original prompt, in Korean — kept verbatim.

1-2-5 · Narrowing the scope to an atom

누가 그냥 todo를 해줘. 비용모델은 1개 강의 수문 조사시 몇명이 몇시간 얼마의 비용인지 부터 아는거로 하자.

새로운 비용모델을 만들자. 즉 매우 지엽적인 수문 1개 조사시에 몇명이 몇시간 얼마의 비용이 들어가는 지. 즉 특정 수문을 현재 조건을 보고 법령에 맞게 실제로 나오는지. 다른 예산은 일단 필요없다.

Original prompt, in Korean — kept verbatim.

1-2-6 · Evidence that cannot be falsified

그 근거라는 부분을 spreadsheet의 일부를 복사붙여넣기 해서 누구나 spreadsheet에서 온 것처럼 해줘. … 새로운 수식이 나오면 그걸 근거를 보여줘

Original prompt, in Korean — kept verbatim.

1-2-7 · Do not trust text extraction — show the source as it is

pdf는 확실히 표로 되어 있으니 읽기가 어렵네. pdf는 스크린샷도 그냥 찍어서 근거로 보여주라

Original prompt, in Korean — kept verbatim.

1-2-8 · Demand the formula, not the result — so the evidence, not the answer, can be compared

현장 출동인원이 몇명이 필요한지는 원래 알 수 없나? … 적용보정계수 같이 수식으로 계산한것들은 그 수식까지 표현해야지. 모든 수식은 다 표현해줘.

Original prompt, in Korean — kept verbatim.

💧 1-3. Case 1 Verification and rollout — four layers before it reaches real work

1-3-1 · How to read this section

This is the core of the work — the validation that AI output passes through before it enters real use. In an environment where data cannot leave the building and a wrong value is immediately a legal and financial risk, reliability was not a feature but a requirement.

Since there is no single correct answer, nothing can be scored. Instead every value is made traceable to its source, so that if something is wrong, a person can find where.

flowchart TB
  A["AI output"]:::src
  A --> B["Source comparison
excerpt keeping row·column coords"]:::dev
  A --> C["Channel verification
PDF as image, table intact"]:::dev
  A --> D["Cross-file invariants
do other files say the same"]:::dev
  A --> E["Independent 2nd estimate
a path that never sees the budget"]:::dev
  B --> F["Put into real use"]:::ok
  C --> F
  D --> F
  E --> F
  D -.->|"mismatches are not hidden"| G["published as known data issues"]:::warn
  classDef src fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.3px;
  classDef dev fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.3px;
  classDef ok fill:#5B3FD9,stroke:#3A2A7A,color:#ffffff,stroke-width:1.4px;
  classDef warn fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;

Below is the actual code — not a claim, but how it runs. (File names and amounts are parameters, so nothing is exposed; the logic is unchanged.)

1-3-2 · Source comparison — show the cited cell in its original grid

flowchart LR
  Q["cell to cite
e.g. E9"]:::inp --> R["raw_slice
keeps Excel row no. · column letter"]:::act
  R --> S["_row_style
called for every table row"]:::act
  S --> T{"is this cell
in the cited list"}:::dec
  T -->|"cell matches"| U["deep yellow + bold"]:::ok
  T -->|"row only"| V["pale yellow"]:::ok
  T -->|"no"| W["no colour"]:::inp
  classDef inp fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.2px;
  classDef act fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.2px;
  classDef dec fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef ok  fill:#eaf6f0,stroke:#0a7048,color:#08512f,stroke-width:1.2px;
  classDef bad fill:#fdeeec,stroke:#a3241d,color:#7a1a15,stroke-width:1.2px;

def evidence(rel_file, sheet_name, row_from, row_to, col_from="A", col_to=None,
             note="", expanded=False, hl_rows=None, hl_cells=None):
    """Attach evidence as an excerpt of the original spreadsheet, highlighting cited rows (pale yellow) and cells (deep yellow)."""
    if hl_rows is None:
        hl_rows = []
    if hl_cells is None:
        hl_cells = []
    rows = set(hl_rows)    # Excel row numbers to highlight
    cells = set(hl_cells)  # cell coordinates to highlight (e.g. "E9")
    df = raw_slice(rel_file, sheet_name, row_from, row_to, col_from, col_to)

    # callback pandas Styler calls per row (same role as a function pointer argument in C)
    def _row_style(row):
        out = []
        for col in df.columns:
            cell_name = col + str(row.name)  # column letter + row number = cell coord (e.g. "E9")
            if cell_name in cells:
                out.append("background-color:#ffd54a; font-weight:600")  # cited cell
            elif row.name in rows:
                out.append("background-color:#fff3c4")                   # cited row
            else:
                out.append("")
        return out
# → an invented coordinate and a real one are separated by colour on screen.

1-3-3 · Channel verification — distrust the extraction path itself

flowchart LR
  A["PDF evidence requested"]:::inp --> B{"is there a rendered
image of that page"}:::dec
  B -->|"yes"| C["show the image as is
table layout survives"]:::ok
  B -->|"no"| D["fall back to extracted text
accept that tables may break"]:::bad
  classDef inp fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.2px;
  classDef act fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.2px;
  classDef dec fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef ok  fill:#eaf6f0,stroke:#0a7048,color:#08512f,stroke-width:1.2px;
  classDef bad fill:#fdeeec,stroke:#a3241d,color:#7a1a15,stroke-width:1.2px;

def evidence_pdf(label, txt_name, note="", expanded=False, hl=None):
    """PDF evidence: show a page screenshot (cited part highlighted). Fall back to text if absent."""
    title = "📎 근거 발췌 — `표준품셈(2023).pdf` " + label
    with st.expander(title, expanded=expanded):
        png = DATA / (txt_name + ".png")
        if png.exists():
            st.image(str(png), width='stretch')      # the image as is — nothing breaks the table
            st.caption("↑ PDF 페이지 스크린샷 (etl.py가 렌더링, 🟡 = 인용 문구). " + note)
        else:
            st.code(_read_txt(txt_name), language=None)  # fallback: machine-extracted text
            st.caption("↑ PDF에서 기계 추출한 원문. " + note)

# raw_slice — slice into a 2-D array keeping Excel coordinates (row number, column letter) alive
data = []           # data[row][col]
row_names = []      # Excel row numbers as the index
for r in range(row_from, row_to + 1):
    line = []
    for c in range(c0, c1 + 1):
        line.append(ws.cell(r, c).value)
    data.append(line)
    row_names.append(r)

1-3-4 · Cross-file invariants — do different files state the same fact

This first checks whether the baseline the last mechanism will use is sound. The per-site unit price in the budget table is not a value to accept just because it is written down, and if the baseline is wrong, however precisely the gap (%) is computed it means nothing.

Where the previous three mechanisms make things checkable by a person, this check catches things even when nobody looks. A self-consistent hallucination cannot pass it.

flowchart TB
  A["row list: records"]:::inp --> B{"any blank among
qty · price · total"}:::dec
  B -->|"yes"| S["not eligible — skip"]:::inp
  B -->|"no"| C["model = qty × price
the total cell is not read"]:::act
  C --> D["err = total − model"]:::act
  D --> E{"is the error below 0.5"}:::dec
  E -->|"yes"| F["exact-match count +1"]:::ok
  E -->|"no"| G["update max error rate, publish"]:::bad
  classDef inp fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.2px;
  classDef act fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.2px;
  classDef dec fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef ok  fill:#eaf6f0,stroke:#0a7048,color:#08512f,stroke-width:1.2px;
  classDef bad fill:#fdeeec,stroke:#a3241d,color:#7a1a15,stroke-width:1.2px;

# Cross-file invariant check: quantity × unit price = total. Recompute the product without reading the total cell, then compare.
records = items.to_dict("records")  # DataFrame → array of dicts (iterated like an array of structs in C)
n_exact = 0         # count of rows matching with zero error
max_err_pct = 0.0   # maximum error rate (%)
for i in range(len(records)):
    row = records[i]
    qty, price, total_val = row["수량"], row["단가_백만원"], row["총계_백만원"]
    if pd.isna(qty) or pd.isna(price) or pd.isna(total_val):
        continue  # if any of the three is blank, the row is not eligible
    model = qty * price          # these are facts stated by different cells, so
    err = total_val - model      # a self-consistent hallucination cannot satisfy this equation
    if abs(err) < 0.5:
        n_exact = n_exact + 1
    if total_val != 0:  # rows with a zero total are excluded from the error rate
        err_pct = abs(err) / total_val * 100.0
        if err_pct > max_err_pct:
            max_err_pct = err_pct

How the 4 mismatches were handled

The [data] / [assumption] labelling convention settled here

1-3-5 · Independent second estimate — reach the same answer by two paths

The same cost is computed by two paths that know nothing of each other. Like debit = credit in double-entry bookkeeping, or two independent witnesses.

flowchart TB
  L["Manual person-days
by grade"]:::inp --> M["× adjustment factor
derived from width band"]:::act
  M --> N["× published wage rates
summed by grade = direct labour"]:::act
  N --> O["× overhead·fee + travel"]:::act
  O --> P["annual cost
the budget has never been read"]:::ok
  P --> R{"gap = annual vs UNIT
how far apart"}:::dec
  U["allocated unit cost UNIT
an entirely separate source"]:::inp -.-> R
  R --> Z["expose the causes as adjustable variables"]:::act
  classDef inp fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.2px;
  classDef act fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.2px;
  classDef dec fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef ok  fill:#eaf6f0,stroke:#0a7048,color:#08512f,stroke-width:1.2px;
  classDef bad fill:#fdeeec,stroke:#a3241d,color:#7a1a15,stroke-width:1.2px;

def once_cost(task_name, fct):
    """Payment for one measurement (travel included, VAT excluded) — the budget is never read."""
    r = labor[labor["업무"] == task_name].iloc[0]  # one row for this task from the manual
    dl = 0.0                                       # running sum of direct labour cost
    for g in GRADES:
        man_days = float(r[g]) * fct               # person-days by grade × adjustment factor
        dl = dl + man_days * wages[g]              # × published wage rate → accumulate
    cost = dl * (1 + ovh_rate) * (1 + fee_rate) + team * fare * 2
    return cost

# Direct labour for per-site fixed tasks: double loop over task rows × grades
fixed_records = fixed_tasks.to_dict("records")
fixed_dl = 0.0
for i in range(len(fixed_records)):
    r = fixed_records[i]
    for g in GRADES:
        fixed_dl = fixed_dl + float(r[g]) * wages[g]

annual = (n_py*cost_py + n_hs*cost_hs + fixed_cost) * vat_mult  # annual cost derived from the law alone
gap = (annual - UNIT) / UNIT * 100      # UNIT = allocated unit cost (value lives elsewhere). Difference between the two paths (%)
# → annual has never seen the budget. That is why the gap is a real signal.

1-3-6 · The screen that passed four layers

1-3-7 · After rollout — decision-maker feedback

The four above are verification before shipping; this is after. What is said here becomes the input to the next cycle.

The analysis closed with a results briefing to the decision-maker, a non-developer. There were three points. (Names and amounts masked; summarized, not verbatim.)

✈️ 1-4. Case 2 Background — finding the right clause in 1,297 pages of regulation

To find and feed, the code has to be prepared in advance. The terms that keep recurring below (1-5·1-6) are exactly those assembly parts.

1-4-1 · What this project is

• Corpus — the US Federal Aviation Regulations, 14 CFR. 1,297 pages in the original.
• Task — a RAG that answers a question by finding the clause (§) that supports it.
• Context — a learning project in a public repository, and a submission to a course contest.
• How it differs from 💧 — data and code are entirely public, so anyone can reproduce the claims in this document.

1-4-2 · What was at stake

• Regulation must not be wrong — pointing at the wrong clause makes someone break a rule in fluent prose.
• Plausible and correct look the same — the sentence reads smoothly even when the clause cited is wrong.
• So judging by eye does not scale.

1-4-3 · A ground-truth set can be built

• The answer can be written down in advance, like "the governing clause is §91.151" — the clause sits in a settled document, so this is comparison, not interpretation.
• With a ground-truth set, verification stops being something you look at and becomes an experiment you run overnight.
• So the bottleneck shifts from "can it be trusted" to "there are too many configurations".

✈️ 1-5. Case 2 Key prompts — asking not about the answer, but where it came from

This section shows the principles set in 1-2 operating in the same places even as the domain changes. Here too the prompts are kept raw, typos and dictation artefacts included.

1-5-1 · Catching a description that differed from the code

Context — the AI described the hybrid search as RRF (rank fusion), and that description went straight into five places in the public documentation: diagrams, glossary, body text. Yet reading it, I never quite got it.

아니, 내가 궁금한건 앙상블을 어떻게 한지 아직 이해안돼. 어느 비율로 어떤 것을 쓰고, 언제 어떤 임베딩을 썼나?

Original prompt, in Korean — kept verbatim.

1-5-2 · Taking back a judgment the AI deferred — rejecting agentic

Context — after implementing agentic RAG itself and showing a quality advantage, the AI handed the decision over with Your call. Which way should we settle on?

지금 에이전틱으로 하면 그래도 10배 정도의 토큰 소모가 있는거 아니가? 우리는 단일 래그로 했을 때 복잡한 문제도 이미 잘 풀었다는것을 확인했어. 이정도 데이터에서는 에이전틱 루프 보다는 단일 래그도 잘 된다면 굳이 토큰 많이 써가면서 에이전트를 붙일 필요성ㅇ ㅣ있을까?

Original prompt, in Korean — kept verbatim.

1-5-3 · Put rules in configuration, not conversation

Context — I was running two sessions at once. One handled experiment code, the other documentation, both on the same repository. That leads to one session committing the other's unfinished work.

그리고 너가 커밋할때는 너가 작업한것만 올려줘. 너가 작업하지 않은건 그대로 둬

claude-memory-backup 에서 커밋 관련된 모든 엠디 파일에 '한 세션에서는 그 세션이 작업한 것만 커밋한다'는 것을 추가해줘.

Original prompt, in Korean — kept verbatim.

✈️ 1-6. Case 2 Verification and rollout — with a ground-truth set, grade hundreds of runs automatically

Where verification in Case 1 was about making things traceable by a person, here it is about making them gradable by a machine. Two mechanisms were built, and 45 configurations were run on top of them.

1-6-1 · A secret exam — set aside questions never used for tuning

Holdout — sitting the exam with questions you tuned on inflates the score. So a few questions are set aside from the start as a "secret exam" used only for grading, never for tuning. I wrote 14 by hand and split them by purpose into three.

The refusal questions weren't there at first — I was only writing in-scope questions until I realized late that "does it avoid making up an answer to an out-of-scope question" also had to be tested, and added them.

1-6-2 · Cheap grading / expensive grading, separated

Because retrieval grading runs in code, all 45 configurations were swept for $0, and the paid generation grading was spent only on the few survivors — the same volume of experiments at roughly a tenth of the cost. That said, neither the generation grading nor checking that the grading is right is free, so this is less "completely free" than calling the expensive grader as little as possible.

1-6-3 · Result — on a tie, the cheaper one

Coverage = the fraction of correct clauses retrieval actually found (found ÷ total correct). MRR = how high the correct answer ranked in the results (closer to 1 the higher). Both are retrieval grading, scored in code — which is why they could be applied to all 45 configurations.

1-6-4 · After rollout — the instructor's score

Same here. After the automatic grading was done, what a person pointed out became the starting point of the next improvement.

1-7. What the two cases leave behind

Different conditions split the verification method. Here is where they split, on one page.

2.Productivity gains using AI

What this topic asks — does the gain remain as team capability rather than personal skill. And for that, how far can work be delegated to AI.

flowchart LR
  M{{"productivity = the scope that reproduces
after the owner changes"}}:::root
  M --> P["Meeting notes
where decisions are made"]:::s2
  P --> A["AI — read and propose
quoting the supporting sentence"]:::s2
  A --> V["Human — judge
approve · question · reject, item by item"]:::gate
  V -->|"only what is approved"| B["GitHub Projects board
where the record lives"]:::s2
  V -.->|"ask back if ambiguous"| P
  classDef root fill:#EFEBFE,color:#3B2A82,stroke:#5B3FD9,stroke-width:2.4px;
  classDef s2 fill:#fdf4e8,stroke:#a8560a,color:#7a3d06,stroke-width:1.3px;
  classDef gate fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.8px;

2-1. Background — the problem the organization had

2-1-1 · Intent stops at one person

[stated] About 40 staff, of whom 3 are developers — the team lead (me) and two team members.

2-1-2 · Why the notes cannot be used as they are

Meetings run over video, following whatever comes to mind. Two things are in that record at once.

2-2. Approach and implementation

2-2-1 · Extraction is AI, Requirement Engineering is human

2-2-2 · Approval gate — how far to delegate, and where to cut

It starts from the premise that AI can be wrong. So what was designed is not "how to instruct it well" but "how to make it come back to a human when it is wrong".

2-2-3 · Task model — put information in the structure, not the title

Moving what was said in a meeting into a Board item, the altitude to write at kept wobbling. You can write the ultimate goal, or a single checklist line.

I fixed it with one test — if you can clearly answer "is this done or not?", it is at task altitude.

2-2-4 · Leak gate — keeping meeting notes from leaving the machine

2-2-5 · Judgment into record

2-3. Effects

2-3-1 · Changes felt in the organization

[stated] Changes felt directly in the organization.

2-3-2 · What the records confirm

[log] Verifiable in the repository and session records.

2-3-3 · What the gates actually caught

This is the evidence that the earlier mechanisms were not decoration. It is the record of the three things named at the head of this section actually being caught — because the proof that a system works is not speed but what a human blocked.

2-3-4 · The metric under test

At the head of this section I set "the scope that reproduces after the owner changes" as the metric. That metric is now being tested in a way I did not design.

2-3-5 · Limits and next steps

This is not a one-time handover but a loop that keeps turning. A person stands where business language becomes development language, and that spot is passed every week. Two things need to rise — verification speed and accuracy. Raising only one means nothing.

3.Tool and model analysis

What this topic asks — are the tools chosen with their limits understood.

flowchart TB
  C1["Tools · models used
what was used"]:::s3 --> C2["Why chosen
why this tool — a strategic choice"]:::s3 --> C3["Strengths · limits
where to trust, where to doubt"]:::s3
  classDef s3 fill:#eef3f6,stroke:#33505f,color:#333C52,stroke-width:1.3px;

3-1. Tools and models used

3-1-1 · How I got here

1. Asking and learning — the arrival of ChatGPT ChatGPT appeared while I was at École 42. I learned concepts by asking them one at a time, and started with copy-and-paste for code. Once search was added, I made it search first for anything current.
2. Does it read everything if you put in a lot? — the Gemini context experiment Exploiting the large input, I fed in meeting transcripts of 1, 2 and 10 hours. Up to one hour it held; from two or three hours content started vanishing, and numbers and relationships in particular came out wrong. A million-token context is not the same as accuracy.
3. Hence RAG — and NotebookLM's opposite limit The conclusion: rather than putting in a lot, find and insert only what is needed. NotebookLM was faithful to its sources but weak at weaving complex context together. That is where I saw that precise citation and broad understanding pull against each other.
4. How far does generation go? — images and video I measured the limits while making learning material. Korean text in infographics barely works (GPT ✗ · Gemini ○), and logical development such as matrices or network flows only looks plausible while being wrong. So I draw those myself with Manim (3Blue1Brown's visualization library).
5. On to agents — Antigravity → Cursor Antigravity gave me my first feel for how an agentic IDE runs, and in Cursor I saw it — what matters is not model performance alone but a design that asks about the workflow first. Attaching a browser, it even checked its own work by screenshot. But the API charges were expensive.
6. Converging on Claude Code — and learning mode A flat rate could carry the same work, and above all the output style could be switched to learning. That was decisive.
7. Now — roles divided Work in Claude Code, current news in Perplexity, long videos and documents in Gemini. And I am checking with local models what kind of constraint on-premise actually is.

3-1-2 · Role of each tool

3-2. Why these were chosen

3-2-1 · learning mode was decisive

Switch the output to learning under /config → Output style and, before giving the answer, it also explains why it did it that way and what should be learned. (The old command was /output-style, deprecated in v2.1.73 and removed in v2.1.91.)

3-2-2 · Cost is part of the choice

3-3. Assessment of the models — strengths and limits

3-3-1 · Strengths — where the model earned its keep

3-3-2 · Limits — observed directly, per model

3-3-3 · When to use what

3-3-4 · The direction I am interested in

The cost verification in §1 could not send client data outside, and the meeting notes in §2 ran into the standard that "company material must not be handed to outside services". Most of the work I have done has been in places like that.

A.Appendix — making a running system verifiable by others

The claim in §2 is about an organization; this appendix is the record of applying the same principle to a system run by one person. It is not an organizational case, so it was moved out of the main text, but it sits on the same line as this document's theme — "make what AI wrote falsifiable by a third party".

The subject is a single-node k3s cluster I have run alone since 2025 — about 20 services under *.bit-habit.com run on it.

A-1 · How I got here

I passed a Linux certification in my first year of middle school. I passed it by rote, and I knew then that I had not understood it. I am thirty-eight now, so for 25 years since, I have been looking for ways to overturn that.

• 2022 · École 42 — set up VMs myself.
• 2023 · wrote Dockerfiles myself.
• 2024 · evaluating other people's k3s assignments made me want to take it up again.
• 2025 · put a server on the Oracle Cloud free tier.
• 2026 · built a k3s cluster on it and run it myself — around 20 services, with certificates, routing, deployment and incidents all in my own hands.

This cluster is the continuation of that, and I imposed the constraint myself — how far can one person get on one free machine. Four ARM cores, 20 services, $0 a month.

A-2 · What I asked the AI to do · did not

The result is infra.bit-habit.com. The source is in the public repository bookseal/bit-habit-infra, and the live state is checked externally every 60 seconds at status.bit-habit.com.

A-3 · Three marks of a falsifiable document

"The documentation is good" is an unfalsifiable claim. Instead I look at whether it is in a form a third party can compare against. Three things went in.

A-4 · Incident postmortem — 200 OK

habit.bit-habit.com served a different site for months. Not an error page — a valid certificate and 200 OK, serving a perfectly wrong site. Because the uptime monitor and my own status page both asked only "did a 200 come back?", the incident was invisible to every automated check.

Diagnosis went by layer-wise bisection — at each step, ask "is the problem above or below this layer" and cut one side away. The decisive move was calling the app's Service directly from inside the cluster. The app was fine, so the fault lay in front of it. The real cause was two independent failures overlapping.

flowchart TB
  B["Browser requests
habit.bit-habit.com"]:::s2b --> T["Traefik reads the routing table"]:::s2b
  T --> F1["Failure ①
rule points at the wrong Service
portfolio ← not the app"]:::bad
  F1 --> N["nginx in the portfolio Pod
receives the request"]:::s2b
  N --> F2["Failure ②
no server_name matches this host,
so it falls back to the default block"]:::bad
  F2 --> R["200 OK + the wrong site
no error anywhere"]:::bad
  APP["Flutter app
fine · healthy · unreachable
0 traffic for 103 days"]:::ghost
  classDef s2b fill:#EFEBFE,stroke:#5B3FD9,color:#3B2A82,stroke-width:1.3px;
  classDef bad fill:#fdeeec,stroke:#a3241d,color:#7a1a15,stroke-width:1.6px;
  classDef ghost fill:#f2f2f4,stroke:#8a8a95,color:#55555f,stroke-dasharray:4 3;

The root cause was not a routing bug. It was a single-source-of-truth bug. The app existed only in the cluster and in no repository at all. So the one line pointing at that app lived in a file that did not know the app existed. An ownerless rule drifts, and nobody notices.

A-5 · Why I built it, and how I used AI

B.Appendix — masking policy

C.Appendix — public material