From dd2bdfa13d9838a9bc4d225341f821e589b6dd9a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gustavo=20Dias=20Apolin=C3=A1rio?= Date: Tue, 25 Aug 2026 17:23:04 -0300 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20aprofunda=20o=20README=20em=20ingl?= =?UTF-8?q?=C3=AAs=20a=20partir=20das=20decis=C3=B5es=20de=20projeto?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Leitura de docs/DECISOES_PROJETO e dos relatórios de mineração antes de reescrever, conforme a regra. O achado principal estava reduzido a uma frase. Ele tem duas partes: a subjetividade do critério prediz o yield na mineração por palavra-chave, com cinco de cinco pontos na ordem prevista, e a ancoragem em identificador de regra de linter restaura o yield dos smells subjetivos, levando o de lista longa de parâmetros de três para setenta e sete por cento. Entram também a separação de papéis entre famílias de modelo, com a troca do alvo de treino para evitar autocircularidade, o reconhecimento da circularidade residual, a rejeição do CodeBLEU por penalizar refatoração correta e diferente, e a linha de mineração de pull requests testada e abandonada. --- README.md | 81 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index bd9bf4d..fa7552b 100644 --- a/README.md +++ b/README.md @@ -13,31 +13,76 @@ Bernardo Vale dos Santos Bento and Filipe Mauro da Terra Caldeira. ## The main finding -Mining commits that **remove a specific linter rule** (Pylint R0913, Ruff PLR2004 and -others) yields 5 to 25 times more valid refactoring pairs than heuristics over commit -messages. The multiplier is not uniform: it scales with how objectively the smell can be -defined. A parameter count is unambiguous, so its yield is high. "Long method" depends on -thresholds, so its yield is low. +Mining refactoring pairs from commit history has a yield problem, and the yield is not uniform +across smells. Using keyword search over commit messages, it tracks **how objectively the smell can +be defined**: -This shaped the whole pipeline. Numbers and reasoning are in `docs/DECISOES_PROJETO.md`. +| Smell | Yield | Why | +|---|---|---| +| Dead Code | 38% | binary: the code is gone or it is not | +| Long Method | 19% | semi-objective: a numeric threshold on length | +| Deep Nesting | 4% | structural but contextual | +| Long Parameter List | 3% | "long" is a judgement call | +| Magic Numbers | 1.2% (n = 1,376) | "magic" depends on the domain | + +Five out of five points fall in the predicted order, monotonically decreasing. + +**But subjectivity turns out not to be destiny.** Anchoring the search on a linter **rule identifier** +instead of on commit-message wording, and filtering the diff for the structural change that the +refactoring implies, restores the yield of the subjective smells: + +| Smell | Keyword mining | Rule-ID anchoring | +|---|---|---| +| Long Parameter List | 3% | **77%** | +| Magic Numbers | 1.2% | **19%** | +| Deep Nesting | 4% | **12%** | +| Long Method | 19% | **42%** | + +Measured over a single batch of 672 judged candidates, of which 180 were real. + +The reading matters more than the numbers. What is subjective is the *natural language* people use +to describe a change in a commit message. The change itself is often perfectly objective, and a +linter rule already encodes that objectivity. Mining against the rule rather than against the prose +converts a subjective criterion into a mineable one. + +The honest note that goes with it: the initial pilot for Dead Code reported 60% and the full batch +came in at 38%, because the first sample was not representative. The qualitative pattern held, the +number did not. + +## Keeping the dataset honest + +Four decisions, each guarding against a specific way this kind of dataset goes wrong. + +**Roles are separated across model families.** The judge is Gemma and it only ever evaluates, never +generates. Generation for two of the tiers uses different families. The fine-tuning target was +**changed** from Qwen2.5-Coder to Stable Code Instruct precisely because the original target shared +a family with one of the generators, which is a path to a model grading its own dialect. + +**Residual circularity is acknowledged rather than hidden.** The structural AST signal used in one +tier's prompt was chosen specifically so the criteria would not come from the judge's own prose. + +**The oracles never touch training data.** PyRef acts as a deterministic validator producing ground +truth, and Sourcery is the comparison baseline that the trained model has to match or beat. A tool +cannot be both the teacher and the exam. -## Dataset construction +**The split is by repository, not by example.** Train and test never share a repository, with +assertions enforcing it, so a model cannot memorise one project's idioms and score well on itself. -The pipeline mined 5,072 candidate pairs. An LLM judge (Gemma, running locally) approved -616 of them. A human quality probe then audited a stratified sample of 50, blind to the AST -proxy, with population precision reweighted by base rate and inter annotator agreement -measured by Cohen's kappa. Two annotators, followed by an LLM assisted re audit. +**Quality was measured before training, deliberately.** A stratified sample of 50 pairs was audited +blind to the AST proxy, with population precision reweighted by base rate and agreement measured by +Cohen's kappa, followed by an LLM-assisted re-audit. Measuring data quality after training the model +tells you nothing you can act on. -Three decisions kept the dataset honest: +## An evaluation metric that was rejected -**Oracle firewall.** PyRef and Sourcery are reserved for evaluation and never produce -training data. A tool cannot be both the teacher and the exam. +CodeBLEU was discarded on purpose: it penalises a refactoring that is correct but different from the +reference, which is the normal case. Evaluation instead uses smell resolution as the primary metric, +plus behaviour preservation and non-introduction of new smells. -**Repository level split.** Train and test never share a repository, so a model cannot -memorise one project's idioms and score well on itself. +## A line that was tested and abandoned -**Quality probe before training.** The sample was audited before any fine tuning started, -deliberately. Measuring the data after training the model tells you nothing you can act on. +Mining pull requests from four large projects was piloted and dropped. Only one of the four carried +usable signal, and the empirical result is recorded rather than the idea being quietly forgotten. ## The five smells From 70a61808927919378fa3e9d30ae29a5c86c0d89c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gustavo=20Dias=20Apolin=C3=A1rio?= Date: Tue, 25 Aug 2026 17:26:29 -0300 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20acrescenta=20o=20problema=20do=20sm?= =?UTF-8?q?ell=20aparente=20ao=20README=20em=20ingl=C3=AAs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Leitura dos documentos de docs que eu havia deixado de fora, entre eles a nota sobre dados negativos. Faltava o problema conceitualmente mais difícil do projeto: distinguir smell real de smell aparente, que é o que separa uma ferramenta de detecção de uma que pode reescrever código sem quebrá-lo. Entram também as duas razões pelas quais gerar os negativos sinteticamente falha, com a distinção entre precisão de rótulo e cobertura do espaço, e a destilação ancorada em quatro movimentos com test set inteiramente real. --- README.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/README.md b/README.md index fa7552b..e5a2d77 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,38 @@ The honest note that goes with it: the initial pilot for Dead Code reported 60% came in at 38%, because the first sample was not representative. The qualitative pattern held, the number did not. +## The harder problem: telling a real smell from an apparent one + +A tool that rewrites code automatically has to answer a question detection alone does not: is this +a smell that should be fixed, or code that only looks like one and must be left alone? + +Nested conditionals that implement a security check. A parameter list frozen by a stable public API. +A magic number that is a protocol code. Each trips a detector, and each would be damaged by the +refactoring the detector implies. A safety gate needs negative examples, and **labelled negative +examples of this kind essentially do not exist**. + +The obvious move is to generate them with a language model and check them by hand. That fails for +two reasons worth separating: + +**The model generates from its own prior, not from the real distribution.** It produces the +prototypical archetype and leaves whole regions of the decision boundary uncovered. + +**Manual review fixes label precision, not coverage.** You cannot verify the absence of something +that was never generated. Reviewing a biased sample carefully yields a carefully verified biased +sample. + +**The approach adopted is anchored distillation**, in four moves. Mine a small real seed of code +where an actual developer judged "this is fine", using linter suppressions and won't-fix markers as +proxy labels. Use that seed as a few-shot anchor so generation orbits the real distribution instead +of the model's prior. Review every example by hand, re-judging "genuinely not a smell" against +"a smell that was merely suppressed". And **hold out a test set that is 100% real, never synthetic**, +so the reported numbers mean something and the synthetic-to-real gap stays measurable. + +One source was examined and used only with caution. A public labelled smell dataset carries a *none* +class that does not separate "not smelly" from "smelly but acceptable in context", which is exactly +the distinction the gate needs, so it is usable only after manual re-labelling of the relevant +slice. + ## Keeping the dataset honest Four decisions, each guarding against a specific way this kind of dataset goes wrong.