Détecteur v1 de mentions de collaboration externe + table amendement_mentions#6
Closed
DavidNO wants to merge 6 commits into
Closed
Détecteur v1 de mentions de collaboration externe + table amendement_mentions#6DavidNO wants to merge 6 commits into
DavidNO wants to merge 6 commits into
Conversation
Exploratory detector (no pre-filter) that sends each amendment's exposé sommaire to an OpenAI-compatible API (Scaleway, OpenRouter...) and extracts declared collaborations with external entities. - Provider-agnostic via LLM_* env vars (base_url/model/key) - Random or deterministic sampling over the amendements table - Defensive JSON parsing (no reliance on json_object mode) - Rate-limit handling: throttle + exponential backoff on 429 - Writes raw results to analysis/output/ (gitignored), prints a frequency summary of the formulations encountered
Stores external-collaboration mentions detected in amendments' exposé sommaire. Analysis table (not in ETL_TABLES): survives rebuilds via create_all while amendements/dossiers are dropped and reloaded. amendementUid is a soft reference to amendements.uid (no ForeignKey), matching the RefUid pattern in the Amendement model, so the referential constraint never blocks the drop of the ETL-managed amendements table. One row per mention; provenance kept via modele/createdAt.
Add --persist flag to the detector: on top of the JSONL output it writes each mention to the amendement_mentions table via SQLAlchemy. Idempotent per amendment (delete-then-insert), commit per row so a partial run keeps what it found. The just recipe now forwards extra flags (--random/--persist).
- Add the missing setup steps (.env copy, starting Postgres via docker compose) - Document the external-collaboration mention detector: LLM_* env config, detect-mentions usage and flags, JSONL/DB outputs - Explain that amendement_mentions is an analysis table that survives rebuilds - Fix the stale Dossier model example (was still named User)
Collaborator
Author
|
Remplacée par #7 : approche regex retenue pour cette itération (déterministe, sans coût d'API). La branche detect-mentions-poc est conservée comme référence — elle a servi à calibrer les motifs et à construire le jeu d'évaluation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Objectif
Première brique d'une partie du POC : détecter les amendements déclarés « travaillés avec » / « inspirés par » une entité externe (lobby, syndicat, association, entreprise…).
Version v1 exploratoire, utilisant uniquement un LLM : chaque amendement est soumis à un LLM pour détecter ce type de déclaration.
Contenu
Détecteur (
analysis/detect_mentions.py)LLM_*(LLM_API_KEY/LLM_BASE_URL/LLM_MODEL), à créer localement…--random) sur tout le corpus.json_object( à changer si le modèle définitif le supporte).--delay) + backoff exponentiel sur 429.analysis/output/, gitignoré) + récap des formulations rencontrées.Table d'analyse (
models/amendement_mention.py→amendement_mentions)amendementUid,citation,formulation,entite,typeEntite,externe,modele,createdAt.ETL_TABLES) : survit aux rebuilds viacreate_all, pendant queamendements/dossierssont recréées.amendementUidest une référence molle (pas deForeignKey) versamendements.uid, sur le modèle desRefUidexistants : évite qu'une contrainte référentielle bloque le drop de la table ETL.Persistance (
--persist)amendement_mentionsen plus du JSONL.Formulations observées (run de 100 amendements aléatoires)
travaillé avec,travaillé en lien avec,en lien avec,en concertation avec,rédigé en collaboration avec,réalisé avec le concours de,issu d'une proposition de,reprend ainsi la demande de,inspirée de,proposé par…Entités réelles détectées : FNSEA, Coordination Rurale, CFDT Défense, SER, Amorce, APPS… Le flag
externedistingue les acteurs privés des organes publics (AID, AMIAD, DGA).Screenshot de la table amendement_mentions
Note
Branche empilée sur
add-amendements-model(#4) : à merger après celle-ci.