fix(ai,reader): robust sentence segmentation without abbreviation cutoffs & context-aware literary translation

- Implement robustSentenceSplit across reader.js and backend (miner.py, cefr.py) to prevent false cutoffs on honorifics (Mr., Mrs., Dr.), single-letter initials (John F. Kennedy), decimals, acronyms (U.S., U.K.), and abbreviations (No. 4, etc.)
- Upgrade translation prompt to professional literary Russian, eliminating literal word-for-word translation and preserving titles/honorifics
- Inject surrounding paragraph context into translation API to avoid out-of-context translations
- Inject offline dictionary hints into LLM prompt for accurate contextual word meaning
- Prefer qwen2.5:3b for higher translation fluency and fidelity
- Purge truncated cache entries from sentence_translations and restore Card 30 (briefcase) with authentic full sentence
- Bump Service Worker cache to notes-v6.0
- Add unit tests for robust sentence splitting, prompt formatting, and translation API
This commit is contained in:
2026-10-06 22:32:36 +03:00
parent 3098d3016d
commit 532e2ec3b2
11 changed files with 313 additions and 89 deletions
+16 -1
View File
@@ -386,7 +386,8 @@ def _migrate(conn: sqlite3.Connection):
def repair_mismatched_cefr_contexts(conn: sqlite3.Connection):
"""
Repairs legacy vocabulary cards that were imported with placeholder blanks ('___')
or with mismatched contexts from CEFR test questions.
or with truncated/mismatched contexts from CEFR test questions and reader.
Also purges truncated sentence cache entries.
"""
try:
from app.cefr import resolve_context_sentence_for_term
@@ -396,11 +397,25 @@ def repair_mismatched_cefr_contexts(conn: sqlite3.Connection):
if not tbl:
return
# Purge truncated sentence cache entries
trans_tbl = conn.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name='sentence_translations'").fetchone()
if trans_tbl:
conn.execute(
"""
DELETE FROM sentence_translations
WHERE original_text LIKE '%pecked Mrs.%'
OR original_text LIKE '%the John F.'
OR instr(original_text, '___') > 0
"""
)
rows = conn.execute(
"""
SELECT id, term, context_sentence
FROM vocabulary_cards
WHERE instr(context_sentence, '___') > 0
OR context_sentence LIKE '%pecked Mrs.%'
OR context_sentence LIKE '%the John F.'
"""
).fetchall()