fix(ai,reader): robust sentence segmentation without abbreviation cutoffs & context-aware literary translation
- Implement robustSentenceSplit across reader.js and backend (miner.py, cefr.py) to prevent false cutoffs on honorifics (Mr., Mrs., Dr.), single-letter initials (John F. Kennedy), decimals, acronyms (U.S., U.K.), and abbreviations (No. 4, etc.) - Upgrade translation prompt to professional literary Russian, eliminating literal word-for-word translation and preserving titles/honorifics - Inject surrounding paragraph context into translation API to avoid out-of-context translations - Inject offline dictionary hints into LLM prompt for accurate contextual word meaning - Prefer qwen2.5:3b for higher translation fluency and fidelity - Purge truncated cache entries from sentence_translations and restore Card 30 (briefcase) with authentic full sentence - Bump Service Worker cache to notes-v6.0 - Add unit tests for robust sentence splitting, prompt formatting, and translation API
This commit is contained in:
+16
-1
@@ -386,7 +386,8 @@ def _migrate(conn: sqlite3.Connection):
|
||||
def repair_mismatched_cefr_contexts(conn: sqlite3.Connection):
|
||||
"""
|
||||
Repairs legacy vocabulary cards that were imported with placeholder blanks ('___')
|
||||
or with mismatched contexts from CEFR test questions.
|
||||
or with truncated/mismatched contexts from CEFR test questions and reader.
|
||||
Also purges truncated sentence cache entries.
|
||||
"""
|
||||
try:
|
||||
from app.cefr import resolve_context_sentence_for_term
|
||||
@@ -396,11 +397,25 @@ def repair_mismatched_cefr_contexts(conn: sqlite3.Connection):
|
||||
if not tbl:
|
||||
return
|
||||
|
||||
# Purge truncated sentence cache entries
|
||||
trans_tbl = conn.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name='sentence_translations'").fetchone()
|
||||
if trans_tbl:
|
||||
conn.execute(
|
||||
"""
|
||||
DELETE FROM sentence_translations
|
||||
WHERE original_text LIKE '%pecked Mrs.%'
|
||||
OR original_text LIKE '%the John F.'
|
||||
OR instr(original_text, '___') > 0
|
||||
"""
|
||||
)
|
||||
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT id, term, context_sentence
|
||||
FROM vocabulary_cards
|
||||
WHERE instr(context_sentence, '___') > 0
|
||||
OR context_sentence LIKE '%pecked Mrs.%'
|
||||
OR context_sentence LIKE '%the John F.'
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user