KakaruHayate 3e8c071e9a
优化日英/中英混合推理分词逻辑
示例

【1pass】(现在版本)
['我是一名粉刷匠,粉刷本领强,', 'I am a painter with strong painting skills', ',', 'A.B.C.D.A', '_', 'B', '—', 'C', '……', 'D.']
['zh', 'en', 'zh', 'en', 'zh', 'en', 'zh', 'en', 'zh', 'en']
【2pass】
['我是一名粉刷匠,粉刷本领强,', 'I am a painter with strong painting skills,', 'A.B.C.D.A_', 'B—', 'C……', 'D.']
['zh', 'en', 'en', 'en', 'en', 'en']
【result】
['我是一名粉刷匠,粉刷本领强,', 'I am a painter with strong painting skills,A.B.C.D.A_B—C……D.']
['zh', 'en']
2024-01-30 22:37:59 +08:00
..
2024-01-28 19:19:57 +08:00
2024-01-16 17:14:18 +01:00
2024-01-16 17:14:18 +01:00
2024-01-25 02:30:37 +08:00
2024-01-26 01:40:18 +08:00
2024-01-30 11:49:41 +08:00
2024-01-29 23:25:08 +08:00
2024-01-16 17:38:48 +08:00
2024-01-25 02:30:08 +08:00
2024-01-28 19:34:03 +08:00
2024-01-24 19:37:47 +08:00
2024-01-23 18:52:03 +08:00