Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

They're rendering Unicode without any markup for language variant.


No, these are completely different, standalone code points, not variant forms of the same code point.

What's actually happening seems to be that the ch_tra model can recognize simplified too and output the corresponding traditional version if the character isn't in the traditional "alphabet"; it doesn't work so well in the other direction.

Example recognizing a partial screenshot of https://chinese.stackexchange.com/a/38707 (anyone can try this on Google Colab, no hardware required; remember to turn on GPU in Runtime -> Change runtime type):

  import easyocr
  import requests

  zhs_reader = easyocr.Reader(['en', 'ch_sim'])
  zht_reader = easyocr.Reader(['en', 'ch_tra'])
  image = requests.get('https://i.imgur.com/HtrpZCZ.png').content
  print('ch_sim:', ' '.join(text for _, text, _ in zhs_reader.readtext(image)))
  print('ch_tra:', ' '.join(text for _, text, _ in zht_reader.readtext(image)))
Results:

  ch_sim: One simplified character may mapping to multiple traditional ones: 皇后->皇后,後夭->后夭 豌鬟->头发,骏财->发财 As reversed, one traditional character may mapping to multiple simplified ones too: 乾燥->干燥, 乾隆->乾隆 嘹望->嘹望,嘹解->了解
  ch_tra: One simplified character may mapping to multiple traditional ones: 皇后->皇后,後天->后天 頭髮->頭發,發財->發財 As reversed, one traditional character may mapping to multiple simplified ones too: 乾燥->干燥, 乾隆->乾隆 瞭望->瞭望, 瞭解->了解
Compare to the original text:

  One simplified character may mapping to multiple traditional ones:

  - 皇后 -> 皇后,後天 -> 后天
  - 頭髮 -> 头发,發財 -> 发财

  As reversed, one traditional character may mapping to multiple simplified ones too:

  - 乾燥 -> 干燥,乾隆 -> 乾隆
  - 瞭望 -> 瞭望,瞭解 -> 了解
Of course, automatic character-to-character conversion from simplified to traditional can be wrong due to ambiguities; excellent examples from above: 头发 => 頭發 (should be 頭髮), 了解 => 了解 (should be 瞭解).


This approach seems a bit weird to me. While I appreciate them separating the models of Traditional and Simplified Chinese, I think I might prefer them to be combined (perhaps even including Japanese Kanji), and instead provide a way for the user to specify which language or regional variant is expected so characters matching the expected variant are simply given a higher score.


Without delving into implementation details, I suspect the ch_tra model was simply trained on a dataset including simplified images with traditional labels.


Funny thing, 夭 and 天 are not the same at all.

It doesn't seem to have a dictionary to do word level matching, only character level.


> Funny thing, 夭 and 天 are not the same at all.

Yes, the simplified model is not that great at recognizing simplified either, at least in this case.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: