{"id":54090,"date":"2021-06-18T10:13:25","date_gmt":"2021-06-18T01:13:25","guid":{"rendered":"https:\/\/learningbox.online\/?p=54090"},"modified":"2021-06-21T08:57:56","modified_gmt":"2021-06-20T23:57:56","slug":"blog_csv-bom-utf8","status":"publish","type":"post","link":"https:\/\/learningbox.online\/en\/column\/blog_csv-bom-utf8\/","title":{"rendered":"CSV and Character Encoding - Avoid Character Garbling with utf8 with BOM"},"content":{"rendered":"

I am the representative, Nishimura! It's been a while since I wrote a blog.
To avoid garbled text in Excel, I added a BOM.<\/p>\n\n\n\n

<\/div>\n\n\n\n

learningBOX is an LMS with multilingual support<\/h2>\n\n\n\n

Our e-learning system, learningBOX, is a multilingual LMS. learningBOX's UI is currently only available in Japanese and English, but the data of teaching materials and learners' answers are flexible enough to support not only Japanese and English, but also Chinese, Korean, Vietnamese, and other languages around the world. However, the data of teaching materials and learners' answers can be flexibly supported not only in Japanese and English but also in Chinese, Korean, Vietnamese and other languages around the world.
In fact, teaching materials created in each country's language are used in training programs for foreigners.<\/p>\n\n\n

\n
\"Multilingual<\/figure>\n<\/div>\n\n\n

Multilingual support in CSV was incomplete.<\/h3>\n\n\n\n

Despite the claim of multilingual support, there was a deficiency in the CSV support. In the currently released version (2.14.28), CSV encoding is fixed to Shift_JIS (Windows-31J).
Therefore, all characters except Japanese, English, and some Chinese and Latin characters will be garbled.<\/p>\n\n\n

\n
\"CSV<\/figure>\n<\/div>\n\n\nBack to Contents<\/a>\n\n\n\n

implemented to be able to choose UTF-8<\/h2>\n\n\n
\n
\"Garbled<\/figure>\n<\/div>\n\n\n

There was a proposal to switch the character encoding of CSV output from learningBOX to UTF-8, but we decided that the impact on users would be too great if we suddenly changed the specification, so we made it possible to choose between UTF-8 and Shift_JIS for the character encoding of CSV. (Default is Shift_JIS)<\/p>\n\n\n\nBack to Contents<\/a>\n\n\n\n

BOM is added to prevent garbled characters in Excel<\/h2>\n\n\n\n

When I open a UTF-8 CSV without BOM in Excel, the characters are garbled. BOM stands for \"Byte Order Mark\" and indicates that the character code of the file is \"UTF-8\".<\/p>\n\n\n\n

Therefore, by using UTF-8 with BOM, you can now open the file in Excel without garbling the characters.<\/p>\n\n\n\n

A long time ago, it was common to avoid garbled characters by making it in UTF-16LE, but relatively recent Excel is now better utf8 with BOM. In addition, the character code of the Web world is becoming unified into utf8, considering the ease of handling outside of Excel, I decided to utf8 instead of UTF-16LE.<\/p>\n\n\n\n

Reference sites:.How to output Unicode csv that opens correctly in both Win and Mac Excel<\/a><\/p>\n\n\n\nBack to Contents<\/a>\n\n\n\n

How to add a BOM in PHP<\/h2>\n\n\n\n

The entity of the BOM is 3 bytes of data and is expressed as \"\\xEF\\xBB\\xBF\\xBF\". In the case of PHP, the BOM can be added by doing the following<\/p>\n\n\n\n

\n

$csv = \"\\xEF\\xBB\\xBF\".$csv;<\/strong><\/code><\/p>\n<\/div>\n\n\n\nBack to Contents<\/a>\n\n\n\n

September<\/h2>\n\n\n\n

Depending on the results of the verification, it is expected to be released in an update next week or the week after.
For more information on the status of the learningBOX release, please click here.Release Note<\/a> <\/p>\n\n\n\nBack to Contents<\/a>\n\n\n\n

We are looking for a learningBOX developer!<\/h2>\n\n\n\n

We are looking for learningBOX developers. We have various openings for backend engineers, frontend engineers, quality assurance engineers, and project managers.
We are looking for development engineers mainly at our headquarters in Tatsu, but we are also actively recruiting infrastructure engineers in Tokyo, so please apply if you are interested, However, if you have experience in Linux operation or have studied Linux or networking at an infrastructure school, you may be welcome!
For more information about employment opportunities at learningBOX, Inc, here<\/a> <\/p>\n\n\n\nBack to Contents<\/a>","protected":false},"excerpt":{"rendered":"This is Mr. Nishimura, Representative Director! It's been a while since I've written a blog, but I've added a BOM to prevent garbled characters in Excel. LearningBOX is an LMS that supports multiple languages...","protected":false},"author":5,"featured_media":78898,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"template-column.php","format":"standard","meta":{"_acf_changed":false,"content-type":"","_lmt_disableupdate":"yes","_lmt_disable":"","advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"_vk_print_noindex":"","_veu_custom_css":"","_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[41],"tags":[269,272,271,178],"class_list":["post-54090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learningbox","tag-csv","tag-elearning-","tag-shift_jis","tag-178"],"acf":[],"modified_by":"i.hirota","jetpack_featured_media_url":"https:\/\/learningbox.online\/wp-content\/uploads\/2020\/07\/english_main.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pgWaOl-e4q","_links":{"self":[{"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/posts\/54090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/comments?post=54090"}],"version-history":[{"count":84,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/posts\/54090\/revisions"}],"predecessor-version":[{"id":119755,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/posts\/54090\/revisions\/119755"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/media\/78898"}],"wp:attachment":[{"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/media?parent=54090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/categories?post=54090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learningbox.online\/en\/wp-json\/wp\/v2\/tags?post=54090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}