Quirks in RPG Maker: Unicode and having fun with codepages

This is the second blog entry from our “Quirks in RPG Maker” series. In the previous article we introduced the non-Unicode application problem. Now let’s explain how this problem has been addressed.

The problem: Non-Unicode games

As explained in the previous article, Unicode is a standard for writing characters and texts using an universal encoding. Before existing an universal standard, some languages were using their own character encoding, sharing often the same bytes representing different glyphs. Games created with a non-Unicode editor contain non-Unicode texts. These games won’t be readable properly under operating systems with a different encoding than the original, unless some conversion between encodings is applied.

Encoding conversion issues

Modern operating systems are Unicode. This helps to convert non-Unicode encodings to Unicode, having enough room to fit any previous encoding to the new universal. However there are multiple issues when conversion standard was being designed.

Non-Unicode Windows Japanese Encoding is Shift JIS. This encoding was similar in the first 127 characters with ASCII (7-bit part), except the backslash (\) showing a yen symbol as a replacement. Originally, the command trigger for show messages is a ¥ followed by a letter, e.g. ¥c[1], in western is \c[1]. This was breaking message codes before. In fact, Japanese Windows shows a Yen sign (¥) as a directory separator even in modern Unicode Windows. In Korean Windows shows a Won sign (₩) instead. The wave dash from Shift JIS was incorrectly mapped to the Unicode full-width tilde, generating issues not only when showing the sign, the problem could affect games with filenames using this character.

The first approach: libiconv and Windows API

libiconv is a library to convert common legacy encodings to Unicode, supporting most non-Unicode Windows encodings. In order to play any non-Unicode game we needed to set the encoding manually before playing the game. There is an EasyRPG specific parameter in RPG_RT.ini to set the encoding by hand.

However, libiconv and Windows API were not supporting a way to detect the encoding. There were issues with yen, won and full-width tilde conversions, having issues when playing Japanese and Korean games.

Games don’t say which encoding are using

There are RPG Maker 2000/2003 games written in many encodings, but there is no a clear way to determine which encoding are using, as they can share the same bytes for different languages.

The solution: ICU

ICU (International Components for Unicode) is a library which brings very good support, including proper encoding conversion for Yen, Won and full-width tilde and an heuristic system for character encoding detection. With this feature the encoding can be autodetected and works fine for most games.

Broken game translations

There are games however, specifically game translations which are mistranslated, having mixed encodings and making the encoding autodetection fail. This is the case for some broken Yume Nikki translations. If you have file not found errors, you probably are using a broken translation. You can try to force manually the encoding in RPG_RT.ini for these particular cases, they may work better.

The ICU encoding heuristics detection gets some text data from the game. Currently uses some texts from the game database terms. Not all are passed because some default database translations from some editors are untranslated from Japanese and ICU would detect these games as Shift JIS when they probably are written using Windows codepage 1252 (western).

Quirks in RPG Maker: RTP and the translation hell

Welcome to our new series “Quirks in RPG Maker”. Here you will read about interesting issues we discovered while developing liblcf and EasyRPG Player.

RTP: “RunTime Package”

Almost all RPG Maker versions for PC bundles a shared game assets package, containing graphics, sounds and music. RTP packages are installed separately and they are downloadable for free from authorized distributor websites. The main motivation for this was intended to have a common asset library (graphics, music, sounds) shared between games to save some disk space in early distribution days (floppy disks, dial-up modems). Modern RPG Maker versions still use this shared way to provide game assets like a shared application library.

RPG Maker 2000 (and 2003) was published in Japan only. All RPG Maker 2000/3 RTP file names are in Japanese. This is OK, nowadays we can read Japanese and other languages across the web without issues. However, things were different around 15 years ago. RPG Maker 2000/2003 installers only work in Japanese Windows (or configured as Japanese), even today they are tricky to install. Though the license didn’t allow to modify the application, there have been unofficial translations of the product, requiring translating all file names to be able to install on non-Japanese Windows.

The problem: Non-unicode applications

Some years ago, there were lots of language specific character encodings around. Operating systems needed to map character symbols for different languages. E.g. Windows used the codepage 1252 for Latin script (western), 1251 for Cyrillic script (Russia, eastern Europe) an dozens more. These encodings shared same byte identification for specific characters, so documents needed to include some encoding identification to be read in systems with different languages and doing a conversion to be read.

Due to early years limitations, Unicode was created to have an universal character encoding for all symbols from all languages. This was awesome, but:
Year 2000. The year of the release of RPG Maker 2000. Popular Windows version that year: Windows 98. Unicode support: no.

Unicode was introduced in Windows NT, but the first user oriented Desktop version with Unicode support (based on Windows NT) was released later: Windows XP (2001).

Being RPG Maker 2000/2003 a non-unicode application supposed lots of problems as usual due to the character encoding legacy. Specific encoding quirks will be extended for the next “Quirks in RPG Maker” article.

RTP translations

Apart of the worldwide repercussion and facts about releasing program translations not needed to mention, the focus on the RTP translation deserves its own space.

The first popular RPG Maker 2000 RTP translation was made by Don Miguel. He translated all 465 filenames, some bugs and inconsistencies but nothing serious… well, he removed 3 files and added 38.

EasyRPG Player has a translation table feature. This allows to play games depending on Japanese filenames with an installed unofficial English RTP translation and vice-versa. If a game uses one of these 38 files added by Don Miguel and the system has the original Japanese RTP, weird things will happen. To prevent missing files, assets have been mapped in the table pointing to other files, but this is not the best solution.

Around 2002, RPG Maker 2003 was announced, after some advice about how bad are translations for the business. Don Miguel said he ceased translating RPG Maker. New, now major problems again.

Multiple RPG Maker 2003 RTP translations

After the RPG Maker 2000 translation release, RPG Maker became very popular around the world. With the RPG Maker 2003 announcement and RTP release, a couple of translators published different RTP translations, even before RPG Maker 2003 translation releases. This race of independent translators became a major mess. Then appeared multiple games depending on RTP assets but incompatible between RTPs. As a result, missing file errors appeared when trying to run them and became a pretty common issue.

EasyRPG Player needed to deal with it, by adding a multiple RTP translation table. Currently we have one translation for RPG Maker 2000 RTP and 3 translations for RPG Maker 2003 RTP. Now it is capable to solve this issue, as a value-added feature.

You can check the EasyRPG wiki for more technical details about RTP and some additional features like 2000 to 2003 migration and shared assets between versions, not mentioned here, but may be mentioned for other articles.