Quirks in RPG Maker: Unicode and having fun with codepages

This is the second blog entry from our “Quirks in RPG Maker” series. In the previous article we introduced the non-Unicode application problem. Now let’s explain how this problem has been addressed.

The problem: Non-Unicode games

As explained in the previous article, Unicode is a standard for writing characters and texts using an universal encoding. Before existing an universal standard, some languages were using their own character encoding, sharing often the same bytes representing different glyphs. Games created with a non-Unicode editor contain non-Unicode texts. These games won’t be readable properly under operating systems with a different encoding than the original, unless some conversion between encodings is applied.

Encoding conversion issues

Modern operating systems are Unicode. This helps to convert non-Unicode encodings to Unicode, having enough room to fit any previous encoding to the new universal. However there are multiple issues when conversion standard was being designed.

Non-Unicode Windows Japanese Encoding is Shift JIS. This encoding was similar in the first 127 characters with ASCII (7-bit part), except the backslash (\) showing a yen symbol as a replacement. Originally, the command trigger for show messages is a ¥ followed by a letter, e.g. ¥c[1], in western is \c[1]. This was breaking message codes before. In fact, Japanese Windows shows a Yen sign (¥) as a directory separator even in modern Unicode Windows. In Korean Windows shows a Won sign (₩) instead. The wave dash from Shift JIS was incorrectly mapped to the Unicode full-width tilde, generating issues not only when showing the sign, the problem could affect games with filenames using this character.

The first approach: libiconv and Windows API

libiconv is a library to convert common legacy encodings to Unicode, supporting most non-Unicode Windows encodings. In order to play any non-Unicode game we needed to set the encoding manually before playing the game. There is an EasyRPG specific parameter in RPG_RT.ini to set the encoding by hand.

However, libiconv and Windows API were not supporting a way to detect the encoding. There were issues with yen, won and full-width tilde conversions, having issues when playing Japanese and Korean games.

Games don’t say which encoding are using

There are RPG Maker 2000/2003 games written in many encodings, but there is no a clear way to determine which encoding are using, as they can share the same bytes for different languages.

The solution: ICU

ICU (International Components for Unicode) is a library which brings very good support, including proper encoding conversion for Yen, Won and full-width tilde and an heuristic system for character encoding detection. With this feature the encoding can be autodetected and works fine for most games.

Broken game translations

There are games however, specifically game translations which are mistranslated, having mixed encodings and making the encoding autodetection fail. This is the case for some broken Yume Nikki translations. If you have file not found errors, you probably are using a broken translation. You can try to force manually the encoding in RPG_RT.ini for these particular cases, they may work better.

The ICU encoding heuristics detection gets some text data from the game. Currently uses some texts from the game database terms. Not all are passed because some default database translations from some editors are untranslated from Japanese and ICU would detect these games as Shift JIS when they probably are written using Windows codepage 1252 (western).

HTML5 Player port

Not interested in any details? Run TestGame in your browser

Some months ago we wrote a post for the April Fools’ Day called HTML5 port plans which was announcing a planned web based EasyRPG Player port.

Because of this date it could be a fake, however some details were really planned about making this real. Using the Emscripten compiler, Player C++ code can be converted to JavaScript code and run in modern web browsers with a faster computer.

This effort could be considered too early to be done, however it is pretty useful, for example, for checking the result online along with the Test Game for fast testing without needing to download Player, game and RTP if not installed in the current machine.

The new port will require a fast machine with a modern browser. Mozilla Firefox (newest version) is recommended, as it is faster than other browsers when running it. Processor requirements are currently high, a +2GHz multi core processor is suggested.

Saving works. The Esc key maps to “Disable fullscreen” in the browser in some cases, use the X key instead to toggle the menu. But even with these flaws it is still a nice proof of concept.

You are invited to try TestGame running in your browser, feel free to share your thoughts and feedback in post comments. Imagine when the JavaScript port becomes more matured you can run RPG maker games directly from your website, how cool is that?

EasyRPG Player 0.2.2 “Brian”

After some months, here is a new maintenance release of the RPG Maker 2000/2003 game interpreter. This release has been focused on bugfixes though it adds some new features.

What is new for users?

Hangul support for Korean games. Before this release, all games using Korean characters were unreadable. The Baekmuk Gulim font glyphs have been added to cover them (#336). Nobody of the team speaks Korean, however we want to support all games possible written in any language.

Gray Garden Korean translation
Korean translation of the Gray Garden game

There is also initial work to add missing Traditional Chinese glyphs, designed by Dmytro Kushnariov (#359)

Fixed major bug regressions introduced in previous releases:

  • Fixed shop’s invalid item bug and refresh sell window when opening it (#305)
  • Fixed sprite flashing (#300)

Fixed picture rotation (position and initial angle) (#310)

Implemented rest (fade out and fade in) in the inn (#313)

Fixed Battle related features:

  • Battle: Added fatigue trigger, fixed hp trigger and cleaned battle_calling to make battle command interpreter work (#326)
  • Battle: Fixed the acquisition of battle backgrounds (#304)

Maps and tile related:

  • Implemented Save (Allow/Forbid/As in parent) map property (#293)
  • Improved tile animation speed (#306)
  • Regenerate autotiles cache after a new chipset is setup (#313)

Messages:

  • Fixed message flags clearing (#313)
  • Show gold window during inn prompt, skip prompt if price is 0 (#313)
  • Implemented closing windows animation (#313)
  • Fixed face not cleared after the event execution ends (#340)
  • Fixed font color when changing game system graphics (#345)
Wadanohara and the Great Blue Sea
Fixed font color was affecting Wadanohara game

Multiple fixes and implementations in the event interpreter:

  • Improve animation speed (#235)
  • Fixed hero facing when triggered (#235)
  • Use the prelock direction for spinning animation like in RPGM (#235)
  • Use middle frame pattern when changing a character’s graphic (#235)
  • Update self movements when GetContinueEvents is true and reset move_route_repeated flag when a new move route is forced (#280)
  • Update interpreter after each autostarting event is set up (#280)
  • Stop event processing during transitions (#313)
  • Prevent destruction of parallel interpreters when map is refreshed and that interpreter is currently running (#325)
  • Fixed delayed deallocation when there are multiple parallel interpreters (#327)
  • Use the through-bool only for Through mode in movement events (#321)
  • Implemented transparent flag for Events (#302)
  • Implemented bush depth for character sprites and fix screen center (#326, #361)

Fixes to menu related issues:

  • Fixed segfault when item ID > max ID in Equip menu (#337)
  • Don’t play menu sound on cancel if menu access is not allowed (#289)
  • Always draw equipment stats (even when ’empty’ item is chosen) (#329)

Added Keypad 0 as Cancel button (#301)

Added MIDI support in OpenAL backend (#308)

Improved loading performance for some games by avoiding recursive folder mapping when not needed (#326)

Building: Patches for SDL and SDL_mixer were backported from Hg for Windows and Android to fix audio and wave issues (#353)

liblcf: Fixed various crashes when loading invalid data size blocks from some games (#120, #121)

Android: Added external gamepad device support.

What is new for developers?

Added code documentation generation option (Doxygen) to Autotools build (#323)