Quirks in RPG Maker: Unicode and having fun with codepages

This is the second blog entry from our “Quirks in RPG Maker” series. In the previous article we introduced the non-Unicode application problem. Now let’s explain how this problem has been addressed.

The problem: Non-Unicode games

As explained in the previous article, Unicode is a standard for writing characters and texts using an universal encoding. Before existing an universal standard, some languages were using their own character encoding, sharing often the same bytes representing different glyphs. Games created with a non-Unicode editor contain non-Unicode texts. These games won’t be readable properly under operating systems with a different encoding than the original, unless some conversion between encodings is applied.

Encoding conversion issues

Modern operating systems are Unicode. This helps to convert non-Unicode encodings to Unicode, having enough room to fit any previous encoding to the new universal. However there are multiple issues when conversion standard was being designed.

Non-Unicode Windows Japanese Encoding is Shift JIS. This encoding was similar in the first 127 characters with ASCII (7-bit part), except the backslash (\) showing a yen symbol as a replacement. Originally, the command trigger for show messages is a ¥ followed by a letter, e.g. ¥c[1], in western is \c[1]. This was breaking message codes before. In fact, Japanese Windows shows a Yen sign (¥) as a directory separator even in modern Unicode Windows. In Korean Windows shows a Won sign (₩) instead. The wave dash from Shift JIS was incorrectly mapped to the Unicode full-width tilde, generating issues not only when showing the sign, the problem could affect games with filenames using this character.

The first approach: libiconv and Windows API

libiconv is a library to convert common legacy encodings to Unicode, supporting most non-Unicode Windows encodings. In order to play any non-Unicode game we needed to set the encoding manually before playing the game. There is an EasyRPG specific parameter in RPG_RT.ini to set the encoding by hand.

However, libiconv and Windows API were not supporting a way to detect the encoding. There were issues with yen, won and full-width tilde conversions, having issues when playing Japanese and Korean games.

Games don’t say which encoding are using

There are RPG Maker 2000/2003 games written in many encodings, but there is no a clear way to determine which encoding are using, as they can share the same bytes for different languages.

The solution: ICU

ICU (International Components for Unicode) is a library which brings very good support, including proper encoding conversion for Yen, Won and full-width tilde and an heuristic system for character encoding detection. With this feature the encoding can be autodetected and works fine for most games.

Broken game translations

There are games however, specifically game translations which are mistranslated, having mixed encodings and making the encoding autodetection fail. This is the case for some broken Yume Nikki translations. If you have file not found errors, you probably are using a broken translation. You can try to force manually the encoding in RPG_RT.ini for these particular cases, they may work better.

The ICU encoding heuristics detection gets some text data from the game. Currently uses some texts from the game database terms. Not all are passed because some default database translations from some editors are untranslated from Japanese and ICU would detect these games as Shift JIS when they probably are written using Windows codepage 1252 (western).

HTML5 Player port

Not interested in any details? Run TestGame in your browser

Some months ago we wrote a post for the April Fools’ Day called HTML5 port plans which was announcing a planned web based EasyRPG Player port.

Because of this date it could be a fake, however some details were really planned about making this real. Using the Emscripten compiler, Player C++ code can be converted to JavaScript code and run in modern web browsers with a faster computer.

This effort could be considered too early to be done, however it is pretty useful, for example, for checking the result online along with the Test Game for fast testing without needing to download Player, game and RTP if not installed in the current machine.

The new port will require a fast machine with a modern browser. Mozilla Firefox (newest version) is recommended, as it is faster than other browsers when running it. Processor requirements are currently high, a +2GHz multi core processor is suggested.

Saving works. The Esc key maps to “Disable fullscreen” in the browser in some cases, use the X key instead to toggle the menu. But even with these flaws it is still a nice proof of concept.

You are invited to try TestGame running in your browser, feel free to share your thoughts and feedback in post comments. Imagine when the JavaScript port becomes more matured you can run RPG maker games directly from your website, how cool is that?

EasyRPG Player 0.2.2 “Brian”

After some months, here is a new maintenance release of the RPG Maker 2000/2003 game interpreter. This release has been focused on bugfixes though it adds some new features.

What is new for users?

Hangul support for Korean games. Before this release, all games using Korean characters were unreadable. The Baekmuk Gulim font glyphs have been added to cover them (#336). Nobody of the team speaks Korean, however we want to support all games possible written in any language.

Gray Garden Korean translation
Korean translation of the Gray Garden game

There is also initial work to add missing Traditional Chinese glyphs, designed by Dmytro Kushnariov (#359)

Fixed major bug regressions introduced in previous releases:

  • Fixed shop’s invalid item bug and refresh sell window when opening it (#305)
  • Fixed sprite flashing (#300)

Fixed picture rotation (position and initial angle) (#310)

Implemented rest (fade out and fade in) in the inn (#313)

Fixed Battle related features:

  • Battle: Added fatigue trigger, fixed hp trigger and cleaned battle_calling to make battle command interpreter work (#326)
  • Battle: Fixed the acquisition of battle backgrounds (#304)

Maps and tile related:

  • Implemented Save (Allow/Forbid/As in parent) map property (#293)
  • Improved tile animation speed (#306)
  • Regenerate autotiles cache after a new chipset is setup (#313)

Messages:

  • Fixed message flags clearing (#313)
  • Show gold window during inn prompt, skip prompt if price is 0 (#313)
  • Implemented closing windows animation (#313)
  • Fixed face not cleared after the event execution ends (#340)
  • Fixed font color when changing game system graphics (#345)
Wadanohara and the Great Blue Sea
Fixed font color was affecting Wadanohara game

Multiple fixes and implementations in the event interpreter:

  • Improve animation speed (#235)
  • Fixed hero facing when triggered (#235)
  • Use the prelock direction for spinning animation like in RPGM (#235)
  • Use middle frame pattern when changing a character’s graphic (#235)
  • Update self movements when GetContinueEvents is true and reset move_route_repeated flag when a new move route is forced (#280)
  • Update interpreter after each autostarting event is set up (#280)
  • Stop event processing during transitions (#313)
  • Prevent destruction of parallel interpreters when map is refreshed and that interpreter is currently running (#325)
  • Fixed delayed deallocation when there are multiple parallel interpreters (#327)
  • Use the through-bool only for Through mode in movement events (#321)
  • Implemented transparent flag for Events (#302)
  • Implemented bush depth for character sprites and fix screen center (#326, #361)

Fixes to menu related issues:

  • Fixed segfault when item ID > max ID in Equip menu (#337)
  • Don’t play menu sound on cancel if menu access is not allowed (#289)
  • Always draw equipment stats (even when ’empty’ item is chosen) (#329)

Added Keypad 0 as Cancel button (#301)

Added MIDI support in OpenAL backend (#308)

Improved loading performance for some games by avoiding recursive folder mapping when not needed (#326)

Building: Patches for SDL and SDL_mixer were backported from Hg for Windows and Android to fix audio and wave issues (#353)

liblcf: Fixed various crashes when loading invalid data size blocks from some games (#120, #121)

Android: Added external gamepad device support.

What is new for developers?

Added code documentation generation option (Doxygen) to Autotools build (#323)

Quirks in RPG Maker: RTP and the translation hell

Welcome to our new series “Quirks in RPG Maker”. Here you will read about interesting issues we discovered while developing liblcf and EasyRPG Player.

RTP: “RunTime Package”

Almost all RPG Maker versions for PC bundles a shared game assets package, containing graphics, sounds and music. RTP packages are installed separately and they are downloadable for free from authorized distributor websites. The main motivation for this was intended to have a common asset library (graphics, music, sounds) shared between games to save some disk space in early distribution days (floppy disks, dial-up modems). Modern RPG Maker versions still use this shared way to provide game assets like a shared application library.

RPG Maker 2000 (and 2003) was published in Japan only. All RPG Maker 2000/3 RTP file names are in Japanese. This is OK, nowadays we can read Japanese and other languages across the web without issues. However, things were different around 15 years ago. RPG Maker 2000/2003 installers only work in Japanese Windows (or configured as Japanese), even today they are tricky to install. Though the license didn’t allow to modify the application, there have been unofficial translations of the product, requiring translating all file names to be able to install on non-Japanese Windows.

The problem: Non-unicode applications

Some years ago, there were lots of language specific character encodings around. Operating systems needed to map character symbols for different languages. E.g. Windows used the codepage 1252 for Latin script (western), 1251 for Cyrillic script (Russia, eastern Europe) an dozens more. These encodings shared same byte identification for specific characters, so documents needed to include some encoding identification to be read in systems with different languages and doing a conversion to be read.

Due to early years limitations, Unicode was created to have an universal character encoding for all symbols from all languages. This was awesome, but:
Year 2000. The year of the release of RPG Maker 2000. Popular Windows version that year: Windows 98. Unicode support: no.

Unicode was introduced in Windows NT, but the first user oriented Desktop version with Unicode support (based on Windows NT) was released later: Windows XP (2001).

Being RPG Maker 2000/2003 a non-unicode application supposed lots of problems as usual due to the character encoding legacy. Specific encoding quirks will be extended for the next “Quirks in RPG Maker” article.

RTP translations

Apart of the worldwide repercussion and facts about releasing program translations not needed to mention, the focus on the RTP translation deserves its own space.

The first popular RPG Maker 2000 RTP translation was made by Don Miguel. He translated all 465 filenames, some bugs and inconsistencies but nothing serious… well, he removed 3 files and added 38.

EasyRPG Player has a translation table feature. This allows to play games depending on Japanese filenames with an installed unofficial English RTP translation and vice-versa. If a game uses one of these 38 files added by Don Miguel and the system has the original Japanese RTP, weird things will happen. To prevent missing files, assets have been mapped in the table pointing to other files, but this is not the best solution.

Around 2002, RPG Maker 2003 was announced, after some advice about how bad are translations for the business. Don Miguel said he ceased translating RPG Maker. New, now major problems again.

Multiple RPG Maker 2003 RTP translations

After the RPG Maker 2000 translation release, RPG Maker became very popular around the world. With the RPG Maker 2003 announcement and RTP release, a couple of translators published different RTP translations, even before RPG Maker 2003 translation releases. This race of independent translators became a major mess. Then appeared multiple games depending on RTP assets but incompatible between RTPs. As a result, missing file errors appeared when trying to run them and became a pretty common issue.

EasyRPG Player needed to deal with it, by adding a multiple RTP translation table. Currently we have one translation for RPG Maker 2000 RTP and 3 translations for RPG Maker 2003 RTP. Now it is capable to solve this issue, as a value-added feature.

You can check the EasyRPG wiki for more technical details about RTP and some additional features like 2000 to 2003 migration and shared assets between versions, not mentioned here, but may be mentioned for other articles.

HTML5 port plans

So much time without updating the blog. Well, we have been pretty busy preparing the upcoming release of EasyRPG Player: 0.1.3.

Version 0.2 will be delayed because of narrower spare time from development team to complete the Battle System. However the next version feature list is yet relevant and will be announced in a not much later blog post.

But now, let’s introduce the yet another next-gen, cloud-enabled, big-data-capable, RPG-as-a-service and The-Internet-Of-Things-embeddable port: EasyRPG Player HTML5 Port (alternate name: EasyRPG.js).

Play your own RPG games in the cloud!

With the HTML5 port you will be able to play your games in any HTML5 compatible browser, share your game with your friends and conquer your neighborhood with your amazing RPG skills.

Let’s make it faster!

Increasing Player popularity with HTML5 will help with development, becoming attractive for Facebook, which could buy our product for a few billions and allowing us to have full time developers improving the Player engine even the Editor.

Run your RPGs everywhere!

The Internet Of Things makes Internet available in unexpected gadgets. With the HTML5 port you will be able to play RPG Maker 2000 and 2003 games in your microwave embedded browser while preparing popcorn!

We plan to release the HTML5 version some day of 2014, expect us!

Experimental build of EasyRPG Editor for Windows

EasyRPG Editor is being made using cross platform components, it could be run at least on GNU/Linux, Windows and Mac OS X. Now you can test the first version built for Windows.

This build is far from being complete, it’s just a platform preview to demonstrate it can be on systems different than GNU/Linux.

Download EasyRPG Editor for Windows

You can open sample_game.rproject from the sample_game folder and load maps into the editor. Most dialog windows are not finished yet. Paint tool (pen) only paints one step each click. Editor will crash when closing project or when selecting the root element of the map tree. These crashes has been fixed already but not in this build.

EasyRPG Editor screen shot on Windows 7

EasyRPG Editor screen shot on GNOME 3.6

Mid 2012 “we’re alive” report

Hi all. We apologize for the lack of updates last 6 months.

We’re busier than ever, even to write this. So, let’s see what’s new since the end of 2011:

General:

  • Moved upstream code to GitHub. [Falc]
  • Added Jenkins build automation system. [take-cheeze]
  • Made optional most CMake build requirements. [take-cheeze, fdelapena]
  • Added GNU build system, to (cross-)compile with ./configure and make. [fdelapena]
  • Wiki documentation updates, add README files. [fdelapena]

EasyRPG Readers:

  • Created python script to (re)generate reader structure with CSV files. [glynnc]

EasyRPG Player:

  • Emulation bug fixes. [Ghabry]
  • Automated nightly builds for PSP and Linux (32 bit). [take-cheeze]

EasyRPG Editor:

  • Initial painting tool. [Elektranox, Falc]
  • Code rewrite. [Falc]

EasyRPG TestGame:

  • Added more image and screen related operations. [elsemieni]
  • Added more tests and RPG_RT.exe stub. [Ghabry]

Things on the way (not in upstream master branch yet):

  • Player: Add debugger (F9 screen). [Ghabry]
  • Web: New end user website. [Falc, fdelapena]
  • Editor: Improve painting tool and major code rewrite. [Falc]
  • Player: Android port. [fdelapena]
  • TestGame: Make it RTP aware, separate 2000 and 2003 projects. [fdelapena]

Wanna help? Contact us via blog comments, forums, or IRC channel.

Editor map rendering

Map rendering is almost completed. The autotiles are not implemented yet, so at the moment they are drawn as normal tiles.

When switching between layers, some tiles are partially hidden in order to highlight those that are drawn on the selected layer. The event layer shows all the tiles and enables a grid.

The map can be displayed at different scales. Scrolling works correctly.

The map size limit is 500×500 tiles, as in RPG Maker. Huge maps do not need a lot of memory. The memory usage remains below 20 MB at the moment.

Planned downtime for server move (done)

While we’re preparing new project improvements, we will do a server move to a new provider.

Server move will provide own IP address for easy-rpg.org, so we will run under SSL for secure browsing for all services (blog, wiki, tracker, forums…) using HTTPS.

Connectivity will be gigabit, bringing faster navigation and downloads.

Downtime will be this week, our prevision is to complete server move before end of month.
Once done, fresh news will follow this November
.

What is new, 2011?

For End Users:
Support for Window_Message (the MessageBox) and related commands [ghabry]
Interpreting many more events [glynnc]
Add GameOver and Title Scene calling [niltonpd]
Enter Hero Name scene [glynnc]
Shop scene [glynnc]
Inn support [glynnc]
Fix Touch events and character drawing [ghabry]
Center Player window on startup under Windows [zhek]
Many free MIDI files replacing the copyrighted ones [Juan Manuel]

For Developers:
Add missing chunks to reader lib [glynnc]
Work on savegame reader started [glynnc]
Refactor Audio to make it SDL-independent [ghabry]
Added more tests to the testmap [ghabry, fdelapena]
More SDL refactoring [vgvgf]
Replace Scene handler with a Stack machine, allows more flexible scene switching [ghabry]
Editor: submitted code refactor of event commands (page 1/3) [fdelapena]

And fixed a bunch of important gameplay related bugs. Now games are playable. Player is gaining alpha status again. A lot more stable than old player version (before code refactor).