Develop your own parser

Introduction

For the last ten years, i have worked in different sides of the localization process from localization companies, big corporation localization groups, automation for localization for localizing windows applications and websites.

In this article, I will present you a localization tool I have created in order to localize online without anything more than your internet browser.

First, I will introduce iL10Nz (say Alliance), the architecture behind it, the different possibilities. Then in the second part, I will describe how to write customized parsers which gives the ability to localize all kind of file types. Finally, I will finish by developing the methods to read translation memory files for any projects.

Then, I will present what needs to be done.

iL10Nz presentation

iL10Nz is a localization tool that helps software, web sites, web application to be used in multiple languages. Localizable files can be created in many formats according to the application, technology used. In this matter, for any localization tool in the business, it is important to create parsers that understand what to translate, what is part of the code or what to avoid. This is a script to read the files and extract what should be changed, translated. iL10Nz is designed on the Front end using YUI library. The back end is using PHP5 to provide all the connectivity to the database and other third party application.

Writing customised parsers

I have written a certain number of parsers that could be helpful for people who are developing open source applications or web sites. At the time i write this article, so far i have created a parser to read po/pot files, php array of translation, js file related to extjs framework.

In the tool environment, a parser is a php include file (.inc) that contains the definition of a specific class object with specific methods that need to be implemented.

The class object is il10nz_Parser and the different methods to implement are : il10nz_Parser (constructor) , parsenow, rebuildnow.

If no customized parser has been uploaded into the projects, the default parser will used.

Here is an example of what needs to be implement

1. <?php

2. /**

3.  * il10nz_Parser.inc iL10Nz Parser class

4.  * 

5.  * This file describes the class used to manage the parser object

6.  * @author Olivier Cheneson <olivier@cheneson.com>

7.  * @version 1.0

8.  * @package il10nz

9.  */

10. class il10nz_Parser{

11. /**

12.  * Objects that represents other classes

13.  */

14. public $Project;

15. /**

16.  * Constructor

17.  */

18. public function il10nz_Parser($_projectobj)

19. {

20. $this->Project = $_projectobj;

21. }

22. /**

23.  *  Public function to parse and extract the localisable strings

24.  * 

25.  * @access Public

26.  * @return integer messagecode or errorcode

27.  */

28. public function parsenow($_filename,$_currenttime ){

29. return 2024;

30. }

31. /**

32.  *  Public function to rebuild the file

33.  * 

34.  * @access Public

35.  * @return integer

36.  */

37. public function rebuildnow($_fileobj,$_targetlang){

38. return 2031;

39. }

40. /**

41.  *  Public function to return the filename for download

42.  * 

43.  * @access Public

44.  * @return string

45.  */

46. public function downloadnow($_fileobj){

47. //location in the server

48. $basename = pathinfo($_fileobj->filename,);

     PATHINFO_BASENAME

49. return 2032;

50. }

51. /**

52.  *  Public function to import translation the file

53.  * 

54.  * @access Public

55.  * @return integer

56.  */

57. public function importtranslation($_fileobj,$_targetlang){

58. return 2125;

59. }

60. }

61. ?>

62. }

1. Methods

This methods is called during the extraction of the localizable strings when you import the file into the projects.

$_filename is the full path of the file to import. It is automatically determined by iL10Nz when you import the file.

$_currenttime is the time stamp used to identify the file that you have just imported in the tool.

So every time you import a file, this is a unique release number of a file in the project.

In the sequence of extraction of the string, you will write the code to read the file and extract the localizable

strings and import these strings into the iL10Nz database.

/*insert string for each languages defined*
$langlist = split(",", $this->Project->targetlanglist);
foreach($langlist as $key => $lgcode){
$sql = "INSERT INTO `il10nz_proj_".$this->Project->projectname."` (id, filename, path,
internfileversion,
 sourcelang, targetlang, source, target, sourcecomment, targetcomment, projectname,
projectversion, createdby,
creationdatetime, lastupdatedby, lastupdatedatetime, locktime, lockedby, translated,
toproofread, user1, user2, user3, user4, user5)";
$sql = $sql  . " VALUES( ";
$sql = $sql  . " x'".$id."', '".pathinfo($_filename, PATHINFO_BASENAME)."',";
$sql = $sql  . " '".$this->Project->importpath."',";
$sql = $sql  .$_currenttime.",";
$sql = $sql  . " '".$this->Project->sourcelang."',";
$sql = $sql  . " '".$lgcode."',";
$sql = $sql  . " x'".$source."', x'".$source."',";
$sql = $sql  . " '',";
$sql = $sql  . " '',";
$sql = $sql  . " '".$this->Project->projectname."',";
$sql = $sql  . " '".$this->Project->projectversion."',";
$sql = $sql  . " '".$this->Project->User->Getusername()."',";
$sql = $sql  . " now(),";
$sql = $sql  . " '".$this->Project->User->Getusername()."',";
$sql = $sql  . " now(),";
$sql = $sql  . " now(),";
$sql = $sql  . " '',";
$sql = $sql  . " '0',";
$sql = $sql  . " '0',";
$sql = $sql  . " '',";
$sql = $sql  . " '',";
$sql = $sql  . " '',";
$sql = $sql  . " '',";
$sql = $sql  . " '')";

Here is the code in the default parser where the localizable strings are inserted into the database

Here is the list of columns in the database that you need to know in order to understand the insertion of the localizable strings:

• po/pot files : In iL10Nz, you are able to import a po/pot file (.po, .pot).

• The po file format is used in the open source industry to internationalize

•  sofware or web application that were generated by gettext library.

#: basidesh.src#RID_STR_FILTER_ALLFILES.string.text
msgid "<All>"
msgstr ""
#: basidesh.src#RID_STR_NOMODULE.string.text
msgid "< No Module >"
msgstr ""
#: basidesh.src#RID_STR_WRONGPASSWORD.string.text
msgid "Incorrect Password"
msgstr ""
#: basidesh.src#RID_STR_OPEN.string.text
msgid "Load"
msgstr ""
#: basidesh.src#RID_STR_SAVE.string.text
msgid "Save"msgstr ""

• PHP Array: Most of the web application will create a php file

• containing a php associated array with the id and localized value

$lang = array();
$lang["title"] = 'AJAX Chat';
$lang['userName'] = 'Username';
$lang['password'] = 'Password';
$lang['login'] = 'Login';
$lang['logout'] = 'Logout';
$lang['channel'] = 'Channel';
$lang['style'] = 'Style';
$lang['language'] = 'Language';

Working with Translation Memory

In iL10Nz environnement, it is possible to import a TM file in TMX (Translation Memory eXchange) format such as the following example.

<?xml version="1.0" encoding="UTF-8"?>
<tmx version="1.3">
<header segtype="sentence" o-tmf="sqlserver" datatype="database" adminlang="en" srclang="en">
<prop type="x-source">en</prop>
<prop type="x-target">fr-fr</prop>
</header>
<body>
<tu tuid="stringid" changeid="" segtype="dbstr">
<tuv xml:lang="en">
<seg>add the string</seg>
</tuv>
<tuv xml:lang="fr">
<seg>ajouter ma chaine de caractère</seg>
</tuv>
<tuv xml:lang="de">
<seg> ly string in german</seg>
</tuv>
</tu>

After uploading the TM File,the resource will be displayed when clicking on CheckTM Button in the Suggestion panel