XarTM - Xaraya/Xarigami Translation Memory

Installation, upgrade and data cleanup

Notes about importing a locale
Updating to newer Xaraya releases
Obsolete files (xartm_errors.txt)
Data cleanup
Upgrade from previous versions

Installation

The program requires PHP and MySQL. Apache 2 and IIS 6 webservers have been tested. Note, that some of the XarTM files are for Web usage, while some others are for command line usage! Command line tools started from Web will display an error message.

1. Copy all files and directories to your webserver's area: bash> mkdir /var/www/xartm
bash> cp translationmemory-1.0/* /var/www/xartm/
2. Edit the settings in config.inc.php: Settings are documented in the file itself.
3. Make your configured "cache" dir writable by the webserver: bash> mkdir /var/www/xartm/cache
bash> chgrp apache /var/www/xartm/cache
bash> chmod g+w /var/www/xartm/cache
4. Set your ADOdb library's path in lib/Common.php: include_once('/usr/share/php/adodb/adodb-exceptions.inc.php');
include_once('/usr/share/php/adodb/adodb.inc.php');
5. Add your locale to db/database.sql's end if not yet there: mysql> INSERT INTO Locales (Name) VALUES ("nl_NL.utf-8");
mysql> INSERT INTO Locales (Name) VALUES ("ph_PH.utf-8");
6. Create an empty database and populate it from db/database.sql: bash> mysqladmin -p create xartm
bash> mysql -p xartm <db/database.sql
7. Install your current translations in the latest Xaraya: Make sure you use the most complete translation pack of your language
8. Generate ALL SKELETONS using the Xaraya Translations module: This is the same as you usually did when using the original module
9. Import the generated skels to XarTM (displays help if no arguments given): bash> php import.php Xaraya hu_HU.utf-8

Notes about importing a locale

NEVER STOP the import process. Starting the import process will mark all your strings as "dead" (unused, outdated). As the import process goes, it restores strings one by one to "useful" state. If you stop the process for some reason, you will have to start and complete it before using XarTM.

Generating the skels is very important, even if you don't have existing translations, because this program imports and works with those skeleton files only. The translation work in XarTM is always stored in the database, and you must export it to get your translated Xaraya locale files!

You can reduce the language pack and database size, if you set generate references to 0 (no references) in the configuration of the Xaraya Translations module, before generating skels.

If you have older backups of your Xaraya locale files, it is a good idea to import some of them first. Import your latest translations after the old backups, and last, import the current Xaraya release's all generated skels, if those also contain your latest translations. This way you can have all older translations of the strings in the translation memory.

Example

Import old locale files backup, let's say you have a file from 04/14/2005.
Import another old backup, you found another zip from 01/25/2006.
Import your current file from Xaraya language downloads.
Install that package in the new Xaraya release, generate all skels, and import those at the end.

Step 3 will make sure that strings lost at skel generation (4) will be available for reuse.

Importing takes long time, especially the first one! 0.5 to 1.5 hours or more.

Updating to newer Xaraya/Xarigami releases

Brief

Export your work from XarTM and install it in new Xaraya
Go to Xaraya's Translations module and generate all skels again
Run the import process again in XarTM

If you import newer skeletons, e.g. when there is a new Xaraya version and you generated all skels again over an exported language pack, the database will keep most of the old strings in "translation memory". (You may even import updated/edited translations.) New imported strings will overwrite the same English strings and their translation, the rest will be marked as dead string, useful for reuse only.

NEVER IMPORT old skeletons after you made translations in XarTM, because you will lose your work! You should import only if you exported the pack, installed it to your Xaraya and generated the skeletons over those exported files!

Obsolete files (xartm_errors.txt)

This assumes that you run your export tasks directly into a Xaraya installation's locale folder. Which is a good idea anyway to immediately test your work.

Xaraya's original Translations module does not delete obsolete files on skel generation. However, XarTM from version 1.1 tries to identify these files (similar to the original module) and will generate a list for you on every export, into the xartm_errors.txt file in your locale's directory.

You must delete the obsolete files manually otherwsie you will be translating unused strings. When you delete an unused file just before your next import process, the importer can mark the strings in deleted files as dead strings.

XarTM export is not able to delete these unnecessary files, because it would not recognize, that you did not install a module, block or theme (eg. xe1 theme!) in your local Xaraya copy, or the file is really gone from a module. Thus the xartm_errors.txt will also list ALL files of the modules what you currently do not have installed in your export Xaraya directory. Do not delete the files of modules, blocks and themes what you did not install in that Xaraya directory if you intend to keep their translation.

There is no perfect stage in the current processes to do this deletion of obsolete files. The best is doing this as a separate task, independent of updating to new releases.

Export translations to a locale folder of the Xaraya version you are working on
Read var/locales/<locale>/xartm_errors.txt, identify truly obsolete files and delete them
Import the whole package again

Cleaning obsolete data

Besides as explained in the previous section, there are other steps you can do to clean up old data.

I think this select would list strings, that don't have any translation but are 'dead' already. So those could be safely deleted on SQL level. NEEDS TESTING, sorry

SELECT Strings.ID, SUBSTRING(Original, 1, 50), p1.Name, p2.Name RootName 
	FROM Projects p1 
	INNER JOIN Projects p2 ON p1.RootID = p2.ID 
	INNER JOIN Strings ON p1.ID = Strings.ProjectID 
	WHERE Strings.Translation = '' AND p2.LastSeen > Strings.LastSeen;

With temp table to avoid mysql subquery reference problem:

DELETE FROM Strings WHERE ID IN 
	(SELECT tempP.tempID FROM 
		(SELECT Strings.ID as tempID 
			FROM Projects p1 
			INNER JOIN Projects p2 ON p1.RootID = p2.ID
			INNER JOIN Strings ON p1.ID = Strings.ProjectID
			WHERE Strings.Translation = '' AND p2.LastSeen > Strings.LastSeen) 
		as tempP);

I think the next select returns those strings that are 'dead' and there is an identical original, identically translated and not dead. Also safe to delete. NEEDS TESTING, sorry

SELECT s1.ID, SUBSTRING(s1.Original, 1, 50) OrigPart, p1.Name, p2.Name RootName 
	FROM Projects p1 
	INNER JOIN Projects p2 ON p1.RootID = p2.ID 
	INNER JOIN Strings s1 ON p1.ID = s1.ProjectID 
	INNER JOIN Strings s2 ON s1.Original = s2.Original AND s1.Translation = s2.Translation AND s1.ID != s2.ID AND s2.LastSeen > p2.LastSeen 
	WHERE s1.Translation != '' AND p2.LastSeen > s1.LastSeen
	GROUP BY s1.ID, OrigPart, p1.Name, p2.Name;

With temp table to avoid mysql subquery reference problem:

DELETE FROM Strings WHERE ID IN 
	(SELECT tempP.tempID FROM 
		(SELECT s1.ID tempID 
			FROM Projects p1 
			INNER JOIN Projects p2 ON p1.RootID = p2.ID 
			INNER JOIN Strings s1 ON p1.ID = s1.ProjectID 
			INNER JOIN Strings s2 ON s1.Original = s2.Original AND s1.Translation = s2.Translation AND s1.ID != s2.ID AND s2.LastSeen > p2.LastSeen 
			WHERE s1.Translation != '' AND p2.LastSeen > s1.LastSeen 
			GROUP BY s1.ID)
		as tempP);

Upgrade from previous versions

Upgrade from any older 1.x to 1.5 for Xarigami support

If you have a database loaded with Xaraya 1.x translations, it's a good start to work on Xarigami translation. If you don't, it's recommended to get a Xaraya and import that. Unfortunately you need a fully functional Xaraya site for that to generate skels.

There is no automatic upgrade

1. Add the new columns

ALTER TABLE Projects ADD COLUMN RootID int NULL DEFAULT NULL;
ALTER TABLE Projects ADD FOREIGN KEY (RootID) References Projects (ID);
ALTER TABLE Strings ADD COLUMN LastModified datetime;

CREATE TRIGGER string_update BEFORE UPDATE ON Strings FOR EACH ROW
BEGIN
IF NEW.Translation != OLD.Translation THEN
SET NEW.LastModified=CURRENT_TIMESTAMP;
END IF;
END;

2. List your roots, there should be only one:

SELECT * FROM Projects WHERE ParentID = 0;

3. Update the RootID value to that root's ID (But keep root project's RootID NULL)

UPDATE Projects SET RootID = 1 WHERE ID > 1;

Copy your Xaraya language pack to Xarigami
Activate the language
In Xarigami Translations module generate skels for all modules, core and all themes
Run command line import.php LOCALE Xarigami

Upgrade to XarTM 1.0

If you used XarTM with date based alpha revisions, before 1.0, you must add one column to the database by a little SQL script and initialize its values. This will change the creation of "index.html" directories back to "index.html" files, as a result of a bugfix.

Errors in previous versions interpreted the empty "index.html" files (appeared in Xaraya 1.1.3) as directory. Exporting will fail if "index.html" files exist. Export will succeed if you delete the "index.html" file, but directory will be created instead. Please make sure you delete those buggy directories when upgrading.

Run the db/upgrade_1.0.sql on your database.: mysql -p xartm <db/upgrade_1.0.sql

Table of contents