Georg Koppen:

Mike Perry:

...

Once we get confirmation on reproducibility (earlier commits were fine, so we should be good), we'll begin testing. Probably Monday/Tuesday.

I got matching builds with the bundles you find on:

https://people.torproject.org/~mikeperry/builds/3.6-beta-1/

Thus, happy testing!

The french build is broken. I have an error message “undefined XML entity” on chrome://torlauncher/content/network-settings-wizard.xul line 130, column 43 on &torSettings.bridgeHelp;

Mark Smith wrote:

On 3/10/14, 1:35 PM, Lunar wrote:

...

The french build is broken. I have an error message “undefined XML entity” on chrome://torlauncher/content/network-settings-wizard.xul line 130, column 43 on &torSettings.bridgeHelp;

There are some new strings in Tor Launcher that need to be translated. I don't know exactly what the interim process is since Mike has always taken care of localization issues for us, but I think English strings are usually pulled in for the untranslated entities.

As far as I can tell, DTD translation files are naively "applied" to the XUL files, without falling back to English for missing entities. If an entity is missing in the DTD, the resulting XUL keeps the entity name as-is, producing invalid XUL like "<tag>&entity_name</tag>". This explains the fatal error that lunar reported.

Problem 1: The above means that when a *new* string is introduced, all existing translations are broken until they are re-imported via `localization/import-translations.sh`, which depends on Transifex having added the corresponding entity to all locales' DTD files. It looks like the latter requires pushing the commits with the new string, and then wait a bit. If that's correct, it makes this situation quite awkward for anyone lacking push rights to your upstream repo, as they cannot package a Tor Launcher with working non-English locales without manually updating every single DTD file.

To make problem 1 less of a problem we could use a tool which can sync all non-English DTD:s to the English one. It be invoked via something like `make update-translations`, making it possible refresh the DTD files so one can package a Tor Launcher that works for all locales without having to involve Transifex (that can wait until right before release time). I'm quite ignorant of XML/DTD stuff, but clearly Transifex must make use of a tool like that. Any ideas about this?

Problem 2: Another issue is that `chrome.manifest` doesn't list English as the first locale. If it did, English would be the fallback for locales not listed in `chrome.manifest` (note that this would not solve problem 1!). For the record, the current fallback is `af`, which doesn't even have a directory in `chrome/locale`, so all unlisted locales are broken.

Problem 3: However, `chrome.manifest` has a pretty exhaustive list of locales, so the broken fallback isn't used in many cases. Unfortunately most of them are pointing either to non-existing directories in `chrome/locale` (fatal error similar to the one from problem 1), and to existing directories which contain translations that are pretty much guaranteed to be outdated (and hence probably hit by problem 1) because they're not updated by `localization/import-translations.sh`. Hence all these locales will experience similar errors to what lunar reported.

For problems 2 and 3 I think some `make` automation can improve the situation quite a bit. See the attached patch for something like that.

I suppose you should see all this as an inspiration for a real fix, as I'm not even sure how you want the translation process to work in the end (perhaps you're fine with the current situation?). For the same reason I've refrained from opening any bugs at this point.

For the record, the current plan for Tails 0.23 (due on 2014-03-19) is to ship a Tor Launcher packaged with my patch applied, since the current state of things breaks Tor Launcher in many locales we care about in Tails:

https://labs.riseup.net/code/issues/6885

(Well, actually Tor Launcher is broken in *all* non-English in Tails due to problem 1 alone, since I introduced a new string. However, your commit a634ce6 fixed this, and my patch fixes the remaining issues, but this still shows the severity of problem 1 for downstreams wishing to package Tor Launcher with potential modifications, like Tails.)

Cheers!

Mike Perry mikeperry@torproject.org writes:

...

a Tor Launcher with working non-English locales without manually updating every single DTD file.

Yes, we have to wait for transifex to at least update the placeholders for us.

Beside the new messages, There are untranslated message/empty messages from 3.5 to be translated in Persian. is it possible that somebody guide me to the transfix repo (? address? or whatever they are called) so I can take care of them?

Thanks a lot, vmon

13/03/14 20:59, Mike Perry wrote:

anonym:

[...]

...

To make problem 1 less of a problem we could use a tool which can sync all non-English DTD:s to the English one. It be invoked via something like `make update-translations`, making it possible refresh the DTD files so one can package a Tor Launcher that works for all locales without having to involve Transifex (that can wait until right before release time). I'm quite ignorant of XML/DTD stuff, but clearly Transifex must make use of a tool like that. Any ideas about this?

Yes, however, it takes a few hours, and has had some issues (which they are in the process of resolving. See https://trac.torproject.org/projects/tor/ticket/9144).

I am not sure if their DTD merge tool is open source/available. Probably?

...

Problem 2: Another issue is that `chrome.manifest` doesn't list English as the first locale. If it did, English would be the fallback for locales not listed in `chrome.manifest` (note that this would not solve problem 1!). For the record, the current fallback is `af`, which doesn't even have a directory in `chrome/locale`, so all unlisted locales are broken.

Ah, I was actually not aware that the first locale was the fallback.

To be honest I was a bit frustrated by the lack of in-depth documentation about this from Mozilla (or if it exists, that they've hidden it so well), so I just went with what I had seen working through experimentation. On second thought, we shouldn't rely on the order. I believe it's just a coincidence that this works at the moment.

I've yet to see it properly documented (except on a third-party site [1]) but it seems that "en-US" is first tried as a fallback. Tor Launcher currently registers "en", which isn't good enough, apparently. Renaming it to "en-US" makes it the fallback no matter in which order the locales are registered.

Because of this, you may want to rename "en" to "en-US" in your translation branches (possibly in Torbutton and any other Firefox/xulrunner repo too, didn't check).

[1] http://babelwiki.babelzilla.org/index.php?title=Create_a_Locale_Support_for_...

Is it per-entity, or is it only used as a fallback if a locale is entirely absent? It sounds like only if a locale is absent?

Yes, only if the locale is absent. There's no per-entity fallback, which is the root cause of problem 1.

...

Problem 3: However, `chrome.manifest` has a pretty exhaustive list of locales, so the broken fallback isn't used in many cases. Unfortunately most of them are pointing either to non-existing directories in `chrome/locale` (fatal error similar to the one from problem 1), and to existing directories which contain translations that are pretty much guaranteed to be outdated (and hence probably hit by problem 1) because they're not updated by `localization/import-translations.sh`. Hence all these locales will experience similar errors to what lunar reported.

For problems 2 and 3 I think some `make` automation can improve the situation quite a bit. See the attached patch for something like that.

Right. This sounds like a good start. I would prefer only one list of locales to update though.. Ideally in a form that makes it easy to cut+paste from the TBB list (ie an env var updated in two places like we have now).

In either case, we should only list locales supported in transifex though, it sounds like.

To avoid problem 3, yes.

An alternative approach to maintaining a static list of supported locales (which is error prone, see commit f9759b6 :)) would be to import and bundle/register all locales available in Transifex:

1. In `import-translations.sh` we import all locales available in Transifex into `chrome/locale`. 2. When packaging we register (in `chrome.manifest`) each locale (i.e. folder) present in `chrome/locale` that has all required translation files (i.e. the same files as the English locale). At the moment e.g. `ca_ES` and `ms` do not have all, so this check is necessary to not break those locales.

This would result in Tor Launcher bundling translations with lots of untranslated strings (possibly 100% untranslated in some) which I'm not sure how you feel about. OTOH, your current process doesn't guarantee anything about that even for the explicitly listed supported locales (except if you manually check). Still, one can easily make it so that only the locales specified by some env var are packaged, which seems to be what you want.

[...]

I am very interested in arriving at a solution that works for both of us out of the box, but I also want to maintain the ability to make it straightforward to verify that the list of locales in TBB, Torbutton, and Tor Launcher are the same. This is most easily done if we have a single line of locales in an env var.

Otherwise, your patch looks like a good direction to head in.

See the `locale_fix` branch on git://git.tails.boum.org/tor-launcher for an improvement using the "alternative approach" mentioned above.

In that branch, building a working package that includes all locales from Transifex' translations amounts to:

make import-translations git add src/chrome/locale git commit -m "Import translations." make package

Note that we have to assume that Transifex' translation repos are up-to-date; we only ensure that each bundled locale has the necessary .dtd and .properties files, not that they have all required entities/strings. That would be a nice fail-safe against problem 1 but we'll have to find the appropriate tools first.

To only bundle selected locales, as you requested, one exports `BUNDLE_LOCALES` appropriately, e.g. with your old list:

export BUNDLE_LOCALES="ar de es fa fr it ko nl pl pt ru vi zh-CN" make package

Unlike the rather optimistic "bundle everything" case, this will throw an error if any of them couldn't be bundled for whatever reason, as a fail-safe. Also note that the "en-US" locale is always automatically included since it's the fallback.

Hopefully this suits your needs.

Cheers!

16/03/14 04:05, Mike Perry wrote:

anonym:

...

13/03/14 20:59, Mike Perry wrote:

...

anonym: I am very interested in arriving at a solution that works for both of us out of the box, but I also want to maintain the ability to make it straightforward to verify that the list of locales in TBB, Torbutton, and Tor Launcher are the same. This is most easily done if we have a single line of locales in an env var.

Otherwise, your patch looks like a good direction to head in.

See the `locale_fix` branch on git://git.tails.boum.org/tor-launcher for an improvement using the "alternative approach" mentioned above.

[...]

...

Hopefully this suits your needs.

This is getting there. Functionally, it seems like exactly the right solution.

However, I noticed that the resulting XPI seems to have a chrome.manifest with multiple locale lines like: locale torlauncher chrome/locale//

This seems bad.

Apparently I forgot to force the push of my local branch, in which I had already fixed that. Now pushed and rebased on most recent upstream.

Can you file a ticket for this branch so we can track the whole translation importation improvement issue there?

Done: #11483

We may want a couple more tweaks before merging this anyway.

Right. Let's continue this on the ticket then, I suppose.

Also, can you squash down your branch to just a commit (or a commit per functional change) rather than preserving the history of the solution itself?

Done in the branch.

Cheers!

On 3/10/14, 1:52 PM, Lunar wrote:

After selecting “fte” as “Use Default Bridges of Type”, the connection progress bar stuck for 105 seconds at 50%. Then it took another 17 seconds to 100%. That's bad. I would probably have closed the app if I was not fiddling with a nearby terminal.

I think this should at least be documented in the release notes (and that's one good condidate for a list of known issues).

This is probably at least partially caused by: https://trac.torproject.org/projects/tor/ticket/9229 I think Mike plans to back-port a tor fix before 3.6 final.

Also, why so many capitalized letters in “Use Default Bridges of Type”?

Thanks for catching our overzealous use of Title Case. I think we should switch to "Use default bridges of type" (although I am not sure if that will cause our translators to have to re-translate that string).