Ticket #809 (closed defect: invalid)

Opened 3 months ago

Last modified 2 months ago

incorrect encoding done by opensync engine

Reported by: pkot Assigned to: dgollub
Priority: high Milestone:
Component: Plugin: gnokii Version: 0.32
Severity: normal Keywords: gnokii encoding
Cc: ion

Description

opensync 0.22 seems to incorrectly use libgnokii. When using gnokii --getphonebook -v with locale set to pl_PL.UTF-8 I get: 

BEGIN:VCARD
VERSION:3.0
FN:DJ Paweł
N:DJ Paweł
TEL;TYPE=PREF,VOICE:+48601123456
X-GSM-MEMORY:ME
X-GSM-LOCATION:238
X-GSM-CALLERGROUP:5
CATEGORIES:brak
TEL;TYPE=VOICE:+48601123456
END:VCARD

When syncing gnokii-sync -> file-sync what I got is: 

BEGIN:VCARD
VERSION:2.1
FN:DJ Pawe?
N:;DJ Pawe?;;;
TEL;VOICE:+48601123456
END:VCARD

This looks as locale was not set for libgnokii usage. gnokii uses for that following code: 

#ifdef ENABLE_NLS
        setlocale(LC_ALL, "");
        bindtextdomain(GETTEXT_PACKAGE, LOCALEDIR);
        textdomain(GETTEXT_PACKAGE);
#endif

If you need any more detailed information I'd be happy to provide it.

Attachments

encoding.patch (0.8 kB) - added by pkot on 10/05/08 19:39:31.

Change History

08/17/08 21:56:45 changed by dgollub

  • status changed from new to assigned.

Interesting! Thanks for the hint i have to look into this topic more closley....

08/17/08 22:02:38 changed by dgollub

ping

08/17/08 23:07:43 changed by tuju

  • milestone changed from 0.40 to 0.38.

08/21/08 16:27:13 changed by tuju

  • milestone deleted.

non-core plugins wont block releases. thus milestone == NULL.

10/04/08 16:54:48 changed by ion

  • cc set to ion.

10/05/08 17:55:06 changed by mbanck

Seems to be an issue in the stable version as well (the orginal report is for opensync-0.22), Ubuntu got this reported as well https://bugs.launchpad.net/bugs/278195

Will this get fixed on the stable branch; or is there at least a work-around for users?

10/05/08 19:39:31 changed by pkot

  • attachment encoding.patch added.

10/05/08 19:40:44 changed by pkot

The attached patch fixes it for me. However it needs more care: to autodetect needed headers and libs.

10/05/08 21:04:35 changed by ion

Thanks for the patch, but it only seems to fix the issue if opensync is running with UTF-8 locales. Everyone should be using UTF-8 locales, but still, this functionality shouldn’t depend on it.

10/05/08 21:23:59 changed by pkot

@ion: Can you show the example when it doesn't work? And if it doesn't work, make sure gnokii --getphonebook works. If both do not work correctly, file a bug for gnokii. It doesn't depend on UTF_8 encoding, just on LC_ALL being set to the encoding you are using with your applications. In fact bindtextdomain() and textdomain() are not needed -- it was simply copied from gnokii app and I didn't bother to tweak the patch as it worked just as I wanted to.

(follow-up: ↓ 11 ) 10/05/08 21:34:02 changed by ion

multisync0.90, create a sync pair, synchronize → everything works

LC_ALL=C multisync0.90, create a sync pair, synchronize → the entries received from the phone have umlauts substituted with question marks.

Locale settings shouldn’t affect the synchronization at all. For instance, evolution understands UTF-8-encoded entries, no matter the locales.

The gnokii-sync source seems to contain the line ‘gn_char_set_encoding("UTF-8");’, but it doesn’t seem to DTRT.

(in reply to: ↑ 10 ; follow-up: ↓ 12 ) 10/05/08 21:43:34 changed by pkot

Replying to ion:

multisync0.90, create a sync pair, synchronize → everything works

With what LC_ALL setting?

LC_ALL=C multisync0.90, create a sync pair, synchronize → the entries received from the phone have umlauts substituted with question marks.

That's expected. LC_ALL=C doesn't know anything about umlauts.

Locale settings shouldn’t affect the synchronization at all. For instance, evolution understands UTF-8-encoded entries, no matter the locales.

I disagree. We indeed might get UTF-8 centric but it would make us dependent on all apps understanding UTF-8 which I think is wrong. Locale says which encoding does the user want/use -- and, for instance, it makes perfect sense for file plugin.

As I said the patch is not perfect but at least it made it work for me (pl_PL.UTF-8).

The gnokii-sync source seems to contain the line ‘gn_char_set_encoding("UTF-8");’, but it doesn’t seem to DTRT.

And which libgnokii version is this? We had problem for 0.6.26 and maybe few earlier versions. Tomorrow there will be new gnokii version.

(in reply to: ↑ 11 ; follow-up: ↓ 14 ) 10/05/08 21:53:32 changed by ion

Replying to pkot:

With what LC_ALL setting?

Empty, but with other locale settings being fi_FI.UTF-8 or en_US.UTF-8

Locale settings shouldn’t affect the synchronization at all. For instance, evolution understands UTF-8-encoded entries, no matter the locales.

I disagree. We indeed might get UTF-8 centric but it would make us dependent on all apps understanding UTF-8 which I think is wrong. Locale says which encoding does the user want/use -- and, for instance, it makes perfect sense for file plugin.

With the file plugin, the vcard entries seem to get correctly marked with the encoding. Whatever program might be used to read the files afterwards should be able to present them with the correct encoding for the user’s terminal. By using UTF-8 for the data, there’s no data loss due to “wrong” locale settings or perhaps broken locales.

And which libgnokii version is this? We had problem for 0.6.26 and maybe few earlier versions. Tomorrow there will be new gnokii version.

The Ubuntu intrepid package libgnokii3 0.6.26.dfsg-2. Isn’t 0.6.26 the newest release?

(follow-up: ↓ 15 ) 10/05/08 23:38:25 changed by pkot

I checked again with gnokii. gn_char_set_encoding("UTF-8") should be enough to provide an output in UTF-8. It might not work earlier due to bug in libgnokii. So hopefuly upgrade to gnokii 0.6.27 will fix the problem.

(in reply to: ↑ 12 ) 10/05/08 23:43:25 changed by pkot

Replying to ion:

Locale settings shouldn’t affect the synchronization at all. For instance, evolution understands UTF-8-encoded entries, no matter the locales.

I disagree. We indeed might get UTF-8 centric but it would make us dependent on all apps understanding UTF-8 which I think is wrong. Locale says which encoding does the user want/use -- and, for instance, it makes perfect sense for file plugin.

With the file plugin, the vcard entries seem to get correctly marked with the encoding. Whatever program might be used to read the files afterwards should be able to present them with the correct encoding for the user’s terminal. By using UTF-8 for the data, there’s no data loss due to “wrong” locale settings or perhaps broken locales.

I fully agree that UTF-8 is superior to ASCII or ISO-whatever. The problem is that the applications may not understand UTF-8 as the input. However it may not be the case for opensync.

On the other hand: I double checked with gnokii and what opensync gnokii-sync plugin does should be just enough to get UTF-8 output. I suggest retesting with current gnokii, 0.6.26 is for sure affected with encoding problems.

And which libgnokii version is this? We had problem for 0.6.26 and maybe few earlier versions. Tomorrow there will be new gnokii version.

The Ubuntu intrepid package libgnokii3 0.6.26.dfsg-2. Isn’t 0.6.26 the newest release?

There was very much development done since then. Tomorrow is 0.6.27 scheduled and it will be equal to current CVS unless anything expected happens.

(in reply to: ↑ 13 ) 10/06/08 06:33:45 changed by ion

  • status changed from assigned to closed.
  • resolution set to invalid.

Replying to pkot:

So hopefuly upgrade to gnokii 0.6.27 will fix the problem.

Awesome. I built gnokii from CVS and rebuilt the sync plugin (without any changes) against it; now synchronization works perfectly.

Resolving as invalid, since the bug was not in the plugin.