vformat plugin asserts because of un-sorted xmlformat in xmlformat_to_vevent conversion

[savago]

I'm getting an assert error:

/home/adenilson/osync/current/trunk/opensync/xmlformat/opensync_xmlformat.c:226:E:osync_xmlformat_search_field: Assertion "xmlformat->sorted" failed

Checking the source code revealed that it expects the xmlformat document to be sorted. But I *do* call osync_xmlformat_sort on my get_changes function (gcalendar.c:297).

I changed the code to print to stdout the xmlDoc and got a XML that successfully validates using xmlformat-event.xsd schema.

What is really weird is that almost the same code is being used by the contacts function successfully.

Attached you can find traces and other information.

[savago]

Traces and other information

[dgollub]

vformat plugin r3826 src/xmlformat-vevent.c:795:

                //  FIXME : handle errors or missing field
                OSyncXMLFieldList *list = osync_xmlformat_search_field(xmlformat, "Method", error, NULL);
                if (list) {

This is the only osync_xmlformat_search_field() i can think of which fits the gdb backtrace. (Btw. please build with debuginfo next time so it's easier to find the corresponding call - this time was easy since it was the only one in vevent-code).

My first guess is that the sorted status gets no copied once multiplied in the engine. I guess it's quite safe to assume that the xmlformat is sorted when it comes from the engine...

Not quite sure yet if it's really engine or just xmlformat-plugin bug - since copying is done via the format-plugin.

[dgollub]

xmlformat copy looks fine. If source xmlformat is sorted the destination format get's also marked as sorted.

[dgollub]

Is this reproducible? Please try this with a current /trunk with tracing enabled and inside gdb.

gdb --args osynctool --sync yourgroup

Once the assert() kicks in please print the xmlformat struct by running:

(gdb) p *xmlformat

I wonder if the structure looks valid at all... maybe something with the conversion path just went completely wrong.

If you're able to reproduce this only with _one_ entry please try also this with gdb:

(gdb) break osync_xmlformat_copy
[... when break poing reached ... ]
(gdb) p *source
[...]
(gdb) continue
[...]

In meanwhile i have certain doubts this is about the engine.

[dgollub]

savago, ping

[gebart]

I've run into this same problem when trying to sync between file-sync and google-calendar. I've added some more trace output to my testing version but it seems that it forgets that the xmlformat is sorted somewhere before the osync_xmlformat_copy()ing starts.

The google-calendar plugin runs osync_xmlformat_sort() on every contact and event right after it parses them so they should have their ->sorted flag set.

I'll post back if I find anything useful.

[gebart]

I've managed to fix this problem by adding a call to osync_xmlformat_sort() in formats/xmlformat.c: demarshal_xmlformat()

I don't know enough about the way opensync works to give an accurate answer (I only started messing around with it two days ago) and my solution is pretty ugly since it doesn't adress the cause of the problem, only the symptoms, but I believe that the reason for this problem is that the xml data is passed as raw xml and not as OSyncXMLFormat.

Below is part of a trace that helped in finding the right place to look for it:

[1230203413.629488]     >>>>>>>  _incoming_dispatch(0x199c5c0)
[1230203413.629516]             Dispatching 0x7f6e04000e90:9(OSYNC_MESSAGE_NEW_CHANGE)
[1230203413.629539]             >>>>>>>  _osync_client_proxy_message_handler(0x7f6e04000e90, 0x199dd10)
[1230203413.629561]                     proxy received command 9
[1230203565.130428]                     >>>>>>>  osync_xmlformat_parse(0x1a60040, 1124, 0x7f6e0ba67f48)
[1230203565.130536]                     <<<<<<<  osync_xmlformat_parse: 0x181e970
[1230203752.847785]                     Data is: 0x181e970, 48
[1230203752.847855]                     >>>>>>>  _osync_engine_receive_change(0x199dd10, 0x17fbb50, 0x1a5ec10)
[1230203752.847890]                             Received change http://www.google.com/calendar/feeds/xxx%40gmail.com/private/full/_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxx, changetype 1, format xmlformat-event, objtype event from member 2
[1230203752.847921]                             common format 0x1817be0 for objtype event
[1230203752.847950]                             converting to common format xmlformat-event

The call to osync_xmlformat_parse() in the trace above is located in formats/xmlformat.c: demarshal_xmlformat() and the ->sorted flag is never restored in that function.

I'd appreciate it if someone else with more knowledge about opensync take a look at the whole passing of xmlformat objects as messages and maybe find a better solution than mine.

[gebart]

patch that cures the symptoms on version 0.38

[dgollub]

Thanks gebart for looking into this!

I committed your patch with some additional nodes how the later solution should look like. r4464

The idea is to avoid any sorting if not required. Instead of using just osync_xmlformat_{assemble,parse} we just also marshal/demarshal the OSyncXMLFormat struct elemenets like sorted. IIRC xmlformat_pasrse/assemble is just some libxml parse/assemble calls wrapped.

I close this ticket now as fixed. Since i plan to address this when splitting the entire xmlformat stuff out of the OpenSync? core. There should be already a ticket for the xmlformat split...

[dgollub]

Reopened due to report on opensync-devel@  http://thread.gmane.org/gmane.comp.misc.opensync.devel/3722

[scriptor]

Patch against opensync_mapping_engine.c to hopefully solve this ticket.

[scriptor]

I have also stumbled over this in various tests and I have silenced this assertion by the following patch:

opensync_mapping_engine.c.diff

Bye, bye.

[scriptor]

Sorry, I attached the wrong file. opensync_mapping_engine.c.diff is for a different ticket.

Here the correct one is xmlformat.c.diff.

[paule]

Updated patch (supercedes xmlformat.c.diff)

[paule]

Updated patch attached. Hopefully we can resolve this one soon - the Opie plugin will not work at all without a fix for it.

[dgollub]

(In [5980]) Applied patch from paule, scriptor which introduces sorting for XMLFormat plugin.

In future versions we should try to find a way to avoid sorting as often as possible.

fixes #923

[dgollub]

(In [5981]) Added TODO comments about avoiding sorting. Sorting is complex: O(N) and causes performance looses...

refs #923