A discussion about filtering on opensync, on IRC. Kept here in the page for reference about some details of the filtering system.

People:

• dgollub - Daniel Gollub
• abauer - Armin Bauer
• boto - Eduardo Habkost

Any refactoring to a more readable and short text explaining the ideas would be welcome. :)

[Qui Fev 16 2006] [09:21:46] <dgollub>	now i can filter with msynctool objtypes which i don't want to sync... very helpfully for my gnokii-sync development
[Qui Fev 16 2006] [09:21:50] <dgollub>	hi boto
[Qui Fev 16 2006] [09:22:52] <dgollub>	abauer: which other changes are planned?
[Qui Fev 16 2006] [09:22:55] <boto>	I didn't get the full discussion, but I don't think the current system of filters are very reliable
[Qui Fev 16 2006] [09:23:31] <boto>	abauer: I was thinking some time ago: what if we use the filters in the mapping_multiply() function instead?
[Qui Fev 16 2006] [09:23:36] <abauer>	dgollub: in the syncengine: finishing the IPC
[Qui Fev 16 2006] [09:23:57] <abauer>	- support for encodings
[Qui Fev 16 2006] [09:24:11] <abauer>	- format registration during the connect function
[Qui Fev 16 2006] [09:24:12] <dgollub>	boto: the basic filters  deny, allow works for me the last few hours very well :)
[Qui Fev 16 2006] [09:24:42] <abauer>	well... these are the major changes im working in the current branch
[Qui Fev 16 2006] [09:24:59] <boto>	dgollub: the problem is when you change an entry on the next synchronization
[Qui Fev 16 2006] [09:25:02] <abauer>	boto: what about the filters? what exactly doesnt work?
[Qui Fev 16 2006] [09:25:24] <boto>	if an entry is not filtered and then it is filtered in the next synchronization, the item simply won't be changed on the other side
[Qui Fev 16 2006] [09:25:36] <boto>	for example, if you filter by category and change an item's category
[Qui Fev 16 2006] [09:26:05] <boto>	and the changetype won't be correct if an entry is filtered when added, but then it isn't filtered when modified
[Qui Fev 16 2006] [09:26:41] <boto>	the logic to make this work better seems to fit on osync_mapping_multiply(). together with better state enum values for mappingentries, so we know if the item is really in the other side or not
[Qui Fev 16 2006] [09:28:36] <boto>	the filters should work good when the filter result never changes for a given object (i.e. if a contact is filtered, it will always filtered on next syncs). but if the filter is based on content, it doesn't seem to be reliable
[Qui Fev 16 2006] [09:30:18] <abauer>	boto: hmm right. i dont know if we have a test case for this already
[Qui Fev 16 2006] [09:32:32] <dgollub>	boto: where is this filter placed in the plugin or in the sync frontend?
[Qui Fev 16 2006] [09:33:07] <boto>	dgollub: what do you mean by "this filter"?
[Qui Fev 16 2006] [09:33:50] <dgollub>	filter which based on content
[Qui Fev 16 2006] [09:34:35] <boto>	dgollub: plugins may register filter functions that may be referred to, in the filters
[Qui Fev 16 2006] [09:36:35] <dgollub>	ah ok ... i thougt about general filters (not depending on content) which are registred by opensync it self or the frontends msynctool multisync0.90 ...
[Qui Fev 16 2006] [09:36:56] <dgollub>	filtering by object type (event, note, contact, ...)
[Qui Fev 16 2006] [09:37:46] <boto>	dgollub: opensync itself and the frontend may also register filters. we just didn't wrote any of them yet  :)
[Qui Fev 16 2006] [09:38:28] <boto>	althought I don't think that the frontend would be a good place to register some filter: it would be better to add it to the opensync tree, not the tree for a specific frontend
[Qui Fev 16 2006] [09:39:20] <dgollub>	yesterday i added a filter for objects function in msynctool and it work quiet well.. in think this kind of general filter will not harm the plugins :)
[Qui Fev 16 2006] [09:40:12] <dgollub>	i think the multisync-gui have this kind of settings without the filtering function in the background
[Qui Fev 16 2006] [09:40:56] <dgollub>	for developing it is very useful to only sync a special object type .. because many unfinished plugins only support only the half of the object types
[Qui Fev 16 2006] [09:41:24] <boto>	dgollub: yes, filtering for objtype is probably the most used (and wanted) feature of filtering
[Qui Fev 16 2006] [09:42:56] <boto>	ah, the filter-for-objtype case is another example of how the filter may not behave as expected: if you add filters for syncing only a given objtype, it won't be synced. but opensync will still ask the plugin for this objtype. if the objtype is completely filtered, we don't want the plugin to read the filtered objtypes either
[Qui Fev 16 2006] [09:43:50] <dgollub>	boto: you are right. but you only have to swap check for converting with filtering ...  so first check if to filter then check if we can convert
[Qui Fev 16 2006] [09:44:59] <boto>	dgollub: I don't see how this swap is related to the problem I described. I am talking about the plugins reading the data for an objtype even if it is filtered
[Qui Fev 16 2006] [09:45:36] <boto>	dgollub: what exactly do you want to fix by swapping filtering and converting? people may want to filter based on the "resulting objformat" after conversion also
[Qui Fev 16 2006] [09:46:41] <boto>	however, calling the filtering function with some parameter saying "we didn't convert yet, don't check the 'dest_objtype' field" may work
[Qui Fev 16 2006] [09:47:43] <boto>	maybe we could do the same for the "reading data on the plugin" stage: on objtype_enabled() function (that is used by the plugins) we may call the filtering function saying "we don't know anything about the data yet, but tell me if my sync member or the objtype is completely filtered"
[Qui Fev 16 2006] [09:48:57] <boto>	abauer: any comments about those ideas?  :)
[Qui Fev 16 2006] [09:53:51] <dgollub>	ok i understand. i ignored what osync_filer_change_allowed() else does.
[Qui Fev 16 2006] [10:10:04] <dgollub>	http://frische.cryptomilch.de/opensync/patches/ - here you can find the patches which brings multisync to filtering objtypes .. but with swapping filtering before changing...
[Qui Fev 16 2006] [10:10:28] <dgollub>	-changing +converting
[Qui Fev 16 2006] [10:38:05] <boto>	dgollub: I didn't understand yet what the filter-before-converting change fixed
[Qui Fev 16 2006] [10:43:40] <boto>	dgollub: just looked at your patch
[Qui Fev 16 2006] [10:43:42] <boto>	dgollub: if you just want to filter by objtype at sync-time, you may use the set_objtype_enabled() functions. they will work better for this task
[Qui Fev 16 2006] [10:44:18] <dgollub>	boto: i take a look on it...
[Qui Fev 16 2006] [10:45:35] <boto>	dgollub: and, the problem of having to swap filtering and conversion probably may be fixed if you add filters equivalent "sourceobjtype == x || destobjtype == x" instead of "srcobjtype == x && srcobjtype == x"
[Qui Fev 16 2006] [10:46:14] <boto>	i.e. call osync_filter_add() twice, with srcobjtype set on one call, and destobjtype set on the other call
[Qui Fev 16 2006] [10:48:36] <dgollub>	for me it fix that a plugin which dont support a objtype don't return an error when osync_change_convert_member_sink dont find a sink member
[Qui Fev 16 2006] [10:49:05] <dgollub>	for example i sync with  kdepim-sync and gnokii-sync...
[Qui Fev 16 2006] [10:49:28] <dgollub>	gnokii-sync only support calendar  ... but kdepim provides also information about notes .. which gnokii-sync can not handle at the moment.
[Qui Fev 16 2006] [10:50:40] <dgollub>	will set_objtype_enabled prevent that osync_change_convert_member_sink fail?
[Qui Fev 16 2006] [10:52:58] Sair	bjohnson deixou este servidor  (Remote closed the connection).
[Qui Fev 16 2006] [11:01:52] <boto>	dgollub: ah, I see
[Qui Fev 16 2006] [11:02:36] <boto>	dgollub: set_objtype_enabled will even prevent that the plugin report the data for the filtered objtype
[Qui Fev 16 2006] [11:03:09] <boto>	of course, the plugin is supposed to always check is_objtype_enabled(). If it doesn't check it, it should be fixed
[Qui Fev 16 2006] [11:03:51] Entrada - 	O spuk- juntou-se a este canal (n=gustavod@unaffiliated/spuk).
[Qui Fev 16 2006] [11:03:54] <dgollub>	i patched my msynctool with set_objtye_enabled. as soon bluetooth is working again we will see if it works :)
[Qui Fev 16 2006] [11:05:35] <dgollub>	boto: ok seems to work :)
[Qui Fev 16 2006] [11:06:43] <boto>	good  :)
[Qui Fev 16 2006] [11:11:06] <dgollub>	ok forget about swapping of filtering and converting - silly idea :P
[Qui Fev 16 2006] [11:17:52] <dgollub>	http://frische.cryptomilch.de/opensync/patches/multisync_filtering-objtypes.patch - short and simple :)

Another discussion about the filtering system:

[Qui Jun 8 2006] [12:57:06] <abauer> btw: i thought again about the filtering system
[Qui Jun 8 2006] [12:57:41] <abauer> i think we should provide several hook points to the filter system
[Qui Jun 8 2006] [12:58:07] <abauer> for example: the first point would be in the plugin process when the change gets reported
[Qui Jun 8 2006] [12:58:46] <abauer> of we filter here, we dont even send the change to the engine
[Qui Jun 8 2006] [12:59:04] <abauer> the next point would be before the conversion
[Qui Jun 8 2006] [12:59:10] <abauer> the next one after the conversion
[Qui Jun 8 2006] [12:59:37] <abauer> the last one would be just before sending to the plugin process (when the change is already converted)
[Qui Jun 8 2006] [12:59:55] <abauer> this way we can get rid of saving the "initial objformat" stuff etc
[Qui Jun 8 2006] [13:57:08] <boto> completely agreed  :)
[Qui Jun 8 2006] [13:57:22] <boto> but, I think we may use the same set of rules for each point
[Qui Jun 8 2006] [13:57:26] <boto> but check them on every point
[Qui Jun 8 2006] [13:57:38] <boto> for example, on plugin change reporting, we will able to handle the rules that has only srcplugin set
[Qui Jun 8 2006] [13:57:59] <boto> on conversion, we will able to handle the rules that have only srcplugin or srcformat set
[Qui Jun 8 2006] [13:58:26] <boto> and so on. we wouldn't need different list of rules, but according to the rules that are set, we may be able to use them
early if possible
[Qui Jun 8 2006] [13:58:45] <boto> these are two approaches. I am not sure which is better
[Qui Jun 8 2006] [13:59:20] <boto> 1. have multiples rulesets, one for each point of the conversion (similar to INPUT/PREROUTING/POSTROUTING/etc on iptables
)
[Qui Jun 8 2006] [14:00:19] <abauer> yes. good idea
[Qui Jun 8 2006] [14:00:33] <boto> 2. have only a ruleset, but if on some early point in time we know a change will be filtered, filter (as in the example a
bove: if the rule is by srcobjtype, we may filter it on get_changeinfo() time, but if it has srcobjformat set, we will need to apply it later)
[Qui Jun 8 2006] [14:00:44] <boto> I have thought more about (2)
[Qui Jun 8 2006] [14:00:56] <boto> I think it would be more intuitive for the user and easier to use by applications
[Qui Jun 8 2006] [14:01:17] <boto> as we will have only one single list of rules but the engine will know when to apply them
[Qui Jun 8 2006] [14:01:25] <boto> well, I need to be away, right now
[Qui Jun 8 2006] [14:01:28] <boto> but I will be back later
[Qui Jun 8 2006] [14:01:39] <boto> you may talk what you think, I will read later :)
[Qui Jun 8 2006] [14:04:46] <abauer> hmm good idea... i actually though about implementing 1)
[Qui Jun 8 2006] [14:04:57] <abauer> but 2) also is not bad
[Qui Jun 8 2006] [14:05:03] <abauer> and shouldnt be hard to implement
[Qui Jun 8 2006] [15:07:54]   * boto is back
[Qui Jun 8 2006] [15:10:03] <boto> yes. this way the filterset could even replace the filter_objtype (objtype_enabled) system
[Qui Jun 8 2006] [15:10:18] <boto> i.e. filters with only srcobjtype set are equivalent to --filter-objtype, currently
[Qui Jun 8 2006] [15:10:47] <boto> so, independently of the decision about (1) and (2), we need to list the points where the filters could be queried. I can
 think of:
[Qui Jun 8 2006] [15:11:02] <boto> - before changes are reported on get_changeinfo() (equivalent to --filter-objtype/objtype_enabled())
[Qui Jun 8 2006] [15:11:09] <boto> - just after a change is reported by get_changeinfo()
[Qui Jun 8 2006] [15:11:28] <boto> - just after the change data is requested from the plugin
[Qui Jun 8 2006] [15:11:44] <boto> - just before (or after) conversion (I am not sure about the "before conversion")
[Qui Jun 8 2006] [15:12:10] <boto> - just before committing to the member