In reply to this post by Tim Lyons [hidden email]>

I have a number of main concerns about DB-API/BSDDB.



(1) Upgrading a database from BSDDB to DB-API.



(2) Using the SQL features for searches and filters.



(3) Transactions for dbapi



(4) Testing.





(1) How is a grampsdb upgraded from BSDDB to DBAPI? Doug's discussion seems to be about upgrading one DBAPI database to another when new fields are added. I am concerned with the initial upgrade. We know that this is the most error prone activity, because of the number of problems that are raised about this on the mailing list. Even though we may tell people to export a Gramps XML file before upgrading Gramps, it is clear that most people don't actually do, this, and many people can't go back to an earlier version of Gramps to do so, and most people expect the upgrade to work by using the new version of Gramps, as it has in the past. I don't see how upgrade can work, as we can't run two different databases at the same time (Using the interfaces as they are at present).



This is a good point, but I think maybe one of documentation. With Gramps 5.0, you'll open up your old databases as you always have. Nothing will be different. But, to get people to use the new database backend, the only method we have currently is for them is to export, create a new database, and then import. We could think of something more specific (eg, a button to "Convert to DB-API") which should be pretty straightforward to implement. But I think we are being cautious... we want to make a solid first step (with the ability to remain with BSDDB in case something bad pops up).

(2) Using the SQL features for searches and filters. One of the virtues of SQL would be that the underlying database mechanisms could be used to make searches and filters much more efficient. This seems like a good thing. I know there has been a lot of discussion about the strategies for using SQL. However, it seems to me (from what little of the discussions I understand) that this is mainly about things like using the SQL queries from the keyboard. I think the searches and filters have to be implemented so that they use SQL, before going any further, because if the internal interface is not suitable for this purpose, it should be changed before release.



There is no internal interface for filters to use the backend directly yet. I looked at this, but it will take some refactoring, I believe. Currently, there is a forced loop to go through all of the data for each filter, and then to filter that. This will need some careful thought to move the WHERE up before that. In addition, whatever changes we make for the filter system while we have BSDDB and DB-API, we will have to duplicate the interface. I'd rather not do it twice, but it may be some time before we remove BSDDB. In any event, filters won't be any faster or slower now.

Even if we never rewrite filters to use SQL (which would be a shame) there are still many reasons to move to DB-API. But, we will rewrite filters. That would be a good next step for 5.1, given that everything works as planned.

(3) Transactions for dbapi. In some situations, the Gramps paradigm where a single editor edits a single object, and the data is simply committed, the simple database paradigm where an update is automatically wrapped in a transaction is entirely suitable. However, some parts of the Gramps data model are highly interdependent, so you can't just update one object without updating others. This applies particularly in import, export and upgrade, but also in some other minor places such as delete (I think). Transactions are provided in Gramps which call the underlying BSDDB to implement ACID for long running updates (as well as simple edits). Are these also implemented for DBAPI? (I don't think so, but I am not sure. This needs to be fixed).





I believe that the Python transaction behavior is the same for BSDDB as for DB-API. That is, if a transaction is interrupted in the same place in the two systems, the result will be the same: either the whole transaction is committed, or it is rolled back. This is true for batch (eg, import) and single edits/deletes.



(4) Testing. Yes please!



I did say that I thought we were not getting the most out of BSDDB, and that it was causing us a lot of problems, so something that had a reputation of being more reliable (like sqlite) sounded like a good idea, and I am enormously impressed by the amount of work that Doug has done on this, but as above there may be a few more things that need to be fixed.



I always think it would be a good idea for others to review code. And more testing. Places where there needs additional tests: are unicode sort orders correct? Are secondary functionality, like undo/redo working correctly? Are things stable on Windows and Mac?

-Doug

regards,

Tim.



If you reply to this email, your message will be added to the discussion below: http://gramps.1791082.n4.nabble.com/Preparation-for-Gramps-5-0-tp4675225p4675258.html

To receive all replies by email,

This email was sent by Tim Lyons (via Nabble)To receive all replies by email, subscribe to this discussion On Fri, Apr 15, 2016 at 10:48 AM, Tim Lyons [via GRAMPS]wrote:

