I made it, but it's quite a bit tricky.

Summary:

You have to change the Books.plist file in ~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books for the books you want to have in a series, and also you have to add rows in the BKSeries-*.sqlite database on ~/Library/Containers/com.apple.iBooksX/Data/Documents/BKSeriesDatabase for them and the series.

Here's how I did it:

The database has 4 tables: ZBKSERIESCHECK, ZBKSERIESITEM, Z_PRIMARYKEY and Z_METADATA. I only changed the first three tables. The "series" is treated just like a book in these tables.

Add the books and the series in table ZBKSERIESCHECK. The table has 5 columns. Z_PK is the primary key. It has to be a sequential number.

Z_ENT and Z_OPT I just completed with a "1".

ZDATECHECKED is just the date the row is created. In my case it was auto-assigned.

ZADAMID is a unique code for the book or series. This is the table with the data I added. The zAdamId I used are from Calibre, so I asume that you can use any number. Keep in mind that those numbers have to be different from the series/books you purchased. The row with the zAdamId "1" is my series, the others are the books. Add the relationships within books and series in the table ZBKSERIESITEM. This table has 16 columns. Z_PK, Z_ENT and Z_OPT are not the same ones defined in the previous table. Z_PK has to be sequential and, Z_ENT and Z_OPT I just completed with "2" and "1" respectively.

ZISCONTAINER defines if a zAdamId is a series or not. "1" is true, "0" is false.

ZPOSITION defines the number of a book in a series, starting with "0" for the first book.

ZADAMID is the same defined in the previous table for the same item.

ZAUTHOR is the author of the series/books.

ZGENRE is the genre of the series/books.

ZSEQUENCEDISPLAYNAME defines how a book is displayed as a member of a series. For example, for the first book, you can define "Book 1" or "Volume 1" or "Vol.1". This string is going to be prepended to the name of the book. This column has to be empty for the series.

ZSERIESADAMID is the series zAdamId of the book. For the series, this is the same as ZADAMID.

ZSERIESTITLE is the name of the series.

ZSORTTITLE and ZTITLE is the name of the book or series. Adjust table Z_PRIMARYKEY. I found that this table has only two rows. One for each of the previous tables. Here you have to change the Z_MAX column to the MAX(*) value of the Z_PK in each of the previous tables. For instance, in my DB, for the table ZBKSERIESCHECK (BKSeriesCheck) the MAX(*) of the column Z_PK is "22". Save the changes. Open the Books.plist file. Look for each of the books of the series and add these keys: seriesAdamId as Number with the value of ZBKSERIESITEM~ZSERIESADAMID.

itemId as Number with the value of ZBKSERIESITEM~ZADAMID.

seriesSequenceNumber as String with the value of ZBKSERIESITEM~ZPOSITION + 1. (Remember that in the table it starts with "0".)

seriesTitle as String with the value of ZBKSERIESITEM~ZTITLE. Save the file. Close iBooks (if you didn't close it from the beginning). Kill the com.apple.BKAgentService process. Start iBooks. You should be able to see your series.

Notes

Those columns I didn't name, I just left them empty and it didn't affect the end result.

I think it's waaay to difficult to put all your books and series properly when you have a lot of them. Hope to see someday something that do all this auto-magically.

This is my first post. I hope everybody finds it useful.