WIP: Sqlite quick search

[julian-klode]

Just filing this now as a replacement for the Xapian search, building the database in-memory for now. This should probably cache in XDG_RUNTIME_DIRECTORY.

TODOs

• Run the main loop when building/searching
• Do some caching
• Locality sorted building such that we get good performance with Acquire::GzipIndexes enabled

[derVedro]

thanks, worked fine for me.

[YuzhongHuangCS]

Appreciate this, when this would be merged?

[amaa-99]

Hi . Thanks for your work on this issue.

I did some tests and, for pattern search (i.e. regex) only on the package names, it seems that a direct search using a for loop and a c++ regex matcher seems to be fast enough and requires minimal implementation:

• We already have the package list in memory, so iterating is very fast.
• There are no indexes that need updating, so no more 'waiting for index updating to finish"...).

With the current synchronous populating of the view, doing a full text regex search on the package description too is not responsive enough (for me) though (a tiny bit too slow for immediate update of the view - a second or so). With asynchronous populating of the view it would probably be responsive enough though.

In the other hand, if using a database the specialty of full text search, in particular when patterns are used (e.g. regex) need to be taken into consideration (otherwise the performance will most likely be worse than the minimal implementation with a for loop standard c++ regex matcher).

For the sqlite case the FTS5 Extension (i.e. "Full-Text Search") is most likely needed.

p.s. Here's the base code that I've used on my tests with std::regex (I tried some variations of this, e.g. searching on the descriptions too, etc...):

bool RPackageLister::limitBySearch(string searchString)
{
    _viewPackages.clear();
    std::regex pattern(searchString, std::regex::optimize | std::regex::nosubs | std::regex::icase);

    for (const RPackage *package : _packages)
    {
        if (package == NULL))
        {  // TODO: Should not happen: Needs to be verified!
            continue;
        }

        if (!_selectedView->hasPackage(package))
        {   // Filter out results that apt doesn't know.
            // TODO: Verify if this check is needed!
            continue;
        }
        
        if (!std::regex_match(package->name(), pattern))
        {
            continue;
        }

        _viewPackages.push_back(package);
     }

     return true;
}

[amaa-99]

I've done some tests with a sqlite fts5 database in memory only (there's no point on persisting that data):

• Populating the database with the package names and descriptions takes up to 2s (with the current 70000 packages found in the debian apt repos). This only needs to be done when the package lists gets updated.
• Quick filter text searches on both the name and description fields takes around 20ms for simple patterns (like e.g. 'gcc').
• The complete indexing and searching is done in-process (no need to spawn any external processes), and there's no access to the filesystem (no files saved, no need to look for cache files...).

On my test implementation the database gets updated at the same point as the view gets updated, so, no more "waiting for index updating to complete".

[julian-klode]

Mostly the reason for fts is that it can do stemming I believe which produces more meaningful results than substrings

[amaa-99]

Hi Julian. Nice to hear from you.

My tests were mainly to get a rough idea of the times involved, both in the minimal code variant using std::regex and with the database.

I think the searching itself can be done synchronously... What were the worst case search times that you got on your tests?

I think we could get by with populating the database synchronously too when updating the view after reading the package cache since the 2s additional delay until the view gets updated is not very noticeable. I'll try to run a test asynchronously populating the database in the background after getting a package cache update to get an idea whether the extra complexity pays off.

About saving the database to file, I think there's no benefit since it anyway needs to get updated when the app starts.
Trying to update an existing one by deleting/inserting elements will most likely take longer then recreating the whole data.

Do you know which search use-cases/patterns would be needed?