Simple Core Data Full Text Search With Swift and SQLite

Written by Emil Loer on Mar 22, 2016 |

In this article I want to show you my approach to providing full text search for Core Data models. It was designed for iOS apps but should work fine on OS X too.

Core Data is really awesome, but there is one part where it really lacks: searching. With NSFetchRequest you can provide a LIKE predicate to search for a substring of a property, but this is not quite flexible. The fetch request also only works on a single entity, so if you want to search within a group of entity classes you either have to use inheritance with a shared property or you're out of luck.

The good thing is that Core Data is built on SQLite and SQLite itself provides a full text search engine. So we can create a search index ourselves and use it in concert with Core Data.

Dependencies

The first thing we have to to is ensure we have access to SQLite in our project. To keep things simple I will be using the FMDB framework which provides a nice Swift/Objective-C wrapper around SQLite. FMDB can be installed with Carthage or CocoaPods, whichever you prefer.

Searchables

After inserting FMDB into our project we have to define some kind of interface for searching. To be able to index any kind of Core Data model we need a couple of extra properties, so let's introduce a protocol for this.

protocol Searchable {
    /// A list of strings that should be indexed.
    var searchableStrings: [String] { get }

    /// An NSURL that can uniquely identify this searchable object.
    var URIRepresentation: NSURL { get }
}

The Searchable protocol will tell our search index what strings should be indexed for our model. Furthermore it provides an URI which is a unique identifier for our model. This means we can fetch the model without knowing the entity. The fetching mechanism provided below can even be extended to fetch non-Core Data objects, so in theory it can be used for anything.

To use the Searchable protocol with NSManagedObject without writing too much code I made a protocol extension that provides the value for URIRepresentation.

extension Searchable where Self: NSManagedObject {
    var URIRepresentation: NSURL {
        return objectID.URIRepresentation()
    }
}

Now any NSManagedObject can be made searchable by adopting this protocol and providing the searchableStrings. An example will be given below.

The search index

In the previous section we've defined what it means to be a searchable object, but we still have to create the full text search index. For this I made the following class:

class SearchIndex {
    let databaseQueue: FMDatabaseQueue

    init() {
        let path = NSFileManager.defaultManager().URLsForDirectory(.LibraryDirectory, inDomains: .UserDomainMask)[0].URLByAppendingPathComponent("SearchIndex.sqlite").path!
        databaseQueue = FMDatabaseQueue(path: path)

        databaseQueue.inTransaction { database, rollback in
            do {
                try database.executeUpdate("CREATE VIRTUAL TABLE IF NOT EXISTS documents USING fts4(tokenize=unicode61, identifier, contents);", values: [])
            } catch {
                print("Warning: Search Index create failed: \(error)")
            }
        }
    }
}

Upon instantiation the SearchIndex class creates or opens a SearchIndex.sqlite file in the app's library directory. It then creates the full text search table if it doesn't exist already. This table is a virtual table, which means that it is actually a group of tables behind the scenes, but we don't have to worry about that.

Now we have an index, but we still have to insert something into it. So let's create a method for this.

/// Insert the searchable object into the index. Updates the searchable if it was already indexed.
func insertSearchable(object: Searchable) {
    databaseQueue.inDatabase { database in
        do {
            try database.executeUpdate("DELETE FROM documents WHERE identifier = ?;", values: [object.URIRepresentation.absoluteString])
            try database.executeUpdate("INSERT INTO documents (identifier, contents) VALUES(?, ?);", values: [object.URIRepresentation.absoluteString, object.searchableStrings.joinWithSeparator(" ")])
        } catch {
            print("Warning: Search Index insert failed: \(error)")
        }
    }
}

The insertSearchable: method inserts a Searchable into the index. Because the FTS engine in SQLite does not support INSERT OR REPLACE we have to remove any existing model from the index first. The automatic replacement of models becomes very useful later on.

Deleting an object from the index is done in the same way:

/// Delete the searchable object from the index.
func deleteSearchable(object: Searchable) {
    databaseQueue.inDatabase { database in
        do {
            try database.executeUpdate("DELETE FROM documents WHERE identifier = ?;", values: [object.URIRepresentation.absoluteString])
        } catch {
            print("Warning: Search Index insert failed: \(error)")
        }
    }
}

Now the only thing that remains is to search for objects. This can be done using the MATCH operator in the following way:

func searchForURIsWithQuery(query: String) -> [NSURL] {
    var objects = [NSURL]()

    databaseQueue.inDatabase { database in
        let results: FMResultSet

        do {
            results = try database.executeQuery("SELECT identifier FROM documents WHERE contents MATCH ?;", values: [query])
        } catch {
            print("Warning: Search Index query failed: \(error)")
            return
        }

        while results.next() {
            objects.append(NSURL(string: results.stringForColumn("identifier"))!)
        }
    }

    return objects
}

The searching itself is actually pretty straight forward. We perform a SELECT query on the virtual table and collect the results in an array which we return. If you want you can improve this method by adding relevance sorting and whatever you like.

When using the query method given above we are left with an array of NSURL instances. We will still have to convert these into NSManagedObject instances (or anything else if you're using some other kind of storage). For this I provide another method that searches directly for Core Data models:

func searchForManagedObjectsWithQuery(query: String, inManagedObjectContext managedObjectContext: NSManagedObjectContext) -> [NSManagedObject] {
    return searchForURIsWithQuery(query).flatMap { uri in
        if let objectID = managedObjectContext.persistentStoreCoordinator?.managedObjectIDForURIRepresentation(uri) {
            return managedObjectContext.objectWithID(objectID)
        } else {
            return nil
        }
    }
}

And that is the entire search index class.

Usage

Now we have our search index ready to use. Let's use it on this example entity:

class Item: NSManagedObject {
    @NSManaged var name: String?
    @NSManaged var description: String?
}

First we have to make the Item searchable by adopting the Searchable protocol.

extension item: Searchable {
    var searchableStrings: [String] {
        return [
            name,
            description
        ].flatMap { $0 }
    }
}

Now we can use it with a search index:

let item: Item // some Item created elsewhere

let index = SearchIndex()

// Insert the item into the index
index.insertSearchable(item)

// Remove the item from the index
index.removeSearchable(item)

// Search the index
let results = index.searchForManagedObjectsWithQuery("hello", inManagedObjectContext: managedObjectContext)

Note that results contains an array of NSManagedObject instances, because our index is capable of searching for multiple classes. This means that you might have to use if let binding to check for the proper types.

Final thoughts

Here I've shown you my approach to adding full text search to a Core Data app. This approach is very flexible because it can be used to search for multiple models simultaneously, and can easily be adapted to suit your specific needs.

If you've enjoyed this post please follow me on Twitter or Facebook. I'd really appreciate it!