Fediverse Search
I've been talking with @GORF about doing some contract work to improve search in Pleroma. Below are my thoughts on search, and some proposals for actions to take. I decided to share this publicly since it's an open-source project, and if we take one of these paths we'll eventually want our changes merged upstreamed.
Fediverse Search
Users join social media sites to blog, network, and fulfill the basic human need to socialize. For newcomers it can be difficult to find friends and discover content that pertains to their interests. Search is one way to improve discoverability. Search isn't just useful to new users, but also to veterans who are trying to find "that one post from 2 weeks ago" or look up current discussion around a particular topic.
The current search implementation has 2 major areas for improvement:
- Content from outside your instance cannot be searched unless it has been federated.
- Search is slow, limited to 20 results, and cannot be filtered.
Note: artificial intelligence is out of scope for now. Maybe in the future we can search for "cat pics" to show actual cat pictures, but for now we'll accept that results are literal.
1. Content from outside your instance cannot be searched unless it has been federated
The Fediverse is expansive, but your instance only knows part of it. Usually your instance only knows about people your users follow, and people those users boost. This impacts search, because users of an instance can only search for content known to the instance.
Proposed solution: search providers
A search provider would be a separate piece of software whose job is to scrape the Fediverse and accumulate as much content as possible. The search provider could be self-hosted like any other free software. Pleroma could be configured to use an external search provider instead of its own database for search results.
It's important to keep the search provider a separate piece of software to not bloat the main Pleroma database, and so multiple servers could share the same search provider. It would implement a documented API so anyone could implement a search provider in any programming language, and other Fediverse backends could add support for search providers, too.
2. Search is slow, limited to 20 results, and cannot be filtered
Ignoring the pains of federation, once there is a lot of searchable content on an instance, the user experience is not very good. Search feels needlessly restrictive, limiting the results to a single page with no way to view more results. It's slow, and it's almost impossible to find "that one post from 2 weeks ago," much less follow a discussion.
Proposed solution: a new search API
We'll design a new search interface that allows endless scrolling. We will support filtering posts by user, hashtag, date range, and instance domain. To support this interface we'll code new API endpoints into Pleroma BE.
Proposals
So where do we go from here? Here are some possible plans.
120 hrs: Federated search
I'll implement search providers to allow searching the broader Fediverse.
- Develop (or adapt) a search provider implementation.
- Expose a documented API.
- Adapt Pleroma to incorporate search providers.
50 hrs: Basic endless scrolling search
I will design a new search UI with endless scrolling.
- Design a new interface.
- Develop an API endpoint for it in Pleroma.
- Adapt Soapbox FE to work with the new API.
60 hrs: Endless scrolling search with advanced filters
I will design a new search UI with endless scrolling and advanced filtering.
- Design a new interface.
- Develop an API endpoint for it in Pleroma.
- Add the ability to filter by username, hashtag, date range, and instance domain.
- Adapt Soapbox FE to work with the new API.