API fork #49

Open
opened 2020-09-19 18:09:25 +00:00 by hj · 11 comments
Member

As it was discussed in the IRC, we most likely want to fork MastoAPI into our own.

Reasons:

  • We would gain control over our own API and can change it the way we want, not just follow whatever Mastodon is doing.
  • Adding more extensions will be much easier, since we won't claim compatibility with clients not explicitly supporting Pleroma
  • We will still support vanilla MastoAPI (+ extensions) for mastodon and legacy clients
  • Maintaining compatibility with older/newer versions of clients/FEs will be easier since we would be able to version our API

How to do it (proposal):

Currently we have /api/v1/ and /api/v2/, as well as /api/pleroma/. We should either make /api/pleroma/v1 or start using version-requesting headers - see https://restfulapi.net/versioning/ . All mastoapi and pleroma endpoints should be copied there, like so:

  • /api/v1/search -> /api/pleroma/v1/search
  • /api/v2/search -> /api/pleroma/v1/search-v2 - all v2 or newer endpoints should have a suffix instead of becoming v2 - remember, it is our version.
  • /api/pleroma/frontend_configurations -> /api/pleroma/v1/frontend_configurations
  • /api/v1/streaming -> /api/pleroma/v1/streaming

The point is - all existing API should remain more-or-less untouched. Adding new features or breaking behavior means we need to create v2 endpoint.

We don't have create /pleroma/v1 immediately but instead create it when we want to make some change into existing API, i.e. consider existing API a "v0"

It doesn't necessary have to be /pleroma/v1, we can use /pleroma-v1, /pv1 or anything else. To avoid confusion we should settle on one versioning method only, i.e. not mixing prefixing /vN and accept-version: 1 header or such.

Examples of application and profits:

Say we want to add Deletes into non-streaming API so that clients not supporting websockets (mobile) and PleromaFE in poll mode (either disabled by user or recovering from WS connection failure) can still receive deletes.

we copy /api/v1/timelines/home (and possibly others) into /api/pleroma/v1/timelines/home:

  • New clients can switch to use new API and support deletes that way.
  • Old clients are unaffected
  • We don't have to test each and every client software in existence to see if it breaks because of it.
  • We don't have to devise a way to extend existing API to make it compatible with everything.

Additional thoughts

Supporting new API is cool, but it also means client software targeting it loses compatibility with older servers, and some developer would prefer to avoid that and would try to figure out if given server supports certain endpoints or not, we should provide an easy way of knowing which endpoints are available, i.e. additional field in /api/v1/instance or similar.

Versioning entire API or just endpoints? Synchronize versions? Semver?

In general there doesn't seem to be much profit in versioning entire API - a lot of endpoints will be identical across versions anyway, versioning endpoints separately also doesn't always make sense since some endpoints are meant to be used together. I would suggest versioning endpoints separately but endpoints meant to be used together should all have matching versions, even if it means skipping a version (i.e. going v1 -> v3), it would sorta be a "version of a feature". I don't see much need for semver either since any number change already implies incompatibility, and i would just prefer using single number.

As it was discussed in the IRC, we most likely want to fork MastoAPI into our own. ## Reasons: - We would gain control over our own API and can change it the way **we** want, not just follow whatever Mastodon is doing. - Adding more extensions will be much easier, since we won't claim compatibility with clients not explicitly supporting Pleroma - We will still support vanilla MastoAPI (+ extensions) for mastodon and legacy clients - Maintaining compatibility with older/newer versions of clients/FEs will be easier since we would be able to version our API ## How to do it (proposal): Currently we have `/api/v1/` and `/api/v2/`, as well as `/api/pleroma/`. We should either make `/api/pleroma/v1` or start using version-requesting headers - see https://restfulapi.net/versioning/ . All mastoapi and pleroma endpoints should be copied there, like so: * `/api/v1/search` -> `/api/pleroma/v1/search` * `/api/v2/search` -> `/api/pleroma/v1/search-v2` - all v2 or newer endpoints should have a suffix instead of becoming v2 - remember, it is **our** version. * `/api/pleroma/frontend_configurations` -> `/api/pleroma/v1/frontend_configurations` * `/api/v1/streaming` -> `/api/pleroma/v1/streaming` The point is - all existing API should remain more-or-less untouched. Adding new features or breaking behavior means we need to create v2 endpoint. We don't have create `/pleroma/v1` immediately but instead create it when we want to make some change into existing API, i.e. consider existing API a "v0" It doesn't necessary have to be `/pleroma/v1`, we can use `/pleroma-v1`, `/pv1` or anything else. To avoid confusion we should settle on one versioning method only, i.e. not mixing prefixing `/vN` and `accept-version: 1` header or such. ## Examples of application and profits: Say we want to add *Deletes* into non-streaming API so that clients not supporting websockets (mobile) and PleromaFE in poll mode (either disabled by user or recovering from WS connection failure) can still receive deletes. we copy `/api/v1/timelines/home` (and possibly others) into `/api/pleroma/v1/timelines/home`: * New clients can switch to use new API and support deletes that way. * Old clients are unaffected * We don't have to test each and every client software in existence to see if it breaks because of it. * We don't have to devise a way to extend existing API to make it compatible with everything. ## Additional thoughts Supporting new API is cool, but it also means client software targeting it loses compatibility with older servers, and some developer would prefer to avoid that and would try to figure out if given server supports certain endpoints or not, we should provide an easy way of knowing which endpoints are available, i.e. additional field in `/api/v1/instance` or similar. ### Versioning entire API or just endpoints? Synchronize versions? Semver? In general there doesn't seem to be much profit in versioning entire API - a lot of endpoints will be identical across versions anyway, versioning endpoints separately also doesn't always make sense since some endpoints are meant to be used together. I would suggest versioning endpoints separately but endpoints meant to be used together should all have matching versions, even if it means skipping a version (i.e. going v1 -> v3), it would sorta be a "version of a feature". I don't see much need for semver either since any number change already implies incompatibility, and i would just prefer using single number.
Member

Currently we have /api/v1/ and /api/v2/, as well as /api/pleroma/. We should either make /api/pleroma/v1 or start using version-requesting headers

There is /api/v1/pleroma/, it's used in chats for example.

Overall, I don't know whether we have enough extension ideas fundamentally incompatible with mastodon clients to warrant the complexity to maintain this. If we can devise a plan to avoid code/openapi spec duplication for extended mastoapi endpoints (probably some clever macros that duplicate mastoapi routing table, but unlike regular mastoapi, call controller actions with a "pleroma_extension" flag or something), then I don't mind having it though.

> Currently we have `/api/v1/` and `/api/v2/`, as well as `/api/pleroma/`. We should either make `/api/pleroma/v1` or start using version-requesting headers There is `/api/v1/pleroma/`, it's used in chats for example. Overall, I don't know whether we have enough extension ideas fundamentally incompatible with mastodon clients to warrant the complexity to maintain this. If we can devise a plan to avoid code/openapi spec duplication for extended mastoapi endpoints (probably some clever macros that duplicate mastoapi routing table, but unlike regular mastoapi, call controller actions with a "pleroma_extension" flag or something), then I don't mind having it though.
Author
Member

code duplication is probably a requirement, not something we want to avoid, since the idea is to lock existing behavior so that changes in v2 would not affect v1. At least that would be ideally.

I have tons of ideas that wouldn't me compatible with existing API or that would benefit from versioning in other ways (for instance - sending fake notification to vanilla MastoAPI to tell user they have unread chats, changing emoji API to have better structure + stickers support without breaking anything)

Our API is messy, there are endpoints that are kinda qvitterAPIish (i.e notifications read) but aren't really. Slowly moving it all into single root (i.e. /api/pleorma/vN) would help.

Also: even if we don't have ideas - mastodon probably does, they already have /v2/search and /v2/media and they can always add new API, even if it would conflict with our code - we already have two emoji apis for instance.

code duplication is probably a requirement, not something we want to avoid, since the idea is to lock existing behavior so that changes in v2 would not affect v1. At least that would be ideally. I have tons of ideas that wouldn't me compatible with existing API or that would benefit from versioning in other ways (for instance - sending fake notification to vanilla MastoAPI to tell user they have unread chats, changing emoji API to have better structure + stickers support without breaking anything) Our API is messy, there are endpoints that are kinda qvitterAPIish (i.e notifications read) but aren't really. Slowly moving it all into single root (i.e. `/api/pleorma/vN`) would help. Also: even if we don't have ideas - mastodon probably does, they already have `/v2/search` and `/v2/media` and they can always add new API, even if it would conflict with our code - we already have two emoji apis for instance.
Author
Member

It's more work either way you look at it - continuing to support MastoAPI+extensions means API will get more and more messy with time due to our work and outside factors (mastodon), it also means more work for API consumers (clients) to figure out how to use it - both in terms of documentation AND in terms of "does this server version supports thing i need or not". So far it's been more or less ok (it wasn't) because only ever PleromaFE used extensions, now we have Soapbox and Husky using our extensions so unless we want our users to suffer and telling them "just update lol" as well as forcing FE/Client devs suffer, we should implement versioning.

It's more work either way you look at it - continuing to support MastoAPI+extensions means API will get more and more messy with time due to our work and outside factors (mastodon), it also means more work for API consumers (clients) to figure out how to use it - both in terms of documentation AND in terms of "does this server version supports thing i need or not". So far it's been more or less ok (it wasn't) because only ever PleromaFE used extensions, now we have Soapbox and Husky using our extensions so unless we want our users to suffer and telling them "just update lol" as well as forcing FE/Client devs suffer, we should implement versioning.
Member

code duplication is probably a requirement, not something we want to avoid, since the idea is to lock existing behavior so that changes in v2 would not affect v1. At least that would be ideally.

Sure, not against duplication on v2. What I mean is, you propose leaving existing pleroma extensions untouched, but duplicating pleroma-extended endpoints under pleroma namespace as well. I don't want duplication in that case.

I have tons of ideas that wouldn't me compatible with existing API or that would benefit from versioning in other ways (for instance - sending fake notification to vanilla MastoAPI to tell user they have unread chats, changing emoji API to have better structure + stickers support without breaking anything)

This makes sense to me, yeah.

Also: even if we don't have ideas - mastodon probably does, they already have /v2/search and /v2/media and they can always add new API, even if it would conflict with our code - we already have two emoji apis for instance.

I get what you mean, but we have 2 emoji APIs because pleroma-fe didn't want to use the mastodon one during qvitterapi days (because of different auth?), unless I am misremembering something.

Our API is messy, there are endpoints that are kinda qvitterAPIish (i.e notifications read) but aren't really. Slowly moving it all into single root (i.e. /api/pleorma/vN) would help.

This is true. Notification read api has a replacement on mastoapi side called markers. I once tried to port pleroma-fe to use it, but turns out it has flaws (updating a marker only updates the marker, doesn't actually set notifs as seen).

Okay, this makes more sense to me now. I am on board with it.

> code duplication is probably a requirement, not something we want to avoid, since the idea is to lock existing behavior so that changes in v2 would not affect v1. At least that would be ideally. Sure, not against duplication on v2. What I mean is, you propose leaving existing pleroma extensions untouched, but duplicating pleroma-extended endpoints under pleroma namespace as well. I don't want duplication in that case. > I have tons of ideas that wouldn't me compatible with existing API or that would benefit from versioning in other ways (for instance - sending fake notification to vanilla MastoAPI to tell user they have unread chats, changing emoji API to have better structure + stickers support without breaking anything) This makes sense to me, yeah. > Also: even if we don't have ideas - mastodon probably does, they already have `/v2/search` and `/v2/media` and they can always add new API, even if it would conflict with our code - we already have two emoji apis for instance. I get what you mean, but we have 2 emoji APIs because pleroma-fe didn't want to use the mastodon one during qvitterapi days (because of different auth?), unless I am misremembering something. > Our API is messy, there are endpoints that are kinda qvitterAPIish (i.e notifications read) but aren't really. Slowly moving it all into single root (i.e. `/api/pleorma/vN`) would help. This is true. Notification read api has a replacement on mastoapi side called markers. I once tried to port pleroma-fe to use it, but turns out it has flaws (updating a marker only updates the marker, doesn't actually set notifs as seen). Okay, this makes more sense to me now. I am on board with it.

I like the idea of not being beholden to whatever Mastodon says. It seems possible to just add /api/pleroma/vN endpoints whenever you need them tho. In fact I thought this was how it already is.

It seems fine to me to add /api/pleroma/v1/notifications to provide a different behavior. What doesn't make sense to me is to duplicate all the existing controllers without changing their behavior.

I like the idea of not being beholden to whatever Mastodon says. It seems possible to just add `/api/pleroma/vN` endpoints whenever you need them tho. In fact I thought this was how it already is. It seems fine to me to add `/api/pleroma/v1/notifications` to provide a different behavior. What doesn't make sense to me is to duplicate all the existing controllers without changing their behavior.
Author
Member

but duplicating pleroma-extended endpoints under pleroma namespace as well. I don't want duplication in that case.

not sure if I understand... What exactly do you want to avoid? Give me an example.

because pleroma-fe didn't want to use the mastodon one during qvitterapi days (because of different auth?), unless I am misremembering something.

No, because Mastodon added custom emoji support much much later, our emoji api is the original one, and i think at the time of migration there were no MastoAPI emoji endpoint at all. This could happen with any other feature we have, i.e. if mastodon starts supporting chats they probably won't be using /v1/pleroma/ and maybe will be using their own structures as well. There were simply no need to switch emoji APIs so far.

>but duplicating pleroma-extended endpoints under pleroma namespace as well. I don't want duplication in that case. not sure if I understand... What exactly do you want to avoid? Give me an example. >because pleroma-fe didn't want to use the mastodon one during qvitterapi days (because of different auth?), unless I am misremembering something. No, because Mastodon added custom emoji support much much later, our emoji api is the original one, and i think at the time of migration there were no MastoAPI emoji endpoint at all. This could happen with any other feature we have, i.e. if mastodon starts supporting chats they probably won't be using `/v1/pleroma/` and maybe will be using their own structures as well. There were simply no need to switch emoji APIs so far.
Member

not sure if I understand... What exactly do you want to avoid? Give me an example.

In the issue description:

Currently we have /api/v1/ and /api/v2/, as well as /api/pleroma/. We should either make /api/pleroma/v1 or start using version-requesting headers - see https://restfulapi.net/versioning/ . All mastoapi and pleroma endpoints should be copied there, like so:

  • /api/v1/search -> /api/pleroma/v1/search

What I want to avoid is having to manually duplicate /api/v1/search to /api/pleroma/v1/search with all the docs and keep them in sync even if we don't have any extensions there. But that's really just implementation details.

> not sure if I understand... What exactly do you want to avoid? Give me an example. In the issue description: > Currently we have `/api/v1/` and `/api/v2/`, as well as `/api/pleroma/`. We should either make `/api/pleroma/v1` or start using version-requesting headers - see https://restfulapi.net/versioning/ . All mastoapi and pleroma endpoints should be copied there, like so: > * `/api/v1/search` -> `/api/pleroma/v1/search` What I want to avoid is having to manually duplicate `/api/v1/search` to `/api/pleroma/v1/search` with all the docs and keep them in sync even if we don't have any extensions there. But that's really just implementation details.
Member

Also not sure about this one, we probably can automate rewrites from /api/pleroma/v1/endpoint -> /api/v1/endpoint for mastoapi endpoints we don't extend. But why do that when clients can just use /api/v1/endpoint and then switch endpoints if we do start to extend it

Also not sure about this one, we probably can automate rewrites from `/api/pleroma/v1/endpoint` -> `/api/v1/endpoint` for mastoapi endpoints we don't extend. But why do that when clients can just use `/api/v1/endpoint` and then switch endpoints if we do start to extend it
Author
Member

yeah if we don't change anything we don't have to copy it. copying and duplicating everything prematurely could be done but it's only an insurance against accidental unintentional changes, so it's not really necessary.

yeah if we don't change anything we don't have to copy it. copying and duplicating everything prematurely could be done but it's only an insurance against accidental unintentional changes, so it's not really necessary.
Member

I'm mostly noted it in pleroma/pleroma#6388. Thanks for moving discussion to another place that I didn't knew about!

From my point, Pleroma needs to continue support Mastodon API but freeze it on 2.7.0 API level and maybe partially upgrade it only if some new feature gets nicely represented by already existing new Mastodon's APIs.

But as I noted in my PR, it should never change it's behavior, only new fields or new optional parameters could be added. If endpoint must change it's behavior for REASONS, like ones that @hj noted, it indeed can be forked to /api/v1/pleroma/$path. Yes, that way, because we already have /api/v1/pleroma/chats, /api/pleroma/admin, /api/v1/pleroma/admin and adding /api/pleroma/v1 will turn it into more chaos that it already is.

Although by applying rules that I suggested it may start to have a meaning. Everything that doesn't have vN in it's path is unversioned and is subject to change. Everything that is accessed through /api/v1 will never change it's behavior. It can be extended by following guidelines I wrote, but will not change in a software breaking way.

Also, why I think optional fields/parameters in Mastodon API instead of special endpoints is a total win? Because it could help with patches to upstream Mastodon apps. It's easier to get support for Pleroma-specific features or bugfixes to accepted, if it would be 10-20 lines of code, instead of copied and pasted code just to support "not boring" Pleroma API endpoints that do the same thing that existing Mastodon API.

Oh, and yes, if Mastodon API just will break itself(deprecation for example), there is no point in doing the same if it already works for us.

I'm mostly noted it in https://git.pleroma.social/pleroma/pleroma/pulls/6388. Thanks for moving discussion to another place that I didn't knew about! From my point, Pleroma needs to continue support Mastodon API but freeze it on 2.7.0 API level and maybe partially upgrade it only if some new feature gets nicely represented by already existing new Mastodon's APIs. But as I noted in my PR, it should never change it's behavior, only new fields or new optional parameters could be added. If endpoint must change it's behavior for REASONS, like ones that @hj noted, it indeed can be forked to `/api/v1/pleroma/$path`. Yes, that way, because we already have `/api/v1/pleroma/chats`, `/api/pleroma/admin`, `/api/v1/pleroma/admin` and adding `/api/pleroma/v1` will turn it into more chaos that it already is. Although by applying rules that I suggested it may start to have a meaning. Everything that doesn't have vN in it's path is unversioned and is subject to change. Everything that is accessed through /api/v1 will never change it's **behavior**. It can be extended by following guidelines I wrote, but will not change in a software breaking way. Also, why I think optional fields/parameters in Mastodon API instead of special endpoints is a total win? Because it could help with patches to upstream Mastodon apps. It's easier to get support for Pleroma-specific features or bugfixes to accepted, if it would be 10-20 lines of code, instead of copied and pasted code just to support "not boring" Pleroma API endpoints that do the same thing that existing Mastodon API. Oh, and yes, if Mastodon API just will break itself(deprecation for example), there is no point in doing the same if it already works for us.
Member

Also, somewhat unrelated to this discussion.

The only thing we missing compared to Mastodon API docs is information when some API extension(endpoint, parameter or field in schema) was actually added.

Most things that are noted in differences_in_mastoapi.md just never implemented by anyone, like previews. And it's impossible to know when this feature was added without digging into git-log.

Also, somewhat unrelated to this discussion. The only thing we missing compared to Mastodon API docs is information when some API extension(endpoint, parameter or field in schema) was actually added. Most things that are noted in `differences_in_mastoapi.md` just never implemented by anyone, like previews. And it's impossible to know when this feature was added without digging into `git-log`.
Sign in to join this conversation.
No labels
BE
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
pleroma/pleroma-meta#49
No description provided.