Skip to content

GitLab

pleroma
pleroma
Merged
Opened by href@hrefMaintainer1 of 2 tasks completed1/2 tasks

Flake Ids for Users and Activities

Closes #450 (closed).

Introduces Flake IDs (decentralized k-ordered ids). They consist of a 64 bit timestamp, 48 bit worker id (takes a random mac address on app start), 16 bit sequence, rendered as a base62 string, stored with the postgres uuid type.

This preserves the possibility to do keysort pagination (as Mast & Twitter API does) while keeping IDs large enough without requiring any coordination (if and only if worker ids are different, and time does not goes backward).

This should also allow us to import previous statuses while keeping ordering by implementing a way to generate Flakes with a previous timestamp (there's no API for that, but I plan to).

Old integers IDs are preserved by cheating and converting the IDs to 64 bits of 0 + 64 bits integer. This is handled on the application side, and theses fake IDs are only generated by the migration. The migration is also entierly done by a pure UPDATE statement so it is quite fast. (Thanks, @mloftis)

Removes a restriction (some queries only used the last 1000 activities). This shouldn't cause any issues as the indexes are correct now, tried on Pleroma.fr without this restriction since two weeks without problems.

This changes the type of id in Twitter API to integer to string. There's no other way-- even if we went to use an integer-based solution such as centralized Snowflakes, it would already bump to 64bit integers and probably break, either on staticly typed apps or in Javascript apps.

However, this causes two issues on the Pleroma FE:

  • it does client-side ordering, and it breaks. The Base62 string representation of the Flake should be sortable, so I don't see why it fucks up the ordering. I tried to look up into the FE code but JS isn't definitely my thing anymore
  • the /notice/:id page fucks up and reloads the API in a loop; couldn't figure out why either.

Mastodon FE is OK, Mast OK.

Tested Clients

  • Mast: OK
  • Roma: OK
  • Tusky: OK
  • Amaroq: OK
  • Mastalab: OK~, some issues being fixed by the author
  • Toot!: NOT OK, apparently relies on IDs being string-like with a 64bit integer inside
  • Subway Tooter: NOT OK, same issues as Toot, but author is fixing it.

TODO

  • waiting for a proper MR in Pleroma FE
  • conversion of older pinned status IDs in user info jsonb
Edited by Hakaba Hitoyo