Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
pleroma
pleroma
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 406
    • Issues 406
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 59
    • Merge Requests 59
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Pleroma
  • pleromapleroma
  • Issues
  • #872

Closed
Open
Opened May 08, 2019 by feld@feldMaintainer

Rework S3 bucket storage hierarchy / strategy?

After discussing the Pleroma S3 usage at work, we have some criticisms for long term archival of these objects especially around backup/restore. We have experience with S3 buckets that have millions of objects and many operations are painfully slow at that size.

Example of a good hierarchy:

        "idx": "t/t/w/ttwi3eda/poster/4.jpg",
        "idx": "t/t/z/ttz6qsiu/small.jpg",
        "idx": "t/u/3/tu3zixdp/large.jpg",
        "idx": "t/u/6/tu68dlz3/large.jpg",
        "idx": "t/u/7/tu7j1i32/large.jpg",
        "idx": "t/u/c/tucz706h/medium.jpg",
        "idx": "t/u/j/tuj95zf9/thumb.jpg",
        "idx": "t/u/m/tumujmfo/poster/5.jpg",

Also suggested condensing our filenames from sha hashes to something denser. "base58 instead of base16" is a recommendation as well.

Edited May 08, 2019 by feld
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: pleroma/pleroma#872