Hotfix hashtag timeline performance
This contains a benchmark that generates a somewhat realistic distribution of hashtags and queries the hashtag timeline.
The result of the benchmark shows that the problem isn't finding the tagged posts, but the join / sorting. I benchmarked for both current develop and the activity-object-id branch, the performance level is the same.
I added a change that makes the hashtag timeline unsorted, which is a hotfix for the problem of hashtag timelines not working at all.
I also backported this to stable and tried it on soykaf, it also fixed the hashtag timelines there.
Benchmarks
develop
RUM enabled: false
09:34:23.361 [warn] Found files in the emoji folder. These will be ignored, please move them to a subdirectory
Found files: test.png
Found 200000 activities, won't generate new ones
Database contains 55 posts tagged with tag_0
Database contains 317 posts tagged with tag_1
Database contains 626 posts tagged with tag_2
Database contains 1137 posts tagged with tag_3
Database contains 1672 posts tagged with tag_4
Database contains 2545 posts tagged with tag_5
Database contains 3438 posts tagged with tag_6
Database contains 4515 posts tagged with tag_7
Database contains 5795 posts tagged with tag_8
Database contains 6887 posts tagged with tag_9
Database contains 8518 posts tagged with tag_10
Database contains 9998 posts tagged with tag_11
Database contains 11581 posts tagged with tag_12
Database contains 13743 posts tagged with tag_13
Database contains 15596 posts tagged with tag_14
Database contains 17848 posts tagged with tag_15
Database contains 20061 posts tagged with tag_16
Database contains 22534 posts tagged with tag_17
Database contains 25236 posts tagged with tag_18
Database contains 27898 posts tagged with tag_19
Database contains 0 posts tagged with tag_20
Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.
Operating System: Linux
CPU Information: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
Number of Available Cores: 8
Available memory: 23.24 GB
Elixir 1.9.4
Erlang 22.1.8
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: For #tag_0, For #tag_1, For #tag_2, For #tag_3, For #tag_4, For #tag_5, For #tag_6, For #tag_7, For #tag_8, For #tag_9, For #tag_10, For #tag_11, For #tag_12, For #tag_13, For #tag_14, For #tag_15, For #tag_16, For #tag_17, For #tag_18, For #tag_19, For #tag_20
Estimated total run time: 2.45 min
Benchmarking Hashtag fetching with input For #tag_0...
Benchmarking Hashtag fetching with input For #tag_1...
Benchmarking Hashtag fetching with input For #tag_2...
Benchmarking Hashtag fetching with input For #tag_3...
Benchmarking Hashtag fetching with input For #tag_4...
Benchmarking Hashtag fetching with input For #tag_5...
Benchmarking Hashtag fetching with input For #tag_6...
Benchmarking Hashtag fetching with input For #tag_7...
Benchmarking Hashtag fetching with input For #tag_8...
Benchmarking Hashtag fetching with input For #tag_9...
Benchmarking Hashtag fetching with input For #tag_10...
Benchmarking Hashtag fetching with input For #tag_11...
Benchmarking Hashtag fetching with input For #tag_12...
Benchmarking Hashtag fetching with input For #tag_13...
Benchmarking Hashtag fetching with input For #tag_14...
Benchmarking Hashtag fetching with input For #tag_15...
Benchmarking Hashtag fetching with input For #tag_16...
Benchmarking Hashtag fetching with input For #tag_17...
Benchmarking Hashtag fetching with input For #tag_18...
Benchmarking Hashtag fetching with input For #tag_19...
Benchmarking Hashtag fetching with input For #tag_20...
##### With input For #tag_0 #####
Name ips average deviation median 99th %
Hashtag fetching 162.15 6.17 ms ±26.61% 5.65 ms 13.68 ms
##### With input For #tag_1 #####
Name ips average deviation median 99th %
Hashtag fetching 82.07 12.18 ms ±16.26% 11.86 ms 20.73 ms
##### With input For #tag_2 #####
Name ips average deviation median 99th %
Hashtag fetching 53.65 18.64 ms ±6.35% 18.43 ms 21.98 ms
##### With input For #tag_3 #####
Name ips average deviation median 99th %
Hashtag fetching 32.45 30.82 ms ±7.30% 30.44 ms 38.60 ms
##### With input For #tag_4 #####
Name ips average deviation median 99th %
Hashtag fetching 21.45 46.61 ms ±10.82% 45.54 ms 64.18 ms
##### With input For #tag_5 #####
Name ips average deviation median 99th %
Hashtag fetching 16.11 62.08 ms ±4.38% 61.29 ms 68.21 ms
##### With input For #tag_6 #####
Name ips average deviation median 99th %
Hashtag fetching 12.25 81.62 ms ±5.06% 80.74 ms 91.05 ms
##### With input For #tag_7 #####
Name ips average deviation median 99th %
Hashtag fetching 8.47 118.07 ms ±10.50% 113.82 ms 146.66 ms
##### With input For #tag_8 #####
Name ips average deviation median 99th %
Hashtag fetching 6.84 146.23 ms ±9.94% 142.76 ms 202.26 ms
##### With input For #tag_9 #####
Name ips average deviation median 99th %
Hashtag fetching 6.37 157.04 ms ±3.87% 154.72 ms 170.89 ms
##### With input For #tag_10 #####
Name ips average deviation median 99th %
Hashtag fetching 4.09 244.36 ms ±4.67% 240.71 ms 269.59 ms
##### With input For #tag_11 #####
Name ips average deviation median 99th %
Hashtag fetching 3.50 285.76 ms ±3.79% 282.21 ms 304.18 ms
##### With input For #tag_12 #####
Name ips average deviation median 99th %
Hashtag fetching 2.91 343.97 ms ±5.53% 338.32 ms 385.85 ms
##### With input For #tag_13 #####
Name ips average deviation median 99th %
Hashtag fetching 1.85 540.26 ms ±9.61% 529.33 ms 616.52 ms
##### With input For #tag_14 #####
Name ips average deviation median 99th %
Hashtag fetching 1.60 623.08 ms ±9.54% 633.49 ms 714.33 ms
##### With input For #tag_15 #####
Name ips average deviation median 99th %
Hashtag fetching 1.29 773.81 ms ±12.76% 751.77 ms 953.57 ms
##### With input For #tag_16 #####
Name ips average deviation median 99th %
Hashtag fetching 1.22 818.46 ms ±4.93% 801.99 ms 873.31 ms
##### With input For #tag_17 #####
Name ips average deviation median 99th %
Hashtag fetching 1.07 938.18 ms ±2.67% 939.52 ms 977.55 ms
##### With input For #tag_18 #####
Name ips average deviation median 99th %
Hashtag fetching 0.92 1.08 s ±11.94% 1.04 s 1.31 s
##### With input For #tag_19 #####
Name ips average deviation median 99th %
Hashtag fetching 0.87 1.15 s ±23.83% 1.01 s 1.57 s
##### With input For #tag_20 #####
Name ips average deviation median 99th %
Hashtag fetching 269.67 3.71 ms ±22.89% 3.53 ms 8.80 msactivity_and_object_id branch
RUM enabled: false
09:21:57.339 [warn] Found files in the emoji folder. These will be ignored, please move them to a subdirectory
Found files: test.png
Found 200000 activities, won't generate new ones
Database contains 55 posts tagged with tag_0
Database contains 280 posts tagged with tag_1
Database contains 602 posts tagged with tag_2
Database contains 1151 posts tagged with tag_3
Database contains 1772 posts tagged with tag_4
Database contains 2550 posts tagged with tag_5
Database contains 3474 posts tagged with tag_6
Database contains 4497 posts tagged with tag_7
Database contains 5614 posts tagged with tag_8
Database contains 6751 posts tagged with tag_9
Database contains 8451 posts tagged with tag_10
Database contains 10293 posts tagged with tag_11
Database contains 11659 posts tagged with tag_12
Database contains 13579 posts tagged with tag_13
Database contains 15675 posts tagged with tag_14
Database contains 17808 posts tagged with tag_15
Database contains 20250 posts tagged with tag_16
Database contains 22542 posts tagged with tag_17
Database contains 25049 posts tagged with tag_18
Database contains 27948 posts tagged with tag_19
Database contains 0 posts tagged with tag_20
Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.
Operating System: Linux
CPU Information: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
Number of Available Cores: 8
Available memory: 23.24 GB
Elixir 1.9.4
Erlang 22.1.8
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: For #tag_0, For #tag_1, For #tag_2, For #tag_3, For #tag_4, For #tag_5, For #tag_6, For #tag_7, For #tag_8, For #tag_9, For #tag_10, For #tag_11, For #tag_12, For #tag_13, For #tag_14, For #tag_15, For #tag_16, For #tag_17, For #tag_18, For #tag_19, For #tag_20
Estimated total run time: 2.45 min
Benchmarking Hashtag fetching with input For #tag_0...
Benchmarking Hashtag fetching with input For #tag_1...
Benchmarking Hashtag fetching with input For #tag_2...
Benchmarking Hashtag fetching with input For #tag_3...
Benchmarking Hashtag fetching with input For #tag_4...
Benchmarking Hashtag fetching with input For #tag_5...
Benchmarking Hashtag fetching with input For #tag_6...
Benchmarking Hashtag fetching with input For #tag_7...
Benchmarking Hashtag fetching with input For #tag_8...
Benchmarking Hashtag fetching with input For #tag_9...
Benchmarking Hashtag fetching with input For #tag_10...
Benchmarking Hashtag fetching with input For #tag_11...
Benchmarking Hashtag fetching with input For #tag_12...
Benchmarking Hashtag fetching with input For #tag_13...
Benchmarking Hashtag fetching with input For #tag_14...
Benchmarking Hashtag fetching with input For #tag_15...
Benchmarking Hashtag fetching with input For #tag_16...
Benchmarking Hashtag fetching with input For #tag_17...
Benchmarking Hashtag fetching with input For #tag_18...
Benchmarking Hashtag fetching with input For #tag_19...
Benchmarking Hashtag fetching with input For #tag_20...
##### With input For #tag_0 #####
Name ips average deviation median 99th %
Hashtag fetching 124.20 8.05 ms ±32.69% 7.25 ms 18.11 ms
##### With input For #tag_1 #####
Name ips average deviation median 99th %
Hashtag fetching 96.76 10.33 ms ±4.65% 10.23 ms 11.96 ms
##### With input For #tag_2 #####
Name ips average deviation median 99th %
Hashtag fetching 53.68 18.63 ms ±17.56% 17.78 ms 31.05 ms
##### With input For #tag_3 #####
Name ips average deviation median 99th %
Hashtag fetching 29.17 34.28 ms ±16.45% 32.05 ms 55.56 ms
##### With input For #tag_4 #####
Name ips average deviation median 99th %
Hashtag fetching 21.91 45.65 ms ±11.68% 43.76 ms 66.92 ms
##### With input For #tag_5 #####
Name ips average deviation median 99th %
Hashtag fetching 16.38 61.05 ms ±4.91% 60.87 ms 73.38 ms
##### With input For #tag_6 #####
Name ips average deviation median 99th %
Hashtag fetching 8.83 113.25 ms ±8.96% 113.46 ms 135.43 ms
##### With input For #tag_7 #####
Name ips average deviation median 99th %
Hashtag fetching 7.71 129.72 ms ±10.31% 132.90 ms 156.56 ms
##### With input For #tag_8 #####
Name ips average deviation median 99th %
Hashtag fetching 5.41 184.74 ms ±14.54% 174.73 ms 267.72 ms
##### With input For #tag_9 #####
Name ips average deviation median 99th %
Hashtag fetching 4.71 212.40 ms ±8.31% 208.08 ms 243.28 ms
##### With input For #tag_10 #####
Name ips average deviation median 99th %
Hashtag fetching 2.79 358.87 ms ±10.97% 342.69 ms 441.10 ms
##### With input For #tag_11 #####
Name ips average deviation median 99th %
Hashtag fetching 2.23 447.47 ms ±12.66% 424.29 ms 529.22 ms
##### With input For #tag_12 #####
Name ips average deviation median 99th %
Hashtag fetching 2.05 488.01 ms ±9.93% 473.63 ms 582.74 ms
##### With input For #tag_13 #####
Name ips average deviation median 99th %
Hashtag fetching 1.75 570.63 ms ±14.94% 593.28 ms 660.21 ms
##### With input For #tag_14 #####
Name ips average deviation median 99th %
Hashtag fetching 1.59 629.25 ms ±10.85% 628.15 ms 743.29 ms
##### With input For #tag_15 #####
Name ips average deviation median 99th %
Hashtag fetching 1.25 799.39 ms ±4.33% 804.37 ms 840.45 ms
##### With input For #tag_16 #####
Name ips average deviation median 99th %
Hashtag fetching 1.40 714.74 ms ±8.31% 719.51 ms 811.06 ms
##### With input For #tag_17 #####
Name ips average deviation median 99th %
Hashtag fetching 1.03 966.82 ms ±13.45% 988.76 ms 1098.87 ms
##### With input For #tag_18 #####
Name ips average deviation median 99th %
Hashtag fetching 1.01 994.90 ms ±8.18% 1018.15 ms 1078.57 ms
##### With input For #tag_19 #####
Name ips average deviation median 99th %
Hashtag fetching 0.83 1.20 s ±13.66% 1.14 s 1.40 s
##### With input For #tag_20 #####
Name ips average deviation median 99th %
Hashtag fetching 212.61 4.70 ms ±28.13% 4.20 ms 8.61 ms
This branch with skip_order set to true, so no ordering takes place
RUM enabled: false
11:33:34.362 [warn] Found files in the emoji folder. These will be ignored, please move them to a subdirectory
Found files: test.png
Found 200000 activities, won't generate new ones
Database contains 55 posts tagged with tag_0
Database contains 317 posts tagged with tag_1
Database contains 626 posts tagged with tag_2
Database contains 1137 posts tagged with tag_3
Database contains 1672 posts tagged with tag_4
Database contains 2545 posts tagged with tag_5
Database contains 3438 posts tagged with tag_6
Database contains 4515 posts tagged with tag_7
Database contains 5795 posts tagged with tag_8
Database contains 6887 posts tagged with tag_9
Database contains 8518 posts tagged with tag_10
Database contains 9998 posts tagged with tag_11
Database contains 11581 posts tagged with tag_12
Database contains 13743 posts tagged with tag_13
Database contains 15596 posts tagged with tag_14
Database contains 17848 posts tagged with tag_15
Database contains 20061 posts tagged with tag_16
Database contains 22534 posts tagged with tag_17
Database contains 25236 posts tagged with tag_18
Database contains 27898 posts tagged with tag_19
Database contains 0 posts tagged with tag_20
Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.
Operating System: Linux
CPU Information: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
Number of Available Cores: 8
Available memory: 23.24 GB
Elixir 1.9.4
Erlang 22.1.8
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: For #tag_0, For #tag_1, For #tag_2, For #tag_3, For #tag_4, For #tag_5, For #tag_6, For #tag_7, For #tag_8, For #tag_9, For #tag_10, For #tag_11, For #tag_12, For #tag_13, For #tag_14, For #tag_15, For #tag_16, For #tag_17, For #tag_18, For #tag_19, For #tag_20
Estimated total run time: 2.45 min
Benchmarking Hashtag fetching with input For #tag_0...
Benchmarking Hashtag fetching with input For #tag_1...
Benchmarking Hashtag fetching with input For #tag_2...
Benchmarking Hashtag fetching with input For #tag_3...
Benchmarking Hashtag fetching with input For #tag_4...
Benchmarking Hashtag fetching with input For #tag_5...
Benchmarking Hashtag fetching with input For #tag_6...
Benchmarking Hashtag fetching with input For #tag_7...
Benchmarking Hashtag fetching with input For #tag_8...
Benchmarking Hashtag fetching with input For #tag_9...
Benchmarking Hashtag fetching with input For #tag_10...
Benchmarking Hashtag fetching with input For #tag_11...
Benchmarking Hashtag fetching with input For #tag_12...
Benchmarking Hashtag fetching with input For #tag_13...
Benchmarking Hashtag fetching with input For #tag_14...
Benchmarking Hashtag fetching with input For #tag_15...
Benchmarking Hashtag fetching with input For #tag_16...
Benchmarking Hashtag fetching with input For #tag_17...
Benchmarking Hashtag fetching with input For #tag_18...
Benchmarking Hashtag fetching with input For #tag_19...
Benchmarking Hashtag fetching with input For #tag_20...
##### With input For #tag_0 #####
Name ips average deviation median 99th %
Hashtag fetching 191.46 5.22 ms ±12.58% 5.10 ms 8.57 ms
##### With input For #tag_1 #####
Name ips average deviation median 99th %
Hashtag fetching 186.06 5.37 ms ±7.21% 5.35 ms 6.73 ms
##### With input For #tag_2 #####
Name ips average deviation median 99th %
Hashtag fetching 179.81 5.56 ms ±6.59% 5.51 ms 6.69 ms
##### With input For #tag_3 #####
Name ips average deviation median 99th %
Hashtag fetching 171.22 5.84 ms ±9.07% 5.77 ms 8.06 ms
##### With input For #tag_4 #####
Name ips average deviation median 99th %
Hashtag fetching 163.24 6.13 ms ±7.52% 6.11 ms 7.37 ms
##### With input For #tag_5 #####
Name ips average deviation median 99th %
Hashtag fetching 159.06 6.29 ms ±6.55% 6.25 ms 7.38 ms
##### With input For #tag_6 #####
Name ips average deviation median 99th %
Hashtag fetching 151.31 6.61 ms ±7.48% 6.54 ms 7.91 ms
##### With input For #tag_7 #####
Name ips average deviation median 99th %
Hashtag fetching 141.73 7.06 ms ±7.16% 7.03 ms 8.38 ms
##### With input For #tag_8 #####
Name ips average deviation median 99th %
Hashtag fetching 135.55 7.38 ms ±7.63% 7.34 ms 9.11 ms
##### With input For #tag_9 #####
Name ips average deviation median 99th %
Hashtag fetching 129.66 7.71 ms ±9.93% 7.57 ms 9.98 ms
##### With input For #tag_10 #####
Name ips average deviation median 99th %
Hashtag fetching 123.43 8.10 ms ±9.06% 7.98 ms 11.36 ms
##### With input For #tag_11 #####
Name ips average deviation median 99th %
Hashtag fetching 110.94 9.01 ms ±8.54% 9.00 ms 11.52 ms
##### With input For #tag_12 #####
Name ips average deviation median 99th %
Hashtag fetching 106.29 9.41 ms ±6.77% 9.38 ms 10.97 ms
##### With input For #tag_13 #####
Name ips average deviation median 99th %
Hashtag fetching 100.08 9.99 ms ±8.01% 9.92 ms 12.72 ms
##### With input For #tag_14 #####
Name ips average deviation median 99th %
Hashtag fetching 95.85 10.43 ms ±8.59% 10.35 ms 13.23 ms
##### With input For #tag_15 #####
Name ips average deviation median 99th %
Hashtag fetching 90.54 11.05 ms ±11.22% 10.92 ms 16.90 ms
##### With input For #tag_16 #####
Name ips average deviation median 99th %
Hashtag fetching 91.79 10.89 ms ±6.60% 10.85 ms 13.02 ms
##### With input For #tag_17 #####
Name ips average deviation median 99th %
Hashtag fetching 77.92 12.83 ms ±6.04% 12.76 ms 15.13 ms
##### With input For #tag_18 #####
Name ips average deviation median 99th %
Hashtag fetching 73.64 13.58 ms ±8.12% 13.37 ms 17.49 ms
##### With input For #tag_19 #####
Name ips average deviation median 99th %
Hashtag fetching 73.75 13.56 ms ±5.89% 13.44 ms 15.67 ms
##### With input For #tag_20 #####
Name ips average deviation median 99th %
Hashtag fetching 255.74 3.91 ms ±11.69% 3.87 ms 5.68 msEdited by lain