friendica (DFRN) - Link zum Originalbeitrag

Federated Pages on Instance insanly slow since yesterday

Hello dear Friendica Support, Friendica Developers and Friendica Admins,
since yesterday the federated pages of my instance are infinitely slow (/community, /network).
I looked at everything in detail and tried everything possible and impossible.
I have reconfigured the database, moved the profile pictures from the database to the file system, reconfigured PHP, etc. All this did not bring any improvement.
None of this brought any improvement.
There are no abnormalities in the PHP, Apache and Mysql log files.
The slow build-up only occurs on the federated pages, i.e. where there is really a lot going on.
There, one PHP process (of many) always hangs on about 90% CPU (per user) until the page is displayed, which can take up to 2 minutes.
The load average is around 3-5%.
Does anyone have similar experiences, or is my hoster simply failing?

Any help would be appreciated 😀

!Friendica Support !Friendica Admins !Friendica Developers

!Friendica Developers !Friendica Admins !Friendica Support

teilten dies erneut

Als Antwort auf Raroun

friendica (DFRN) - Link zum Originalbeitrag

Raroun

The Database Settings are looking fine (i´m familiar with database settings) i just activated the slow query log.
But it seems php is taking much time to fullfill a request.
Per User one php thread is working like 1-2 Minutes for displaying the global timeline.
Probably its my hoster.
I Ordered a new dedicated Server - just in case.

teilten dies erneut

Als Antwort auf Raroun

friendica - Link zum Originalbeitrag

Damien Goutte-Gattat

@Raroun I have observed some performance problems as well after upgrading to 2022.10, reported here. I am not sure if they are related to what you are seeing – in my case, seemingly all pages were slow to load, not only the federated pages (but maybe the other pages were slow because of the high server load caused by the federated pages… I don’t know).
@Raroun

teilten dies erneut

Als Antwort auf Damien Goutte-Gattat

friendica (DFRN) - Link zum Originalbeitrag

Raroun

In my case the Server load is at 5%-10% in average. The database is returning the query’s in a normal timeframe but php takes 1-2 minutes to generated / calculate the delivered page to the user.
I also noticed that that there is high activity on the network. I get like 300 Notifications per hour at least.
I didn’t changed something, the federated sites just went insanely slow since yesterday.

teilten dies erneut

Als Antwort auf Raroun

friendica - Link zum Originalbeitrag

Damien Goutte-Gattat

@Raroun If it’s really the same problem, it may not come from your hoster then. In my case, my instance is already (always has been in fact) hosted on a dedicated virtual server. It’s a small VPS (2 CPU cores, 4 GB of RAM, 80 GB of disk space), but until now it has always been enough for my mono-user instance.
@Raroun

teilten dies erneut

Unbekannter Ursprungsbeitrag

friendica (DFRN) - Link zum Originalbeitrag

Raroun

!Michael Vogel
Hi 😀
So i upgraded my whole instance to a new Server - with php 8 and really fast hardware (24 Threads, 128gb Ram, NVME) - but its the same as before.
Php process is stalling at 100% (like an infinte loop) until it reaches max execution timeout.
Is there a possibilty to go back to 2022.06 just for testing purposes?
!Friendica Developers
!Friendica Developers !Michael 🇺🇦

teilten dies erneut

Unbekannter Ursprungsbeitrag

friendica (DFRN) - Link zum Originalbeitrag

Raroun

!Michael Vogel
The only Statement that appears sometimes in the slow query log is this one:
# Query_time: 1.600431 Lock_time: 0.000767 Rows_sent: 47824 Rows_examined: 1005873
# Rows_affected: 0 Bytes_sent: 47378874
use db7263723a1;
SET timestamp=1668536191;
SELECT `uid`, `id`, `parent`, `guid`, `network`, `gravity`, `uri-id`, `uri`, `thr-parent-id`, `thr-parent`, `parent-uri-id`, `parent-uri`, `conversation`, `commented`, `created`, `edited`, `received`, `verb`, `object-type`, `postopts`, `plink`, `wall`, `private`, `starred`, `origin`, `parent-origin`, `title`, `body`, `language`, `content-warning`, `location`, `coord`, `app`, `rendered-hash`, `rendered-html`, `object`, `quote-uri`, `quote-uri-id`, `allow_cid`, `allow_gid`, `deny_cid`, `deny_gid`, `mention`, `global`, `author-id`, `author-link`, `author-name`, `author-avatar`, `author-network`, `author-updated`, `author-gsid`, `author-addr`, `author-uri-id`, `owner-id`, `owner-link`, `owner-name`, `owner-avatar`, `owner-network`, `owner-contact-type`, `owner-updated`, `causer-id`, `causer-link`, `causer-name`, `causer-avatar`, `causer-contact-type`, `causer-network`, `contact-id`, `contact-uid`, `contact-link`, `contact-name`, `contact-avatar`, `writable`, `self`, `cid`, `alias`, `event-created`, `event-edited`, `event-start`, `event-finish`, `event-summary`, `event-desc`, `event-location`, `event-type`, `event-nofinish`, `event-ignore`, `event-id`, `question-id`, `question-multiple`, `question-voters`, `question-end-time`, `has-categories`, `has-media`, `delivery_queue_count`, `delivery_queue_done`, `delivery_queue_failed`, `featured`, `contact-uid`, `gravity`, `post-type`, `post-reason` FROM `post-user-view` WHERE (`parent-uri-id` IN (4511094, 4809812, 4907690, 3635238, 4967581, 4907047, 4943697, 4814005, 4966482, 4914065, 4965670, 4911685, 4916408, 4963419, 4963170, 4726254, 4956288, 4961280, 4960948, 4953203, 4946122, 4943639, 4956766, 4940726, 4918454, 4932644, 4909260, 4954533, 4953931, 4946145, 4946427, 4953058, 4932005, 4907437, 4885029, 4920727, 4915707, 4952289, 4886192, 4926705, 4870458, 3558348, 4951081, 4909157, 4895268, 4768035, 4927956, 4249269, 4943757, 4945693, 4912871, 4925983, 4943671, 4927997, 4910104, 4928135, 4884978, 4907710, 4496324, 4847643, 4890237, 4911933, 971843, 4498373, 4915994, 4917334, 4916702, 4676426, 4916492, 4912754, 4928229, 4927746, 4816024, 4926383, 4926344, 4830339, 4926288, 4926215, 4926163, 4925904, 4925973, 4925952, 4925907, 4925894, 4925859, 4919267, 4925610, 4925487, 4925493, 4925450, 4925420, 4833863, 4108117, 4924164, 4923547, 4923380, 4923372, 4915966, 4909130, 4922017) AND `uid` IN (0, 2) AND (`vid` != 6 OR `vid` IS NULL) AND `visible` AND NOT `deleted` AND NOT `author-blocked` AND NOT `owner-blocked` AND (NOT `causer-blocked` OR `causer-id` = 0 OR `causer-id` IS NULL) AND NOT `contact-blocked` AND ((NOT `contact-readonly` AND NOT `contact-pending` AND (`contact-rel` IN (2, 3))) OR `self` OR `gravity` != 0 OR `contact-uid` = 0) AND NOT `post-user-view`.`uri-id` IN (SELECT `uri-id` FROM `post-user` WHERE `uid` = 2 AND `hidden`) AND NOT `author-id` IN (SELECT `cid` FROM `user-contact` WHERE `uid` = 2 AND `blocked` AND `cid` = `author-id`) AND NOT `owner-id` IN (SELECT `cid` FROM `user-contact` WHERE `uid` = 2 AND `blocked` AND `cid` = `owner-id`) AND NOT (`gravity` = 0 AND `author-id` IN (SELECT `cid` FROM `user-contact` WHERE `uid` = 2 AND `ignored` AND `cid` = `author-id`)) AND NOT (`gravity` = 0 AND `owner-id` IN (SELECT `cid` FROM `user-contact` WHERE `uid` = 2 AND `ignored` AND `cid` = `owner-id`))) ORDER BY `uri-id` DESC, `uid` DESC;
!Michael 🇺🇦

teilten dies erneut

Unbekannter Ursprungsbeitrag

friendica (DFRN) - Link zum Originalbeitrag

Raroun

Thought so, probaly that output from the "rendertime" addon helps?
Datenbank: 3.991/0, Netzwerk: 0, Darstellung: 143.34, Sitzung: 0, I/O: 0.02, Sonstiges: 1.42, Gesamt: 148.77
Class-Init: 0.009, Boot: 0.024, Init: 0, Inhalt: 148.731, Sonstiges: 0.015, Gesamt: 148.779

teilten dies erneut

Als Antwort auf Raroun

friendica (DFRN) - Link zum Originalbeitrag

Raroun

Tried nearly everything the last 3 days.
Switched all PHP Versions from 7.1 to 8.0, with fastgci with mpm Prefork, its allways the same result.
Disabled all Friendica Addons: same result
Configured Hugetables: same result
Adjusted nearly every value in php.ini and pfp-fpm.ini: same result
PHP Process is stuck for around a minute an then it tries to allocate more Memory.
Strace output:
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc10ed31000
munmap(0x7fc10ed31000, 2097152) = 0
mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc100801000
munmap(0x7fc100801000, 2093056) = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc10ed31000
munmap(0x7fc10ed31000, 2097152) = 0
mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc100601000
munmap(0x7fc100601000, 2093056) = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc10ed31000
munmap(0x7fc10ed31000, 2097152) = 0
mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc100401000
munmap(0x7fc100401000, 2093056) = 0
That allocation goes on for another minute, after that the Page is instantly shown.
I am desperate now 🙁

teilten dies erneut

⇧