BP Group forums will not read topics that have a large amount of content text

[burtadsit]

westpointer reported problems with a specific topic on a site. See: http://buddypress.org/forums/topic.php?id=1807

His replies in that specific topic totaled ~80K+. Tbey were visible and usable in bbpress but bp reported "There are no posts for this topic." when it's obvious there were.

After looking around at this issue westpointer and I debugged the problem and it turns out that the data was coming across from bbpress to bp but the ixr client call was failing with the error code IXR_Error(-32700, 'parse error. not well formed');

That means that the data got to the client but the client library was having some kind of problem with it after it got there. Eventually we tracked it down to the parse() function in the IXR_Message class in class-IXR.php. parse() gets the xml data and that function was returning false to query() in IXR_Client making the query fail.

In parse() the code that does the parsing of the xml data was not failing:

`

if (!xml_parse($this->_parser, $this->message)) {

/* die(sprintf('XML error: %s at line %d',

xml_error_string(xml_get_error_code($this->_parser)),
xml_get_current_line_number($this->_parser))); */

return false;

}

`

It was never getting there. The only other place in that function that it could gracefully fail was at the top of parse().

`

first remove the XML declaration

$this->message = preg_replace('/<\?xml(.*)?\?'.'>/', , $this->message);
if (trim($this->message) == ) {

return false;

}

`

That is the first thing that parse() does and it returns false because trim() is failing on the 80K string. We checked for data actually in $this->message above that on entry to the fn and after the preg_replace() call. The data was getting there, going successfully through preg_replace() and trim() came up with which caused the fn to fail. Eventually all the way back up the chain to bp.

Any xmlrpc call that results in a big chunk of data coming across will fail this way. Don't know if the actual parser can handle that much data but it never gets the chance to find out.

This must be related to some magic php number like 32K or 64K where goofy things begin to happen. While we were scrounging around trying to pin this down we found this which describes a similar problem at about 80K. ​http://bugs.php.net/bug.php?id=46987

Andy wanted to be reminded of this call in BP_Forums_Template_Topic class:

`
$this->total_post_count = count( bp_forums_get_posts( $topic_id ) );
`

That gets the entire topic and bypasses pagination. That will cause the xml client library to fail if the topic has a large amount of total text.

I'm not sure there is a workaround for this issue other than always use pagination when getting xmlrpc data. I think that spot above is the only place we found where pagination wasn't being used.

I thought that this might be solved by using the multicall feature of the library. Where multiple xmlrpc calls are queued and then one call is made and all the returns from these multiple calls are sent back to the client.

That wont help really. The first call that gets more than a happy amount of data will fail. We aren't doing multiple different things and we are trying to reduce traffic. We are doing one thing that isn't handled by the xmlrpc client lib very well.

Site admins should be made aware of this and stay on top of large topics. It's not the number of replies in the topic, it's the total size of one page of data.

[burtadsit]

The ticket didn't handle my double single quotes happily and turned this line into something odd:

The data was getting there, going successfully through preg_replace() and trim() came up with an empty string which caused the fn to fail. Eventually all the way back up the chain to bp.

That's what it should read. The italics weren't mine. If I really want to emphasize something I'll just talk louder than the others in the room, thereby making me more right than they are.

[apeatling]

This line has been replaced:
$this->total_post_count = count( bp_forums_get_posts( $topic_id ) );

With a total post count from the topic object and it still breaks, so we can scratch that off as a potential reason.

[westpointer]

Andy and Burt,

Thank you very much! The file you sent me yesterday did fix the problem with my large thread. I'd set pagination to occur at 50 posts. Once I reduced that to 40 posts, the topic started working again (​http://buglenotes.com/groups/free-for-all/forum/topic/79)

One time to note, bp_the_topic_latest_post_excerpt() is still broken for large topics. See ​http://buglenotes.com/groups/free-for-all at the Obama Chronicles II topic.

[apeatling]

Will take a look at bp_the_topic_latest_post_excerpt() but I'll close this as fixed for now.