Double-byte characters in post url turned to mess code

[andrepan]

Enviroment

BuddyPress 1.1.2/WPMU 2.8.5.2

---

Sympton

Upgraded from 1.1.1/2.8.4a via admin panel automatically, the upgrade went smoothly with no error popup.

But after that, a tricky bug was found with Site Wide Activity widget on home page, the URLs that contain double-byte characters and full mode punctuations (say, Chinese characters I'm using) somewhat messed up, and the links lead to page not found notification, I believe those links are all OK in 1.1.1.

This bug affects the Site Wide Activity widget on home page (so far I know, the Recent Site Wide Posts is not affected), Activity/Profile page under Account menu.

I have a screenshot attached.

BTW, I have /%year%/%monthnum%/%day%/%postname%/ for permalink.

Best,
Andre

[levinng]

It caused by add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); function in buddypress/bp-activity/bp-activity-filters.php

It stripped %ad to somewhat another string by calling wp_kses_split function

I was changed the line 413 in /wp-include/kses.php

from

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*(>|$)|>))%',

to

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*^>))%',

Would any wordpress expert confirm my modification is make sense and no side effect.

Thanks alot!

[levinng]

Replying to levinng:

It caused by add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); function in buddypress/bp-activity/bp-activity-filters.php

It stripped %ad to somewhat another string by calling wp_kses_split function

I was changed the line 413 in /wp-include/kses.php

from

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*(>|$)|>))%',

to

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*^>))%',

Would any wordpress expert confirm my modification is make sense and no side effect.

Thanks alot!

Or comment out add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); without touch the core.

[andrepan]

Replying to levinng:

Replying to levinng:

It caused by add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); function in buddypress/bp-activity/bp-activity-filters.php

It stripped %ad to somewhat another string by calling wp_kses_split function

I was changed the line 413 in /wp-include/kses.php

from

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*(>|$)|>))%',

to

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*^>))%',

Would any wordpress expert confirm my modification is make sense and no side effect.

Thanks alot!

Or comment out add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); without touch the core.

Hi, levinng,
It seems both don't work for me

[levinng]

@andrepan

I tired to copy your blog post content to my testing platform, it seems work for me, are you sure 'bp-activity-filters.php' first 3line is like that? or maybe it only work for BP 1.1.2. Gd luck!

/* Apply WordPress defined filters */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */

[andrepan]

Replying to levinng:

@andrepan

I tired to copy your blog post content to my testing platform, it seems work for me, are you sure 'bp-activity-filters.php' first 3line is like that? or maybe it only work for BP 1.1.2. Gd luck!

/* Apply WordPress defined filters */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */

@levinng
Yes, I'm quite sure it's been modified like that, nothing happened, and I'm using 1.1.2 without cache.

Thanks for follow up :)

[levinng]

Replying to andrepan:

Replying to levinng:

@andrepan

I tired to copy your blog post content to my testing platform, it seems work for me, are you sure 'bp-activity-filters.php' first 3line is like that? or maybe it only work for BP 1.1.2. Gd luck!

/* Apply WordPress defined filters */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */

@levinng
Yes, I'm quite sure it's been modified like that, nothing happened, and I'm using 1.1.2 without cache.

Thanks for follow up :)

Could it be a unclean 1.1.2 version when you using automatic upgrade?, if you don't mind, could you send me your buddypress plugin to levin@… or on the other hand, you could download and test a clean 1.1.2. Thus did you replace the theme after you upgrade 1.1.2? i'm not sure it is related or not.

[andrepan]

Replying to levinng:

Replying to andrepan:

Replying to levinng:

@andrepan

I tired to copy your blog post content to my testing platform, it seems work for me, are you sure 'bp-activity-filters.php' first 3line is like that? or maybe it only work for BP 1.1.2. Gd luck!

/* Apply WordPress defined filters */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */

@levinng
Yes, I'm quite sure it's been modified like that, nothing happened, and I'm using 1.1.2 without cache.

Thanks for follow up :)

Could it be a unclean 1.1.2 version when you using automatic upgrade?, if you don't mind, could you send me your buddypress plugin to levin@… or on the other hand, you could download and test a clean 1.1.2. Thus did you replace the theme after you upgrade 1.1.2? i'm not sure it is related or not.

@levinng I'm so so sorry to make you confused, your solution is verified now, I finally found myself very stupid to modify codes directly in FTP Client but on the file of anthor BuddyPress installation, the multi-tab view of my FTP client trapped me in since I have two remote server tab installed with BP side by side, dead-stupid mistake, anyway, your methodology works 100%! Hope the dev team could have a look at this ticket and roll out a final solution.

Thanks man! Maybe next time we can have a meetup while I'm in HK :)

[levinng]

Hm... btw it just a interim solution, the problem consist two part.

1. add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 );

Why call it twice in bp-activity-filters.php

2. when parsing with bp_activity_filter_kses(wp_kses) in bp-activity-filters.php, I'd tried to echo $content before "return wp_kses( $content, $activity_allowedtags );" , however the echo-ed content didn't contain any mess up character, It just messing the title link retrieve by activity widget.

3. When i remove all additional $activity_allowedtags in bp_activity_filter_kses like below

function bp_activity_filter_kses( $content ) {
global $allowedtags;

        $activity_allowedtags = $allowedtags;
        return wp_kses( $content, $activity_allowedtags );
}

But the result still have those mess up code appears in link. I think it is already out of my knowledge.

4. So I have a last try, I changed the line 413 in /wp-include/kses.php

from

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*(>|$)|>))%',

to

        return preg_replace_callback('%((<!--.*?(-->|$))|(<[^>]*^>))%',

it could be fixed the problem in BP, but i've no idea if it suffer any side effects, I shouldn't touch wp core, because the defects didn't appear in WP alone.

[johnjamesjacoby]

What is the resolution to this?

Are there steps anyone can walk me through to duplicate the environment you're using BuddyPress in to test this more completely?

[levinng]

Replying to johnjamesjacoby:

What is the resolution to this?

Are there steps anyone can walk me through to duplicate the environment you're using BuddyPress in to test this more completely?

Hi jjj,

you could write a new post and enter the title with "學" chinese character word without double quote, save it and return to home page, when you move over the post link title in site wide activity widget, you may notice � character in the url instead of that chinese slug. However, I cannot found any �, when using wordpress alone.

Not just one chinese word affected so far i've seen, but the unicode RAW url contains "%ad" will most likely mess up.

The interim resolution to this issues is comment out the wordpress defined filters in bp-activity-filters.php, or modify the preg_replace_callback in WP's kse.

/* Apply WordPress defined filters */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */
/* add_filter( 'bp_get_activity_content', 'bp_activity_filter_kses', 1 ); */

[levinng]

FYI,
Tested on BP 1.1.2, WPMU 2.8.5 and WPMU 2.8.5.2

[cnorris23]

I tested this on 1.2.5, and it seems to be fixed. I know some code went into a 1.2 point release that dealt with this type of issue. @andrepan and @levinng, could one of you report back as to the status of this issue?

[cnorris23]

Not the final 1.2.5, but the nearly final 1.2.5 to specify.

[levinng]

Hi cnorris23,

This bug is no longer valid in 1.2 branch, however another long lasting RSS bug still persist in 1.2 and remains unfixed. If you have time, please take a look ​http://trac.buddypress.org/ticket/1983

Cheers,
Levin

[cnorris23]

Going to go ahead and close this. @andrepan and @levinng, if for some reason this hasn't actually been fixed, please reopen.