New component to manage BuddyPress "attachments"

[imath]

We've been talking about managing media in core in previous dev-chats. I thought about it, explored, and began building something a bit messy at the moment but working ;)
Though, i think it's too early to "attach" (!) a first patch. First, i'd like to have your opinion on the way i see things in this area.

My main idea is to have a component that would help others to deal with files. So far, BuddyPress is already dealing with image files for groups & users avatar management.
So i thought, let's start from here and with this kind of files : images.

Avatar "management" in WordPress Administration screens

I've been working in this area for wp-admin/profile feature and the signups management screen #5374 (again a huge thanks to boonebgorges for reviewing and improving it). And i thought, in this profile and the group administration screen, it could be interesting to give a way to the community administrator to deal with these avatars without going back to the front office of his website.
To do so i'm using a customized WP Media Editor without changing anything to the core avatars API.

Avatar metabox, ​full version of the screen cap

Avatar "editor", ​full version of the screen cap

A unique function to add an attachment

I like the way wp_editor adds a wysiwyg editor and i thought, i could be interesting to have such a way to launch our "attachments" management feature. So i built a class to enqueue script, css and to add the needed settings and strings to the media editor. Thanks to it we could be able to add an avatar to a group or a member in front or backend of the blog and we could extend it to manage other attachments for the members, the message component and why not any other component created by a plugin.

A Custom Post Type to organize components attachments

So i kept on exploring and came to the conclusion that the best way to handle attachments was to benefit from the WordPress attachment API by creating a "container" object to hold a bunch of files. The huge advantage of doing so is that a lot is already done by WordPress. We just need to add some hooks at key points, map some capacities and a custom BackBone script to extend the media editor views (library, uploader, select..). So i built an example of extending the members component to provide a "Photoset" feature to the community users. From their profile page, they can create a photoset (or album) and then add photos to it. Here are some screen captures of the different part of this example.

the photo uploader, ​full version of the screen cap

the tab to manage the photos, ​full version of the screen cap

the user's attachment page, ​full version of the screen cap

the directory structure

Before going on, i'd like to have your feedbacks / opinions about this idea :

1. What do you think of it globally ?
2. I think it would be interesting to find a very generic term for the attachments container so that it could be used for other type of files than images in the future if we need to (it can be used for organizing the attached files to a private message...). If so what would it be named like ?
3. Has_directory : i may be wrong on the subject, but as notifications or messages, i believe attachments shouldn't have a directory page
4. What would be the "community actions" : share in activities ? favorite, like, download counter ...?
5. I think we must try to build a kind of "BuddyPress Attachments API" to help plugins to rely on it for the custom components they're creating.

This is it for now, i'm curious about your feedbacks. You can watch a demo of the different parts i've talked about from my vimeo account :
​http://vimeo.com/imath/attachments

[johnjamesjacoby]

This is a great first pass. I like the direction with making it a general attachments component that integrates with some of the other components VS hardcoding attachment integration into each one. I also like the idea of Gravatar uploads being owned by this component, and taken out of BP Core (similar to Notifications.)

To your points specifically

1. I like it.
2. "Attachment" is a universally recognized word, and balances nicely with WordPress's "Media" which feels homogenized and ambiguous.
3. No urgent need for a directory, but I expect we'll see lots of requests for it in core to address the "Pinterest" style social networks.
4. Fine with Activity's traditional Comments & Favorite. Unsure anything would be both worth adding and specific to Attachments.
5. This is the biggest sticking point. We need a flexible API that can be extended/replaced/pluggable and iterated on with relative ease. Filterable upload paths, uses WordPress core API's to it works with any other media plugins, etc...

I agree also that "Photo sets" needs a new name if we use them. Not hugely convinced we should restrict attachments to living in any one bucket/category/album/group. Can imagine it being convenient to the user to show a UI that can filter attachments by the component they were used in, and being able to see attachments in: Private Messages, Activity Stream updates, Groups, Profile, Blogs, and then just general Attachments. Even in that scenario, I'm imagining "component" as a taxonomy term, not a literal hardwired attachment-to-component connection. That would enable custom taxonomy types for even more elaborate attachment groupings.

[boonebgorges]

Very, very cool!

2. "Attachment" seems good to me too.
3. What jjj said. Maybe set to false for now, but keep in consideration for future.

I think that the UI around this seems really pretty great, and I love how you're integrating the WP tools. My big concerns surround the API (your question 5) and privacy.

API

I like jjj's suggestion that attachments could be associated with more than one item. This suggests using taxonomies to represent these connections. (Though this may introduce issues regarding storage location.)

At a low level, perhaps we could wrap much of the functionality into a BP_Attachment class that represents specific attachment objects. It would have normal CRUD methods, in addition to specialized methods like:

• associate_with_item( $item_id )
• disassociate_with_item( $item_id )
• handle_upload( $_FILES )
• handle_crop( $args )
• get_url()
• get_path()

Then, at a somewhat higher level, we'd have easy-to-use functions for plugin developers who want to implement various aspects of attachments into their plugins (we would also use them internally throughout the components):

• template loop bp_has_attachments( array( 'component' => $component, 'user_id' => $user_id, etc ) )
• bp_load_attachments_assets() or something like that, which could be used to enqueue the necessary javascript and CSS requirements for the front-end
• procedural wrappers for specific BP_Attachment actions

Bottom line is: WP handles much of the very low-level stuff (like file handling), and much of the front-end stuff (through the wp.editor backbone framework). So we'll be building wrappers that provide the BP-specific middleware, like association with BP groups, users, etc. Ideally, it'll be easier to use than WP's core libraries as well, because we have the advantage of hindsight when building :)

Privacy

imath, you and I have dealt with this and talked about it before. Privacy is going to be the very first question we are asked about attachments. The basic situation is this: The only way we can protect files is if they are served through BuddyPress, and the webserver software is prevented from serving them directly. This means we need a combination of webserver rules (like .htaccess or nginx.conf rules) + a download handler script. However, doing this introduces lots of complications:

• Serving through BP means that BP/WP has to load. In the case of an album with 20 thumbnails, this could mean that WP loads 21 times on a single pageload, which is unacceptable in most cases. So we may want to disable embedded items/thumbnails in cases where the webserver cannot serve the files directly.
• We have to account for different webserver software. Apache is fairly straightforward. mod_rewrite is already required for BP, and Apache is nearly always configured to allow the correct overrides via directory-specific .htaccess. So, we can create an .htaccess in the top-level directory that blocks direct access. And if we are smart about the way we store files, we may be able to protect only those files that are associated with "private" items, allowing Apache to serve the others directly; this is how BuddyPress Docs does it ​https://github.com/boonebgorges/buddypress-docs/blob/1.6.x/includes/attachments.php#L199. nginx does not allow for per-directory drop-in rewrites; they all must be configured manually in nginx.conf or another global file. This means that either all uploads are protected, or all of them are unprotected. The dev version of BuddyPress Docs does some detection ​https://github.com/boonebgorges/buddypress-docs/blob/1.6.x/includes/attachments.php#L731 and makes some suggestions for these cases ​https://github.com/boonebgorges/buddypress-docs/blob/1.6.x/includes/attachments.php#L703. And then there's IIS :)
• If attachments can be associated with two items, and one item is private while the other is public, how do you handle permissions for that item?
• etc

We could consider doing v1 of this feature with zero privacy controls. But: (1) we would have to document this very carefully, because people will expect that, by default, things like message attachments and private group attachments will be private; and (2) even if it's not going to be implemented in v1, we should think about what the eventual solution may look like, to make sure that our architecture allows for it.

Any thoughts that people might have about this are welcome. I know that imath (BuddyDrive) and I (BuddyPress Docs) have thought about this issue extensively, but maybe other have too :)

[modemlooper]

If this isn't created from a mobile first approach it will not be very useful. The majority of the internet is accessed from mobile devices now. Nobody takes pictures with a computer. They do it all from mobile devices, Instagram etc.

That being said, if you build it to have and externally accessible API I will build a photo app for App Stores.

[imath]

Two weeks have passed since the publication of this ticket. I thought it was important I make a first "report" of my developments progress for this component.
We decided in a dev-chat that this component would be developed as a "feature as a plugin" component. This is, in my opinion, a very good approach because it will enable us to soon work more collaboratively on the subject without affecting the BuddyPress core.
Before i publish a first version of the plugin on the BuddyPress github repository, which I hope will happen in the coming days, I really want it to be as clean as possible. So I realize it’s taking me more time than I thought, sorry.

PS: primarly wrote this in french and translated it. If like me you're more comfortable with this language you can read my first version ​here ;)

New "feature as a plugin" component for BuddyPress

Since it is possible that the component integrates BuddyPress core when it will be mature enough, I think the activation process must be the same than for the other components : by integrating the components list of the BuddyPress administration settings page (/ wp-admin/options-general.php?page=bp-components). I see two interests :

1. only some portions of the code will need to be changed once integrated in core (such as the path to the include directory of the component files)
2. it allows to apply specific settings in conjunction with the components from being activated and would be likely to use the "attachments" component. These settings will directly be handled from this administration page.

In my local development environment, I created a specific "loader" that allows me to include it in components list. However, it will be necessary to make some adjustments to core to make this possible. In Attachments, you will find a patch (5429.diff) detailing those needs.

About the BuddyPress Attachment API

The plugin is based on a post_type I named bp_attachment, it draws heavily on the WordPress post type attachment we could have used to save time. However, I think there are more advantages to separate things.

1. using the attachment post type would have generated the display of items in the list of media in the administration and during the use of the media editor by the administrator. This might lead to confusion and would need to filter the different queries.
2. Our custom post type can enjoy a custom taxonomy I called bp_component and allow us to categorize the attachments depending on the components they’re linked to.

The bp_attachment post type will have a term attribute ( groups , messages ... ) and it will be possible to enrich it with metas referencing the items of the other components it is attached to, eg group_id = 1. The plugin also provides the ability to attach multiple items to an attachment. Indeed, it seems important to be able to offer users to link a file to several groups for example. This is a request that I was often made to BuddyDrive. So for the BuddyPress attachments component it will happen :)
Ideally, I think, the other components should indicate they wish to use the attachments, we will see that later. In the meantime, the plugin is handling it for a single component : 'groups'.
I preferred to start humbly and see for the other components a bit later. The 'groups' component is interesting because it acts as a container for other features and its visibility attributes can be exploited later in conjonction with the status of the attachment which defaults to 'inherit'.

PHP part

I preferred to cut different parts of the API in different classes in the first place. It is always possible to combine them into one at some point if needed.

1. BP_Attachments_Upload is responsible for managing the uploads. It includes different filters and actions to enjoy the WordPress Attachments API to our advantage to create different metas and associations to the terms of our custom taxonomy .
2. BP_Attachments_Browser sets up and launches the WordPress media editor to allow its use for our attachments component.
3. BP_Attachments is the main CRUD class for attachments. So far I have completed the R and D, C is provided by BP_Attachments_Upload and U is one of the things I still have to work on in the coming days. The different templates and the WP_List_Table in administration screens will use it.
4. I still have to create the class BP_Attachments_Groups to manage the integration of the component within groups.

Each of these classes have "wrapper" functions.
About front end screens: to maximize compatibility with themes, I took the advantage of using the "plugins" templates provided by BuddyPress:

• 'members/single/plugins' for profile pages
• 'groups/single/plugins' for groups pages

H3 tag around the hook do_action('bp_template_title') is a bit annoying , but it's the only drawback I see to this method.
In profile pages, I use $ _GET variables to start. This principle, however, remains interesting if we opt for the use of Ajax in this section to provide a fallback mechanism, in case javascript is disabled or broken. For templates, I use the same markup than other components loops, replacing the avatar by a thumbnail of the photo or an icon file provided by WordPress .
Secondary navigation can provide tabs to view all attachments, those he shared in groups. Eventually we can imagine other tabs for "private" files, shared with friends, etc. In this part of the plugin, I have to build templates for groups and those to display the file or edit it. In the edit template file I plan to list the components linking the file to allow the user to eventually detach his file from some or attach it to others. Finally, the use of a custom post type raises two concerns:

1. The post type is only available on the blog ( root_blog ) on which it is initialized. This means that the use from a child site is not intended or requires the use of the switch_to_blog() function.
2. This is not necessarily linked, but that's what made me think of it. We must create a mechanism for users to verify capabilities. Like checking a user can upload attachments on their profile, to a group, can edit, delete, etc. I started to build a mapping of capabilities that will be important to check when the time comes. I think it will be a priority following the publication of the first version of the plugin.

On this second point, I think we should change the bp_current_user_can() function to allow it to pass arguments in order to check a user can create an attachment for a group_id for instance. I created this adaptation on my local dev. It is integrated in the patch attached to the ticket (5429.diff).
In the particular case of groups or users avatars, for now I only need a small change in core. In bp_core_avatar_handle_crop() if bp_core_delete_existing_avatar() could include the item_id argument, it would be nice. I’m still wondering about bp_core_avatar_handle_upload() as the one I use in the plugin is almost a copy of it. Difference is my use of new WP_Error instead of bp_core_add_message().
These attachments will be a little special because they will not create a bp_attachment in the posts table or in component upload directories. They will use the current logic. My choice is guided by a simplification of the Media Editor for this type of attachment that incorporates two tabs: Upload and Crop. Which brings me to the most complex part to me!

JavaScript / Ajax / WP Media Editor

Since we use a post_type != 'attachment' it forced me to completely overload the BackBone WordPress model to replace Attachments, Attachment & Query by BuddyPress specific ones eg bpAttachments....
The main reason is that WordPress is blocking Ajax actions (get/save) by early bailing after a post_type check :

if ( 'attachment' != $post['post_type'] )
    wp_send_json_error();

It was also necessary that I "copy and adapt" some functions like the one that prepares attachments for javascript and the one that builds the url to the uploaded file, for the same reasons .
About Query I need to integrate arguments to filter attachments by user, component or item_id.
These changes were quite disturbing for my views "extends". I am just starting to glimpse the tunnel! (French expression! not sure it’s great in English: "voir le bout du tunnel")

Another point, version 3.9 of WordPress added, among others, image editing functions directly inside the Media Editor. For now I am not too focused on the potential of this new view, but I plan to explore it more deeply soon especially to avoid using jCrop as ​ImgAreaSelect is included in the editor.
The introduction of this new feature has temporarily "broken", the mechanism I had developed and more globally using the "Add Media" button in the WP_Editor in front end. I proposed a patch on WordPress trac ​27372.diff. I am confident about the resolution of the ​21811 ticket by the release of 3.9.
However, this side effect has prompted me to design a fallback mechanism to keep on being able to create attachments even if Javascript is unavailable or broken.

Privacy

"And last but not least!" boonebgorges is right to focus on this point as we will be expected on the subject, that's for sure. Unfortunately, in this area, even though WordPress provides a "private" status for its media, it provides no mechanism to ensure that access to the files is actually private. We faced this problem for our respective plugins BP Docs and BuddyDrive. I’ve explored how Easy Digital Downloads or WooCommerce was dealing with it, it is very similar. It does not cover all situations, especially when the server software is not Apache.

At the same time, it is likely that the majority of WordPress configurations use Apache. We can therefore use the .htaccess trick by default and pair it with a mechanism to check files are not directly reachable prior to the ability to activate the "private" status of attachments. I looked at the links you've added to your comment, boonebgorges, and I think you’re right, our only option is to help administrators to configure their server so that access to files can be "private".

For instance we could move an image file in a subdirectory of the main private directory. Then from the settings of the component, we could attempt to display the image which if actually loaded would indicate that the server configuration does not provide ability to share private attachments. A tutorial could then be published on the codex to list the available ways to make it happen.

It seems to me that the best practice is to move the main private upload directory out of the web directory while giving proper permissions to the www user to write / edit and delete folders and files within that particular directory.

Then there is the problem of displaying images or other media. Since in such a case, it is easy to force a download, the display is actually more problematic especially if we must check each thumbnail upon current logged in user.

I think the easiest way is to use a general thumbnail to indicate a file is private. In a similar way WordPress does for files that are not images. Then we should be able to display the image in its preview or unique view to limit the impact of having to load WordPress to check file access. About images, I thought, for a very small time, about using the post_content field of our bp_attachment post type to fill it with the base64 encoded version of the image, but I think it may be too heavy in particular when generating these encoded images.
About the plugin, I’ve included a specific directory for private attachments. The idea is to use it instead of the public one according to the status selected of the bp_attachment post_type. In the first version I will publish on github, I think I will leave this part as a "todo" to enjoy your inputs on the subject. This is priority number 2 after capabilities management :)

To end this "report": I think I've completed the first version of the plugin by 60%, and I hope to be able to create the github repository within the next ten days.

5329.diff contains core adaptations I believe are interesting to be published into trunk before the 2.0 release. As explained above about avatars, I will eventually complete it with another patch soon.

[slaFFik]

10 days passed... Waiting for the repository :)

[imath]

Replying to slaFFik:

10 days passed... Waiting for the repository :)

Yeah, sorry i'm at 80% of a first version, i hope to create the repo soon, so that you will be able to test/contribute to it ;)

[imath]

I think very first alpha version of "BP Attachments" is almost ready to be published in a github repository.

It will need some more work about the privacy, Backbone, the capability mapping... It will also need a specific Administration screen, i just created the admin menus. But it's working : it manages the avatars for groups (except during the creation process) and users and it adds a nav to user's profile and group to upload files.

The very last steps, i think we need:

• the patch 5429.01.diff to be committed
• and the ticket #5380 to be solved

So that it'll be easier to test.

In 5429.01.diff, i've tried to modify core the less possible. I simply need an extra action, filter and argument to be passed in a function.

I've also quickly made an awful "frenglish" screencast to demo it :
​demo of BP Attachments

[boonebgorges]

5429.02.diff​ is an update of 01.diff with the following changes:

• Make the filter more general-purpose
• Don't bother passing anything to the new action (it's all available via buddypress(), and we don't know what use people will put the new action to anyway)

If these will work for your purpose, go ahead and commit it.

[imath]

Replying to boonebgorges:

If these will work for your purpose, go ahead and commit it.

It's working great :) i'll commit it tonight.

I have a BuddyPress warning due to #5380 as the current group is not defined in Group Administration. In ​5380.01.diff, i've solve it by being able to pass the group_id to the function bp_get_group_has_avatar()

I'll create the repo before dev-chat ;)

[imath]

In 8160:

Adapt core to allow BP Attachments to be into the list of optional components

So that next "feature as a plugin" Attachments can be activated just like the other BuddyPress components, we need to create two new hooks.

props johnjamesjacoby, boonebgorges, imath.

See #5429

[imath]

The Github repository is created and available here:
​https://github.com/buddypress/bp-attachments

[mazzamuz]

I love what you guys are doing.

Just a few thoughts as I've been looking at this rather extensively...

​http://www.mertyazicioglu.com/projects/image-rotation-fixer/
"Automatically fixes the rotation of JPEG images using PHP's EXIF extension, immediately after they are uploaded to the server." - its in the wp repository.

I love the way ​http://www.buddyboss.com/features/walls/ did this with adding images to the activity feed. (​http://www.plupload.com/)

​http://buddydev.com/ has good popular ideas around avatars

Remember web performance optimization ​https://wordpress.org/plugins/w3-total-cache/ and the like and hosting content else where.

I'm working on a project which needs "hi-res" (1024px+) images attached to group + members profiles. it would be great if the group/member avatar functions could return a "bigger" or custom image crop.

I cant seem to find basic documentation to retrieve the images via loop / ID. ie:
get_pic_by_attachment_id($attid);
get_pics_by_group_id($groupid);
get_pics_by_group_slug($groupslug);
get_pics_by_member_id($memberid);

thats it for now

[sooskriszta]

@imath As discussed a couple of weeks ago in ​devchat, @sbrajesh might be helpful in this pursuit.
​http://buddydev.com/downloads/mediapress/versions/mediapress-1-0-beta1.zip

• User media/gallery/album
• Group media/gallery/album
• Photo, Audio, Video
• Drag and Drop Uploader
• Upload from Activity
• Automatic Wall galleries creation (like Facebook) for Groups/Users
• Grid Layout for media
• Playlist view for audio/Video Galleries

sbrajesh:

I have tried to architect in the way that in future it may be considered for the core...I will be happy to contribute it to the core :)

[imath]

Replying to sooskriszta:

@imath As discussed a couple of weeks ago in ​devchat, @sbrajesh might be helpful in this pursuit.

Of course, i'm convinced of it :) sbrajesh seems to have done an amazing work with the plugin. I'll test it asap. Had a first quick look on the javascript files, and i think we took two different roads. In BP Attachments i'm using BackBone to build a BP Attachments editor looking like the WordPress built in one.

I have tried to architect in the way that in future it may be considered for the core...I will be happy to contribute it to the core :)

sbrajesh you are very welcome, i'm sure there are things we can learn from your great work. I'd be very happy to have your help on this ​file, in particular, and all the rest actually :)

[johnjamesjacoby]

Moving to 2.3 to discuss getting this in early.

[imath]

In this ticket #6004, i've closed, there's an interesting discussion between boonebgorges and me about

Attachments as "library" rather than "component"

[imath]

With the introduction of the Attachments API #6278, i guess we will be able to work on this feature in a future release, if our "appetite" or the community needs for such a component to be included in core have grown :)

[DJPaul]

We're closing this ticket because it has not received any contribution or comments for at least two years. We have decided that it is better to close tickets that are good ideas, which have not gotten (or are unlikely to get) contributions, rather than keep things open indefinitely. This will help us share a more realistic roadmap for BuddyPress with you.

Everyone very much appreciates the time and effort that you spent sharing your idea with us. On behalf of the entire BuddyPress team, thank you.

If you feel strongly that this enhancement should still be added to BuddyPress, and you are able to contribute effort towards it, we encourage you to re-open the ticket, or start a discussion about it in our Slack channel. Please consider that time has proven that good ideas without contributions do not get built.

For more information, see ​https://bpdevel.wordpress.com/2018/01/21/our-awaiting-contributions-milestone-contains/
or find us on Slack, in the #buddypress channel: ​https://make.wordpress.org/chat/

[teeboy4real]

Can this be re-opened again, @imath as started a complete rewrite of the plugin which is in progress ​https://github.com/buddypress/bp-attachments

[imath]

Hi @teeboy4real

No worries, we're following progress about an attachment component form this ticket #8022