Optimizing Media with the Attachment Library

[edfullman1]

Thank you very much for all the hard work with: Avatars, Cover Photos.

I'd like to suggest that all the great work on the Attachment Library is inevitably going to drive more media being stored on Wordpress Servers. While Cover Photos are a relatively new addition to Buddypress very likely some kind of multi-part file like on IOS or even video will be the next request. As a result storage, dev-ops complexity, media integrity, and the general limitations of optimizing a Wordpress/Buddypress will be impacted.

In general, moving media off of a Wordpress server to S3 and other solutions is an objective for developer/dev-ops in order to minimize the on-server file operations, and improve performance. There a number of plugin makers that are busily building solutions to move the media library to S3 and other off-server storage locations.

If the avatars and cover photos could be saved in a part of the media library then they would automatically be candidates to be moved off the server by these plugins. Getting existing data into the library would be the challenge, and then reverting the whole process for someone that didn't like the outcome would be an additional requirement. However, in general this would be a one-way path for most.

Since there is so much work going on inside the attachments library this seems like a reasonable possibility, and an easy opportunity for an improvement.

[boonebgorges]

BuddyPress uses wp_upload_dir() to determine the upload path for images, and uses wp_handle_upload() to handle the upload process itself. I *believe* that, as a result, media management plugins like those that integrate with S3 should automatically work with BuddyPress uploads.

This is not the same thing as integration with the Media Library interface. Among other things, WordPress uses post objects with the post_type 'attachment' to display content in the Media Library. I'm not sure whether this is desirable for BuddyPress content.

I bet imath will have something more to say about this.

[imath]

Hi thank you for your feedback @edfullman1
and thank you @boonebgorges for your great reply. Before adding anything to it i need to make sure i completely understand the description of the ticket.

If the avatars and cover photos could be saved in a part of the media library then they would automatically be candidates to be moved off the server by these plugins

What do you call the "media library" ? As Boone said we're using WordPress process to upload files and they are written in /wp-content/uploads so i have difficulties to see where the problem is.

Could you give an example of plugin?

If i understand well you suggest to use the WordPress 'attachment' post type to organize the files members uploaded for their avatars or cover images ?

[imath]

Ok i think i begin to understand the problem better. Just did a plugin search and found this file:
​https://plugins.trac.wordpress.org/browser/amazon-s3-and-cloudfront/trunk/classes/amazon-s3-and-cloudfront.php

I see the plugin is not filtering wp_upload_dir() but instead it's :

• filtering wp_get_attachment_url to use the amazon url to the file
• filtering wp_update_attachment_metadata to copy the file to amazon and i guess edit the metadata and then eventually remove the local file.

I'll try to look for another plugins like this one to have a better idea about how they manage uploads and if they all do the same.

On the other hand, we could also consider plugins can add feature to handle how BuddyPress is managing avatars and cover images.

[edfullman1]

Thanks for replying.

Sorry to go dark. I've been in the middle of a deployment for the last week.

I've done a lot of research on this issue, and here are the top 3 plugins for this type of feature.
Rtcamp's Rtamazon-S3 at ​http://docs.rtcamp.com/rtamazon-s3/
Delicious Brains WP-Offload-s3 ​https://deliciousbrains.com/wp-offload-s3/ (this is a fork of amazon-s3-and-cloudfront.php)
Humanmade's S3-Uploads ​https://github.com/humanmade/S3-Uploads

All of these plugins basically work the same way with the exception of S3-Uploads. Basically none of them can handle the avatars and cover photos, only media files stored with posts.

I've worked with S3-Uploads, and like the concept. However, the plugin seems to brute force the translation of URLs, and then tries to work around the process of media uploaded to a tmp directory. This causes Wordpress (specifically file.php) to throw an error. This is only a problem with dynamic files in the upload directory, as well as the avatars and cover photo uploads. The attachments to posts don't have this problem.

My personal goal with our online service is to have all the media (including Avatars and Cover Photos) up on Amazon S3 and then eventually on Cloudfront once we get out of beta.

[boonebgorges]

All of these plugins basically work the same way with the exception of S3-Uploads. Basically none of them can handle the avatars and cover photos, only media files stored with posts.

At a glance, it looks like S3-Uploads ought to be able to handle BP uploads. It filters wp_upload_dir(), replacing WP's native file storage with an s3:// stream wrapper. This ought to "just work". I'll try to find some time to test it myself, but maybe in the meantime you can say more about what you mean by "dynamic files in the upload directory". Does this just mean files that are not represented by an attachment-post and the associated metadata?

It'd be an interesting exercise to make BP uploads work with the other S3 plugins *without* requiring the standard WP media treatment for BP content. It'd definitely be possible to fake the creation of attachment metadata - or perhaps to *actually* create it, and then immediately delete it after S3 has done its thing - without needing post_type=attachment. wp_get_attachment_url() obviously doesn't work without a post. It'd be easy enough to write a wrapper that made it work, though I know that the goal is to make it completely seamless with these drop-ins.

I'm pretty wary of going down the road of using WP's standard attachment functions for BP content. Stuff like cover photos doesn't seem like appropriate content for the Media Library. Adding it there will add a ton of bloat to wp_posts and wp_metadata. And it'll mean that we have to do various tricks to keep that content *out* of the regular Media Library view, since you presumably will not want to see thousands of avatars when finding a picture of your cat to embed in a blog post :) So, if there's a lightweight way to work alongside these S3 plugins without resorting to the full Media Library treatment, it'd be strongly preferred.

[imath]

Thanks for your reply @edfullman1

All of these plugins basically work the same way with the exception of S3-Uploads. Basically none of them can handle the avatars and cover photos, only media files stored with posts.

This is inspiring me 2 things :

1. I've been working on a plugin lately that changes the avatar dir / avatar url according to the requested object and item id (eg: group having the ID 1 or user having the ID 42..). And BuddyPress is including the needed filters to achieve what you're suggesting for avatars and cover images.
2. The ​BP Attachments plugin is using a very similar way WordPress uses for their attachments, it would be interesting to see how it behaves with these plugins (you'll need latest trunk to test this plugin).

What boonebgorges says here:

And it'll mean that we have to do various tricks to keep that content *out* of the regular Media Library view, since you presumably will not want to see thousands of avatars when finding a picture of your cat to embed in a blog post :)

is very important i agree at 100%.

Cover images or Avatars don't need to be "organized" in the database, because it's basically one file (or 2 files for the avatar - thumb/full) in a directory restricted to the object. So adding entries to the database, imho, would really be too bad. Of course i'm really open to hear and take in account arguments proving i'm wrong :)

The approach for the BP Attachments plugin is a bit different because one user can share more than one file (hopefully!). And i think keeping the files generated by this plugin out of the WordPress Media Library is *still* very important.

I see the WordPress Media Library more like public content to use to "prettify" your posts and pages.

The *future* BuddyPress Attachments Library will have to face more complexity, starting with "Privacy" and the very big difference is: it will target BuddyPress specific objects like Users, Groups, Private messages... which all *are not* post types :). As a result these are not content to use to "prettify" a blog post or a page and shouldn't introduce any confusion being in the WordPress Media Library loop (not to mention potential troubles with Attachments gallery plugin!). Of course the Administrator should be able to moderate these contents but i really think it's best to do it somewhere else in the Administration.

If you are interested, we've been talking about it on this ticket #5429, and i would love if you could contribute to the BP Attachments plugin: because i strongly believe the enhancement you're suggesting is more relative to it ;)

[edfullman1]

Thanks,

I looked at the repo, and it looks promising. Photo's on activity updates :)

So, I'll try to stand up a test server this weekend, and see how this plugin works with S3-Upload, and I'll try rtAmazon-S3 if I get a chance.

Ed.

[edfullman1]

You may also want to look at rtCamp's Media Plugin for Buddypress and bbPress

​https://wordpress.org/plugins/buddypress-media/

This delivers some of the goals of #5429. However, interestingly it still doesn't add value to avatars and cover photos.

[edfullman1]

I had a few minutes, and implemented BP-Attachments. Basically out of the box it has the same problem as Buddypress without the plugin. The challenge for example with S3-Uploads is that it is translating URLs and when Wordpress tries to envoke move() it fails, because I suppose it looks for the the uploaded file on the server versus on S3.

I didn't install Buddypress 2.4.0-alpha. Can the plugin function without it?

[DJPaul]

Yeah, to confirm: I tried with s3-uploads, doesn't work out the box.

[DJPaul]

I'm going to close this ticket for now. We don't want to implement with the WP Media Library for the discussed reasons, and there's at least one plugin that's testing this approach to see how well it works. We *do* need to support media services such as the S3 Uploads plugin as much and as soon as possible, but that ought to have its own ticket.

Thanks for the discussion, @edfullman1 !