But first, let’s dive deeper into what the WordPress Media Experiments plugin is all about and why it’s making waves. 🏄

WordPress Media Experiments: in a nutshell 🐿️

Before we give you a full rundown, let’s give you a quick overview of what the WordPress Media Experiments plugin does and how to install it.

The plugin is a way to optimize images directly within the Block Editor. It’s one area that isn’t yet possible in WordPress, and we will all usually use third-party solutions to do image optimization site-wide (for instance, via Optimole). However, this plugin carries out all the optimization behind the scenes, which is what makes it original.

The plugin can do much more than simple image optimization, though. For instance, it lets you generate thumbnails, convert GIFs to video, and much more. In fact, WordPress Media Experiments is aiming to achieve a lot, and it could be that plenty of this eventually ends up in the WordPress core.

However, there are also some caveats to note. First off, it’s an experimental plugin, which means it’s not in the WordPress plugin directory.

If you want to try it out, you’ll have to get the archive from GitHub – plus probably do a test run on a dev site instead of your production website (but you probably know that).

Click here if you’re not sure how to get a GitHub plugin installed on WordPress:

Before you start, you’ll need to use PHP 8.1 or higher, as you won’t be able to activate the plugin on a lower version.

Go to the plugin’s repo at https://github.com/swissspidy/media-experiments.

Download and extract the ZIP file from the dedicated GitHub page. While there’s a build process for the more stable version, we recommend the ‘nightly’ build, as you can download and install it in a simpler fashion:

From here, log into your site using Secure File Transfer Protocol (SFTP), and drop the plugin folder into your wp-content/plugins directory:

You can activate the plugin in the same way as any other and begin to try it out!

👉 In case you want to learn more on how to install WordPress themes and plugins from GitHub, read this.

Now that we’ve got the installation out of the way, here’s what’s waiting for you on the inside:

Touring the WordPress Media Experiments plugin with Pascal Birchler

The WordPress Media Experiments plugin comes courtesy of Pascal Birchler.

He’s a Swiss Google engineer who contributes to WordPress in a number of ways.

For instance, he had hands-on involvement in embedding content within WordPress, the WP-CLI, and a whole host of other facets. If you don’t already know about it, his periodic table of WordPress plugins is well worth a look:

As such, it’s no surprise that he also had a way to optimize media within the Block Editor near completion. In fact, this is a great place to start in our Pascal Birchler interview, so let’s do it!

On building things for future scenarios

“How did you start working on this project, and what’s the development context of the plugin?”

I’ve been a long-time WordPress contributor to core, translations and everything related to multilingual, including the Core Performance Team. We look at ways to improve the front end performance of WordPress for the millions of sites that use it. Image compression and formats are a part of that.

I had experience working on another WordPress plugin with similar media features. Matt Mullenweg expressed some interest about compressing the images directly in the Block Editor. I already built this! After some polish and bug fixes, I put out an explanatory video.

What Media Experiments does at its core

“How does the plugin handle the image processing and compression?”

In a nutshell, the plugin moves all image processing from the server to the client. When you add a new image to a post (or want to compress an existing one), all optimizations will happen in the browser before the final image reaches the server. The same goes for all the thumbnail versions of an image.

This is possible thanks to a new, custom upload queue system I’ve built. There are also ports of some open-source image libraries to WebAssembly so you can use them on the web.

Note that once you compress or optimize an image, it’s a permanent change. However, you can retrieve the original version if you change your mind.

You can find any options within the Media Experiments metabox within the Block or Site Editor:

This will change in a dynamic way based on the media you’re using or whether you’re loading an Image Block. Once you complete an image upload, you get to confirm whether you want to use an optimized version on your site:

We’ll showcase a few more of these specific screens through our interview with Pascal Birchler, so keep your eyes peeled for them.

Developing for a near-infinite number of devices, displays, and standards

“What were the most significant technical challenges faced during the development of this plugin, especially in implementing the client-side image processing feature?”

Most of the challenges arise from having to support so many browsers, devices, and image formats – all with their subtle differences. Ensuring that the compression works cross-browser, cross-viewport, and for a broad set of file types is challenging. It’s also very time-consuming.

Another challenge was the introduction of ‘cross-origin isolation’ to the Block Editor, as the WebAssembly image libraries require this security feature. Because the Block Editor is highly customizable and extensible through plugins, it is important that the security hardening doesn’t break any other on-site plugins.

The benefits to the end user when using image compression directly within the Block Editor

Right now, we have a good system for optimizing images using tools such as Optimole or others. However, the ability to achieve all of this behind the scenes within the Block Editor has clear advantages.

Here’s Pascal to explain:

“What’s the impact on loading times from the user’s point of view as opposed to using a dedicated media optimization plugin?”

Client-side image compression like this impacts the load time in two areas:

1. In the editor, image uploads will be much faster as you don’t have to wait for the server to finalize the upload and correlate all the thumbnails. You can continue editing your post while the browser is doing the heavy lifting for you. Other plugins still do everything on the server, whether that’s your own or via an API.
2. When you read a post on the front end, the load time will improve thanks to smaller image file sizes. The improvements will be similar regardless of which plugin you use, as the output image will always have optimization.

Of course, this is a positive for those who wish to try it out. However, in the future, the functionality will be even better – especially if this gets into the WordPress core.

How security is vital for WordPress Media Experiments

“What security measures are in place to protect against potential vulnerabilities, given that the plugin handles external media imports and transcoding operations?”

Right now, there are multiple safety nets in place. Cross-origin isolation instructs the browser to isolate the website from browsing contexts and lets you block the loading of unwanted resources.

Second, there are multiple client-side file type checks in JavaScript to be able to filter out any unwanted media as early as possible. The plugin also relies on the existing strict media handling in WordPress itself. This will reject anything suspicious on the server if needed.

On whether the Media Experiments functionality will make it into the WordPress core

Most WordPress features don’t just appear in a point release. It takes months (even years) of hard work to develop the functionality, which then goes through a thorough review and beta testing process.

Pascal has his own thoughts on whether WordPress Media Experiments will ever become a native feature:

“Do you think the image compression feature is a good candidate for the WordPress core?”

I think it should be possible to build something that is valuable and useful for all WordPress sites. Of course, this means elements such as the video recording or other things I’ve shown in the demo are probably not good candidates for inclusion in core. However, the image compression to me is a ‘no-brainer’: it’s useful, it makes Images faster, it makes your site faster, it reduces stress for the server.

This is arguably one area where WordPress lacks, so Pascal’s thoughts make a lot of sense. Who knows: we may see this end up in a WordPress release within the next year or two.

What it will take for Media Experiments to get into core

“How does a feature get into the WordPress core, along with the approval and committal process?”

…It depends on the size of the feature, I would say. For very small changes (like an improvement) you open a ticket on Trac and someone may feel as though it’s a good idea. For bigger changes, you should maybe write a proposal blog post.

With the Block Editor, there’s another approach because the Gutenberg plugin is a great way to experiment with new features. They may not even be fully ready yet, such as these media experiments. Over multiple Gutenberg plugin releases, we can further improve the feature and eventually add it to the core if it’s good enough.

User Interface (UI) design is a vital component of any plugin

While it should be that functionality wins when it comes to whether the team accepts new features, the reality is that visuals matter too. As such, Pascal had to consider UI design as well as feature design.

As a developer first and foremost, he admits this was a tough job:

“Which was the most challenging part of developing the UI, and what was your most difficult decision?”

I am not a designer! When I began, I first had to get up to speed on the latest changes in the Gutenberg design system, which goes through constant refinement.

Overall, I tried my best to make the whole experience as seamless as possible. There’s a lot of complexity involved with things such as image compression, conversion, recording videos within the editor, or uploading via a QR code. I attempted to build this in a way that feels very natural and ‘WordPress-like’ to users, rather than something ‘foreign.’

What’s exciting about WordPress Media Experiments, besides the image compression?

WordPress Media Experiments packs in a lot of functionality for what is an image compressor at its core. However, there’s plenty to consider, such as its thumbnail generation and GIF conversion.

In fact, it’s the former feature Pascal believes is a clear winner:

“Out of all the plugin features – besides the image compression – which ones excite you the most?”

That’s a tough one – it’s like asking who your favorite child is or something! There are many features in [the plugin], and I don’t think I’ve even shown all of them in the demo.

The client-side thumbnail generation is probably the nicest one because it’s basically invisible. You just upload the image, and everything happens for you.

This facet is something Matt Mullenweg talks about in the State of the Word as an example of what we can still achieve with WordPress – especially the impact on server stress.

Combining Artificial Intelligence (AI) with Media Experiments

WordPress Media Experiments uses AI with regards to its speech-to-text recognition. However, this isn’t all. The plugin will use AI to create video captions and ‘alt’ text too.

Even this simple implementation offers complexity, as Pascal explains:

“What AI does the speech-to-text recognition use?”

It’s basically a company that built open-source machine learning models for speech-to-text generation. They have these open source models for multiple languages…I’m using those to generate the subtitles.

What I really like about it is that it does not talk to any server: it’s all in your browser. [There are] no API keys, and it works offline. Of course, there are limitations because if you have a server that you can talk to, there’s more you can do.

I think it shows what’s possible, and maybe the possibilities in browser is an area where more people will invest some time in the WordPress space, rather than just using ChatGPT.

Of course, AI is something that many plugins are starting to implement, so any new WordPress features also need to consider whether they can offer a benefit. We like the idea of AI alt text generation as it is often a time sink and requires some knowledge to get right for accessibility reasons. This is definitely one area we are excited about, and it could provide a tangible and superb positive for practically all users.

Improving WordPress Media Experiments to capture attention now

WordPress Media Experiments is akin to a ‘live document,’ in that it won’t likely see completion until it becomes part of the WordPress core.

As such, there is a lot to implement and improve upon with this proof of concept plugin:

“Are you planning to continue adding features to the plugin, or will you focus on trying to bring specific features into the WordPress core?”

There are still a lot of things that need polishing or adding to the plugin, but at the same time, I want to bring this to as many people as possible. Doing the former helps to further improve the architecture and uncover any roadblocks that might still exist.

That said, not all the current features necessarily make sense for core inclusion, at least not for an initial version. So right now, I’m trying to figure out what that initial version in the core could look like.

Future functionality within WordPress Media Experiments

Along with updating the plugin in order to make improvements, the feature set will also need additions. This is something Pascal is constantly working on:

“What other exciting things are you planning to bring to the plugin?”

I did get one feature request/idea over the weekend after my Tweet, and I’m now building it. So, let’s say you write a new blog post in WordPress on your computer. Then, you want to add an image to it that’s on your phone. How do you get the image to the computer?

My solution is to click a button in the editor and get a QR code. You scan this with your phone, select the picture, and the plugin will upload it to WordPress. The editor will refresh and insert the image. You could even send the link or code to someone else, for example. Maybe this will become my favorite feature!

[QR code uploading] is not something I’ve seen done before, whereas there are already solutions for compression. I think this is definitely new.

However, where does Pascal see the future of WordPress Media Experiments heading? It’s no surprise that he is thinking about the end-user experience:

There are so many other things you could build, such as enhanced video editing features or a way to optimize your entire media library. First though, I think I am going to start with something simpler: improved thumbnail cropping using so-called ‘saliency detection.’

Today, when you upload a photo of an animal or a person where the object is not nicely in the center, it could be completely missing from the thumbnail version WordPress generates. It’s much better to focus on the most salient region of an image instead, such as a person’s face. A tiny detail like this could make thumbnails much more pleasing.

Wrapping up 🎁

The WordPress Media Experiments plugin brings a potentially revolutionary piece of functionality to the Block Editor. Image compression and thumbnail generation are things you often need to take extra steps to achieve. However, as our interview with Pascal Birchler shows, this technology could soon be within the WordPress core. Certainly, the developer is working hard to refine and revise the plugin, with the result being something we will all benefit from in years to come. Even better, because it’s an open-source solution, anyone can contribute and improve upon the plugin’s functionality!

What was your favorite part about our Pascal Birchler interview, and what excites you about the WordPress Media Experiments plugin? Share your thoughts with us in the comments section below!