feat!: Run plugin on live deploy URL by default #588

aitchiss · 2023-07-10T13:09:46Z

Closes #587

We previous released the ability to run the plugin during the onSuccess build step as experimental/undocumented functionality. We've had good feedback from users and are now ready to roll this out as the default behaviour.

⚠️ This PR has breaking changes

The plugin will now run after a deploy has completed auditing the live deploy rather than the statically served build folder. This will:

Allow the plugin to be used on ISR/SSR sites
Speed up "time to deploy" since the plugin run no longer holds up the deploy
Allow the plugin to run in an environment truer to production

The main side effect of this change is that if score thresholds have been configured, the plugin will no longer fail the deploy if those scores are not met.

To continue running the plugin onPostBuild (i.e. before the deploy is complete, using the statically served build folder) and maintain the "fail on thresholds" functionality, set fail_deploy_on_score_thresholds to 'true' in netlify.toml:

[[plugins]]
  package = "@netlify/plugin-lighthouse"

  # Set the plugin to run prior to deploy, failing the build if minimum thresholds aren't set 
  [plugins.inputs]
    fail_deploy_on_score_thresholds = "true"

  # Set minimum thresholds for any report area
  [plugins.inputs.thresholds]
    performance = 0.9

If your configuration currently relies on serveDir to ensure the correct page(s) are audited, please note this will have no impact on the new default plugin behaviour. Either continue to run during the onPostBuild stage via the above steps, or amend your path input(s) to ensure the full page route is specified.

For example -

Old configuration relying on serveDir:

[[plugins]]
  package = "@netlify/plugin-lighthouse"

  # Serves the 'pages' directory and audits the file about.html
  # On the live deploy this may correspond to http://1234--yoursite.netlify.app/pages/about
  [[plugins.inputs.audits]]
    serveDir = "pages"
    path = "about.html"

New configuration not relying on serveDir:

[[plugins]]
  package = "@netlify/plugin-lighthouse"

  [[plugins.inputs.audits]]
    path = "pages/about"

…y_on_failed_threshold for onPostBuild event

netlify · 2023-07-10T13:09:53Z

👷 Deploy Preview for plugin-lighthouse processing.

Name	Link
🔨 Latest commit	`c2daeae`
🔍 Latest deploy log	https://app.netlify.com/sites/plugin-lighthouse/deploys/64ae6005b8c60500084e6145

aitchiss · 2023-07-10T13:11:51Z

README.md

    output_path = "reports/route1.html"

-  # to audit an HTML file other than index.html in the build directory


A couple of options are specific only to the onPostBuild behaviour (e.g. which directory to serve, file names other than index.html) so have been moved to the new/separate section

aitchiss · 2023-07-10T13:15:08Z

src/index.js

-    process.env.LIGHTHOUSE_RUN_ON_SUCCESS === 'true'
-      ? 'onSuccess'
-      : 'onPostBuild';
+    inputs?.fail_deploy_on_score_thresholds === 'true'


This is the main change in this PR - pretty much everything else is docs changes or updating tests to include fail_deploy_on_score_thresholds / remove the old env variable for onSuccess

…tBuild

aitchiss · 2023-07-10T13:44:42Z

Test steps for this PR

Check the new default behaviour

Using a test site,

Install this branch version of the plugin
Make sure you don't have the fail_deploy_on_score_thresholds input set
Deploy your test site
Lighthouse output should appear during the onSuccess step
Lighthouse reports should appear in the UI
Viewing the full lighthouse report should show the deploy URL in the header (not localhost)
Check setting thresholds that your deploy won't achieve. The deploy should complete as normal, but the plugin runs in the deploy summary should note the failing threshold
Check you can still audit multiple pages by specifying them in the netlify.toml and they all run onSuccess

Check the old `onPostBuild` behaviour

Using a test site

Install this branch version of the plugin
Set the plugin input fail_deploy_on_score_thresholds = "true" as per the instructions above
Deploy your test site
Lighthouse output should appear during the onPostBuild step (and the deploy should take longer while this runs)
Lighthouse reports should appear in the UI
Viewing the full lighthouse report should show localhost in the header (not the deploy URL)
Check setting thresholds that your deploy won't achieve. The deploy should fail as a result of the thresholds
Check you can still audit multiple pages by specifying them in the netlify.toml and they all run onPostBuild

Extra credit

Run the plugin on some test sites that rely on SSR/ISR and check the pages were successfully audited
Try settings some other configuration options as per the new README and make sure they're applied
serveDir should have no impact unless fail_deploy_on_score_thresholds is also set

aitchiss · 2023-07-10T14:06:30Z

README.md

@@ -173,21 +184,13 @@ yarn local

 ## Preview Lighthouse results within the Netlify UI

-Netlify offers an experimental feature through Netlify Labs that allows you to view Lighthouse scores for each of your builds on your site's Deploy Details page with a much richer format.


This section was incorrectly referencing the feature being in Labs, seems it got missed in a previous cleanup. I've also added some new screenshots to bring them a bit more up to date

aitchiss · 2023-07-10T14:07:09Z

package-lock.json

@@ -6,7 +6,7 @@
  "packages": {
    "": {
      "name": "@netlify/plugin-lighthouse",
-      "version": "4.0.7",
+      "version": "4.1.1",


Missed by previous release(s)

aitchiss · 2023-07-11T13:15:53Z

All manual tests/edge cases checked, just putting a pin in this for now so docs has a chance to weigh in 📌

jackbrewer · 2023-07-11T13:21:44Z

README.md

+#### Fail a deploy based on score thresholds
+
+By default, the lighthouse plugin will run _after_ your deploy has been successful, auditing the live deploy content.
+


Is it worth adding a note about the limitations, e.g. the site needs to be pre-built – no SSR etc

Great callout! I've added a sentence that uses similar wording to what we currently have on the docs page 👍

jackbrewer

LGTM! Thank you so much for getting this over the line 🚀

klavavej

The changes you made to the README look good to me, but I found a few pre-existing things in code blocks that could use further tidying up to align with the other changes you made.

klavavej · 2023-07-11T21:54:15Z

README.md

+
+By default, the lighthouse plugin will run _after_ your deploy has been successful, auditing the live deploy content.
+
+To run the plugin _before_ the deploy is live, use the `fail_deploy_on_score_thresholds` input to instead run during the `onPostBuild` event. 


Earlier on the page, under "The lighthouse scores are automatically printed to the Deploy log in the Netlify UI. For example:" the example deploy log output shows the plugin running at the onPostBuild stage

Since this is no longer the default behavior, and this option isn't introduced until after the deploy log example, it could be helpful to update the deploy log example to be more representative of what folks get by default.

klavavej · 2023-07-11T21:59:01Z

README.md

+To run the plugin _before_ the deploy is live, use the `fail_deploy_on_score_thresholds` input to instead run during the `onPostBuild` event. 
+This will statically serve your build output folder, and audit the `index.html` (or other file if specified as below). Please note that sites or site paths using SSR/ISR (server-side rendering or Incremental Static Regeneration) cannot be served and audited in this way.
+
+Using this configuration, if minimum threshold scores are supplied and not met, the deploy will fail. Set the threshold based on `performance`, `accessibility`, `best-practices`, `seo`, or `pwa`.


earlier on the page, the code sample under "Then add the plugin to your netlify.toml configuration file:" shows thresholds being configured without fail_deploy_on_score_thresholds = "true"

Since the thresholds won't do anything without fail_deploy_on_score_thresholds = "true", I think it would be good to remove them from that initial toml sample

aitchiss · 2023-07-12T08:11:05Z

The changes you made to the README look good to me, but I found a few pre-existing things in code blocks that could use further tidying up to align with the other changes you made.

Great call outs - thank you! I've updated those sections now

klavavej

Thanks for the changes - LGTM!

dsolanorush · 2023-07-13T15:07:28Z

Great work @aitchiss!

1. Lighthouse plugin config - set `fail_deploy_on_score_thresholds = "true”` in `[plugins.inputs]` to get some of the old behavior back. mastodon discussion: https://mastodon.social/@rdela/112208080966164909 - PR 588 netlify/netlify-plugin-lighthouse#588 Looks like a bunch of the config options and behavior changed and then netlify disabled issues - netlify-plugin-lighthouse is now in maintenance(-only) mode https://github.com/netlify/netlify-plugin-lighthouse/blob/main/CONTRIBUTING.md https://fosstodon.org/@eleventy/112208087737676821 - explain lowering of seo failure threshold due to `x-robots-tag: noindex` header set on deploy previews 2. about page file links - these worked in README but needed full URLs to trunk branch for the site

* fix: Lighthouse plugin config and about page file links 1. Lighthouse plugin config - set `fail_deploy_on_score_thresholds = "true”` in `[plugins.inputs]` to get some of the old behavior back. mastodon discussion: https://mastodon.social/@rdela/112208080966164909 - PR 588 netlify/netlify-plugin-lighthouse#588 Looks like a bunch of the config options and behavior changed and then netlify disabled issues - netlify-plugin-lighthouse is now in maintenance(-only) mode https://github.com/netlify/netlify-plugin-lighthouse/blob/main/CONTRIBUTING.md https://fosstodon.org/@eleventy/112208087737676821 - explain lowering of seo failure threshold due to `x-robots-tag: noindex` header set on deploy previews 2. about page file links - these worked in README but needed full URLs to trunk branch for the site

aitchiss added 4 commits July 10, 2023 13:43

make onSuccess the default event, update inputs to accept block_deplo…

c22cd16

…y_on_failed_threshold for onPostBuild event

woops, formatting

9e33c9a

Update README

c5751e8

rename input to fail_deploy_on_score_thresholds

9b66701

github-actions bot added the type: feature label Jul 10, 2023

aitchiss commented Jul 10, 2023

View reviewed changes

fix formatting

841ce2f

aitchiss commented Jul 10, 2023

View reviewed changes

aitchiss added 2 commits July 10, 2023 14:24

run onPostBuild for deploy preview cypress tests

7fa7d3a

add a comment to explain our smoke tests need the plugin to run onPos…

8e7d31f

…tBuild

aitchiss commented Jul 10, 2023

View reviewed changes

aitchiss self-assigned this Jul 11, 2023

benedfit previously approved these changes Jul 11, 2023

View reviewed changes

jackbrewer reviewed Jul 11, 2023

View reviewed changes

jackbrewer previously approved these changes Jul 11, 2023

View reviewed changes

add caveat to new input section of readme

acd78ec

aitchiss dismissed stale reviews from jackbrewer and benedfit via acd78ec July 11, 2023 13:26

benedfit previously approved these changes Jul 11, 2023

View reviewed changes

klavavej self-requested a review July 11, 2023 20:39

klavavej previously approved these changes Jul 11, 2023

View reviewed changes

apply readme improvements from review

c2daeae

aitchiss dismissed stale reviews from klavavej and benedfit via c2daeae July 12, 2023 08:10

klavavej self-requested a review July 12, 2023 14:26

klavavej approved these changes Jul 12, 2023

View reviewed changes

aitchiss merged commit 1116f78 into main Jul 13, 2023

aitchiss deleted the sa/587/makeRunOnSuccessTheDefault branch July 13, 2023 12:29

token-generator-app bot mentioned this pull request Jul 13, 2023

chore(main): release 5.0.0 #591

Merged

rdela mentioned this pull request Apr 3, 2024

Lighthouse plugin config and about page file links rdela/eleventeen#16

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Run plugin on live deploy URL by default #588

feat!: Run plugin on live deploy URL by default #588

aitchiss commented Jul 10, 2023 •

edited

Loading

netlify bot commented Jul 10, 2023 •

edited

Loading

aitchiss Jul 10, 2023

aitchiss Jul 10, 2023

aitchiss commented Jul 10, 2023 •

edited

Loading

aitchiss Jul 10, 2023

aitchiss Jul 10, 2023

aitchiss commented Jul 11, 2023

jackbrewer Jul 11, 2023

aitchiss Jul 11, 2023

jackbrewer left a comment

klavavej left a comment

klavavej Jul 11, 2023

klavavej Jul 11, 2023

aitchiss commented Jul 12, 2023

klavavej left a comment

dsolanorush commented Jul 13, 2023

		output_path = "reports/route1.html"

		# to audit an HTML file other than index.html in the build directory

		@@ -173,21 +184,13 @@ yarn local

		## Preview Lighthouse results within the Netlify UI

		Netlify offers an experimental feature through Netlify Labs that allows you to view Lighthouse scores for each of your builds on your site's Deploy Details page with a much richer format.

		#### Fail a deploy based on score thresholds

		By default, the lighthouse plugin will run _after_ your deploy has been successful, auditing the live deploy content.


		By default, the lighthouse plugin will run _after_ your deploy has been successful, auditing the live deploy content.

		To run the plugin _before_ the deploy is live, use the `fail_deploy_on_score_thresholds` input to instead run during the `onPostBuild` event.

feat!: Run plugin on live deploy URL by default #588

feat!: Run plugin on live deploy URL by default #588

Conversation

aitchiss commented Jul 10, 2023 • edited Loading

netlify bot commented Jul 10, 2023 • edited Loading

👷 Deploy Preview for plugin-lighthouse processing.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aitchiss commented Jul 10, 2023 • edited Loading

Test steps for this PR

Check the new default behaviour

Check the old onPostBuild behaviour

Extra credit

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aitchiss commented Jul 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackbrewer left a comment

Choose a reason for hiding this comment

klavavej left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aitchiss commented Jul 12, 2023

klavavej left a comment

Choose a reason for hiding this comment

dsolanorush commented Jul 13, 2023

aitchiss commented Jul 10, 2023 •

edited

Loading

netlify bot commented Jul 10, 2023 •

edited

Loading

aitchiss commented Jul 10, 2023 •

edited

Loading

Check the old `onPostBuild` behaviour