feat: add new package `rule-tester` #6777

bradzacher · 2023-03-27T02:01:03Z

PR Checklist

Steps in Contributing were taken

References:

Overview

The changes I implemented in #5750 were pretty complex to build because I was working around how the base RuleTester class did things. I ran into a lot of weirdness due to things like the class expecting specific properties on objects.

In looking to implement #6499 - I don't want to hit the same issues, so I think it's best if we just draw a line in the sand and fork it so that we can build our enhancements directly into the tooling.

This is also good because it means that we can provide a new baseline for testing and don't need to worry about what the installed ESLint version supports - which will make our eventual dependency testing easier.

This PR:

fork the base class and adds types
- The logic itself is unchanged, but I did move some things around to make it a bit easier to type and manage.
clones the base eslint test for RuleTester
- This test is completely unchanged because it'll take a complete rewrite to make it jest-compatible.
adds the changes we've added in our existing subclass fork:
- adds afterAll as additional static "test framework providers" to the class.
- adds a new per-test option - skip which runs the test using it.skip.
- adds automatic parser cache cleaning using afterAll.
- adds automatic file naming for type-aware linting
  - I implemented this slightly differently than before:
    - it uses a different default name for tsx files now
    - the default filenames can be overridden in the constructor (previously you HAD to do it per test).
- adds per-test dependency constraints
  - I implemented this slightly differently than before:
    Before our hands were tied and I had to lean on setting only on every test to filter out the skipped tests - now with the new skip functionality we only need set skip on these tests which is a much, much nicer implementation!
adds new features
- adds a new per-test option - skip which runs the test case using it.skip.
- adds itSkip and describeSkip as additional static "test framework providers" to the class.
- adds a new defaultFilenames constructor option to allow configuring the default filenames used for type-aware testing.

Documentation:
https://deploy-preview-6777--typescript-eslint.netlify.app/architecture/rule-tester

TODO:

add a new package
fork the base class
implement the changes we have in our current subclass fork
- implement automatic parser cache cleaning
- implement automatic file naming for type-aware linting
- implement the dependency checks from feat(utils): add dependency constraint filtering for RuleTester #5750
~~migrate codebase to this ruletester~~
- i'll do this in a separate PR
~~implement snapshot testing Enhancement: snapshot testing for lint rule failures #6499~~
- i'll do this in a separate PR

typescript-eslint · 2023-03-27T02:01:05Z

Thanks for the PR, @bradzacher!

typescript-eslint is a 100% community driven project, and we are incredibly grateful that you are contributing to that community.

The core maintainers work on this in their personal time, so please understand that it may not be possible for them to review your work immediately.

Thanks again!

🙏 Please, if you or your company is finding typescript-eslint valuable, help us sustain the project by sponsoring it transparently on https://opencollective.com/typescript-eslint.

netlify · 2023-03-27T02:01:10Z

✅ Deploy Preview for typescript-eslint ready!

Name	Link
🔨 Latest commit	`d5a1839`
🔍 Latest deploy log	https://app.netlify.com/sites/typescript-eslint/deploys/6449d6e3d9b33600080fa9d4
😎 Deploy Preview	https://deploy-preview-6777--typescript-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

nx-cloud · 2023-03-27T02:04:31Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit d5a1839. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch

✅ Successfully ran 32 targets

Sent with 💌 from NxCloud.

packages/eslint-plugin-tslint/src/rules/config.ts

packages/utils/src/ts-eslint/Rule.ts

codecov · 2023-04-02T00:59:11Z

Codecov Report

Merging #6777 (d5a1839) into v6 (3e3b789) will increase coverage by 0.10%.
The diff coverage is 20.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##               v6    #6777      +/-   ##
==========================================
+ Coverage   87.51%   87.61%   +0.10%     
==========================================
  Files         376      375       -1     
  Lines       12939    12923      -16     
  Branches     3820     3821       +1     
==========================================
- Hits        11323    11322       -1     
+ Misses       1231     1216      -15     
  Partials      385      385

Flag	Coverage Δ
unittest	`87.61% <20.00%> (+0.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
packages/utils/src/ts-eslint/Linter.ts	`50.00% <ø> (ø)`
packages/utils/src/ts-eslint/RuleTester.ts	`100.00% <ø> (ø)`
packages/typescript-estree/src/simple-traverse.ts	`71.87% <20.00%> (-4.80%)`	⬇️

JoshuaKGoldberg

Whew, what a PR! This is great - looking forward to having native dependency contraints!

JoshuaKGoldberg · 2023-04-03T16:14:56Z

docs/architecture/Rule_Tester.mdx

+
+import CodeBlock from '@theme/CodeBlock';
+
+# `@typescript-eslint/rule-tester`


[Docs] Maybe we should explicitly call this out as beta/wip/similar, so folks don't see this and worry they should immediately switch to it?

As per our chats on discord - @armano2 and I think we should push to make this a breaking change replacement in v6 - people should switch to it cos it's better in many ways and it should mostly be a 1:1 replacement.

Also switching to the package means people get the only: boolean test prop for free, regardless of their ESLint version.

docs/architecture/Rule_Tester.mdx

packages/rule-tester/LICENSE

packages/rule-tester/package.json

JoshuaKGoldberg · 2023-04-03T16:39:16Z

packages/rule-tester/src/RuleTester.ts

+// we intentionally import from eslint here because we need to use the same class
+// that ESLint uses, not our custom override typed version
+import { SourceCode } from 'eslint';
+import merge from 'lodash.merge';


[Question] Should this be further forked to do the same lodash import style as our other files?

I thought about that - but ESLint has this exact import for its rule tester.
So I figured that we'd just use the exact same code to avoid a duplicate install

JoshuaKGoldberg · 2023-04-03T16:42:05Z

packages/utils/src/ts-eslint/Rule.ts

+  [nodeSelector: string]: RuleFunction | undefined;
+}
+// Interface to merge into for anyone that wants to add more selectors
+interface RuleListenerExtension {}


[Docs] This feels like something that should be mentioned in the docs page?

That was just me adding some future-proofing in case anyone was doing something weird.
Actually don't need it - but I think it's fine to leave it undocumented for the moment.

JoshuaKGoldberg · 2023-04-03T16:43:20Z

packages/utils/src/json-schema.ts

@@ -1,24 +1,2 @@
-// Note - @types/json-schema@7.0.4 added some function declarations to the type package
-// If we do export *, then it will also export these function declarations.
-// This will cause typescript to not scrub the require from the build, breaking anyone who doesn't have it as a dependency


[Question] Is this comment no longer necessary?

yeah because in the olden days we did export * from 'json-schema' and we'd export the types for functions that don't actually exist.
But now a days we can do export type * and it'll not include the runtime bits!

Though #6963 would just be better than making this change at all TBH

packages/rule-tester/typings/eslint.d.ts

add a new package and fork the base class

d2e7f15

bradzacher added enhancement New feature or request DO NOT MERGE PRs which should not be merged yet labels Mar 27, 2023

add our current updated features

6aa9edc

bradzacher force-pushed the fork-rule-tester branch from 3b8c1ac to 6aa9edc Compare March 28, 2023 13:10

tweak types

38d8019

armano2 reviewed Mar 28, 2023

View reviewed changes

packages/eslint-plugin-tslint/src/rules/config.ts Outdated Show resolved Hide resolved

armano2 reviewed Mar 28, 2023

View reviewed changes

packages/utils/src/ts-eslint/Rule.ts Show resolved Hide resolved

bradzacher added 4 commits April 1, 2023 09:44

move util

3f9fa9a

Merge branch 'v6' into fork-rule-tester

9992ffd

revert changes to tslint plugin

35610ac

yarn deduplicate

ea1bf0a

bradzacher marked this pull request as ready for review March 31, 2023 23:58

bradzacher removed the DO NOT MERGE PRs which should not be merged yet label Mar 31, 2023

bradzacher added 3 commits April 1, 2023 10:55

small fixes

6655d19

fix missing 'skip' in allowlist

df9da2c

add docs

0af3ea9

bradzacher mentioned this pull request Apr 2, 2023

feat: remove RuleTester in /utils in favour of the new /rule-tester package #6816

Merged

1 task

add missing sidebar link

af11858

JoshuaKGoldberg approved these changes Apr 3, 2023

View reviewed changes

Merge branch 'v6' into fork-rule-tester

66fc363

bradzacher added this to the 6.0.0 milestone Apr 27, 2023

review

d5a1839

bradzacher changed the title ~~feat: add new package for our rule-tester fork~~ feat: add new package rule-tester Apr 27, 2023

bradzacher merged commit 2ce1c1d into v6 Apr 27, 2023
44 of 46 checks passed

bradzacher deleted the fork-rule-tester branch April 27, 2023 03:12

github-actions bot locked as resolved and limited conversation to collaborators May 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add new package `rule-tester` #6777

feat: add new package `rule-tester` #6777

bradzacher commented Mar 27, 2023 •

edited

typescript-eslint bot commented Mar 27, 2023

netlify bot commented Mar 27, 2023 •

edited

nx-cloud bot commented Mar 27, 2023 •

edited

codecov bot commented Apr 2, 2023 •

edited

JoshuaKGoldberg left a comment

JoshuaKGoldberg Apr 3, 2023 •

edited

bradzacher Apr 27, 2023

JoshuaKGoldberg Apr 3, 2023

bradzacher Apr 27, 2023

JoshuaKGoldberg Apr 3, 2023

bradzacher Apr 27, 2023

JoshuaKGoldberg Apr 3, 2023

bradzacher Apr 27, 2023


		import CodeBlock from '@theme/CodeBlock';

		# `@typescript-eslint/rule-tester`

feat: add new package rule-tester #6777

feat: add new package rule-tester #6777

Conversation

bradzacher commented Mar 27, 2023 • edited

PR Checklist

Overview

typescript-eslint bot commented Mar 27, 2023

netlify bot commented Mar 27, 2023 • edited

✅ Deploy Preview for typescript-eslint ready!

nx-cloud bot commented Mar 27, 2023 • edited

☁️ Nx Cloud Report

codecov bot commented Apr 2, 2023 • edited

Codecov Report

JoshuaKGoldberg left a comment

Choose a reason for hiding this comment

JoshuaKGoldberg Apr 3, 2023 • edited

Choose a reason for hiding this comment

bradzacher Apr 27, 2023

Choose a reason for hiding this comment

JoshuaKGoldberg Apr 3, 2023

Choose a reason for hiding this comment

bradzacher Apr 27, 2023

Choose a reason for hiding this comment

JoshuaKGoldberg Apr 3, 2023

Choose a reason for hiding this comment

bradzacher Apr 27, 2023

Choose a reason for hiding this comment

JoshuaKGoldberg Apr 3, 2023

Choose a reason for hiding this comment

bradzacher Apr 27, 2023

Choose a reason for hiding this comment

feat: add new package `rule-tester` #6777

feat: add new package `rule-tester` #6777

bradzacher commented Mar 27, 2023 •

edited

netlify bot commented Mar 27, 2023 •

edited

nx-cloud bot commented Mar 27, 2023 •

edited

codecov bot commented Apr 2, 2023 •

edited

JoshuaKGoldberg Apr 3, 2023 •

edited