Cannot run my test without affecting all other tests

[RRosio]

My test needs additional JupyterServer flags (--LabApp.custom_css=True), and I don't know how to add them without affecting the server configuration for all the other tests.

• What is the best way to run a server with a custom configuration for my test?
• Where does the test go in the Jupyterlab repo? I am trying to write a Galata test to ensure that the custom CSS is applied. The workflow file for Galata tests starts the server at this line, I don't see a way to run my test without making a new workflow file/step that starts a different server for use with my test.

I would like to make a write up of this process so that others using Galata can refer to it later on. Thank you!

[krassowski]

I don't know how to add them without affecting the server configuration for all the other tests.

One approach which circumvents the problem would be having the custom CSS always on and then defining the custom CSS as:

body[data-testing-custom-css] .jp-Cell {
   background: red
}

and then in your integration test set the data attribute on body to enable the tests.

[krassowski]

(for reference the configuration is implemented in)

jupyterlab/jupyterlab/galata/__init__.py

Lines 10 to 40 in 453adc4

	def configure_jupyter_server(c):
	"""Helper to configure the Jupyter Server for integration testing
	with Galata.
	By default the tests will be executed in the OS temporary folder. You
	can override that folder by setting the environment variable ``JUPYTERLAB_GALATA_ROOT_DIR``.
	.. warning::
	Never use this configuration in production as it will remove all security protections.
	"""
	# Test if we are running in a docker
	if getpass.getuser() == "jovyan":
	c.ServerApp.ip = "0.0.0.0" # noqa S104
	c.ServerApp.port = 8888
	c.ServerApp.port_retries = 0
	c.ServerApp.open_browser = False
	# Add test helpers extension shipped with JupyterLab.
	# You can replace the following line by the two following one
	# import jupyterlab
	# c.LabServerApp.extra_labextensions_path = str(Path(jupyterlab.__file__).parent / "galata")
	c.LabServerApp.extra_labextensions_path = str(Path(__file__).parent)
	c.ServerApp.root_dir = os.environ.get(
	"JUPYTERLAB_GALATA_ROOT_DIR", mkdtemp(prefix="galata-test-")
	)
	c.ServerApp.token = ""
	c.ServerApp.password = ""
	c.ServerApp.disable_check_xsrf = True
	c.LabApp.expose_app_in_browser = True

[krassowski]

Also, if you want to test server startup of the server with different options you can add a jest test using the testing utility made for exactly this purpose:

jupyterlab/packages/testing/src/start_jupyter_server.ts

Lines 18 to 49 in c106f0a

	/**
	* A Jupyter Server that runs as a child process.
	*
	* ### Notes
	* There can only be one running server at a time, since
	* PageConfig is global. Any classes that use `ServerConnection.ISettings`
	* such as `ServiceManager` should be instantiated after the server
	* has fully started so they pick up the right `PageConfig`.
	*
	* #### Example
	* ```typescript
	* const server = new JupyterServer();
	*
	* beforeAll(async () => {
	* await server.start();
	* }, 30000);
	*
	* afterAll(async () => {
	* await server.shutdown();
	* });
	* ```
	*
	*/
	export class JupyterServer {
	/**
	* Start the server.
	*
	* @returns A promise that resolves with the url of the server
	*
	* @throws Error if another server is still running.
	*/
	async start(options: Partial<JupyterServer.IOptions> = {}): Promise<string> {

Does the above help @RRosio?

[ericsnekbytes]

Adding a question here. As far as I know, these are the pieces of Lab's testing infrastructure:

• playwright
• galata (playwright utility layer + python wrapper)
• jest (just found this out from your comment, I haven't found much, maybe only this, in the JupyterLab docs for this, am I missing it?)

Are there any other pieces? Thanks :D

[RRosio]

Thank you @krassowski this has all been very helpful! I have been trying out the suggestion you made with the data attribute on the CSS, as it seems more approachable to me. I am currently setting the --LabApp.custom_css=True flag in the galata/__init__.py. Then I am working on a test in galata/test/documentation that will take screenshots (I am having some issues with running the tests atm). Currently I am adding the data attribute defined custom CSS to the placeholder file, custom.css, that I have added in my PR, and in the test itself, setting the data attribute on the body as you mentioned.

If I understand, I can use the jest test to check that the Server starts up with my additional flag? I am inclined to think that the test should check that the handlers have been appended by making a request to that endpoint, is that the sort of test you are referring to? Also, once I have checked that, would it be a test that I should include in my PR?

[krassowski]

jest (just found this out from your comment, I haven't found much, maybe only this, in the JupyterLab docs for this, am I missing it?)

Every core package includes a test folder (for example packages/cells/test); they are also mentioned in contributing docs in Build and run the tests (which should be slightly reworded - it appears to have been written before galata introduction - and be moved out of "Installing JupyterLab" for visibility. But no one is hiding these tests really ;)

There is also pytest used on the backend. I would not split galata and playwright into separate subpoints; there is no python wrapper around playwright at least nothing that I would call that; galata is just a set of helpers for playwright which is a pretty typical use case of playwright I think.

Briefly (this possibly should be turned into a PR once I or someone else have more time):

• front-end visual tests:
  • use cases: visual regression testing, high level integration testing (end-to-end), screenshots for documentation
  • stack: playwright/galata
  • server: single instance running for all the tests, fixed configuration
  • mocked: settings/server responses/layout as needed
  • where: galata
• front-end code tests:
  • use cases: front-end unit testing, low level integration testing
  • stack: jest, jest-dom, simulate-event (in the past used to include karma)
  • server: only started for tests which need it, started and stopped as many times as needed, configuration can be customized for each test
  • mocked: DOM objects, user interactions
  • where: in packages/<core-package-name>/test
• back-end code tests:
  • use cases: back-end unit testing, CLI testing
  • stack: pytest, pytest-jupyter
  • server: started if fixture requested (there are two fixtures from jupyterlab-server, one allows configuration and the other not, but only one seems to be used)
  • mocked: backend and remote stuff like PyPI registry to test extension manager
  • where: jupyterlab/tests
• less relevant to most contributors:
  • examples tests which check a core subset of public API that we advertise in examples
  • smoke tests e.g. jupyterlab/browser_check.py

[krassowski]

By the way, a way to familiarise yourself with the codebase which worked very well for me was gradually helping with pull request review, especially these tagged as Needs Review ;)

[krassowski]

I am inclined to think that the test should check that the handlers have been appended by making a request to that endpoint, is that the sort of test you are referring to?

Yes.

If I understand, I can use the jest test to check that the Server starts up with my additional flag?

Basically yes, this was my thinking; more specifically whether the handler is available and whether server returns the css file depending on the flag setting (as you described). Perhaps it better belongs on the pytest level though?

[RRosio]

Perhaps it better belongs on the pytest level though?
Thank you @krassowski!

I have one more question about the Galata tests. I am trying to run the tests locally to first generate the image that my test will later use for comparison, but I am running into either 403 or 404 errors. The recordings show the Jupyterlab application in the browser and a Launcher Tab opened. Below are some of the errors. Maybe it is something in either my environment that is not correct or the steps that I am taking to run the tests.

[W 2023-11-06 12:13:38.719 ServerApp] 404 GET /api/contents/test-documentation-custom_css-Use-custom-CSS-layout-should-apply-custom-CSS-to--documentation?type=directory&content=1 (::1): No such file or directory: test-documentation-custom_css-Use-custom-CSS-layout-should-apply-custom-CSS-to--documentation
[W 2023-11-06 12:13:38.720 ServerApp] wrote error: 'No such file or directory: test-documentation-custom_css-Use-custom-CSS-layout-should-apply-custom-CSS-to--documentation'
    Traceback (most recent call last):
      File "/Users/rosioreyes/miniconda3/envs/lab4_custom_css/lib/python3.12/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/Users/rosioreyes/miniconda3/envs/lab4_custom_css/lib/python3.12/site-packages/jupyter_server/services/contents/handlers.py", line 121, in get
        model = await ensure_async(
                ^^^^^^^^^^^^^^^^^^^
      File "/Users/rosioreyes/miniconda3/envs/lab4_custom_css/lib/python3.12/site-packages/jupyter_core/utils/__init__.py", line 189, in ensure_async
        result = await obj
                 ^^^^^^^^^
      File "/Users/rosioreyes/miniconda3/envs/lab4_custom_css/lib/python3.12/site-packages/jupyter_server/services/contents/filemanager.py", line 863, in get
        raise web.HTTPError(404, "No such file or directory: %s" % path)
    tornado.web.HTTPError: HTTP 404: Not Found (No such file or directory: test-documentation-custom_css-Use-custom-CSS-layout-should-apply-custom-CSS-to--documentation)

Running 1 test using 1 worker

  ✘  1 [documentation] › test/documentation/custom_css.test.ts:13:7 › Use custom CSS layout › should apply custom CSS  (1.5m)


  1) [documentation] › test/documentation/custom_css.test.ts:13:7 › Use custom CSS layout › should apply custom CSS  

    Test timeout of 90000ms exceeded.

    Error: page.$: Target page, context or browser has been closed

       at src/helpers/activity.ts:58

      56 |       ? Utils.xpBuildActivityTabSelector(name)
      57 |       : Utils.xpBuildActiveActivityTabSelector();
    > 58 |     return page.$(`xpath=${tabSelector}`);
         |                 ^
      59 |   }
      60 |
      61 |   /**

        at ActivityHelper.getTab (/Users/rosioreyes/Desktop/code/jupyter/jupyterlab/galata/src/helpers/activity.ts:58:17)
        at ActivityHelper.isTabActive (/Users/rosioreyes/Desktop/code/jupyter/jupyterlab/galata/src/helpers/activity.ts:37:28)
        at fn (/Users/rosioreyes/Desktop/code/jupyter/jupyterlab/galata/src/fixtures.ts:328:33)
        at Timeout.check (/Users/rosioreyes/Desktop/code/jupyter/jupyterlab/galata/src/utils.ts:166:33)

    attachment #1: video (video/webm) ──────────────────────────────────────────────────────────────
    test-results/test-documentation-custom_css-Use-custom-CSS-layout-should-apply-custom-CSS--documentation/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 failed
    [documentation] › test/documentation/custom_css.test.ts:13:7 › Use custom CSS layout › should apply custom CSS  

The steps I take to set up the tests
1. Build dev_mode with jlpm run build
2. Launch Jupyter lab jlpm run start or jlpm run start:doc
3. Install Chromium with jlpm playwright install chromium
4. Build Galata with jlpm run build
5. Run tests with jlpm run test:doc or jlpm playwright test documentation/custom_css.test.ts

[krassowski]

The 404 errors are unrelated and are not really errors. I believe this is problem with jupyter-server and tracked in:

• Save-As should use a head request (it does not need the body) and get should catch non-existing. #15153
• Don't let ContentsHandler's internal 404 bubble up. jupyter-server/jupyter_server#1328

The other issue is looks similar to what I am trying to fix in #15355. Can you try to apply the changes to activity.ts from that PR locally, run jlpm run build in galata folder and se if it helps?