Better documentation for Rust

[derMihai]

Description

Some background

Currently, getting anything more complex than examples/rust-hello-world running is undocumented, and really hard without advanced knowledge of how the RIOT build-system works.

I personally have only limited experience with Rust, Cargo and RIOT. In the following, I will list some difficulties I encountered.

Areas that require documentation

Creating a RIOT Rust module or RIOT wrapper

• how to properly create a Rust RIOT module or extend the riot-wrappers crate

Setting up out-of-tree builds

• setting up a OOT rust-application is quite straight-forward, but some stuff is still difficult to find out
  • e.g. how to declare used modules (e.g. in Makefile, Cargo.toml)?
• setting up a Rust OOT RIOT module is unclear. Extending the module creation documentation would be helpful.

Using RIOT C-modules that are neither in riot-wrappers nor in riot-sys

• being able to access any C RIOT module is necessary for any serious project
• there is a bit of documentation in riot-sys that points out that extra headers can just be added in riot-headers.h.
• how should then riot-headers.h be extended?
  1. changing it directly in the .cargo registry works, but this is no clean solution
  2. cloning riot-wrappers and riot-sys from git, modifying them, and then pointing to them in Cargo.toml kinda works. Even if I check-out to the exact version stated in the .cargo registry, I get some warnings (e.g. gnrc_netif_dev_is_6lo redeclared with a different signature) -- this solution cannot be right either

Mismatched riot-sys exports:

• Example: both riot_sys::mutex_lock() and riot_sys::inline::mutex_lock() work with riot_sys::inline::mutex_t and will not compile when supplied with riot_sys::mutex. But using riot_sys::inline::mutex_t will then not work with e.g. riot_sys::ztimer_mutex_lock(). What is the workaround in this case?

Getting some sweet language support

• Rust is (at least for me) no language to use without proper integration in a code editor or IDE
  • rust-analyzer can be configured independent of the editor choice by setting the environment variables in .cargo/config.toml see discussion below
  • LSP support for Rust app development
  • LSP support for riot-sys in Rust module developement
• my experience with vscode: the C bindings generated by riot-sys in ./target are not seen by rust-analyzer. This perhaps boils down to the fact that this is no pure cargo build.
• would be nice to have a proper integration in at least one code editor - seems that rust-analyzer can be configured independently

[derMihai]

@chrysn

[chrysn]

(Picking bits of this between tasks; thanks for tagging me, this needs comprehensive addressing, but bits for now)

• on editing: I'm rarely using editor integration, and given you ask about IDEs my go-to answer of "VIM with rust-analyzer" is likely not helpful. I haven't had my hands on IDEs since Eclipse many years ago -- what configuration do they take these days? Do they take a sh line that they can execute as a language server? Can they pass environment variables to the language server? Do they expect something like compile-commands.json? (Once we know what we need, we can make the build system export everything that is set up to run cargo to that format).
• For modules, the best we have right now is the drivers/lsm303agr example. As for the infrastructure for the module grep for lsm303agr in the tree -- yes, that's way too many occurrences still. (This is clearly not the fix, just the best pointer I can give without writing the rest of the docs)
• Writing wrappers outside of riot-wrappers should work just fine, with using riot-sys as a dependency. The magic riot-wrappers does is primarily about conditionally defining Rust modules depending on which RIOT modules are around -- that's a pattern a non-riot-wrappers module wouldn't use anyway.
• riot-headers.h should eventually be just a "riot-all.h" (but RIOT doesn't have that, so the next best thing is to add in header files as they are needed). The way I'd recommend right now is to fork riot-sys, [patch.crates-io] it to your git branch, and upstream any additions ASAP.

[chrysn]

I get some warnings (e.g. gnrc_netif_dev_is_6lo redeclared with a different signature) -- this solution cannot be right either

That is OK. Thing is, due to RIOT's prevalence of macros we need to run both binden to and c2rust to translate header files, and they each define their own types for structs, and their own functions, and they differ b/c the types differ in identity (but not in layout).

You're only seeing these warnings now b/c cargo won't show you warnigns originating from crates you take from crates.io or git, whereas it will show you warnings from crates you have checked out (expecting that given they're local, you may be in a position to fix them).

[derMihai]

@chrysn thanks for the fast reply, this is valuable information. I will try out your suggestions and write down a log, perhaps that will help with building the documentation later.

About editing and editor/IDE support: seems like rust-analyzer supports lots of editors. Like I mentioned, I use rust-analyzer in vscode, but it fails to discover the C bindings. If rust-analyzer works for you in vim, then I suspect there is a configuration issue. If you use a custom configuration, could you share it please?

[chrysn]

In vim I can specify a custom invocation as

let g:LanguageClient_serverCommands = {
      \ 'rust': ['~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rust-analyzer'],
      \ }

(which is not really what fixes things, for that I'd need to export the per-project environment variables that could be copied out from make all QUIET=0). Question is, do other tools such as vscode need this in the same form (providing a sh invocation), or do they take environment variables, or do we need to pull out the big hammer and provide a rust-analyzer script that invokes the real rust-analyzer but is placed earlier in the $PATH (as you say vscode, I'd guess that's on Windows, so %PATH%?)?

[derMihai]

Indeed, rust-analyzer is failing when building build.rs in riot-sys, because env variables are missing ( Please pass in RIOT_CC; see README.md for details.: NotPresent).

According to the rust-analyzer documentation, it calls cargo check --quiet --workspace --message-format=json --all-targets [+ other options] for that, but that can be changed to something completely different. So maybe a separate make target (e.g. cargo_check) could be defined that calls check instead of build? And then rust-analyzer can can call that. This solution would be independent of editor choice.

[derMihai]

I modified settings.json to the following, and it did the trick:

{
    "rust-analyzer.cargo.extraEnv": {
        "CC": "",
        "CFLAGS": "",
        "CPPFLAGS": "",
        "CXXFLAGS" : "",
        "RIOT_COMPILE_COMMANDS_JSON" : "/home/mihai.renea@ml-pa.loc/proj/lara_test_rust/bin/native/cargo-compile-commands.json",
        "RIOT_USEMODULE" : "at auto_init auto_init_ztimer board board_common_init core core_idle_thread core_init core_lib core_msg core_panic core_thread cpu fmt frac isrpipe isrpipe_read_timeout libc native_drivers periph periph_common periph_gpio periph_gpio_linux periph_init periph_init_gpio periph_init_gpio_linux periph_init_led0 periph_init_led1 periph_init_led2 periph_init_led3 periph_init_led4 periph_init_led5 periph_init_led6 periph_init_led7 periph_init_leds periph_init_pm periph_init_timer periph_init_uart periph_pm periph_timer periph_uart preprocessor preprocessor_successor stdio_native sys tsrb ztimer ztimer_convert ztimer_convert_frac ztimer_convert_shift ztimer_core ztimer_extend ztimer_init ztimer_msec ztimer_periph_timer ztimer_usec"
    },
    "rust-analyzer.check.overrideCommand": [ 
        "cargo", "check", "--target", "i686-unknown-linux-gnu", "--message-format=json"],
    "rust-analyzer.cargo.buildScripts.overrideCommand": [
        "cargo", "check", "--target", "i686-unknown-linux-gnu", "--message-format=json"
    ]
}

So this confirms the missing env variables where the problem. But this is no clean solution. Sadly, rust-analyzer does not support a unified configuration file across editors, although this feature seems to be planned.

I think the solution would be a wrapper around cargo check that sets the environment variables. This wrapper can then be set in rust-analyzer.check.overrideCommand and rust-analyzer.cargo.buildScripts.overrideCommand.

[chrysn]

What scares me a bit about configuring editors to run some wrapper are the security implications. Once your IDE is configured to run some wrapper for cargo-check (or some command that generates environment variables), all of a sudden opening a file in the editor becomes a vector for code execution. Sure, that's also the case when running make, but that is a well-understood risk, whereas opening a file in an editor should really not allow code execution (hello Office macros).

I'd like to better understand how this is used in IDEs -- are these per-project settings.json that are configured, or is this a global file? What are the general security expectations when users open projects in an IDE?

[derMihai]

I understand. For the record, cargo check is already calling pretty much any build.rs it can find, so rust-analyzer too, indirectly. Quite alarming, tbh. What I had in mind is the following:

1. make target that exports RIOT build environment variables to a file, say env.json
2. script that we can prepend to cargo check that reads env.json and sets up the environment variables

The only risk I see (please correct me if I'm wrong) is someone modifying env.json, so our script ends up setting up potentially malicious environment variables.

Vscode opens any new workspace in restricted mode -- that is, it should disable anything that could execute code, e.g. build scripts, extension features. The user must always actively mark the workspace as trusted. settings.json can be set both globally and per workspace. The workspace settings override the global ones.

That been said, I personally use vscode exclusively for convenience. I am willing to try others in the hope that I will find something else that works and isn't Microsoft. So I am really in favor of not assuming any editor, and finding a solution that safely and securely works for everyone.

[chrysn]

Right -- so given that, anything that calls rust-analyzer needs to sandbox it, and that also goes for any wrapper to rust-analyzer.

I think a good path forward would be to provide some make targets (make cargo-check, make cargo-update, make rustanalyzer) along with a more generic target that just prints what you need to prefix to your custom calls.

As for vscode ... I don't care what people use, but I do care that if that environment's behavior is sensible, it should work with our code :-)

[chrysn]

So, how do we get from where we are now to easier use? I've looked at the settings.json you've shown, and make rust-analyzer won't cut it there.

As a first step, in #20319, I'm adding output to make info-rust, which contains all the data you needed for your settings.json. In parallel, with the changes of RIOT-OS/rust-riot-wrappers#79 (which technically do break the output of this, but it's both in PR stage right now), the long modules line that would change often can be replaced with a short DEP_RIOT_SYS_CFLAGS=.../riotbuild.h line, which is just a function of location and board name. rust-analyzer hopefully uses the cargo output of build scripts that tell it which files to watch for changes. Altering the Makefile will not immediately update the riotbuild.h, but at least as soon as you build/flash, things should be automatic. Could you have a look at those PRs as to whether those would work for you?

[derMihai]

I looks good, my only suggestion would be that the contents of $(USEMODULE) be exported into a file, so I can cat it directly when setting the environment, and don't have to copy-paste it every time the module list changes. But if RIOT-OS/rust-riot-wrappers#79 addresses this anyway (with DEP_RIOT_SYS_CFLAGS) then I guess it can stay like that for now.

[chrysn]

Yeah, then I'll just go that route.

Next up:

• Can we get around the need for a (VSCode specific) settings.json by providing a suitable compile-commands.json? That has been said to be used by multiple IDEs, but using none, I wouldn't know.
• If not, what is the search pth for settings.json? Can it just be next to the application's Makefile?

[derMihai]

I think we're mixing up stuff here. compile-commands.json is not used by the IDE, but the language server. In this case, rust-analyzer is the language server, but it doesn't understand compile-commands.json. It looks like it has its own config file for non-Cargo projects, but I don't know if we want to go down that rabbit-hole. What we need is to set up the environment variables for rust-analyzer. And because the IDE calls rust analyzer, it has to set it up whenever it calls rust-analyzer. As long as it doesn't need to be done each time something changes (e.g. module list), I think that manually setting up the environment variables in a editor-specific way is actually a good solution.

Sorry for being verbose, I just want to make sure we are on the same page, and I didn't understand something wrong.

Speaking of settings.json, that is found in .vscode/settings.json in the root of the project. In my case, the root of the project is always the RIOT application directory.

[chrysn]

Thanks for elaborating, I was unaware of the scope of compile-commands.json and of what goes into rust-analyzer's startup.

#20319 is now up to date, and contains a documentation section on IDE/editor setup. Can you check whether that works for you, maybe as a review there?

[derMihai]

So, RIOT_USEMODULE must still be set, but is not printed by info-rust.

[chrysn]

#20319 updated riot-sys and -wrappers to the latest git version; did your application follow that?

[derMihai]

Oh, my bad!
Yes, I pulled the last changes and now it works just fine!

[chrysn]

I've taken the liberty of editing your original post into a checklist, and ticking off "needs proper integration with IDEs" (while leaving "with at least one editor" and "vscode" unchecked because they don't work out of the box yet).

Any priorities what to tackle next?

My suggestion is on the riot-headers.h -- your option ii is what I think is the right way, and the only reason the warnings show is that once you're working from a checkout you get to see warnings that were previously not shown (compiler goes "Oh you got it locally, great, so now that your attention is at it, you may want to know that things are not ideal there"). Reducing the number of warnings is certainly an important task on its own, but it may suffice to explain them (and at any rate, a PR adding a header file can be merged within minutes, no discussion there if it builds). Do you agree with that resolution?

[derMihai]

Yes, I also think riot-headers.h is the next big hurdle. Solution ii is fine for the moment, but in the long run, it would be convenient to pull riot-sys from crates.io. Any way of generating that header on the fly?

[Remmirad]

Regarding an IDE independent way to pass env variables to rust-analyzer one could use the [env] part from <Project>/.cargo/config.toml rust-analyzer uses those (Or rather cargo check does).

[derMihai]

Regarding an IDE independent way to pass env variables to rust-analyzer one could use the [env] part from <Project>/.cargo/config.toml rust-analyzer uses those (Or rather cargo check does).

I just tried this out. Put the following in .cargo/config.toml

[env]
RIOTBUILD_CONFIG_HEADER_C = "<Project>/bin/native/riotbuild/riotbuild.h" 
RIOT_COMPILE_COMMANDS_JSON = "<Project>/bin/native/cargo-compile-commands.json"

[build]
target = "i686-unknown-linux-gnu" # I test this on native
message-format = "json"

And it worked like a charm for both VSCode and neovim, with zero extra project-specific configuration. This is great!

[derMihai]

I was trying things out once again, and there is a case where rust-analyzer still fails, and that is on external rust module creation. But this depends on where the module is located.

1. external app + module in a sub-directory of the app, then everything is ok
2. external app + module located somewhere else, then rust-analyzer fails to find the riot-sys symbols. They are still available when working on the app's source files, but not for the module.

This behavior is independent of the editor, I tried it out with both neovim and vscode. The program compiles otherwise just fine. Sym-linking the module in the application directory doesn't work either.

Another observation is that, in case 1. (module located in the app directory), the path to riot-sys has to be overriden in the Cargo.toml of the module too, i.e. the following snippet

[patch.crates-io]
# we modify the riot-sys crate to include the headers we need
riot-sys = { path = "../../modules/rust-riot-sys" }

has to be present int both app's and module's Cargo.toml. For successful compilation, it is only needed in the app's Cargo.toml.

This, and the fact that rust-analyzer starts a new scan when jumping from an app source-file to a module's makes me suspect that rust analyzer tries to analyze the module as a standalone-thing, but that is not possible because all the bindings are in the app's bin/ folder.

UPDATE:
Seems that what I was suspecting was true. On default, rust-analyzer indeed runs per-workspace - so it tries to run the build.rs in riot-sys for the module too, but the enviroinment variables are not set there. Copying the .cargo/config.toml from the app into the modules's root will do the trick, but this is not a clean solution.
The reason why case 1. just worked boils down to how cargo searches for configuration files. TLDR: cargo checks for config files in parent directories, so it hits the .cargo/config.toml in the app's directory.
Placing the .cargo/config.toml in a directory above both the app and the module will work too, ofc, but this is once again not a clean solution.