Add SBGP extension handling #2053

thillux · 2023-10-12T13:04:47Z

This PR adds support for parsing SBGP ASNs and IP-Ranges. A basic test is also added.

wiktor-k · 2023-10-12T13:28:32Z

FTR I think the job fails because of issues fixed in #2052 (just saying that it's nothing that you've introduced).

thillux · 2023-10-12T13:29:58Z

FTR I think the job fails because of issues fixed in #2052 (just saying that it's nothing that you've introduced).

Thank you for the hint!

Signed-off-by: Markus Theil <theil.markus@gmail.com>

thillux · 2023-10-12T16:52:00Z

This currently depends on #2055. After it is merged, I'll cleanup my temporary commits.

Signed-off-by: Markus Theil <theil.markus@gmail.com>

thillux · 2023-10-14T10:41:34Z

This currently depends on #2055. After it is merged, I'll cleanup my temporary commits.

I found another fix for the compilation with Rust 1.56. This PR is ready now from my side.

sfackler · 2023-10-14T22:43:33Z

openssl-sys/src/handwritten/x509_sbgp.rs

+ * At the moment this is coded for simplicity, not speed.
+ */
+#[cfg(ossl110)]
+fn addr_expand(addr: *mut u8, bs: *const ASN1_BIT_STRING, length: isize, fill: u8) -> bool {


This doesn't seem like something that should be in openssl-sys.

This function, and associated functions , has now been replaced with a FFI binding to X509v3_addr_get_range and X509v3_addr_get_afi.

sfackler · 2023-10-14T22:44:52Z

openssl/src/x509/extension.rs

+            };
+            append(&mut value, &mut first, true, &asn_format);
+        }
+        X509Extension::new_nid(None, Some(ctx), Nid::SBGP_AUTONOMOUSSYSNUM, &value)


We need to move away from the string APIs for x509 extensions, and instead directly work with the relevant OpenSSL types.

sfackler · 2023-10-14T22:45:24Z

openssl/src/x509/sbgp.rs

+        }
+    }
+
+    pub fn ranges(&self) -> Option<Vec<(u32, u32)>> {


Does this correspond to some OpenSSL function?

Same question for the other methods in this file.

These function correspond to X509v3_addr_get_range, but executed for the whole extension, since RFC 3779 encodings are rather tedious to work with. Same thing applies to the other functions in this file, they just parse the openssl-internal representation into their Rust-equivalents.

botovq · 2023-10-21T09:02:42Z

This is a very dangerous API full of traps and bugs. It needs to be wrapped very carefully. What is the use case you want to support by exposing these? Do you have some working code I could look at?

- X509v3_addr_get_range - X509v3_addr_get_afi

thillux · 2023-11-15T10:37:58Z

@botovq We are currently in the process of reworking this PR. Our use case is certifying address ranges and ASNs in a datacenter fabric setting. Sadly I have no public code that I can share.

@PetrichorIT is currently working on this.

botovq

I really think it is a bad idea to expose this API. What stops you from using rpki-rs for this?

As already mentioned, and you will have no doubt found while developing this, the OpenSSL RFC 3779 API is really bad. And I am not aware of any OpenSSL developer with domain expertise to speak of. So that's not going to change anytime soon.

You don't magically get memory safety by using this via ffi from Rust and then omitting error checks for functions that allocate internally. Very much the opposite of that.

There are omissions of necessary features, missing checks, etc. This all looks taylored to a narrow use case. I do not think this is anywhere close to being ready. People who will try to use this in the future will have a very hard time interoperating with modern RPKI validators.

botovq · 2023-11-15T12:37:07Z

openssl/src/x509/extension.rs

+    /// Construct a new `SbgpAsIdentifier` extension.
+    pub fn new() -> SbgpAsIdentifier {
+        SbgpAsIdentifier {
+            critical: false,


I don't think this should ever be non-critical. At least default to critical since that's what RPKI certs should contain. It's a MUST in RFC 6487 section 2, while RFC 3779 marks it as SHOULD, hence false is a poor default.

yes, critical=true is now the default

botovq · 2023-11-15T12:38:01Z

openssl/src/x509/extension.rs

+    }
+
+    /// Sets the `critical` flag to `true`. The extension will be critical.
+    pub fn critical(&mut self) -> &mut SbgpAsIdentifier {


Why do you need this knob at all?

In its newer version fn critical(&mut self, value: bool) is might be useful to have this configuration option for some legacy systems ? I personally cannot think of a use case, but just in case?

botovq · 2023-11-15T12:45:30Z

openssl/src/x509/extension.rs

+                if min == max {
+                    ffi::X509v3_asid_add_id_or_range(asid, 0, min.as_ptr(), std::ptr::null_mut());
+                } else {
+                    ffi::X509v3_asid_add_id_or_range(asid, 0, min.as_ptr(), max.as_ptr());


These can fail and will sometimes free min and/or max and sometimes they won't. In the former case you'll double free below.

Definitely needs error checking, but I'm unsure how you can avoid the double free without panicking and retain memory safety.

The newer implementation should prevent double frees + free all nessecary allocations.

botovq · 2023-11-15T12:50:02Z

openssl/src/x509/extension.rs

+    /// Construct a new `SbgpIpAddressIdentifier` extension.
+    pub fn new() -> SbgpIpAddressIdentifier {
+        SbgpIpAddressIdentifier {
+            critical: false,


Again, default to critical

botovq · 2023-11-15T14:24:58Z

openssl/src/x509/extension.rs

+                std::mem::forget(max);
+            }
+
+            X509Extension::new_internal(Nid::SBGP_AUTONOMOUSSYSNUM, self.critical, asid as *mut _)


Before doing this you should check that your extension data is canonical, otherwise you produce an extension with invalid DER encoding.

botovq · 2023-11-15T14:26:15Z

openssl/src/x509/extension.rs

+        self.asn.push((asn_min, asn_max));
+        self
+    }
+


add_inherit() is missing in the builder.

botovq · 2023-11-18T10:09:14Z

openssl-sys/src/handwritten/x509_sbgp.rs

+use libc::*;
+
+#[repr(C)]
+#[cfg(ossl110)]


All these should probably condition on OPENSSL_NO_RFC3779 as well.

PetrichorIT · 2023-11-29T15:53:08Z

Given the current state of the RFC 3779 API, I think there are two options:

Either we only expose a certain subset of the RFC 3779 API that we can assure to be "safe", or we expose the entire API with all meaningful features and checks and try to build a more abstract API around it, to ensure safety.

I personally prefer a second approach.
If we limit the ways we use the API surface of openssl when interacting with the prime suspects
(X509v3_asid_add_id_or_range, ...), I think the API could be safely represented in Rust.
For the time being, separating the creational API and the read API seems like a good idea
to manage the complexity inherent to this problem.

I'd like to try this approach and see where it leads.

thillux · 2023-11-29T16:24:17Z

I really think it is a bad idea to expose this API. What stops you from using rpki-rs for this?

We have custom OpenSSL providers for random number generation we'd like to use here.
We already use OpenSSL with RFC3779 support in a C++-based application, which consumes certificates from our Rust-based CA.

As already mentioned, and you will have no doubt found while developing this, the OpenSSL RFC 3779 API is really bad. And I am not aware of any OpenSSL developer with domain expertise to speak of. So that's not going to change anytime soon.

We are aware of this and never use OpenSSL without address sanitizers as the bare minimum during development in our C++ applications using OpenSSL and RFC3779.

You don't magically get memory safety by using this via ffi from Rust and then omitting error checks for functions that allocate internally. Very much the opposite of that.

Sure, due to memory safety reasons we chose Rust for our CA application, but it does not completely outweigh missing features in the OpenSSL API subset exposed for our use case.

There are omissions of necessary features, missing checks, etc. This all looks taylored to a narrow use case. I do not think this is anywhere close to being ready. People who will try to use this in the future will have a very hard time interoperating with modern RPKI validators.

We currently do not deploy a full RPKI and are a bit taylored yes. But we are eager to make this PR usable in a broader fashion. If somebody steps in and tells us, what is missing in his opinion.

botovq

Thanks for the explanations. I won't have time for a thorough review in the next days/week, so I only left a few quick comments.

I think we should aim at a good middle ground between the two options you describe: let's make it flexible enough to be generally usable and correct, but let's make it easy enough to use by not exposing things that aren't used in the wild.

So, for example, you currently don't expose the SAFI for IPAddressFamily and you don't expose the RDI variant for ASIdentifiers (which I think is good). In a similar vein, let's not expose criticality as a knob.

botovq · 2023-11-29T16:20:28Z

openssl/src/x509/extension.rs

+pub struct SbgpAsIdentifier {
+    critical: bool,
+    asn: Vec<(u32, u32)>,
+    inherit: bool,


I think it is good hygiene to make illegal states non-representable as far as possible/reasonable. The ASIdentifiers should either inherit or contain a (non-empty) list of ASIds and ASRanges. So I think this should be an enum with an Inherit and an ASNumber variant that contains the Vec<(u32, u32)> with the list of ASIds and ASNumbers.

I think this is a good idea. However I think patterns like type-state would be overkill in this scenario, so assertions / panics remain nessecary to prevent silent failures.

botovq · 2023-11-29T16:33:10Z

openssl/src/x509/extension.rs

@@ -428,6 +443,258 @@ impl AuthorityKeyIdentifier {
    }
 }

+#[cfg(ossl110)]
+pub struct SbgpAsIdentifier {
+    critical: bool,


I would suggest that you omit this field. If anyone really needs to fiddle with criticality, they can readily add the field and accessor with a solid justification.

botovq · 2023-11-29T16:40:14Z

openssl/src/x509/extension.rs

+pub struct SbgpIpAddressIdentifier {
+    critical: bool,
+    ip_ranges: Vec<(IpAddr, IpAddr)>,
+    inherit: Option<IPVersion>,


Similarly here, let's omit the criticality marker and for the sake of correctness, I think this should have two optional enums, one for IPv4 and one for IPv6. Each of these enums contains either an Inherit marker or a vector of IP ranges of the correct kind.

Unless I'm missing something, this struct doesn't currently allow representing inheritance of both IPv4 and IPv6.

PetrichorIT · 2024-02-01T11:15:02Z

So this PR should be ready now.

In regards to the extension builders, criticality will no longer be a configuration option,
and all builder should now make invalid states irrepresentable. To prevent missuse of the current
API , builders will panic if an operation results in a loss of information, so e.g.

SbgpAsIdentifier::new()
    .add_asn(1234)
    .add_inherit()
    .build()

would panic, since the assigned AS number would be lost when the inherit flag is set.

The functions X509V3_{addr, asid}_{inherits, subset, is_canonical} are now represented in the API.
X509V3_{addr, asid}_canonize is not publicly exposed, since the builder only needs to use it internally.
In general, i think is good to keep the read API and write/builder API seperate, since mutating a raw extension
could go wrong in a number of ways and we dont really want to expose the underlying datastructures to the user.

In regards to X509V3_asid_subset, since this API is faulty in 1.1.0, it was moved to 1.1.1 (openssl/openssl#18514).

The APIs for interacting with store contextes and validation paths X509v3_{asid, addr}_{validate_path, validate_resource_set}
are not supported, since they seem like a source of unwanted complexity regarding this PR. They are however bound in openssl-sys,
for completions sake.

@botovq

botovq · 2024-02-01T11:17:55Z

Thanks. I will try to find time to take a look soon.

botovq · 2024-02-29T09:33:24Z

Apologies for the delay. I haven't forgotten. I won't be able to review this properly before mid-March.

HolyShitMan · 2024-03-26T11:50:30Z

Apologies for the delay. I haven't forgotten. I won't be able to review this properly before mid-March.

Did you find any time to review the PR?

Greetings

botovq · 2024-03-26T11:57:48Z

On Tue, Mar 26, 2024 at 04:50:52AM -0700, Simon Buttgereit wrote: > Apologies for the delay. I haven't forgotten. I won't be able to review this properly before mid-March. Did you find any time to review the PR?

I did. Not quite through yet.

HolyShitMan · 2024-04-24T08:45:51Z

Knowing that my request may be annoying: I would like to kindly remind you that this pull request is still open.

botovq

Ignoring the comment on X509_get_ext_d2i(), the use of the C API looks about as sane as it can be now.

Regress coverage is decent. As noted, various panic strings need a typo fix (allready -> already). Whether rust-openssl would like panics or different error handling is up to the maintainers of the crate -- as are other API concerns on the rust side of things.

As far as the reservations I had are concerned, this should be ready to proceed.

botovq · 2024-04-24T10:16:51Z

openssl/src/x509/sbgp.rs

+                ffi::NID_sbgp_autonomousSysNum,
+                std::ptr::null_mut(),
+                std::ptr::null_mut(),
+            );


X509_get_ext_d2i is a tricky API to use correctly. As far as I can see, no caller in rust-openssl gets it right.

This can return NULL for several reasons: the benign reason is that there is no extension corresponding to the nid in the cert. However, there could also be a library-internal error (malformed extension, memory allocation error, ...), or other errors, like multiple extensions of the same type, which this idiom silently ignores.

The correct way is to pass in a pointer crit to a c_int as third argument. If NULL is returned and crit is set to anything but -1, an error should be raised.

Not sure if this should be fixed in this PR or if rust-openssl wants to land this as-is and fix the issue in a sweep.

After considering this further, if there would be a minimally invasive solution available, I'd opt to address it within this PR. However, it seems that we require some new Error construct due to the fact that X509_get_ext_d2i doesn't set the internal OpenSSL error, making the ErrorStack inapplicable.

I'm unsure if this issue extends to other OpenSSL functions as well, where a new Error construct would also help to handle return values properly. After all, I propose addressing this in a separate PR to avoid overloading the current one.

In the meantime of the last few months we also got some additional X509 functionality.
Yet, I suggest following the same procedure: let's complete the current task first, and then I'll create an additional PR specifically for the new functions.

botovq · 2024-04-24T10:19:34Z

openssl/src/x509/extension.rs

+    pub fn add_inherit(&mut self) -> &mut SbgpAsIdentifier {
+        if let SbgpAsIdentifierOrInherit::List(ref l) = self.0 {
+            if !l.is_empty() {
+                panic!("cannot set extension to inherit, list allready contains elements");


Suggested change

panic!("cannot set extension to inherit, list allready contains elements");

panic!("cannot set extension to inherit, list already contains elements");

Similar typos are throughout the panic strings

No problem, I will fix all these

nomick and others added 6 commits October 12, 2023 15:51

read AS numbers

c160383

IP address range parsing

525dcbb

add possibility to define sbgp extensions

a7dc993

some style fixes

187ee6d

fixes for current master and OpenSSL 3

d27ac33

Signed-off-by: Markus Theil <theil.markus@gmail.com>

add test for SBGP extension

e93ab39

Signed-off-by: Markus Theil <theil.markus@gmail.com>

thillux force-pushed the sbgp-extension branch 2 times, most recently from 3bf5d75 to d759114 Compare October 12, 2023 14:27

thillux force-pushed the sbgp-extension branch from 5ff5364 to 6c1c43a Compare October 12, 2023 16:57

mark supported OpenSSL version range, make CI happy

815a951

Signed-off-by: Markus Theil <theil.markus@gmail.com>

thillux force-pushed the sbgp-extension branch from 6c1c43a to 815a951 Compare October 12, 2023 17:03

thillux changed the title ~~Add S-BGP extension handling~~ Add SBGP extension handling Oct 12, 2023

fix formatting for older rust versions

df85aef

Signed-off-by: Markus Theil <theil.markus@gmail.com>

sfackler reviewed Oct 14, 2023

View reviewed changes

PetrichorIT added 5 commits October 25, 2023 16:40

removed local reimpls of ossl functions

78c4b4e

- X509v3_addr_get_range - X509v3_addr_get_afi

API refactoring, removed unessecary components

95f97b3

switched to internal types for sbgp extension building

98bff2f

simplified builder, added more tests

a24c404

removed random whitespace change in systest/build

c4a61d1

botovq mentioned this pull request Nov 11, 2023

Potential use after free in callers of X509v3_asid_add_id_or_range() openssl/openssl#22700

Closed

PetrichorIT added 2 commits November 15, 2023 13:28

appeased clippy & added missing cfg(ossl110)

54c6e48

fix unused import

938a715

botovq suggested changes Nov 18, 2023

View reviewed changes

PetrichorIT added 2 commits November 27, 2023 13:34

fixed inherit & critical defaults & canonical

c5cad36

fix leak, deferred allocation of max

0dd8c8d

botovq reviewed Nov 29, 2023

View reviewed changes

PetrichorIT added 12 commits December 4, 2023 11:45

simplified builder, removed nonrepr states

e4c23d1

fixed cfg statements

2386460

added assertions & panics + renamed some types

49fec48

docs & AS number extension subset_of test

cc295b5

added SBGP IP ext, misssing bindings

b6f9916

added missing bindings + fixed fmt

7949ec4

added OPENSSL_NO_RFC3779 to tests

ba3bf9f

tests missing cfg flags + simplified use stmts

fa9e0b3

Merge branch 'master' into sbgp-extension

7cb5e43

fixed missing cfg flags

8f32bb9

fixed X509V3_asid_subset, no longer available in 1.1.0

6cc634c

fixed rustfmt

7bce3d5

botovq reviewed Apr 24, 2024

View reviewed changes

fix typos & small clippy fix

6dcf781

	panic!("cannot set extension to inherit, list allready contains elements");
	panic!("cannot set extension to inherit, list already contains elements");

Add SBGP extension handling #2053

Are you sure you want to change the base?

Add SBGP extension handling #2053

Conversation

thillux commented Oct 12, 2023

wiktor-k commented Oct 12, 2023

thillux commented Oct 12, 2023

thillux commented Oct 12, 2023

thillux commented Oct 14, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

botovq commented Oct 21, 2023

thillux commented Nov 15, 2023 • edited

botovq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PetrichorIT commented Nov 29, 2023

thillux commented Nov 29, 2023

botovq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

botovq Nov 29, 2023 • edited

Choose a reason for hiding this comment

PetrichorIT commented Feb 1, 2024

botovq commented Feb 1, 2024 via email

botovq commented Feb 29, 2024

HolyShitMan commented Mar 26, 2024

botovq commented Mar 26, 2024 via email

HolyShitMan commented Apr 24, 2024

botovq left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thillux commented Oct 14, 2023 •

edited

thillux commented Nov 15, 2023 •

edited

botovq Nov 29, 2023 •

edited