I have zero experience with nix unfortunately.

Ah, no worries. Can I leave this open in case anyone else knows?

I have the same problem. When setting OPENSSL_DIR to the install location of openssl, it still fails with:

error: failed to run custom build command for `openssl-sys v0.9.75`

Caused by:
  process didn't exit successfully: `/tmp/cargo-install4GB1Io/release/build/openssl-sys-baf4c06737a35cbf/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR = /nix/store/0amx6k0b9i34kj5rbfq359s95mzj4xg8-openssl-1.1.1p-bin/bin/

  --- stderr
  thread 'main' panicked at 'OpenSSL include directory does not exist: /nix/store/0amx6k0b9i34kj5rbfq359s95mzj4xg8-openssl-1.1.1p-bin/bin/include', /home/fkarg/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.75/build/main.rs:74:9

And sure enough, the include directory does not exist:

$ ls -lah /nix/store/0amx6k0b9i34kj5rbfq359s95mzj4xg8-openssl-1.1.1p-bin/bin/
Permissions Size User Date Modified Name
.r-xr-xr-x  6.8k root  1 Jan  1970  c_rehash*
.r-xr-xr-x  771k root  1 Jan  1970  openssl*

Help would very much be appreciated. I think there's something else that's missing here.

I'm also interested in getting this working.

I've had some success with nix-shell -p libiconv openssl pkgconfig

I, too, don't know about nix, but the error means rustc cannot find the (openssl-dev) .lib files where it is looking for them. I had similar error on debian, fixed it simply by sudo apt install openssl-dev. Maybe rustc also doesn't know all about nix?

This fixed the issue of Rust/pkg-config finding openssl for me: https://discourse.nixos.org/t/rust-openssl-woes/12340/3
Btw, I am using flake and NixOS, not using per project Nix configuration.

I tried to build aptos-lab/aptos-core to get the CLI tool aptos on NixOS with Home Manager and also ran into this crate's issue. @helinwang's setting PKG_CONFIG_PATH = "${pkgs.openssl.dev}/lib/pkgconfig"; did not work for me. Manually setting export PKG_CONFIG_PATH=/nix/store/g8jc4yh3hgvwjp2q420ddgvc3gb2rpkd-openssl-3.0.8-dev/lib/pkgconfig did seem to help.

In my experience, this works with a combination of:

rustPlatform.buildRustPackage {
  nativeBuildInputs = lib.optionals stdenv.isLinux [ pkg-config ];
  # Needed to get openssl-sys to use pkg-config.
  # Doesn't seem to like OpenSSL 3
  OPENSSL_NO_VENDOR = 1;

  buildInputs = lib.optionals stdenv.isLinux [ openssl_1_1 ];

  ...
}

If anyone reading this issue has had the following issue: Package openssl was not found in the pkg-config search path.

The solution I found was setting OPENSSL_DEV=pkgs.openssl.dev; in environment.variables (alongside installing openssl and pkg-config, of course)

What seems to work for me was export PKG_CONFIG_PATH=/nix/store/z4vl5vjz1a479wmbm10mhdpqygg5b3vl-system-path/lib/pkgconfig:$PKG_CONFIG_PATH

which I found with find /nix/store -name "openssl*" because the /run/current-system/sw/lib/ path was always failing

Here is what I needed to put in my configuration.nix to get it working:

# configuration.nix
{ pkgs, ... }: {
  environment.systemPackages = [ pkgs.pkg-config ];
  # set environment variable for fish shell
  programs.fish.shellInit = "export PKG_CONFIG_PATH=${pkgs.openssl.dev}/lib/pkgconfig";
}

For folks trying to build this crate with the vendored flag on nix: it's currently impossible (hence why many people shared solutions using OPENSSL_NO_VENDOR = 1), but I opened #2122 to fix this.

  environment.systemPackages = with pkgs; [
    # ...
    openssl
  ];
  environment.variables = {
    PKG_CONFIG_PATH="${pkgs.openssl.dev}/lib/pkgconfig";
  };

sudo nixos-rebuild switch then relog.

  environment.systemPackages = with pkgs; [
    # ...
    openssl
  ];
  environment.variables = {
    PKG_CONFIG_PATH="${pkgs.openssl.dev}/lib/pkgconfig";
  };

sudo nixos-rebuild switch then relog.

Can confirm that setting this environment variable makes this crate compile on NixOS.