tab.find_elements() performance is 150x worse than JS

[bricef]

Issue

Running tab.find_elements() takes >150 times as long as running document.querySelectorAll() in Javascript.

I'm not sure if I'm doing something wrong or if this the expected performance, but this performance is prohibitive for my use case. Are there performance tricks I don't know about? The code seems to delegate to the browser's document.querySelectorAll() so I'm not sure where the time is being spent.

On the same url:

Rust

let start = Instant::now();
let es = tab.find_elements("[href]")?;
let duration = start.elapsed();
println!("Time elapsed in find_elements() is: {:?}", duration);

Output

Time elapsed in find_elements() is: 46.614745743s

JS

console.time("test"); 
document.querySelectorAll("[href]");
console.timeEnd("test")

Output

test: 0.260009765625 ms

Investigations

Looking at the headless_chrome source leads me to browser::transport::Transport.call_method(), but it's not clear where the time is being spent. I'm still starting on my rust journey, so I'm not familiar with the profiler tooling at this time to understand the details of this performance issue.

[bricef]

Update

I created a minimum viable test case:

use std::time::Instant;
use headless_chrome::Browser;

#[tokio::main]
async fn main() -> Result<(), anyhow::Error>{
    let browser = Browser::default()?;
    let tab = browser.new_tab()?;
    tab.navigate_to("https://en.wikipedia.org")?;
    tab.wait_until_navigated()?;

    let start = Instant::now();
    let _es = tab.find_elements("[href]")?;
    let duration = start.elapsed();

    println!("Time elapsed in find_elements() is: {:?}", duration);
    Ok(())
}

Compiling it with debug symbols and running perf/flamegraph I get the following:

Which suggests that the majority of the time is spent in chrome. I'm not sure why chrome would take so long, and I'm running against a production build of chromium (so no debug symbols...). Will continue to investigate.

[bricef]

On the headless_chrome crate front, the de/serialization of events from the websocket seems to be the hot zone, which is not unexpected.

[chenshuiluke]

Did you ever end up getting further in your investigation?

[juddbaguio]

Any improvements on this?