Skip to content

Commit

Permalink
feat(is_pre_process): add config item
Browse files Browse the repository at this point in the history
add `is_pre_process` and make  it configurable
  • Loading branch information
Dirreke committed Dec 6, 2023
1 parent 4f80661 commit 859bf93
Show file tree
Hide file tree
Showing 10 changed files with 90 additions and 28 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ Cargo.lock
.idea/
*.iml
.vscode/
log/
24 changes: 20 additions & 4 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ my_rolling_appender:
base: 1
count: 5
pattern: "logs/test.{}.log"
is_pre_process: false
```

The new component is the _policy_ field. A policy must have `kind` like most
Expand Down Expand Up @@ -204,12 +205,16 @@ unit is not specified), case does not matter:
- month[s]
- year[s]

> note: The log file will be rolled at the integer time. For example, if the
> Note: The log file will be rolled at the integer time. For example, if the
`limit` is set to `2 day`, the log file will be rolled at 0:00 every other a
day, regardless of the time `log4rs` was started or the log file was created.
This means that the initial log file will be likely rolled before the limit
is reached.

> Note: Setting "roller.is_pre_process"(In the next section, we will introduce)
to true can ensure that the logs are rolled in the correct position instead of
leaving a single line of logs in the previous log file.

i.e.

```yml
Expand All @@ -219,9 +224,9 @@ trigger:
```

The _roller_ field supports two types: delete, and fixed_window. The delete
roller does not take any other configuration fields. The fixed_window roller
supports three fields: pattern, base, and count. The most current log file will
always have the _base_ index.
roller supports one fileds: is_pre_process. The fixed_window roller supports
four fields: pattern, base, count and is_pre_process. The most current log
file will always have the _base_ index.

The _pattern_ field is used to rename files. The pattern must contain the
double curly brace `{}`. For example `archive/foo.{}.log`. Each instance of
Expand All @@ -238,6 +243,15 @@ The _count_ field is the exclusive maximum index used to name rolling files.
However, be warned that the roller renames every file when a log rolls over.
Having a large count value can negatively impact performance.

The _is_pre_process_ filed is optional. It supports two value: true and false.
It defaults to false, which means that the file will be rolled after the file
get written.

> Note: The default value have higher performance. However, if you use the
`triger: time`, setting it to true will ensure that the logs are rolled in the
correct position instead of leaving a single line of logs in the previous log
file.

i.e.

```yml
Expand All @@ -246,13 +260,15 @@ roller:
base: 1
count: 5
pattern: "archive/journey-service.{}.log"
is_pre_process: false
```

or

```yml
roller:
kind: delete
is_pre_process: false
```

## Refresh Rate
Expand Down
4 changes: 2 additions & 2 deletions examples/compile_time_config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ fn main() {
let config = serde_yaml::from_str(config_str).unwrap();
log4rs::init_raw_config(config).unwrap();

info!("Goes to console");
error!("Goes to console");
info!("Goes to console, file and rolling file");
error!("Goes to console, file and rolling file");
trace!("Doesn't go to console as it is filtered out");
}
7 changes: 4 additions & 3 deletions examples/sample_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ appenders:
level: info
file:
kind: file
path: "log/log.log"
path: "log/file.log"
encoder:
pattern: "[{d(%Y-%m-%dT%H:%M:%S%.6f)} {h({l}):<5.5} {M}] {m}{n}"
rollingfile:
kind: rolling_file
path: "log/log2.log"
path: "log/rolling_file.log"
encoder:
pattern: "[{d(%Y-%m-%dT%H:%M:%S%.6f)} {h({l}):<5.5} {M}] {m}{n}"
policy:
Expand All @@ -22,9 +22,10 @@ appenders:
limit: 1 minute
roller:
kind: fixed_window
pattern: "log/old-log-{}.log"
pattern: "log/old-rolling_file-{}.log"
base: 0
count: 2
is_pre_process: true
root:
level: info
appenders:
Expand Down
45 changes: 31 additions & 14 deletions src/append/rolling_file/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -167,25 +167,39 @@ impl Append for RollingFileAppender {
// TODO(eas): Perhaps this is better as a concurrent queue?
let mut writer = self.writer.lock();

let is_pre_process = self.policy.is_pre_process();
let log_writer = self.get_writer(&mut writer)?;
let len = log_writer.len;

let mut file = LogFile {
writer: &mut writer,
path: &self.path,
len,
};
if is_pre_process {
let len = log_writer.len;

let mut file = LogFile {
writer: &mut writer,
path: &self.path,
len,
};

// TODO(eas): Idea: make this optionally return a future, and if so, we initialize a queue for
// data that comes in while we are processing the file rotation.

// TODO(eas): Idea: make this optionally return a future, and if so, we initialize a queue for
// data that comes in while we are processing the file rotation.
self.policy.process(&mut file)?;

//first, rotate
self.policy.process(&mut file)?;
let log_writer_new = self.get_writer(&mut writer)?;
self.encoder.encode(log_writer_new, record)?;
log_writer_new.flush()?;
} else {
self.encoder.encode(log_writer, record)?;
log_writer.flush()?;
let len = log_writer.len;

//second, write
let writer_file = self.get_writer(&mut writer)?;
self.encoder.encode(writer_file, record)?;
writer_file.flush()?;
let mut file = LogFile {
writer: &mut writer,
path: &self.path,
len,
};

self.policy.process(&mut file)?;
}

Ok(())
}
Expand Down Expand Up @@ -410,6 +424,9 @@ appenders:
fn process(&self, _: &mut LogFile) -> anyhow::Result<()> {
Ok(())
}
fn is_pre_process(&self) -> bool {
false
}
}

#[test]
Expand Down
4 changes: 4 additions & 0 deletions src/append/rolling_file/policy/compound/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ impl Policy for CompoundPolicy {
}
Ok(())
}

fn is_pre_process(&self) -> bool {
self.roller.is_pre_process()
}
}

/// A deserializer for the `CompoundPolicyDeserializer`.
Expand Down
11 changes: 8 additions & 3 deletions src/append/rolling_file/policy/compound/roll/delete.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,23 @@ use crate::config::{Deserialize, Deserializers};
#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug, Default, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRollerConfig {
#[serde(skip_deserializing)]
_p: (),
is_pre_process: Option<bool>,
}

/// A roller which deletes the log file.
#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug, Default)]
pub struct DeleteRoller(());
pub struct DeleteRoller {
is_pre_process: bool,
}

impl Roll for DeleteRoller {
fn roll(&self, file: &Path) -> anyhow::Result<()> {
fs::remove_file(file).map_err(Into::into)
}

fn is_pre_process(&self) -> bool {
self.is_pre_process
}
}

impl DeleteRoller {
Expand Down
18 changes: 16 additions & 2 deletions src/append/rolling_file/policy/compound/roll/fixed_window.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ pub struct FixedWindowRollerConfig {
pattern: String,
base: Option<u32>,
count: u32,
is_pre_process: Option<bool>,
}

#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug)]
Expand Down Expand Up @@ -87,14 +88,15 @@ pub struct FixedWindowRoller {
compression: Compression,
base: u32,
count: u32,
is_pre_process: bool,
#[cfg(feature = "background_rotation")]
cond_pair: Arc<(Mutex<bool>, Condvar)>,
}

impl FixedWindowRoller {
/// Returns a new builder for the `FixedWindowRoller`.
pub fn builder() -> FixedWindowRollerBuilder {
FixedWindowRollerBuilder { base: 0 }
FixedWindowRollerBuilder::default()
}
}

Expand Down Expand Up @@ -155,6 +157,10 @@ impl Roll for FixedWindowRoller {

Ok(())
}

fn is_pre_process(&self) -> bool {
self.is_pre_process
}
}

fn move_file<P, Q>(src: P, dst: Q) -> io::Result<()>
Expand Down Expand Up @@ -239,6 +245,7 @@ fn rotate(
#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug, Default)]
pub struct FixedWindowRollerBuilder {
base: u32,
is_pre_process: bool,
}

impl FixedWindowRollerBuilder {
Expand All @@ -249,7 +256,13 @@ impl FixedWindowRollerBuilder {
self.base = base;
self
}

/// Sets the is_pre_process flag for log files.
///
/// Defaults to fa;se
pub fn is_pre_process(mut self, b: bool) -> FixedWindowRollerBuilder {
self.is_pre_process = b;
self
}
/// Constructs a new `FixedWindowRoller`.
///
/// `pattern` is either an absolute path or lacking a leading `/`, relative
Expand Down Expand Up @@ -283,6 +296,7 @@ impl FixedWindowRollerBuilder {
compression,
base: self.base,
count,
is_pre_process: self.is_pre_process,
#[cfg(feature = "background_rotation")]
cond_pair: Arc::new((Mutex::new(true), Condvar::new())),
})
Expand Down
2 changes: 2 additions & 0 deletions src/append/rolling_file/policy/compound/roll/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ pub trait Roll: fmt::Debug + Send + Sync + 'static {
/// If this method returns successfully, there *must* no longer be a file
/// at the specified location.
fn roll(&self, file: &Path) -> anyhow::Result<()>;
/// Return the config `Roll.is_pre_process` value
fn is_pre_process(&self) -> bool;
}

#[cfg(feature = "config_parsing")]
Expand Down
2 changes: 2 additions & 0 deletions src/append/rolling_file/policy/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ pub trait Policy: Sync + Send + 'static + fmt::Debug {
/// This method is called after each log event. It is provided a reference
/// to the current log file.
fn process(&self, log: &mut LogFile) -> anyhow::Result<()>;
/// Return the config `Roll.is_pre_process` value
fn is_pre_process(&self) -> bool;
}

#[cfg(feature = "config_parsing")]
Expand Down

0 comments on commit 859bf93

Please sign in to comment.