Describe your environment
- Platform: macOS
- opentelemetry-cpp version: main branch
- Component:
sdk/src/logs/batch_log_record_processor.cc
- BatchLogRecordProcessor configuration:
max_queue_size = 8192max_export_batch_size = 2048schedule_delay_millis = 5000
Steps to reproduce
- Configure
BatchLogRecordProcessor with a LogRecordExporter that counts export calls and records per call.
- Produce log records at a constant rate (e.g. 5,000 logs/s) for 10 seconds.
- Observe the number of export requests and the average number of logs per request.
- Produce 50,000 log records without calling
ForceFlush().
What is the expected behavior?
With max_export_batch_size = 2048, exports should be close to 2048 records each. After #4466, BatchSpanProcessor behaves this way: on a normal wakeup it exports at most one batch, and only ForceFlush() / Shutdown() drain the entire buffer.
BatchLogRecordProcessor should follow the same semantics.
What is the actual behavior?
BatchLogRecordProcessor::Export() still drains the whole buffer in a tight loop:
void BatchLogRecordProcessor::Export()
{
do {
bool notify_force_flush =
synchronization_data_->is_force_flush_pending.exchange(false, std::memory_order_acq_rel);
if (notify_force_flush) {
num_records_to_export = buffer_.size();
} else {
num_records_to_export =
buffer_.size() >= max_export_batch_size_ ? max_export_batch_size_ : buffer_.size();
}
// ... consume & export ...
} while (true);
}And the worker wakes up as soon as the buffer is non-empty:
synchronization_data_->cv.wait_for(lk, timeout, [this] {
...
return !buffer_.empty();
});Measured result
| Metric |
Value |
| Logs produced |
50,000 |
| Export calls |
2,332 |
| Average batch size |
21.4 |
| Full batches (>=2048) |
1 |
| Tiny batches (<100) |
2,331 |
The first export is 2048 records; every subsequent export in the same wakeup is ~20–25 records because the worker keeps draining the buffer instead of waiting for the next full batch.
Additional context
Describe your environment
sdk/src/logs/batch_log_record_processor.ccmax_queue_size = 8192max_export_batch_size = 2048schedule_delay_millis = 5000Steps to reproduce
BatchLogRecordProcessorwith aLogRecordExporterthat counts export calls and records per call.ForceFlush().What is the expected behavior?
With
max_export_batch_size = 2048, exports should be close to 2048 records each. After #4466,BatchSpanProcessorbehaves this way: on a normal wakeup it exports at most one batch, and onlyForceFlush()/Shutdown()drain the entire buffer.BatchLogRecordProcessorshould follow the same semantics.What is the actual behavior?
BatchLogRecordProcessor::Export()still drains the whole buffer in a tight loop:And the worker wakes up as soon as the buffer is non-empty:
Measured result
The first export is 2048 records; every subsequent export in the same wakeup is ~20–25 records because the worker keeps draining the buffer instead of waiting for the next full batch.
Additional context
BatchSpanProcessordrain the queue in a tight loop instead of waiting for the next full batch #4449 reported the same problem forBatchSpanProcessor.Export()drain the buffer only forForceFlush()/Shutdown().sdk/src/logs/batch_log_record_processor.cc) still uses the old implementation and should be aligned with the trace path.