Class LogBufferManager
log records are written to disk as blocks of data. Block size is a multiple of 512 data to assure optimum disk performance.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionclass
static interface
(package private) class
helper thread to flush buffers that have threads waiting longer than configured maximum.class
static interface
class
static interface
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate LogBuffer[]
array of all LogBuffer objects allocated.private final Object
mutex for synchronizing access to buffers list.(package private) int
number of buffers waiting to be forced.private LogBuffer
The LogBuffer that is currently being filled.(package private) final LogBufferManager.FlushManager
thread used to flush long waiting buffersprivate static final String
name of flush manager threadprivate final boolean
private long
number of times channel.force() called.(package private) long
private final Object
mutex for synchronizing threads through the portion of force() that forces the channel.(package private) long
(package private) long
(package private) long
(package private) long
private LogBuffer[]
queue of buffers waiting to be written.(package private) final boolean
indicates if a force() must be called in the flush() method.private int
next get workerID from forceQueue .private int
next put workerID into forceQueue .private LogBuffer[]
array of LogBuffer objects available for fillingprivate int
number of times buffer size was increased because of threads waiting for buffers.private boolean
boolean is set true when an IOException is returned by a write or force to a log file.private IOException
The last IOException returned to the logger(package private) int
last BSN forced to log.private long
time of last force used to compute totalTimeBetweenForceprivate LogFileManager
reference to LogFileManager that owns this Buffer Manager instance.private int
maximum number of buffers forced by channel.force()private int
private long
private long
maximum time (ms) for any single writeprivate int
minimum number of buffers forced by channel.force().private long
(package private) int
next block sequence number for fillBuffer.(package private) short
workerID into freeBuffer list maintained in getBuffer.(package private) int
next BSN to be written to log.private long
number of times buffer was forced because it is full.(package private) long
LogBuffer.tod from previous buffer written.private int
number of threads waiting for a forceprivate int
private long
total amount of time spent in channel.force();private long
private long
total time between channel.force() callsprivate long
total amount of time (ms) spent waiting for the forceMangerLockprivate long
total amount of time spent in channel.write();private long
number of times there were no buffers available.private long
number of times channel.write() called. -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescription(package private) int
bsnFromMark
(long mark) returns the BSN value portion of a log key mark .(package private) void
close()
Shutdown any threads that are started by this LogBufferManager instance.private String
doubleToString
(double val, int decimalPlaces) convert a double to String with fixed number of decimal places(package private) final long
elapsedTime
(long startTime) compute elapsed time for an event(package private) void
flushAll()
flush active buffers to disk and wait for all LogBuffers to be returned to the freeBuffer pool.private void
force
(boolean timeout) forces buffer to disk.(package private) void
Force the current buffer to disk before starting a replay().(package private) void
Add a buffer to the forceQueue.private LogBuffer
returns a LogBuffer to be filled.(package private) LogBuffer
getLogBuffer
(int index) return a new instance of LogBuffer.(package private) String
getStats()
Returns an XML node containing statistics for the LogBufferManager.final long
provides synchronized access to waitForBuffer(package private) void
init
(LogFileManager lfm, int bsn) perform initialization following reposition of LogFileManager.(package private) long
markFromBsn
(int bsn, int offset) generate a log mark (log key).(package private) void
open()
Allocate pool of IO buffers for Logger.(package private) long
put
(short type, byte[][] data, boolean sync) writes data byte[][] to log and returns a log key.private void
releaseBuffer
(LogBuffer buffer) decrements count of threads waiting on this buffer.(package private) void
replay
(ReplayListener listener, long mark, boolean replayCtrlRecords) Replays log from requested mark forward to end of log.private void
Waits for logBuffer to be forced to disk.
-
Field Details
-
flushPartialBuffers
private final boolean flushPartialBuffers- See Also:
-
haveIOException
private boolean haveIOExceptionboolean is set true when an IOException is returned by a write or force to a log file.Any attempt to write or force after haveIOException becomes true should result in an IOException being returned to the caller.
-
ioexception
The last IOException returned to the logger -
bufferManagerLock
mutex for synchronizing access to buffers list.also synchronizes access to fqPut in routines that put LogBuffers into the forceQueue[].
-
forceManagerLock
mutex for synchronizing threads through the portion of force() that forces the channel. -
lfm
reference to LogFileManager that owns this Buffer Manager instance.- See Also:
-
forceRequired
final boolean forceRequiredindicates if a force() must be called in the flush() method.Set false in constructor if config.getLogFileMode() is "rwd".
-
fillBuffer
The LogBuffer that is currently being filled. -
freeBuffer
array of LogBuffer objects available for filling -
bufferList
array of all LogBuffer objects allocated.Used to find and debug buffers that are not in the freeBuffer list if logger hangs waiting for buffers to be returned to the freeBuffer pool.
-
nextIndex
short nextIndexworkerID into freeBuffer list maintained in getBuffer. -
waitForBuffer
private long waitForBuffernumber of times there were no buffers available.The FlushManager thread monitors this field to determine if the buffer pool needs to be grown.
-
noRoomInBuffer
private long noRoomInBuffernumber of times buffer was forced because it is full. -
growPoolCounter
private int growPoolCounternumber of times buffer size was increased because of threads waiting for buffers. -
nextFillBSN
int nextFillBSNnext block sequence number for fillBuffer. -
nextWriteBSN
int nextWriteBSNnext BSN to be written to log.synchronized by forceManagerLock
-
prevWriteTOD
long prevWriteTODLogBuffer.tod from previous buffer written.maintained in force() method. Used to check against decrement in TOD field. Added to help investigate BUG 303907
-
buffersWaitingForce
int buffersWaitingForcenumber of buffers waiting to be forced.synchronized by bufferManagerLock.
incremented in put() and decremented in releaseBuffer(). When a thread calls put() with sync parameter set true, and buffersWaitingForce is also zero, then put() causes the buffer to be forced immediately. This strategy minimizes latency in situations of low load, such as a single thread running.
-
lastForceBSN
int lastForceBSNlast BSN forced to log.synchronized by forceManagerLock
-
forceCount
private long forceCountnumber of times channel.force() called. -
writeCount
private long writeCountnumber of times channel.write() called. -
minBuffersForced
private int minBuffersForcedminimum number of buffers forced by channel.force(). -
maxBuffersForced
private int maxBuffersForcedmaximum number of buffers forced by channel.force() -
totalForceTime
private long totalForceTimetotal amount of time spent in channel.force(); -
totalWriteTime
private long totalWriteTimetotal amount of time spent in channel.write(); -
maxWriteTime
private long maxWriteTimemaximum time (ms) for any single write -
totalWaitForWriteLockTime
private long totalWaitForWriteLockTimetotal amount of time (ms) spent waiting for the forceMangerLock -
totalTimeBetweenForce
private long totalTimeBetweenForcetotal time between channel.force() calls -
minTimeBetweenForce
private long minTimeBetweenForce -
maxTimeBetweenForce
private long maxTimeBetweenForce -
lastForceTOD
private long lastForceTODtime of last force used to compute totalTimeBetweenForce -
threadsWaitingForce
private int threadsWaitingForcenumber of threads waiting for a force -
maxThreadsWaitingForce
private int maxThreadsWaitingForce -
totalThreadsWaitingForce
private long totalThreadsWaitingForce -
threadsWaitingForceThreshold
private int threadsWaitingForceThreshold -
forceOnTimeout
long forceOnTimeout -
forceNoWaitingThreads
long forceNoWaitingThreads -
forceHalfOfBuffers
long forceHalfOfBuffers -
forceMaxWaitingThreads
long forceMaxWaitingThreads -
forceOnFileSwitch
long forceOnFileSwitch -
flushManager
thread used to flush long waiting buffers -
flushManagerName
name of flush manager thread- See Also:
-
forceQueue
queue of buffers waiting to be written. The queue guarantees that buffers are written to disk in BSN order. Buffers are placed into the forceQueue using the fqPut workerID, and removed from the forceQueue using the fqGet workerID. Access to these two workerID members is synchronized using separate objects to allow most threads to be storing log records while a single thread is blocked waiting for a physical force.Buffers are added to the queue when put() detects the buffer is full, and when the FlushManager thread detects that a buffer has waited too long to be written. The fqPut member is the workerID of the next location in forceQueue to put a LogBuffer that is to be written. fqPut is protected by bufferManagerLock .
Buffers are removed from the queue in force() and written to disk. The fqGet member is an workerID to the next buffer to remove from the forceQueue. fqGet is protected by forceManagerLock .
The size of forceQueue[] is one larger than the size of freeBuffer[] so that fqPut == fqGet always means the queue is empty.
-
fqPut
private int fqPutnext put workerID into forceQueue .synchronized by bufferManagerLock.
-
fqGet
private int fqGetnext get workerID from forceQueue .synchronized by forceManagerLock.
-
-
Constructor Details
-
LogBufferManager
LogBufferManager(Configuration config) - Parameters:
config
- Configuration object
-
-
Method Details
-
elapsedTime
final long elapsedTime(long startTime) compute elapsed time for an event- Parameters:
startTime
- time event began- Returns:
- elapsed time (System.currentTimeMillis() - startTime)
-
force
forces buffer to disk.batches multiple buffers into a single force when possible.
Design Note:
It was suggested that using forceManagerLock to control writes from the forceQueue[] and forces would reduce overlap due to the amount of time that forceManagerLock is shut while channel.force() is active.Experimented with using two separate locks to manage the channel.write() and the channel.force() calls, but it appears that thread calling channel.force() will block another thread trying to call channel.write() so both locks end up being shut anyway. Since two locks did not provide any measurable benefit, it seems best to use a single forceManagerLock to keep the code simple.
- Throws:
IOException
InterruptedException
-
sync
Waits for logBuffer to be forced to disk.No monitors are owned when routine is entered.
Prior to calling sync(), the thread called put() with sync param set true to register the fact that the thread would wait for the force.
- Throws:
IOException
InterruptedException
-
releaseBuffer
decrements count of threads waiting on this buffer.If count goes to zero, buffer is returned to the freeBuffer list, and any threads waiting for a free buffer are notified.
- Parameters:
buffer
- LogBuffer to be released- See Also:
-
getFillBuffer
returns a LogBuffer to be filled.PRECONDITION: caller holds bufferManagerLock monitor.
- Returns:
- a LogBuffer to be filled.
- Throws:
LogFileOverflowException
-
getLogBuffer
return a new instance of LogBuffer.Actual LogBuffer implementation class is specified by configuration.
- Returns:
- a new instance of LogBuffer
- Throws:
ClassNotFoundException
-
fqAdd
Add a buffer to the forceQueue.PRECONDITION: bufferManagerLock owned by caller
- Parameters:
buffer
- LogBuffer to be added to the forceQueue
-
put
long put(short type, byte[][] data, boolean sync) throws LogRecordSizeException, LogFileOverflowException, InterruptedException, IOException writes data byte[][] to log and returns a log key.waits for IO to complete if sync is true.
MG 27/Jan/05 modified code to force buffer if caller has set sync == true, and there are no buffers waiting to be written. This causes buffers to be written immediately in a single threaded and/or low volume situation. Change suggested by developers at ApacheCon and at ObjectWebCon. This feature is disabled by default and is enabled by setting the log configuration property XXX to true.
- Returns:
- token reference (log key) for record just written
- Throws:
LogRecordSizeException
- when size of byte[] is larger than the maximum possible record for the configured buffer size.LogFileOverflowException
InterruptedException
IOException
- See Also:
-
forceCurrentBuffer
Force the current buffer to disk before starting a replay().- Throws:
IOException
-
replay
void replay(ReplayListener listener, long mark, boolean replayCtrlRecords) throws LogConfigurationException, InvalidLogKeyException Replays log from requested mark forward to end of log.Blocks caller until replay completes due to end of log, or an exception is passed to listener.onError().
- Parameters:
listener
- ReplayListener to receive notifications for each log record.mark
- log key for the first record to be replayed.If mark is zero then the entire active log is replayed.
replayCtrlRecords
- indicates whether to return control records.used by utility routines such as CopyLog.
- Throws:
InvalidLogKeyException
- if the requested key is not found in the log.LogConfigurationException
- See Also:
-
open
Allocate pool of IO buffers for Logger.The LogBufferManager class is a generalized manager for any type of LogBuffer. The class name for the type of LogBuffer to use is specified by configuration parameters.
- Throws:
ClassNotFoundException
- if the configured LogBuffer class cannot be found.
-
close
void close()Shutdown any threads that are started by this LogBufferManager instance. -
init
perform initialization following reposition of LogFileManager.- Parameters:
lfm
- LogFileManager used by the buffer manager to obtain log files for writing buffers.bsn
- last Block Sequence Number written by Logger.
-
flushAll
flush active buffers to disk and wait for all LogBuffers to be returned to the freeBuffer pool.May be called multiple times.
- Throws:
IOException
-
doubleToString
convert a double to String with fixed number of decimal places- Parameters:
val
- double to be converteddecimalPlaces
- number of decimal places in output- Returns:
- String result of conversion
-
getStats
String getStats()Returns an XML node containing statistics for the LogBufferManager.The nested
element contains entries for each LogBuffer object in the buffer pool. - Returns:
- a String containing statistics.
-
bsnFromMark
int bsnFromMark(long mark) returns the BSN value portion of a log key mark .- Parameters:
mark
- log key or log mark to extract BSN from.- Returns:
- BSN portion of mark
-
markFromBsn
long markFromBsn(int bsn, int offset) generate a log mark (log key).- Parameters:
bsn
- Block Sequence Number.offset
- offset within block.May be zero to allow access to the beginning of a block.
- Returns:
- a log key.
-
getWaitForBuffer
public final long getWaitForBuffer()provides synchronized access to waitForBuffer- Returns:
- the current value of waitForBuffer
-