Skip to content

Commit

Permalink
Support custom backup directory and time format
Browse files Browse the repository at this point in the history
yinonavraham committed Sep 18, 2019
1 parent 3d09ab9 commit 8511fb9
Showing 3 changed files with 303 additions and 158 deletions.
59 changes: 29 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
# lumberjack [![GoDoc](https://godoc.org/gopkg.in/natefinch/lumberjack.v2?status.png)](https://godoc.org/gopkg.in/natefinch/lumberjack.v2) [![Build Status](https://travis-ci.org/natefinch/lumberjack.svg?branch=v2.0)](https://travis-ci.org/natefinch/lumberjack) [![Build status](https://ci.appveyor.com/api/projects/status/00gchpxtg4gkrt5d)](https://ci.appveyor.com/project/natefinch/lumberjack) [![Coverage Status](https://coveralls.io/repos/natefinch/lumberjack/badge.svg?branch=v2.0)](https://coveralls.io/r/natefinch/lumberjack?branch=v2.0)
# lumberjack [![GoDoc](https://godoc.org/github.com/jfrog/lumberjack?status.png)](https://godoc.org/github.com/jfrog/lumberjack)

This package was forked from: https://github.com/natefinch/lumberjack, version 2.0

### Lumberjack is a Go package for writing logs to rolling files.

Package lumberjack provides a rolling logger.

Note that this is v2.0 of lumberjack, and should be imported using gopkg.in
thusly:

import "gopkg.in/natefinch/lumberjack.v2"
The package should be imported using the following:

The package name remains simply lumberjack, and the code resides at
https://github.com/natefinch/lumberjack under the v2.0 branch.
import "github.com/jfrog/lumberjack/v2"

Lumberjack is intended to be one part of a logging infrastructure.
It is not an all-in-one solution, but instead is a pluggable
@@ -47,8 +45,8 @@ log.SetOutput(&lumberjack.Logger{
``` go
type Logger struct {
// Filename is the file to write logs to. Backup log files will be retained
// in the same directory. It uses <processname>-lumberjack.log in
// os.TempDir() if empty.
// in the same directory, or where defined by `BackupDir`.
// It uses <processname>-lumberjack.log in os.TempDir() if empty.
Filename string `json:"filename" yaml:"filename"`

// MaxSize is the maximum size in megabytes of the log file before it gets
@@ -75,40 +73,52 @@ type Logger struct {
// Compress determines if the rotated log files should be compressed
// using gzip. The default is not to perform compression.
Compress bool `json:"compress" yaml:"compress"`

// TimeFormat determines the format to use for formatting the timestamp in
// backup files. The default format is defined in `DefaultTimeFormat`.
TimeFormat string `json:"timeformat" yaml:"timeformat"`

// BackupDir is the directory where backup files shall be saved to. The
// default is empty string which is resolved to where the active log file
// is located.
BackupDir string `json:"backupdir" yaml:"backupdir"`
// contains filtered or unexported fields
}
```
Logger is an io.WriteCloser that writes to the specified filename.
Logger is an `io.WriteCloser` that writes to the specified filename.

Logger opens or creates the logfile on first Write. If the file exists and
is less than MaxSize megabytes, lumberjack will open and append to that file.
If the file exists and its size is >= MaxSize megabytes, the file is renamed
is less than `MaxSize` megabytes, lumberjack will open and append to that file.
If the file exists and its size is `>= MaxSize` megabytes, the file is renamed
by putting the current time in a timestamp in the name immediately before the
file's extension (or the end of the filename if there's no extension). A new
log file is then created using original filename.

Whenever a write would cause the current log file exceed MaxSize megabytes,
Whenever a write would cause the current log file exceed `MaxSize` megabytes,
the current file is closed, renamed, and a new log file created with the
original name. Thus, the filename you give Logger is always the "current" log
original name. Thus, the filename you give `Logger` is always the "current" log
file.

Backups use the log file name given to Logger, in the form `name-timestamp.ext`
where name is the filename without the extension, timestamp is the time at which
the log was rotated formatted with the time.Time format of
the log was rotated formatted with the default `time.Time` format of
`2006-01-02T15-04-05.000` and the extension is the original extension. For
example, if your Logger.Filename is `/var/log/foo/server.log`, a backup created
example, if your `Logger.Filename` is `/var/log/foo/server.log`, a backup created
at 6:30pm on Nov 11 2016 would use the filename
`/var/log/foo/server-2016-11-04T18-30-00.000.log`

The backup files name and location can be customized using the `Logger`'s `BackupDir`
and `TimeFormat` optional fields.

### Cleaning Up Old Log Files
Whenever a new logfile gets created, old log files may be deleted. The most
recent files according to the encoded timestamp will be retained, up to a
number equal to MaxBackups (or all of them if MaxBackups is 0). Any files
number equal to `MaxBackups` (or all of them if `MaxBackups` is `0`). Any files
with an encoded timestamp older than MaxAge days are deleted, regardless of
MaxBackups. Note that the time encoded in the timestamp is the rotation
`MaxBackups`. Note that the time encoded in the timestamp is the rotation
time, which may differ from the last time that file was written to.

If MaxBackups and MaxAge are both 0, no old log files will be deleted.
If `MaxBackups` and `MaxAge` are both `0`, no old log files will be deleted.



@@ -166,14 +176,3 @@ Write implements io.Writer. If a write would cause the log file to be larger
than MaxSize, the file is closed, renamed to include a timestamp of the
current time, and a new log file is created using the original log file name.
If the length of the write is greater than MaxSize, an error is returned.









- - -
Generated by [godoc2md](http://godoc.org/github.com/davecheney/godoc2md)
76 changes: 53 additions & 23 deletions lumberjack.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
// Package lumberjack provides a rolling logger.
//
// Note that this is v2.0 of lumberjack, and should be imported using gopkg.in
// thusly:
// The package should be imported using the following:
//
// import "gopkg.in/natefinch/lumberjack.v2"
// import "github.com/jfrog/lumberjack/v2"
//
// The package name remains simply lumberjack, and the code resides at
// https://github.com/natefinch/lumberjack under the v2.0 branch.
@@ -36,9 +35,9 @@ import (
)

const (
backupTimeFormat = "2006-01-02T15-04-05.000"
compressSuffix = ".gz"
defaultMaxSize = 100
DefaultTimeFormat = "2006-01-02T15-04-05.000"
compressSuffix = ".gz"
defaultMaxSize = 100
)

// ensure we always implement io.WriteCloser
@@ -58,13 +57,16 @@ var _ io.WriteCloser = (*Logger)(nil)
// original name. Thus, the filename you give Logger is always the "current" log
// file.
//
// Backups use the log file name given to Logger, in the form
// `name-timestamp.ext` where name is the filename without the extension,
// timestamp is the time at which the log was rotated formatted with the
// time.Time format of `2006-01-02T15-04-05.000` and the extension is the
// original extension. For example, if your Logger.Filename is
// `/var/log/foo/server.log`, a backup created at 6:30pm on Nov 11 2016 would
// use the filename `/var/log/foo/server-2016-11-04T18-30-00.000.log`
// Backups use the log file name given to Logger, in the form `name-timestamp.ext`
// where name is the filename without the extension, timestamp is the time at which
// the log was rotated formatted with the default time.Time format of
// `2006-01-02T15-04-05.000` and the extension is the original extension. For
// example, if your `Logger.Filename` is `/var/log/foo/server.log`, a backup created
// at 6:30pm on Nov 11 2016 would use the filename
// `/var/log/foo/server-2016-11-04T18-30-00.000.log`
//
// The backup files name and location can be customized using the Logger's BackupDir
// and TimeFormat optional fields.
//
// Cleaning Up Old Log Files
//
@@ -78,8 +80,8 @@ var _ io.WriteCloser = (*Logger)(nil)
// If MaxBackups and MaxAge are both 0, no old log files will be deleted.
type Logger struct {
// Filename is the file to write logs to. Backup log files will be retained
// in the same directory. It uses <processname>-lumberjack.log in
// os.TempDir() if empty.
// in the same directory, or where defined by `BackupDir`.
// It uses <processname>-lumberjack.log in os.TempDir() if empty.
Filename string `json:"filename" yaml:"filename"`

// MaxSize is the maximum size in megabytes of the log file before it gets
@@ -107,6 +109,15 @@ type Logger struct {
// using gzip. The default is not to perform compression.
Compress bool `json:"compress" yaml:"compress"`

// TimeFormat determines the format to use for formatting the timestamp in
// backup files. The default format is defined in `DefaultTimeFormat`.
TimeFormat string `json:"timeformat" yaml:"timeformat"`

// BackupDir is the directory where backup files shall be saved to. The
// default is empty string which is resolved to where the active log file
// is located.
BackupDir string `json:"backupdir" yaml:"backupdir"`

size int64
file *os.File
mu sync.Mutex
@@ -218,7 +229,11 @@ func (l *Logger) openNew() error {
// Copy the mode off the old logfile.
mode = info.Mode()
// move the existing file
newname := backupName(name, l.LocalTime)
newname := l.backupName(name, l.LocalTime)
err := os.MkdirAll(filepath.Dir(newname), 0755)
if err != nil {
return fmt.Errorf("can't make directories for backup logfile: %s", err)
}
if err := os.Rename(name, newname); err != nil {
return fmt.Errorf("can't rename log file: %s", err)
}
@@ -244,8 +259,8 @@ func (l *Logger) openNew() error {
// backupName creates a new filename from the given name, inserting a timestamp
// between the filename and the extension, using the local time if requested
// (otherwise UTC).
func backupName(name string, local bool) string {
dir := filepath.Dir(name)
func (l *Logger) backupName(name string, local bool) string {
dir := l.backupDir()
filename := filepath.Base(name)
ext := filepath.Ext(filename)
prefix := filename[:len(filename)-len(ext)]
@@ -254,10 +269,24 @@ func backupName(name string, local bool) string {
t = t.UTC()
}

timestamp := t.Format(backupTimeFormat)
timestamp := t.Format(l.timeFormat())
return filepath.Join(dir, fmt.Sprintf("%s-%s%s", prefix, timestamp, ext))
}

func (l *Logger) backupDir() string {
if l.BackupDir != "" {
return l.BackupDir
}
return l.dir()
}

func (l *Logger) timeFormat() string {
if l.TimeFormat != "" {
return l.TimeFormat
}
return DefaultTimeFormat
}

// openExistingOrNew opens the logfile if it exists and if the current write
// would not put it over MaxSize. If there is no such file or the write would
// put it over the MaxSize, a new file is created.
@@ -311,6 +340,7 @@ func (l *Logger) millRunOnce() error {
return err
}

backupDir := l.backupDir()
var compress, remove []logInfo

if l.MaxBackups > 0 && l.MaxBackups < len(files) {
@@ -357,13 +387,13 @@ func (l *Logger) millRunOnce() error {
}

for _, f := range remove {
errRemove := os.Remove(filepath.Join(l.dir(), f.Name()))
errRemove := os.Remove(filepath.Join(backupDir, f.Name()))
if err == nil && errRemove != nil {
err = errRemove
}
}
for _, f := range compress {
fn := filepath.Join(l.dir(), f.Name())
fn := filepath.Join(backupDir, f.Name())
errCompress := compressLogFile(fn, fn+compressSuffix)
if err == nil && errCompress != nil {
err = errCompress
@@ -398,7 +428,7 @@ func (l *Logger) mill() {
// oldLogFiles returns the list of backup log files stored in the same
// directory as the current log file, sorted by ModTime
func (l *Logger) oldLogFiles() ([]logInfo, error) {
files, err := ioutil.ReadDir(l.dir())
files, err := ioutil.ReadDir(l.backupDir())
if err != nil {
return nil, fmt.Errorf("can't read log file directory: %s", err)
}
@@ -438,7 +468,7 @@ func (l *Logger) timeFromName(filename, prefix, ext string) (time.Time, error) {
return time.Time{}, errors.New("mismatched extension")
}
ts := filename[len(prefix) : len(filename)-len(ext)]
return time.Parse(backupTimeFormat, ts)
return time.Parse(l.timeFormat(), ts)
}

// max returns the maximum size in bytes of log files before rolling.
326 changes: 221 additions & 105 deletions lumberjack_test.go
Original file line number Diff line number Diff line change
@@ -96,7 +96,7 @@ func TestWriteTooLong(t *testing.T) {

func TestMakeLogDir(t *testing.T) {
currentTime = fakeTime
dir := time.Now().Format("TestMakeLogDir" + backupTimeFormat)
dir := time.Now().Format("TestMakeLogDir" + DefaultTimeFormat)
dir = filepath.Join(os.TempDir(), dir)
defer os.RemoveAll(dir)
filename := logFile(dir)
@@ -445,43 +445,60 @@ func TestMaxAge(t *testing.T) {
}

func TestOldLogFiles(t *testing.T) {
currentTime = fakeTime
megabyte = 1

dir := makeTempDir("TestOldLogFiles", t)
defer os.RemoveAll(dir)

filename := logFile(dir)
data := []byte("data")
err := ioutil.WriteFile(filename, data, 07)
isNil(err, t)

// This gives us a time with the same precision as the time we get from the
// timestamp in the name.
t1, err := time.Parse(backupTimeFormat, fakeTime().UTC().Format(backupTimeFormat))
isNil(err, t)

backup := backupFile(dir)
err = ioutil.WriteFile(backup, data, 07)
isNil(err, t)

newFakeTime()

t2, err := time.Parse(backupTimeFormat, fakeTime().UTC().Format(backupTimeFormat))
isNil(err, t)

backup2 := backupFile(dir)
err = ioutil.WriteFile(backup2, data, 07)
isNil(err, t)

l := &Logger{Filename: filename}
files, err := l.oldLogFiles()
isNil(err, t)
equals(2, len(files), t)

// should be sorted by newest file first, which would be t2
equals(t2, files[0].timestamp, t)
equals(t1, files[1].timestamp, t)
forEachBackupTestSpec(t, func(t *testing.T, test backupTestSpec) {
currentTime = fakeTime
megabyte = 1

dir := makeTempDir("TestOldLogFiles", t)
defer os.RemoveAll(dir)
var backupDir string
effectiveBackupDir := dir
if test.customBackupDir {
backupDir = makeTempDir("TestOldLogFilesBackup", t)
defer os.RemoveAll(backupDir)
effectiveBackupDir = backupDir
}

filename := logFile(dir)
data := []byte("data")
err := ioutil.WriteFile(filename, data, 07)
isNil(err, t)

// This gives us a time with the same precision as the time we get from the
// timestamp in the name.
getTime := func() time.Time {
theTime := fakeTime()
if !test.local {
theTime = theTime.UTC()
}
theTime, err := time.Parse(test.timeFormat, theTime.Format(test.timeFormat))
isNil(err, t)
return theTime
}

t1 := getTime()

backup := backupFile(effectiveBackupDir, withLocalTime(test.local), withTimeFormat(test.timeFormat))
err = ioutil.WriteFile(backup, data, 07)
isNil(err, t)

newFakeTime()

t2 := getTime()

backup2 := backupFile(effectiveBackupDir, withLocalTime(test.local), withTimeFormat(test.timeFormat))
err = ioutil.WriteFile(backup2, data, 07)
isNil(err, t)

l := &Logger{Filename: filename, LocalTime: test.local, TimeFormat: test.timeFormat, BackupDir: backupDir}
files, err := l.oldLogFiles()
isNil(err, t)
equals(2, len(files), t)

// should be sorted by newest file first, which would be t2
equals(t2, files[0].timestamp, t)
equals(t1, files[1].timestamp, t)
})
}

func TestTimeFromName(t *testing.T) {
@@ -530,64 +547,87 @@ func TestLocalTime(t *testing.T) {
equals(len(b2), n2, t)

existsWithContent(logFile(dir), b2, t)
existsWithContent(backupFileLocal(dir), b, t)
existsWithContent(backupFile(dir, withLocalTime(true)), b, t)
}

func TestRotate(t *testing.T) {
currentTime = fakeTime
dir := makeTempDir("TestRotate", t)
defer os.RemoveAll(dir)

filename := logFile(dir)

l := &Logger{
Filename: filename,
MaxBackups: 1,
MaxSize: 100, // megabytes
}
defer l.Close()
b := []byte("boo!")
n, err := l.Write(b)
isNil(err, t)
equals(len(b), n, t)

existsWithContent(filename, b, t)
fileCount(dir, 1, t)

newFakeTime()

err = l.Rotate()
isNil(err, t)

// we need to wait a little bit since the files get deleted on a different
// goroutine.
<-time.After(10 * time.Millisecond)

filename2 := backupFile(dir)
existsWithContent(filename2, b, t)
existsWithContent(filename, []byte{}, t)
fileCount(dir, 2, t)
newFakeTime()

err = l.Rotate()
isNil(err, t)

// we need to wait a little bit since the files get deleted on a different
// goroutine.
<-time.After(10 * time.Millisecond)

filename3 := backupFile(dir)
existsWithContent(filename3, []byte{}, t)
existsWithContent(filename, []byte{}, t)
fileCount(dir, 2, t)

b2 := []byte("foooooo!")
n, err = l.Write(b2)
isNil(err, t)
equals(len(b2), n, t)

// this will use the new fake time
existsWithContent(filename, b2, t)
forEachBackupTestSpec(t, func(t *testing.T, test backupTestSpec) {
currentTime = fakeTime
dir := makeTempDir("TestRotate", t)
defer os.RemoveAll(dir)
var backupDir string
effectiveBackupDir := dir
if test.customBackupDir {
// Temp non-existing dir - expected to be created on rotate
backupDir = filepath.Join(makeTempDir("TestOldLogFilesBackup", t), "backups")
defer os.RemoveAll(backupDir)
effectiveBackupDir = backupDir
}

filename := logFile(dir)

l := &Logger{
Filename: filename,
MaxBackups: 1,
MaxSize: 100, // megabytes
BackupDir: backupDir,
TimeFormat: test.timeFormat,
LocalTime: test.local,
}
defer l.Close()
b := []byte("boo!")
n, err := l.Write(b)
isNil(err, t)
equals(len(b), n, t)

existsWithContent(filename, b, t)
fileCount(dir, 1, t)

newFakeTime()

err = l.Rotate()
isNil(err, t)

// we need to wait a little bit since the files get deleted on a different
// goroutine.
<-time.After(10 * time.Millisecond)

filename2 := backupFile(effectiveBackupDir, withLocalTime(test.local), withTimeFormat(test.timeFormat))
existsWithContent(filename2, b, t)
existsWithContent(filename, []byte{}, t)
if test.customBackupDir {
fileCount(dir, 1, t)
fileCount(effectiveBackupDir, 1, t)
} else {
fileCount(dir, 2, t)
}
newFakeTime()

err = l.Rotate()
isNil(err, t)

// we need to wait a little bit since the files get deleted on a different
// goroutine.
<-time.After(10 * time.Millisecond)

filename3 := backupFile(effectiveBackupDir, withLocalTime(test.local), withTimeFormat(test.timeFormat))
existsWithContent(filename3, []byte{}, t)
existsWithContent(filename, []byte{}, t)
if test.customBackupDir {
fileCount(dir, 1, t)
fileCount(effectiveBackupDir, 1, t)
} else {
fileCount(dir, 2, t)
}

b2 := []byte("foooooo!")
n, err = l.Write(b2)
isNil(err, t)
equals(len(b2), n, t)

// this will use the new fake time
existsWithContent(filename, b2, t)
})
}

func TestCompressOnRotate(t *testing.T) {
@@ -696,7 +736,9 @@ func TestJson(t *testing.T) {
"maxage": 10,
"maxbackups": 3,
"localtime": true,
"compress": true
"compress": true,
"timeformat": "1:2.3",
"backupdir": "bar"
}`[1:])

l := Logger{}
@@ -708,6 +750,8 @@ func TestJson(t *testing.T) {
equals(3, l.MaxBackups, t)
equals(true, l.LocalTime, t)
equals(true, l.Compress, t)
equals("1:2.3", l.TimeFormat, t)
equals("bar", l.BackupDir, t)
}

func TestYaml(t *testing.T) {
@@ -717,7 +761,9 @@ maxsize: 5
maxage: 10
maxbackups: 3
localtime: true
compress: true`[1:])
compress: true
timeformat: 1:2.3
backupdir: bar`[1:])

l := Logger{}
err := yaml.Unmarshal(data, &l)
@@ -728,6 +774,8 @@ compress: true`[1:])
equals(3, l.MaxBackups, t)
equals(true, l.LocalTime, t)
equals(true, l.Compress, t)
equals("1:2.3", l.TimeFormat, t)
equals("bar", l.BackupDir, t)
}

func TestToml(t *testing.T) {
@@ -737,7 +785,9 @@ maxsize = 5
maxage = 10
maxbackups = 3
localtime = true
compress = true`[1:]
compress = true
timeformat = "1:2.3"
backupdir = "bar"`[1:]

l := Logger{}
md, err := toml.Decode(data, &l)
@@ -748,14 +798,60 @@ compress = true`[1:]
equals(3, l.MaxBackups, t)
equals(true, l.LocalTime, t)
equals(true, l.Compress, t)
equals("1:2.3", l.TimeFormat, t)
equals("bar", l.BackupDir, t)
equals(0, len(md.Undecoded()), t)
}

func forEachBackupTestSpec(t *testing.T, do func(t *testing.T, test backupTestSpec)) {
for _, test := range backupTestSpecs() {
t.Run(test.name, func(t *testing.T) {
do(t, test)
})
}
}

type backupTestSpec struct {
name string
local bool
timeFormat string
customBackupDir bool
}

func backupTestSpecs() []backupTestSpec {
return []backupTestSpec{
{
name: "Default time format, UTC, default backup dir",
local: false,
timeFormat: DefaultTimeFormat,
customBackupDir: false,
},
{
name: "Default time format, local time, custom backup dir",
local: true,
timeFormat: DefaultTimeFormat,
customBackupDir: true,
},
{
name: "Custom time format, UTC, custom backup dir",
local: false,
timeFormat: "20060102150405000",
customBackupDir: true,
},
{
name: "Default time format, local time, default backup dir",
local: true,
timeFormat: "2006.01.02.15.04.05.000",
customBackupDir: false,
},
}
}

// makeTempDir creates a file with a semi-unique name in the OS temp directory.
// It should be based on the name of the test, to keep parallel tests from
// colliding, and must be cleaned up after the test is finished.
func makeTempDir(name string, t testing.TB) string {
dir := time.Now().Format(name + backupTimeFormat)
dir := time.Now().Format(name + DefaultTimeFormat)
dir = filepath.Join(os.TempDir(), dir)
isNilUp(os.Mkdir(dir, 0700), t, 1)
return dir
@@ -778,18 +874,38 @@ func logFile(dir string) string {
return filepath.Join(dir, "foobar.log")
}

func backupFile(dir string) string {
return filepath.Join(dir, "foobar-"+fakeTime().UTC().Format(backupTimeFormat)+".log")
func backupFile(dir string, opts ...backupFileOpt) string {
options := backupFileOpts{
local: false,
timeFormat: DefaultTimeFormat,
}
for _, opt := range opts {
opt(&options)
}
currTime := fakeTime()
if !options.local {
currTime = currTime.UTC()
}
return filepath.Join(dir, "foobar-"+currTime.Format(options.timeFormat)+".log")
}

type backupFileOpts struct {
local bool
timeFormat string
}

func backupFileLocal(dir string) string {
return filepath.Join(dir, "foobar-"+fakeTime().Format(backupTimeFormat)+".log")
type backupFileOpt func(opts *backupFileOpts)

func withLocalTime(local bool) backupFileOpt {
return func(opts *backupFileOpts) {
opts.local = local
}
}

// logFileLocal returns the log file name in the given directory for the current
// fake time using the local timezone.
func logFileLocal(dir string) string {
return filepath.Join(dir, fakeTime().Format(backupTimeFormat))
func withTimeFormat(format string) backupFileOpt {
return func(opts *backupFileOpts) {
opts.timeFormat = format
}
}

// fileCount checks that the number of files in the directory is exp.

0 comments on commit 8511fb9

Please sign in to comment.