Skip to content

Commit

Permalink
Merge pull request #238 from nono/ignore
Browse files Browse the repository at this point in the history
Allow users to ignore somes files listed in .cozyignore
  • Loading branch information
kosssi committed Mar 18, 2016
2 parents fd4ae6c + 52811e3 commit a445986
Show file tree
Hide file tree
Showing 12 changed files with 637 additions and 51 deletions.
30 changes: 21 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[Cozy][0] Desktop <sup>(alpha)</sup>
=================
====================================

[![Build Status][1]][2]

Expand Down Expand Up @@ -35,12 +35,14 @@ sudo npm install cozy-desktop -g
CLI Running
-----------

Configure it with your remote Cozy
Configure it with your remote Cozy and your local directory:

```bash
cozy-desktop add-remote-cozy https://url.of.my.cozy/ /sync/directory
cozy-desktop add-remote-cozy https://url.of.my.cozy/ ~/cozy
```

It will synchronize your local directory `~/cozy` with your remote cozy.

Then start synchronization daemon:

```bash
Expand All @@ -57,6 +59,19 @@ cozy-desktop -h
Advanced use cases
------------------

It's possible to make cozy-desktop ignore some files and folders by using a
`.cozyignore` file. It works pretty much like a `.gitignore`, ie you put
patterns in this file to ignore. The rules for patterns are the same, so you
can look at
[git documentation](https://git-scm.com/docs/gitignore/#_pattern_format) to
see for their format. For example:

```bash
*.mp4
heavy-*
/tmp
```

Cozy-desktop keeps the metadata in a pouchdb database. If you want to use
several synchronized directories, you'll have to tell cozy-desktop to keeps
its metadata database somewhere else. The `COZY_DESKTOP_DIR` env variable has
Expand All @@ -78,18 +93,15 @@ Cozy-desktop is designed to synchronize files and folders between a remote
cozy instance and a local hard drive, for a personal usage. We tried to make
it simple and easy. So, it has some limitations:

- It's only a command-line interface and it works only on Linux for the moment.
We are working to improve this in the next weeks.

- It's all or nothing for files and folders to synchronize, but we have on our
roadmap to add a mean to select which files and folders to synchronize.
- It's only a command-line interface and it is tested only on Linux for the
moment. We are working to improve this in the next weeks.

- Files and folders named like this are ignored:
- `.cozy-desktop` (they are used to keep internal state)
- `_design` (special meaning for pouchdb/couchdb)

- It's not a particularly good idea to share code with cozy-desktop:
- `node_modules` can't be ignored for the moment and have tons of small files
- `node_modules` have tons of small files
- compiled code often has to be recompile to works on another environment
- git and other VCS are not meant to be share like this. You may lose your
work if you make changes on two laptops synchronized by cozy-desktop (it's
Expand Down
16 changes: 16 additions & 0 deletions doc/debug.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,19 @@ You can launch cozy-desktop directly in coffee:
```bash
DEBUG=true node_modules/.bin/coffee src/bin/cli.coffee sync
```


Debug ignored files
-------------------

You can list the files and folder that are synchronized with:

```bash
cozy-desktop ls
```

And those which are ignored:

```bash
cozy-desktop ls --ignored
```
46 changes: 46 additions & 0 deletions doc/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,52 @@ secure. And bugs in this part mean losing data, which is very bad. So, we
don't try to be smart and prefer a robust solution.


Ignores
-------

Cozy-desktop can ignore some files and folders with a `.cozyignore` file. This
file is read only at the startup of Cozy-desktop. So, if this file is
modified, cozy-desktop has to be relaunched for the changes to be effective.

There 4 places where ignoring files and folders can have a meaning:

- when a change is detected on the local file system and cozy-desktop is going
to save it in its internal pouchdb
- when a change is detected on the remote cozy and cozy-desktop is going to
save it in its internal pouchdb
- when a change is taken from the pouchdb and cozy-desktop is going to apply
on the local file system
- when a change is taken from the pouchdb and cozy-desktop is going to apply
on the remote cozy.

Even with the first two checks, pouchdb can have a change for an ignored file
from a previous run of cozy-desktop where the file was not yet ignored. So, we
have to implement the last two checks. It is enough for a file created on one
side (local or remote) won't be replicated on the other side if it is ignored.

But, there is a special case: conflicts are treated ahead of pouchdb. So, if a
file is created in both the local file system and the remote cozy (with
different contents) is ignored, the conflict will still be resolved by
renaming if we implement only the last two checks. We have to avoid that by
also implementing at least one of the first two checks.

In practice, it's really convenient to let the changes from the remote couchdb
flows to pouchdb, even for ignored files, as it is very costly to find them
later if `.cozyignore` has changed. And it's a lot easier to detect local
files that were ignored but are no longer at the startup, as cozy-desktop
already does a full scan of the local file system at that moment.

Thus, cozy-desktop has a check for ignored files and folder in three of the
four relevant places:

- when a change is detected on the local file system and cozy-desktop is going
to save it in its internal pouchdb
- when a change is taken from the pouchdb and cozy-desktop is going to apply
on the local file system
- when a change is taken from the pouchdb and cozy-desktop is going to apply
on the remote cozy.


Documents schema
----------------

Expand Down
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@
"fs-extra": "0.26.7",
"lodash.isequal": "4.1.1",
"lodash.pick": "4.1.0",
"micromatch": "2.3.7",
"mime": "1.3.4",
"node-uuid": "1.4.7",
"path-extra": "3.0.0",
"pouchdb": "5.3.1",
"printit": "0.1.18",
"progress": "1.1.8",
"read": "1.0.7",
"readdirp": "2.0.0",
"request-json-light": "0.5.22"
},
"devDependencies": {
Expand Down
46 changes: 39 additions & 7 deletions src/app.coffee
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
async = require 'async'
path = require 'path-extra'
os = require 'os'
url = require 'url'
log = require('printit')
async = require 'async'
fs = require 'fs'
os = require 'os'
path = require 'path-extra'
readdirp = require 'readdirp'
url = require 'url'
log = require('printit')
prefix: 'Cozy Desktop '
date: true

Config = require './config'
Devices = require './devices'
Pouch = require './pouch'
Ignore = require './ignore'
Merge = require './merge'
Prep = require './prep'
Local = require './local'
Expand Down Expand Up @@ -107,13 +110,24 @@ class App
callback? err


# Load ignore rules
loadIgnore: ->
try
ignored = fs.readFileSync(path.join @basePath, '.cozyignore')
ignored = ignored.toString().split('\n')
catch error
ignored = []
@ignore = new Ignore(ignored).addDefaultRules()


# Instanciate some objects before sync
instanciate: ->
@loadIgnore()
@merge = new Merge @pouch
@prep = new Prep @merge
@prep = new Prep @merge, @ignore
@local = @merge.local = new Local @config, @prep, @pouch
@remote = @merge.remote = new Remote @config, @prep, @pouch
@sync = new Sync @pouch, @local, @remote
@sync = new Sync @pouch, @local, @remote, @ignore


# Start the synchronization
Expand Down Expand Up @@ -149,6 +163,24 @@ class App
@local?.watcher.debug()


# Call the callback for each file
walkFiles: (args, callback) ->
@loadIgnore()
options =
root: @basePath
directoryFilter: '!.cozy-desktop'
entryType: 'both'
readdirp options
.on 'warn', (err) -> log.warn err
.on 'error', (err) -> log.error err
.on 'data', (data) =>
doc =
_id: data.path
docType: if data.stat.isFile() then 'file' else 'folder'
if @ignore.isIgnored(doc) is args.ignored?
callback data.path


# Recreate the local pouch database
resetDatabase: (callback) =>
@askConfirmation (err, ok) =>
Expand Down
10 changes: 9 additions & 1 deletion src/bin/cli.coffee
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ sync = (mode, args) ->


program
.command 'add-remote-cozy <url> <syncPath>'
.command 'add-remote-cozy <url> <localSyncPath>'
.description 'Configure current device to sync with given cozy'
.option '-d, --deviceName [deviceName]', 'device name to deal with'
.action (url, syncPath, args) ->
Expand Down Expand Up @@ -106,6 +106,14 @@ program
The README has more instructions about that.
"""

program
.command 'ls'
.description 'List local files that are synchronized with the remote cozy'
.option('-i, --ignored', 'List ignored files')
.action (args) ->
app.walkFiles args, (file) ->
console.log file

program
.command 'reset-database'
.description 'Recreates the local database'
Expand Down
117 changes: 117 additions & 0 deletions src/ignore.coffee
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
{basename,dirname} = require('path')
matcher = require('micromatch').matcher

# Cozy-desktop can ignore some files and folders from a list of patterns in the
# cozyignore file. This class can be used to know if a file/folder is ignored.
#
# See https://git-scm.com/docs/gitignore/#_pattern_format
class Ignore

# See https://github.com/github/gitignore/tree/master/Global
DefaultRules = [
# Dropbox
'.dropbox'
'.dropbox.attr'
'.dropbox.cache'

# Eclipse, SublimeText and many others
'*.tmp'
'*.bak'

# Emacs
'*~'
'\\#*\\#'

# LibreOffice
'.~lock.*#'

# Linux
'.fuse_hidden*'
'.Trash-*'

# Microsoft Office
'~$*.{doc,xls,ppt}*'

# OSX
'.DS_Store'
'.DocumentRevisions-V100'
'.fseventsd'
'.Spotlight-V100'
'.TemporaryItems'
'.Trashes'
'.VolumeIcon.icns'

# Vim
'*.sw[px]'

# Windows
'Thumbs.db'
'ehthumbs.db'
]

# See https://github.com/jonschlinkert/micromatch#options
MicromatchOptions =
noextglob: true

# Load patterns for detecting ignored files and folders
constructor: (lines) ->
@patterns = []
for line in lines
continue if line is '' # Blank line
continue if line[0] is '#' # Comments
@patterns.push @buildPattern line

# Parse a line and build the corresponding pattern
buildPattern: (line) ->
folder = false
negate = false
noslash = line.indexOf('/') is -1
if line.indexOf('**') isnt -1 # Detect two asterisks
fullpath = true
noslash = false
if line[0] is '!' # Detect bang prefix
line = line.slice 1
negate = true
if line[0] is '/' # Detect leading slash
line = line.slice 1
if line[line.length-1] is '/' # Detect trailing slash
line = line.slice 0, line.length-1
folder = true
line = line.replace /^\\/, '' # Remove leading escaping char
line = line.replace /\s*$/, '' # Remove trailing spaces
pattern =
match: matcher line, MicromatchOptions
basename: noslash # The pattern can match only the basename
folder: folder # The pattern will only match a folder
negate: negate # The pattern is negated
return pattern

# Add some rules for things that should be always ignored (temporary
# files, thumbnails db, trash, etc.)
addDefaultRules: ->
morePatterns = (@buildPattern rule for rule in DefaultRules)
@patterns = morePatterns.concat @patterns
return @

# Return true if the doc matches the pattern
match: (path, isFolder, pattern) ->
if pattern.basename
return true if pattern.match basename path
if isFolder or not pattern.folder
return true if pattern.match path
parent = dirname path
return false if parent is '.'
return @match parent, true, pattern

# Return true if the given file/folder path should be ignored
isIgnored: (doc) ->
result = false
for pattern in @patterns
if pattern.negate
result and= not @match doc._id, doc.docType is 'folder', pattern
else
result or= @match doc._id, doc.docType is 'folder', pattern
return result


module.exports = Ignore
Loading

0 comments on commit a445986

Please sign in to comment.