Andreas/wekan
1
0
Fork 0
mirror of https://github.com/wekan/wekan.git synced 2025-12-24 03:10:12 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

Fixed Non-ASCII attachment filename will crash when downloading.

Browse source 
Thanks to xet7 !

Fixes #2759
This commit is contained in:
Lauri Ojansivu 2021-04-29 13:26:49 +03:00
parent 843ff8eaaa
commit c2da477735
277 changed files with 30568 additions and 52 deletions
Download patch file Download diff file Expand all files Collapse all files

5
packages/wekan-cfs-temp-store/.travis.yml Normal file
View file

@ -0,0 +1,5 @@
language: node_js
node_js:
- "0.10"
before_install:
- "curl -L http://git.io/s0Zu-w | /bin/sh"

169
packages/wekan-cfs-temp-store/CHANGELOG.md Normal file
View file

@ -0,0 +1,169 @@
# Changelog
## vCurrent
## [v0.1.2] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.1.2)
#### 17/12/14 by Morten Henriksen
## [v0.1.1] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.1.1)
#### 17/12/14 by Morten Henriksen
- mbr update, remove versions.json
- Bump to version 0.1.1
## [v0.1.0] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.1.0)
#### 17/12/14 by Morten Henriksen
- mbr update versions and fix warnings
- fix 0.9.1 package scope
- don't rely on package names; fix for 0.9.1
- 0.9.1 support
## [v0.0.29] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.29)
#### 28/08/14 by Morten Henriksen
- Meteor Package System Update
## [v0.0.28] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.28)
#### 27/08/14 by Eric Dobbertin
- change package name to lowercase
## [v0.0.27] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.27)
#### 17/06/14 by Eric Dobbertin
- add `FS.TempStore.removeAll` method
## [v0.0.26] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.26)
#### 30/04/14 by Eric Dobbertin
## [v0.0.25] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.25)
#### 30/04/14 by Eric Dobbertin
- use third-party combined-stream node pkg as attempt to resolve pesky streaming issues
## [v0.0.24] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.24)
#### 29/04/14 by Eric Dobbertin
- generate api docs
- fileKey methods now expect an FS.File always, so we give them one
- small FS.File API change
## [v0.0.23] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.23)
#### 12/04/14 by Eric Dobbertin
- test for packages since we're assigning default error functions for stores
## [v0.0.22] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.22)
#### 12/04/14 by Eric Dobbertin
- avoid errors if file already removed from temp store
## [v0.0.21] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.21)
#### 12/04/14 by Eric Dobbertin
## [v0.0.20] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.20)
#### 08/04/14 by Eric Dobbertin
- cleanup stored/uploaded events and further improve chunk tracking
## [v0.0.19] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.19)
#### 08/04/14 by Eric Dobbertin
- use internal tracking collection
- Have TempStore set the size
- Add the SA on stored result
- allow unset chunkSum
## [v0.0.18] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.18)
#### 06/04/14 by Eric Dobbertin
- delete chunkCount and chunkSize props from fileObj after upload is complete
## [v0.0.17] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.17)
#### 06/04/14 by Eric Dobbertin
- We now wait to mount storage until it's needed (first upload begins); this ensures that we are able to accurately check for the cfs-worker package, which loads after this one. It also makes the code a bit cleaner.
## [v0.0.16] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.16)
#### 04/04/14 by Morten Henriksen
- Temporary workaround: We currently we generate a mongoId if gridFS is used for TempStore
- Note: At the moment tempStore will only use gridfs if no filesystem is installed
## [v0.0.15] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.15)
#### 02/04/14 by Morten Henriksen
- Use the stored event and object instead (result object is not used at the moment - but we could store an id at some point)
## [v0.0.14] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.14)
#### 31/03/14 by Eric Dobbertin
- use latest releases
- use latest releases
## [v0.0.13] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.13)
#### 31/03/14 by Morten Henriksen
- Try to use latest when using weak deps
## [v0.0.12] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.12)
#### 30/03/14 by Morten Henriksen
## [v0.0.11] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.11)
#### 30/03/14 by Morten Henriksen
- Set noon callback - we just want the file gone
## [v0.0.10] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.10)
#### 29/03/14 by Morten Henriksen
- add filesystem and gridfs as weak deps
## [v0.0.9] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.9)
#### 29/03/14 by Morten Henriksen
- Add check to see if FS.TempStore.Storage is set
## [v0.0.8] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.8)
#### 29/03/14 by Morten Henriksen
- Converting TempStore to use SA api
## [v0.0.7] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.7)
#### 25/03/14 by Morten Henriksen
- use `new Date`
- Have TempStore emit relevant events
## [v0.0.6] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.6)
#### 23/03/14 by Morten Henriksen
- Rollback to specific git dependency
- use collectionFS travis version force update
## [v0.0.5] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.5)
#### 22/03/14 by Morten Henriksen
- try to fix travis test by using general package references
## [v0.0.4] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.4)
#### 21/03/14 by Morten Henriksen
- fix chunk files not actually being deleted
## [v0.0.3] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.3)
#### 18/03/14 by Morten Henriksen
- * TempStore is now an EventEmitter * progress event * uploaded * (start) should perhaps be created * remove * Added FS.TempStore.listParts - will return lookup object listing the parts already uploaded
- Allow chunk to be undefined an thereby have the createWriteStream follow normal streaming api
- Allow undefined in chunkPath
- added comments
- bug hunting
- Add streaming WIP
- rename temp store collection to 'cfs.tempstore'
- fix ensureForFile
- track tempstore chunks in our own collection rather than in the file object
- change to accept buffer; less converting
- prevent bytesUploaded from getting bigger than size
## [v0.0.2] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.2)
#### 15/02/14 by Morten Henriksen
- fix typo
## [v0.0.1] (https://github.com/zcfs/Meteor-cfs-tempstore/tree/v0.0.1)
#### 13/02/14 by Morten Henriksen
- init commit

20
packages/wekan-cfs-temp-store/LICENSE.md Normal file
View file

@ -0,0 +1,20 @@
The MIT License (MIT)
Copyright (c) 2013 [@raix](https://github.com/raix) and [@aldeed](https://github.com/aldeed), aka Morten N.O. Nørgaard Henriksen, mh@gi-software.com
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

24
packages/wekan-cfs-temp-store/README.md Normal file
View file

@ -0,0 +1,24 @@
wekan-cfs-tempstore
=========================
This is a Meteor package used by
[CollectionFS](https://github.com/zcfs/Meteor-CollectionFS). It provides
an API for quickly storing chunks of file data in temporary files. If also supports deleting those chunks, and combining them into one
binary object and attaching it to an FS.File instance.
You don't need to manually add this package to your app, but you could replace
this package with your own if you want to handle temporary storage in another
way.
> `FS.TempStore` uses the `wekan-cfs-storage-adapter` compatible Storage Adapters, both `FS.Store.FileSystem` and `FS.Store.GridFS` will be defaulted. *for more information read the [internal.api.md](internal.api.md)*
##Documentation
[API Documentation](api.md)
##Contribute
Here's the [complete API documentation](internal.api.md), including private methods.
Update docs, `npm install docmeteor`
```bash
$ docmeteor
```

112
packages/wekan-cfs-temp-store/api.md Normal file
View file

@ -0,0 +1,112 @@
## cfs-tempstore Public API ##
CollectionFS, temporary storage
_API documentation automatically generated by [docmeteor](https://github.com/raix/docmeteor)._
##Temporary Storage
Temporary storage is used for chunked uploads until all chunks are received
and all copies have been made or given up. In some cases, the original file
is stored only in temporary storage (for example, if all copies do some
manipulation in beforeSave). This is why we use the temporary file as the
basis for each saved copy, and then remove it after all copies are saved.
Every chunk is saved as an individual temporary file. This is safer than
attempting to write multiple incoming chunks to different positions in a
single temporary file, which can lead to write conflicts.
Using temp files also allows us to easily resume uploads, even if the server
restarts, and to keep the working memory clear.
The FS.TempStore emits events that others are able to listen to
-
### <a name="FS.TempStore"></a>*fs*.TempStore {object}&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This property __TempStore__ is defined in `FS`*
it's an event emitter*
> ```FS.TempStore = new EventEmitter();``` [tempStore.js:28](tempStore.js#L28)
-
We will not mount a storage adapter until needed. This allows us to check for the
existance of FS.FileWorker, which is loaded after this package because it
depends on this package.
-
XXX: TODO
FS.TempStore.on('stored', function(fileObj, chunkCount, result) {
This should work if we pass on result from the SA on stored event...
fileObj.update({ $set: { chunkSum: 1, chunkCount: chunkCount, size: result.size } });
});
Stream implementation
-
### <a name="FS.TempStore.removeFile"></a>*fsTempstore*.removeFile(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __removeFile__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
This function removes the file from tempstorage - it cares not if file is
already removed or not found, goal is reached anyway.
> ```FS.TempStore.removeFile = function(fileObj) { ...``` [tempStore.js:169](tempStore.js#L169)
-
### <a name="FS.TempStore.createWriteStream"></a>*fsTempstore*.createWriteStream(fileObj, [options])&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __createWriteStream__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
File to store in temporary storage
* __options__ *{[Number ](#Number )|[ String](# String)}* (Optional)
__Returns__ *{Stream}*
Writeable stream
`options` of different types mean differnt things:
`undefined` We store the file in one part
(Normal server-side api usage)*
`Number` the number is the part number total
(multipart uploads will use this api)*
`String` the string is the name of the `store` that wants to store file data
(stores that want to sync their data to the rest of the files stores will use this)*
> Note: fileObj must be mounted on a `FS.Collection`, it makes no sense to store otherwise
> ```FS.TempStore.createWriteStream = function(fileObj, options) { ...``` [tempStore.js:217](tempStore.js#L217)
-
### <a name="FS.TempStore.createReadStream"></a>*fsTempstore*.createReadStream(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __createReadStream__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
The file to read
__Returns__ *{Stream}*
Returns readable stream
> ```FS.TempStore.createReadStream = function(fileObj) { ...``` [tempStore.js:313](tempStore.js#L313)

225
packages/wekan-cfs-temp-store/internal.api.md Normal file
View file

@ -0,0 +1,225 @@
## Public and Private API ##
_API documentation automatically generated by [docmeteor](https://github.com/raix/docmeteor)._
***
__File: ["tempStore.js"](tempStore.js) Where: {server}__
***
##Temporary Storage
Temporary storage is used for chunked uploads until all chunks are received
and all copies have been made or given up. In some cases, the original file
is stored only in temporary storage (for example, if all copies do some
manipulation in beforeSave). This is why we use the temporary file as the
basis for each saved copy, and then remove it after all copies are saved.
Every chunk is saved as an individual temporary file. This is safer than
attempting to write multiple incoming chunks to different positions in a
single temporary file, which can lead to write conflicts.
Using temp files also allows us to easily resume uploads, even if the server
restarts, and to keep the working memory clear.
The FS.TempStore emits events that others are able to listen to
-
### <a name="FS.TempStore"></a>*fs*.TempStore {object}&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This property __TempStore__ is defined in `FS`*
it's an event emitter*
> ```FS.TempStore = new EventEmitter();``` [tempStore.js:28](tempStore.js#L28)
-
### <a name="FS.TempStore.Storage"></a>*fsTempstore*.Storage {StorageAdapter}&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This property is private*
*This property __Storage__ is defined in `FS.TempStore`*
This property is set to either `FS.Store.FileSystem` or `FS.Store.GridFS`
__When and why:__
We normally default to `cfs-filesystem` unless its not installed. *(we default to gridfs if installed)*
But if `cfs-gridfs` and `cfs-worker` is installed we default to `cfs-gridfs`
If `cfs-gridfs` and `cfs-filesystem` is not installed we log a warning.
the user can set `FS.TempStore.Storage` them selfs eg.:
```js
// Its important to set `internal: true` this lets the SA know that we
// are using this internally and it will give us direct SA api
FS.TempStore.Storage = new FS.Store.GridFS('_tempstore', { internal: true });
```
> Note: This is considered as `advanced` use, its not a common pattern.
> ```FS.TempStore.Storage = null;``` [tempStore.js:54](tempStore.js#L54)
-
We will not mount a storage adapter until needed. This allows us to check for the
existance of FS.FileWorker, which is loaded after this package because it
depends on this package.
-
XXX: TODO
FS.TempStore.on('stored', function(fileObj, chunkCount, result) {
This should work if we pass on result from the SA on stored event...
fileObj.update({ $set: { chunkSum: 1, chunkCount: chunkCount, size: result.size } });
});
Stream implementation
-
### <a name="_chunkPath"></a>_chunkPath([n])&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method is private*
__Arguments__
* __n__ *{Number}* (Optional)
Chunk number
__Returns__ *{String}*
Chunk naming convention
> ```_chunkPath = function(n) { ...``` [tempStore.js:104](tempStore.js#L104)
-
### <a name="_fileReference"></a>_fileReference(fileObj, chunk)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method is private*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
* __chunk__ *{Number}*
__Returns__ *{String}*
Generated SA specific fileKey for the chunk
Note: Calling function should call mountStorage() first, and
make sure that fileObj is mounted.
> ```_fileReference = function(fileObj, chunk, existing) { ...``` [tempStore.js:118](tempStore.js#L118)
-
### <a name="FS.TempStore.exists"></a>*fsTempstore*.exists(File)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __exists__ is defined in `FS.TempStore`*
__Arguments__
* __File__ *{[FS.File](#FS.File)}*
object
__Returns__ *{Boolean}*
Is this file, or parts of it, currently stored in the TempStore
> ```FS.TempStore.exists = function(fileObj) { ...``` [tempStore.js:145](tempStore.js#L145)
-
### <a name="FS.TempStore.listParts"></a>*fsTempstore*.listParts(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __listParts__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
__Returns__ *{Object}*
of parts already stored
__TODO__
```
* This is not yet implemented, milestone 1.1.0
```
> ```FS.TempStore.listParts = function(fileObj) { ...``` [tempStore.js:156](tempStore.js#L156)
-
### <a name="FS.TempStore.removeFile"></a>*fsTempstore*.removeFile(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __removeFile__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
This function removes the file from tempstorage - it cares not if file is
already removed or not found, goal is reached anyway.
> ```FS.TempStore.removeFile = function(fileObj) { ...``` [tempStore.js:169](tempStore.js#L169)
-
### <a name="FS.TempStore.createWriteStream"></a>*fsTempstore*.createWriteStream(fileObj, [options])&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __createWriteStream__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
File to store in temporary storage
* __options__ *{[Number ](#Number )|[ String](# String)}* (Optional)
__Returns__ *{Stream}*
Writeable stream
`options` of different types mean differnt things:
`undefined` We store the file in one part
(Normal server-side api usage)*
`Number` the number is the part number total
(multipart uploads will use this api)*
`String` the string is the name of the `store` that wants to store file data
(stores that want to sync their data to the rest of the files stores will use this)*
> Note: fileObj must be mounted on a `FS.Collection`, it makes no sense to store otherwise
> ```FS.TempStore.createWriteStream = function(fileObj, options) { ...``` [tempStore.js:217](tempStore.js#L217)
-
### <a name="FS.TempStore.createReadStream"></a>*fsTempstore*.createReadStream(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
*This method __createReadStream__ is defined in `FS.TempStore`*
__Arguments__
* __fileObj__ *{[FS.File](#FS.File)}*
The file to read
__Returns__ *{Stream}*
Returns readable stream
> ```FS.TempStore.createReadStream = function(fileObj) { ...``` [tempStore.js:313](tempStore.js#L313)

32
packages/wekan-cfs-temp-store/package.js Normal file
View file

@ -0,0 +1,32 @@
Package.describe({
git: 'https://github.com/zcfs/Meteor-cfs-tempstore.git',
name: 'wekan-cfs-tempstore',
version: '0.1.6',
summary: 'CollectionFS, temporary storage'
});
Npm.depends({
'combined-stream': '0.0.4'
});
Package.onUse(function(api) {
api.use(['wekan-cfs-base-package@0.0.30', 'wekan-cfs-file@0.1.16', 'ecmascript@0.1.0']);
api.use('wekan-cfs-filesystem@0.1.2', { weak: true });
api.use('wekan-cfs-gridfs@0.0.30', { weak: true });
api.use('mongo@1.0.0');
api.addFiles([
'tempStore.js'
], 'server');
});
// Package.on_test(function (api) {
// api.use('collectionfs');
// api.use('test-helpers', 'server');
// api.use(['tinytest', 'underscore', 'ejson', 'ordered-dict',
// 'random', 'deps']);
// api.addFiles('tests/server-tests.js', 'server');
// });

395
packages/wekan-cfs-temp-store/tempStore.js Normal file
View file

@ -0,0 +1,395 @@
// ##Temporary Storage
//
// Temporary storage is used for chunked uploads until all chunks are received
// and all copies have been made or given up. In some cases, the original file
// is stored only in temporary storage (for example, if all copies do some
// manipulation in beforeSave). This is why we use the temporary file as the
// basis for each saved copy, and then remove it after all copies are saved.
//
// Every chunk is saved as an individual temporary file. This is safer than
// attempting to write multiple incoming chunks to different positions in a
// single temporary file, which can lead to write conflicts.
//
// Using temp files also allows us to easily resume uploads, even if the server
// restarts, and to keep the working memory clear.
// The FS.TempStore emits events that others are able to listen to
var EventEmitter = Npm.require('events').EventEmitter;
// We have a special stream concating all chunk files into one readable stream
var CombinedStream = Npm.require('combined-stream');
/** @namespace FS.TempStore
* @property FS.TempStore
* @type {object}
* @public
* @summary An event emitter
*/
FS.TempStore = new EventEmitter();
// Create a tracker collection for keeping track of all chunks for any files that are currently in the temp store
var tracker = FS.TempStore.Tracker = new Mongo.Collection('cfs._tempstore.chunks');
/**
* @property FS.TempStore.Storage
* @type {StorageAdapter}
* @namespace FS.TempStore
* @private
* @summary This property is set to either `FS.Store.FileSystem` or `FS.Store.GridFS`
*
* __When and why:__
* We normally default to `cfs-filesystem` unless its not installed. *(we default to gridfs if installed)*
* But if `cfs-gridfs` and `cfs-worker` is installed we default to `cfs-gridfs`
*
* If `cfs-gridfs` and `cfs-filesystem` is not installed we log a warning.
* the user can set `FS.TempStore.Storage` them selfs eg.:
* ```js
* // Its important to set `internal: true` this lets the SA know that we
* // are using this internally and it will give us direct SA api
* FS.TempStore.Storage = new FS.Store.GridFS('_tempstore', { internal: true });
* ```
*
* > Note: This is considered as `advanced` use, its not a common pattern.
*/
FS.TempStore.Storage = null;
// We will not mount a storage adapter until needed. This allows us to check for the
// existance of FS.FileWorker, which is loaded after this package because it
// depends on this package.
function mountStorage() {
if (FS.TempStore.Storage) return;
// XXX: We could replace this test, testing the FS scope for grifFS etc.
// This is on the todo later when we get "stable"
if (Package["wekan-cfs-gridfs"] && (Package["wekan-cfs-worker"] || !Package["wekan-cfs-filesystem"])) {
// If the file worker is installed we would prefer to use the gridfs sa
// for scalability. We also default to gridfs if filesystem is not found
// Use the gridfs
FS.TempStore.Storage = new FS.Store.GridFS('_tempstore', { internal: true });
} else if (Package["wekan-cfs-filesystem"]) {
// use the Filesystem
FS.TempStore.Storage = new FS.Store.FileSystem('_tempstore', { internal: true });
} else {
throw new Error('FS.TempStore.Storage is not set: Install wekan-cfs-filesystem or wekan-cfs-gridfs or set it manually');
}
FS.debug && console.log('TempStore is mounted on', FS.TempStore.Storage.typeName);
}
function mountFile(fileObj, name) {
if (!fileObj.isMounted()) {
throw new Error(name + ' cannot work with unmounted file');
}
}
// We update the fileObj on progress
FS.TempStore.on('progress', function(fileObj, chunkNum, count, total, result) {
FS.debug && console.log('TempStore progress: Received ' + count + ' of ' + total + ' chunks for ' + fileObj.name());
});
// XXX: TODO
// FS.TempStore.on('stored', function(fileObj, chunkCount, result) {
// // This should work if we pass on result from the SA on stored event...
// fileObj.update({ $set: { chunkSum: 1, chunkCount: chunkCount, size: result.size } });
// });
// Stream implementation
/**
* @method _chunkPath
* @private
* @param {Number} [n] Chunk number
* @returns {String} Chunk naming convention
*/
_chunkPath = function(n) {
return (n || 0) + '.chunk';
};
/**
* @method _fileReference
* @param {FS.File} fileObj
* @param {Number} chunk
* @private
* @returns {String} Generated SA specific fileKey for the chunk
*
* Note: Calling function should call mountStorage() first, and
* make sure that fileObj is mounted.
*/
_fileReference = function(fileObj, chunk, existing) {
// Maybe it's a chunk we've already saved
existing = existing || tracker.findOne({fileId: fileObj._id, collectionName: fileObj.collectionName});
// Make a temporary fileObj just for fileKey generation
var tempFileObj = new FS.File({
collectionName: fileObj.collectionName,
_id: fileObj._id,
original: {
name: _chunkPath(chunk)
},
copies: {
_tempstore: {
key: existing && existing.keys[chunk]
}
}
});
// Return a fitting fileKey SA specific
return FS.TempStore.Storage.adapter.fileKey(tempFileObj);
};
/**
* @method FS.TempStore.exists
* @param {FS.File} File object
* @returns {Boolean} Is this file, or parts of it, currently stored in the TempStore
*/
FS.TempStore.exists = function(fileObj) {
var existing = tracker.findOne({fileId: fileObj._id, collectionName: fileObj.collectionName});
return !!existing;
};
/**
* @method FS.TempStore.listParts
* @param {FS.File} fileObj
* @returns {Object} of parts already stored
* @todo This is not yet implemented, milestone 1.1.0
*/
FS.TempStore.listParts = function fsTempStoreListParts(fileObj) {
var self = this;
console.warn('This function is not correctly implemented using SA in TempStore');
//XXX This function might be necessary for resume. Not currently supported.
};
/**
* @method FS.TempStore.removeFile
* @public
* @param {FS.File} fileObj
* This function removes the file from tempstorage - it cares not if file is
* already removed or not found, goal is reached anyway.
*/
FS.TempStore.removeFile = function fsTempStoreRemoveFile(fileObj) {
var self = this;
// Ensure that we have a storage adapter mounted; if not, throw an error.
mountStorage();
// If fileObj is not mounted or can't be, throw an error
mountFile(fileObj, 'FS.TempStore.removeFile');
// Emit event
self.emit('remove', fileObj);
var chunkInfo = tracker.findOne({
fileId: fileObj._id,
collectionName: fileObj.collectionName
});
if (chunkInfo) {
// Unlink each file
FS.Utility.each(chunkInfo.keys || {}, function (key, chunk) {
var fileKey = _fileReference(fileObj, chunk, chunkInfo);
FS.TempStore.Storage.adapter.remove(fileKey, FS.Utility.noop);
});
// Remove fileObj from tracker collection, too
tracker.remove({_id: chunkInfo._id});
}
};
/**
* @method FS.TempStore.removeAll
* @public
* @summary This function removes all files from tempstorage - it cares not if file is
* already removed or not found, goal is reached anyway.
*/
FS.TempStore.removeAll = function fsTempStoreRemoveAll() {
var self = this;
// Ensure that we have a storage adapter mounted; if not, throw an error.
mountStorage();
tracker.find().forEach(function (chunkInfo) {
// Unlink each file
FS.Utility.each(chunkInfo.keys || {}, function (key, chunk) {
var fileKey = _fileReference({_id: chunkInfo.fileId, collectionName: chunkInfo.collectionName}, chunk, chunkInfo);
FS.TempStore.Storage.adapter.remove(fileKey, FS.Utility.noop);
});
// Remove from tracker collection, too
tracker.remove({_id: chunkInfo._id});
});
};
/**
* @method FS.TempStore.createWriteStream
* @public
* @param {FS.File} fileObj File to store in temporary storage
* @param {Number | String} [options]
* @returns {Stream} Writeable stream
*
* `options` of different types mean differnt things:
* * `undefined` We store the file in one part
* *(Normal server-side api usage)*
* * `Number` the number is the part number total
* *(multipart uploads will use this api)*
* * `String` the string is the name of the `store` that wants to store file data
* *(stores that want to sync their data to the rest of the files stores will use this)*
*
* > Note: fileObj must be mounted on a `FS.Collection`, it makes no sense to store otherwise
*/
FS.TempStore.createWriteStream = function(fileObj, options) {
var self = this;
// Ensure that we have a storage adapter mounted; if not, throw an error.
mountStorage();
// If fileObj is not mounted or can't be, throw an error
mountFile(fileObj, 'FS.TempStore.createWriteStream');
// Cache the selector for use multiple times below
var selector = {fileId: fileObj._id, collectionName: fileObj.collectionName};
// TODO, should pass in chunkSum so we don't need to use FS.File for it
var chunkSum = fileObj.chunkSum || 1;
// Add fileObj to tracker collection
tracker.upsert(selector, {$setOnInsert: {keys: {}}});
// Determine how we're using the writeStream
var isOnePart = false, isMultiPart = false, isStoreSync = false, chunkNum = 0;
if (options === +options) {
isMultiPart = true;
chunkNum = options;
} else if (options === ''+options) {
isStoreSync = true;
} else {
isOnePart = true;
}
// XXX: it should be possible for a store to sync by storing data into the
// tempstore - this could be done nicely by setting the store name as string
// in the chunk variable?
// This store name could be passed on the the fileworker via the uploaded
// event
// So the uploaded event can return:
// undefined - if data is stored into and should sync out to all storage adapters
// number - if a chunk has been uploaded
// string - if a storage adapter wants to sync its data to the other SA's
// Find a nice location for the chunk data
var fileKey = _fileReference(fileObj, chunkNum);
// Create the stream as Meteor safe stream
var writeStream = FS.TempStore.Storage.adapter.createWriteStream(fileKey);
// When the stream closes we update the chunkCount
writeStream.safeOn('stored', function(result) {
// Save key in tracker document
var setObj = {};
setObj['keys.' + chunkNum] = result.fileKey;
tracker.update(selector, {$set: setObj});
var temp = tracker.findOne(selector);
if (!temp) {
FS.debug && console.log('NOT FOUND FROM TEMPSTORE => EXIT (REMOVED)');
return;
}
// Get updated chunkCount
var chunkCount = FS.Utility.size(temp.keys);
// Progress
self.emit('progress', fileObj, chunkNum, chunkCount, chunkSum, result);
var modifier = { $set: {} };
if (!fileObj.instance_id) {
modifier.$set.instance_id = process.env.COLLECTIONFS_ENV_NAME_UNIQUE_ID ? process.env[process.env.COLLECTIONFS_ENV_NAME_UNIQUE_ID] : process.env.METEOR_PARENT_PID;
}
// If upload is completed
if (chunkCount === chunkSum) {
// We no longer need the chunk info
modifier.$unset = {chunkCount: 1, chunkSum: 1, chunkSize: 1};
// Check if the file has been uploaded before
if (typeof fileObj.uploadedAt === 'undefined') {
// We set the uploadedAt date
modifier.$set.uploadedAt = new Date();
} else {
// We have been uploaded so an event were file data is updated is
// called synchronizing - so this must be a synchronizedAt?
modifier.$set.synchronizedAt = new Date();
}
// Update the fileObject
fileObj.update(modifier);
// Fire ending events
var eventName = isStoreSync ? 'synchronized' : 'stored';
self.emit(eventName, fileObj, result);
// XXX is emitting "ready" necessary?
self.emit('ready', fileObj, chunkCount, result);
} else {
// Update the chunkCount on the fileObject
modifier.$set.chunkCount = chunkCount;
fileObj.update(modifier);
}
});
// Emit errors
writeStream.on('error', function (error) {
FS.debug && console.log('TempStore writeStream error:', error);
self.emit('error', error, fileObj);
});
return writeStream;
};
/**
* @method FS.TempStore.createReadStream
* @public
* @param {FS.File} fileObj The file to read
* @return {Stream} Returns readable stream
*
*/
FS.TempStore.createReadStream = function(fileObj) {
// Ensure that we have a storage adapter mounted; if not, throw an error.
mountStorage();
// If fileObj is not mounted or can't be, throw an error
mountFile(fileObj, 'FS.TempStore.createReadStream');
FS.debug && console.log('FS.TempStore creating read stream for ' + fileObj._id);
// Determine how many total chunks there are from the tracker collection
var chunkInfo = tracker.findOne({fileId: fileObj._id, collectionName: fileObj.collectionName}) || {};
var totalChunks = FS.Utility.size(chunkInfo.keys);
function getNextStreamFunc(chunk) {
return Meteor.bindEnvironment(function(next) {
var fileKey = _fileReference(fileObj, chunk);
var chunkReadStream = FS.TempStore.Storage.adapter.createReadStream(fileKey);
next(chunkReadStream);
}, function (error) {
throw error;
});
}
// Make a combined stream
var combinedStream = CombinedStream.create();
// Add each chunk stream to the combined stream when the previous chunk stream ends
var currentChunk = 0;
for (var chunk = 0; chunk < totalChunks; chunk++) {
combinedStream.append(getNextStreamFunc(chunk));
}
// Return the combined stream
return combinedStream;
};

39
packages/wekan-cfs-temp-store/tests/server-tests.js Normal file
View file

@ -0,0 +1,39 @@
function equals(a, b) {
return !!(EJSON.stringify(a) === EJSON.stringify(b));
}
Tinytest.add('cfs-tempstore - server - test environment', function(test) {
test.isTrue(typeof FS.Collection !== 'undefined', 'test environment not initialized FS.Collection');
});
/*
* This is a server-only package so only server tests are needed.
* Need to test each API method:
* FS.TempStore.saveChunk
* FS.TempStore.getDataForFile
* FS.TempStore.getDataForFileSync
* FS.TempStore.deleteChunks
* FS.TempStore.ensureForFile
*
*/
//Test API:
//test.isFalse(v, msg)
//test.isTrue(v, msg)
//test.equalactual, expected, message, not
//test.length(obj, len)
//test.include(s, v)
//test.isNaN(v, msg)
//test.isUndefined(v, msg)
//test.isNotNull
//test.isNull
//test.throws(func)
//test.instanceOf(obj, klass)
//test.notEqual(actual, expected, message)
//test.runId()
//test.exception(exception)
//test.expect_fail()
//test.ok(doc)
//test.fail(doc)
//test.equal(a, b, msg)