Centralized File System Overview

In Community Server 2007 and earlier versions, the many file-related features such as post attachments, site files, blog files, avatars, and role icons implemented their own file management logic and supported the following:

• Storing files in the database;
• Storing files in the file system; or,
• Configurable options allowing file storage in the database and/or file system. 

Adding new file-based features in the core application or in add-ons required implementing custom file management logic. Database-based storage was the default for post attachments. Write permissions needed to be granted in many locations within the CS application. 

Storing files in external systems such as Amazon S3 required modifications to the custom file management logic used for each file-based feature. 

In Community Server 2008, Telligent has implemented the Centralized File System (or Centralized File Storage or CFS) to centralize file management, which has provided several advantages:

• One instance of file management logic used for all file-based features.
• Supporting file storage providers so that new file storage services, such as Amazon S3, can be used to store files.
• It is easier to add file-based features to Community Server both in the core product as well as in add-ons.

The CFS sits deep in the Community Server API, between the API for file-based features and the configured storage providers:

Interacting with individual file-based features has changed very little with Community Server 2008. The individual file-based features, however, use the CFS instead of their own custom file management logic.

Users and developers do not directly interact with the CFS except when they need to implement new file-based functionality for direct access to the CFS. For example, to add/update/delete post attachments and their content, code should not access the CFS; instead, it should access the add/update/delete methods of the PostAttachments class, which themselves interact with the CFS file store for post attachments. 

When implementing a new feature, such as video transcoding, you may want to have a new file store and interact with the CFS directly as a new file-based feature.

CFS Terminology

The CFS uses file store keys, paths, and filenames to store files:

File Store Key

The file store key is the name identified on the <fileStore /> node representing the file store in the communityserver.config file. It represents a unique set of files that are managed independently of other file stores.

File store keys must be between 3 and 255 characters and cannot contain any characters that are not valid in windows file system paths.

Path

Paths are similar to directories, but are technically used more like keys. Paths can represent multiple values separated by periods, which are used for defining sub-paths and are accessible via the CentralizedFileSystemProvider. DirectorySeparator property. For example, if a common file system path is my\files\are\here, the similar CFS path would be my.files.are.here.

There is a technical difference between directories and CFS paths, however. When querying for files in a directory, files are generally accessed within a single folder, as with my\files\are. In the CFS, because paths are treated as keys, files can also be accessed using partial paths, such as my.files.a, which would match files in the my.files.are path, as well as the my.files.and and my.files.any paths.

Paths must be between 0 and 769 characters and cannot contain any characters that are not valid in windows file system paths.

Filename

Filenames are the names of the files stored in the CFS. They must be between 1 and 255 characters in length and must not contain any characters that are invalid in window file system file names.

File Stores in the CFS

Each file-based feature can use one or more file stores in the CFS. Each file store is defined within the communityserver.config file by a <fileStore /> node within the <CentralizedFileStorage /> node. 

Each file store must define two attributes: name and type.  Additionally, you can optionally set the downloadValidatorType attribute.

name

The name attribute defines the name of the file store (also referred to as the file store key). This is how the code used to implement file-based features identifies the file store when it interacts with the CFS. 

type

The type attribute identifies the CFS file storage provider to use for storing files for the file store. 

CFS file storage providers are .net classes that inherit and implement the CommunityServer.Components.CentralizedFileStorageProvider base class (in the CommunityServer.Components.dll assembly). Community Server 2008 ships with two CFS file storage providers: AmazonS3FileStorageProvider and FileSystemFileStorageProvider.

Each file store can define its own file storage provider with its own configuration. For example, you can store post attachments in Amazon S3 but you can store user avatars in the local file system. By default, Community Server 2008 uses the file system file storage provider for all file stores and stores files in the filestorage/ folder.

downloadValidatorType

Each file store can optionally identify a download validator, which is a .net class that implements the CommunityServer.Components.IcentralizedFileAccess Validator interface in the CommunityServer.Components.dll assembly. 

A download validator implements logic that identifies, for the file store, whether a file should be accessible. In Community Server 2008, only post attachments identify a download validator to ensure that users have view and/or read permission to the attachment's associated post before the attachment can be downloaded.

Additional Attributes

Each provider requires that additional attributes be defined on the <fileStore /> node. These additional attributes are documented in the comments just above the <CentralizedFileStorage /> node in the communityserver.config file included with Community Server 2008.

Creating New CFS File Stores

Creating a new CFS file store is as simple as adding a new <fileStore /> node to the <CentralizedFileStorage /> node of the communityserver.config file with a new name. When the <fileStore /> node exists, you can use that file store to add/delete/query files.

The existing file stores are named according to the full name of the class in the Community Server API that interacts with the file store. For example, you create/update/delete post attachments via the CommunityServer.Components.PostAttachments class in the Community Server API so the CFS file store used to store post attachment files is named CommunityServer.Components.PostAttachments. Telligent recommends that you continue this naming convention when adding additional file stores to the CFS.

Coding Against a CFS File Store

Telligent recommends using the CFS for file storage when you implement new file-based features for Community Server. The CFS provides a simple API for adding, deleting and querying files and paths.

CFS exposes the following methods, all of which must be supported by all CFS providers:  GetFile, GetFiles, GetPaths, AddPath, AddUpdateFile, Delete. 

To interact with these methods, the code must first retrieve the file storage provider for a specific file store. This is done using the file store key, as in:

CentralizedFileSystemProvider myFileStore = CentralizedFileSystemProvider.Instance("FILE_STORE_KEY");

This will load the file storage provider for the “FILE_STORE_KEY” file store. There must be a corresponding <fileStore /> node in the communityserver.config file named "FILE_STORE_KEY; otherwise, the Instance method will return null. The file management methods can then be called on the myFileStore object. Any operations performed using the myFileStore object are performed within the context of the file store named "FILE_STORE_KEY".

Get Files

ICentralizedFile GetFile(string path, string fileName)

ICentralizedFile file is an abstract representation of a file and supports the following:

• Retrieving the ContentLength, FileName, Path, and FileStoreKey of the file
• Retrieving a stream of the content of the file via the OpenReadStream method
• Getting a URL to download a file via the GetDownloadUrl method

This gets the file store's provider's implementation of the ICentralizedFile interface representing the file with the given path and file name. If the file is not found, the GetFile method returns null.

List Files

List<ICentralizedFile> GetFiles(PathSearchOption searchOption)

List<ICentralizedFile> GetFiles(string path, PathSearchOption searchOption)

These get a list of files for the given path matching the defined search option (TopLevelPathOnly, AllPaths). When using the TopLevelPathOnlysearch option, the file store works like a file system. When using the AllPathssearch option, the file store works more like a key-based system. 

Note that paths are technically keys, not directories. If you have a file in the path my.files.are.here, the file can be retrieved from the path my.files or my.fil when using the AllPaths search option.

List Strings

List<string> GetPaths()

List<string> GetPaths(string path)

These get a list of paths within the defined path. When you omit the path, CFS retrieves all top-level paths. Paths are separated using periods (identified by the CentralizedFileSystemProvider.DirectorySeparator property).

AddPath

void AddPath(string path)

This adds a path. In CFS, there is no concept of parent folder and child folder; instead, paths are treated as keys. So you can add a path named my.files.are.here just as easily as incrementally adding my, my.files, my.files.are, and finally my.files.are.here. The CFS will handle either case. 

AddUpdateFile

ICentralizedFile AddUpdateFile(string path, string fileName, Stream contentStream)

This adds or updates a file to the file store. If a file with the same path and fileName already exists, it is overwritten. The path does not need to be manually created.

voidDelete

void Delete()

void Delete(string path)

void Delete(string path, string fileName)

These delete a file or path. If the path is omitted, CFS deletes all paths and files from the file store.

Implementing New CFS File Storage Providers

You can implement new CFS file storage providers by inheriting from CommunityServer.Components.CentralizedFileStorageProvider. Here are a few notes regarding implementing custom CFS storage providers:

• Are You Sure?

You do not need to implement new CFS file storage providers to store files in the CFS. Only when you want to store files outside of the file system or Amazon S3 is a new file storage provider required.

• Resources

Because creating new CFS file storage providers should be rare, Telligent cannot list all of the considerations here. Instead, we suggest reviewing the notes and comments in the CentralizedFileSystemProvider class in the Community Server 2008 SDK and the two implementations included in it for specific information and implementation guidelines for each method.

• IcentralizedFile

Each CentralizedFileSystemProvider must also do the following:

• Implement the ICentralizedFile interface
• Support reading files as a stream (via the ICentralizedFile.OpenReadStream method)
• Download files via a URL (via the ICentralizedFile.GetDownloadUrl method).

All CFS functionality should be fully implemented to support all of the ways a file store can be used.

General CFS Notes

Here are a few general notes about using the CFS in Community Server 2008:

• Avoid publishing the ICentralizedFile.GetDownloadUrl()

Generally speaking, you should use the CentralizedFileStorageProvider.GetGenericDownloadUrl(ICentralizedFile)method when rendering download links to a file. The URL generated by this method is generic and, when accessed, will determine the current file storage provider for the file and redirect to it. This allows you to change the storage provider without having to update all existing links to files (since ICentralizedFile.GetDownloadUrl()would otherwise return a storage-provider-specific URL that would not be valid when the storage provider is changed).

• Avoid calling ICentralizedFile.OpenReadStream()

Whenever possible, render a URL and do not read the contents of the file. In most cases, a URL can be rendered to a user instead of reading a file and writing it manually. In some cases, however, this is not possible, such as when resizing an image.

• When a URL is really a CFS-stored File

When processing URLs, the CentralizedFileStorageProvider class exposes a few static methods that can help to translate a URL into a CFS-stored ICentralizedFile: bool IsCentralizedFile(string url) and ICentralizedFile GetCentralizedFileByUrl(string url). These methods can be used to detect and retrieve CFS files by their URL. Note that these methods only work with URLs generated by the CentralizedFileStorageProvider.GetGenericDownloadUrl method.