Microsoft File Connector

Installation and Configuration

Video Tutorial „Set up a basic Microsoft File Connector”

This Video describes how to configure the Microsoft File Connector. See what preconditions are necessary and how to configure the index. Also have a look at the Active Directory Based Authentication, as well as LDAP and how to analyze crawled documents and crawl runs in app.telemetry.

https://www.youtube.com/watch?v=S2JCrM98W30

Configuration of Mindbreeze

Click on the “Indices” tab and then on the “Add new index” symbol to create a new index.

Enter the index path, e.g. “/data/indices/filesystem”. Change the Display Name of the Index Service and the related Filter Service if necessary.

Add a new data source with the symbol “Add new custom source” at the bottom right.

• „Ignore Category Instance”: If there are multiple File Crawlers configured on an Index, the search will noch be restricted to the concrete Category Instance of this Crawler.
• „Authorization Service“: Currently we provide no Authorization Serivce for Microsoft File.

Configuration of Data Source

Caching Principal Resoution Service

To use the Caching Principal Resolution Service you have to select CachingLdapPrincipalResoution. Then it is used to resolve the AD group membership of a user in the search.

For more details click here Caching Principal Resolution Service.

Sources

• “Root Directories (UNC Path)”: Directories should be separated by new line (maximum 24 directories). For crawling using “Content Location Optimization” option in Linux make sure that root paths are mounted.
• “Supports SMB2”: The crawler uses SMB2 protocol by selecting this option.
• “Disable SMB Packet Signing”: SMB packets will be sent without signature and the signature of receiving SMB packets will not be verified.
• “Disable SMB2 Multi-Protocol Negotiate”: On servers that support only SMB1 this option allows for better error messages.
• “SMB Client Transaction Timeout”: Thread timeout in seconds for SMB Connections.
• “SMB Client Socket Timeout”: Socket timeout for all SMB connections made.
• “Crawl Last Modified Directory Files First”: Files and sub directories of a directory will be sorted based on their modification date during traversal of a directory.
• “Root Traversal Threads Count”: The maximal number of threads that traverse root directories configured in “Root Directories” field.
• “Documents Dispatcher Threads Count”: The maximal number of threads that send documents from “Documents Dispatcher Queue” to the index.
• “Documents Dispatcher Queue Size”: The maximal length of the queue, where the directories together and files are queued for “Document Dispatcher Threads” to be dispatched to index.
• “Directory Files Lister Threads Count”: The maximal number of threads that takes directories from “Directory Files Lister Queue” and fetched their subdirectories, documents and ACLs. Subdirectories are enqueued into the “Directory Files Lister Queue”.
• “Directory Files Lister Queue Size”: The maximal length of the queue, where directories are queued for fetching their subdirectories, files and ACLs.
• “Includes”: Only those files and directories are crawled whose path match this pattern (case sensitive).
• “Document Size Limit (MB)”: The Crawler will ignore documents larger than this Size. If this limit is changed, the limit and the rpc-timeout of the filter service should be adapted as well.
• “Maximum Crawled Document Size in MB.”: If set with a positive value, the crawler sends documents with larger content in MB as the configured value with empty content to the index.
• “Include Patterns”: With “regexpIgnoreCase:” allows to use case insensitive regular expressions.
• “Excludes”: Those files and directories whose path match this pattern are not crawled (case sensitive).
• “Exclude Patterns”: With “regexpIgnoreCase:” allows to use case insensitive regular expressions.
• “Exclude Directories”: If selected directories are not crawled.
• “Always Use Directory Rights”: If selected direct security permissions on files itself are ignored.
• “Full Traversal Interval (Hours)”: Interval between two full traversals of all documents in file share. Modified documents are crawled during incremental traversal after “Crawler Interval”.
  The default setting (-1) is fine for most installations, which is a full traversal after “Crawler Interval”. However for large file shares incremental traversal can improve performance of crawler but deleted documents (deleted from file share) remain in index until next full traversal.
• “Remove Deleted Documents from Index”: By selecting this option documents which are deleted from file share are deleted from index at the end of a full traversal.

Security Rights Settings

• “ACL Security Level”: Drei Ebenen können ausgewählt werden: Share, Directory und File.

• “LSA/SAM Desired Access”: Sometimes there are local groups in ACLs of the documents. In order to resolve the domain users or domain groups inside these local groups there is an access to LSA (Local Security Authority) and SAM (Service Account Manager) using the RPC-SMB protocol needed. It is therefore necessary that the crawler service user has one of the following access permissions to LSA and SAM: Maximum allowed, Generic all, Generic execute, Generic Read or Read Control. For crawling NetApp share Read Control may be needed as Desired LSA/SAM Access to be selected.
• “Resolve All Domains”: To correctly assign the file permissions (ACLs) of different domains, select the Resolve All Domains option. For this it is necessary that either the LDAP servers of these domains are configured directly under "LDAP Server" or can be resolved via DNS SRV Records from AD using LDAP. Therefore, the domains should be configured in the Network Tab under LDAP Setting. If "Resolve All Domains" is not selected, only the ACLs from the File Share Server domain will be resolved correctly.

Trustee Information Settings

• “Trustee Information File Path”: Path to the Trustee information file which can reside in a local folder or an UNC-Path.
• “Trustee Volume Path”: The Volume-Path of the Root folder in the Trustee information file. Can be omitted if the Root folder is equal to the Volume-Path.

Extensions (Index File Lister)

These are plugins that can be provided by Mindbreeze to cover special use cases. The files are not indexed by classical "browsing" through the file trees, but a file or a database or something similar is bound, which contains a list of files to be indexed. So only the URLs files of these lists are indexed instead of "browsing" through all trees. This mechanism is similar to Sitemaps in the Web Connector.

Microsoft File Connector provides the Interface IndexFileListerPlugin (index-filelister-spi.jar) to list documents together with additional properties from an index file for crawling.

public interface IndexFileListerPlugin {

boolean isIndexFile(ReadonlyFile file);

void init(Properties properties);

Collection<Map.Entry<ReadonlyFile, TypesProtos.Item>> listIndexFile(FilesystemContext context, ReadonlyFile  indexFile);

}

messdk-generated.jar and protobuf-java-3.0.0.jar from Java service API together with index-filelister-spi.jar are needed to implement the IndexFileListerPlugin. After implementing the plugin, it should be configured as follows. Provide the path of JAR file containing the implementation in the „Index File Lister Plugin“ field. The optional „Index File Lister Plugin Property“ fields define the properties needed by the plugin to be initialized with.

Index Files are queued „Queue Size“ during directory traversal, which are then handled by parallel threads „Thread Count”. The option „Skip unchanged Index File Listing during Incremental Traversal“ should be selected only if the option „Full Traversal Interval” is also configured. By this means index files which are not changed are ignored during incremental traversals “Crawler Interval”.

The Microsoft File Connector contains a preconfigured content mapping file (XML) which provides necessary rules to be applied on documents according to their content type. Sometimes it is necessary to change these rules and save this mapping file in separate location. In order to use this modified mapping file, it is necessary to configure this file’s location in “Content Type Mapping Description File”.

Content Location Optimization

For crawling large files it is beneficial to use Content Location Optimzation. For example if you want to crawl Outlook PST Files.

Configure the mount point according to the screenshot above. The following configuration Options are needed:

• “Root Directory (UNC Path)”: Use the same root path you used in the source config above.
• “Root Directory (Mount Path)”: The local path to which the UNC Path is mounted.
• “Files Pattern (Regex)”: A regex pattern matchting those files which should be indexed using Content Location Optimization.

The Content Location Optimization feature requires that the UNC Path is mounted locally. This can be configured using the System Configuration in the Management Center:

1. Create the local folder using Filemin:
   
   
2. Grant Permissions to the Mindbreeze user (mes):
   
3. Add a CIFS mount using the “Disk and Network Filesystems” Module:
   
4. Configure the mount:
   
5. After you press “Create” the Network filesystem will be mounted and is ready for use.

Crawling Outlook PST Files

In addition to the File Crawler configuration above you also need to add an Outlook PST Datasource to crawl PST Files remove “Default” from Category Instance field.

And finally ensure that a Filter Plugin is enabled for .pst extension.

Credentials

The user must have read access permission in order to crawl a share.

• “Username”:
• “Domain”:
• “Password”:
• “LDAP Server”: Provide this only if you want to override LDAP setting configured under “Network Setting” tab.

Additional Settings

• “Always Update Files Matching Regex”: Documents matching this regex will always be sent to filter service no matter if they were changed or not.
• “Ignore Content of Documents without Extension”: If this checkbox is selected than the automatic mimetype detection is deactivated for documents without extension. The contents of these documents are not indexed.
• “Disable Default Extension”: If selected documents which have no extension and an automatic mime type detection attempt is failed have no extension.
• “Fetch Preview Content from Datasource”: To provide PDF Preview for PDF documents, the binary content of PDF documents is stored in the index. If this option is selected, the binary content will be fetched directly from datasource instead. The storing of the PDF Content in the index can then be disabled in the Filter Configuration reducing the disk usage of the index.
• “Enable Heap Dump On OutOfMemory”: If the crawler needs more memory than configured in Plugins.xml <vm_arg> a heap dump is generated in log directory.
• “Max. Retry Duration by Filter Connection Problems”: Maximum amount of time the crawler is allowed to retry sending a document to the filter service during connection problems.
• “Retry Interval during Repository Connection Problems”: The amount of time the crawler waits before retrying to connect to the data source during connection problems.
• “Max. Retry Duration during Repository Connection Problems”: Maximum amount of time the crawler is allowed to retry connecting to the data source during connection problems.

Open Search Results

Search results from a Microsoft File datasource (Microsoft Word, Microsoft Excel and Microsoft Powerpoint) are opened on Windows 10 directly in the respective program if the current user is signed in to the respective fileserver and Microsoft Office 2019 is installed.