Configuration
Connector for GraphQL

Introduction

The Connector for GraphQL connects the GraphQL API with the Mindbreeze InSpire search. This allows items that were fetched via GraphQL to be used in Mindbreeze Insight Apps. The connector iterates over all objects retrieved from the GraphQL API and transforms them into indexable entities for the Mindbreeze search index.

Administrators can start with the configuration immediately, since the Connector for GraphQL is included in Mindbreeze InSpire by default.

Core Concepts

Data Acquisition

Authentication: The connector handles the necessary security handshake to access the GraphQL endpoint, ensuring authorized data retrieval.
GraphQL Queries: The connector uses configurable GraphQL queries to define exactly which fields and relationships should be fetched from the server.
Pagination: To handle large datasets, the connector implements pagination logic, iterating through result sets to ensure all available items are discovered and indexed.
Items: Each distinct entity returned by the GraphQL query (e.g., a "User", "Product", or "Document") is treated as an individual "Item" and mapped to a single Mindbreeze documents.

Data Transformation & Extraction

JSONPath: The connector heavily uses JSONPath (RFC 9535) syntax. This allows administrators to define the path to a specific value (e.g., $.user.profile.email) via configuration.
JSONPointer (Deprecated): The connector supports JSONPointer (RFC 6901) syntax. This allows administrators to define the path to a specific value (e.g., /user/profile/email) via configuration.
Pivoting: A normalization step that transforms arrays of key-value pairs into objects.
This simplifies subsequent metadata extraction.
Metadata Extraction: All data provided in an item is extracted and stored as metadata. Complex metadata is flattened to ease the access.

i18n Metadata Extraction: Mindbreeze i18n metadata is extracted from multi-language fields, allowing metadata displayed to users in the correct language.
Flattening: Recursively traverses nested JSON trees and converts them into underscore-separated properties (e.g., address_city), flattening nested data into simple metadata for more convenient access.
ACLs (Access Control Lists): The connector extracts security principals from the JSON response and maps them to GRANT or DENY permissions, ensuring that only authorized users can see the resulting search entries.

Data Processing Pipeline

The crawler orchestrates the retrieval of data from a GraphQL API and transforms it into indexed documents through a sequential pipeline of metadata enrichment and flattening.

Authentication: Fetch an OAuth token to authorize API requests.
Data Retrieval: Request objects from GraphQL using a defined query.
Pagination Management: Large result sets are supported by using offset, cursor, or single-page retrieval. Note that pagination is handled independently; therefore, pivoting cannot be applied to the JSON items returned by the API.
Item Processing: Iterate through every retrieved item and perform the following sequence:
1. Pivoting: Execute sequential data transformations where each step builds on the previous.
2. i18n Extraction: Extract internationalization (i18n) metadata based on the pivoting results.
3. Main Metadata Assignment: Set primary metadata based on the pivoting results.
4. ACL Metadata Assignment: Define access control lists (ACL) based on the pivoting results.
5. Mindbreeze Item Metadata Extraction / Flattening: Convert nested structures into simple, flat metadata and a Mindbreeze item metadata.
6. Indexing: Commit the final processed document to the search index.

Configuration

This chapter explains the basic configuration of the Connector for GraphQL in a Quickstart Guide. If you need additional information for your use case, you will find a list of all settings in the following chapters.

Quickstart Guide

Step 1: Creation of the index

Add a new index in the tab “Indices” using the button “+ Add Index”. Select the desired “Index Node” and “Client Service” and select the option “Connector for GraphQL” in “Data Source”. Then confirm your entries with “Apply”.

Step 2: Configuration of the data source

In the next step, the data source must be configured in the section “Data Sources”.

The following settings must be set in the following sections to create a basic configuration:

Setting	Example
Endpoint for GraphQL	https://company-dev.atlassian.net/gateway/api/graphql

Setting

Example

GraphQL Query

query custom_query_name($cloudID: ID!, $cursor: String) {
    jira {
        issueSearchStable(
            cloudId: $cloudID
            first: 50
            after: $cursor
            issueSearchInput: {jql: "project = ‘DEV’"}
        ){
        totalCount
        edges {
            node {...}
        }
        pageInfo {
            hasNextPage
            endCursor
        }
     }
   }
}

Setting	Example
Pagination Type	Choose the pagination type to be used: Cursor Offset
Results Path	Cursor based Pagination: $.data.jira.issueSearchStable.edges Offset based Pagination: $.data.GlideRecord_Query.kb_knowledge[‘_results’]
Cursor pagination End Condition	Only necessary when the setting “Pagination Type” is set to the option “Cursor”. The following options are available: “Has Next Flag” “No end cursor” “No results”

Setting	Example
Key Path	$.sys_id.value

Save the configuration with “Save.” The basic configuration for the Connector for GraphQL is now completed.

Available settings

Section “Performance Settings” (Advanced Settings)

Setting	Description	Default setting
Concurrent Filter and Index Dispatch Threads	The number of threads dispatching data to the filter service and index service. If the number is smaller or equal to zero, serial dispatching is caused.	10
Connection Timeout (Seconds)	Defines in seconds how long the connector will wait for a response before it cancels the API call.	30
Maximum Fetch Retries	Defines the maximum number of retries that will be attempted, when the server sends certain throttling responses (e.g., 429).	5
User Agent	Defines the header of the User Agent to be used for API calls.	NON\|SV\|Mindbreeze\|ConnectorForGraphQL/1.0
Trust all SSL certificates	Allows the use of unsecured connections. This can be used for test systems, for example. Attention: Must not be enabled in the production environment.	Deactivated

Section “Diagnostic Settings”

Setting	Description	Default Setting/Example
Log all Requests	When this setting is enabled, all requests to the GraphQL API are logged in a file named “request-log.csv.”	Deactivated
Dump Network Traffic	When this setting is enabled, the system saves a log file for every GraphQL network request, including the full request, response bodies and headers. Attention: Be aware that secrets like authentication headers might also be written to the network dump files.	Deactivated
Dump Pivoting Results	When this setting is enabled, metadata pivoting intermediate data structures are logged in the folder “pivoting-dump”.	Deactivated

Section “GraphQL Connection”

Setting

Description

Default Setting/Example

Endpoint for GraphQL

Defines the URL where GraphQL queries are sent to.

Hint: The structure of the endpoint is dependent on the used data source, as shown in the column “Default Setting/Example”.

Example of a ServiceNow endpoint:

https://dev123456.service-now.com/now/nav/ui/classic/params/target/graphql-explorer.do

Example of an Atlassian Jira endpoint:

https://company-dev.atlassian.net/gateway/api/graphql

GraphQL Credential

Defines the credentials used to authenticate requests to the GraphQL endpoint.

The following types of Credentials are supported:

Type	Used for…
Username/ Password	BASIC-Authentication
Password	Static Token/API Key-Authentication
OAuth 2	OAuth2 Authentication

Default Setting:
None

Section “Query”

Setting	Description	Example
GraphQL Query	Defines the GraphQL query.	Query examples can be found in the chapters Cursor-based Pagination and Offset-based Pagination.
Query Variable Value JSON	Defines custom query variables that can be used in the GraphQL query.	{“limit“:3} For more context, see the Query chapter in the chapters Cursor-based Pagination and Offset-based Pagination.
Request Headers	Defines custom headers that will be sent to the GraphQL endpoint.	"X-ExperimentalApi": "JiraIssueSearch"

Section “Pagination”

Setting

Description

Default Setting/Example

Pagination Type*

This setting defines which type of pagination should be used and whether the query utilizes cursor-based or offset-based pagination.

The following options are available:

Option	Description
None	No pagination is done.
Cursor	The page iteration is done with cursors.
Offset	The page iteration is done with offsets (e.g. start, limit).

None

Results Path*

A JSON-Path to the JSON value that defines the results array.

Cursor Paged Pagination:

$.data.jira.issueSearchStable.edges

Offset Paged Pagination:

$.data.GlideRecord_Query.kb_knowledge[‘_results’]

Cursor pagination End Condition*

This setting defines the condition used to determine when all available documents have been retrieved.

Hint: Is not supported with offset pagination.

Option	Description
Has Next Flag	A Boolean flag is present that states the presence of another page.
No end Cursor	The last page won’t return a next cursor.
No results	No returned results means that no results are left.

Default setting: “Has Next Flag”

Values:
“Has Next Flag”, “No end cursor”, “No results”

Next Page Path

A JSON-Path to the JSON value that defines, if there is a next page.

Example:

$.data.jira.issueSearchStable.pageInfo.hasNextPage

For more information, see the Mindbreeze Configuration for Cursor Paged Pagination.

Next Cursor Path

A JSON-Path to the JSON value that defines the next cursor.

Example:

$.data.jira.issueSearchStable.pageInfo.endCursor

For more information, see the Mindbreeze Configuration for Cursor Paged Pagination.

Paging Variable Name

Defines the variable that is sent with the query to control the next page iteration. The variable to be defined depends on the configured query and its properties.

Example for Cursor Paged Pagination:

cursor

Example for Offset Paged Pagination:

offset

Section “Metadata Settings”

Setting

Description

Default Setting/Example

Pivoting JSON Transformation

A JSON path to the JSON Array that should be pivoted.

Option	Description
Pivoting Array Path	A JSON path to the JSON Array that should be pivoted.
Pivoting Key Path	A JSON path to the JSON Key that should be used for pivoting.
Pivoting Value Path	A JSON path to the JSON Value that should be used for pivoting. If this field is left blank, the entire element is used as the value for the pivot.

See the chapter Examples for Pivoting.

Key Path*

A JSON path to the JSON value that acts as the unique key for Mindbreeze documents sent to the Mindbreeze filter and index service by the connector.

Example:

$.sys_id.value

For more context, see the code example in the “Result” chapter in the chapter Offset-based Pagination.

URL Path

A JSON path to the JSON value that should be used as Mindbreeze document URL.

Example:

$.url.value

Title Path

A JSON path to the JSON value that defines the document’s title.

If the title should be i18n metadata, the same path needs to be configured for the setting “I18N Metadata Extraction”.

Example:

$.text.value

For more context, see the code example in the “Result” chapter in the chapter Offset-based Pagination.

‘Created Date’ Path

A JSON path to the JSON value that specifies the document’s creation date.

Example:

$.sys_created_on.value

For more context, see the code example in the “Result” chapter in the chapter Offset-based Pagination.

‘Modified Date’ Path

A JSON path to the JSON value that specifies the document’s last modification date.

Example:

$.sys_updated_on.value

For more context, see the code example in the “Result” chapter in the chapter Offset-based Pagination.

Date Format

The date pattern used to parse the values provided by the created and modified date paths.

Hint: By default, the date format according to ISO 8601 is used. If a custom date format is needed, configure the date format here.

Example:

MM-dd'T'HH:mm'Z'

Default Timezone

The default timezone that is used for parsing dates without timezone.

Example:

America/Phoenix

UTC

Extension Path

A JSON path to the JSON value that defines the file extension for the document.

Example:

$.extension.value

Default Extension

The fallback file extension used, if the previous setting “Extension Path” is configured or if the JSON does not contain the required value.

Default setting:

Json txt

I18N Metadata Extraction

Specifies the metadata containing the i18n values.

If the data is provided as a JSON array, it must be pivoted first. All keys and values must be strings, and keys must comply with the ISO 639 standard.

Option	Description
I18N Path	A JSON path to the JSON object that contains the i18n metadata.

See examples in the chapter Examples for i18n.

GraphQL Prefix

(Advanced Settings)

This setting prepends a GraphQL prefix to the metadata.

Default setting:

grql

Section “Content Settings”

The content for the documents will be provided by the JSONFilter.

Setting	Description	Default Setting/Example
Content Path	A JSON path to the object that contains the content.	Example: $.document.content
Content Extraction Paths	Multiple JSON paths, separated by newline, that localize the content in the JSON object provided by the setting “Content Path”.	Default: $ (everything)
Flatten Metadata	If checked, metadata will be flattened for the object provided by the setting “Content Path”.	Default: false
Content Metadata Name	The name for the content item metadata provided by the JSONFilter.	Default: json

Section “ACL Settings”

Setting

Description

Example

Enable Default ACLs

When this setting is enabled, the default principals from the following setting “Default ACL Principals” will be added as GRANT permissions for all documents.

The principals are separated by newline.

Default setting:

Deactivated

Default ACL Principals

The principals defined in this setting will be added to the document’s GRANT principals.

Example:

marketing_group

developer_group

ACLs

Defines the property used to populate the Access Control List.

Option	Description
ACL Path	A JSON path to the JSON value that specifies the document’s permissions / ACLs.
ACL Separator	The delimiter that is used for splitting ACL entries.
ACL Type	GRANT/DENY

Hinweis:
Per default the values are not split, so “group_x,group_y” would be added as a single principal but with delimiter “,” the principals “group_x” and “group_y” would be added.

Grant access to ‘everyone’ per Default

When this setting is enabled, search result access is granted to all users by default.

This might be useful for public data.

Default setting:

Deactivated

Examples of cursor based and offset based pagination

Cursor-based Pagination

The following examples utilize queries and responses from Atlassian Jira for illustrative purposes.

Please note that this is just an example. For actual production crawling use cases, please use the dedicated Mindbreeze Atlassian Jira Connector instead to crawl Jira documents.

Query

The following query is defined:

query custom_query_name($cloudID: ID!, $cursor: String) {

jira {

issueSearchStable(

cloudId: $cloudID

first: 50

after: $cursor

issueSearchInput: {jql: "project = 'DEV'"}

){

totalCount

edges {

node {

summary

}

pageInfo {

hasNextPage

endCursor

}

The following configuration options are in the defined query:

Configuration option	Defined in the query as…
Query variables	cloudID
Paging variable name	cursor
Results path	edges
Next page path / Next cursor path	hasNextPage endCursor

Result

To see if the defined query creates the output that is needed, any GraphQL explorer can be used. Due to how the query is defined, the following query variable and header are also needed:

Query variable	{„cloudID“: „my-organization-1234“}
Headers	{"X-ExperimentalApi" : "JiraIssueSearch"}

The result then looks like this:

Since the defined query returns the expected results, the next step is to configure the query and the cursor-based pagination in Mindbreeze InSpire.

Mindbreeze Configuration

To fully configure the defined query and the cursor-based pagination in Mindbreeze InSpire, the following settings must be configured in the following sections.

Section “Query”:

Setting

Entry

GraphQL Query

As defined in the previous chapter “Query”:

query custom_query_name($cloudID: ID!, $cursor: String) {

jira {

issueSearchStable(

cloudId: $cloudID

first: 50

after: $cursor

issueSearchInput: {jql: "project = 'DEV'"}

){

totalCount

edges {

node {

summary

}

pageInfo {

hasNextPage

endCursor

}

Query Variable Value JSON

{„cloudID“: „my-organization-1234“}

Static variables (e.g., IDs or Project Names) should be configured within the „Query Variables” setting.

Hint: Dynamic variables (e.g., cursors or offsets) must not be defined here, as their values change during execution.

Regardless of the type, all variables must be defined as parameters within the query to ensure they can be modified during the crawling process.

Section “Pagination”:

Configuration Option	Example value	Description
Pagination Type	CURSOR	The paging is done via cursor.
Results Path	$.data.jira.issueSearchStable.edges	The “edges” contain the results array.
Next Page Path	$.data.jira.issueSearchStable.pageInfo.hasNextPage	An identifier checks, if there is a next page.
Next Cursor Path	$.data.jira.issueSearchStable.pageInfo.endCursor	The cursor fetches the next page.
Paging Variable Name	cursor	This variable is defined in the query and is used to inject the new cursor that is fetched while crawling.

The Mindbreeze configuration is then completed and can be saved.

Offset-based Pagination

The following examples utilize queries and responses from ServiceNow for illustrative purposes.

Please note that this is just an example. For actual production crawling use cases, please use the dedicated Mindbreeze InSpire ServiceNow Connector instead to crawl ServiceNow documents.

Query

The following query is defined:

query get_knowledge_articles($offset: Int) {
    GlideRecord_Query {
        kb_knowledge(pagination: {limit: 2, offset: $offset}) {
            _results {
                sys_id {
                    value
                }
                text {
                    value
                }
                sys_updated_on {
                    value
                }
                sys_created_on {
                    value
                }
            }
        }
    }
}

The following configuration options are in the defined query:

Configuration option	Defined in the query as…
Paging variable name	offset
Results path	_results

Result

To see if the defined query creates the output that is needed, any GraphQL explorer can be used. Due to how the query is defined, only the query itself will be needed to get the following result:

Since the defined query returns the expected results, the next step is to configure the query and the offset-based pagination in Mindbreeze InSpire.

Mindbreeze Configuration

To fully configure the defined query and the offset-based pagination in Mindbreeze InSpire, the following settings must be configured in the following sections.

Section “Query”:

Setting

Entry

GraphQL Query

As defined in the previous chapter “Query”:

Section „Pagination“:

Configuration Option	Example value	Description
Pagination Type	OFFSET	The paging is done via offset.
Results Path	$.data.GlideRecord_Query.kb_knowledge[‘_results’]	“_results” contain the results array.
Next Page Path		Not used with offset pagination.
Next Cursor Path		Not used with offset pagination.
Paging Variable Name	offset	This variable is defined in the query and is used to inject the new offset that is calculated while crawling.

The Mindbreeze configuration is then completed and can be saved.

Examples for Pivoting

“Pivoting” refers to the process of converting a JSON array into a JSON object.

Key/Value Pivoting

Mindbreeze configuration:

Key Pivoting 1

Mindbreeze configuration:

Key Pivoting 2

Mindbreeze configuration:

Examples for i18n

The Connector for GraphQL supports i18n metadata that can be configured in the metadata settings. Only JSON objects are supported as i18n metadata. All keys and values must be strings, and keys must comply with the ISO 639 standard.

i18n Object metadata

Original:
{

"custom_objects": {

"i18n": {

"en": "Account ID",

"es": "ID de cuenta",

"fr": "ID du compte"

}

Mindbreeze configuration:

I18N Array metadata

If i18n metadata is provided in a JSON array, it has to be pivoted first in order to resolve it.

Original:
{

"custom_objects": {

"i18n": [{

"lang": "en",

"val": "Account ID"

}, {

"lang": "es",

"val": "ID de cuenta"

}, {

"lang": "fr",

"val": "ID du compte"

}

]

}

Mindbreeze configuration for pivoting:

Pivoting Result:

{

"custom_objects": {

"i18n": {

"en": "Account ID",

"es": "ID de cuenta",

"fr": "ID du compte"

}

Mindbreeze configuration for i18n:

i18n Title

If i18n metadata should be used as title metadata (multilingual title) the JSON path has to be configured in the settings “Title Path” and “I18N Path”.

Configuration
Connector for GraphQL

Introduction

Core Concepts

Data Acquisition

Data Transformation & Extraction

Data Processing Pipeline

Configuration

Quickstart Guide

Step 1: Creation of the index

Step 2: Configuration of the data source

Available settings

Section “Performance Settings” (Advanced Settings)

Section “Diagnostic Settings”

Section “GraphQL Connection”

Section “Query”

Section “Pagination”

Section “Metadata Settings”

Section “Content Settings”

Section “ACL Settings”

Examples of cursor based and offset based pagination

Cursor-based Pagination

Query

Result

Mindbreeze Configuration

Offset-based Pagination

Query

Result

Mindbreeze Configuration

Examples for Pivoting

Key/Value Pivoting

Key Pivoting 1

Key Pivoting 2

Examples for i18n

i18n Object metadata

I18N Array metadata

i18n Title

Download PDF

Download PDF

{{{i18n.refineSearch}}}

Configuration Connector for GraphQL

Introduction

Core Concepts

Data Acquisition

Data Transformation & Extraction

Data Processing Pipeline

Configuration

Quickstart Guide

Step 1: Creation of the index

Step 2: Configuration of the data source

Available settings

Section “Performance Settings” (Advanced Settings)

Section “Diagnostic Settings”

Section “GraphQL Connection”

Section “Query”

Section “Pagination”

Section “Metadata Settings”

Section “Content Settings”

Section “ACL Settings”

Examples of cursor based and offset based pagination

Cursor-based Pagination

Query

Result

Mindbreeze Configuration

Offset-based Pagination

Query

Result

Mindbreeze Configuration

Examples for Pivoting

Key/Value Pivoting

Key Pivoting 1

Key Pivoting 2

Examples for i18n

i18n Object metadata

I18N Array metadata

i18n Title

Download PDF

Download PDF

Configuration
Connector for GraphQL