Configuration
Connector for GraphQL

Introduction

The Connector for GraphQL connects the GraphQL API with the Mindbreeze InSpire search. This allows items that were fetched via GraphQL to be used in Mindbreeze Insight Apps.

The connector iterates over all objects retrieved from the GraphQL API and transforms them into Indexable entities for the Mindbreeze search index.

Administrators can start with the configuration immediately, since the Connector for GraphQL is included in Mindbreeze InSpire by default.

Core Concepts

Data Acquisition

Authentication: The connector handles the necessary security handshake to access the GraphQL endpoint, ensuring authorized data retrieval.
GraphQL Queries: The connector uses configurable GraphQL queries to define exactly which fields and relationships should be fetched from the server.
Pagination: To handle large datasets, the connector implements pagination logic, iterating through result sets to ensure all available items are discovered and indexed.
Items: Each distinct entity returned by the GraphQL query (e.g., a "User", "Product", or "Document") is treated as an individual "Item" and mapped to a single Mindbreeze documents.

Data Transformation & Extraction

JSONPointer: The connector heavily uses JSONPointer (RFC 6901) syntax. This allows administrators to define the path to a specific value (e.g., /user/profile/email) via configuration.
Pivoting: A normalization step that transforms arrays of key-value pairs into objects.
This simplifies subsequent metadata extraction.
Metadata Extraction: All data provided in an item, is extracted and stored as metadata. Complex metadata is flattened to ease the access.
i18n Metadata Extraction: Mindbreeze i18n metadata is extracted from multi-language fields, allowing metadata displayed to users in the correct language.
Flattening: Recursively traverses nested JSON trees and converts them into underscore-separated properties (e.g., address_city), flattening nested data into simple metadata for more convenient access.
ACLs (Access Control Lists): The connector extracts security principals from the JSON response and maps them to GRANT or DENY permissions, ensuring that only authorized users can see the resulting search entries.

Data Processing Pipeline

The crawler orchestrates the retrieval of data from a GraphQL API and transforms it into indexed documents through a sequential pipeline of metadata enrichment and flattening.

Authentication: Fetch an OAuth token to authorize API requests.
Data Retrieval: Request objects from GraphQL using a defined query.
Pagination Management: Large result sets are supported by using offset, cursor, or single-page retrieval. Note that pagination is handled independently; therefore, pivoting cannot be applied to the JSON items returned by the API.
Item Processing: Iterate through every retrieved item and perform the following sequence:
1. Pivoting: Execute sequential data transformations where each step builds on the previous.
2. Main Metadata Assignment: Set primary metadata based on the pivoting results.
3. ACL Metadata Assignment: Define access control lists (ACL) based on the pivoting results.
4. i18n Extraction: Extract internationalization (i18n) metadata based on the pivoting results.
5. Mindbreeze Item Metadata Extraction / Flattening: Convert nested structures into simple, flat metadata and a Mindbreeze item metadata.
6. Indexing: Commit the final processed document to the search index.

Configuration

This chapter explains the basic configuration of the Connector for GraphQL in a Quickstart Guide. If you need additional information for your use case, you will find a list of all settings in the following chapters.

Quickstart Guide

Step 1: Creation of the index

Add a new index in the tab “Indices” using the button “+Add Index”. Select the desired “Index Node” and “Client Service” and select the option “Connector for GraphQL” in “Data Source”. Then confirm your entries with “Apply”.

After the index is created, you can, for example, change the name of the index or apply a filter service.

Step 2: Configuration of the data source

In the next step, the data source must be configured in the section “Data Sources”.

The following settings must be set in the following sections to create a basic configuration:

Setting

Entry

Endpoint for GraphQL

Example:

https://company-dev.atlassian.net/gateway/api/graphql

Setting

Entry

GraphQL Query

Example:

query custom_query_name($cloudID: ID!, $cursor: String) {
    jira {
        issueSearchStable(
            cloudId: $cloudID
            first: 50
            after: $cursor
            issueSearchInput: {jql: "project = ‘DEV’"}
        ){
        totalCount
        edges {
            node {...}
        }
        pageInfo {
            hasNextPage
            endCursor
        }
     }
   }
}

Setting	Entry
Pagination Type	Choose the pagination type to be used: Cursor Offset
Results Pointer	Example for Cursor Paged Pagination: /data/jira/issueSearchStable/edges Example for Offset Paged Pagination: /data/GlideRecord_Query/kb_knowledge/_results
Cursor pagination End Condition	Only necessary when the setting “Pagination Type” is set to the option “Cursor”. The following options are available: “Has Next Flag” “No end cursor” “No results”

Setting

Entry

Key Pointer

Example:

/sys_id/value

Save the configuration with “Save.” The basic configuration for the Connector for GraphQL is now completed.

Available settings

Section “Performance Settings” (Advanced Settings)

Setting	Description	Default setting
Concurrent Filter and Index Dispatch Threads	The number of threads dispatching data to the filter service and index service. If the number is smaller or equal to zero, serial dispatching is caused.	Default setting: 10
Connection Timeout (Seconds)	Defines in seconds how long the connector will wait for a response before it cancels the API call.	Default setting: 30
Maximum Fetch Retries	Defines the maximum number of retries that will be attempted, when the server sends certain throttling responses (e.g., 429).	Default setting: 5
User Agent	Defines the header of the User Agent to be used for API calls.	Default setting: NON\|SV\|Mindbreeze\|ConnectorForGraphQL/1.0
Trust all SSL certificates	Allows the use of unsecured connections. This can be used for test systems, for example. Attention: Must not be enabled in the production environment.	Default setting: Deactivated

Section “Diagnostic Settings”

Setting	Description	Default setting/Example
Log all Requests	When this setting is enabled, all requests to the GraphQL API are logged in a file named “request-log.csv.”	Default setting: Deactivated
Dump Network Traffic	When this setting is enabled, the system saves a log file for every GraphQL network request, including the full request, response bodies and headers. Attention: Be aware that secrets like authentication headers might also be written to the network dump files.	Default setting: Deactivated
Dump Pivoting Results	When this setting is enabled, metadata pivoting intermediate data structures are logged in the folder ‘pivoting-dump’	Default setting: Deactivated

Section “GraphQL Connection”

Setting

Description

Default setting/Example

Endpoint for GraphQL

Defines the URL where GraphQL queries are sent to.

Hint: The structure of the endpoint is dependent on the used data source, as shown in the column “Default setting/Example”.

Example of a ServiceNow endpoint:

https://dev123456.service-now.com/now/nav/ui/classic/params/target/graphql-explorer.do

Example of an Atlassian Jira endpoint:

https://company-dev.atlassian.net/gateway/api/graphql

GraphQL Credential

Defines the credentials used to authenticate requests to the GraphQL endpoint.

The following types of Credentials are supported:

Type	Used for…
Username/Password	BASIC-Authentication
Password	Static Token/API Key-Authentication
OAuth 2	OAuth2 Authentication

Default setting:

None

Section “Query”

Setting	Description	Example
GraphQL Query	Defines the GraphQL query.	See the chapter Query.
Query Variable Value JSON	Defines custom query variables that can be used in the GraphQL query.	Example: {“limit“:3} For more context, see the chapter Query.
Request Headers	Defines custom headers that will be sent to the GraphQL endpoint.	Example: "X-ExperimentalApi": "JiraIssueSearch"

Section “Pagination”

Setting

Description

Default setting/Example

Pagination Type*

This setting specifies whether the query utilizes cursor-based or offset-based pagination.

Defines the type of pagination to be used.

The following options are available:

Option	Description
None	No pagination is done
Cursor	The page iteration is done with cursors
Offset	The page iteration is done with offsets (e.g.: start, limit)

Default setting:

Cursor

Results Pointer*

A JSON-Pointer to the JSON value that defines the results array.

Example for Cursor Paged Pagination:

/data/jira/issueSearchStable/edges

Example for Offset Paged Pagination:

/data/GlideRecord_Query/kb_knowledge/_results

Cursor pagination End Condition*

This setting defines the condition used to determine when all available documents have been retrieved.

Hint: Is not supported with offset pagination.

Option	Description
Has Next Flag	A Boolean flag is present that states the presence of another page
No end Cursor	The last page won’t return a next cursor.
No results	No returned results – no results left

Default setting: “Has Next Flag”

Values:
“Has Next Flag”, “No end cursor”, “No results”

Next Page Pointer

A JSON-Pointer to the JSON value that defines, if there is a next page.

Example:

/data/jira/issueSearchStable/pageInfo/hasNextPage

For more information, see the Mindbreeze Configuration for Cursor Paged Pagination.

Next Cursor Pointer

A JSON-Pointer to the JSON value that defines the next cursor.

Example:

/data/jira/issueSearchStable/pageInfo/endCursor

For more information, see the Mindbreeze Configuration for Cursor Paged Pagination.

Paging Variable Name

Defines the variable that is sent with the query to control the next page iteration. The variable to be defined depends on the configured query and its properties.

Example for Cursor Paged Pagination:

cursor

Example for Offset Paged Pagination:

offset

Section “Metadata Settings”

Setting

Description

Default setting/Example

Pivoting JSON Transformation

A JSON pointer to the JSON Array that should be pivoted.

Option	Description
Pivoting Array Pointer	A JSON pointer to the JSON Array that should be pivoted.
Pivoting Key Pointer	A JSON pointer to the JSON Key that should be used for pivoting.
Pivoting Value Pointer	A JSON pointer to the JSON Value that should be used for pivoting. If this field is left blank, the entire element is used as the value for the pivot.

Example:

See 2.4 Examples for Pivoting

Key Pointer*

A JSON pointer to the JSON value that acts as the unique key for Mindbreeze documents sent to the Mindbreeze filter and index service by the connector.

Example:

/sys_id/value

For more context, see the code example in the chapter Result for Offset Paged Pagination.

URL Pointer

A JSON pointer to the JSON value that should be used as Mindbreeze document URL.

Example:

/url/value

Title Pointer

A JSON pointer to the JSON value that defines the document’s title.

Example:

/text/value

For more context, see the code example in the chapter Result for Offset Paged Pagination.

Content Pointer

A JSON pointer to the JSON value that contains the document’s primary content.

Example:

/content/value

‘Created Date’ Pointer

A JSON pointer to the JSON value that specifies the document’s creation date.

Example:

/sys_created_on/value

For more context, see the code example in the chapter Result for Offset Paged Pagination.

‘Modified Date’ Pointer

A JSON pointer to the JSON value that specifies the document’s last modification date.

Example:

/sys_updated_on/value

For more context, see the code example in the chapter Result for Offset Paged Pagination.

Date Format

The date pattern used to parse the values provided by the created and modified date pointers.

Hint: By default, the date format according to ISO 8601 is used. If a custom date format is needed, configure the date format here.

Example:

MM-dd'T'HH:mm'Z'

Default Timezone

The default timezone that is used for parsing dates without timezone

Example:

America/Phoenix

UTC

Extension Pointer

A JSON pointer to the JSON value that defines the file extension for the document.

Example:

/extension/value

Default Extension

The fallback file extension used, if the previous setting “Extension Pointer” is configured or if the JSON does not contain the required value.

Default setting:

txt

I18N Metadata Extraction

Specify the metadata containing the i18n values.
If the data is provided as a JSON array, it must be pivoted first.
All keys and values must be strings, and keys must comply with the ISO 639 standard.

Option	Description
I18N Pointer	A JSON pointer to the JSON Object that contains the I18N metadata.

Example:

See 2.5 Examples for I18N

GraphQL Prefix

(Advanced Settings)

This setting prepends a GraphQL prefix to the metadata.

Default setting:

grql

Section “ACL Settings”

Setting

Description

Example

Enable Default ACLs

When this setting is enabled, the default principals from the setting ‘Default ACL Principals’ will be added as GRANT permissions for all documents.

The principals are separated by newline.

Default setting:

Deactivated

Default ACL Principals

The principals defined in this setting will be added to the document’s GRANT principals.

Example:

marketing_group

developer_group

ACLs

Defines the property used to populate the Access Control List.

Option	Description
ACL Pointer	A JSON pointer to the JSON value that specifies the document’s permissions / ACLs
ACL Separator	The delimiter that is used for splitting ACL entries.
ACL Type	GRANT/DENY

Note:
Per default the values are not split so “group_x,group_y” would be added as a single principal but with delimiter “,” the principals “group_x” and “group_y” would be added.

Grant access to ‘everyone’ per Default

When this setting is enabled, search result access is granted to all user s by default.

This might be useful for public data.

Default setting:

Deactivated

Examples of cursor paged and offset paged pagination

Cursor Paged Pagination

The following examples utilize queries and responses from Atlassian Jira for illustrative purposes.

Please note that this is just an example. For actual production crawling use cases, please use the dedicated Mindbreeze Atlassian Jira Connector instead to crawl Jira documents.

Query

Configuration option:

Query Variables
Paging Variable Name
Results Pointer
Next Page Pointer / Next Cursor Pointer

Result

{
    "data": {
        "jira": {
            "issueSearchStable": {
                "totalCount": 2,
                "edges": [
                    {"<<ITEM-1>>"},
                    {"<<ITEM-2>>"}
                ],
                "pageInfo": {
                    "hasNextPage": true,
                    "endCursor": "<<END-CURSOR>>"
                }
            }
        }
    }
}

Mindbreeze Configuration

Query Variable Value JSON :

{„cloudID“: „my-organization-1234“}

Static variables (e.g., IDs or Project Names) should be configured within the „Query Variables” setting.

Hint: Dynamic variables (e.g., cursors or offsets) must not be defined here, as their values change during execution.

Regardless of the type, all variables must be defined as parameters within the query to ensure they can be modified during the crawling process.

Pagination configuration:

Configuration option	Example value	Description
Query supports Cursor Pagination	true	The paging is done via cursor.
Query supports Offset Pagination	false	The paging is not done via offset.
Results Pointer	/data/jira/issueSearchStable/edges	The edges contain the results array.
Next Page Pointer	/data/jira/issueSearchStable/pageInfo/hasNextPage	An identifier checks, if there is a next page.
Next Cursor Pointer	/data/jira/issueSearchStable/pageInfo/endCursor	The cursor fetches the next page.
Paging Variable Name	cursor	This variable is defined in the query and is used to inject the new cursor that is fetched while crawling.

Offset Paged Pagination

The following examples utilize queries and responses from ServiceNow for illustrative purposes.

Please note that this is just an example. For actual production crawling use cases, please use the dedicated Mindbreeze ServiceNow Connector instead to crawl ServiceNow documents.

Query

query get_knowledge_articles($offset: Int) {
    GlideRecord_Query {
        kb_knowledge(pagination: {limit: 2, offset: $offset}) {
            _results {
                sys_id {
                    value
                }
                text {
                    value
                }
                sys_updated_on {
                    value
                }
                sys_created_on {
                    value
                }
            }
        }
    }
}

Configuration option:

Paging Variable Name
Results Pointer

Result

{
    "data": {
        "GlideRecord_Query": {
            "kb_knowledge": {
                "_results": [
                    {
                        "sys_id": {
                            "value": "0b48fd75474321009db4b5b08b9a71c2"
                        },
                        "text": {
                            "value": "<p>CONTENTp>"
                        },
                        "sys_updated_on": {
                            "value": "2014-12-19 15:54:36"
                        },
                        "sys_created_on": {
                            "value": "2014-09-09 23:31:10"
                        }
                    },
                    { <<ITEM-2>> }

]

}

Mindbreeze Configuration

Pagination configuration:

Configuration option	Example value	Description
Query supports Cursor Pagination	False	The paging is not done via cursor.
Query supports Offset Pagination	True	The paging is done via offset.
Results Pointer	/data/GlideRecord_Query/kb_knowledge/_results	“_results” contain the results array.
Next Page Pointer		Not used with offset pagination.
Next Cursor Pointer		Not used with offset pagination.
Paging Variable Name	offset	This variable is defined in the query and is used to inject the new offset that is calculated while crawling.

Examples for Pivoting

“Pivoting” refers to the process of converting a JSON array into a JSON object.

Key/Value Pivoting

Original:

{
  "custom_objects": [
    {
      "key": "a",
      "value": "1"
    },
    {
      "key": "b",
      "value": "2"
    }
  ]
}

Transformed:
{
  "custom_objects": {
    "a": "1",
    "b": "2"
  }

}

Config:

Key Pivoting 1

Original:
{

"custom_objects": [

{

"key": "a",

"value": "1",

"data_type": "int",

"last_updated": "2023-10-27T10:00:00Z",

"version": 1,

"source": "system_default"

{

"key": "b",

"value": "2",

"data_type": "int",

"last_updated": "2023-10-27T11:30:00Z",

"version": 2,

"source": "user_input"

}

]

}

Transformed:
{

"custom_objects": {

"a": {

"key": "a",

"value": "1",

"data_type": "int",

"last_updated": "2023-10-27T10:00:00Z",

"version": 1,

"source": "system_default"

"b": {

"key": "b",

"value": "2",

"data_type": "int",

"last_updated": "2023-10-27T11:30:00Z",

"version": 2,

"source": "user_input"

}

}

Config:

Key Pivoting 2

Original:
{

"custom_objects": [

{

"key": "a"

{

"key": "b"

}

]

}

Transformed:
{

"custom_objects": {

"a": {

"key": "a"

"b": {

"key": "b"

}

}

Config:

Examples for I18N

The connector supports i18n metadata that can be configured in the metadata settings.

Only JSON objects are supported as i18n metadata. All keys and values must be strings, and keys must comply with the ISO 639 standard.

I18N Object metadata

Original:
{

"custom_objects": {

"i18n": {

"en": "Account ID",

"es": "ID de cuenta",

"fr": "ID du compte"

}

}

Config:

I18N Array metadata

If i18n metadata is provided in a JSON array it has to be pivoted first in order to resolve it.

Original:
{

"custom_objects": {

"i18n": [{

"lang": "en",

"val": "Account ID"

}, {

"lang": "es",

"val": "ID de cuenta"

}, {

"lang": "fr",

"val": "ID du compte"

}

]

}

Pivoting Config:

Pivoting Result:

{

"custom_objects": {

"i18n": {

"en": "Account ID",

"es": "ID de cuenta",

"fr": "ID du compte"

}

I18N Config

Configuration
Connector for GraphQL

Introduction

Core Concepts

Data Acquisition

Data Transformation & Extraction

Data Processing Pipeline

Configuration

Quickstart Guide

Step 1: Creation of the index

Step 2: Configuration of the data source

Available settings

Section “Performance Settings” (Advanced Settings)

Section “Diagnostic Settings”

Section “GraphQL Connection”

Section “Query”

Section “Pagination”

Section “Metadata Settings”

Section “ACL Settings”

Examples of cursor paged and offset paged pagination

Cursor Paged Pagination

Query

Result

Mindbreeze Configuration

Offset Paged Pagination

Query

Result

Mindbreeze Configuration

Examples for Pivoting

Key/Value Pivoting

Key Pivoting 1

Key Pivoting 2

Examples for I18N

I18N Object metadata

I18N Array metadata

Download PDF

Download PDF

{{{i18n.refineSearch}}}

Configuration Connector for GraphQL

Introduction

Core Concepts

Data Acquisition

Data Transformation & Extraction

Data Processing Pipeline

Configuration

Quickstart Guide

Step 1: Creation of the index

Step 2: Configuration of the data source

Available settings

Section “Performance Settings” (Advanced Settings)

Section “Diagnostic Settings”

Section “GraphQL Connection”

Section “Query”

Section “Pagination”

Section “Metadata Settings”

Section “ACL Settings”

Examples of cursor paged and offset paged pagination

Cursor Paged Pagination

Query

Result

Mindbreeze Configuration

Offset Paged Pagination

Query

Result

Mindbreeze Configuration

Examples for Pivoting

Key/Value Pivoting

Key Pivoting 1

Key Pivoting 2

Examples for I18N

I18N Object metadata

I18N Array metadata

Download PDF

Download PDF

Configuration
Connector for GraphQL