Tokenize (open)

Returns an array of strings by splitting a string into sub strings.

Syntax

// Core function, no LOADLIB necessary

STRING ARRAY FUNCTION Tokenize(STRING text, STRING separator)

Parameters

STRING text

String to split into tokens

STRING separator

Case-sensitive separator between the tokens

Return value

STRING ARRAY

A string array with the separated tokens

Description

Tokenize splits a string into substrings, divided on the place of the seperator. When found, the separator is removed from the string and the substrings are returned in an array.

If the seperator is not found, the original string in split into its UTF-8 characters.

Examples

// returns {"a","b","c","d","e"}
STRING ARRAY example1 := Tokenize("a-b-c-d-e", "-");

// returns {"A","B","C","D","E"}
STRING ARRAY example2 := Tokenize("A&&B&&C&&D&&E", "&&");

// returns the entire string, because seperator "+" is not found
STRING ARRAY example3 := Tokenize("a-b-c-d-e", "+");

// returns an empty array because string and seperator are the same
STRING ARRAY example4 := Tokenize("a/b/c/d/e", "a/b/c/d/e");

// returns the entire string, because seperator "a" is not found
STRING ARRAY example5 := Tokenize("A-B-C-D-E-F", "a");

// Splits the string into its UTF-8 characters
STRING ARRAY example5 := Tokenize("reëel", "");
// Results in [ "r", "e", "ë", "e", "l" ]