PivotalĀ® GPText 3.3.1 Release Notes

This document contains release information for Pivotal GPText 3.3.1

Released: September 2019

About Pivotal GPText

Pivotal GPText joins the Greenplum Database massively parallel-processing database server with Apache SolrCloud enterprise search and the Apache MADlib Analytics Library to provide large-scale analytics processing and business decision support. GPText includes free text search as well as support for text analysis.

GPText includes the following features:

  • The GPText database schema provides in-database access to Apache Solr indexing and searching
  • Build indexes with database data or external documents and search with the GPText API
  • Custom tokenizers for international text and social media text
  • A Universal Query Processor that accepts queries with mixed syntax from supported Solr query processors
  • Faceted search results
  • Term highlighting in results
  • Natural language processing, including part-of-speech tagging and named entity extraction
  • Greater emphasis on high availability

The GPText management utility suite includes command-line utilities to perform the following tasks:

  • Start, stop, and monitor ZooKeeper and GPText nodes
  • Configure GPText nodes and indexes
  • Add and delete replicas for index shards
  • Back up and restore GPText indexes
  • Recover a GPText node
  • Expand the GPText cluster by adding GPText nodes


Installing GPText also installs Apache Solr Cloud and, optionally, Apache ZooKeeper.

Following are GPText installation prerequisites.

  • GPText runs on Red Hat Enterprise Linux 5.x, 6.x, and 7.x.
  • GPText runs on Greenplum Database version 4.3.6 or higher, Greenplum Database 5, or Greenplum Database 6. Greenplum Database 6 requires at least GPText 3.3.1.
  • Install Java JRE 1.8.x and add the bin directory to the PATH on all hosts in the cluster. GPText is tested with Oracle Java 1.8 and OpenJDK 1.8.
  • Install and configure your Greenplum Database system before you install GPText. See the Pivotal Greenplum Database Installation Guide at https://gpdb.docs.pivotal.io.
  • Ensure that nc (netcat) is installed on all Greenplum cluster hosts (sudo yum install nc).
  • Installing lsof on all cluster hosts is recommended (sudo yum install lsof).
  • GPText cannot be installed onto a shared NFS mount.
  • GPText nodes can be installed on the Greenplum Database cluster hosts alongside the Greenplum segments or on additional, non-database hosts accessible on the Greenplum cluster network. All hosts participating in the GPText system must have the same operating system and configuration and have passwordless-ssh access for the gpadmin user. See the Pivotal Greenplum Database Installation Guide for instructions to configure hosts.
  • If you plan to place GPText nodes on the Greenplum Database segment hosts, ensure that you reserve memory for GPText use when you configure Greenplum Database. To determine the memory to set aside for GPText, multiply the number of GPText nodes to create on each Greenplum segment host by the JVM maximum size. Subtract this memory from the physical RAM when calculating the value for the Greenplum Database gp_vmem_protect_limit server configuration parameter. See the Greenplum Database server configuration parameter gp_vmem_protect_limit in the Greenplum Database Reference Guide for recommended memory calculation formulas or visit the GPDB Virtual Memory Calculator web site.
  • Apache Solr requires a ZooKeeper cluster with at minimum three nodes (five nodes recommended). You can install a “binding” ZooKeeper cluster with GPText on the Greenplum cluster hosts, or you can use an existing ZooKeeper cluster. When deployed alongside Greenplum Database segments, ZooKeeper performance can be affected under heavy database load. For best performance, install a ZooKeeper cluster on separate hosts with network connectivity to the Greenplum network.

New Features and Enhancements in GPText 3.3.1

Using GPText 3.3 with Greenplum Database 6.0

GPText 3.3.1 can be installed on a Greenplum Database 6 system with Java 8.

A GPText binary distribution has been added to Pivotal Network for Red Hat 7/CentOS 7 with Greenplum Database 6.

Note: The “Greenplum Text 3.3.1 for RHEL 7” distribution is for Greenplum Database 6.x only. Download the RHEL 6 distribution if you are installing GPText into a Greenplum Database 5.x system.

Following are differences using GPText with Greenplum Database 6 than with earlier Greenplum Database releases:

  • The custom_variable_classes server configuration parameter has been removed in Greenplum Database 6. With earlier Greenplum Database versions, it was necessary to add 'gptext' to this parameter in order to set GPText configuration parameters. Greenplum Database 6 allows you to set configuration parameters in a database session without declaring a variable class.

  • In Greenplum Database 4 and 5, the default output format for the binary data type bytea is the PostgreSQL escape format, a sequence of ASCII characters with escape sequences where bytes cannot be represented with ASCII. In Greenplum Database 6, the default output format is the hex format, which represents each byte with hexadecimal digits. In Greenplum Database 5, the hex output format can be specified by setting the bytea_output configuration parameter to hex. To produce the same output in Greenplum Database 4, 5, and 6, you can set the bytea_output configuration parameter to escape.

Custom Configuration Directory

A new optional installation parameter, GPTEXT_CUSTOM_CONFIG_DIR, can be set in the gptext_install_config file to specify a directory to store custom configuration files.

By default, GPText saves custom configuration files under the $GPTEXTHOME/share/ directory on each Solr host, for example $GPTEXTHOME/share/external_.

To specify a different directory to store external configuration files, before you run the GPText installer, uncomment the GPTEXT_CUSTOM_CONFIG_DIR parameter in the gptext_install_config file and specify the full path to the directory. For example:


The gpadmin user must have the OS permissions required to create the directory.

If the parameter is set, the GPText installer will create the custom configuration directory on every Solr host. Configuration files you upload using the gptext-external upload command will be stored under this directory on every Solr host to allow Solr to access the external document source from every host. For example if the GPTEXT_CUSTOM_CONFIG_DIR parameter is set to /home/gpadmin/config_dir when you install GPText, an s3 configuration with the name s3_conf will be saved in the directory /home/gpadmin/config_dir/external_source/s3/s3_conf on each host.

New Features and Enhancements in GPText 3.2.0

The GPText 3.2.0 release provides the following features and enhancements.


GPText 3.2.0 enables lemmatizing terms in GPText indexes. You can define Solr analysis chains that include the Apache OpenNLP parts-of-speech filter and the new GPText WordNetLemmatizer filter, which replaces terms with the root form of the term. The WordNetLemmatizer filter uses a lexical database from the Princeton University WordNet® project to determine the root form.

GPText Configuration Files Location

GPText now saves configuration files gptext.conf, gptxtenvs.conf, and zookeeper.conf only in the Greenplum Database master and standby master directories. The gptext.conf file is no longer saved in each segment data directory.

Flexible Sharding

By default, GPText creates one Solr index shard for each Greenplum Database primary segment. You can now specify a smaller number of shards by setting the gptext.idx_num_shards parameter to the number of shards you want before you create the index. This works for both regular GPText indexes and external indexes.

When gptext.idx_num_shards is set to the default (0), GPText configures the index to use the Solr implicit router, with one shard per Greenplum Database segment. When the gptext.idx_num_shards parameter is changed to the number of shards desired, GPText creates the index using the Solr compositeId router to route documents to shards. The compositeId router does not support duplicate IDs, so if you set the if_check_id_uniqueness argument to false when you call the gptext.create_index() function the implicit router is used, and the index will have one shard per Greenplum Database segment.

The content_id column is removed from the output of the gptext.index_status() and gptext.index_summary() functions, since Greenplum Database segments are not always associated with a single index shard.

See Specifying the Number of Shards for more information about this feature.

gptext-recover Utility

When using the -f (--force) option, the gptext-recover utility now verifies that there are no indexes in a red state before proceeding. If any index is down, the utility exits.

ZooKeeper Upgrade

Apache ZooKeeper included with GPText 3.2.0 has been upgraded to version 3.4.11. This ZooKeeper release includes bug fixes that resolve an inconsistent cluster issue with GPText(MPP-29742).

New Features and Enhancements in GPText 3.1.0

The GPText 3.1.0 release provides the following features and enhancements.

Improvements to aid in developing and testing analyzer chains

  • The new gptext.list_field_types() function lists the field types defined in the managed-schema configuration file for an index.

  • The new gptext.get_field_type() function displays the index and query analyzer chains for a field type in JSON format.

  • The new gptext.analyzer() function shows the index or query analyzer chain output for a given field type and input text. This function is useful for testing and debugging analyzer chains interactively without modifying the index.

Part-of-speech tagging and named entity recognition

  • GPText includes OpenNLP libraries and analyzer classes to classify indexed terms’ parts-of-speech (POS), and to recognize named entities, such as the names of persons, locations, and organizations (NER). GPText saves NER terms in the field’s terms vector, prepended with a code to identify the type of entity recognized. This allows searching documents by entity type.

  • The new gptext.ner_terms() function lists NER-tagged terms for documents that match a query.

  • GPText includes the OpenNLP models for the English language. You can download models for other languages from the OpenNLP web site and use them with GPText.

Other enhancements and fixes

  • The first argument of the gptext.terms() function, an anytable data type, has been made optional.

  • Fixed an error where the gptext.partition_status() function displayed partition information for an index after it was dropped.

Apache Solr updated to Solr version 7.3

GPText 3.1.0 includes Apache Solr 7.3. See the following release documents for information about the Solr 7.3 release.

Following are GPText changes and Solr usage notes related to the Solr 7.3 upgrade.

  • GPText server-side components are rebuilt and tested with the new Solr JAR files.

  • The managed-schema, solrconfig.xml and other collection configuration files are updated.

  • The top-level <highlighting> element in solrconfig.xml is now officially deprecated in favor of the equivalent <searchComponent> syntax. This element has been out of use in default Solr installations for several releases already.

  • The legacyCloud parameter now defaults to false. If an entry for a replica does not exist in state.json, that replica will not be registered. This may affect users who bring up replicas and they are automatically registered as a part of a shard. It is possible to revert to the old behavior by setting the property legacyCloud=true in the cluster properties by running the following command in the GPText installation directory:

    $ ./server/scripts/cloud-scripts/zkcli.sh -zkhost -cmd clusterprop -name legacyCloud -val true
  • With earlier Solr releases, if you drop an index while a Solr node with a replica of the index is down, when the down node comes back on-line, the index comes back and cannot be deleted. Solr 7 fixes this bug. The GPText workaround for this bug is removed.

  • PointFields are default numeric types. Solr has implemented *PointField types across the board, to replace Trie* based numeric fields. All Trie* fields are now considered deprecated, and will be removed in Solr 8. If you are using Trie* fields in your schema, you should consider moving to PointFields as soon as feasible. Changing to the new PointField types will require you to re-index your data.

  • The following spatial-related fields have been deprecated:
    GeoHashField FieldType
    Use one of these field types instead:

  • To improve parameter consistency in the Collections API, the parameter names fromNode for the MOVEREPLICA command and source, and target for the REPLACENODE command have been deprecated and replaced with sourceNode and targetNode instead. The old names will continue to work for backwards compatibility, but they will be removed in Solr 8.

  • The replica core name has changed from <collection_name>_shard#_replica# to <collection_name>_shard#_replica_<node_type>#. For example, demo.wikipedia.articles_shard0_replica1 becomes demo.wikipedia.articles_shard0_replica_n1.

New Features and Enhancements in GPText 3.0.0

  • GPText 3.0.0 allows adding documents stored in Amazon Web Services S3 buckets to a GPText external index. This enhancement includes changes to enable uploading AWS credentials to ZooKeeper and support for the s3 document source type for the gptext.external_login(), gptext.external_logout(), gptext.index_external(), and gptext.index_external_dir() GPText functions.

  • The gptext-state utility with the --index (-i) option now includes the date and time the GPText index was last modified.

New Features and Enhancements in GPText 2.4.0

  • GPText 2.4.0 allows adding documents stored in an authenticated FTP server to a GPText external index. This enhancement includes changes to add support for the ftp type to the gptext.external upload command-line utility and the gptext.external_login(), gptext.external_logout(), gptext.index_external(), and gptext.index_external_dir() GPText functions.

New Features and Enhancements in GPText 2.3.1

  • The gptext-backup command-line utility can now back up GPText indexes to local GPText cluster storage as well as a directory on a shared drive. For local backups, backup metadata and the index configuration files are backed up to the Greenplum Database master data directory and index shards are backed up in the segment data directories on each host.
  • The gptext-backup utility has a new option to back up just the index configuration files from ZooKeeper, with no index data.
  • The gptext-restore uility is updated to restore backups created on local cluster storage.
  • The gptext-restore utility has a new option to restore only the configuration files from a backup. This option loads the configuration files into ZooKeeper and creates an empty GPText index.

New Features and Enhancements in GPText 2.3.0

Revised gptext-config Utility Syntax

The gptext-config command-line utility was revised to have a more user-friendly syntax.

A new list subcommand was added to gptext-config you can use to list all of the configuration files for a specified GPText index.

$ gptext-config list -i <index-name>

Index Documents in a Hadoop File System (hdfs) Document Source

GPText 2.3.0 enables you to add documents stored in a hdfs system to a GPText external index.

  • The new gptext-external command-line utility uploads Hadoop configuration and authentication files to a named configuration in ZooKeeper. The utility has subcommands upload, list, and delete to manage the configurations you have uploaded.
  • The new gptext.external_login() function logs in to the hdfs system using the named configuration you have uploaded. You can log in to only one external document source at a time.
  • Use URLs of the form hdfs://<url> with the gptext.index() and gptext.index_external() functions to add documents to a GPText external index.
  • Use the new gptext.index_external_dir() function to add all documents in an hdfs directory to a GPText external index.
  • Log out of the hdfs external document source with the new gptext.external_logout() function.

See Authenticating with an External Document Source for steps to enable access to an hdfs document source.

Known Issues

See the Apache Jira for known issues in Apache Solr.

Following are known issues in GPText. Workarounds are provided when available.

gptext-migrator Fails to Install the GPText Shared Libary After Greenplum Database Upgrade

When you upgrade Greenplum Database and then migrate your existing GPText installation to the new Greenplum Database installation, the gptext-migrator utility in some cases fails to install the GPText UDF library to the new Greenplum Database $GPHOME/lib/postgresql directory. gptext-migrator outputs the message [INFO]:-UDF libraries are installed in $GPTXTHOME/lib, don't need to migrate. The message is correct only if you installed GPText binaries to a shared drive following the Optional Two-Part GPText Installation installation method.


  1. Create a host file containing a list of all Greenplum Database hosts.

  2. Make sure the gpadmin user has write permission in the $GPHOME/lib/postgresql directory of the new Greenplum Database installation directory on every Greenplum Database host.

  3. Use the gpscp utility to copy the GPText UDF library from the old Greenplum Database installation to the new Greenplum Database installation.

    $ gpscp -f hostfile /usr/local/greenplum-db-<old-version>/lib/postgresql/gptext*.so \

See Upgrading GPText for more inforaation.

Wildcards in GPText Search Options

Solr does not return all fields when the fl Solr search option contains a wildcard that matches field names. For example, given a table with columns contenta and contentb, specifying fl=contenta,contentb,(sum,1,1) correctly returns three fields. Specifying fl=cont*,sum(1,1) correctly returns contenta and contentb, but omits the pseudo-field sum(1,1).

Specifying a wildcard to match all fields (fl=*,sum(1,1)) also omits the pseudo-field.

Index Load Failure After Configuration File Error

If Solr fails to load an index because of a configuration file error, and then the index is dropped without first correcting the configuration file error, the index cannot be recreated until GPText is restarted. This can happen if you edit managed-schema or solrconfig.xml and introduce an XML syntax error or a typo in configuration values.


  1. When an index fails to load, check the Solr log to find the cause.
  2. If the cause is a configuration file error, such as invalid XML, use the gptext-config utility to edit the file and fix the error. Dropping the index without first correcting the error is not recommended.
  3. If you have dropped an index that failed to load without first correcting the cause of the failure, you must restart GPText before you can recreate the index. Run gptext-start -r to restart GPText.

Startup Failure with Large Numbers of Indexes

When there is a large number of Solr cores, Solr Cloud can fail to restart successfully, with error messages indicating failure to elect leaders for shards. This is a known Solr issue; see https://issues.apache.org/jira/browse/SOLR-5990 in the Apache Solr Jira for an example. Because of this issue, it is recommended to avoid designing GPText applications that create large numbers of indexes, shards, and replicas. The number of cores you can create before you observe this behavior is hardware dependent, so you should test to determine your system’s limits. You can create and successfully operate a larger numbers of indexes than can be restarted successfully later, so be sure to test restarting GPText to determine a practical limit.

Setting GPText Configuration Parameters Without First Setting custom_variable_classes

In Greenplum Database versions before Greenplum Database 6, if the custom_variable_classes Greenplum Database server configuration parameter does not include the value “gptext”, attempting to set a GPText configuration parameter returns an error message, for example:

mydb-# set gptext.replication_factor = 4;
WARNING:  Please logon again to make GUC setting take effect. (GucValue.h:301)
WARNING:  Please logon again to make GUC setting take effect. (GucValue.h:301)
ERROR: unrecognized configuration parameter "gptext.replication_factor"

In GPText 2.0, in addition to the error message, the value of the configuration parameter persisted in ZooKeeper is zero, replacing the previous value of the parameter.

 mydb-# show gptext.replication_factor;

Beginning with GPText 2.1, the error message is still generated, however the value saved in ZooKeeper is the value specified in the set command, 4 in the preceding example.

To prevent the error message, before setting any GPText configuration parameters, use the gpconfig command-line utility to set the custom_variable_classes configuration parameter:

$ gpconfig -c custom_variable_classes -v 'gptext'

In Greenplum Database 6.0, the custom_variable_classes configuration parameter is removed and custom parameters can be set without errors.