9.1.13-h3 Known Issues

[Consuelo Dular]

Idon't know if this is your exact issue, but it seems as if we are tracking something internally. PAN-194219, the software packet buffers are depleting erroneously during HTTP/2 inspection only post 9.1.14 upgrade, and subsequently not decrypting.

It appears we are seeing upgrade requests from servers through a TLS session, TLS1.1 requests to 1.2 or in some cases 1.3, this is not allowed per RFC. The firewall reads the request as a header frame, and then tries to extract sequential data from the incoming packets of the session. Since the session isn't terminated, the bogus packet(s) aren't dropped, session depletion happens.

Indeed, it is now documented in the known issues. If your security requirements allow for HTTP1.1 / strip ALPN workaround that will suffice. If not, we are still researching the TLS upgrade requests and out of order queue.

Technically, Sphinx is a standalone software package providesfast and relevant full-text search functionality to client applications.It was specially designed to integrate well with SQL databases storingthe data, and to be easily accessed by scripting languages. However, Sphinxdoes not depend on nor require any specific database to function.

Applications can access Sphinx search daemon (searchd) using any ofthe three different access methods: a) via Sphinx own implementation of MySQL network protocol (using a small SQL subset called SphinxQL, this is recommendedway), b) via native search API (SphinxAPI) or c) via MySQL server with apluggable storage engine (SphinxSE).

Official native SphinxAPI implementations for PHP, Perl, Python, Ruby and Java are included within the distribution package. API is very lightweightso porting it to a new language is known to take a few hours or days.Third party API ports and plugins exist for Perl, C#, Haskell,Ruby-on-Rails, and possibly other languages and frameworks.

Starting from version 1.10-beta, Sphinx supports two different indexingbackends: "disk" index backend, and "realtime" (RT) index backend.Disk indexes support online full-text index rebuilds, but online updatescan only be done on non-text (attribute) data. RT indexes additionallyallow for online full-text index updates. Previous versions onlysupported disk indexes.

Data can be loaded into disk indexes using a so-called data source.Built-in sources can fetch data directly from MySQL, PostgreSQL, MSSQL, ODBCcompliant database (Oracle, etc) or a pipe in TSV or a custom XML format.Adding new data sources drivers (eg. to natively support other DBMSes)is designed to be as easy as possible. RT indexes, as of 1.10-beta,can only be populated using SphinxQL.

This program is free software; you can redistribute it and/or modifyit under the terms of the GNU General Public License as published bythe Free Software Foundation; either version 2 of the License,or (at your option) any later version. See COPYING file for details.

This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITYor FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License formore details.

Sphinx development was started back in 2001, because I didn't manageto find an acceptable search solution (for a database driven Web site)which would meet my requirements. Actually, each and every important aspect was a problem:

On Windows, you will need Microsoft Visual C/C++ Studio .NET 2005 or above.Other compilers/environments will probably work as well, but for thetime being, you will have to build makefile (or other environmentspecific project files) manually.

Extract everything from the distribution tarball (haven't you already?) and go to the sphinx subdirectory. (We are using version 2.2.11-dev here for the sake of example only; be sure to change this to a specific version you're using.)

this means that some compile-time type size check failed.The most probable reason is that off_t type is less than 64-biton your system. As a quick hack, you can edit sphinx.h and replace off_twith DWORD in a typedef for SphOffset_t, but note that this will prohibityou from using full-text indexes larger than 2 GB. Even if the hack helps,please report such issues, providing the exact error message andcompiler/OS details, so I could properly fix them in next releases.

Installing Sphinx on a Windows server is often easier than installing on a Linux environment;unless you are preparing code patches, you can use the pre-compiled binary files from the Downloadsarea on the website.

Extract everything from the .zip file you have downloaded - sphinx-2.2.11-dev-win32.zip, or sphinx-2.2.11-dev-win32-pgsql.zip if you need PostgresSQL support as well. (We are using version 2.2.11-dev here for the sake of example only; be sure to change this to a specific version you're using.) You can use Windows Explorer in Windows XP and up to extract the files, or a freeware package like 7Zip to open the archive.

For the remainder of this guide, we will assume that the folders are unzipped into C:\Sphinx, such that searchd.exe can be found in C:\Sphinx\bin\searchd.exe. If you decide to use any different location for the folders or configuration file, please change it accordingly.

The searchd service will now be listed in the Services panel within the Management Console, available from Administrative Tools. It will not have been started, as you will need to configure it and build your indexes with indexer before starting the service. A guide to do this can be found under Quick tour.

During the next steps of the install (which involve running indexer pretty much as you would on Linux) you may find that you get an error relating to libmysql.dll not being found. If you have MySQL installed, you should find a copy of this library in your Windows directory, or sometimes in Windows\System32, or failing that in the MySQL core directories. If you do receive an error please copy libmysql.dll into the bin directory.

32-bit document IDs are now deprecated. Our binary releasesare now all built with 64-bit IDs by default. Note that they can stillload older indexes with 32-bit IDs, but that support will eventually beremoved. In fact, that was deprecated awhile ago, but now we just want tomake it clear: we don't see any sense in trying to save your server's RAMthis way.

dict=crc is now deprecated. It has a bunch of limitations,the most important ones being keyword collisions, and no (good) wildcardmatching support. You can read more about those limitations in ourdocumentation.

charset_type=sbcs is now deprecated, we're slowly switchingto UTF-only. Even if your database is SBCS (likely for legacy reasonstoo, eh?), this should be absolutely trivial to workaround, just add apre-query to fetch your data in UTF-8 and you're all set. Also, in fact,our current UTF-8 tokenizer is even faster than the SBCS one.

enable_star is deprecated now. Previous default mode wasenable_star=0 which was due to compatibility with a very old Sphinxversion. Such implicit star search isn't very intuitive. So, we've decidedto eventually remove it and have marked it as deprecated just recently. We planto totally remove this configuration key in the 2.2.X branch.

str2ordinal attributes are deprecated. This feature allowsyou to perform sorting by a string. But it's also possible to do this withordinary string attributes, which is much easier to use. str2ordinal onlycovers a small part of this functionality and is not needed now.

str2wordcount attributes are deprecated.index_field_lengths=1will create an integer attribute with field length set automatically and werecommend to use this configuration key when you need to store fieldlengths. Also, index_field_lengths=1 allows you to use new ranking formulaslike BM25F().

hit_format is deprecated. This is a hidden configurationkey - it's not mentioned in our documentation. But, it's there and it'spossible that someone may use it. And now we're urging you: don't use it.The default value is 'inline' and it's a new standard. 'plain' hit_formatis obsolete and will be removed in the near future.

None of the different querying methods are deprecated, but as ofversion 2.2.1-beta, SphinxQL is the most advanced method. We plan toremove SphinxAPI and Sphinx SE someday so it would be a good idea tostart using SphinxQL.

Sample configuration file, sphinx.conf.dist, which has all the options documented, is created by configure. Copy and edit that sample file to make your own configuration: (assuming Sphinx is installed into /usr/local/sphinx/)

The data to be indexed can generally come from very differentsources: SQL databases, plain text files, HTML files, mailboxes,and so on. From Sphinx point of view, the data it indexes is aset of structured documents, each of which has thesame set of fields and attributes.This is similar to SQL, where each row would correspond to a document,and each column to either a field or an attribute.

Depending on what source Sphinx should get the data from,different code is required to fetch the data and prepare it for indexing.This code is called data source driver (or simplydriver or data source for brevity).

There can be as many sources per index as necessary. They will besequentially processed in the very same order which was specified inindex definition. All the documents coming from those sourceswill be merged as if they were coming from a single source.

Fields are named, and you can limit your searches to a singlefield (eg. search through "title" only) or a subset of fields(eg. to "title" and "abstract" only). Sphinx index format generallysupports up to 256 fields. However, up to version 2.0.1-beta indexeswere forcibly limited by 32 fields, because of certain complicationsin the matching engine. Full support for up to 256 fields was addedin version 2.0.2-beta.

Note that the original contents of the fields are not storedin the Sphinx index. The text that you send to Sphinx gets processed,and a full-text index (a special data structure that enables quicksearches for a keyword) gets built from that text. But the originaltext contents are then simply discarded. Sphinx assumes that you storethose contents elsewhere anyway.

3a8082e126