What happens at a Lucene/Solr Hackday

Charlie HullSeptember 23, 2019

For the last few years I’ve run free Lucene Hackdays around the same time of year as the largest conference in our open source search sector, Activate (previously known as Lucene Revolution and run by Lucidworks). The idea is that if people are in town for a conference anyway, they might like to get together with other Lucene/Solr enthusiasts and work together on some problems. This year, as Activate was in Washington DC, OSC organised a hackday kindly hosted by the American Geophysical Union at their new (and highly eco-friendly!) new headquarters building. Around 20 people attended including several Lucene/Solr committers and four of us from the OSC team.

I’ve listed below some of what we worked on, with links to related JIRA issues and other supporting material. On the day we used Relevance Slack to communicate (check out the #lucene-hackday channel) - if you want more information this is the best place to ask. Thanks to everyone who came and hacked, it was a lot of fun! Note that we didn’t have time to work on everything but I’ve listed all the ideas anyway in case someone else is inspired to continue. Also, the following is based on my notes taken during the day and the shared GoogleDoc we used, so do please forgive any inaccuracies or mistakes.

Things we worked on

1. Andy Webb suggested two ideas:

a. add support for BooleanSimilarity to Solr, for situations when you don’t want document/term frequency to influence scores (ref LUCENE-5867): “I’ve raised SOLR-13751 which links to https://github.com/apache/lucene-solr/pull/867 - just running ant test now. It’d be great to get further feedback/comments on this - this is my first Solr PR!”

b. Solr admin UI tweaks such as alphabetical ordering of collections in the drop-down lists. Bob Bunch took this on: ”Here are a couple of patches (master branch; I’m not set up to contribute yet). This fixes:

  • Collections page, order of collections
  • Collections drop-list, order of collections
  • Cores drop-list, order of cores“
diff --git a/solr/webapp/web/index.html b/solr/webapp/web/index.html
index 7fe1381763c..11157402269 100644
--- a/solr/webapp/web/index.html
+++ b/solr/webapp/web/index.html
@@ -182,7 +182,7 @@ limitations under the License.
                         ng-model="currentCollection"
                         chosen
                         ng-change="showCollection(currentCollection)"
-                        ng-options="collection.name for collection in aliases_and_collections"</select
+                        ng-options="collection.name for collection in aliases_and_collections | orderBy: 'name'"</select
               </div
               <p id="has-no-collections" ng-show="collections.length==0"<a href="#/~collections"
                 No collections available
@@ -208,7 +208,7 @@ limitations under the License.
                         ng-model="currentCore"
                         chosen
                         ng-change="showCore(currentCore)"
-                        ng-options="core.name for core in cores"</select
+                        ng-options="core.name for core in cores | orderBy: 'name'"</select
               </div
               <p id="has-no-cores" ng-show="cores.length==0"<a href="#/~cores"
                 No cores available
diff --git a/solr/webapp/web/partials/collections.html b/solr/webapp/web/partials/collections.html
index 198030c744f..8c9cebc81c7 100644
--- a/solr/webapp/web/partials/collections.html
+++ b/solr/webapp/web/partials/collections.html
@@ -88,7 +88,7 @@ limitations under the License.
           <input type="text" name="alias" ng-model="aliasToCreate" id="alias"</p
 
         <p class="clearfix"<label for="aliasCollections"Collections:</label
-          <select multiple id="aliasCollections" ng-model="aliasCollections" ng-options="collection.name for collection in collections" class="other"
+          <select multiple id="aliasCollections" ng-model="aliasCollections" ng-options="collection.name for collection in collections | orderBy: 'name'" class="other"
           </select</p
@@ -382,7 +382,7 @@ limitations under the License.
   <div id="navigation" class="requires-core clearfix"
     <button id="add" class="action" ng-click="showAddCollection()"<spanAdd Collection</span</button
     <ul
-      <li ng-repeat="c in collections" ng-class="{current: collection.name == c.name && collection.type === 'collection'}"<a href="#~collections/"</a</li
+      <li ng-repeat="c in collections | orderBy: 'name'" ng-class="{current: collection.name == c.name && collection.type === 'collection'}"<a href="#~collections/"</a</li
     </ul
     <hr/
     <button id="create-alias" class="action requires-core" ng-click="toggleCreateAlias()" ng-disabled="collections.length == 0"<spanCreate Alias</span</button>
diff --git a/solr/webapp/web/partials/cores.html b/solr/webapp/web/partials/cores.html
index 1615769564f..4dccc313449 100644
--- a/solr/webapp/web/partials/cores.html
+++ b/solr/webapp/web/partials/cores.html
@@ -217,7 +217,7 @@ limitations under the License.
 
   <div id="navigation" class="requires-core clearfix" ng-show="hasCores"
     <ul
-      <li ng-repeat="c in cores" ng-class="{current: core.name == c.name}"<a href="#~cores/"</a</li
+      <li ng-repeat="c in cores | orderBy: 'name'" ng-class="{current: core.name == c.name}"<a href="#~cores/"</a</li
     </ul
   </div>

2. Bob Bunch also suggested:

a. To work more on monitoring JVM full garbage collects and shard/core status at scale. (we didn’t get around to this)

b. Exposure testing - we resolved a major bug in Gus’ nifty cloud.sh script - SOLR-11492. The -c option (in conjunction with a restart) caused a very bad scenario, even on *nix platforms! (Bob’s Macbook got so warm it started restarting itself and one of the USB ports stopped working for a while!)

3. Anshum Gupta suggested:

a. Drive forward Mark Millers’ Gradle effort (a huge project to change Lucene/Solr’s build system - Mark has put months of work into this and it promises to shorten build times considerably). A group of people worked on testing the Gradle build system to identify any issues and to become familiar with the new system.

b. SOLR-13527 (we didn’t get around to this)

c. Get GitHub to run ‘ant precommit’ as an action for all pull requests - SOLR-13753 - this has been committed to Github here.

4. Bertrand Rigaldies suggested:

Return the offset of snippets in Solr’s highlighter. A large team worked on this. We began with setting up a Highlighter formatter using the original highlighter in Solr. An example showing this approach is here.

screen2.png

David Smiley began implementing an “extended” mode to the unified highlighter that offers similar functionality - SOLR-1954

The innovation in the “extended” unified highlighter that David prototyped was in returning the passages’ metadata out-of-band of, or in addition to, the formatted snippets (See screenshot below).

screen.png

5. Eric Pugh and Peter Fries worked on upgrading the Clade taxonomy tool from Python2 to Python 3. (Clade was a project created by Flax to use Solr to classify data into a taxonomy).

6. Joe Ye worked on the Marple index inspector tool to allow it to read older versions of Lucene indexes. Joe had worked on this at a previous Hackday but had lost all his work when his laptop was subsequently stolen!

Things we didn’t get a chance to work on

The following suggestions were made but we didn’t have time to look at them. If you have a particular interest in an issue, or can help, you could either contribute to the JIRA ticket or perhaps contact the author directly via Relevance Slack.

7. Andy Hind suggested:

a. A fix for MLT using corpus size - SOLR-13752

b. Identify known vulnerabilities in dependencies while building Solr using Gradle.

8. Gus Heck suggested: Hardcoded timeouts - SOLR-13457

9. Michael Gibney suggested: Facet caching/singlePass multi-domain facet collection (relatedness) - SOLR-13132

10. Shyamsunder Mutcha suggested: JSON Facet API Java objects - “I don’t see a feature to write JSON facet queries as Query objects.”

Summary

As usual, we had considerably more ideas than time but everyone worked hard on what we selected - thankyou all for your efforts. Perhaps more importantly everyone got a chance to meet or catch up with others in the Lucene/Solr community, discuss problems and potential solutions and give some insights into how they use Lucene/Solr.

We’ll be running more Hackdays - watch this space! If you’re interested in supporting or running a Hackday and empowering your search team, get in touch.


More blog articles:


We empower great search teams!

We help teams that use Solr and Elasticsearch become more capable through consulting and training.

Services Training