SAIC has been a leading developer of intelligence analysis software for decades. In fact, their Pathfinder application has seen continual usage at a variety of agencies since the early 90s. It has been periodically updated with newer and better technologies as they emerged, and OpenSource Connections was an integral part of one of these latest efforts. Over the course of fourteen months, across two project teams, we helped update both the front and back ends of the application.
The front-end team improved the search functionality by developing a .Net / Service Oriented Architecture-based user interface. To ease the transition to the new interface, old views were ported, and new views were built to maximize the new capabilities, including toolbars, applets, plug-ins etc. The updated interface decreased the technical-know-how required of the analysts performing the queries, thereby allowing them to focus on the information related to the search rather than on the technical parameters required to perform the search.
For full-text searching, the open source Lucene project has singled itself out as an invaluable tool. Its speed and accuracy are second to none. It is also documented to work well with a variety of programming languages, which grants it greater future flexibility. SAIC had previously integrated it into Pathfinder by enhancing its query language and plugging it into their data feeds, but the new implementation needed to be tied directly to the central Oracle database.
To accomplish this, the back-end team indexed the text with Lucene, but only stored the record identity in the index. Searching would take place completely within Lucene, but whenever the actual text was called for it would be retrieved from Oracle. Since the database was already integrated into a great number of data feeds, this instantly added them to Pathfinder and allowed it to leverage the data access security protocols already in place. This solution not only saves the client money in its lack of licensing fees but offers better forward compatibility.
Finally, we deployed Xen virtualization software for use by both teams as a development and testing platform. This saved additional dollars by effectively turning two servers into twenty and allowed the teams to quickly create whatever server environment their task called for.