Integrating FredHopper into a hybris Marketplace

In this blog article we discuss how Fredhopper, an advanced site search and merchandising product, can be integrated into the hybris eCommerce suite not only to search for products, but to create cross selling and campaigns as well. In the used scenario hybris is the foundation of a marketplace with a few million products from thousands of vendors.

The standard edition of hybris uses the open source Lucene-based Solr enterprise search server. But for this project Fredhopperwill be be used instead. The article explains how Fredhopper can be provided with all the required data from the hybris system.

Fredhopper is a commercial Online Marketing Suite with a focus on

  • On-site Search,
  • On-site Targeting, and
  • Predictive Targeting.

Fredhopper also delivers the navigational structure for our project and all other navigation-related items like breadcrumbs, etc.

A marketplace consists not only of one huge catalog, which lists all products from all marketplace vendors. It also contains individual vendor shops, where the vendors can have their own catalog structure, their own navigation and their own search. All ths is also handled by Fredhopper.

A marketplace with two vendors (dealers).

For example, consider the main catalog in the figure above. It can contain an audio player, which is offered by different vendors. Therefore, the audio player must occur in the main catalog and also in the vendor catalogs. In the context of a vendor shop an audio player should be displayed in that context only, while the main catalog of the marketplace will display the product offered by multiple vendors at different prices where the offer with the lowest price will be displayed in the standard result list (either via search or navigation).

Why Fredhopper?

First, Fredhopper can do much more than searching. It can also be used to create campaigns for upselling, has a recommendation feature and offers sophisticated auto-correction for spelling, etc. in a lot of languages, including those used in the marketplace.

Illustrations of Fredhopper-based offers and recommendations.

Secondly, changes in its behavior can be easily configured by marketing users without a need for technical support or code changes. This flexibility was the reason why we switched from the provided Solr product to Fredhopper.

Integrating Fredhopper into hybris

Before I discuss the details of the integration, let’s take a look at the hardware and infrastructure we had to integrate. As the following diagram shows, Fredhopper is deployed on different servers where each server has a specific function (Navigation/search and quick search).

Distributed Fredhopper deployment for Navigation/search and quick search.

To use the Fredhopper features like navigation services, search services, campaigns, etc. we developed a suite of components for the hybris WCMS (Web Content Management System):

  • Components used to display categories
  • Search components
  • Navigation components
  • Recommendation components

All those components are “regular” hybris components that query the Fredhopper index.

The Fredhopper servers receive requests from and deliver responses to the hybris servers and not directly to the user. The hybris servers use web services to communicate with the Fredhopper servers. Fredhopper provides a Java-based web service library to send queries. We wrote our own wrapper to encapsulate all the technical details and created an API where all project-based queries are encapsulated.

The following snippet shows, how to use this wrapper (NavigationHelper) to communicate with Fredhopper. It’s an excerpt of our WCMS search component:

@RequestMapping("/ProductsSearchComponentController.html")
public String showSearchNavigation(final ModelMap model, final HttpServletRequest request) throws WebException {

    // ProductsListComponentModel is a hybris component
	ProductsListComponentModel component = (ProductsListComponentModel) request
			.getAttribute(FrontendConstants.CMSConstants.COMPONENT_KEY);
	long startTime = System.currentTimeMillis();
	NavigationState navigationState = this.navigationHelper.getOrCreateNavigationState(request);

	// --- Calls a Fredhopper web service through our navigationHelper wrapper
	// NavigationResult is a DTO, result from Fredhopper
	NavigationResult navigationResult = this.navigationHelper
			.getOrCreateNavigationResult(request, navigationState);

	try {
		this.navigationHelper.getSelectedCategory(request, navigationState);
	} catch (final UnknownIdentifierException e) {
		Logger.getLogger(this.getClass()).warn("No navigation category with code '" +
				navigationState.getCategoryPath() + "' found");
		throw new WebException("Category not found", HttpServletResponse.SC_NOT_FOUND, e);
	}

	// --- Build up Spring MVC view model
	model.put("orderByList", SearchOrderEnumeration.getSortOptions(navigationState.getVendorShop() != null));
	model.put("pageSizeList", pageSizeList);
	model.put("isSearch", Boolean.valueOf(navigationState.getSearchTerm() != null));
	model.put("productsPagination", this.paginationHelper.pagination((int) navigationResult.getTotalCount(),
			navigationResult.getPageSize(), navigationResult.getPage()));
	if (navigationState.getSearchTerm() != null) {
		model.put("suggestionList", this.navigationHelper.getSuggestions(navigationState.getSearchTerm()));
		model.put(CampaignComponentConstants.CONTEXT_CAMPAIGN_ELEMENT,
				searchCampaign(navigationResult.getCampaigns(), component.getCampaignName()));
	}
	if (Config.getBoolean(NavigationHelper.FREDHOPPER_LOG_TIME, false)) {
		Logger.getLogger(this.getClass()).warn
				("FREDHOPPER_TIME: showSearchNavigation took " +
						(System.currentTimeMillis() - startTime) + "ms");
	}

	if (navigationState.getVendorShop() == null) {
		return FrontendConstants.CMS2_COMPONENT_VIEW_PATH + COMPONENT_JSP_PATH;
	} else {
		model.put("contextVendorShop", navigationState.getVendorShop());
		return FrontendConstants.CMS2_COMPONENT_VIEW_PATH + COMPONENT_FOR_VENDOR_SHOP_JSP_PATH;
	}
}

The class NavigationHelper calls the Fredhopper Navigation Service. This service is used to create the “real” communication with the Fredhopper Java Webservice Client like this:

public NavigationResult navigate(final NavigationState state) throws NavigationException {
	...
	// Mappingcode to create the query String going to Fredhopper from the NavigationState Object
	final Page page = this.runQuery(query);
	// Mapping Code from FredHopper Page Object to "NavigationResult" on our side
	...
}

In the runQuery() method we really execute the webservice call, as shown below:

private Page runQuery(final Query query) {
	final String queryString = query.toQueryString();
	final FASWebService fasService = getFasWebService();
	return fasService.getAll(queryString);
}

The FASWebService class is the given Fredhopper class, which returns a FredHopper com.fredhopper.webservice.client.Page object which is used to fill our NavigationResult object.

Starting from this base, we can easily provide a toolbox consisting of Fredhopper WCMS components.

Updating the Fredhopper search index

To export the data from hybris to Fredhopper, we use the existing Solr features from hybris (part of the Solr extension), but extended them for usage with Fredhopper. From here on we start an export which exports all the products, row by row.

If a product has multiple offers from different vendors, the cheapest product will be selected for the export. This is because, according to the current business rules, the cheapest offer be displayed as the default. The idea is that the frontend will retrieve the other offers from the database and not from the Fredhopper index, because the index generation will be to huge and therefore too slow otherwise.

Using Kettle ETL Jobs for exporting and updating the Fredhopper search index.

We export the current product list into a CSV file which is then fed into a Kettle ETL Job which imports the data into the Fredhopper index, which is completely regenerated. Pentaho Kettle is a data integration framework for Extraction, Transformation and Loading (ETL) Jobs. This job is triggerd once a week.

We also run two other jobs on a more regular basis which handle delta updates for new, modified and deleted products. This is our solution to keep Fredhopper up-to-date even for huge data stores.

The first incremental job searches for all modifications in the data model (like updated name, change of category etc) to be exported. We get a big search result here and the job takes a few hours to run. It is triggered once a day.

For information about price and availability we use a second incremental job that runs multiple times a day. Fredhopper doesn’t differentiate between price rows and products so this second job only monitors price rows and only the attributes of price rows (like price and availability) are considered.

Scaling the Fredhopper Integration to huge Marketplaces

As mentioned above and shown in the first diagram, there are two contexts, the general marketplace context, where the customer is browsing the marketplace, and the shop context for the specific vendor stores. Therefore we use a mixed mode: we use Fredhopper to create the product listing pages but load vendor-specific product attributes like vendor prices and stock availability directly from the database.

As a result we have a fast, reliable and up-to-date search and navigation engine which fulfils the specific requirements of a broad marketplace, namely to display a large number of products in varying context, extremely well.

Share

Leave a Reply

*

3 Responses to “Integrating FredHopper into a hybris Marketplace”

  1. Amit Nath says:

    This post was helpful. Specially updating part of Fredhopper index. How complex is it to set up Fredhopper cluster one for Navigational Search and one for Quick Search as mentioned in the diagram? This seems to be difficult on Hybris-SOLR integration.

  2. Imad Eddine says:

    Hello,

    We are a hybris integrator and we have integrated fredhopper in 2 projects. It’s a great solution !