How to make Drupal Search API index the full rendered HTML content

, - Posted by


Drupal is somehow a selected content management system by a lot of government bodies in Western Australia. While it’s a great little system, I still feel that it needs a lot of improvements especially around the search area.

One thing that struck me was the fact that it did not index the final rendered HTML content. Yes it can index content fields fine, but it definitely does not index the final rendered HTML – which sucks in my opinion. The reason is, you would have had a lot of custom content blocks dropped onto the different content placeholders which would appear on your content pages. But, since the only content indexed is your main page content field, when you try to look for a text within your rendered custom blocks, the search results won’t return the corresponding page(s) where the blocks may be loaded to.

I’ve installed Search API and have tried using the default index or even SOLR! And yet both came to no avail. The main reason behind it is not so much about the search technology, but rather the content passed from Drupal to these search indexer does not contain the full rendered HTML content. Therefore, no matter what search technology you use, it won’t return the proper results since the content is never indexed in the first place.


Finally, after some fiddling around, I found a workable solution! And no, it’s not installing Google search but rather a modification to the search API file.

Search API introduced a search field called “Rendered Item”. You can add this field in the Fields area of your index, as well as your Search API Pages. The idea behind it is great, that is to render the actual HTML output of the field. Therefore, I hijack this field to index the actual page’s HTML output instead of just the field’s HTML output – if that makes sense?

Anyway, the Rendered Item column is contained within this file:


This is the line that actually adds the field value into the index:

public function addFieldValues(ItemInterface $item) 

So, I hijack it to include the rendered page HTML. I’m using this method to download the rendered page HTML:


My method now looks like below. Please look at the line in red and green.

function SiteURL()
$protocol = (!empty($_SERVER[‘HTTPS’]) && $_SERVER[‘HTTPS’] !== ‘off’ || $_SERVER[‘SERVER_PORT’] == 443) ? “https://” : “http://”;
$domainName = $_SERVER[‘HTTP_HOST’].’/’;
return $protocol.$domainName;

* {@inheritdoc}
public function addFieldValues(ItemInterface $item) {
$original_user = $this->currentUser->getAccount();

// Switch to the default theme in case the admin theme is enabled.
$active_theme = $this->getThemeManager()->getActiveTheme();
$default_theme = $this->getConfigFactory()
$default_theme = $this->getThemeInitializer()

// Count of items that don’t have a view mode.
$unset_view_modes = 0;

$fields = $this->getFieldsHelper()
->filterForPropertyPath($item->getFields(), NULL, ‘rendered_item’);
foreach ($fields as $field) {

$idString = $item->getId() . ”;

if(strpos($idString, ‘node/’) != false)
$idString = str_replace(“entity:node/”, “”, $idString);
$idString = str_replace(“:en”, “”, $idString);

$html = \file_get_contents($this->SiteURL() . ‘/node/’ . $idString);
$configuration = $field->getConfiguration();

// Change the current user to our dummy implementation to ensure we are
// using the configured roles.
$this->currentUser->setAccount(new UserSession([‘roles’ => $configuration[‘roles’]]));

$datasource_id = $item->getDatasourceId();
$datasource = $item->getDatasource();
$bundle = $datasource->getItemBundle($item->getOriginalObject());
// When no view mode has been set for the bundle, or it has been set to
// “Don’t include the rendered item”, skip this item.
if (empty($configuration[‘view_mode’][$datasource_id][$bundle])) {
// If it was really not set, also notify the user through the log.
if (!isset($configuration[‘view_mode’][$datasource_id][$bundle])) {
else {
$view_mode = (string) $configuration[‘view_mode’][$datasource_id][$bundle];

$value = (string) $this->getRenderer()->renderPlain($build);
if ($value) {


// Restore the original user.
// Restore the original theme.

if ($unset_view_modes > 0) {
$context = [
‘%index’ => $this->index->label(),
‘%processor’ => $this->label(),
‘@count’ => $unset_view_modes,
$this->getLogger()->warning(‘Warning: While indexing items on search index %index, @count item(s) did not have a view mode configured for one or more “Rendered item” fields.’, $context);

Basically, I’m just telling Drupal, if you load a content node, download the full page HTML output and add it to the index.

if(strpos($idString, ‘node/’) != false)
$idString = str_replace(“entity:node/”, “”, $idString);
$idString = str_replace(“:en”, “”, $idString);

$html = \file_get_contents($this->SiteURL() . ‘/node/’ . $idString);

The addValue(string) is the method that adds any string to the index.


Delete all item from the index, then do a full re-index 🙂

NOTE: As this is updating Search API’s system file, when you upgrade the module, ensure you re-apply the changes because it would have been overridden by the upgrade.


Hope this helps,



How to Install Sitecore Experience Platform 9.0 Update 1

, - Posted by


Sitecore has released version 9.0 not too long ago. I should say, I’m probably quiet late coming into the game because by the time I wanted to install it, they have already released the Update 1. In this article I just would like to go through with you the install process just simply because it’s a b**ch. They have now get rid of the EXE and using Powershell instead.

Maybe I’m just whining, but for a $90k+ product I would expect better, a wizard or some sort. SharePoint at least provides us with wizard and that helps us a lot in getting it up and running pretty quickly.

Anyway, let me just run you through my install process.

PS: This is for Single Instance install and not for Azure or for the scalable option.


  1. Download the Installation Guide PDF from Sitecore website
  2. Install SQL 2016, I used Developer edition. You do NOT want to use SQL 2012 because install will fail!
  3. If your SQL is separate from your Sitecore server, Ensure your SQL 2016 can be discovered through network. Go to SQL Configuration Manager and ensure TCP/IP protocol is enabled. Then also check through SQL Management Studio that both Windows Authentication and SQL Authentication are enabled for your DB server, and “Allow remote connection” is also ticked.
  4. From within Sitecore PDF document ensure the prerequisites specified in 2.3.2 is met.
  5. Install SOLR and enable SSL! Sitecore 9 is now using SOLR and not Lucene.


  1. Please ensure that you DO NOT USE SOLR 7 or it won’t be compatible! You would only find that out during the Sitecore Powershell script execution. For me, I used 6.6.1 and that worked fine.
  2. To install SOLR I downloaded the ZIP package instead of the Bitnami executable version. Download from this link:
  3. Then unzip it to, let’s say C:\SOLR661
  4. Next, you need to add Sitecore’s Schema.XML into C:\SOLR661\server\solr\configsets\basic_configs\conf. Lucky I’ve made this available myself through my article. Therefore, please download from here and put it into the basic_configs\conf and rename it to schema.xml.
  5. Once that’s done, you’ll now need to enable SSL. Before I let you reading the SOLR article on enabling SSL, I would like to warn you that you need to make the SOLR SSL port to 8989! The article is here: Where it says “bin\solr.cmd -p 8984” in the article, you want to change the port number to “bin\solr.cmd -p 8989”.
  6. Once you’ve done that, during the process it would have created a solr-ssl.keystore.p12 file. Add and install this certificate into your Personal and Trusted Root Certificate area.
  7. Check if SOLR can be accessed using SSL on port 8989 by going to https://localhost:8989/solr. If it works, next thing is to make SOLR run as Windows Service. At the moment, after step 5, SOLR is running as an executable service however you don’t know what the service name is which we will need for the Powershell script to work. So, let’s stop this service. To do this, open Command Prompt and navigate to C:\SOLR661\bin. Then run solr.cmd stop -p 8989. This will stop the SOLR service.
  8. To install SOLR as Windows Service use this tool:
  9. Once installed, through Command Prompt run: “c:\Program Files\nssm\win64\nssm” install solr661
  10. A dialog will come up and simply select the “solr.cmd” file and in the arguments, ensure it is start -f -p 8989. Below is the example screenshot:
  11. Click Install Service. Then your SOLR service is available as a Windows Service:
  12. Start it. Then confirm if SOLR is now running again in SSL port 8989 ie. https://localhost:8989/solr
  13. Once it’s working, now we’re ready to install Sitecore 9.0! Now go back to Sitecore Downloads page and download the “Packages for XP Single“.
  14. Ensure Sitecore Installation Framework (SIF) for Powershell is enabled! Follow the instructions from this document: Sitecore-Installation-Framework-Configuration-Guide-1.1.pdf. This is all available through the Sitecore Downloads page.
  15. If SIF is installed correctly, you’ll be able to run this command from Powershell to verify: Get-Module SitecoreInstallFramework –ListAvailable
  16. If all looks good, lets now unzip the Sitecore 9.0.1 rev. 171219 (WDP XP0 Packages).zip you download as part of step 12. Let’s unpack them to c:\resourcefiles.
  17. Another thing you want to install is SQLCMD. You can download from here:
  18. Once installed, verify if SQLCMD is available through System Path. To do that, Open Command Prompt and simply type “sqlcmd”. If it says command not recognised then you need to add the path in your Environment Variables. This is needed because Powershell script uses sqlcmd to execute some of its SQL statements!
  19. OK let’s now look at the unzipped file from Step 15. You should see 3 files. What you want to unzip further is XP0 Configuration files 9.0.1 rev. which will contain the SIF Configuration files (in .json extension). Put them on all in the same folder C:\resourcefiles.
  20. Now, go back to your Sitecore install PDF and look at section 4.2.2 item 4. Sitecore has provided a set of script to call those JSON files all in the correct order. If you copy paste from the PDF directly, it contains errors, mainly spacing and text encoding. However, without fixing this, you’ll be in trouble. For example, the Package variables goes like below: 
  21. Package = “$PSScriptRoot\Sitecore 9.0.1 rev. 171219


  22. While it should be:
  23. Package = “$PSScriptRoot\Sitecore 9.0.1 rev. 171219 (OnPrem)” ==> All in one line!
  24. So you’ll need to be fixing all the spacing issue. Finally, the part you want to modify is this section:
  25. #define parameters
    $prefix = “TFS” ==> This would be your Sitecore instance name
    $PSScriptRoot = “C:\resourcefiles”
    $XConnectCollectionService = “$prefix.xconnect”
    $sitecoreSiteName = “$”
    $SolrUrl = “https://localhost:8989/solr”
    $SolrRoot = “C:\Solr-6.6.1”
    $SolrService = “solr661==> This will be using the Windows Service name you created previously!
    $SqlServer = “TFS\TFSDEV01”
    $SqlAdminUser = “sa”
  26. Save the file as install.ps1 and place it in C:\resourcefiles.
  27. Once that’s all done, run the install.ps1 from Powershell. The whole process would take up to 30 minutes to finish depending on how fast your server is.

OK, during the install process I encountered the following issues:

  • Using SQL 2012
  • SQLCMD not installed
  • Using the latest SOLR 7 instead of 6.6.1 or 6.6.2

The script would eventually fail if you use any of the above.


Once the install is finished, go to your C:\inetpub\wwwroot folder. Your Sitecore website is now fully installed and running.

Before you get too excited, the analytics database is no longer using MongoDB, rather it creates an “xconnect” website. In IIS, Sitecore creates 2 websites for me: ==> Sitecore
https://tfs.xconnect ==> The xconnect/analytics website

A lot of people including myself encountered issue trying to connect to the xconnect website. The issue is centered around certificate! Without the xconnect running, your analytics would be disabled, and you would see tons of errors in the log file. Experience Analytics will also show error messages.

So, try to go to your xconnect website https://tfs.xconnect. For me, it asked me to specify client certificate. I’m running Windows 10 and by default it has 2 client certificates available. If you select any of them it would fail, because what you need to use is Sitecore’s XConnect client certificate. The error message you would get is:

HTTP Error 403.16 – Forbidden Your client certificate is either not trusted or is invalid.

As part of your JSON install previously, it runs xconnect-createcert.json which would have created the client certificate for you. Except, it doesn’t add it to the browser. So, to add the XConnect client certificate:

  1. Open Command Prompt and type in “mmc”
  2. Add the Certificates snap-in
  3. In your Personal tab, you would have seen the XConnect client certificate. Download it and ensure you include the private key, thus will ask you to save it as a pfx file. Tick the “Include all certificates in the certificate path if possible” and “Export all extended properties”. Save it to a file.
  4. Then go to Internet Explorer > Internet Options > Content > Certificates. Under Personal tab simply Import the client certificate.
  5. Close your browser and perform IISRESET.
  6. Visit your xconnect website again, https://tfs.xconnect for me. It should show some numbers on the screen if successful.
  7. Finally, go to Sitecore and confirm if Experience Analytics works.

Done…you’re all setup now. You now can run the post deployment steps such as building index, etc.


Along the way, some of the “FilePermission” steps may fail. It did not fail on my Windows 10 laptop, but it did on Windows Server 2012 R2. Simply comment them out:

//”SetAppPoolCertStorePermissions”: {
// Set permissions for the App Pool User to access the client certificate.
// “Type”: “FilePermissions“,
// “Params”: {
// “Path”: “[ResolveCertificatePath(variable(‘Security.XConnect.CertificatePath’))]”,
// “Rights”: [
// {
// “User”: “[concat(‘IIS AppPool\\’, parameter(‘SiteName’))]”,
// “FileSystemRights”: [ “Read” ],
// “InheritanceFlags”: [ “None” ]
// }
// ]
// }

There are multiple of them found in xconnect-xp0.json and sitecore-xp0.json. I don’t think this is a big deal because basically it just gives file permission to the Sitecore folder as well as the XConnect certificate which you can do manually. The main thing is to get the install finished first.


Hope this helps!


Launching My New Weight Loss Journey and Healthy Lifestyle Blog!

, - Posted by

Hi everyone,

I’m pleased to announce that I have now started a new weight loss journey and healthy lifestyle blog! I will also continue writing my cycling articles there. Please visit:

Please also follow me on the social media platforms below:

Youtube –

Twitter –

Instagram –

Facebook –




SharePoint calling 3rd party SOAP – An existing connection was forcibly closed by the remote host

, - Posted by


I have a custom ASMX deployed to SharePoint, within that ASMX was a code to call an another SOAP service (written in PHP). On my mockup ASP.NET website all works well. As soon as I transferred the code to SharePoint, I get this error even though I’ve tried to add the SSL certificate of the third party SOAP call to “Manage trusts” section, etc it still didn’t work.

Exception information:
Exception type: WebException
Exception message: The underlying connection was closed: An unexpected error occurred on a send.
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)


The resolution as apparently very simple!

  1. Add “using System.Net;”
  2. Then before you instantiate your web service, you add this line:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

MyWebService service = new MyWebService();






The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob Generic failure

, - Posted by


My client said to me that their SharePoint user profile is not synching. Upon checking, the “User Profile Synchronization Service” is started, and the User Profile Application Service is also running fine.

However, when I clicked the “Start full synchronization”, nothing happens. The status does not change, it remains in “Idle”. Then I checked in Event Viewer and I get the error of:

The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob threw an exception. More information is included below.

Generic Failure


Finally I got it working by doing the following:

  1. Clear SP cache
    1. Go to services.msc and Stop SharePoint 2010 Administration service
    2. Stop SharePoint 2010 Timer
    4. Delete ALL XMLs including cache.ini.
    5. Start SharePoint 2010 Timer
    6. Start SharePoint 2010 Administration
  2. Go to Central Admin > Manage services on server, stop the User Profile Synchronization service
  3. Restart User Profile Synchronization Service
  4. Re-run User Profile Full Synch




Our Services

We provides you the best Services in our themes.

  • Click on the link below to see a full list of clients which we have developed solutions and provided consultancy for.


  • We are solution-centered and not application-centered.


  • Being creative and having fun and yet still delivering a fantastic service is the center of our values.


  • TFS Consulting Services guarantees delivery that is within budget and deadline or you engage us for free.


Implementing IT does not have to be difficult.

As long as you have the right methodologies

We have heard a lot of complaints from our clients that IT a lot of the times give them headache. The issues range from over-budget implementation, server is too hard to maintain, application is not user friendly, features not complete and many others. If you have ever experienced similar situations, don’t worry. This is why TFS Consulting Services is here. We exist to help clients implementing a successful IT solution. We have various methodologies which we have proven working in delivering a successful IT implementation. Below is the list of some of our key service offerings:
  • Planning and Methodologies

    Implementing IT solution does not have to be difficult. TFS Consulting Services has a lot of resources on planning and methodologies that will ensure successful delivery of your IT solution. TFS Consulting Services has been around in the web industry for more than 10 years and has experienced all the successes and failures of various type of IT deployment.

    read more

  • Technical Resource

    Do you need a technical resource? TFS Consulting Services can also provide you with technical resource for developing ASP.NET (C# and VB.NET), SharePoint (2003, 2007, 2010, 2013) and MS CRM applications. Our resource is an Microsoft Certified Personnel (MVP) and Microsoft Certified Technology Specialist (MCTS) in all ASP.NET, SharePoint and CRM.

    read more

  • IT Consulting & Advice

    Make sure your IT implementation is robust and scalable. TFS Consulting Services can provide consulting and advice on industry’s best practice on various web-related areas such as website security, design and usability, application-specific (such as SharePoint)’s best practice, Search Engine Optimisation (SEO), coding standards and many others.

    read more

  • Solution Development

    Finally TFS Consulting Services provides you with solution development service. We mainly work with Microsoft technologies (ie. .NET and SQL Server), however we are also capable of developing with PHP and MySQL. If you ever need any business process automation, integration and solution development work,  we are the trusted expert you should go to.

    read more

For more detailed service offerings please visit our Solutions page.


  • I’m happy to recommend Tommy as a knowledgeable and diligent developer.

    Mike Stringfellow, Vivid Group
  • Tommy has a lot of great ideas that can be delivered into great products. It’s a pleasure working with him because he has a broad knowledge about available technologies out there and he knows what’s best for the client’s needs. He just knows how to work effectively and efficiently.

    Teddy Segoro, Student Edge
  • I’ve worked with Tommy over the past 6 months and have found his approach to development (especially SharePoint) absolutely outstanding. Tommy goes beyond the boundries of typical software development with his ability understand what a client requires and then build it into fully fledged software solution. Coupled with his professional “Best Practice” approach, you get Continue Reading

    Michael Bonham, DSC-IT

Contact us

Tommy Segoro
+61 404 457 754


© TFS Consulting Services 2018. All rights reserved.