search_expand_cjk

• Drupal 4.7
• Drupal 5
• Drupal 6
• Drupal 7

Versions

4.7 – 7

search_expand_cjk($matches)

Basic CJK tokenizer. Simply splits a string into consecutive, overlapping sequences of characters ('minimum_word_size' long).

Code

modules/search.module, line 370

<?php
function search_expand_cjk($matches) {
  $min = variable_get('minimum_word_size', 3);
  $str = $matches[0];
  $l = drupal_strlen($str);
  // Passthrough short words
  if ($l <= $min) {
    return ' '. $str .' ';
  }
  $tokens = ' ';
  // FIFO queue of characters
  $chars = array();
  // Begin loop
  for ($i = 0; $i < $l; ++$i) {
    // Grab next character
    $current = drupal_substr($str, 0, 1);
    $str = substr($str, strlen($current));
    $chars[] = $current;
    if ($i >= $min - 1) {
      $tokens .= implode('', $chars) .' ';
      array_shift($chars);
    }
  }
  return $tokens;
}
?>

Login or register to post comments