Merging function get_keyword() and function keyword_count()

[splifingate]

It seems duplicitous, to me, that there would be two distinct functions() pertaining to a subjectively singular process.

Just trying to determine the reasons behind the separate functions(), and getting my mind around the idea of merging the two...

...could/should this be a Thing?

[danpros]

Yes we need to improve the search functions/functionality, previously we only search the title and tags based on the filename because I don't want to open each file to find the desired word. Imagine if we have 10K articles, opening 10K files one by one will definitely slow things down a lot.

I'm going to create some kind of new method for this. So it's almost similar to page views.

There is a kind of search index that contains each article content. When we create an article, update/edit/delete. We will automatically add this to the search index. We can also manually add this.

There are two options, search only title and tags, or title, tags and content (more RAM used).

[taufik-nurrohman]

The solution is to read the file line by line but not storing it into memory. Just check if a query is found in the string then stop the loop, continue to the next file ^{1 2}

if (is_file($path) && $h = fopen($path, 'r')) {
    while (false !== ($v = fgets($h, 4096)) {
        // Find the query in `$v`
        if (has_query_in_file_stream($v)) {
            add_file_path_to_queue($path);
            break;
        }
    }
    fclose($h);
}

Footnotes

1. https://github.com/mecha-cms/mecha/blob/v3.2.0/engine/f.php#L1410 ↩

2. https://github.com/mecha-cms/mecha/blob/v3.2.0/engine/f.php#L1142 ↩

[danpros]

@taufik-nurrohman Thanks for the reference.

For performance reasons it seems best to put the contents into a single file in json format (id => content), sure more RAM during read or update the index. I tried and for 150 articles between 300-500 words, the file created was around 400-600KB.

[taufik-nurrohman]

No problem. It seems that HTMLy is more into improving the performance by file cache (requires more storage) than leveraging the RAM (less storage). But I understand.

[splifingate]

"There are two options, search only title and tags, or title, tags and content (more RAM used)."

I admit to using systems and resources where CPU and RAM are not really an issue . . . this idea has not been a personal consideration since the 486-DX Days 🤷

It just makes more sense, to me--in the present moment--to get my mind around the idea that the get_keyword($keyword, $page, $perpage) and keyword_count($keyword) functions are co-joined (or, at least, share the same Space).

Why are these two functions() separate?

[splifingate]

For that matter: since the advanced functionality of AdminLTE is central to the Core, I am wondering if a few extra MB of space is really a limiting factor....

[danpros]

Why are these two functions() separate?

I am introducing the search count lately, so I just using two existing functions.

By using the old model (only title and tags) the performance will not be felt but with the new type it will be quite noticeable (opening files one by one).

I have made it in local development (only using one function to get the search result). With index etc.

Current code in my local dev:

// Return search page.
function get_keyword($keyword, $page, $perpage)
{
    $posts = get_blog_posts();

    $tmp = array();
    $search = array();
    
    $searchFile = "content/data/search.json";
    if (file_exists($searchFile)) {
        $search = json_decode(file_get_contents($searchFile), true);
    }

    foreach ($posts as $index => $v) {
        $arr = explode('_', $v['filename']);
        
        if (isset($search[$arr[2]])) {
            $filter = $search[$arr[2]];
        } else {
            $filter = $arr[1] . ' ' . $arr[2];
        }

        $findRxWhole = '\b' . preg_quote($keyword, '~') . '\b'; // whole-words only

        $findRx = "~{$findRxWhole}~iu"; // Case-insensitive and UTF-8 mode
        
        if (preg_match ($findRx, $filter)) {
            if (!in_array($v, $tmp)) {
                $tmp[] = $v;
            }
        }
        
    }

    if (empty($tmp)) {
        return false;
    }

    return $tmp = array(get_posts($tmp, $page, $perpage), count($tmp));

}

In htmly.php previously:

$posts = get_keyword($keyword, $page, $perpage);

Since now returning an array we need to use $posts[0] for the posts and $posts[1] for the count.

    $total = '';

    $posts = get_keyword($keyword, $page, $perpage);
    if ($posts) { 
        $total = $posts[1];
    }

I will commit it to the repo after its done.

Note: The search index using post-slug => content (key => value).

[splifingate]

Looking forward to seeing just how this develops ;)

[danpros]

Please test the latest branch e4dfc42

Enable the fulltext search option using admin/config. You will find a new link /admin/search on sidebar, here you can find posts that have not been indexed, you can also add them.

At the moment it is still very simple. When we add, editing a post, it will automatically added to the index.

[splifingate]

I have been otherwise consumed with activities that demand my attentions.

When I catch a breath, I will resume ;)

[splifingate]

So; I have revisited your idea, but I have no real understanding of what I need to do to be writing upon the same page as you...

...why does this functionality need a complete re-write?

[splifingate]

I understand little about =>

[danpros]

Hello,

The get_keyword() function in the latest version is more efficient, as you stated earlier. Previously, to get the count number, we had to call keyword_count() and in the last version it is not necessary. The keyword_count() function still exists but is no longer needed.

// Return search page.
function get_keyword($keyword, $page, $perpage)
{
    
    if (strlen($keyword) < 3) {
        return array();
    }

    $posts = get_blog_posts();

    $tmp = array();
    $search = array();
    
    $searchFile = "content/data/search.json";
    
    if (file_exists($searchFile) && config('fulltext.search') == "true") {
        $search = json_decode(file_get_contents($searchFile), true);
    }

    foreach ($posts as $index => $v) {
        $arr = explode('_', $v['filename']);
        
        if (isset($search['post_' . $arr[2]])) {
            $filter = $search['post_' . $arr[2]];
        } else {
            $filter = $arr[1] . ' ' . $arr[2];
            $keyword = remove_accent($keyword);
        }

        if (stripos($filter, trim($keyword)) !== false) {
            if (!in_array($v, $tmp)) {
                $tmp[] = $v;
            }
        }
        
    }

    if (empty($tmp)) {
        return $tmp;
    }

    return $tmp = array(get_posts($tmp, $page, $perpage), count($tmp));

}