Elasticsearch 500萬索引批量存儲php的示例分析

這篇文章給大家分享的是有關Elasticsearch 500萬索引批量存儲php的示例分析的內容。小編覺得挺實用的，因此分享給大家做個參考，一起跟隨小編過來看看吧。

引用文件

dict.txt.cache.json, pinyin.php 云盤下載地址:

http://pan.baidu.com/s/1c1W6fQC

索引生成測試代碼 php

單條索引生成,速度較慢  elastic.php

<?php
use Elasticsearch\ClientBuilder;
require 'vendor/autoload.php';

include('pinyin.php');
ini_set('memory_limit','10280M');
$start = "開始時間:" . time() . PHP_EOL;
echo $start;
$client = ClientBuilder::create()->build();

$dict = array_map(function($str){
    return str_ireplace('.', '', $str);},
    array_keys(json_decode(file_get_contents('./dict.txt.cache.json'),
        true)
    )
);
$a = 1;
do {
    $params = [
        'index' => 'my_index',
        'type' => 'my_type',
        'id' => $a,
        'body' => [
            'user_id' =>  mt_rand(100000000, 2000000000).'@qq.com',
            'name' => $tmpname = $dict[mt_rand(1000,360000)],
            'py' => \Utils\Pinyin::conv($tmpname),
            'phone' => array_rand(array_flip([13,15,17,18])) . mt_rand(100000000, 999999999),
            'mail' => substr(
                    str_shuffle(
                        "0123456789abcdefghijklmnopqrstuvwxyz"),
                    0, mt_rand(5,20)
                ) . '@' . array_rand(
                    array_flip(
                        [
                            '163.com',
                            '126.com',
                            'yeah.net',
                            'qq.com',
                            'foxmail.com',
                            'gmail.com',
                            'yahoo.com',
                            'hotmail.com',
                            'sina.com',
                            'sina.cn',
                            'sina.com.cn'
                        ]
                    )
                ),
            'appoint' => implode(
                '-',
                array_map(function() use($dict) {
                    return $dict[mt_rand(1000,360000)] . (mt_rand(0,100) > 60 ?
                        $dict[mt_rand(1000,360000)] : '') . (mt_rand(0,100) > 80 ?
                        $dict[mt_rand(1000,360000)] : '');},
                    array_pad([], mt_rand(2,5), '')
                )
            ),
        ]
    ];
    //創建索引
    $response = $client->index($params);

    $a = isset($a) ? ++$a : 1;
    $a % 50 === 0 ? print(($b = isset($b) ? ++$b : 1).'%'.PHP_EOL) : '';
} while ($a <= 5000);
$end = "結束時間:" . time() . PHP_EOL;
echo $end;
echo "耗時:" . ($end - $start);

批量索引生成  elasticBulk.php

<?php
/**
 * 500萬用戶模擬數據批量新增到 elasticsearch demo
 */
use Elasticsearch\ClientBuilder;
require 'vendor/autoload.php';

include('pinyin.php');
ini_set('memory_limit','10280M');

$connectionPool = '\Elasticsearch\ConnectionPool\StaticNoPingConnectionPool';
$selector = '\Elasticsearch\ConnectionPool\Selectors\StickyRoundRobinSelector';
$client = ClientBuilder::create()
    ->setRetries(10)//重試次數，默認重試次數為集群節點數
    ->setConnectionPool($connectionPool)
    ->setSelector($selector)
    ->build();

$dict = array_map(function($str){
    return str_ireplace('.', '', $str);},
    array_keys(json_decode(file_get_contents('./dict.txt.cache.json'),
            true)
    )
);

echo $start = "開始時間:" . time() . PHP_EOL;
createData($client, $dict);
echo $end = "結束時間:" . time() . PHP_EOL;
echo "耗時:" . $end - $start;

function createData($client, $dict){
    $bulk = array('index'=>'my_index4','type'=>'my_type4');
    //bulk批量生成
    for($j = 0;$j <= 99; $j++) {
        for($i = $j * 50000 + 1; $i <= $j * 50000 + 50000; $i ++) {
            $bulk['body'][]=array(
                'index' => array(
                    '_id'=>$i
                ),
                'type' => 'blocking'
            );

            $bulk['body'][] = [
                'user_id' =>  mt_rand(100000000, 2000000000).'@qq.com',
                'name' => $tmpname = $dict[mt_rand(1000,360000)],
                'py' => \Utils\Pinyin::conv($tmpname),
                'phone' => array_rand(array_flip([13,15,17,18])) . mt_rand(100000000, 999999999),
                'mail' => substr(
                        str_shuffle(
                            "0123456789abcdefghijklmnopqrstuvwxyz"),
                        0, mt_rand(5,20)
                    ) . '@' . array_rand(
                        array_flip(
                            [
                                '163.com',
                                '126.com',
                                'yeah.net',
                                'qq.com',
                                'foxmail.com',
                                'gmail.com',
                                'yahoo.com',
                                'hotmail.com',
                                'sina.com',
                                'sina.cn',
                                'sina.com.cn'
                            ]
                        )
                    ),
                'appoint' => implode(
                    '-',
                    array_map(function() use($dict) {
                        return $dict[mt_rand(1000,360000)] . (mt_rand(0,100) > 60 ?
                            $dict[mt_rand(1000,360000)] : '') . (mt_rand(0,100) > 80 ?
                            $dict[mt_rand(1000,360000)] : '');},
                        array_pad([], mt_rand(2,5), '')
                    )
                ),
            ];
        }
        $client->bulk($bulk);

        //進度統計
        print($j + 1).'%'.PHP_EOL;
    }
}

單節點測試結論

1.單條性能約100條/s , 批量性能約5000條/s

2.單節點批量處理一次性bulk數據上限35萬，否則報錯：PHP Fatal error:  Uncaught exception 'Elasticsearch\Common\Exceptions\NoNodesAvailableException' with message 'No alive nodes found in your cluster

3. 500萬全真模擬數據占空間2G

4.查詢性能:

第一次稍慢，但在１秒以內

第二次極速

感謝各位的閱讀！關于“Elasticsearch 500萬索引批量存儲php的示例分析”這篇文章就分享到這里了，希望以上內容可以對大家有一定的幫助，讓大家可以學到更多知識，如果覺得文章不錯，可以把它分享出去讓更多的人看到吧！