性能检测架构-Kafka

  1. 启动
    bin/kafka-server-start.sh -daemon config/server.properties
  2. 关闭
    bin/kafka-server-stop.sh
  3. 创建消息队列
    bin/kafka-topics.sh –create –topic log-events –bootstrap-server localhost:9092
  4. 查看消息队列详情
    bin/kafka-topics.sh –describe –topic log-events –bootstrap-server localhost:9092
  5. 发消息
    bin/kafka-console-producer.sh –topic log-events –bootstrap-server localhost:9092
  6. 收消息
    bin/kafka-console-consumer.sh –topic log-events –from-beginning –bootstrap-server localhost:9092

参考

  1. MapReduce服务 [https://doc.hcs.huawei.com/zh-cn/devg/mrs/mrs_07_450066.html]

组织 团队 工程
/api/0/organizations/{organization_id_or_slug}/teams/

  1. Create a new project bound to a team
    /api/0/teams/{organization_id_or_slug}/{team_id_or_slug}/projects/

  2. doc js [https://docs.sentry.io/platforms/javascript/apis/]

  3. report crash
    https://apm.kch8.top/api/2/envelope/?sentry_version=7&sentry_key=8603211d236e4cbfbb535c9cb0f9db99&sentry_client=sentry.javascript.browser%2F9.15.0

1
2
3
{"event_id":"eedf7cdfd4f243d0928805be94802369","sent_at":"2025-04-30T02:52:19.365Z","sdk":{"name":"sentry.javascript.browser","version":"9.15.0"},"trace":{"environment":"production","public_key":"8603211d236e4cbfbb535c9cb0f9db99","trace_id":"5bba074d124d472d8b974d6c0512538f"}}
{"type":"event"}
{"exception":{"values":[{"type":"TypeError","value":"a.split is not a function","stacktrace":{"frames":[{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"HTMLDivElement.invoker","in_app":true,"lineno":10954,"colno":5},{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"callWithAsyncErrorHandling","in_app":true,"lineno":2022,"colno":17},{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"callWithErrorHandling","in_app":true,"lineno":2015,"colno":19},{"filename":"http://localhost:5173/src/components/HelloWorld.vue?t=1745981533171","function":"handleApmCrash","in_app":true,"lineno":69,"colno":9}]},"mechanism":{"type":"instrument","handled":false,"data":{"function":"addEventListener","handler":"invoker","target":"EventTarget"}}}]},"level":"error","event_id":"eedf7cdfd4f243d0928805be94802369","platform":"javascript","request":{"url":"http://localhost:5173/","headers":{"Referer":"http://localhost:5173/","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36 Edg/135.0.0.0"}},"timestamp":1745981539.364,"environment":"production","sdk":{"integrations":["InboundFilters","FunctionToString","BrowserApiErrors","Breadcrumbs","GlobalHandlers","LinkedErrors","Dedupe","HttpContext","BrowserSession"],"name":"sentry.javascript.browser","version":"9.15.0","packages":[{"name":"npm:@sentry/browser","version":"9.15.0"}]},"breadcrumbs":[{"timestamp":1745981262.448,"category":"console","data":{"arguments":["message event:worker -> main streamId:undefined stream:undefined callbackId:undefined callback:undefined"],"logger":"console"},"level":"log","message":"message event:worker -> main streamId:undefined stream:undefined callbackId:undefined callback:undefined"},{"timestamp":1745981262.449,"category":"console","data":{"arguments":["worker ready"],"logger":"console"},"level":"log","message":"worker ready"},{"timestamp":1745981271.836,"category":"ui.click","message":"body"},{"timestamp":1745981272.684,"category":"ui.click","message":"body > div#app > div.btn"},{"timestamp":1745981272.689,"category":"sentry.event","event_id":"1d406445cee646d4ba2cd381ba3a2cf9","level":"info","message":"Hello, world!"},{"timestamp":1745981533.283,"category":"console","data":{"arguments":["[vite]","hot updated: /src/components/HelloWorld.vue"],"logger":"console"},"level":"debug","message":"[vite] hot updated: /src/components/HelloWorld.vue"},{"timestamp":1745981539.36,"category":"ui.click","message":"body > div#app > div.btn"},{"timestamp":1745981539.361,"category":"console","data":{"arguments":["[Vue warn]: Unhandled error during execution of native event handler","\n"," at <HelloWorld","msg=\"Fund\"",">","\n"," at <App>"],"logger":"console"},"level":"warning","message":"[Vue warn]: Unhandled error during execution of native event handler \n at <HelloWorld msg=\"Fund\" > \n at <App>"}],"extra":{"arguments":[{"type":"click","target":"body > div#app > div.btn","currentTarget":"body > div#app > div.btn","isTrusted":true,"_vts":1745981539360}]},"contexts":{"trace":{"trace_id":"5bba074d124d472d8b974d6c0512538f","span_id":"a33ad37f768a7669"}}}
1
2
3
4
5
6
7
8
9
10
11
CREATE ROUTINE LOAD log_db.jfzfund_routine ON jfzfund
COLUMNS(dt)
PROPERTIES(
"format"="json",
"jsonpaths"="[\"$.dt\"]"
)
FROM KAFKA(
"kafka_broker_list" = "127.0.0.1:9092",
"kafka_topic" = "log-events",
"property.kafka_default_offsets" = "OFFSET_BEGINNING"
);
1
{"dt":"2025-04-26 22:16:19"}

性能检测架构-Doris

准备

  1. macOS

源码编译

  1. 下载仓库
    1
    https://github.com/apache/doris.git
  2. 编译
  • 预安装依赖

    1
    2
    brew install automake autoconf libtool pkg-config texinfo coreutils gnu-getopt \
    python@3 cmake ninja ccache bison byacc gettext wget pcre maven llvm@16 openjdk@17 npm
  • 设置环境变量

    1
    2
    设置JAVA_HOME环境变量,用java17
    设置node环境变量,用node18
  • 修改build.sh

    1
    去掉 -Werror
  • 执行编译

    1
    sh build.sh --broker  --spark-dpp --hive-udf  --be-java-extensions --be --fe

启动

  1. 调大file descriptors
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # 通过 ulimit 命令调大 file descriptors limit 限制大小
    ulimit -n 65536
    # 查看是否生效
    $ ulimit -n

    # 将该配置写到到启动脚本中,以便下次打开终端会话时不需要再次设置
    # 如果是 bash,执行下面语句
    echo 'ulimit -n 65536' >>~/.bashrc
    # 如果是 zsh,执行下面语句
    echo 'ulimit -n 65536' >>~/.zshrc
  2. 启动BE
    1
    2
    3
    4
    5
    6
    cd output/be/conf
    修改
    priority_networks = 127.0.0.1/32

    cd output/be/bin
    sh ./start_be.sh --daemon
  3. 启动FE
    1
    2
    3
    4
    5
    6
    cd output/fe/conf
    修改
    priority_networks = 127.0.0.1/32

    cd output/fe/bin
    sh ./start_fe.sh --daemon

打开编辑后台

  1. http://127.0.0.1:8030

账号:root
密码为空

  1. 点击playground,注册backend
    1
    2
    3
    4
    ALTER SYSTEM ADD BACKEND "127.0.0.1:9050";
    `

    ## 插入数据
    curl –location-trusted -u root: -H “format:json” -H “read_json_by_line:true” -H “load_to_single_tablet:true” -H “strict_mode:false” -H “timeout:600” -H “expect:100-continue” -T /Users/mac/Documents/Web/doris-master/output/test/log.json http://127.0.0.1:8030/api/log_db/log_table/_stream_load
    1
    2
    3
    4
    5
    6
    7
    8
    9

    ##
    1. 采集 app
    2. 传输 kaffa
    3. 解析 flink
    4. 存储 doris
    5. 展示 grafana

    ## 常见命令
    mysql -uroot -P9030 -h127.0.0.1 -e “show backends;”
    show VARIABLES like ‘%insert%’;
    show partitions from log_table;
    drop table log_table;
    1
    2
    3
    4
    5
    6
    7

    ## 参考
    1. 导入配置参数 [https://doris.apache.org/zh-CN/docs/data-operate/import/import-way/stream-load-manual#%E5%AF%BC%E5%85%A5%E9%85%8D%E7%BD%AE%E5%8F%82%E6%95%B0]

    2. 网易云改造 [https://www.selectdb.com/blog/1369]


    CREATE TABLE jfzfund_998866 (
    application_id VARCHAR(32) NULL COMMENT ‘应用id’,
    log_type VARCHAR(12) NULL COMMENT ‘日志类型/jm/tm’,
    container_id VARCHAR(32) NULL COMMENT ‘container_id’,
    logs_timestamp BIGINT NULL COMMENT ‘日志产生时间’,
    log_level VARCHAR(10) NULL COMMENT ‘日志级别’,
    host_name VARCHAR(32) NULL COMMENT ‘主机名’,
    exception_log TEXT NULL COMMENT ‘异常日志’,
    job_id INT NULL COMMENT ‘任务id’,
    message TEXT NULL COMMENT ‘日志内容’,
    tag TEXT NULL COMMENT ‘日志关键指标’,
    log_file_path TEXT NULL COMMENT ‘日志存储路径’,
    exception_class_name TEXT NULL COMMENT ‘异常类名’,
    exception_type TEXT NULL COMMENT ‘异常类型’,
    exception_message TEXT NULL COMMENT ‘异常msg’,
    exception_caused_by TEXT NULL COMMENT ‘异常caused_by’,
    dt date NULL COMMENT ‘天’,
    hh TEXT NULL COMMENT ‘小时’,
    mm TEXT NULL COMMENT ‘分钟’,
    INDEX idx_exception_message (exception_message) USING INVERTED PROPERTIES(“parser” = “english”),
    INDEX idx_message (message) USING INVERTED PROPERTIES(“parser” = “english”)
    ) ENGINE=OLAP
    DUPLICATE KEY(application_id, log_type, container_id, logs_timestamp, log_level, host_name)
    COMMENT ‘OLAP’
    PARTITION BY RANGE(dt)()
    DISTRIBUTED BY RANDOM BUCKETS 15
    PROPERTIES (
    “replication_num” = “1”,
    “dynamic_partition.enable” = “true”,
    “dynamic_partition.time_unit” = “DAY”,
    “dynamic_partition.start” = “-15”,
    “dynamic_partition.end” = “2”,
    “dynamic_partition.prefix” = “p”,
    “dynamic_partition.buckets” = “15”,
    “dynamic_partition.create_history_partition” = “true”,
    “compression” = “ZSTD”,
    “compaction_policy” = “time_series”
    );

无类别域间路由介绍

CIDR主要是一个按位的、基于前缀的,用于解释IP地址的标准。它通过把多个地址块组合到一个路由表表项而使得路由更加方便。这些地址块叫做CIDR地址块、CIDR块、CIDR区块或IP段。当用二进制表示这些地址时,它们有着在开头部分的一系列相同的位。IPv4的CIDR地址块的表示方法和IPv4地址的表示方法是相似的:由四部分组成的点分十进制地址,后跟一个斜线,最后是范围在0到32之间的一个数字:A.B.C.D/N。点分十进制的部分和IPv4地址一样是一个被分成四个八位位组的32位二进制数。斜线后面的数字就是前缀长度,也就是从左到右,被地址块里的地址所共享的位的数目。当只需说明大概时,十进制部分有时会被省略,因此,/20就表示一个前缀长度是20的CIDR地址块。如果一个IP地址的前N位与一个CIDR地址块的前缀是相同的话,那么就说这个地址属于这个CIDR地址块,也可以说是与CIDR地址块的前缀匹配。所以,要理解CIDR,就要把地址写成二进制的形式。因为IPv4地址的长度总是32位,N位长的CIDR前缀就意味着地址里前N位匹配,后
32

N
{\displaystyle 32-N}位不匹配。这些位有
2
(
32

N
)
{\displaystyle 2^{(32-N)}}种不同的组合,即
2(32−N)
{\displaystyle 2^{(32-N)}}个IPv4地址与CIDR地址块的前缀匹配。前缀越短就能匹配越多的地址,越长就匹配得越少。一个地址可能与多个长度不同的CIDR前缀匹配。CIDR也用在IPv6中。因为位数的非常多,所以在IPv6中,前缀长度的范围是从0到128。这里也用同样的方法来表示一个地址:前缀写作一个IPv6的地址,后跟一个斜线,最后是前缀的位数。


util-safe-delete-file

1
ls -l | awk '{ print "rm -f ", $9 }' | sh

harmony-hvigor-plugin

  1. 重命名插件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    export function renameAppPlugin(str?: string): HvigorPlugin {
    return {
    pluginId: 'RenameAppPlugin',
    apply(currentNode: HvigorNode) {
    const appContext = currentNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext
    const appJson = appContext.getAppJsonOpt()
    const appVersion = appJson['app']['versionName']
    console.log('RenameAppPlugin apply start, appVersion:', appVersion)
    const product = appContext.getCurrentProduct()
    const productName = product.getProductName()
    const bundleType = product.getBundleType()
    console.log('RenameAppPlugin apply start, productName:', productName, bundleType, product.getBundleName())

    currentNode.registerTask({
    // 编写自定义任务
    name: 'renameAppTask',
    run: (taskContext) => {
    console.log('renameHarTask run start', taskContext.moduleName, taskContext.modulePath)
    const formatZero = (d: number): string => {
    return d < 10 ? `0${d}` : `${d}`
    }
    const formatDate = (): string => {
    const date = new Date()
    return `${date.getFullYear()}${formatZero(date.getMonth() +
    1)}${formatZero(date.getDate())}${formatZero(date.getHours())}${formatZero(date.getMinutes())}`
    }

    const rename = (signType: string) => {
    const outputPath = `/build/outputs/${productName}/`
    const sourceName = `${taskContext.moduleName}-${productName}-${signType}.${bundleType}`
    const sourceFile = taskContext.modulePath + outputPath + sourceName
    const targetPath = taskContext.modulePath + outputPath
    const targetName = `${taskContext.moduleName}-${productName}-${appVersion}-${formatDate()}-${signType}.${bundleType}`
    const targetFile = targetPath + targetName
    console.log('renameAppTask sourceFile:', sourceFile)
    console.log('renameAppTask targetFile:', targetFile)

    // 创建目录
    fs.mkdir(targetPath, { recursive: true }, (err) => {
    if (err) {
    console.log('renameAppTask mkdir err: ' + err)
    }
    fs.rename(sourceFile, targetFile, (err) => {
    console.log('renameAppTask rename err: ' + err)
    })
    })
    }

    rename('signed')
    rename('unsigned')
    },
    dependencies: ['PackageApp'],
    postDependencies: ['assembleApp']
    })
    }
    }
    }

  2. 输出重命名

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    hvigor.getRootNode().afterNodeEvaluate(rootNode => {
    // 获取app插件的上下文对象
    const appContext = rootNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext;
    const projectName = appContext.getProjectName()
    const product = appContext.getCurrentProduct()
    const productName = product.getProductName()

    const appJson = appContext.getAppJsonOpt();
    const appVersion = appJson['app']['versionName']
    console.log('renameAppPlugin getRootNode, appJson:', appJson)
    console.log('renameAppPlugin getRootNode, productName:', productName, projectName)

    const buildProfileJson = appContext.getBuildProfileOpt();
    // console.log('renameAppPlugin getRootNode start, buildProfileJson:', buildProfileJson)
    const productsInBuildProfile = buildProfileJson['app']['products']
    console.log('renameAppPlugin getRootNode, productsInBuildProfile:', productsInBuildProfile)

    const formatZero = (d: number): string => {
    return d < 10 ? `0${d}` : `${d}`
    }
    const formatDate = (): string => {
    const date = new Date()
    return `${date.getFullYear()}${formatZero(date.getMonth() +
    1)}${formatZero(date.getDate())}${formatZero(date.getHours())}${formatZero(date.getMinutes())}`
    }

    for (const product of productsInBuildProfile) {
    if (!product['output']) {
    product['output'] = { 'artifactName': '' }
    }
    product['output']['artifactName'] += `${projectName}-${productName}-${appVersion}-${formatDate()}`
    console.log('renameAppPlugin getRootNode, output name:', product['output']['artifactName'])
    }

    console.log('renameAppPlugin getRootNode, is assembleApp task:', hvigor.isCommandEntryTask('assembleApp'), hvigor.getCommandEntryTask());
    if (hvigor.isCommandEntryTask('assembleApp') && productName === 'prod') {
    const foundProduct = productsInBuildProfile.find(product => product['name'] === 'prod')
    if (foundProduct) {
    foundProduct['signingConfig'] = 'prod'
    appContext.setBuildProfileOpt(buildProfileJson)
    console.log('renameAppPlugin assemble prod app, change to prod sign', foundProduct)
    }
    }
    })

  3. 动态添加依赖

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    // https://developer.huawei.com/consumer/cn/doc/harmonyos-guides-V5/ide-build-expanding-context-V5
    // function jfzDependPlugin(): HvigorPlugin {
    // return {
    // pluginId: 'jfzDependPlugin',
    // async apply(currentNode: HvigorNode): Promise<void> {
    // console.log('jfzDependPlugin', currentNode)
    // const appContext = currentNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext;
    // console.log('jfzDependPlugin appContext', appContext)
    // const dependency = appContext.getDependenciesOpt({}); //获取dependency依赖
    // // dependency["library"]="file:library.har"
    // console.log(dependency);
    // appContext.setDependenciesOpt(dependency ) //修改dependency依赖
    // }
    // }
    // }

pyenv安装python错误总结

  1. No rule to make target pyconfig.h', needed by Programs/python.o’
    a. 查看 diff 命令指向,如which diff
    如果输出为:$HOME/Library/OpenHarmony/Sdk/12/toolchains/diff这个,会导致编译失败
    把diff指向系统的diff,如:/usr/bin/diff
    b. 删除上次缓存,python-build*目录