1. 启动
   bin/kafka-server-start.sh -daemon config/server.properties
2. 关闭
   bin/kafka-server-stop.sh
3. 创建消息队列
   bin/kafka-topics.sh –create –topic log-events –bootstrap-server localhost:9092
4. 查看消息队列详情
   bin/kafka-topics.sh –describe –topic log-events –bootstrap-server localhost:9092
5. 发消息
   bin/kafka-console-producer.sh –topic log-events –bootstrap-server localhost:9092
6. 收消息
   bin/kafka-console-consumer.sh –topic log-events –from-beginning –bootstrap-server localhost:9092

参考

1. MapReduce服务 [https://doc.hcs.huawei.com/zh-cn/devg/mrs/mrs_07_450066.html]

组织 团队 工程
/api/0/organizations/{organization_id_or_slug}/teams/

1. Create a new project bound to a team
   /api/0/teams/{organization_id_or_slug}/{team_id_or_slug}/projects/

2. doc js [https://docs.sentry.io/platforms/javascript/apis/]

3. report crash
   https://apm.kch8.top/api/2/envelope/?sentry_version=7&sentry_key=8603211d236e4cbfbb535c9cb0f9db99&sentry_client=sentry.javascript.browser%2F9.15.0

1
2
3

{"event_id":"eedf7cdfd4f243d0928805be94802369","sent_at":"2025-04-30T02:52:19.365Z","sdk":{"name":"sentry.javascript.browser","version":"9.15.0"},"trace":{"environment":"production","public_key":"8603211d236e4cbfbb535c9cb0f9db99","trace_id":"5bba074d124d472d8b974d6c0512538f"}}
{"type":"event"}
{"exception":{"values":[{"type":"TypeError","value":"a.split is not a function","stacktrace":{"frames":[{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"HTMLDivElement.invoker","in_app":true,"lineno":10954,"colno":5},{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"callWithAsyncErrorHandling","in_app":true,"lineno":2022,"colno":17},{"filename":"http://localhost:5173/node_modules/.vite/deps/vue.js?v=fcd7ae09","function":"callWithErrorHandling","in_app":true,"lineno":2015,"colno":19},{"filename":"http://localhost:5173/src/components/HelloWorld.vue?t=1745981533171","function":"handleApmCrash","in_app":true,"lineno":69,"colno":9}]},"mechanism":{"type":"instrument","handled":false,"data":{"function":"addEventListener","handler":"invoker","target":"EventTarget"}}}]},"level":"error","event_id":"eedf7cdfd4f243d0928805be94802369","platform":"javascript","request":{"url":"http://localhost:5173/","headers":{"Referer":"http://localhost:5173/","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36 Edg/135.0.0.0"}},"timestamp":1745981539.364,"environment":"production","sdk":{"integrations":["InboundFilters","FunctionToString","BrowserApiErrors","Breadcrumbs","GlobalHandlers","LinkedErrors","Dedupe","HttpContext","BrowserSession"],"name":"sentry.javascript.browser","version":"9.15.0","packages":[{"name":"npm:@sentry/browser","version":"9.15.0"}]},"breadcrumbs":[{"timestamp":1745981262.448,"category":"console","data":{"arguments":["message event:worker -> main streamId:undefined stream:undefined callbackId:undefined callback:undefined"],"logger":"console"},"level":"log","message":"message event:worker -> main streamId:undefined stream:undefined callbackId:undefined callback:undefined"},{"timestamp":1745981262.449,"category":"console","data":{"arguments":["worker ready"],"logger":"console"},"level":"log","message":"worker ready"},{"timestamp":1745981271.836,"category":"ui.click","message":"body"},{"timestamp":1745981272.684,"category":"ui.click","message":"body > div#app > div.btn"},{"timestamp":1745981272.689,"category":"sentry.event","event_id":"1d406445cee646d4ba2cd381ba3a2cf9","level":"info","message":"Hello, world!"},{"timestamp":1745981533.283,"category":"console","data":{"arguments":["[vite]","hot updated: /src/components/HelloWorld.vue"],"logger":"console"},"level":"debug","message":"[vite] hot updated: /src/components/HelloWorld.vue"},{"timestamp":1745981539.36,"category":"ui.click","message":"body > div#app > div.btn"},{"timestamp":1745981539.361,"category":"console","data":{"arguments":["[Vue warn]: Unhandled error during execution of native event handler","\n"," at <HelloWorld","msg=\"Fund\"",">","\n"," at <App>"],"logger":"console"},"level":"warning","message":"[Vue warn]: Unhandled error during execution of native event handler \n  at <HelloWorld msg=\"Fund\" > \n  at <App>"}],"extra":{"arguments":[{"type":"click","target":"body > div#app > div.btn","currentTarget":"body > div#app > div.btn","isTrusted":true,"_vts":1745981539360}]},"contexts":{"trace":{"trace_id":"5bba074d124d472d8b974d6c0512538f","span_id":"a33ad37f768a7669"}}}

1
2
3
4
5
6
7
8
9
10
11

CREATE ROUTINE LOAD log_db.jfzfund_routine ON jfzfund
COLUMNS(dt)
PROPERTIES(
    "format"="json",
    "jsonpaths"="[\"$.dt\"]"
)
FROM KAFKA(
    "kafka_broker_list" = "127.0.0.1:9092",
    "kafka_topic" = "log-events",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);

1

{"dt":"2025-04-26 22:16:19"}

准备

1. macOS

源码编译

1. 下载仓库

   1	https://github.com/apache/doris.git

2. 编译

• 预安装依赖

  1 2	brew install automake autoconf libtool pkg-config texinfo coreutils gnu-getopt \ python@3 cmake ninja ccache bison byacc gettext wget pcre maven llvm@16 openjdk@17 npm

• 设置环境变量

  1 2	设置JAVA_HOME环境变量，用java17 设置node环境变量，用node18

• 修改build.sh

  1	去掉 -Werror

• 执行编译

  1	sh build.sh --broker --spark-dpp --hive-udf --be-java-extensions --be --fe

启动

1. 调大file descriptors

   1 2 3 4 5 6 7 8 9 10

   # 通过 ulimit 命令调大 file descriptors limit 限制大小 ulimit -n 65536 # 查看是否生效 $ ulimit -n # 将该配置写到到启动脚本中，以便下次打开终端会话时不需要再次设置 # 如果是 bash，执行下面语句 echo 'ulimit -n 65536' >>~/.bashrc # 如果是 zsh，执行下面语句 echo 'ulimit -n 65536' >>~/.zshrc

2. 启动BE

   1 2 3 4 5 6

   cd output/be/conf 修改 priority_networks = 127.0.0.1/32 cd output/be/bin sh ./start_be.sh --daemon

3. 启动FE

   1 2 3 4 5 6

   cd output/fe/conf 修改 priority_networks = 127.0.0.1/32 cd output/fe/bin sh ./start_fe.sh --daemon

打开编辑后台

1. http://127.0.0.1:8030

账号:root
密码为空

2. 点击playground，注册backend

   1 2 3 4

   ALTER SYSTEM ADD BACKEND "127.0.0.1:9050"; ` ## 插入数据

   curl –location-trusted -u root: -H “format:json” -H “read_json_by_line:true” -H “load_to_single_tablet:true” -H “strict_mode:false” -H “timeout:600” -H “expect:100-continue” -T /Users/mac/Documents/Web/doris-master/output/test/log.json http://127.0.0.1:8030/api/log_db/log_table/_stream_load

   1 2 3 4 5 6 7 8 9

   ## 1. 采集 app 2. 传输 kaffa 3. 解析 flink 4. 存储 doris 5. 展示 grafana ## 常见命令

   mysql -uroot -P9030 -h127.0.0.1 -e “show backends;”
   show VARIABLES like ‘%insert%’;
   show partitions from log_table;
   drop table log_table;

   1 2 3 4 5 6 7

   ## 参考 1. 导入配置参数 [https://doris.apache.org/zh-CN/docs/data-operate/import/import-way/stream-load-manual#%E5%AF%BC%E5%85%A5%E9%85%8D%E7%BD%AE%E5%8F%82%E6%95%B0] 2. 网易云改造 [https://www.selectdb.com/blog/1369]

   CREATE TABLE jfzfund_998866 (
   application_id VARCHAR(32) NULL COMMENT ‘应用id’,
   log_type VARCHAR(12) NULL COMMENT ‘日志类型/jm/tm’,
   container_id VARCHAR(32) NULL COMMENT ‘container_id’,
   logs_timestamp BIGINT NULL COMMENT ‘日志产生时间’,
   log_level VARCHAR(10) NULL COMMENT ‘日志级别’,
   host_name VARCHAR(32) NULL COMMENT ‘主机名’,
   exception_log TEXT NULL COMMENT ‘异常日志’,
   job_id INT NULL COMMENT ‘任务id’,
   message TEXT NULL COMMENT ‘日志内容’,
   tag TEXT NULL COMMENT ‘日志关键指标’,
   log_file_path TEXT NULL COMMENT ‘日志存储路径’,
   exception_class_name TEXT NULL COMMENT ‘异常类名’,
   exception_type TEXT NULL COMMENT ‘异常类型’,
   exception_message TEXT NULL COMMENT ‘异常msg’,
   exception_caused_by TEXT NULL COMMENT ‘异常caused_by’,
   dt date NULL COMMENT ‘天’,
   hh TEXT NULL COMMENT ‘小时’,
   mm TEXT NULL COMMENT ‘分钟’,
   INDEX idx_exception_message (exception_message) USING INVERTED PROPERTIES(“parser” = “english”),
   INDEX idx_message (message) USING INVERTED PROPERTIES(“parser” = “english”)
   ) ENGINE=OLAP
   DUPLICATE KEY(application_id, log_type, container_id, logs_timestamp, log_level, host_name)
   COMMENT ‘OLAP’
   PARTITION BY RANGE(dt)()
   DISTRIBUTED BY RANDOM BUCKETS 15
   PROPERTIES (
   “replication_num” = “1”,
   “dynamic_partition.enable” = “true”,
   “dynamic_partition.time_unit” = “DAY”,
   “dynamic_partition.start” = “-15”,
   “dynamic_partition.end” = “2”,
   “dynamic_partition.prefix” = “p”,
   “dynamic_partition.buckets” = “15”,
   “dynamic_partition.create_history_partition” = “true”,
   “compression” = “ZSTD”,
   “compaction_policy” = “time_series”
   );

https://stackoverflow.com/questions/69012676/install-older-ruby-versions-on-a-m1-macbook

1
2

brew install rbenv
RUBY_CFLAGS="-w" rbenv install 2.7.4

CIDR主要是一个按位的、基于前缀的，用于解释IP地址的标准。它通过把多个地址块组合到一个路由表表项而使得路由更加方便。这些地址块叫做CIDR地址块、CIDR块、CIDR区块或IP段。当用二进制表示这些地址时，它们有着在开头部分的一系列相同的位。IPv4的CIDR地址块的表示方法和IPv4地址的表示方法是相似的：由四部分组成的点分十进制地址，后跟一个斜线，最后是范围在0到32之间的一个数字：A.B.C.D/N。点分十进制的部分和IPv4地址一样是一个被分成四个八位位组的32位二进制数。斜线后面的数字就是前缀长度，也就是从左到右，被地址块里的地址所共享的位的数目。当只需说明大概时，十进制部分有时会被省略，因此，/20就表示一个前缀长度是20的CIDR地址块。如果一个IP地址的前N位与一个CIDR地址块的前缀是相同的话，那么就说这个地址属于这个CIDR地址块，也可以说是与CIDR地址块的前缀匹配。所以，要理解CIDR，就要把地址写成二进制的形式。因为IPv4地址的长度总是32位，N位长的CIDR前缀就意味着地址里前N位匹配，后
32
−
N
{\displaystyle 32-N}位不匹配。这些位有
2
(
32
−
N
)
{\displaystyle 2^{(32-N)}}种不同的组合，即
2(32−N)
{\displaystyle 2^{(32-N)}}个IPv4地址与CIDR地址块的前缀匹配。前缀越短就能匹配越多的地址，越长就匹配得越少。一个地址可能与多个长度不同的CIDR前缀匹配。CIDR也用在IPv6中。因为位数的非常多，所以在IPv6中，前缀长度的范围是从0到128。这里也用同样的方法来表示一个地址：前缀写作一个IPv6的地址，后跟一个斜线，最后是前缀的位数。

1. 重命名插件

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58

   export function renameAppPlugin(str?: string): HvigorPlugin { return { pluginId: 'RenameAppPlugin', apply(currentNode: HvigorNode) { const appContext = currentNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext const appJson = appContext.getAppJsonOpt() const appVersion = appJson['app']['versionName'] console.log('RenameAppPlugin apply start, appVersion:', appVersion) const product = appContext.getCurrentProduct() const productName = product.getProductName() const bundleType = product.getBundleType() console.log('RenameAppPlugin apply start, productName:', productName, bundleType, product.getBundleName()) currentNode.registerTask({ // 编写自定义任务 name: 'renameAppTask', run: (taskContext) => { console.log('renameHarTask run start', taskContext.moduleName, taskContext.modulePath) const formatZero = (d: number): string => { return d < 10 ? `0${d}` : `${d}` } const formatDate = (): string => { const date = new Date() return `${date.getFullYear()}${formatZero(date.getMonth() + 1)}${formatZero(date.getDate())}${formatZero(date.getHours())}${formatZero(date.getMinutes())}` } const rename = (signType: string) => { const outputPath = `/build/outputs/${productName}/` const sourceName = `${taskContext.moduleName}-${productName}-${signType}.${bundleType}` const sourceFile = taskContext.modulePath + outputPath + sourceName const targetPath = taskContext.modulePath + outputPath const targetName = `${taskContext.moduleName}-${productName}-${appVersion}-${formatDate()}-${signType}.${bundleType}` const targetFile = targetPath + targetName console.log('renameAppTask sourceFile:', sourceFile) console.log('renameAppTask targetFile:', targetFile) // 创建目录 fs.mkdir(targetPath, { recursive: true }, (err) => { if (err) { console.log('renameAppTask mkdir err: ' + err) } fs.rename(sourceFile, targetFile, (err) => { console.log('renameAppTask rename err: ' + err) }) }) } rename('signed') rename('unsigned') }, dependencies: ['PackageApp'], postDependencies: ['assembleApp'] }) } } }

2. 输出重命名

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

   hvigor.getRootNode().afterNodeEvaluate(rootNode => { // 获取app插件的上下文对象 const appContext = rootNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext; const projectName = appContext.getProjectName() const product = appContext.getCurrentProduct() const productName = product.getProductName() const appJson = appContext.getAppJsonOpt(); const appVersion = appJson['app']['versionName'] console.log('renameAppPlugin getRootNode, appJson:', appJson) console.log('renameAppPlugin getRootNode, productName:', productName, projectName) const buildProfileJson = appContext.getBuildProfileOpt(); // console.log('renameAppPlugin getRootNode start, buildProfileJson:', buildProfileJson) const productsInBuildProfile = buildProfileJson['app']['products'] console.log('renameAppPlugin getRootNode, productsInBuildProfile:', productsInBuildProfile) const formatZero = (d: number): string => { return d < 10 ? `0${d}` : `${d}` } const formatDate = (): string => { const date = new Date() return `${date.getFullYear()}${formatZero(date.getMonth() + 1)}${formatZero(date.getDate())}${formatZero(date.getHours())}${formatZero(date.getMinutes())}` } for (const product of productsInBuildProfile) { if (!product['output']) { product['output'] = { 'artifactName': '' } } product['output']['artifactName'] += `${projectName}-${productName}-${appVersion}-${formatDate()}` console.log('renameAppPlugin getRootNode, output name:', product['output']['artifactName']) } console.log('renameAppPlugin getRootNode, is assembleApp task:', hvigor.isCommandEntryTask('assembleApp'), hvigor.getCommandEntryTask()); if (hvigor.isCommandEntryTask('assembleApp') && productName === 'prod') { const foundProduct = productsInBuildProfile.find(product => product['name'] === 'prod') if (foundProduct) { foundProduct['signingConfig'] = 'prod' appContext.setBuildProfileOpt(buildProfileJson) console.log('renameAppPlugin assemble prod app, change to prod sign', foundProduct) } } })

3. 动态添加依赖

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

   // https://developer.huawei.com/consumer/cn/doc/harmonyos-guides-V5/ide-build-expanding-context-V5 // function jfzDependPlugin(): HvigorPlugin { // return { // pluginId: 'jfzDependPlugin', // async apply(currentNode: HvigorNode): Promise<void> { // console.log('jfzDependPlugin', currentNode) // const appContext = currentNode.getContext(OhosPluginId.OHOS_APP_PLUGIN) as OhosAppContext; // console.log('jfzDependPlugin appContext', appContext) // const dependency = appContext.getDependenciesOpt({}); //获取dependency依赖 // // dependency["library"]="file:library.har" // console.log(dependency); // appContext.setDependenciesOpt(dependency ) //修改dependency依赖 // } // } // }

1. TS中的问号 ? 与感叹号 ! 是什么意思
   https://www.jianshu.com/p/4160800f9948

https://doc.qt.io/qtforpython-6.5/tutorials/debugging/vscode/vscode.html

1. No rule to make target pyconfig.h', needed by Programs/python.o’
   a. 查看 diff 命令指向，如which diff
   如果输出为：$HOME/Library/OpenHarmony/Sdk/12/toolchains/diff这个，会导致编译失败
   把diff指向系统的diff，如：/usr/bin/diff
   b. 删除上次缓存，python-build*目录

http://www.52im.net/thread-2172-1-1.html