工具Tools
監控儀表板怎麼看Reading the monitoring dashboard
團隊有一個監控儀表板(CLP Monitor),集中顯示 SW 與 Ava 各環境的機器數據:服務有沒有在跑、佇列有沒有積壓、資料庫和快取吃不吃緊、簡訊供應商的帳號狀態等。它是唯讀看板,不會改動任何系統,放心點、放心切。網址和登入方式請向主管索取。The team has a monitoring dashboard (CLP Monitor) that gathers machine data for every SW and Ava environment: whether services are running, whether queues are backing up, how stressed the database and cache are, the SMS provider's account status, and more. It's a read-only board — it changes nothing, so click around freely. Ask your manager for the address and sign-in.
這頁是寫給不是工程師的人(PM、QA、新人)看的。你不需要懂每個技術名詞,只要學會回答三個問題:這個紅色是真的有事嗎?、客戶會不會感覺到?、要不要通知人?This page is written for non-engineers (PMs, QA, newcomers). You don't need every technical term — just learn to answer three questions: is this red real?, will customers feel it?, should I tell someone?
1. 最重要的一件事:紅色「異常」標籤只看「最後一個點」1. The one thing to know: the red label only checks the last point
每張圖標題前的小標籤(正常 / 注意 / 異常),只看圖最右邊那一個點有沒有碰到紅線。它不知道現在是不是每天固定的忙碌時段,也不管這個狀況持續了多久。所以每天中午到傍晚,常常會一整排紅,大部分是正常的。標籤是「提醒你看一下」,不是「已經出事」。The small label before each chart title (Normal / Caution / Alarm) only checks whether the rightmost point touches the red line. It doesn't know whether this is the daily busy period or how long the situation has lasted. So from noon to early evening a whole row of charts often goes red, and most of it is normal. The label means "take a look", not "something broke".
實例:為什麼下午三點這三張圖都是「異常」?
・Horizon 機器數、ECS 服務容量:綠線在最右邊衝到 30、碰到紅色虛線「max 30」。30 是最多能開的台數,碰到上限就被標紅。但看整張圖:前一天下午也是 30、晚上慢慢降回個位數 —— 這是每天中午固定把機器開滿,準備處理每日排程的大量工作,是安排好的。
・Trigger 佇列等待:綠線在最右邊衝到 700–800 秒,超過紅線 300 秒。這是「每日工作排隊要等多久」,每天中午開跑後排隊時間會拉長。看最左邊:前一天同一時間也衝到快 1,000 秒,一小時內就降回 0。
結論:三張都是每天的固定型態,不用處理。
一天的節奏長這樣(台灣時間,以 2026-10 上旬實際資料為例):約 11:30 機器先開到上限(先準備好人手)→ 13:30 起工作開始大量進來 → 15:00 左右排隊最多 → 16:15 左右排完 → 約 17:30 才開始慢慢減少機器,一路降到晚上 22:30 左右剩個位數。所以白線會在高處停將近 6 小時,中間真正忙的是 13:30–16:15。這是安排好的,不是機器降不下來。
⏰ 這個排程是照美國時間設定的,美國夏令時間結束後(11 月初起),整段在台灣時間會晚 1 小時,不要以為是異常。
要擔心的是:過了傍晚還不降,或是比前一天同時段高很多、而且還在往上爬。Example: why are these three charts "Alarm" at 3 pm?
・Horizon machines, ECS service capacity: the green line at the right edge reaches 30 and touches the red dashed "max 30". 30 is the most machines allowed, so touching it turns the label red. But look at the whole chart: yesterday afternoon it was also 30 and drifted down to single digits overnight — machines are scaled to the maximum every day at noon to handle the daily batch, on purpose.
・Trigger queue wait: the green line at the right edge reaches 700–800 seconds, above the 300-second red line. It's "how long daily jobs wait in line", which grows after the noon kickoff. Look at the left edge: yesterday at the same time it hit nearly 1,000 seconds and was back to 0 within an hour.
Verdict: all three are the normal daily pattern — nothing to do.
A typical day (Taipei time, from early October 2026 data): around 11:30 machines go to the maximum (staffing up in advance) → work pours in from 13:30 → the line peaks around 15:00 → it's cleared around 16:15 → machines only start scaling down around 17:30, reaching single digits around 22:30. So the white line sits high for almost 6 hours, with the real rush between 13:30 and 16:15. This is planned — not machines failing to scale down.
⏰ The schedule follows US time, so after US daylight saving ends (early November) the whole pattern shifts 1 hour later in Taipei time — that's not an anomaly.
Worry if it doesn't come down after the early evening, or is much higher than yesterday at the same time and still climbing.
小技巧:點圖例(圖上方的色塊名稱)可以隱藏 / 顯示那條線,有刪除線的就是被隱藏了 —— 線很多時,一次只留要比的兩條最清楚。滑鼠停在標籤上,會顯示它是因為哪條線、哪個數字被判紅;停在圖上,會顯示那個時間點的台灣 / 美西時間和數值。把時間範圍切到 24h 或 3d,就能拿今天跟前幾天同一時段比。Tip: click a legend item (the colored names above the chart) to hide / show that line — struck-through means hidden; with many lines, keep just the two you're comparing. Hover the label to see which line and number made it red; hover the chart for the Taipei / US Pacific time and value at that point. Switch the range to 24h or 3d to compare today with the same time on previous days.
2. 看到紅色時,問自己三個問題2. When you see red, ask three questions
flowchart TD
A["看到紅色標籤"] --> Q1{"① 現在是不是<br>每天的忙碌時段?<br>(圖上淡灰底色)"}
Q1 -->|是| Q1b{"前幾天同一時段<br>也長這樣嗎?"}
Q1b -->|是| OK["固定型態,不用處理<br>傍晚再看一次有沒有降"]
Q1b -->|比平常高很多| W["值得注意 → 第 6 節"]
Q1 -->|不是| Q2{"② 是一下子<br>還是一直這樣?"}
Q2 -->|一下子就回來| OK2["短暫波動,不用處理"]
Q2 -->|持續 15 分鐘以上| Q3{"③ 是第 3 節<br>「客戶會感覺到」的那幾種嗎?"}
Q3 -->|是| NOW["立刻通報 → 第 6 節"]
Q3 -->|不是 / 不確定| W
flowchart TD
A["See a red label"] --> Q1{"① Is it the daily<br>busy period?<br>(light gray shading)"}
Q1 -->|yes| Q1b{"Did previous days<br>look the same then?"}
Q1b -->|yes| OK["Normal pattern, nothing to do<br>check again in the evening"]
Q1b -->|much higher than usual| W["Worth flagging → section 6"]
Q1 -->|no| Q2{"② A blip<br>or ongoing?"}
Q2 -->|recovers quickly| OK2["Brief wobble, nothing to do"]
Q2 -->|15+ minutes| Q3{"③ One of section 3's<br>'customers will feel it'?"}
Q3 -->|yes| NOW["Report now → section 6"]
Q3 -->|no / unsure| W
文字版。① 現在是每日忙碌時段(淡灰底色,約台灣 12:00–17:00)嗎?是的話,跟前幾天同時段比,一樣就沒事。② 不是忙碌時段的話,是一下子就回來,還是持續 15 分鐘以上?一下子的不用管。③ 持續的話,對照第 3 節,是「客戶會感覺到」的就立刻通報,其他的整理好再回報。Text version. ① Is it the daily busy period (light gray, about 12:00–17:00 Taipei)? If so, compare with the same time on previous days — the same means fine. ② If not, is it a blip or 15+ minutes? Ignore blips. ③ If ongoing, check section 3: if customers will feel it, report immediately; otherwise write it up and report.
| 圖上的底色Chart shading | 意思Meaning |
|---|---|
| 淡灰(約台灣 12:00–17:00)Light gray (about 12:00–17:00 Taipei) | 每日忙碌時段:系統在跑每天固定的大量工作(美國那邊是半夜)。數字衝高很正常,客戶通常感覺不到。Daily busy period: the system runs its big daily jobs (it's the middle of the night in the US). High numbers are normal; customers usually don't notice. |
| 深灰(約台灣 22:00–翌日 06:00)Dark gray (about 22:00–06:00 Taipei) | 美國上班時間:客戶正在用。這段出問題客戶會直接感覺到,要特別留意。US business hours: customers are using it. Problems here are felt directly — watch closely. |
| 頂部狀態帶的空白格Blank cells in the top status band | 那 5 分鐘沒收到資料。不是「沒事」,是「不知道」。一大段空白要回報。No data for those 5 minutes. Not "fine" — "unknown". Report a long blank stretch. |
3. 客戶會感覺到的:看到就立刻通報3. Customers will feel it: report immediately
| 圖上看到On the chart | 白話In plain words | 客戶會怎樣What customers see |
|---|---|---|
| ALB Target 健康度的 healthy 掉到 0ALB target healthy drops to 0 | 能接客的網站機器一台都沒有No web machines able to serve | 網站 / 系統打不開The site / system won't load |
| ALB 的 p95 持續超過 3 秒(15 分鐘以上,單根尖峰不算),或 5xx 暴增且同時 healthy 掉到 0ALB p95 over 3 s for 15+ minutes (single spikes don't count), or a 5xx surge while healthy drops to 0 | 大部分人的頁面要等 3 秒以上;或系統大量回錯誤Most page loads take 3+ seconds, or the system returns lots of errors | 很慢、按了沒反應、出現錯誤頁Very slow, clicks do nothing, error pages |
| ECS 服務容量:desired 和 running 兩條線一直分開、合不起來ECS capacity: desired and running stay apart | 「想要幾台」和「實際有幾台」對不上 —— 機器開不起來"Wanted" and "actually running" don't match — machines won't start | 越來越慢,最後可能整個停擺(曾經因此中斷好幾個小時)Slower and slower, possibly a full outage (this once lasted hours) |
| 非忙碌時段,Queue Wait 的 high / highest 超過 300 秒紅線Outside the busy period, high / highest queue wait over the 300 s line | 緊急的工作排隊超過 5 分鐘,可能開始有工作逾時失敗(對照「錯誤計數」的佇列逾時確認)Urgent jobs wait over 5 minutes; jobs may start timing out (confirm with queue timeouts in Error Counts) | 簡訊、回覆、通話延遲或沒發出去Texts, replies or calls delayed or never sent |
| 客戶流程 Flow:交接處一邊變 0、另一邊還在動;或即時觸發(realtime)有積壓Flow: one side of a handoff at 0 while the other moves; or a realtime trigger backlog | SW 送出去了但 Ava 沒收到(或反過來);即時的工作卡住SW sent it but Ava didn't receive it (or vice versa); realtime work stuck | 新名單進來沒有收到第一則訊息New leads never get the first message |
| Twilio:出現 32005 / 10005 或 disabled = yesTwilio: 32005 / 10005 or disabled = yes | 電信商把某個帳號的打電話功能關掉了(就算標的是 warning 也是真的)The carrier turned off calling for an account (real even though it's labeled warning) | 那個帳號的客戶打不出 / 接不到電話That account can't place or receive calls |
4. 值得回報、但不用半夜叫人的4. Worth reporting, no need to wake anyone
- 忙碌時段過了(傍晚以後)數字還不降,或比前幾天同時段高很多、還在爬。Numbers still high after the busy period (evening), or much higher than previous days and still climbing.
- Trigger 佇列長度一直往上漲,但旁邊的 worker 數沒有跟著增加 —— 代表系統沒有自動加人手。這個曾經導致暫存區被塞爆、整個系統出問題,如果在中午到傍晚看到,請不要等。Trigger queue length keeps rising while worker count stays flat — the system isn't adding hands automatically. This once filled the cache and broke things; if you see it between noon and evening, don't wait.
- 資料庫的死鎖(deadlock)大於 0:兩件工作互相卡住,其中一件被系統取消了 —— 代表有工作失敗。Database deadlocks above 0: two jobs blocked each other and one was cancelled — a job failed.
- 錯誤計數裡的 blacklist 失敗大於 0:勿擾名單(DNC)檢查沒做到,有合規風險。Blacklist failures above 0 in error counts: the do-not-contact (DNC) check didn't run — a compliance risk.
- 搜尋引擎(Elasticsearch)狀態變 red,或寫入量掉到接近 0、但搜尋量還正常:資料寫不進去,查得到的會是舊資料。Search engine (Elasticsearch) turns red, or writes drop near 0 while searches continue: data isn't being saved, so lookups show stale data.
- 頂部狀態帶一大段空白:監控本身沒收到資料,我們看不到系統狀況。A long blank stretch in the status band: the monitor itself got no data, so we're blind.
5. 常見的「看起來很嚇人,其實沒事」5. Looks scary, actually fine
| 你看到的What you see | 為什麼沒事Why it's fine |
|---|---|
| 中午開始機器數衝到上限(30)、各種線一起衝高Machines hit the max (30) from noon and everything spikes | 每天固定把機器開滿處理每日工作(見第 1 節實例)Machines are maxed every day for the daily batch (see section 1) |
| Trigger 佇列等待:每天下午 daily wait 衝過「300s job timeout」紅線,最高到 1,000 秒左右Trigger queue wait: every afternoon daily wait crosses the "300s job timeout" line, peaking around 1,000 s | 每日工作排隊的固定型態(約 14:00–16:00 超過紅線,16:00 左右回到 0)。紅線名字叫「job timeout」容易嚇到人,但實際對照近 3 天的資料,這段時間「錯誤計數」裡的佇列逾時是 0,沒有看到工作因此失敗。要擔心的是傍晚以後還降不回 0,或同時「錯誤計數」裡的佇列逾時明顯變多。The daily batch queue's normal pattern (over the line about 14:00–16:00, back to 0 around 16:00). The line is named "job timeout", which sounds scary, but over the last 3 days queue timeouts in Error Counts were 0 during this period — no jobs were seen failing from it. Worry if it isn't back to 0 by evening, or queue timeouts in Error Counts rise at the same time. |
| RDS Serverless v2 reader:ACU% 永遠是 100%、永遠「異常」RDS Serverless v2 reader: ACU% always 100%, always "Alarm" | 這是資料庫的「唯讀分身」。它被固定在一個容量,這個百分比的算法讓它永遠顯示 100%,所以標籤永遠是紅的 —— 這張圖不能拿來判斷健康(圖的說明也寫了「別再拿它判斷健康」)。近 3 天的資料是一條完全平的線,實際查詢量幾乎是 0,客戶不受影響。工程師已在 2026-10-05 結案追蹤,看到它紅可以忽略。This is the database's read-only copy. It's pinned at one capacity, and the way this percentage is computed makes it always show 100%, so the label is always red — this chart can't be used to judge health (its own note says so). Three days of data are a perfectly flat line with almost no queries; customers aren't affected. Engineers closed this out on 2026-10-05 — ignore the red. |
| ALB p95 回應時間:標籤常在「注意」和「異常」之間跳,線大多在 1–3 秒、三不五時冒一根超過 3 秒ALB p95 response time: the label flips between Caution and Alarm; the line mostly sits at 1–3 s with occasional spikes over 3 s | 這是過去一個月的常態(2026-09 至 10 月初:每天約一半以上的時間高於 1 秒「正常線」、約 2–10% 的時間單點超過 3 秒)。標籤只看最後一點,所以剛好落在尖峰上就會變紅、下一輪又回到「注意」。要看的是「突然比平常高很多,而且持續」,例如整段停在 3 秒以上 15 分鐘以上。 ⚠️ 但 5xx 出現高柱時不要忽略:對照同時段「ALB Target 健康度」,如果 healthy 掉到 0,就是真的對外中斷,即使幾分鐘就恢復也要回報。This has been normal for the past month (Sept to early Oct 2026: more than half of each day above the 1 s "normal" line, single points over 3 s about 2–10% of the time). The label only checks the last point, so landing on a spike turns it red and the next refresh brings back Caution. What matters is "suddenly much higher than usual and sustained" — e.g. staying above 3 s for 15+ minutes. ⚠️ But don't ignore a tall 5xx bar: check ALB Target health at the same time — if healthy dropped to 0, it was a real outage; report it even if it recovered within minutes. |
| 「daily delayed」有幾百萬筆"daily delayed" in the millions | 那是預約在未來才做的工作,不是塞車。圖的標題也寫著「幾百萬是常態」Those are scheduled for later, not a jam. The chart title even says millions is normal |
| 每個整點某條線跳一下A line blips every hour on the hour | 整點固定有資料進來,幾分鐘內就消化完Data arrives every hour and clears within minutes |
| ECS 服務容量:某個時段兩條線一直合不起來ECS service capacity: two lines never meet during some period | 先看是不是比錯線。這張圖同時畫了兩個服務(horizon、backend),各有 desired / running 共 4 條線,顏色很接近。要比的是同一個服務的兩條:horizon desired ↔ horizon running、backend desired ↔ backend running。拿 horizon running 去比 backend desired,本來就不會一樣(一個是背景工人、一個是網站機器)。圖例上有刪除線的代表被點掉隱藏了,再點一次就會出現。實際查過近 3 天的資料,同一服務的兩條線最長只差約 3 分鐘(擴縮過渡)。First check whether you're comparing the wrong lines. This chart draws two services (horizon, backend), each with desired / running — four similar-colored lines. Compare within one service: horizon desired ↔ horizon running, backend desired ↔ backend running. Horizon running vs backend desired will never match (background workers vs web machines). A struck-through legend item is hidden; click it to show it again. Three days of data checked: within one service the lines differ for at most about 3 minutes (scaling transition). |
| 有工程師在部署時,線短暫分開、亮一下紅Lines briefly split or go red during a deploy | 換版本的過渡期,幾分鐘就回來Version-switch transition; back within minutes |
| 圖上寫「Maximum」的那條線越過紅線A line labeled "Maximum" crosses the red line | 紅線是給「Average 平均」用的;Maximum 是瞬間最高值,本來就比較高Red lines are for the Average; Maximum is an instantaneous peak and naturally higher |
| STG 頁很吵、常有尖峰STG is noisy with spikes | STG 是測試環境,通常是有人在測。客戶看不到STG is the test environment — usually someone testing. Customers never see it |
| Ava 頁某些錯誤線永遠是 0(例如 blacklist)Some error lines always 0 on Ava tabs (e.g. blacklist) | 那是 SW 才有的功能,Ava 本來就沒有這筆資料。不代表 Ava 合規沒問題That's an SW-only feature, so Ava has no data. It does not mean Ava compliance is fine |
| Datadog 有一堆「No Data」Lots of "No Data" in Datadog | 大多是被暫停的檢查,根本沒在跑Mostly paused checks that aren't running at all |
| Fin 今天的金額寫 Estimated、昨天的還在變Fin shows today as Estimated, yesterday still changing | 雲端帳單隔天才結算Cloud bills settle a day later |
| 很久以前的圖是空的Charts empty far back | 那時還沒開始收這個數據。空白是「不知道」,不是「當時很健康」Collection hadn't started yet. Blank means unknown, not healthy |
6. 有問題時怎麼做6. What to do when there's a problem
- 先照第 2 節的三個問題判斷,排除資料中斷、批次窗口、部署過渡。也可以到「排程」頁看那個時間有沒有已知排程。Go through section 2's three questions first to rule out data gaps, batch windows and deploy transitions. The Schedule tab shows whether something is planned at that time.
- 展開那張圖的「完整判讀」,很多圖都寫了下一步要看哪張圖一起判斷(例如容量線要配待啟動數、回應時間要配健康主機數)。Expand that chart's "Full notes" — many say which chart to read alongside (e.g. capacity with pending, response time with healthy targets).
- 截圖時把時間範圍和分頁名稱一起截進去,記下開始時間(台灣時間)和現在是否還在持續。Include the time range and tab name in the screenshot, and note the start time (Taipei) and whether it's still ongoing.
- 通報:(通報管道與聯絡人待補)Report it: (channel and contacts to be added)
不要自己動手修 production。儀表板是唯讀的;重開服務、調整容量、刪資料都由負責的工程師處理。你的任務是早一點發現、講清楚看到什麼。如果是客戶尖峰時段(深底色)出現對外中斷的跡象(健康主機數 = 0、回應時間 > 3 秒、5xx 暴增),請不要等,立刻通報。Don't try to fix production yourself. The dashboard is read-only; restarts, capacity changes and data deletion are handled by the responsible engineers. Your job is to spot it early and describe clearly what you saw. If there are signs of a public outage during the customer peak (dark shading) — healthy targets = 0, response time > 3 s, a 5xx surge — don't wait; report immediately.
看不懂也是有價值的回報。這個儀表板還在持續改進,如果某個名詞、某張圖看不懂,記下來回報給維護的人 —— 它直接告訴團隊哪裡標示得不夠清楚。"I don't understand this" is a valuable report too. The dashboard is still evolving; if a term or chart makes no sense, note it and tell the maintainers — it shows the team exactly where the labeling falls short.
名詞白話對照Plain-language glossary
| 圖上寫的On the chart | 白話Plain meaning |
|---|---|
| desired / running | 系統「想要」開幾台機器 / 「實際」開了幾台。兩條線疊在一起 = 正常How many machines the system wants / actually has. Overlapping = normal |
| task、worker processes、Horizon | 幫忙在背景做事的機器 / 工人(發簡訊、處理名單…)Machines / workers doing background work (sending texts, processing lists…) |
| max 30(紅色虛線) | 最多允許開幾台The most machines allowed |
| Queue(佇列)/ Wait / Length | 待辦工作的排隊隊伍 / 要排多久 / 隊伍多長The line of pending work / how long the wait is / how long the line is |
| delayed | 預約在未來才要做的工作(不算排隊)Work scheduled for later (not in line yet) |
| Trigger(daily / realtime) | 依 Segment 規則觸發的工作;daily = 每天固定跑一次,realtime = 名單一進來就跑Work triggered by Segment rules; daily = once a day, realtime = as soon as a contact arrives |
| Redis | 暫存區,放排隊中的工作和快取資料。塞滿了工作會寫不進去Short-term storage for queued work and cached data. If full, work can't be added |
| RDS | 資料庫,存聯絡人、對話等正式資料The database holding contacts, conversations and other real data |
| Elasticsearch(ES) | 搜尋引擎,Contacts / Actions 頁的查詢靠它The search engine behind Contacts / Actions lookups |
| ALB、healthy / unhealthy | 網站的總機,把使用者分配到網站機器上 / 能接客的、不能接客的機器數The site's switchboard that routes users to web machines / how many can or can't serve |
| p95 | 「100 個人裡第 95 慢的人」等了多久 —— 代表大多數人的感受How long the 95th-slowest of 100 users waited — what most people feel |
| 5xx | 系統內部出錯的回應Responses where the system itself errored |
| Average / Maximum | 平均值 / 瞬間最高值。紅線看 AverageAverage / instantaneous peak. Red lines apply to the Average |
| CLO / CL1 / STG | 外部正式環境(真客戶)/ 內部正式環境 / 測試環境External production (real customers) / internal production / staging |
分頁導覽:要看什麼去哪一頁Which tab for what
| 分頁Tab | 看什麼What it shows |
|---|---|
| CLO / CL1 | SW 的外部 production / 內部 production:服務容量、佇列、快取、資料庫、網站回應、搜尋引擎SW external / internal production: service capacity, queues, cache, database, site response, search engine |
| AVA CLO / AVA CL1 | Ava 的外部 / 內部 production,同一套圖Ava external / internal production, same set of charts |
| STG | Staging。流量來自 QA 和開發,不是客戶Staging. Traffic comes from QA and developers, not customers |
| Twilio | 簡訊 / 語音供應商所有帳號的狀態:語音有沒有被關、告警代碼、通話失敗數All SMS / voice provider accounts: voice disabled or not, alert codes, failed calls |
| Datadog | 各告警監控的狀態總表,以及應用錯誤、5xx、逾時工作的摘要Status of all alert monitors, plus summaries of app errors, 5xx and timed-out jobs |
| 客戶流程 FlowFlow | 「建立聯絡人」這條主流程從進來到送出,每一段有沒有在動Whether each stage of the main "create contact" flow is moving, from intake to output |
| 排程 Schedule、維護時段 HoursSchedule, Hours | 每天固定什麼時候有批次、擴縮、備份 —— 判斷「這個尖峰是不是排程造成的」先來這裡;Hours 則建議哪個時段適合維護When batches, scaling and backups happen each day — check here first when asking "is this spike just a schedule?"; Hours suggests maintenance windows |
| 差異 DiffDiff | 各環境的資料庫、索引、版本、設定並排比較。紅 = 型別衝突、部署會出事;黃 = 某邊有某邊沒有Side-by-side DB, index, version and config across environments. Red = type conflict that will break a deploy; yellow = present on one side only |
| 系統資訊、架構、資料、Fin、待辦System, Architecture, Data, Fin, TODO | 裝置盤點、整體架構圖(新人了解系統很好用)、資料容量與備份、雲端成本、團隊待辦。日常判斷健康度用不太到Inventory, an architecture map (great for newcomers), storage and backups, cloud cost, team to-dos. Rarely needed for day-to-day health checks |
| 問診 ChatChat | 勾選資料來源後用文字問「現在狀況如何」。模型只看得到勾選的來源 ——回答出現你沒勾的數字,就是它在編。勾越多越慢,等 1–5 分鐘是正常的Pick data sources, then ask "how are things?" The model only sees what you ticked — a number from a source you didn't tick is made up. More sources = slower; 1–5 minutes is normal |
進階:給工程師的完整判讀對照(含資料庫、快取、搜尋引擎細項)Advanced: full reading guide for engineers (database, cache, search-engine details)
共通元素Shared elements
| 元素Element | 怎麼讀How to read it |
|---|---|
| 頂部狀態帶Top status band | 過去 24 小時,每格 5 分鐘。 ・空白格 = 那 5 分鐘沒收到資料,是「記錄中斷」,不是「沒事」。 ・琥珀 = 重要佇列開始積壓;紅 = 積壓超過工作逾時,或關鍵服務掉到 0 台(依分頁不同)。 ・點任一格,下面的圖會跳到那個時段前後各 30 分鐘。Last 24 hours, one cell per 5 minutes. ・A blank cell = no data arrived in those 5 minutes — a recording gap, not "all clear". ・Amber = an important queue is starting to back up; red = backlog past the job timeout, or a key service down to 0 (varies by tab). ・Click a cell to jump the charts to 30 minutes either side. |
| 圖上的灰色底色Gray shading on charts | 只是時間脈絡,不是健康狀態(所以刻意不上彩色): ・淡底色 = 每日批次窗口(約台灣 12:00–17:00):每日的大量排程工作在這時跑,各種數字衝高是常態,客戶通常無感。 ・深底色 = 美國白天客戶尖峰(約台灣 22:00–翌日 06:00):客戶有感的時段,這時要特別盯「服務有沒有在跑」和「網站回應時間 / 錯誤」。Time context only, not health (deliberately uncolored): ・Light shading = daily batch window (about 12:00–17:00 Taipei): the big daily scheduled jobs run now, so spikes are normal and customers usually don't notice. ・Dark shading = US daytime peak (about 22:00–06:00 Taipei): customers feel problems here — watch "are services running" and "site response time / errors" closely. |
| 每張圖的狀態標籤Each chart's status pill | 「讀取中」= 還在載;變成彩色(例如「注意」)才是要看的;「無門檻」= 這張圖還沒量出可靠的基線,刻意不畫門檻線 —— 看形狀和趨勢,不要自己套數字。"Loading" = still loading; colored (e.g. "Caution") = look at it; "No threshold" = no reliable baseline yet, so no line is drawn on purpose — read the shape and trend; don't invent a number. |
時間軸是台灣時間(滑鼠停在圖上可看美西時間),頁面每 30 秒自動更新。上方可切 1 小時到 7 天的範圍。The time axis is Taipei time (hover for US Pacific); the page refreshes every 30 seconds. Choose a range from 1 hour to 7 days at the top.
逐項:真的有問題的型態Item by item: real problem patterns
| 看哪裡Where | 正常Normal | 有問題的型態Problem pattern |
|---|---|---|
| 服務容量(要幾台 vs 實際幾台)Service capacity (wanted vs running) | 兩條線重疊;短暫分開是擴縮中Lines overlap; brief gaps while scaling | 兩條線持續分開、不收斂 = 擴不起來。曾經造成數小時的對外中斷Lines stay apart and don't converge = can't scale; this once caused an hours-long outage |
| 待啟動數(Pending)Pending tasks | 貼著 0;部署時偶爾冒一下At 0; occasional blips during deploys | 持續 > 0 而實際台數不動 = 服務一直起不來Sustained > 0 while running stays flat = services keep failing to start |
| 佇列等待時間(高優先)Queue wait (high priority) | 接近 0 秒Near 0 seconds | > 1 分鐘開始積壓;> 5 分鐘會超過工作逾時,工作直接失敗> 1 minute = backing up; > 5 minutes exceeds the job timeout and jobs fail |
| 每日 trigger 佇列 + 工作台數Daily trigger queue + workers | 每天中午衝高,約一小時內排掉Spikes around noon, drains within about an hour | 長度一路墊高,但工作台數沒跟著上去 = 自動擴縮沒觸發,要人工介入。曾經因此把快取塞爆Length keeps climbing while worker count stays flat = autoscaling didn't fire; needs a human. This once filled the cache completely |
| 快取記憶體Cache memory | < 70%;觸頂但「逐出數」同步上升 = 正常回收< 70%; at the top with evictions rising too = normal recycling | ≥ 95% 但逐出數沒跟上 = 塞滿了清不掉的東西,寫入會直接失敗;多個節點的線分岔 = 複寫異常≥ 95% with evictions not rising = filled with things that can't be evicted, writes will fail; node lines diverging = replication trouble |
| 資料庫 CPU / 負載Database CPU / load | 平均 < 70%(批次時偏高屬常態)Average < 70% (higher during batches is normal) | 平均 > 85% 持續;「每顆 CPU 的活躍連線」平均持續 ≥ 1.0 = 吃滿Average > 85% sustained; "active sessions per vCPU" average sustained ≥ 1.0 = saturated |
| 資料庫死鎖Database deadlocks | 0 | > 0 就是事故:交易已被回滾,代表有工作失敗了Any > 0 is an incident: a transaction was rolled back, so a job failed |
| 資料庫複寫停止旗標Replication-stopped flag | 0 | = 1 持續 10 分鐘(短暫一下見第 5 節)= 1 for 10 minutes (brief blips: see section 5) |
| 網站健康主機數Healthy web targets | 健康數 = 執行中數,不健康 = 0Healthy = running, unhealthy = 0 | 健康數 = 0 = 對外中斷;健康數持續少於執行中數Healthy = 0 = public outage; healthy persistently below running |
| 網站回應時間 / 5xxSite response time / 5xx | p95 < 1 秒p95 < 1 s | 1–3 秒劣化;> 3 秒嚴重;5xx 每 5 分鐘明顯變多1–3 s degraded; > 3 s severe; 5xx clearly rising per 5 minutes |
| 錯誤計數Error counts | 佇列逾時維持在日常背景值Queue timeouts at the usual background level | 佇列逾時明顯上升;黑名單(DNC)檢查失敗 > 0 = 合規風險Queue timeouts clearly rising; blacklist (DNC) check failures > 0 = compliance risk |
| 搜尋引擎(ES)Search engine (ES) | 叢集綠燈、拒絕數 0、4xx/5xx 平穩Cluster green, rejections 0, 4xx/5xx flat | red = 硬故障;拒絕數 > 0 = 已飽和;4xx/5xx 連續兩格越線(這組響了就是真問題);寫入量掉到接近 0 但搜尋量還在 = 寫入路徑壞了;磁碟餘裕低於 10% 線red = hard failure; rejections > 0 = saturated; 4xx/5xx over the line two buckets in a row (when this fires it's real); writes near 0 while searches continue = write path broken; disk headroom under the 10% line |
| 主機狀態檢查Host status check | 0 | = 1 是機器層級故障,不是負載高,看到就要處理= 1 is a machine-level failure, not load — act on it |
| Twilio | 語音被關告警為 0、disabled = noVoice-disabled alerts 0, disabled = no | 出現「語音被關」告警(32005 / 10005)或 disabled = yes。注意 32005 本身等級是 warning,warning 不代表沒事Voice-disabled alerts (32005 / 10005) or disabled = yes. Note 32005 is itself a warning — warning doesn't mean fine |
| 客戶流程 FlowFlow | 每一段都有在動Every stage moving | 交接處一邊變 0、另一邊還在動;即時觸發佇列有積壓(客戶當下就受影響)One side of a handoff drops to 0 while the other keeps moving; backlog in the realtime trigger queue (customers affected right now) |
逐項:看起來異常但沒事Item by item: looks wrong but fine
| 你看到的What you see | 為什麼沒事Why it's fine |
|---|---|
| 淡底色時段裡各種數字衝高(台數、佇列、資料庫 CPU)Everything spikes inside the light shading (workers, queues, DB CPU) | 每日批次窗口的已知型態。要看的是窗口結束後有沒有回落Known daily batch pattern. What matters is whether it comes back down afterwards |
| 每天中午 trigger 佇列衝到很高Trigger queue very high around noon | 每日排程開跑,一小時內排掉就正常。工作台數上去後長度還漲一下也正常(每筆工作會再派出新工作)The daily schedule kicks off; draining within an hour is normal. A short rise even after workers scale up is also normal (each job spawns more) |
| 「延遲中 delayed」的工作有幾百萬筆Millions of "delayed" jobs | 那是排在未來執行的工作,不是積壓。真正的積壓看「等待中 waiting」Those are scheduled for later, not a backlog. Real backlog is "waiting" |
| 每個整點佇列跳一下A queue blip every hour on the hour | 整點的資料上傳 / 重新平衡,幾分鐘內清空屬預期Hourly uploads / rebalancing; clearing within minutes is expected |
| 部署時容量線短暫分開、待啟動數冒一下、健康主機數暫時變少During a deploy: capacity lines briefly apart, pending blips, fewer healthy targets | 滾動部署的過渡態。要看有沒有「持續」Rolling-deploy transition. Only "sustained" matters |
| 某條「Maximum」線越過門檻A "Maximum" line crosses the threshold | 門檻只對 Average 有效。Maximum 是瞬間最大值,天生比較高(例如搜尋引擎記憶體的 Max 天天摸到門檻)。這個誤判團隊踩過好幾次Thresholds apply to Average only. Maximum is an instantaneous peak and naturally higher (e.g. search-engine memory Max touches the line daily). The team has fallen for this several times |
| 資料庫 swap 大於 0Database swap above 0 | swap 是存量不是速率,大多數主機都 > 0。只有「swap 持續增加 + 可用記憶體很低」才是真壓力Swap is a level, not a rate; most hosts are > 0. Only "swap steadily growing + low free memory" is real pressure |
| 「被阻塞的交易」常常 > 0"Blocked transactions" often > 0 | 正常的等鎖。只有死鎖 > 0 才是事故Normal lock waits. Only deadlocks > 0 are incidents |
| 複寫停止旗標短暫出現 -1Replication flag briefly shows -1 | 每週例行維護時會出現,刻意設計成不告警Happens during weekly maintenance; deliberately doesn't alert |
| 某條告警線一直是 100%、一直響An alert line stuck at 100%, always firing | 有些比例指標的設計本來就會貼頂(圖下說明會寫「不要再拿它判斷健康」),改看旁邊的實際值Some ratio metrics sit at the top by design (the notes say "don't judge health by this"); read the actual value beside it |
| 低流量環境的回應時間、搜尋延遲忽高忽低Response time or search latency jumping around on low-traffic environments | 樣本少,一個慢請求就會讓線跳起來,不代表容量問題Small samples — one slow request moves the line; not a capacity issue |
| STG 的尖峰、錯誤很吵Spikes and noisy errors on STG | 通常是有人在測或灌測試資料。STG 的價值是部署前後對照:平常吵吵的曲線突然變成持續的高原,才是真的壞了Usually someone testing or loading data. STG's value is before/after deploy comparison: a noisy line turning into a sustained plateau is what's actually broken |
| 某些錯誤線在 Ava 頁永遠是 0(例如黑名單、佇列逾時)Some error lines always 0 on Ava tabs (e.g. blacklist, queue timeouts) | 那是 SW 才有的功能,在 Ava 是「無資料」。不要讀成「Ava 合規檢查正常」Those are SW-only features, so on Ava it's "no data". Don't read it as "Ava compliance is fine" |
| Datadog 有一堆「No Data」Lots of "No Data" in Datadog | 大多是標了「已暫停」的檢查 —— 檢查根本沒在跑,不是服務沒回應Mostly checks marked "paused" — the check isn't running, not the service failing to respond |
| Twilio 告警的「內容」欄空白、或只掃了少數帳號Twilio alert text blank, or only a few accounts scanned | 供應商本來就沒填說明;「focus 掃」只掃部分帳號,完整資料等下一輪(約 15 分鐘)。數字後面有「+」代表被截斷,不是確切值The provider leaves it empty; a "focus" scan covers only some accounts, the full one comes next round (~15 min). A "+" after a number means truncated, not exact |
| Flow 頁某些數字比預期小,或 CLO 是 CL1 的兩倍Flow numbers smaller than expected, or CLO double CL1 | 那是日誌事件數,不是筆數(一個工作可能帶很多聯絡人);排程類數字只證明排程還活著;兩台主機都跑排程會變兩倍。超過 5 分鐘沒更新的讀數會變黃底,不要當現在的值They're log-event counts, not record counts (one job may carry many contacts); schedule counts only prove the schedule is alive; two hosts running it doubles the number. Readings older than 5 minutes turn yellow — don't treat them as current |
| Fin 今天的成本是 Estimated、昨天的還會變Fin shows today as Estimated and yesterday still changing | 雲端帳單隔天才結算,本來就會變Cloud billing settles a day later; it's expected to change |
| 很早以前的圖是空的Charts empty far back in time | 很多指標是後來才開始收的。更早的空白是「盲區」,不是「當時健康」Many metrics started being collected later. Earlier blanks are blind spots, not "healthy back then" |