QA 工程師QA engineers
auto_cl 測試在做什麼What the auto_cl tests do
auto_cl 是 ContactLoop 的自動化測試:用程式開一個 Chromium 瀏覽器,像真人一樣登入 SW Manager 和 Ava Messenger、匯入聯絡人、收發簡訊、切換頁面,邊做邊檢查結果對不對;跑完自動產生報告、錄影,並發 Email 與 Teams 通知。用 Python + pytest + Playwright 寫的。怎麼安裝、怎麼跑,見 從 0 跑出第一套測試;這頁講「跑起來之後每一步在做什麼」。auto_cl is ContactLoop's automated test suite: it opens a Chromium browser and, like a person would, signs in to SW Manager and Ava Messenger, imports contacts, sends and reads texts, and switches pages — checking the results as it goes. When it finishes it writes a report and a video and sends Email and Teams notifications. It's written in Python with pytest and Playwright. For installing and running it see Run your first test suite; this page explains what each step does once it's running.
1. 整個程式怎麼運作1. How the whole run works
flowchart TD
A["執行 pytest --env=stg / cl1 / prod<br>(或 scripts/run_stg.sh 等)"] --> B["開場:清掉上一次留下的<br>問題紀錄檔"]
B --> C["排順序:test_ip 排最前、<br>test_ava_link 排最後,其他依檔名"]
C --> D["開一個 Chromium 瀏覽器<br>全程錄影 + 記錄所有網路請求"]
D --> T1["① test_ip<br>確認這台電腦目前的 IP"]
T1 --> T2["② test_api_metrics<br>API 效能統計工具的單元測試"]
T2 --> T3["③ test_sw_nucleus<br>SW 匯入 → 簡訊 → 模擬回覆 → Ava 對照"]
T3 --> T4["④ test_sw_page_api_checks<br>逐頁切換,檢查每頁打的 API"]
T4 --> T5["⑤ test_sw_segments<br>匯入前後 Segment 人數有沒有正確增加"]
T5 --> T6["⑥ test_ava_link<br>點 Ava 對話裡的連結,看打不打得開<br>(CircleCI 只在 prod 跑)"]
T6 --> E["關閉瀏覽器,存影片<br>產生 report.html / junit.xml / test.log"]
E --> F["收尾:整理 API 效能資料<br>(最慢的 5 支 API)"]
F --> G["歸檔到 runs/ 並上傳報告、log、影片"]
G --> H["寄 Email 結果通知"]
H --> I{"有設定 Teams?"}
I -->|有| J["發 Teams 通知"]
I -->|沒有| K["結束"]
J --> K
flowchart TD
A["Run pytest --env=stg / cl1 / prod<br>(or scripts/run_stg.sh etc.)"] --> B["Start: clear the previous<br>run's issue files"]
B --> C["Order: test_ip first,<br>test_ava_link last, others by file name"]
C --> D["Open one Chromium browser<br>recording video + all network requests"]
D --> T1["① test_ip<br>check this machine's IP"]
T1 --> T2["② test_api_metrics<br>unit tests for the API metrics tool"]
T2 --> T3["③ test_sw_nucleus<br>SW import → SMS → simulated replies → Ava check"]
T3 --> T4["④ test_sw_page_api_checks<br>switch pages, check each page's API"]
T4 --> T5["⑤ test_sw_segments<br>do segment counts rise correctly after import"]
T5 --> T6["⑥ test_ava_link<br>open links in Ava conversations<br>(CircleCI runs it on prod only)"]
T6 --> E["Close browser, save video<br>write report.html / junit.xml / test.log"]
E --> F["Wrap-up: summarize API timings<br>(5 slowest APIs)"]
F --> G["Archive to runs/ and upload report, log, video"]
G --> H["Send the result Email"]
H --> I{"Teams configured?"}
I -->|yes| J["Send Teams notification"]
I -->|no| K["Done"]
J --> K
文字版。執行 pytest --env=環境(預設 stg)後:先清掉上次的問題紀錄檔 → 把測試排好順序(test_ip 最先、test_ava_link 最後,中間其他檔依檔名排序)→ 開一個 Chromium 瀏覽器,全程錄影並記錄網路請求 → 依序執行 6 個測試檔(所有測試共用同一個瀏覽器分頁)→ 關閉瀏覽器並把影片存到 videos/,產生報告 → 整理 API 效能資料 → 歸檔到 runs/ 並上傳 → 寄 Email,有設定的話再發 Teams 通知。瀏覽器預設會顯示畫面,設定 HEADLESS=true 才在背景跑。Text version. After pytest --env=<env> (default stg): clear the previous issue files → order the tests (test_ip first, test_ava_link last, the rest by file name) → open one Chromium browser, recording video and network requests → run the 6 test files in order (all tests share one browser tab) → close the browser, save the video to videos/ and write the reports → summarize API timings → archive to runs/ and upload → send the Email, and a Teams notification if configured. The browser is visible by default; set HEADLESS=true to run it in the background.
CircleCI 上 stg、cl1 不跑 ⑥ test_ava_link(設定裡用 --ignore 排除),只有 prod 會跑,所以 prod 報告比另外兩個多 3 項。自己在本機跑 pytest --env=stg 則 6 個檔都會跑。On CircleCI, stg and cl1 skip ⑥ test_ava_link (excluded with --ignore); only prod runs it, so the prod report has 3 more rows. Running pytest --env=stg locally runs all 6 files.
README 寫的順序少了一個檔。README 列 5 個測試檔,但實際上 test_api_metrics.py(單元測試,不開瀏覽器)也會跑,依檔名排在 test_sw_nucleus 前面。The README's order is missing one file. It lists 5 test files, but test_api_metrics.py (unit tests, no browser) also runs, placed before test_sw_nucleus by file name.
2. 六個測試檔,依執行順序2. The six test files, in run order
| # | 檔案File | 在做什麼(步驟)What it does (steps) |
|---|---|---|
| ① | test_ip.py | 開一個查 IP 的網頁,把這台電腦目前對外的 IP 記進 log。用來確認從哪個網路在跑(有些環境會限制來源 IP)。Opens an IP-lookup page and logs this machine's public IP — to confirm which network the run comes from (some environments restrict source IPs). |
| ② | test_api_metrics.py | 「API 效能統計工具」本身的單元測試(網址正規化、平均與 P95 計算、存檔、讀取網路紀錄檔時的各種異常)。不開瀏覽器、不碰任何環境,幾秒就跑完。Unit tests for the API metrics tool itself (URL normalization, average and P95 math, saving, edge cases when reading the network log). No browser, no environment; done in seconds. |
| ③ | test_sw_nucleus.py | 最完整的端到端流程:1 SW 登入 → 2 選帳號 → 3 匯入一筆聯絡人 → 4 透過簡訊服務 Quo 讀簡訊,確認 SW 發出的第一則簡訊有送達(最多等約 5 分鐘)→ 5 模擬聯絡人連續回幾則訊息(從語句池隨機挑不重複的句子),每則都等到系統自動回覆才送下一則 → 6 Ava 登入 → 7 選帳號、打開那段對話、把訊息抓下來 → 8 比對:SW 的第一則訊息與模擬的回覆,在 Ava 對話頁上是不是都一字不差出現。The most complete end-to-end flow: 1 SW sign-in → 2 pick the account → 3 import a contact → 4 read texts via the Quo SMS service to confirm SW's first text arrived (up to ~5 minutes) → 5 simulate the contact replying several times (random, non-repeating lines from a pool), waiting for each automatic reply before the next → 6 Ava sign-in → 7 pick the account, open that conversation, capture the messages → 8 compare: do SW's first message and the simulated replies all appear verbatim on the Ava conversation page. |
| ④ | test_sw_page_api_checks.py | 登入 SW、選帳號,然後一頁一頁切換 Contacts、Actions、Reports 等核心頁面,攔截每頁載入時打出去的 API,檢查狀態碼正常、回應是合法資料、而且有該有的欄位。測試報告上的 20 項就是這一檔,逐項說明在下面第 3 節。Signs in to SW, picks the account, then switches page by page through Contacts, Actions, Reports and other core pages, capturing the API each page calls and checking the status, that the response is valid data, and that the expected fields exist. The 20 rows in the test report are this file — see section 3. |
| ⑤ | test_sw_segments.py | 1 SW 登入 → 2 選帳號 → 3 用 API 查一次各 Segment 的人數(匯入前)→ 4 匯入一筆聯絡人 → 5 等一下(cl1 等 60 秒、其他 30 秒)再查一次,檢查人數有沒有照規則增加:第 1 個 Segment 一定 +1;另外三個 Segment 分別在 email、姓氏、名字符合條件時 +1。用來確認「聯絡人進來後有被正確分到 Segment」。1 SW sign-in → 2 pick the account → 3 query each segment's count via the API (before import) → 4 import a contact → 5 wait (60 s on cl1, 30 s elsewhere) and query again, checking counts rose by the rules: the first segment always +1; three others +1 when the email, last name or first name matches. Confirms new contacts land in the right segments. |
| ⑥ | test_ava_link.py | 1 Ava 登入 → 2 逐一選帳號、進 Conversations、隨機抽幾段對話 → 3 點開每段對話裡 Ava 發出的連結,看能不能正常打開(打不開、跳到瀏覽器錯誤頁、或 HTTP 400 以上都算異常)。抽到的對話沒有 Ava 連結時,會從剩下的對話隨機補一段。異常會整理成表格放進通知。1 Ava sign-in → 2 for each account, open Conversations and randomly sample a few → 3 open the link Ava sent in each one and check it loads (failing to open, landing on a browser error page, or HTTP 400+ counts as a problem). If a sampled conversation has no Ava link, another one is picked at random. Problems are tabulated in the notification. |
③④⑤⑥ 都是「同一個瀏覽器分頁、照步驟往下走」:前一步失敗,後面依賴它的步驟通常也會跟著失敗,看報告時先找第一個失敗的步驟。③④⑤⑥ all walk through steps in the same browser tab: when one step fails, the steps that depend on it usually fail too — in the report, look for the first failure.
報告上每一項對應哪個檔(依報告順序)Which file each report row comes from (in report order)
| # | 報告上的名稱Names in the report | 項數Rows | 說明Notes |
|---|---|---|---|
| ① | test_ip | 1 | 查 IPIP check |
| ② | test_replaces_numeric_segments … test_skips_malformed_entry_without_crashing | 16 | API 效能統計工具的單元測試,每項幾毫秒。名稱裡的 segments 指的是「網址的片段」,跟 SW 的 Segment 無關Unit tests for the API metrics tool, a few ms each. "segments" here means URL path segments, unrelated to SW Segments |
| ③ | test_sw_nucleus_login → test_sw_nucleus_select_account → test_import_contacts → test_verify_sms → test_simulate_lead_messages → test_ava_nucleus_login → test_ava_nucleus_select_account → test_compare_messages | 8 | 對應上表 ③ 的 8 個步驟。test_verify_sms 約 30 秒、test_simulate_lead_messages 約 1 分半,是整套最花時間的部分之一The 8 steps of ③ above. test_verify_sms takes ~30 s and test_simulate_lead_messages ~1.5 min — among the slowest parts |
| ④ | test_login → test_select_account → test_page_triggers_expected_api[…] ×17 → test_actions_3month_range_completeness_not_capped | 20 | 逐項說明見第 3 節See section 3 |
| ⑤ | test_sw_segments_login → test_sw_segments_select_account → test_sw_segments_count → test_import_contacts → test_sw_segments_count2 | 5 | count = 匯入前查人數,count2 = 匯入後再查並比對。test_import_contacts 在 ③ 和 ⑤ 各出現一次,是兩個不同檔的同名步驟count = before import, count2 = after import and compare. test_import_contacts appears in both ③ and ⑤ — same name, different files |
| ⑥ | test_ava_link_login → test_ava_link_select_and_filter → test_ava_link_click_links | 3 | 只在 prod 報告出現;點連結那一步約 10 分鐘以上,是最久的一項Only in the prod report; the link-clicking step takes 10+ minutes, the longest row |
合計:stg、cl1 共 51 項,prod 共 54 項。Total: 51 rows on stg and cl1, 54 on prod.
3. 測試報告上的 20 項(依報告順序)3. The 20 rows in the test report (in report order)
這 20 項都來自 test_sw_page_api_checks.py。第 3~19 項做的事都一樣:打開那一頁 → 等那一頁該打的 API 出現 → 檢查回應;等不到或回應不對會重新整理再試,最多 3 次。同一頁常會連續打好幾次同一支 API,只要其中有一支正常就算通過。All 20 rows come from test_sw_page_api_checks.py. Rows 3–19 all do the same thing: open the page → wait for the API that page should call → check the response; if it doesn't arrive or is wrong, reload and retry, up to 3 times. A page often calls the same API several times — one good response is enough to pass.
| # | 報告上的名稱Name in the report | 檢查什麼What it checks | stg | cl1 / prod |
|---|---|---|---|---|
| 1 | test_login | 登入 SWSign in to SW | Passed | Passed |
| 2 | test_select_account | 選測試帳號,並記下選帳號時遇到的是新版還是舊版介面(舊版介面 → 後面的頁面檢查全部 Skipped)Pick the test account and note whether the new or old UI appeared (old UI → every page check below is Skipped) | Passed | Passed |
| 3 | [Home] | 首頁有打出 Actions 或 Contacts 的資料 APIHome calls the Actions or Contacts data API | Passed | Passed |
| 4 | [Contacts-27] | Contacts 打新版表格 API,回應有表格資料與總筆數Contacts calls the new grid API; the response has rows and a total | Passed | XPassed ¹ |
| 5 | [Contacts - Marketing Sources] | 同上(新版表格 API)Same (new grid API) | Passed | XPassed ¹ |
| 6 | [Contacts - Endpoint Attributes] | 同上Same | Passed | XFailed ² |
| 7 | [Contacts - Technographics] | 同上Same | Passed | XFailed ² |
| 8 | [Contacts - API] | 選單裡的「API」其實是 API 文件頁,檢查它有讀到帳號的 API Key 資料The menu's "API" is the API doc page; checks it loads the account's API key data | Passed | Passed |
| 9 | [Contacts (old)] | 舊版 Contacts 打舊版清單 API(一頁 20 筆),回應有聯絡人清單與分頁資訊Old Contacts calls the old list API (20 per page); the response has the contacts and paging info | Passed | Passed |
| 10 | [Contacts Grid (old)] | 舊的 Contacts Grid 頁打新版表格 APIThe old Contacts Grid page calls the new grid API | Passed | XFailed ³ |
| 11 | [Contacts-26 (old)] | 舊的 Contacts-26 頁打新版表格 APIThe old Contacts-26 page calls the new grid API | Passed | XFailed ³ |
| 12 | [Actions] | Actions 打新版 Actions 表格 API,回應有總計(Total)與「資料是否完整」資訊Actions calls the new Actions grid API; the response has totals and a completeness flag | Passed | XFailed ⁴ |
| 13 | [Actions - Marketing Sources] | 這頁目前新舊兩支 API 都會打,檢查舊版那支有正常回應This page currently calls both the new and old API; checks the old one responds | Passed | Passed |
| 14 | [Actions - Endpoint Breakdowns] | 打新版 Actions 表格 APICalls the new Actions grid API | Passed | XFailed ⁴ |
| 15 | [Actions - Technographics] | 同上Same | Passed | XFailed ⁴ |
| 16 | [Action Deep Dive] | 同上Same | Passed | XFailed ⁴ |
| 17 | [Action Report (legacy)] | 舊版 Action Report 打舊版報表 APIThe old Action Report calls the old report API | Passed | Passed |
| 18 | [Contact Report (legacy)] | Contact Report 打聯絡人報表 APIContact Report calls the contact report API | Passed | Passed |
| 19 | [Visitor Report (legacy)] | Visitor Report 打訪客報表 APIVisitor Report calls the visitor report API | Passed | Passed |
| 20 | test_actions_3month_range_completeness_not_capped | 回歸測試:用近 3 個月的日期區間開 Actions,檢查回應說「資料沒有被截斷」。對應已知 bug(Trello f1cfO0nU:區間太大時 Total: Filtered 會失準)Regression test: open Actions with a ~3-month range and check the response says the data wasn't truncated. Tracks a known bug (Trello f1cfO0nU: Total: Filtered drifts on large ranges) | XPassed ⁵ | XFailed ⁵ |
- 1
保險用的 xfail:這兩頁即使旗標關著通常也會打新版並通過,所以 cl1、prod 都顯示 XPassed。標記當初是為了保護 prod 第一次換新版時不要讓 CI 變紅;prod 換版後行為已跟 cl1 一樣。Precautionary xfail: these two pages usually call the new API and pass even with the flag off, so cl1 and prod both show XPassed. The mark was added to protect prod's first switch to the new UI; since the switch, prod behaves like cl1.
- 2
Contacts 新版功能的旗標在 cl1 / prod 沒開(功能還沒完成,刻意關著),所以這兩頁還在打舊版 API,攔不到新版的。不是測試程式壞掉、也不是部署慢,等旗標打開才會自動變 XPassed。The Contacts new-grid flag is off on cl1 / prod (feature unfinished, off on purpose), so these pages still call the old API and the new one never appears. It's not a test bug or a slow deploy — it turns XPassed once the flag is on.
- 3
在 cl1 版的前端,這兩個舊網址會直接轉到 Contacts-27,已經不是獨立頁面(SW 選單裡也已經拿掉);stg 還保留著。屬環境版本差異。On the cl1 frontend these two old URLs redirect to Contacts-27 and are no longer pages of their own (already removed from the SW menu); stg still keeps them. An environment version difference.
- 4
Actions 新版功能的旗標在 cl1 / prod 沒開,這些頁還在打舊版 API。同 ²,旗標打開才會變 XPassed。The Actions new-grid flag is off on cl1 / prod, so these pages still call the old API. As with ², they turn XPassed once the flag is on.
- 5
已知 bug 的回歸測試,標記為 xfail。stg 目前跑出 XPassed(資料沒被截斷,代表 stg 上已經不再重現);cl1、prod 仍是 XFailed(還會截斷)。等每個環境都變 XPassed,就可以把這個標記拿掉。Regression test for a known bug, marked xfail. stg currently shows XPassed (no truncation — it no longer reproduces on stg); cl1 and prod still show XFailed (still truncating). Once every environment shows XPassed, the mark can be removed.
上表是現行(2026-10-01)stg、cl1、prod 三份報告的結果,cl1 和 prod 完全相同。旗標打開、bug 修好或部署版本變動時結果會跟著變,看到跟上表不同時,先對照上面的註解判斷是「好消息」還是真的壞了。如果在第 2 項偵測到舊版介面,第 3~20 項會全部顯示 Skipped。The table reflects the current (2026-10-01) stg, cl1 and prod reports; cl1 and prod are identical. Results change when flags are turned on, bugs are fixed or deployments change — if you see something different, use the notes above to tell good news from a real break. If row 2 detects the old UI, rows 3–20 all show Skipped.
報告開頭那行數字怎麼看Reading the summary line
例如 cl1:「ALL PASSED — 共 51 項:✅ 40 passed, ❌ 0 failed, ⚠️ 0 error」。passed 只算真正的 Passed,XFailed / XPassed 不算在裡面,所以 40 + 11(2 個 XPassed + 9 個 XFailed)= 51。只要 failed 和 error 都是 0,就會顯示 ALL PASSED。三個環境的現行數字:For example cl1: "ALL PASSED — 51 rows: ✅ 40 passed, ❌ 0 failed, ⚠️ 0 error". "passed" counts only real Passed rows, not XFailed / XPassed, so 40 + 11 (2 XPassed + 9 XFailed) = 51. It shows ALL PASSED whenever failed and error are both 0. Current numbers:
| 環境Env | 共Total | passed | XPassed | XFailed |
|---|---|---|---|---|
| stg | 51 | 50 | 1 | 0 |
| cl1 | 51 | 40 | 2 | 9 |
| prod | 54 | 43 | 2 | 9 |
4. 報告上的結果代號4. Result labels in the report
| 代號Label | 意思Meaning | 要不要處理Action needed? |
|---|---|---|
| Passed | 通過Passed | 不用No |
| Failed | 失敗,而且不是已知狀況Failed, and not a known condition | 要看:會被寫進問題紀錄檔,出現在 Email / Teams 通知的表格裡Yes: it's written to the issue file and shown in the Email / Teams table |
| XFailed | 「預期中」的失敗:已知的環境差異或已知 bugAn expected failure: a known environment difference or known bug | 不用,不會讓 CI 變紅,也不會出現在通知的問題表格No — it doesn't turn CI red or appear in the notification's issue table |
| XPassed | 原本預期會失敗,結果通過了Expected to fail, but passed | 好消息:代表旗標已經打開或 bug 已修,可以把那筆 xfail 標記拿掉Good news: the flag is on or the bug is fixed — the xfail mark can be removed |
| Skipped | 被略過(例如環境還是舊版介面)Skipped (e.g. the environment still has the old UI) | 看原因,通常是正常的Check the reason; usually expected |
5. 跑完會留下什麼5. What a run leaves behind
| 檔案 / 位置File / location | 內容Contents |
|---|---|
reports/report.html | 測試報告(就是上面那 20 項那份),單一檔案可以直接用瀏覽器開The test report (the one with the 20 rows above), a single file you can open in a browser |
reports/junit.xml | 同樣的結果,給 CI 讀的格式The same results in a CI-readable format |
reports/test.log | 每一步的詳細紀錄,出錯時看這裡Step-by-step log — look here when something fails |
videos/ | 整次執行的錄影(檔名帶環境與時間)A video of the whole run (named with environment and time) |
reports/sw_page_api_issues.json、ava_link_issues.json | 頁面 API 檢查、Ava 連結檢查的問題清單(有問題才會產生;已知的 xfail 不算)Issue lists for the page API checks and Ava link checks (only created when there are problems; known xfails excluded) |
reports/api_metrics.json | 整次執行所有 API 的耗時統計,通知裡會列最慢的 5 支Timing stats for every API in the run; the notification lists the 5 slowest |
runs/<run_id>/ | 每次執行的歸檔(報告、log、執行資訊),也會上傳一份到雲端,通知裡附連結An archive per run (report, log, run info), also uploaded to the cloud with links in the notification |
| Email / Teams 通知Email / Teams notification | 通過 / 失敗數量摘要、錯誤訊息、頁面 API 與 Ava 連結的問題表格、最慢的 5 支 APIPass / fail summary, error messages, page-API and Ava-link issue tables, the 5 slowest APIs |
每次執行都會覆蓋 reports/ 裡的檔案,要留存請看 runs/ 的歸檔。另外,如果某次執行中途沒有產生新報告,歸檔時可能會帶到上一次的報告 —— 對照報告內容時,以報告裡實際的頁面網址判斷是哪個環境。Every run overwrites the files in reports/; keep results via the runs/ archive. Also, if a run stops before writing a new report, the archive may pick up the previous report — when comparing, judge the environment by the page URLs inside the report.