據(jù)提取實(shí)戰(zhàn)：基于 browser-harness 的四種無瀏覽器抓取模式)
Macrotrends 歷史金融數(shù)據(jù)提取實(shí)戰(zhàn)基于 browser-harness 的四種無瀏覽器抓取模式【免費(fèi)下載鏈接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.項(xiàng)目地址: https://gitcode.com/gh_mirrors/br/browser-harnessMacrotrendshttps://www.macrotrends.net提供長期歷史金融與經(jīng)濟(jì)圖表數(shù)據(jù)覆蓋股票 OHLCV 價(jià)格、市值、市盈率、營收、利潤率、股息率、SP 500 指數(shù)、聯(lián)邦基金利率、美債收益率、CPI、黃金價(jià)格等海量序列。本文基于本倉庫agent-workspace/domain-skills/macrotrends/scraping.md的實(shí)戰(zhàn)驗(yàn)證結(jié)論系統(tǒng)講解在 browser-harness 項(xiàng)目中用純 HTTP 方式無需啟動(dòng)瀏覽器提取這些數(shù)據(jù)的四種訪問模式、完整可運(yùn)行代碼、字段含義與反爬注意事項(xiàng)。讀完本文你將能夠直接復(fù)制代碼塊抓取任意美股的歷史行情與基本面數(shù)據(jù)、指數(shù)圖表數(shù)據(jù)和經(jīng)濟(jì)指標(biāo)時(shí)間序列。為什么 Macrotrends 只讀任務(wù)不需要瀏覽器browser-harness 的核心定位是通過 CDP 直接控制真實(shí)瀏覽器完成交互型任務(wù)而它的 SKILL.md 明確寫有一條原則A basic fetch of public information needs no browser. If a plain HTTP request can read it — a public page, an API, docs — usecurlor your fetch tool, and leave the browser alone.獲取公開信息不需要瀏覽器能用普通 HTTP 讀到的內(nèi)容就用 HTTP把瀏覽器留給需要點(diǎn)擊、輸入、登錄態(tài)、JS 渲染或反爬頁面的場景。Macrotrends 的所有數(shù)據(jù)端點(diǎn)都滿足這一判斷它們要么是直接返回內(nèi)嵌 JS 數(shù)組變量的 iframe PHP 頁面要么是返回 JSON 的/economic-data/API全部可以通過http_get直接訪問無需瀏覽器。本倉庫的領(lǐng)域技能文檔 scraping.md 對(duì)所有結(jié)果均已在真實(shí)站點(diǎn)上驗(yàn)證文檔標(biāo)注驗(yàn)證日期為 2026-04-18。http_get是 browser-harness 為純 HTTP 無瀏覽器場景提供的現(xiàn)成輔助函數(shù)實(shí)現(xiàn)在 src/browser_harness/helpers.pydef http_get(url, headersNone, timeout20.0): Pure HTTP — no browser. Use for static pages / APIs. Wrap in ThreadPoolExecutor for bulk. When BROWSER_USE_API_KEY is set, routes through the fetch-use proxy (handles bot detection, residential proxies, retries). Falls back to local urllib otherwise. if os.environ.get(BROWSER_USE_API_KEY): try: from fetch_use import fetch_sync return fetch_sync(url, headersheaders, timeout_msint(timeout * 1000)).text except ImportError: pass import gzip h {User-Agent: Mozilla/5.0, Accept-Encoding: gzip} if headers: h.update(headers) with urllib.request.urlopen(urllib.request.Request(url, headersh), timeouttimeout) as r: data r.read() if r.headers.get(Content-Encoding) gzip: data gzip.decompress(data) return data.decode()從源碼可以看到三個(gè)對(duì)本文至關(guān)重要的實(shí)現(xiàn)細(xì)節(jié)默認(rèn) UA 就是Mozilla/5.0——這與 Macrotrends 文檔所有端點(diǎn)用默認(rèn) UA 即可的結(jié)論吻合自動(dòng)處理 gzip 解壓——/economic-data/API 返回 gzip 壓縮的 JSONhttp_get已經(jīng)內(nèi)置解壓邏輯設(shè)置了BROWSER_USE_API_KEY時(shí)自動(dòng)路由到 fetch-use 代理——若你的抓取遇到反爬可在不修改代碼的情況下通過代理通道獲得住宅 IP、重試與機(jī)器人檢測規(guī)避能力。另外領(lǐng)域技能中的from helpers import http_get來自 agent_helpers.py 的加載機(jī)制當(dāng)BH_AGENT_WORKSPACE指向agent-workspace目錄時(shí)helpers.py 的_load_agent_helpers()會(huì)把a(bǔ)gent_helpers.py中定義的輔助函數(shù)動(dòng)態(tài)注入browser_harness.helpers命名空間。領(lǐng)域技能文檔即以此方式復(fù)用同一套 HTTP 工具。第一步根據(jù)目標(biāo)數(shù)據(jù)選擇訪問模式Macrotrends 的數(shù)據(jù)分為四大類對(duì)應(yīng)四種不同的訪問模式文檔給出的選擇表如下表格中的延遲為文檔實(shí)測值供參考目標(biāo)模式延遲數(shù)據(jù)變量股票 OHLCV 價(jià)格歷史直接 iframe PHP~190msdataDaily股票市值日頻直接 iframe PHP~200mschartData股票基本面PE、營收、利潤率直接 iframe PHP~140mschartDataSP 500 / 綜合指數(shù)圖表chart_iframe_comp.php~90msoriginalData經(jīng)濟(jì)指標(biāo)利率、收益率、CPI/economic-data/JSON API~150msdata[]數(shù)組黃金、大宗商品價(jià)格兩條路徑均可~150msdata[]或originalData核心結(jié)論Macrotrends 只讀任務(wù)絕不使用瀏覽器。所有端點(diǎn)均可用默認(rèn)Mozilla/5.0UA 通過http_get訪問只有個(gè)別頁面偶爾返回 403 時(shí)才需要切換到 Chrome UA詳見后文限流與反爬一節(jié)。Pattern 1股票價(jià)格歷史OHLCV股票價(jià)格歷史是最常用的數(shù)據(jù)其 iframe URL 可以直接構(gòu)造無需先抓取主頁面。URL 格式為https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t{TICKER}yb{N}完整可運(yùn)行代碼如下import json, re from helpers import http_get def get_stock_ohlcv(ticker: str, years_back: int None) - list[dict]: Returns daily OHLCV records for any US stock. ticker: uppercase ticker symbol, e.g. AAPL, MSFT, TSLA, NVDA years_back: number of years of history (1~250 records, 15~3772 records). Omit (None) to get ALL available history (AAPL goes back to 1980). url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t{ticker} if years_back: url fyb{years_back} html http_get(url) m re.search(rvar\sdataDaily\s*\s*\[, html) if not m: raise ValueError(fNo dataDaily found for ticker {ticker!r}) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # Usage records get_stock_ohlcv(AAPL, years_back15) # [{d: 2011-04-18, o: 9.771, h: 9.9547, l: 9.593, c: 9.9433, v: 18.275}, ...] latest records[-1] # {d: 2026-04-17, o: 266.96, h: 272.3, l: 266.72, c: 270.23, # v: 55.211, ma50: 260.554, ma200: 251.828} print(f{latest[d]}: close${latest[c]} vol{latest[v]}M shares)解析邏輯說明正則先定位var dataDaily [的位置然后用手寫的括號(hào)計(jì)數(shù)bc找到與之配對(duì)的結(jié)束]最后把截取的 JSON 文本交給json.loads。這里特意不用re.DOTALL貪婪匹配是因?yàn)閐ataDaily數(shù)組可能高達(dá) ~450KB3772 條 OHLCV 記錄括號(hào)計(jì)數(shù)法是 O(n) 的線性掃描速度更快。dataDaily 字段參考字段含義類型d日期YYYY-MM-DDstro開盤價(jià)已按拆股調(diào)整str/floath最高價(jià)str/floatl最低價(jià)str/floatc收盤價(jià)str/floatv成交量單位百萬股str/floatma5050 日均線str/float僅出現(xiàn)在近期記錄中ma200200 日均線str/float僅出現(xiàn)在近期記錄中注意所有價(jià)格值都是字符串需要自行float()轉(zhuǎn)換。成交量單位是百萬股55.211表示當(dāng)日成交 5521.1 萬股。已驗(yàn)證的 ticker2026-04-18 實(shí)測以下 ticker 均通過直接構(gòu)造 iframe URL 驗(yàn)證成功無需先抓取主頁面# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc. # 3772 records for yb15 (goes back to 2011-04-18) # AAPL full history: 11428 records back to 1980-12-12Pattern 2股票基本面PE、營收、市值、利潤率基本面數(shù)據(jù)分散在幾個(gè)不同的 PHP 文件中均支持直接構(gòu)造 URL。這些文件返回的都是var chartData [...]變量解析邏輯與 Pattern 1 完全一致。市值日頻單位十億美元import json, re from helpers import http_get def get_market_cap(ticker: str, years_back: int 15) - list[dict]: url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t{ticker}yb{years_back} html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) data get_market_cap(AAPL) # [{date: 2026-04-15, v1: 3929.35}, {date: 2026-04-16, v1: 3884.67}, ...] # v1 market cap in billions USDPE 比率、營收、流動(dòng)比率季度/年度基本面import json, re from helpers import http_get def get_fundamental(ticker: str, metric_type: str, statement: str, freq: str Q, years_back: int 15) - list[dict]: freq: Q quarterly, A annual url ( fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ ffundamental_iframe.php?t{ticker}type{metric_type}statement{statement} ffreq{freq}subyb{years_back} ) html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # PE ratio pe get_fundamental(AAPL, pe-ratio, price-ratios) # [{date: 2025-09-30, v1: 254.146, v2: 7.46, v3: 34.07}, ...] # v1 stock price, v2 quarterly EPS, v3 PE ratio # Revenue rev get_fundamental(AAPL, revenue, income-statement) # [{date: 2025-12-31, v1: 435.617, v2: 143.756, v3: 15.65}, ...] # v1 TTM revenue ($B), v2 quarterly revenue ($B), v3 YoY growth % # Total assets assets get_fundamental(AAPL, total-assets, balance-sheet) # Current ratio ratio get_fundamental(AAPL, current-ratio, ratios)注意fundamental_iframe.php的 URL 中sub是空參數(shù)保留占位freq取Q季度或A年度。不同type/statement組合返回的v1/v2/v3含義不同詳見下文 Gotchas 的字段對(duì)照。利潤率def get_profit_margins(ticker: str, years_back: int 15) - list[dict]: url ( fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ ffundamental_metric.php?t{ticker}chartprofit-marginsubyb{years_back} ) html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) margins get_profit_margins(AAPL) # [{date: 2025-12-31, v1: 47.33, v2: 32.38, v3: 27.04}, ...] # v1 gross margin %, v2 operating margin %, v3 net margin %股息率def get_dividend_yield(ticker: str, years_back: int 15) - list[dict]: url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t{ticker}yb{years_back} html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) dy get_dividend_yield(AAPL) # [{date: 2026-04-17, c: 270.23, ttm_d: 1.03848, ttm_dy: 0.3843}, ...] # c stock price, ttm_d TTM dividend ($), ttm_dy TTM yield (%)股票指標(biāo) URL 參考所有股票類端點(diǎn)共用同一個(gè) URL 前綴https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/并統(tǒng)一接收?t{TICKER}yb{N}參數(shù)基本面類多一個(gè)sub空參指標(biāo)PHP 文件額外參數(shù)股票價(jià)格 OHLCVstock_price_history.php—市值日頻market_cap.php—股息率dividend_yield.php—拆股含于價(jià)格歷史stock_splits.php—PE 比率fundamental_iframe.phptypepe-ratiostatementprice-ratios營收fundamental_iframe.phptyperevenuestatementincome-statement總資產(chǎn)fundamental_iframe.phptypetotal-assetsstatementbalance-sheet流動(dòng)比率fundamental_iframe.phptypecurrent-ratiostatementratios利潤率fundamental_metric.phpchartprofit-marginPattern 3指數(shù)與綜合圖表SP 500、Shiller PE 等指數(shù)類頁面如 SP 500 PE 比率、黃金百年價(jià)格把圖表數(shù)據(jù)內(nèi)嵌在chart_iframe_comp.php中數(shù)據(jù)變量名是originalData。與股票類不同這類 URL 需要頁面 ID URL slug兩個(gè)參數(shù)https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{page_id}url{url_slug}import json, re from helpers import http_get def extract_index_chart(page_id: int, url_slug: str) - list[dict]: page_id: the numeric ID from the page URL, e.g. 2577 url_slug: last segment of the page URL, e.g. sp500-pe-ratio-price-to-earnings-chart url fhttps://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{page_id}url{url_slug} html http_get(url) m re.search(rvar\soriginalData\s*\s*\[, html) if not m: raise ValueError(originalData not found — this page may use a different pattern) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # SP 500 PE ratio (1180 monthly records, 1927-2026) pe_data extract_index_chart(2577, sp500-pe-ratio-price-to-earnings-chart) # [{date: 1927-12-01, close: 15.9099}, ..., {date: 2026-03-01, close: 27.8925}] # close is the PE ratio value # Gold prices (1336 monthly records, 1915-2026) gold_data extract_index_chart(1333, historical-gold-prices-100-year-chart) # [{id: GOLDAMGBD228NLBM, date: 1915-01-01, close: 629.36, close1: 19.250}, ...] # close inflation-adjusted price, close1 nominal USD price print(fLatest SP PE: {pe_data[-1]}) # {date: 2026-03-01, close: 27.8925} print(fLatest gold: {gold_data[-1]}) # {id: ..., date: 2026-04-01, close: 5177.19, close1: 5177.190}檢測頁面屬于哪種模式不是所有macrotrends.net/NNNN/slug頁面都用chart_iframe_comp.php——有些經(jīng)濟(jì)頁面走generateChart()/economic-data/API?？梢韵茸ブ黜撁?HTML根據(jù)特征字符串判斷def get_page_pattern(page_url: str) - str: html http_get(page_url) if chart_iframe_comp.php in html: return index_chart # use extract_index_chart() elif generateChart in html and highchartsURL in html: return economic_api # use get_economic_data() elif /production/stocks/desktop/PRODUCTION/ in html: return stock_iframe # use get_stock_ohlcv() etc. return unknown從頁面 URL 提取 ID 和 slugimport re from helpers import http_get page_url https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart html http_get(page_url) # Option A: parse from the iframe src in the HTML m re.search(rchart_iframe_comp\.php\?id(\d)url([^]), html) if m: page_id, url_slug int(m.group(1)), m.group(2) # Option B: derive from the page URL (works when slug matches) import urllib.parse parts page_url.rstrip(/).split(/) page_id int(parts[-2]) # 2577 url_slug parts[-1] # sp500-pe-ratio-price-to-earnings-chartPattern 4經(jīng)濟(jì)指標(biāo) JSON API使用generateChart()的頁面其數(shù)據(jù)來自/economic-data/{pageID}/{freq}這個(gè) JSON 端點(diǎn)。該端點(diǎn)要求Referer頭必須匹配對(duì)應(yīng)的頁面 URL否則部分頁面會(huì)返回 403。響應(yīng)默認(rèn)是 gzip 壓縮的http_get已內(nèi)置解壓下面示例用標(biāo)準(zhǔn)庫手寫一遍以展示完整鏈路import json, datetime, gzip, urllib.request from helpers import http_get def get_economic_data(page_id: int, referer_url: str, freq: str D) - dict: page_id: numeric ID from the page URL (e.g. 2015 for Fed Funds Rate) referer_url: the full page URL — required as Referer header freq: D daily, M monthly (not all support both) Returns {data: [[ts_ms, value], ...], metadata: {...}} url fhttps://www.macrotrends.net/economic-data/{page_id}/{freq} headers { User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36, Accept: application/json, */*, Accept-Encoding: gzip, Referer: referer_url, } with urllib.request.urlopen(urllib.request.Request(url, headersheaders), timeout20) as r: raw r.read() if r.headers.get(Content-Encoding) gzip: raw gzip.decompress(raw) result json.loads(raw) if result is None: raise ValueError(fpageID{page_id} does not support freq{freq!r}) return result # Fed Funds Rate (daily, 25319 records) ffr get_economic_data(2015, https://www.macrotrends.net/2015/fed-funds-rate-historical-chart, freqD) print(ffr[metadata][name]) # Fed Funds Interest Rate print(ffr[metadata][label]) # % # Convert timestamps to dates for ts_ms, value in ffr[data][-3:]: dt datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC) print(f{dt.strftime(%Y-%m-%d)}: {value}%) # 2026-04-13: 3.64% # 2026-04-14: 3.64% # 2026-04-15: 3.64% # 10-Year Treasury yield (daily, 16074 records) t10 get_economic_data(2016, https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart, freqD) # Last: 2026-04-15: 4.29% # Gold prices (monthly, 1336 records, 1915-present) — template5 gold get_economic_data(1333, https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart, freqM) # metadata: {name: Gold Prices, currency: $, label: } # US Unemployment Rate (monthly, 938 records) unemp get_economic_data(1316, https://www.macrotrends.net/1316/us-national-unemployment-rate, freqM) # metadata: {name: U.S. Unemployment Rate, label: %} # Debt-to-GDP ratio (monthly, 712 records) debt_gdp get_economic_data(1381, https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart, freqM)返回結(jié)構(gòu)是{data: [[毫秒時(shí)間戳, 數(shù)值], ...], metadata: {...}}data內(nèi)的時(shí)間戳需除以 1000 后轉(zhuǎn) datetime。metadata 字段說明{ name: Fed Funds Interest Rate, # chart title tableHeaderName: Fed Funds Interest Rate, currency: , # $ for dollar-denominated series label: %, # units label chartType: line, mobileChartType: line, lineWidth: 2, positiveColor: #2caffe, negativeColor: , decimals: , chartScale: linear, seriesUnits: }其中name圖表標(biāo)題、currency美元計(jì)價(jià)序列為$、label單位標(biāo)簽如%是最常用于校驗(yàn)數(shù)據(jù)正確性的字段?？捎妙l率代碼代碼含義備注D日頻大多數(shù)序列支持M月頻不可用時(shí)返回nullQ季頻通常為null——用M代替A年頻通常為null——用M代替DEFAULT默認(rèn)通常是月頻大多數(shù)序列與M相同INDEXMONTHLY月度指數(shù)收盤部分商品/指數(shù)序列INDEXDAILY日度指數(shù)部分序列DAILYEXCHANGERATE日頻外匯匯率貨幣對(duì)10YD10 年日頻專門序列實(shí)操建議先試D得到null再回退到M。大多數(shù)經(jīng)濟(jì)序列只有D和M穩(wěn)定返回?cái)?shù)據(jù)Q、A對(duì)多數(shù)指標(biāo)都返回null。已知經(jīng)濟(jì)頁面 IDIDURL slug描述1316us-national-unemployment-rate美國失業(yè)率月頻回溯至 1948 年1333historical-gold-prices-100-year-chart黃金價(jià)格月頻回溯至 1915 年1381debt-to-gdp-ratio-historical-chart美國債務(wù)/GDP 比率2015fed-funds-rate-historical-chart聯(lián)邦基金利率日頻回溯至 1954 年201610-year-treasury-bond-rate-yield-chart10 年期美債收益率日頻回溯至 1962 年2577sp500-pe-ratio-price-to-earnings-chartSP 500 PE 比率使用chart_iframe_comp.php通用提取輔助函數(shù)前面四個(gè)模式的解析邏輯高度一致正則定位變量名 → 括號(hào)計(jì)數(shù)找到數(shù)組邊界 →json.loads可以收斂為一個(gè)通用函數(shù)一次性覆蓋dataDaily、chartData、originalData三種內(nèi)嵌 JS 變量import json, re from helpers import http_get def extract_chart_var(html: str, var_name: str) - list: Extract a JS array variable from Macrotrends iframe HTML. m re.search(rfvar\s{re.escape(var_name)}\s*\s*\[, html) if not m: return [] si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: return json.loads(html[si:j1]) return [] # Works for dataDaily, chartData, or originalData: html http_get(https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?tAAPLyb15) daily extract_chart_var(html, dataDaily) html2 http_get(https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id2577urlsp500-pe-ratio-price-to-earnings-chart) pe_data extract_chart_var(html2, originalData)URL 構(gòu)造速查股票類頁面STOCK_BASE https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ # Price history OHLCV f{STOCK_BASE}stock_price_history.php?t{ticker} # all history f{STOCK_BASE}stock_price_history.php?t{ticker}yb{years} # last N years # Market cap f{STOCK_BASE}market_cap.php?t{ticker}yb{years} # Fundamentals f{STOCK_BASE}fundamental_iframe.php?t{ticker}type{type}statement{stmt}freq{freq}subyb{years} # type/statement combos: pe-ratio/price-ratios, revenue/income-statement, # total-assets/balance-sheet, current-ratio/ratios # Metrics f{STOCK_BASE}fundamental_metric.php?t{ticker}chart{metric}subyb{years} # metrics: profit-margin # Dividend yield f{STOCK_BASE}dividend_yield.php?t{ticker}yb{years}經(jīng)濟(jì) / 指數(shù)頁面# From numeric ID URL slug (read from page source or page URL) fhttps://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{id}url{slug} # Economic indicator JSON API (requires Referer header) fhttps://www.macrotrends.net/economic-data/{page_id}/{freq}限流與反爬實(shí)測結(jié)論根據(jù)文檔在 2026-04-18 的實(shí)測未觀察到任何限流對(duì)同一股票 iframe 連續(xù)發(fā)起 10 次快速請求總耗時(shí) 1.8 秒沒有觸發(fā)限速、驗(yàn)證碼或 429 錯(cuò)誤默認(rèn) UAMozilla/5.0對(duì)絕大多數(shù)端點(diǎn)可用所有 iframe PHP 數(shù)據(jù)文件從未出現(xiàn) 403部分主頁面 HTML 需要 Chrome UA抓取/stocks/charts/...或/2015/...這類包裝頁遇到 403 時(shí)切換為headers {User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36}/economic-data/{id}/{freq}必須攜帶Referer把頁面 URL 放進(jìn)Referer頭缺少時(shí)請求雖然被放行但部分頁面會(huì)返回 403任何端點(diǎn)都不需要 cookie、會(huì)話或鑒權(quán) token。如果本地直連仍遇到反爬攔截還可以利用 http_get 的代理路由能力設(shè)置BROWSER_USE_API_KEY環(huán)境變量后http_get會(huì)自動(dòng)改走 fetch-use 代理提供機(jī)器人檢測規(guī)避、住宅代理與重試代碼本身無需改動(dòng)。Gotchas容易踩的坑主頁面 URL ≠ 數(shù)據(jù)頁面 URL。部分 URL 會(huì)重定向到其他內(nèi)容例如/1316/us-national-debt-by-year實(shí)際會(huì)重定向到/1316/us-national-unemployment-rate。如果返回的數(shù)據(jù)看起來不對(duì)務(wù)必檢查最終 URLr.url并把最終 URL 作為Referer。yb參數(shù)控制歷史深度yb1→ 約 250 條記錄最近一年yb15→ 約 3772 條記錄最近 15 年省略 → 全部歷史AAPL 有 11428 條記錄回溯至 1980 年對(duì)大多數(shù)查詢這也是默認(rèn)行為經(jīng)濟(jì)頁面存在兩種 iframe 模式。macrotrends.net/NNNN/slug頁面要么用chart_iframe_comp.php變量是originalData要么用generateChart/economic-data/API。抓主頁面 HTML 判斷if chart_iframe_comp.php in html: # use extract_index_chart() elif highchartsURL in html: # use get_economic_data()黃金數(shù)據(jù)有兩列價(jià)格{id: GOLDAMGBD228NLBM, date: 2026-04-01, close: 5177.19, close1: 5177.190} # close inflation-adjusted price (base year adjusts over time) # close1 nominal USD price (the raw market price)經(jīng)濟(jì) API 的頻率代碼。只有D和M在大多數(shù)序列上穩(wěn)定返回?cái)?shù)據(jù)A、Q對(duì)多數(shù)經(jīng)濟(jì)指標(biāo)返回null。優(yōu)先試D。chartData的字段隨指標(biāo)變化market_cap.php→{date, v1}v1 市值單位十億美元fundamental_iframe.phptypepe-ratio →{date, v1, v2, v3}股價(jià)、EPS、PEfundamental_iframe.phptyperevenue →{date, v1, v2, v3}TTM 營收、季度營收、同比增速 %fundamental_metric.phpchartprofit-margin →{date, v1, v2, v3}毛利率 %、營業(yè)利潤率 %、凈利率 %dividend_yield.php→{date, c, ttm_d, ttm_dy}股價(jià)、TTM 股息、TTM 股息率 %大數(shù)組必須用括號(hào)計(jì)數(shù)。股票 iframe 的var dataDaily [...]約 450KB、3772 條 OHLCV 記錄。re.DOTALL貪婪正則雖然能工作但很慢本文所有示例采用的bc括號(hào)計(jì)數(shù)法是 O(n) 線性掃描速度快得多。沒有公開的 ticker 查詢 API。如果需要查找公司 slug可訪問搜索端點(diǎn)https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?vYYYYMMDD但股票價(jià)格 iframe 只需要 ticker 符號(hào)?tAAPL不需要 slug。與 browser-harness 的協(xié)作邊界最后回到 harness 的整體使用邊界本技能文檔只解決 Macrotrends 的只讀數(shù)據(jù)提取場景全程用http_get即可。按照 SKILL.md 的指引只有當(dāng)任務(wù)升級(jí)為交互型操作例如需要在 Macrotrends 頁面上點(diǎn)擊圖表控件、滾動(dòng)加載、操作已登錄會(huì)話時(shí)才應(yīng)升級(jí)到瀏覽器模式——而此時(shí)可以從 helpers.py 中按需調(diào)用new_tab、goto_url、js、click_at_xy等 CDP 輔助函數(shù)。日常的批量抓取如用ThreadPoolExecutor并發(fā)拉取多只股票則完全落在http_get的職責(zé)范圍內(nèi)既不消耗瀏覽器資源也不會(huì)觸發(fā)任何瀏覽器級(jí)反爬?！久赓M(fèi)下載鏈接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.項(xiàng)目地址: https://gitcode.com/gh_mirrors/br/browser-harness創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考 Macrotrends 歷史金融數(shù)據(jù)提取實(shí)戰(zhàn)：基于 browser-harness 的四種無瀏覽器抓取模式
Macrotrends 歷史金融數(shù)據(jù)提取實(shí)戰(zhàn)基于 browser-harness 的四種無瀏覽器抓取模式【免費(fèi)下載鏈接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.項(xiàng)目地址: https://gitcode.com/gh_mirrors/br/browser-harnessMacrotrendshttps://www.macrotrends.net提供長期歷史金融與經(jīng)濟(jì)圖表數(shù)據(jù)覆蓋股票 OHLCV 價(jià)格、市值、市盈率、營收、利潤率、股息率、SP 500 指數(shù)、聯(lián)邦基金利率、美債收益率、CPI、黃金價(jià)格等海量序列。本文基于本倉庫agent-workspace/domain-skills/macrotrends/scraping.md的實(shí)戰(zhàn)驗(yàn)證結(jié)論系統(tǒng)講解在 browser-harness 項(xiàng)目中用純 HTTP 方式無需啟動(dòng)瀏覽器提取這些數(shù)據(jù)的四種訪問模式、完整可運(yùn)行代碼、字段含義與反爬注意事項(xiàng)。讀完本文你將能夠直接復(fù)制代碼塊抓取任意美股的歷史行情與基本面數(shù)據(jù)、指數(shù)圖表數(shù)據(jù)和經(jīng)濟(jì)指標(biāo)時(shí)間序列。為什么 Macrotrends 只讀任務(wù)不需要瀏覽器browser-harness 的核心定位是通過 CDP 直接控制真實(shí)瀏覽器完成交互型任務(wù)而它的 SKILL.md 明確寫有一條原則A basic fetch of public information needs no browser. If a plain HTTP request can read it — a public page, an API, docs — usecurlor your fetch tool, and leave the browser alone.獲取公開信息不需要瀏覽器能用普通 HTTP 讀到的內(nèi)容就用 HTTP把瀏覽器留給需要點(diǎn)擊、輸入、登錄態(tài)、JS 渲染或反爬頁面的場景。Macrotrends 的所有數(shù)據(jù)端點(diǎn)都滿足這一判斷它們要么是直接返回內(nèi)嵌 JS 數(shù)組變量的 iframe PHP 頁面要么是返回 JSON 的/economic-data/API全部可以通過http_get直接訪問無需瀏覽器。本倉庫的領(lǐng)域技能文檔 scraping.md 對(duì)所有結(jié)果均已在真實(shí)站點(diǎn)上驗(yàn)證文檔標(biāo)注驗(yàn)證日期為 2026-04-18。http_get是 browser-harness 為純 HTTP 無瀏覽器場景提供的現(xiàn)成輔助函數(shù)實(shí)現(xiàn)在 src/browser_harness/helpers.pydef http_get(url, headersNone, timeout20.0): Pure HTTP — no browser. Use for static pages / APIs. Wrap in ThreadPoolExecutor for bulk. When BROWSER_USE_API_KEY is set, routes through the fetch-use proxy (handles bot detection, residential proxies, retries). Falls back to local urllib otherwise. if os.environ.get(BROWSER_USE_API_KEY): try: from fetch_use import fetch_sync return fetch_sync(url, headersheaders, timeout_msint(timeout * 1000)).text except ImportError: pass import gzip h {User-Agent: Mozilla/5.0, Accept-Encoding: gzip} if headers: h.update(headers) with urllib.request.urlopen(urllib.request.Request(url, headersh), timeouttimeout) as r: data r.read() if r.headers.get(Content-Encoding) gzip: data gzip.decompress(data) return data.decode()從源碼可以看到三個(gè)對(duì)本文至關(guān)重要的實(shí)現(xiàn)細(xì)節(jié)默認(rèn) UA 就是Mozilla/5.0——這與 Macrotrends 文檔所有端點(diǎn)用默認(rèn) UA 即可的結(jié)論吻合自動(dòng)處理 gzip 解壓——/economic-data/API 返回 gzip 壓縮的 JSONhttp_get已經(jīng)內(nèi)置解壓邏輯設(shè)置了BROWSER_USE_API_KEY時(shí)自動(dòng)路由到 fetch-use 代理——若你的抓取遇到反爬可在不修改代碼的情況下通過代理通道獲得住宅 IP、重試與機(jī)器人檢測規(guī)避能力。另外領(lǐng)域技能中的from helpers import http_get來自 agent_helpers.py 的加載機(jī)制當(dāng)BH_AGENT_WORKSPACE指向agent-workspace目錄時(shí)helpers.py 的_load_agent_helpers()會(huì)把a(bǔ)gent_helpers.py中定義的輔助函數(shù)動(dòng)態(tài)注入browser_harness.helpers命名空間。領(lǐng)域技能文檔即以此方式復(fù)用同一套 HTTP 工具。第一步根據(jù)目標(biāo)數(shù)據(jù)選擇訪問模式Macrotrends 的數(shù)據(jù)分為四大類對(duì)應(yīng)四種不同的訪問模式文檔給出的選擇表如下表格中的延遲為文檔實(shí)測值供參考目標(biāo)模式延遲數(shù)據(jù)變量股票 OHLCV 價(jià)格歷史直接 iframe PHP~190msdataDaily股票市值日頻直接 iframe PHP~200mschartData股票基本面PE、營收、利潤率直接 iframe PHP~140mschartDataSP 500 / 綜合指數(shù)圖表chart_iframe_comp.php~90msoriginalData經(jīng)濟(jì)指標(biāo)利率、收益率、CPI/economic-data/JSON API~150msdata[]數(shù)組黃金、大宗商品價(jià)格兩條路徑均可~150msdata[]或originalData核心結(jié)論Macrotrends 只讀任務(wù)絕不使用瀏覽器。所有端點(diǎn)均可用默認(rèn)Mozilla/5.0UA 通過http_get訪問只有個(gè)別頁面偶爾返回 403 時(shí)才需要切換到 Chrome UA詳見后文限流與反爬一節(jié)。Pattern 1股票價(jià)格歷史OHLCV股票價(jià)格歷史是最常用的數(shù)據(jù)其 iframe URL 可以直接構(gòu)造無需先抓取主頁面。URL 格式為https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t{TICKER}yb{N}完整可運(yùn)行代碼如下import json, re from helpers import http_get def get_stock_ohlcv(ticker: str, years_back: int None) - list[dict]: Returns daily OHLCV records for any US stock. ticker: uppercase ticker symbol, e.g. AAPL, MSFT, TSLA, NVDA years_back: number of years of history (1~250 records, 15~3772 records). Omit (None) to get ALL available history (AAPL goes back to 1980). url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t{ticker} if years_back: url fyb{years_back} html http_get(url) m re.search(rvar\sdataDaily\s*\s*\[, html) if not m: raise ValueError(fNo dataDaily found for ticker {ticker!r}) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # Usage records get_stock_ohlcv(AAPL, years_back15) # [{d: 2011-04-18, o: 9.771, h: 9.9547, l: 9.593, c: 9.9433, v: 18.275}, ...] latest records[-1] # {d: 2026-04-17, o: 266.96, h: 272.3, l: 266.72, c: 270.23, # v: 55.211, ma50: 260.554, ma200: 251.828} print(f{latest[d]}: close${latest[c]} vol{latest[v]}M shares)解析邏輯說明正則先定位var dataDaily [的位置然后用手寫的括號(hào)計(jì)數(shù)bc找到與之配對(duì)的結(jié)束]最后把截取的 JSON 文本交給json.loads。這里特意不用re.DOTALL貪婪匹配是因?yàn)閐ataDaily數(shù)組可能高達(dá) ~450KB3772 條 OHLCV 記錄括號(hào)計(jì)數(shù)法是 O(n) 的線性掃描速度更快。dataDaily 字段參考字段含義類型d日期YYYY-MM-DDstro開盤價(jià)已按拆股調(diào)整str/floath最高價(jià)str/floatl最低價(jià)str/floatc收盤價(jià)str/floatv成交量單位百萬股str/floatma5050 日均線str/float僅出現(xiàn)在近期記錄中ma200200 日均線str/float僅出現(xiàn)在近期記錄中注意所有價(jià)格值都是字符串需要自行float()轉(zhuǎn)換。成交量單位是百萬股55.211表示當(dāng)日成交 5521.1 萬股。已驗(yàn)證的 ticker2026-04-18 實(shí)測以下 ticker 均通過直接構(gòu)造 iframe URL 驗(yàn)證成功無需先抓取主頁面# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc. # 3772 records for yb15 (goes back to 2011-04-18) # AAPL full history: 11428 records back to 1980-12-12Pattern 2股票基本面PE、營收、市值、利潤率基本面數(shù)據(jù)分散在幾個(gè)不同的 PHP 文件中均支持直接構(gòu)造 URL。這些文件返回的都是var chartData [...]變量解析邏輯與 Pattern 1 完全一致。市值日頻單位十億美元import json, re from helpers import http_get def get_market_cap(ticker: str, years_back: int 15) - list[dict]: url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t{ticker}yb{years_back} html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) data get_market_cap(AAPL) # [{date: 2026-04-15, v1: 3929.35}, {date: 2026-04-16, v1: 3884.67}, ...] # v1 market cap in billions USDPE 比率、營收、流動(dòng)比率季度/年度基本面import json, re from helpers import http_get def get_fundamental(ticker: str, metric_type: str, statement: str, freq: str Q, years_back: int 15) - list[dict]: freq: Q quarterly, A annual url ( fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ ffundamental_iframe.php?t{ticker}type{metric_type}statement{statement} ffreq{freq}subyb{years_back} ) html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # PE ratio pe get_fundamental(AAPL, pe-ratio, price-ratios) # [{date: 2025-09-30, v1: 254.146, v2: 7.46, v3: 34.07}, ...] # v1 stock price, v2 quarterly EPS, v3 PE ratio # Revenue rev get_fundamental(AAPL, revenue, income-statement) # [{date: 2025-12-31, v1: 435.617, v2: 143.756, v3: 15.65}, ...] # v1 TTM revenue ($B), v2 quarterly revenue ($B), v3 YoY growth % # Total assets assets get_fundamental(AAPL, total-assets, balance-sheet) # Current ratio ratio get_fundamental(AAPL, current-ratio, ratios)注意fundamental_iframe.php的 URL 中sub是空參數(shù)保留占位freq取Q季度或A年度。不同type/statement組合返回的v1/v2/v3含義不同詳見下文 Gotchas 的字段對(duì)照。利潤率def get_profit_margins(ticker: str, years_back: int 15) - list[dict]: url ( fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ ffundamental_metric.php?t{ticker}chartprofit-marginsubyb{years_back} ) html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) margins get_profit_margins(AAPL) # [{date: 2025-12-31, v1: 47.33, v2: 32.38, v3: 27.04}, ...] # v1 gross margin %, v2 operating margin %, v3 net margin %股息率def get_dividend_yield(ticker: str, years_back: int 15) - list[dict]: url fhttps://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t{ticker}yb{years_back} html http_get(url) m re.search(rvar\schartData\s*\s*\[, html) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) dy get_dividend_yield(AAPL) # [{date: 2026-04-17, c: 270.23, ttm_d: 1.03848, ttm_dy: 0.3843}, ...] # c stock price, ttm_d TTM dividend ($), ttm_dy TTM yield (%)股票指標(biāo) URL 參考所有股票類端點(diǎn)共用同一個(gè) URL 前綴https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/并統(tǒng)一接收?t{TICKER}yb{N}參數(shù)基本面類多一個(gè)sub空參指標(biāo)PHP 文件額外參數(shù)股票價(jià)格 OHLCVstock_price_history.php—市值日頻market_cap.php—股息率dividend_yield.php—拆股含于價(jià)格歷史stock_splits.php—PE 比率fundamental_iframe.phptypepe-ratiostatementprice-ratios營收fundamental_iframe.phptyperevenuestatementincome-statement總資產(chǎn)fundamental_iframe.phptypetotal-assetsstatementbalance-sheet流動(dòng)比率fundamental_iframe.phptypecurrent-ratiostatementratios利潤率fundamental_metric.phpchartprofit-marginPattern 3指數(shù)與綜合圖表SP 500、Shiller PE 等指數(shù)類頁面如 SP 500 PE 比率、黃金百年價(jià)格把圖表數(shù)據(jù)內(nèi)嵌在chart_iframe_comp.php中數(shù)據(jù)變量名是originalData。與股票類不同這類 URL 需要頁面 ID URL slug兩個(gè)參數(shù)https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{page_id}url{url_slug}import json, re from helpers import http_get def extract_index_chart(page_id: int, url_slug: str) - list[dict]: page_id: the numeric ID from the page URL, e.g. 2577 url_slug: last segment of the page URL, e.g. sp500-pe-ratio-price-to-earnings-chart url fhttps://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{page_id}url{url_slug} html http_get(url) m re.search(rvar\soriginalData\s*\s*\[, html) if not m: raise ValueError(originalData not found — this page may use a different pattern) si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: ei j; break return json.loads(html[si:ei1]) # SP 500 PE ratio (1180 monthly records, 1927-2026) pe_data extract_index_chart(2577, sp500-pe-ratio-price-to-earnings-chart) # [{date: 1927-12-01, close: 15.9099}, ..., {date: 2026-03-01, close: 27.8925}] # close is the PE ratio value # Gold prices (1336 monthly records, 1915-2026) gold_data extract_index_chart(1333, historical-gold-prices-100-year-chart) # [{id: GOLDAMGBD228NLBM, date: 1915-01-01, close: 629.36, close1: 19.250}, ...] # close inflation-adjusted price, close1 nominal USD price print(fLatest SP PE: {pe_data[-1]}) # {date: 2026-03-01, close: 27.8925} print(fLatest gold: {gold_data[-1]}) # {id: ..., date: 2026-04-01, close: 5177.19, close1: 5177.190}檢測頁面屬于哪種模式不是所有macrotrends.net/NNNN/slug頁面都用chart_iframe_comp.php——有些經(jīng)濟(jì)頁面走generateChart()/economic-data/API?？梢韵茸ブ黜撁?HTML根據(jù)特征字符串判斷def get_page_pattern(page_url: str) - str: html http_get(page_url) if chart_iframe_comp.php in html: return index_chart # use extract_index_chart() elif generateChart in html and highchartsURL in html: return economic_api # use get_economic_data() elif /production/stocks/desktop/PRODUCTION/ in html: return stock_iframe # use get_stock_ohlcv() etc. return unknown從頁面 URL 提取 ID 和 slugimport re from helpers import http_get page_url https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart html http_get(page_url) # Option A: parse from the iframe src in the HTML m re.search(rchart_iframe_comp\.php\?id(\d)url([^]), html) if m: page_id, url_slug int(m.group(1)), m.group(2) # Option B: derive from the page URL (works when slug matches) import urllib.parse parts page_url.rstrip(/).split(/) page_id int(parts[-2]) # 2577 url_slug parts[-1] # sp500-pe-ratio-price-to-earnings-chartPattern 4經(jīng)濟(jì)指標(biāo) JSON API使用generateChart()的頁面其數(shù)據(jù)來自/economic-data/{pageID}/{freq}這個(gè) JSON 端點(diǎn)。該端點(diǎn)要求Referer頭必須匹配對(duì)應(yīng)的頁面 URL否則部分頁面會(huì)返回 403。響應(yīng)默認(rèn)是 gzip 壓縮的http_get已內(nèi)置解壓下面示例用標(biāo)準(zhǔn)庫手寫一遍以展示完整鏈路import json, datetime, gzip, urllib.request from helpers import http_get def get_economic_data(page_id: int, referer_url: str, freq: str D) - dict: page_id: numeric ID from the page URL (e.g. 2015 for Fed Funds Rate) referer_url: the full page URL — required as Referer header freq: D daily, M monthly (not all support both) Returns {data: [[ts_ms, value], ...], metadata: {...}} url fhttps://www.macrotrends.net/economic-data/{page_id}/{freq} headers { User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36, Accept: application/json, */*, Accept-Encoding: gzip, Referer: referer_url, } with urllib.request.urlopen(urllib.request.Request(url, headersheaders), timeout20) as r: raw r.read() if r.headers.get(Content-Encoding) gzip: raw gzip.decompress(raw) result json.loads(raw) if result is None: raise ValueError(fpageID{page_id} does not support freq{freq!r}) return result # Fed Funds Rate (daily, 25319 records) ffr get_economic_data(2015, https://www.macrotrends.net/2015/fed-funds-rate-historical-chart, freqD) print(ffr[metadata][name]) # Fed Funds Interest Rate print(ffr[metadata][label]) # % # Convert timestamps to dates for ts_ms, value in ffr[data][-3:]: dt datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC) print(f{dt.strftime(%Y-%m-%d)}: {value}%) # 2026-04-13: 3.64% # 2026-04-14: 3.64% # 2026-04-15: 3.64% # 10-Year Treasury yield (daily, 16074 records) t10 get_economic_data(2016, https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart, freqD) # Last: 2026-04-15: 4.29% # Gold prices (monthly, 1336 records, 1915-present) — template5 gold get_economic_data(1333, https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart, freqM) # metadata: {name: Gold Prices, currency: $, label: } # US Unemployment Rate (monthly, 938 records) unemp get_economic_data(1316, https://www.macrotrends.net/1316/us-national-unemployment-rate, freqM) # metadata: {name: U.S. Unemployment Rate, label: %} # Debt-to-GDP ratio (monthly, 712 records) debt_gdp get_economic_data(1381, https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart, freqM)返回結(jié)構(gòu)是{data: [[毫秒時(shí)間戳, 數(shù)值], ...], metadata: {...}}data內(nèi)的時(shí)間戳需除以 1000 后轉(zhuǎn) datetime。metadata 字段說明{ name: Fed Funds Interest Rate, # chart title tableHeaderName: Fed Funds Interest Rate, currency: , # $ for dollar-denominated series label: %, # units label chartType: line, mobileChartType: line, lineWidth: 2, positiveColor: #2caffe, negativeColor: , decimals: , chartScale: linear, seriesUnits: }其中name圖表標(biāo)題、currency美元計(jì)價(jià)序列為$、label單位標(biāo)簽如%是最常用于校驗(yàn)數(shù)據(jù)正確性的字段?？捎妙l率代碼代碼含義備注D日頻大多數(shù)序列支持M月頻不可用時(shí)返回nullQ季頻通常為null——用M代替A年頻通常為null——用M代替DEFAULT默認(rèn)通常是月頻大多數(shù)序列與M相同INDEXMONTHLY月度指數(shù)收盤部分商品/指數(shù)序列INDEXDAILY日度指數(shù)部分序列DAILYEXCHANGERATE日頻外匯匯率貨幣對(duì)10YD10 年日頻專門序列實(shí)操建議先試D得到null再回退到M。大多數(shù)經(jīng)濟(jì)序列只有D和M穩(wěn)定返回?cái)?shù)據(jù)Q、A對(duì)多數(shù)指標(biāo)都返回null。已知經(jīng)濟(jì)頁面 IDIDURL slug描述1316us-national-unemployment-rate美國失業(yè)率月頻回溯至 1948 年1333historical-gold-prices-100-year-chart黃金價(jià)格月頻回溯至 1915 年1381debt-to-gdp-ratio-historical-chart美國債務(wù)/GDP 比率2015fed-funds-rate-historical-chart聯(lián)邦基金利率日頻回溯至 1954 年201610-year-treasury-bond-rate-yield-chart10 年期美債收益率日頻回溯至 1962 年2577sp500-pe-ratio-price-to-earnings-chartSP 500 PE 比率使用chart_iframe_comp.php通用提取輔助函數(shù)前面四個(gè)模式的解析邏輯高度一致正則定位變量名 → 括號(hào)計(jì)數(shù)找到數(shù)組邊界 →json.loads可以收斂為一個(gè)通用函數(shù)一次性覆蓋dataDaily、chartData、originalData三種內(nèi)嵌 JS 變量import json, re from helpers import http_get def extract_chart_var(html: str, var_name: str) - list: Extract a JS array variable from Macrotrends iframe HTML. m re.search(rfvar\s{re.escape(var_name)}\s*\s*\[, html) if not m: return [] si html.index([, m.start()) bc 0 for j, ch in enumerate(html[si:], si): if ch [: bc 1 elif ch ]: bc - 1 if bc 0: return json.loads(html[si:j1]) return [] # Works for dataDaily, chartData, or originalData: html http_get(https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?tAAPLyb15) daily extract_chart_var(html, dataDaily) html2 http_get(https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id2577urlsp500-pe-ratio-price-to-earnings-chart) pe_data extract_chart_var(html2, originalData)URL 構(gòu)造速查股票類頁面STOCK_BASE https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ # Price history OHLCV f{STOCK_BASE}stock_price_history.php?t{ticker} # all history f{STOCK_BASE}stock_price_history.php?t{ticker}yb{years} # last N years # Market cap f{STOCK_BASE}market_cap.php?t{ticker}yb{years} # Fundamentals f{STOCK_BASE}fundamental_iframe.php?t{ticker}type{type}statement{stmt}freq{freq}subyb{years} # type/statement combos: pe-ratio/price-ratios, revenue/income-statement, # total-assets/balance-sheet, current-ratio/ratios # Metrics f{STOCK_BASE}fundamental_metric.php?t{ticker}chart{metric}subyb{years} # metrics: profit-margin # Dividend yield f{STOCK_BASE}dividend_yield.php?t{ticker}yb{years}經(jīng)濟(jì) / 指數(shù)頁面# From numeric ID URL slug (read from page source or page URL) fhttps://www.macrotrends.net/assets/php/chart_iframe_comp.php?id{id}url{slug} # Economic indicator JSON API (requires Referer header) fhttps://www.macrotrends.net/economic-data/{page_id}/{freq}限流與反爬實(shí)測結(jié)論根據(jù)文檔在 2026-04-18 的實(shí)測未觀察到任何限流對(duì)同一股票 iframe 連續(xù)發(fā)起 10 次快速請求總耗時(shí) 1.8 秒沒有觸發(fā)限速、驗(yàn)證碼或 429 錯(cuò)誤默認(rèn) UAMozilla/5.0對(duì)絕大多數(shù)端點(diǎn)可用所有 iframe PHP 數(shù)據(jù)文件從未出現(xiàn) 403部分主頁面 HTML 需要 Chrome UA抓取/stocks/charts/...或/2015/...這類包裝頁遇到 403 時(shí)切換為headers {User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36}/economic-data/{id}/{freq}必須攜帶Referer把頁面 URL 放進(jìn)Referer頭缺少時(shí)請求雖然被放行但部分頁面會(huì)返回 403任何端點(diǎn)都不需要 cookie、會(huì)話或鑒權(quán) token。如果本地直連仍遇到反爬攔截還可以利用 http_get 的代理路由能力設(shè)置BROWSER_USE_API_KEY環(huán)境變量后http_get會(huì)自動(dòng)改走 fetch-use 代理提供機(jī)器人檢測規(guī)避、住宅代理與重試代碼本身無需改動(dòng)。Gotchas容易踩的坑主頁面 URL ≠ 數(shù)據(jù)頁面 URL。部分 URL 會(huì)重定向到其他內(nèi)容例如/1316/us-national-debt-by-year實(shí)際會(huì)重定向到/1316/us-national-unemployment-rate。如果返回的數(shù)據(jù)看起來不對(duì)務(wù)必檢查最終 URLr.url并把最終 URL 作為Referer。yb參數(shù)控制歷史深度yb1→ 約 250 條記錄最近一年yb15→ 約 3772 條記錄最近 15 年省略 → 全部歷史AAPL 有 11428 條記錄回溯至 1980 年對(duì)大多數(shù)查詢這也是默認(rèn)行為經(jīng)濟(jì)頁面存在兩種 iframe 模式。macrotrends.net/NNNN/slug頁面要么用chart_iframe_comp.php變量是originalData要么用generateChart/economic-data/API。抓主頁面 HTML 判斷if chart_iframe_comp.php in html: # use extract_index_chart() elif highchartsURL in html: # use get_economic_data()黃金數(shù)據(jù)有兩列價(jià)格{id: GOLDAMGBD228NLBM, date: 2026-04-01, close: 5177.19, close1: 5177.190} # close inflation-adjusted price (base year adjusts over time) # close1 nominal USD price (the raw market price)經(jīng)濟(jì) API 的頻率代碼。只有D和M在大多數(shù)序列上穩(wěn)定返回?cái)?shù)據(jù)A、Q對(duì)多數(shù)經(jīng)濟(jì)指標(biāo)返回null。優(yōu)先試D。chartData的字段隨指標(biāo)變化market_cap.php→{date, v1}v1 市值單位十億美元fundamental_iframe.phptypepe-ratio →{date, v1, v2, v3}股價(jià)、EPS、PEfundamental_iframe.phptyperevenue →{date, v1, v2, v3}TTM 營收、季度營收、同比增速 %fundamental_metric.phpchartprofit-margin →{date, v1, v2, v3}毛利率 %、營業(yè)利潤率 %、凈利率 %dividend_yield.php→{date, c, ttm_d, ttm_dy}股價(jià)、TTM 股息、TTM 股息率 %大數(shù)組必須用括號(hào)計(jì)數(shù)。股票 iframe 的var dataDaily [...]約 450KB、3772 條 OHLCV 記錄。re.DOTALL貪婪正則雖然能工作但很慢本文所有示例采用的bc括號(hào)計(jì)數(shù)法是 O(n) 線性掃描速度快得多。沒有公開的 ticker 查詢 API。如果需要查找公司 slug可訪問搜索端點(diǎn)https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?vYYYYMMDD但股票價(jià)格 iframe 只需要 ticker 符號(hào)?tAAPL不需要 slug。與 browser-harness 的協(xié)作邊界最后回到 harness 的整體使用邊界本技能文檔只解決 Macrotrends 的只讀數(shù)據(jù)提取場景全程用http_get即可。按照 SKILL.md 的指引只有當(dāng)任務(wù)升級(jí)為交互型操作例如需要在 Macrotrends 頁面上點(diǎn)擊圖表控件、滾動(dòng)加載、操作已登錄會(huì)話時(shí)才應(yīng)升級(jí)到瀏覽器模式——而此時(shí)可以從 helpers.py 中按需調(diào)用new_tab、goto_url、js、click_at_xy等 CDP 輔助函數(shù)。日常的批量抓取如用ThreadPoolExecutor并發(fā)拉取多只股票則完全落在http_get的職責(zé)范圍內(nèi)既不消耗瀏覽器資源也不會(huì)觸發(fā)任何瀏覽器級(jí)反爬?！久赓M(fèi)下載鏈接】browser-harnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.項(xiàng)目地址: https://gitcode.com/gh_mirrors/br/browser-harness創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考