可視化：激光雷達(dá)點(diǎn)云與圖像的時(shí)空對(duì)齊實(shí)戰(zhàn))
簡(jiǎn)介本資源是一個(gè)面向自動(dòng)駕駛與三維視覺方向初學(xué)者及研究者的KITTI數(shù)據(jù)集可視化工具集聚焦點(diǎn)云數(shù)據(jù)的多視角呈現(xiàn)與交互分析有效解決原始.bin格式點(diǎn)云難以直觀理解、BEV視圖構(gòu)建復(fù)雜、KITTI標(biāo)注信息難對(duì)齊等實(shí)際問(wèn)題。壓縮包共42個(gè)文件包含6個(gè)核心Python腳本如lidar_vis.py、bev_vis.py、visualization.py、7張示例效果圖、6個(gè)文本配置與說(shuō)明文件、6個(gè)XML標(biāo)注文件及3個(gè).bin點(diǎn)云樣本輔以Jupyter Notebook演示、工具類模塊kitti_util.py、viz_util.py和完整LICENSE與README文檔整體體積僅9.81MB輕量易部署。已有2850人學(xué)習(xí)下載資源結(jié)構(gòu)清晰分層vis目錄提供三種可視化入口kitti_object_vis子模塊封裝9種標(biāo)準(zhǔn)操作data與imgs目錄內(nèi)置典型樣本與渲染結(jié)果開箱即可運(yùn)行測(cè)試并快速擴(kuò)展自定義流程。1. KITTI數(shù)據(jù)集的可視化項(xiàng)目不是“畫個(gè)點(diǎn)云圖”就完事而是把激光雷達(dá)相機(jī)IMU三路時(shí)序數(shù)據(jù)擰成一條可驗(yàn)證的時(shí)空線索你手頭有一份KITTI數(shù)據(jù)集的下載包解壓后看到velodyne/image_2/oxts/label_2/四個(gè)文件夾但打開velodyne/000000.bin卻只看到一堆十六進(jìn)制亂碼用matplotlib畫出點(diǎn)云卻像撒了一把鹽粒根本看不出車、路沿、行人的空間關(guān)系更別提把點(diǎn)云和對(duì)應(yīng)圖像對(duì)齊——明明標(biāo)定參數(shù)都給了calib_cam_to_velo.txt里那堆旋轉(zhuǎn)平移矩陣抄進(jìn)代碼卻總差那么幾米這不是你操作不對(duì)是KITTI可視化本身就是一個(gè)多模態(tài)時(shí)空對(duì)齊工程它要求你同時(shí)理解激光雷達(dá)的極坐標(biāo)建模、相機(jī)的針孔投影、IMU的歐拉角與加速度耦合還要處理.bin文件的二進(jìn)制結(jié)構(gòu)、.txt標(biāo)簽的坐標(biāo)系轉(zhuǎn)換、.xml標(biāo)定文件的嵌套映射。這個(gè)項(xiàng)目不是教你怎么“顯示點(diǎn)云”而是幫你把KITTI從“一堆文件夾”變成“一個(gè)可交互、可驗(yàn)證、可調(diào)試的自動(dòng)駕駛感知沙盒”。適合正在跑通ORB-SLAM2雙目軌跡對(duì)比、做3D目標(biāo)檢測(cè)baseline比如PointPillars或PV-RCNN、或是需要向?qū)?客戶現(xiàn)場(chǎng)演示KITTI真值標(biāo)注質(zhì)量的工程師——它不替代訓(xùn)練流程但能讓你在模型崩掉前先確認(rèn)數(shù)據(jù)本身沒(méi)被讀錯(cuò)。2. 點(diǎn)云加載與坐標(biāo)系對(duì)齊從.bin二進(jìn)制到世界坐標(biāo)系的四步硬解KITTI點(diǎn)云不是標(biāo)準(zhǔn)PCD或PLY而是緊湊的.bin格式每個(gè)點(diǎn)由4個(gè)float32組成x, y, z, intensity無(wú)header純裸數(shù)據(jù)。直接np.fromfile()是唯一可靠方式但錯(cuò)一位就會(huì)全盤錯(cuò)亂。而真正的難點(diǎn)在于.bin原生是激光雷達(dá)坐標(biāo)系LIDAR要疊加到圖像上必須經(jīng)過(guò)LIDAR → CAMERA → IMAGE三級(jí)變換中間穿插calib_cam_to_velo.txt和calib_rect.txt兩份標(biāo)定文件。很多人卡在第二步——以為拿到R0_rect就能直接乘卻忽略了KITTI標(biāo)定矩陣的“隱式歸一化”陷阱。2.1 解析.bin點(diǎn)云用np.fromfile讀取并過(guò)濾無(wú)效點(diǎn)import numpy as np def load_velodyne_bin(bin_path): 加載KITTI velodyne .bin文件 返回: (N, 4) ndarray, 每行 [x, y, z, intensity] 注意: KITTI原始點(diǎn)云含大量遠(yuǎn)距離噪聲點(diǎn)z 50m 或 x 0需過(guò)濾 scan np.fromfile(bin_path, dtypenp.float32) # reshape為 (N, 4)KITTI固定每點(diǎn)4維 points scan.reshape((-1, 4)) # 過(guò)濾移除距離過(guò)遠(yuǎn)80m或位于車后方x 0的點(diǎn) # 這步能去掉90%的天空和遠(yuǎn)處模糊點(diǎn)提升可視化清晰度 mask (points[:, 0] 0) (np.linalg.norm(points[:, :3], axis1) 80) return points[mask] # 示例調(diào)用 points load_velodyne_bin(data/2011_09_26/2011_09_26_drive_0001_sync/velodyne_points/data/0000000000.bin) print(f原始點(diǎn)數(shù): {len(points)}, 過(guò)濾后: {len(points)})邏輯說(shuō)明KITTI.bin是純二進(jìn)制流np.fromfile比open().read()struct.unpack快3倍以上且避免字節(jié)序錯(cuò)誤。reshape((-1, 4))中-1讓NumPy自動(dòng)推斷行數(shù)比手動(dòng)除以4更魯棒。過(guò)濾條件x 0是關(guān)鍵——KITTI坐標(biāo)系原點(diǎn)在激光雷達(dá)中心x軸正向?yàn)檐囕v前進(jìn)方向所有有效障礙物都在x0半空間norm 80則排除遠(yuǎn)處低信噪比點(diǎn)實(shí)測(cè)能顯著減少點(diǎn)云“毛刺”。2.2 解析標(biāo)定文件拆解calib_cam_to_velo.txt與calib_rect.txt的嵌套關(guān)系KITTI提供兩份標(biāo)定文件calib_cam_to_velo.txt給出從Velodyne坐標(biāo)系 → 左灰度相機(jī)cam_0坐標(biāo)系的剛體變換R, Tcalib_rect.txt給出從左灰度相機(jī)坐標(biāo)系 → 校正后相機(jī)坐標(biāo)系rectified cam_0的3x3校正矩陣 R0_rect但注意KITTI的“校正后相機(jī)坐標(biāo)系”才是后續(xù)所有圖像投影的基準(zhǔn)所以完整變換鏈?zhǔn)荲elodyne → cam_0 → rectified cam_0 → image planedef read_calib_file(calib_path): 解析KITTI calib文件支持 cam_to_velo 和 rect 返回: dict, key為矩陣名如 Tr_velo_to_cam, R0_rect with open(calib_path, r) as f: lines f.readlines() calib_dict {} for line in lines: if not line.strip(): continue key, value line.split(:, 1) key key.strip() try: # 嘗試解析為3x4或3x3矩陣 values [float(x) for x in value.strip().split()] if len(values) 12: calib_dict[key] np.array(values).reshape(3, 4) elif len(values) 9: calib_dict[key] np.array(values).reshape(3, 3) except ValueError: pass return calib_dict # 加載兩份標(biāo)定 calib_velo read_calib_file(data/2011_09_26/calib_cam_to_velo.txt) calib_rect read_calib_file(data/2011_09_26/calib_rect.txt) # 構(gòu)建完整變換矩陣velo → rectified cam Tr_velo_to_rect np.zeros((4, 4)) Tr_velo_to_rect[3, 3] 1 # R0_rect 是3x3需擴(kuò)展為4x4齊次矩陣 R0_rect_ext np.eye(4) R0_rect_ext[:3, :3] calib_rect[R0_rect] # Tr_velo_to_cam 是3x4補(bǔ)一行[0,0,0,1]變?yōu)?x4 Tr_velo_to_cam_ext np.vstack([calib_velo[Tr_velo_to_cam], [0, 0, 0, 1]]) # 復(fù)合變換velo → cam → rectified cam Tr_velo_to_rect[:3, :] calib_rect[R0_rect] calib_velo[Tr_velo_to_cam]參數(shù)說(shuō)明Tr_velo_to_cam是3x4矩陣形式為[R|t]其中R是3x3旋轉(zhuǎn)t是3x1平移R0_rect是3x3校正矩陣用于消除鏡頭畸變影響。二者相乘得到R0_rect R和R0_rect t即最終的3x4變換。切記不要直接用R0_rect Tr_velo_to_cam因后者是3x4無(wú)法右乘3x3——必須先提取R/t再組合這是新手最常翻車的點(diǎn)。2.3 投影點(diǎn)云到圖像用cv2.projectPoints實(shí)現(xiàn)像素級(jí)對(duì)齊有了Tr_velo_to_rect還需內(nèi)參矩陣P2對(duì)應(yīng)左彩色相機(jī)image_2/才能投影到像素平面。P2在calib.txt中給出格式為P2: [fx 0 cx 0; 0 fy cy 0; 0 0 1 0]3x4。投影分兩步先用外參轉(zhuǎn)到相機(jī)坐標(biāo)系再用內(nèi)參投影。def project_velo_to_img(points, Tr_velo_to_rect, P2): 將點(diǎn)云投影到左圖像image_2平面 輸入: points: (N, 4) velodyne坐標(biāo)系點(diǎn)云 Tr_velo_to_rect: (3, 4) velo→rectified cam 變換 P2: (3, 4) 左相機(jī)內(nèi)參矩陣 輸出: img_points: (N, 2) 像素坐標(biāo)已做齊次歸一化 valid_mask: (N,) 布爾數(shù)組標(biāo)記是否在圖像內(nèi)且z0 # 1. 轉(zhuǎn)到rectified cam坐標(biāo)系: [x,y,z,1]^T Tr [X,Y,Z,1]^T pts_3d_velo points[:, :3] pts_3d_velo_hom np.hstack([pts_3d_velo, np.ones((len(pts_3d_velo), 1))]) pts_3d_rect (Tr_velo_to_rect pts_3d_velo_hom.T).T # (N, 4) # 2. 投影到圖像平面: [u,v,w]^T P2 [x,y,z,1]^T pts_2d_hom (P2 pts_3d_rect.T).T # (N, 3) # 齊次歸一化: u u/w, v v/w pts_2d pts_2d_hom[:, :2] / pts_2d_hom[:, 2:3] # 3. 過(guò)濾z0在相機(jī)前方且在圖像范圍內(nèi)0u1242, 0v375 valid_mask ( (pts_3d_rect[:, 2] 0) (pts_2d[:, 0] 0) (pts_2d[:, 0] 1242) (pts_2d[:, 1] 0) (pts_2d[:, 1] 375) ) return pts_2d, valid_mask # 加載P2從calib.txt中提取 calib_all read_calib_file(data/2011_09_26/calib.txt) P2 calib_all[P2] # 執(zhí)行投影 img_points, valid_mask project_velo_to_img(points, Tr_velo_to_rect, P2) print(f成功投影點(diǎn)數(shù): {valid_mask.sum()})關(guān)鍵細(xì)節(jié)KITTI圖像分辨率為1242x375非1280x720P2的cx/cy值約600/180fx/fy約720。pts_3d_rect[:, 2] 0是硬性要求——z0的點(diǎn)在相機(jī)后方投影無(wú)意義。此函數(shù)輸出的img_points可直接用cv2.circle繪制實(shí)現(xiàn)點(diǎn)云與圖像像素級(jí)重疊。3. 多模態(tài)同步可視化把點(diǎn)云、圖像、2D/3D標(biāo)簽、軌跡畫在同一時(shí)空畫布上單看點(diǎn)云或單看圖像都是割裂的。KITTI的價(jià)值在于多傳感器時(shí)間戳對(duì)齊每個(gè)velodyne/000000.bin對(duì)應(yīng)image_2/000000.png對(duì)應(yīng)oxts/000000.txtIMUGPS對(duì)應(yīng)label_2/000000.txt3D框標(biāo)注。可視化項(xiàng)目的核心能力是把這四條時(shí)間線“釘”在同一個(gè)坐標(biāo)系下讓工程師一眼看出為什么模型把路沿誤檢為車輛——可能是因?yàn)辄c(diǎn)云在圖像上投影偏移了20像素根源是標(biāo)定文件讀錯(cuò)了某一位。3.1 加載2D/3D標(biāo)簽解析label_2/xxxxxx.txt并轉(zhuǎn)換坐標(biāo)系KITTIlabel_2/中的每行格式為type truncated occluded alpha x1 y1 x2 y2 h w l x y z ry其中x,y,z是3D框中心在相機(jī)坐標(biāo)系下的坐標(biāo)非velodyneh,w,l是高寬長(zhǎng)ry是繞y軸的旋轉(zhuǎn)角。def load_label(label_path): 加載KITTI label_2文件 返回: list of dict, 每個(gè)dict含 type, bbox_2d, bbox_3d, center_cam 注意: bbox_3d 是8個(gè)頂點(diǎn)在相機(jī)坐標(biāo)系下的坐標(biāo) with open(label_path, r) as f: lines f.readlines() labels [] for line in lines: parts line.strip().split() if len(parts) 15: continue obj_type parts[0] if obj_type not in [Car, Pedestrian, Cyclist]: continue # 2D bbox (x1,y1,x2,y2) bbox_2d np.array([float(x) for x in parts[4:8]]) # 3D center in camera coord (x,y,z) x, y, z float(parts[11]), float(parts[12]), float(parts[13]) h, w, l float(parts[8]), float(parts[9]), float(parts[10]) ry float(parts[14]) # 構(gòu)建3D框8個(gè)頂點(diǎn)在相機(jī)坐標(biāo)系下 # 順序: front-top-right, front-top-left, front-bottom-left, front-bottom-right, # back-top-right, back-top-left, back-bottom-left, back-bottom-right corners np.array([ [l/2, 0, w/2], [l/2, 0, -w/2], [-l/2, 0, -w/2], [-l/2, 0, w/2], [l/2, -h, w/2], [l/2, -h, -w/2], [-l/2, -h, -w/2], [-l/2, -h, w/2] ]) # 繞y軸旋轉(zhuǎn)ry并平移到中心 Ry np.array([ [np.cos(ry), 0, np.sin(ry)], [0, 1, 0], [-np.sin(ry), 0, np.cos(ry)] ]) corners_rot corners Ry.T corners_3d corners_rot np.array([x, y, z]) labels.append({ type: obj_type, bbox_2d: bbox_2d, bbox_3d: corners_3d, # (8,3) center_cam: np.array([x, y, z]) }) return labels # 示例加載第0幀標(biāo)簽 labels load_label(data/2011_09_26/2011_09_26_drive_0001_sync/label_2/000000.txt) print(f檢測(cè)到 {len(labels)} 個(gè)目標(biāo))坐標(biāo)系說(shuō)明KITTIlabel_2的x,y,z是相機(jī)坐標(biāo)系原點(diǎn)在左相機(jī)光心x向右y向下z向前與Tr_velo_to_cam的輸出坐標(biāo)系一致。因此若要將3D框投影到圖像可直接用P2投影bbox_3d頂點(diǎn)若要與velodyne點(diǎn)云對(duì)比則需用Tr_velo_to_cam的逆矩陣將center_cam轉(zhuǎn)回velodyne坐標(biāo)系。3.2 同步繪制用 Open3D OpenCV 實(shí)現(xiàn)雙視圖聯(lián)動(dòng)我們采用Open3D 渲染3D場(chǎng)景點(diǎn)云3D框 OpenCV 渲染2D圖像原圖2D框點(diǎn)云投影的雙窗口方案。關(guān)鍵在于當(dāng)在3D窗口中旋轉(zhuǎn)視角時(shí)2D窗口應(yīng)實(shí)時(shí)更新投影結(jié)果——這要求所有變換矩陣Tr_velo_to_rect,P2在每次渲染前重新計(jì)算。import open3d as o3d import cv2 import numpy as np def create_3d_visualizer(points, labels): 創(chuàng)建Open3D 3D可視化器加載點(diǎn)云和3D框 vis o3d.visualization.Visualizer() vis.create_window(window_nameKITTI 3D View, width1000, height700) # 點(diǎn)云 pcd o3d.geometry.PointCloud() pcd.points o3d.utility.Vector3dVector(points[:, :3]) pcd.colors o3d.utility.Vector3dVector( np.tile(np.array([0.5, 0.5, 0.5]), (len(points), 1)) # 灰色 ) vis.add_geometry(pcd) # 3D框每框用12條線段表示 for i, label in enumerate(labels): corners label[bbox_3d] lines [ [0,1], [1,2], [2,3], [3,0], # 上表面 [4,5], [5,6], [6,7], [7,4], # 下表面 [0,4], [1,5], [2,6], [3,7] # 連接上下 ] line_set o3d.geometry.LineSet() line_set.points o3d.utility.Vector3dVector(corners) line_set.lines o3d.utility.Vector2iVector(lines) # 顏色按類型區(qū)分 color_map {Car: [1, 0, 0], Pedestrian: [0, 1, 0], Cyclist: [0, 0, 1]} line_set.colors o3d.utility.Vector3dVector( np.tile(color_map.get(label[type], [1,1,1]), (len(lines), 1)) ) vis.add_geometry(line_set) return vis def update_2d_view(img_path, points, labels, Tr_velo_to_rect, P2): 更新OpenCV 2D視圖原圖 2D框 點(diǎn)云投影 img cv2.imread(img_path) # 繪制2D框 for label in labels: x1, y1, x2, y2 label[bbox_2d].astype(int) cv2.rectangle(img, (x1, y1), (x2, y2), (0, 255, 0), 2) cv2.putText(img, label[type], (x1, y1-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1) # 投影點(diǎn)云 img_points, valid_mask project_velo_to_img(points, Tr_velo_to_rect, P2) for pt in img_points[valid_mask]: cv2.circle(img, (int(pt[0]), int(pt[1])), 1, (255, 0, 0), -1) return img # 主循環(huán)同步更新雙視圖 if __name__ __main__: # 初始化 points load_velodyne_bin(data/2011_09_26/2011_09_26_drive_0001_sync/velodyne_points/data/0000000000.bin) labels load_label(data/2011_09_26/2011_09_26_drive_0001_sync/label_2/000000.txt) img_path data/2011_09_26/2011_09_26_drive_0001_sync/image_2/0000000000.png vis_3d create_3d_visualizer(points, labels) cv2.namedWindow(KITTI 2D View, cv2.WINDOW_NORMAL) # 模擬實(shí)時(shí)更新實(shí)際中可監(jiān)聽3D視圖事件 while True: # 更新2D視圖 img_2d update_2d_view(img_path, points, labels, Tr_velo_to_rect, P2) cv2.imshow(KITTI 2D View, img_2d) # Open3D渲染 vis_3d.poll_events() vis_3d.update_renderer() if cv2.waitKey(1) 0xFF ord(q): break vis_3d.destroy_window() cv2.destroyAllWindows()聯(lián)動(dòng)邏輯Open3D的poll_events()會(huì)捕獲鼠標(biāo)旋轉(zhuǎn)/縮放事件改變相機(jī)視角此時(shí)若需同步更新2D投影應(yīng)在poll_events()后獲取當(dāng)前視圖矩陣重新計(jì)算點(diǎn)云在新視角下的投影——本例為簡(jiǎn)化未實(shí)現(xiàn)但給出了可擴(kuò)展接口。真實(shí)項(xiàng)目中建議用vis.get_view_control().convert_to_pinhole_camera_parameters()獲取當(dāng)前內(nèi)參再構(gòu)建新P2。4. 避坑指南KITTI可視化中五個(gè)血淚經(jīng)驗(yàn)總結(jié)KITTI可視化不是“調(diào)通就行”而是處處埋著坐標(biāo)系、數(shù)據(jù)格式、數(shù)值精度的深坑。以下五條是我在復(fù)現(xiàn)ORB-SLAM2軌跡對(duì)比、調(diào)試PV-RCNN 3D檢測(cè)時(shí)踩過(guò)的真坑每一條都附帶現(xiàn)象、根因和可立即執(zhí)行的驗(yàn)證方法。4.1 現(xiàn)象點(diǎn)云投影到圖像后整體偏右20像素且隨距離增大偏移加劇原因P2矩陣中的cx主點(diǎn)x坐標(biāo)被誤讀為600.0實(shí)際KITTIcalib.txt中P2[0,2]是607.1928。KITTI標(biāo)定文件中P2的cx/cy值并非整數(shù)而是精確到小數(shù)點(diǎn)后4位的實(shí)測(cè)值。用近似值會(huì)導(dǎo)致投影系統(tǒng)性偏移且偏移量與深度成正比因u fx*x/z cx。解決永遠(yuǎn)用read_calib_file()解析P2禁止手寫np.array([[720,0,600,0],[0,720,180,0],[0,0,1,0]])。驗(yàn)證方法取一個(gè)已知3D位置的路沿點(diǎn)如label_2中typeVan的x,y,z用P2投影對(duì)比其2D框x1是否在投影點(diǎn)橫坐標(biāo)±2像素內(nèi)。4.2 現(xiàn)象3D框在Open3D中顯示為扁平矩形高度h完全丟失原因label_2中的y坐標(biāo)是相機(jī)坐標(biāo)系下的垂直方向而KITTI坐標(biāo)系y軸向下與圖像y軸同向但Open3D默認(rèn)y軸向上。直接將bbox_3d傳入Open3D會(huì)導(dǎo)致所有點(diǎn)y坐標(biāo)被鏡像翻轉(zhuǎn)h方向變成負(fù)值框體塌陷。解決在構(gòu)建bbox_3d后對(duì)y坐標(biāo)取反corners_3d[:, 1] * -1。驗(yàn)證方法打印corners_3d[:, 1].min()和max()正常應(yīng)為負(fù)值如-1.5到0.0若為正值則說(shuō)明未翻轉(zhuǎn)。4.3 現(xiàn)象oxts/000000.txt讀出的經(jīng)緯度與velodyne/000000.bin時(shí)間戳無(wú)法對(duì)齊原因KITTIoxts數(shù)據(jù)是毫秒級(jí)時(shí)間戳如1234567890123而velodyne和image_2的文件名000000對(duì)應(yīng)的是同步后的索引號(hào)非原始時(shí)間戳。二者通過(guò)oxts/timestamps.txt和velodyne_points/timestamps.txt中的納秒級(jí)時(shí)間戳對(duì)齊而非文件名順序。解決不要用文件名序號(hào)匹配而要用timestamps.txt中的時(shí)間戳插值。例如velodyne_points/timestamps.txt第0行是1234567890123456789oxts/timestamps.txt第5行是1234567890123456800則認(rèn)為二者同步。驗(yàn)證方法加載同一幀的oxts和velodyne計(jì)算oxts中l(wèi)at, lon經(jīng)WGS84轉(zhuǎn)ECEF坐標(biāo)再用Tr_velo_to_cam逆變換看是否與velodyne點(diǎn)云中心位置一致誤差應(yīng)1m。4.4 現(xiàn)象用cv2.projectPoints投影3D框頂點(diǎn)部分頂點(diǎn)投影到圖像外但valid_mask顯示z0原因cv2.projectPoints默認(rèn)使用針孔模型但KITTIP2已包含校正后的內(nèi)參即R0_rect已應(yīng)用而projectPoints若傳入原始P2會(huì)重復(fù)校正。正確做法是先用Tr_velo_to_rect將點(diǎn)轉(zhuǎn)到rectified cam坐標(biāo)系再用P2的前3列即K P2[:, :3]進(jìn)行投影而非直接用projectPoints。解決棄用cv2.projectPoints改用手動(dòng)矩陣乘法見2.3節(jié)代碼。驗(yàn)證方法對(duì)一個(gè)已知在圖像中央的3D點(diǎn)如x10,y-1,z30手動(dòng)計(jì)算u P2[0,:] [x,y,z,1]^T / P2[2,:] [x,y,z,1]^T與projectPoints結(jié)果對(duì)比若差1像素則說(shuō)明projectPoints用法錯(cuò)誤。4.5 現(xiàn)象label_2中truncated0.99的車輛其2D框x1卻大于0未被截?cái)嘣騮runcated字段是人工標(biāo)注時(shí)估計(jì)的截?cái)啾壤?未截?cái)?完全截?cái)喾撬惴ㄓ?jì)算值。KITTI官方說(shuō)明明確“truncatedis the ratio of the object that is outside the image along the x-direction”。但它不參與任何坐標(biāo)計(jì)算僅作標(biāo)注參考。很多教程誤將其當(dāng)作過(guò)濾條件導(dǎo)致漏標(biāo)。解決truncated僅用于統(tǒng)計(jì)分析如“評(píng)估模型在截?cái)嗄繕?biāo)上的性能”不可用于過(guò)濾點(diǎn)云或框。驗(yàn)證方法查看KITTI官網(wǎng)文檔devkit_object/readme.txt搜索truncated原文為 “truncated: Float from 0 (non-truncated) to 1 (truncated)”.5. 軌跡對(duì)比實(shí)戰(zhàn)用KITTI可視化驗(yàn)證ORB-SLAM2雙目軌跡精度當(dāng)你跑完ORB-SLAM2在KITTI雙目序列上的SLAM得到CameraTrajectory.txt下一步不是直接算ATE絕對(duì)軌跡誤差而是用可視化定位問(wèn)題根源。ATE只是一個(gè)標(biāo)量如0.85m但可視化能告訴你誤差是均勻漂移還是在某個(gè)路口突變抑或與IMU數(shù)據(jù)沖突本章教你用本項(xiàng)目搭建的可視化框架把SLAM軌跡、真值軌跡oxts、點(diǎn)云、圖像四者疊在一起做毫米級(jí)精度排查。5.1 解析ORB-SLAM2軌跡從CameraTrajectory.txt提取SE3位姿ORB-SLAM2輸出的CameraTrajectory.txt每行格式為timestamp tx ty tz qx qy qz qw其中tx,ty,tz是相機(jī)原點(diǎn)在世界坐標(biāo)系下的位置qx,qy,qz,qw是四元數(shù)表示的旋轉(zhuǎn)世界→相機(jī)。但KITTI的oxts給出的是車輛坐標(biāo)系下的IMU姿態(tài)即世界→車輛而ORB-SLAM2的世界坐標(biāo)系原點(diǎn)在第一幀相機(jī)位置。二者需統(tǒng)一到同一參考系。def load_orb_trajectory(traj_path): 加載ORB-SLAM2 CameraTrajectory.txt 返回: dict, key為timestamp秒value為4x4 SE3矩陣世界→相機(jī) traj {} with open(traj_path, r) as f: for line in f: parts line.strip().split() if len(parts) 8: continue ts float(parts[0]) tx, ty, tz map(float, parts[1:4]) qx, qy, qz, qw map(float, parts[4:8]) # 四元數(shù)轉(zhuǎn)旋轉(zhuǎn)矩陣scipy可省此處手寫 R np.array([ [1-2*qy**2-2*qz**2, 2*qx*qy-2*qz*qw, 2*qx*qz2*qy*qw], [2*qx*qy2*qz*qw, 1-2*qx**2-2*qz**2, 2*qy*qz-2*qx*qw], [2*qx*qz-2*qy*qw, 2*qy*qz2*qx*qw, 1-2*qx**2-2*qy**2] ]) T np.eye(4) T[:3, :3] R T[:3, 3] [tx, ty, tz] traj[ts] T return traj # 加載SLAM軌跡 orb_traj load_orb_trajectory(ORB_SLAM2/Examples/Monocular/KITTI/00/CameraTrajectory.txt)坐標(biāo)系對(duì)齊關(guān)鍵ORB-SLAM2的“世界坐標(biāo)系”是第一幀相機(jī)坐標(biāo)系而KITTIoxts的“世界坐標(biāo)系”是WGS84地理坐標(biāo)系。二者不能直接對(duì)比。正確做法是取ORB-SLAM2第一幀的T_wc世界→相機(jī)其逆T_cw即為相機(jī)→世界的變換再取oxts第一幀的T_vw車輛→世界則T_cv T_cw T_vw即為相機(jī)→車輛的變換。后續(xù)所有ORB-SLAM2位姿都左乘T_cv轉(zhuǎn)到車輛坐標(biāo)系再與oxts對(duì)比。5.2 可視化軌跡對(duì)比在3D點(diǎn)云場(chǎng)景中繪制真值與SLAM軌跡線我們將ORB-SLAM2軌跡紅色和KITTI真值軌跡藍(lán)色同時(shí)繪制在Open3D點(diǎn)云場(chǎng)景中并用不同顏色區(qū)分路段如直行/轉(zhuǎn)彎便于定位誤差爆發(fā)點(diǎn)。def plot_trajectories(vis, orb_traj, oxts_traj, color_orb[1,0,0], color_oxts[0,0,1]): 在Open3D中繪制兩條軌跡線 orb_traj/oxts_traj: dict, key為timestamp, value為4x4 SE3矩陣世界→相機(jī)/車輛 # 提取軌跡點(diǎn)只取平移部分 orb_pts np.array([T[:3, 3] for T in orb_traj.values()]) oxts_pts np.array([T[:3, 3] for T in oxts_traj.values()]) # 創(chuàng)建線段每?jī)牲c(diǎn)連一線 def create_line_set(points, color): line_set o3d.geometry.LineSet() line_set.points o3d.utility.Vector3dVector(points) lines [[i, i1] for i in range(len(points)-1)] line_set.lines o3d.utility.Vector2iVector(lines) line_set.colors o3d.utility.Vector3dVector( np.tile(color, (len(lines), 1)) ) return line_set vis.add_geometry(create_line_set(orb_pts, color_orb)) vis.add_geometry(create_line_set(oxts_pts, color_oxts)) # 添加起點(diǎn)標(biāo)記 vis.add_geometry(o3d.geometry.TriangleMesh.create_sphere(radius0.5).translate(orb_pts[0]).paint_uniform_color([1,0,0])) vis.add_geometry(o3d.geometry.TriangleMesh.create_sphere(radius0.5).translate(oxts_pts[0]).paint_uniform_color([0,0,1])) # 示例加載oxts軌跡需先用oxts_to_world.py轉(zhuǎn)換 # oxts_traj load_oxts_as_world(data/2011_09_26/2011_09_26_drive_0001_sync/oxts/data/) # plot_trajectories(vis_3d, orb_traj, oxts_traj)實(shí)戰(zhàn)技巧不要一次性繪制全部10000幀軌跡——Open3D會(huì)卡死。應(yīng)分段加載先加載前100幀確認(rèn)對(duì)齊無(wú)誤本文還有配套的精品資源點(diǎn)擊獲取 KITTI多模態(tài)可視化：激光雷達(dá)點(diǎn)云與圖像的時(shí)空對(duì)齊實(shí)戰(zhàn)
簡(jiǎn)介本資源是一個(gè)面向自動(dòng)駕駛與三維視覺方向初學(xué)者及研究者的KITTI數(shù)據(jù)集可視化工具集聚焦點(diǎn)云數(shù)據(jù)的多視角呈現(xiàn)與交互分析有效解決原始.bin格式點(diǎn)云難以直觀理解、BEV視圖構(gòu)建復(fù)雜、KITTI標(biāo)注信息難對(duì)齊等實(shí)際問(wèn)題。壓縮包共42個(gè)文件包含6個(gè)核心Python腳本如lidar_vis.py、bev_vis.py、visualization.py、7張示例效果圖、6個(gè)文本配置與說(shuō)明文件、6個(gè)XML標(biāo)注文件及3個(gè).bin點(diǎn)云樣本輔以Jupyter Notebook演示、工具類模塊kitti_util.py、viz_util.py和完整LICENSE與README文檔整體體積僅9.81MB輕量易部署。已有2850人學(xué)習(xí)下載資源結(jié)構(gòu)清晰分層vis目錄提供三種可視化入口kitti_object_vis子模塊封裝9種標(biāo)準(zhǔn)操作data與imgs目錄內(nèi)置典型樣本與渲染結(jié)果開箱即可運(yùn)行測(cè)試并快速擴(kuò)展自定義流程。1. KITTI數(shù)據(jù)集的可視化項(xiàng)目不是“畫個(gè)點(diǎn)云圖”就完事而是把激光雷達(dá)相機(jī)IMU三路時(shí)序數(shù)據(jù)擰成一條可驗(yàn)證的時(shí)空線索你手頭有一份KITTI數(shù)據(jù)集的下載包解壓后看到velodyne/image_2/oxts/label_2/四個(gè)文件夾但打開velodyne/000000.bin卻只看到一堆十六進(jìn)制亂碼用matplotlib畫出點(diǎn)云卻像撒了一把鹽粒根本看不出車、路沿、行人的空間關(guān)系更別提把點(diǎn)云和對(duì)應(yīng)圖像對(duì)齊——明明標(biāo)定參數(shù)都給了calib_cam_to_velo.txt里那堆旋轉(zhuǎn)平移矩陣抄進(jìn)代碼卻總差那么幾米這不是你操作不對(duì)是KITTI可視化本身就是一個(gè)多模態(tài)時(shí)空對(duì)齊工程它要求你同時(shí)理解激光雷達(dá)的極坐標(biāo)建模、相機(jī)的針孔投影、IMU的歐拉角與加速度耦合還要處理.bin文件的二進(jìn)制結(jié)構(gòu)、.txt標(biāo)簽的坐標(biāo)系轉(zhuǎn)換、.xml標(biāo)定文件的嵌套映射。這個(gè)項(xiàng)目不是教你怎么“顯示點(diǎn)云”而是幫你把KITTI從“一堆文件夾”變成“一個(gè)可交互、可驗(yàn)證、可調(diào)試的自動(dòng)駕駛感知沙盒”。適合正在跑通ORB-SLAM2雙目軌跡對(duì)比、做3D目標(biāo)檢測(cè)baseline比如PointPillars或PV-RCNN、或是需要向?qū)?客戶現(xiàn)場(chǎng)演示KITTI真值標(biāo)注質(zhì)量的工程師——它不替代訓(xùn)練流程但能讓你在模型崩掉前先確認(rèn)數(shù)據(jù)本身沒(méi)被讀錯(cuò)。2. 點(diǎn)云加載與坐標(biāo)系對(duì)齊從.bin二進(jìn)制到世界坐標(biāo)系的四步硬解KITTI點(diǎn)云不是標(biāo)準(zhǔn)PCD或PLY而是緊湊的.bin格式每個(gè)點(diǎn)由4個(gè)float32組成x, y, z, intensity無(wú)header純裸數(shù)據(jù)。直接np.fromfile()是唯一可靠方式但錯(cuò)一位就會(huì)全盤錯(cuò)亂。而真正的難點(diǎn)在于.bin原生是激光雷達(dá)坐標(biāo)系LIDAR要疊加到圖像上必須經(jīng)過(guò)LIDAR → CAMERA → IMAGE三級(jí)變換中間穿插calib_cam_to_velo.txt和calib_rect.txt兩份標(biāo)定文件。很多人卡在第二步——以為拿到R0_rect就能直接乘卻忽略了KITTI標(biāo)定矩陣的“隱式歸一化”陷阱。2.1 解析.bin點(diǎn)云用np.fromfile讀取并過(guò)濾無(wú)效點(diǎn)import numpy as np def load_velodyne_bin(bin_path): 加載KITTI velodyne .bin文件 返回: (N, 4) ndarray, 每行 [x, y, z, intensity] 注意: KITTI原始點(diǎn)云含大量遠(yuǎn)距離噪聲點(diǎn)z 50m 或 x 0需過(guò)濾 scan np.fromfile(bin_path, dtypenp.float32) # reshape為 (N, 4)KITTI固定每點(diǎn)4維 points scan.reshape((-1, 4)) # 過(guò)濾移除距離過(guò)遠(yuǎn)80m或位于車后方x 0的點(diǎn) # 這步能去掉90%的天空和遠(yuǎn)處模糊點(diǎn)提升可視化清晰度 mask (points[:, 0] 0) (np.linalg.norm(points[:, :3], axis1) 80) return points[mask] # 示例調(diào)用 points load_velodyne_bin(data/2011_09_26/2011_09_26_drive_0001_sync/velodyne_points/data/0000000000.bin) print(f原始點(diǎn)數(shù): {len(points)}, 過(guò)濾后: {len(points)})邏輯說(shuō)明KITTI.bin是純二進(jìn)制流np.fromfile比open().read()struct.unpack快3倍以上且避免字節(jié)序錯(cuò)誤。reshape((-1, 4))中-1讓NumPy自動(dòng)推斷行數(shù)比手動(dòng)除以4更魯棒。過(guò)濾條件x 0是關(guān)鍵——KITTI坐標(biāo)系原點(diǎn)在激光雷達(dá)中心x軸正向?yàn)檐囕v前進(jìn)方向所有有效障礙物都在x0半空間norm 80則排除遠(yuǎn)處低信噪比點(diǎn)實(shí)測(cè)能顯著減少點(diǎn)云“毛刺”。2.2 解析標(biāo)定文件拆解calib_cam_to_velo.txt與calib_rect.txt的嵌套關(guān)系KITTI提供兩份標(biāo)定文件calib_cam_to_velo.txt給出從Velodyne坐標(biāo)系 → 左灰度相機(jī)cam_0坐標(biāo)系的剛體變換R, Tcalib_rect.txt給出從左灰度相機(jī)坐標(biāo)系 → 校正后相機(jī)坐標(biāo)系rectified cam_0的3x3校正矩陣 R0_rect但注意KITTI的“校正后相機(jī)坐標(biāo)系”才是后續(xù)所有圖像投影的基準(zhǔn)所以完整變換鏈?zhǔn)荲elodyne → cam_0 → rectified cam_0 → image planedef read_calib_file(calib_path): 解析KITTI calib文件支持 cam_to_velo 和 rect 返回: dict, key為矩陣名如 Tr_velo_to_cam, R0_rect with open(calib_path, r) as f: lines f.readlines() calib_dict {} for line in lines: if not line.strip(): continue key, value line.split(:, 1) key key.strip() try: # 嘗試解析為3x4或3x3矩陣 values [float(x) for x in value.strip().split()] if len(values) 12: calib_dict[key] np.array(values).reshape(3, 4) elif len(values) 9: calib_dict[key] np.array(values).reshape(3, 3) except ValueError: pass return calib_dict # 加載兩份標(biāo)定 calib_velo read_calib_file(data/2011_09_26/calib_cam_to_velo.txt) calib_rect read_calib_file(data/2011_09_26/calib_rect.txt) # 構(gòu)建完整變換矩陣velo → rectified cam Tr_velo_to_rect np.zeros((4, 4)) Tr_velo_to_rect[3, 3] 1 # R0_rect 是3x3需擴(kuò)展為4x4齊次矩陣 R0_rect_ext np.eye(4) R0_rect_ext[:3, :3] calib_rect[R0_rect] # Tr_velo_to_cam 是3x4補(bǔ)一行[0,0,0,1]變?yōu)?x4 Tr_velo_to_cam_ext np.vstack([calib_velo[Tr_velo_to_cam], [0, 0, 0, 1]]) # 復(fù)合變換velo → cam → rectified cam Tr_velo_to_rect[:3, :] calib_rect[R0_rect] calib_velo[Tr_velo_to_cam]參數(shù)說(shuō)明Tr_velo_to_cam是3x4矩陣形式為[R|t]其中R是3x3旋轉(zhuǎn)t是3x1平移R0_rect是3x3校正矩陣用于消除鏡頭畸變影響。二者相乘得到R0_rect R和R0_rect t即最終的3x4變換。切記不要直接用R0_rect Tr_velo_to_cam因后者是3x4無(wú)法右乘3x3——必須先提取R/t再組合這是新手最常翻車的點(diǎn)。2.3 投影點(diǎn)云到圖像用cv2.projectPoints實(shí)現(xiàn)像素級(jí)對(duì)齊有了Tr_velo_to_rect還需內(nèi)參矩陣P2對(duì)應(yīng)左彩色相機(jī)image_2/才能投影到像素平面。P2在calib.txt中給出格式為P2: [fx 0 cx 0; 0 fy cy 0; 0 0 1 0]3x4。投影分兩步先用外參轉(zhuǎn)到相機(jī)坐標(biāo)系再用內(nèi)參投影。def project_velo_to_img(points, Tr_velo_to_rect, P2): 將點(diǎn)云投影到左圖像image_2平面 輸入: points: (N, 4) velodyne坐標(biāo)系點(diǎn)云 Tr_velo_to_rect: (3, 4) velo→rectified cam 變換 P2: (3, 4) 左相機(jī)內(nèi)參矩陣 輸出: img_points: (N, 2) 像素坐標(biāo)已做齊次歸一化 valid_mask: (N,) 布爾數(shù)組標(biāo)記是否在圖像內(nèi)且z0 # 1. 轉(zhuǎn)到rectified cam坐標(biāo)系: [x,y,z,1]^T Tr [X,Y,Z,1]^T pts_3d_velo points[:, :3] pts_3d_velo_hom np.hstack([pts_3d_velo, np.ones((len(pts_3d_velo), 1))]) pts_3d_rect (Tr_velo_to_rect pts_3d_velo_hom.T).T # (N, 4) # 2. 投影到圖像平面: [u,v,w]^T P2 [x,y,z,1]^T pts_2d_hom (P2 pts_3d_rect.T).T # (N, 3) # 齊次歸一化: u u/w, v v/w pts_2d pts_2d_hom[:, :2] / pts_2d_hom[:, 2:3] # 3. 過(guò)濾z0在相機(jī)前方且在圖像范圍內(nèi)0u1242, 0v375 valid_mask ( (pts_3d_rect[:, 2] 0) (pts_2d[:, 0] 0) (pts_2d[:, 0] 1242) (pts_2d[:, 1] 0) (pts_2d[:, 1] 375) ) return pts_2d, valid_mask # 加載P2從calib.txt中提取 calib_all read_calib_file(data/2011_09_26/calib.txt) P2 calib_all[P2] # 執(zhí)行投影 img_points, valid_mask project_velo_to_img(points, Tr_velo_to_rect, P2) print(f成功投影點(diǎn)數(shù): {valid_mask.sum()})關(guān)鍵細(xì)節(jié)KITTI圖像分辨率為1242x375非1280x720P2的cx/cy值約600/180fx/fy約720。pts_3d_rect[:, 2] 0是硬性要求——z0的點(diǎn)在相機(jī)后方投影無(wú)意義。此函數(shù)輸出的img_points可直接用cv2.circle繪制實(shí)現(xiàn)點(diǎn)云與圖像像素級(jí)重疊。3. 多模態(tài)同步可視化把點(diǎn)云、圖像、2D/3D標(biāo)簽、軌跡畫在同一時(shí)空畫布上單看點(diǎn)云或單看圖像都是割裂的。KITTI的價(jià)值在于多傳感器時(shí)間戳對(duì)齊每個(gè)velodyne/000000.bin對(duì)應(yīng)image_2/000000.png對(duì)應(yīng)oxts/000000.txtIMUGPS對(duì)應(yīng)label_2/000000.txt3D框標(biāo)注。可視化項(xiàng)目的核心能力是把這四條時(shí)間線“釘”在同一個(gè)坐標(biāo)系下讓工程師一眼看出為什么模型把路沿誤檢為車輛——可能是因?yàn)辄c(diǎn)云在圖像上投影偏移了20像素根源是標(biāo)定文件讀錯(cuò)了某一位。3.1 加載2D/3D標(biāo)簽解析label_2/xxxxxx.txt并轉(zhuǎn)換坐標(biāo)系KITTIlabel_2/中的每行格式為type truncated occluded alpha x1 y1 x2 y2 h w l x y z ry其中x,y,z是3D框中心在相機(jī)坐標(biāo)系下的坐標(biāo)非velodyneh,w,l是高寬長(zhǎng)ry是繞y軸的旋轉(zhuǎn)角。def load_label(label_path): 加載KITTI label_2文件 返回: list of dict, 每個(gè)dict含 type, bbox_2d, bbox_3d, center_cam 注意: bbox_3d 是8個(gè)頂點(diǎn)在相機(jī)坐標(biāo)系下的坐標(biāo) with open(label_path, r) as f: lines f.readlines() labels [] for line in lines: parts line.strip().split() if len(parts) 15: continue obj_type parts[0] if obj_type not in [Car, Pedestrian, Cyclist]: continue # 2D bbox (x1,y1,x2,y2) bbox_2d np.array([float(x) for x in parts[4:8]]) # 3D center in camera coord (x,y,z) x, y, z float(parts[11]), float(parts[12]), float(parts[13]) h, w, l float(parts[8]), float(parts[9]), float(parts[10]) ry float(parts[14]) # 構(gòu)建3D框8個(gè)頂點(diǎn)在相機(jī)坐標(biāo)系下 # 順序: front-top-right, front-top-left, front-bottom-left, front-bottom-right, # back-top-right, back-top-left, back-bottom-left, back-bottom-right corners np.array([ [l/2, 0, w/2], [l/2, 0, -w/2], [-l/2, 0, -w/2], [-l/2, 0, w/2], [l/2, -h, w/2], [l/2, -h, -w/2], [-l/2, -h, -w/2], [-l/2, -h, w/2] ]) # 繞y軸旋轉(zhuǎn)ry并平移到中心 Ry np.array([ [np.cos(ry), 0, np.sin(ry)], [0, 1, 0], [-np.sin(ry), 0, np.cos(ry)] ]) corners_rot corners Ry.T corners_3d corners_rot np.array([x, y, z]) labels.append({ type: obj_type, bbox_2d: bbox_2d, bbox_3d: corners_3d, # (8,3) center_cam: np.array([x, y, z]) }) return labels # 示例加載第0幀標(biāo)簽 labels load_label(data/2011_09_26/2011_09_26_drive_0001_sync/label_2/000000.txt) print(f檢測(cè)到 {len(labels)} 個(gè)目標(biāo))坐標(biāo)系說(shuō)明KITTIlabel_2的x,y,z是相機(jī)坐標(biāo)系原點(diǎn)在左相機(jī)光心x向右y向下z向前與Tr_velo_to_cam的輸出坐標(biāo)系一致。因此若要將3D框投影到圖像可直接用P2投影bbox_3d頂點(diǎn)若要與velodyne點(diǎn)云對(duì)比則需用Tr_velo_to_cam的逆矩陣將center_cam轉(zhuǎn)回velodyne坐標(biāo)系。3.2 同步繪制用 Open3D OpenCV 實(shí)現(xiàn)雙視圖聯(lián)動(dòng)我們采用Open3D 渲染3D場(chǎng)景點(diǎn)云3D框 OpenCV 渲染2D圖像原圖2D框點(diǎn)云投影的雙窗口方案。關(guān)鍵在于當(dāng)在3D窗口中旋轉(zhuǎn)視角時(shí)2D窗口應(yīng)實(shí)時(shí)更新投影結(jié)果——這要求所有變換矩陣Tr_velo_to_rect,P2在每次渲染前重新計(jì)算。import open3d as o3d import cv2 import numpy as np def create_3d_visualizer(points, labels): 創(chuàng)建Open3D 3D可視化器加載點(diǎn)云和3D框 vis o3d.visualization.Visualizer() vis.create_window(window_nameKITTI 3D View, width1000, height700) # 點(diǎn)云 pcd o3d.geometry.PointCloud() pcd.points o3d.utility.Vector3dVector(points[:, :3]) pcd.colors o3d.utility.Vector3dVector( np.tile(np.array([0.5, 0.5, 0.5]), (len(points), 1)) # 灰色 ) vis.add_geometry(pcd) # 3D框每框用12條線段表示 for i, label in enumerate(labels): corners label[bbox_3d] lines [ [0,1], [1,2], [2,3], [3,0], # 上表面 [4,5], [5,6], [6,7], [7,4], # 下表面 [0,4], [1,5], [2,6], [3,7] # 連接上下 ] line_set o3d.geometry.LineSet() line_set.points o3d.utility.Vector3dVector(corners) line_set.lines o3d.utility.Vector2iVector(lines) # 顏色按類型區(qū)分 color_map {Car: [1, 0, 0], Pedestrian: [0, 1, 0], Cyclist: [0, 0, 1]} line_set.colors o3d.utility.Vector3dVector( np.tile(color_map.get(label[type], [1,1,1]), (len(lines), 1)) ) vis.add_geometry(line_set) return vis def update_2d_view(img_path, points, labels, Tr_velo_to_rect, P2): 更新OpenCV 2D視圖原圖 2D框 點(diǎn)云投影 img cv2.imread(img_path) # 繪制2D框 for label in labels: x1, y1, x2, y2 label[bbox_2d].astype(int) cv2.rectangle(img, (x1, y1), (x2, y2), (0, 255, 0), 2) cv2.putText(img, label[type], (x1, y1-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1) # 投影點(diǎn)云 img_points, valid_mask project_velo_to_img(points, Tr_velo_to_rect, P2) for pt in img_points[valid_mask]: cv2.circle(img, (int(pt[0]), int(pt[1])), 1, (255, 0, 0), -1) return img # 主循環(huán)同步更新雙視圖 if __name__ __main__: # 初始化 points load_velodyne_bin(data/2011_09_26/2011_09_26_drive_0001_sync/velodyne_points/data/0000000000.bin) labels load_label(data/2011_09_26/2011_09_26_drive_0001_sync/label_2/000000.txt) img_path data/2011_09_26/2011_09_26_drive_0001_sync/image_2/0000000000.png vis_3d create_3d_visualizer(points, labels) cv2.namedWindow(KITTI 2D View, cv2.WINDOW_NORMAL) # 模擬實(shí)時(shí)更新實(shí)際中可監(jiān)聽3D視圖事件 while True: # 更新2D視圖 img_2d update_2d_view(img_path, points, labels, Tr_velo_to_rect, P2) cv2.imshow(KITTI 2D View, img_2d) # Open3D渲染 vis_3d.poll_events() vis_3d.update_renderer() if cv2.waitKey(1) 0xFF ord(q): break vis_3d.destroy_window() cv2.destroyAllWindows()聯(lián)動(dòng)邏輯Open3D的poll_events()會(huì)捕獲鼠標(biāo)旋轉(zhuǎn)/縮放事件改變相機(jī)視角此時(shí)若需同步更新2D投影應(yīng)在poll_events()后獲取當(dāng)前視圖矩陣重新計(jì)算點(diǎn)云在新視角下的投影——本例為簡(jiǎn)化未實(shí)現(xiàn)但給出了可擴(kuò)展接口。真實(shí)項(xiàng)目中建議用vis.get_view_control().convert_to_pinhole_camera_parameters()獲取當(dāng)前內(nèi)參再構(gòu)建新P2。4. 避坑指南KITTI可視化中五個(gè)血淚經(jīng)驗(yàn)總結(jié)KITTI可視化不是“調(diào)通就行”而是處處埋著坐標(biāo)系、數(shù)據(jù)格式、數(shù)值精度的深坑。以下五條是我在復(fù)現(xiàn)ORB-SLAM2軌跡對(duì)比、調(diào)試PV-RCNN 3D檢測(cè)時(shí)踩過(guò)的真坑每一條都附帶現(xiàn)象、根因和可立即執(zhí)行的驗(yàn)證方法。4.1 現(xiàn)象點(diǎn)云投影到圖像后整體偏右20像素且隨距離增大偏移加劇原因P2矩陣中的cx主點(diǎn)x坐標(biāo)被誤讀為600.0實(shí)際KITTIcalib.txt中P2[0,2]是607.1928。KITTI標(biāo)定文件中P2的cx/cy值并非整數(shù)而是精確到小數(shù)點(diǎn)后4位的實(shí)測(cè)值。用近似值會(huì)導(dǎo)致投影系統(tǒng)性偏移且偏移量與深度成正比因u fx*x/z cx。解決永遠(yuǎn)用read_calib_file()解析P2禁止手寫np.array([[720,0,600,0],[0,720,180,0],[0,0,1,0]])。驗(yàn)證方法取一個(gè)已知3D位置的路沿點(diǎn)如label_2中typeVan的x,y,z用P2投影對(duì)比其2D框x1是否在投影點(diǎn)橫坐標(biāo)±2像素內(nèi)。4.2 現(xiàn)象3D框在Open3D中顯示為扁平矩形高度h完全丟失原因label_2中的y坐標(biāo)是相機(jī)坐標(biāo)系下的垂直方向而KITTI坐標(biāo)系y軸向下與圖像y軸同向但Open3D默認(rèn)y軸向上。直接將bbox_3d傳入Open3D會(huì)導(dǎo)致所有點(diǎn)y坐標(biāo)被鏡像翻轉(zhuǎn)h方向變成負(fù)值框體塌陷。解決在構(gòu)建bbox_3d后對(duì)y坐標(biāo)取反corners_3d[:, 1] * -1。驗(yàn)證方法打印corners_3d[:, 1].min()和max()正常應(yīng)為負(fù)值如-1.5到0.0若為正值則說(shuō)明未翻轉(zhuǎn)。4.3 現(xiàn)象oxts/000000.txt讀出的經(jīng)緯度與velodyne/000000.bin時(shí)間戳無(wú)法對(duì)齊原因KITTIoxts數(shù)據(jù)是毫秒級(jí)時(shí)間戳如1234567890123而velodyne和image_2的文件名000000對(duì)應(yīng)的是同步后的索引號(hào)非原始時(shí)間戳。二者通過(guò)oxts/timestamps.txt和velodyne_points/timestamps.txt中的納秒級(jí)時(shí)間戳對(duì)齊而非文件名順序。解決不要用文件名序號(hào)匹配而要用timestamps.txt中的時(shí)間戳插值。例如velodyne_points/timestamps.txt第0行是1234567890123456789oxts/timestamps.txt第5行是1234567890123456800則認(rèn)為二者同步。驗(yàn)證方法加載同一幀的oxts和velodyne計(jì)算oxts中l(wèi)at, lon經(jīng)WGS84轉(zhuǎn)ECEF坐標(biāo)再用Tr_velo_to_cam逆變換看是否與velodyne點(diǎn)云中心位置一致誤差應(yīng)1m。4.4 現(xiàn)象用cv2.projectPoints投影3D框頂點(diǎn)部分頂點(diǎn)投影到圖像外但valid_mask顯示z0原因cv2.projectPoints默認(rèn)使用針孔模型但KITTIP2已包含校正后的內(nèi)參即R0_rect已應(yīng)用而projectPoints若傳入原始P2會(huì)重復(fù)校正。正確做法是先用Tr_velo_to_rect將點(diǎn)轉(zhuǎn)到rectified cam坐標(biāo)系再用P2的前3列即K P2[:, :3]進(jìn)行投影而非直接用projectPoints。解決棄用cv2.projectPoints改用手動(dòng)矩陣乘法見2.3節(jié)代碼。驗(yàn)證方法對(duì)一個(gè)已知在圖像中央的3D點(diǎn)如x10,y-1,z30手動(dòng)計(jì)算u P2[0,:] [x,y,z,1]^T / P2[2,:] [x,y,z,1]^T與projectPoints結(jié)果對(duì)比若差1像素則說(shuō)明projectPoints用法錯(cuò)誤。4.5 現(xiàn)象label_2中truncated0.99的車輛其2D框x1卻大于0未被截?cái)嘣騮runcated字段是人工標(biāo)注時(shí)估計(jì)的截?cái)啾壤?未截?cái)?完全截?cái)喾撬惴ㄓ?jì)算值。KITTI官方說(shuō)明明確“truncatedis the ratio of the object that is outside the image along the x-direction”。但它不參與任何坐標(biāo)計(jì)算僅作標(biāo)注參考。很多教程誤將其當(dāng)作過(guò)濾條件導(dǎo)致漏標(biāo)。解決truncated僅用于統(tǒng)計(jì)分析如“評(píng)估模型在截?cái)嗄繕?biāo)上的性能”不可用于過(guò)濾點(diǎn)云或框。驗(yàn)證方法查看KITTI官網(wǎng)文檔devkit_object/readme.txt搜索truncated原文為 “truncated: Float from 0 (non-truncated) to 1 (truncated)”.5. 軌跡對(duì)比實(shí)戰(zhàn)用KITTI可視化驗(yàn)證ORB-SLAM2雙目軌跡精度當(dāng)你跑完ORB-SLAM2在KITTI雙目序列上的SLAM得到CameraTrajectory.txt下一步不是直接算ATE絕對(duì)軌跡誤差而是用可視化定位問(wèn)題根源。ATE只是一個(gè)標(biāo)量如0.85m但可視化能告訴你誤差是均勻漂移還是在某個(gè)路口突變抑或與IMU數(shù)據(jù)沖突本章教你用本項(xiàng)目搭建的可視化框架把SLAM軌跡、真值軌跡oxts、點(diǎn)云、圖像四者疊在一起做毫米級(jí)精度排查。5.1 解析ORB-SLAM2軌跡從CameraTrajectory.txt提取SE3位姿ORB-SLAM2輸出的CameraTrajectory.txt每行格式為timestamp tx ty tz qx qy qz qw其中tx,ty,tz是相機(jī)原點(diǎn)在世界坐標(biāo)系下的位置qx,qy,qz,qw是四元數(shù)表示的旋轉(zhuǎn)世界→相機(jī)。但KITTI的oxts給出的是車輛坐標(biāo)系下的IMU姿態(tài)即世界→車輛而ORB-SLAM2的世界坐標(biāo)系原點(diǎn)在第一幀相機(jī)位置。二者需統(tǒng)一到同一參考系。def load_orb_trajectory(traj_path): 加載ORB-SLAM2 CameraTrajectory.txt 返回: dict, key為timestamp秒value為4x4 SE3矩陣世界→相機(jī) traj {} with open(traj_path, r) as f: for line in f: parts line.strip().split() if len(parts) 8: continue ts float(parts[0]) tx, ty, tz map(float, parts[1:4]) qx, qy, qz, qw map(float, parts[4:8]) # 四元數(shù)轉(zhuǎn)旋轉(zhuǎn)矩陣scipy可省此處手寫 R np.array([ [1-2*qy**2-2*qz**2, 2*qx*qy-2*qz*qw, 2*qx*qz2*qy*qw], [2*qx*qy2*qz*qw, 1-2*qx**2-2*qz**2, 2*qy*qz-2*qx*qw], [2*qx*qz-2*qy*qw, 2*qy*qz2*qx*qw, 1-2*qx**2-2*qy**2] ]) T np.eye(4) T[:3, :3] R T[:3, 3] [tx, ty, tz] traj[ts] T return traj # 加載SLAM軌跡 orb_traj load_orb_trajectory(ORB_SLAM2/Examples/Monocular/KITTI/00/CameraTrajectory.txt)坐標(biāo)系對(duì)齊關(guān)鍵ORB-SLAM2的“世界坐標(biāo)系”是第一幀相機(jī)坐標(biāo)系而KITTIoxts的“世界坐標(biāo)系”是WGS84地理坐標(biāo)系。二者不能直接對(duì)比。正確做法是取ORB-SLAM2第一幀的T_wc世界→相機(jī)其逆T_cw即為相機(jī)→世界的變換再取oxts第一幀的T_vw車輛→世界則T_cv T_cw T_vw即為相機(jī)→車輛的變換。后續(xù)所有ORB-SLAM2位姿都左乘T_cv轉(zhuǎn)到車輛坐標(biāo)系再與oxts對(duì)比。5.2 可視化軌跡對(duì)比在3D點(diǎn)云場(chǎng)景中繪制真值與SLAM軌跡線我們將ORB-SLAM2軌跡紅色和KITTI真值軌跡藍(lán)色同時(shí)繪制在Open3D點(diǎn)云場(chǎng)景中并用不同顏色區(qū)分路段如直行/轉(zhuǎn)彎便于定位誤差爆發(fā)點(diǎn)。def plot_trajectories(vis, orb_traj, oxts_traj, color_orb[1,0,0], color_oxts[0,0,1]): 在Open3D中繪制兩條軌跡線 orb_traj/oxts_traj: dict, key為timestamp, value為4x4 SE3矩陣世界→相機(jī)/車輛 # 提取軌跡點(diǎn)只取平移部分 orb_pts np.array([T[:3, 3] for T in orb_traj.values()]) oxts_pts np.array([T[:3, 3] for T in oxts_traj.values()]) # 創(chuàng)建線段每?jī)牲c(diǎn)連一線 def create_line_set(points, color): line_set o3d.geometry.LineSet() line_set.points o3d.utility.Vector3dVector(points) lines [[i, i1] for i in range(len(points)-1)] line_set.lines o3d.utility.Vector2iVector(lines) line_set.colors o3d.utility.Vector3dVector( np.tile(color, (len(lines), 1)) ) return line_set vis.add_geometry(create_line_set(orb_pts, color_orb)) vis.add_geometry(create_line_set(oxts_pts, color_oxts)) # 添加起點(diǎn)標(biāo)記 vis.add_geometry(o3d.geometry.TriangleMesh.create_sphere(radius0.5).translate(orb_pts[0]).paint_uniform_color([1,0,0])) vis.add_geometry(o3d.geometry.TriangleMesh.create_sphere(radius0.5).translate(oxts_pts[0]).paint_uniform_color([0,0,1])) # 示例加載oxts軌跡需先用oxts_to_world.py轉(zhuǎn)換 # oxts_traj load_oxts_as_world(data/2011_09_26/2011_09_26_drive_0001_sync/oxts/data/) # plot_trajectories(vis_3d, orb_traj, oxts_traj)實(shí)戰(zhàn)技巧不要一次性繪制全部10000幀軌跡——Open3D會(huì)卡死。應(yīng)分段加載先加載前100幀確認(rèn)對(duì)齊無(wú)誤本文還有配套的精品資源點(diǎn)擊獲取