Paper-grade results · 2026-06-15

Sprint 2 Main Matrix 결과

S3/S4 backbone과 full/partial/ArcFace/LoRA 조건을 같은 fold 정책으로 비교한 40-run main matrix 결과다. 이 문서는 결론을 성급히 확정하기보다, 어떤 결과를 main evidence로 쓸 수 있고 무엇이 아직 deployment/runtime pending인지 분리한다.

이번 결과의 판정

보관된 metric run52

W&B export + metrics.json 기준

main matrix40/40

8 configs x 5 folds 완료

screening/legacy12

결론용 비교에서 분리 보존

primary selection99.05%

mobileclip2_s4_server_full_ce_hardneg

연구 질문과 가설

질문

MobileCLIP2-S3와 S4 중 어떤 backbone이 우리 데이터셋에서 더 안정적인 Top-1 retrieval 성능을 내는가?

같은 backbone 안에서 full unfreeze, partial unfreeze, ArcFace, LoRA 중 어떤 학습 방식이 더 적합한가?

가설

작은 데이터셋에서는 full unfreeze가 representation을 더 직접적으로 조정할 수 있지만, overfit 위험도 함께 커진다.

LoRA는 가볍지만 현재 데이터와 목적에서는 full/partial보다 낮을 수 있으며, ArcFace는 class margin에는 도움을 줄 수 있으나 retrieval 전체 성능은 검증이 필요하다.

Validation Top-1 mean 비교

Primary metric은 validation Top-1 mean이다. Test Top-1은 reference로 함께 보되, 모델 선택 규칙을 test에 맞춰 뒤집지 않는다.

s4_server_full_ce_hardneg

99.05%

s3_server_full_ce_hardneg

98.85%

s3_server_partial_ce_hardneg

98.74%

s3_server_partial_arcface_hardneg

98.55%

s4_server_partial_arcface_hardneg

98.17%

s4_server_partial_ce_hardneg

97.84%

s4_server_lora_ce_hardneg

96.67%

s3_server_lora_ce_hardneg

95.81%

핵심 그래프 2개로 보는 보조 판단

논문식 결과 해석에서는 Top-1 평균만으로 끝내지 않는다. macro F1은 class imbalance와 약한 class를 함께 반영하고, low-margin count는 앱에서 ambiguous가 늘어날 위험을 보여준다.

Test macro F1 mean

높을수록 좋다. class별 균형 성능을 보는 보조 지표다.

S3 partial ArcFace

97.67%

S3 full CE

97.41%

S3 partial CE

97.16%

S4 full CE

97.11%

S4 partial ArcFace

96.41%

S4 partial CE

94.70%

S4 LoRA

93.21%

S3 LoRA

91.36%

Test low-margin total

낮을수록 좋다. 후보 점수가 붙어 ambiguous가 늘어날 수 있는 위험 신호다.

S3 full CE

S4 full CE

S3 partial CE

S3 partial ArcFace

S4 partial ArcFace

S4 partial CE

S4 LoRA

S3 LoRA

Config별 랭킹 요약

요약 영역은 순위와 핵심 지표만 빠르게 읽도록 카드형으로 정리했다. 전체 fold별 raw 값은 아래 표에서 검산한다.

S4 full CE + hard negative

s4_server_full

Val Top-1 mean99.05%std 0.15%

Test Top-1 mean98.67%std 0.11%

Test macro F197.11%5 folds

Low-margin6test total

Best by validation fold0

run name 보기

mobileclip2_s4_server_full_ce_hardneg_fold0_20260612_003526

S3 full CE + hard negative

s3_server_full

Val Top-1 mean98.85%std 0.42%

Test Top-1 mean98.73%std 0.28%

Test macro F197.41%5 folds

Low-margin4test total

Best by validation fold3

run name 보기

mobileclip2_s3_server_full_ce_hardneg_fold3_20260611_214421

S3 partial CE + hard negative

s3_server_partial

Val Top-1 mean98.74%std 0.47%

Test Top-1 mean98.63%std 0.17%

Test macro F197.16%5 folds

Low-margin8test total

Best by validation fold1

run name 보기

mobileclip2_s3_server_partial_ce_hardneg_fold1_20260612_110637

S3 partial ArcFace + hard negative

s3_server_partial_arcface

Val Top-1 mean98.55%std 0.35%

Test Top-1 mean98.56%std 0.14%

Test macro F197.67%5 folds

Low-margin8test total

Best by validation fold1

run name 보기

mobileclip2_s3_server_partial_arcface_hardneg_fold1_20260614_083236

S4 partial ArcFace + hard negative

s4_server_partial_arcface

Val Top-1 mean98.17%std 0.46%

Test Top-1 mean98.09%std 0.39%

Test macro F196.41%5 folds

Low-margin18test total

Best by validation fold1

run name 보기

mobileclip2_s4_server_partial_arcface_hardneg_fold1_20260614_232304

S4 partial CE + hard negative

s4_server_partial

Val Top-1 mean97.84%std 0.84%

Test Top-1 mean97.61%std 0.91%

Test macro F194.70%5 folds

Low-margin34test total

Best by validation fold1

run name 보기

mobileclip2_s4_server_partial_ce_hardneg_fold1_20260614_165542

S4 LoRA CE + hard negative

s4_server_lora

Val Top-1 mean96.67%std 0.73%

Test Top-1 mean96.59%std 0.74%

Test macro F193.21%5 folds

Low-margin49test total

Best by validation fold1

run name 보기

mobileclip2_s4_server_lora_ce_hardneg_fold1_20260615_052650

S3 LoRA CE + hard negative

s3_server_lora

Val Top-1 mean95.81%std 0.41%

Test Top-1 mean95.92%std 0.40%

Test macro F191.36%5 folds

Low-margin68test total

Best by validation fold1

run name 보기

mobileclip2_s3_server_lora_ce_hardneg_fold1_20260614_124202

현재 해석

validation mean 기준: mobileclip2_s4_server_full_ce_hardneg가 가장 높다.
macro F1 기준: mobileclip2_s3_server_partial_arcface_hardneg가 가장 높지만, validation mean은 4위이므로 최종 우열은 단일 보조 지표만으로 뒤집지 않는다.
low-margin 기준: mobileclip2_s3_server_full_ce_hardneg가 가장 낮고, mobileclip2_s4_server_full_ce_hardneg도 안정권이다. LoRA 계열은 low-margin risk가 크다.
single run 기준: validation Top-1 최고 run은 mobileclip2_s3_server_full_ce_hardneg_fold3_20260611_214421이다. 단일 fold 최고값은 배포 후보 판단에는 참고값이며, 논문식 결론은 5-fold 평균을 우선한다.
앱 후보와의 관계: 기존 앱 후보 mobileclip2_s3_server_full_ce_hardneg_fold3_20260611_214421은 single-run validation이 높고 이미 artifact화된 후보지만, 40-run matrix의 평균 비교 결과와 별도로 runtime/parity 검증이 필요하다.
text retrieval: 현재 metric의 text_query_retrieval.query_count는 0이므로, 텍스트 검색 평가는 이 표에서 결론 내리지 않는다.

Per-class 약점 후보

상위 2개 config의 test per-class F1 평균에서 낮은 class를 먼저 본다. 이 목록은 데이터/라벨/혼동쌍 점검 후보이지, 단독 원인 판정은 아니다.

S4 full CE + hard negative 낮은 test F1 class

class_id	mean test F1	fold count
`deoksugung_daehanmun`	71.67%	5
`cheonggyecheon`	92.87%	5
`changgyeonggung_myeongjeongmun`	93.24%	5
`changgyeonggung_myeongjeongjeon`	93.38%	5
`changgyeonggung`	95.83%	5
`gyeongbokgung_geunjeongjeon`	95.93%	5
`naksan_park`	98.09%	5
`myeongdong_cathedral`	98.32%	5

S3 full CE + hard negative 낮은 test F1 class

class_id	mean test F1	fold count
`deoksugung_daehanmun`	80.95%	5
`changgyeonggung_myeongjeongmun`	90.54%	5
`cheonggyecheon`	91.36%	5
`changgyeonggung_myeongjeongjeon`	92.05%	5
`changgyeonggung`	95.77%	5
`gyeongbokgung_geunjeongjeon`	96.54%	5
`myeongdong_cathedral`	98.20%	5
`deoksugung_hamnyeongjeon`	98.81%	5

Fold-level raw table

논문 표에는 평균과 표준편차만 들어가도, 내부 기록에는 fold별 수치를 남겨 재현과 검산이 가능해야 한다.

config	fold	run	epoch	val Top-1	val Top-3	test Top-1	test Top-3	test macro F1	test low margin
`mobileclip2_s3_server_full_ce_hardneg`	0	`mobileclip2_s3_server_full_ce_hardneg_fold0_20260611_173848`	8	98.55%	99.73%	98.54%	99.69%	97.20%	2
`mobileclip2_s3_server_full_ce_hardneg`	1	`mobileclip2_s3_server_full_ce_hardneg_fold1_20260611_190041`	22	99.09%	99.36%	99.17%	99.79%	98.74%	1
`mobileclip2_s3_server_full_ce_hardneg`	2	`mobileclip2_s3_server_full_ce_hardneg_fold2_20260611_202226`	19	98.26%	99.36%	98.75%	99.58%	97.68%	0
`mobileclip2_s3_server_full_ce_hardneg`	3	`mobileclip2_s3_server_full_ce_hardneg_fold3_20260611_214421`	11	99.26%	99.82%	98.75%	99.79%	96.86%	1
`mobileclip2_s3_server_full_ce_hardneg`	4	`mobileclip2_s3_server_full_ce_hardneg_fold4_20260611_230857`	28	99.07%	99.81%	98.44%	99.69%	96.54%	0
`mobileclip2_s3_server_lora_ce_hardneg`	0	`mobileclip2_s3_server_lora_ce_hardneg_fold0_20260614_115759`	28	95.37%	98.82%	96.05%	99.27%	92.60%	17
`mobileclip2_s3_server_lora_ce_hardneg`	1	`mobileclip2_s3_server_lora_ce_hardneg_fold1_20260614_124202`	19	96.36%	98.55%	96.05%	99.17%	90.24%	15
`mobileclip2_s3_server_lora_ce_hardneg`	2	`mobileclip2_s3_server_lora_ce_hardneg_fold2_20260614_132535`	27	95.61%	98.81%	96.15%	98.65%	90.41%	12
`mobileclip2_s3_server_lora_ce_hardneg`	3	`mobileclip2_s3_server_lora_ce_hardneg_fold3_20260614_140937`	26	95.58%	98.80%	95.21%	98.75%	89.49%	11
`mobileclip2_s3_server_lora_ce_hardneg`	4	`mobileclip2_s3_server_lora_ce_hardneg_fold4_20260614_145413`	26	96.11%	98.33%	96.15%	98.96%	94.06%	13
`mobileclip2_s3_server_partial_arcface_hardneg`	0	`mobileclip2_s3_server_partial_arcface_hardneg_fold0_20260614_074157`	27	98.37%	99.46%	98.34%	99.58%	97.13%	1
`mobileclip2_s3_server_partial_arcface_hardneg`	1	`mobileclip2_s3_server_partial_arcface_hardneg_fold1_20260614_083236`	21	98.91%	99.36%	98.65%	99.69%	97.95%	0
`mobileclip2_s3_server_partial_arcface_hardneg`	2	`mobileclip2_s3_server_partial_arcface_hardneg_fold2_20260614_092247`	21	98.08%	99.45%	98.65%	99.58%	97.65%	0
`mobileclip2_s3_server_partial_arcface_hardneg`	3	`mobileclip2_s3_server_partial_arcface_hardneg_fold3_20260614_101401`	23	98.90%	99.45%	98.65%	99.90%	97.61%	4
`mobileclip2_s3_server_partial_arcface_hardneg`	4	`mobileclip2_s3_server_partial_arcface_hardneg_fold4_20260614_110518`	16	98.52%	99.44%	98.54%	99.58%	97.99%	3
`mobileclip2_s3_server_partial_ce_hardneg`	0	`mobileclip2_s3_server_partial_ce_hardneg_fold0_20260612_094129`	27	98.91%	99.64%	98.44%	99.79%	96.56%	2
`mobileclip2_s3_server_partial_ce_hardneg`	1	`mobileclip2_s3_server_partial_ce_hardneg_fold1_20260612_110637`	28	99.18%	99.64%	98.75%	99.79%	97.11%	1
`mobileclip2_s3_server_partial_ce_hardneg`	2	`mobileclip2_s3_server_partial_ce_hardneg_fold2_20260612_115459`	28	97.99%	99.09%	98.75%	99.69%	97.15%	1
`mobileclip2_s3_server_partial_ce_hardneg`	3	`mobileclip2_s3_server_partial_ce_hardneg_fold3_20260612_124245`	25	98.99%	99.36%	98.44%	99.58%	97.10%	3
`mobileclip2_s3_server_partial_ce_hardneg`	4	`mobileclip2_s3_server_partial_ce_hardneg_fold4_20260612_133018`	19	98.61%	99.72%	98.75%	99.79%	97.86%	1
`mobileclip2_s4_server_full_ce_hardneg`	0	`mobileclip2_s4_server_full_ce_hardneg_fold0_20260612_003526`	11	99.18%	99.55%	98.54%	99.48%	97.08%	1
`mobileclip2_s4_server_full_ce_hardneg`	1	`mobileclip2_s4_server_full_ce_hardneg_fold1_20260612_022316`	26	99.00%	99.45%	98.65%	99.58%	96.96%	1
`mobileclip2_s4_server_full_ce_hardneg`	2	`mobileclip2_s4_server_full_ce_hardneg_fold2_20260612_041147`	24	98.81%	99.36%	98.65%	99.58%	97.12%	2
`mobileclip2_s4_server_full_ce_hardneg`	3	`mobileclip2_s4_server_full_ce_hardneg_fold3_20260612_055920`	20	99.17%	99.36%	98.65%	99.58%	96.81%	1
`mobileclip2_s4_server_full_ce_hardneg`	4	`mobileclip2_s4_server_full_ce_hardneg_fold4_20260612_074252`	30	99.07%	99.91%	98.86%	99.69%	97.58%	1
`mobileclip2_s4_server_lora_ce_hardneg`	0	`mobileclip2_s4_server_lora_ce_hardneg_fold0_20260615_043734`	19	95.82%	98.64%	95.84%	98.96%	92.07%	10
`mobileclip2_s4_server_lora_ce_hardneg`	1	`mobileclip2_s4_server_lora_ce_hardneg_fold1_20260615_052650`	26	97.64%	99.00%	96.98%	99.17%	92.13%	11
`mobileclip2_s4_server_lora_ce_hardneg`	2	`mobileclip2_s4_server_lora_ce_hardneg_fold2_20260615_061625`	26	97.17%	98.54%	97.19%	98.75%	95.50%	9
`mobileclip2_s4_server_lora_ce_hardneg`	3	`mobileclip2_s4_server_lora_ce_hardneg_fold3_20260615_070528`	30	96.41%	98.71%	95.73%	98.75%	92.28%	11
`mobileclip2_s4_server_lora_ce_hardneg`	4	`mobileclip2_s4_server_lora_ce_hardneg_fold4_20260615_075358`	27	96.30%	98.70%	97.19%	99.58%	94.09%	8
`mobileclip2_s4_server_partial_arcface_hardneg`	0	`mobileclip2_s4_server_partial_arcface_hardneg_fold0_20260614_220534`	22	98.55%	99.27%	97.92%	98.96%	95.33%	4
`mobileclip2_s4_server_partial_arcface_hardneg`	1	`mobileclip2_s4_server_partial_arcface_hardneg_fold1_20260614_232304`	28	98.64%	99.27%	98.44%	99.06%	97.43%	4
`mobileclip2_s4_server_partial_arcface_hardneg`	2	`mobileclip2_s4_server_partial_arcface_hardneg_fold2_20260615_004042`	24	97.53%	99.18%	97.92%	98.86%	97.10%	4
`mobileclip2_s4_server_partial_arcface_hardneg`	3	`mobileclip2_s4_server_partial_arcface_hardneg_fold3_20260615_015906`	17	97.88%	98.90%	97.61%	98.75%	94.45%	1
`mobileclip2_s4_server_partial_arcface_hardneg`	4	`mobileclip2_s4_server_partial_arcface_hardneg_fold4_20260615_031748`	25	98.24%	98.89%	98.54%	99.27%	97.73%	5
`mobileclip2_s4_server_partial_ce_hardneg`	0	`mobileclip2_s4_server_partial_ce_hardneg_fold0_20260614_153813`	16	98.09%	99.00%	97.40%	99.17%	94.88%	4
`mobileclip2_s4_server_partial_ce_hardneg`	1	`mobileclip2_s4_server_partial_ce_hardneg_fold1_20260614_165542`	28	98.36%	98.91%	97.92%	99.38%	95.08%	3
`mobileclip2_s4_server_partial_ce_hardneg`	2	`mobileclip2_s4_server_partial_ce_hardneg_fold2_20260614_181251`	4	96.34%	98.72%	96.15%	97.92%	90.83%	20
`mobileclip2_s4_server_partial_ce_hardneg`	3	`mobileclip2_s4_server_partial_ce_hardneg_fold3_20260614_193007`	8	98.16%	99.45%	98.54%	99.69%	97.17%	4
`mobileclip2_s4_server_partial_ce_hardneg`	4	`mobileclip2_s4_server_partial_ce_hardneg_fold4_20260614_204710`	27	98.24%	99.63%	98.02%	99.38%	95.55%	3

Screening / legacy로 분리한 run

아래 run은 보존하지만 main matrix 결론에는 직접 섞지 않는다. 설정명, unfreeze 범위, 실행 시점이 main matrix와 다르기 때문이다.

config	run count	category
`mobileclip2_s3_server_full`	5	screening / legacy
`mobileclip2_s4_multitask_lora_ce_hardneg`	1	screening / legacy
`mobileclip2_s4_multitask_partial_unfreeze_arcface_hardneg`	5	screening / legacy
`mobileclip2_s4_multitask_partial_unfreeze_ce_hardneg`	1	screening / legacy

재현성과 남은 검증

covered W&B export 폴더와 각 run의 metrics.json을 source로 남겼다.
covered 40-run main matrix와 12개 screening/legacy run을 분리했다.
pending dataset fingerprint/checksum, split manifest hash, exact commit hash는 별도 artifact ledger에서 보강해야 한다.
pending ONNX/FP16 export parity, Android runtime latency, text semantic query regression은 deployment/runtime 평가 문서에서 따로 검증한다.

기계가 읽을 수 있는 요약 파일: sprint2-main-matrix-results-2026-06-15.summary.json