Authors: Tiago Rodrigues, Eva Laussac, Everton Gomede · UTFPR, Cornélio Procópio, Brazil
Can a lightweight lane detector that looks at the previous frames beat the single-frame UFLD baseline? UFLD (ResNet-18) and a small CNN, each with temporal variants, trained on the ELAS ego-lane dataset and tested on three road scenes never used for training or model selection.
dataset ELAS11 models2 seedsheld-out scenes BR_S02 · VIX_S05 · VV_S03NVIDIA GeForce RTX 3050GPU time 4 h 47 minindicative pilot, not the full protocol
Where the research stands
Question: can a lightweight lane detector that also looks at the previous frames beat the single-frame UFLD baseline, and run on an in-car embedded system? Everything below comes from pilot runs on ELAS (2–3 seeds, short training, no hyper-parameter search), so it shows directions, not final claims.
Overfitting fixed0.07 → 0.86UFLD baseline, held-out lane F1: original augmentation vs + geometric augmentation (3 seeds)
Clean frames0.881 vs 0.895UFLD baseline vs the best temporal model (UFLD v0.7 ConvGRU): within seed noise (2 seeds)
Occluded current frame0.761 → 0.875UFLD baseline vs UFLD v0.3 gated when the current frame is occluded and the earlier frames are clean: +0.114, in every seed
Kalman tracker on the output−21 % jitterfor every model, with unchanged lane F1: the cheap temporal reference a learned model has to beat
Lightweight model0.850 · 2.8 MLite v0.5 lane F1 and parameters, against 0.881 with 21.8 M for the UFLD baseline (8× fewer parameters)
1 · Overfitting: what fixed it
UFLD baseline trained with each recipe (same split, seeds and epochs); held-out lane F1. Geometric augmentation (shift, zoom, rotation, perspective) is the fix; flips and heavy regularisation are not.
2 · Where time helps: an occluded current frame
Held-out lane F1 with clean frames (hollow) and with the current frame occluded by boxes while the earlier frames stay clean (filled); the number is the occluded F1 and its change. Single-frame models only see the damaged image.
3 · Accuracy against size
Held-out lane F1 (clean frames) and parameters. Compute per frame: UFLD 7.1 GMAC, lite 2.0 GMAC; a temporal model streaming with cached features adds only its fusion (it recomputes 3 frames otherwise). Lite models from elas_lite_long (longer training).
4 · How far back to look (UFLD v0.4 coord)
Held-out lane F1 of UFLD v0.4 coord minus the UFLD baseline for each history length and spacing (history frame k = t − k·step; same split and baseline for every arm). Spacing changes little; five frames are worse than three.
Desafios
Base de dados pequena. O protocolo usa 10 cenas do ELAS, e só 3 ficam reservadas para teste. A UFLD decorava a posição das faixas até entrar o aumento de dados geométrico.
Quadros limpos escondem o efeito temporal. Com quadros limpos, os modelos temporais UFLD ficam a cerca de ±0,02 de lane F1 da baseline. Duas sementes não bastam para separar isso do ruído.
Os modelos leves treinam devagar e de forma instável. Treinados do zero, precisam de 16 épocas ou mais. O lite v0.5 sem histórico real (o controle de capacidade) marcou 0,877 em uma semente e 0,627 na outra. Com quadros limpos, ainda não se sabe quanto do ganho vem dos quadros anteriores; com o quadro atual ocluído, o histórico vale +0,17 e +0,38 nas duas sementes.
Memória em sequências longas. Os modelos recorrentes derivam quando o estado é mantido ao longo de uma cena inteira (lite v0.6: −0,48 de lane F1). Eles foram treinados só com clipes de 3 quadros.
Mais histórico não é melhor. Espaçar os quadros do histórico de 2, 5 ou 10 quadros não muda nada; usar 5 quadros em vez de 3 reduz o lane F1.
Uma única base de dados até agora. CULane, TuSimple e OpenLane ainda não foram baixadas. Os leitores estão prontos e foram testados em cópias sintéticas da estrutura delas.
Ainda sem medição embarcada. Os números da Jetson são simulações com fatores de lentidão assumidos, não medições.
Próximos passos
Rodar o protocolo completo no ELAS: 6 sementes, até 50 épocas e o mesmo orçamento de busca de hiperparâmetros para todos os modelos (cerca de 4–5 dias na RTX 3050).
Tornar a robustez um resultado principal: primeiro as corrupções sintéticas, depois oclusões naturais, noite e chuva.
Baixar CULane e TuSimple (o guia de download está pronto), pré-treinar nelas, ajustar no ELAS e reportar as métricas oficiais delas.
Resolver a questão dos modelos leves: mais sementes e treino mais longo para o lite v0.5 contra seu controle de capacidade, e treinar os modelos recorrentes com sequências longas.
Última fase, embarcado: exportar o melhor modelo temporal leve para TensorRT (FP16/INT8) e medir latência, memória e consumo de energia em uma Jetson.
Challenges
Small dataset. The protocol uses 10 ELAS scenes, and only 3 are held out for testing. UFLD memorised lane positions until geometric augmentation was added.
Clean frames hide the temporal effect. On clean held-out frames the temporal UFLD models stay within about ±0.02 lane F1 of the baseline. Two seeds cannot separate that from noise.
The lightweight models train slowly and unstably. Trained from scratch, they need 16 epochs or more. Lite v0.5 without real history (the capacity control) scored 0.877 in one seed and 0.627 in the other. On clean frames it is still open how much of the gain comes from the earlier frames; with an occluded current frame, the history is worth +0.17 and +0.38 in both seeds.
Memory over long sequences. The recurrent models drift when their state is carried across a whole scene (lite v0.6: −0.48 lane F1). They were trained on 3-frame clips only.
More history is not better. Spacing the history frames 2, 5 or 10 frames apart changes nothing; 5 frames instead of 3 costs lane F1.
One dataset so far. CULane, TuSimple and OpenLane are not downloaded yet. Their readers are ready and were tested on synthetic copies of their layouts.
No embedded measurement yet. The Jetson numbers are simulations based on assumed slowdowns, not measurements.
Next steps
Run the full protocol on ELAS: 6 seeds, up to 50 epochs, and the same hyper-parameter search budget for every model (about 4–5 days on the RTX 3050).
Make robustness a primary result: the synthetic corruptions now, then natural occlusions, night and rain frames.
Download CULane and TuSimple (the download guide is ready), pre-train on them, fine-tune on ELAS, and report their official metrics.
Settle the lightweight question: more seeds and longer training for lite v0.5 against its capacity control, and train the recurrent models on long sequences.
Last phase, embedded: export the best lightweight temporal model to TensorRT (FP16/INT8) and measure latency, memory and power on a Jetson.
Qual modelo comparar com a baselineWhich model to compare with the baseline
Lane F1 nas cenas de teste nunca vistas, média de 2 sementes; o quadro atual ocluído vem da avaliação de robustez (o histórico continua limpo). Resultados de piloto: indicam a direção, não provam.Lane F1 on the held-out scenes, mean of 2 seeds; the occluded current frame comes from the robustness evaluation (the history stays clean). Pilot results: they show the direction, they do not prove it.Modelos lite: execução elas_lite_long (treino mais longo).Lite models: run elas_lite_long (longer training).
SituaçãoSituation
Supera a baseline?Better than the baseline?
Melhor modeloBest model
NúmerosNumbers
Quadros limposClean frames
Não claramenteNot clearly
nenhum aindanone yet
UFLD v0.7 ConvGRU: +0,014 (0,895 × 0,881); sem o histórico perde só 0,005, então o ganho não é temporalUFLD v0.7 ConvGRU: +0.014 (0.895 vs 0.881); without its history it loses only 0.005, so the gain is not temporal
Quadro atual ocluído (histórico limpo)Occluded current frame (clean history)
Sim, em todas as sementesYes, in every seed
UFLD v0.3 gatedUFLD v0.3 gated
UFLD v0.3 gated +0,114, UFLD v0.6 aligned +0,104, UFLD v0.7 ConvGRU +0,041 (baseline 0,761; controle com o mesmo treino 0,742)UFLD v0.3 gated +0.114, UFLD v0.6 aligned +0.104, UFLD v0.7 ConvGRU +0.041 (baseline 0.761; equal-training control 0.742)
Quadro atual borrado, escuro ou com ruídoBlurred, dark or noisy current frame
NãoNo
nenhumnone
todas as diferenças entre -0,010 e +0,024every difference between -0.010 and +0.024
Cena de teste com oclusões reaisHeld-out scene with real occlusions
Sim, mas é uma única cenaYes, but it is a single scene
Modelo leve para embarcado (× UFLD baseline)Lightweight model for embedded use (vs UFLD baseline)
Sob oclusão sim; em quadros limpos nãoUnder occlusion yes; on clean frames no
Lite v0.5 warpedLite v0.5 warped
limpos -0,031, ocluídos +0,068, 8× menos parâmetros (2,8 M × 21,8 M); treinos diferentes: até 16 × 4 épocasclean -0.031, occluded +0.068, 8× fewer parameters (2.8 M vs 21.8 M); different training: up to 16 vs 4 epochs
O modelo leve usa o histórico? (× mesmas camadas sem histórico, com oclusão)Does the lite model use its history? (vs same layers without history, occluded)
Comparação principal: UFLD v0.3 (gated) × UFLD baseline. Mesma rede e mesmo treino, com praticamente nenhum parâmetro a mais; qualquer diferença vem do uso dos quadros anteriores. É o modelo que mais mantém as faixas quando o quadro atual está ocluído, e empata com a baseline em quadros limpos.
Comparação secundária, para o embarcado: lite v0.5 × UFLD baseline, e lite v0.5 × o mesmo modelo sem histórico. Mostra o custo de um modelo muito menor e, pelo controle sem histórico, que a robustez vem do tempo.
Declarar antes da execução completa. Esta escolha foi feita olhando o teste; para a afirmação ser válida, declare agora as comparações principais e a métrica com oclusão como métrica principal, e use as 6 sementes da execução completa para confirmar ou rejeitar.
Condição para comparar as famílias: treinar UFLD e lite com o mesmo protocolo (mesmas épocas máximas e busca de hiperparâmetros) antes de colocar os dois lado a lado num artigo.
Recommendation
Primary comparison: UFLD v0.3 (gated) vs the UFLD baseline. Same network and same training, with almost no extra parameters, so any difference comes from using the earlier frames. It is the model that keeps the lanes best when the current frame is occluded, and it ties the baseline on clean frames.
Secondary comparison, for the embedded goal: lite v0.5 vs the UFLD baseline, and lite v0.5 vs the same model without history. It shows the cost of a much smaller model and, through the no-history control, that the robustness comes from time.
Declare before the full run. This choice was made by looking at the test set. For the claim to hold, declare the primary comparisons now, with occluded-frame F1 as a primary metric, and let the 6 seeds of the full run confirm or reject them.
Condition for comparing the families: train UFLD and lite under the same protocol (same maximum epochs and hyper-parameter search) before putting them side by side in a paper.
Os candidatos lado a ladoThe candidates side by side
ModeloModel
F1 limposClean F1
F1 ocluídosOccluded F1
Δ oclusão × baselineΔ occluded vs baseline
Parâmetros (M)Parameters (M)
Perda sem históricoCost without history
NotaNote
UFLD baseline
0.881
0.761
—
21.8
—
referência de um quadrosingle-frame reference
UFLD baseline +CT
0.883
0.742
-0.018
21.8
—
controle: o mesmo treino extra dos modelos temporais, sem históricocontrol: the temporal models' extra training, without history
UFLD v0.2 weighted
0.872
0.756
-0.005
21.8
0.001
sem ganho com oclusãono gain under occlusion
UFLD v0.3 gated
0.883
0.875
+0.114
21.9
0.012
recomendado: robusto à oclusão, quase sem custo extrarecommended: robust to occlusion, almost no extra cost
UFLD v0.4 coord
0.880
0.765
+0.004
21.8
0.003
sem ganho com oclusão; escolhido na validação limpa para a ablação de históricono gain under occlusion; chosen on clean validation for the history ablation
UFLD v0.6 aligned
0.880
0.864
+0.104
23.5
0.008
quase tão robusto quanto a v0.3, porém maior e mais complexonearly as robust as v0.3, but larger and more complex
UFLD v0.7 ConvGRU
0.895
0.802
+0.041
22.1
0.005
melhor em quadros limpos, mas o ganho não é temporal; o estado deriva em sequências longasbest on clean frames, but the gain is not temporal; its state drifts over long sequences
Lite baseline
0.322
0.254
-0.507
2.5
—
referência leve de um quadrolightweight single-frame reference
Lite baseline +CT
0.322
0.258
-0.503
2.5
—
controle de treino da família litetraining control of the lite family
Lite v0.5 warped
0.850
0.829
+0.068
2.8
0.004
candidato embarcado: usa o histórico sob oclusãoembedded candidate: uses its history under occlusion
Lite v0.5 static
0.752
0.554
-0.206
2.8
—
controle de capacidade: mesmas camadas, sem histórico; instávelcapacity control: same layers, no history; unstable
Lite v0.6 ConvGRU
0.561
0.554
-0.206
2.7
0.009
recorrente leve; ainda fracolightweight recurrent; still weak
“Perda sem histórico”: quanto o lane F1 cai quando o histórico é trocado pelo quadro atual no teste (quadros limpos).“Cost without history”: how much lane F1 drops when the history is replaced by the current frame at test time (clean frames).
O que os resultados mostram
O aumento de dados geométrico elimina o overfitting da UFLD. Na ablação de aumento de dados (UFLD baseline, 3 sementes, mesma divisão e mesmo orçamento), pequenos deslocamentos, zooms, rotações e mudanças de perspectiva aleatórios, aplicados ao clipe inteiro, elevam o melhor lane F1 de validação de 0,161 para 0,772 e o lane F1 nas cenas de teste nunca vistas de 0,068 para 0,858. A melhor época passa da 1 para a 4, a última. Só as mudanças fotométricas chegam a 0,150 no teste; o espelhamento não acrescenta nada (0,769). A receita do segundo piloto foi escolhida na validação: aumento geométrico mais degradação do quadro atual, para todos os modelos.
Dropout + label smoothing + weight decay juntos pioram a UFLD (0,002 no teste). As faixas continuam sendo previstas em todos os quadros, mas no lugar errado. A causa provável é o label smoothing: ele achata a distribuição sobre as 100 células da grade e puxa o soft-argmax para o centro da imagem. Como o braço muda três coisas ao mesmo tempo, isto é uma hipótese.
Nenhum outro remédio isolado resolve o overfitting. Outros cinco braços da mesma ablação mudam uma única coisa na receita original (3 sementes): taxa de aprendizado do backbone 10× menor (lane F1 no teste 0,050), stem e primeiro estágio da ResNet congelados (0,083), degradação do quadro atual (0,094), oito cenas de treino a mais (0,129) e uma cabeça de 256 unidades em vez de 2048 (0,211). Todos continuam com o melhor resultado na época 1 ou 2. Só o aumento de dados geométrico (0,858) elimina o problema, e por isso ele continua sendo a receita de todos os modelos.
Com a nova receita, a UFLD baseline chega a 0,881 de lane F1 no teste (0,849 nas cenas já vistas), contra 0,057 no primeiro piloto. O segundo piloto treinou 11 modelos × 2 sementes em 4 h 47 min na RTX 3050.
Os modelos temporais UFLD empatam com a baseline, mas não a superam com clareza. A v0.2, a v0.3, a v0.4 e a v0.6 ficam entre 0,872 e 0,883; o controle com o mesmo treino extra fica em 0,883. A UFLD v0.7 (ConvGRU) é a mais alta, com 0,895 (+0,014 sobre a baseline, +0,012 sobre o controle). Porém, trocar o histórico dela pelo quadro atual no teste custa só 0,005, então esse ganho não é claramente temporal.
Os modelos lite ficam subtreinados após 4 épocas da nova receita (eles treinam do zero): lite baseline 0,218, +CT 0,312. O lite v0.5 chega a 0,700 e o lite v0.6 a 0,541, bem acima de ambos, mas trocar o histórico pelo quadro atual custa só 0,020 e 0,017. A maior parte da vantagem vem, portanto, das camadas extras de fusão, e não do tempo.
Com 16 épocas, a família lite se aproxima da UFLD com 8× menos parâmetros. A execução lite mais longa (elas_lite_long, 2 sementes, parada antecipada) leva o lite v0.5 a 0,850 de lane F1 no teste (0,844 / 0,856) com 2,8 M de parâmetros, contra 0,881 da UFLD baseline com 21,8 M. O lite baseline fica em 0,322, e mais treino não o ajuda (+CT 0,322). O controle de capacidade (lite v0.5 com o quadro atual em todas as posições, as mesmas camadas) é instável: 0,877 na semente 1 e 0,627 na semente 2. Por isso, com quadros limpos, quanto do ganho do lite v0.5 vem dos quadros anteriores ainda está em aberto (+0,098 em média; −0,033 e +0,229 por semente). O lite v0.6 (ConvGRU) chega a 0,561.
O filtro de Kalman dá a suavidade de graça. Em todos os modelos, ele reduz o jitter (a oscilação da posição prevista entre quadros) em cerca de 20 % (UFLD baseline: de 1,51 para 1,19 px) sem alterar o lane F1. Os modelos temporais UFLD aprendidos oscilam tanto quanto a baseline (1,45 a 1,75 px contra 1,51 px); portanto, é o filtro, e não a fusão, que deixa a saída mais suave.
Manter o estado recorrente ao longo de uma cena inteira ainda não funciona. Treinada com clipes de 3 quadros, a ConvGRU deriva quando o estado nunca é reiniciado: a UFLD v0.7 perde 0,028 e o lite v0.6 perde 0,483 de lane F1 nesse modo. Para usar um único estado contínuo no carro, é preciso antes treinar com sequências longas (retropropagação truncada no tempo).
A oclusão é onde o tempo pode ajudar. Na única cena de teste marcada como "oclusão" (VIX_S05), a UFLD v0.7 marca 0,792, contra 0,741 da baseline e 0,772 do controle com o mesmo treino extra; nas outras duas cenas a baseline é a melhor (0,948). Uma única cena é anedótica.
Quando o quadro atual está ocluído, os modelos temporais mantêm as faixas. A avaliação de robustez (python -m tac_ufld robustness) reavaliou todos os checkpoints nos 2.610 quadros de teste. Em cada quadro, a imagem atual foi coberta por retângulos, borrada, escurecida ou recebeu ruído, enquanto os quadros do histórico ficaram limpos; todos os modelos viram as mesmas corrupções. Com oclusão (673 quadros), a UFLD v0.3 mantém 0,875 de lane F1 (sementes 0,865 / 0,886) e a v0.6 0,864 (0,857 / 0,871), contra 0,761 da baseline (0,753 / 0,768) e 0,742 do controle com o mesmo treino extra. São +0,11 nas duas sementes; mais treino, sozinho, não fecha a diferença. Em todos os quadros degradados, a v0.3 chega a 0,886 e a v0.6 a 0,884, contra 0,850 da baseline; para a v0.3 e a v0.6, isso fica levemente acima do F1 com quadros limpos (0,883 e 0,880). Desfoque, escurecimento e ruído quase não afetam nenhum modelo UFLD (dentro de ±0,03). A v0.7 ganha menos com oclusão (0,802), e os modelos lite perdem de 0,02 a 0,07, como suas baselines. O filtro de Kalman não acrescenta nada aqui. Ressalvas: 2 sementes, um único conjunto fixo de corrupções sintéticas, e os modelos foram treinados com degradação do quadro atual; então isto mede o que esse treino ensina, e não oclusões naturais.
Com o quadro atual ocluído, o modelo lite claramente usa o histórico. Na avaliação de robustez da execução lite longa, o lite v0.5 mantém 0,829 de lane F1 com oclusão (sementes 0,817 / 0,840, só 0,02 abaixo do valor com quadros limpos). O controle de capacidade, com as mesmas camadas mas o quadro atual em todas as posições, cai para 0,554 (0,648 / 0,461). São +0,17 e +0,38 a favor do histórico real, nas duas sementes. O lite baseline fica perto de 0,25 e o lite v0.6 perto de 0,55. Mesmas ressalvas da UFLD: 2 sementes, oclusões sintéticas.
Olhar mais para trás não ajuda; olhar mais quadros atrapalha. A ablação de histórico (ablation_history, UFLD v0.4, o modelo temporal escolhido na validação do piloto 2, 2 sementes) usa sua própria divisão, com blocos de 180 quadros, na qual a baseline marca 0,831. A v0.4 ganha +0,029 sobre a baseline com 3 quadros espaçados de 2, +0,028 espaçados de 5 e +0,027 espaçados de 10, mas só +0,004 com 5 quadros espaçados de 5. O espaçamento do histórico quase não importa nessa faixa; uma janela mais longa piora a v0.4.
Custo. Com as features do histórico em cache, cada modelo temporal UFLD custa o mesmo que sua baseline (7,1 GMACs por quadro) mais a fusão. Por clipe de 3 quadros, a v0.7 precisa de 21,5 GMACs e 22,1 M de parâmetros (+0,3 M), e a v0.6 de 21,8 GMACs e 23,5 M (+1,7 M). Os modelos lite precisam de 2,0 GMACs por quadro e de 2,5 a 2,8 M de parâmetros.
What the results show
Geometric augmentation removes the UFLD overfitting. In the augmentation ablation (UFLD baseline, 3 seeds, same split and budget), small random shifts, zooms, rotations and perspective changes applied to the whole clip lift the best validation lane F1 from 0.161 to 0.772 and the held-out lane F1 from 0.068 to 0.858. The best epoch moves from 1 to 4, the last one. Photometric changes alone reach 0.150 held-out; flips add nothing (0.769). The recipe of the second pilot was chosen on validation: geometric augmentation plus current-frame degradation, for every model.
Dropout + label smoothing + weight decay together make UFLD worse (held-out 0.002). Lanes are still predicted in every frame but misplaced. The likely cause is label smoothing, which flattens the distribution over the 100 grid cells and pulls the soft-argmax towards the image centre; the arm changes three things at once, so this is a hypothesis.
No other single remedy fixes the overfitting. Five more arms of the same ablation change one thing each in the original recipe (3 seeds): a 10x lower backbone learning rate (held-out lane F1 0.050), frozen stem and first ResNet stage (0.083), current-frame degradation (0.094), eight more training scenes (0.129) and a 256-unit head instead of 2048 (0.211). Every one still peaks at epoch 1 or 2. Only geometric augmentation (0.858) removes the problem, so it remains the recipe for every model.
With the new recipe the UFLD baseline reaches 0.881 held-out lane F1 (0.849 on seen scenes), against 0.057 in the first pilot. The second pilot trained 11 models × 2 seeds in 4 h 47 min on the RTX 3050.
The UFLD temporal models match the baseline but do not clearly beat it. v0.2, v0.3, v0.4 and v0.6 land between 0.872 and 0.883, the equal-training control at 0.883. UFLD v0.7 (ConvGRU) is highest at 0.895 (+0.014 over the baseline, +0.012 over the control), but replacing its history by the current frame at test time costs only 0.005, so that gain is not clearly temporal.
The lite models are under-trained after 4 epochs of the new recipe (they train from scratch): lite baseline 0.218, +CT 0.312. Lite v0.5 reaches 0.700 and lite v0.6 0.541, far above both, but replacing their history by the current frame costs only 0.020 and 0.017. Most of their lead therefore comes from the extra fusion layers, not from time.
With 16 epochs the lite family comes close to UFLD, with 8× fewer parameters. The longer lite run (elas_lite_long, 2 seeds, early stopping) puts lite v0.5 at 0.850 held-out lane F1 (0.844 / 0.856) with 2.8 M parameters, against 0.881 for the UFLD baseline with 21.8 M. The lite baseline stays at 0.322 and more training does not help it (+CT 0.322). The capacity control (lite v0.5 with the current frame in every position, the same layers) is unstable: 0.877 in seed 1 and 0.627 in seed 2. On clean frames, the share of lite v0.5's gain that comes from the earlier frames is therefore still open (+0.098 on average, −0.033 and +0.229 per seed). Lite v0.6 (ConvGRU) reaches 0.561.
The Kalman tracker gives the smoothness for free. On every model it lowers jitter by about 20 % (UFLD baseline 1.51 to 1.19 px) with unchanged lane F1. The learned temporal UFLD models are about as steady as their baseline (1.45 to 1.75 px against 1.51 px), so the tracker, not the fusion, is what makes the output smoother.
Carrying the recurrent state over a whole scene does not work yet. Trained on 3-frame clips, the ConvGRU drifts when its state is never reset: UFLD v0.7 loses 0.028 and lite v0.6 0.483 lane F1 in carry mode. One-state deployment needs training on long sequences (truncated back-propagation through time) first.
Occlusion is where time might help. In the one held-out scene tagged "occlusion" (VIX_S05), UFLD v0.7 scores 0.792 against 0.741 for the baseline and 0.772 for the equal-training control; in the other two scenes the baseline is best (0.948). One scene is anecdotal.
When the current frame is occluded, the temporal models keep the lanes. The robustness evaluation (python -m tac_ufld robustness) re-ran every checkpoint on the 2,610 held-out frames. In each frame the current image was occluded by boxes, blurred, darkened or made noisy, while the history frames stayed clean; every model saw the same corruptions. Under occlusion (673 frames), UFLD v0.3 keeps 0.875 lane F1 (seeds 0.865 / 0.886) and v0.6 0.864 (0.857 / 0.871), against 0.761 for the baseline (0.753 / 0.768) and 0.742 for the equal-training control. That is +0.11 in both seeds, so more training alone does not close the gap. Over all degraded frames, v0.3 reaches 0.886 and v0.6 0.884 against 0.850 for the baseline. For v0.3 and v0.6 that is marginally above their clean-frame F1 (0.883 and 0.880). Blur, darkening and noise barely hurt any UFLD model (within ±0.03). v0.7 gains less under occlusion (0.802), and the lite models lose 0.02–0.07 like their baselines. The Kalman tracker adds nothing here. Caveats: 2 seeds, one fixed set of synthetic corruptions, and the models were trained with current-frame degradation, so this measures what that training teaches, not natural occlusions.
With an occluded current frame, the lite model clearly uses its history. In the robustness evaluation of the longer lite run, lite v0.5 keeps 0.829 lane F1 under occlusion (seeds 0.817 / 0.840, only 0.02 below clean). The capacity control, with the same layers but the current frame in every position, falls to 0.554 (0.648 / 0.461). That is +0.17 and +0.38 for the real history, in both seeds. The lite baseline stays near 0.25 and lite v0.6 near 0.55. Same caveats as for UFLD: 2 seeds, synthetic occlusions.
Looking further back does not help; looking at more frames hurts. The history ablation (ablation_history, UFLD v0.4, the temporal model chosen on pilot-2 validation, 2 seeds) uses its own split with 180-frame blocks, where the baseline scores 0.831. It gains +0.029 over the baseline with 3 frames 2 apart, +0.028 with 5 apart and +0.027 with 10 apart, but only +0.004 with 5 frames 5 apart. The spacing of the history barely matters in this range; a longer window makes v0.4 worse.
Cost. With cached history features each temporal UFLD model costs its baseline (7.1 GMACs per frame) plus the fusion; per 3-frame clip v0.7 needs 21.5 GMACs and 22.1 M parameters (+0.3 M), v0.6 21.8 GMACs and 23.5 M (+1.7 M). The lite models need 2.0 GMACs per frame and 2.5 to 2.8 M parameters.
The models
Two families, each a single-frame baseline and temporal variants built on it. All of them read the current frame and predict the two ego lanes of that frame; the temporal ones also read earlier frames.
Every temporal model keeps its baseline's backbone and head and only adds a fusion step between them. It is compared with its own baseline on the same seeds.
Three questions separate the temporal models: do they align the frames before fusing (v0.5, v0.6), how do they remember (a fixed 3-frame window for v0.2 to v0.6, a recurrent state for v0.7 and lite v0.6), and what do they cost (UFLD about 7 GMACs per frame, lite about 2).
With feature caching each frame is encoded once, so a temporal model costs its baseline plus the fusion step, not three backbones.
The controls are not candidates. +CT separates a temporal gain from extra training, lite v0.5 static separates it from the extra fusion layers, and the Kalman tracker shows what simple output filtering already gives.
Model
Role
Frames used
How history enters
Aligns motion
Memory
Params (M)
UFLD family (ResNet-18, ImageNet)
UFLD baseline
baseline
t
none
no
none
21.78
UFLD baseline +CT
control
t
none
no
none
21.78
UFLD v0.2 weighted
temporal
t-4, t-2, t
weighted average
no
3-frame window
21.78
UFLD v0.3 gated
temporal
t-4, t-2, t
per-cell gates
no
3-frame window
21.91
UFLD v0.4 coord
temporal
t-4, t-2, t
weighted average
no
3-frame window
21.78
UFLD v0.6 aligned
temporal
t-4, t-2, t
warp + gate
yes (learned flow)
3-frame window
23.50
UFLD v0.7 ConvGRU
temporal
t-4, t-2, t
ConvGRU
no
recurrent state
22.07
Lite family (4-block CNN, from scratch)
Lite baseline
baseline
t
none
no
none
2.49
Lite baseline +CT
control
t
none
no
none
2.49
Lite v0.5 warped
temporal
t-4, t-2, t
warp + gate
yes (learned flow)
3-frame window
2.84
Lite v0.6 ConvGRU
temporal
t-4, t-2, t
ConvGRU
no
recurrent state
2.73
UFLD family
UFLD baseline
baselineOfficial UFLD (Qin et al., ECCV 2020), supervisor's notebook21.78 M params · 7.1 GMACs per clip
What it doesUltra-Fast Lane Detection with an ImageNet-pretrained ResNet-18. Lane detection is row-wise classification: for each of 18 image rows and each lane slot the head picks one of 100 horizontal cells, or "no lane". One frame in, no memory.
DifferenceThe reference of the UFLD family. Every UFLD temporal model starts from its trained weights and is compared with it on the same seeds.
UFLD baseline +CT
controlv0.4 control21.78 M params · 7.1 GMACs per clip
What it doesThe same network trained a second time from its own best checkpoint, with exactly the schedule the temporal variants get after their warm start.
DifferenceA control, not a candidate. A temporal model that beats the baseline but not this control gained from extra training, not from time.
UFLD v0.2 weighted
temporalSupervisor's notebook21.78 M params · 21.3 GMACs per clip, 3 frames per clip
What it doesEncodes the three frames with the shared ResNet and averages the feature maps with three learned weights (starting at 0.1 / 0.2 / 0.7) before the UFLD head. Trained with an extra existence loss and a temporal-consistency loss.
DifferenceThe simplest fusion: one weight per frame, the same at every pixel. Lane markings that moved between frames are averaged at different positions.
UFLD v0.3 gated
temporalSupervisor's notebook21.91 M params · 21.4 GMACs per clip, 3 frames per clip
What it doesLike v0.2, but a small convolutional network predicts a softmax gate per frame and per feature cell, biased towards the current frame.
DifferenceCan use history only where the current frame is weak, such as an occluded cell. Like v0.2 it fuses feature maps that are not aligned.
UFLD v0.4 coord
temporalSupervisor's notebook21.78 M params · 21.3 GMACs per clip, 3 frames per clip
What it doesv0.2's network with a different loss: a soft-argmax coordinate term penalises the lane position error in pixels, and the existence loss is re-weighted.
DifferenceSame fusion as v0.2. It tests whether a position-aware loss helps; the network itself is unchanged.
UFLD v0.6 aligned
temporalNew in v0.4 (port of lite v0.5)23.50 M params · 21.8 GMACs per clip, 3 frames per clip
What it doesA learned flow field (up to 4 cells of the 12x16 feature grid, about 128 px) warps each history map onto the current frame. The aligned history is compressed and blended in through a per-cell gate that starts almost closed.
DifferenceThe only UFLD model that aligns frames before fusing, so moving markings are not blurred. It adds about 1.7 M parameters to the baseline.
UFLD v0.7 ConvGRU
temporalNew in v0.422.07 M params · 21.5 GMACs per clip, 3 frames per clip
What it doesA convolutional GRU (64 channels) runs over the reduced feature maps from the oldest frame to the current one. Its final state is projected back and added to the current features. The projection starts at zero, so the untrained model is exactly the baseline.
DifferenceRemembers through a gated recurrence instead of a weighted sum. A deployment can carry one hidden state per camera instead of storing past feature maps.
Lite family
Lite baseline
baselineELAS development script2.49 M params · 2.0 GMACs per clip
What it doesA small CNN: four stride-2 convolution blocks (32 to 128 channels) and a head that pools the features to the 18x100 row-anchor grid and classifies every row with a shared MLP. Trained from scratch, without ImageNet.
DifferenceThe reference of the lite family: about 8 times fewer parameters and 3 to 4 times less compute than UFLD.
Lite baseline +CT
controlv0.4 control2.49 M params · 2.0 GMACs per clip
What it doesThe lite baseline trained a second time from its own best checkpoint with the temporal variants' schedule.
DifferenceThe equal-training control for lite v0.5 and lite v0.6.
Lite v0.5 warped
temporalELAS development script (refactored)2.84 M params · 6.0 GMACs per clip, 3 frames per clip
What it doesThe lite baseline plus warped residual fusion: a learned flow (up to 8 cells at stride 16, about 128 px) aligns each history feature map to the current one, and a per-pixel gate that starts closed mixes the aligned history in.
DifferenceSame fusion idea as UFLD v0.6 on a network that is 8 times smaller.
Lite v0.6 ConvGRU
temporalNew in v0.42.73 M params · 6.1 GMACs per clip, 3 frames per clip
What it doesThe lite baseline plus the ConvGRU fusion of UFLD v0.7 (64 hidden channels, read-out starting at zero).
DifferenceRecurrent memory on the smallest network: the candidate for an embedded system that keeps one state tensor per camera. Unlike v0.5 it does not align frames explicitly.
Reference, not a model: Any model + Kalman tracker: a causal constant-velocity Kalman filter on every lane point, with existence smoothing, tuned on validation. It costs microseconds per frame and is the bar a learned temporal model has to clear.
Lane F1 on unseen roads
CULane-style lane F1: each lane drawn 12 px wide (30 px at 1640 px, scaled to 640 px), matched one-to-one, correct when IoU ≥ 0.5. Post-processing tuned on validation only. On ELAS every model predicts both ego lanes in every frame, so precision = recall = F1: the errors are misplaced lanes, not missed ones.
Held-out test scenes (unseen roads)
Seen-scene test blocks
UFLD familyLite familybars = mean over seeds, dots = individual seeds
Held-out test scenes, all metrics (mean ± std over seeds)
Model
Lane F1 @0.5
Lane F1 @0.35
Pixel F1
Anchor F1
Jitter px
FP
FN
UFLD baseline
0.881 ± 0.001
0.962 ± 0.008
0.812 ± 0.004
0.951 ± 0.001
1.51 ± 0.01
622 ± 4
622 ± 4
UFLD baseline +CT
0.883 ± 0.005
0.963 ± 0.004
0.809 ± 0.001
0.949 ± 0.001
1.81 ± 0.00
610 ± 27
610 ± 27
UFLD v0.2 weighted
0.872 ± 0.007
0.954 ± 0.010
0.806 ± 0.007
0.948 ± 0.003
1.45 ± 0.02
670 ± 37
670 ± 37
UFLD v0.3 gated
0.883 ± 0.015
0.958 ± 0.005
0.814 ± 0.008
0.950 ± 0.001
1.58 ± 0.04
608 ± 77
608 ± 77
UFLD v0.4 coord
0.880 ± 0.023
0.954 ± 0.014
0.808 ± 0.012
0.949 ± 0.004
1.48 ± 0.15
626 ± 122
626 ± 122
UFLD v0.6 aligned
0.880 ± 0.011
0.957 ± 0.011
0.811 ± 0.009
0.949 ± 0.005
1.75 ± 0.11
626 ± 59
626 ± 59
UFLD v0.7 ConvGRU
0.895 ± 0.014
0.961 ± 0.011
0.816 ± 0.008
0.951 ± 0.004
1.63 ± 0.11
549 ± 75
549 ± 75
Lite baseline
0.218 ± 0.010
0.370 ± 0.000
0.408 ± 0.008
0.566 ± 0.019
1.77 ± 0.21
4078 ± 54
4078 ± 54
Lite baseline +CT
0.312 ± 0.061
0.482 ± 0.057
0.483 ± 0.056
0.663 ± 0.042
2.68 ± 0.12
3586 ± 317
3586 ± 317
Lite v0.5 warped
0.700 ± 0.012
0.899 ± 0.003
0.722 ± 0.000
0.903 ± 0.016
1.83 ± 0.04
1562 ± 62
1562 ± 62
Lite v0.6 ConvGRU
0.541 ± 0.011
0.754 ± 0.009
0.648 ± 0.002
0.814 ± 0.012
2.00 ± 0.01
2394 ± 56
2394 ± 56
Seen-scene test
Model
Lane F1 @0.5
Lane F1 @0.35
Pixel F1
Anchor F1
Jitter px
FP
FN
UFLD baseline
0.849 ± 0.001
0.896 ± 0.013
0.807 ± 0.003
0.933 ± 0.005
2.01 ± 0.01
190 ± 1
190 ± 1
UFLD baseline +CT
0.847 ± 0.002
0.904 ± 0.003
0.810 ± 0.001
0.936 ± 0.000
2.36 ± 0.05
192 ± 3
192 ± 3
UFLD v0.2 weighted
0.845 ± 0.005
0.896 ± 0.005
0.810 ± 0.005
0.932 ± 0.005
1.88 ± 0.16
195 ± 6
195 ± 6
UFLD v0.3 gated
0.855 ± 0.007
0.902 ± 0.008
0.817 ± 0.008
0.935 ± 0.005
2.08 ± 0.07
182 ± 9
182 ± 9
UFLD v0.4 coord
0.841 ± 0.023
0.900 ± 0.015
0.810 ± 0.015
0.929 ± 0.008
1.93 ± 0.03
200 ± 28
200 ± 29
UFLD v0.6 aligned
0.850 ± 0.003
0.898 ± 0.003
0.814 ± 0.005
0.931 ± 0.003
2.40 ± 0.28
187 ± 4
189 ± 4
UFLD v0.7 ConvGRU
0.851 ± 0.002
0.899 ± 0.008
0.815 ± 0.008
0.934 ± 0.002
2.14 ± 0.19
188 ± 2
188 ± 2
Lite baseline
0.169 ± 0.014
0.438 ± 0.028
0.451 ± 0.012
0.648 ± 0.000
1.83 ± 0.29
1044 ± 17
1044 ± 17
Lite baseline +CT
0.389 ± 0.028
0.615 ± 0.004
0.567 ± 0.005
0.741 ± 0.010
2.61 ± 0.09
768 ± 35
768 ± 35
Lite v0.5 warped
0.793 ± 0.012
0.858 ± 0.004
0.738 ± 0.011
0.906 ± 0.003
2.07 ± 0.30
260 ± 16
260 ± 16
Lite v0.6 ConvGRU
0.685 ± 0.003
0.801 ± 0.021
0.694 ± 0.002
0.865 ± 0.003
2.02 ± 0.00
396 ± 4
396 ± 4
Does temporal context help?
Each temporal model is compared with the single-frame model of its own family on the same seeds (exact Wilcoxon, Holm correction). With 2 seeds no difference can reach p < 0.05 unless there are at least 6 seeds, so the verdicts here are indicative.
Temporal model minus its single-frame baseline (held-out lane F1)
Temporal model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
UFLD v0.2 weighted
UFLD baseline
2
-0.009
[-0.081, +0.063]
0.500
1.000
underpowered
UFLD v0.3 gated
UFLD baseline
2
+0.003
[-0.138, +0.143]
1.000
1.000
underpowered
UFLD v0.4 coord
UFLD baseline
2
-0.001
[-0.218, +0.216]
1.000
1.000
underpowered
UFLD v0.6 aligned
UFLD baseline
2
-0.001
[-0.110, +0.109]
1.000
1.000
underpowered
UFLD v0.7 ConvGRU
UFLD baseline
2
+0.014
[-0.123, +0.151]
0.500
1.000
underpowered
Lite v0.5 warped
Lite baseline
2
+0.483
[+0.283, +0.683]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline
2
+0.323
[+0.133, +0.513]
0.500
1.000
underpowered
Equal-training control
Temporal models start from their baseline's best checkpoint and train further. The +CT baseline gets the same extra training; a temporal gain that survives this comparison is not an effect of more epochs.
Model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
UFLD v0.2 weighted
UFLD baseline +CT
2
-0.011
[-0.030, +0.007]
0.500
1.000
underpowered
UFLD v0.3 gated
UFLD baseline +CT
2
+0.000
[-0.086, +0.087]
1.000
1.000
underpowered
UFLD v0.4 coord
UFLD baseline +CT
2
-0.003
[-0.166, +0.160]
1.000
1.000
underpowered
UFLD v0.6 aligned
UFLD baseline +CT
2
-0.003
[-0.059, +0.053]
1.000
1.000
underpowered
UFLD v0.7 ConvGRU
UFLD baseline +CT
2
+0.012
[-0.071, +0.095]
0.500
1.000
underpowered
Lite v0.5 warped
Lite baseline +CT
2
+0.388
[-0.265, +1.042]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline +CT
2
+0.229
[-0.415, +0.872]
0.500
1.000
underpowered
UFLD baseline +CT
UFLD baseline
2
+0.002
[-0.051, +0.056]
1.000
1.000
underpowered
Lite baseline +CT
Lite baseline
2
+0.094
[-0.359, +0.548]
0.500
1.000
underpowered
Against the Kalman tracker
The same predictions filtered by the output Kalman tracker (tuned on validation). This is what time gives almost for free; a learned temporal model should beat its baseline + Kalman (paired table below).
Model
Lane F1
Lane F1 + Kalman
Jitter px
Jitter px + Kalman
UFLD baseline
0.881
0.881
1.51
1.19
UFLD baseline +CT
0.883
0.884
1.81
1.44
UFLD v0.2 weighted
0.872
0.874
1.45
1.19
UFLD v0.3 gated
0.883
0.883
1.58
1.25
UFLD v0.4 coord
0.880
0.880
1.48
1.20
UFLD v0.6 aligned
0.880
0.879
1.75
1.39
UFLD v0.7 ConvGRU
0.895
0.894
1.63
1.31
Lite baseline
0.218
0.215
1.77
1.26
Lite baseline +CT
0.312
0.309
2.68
2.11
Lite v0.5 warped
0.700
0.700
1.83
1.46
Lite v0.6 ConvGRU
0.541
0.540
2.00
1.57
Temporal model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
UFLD v0.2 weighted
UFLD baseline + Kalman
2
-0.010
[-0.089, +0.069]
0.500
1.000
underpowered
UFLD v0.3 gated
UFLD baseline + Kalman
2
+0.002
[-0.146, +0.149]
1.000
1.000
underpowered
UFLD v0.4 coord
UFLD baseline + Kalman
2
-0.002
[-0.226, +0.223]
1.000
1.000
underpowered
UFLD v0.6 aligned
UFLD baseline + Kalman
2
-0.002
[-0.119, +0.115]
1.000
1.000
underpowered
UFLD v0.7 ConvGRU
UFLD baseline + Kalman
2
+0.013
[-0.131, +0.157]
0.500
1.000
underpowered
Lite v0.5 warped
Lite baseline + Kalman
2
+0.486
[+0.321, +0.650]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline + Kalman
2
+0.326
[+0.171, +0.481]
0.500
1.000
underpowered
When the current frame is degraded
The held-out test scenes again, but every current frame is corrupted (an occluding box, strong blur, darkening or noise; the same corruption for every model) while the earlier frames stay clean. This is where earlier frames should pay off: a single-frame model only sees the damaged image. Validation-tuned post-processing, nothing re-tuned.
Held-out lane F1 with a degraded current frame
Model
Clean
Degraded
occlude
blur
darken
noise
Degraded + Kalman
UFLD baseline
0.881
0.850
0.761
0.891
0.869
0.884
0.852
UFLD baseline +CT
0.883
0.849
0.742
0.894
0.873
0.891
0.850
UFLD v0.2 weighted
0.872
0.847
0.756
0.901
0.859
0.878
0.846
UFLD v0.3 gated
0.883
0.886
0.875
0.902
0.872
0.893
0.888
UFLD v0.4 coord
0.880
0.855
0.765
0.899
0.874
0.885
0.850
UFLD v0.6 aligned
0.880
0.884
0.864
0.907
0.874
0.890
0.886
UFLD v0.7 ConvGRU
0.895
0.874
0.802
0.915
0.883
0.899
0.873
Lite baseline
0.218
0.163
0.147
0.188
0.150
0.169
0.179
Lite baseline +CT
0.312
0.243
0.231
0.259
0.233
0.250
0.261
Lite v0.5 warped
0.700
0.679
0.670
0.702
0.671
0.673
0.680
Lite v0.6 ConvGRU
0.541
0.520
0.535
0.525
0.523
0.498
0.522
Recurrent models with a carried state
Held-out lane F1 when the ConvGRU keeps one state per stream across the whole scene, minus the window mode used in training (a zero state at every clip).
Model
Carry − window
Seeds
Lite v0.6 ConvGRU
-0.4832
2
UFLD v0.7 ConvGRU
-0.0280
2
Is it the history?
Test-time ablation: history frames replaced by the current frame. The gain is what the real history contributes to each trained model.
Temporal model
F1 gain from real history
Seeds
Lite v0.5 warped
+0.0200
2
Lite v0.6 ConvGRU
+0.0173
2
UFLD v0.2 weighted
+0.0006
2
UFLD v0.3 gated
+0.0121
2
UFLD v0.4 coord
+0.0033
2
UFLD v0.6 aligned
+0.0076
2
UFLD v0.7 ConvGRU
+0.0049
2
Where temporal information should help
Held-out lane F1 per scene condition from the ELAS scene tags (whole scenes, so these are three scenes, not frame-level subsets) and jitter, the mean frame-to-frame change of the predicted lane position (lower is steadier; it also contains real lane motion).
Model
occlusion
transition
jitter px
UFLD baseline
0.741
0.948
1.51
UFLD baseline +CT
0.772
0.937
1.81
UFLD v0.2 weighted
0.742
0.934
1.45
UFLD v0.3 gated
0.771
0.938
1.58
UFLD v0.4 coord
0.766
0.935
1.48
UFLD v0.6 aligned
0.760
0.938
1.75
UFLD v0.7 ConvGRU
0.791
0.945
1.63
Lite baseline
0.060
0.294
1.77
Lite baseline +CT
0.119
0.405
2.68
Lite v0.5 warped
0.435
0.829
1.83
Lite v0.6 ConvGRU
0.250
0.681
2.00
The lite family with a longer budget
Lite family, v0.4 recipe, up to 16 epochs (patience 4), 2 seeds, no HPO. Indicative only.
Head to head on the held-out roads
Same split, recipe and evaluation; the numbers in brackets are the maximum epochs of each run (early stopping may end sooner). The lite models come from elas_lite_long, the others from elas_pilot_v2.
Held-out test scenes (mean ± std over seeds)
Model
Lane F1 @0.5
Lane F1 @0.35
Pixel F1
Anchor F1
Jitter px
FP
FN
Lite baseline
0.322 ± 0.018
0.518 ± 0.025
0.497 ± 0.009
0.676 ± 0.007
2.83 ± 0.23
3532 ± 94
3532 ± 94
Lite baseline +CT
0.322 ± 0.020
0.519 ± 0.027
0.506 ± 0.003
0.674 ± 0.006
3.32 ± 0.30
3534 ± 105
3534 ± 105
Lite v0.5 warped
0.850 ± 0.009
0.935 ± 0.013
0.788 ± 0.009
0.940 ± 0.011
2.19 ± 0.08
782 ± 47
782 ± 47
Lite v0.5 static
0.752 ± 0.177
0.876 ± 0.116
0.735 ± 0.074
0.899 ± 0.065
2.89 ± 0.63
1292 ± 923
1292 ± 923
Lite v0.6 ConvGRU
0.561 ± 0.027
0.726 ± 0.037
0.646 ± 0.015
0.811 ± 0.001
2.59 ± 0.32
2286 ± 141
2286 ± 141
Temporal model minus its single-frame baseline (held-out lane F1)
Temporal model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
Lite v0.5 warped
Lite baseline
2
+0.528
[+0.446, +0.609]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline
2
+0.239
[-0.167, +0.645]
0.500
1.000
underpowered
Equal-training control
Temporal models start from their baseline's best checkpoint and train further. The +CT baseline gets the same extra training; a temporal gain that survives this comparison is not an effect of more epochs.
Model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
Lite v0.5 warped
Lite baseline +CT
2
+0.528
[+0.427, +0.629]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline +CT
2
+0.239
[-0.186, +0.665]
0.500
1.000
underpowered
Lite baseline +CT
Lite baseline
2
-0.000
[-0.020, +0.019]
1.000
1.000
underpowered
Capacity control: extra layers or earlier frames?
Lite v0.5 static is lite v0.5 with every history frame replaced by the current frame, in training and at test time: the same layers, warm start and budget, but no temporal information. "Lite v0.5 − static" is what the earlier frames add; "static − baseline" is what the extra fusion layers add on their own.
Model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
Lite v0.5 static
Lite baseline
2
+0.430
[-1.324, +2.184]
0.500
1.000
underpowered
Lite v0.5 warped
Lite v0.5 static
2
+0.098
[-1.575, +1.770]
1.000
1.000
underpowered
Against the Kalman tracker
The same predictions filtered by the output Kalman tracker (tuned on validation). This is what time gives almost for free; a learned temporal model should beat its baseline + Kalman (paired table below).
Model
Lane F1
Lane F1 + Kalman
Jitter px
Jitter px + Kalman
Lite baseline
0.322
0.321
2.83
1.94
Lite baseline +CT
0.322
0.322
3.32
1.87
Lite v0.5 warped
0.850
0.847
2.19
1.74
Lite v0.5 static
0.752
0.750
2.89
2.31
Lite v0.6 ConvGRU
0.561
0.558
2.59
1.97
Temporal model
Compared with
Seeds
Δ lane F1
95 % CI
Wilcoxon p
Holm p
Verdict
Lite v0.5 warped
Lite baseline + Kalman
2
+0.529
[+0.434, +0.624]
0.500
1.000
underpowered
Lite v0.5 static
Lite baseline + Kalman
2
+0.432
[-1.336, +2.199]
0.500
1.000
underpowered
Lite v0.6 ConvGRU
Lite baseline + Kalman
2
+0.241
[-0.179, +0.660]
0.500
1.000
underpowered
When the current frame is degraded
The held-out test scenes again, but every current frame is corrupted (an occluding box, strong blur, darkening or noise; the same corruption for every model) while the earlier frames stay clean. This is where earlier frames should pay off: a single-frame model only sees the damaged image. Validation-tuned post-processing, nothing re-tuned.
Held-out lane F1 with a degraded current frame
Model
Clean
Degraded
occlude
blur
darken
noise
Degraded + Kalman
Lite baseline
0.322
0.261
0.254
0.258
0.263
0.268
0.277
Lite baseline +CT
0.322
0.271
0.258
0.270
0.287
0.272
0.287
Lite v0.5 warped
0.850
0.842
0.829
0.866
0.836
0.838
0.840
Lite v0.5 static
0.752
0.679
0.554
0.731
0.741
0.697
0.688
Lite v0.6 ConvGRU
0.561
0.563
0.554
0.560
0.591
0.547
0.561
Recurrent models with a carried state
Held-out lane F1 when the ConvGRU keeps one state per stream across the whole scene, minus the window mode used in training (a zero state at every clip).
Model
Carry − window
Seeds
Lite v0.6 ConvGRU
-0.1540
2
Is it the history?
Test-time ablation: history frames replaced by the current frame. The gain is what the real history contributes to each trained model.
Temporal model
F1 gain from real history
Seeds
Lite v0.5 warped
+0.0042
2
Lite v0.6 ConvGRU
+0.0086
2
Fixing the UFLD overfitting
The first pilot showed every UFLD run peaking at epoch 1. Each arm below changes one thing in the training of the UFLD baseline (same split, seeds, epochs and evaluation) and is compared with the original recipe (photometric augmentation). The recipe for later runs is chosen on the validation column; the held-out column is reported, never used for choosing. "Best epochs" are the validation-selected epochs per seed; the "more scenes" arm has its own, larger validation set. Three seeds cannot reach significance, so the verdicts are indicative.
Held-out lane F1 per arm
Arm
Validation lane F1
Held-out lane F1
Best epochs
Held-out Δ vs reference
geometric
0.772 ± 0.007
0.858 ± 0.002
4, 4, 4
+0.790 underpowered
geometric flip
0.768 ± 0.001
0.769 ± 0.016
4, 4, 4
+0.701 underpowered
more scenes (18)
0.288 ± 0.032
0.129 ± 0.023
1, 2, 2
+0.061 underpowered
small head (256)
0.248 ± 0.026
0.210 ± 0.023
1, 1, 1
+0.142 underpowered
photometric extended
0.175 ± 0.007
0.150 ± 0.016
1, 1, 1
+0.082 underpowered
frozen stem + layer1
0.170 ± 0.021
0.083 ± 0.012
1, 1, 1
+0.015 underpowered
current-frame degradation
0.163 ± 0.020
0.094 ± 0.040
1, 1, 1
+0.026 underpowered
photometric
0.161 ± 0.034
0.068 ± 0.044
2, 2, 1
reference
none
0.150 ± 0.022
0.068 ± 0.043
2, 1, 1
+0.000 underpowered
backbone lr ×0.1
0.149 ± 0.014
0.050 ± 0.004
1, 1, 1
-0.019 underpowered
regularised
0.019 ± 0.023
0.002 ± 0.003
1, 1, 1
-0.066 underpowered
How far back to look
History-length ablation: every arm uses the same split (purge gap large enough for the longest history) and the same trained baseline, so only the temporal context changes. Cells are means over seeds on the held-out scenes; "not run" arms can be added with ablate --only.
UFLD v0.4 coord: lane F1 gain over UFLD baseline
UFLD v0.4 coord: held-out lane F1
Training curves
Temporal models and +CT controls start from their family baseline's best checkpoint.
UFLD baseline
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD baseline +CT
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD v0.2 weighted
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD v0.3 gated
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD v0.4 coord
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD v0.6 aligned
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
UFLD v0.7 ConvGRU
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
Lite baseline
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
Lite baseline +CT
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
Lite v0.5 warped
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
Lite v0.6 ConvGRU
validation lane F1 per epoch; solid = first seed, dashed = second, dot = best epoch (the checkpoint kept)
Streaming on a held-out road
240 consecutive frames of scene BR_S02 (never seen in training), run frame by frame through the streaming detector: left UFLD baseline, right Lite v0.5 with cached history features (first pilot checkpoints). Red = ego-left slot, yellow = ego-right slot, numbers = mean existence probability of each lane.
Latency and real-time budgets
Measured at batch 1 on NVIDIA GeForce RTX 3050 with 640×480 frames of the held-out clip (first pilot checkpoints). "cached" reuses the stored features of previous frames; "recompute" re-encodes all three frames every time. Preprocessing (a PIL bilinear resize on the CPU, identical to training) is the largest cost.
preprocessmodelpostprocess
Simulated 30 FPS camera: service times = measured × slowdown. The Jetson slowdowns are assumptions from peak-throughput ratios, not measurements on a Jetson. "History fallbacks" are frames whose history frames were skipped because the worker was busy.
Profile
Slowdown
Model
Mode
Processed FPS
p95 latency ms
History fallbacks
Budgets
desktop
×1.0
UFLD baseline
single
30.0
22.3
0 %
met
desktop
×1.0
UFLD v0.2 weighted
cached
30.0
22.0
0 %
met
desktop
×1.0
UFLD v0.2 weighted
recompute
30.0
30.1
0 %
met
desktop
×1.0
UFLD v0.3 gated
cached
30.0
22.6
0 %
met
desktop
×1.0
UFLD v0.3 gated
recompute
30.0
30.3
0 %
met
desktop
×1.0
UFLD v0.4 coord
cached
30.0
21.7
0 %
met
desktop
×1.0
UFLD v0.4 coord
recompute
30.0
38.5
0 %
met
desktop
×1.0
Lite baseline
single
30.0
18.2
0 %
met
desktop
×1.0
Lite v0.5 warped
cached
30.0
24.1
0 %
met
desktop
×1.0
Lite v0.5 warped
recompute
30.0
33.7
0 %
met
jetson_agx_orin_assumed
×1.7
UFLD baseline
single
29.0
50.4
0 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.2 weighted
cached
28.7
59.1
4 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.2 weighted
recompute
24.9
75.4
18 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.3 gated
cached
28.4
61.3
5 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.3 gated
recompute
24.2
77.2
21 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.4 coord
cached
29.0
51.6
3 %
missed
jetson_agx_orin_assumed
×1.7
UFLD v0.4 coord
recompute
25.2
74.2
17 %
missed
jetson_agx_orin_assumed
×1.7
Lite baseline
single
29.0
33.8
0 %
met
jetson_agx_orin_assumed
×1.7
Lite v0.5 warped
cached
27.2
68.8
10 %
missed
jetson_agx_orin_assumed
×1.7
Lite v0.5 warped
recompute
20.5
85.8
33 %
missed
jetson_orin_nano_assumed
×7.0
UFLD baseline
single
7.8
186.5
0 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.2 weighted
cached
7.6
196.5
82 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.2 weighted
recompute
6.1
218.1
85 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.3 gated
cached
7.4
200.6
79 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.3 gated
recompute
5.8
228.7
92 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.4 coord
cached
7.5
202.6
79 %
missed
jetson_orin_nano_assumed
×7.0
UFLD v0.4 coord
recompute
5.9
240.6
87 %
missed
jetson_orin_nano_assumed
×7.0
Lite baseline
single
9.4
168.7
0 %
missed
jetson_orin_nano_assumed
×7.0
Lite v0.5 warped
cached
7.7
199.3
80 %
missed
jetson_orin_nano_assumed
×7.0
Lite v0.5 warped
recompute
5.8
243.9
87 %
missed
Export and precision checks
Each exported model runs the same frames as PyTorch through the same streaming code (first pilot checkpoints). Val lane F1 is measured on 240 validation frames with the exact training-time temporal context; the difference to PyTorch FP32 is in brackets. TensorRT ran on the development GPU, not on a Jetson.
Model
Backend
max |Δ logits|
max |Δ exist|
mean lane Δx px
Val lane F1
Numerics
Lite v0.5 warped
onnxruntime fp32
7.6e-06
3.6e-07
0.00
0.5636 (+0.0000)
within tolerance
Lite v0.5 warped
onnxruntime int8
7.6e-06
6.4e-02
1.36
0.5570 (-0.0066)
outside tolerance
Lite v0.5 warped
tensorrt fp32
1.8e-03
1.2e-04
0.00
0.5636 (+0.0000)
outside tolerance
Lite v0.5 warped
tensorrt fp16
5.9e-02
1.3e-03
0.01
0.5636 (+0.0000)
outside tolerance
Lite v0.5 warped
tensorrt int8
1.3e-02
2.1e-02
0.36
0.5658 (+0.0022)
outside tolerance
UFLD baseline
onnxruntime fp32
2.4e-06
3.0e-07
0.00
0.1732 (+0.0000)
within tolerance
UFLD baseline
onnxruntime int8
3.5e-01
3.6e-02
2.29
0.1776 (+0.0044)
outside tolerance
UFLD baseline
tensorrt fp32
4.7e-04
6.4e-05
0.00
0.1732 (+0.0000)
within tolerance
UFLD baseline
tensorrt fp16
8.7e-03
6.2e-04
0.02
0.1732 (+0.0000)
within tolerance
UFLD baseline
tensorrt int8
1.7e-01
1.7e-02
0.54
0.1732 (+0.0000)
outside tolerance
UFLD v0.2 weighted
onnxruntime fp32
1.4e-06
1.8e-07
0.00
0.0789 (+0.0000)
within tolerance
UFLD v0.2 weighted
onnxruntime int8
1.4e-06
5.1e-03
0.72
0.0636 (-0.0154)
outside tolerance
UFLD v0.2 weighted
tensorrt fp32
5.5e-04
2.5e-05
0.00
0.0789 (+0.0000)
within tolerance
UFLD v0.2 weighted
tensorrt fp16
5.7e-03
3.7e-04
0.01
0.0789 (+0.0000)
within tolerance
UFLD v0.2 weighted
tensorrt int8
6.9e-04
1.0e-02
0.41
0.0768 (-0.0022)
outside tolerance
Examples from the held-out scenes
UFLD baseline, held-out test, a typical success: original | ground truth (green) | prediction (red) - VV_S03_786UFLD baseline, held-out test, a typical error: original | ground truth (green) | prediction (red) - VIX_S05_52UFLD v0.7 ConvGRU, held-out test, a typical success: original | ground truth (green) | prediction (red) - VV_S03_775UFLD v0.7 ConvGRU, held-out test, a typical error: original | ground truth (green) | prediction (red) - VIX_S05_51
Roadmap
First make a lightweight temporal model beat the UFLD baseline under the full protocol; only then optimise for an in-car embedded system.
Phase 1: protocol and first pilot (v0.3)
doneLeakage-safe ELAS protocol: three held-out road scenes, 60-frame blocks with a purge gap, six seeds, HPO with the same budget for every model, validation-only selection.
doneGPU pilot (2 seeds, up to 4 epochs): lite v0.5 is best on unseen roads; every UFLD run peaks at epoch 1 and then overfits; history adds little to any model.
doneDeployment tooling: ONNX, TensorRT FP32 / FP16 / INT8 on the desktop GPU, streaming with cached history features.
Phase 2: make a lightweight temporal model beat the UFLD baseline (v0.4)
doneFix the UFLD overfitting: geometric augmentation lifts the UFLD baseline from 0.07 to 0.86 held-out lane F1 (configs/ablations/augmentation.yaml, chosen on validation); extended photometric augmentation, flips and dropout + label smoothing do not help.
doneThe other overfitting levers as single changes to the original recipe: a 10x lower backbone learning rate, frozen early layers, a smaller UFLD head, current-frame degradation and 8 more ELAS training scenes. None fixes the overfitting on its own (held-out lane F1 0.05 to 0.21, best epoch still 1 or 2); geometric augmentation stays the recipe. Cross-dataset initialisation (model.init_checkpoint) is implemented for when CULane / TuSimple weights are available.
doneLonger history (configs/ablations/history.yaml, UFLD v0.4, 4 arms): steps of 2, 5 and 10 frames give the same gain over the baseline (about +0.03); 5 frames instead of 3 remove it. More arms of the grid (steps 1 to 15, 2 to 5 frames) can be added with ablate --only.
doneCurrent-frame degradation: only the current frame is occluded, blurred, darkened or made noisy while the history stays clean, so a temporal model must use its history (data.augmentation.current_frame_prob).
doneAlign before fusing: UFLD v0.6 (lite v0.5's learned warp and gate on UFLD features).
doneRecurrent state: UFLD v0.7 and lite v0.6 (ConvGRU); streaming can carry one state per camera (--mode carry).
doneEqual-training controls: the +CT baselines get the same extra training as the temporal models.
doneEvaluate where time should help: lane F1 per scene condition, jitter, and a validation-tuned Kalman tracker as the cheap temporal reference.
doneSecond pilot with the v0.4 models and recipe (configs/elas_pilot_v2.yaml): UFLD baseline 0.881, best temporal UFLD (v0.7) 0.895; the lite models need more epochs.
doneCapacity control lite_v05_static: lite v0.5 fed the current frame in every position, to separate a temporal gain from the extra fusion layers.
doneLite family with a longer budget (up to 16 epochs) and the capacity control (configs/elas_lite_long.yaml): lite v0.5 reaches 0.850 with 8x fewer parameters than UFLD; under an occluded current frame its history is worth +0.17 and +0.38 over the no-history control.
doneRobustness evaluation: current frame occluded, blurred, darkened or noisy with a clean history. Under occlusion UFLD v0.3 keeps 0.875 lane F1 against 0.761 for the baseline, in both seeds.
nextTrain the recurrent models on long sequences so that one carried state per camera works (carry mode drifts after 3-frame training).
Phase 3: full protocol (supervisor)
nextFull run: 6 seeds, HPO, up to 50 epochs, with the recipe and models chosen from the pilots (configs/elas.yaml).
laterPretraining on CULane / TuSimple, then fine-tuning on ELAS (needs the datasets or the official UFLD weights).
laterMeasure on the real device: build the TensorRT engines on a Jetson and run the measurement script; today's Jetson figures are simulations with assumed slowdowns.
laterTake preprocessing off the CPU: the CPU resize is about 9 ms of a 14 to 19 ms frame. Resize on the camera's image processor or the GPU with zero-copy buffers, and retrain with that same resize.
laterCrop to the road: the row anchors cover only the lower 41 % of the image, so cropping cuts compute by about 2.4 times; a lower input resolution is the second option. Both need retraining.
laterTensorRT FP16 by default (lossless in the pilot); INT8 only if FP16 is too slow, calibrated on the car's camera or with quantization-aware training for the lite models.
laterPrefer the lite family: about 8 times fewer parameters and 3.6 times less compute than UFLD.
laterRobustness to dropped frames: train with frame drops and space the history by timestamps, since the car's camera may run at a different frame rate than ELAS.
laterProduction runtime: C++ TensorRT or DeepStream instead of Python, capture / inference / post-processing in parallel, always process the latest frame, locked clocks, thermal tests inside the enclosure.
Reproduce and continue
python -m tac_ufld doctor
python -m pytest -q
python -m tac_ufld ablate --spec configs/ablations/augmentation.yaml --confirm # overfitting remedies
python -m tac_ufld run --config configs/elas_pilot_v2.yaml # pilot with the v0.4 models
python -m tac_ufld ablate --spec configs/ablations/history.yaml --confirm # history length
python -m tac_ufld run --config configs/elas.yaml --confirm # full protocol
python -m tac_ufld ui # inspect checkpoints
python -m tac_ufld site --runs results/elas_pilot_v2 # rebuild this page
The full experiment differs from the pilots in seeds (6 instead of 2), epochs (up to 50 with patience 10) and hyper-parameter search (the same budget for every model). Only the full run can support claims.
