what the title says. this is a pretty big paper in the deep learning anomaly detection space, accepted at the International Conference on Knowledge Discovery & Data Mining (KDD 2023) with hundreds of references. [arxiv]
When i cloned and run their source code [repo], the model seems to be working fine, that is until you look closer at the code and results.
The original source code does not log the loss progress which i found questionable so i started logging it myself, and the results where quite interesting.
==================== Train ===================
speed: 0.6970s/iter; left time: 3098.7351s
speed: 0.6970s/iter; left time: 3098.7351s
series loss: 17.188232421875, prior loss: 17.188232421875, loss: 0.0
speed: 0.3407s/iter; left time: 1480.7512s
speed: 0.3407s/iter; left time: 1480.7512s
series loss: 19.67855453491211, prior loss: 19.67855453491211, loss: 0.0
speed: 0.3405s/iter; left time: 1445.5816s
speed: 0.3405s/iter; left time: 1445.5816s
series loss: 20.960140228271484, prior loss: 20.960140228271484, loss: 0.0
speed: 0.3406s/iter; left time: 1412.3307s
speed: 0.3406s/iter; left time: 1412.3307s
series loss: 21.283531188964844, prior loss: 21.283531188964844, loss: 0.0
speed: 0.3404s/iter; left time: 1377.3572s
speed: 0.3404s/iter; left time: 1377.3572s
series loss: 21.437604904174805, prior loss: 21.437604904174805, loss: 0.0
speed: 0.3405s/iter; left time: 1343.4382s
speed: 0.3405s/iter; left time: 1343.4382s
series loss: 21.535520553588867, prior loss: 21.535520553588867, loss: 0.0
speed: 0.3406s/iter; left time: 1309.8116s
speed: 0.3406s/iter; left time: 1309.8116s
series loss: 21.596588134765625, prior loss: 21.596588134765625, loss: 0.0
speed: 0.3405s/iter; left time: 1275.4794s
speed: 0.3405s/iter; left time: 1275.4794s
series loss: 21.639469146728516, prior loss: 21.639469146728516, loss: 0.0
speed: 0.3414s/iter; left time: 1244.8077s
speed: 0.3414s/iter; left time: 1244.8077s
series loss: 21.66444969177246, prior loss: 21.66444969177246, loss: 0.0
Epoch: 1, Cost time: 520.156s
vali 1 loss: 0.0, vali 2 loss: nan
speed: 2.4402s/iter; left time: 8630.9654s
speed: 2.4402s/iter; left time: 8630.9654s
series loss: 21.682228088378906, prior loss: 21.682228088378906, loss: 0.0
speed: 0.3417s/iter; left time: 1174.2999s
speed: 0.3417s/iter; left time: 1174.2999s
series loss: 21.693164825439453, prior loss: 21.693164825439453, loss: 0.0
speed: 0.3431s/iter; left time: 1144.8169s
speed: 0.3431s/iter; left time: 1144.8169s
series loss: 21.697399139404297, prior loss: 21.697399139404297, loss: 0.0
speed: 0.3453s/iter; left time: 1117.8676s
speed: 0.3453s/iter; left time: 1117.8676s
series loss: 21.699926376342773, prior loss: 21.699926376342773, loss: 0.0
speed: 0.3406s/iter; left time: 1068.4710s
speed: 0.3406s/iter; left time: 1068.4710s
series loss: 21.702926635742188, prior loss: 21.702926635742188, loss: 0.0
speed: 0.3222s/iter; left time: 978.5355s
speed: 0.3222s/iter; left time: 978.5355s
series loss: 21.698776245117188, prior loss: 21.698776245117188, loss: 0.0
speed: 0.3220s/iter; left time: 945.7742s
speed: 0.3220s/iter; left time: 945.7742s
series loss: 21.703542709350586, prior loss: 21.703542709350586, loss: 0.0
speed: 0.3222s/iter; left time: 913.9631s
speed: 0.3222s/iter; left time: 913.9631s
series loss: 21.700130462646484, prior loss: 21.700130462646484, loss: 0.0
speed: 0.3219s/iter; left time: 881.0818s
speed: 0.3219s/iter; left time: 881.0818s
series loss: 21.70258140563965, prior loss: 21.70258140563965, loss: 0.0
Epoch: 2, Cost time: 496.539s
vali 1 loss: 0.0, vali 2 loss: nan
speed: 2.2717s/iter; left time: 5970.0229s
speed: 2.2717s/iter; left time: 5970.0229s
series loss: 21.702590942382812, prior loss: 21.702590942382812, loss: 0.0
speed: 0.3215s/iter; left time: 812.6774s
speed: 0.3215s/iter; left time: 812.6774s
series loss: 21.70191764831543, prior loss: 21.70191764831543, loss: 0.0
speed: 0.3214s/iter; left time: 780.2721s
speed: 0.3214s/iter; left time: 780.2721s
series loss: 21.703283309936523, prior loss: 21.703283309936523, loss: 0.0
speed: 0.3227s/iter; left time: 751.1521s
speed: 0.3227s/iter; left time: 751.1521s
series loss: 21.701114654541016, prior loss: 21.701114654541016, loss: 0.0
speed: 0.3225s/iter; left time: 718.4966s
speed: 0.3225s/iter; left time: 718.4966s
series loss: 21.702499389648438, prior loss: 21.702499389648438, loss: 0.0
speed: 0.3226s/iter; left time: 686.5956s
speed: 0.3226s/iter; left time: 686.5956s
series loss: 21.704504013061523, prior loss: 21.704504013061523, loss: 0.0
speed: 0.3226s/iter; left time: 654.1541s
speed: 0.3226s/iter; left time: 654.1541s
series loss: 21.70374298095703, prior loss: 21.70374298095703, loss: 0.0
speed: 0.3224s/iter; left time: 621.5679s
speed: 0.3224s/iter; left time: 621.5679s
series loss: 21.70180892944336, prior loss: 21.70180892944336, loss: 0.0
speed: 0.3224s/iter; left time: 589.3803s
speed: 0.3224s/iter; left time: 589.3803s
series loss: 21.70216178894043, prior loss: 21.70216178894043, loss: 0.0
Epoch: 3, Cost time: 483.480s
vali 1 loss: 0.0, vali 2 loss: nan
speed: 2.2459s/iter; left time: 3860.6496s
speed: 2.2459s/iter; left time: 3860.6496s
series loss: 21.701725006103516, prior loss: 21.701725006103516, loss: 0.0
speed: 0.3214s/iter; left time: 520.3479s
speed: 0.3214s/iter; left time: 520.3479s
series loss: 21.700902938842773, prior loss: 21.700902938842773, loss: 0.0
speed: 0.3216s/iter; left time: 488.5637s
speed: 0.3216s/iter; left time: 488.5637s
series loss: 21.70222282409668, prior loss: 21.70222282409668, loss: 0.0
speed: 0.3225s/iter; left time: 457.6450s
speed: 0.3225s/iter; left time: 457.6450s
series loss: 21.701622009277344, prior loss: 21.701622009277344, loss: 0.0
speed: 0.3226s/iter; left time: 425.4485s
speed: 0.3226s/iter; left time: 425.4485s
series loss: 21.701330184936523, prior loss: 21.701330184936523, loss: 0.0
speed: 0.3224s/iter; left time: 393.0522s
speed: 0.3224s/iter; left time: 393.0522s
series loss: 21.703283309936523, prior loss: 21.703283309936523, loss: 0.0
speed: 0.3224s/iter; left time: 360.7532s
speed: 0.3224s/iter; left time: 360.7532s
series loss: 21.7034912109375, prior loss: 21.7034912109375, loss: 0.0
speed: 0.3225s/iter; left time: 328.6039s
speed: 0.3225s/iter; left time: 328.6039s
series loss: 21.702743530273438, prior loss: 21.702743530273438, loss: 0.0
speed: 0.3223s/iter; left time: 296.1858s
speed: 0.3223s/iter; left time: 296.1858s
series loss: 21.703426361083984, prior loss: 21.703426361083984, loss: 0.0
Epoch: 4, Cost time: 482.616s
vali 1 loss: 0.0, vali 2 loss: nan
speed: 2.2444s/iter; left time: 1817.9376s
speed: 2.2444s/iter; left time: 1817.9376s
series loss: 21.703350067138672, prior loss: 21.703350067138672, loss: 0.0
speed: 0.3215s/iter; left time: 228.2310s
speed: 0.3215s/iter; left time: 228.2310s
series loss: 21.70497703552246, prior loss: 21.70497703552246, loss: 0.0
speed: 0.3214s/iter; left time: 196.0459s
speed: 0.3214s/iter; left time: 196.0459s
series loss: 21.704113006591797, prior loss: 21.704113006591797, loss: 0.0
speed: 0.3215s/iter; left time: 163.9427s
speed: 0.3215s/iter; left time: 163.9427s
series loss: 21.70448112487793, prior loss: 21.70448112487793, loss: 0.0
speed: 0.3213s/iter; left time: 131.7412s
speed: 0.3213s/iter; left time: 131.7412s
series loss: 21.701704025268555, prior loss: 21.701704025268555, loss: 0.0
speed: 0.3214s/iter; left time: 99.6477s
speed: 0.3214s/iter; left time: 99.6477s
series loss: 21.704320907592773, prior loss: 21.704320907592773, loss: 0.0
speed: 0.3220s/iter; left time: 67.6261s
speed: 0.3220s/iter; left time: 67.6261s
series loss: 21.703289031982422, prior loss: 21.703289031982422, loss: 0.0
speed: 0.3226s/iter; left time: 35.4901s
speed: 0.3226s/iter; left time: 35.4901s
series loss: 21.703575134277344, prior loss: 21.703575134277344, loss: 0.0
speed: 0.3227s/iter; left time: 3.2270s
speed: 0.3227s/iter; left time: 3.2270s
series loss: 21.700424194335938, prior loss: 21.700424194335938, loss: 0.0
Epoch: 5, Cost time: 482.731s
vali 1 loss: 0.0, vali 2 loss: nan
The model is learning nothing much at all.
this made me look deeper into the source code and here are other things i found :
Early stoping and validation is useless since validation loss 2 is not even calculated.
The validation is also using the test dataset, not that matters since the modal is not really learning.
the Revin normalisation is not really getting applied correctly, since its module is getting reinitialised every forward.
So this paper just uses the generous point adjust metric to get high results? while the model itself is not really doing anything? im i reading this wrong?.
[link] [comments]
