[bug] Hallucination out of the GAN when there shouldn't be...

frank commented

2026-10-02 14:30:47 +00:00

Owner

Attached is the sound sample I'd initially ran with for smoke testing with fargan_smoketest (which is a blindingly braindead simple app that reads in a .f32 type floats file plays the dump from there at the desired rate per msec to the FARGAN engine and dumps out a 16-bit PCM output file to stdout.

Converted to .f32 from the 16 kHz input .s16 file, it should (and does...) playback correctly with identical quality. It should be nearly flawless in reproduction as there's no data dropouts, etc. It is not. The sample inputs was a Harvard sentences selection, no background noise to confuse the synth. Several seconds in, abut midpoint of the second sentence uttered, the GAN hallucinates the same way on both Opus' tool and the test-wedge I wrote for the library. It slurs the output and begins to talk a bit like the town drunk. Intelligible, but definitely not the near perfection that both tools started with. It's not a failing in my test tool as the Opus tooling does the exact same thing.

Suspect a mix of inadequate training (It probably was optimized to just under a second of performance and starts wandering off into the rough once you're past that window and don't do some sort of recock of the NN for pauses, etc. Likely needs a longer loop window in training and needs pause/energy detection to reset the synth so it doesn't do that. Might be analysis, though that's going to be a tougher nut to crack and we need the first two anyhow, even if it doesn't fix.

Attached is the sound sample I'd initially ran with for smoke testing with fargan_smoketest (which is a blindingly braindead simple app that reads in a .f32 type floats file plays the dump from there at the desired rate per msec to the FARGAN engine and dumps out a 16-bit PCM output file to stdout. Converted to .f32 from the 16 kHz input .s16 file, it should (and does...) playback correctly with identical quality. It should be nearly flawless in reproduction as there's no data dropouts, etc. It is not. The sample inputs was a Harvard sentences selection, no background noise to confuse the synth. Several seconds in, abut midpoint of the second sentence uttered, the GAN hallucinates the same way on both Opus' tool and the test-wedge I wrote for the library. It slurs the output and begins to talk a bit like the town drunk. Intelligible, but definitely not the near perfection that both tools started with. It's not a failing in my test tool as the Opus tooling does the exact same thing. Suspect a mix of inadequate training (It probably was optimized to just under a second of performance and starts wandering off into the rough once you're past that window and don't do some sort of recock of the NN for pauses, etc. Likely needs a longer loop window in training and needs pause/energy detection to reset the synth so it doesn't do that. Might be analysis, though that's going to be a tougher nut to crack and we need the first two anyhow, even if it doesn't fix.

frank self-assigned this

2026-10-02 14:52:01 +00:00

frank changed title from ~~Hallucination out of the GAN when there shouldn't be...~~ to [bug] Hallucination out of the GAN when there shouldn't be...

2026-10-02 14:56:31 +00:00

Rows
Columns

[bug] Hallucination out of the GAN when there shouldn't be... #1