3D Sound System

[EgonOlsen]

I've added an explosion sound for the bombs as a starter. All worked fine so far! 

[EgonOlsen]

I've added multiple sounds for all kinds of game events: Your SoundSystem works great so far....

I've noticed one strange thing though and that is if you exit the application without calling cleanUp(), one of my machines (with onboard sound) hangs. It returns to the desktop, but you can't do anything on it. Not even the mouse is moving any longer. This is no problem as i can just call cleanUp() and all is fine, but i found it a bit strange anyway.

[paulscode]

I've added multiple sounds for all kinds of game events: Your SoundSystem works great so far....

I've noticed one strange thing though and that is if you exit the application without calling cleanUp(), one of my machines (with onboard sound) hangs. It returns to the desktop, but you can't do anything on it. Not even the mouse is moving any longer. This is no problem as i can just call cleanUp() and all is fine, but i found it a bit strange anyway.

I am not sure why that would happen.  I do know that if you are using the OpenAL part, the cleanUp() method calls AL10.alDeleteSources() to remove the channels, AL10.alDeleteBuffers() to remove the loaded sound buffers, and AL.destroy() to shut down OpenAL.  Perhaps since OpenAL makes use of a .dll, there may be artifacts from that in memory if OpenAL is not shut down properly?  I will definitely make a note in the documentation on this, though.  Thanks!

[paulscode]

Today I decided to take a break from the JavaDoc for a while and run some tests and fix bugs.

1) I solved the Javasound OGG sample-rate bug.  After tinkering around a bit, I found that for stereo OGGs, you have to multiply the sample rate by four, but for mono OGGs no change is required.  I am not sure why this is the case (I couldn't find any documentation about this issue online), so I really can't say with 100% certainty that the problem is gone for good, but I downloaded several stereo and mono oggs from a variety of places, and they are all playing at the correct speed in both OpenAL and JavaSound, streaming and non-streaming.

2) I added quickStream() methods, which take the same parameters as quickPlay() except they stream the source.

3) I created a class called SoundSystemLogger, which handles all output messages.  It has methods for handling 3 kinds of messages: Normal, Important, and Error.  I added a setLogger() method in SoundSystemConfig, which you have the option of calling before instantiating the SoundSystem.  This will allow the user to extend the SoundSystemLogger class and override its methods to handle output messages any way they like (for example error messages could be handed off to your own error logger which prints them in a dialog box instead of the Java Console).  If no SoundSystemLogger is set before creating the SoundSystem, then the basic SoundSystemLogger class is used as the default message logger for the SoundSystem.

4) I noticed a bug that occurs when switching between libraries.  Basically any source that is being streamed from a seperate thread (streaming sources in OpenAL, and both types of sources in JavaSound) will cause an exception to be thrown when the playing library's "cleanup()" method is called before initializing the new library.  This is because the source object may become null between when I check if the source is null and when I use the source (a thread synchronization problem).  I think I can solve this by something like Thread.sleep(20); after interrupting the StreamThread so it has time to shut down, before removing the source object.

I plan to release a new version of SoundSystem tomorrow hopefully after fixing the thread problem and looking into the JavaSound rolloff attenuation bug.  I'll take a look at the pause issue again, but if there isn't a quick fix, I'll wait until the following release to fix that problem.

[paulscode]

SoundSystem Beta Release

JAR:
http://www.paulscode.com/libs/SoundSystem/16AUG2008/SoundSystem.jar

Source Code:
http://www.paulscode.com/source/SoundSystem/16AUG2008/SoundSystemSource.zip

Let's see, there have been quite a few changes since the first release, so I'll try to list everything here.

1) Sample rate (playback speed) for stereo .oggs is now working for most .ogg files.  I have found one stereo .ogg file that still plays too slowly, and I have also found some monotone .oggs that do not play at all (most monotone oggs are working fine, though).  Strangely, one of the .ogg files that is not working is the "gunshot" sound effect I used in one of my previous demo applets.  Since the file used to load and play properly, this seems to indicate that I may have broken something in the .ogg loading code.  The good thing about that is I still have the source code for that demo applet.  I should be able to compare line-by-line with the SoundSystem source code to locate the problem.  This is an issue I will continue to work on, but for now it is acceptible.  95% of .ogg files play correctly for both OpenAL and Javasound (and I haven't found any sound files that "work for one but not the other").

2) There are new methods for creating fast sources: quickStream(), which I mentioned in my last post, and backgroundMusic().  backgroundMusic() creates a permanant, priority, streaming, looping, non-attenuating source and plays it.  You can specify a source name or not, and you can also specify looping or not.  The easiest way to use it is just backgroundMusic( "music.ogg" );.

3) I fixed a number of null pointer exceptions caused when shutting things down or switching between libraries.  I will continue to correct problems like these as they pop up, but for now I believe I got most of them.

4) There is a new SoundSystemLogger class for handling messages, as I mentioned in my last post.  There are still "println's" floating around that I am working on removing.  The next release will have the logger fully implemented in all classes.

5) I fixed a mirror-image bug with Javasound panning (sources to the right were playing in the left speaker and vice versa).  Works correctly now.

6) I did some tweaking on the Javasound Rolloff attenuation, and I got it really close to OpenAL's rolloff attenuation.  It's a far cry better than it was before.  In fact, all the 3D effects I am emulating in JavaSound are pretty much dead-on matches to the built-in OpenAL effects (listener orientation, 3D panning, attenuation).  On my PC I'd have trouble telling which library is active if I didn't write the code myself. In fact 3D panning is better in JavaSound, because it works for both stereo and monotone sounds ( ok, enough bragging )

7) I made a major change to the way all sound files are loaded.  Before, there used to be four slightly different ways of loading sound files due to the fact that there are two supported formats and two libraries.  Now there is a file loader class which provides generic methods for reading from sound files regardless of their format.  It handles the format-specific details in the background.  This one class has been fully implemented for all file loading: wav or ogg, streaming or nonstreaming, OpenAL or Javasound.  One good thing about this is now sounds will play at the same sample rate for both OpenAL and Javasound, so even if I don't ever fix the sample rate bug, a user could potentially just change the speed of his sound file to compensate for the bug.  The best thing about this new loader, though, is that it can stream both wavs and oggs (before you could only stream oggs).

I spent most of today trying to get pause to work.  I didn't think I was even going to get a release posted today because I had things torn apart.  I am still nowhere close to getting pause to work.  It is deceptively tricky.  At the moment I have sources so they pause, but when they start back up, some of the queued sound data is getting lost.  For non-looping sources it sounds like the sound gets cut off soon after resuming after a pause.  For looping sources, it sounds like they are getting rewinded (thats in Javasound, non-streaming sources.  I haven't got around to looking at the other situations just yet).  I will work on this some more, but I am kind of beginning to wonder if this function is all that vital.  Pause is good if you are making an audio player, but for video games, I am not sure where you would need to pause a sound (stop and play should be sufficient for most games).  Does anybody else have some feedback on this issue?  I just still haven't decided if a pause function is really worth all the effort.

--UPDATE--
I believe I figured out why I am having a problem with pausing.  I think it is related to the SourceDataLine.stop() method being called from the command thread while data is being fed into the line from the stream thread.  The stop() method ends both input and output, so maybe I just need to make sure input has completed before calling stop().  I could have the command thread set a boolean to pause the source, then the stream thread can actually call stop() when it is finished inputing bytes.  I'll try this out and let you know how it goes.

[paulscode]

I discovered a major bug today, which I assume is related to the new SoundFileLoader class I added.  Basically, if a streaming source plays through to the end, any future attempts to play that same source will result in no sound being played.  This is, of course, a must-fix bug.  I am attempting to track the problem down now, and I'll post an updated version of SoundSystem as soon as I get this fixed.

[paulscode]

I spent all day on the streaming source replay bug today.  Turns out it is actually two problems, one for Javasound and one for OpenAL.  The Javasound problem was that when you call SourceDataLine.flush() from one thread before it ends, then you write data on the line from a new thread, the line keeps on flushing so nothing ever gets played (strangely, this happens even if you wait a really long time before trying to write data to the line).  The solution is to only call SourceDataLine.stop() from the first thread.  Then from the second thread, call SourceDataLine.flush() and SourceDataLine.drain() (not sure why I have to drain, but it doesn't work unless I do).  Then you can SourceDaraLine.start() and begin writing the data.  Strange problem, and it was rather difficult to find.

Obviously, this is not the same problem happening for OpenAL, since it doesn't use SourceDataLines.  This second problem behaves exactly like the first problem, though.  I have not solved it yet, although I spent most of the day trying to.  I have tracked it down to queuing and processing stream buffers.  Basically, the first time the sound plays, streambuffers are processed and played no problem.  If the stream is looping, I can go back to the beginning of the stream and immediately replay (from the same thread), no problem.  However, if the stream ends and the thread stops, then when I create a new thread later to restream the sound, I can queue stream buffers without any errors, but they are processed and discarded super-fast and no sound is being produced.  Strangely, this looks a lot like what was happening in Javasound to me.  Except for OpenAL there doesn't seem to be anything equivalent to flush() and drain().  I would like to find what is causing this bug, but worst case scenario, I could simply redesign the stream thread so it sleeps rather than shutting down (would require some rearranging of thing, but not super difficult).  It would be interrupted when the source needed to be restreamed, and shut down when cleanUp() is called.  If I do decide to do it this way, I will probably also change it for Javasound as well just so it matches.  Come to think of it, that is probably a better way to handle threads for streaming sources, anyway.

My guess is this problem has most likely been around since I started doing "channel reserving".  I just hadn't noticed it until now, because I simply never ran a test where a streaming source was played all the way through, then replayed.  I definitely can not assume that just because something worked in SoundManager, that it should also work in SoundSystem.  They are really two completely different beasts.

One happy note, though:  I got the pause function to work.  It's a little unresponsive in Javasound.  This is because the SourceDataLine feeds data into the Mixer, so when you pause the SourceDataLine, the sound stops playing a second later (this is also true for stop).  It can be tweaked a little by messign with the streaming buffer size, but it there is no way to get completely around the problem since it is a limitation of the library.  It works well enough, and like I said before, I really can't think of why it would be needed in a game anyway.

[paulscode]

I would like to find what is causing this bug, but worst case scenario, I could simply redesign the stream thread so it sleeps rather than shutting down (would require some rearranging of thing, but not super difficult).  It would be interrupted when the source needed to be restreamed, and shut down when cleanUp() is called.

On second thought, that idea will not work either in the new "permanant channel" infrastructure.  Since "sources" hop around from channel to channel, there is likely be more than one source (and therefore more than one stream-thread) playing on a channel throughout its life, so the second time something tries to stream on a given channel, it will not produce any sound.

And to further complicate things, I came across the following post on the lwjgl forums:

AL_INVALID_OPERATION can occur for a few reasons:
...
Trying to alSourceQueue one or more Buffers on a Source that already has
a Buffer attached to it using alSourcei AL_BUFFER.    (If you are streaming
data to a Source you should always call 'alSourceQueue' to add Buffers to
the Source.)
...

Because the OpenAL sources (channels) are permanant, this means that after playing a non-streaming "source" on a given channel, attempting to play a streaming one on that same channel would result in an AL_INVALID_OPERATION error.  I haven't tested to see if this is true, but I am willing to bet that it is.  This, of course, means that I will have to stream all sources just as I am doing in Javasound now.  A "normal" source has the data pre-loaded in a buffer, while a "streaming" source reads in small chunks of data directly from the file as it plays.  Both will need to use alSourceQueue to queue the data like a streaming source, regardless of where that data is coming from.

So what this all means is that I am going to have to make another fundamental change to how the SoundSystem is designed.  My plan is to eliminate the entire "stream-thread" concept, and instead have "channel-threads".  Basically there would be 32 (or however many channels there are) threads, one attached to each audio channel.  They would use an interface much like the "Command Queue", where sources would tell whichever channel they are assigned to what they would like to do, and the "channel-thread" will process those commands.  Commands will be straightforward, like "stream filename", "play sound-buffer x", "stop", "pause", "rewind", "flush", and "shut down".  The thread will handle the library-specific details like how to feed data into the stream and the like.

This change is quite major, and it will most likely take some time to finish and debug.  I am going to spend an great deal of time making sure this new thread infrastructure is completely synchronized and stable.   Ultimately, the changes will all be in the background.  The user-interface for SoundSystem will not change, so feel free to use one of the two initial versions in the mean time just to get used to using the library.  No need to worry that everything will be broken when the next release comes out.  Hopefully this is the last major change I will be making to the SoundSystem.  I'd rather focus on fixing minor bugs (that's a lot more fun).

[EgonOlsen]

32 threads? How many do you spawn in the current implementation? 32 sounds like a bad idea to me...

[paulscode]

32 threads? How many do you spawn in the current implementation? 32 sounds like a bad idea to me...

Currently, as many threads spawn as there are sources actively streaming (up to 32 at any given time).  If you are using non-streaming sources in OpenAL, then no additional threads are spawning (except, of course for the main "Command Queue" thread).
--EDIT--
This is significantly fewer than were spawned by the SoundManager class.  For example, in my "flying boxes" demo applet, there were 500 threads running at once.
--EDIT #2--
Oh, but then most of those sources were being culled, so most of the threads were shut down.  So again 32 Threads were running at any given time.  Disregard my last statement

Is there a lot of overhead with using 32 threads or something, or are you referring to the difficulty with synchronizing that many? I wasn't aware that running 32 threads was extreme (I am rather new to the whole thread business, but I thought computers are used to handling hundreds of threads at a time).  If you think that 32 threads is too much overhead, I will come up with a new solution that somehow uses a single thread to handle all channels.  I think it could be done by iterating through a channel list, processing one queued command each or feeding a chunk of data into the stream, then looping back to the first channel.  It would need to be somewhat intelligent, though, to avoid gaps in the audio stream.  For example, it takes a noticable amount of time to load a sound file that has not yet been loaded, so the ChannelThread would need to be able to say, "Ok, I'll tell the CommandThread to load that sound.  In the mean time, let me move on to the next channel and take care of this one later when that sound buffer is ready."

I'll have to see if one thread is fast enough to avoid gaps in the audio stream.  I may have to do something like assigning a thread to like every 4 channels or something if one tread is not enough to handle the work load.

Thank for the feedback!

[paulscode]

While we are on the subject of threads, I decided to make a basic thread class which takes care of all the normal synchronization stuff, and extend it in any future threads I create.  I was wondering if some of you Java gurus out there could take a look at it and let me know if it looks sound to you, or if you see any problems.  I know this is more related to Java and not about the SoundSystem, but any and all comments or suggestions on the subject of threads would be greatly appreciated!

Code Select Expand


package paulscode.sound;


public class SimpleThread extends Thread
{
    // thread running or not:
    private boolean alive = true;
    // thread ending or not:
    private boolean kill = false;
   
    // clear any memory used by the thread.
    // MUST call super.cleanUp() at the bottom of Overriden cleanUp() method!!!
    public void cleanUp()
    {
        kill( true, true );  // tread needs to shut down
        alive( false, true );  // thread has ended
    }
   
    @Override
    public void run()
    {
        /*  How the run() method should be set up:  */
       
        // Do your stuff here.  Remember to check dying() often to know when
        // the user wants the thread to shut down.
       
        // MUST call cleanUp() at the bottom of Overridden run() method!!!!!
        cleanUp();  // clears memory and sets status to dead.
    }
   
    // calls the rerun() method on a seperate thread, which calls run() when
    // the previous thread finishes:
    public void restart()
    {
        new Thread()
        {
            @Override
            public void run()
            {
                rerun();
            }
        }.start();
    }

    // kills the previous thread, waits for it to end, and then calls run():
    private void rerun()
    {
        kill( true, true );
        while( alive( false, false ) )
        {
            snooze( 100 );
        }
        alive( true, true );
        kill( false, true );
        run();
    }
   
    // returns true if thread is alive, false if it is dead:
    public boolean alive()
    {
        return alive( false, false );
    }
   
    // sets dying() to true, letting the thread know it needs to shut down:
    public void kill()
    {
        kill( true, true );
    }
   
    // returns true if user requested the thread to be shut down:
    protected boolean dying()
    {
        return kill( false, false );
    }
   
    // sets or returns varriable 'alive'
    private synchronized boolean alive( boolean b, boolean toChange )
    {
        if( toChange )
            alive = b;
        return alive;
    }
    // sets or returns varriable 'kill'
    private synchronized boolean kill( boolean b, boolean toChange )
    {
        if( toChange )
            kill = b;
        return kill;
    }
   
    // Sleeps for the specified number of milliseconds:
    protected void snooze( long milliseconds )
    {
        try
        {
            Thread.sleep( milliseconds );
        }
        catch( InterruptedException e ){}
    }
}


[paulscode]

Ok, so here is Plan B.  Forget the whole command-queueing idea - its overly complicated.  The only thing that absolutely must be done from a single thread is feeding chunks of data into an audio stream.  Everything else can be done from the Command Queue thread that is already in place and working nicely.  What I will do is create a single Channel Thread which will maintain a list of actively streaming channels.  It will iterate through them feeding them chunks of data.  When the end of the stream for a particular channel is reached, it will be dropped from the list.  I like this much better than my first idea - simpler is always good.

[EgonOlsen]

Ok, so here is Plan B.  Forget the whole command-queueing idea - its overly complicated.  The only thing that absolutely must be done from a single thread is feeding chunks of data into an audio stream.  Everything else can be done from the Command Queue thread that is already in place and working nicely.  What I will do is create a single Channel Thread which will maintain a list of actively streaming channels.  It will iterate through them feeding them chunks of data.  When the end of the stream for a particular channel is reached, it will be dropped from the list.  I like this much better than my first idea - simpler is always good.

Sounds much better too me than spawing lots of thread. The problems with so many threads is, that you

a) are using more system resources for context switches and thread management
b) are getting a highly non-deterministic behaviour. Java doesn't guarantee that a particular thread gets called in a time frame. The more you use, the higher the chance that a thread is late...especially when run on single-core, low-spec cpus

I don't see anything wrong with spawning a thread for each streaming source as you usually don't have that much in a game IMHO. I do see something wrong with spawning 32 of them. It's always better to iterate through a pool of <whatever> than to start a single worker thread per <whatever>.

[paulscode]

highly non-deterministic behaviour. Java doesn't guarantee that a particular thread gets called in a time frame. The more you use, the higher the chance that a thread is late

That is good to know.  Thanks!

I actually prefer to use as few threads as possible, anyway.  Saves me the headache of trying to remember what all is running at any given time and could potentially have a conflict over a particular resource.  So thanks for the comment about 32 threads being a bad idea - made me stop to think of something better and a lot simpler.

So I started on implementing Plan B yesterday, and it looks like it is actually going to be much easier to do than I originally thought  There is really not all that much I'll have to add - I'm mostly cutting out old classes and moving sections of code from one place to another.  And the best part is that there will never be more than 3 threads running at any given time:  The user's main thread, the command thread, and the channel thread.  If none of the sound sources are moving, the command thread will be sleeping, and if there are no sources playing, the channel thread will be sleeping.  I think this is probably the most efficient way to do things.

[paulscode]

Initial tests of the new single-stream-thread infrastructure have not been promising.  Javasound has skipping and strange problems where the last part of a sound repeats itself at the end of the stream like an echo.  OpenAL is not playing at all.  For now, I can't say one way or the other if there is a problem with the stream thread not being able to keep up with 32 sources.  If that were the case, though, I would expect only to see gaps in the stream, not the echo thing or muted sound.  I will just have to dig into each problem further to see what I can do to fix things.

In the mean time, I am also keeping an eye out for any way possible to reduce the amount of time the stream thread spends on each channel as it loops through them.  The faster it can make it through every source, the less likely there will be problems with pauses, clicks, etc. during playback.  One of the things I thought of is use synchronized methods interfacing booleans that can be set by the command thread to tell the stream thread things like if the source is paused or stopped, so the stream thread does not have to spend any time calling the slower library-specific functions for checking the state of each source.  If the paused boolean is true, it can immediately move on to the next source.  If stopped boolean is true, it can drop that source from its list.  That way, it can focus all its energy into feeding data to each channel and moving on to the next one.