3D Sound System

[paulscode]

It wasn't working on my computer for some reason, it just kept scrolling.

Are you talking about the graphics or the sound (what is scrolling?)
If the problem is with the sound, is it just the MIDI, or are the other sound effects not working as well?  Are there any error messages or exceptions printing out to the Java console?
If you are referring to the graphics flicker, that is because they are not buffered.  I really should change that.  I originally just threw this applet together to replicate a bug that Egon was having with the Robombs sound effects a while back.

What about streaming as opposed to keeping it in memory?

MIDI can't be streamed.  Well, depending on your definition of streaming, that is (you could say that MIDI is always streamed).  Here's how it works.  A MIDI file does not contain any audio data at all.  Instead, it is full of commands called MIDI events.  These say higher-level things like "Play a high C on a guitar for one second".  These commands are loaded from the file into something called a "sequence".  Java sound has a "sequencer", which sends the MIDI events to a Receiver.  The receiver takes the events and tells the synthesizer what to do.  The synthesizer generates the actual audio.

So it appears to me that the problem is coming either from the sequencer taking too long to generate the MIDI events or the synthesizer taking too long to generate the audio.  More specifically, while JavaSound is somehow busy with a MIDI event, it doesn't get around to starting playback on the clips.  Running the entire process on a seperate thread does not seem to help, so something internal to JavaSound between MIDI and clip processing appears to be running on the same thread.  I do not seem to have any control over that, although I am looking into this some more.  If I can find some source code for playing clips and MIDI at the same time, it might help.

I'm pretty sure I've played java games that had music and sound effects.

Yes, of course, which is why this problem is so perplexing.  I know it is possible, I just can't figure out how.

[paulscode]

I thought of some things to test which should help me narrow down where the MIDI/Clip hang-up problem is occurring:

1) Check if the command thread is actually running the Clip "play" commands at the proper time, and if so, is it taking a long time to process those commands.

2) Check if the hang-up problem only occurs for Clips, or if it also affects SourceDataLines as well.

3) Determine if there is an actual line of code that is taking a long time to run, or if the hang-up is completely internal.

4) Determine if the problem still occurs when using a Microsoft Windows-specific synthesizer instead of the Java synthesizer.

5) Rule out conflicts with SoundSystem code, such as the command thread.  To do this, I will create a bare-bones test case from scratch which does not use the SoundSystem library.

6) Rule out conflicts with non-sound related stuff, such as KeyEvents.

7)  Determine if the problem is with the sequencer or the synthesizer.  To do this, I will write a test program using the software synthesizer I talked about instead of the internal JavaSound synthesizer.  This would give me the ability to control what thread synthesis is happening on.  Obviously I won't be able to use this in the SoundSystem library, but it should help me verify if the synthesizer is the problem.

[fireside]

Are you talking about the graphics or the sound (what is scrolling?)

It's the graphics that appear to be scrolling.  I'll try it again.  I could just see the text choices going by really fast, like tv's in the old days when the vertical hold was off.

[paulscode]

Oh, ok, I have the graphics buffered now so you can read the commands 

[fireside]

Yes, there does seem to be a slight delay when using java sound.

[paulscode]

I'm going to look into an applet-specific option for MIDI.  Instead of using JavaSound to play MIDI, I will have the applet communicate back with the browser, and let the browser handle playing the MIDI file.  I've been playing around with applet / javascript interaction and it seems like this should be pretty simple (basically you'd interact with hidden <embed> tags - one for each MIDI file you want the applet to be able to play).  This option would of course require you to place the javascript code into the html page where the applet resides.  I will also need to make sure it is compatible with all the major browsers.  I am not actually going to include this in the SoundSystem library, but I'll release it as a seperate JAR that should go nicely with the small version of SoundSystem for use in applets (assuming of course that I can get it to work and it doesn't have any compatibility or delay problems).

[fireside]

Sounds good.  Hope it works out.

[paulscode]

My initial attempts with the javascript MIDI stuff have been somewhat problematic.  It took me a while to find a method for controling MIDI via javascript which works in all the major browsers.  I eventually did find one.  Problem is it adds a new hiden element to the page each time play() is called, which causes the applet to loose focus, thus requiring you to either request focus again or click on the applet's frame.

One good thing I have discovered though is that the sound effects delay problem is gone when using the browser to play MIDI, so I am going to tinker around with this thing for a while to see if I can get it to behave properly.

[fireside]

That is a problem with Java.  I found a getfocus or something like that, but when I tried it, it didn't work.  The only way to really get the focus in an applet is to click inside it that I've found. 

[EgonOlsen]

If your applet is signed anyway, you could use java.awt.Robot to let the applet click itself. Quite hacky, but possible...

[paulscode]

Ok, I think I have the Javascript MIDI library behaving in an acceptable manner finally.  The losing-focus problem has been solved.  In case you were wondering why it took me so long, it is extremely difficult to get something that behaves the same way in all browsers.  Oh, and I really hate Microsoft, btw.   

Anyway...  Here it is:
Javascript and required files

Demo Applet demonstrating use of the javascript alongside the small version of Sound System
(Java Source, HTML Source)

To use this javascript, make sure the files playmidi.js and silence.mid are in the same directory, and place the following line of code somewhere on your HTML page:

Code Select Expand

<script type='text/javascript' src='playmidi.js'> </script>

Include the following methods in your extended JApplet class.  These are just simple interfaces to the Javascript methods (the method names are pretty self-explanatory):

Code Select Expand

    public void loadMIDI( String fileUrl, float gain, boolean loop )
    {
        if( fileUrl == null )
            return;
       
        int volume = (int) (gain * 100.0f);
       
        if( volume > 100 )
            volume = 100;
        if( volume < 0 )
            volume = 0;
       
        String loopString;

        if( loop )
            loopString = "true";
        else
            loopString = "false";
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[3];
        args[0] = fileUrl;
        args[1] = String.valueOf( volume );
        args[2] = loopString;
       
        mainWindow.call( "loadMIDI", args );
    }
   
    public void playMIDI( String fileUrl )
    {
        if( fileUrl == null )
            return;
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[1];
        args[0] = fileUrl;
       
        mainWindow.call( "playMIDI", args );
       
        this.requestFocus();
    }
   
    public void pauseMIDI( String fileUrl )
    {
        if( fileUrl == null )
            return;
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[1];
        args[0] = fileUrl;

        mainWindow.call( "pauseMIDI", args );
       
        this.requestFocus();
    }
   
    public void stopMIDI( String fileUrl )
    {
        if( fileUrl == null )
            return;
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[1];
        args[0] = fileUrl;

        mainWindow.call( "stopMIDI", args );
       
        this.requestFocus();
    }
   
    public void rewindMIDI( String fileUrl )
    {
        if( fileUrl == null )
            return;
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[1];
        args[0] = fileUrl;

        mainWindow.call( "rewindMIDI", args );
       
        this.requestFocus();
    }
   
    public void setMIDIAttribute( String fileUrl, String attribute, String value )
    {
        if( fileUrl == null )
            return;
       
        JSObject mainWindow = JSObject.getWindow( this );
        String[] args = new String[3];
        args[0] = fileUrl;
        args[1] = attribute;
        args[2] = value;
       
        mainWindow.call( "setMIDIAttribute", args );
    }

To be able to compile your applet, you will also need to include the file 'plugin.jar', which can be found in your jre directory under the 'lib' subdirectory.  Also, import netscape.javascript.JSObject at the top of your code.  You do NOT need to include 'plugin.jar' in your applet's archive list at runtime - it will already be loaded when running the applet from inside a browser

I have the javascript working for all the major browsers using the QuickTime audio pluggin for MIDI, which is the default for most browsers, so most people are likely to be using it.  The javascript may works with some other audio pluggins as well, but I haven't taken the time to test this yet.  Problem is each pluggin uses its own method names for the controls (play() Play() or doPlay(), for example) so I'd have to check what audio pluggin a user has and call the correct methods for it).  I may come back at some point to work on that, but for now it is probably going to work 90% of the time.

A couple of limitations / bugs with this javascript:
1) Apparently a "hidden" embed tag cannot be played in browsers based on Netscape technology (Firefox for example), so I had to make the player visible.  A width and height to zero works fine in Firefox, but it does not compute in Internet Explorer, so I was forced to go with a width and height of 1.  Therefore, the browser displays a single pixel for each embed tag wherever you place the javascript on your page.  For that reason, I recommend placing the script at the bottom of the page (right before closing the </body>)  Or if you have a better place on the page, just put it wherever.  I'm going to tinker around with this some more to see if I can figure a way to make the thing completely invisible (I might have to brush up on my HTML skills).
2) With the QuickTime pluggin (possibly other pluggins as well), you can't change the "volume" and "looping" attributes via javascript once your <embed> tag has been created.  For that reason, you must specify the volume and looping attributes of your MIDI file when you load it, and they can not be changed later.
3) The user has complete control over the MIDI playback, so if they have music or embed disabled in their browser, or their Synth volume control is muted, there's not much you can do about it.
4) I was having problems with loud scratching sounds in some browsers the first time a sound is played.  To compensate, I had to make the javascript automatically play a silent MIDI file when it starts up.  This is completely transparent, but you are required to place the file "silence.mid" wherever the javascript is located.  On one of my test machines, I noticed the scratchiness had popped back up when loading the sounds from inside an applet rather than from the HTML.  If this starts happening a lot, a workaround would be to pre-load the MIDI files directly from the HTML rather then from inside the applet:

Code Select Expand

<script type='text/javascript' src='playmidi.js'> </script>
<script type='text/javascript'>
<!--
loadMIDI( 'theme.mid', 100, true );
loadMIDI( 'Bach.mid', 40, false );
//-->
</script>

[fireside]

It says I need to install apple Quicktime plugin.  Would that be right? 

[fireside]

Apparently Firefox doesn't come with a midi player.  I realize you just did a ton of work on it, but I'm not sure I would want to ask people to download Quicktime if they have Firefox on their computer which is over 20 per cent right now.  I have to think about it a little bit.  I'm really sorry about this.  I had no idea it would be this complicated.  I should have just shut up.

[paulscode]

Yes, Quicktime or any other pluggin for playing MIDI is required for this script to work in any browser, not just Firefox.  No problem, I learned a lot making this script.  BTW, there is another way to do it using flash (would require a Flash pluggin to be installed, of course).  I didn't choose that route because I figured users are more likely to have Quicktime installed than Flash.  If you prefer, I can make a Flash version of the script.

[paulscode]

I had another thought.  I could use the <BGSOUND> tag, which only works in Internet Explorer.  As you say, that would exclude over 20% of users, but at least it would be silent and not ask them to install anything.

I could also make a hybrid, which would use <BGSOUND> if the user was running Internet Explorer, and <EMBED> (either QT or Flash) if the user was running a different browser like Firefox.  That would reduce the users requiring a pluggin installed from 100% down to 20%.