Try modulating audio stream for video call

[sansyrox]

Is your feature request related to a problem? Please describe.
Currently, there is no feature to modulate the audio stream of video calls

Describe the solution you'd like
Ability to change the audio voice

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Approach to be followed (optional)
A clear and concise description of approach to be followed.

Additional context
Add any other context or screenshots about the feature request here.

[ZohebMOPO]

@sansyrox I am kinda busy this weekend because schools opened. So I can't guarantee when the feature will be done.

[sansyrox]

That's alright, @ZohebMOPO . You can take your time 😄

[ZohebMOPO]

Tysm!

[ZohebMOPO]

@sansyrox I was going through some articles about this feature. I am not sure(underconfident) if I could add this feature.Sorry for that. Btw, this was the article that I was reading.

[ZohebMOPO]

There's a npm package that does this thing but it is binded with redux :(

[sansyrox]

@ZohebMOPO , a redux binding is not required and the module will be only helpful for the dispatching of events. Making it very slow.

The important part is an event like this. We shouldn't use a state library or rather any state module for this in real time audio.

let audioEvent: AudioEvent = (audioCtx, getCurrentTime) => {
  let oscillator: OscillatorNode = audioCtx.createOscillator();
  let gainNode: GainNode = audioCtx.createGain();

  oscillator.connect(gainNode);
  gainNode.connect(audioCtx.destination);

  oscillator.type = 'square';
  oscillator.frequency.value = 100;
  oscillator.start(getCurrentTime() + 500); // wait half a second, then make sound.

  gainNode.gain.value = 0.1;
};

Even with the above module we would have to implement the events ourself, which are the real difficult parts.

@sansyrox I was going through some articles about this feature. I am not sure(underconfident) if I could add this feature.Sorry for that. Btw, this was the article that I was reading.

@raghavdhingra and I will help you for sure with your PR.

[sansyrox]

Also, try implementing any kind of real time audio modulation even in a sandbox (using an earlier video as well that you shared with us) . We will help you out where you get stuck.

We can brainstorm a different strategy next week otherwise.

[ZohebMOPO]

Hmm sure. Tysm!

[raghavdhingra]

@raghavdhingra and I will help you for sure with your PR.
Definitely

[ZohebMOPO]

@raghavdhingra

A bug is coming up and I can't find a way to fix that up.

class Controller extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      audio: null,
    };
  }
  async componentDidMount() {
    this.audio = new window.AudioContext();
    this.source = await this.audio.createMediaStreamSource(
      this.getMicrophone()
    );
    this.node = this.audio.createGain();
  }

  async getMicrophone() {
    const audio = await navigator.mediaDevices.getUserMedia({
      audio: true,
      video: false,
    });
    return audio;
  }

  Volume(e) {
    const value = parseInt(e.target.value);

    this.node.gain.value = value;
  }

  render() {
    return (
      <div>
        <button data-playing="false" role="switch" aria-checked="false">
          <input
            type="range"
            id="volume"
            min="0"
            max="2"
            value="1"
            step="0.01"
          />
          <span>Volume</span>
        </button>
      </div>
    );
  }
}

export default Controller;

This is the code for volume

[sansyrox]

This is because you don’t have an onchange field on input.

…

On Thu, 8 Jul 2021 at 6:51 PM, Zoheb Alli Khan ***@***.***> wrote: @raghavdhingra <https://github.com/raghavdhingra> [image: ezgif com-gif-maker] <https://user-images.githubusercontent.com/72716462/124928450-39016c00-e01d-11eb-922e-dc2878bbb731.gif> A bug is coming up and I can't find a way to fix that up. class Controller extends React.Component { constructor(props) { super(props); this.state = { audio: null, }; } async componentDidMount() { this.audio = new window.AudioContext(); this.source = await this.audio.createMediaStreamSource( this.getMicrophone() ); this.node = this.audio.createGain(); } async getMicrophone() { const audio = await navigator.mediaDevices.getUserMedia({ audio: true, video: false, }); return audio; } Volume(e) { const value = parseInt(e.target.value); this.node.gain.value = value; } render() { return ( <div> <button data-playing="false" role="switch" aria-checked="false"> <input type="range" id="volume" min="0" max="2" value="1" step="0.01" /> <span>Volume</span> </button> </div> ); } } export default Controller; This is the code for volume — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHEOIBSQ64FMKC6LQY2TFK3TWWQ4LANCNFSM47ABOIXA> .

[sansyrox]

const audio = await this. getMicrophone()
this.source = await this.audio.createMeid..(audio)

[ZohebMOPO]

class Audio extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      audio: null,
    };

    this.toggleMicrophone = this.toggleMicrophone.bind(this);
    this.canvas = React.createRef();
    this.vol = React.createRef();
    this.bass = React.createRef();
    this.mid = React.createRef();
    this.treble = React.createRef();
  }

  async componentDidMount() {
    this.audioContext = new window.AudioContext();
    this.analyserNode = new window.AnalyserNode(this.audioContext, {
      fftSize: 256,
    });
    if (this.audioContext.state === "suspended") {
      await this.audioContext.resume();
    }
    this.mic = await this.getMicrophone();
    this.source = this.audioContext.createMediaStreamSource(this.mic);
    this.source.connect(this.audioContext.destination);
    this.source.connect(this.analyserNode);
    this.gainNode = new window.GainNode(this.audioContext, {
      gain: this.vol.current,
    });
    this.bassEQ = new window.BiquadFilterNode(this.audioContext, {
      type: "lowshelf",
      frequency: 500,
      gain: this.bass.current,
    });
    this.midEQ = new window.BiquadFilterNode(this.audioContext, {
      type: "peaking",
      Q: Math.SQRT1_2,
      frequency: 1500,
      gain: this.mid,
    });
    this.trebleEQ = new window.BiquadFilterNode(this.audioContext, {
      type: "highshelf",
      frequency: 3000,
      gain: this.treble.current,
    });
  }

  voll(e) {
    const val = parseFloat(e.target.value);
    this.gainNode.gain.setTargetAtTime(
      val,
      this.audioContext.currentTime,
      0.01
    );
  }

  basss(e) {
    const bass = parseInt(e.target.value);
    this.midEQ.gain.setTargetAtTime(bass, this.audioContext.currentTime, 0.01);
  }

  midd(e) {
    const mid = parseInt(e.target.value);
    this.midEQ.gain.setTargetAtTime(mid, this.audioContext.currentTime, 0.01);
  }

  treblee(e) {
    const treble = parseInt(e.target.value);
    this.trebleEQ.gain.setTargetAtTime(
      treble,
      this.audioContext.currentTime,
      0.01
    );
  }

  async getMicrophone() {
    const audio = await navigator.mediaDevices.getUserMedia({
      audio: {
        echoCancellation: false,
        latency: 0,
      },
      video: false,
    });

    this.setState({ audio });
    return audio;
  }

  stopMicrophone() {
    this.state.audio.getTracks().forEach((track) => track.stop());
    this.setState({ audio: null });
  }

  toggleMicrophone() {
    if (this.state.audio) {
      this.stopMicrophone();
    } else {
      this.getMicrophone();
    }
  }

  drawVisualizer() {
    requestAnimationFrame(this.drawVisualizer);

    const bufferLength = this.analyserNode.frequencyBinCount;
    const dataArray = new Uint8Array(bufferLength);
    this.analyserNode.getByteFrequencyData(dataArray);
    const canvas = this.canvas.current;
    const height = canvas.height;
    const width = canvas.width;
    const barWidth = width / bufferLength;

    const canvasContext = canvas.getContext("2d");
    canvasContext.clearReact(0, 0, width, height);
    dataArray.forEach((item, index) => {
      const y = ((item / 255) * height) / 2;
      const x = barWidth * index;

      canvasContext.fillStyle = "rgb(0, 0, 0)";
      canvasContext.filRect(x, height - y, barWidth, y);
    });
  }

  componentDidUpdate() {
    this.drawVisualizer();
  }
  render() {
    return (
      <div>
        <button onClick={this.toggleMicrophone}>
          {this.state.audio ? "Stop microphone" : "Get microphone input"}
        </button>
        <canvas width="300" height="300" ref={this.canvas}></canvas>
        <label for="volume">Volume</label>
        <input
          type="range"
          id="volume"
          min="0"
          max="1"
          step="0.1"
          ref={this.vol}
          onChange={this.voll}
        />
        <label for="bass">Bass</label>
        <input
          type="range"
          id="bass"
          min="0"
          max="1"
          step="0.1"
          ref={this.bass}
          onChange={this.basss}
        />
        <label for="mid">Mid</label>
        <input
          type="range"
          id="mid"
          min="0"
          max="1"
          step="0.1"
          ref={this.mid}
          onChange={this.midd}
        />
        <label for="treble">Treble</label>
        <input
          type="range"
          id="treble"
          min="0"
          max="1"
          step="0.1"
          ref={this.treble}
          onScroll={this.treblee}
        />
      </div>
    );
  }
}

export default Audio; ```



Is there anything I am doing wrong? @raghavdhingra 

[sansyrox]

@raghavdhingra ^^

[raghavdhingra]

@ZohebMOPO
this.setState({ audio });
We can not store audio within the state, hooks required time to update, we can not put a stream within the state.

[raghavdhingra]

There are 3 ways to store a stream,

1. Use useRef for storing it
2. You can use the global window variable, window.audioStream = audio
3. Make a local variable for storing the value

[ZohebMOPO]

First of all, I guess it's a class component, so useRef will not work. and for the window variable, there's no audioStream method. I guess I am dumb, Idk but I can't figure this one out :(((