Merge branch 'develop'

.github/workflows/release.yml

@@ -31,7 +31,7 @@ jobs:
       - name: Build with PyInstaller
         run: |
           .\venv\Scripts\Activate
-          pyinstaller main.py --name WingmanAiCore --noconfirm --icon assets/wingman-ai.ico --paths venv/Lib/site-packages --add-data "venv/Lib/site-packages/azure/cognitiveservices/speech;azure/cognitiveservices/speech" --add-data "assets;assets" --add-data "services;services" --add-data "wingmen;wingmen" --add-data "templates;templates" --add-data "audio_samples;audio_samples" --add-data "LICENSE;."
+          pyinstaller main.py --name WingmanAiCore --noconfirm --icon assets/wingman-ai.ico --paths venv/Lib/site-packages --add-data "venv/Lib/site-packages/azure/cognitiveservices/speech;azure/cognitiveservices/speech" --add-data "assets;assets" --add-data "services;services" --add-data "wingmen;wingmen" --add-data "templates;templates" --add-data "audio_samples;audio_samples" --add-data "LICENSE;." --add-data "lib/python3.dll;." --hidden-import urllib --hidden-import urllib.robotparser
 
       - name: Upload Windows Exe Artifact
         uses: actions/upload-artifact@v4

.gitignore

@@ -1,6 +1,6 @@
 # Byte-compiled / optimized / DLL files
 __pycache__/
-*.py[cod]
+*.pyc
 *$py.class
 
 # C extensions
@@ -165,17 +165,7 @@ cython_debug/
 # Mac stuff
 .DS_Store
 
-# Wingman AI generated files
-*.exe
-configs/configs/config.*.yaml
-audio_output
-config.yaml
-apikeys.yaml
-api_keys.yaml
-
-configs/system/secrets.yaml
-
 .frontmatter/database/taxonomyDb.json
-
 frontmatter.json
+
 skills/**/dependencies

.vscode/extensions.json

@@ -1,7 +1,9 @@
 {
     "recommendations": [
         "ms-python.black-formatter",
-        "ms-python.python",
-        "ms-python.pylint"
+        "ms-python.pylint",
+        "ms-python.vscode-pylance",
+        "ms-python.debugpy",
+        "njpwerner.autodocstring"
     ]
 }

README.md

@@ -47,10 +47,11 @@ Since version 2.0, Wingman AI Core acts as a "backend" API (using FastAPI and Py
   - **Instant activation**: Users can (almost) instantly trigger commands by saying exact phrases.
   - Optional: Predetermined responses
 - **Custom Wingman** support: Developers can easily plug-in their own Python scripts with custom implementations
+- **Skills** that can do almost anything. Think Alexa... but better.
 - **directory/file-based configuration** for different use cases (e.g. games) and Wingmen. No database needed.
 - Wingman AI Core exposes a lot of its functionality via **REST services** (with an OpenAPI/Swagger spec) and can send and receive messages from clients, games etc. using **WebSockets**.
 
-We (Team ShipBit) offer an additional [client with a neat GUI](https://www.shipbit.de/wingman-ai) that you can use to configure everything in Wingman AI Core.
+We (Team ShipBit) offer an additional [client with a neat GUI](https://www.wingman-ai.com) that you can use to configure everything in Wingman AI Core.
 
 <img src="assets/wingman-ui-1.png" width="23%"></img> <img src="assets/wingman-ui-3.png" width="23%"></img> <img src="assets/wingman-ui-4.png" width="23%"></img> <img src="assets/wingman-ui-2.png" width="23%"></img>
 
@@ -75,28 +76,16 @@ If you're a developer, you can just clone the repository and start building your
 
 ### Gamers & other interested people
 
-If you're not a developer, you can start with pre-built Wingmen from us or from the community and adapt them to your needs. Since version 2, we offer an [eay-to-use client](https://www.shipbit.de/wingman-ai) for Windows that you can use to cofigure every single detail of your Wingmen. It also handles multiple configurations and offers system-wide settings like audio device selection.
+If you're not a developer, you can start with pre-built Wingmen from us or from the community and adapt them to your needs. Since version 2, we offer an [eay-to-use client](https://www.wingman-ai.com) for Windows that you can use to cofigure every single detail of your Wingmen. It also handles multiple configurations and offers system-wide settings like audio device selection.
 
 ## Providers & cost
 
-Wingman AI Core is free but the AI providers you'll be using might not be. We know that this is a big concern for many people, so we want to offer an easier solution. We're working on "Wingman Pro" which will offer a subscription-based service with a flat fee for all the AI providers you need (and additional GUI features). This way, you won't have to worry about intransparent "pay-per-use" costs. But we're not ready yet.
+Wingman AI Core is free but the AI providers you'll be using might not be. We know that this is a big concern for many people, so we are offering "Wingman Pro" which is a subscription-based service with a flat fee for all the AI providers you need (and additional GUI features). This way, you won't have to worry about intransparent "pay-per-use" costs.
 
-### Unlimited access for Patreons for just $5/month
-
-**Until Wingman Pro is ready**, we offer our [Patreon](https://www.patreon.com/ShipBit) supporters unlimited access to the following services via our Azure infrastructure:
-
-- Open AI GPT-3.5 Turbo via Azure
-- Open AI Whisper via Azure (STT)
-- Azure Speech (STT)
-- Azure TTS
+Check out the pricing and features here: [Wingman AI Pro](https://www.wingman-ai.com)
 
 Wingman AI also supports local providers that you have to setup on your own but can then use and connect with our client for free:
 
-- [whispercpp](https://github.com/ggerganov/whisper.cpp) (STT)
-- [XVASynth](https://store.steampowered.com/app/1765720/xVASynth/) (TTS)
-
-You can basically get everything (except 11Labs and OpenAI TTS) for $5 per month using one of our [Patreon packages](https://www.patreon.com/shipbit/membership). We also offer [one-time purchases](https://www.patreon.com/shipbit/shop) for people who hate subscriptions.
-
 ### Other providers
 
 You can also use your own API key to use the following services:
@@ -190,16 +179,13 @@ For updates and more information, visit the [StarHead website](https://star-head
 
 ### Noteworthy community projects
 
-- [UEXCorp](https://discord.com/channels/1173573578604687360/1179594417926066196) by @JayMatthew: A custom Wingman that utilizes the UEX Corp API to pull live data for Star Citizen. Think StarHead on steroids.
-- [Cora](https://discord.com/channels/1173573578604687360/1205649611470016512) by @eXpG_kalumet: A fork offering automatic keybinding, multiple Wingmen using a single key, bi-directional UEXCorp communication, screenshot analysis and much more for Star Citizen. Note that Cora is a standalone fork that you cannot easily integrate into the latest Wingman AI Core release.
-
-Cora Showcase Video:
-
-[![IMAGE ALT TEXT](https://img.youtube.com/vi/5eE5VLuKtTw/0.jpg)](https://www.youtube.com/watch?v=5eE5VLuKtTw 'Wingman AI Release Trailer')
+- [UEXCorp](https://discord.com/channels/1173573578604687360/1179594417926066196) by @JayMatthew: A former Custom Wingman, now Skill that utilizes the UEX Corp API to pull live data for Star Citizen. Think StarHead on steroids.
+- [Clippy](https://discord.com/channels/1173573578604687360/1241854342282219662) by @teddybear082: A tribute Skill to the sketchy Microsoft assistant we all used to hate.
+- [WebSearch](https://discord.com/channels/1173573578604687360/1245432544946688081) by @teddybear082: A Skill that can pull data from websites (and quote the sources) for you.
 
 ## Can I configure Wingman AI Core without using your client?
 
-Yes, you can! You can edit all the configs in your `%APP_DATA%/Roaming/ShipBit/WingmanAI/[version]` directory.
+Yes, you can! You can edit all the configs in your `%APP_DATA%\ShipBit\WingmanAI\[version]` directory.
 
 The YAML configs are very indentation-sensitive, so please be careful. We recommend using [VSCode](https://code.visualstudio.com/) with the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) to edit them.
 
@@ -249,7 +235,7 @@ If you want to read some code first and understand how it all works, we recommen
 
 - `http://127.0.0.1:8000/docs` - The OpenAPI (ex: Swagger) spec
 - `wingman_core.py` - most of the public API endpoints that Wingman AI exposes
-- The config files in `%APP_DATA%/Roaming/ShipBit/WingmanAI/[version]` to get an idea of what's configurable.
+- The config files in `%APP_DATA%\ShipBit\WingmanAI\[version]` to get an idea of what's configurable.
 - `Wingman.py` - the base class for all Wingmen
 - `OpenAIWingman.py` - derived from Wingman, using all the providers
 - `Tower.py` - the factory that creates Wingmen
@@ -289,7 +275,7 @@ This list will inevitably remain incomplete. If you miss your name here, please
 
 #### Special thanks
 
-- [**JayMatthew aka SawPsyder**](https://robertsspaceindustries.com/citizens/JayMatthew) and @teddybear082 for outstanding moderation in Discord, constant feedback and valuable core contributions
+- [**JayMatthew aka SawPsyder**](https://robertsspaceindustries.com/citizens/JayMatthew), @teddybear082 and @Thaendril for outstanding moderation in Discord, constant feedback and valuable Core & Skill contributions
 - @lugia19 for developing and improving the amazing [elevenlabslib](https://github.com/lugia19/elevenlabslib).
 - [Knebel](https://www.youtube.com/@Knebel_DE) who helped us kickstart Wingman AI by showing it on stream and grants us access to the [StarHead API](https://star-head.de/) for Star Citizen.
 - @Zatecc from [UEX Corp](https://uexcorp.space/) who supports our community developers and Wingmen with live trading data for Star Citizen using the [UEX Corp API](https://uexcorp.space/api.html).

api/commands.py

@@ -44,6 +44,7 @@ class LogCommand(WebSocketCommandModel):
     source: LogSource = "system"
     tag: Optional[CommandTag] = None
     skill_name: Optional[str] = None
+    additional_data: Optional[dict] = None
 
 
 class PromptSecretCommand(WebSocketCommandModel):

api/enums.py

@@ -43,6 +43,7 @@ class CommandTag(Enum):
 
 class CustomPropertyType(Enum):
     STRING = "string"
+    TEXTAREA = "textarea"
     NUMBER = "number"
     BOOLEAN = "boolean"
     SINGLE_SELECT = "single_select"
@@ -113,9 +114,10 @@ class OpenAiTtsVoice(Enum):
 
 
 class SoundEffect(Enum):
-    ROBOT = "ROBOT"
-    RADIO = "RADIO"
-    INTERIOR_HELMET = "INTERIOR_HELMET"
+    AI = "AI"
+    LOW_QUALITY_RADIO = "LOW_QUALITY_RADIO"
+    MEDIUM_QUALITY_RADIO = "MEDIUM_QUALITY_RADIO"
+    HIGH_END_RADIO = "HIGH_END_RADIO"
     INTERIOR_SMALL = "INTERIOR_SMALL"
     INTERIOR_MEDIUM = "INTERIOR_MEDIUM"
     INTERIOR_LARGE = "INTERIOR_LARGE"
@@ -154,6 +156,10 @@ class ConversationProvider(Enum):
     AZURE = "azure"
     WINGMAN_PRO = "wingman_pro"
 
+class ImageGenerationProvider(Enum):
+    OPENAI = "openai"
+    WINGMAN_PRO = "wingman_pro"
+
 
 class SummarizeProvider(Enum):
     OPENAI = "openai"
@@ -274,6 +280,9 @@ class VoiceActivationSttProviderEnumModel(BaseEnumModel):
 class ConversationProviderEnumModel(BaseEnumModel):
     conversation_provider: ConversationProvider
 
+class ImageGenerationProviderEnumModel(BaseEnumModel):
+    image_generation_provider: ImageGenerationProvider
+
 
 class SummarizeProviderEnumModel(BaseEnumModel):
     summarize_provider: SummarizeProvider

api/interface.py

@@ -10,6 +10,7 @@
     AzureRegion,
     ConversationProvider,
     GroqModel,
+    ImageGenerationProvider,
     MistralModel,
     CustomPropertyType,
     TtsVoiceGender,
@@ -315,7 +316,10 @@ class WingmanProSettings(BaseModel):
 
 class SoundConfig(BaseModel):
     play_beep: bool
-    """adds a beep/Quindar sound before and after the wingman talks"""
+    """adds a Beep/Quindar sound before and after the wingman talks"""
+
+    play_beep_apollo: bool
+    """adds a Apollo Beep sound before and after the wingman talks"""
 
     effects: list[SoundEffect]
     """You can put as many sound effects here as you want. They stack and are added in the defined order here."""
@@ -352,6 +356,8 @@ class FeaturesConfig(BaseModel):
     conversation_provider: ConversationProvider
     summarize_provider: SummarizeProvider
     remember_messages: Optional[int] = None
+    image_generation_provider: ImageGenerationProvider
+    use_generic_instant_responses: bool
 
 
 class CommandKeyboardConfig(BaseModel):
@@ -424,12 +430,9 @@ class CommandConfig(BaseModel):
     """The actions to execute when the command is called. You can use keyboard, mouse and wait actions here."""
 
 
-class CustomWingmanClassConfig(BaseModel):
+class CustomClassConfig(BaseModel):
     module: str
-    """Where your code is located. Use '.' as path separator!
-    We advise you to put all your custom wingmen into the /wingmen directory.
-    "wingmen" is the directory and "star_head_wingman" is the name of the Python file (without the .py extension).
-    """
+    """Where your code is located. Use '.' as path separator!"""
 
     name: str
     """The name of your class within your file/module."""
@@ -440,19 +443,19 @@ class LabelValuePair(BaseModel):
     value: str | int | float | bool
 
 
-class CustomWingmanProperty(BaseModel):
+class CustomProperty(BaseModel):
     id: str
     """The name of the property. Has to be unique"""
     name: str
     """The "friendly" name of the property, displayed in the UI."""
     value: str | int | float | bool | None
     """The value of the property"""
+    property_type: CustomPropertyType
+    """Determines the type of the property and which controls to render in the UI."""
     hint: Optional[str] = None
     """A hint for the user, displayed in the UI."""
     required: Optional[bool] = False
     """Marks the property as required in the UI."""
-    property_type: Optional[CustomPropertyType] = CustomPropertyType.STRING
-    """Determines the type of the property and which controls to render in the UI."""
     options: Optional[list[LabelValuePair]] = None
     """If property_type is set to 'single_select', you can provide options here."""
 
@@ -467,15 +470,14 @@ class SkillExample(BaseModel):
     answer: LocalizedMetadata
 
 
-class SkillConfig(CustomWingmanClassConfig):
+class SkillConfig(CustomClassConfig):
+    description: LocalizedMetadata
     prompt: Optional[str] = None
     """An additional prompt that extends the system prompt of the Wingman."""
-    custom_properties: Optional[list[CustomWingmanProperty]] = None
+    custom_properties: Optional[list[CustomProperty]] = None
     """You can add custom properties here to use in your custom skill class."""
-    description: Optional[LocalizedMetadata] = None
     hint: Optional[LocalizedMetadata] = None
     examples: Optional[list[SkillExample]] = None
-    commands: Optional[list[CommandConfig]] = None
 
 
 class SkillBase(BaseModel):
@@ -522,12 +524,12 @@ def __getitem__(self, item):
     def __setitem__(self, key, value):
         self.extra_properties[key] = value
 
-    custom_properties: Optional[list[CustomWingmanProperty]] = None
+    custom_properties: Optional[list[CustomProperty]] = None
     """You can add custom properties here to use in your custom wingman class."""
 
     disabled: Optional[bool] = False
     """Set this to true if you want to disable this wingman. You can also just remove it from the config."""
-    custom_class: Optional[CustomWingmanClassConfig] = None
+    custom_class: Optional[CustomClassConfig] = None
     """If you want to use a custom Wingman (Python) class, you can specify it here."""
     name: str
     """The "friendly" name of this Wingman. Can be changed by the user."""

providers/elevenlabs.py

@@ -3,7 +3,7 @@
     GenerationOptions,
     PlaybackOptions,
 )
-from api.enums import ElevenlabsModel, WingmanInitializationErrorType
+from api.enums import ElevenlabsModel, SoundEffect, WingmanInitializationErrorType
 from api.interface import ElevenlabsConfig, SoundConfig, WingmanInitializationError
 from services.audio_player import AudioPlayer
 from services.secret_keeper import SecretKeeper
@@ -52,15 +52,30 @@ async def play_audio(
 
         def notify_playback_finished():
             audio_player.playback_events.unsubscribe("finished", playback_finished)
+
+            contains_high_end_radio = SoundEffect.HIGH_END_RADIO in sound_config.effects
+            if contains_high_end_radio:
+                audio_player.play_wav("Radio_Static_Beep.wav")
+
             if sound_config.play_beep:
-                audio_player.play_beep()
+                audio_player.play_wav("beep.wav")
+            elif sound_config.play_beep_apollo:
+                audio_player.play_wav("Apollo_Beep.wav")
+
             WebSocketUser.ensure_async(
                 audio_player.notify_playback_finished(wingman_name)
             )
 
         def notify_playback_started():
             if sound_config.play_beep:
-                audio_player.play_beep()
+                audio_player.play_wav("beep.wav")
+            elif sound_config.play_beep_apollo:
+                audio_player.play_wav("Apollo_Beep.wav")
+
+            contains_high_end_radio = SoundEffect.HIGH_END_RADIO in sound_config.effects
+            if contains_high_end_radio:
+                audio_player.play_wav("Radio_Static_Beep.wav")
+
             WebSocketUser.ensure_async(
                 audio_player.notify_playback_started(wingman_name)
             )

providers/open_ai.py

@@ -285,6 +285,7 @@ def buffer_callback(audio_buffer):
                     buffer_callback,
                     sound_config,
                     wingman_name=wingman_name,
+                    use_gain_boost=True,  # "Azure Streaming" low gain workaround
                 )
             else:
                 await audio_player.play_with_effects(

providers/wingman_pro.py

@@ -14,7 +14,7 @@
 
 class WingmanPro:
     def __init__(
-        self, wingman_name: str, settings: WingmanProSettings, timeout: int = 30
+        self, wingman_name: str, settings: WingmanProSettings, timeout: int = 120
     ):
         self.wingman_name: str = wingman_name
         self.settings: WingmanProSettings = settings
@@ -98,7 +98,7 @@ def ask(
             json=data,
             timeout=self.timeout,
         )
-        if response.status_code == 403:
+        if response.status_code == 401 or response.status_code == 403:
             self.send_unauthorized_error()
             return None
         else:
@@ -178,6 +178,7 @@ def buffer_callback(audio_buffer):
                 buffer_callback=buffer_callback,
                 config=sound_config,
                 wingman_name=wingman_name,
+                use_gain_boost=True,  # "Azure Streaming" low gain workaround
             )
         else:  # non-streaming
             response = requests.post(
@@ -234,6 +235,30 @@ async def generate_openai_speech(
                 wingman_name=wingman_name,
             )
 
+    async def generate_image(
+        self,
+        text: str,
+    ):
+        data = {
+            "text": text,
+        }
+        response = requests.post(
+            url=f"{self.settings.base_url}/generate-image",
+            params={
+                "region": self.settings.region.value,
+            },
+            headers=self._get_headers(),
+            json=data,
+            timeout=self.timeout,
+        )
+        if response is not None:
+            if response.status_code == 403:
+                self.send_unauthorized_error()
+                return
+            else:
+                response.raise_for_status()
+            return response.text
+
     def get_available_voices(self, locale: str = ""):
         response = requests.get(
             url=f"{self.settings.base_url}/azure-voices",