Voice chat

Feature request. Would it be possible to add elevenlabs_tts and whisper_stt to the discord bot so you could talk to it and have it reply with voice? I know these features work with the web ui.

Yup, it's on the roadmap!  (You know, the one in my head)..
It had been dependent on me getting more hardware, which just happened to show up yesterday!
I'll likely want to get some bugfixes and plugin integration working first, but I plan to get started on it in not too long.

 Thank you for writing this code. Cant wait.

Thanks! :). Let's keep this open until I get it in, for tracking.

Always welcome to have others contribute as well!  I don't have to be a hero, just trying to get it off the ground, really. :)
Personally what I want is an OSS solution, and open to package suggestions.  We'll need to include packages for:

stt
tts
(possibly) wakeword detection, but only if the tts solution is too expensive to run continually

So far it seems like OpenAI's whisper is the best option for STT, and is MIT license.
https://github.com/openai/whisper
There were a few other options I looked at as well, such as https://github.com/mozilla/DeepSpeech and https://github.com/coqui-ai/STT (by some of the same people that did DeepSpeech).  But it seems CoquiAI has decided not to invest any more in STT, or so they told me.  Other options didn't have a great accuracy rate, at least as far as my testing went.
An alternative to whisper would be ESPNet (https://github.com/espnet/espnet), but I haven't played around with that yet.  It's under Apache 2.0, so the license works out.
In terms of TTS, tortoise-tts has the best results I've seen of any of the OSS projects.  It's Apache 2.0 as well.  I believe ESPNet may also offer something here.
Commercial offerings produce better-sounding TTS results, at least to me, but I'm not happy depending on them for a few reasons:

this project is all about self-hosting
the space is changing so fast that I suspect a year from now commercial offerings will be completely different, so it's a moving target at best

That said, it might make sense further down the line to also support 3rd party offerings for either of these features, simply because of the hardware requirements.  Running Oobabooga +  whisper + tortoisetts may be a pretty demanding on a single-gpu system, especially if inference needs to happen fast enough to engage in a live voice conversation.  Hence my desire to wait until I had more hardware available myself.
But for now, I would like to focus on OSS, self-hosted offerings for this feature.
Would welcome any thoughts or experiences in this area that others have!

I agree totally with your words on TTS. I've actually been messing with the STT whisper options on oobabooga a little and I'm not disappointed at all. Not sure how you'll implement the listening and responding... unless you plan on having it join a VC, listen, respond, like an always available assistant (or nuisance, like one named neuro-sama).
Personally, I am using an AMD MI60 (32GB) GPU for my text generation, image generation, and voice generation, sitting at about 53% VRAM usage with a 30B 4bit model loaded. I don't have any issues running all these demanding models, but I understand my case may be an outlier.
I haven't heard or used espnet, I'll test it out see if I see results.

Bark TTS can sometimes produce very good results and is open source.  https://github.com/suno-ai/bark
I have played with this one a little, I think oobabooga might have implemented some support..  I am not sure I will have to look into it.  I have heard of another TTS that is practically real time

Tortoise-TTS has an API on it too.

Making some progress!
Things I've learned:

discord.py only implements sending audio, but not receiving / decoding audio.  It could be done, but it looks like there are better other libraries which already work.  In particular, songbird looks nice, and I have it able to parse audio when oobabot has joined an audio channel.

tortoise-tts is slow.  Will need a different library for actual production use.  Punting on this decision for now.

Whisper is terrific -- impressively accurate and fast.  Whisper.cpp is very impressive too.  It's only a few MB of footprint (as opposed to ~GB for the torchaudio setup that GPU-whisper uses).  And it's actually pretty fast.  Going to try and use that, as it would let the whole setup work on a single-GPU system.  It can run on a huge variety of systems, from mobile phones on up, with extra acceleration for Apple silicon macs, NVIDIA GPUs, and OpenCL GPUs.

Current plan is to add a binary component which will be the "driver" part of Songbird plus Whisper.cpp.  Its job will be to take in audio streams, send them to whisper.cpp and generate a transcript.  It will also be responsible for playing the response output, however that gets generated.  The MVP version will just likely use something simple, to be improved later ofc.
I'll take a look at Bark, etc. when I get to that stage.  Thanks for the recommendations!

And... the very first speech-to-text where oobabot is listening to the audio from a discord voice channel is working!

Is this actually implemented now? Any updates from a few weeks back? Would love this feature

Guessing this bad boy cant post audio to the chat itself? How do you have it running in a voice room, then?

Update:
I've been hard at work on this.  It's a little tricky to get right, and took a few rounds to get to a place where I'm happy with.
Progress as of 6/22

[✅ done] bot can join and leave voice channels
[✅ done] bot can transcribe audio
[✅ done] bot transcribes audio into text reliably
[✅ done] bot can generate text responses from spoken wake words
[🚧 in progress] integrate voice work into oobabot-plugin
[todo] initial Text To Speech to post voice replies, using system text-to-speech (e.g. osx "say")
[todo] fancier TTS using tbd AI-driven fanciness
...
[todo] profit!  (I mean, release for free)

Most of the work so far has been happening in https://github.com/chrisrude/discriviner
and on the oobabot-audio fork in this repo.
Discrivener is a separate CLI app that joins the Discord audio channel and transcribes audio.  It runs realtime on CPU only, so can work on single-GPU systems.   It's written in Rust, rather than python, for performance reasons.  The underlying transciption is done by whisper.cpp.

If you want to check out what's been done so far on the python side, I just merged the audio branch back into main.
https://github.com/chrisrude/oobabot/pull/52

For the super intrepid, there are bleeding-edge binaries available at github.com/chrisrude/discrivener
Here's a brief preview of how to test this out.  Currently, only Intel-based Linux and OSX are supported.
Download discrivener-json binary
Currently there are binaries for 64-bit x86 Linux (kernel 3.2+, glibc 2.17+) as well as 64-bit Intel macOS (10.7+, Lion+).  You can download them here.
Download model
Download a model of your choice from https://ggml.ggerganov.com/.
I recommend ggml-model-whisper-base.en-q5_1.bin as a starting point.
Configure oobabot
First, upgrade oobabot to version 0.2.0 (or later).
If you're using the command-line oobabot, you'll want to regenerate your config.yml with
mv config.yml config.yml.orig && oobabot -c config.yml.orig --gen > config.yml

If you're using the oobabot-plugin, just upgrade to version 0.2.0 of the oobabot-plugin, then switch to the Advanced tab.
Now, in your config.yml, under discord:, there should be two new keys:
discrivener_location -- set this to the full path and filename of the discrivener-json-* file you downloaded
discrivener_model_location -- set this to the full path and filename of the ggml-model-*.bin file you downloaded
Restart oobabot, or else restart the oobabooga service (needed to show the new 'Audio' tab).
Your bot should restart and if everything works, you should see something like:
2023-07-07 03:40:50,320  INFO Discrivener found at ...some path.../discrivener-json
2023-07-07 03:40:50,320  INFO Discrivener model found at ...some path.../ggml-base.en.bin

And then later:
2023-07-07 03:40:53,224 DEBUG Registering audio commands
2023-07-07 03:40:53,773  INFO Registered command: lobotomize: Erase (bot)'s memory of any message before now in this channel.
2023-07-07 03:40:53,773  INFO Registered command: say: Force (bot) to say the provided message.
2023-07-07 03:40:53,774  INFO Registered command: join_voice: Have (bot) join the voice channel you are in right now.
2023-07-07 03:40:53,774  INFO Registered command: leave_voice: Have (bot) leave the voice channel it is in.

If this is your first time, it may take 5 to 10 minutes for Discord to recognize the two new commands: /join_voice and /leave_voice.
Then, to test:

join a discord voice channel with your user account.  This currently must be a voice channel in a Discord server that the bot has access to
under that same account, send the bot the command /join_voice.  It doesn't matter what channel you use to send this command from.

The bot should look for the voice channel you're currently in and join it.
The bot will then stay in that channel until it receives a /leave_voice command, or until the bot is stopped.
Talk to the bot!
If you're using the oobabooga-plugin (recommended), you'll now see audio logged to the "Audio" tab under a few second of latency.
If you are the only other person in the channel, the bot will respond to every message it hears.
If there is more than one human in the channel, the bot will only respond if it receives a wakeword, and then with a very low percentage chance afterwards.  This is still a work in progress.
This is very very early, and has some known limitations:

the voice synthesis is crude, and sounds like Steven Hawking's wheelchair.  A better-sounding voice is in the works, but I wanted to start with something workable
the bot doesn't have social skills.  In a large group, it will insert random messages and generally be annoying.  One-on-one conversations are much better.  As such, it's tuned to always respond to anything you say if you're the only human in the voice channel.  But if there is more than one, it will only respond to wakewords, and then VERY occasionally to random messages after that.

Better documentation and builds are in the works, I just wanted to put this out to see if it works for anyone other than me. :)

For the super intrepid, there are bleeding-edge binaries available at github.com/chrisrude/discrivener.
Here's a brief preview of how to test this out.  Currently, only Intel-based Linux and OSX are supported.
Download a discrivener-json binary
Currently there are binaries for 64-bit x86 Linux (kernel 3.2+, glibc 2.17+) as well as 64-bit Intel macOS (10.7+, Lion+).  You can download them here.
Download and extract espeak-ng-data.tar.gz
From the same page, download espeak-ng-data.tar.gz and extract it under /usr/local/share.  You should wind up with a /usr/local/share/espeak-ng-data with a bunch of *_data files in it.
Download model
Download a model of your choice from https://ggml.ggerganov.com/.
I recommend ggml-model-whisper-base.en-q5_1.bin as a starting point.
Configure oobabot
First, upgrade oobabot to version 0.2.0 (or later).
If you're using the command-line oobabot, you'll want to regenerate your config.yml with
mv config.yml config.yml.orig && oobabot -c config.yml.orig --gen > config.yml

If you're using the oobabot-plugin, just upgrade to version 0.2.0 of the oobabot-plugin, then switch to the Advanced tab.
Now, in your config.yml, under discord:, there should be two new keys:
discrivener_location -- set this to the full path and filename of the discrivener-json-* file you downloaded
discrivener_model_location -- set this to the full path and filename of the ggml-model-*.bin file you downloaded
Restart oobabot, or else restart the oobabooga service (needed to show the new 'Audio' tab).
Your bot should restart and if everything works, you should see something like:
2023-07-07 03:40:50,320  INFO Discrivener found at ...some path.../discrivener-json
2023-07-07 03:40:50,320  INFO Discrivener model found at ...some path.../ggml-base.en.bin

And then later:
2023-07-07 03:40:53,224 DEBUG Registering audio commands
2023-07-07 03:40:53,773  INFO Registered command: lobotomize: Erase (bot)'s memory of any message before now in this channel.
2023-07-07 03:40:53,773  INFO Registered command: say: Force (bot) to say the provided message.
2023-07-07 03:40:53,774  INFO Registered command: join_voice: Have (bot) join the voice channel you are in right now.
2023-07-07 03:40:53,774  INFO Registered command: leave_voice: Have (bot) leave the voice channel it is in.

If this is your first time, it may take 5 to 10 minutes for Discord to recognize the two new commands: /join_voice and /leave_voice.
Then, to test:

join a discord voice channel with your user account.  This currently must be a voice channel in a Discord server that the bot has access to
under that same account, send the bot the command /join_voice.  It doesn't matter what channel you use to send this command from.

The bot should look for the voice channel you're currently in and join it.
The bot will then stay in that channel until it receives a /leave_voice command, or until the bot is stopped.
Talk to the bot!
If you're using the oobabooga-plugin (recommended), you'll now see audio logged to the "Audio" tab under a few second of latency.
If you are the only other person in the channel, the bot will respond to every message it hears.
If there is more than one human in the channel, the bot will only respond if it receives a wakeword, and then with a very low percentage chance afterwards.  This is still a work in progress.

Example of what the transcript UI looks like (using oobabot-plugin):
This is what the bot is hearing and saying, the transcript just helps you see what's going on.

Thanks @chrisrude for this
It's soo coool bro I will try it out

btw, I've been having occasional issues using the uploaded discrivener binaries.  If you're seeing weird audio glitching, just let me know and don't spend too much time on it, it's likely a problem with the compilation process.
Things work fine when I build locally, but it's harder to make the binary work well on distribution.

For TTS I have been messing around with silero-api-server and it has been pretty solid.

For TTS I have been messing around with silero-api-server and it has been pretty solid.

I heard it performs pretty good, but it's still a non open source platform.

For TTS I have been messing around with silero-api-server and it has been pretty solid.

I heard it performs pretty good, but it's still a non open source platform.

What do you mean? Like the models?

For TTS I have been messing around with silero-api-server and it has been pretty solid.

I heard it performs pretty good, but it's still a non open source platform.

What do you mean? Like the models?

Sorry, I was thinking of something else. I just looked at it and I'll test it out tomorrow if I get the chance.
Looks promising, what's the delay like? Any options for custom trained voices?

Yeah someone made a server API for Silero to use to use in Silly Tavern since TGW doesn't stream audio. On a Xeon E5-2430 (6 core 12 threads) which is from 2012 I get about 8x time real time speed (1 second for 8 seconds of audio). Custom voices are theoretically possible but I have heard you need 5 or more hours of good audio to train it. It comes with 117 English voices though.

For TTS I have been messing around with silero-api-server and it has been pretty solid.

I heard it performs pretty good, but it's still a non open source platform.

What do you mean? Like the models?

Sorry, I was thinking of something else. I just looked at it and I'll test it out tomorrow if I get the chance. Looks promising, what's the delay like? Any options for custom trained voices?

Yeah someone made a server API for Silero to use to use in Silly Tavern since TGW doesn't stream audio. On a Xeon E5-2430 (6 core 12 threads) which is from 2012 I get about 8x time real time speed (1 second for 8 seconds of audio). Custom voices are theoretically possible but I have heard you need 5 or more hours of good audio to train it. It comes with 117 English voices though.

Looks like it works pretty good, actually. Unfortunately it currently breaks the bot if it's enabled in webui.

Looks like it works pretty good, actually. Unfortunately it currently breaks the bot if it's enabled in webui.

In the WebUI having it enabled breaks anything that uses the API. So, that is why you use the actual API server not the one that comes with the WebUI. https://github.com/ouoertheo/silero-api-server

Looks like it works pretty good, actually. Unfortunately it currently breaks the bot if it's enabled in webui.

In the WebUI having it enabled breaks anything that uses the API. So, that is why you use the actual API server not the one that comes with the WebUI. https://github.com/ouoertheo/silero-api-server

I gotcha, so the API server is only sent text by sillytavern running on the client device (currently), but this functionality could definitely be put into oobabot.
This is pretty promising, I'm wondering how far @chrisrude is in their current implementation of the other TTS/STT solution.

Looks like it works pretty good, actually. Unfortunately it currently breaks the bot if it's enabled in webui.

In the WebUI having it enabled breaks anything that uses the API. So, that is why you use the actual API server not the one that comes with the WebUI. https://github.com/ouoertheo/silero-api-server

I gotcha, so the API server is only sent text by sillytavern running on the client device (currently), but this functionality could definitely be put into oobabot. This is pretty promising, I'm wondering how far @chrisrude is in their current implementation of the other TTS/STT solution.

I have made a couple of things using it like a Alex like thing and a endless conversation bot. I have it so the is audio streamed directly from the server so there is no temp file. All it needs is the text to made into speech, the language, and what voice it is going to use (It also takes "random" as a speaker). Plus of course what the host is too which makes it so you can run it on another separate PC. I am not sure how Discord's API handles audio playback but I feel like it would be a good fit for this project as it is fast and not bad sounding. Silero also have a lightweight STT that I have used but if you are running it on a server, Wisper would likely be better in terms of not as high error rate.

I am not sure how Discord's API handles audio playback

The dev is currently implementing his own playback, but it's the same idea as music streaming bots on discord. You can have them stream audio directly to the voice channel with no problem. Definitely looking forward to seeing if this gets added, since the voice generation is very quick!

I am not sure how Discord's API handles audio playback

The dev is currently implementing his own playback, but it's the same idea as music streaming bots on discord. You can have them stream audio directly to the voice channel with no problem. Definitely looking forward to seeing if this gets added, since the voice generation is very quick!

Would it be possible to add https://github.com/BuffMcBigHuge/text-generation-webui-edge-tts to the discord bot so the bot could answer with custom voices?


Error to load an app: MissingPluginException

After installing Flet support through the Ubuntu 22.04 LTS PIP, every attempt to load an application with python3 app_name.py just loads an empty window with the loading in progress icon and no content. The error message below is displayed.
[ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: MissingPluginException(No implementation found for method isFocused on channel window_manager)
The webapp version always loads successfully in the browser. What am I missing?
PS: I also have Flutter installed. Could this be causing some bad behavior?

Will fix that shortly!

It's been fix. Upgrade Flet module to 0.1.38.


Refactored project to incorporate a build system

The project now uses Autotools as its build system, and running the
usual './configure', 'make', 'make install' results in the building of a
shared library that gets installed in /usr/local/lib by default, as well
as the genann.h header, which gets installed in /usr/local/include.
All of the usual configuration flexibility is present, including the
addition of gmp and mpfr simply as a demonstration of the
auto-configuration ability of the system. The build system looks for gmp
and mpfr if the user specifies --with-gmp or --with-mpfr and looks for
the installation. If found, it automatically links the target to those
libraries.
The usual variables are configurable (CC, CFLAGS, CPPFLAGS, LDFLAGS,
LIBS), and the configuration even allows specifying the activation
function through there with ACTIVATION_FUNCTION=SIGMOID, although LINEAR
and THRESHOLD are also options. Specifying these options defines the
necessary preprocessor symbol.
Running 'make check' builds the test, links it against the library, and
runs it.
Running 'make examples' builds all four examples.

It looks like you put a lot of work into this, but I don't think a 2-file library needs a build system of this caliber. The point of keeping the library in a single .c file is that a user can just do gcc -c genann.c -o genann.o. They don't need a several hundred line long build system.
Great job on the documentation though, most PRs skip over that.

Yea, I completely understand. I've been on an autotools kick recently, and I came across a comment asking for building a shared library, so I figured I'd get some good practice out of it. I definitely agree it's probably overkill though, haha.
I love the library, by the way, it really is awesome.


Parameter for text align

I am having issues when I am using some specific strings (e.g. "Updating queue number...") as progress messages where it turns out to be weird looking because of some weird text alignment. (See attached pic)

I checked the code and it confirmed that it was using justify text alignment.
Text(_dialogMessage, textAlign: TextAlign.justify, style: _messageStyle)
I hope you can add an optional parameter for setting the text alignment so I can easily adjust it accordingly.
Thanks! :)

Please checkout latest version v1.2.2, I am closing the issue now, feel free to re-open if it doesn't solve your requirement


[Theme Request] Kirigiri Kyoko (Danganronpa)

Who/What do you want to be featured?
Kirigiri Kyoko (Danganronpa)
What do they look like?

What type of theme?

[ ] Dark Theme

Additional Information (Optional)

I like the vibe she gives off. Her color palette is similar to some of the existing themes though, so I might need to imagine something different, or figure out a way to easily introduce similar colored girls .


Roll engine 949fb0fb4cbe..d88e0b8229ec (1 commits)

https://github.com/flutter/engine/compare/949fb0fb4cbe...d88e0b8229ec
git log 949fb0fb4cbe..d88e0b8229ec --no-merges --oneline
d88e0b822 Roll src/third_party/skia 2f8c60e28682..ca858b68c34b (2 commits) (flutter/engine#6142)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

@rmistry @liyuqian : New roll has been created by https://autoroll.skia.org/r/flutter-engine-flutter-autoroll

Auto-roller completed checks. About to merge.


Use right API for websocket event subscribing

This only a documentation change.
onmessage,onerror,onopen,onclose doesn't exists anymore and according to this example https://github.com/facebook/react-native/blob/master/Examples/UIExplorer/WebSocketExample.js#L114 addEventListener should be used instead

onopen etc are defined here: https://github.com/facebook/react-native/blob/2525feb37f7b9b3a18625d58e9b9013573c8c129/Libraries/WebSocket/WebSocket.js#L73-L76 and are implemented by EventTarget. I think they're still the preferred public interface.


Add 'T | PromiseLike' inference from awaited types

This re-introduces the T | PromiseLike<T> inference heuristic previously added in the awaited type PR. This heuristic improves type inference for cases where we are inferring to a union consisting of a single type variable T and one or more Promise-like types whose "promised type" (i.e., the type of the value parameter to the onfulfilled callback) is precisely T. This primarily impacts inference for cases like Promise.resolve and Promise#then:
declare const p4: Promise<number> | Promise<string>;

// infers `number | string`
const p5 = Promise.resolve(p4);

One minor change vs. the implementation in the awaited types implementation is that this version detects unions with more than two constituents (i.e., T | PromiseLike<T> | Promise<T>) as long as all of the constituents are essentially promises for T.

@typescript-bot perf test
@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

Hmm. The failing JQuery tests are falling afoul of union subtype reduction. The then method is declared with something like this:
then<ARD = never, AJD = never, ...>(
  doneFilter: (t: TR, u: UR, v: VR, ...s: SR[]) => PromiseBase<ARD, AJD, ...> | Thenable<ARD> | ARD,
  failFilter: (t: TJ, u: UJ, v: VJ, ...s: SJ[]) => PromiseBase<ARF, ...> | Thenable<ARF> | ARF,
  progressFilter: (t: TN, u: UN, v: VN, ...s: SN[]) => PromiseBase<ARP, ...> | Thenable<ANP> | ANP
): PromiseBase<ARD | ARF | ARP, AJD | AJF | AJP, ...>;

Since PromiseBase<ARD, AJD, ...> is assignable to Thenable<ARD>, we drop PromiseBase during subtype reduction, so its not available as a candidate for inference.

@typescript-bot perf test
@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

@typescript-bot run dt
@typescript-bot test this
@typescript-bot user test this

@typescript-bot user test this

RWC Baselines look good.
User test baselines look good
DT baselines are clean.

@ahejlsberg, @weswigham I'd appreciate if you could take a look.

@rbuckton is this for 3.9 or 4.0?

I was hoping for 3.9, as this was initially part of awaited and does fix some inference-related errors in RWC, but I'd defer to @DanielRosenwasser to weigh on on this. I'll have to dig through our issues list for related issues though.

I think we should take this as an item in the design meeting to understand whether it goes in regardless of awaited. It's way too late in the game to add it to 3.9 IMO.

Unfortunately, we never finished reviewing this PR. It is pretty old now, so I'm going to close it to reduce the number of open PRs.


Thanks for the info and pictures!

Thanks for the info and pictures!
Could you let me know what the AOV/ FOV is on your Blender camera?
Originally posted by @JustinPedersen96 in https://github.com/JustinPedersen/maya_fspy/issues/10#issuecomment-869478005
Hi @JustinPedersen

Another example


Failed to connect to Pusher.

Trying at the moment to get the WebSockets working as a replacement for Pusher but I always receive the error message Failed to connect to Pusher. when trying to follow the steps in https://beyondco.de/docs/laravel-websockets/basic-usage/pusher#pusher-configuration.
        'pusher' => [
            'driver'  => 'pusher',
            'key'     => env('PUSHER_APP_KEY'),
            'secret'  => env('PUSHER_APP_SECRET'),
            'app_id'  => env('PUSHER_APP_ID'),
            'options' => [
                'cluster'      => env('PUSHER_APP_CLUSTER'),
                'useTLS'       => true,
                'host'         => 'valet.host',
                'port'         => 6001,
                'scheme'       => 'https',
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => env('APP_ENV') !== 'local',
                    CURLOPT_SSL_VERIFYPEER => env('APP_ENV') !== 'local',
                ],
            ],
        ],

Is there any configuration that I am overlooking or misusing? Normal pusher works fine and the WebSockets are also running without issues.

Perhaps you need to set the scheme to http if you don't use SSL.

I am using HTTPS both with Valet and the production system and configured the SSL certificates.

Having self-signed certificates need the curl_options to have them set to true.

Tried that before and tried it again but that results in the same issue.
// With curl_options explicitly set to true
            'options' => [
                'cluster'      => env('PUSHER_APP_CLUSTER'),
                'useTLS'       => true,
                'host'         => '<IP_ADDRESS>',
                'port'         => 6001,
                'scheme'       => 'https',
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => true,
                    CURLOPT_SSL_VERIFYPEER => true,
                ],
            ],

// Without curl_options, letting it use defaults
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'useTLS'  => true,
                'host'    => '<IP_ADDRESS>',
                'port'    => 6001,
                'scheme'  => 'https',
            ],

Also tried to set useTLS to false to avoid wrong ports but the same issue persists.

Sorry for disorienting you, but values should be false so they won't check for additional SSL verifications.

That's what the behaviour at the moment is when I run it locally because I use env('APP_ENV') !== 'local' which will only enable the SSL checks in production/staging.

Is your websockets server running?

Yeah, when I manually post it I can see that it receives the requests by throwing an error because I am omitting the signature.
>>> https://valet.test:6001/apps/ID/events
Starting the WebSocket server on port 6001...
Exception `Symfony\Component\HttpKernel\Exception\HttpException` thrown: `Invalid auth signature provided.`
Exception `Symfony\Component\HttpKernel\Exception\HttpException` thrown: `Invalid auth signature provided.`
Exception `Symfony\Component\HttpKernel\Exception\HttpException` thrown: `Invalid auth signature provided.`

Based on some logging it looks like auth_signature is always empty and thus a 401 is thrown which is just being swallowed into a generic request failure without more details.

Looks like this is a known issue with the combination of Valet, Guzzle and curl https://github.com/laravel/valet/issues/736#issuecomment-511789470.

I had the same problem, websockets was not running in the server, run: php artisan websockets:serve


WIP: Starting to bump the KafkaChannel

Signed-off-by: Matthias Wessendorf<EMAIL_ADDRESS>Fixes #
Proposed Changes

bump types to v1beta1

Release Note

NONE (missing reconcilers, etc. just the types).

Docs

We also need the conversion funcs between v1alpha1 / v1beta1.
Something like this might help with that?
https://github.com/knative/eventing/pull/2621/files

yeah, I forgot the conversion, but will add ;-)
On Thu 11. Jun 2020 at 17:50, Ville Aikas<EMAIL_ADDRESS>wrote:

We also need the conversion funcs between v1alpha1 / v1beta1.
Something like this might help with that?
https://github.com/knative/eventing/pull/2621/files
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/knative/eventing-contrib/pull/1308#issuecomment-642761022,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABGPTUSYAHV7HVI7ULVOI3RWD4MHANCNFSM4NZXFTAQ
.

--
Sent from Gmail Mobile

/test pull-knative-eventing-contrib-unit-tests

/close
@matzew asked me to take over this work as he's gonna leave for PTO soon


Reorganise es funcs

This reorganises the ES functions used to make a search.
At a very high level it:

renames function names from _get_... to _build_...
merges some functions
renames variables and params to keep them consistent
adds some more tests

Codecov Report

Merging #905 into develop will decrease coverage by 0.06%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop     #905      +/-   ##
===========================================
- Coverage    96.66%   96.59%   -0.07%
===========================================
  Files          168      167       -1
  Lines         4884     4851      -33
  Branches       470      468       -2
===========================================
- Hits          4721     4686      -35
- Misses          99      103       +4
+ Partials        64       62       -2

Impacted Files
Coverage Δ

datahub/search/query_builder.py
98.08% <100%> (+0.57%)
:arrow_up:

datahub/core/fields.py
78.94% <0%> (-17.72%)
:arrow_down:

datahub/core/admin.py
82.6% <0%> (ø)
:arrow_up:

datahub/metadata/admin.py
100% <0%> (ø)
:arrow_up:

datahub/metadata/serializers.py
100% <0%> (ø)
:arrow_up:

datahub/metadata/views.py
100% <0%> (ø)
:arrow_up:

datahub/metadata/registry.py
100% <0%> (ø)
:arrow_up:

datahub/metadata/metadata.py
100% <0%> (ø)
:arrow_up:

datahub/metadata/filters.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b86771c...9a69075. Read the comment docs.

@reupen @elcct have another look.
I don't feel like I understand all the places where aggregations are used so I'd rather not touch split vs rsplit stuff. Probably one of you could take a better look at that.


Omar's Open Source Pull Request

Well done!  Your PR was merged- congrats!  Very excited that you found a project you were interested in and made a real contribution.


EnumMemeber containing comma in the value field is not deserialized correctly but is serialized correctly

Checked with following runtimes:
https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure/3.3.12
https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime/2.3.11
The enum is generated like this:
    [JsonConverter(typeof(StringEnumConverter))]
    public enum ResourceIdentityType
    {
        [EnumMember(Value = "SystemAssigned")]
        SystemAssigned,
        [EnumMember(Value = "UserAssigned")]
        UserAssigned,
        [EnumMember(Value = "SystemAssigned, UserAssigned")]
        SystemAssignedUserAssigned,
        [EnumMember(Value = "None")]
        None
    }

if the value SystemAssignedUserAssigned of the enum is serialized it has correct network payload value of SystemAssigned, UserAssigned but when the same value is received from server it is deserialized as UserAssigned .
I'm pretty sure this happens in the autogenerated word as well.
Repro steps in Fluent test:

Clone https://github.com/Azure/azure-libraries-for-net/tree/Fluent-v1.8  (tag is important)
Open in the root folder Fluent.Tests.sln
After build open \src\ResourceManagement\Compute\Generated\VirtualMachinesOperations.cs file and put a breakpoint at line 1907.
Open test explorer window and start debuging Fluent.Tests.Compute.VirtualMachine.EMSILMSIOperations.CanCreateWithLMSIAndEMSI test
When your breakpoint will be activated you will see

                _responseContent = await _httpResponse.Content.ReadAsStringAsync().ConfigureAwait(false);
                try
                {
                    _result.Body = Rest.Serialization.SafeJsonConvert.DeserializeObject<VirtualMachineInner>(_responseContent, Client.DeserializationSettings);
                }

that _responseContent contains
...
 "identity": {
    "type": "SystemAssigned, UserAssigned",
...

but _result.Body.Identity.Type will have value UserAssigned

Would you mind trying by editing the code and adding the [DataMember] attribute on the ResourceIdentityType and let me know if that makes it deserialize it appropriately.

Does not help.
    /// <summary>
    /// Defines values for ResourceIdentityType.
    /// </summary>
    [JsonConverter(typeof(StringEnumConverter))]
    [DataContract]
    public enum ResourceIdentityType
    {
        [EnumMember(Value = "SystemAssigned")]
        [DataMember]
        SystemAssigned,
        [EnumMember(Value = "UserAssigned")]
        [DataMember]
        UserAssigned,
        [EnumMember(Value = "SystemAssigned, UserAssigned")]
        [DataMember]
        SystemAssignedUserAssigned,
        [EnumMember(Value = "None")]
        [DataMember]
        None
    }

Same issue. BTW [DataMember]  is not applicable to the class or enum definition. Only to it's properties/members.

@hovsepm this bug is fixed in newtonsoft 10.0.3
So at this point .NET SDK targeting .NET 4.5.2 will not be fixed, but all SDKs that are targeting .NET Core can upgrade to newtonsoft 10.0.3 or above and this will not be an issue.
.Net SDK targeting .NET 4.5.2 cannot be upgraded due to powershell restriction and hence will have to wait until later part of year 2018


[libgnutls] Static use depends on -ldl, not carried in pc file

Describe the bug
While trying to update port libmicrohttpd, I stumbled over unresolved symbols when using static build of port libgnutls in config.log:
configure:24362: checking whether GnuTLS is usable
configure:24385: gcc -o conftest -I/home/kpa/Projekte/installed/x64-linux/debug/lib/pkgconfig/../../../include -fPIC -g -fno-strict-aliasing  -L/home/kpa/Projekte/installed/x64-linux/debug/lib  conftest.c  -L/home/kpa/Projekte/installed/x64-linux/debug/lib/pkgconfig/../../lib -lgnutls -pthread -ltspi -lgmp -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 >&5
/home/kpa/Projekte/installed/x64-linux/debug/lib/libgnutls.a(tpm.o): In function `check_init':
/home/kpa/Projekte/buildtrees/libgnutls/x64-linux-dbg/lib/.././../src/3.6.15-4050d06d46.clean/lib/tpm.c:123: undefined reference to `dlopen'
/home/kpa/Projekte/buildtrees/libgnutls/x64-linux-dbg/lib/.././../src/3.6.15-4050d06d46.clean/lib/tpm.c:129: undefined reference to `dlsym'
/home/kpa/Projekte/buildtrees/libgnutls/x64-linux-dbg/lib/.././../src/3.6.15-4050d06d46.clean/lib/tpm.c:129: undefined reference to `dlclose'
...
collect2: error: ld returned 1 exit status
...
configure:24397: result: no
configure:24407: WARNING: pkg-config reports that GnuTLS is present, but GnuTLS can't be used

I'm not sure if this is desired for the static build, or if it is just a bug in the .pc files.
Environment

OS: linux
Triplet: default (x64-linux)

To Reproduce

Add libgnutls dependency to libmicrohttpd
Add --with-https to OPTIONS for vcpkg_configure_make in libmicrohttpd portfile.
./vcpkg install libmicrohttpd
Inspect config.log

Expected behavior
No undefined references when consuming libgnutls via .pc files in a static build

I just tried the same under Ubuntu 18.04.5 LTS and getting:
configure:24362: checking whether GnuTLS is usable
configure:24385: gcc -o conftest -I/home/cromo/vcpkg/installed/x64-linux/debug/../include -fPIC -g -fno-strict-aliasing  -L/home/cromo/vcpkg/installed/x64-linux/d
ebug/lib  conftest.c  -L/home/cromo/vcpkg/installed/x64-linux/debug/lib -lgnutls -pthread -lgmp -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 >&5
configure:24385: $? = 0
configure:24387: result: yes
configure:24714: checking for gnutls_privkey_import_x509_raw()
configure:24743: gcc -o conftest -I/home/cromo/vcpkg/installed/x64-linux/debug/../include -fPIC -g -fno-strict-aliasing -I/home/cromo/vcpkg/installed/x64-linux/de
bug/../include  -L/home/cromo/vcpkg/installed/x64-linux/debug/lib -lgnutls -pthread -lgmp -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 -L/home/cromo/vcpkg/ins
talled/x64-linux/debug/lib  conftest.c  -L/home/cromo/vcpkg/installed/x64-linux/debug/lib -lgnutls -pthread -lgmp -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2
 >&5
configure:24743: $? = 0
configure:24750: result: yes

@wrobelda Briefly, do you want to say you cannot reproduce the issue?
The build of gnutls might have enabled additional features based on system introspection. This would still be a port bug. Maybe we should compare stdout of configure before entering config.log.

@dg0yt yes, precisely – I cannot reproduce that behavior. Attaching my gnutls bulld logs.
Archive.zip

@wrobelda Thanks for sharing the logs. I could spot a difference, and it is already present in your comment:
On my system, vcpkg port libgnutls carries -ltspi which is a system library discovered during libgnutls build.
So this a bug in port libgnutls. There is no such port in vcpkg. I will prepare a PR to build libgnutls --without-tpm.


Update to ungoogled-chromium 120.0.6099.129

No modification aside from the submodule bump is needed.

Builds and runs fine locally.

Is there anything that's holding this PR back? /cc @PF4Public

Sorry @Cubik65536 Busy times :)

aw damn, totally didn't have this on my radar anymore

aw damn, totally didn't have this on my radar anymore

Maybe we can have an auto PR reviewer request or something similar for notifications?

I am subscribed on the whole organization, that with other open source projects (mostly specific issues / prs on those cases) I work on or am subscribed to amounts to quite a lot of emails that I'm currently a bit behind on. (2.6k since 2023-11-01 in total)

It seems the build is failing
https://github.com/ungoogled-software/ungoogled-chromium-macos/actions/runs/7361151009/job/20038154725
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?

@Cubik65536  Wasn't this failure fixed recently?

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?

@Cubik65536 Wasn't this failure fixed recently?

It was.
The issue was fixed by reverting Python for our CI to 3.11, not sure if that got bumped up in some way to 3.12 again and caused the issue, maybe we can still try to revert it in some way.
I'm off for now and I'll (re)investigate on this probably 1 or 2 days later.

Just a quick update, I'm a bit busy these days and I will try to put up a fix by this weekend.

Just a quick update, I'm a bit busy these days and I will try to put up a fix by this weekend.

No worries, take your time, it is a volunteer work after all.

It seems the build is failing
ungoogled-software/ungoogled-chromium-macos/actions/runs/7361151009/job/20038154725
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?

#139 is up for a fix


RDNAM parsing is not robust enough

          I have the same issue as above. I am not sure what exactly is meant by adding '(NON-BEAUTIFIED)' to the prompt . Could someone please advise and may be update the repo?

`>>> from ragelo import get_retrieval_evaluator

evaluator = get_retrieval_evaluator("RDNAM", llm_provider="openai")
raw_answer, processed_answer = evaluator.evaluate(query="What is the capital of France?", document='Lyon is the second largest city in France.')
Failed to PARSE answer for qid: <no_qid> document id: {document.did}
Raw answer: {"O": 0}
Traceback (most recent call last):
File "", line 1, in
File "/home/Ubuntu/miniconda3/envs/ragelo/lib/python3.11/site-packages/ragelo/evaluators/retrieval_evaluators/base_retrieval_evaluator.py", line 200, in evaluate
raise ValueError(
ValueError: ('Failed to evaluate qid: <no_qid> did: <no_did>', 'Exception: Failed to parse answer: {{"O": 0}')
`

Originally posted by @fahdmirza in https://github.com/zetaalphavector/RAGElo/issues/30#issuecomment-2187909933

As for the quality of RDNAM, it is more of a "proof-of-concept" to see if we could handle that, as it was published. But, Ideally, we want to move to JSON mode on OpenAI when needed, specially with this use case.


sentry_sdk.init breaks import exceptiongroup in virtualenv activated with activate_this.py

How do you use Sentry?
Sentry Saas (sentry.io)
Version
2.2.1
Steps to Reproduce
$ docker run --rm -it ubuntu:22.04
root@e264f830878b:/# apt update
root@e264f830878b:/# apt install -y python3-apport virtualenv
root@e264f830878b:/# virtualenv venv
root@e264f830878b:/# venv/bin/pip install exceptiongroup sentry-sdk
…
Successfully installed certifi-2024.2.2 exceptiongroup-1.2.1 sentry-sdk-2.2.1 urllib3-2.2.1
root@e264f830878b:/# cat > test.py <<EOF
exec(open("venv/bin/activate_this.py").read(), {"__file__": "venv/bin/activate_this.py"})
import sentry_sdk
<EMAIL_ADDRESS>import exceptiongroup
EOF
root@e264f830878b:/# python3 test.py

Expected Result
No error.
Actual Result
Traceback (most recent call last):
  File "//test.py", line 4, in <module>
    import exceptiongroup
  File "/venv/lib/python3.10/site-packages/exceptiongroup/__init__.py", line 20, in <module>
    from ._formatting import (
  File "/venv/lib/python3.10/site-packages/exceptiongroup/_formatting.py", line 394, in <module>
    assert sys.excepthook is apport_python_hook.apport_excepthook
AssertionError
Sentry is attempting to send 2 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit

The relevant code within exceptiongroup is
if getattr(sys.excepthook, "__name__", None) in (
    "apport_excepthook",
    # on ubuntu 22.10 the hook was renamed to partial_apport_excepthook
    "partial_apport_excepthook",
):
    …
    import apport_python_hook

    assert sys.excepthook is apport_python_hook.apport_excepthook

which fails because Sentry has patched sys.excepthook but retained the same __name__, due to functools.wraps within the ensure_integration_enabled decorator, used by _make_excepthook as of

#2906

(cc @sentrivana)
This is arguably poor code within exceptiongroup (I opened agronholm/exceptiongroup#123), but Sentry should avoid breaking it since it’s a popular library; for example, it’s a dependency of IPython.

Hey @andersk
Thanks for reporting this and for the great reproduction!
Because of the great reproduction, it was easy to come of with a fix!

Released a fix for this in 2.3.1: https://github.com/getsentry/sentry-python/blob/master/CHANGELOG.md


This updates the changelog and has setup.py use it.

Before:
`pylibmc` is a Python client for `memcached <http://memcached.org/>`_ written in C.

See `the documentation at sendapatch.se/projects/pylibmc/`__ for more information.

__ http://sendapatch.se/projects/pylibmc/

.. image:: https://travis-ci.org/lericson/pylibmc.png?branch=master
   :target: https://travis-ci.org/lericson/pylibmc

New in version 1.4.0
====================

Brace yourself, Python 3.x support has come!

Thanks to everybody involved in this project; this release involves less
authors but **a lot** more work per person. Thanks especially to Harvey Falcic
for the work he put in, without which there wouldn't be any Python 3.x support.
Also thanks to Sergey Pashinin for the initial stab at the problem.

Other than that, we had miscellaneous bug fixes, testing improvements, and
documentation updates.

Last but not least I would like to ask for your support in this project, either
by helping out with development, testing, documentation or anything at all; or
simply by donating some `magic internet money`__ to the project's Bitcoin
address `12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h`__.

__ http://static.adzerk.net/Advertisers/5af77cf0094d4303bb308b955dd05992.jpg
__ bitcoin:12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h

Maintainer
==========

- Website: `sendapatch.se/ <http://sendapatch.se/>`_
- Github: `github.com/lericson <http://github.com/lericson>`_
- IRC: ``lericson`` in ``#sendapatch`` on ``chat.freenode.net``
- E-mail: ``ludvig`` at ``sendapatch.se``

------

.. image:: http://www.smbc-comics.com/comics/20110908.gif
   :target: http://www.smbc-comics.com/index.php?db=comics&id=2362
   :align: center

After:
`pylibmc` is a Python client for `memcached <http://memcached.org/>`_ written in C.

See `the documentation at sendapatch.se/projects/pylibmc/`__ for more information.

__ http://sendapatch.se/projects/pylibmc/

.. image:: https://travis-ci.org/lericson/pylibmc.png?branch=master
   :target: https://travis-ci.org/lericson/pylibmc

Maintainer
==========

- Website: `sendapatch.se/ <http://sendapatch.se/>`_
- Github: `github.com/lericson <http://github.com/lericson>`_
- IRC: ``lericson`` in ``#sendapatch`` on ``chat.freenode.net``
- E-mail: ``ludvig`` at ``sendapatch.se``

------

.. image:: http://www.smbc-comics.com/comics/20110908.gif
   :target: http://www.smbc-comics.com/index.php?db=comics&id=2362
   :align: center

New in version 1.4.3
--------------------
- Fixed several major memory leaks

New in version 1.4.0
--------------------

Brace yourself, Python 3.x support has come!

Thanks to everybody involved in this project; this release involves less
authors but **a lot** more work per person. Thanks especially to Harvey Falcic
for the work he put in, without which there wouldn't be any Python 3.x support.
Also thanks to Sergey Pashinin for the initial stab at the problem.

Other than that, we had miscellaneous bug fixes, testing improvements, and
documentation updates.

Last but not least I would like to ask for your support in this project, either
by helping out with development, testing, documentation or anything at all; or
simply by donating some `magic internet money`__ to the project's Bitcoin
address `12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h`__

__ http://static.adzerk.net/Advertisers/5af77cf0094d4303bb308b955dd05992.jpg
__ bitcoin:12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h

New in version 1.3.0
--------------------

Because there ain't nothing better than releasing software in the spring time.

Lots of improvements have come about in just about every corner of the library,
thanks to *eighteen different authors* over almost *20 pull requests*. Amazing.

- Added touch support
- Added compress_level
- Added weighted distribution
- Added automated benchmarking utility
- Added Travis CI and build status
- Added behavior dead_timeout
- Added behaviors tcp_keepalive
- Added behavior callback_prefix_key
- Lots of bug tuning, fixes, tests, and documentation

Lastly, thanks in particular to the authors of this release: Abramowitz,
Baklanov, Bergström, Borisov, Branson, Brown, Ericson, Hansche, Hlodversson,
King, Kowalak, McFague, Moura, Noguchi, Shurter, Williams and Wong.

@lericson We were on 1.4.1 because there wasn't a changelog that discussed how important upgrading to 1.4.3 was.  We've had scripts that restarted our applications for months because of memory leaks, so having 1.4.3 is HUGE!  I figure others might be interested in seeing the changelog updated as well.

Sounds like it warrants the release of pylibmc 1.5.0, to be honest! I'll get to releasing it in coming week.
-Ludvig

On 17 jun 2015, at 19:12, John Anderson<EMAIL_ADDRESS>wrote:
@lericson We were on 1.4.1 because there wasn't a changelog that discussed how important upgrading to 1.4.3 was. We've had scripts that restarted our applications for months because of memory leaks, so having 1.4.3 is HUGE! I figure others might be interested in seeing the changelog updated as well.
—
Reply to this email directly or view it on GitHub.

Alright, I released 1.5.0. :)


Can tsffs report the coverage rate of fuzz？

I tried the two demos you provided：

fuzz EDK2 UEFI Application

fuzz platform BIOS FD image

None of this gives me coverage information, so what do I need to do？

There is a script, scripts/cov.sh, that should produce a coverage report, but it wasn't working for me so I also am curious about generating coverage information.
[root@7cfda620aa4b example]# ./../../tsffs/scripts/cov.sh
/workspace/tsffs/scripts /workspace/projects/example
info: cargo-llvm-cov currently setting cfg(coverage); you can opt-out it by passing --no-cfg-coverage
I will run `rustup component add llvm-tools-preview --toolchain stable-x86_64-unknown-linux-gnu` to install the `llvm-tools-preview` component for the selected toolchain.
Proceed? [Y/n] y
info: downloading component 'llvm-tools'
info: installing component 'llvm-tools'
 29.6 MiB /  29.6 MiB (100 %)  25.4 MiB/s in  1s ETA:  0s
error: none of the selected packages contains these features: 6.0.169
error: process didn't exit successfully: `/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo test --tests --manifest-path /workspace/tsffs/Cargo.toml --target-dir /workspace/tsffs/target/llvm-cov-target --features=6.0.169` (exit status: 101)

cov.sh is just for test coverage of tsffs, so it won't give what you want. Tsffs logs to log.json new coverage/findings by edge, that combined with the timestamp is enough to get cov/time info, I can provide a simple script if desired.

I tested the log_level parameter as follows, but I still didn't see the cov info.
Is there a problem with the usage, or output the cov info at another location？
$ vim fuzz.simics
+    tsffs.log-level 2
or
+    @tsffs.log_level = 2

You're getting corpus entries so you should be seeing the output -- is there a log.json in your project directory? If not, you probably need to update your TSFFS build, this feature was added recently.


Support @default trait

Evolving Models
In theory, Smithy helps developers avoid Breaking changes.
It does this by publishing tooling and guides that dictate what changes are breaking vs those that are not.
In particular Update structures is of interest.

The following changes to structure shapes are backward-compatible:
…
2. Removing the required trait from a structure member and replacing it with the default trait (assuming the member was not marked with the clientOptional trait).

However, this is not possible for Smithy-Dafny consumers at this time,
because Smithy-Dafny does not support the default trait.
Example case
In MPL#594,
Crypto Tools removed the @required trait from a String, ddbTableName.
This passed all of the MPLs CI,
but once Crypto Tools started testing against the DB-ESDK,
Crypto Tools realized the Dafny type had subtly changed from string to Option<string>.
See DB-ESDK#1313.
This is only test code,
so Crypto Tools may not be blocked...
But Smithy cannot be fully utilized with out the @default trait.

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

In the case of this String,
rather than just removing @required,
Crypto Tools could replace @required with @default(""),
and then let the business logic treat "" as None.
The business logic already is required to assert 3 <= |ddbTableName| <= 255,
or something like that.
But the Dafny type would not change from string to Optional<string>.

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

This is likely a separate issue, but there is a similar complexity around @extendable rescues. How can you add new operations to such a shape?

Well, the breakage would have preventable IF and ONLY IF Smithy-Dafny respects @default...

The breakage would have been caught at the time of removing @required if smithy diff was run on the change.
Still being able to make the change anyway by also adding @default depends on smithy-dafny supporting it, true. But at least you would have known before changing MPL that you were going to break consumers.

Upgrading this to a soundness bug instead of a feature request, see #746


login interface

Please answer these questions before submitting your issue.

Why do you submit this issue?
Question

Question

What do you want to know?
Hi Mr.Wu , I was wondering if skywalking8.1.0 has a login interface . If there is , then which config file is to set up the user and password ? Thanks.

Requirement or improvement

I just upgrade the skw 6.1.0 to 8.1.0 and I can not find my log in interface . Neither could I set up in webapp.yml

No, there isn't. The old login is not real.

Use APISIX, Nginx or other proxy tech to set up a login page.

roger that


Fix go test never fails

@cnkarz go tests in kafkautils package were never run because you didn't write the TestMain properly.
You need to add this line at the end of TestMain function: os.Exit(m.Run())
After I've fixed TestMain function, go tests in kafkautils package failed. Can you fix them by adding new commit to this PR?

@cnkarz go tests in kafkautils package were never run because you didn't write the TestMain properly.
You need to add this line at the end of TestMain function: os.Exit(m.Run())
After I've fixed TestMain function, go tests in kafkautils package failed. Can you fix them by adding new commit to this PR?

why didn't you fix the tests?

why didn't you fix the tests?

I could fix them but I thought it would be better to let you fix them since you are the author.


only package if config allows

Overview
The package command should not do anything if the project has explicitly disabled packaging in config with package_lambda: false.

/do-e2e-tests


Syntax highlighting bug on equality operator

https://user-images.githubusercontent.com/16399020/105840491-113dbf00-600e-11eb-95ba-0e5a1427c14a.mp4

@Alhadis If you have free time

@Alhadis If you have free time


Trying to run frontend getting lot of errors

Could you please give a more detailed approach to run frontend, its giving a lot of errors? Getting a lot of errors regarding imports as well.

The fullstack version isn't working yet, only the standalone script. I've added a note in the readme.


Cut off Text

Sometimes, the items title is cut off like in this example:
https://goo.gl/photos/aQz2fQ5qHw7aq67S9
The code which has been used is exactly the same as in the previous issue with the missing ripple.

@Florianisme i will look in to that and before that can you give me your device specifications

I'm using the OnePlus One 64GB, DPI is set to default and no font settings have been changed

@Florianisme thank you i will take a look in to the issue.

@Florianisme fixed the issue today night update will include fixes to both your bugs.
Thanks for your support.

@Florianisme 1.2.2 is out with the fix.


Feat/change persistence of streams

Feature or Problem

cli toggle to modify the nats stream persistence between file and memory.
I do seek some guidance regarding testing here. Since we have multiple tests which have the file - default - mode enabled, should I create another suite or how do you think should I add tests for this?
Thank you
Related Issues

#479
Release Information

Consumer Impact

Testing

Unit Test(s)

Acceptance or Integration

Manual Verification

hey @markkovari thank you for this submission! Upon a first look, I only found a spelling issue that needs to be addressed. Otherwise, the changes made should do the trick.
testing wise, I believe we want the following

ensure that the flag (parsing) works... as in, when no specified -> expected default value & when specified -> maps to expected values (File, Memory)
ensure all the existing tests work as expected (no regression bugs)... this is basically just running the test suite locally
To check that we have a functional Memory mode, ensure that there is a file when using File mode and the absence of said file otherwise (ie, Memory mode). This entails figuring out how to find/peek/inspect a JetStream persistence file for a given stream.


Load large text file for LM pre-training resulting in OOM

I tried to pretrain Longformer using transformers and datasets. But I got OOM issues with loading a large text file. My script is almost like this:
from datasets import load_dataset

@dataclass
class DataCollatorForDatasetsLanguageModeling(DataCollatorForLanguageModeling):
    """
    Data collator used for language modeling based on DataCollatorForLazyLanguageModeling
    - collates batches of tensors, honoring their tokenizer's pad_token
    - preprocesses batches for masked language modeling
    """

    block_size: int = 512

    def __call__(self, examples: List[dict]) -> Dict[str, torch.Tensor]:
        examples = [example['text'] for example in examples]
        batch, attention_mask = self._tensorize_batch(examples)
        if self.mlm:
            inputs, labels = self.mask_tokens(batch)
            return {"input_ids": inputs, "labels": labels}
        else:
            labels = batch.clone().detach()
            if self.tokenizer.pad_token_id is not None:
                labels[labels == self.tokenizer.pad_token_id] = -100
            return {"input_ids": batch, "labels": labels}

    def _tensorize_batch(self, examples: List[str]) -> Tuple[torch.Tensor, torch.Tensor]:

        if self.tokenizer._pad_token is None:
            raise ValueError(
                "You are attempting to pad samples but the tokenizer you are using"
                f" ({self.tokenizer.__class__.__name__}) does not have one."
            )

        tensor_examples = self.tokenizer.batch_encode_plus(
            [ex for ex in examples if ex],
            max_length=self.block_size,
            return_tensors="pt",
            pad_to_max_length=True,
            return_attention_mask=True,
            truncation=True,
        )

        input_ids, attention_mask = tensor_examples["input_ids"], tensor_examples["attention_mask"]
        return input_ids, attention_mask

dataset = load_dataset('text', data_files='train.txt',cache_dir="./", , split='train')
data_collator = DataCollatorForDatasetsLanguageModeling(tokenizer=tokenizer, mlm=True,
                      mlm_probability=0.15, block_size=tokenizer.max_len)
trainer = Trainer(model=model, args=args, data_collator=data_collator,
                      train_dataset=train_dataset, prediction_loss_only=True, )
trainer.train(model_path=model_path)

This train.txt is about 1.1GB and has 90k lines where each line is a sequence of 4k words.
During training, the memory usage increased fast as the following graph and resulted in OOM before the finish of training.

Could you please give me any suggestions on why this happened and how to fix it?
Thanks.

Not sure what could cause that on the datasets side. Could this be a Trainer issue ? cc @julien-c @sgugger ?

There was a memory leak issue fixed recently in master. You should install from source and see if it fixes your problem.

@sgugger Thanks for your comments. I have install from source code as you told, but the problem is still there.
To reproduce the issue, just replace the  these lines with:
(load_dataset and DataCollatorForDatasetsLanguageModeling as above mentioned)
    dataset = load_dataset("bookcorpus")
    dataset = dataset.train_test_split(test_size=0.1)
    train_dataset = dataset['train']
    eval_dataset = dataset['test'] if training_args.do_eval else None

    data_collator = DataCollatorForDatasetsLanguageModeling(
        tokenizer=tokenizer,
        mlm=data_args.mlm,
        mlm_probability=data_args.mlm_probability,
        block_size=data_args.block_size

and run by:
python run_language_modeling.py
--output_dir=output \
--model_type=bert \
--model_name_or_path=bert-base-uncased \
--do_train \
--do_eval \
--mlm

Same here. Pre-training on wikitext-103 to do some test. At the end of the training it takes 32GB of RAM + ~30GB of SWAP. I installed dataset==1.1.0, not built from source. I will try uninstalling and building from source when it finish.

This seems to be on the transformers library side.
If you have more informations (pip env) or even better, a colab reproducing the error we can investigate.

It seems like it's solved with freshed versions of transformers. I have tried to replicate the error doing a fresh pip install transformers & datasets on colab and the error doesn't continue. On colab it keeps stable on 5GB! (Y)

@gaceladri witch version transformers and datasets  are you using now? I want to try again. Thanks.

transformers==3.3.1
datasets==1.1.0
tokenizers==0.8.1rc2

It does not happen to me anymore. Can we close? @leethu2012

It does not happen to me anymore. Can we close? @leethu2012

It's happening to me again. After 4 hours of pre-training, my ram memory gets full and the kernel dies. I am using the last transformers version as today. 4.4.0 and the last version of datasets 1.2.1, both installed from master. The memory consumption keeps increasing.

It looks like it is something from pytorch/python itself :face_with_head_bandage:  https://github.com/pytorch/pytorch/issues/13246

Thanks for the investigation @gaceladri
Apparently this happens when num_workers>0 and has to do with objects being copied-on-write.
Did you try setting num_workers to 0 @gaceladri ?
If the issue doesn't happen with num_workers=0 then this would confirm that it's indeed related to this python/pytorch issue.
Since a Dataset object is a wrapper of a pyarrow Table, we should investigate if the data being copied comes from the Table itself or from metadata in the Dataset object. If it comes from the metadata in the Dataset object, we should be able to implement a workaround. But if it comes from the Table, we'll need to see with the pyarrow team what we can do...

@lhoestq I have tried and it keeps increasing also with dataloader_num_workers=0

Hmmm so this might come from another issue...
Since it doesn't seem to be related to multiprocessing it should be easier to investigate though.
Do you have some ideas @gaceladri ?

I looked quickly to a previously spoted bug in my env wandb /sdk/interface/interface.py, because sometimes when I load the dataset I got a multiprocessing error at line 510 in wandb...interface.py
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<timed eval> in <module>

~/anaconda3/envs/tfm/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path, trial)
    877             print(len(epoch_iterator))
    878
--> 879             for step, inputs in enumerate(epoch_iterator):
    880
    881                 start_step = time.time()

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
   1083             else:
   1084                 del self._task_info[idx]
-> 1085                 return self._process_data(data)
   1086
   1087     def _try_put_index(self):

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1109         self._try_put_index()
   1110         if isinstance(data, ExceptionWrapper):
-> 1111             data.reraise()
   1112         return data
   1113

~/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--> 428         raise self.exc_type(msg)
    429
    430

AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1083, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1070, in _getitem
    format_kwargs=format_kwargs,
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 886, in _convert_outputs
    v = map_nested(command, v, **map_nested_kwargs)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 847, in command
    return torch.tensor(x, **format_kwargs)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 101, in _showwarnmsg
    _showwarnmsg_impl(msg)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/warnings.py", line 30, in _showwarnmsg_impl
    file.write(text)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/lib/redirect.py", line 100, in new_write
    cb(name, data)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 729, in _console_callback
    self._backend.interface.publish_output(name, data)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 186, in publish_output
    self._publish_output(o)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 191, in _publish_output
    self._publish(rec)
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/site-packages/wandb/sdk/interface/interface.py", line 510, in _publish
    if self._process and not self._process.is_alive():
  File "/home/ad/anaconda3/envs/tfm/lib/python3.6/multiprocessing/process.py", line 134, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

My workaround was to just comment those lines without looking to much into consecuences:
def _publish(self, record: pb.Record, local: bool = None) -> None:
        #if self._process and not self._process.is_alive():
        #    raise Exception("The wandb backend process has shutdown")

It worked so far... I need to try running without wandb and see if it could be causing something wrong with multiprocessing. I am going to try to launch the training setting wandb to false and I will let you know again.

@lhoestq But despite this, I got lost into the class Dataset() reading the pyarrow files.

Or maybe wandb uses multiprocessing ? One process for wandb logging and one for actual training ? If this is the case then even setting num_workers=0 would cause the process to be forked for wandb and therefore cause the memory issue.

@lhoestq could be, but if we set wandb to false this should not happen. I am going to try.

@lhoestq It keeps happening. I have uninstalled wandb from my env, setted %env WANDB_DISABLED=true on my notebook, and commented this func:
def get_available_reporting_integrations():
    integrations = []
    if is_azureml_available():
        integrations.append("azure_ml")
    if is_comet_available():
        integrations.append("comet_ml")
    if is_mlflow_available():
        integrations.append("mlflow")
    if is_tensorboard_available():
        integrations.append("tensorboard")
    # if is_wandb_available():
    #     integrations.append("wandb")
    return integrations

As a fast test and it keeps increasing the ram memory. Wandb could not be the blameworthy here.

Thanks for checking @gaceladri . Let's investigate the single process setting then.
If you have some sort of colab notebook with a minimal code example that shows this behavior feel free to share it @gaceladri so that we can play around with it to find what causes this. Otherwise I'll probably try to reproduce on my side at one point

@lhoestq sure. Here you have https://colab.research.google.com/drive/1ba09ZOpyHGAOQLcsxiQAHRXl10qnMU5o?usp=sharing let me know if the link works and it reproduces the issue. To me, it reproduces the issue, since if you start the training the ram memory keeps increasing.
Let me know. Thanks!

Could the bug be comming from tokenizers?
I got this warning at the terminal from my jupyter notebook:
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

I've never experienced memory issues with tokenizers so I don't know
Cc @n1t0 are you aware of any issue that would cause memory to keep increasing when the tokenizer is used in the Data Collator for language modeling ?

@lhoestq Thanks for pointing to n1t0, just to clarify. That warning was doing fine-tuning, without collator.

from datasets import load_dataset, load_metric
import numpy as np

GLUE_TASKS = [
    "cola",
    "mnli",
    "mnli-mm",
    "mrpc",
    "qnli",
    "qqp",
    "rte",
    "sst2",
    "stsb",
    "wnli",
]
task = "mnli"
actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset("glue", actual_task)
metric = load_metric("glue", actual_task)
batch_size = 16
attention_type = "linear"

from transformers.models.mobilebert_mod import (
    MobileBertForSequenceClassification,
    MobileBertTokenizerFast,
)
from transformers.models.mobilebert_mod.configuration_mobilebert import (
    MobileBertConfigMod,
)
from transformers import TrainingArguments, Trainer

num_labels = 3 if task.startswith("mnli") else 1 if task == "stsb" else 2
tokenizer = MobileBertTokenizerFast.from_pretrained(
    "/media/ad/00b5422b-9d54-4449-8b5d-08eab5cdac8c/training_trfm/big_linear_layerdrop_shared/checkpoint-23000/",
    max_len=512,
)
model = MobileBertForSequenceClassification.from_pretrained(
    "/media/ad/00b5422b-9d54-4449-8b5d-08eab5cdac8c/training_trfm/big_linear_layerdrop_shared/checkpoint-23000/",
    num_labels=num_labels,
)
print(model.num_parameters())

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

sentence1_key, sentence2_key = task_to_keys[task]
if sentence2_key is None:
    print(f"Sentence: {dataset['train'][0][sentence1_key]}")
else:
    print(f"Sentence 1: {dataset['train'][0][sentence1_key]}")
    print(f"Sentence 2: {dataset['train'][0][sentence2_key]}")

def preprocess_function(examples):
    if sentence2_key is None:
        return tokenizer(examples[sentence1_key], truncation=True)
    return tokenizer(examples[sentence1_key], examples[sentence2_key], truncation=True)

encoded_dataset = dataset.map(preprocess_function, batched=True)
metric_name = (
    "pearson"
    if task == "stsb"
    else "matthews_correlation"
    if task == "cola"
    else "accuracy"
)

args = TrainingArguments(
    f"test-glue/{task}_{attention_type}",
    evaluation_strategy="steps",
    learning_rate=1e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    logging_steps=200,
    num_train_epochs=5,
    gradient_accumulation_steps=1,
    warmup_steps=10000,
    fp16=True,
    dataloader_num_workers=10,
    weight_decay=0.1,
    load_best_model_at_end=True,
    metric_for_best_model=metric_name,
)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    if task != "stsb":
        predictions = np.argmax(predictions, axis=1)
    else:
        predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

validation_key = (
    "validation_mismatched"
    if task == "mnli-mm"
    else "validation_matched"
    if task == "mnli"
    else "validation"
)

trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset[validation_key],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()

Now, I have come back to pre-training. The changes that I think I have done are: not formatting the dataset to torch: big_dataset.set_format(type='torch', columns=["text", "input_ids", "attention_mask", "token_type_ids"]) so maybe some column is dropped and not freezed in memory and now I have not setted any validation dataset in the trainer.
My validation dataset before:
book_corpus_eval = load_dataset(
    "bookcorpus",
    "plain_text",
    cache_dir="/home/ad/Desktop/bookcorpus",
    split="train[98:99%]",
)
book_corpus_eval = book_corpus_eval.map(encode, batched=True)
book_corpus_eval.set_format(
    type="torch", columns=["text", "input_ids", "attention_mask", "token_type_ids"]
)
**book_corpus_eval = book_corpus_eval.select([i for i in range(1500)])**

Maybe selecting or indexing the dataset before feeding it to the trainer, do something strange.
My trainer now:

big_dataset = load_from_disk("/home/ad/Desktop/35percent_data.arrow/")

from transformers import DataCollatorForWholeWordMask

data_collator = DataCollatorForWholeWordMask(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./big_linear_layerdrop_shared_silu_secondtry",
    overwrite_output_dir=True,
    per_device_train_batch_size=60,
    per_device_eval_batch_size=60,
    save_steps=500,
    save_total_limit=10,
    logging_first_step=True,
    logging_steps=100,
#     evaluation_strategy='steps',
#     eval_steps=250,
    gradient_accumulation_steps=8,
    fp16=True,
    dataloader_num_workers=10,
    warmup_steps=15000,
    learning_rate=6e-4,
    adam_epsilon=1e-6,
    adam_beta2=0.98,
    weight_decay=0.01,
    max_grad_norm=1.0,
    max_steps=500000,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=big_dataset,
#     eval_dataset=book_corpus_eval,
    tokenizer=tokenizer)

import wandb
wandb.login()

trainer.train()

And surprisingly, the ram now keeps going up and down. The training is up now for 12h without collapse the ram. I don't know what could cause the leakage. :mag:

Thanks for sharing your results.
So you still had the issue for fine-tuning ?
And the issue still appears with a bare-bone dataset from an arrow file...

Yes, on both cases. Fine-tuning a pre-trained model and pre-training from scratch with a local arrow file already pre-processed.


Import teensy cores and libraries as submodules

This only affects building using the makefile. Using the Arduino IDE should still be supported, provided one has installed Teensyduino and the necessary libraries.
The motivation for this change is to be able to use a fork of the teensy3 cores that has more usb buffers, so as to support larger sliding window sizes. (See https://github.com/mitchgu/TAMProxy-pyHost/issues/24)
A nice side affect is that the build directory is now neater because all the extra libraries are in relative paths.
Unfortunately, eliminating the Teensyduino/Arduino IDE dependency altogether is still difficult because of Windows makefile support and the size, complexity, and platform dependency of the arm compilers.

I'm mainly waiting on actual testing with all the devices before I merge this.

Why did windows support get dropped? I could conceivably get mingw in a state to compile this with the old makefil

The code that was in there was pretty minimal and never worked in the first place, so I figured it'd be better to remove it and raise an error until someone (maybe you) figures out a proper way to do it.

Is this makefile your creation, or did it come from somewhere upstream?

It's derived from https://github.com/apmorton/teensy-template, which also did not support Windows


Add eslintrc generation

@btd
I don't think @sindresorhus is interested in this as this is probably supposed to be minimum. https://github.com/yeoman/generator-node/issues/66 They should be so maybe create a PR there?

Yep @stevemao i already understand it) I just locally use my patched version. I do not think they will add it also.

@btd Why? They mentioned eslint there. But I think it'll be lower priority but they will eventually support it.

@stevemao because of this cite:

jshint (we also have eslint on yeoman-generator, but we'll make a cut and probably keep only one)

Actually i think i will do own generator with some more options.

I'm planning to support ESLint. Just not ready to do it quite yet.


Cannot add a static ip from other resource group with managed identities

Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 0, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"LinkedAuthorizationFailed","message":"The client 'xxbeb' with object id 'xxbeb' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/virtualMachines/write' on scope '/subscriptions/xx8/resourceGroups/MC_resourcegroup/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-36869957-vmss/virtualMachines/0'; however, it does not have permission to perform action 'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions/xxx8/resourceGroups/resourcegroup/providers/Microsoft.Network/virtualNetworks/xx-vnet/subnets/xxvnetsubnet' or the linked scope(s) are invalid."}}
My vnet is in the same resource group as my k8 cluster with vnet
My static ip is in other resource group.
I have Network Contributor assigned to my static ip, not to my vnet. After adding Network Contributor to the vnet is works. I will once try a clean install. I believe this is not properly listed in the documentation...
`echo $NETWORK_RESOURCE_GROUP
CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show -g ${RESOURCE_GROUP} -n ${CLUSTER_MANAGED_IDENTITY_NAME} --query principalId -otsv)
echo $CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID
az role assignment create --assignee $CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID --role "Network Contributor" --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$NETWORK_RESOURCE_GROUP
export CLUSTER_MANAGED_IDENTITY_ID="$(az identity show -g ${RESOURCE_GROUP} -n ${CLUSTER_MANAGED_IDENTITY_NAME} --query id -otsv)"
echo $CLUSTER_MANAGED_IDENTITY_ID
VNET_ID=$(az network vnet show --resource-group ${RESOURCE_GROUP} --name ${CLUSTER_VNET} --query id -o tsv)
echo $VNET_ID
az role assignment create --role "Network Contributor" --assignee ${CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID} --scope $VNET_ID
az role assignment delete --role "Reader" --assignee ${CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID} --scope $VNET_ID
SUBNET_ID=$(az network vnet subnet show --resource-group ${RESOURCE_GROUP} --vnet-name $CLUSTER_VNET --name $CLUSTER_VNET_SUBNET_NAME --query id -o tsv)
echo $SUBNET_ID
az role assignment create --role "Network Contributor" --assignee ${CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID} --scope $VNET_ID `

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6a9c74a8-9f1a-3d3d-3aed-f8ec45f92e05
Version Independent ID: 01e98043-dc80-490e-a761-8d558ff4e6df
Content: Use managed identities in Azure Kubernetes Service - Azure Kubernetes Service
Content Source: articles/aks/use-managed-identity.md
Service: container-service
GitHub Login: @mlearned
Microsoft Alias: mlearned

@V4A001 Thanks for your comment! We will review and provide an update as appropriate.

@V4A001 I tried to do repro on my side but I was not able to reproduce the issue. As mentioned by you in the issue can you please confirm if after clean installation were you able to add static ip from other resource groups with managed identities.
Awaiting your inputs so I can further have discussion on this with Product Group to make necessary changes in documentation.
Thanks

My bad

I am able to  use another resource group;
`echo $NETWORK_RESOURCE_GROUP
CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show -g ${RESOURCE_GROUP} -n ${CLUSTER_MANAGED_IDENTITY_NAME} --query principalId -otsv)
echo $CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID
az role assignment create --assignee $CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID --role "Network Contributor" --scope /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$NETWORK_RESOURCE_GROUP
VNET_ID=$(az network vnet show --resource-group ${RESOURCE_GROUP} --name ${CLUSTER_VNET} --query id -o tsv)
echo $VNET_ID
az role assignment create --role "Network Contributor" --assignee ${CLUSTER_MANAGED_IDENTITY_PRINCIPAL_ID} --scope $VNET_ID `


CSharp - Path Parameters should keep their name and descriptions

Using the OpenAPI 3.0.0 petstore sample
kiota generate -d "https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v3.0/petstore.json" -l CSharp

The part of interest is
"paths": {
  "/pets/{petId}": {
    "get": {
      "parameters": [
        {
          "name": "petId",
          "in": "path",
          "required": true,
          "description": "The id of the pet to retrieve",
          "schema": {
            "type": "string"
          }
        }
      ]
    }
  }
}

Current Behavior
The following block of code is created for the indexer
/// Gets an item from the ApiSdk.pets.item collection
public WithPetItemRequestBuilder this[string position] { get {
    var urlTplParams = new Dictionary<string, object>(PathParameters);
    if (!string.IsNullOrWhiteSpace(position)) urlTplParams.Add("petId", position);
    return new WithPetItemRequestBuilder(urlTplParams, RequestAdapter);
} }

Expected Behavior
I would like for the position variable to be named the same as the path parameter petId. I would also like for path parameter's description to be kept and placed in a corresponding <param/> documentation.
/// Gets an item from the ApiSdk.pets.item collection
/// <param name="petId">The id of the pet to retrieve</param>
public WithPetItemRequestBuilder this[string petId] { get {
    var urlTplParams = new Dictionary<string, object>(PathParameters);
    if (!string.IsNullOrWhiteSpace(petId)) urlTplParams.Add("petId", petId);
    return new WithPetItemRequestBuilder(urlTplParams, RequestAdapter);
} }

Thanks for your interest in kiota and for reporting this.
This is something that could be easily fixed by adding a ParameterDescription property on the code indexer DOM element and populating that property.
https://github.com/microsoft/kiota/blob/9f3c55fca68a0ec94abab7edcde0cc45463a9c67/src/Kiota.Builder/KiotaBuilder.cs#L973
https://github.com/microsoft/kiota/blob/9f3c55fca68a0ec94abab7edcde0cc45463a9c67/src/Kiota.Builder/CodeDOM/CodeIndexer.cs#L4
Then we need to make sure we carry the information when we convert the indexer to a method in other languages
https://github.com/microsoft/kiota/blob/9f3c55fca68a0ec94abab7edcde0cc45463a9c67/src/Kiota.Builder/CodeDOM/CodeMethod.cs#L83
And lastly we need to make sure we write the information in dotnet (the other languages will rely on the code parameter information already)
https://github.com/microsoft/kiota/blob/9f3c55fca68a0ec94abab7edcde0cc45463a9c67/src/Kiota.Builder/Writers/CSharp/CodeIndexerWriter.cs#L15
Is this something you'd be willing to submit a pull request for?

@baywet Please review based on the 1.5 release improvements. Thanks!

The 1.5 improvements didn't impact this issue much: We now support type specific indexer parameters (not just string anymore, but also int etc...). #2594
We can't change the parameter name, it'd be a breaking change even though it's unlikely anybody has a direct reference to it. I created #3070 to follow up on this during v2.
We should still carry on the description of the parameter for a better developer experience. I'll work on that for the next release.


Support for returning multiple outputs

This PR makes it possible to return more than one output from Cog, without having to wrap the outputs in a zip file.
Python API changes
The predict() function treats dict outputs as multiple outputs. This is a change in behavior from before, when dicts were returned as json.
You can mix pathlib.Paths and other non-Path types. The following is a valid response:
def predict(self):
    return {
        "file-output1": Path(path1),
        "file-output2": Path(path2),
        "text-output": "foobar",
        "list-output": [1, 2, 3],
    }

Server changes
The built-in HTTP server returns multiple outputs as multipart/form-data.
The Redis queue worker returns multiple outputs as a new multi_values field in the response struct. The response struct now is:
type InferResponse struct {
	Error       string        `json:"error"`
	File        *InferFile    `json:"file"`
	Value       string        `json:"value"`
	MultiValues []*MultiValue `json:"multi_values"`
	Status      string        `json:"status"`
}

type MultiValue struct {
	Name  string     `json:"name"`
	File  *InferFile `json:"file"`
	Value string     `json:"value"`
}

type InferFile struct {
	URL  string `json:"url"`
	Name string `json:"name"`
}

UI changes
cog predict can write multiple outputs to stdout if the outputs are non-Path or textual Path outputs.
For example, given the following predict.py:
    @cog.input("input", type=str)
    def predict(self, input):
        out_dir = Path(tempfile.mkdtemp())
        out_path1 = out_dir / "first.txt"
        out_path2 = out_dir / "second.txt"
        with open(out_path1, "w") as f:
            f.write(f"one {input}")
        with open(out_path2, "w") as f:
            f.write(f"two {input}")

        return {"first": out_path1, "second": out_path2, "third": "just text"}

cog predict -i input=hello outputs:
first:
one hello
second:
two hello
third:
just text

Non-textual Path outputs are written to files named after the key in the response dict.
If the user passes the -o flag to cog predict, the outputs get named <output-name>.<dict-key>.<extension>.

Signed-off-by: andreasjansson<EMAIL_ADDRESS>

This is now being incorporated into #407.


NoRecordsContent is displayed in table before data is loaded

Bug type
Component
Component name
MudTable
What happened?
Content from NoRecordsContent render fragment is displayed incorrectly for a brief moment before data is loaded. This happens when the table first loads. It can be seen in the "Server Side Filtering, Sorting and Pagination" example from docs.
Expected behavior
I would expect LoadingContent to be displayed instead of NoRecordsContent.
Reproduction link
https://try.mudblazor.com/snippet/cEcQErPZGAtLygdP
Reproduction steps
Press run.
Relevant log output
No response
Version (bug)
6.0.11
Version (working)
No response
What browsers are you seeing the problem on?
Chrome
On what operating system are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

The expected behavior can be achieved by setting "Loading" parameter to true. I didn't realize that Loading defaults to false since there is loading animation during search even if Loading is not set explicitly. So my bad - this is not a bug. I just didn't read docs properly.
With that said, I think there should be consistency with application of Loading parameter. Loading animation (or lack there of) during initial load and data load triggered by paging or search should IMO all be affected by Loading parameter. In docs it says "Show a loading animation, if true."


Generate settings & alarm limits change events in backend simulator

This PR:
Updates protocol buffer message definitions:

Reorders fields so that HFNC-related fields come first
Updates the ventilation modes enum, including adding and removing some modes
Organizes the log event codes enum into sections, and adds values for control and alarm limits events
Defines a new log event type enum value for alarm limits
Adds more old/new fields for different data types into log events, instead of just having old_value and new_value, and reorders fields

Updates the frontend:

Reorders the cases of switch statements for ventilation mode, to match the order of values in the ventilation modes protobuf enum
Updates the modes page layout to show the new modes and to remove modes which will not be implemented (this change is untested, because I have no way of viewing the modes page)
In the Event Log, renames the Alarm column to the Event column; makes patient alarms be shown in red rather than green; changes the format of the date/time column to display the date first, since date has a higher priority than time in the column's sort order; adds support for alarm limits change events
In the Event Log, fixes capitalization for event labels and details; makes details strings more brief; adds missing units into details strings; adds a special detail string for ventilation started/stopped events; removes undesirable space between value and unit when the unit is a percentage; reorganizes event type definitions to match the organization of the log event codes enum
Fixes a misspelling of "Cannula"

Updates the backend:

Splits off events log management functionality from alarms management
Generates parameter change log events for FiO2 and Flow Rate
Generates alarm limits change log events for FiO2, SpO2, and Heart Rate
Adds backend simulation and alarms for heart rate
Disables many unit tests in backend/tests/unit/protocols/test_file.py because I didn't write those tests and I don't have instructions for how to update the test examples for the new protobuf field orderings. This will need to be fixed in a future PR, and it should be fixed before any changes are made to the filesystem-related protocols. (@Tanmay06 do you have any personal notes about how you generated these examples and/or how I should go about updating them? If not, @rohanpurohit this might be a new task for you, though lower priority than firmware-related tests)

@Sudhir-dev Can you try running the frontend with the backend simulator to see if you can catch any issues or suspicious behaviors which might've slipped past me? And can you also review the changes to the protobuf definitions and the events log-related frontend files, and raise any questions you might have?

For records-keeping:

This project is licensed under Apache License v2.0 for any software, and Solderpad Hardware License v2.1 for any hardware - do you agree that your contributions to this project will be under these licenses, too? Yes
Were any of these contributions also part of work you did for an employer or a client? No
Does this work include, or is it based on, any third-party work which you did not create? No


How to get the valule using ng-model?

Not able to get the selected date or time with ng-model

Hi! You can bind date or time value using date="dateobject" (or similarly time="timeobject") instead of using ng-model directive.
Let me know if that works for you.

Thanks a lot Gleb!!!!!
It worked for me...

Great, glad it worked out :)


Initial public version

initial clusterview plugin cleaned up for public release and for submission to the grafana community.

compatibility check looks to be failing due to a bug Grafana issue 380. I assume Grafana PR 443 will fix it (as soon as it is merged).


Is there a way to work with Terraform workspaces via Terratest?

Question:
In our Terraform scripts, we are using a concept of workspaces. For example, it's part of naming convention of resources:
resource "oci_core_instance" "CPServer" {
  display_name = "cpServer${count.index}-${terraform.workspace}"
  # other configuration
}

Is there a way to work with workspaces in the Terratest test? E.g. switch to the existing, or create the new one.
Is it even a valid use case?

I think that RunTerraformCommand should be fine for now. :+1:
Would be OK if I write some proposal/PR for those functions? Something like:

WorkspaceNew(t *testing.T, name string) string,
WorkspaceSelect(t *testing.T, name string) string,

or maybe just:

WorkspaceSelectOrNew(t *testing.T, name string) string.

I guess, they could be modeled like Destroy.

Sure, that would be great 👍

New function WorkspaceSelectOrNew satisfies the initial question.