Kubevirt vmi instance still running without operator

What happened:
A clear and concise description of what the bug is.
I cant delete kubevirt or vms at all the ns is stuck in terminating and finalizers did not work
What you expected to happen:
A clear and concise description of what you expected to happen.
Error from server (InternalError): error when replacing "/tmp/kubectl-edit-4069980125.yaml": Internal error occurred: failed calling webhook "virtualmachines-mutator.kubevirt.io": failed to call webhook: Post "https://virt-api.kubevirt.svc:443/virtualmachines-mutate?timeout=10s": Forbidden
How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.
k delete -f focal.yaml
k delete -f kubevirt-cr.yaml
k delete -f kubevirt-operator.yaml
k delete -f kubevirt-cr.yaml
Additional context:
Add any other context about the problem here.
Environment:

KubeVirt version (use virtctl version): N/A

Client Version: version.Info{GitVersion:"v1.3.1", GitCommit:"ed1e7ae8548d319fa7aacf315ad198f7241287c5", GitTreeState:"clean", BuildDate:"2024-08-22T08:52:25Z", GoVersion:"go1.22.2 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes version (use kubectl version): N/A

Client Version: v1.29.9
Kustomize Version: v5.0.4
Server Version: v1.29.9

VM or VMI specifications: N/A

Cloud provider or hardware configuration: N/A

OS (e.g. from /etc/os-release): N/A

focal ubuntu 20.04

Kernel (e.g. uname -a): N/A
5.15.0-124-generic

Install tools: N/A

Others: N/A

@adityakiwi Thanks for raising this bug. Did the dosubot recommendation help?


Update snapshot tests to integrate new custom-json-diff functionality.

Changes

I have ugraded the functionality in custom-json-diff to better handle bom diffing
Handles version issue
Produces an HTML report in addition to the JSON
Comparison is of defined bom objects which allows for better control for different use cases

Notes on the current configuration of the tests

Will pass if the newly generated bom has the same components or dependencies at equal or greater versions. This also means that purl and bom-ref will not be used in the comparison given they will not be the same in such a scenario.
Will pass if the snapshot bom has empty fields and the newly generated bom has populated these fields
Includes licenses, properties, and evidence in comparison but not hashes

@prabhu @setchy I have tested with the snapshots, updated the ones which required updating (properties expanded for a couple) and there's one thing I need to check on.
Change of scope from optional to required for bouncy castles dependencies for netty. The netty pom indicates these are optional. Thoughts?
{
  "bom-ref": "pkg:maven/org.bouncycastle/bcpkix-jdk15on@1.69?type=jar",
  "description": "The Bouncy Castle Java APIs for CMS, PKCS, EAC, TSP, CMP, CRMF, OCSP, and certificate generation. This jar contains APIs for JDK 1.5 and up. The APIs can be used in conjunction with a JCE/JCA provider such as the one provided with the Bouncy Castle Cryptography APIs.",
  "name": "bcpkix-jdk15on",
  "purl": "pkg:maven/org.bouncycastle/bcpkix-jdk15on@1.69?type=jar",
  "scope": "optional",
  "type": "framework",
  "version": "1.69"
}

{
  "bom-ref": "pkg:maven/org.bouncycastle/bcpkix-jdk15on@1.69?type=jar",
  "description": "The Bouncy Castle Java APIs for CMS, PKCS, EAC, TSP, CMP, CRMF, OCSP, and certificate generation. This jar contains APIs for JDK 1.5 and up. The APIs can be used in conjunction with a JCE/JCA provider such as the one provided with the Bouncy Castle Cryptography APIs.",
  "name": "bcpkix-jdk15on",
  "purl": "pkg:maven/org.bouncycastle/bcpkix-jdk15on@1.69?type=jar",
  "scope": "required",
  "type": "framework",
  "version": "1.69"
}

<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcpkix-jdk15on</artifactId>
  <version>1.69</version>
  <scope>compile</scope>
  <optional>true</optional>
</dependency>

<!--
  Completely optional and only needed for OCSP stapling to construct and
  parse OCSP requests and responses.
-->
<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcprov-jdk15on</artifactId>
  <version>1.69</version>
  <scope>compile</scope>
  <optional>true</optional>
</dependency>

<!--
  Completely optional and only needed for ALPN.
-->
<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bctls-jdk15on</artifactId>
  <version>1.69</version>
  <scope>compile</scope>
  <optional>true</optional>
</dependency>

Closes #1112

@cerrussell, Impressive work!

Change of scope from optional to required for bouncy castles dependencies for netty

I think the cyclonedx maven plugin currently over-reports required dependencies. I noticed this while trying the new maven deps tree approach using the environment variable PREFER_MAVEN_DEPS_TREE=true. Are you seeing the differences upon repeated runs for the same project?

@prabhu I don't have PREFER_MAVEN_DEPS_TREE=true but I get this for netty@7ad2b91515b3affaeadb4b2975cd6d2a8342c403

@cerrussell any updates? Did you get a chance to test the last two versions of the cyclonedx maven plugin directly?

@prabhu I ran the maven plugin directly and got the same thing. However, I did a bit of digging and I think what's actually happened is the devs at netty thought they had made these dependencies optional but because of the way one of their plugins worked, they were mandatory. This commit (Remove unwanted mandatory dependency in OSGi) details what happened and it makes sense why cyclonedx maven plugin changed what were intended to be optional dependencies to required... it is actually correct.
I'm therefore going to update the netty bom and then we should be good to go.

@prabhu @setchy ready for review


Mypy run reveals a bunch of type errors

Hi,
Cool project by the way.
I was messing around with my fork of your project (not many changes mostly just running black and a couple of tweaks), and the I ran mypy:
mypy src

and I got the following (I fixed a couple, so this list should be a little longer on your version). I definitely recommend checking this out since you need to have rigor in your type system and vs code doesn't pick up on these.
src/cfile/core.py:469: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
src/cfile/core.py:477: error: Item "Comment" of "Comment | Statement | Sequence" has no attribute "append"  [union-attr]
src/cfile/core.py:477: error: Item "Statement" of "Comment | Statement | Sequence" has no attribute "append"  [union-attr]
src/cfile/core.py:484: error: Item "Comment" of "Comment | Statement | Sequence" has no attribute "extend"  [union-attr]
src/cfile/core.py:484: error: Item "Statement" of "Comment | Statement | Sequence" has no attribute "extend"  [union-attr]
src/cfile/writer.py:34: error: Incompatible types in assignment (expression has type "None", variable has type "str")  [assignment]
src/cfile/writer.py:35: error: Incompatible types in assignment (expression has type "None", variable has type "TextIO")  [assignment]
src/cfile/writer.py:141: error: "TextIO" has no attribute "getvalue"  [attr-defined]
src/cfile/writer.py:148: error: "TextIO" has no attribute "getvalue"  [attr-defined]
src/cfile/writer.py:155: error: Cannot call function of unknown type  [operator]
src/cfile/writer.py:162: error: Item "Comment" of "Comment | Statement | Sequence" has no attribute "__iter__" (not iterable)  [union-attr]
src/cfile/writer.py:162: error: Item "Statement" of "Comment | Statement | Sequence" has no attribute "__iter__" (not iterable)  [union-attr]
src/cfile/writer.py:162: error: Item "Sequence" of "Comment | Statement | Sequence" has no attribute "__iter__" (not iterable)  [union-attr]
src/cfile/writer.py:189: error: Cannot call function of unknown type  [operator]
src/cfile/writer.py:212: error: Cannot call function of unknown type  [operator]
src/cfile/writer.py:317: error: Argument 1 to "_write_type" of "Writer" has incompatible type "Type | Struct"; expected "Type | StructRef"  [arg-type]
src/cfile/writer.py:341: error: Item "Struct" of "Type | Struct" has no attribute "pointer"  [union-attr]
src/cfile/writer.py:351: error: Item "Struct" of "Type | Struct" has no attribute "pointer"  [union-attr]
src/cfile/writer.py:437: error: Argument 1 to "len" has incompatible type "Comment | Statement | Sequence"; expected "Sized"  [arg-type]
src/cfile/factory.py:34: error: Incompatible return value type (got "Line", expected "Blank")  [return-value]
src/cfile/factory.py:99: error: Invalid type comment or annotation  [valid-type]
src/cfile/factory.py:99: note: Suggestion: use core.Block[...] instead of core.Block(...)
src/cfile/factory.py:114: error: Argument 2 to "Function" has incompatible type "str | Type | Struct | None"; expected "str | Type | None"  [arg-type]
src/cfile/factory.py:126: error: Argument 1 to "Type" has incompatible type "str | Type | Struct"; expected "str | Type"  [arg-type]
Found 22 errors in 3 files (checked 5 source files)

Thanks for the feedback, I will try to check against mypy in the future.
Currently I'm doing a major refactoring of the code due to a fundamental design issue I found a while back.
It's going to take some time to get it fixed.

Sounds good. Happy to provide feedback if required.

I fixed as much as I could. The remaining issue I believe is some bug in mypy. Perhaps I'll find a workaround for it later.

Sounds good!


Crash when editing third slot; inability to save url without setting interval and power

Trying to save data to the third slot (slot 2) sometimes (but not always!) crashes with the following error:
Process: no.nordicsemi.android.nrfbeacon.nearby, PID: 4368
 java.lang.IndexOutOfBoundsException: Invalid index 2, size is 1
     at java.util.ArrayList.throwIndexOutOfBoundsException(ArrayList.java:255)
     at java.util.ArrayList.set(ArrayList.java:481)
     at no.nordicsemi.android.nrfbeacon.nearby.update.UpdateFragment.configureUrlSlot(UpdateFragment.java:1555)
     at no.nordicsemi.android.nrfbeacon.nearby.update.ReadWriteAdvertisementSlotDialogFragment$2.onClick(ReadWriteAdvertisementSlotDialogFragment.java:285)
     at android.view.View.performClick(View.java:5204)
     at android.view.View$PerformClick.run(View.java:21153)
     at android.os.Handler.handleCallback(Handler.java:739)
     at android.os.Handler.dispatchMessage(Handler.java:95)
     at android.os.Looper.loop(Looper.java:148)
     at android.app.ActivityThread.main(ActivityThread.java:5417)
     at java.lang.reflect.Method.invoke(Native Method)
     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

In addition, some beacons come with both interval and power set to zero in the second and third slots - currently, it is not possible to configure a URL in these slots before setting the interval and power (pressing configure has no effect - if this is a technical limitation, could it instead show a hint?).

Hi Simon,
I have never come across this scenario but configuring URL has been working
without any issues with the current version. I can look this up on Monday
and get back.
On 7 Oct 2016 5:54 pm, "Simon Robinson"<EMAIL_ADDRESS>wrote:
Trying to save data to the third slot (slot 2) sometimes (but not always!)
crashes with the following error:
Process: no.nordicsemi.android.nrfbeacon.nearby, PID: 4368
java.lang.IndexOutOfBoundsException: Invalid index 2, size is 1
at java.util.ArrayList.throwIndexOutOfBoundsException(ArrayList.java:255)
at java.util.ArrayList.set(ArrayList.java:481)
at no.nordicsemi.android.nrfbeacon.nearby.update.UpdateFragment.configureUrlSlot(UpdateFragment.java:1555)
at no.nordicsemi.android.nrfbeacon.nearby.update.ReadWriteAdvertisementSlotDialogFragment$2.onClick(ReadWriteAdvertisementSlotDialogFragment.java:285)
at android.view.View.performClick(View.java:5204)
at android.view.View$PerformClick.run(View.java:21153)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:148)
at android.app.ActivityThread.main(ActivityThread.java:5417)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
In addition, some beacons come with both interval and power set to zero in
the second and third slots - currently, it is not possible to configure a
URL in these slots before setting the interval and power (pressing
configure has no effect - if this is a technical limitation, could it
instead show a hint?).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NordicSemiconductor/Android-nRF-Beacon-for-Eddystone/issues/6,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AISvDbtWmL5UcHE4sbRkayl1I8a7qNCYks5qxms1gaJpZM4KRL79
.

@simonrob are you able to reproduce this error? I'm unable to reproduce it on my end. If possible could you please send me the steps to reproduce? Thanks

@roshanrajaratnam - yes, but on closer inspection it may be related to the prefix selector rather than the configuration of the third slot. It also happens for slot 1 as well as slot 2.
The simplest way I can find to reproduce it is to connect to a beacon that already has three slots configured as URLs, switch to slot 2, then use the prefix selector to change from, e.g., https://www. to https:// - pressing configure causes this crash every time for me.
Let me know if you need any more information.

Great thanks for the input, i've tried the same steps as you have mentioned, but I am still unable to reproduce the error on my end. Are you using the latest version of the app (V2.2)?

Yes - it's the latest version of the app. And you're right about the prefix (sometimes changing the URL text crashes the app too). The key step seems to be changing slots then trying to update the URL.
Could it be beacon related somehow? (I'm using one of these: https://fccid.io/pdf.php?id=2968851)

That would be my guess as well because the steps don't seem to reproduce for me. I wonder if this beacon supports the eddystone firmware? Have you flashed the beacon with the Eddystone firmware?

It's one of the Eddystone beacons given out by the Physical Web team so I'd have thought it would already have that firmware (when the app doesn't crash, the beacon does function normally with all slots and data types etc).
Happy to try updating the firmware if you can point me towards where to get the firmware and basic instructions though.

Right. Perhaps you could also try viewing the broadcast capabilities of this beacon on the update tab and send a screenshot please? The app should ideally work with the standard Eddystone Gatt Configuration Service Spec.

Here's the broadcast capabilities page. Interestingly, the slot information (also below) shows that only slot 0 is configured - this is not the case (all three slots are configured and broadcasting). On a possibly related note, I haven't ever seen the beacon in the nRF app's beacons tab, though it does show up in Android's Nearby discoveries app.

This is interesting. I think this seems to be something to do with the beacon firmware itself. All these features are fully functional on my end. However the beacons tab will only display, UID and EID beacons that are registered on the proximity API. In addition , for this to work you will have to compile the source code with your own API key by enabling the nearby api. The reason being you need write permissions to register the beacons hence it will not show up on your beacons tab.

Ah, ok - looks like it might be best for me to wait for things to become a little more established/settled then. Feel free to close this issue - the problem is only intermittent, and seems to be beacon rather than app-related. Thanks for the help!

No problem! Please do let me know if you need any more information. I will keep you posted for new updates on our end!


Deadlock when accessing constant data through misaligned pointer generated at compile time

Hello,
Thanks for your awesome platform integration of the RP2040 core earlephillower (until now that I discovered it I was simply building & uploading through the Arduino CLI and editing code + debugging with custom configuration in VS Code x)).
I have an issue though I noticed with a small pet project I started sunday, freshly installing everything (so it's latest version of PlatformIO + your platform + earlephillower core & C++ compilers on Windows).
I'm not sure if I was only lucky until now and it's just the way I handle constant values that is wrong, but I have an issue now where randomly the compiler emits pointers on constant values with the LSB set to 1 instead of 0 (causing the program to crash).
To illustrate simply, I have in a .h file something like this (the content doesn't matter from what I noticed):
static const unsigned char myBigConstantAsset[] = { /* generatedBytes */ };

Then I later retrieve a pointer on it to feed it to some library or do some processing, for example just doing this:
const uint16_t * data = (const uint16_t *) myBigConstantAsset; // pointer on 16-bit 565 RGB image data for example
uint16_t color = data[0]; // Can crash immediately here, using the pointer without doing anything in-between that could corrupt or overwrite it

Will work if data has LSB to 0 (which is normal) but crash if the LSB is randomly 1 (which is wrong for data access, I remember a long time ago doing some trickery to manually set the LSB to 1 on an ARM application to get a function pointer - does that mean by the way the compiler sometime treats my data pointer as a function pointer ?!).
I use this configuration in my platform.ini file that should be fairly basic:
[env]
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
framework = arduino
board_build.core = earlephilhower
board_build.filesystem_size = 0m
board_build.mcu = rp2040
board_build.f_cpu = 150000000L
lib_deps =
	SPI
<EMAIL_ADDRESS>build_flags =
	-O2
  	-fpermissive
	-DUSER_SETUP_LOADED=1
	-DST7735_DRIVER=1
	-DST7735_GREENTAB3=1
	-DTFT_RGB_ORDER=TFT_BGR
	-DTFT_WIDTH=128
	-DTFT_HEIGHT=128
	-DRP2040_PIO_SPI=1
	-DRP2040_PIO_CLK_DIV=1
	-DTFT_DC=3
	-DTFT_SCLK=4
	-DTFT_MOSI=5
	-DTFT_MISO=-1
	-DTFT_CS=7
	-DTFT_RST=8
	-DTFT_BL=9
	-DTFT_BACKLIGHT_ON=1

[env:pico]
board = pico

I'm confused because I used several times embedded constant values (I also use a lot of contexpr computed values by the way) instead of emulated FS (I know I'm lazy on this one) and it's the first time I have this issue :/
And it's not super easy to reproduce to compare with other platforms as just cleaning sometime or anything that change the emitted binary also change the emitted address of the pointers and can produce them as correctly usable.
For now I settled with this workaround to always remove the LSB if it's set to 1:
const uint16_t * myPointer = (const uint16_t *)((size_t) sheet.data & 0xFFFFFFFE) // Casting pointer to size_t + applying binary mask to remove LSB and cast back to original type

Which seems to work since now my code runs without anymore issue even when I still saw the original address having the LSB set to 1
I'll still try to reproduce a custom configuration on the Arduino IDE with a custom board definition reusing the core of earlephillower and as much as possible the same compilation options to see if I'm able to also reproduce the issue.

Workaround 2: simply using alignas(sizeof(void *)) (depending on the context such as DMA, other types of aligment might be more suitable) so I'm simply doing my own PROGMEM definition

const uint16_t * data = (const uint16_t *) myBigConstantAsset; // pointer on 16-bit 565 RGB image data for example
uint16_t color = data[0]; // Can crash immediately here, using the pointer without doing anything in-between that could corrupt or overwrite it

Pretty sure the Cortex-M0+ can't handle unaligned access. Accessing a uint16_t (2 byte value) needs a 2-byte alignment. Adding that to the myBigConstantAsset definition should fix the issue as you already confirmed.

const uint16_t * data = (const uint16_t *) myBigConstantAsset; // pointer on 16-bit 565 RGB image data for example
uint16_t color = data[0]; // Can crash immediately here, using the pointer without doing anything in-between that could corrupt or overwrite it

Pretty sure the Cortex-M0+ can't handle unaligned access. Accessing a uint16_t (2 byte value) needs a 2-byte alignment. Adding that to the myBigConstantAsset definition should fix the issue as you already confirmed.

Exact, it can't.
Still it's pretty strange that GCC emitted this code, something must have


The Project Configurator

The Project Configurator is an executable bash script in the root of the repository named config which prepares the environment for Release Manager to be installed. It:

[x] Checks for required packages an installs those not present
[x] Exports essential PATH information to the environment and ensures persistence
[x] Creates all necessary directories
[x] Acts as a pivot-point from which further actions can be launched from the command-line:

[x] Install RM - utilizes Installer
[x] Update RM - utilizes Installer
[x] Uninstall RM - utilizes Installer

Complete.  Shouldn't need anything for quite some time.


Provides a list of power toy shortcut keys on one page

Description of the new feature / enhancement
make people Find the functions you want to use more quickly
Scenario when this would be used?
It is very time-consuming to search for the desired function activation method page by page.
Supporting information
No response

Hello and thanks for your message. I have an idea for this in #24880, what do you think about this?
CC @crutkas @niels9001 I lost count of the number of requests for this 😆

This is a duplicate for a couple of different issues. The new dashboard, as well as the shortcut guide #15405.

/dup #26816


Add GHA to lint TypeScript code

Proposed changes
What changed
npm linting tasks are not being run by the gradle build system. While we have this combined build system, we'll need to run some TypeScript specific actions separatgely.
This PR can be used as a basis for recording Sonarcloud results using the Monorepo approach
Note: This PR will be marked as failing, until any linting issues are corrected in the main branch

Why did it change

Issue tracking

OJ-XXXX

Checklists
Environment variables or secrets

[ ] No environment variables or secrets were added or changed

[ ] Documented in the README
[ ] Added to deployment repository
[ ] Added to local startup repository

Other considerations

[ ] Update README with any new instructions or tasks

Closing in favour of #132


[Feature]: chezmoi cd like command -> vfox cd

1. Your usage scenarios?
Let users know where the SDK is installed.
2. What is your expected outcome?
case 1:
vfox cd
# Equivalent to
cd $HOME/.version-fox

case 2:
vfox cd nodejs
# Equivalent to
cd $HOME/.version-fox/cache/nodejs/<current-used-version-path>

reference: https://www.chezmoi.io/reference/commands/cd/

This should only output the SDK installation path, there is no way to switch the current directory automatically, right?
If so, I think which or where command might be better.

This should only output the SDK installation path, there is no way to switch the current directory automatically, right?
If so, I think which or where command might be better.

In my scenario, I mainly want to modify the configuration file in VFOX_HOME($HOME/.version-fox). I don't want to remember the management directory path of vofx every time and then cd into it. If I change the VFOX_HOME path, the workflow will become more complex.

This should only output the SDK installation path, there is no way to switch the current directory automatically, right?
If so, I think which or where command might be better.

In my scenario, I mainly want to modify the configuration file in VFOX_HOME($HOME/.version-fox). I don't want to remember the management directory path of vofx every time and then cd into it. If I change the VFOX_HOME path, the workflow will become more complex.

Ok, I got it, I don't object to this feature, you can go ahead and implement it ;)


Move Planner

Would probably require a good amount of work in order to allow this to happen.

Yes, planner is a very useful feature. It seems that planner on the web is occasionally buggy, it would be great if planner could be applied on the Player. Looking forward to it!


Consider moving Section's methods into a trait

Here is what I am imagining:

Move all of Section's methods to a trait SectionMethods
Make an impl SectionMethods for Section { ... }

This will make it nicer to mixin new methods into Section, similar to the way @JimBlandy used inheritence and subclassing in the breakpad tests, eg http://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/dwarf/dwarf2reader_cfi_unittest.cc#271
Once this is done, one could add new methods like so:
trait CfiMethods: SectionMethods {
    fn cie(self, ...) -> Self {
        // Default trait method implementation here
    }

    fn fde(self, ...) -> Self {
        // Default trait method implementation here
    }

    // Etc...
}

// Nothing else needed because we have defaults for the trait methods
impl CfiMethods for Section {}

The default trait method implementations can use the SectionMethods methods
Returning Self allows returning the actual concrete "subclass" type, rather than the "base" type so it allows chaining even with "subclasses" (which is an annoyance that @JimBlandy had with breakpad + subclassing)
Would it be better to define a CfiSection and impl SectionMethods and impl CfiMethods for CfiSection? Wouldn't that provide better namespacing? No, we only get trait methods if they are being use Traited, so if you don't want them, then don't use Trait in that module. Also, what I proposed here allows reuse of the original SectionMethods implementation instead of requiring all new types to also implement that trait.

So here are all the Section subclasses I could find in the Breakpad source:
https://chromium.googlesource.com/breakpad/breakpad/+/master/src/common/mac/macho_reader_unittest.cc#501
https://chromium.googlesource.com/breakpad/breakpad/+/master/src/processor/synth_minidump.h
https://chromium.googlesource.com/breakpad/breakpad/+/master/src/common/linux/synth_elf.h
https://chromium.googlesource.com/breakpad/breakpad/+/master/src/common/dwarf/dwarf2reader_test_common.h
One thing that I think won't work with your proposal is that many of these subclasses have their own internal state. Additionally, it seems like we'd actually lose some type safety vs. the C++ version, since many of the subclasses have methods for appending other Section subclasses with special handling (like the stuff in synth_minidump.h).

Yes, it doesn't assume any extra state, just a collection of helper methods to make writing certain forms easier.
I'm not sure how to handle those things...

I mean, if everything is a trait, one could always re-implement those traits for their own types, but that is the kind of boilerplate I was hoping to avoid...

We discussed on IRC maybe adding something so you could do
struct MySection {
  section: Section,
}

impl_section_for!(MySection, section);

...but I realize that given the existing Section API which takes ownership of self in all the methods and returns ownership is not really conducive to doing this.

So it struck me that the C++ examples there have two distinct use cases, they're just conflated by the inheritance. Some of those are "we want new convenience methods on Section" and some of those are "we want to encapsulate a Section as a new type and late be appended to another Section." I think we can support both of those cases without significant changes.
First of all, I realized I had not implemented Section::Append(const Section&) from the C++ version, so I implemented Section::append_section: https://github.com/luser/rust-test-assembler/commit/2d9cfdd178a770bdd08cb3d98bb0150a729ff2b4 . It takes an Into<Section>, so you should be able to write things in the latter class simply by encapsulating a Section and then implementing From<T> for your type, like:
struct MyThing { section: Section }

impl From<MyThing> for Section {
  fn from(t: MyThing) -> Section {
    let MyThing { section, .. } = t;
    section
  }
}

fn do_something() {
  let mut s = Section::new();
  let t = MyThing::new();
  // do something with t
  s = s.append_section(t);
  // do something with s
}

It might be a pain to use a Section as a member because of the ownership, but if MyThing is going to implement its API methods the same way to allow chaining it might not be too bad, something like:
impl MyThing {
  fn do_a_thing(self, x: u32) -> MyThing {
    let MyThing { mut section, .. } = self;
    section = section.L32(x);
    MyThing { section: section }
  }
}

For the other case (just adding utility methods to Section, like in your initial comment), I think that's already handled very nicely by traits. I wrote a test just to prove to myself that it would work well: https://github.com/luser/rust-test-assembler/commit/ed7acbf58a0d3385f5a82bb986b1293163b751b1
Does that all sound reasonable to you? I suppose we'll have to try implementing some real-world uses of these to find out if the API is suitable.

I started a port of SynthMinidump on my flights to and from Portland, and it seems to be pretty workable with the current API: https://github.com/luser/rust-minidump/blob/master/src/synth_minidump.rs
Given that the "additional helper methods" use case is handled well by traits, and that the "encapsulate Section" use case seems to be workable, I'm going to claim that we don't need to significantly change the API right now.

This conversation brought up a bunch of angles that hadn't occurred to me, but I still think extension traits aren't quite the right thing. If I wanted to write tests for, say, a TrueType parser, I could certainly write an extension trait that adds methods for building various kinds of TrueType tables and so on to a Section. But then those methods would be callable on any Section at all, even those representing other sorts of files. I'd like to have types TrueTypeFile, TrueTypeCollectionFile, and so on, each of which only lets me append things that belong in it.
Would it work to have the dumb D32 methods in a trait, with a default impl for types that implement AsMut<Section>? Or perhaps a custom AsSection trait, to make the opt-in explicit? Would that run afoul of trait coherence rules?


Needs updating

here is the new way to add loops:
https://powerusers.microsoft.com/t5/General-Power-Automate/Add-a-Condition-has-moved-in-Flow/td-p/165401

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 39d79925-8b72-d4b0-e3fb-1b9164ecaaf3
Version Independent ID: 06c8759f-1adf-470f-bdba-1f3af6f0a523
Content: Use the apply to each action to loop through an array of items. - Power Automate
Content Source: articles/apply-to-each.md
Service: flow
GitHub Login: @MSFTMan
Microsoft Alias: deonhe

@goyaweb Thanks for reporting this issue. We are updating this document and the updates will be live soon.


HTML5 Push Notifications Broken in 0.87.1

Home Assistant release with the issue:0.87.1
Last working Home Assistant release (if known):0.87.0
Operating environment (Hass.io/Docker/Windows/etc.):Docker
Please provide details about your environment.
Component/platform:HTML5 Push Notifications
Please add the link to the documentation at https://www.home-assistant.io/components/ of the component/platform in question.
https://www.home-assistant.io/components/notify.html5/
Following upgrade to 0.87.1 HTML5 notification stopped working. Test calling from dev tools generates "failed to call........"
See associated log:
2019-02-12 13:20:22 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.22893546415328] Error handling message: {'type': 'call_service', 'domain': 'notify', 'service': 'html5_juan_samsung_phone', 'service_data': {'message': 'test'}, 'id': 119}
 Traceback (most recent call last):
 File "/usr/src/app/homeassistant/components/websocket_api/decorators.py", line 17, in _handle_async_response await func(hass, connection, msg)
 File "/usr/src/app/homeassistant/components/websocket_api/commands.py", line 148, in handle_call_service connection.context(msg))
 File "/usr/src/app/homeassistant/core.py", line 1130, in async_call self._execute_service(handler, service_call))
 File "/usr/src/app/homeassistant/core.py", line 1152, in _execute_service await handler.func(service_call)
 File "/usr/src/app/homeassistant/components/notify/__init__.py", line 117, in async_notify_message await notify_service.async_send_message(**kwargs)
 File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs)
 File "/usr/src/app/homeassistant/components/notify/html5.py", line 451, in send_message self._push_message(payload, **kwargs)
 File "/usr/src/app/homeassistant/components/notify/html5.py", line 498, in _push_message json.dumps(payload), gcm_key=gcm_key, ttl='86400'
 File "/usr/local/lib/python3.6/site-packages/pywebpush/__init__.py", line 256, in send encoded = self.encode(data)
 File "/usr/local/lib/python3.6/site-packages/pywebpush/__init__.py", line 184, in encode version=content_encoding)
 File "/usr/local/lib/python3.6/site-packages/http_ece/__init__.py", line 375, in encrypt keyid=keyid, keylabel=keylabel)
 File "/usr/local/lib/python3.6/site-packages/http_ece/__init__.py", line 113, in derive_key keylabel=keylabel)
 File "/usr/local/lib/python3.6/site-packages/http_ece/__init__.py", line 78, in derive_dh pubkey = ec.EllipticCurvePublicKey.from_encoded_point(
 AttributeError: type object 'EllipticCurvePublicKey' has no attribute 'from_encoded_point'

Might be caused by web-push-libs/pywebpush#101

HA pinned cryptography==2.3.1 this from_encoded_point method moved to EllipticCurvePublicKey class at cryptography>=2.5

Please try to manual upgrade cryptography via pip install cryptography==2.5

That fixes the issue for me (same environment as @juan11perez )

I confirm manual upgrade cryptography via pip install cryptography==2.5 also fixes it for me.

It'd be great to have a fix for this in the "latest" Docker image, though. "rc" seems to already have moved on.

It'd be great to have a fix for this in the "latest" Docker image, though. "rc" seems to already have moved on.

You can fix the image yourself:
docker exec -i -t <home assistant container name> /bin/bash
pip install cryptography==2.5


can anybody help me?

Could someone help me have tecogan please? I don't know what to do. Where do I begin?

You could follow the installation instructions of my TecoGAN Docker fork.
Using this setup should make it much easier to get started.
https://github.com/tom-doerr/TecoGAN

@tom-doerr thanks

@tom-doerr i use windows

Then you might want to look into Windows Subsystem for Linux

@tom-doerr I think I have managed to do that part, now in this command "docker run --gpus all -it --mount src = $ ( pwd ) , target = / TecoGAN, type = bind -w / TecoGAN tecogan_image bash" gives me this error "invalid argument "src" for "--mount" flag: invalid field 'src' must be a key=value pair
See 'docker run --help'."

@tom-doerr I think I have managed to do that part, now in this command "docker run --gpus all -it --mount src = $ ( pwd ) , target = / TecoGAN, type = bind -w / TecoGAN tecogan_image bash" gives me this error "invalid argument "src" for "--mount" flag: invalid field 'src' must be a key=value pair
See 'docker run --help'."

Is it possible that you are running that command in the normal Windows terminal instead of in the Windows Subsystem for Linux?

You could follow the installation instructions of my TecoGAN Docker fork.
Using this setup should make it much easier to get started.
https://github.com/tom-doerr/TecoGAN

Thanks for all


Redshift query step with csv serializer

First reusable step for making SQL query to redshift and outputting CSV file as a datum (optionally datum alias).
Logs the query and a truncated preview of the results.

Note: I wasn't able to test IMDSv2, so that part is written on faith only.
I need some help setting up staging workers so that they can do it as I believe they need some extra configuration (?)


fix(FormattingToolbar): fixes change of viewport for link modal - I376

Signed-off-by: elit-altum<EMAIL_ADDRESS>Issue #376

Changes

Added the window.scroll() method to scroll to the HyperlinkModal's viewport
Also changed the HyperLinkWrapper and HyperLinkCaret styles to tab of 2.

Flags

Was the indentation of the fields intentionally of 4 tab spaces? If yes, I will revert it!

@irmerk @DianaLease Please review!

A couple issues I spot on first pass:

The modal (and it's caret) are off center
The width is off

(1)

(2)

@irmerk Sorry I didn't see that, my bad. Just fixed 1.
Can you tell me how to recreate the second instance ? I am not able to do that.

Ah the second one isn't an issue anymore. A new issue I'm finding where the screen shoots down when clicking a link:

Ah the second one isn't an issue anymore. A new issue I'm finding where the screen shoots down when clicking a link:

@irmerk If you press Enter near the end of the page on any text. The page scrolls down to that viewport and brings it to center of the screen is that expected behavior ?

@irmerk One more thing, right now the z-index is such that the link modal is on top of the formatting toolbar icons. Is that expected behavior or should I fix it?

@Michael-Grover do you have any input on the last two comments from @elit-altum?

It looks like on Google docs, the screen never moves when the link modal pops up, even if the link is near the bottom of the page. I think we should use that behavior, so that the screen never unexpectedly moves

@Michael-Grover This can be implemented but I am running into some issues.

The value of window.scrollY is always coming out to be as much as the end of the page if I use it the the useEffect() hook which displays the modal.
Outside the hook I tried using state to store the window.scrollY value and scroll to that after the modal opens. But if I scroll the page with the opened modal, the editor crashes with an error tried to update state of an unmounted component is there a workaround for any of these problems ?

The value of window.scrollY is always coming out to be as much as the end of the page if I use it the the useEffect() hook which displays the modal.

Hmm, so the window.scrollY logic is to keep it from going to the top of the page when a link is clicked? Is it possible to prevent the scroll to the top of the page without doing this? Or maybe only do this if the modal is calculated to be outside of the current window view?

@irmerk This is possible from the second logic of using the state to hold the window.scrollY. It sounds weird but is there anyway I can just use the value of state once and then not update it later ?

@irmerk If you press Enter near the end of the page on any text. The page scrolls down to that viewport and brings it to center of the screen is that expected behavior ?

Can you please tell me about this @irmerk so I can see if this is related?

@irmerk This is possible from the second logic of using the state to hold the window.scrollY. It sounds weird but is there anyway I can just use the value of state once and then not update it later ?

Hmm well I think you can use useState and not use the second argument returned:
const [wscroll, setWscroll] = useState(...window.scrollY)

If you never change this with setWscroll, it should stay the same? Though I might not be fully understanding.

@irmerk This is possible from the second logic of using the state to hold the window.scrollY. It sounds weird but is there anyway I can just use the value of state once and then not update it later ?

Hmm well I think you can use useState and not use the second argument returned:
const [wscroll, setWscroll] = useState(...window.scrollY)

If you never change this with setWscroll, it should stay the same? Though I might not be fully understanding.

@irmerk this does not seem to work as window.scrollY will be used for the initial state setting only. So even if the user does scroll down, the value is still what it was before.

Can someone clarify what functionality is trying to be achieved with the scroll stuff?

@Michael-Grover When the link modal opens, the screen is scrolling to the top of the page. This could be due to the modal positioning and body height. But what we are trying to achieve is to manually scroll to a particular point on the page and overriding the scroll to top bug.
Hope this helps!

It looks like on Google docs, the screen never moves when the link modal pops up, even if the link is near the bottom of the page. I think we should use that behavior, so that the screen never unexpectedly moves

I don't think we should move the screen when opening a link. Google docs already has a nice way of handling link modals, I think we should copy that.

When the screen moves unexpectedly it can be disorienting and distracting for users

@Michael-Grover I have been trying to recreate that only!

sorry I misunderstood. Sounds good

Unsure what else to suggest on this one. @DianaLease any thoughts?

Played around with this locally and it seems that the page jumping issue is stemming from us setting the focus to be on the URL field when the modal opens. I'm wondering if we should just remove that functionality for now @Michael-Grover ? This page jumping bug seems like higher priority.

@DianaLease I think it's important that people can create a link without using their mouse to select the link input. Maybe I'm wrong but this requires us to set the focus on the URL field when the modal opens, right? Many people like using keyboard shortcuts to make a link, and when products mess with the basic steps for creating link it can be annoying.

Write text that I want to turn into a link
press alt+shift+backspace to highlight text
press cmd+k to make link
Type URL
Press enter to apply url
Press right arrow to deselect highlighted text and move text cursor back to the end of previously highlighted text
Continue typing

This sequence of key strokes is identical on Apple Mail, Gmail web client, Google docs, and Microsoft Word (sometimes step 6 is done automatically), and most other editors. To insert an additional step of having to move your mouse to click on the URL field, or having to tab a few times to focus the URL field, would throw a lot of people off. Slack, for example, does not use cmd+K as their hyperlink shortcut and it always trips me up when I try to make a link.

@Michael-Grover You'd still be able to add a URL with your keyboard. You'd just have to hit tab after cmd+k

I still think that's very disruptive to peoples' flow when they are used to being able to add a link in one second from muscle memory. Would people necessarily know that clicking tab will focus the correct field? I think they may just use their mouse rather than make an assumption of where focus will move when pressing tab.
Is the main problem that when the hyperlink field goes off-screen, the screen moves? Can we do what Google does and make the modal appear above the selected text when it is at the bottom of the screen?

If there is space for modal below highlighted text, put the modal below

If there is not space for modal below highlighted text, put modal above

If the highlighted text is off screen, move screen to text such that the text is in the middle of the screen and place modal above

@Michael-Grover I think I found a solution that allows us to keep the focus and not have the page jump. @elit-altum I'll put in a PR and tag you to check it out if that's alright

It'll be great if @DianaLease's PR works, but I think for context this is two separate issues. @Michael-Grover I agree that that is the ideal outcome, but we want to keep these PRs and Issues as narrow focused as possible. So if this would solve 1/2 problems, we shouldn't block it being merged and can follow up on the remaining issue with a new GitHub Issue.

PR here: https://github.com/accordproject/markdown-editor/pull/388


Jumpbug prediction

Hello. I cant add a simple jumpbug using engine prediction, it dont work for some reason.
The code:
  if (!config->misc.jumpbugkey.isDown())
    return;
  if (!EnginePrediction::getFlags() & 1 && localPlayer->flags() & 1)
  {
    cmd->buttons |= UserCmd::IN_DUCK;
    cmd->buttons &= UserCmd::IN_JUMP;
  }
}

I don't know how to fix it

are you putting it between the flags?

are you putting it between the flags?

Yes


Create nginx_upstream.md

添加 Nginx负载 功能

I have create a PR in MeteorGX/v2ray-step-by-step#1 which contain some format improvement, would you like to merge it?

I have create a PR in MeteorGX#1 which contain some format improvement, would you like to merge it?

sorry, that don't need PR, because there is too much content


flutter_view samples should show navigation between native and flutter

We are working on the new flutter_view sample, which splits out the "view composition" part of hello_services. Once that is done, we should consider also demonstrating how to navigate between "full screen" native (iOS/Android) and Flutter views.

Done in #10386.


merge from konsum-frontend-svelte

package.json

chore(deps): add ^8.3.7 remove ^8.3.4 (devDependencies.npm-pkgbuild)

:tada: This PR is included in version 1.31.14 :tada:
The release is available on:

v1.31.14
npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Support creating table for CDC connectors with the * syntax

Is your feature request related to a problem? Please describe.
When using a mysql cdc source after crfeating a source with
CREATE SOURCE test WITH (
  connector = 'mysql-cdc',
  hostname = 'xxx,
  port = '3306',
  username = 'xxx',
  password = 'xxx',
  database.name = 'auto',
  server.id = 5888
);

To create a table from the data base you have to specify all the columns explicitly in table creation
CREATE TABLE IF NOT EXISTS my_products (
    id STRING,
    -- name of the product
    -- format: <group> <type> (<import/domestic>)
    -- e.g. KNOT Type III (import)
    name STRING,
    price DECIMAL,
    PRIMARY KEY (id)
) FROM test TABLE 'auto.products'

Describe the solution you'd like
I would like to able to create the table using a select * syntax to automatically detect the columns in the table and create downstream tables
CREATE TABLE IF NOT EXISTS my_products (
    PRIMARY KEY (id),
    *
) FROM test TABLE 'auto.products'

Describe alternatives you've considered
No response
Additional context
This is is that when we have a huge number of tables with 100+ columns we can avoid the tedium of write scripts to auto generate schema for create table syntax

I would like to able to create the table using a select * syntax to automatically detect the columns in the table and create downstream tables

There's a side effect that * looks to imply that the schema should always be synced. May I ask about the desired behavior when there's a schema change in the source database (MySQL)? Would you like RisingWave to automatically sync the schema, or would you prefer to make manual schema changes?

This is is that when we have a huge number of tables with 100+ columns we can avoid the tedium of write scripts to auto generate schema for create table syntax

Do you think it would be helpful to have a script that fetches the schema and converts it to RisingWave DDL?

I would like to able to create the table using a select * syntax to automatically detect the columns in the table and create downstream tables

There's a side effect that * looks to imply that the schema should always be synced. May I ask about the desired behavior when there's a schema change in the source database (MySQL)? Would you like RisingWave to automatically sync the schema, or would you prefer to make manual schema changes?

Good point.
Making it sync with upstream will introduce much more complexity. I suppose the direct requirement here is just pulling the schema at this moment from upstream DB, and we can work it out quickly. Furthermore, this is also consistent with the "external catalog" we were discussing recently, which makes a unified framework to read the upstream schema and create tables accordingly.

Agree with Eric's idea on only pulling the full schema at the first time setting up the CDC table.
After that, the table schema should be static unless manually updated.
Automatic schema change is another topic to concern.

In an ideal world schema change would be better appended automatically.
If it just syncs the first time , how would I add new columns in case of a schema change?
Can I manually add changes?

Do you think it would be helpful to have a script that fetches the schema and converts it to RisingWave DDL?

This might be a good easy solution too!

Yeah, that's an ideal solution and also aligns with our vision.
However, automatic schema change might require substantial effort.
At least for now, we suggest manually managing schema changes via https://docs.risingwave.com/docs/current/sql-alter-table/#add-column, which is the common practice in most traditional databases.


performance: optimize by protobuf context-type.

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

replace #1596
Fixes #

is there before-and-after data comparison?

/retest

current gateway API does not support protobuf. raised issue https://github.com/kubernetes-sigs/gateway-api/issues/3346#issue-2535691054

xref: https://github.com/envoyproxy/gateway/issues/4283#issuecomment-2370314094
https://github.com/envoyproxy/gateway/issues/4283#issuecomment-2378284177
EG has already support multi content-type by controller-runtime


Add support for key sharing to KVL 4000

This commit introduces support for key sharing from KFDtool software to the Motorola KVL 4000.
Sharing is implemented as an automatic feature when using Keyload or Multiple Keyload functionality; the software detects whether the connected device is a KVL, and adjusts its behaviour appropriately.
The main differences are a new opcode (D1 indicating a KVL vs D0 indicating an MR), and limiting the KMM size for transfers to a KVL.
The latter is required because the KFD* hardware has a limited onboard buffer, and the KVL does not like when there is too much delay between bytes on the TWI data line. A limited KMM size avoids calls being split over multiple CMD_SEND_BYTES commands.

LGTM, no impacts to TWI code and I have no way to test DLI mode for breaking changes.

LGTM, we will monitor for DLI regression reports as I do not have hardware to be able to test either.


[App]: Runs do not appear in the webinterface

Current Behavior
When I create new sweeps and run them, the runs do not appear in the webinterface, even though I can locally see that they are running. However the run count clearly shows that there are runs, the table just does not display any.
Expected Behavior
I expect to see the runs that have been started and recognized by WandB.
Steps To Reproduce
Create a sweep and start agents.
Screenshots

Environment
OS: MacOS
Browsers: FireFox
Update:
Apparently the runs appear as soon as they are finished, but not while they are running. As soon as one run is finished, the other 'running' runs are also appearing.

I have the same issue.

Hello @christophe-rx and @zimmer-m !
To be clear, this is an unintended behavior for sweeps to only be updated at the end of the sweep. Each run within a sweep should show up in the UI as it is being executed. If you are still experiencing this, could you share a link to the sweep that is not showing up? I have tested this on my end on wandb version 0.15.10 and this behavior is not appearing for me so could you also provide your SDK versions?

Raphael San Andres commented:
Hi, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

Hi, I am facing this issue today. It's urgent.
Thanks,
Ayush

I'm also facing this problem - it started around ~1h ago. I guess it might be a server-side problem.

We are closing this issue due to inactivity. If you feel that it merits additional review, please provide a reproduction and add a comment explaining why. Thank you.


Stat number of retries performed during join process

This stat will give us a better idea about how successful the join fan-out is for all Ringpop-based applications.
@uber/ringpop

ship it

wewt


Disconnects when using on AWS instance via Remote

Issue Type: Bug
Connection to the server (using a key pair) works without problems, but when and only when the Comment Anchors extension is installed, the connection will drop a few moments in (when comment anchors is ~7% into scanning a large workspace), forcing a reload of the VS window- effectively renders the extension unusable.
Extension version: 1.6.1
Remote version: 0.49.0
VS Code version: Code 1.42.1 (c47d83b293181d9be64f27ff093689e8e7aed054, 2020-02-11T14:50:36.977Z)
OS version: Linux x64 5.3.0-40-generic
Remote OS version: Linux x64 4.15.0-1058-aws
Remote OS version: Linux x64 4.15.0-1060-aws

Have you tried disabling workspace anchors by setting commentAnchors.workspace.enabled to false in your settings?


Added Results Scraping after a suite is run

After the usual run of teuthology-results, the scraper script produces a summary of the results and stores it in scrape.log inside the archive-dir of the results.
Changes made

Added the scraper script from https://github.com/jcsp/scrape as teuthology/scrape.py.

Modified the script at line 470 to include printing of the scrape logs.

Modified teuthology/results.py to run the scraper after every result generation and emailing.

Testing
Since I don't have access to a paddle database, I did the following to test:

Installed the package using pip install ..

In the site-packages, commented out / filled with dummy data, the lines that required calling the endpoints on the database.

Then ran:

teuthology-results --archive-dir results --name rados --email<EMAIL_ADDRESS>--dry-run --timeout 2

Here results is a sample results directory given by @jdurgin
Signed-off-by: Shubham Mishra<EMAIL_ADDRESS>

Can one of the admins verify this patch?

jenkins: ok to test

Sorry, I wrongly ran git rebase with HEAD~4. That is why previous history of other commits is coming here.

@jdurgin Please review.

could you add a "Signed-off-by" line at the end of your commit message? "git commit -s" will do the trick for you. see https://github.com/ceph/ceph/blob/master/SubmittingPatches.rst#1-sign-your-work

could you add a "Signed-off-by" line at the end of your commit message? "git commit -s" will do the trick for you. see https://github.com/ceph/ceph/blob/master/SubmittingPatches.rst#1-sign-your-work

Added manually for now. Will keep in mind from now on. Thanks

@grapheo12 please add it to the commit messages - you can do this by interactive rebase, e.g. git rebase -i origin/master, mark both commits to be edited, and for each one git commit --amend -s && git rebase --continue

@jdurgin Done.

ok to test

ok to test

I wonder if this code has a unit tests?

@kshtsk the goal for this PR is just to get scrape into the repo - it's been quite helpful but many users of teuthology are unaware of it - we can continue improving it in other PRs. I'm happy merging it as-is and improving tests and style in further PRs. Is that ok with you?

@kshtsk the goal for this PR is just to get scrape into the repo - it's been quite helpful but many users of teuthology are unaware of it - we can continue improving it in other PRs. I'm happy merging it as-is and improving tests and style in further PRs. Is that ok with you?

it is okay as soon as it is py2/py3 compatible

E     File "/home/jenkins-build/build/workspace/teuthology-pull-requests/teuthology/scrape.py", line 147
E       return "common/lockdep" in job.get_assertion():
E                                                     ^
E   SyntaxError: invalid syntax

@kshtsk @jdurgin Please review.

@jdurgin is this code supposed to be running for each teuthology job?

@jdurgin is this code supposed to be running for each teuthology job?

Just once for each run, it checks all jobs in a run at once.

Well then this code for sure will break py3 based teuthology setup, we've recently merged last portion of py3 compatible code, and we cannot switch our sepia to py3 yet know, so we enable py3 unit/functional test for teuthology, so it is highly desired any new code have a test covering real py execution, because static analysis tools does not catch issues like I mentioned in a review.

If it is needed, I can write unit tests for it. Please guide on what kinds of checks are required.

If it is needed, I can write unit tests for it. Please guide on what kinds of checks are required.

we basically need a functional test, since the code involve file operations, and system grep are involved, it is probably makes sense to have some shorten variant of real logs would be great to have parsed.

Should I do that on a separate PR?

Should I do that on a separate PR?

It would be highly appreciated to have it within this PR, as I said, it will break py3 part if we not test it and merge.

@susebot run deploy

Not blocking this PR, but please add a follow on PR to document this as well. Thanks.

@vasukulkarni @kshtsk I am disabling the integration for now.
Should I put the unit tests, integration part and the documentation together in the next PR?

@vasukulkarni @kshtsk I am disabling the integration for now.
Should I put the unit tests, integration part and the documentation together in the next PR?

Sure that works!

@grapheo12 do you still plan to add tests as a separate PR or anything like that?

@kshtsk Yes I do plan to work on that. But due to my studies, I will not be able to concentrate on it till the 1st week of May. I'll definitely work on it after that. Please give me some time if that is not a problem.

@kshtsk Yes I do plan to work on that. But due to my studies, I will not be able to concentrate on it till the 1st week of May. I'll definitely work on it after that. Please give me some time if that is not a problem.

np, take your time, it was just a heads-up

A month passed from the 1 May, there is a conflict already, also the master branch is switched to py3. So we want this merged, we need to:

resolve conflicts
add tests so we can cover py2 and py3 testing coverage so we can back port this to py2 as well.

@kshtsk Sorry for the delay. I started looking into it a week or so back. But could not decide what kind of tests to write for it. It'd help if you can provide me the with a brief overview of the test functions I need to write. I have no background in writing tests. Sorry again.

The code is compatible with both py2 and py3 I guess, based on the changes I made after the comments I received in this thread.

@kshtsk @jdurgin Please review if I am going in the right direction in writing the test.

retest this please.

hey @grapheo12 I'm sorry for so long response, however, I tried to run tests and can see messages like that:
    self.config = yaml.load(open(os.path.join(self.path, "config.yaml"), 'r'))
  /home/kyr/kshtsk/teuthology/teuthology/scrape.py:271: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    self.summary_data = yaml.load(open(summary_path, 'r'))

Can we get rid of it.

@grapheo12 and I guess we can try and enable it.

I am also integrating the upstream changes to the script with this.

@susebot run deploy

https://ceph-ci.suse.de/job/teuthology-deploy-ovh/269/artifact/logs/run/scrape.log/*view*/
Here is the log it generated.

Corresponding result.log:
https://ceph-ci.suse.de/job/teuthology-deploy-ovh/269/artifact/logs/run/results.log/*view*/

@grapheo12 I'm sorry I removed your comment with that big diff since it is not relevant and present a noise.

I cleaned the DNM tag.

@grapheo12 maybe you can squash the commits?

@kshtsk All commits into 1? Or do you want me to keep some specific stages?

@kshtsk All commits into 1? Or do you want me to keep some specific stages?

all in one, since it is initial version, I don't have any tender feelings for the hardening of the first version :-) And it would be good to have it monolithic.

@susebot run deploy

Is this somehow error related to this PR:
teuthology.exceptions.CommandFailedError: Command failed on target-pr-1432-045 with status 1: 'sudo yum -y install ceph-radosgw'

@grapheo12 I don't think so, but is nice to see how the scrape parses errors:
https://ceph-ci.suse.de/job/teuthology-deploy-ovh/270/artifact/logs/run/scrape.log/*view*/

The failure is related to fedora repos accessibility (see report https://ceph-ci.suse.de/job/teuthology-deploy-ovh/270/artifact/logs/run/teuthology-job-3.html):
2020-06-26T15:32:46.188 INFO:teuthology.orchestra.run.target-pr-1432-045.stdout:You can remove cached packages by executing 'dnf clean packages'.
2020-06-26T15:32:46.208 INFO:teuthology.orchestra.run.target-pr-1432-045.stderr:Error: Error downloading packages:
2020-06-26T15:32:46.209 INFO:teuthology.orchestra.run.target-pr-1432-045.stderr:  Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=centos
2020-06-26T15:32:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1

I would change the commit title to results: add scrape after run finished so it is clear to which teuthology package the scrape is added.
I am fine this PR to go, still @jdurgin review required.

Apologies from me to drag this for so long :sweat_smile:


Molecule 2.22rc3: Vagrant driver regression

Issue Type

Bug report

Molecule and Ansible details
Ansible: 2.8.1
Molecule 2.22rc3
Python: 2.7.12
Vagrant: 2.0.2
Virtualbox: 5.2.30r130521
Ubuntu 16.04.6 LTS x86_64
Molecule installation method (one of):

pip

Ansible installation method (one of):

pip

Detail any linters or test runners used:
Desired Behavior
Running Molecule test on a sample role with a Vagrant driver and a Virtualbox provider should result in a Virtualbox VM being brought up at some point.
Actual Behaviour
Debug output snippet:
    TASK [Create molecule instance(s)] *********************************************
    task path: /usr/local/lib/python2.7/dist-packages/molecule/provisioner/ansible/playbooks/vagrant/create.yml:8
    <<IP_ADDRESS>> ESTABLISH LOCAL CONNECTION FOR USER: go
    <<IP_ADDRESS>> EXEC /bin/sh -c 'echo ~go && sleep 0'
    <<IP_ADDRESS>> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038 `" && echo ansible-tmp-1562189970.7-142759633793038="` echo /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038 `" ) && sleep 0'
    Using module file /usr/local/lib/python2.7/dist-packages/molecule/provisioner/ansible/plugins/libraries/molecule_vagrant.py
    <<IP_ADDRESS>> PUT /var/go/.ansible/tmp/ansible-local-10147KOol3i/tmpmVJNT9 TO /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038/AnsiballZ_molecule_vagrant.py
    <<IP_ADDRESS>> EXEC /bin/sh -c 'chmod u+x /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038/ /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038/AnsiballZ_molecule_vagrant.py && sleep 0'
    <<IP_ADDRESS>> EXEC /bin/sh -c '/usr/bin/python2 /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038/AnsiballZ_molecule_vagrant.py && sleep 0'
    <<IP_ADDRESS>> EXEC /bin/sh -c 'rm -f -r /var/go/.ansible/tmp/ansible-tmp-1562189970.7-142759633793038/ > /dev/null 2>&1 && sleep 0'
    The full traceback is:
    WARNING: The below traceback may *not* be related to the actual failure.
      File "/tmp/ansible_molecule_vagrant_payload_VrLKoY/__main__.py", line 395, in up
        self._vagrant.up(provision=provision)
      File "/usr/local/lib/python2.7/dist-packages/vagrant/__init__.py", line 337, in up
        self._call_vagrant_command(args)
      File "/usr/local/lib/python2.7/dist-packages/vagrant/__init__.py", line 963, in _call_vagrant_command
        stderr=err_fh, env=self.env)
      File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
        raise CalledProcessError(retcode, cmd)

    failed: [localhost] (item={'box': u'ubuntu/xenial64', 'name': u'Ubuntu'}) => {
        "ansible_loop_var": "item",
        "changed": false,
        "invocation": {
            "module_args": {
                "config_options": {},
                "force_stop": false,
                "instance_interfaces": [],
                "instance_name": "Ubuntu",
                "instance_raw_config_args": null,
                "platform_box": "ubuntu/xenial64",
                "platform_box_url": null,
                "platform_box_version": null,
                "provider_cpus": 2,
                "provider_memory": 512,
                "provider_name": "virtualbox",
                "provider_options": {},
                "provider_override_args": null,
                "provider_raw_config_args": null,
                "provision": false,
                "state": "up"
            }
        },
        "item": {
            "box": "ubuntu/xenial64",
            "name": "Ubuntu"
        },
        "msg": "ERROR: See log file '/tmp/molecule/test-role-vagrant/default/vagrant-Ubuntu.err'"
    }

Content of sample error file identified in Molecule output:
### 2019-07-03 10:08:56 ###
### 2019-07-03 10:08:56 ###
A Vagrant environment or target machine is required to run this
command. Run `vagrant init` to create a new Vagrant environment. Or,
get an ID of a target machine from `vagrant global-status` to run
this command on. A final option is to change to a directory with a
Vagrantfile and to try again.

Note that with the same sample role and environment, rolling back to Molecule 2.20.1 via pip install molecule==2.20.1 fixed the problem and Vagrant boxes were again created successfully.

Can anyone using the Vagrant driver comment on this? Can anyone reproduce such an error with the latest release candidate? I don't use this driver, so I can't comment on this.

I have got the same Issue with vagrant driver. But instead I use the libvirt provider. (Provider doesn't matter I think...)
It is very likely that the invokation of the vagrant command via "vagrant-molecule" has not been adapted to the new directory structure.
The vagrant command needs to be invoked where the Vagrantfile is located. With the introduction of the new directory structure this location changed from /tmp/molecule/[role name]/[scenario name] (1.20.1) to
/tmp/mol.[user]/molecule/[role name]/[scenario name]/molecule/vagrant/[scenario name] (1.22rc3).
As far as I understand, the working directory for the vagrant call needs to be adapted to the correct path.

That's awesome @jonasjoest, you're probably right. Can anyone pick this up?
I'll try to get to it sometime soon ...
This should block any further release @ssbarnea.

After further investigation it turned out that the env variable MOLECULE_EPHEMERAL_DIRECTORY is still set to /tmp/mol.[user]/molecule/[role name]/[scenario name].
The molecule-vagrant - Ansible module assembles the vagrant CWD with that MOLECULE_EPHEMERAL_DIRECTORY  env variable:
https://github.com/ansible/molecule/blob/55f3b11c21379bacd82db5601d648a6c3e1156c6/molecule/provisioner/ansible/plugins/libraries/molecule_vagrant.py#L474
But instead the vagrant driver uses the config attribute self._config.scenario.ephemeral_directory to create the Vagrantfile.
https://github.com/ansible/molecule/blob/55f3b11c21379bacd82db5601d648a6c3e1156c6/molecule/driver/vagrant.py#L198
Since the Vagrantfile is in the end located at
/tmp/mol.[user]/molecule/[role name]/[scenario name]/molecule/vagrant/[scenario name]
and the env variable points to
/tmp/mol.[user]/molecule/[role name]/[scenario name]
there has to be a mismatch between environment variables and config attributes as far as I understand.
So this issue is perhaps not only vagrant-driver-specific. Maybe you can try to confirm that the config attribute in python does not equal the environment variable in terms of the ephemeral directory. I am currently not able to test this though.

Any patch would be welcomed, I will review it with priority.

@ssbarnea we need to fix this before make any more release candidates.

@ssbarnea please see https://github.com/ansible/molecule/pull/2170#pullrequestreview-270474667. I think we should merge that fix and make a final release candidate.


[ECS] Overriden skip in predefined sets

In some predefined sets you're using the skip option. For instance here:
https://github.com/symplify/symplify/blob/44354dd1ef1b6c62b1285c4c4ceb199de2c07016/packages/easy-coding-standard/config/set/common/control-structures.php#L69-L71
This however isn't ideal since the skip setting is later changed by either another set or the user's own skip settings so this gets lost and ECS then reports errors that should have been skipped.

Yes, it's not optimal. It's to avoid including this skip manually in every single import of such config.

Looking into it more I guess you can't really do much else eh... \Symfony\Component\DependencyInjection\Loader\Configurator\ContainerConfigurator doesn't really give you any good API for that.
The only possibility I can think of is not using the skip option in sets but instead calling a method on some service that would add it to the skip parameter later.

Exactly. This issue is known for years:

https://github.com/symplify/symplify/issues/2906

Until someone adds it to Symfony core, there is nothing we can do


Structured fields not logged if they end with a quote

NLog version: 5.2.4
Platform: .Net Framework 4.7.2 and 4.8.
Current NLog config
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      internalLogLevel="Trace"
      internalLogFile="internal.txt">

  <targets>
    <target name="logconsole" xsi:type="Console" />
    <target xsi:type="File"
            name="jsonLog"
            fileName="json.log">
      <layout type="JsonLayout"
              IncludeAllProperties="true"
              ExcludeEmptyProperties="true"
              />
    </target>
  </targets>

  <rules>
    <logger name="*" minlevel="Info" writeTo="logconsole" />
    <logger name="json" writeTo="jsonLog"/>
  </rules>
</nlog>

Source code
private static readonly Logger Logger = LogManager.GetCurrentClassLogger();
static readonly Logger jsonlog = LogManager.GetLogger("json");
static void Main() {
    try {
        Logger.Info("Hello world");
        var accessLogEventInfo = new LogEventInfo(LogLevel.Info, null, null);
        accessLogEventInfo.Properties["noquote"] = "No quote:";
        accessLogEventInfo.Properties["endquote"] = "I quote: \"";
        jsonlog.Log(accessLogEventInfo);
    }
    catch (Exception ex) {
        Logger.Error(ex, "Goodbye cruel world");
    }
}

What is the current result?
{ "noquote": "No quote:" }

What is the expected result?
{ "noquote": "No quote:", "endquote": "I quote: \"" }

The workaround is to configure ExcludeEmptyProperties="false".

Did you checked the Internal log?  Yes:  the two "internal.txt"s from runs with ExcludeEmptyProperties true and false are identical.

This did not work in version 4, either.  I am not aware of a version where it did.

Can you help us by writing an unit test?  I could, but have not yet.

I feel like I'm missing something obvious here, since it's unlikely I'm the first to strike this problem.  So if this is a misconfiguration rather than a bug, apols.

Hi, I can look at it if I may.

@Aragorn159 You don't have to ask for permission to make an investigation.
Anyway it looks like the result is this:
{{ "noquote": "No quote:", "endquote": "I quote: \""}

And then there is this validation, which make faulty decision:
            if (ExcludeEmptyProperties && (sb[sb.Length-1] == '"' && sb[sb.Length-2] == '"'))
            {
                sb.Length = initialLength;
            }

Created bug-fix #5365

Thank you for reporting this bug, and providing the wonderful details. Fix will be included in the next release of NLog.

NLog v5.2.5 has been released - https://www.nuget.org/packages/NLog/5.2.5#release-body-tab


Draw line between two hotspots

Hi,
I am implementing Marzipano in my project and need to draw line between two hotspots. I am not getting any way to draw line. Please help me to draw line between hotspots.
Thanks

The Marzipano API doesn't support drawing arbitrary geometric objects. That said, if you're ok with the line being straight in screen space (not in spherical space), you could probably accomplish it with something like the following:

Superimpose a <canvas> with a 2D rendering context (https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D) on top of the viewer element;
Listen to the viewChange event to detect view changes, then use the coordinatesToScreen to convert the spherical coordinates of the two endpoints into screen coordinates;
Draw a line between the two endpoints on the 2D canvas, leaving the background transparent so you can see the pano underneath.

Closing due to lack of activity.

Hi,
Thanks for this help.
To draw a line I draw two hotspots on marzipano and draw div with border and transform: rotate(angle) between these two hotspots  and it works fine.
var angle = Math.atan2(pointB.top - pointA.top, pointB.left - pointA.left) * 180 / Math.PI
But now the issue is when we move marzipano up/ down at some extend point1 or point2 goes out of screen and line position distorted.
Please help he how to fix this issue.

Do you have a demo link we can take a look at?

Hi
Basically we want to draw line between 2 hot spots. but while any of our hotspot get out of screen it's display changes to None. So we want to set it display block, so that we will get it's coordinates.
We have made changes in hotspot.js and set this.visible=true  , still it changes to none.
Please help me so that I will draw lines as per our requirement.
Thanks

I didn't quite understand how you are trying to implement this. You mentioned that you "draw a div with border and transform: rotate(angle) between these two hotspots and it works fine". Are you drawing it on a canvas? Using CSS transforms? A demo link would help.

Hi,
For drawing line between two points first of all I draw two hotspots. Then I get screen coordinates of these two points and draw a div with rotation and width.  On viewChange I am drawing line between these two points. For some extend it works fine, but after some extend any of my hotspot display changes to none due to which I am not able to find it's coordinates (offset) so now I get coordinates (0, 0) due to which my line get distorted.
So now I do not want to change my hotspot display so that I will get its coordinates. Please help me resolve this issue. It's very urgent for me.
Thanks

How are you getting the hotspot screen coordinates? (RectilinearView.coordinatesToScreen)[http://www.marzipano.net/reference/RectilinearView.html#coordinatesToScreen] should always return a value unless the hotspot is behind the camera. If the hotspot is behind the camera, I don't think there's any sensible value to return for the screen coordinates.


Redis client library is outdated, doesn't support cluster mode

Well, topic says it all - https://github.com/uber/makisu/blob/master/Gopkg.lock#L173-L182 the repo with client library is archived in favor of upstream https://github.com/gomodule/redigo. Apparently, upstream repo also does not support cluster mode, so it might be beneficial to migrate to well-maintained library like this one https://github.com/go-redis/redis, to kill two birds with one stone.

In practice, this what happens when you try to run makisu against Redis cluster:
{"level":"error","ts":1545209445.746716,"msg":"Failed to push cache: store tag mapping (65b6ee26,902f6a63afa7effffc16c0c14bb3dbf5bbe4926dd689c2aaddc8e792044fb5ab,5c20fb913c01b4f9b748d2fcfcb455f77a8a709ec86300a98b80280d730e7616): MOVED 15630 x.x.x.x:6379; store tag mapping (d0b17d2e,f7dae4047cccd0e0ce35a4e758f2b7b60707b9f54ba5cc52c53837a6acf31ca5,f9a83abf7e5603e1bd3d8242525dab840dba8b23921c0385460b074f06c8aa0a): MOVED 8778 x.x.x.x:6379"}

Need a few more days to create a new release.


Provide "require signed commits" on github_branch_protection

This commit provides changing the toggle to require signed commits on
branch protection.
The API for signed commits
is separate endpoint from the branch protection url, and does not come on the same payload.
This means that we need to make an extra request to read and modify it.
This changes requires go-github at least v19.
Needs:

[ ] Update go-github version: https://github.com/terraform-providers/terraform-provider-github/pull/207
[ ] Provide default required reviewers count to avoid GitHub update error: https://github.com/terraform-providers/terraform-provider-github/pull/211

Closes https://github.com/terraform-providers/terraform-provider-github/issues/87

I'll gladly rebase or pick the last commit apart if necessary, but updating the version of go-github is important for it. After #207 lands, I'll gladly update the PR.
Note: precompiled plugin to validate the changes here


Getting error (LayoutException: RecyclerListView needs to have a bounded size. Currently height or width is 0)

Facing same issue ...found any solution for this?please suggest


Improve handling of e-mail related properties

Original issue 113 created by soi-toolkit on 2011-03-13T08:33:19.000Z:
The default values of e-mail related properties (IMAP, POP3, SMTP) are today hardcoded and needs to be handled in the same way as properties for other transports, i.e. environment specific properties (e.g. host names, ports and default email-addresses) need to be configurable in the eclipse plugin preference page and security related properties (username and passwords) needs to be alerted when created in the security property file.

Won't fix.
The v2.x generator templates no longer have support for IMAP, POP, SMTP.


Update README - highlight "winning" metrics

Should make it more digestable

Closing this. PR to change the generation and add highlighting will be welcomed.


This article should include the updated nested field copy and replace

Type of issue
Outdated information
Description
The information in the announcement for Nested record field copy and update should be added to this article.
Page URL
https://learn.microsoft.com/en-ca/dotnet/fsharp/language-reference/copy-and-update-record-expressions
Content source URL
https://github.com/dotnet/docs/blob/main/docs/fsharp/language-reference/copy-and-update-record-expressions.md
Document Version Independent Id
2499d6cc-0b9b-cce6-f8f8-a6e09964b444
Article author
@ChrSteinert
Metadata

ID: 474586f3-428e-e787-e0ae-5b39331635cc
Service: dotnet-fsharp

Related Issues

Hi @BillWagner / @ChrSteinert, I will work on this and raise PR soon.


MLT options not showing in token dialog box

I am using Multilevel Tokens v1.4.2 with Foundry v0.7.10, hosted on The Forge. However, with the mod enabled, I do not see the MLT tab in the token dialog box. This happens even when MLT is the only mod enabled in my game. I believe this started after I updated Foundry to 0.7.10.

Afraid I can't reproduce this issue - shows up fine for me with foundry 0.7.10 on the forge.

token dialog box

MLT is configured in the settings for drawings, not tokens, though, are you looking in the right place?

Oh my gosh.  I completely forgot where to look.
The tab for MLT is right there on the drawing dialog box.
Thank you for pointing that out. I'm sorry to have wasted your time.


Snippet: createDepositAddress()

createDepositAddress();
async function createDepositAddress() {

    let response = undefined
    const currencyCode = 'USDT'
    const amount = 1.2345

    const Exchange = ccxt["binance"];
    const exchange = new Exchange({ enableRateLimit: true, 'timeout': 30000 }); //(Have Also set API keys)
    exchange.timeout = 30000;
    try {

        if (exchange.has['fetchDepositAddress']) {
            response = await exchange.fetchDepositAddress(currencyCode)
        } else if (exchange.has['createDepositAddress']) {
            response = await exchange.createDepositAddress(currencyCode)
        }
        console.log(
            'In order to deposit ' + amount.toString() + ' ' + currencyCode + ' to ' + exchange.id +
            ', please send your ' + amount.toString() + ' ' + currencyCode + ' to the address provided by ' +
            exchange.id + ' exchange for depositing: ' + response['address'] + (response['tag'] || '') + ' ' + "Network: " + response["network"] + ' ' + "Info: " + response["info"])

    } catch (e) { console.log(e.constructor.name, e.message) }
}

Source: https://github.com/ccxt/ccxt/issues/12378#issue-1172751000

Note:
Some exchanges have a method called exchange.fetchDepositAddressesByNetwork(), that method allows fetching all networks at once, but in general it is rare, and very few exchanges will support it. You cannot return all networks at once from exchange.createDepositAddress() or from exchange.fetchDepositAddress(), but you can specify a network parameter to exchange.createDepositAddress() or to exchange.fetchDepositAddress() to override the default network returned from the exchange:
await exchange.fetchDepositAddress(currencyCode, { 'network: 'TRC20' })

Also, if the exchange does not have exchange.fetchDepositAddressesByNetwork(), then also check exchange.currencies[currencyCode] for a networks property inside the currency in question. Some exchanges will provide the networks info inside the currencies-info. Note, that you have to exchange.loadMarkets() before accessing markets and currencies. Hope that answers your question, let us know if not.
Source: https://github.com/ccxt/ccxt/issues/12378#issuecomment-1072408421


Reviw python version support

Validate works with Python 3.12 remove support for deprecated versions of python

Update so tests under multi versions using tox and move to pyproject.toml format.


Ollama not use GPU

What is the issue?
I am using Ollama , it use CPU only and not use GPU, although I installed cuda v 12.5 and cudnn v 9.2.0 and I can check that python using gpu in liabrary like pytourch (result of command (>>> print(torch.backends.cudnn.is_available())
True, ), I have Nvidia 1050 ti and I am trying to runn llama3 8B model, i found this warning in ollamam server log "level=WARN source=gpu.go:177 msg="CPU does not have AVX or AVX2, disabling GPU support."
OS
Windows
GPU
Nvidia
CPU
AMD
Ollama version
0.1.43

解决了么？我也遇到同样的问题

I also encountered the same problemI also encountered the same problem, my Ollama version is the Windows installation version, version number 0.1.43

I also encountered the same problemI also encountered the same problem, my Ollama version is the Windows installation version, version number 0.1.44

Getting the same error, I'm running ollama on docker on arch linux, I've already installed nvidia-container-toolkit, and I'm able to run nvidia-smi command inside the container, and under the processes using GPU in nvidia-smi, there's no process using GPU inside the container, while I'm using ollama to run llama3.

Unfortunately you're running this on some pretty old hardware. The RTX 1050 Ti only has 4 GB of VRAM, so llama3 8B would be tight even at 4 bit quantization even if you had AVX on the CPU. That said, this is a dupe of #2187 . In the future we'll make it easy for you to recompile Ollama on your own w/o AVX support but w/ CUDA support.

@pdevine can you tell me why I'm not able to run it on my pc, I got a RTX 4080 Ti.

@pdevine Can ollama run llama3 utilizing the GPU? I am running ollama on an iMac 27" with an AMD Radeon Pro 5700 XT.  llama.cpp knows about the AMD GPU.
E.g.
 ./main -m /Users/davidlaxer/llama.cpp/models/7B/ggml-model-q4_0.gguf -n 128 -ngl 1
Log start
main: build = 3051 (5921b8f0)
main: built with Apple clang version 15.0.0 (clang-15<IP_ADDRESS>) for x86_64-apple-darwin23.5.0
main: seed  =<PHONE_NUMBER>
llama_model_loader: loaded meta data with 16 key-value pairs and 291 tensors from /Users/davidlaxer/llama.cpp/models/7B/ggml-model-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = models
llama_model_loader: - kv   2:                       llama.context_length u32              = 2048
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 259
llm_load_vocab: token to piece cache size = 0.3368 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 2048
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
llm_load_print_meta: general.name     = models
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.30 MiB
ggml_backend_metal_log_allocated_size: allocated buffer, size =   108.60 MiB, (  108.60 / 16368.00)
llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/33 layers to GPU
llm_load_tensors:      Metal buffer size =   108.60 MiB
llm_load_tensors:        CPU buffer size =  3539.27 MiB
.................................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: AMD Radeon Pro 5700 XT
ggml_metal_init: picking default device: AMD Radeon Pro 5700 XT
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/davidlaxer/ollama/llm/llama.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   AMD Radeon Pro 5700 XT
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = false
ggml_metal_init: hasUnifiedMemory              = false
ggml_metal_init: recommendedMaxWorkingSetSize  = 17163.09 MB
ggml_metal_init: skipping kernel_mul_mm_f32_f32                    (not supported)
ggml_metal_init: skipping kernel_mul_mm_f16_f32                    (not supported)
ggml_metal_init: skipping kernel_mul_mm_q4_0
...

I'm having a similar issue. Two days ago I have started ollama (0.1.44) with Docker, used it for some text generation with llama3:8b-instruct-q8_0, everything went fine and it was generated on two GPUs.
Today I wanted to use it again, but it did the generation on a CPU instead of GPU.  Same thing happened when I tried to use an embedding model. The solution was to restart the docker container...

@dvdblk if that happens again, you may want to check the host and see if the nvidia_uvm driver unloaded.  We recently had to add this new logic to our install script to make sure things stay loaded on driver 555 or newer.

@dvdblk if that happens again, you may want to check the host and see if the nvidia_uvm driver unloaded. We recently had to add this new logic to our install script to make sure things stay loaded on driver 555 or newer.

Thanks for the input. I'm on 535.171.04 right now. I vaguely remember an nvidia-driver issue from last year where the driver would stop working (unload?) after some time and it required a system reboot, but I have fixed this long time ago.
I'm not sure how to reproduce ollama losing access to GPU but the container was up for 40+ hours and until it got restarted the generation was done on CPU. I also have a few other containers running that use very small amount of vram on GPUs and they kept working while ollama started suddenly using CPU. Will try to get some logs the next time.


Reorg repo and README

Here are some ideas I had to reorganize this repo.

a few pointers from Irene

keep the old intro text

Left some comments but overall I like this change a lot!

Thanks for the review @keiko713 ! I'll incorporate these changes.

Okiedoke, this should be good now.


Duplicate Users in staging DB

In the staging db, there are some duplicate User email and name records.
I believe this issue is caused by the async execution of the list of saveCommit promises in src/listeners.ts.
Might also want to add a unique constraint on the email, name columns.

Was able to duplicate the bug on development environment after increasing the network latency of local containers.

May I know what's your planned solution to fix the issue?

Issue is fixed by having the promises execute sequentially.
I need to write some SQL to clean up the duplicate data, should I include this in the repo?

Issue is fixed by having the promises execute sequentially. I also want to add a unique constraint on user email/name columns so that this throws an error in the future.

I agree with this fix. Another thing to note is that it's better to run the each save commit action in a transaction, to ensure each save is atomic.

I need to write some SQL or use Knex to clean up the duplicate data, should I include this in the repo?

This is a data migration, so you can create a new migration for this. I suppose the migration should:

clean up the duplicated data; and
setup unique constraints.