Specify provisioning profile in Xcode 8 with gym

With Xcode 8 and manual code signing, a new build parameter is used to specify a provisioning profile to sign with.
It's all covered in the code signing guide, but for the ones that are not yet using match, it would be useful to have a gym parameter reflecting the change in Xcode. And then our Fastfiles could have something like this:
gym({
    workspace: "AwesomeProject.xcworkspace",
    scheme: "AwesomeScheme",
    configuration: "Release",
    provisioning_profile_specifier: "Foo AppStore"
  })

Right now, my solution involves the xcargs parameter, like this:
  gym({
    workspace: "AwesomeProject.xcworkspace",
    scheme: "AwesomeScheme",
    configuration: "Release",
    xcargs: "PROVISIONING_PROFILE_SPECIFIER='Foo AppStore'"
  })

But it wasn't easy to figure it out for me and I think that a proper gym parameter with README documentation could be more discoverable.
@KrauseFx What do you think?
I can a open PR with the implementation if y'all agree it makes sense adding this parameter.

@chrisfsampaio Thanks for reaching out. I want to leave this open for feedback before giving you the go ahead for a PR. The team will take a look here.

+1

+1

@chrisfsampaio Thank you so much, this was exactly what i needed.
No I can enjoy my friday afternoon drinks 👍

Thanks 👍 How does this work if you have multiple targets?

We don't use fastlane, only the fastlane tools (deliver / match / gym) to deliver our apps. In gym we specify the parameters manually for each target.

@KrauseFx with multiple targets, this would set the same provisioning profile for every target under the selected scheme. We can make that explicit in the parameter documentation.
In this particular case it works well because we only have one main target, which I would guess is the setup for most of the apps. Once it gets more complex, with multiple targets and different configurations (e.g. AppStore, Debug, Release Candidate), we should advice the use of match or the automatic system embedded in Xcode, since it wouldn't make sense to use the same provisioning profile for everything.

@chrisfsampaio In my project, my share extension target and app's target using different provisioning profile, how can I use your solution?

@marcuswu0814 The short answer is you can't 😕. Unfortunately xcodebuild doesn't provide an API to manipulate build settings per target.
If you want to "manually" manage code signing, you need to provide the correct provisioning profile names for each target in Xcode on the Build Settings tab.
You can also use match if you're working within a team. You can find a lot of useful information in the Code Signing Troubleshooting Guide.

This is all actually described in https://docs.fastlane.tools/codesigning/GettingStarted/, in the Xcode docs it is described how you can set up code signing for multiple targets 👍

I'll close the issue for now, as fastlane's recommended setup is to specify either the profile in Xcode 8, or to use environment variables in Xcode 7 and below

👍

@chrisfsampaio  - thanks for this issue it gave me a workaround to get things working in my situation.  In my case i have 2 targets with 3 different profiles per target in my code.
Using the     xcargos: "PROVISIONING_PROFILE_SPECIFIER='Foo AppStore'" helped things along.

@KrauseFx I have an app that has over 40 instances in the store, now using Xcode 8 this is going to be a nightmare if I can do this programmatically :/

I'm in a similar situation to @RyanCopley. We have 3 targets (app, watch app, watch extension) and use fastlane to build many instances of the same product for clients. Each of these instances have their own bundle ids / provisioning profiles and up until now we have generated different .env files to manage this.
Along with this we were using sigh UDID and environment variables to select the correct provisioning profiles for the targets in the project. And using the Other field to point at the environment variable.
ENV["XX_WATCHKIT_TARGET_PROFILE"] = sigh(
      app_identifier: "#{ENV["APP_IDENTIFIER"]}.watchkitapp",
      filename: "#{ENV["APP_IDENTIFIER"]}.watchkitapp.mobileprovision",
      output_path: ENV["ARTIFACT_OUTPUT_PATH"]
    )

I understand that with Xcode 8 we should be now using the PROVISIONING_PROFILE_SPECIFIER and have successfully migrated to this on a much simpler vanilla project. What is the approach on Xcode 8?
sigh returns the UDID, is there a way to retrieve the profile name from sigh so I can place this in an environment variable.

@danielbowden

sigh returns the UDID, is there a way to retrieve the profile name from sigh so I can place this in an environment variable?

checkout method in https://github.com/fastlane/fastlane/issues/6245

Interesting, thanks @puttin. I was just looking at how match does it internally too.
https://github.com/fastlane/fastlane/blob/master/match/lib/match/runner.rb#L102-L116
It already handles the parsing, via
FastlaneCore::ProvisioningProfile.parse(profile)

Provided I could get the correct profile path to pass to it, I could do.
 parsed = FastlaneCore::ProvisioningProfile.parse(profile)
 name = parsed["Name"]

Though that may be a method better suited to be on provisioning_profile.rb like udid is:
https://github.com/fastlane/fastlane/blob/fa5fc3ef7ef22c9e6b760f0e15b7f2498beb151e/fastlane_core/lib/fastlane_core/provisioning_profile.rb#L37-L39
Is there a safer way to get provisioning profile path than what you have provided?
profiles_path = File.expand_path("~") + "/Library/MobileDevice/Provisioning Profiles/*.mobileprovision"
  profile_paths = Dir[profiles_path]

As per my sigh example in my previous comment above I guess I already have specified the filename and output_path so I could take it from there?

@danielbowden It seems that my Xcode only use profile in the library position.
You can modify the method use your dir to try.

Hey @jeeftor! That sounds exactly like the solution I need for the sticker extensions which has two targets. One main app and one extensions with two different profiles. What did you setup look like?

In line with my comment above I have created a Fastlane action to retrieve the provisioning profile specifier from a provided profile.
I am happy to put this up as an actions PR if others find this useful. There is a gist of it here: https://gist.github.com/danielbowden/7cb4a5c71a0372ea3b7ceeae1c71022a
module Fastlane
  module Actions
    module SharedValues
      GET_PROVISIONING_PROFILE_SPECIFIER_CUSTOM_VALUE = :GET_PROVISIONING_PROFILE_SPECIFIER_CUSTOM_VALUE
    end

    class GetProvisioningProfileSpecifierAction < Action
      def self.run(params)
        profile_path = params[:profile]
        UI.message "Provisioning profile: #{profile_path}"

        parsed = FastlaneCore::ProvisioningProfile.parse(profile_path)
        parsed["Name"]
      end

      #####################################################
      # @!group Documentation
      #####################################################

      def self.description
       "Returns the provisioning profile specifier for the provided provisioning profile"
      end

      def self.details
        "Returns the provisioning profile specifier for the provided provisioning profile"
      end

      def self.available_options
        [
          FastlaneCore::ConfigItem.new(key: :profile,
                                       env_name: "FL_GET_PROVISIONING_PROFILE_SPECIFIER_PROFILE_PATH",
                                       description: "The path to a provisioning profile",
                                       optional: false,
                                       verify_block: proc do |value|
                                          UI.user_error!("Couldn't find provisioning profile at path `#{value}`") unless File.exist?(value)
                                       end
                                      )
        ]
      end

      def self.output
        [
          ['GET_PROVISIONING_PROFILE_SPECIFIER', 'The corresponding provisioning profile specifier']
        ]
      end

      def self.return_value
        "The returned provisioning profile specifier is to be used in an environment variable for Xcode 8+ to use in order to select the correct provisioning profile"
      end

      def self.authors
        ["danielbowden"]
      end

      def self.is_supported?(platform)
        platform == :ios
      end
    end
  end
end

I have run this on several different projects using the Xcode 8 build tools. The projects have 3 targets (app, watch app, watch extension) and I've verified the embedded provisioning profile in each of these packaged payloads is as expected.
My Xcode 8 project specifies each of the provisioning profiles using environment variables similar to the way it was done in Xcode 7, though this time it uses the name specifier. See screenshot

My Fastfile uses the profile retrieved via sigh and passes it to my action get_provisioning_profile_specifier profile: <profile_path>
sigh(
      app_identifier: "#{ENV["APP_IDENTIFIER"]}.watchkitapp",
      filename: "#{ENV["APP_IDENTIFIER"]}.watchkitapp.mobileprovision",
      output_path: ENV["ARTIFACT_OUTPUT_PATH"]
)

ENV["AS_WATCHKIT_TARGET_PROFILE"] = get_provisioning_profile_specifier profile: File.join("#{ENV["ARTIFACT_OUTPUT_PATH"]}", "#{ENV["APP_IDENTIFIER"]}.watchkitapp.mobileprovision")

UI.message "ENV AS_WATCHKIT_TARGET_PROFILE: #{ENV["AS_WATCHKIT_TARGET_PROFILE"]}".blue

I actually wrote up what I ended up doing in a medium article<EMAIL_ADDRESS>Basically what I wrote here didn't actually end up working for me.  My approach was to actually manually rewrite the project file and use manual signing.  But it works well.

@jeeftor Thanks so much for sharing. Really glad to have you as part of our community 👍

@jeeftor what's the reason you're not using environment variables as described here? This way you don't have modify and revert the project file.

That is weird that when I configured
"xcargs "PROVISIONING_PROFILE_SPECIFIER='#{ENV["PROVISIONING_PROFILE_SPECIFIER"]}'""
in Gymfile, it build forever with Jenkins and take me 5h11m in local (running in my laptop) to build app.
Gymfile:
output_directory "./build/"
output_name "360"
include_symbols "false"
include_bitcode "false"
codesigning_identity  (ENV["CODE_SIGN_IDENTITY"])
# provisioning_profile_path ENV["SIGH_OUTPUT_PATH"]
xcargs "PROVISIONING_PROFILE_SPECIFIER='#{ENV["PROVISIONING_PROFILE_SPECIFIER"]}'"
configuration "Fastlane"

Fastlane: 1.105.3

@KrauseFx @TKBurner Is there any way to specify the target when I set provisioning profile with provisioning_profile_specifier? Because I got extension in side my project, each target should be set with one unique provisioning profile, separately.
below is my gym content, I got it working for those projects without multiple targets. And I already try the action update_project_provisioning, which didn't work for Xcode 8 anymore.
defined = output+" PROVISIONING_PROFILE_SPECIFIER='#{ENV["BUILD_PROVISIONING_PROFILE_TARGET"]}' DEVELOPMENT_TEAM='$(sigh_#{options[:app_identifier]}_adhoc_team-id)'"
    base_gym(
      configuration: "Release",
      user_defined: defined,
      postfix: postfixString,
      workspace: options[:workspace],
      xcodeproj: options[:xcodeproj],
      scheme: options[:scheme],
      package_name: options[:package_name],
      export_method: "ad-hoc",
      codesigning_identity: "#{ENV["BUILD_CODESIGNING_IDENTITY"]}"
      )

@zake1989 I think you'll need to setup manual code signing in Xcode for your project and then it'll work properly with gym. Let me know how it goes!

@zake1989 @TKBurner If you are doing multiple build steps and need to sign a project in a CI environment before you can pass it to gym, use this: https://gist.github.com/vbresults/9da2b42e0fabf111a100e05c502aba98


Added info about VS for Mac

Thanks for the info.
There are a couple of minor changes I would like to be made before merging:

typo "Duppressor"
make it a "##" heading and move it to after the "Visual Studio" paragraph
change the "Visual Studio" heading to "Visual Studio (for Windows)"

I would not bother you with these if I could make the changes myself.
Could you please check if you enabled "allow changes from maintainer" on this PR?
Or tell me how to add commits to the PR otherwise?


Ease 2-3 transition by asserting on utf8/os mismatch

This will help make sure people creating a PathBuf will set AllowInvalidUtf8 on the arg.
This will also make it so we can work towards https://github.com/clap-rs/clap/issues/2683 without breaking compatibility

If someone needs to generically operate on ArgMatches, we can always add a raw data accessor in the future.


Passing arguments to docker run

Hey!
I have a question, that may sound dumb, but who knows. I am looking for a way to pass command line arguments to docker run when starting a container. Is that possible right now?
Thanks in advance.

Assuming you are referring to function containers and not the service continer, Fn doesn't let  you specify arbitrary docker arguments, typically these will be exposed via features of Fn itself.
Fn aims to make functions portable, scalable, secure and consistent across a large range of environments,  (including, eventually environments that aren't running docker (e.g. Kubernetes, clear containers etc.)).
In order to do that we have to restrict the means by which functions  can become specific to their environments by definition - unfortunately adding arbitrary docker commands would allow functions to (e.g.) mount containers and open ports which in turn could compromise the portability of those functions.

If you are interested in adding a specific feature here please say though  - there are a lot of features that can feasibly be added without exposing docker args to users.

the last question has been answered: https://stackoverflow.com/questions/51441703/how-to-change-driver-from-docker-to-mocker/51565330#51565330 -- as far as hooking that up to fn run, that is a whole can of worms we can open on the CLI repo if we get there :)
our stance is as stated, we can add features for certain things but it's not easy for us to add arbitrary flags around docker run in function invocations for the service itself (as we're not shelling out to docker run). thanks!


🏷 Add pipeline debugging

✨ What's this?
This adds (optional) debug groups to our pipeline and object labels to pipeline textures and render targets.
🔍 Why do we want this?
This makes debugging in tools like RenderDoc significantly easier since it is much easier to find specific draw calls and to generally verify that intended rendering steps are happening, and in what order.
🏗 How is it done?
Using:
https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glPushDebugGroup.xhtml
https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glPopDebugGroup.xhtml
https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glObjectLabel.xhtml
See also:
https://www.khronos.org/opengl/wiki/Debug_Output
💥 Breaking changes
New optional parameters are binary breaking, but the changes are code compatible.
🔬 Why not another way?
The used GL API is the way to do this, and using the existing pipeline functionality felt natural.
Perhaps having two different patterns to give labels for textures and rendertargets (first vs last/optional parameter) isn't perfect, but I'm happy to change that signature down the line. I think it's ok for now.
🦋 Side effects
I was able to find a rendering bug much faster and easier.
💡 Review hints
I tested this with RenderDoc and TD, where you can also see how these new APIs are used: https://github.com/beardgame/td/commit/c80add24a8d42ba8e1495803aaa1c49d2269465e

Yap. Had to make it more complicated because Apple (and perhaps others?) doesn't support the needed extension. Happy to hear thoughts on the pattern used in that last commit.


Condense the file_map and file_stats_map

The stats for priority, access times, and size can be stored in the file_map alongside the files themselves.
This will simplify the implementation, while probably not having a significant impact on performance.

Implemented.


Interrupt latency using the vectored feature (especially Xtensa)

With these two non-scientifical tests
Vectored
#![no_std]
#![no_main]

use core::cell::RefCell;

use critical_section::Mutex;
use esp32s3_hal::{
    clock::ClockControl,
    interrupt::{self},
    peripherals::{self, Peripherals},
    prelude::*,
    system::{SoftwareInterrupt, SoftwareInterruptControl},
    Delay,
};
use esp_backtrace as _;

static SWINT: Mutex<RefCell<Option<SoftwareInterruptControl>>> = Mutex::new(RefCell::new(None));

#[entry]
fn main() -> ! {
    let peripherals = Peripherals::take();
    let system = peripherals.SYSTEM.split();
    let sw_int = system.software_interrupt_control;
    let clocks = ClockControl::boot_defaults(system.clock_control).freeze();

    critical_section::with(|cs| SWINT.borrow_ref_mut(cs).replace(sw_int));

    interrupt::enable(
        peripherals::Interrupt::FROM_CPU_INTR0,
        interrupt::Priority::Priority3,
    )
    .unwrap();

    let mut delay = Delay::new(&clocks);
    loop {
        delay.delay_ms(500u32);
        critical_section::with(|cs| {
            unsafe {
                CLCK = esp32s3_hal::xtensa_lx::timer::get_cycle_count();
            }
            SWINT
                .borrow_ref_mut(cs)
                .as_mut()
                .unwrap()
                .raise(SoftwareInterrupt::SoftwareInterrupt0);
        });
    }
}

static mut CLCK: u32 = 0u32;

#[interrupt]
fn FROM_CPU_INTR0() {
    let diff = unsafe { esp32s3_hal::xtensa_lx::timer::get_cycle_count() - CLCK };
    esp_println::println!("diff = {}", diff);
    critical_section::with(|cs| {
        SWINT
            .borrow_ref_mut(cs)
            .as_mut()
            .unwrap()
            .reset(SoftwareInterrupt::SoftwareInterrupt0);
    });
}

Non-vectored
#![no_std]
#![no_main]

use core::cell::RefCell;

use critical_section::Mutex;
use esp32s3_hal::{
    clock::ClockControl,
    interrupt::{self},
    peripherals::{self, Peripherals},
    prelude::*,
    system::{SoftwareInterrupt, SoftwareInterruptControl},
    Delay,
};
use esp_backtrace as _;

static SWINT: Mutex<RefCell<Option<SoftwareInterruptControl>>> = Mutex::new(RefCell::new(None));

#[entry]
fn main() -> ! {
    let peripherals = Peripherals::take();
    let system = peripherals.SYSTEM.split();
    let sw_int = system.software_interrupt_control;
    let clocks = ClockControl::boot_defaults(system.clock_control).freeze();

    critical_section::with(|cs| SWINT.borrow_ref_mut(cs).replace(sw_int));

    unsafe {
        interrupt::disable(esp32s3_hal::get_core(), peripherals::Interrupt::SDIO_HOST);

        interrupt::map(esp32s3_hal::get_core(), peripherals::Interrupt::FROM_CPU_INTR0, interrupt::CpuInterrupt::Interrupt23LevelPriority3);
        esp32s3_hal::xtensa_lx::interrupt::enable_mask(
            esp32s3_hal::xtensa_lx::interrupt::get_mask() | 1 << interrupt::CpuInterrupt::Interrupt23LevelPriority3 as u32,
        );

    }

    let mut delay = Delay::new(&clocks);
    loop {
        delay.delay_ms(500u32);
        critical_section::with(|cs| {
            unsafe {
                CLCK = esp32s3_hal::xtensa_lx::timer::get_cycle_count();
            }
            SWINT
                .borrow_ref_mut(cs)
                .as_mut()
                .unwrap()
                .raise(SoftwareInterrupt::SoftwareInterrupt0);
        });
    }
}

static mut CLCK: u32 = 0u32;

#[no_mangle]
extern "C" fn level3_interrupt() {
    let diff = unsafe { esp32s3_hal::xtensa_lx::timer::get_cycle_count() - CLCK };
    esp_println::println!("diff = {}", diff);
    critical_section::with(|cs| {
        SWINT
            .borrow_ref_mut(cs)
            .as_mut()
            .unwrap()
            .reset(SoftwareInterrupt::SoftwareInterrupt0);
    });
}

I get these results
with saving floats
vectored = 1111 cycles
non-vectored = 214
w/o saving floats
vectored = 1096
non-vectored = 199
w/o saving floats, w/o spilling registers
vectored = 1043
non-vectored = 146
Latency is more than five times with vectoring enabled.
While it's more or less possible to use vectored and non-vectored interrupts together on RISC-V that is not possible in a sane way for Xtensa currently. This hurts esp-wifi a lot but also hurts async performance.

While it's probably still worth to optimize things, regarding esp-wifi on Xtensa this might be a red herring.
Replacing/adding this to the top of handle_interrupts should be the most naive and fasted dispatching to the handlers:
        if level == 1 {
            if interrupt::get() & (1 << 6) != 0 {
                interrupt::clear(1 << 6);
                xtensa_lx_rt::Timer0(1, save_frame);
            }
        }

        if level == 3 {
            if interrupt::get() & (1 << 29) != 0 {
                interrupt::clear(1 << 29);
                xtensa_lx_rt::Software1(3, save_frame);
            }
        }

        let status = get_status(crate::get_core());
        if status == 0 {
            return;
        }

        if level == 1 {
            if status & (1 << Interrupt::WIFI_MAC as u128) != 0 {
                handle_interrupt(1, Interrupt::WIFI_MAC, save_frame);
            }
            if status & (1 << Interrupt::WIFI_PWR as u128) != 0 {
                handle_interrupt(1, Interrupt::WIFI_PWR, save_frame);
            }
        }

        if level == 2 {
            if status & (1 << Interrupt::TG1_T0_LEVEL as u128) != 0 {
                handle_interrupt(2, Interrupt::TG1_T0_LEVEL, save_frame);
            }
        }

        if level == 3 {
            if status & (1 << Interrupt::FROM_CPU_INTR0 as u128) != 0 {
                handle_interrupt(3, Interrupt::FROM_CPU_INTR0, save_frame);
            }

            if status & (1 << Interrupt::TG0_T0_LEVEL as u128) != 0 {
                handle_interrupt(3, Interrupt::TG0_T0_LEVEL, save_frame);
            }
        }

        return;

but using this with esp-wifi doesn't improve the benchmark results (seems it gets even worse)

Do you see the same results on the esp32 or esp32s2? In my testing is see much worse results on the s3, which makes me think that we have a general xtensa issue, and a esp32s3 specific issue. Maybe this might help the other chips?

Maybe my results were not correct yesterday - testing again today using my Android phone as the bench-server (https://github.com/bjoernQ/android-tcp-bench-server) shows a clear indication that at least the download rates get much better,
Original
connecting to Address([192, 168, 12, 44]):4321...
connected, testing...
download: 676 kB/s
Testing upload...
connecting to Address([192, 168, 12, 44]):4322...
connected, testing...
upload: 649 kB/s
Testing upload+download...
connecting to Address([192, 168, 12, 44]):4323...
connected, testing...
upload+download: 533 kB/s
Testing download...
connecting to Address([192, 168, 12, 44]):4321...
connected, testing...
download: 667 kB/s
Testing upload...
connecting to Address([192, 168, 12, 44]):4322...
connected, testing...
upload: 645 kB/s
Testing upload+download...
connecting to Address([192, 168, 12, 44]):4323...
connected, testing...

With the optimized/hacked interrupt handling
Testing download...
connecting to Address([192, 168, 12, 44]):4321...
connected, testing...
download: 931 kB/s
Testing upload...
connecting to Address([192, 168, 12, 44]):4322...
connected, testing...
upload: 651 kB/s
Testing upload+download...
connecting to Address([192, 168, 12, 44]):4323...
connected, testing...
upload+download: 519 kB/s
Testing download...
connecting to Address([192, 168, 12, 44]):4321...
connected, testing...
download: 916 kB/s
Testing upload...
connecting to Address([192, 168, 12, 44]):4322...
connected, testing...
upload: 653 kB/s
Testing upload+download...
connecting to Address([192, 168, 12, 44]):4323...
connected, testing...
upload+download: 529 kB/s
Testing download...
connecting to Address([192, 168, 12, 44]):4321...
connected, testing...
download: 919 kB/s
Testing upload...
connecting to Address([192, 168, 12, 44]):4322...
connected, testing...

Just for reference (on my network), current main with an esp32:
Got IP: <IP_ADDRESS>/24
Testing download...
connecting to Address([192, 168, 0, 24]):4321...
connected, testing...
download: 943 kB/s
Testing upload...
connecting to Address([192, 168, 0, 24]):4322...
connected, testing...
upload: 771 kB/s
Testing upload+download...
connecting to Address([192, 168, 0, 24]):4323...
connected, testing...
upload+download: 616 kB/s

ESP32S3 (powered through uart, not USB serial jtag):
Testing download...
connecting to Address([192, 168, 0, 24]):4321...
connected, testing...
download: 562 kB/s
Testing upload...
connecting to Address([192, 168, 0, 24]):4322...
connected, testing...
upload: 387 kB/s
Testing upload+download...
connecting to Address([192, 168, 0, 24]):4323...
connected, testing...
upload+download: 311 kB/s

Both with the following configuration:
[esp-wifi]
rx_queue_size = 32
tx_queue_size = 6
static_rx_buf_num = 16
dynamic_rx_buf_num = 32
ampdu_rx_enable = 0
ampdu_tx_enable = 0
rx_ba_win = 32
max_burst_size = 6
heap_size = 100000

I feel like there must be some kind of CPU/Flash/Cache issue with the S3 to see this much of a difference, and maybe the more optimized interrupt you've been handling masks this issue a bit because it's less CPU bound :thinking:.

Is is possible that one of the critical code path is not in IRAM for the ESP32-S3? I pretty much discovered the same thing, interrupt (and thus embassy) is very slow

Is is possible that one of the critical code path is not in IRAM for the ESP32-S3? I pretty much discovered the same thing, interrupt (and thus embassy) is very slow

Cache misses would definitely increase latency a lot - #1169 should make a huge difference

@bjoernQ I already enabled 32K ICACHE in my fork months ago, but latencies are still unpredictable...

I decided to look at this more in general and less in the direction of esp-wifi again.
My test setup is as this: have an GPIO interrupt trigger at falling edge of the boot button. The handler just toggles GPIO 2. Attached a logic analyzer and configured accordingly (400MHz sample rate, setting the right trigger) to measure the time from the falling edge of the boot button to the toggle of GPIO2. This measures a lot more than just the interrupt latency but should be good enough.
To get reasonable results things called by the user-handler is moved to RAM - for critical-section it was done by cloning the repo locally. (But could and should be done via linker-script like outlined at the end of this text)
I again checked how much the Xtensa register spilling changes things. In my experiments it's 0.5us. Sounds a lot but compared to the overall latency it's not really much.
First tested ESP32-S3
The first triggering gives 62.5us (!!!). From the second triggering on, it's down to 14.5us. Since besides the interrupts it's just running an empty loop it seems that the first triggering ran into cache misses. In a real application, things will be much worse (i.e. every interrupt invocation could see such latency)
Let's verify: set flash frequency to 20 MHz and .... the first triggering is 110us, after that it's 14.5 again. So, it is flash access, definitely.
I changed HAL code to just call the user-handler directly in __level_2_interrupt and latency went down to 5.1us also for the first invocation. So, the code dispatching the interrupt is accessing flash. (Changed it back for the next tests)
We assumed RISC-V is fine, let's check ESP32-C3
First invocation is 58.3us, then it stays at 18us. Seems like the same problem, then. Double checking with 20MHz flash frequency shows 98.5us for the first invocation, then 18us.
Looking at the dis-assembly of the vectoring code revealed it's also calling code in flash

core::panicking::panic
core::panicking::panic_bounds_check
memset
memcpy

We can get rid of the panicking stuff via panic-immediate-abort. In theory memset and memcpy should be weak symbols. I tried to provide these functions in ROM but it just didn't work. 🤷♂
What worked for ESP32-C3 was omitting compiler-builtins from .text ( *(EXCLUDE_FILE (*compiler_builtins*.o) .literal .text .literal.* .text.*) ) and include it in .rwtext ( *compiler_builtins*.o(.literal .text .literal.* .text.*) )
That together with panic-immediate-abort gets the first triggering on C3 down to 22.9us. Still more than the next invocations but from the disassembly there shouldn't be calls to code in flash. 🤔
On ESP32-S3 gnu-ld ignored not placing compiler-builtins in .text - what seems to work there is
    EXCLUDE_FILE (*compiler_builtins*.o)
    *(.literal .text .literal.* .text.*)

(but that in turn wasn't working for lld)
On S3 it gets down to 25.8us for the first invocation, then down to 14.5
So, there is still some flash access! Probably not code, but data. Looking at the assembly again showed some access to DROM addresses.
Probably having an option to place compiler-builtins in RAM would be a good thing no matter what. We should also have an option to place the code for critical-section in RAM.
We need to investigate what is actually accessed in DROM and what we can do about it.
We could also rewrite the vectored interrupt dispatching in assembly but that's quite some effort and harder to maintain

So, I looked a bit into the DROM access.
One easy thing is CPU_INTERRUPT_LEVELS - the const array is placed in DROM and seems nothing can be done about it when const apparently. (link_section attribute is not possible for const)
What worked was making it static and specifying the `#[link_section = ".rwtext"]``
But that is not enough.
It seems that at least switch-tables are placed in .rodata always.
I don't see a way to completely avoid hitting flash currently. (Me not seeing it doesn't mean there isn't a way)
Falling back to (inline) assembly would give us full control and is the only option I see right now. Besides the downside of having less maintainable code it's quite probable that user code will hit the flash when we call it. But might still we worth it just for the async handlers we include in the HAL (if we can make them flash access free)
However, it's at least possible to cut down the worst-case latency with the (moderate) changes described in the comment above
UPDATE: I changed the order of text+rodata and rwtext+rwdata in esp32s3.x and let .data include *(.rodata..Lswitch.table.*)  which gets me down to 18.55us (on ESP32-S3) when triggering the interrupt for the first time - there is still flash data access but it got slightly better (but sacrifice a chunk of RAM)

Thank you for looking into this @bjoernQ! Really helpful!
I think we want to avoid writing this all in assembly, I think later we should open up the option for "high" level interrupts written in asm, but in general, I think we want to keep our normal interrupt handling in Rust. I looked around the esp forums for interrupt latency topics, I saw values from 2us to 15us, at least we're heading this way now, and going from 62us to 15 is a big win!
I think some actionable items from your findings would be:

[ ] Find a way to use ROM memset/memcpy - if it can't work, we should ensure memcpy and memset are in RAM somehow
[ ] Replace consts this static's if they're referenced in interrupt handling code
[ ] Find a way to get LLVM to either

Put jump tables in RAM
Don't emit jump tables for specific functions
Linker magic to place them in RAM on a best effort basis

@bjoernQ, if possible, would you be able to your benchmarking code, either in a gist or whatever is more suitable so we can have a reference when we begin to implement these things?
cc: @sethp, I know its been a while since you've poked your head into esp-land, but I noticed you made this post: https://discourse.llvm.org/t/moving-jump-tables-sections-on-embedded-systems/70226 and I was wondering if you made any headway in your project.

Here is the code for what I used to benchmark: https://gist.github.com/bjoernQ/4bc8236b926803e6aa22960880af70cd
it's good to use a logic-analyzer which supports a reasonable high sample rate (I used DSLogic Pro at 400MHz)

For not emitting jump-tables there is
"-Z", "no-jump-tables", - RFC: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/no-jump-tables.html
Currently it seems there is no support for doing that on a function level: https://github.com/rust-lang/rust/blob/master/compiler/rustc_middle/src/middle/codegen_fn_attrs.rs

Wow, awesome work @bjoernQ ! We never got as precise as measuring latency in seconds, just by proxy (i.e. whether or not our output was "flickering" when we were blowing our real-time budget on cache misses). It's very cool to see these numbers, and thank you for sharing the benchmark! Your analysis seems like it matches more or less what I found working on #534 : flash is cache thrash, and rustc loves to sneak in various accesses.

Currently it seems there is no support for doing that on a function level: https://github.com/rust-lang/rust/blob/master/compiler/rustc_middle/src/middle/codegen_fn_attrs.rs

Yeah, that's a LLVM limitation, I believe: I'm not quite sure what optimization pass is producing those jump tables, but I couldn't find a way to configure it at a more granular level than global. The unfortunate part is that setting that in the build flags for esp-hal does nothing for any consumers of the crate, IIRC: the build flags aren't carried over.
There is -C llvm-args=--max-jump-table-size=0 you could play with (or, possibly, --min-jump-table-entries= but that was ignored when I last tried it). A half-baked idea I had was to set that threshold just below whatever size start_trap_rust_hal would emit, since it's a fairly large number, but that just would've moved the problem for us to our handler.
There's also a very suspiciously named "jump tables in function section".... thing, down at the LLVM level; I didn't fully trace that through to identify where/how to turn it on, but this signature suggests it's also configurable per-function: https://github.com/llvm/llvm-project/blob/9a9aa41dea83039154601082b1aa2c56e35a5a17/llvm/include/llvm/CodeGen/TargetLoweringObjectFileImpl.h#L83-L84 (though maybe not at the rust source level).

I don't see a way to completely avoid hitting flash currently. (Me not seeing it doesn't mean there isn't a way)

The results I found were similar to yours; I thought there might be a path through LLVM, which leads me to:

cc: @sethp, I know its been a while since you've poked your head into esp-land, but I noticed you made this post: https://discourse.llvm.org/t/moving-jump-tables-sections-on-embedded-systems/70226 and I was wondering if you made any headway in your project.

Not the ELF post-processor or LLVM change[^1], but I did find a way to route around it. I tried a few different rust source-level constructs to avoid the jump table, but the optimizer was too clever. That did lead me to the realization that we already have implemented a jump table at the rust source level, though: peripherals::__EXTERNAL_INTERRUPTS! And, since that's a static, #[link_section = "..."] works just fine for section placement.
So changing the troublesome match from:
match code {
1 => interrupt1(...),
...
}

to something like:
HANDLER_VECTOR[code](...) // given HANDLER_VECTOR[1] == interrupt1 , etc.

ought to avoid the whole jump table placement question entirely.[^2] Populating HANDLER_VECTOR is another story, as is bounds-checking, but that seemed the most viable path forward to me at the time.
[^1]: I didn't pursue it because realizing that change in our downstream project would have been a lot of work, but perhaps the situation is different for Xtensa? Teaching the TargetLoweringObjectFileELF to emit its jump tables into .rodata.jtable or something that can be targeted with a linker script should be straightforward, it's this one line that needs to change.
[^2]: Keen eyed readers may note that looks an awful lot like the hardware's interrupt vector, modulo some trampolining. That's what rust-lang/rust#111891 was all about: avoiding start_trap_rust_hal entirely by slotting extern "riscv-interrupt-m" fn interruptN into the hardware vector directly. I have a prototype laying around on a disk somewhere that shows with some weak symbols and a little more finessing with #[inline] it's possible to avoid saving more registers than the entire handling path uses, which can dramatically lowers not just interrupt-to-entry latency but end-to-end interrupt handling latency overall for simple handlers (like something that queues async work for later?). If you'd like, I can dig up the details: unfortunately, I don't think I wrote down the path from #111891 through to the HAL anywhere public, but I should have some notes on it lying around.

Had another look at this.
"Replace consts with static's if they're referenced in interrupt handling code" doesn't have a huge impact but increases the RAM usage - we probably don't want to do this by default (would mean a new feature or we need the build-time configuration)
Not emitting jump tables has some impact on the initial invocation but impacts performance in a negative way overall. Also, it's something only the binary crate can configure.
Moving jump tables etc. to RAM might be a good option but again, increases RAM usage which should be an opt-in / opt-out but we shouldn't just do it unconditionally.

I will definitely get back to this once we have a sane way to do fine grained configuration by the user (i.e. not hundreds of Cargo feature 😄)

I wonder how hard it is to make a feature that puts the jump table in RAM? I am trying to build a Rust-based flight controller and kept bumping into unexpected latencies because I am running the two cores on the ESP32-S3 almost in full, so as @sethp said I have huge cache pressure. Since we have 4bytes for each interrupt and total 128 interrupts max, a 512 byte RAM price is a great deal for deterministic latency.

Such functionality wouldn't be too hard to do via the linker script. It's just that we currently try to avoid introducing new features but want that build-time config system

Going to close this in favor of #1747


feat: byte to 64

Pull Request Test Coverage Report for Build 3536
Details

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.007%) to 82.282%

Totals

Change from base Build 3532:
0.007%

Covered Lines:
4189

Relevant Lines:
5091

💛  - Coveralls


Idea submission form

Add an idea submission form to the website which emails out the team or sends a message to slack.
"Hey you've got an idea to make Fort Collins better? Let us know your thoughts."

Instead of email or Slack, what if the output could be a GitHub issue or a comment on https://github.com/CodeForFoco/org/issues/17? I guess it would require an account to post as, but just a thought.

@abettermap I really like that idea.
Email is simple to implement, we could just use FormSpree.
Slack can be simple if we use a tool like this. Alternatively, building our own integration isn't too difficult, it just would require a backend API to securely hold our Slack API keys.
Your Github suggestion is my favorite. The first solution that comes to mind, create an account (e.g. CFFC-Bot) and use it to create issues / comments via the API, create a backend API to securely hold our Slack API keys.

Yayuh, I like it. Let's set this up Thursday at the meeting if we have time.

Hello,
I did a little research on this, and, according to me, it is not possible to use the GitHub API because for that, we need to use a library like Octokit (https://github.com/octokit/octokit.rb) which is the library suggested by GitHub to use with Ruby (https://developer.github.com/libraries/).
Using this library means that we should be able to execute ruby code on the server, and that is not possible. At least not in GitHub.
FormBot looks nice and easy to do. I'm going to work on it.
Thank you.

Ah, bummer. Would this be an option? https://coderwall.com/p/8lq1ba/how-to-create-a-contact-form-for-a-github-pages-served-jekyll-website
Also, do you have a link to FormBot? I only found a Chrome extension which looks like it automates form populating rather than building: https://chrome.google.com/webstore/detail/formbot/ilnjaomgpkbmapjachcppgabkidbmnjc?hl=en

Hello,
This is the link: https://stacktodo.com/tools/form

Yeah definitely not what I was looking at haha. That looks promising so I'd say it's certainly an option, assuming the Heroku thing from my link isn't worth it or isn't feasible (it does sound kinda overkill to me).
A similar option would be Google Forms, which would let us store the ideas in a spreadsheet. I guess it depends what we plan on doing with the ideas later. Slack = better for discussion, Google Sheets = easier storage and analysis.
Another bonus to the Google approach is that the output could be public, which seems like something we/they would want. We could provide a link to the spreadsheet below the form saying "View current ideas" or something, to maybe give users additional ideas or to see if theirs has already been suggested.

Using Heroku sound little overkill indeed but I think it is the only way to execute code.
Maybe, based on that article, then is possible to call GitHub API from Heroku and be able to create issues / comments from the website...

We have a free tier Heroku instance running Node app for our Slack invites.

Repo
Live Site

We could add another API endpoint to handle idea submissions.
Might be overkill but could be fun.

If we do go this route we could set up a CORs policy for the heroku domain and the node-github library.

cors
node-github

@afibarra Sounds like we are going to try the Heroku thing. Do you want me to assign this issue to you?
@JordanMajd Whoever is doing this would need Heroku credentials, yeah?

@abettermap @JordanMajd Yes, assign it to me
Actually, I'm think I finished it. But it is a java application, not a Node.js one. I don't have much time to learn Node.js :). Even though, it took me some time to figure it out how to deploy a Java EE app in Heroku, because Heroku only runs Java SE applications.
I tested the app against one of my public GitHub repositories.
I tested using curl as follows: curl --data "issueTitle=New Issue1&userComment=User Comment1" http://salty-journey-24179.herokuapp.com/createIssue
And this is what you can see in the Heroku server log file:
2017-02-12T06:12:04.151234+00:00 app[web.1]: 2017-02-12 06:12:04,151 INFO  [com.jcabi.http.request.BaseRequest] (default task-1) #fetch(POST api.github.com /repos/afibarra/afibarra.github.io/issues): [201 Created] in 489ms
2017-02-12T06:12:04.238222+00:00 app[web.1]: 2017-02-12 06:12:04,237 INFO  [com.jcabi.http.request.BaseRequest] (default task-1) #fetch(GET api.github.com /repos/afibarra/afibarra.github.io/issues/7): [200 OK] in 70ms
According to me it is possible to transfer the ownership of a GitHub repo and a Heroku App as well to other person, in this case, you of course.
Thank you,
Armando

Nice work! We will want to include a label of some sort as well, probably project idea, and maybe use some kind of comment template to present the info neatly (including the idea's category or other meta stuff). We might also need a way to contact the submitter without exposing their email publicly, assuming they want to be contacted.
I will defer to @JordanMajd for the Heroku stuff and repo transfer as I know nothing about it, but it sounds like the Java approach is a nice solution.
I don't think I can assign it to you if you're not a repo member but it's all yours.

@afibarra good work!
I'll look into what it takes to transfer the repository and the heroku instance.

The issue submission backened is done thanks to @afibarra and can be found here.
The next steps are:

Create gh account to use as submission bot
Deploy backend to heroku
Create front end form on website.

Closed.
The backend can be found here. It still needs to be deployed but that is an issue tracked in it's own repo.


"mouseenter" not triggering the "mouseEnter" function of the component

Hi,
I am trying to run a ember acceptance test and started encountering the issue when I tried to execute the command
triggerEvent('#my-tag-1', 'mouseenter')
The component associated with the element has the function
mouseEnter() {
  console.log('mouse entered');
}

When I am running the program in development environment, the event is getting triggered and the console is also getting logged. But when running ember test it fails to trigger the mouseEnter event of the component.
I have double checked that element with id #my-tag-1 exists on DOM while running the tests. But still when triggering the mouseenter event, the component is not capturing it.
my ember version and node version are
ember-cli: 2.9.1
http_parser: 2.7.0
node: 4.5.0
v8: <IP_ADDRESS>
uv: 1.9.1
zlib: 1.2.8
ares: 1.10.1-DEV
icu: 56.1
modules: 46
openssl: 1.0.2h
os: darwin x64
v4.5.0

This was working fine when I was working with ember#2.4.6 and ember-qunit#1.4.0, but started to break after I upgraded to ember#2.8-lts and ember-qunit#3.0.1

After playing around using Ember.$ and Jquery's trigger the mouseEnter function of the component is getting executed from the acceptance test, not sure why triggerEvent is not working

Can you share your solution with Ember.$

I'm seeing this issue too. I still think triggerEvent should be the appropriate solution, so I'd like to understand why this doesn't work.

I think this is probably the wrong place for the issue, however. The method is actually in Ember itself: https://github.com/emberjs/ember.js/blob/26a8b5bb022a63e57751259e65c6088ee3645d1a/packages/ember-testing/lib/helpers/trigger_event.js

My best guess right now is that this doesn't work because the mouseenter event is not trusted.

I think this is probably the wrong place for the issue, however.

Yeah, since the helper isn't actually defined here it may be best to move the issue to the Ember repo itself. You could start by adding a failing test here.
It would be good to know if this issue manifests itself when using ember-mocha as well (I assume it does).

There's a neat blog post by Marcellin that can circumvent this.
http://www.marcellin.me/blog/acceptance-testing-bh/

@tchan that's essentially how we worked around it. Thanks :)


Uncaught TypeError: Cannot read property 'init' of undefined

Hi,
I'm trying to use this library for the first time in my react app and I have the error message below when I click on the button "Pick a File":
Uncaught TypeError: Cannot read property 'init' of undefined
    at t.r.initClient (filestack-react.js:1)
    at r.onClickPick (filestack-react.js:1)
    at HTMLUnknownElement.callCallback (react-dom.development.js:100)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:138)
    at Object.invokeGuardedCallback (react-dom.development.js:187)
    at Object.invokeGuardedCallbackAndCatchFirstError (react-dom.development.js:201)
    at executeDispatch (react-dom.development.js:461)
    at executeDispatchesInOrder (react-dom.development.js:483)
    at executeDispatchesAndRelease (react-dom.development.js:581)
    at executeDispatchesAndReleaseTopLevel (react-dom.development.js:592)
    at forEachAccumulated (react-dom.development.js:562)
    at runEventsInBatch (react-dom.development.js:723)
    at runExtractedEventsInBatch (react-dom.development.js:732)
    at handleTopLevel (react-dom.development.js:4472)
    at batchedUpdates$1 (react-dom.development.js:16537)
    at batchedUpdates (react-dom.development.js:2131)
    at dispatchEvent (react-dom.development.js:4551)
    at interactiveUpdates$1 (react-dom.development.js:16592)
    at interactiveUpdates (react-dom.development.js:2150)
    at dispatchInteractiveEvent (react-dom.development.js:4528)

I am simply adding the following to my application:
import ReactFilestack from 'filestack-react'

const onSuccess = result => {
  console.log('result', result);
};

const options = {
  accept: 'image/*',
  maxFiles: 5,
  storeTo: {
    location: 's3',
  },
};
const apiKey = "MY API KEY"

<form>

Test

 <ReactFilestack
     apikey={apiKey}
      mode="upload"
     options={options}
     onSuccess={onSuccess}
 />
</form>

Is it maybe linked to my version of React ? I'm using create-react-app with react v.16.4.0
Thanks in advance for your help!

@chdecultot Thanks for reporting this issue! It looks like this might be related to the recent release of filestack-js 1.0. It's likely anyone following the readme instructions since that release will have run into issues because the peer dependency was still on a pre-1.0 of filestack-js, and this library needed some patching to accommodate the changes in 1.0.
We just published filestack-react 2.0.0 today, which should now support filestack-js 1.0 and React 16. Can you try upgrading to this new version and confirm if this resolves the issue? And do make sure your peer dependencies are correct as well.

Hi @velveteer,
Thanks a lot!
Now I get another error:
TypeError: File argument is not a valid Blob
    at getFile (index.esm.js:6926)
    at upload (index.esm.js:7633)
    at Client../node_modules/filestack-js/build/browser/index.esm.js.Client.upload (index.esm.js:8601)
    at t.r.initClient (filestack-react.js:1)
    at r.onClickPick (filestack-react.js:1)
    at HTMLUnknownElement.callCallback (react-dom.development.js:100)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:138)
    at Object.invokeGuardedCallback (react-dom.development.js:187)
    at Object.invokeGuardedCallbackAndCatchFirstError (react-dom.development.js:201)
    at executeDispatch (react-dom.development.js:461)
    at executeDispatchesInOrder (react-dom.development.js:483)
    at executeDispatchesAndRelease (react-dom.development.js:581)
    at executeDispatchesAndReleaseTopLevel (react-dom.development.js:592)
    at forEachAccumulated (react-dom.development.js:562)
    at runEventsInBatch (react-dom.development.js:723)
    at runExtractedEventsInBatch (react-dom.development.js:732)
    at handleTopLevel (react-dom.development.js:4472)
    at batchedUpdates$1 (react-dom.development.js:16537)
    at batchedUpdates (react-dom.development.js:2131)
    at dispatchEvent (react-dom.development.js:4551)
    at interactiveUpdates$1 (react-dom.development.js:16592)
    at interactiveUpdates (react-dom.development.js:2150)
    at dispatchInteractiveEvent (react-dom.development.js:4528)

But if I remove the "mode='upload'" prop the filePicker opens up correctly.
I haven't yet tested the rest of the integration.

Yes that's why I was just understanding by reading the filestack-js documentation... actually I didn't pass any file prop, so that's why it was failing.
When using the filepicker it seems to be working perfectly fine, my files are uploaded to my S3 bucket as expected.
Thanks a lot for your help, we can close this issue now.


Night CI 2023-07-13: Failed

action url: https://github.com/spidernet-io/spiderpool/actions/runs/5547523192

There are two problems here:

Unit testing encountered a non-production problem: Error: Codecov: Failed to properly upload:

    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2023-07-13T20:27:57.990Z] ['verbose'] End of uploader: 10091 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v3.1.4/dist/codecov' failed with exit code 255
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Upload to Codecov

An error is reported in e2e because the same subnet is created. The problem looks like this: subnetName := "v6-ss-" + tools.RandomName() is named using timestamps, and ginkgo runs concurrently, resulting in Duplicate name.

 [FAILED] Unexpected error:
      <*errors.StatusError | 0xc00015e640>:
      spidersubnets.spiderpool.spidernet.io "v6-ss-73-592166906" already exists
      {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "spidersubnets.spiderpool.spidernet.io \"v6-ss-73-592166906\" already exists",
              Reason: "AlreadyExists",
              Details: {
                  Name: "v6-ss-73-592166906",
                  Group: "spiderpool.spidernet.io",
                  Kind: "spidersubnets",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 409,
          },
      }
  occurred
  In [BeforeEach] at: /home/runner/work/spiderpool/spiderpool/test/e2e/affinity/affinity_test.go:36


Make proper Arrays.toString() calls instead of .toString, which is im…

…proper for arrays

@ryanmkurtz Fixed!


Running tests

There seems to be tests in the repository, yet no instructions on how to run/use/extend them.
I suggest adding a Makefile or similar that downloads bats and runs the test

nvm! it was in the README, I just missed the whole submodule part


MSSQL migrations failing after first migration after baselining

Bug description
When using MSSQL, the first migration I run after baselining works, however any subsequent ones fail.
I am using prisma on a db with existing data, in hopes of keeping my workflow as simple as possible, I am hoping to avoid having to run a manual data migration. I am using a backup of the existing data in a local instance of Azure SQL Edge (as I'm on an M1). I have the shadowdb set up as copy of the db I am working on.
@janpio as discussed, tagging you here
How to reproduce
npx prisma db pull
npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script > prisma/migrations/init/migration.sql
npx prisma migrate resolve --applied init
npx prisma migrate dev

After which, I make a change to the schema (all the values for this column are "false") and run
npx prisma migrate dev -n var_to_bool_users

which generates the migration below
// migration.sql

/*
  Warnings:

  - Made the column `twoFAEnabled` on table `Users` required. This step will fail if there are existing NULL values in that column.

*/
BEGIN TRY

BEGIN TRAN;

-- AlterTable
ALTER TABLE [dbo].[Users] DROP CONSTRAINT [DF_Users_twoFAEnabled];
ALTER TABLE [dbo].[Users] ALTER COLUMN [twoFAEnabled] BIT NOT NULL;
ALTER TABLE [dbo].[Users] ADD CONSTRAINT [DF_Users_twoFAEnabled] DEFAULT 0 FOR [twoFAEnabled];

COMMIT TRAN;

END TRY
BEGIN CATCH

IF @@TRANCOUNT > 0
BEGIN
    ROLLBACK TRAN;
END;
THROW

END CATCH

This migration runs and is successfully applied to the db. After repeating this process, I see the data is dropped on the shadowdb, the tables all still exist at this point.

Any migrations that run after this, I get this error (or similar) along with all the tables on the shadowdb being dropped.
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db" - SQL Server

Error: P3006

Migration `20220706080149_var_to_bool_users` failed to apply cleanly to the shadow database.
Error:
Cannot find the object "dbo.Users" because it does not exist or you do not have permissions.
   0: migration_core::state::DevDiagnostic
             at migration-engine/core/src/state.rs:250

Expected behavior
I expect subsequent migrations after the first to not error out
Prisma information
Config used
// .env file with fake creds 😄
DATABASE_URL="sqlserver://localhost:1433;database=PROD;user=sa;pwd=bigStrongPwd1234;"
SHADOW_DATABASE_URL="sqlserver://localhost:1433;database=PRODshadow;user=sa;pwd=bigStrongPwd1234;"

// schema.prisma excerpt
datasource db {
  provider          = "sqlserver"
  url               = env("DATABASE_URL")
  shadowDatabaseUrl = env("SHADOW_DATABASE_URL")
}

model Users {
  userID                 Int       @id(map: "PK_Users") @default(autoincrement())
  companyRole            String?   @db.VarChar(150)
  email                  String?   @db.VarChar(150)
  password               String?   @db.VarChar(150)
  salt                   String?   @db.VarChar(150)
  firstName              String?   @db.VarChar(150)
  lastName               String?   @db.VarChar(150)
  regType                String?   @db.VarChar(150)
  tacAccept              String?   @db.VarChar(5)
  tacAcceptDate          DateTime? @db.SmallDateTime
  userActive             String?   @db.VarChar(5)
  dateAdded              DateTime? @db.SmallDateTime
  companyId              Int?
  emailConfirmGUID       String?   @db.VarChar(150)
  emailConfirmDate       DateTime? @db.SmallDateTime
  emailConfirmed         String?   @default("false", map: "DF_Users_emailConfirmed") @db.VarChar(5)
  dashboardStepsComplete String?   @default("false", map: "DF_Users_dashboardStepsComplete_1") @db.VarChar(5)
  lastLoginDate          DateTime? @db.SmallDateTime
  viewRiskMatrix         String?   @db.VarChar(5)
  userGUID               String?   @default(dbgenerated("newid()"), map: "DF_Users_userGUID") @db.UniqueIdentifier
  partnerGUID            String?   @db.VarChar(150)
  twoFAEnabled           String?   @default("false", map: "DF_Users_twoFAEnabled") @db.VarChar(5)
}

Environment & setup

OS: MacOS
Database: MSSQL
Node.js version: 17.4

Prisma Version
Tried this on both 3.15 and 4.00

To isolate the problem, and as you are working with a backup of your actual database (which is a great practice - never do this with the live system!) I suggest you deleted everything besides Users and then try your full process again. (The shadow database can be empty by the way - Prisma Migrate takes care itself with bootstrapping that from nothing every time.) Can you still reproduce the problem? If so, can you just share the generated .sql file with us? Then we can just reproduce ourselves.

Sure thing. Would you mind if I threw it in a password-protected file and DMed you the password on Twitter or something, as it's unsanitised data 😬 Apologies if this is bridging work/life here - happy to do any alternative.
I have replicated it with just the users table (only 400 records) - in the morning I will try and do it with super simple seeded data.

<EMAIL_ADDRESS>is always there for these cases - but Twitter DM is also fine.

Hey, any update regarding this? Facing a similar kind of issue.

Took only 2 years, but I can reproduce this:
PS C:\Users\Jan\Documents\throwaway\14246> npx prisma migrate dev
Environment variables loaded from .env
Prisma schema loaded from prisma\schema.prisma
Datasource "db": SQL Server database

Error: P3006

Migration `20240522204011_change` failed to apply cleanly to the shadow database.
Error:
Cannot find the object "dbo.Users" because it does not exist or you do not have permissions.
   0: schema_core::state::DevDiagnostic
             at schema-engine\core\src\state.rs:268

Whole reproduction for the future:

Create a new database, apply this SQL:BEGIN TRY

BEGIN TRAN;

-- CreateTable
CREATE TABLE [dbo].[Users] (
    [userID] INT NOT NULL IDENTITY(1,1),
    [companyRole] VARCHAR(150),
    [email] VARCHAR(150),
    [password] VARCHAR(150),
    [salt] VARCHAR(150),
    [firstName] VARCHAR(150),
    [lastName] VARCHAR(150),
    [regType] VARCHAR(150),
    [tacAccept] VARCHAR(5),
    [tacAcceptDate] SMALLDATETIME,
    [userActive] VARCHAR(5),
    [dateAdded] SMALLDATETIME,
    [companyId] INT,
    [emailConfirmGUID] VARCHAR(150),
    [emailConfirmDate] SMALLDATETIME,
    [emailConfirmed] VARCHAR(5) CONSTRAINT [DF_Users_emailConfirmed] DEFAULT 'false',
    [dashboardStepsComplete] VARCHAR(5) CONSTRAINT [DF_Users_dashboardStepsComplete_1] DEFAULT 'false',
    [lastLoginDate] SMALLDATETIME,
    [viewRiskMatrix] VARCHAR(5),
    [userGUID] UNIQUEIDENTIFIER CONSTRAINT [DF_Users_userGUID] DEFAULT newid(),
    [partnerGUID] VARCHAR(150),
    [twoFAEnabled] VARCHAR(5) CONSTRAINT [DF_Users_twoFAEnabled] DEFAULT 'false',
    CONSTRAINT [PK_Users] PRIMARY KEY CLUSTERED ([userID])
);

COMMIT TRAN;

END TRY
BEGIN CATCH

IF @@TRANCOUNT > 0
BEGIN
    ROLLBACK TRAN;
END;
THROW

END CATCH

npx prisma db pull
Creates this Prisma schema:generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "sqlserver"
  url      = env("DATABASE_URL")
}

model Users {
  userID                 Int       @id(map: "PK_Users") @default(autoincrement())
  companyRole            String?   @db.VarChar(150)
  email                  String?   @db.VarChar(150)
  password               String?   @db.VarChar(150)
  salt                   String?   @db.VarChar(150)
  firstName              String?   @db.VarChar(150)
  lastName               String?   @db.VarChar(150)
  regType                String?   @db.VarChar(150)
  tacAccept              String?   @db.VarChar(5)
  tacAcceptDate          DateTime? @db.SmallDateTime
  userActive             String?   @db.VarChar(5)
  dateAdded              DateTime? @db.SmallDateTime
  companyId              Int?
  emailConfirmGUID       String?   @db.VarChar(150)
  emailConfirmDate       DateTime? @db.SmallDateTime
  emailConfirmed         String?   @default("false", map: "DF_Users_emailConfirmed") @db.VarChar(5)
  dashboardStepsComplete String?   @default("false", map: "DF_Users_dashboardStepsComplete_1") @db.VarChar(5)
  lastLoginDate          DateTime? @db.SmallDateTime
  viewRiskMatrix         String?   @db.VarChar(5)
  userGUID               String?   @default(dbgenerated("newid()"), map: "DF_Users_userGUID") @db.UniqueIdentifier
  partnerGUID            String?   @db.VarChar(150)
  twoFAEnabled           String?   @default("false", map: "DF_Users_twoFAEnabled") @db.VarChar(5)
}

npx prisma migrate diff --from-empty --to-schema-datamodel prisma/schema.prisma --script > prisma/migrations/init/migration.sql
Creates this migration.sql:BEGIN TRY

BEGIN TRAN;

-- CreateTable
CREATE TABLE [dbo].[Users] (
    [userID] INT NOT NULL IDENTITY(1,1),
    [companyRole] VARCHAR(150),
    [email] VARCHAR(150),
    [password] VARCHAR(150),
    [salt] VARCHAR(150),
    [firstName] VARCHAR(150),
    [lastName] VARCHAR(150),
    [regType] VARCHAR(150),
    [tacAccept] VARCHAR(5),
    [tacAcceptDate] SMALLDATETIME,
    [userActive] VARCHAR(5),
    [dateAdded] SMALLDATETIME,
    [companyId] INT,
    [emailConfirmGUID] VARCHAR(150),
    [emailConfirmDate] SMALLDATETIME,
    [emailConfirmed] VARCHAR(5) CONSTRAINT [DF_Users_emailConfirmed] DEFAULT 'false',
    [dashboardStepsComplete] VARCHAR(5) CONSTRAINT [DF_Users_dashboardStepsComplete_1] DEFAULT 'false',
    [lastLoginDate] SMALLDATETIME,
    [viewRiskMatrix] VARCHAR(5),
    [userGUID] UNIQUEIDENTIFIER CONSTRAINT [DF_Users_userGUID] DEFAULT newid(),
    [partnerGUID] VARCHAR(150),
    [twoFAEnabled] VARCHAR(5) CONSTRAINT [DF_Users_twoFAEnabled] DEFAULT 'false',
    CONSTRAINT [PK_Users] PRIMARY KEY CLUSTERED ([userID])
);

COMMIT TRAN;

END TRY
BEGIN CATCH

IF @@TRANCOUNT > 0
BEGIN
    ROLLBACK TRAN;
END;
THROW

END CATCH

npx prisma migrate resolve --applied init
Change the Prisma schema to:generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "sqlserver"
  url      = env("DATABASE_URL")
}

model Users {
  userID                 Int       @id(map: "PK_Users") @default(autoincrement())
  companyRole            String?   @db.VarChar(150)
  email                  String?   @db.VarChar(150)
  password               String?   @db.VarChar(150)
  salt                   String?   @db.VarChar(150)
  firstName              String?   @db.VarChar(150)
  lastName               String?   @db.VarChar(150)
  regType                String?   @db.VarChar(150)
  tacAccept              String?   @db.VarChar(5)
  tacAcceptDate          DateTime? @db.SmallDateTime
  userActive             String?   @db.VarChar(5)
  dateAdded              DateTime? @db.SmallDateTime
  companyId              Int?
  emailConfirmGUID       String?   @db.VarChar(150)
  emailConfirmDate       DateTime? @db.SmallDateTime
  emailConfirmed         String?   @default("false", map: "DF_Users_emailConfirmed") @db.VarChar(5)
  dashboardStepsComplete String?   @default("false", map: "DF_Users_dashboardStepsComplete_1") @db.VarChar(5)
  lastLoginDate          DateTime? @db.SmallDateTime
  viewRiskMatrix         String?   @db.VarChar(5)
  userGUID               String?   @default(dbgenerated("newid()"), map: "DF_Users_userGUID") @db.UniqueIdentifier
  partnerGUID            String?   @db.VarChar(150)
  twoFAEnabled           Boolean?  @default(false, map: "DF_Users_twoFAEnabled")
}

(only change is in last line)
npx prisma migrate dev
Creates this new migration.sql:/*
  Warnings:

  - You are about to alter the column `twoFAEnabled` on the `Users` table. The data in that column could be lost. The data in that column will be cast from `VarChar(5)` to `Bit`.

*/
BEGIN TRY

BEGIN TRAN;

-- AlterTable
ALTER TABLE [dbo].[Users] DROP CONSTRAINT [DF_Users_twoFAEnabled];
ALTER TABLE [dbo].[Users] ALTER COLUMN [twoFAEnabled] BIT NULL;
ALTER TABLE [dbo].[Users] ADD CONSTRAINT [DF_Users_twoFAEnabled] DEFAULT 0 FOR [twoFAEnabled];

COMMIT TRAN;

END TRY
BEGIN CATCH

IF @@TRANCOUNT > 0
BEGIN
    ROLLBACK TRAN;
END;
THROW

END CATCH

Another run of migration.sql triggers the error.

Running the two SQL scripts manually on a new database works just fine. No idea what is different with the shadow database and why that fails :/


Use the PhpDocExtractor in addition to the ReflectionExtractor.

I've added the PhpDocExtractor in addition to the ReflectionExtractor which allows you to use the phpdoc to decode data properly. That also works in tandem with the array denormalizer (which I've added to the verbs.php). Its typically default behaviour to include the PhpDocExtractor when the https://symfony.com/doc/current/components/property_info.html#phpdocextractor
I also fixed a bug with the NormalizeToPropertiesAndClassName. If the required parameters count is 0 and there's data, the Arr::has( will return true causing it to fail for no reason. I found that because the DTO class wouldn't serialise for me. I believe the allows us to store a serializable class as a property test isn't actually testing anything, otherwise it should've caught it - I wasn't sure so I didn't delete it.
I also believe now the deserialiser is more robust, the BackedEnum code can be removed and I wrote a test to try prove to myself its working fine - but do as you please with that.

@nick-potts thanks for your patience on this one. I was having a hard time understanding the scope of this PR. Would you mind adding a test that fails when the array denormalizer is missing to demonstrate the value of adding that?

The initial request I made had a bunch of changes that were later removed so the initial comment doesn't make too much sense - the latest is purely stuff that allows the PhpDoc Extractor to work.
The test that was added fails without the array denormaliser - the big benefit from the phpdoc extractor is that it can deserialise arrays using just the phpdoc. From the DTO it tests against, you can see how this works.
/** @var DTO[] $dtos */
public array $dtos = [new DTO]

Without the PhpDoc extractor or the array denormaliser, this property wouldn't get deserialised properly.

Thanks! Sorry it took so long to get this merged.


HBO Max Service Error Even after using VMP Signing Service. Netflix works fine.

Hi everyone!
I just tried out my application for the first time today using HBO Max, however I am getting this error:
Service error:
H-25e9b383-fb97-4707-8ac9-f88935070383
Netflix works fine inside of my application for both development as well as packaged versions, however HBO Max is giving me this error. I have tested on both Electron versions 9.4.3 as well as 12.0.4. I am almost certain that this is a Widevine problem with castlabs. I am using the vmp signing service to sign my application.
Thanks!

It appears that Brave Browser also incorporates Widevine for their viewership, of which works with HBO Max. Brave, similar to electron, is built on top of Chromium so shouldn't there be a way for Electron to work with HBO Max as well?

Did you figure this out?

Yes I did! It was an issue with my electron code, not Castlabs!

@ryankashi - we are having a similar issue. Can you please tell us how you fixed it? Thanks!


Reduce_sum and reduce_mean layers for Keras 3

Good morning,
Tensorflow recently moved to Keras 3. Before 2.16 (I believe), it was on Keras 2, and was allowing steps like tf.reduce_sum in the composition of a model.
Now, everything has to be a Layer of custom Layer.
So, I do get that there are some alternatives, such as using a custom Layer or a Lambda, but reduce_sum/reduce_mean are useful functions (e.g. getting the average of a bunch of embeddings), and custom Layer or Lambda have their own limitations which make things a lot less frictionless (e.g. serializing/deserializing).
Would be nice to have real layers for those?

There are equivalents in Numpy. See here.

@ghsanti , Thanks for pointing it out.
@fburgaud , With the multi backend support in Keras 3, in order to follow similar naming convention and implementation among Numpy, PyTorch, Jax and TensorFlow it has been named as ops.mean and ops.sum


是否有Space组件的规划

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

有，规划中


stop collet Retainer when the player is not in their homeworld

Thanks :D i wasnt thinking about world travel. Will release a new version soon.


github: add release-note description format to pull request template

Signed-off-by: Cadmus<EMAIL_ADDRESS>
What problem does this PR solve?
Help Tichi to realize the information collection function of release note.
https://github.com/ti-community-infra/configs/pull/326
Problem Summary:
What is changed and how it works?
Add a release note reference to pull_ request_template.md
What's Changed:
Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Breaking backward compatibility

Release note
Please add a release note.
If you don't think this PR needs a release note then fill it with None.

/run-check-release-note

/cc @andylokandy
Could you please take a look?

/hold
Needs test.

/run-check-release-note

/run-check-release-note

/merge

/unhold

/run-test


Add Engineering Strategy

[Insert URL to the list here]
https://github.com/aleixmorgadas/awesome-engineering-strategy
[Explain what this list is about and why it should be included here]
Useful Engineer Strategy resources. It contains Books list, posts, and open Engineering Strategies and Open Architectures.
Reviewed PR:

https://github.com/sindresorhus/awesome/pull/2316#issuecomment-1183511728
https://github.com/sindresorhus/awesome/pull/2302#issuecomment-1185483548
https://github.com/sindresorhus/awesome/pull/2370#pullrequestreview-1100158778

By submitting this pull request I confirm I've read and complied with the below requirements 🖖
Please read it multiple times. I spent a lot of time on these guidelines and most people miss a lot.
Requirements for your pull request

Don't waste my time. Do a good job, adhere to all the guidelines, and be responsive.
You have to review at least 2 other open pull requests.
Try to prioritize unreviewed PRs, but you can also add more comments to reviewed PRs. Go through the below list when reviewing. This requirement is meant to help make the Awesome project self-sustaining. Comment here which PRs you reviewed. You're expected to put a good effort into this and to be thorough. Look at previous PR reviews for inspiration. Just commenting “looks good” or simply marking the pull request as approved does not count! You have to actually point out mistakes or improvement suggestions.
You have read and understood the instructions for creating a list.
This pull request has a title in the format Add Name of List.

✅ Add Swift
✅ Add Software Architecture
❌ Update readme.md
❌ Add Awesome Swift
❌ Add swift
❌ add Swift
❌ Adding Swift
❌ Added Swift

Your entry here should include a short description about the project/theme of the list. It should not describe the list itself. The first character should be uppercase and the description should end in a dot. It should be an objective description and not a tagline or marketing blurb.

✅ - [iOS](…) - Mobile operating system for Apple phones and tablets.
✅ - [Framer](…) - Prototyping interactive UI designs.
❌ - [iOS](…) - Resources and tools for iOS development.
❌ - [Framer](…)
❌ - [Framer](…) - prototyping interactive UI designs

Your entry should be added at the bottom of the appropriate category.
The title of your entry should be title-cased and the URL to your list should end in #readme.

Example: - [Software Architecture](https://github.com/simskij/awesome-software-architecture#readme) - The discipline of designing and building software.

The suggested Awesome list complies with the below requirements.

Requirements for your Awesome list

Has been around for at least 30 days.That means 30 days from either the first real commit or when it was open-sourced. Whatever is most recent.
Don't open a Draft / WIP pull request while you work on the guidelines. A pull request should be 100% ready and should adhere to all the guidelines when you open it. Instead use #2242 for incubation visibility.
Run awesome-lint on your list and fix the reported issues. If there are false-positives or things that cannot/shouldn't be fixed, please report it.
The default branch should be named main, not master.
Includes a succinct description of the project/theme at the top of the readme. (Example)

✅ Mobile operating system for Apple phones and tablets.
✅ Prototyping interactive UI designs.
❌ Resources and tools for iOS development.
❌ Awesome Framer packages and tools.

It's the result of hard work and the best I could possibly produce.
If you have not put in considerable effort into your list, your pull request will be immediately closed.
The repo name of your list should be in lowercase slug format: awesome-name-of-list.

✅ awesome-swift
✅ awesome-web-typography
❌ awesome-Swift
❌ AwesomeWebTypography

The heading title of your list should be in title case format: # Awesome Name of List.

✅ # Awesome Swift
✅ # Awesome Web Typography
❌ # awesome-swift
❌ # AwesomeSwift

Non-generated Markdown file in a GitHub repo.
The repo should have awesome-list & awesome as GitHub topics. I encourage you to add more relevant topics.
Not a duplicate. Please search for existing submissions.
Only has awesome items. Awesome lists are curations of the best, not everything.
Does not contain items that are unmaintained, has archived repo, deprecated, or missing docs. If you really need to include such items, they should be in a separate Markdown file.
Includes a project logo/illustration whenever possible.

Either centered, fullwidth, or placed at the top-right of the readme. (Example)
The image should link to the project website or any relevant website.
The image should be high-DPI. Set it to maximum half the width of the original image.

Entries have a description, unless the title is descriptive enough by itself. It rarely is though.
Includes the Awesome badge.

Should be placed on the right side of the readme heading.

Can be placed centered if the list has a centered graphics header.

Should link back to this list.

Has a Table of Contents section.

Should be named Contents, not Table of Contents.
Should be the first section in the list.
Should only have one level of nested lists, preferably none.
Must not feature Contributing or Footnotes sections.

Has an appropriate license.

We strongly recommend the CC0 license, but any Creative Commons license will work.

Tip: You can quickly add it to your repo by going to this URL: https://github.com/<user>/<repo>/community/license/new?branch=main&template=cc0-1.0 (replace <user> and <repo> accordingly).

A code license like MIT, BSD, Apache, GPL, etc, is not acceptable. Neither are WTFPL and Unlicense.
Place a file named license or LICENSE in the repo root with the license text.
Do not add the license name, text, or a Licence section to the readme. GitHub already shows the license name and link to the full text at the top of the repo.
To verify that you've read all the guidelines, please comment on your pull request with just the word unicorn.

Has contribution guidelines.

The file should be named contributing.md. Casing is up to you.
It can optionally be linked from the readme in a dedicated section titled Contributing, positioned at the top or bottom of the main content.
The section should not appear in the Table of Contents.

All non-important but necessary content (like extra copyright notices, hyperlinks to sources, pointers to expansive content, etc) should be grouped in a Footnotes section at the bottom of the readme. The section should not be present in the Table of Contents.
Has consistent formatting and proper spelling/grammar.

The link and description are separated by a dash. Example: - [AVA](…) - JavaScript test runner.
The description starts with an uppercase character and ends with a period.
Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

Doesn't use hard-wrapping.
Doesn't include a Travis badge.You can still use Travis for list linting, but the badge has no value in the readme.
Doesn't include an Inspired by awesome-foo or Inspired by the Awesome project kinda link at the top of the readme. The Awesome badge is enough.

Go to the top and read it again.

'unicorn`

Thanks for making an Awesome list! 🙌
It looks like you didn't read the guidelines closely enough. I noticed multiple things that are not followed. Try going through the list point for point to ensure you follow it. I spent a lot of time creating the guidelines so I wouldn't have to comment on common mistakes, and rather spend my time improving Awesome.

Thanks for making this Awesome list!

Thank you for the feedback @sindresorhus 🧡
I identified that I didn't follow the - <description>. or adding the #readme at the end of the URL. I fixed them two.
I also followed those directions in awesome-engineering-strategy.
I hope this time I didn't miss any guideline ^_^

Your entry here should include a short description about the project/theme of the list. It should not describe the list itself.

Includes a succinct description of the project/theme at the top of the readme. (Example)

This should describe the topic.

Some links are missing a description.

The readme should only have English links. For other languages, it's better to create a separate Markdown file and link to it from the main list.

Some of the descriptions needs work:

A list of Patterns to understand and address the Business needs from an Engineering point of view.

None of the words here should be capitalized. Applies to other descriptions too.

Some of the descriptions needs work:

A list of Patterns to understand and address the Business needs from an Engineering point of view.

None of the words here should be capitalized. Applies to other descriptions too.

I confirm I've read and complied with the below requirements 🖖

@sindresorhus. Thank you for the clarifications and feedback. I improved the README.md 👍

Good night @sindresorhus. I applied all the changes and also took some time to review other PR based on what I learned during the process.
Do you think it follows all the guidelines? 😄

Hey,
I would remove that part in your readme:

Engineering Strategy is very needed nowadays in every technological company.

it seems this is not adding anything to the reader and for brevity purpose I would remove it.
Also I came across this article recently, https://lethain.com/how-to-safely-think-in-systems/, if you think this is useful happy to make a PR to your repo.

I updated the readme based on @cosydney feedback 👍

Includes a succinct description of the project/theme at the top of the readme.

https://github.com/aleixmorgadas/awesome-engineering-strategy/blob/main/contributing.md is just copy-pasted and have broken links. The contributions guidelines should be customized for your list.

https://github.com/aleixmorgadas/awesome-engineering-strategy/blob/main/contributing.md is just copy-pasted and have broken links. The contributions guidelines should be customized for your list.

Adapted for my list 👍

@sindresorhus, I think I have finally applied all the guidelines and fixes. Thank you for being so patient 🙏

GitPod => Gitpod

Done ✅

Tweet: https://twitter.com/awesome__re/status/1603895136316002304


[Feature][datasource] Simply create a new type of data source

Describe the feature
Create a data source, add a new type of data source, directly add the driver class on the page, and store the data source driver package in the lib directory. It is more convenient and flexible to deal with new types of data sources.
Is your feature request related to a problem? Please describe.
At present, a new type of data source is newly added, which will involve the modification, compilation, and release of the front-end VUE and the back-end JAVA. There will be many problems and difficulties in this process.
Describe the solution you'd like
DS can support new types of data sources. You can add a data source driver class on the page, and store the corresponding data source driver package in the lib directory to use the new data source.
Additional context

Commonly used data sources, select the data source type, you can bring out the driver class, connect to url information
Customize the data source, the user fills in the corresponding driver class, and puts the data source driver package in the lib directory
Fill in the data source ip, port and other information to automatically bind the variables on the connection url in both directions
Relevant parameter information can be added after connecting to the url

描述功能
创建数据源，新增一种新类型的数据源，直接在页面添加driver class,lib目录下存放数据源驱动包。在面对新类型的数据源处理起来更方便，灵活。
您的功能请求与问题有关吗？ 请描述。
目前新增加一种新的类型数据源，会涉及前端VUE，后端JAVA的修改，编译，发布版本。这过程中会存在很多问题难点。
描述您想要的解决方案
DS 可以支持新的类型数据源，可以在页面添加数据源driver class，lib目录下存放相应的数据源驱动包，即可使用新的数据源。
其他内容
在此处添加有关功能请求的任何其他上下文或屏幕截图。
1.常用数据源，选择数据源类型，可以带出driver class，连接url 信息
2.自定义数据源，用户填写相应driver class，将数据源驱动包放在lib目录下
3.填写数据源ip,端口等信息可自动在连接url 上变量双向绑定
4.连接url后可以追加相关参数信息

选择常用数据源

填写数据源信息

选择自定义数据源或新类型数据源，数据源信息需要自行填写

＃1390

I will finish it

+1，any design doc about this feature? It maybe need dynamic load class, if there is design doc, it will be better.


Configuration file for each driver

Instead of having a StarMe.cfg file we should have something like EW.cfg for the EW driver, synth.cfg for the synthesis driver, and abund.cfg for the abundances driver. The drivers were introduced in commit bc0f0ee and a8f5c9b.

For abundances, this is moved to issue #10

Is this relevant anymore? At the moment we use the same configuration file for all (except ARES), namely StarMe.cfg. Is this optimal, or should we change it?

Probably we should change for each Driver since each of them has different options.


Add Capture VPC Resource Spinnup/Teardown to CLI

Description
This task is to begin building out the management code for the Capture VPC.  We'll add CDK resources to the repo for the Capture VPC and update the CLI to be able to spin them up and naively tear them down.  This is expected to include:

An Arkime Capture Node Docker Image
AWS Fargate task(s) running the Arkime capture image
An OpenSearch Domain to receive metadata, probably in the same VPC
A VPC for the Fargate Tasks/Domain to live in
An S3 Bucket for the raw packet capture data

The goal here is to get the basic bones in place.  While basic effort will be given to wiring the components together, they are not expected to all perform cohesively at the end of this task.
We want the captured data/metadata to be retained upon spindown of the Arkime Cluster, so you should explore how many CloudFormation stacks to split this across.
Acceptance Criteria

Able to create/destroy the Arkime Capture VPC and its accompanying resources via the CLI
Captured data is retained on termination of the Cluster
NON-GOAL: all components are running/configured correctly and wired toegether

Picking up this task.

Posted the first PR for this, laying some more groundwork.  As I explain in the PR, there's some trickiness in how the Python and TypeScript sides of this thing interact with each other.
https://github.com/arkime/cloud-demo/pull/15

Made some good progress on this task; have a new PR up.  Have Create/Destroy Cluster working, with the caveat that there is some remaining work to correctly configure the Capture Nodes.  I ran into trouble getting Arkime installed/configured in the Docker containers and decided to circle back on that later as it wasn't currently a blocker.  It's still not in a place where I'm happy with it to wrap this task, but I'm gonna postpone completing this a bit to work on something else I can make faster progress on (https://github.com/arkime/cloud-demo/issues/6).

https://github.com/arkime/cloud-demo/pull/16

PR posted to resolve the task: https://github.com/arkime/cloud-demo/pull/19

PR merged; resolving.


Device auth flow / Headless auth

TL;DR
This PR implements DeviceAuth flow for Pyflyte - which supports headless authentication
Config
❯ cat ~/.flyte/demo-config.yaml
admin:
  endpoint: dns:///xyz
  authType: DeviceFlow

Sample output
❯ pyflyte run --remote hello.py wf

To Authenticate navigate in a browser to the following URL: https://xyz/activate and enter code: SCWBRTNH
OR copy paste the following URL: https://xyz/activate?user_code=SCWBRTNH

Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication Pending...
Authentication successful!
Go to https://xyz/console/projects/flytesnacks/domains/development/executions/f8cdccb9a534448e4883 to see execution in the console.

Type

[ ] Bug Fix
[x] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[x] Unit tests added
[x] Code documentation added
[x] Any pending items have an associated Issue

Tracking Issue
https://github.com/flyteorg/flyte/issues/3484

This is awesome. Thank you!


[SPARK-19307][pyspark] Make sure user conf is propagated to SparkContext.

The code was failing to propagate the user conf in the case where the
JVM was already initialized, which happens when a user submits a
python script via spark-submit.
Tested with new unit test and by running a python script in a real cluster.

Test build #71876 has started for PR 16682 at commit 35eeafe.

Test build #71876 has finished for PR 16682 at commit 35eeafe.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71876/
Test FAILed.

Merged build finished. Test FAILed.

Test build #71882 has started for PR 16682 at commit cef5136.

Test build #71882 has finished for PR 16682 at commit cef5136.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71882/
Test FAILed.

Test build #71893 has started for PR 16682 at commit ef8349a.

Test build #71893 has finished for PR 16682 at commit ef8349a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71893/
Test PASSed.

@holdenk @zjffdu

LGTM

Merging to master / 2.1.


Wrong .predict_proba() output shape for Classifier?

I'm trying the .predict_proba() method on a multi-label problem. And while the method returns the appropriate values, instead of returning them as [n_samples, n_classes] it returns them as [n_samples, n_probabilities]
Here is the output from a sample from a simple multi-label problem with 4 unique classes:
[[0.50014246  0.49985754  0.50011709  0.49988291  0.50063539  0.49936461    0.49984055  0.50015945]      [0.50018613  0.49981387  0.50015697  0.49984303  0.50078478  0.49921522    0.49983006  0.50016994]]
While the values in each single array are correct, they aren't paired (e.g., [0.5001, 0.4998], [0.5001, 0.4998]), which would then make the size [n_samples, n_classes]. Was this intended? I apologize for any confusion.
-ralston3

Thanks, it's a good question. I think you're right, returning sub-arrays would be more intuitive.
The most important decision factor is preserving compatibility with other sklearn classifiers and the rest of the functionality. Do you know how those work and what format they output? Are there other functions (e.g. grid_search) that would break if we changed the output?

alexjc,
Many (if not all) sklearn classifiers (unless there's a real niche one that I'm forgetting about), output .predict_proba() with sub-arrays. grid_search itself actually outputs sub-arrays as well. So I think you'd be okay to do the same (it would also make things much more consistent with scikit).
In regards to your edit:  the .predict_proba() output for the same type of simple problem I mentioned above, using a ensemble.RandomForestClassifier() outputs:
[array([[ 0.5,  0.5],         [ 0.7,  0.3]]),   array([[ 0.4,  0.6],         [ 0.5,  0.5]]),   array([[ 0.7,  0.3],         [ 0.7,  0.3]]),   array([[ 0.7,  0.3],         [ 0.6,  0.4]])]
a list of n outputs (4 since the toy problem contains 4 unique classes) by n_classes  (classes being 2 [1 or 0 since an instance either belongs to a class or doesn't]).
While in the case of the sknn.Classifier, .predict_proba() for that same problem would output:
[[0.50014246 0.49985754 0.50011709 0.49988291 0.50063539 0.49936461 0.49984055 0.50015945] [0.50018613 0.49981387 0.50015697 0.49984303 0.50078478 0.49921522 0.49983006 0.50016994]]
(I actually used dummy numbers but the shapes are the exact same as the output)
So with the ensemble.RandomForestClassifier() output I can read that theres a 50/50 chance that the first test instance belongs to class 1. I can also tell that there is a 70/30 chance that the second test instance belongs to class1. Further, I can see that there's a 40/60 chance that the first test instance belongs to class 2...and so on and so forth.
With the sknn.Classifier output it's just a bit confusing as to what I'm reading since the probabilities aren't in sub-arrays.
If that's what you guys intended then I apologizes for this hella long post, but I just wanted to make sure before I put it in the toolbox. The whole multi-label probability thing is definitely a brain twister.

That sounds reasonable. It was an array because that's how it comes out of the underlying Regressor. Converting it into a list of arrays would work and solves the problem of having different class numbers per output index.
It shouldn't be too much work. If you want to tackle it you're welcome to!


Add universitas muhammadiyah kudus

Fixing Domain Requirements based on reply : https://github.com/JetBrains/swot/pull/23780#issuecomment-2367765119

@ulinnuhadeni The first verification stage is complete: the official domain for  Universitas Muhammadiyah Kudus is sipenmaru.umku.ac.id, to the best of my knowledge. This is just an informational note. The pull request is still under review. The review may take some more time. I greatly appreciate your patience. Here is the proof of domain ownership: https://sipenmaru.umku.ac.id/

the official domain for Universitas Muhammadiyah Kudus is https://umku.ac.id

@ulinnuhadeni Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.


Data gap 11.3.1

Filled data gap, however cannot find original email with contacts. Made note of this in GIF - Documentation.

Done in https://github.com/sdg-data-canada-odd-donnees/sdg-data-donnees-odd/commit/9810bb5d472a8b9cb45c0ca339a7bd72d2dab0ff


The name of the mod incorrectly appears when it shows up in mod mismatch warning.

Thank you for the report, but that is not my problem.
I have never named my Mod Blaze Powder In End Cities, it can be Config API, because I copied over a lot things from there, but a name wich is not one of my projects and is in version 1.0.0 is not possible.


[APIM 3.2.0] Admin Portal Theming

Description:
The Logo of the Admin portal is hardcoded in the React Component Navigator.jsx. Hence, changing the logo URL in the defaultTheme.js of the Admin portal is not reflected.
Furthermore, the appBar customizations made in the Admin portal are not reflected due to the Styles being configured in the React components without any reference to the defaultTheme.js artifact.
Steps to reproduce:

Change the Logo URL of the Admin portal in <apim>/repository/deployment/server/jaggeryapps/admin/site/public/conf/defaultTheme.js

Affected Product Version:

APIM 3.2.0

This issue fixed in latest pack, hence closing


Update DetectRTC.js

Support listing input + output devices, that have the same label and if, but differ only in device kind.
I don't know how this PR will work with the tests, as I couldnt run them locally.

The tests are failing due to authorization issues. @muaz-khan is there anything I can do to help them pass?


How do we install the wego UI in a running weave gitops managed cluster

As a Platform Operator I would like to use the wego ui install command to setup the wego UI at a known URL inside the cluster.
Ingress is a separate issue.   In amazon this should be as simple as adding a load balancer config.
Output of the spike is a definition of how the cluster installs into the system.
(At the end of the day you need API server in the right place, and somewhere you can get the HTML)

We can install the UI at wego install time, but when we get to multi-cluster support we may want to install in just one of the clusters.
Also if teams move to the enterprise product they may want to move where the UI is installed.
But to be clear, we can install via the install command now.

Closed by https://github.com/weaveworks/weave-gitops/issues/655


在GitHub上找不到“ask chatgpt”图标

Describe the bug
问题描述
A clear and concise description of what the bug is.
To Reproduce
如何复现
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
期望行为
A clear and concise description of what you expected to happen.
Screenshots
截图说明
If applicable, add screenshots to help explain your problem.
Please complete the following information):
请补全以下内容

OS: [e.g. Windows]
Browser: [e.g. chrome, safari]
Extension Version: [e.g. v2.0.2]

Additional context
其他
Add any other context about the problem here.

仅限commit分析, 代码自己框选询问即可

如无问题我先关闭了, 有问题可以直接问