[release-1.10] feat: annotations on gateway serviceaccount

task: cherrypick github.com/istio/istio/pull/33914
Please provide a description of this PR:
Manual cherrypick required.
Resolves the issue #33914 failed to apply on top of branch "release-1.10" on https://github.com/istio/istio/issues/34837

Hi @bryankaraffa. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jacob-delgado -- Please  let me know if there is anything else I can help to get this approved and merged?  Thanks in advance

/ok-to-test

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
992004fcd6b03a843aa8bb1ad3f04252e8b57b88
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
82c4cd80453b131148a854cec8cb6969583d74c0
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio_release-1.10

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
82c4cd80453b131148a854cec8cb6969583d74c0
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
ae6c2294e119ca708ea9c96e45bb312e72a1646c
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Not sure what's going on here..
Executing: cd .; gh pr view https://github.com/istio/istio/pull/34969 --json files
{"files":null}
Found 0 files.

But I see releasenotes/notes/34969.yaml on the PR / GitHub.com UI

/test release-notes_istio_release-1.10

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
ae6c2294e119ca708ea9c96e45bb312e72a1646c
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio_release-1.10

@bryankaraffa: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
68cbb6afee19ab5e176d6854712242ea7e5c235f
link
/test release-notes_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bryankaraffa: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio_release-1.10
68cbb6afee19ab5e176d6854712242ea7e5c235f
link
/test release-notes_istio_release-1.10

integ-pilot-multicluster-tests_istio_release-1.10
68cbb6afee19ab5e176d6854712242ea7e5c235f
link
/test integ-pilot-multicluster-tests_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-pilot-multicluster-tests_istio_release-1.10

/retest


Decouple nuget packages from log4net

Issue
The nuget packages MUnique.OpenMU.* have a reference to log4net.
This forces every consumer to reference and use log4net as well.
Expected behavior
No reference to log4net. Instead, reference a logging abstraction, e.g. Microsoft.Extensions.Logging.Abstractions.
Additional context
See also:

https://blog.stephencleary.com/2018/05/microsoft-extensions-logging-part-1-introduction.html
https://blog.stephencleary.com/2018/06/microsoft-extensions-logging-part-2-types.html
https://stackify.com/net-core-loggerfactory-use-correctly/  Paragraph "Accessing the Logging API Outside of a MVC Controller"

As the packages make no use of an DI/IoC Container/Framework, the solution might be a bit tricky.

I started removing references of log4net from all projects, except the main entry points, of course.
It's going well and in this process I'm trying to move the architecture more towards using dependency injection, too.


[Slider, ColorSlider, ColorArea, ColorWheel, ColorHandle] "touch-action: none" missing in a number of patterns

Description
Slider, ColorSlider, ColorArea, ColorWheel, ColorHandle all deserve to have their touch-action property manages so that trying to set the value of those interfaces doesn't cause page scrolling in touch context.
Expected behavior
Setting values, and touching the "track", on these elements do not scroll the page in touch environments.
refs  https://github.com/adobe/spectrum-web-components/issues/1454

Reference: https://github.com/adobe/spectrum-web-components/pull/1455/files


admission timeout should not crash on panic

/assign @benluddy @dgrisonnet @dinhxuanvu

I am fine with fixing the panic, but I think that in the long-run we should work on removing this admission plugin.
It appears to me that it acts as a safeguard to prevent malicious/erroneous client from congesting the server, but it also prevents normal clients from executing any kind of long-running requests that would go through the admission plugin.
It would be better in my opinion to either implement a mechanism detecting clients congesting the server and block them or add an alert that would detect congestion ahead of time and provide users with a way to know which client is behaving incorrectly.

/retitle OCPBUGS-5991: admission timeout should not crash on panic


Update expressroute-locations.md

Added URL for Comcast listing

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@jaimesc : Thanks for your contribution to the Azure documentation! The author, @cherylmc, has been notified to review your proposed change.

#sign-off


The stats data includes strange strings like ` + 4 modules'

Bug report

What is the current behavior?
The stats data includes strange string:
{
  "moduleId": 14,
  "moduleIdentifier": "/projects/sbt-vuefy/node_modules/vue-loader/lib/index.js??vue-loader-options!/projects/sbt-vuefy/src/test/scala/givers/vuefy/assets/vue/component-a.vue e03f6658ab7c5e6582e13283490782d2",
  "module": "./vue/component-a.vue + 4 modules",
  "moduleName": "./vue/component-a.vue + 4 modules",
  "type": "harmony import specifier",
  "userRequest": "./dependencies/component-b.vue",
  "loc": "7:29-39"
}

If the current behavior is a bug, please provide the steps to reproduce.
The stats data above can be generated using the below plugin:
class Plugin {
  apply(compiler) {
    compiler.hooks.emit.tap("emit-hook", (compilation) => {
      for (let module of compilation.getStats().toJson().modules) {
        for (let reason of module.reasons) {
          console.log(JSON.stringify(reason));
        }
      }
    });
  }
}

What is the expected behavior?
Can we remove the string   + 4 modules in  "moduleName": "./vue/component-a.vue + 4 modules"?

Other relevant information:
webpack version:  4.27.1
Node.js version: 9.0.0
Operating System: Ubuntu 18.04 LTS
Additional tools: I'm using vue-loader with webpack. So, I'm not sure who adds these strange strings.

This simply means that the module was concatenated to include 4 other modules. You can disable this optimization if you want to see the correct dependency tree in the stats data.

Thank you @niieani. So, it is not a bug.
I have two follow-up questions:

How do I turn of the optimization you are talking about?
Where is the location of the code that is responsible? (I'd like to learn).

Thank you again.

Here you can turn it off:
https://webpack.js.org/configuration/optimization/#optimization-concatenatemodules
This is where it comes from in the code:
https://github.com/webpack/webpack/blob/d5e26f728adb63a1fca080ef728fd627952a921d/lib/optimize/ConcatenatedModule.js#L393-L398


API documentation for version 5

Hi
Where to find documentation for the API?
There used to be something under the Wiki

See https://tesseract-ocr.github.io/tessdoc/
https://tesseract-ocr.github.io/tessapi/5.x/index.html
I am not sure whether it has been updated recently.

Also see
https://abi-laboratory.pro/?view=timeline&l=tesseract
https://abi-laboratory.pro/index.php?view=changelog&l=tesseract&v=current

Is there some examples available like there used to be? :)

https://tesseract-ocr.github.io/tessdoc/#usage

For the most updated API documentation you can simply read the header files:
https://github.com/tesseract-ocr/tesseract/tree/master/include/tesseract

For the most updated API documentation you can simply read the header files:
https://github.com/tesseract-ocr/tesseract/tree/master/include/tesseract

It would be great though if it was possible to get some simple best practice examples on the new API v5 :)


Rename "function" column to "job_title"

Fix #162
Note that the "function" column is not deleted at this point, we probably should not delete until at least the next version.

Coverage remained the same at 52.924% when pulling fbb249ab2142f98c3c03eb436e19b5b5b8d67b8e on RLejolivet:rename-function-column into 13768d24ac1bcbc3347dc4bf5d41247484a7c1ab on blossom-project:master.


Add connector

@bsmithgall Forgot to remake this PR.  Please feel free to take a look whenever you have the time.  Note: there are no tests written yet - I plan to do that Monday - but I figured at least this way you can take  look.

This is a good start I think -- but I think it could be improved a little bit.

I think you should either explicitly enforce having a connector object instantiated via the enforce_full_pipeline method on the pipeline object itself, or alternatively allow no connectors.
If you go with the former, there is a good amount of duplication between a Connector object and an Extractor object which makes me think that maybe we are overengineering this here? Maybe instead we should just dump the Connector class and have a base-class method on the Extractor class called connect and let it be implemented by different subclasses or handled via mixin.

Thinking about it a bit more, I'm starting to lean towards destroying the Connector class (sorry!) -- if you look at the existing codebase, there's just a lot of pass-between from the two classes. I think answer provides some context about what I'm thinking.
Basically, you shouldn't have the target needed to be thrown into multiple places. It should only have to be declared once and not necessarily shared among the pipeline. If it is, it feels like the concerns aren't properly separated.

@bsmithgall That makes a lot of sense. I definitely kept getting the vibe that it wasn't making things simpler, but couldn't think of a good alternative.  I'll work on implementing what you described and I'll let you know when I make any significant commits if you'd like to take a look.

:+1:
If you want you could also take a crack at #15 and I could do some work on this, or I could take that from you as I think those are the last two big ones.

@bsmithgall That works for me.  I already stared a bit on #15, so I'll finish that up.

@saylorsd I am working on an implementation similar to this that I think is a bit promising. Do you have some time this afternoon to hop on a call and talk through implementation of FTP (for #17) or HTTP (for #32)

@bsmithgall Sounds good! I'm free until 4, so whenever works for you will probably work for me.

Because this branch has failing tests & merge conflicts, I'm going to close it and open a different PR if that's alright.

@saylorsd take a look here

:+1: Looks good. Are you waiting for the other two know connectors to be implemented before you make the pull request?

Yeah or at least one
On Jan 27, 2016 1:21 PM, "Steven Saylor"<EMAIL_ADDRESS>wrote:

[image: :+1:] Looks good. Are you waiting for the other two know
connectors to be implemented before you make the pull request?
—
Reply to this email directly or view it on GitHub
https://github.com/UCSUR-Pitt/wprdc-etl/pull/26#issuecomment-175808842.

Sounds good.  Do you plan on/already started doing so or mind if I take a swing at it?

Go for it. I'll ping you when I'm home and we can pair on the rest.
On Jan 27, 2016 1:24 PM, "Steven Saylor"<EMAIL_ADDRESS>wrote:

Sounds good. Do you plan on/already started doing so or mind if I take a
swing at it?
—
Reply to this email directly or view it on GitHub
https://github.com/UCSUR-Pitt/wprdc-etl/pull/26#issuecomment-175809941.


S3Bucket validation transient issue

Hello @jnonino ,
Since your udpate, I have issue during sonarqube deployment :
 Error: failure configuring LB attributes: InvalidConfigurationRequest: S3Bucket validation transient issue
      status code: 400, request id: 391d3af7-99cd-4cd8-9617-15557228d1ce

  with module.sonar.module.ecs_fargate.module.ecs-fargate-service.module.ecs-alb.aws_lb.lb,
  on .terraform/modules/sonar.ecs_fargate.ecs-fargate-service.ecs-alb/main.tf line 26, in resource "aws_lb" "lb":
  26: resource "aws_lb" "lb" {

Works if I disable S3 export log from LB.

Yes, seems to be an issue there. I found this while searching about it AWS S3 Bucket for ALB Transient Issue
. When I have some time I will try to find a way to solve it.

Hi, how are you? Can you check if change in latest version fixed the issue?

I'm getting exactly the same error with latest version.
Also the bucket s3 arn should be provided as output so lifecycle can be configured.

Also getting the error when using an S3 bucket in another region
│ Error: failure configuring LB attributes: InvalidConfigurationRequest: S3Bucket validation transient issue
│       status code: 400, request id: 8db33ae0-168a-42c2-bd37-109b1d2c6664

I also got that error when the bucket where the elb logs are sent had the encryption option "Server-side encryption with AWS Key Management Service keys (SSE-KMS)" with an aws managed key, and the error disappeared after switching the bucket to "Server-side encryption with Amazon S3 managed keys (SSE-S3)".
So I think the bug/error is on AWS side with the bucket encryption and not on terraform side.


Separate command-line option data into its own module

Breaks down the command-line options configuration object, validation function, and other pieces that would be convenient to anyone who happened to write their own alternative to cli/index.js.

@facebook-github-bot shipit

@facebook-github-bot shipit


Tauri throws an error relating to that I cannot name window with "main", cause there is already one.

So, I am using the localhost-plugin for tauri app, because I got a problem with mixed content trying to reach API endpoints. Everything works just fine, but I gotta set the window name to something different than "main", because it's already used by actual main window. As it turns out, it cannot be overwritten. So, when I run my app I am getting 2 windows. The main with not working API and the second with working API.
How do I resolve this problem to let it on "main"?

thread 'main' panicked at 'error while running tauri application: Setup(SetupError(WindowLabelAlreadyExists("main")))', src\main.rs:30:6
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

// Prevents additional console window on Windows in release, DO NOT REMOVE!!
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]

// Learn more about Tauri commands at https://tauri.app/v1/guides/features/command

use tauri::{utils::config::AppUrl, window::WindowBuilder, WindowUrl};

fn main() {
  let port = portpicker::pick_unused_port().expect("failed to find unused port");

  let mut context = tauri::generate_context!();
  let url = format!("http://localhost:{}", port).parse().unwrap();
  let window_url = WindowUrl::External(url);
  // rewrite the config so the IPC is enabled on this URL
  context.config_mut().build.dist_dir = AppUrl::Url(window_url.clone());

  tauri::Builder::default()
    .plugin(tauri_plugin_localhost::Builder::new(port).build())
    .setup(move |app| {
      WindowBuilder::new(
        app,
        "main",
        window_url
      )
      .title("Localhost Example")
      .build()?;
      Ok(())
    })
    .run(context)
    .expect("error while running tauri application");
}

You'll need to either remove the window config from the tauri.conf.json file and only spawn the window from rust, or delete the .setup stuff from the code and only use the tauri config to spawn the window (https://github.com/tauri-apps/plugins-workspace/issues/148#issuecomment-1692776095) - otherwise you're indeed creating 2 windows here.


[crypto] compilation fix for clang10 -Wdeprecated-copy (delete KeyMat…

Deleting a copy constructor without specifying an explicit assignment operator causes an error in clang10 (-Wdeprecated-copy). Since explicit assignment operator defined in OPENTHREAD_CONFIG_PLATFORM_KEY_REFERENCES_ENABLE case, added the copy constructor deletion also under the same. If the macro is not enabled, implicit copy constructor and assignment operator would take effect.

@sarveshkumarv , should we close this PR in favor of #7487 ?

@jwhui yes, closed the PR. thanks.


Implementation of Super Mario Bros

Implementation of Super Mario Bros.
Includes support for the following ROM versions:

(World)
(Japan, USA)
(E)

Has the following 4 types of quests:

Score - Collect 100,000 points
Worlds - Complete one of Bowser's Castles
Streak - Complete five consecutive levels without dying (warps not included)
Damageless - Complete a level without dying or taking damage

Hi, I cloned your branch, i encountered some bugs
BizHawk 2.9.1, NesHawk, Super Mario Bros (Japan, USA) rom

What specifically had you done when it happened? I tried to account for a max score increase to prevent RAM manipulation, but that value must've been too low.

I just played he game, score wasn't increasing and streak / damageless went up without me doing anything

I'll have to check tomorrow, but it's odd that the level related things were incrementing on their own and the score was spamming that message. I put in some code to try and prevent that sort of message spam when the value was too high.
Had you advanced beyond the title screen perhaps?


Load extensions on runtime

Previous PR had messed up branches. See for reference. https://github.com/OP-Engineering/op-sqlite/pull/38

@ospfranco Hi, after updating to 2.0.15 I get Use of undeclared identifier 'sqlite3_enable_load_extension' and Use of undeclared identifier 'sqlite3_load_extension' on iOS on the bridge:

I'm using rn 0.73.2 and iOS 17.2 on Xcode 15.2

Are you using the Phone embedded sqlite or any flags?

Not embedded, no special flags on pod install. I have cleaned the Product and Derived Cache. Downgrading to 2.0.13 works as a charm.

Just created a new expo project and installed 2.0.15 and it is compiling without issues. Maybe you can create a reproduction repo.

Thank you for looking into this. I created an empty bare react-native project with expo-modules-core (that's what I do in current app) and it compiled just fine.
What I've tried:

delete yarn.lock and node_modules
delete Podfile.lock and Pods folder
pod deintegrate
delete Derived Data, clean build folder
install packages again
pod install

Cannot get it to compile.
Tried with OP_SQLITE_USE_PHONE_VERSION=1 npx pod-install and the same error pops.
This is my Podfile:

Click me
require File.join(File.dirname(`node --print "require.resolve('expo/package.json')"`), "scripts/autolinking")

# required for react-native-permissions
def node_require(script)
  # Resolve script with node to allow for hoisting
  require Pod::Executable.execute_command('node', ['-p',
   "require.resolve(
     '#{script}',
     {paths: [process.argv[1]]},
   )", __dir__]).strip
end

node_require('react-native/scripts/react_native_pods.rb')
node_require('react-native-permissions/scripts/setup.rb')

platform :ios, '13.4'
prepare_react_native_project!
$RNVideoUseVideoCaching=true
flipper_config = ENV['NO_FLIPPER'] == "1" ? FlipperConfiguration.disabled : FlipperConfiguration.enabled

setup_permissions([
 'Camera',
 'LocationWhenInUse',
 'Microphone',
 'Notifications',
 'PhotoLibrary'
])

flipper_config = ENV['NO_FLIPPER'] == "1" ? FlipperConfiguration.disabled : FlipperConfiguration.enabled

linkage = ENV['USE_FRAMEWORKS']
if linkage != nil
  Pod::UI.puts "Configuring Pod with #{linkage}ally linked Frameworks".green
  use_frameworks! :linkage => linkage.to_sym
end

IS_CI = ENV['APPCENTER_BUILD_ID']

target 'app' do
  use_expo_modules!
  post_integrate do |installer|
  	begin
  		expo_patch_react_imports!(installer)
  	rescue => e
  		Pod::UI.warn e
  	end
  end
  config = use_native_modules!
  use_frameworks! :linkage => :static

    use_react_native!(
      :path => config[:reactNativePath],
      # Note that if you have use_frameworks! enabled, Flipper will not work and
      # you should disable the next line.
      :flipper_configuration =>  FlipperConfiguration.disabled,
      # An absolute path to your application root.
      :app_path => "#{Pod::Config.instance.installation_root}/.."
    )

  pod 'FirebaseAppCheckInterop', '=' + "10.19.0"
  pod 'FirebaseAuthInterop', '=' + "10.19.0"
  pod 'FirebaseMessagingInterop', '=' + "10.19.0"
  pod 'FirebaseSharedSwift', '=' + "10.19.0"
  pod 'FirebaseCoreInternal', '=' + "10.19.0"
  pod 'FirebaseInstallations', '=' + "10.19.0"
  pod 'SDWebImageWebPCoder', '=' + "0.14.1"
  $VCDisableFrameProcessors = true
  $RNVideoUseVideoCaching=true

  $FirebaseSDKVersion = '10.19.0'

  pod 'RNLocalize', :path => '../node_modules/react-native-localize'
  pod 'react-native-image-crop-tools', :path => '../node_modules/react-native-image-crop-tools'

  target 'appTests' do
  	inherit! :complete
  	# Pods for testing
  end

  post_install do |installer|
  	react_native_post_install(
  		installer,
  		config[:reactNativePath],
  		:mac_catalyst_enabled => false
  	)
  end
end

It seems it does not like use_frameworks! :linkage => :static. I updated the repro repo Podfile to include it and it breaks.

That's weird, considering the source files are compiled with the rest of the code. Please create a new issue and I will take a look when I can although the use framework issues are usually not fixable.

@ospfranco done, linked issue: https://github.com/OP-Engineering/op-sqlite/issues/42


Update the icons package docs

### Tasks
- [ ] On the page `Serve the icons within your application` update the sections `Angular wrapper package` and `Declare manually Web components` to point out, this are two different ways how to add components to your angular app.
- [ ] Move the section `Serve the icons within your application` from the page `Getting Started/Angular` to the page `Getting Started/Packages/Icons` and merge it with the section `Copy & paste the icons with a postinstall script`.
- [ ] Update the title of the merged section to something like `Selfhosting the icons`.
- [ ] Point out, the difference between a vanillaJS setup (which will need the postinstall script) and the angular way to serve the icons.
- [ ] Point out, that the copied `/assets/post-icons` folder needs to be served by a dev server and be available in the dist folder, to be accessible from the int/prod server.
- [ ] Remove the section `Serve the icons wihin an Angular application` above.

@oliverschuerch we were not sure what to discuss about for this ticket, so we skipped it.

Not relevant anymore


Open IFTTT Twitter > Facebook

Good and or instagram to both...do we have instagram?
On Saturday, February 13, 2016, Clément Renaud<EMAIL_ADDRESS>wrote:

—
Reply to this email directly or view it on GitHub
https://github.com/protocultural/protocultural.org/issues/28.

No instagram I think

this guy https://www.instagram.com/protocultural/

K then we ignore
On Saturday, February 13, 2016, Clément Renaud<EMAIL_ADDRESS>wrote:

this guy https://www.instagram.com/protocultural/
—
Reply to this email directly or view it on GitHub
https://github.com/protocultural/protocultural.org/issues/28#issuecomment-183617085
.


System tests with a real AreaDetector IOC

This issue captures a discussion with @jwlodek @mrakitin @evalott100 @prjemian and others.
Despite our best efforts, updates to ophyd-async are still reasonably likely to break for someone somewhere, primarily because IOCs are fiddly, idiosyncratic, have no defined schemas and can vary per facility. @coretl actually identified this as a major risk to the project in a review a few weeks ago. This brittleness is particularly evident with AreaDetectors, so if we're going to put any work into mitigating this problem, that would be the low-hanging fruit to start with.
@jwlodek's proposal was a set of system tests that run in CI against a containerised simulated areadetector. @gilesknap could probably comment on how easy such a thing would be to set up. The main flaw is that it would only pick up problems with ADCore, and not any specific detectors, but it would be a start.
Comment if I've missed anything :)

Discussed this with @DominicOram in relation to an Odin system test. To summarize:

We like the idea of separating out tests/ and system-tests/ so the existing test suite isn't burdensome to run
Anything in system-tests/ will have dependent services like the EPICS containerised IOC
These services should be run up in the same way as @garryod does for his fake databases, can't remember if it was docker compose or a GitHub actions construct

Yep, Garry pointed me at the repos. I will do some of this as part of https://github.com/DiamondLightSource/dodal/issues/700 but will probably spin out a new issue for the CI

I like the idea of a separate system_tests that have some more rigorous "real" tests with external service dependencies. Something we can run before releases to better catch issues without compromising the simple/fast current suite of tests.

@jwlodek agreed but I think running it just around releases is not often enough. Given containerised IOCs etc. there's no reason not to also run it in CI for at least every PR.


is there a mail list for this project?

I currently working on a deep learning project using time series data. I am interested in learning more and possibly contributing to this project. What is the best way to ask questions
for example does spark-deep-learning scale out linearly?
Happy Thanksgiving
Andy

Here is the brand-new mailing list: https://groups.google.com/forum/#!forum/dl-pipelines-users. Thanks for asking :-)
For the linear scale-out, inference should scale out mostly linearly, but there can be some overhead, and as usual with Spark good partition number matters. Is there a particular type of workload you're thinking about? (We can move the conversation to the mailing list if you want)


chore: create appsmith schema for postgres

Description
The current state is default schema or public schema. This schema is accessible by default when user connects to the pg database. Hence create appsmith schema for Appsmith server to use. This is to avoid anyone accidentally modifying the appsmith data.
Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!TIP]
🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/11104824772
Commit: 80da201f84ad4ca87e76d40dbedc2e06108f88df
Cypress dashboard.
Tags: @tag.Sanity
Spec:
Mon, 30 Sep 2024 11:39:09 UTC

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[ ] No

Summary by CodeRabbit
Summary by CodeRabbit

New Features

Introduced a script to initialize the PostgreSQL database schema for Appsmith.
Added utilities for managing PostgreSQL database connections, including availability checks and parameter extraction.
Enhanced scripts for managing PostgreSQL connections and initialization.
Improved environment configuration for PostgreSQL database connections, including automatic password generation for local setups.
Updated JDBC URL handling to include schema parameters for PostgreSQL connections.

Bug Fixes

Improved error handling and connection retry mechanisms for PostgreSQL setup.

Documentation

Updated comments and logging for better clarity on database operations.

/build-deploy-preview skip-tests=true

/build-deploy-preview skip-tests=true

@abhvsn I have tested this change by manually verifying the information table to check the schema used for creation.

/build-deploy-preview skip-tests=true

/build-deploy-preview skip-tests=true


removing staging from the iaf template

remove all internal references

@bmos299 This fix will break things.  The staging pull secret is required to install the Demo Cartridge.  Need to keep it here so that both pull secrets are set and only one reboot is necessary.


Add BinaryClassificationResult

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.70%. Comparing base (b0844ad) to head (a9356bc).

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #39       +/-   ##
===========================================
+ Coverage   80.00%   93.70%   +13.70%
===========================================
  Files           9       10        +1
  Lines         100      143       +43
  Branches       12       18        +6
===========================================
+ Hits           80      134       +54
+ Misses         20        9       -11

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


New list: Awesome Conversational AI

I have started a new awesome list: https://github.com/jyguyomarch/awesome-conversational-ai
Found some very good lists on ML and NLP, but none on Conversational AI or related. Thought I would post here to get early feedback, or just in case someone knows of an existing list. If you want to contribute as usual feel free to PR.
I'll close this issue in a bit less than 30 days once the list is mature enough for a PR. And if it is not the right way to use issues in this repo, feel free to close it of course.

I'm trying to learn GitHub since I dared do mistakes learn cool diagrams, not sure Traumatic Brain Injury can deal with much but seeing your posts are interesting. And it heals and lessen pain. Thank you.

@jyguyomarch
Would this be useful?

@bolshoytoster definitely. Thx for that!

Good for you

I am looking for others who would like to build out a website domain called NicelyUnited.com This would be complex however I know we have the global reach to make this happen.  Anyone who wants to collaborate and create something that will be uniquely ours please refer to this post.  My email is<EMAIL_ADDRESS>

Closing this issue as I have created a PR #2160

Thank you
Get Outlook for iOShttps://aka.ms/o0ukef

From: jyguyomarch @.>
Sent: Saturday, January 22, 2022 8:09:37 PM
To: sindresorhus/awesome @.>
Cc: RockSolid11212 @.>; Comment @.>
Subject: Re: [sindresorhus/awesome] New list: Awesome Conversational AI (Issue #2130)
Closed #2130https://github.com/sindresorhus/awesome/issues/2130.
—
Reply to this email directly, view it on GitHubhttps://github.com/sindresorhus/awesome/issues/2130#event-5937672614, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV76ETXIZRSOFHMBEIWHCH3UXNPODANCNFSM5KXCOJZA.
You are receiving this because you commented.Message ID: @.***>


[BUG] 'pandas._libs.tslibs.offsets.MonthBegin' object has no attribute '_period_dtype_code'

My data is sorted by month frequency like 1844-01-01,1844-02-01, and I used ignore_time_index=True, and I ran it exactly as the example on the official website, but I still got the error：
'pandas._libs.tslibs.offsets.MonthBegin' object has no attribute '_period_dtype_code'
Traceback (most recent call last):
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/timeseries/models/abstract/model_trial.py", line 38, in model_trial
model = fit_and_save_model(
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/timeseries/models/abstract/model_trial.py", line 72, in fit_and_save_model
model.fit(
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/timeseries/models/abstract/abstract_timeseries_model.py", line 188, in fit
return super().fit(**kwargs)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/core/models/abstract/abstract_model.py", line 587, in fit
out = self._fit(**kwargs)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/timeseries/models/sktime/abstract_sktime.py", line 140, in _fit
self._to_skt_data_frame(train_data[[self.target]]), fh=self._fh()
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/autogluon/timeseries/models/sktime/abstract_sktime.py", line 95, in _to_skt_data_frame
data.index.get_level_values(1).to_period(freq=data.freq),  # noqa
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/indexes/extension.py", line 95, in method
result = attr(self._data, *args, **kwargs)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 84, in method
return meth(self, *args, **kwargs)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 1184, in to_period
return PeriodArray._from_datetime64(self._ndarray, freq, tz=self.tz)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/period.py", line 286, in _from_datetime64
data, freq = dt64arr_to_periodarr(data, freq, tz)
File "/root/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/period.py", line 1119, in dt64arr_to_periodarr
base = freq._period_dtype_code
AttributeError: 'pandas._libs.tslibs.offsets.MonthBegin' object has no attribute '_period_dtype_code'
please help me

Hi,

What version of AutoGluon are you using?
Can you please share the code that you ran that resulted in this error?

Thanks for the reply ！

version is 0.5.2 (I originally thought I was on the latest version, I will try to upgrade to fix this)
the code is exactly the same as the quickstart on the official site

I am excited and guilty to reply to you, my model started training very successfully, I am sorry it was my oversight

No worries


Error: "SiriShortcuts" plugin is not implemented on web with SvelteKit Capacitor

Trying to build a SvelteKit Capacitor iOS app along the lines of examples [1, 2]. I don't understand why I receive a web error.
Is there a way to work around the error? After building the app, it is run inside Capacitor iOS.

Trying to work around by using:
	import { Capacitor } from '@capacitor/core';
	import { SiriShortcuts } from 'capacitor-plugin-siri-shorts';

	if (Capacitor.isNativePlatform()) {
		SiriShortcuts.donate({
			persistentIdentifier: 'shortcut',
			title: 'Fetch updates'
		});
	}


MongoDB connector: Add 'push' command support for embedded objects

Suggested solution
Currently, Prisma does not support a 'push' command for array in embedded objects. It would be extremely useful to have a native 'push' command that allows us to update and add elements directly within array of embedded objects in the database.
Alternatives
As an alternative, we can first fire a find command, and then alter the array followed by a set command for the embedded object.  However, these approaches are not as clean, efficient,  as having a dedicated 'push' command in Prisma.
Additional context
The 'push' command for embedded objects would streamline the process of adding new items to an embedded array within a Prisma model. Currently, developers have to manually fetch the existing embedded object, update it in the application layer, and then save it back to the database. This process is cumbersome, especially when dealing with large arrays or frequent updates.
Having a 'push' command will not only improve developer productivity but also optimize database performance by reducing unnecessary reads.
Additionally, this feature aligns with the general expectations of modern ORMs and helps Prisma stay competitive in the market.
Below code doesn't work, a push inside embedded object should be supported.
return this.prisma.path.update({

      where: {
        id: id,
      },

      data: {
        obj1: {
          objarray: {
            push: {
              paramObj,
            },
          },
        },
      },
    });

What exactly are obj1 and objarray in your example?
Because fundamentally we do support push for composite types (embedded documents in MongoDB): https://www.prisma.io/docs/reference/api-reference/prisma-client-reference#push-1
const product = prisma.product.update({
  where: {
    id: 10,
  },
  data: {
    photos: {
      // Push a photo to the end of the photos list
      push: [{ height: 100, width: 200, url: '1.jpg' }],
    },
  },
})

@janpio Thanks for the reply. The example that you shared is an array of composite type. In my case, the composite array is inside the embedded object. However, the page above doesn't has schema of the product.
My schema would look something like this: So, I am trying to push a photo in the embedded object Product present in User Model.
`
type Photo {
id : String
url: String
}
type Product {
photos: Photo []
}
model User {
product: Product;
}
`

Ah ok, so you are trying to push 1 value inside of an array which is part of an embbedded document / composite type - do I understand that correctly?
I think that would indeed require reading the full embedded document, then using push on its list field (photos), and then
update that whole object back. And you want to avoid the need to do all this manually, and be able to just use 1 method directly, correct?

@janpio yes, exactly. It seems it's not possible right now.

Thanks, valid feature request then. Thanks.


Can we combine weights with heuristics? (A* with monus-weighted-search)

I'm interested in monus-weighted-search is the context of searches with heuristics - specifically A* search. I understand A* search as an extension of Dijkstra's algorithm where we track the cost of the path so far, but we weight our node expansion with a heuristic (the estimated remaining cost to the goal).
My attempt so far is to work with
data Cost a = Cost a (Maybe a)

Here, the first a is our incurred cost, and the second Maybe a is our estimated total cost to the goal, with Nothing meaning an infinite cost. Ordering is defined as ordering on the sum of the cost and estimate, and the monoid operation is the product of Sum and Min monoids (so combine costs, and take the minimum estimated cost).
However, I'm stuck on a sensible Monus for this (if there is any at all). My intuition as to what is happening in monus-weighted-search is we essentially have a forest of trees to explore. When we use popMin, we find a root in this forest with the lowest weight, and then sort of "splice in" the siblings of that root, and adjust all other roots with |-| to decrement their weight. In this case, I would have to reduce the cost and estimate for each node, but I'm not sure that can be done with |-| as |-| is commutative, and this operation isn't:
Cost a a' |-| Cost b b' = Cost (a |-| b) (a' |-| (fmap (|-| a) a')

Notice that we had to choose a for the new final Cost, completely ignoring b'.
Were |-| no longer commutative, I think this could work, however. The only assumption is commutativity appears in <||>:

(x, xv) <||> (y, yv)
  | x <= y    = (x, HeapT (ListT (pure ((x |-| y :< yv) :- runHeapT xv))))
  | otherwise = (y, HeapT (ListT (pure ((x |-| y :< xv) :- runHeapT yv))))

Notice how both branches call x |-| y. I think if we instead did:
(x, xv) <||> (y, yv)
  | x <= y    = (x, HeapT (ListT (pure ((x |-| y :< yv) :- runHeapT xv))))
  | otherwise = (y, HeapT (ListT (pure ((y |-| x :< xv) :- runHeapT yv))))

then things might work out.
Alternatively, maybe it's better to instead have something like:
HeapT w m a = ListT m (Node (w, Maybe w) a (HeapT w m a))

where each Node now has a cost and an estimate, allowing us to operate on the estimated cost independently.
I'd love to hear your thoughts!

Aha, I just read in the conclusion of the paper:

One drawback of our approach is that it does not seem possible to formulate a monus for heuristic-augmented search algorithms, like A*: we would be interested in generalisations of the monus algebra which might allow for the inclusion of such heuristics

Yes, unfortunately I can't figure out a way to add heuristics to the weight. The issue isn't just that |-| isn't commutative, it's that there's no implementation of |-| that satisfies:
(x <> y) |-| y = x

I haven't looked into the issue in a while, so I might be misremembering, but I am pretty sure that there's no general Monus (a,b) given Monus a and Monus b.
My intuition for how to add heuristics is not to pair them up with the weight, but rather to put them in a different part of the algorithm altogether.

My intuition for how to add heuristics is not to pair them up with the weight, but rather to put them in a different part of the algorithm altogether.

Yea, I did also consider this (re-evaluate the heuristic at each popMin call) but I couldn't see a way to extract any information from each HeapT subtree to compute a heuristic with.


Change expand group speed

How change expand group speed or delete expand animation (etc speed) absolutely.
Etc, like ExpandableListView behavior ...

Use RecyclerView.ItemAnimator's methods.

RecyclerView.ItemAnimator.setMoveDuration(long moveDuration)
RecyclerView.ItemAnimator.setRemoveDuration(long removeDuration)
RecyclerView.ItemAnimator.setAddDuration(long addDuration)


Is wasm-eval in content security policy needed?

Blocking https://github.com/oasisprotocol/oasis-wallet-ext/issues/1
https://github.com/oasisprotocol/oasis-wallet-ext/blob/f912f132cc22bec5d70541ddf46b90d7d28c6c6d/public/manifest.json#L35
There are no wasm files in repo. Is there a dependency the requires this?
I found<EMAIL_ADDRESS>so maybe only needed in development?

Also object-src.
I'd use "default-src 'self'; frame-ancestors 'none';" as per https://developer.chrome.com/docs/extensions/mv3/security/#content_security_policy

OK， If it is not used, I will remove it

ok ,well done

I set it to. "content_security_policy": "script-src 'self';",
because default-src 'self' will lost my css .
frame-ancestors we have Dapp  , so we supply  our iframe  to all web if set our chrome-id . if set this . it cannot work

ah, okay


build badge links to 404

Hi @caryan, thanks for reminding about this.
Right now the semaphore build is still under our private semaphore organization and I'm looking into how to make public.
This will make it easier to see builds on submitted PRs.  Currently, the badge on the github readme is accurately and publicaly updated based on the master build.

Ok great! The settings should be fixed.  Do re-open if there are still difficulties.


speikkonu/wideimage abandoned

speikkonu/wideimage is abandoned - vegas-cmf/media should use another library for thumbnail generation instead

Imagick ?
Phalcon\Image\Adapter\Imagick

:+1: for Imagick.


fix: fix build with forward ref and modify configs and beautify

Changes files beatify
Added babel config file
Modified package.json babel config and added clean script
Fixed forwardRef types in carousel

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: shayan-ginbits
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @gregpavl

@shayan-ginbits: GitHub didn't allow me to request PR reviews from the following users: daniellalasa2.
Note that only ginbits members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @daniellalasa2

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[APPROVALNOTIFIER] This PR is APPROVED
Approval requirements bypassed by manually added approval.
This pull-request has been approved by: shayan-ginbits
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Best practice query of foundation_and_overrides.scss

Shouldn't foundation_and_overrides.scss be prefixed with an _ as it is a partial that's imported in your own scss file? Perhaps it should really be _foundation-and-overrides.scss really?

Partials are parts of a project that may have dependencies in it. The main file of a package is not a partial as it does not rely on anything, even if the purpose of the package is to be included somewhere.


stm32h7/linum-stm32h753bi: fixed leds example

Summary
Fixed leds example.
Impact
fix build of example.
Testing
Run the build command.
./tools/configure.sh linum-stm32h753bi:leds

@xiaoxiang781216 @halyssonJr   there is some issue with LVGL on ESP32 board: (not related to his PR)
Configuration/Tool: esp32-2432S028/nsh
Cleaning...
find: './lvgl/demos': No such file or directory
find: './lvgl/examples': No such file or directory
find: './lvgl/src/extra': No such file or directory


Unit Test Ansible Roles used in platforms/shared/configuration/environment-setup.yaml

Description
As a developer I want to test environment playbook statically so that I can integrate with CI tools.
Use molecule (https://molecule.readthedocs.io/en/stable/index.html) to test Ansible roles.
Acceptance Criteria

molecule folder created under platforms/shared/configuration
At least one test scenario created to test all roles called by environment-setup.yaml playbook
Test scenario should include verification steps for each role.
molecule test should pass
All yaml lint issues and ansible code-quality issues should be fixed.

Closed as per PR


How to get absolute distances on your own dataset?

Hi author, If I collect some data by myself (including depth information labels), how can I get the absolute depth distance on your project?

Hi, the method itself is to generate relative depth values. If you have metric depth values, you might try scaling the relative depth values using a scaling factor. But the result might be not accurate enough.

I am a newbie, how should I calculate the scale factor, can you tell me where is the code to calculate the scale factor or tell me how to calculate the scale factor, thank you very much!

Hi, I don't have this code. The question is out of the scope of this repo.
Median scaling might be a good starting point to your question.

Is it possible to refer to this part of the code to statistically calculate the scaling factor that corresponds to my own data.

Yes, you can have a try, but it might not be accurate enough.

So when doing model inference I don't have a depth of gt then I can't calculate the absolute distance? In other words, this scaling factor can only be calculated when doing evaluation.

If you don't have the ground-truth then you cannot use this median scaling.


Update CLI for redisenterprise 2023-03-01-preview using aaz

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

redisenterprise

Hi @revanthballa1188, please clean up the commands in generated and manual folders as well. BTW it seems not necessary to update quantum extension

@kairu-ms , I did not understand what you said? what do you mean by clean up generated folders?
I will remove the quantum folder.
Also, I need help with removing the database_name as required for the commands, It should be taken automatically from the resource as we have just one default now right now


Standardize type declaration indentation with tabs

A lot of files use just tabs, a lot just spaces, and a few use a mix. This standardizes it to just be tabs

Can you make eslint check those too? Because when we upgrade the eslint-config-chartjs it will want spaces.

Can you make eslint check those too? Because when we upgrade the eslint-config-chartjs it will want spaces.


Fixing AppHarbor Build Issue

For posterity, here is where I got this solution from: https://support.appharbor.com/discussions/problems/78633-cant-build-aspnet-mvc-project-generated-from-vstudio-2015-enterprise
And specifically from this commit: https://github.com/runesoerensen/roslyn-sample/commit/ba145b4c164e07e9bb2c931581ebac90863775bc


In the StorageAccountHostsMixin class of the python sdk, there is no correct way to parse account name from the Azurite storage blob endpoint

Error Description:
Azurite is an Azure Storage emulator.
In the StorageAccountHostsMixin class of the python sdk, there is no correct way to parse account name from the Azurite storage blob endpoint.
In Azurite, storage blob endpoint constructed is shown as following:
https://<IP_ADDRESS>:10000/<account-name>/<resource-path>
In Azure SDK for python, storage blob endpoint constructed is shown as following:
https://<account-name>.blob.core.windows.net/<resource-path>
To Reproduce:
A demo code is shown as following:
from azure.storage.blob import BlobClient

account_url = "https://<IP_ADDRESS>:10000/devstoreaccount1"+"/foourl/barurl"
access_key = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
blob_client = BlobClient(account_url=account_url, credential=access_key)
print(blob_client.account_name=="devstoreaccount1")

Error Track:
The process code for parse account name from the Azurite storage blob endpoint is shown as following:
https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/storage/azure-storage-blob/azure/storage/blob/_shared/base_client.py#L87

Expected Behavior:
there is a correct way to parse account name from the Azurite storage blob endpoint in python SDK. Finally, able to successfully access azurite.
@dw511214992, @ramya-rao-a, @kyle-patterson, @tg-msft for notification.

@ramya-rao-a, @kyle-patterson, @tg-msft, @dw511214992 Any ideas about this issue?

@tasherif-msft Do you have any progress on this issue?

@tasherif-msft Do you have any progress on this issue?

Hi @zzhxiaofeng @v-xuto, sorry for the long delay on this but I wanted to provide you with the solution to this if you were still looking.
In the sample, you are setting the account URL to also contain the blob path. This is not valid as the account URL should just be the plain account URL, https://<IP_ADDRESS>:10000/<account-name> in this case. To provide the blob path, you can either use the other params of the BlobClient constructor or the from_blob_url function on the BlobClient. Here is a sample which assumes your container is called foourl and your blob name is barurl (based on the provided sample).
from azure.storage.blob import BlobClient

account_url = "https://<IP_ADDRESS>:10000/devstoreaccount1"
access_key = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="

# Option 1: Provide container name and blob name to BlobClient. These are required parameters anyway
blob_client = BlobClient(account_url=account_url, container_name="foourl", blob_name="bar_url", credential=access_key)

# Option 2: Use the from_blob_url helper function
blob_url = "https://<IP_ADDRESS>:10000/devstoreaccount1" + "/foourl/barurl"
blob_client = BlobClient.from_blob_url(blob_url, credential=access_key)

Hopefully this helps and let me know if you have any questions. Thanks!

@jalauzon-msft Thanks for your contribution. This issue has been fixed. I will close the issue now.


Merge latest Compatible to berkeley on 9/8/2023

Explain your changes:
This PR merges latest compatible to berkeley to resolve the merge conflict in #13654
Explain how you tested your changes:
*
Checklist:

[ ] Dependency versions are unchanged

Notify Velocity team if dependencies must change in CI

[ ] Modified the current draft of release notes with details on what is completed or incomplete within this project
[ ] Document code purpose, how to use it

Mention expected invariants, implicit constraints

[ ] Tests were added for the new behavior

Document test purpose, significance of failures
Test names should reflect their purpose

[ ] All tests pass (CI will check this if you didn't)
[ ] Serialized types are in stable-versioned modules
[ ] Does this close issues? List them

Closes #0000

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!approved-for-mainnet


can clientconfig be used for Context settings?

Hi,
I'm using CC now for managing the emailaddress where forms are being send to.
Right now i need a different email address,  for different contexts.
Is it possible to create a setting for that has the same key but a different value per context?
TNX
RDG

Not with ClientConfig. See #4


portaudio and punkt issue

First of all, what you have done here is awesome. Thank you. This is by far one of the most simple ways to run ollama with whisper that I have seen. But I have run into a few issues.

When installing the dependencies via requirements.txt, pyaudio had issues installing, due to portaudio not present. Installing portaudio separately resolved the issue. Is it possible to include that in the requirements.txt file?

when running app.py, it seems like punkt is missing. Installing it seemed to sort it out.

Is there a way to have it install using a different default model instead of llama2. Something more lightweight like phi2? I'm trying to run this on a pi 5.

ollama = Ollama(model="model_name")
Use ollama as llm property in ConversationChain

Hi @adjei7,
Thanks for your comment, here is some advices:

Can you try using poetry instead, this project was built and test under poetry env
This punk downloading is required for the first time of nltk running
You can use Ollama with model specific name as @exportio's comment

Hey @adjei7
You could try this: https://github.com/huggingface/parler-tts


Ethan: Watch (?-i:8YC547KB)(?# gg.sports.gg)

Ethan requests the watch of the watch_keyword (?-i:8YC547KB)(?# gg\.sports\.gg). See the MS search here and the Stack Exchange search in text, in URLs, and in code.

(?-i:8YC547KB)(?# gg\.sports\.gg) has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Rejected by Ollie in Charcoal HQ. Reason: 'Domains that have been seen in spam, meet the MS criteria, and have a relatively low FP rate can be watched.  They'll also catch more spam than a more specific watch such as this one.'


[WiP] Reversion transaction

Wrapping Page._apply_revision in transaction should avoid damages if reversion fails
Fix #3811

@mkoistinen @ojii I'm a bit halted here. Can't understand why reverting in the transaction (https://github.com/divio/django-cms/pull/4235/files#diff-960fbbc1702d77aa625a17bcb0c37191R1341) does not work on MySQL (and mysql does not make any easier by not having a query error log ...)

how does it fail? Does it raise an exception (inside the try/except)? Have you tried enabling db debugging in django and looking at connection.queries, maybe they have a hint.
General (not related to your problem) observations:
Instead of blank try/except and returning True/False/None, why not let it raise the exception and check for True/False/Exception in the calling function? I find using bool/None as return values a bit risky (due to the falsyness of None).
And a small nitpick, instead of assertTrue(x != y), use assertNotEqual(x, y).
PS: Installing mysql :(

@ojii transaction is set to needs_rollback=True during target_revision.revert(True) (here https://github.com/django/django/blob/stable/1.8.x/django/db/transaction.py#L249) with no error reported by the mysql driver apparently because savepoints id has None value (https://github.com/django/django/blob/stable/1.8.x/django/db/transaction.py#L200) :/
I tried debugging this with no success, any idea?
Letting the Exception bubble might be an option, at this stage I just tried to see if using a transaction would fix the issue (which it does, except the weird mysql behavior ...)

I'll try to dig in as soon as I get --parallel to work with mysql in better-tests ;-)

For the scope of this PR you can just run cms.tests.reversion_tests.ReversionTestCase.test_undo_redo

@mkoistinen the only solution is to ignore transactions in case of Mysql
Problem is this set of queries:
             DELETE FROM `cms_page_placeholders` WHERE `cms_page_placeholders`.`page_id` = 1
             SELECT `cms_page_placeholders`.`placeholder_id` FROM `cms_page_placeholders` WHERE (`cms_page_placeholders`.`placeholder_id` IN (2, 3) AND `cms_page_placeholders`.`page_id` = 1)
             INSERT INTO `cms_page_placeholders` (`page_id`, `placeholder_id`) VALUES (1, 2), (1, 3)

Probably due to isolation level, the last one fails due to duplicate values, which should not occurr because the records is dropped in query n. 1 ... I give up on understanding the mysql behavior

Closing this in favor of new reversion support in 3.2.1

Coverage decreased (-0.2%) to 90.898% when pulling fba86f5bac120012c160600557d61da5491c29ae on yakky:feature/reversion_transaction into 63d518cfc278ab10a1613c7f0a1f31deac718939 on divio:support/3.1.x.

Coverage decreased (-0.2%) to 90.898% when pulling fba86f5bac120012c160600557d61da5491c29ae on yakky:feature/reversion_transaction into 63d518cfc278ab10a1613c7f0a1f31deac718939 on divio:support/3.1.x.

Coverage decreased (-0.2%) to 90.898% when pulling fba86f5bac120012c160600557d61da5491c29ae on yakky:feature/reversion_transaction into 63d518cfc278ab10a1613c7f0a1f31deac718939 on divio:support/3.1.x.


Update name of the parameter "table_structure_options.do_cell_matching" to better reflect meaning

Currently from it's name it is unclear what it should do, and we need to give it a better name.
Actutal meaning:
do_cell_matching = True - TableModel uses page cells, generated by parser, for post processing and cell text matching.

True - is default.
Post processing improves structure recognition in many difficult cases for tables.
This should be used with docling_parse_backend

do_cell_matching = False - TableModel doesn't do post-processing.

TableModel uses inferred table-cells bboxes to request PDF backend a text from given bbox.
This should be used with pypdfium2_backend.
Make sure it's False when OCR enabled, otherwise PDF backend will return empty string on a given bbox, because there are no programmatic cells with text.

@maxmnemonic @cau-git Is this something we can close with the docling-core v2?

Looking at it...

@PeterStaar-IBM, @cau-git, I have a better name - respect_pdf_cell_boundaries, that would reflect its meaning better.
However this parameter largely not in use, by default it's enabled, because our docling-parse works fine and produces good and discreet PDF cells. So I propose we don't change it, and close the issue.

@maxmnemonic @cau-git


R image: update python, pangeo-notebook and rstudio-server

Update these core packages to recent versions. First time we update these since 2022.
See https://github.com/oceanhackweek/jupyter-image/issues/90#issuecomment-2474926338
This update is in support of the OHW-espanol event in a couple of weeks. ref https://github.com/oceanhackweek/Hub-Management/issues/6

@abkfenris the build_push check failed:
Traceback (most recent call last):
  File "/opt/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

I built the image locally (make r-build) without any problems. Could you help me debug this??
Thanks!

You'd think conda would be able to find conda...

Hmm, it's a pretty old miniconda image, so I wonder if there are somehow multiple versions of conda conflicting.
--- a/r/Dockerfile
+++ b/r/Dockerfile
@@ -1,5 +1,5 @@
 #syntax=docker/dockerfile:1.3
-FROM continuumio/miniconda3:4.12.0@sha256:58b1c7df8d69655ffec017ede784a075e3c2e9feff0fc50ef65300fc75aa45ae
+FROM continuumio/miniconda3:24.9.2-0@sha256:39b820af05f0f9bb986ca145a0d077bbd3bd866256277dd869fefb93bd03cd21

 ENV NB_USER jovyan

I tried changing a handful of things, but in my tinkering this morning I couldn't get Rstudio to launch from Jupyterlab by running make r-lab.

Thanks. I should've tried to running make r-lab too, but forgot about it. I tried it now, and sure enough, I run into the same error as on the GH action (and same one you're seeing, presumably):
File "/opt/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

Hmm.

I landed on a solution: update conda before doing conda install. After adding conda update -n root conda && \ right before conda install, I'm able to launch jupyter lab with make r-lab!
But on to the next problem: when I launch the RStudio kernel from jlab, I get a 500: Internal Server Error. Sigh.
And the build_push check failed with the same error as before. So the conda update fixed that immediate problem on my end, locally, but not on the CI. Hmm.
Actually, re-reading your message:

I tried changing a handful of things, but in my tinkering this morning I couldn't get Rstudio to launch from Jupyterlab by running make r-lab.

I now see that you were already at that stage, where you could launch jlab but not launch Rstudio from jlab. I thought what you meant was that you couldn't launch jlab. I'll keep digging.

I updated the miniconda image to the version you listed, then explicitly added two library deb packages that were causing more subtle errors (including with nb_conda_kernels, I think): libarchive13 and libfmt-dev.
The CI tests are passing now! But Rstudio is still not launching successfully from jupyter lab. Rather than an immediate 500 error, as before, there's a wait of a few seconds, then the 500 error plus "The error was: could not start rstudio in time".
I added libarchive13 to try to address this error when launching jupyter lab with make r-lab:
conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory)

This error doesn't prevent jlab from launching and behaving normally (it seems). But I figured it's something I could address. Adding libarchive13 didn't fix it (I also tried libarchive-dev). I'm going to try one more strategy I just ran into to solve the libarchive.so error.

If nb_conda_kernels is causing issues, we should be able to get rid of it for this image. It shouldn't be necessary to run Rstudio. I think I tested with it removed, but still couldn't get anything other than a 500 trying to launch Rstudio.

I might explore the links off of here: https://discourse.jupyter.org/t/a-template-repository-for-running-latest-rstudio-r-on-mybinder-org-your-jupyterhub/12235

I've pushed my last commits for the day. I've now eliminated all conda / libmamba errors. The last set of errors involved conflicts between libmamba and libarchive coming from conda-forge vs main channels.
There are no clear errors left in the run-time logs from make r-build and make r-lab, other than these warnings on make r-lab, which I assume are not critical.
[W 2024-11-14 23:25:32.326 ServerApp] No web browser found: Error('could not locate runnable browser').
...
[W 2024-11-14 23:26:33.071 LabApp] Could not determine jupyterlab build status without nodejs

BUT, RStudio still doesn't launch :disappointed:. I still get this behavior: there's a wait of a few seconds, then the 500 error plus "The error was: could not start rstudio in time".
And the errors emitted when I click on the RStudio launcher in Jupyter Lab:
r-1  | [E 2024-11-14 23:26:54.725 ServerApp] {
r-1  |       "Host": "<IP_ADDRESS>:8080",
r-1  |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
r-1  |       "Referer": "http://<IP_ADDRESS>:8080/lab",
r-1  |       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0"
r-1  |     }
r-1  | [E 2024-11-14 23:26:54.725 ServerApp] 500 GET /rstudio/<EMAIL_ADDRESS>15033.48ms referer=http://<IP_ADDRESS>:8080/lab

Sigh, the build_push check failed; I was expecting it to work. But it looks like it's a temporary blip on the CI:
ERROR: failed to solve: rpc error: code = Unknown desc = write /opt/conda/lib/python3.12/site-packages/babel/locale-data/ta.dat: no space left on device
Error: buildx failed with: ERROR: failed to solve: rpc error: code = Unknown desc = write /opt/conda/lib/python3.12/site-packages/babel/locale-data/ta.dat: no space left on device

I left nb_conda_kernels on the conda env, and everything is fine.

I might explore the links off of here: https://discourse.jupyter.org/t/a-template-repository-for-running-latest-rstudio-r-on-mybinder-org-your-jupyterhub/12235

I saw that earlier, but I didn't look into it.

FYI the "no space left on device" error I ran into earlier turned out to be a very real error.
It looks like it's a docker error involving the GitHub limit of 14 GB for images. But I implemented a solution (adding this step: rm -rf /opt/hostedtoolcache in build-image.yml) that apparently worked, since the build check got through. For reference:

https://github.com/orgs/community/discussions/25678#discussioncomment-5242449
https://github.com/actions/runner-images/issues/2875

That's just a CI problem, though. I'm still working on the core problem of not being able to launch RStudio from Jupyter Lab in local tests. I'll try some things today.

Ya, I've run into that a few times as they update the underlying image and have lots of language versions installed.
It might also be possibly to build on top of rocker, and then launch right into RStudio. https://github.com/search?q=repo%3A2i2c-org%2Finfrastructure rstudio&type=code

https://infrastructure.2i2c.org/howto/features/rocker/

I started looking into rocker a couple of days ago. It does look like the way to go going forward. But for me at least, making that switch now would be taking on too much beyond my depth for the upcoming event (starts next Monday 11/25).
I've tried downgrading pangeo-notebook, the Python version and the rstudio-server package, but still haven't made progress. I'm very, very close to giving up and falling back to our old setup. I'll ask on the 2i2c Slack (that I just joined) and see if that leads somewhere, before giving up by tomorrow.

@emiliom :wave:
@jnywong shared this in our 2i2c Slack, and I took a quick look.
When I try this PR, the jupyter server instance throws a traceback:
2024-11-20T09:43:44.819637Z [rserver] ERROR system error 13 (Permission denied) [path: /home/jovyan/.cache/rstudio, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:848; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:845
TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/rstudio/logging.conf'. Logging to '/home/jovyan/.local/share/rstudio/log/rserver.log'.
rserver[1831]: ERROR system error 13 (Permission denied) [path: /home/jovyan/.cache/rstudio, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:848; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:845

If you chown the cache directory such that NB_USER has the appropriate permissions, you will then be able to launch RStudio from within the image:
RUN chown -R ${NB_USER} ${HOME}/.cache
USER ${NB_USER}

Long-term, I agree that moving to something like Rocker seems like a good suggestion, but hopefully this unblocks you? Feel free to ping me if you run into any more problems. I'm not an R expert, but I am very familiar with containers & the Jupyter stack.

OMG, that did it!!! Thank you so much @agoose77!! This is a huge relief for me.
I'll update the PR. Again, THANK YOU!!

@emiliom a pleasure! I haven't tested beyond "does it launch into RStudio". Keeping fingers crossed :crossed_fingers:


Fix several issues with search linking

Patch search result return with project and subproject urls
Provide fallback to old query in more cases
More defensive tests whether we should even use new search, or if theme has
mangled our attempts to insert information

:+1:


请问如何关闭断点下载

暂时不太需要这个功能…但文档找不到如何设置强制关闭断点下载的支持

在下载之前通过 FileDownloader#clear(id, targetFilePath)方法可以有效的删除断点数据，这样接下来下载该文件就不再从断点下载了:
/**
 * Clear the data with the provided {@code id}.
 * Normally used to deleting the data in filedownloader database, when it is paused or in
 * downloading status. If you want to re-download it clearly.
 *

 * Note: YOU NO NEED to clear the data when it is already completed downloading,
 * because the data would be deleted when it completed downloading automatically by FileDownloader.
 *

 * If there are tasks with the {@code id} in downloading, will be paused first;
 * If delete the data with the {@code id} in the filedownloader database successfully, will try
 * to delete its intermediate downloading file and downloaded file.
 *
 * @param id             the download {@code id}.
 * @param targetFilePath the target path.
 * @return {@code true} if the data with the {@code id} in filedownloader database was deleted,
 * and tasks with the {@code id} was paused; {@code false} otherwise.
 */
public boolean clear(final int id, final String targetFilePath)


Support "offline" blob uploads with detached container

Feature
Our recommended path for offline document creation using a detached container does not currently work with SharedTree.  This is because the Fluid runtime currently does not support uploading blobs while in a detached state.  This is problematic for the SharedTree DDS, which uses Blobs both at the public API layer as well as internally while summarizing.
One potential solution is for the runtime to permit blob uploads while detached, temporarily storing them in memory or persistent storage.  During attachment, these blobs will be uploaded to the service prior to sending the initial summary and the IFluidHandle patched with resulting 'absoluteUri'.  (For correctness, absoluteUri should be made opaque.)

can someone expand the scenarios here? is it just detached? what about serialize? same session only? different sessions?

Hi @anthony-murphy, thanks for looking at this.  Here's more information about this scenario (along with some thinking of what a solution might look like)--hopefully it will address some of the questions.
The scenario we'd like to support is as follows--the host allows editing a detached container, and relies on ExternalContainer.serialize() along with some other storage option of its choice (e.g. IndexedDB) for persistence.  At some point in the future, the host may choose to attach the container to a driver, assumed here odsp-driver.
The following issues currently exist with blobs:

When detached, runtime.uploadBlob() doesn't work, and throws an exception.
Assuming this problem were fixed and runtime.uploadBlob() stored the blobs locally, the blob handles returned would end up in ops (in the serialized container), but wouldn't refer to data on Sharepoint.  This would create a problem when the container was later attached to odsp-driver (i.e., the blob handles would be meaningless for other clients that don't have or know about the local blobs).

On point (2)--when we attach later to odsp-driver, we'd like to upload all the blobs to Sharepoint and then replace the old handles in the ops with the new Sharepoint handles, but there's some question around the best way to do that.
Here's a sketch of a solution that I've proposed elsewhere--at some point when the host creates the detached container, it provides an instance of an interface (maybe IDetachedBlobStore) with the following methods:

storeBlob() -- stores blob of data, returns a (serializable, string or similar) handle
retrieveBlobForHandle() -- given handle, returns blob
listBlobHandles() -- returns a list of all handles for all stored blobs

If the container is in a detached state, then the Fluid runtime uses storeBlob() and retrieveBlobForHandle() as listed above to implement runtime.uploadBlob() and IFluidHandle.get().
When the container is attached, listBlobHandles() is used to upload all the blobs to Sharepoint.  At this point, the runtime has a correspondence map from the old, local handles to the new Sharepoint handles.  So the question at this point is how to map old handles in ops to the new handles.
Here's one idea I've floated--the runtime somehow attaches the map with the container data in the driver, either as an op or using the container quorum feature, the outcome is that any client attached to the container can get the map.  We assume that every DDS that stores handles in ops will deserialize them to an IFluidHandle via IFluidSerializer.parse().  The idea is that under the hood, if the container is attached, IFluidSerializer.parse() checks if the handle is a local handle, and uses the knowledge of the mapping to translate to a Sharepoint-meaningful IFluidHandle.
Anyways, thanks again for looking at this, and please let me know if I can clarify the scenario further.  The approaches above are just to help sketch out what we're hoping for (i.e. as an API consumer), I'm definitely assuming your thinking on how to go about doing it is likely better than mine, here.
Thanks again!

Also--another couple of quick comments:

Currently it's a pattern with odsp-driver for new container instances to allow the user to edit the container locally in a detached state before it's sync'ed to odsp.  This means that if the user does anything in this window that triggers blob upload, currently it would cause a failure.  Am imagining that a solution to the above problem could be leveraged to cover this edge case.
The high level scenario:

the host allows editing a detached container, and relies on ExternalContainer.serialize() along with some other storage option of its choice (e.g. IndexedDB) for persistence
might also be addressed by allowing a container to be detached from one driver and reattached to another one, however I believe the problem of moving the blobs will still be essentially the same as that described here, using ExternalContainer.serialize().

Thanks again so much.

There are multiple directions we can go here, it would be great to first understand constraints.
Some number of questions:

The timeline, to dogfood and to production. Note that any server changes (if they are required) take a long time to production, so we likely need to look into both long term strategy here as well as short term solution we can achieve in given time frame.
User experience while file is not fully uploaded. With big sizes we can't rely on single roundtrip to upload 400Mb of data, and there is no way to get streaming API stood up for file creation in this short time frame, so we should rely on multiple uploads and presence of the file in some state during this time, as well as possibility of client crashing / being closed. There are couple questions that come here:

What should be user experience if user attempts to open such partially uploaded file?
Do we want to give it some non-fluid extension and rename at the end of the process not to allow user to open it with Whiteboard app?

We would need to have some basic understanding on distribution of blob counts, sizes, and type of data (binary vs. unicode) to better understand pros/cons of various solutions.

As for workflow, it likely will look like this:

A file is created, either zero-byte file, or with some initial (empty) summary (to make it valid fluid file).
A number of binary blobs are uploaded, one by one sequentially (ODSP does not support concurrent uploads). These are called "attachment" blobs.

this is the most efficient process to upload bits for binary large payloads, and very inefficient for small blobs.
attachment blobs are not included into TreesLatest (aka snapshot) response when client loads this file later - they are loaded by client on demand.

An incremental summary upload can be used (see issue #5690) to upload number of blobs in aggregate, i.e. many blobs per request

this is efficient for small blobs, especially if they are not binary, i.e. do not need base64 encoding.
blobs uploaded this way would be included into TreesLatest (aka snapshot) future response. This might be not desirable, and it's possible to implement new ODSP feature to mark some of these blobs to be treated as "attachment" blobs.

Once the process is complete, a final summary is uploaded, and file is renamed to have .fluid extension (if needed, depending UX question / answer above).

The work on runtime side would include

Ability to "upload" blobs in detached state and get handles back.

With us moving toward hiding handle.absolutePath (it's already marked deprecated), we can make absolutePath property throw if it is accessed before blob was truly uploaded and got ID. We can ensure though such checks that serialization of handles happens only after blobs were actually uploaded (i.e. DDSs do not attempt to serialize handle when handle is stored in DDS, only at summarization time).
That said, current design assumes that once container starts file creation process, container switched to "attaching" state and all DDSs start to create ops in response to local changes. This means serialization of handles and need to have blob IDs, potentially before blob upload was completed. Some redirection schema is possible to avoid this problem (similar to how it is described above), but it complicates whole picture. We can avoid these problems if FF works on Container that host has no access to, i.e. it is created by cloning or from earlier serialized state (we leverage similar pattern in data loss prevention flow).

I'm sure I'm missing some key things here, but worth writing it out as is - I'll keep adding to this write up.
Overall - it's a lot of work no matter how we look into it.

@coclauso, @DLehenbauer, I've updated my long post above. It would be great for you to take a look and comment, especially on perf aspect of it - I think it would be the most challenging part of whole effort. We could run some experiment with some workloads approximating what users would experience RE uploading that much data and that many blobs in one go, to get a scenes if current approach will work, or not.
Number of roundtrips likely can be addressed in reasonable amount of time by adding multi-blob upload API or leveraging summary blob upload path. But there is really no solution for bandwidth here, and we can't leverage multiple parallel uploads (as SPO can't handle it), so we are limited to what single connection can give us. Note that existing file flow already serializes all attachment uploads, so you can experience it in existing product by ensuring you create scenario that hits many uploads at once.
If the whole process and perf looks reasonable, we could start breaking it into individual work items.

Hi @vladsud ,
Thanks for your response.  I’ll try to go through this and add my own comments, for whatever light they can shed.  Please point out anything I don’t address well.  It’s also possible that @DLehenbauer can reply to some portions.

The timeline, to dogfood and to production

I’ll start a separate communication for this with folks who can provide more details.

User experience while file is not fully uploaded

What should be user experience if user attempts to open such partially uploaded file?

Do we want to give it some non-fluid extension and rename at the end of the process not to allow user to open it with the app?

I could imagine something like .[extension].partial or .[extension].incomplete until it’s fully uploaded.  On our side, we’re planning to delete the local data once it’s been transferred to Sharepoint.  We should make it such that in the case where the app crashes mid-upload, then that wouldn’t happen, and the next time the app was launched, the upload would retry.

We would need to have some basic understanding on distribution of blob counts, sizes, and type of data (binary vs. unicode) to better understand pros/cons of various solutions.

This might be partially addressed by a comment that follows.  In our application, we reserve blobbing for large chunks of data.  I believe they’re binary (i.e. we don’t base64 encode) but could double check.

As for workflow, and possible designs, here are some thoughts:

Most of this seems fine, I’d comment on this, though:

An incremental summary upload can be used (see issue #5690) to upload number of blobs in aggregate, i.e. many blobs per request

Right now, we’re using blobs to refer to data that should be stored as Sharepoint attachment, and our application has heuristic code to only blob large chunks of data, not small chunks.  My suggestion here would be for simplicity not to consider the case of a “small blob” at the runtime level and leave it to the API consumer to ensure that only large blocks of data are blobbed.

Once attachment process starts, new blob creation would need to wait till file is created (but not wait for whole creation sequence to finish, i.e. final snapshot upload).

I think the question here is, while the document is being transferred to Sharepoint, is it acceptable for some app functionality to block until the transfer is complete?  If we decide it is, for simplicity of the API consumer, I'm wondering if the best way to expose this to the app would be to have the async runtime.uploadBlob() function block until the attachment process is complete?

Full creation process would need to either grab the summary at the start of the process and upload it after all attachment blobs were uploaded.

If allowing blob creation during the attachment process rather than blocking isn’t too much harder, it might be worth doing.  It could provide a smoother user experience during transfer.
Let me know if there's anything else I can provide that can help.
Thanks,
Cooper

Also, on this:

Number of roundtrips likely can be addressed in reasonable amount of time by adding multi-blob upload API or leveraging summary blob upload path. But there is really no solution for bandwidth here, and we can't leverage multiple parallel uploads (as SPO can't handle it)

It's current the case that attaching a container to Sharepoint takes a number of seconds (approx. 5 or so), generally apps allow the user to edit the document seamlessly during this time so the experience is still good.  If this fundamental SPO bandwidth bottleneck were just manifested as a longer attachment waiting time, say a few seconds more, then I'd imagine the existing mechanisms that mask this from the user would be fine here.
This issue relates back to these earlier comments:

I think the question here is, while the document is being transferred to Sharepoint, is it acceptable for some app functionality to block until the transfer is complete?

If allowing blob creation during the attachment process rather than blocking isn’t too much harder, it might be worth doing. It could provide a smoother user experience during transfer.

It could be the right approach here is to allow for the app to create additional blobs during attachment.

Thanks! So here is modified plan:
Pre-work:
0. Modify upload blob API to work in detached mode by

holding data to be uploaded
Issuing handles that go through redirection (i.e. in the form of /_blobs/<guid>, where guid is the unique local ID of the blob, not current blob ID from service). This is required for serialization of detached container. Back compat needs to be considered with existing format.
Implement detached serialization & rehydration of binary attached blobs (they would need to be base64 encoded if using current approach, see more below).

Upload process:

Container transitions to Attaching state.
A file is created

It's zero byte file. This is natively supported by runtime/loader/driver, i.e. if there are no attachment blobs to upload, then we leverage existing workflow of creation from summary, otherwise we use this workflow.
It has some non-fluid extension. FF supports this flow, but app controls naming / renaming.

Upload every attachment blob, sequentially.

this process is done when there are no non-uploaded blobs. This includes any new blobs that showed up while # 1-3 was in flight.
Making local changes while upload is happening is supported, however they will not be reflected in history of the file (as ops), as snapshot is grabbed at the very end.

Summary is created, and uploaded to file.

Container is transitioned to Attaching state at the beginning of this process.
#5690 work is used to get incremental summaries if needed (would be nice to confirm if that's a requirement).

When summary upload is is over

Container transitions to Attached state.
File is renamed to have .fluid extension (performed by app).

Detached serialization / rehydration is easy and small part of it (see prep work), however size should be considered (i.e. is it Ok to serialize all the content of container into single string). Even if it works, it would likely be super-inefficient (too many memory copies, base64 encoding). Future plan likely needs to involve something more complicated, where serialization is done through some interface that supports working with individual binary blobs (this should be considered, but handled outside of this item as future improvement).

Detached serialization / rehydration is easy and small part of it (see prep work), however size should be considered (i.e. is it Ok to serialize all the content of container into single string). Even if it works, it would likely be super-inefficient (too many memory copies, base64 encoding). Future plan likely needs to involve something more complicated, where serialization is done through some interface that supports working with individual binary blobs (this should be considered, but handled outside of this item as future improvement).

It sounds like we're deciding to boot implementing this suggestion from earlier:

Here's a sketch of a solution that I've proposed elsewhere--at some point when the host creates the detached container, it provides an instance of an interface (maybe IDetachedBlobStore) with the following methods:...

A comment I would make is that if this isn't too difficult, it might be worth doing earlier, the reason is that if a product ships at any point with blobs in the serialized snapshot, then whatever functionality is required to deserialize them from this source has to be maintained indefinitely into the future, so there could be some value in doing this in the first version.

Agree, though I'd point out we are already sort of in that bucket (if anybody uses string-based functionality, then their drafts would either be lost, or we would need to build an adapter to convert it to this new format).
Clearly addressing it sooner is better :)
I'd say that we can go with prototype (hidden under experimental / deprecated façade) that uses current string approach to get feedback about overall process (and find first set of bugs), and keep working on moving toward something like IDetachedBlobStore
That said, I'd modify it to have only 4 methods (blob-related):

read(blobid) -> content
write(content) -> blobId.
readSummary()
writeSummary()

No listing, no handles.
Maybe add delete() for future (if we ever implement GC for detached state, host can delete everything related to a file on successful file creation).

@danielroney - FYI: Breakdown of work items below.
Phase1: Support blobs while detached

Define IDetachedBlobStorage with create/read methods
Plumb IDetachedBlobStorage from host to ContainerStorageAdapter
Use IDetachedBlobStorage to persist create/read blobs when detached.
Tests (incl. mock implementation of Node.js)

Phase 2: Upload blobs on attachment

Change ODSP workflow when attaching with blobs

Create 0B file with non-.fluid extension
Process queue of pending blob uploads
Create/upload summary
Rename file to .fluid extension

Tests for new creation flow
Implement redirection for detached blob URLs

Implement hook for uploading blobs on Container.attach()
As detached blobs upload, add their before/after URLs to a table and reclaim detached storage.
Use the table to resolve requests for "before" URLs.
Add snapshot/load APIs to IDetachedBlobStorage
Include blob redirection table in summary.

Tests for Blob redirection

Open issues:

Figure out the appropriate layer to perform uploading of detached blobs and triggering the alternate ODSP creation flow
Lower priority, but we should consider if detached blobs should be deduped and/or GC'ed.


Add support for Objective-C

[ ] Extend NSObject
[ ] Add documentation for using in an Objective-C project
[ ] Update README to refer to Swift and Objective-C applications

What's the word on this ?
How would I go about trying to do this on my own, what is supposed to extend NSObject, etc ?

This should be a fairly straightforward step. Making all aspects of this API available from Objective C will only require

NSObject inheritance
@objc markers
where such functionality is desired

Source

I'm going to close this issue. We'll be embracing Swift moving forward--especially open source Swift and Swift 3.0. By dropping explicit bridging support to Objective-C, we can better embrace Swifty conventions.
We understand that many legacy apps are still using Objective-C, but may want to use this SDK. In these situations, it is best to take advantage of the ability to mix Swift and Objective-C code in the same project.


another fix to OSS tests

Test Plan:

Circle should be gree
./gradlew  :ReactAndroid:assembleDebugAndroidTest

By analyzing the blame information on this pull request, we identified @cjhopman and @mkonicek to be potential reviewers.

@facebook-github-bot shipit

Thanks for importing. If you are an FB employee go to Phabricator to review.


add husl requirement

Can you provide an explanation why this change is needed?

I can't "pip install ggplot" and immediately import ggplot to use it without installing husl first. I get "ImportError: No module named 'husl'".

See https://github.com/yhat/ggplot/pull/369
Has the location of the husl.py file been changed?

After I installed ggplot in a fresh virtual environment on python 3.4.3 using the command
pip install ggplot

I faced the same issue as documented by @mcsalgado.
For new users, this might be confusing to have to read a traceback and figure out to install husl as well. The issue can be wholly avoided if we add husl to setup.py as a requirement. its already specified in travis.yml as a dependency


Override LazyTraceThreadPoolTaskExecutor.getQueueSize() to return del…

fix for https://github.com/spring-cloud/spring-cloud-sleuth/issues/2315

Hey, thanks for the PR @vg2902 . Can you write a test for this one too please?

Hey, thanks for the PR @vg2902 . Can you write a test for this one too please?

np, will add one shortly

I've added a test, please have a look


workbench.action.editor.nextChange is slow in 1.75.0

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.75
OS Version: Windows 10

Steps to Reproduce:

Open a project in VSCode that's versioned by git.
Make a change.
Use workbench.action.editor.nextChange, (Alt+F5).
Measure the time it takes to move your cursor to the next change. You should notice a significant delay in version 1.75 compared to version 1.74.

I've only tested this in a single repository with ~30k LoC and a few hundred commits.

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.75.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

@hediet is this something that falls under your area of ownership?

Can confirm. In 1.74 it was instant, whereas now it takes >1sec even in a new project of a single file.

I'm also seeing this perf regression with 1.75.
Side observation: as noted, alt+f5 is slow in 1.75. However, alt+f3 (related functionality -- 'Show Next Change') still feels instant. (Not sure if this is relevant or helpful.)

For me both workbench.action.editor.nextChange and workbench.action.compareEditor.previousChange is slow. It doesn't matter which key combination I bind this command to. Repository size seems to have no (noticeable) effect. Disabling GitLens has no effect either.

I'm seeing this issue too, with both workbench.action.editor.nextChange and workbench.action.editor.previousChange.
@hediet I noticed you assigned the diff-editor label. It's worth noting that the comparable commands for the diff editor (workbench.action.compareEditor.nextChange and workbench.action.compareEditor.previousChange) do not seem to be exhibiting the same slowness, so this issue seems to only be happening in the editor, not the diff editor.

I only just noticed this myself after switching my keybindings because editor.action.dirtydiff.next & editor.action.dirtydiff.previous stopped working since 1.76.0. It's frustrating having to open up the diff editor every time to see things quickly.


The service returns a 500 error code when the token is invalid

The issue seems to be related to the gateway

The error is due to the pipeline assigned to the service


Encoder error

plugins.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.6")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.2.0")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.10"

build.sbt
ThisBuild / version := "1.0.0"
ThisBuild / scalaVersion := "2.12.15"

val sparkVersion = "3.2.1"

lazy val root = (project in file("."))
  .settings(
    name := "kafka-consumer",
    Compile / PB.targets := Seq(
      scalapb.gen() -> (Compile / sourceManaged).value / "scalapb"
    ),
    libraryDependencies ++= Seq(

      "com.thesamet.scalapb" %% "sparksql32-scalapb0_11" % "1.0.0-M1",
      "com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"
    ),

    assembly / assemblyShadeRules := Seq(
      ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.@1").inAll,
      ShadeRule.rename("scala.collection.compat.**" -> "shadecompat.@1").inAll
    )
)

code
import msgs_pb.msgs._
import scalapb.spark.Implicits._

val spark = SparkSession.builder()
    .master("local[*]")
    .appName("kafka-consumer")
    .getOrCreate()

  val df = spark
    .read
    .format("kafka")
    .option("subscribe", "test-dev-1")
    .option("startingOffsets", "earliest")
    .load()

val decoded = df
    .map(row => {
      val value = Msg.parseFrom(row.getAs[Array[Byte]]("value"))
      value
    })

decoded.show()

msgs.proto
syntax = "proto3";
package msgs;
import "google/protobuf/timestamp.proto";

message Msg {
  fixed64 id = 1;
  google.protobuf.Timestamp created = 2;
  repeated Dev devices = 3;
}

message Dev {
  optional google.protobuf.Timestamp received = 1;
  fixed64 received_device_id = 2;
  fixed64 device_id = 3;
  optional fixed64 parent_id = 4;
}

Works properly with spark 3.1.x and "sparksql31-scalapb0_11" % "1.0.0-M1" but with spark 3.2.x and "sparksql32-scalapb0_11" % "1.0.0-M1" wil fail with
Exception in thread "main" java.lang.NoSuchMethodError: 'void org.apache.spark.sql.catalyst.expressions.objects.Invoke.<init>(org.apache.spark.sql.catalyst.expressions.Expression, java.lang.String, org.apache.spark.sql.types.DataType, scala.collection.Seq, scala.collection.Seq, boolean, boolean)'

and issue is with encoding parsed message in map function
    .map(row => {
      val value = Msg.parseFrom(row.getAs[Array[Byte]]("value"))
      value  // <-- Fails here
    })

One update.
It does work with spark 3.2.0 and only 3.2.1 is not working :(

Spark 3.2.0 and Spark 3.2.1 has binary incompatibilities in its internal APIs, so it is impossible to produce a single version that seamlessly works with both versions of Spark. I have created a new release (1.0.0) compatible with Spark 3.2.1, and updated the docs at https://scalapb.github.io/docs/sparksql#setting-up-your-project. Existing sparksql version 1.0.0-M1 is pretty much the same as 1.0.0 except it is built for Spark 3.2.0.

Solved in 4h 😮
I can only confirm that it works :)
👍