Failed to complete amalgamation for Android

I followed the readme for the amalgamation for android, but failed to build it correctly.
I built the standalone toolchain:
export SYSROOT=$NDK/platforms/android-19/arch-arm
$NDK/build/tools/make-standalone-toolchain.sh --arch=arm --platform=android-19 --install-dir=$USERSPACE/NDK_standaloneToolchain/arm_android19 --toolchain=arm-linux-androideabi-4.9
Then built OpenBLAS for Android according to the instructions the readme refers to:
export PATH=$PATH:$USERSPACE/NDK_standaloneToolchain/arm_android19/bin
make HOSTCC=gcc CC=arm-linux-androideabi-gcc NO_LAPACK=1 TARGET=ARMV7
(Note that I used the stanadalone toolchain I built before as CC, assuming this is why I built it.. not very clear in the instructions)
Then I tried building MxNet amalgamation.
I edited the make file so it would compile using the standalone toolchain:
export CXX=/home/ubuntu/NDK_standaloneToolchain/arm_android19/bin/arm-linux-androideabi-g++
export SYS_ROOT=/home/ubuntu/NDK_standaloneToolchain/arm_android19/sysroot
export INCLUDE=/home/ubuntu/NDK_standaloneToolchain/arm_android19/include/c++/4.9
and edtted line 44:
${CXX} ${CFLAGS} -fPIC -o $@ -c jni/predictor.cc --sysroot=${SYS_ROOT} -I ${INCLUDE}
The additional --sysroot and -I are so it find certain includes in the cc files.
Then I had to comment out line 32 in mxnet_predict-all.cc because I couldn't find emmintrin.h anywhere...
I didn't see any of the includes the readme says to remove.
after finally passing all the compiler errors, the command make ANDROID=1 fails with:
warning: skipping incompatible <path..>/mxnet/3rdparty/OpenBLAS/libopenblas.so while searching for openblas
error: cannot find -lopenblas
I assumed this meant OpenBLAS wasn't build how the make expects it to, so I tried several time to build the standalone toolchain in various forms, and the build OpenBLAS again, to no avail.
I find this quite confusing, as I'm using the same toolchain to build OpenBLAS and the mxnet .so..
Any help much appreciated.

@tqchen Is there any android guy in our team could help on this?

I just managed to get past this incompatible lib issue. It was due to to much garbage in my OpenBLAS playground.. It compiles now, thanks to Leliana (Android example authour). I have more issues, but this was my bad. My apologies.
I'll close this, but if anyone would like to tell me if its OK that OpenBLAS only generates libopenblas.a file and no libopenblas.so? Amalgamation builds, but when I try to use the result .so I get linking errors..

@yonadavs
I am facing similar issues can you help me out?
following is the error when I diod make ANDROID=1
tmp/my-android-toolchain/bin/arm-linux-androideabi-g++ -std=c++11 -Wno-unknown-pragmas -Wall -I/opt/openblas_android -mhard-float -D_NDK_MATH_NO_SOFTFP=1 -O3 -fPIC -o jni_libmxnet_predict.o -c jni/predictor.cc --sysroot=/tmp/my-android-toolchain/sysroot -I /tmp/my-android-toolchain/include/c++/4.9
In file included from jni/predictor.cc:4:0:
jni/../mxnet_predict-all.cc:21:19: fatal error: cblas.h: No such file or directory
#include <cblas.h>
^
compilation terminated.

I git clone openblas again and built it for android. That got it working. Now how to use the /mxnet/amalgamation/../lib/libmxnet_predict.so file which is generated after all the processes.

Hi, I'm encountering the same problem, what do you mean by "too much garbage in my OpenBLAS playground"?
BTW, I'm confused too about OpenBLAS generating only a .a file (is this however the file that needs to be pased to the Amalgamation?)

What I meant was that after trying various builds, and changing some files, it was full of build garbage. try make clean, or delete OpenBLAS, and clone it again.

yes, the .a file is what you want.

Thanks for your quick answer. I rebuilt OpenBLAS and the standalone toolchain but I get the same error (skipping incompatible...)
Any ideas?

Did you make sure to set the CC and CXX as shown in the standalone toolchain instruction? (it's near the bottom of the page).
also, be sure to pass your custom android-gcc as the parameter to to OpenBLAS make.
My make command looked like this:
export PATH=$PATH:/home/ubuntu/NDK_standaloneToolchain/arm_android19/bin
make HOSTCC=gcc CC=arm-linux-androideabi-gcc NO_LAPACK=1 TARGET=ARMV7

Quite sure, I wasn't actually specifying the CXX variable, but it doesn't works even in this case. Could this depend on which android API I choose creating the standalone toolchain? (I'm doing this with android-21).
BTW, more in general, the amalgamation section seems quite outdated w.r.t. the rest of mxnet in the last releases (i.e. the includes that the readme says to remove)

AFAIK the amalgamation readme is a contribution by someone who went through all you are going through now, and posted a guide. I guess this is why it isn't updated.
I assume you mean NDK API 21, and not Android API 21, those a re 2 separate things..
Any way, I don't think there should be a problem with using API 21, as that's how the WhatIsIt demo was built.
It's been a  while since I messed with this, so all I can recommend now is make sure you set the system path var to include your toolchain/bin, and to use the make command as I wrote earlier..

Seems I solved the compilation problem, but not sure why at some point the OpenBLAS library was fine w.r.t. the previous attempts. My guess is that since I started over from the very beginning (standalone toolchain creation, oblas compilation etc...) using always the same terminal instance, all the environment variables were correctly set. I'm available to write what I did step-by-step if someone is interested. Also, remember to check if the C++ standard library is correctly configured in the creation of the standalone toolchain, and try to link it in the amalgamation process (I added -lstdc++ in the Makefile)
Also, it seems that the jni_libmxnet_predict.so is needed in your Android Project to make it work, and not only the libmxnet_predict.so file.

Just to correct the last statement: ONLY the jni_libmxnet_predict.so is needed in the android project.

Precisely, and I spent a lot of time figuring it out since the WhatsThis example was somehow misleading (was it renamed to libmxnet_predict.so?), but maybe this is because I have zero experience with android / ndk.
Also, this should be the solution for https://github.com/dmlc/mxnet/issues/1747 (my problem raised trying to substitute the models in WhatsThis)

@iluvatar593 can you please share the steps you followed for building mxnet for android?


submitting-changes.chapter.md: Suggest adding cosmetic changes to .git-blame-ignore-revs

Closing since the main issue I wanted to fix was resolved in https://github.com/NixOS/nixpkgs/pull/188277 anyway, and since I don't think anybody wants a protracted discussion on git pedantry.

@leungbk I still think it's worthwhile to document this, at least for future reference. I only learned of that feature very recently, and only because I stood in the firehose of nixpkgs pull requests for a while. You can still remove the specific change and keep the update to the manual.


Real-Time, I/O, and Multitasking: Julia for Medical Imaging | Niklas Hackelberg | JuliaCon 2022

Video Title:  Real-Time, I/O, and Multitasking: Julia for Medical Imaging | Niklas Hackelberg | JuliaCon 2022
Video Link: https://www.youtube.com/watch?v=chZfGfAAnpE
Content:
00:00 Welcome!
00:34 Motivation: Medical Imaging
00:50 Motivation: Imaging Scanner
01:36 Motivation: Challenges
03:13 Architecture and Packages
04:40 Network Communication Optimizations
05:40 High Performance Sample Transmission
06:08 MPIMeasurements.jl: Main Data Structures
06:49 Threading Model
08:00 Threading Example

Added, thanks!

Ah @logankilpatrick, I just noticed that I did the timestamp the wrong way around and listed the end and not the start of a section.
Here is the corrected version:
00:00 Welcome!
00:10 Motivation: Medical Imaging
00:34 Motivation: Imaging Scanner
00:50 Motivation: Challenges
01:36 Architecture
03:13 Network Communication Optimizations
04:40 High Performance Sample Transmission
05:40 Imaging Scanner Data Structures
06:08 Threading Model
06:49 Threading Example


IPERF drop-hunter enhancements

Currently IPERF drop-hunter hunts for '0' packet drop. Practically hunting for drop by % (0.01%) is more useful as the absolute 0 drop is a very hard criteria that sometime requires lowering the throughput by a magnitude to meet the condition which could hide the big picture.
While at it there are a few fine tweaks it may consider.

ramp-up time: to warm the system up.
mixed param unit i.e.   { "arg": "bitrate-range", "vals": ["100M-1000M"], "role": "client" } - Can range be a mix of M and G

1 - drop by percentage has been and merged.
2 - Ramp-up time feature has been merged by "--o"
3 - mixed param unit support is unnecessary.


Add a profile selection/display toolbar for PlotWindow + ImageView widget with side histograms

A Toolbar with horizontal, vertical and free profile line with settable width using image.bilinear module that can be attached to a PlotWindow.

A Toolbar to display and control the limits of a Plot.

ImageView widget: plot an image with side histograms (taken from PyMca) with a def main(argv) and if name == 'main': to start the widget and display an image file

Skip GUI tests when DISPLAY is not defined on Linux (e.g., running them on a server)

Closes #69

Fix run_tests.py coverage output

1D ROI usage should not be there.

Profile Menu. It should say "Free" and not "Fee"

Can the radar view have a wider pen? It is difficult to be seen on my screen.

Radar view: It would be convenient to switch to a hand cursor when one can move the viewing zone or (even simpler) to move the rectangle when moving (a left-clicked) mouse in the radar view even if not on the rectangle.

Addressed the 4 comments.
For rectangle width, it is pixels on the screen.


Title: Javascript tool that can Encode & Decode Text

Is your feature request related to a problem? Please describe.
This tool Can Encode as Well as decode text in Several formats.
Describe the solution you'd like.
I have added Several modules that can handle Encoding & Decoding formats such as base64, url-encode, unicode, html entity, binary & hex.
Describe alternatives you've considered.
The text encoding are very essential for cyber-security enthusiasts as they help in various security research such as bypassing Firewalls.
Add any other context or screenshots about the feature request here.

Pease assign to me


Apple.BcUpdate version 1.1

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/111864&drop=dogfoodAlpha

Manual Validation failed with:

[Policy] Hardware

Supporting custom hardware in validation is "Not Planned".

https://github.com/microsoft/winget-pkgs/issues/199504


Not support MV3 native ES background worker

Blocked by https://github.com/w3c/ServiceWorker/issues/1356, otherwise, we need to emit import statements which is not easy in a webpack bundle.

I think that maybe it already works with this, regardless if manifest version:

Yep: https://github.com/pixiebrix/pixiebrix-extension/pull/7472

I'm unsure if your comment relates to this issue, but I'll write down what I recently found as a memo.
Firstly, Chrome supports service workers with native ES modules now, you need to write your manifest like this:
  "background": {
    "service_worker": "background.worker.js",
    "type": "module"
  },

Then the service worker can use import or export statements, not importScripts.
Currently, we use JSONP-style (the default of webpack) to load chunks, this way works well, so even if this issue is not resolved, this plugin is ok to use.
// JSONP-style chunk format
"use strict";
(globalThis["webpackChunk"] = globalThis["webpackChunk"] || []).push(...)

As an ES Module enthusiast, I hope we can support ES Module as the chunk format, and also as the chunk loading format.
// ES Module chunk format
export const id = 232;
export const ids = [232];
export const modules = {

/***/ 232:
/***/ (() => {
/***/ })

};

If the chunk format becomes ESM, the only way to load it is static import or dynamic import.
Dynamic import already works to load JSONP-style chunks in the content script (https://github.com/awesome-webextension/webpack-target-webextension#content-script), the problem is dynamic import is not allowed in an ES Module service worker even if the dynamic import is called before first event loop ends. This requires me to emit not dynamic import, but static imports. Webpack now only supports dynamic import (see __webpack_require__.f.j if you're interested) those chunks.
This becomes harder, but not impossible. I think it needs a lot of time to figure out how to make this work. I can emit import statements at the top level, but I need to know their file name, but unluck, file names are generated from a runtime function (__webpack_require__.u) and it's very hard to do it in the compile time.
Since everything works well today, this is not very urgent to fix.

Then the service worker can use import or export statements, not importScripts.

That's very good to know. Parcel forced me to use that syntax but never investigated it. I'll start using it in webpack too because importScript breaks "Pause on uncaught errors" in the dev tools, because the debugger pauses on importScript, regardless of the actual position and source maps.

dynamic import

That didn't seem to work for me 🤷♂ I got the same error as before

Changing it to import statement fixed it.

So technically now I'm using a native ES background worker in webpack-target-webextension. This issue is fixed? Maybe it just needs some documentation?

Native ES service worker works, dynamic import does not.
While using this plugin, compiled ES modules (including dynamic import) work after bundling, loading by importScritps. Native ES service worker does not work with this plugin currently (the screenshot you gave).

I just took another look of this. This is not possible until there is ES Module support in content script.

e is ES Module support in content script

I think you meant "in background workers" right?

e is ES Module support in content script

I think you meant "in background workers" right?

No, to share chunks between the background and content script, they must use the format that both environments support. Now ES Module is only supported in the background worker and not supported in the content script, so there is no meaning in investigating this.

What do you mean? I definitely am using both static and dynamic ESM in content scripts:
https://robwu.nl/crxviewer/?crx=https%3A%2F%2Fchromewebstore.google.com%2Fdetail%2Frefined-github%2Fhlepfoohegkhhmjieoechaddaejaokhf%3Fhl%3Den
See content-script.js.
There are some tricks though:

you need to use import(chrome.runtime.getURL('actual-content-script.js')) in your entry point in order to later be able to use import statements
all JS files need to be in web_accessible_resources

hmm wait you're right, an extra file will be enough to load ESM in content script.


test: Add standalone L4LB XDP tests in a form of Github Action

This commit introduces a new GH action called Cilium L4LB XDP which is responsible for running the standalone LB tests.
The action starts a Fedora VM with vagrant. We do that because we need to run Kind on cgroupv2-only machine (otherwise, bpf_sock which is required by the LB health check is not guaranteed to work). Unfortunately, GH Action does not support any runner with cgroupv2-only. So instead we run Fedora 34 which has cgroupv1 disabled on the MacOS runner which supports nested virtualisation.
For now the test issues 10 requests to LB VIP from the Fedora VM. See test.sh for more details.

ci-l4lb

ci-l4lb

That won't work anymore, not until this PR is merged in master.

The test has passed https://github.com/cilium/cilium/pull/16338/checks?check_run_id=2720855229.
@aanm I made the standalone LB to be installed via helm.

Also, does either EKS/AKS/GKE support cgroupsv2 yet? A hosted provider is always preferred that can be used instead. Running kind for some of the most basic test in GitHub Actions is okay, but it doesn't really scale, running kind inside VMs will be even harder to scale, the resources GitHub Actions provide are quite limited in.

@errordeveloper Thanks for the review!

I would imagine Kata Containers would be more suitable for this purpose, would you mind to explore it?

I haven't checked it yet. I took the same path as the Kind and containerd projects to test with cgroupv2-only. As a follow-up, I will evaluate whether Kata Containers could be used for the same.

Also, does either EKS/AKS/GKE support cgroupsv2 yet?

I haven't looked. IMHO, I prefer the current approach, as it involves few actors than creating a VM on cloud provider.

As a follow-up, I will evaluate whether Kata Containers could be used for the same.

@errordeveloper It seems that Kata is not supported on OS X. Running it on a regular runner would be very slow (due to missing nested virtualization support). An alternative would be to run on a cloud provider VM, but I'd like to avoid this complexity.

...I'd like to avoid this complexity.

I can see how this solution seems less complex, but personally I actually find it rather quite convoluted. I would recommend using any hosted solution instead of using this sort of trick to make something work inside GitHub. I am not blocking the PR, just wanted to make my view clear.

Thanks everyone for reviewing! Addressed all feedback, got all ACKs. Marking ready to merge.


Updated index.html (solving issue #966)

solving issue #966

🛠 Issue (966)

Closes #966
Description
Corrected syntax error.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[ ] I have made this from my own
[ ] I have taken help from some online resourses
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] The title of my pull request is a short description of the requested changes.

ATTACH SCREEN-SHOTS / DEPLOYMENT LINK
--NA--

Hey @TusharKesarwani  can you please review my PR.


[tool] Proposal to multiple defines for --dart-define-from-file

Overview
Make --dart-define-from-file option define multiple times.
Use case
In case of changing values for each environment of flavors and build types, multi --dart-define-from-file is needed.
For example,
// development x debug
flutter build apk --dart-define-from-file=define/flavor/development/android.json --dart-define-from-file=define/flavor/debug/android.json`.

// production x debug
flutter build apk --dart-define-from-file=define/flavor/production/android.json --dart-define-from-file=define/flavor/debug/android.json`.

packages
└ app
　 └ define
　 　 ├ build
　 　 │ ├ debug
　 　 │ │ ├ android.json
　 　 │ │ └ ios.json
　 　 │ └ release
　 　 │ 　 ├ android.json
　 　 │ 　 └ ios.json
　 　 └ flavor
　 　 　 ├ development
　 　 　 │ ├ android.json
　 　 　 │ └ ios.json
　 　 　 └ production
　 　 　 　 ├ android.json
　 　 　 　 └ ios.json

Links

https://github.com/flutter/flutter/pull/108098

Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is test-exempt.
[x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Thank you for contributing! You will need to get all CI tests (other than luci-flutter) passing before this will PR will be reviewed.

@christopherfujino
I have committed some additional changes and ready for review.

@christopherfujino
I have committed some additional changes and ready for reviews.

@christopherfujino
Thank you for your reviews. I'll try to update them.

@christopherfujino
I updated from your reviews. I created two helper functions to extract dart defines because flutter_command.dart needs defineConfigJsonMap as a parameter of BuildInfo. What do you think about it?

@andrewkolos can you take another look and, if you approve, add autosubmit?

Updated from upstream/master in an effort to unstuck the Google testing check.


CLI commands with -h flag still run

Generally, CLI commands run with a -h  flag have a behavior expectation that the action will not take place, and that a help message will be displayed instead.
Currently, a command like cdk deploy -h will run the deployment command.
While it's very reasonable for a flag to throw an error if desired (such as "use a different command structure to see a help message"), this definitely led to customer surprise for me.

Agreed, sounds like a bug.


Update documentation to show orm and root_validator interaction

Initial Checks

[X] I have searched Google & GitHub for similar requests and couldn't find anything
[X] I have read and followed the docs and still think this feature is missing

Description
On https://github.com/pydantic/pydantic/pull/822/files a test show how to use the form_orm method along with root_validator, but nowhere in the documentation around validators this gets exemplified.
The intuitive way to do it is
  @root_validator(pre=True)
  def change_input_data(cls, value):
      value['z'] = value['x'] + value['y']
      return value

But this would result in an error:
GetterDict' object does not support item assignment (type=type_error)
The correct implementation is the one stated on the tests:
@root_validator(pre=True)
def change_input_data(cls, value):
    nonlocal validator_value
    validator_value = value
      return {**value, 'z': value['x'] + value['y']}

Affected Components

[ ] Compatibility between releases
[ ] Data validation/parsing
[ ] Data serialization - .dict() and .json()
[ ] JSON Schema
[ ] Dataclasses
[ ] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[X] Generic Models
[ ] Other Model behaviour - construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

This is fixed in v2 where post root validators will operate on a dict.
In terms of pre root validators, the object could support assignment, though I agree it's unlikely.
The correct implemented is definitely not to use nonlocal


Enable building FAISS main statically

Port https://github.com/rapidsai/raft/pull/2323 PR from RAFT
[Cleans up a collection of anti-patterns in the cuvs CMake code while also enabling building faiss from latest main]

/merge


Find Website template to use

As a developer, I want a template for a website that is nice looking, intuitive, and flexible so that I can develop a user friendly experience which can be changed in look and function as I need

Material Bootstrap - https://fezvrasta.github.io/bootstrap-material-design/

https://bootswatch.com/


Data Semantics - Partial types and methods

Scope

[x] add a placeholder to the project
[x] add code example
[x] add unit tests

It is a partial task of Prepare examples for 2. Data Semantics #98.

Dane


[Refactor][Framework] Break up Blueprint config in the database

What and why to refactor
Blueprints are currently stored in a monolithic fashion in the database, with the plan and settings stored as encrypted json blobs within them. The blob structure makes searching for blueprints by some blob attribute impossible except by querying for all blueprints and brute-force searching through them to filter for the one(s) we're looking for. This is bad for performance obviously.
Describe the solution you'd like
Break the plan and settings up into other tables and link them to the BP table via IDs. We'll have to decide which columns of these tables should be encrypted. At the moment, the main use case will be having at least a blueprint_scopes table, which will be referenced by blueprint_connections, referenced by blueprint_settings. Having these tables will be the minimum requirement of this ticket.
Related issues
#5153 - introduced brute-force searching for blueprints by scopes. This ticket should optimize that.
Additional context
n/a

Good idea.
I don't think encryption is necessary for the settings field anymore, all sensitive information had been moved to other places already IIRC.

@keon94 I think this is an important refactor we should do ASAP, would you mind posting your design first?

@klesh Here's what I have in mind (simplified):

Not sure versioning would make sense anymore? Because if we are normalizing data across SQL tables, then we automatically take care of that with migrations. I've also kept the Pipeline plan internals untouched (still JSON), but maybe we can think of having them normalized too (probably another PR to keep things simpler).

@keon94 LGTM overall, some minor suggestion:

the blueprint_connections.settings_id should be setting_id singular.
we can drop the version column since it makes no sense without JSON 's flexibility, and we can migrate those tables in our migration scripts after the refactor.
either convention: blueprint_setting_id blueprint_connection_id or setting_id connection_id is good for me, but no MIX.
we can leave the plan out of this refactor and make another one for it. I would prefer we have pipeline_stages pipeline_tasks alongside pipelines as they are the models we used in our code to represent the pipeline plan, essentially, a pipeline plan is an array of pipeline stages, and a pipeline stage is an array of pipeline tasks

@klesh About blueprint_scopes, the blueprint_connection_id would be referring to blueprint_connections.id field not blueprint_connections.connection_id. So I am trying to make the distinction.

@klesh About blueprint_scopes, the blueprint_connection_id would be referring to blueprint_connections.id field not blueprint_connections.connection_id. So I am trying to make the distinction.

You are right.
Do we need blueprint_scopes.blueprint_id?

@klesh blueprint_id is technically not needed (not even in connections) but I decided to put it there as a convenience since it's going to be immutable and we'll be often querying based on it (otherwise we'd need to do a bunch of Joins).
plan_id replaces plan in the blueprint table after th refactor. Plans will get their own table. (another ticket)

@klesh blueprint_id is technically not needed (not even in connections) but I decided to put it there as a convenience since it's going to be immutable and we'll be often querying based on it (otherwise we'd need to do a bunch of Joins). plan_id replaces plan in the blueprint table after th refactor. Plans will get their own table. (another ticket)

I see, let's tackle it one by one. bp.settings only for now.


Unable to convert between caps formats [GRAY12p, 1920x1080, 8192/225]

Describe the bug
Using camera: TIS_DMK_39GX265_Z20_TI
I am trying to upgrade from tiscamera 0.14.0 to 1.1.0.
My code looks like this:
pipeline: Gst.Pipeline = Gst.parse_launch("tcambin serial=... name=source ! video/x-raw,format=GRAY12p,width=1920,height=1080,framerate=8192/225 ! valve name=valve ! appsink name=sink") state = pipeline.set_state(Gst.State.READY)
the returned state appears to be Gst.StateChangeReturn.FAILURE
using Gst.debug_set_default_threshold(Gst.DebugLevel.WARNING) I see:
0:01:12.917753128   346 0x7f64a8057980 WARN             tcamconvert tcamconvert.cpp:214:gst_tcamconvert_transform_caps:<tcambin-converter> error: Unable to convert between caps formats 0:01:12.932470709   346 0x7f64a8057980 WARN                GST_PADS gstpad.c:4231:gst_pad_peer_query:<source:src> could not send sticky events
Using tcam-ctrl -c I can see the caps I use, but using the function print_formats from examples/python/04-list-formats.py did not show GRAY12p at all.
computer used (please complete the following information):

OS: Ubuntu 20.04
Architecture/Platform: amd64 Desktop
Camera  TIS_DMK_39GX265_Z20_TI
tiscamera version
Tcam:	1.1.0_master/70abad2a_rev_4139
Aravis:	0.8_version_e977fa4
Modules:	gst_aravis_arvu3v_v4l2_libusb_tools_doc_tests

The failure is returned, because the valve prevents the start. The appsink does not get an image and therefore does not configure itself (or something like that.) I detected this behaviour, when I changed from Gstreamer 0.10 to version 1.0.
I found no workaround.
Stefan

I removed the valve from the pipeline and the error still persists:
pipeline is now: tcambin serial=... name=source ! video/x-raw,format=GRAY12p,width=1920,height=1080,framerate=8192/225 ! appsink name=sink
then setting state to READY works
but setting state to PLAYING afterwards fails:
tcamconvert tcamconvert.cpp:214:gst_tcamconvert_transform_caps:<tcambin-converter> error: Unable to convert between caps formats GST_PADS gstpad.c:4231:gst_pad_peer_query:<source:src> could not send sticky events basesrc gstbasesrc.c:3072:gst_base_src_loop:<source> error: Internal data stream error basesrc gstbasesrc.c:3072:gst_base_src_loop:<source> error: streaming stopped, reason not-negotiated (-4)

Ouch, I am sorry. tcambin does not like GRAY12p. tcamsrc does like it.
tcambin serial=... name=source ! video/x-raw,format=BGRx
or with tcamdutils
tcambin serial=... name=source ! video/x-raw,format=RGBx64
Stefan

changing tcambin serial=... ! video/x-raw,format=GRAY12p to tcamsrc serial=... ! video/x-raw,format=GRAY12p solved this issue for me


Update Link for new Version

Currently the extensions informs the user that a newer version is available (for extension and/or webserver). This will block showing the current over time.
The extension should provide in the pop up and/or in the over time display a link to get to the newest version.
This could be a clickable link:

Won't be necessary for new version since the extension will auto update.


[SPARK-46048][PYTHON][SQL] Support DataFrame.groupingSets in PySpark

What changes were proposed in this pull request?
https://github.com/apache/spark/pull/43813 added Scala API of DataFrame.groupingSets. This PR proposes to have the same API in PySpark (non-Spark Connect for now).
Why are the changes needed?
For feature parity.
Does this PR introduce any user-facing change?
Yes, it adds a new Python API DataFrame.groupingSets that is equivalent to GROUPING SETS.
How was this patch tested?
Doctests were added.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @zhengruifeng

Merged to master.


Requirement to use camelCase

I would like to consider forcing the use of camelCase naming in this standard.
As it's JSON as in JavaScript Object Notation and it might be one of the only thing that's consistent in JavaScript - example - it is Array.prototype.forEach and not Array.prototype.for_each.
(I doubt this hasn't been discussed here before, but I couldn't find anything about it.)

If the spec makes a recommendation using 'should' then some 'authors of generic client libraries' are more likely to only support the recommended style, perhaps out of personal preference and/or natural laziness, and the 'should' in the spec will be used to defend that position. Then people wanting to provide an API to a large system that uses the other style are less likely to choose JSON-API.
In my view:

The way multi-word key names are composed should be outside the scope of the JSON-API spec. It's just a name.
Adoption of JSON-API will almost inevitably be hindered by recommending one style over another.
There are more important issues to discuss and resolve.

We've gone with a "recommendation" for dasherized. This is not a part of the base spec and of course won't be enforced. In #341, we originally made this a requirement of the spec, but there was a lot of pushback and it was moved. Ultimately, names and URLs are opaque to the spec.

I made a PR for this https://github.com/json-api/json-api/pull/1247
And agree, it should use camelCase and  Google JSON Style Guide says the same:

Property names must conform to the following guidelines:

Property names should be meaningful names with defined semantics.
Property names must be camel-cased, ascii strings.
The first character must be a letter, an underscore (_) or a dollar sign ($).
Subsequent characters can be a letter, a digit, an underscore, or a dollar sign.
Reserved JavaScript keywords should be avoided (A list of reserved JavaScript keywords can be found below).
These guidelines mirror the guidelines for naming JavaScript identifiers. This allows JavaScript clients to access properties using dot notation. (for example, result.thisIsAnInstanceVariable).

Here's an example of an object with one property:

{
  "thisPropertyIsAnIdentifier": "identifier value"
}

This should be reason enough.


automation-nation.org shut down

http://automation-nation.org/superputty-tutorial, listed in README.md, is no longer available as the site is defunct.

Thanks, Removed the dead link from the Documentation Wiki page. Let me know if you run across it anywhere else.


Has one association

I am doing indexing for Listing model. And listing model has one association, has_one location.
I wanted to send multiple location ids in index, then how I need to define it in index.
location_ids = [3,25,15,45,67]
I would like to search with this for listing model.
Can anyone suggest?

Hi @pinkal-bacancy - you'll need to add an attribute with the location id to your index definition - probably along the lines of:
has location.id, :as => :location_id

And then when you're searching you can specify multiple values, and it should return listings that match any of the locations:
Listing.search :with => {:location_id => [3, 25, 15, 45, 67]}

Thank you for a quick response!
Yes, I used the same. But it doesn't give me the records based on these location ids.
D, [2020-04-24T14:50:17.365503 #6044] DEBUG -- :   Sphinx Query (1.6ms)  SELECT * FROM listing_core, listing_delta WHERE community_id = 1 AND location_id IN (3, 25, 11, 30, 33, 34, 35, 37, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 85, 89, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 115, 116, 117, 118, 119, 122, 123, 124, 125, 126, 127, 128, 129, 131, 132, 133, 135, 136, 137, 138, 140, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 161, 162, 167, 70, 83, 84, 86, 87, 88, 90, 91, 107, 108, 109, 110, 111, 112, 114, 156, 157, 166, 27, 31, 32, 52, 9, 10, 17, 163, 164, 165, 4, 5, 19, 20, 21, 160, 168, 169, 170, 120, 24, 121) AND sphinx_deleted = 0 ORDER BY sort_date DESC LIMIT 0, 10 OPTION field_weights=(title=10, category=8, description=3), max_query_time=1000
D, [2020-04-24T14:50:17.366363 #6044] DEBUG -- :   Sphinx  Found 129 results
D, [2020-04-24T14:50:17.377255 #6044] DEBUG -- :   Listing Load (9.4ms)  SELECT listings.* FROM listings WHERE listings.id IN (207, 206, 205, 184, 183, 179, 178, 174, 173, 171)
D, [2020-04-24T14:50:17.488634 #6044] DEBUG -- :   Person Load (3.4ms)  SELECT people.* FROM people WHERE people.id IN ('Zo32jxuMcrZ5UZExr0sMog', 'sBfz9IWTd6KdWz7vDJ_EmA')
D, [2020-04-24T14:50:17.521495 #6044] DEBUG -- :   ListingImage Load (23.0ms)  SELECT listing_images.* FROM listing_images WHERE (error IS NULL) AND listing_images.listing_id IN (171, 173, 174, 178, 179, 183, 184, 205, 206, 207) ORDER BY position
D, [2020-04-24T14:50:17.535922 #6044] DEBUG -- :   Testimonial Load (4.5ms)  SELECT testimonials.* FROM testimonials WHERE testimonials.blocked = 0 AND testimonials.receiver_id IN ('Zo32jxuMcrZ5UZExr0sMog', 'sBfz9IWTd6KdWz7vDJ_EmA') ORDER BY id DESC
D, [2020-04-24T14:50:17.544265 #6044] DEBUG -- :   HABTM_Categories Load (3.2ms)  SELECT categories_listings.* FROM categories_listings WHERE categories_listings.listing_id IN (171, 173, 174, 178, 179, 183, 184, 205, 206, 207)
D, [2020-04-24T14:50:17.554028 #6044] DEBUG -- :   Category Load (3.6ms)  SELECT categories.* FROM categories WHERE categories.id IN (4, 8, 2, 3, 1, 5, 6, 7, 9, 12, 11, 10)
D, [2020-04-24T14:50:17.560558 #6044] DEBUG -- :   Location Load (3.4ms)  SELECT locations.* FROM locations WHERE locations.listing_id IN (171, 173, 174, 178, 179, 183, 184, 205, 206, 207)
D, [2020-04-24T14:50:17.610158 #6044] DEBUG -- :   FavouriteListing Load (3.5ms)  SELECT favourite_listings.* FROM favourite_listings WHERE favourite_listings.listing_id IN (171, 173, 174, 178, 179, 183, 184, 205, 206, 207)

Is there any way to display listing based on the order of the location ID? For exampple if 3 is the first element of location id then, it should give the listing of that location first.

Yes, you can sort by any attribute - just use the :order option in your search call. To expand on the previous example:
Listing.search(
  :with => {:location_id => [3, 25, 15, 45, 67]},
  :order => :location_id
)

It is working for only one record of location. The association is, the listing has_on location.
If I am sending an array of location id then it is not giving record in order by location ids.

I would like to display listings records in following manner,
if I have array of location_id = [3,25,15,45]
then it should give the listing of location 3 first and second is of 25's listing and so on....

Ah, I'm afraid that's not possible with Sphinx or Thinking Sphinx (or ActiveRecord, I believe). You'll need to do that sorting yourself once you have the results.

Yeah... you are right. But after applying sphinx search, it's converting records based on pagination. I want listing with id 14 into the result, but it is giving me [205, 207, 206, 205, 203] like this due to pagination and ordering. So I am finding if there is any way to do custom ordering with sphinx or not. So i can get the listing which I need.

I think you might need to conduct the search with each single location separately, to ensure you're paginating through one entirely before moving on to the next.

You could change the pagination page size to something very large, but that's only useful if you're always sure that you'll never be over that limit.

Oops, I can't define size statically.

Issue resolved.
I removed pagination from sphinx search and give dynamic limit. And apply pagination on the result of sphinx search.
And this works for me.

Good to know you've found a way through :)


Cannot read property 'split' of undefined

TypeError: Cannot read property 'split' of undefined
at PHPCSFixer.module.exports.Beautifier.findFile (/home/gadjet/.atom/packages/atom-beautify/src/beautifiers/beautifier.coffee:138:25)
at PHPCSFixer.module.exports.PHPCSFixer.beautify (/home/gadjet/.atom/packages/atom-beautify/src/beautifiers/php-cs-fixer.coffee:65:39)
at /home/gadjet/.atom/packages/atom-beautify/src/beautifiers/index.coffee:363:28
at tryCatcher (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:517:31)
at Promise._settlePromise (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:574:18)
at Promise._settlePromiseCtx (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/promise.js:611:10)
at _drainQueueStep (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:142:12)
at _drainQueue (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:147:5)
at Async.drainQueues (/home/gadjet/.atom/packages/atom-beautify/node_modules/bluebird/js/release/async.js:17:14)
at

got this issue some time ago ,
just go to beautify setting,
remove cs-fixer phar path from exutable, and put into php setting,
at this point beautify will notice you that is deprecated,
just go to to you user folder (i'm assuming you are uing win10) ".../.atom\packages\atom-beautify\src\beautifiers" and edit file "php-cs-fixer.coffee" at line 46 from "cs_fixer_path: true" to "cs_fixer_path: false"
and it should work again


Use custom functions/extensions with go-sqlite3 driver

go-sqlite3 supports easy definition of custom functions:
https://godoc.org/github.com/mattn/go-sqlite3#hdr-Go_SQlite3_Extensions
Currently cannot use the approach with xorm as:
regex = func(re, s string) (bool, error) {
	return regexp.MatchString(re, s)
}
sql.Register("sqlite3_with_go_func",
		&sqlite3.SQLiteDriver{
				ConnectHook: func(conn *sqlite3.SQLiteConn) error {
					return conn.RegisterFunc("regexp", regex, true)
				},
		})
orm, err := xorm.NewEngine("sqlite3_with_go_func", vulnDbPath)
if err != nil {
	return err
}

Fails as xorm appears only to work with "sqlite3".  However, I cannot use "sqlite3" as that gives the error:
panic: sql: Register called twice for driver sqlite3

Any good way around this?

the library should implement https://golang.org/src/database/sql/sql.go?s=19056:19091#L649
also see https://github.com/golang/go/issues/27152


Feature: ability to post-process responses from LLM in voice assistant

Sometimes it's desirable to control or change the output from LLMs before TTS.

Similar: https://github.com/livekit/agents/issues/451

@davidzhao I'm still working on the same thing (discussed in slack), but the changes I've made in my code are specific to my requirements and not generalised. I'm not sure if that thing can help you. I'm eagerly waiting for the solution of this.

@theomonnom why this marked as completed?
can we use this feature now? is there any docs about this? any information will be appreciated:)

you'd want to use the before_tts_cb callback with VoiceAssistant. you can reference this example here


Update RULES.md

Hackathon Submission: <PROJECT_NAME>
Name of Team Lead
Enter your full name

GitHub handles of Team Members
Enter the GitHub handles of all your team members (including yourself) as separate bullet points

Are all of your team members registered on our Hackathon Signup page?
This is a mandatory step for all participants to be considered eligible for judging

[x] Yes
[ ] No

Rules and Code of Conduct
By submitting this PR, you agree to follow our Rules and Code of Conduct.

[x] Yes

Anything Else?
Any other feedback, queries or information you would like to share with us?

@ChiragTankan this is an empty PR mentioning an update to the rules and an irrelevant PR description, therefore I'm closing this PR

@adityaoberai  I am unable to submit this one  I have created my submission file in my forked repo in submissions but not able to see my name in pull request in official repo how should I do ?

@adityaoberai  I am unable to submit this one  I have created my submission file in my forked repo in submissions but not able to see my name in pull request in official repo how should I do ?

Can you share a link of your fork with me once?

Sure ,
https://github.com/ChiragTankan/htf24-hackathon-submissions

@ChiragTankan this should help 👇
https://github.com/user-attachments/assets/41da05c5-1dbf-4b58-8fdf-2475217d5e7e


Enter "the waddling"

https://github.com/palewire/cummings.ee/blob/main/_data/poems/is-5/one-vii.yaml
https://archive.org/details/is-5-ee-cummings/page/n25/mode/2up

Instructions on how to contribute here: https://github.com/palewire/cummings.ee/blob/main/CONTRIBUTING.md

This one also qualified as a duplicate as described in #294
https://cummings.ee/book/and/poem/portraits-iv/


Helm repository / Versioned download URL

Is there a public Helm repo available for this chart: https://downloads.openappsec.io/packages/helm-charts/nginx-ingress/open-appsec-k8s-nginx-ingress-latest.tgz
Or is there maybe a versioned URL like: https://downloads.openappsec.io/packages/helm-charts/nginx-ingress/open-appsec-k8s-nginx-ingress-<version>.tgz
Also, why is the chart in this repo not up to date? (Version 4.8.3 instead of 4.8.4)
https://github.com/openappsec/openappsec/tree/main/build_system/charts/open-appsec-k8s-nginx-ingress
I want to deploy the helm chart using terraform, currently I need to store the whole .tgz file in my repo.
Awesome project, thanks in advance.

Hi @swissbuechi,
Thank you for reaching out and for your interest in our project! We appreciate your support.
Regarding your inquiry about a public Helm repo or versioned URLs for our NGINX Ingress Helm chart, currently, we don't have a public Helm repo or versioned URLs. We understand the importance of these features and are actively working on adding them in the near future.
As for the discrepancy in versions between the Helm versions, we are constantly working to ensure that the latest versions are reflected in the repository and will have this updated soon.
If you have any specific requirements or encounter issues, please let us know, and we'll be happy to assist you. Thank you for your understanding.
Best regards,
Oriane
open-appsec team

Thank you for your reply.
I'm looking forward to these features being implemented.

Hi @swissbuechi,
We've updated the Helm Chart and it's now based on version 4.9.0 NGINX Ingress Controller.
Also, starting from this open-appsec version you can pull a versioned helm, just replace the latest tag with our version (for example https://downloads.openappsec.io/packages/helm-charts/nginx-ingress/open-appsec-k8s-nginx-ingress-1.1.3.tgz)
Have a happy new year,
Oriane
open-appsec team

Hello! Any news about the public Helm repository @orianelou ? I want to try openappsec with Flux in my k8s cluster, but I cant do GitOps without this repo :)
Thank you!


Fix configuration

@cailinw Please look through this carefully! :slightly_smiling_face:

@cailinw It was not a specific bug but configure was more complicated than it needed to be and I think there would be some issues in the first-time setup for a user. I tested this new implementation even with the banyanconfig.toml file removed.


issue with react-app

I installed react-app globally, then in an empty folder ran react-app new.
then got the below
Scaffolding a new JavaScript application in /Users/admin/Documents/Development_Stuff/B/ra
Installing 'react-app-tools' from npm... This may take a couple minutes.

><EMAIL_ADDRESS>install /Users/admin/Documents/Development_Stuff/B/ra/node_modules/fsevents
> node install

[fsevents] Success: "/Users/admin/Documents/Development_Stuff/B/ra/node_modules/fsevents/lib/binding/Release/node-v57-darwin-x64/fse.node" already installed
Pass --update-binary to reinstall or --build-from-source to recompile

><EMAIL_ADDRESS>postinstall /Users/admin/Documents/Development_Stuff/B/ra/node_modules/webpack/node_modules/uglifyjs-webpack-plugin
> node lib/post_install.js

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.
npm WARN ra No description
npm WARN ra No repository field.
npm WARN ra No license field.

+<EMAIL_ADDRESS>added 1758 packages from 885 contributors in 76.217s
Error: Cannot find module '/Users/admin/Documents/Development_Stuff/B/ra/node_modules/react-app-tools/scripts/new'
    at Function.Module._resolveFilename (module.js:538:15)
    at Function.Module._load (module.js:468:25)
    at Module.require (module.js:587:17)
    at require (internal/module.js:11:18)
    at Promise.resolve.then.then.then (/Users/admin/.config/yarn/global/node_modules/react-app/cli.js:107:13)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

@joshua1 you can bootstrap a new empty project by running:
$ yarn create react-app --scripts-version<EMAIL_ADDRESS>hello-world
$ cd ./hello-world
$ yarn start

Just replace hello-world with the name of your project. Alternatively, you can start from this boilerplate that has core web application building blocks in place (authentication, GraphQL API endpoint, etc.):
$ git clone -o react-firebase-starter https://github.com/kriasoft/react-firebase-starter.git hello-world
$ cd ./hello-world
$ yarn setup
$ yarn start


Even more metrowerks ps2 compilers

Awesome, I see MWCC from R3.6 Hotfix 3 ( 3.0.1 build 75), R3.8.1 MPTP1 (Version 3.0.1 build 103), one from R3.51 (Version 3.0 build 50), and one from R3.5 (Version 3.0 build 38).
There are two that could be added:

R3.51 Version 3.0.1 build 51 (builds 50 and 52 are added, but 51 is also in this version under 3.0.1)
R3.5 Version 3.0.1 build 44 (build 38 from this release is already added).

Great additions, thank you!

I think this is ready to go (3.0.1 build 44 was already added)

Looks great to me, thank you!


?POW

Love the new firmware. works flawlessly without crashing which has always been a problem for my sonoffs.
Is there any work in the pipeline for the Sonoff POW?
Thanks,
Sam

Hi yes my plan is to support it but I just haven't had rime sorry. I will get around to it though at some stage.


Provide succinct example of extending models in Readme or Wiki

Monkey patching / extending models seems to be a bit tricky in Rails, though I'm sure there's a conventional approach you might recommend for the Gutentag models.
I'm trying to override #to_params on Gutentag::Tag in an initializer, but I believe it's being canceled by Rails's loading process, and I'm going to hack around it for now as I can't work it out.
I understand solving that problem isn't within the scope of this project, but I thought that a simple example of the conventional pattern for extending the models might be a good addition to the readme or wiki here, as it seems like a potentially common use case.

Hi Luke,
It is a bit of an edge case, but would nice to have it noted somewhere. Did you end up figuring out a neat solution? I'm wondering if the best way is to include a module into Gutentag::Tag within an after_initialize block…
# in config/initializers/gutentag.rb or similar

Rails.application.config.after_initialize do
  Gutentag::Tag.include TagParams
end

Hey Pat, yep I did just get it working to extract a couple of methods (your after_initialize tip here was the key :smile: ):
# in config/initializers/gutentag.rb

Rails.application.config.after_initialize do
  Gutentag::Tag.instance_eval do
    def find_by_name_param!(string)
      self.find_by_name!(string.tr('+', ' '))
    end
  end

  Gutentag::Tag.class_eval do
    def to_param
      self.name.tr(' ', '+')
    end
  end
end

Do you think that would be useful as an example of extending the classes? You might prefer the module inclusion, which seems cleaner maybe.

Turned out that wasn't a working solution because in dev, rails reloads the models between page loads and blows away our changes :) Here's what I have working now:
# in config/initializers/gutentag.rb

require 'tag'

Rails.application.config.to_prepare do
  Gutentag::Tag.include Tag
end

And
# in lib/tag.rb

module Tag
  def self.included base
    base.extend ClassMethods
    base.send :include, InstanceMethods
  end

  module ClassMethods
    def find_by_name_param!(string)
      self.find_by_name!(string.tr('+', ' '))
    end
  end

  module InstanceMethods
    def to_param
      self.name.tr(' ', '+')
    end
  end
end

This is now in the README :) https://github.com/pat/gutentag#extending

Very nice :) Thanks @pat :revolving_hearts: this has been really useful for a project I'm working on, tracking the public lobbying activities of the fossil fuels lobby.

Oh interesting! Is that deployed publicly anywhere?

Oh interesting! Is that deployed publicly anywhere?

Not yet, but I'll let you know :)


Fix Cardinal Components Block syncing

Fixes CCA block components not syncing due to the CCA mixin being indirectly (this method calls a method CCA mixins into) overwritten here.

I've just found that block entities placed/added after the chunk load still don't work. Will try to fix it soon.


Running Background Jobs - Worker Tier & SQS or Sidekiq/Suckerpunch/Reque/DelayedJob?

Whats the best way to run the Background Jobs?
Should we go for
Worker Tier & SQS or
Sidekiq/Suckerpunch/Reque/DelayedJob?
Also, when we deploy using eb:deploy, we need to set tier to WorkerTier and deploy is it? OR can the eb deploy for both Web & Worker tiers are the same time, if so any such example configuration?
What settings are there for configuring worker tier?
Thank you much in advance.

I haven't deployed a worker tier yet but our scalability is planned around sidekiq pro with elasticache(redis).  I plan to do some testing around what is the best for a worker tier, and we are considering alternatives such as EB for web only with docker containers for workers.  But nonetheless, I've not deployed and tested that scenario, only done enough to implement sidekiq/redis in development/testing.
In my experience, sidekiq is great and sidekiq pro gets you the bulletproof features.  The simple offloading of work to sidekiq/redis can add dramatic scale to this sort of architecture and reduce necessary running instances.  There is a SEDA (synchronous event driven architecture) paper out there somewhere that is really applicable to this approach.  See http://www.eecs.harvard.edu/~mdw/proj/seda/
I've done some good work with rails/docker here:  https://github.com/alienfast/docker-rails
I'm really interested in containers, and in essence you could view EB as a container with more orchestration.  I think eventually EB could get phased out with containers taking over, but it will take a while before the orchestration meets the level that EB provides.  I do however think that containers might be where they need to be in order to run the worker tier...definitely worth looking at.

We currently use Sidekiq with non aws infrastructure on a separate server. We are moving to AWS.
How can we configure the EB environment a dedicated instance for Sidekiq where no other web requests are processed by this instance? Leader only flag also will not help as such, as it's part of web tier. Under load Sidekiq stops as it fights for resources when web requests are keeping the instance busy.
Any sample configuration for Worker tier?
I will try docker rails for next phase of our infra rollout.

On Feb 9, 2016, at 3:37 AM, Kevin Ross<EMAIL_ADDRESS>wrote:
I haven't deployed a worker tier yet but our scalability is planned around sidekiq pro with elasticache(redis). I plan to do some testing around what is the best for a worker tier, and we are considering alternatives such as EB for web only with docker containers for workers. But nonetheless, I've not deployed and tested that scenario, only done enough to implement sidekiq/redis in development/testing.
In my experience, sidekiq is great and sidekiq pro gets you the bulletproof features. The simple offloading of work to sidekiq/redis can add dramatic scale to this sort of architecture and reduce necessary running instances. There is a SEDA (synchronous event driven architecture) paper out there somewhere that is really applicable to this approach. See http://www.eecs.harvard.edu/~mdw/proj/seda/
I've done some good work with rails/docker here:  https://github.com/alienfast/docker-rails
I'm really interested in containers, and in essence you could view EB as a container with more orchestration. I think eventually EB could get phased out with containers taking over, but it will take a while before the orchestration meets the level that EB provides. I do however think that containers might be where they need to be in order to run the worker tier...definitely worth looking at.
—
Reply to this email directly or view it on GitHub.

See
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
And
https://github.com/ThoughtWorksStudios/eb_deployer/issues/34#issuecomment-76432311


Folder side resets everytime i join a new file

if i open a file, all the folders reset to their initial state (all closed with an exeption, i want for them to keep  however i have them between pages   (if i go to another folder, they don't change)

I agree, that would be much better user experience. I will need to find some way to keep that state across page reloads. I will look into it.

yes this would be great,
created an issue for this before I saw this one, closed now (https://github.com/oleeskild/obsidian-digital-garden/issues/110)

I've just released version 1.21.0 of the template. The filetree should now remember the state that it's in across page reloads. The template can be updated by following the steps in the documentation. Let me know if it works as you expected 😄

Super Nice, working as expected!


Added playbooks to enable and disable passwordless ssh for root user.

Tested on dev cluster.

Created SSH key pair for root.
Ran playbook to enable password less ssh for root.
Became root, SSH'd to all other nodes on the cluster (login, backup, two compute nodes).
Ran playbook to disable password less ssh for root.
Became root, confirmed I can no longer SSH to all other nodes on the cluster.

Added plays to add and remove a SSH splash screen warning on bastion node.

Rebased to one commit and merged.


Rethink the meaning of tracking a pods state

If the perceptor data model isn't up to date, or if some how a perceiver knows of a pod but not of its containers, then you can trick perceptor into thinking a pod is free of vulnerabilities even though its just missing information content.
Lets rethink what it means to annotate the overall status of a pod: Or if its even worth doing at all, since the containers are the source of all the threats we care about currently.

If we stop sending pod information to the perceptor it makes the perceptor's job 1 thing, dealing with images.  That's what it cares about and really why the perceivers care about too.
The downside to doing that is that the logic in the perceptor to deal with rollup information and which pods have what images gets moved into the perceiver.  IMO, that is where it should reside anyway since a pod is a concept in the cluster the perceiver is talking to.

I think consensus is that we send pod info, but its really a passthrough thing.  perceptor shouldn't interpret that information in anyway.


Lecture 6: Learning Partially Observed GM and the EM Algorithm

https://hwehry.github.io/pgm-spring-2019/notes/lecture-06/

https://hwehry.github.io/pgm-spring-2019/

@hwehry, please address the questions/comments raised in the review. Thanks!

I apologize! I missed these notifications and will respond shortly.
On Tue, Feb 19, 2019 at 11:17 AM Maruan<EMAIL_ADDRESS>wrote:

@hwehry https://github.com/hwehry, please address the
questions/comments raised in the review. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sailinglab/pgm-spring-2019/pull/6#issuecomment-465200180,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ak2H52pQ_CbRd4nOVnKLmXvAqgiNfeqCks5vPCOtgaJpZM4a4nSR
.


How to rollup scss file without compiling with node-sass

Does rollup-plugin-scss always compile scss to css ? is there a way we can rollup scss file and generate bundle.scss in dist folder ?

Fixed it using.  https://github.com/weizhenye/rollup-plugin-bundle-scss


[Suggestion] Various Solderer Recipe Rearrangements.

The Solderer supports Shift clicking fairly nicely, except some components can be inserted into multiple slots.  This causes some minor annoyances with manual use or simple automation (like hoppers).  A few recipes could be rearranged to remove the more common one(s), or changes to many recipes to remove all but one of them.
Current Conflicts as of Refined Storage 1.3.5:
Redstone (All Processors and All Upgrades)
Crafting Table (Crafting Grid and Crafting Upgrade)
Basic Processor (Storage Blocks and Interface)
Buckets (Fluid Interface, Fluid Grid) - Impossible to fix without changing the ingredients of the Fluid Interface
Possible Change 1:  Minor, only fixes Redstone and optionally Crafting Tables.
All Upgrades:
A) Blank Upgrade on top, Redstone in the middle, and special material on the bottom.  Solves both Redstone and the Crafting Table conflicts.
B) Special Material on top, Redstone in the middle, and the blank Upgrade on the bottom.  This would be similar to crafting Processors, but would leave the Crafting Table conflict in.
Possible Change 2: Major (I think this removes all conflicts except the Fluid Interface)
Top Slot: Special Materials (Chests, Crafting Tables, Ender Pearls, Sugar, buckets, etc.), Printed Processors, Importer, Storage Parts
Middle Slot: Redstone, All Processors, Diamond, Gold, Iron, silicon.  Interface for Fluid Interface.
Bottom Slot: Machine Casings, Grids, blank Upgrades, Exporter, Printed Silicon.  Bucket for Fluid Interface.

Thanks for the good report, listing all conflicts. This made it easier for me to fix.
Can you double check these changes for me:

the redstone for upgrades is now in the middle slot
the crafting table for the crafting grid is moved to the top slot, to stop conflict with upgrades
pattern in pattern grid is moved to the top slot as well, for consistency
bucket in fluid grid is moved to the top slot as well, for consistency
interface processor is now on top, to stop the conflict with the storage blocks
fluid interface now takes a bucket, redstone in the middle, and an interface on the bottom.

Looks good.  The only change I would still suggest is Chest in Disk Drive should be moved to the top slot as well, for consistency.  ~line 290

Fixed, thanks.

Oh, wait. whoops.  With that rearrangement, Machine Casings can go in either the middle or bottom (Storage Blocks vs Grid/Disk Drive)


Error: robot.urdf.xacro could not be found

Hi Sammy,
thank you very much for this script! It helps me a lot!
I have the error that the file robot.urdf.xacro could not be found after the build when I run the startup file.
If I manually rename the file in the installation folder, it works.
Now I have added line 105 and it also works with the script:

I work with ROS2 Humble.
Maybe someone else has the same problem and it helps.
All the best,
Daniel

Thanks for reporting this bug and the solution. The latest commit should fix it.


terraform-providers.scalr: init at 1.0.4

Description of changes
Adds the scalr Terraform provider to terraform-providers package set.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Use the update-provider script.

I think it might even be better if we add a condition to the script to only update the ones with terraform.io as a substring of the homepage.

I think it might even be better if we add a condition to the script to only update the ones with terraform.io as a substring of the homepage.

No, we've previously removed a provider that was hosted on a different registry that didn't work with the update script. https://github.com/NixOS/nixpkgs/pull/155959

I think it might even be better if we add a condition to the script to only update the ones with terraform.io as a substring of the homepage.

No, we've previously removed a provider that was hosted on a different registry that didn't work with the update script. #155959

I could work on adding a special scalr condition to that script if that would be acceptable? I can get the latest version from GitHub's release endpoint since it is hosted there.

I'm trying to find a good way to get this provider supported as it's something I use daily - the company I work for uses Scalr because it is massively cost-advantageous over TFC's business pricing. I'd love if you can provide guidance on how to do so in the least intrusive manner.

As I said above I'm not enthusiastic about adding it. I don't want to end up with a collection of registries that only publish a single provider as it will be a maintenance burden.
We don't need to add it to nixpkgs for your company to use it, can do so via a flake or overlay.

Sure, I get that it's possible to maintain custom config, but I'd love to contribute to Nix for everyone to enjoy. Scalr is a special case here because Scalr will almost certainly never be in the official registry due to the fact that they compete with Terraform Cloud. I don't even know if Hashicorp would allow them to publish their module to TFC, but even so, they probably host their own as a safety net so that Hashicorp couldn't remove it.

Using https://api.github.com/repos/Scalr/terraform-provider-scalr/releases/latest - we can grab the tag attribute to get the version. Perhaps with jq.

Scalr is a special case here because Scalr will almost certainly never be in the official registry due to the fact that they compete with Terraform Cloud.

Every provider that can't or won't publish on the official registry is a "special case". Doesn't change the fact that each one we add is a maintenance burden and once we add one we have no grounds to reject others.

Scalr is a special case here because Scalr will almost certainly never be in the official registry due to the fact that they compete with Terraform Cloud.

Every provider that can't or won't publish on the official registry is a "special case". Doesn't change the fact that each one we add is a maintenance burden and once we add one we have no grounds to reject others.

I've been experimenting with handling GitHub as a source, I think I have something that won't be completely unmaintainable. Let me finish this up and I'll post it for comments.

@zowoq - I think this handles providers on GitHub in a generic enough way that as long as the release uses a semver, it should work correctly and allow us to have overrides in the update script itself.

Sorry but what you've pushed doesn't change what I said above, this when it isn't going to be merged. Closing.

Sorry but what you've pushed doesn't change what I said above, this isn't going to be merged. Closing.

I think a lot of the concern you had was that it didn't work with the update script in the past, correct? This PR made it work with the update script without breaking existing functionality, retrieving the latest version. It'd also be possible to reuse this code for almost any repository that happens to be hosted on GitHub by simply adding a few values to the array in the shell script.
Because the update-packages script would be able to fetch all of the relevant values, it could update packages.json using the existing workflow, making it fairly easy for this provider to be maintained.
I'd love for you to reconsider.

Currently we have 140 providers, 137 work as is and 3 require a single line fix.
This adds a bunch of complexity that will only be of immediate benefit to one or two providers and will quite likely be of little benefit long term while increasing the overall maintenance burden.

Currently we have 140 providers, 137 work as is and 3 require a single line fix.
This adds a bunch of complexity that will only be of immediate benefit to one or two providers and will quite likely be of little benefit long term while increasing the overall maintenance burden.

I completely understand where you are coming from. On the flip side, there are thousands of default.nix files with logic and workarounds that are only of benefit to one package. I'd hardly say that doesn't make it a bad idea to package them if someone is willing to maintain it (as I am willing to maintain my code in this case), and if there are users of the package/provider that would benefit from it (and there are, evidenced by the fact that these packages have requests on nixpkgs). What if I were to break out the GitHub packages into a "terraform-providers-github" folder and have a separate script and workflow for them that I maintain and not touch your update script at all? We'd lose the benefit of having it all in one place, but I could maintain it independently and you wouldn't have to worry about bugs in my code breaking existing TFC packages.

This wouldn't actually change anything, they'd still need to use mkProvider so they can be in the terraform-provider namespace.
Offering to maintain it is appreciated but doesn't change that I'd still have to maintain it regardless as couldn't make changes to one and not the other.

That's actually an extremely good point. So there'd be no benefits from splitting them out into different folders. I feel like I need to brainstorm on this some more. The burden of maintaining workarounds is largely due to conditional logic in the update script, right? What if we could somehow keep all the conditional logic for github packages separate and keep the existing logic intact, perhaps with some sort of wrapper that called common functions but overrode the few things that are special to github repo packages?

What if we could somehow keep all the conditional logic for github packages separate and keep the existing logic intact, perhaps with some sort of wrapper that called common functions but overrode the few things that are special to github repo packages?

My point of view is that adding providers that don't use the official registry is basically just doubling the complexity of the packaging and scripts, regardless of how simple it may be.

I don't view things exactly as zowoq. nixpkgs is here to provide utility, and Scalr is an important player in the Terraform ecosystem. Because we have a flat namespace here, there are some restrictions around name clashes, but we'll unlikely hit that with this provider.
Since we are building from source, it might be interesting to explore how we can scan the github tags on all the providers instead of querying the registry. That way it reduces the number of branches in the update code, and we could start supporting more registries.
@dylanmtaylor if the registry supports the same query methods as the official one, I'm happy to accept a PR at https://github.com/nix-community/nixpkgs-terraform-providers-bin

I don't view things exactly as zowoq. nixpkgs is here to provide utility, and Scalr is an important player in the Terraform ecosystem. Because we have a flat namespace here, there are some restrictions around name clashes, but we'll unlikely hit that with this provider.
Since we are building from source, it might be interesting to explore how we can scan the github tags on all the providers instead of querying the registry. That way it reduces the number of branches in the update code, and we could start supporting more registries.
@dylanmtaylor if the registry supports the same query methods as the official one, I'm happy to accept a PR at https://github.com/nix-community/nixpkgs-terraform-providers-bin

I love that you see that the Scalr provider would provide value here. Can we re-open this pull request so we can continue iterating on this idea? I don't know if it supports the same query methods since I just grabbed it from GitHub and built it that way, but it does work with the official Terraform client, so I imagine they at least partially reversed-engineered and implemented the TFC one. Perhaps you can help me with an investigation into that?
Thanks.

Can we re-open this pull request so we can continue iterating on this idea?

No, my opinion on this hasn't changed. One or two providers isn't worth the ongoing maintenance.

Since we are building from source, it might be interesting to explore how we can scan the github tags on all the providers instead of querying the registry. That way it reduces the number of branches in the update code, and we could start supporting more registries.

Honestly I think is is a bit of a policy change since the beginning of last year when we removed providers that weren't on the registry.
@zimbatm If you want to move forward with this idea do you want to take over as maintainer/codeowner?

Can we re-open this pull request so we can continue iterating on this idea?

No, my opinion on this hasn't changed. One or two providers isn't worth the ongoing maintenance.

As @zimbatm said, Scalr is a major player in the Terraform ecosystem. I feel like the benefit to the community would be substantial and with further refinement, I bet we could reduce the amount of conditional logic such that it is even less than what I described in #222462. I'd like to open this PR back up, play around with some ideas (I would prefer to not open further PRs as it feels spammy here and the discussion can live in one place), and try to come to a solution that is more palatable. As I said earlier, I'm happy to be the maintainer of my changes.

As I said earlier, I'm happy to be the maintainer of my changes.

And as I said earlier, I'll end up having to maintain it regardless.

So I just tried something - I messaged Scalr via one of my contacts there from work and asked if they would consider publishing to TFC. I imagine the answer might be no, as I don't think I'm the first one with this request, but it would eliminate the need for a workaround if they agree. I do understand your concern but I feel like the level of additional maintenance burden might be less than you are expecting. Regrettably, I don't feel like I will be able to convince you of this without trying to implement the solution proposed or one that we come up with together. If you're receptive to this idea, you can add me as a co-maintainer for the providers and I could help solution something that would work.

I imagine that as established maintainers in the NixOS community, you have a large maintenance burden already, whereas I am fresh to the community and only have/want to have a handful of things that I maintain. Right now, this is just aws-workspaces and scalr-cli, but I am also hoping to take on gnome-extensions-cli and a couple of meraki packages as well as possibly chef-workstation. As they say in the development industry, I 'have the bandwidth' to take on (co-)maintainership of something like this Terraform provider script. I feel like what you have already is incredibly well designed, and adding the logic to pull the special providers from GitHub isn't a huge undertaking. I also have extensive experience with GitHub actions and wouldn't mind troubleshooting that. I'm happy to work alongside you on this. Let me know what you're receptive too here - I don't plan to just merge in my PR and disappear.

To add some historical context; Terraform introduced the registry around 0.11. We supported the old system and the registry for a while, leading to complicated code. Once 1.0 came out, zowoq helped a lot to clean things and remove the old packages that didn't migrate to the registry. That's when we agreed not to include packages outside the registry. And generally, to avoid packages with overrides and they create more maintenance burden. zowoq also set up GitHub Actions to automate the updates, helped a lot on the nixpkgs-terraform-providers-bin repo and also takes really good care of that part of nixpkgs.
@zowoq to me, the policy made sense in that context. I don't mind re-evaluating it, as long as the new policy is sustainable. At the same time, I also respect that you shouldn't be forced to maintain more code than you are comfortable with (this reminds me of the GHA discussions we were having as well). Ideally, we find a solution that satisfies you both.
Regarding the design, we hit a local maxima, but I think there is still room to improve things.
One would be to split the fat providers.json into one JSON per provider + a builtins.readDir to load them all. This makes it easier to work with individual providers, and also reduces the risk of merge conflicts. It also makes it easier to see all the available providers by navigating in nixpkgs.
Another one would be if we can use the ryantm updaters bot. See how each provider can expose an updateScript, so we don't have to maintain our own github actions script.
I think these two changes would make it easier to split the maintenance further down as well. What do you think? Making room for new contributors is also important.

I messaged Scalr via one of my contacts there from work and asked if they would consider publishing to TFC.

Until we have an answer about this I don't see that we need to discuss anything TBH, if it does get added to the registry then there really isn't any reason to make changes here.
That said I will comment on two things:

One would be to split the fat providers.json into one JSON per provider + a builtins.readDir to load them all.

I've thought about this before. I've found make changes across all of the providers to be very easy with everything in one file, I'd be strongly opposed to splitting it up.

See how each provider can expose an updateScript, so we don't have to maintain our own github actions script.

The action just runs all of the updateScripts at once.


Attempted fix

@david-a-wheeler

Here's hoping!

Drat, failed.  See:
https://circleci.com/gh/linuxfoundation/cii-best-practices-badge/889

Appears to be:
https://discuss.circleci.com/t/error-running-bundle-exec-rake-command/3760

It rebuilds fine if you rebuild without the cache.  It looks like CircleCI overwrites the old cache when you force it to rebuild without the cache; if so, then rebuilding without the cache should fix the problem (at least for this branch).

Looks like we're getting good rebuilds.  I'm crossing my fingers...!


update: analyticdb+postgresql vectore

Describe what this PR does / why we need it
Does this pull request fix one issue?

Describe how you did it
add analyticdb api
Describe how to verify it
use springboot test
Special notes for reviews

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please run mvn spring-javaformat:validate -pl spring-ai-alibaba-core to reformat code.

pull once again


what's in a name?

Apparently jishaku means "magnet" in Japanese. Why'd you choose that name? Just curious.

Discussed on Discord. It's an eval cog that you can just "stick on" to quickly diagnose a problem.


Add gomega checks into the HTTP Error TC in processor test

Coverage increased (+0.04%) to 77.149% when pulling 260f41a9097d1fbed0729d1ec1c7a4c5ec9e4a4d on jmedved:contiv-crd into 1cf001fa34249494c651e698a085067994dc48c9 on contiv:contiv-crd.


Df 105

This connects to #105

@vijaypoojari -> please add PR description

Please run mvn license:format and add commit with fixed license headers


Overlay does not open with Firefox on Android

It seams that the plugin does not work with Firefox on Android. On http://airbnb.io/react-dates no date overlay appears if I tab on State date or End date.
You can easily simulate this behavior if you open http://airbnb.io/react-dates in Firefox (in my case v48), turn on Responsive view with Ctrl+Shift+M and enable Simulate touch events.

Ooh, that's not good and could explain a lot of data for us. D: I will dig into this.

@majapw I tracked this bug down and found the problem: https://github.com/airbnb/react-dates/blob/master/src/components/DateInput.jsx#L124
It seems that disabled input fields cannot trigger events in Firefox with Simulate touch events on. Demonstration: http://www.webpackbin.com/41VTyxNgz

Makes sense. I thought that I had a click handler above the input itself (on the surrounding div) to deal with this particular scenario (although it may have disappeared at some point in the grind).
As it is, if we can find a way to hide the keyboard and cursor on mobile devices consistently, I'm happy to replace the disabled property with that as that was mainly what I was trying to address.

The same error occurs on Windows 10 with current Firefox version (49). I try to find a solution and will submit a PR asap.

@majapw What's about registering an onClick handler on DateInput__display-text ?
<div
  className={cx('DateInput', {
    'DateInput--with-caret': showCaret && focused,
    'DateInput--disabled': disabled,
  })}
>
  <label className="DateInput__label" htmlFor={id}>
    {placeholder}
  </label>

  <input
    className="DateInput__input"
   ....
  />

  <div
    className={cx('DateInput__display-text', {
      'DateInput__display-text--has-input': !!value,
      'DateInput__display-text--focused': focused,
      'DateInput__display-text--disabled': disabled,
    })}
    onClick={onFocus}
  >
    {value || placeholder}
  </div>
</div>

Hey @davincho, can you check if this is still an issue after https://github.com/airbnb/react-dates/pull/174 has been merged in?

Looks like the issue has been resolved! (checked on Linux with most recent Firefox with 'Simulate touch events' enabled and on Firefox for Android). Thanks!


Auto device placement for distributed runtime

In a distributed TF setting, we need to place variables and ops to different devices. This is annoying to manually assign each variable and op, especially when we have GPU resources in our environment.
TF offer a context named tf.train.replica_device_setter which place variables to ps devices in round-robin manner, and this is helpful but not enough. @mrry can you shed some light on the auto distributed devices placement problem?

@bhack this is the best I can do for now, pls feel free to correct me and provide some more thoughts. thx.

Thank you @windreamer. I also  find  this topic quite crucial for the "user experience" when we have distributed devices. Generally on cluster we could have really unbalanced GPU/CPU scalar resources ratio and there are different consideration to do for data parallel and model parallel approach. So it is really interesting to have some feedbacks. /cc @mtamburrano @lenlen

@vrv is working on improving the device placement so that policies will be more declarative, rather than binding to specific devices at graph construction time. This should address some of the limitations of tf.train.replica_device_setter(). If you had specific requirements, it would be good to collect them here, so that we could take them into consideration in the redesign.

I'm facing the bug here. I'm wondering if it is caused by the tf.train.replica_device_setter. If not, what is the most possible reason should it be caused ? Thank you.

See the discussion on that issue. The problem is most likely caused by the fact that tf.train.SyncReplicasOptimizer bakes the number of workers into the checkpoint, which is another issue that should probably be addressed.

Indeed, this is a much bigger project that we'll chip away at over time (optimizing for runtime across a distributed cluster is kind of a hard problem, and we're not good enough at it that manual placement typically produces better results).  We'd love to do a better job of this, so we'll be looking at this over the next little while.  Happy to brainstorm ideas here.

/cc @robertnishihara @maxpumperla

It would be better if parameter can be partitioned into pieces and well managed.

Closing due to inactivity. Feel free to open a new github issue if the problem still persists in recent versions.

@drpngx From last @vrv comment we are waiting news from google on this. Any progress or public roadmap?

@benoitsteiner is working on better cost models for auto-assignment I think, but I don't know what the expected roadmap for that work is.
C++ shape inference theoretically makes it possible to do an initial-placement algorithm that is better than the simple_placer, but right now the python runtime still maintains shape information that is not propagated to C++.  I think we need to wait for the control flow ops to be implemented via the C-API before we could make any more progress on initial static shape inference -- I believe the C-API stuff is being worked on.  There are probably some other ways to start playing around with this though (e.g., via add_shapes argument to as_graph_def()), but nobody I know is working on that.

I don't really know how you are solving this in production or managed cloud :) I know that you cannot disclose it but I don't think that you are handling manual placement.

We are now able to predict memory usage and performance for each operation of a TensorFlow graph. With this data we can start the actual work on optimizing TensorFlow models. We're completing work on a first optimizer that improves the performance of several models by about 20%, and we've also started to find a good device placement in parallel.
We'll start open sourcing this work as it matures. It's hard to predict how quickly research projects will succeed so unfortunately I don't have a roadmap to share.

@benoitsteiner Thank you for the update. We can live this open to track relevant progress.

/cc @jhseu

Probably through hacks and common deployment patterns. :P

@vrv I've seen your deleted comment. Another hypothesis are a secret army of manual Oompa-Loompa device placers ;)

Indeed, Oompa-Loompa's are not simple people.
(My deleted comment was in response to "I don't really know how you are solving this in production" to which I responded "hacks and common deployment scenarios".  I didn't see all the other comments after and it looked out of context ;)

:smile:

Since #4552 was closed and referenced here, I would just like to point out that there's a separate issue having nothing to do with distributed computing, and that is to prevent TF from scheduling too many concurrent operations at once such that it runs out of memory (on a single GPU). I have run into this problem many times where every single op can fit onto memory but because TF is overly aggressive about concurrency I end up not being able to run the model.

May I know the progress on this issue? Auto placement is a really nice shot for TF guys to benefit its distributed execution model. We have tried to do some kind of alike work by mixing data-parallelism with model-parallelism to get a trade-off between computation and communication and see promising improvement against standard data-parallelism implementation, also we are trying to automate this work. So maybe there are some potential collaboration with this issue.
@yuefengz @benoitsteiner

We've made a lot of progress on the placement side, which will be detailed in our upcoming ICML paper.
We've also automated data-parallelism in the  grappler auto parallelizer. We've tested it successfully on several models and we'll start documenting it as soon as possible.
We're open to a collaboration. Is there any documentation or paper you can point us to to get started thinking about what a common solution could look like ?

/cc @klueska Any feedback?

Got the name of the paper: Device Placement Optimization with Reinforcement Learning. Looking forward to the preprint!

@benoitsteiner
Thanks for the nice progress.
Our work haven't been recorded into any paper since we are still on the way with it.
As to the grappler auto parallelizer, we do have taken a look at its implementation. It is nice and somehow is what we want, although currently it only support naive BSP auto placement(Is my understanding correct?). My question is that why not to implement the grappler auto-parallelizer in python side, since in python side, there are more graph metadata and the processing overhead should be less than implementing it in c++ side.
As to the starting point of our solution, we are working on designing a cost model to evaluate the potential placement solution to help find a sub-optimal one since we think the auto-placement problem actually is a NP-hard problem, it is non-trivial to find the optimal solution.
We have added hybrid-parallelism for AlexNet into python side(actually transform the placement problem into a graph re-written problem), which is a mixture of data-parallelism and model parallelism and the speed-up against standard implementation is significant. The implementation overhead actually is not quite large. And also we are trying to follow this strategy to make it as automatic as possible, also as general as possible.
Not sure how far your device placement optimization work has gone.
As to your nice paper(Device Placement Optimization with Reinforcement Learning) to be published on ICML 2017, will you put it on arXiv before the conference?
Thanks

Our Device Placement Optimization with Reinforcement Learning paper has just been published on arXiv.

@benoitsteiner
Thanks for the update, already taken a go-through of your nice paper and really appreciate your fancy RL-based idea illustrated in the paper.
It looks that so far the device placement optimization is still time consuming(around 20 hour with 80 GPUs), so will you integrate it into the TF master revision? Also may I know inside Google how often you will use this RL-based automatic placement solution to accelerate the model training process?
Thanks

hi, why not take a little step forward,  like users should explicitly declare where the computation should happens (cpu, gpu, ps), but just with a 'virtual' device id, then tf automatically schedule the sub graphs to any free devices, or a preferred device list would be even better for migration between distrubuted, multi-gpus, single gpu and cpu only. And I think this would be much easier for us to try those training tf models in our own environment.
PS, I really don't want to try the distributed tf after I checked the tutorials, -,-

I read the paper. I only wish there was a bit more detail on the RL training procedure. My I ask when would we be able to read some more documentation on this topic?

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

This issue could have some connection with https://github.com/tensorflow/k8s/issues/35 /cc @jlewi

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Closing this issue as it's too broad. For the common case where graphs have one compute device and many parameter server devices we currently support automatic placement. For more complicated cases it's an open research problem how to do this.

@benoitsteiner    I have read your  Device Placement Optimization with Reinforcement Learning paper, but I have some questions about it. First, this paper has given the cost function and used adam optimizer to update parameters, what's the use of formula (3) in this paper? Second, how can I calculate the probability of placement P? Finally, is the code for this paper open source? Thank you.

https://ai.googleblog.com/2020/12/end-to-end-transferable-deep-rl-for.html

https://arxiv.org/abs/2201.12023

https://arxiv.org/abs/2201.12023
Hello, could I ask if there is a plan of open source the code? I am curious about what your paper mentioned :

Since there are only less than 80 kinds of common primitive operators in DL graphs, we can manually enumerate the possible parallel algorithms for all primitive operators.

Is there any doc or code relating to this?


Imagepicker isn't working with the video saved after trimming

Steps to reproduce -

Use imagepicker.pickVideo to select video from file manager
video_player plays the video successfully
Use trimmer to trim the picked video
Save the trimmed video
imagepicker.pickVideo the trimmed video to select the trimmed video now

Observation -

Trimmed video is not initialisating and is not playable in the video_player.

Code to save the video -
await widget._trimmer
        .saveTrimmedVideo(
            startValue: _startValue,
            endValue: _endValue,
            outputFormat: FileFormat.mp4,
            videoFileName: 'Trimmed_${new DateTime.now()}',
            videoFolderName: "Videos/Trimmed",
            storageDir: StorageDir.externalStorageDirectory)
        .then((value) {
      setState(() {
        _progressVisibility = false;
        _value = value;
      });
    });

Please let me know if you need more details on reproducing this issue?
Thanks!

@sbis04

This could be related - https://stackoverflow.com/questions/48356188/flutter-video-player-not-playing-video-saved-to-application-directory

Actually I figured out the reason after a bit of manual testing. If I trim the video to less than 1s and between 0s & 1s, the duration of the video shows up as zero in the Android File manager and hence the video player is unable to initialise that particular video.

@sbis04  Is there a way to figure out Duration of the video after trimming ?

Hi @saurabhrayakwar, currently there is no way to set the maximum trimmed video duration. I know this is a really important feature and is currently in development. However, you are open to submit PR.

Closing this issue as there is already an issue regarding this.
Duplicate of: #4


Convert Registry readonly public fields to properties with explicit getter/init behavior

          property would be better, as it gets public; better read-only if never changed.

Originally posted by @vlada-shubina in https://github.com/dotnet/sdk-container-builds/pull/319#discussion_r1096321506
This is found by CA1051, which we should enable as part of the PR solving this.

Also address: https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1051
https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2211 - Non-constant fields should not be visible in scope of this issue.


Force turnServers.urls into an array

If a single server is given a string can be used which must here be forced into an array.

Fixes #238

Rather than use toString to test if it is not an Array, I'd much rather enforce the 'string or Array' logic using typeof and instance of, respectively, as we do elsewhere in the library (see UA configuration parsing, for example).
That being said, I posted some questions on Issue #238 itself that should be answered before we consider a pull request.


⚠ Fortnite has degraded performance

In ebfdbe7, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1745 ms

Resolved: Fortnite performance has improved in a206b4c.