[QUESTION]

I have used c++ producer sdk for raspberry pi for live video and audio streaming .I have installed Ubuntu os in raspberry pi4 model and i am successfully access the live video streaming with this command (./kvs_gstreamer_sample kvs_example_camera_stream) but while i want to access  audio with video i am not able to access the live video and audio stream with this command  and configured these command in .sh file.
Note =>but normal video stream working fine
kvs_gstreamer_audio_video_sample kvs_example_camera_stream
configured credential
export AWS_DEFAULT_REGION=ap-south-1
export AWS_ACCESS_KEY_ID= ***********************
export AWS_SECRET_ACCESS_KEY=*******************
export GST_PLUGIN_PATH=pwd/amazon-kinesis-video-streams-producer-sdk-cpp/build
export LD_LIBRARY_PATH=pwd/open-source/local/lib
export AWS_KVS_AUDIO_DEVICE=hw:2,0
cd amazon-kinesis-video-streams-producer-sdk-cpp/build/
./kvs_gstreamer_audio_video_sample kvs_example_camera_stream
getting error--------------------------------
bash: cd: amazon-kinesis-video-streams-producer-sdk-cpp/build/: No such file or directory
[INFO ] [03-10-2023 13:00:54:611.167 GMT] Using region: ap-south-1
[INFO ] [03-10-2023 13:00:54:612.026 GMT] Using aws credentials for Kinesis Video Streams
[INFO ] [03-10-2023 13:00:54:612.382 GMT] No session token was detected.
[INFO ] [03-10-2023 13:00:54:616.691 GMT] createKinesisVideoClient(): Creating Kinesis Video Client
[WARN ] [03-10-2023 13:00:54:616.977 GMT] fixupClientInfo(): Connection timeout is invalid...setting to default
[WARN ] [03-10-2023 13:00:54:617.105 GMT] fixupClientInfo(): Completion timeout is invalid...setting to default
[INFO ] [03-10-2023 13:00:54:617.245 GMT] heapInitialize(): Initializing native heap with limit size 134217728, spill ratio 0% and flags 0x00000001
[INFO ] [03-10-2023 13:00:54:617.345 GMT] heapInitialize(): Creating AIV heap.
[INFO ] [03-10-2023 13:00:54:617.519 GMT] heapInitialize(): Heap is initialized OK
[DEBUG] [03-10-2023 13:00:54:618.034 GMT] getSecurityTokenHandler invoked
[DEBUG] [03-10-2023 13:00:54:618.356 GMT] Refreshing credentials. Force refreshing: 0 Now time is:<PHONE_NUMBER>618221939 Expiration: 0
[INFO ] [03-10-2023 13:00:54:618.561 GMT] New credentials expiration is<PHONE_NUMBER>
[INFO ] [03-10-2023 13:00:54:618.750 GMT] createDeviceResultEvent(): Create device result event.
[DEBUG] [03-10-2023 13:00:54:618.902 GMT] clientReadyHandler invoked
[DEBUG] [03-10-2023 13:00:54:619.135 GMT] Client is ready
[INFO ] [03-10-2023 13:00:54:619.585 GMT] Creating Kinesis Video Stream kvs_example_camera_stream
[INFO ] [03-10-2023 13:00:54:619.920 GMT] createKinesisVideoStream(): Creating Kinesis Video Stream.

[WARN ] [03-10-2023 13:00:54:984.423 GMT] describeStreamResult(): Retention period returned from the DescribeStream call doesn't match the one specified in the StreamInfo
[WARN ] [03-10-2023 13:00:54:984.489 GMT] describeStreamResult(): Content type returned from the DescribeStream(video/h264) call doesn't match the one specified in the StreamInfo(video/h264,audio/aac)
[WARN ] [03-10-2023 13:00:54:984.539 GMT] describeStreamResult(): [kvs_example_camera_stream] Retention period returned from the DescribeStream call doesn't match the one specified in the StreamInfo
[WARN ] [03-10-2023 13:00:54:984.587 GMT] describeStreamResult(): [kvs_example_camera_stream] Content type returned from the DescribeStream call doesn't match the one specified in the StreamInfo
[INFO ] [03-10-2023 13:00:55:338.168 GMT] writeHeaderCallback(): RequestId: a2311647-6a2b-4505-a4d0-e08d33884326
[DEBUG] [03-10-2023 13:00:55:338.556 GMT] getStreamingEndpointCurlHandler(): [kvs_example_camera_stream] GetStreamingEndpoint API response: {"DataEndpoint":"https://s-703775db.kinesisvideo.ap-south-1.amazonaws.com"}
[INFO ] [03-10-2023 13:00:55:344.436 GMT] getStreamingEndpointResultEvent(): Get streaming endpoint result event.
[DEBUG] [03-10-2023 13:00:55:344.805 GMT] getStreamingTokenHandler invoked
[DEBUG] [03-10-2023 13:00:55:344.957 GMT] Refreshing credentials. Force refreshing: 1 Now time is:<PHONE_NUMBER>344935848 Expiration:<PHONE_NUMBER>
[INFO ] [03-10-2023 13:00:55:345.076 GMT] New credentials expiration is<PHONE_NUMBER>
[INFO ] [03-10-2023 13:00:55:345.217 GMT] getStreamingTokenResultEvent(): Get streaming token result event.
[DEBUG] [03-10-2023 13:00:55:345.385 GMT] streamReadyHandler invoked
[DEBUG] [03-10-2023 13:00:55:345.999 GMT] Stream is ready: kvs_example_camera_stream
[DEBUG] [03-10-2023 13:00:55:474.421 GMT] appsink configured
[INFO ] [03-10-2023 13:00:55:598.252 GMT] Using default video device
[INFO ] [03-10-2023 13:00:55:602.745 GMT] Using audio device hw:2,0
Not all elements could be created:
[INFO ] [03-10-2023 13:00:55:606.321 GMT] stopKinesisVideoStreamSync(): Synchronously stopping Kinesis Video Stream 0000aaaaeb434280.
[DEBUG] [03-10-2023 13:00:55:606.451 GMT] streamClosedHandler invoked for upload handle:<PHONE_NUMBER>3709551615
[INFO ] [03-10-2023 13:00:55:606.527 GMT] Freeing Kinesis Video Stream for kvs_example_camera_stream
[INFO ] [03-10-2023 13:00:55:606.588 GMT] freeKinesisVideoStream(): Freeing Kinesis Video stream.
[DEBUG] [03-10-2023 13:00:55:606.628 GMT] freeStream(): [kvs_example_camera_stream] Freeing stream
[DEBUG] [03-10-2023 13:00:55:606.669 GMT] curlApiCallbacksShutdownActiveRequests(): pActiveRequests hashtable is empty
[DEBUG] [03-10-2023 13:00:56:007.183 GMT] freeStream(): [kvs_example_camera_stream] Completed freeing stream
[INFO ] [03-10-2023 13:00:56:007.661 GMT] Freeing Kinesis Video Stream for kvs_example_camera_stream
[INFO ] [03-10-2023 13:00:56:007.892 GMT] freeKinesisVideoClient(): Freeing Kinesis Video Client
[DEBUG] [03-10-2023 13:00:56:008.020 GMT] curlApiCallbacksShutdownActiveRequests(): pActiveRequests hashtable is empty
[DEBUG] [03-10-2023 13:00:56:408.752 GMT] freeKinesisVideoClientInternal(): Total allocated memory 0
[INFO ] [03-10-2023 13:00:56:409.092 GMT] Completed freeing client
[WARN ] [03-10-2023 13:00:56:409.286 GMT] curlApiCallbacksShutdown(): curlApiCallbacksShutdown called when already in progress of shutting down
also applied the second method but it will also does not work so kindy suggest with above method
export GST_PLUGIN_PATH=pwd/build
export LD_LIBRARY_PATH=pwd/open-source/local/lib
gst-launch-1.0 -v v4l2src device=/dev/video0 ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency ! h264parse ! kvssink stream-name="kvs_example_camera_stream" access-key="****************" secret-key="************************** alsasrc device=hw:2,0 format=S16LE rate=44100 ! audioconvert ! voaacenc ! queue ! kvssink

Closing this as it is a duplicate of https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/discussions/1081


x/tools/cmd/gopls: check command report function redeclared with another build tag

Please answer these questions before submitting your issue. Thanks!
What did you do?
In repository https://github.com/wingyplus/wire-tut. I have 2 InitializeEvent functions, one come from me write by my hand (di.go) and one generated from google/wire command.
When I ran gopls check ./di.go.
What did you expect to see?
No report anything because InitializeEvent in di.go has a wireinject build tag.
What did you see instead?
gopls report:
$ gopls check ./di.go
/Users/thanabodee/workspace/wire-tut/di.go:7:6-21: InitializeEvent redeclared in this block

Build info
golang.org/x/tools/cmd/gopls
<EMAIL_ADDRESS>h1:X0hWR11/1QMjiO+1j6iB5GYJWYvBOl7/VAI3u0SXMno=
<EMAIL_ADDRESS>h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=

Go info
go version devel +16bf0d5e03 Thu May 2 23:12:11 2019 +0000 darwin/amd64

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/thanabodee/Library/Caches/go-build"
GOENV="/Users/thanabodee/Library/Preferences/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/thanabodee"
GOPROXY="direct"
GOROOT="/Users/thanabodee/tools/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/Users/thanabodee/tools/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/thanabodee/workspace/wire-tut/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/x7/tl6gyzn92vzcr35t94xgt6y00000gp/T/go-build053619932=/tmp/go-build -gno-record-gcc-switches -fno-common"

This is because gopls does not very good at coping with build tags
we are tracking that in #29202 and although this is an interestingly different symptom I don't think it a different root cause.

I see. Thanks for your clarify.


Setting check run to neutral when timeout if not failure

Current dependencies on/for this PR:

main

PR #695

PR #696   👈

This comment was auto-generated by Graphite.


Implement Upgrade Action

The CNAB spec specifies that an invocation image should implement an upgrade action. We should implement that here to be compliant, even if it means doing nothing for now.

This was implemented in https://github.com/deislabs/porter-helm/pull/13; closing.


[mountvol] Fix typo in syntax summary

Fix a typo involving < angle brackets >.

@glenn-slayden : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Automate creation of release artifacts

This sets up a new workflow action that triggers when a new tag is created, building the Maven repo and the Figma resources and uploading them to the GitHub release for that tag. The Gradle build will have the version set to the tag's name, so if a tag named "v0.20.0-rc03", the DesignCompose libraries will be version "0.20.0-rc03". (No more making commits to set the version to rc02 or rc03 or whatever).

I did some test releases in a fork of the repo. Here's a test using the latest push: https://github.com/timothyfroehlich/automotive-design-compose/releases/tag/v0.2.test2


Error: Plugin docusaurus-plugin-google-analytics is used 2 times with id=default.

🐛 Bug Report
I upgraded to the latest version and got this
Error: Plugin docusaurus-plugin-google-analytics is used 2 times with id=default.
To use the same plugin multiple times on a Docusaurus site, you need to assign a unique id to each plugin instance.

Have you read the Contributing Guidelines on issues?
Yes
To Reproduce
Install google analytics plugin on a docusaurus instance (it might happen with other plugins but we only use the GA one)
Try it here https://github.com/openmultiplayer/wiki/ using commit 2b2321341d32121ff3195fc801d0831b1693ce10
I fixed it by giving the GA plugin an explicit ID: https://github.com/openmultiplayer/wiki/commit/6fada686d8a85fab6cf6b94cb8f5734e29d77a1e
Your Environment

Docusaurus version used: v2 alpha 66
Environment name and version (e.g. Chrome 78.0.3904.108, Node.js 10.17.0): latest stable node
Operating system and version (desktop or mobile): windows and vercel, so linux I think

Hi,
If you use the classic preset with googleAnalytics themeConfig, you already use the google analytics plugin, as it's included in the preset.
So you actually have 2 instances of the google analytics plugin running on your site, which is likely not something you want right?
I'll improve the error message to make this clearer

Ah, I didn't know it was added by default now. Thanks1

it is added only when your preset classic has the googleAnalytics config attribute

Hello @slorber,
Sorry to comment on this closed issue, but I wonder where to put exactly the googleAnalytics config with preset classic?
I've tried:
presets: [
    [
      '@docusaurus/preset-classic',
      {
        themeConfig: {
          googleAnalytics: {
            trackingID: 'G-ZGCNWVF92C',
            // Optional fields.
            anonymizeIP: true, // Should IPs be anonymized?
          },
        },
      },
    ],
  ],

And:
presets: [
    [
      '@docusaurus/preset-classic',
      {
        googleAnalytics: {
          trackingID: 'G-ZGCNWVF92C',
          // Optional fields.
          anonymizeIP: true, // Should IPs be anonymized?
        },
      },
    ],
  ],

And:
module.exports = {
  themeConfig: {
    googleAnalytics: {
      trackingID: 'G-ZGCNWVF92C',
      // Optional fields.
      anonymizeIP: true, // Should IPs be anonymized?
     },
  },
}

But no success so far 😄
Version: 2.0.0-alpha.69
Thank you for your help!

https://v2.docusaurus.io/docs/api/plugins/@docusaurus/plugin-google-analytics/#configuration
this one should work:
module.exports = {
  themeConfig: {
    googleAnalytics: {
      trackingID: 'XXX',
      // Optional fields.
      anonymizeIP: true, // Should IPs be anonymized?
     },
  },
}

Note: it is only enabled on the production build, so you'll have to use docusaurus build && docusaurus serve to test that

Thanks @slorber for the quick answer. 😉
The "working" solution was the one I tried first but alas it did and still do not work.
I'm using a GitHub workflow to deploy my documentation on the gh-pages branch. It runs the following instructions at the end:
yarn install --frozen-lockfile
yarn deploy

So either:

There is a bug (I doubt it)
I did not configure my Google Analytics correctly (I doubt it)
It cannot work with GitHub hosting?
Should I configure gtag too? (https://v2.docusaurus.io/docs/api/plugins/@docusaurus/plugin-google-gtag/)

Thanks for your time! 😄

@gulien I will only be able to help you if you provide me a repro using the config I gave you + a live deployment where I can indeed see it does not work.

My bad the documentation source code is located here:
https://github.com/thecodingmachine/symfony-boilerplate/tree/master/docs
And the documentation is available here: https://thecodingmachine.github.io/symfony-boilerplate/

This works, as you can see

Ok so GA do not show anything, so I guess I actually did not configura GA correctly on my side...
Sorry for that, have a nice day :)

that's probably it yes :) afaik it works on RN website

I am using @docusaurus/preset-classic, in my case, it is not collecting data. I tried to google analytics as well as gtag method but neither one is working.

@adsazad there were changes recently on this (config was moved from themeConfig to plugin options), and we can't help if you don't provide your docusaurus version and provide more context
If you follow the official doc of your exact docusaurus version (not production doc), you should be able to make it work and see GA requests in the network tab
Also note that Google launched Google Analytics 4, to progressively replace all the former analytics generations, and we don't support that yet. There's an issue to support it here: https://github.com/facebook/docusaurus/issues/7221


Sound problem

Hello everyone when I launch parsec soda and the people I am on a call with join there is a crackling sound can you help me please

This is a known issue. The crackling occurs due to how Smash Soda mixes audio...it's on my todo list though, will sort it soon!


Support create empty file and open for write/RW workloads

What changes are proposed in this pull request?
Fixes #15569
Support FIO sequential write test
fio --name=sequential-write --directory=/mnt/alluxio-fuse --rw=write --refill_buffers --bs=4M --size=4G --end_fsync=1
Why are the changes needed?
The sequential write workload failed with
Fuse.Create(path=/sequential-write.0.0,mode=100644)
Fuse.Flush(path=/sequential-write.0.0,fd=1)
Fuse.Release(path=/sequential-write.0.0,fd=1)
Fuse.Open(path=/sequential-write.0.0,flags=0x8002(READ_WRITE))
Fuse.Write(path=/sequential-write.0.0,fd=2,size=131056,offset=0)
Cannot overwrite existing file /sequential-write.0.0 without O_TRUNC flag or fuse.truncate(size=0), flag 0x8002

and Alluxio do not support create empty file then open for write
Does this PR introduce any user facing changes?
Support create empty file and open for write or open for readOrWrite

Validated that with this change can help pass the sequential write test
[alluxio@Ec2Cluster-worker-0 alluxio]$ fio --name=sequential-write --directory=/mnt/alluxio-fuse --rw=write --refill_buffers --bs=4M --size=4G --end_fsync=1
sequential-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
sequential-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=784MiB/s][r=0,w=196 IOPS][eta 00m:00s]
sequential-write: (groupid=0, jobs=1): err= 0: pid=12154: Thu Jun 16 22:46:48 2022
  write: IOPS=179, BW=719MiB/s (754MB/s)(4096MiB/5700msec)
    clat (msec): min=3, max=140, avg= 4.85, stdev= 5.91
     lat (msec): min=3, max=140, avg= 4.85, stdev= 5.91
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    6], 95.00th=[   17],
     | 99.00th=[   25], 99.50th=[   27], 99.90th=[   29], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=393216, max=819200, per=99.49%, avg=732066.91, stdev=122894.89, samples=11
   iops        : min=   96, max=  200, avg=178.73, stdev=30.00, samples=11
  lat (msec)   : 4=84.86%, 10=8.69%, 20=3.81%, 50=2.54%, 250=0.10%
  cpu          : usr=14.25%, sys=2.83%, ctx=33797, majf=0, minf=31
  IO depths    : 1=100.1%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=719MiB/s (754MB/s), 719MiB/s-719MiB/s (754MB/s-754MB/s), io=4096MiB (4295MB), run=5700-5700msec

Error: 2.814 [ERROR] alluxio.util.JvmPauseMonitorTest.testMockedInfoPause  Time elapsed: 1.609 s  <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at alluxio.util.JvmPauseMonitorTest.testMockedInfoPause(JvmPauseMonitorTest.java:106)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at

alluxio-bot, merge this please


[DISCUSSION] Lawnicons not applying dark-theme icon pack to other launchers

About this discussion
I've just updated to the latest version (2.0.0) on my Android 11 (OxygenOS 11) device.
The update means icons can be applied to launchers beside Lawnchair.
I read that:

Lawnicons will have a dark or white background depending on the system theme

My device has a systemwide dark theme which supports all apps with this feature. However, I am unable to view dark icons regardless of the system theme's state (both dark and light).
Applying Lawnicons to launchers (except Lawnchair) shows light-themed icons:

Lawnicons applied to other launcher (systemwide dark theme)
I also noticed that the About tab of the Lawnicons app shows the app icon in a light background when the systemwide dark theme is enabled and vice versa:

Lawnicons About tab (systemwide dark theme)

Lawnicons About tab (systemwide light theme)
Are the icons (when applied to other launchers) also expected to be black and white for devices that don't natively support material you (Android 11 and below), or does the app have support for wallpaper-based colouring on all devices (like Lawnchair)? I am unable to produce dark-themed icons, so I am unable to tell myself.

I am also experiencing same issue, MIUI 12.5 Android 11.

Pixel 7 Pro with GrapheneOS and Lawnchair:
When I set an application's icon manually, it does not adapt to changes between light and dark themes, e.g. when I set the light themed icon while my system theme is light, it remains light even if my system switches to a dark theme afterwards and vice versa.
When I press and hold on the icon in order to change it via "Customize", the icon shown in the pop-up window respects the system theme regardless of the icon theme on the home screen.
Everything works perfectly for icons that are set automatically because the app itself has an icon in the icon pack.

The same in Android13,OnePlus 8 Pro,Nameless Launcher

The core issue in the first post is fixed in the latest version (2.1). The colors for pre-A12 devices was reversed, and now fixed in that version.
As for the themed icons not properly changing colors (A12+ only), try restarting Lawnchair, change the icon pack (to trigger icon cache reset), re-apply the icons.

As for the themed icons not properly changing colors (A12+ only), try restarting Lawnchair, change the icon pack (to trigger icon cache reset), re-apply the icons.

Yeah, It works if you change the icon pack a few times and restart Lawnchair, but still it's really bad UX. I hope there's a way to clear cache automatically on wallpaper or system color change


scam nsfw from twitter

NSFW spam shared on twitter fakes with nudes.

quinni.cyou
NSFW
20220902-1662123080

alenai.cyou same as above

Thank you Tomasz!
Merging.


Testing new endpoints for v1.1.0 release

The develop branch has quite a few more endpoints than the main branch, and it's time to merge them to create a new release.
This is a call for testing - please try to break the endpoints below!
Any comments, issues, feedback or insights are welcome as issue comments here.
Any typos or inconsistencies are an extra bonus.
When the PR is merged for the next version, it will automatically close this issue.

List of new endpoints found on the dev deployment (all the details are in the docs):

https://citycoins-api.citycoins.workers.dev/mining/get-block-winner-id/{cityname}/{blockheight}
https://citycoins-api.citycoins.workers.dev/mining/get-last-high-value-at-block/{cityname}/{blockheight}
https://citycoins-api.citycoins.workers.dev/mining/has-mined-at-block/{cityname}/{blockheight}/{userid}
https://citycoins-api.citycoins.workers.dev/mining-claims/can-claim-mining-reward/{cityname}/{blockheight}/{address}
https://citycoins-api.citycoins.workers.dev/mining-claims/is-block-winner/{cityname}/{blockheight}/{address}
https://citycoins-api.citycoins.workers.dev/stacking/stacking-active-at-cycle/{cityname}/{cycleid}
https://citycoins-api.citycoins.workers.dev/stacking-claims/get-stacking-reward/{cityname}/{cycleid}/{userid}
https://citycoins-api.citycoins.workers.dev/tools/prices/{cityname}/{currency}
https://citycoins-api.citycoins.workers.dev/tools/proof-of-hodl/{cityname}/{address}

Outstanding tasks before merge:

[ ] update package.json version number
[ ] update OpenAPI version number

Also noting a few things that need to be updated in the docs as well before this gets merged:

[ ] add links to new OpenAPI tags for Mining and Stacking claims pages
[ ] move get-stacking-reward to Stacking Claims page

Found one, using a value that is not a valid Stacks address crashes this endpoint:
https://citycoins-api.citycoins.workers.dev/tools/proof-of-hodl/mia/test
Should return either the default (false) or throw a 404 for stacks address not found. Leaning towards the former.


Bring in Optics f2py update

For some reason, this branch seems to have gotten lost. Mea culpa.
So, let's bring it in.
Closes #36

I'm throwing a blocker on this until I can test to be sure it works. Because f2py doesn't run on our CI


[Task]: Validate if access given previous to the sheets can be invoked or not?

Is there an existing issue for this?

[X] I have searched the existing issues

SubTasks
Check if access given to the sheets previously can be revoked or not?

Keeping this open as we have asked the question related to this on google drive forum

We have not received any response on the question we had asked on google drive community forum. Alternatively we decided to save authorised file ids in the datasource, thus associating file ids with datasource and hence solving the issue of which files were authorised for any given datasource. Hence closing this issue


Code actions widget has hover feedback on headers which are not interactable

Version: 1.73.0-insider (user setup)
Commit: fad3a77833b9249158dfd88477114a06435e46a2
Date: 2022-10-07T05:18:17.049Z
Electron: 19.0.17
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: Yes

Hm, can't repro this on mac or PC. Are you seeing this in every instance of the code actions widget?

Seems to be, I can't DOM inspect it easily as it relies on focus

This also allows keyboard focus which is also bad because it's non-interactable. cc @meganrogge

cannot reproduce on macOS

If it's only in production and !important fixed I'm guessing the CSS order is different when bundled? The hover rule probably just needs something like :not(.header)

@Tyriar I wasn't able to repro in mac or pc with either insiders or in production today. Have you still been running into this? I did have a fix on https://github.com/microsoft/vscode/issues/187492 which should be in insiders now, but I'll keep digging to see if it may be related.

@justschen it's working for me now. Something you might want to consider changing is the blue color for the headers as blue text typically indicates a link. Maybe a small topic for the UX sync?


Fix hex escape to not include letters after "F"

See the docs of Oniguruma for their definition of \h:
https://github.com/kkos/oniguruma/blob/master/doc/RE#L68

Thanks! Quite right, not sure how I made that mistake.


WIP: Add C++ Pattern Validation

Envoy now uses RE2 as a safe regex engine instead of std::regex (https://github.com/envoyproxy/envoy/pull/7878). Because PGV already requires patterns to use RE2 syntax, one option is to use RE2 for C++ patterns as well. This implements it, for use in strings, bytes, repeated items, and may key/value pattern validation.
Implements https://github.com/envoyproxy/protoc-gen-validate/issues/22
WIP: I ran in to difficulty creating the regex because a regex containing a null character would get cut off... for example, the ascii character test used the pattern, ^[\x00-x7f]+$, and consuming this as a string resulted in creating a null-terminated string pattern ^[ instead of the actual pattern. I think this might be a problem across most of the C++ code? That's why there's a terrible string construction in the pattern creation.
Signed-off-by: Asra Ali<EMAIL_ADDRESS>

@akonradi

@rmichela @rodaine would either one of you be able to give this a pass over? thank you!

This looks fine to me, but I'm not terribly familiar with C++ enough to say if this is ideal or not.
Looking in Envoy, the RE2 usages use a StringPiece. Is that applicable here? https://github.com/envoyproxy/envoy/blob/master/source/common/common/regex.cc#L47

(Sorry just getting back to this after KubeCon) Oh, thanks Chris and Alex. I think we can get the best of both worlds with that re2::StringPiece. Should avoid the copy. I'll update and test using that for both the matcher and the value.

I used re2::StringPiece where I could to avoid copies. I do have a question, seems like the pattern is to panic if a regex is invalid, so I eliminated re2::Quiet, it'll throw an error if it's an invalid pattern like the go code does.

I assume we're validating the regex when generating the code, so we shouldn't see invalid regexes during validation. I think the behavior of failing loudly is reasonable.


Update regenerate.sh to include all recursive subdirs containing a sqlc.json

This includes the postgresql and mysql subdirectories of some of the test cases.

This change is

Thanks!


Added new files

Is all good - let me know.

Hey there,
Do you see the changes I brought about?

ionut.md


Same as last report

Exception during loading of package
Traceback (most recent call last):
  File "/data/venv/lib/python3.9/site-packages/redbot/core/core_commands.py", line 188, in _load
    await bot.load_extension(spec)
  File "/data/venv/lib/python3.9/site-packages/redbot/core/bot.py", line 1680, in load_extension
    lib = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 529, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1029, in load_module
  File "<frozen importlib._bootstrap_external>", line 854, in load_module
  File "<frozen importlib._bootstrap>", line 274, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 711, in _load
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/data/cogs/CogManager/cogs/assistant/__init__.py", line 6, in <module>
    from .assistant import Assistant
  File "/data/cogs/CogManager/cogs/assistant/assistant.py", line 7, in <module>
    from .abc import CompositeMetaClass
  File "/data/cogs/CogManager/cogs/assistant/abc.py", line 7, in <module>
    from .models import DB, Conversations, GuildSettings
  File "/data/cogs/CogManager/cogs/assistant/models.py", line 6, in <module>
    from openai.embeddings_utils import cosine_similarity
  File "/data/cogs/Downloader/lib/openai/embeddings_utils.py", line 5, in <module>
    import plotly.express as px
ModuleNotFoundError: No module named 'plotly'

Seems to also require plotly

added plotly to requirements info json, i guess the openai lib doesnt include those like its supposed to, just message me in the cog support server in my channel if theres another missing dependency


Add the postgresql94 SCL yum repo for post deployment updates.

cc @abellotti @carbonin

What was Please also add to "build time repos" for? @abellotti


Compress fonts [was; Compress WOFF and PNG fonts?]

I ran woff-compress on the WOFF fonts and zopflipng on the PNG fonts and achieved a size reduction of ~3.5% for WOFF files and ~25% for PNG files (I don't know if the PNG fonts are still relevant nowadays). 3.5% is not much, but it has to be done only once and thus might still be worth doing to save some bandwidth.

~3.5% for WOFF

Nice. We are re-working the fonts for v3 and that will be useful to do.
Also, WOFF2 is important (see the relevant branch for a test).

I don't know if the PNG fonts are still relevant nowadays

This came up in https://github.com/mathjax/MathJax/pull/1613.
They are rarely used these days (badly served webfonts, user settings blocking webfonts). The review of a change would be quite time consuming (and necessary because bugs in zopfli compressors do exist). The PNG fonts will also be dropped from v3.
Overall, it seems not worth spending the necessary resources.

Agreed: the png fonts will not be part of v3, and are rarely used even in v2, so not worth spending the effort on them.

Renamed and re-labeled.

See #1990.


Updating dependencies

This commit updates the dependencies of Aria Templates.
Before integrating it, we should:

[x] publish a new version of at-packager
[x] update the version of at-packager in package.json (so that we do not have a git+https url)

@fbasso Thank you for your review!

The travis build failure is caused by the following bug in version 1.7.4 of UglifyJS: https://github.com/mishoo/UglifyJS2/issues/1345


Adding Tradiio connector

Adding Tradiio.com connector to web scrobbler.
Tradiio is a new music discovery App, where users decide and help upcoming artists rise.
Become more than a passive listener. Play your part in your new favourite artists’ careers.
Artists that perform well, get opportunities such as label deals, festival perfomances, music videos and even radio play.

Connector is not defined?
Hi David is this a major problem?

This looks good.
The reason it's failing jshint is because you changed the inline configuration.  The following line tells the  jshinter that Connector is a global variable:
/* global Connector */

which you changed to /* Tradiio Connector */.  Just change it back and it should pass.
To make the connector more complete, I would suggest adding the current time and track art functionality.  Current time will allow the extension to detect repeated plays.
Connector.currentTimeSelector = '.songDuration.s-progress';

Connector.getTrackArt = function () {
	return $('.cover-small img').attr('src').replace('-square.jpg', '-normal.jpg');
};

Cheers.

Hi @Paszt Many thanks for your help.
Fixed

@andremoniz - Why did you change the value of currentTimeSelector?  It won't work with the changes you made.  currentTimeSelector is expecting a jquery selector that will find the element containing track current time in h:m:s format.  The value of '.leftmusicplayer .progress-play' is the progress bar which will always have a text value of white space.  '.songDuration.s-progress' will give you the current time.

Fixed

@andremoniz did you push the change?

I did a new pull request


Corrected scale representation for NUMERIC/DECIMAL

@gunnarmorling This PR is a proposal to fix the issue for the missing trailing zeros in the double representation of a NUMERIC/DECIMAL data type in PostgreSql.
The bug is reported in the Issue DBZ-287.

Awesome, thanks a lot! I'm at conference today, but I'm going to review
this asap.

Any updates on merging this PR? This is a blocker for us.

I am sorry for the delay, Gunnar is away this week. I'll take look at it tomrrow. Is there any chance you could provide integration test for the functionality. Also just a reminder - having the issue merged does not mean that we will release the new version with the fix - the release date for 0.5.2 has not been set yet.

@user453584 @jpechane I am on vacation till 27 of July, anyway I'll try to provide integration tests in the next days.

@jpechane I have added integration test for the fix and verified that it correctly check the scenario fixed with this PR, so it fails with the actual master and passes with this branch.

In terms of a release, I would think that we can ship a 0.5.2 some time next week, including this change and hopefully a few others from the pending PRs. They warrant another bugfix release for sure.

🙌

@gunnarmorling I've squashed all my commits in just one commit, hope it is as you expected. I found a different solution for the fixing, not using the string but with the more elegant usage of MathContext: just changing the scale to the existing decimal value unfortunately does not solve the problem.

Rebased and applied. Thanks a lot, @mcapitanio! Note that I changed the implementation to use setScale(), as I think it's better to avoid the double conversion of the previous approach. The tests pass (also changed them to use a larger scale), so it should be good. Thanks again!

@gunnarmorling Absolutely, the setScale is the best solution, I didn't realize it was necessary to make the re-assignement! Hope in helping Debezium again, Postgres connector is so crucial for us!

Awesome, your contributions are always welcome! A release for shipping this
fix will be done soon.

@gunnarmorling Thanks for helping this issue along (and @mcapitanio for the PR). @gunnarmorling What is the status of the 0.5.2 release? This bug has been a blocker for us, so we're hoping to get the new release pulled down as soon as possible

Hi, the 0.5.2 release is scheduled for next week. If you like, you can download the latest snaphot build from here: https://oss.sonatype.org/content/repositories/snapshots/io/debezium/.


Hierarchy redo

With 1.20 features copy and for all sections above them.

This whole PR is wrong. there is only one folder you can change and that's data colleciton!


[Core] NRE when using an invalid method for entryPoint

I had the wrong namespace in my manually-authored function.json, and I just got a NRE when running the Core Tools:
Listening on http://localhost:7071/
Hit CTRL-C to exit...
Object reference not set to an instance of an object.

@MikeStall this may be an issue in full framework as well:

System.NullReferenceException : Object reference not set to an instance of an object.
at Microsoft.Azure.WebJobs.Script.UserTypeScanner.GetPossibleExtensionAssemblies(IEnumerable1 userTypes) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\UserTypeScanner.cs : 21    at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadDirectlyReferencesExtensions(IEnumerable1 userTypes) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 718
at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize() at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 505
at Microsoft.Azure.WebJobs.Script.ScriptHost.Create(IScriptHostEnvironment environment,IScriptEventManager eventManager,ScriptHostConfiguration scriptConfig,ScriptSettingsManager settingsManager) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 937

It does and it's not clear what the problem is.

I ran into this as well. In mu case, it seems the cause is that when you delete a Function in VS and republish, the old function folder doesn't get removed, so it ends up with a function.json that points to nothing. e.g. I have:
{
  "generatedBy": "Microsoft.NET.Sdk.Functions-<IP_ADDRESS>",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "11 14 17 * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "..\\bin\\FunctionAppVS2017.dll",
  "entryPoint": "FunctionAppVS2017.MyTimer.Run"
}

But FunctionAppVS2017.MyTimer.Run no longer exists in the assembly.
We need to fix the runtime so it quietly ignores that function and doesn't fail the whole startup (in addition to fixing VS publish: https://github.com/Azure/Azure-Functions/issues/469).


Any plans to add CDC linting?

Feature Description
With a good number of opensource RTL developers being converted software engineers, a CDC validation option would greatly help improve the quality of IP in opensource domain.
Is there any thinking in that direction within Yosys dev community?

Clock domain crossing correctness can be implemented prior to passing RTLIL or Verilog to Yosys by language specific tooling, for example, see Amaranth HDL. To my knowledge it's not within scope of any current efforts to integrate related features to Yosys. If anybody is interested in building and maintaining something like that, feel free to ask to reopen this issue

Actually we need to determine clock domains for various functionality inside yosys, which is currently done in an ad-hoc manner, but I'd like to eventually build a helper for (similar to the sigmap and modwalker utilities). Once you have a utility that can tell you for each wire bit the clock domains it's driven from and used in, writing a pass that highlights CDC locations would be the easy part. I've been meaning to look at that for about a year now, there's just never the time... (cf https://github.com/YosysHQ/yosys/discussions/3956 also)

FWIW Verilator had CDC warnings but they were rarely used, and lost the maintainer, so were removed.

It's worth a ton!!!

Any chance to revive the CDC linting activity within Verilator?

Similarly to what we've recently seen for language linting, second opinion is a welcome option.

See discussion #3956 CDC detection with yosys.  I have a working tool "cdc_snitch" posted and documented.
By "working" I mean "works for me", and in fact deployed in production CI.  I've had no contributions or feedback from others, yet.  It would be great to get more people involved!

Great lead! We will be trying it on all of our active projects, and are also spreading the word for the others to kick some CDC tires with it.

Any additional insights you can share on ...and in fact deployed in production CI... ?!

Any additional insights you can share on ...and in fact deployed in production CI... ?!

In Bedrock and an additional internal project, we check the portable layer of synthesizable Verilog with cdc_snitch, and don't pass CI unless cdc_snitch reports zero BAD.  Sorry, the output of that check is not visible publicly. The gitlab CI job definitions are in (https://github.com/BerkeleyLab/Bedrock/blob/master/.gitlab/ci/cdc_check.gitlab-ci.yml).
In Bedrock, those two code bases have 3001 and 5211 (non-blank, non-comment) lines of code, with some overlap.
The additional internal project checks 15711 LoC, that's heavy on the DSP.
If you peek at our gitlab CI configuration, you can tell that we take CI pretty seriously.
This includes hardware-in-the-loop testing.
Here's a cut-and-paste of the relevant output from the last master-branch CI run, job badger_cdc:
yosys --version
Yosys 0.23 (git sha1 7ce5011c24b)
yosys -q -p "read_verilog -DBUGGY_FORLOOP hw_test.v hw_test_skin.v rtefi_blob.v /builds/hdl-libraries/bedrock/badger/rtefi_center.v /builds/hdl-libraries/bedrock/badger/scanner.v /builds/hdl-libraries/bedrock/badger/pbuf_writer.v /builds/hdl-libraries/bedrock/badger/udp_port_cam.v /builds/hdl-libraries/bedrock/badger/crc8e_guts.v /builds/hdl-libraries/bedrock/badger/construct.v construct_tx_table.v /builds/hdl-libraries/bedrock/badger/ones_chksum.v /builds/hdl-libraries/bedrock/badger/xformer.v /builds/hdl-libraries/bedrock/badger/ethernet_crc_add.v /builds/hdl-libraries/bedrock/badger/hack_icmp_cksum.v /builds/hdl-libraries/bedrock/dsp/reg_delay.v /builds/hdl-libraries/bedrock/dsp/reg_tech_cdc.v /builds/hdl-libraries/bedrock/badger/mac_subset.v /builds/hdl-libraries/bedrock/badger/precog.v /builds/hdl-libraries/bedrock/badger/test_tx_mac.v /builds/hdl-libraries/bedrock/badger/hello.v /builds/hdl-libraries/bedrock/badger/speed_test.v /builds/hdl-libraries/bedrock/badger/mem_gateway.v /builds/hdl-libraries/bedrock/badger/spi_flash.v /builds/hdl-libraries/bedrock/badger/spi_flash_engine.v mac_compat_dpram.v /builds/hdl-libraries/bedrock/homeless/activity.v /builds/hdl-libraries/bedrock/badger/base_rx_mac.v /builds/hdl-libraries/bedrock/badger/packet_categorize.v /builds/hdl-libraries/bedrock/dsp/data_xdomain.v /builds/hdl-libraries/bedrock/dsp/flag_xdomain.v lb_demo_slave.v /builds/hdl-libraries/bedrock/dsp/freq_count.v /builds/hdl-libraries/bedrock/dsp/freq_gcount.v fake_config_romx.v /builds/hdl-libraries/bedrock/dsp/dpram.v /builds/hdl-libraries/bedrock/homeless/multi_counter.v spi_gate.v; script /builds/hdl-libraries/bedrock/build-tools/cdc_snitch_proc.ys; write_json hw_test_yosys.json"
python3 /builds/hdl-libraries/bedrock/build-tools/cdc_snitch.py hw_test_yosys.json -o hw_test_cdc.txt
json from Yosys 0.23 (git sha1 7ce5011c24b)
modules: hw_test_skin
OK1: 2773  CDC: 15  OKX: 24  BAD: 0

Bedrock seems like a very interesting library project. Have you seen @obruendl 's OpenLogic library? He is also looking into "pro"-grade CI.
BTW, Bedrock Yosys is curiously ancient. Any reason for such a long leg?

Bedrock seems like a very interesting library project.

Thanks!

Have you seen @obruendl 's OpenLogic library?

Maybe?  The licenses prevent the projects from merging.  He's welcome to borrow code from us.

Bedrock Yosys is curiously ancient. Any reason for such a long leg?

The hint is in bedrock's Dockerfile:
FROM debian:bookworm-slim as testing_base_bookworm
apt-get install -y [...] yosys

If sufficiently motivated (as we were back in the bullseye days), we could build a fresh (pinned) copy of yosys.
But our code base, including tests, does work with stock yosys-0.23.  Stability is important to us.  I have been testing newer tools outside of CI, so when trixie comes out, we hope to accomplish a smooth transition.


Record types don't exist

Heya, @bensku is there any plans to add support for record types?

Also, is it possible to emit JSON? I want to use this project to create 'bindings' for another language I'm working on to interface with Java, and this would really be useful

Hm fair enough, thanks! I've actually decided to use reflection to map out Java arguments and types, parameter names are included in bytecode if you specify it in javac so it's not too much of an issue thankfully

For record, I built this to avoid loading classes (potentially with side effects in static initializers) and to convert Javadoc to JSDoc. If you don't need the latter, it is probably much easier to parse bytecode or (as you do), use reflection.


feat: graph view with cytoscape

Add option to try out cytoscape graph

So good! Just make it the default I reckon.


Need to change the websocket URL displayed in store.

Description:
We need to have a way to change or customize the web socket URLs that display on APIM-Store.

Fixed with https://github.com/wso2/carbon-apimgt/pull/5964/files


Any example how to use this feature?

I tried to setup a Page to PDF from filter to view/page but it didn't work.

Hi @hardc0l2e, you can use this plugin by triggering it as an action:

Create a view and place an "action" on it.
click on the placed button. On the right side you will see some configuration pane for it.
Click on the "Action" drop-down in the config pane and select "page_to_pdf"
Set further settings and save
You can now click on the button of your page and retrieve the pdf file


SmartSelect.single can't use modalHeaderBuilder.

Is that a bug?
When I use modalHeaderBuilder with SmartSelect.single, it will throw an error "_TypeError (type '(BuildContext, S2SingleState) => Column' is not a subtype of type '((BuildContext, S2SingleState<String?>) => Widget)?')".

null-safety migration issues
@richard1991 try this one if it solves the issue
https://pub.dev/packages/flutter_awesome_select/versions/6.1.1-beta


SmallDataIO::get_specific_data_line reads data as coords

Hey,
I noticed the SmallDataIO::get_specific_data_line looks for the line corresponding to a specific 'time' requested as an argument, but this is searched through the whole line and not just the first column, which means that if by any change the coordinate (time) happens to match some of the data on a given line (but not the actual time) then SmallDataIO will think it found the correct line.
Explicit example:

BBH example with cheap params
Punctures are at (6,8,0) and (10,8,0)
I ran and decided to restart at t=8 (not fine tuned, I actually want to restart at that point!)
The Puncture tracker was for some reason setting the punctures to (6,8,0) and (10,8,0) (the ones of t=0)
Why? Because the line of t=0:

#          time                 x_1                 y_1                 z_1                 x_2                 y_2                 z_2
   0.0000000000    6.0000000000e+00    8.0000000000e+00    0.0000000000e+00    1.0000000000e+01    8.0000000000e+00    0.0000000000e+00

Indeed has an 8.0000000000, so the SmallDataIO::get_specific_data_line stops there and assumes it found t=8, since "8" is in that line
Explicit place where the bug is in SmallDataIO.cpp:
       while (std::getline(m_file, line))
        {
            if (!(line.find(coords_string) == std::string::npos))
            {

I guess we need to replace this by looking just at the first column.

Well, not necessarily just the first column. It will depend on the length of a_coords. But yes, I agree that we should restrict to the substring corresponding to the "coords" columns only.


Set Programmed Condition in Gateway Status

Gateway API prescribes that Programmed condition is present on Gateway resources
https://github.com/kubernetes-sigs/gateway-api/blob/c67f3fd31976ea74ea06a78c2a88ae38df95a41d/apis/v1beta1/gateway_types.go#L519-L569
Note: this is required by conformance tests https://github.com/kubernetes-sigs/gateway-api/blob/6efcfdf8aff7aa7c3ed8deb956e5b93eed779234/conformance/utils/kubernetes/helpers.go#L216
Acceptance Criteria:

Implement Programmed condition
Update the docs https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/docs/gateway-api-compatibility.md#gateway

@pleshakov is this a blocker for running conformance tests?

@kate-osborn
yep. added a note.


Make Spec, Frames and AxesPoints generic

Make AxesPoints key a generic type rather than a string
Propagate genericness through Frames, Spec and all subclasses of Spec such that you can now have, for example, spec: Spec[str] or spec: Spec[Motor]
Set service code to explicitly use Spec[str]
Fix unit test debug configuration

Currently getting issues with generic union, see: https://github.com/wyfo/apischema/issues/207

@thomascobb I think this is nearly there but it may be worth a rethink. Mypy wants type annotations for specs much more often (and in more awkward places) than I had hoped.
One reason for this is that it's not good at retroactively inferring types. So if I do
empty = Static.duration(0.5) * Line("x", 0, 1, 2)

It will demand a type annotation because it cannot infer that the axes for the Static part of the expression are of type str from the fact that I passed a string to the Line.
Another problem is the fact that TypeVars need to be bound inside a class.
For example, for line:
AAxis = A[K, schema(description="An identifier for what to move")]

@dataclass
class Line(Spec[K]):
    ...

Putting K in AAxis isn't good enough, the reference to K needs to be inside Line for mypy to bind it. Otherwise you have to type such things as
# I'm passing a string so mypy should know that K=str, but it doesn't because the reference to K is outside Line
spec: Line[str] = Line("x", 0, 10, 5)

We can fix this by moving AAxis into Line but that requires some duplication and I'm not sure how often this pattern is used in the rest of the library.
What are your thoughts?

The second problem was fixed with
class Line:
    axis: AAxis[K]


Hardware keyboard auto blur issue on keyboard hide #289

This pull request fixes the issue https://github.com/ionic-team/ionic-plugin-keyboard/issues/289

We've tried it, and I can confirm this pull request addresses the issue. Thanks, @onderceylan, for the fix!

Glad to see it validated @uniphonic, hope it will be merged soon.


update LICENSE year

There is nothing wrong with this PR, but I don’t think you have a correct understanding of how to contribute to an open source project.


Find surrounding word

When cursor is placed inside a word (but word is not selected), use that word as the selection.
Example:

Type "embolden"
Place cursor between "b" and "o"
Press ctrl+b
Expected: text is replaced with "embolden"

I think that a great addition would be for block-kind of commands to wrap the whole line, so you have selection like:
Some ran^dom text

And press toggle ordered list, it makes:
1. Some ran^dom text

Resolved in v0.8.0


Embed universal viewer instead of OSD.

Closes #108.

Coverage increased (+0.005%) to 97.877% when pulling 5df0be882654c86bbf8e347d284d6479dbceefff on embed_uv_for_item_embed into 1ca58f0c1d78807772455cbeac76b033f3e8003c on master.


Could not get a resource from the pool and  java.util.NoSuchElementException: Timeout waiting for idle object

Expected behavior
can get jedis with j3ds:2.9.0
Actual behavior
use try-with-close to get jedis instance by:
try (Jedis jedis = getJedisInstance()) {
bvalue = jedis.get(bkey);
} catch (Exception e) {
logger.warn("Exception *****", id, e);
}
but get exception in log：
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
......
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
... 46 common frames omitted
Steps to reproduce:
Now can't reproduce it
Redis / Jedis Configuration
Jedis version:
2.9.0
Redis version:
4.0.8
Java version:
openjdk:17.0_45

What's getJedisInstance doing?.

@xiaoxianglee The exception is thrown by commons-pool2-2.4.2 and there can be many reasons for that. It can even be due to some rare system issues. Without any specific reason and/or reproducing the issue, it can hardly be fixed.


Add tooltips to describe fields

It would be great if it was possible to annotate protobuf fields and have those fields appear as tooltips in the generated web form. This could be used to provide instructions and act as living documentation.

field comments are now generated into the form as tooltips.
Also proto2's defaults are now filled into the form (proto3 does not have defaults).
@gpaul you might find this useful.

I have made some backwards incompatible changes for custom fields, but the form_test.go shows how to use it with the new API.  This was necessary to get the comments.


DataCollatorForCompletionOnlyLM instruction token masking fails if first occurence of instruction is marked differently

The current code in DataCollatorForCompletionOnlyLM assumes that the first deteced occurence of instruction_template comes before the first detected occurence of response_template. This is reasonable, since in current applications conversations are initiated by the user, not the assistant. However, this can fail if the first instruction is marked differently from all the other instructions, which can if a context-sensitive tokenizer such as Llama-2 tokenizes the instruction_template differently at the start of a string than in the middle.
In particular this happens in practice with TinyLlama: <|user|> gets tokenized as 529, 29989, 1792, 29989, 29958 at the start of a conversation, but as 29966, 29989, 1792, 29989, 29958 in later messages.
Reproduction snippet:
from transformers import AutoTokenizer
from trl import DataCollatorForCompletionOnlyLM

tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

chat = [
    {"role": "user", "content": "Which is bigger, the moon or the sun?"},
    {"role": "assistant", "content": "The sun."},
    {"role": "user", "content": "Really?"},
    {"role": "assistant", "content": "Yes."},
    {"role": "user", "content": "I don't believe you."},
    {"role": "assistant", "content": "That's okay."},
]

collator = DataCollatorForCompletionOnlyLM(response_template=[29966, 29989, 465, 22137, 29989, 29958, 13], instruction_template=[29871, 13, 29966, 29989, 1792, 29989, 29958], tokenizer=tokenizer)

collator([tokenizer.apply_chat_template(chat)])
# This doesn't mask user messages 2 & 3.

PR #1185 fixes this, and makes the above snippet mask out all the user messages correctly.

@younesbelkada While we're talking about this anyway: Would it potentially make sense to add a more flexible system as an alternative to matching strings here? I have in mind something like passing a function to the data collator that maps example to a list of (first_token_idx, last_token_idx)-pairs that mark each part of the example that is assistant-generated, i.e. should be unmasked.
I'm asking for two reasons:

There's still many edge cases remaining, given the wide variety of chat templates. (Case in point: Even the code snipped above is still not 100% correct - the chat template inserts a newline after the final EOS token, which IMHO should be masked if the final turn is from the assistant, but isn't, and I don't see a way of doing that with the template system.)
Doing that would allow re-use of that function in other parts of a project, e.g. for reward or RL training, unit-testing, etc.

I will probably implement something like this for a project I'm working on - I'd be happy to open a PR once it's done, if you think that would be useful to have in the library.
(Asking this here, as it's broadly related to the issue, but not to the fix in the PR.)


Kubernetes deployment

Is some content missing, wrong or not clear?
Based on https://docs.open-metadata.org/deployment/kubernetes
Following the mentioned documentation does not provide plug&play experience.
Here are the points that need clarification:

By default all services are not accessible from the other pods. mysql, elasticsearch, openmetadata-dependencies-web, openmetadata itself. Only MySQL has networkPolicy template file and hence can be enabled by setting mysql.networkPolicy.enabled: true in ./chart/deps/values.yaml. All the other require specifying networkPolicy explicitly.
Default password for Airflow and MySQL are configurable, but not for OpenMetadata itself
OpenMetadata uses single host name for both addressing ingestion airflow internally and generating url for ingestion DAGs. It makes the ingestion DAGs' links in OpenMetadata UI useless since they are not accessible out of cluster.

Describe the solution you'd like
I see possible fix as follows:

Add networkPolicy templates  and add corresponding switchers in values.yaml
Add parameters into values.yaml
add guidance on possible alternatives

@akash-jain-10 lets follow-up on this one and close it

Hello @kananmammadli-vonage . Thanks for bringing this up. Will take a look at adding network policy to OpenMetadata Helm Charts as part of Point 1. Could you please create an issue ticket here ? Thanks.
Will make sure to add a documentation for configuring/ updating default password via Bare Metal, Docker, K8s deployments with Basic Auth.
For Point 3, will take a look and prioritize getting this seperated. Current alternatives is to setup / map airflow webserver over  dns name and use that with openmetadata configuration.

Closing this issue and concluding things here -

We now have Network Policies available for OpenMetadata Helm Chart. https://github.com/open-metadata/openmetadata-helm-charts/pull/126
We have added documentation for usage of custom passwords with initial admins with Basic Auth SSO in docs here
Airflow UI should be available over a DNS which can be used in configuring the Airflow Endpoint URL for OpenMetadata. This fixes the Issue for URL links which will not be broken anymore.


feat: Add the roses style text

Roses
Add a separate Roses component to let users use the following styled text, with a layered text shadow effect.
Courtesy : https://codepen.io/TajShireen/pen/abzmoRE

Feature merged and released


LibWeb: Return used values for grid tracks in getComputedStyle()

That's awkward, but getComputedStyle needs to return used track values for gridTemplateColumns and gridTemplateRows properties. This change implements it by saving style values with used values into layout state, so it could be assigned to paintables during LayoutState::commit() and later accessed by style_value_for_property().
I haven't seen it used in the wild, but WPT grid tests extensively use it. For example this change helps to go from 0/10 to 8/10 on this test: https://wpt.live/css/css-grid/layout-algorithm/grid-fit-content-percentage.html

(Follow-up: #1454)


not enough values to unpack (expected 3, got 2) for test script

i_auroc, p_auroc, pro_auroc = SimpleNet.test(dataloaders["training"], dataloaders["testing"], save_segmentation_images)
ValueError: not enough values to unpack (expected 3, got 2)
while in test:
only return two values
return auroc, full_pixel_auroc

我也遇到了同样的问题

Getting same Error. How to solved it ?

The evaluation process is included in "train" method.


Ability to add trackers manually should be restricted to multi-sig admin group

Currently, trackers can be manually added by O/S operators and the admin group. This should be restricted to the admin group only because an attacker must compromise multiple signers to submit a false tracker. Under the current model a single operator being compromised could result in submission of false or malicious trackers.
zetacored tx crosschain add-to-out-tx-tracker

Closing this in Favor of
https://github.com/zeta-chain/node/issues/1318


Message from a fellow developer

Greetings!
I also use Qlik Sense and have for many years,
and if you need perpetual licenses for any Qlik products and for any number of users,
please contact my Telegram: my Telegram
And if you are not interested, then I apologize for the intrusion into your repository ;-)

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."


Only conditionally call H5Pset_evict_on_close

Closes #259

Looks like MSRV tests fail because the most recent half (required by feature="f16") bumped their MSRV to 1.70 🤔 Wonder if we should do the same then

Alternatively make MSRV only work for the minimal set of features, but I'd rather bump it up to 1.70 even with 1.70 being quite recent (1st of June)


Conntrack table GC cause regenerating endpoint failed

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Hi,
Recently we found a problem: when creating deployments, sometimes pods stuck at ContainerCreating state for a long time like 1 hour, and cilium agent keeps print logs like JoinEP: Failed to load program and Failed to replace Qdisc for lxc... link not found. Metrics show that there are many endpoint creation requests are in-flight, so I think the failed requests are continuously retied.
Besides, We observe that CEP CR is successefully created, that node's kernel log shows that lxc device has ever been created. In the endpoint creation process, the waitingForCTCleaning stat indicates it takes 1-2 minutes in cilium agent logs.
kubectl describe pod shows several Failed to create pod sandbox... Unable to create endpoint: Cilium API client timeout exceeded events. I guess that kubelet deletes and recreates the pod sandbox when it waits for CNI response and timeouts. The deletion request removes the lxc before GC is done and eBPF binary loaded.
Is the presumption reasonable? Would it be helpful to reduce ct GC time if we lower the bpf-ct-timeout or conntrack-gc-interval? Thanks.
Cilium Version
1.11.15, but may affect the recent versions
Kernel Version
5.10
Kubernetes Version
1.24
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

v1.11.15 is quite an old version -- can you try two things

reproducing this on the latest v1.11 version
capturing a sysdump on a node experiencing the issue

Thanks!

@squeed Hi, thank you for your reply, sorry for that maybe I express in a misleading way. Actually I want to ask for an answer for my presumption rather than ask for a bug fix. It is very difficult for me to put up the sysdump, because there are strict constraints to export data from our product env.  I think this issue can be closed.

Well, it seems like you're hitting some kind of performance issue, which would be good to clean up, especially if you're able to reproduce this on a more recent version.
If you don't mind, could you capture a CPU profile and heap dump (instructions here) on a node experiencing this issue? That would definitely help tackle the problem

For anyone who might be interested, set the conntrack-gc-interval to non-zero seems to fix our problem. Besides, I lower the bpf-ct-timeout and increase the endpoint-queue-size. If conntrack-gc-interval is zero, the conntrack gc interval is calculated dynamically, which has some problem mentioned in other issue.


oni.statusbar.enabled

It seems there are two settings that control this, oni.statusbar.enabled and statusbar.enabled.
Setting the former to false doesn't do anything on its own. I have to set statusbar.enabled to false instead. Moreover, if statusbar is enabled but sidebar is not, the statusbar shows incomplete infos.
Just reporting this because it seems a bug, I don't use neither sidebar nor statusbar (please keep this possibility...).

Thanks for catching this @mg979 ! Looks like it was an error on our features page:  https://github.com/onivim/oni/wiki/Features#status-bar
Just corrected it.

I think there is some general problem with sidebar.enabled. It seems it affects more than just sidebar, because if it's false, quickinfo doesn't work anymore, and also the statusbar infos are incomplete.

This is the config I'm using (the rest is defalult)
module.exports = {
    activate,
    deactivate,
    //add custom config here, such as

    //////////////////
    // default

    "ui.colorscheme": "nord",
    "oni.useDefaultConfig": true,
    "editor.fontFamily": "Monaco",
    "editor.clipboard.enabled": false,
    "ui.animations.enabled": true,
    "ui.fontSmoothing": "auto",

    //////////////////
    // working

    "statusbar.enabled": false,
    "oni.hideMenu": true,
    "tabs.mode": "native",
    "commandline.mode": false, // Do not override commandline UI
    "wildmenu.mode": false, // Do not override wildmenu UI

    ////////////////////////////////////
    // quickinfo not working

    //"oni.useDefaultConfig": false,
    //"oni.loadInitVim": true,
    //"sidebar.enabled": false,
}

If I set
"oni.useDefaultConfig": false

this loads nvim init.vim, so not sure what
oni.loadInitVim": true

would add to this.
In any case, loading my init.vim alone breaks quickinfo.
Also setting this alone breaks quickinfo:
"sidebar.enabled": false

What version of Oni are you running @mg979 ?
We have #1548 and #1562 relating to the issues you are seeing, where the first is fixed now in master. I think the second has an on-going fix as well.
As for the other bits, some users will want to have both the Oni default config + there own init.vim which is why there is two config options. Also, oni.loadInitVim accepts a path to an init.vim such that you can have an Oni specific config rather than using the system default.

Thanks. I'm running Oni 0.3.0 (latest release).


Issue on minimum sequence length

I'm currently using "find SNPs by sequence." It appears that the minimum sequence length is set to 20. Would it be possible to adjust it to a shorter length, such as 16? Thank you!

Hello,
Find SNPs by sequence need to blast your sequence to a reference in order to know the localisation of your sequence. I choose to use the library gget to do it since it call a external service and remove the need of having the reference to perform the blast.
There are some major drawbacks with this approach:

The external service need to be up and an internet connection is needed.
We are limited to only human genome.
We can't use a sequence shorter than 20 bases.

A solution to perform the blast is  to be considered but as drawback you will need an access to the genome reference as it is currently the case to perform the PCR in silico.
Maybe the best is to have the two solution and let the user choose the best suited for his case.
I will work on a solution this week, or feel free to propose a PR :)

I appreciate the information you've shared, and I understand it well.
I've observed that the genomic locations are obtained from the UCSC Genome Browser API using the gget library. However, this approach is limited to the Human genome, with a minimum sequence length requirement of 20bp. So far, I haven't come across any alternative APIs that can offer similar information for shorter sequences. If you can work out a solution, that would be fantastic. Using a local version of a reference genome along with software like blast/blat shouldn't be a problem for us.
Thank you so much!

From: Benoitdw @.>
Sent: Friday, April 12, 2024 3:10 PM
To: Benoitdw/SNPrimer @.>
Cc: JiguiS @.>; Author @.>
Subject: Re: [Benoitdw/SNPrimer] Issue on minimum sequence length (Issue #3)
Hello,
Find SNPs by sequence need to blast your sequence to a reference in order to know the localisation of your sequence. I choose to use the library gget to do it since it call a external service and remove the need of having the reference to perform the blast.
There are some major drawbacks with this approach:

The external service need to be up and an internet connection is needed.
We are limited to only human genome.
We can't use a sequence shorter than 20 bases.

A solution to perform the blast is to be considered but as drawback you will need an access to the genome reference as it is currently the case to perform the PCR in silico.
Maybe the best is to have the two solution and let the user choose the best suited for his case.
I will work on a solution this week, or feel free to propose a PR :)
—
Reply to this email directly, view it on GitHubhttps://github.com/Benoitdw/SNPrimer/issues/3#issuecomment-2051950605, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BHNRGBN7NMTMNFRSADHKNULY472M7AVCNFSM6AAAAABGCRIFN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRHE2TANRQGU.
You are receiving this because you authored the thread.Message ID: @.***>


Do not change vendor.
What did you see instead?
Vendor changed.

Can you elaborate on what change happened, after what command, and why it's unexpected?

After go mod vendor, there are lots of changes inVendor.
Since i only want move from dep to module, all things about dep like Vendor should stay as they were.

go mod vendor is more effective than dep at removing unneded files from the vendor folder. However, if the go mod init step went well, there should be no other meaningful change to the vendor folder.
If there is and you believe it is a bug, we will need instructions to reproduce the issues, and details on what changes.

I reproduce it:
$ git clone https://github.com/cortexproject/cortex.git
$ cd corex
$ GO111MODULE=on go mod init
go: creating new go.mod: module github.com/cortexproject/cortex
go: copying requirements from Gopkg.lock
$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	go.mod
$ GO111MODULE=on go mod vendor
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   vendor/cloud.google.com/go/bigtable/admin.go
#	modified:   vendor/cloud.google.com/go/bigtable/bigtable.go
#	modified:   vendor/cloud.google.com/go/bigtable/bttest/inmem.go
#	modified:   vendor/cloud.google.com/go/bigtable/filter.go
......
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	go.mod
#	go.sum
#	vendor/cloud.google.com/go/bigtable/go18.go

There is two question:

go mod init do not generate go.sum
go mod vendor change lots of things in Vendor

@FiloSottile hi, any input?

go mod init does not promise to exactly reproduce the configuration of dep. It roughs in a starting point, from which the result is computed using minimal version selection and the transitive go.mod files of your dependencies.
To my knowledge, dep makes no effort to ingest or interpret the dependencies declared in go.mod files, so if any dependency has such a file, module mode can produce a substantially different result from dep. So merely being different from dep is not an error.
If you believe that the module-mode result is erroneous, please provide specific detail for which module's (or modules') requirements are not being followed. (#29773 is the only such issue that I know of at the moment.)

@bcmills hi, my key issue is that why go mod modify things under vendor, go mod do not need them any more, right?

Hi, i do below test in clean env:
$ docker run golang:1.12.1 /bin/bash
# go get github.com/cortexproject/cortex
# export GO111MODULE=on
# go mod init
# go mod vendor
# go mod tidy

Then i can see lots of changes in vendor.
@bcmills i test 3 times, reproduce every time.

@bcmills hi, any update?

Your steps to reproduce explicitly invoke go mod vendor. The whole point of go mod vendor is to update the vendor directory.
As noted in https://github.com/golang/go/issues/31223#issuecomment-480818671: if you believe that the module-mode result is erroneous, please provide specific detail for which module's (or modules') requirements are not being followed.

Get it, go mod vendor not only get things from vendor and also update it.
But it may be confused for users who want to move dep to go mod since after move the vendor would be useless for them, so update vendor could be meaningless for them.
Add an option to avoid changes in vendor could be better i think.

go mod vendor does not “get things from vendor” at all: its only purpose is to update the vendor directory, and it currently begins that process by deleting it entirely (without reading from it) before repopulating it.
If you want to avoid changes in vendor, then simply don't run go mod vendor. (But see also #30240.)


[v2] Timeline problems

I have a strange problem with timeline version2 using md thmes.
The year and month text stay appear / disappear.
I have tested in emulator and my iphone / ipad the same problem.
https://imgur.com/a/60lP0

Only to complement, It happens in the iOS themes too.

See this issue too, looks like some iOS-specific rendering issue

Yes, I think so.

@nolimits4web,
I have tested and the [v1] don't have this issue.

I get this is 1.6.5

Issue is closed because of outdated/irrelevant/not actual/needed more information/inactivity.
If this issue is still actual and reproducible for latest version of Framework7, please create new issue and fill the issue template correctly:

Clearly describe the issue including steps to reproduce when it is a bug.
Make sure you fill in the earliest version that you know has the issue.
Provide live link or JSFiddle/Codepen or website with issue


crash

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
1 launch
Expected behavior
it launch
Screenshots
If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):

OS: Windows
Modpack Version: [1.9.18+forge.1.20.1
Additional context
Add any other context about the problem here.

Remove Canary. There's already Radium installed. Radium and Canary do the same thing but Radium > Canary.


Message original sender

When message is being sent by a provider, the original message sender msa should get stored.

Hey @enddynayn I was just messing around with this and wondering the same thing. If alice is a provider of like 4 msa_id's it just seems to use the most recently delegated id. How would it decide though? Alice is signing the Messages.add transactions in order to pay the fees, so I don't see where the delegator is involved. Should the delegator by signing something?

Let's say that Alice is a provider to Bob. In other words, Bob is a delegator and Alice is the delegate.
If Alice wants to sends a message on behalf of Bob the second parameter of Messages.add, on_behalf, is populated with Bobs MsaId. We can verify that the message is valid, offchain, by checking that Alice was indeed a provider to Bob.

@enddynayn I only see schemaId and message as parameters for Messages.add
I'm confused as to what you mean by the second parameter on_behalf

@enddynayn  you mean if on_behalf is provided, do we need to store if message was sent on behalf of "this msa _id"? to be clear


This should be moved to the parent ESAPI project

Hi Karl - this is awesome! I haven't seen some of this code in ages! :) I would love to move this project over to the official ESAPI Github and help foster some community involvement in the project!
https://github.com/ESAPI
If you would like to get this moving let me know and we'll get it moved over! Great work by the way!

Thank you.
Sure, we can move it so it is under the same umbrella that the others are.

Can you transfer the repository to me and I will move it to the ESAPI org, then add you to it

On May 4, 2015, at 5:17 AM, Karl Düüna<EMAIL_ADDRESS>wrote:
Thank you.
Sure, we can move it so it is under the same umbrella that the others are.
—
Reply to this email directly or view it on GitHub https://github.com/DeadAlready/node-esapi/issues/1#issuecomment-98679901.

Can you transfer the repository to me and I will move it to the ESAPI org, then add you to the repo

On May 4, 2015, at 5:17 AM, Karl Düüna<EMAIL_ADDRESS>wrote:
Thank you.
Sure, we can move it so it is under the same umbrella that the others are.
—
Reply to this email directly or view it on GitHub https://github.com/DeadAlready/node-esapi/issues/1#issuecomment-98679901.

It looks that this repo was actually transferred to parent ESAPI account, finally. This issue should be closed, I think.


Duck typing conversions

A lot of the design in the Framework and StarML is oriented around Pintail and its requirements, but it recently occurred to me that after data binding has occurred, Pintail is entirely irrelevant; that is, view bindings have direct access to the bound context and so Pintail wouldn't help with "proxying" at that point even if the types were interfaces.
So really, it's up to StardewUI to decide whether or not to support this fuzzy type conversion. There's a case to be made that it should. For example, suppose the client defines:
public record Sprite(Texture2D Texture, Rectangle SourceRect, Vector4 FixedEdges);

or even
public class CropData
{
    public Texture2D Texture { get; set; }
    public Rectangle SourceRect { get; set; }
    // other properties
}

These aren't assignment-compatible with Sprite, but their properties are assignment-compatible with all the required properties of Sprite. Having some form of duck typing is really not a far cry from what the Framework is already doing with property assignments and conversions.
A possible specification would go as follows:

Disallow conversions to any primitive destination type (int, float, etc.) since that can never succeed and we don't want to waste the cycles checking. struct types are fine, just not primitives.
For each constructor on the target type, match arguments with property names of the source type. Either case-insensitive comparison or camelCase to PascalCase conversion.
Choose the constructor with the highest count of matched arguments. (We intentionally do this before validation, because I don't like the idea of ambiguous and implicit fallback behavior; either the constructor that one would expect to match does match, or it's an error.)
Validate that each source property is conversion-compatible, either having the exact type required or having a source->dest converter available. This includes optional arguments; it is OK to simply lack a property corresponding to an optional arg, but not OK to have an incompatible property.
For any additional properties on the target type that are (a) public, (b) writable and (c) not already specified in a ctor argument with the same name, do the same mapping and validation from source to destination in the previous step.
Generate and cache the converter one time like we do for binding types everywhere else.

Additionally, if the only matching constructor (or the only constructor at all) is a default parameterless constructor, and there aren't any properties on the source type matching properties on the destination type, then either don't allow the conversion, or log a prominent warning (one time) that the conversion is meaningless and therefore probably wrong.
This might need to handle fields on the source type, too. While I'm not crazy about it due to the performance issues, I expect it's just going to be a thing that comes up. (See rationale in #15).
This specification heavily favors record types, in which ctor args are always the same as property names, and that's intended. It can still be made to work with other classes, though, by using case-insensitive comparisons or case transformations as mentioned earlier.
Anticipated challenges include:

Recursive duck typing could lead to infinite recursion if there are any cycles in the graph (doesn't have to be an object cycle, just a cycle in the types themselves). We can either prevent recursion entirely by making the duck-type converter factory a decorator around the regular one - meaning, only the top-level conversion can be duck-typed while all properties/args/etc. cannot - or, if that's too restrictive, maintaining a "path" during the validation phase and bail out (error) if we reach a type that's already in the process of validation.
Collections are hard, as always. This is a preexisting issue with data bindings as well. Can maybe be deferred since I don't think any really important property types involve collections, except for drop-downs (#9). If we want to solve it, then can start by solving it for covariant types (IEnumerable, IReadOnlyList) and worry about adapters for invariant types later.
Error logs currently use the short type names, but this would make it even more important to use full type names; we don't want messages saying "Sprite cannot be converted to Sprite".

Side note: while the practice should be heavily discouraged, this also solves the problem with devs importing the Core Library as a Shared Project and trying to use those types; since they are in fact the same logical types, they should always be duck-type compatible.

Committed a working implementation for this. Don't yet know how well it works, but the existing types give a decent amount of stress testing already with default parameters, nullable value types, etc., so it should cover a wide range of cases.
Probably needs a quick doc update on the type conversions so I won't close this until that's added.

Was getting ready to close this, and then read over the notes again and realized that I completely forgot about collections, so this could drag on for quite a while longer, unfortunately.
Still, it's in a pretty good state as most (maybe all?) of the types that would actually benefit from this conversion don't use collections anyway: Sprite, Edges, Bounds, etc.


Update remote rules nightly

What would it take to run a nightly job to update the remote rule sets?
Adding sources to the remote rules makes the unit_test fail.
/opt/binaryalert/rules/clone_rules.py
REMOTE_RULE_SOURCES = {
    'https://github.com/Neo23x0/signature-base.git': ['yara'],
    'https://github.com/YARA-Rules/rules.git': ['CVE_Rules'],
    'https://github.com/SupportIntelligence/Icewater.git': ['']
}

$ ./manage.py unit_test
.........................................................................F
======================================================================
FAIL: test_update_rules (tests.rules.update_rules_test.UpdateRulesTest)
Verify which rules files were saved and deleted.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "/opt/binaryalert/tests/rules/update_rules_test.py", line 52, in test_update_rules
    self.assertEqual(expected_files, set(compile_rules._find_yara_files()))
AssertionError: Items in the second set but not the first:
'github.com/SupportIntelligence/Icewater.git/CVE_Rules/cloned.yara'

----------------------------------------------------------------------
Ran 74 tests in 18.957s

FAILED (failures=1)
TEST FAILED: Unit tests failed

Need a way to make sure all of the python libraries are available for the rules.
/opt/binaryalert/rules/clone_rules.py
REMOTE_RULE_SOURCES = {
    'https://github.com/Neo23x0/signature-base.git': ['yara'],
    'https://github.com/YARA-Rules/rules.git': [''],
    'https://github.com/SupportIntelligence/Icewater.git': ['']
}

$ ./manage.py compile_rules
Traceback (most recent call last):
  File "./manage.py", line 495, in <module>
    main()
  File "./manage.py", line 491, in main
    manager.run(args.command)
  File "./manage.py", line 352, in run
    getattr(self, command)()  # Command validation already happened in the ArgumentParser.
  File "./manage.py", line 421, in compile_rules
    compile_rules.compile_rules(COMPILED_RULES_FILENAME)
  File "/opt/binaryalert/rules/compile_rules.py", line 36, in compile_rules
    externals={'extension': '', 'filename': '', 'filepath': '', 'filetype': ''})
yara.SyntaxError: ./Mobile_Malware/Android_FakeApps.yar(101): invalid field name "app_name"

Would it be better to remove the rules or install the missing python libraries?
/opt/binaryalert/rules/compile_rules.py
    for line in yara_filepaths:
        try:
            test = yara.compile(RULES_DIR+'/'+line)
        except:
            os.remove(RULES_DIR+'/'+line)

    yara_filepaths = {relative_path: os.path.join(RULES_DIR, relative_path)
                      for relative_path in _find_yara_files()}

Compile requires enough memory to complete.   These rules required a t2.small to build.
$ ./manage.py compile_rules
Traceback (most recent call last):
  File "./manage.py", line 495, in <module>
    main()
  File "./manage.py", line 491, in main
    manager.run(args.command)
  File "./manage.py", line 352, in run
    getattr(self, command)()  # Command validation already happened in the ArgumentParser.
  File "./manage.py", line 421, in compile_rules
    compile_rules.compile_rules(COMPILED_RULES_FILENAME)
  File "/opt/binaryalert/rules/compile_rules.py", line 45, in compile_rules
    externals={'extension': '', 'filename': '', 'filepath': '', 'filetype': ''})
MemoryError

Only a certain number of rules can apply before receiving this error.
$ ./manage.py apply
Traceback (most recent call last):
  File "./manage.py", line 495, in <module>
    main()
  File "./manage.py", line 491, in main
    manager.run(args.command)
  File "./manage.py", line 352, in run
    getattr(self, command)()  # Command validation already happened in the ArgumentParser.
  File "./manage.py", line 382, in apply
    subprocess.check_call(['terraform', 'apply', '-auto-approve=false'])
  File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['terraform', 'apply', '-auto-approve=false']' returned non-zero exit status 1

Thanks for all the feedback! We've been wanting to improve the entire rule sourcing process for awhile now, and I'm excited to get some feedback in that regard. Before going into some specific issues you've brought up, let me ask this:
Would it be helpful if rules were stored separately? For example, if YARA rules were just stored in an S3 bucket, then they could be updated without re-deploying BinaryAlert. This adds some latency to the analyzers (which have to download/decompress the compiled rules file once per Lambda container), but then you would not need a terraform apply to update YARA rules. This is something that has come up before, and we'd love any discussion about this approach.
Another benefit of storing YARA rules in S3 is that a Lambda function could run on a regular interval (e.g. daily) to automatically update the rules, obviating the need for an EC2 instance entirely (serverless FTW!) The downside is that this automatically bundles untrusted rules files from the Internet. Malicious YARA rules could, for example, cause buffer overflows in YARA to gain control over the analyzer Lambda execution, which has access to the S3 bucket with your files. Thoughts?
Now, let's go into some specific issues you've encountered:

Changing rule sources should not break unit tests. This can be easily fixed, which I'm happy to do.
The androguard library is not included, which breaks some mobile YARA rules. This is a YARA library, not a Python library, and unfortunately, androguard has a somewhat tricky install process which involves modifying YARA source files and including the cuckoo library. We can add this to the backlog, but this leads to:
Rule source declaration should be more expressive. In particular, there should be a way to ignore specific rule files or even specific rules. Right now, you can only specify a top-level repo and a set of subfolders to clone. Since only .yar and .yara files are bundled by BinaryAlert, you can manually rename rules files something like .yara.DISABLED to ignore them, but it would be great if they were just removed during the clone process.
Rule compilation takes a fair amount of memory. I'm not sure of an easy way around this one. Pre-compiling the rules saves the analyzers a significant amount of time (and therefore $$), and so far we've been able to compile even tens of thousands of rules with only a few GB of RAM. I'll be sure to update the documentation to list the memory requirement (and minimal EC2 machine size).

Next week, I'll open individual issues to address the points above, as well as potentially separating the rules.

Closing this as part of the more expressive YARA rule cloning from #98 and #99, but feel free to open another issue about any specific follow-ups!
For example, if loading YARA rules from S3 is something that you would want, go ahead and open an issue for it. We've found the local rules in the repo to be the most effective so far


Future updates in scikit-learn could brake behaviour of energy reconstruction

In the framework of use of protopipe (or any other pipeline which makes use of basic ctapipe's functionalities to estimate the energy of showers), a call to ctapipe's energy_regressor.py is made.
During such operation, the 'joblib' method is imported explicitly from scikit-learn (line #300 in reco/regresson_classifier_base.py - at least).
I recently got a warning explaining that this method will be removed in future versions of the library- possibly because it will become (or is already) a separate package.
With this issue I then wish to report that if ctapipe's environment gets updated, it might stop working.
Running from the GRID, I got many empty files because of this: apparently there the sciki-learn package has been already updated.
I then suggest to update the outdated line (and any other one which does the same) to import joblib (of course, you would also need to add the joblib package to the list of required packages in setup.py).

@HealthyPear could you write a minimal example that raises this warning?

Dear Maximilian,
I am afraid "minimal" is a big approximation :(
For the moment I try to give you at least the Traceback of the issue that I got from DIRAC.
During the generation of DL1 data for classification in protopipe, the estimated energy is used.
When doing this, from write_dl1.py the following piece of code is selected,
from ctapipe.reco.energy_regressor import * (this is bad, I know...)
regressor = EnergyRegressor.load(reg_file, cam_id_list=args.cam_ids)
where args.cam_ids is something like "LSTcam" and reg_file is the file generated by the script build_model.py (please, refer to the - now public - relevant protopipe's documentation / code).
From here the rest of the Traceback is fully-ctapipe dependent (mind that here the relevant ctapipe is the one on the DIRAC environment, which has been evidently updated),
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/site-packages/ctapipe/reco/energy_regressor.py", line 156, in load
    return super().load(path, cam_id_list, unit)
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/site-packages/ctapipe/reco/regressor_classifier_base.py", line 311, in load
    self.model_dict[key] = joblib.load(path.format(cam_id=key))
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 598, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 526, in _unpickle
    obj = unpickler.load()
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/pickle.py", line 1050, in load
    dispatch[key[0]](self)
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/pickle.py", line 1338, in load_global
    klass = self.find_class(module, name)
  File "/cvmfs/cta.in2p3.fr/software/miniconda/envs/ctapipe_v0.6.2/lib/python3.6/pickle.py", line 1388, in find_class
    __import__(module, level=0)
ModuleNotFoundError: No module named 'joblib'

Well no big deal, just import joblib instead of from sklearn.externals import joblib

It's kind of a big deal because scikit learn came with it's own version of joblib so a fixed version of joblib was guaranteed for a fixed version of scikit learn. If this is no longer the case, we need to fix a version of joblib to ctapipe or a processing so we can load the models back.

Dear Maximilian,

I am afraid "minimal" is a big approximation :(
For the moment I try to give you at least the Traceback of the issue that I got from DIRAC.

I was asking about the changes in sklearn.
So am I right that the minimal example would be

from sklearn.externals import joblib

?

I see.
This is weird though because I can still from sklearn.externals import joblib with the latest (0.20.3) version of scikit-learn... it's more an installation issue I think?

@MaxNoe
Concerning the version issue, if joblib is a dependency of sklearn, the right version will be installed when scikit-learn is installed, no?
And we don't need to add joblib to ctapipe's dependency list.

For ref:
https://github.com/scikit-learn/scikit-learn/pull/13531

Concerning the version issue, if joblib is a dependency of sklearn, the right version will be installed when scikit-learn is installed, no?

The problem ja, sklearn is basically compatible  with any joblib version but a model has to be loaded with the same joblib version that was used to dump it.
So the dependency is not fixed for sklearn, but it might make sense to fix it for ctapipe to make sure a model that was trained with a certain version of ctapipe is loadable with that ctapipe version.
At least for a specific analysis the joblib version must be fixed.

Also from the joblib docs:

As of Python 3.8 and numpy 1.16, pickle protocol 5 introduced in PEP 574 supports efficient serialization and de-serialization for large data buffers natively using the standard library:

pickle.dump(large_object, fileobj, p

This implies another issue as well: we should use a standard (and non-version-specific) way of storing models.  I recall that @mackaiver had used a library that writes them as some industry-standard XML or something.
In any case, we can try to specifically fix the version of joblib and sklearn in both setup.py and in the conda package to avoid this problem.

This implies another issue as well: we should use a standard (and non-version-specific) way of storing models. I recall that @mackaiver had used a library that writes them as some industry-standard XML or something.

Yes, we used pmml for storing models to be interoperable with java:
https://de.wikipedia.org/wiki/Predictive_Model_Markup_Language
This is however considered obsolete by the people who developed it but for the successor, there is basically no documentation and no scikit-learn plugin.
Both projects seem rather dead at this point, which is a shame.

As documentation, here are a few links (from discussion with Max):

https://www.andrey-melentyev.com/model-interoperability.html#designated-model-interchange-formats
https://github.com/onnx/onnx
https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language
http://dmg.org/pmml/v4-3/GeneralStructure.html
http://github.com/fact-project/aict-tools

It's too bad PMML/PFA seem to be dead - those looked nice.  I guess ONNX is the way to go, though it seems more complex as it's not just a format, but also a runtime for predicting... It would be nicest to just store in ONNX format and then read back and use sklearn (leaving open the option to also use their runtime in the future)

Without any easy dump format/solution, wouldn't it be easier to just fix a joblib version in ctapipe?

I just checked and I don't see why we are having this problem. The latest scikit-learn package automatically depends on joblib, so it's hard to have it installed without joblib there already:
conda update scikit-learn                 (cta-dev) 
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /Users/kkosack/anaconda3/envs/cta-dev

  added / updated specs:
    - scikit-learn
...
The following NEW packages will be INSTALLED:

  joblib             conda-forge/noarch::joblib-0.13.2-py_0

What I will do right now is create a new package for ctapipe-0.6.2 that fixes the version of scikit-learn to 0.21.3 (which seems to be the latest), and thus also installs a compatible joblib.  Once that's done, on the grid we can update the environment and it should all work.

Or we can stick with 0.20.3, which doesn't need an external joblib package (either way is fine as long as we are consistent)

It's too bad PMML/PFA seem to be dead - those looked nice

The rumors about PMML's death are greatly exaggerated. There's a minor PMML 4.4 schema version update coming up shortly, with a major PMML 5.0 schema version upgrade already in the works (the development happens on a legacy SourceForge and/or http://mantis.dmg.org environments, not GitHub).
The liveliness of a standard does not depend on its regulatory body (DMG.org is a closed group, and based on my recent experience, they are not accepting new members), but on the people using it to solve their everyday problems. The PMML standard is doing very nicely in this regard.

I just checked and I don't see why we are having this problem.

We are not having this problem I suppose.
The way to go forward is:
Require joblib or a (as soon as released) a scikit-learn version that depends explicitly on joblib
from joblib import load, dump
instead of from sklearn.externals import joblib
Problems can only arise when doing the first with an old scikitlearn version or the second with a newer scikit learn version. The first case would be remedied by explicitly requiring joblib.
(which we should do, we import it directly)

@kosack , indeed, there is no real problem from the point of view of joblib version, or joblib being in or outside sklearn. The question raise by Michele came from the fact that he is building his models with ctapipe v0.6.1 (because he cannot install v0.6.2 on his Mac because of the missing eventio package) and then needs to run his models on the grid with v0.6.2 that has a different version of joblib, hence the crash.
As you have opened #1091 , and as there is no real problem, I think this issue can be closed and probably we can try to fix Michele's specific problem on the redmine issue: https://forge.in2p3.fr/issues/37680

@bregeon Yesterday Karl uploaded the version 0.11 of eventio on Anaconda Cloud, hence I was able to install the stable version of ctapipe 0.6.2 from there (now I should have the same version as on DIRAC).
A problem in any case remains, as far as I understand: either we freeze the version of scikit-learn or we have to have find a way to keep up with the changes.
This is because 1) of the joblib dependence, since newer versions of sklearn drop it 2) because even if maybe now the difference is small betwee sklearn 0.20 and 0.21, perhpas in the future the difference will be bigger and models pickled from an older version could stop being imported or (even worst) give wrong results.

This is fixed, master used joblib directly