Implement support for --compression in lxc export

Required information

Distribution: ubuntu
Distribution version: 19.10
The output of

lxc-start --version
3.0.3

Issue description
Please add --compression off switch to the lxc export command
I export large images from my laptop to another computer on my local network.
gzip puts a lot of load on the laptop CPU. I would prefer to have an option to switch off the compression since moving large images in a local network is not a problem.

This should be implemented the same way things are done for lxc publish.
Note that a current workaround is to set backups.compression_algorithm as a global setting, if you set it to none then it won't compress.

thanks @stgraber I'll try to use workaround until the solution is implemented

hey @stgraber would something like this work? https://github.com/lxc/lxd/compare/master...srados:no-compression-backup
it's not finished but I wanted to see if this is the way to go.

oh, or shall I copy the lxc publish approach to maintain uniformity?

I'd prefer it work the same way we've done for lxc publish.

Assigning to you since you seem to have most of the pieces figured out already :)

As far as how we usually split commits for this type of work, it'd look like:

api: Add backup_compression_algorithm API extension

Adds that string to shared/version/api.go
Adds a description in doc/api-extensions.md

shared/api: Add CompressionAlgorithm to InstanceBackupsPost

Add the new field to InstanceBackupsPost in shared/api/instance_backup.go
The field should be added at the end of the struct, after a API extension: backup_compression_algorithm comment which is itself preceded by a line break (see image.go for examples).

lxd/backups: Add support for CompressionAlgorithm

The actual change to the creation logic

lxc/export: Add --compression option

The change to the client to set the option based on the command line argument

i18n: Update translation templates

As the previous commit changes translated strings, you need a commit that updates the translation templates. You can generate the content of this commit by running make i18n.

Let me know if you need any help with this!

Thanks!

I may ping you in order to find out how you test the changes, otherwise I think I'll be fine ;)

@stgraber shall I log backup_compression_algorithm to instances_backups table? If yes, where do I create migration to add new field to that table?

No, we don't need to record the compression algorithm, we have logic to figure it out from the on-disk files.

is there an easy way to test my changes? I ran make and I have binaries, but I'm not sure how to set the whole system up

If you're using the snap, you can move your locally built binaries to:

/var/snap/lxd/common/lxd.debug
/var/snap/lxd/common/lxc.debug

Then reload LXD with systemctl reload snap.lxd.daemon which will then have it start from the debug binaries.

Thanks for the tip...this would definitely help me stop programming in the dark :)

Seems to be working well from a quick local test.

Great. Thanks for all the help and such an awesome project!

Np, thanks for the contribution!


function list on the left should become scroll-able if we have more functions that the space can display

Verified. One might argue that the New Function UI should not scroll, but right now, it's too minor to worry about.


[HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table

What is the purpose of the pull request
The table created by CTAS has missed the record fields in the hoodie.properties which will lead the crash for merge & update. This PR try to fix this bug.
Options
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

I will land it after the Ci has pass to avoid block others.


added location permissions

ACCESS_FINE_LOCATION was the only one needed to get past the bottleneck, but I added two others from the tcnclient AndroidManifest.xml -- don't know if that is correct

We don't need this anymore because Exposure Notification library handles this for us. Consequently, new design doesn't have the permission part as well:
https://www.figma.com/file/I4OOjT4VhsSberDIAZTkcH/Covid-Watch-App-v2.0?node-id=0%3A1


Issue with Mastodon API

$pleroma-bot --archive /home/gummipunkt/twitter-2022-11-04-89ab2.zip

ℹ 2022-11-08 10:54:47,756 - pleroma_bot - INFO - config path: /home/user/pleroma/config.yml
ℹ 2022-11-08 10:54:47,756 - pleroma_bot - INFO - tweets temp folder: /home/user/pleroma/tweets
ℹ 2022-11-08 10:54:47,760 - pleroma_bot - INFO - ======================================
ℹ 2022-11-08 10:54:47,760 - pleroma_bot - INFO - Processing user:       user
✖ 2022-11-08 10:54:48,391 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:571)

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://domain.tld/api/v1/accounts/mastodon_user/statuses

--> config.yml
# Change this to your target Fediverse instance
pleroma_base_url: https://domain.tld
# How many tweets to get in every execution
# Twitter's API hard limit is 3,200
max_tweets: 40000
# Twitter bearer token
twitter_token: AAAAAAAAAAAAAAAAAAAAALREALLYLONGBEARERTOKEN
users:
- twitter_username: twitterusername
  pleroma_username: mastodonusername
  # Mastodon/Pleroma bearer token
  pleroma_token: dfdC9QWBcTeGLp-X8-notaslongastwitterbearertoken

Hi!
Hmm, is the value of pleroma_username on your config the username of your Mastodon user by any chance?
Please make sure it's actually the Mastodon account ID:
https://robertoszek.github.io/pleroma-bot/gettingstarted/configuration/#mastodon
Due to how Mastodon's API works, the ID of the user is needed instead of the username itself.
Could you double check if that's the case in your config?

Right, my mistake. Thanks. Next problem:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/twitter-2022-11-04-89ae0c61e6b2e1914fb9a52867a002231d/data/tweet.js'
Twitter changed a few months ago tweet.js to tweets.js
Thanks.

Huh, I wonder if they also messed around with the formatting. I'll need to look into it. Thanks for bringing it to my attention!
I've requested a new archive to double check, it'll take 24hrs to process though.
In the meantime, would you mind checking if changing the path from tweet.js to tweets.js is enough?
You can do so by installing this release candidate version with the change:
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc9

Hi, Thank you for looking into that. Yeah, already tried that:

 File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 559, in main
    user.check_pinned(posted)
  File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/_utils.py", line 356, in check_pinned
    pinned_post = self.pin(id_post_to_pin)
  File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/_utils.py", line 630, in pin
    pin_id = self.pin_pleroma(id_post)
  File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/_pin.py", line 133, in pin_pleroma
    self.unpin_pleroma(pinned_file)
  File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/_pin.py", line 179, in unpin_pleroma
    pinned = _find_pinned(self, pinned_file)
  File "/home/gummipunkt/.local/lib/python3.8/site-packages/pleroma_bot/_pin.py", line 200, in _find_pinned
    if post["pinned"]:
KeyError: 'pinned'

Finally got my hands on a new archive, apart from changing the filename of tweet.js to tweets.js the formatting doesn't seem to have changed much.
To make sure I can try to reproduce your issue, please remove any residual folders you may have on your machine from previous failed runs (remove the folders tweets and users and their contents if they exist).
Could you try also running it with the flag --forceDate?
$ pleroma-bot --archive /home/gummipunkt/twitter-2022-11-04-89ab2.zip --forceDate

I'm wondering if maybe the Mastodon account has posts more recent than the archive, or it's reporting as such.
If after all that still fails at the same point, please give this version a try and report back if it makes any difference:
 $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc10

Thanks, with --forceDate it works!


Add a server command for hosting log events

This should probably be implemented as a runtimepy/conntextual application?
Can get command-line arguments from the YAML and can specify server and client connections etc.
This really just means that for now, we only need to implement the CombatLogEventConnection (do we need specific ones for TCP and WebSocket?).

Started dev here: https://github.com/vkottler/gnomish-army-knife/pull/4
Next up things:

set up unit testing so we can run our LogServer task, ideally using the test data
set up a client and server pair (one is the forwarder, one the consumer / disk writer?) in the config
server should wait for a new log file to show up when it finishes processing the newest one?

Depends on https://github.com/vkottler/runtimepy/issues/154.

The arbiter task implementation should now work for both incoming server connections (if a server is configured) and out-going client connections (e.g. to some central log ingester).
We need to change it to perform a connection search on a recurring basis though

Next step is to test (using test data) with a client connected. We should also create a script / config that should run while we're actually playing!


Refactor JS to prepare for JS bundling migrationg

Description
Currently some JS files rely on the fact that JS files loaded individually and that they execute in global environment. With JS bundling that will no longer be the case as there each JS file ends up being executed in its own function scope (imitating execution within a module).
This PR fixes 2 cases where we rely on files being loaded in global environment. Also removes few unnecessary local variables during controller instantiations.
Checklist
General

[x] Added the correct label: < feature | bug | dependencies | infrastructure | ignore-for-release >
[ ] Created tests which fail without the change (if possible)
[ ] Extended the README / documentation, if necessary

User visible changes

[ ] Wrote browser tests using the validateAccessibility method
[ ] Manually tested at 200% size
[ ] Manually evaluated tab order

New Features

[ ] Add new FeatureFlag env vars to server/conf/application.conf
[ ] Conditioned new functionality on a FeatureFlag
[ ] Wrote browser tests with the feature flag off and on, etc.

Issue(s) this completes
Issue #3864

Codecov Report

Merging #3910 (f575849) into main (2fb479e) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #3910   +/-   ##
=========================================
  Coverage     80.79%   80.79%
  Complexity     2454     2454
=========================================
  Files           297      297
  Lines          9668     9668
  Branches        659      659
=========================================
  Hits           7811     7811
  Misses         1602     1602
  Partials        255      255

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Enable changing output and source directory from CLI

We should make it easier for people to specify some of the core paths from the CLI. Reported by @skoop.
These values:
https://github.com/sculpin/sculpin/blob/master/src/Sculpin/Bundle/SculpinBundle/DependencyInjection/Configuration.php

Makes perfect sense to me, tbh I thought it was already in there :+1:

Me too!

:+1:

Hi there,
Not sure this issue is really clear, or is even an easy pick. If I'm not wrong, containers are compiled and then are immutables. When they are compiled, we don't have access to an InputInterface object, so there's no way I know to populate containers with CLI parameters before compilation.
Is there any way to overwrite default values with cli parameters without a massive code duplication (checking for cli parameters every time source or output dir is needed)?

From http://symfony.com/doc/current/components/dependency_injection/parameters.html :
$container->setParameter('mailer.transport', 'sendmail');
Or in this case:
$container->setParameter('sculpin.output_dir', 'some/dir);
That should do the trick nicely in the command. I'll see if I can add a PR for that.

This was neither clear nor an easy pick, but it's in Sculpin 3.0RC1 (after some ugly hacks) so closing this issue. :)


missing cfg flags with cmake build

Thanks for all your work on esp-rs!
esp-idf-svc has a bunch of cfg flags, eg esp_idf_comp_esp_netif_enabled, that normally get set by some automatic process I don't understand.
This doesn't work with cmake projects from this template. If you:

create a std project called "test" with cargo generate --vcs none --git https://github.com/esp-rs/esp-idf-template cmake
add esp-idf-svc = "0.42.5" to  components/rust-test/Cargo.toml
add use esp_idf_svc::netif; to components/rust-test/src/lib.rs

then you get this error:
error[E0432]: unresolved import `esp_idf_svc::netif`
 --> src/lib.rs:2:5
  |
2 | use esp_idf_svc::netif;
  |     ^^^^^^^^^^^^^^^^^^ no `netif` in the root

If you brute-force it, by adding set(ESP_RUSTFLAGS "--cfg esp_idf_comp_esp_netif_enabled") to components/rust-test/CMakeLists.txt, then this shifts the error onto other missing flags (such as esp_idf_comp_esp_event_enabled).
Any ideas?

esp_idf_comp_* flags get auto-defined by the build system when you enable the corresponding ESP IDF component as part of the CMake build.

Aha! Thank you!


MSSql: support windows authentication

Right now, we're using tedious a pure Javascript TDS driver. Although there is work in progress to support windows authentication (on windows only). See:

https://github.com/tediousjs/tedious/issues/415
https://github.com/tediousjs/tedious/issues/246
https://github.com/tediousjs/tedious/pull/497
https://github.com/tediousjs/tedious/commit/f5a2260f
https://github.com/tediousjs/tedious/pull/624
https://github.com/tediousjs/tedious/issues/512
https://github.com/tediousjs/tedious/issues/690

Alternative drivers:

msnodesqlv8 (windows only, native module, requires Visual Studio 2015/2017)

@n-riesco  Any progess on the Windows Authentification issue?

@pabucur No, no progress. I've just checked and I haven't seen any work on mssql in https://github.com/tediousjs/tedious since April.

Another existing Plotly client has also requested this feature.
cc @n-riesco @nicolaskruchten @sudburyrob


Support using auto with Postgres ArrayFields

Description
Previously, any django.contrib.postgres.fields.ArrayFields used in models had to be typed manually (say, as list[MyEnum]). Now they're compatible with auto, and resolve to a List of the type of base_field.
Types of Changes

[ ] Core
[ ] Bugfix
[X] New feature
[X] Enhancement/optimization
[ ] Documentation

Issues Fixed or Closed by This PR
Fix https://github.com/strawberry-graphql/strawberry-graphql-django/issues/200
Checklist

[X] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[X] I have read the CONTRIBUTING document.
[X] I have added tests to cover my changes.
[X] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Currently it seems like there's two issues here:

I'm always adding Postgres dependencies to dev installs. I'd imagine we'd want to make sure tests pass w/o Postgres deps installed (for if they're not at runtime) - is there any precedent for splitting dev dependencies like this? I've only seen Django/Python versions handled specially, not any other optional dependencies.
Even with Postgres dependencies added (verified w/ Poetry output in GH Actions logs), this is still failing - even though it passes locally. I'll dig more into this soon.

Shouldn't we use psycopg3 ?
Django by default now uses psycopg3

Shouldn't we use psycopg3 ?
Django by default now uses psycopg3

The dependency here is a "dev" one and mostly for tests, meaning that if we used psycopg3 we would not be able to test on django 4.1 or older

Why was this closed and not merged?
Is there missing todos here?

@bellini666 do you think we should get this in?

@bellini666 do you think we should get this in?

Yes I do! It makes sense
Not sure why @colinmcdonald22 closed this, but I can take over it to ensure it will get in

@bellini666
Maybe this PR should be re-opened so it won't be forgotten ?

I need this!

I need this!

You shall have this https://github.com/strawberry-graphql/strawberry-django/pull/567


OpenVPN - Client Specific Overrides are not applied

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
OpenVPN settings for (Client Specific Overrides) in the ccd folder are not correctly applied.
To Reproduce
Steps to reproduce the behavior:

Create a new OpenVPN Server
Create Client Specific Overrides (for 4 Clients)

I only setting this option for the common name (user)
ifconfig-push <IP_ADDRESS> <IP_ADDRESS>;
Expected behavior
Clients who are connecting should apply the settings from the Client Specific Overrides.
Describe alternatives you considered
I checked the folder inside /var/etc/openvpn-csc/2 and only 2 out of 4 config where created. All 4 clients where connected,
but as stated only two where using the right ip. After creating the config inside /var/etc/openvpn-csc/2 manually for
the other two clients the settings are applied.
Screenshots
Relevant log files
Additional context
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021
VMware esxi 7.0
Intel(R) Xeon(R) E-2124 CPU @ 3.30GHz (4 cores)
VMXNET3

I checked the folder inside /var/etc/openvpn-csc/2 and only 2 out of 4 config where created.

Best check the openvpn logs after login, since client specific overrides can be dynamic (radius for example), these are generated during login.
https://github.com/opnsense/core/blob/2e8e0adb81995e760ab821f047525fc192742ff3/src/etc/inc/plugins.inc.d/openvpn/auth-user.php#L144


More cats 🐱

Thanks for your valuable input and very well written, exhaustive issue.
More cats 🐱 was addressed in dde4d4841812cf02b7081960c10ab2a5497bc982, thanks to the generous image contributions of @danielSWolf.


Throw or return null when message has been sent entirely?

Should we throw an error or just return null?
See: https://github.com/saltyrtc/chunked-dc-js/blob/673ffd4fb259532625bc307bb25ded830e394cd6/chunked-dc.ts#L52:L54

Obsoleted by #8.


JWS/JWE evolutions

At the moment, the loader returns a JWE/JWS or an array of them.
These objects must be updated to handle all recipients/signatures. This way the loader wil always return a JWE or a JWS object.
The JWS/JWE object must have getters to access on shared unprotected headers per recipients, encrypted key.
They should be have converter included to return a JSON Serialization representation (full, flattened or compact). When JWEs are decrypted, a recipient could be easily added.
Key used to decrypt or verify the signature should be available.

Done. See branch v2.0.x.


Add user avatar to the navbar

Fixes: #2305
@timabbott please have a look and let me know if any more changes are to be made.

Automated message from Dropbox CLA bot
@aakash-cr7, it looks like you've already signed the Dropbox CLA. Thanks!

@timabbott reminder: This is completed.

Sorry I didn't get a chance to look at it when you first posted it.  @aakash-cr7 can you rebase?  This branch is very old.

Yeah sure @timabbott . I'll rebase it and let you know.

@timabbott I rebased the branch, but the checks doesn't seem to come through.

@aakash-cr7 Would you mind trying again to make it easier for me to test? Thanks!

I think this project has been replaced by #4083.
Thanks again for your work on this @aakash-cr7, and I'm sorry this PR got stuck waiting on me to find time for it :(.


chore(deps): update common-dev-assets digest to cdd71fc

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
a7345c4 -> cdd71fc

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 6.3.0 :tada:
The release is available on:

GitHub release
v6.3.0

Your semantic-release bot :package::rocket:


Open source the dataset with non-commercial license

I believe functionary dataset could significantly contribute to the research and development efforts of the community working on LLMs. Your work has already shown immense potential, and making the dataset available could further accelerate innovation and collaboration in this space. Thanks in advance.

Hi Berat!
We don't plan to release the dataset. Thank you for your interest and suggestion!


Running models on Windows on Snapdragon

I would like to run the controlnet(https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/controlnet_quantized/README.md) models on the HTP on snapdragon with windows. Is it possible to run the exported binary cache files of the controlnet models on snapdragon with windows?
In the docs I saw that the only support runtime is qnn_lib_aarch64_android for job compilation.

Hi @kristoftunner you can run exported context binary files on snapdragon with windows. Please make sure you are also using QNN SDK 2.19
We are in the actively validating these models on snapdragon on windows, it would be great if you can give it a spin and let us know how it goes :)

@bhushan23  thx for the quick feedback, I will update you here once I have it running.

Hi @kristoftunner , a clarification. You will need to use --target_runtime qnn_bin to get a cross platform asset.

@bhushan23 the model binaries are working under windows, I tried it out on a volterra machine, thx


Youth national data demographics

@longhotsummer
Not the most elegant, but at least it's temporary : )

Cool!


Match telescope descriptions and camera descriptions by an index, not by name

We should save an additional table where we store for each telescope id, the index for the camera description and the optics description.
At the moment, we assume that the tel_description string is unique.
But this is not the case for simulations of the LST / MAGIC array since there, the two MAGIC telescopes differ only in mirror area.
This is currently not possible and reading back the instrument description will not yield the correct result.

I think we assume this in quite a lot of places. But I think this is really an anti-pattern.
We rely on something that comes from the guessing part of the names rather than on something well-defined like telescope ids and indices.
To be safe, we could just store everything by telescope_id, which would be a bit larger, but ultimately the safe bet.

Example:
In [3]: s = SimTelEventSource('/fefs/aswg/workspace/MC_common/corsika7.7_simtelarray_20200629/prod5/4LSTs_MAGIC/gamma/zenith_20deg/south_point
   ...: ing/sim_telarray/off0.4deg/data/gamma_20deg_180deg_run1___cta-prod5-lapalma_4LSTs_MAGIC_desert-2158m_4LSTs_MAGIC_mono_off0.4.simtel.gz
   ...: ')

In [4]: s.subarray.to_hdf('test.hdf5')

In [5]: s2 = s.subarray.from_hdf('test.hdf5')

In [6]: s2 == s
Out[6]: False

An alternative would be to include mirror area and and/or num_mirror_tiles in the guessing, but like I said, I don't like relying on the guessing at all.
It also makes it impossible match the telescopes without the guessing code inside ctapipe, which is also a nogo, I think.


Irregular - Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Only from linux

Hi
I've a big problem with PHPSQLSRV, and SQLMCD on linux, too. We have irregular (mostly every 30-40 or X  minutes) no connection to our SQL 2016 windows-server. This is the error message of sqlcmd:

Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Timeout error [258]. .
Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Unable to complete login process due to delay in prelogin response.
[2017-07-07 14:00:26] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S <IP_ADDRESS> -d easy_prod -U myUser -P myPassword -l 2 -t 2 -Q 'SELECT @@VERSION''

I've created a BASH-script, that runs every 5 seconds SQLCMD:

#!/bin/bash
safeRunCommand() {
dt=$(date '+%Y-%m-%d %H:%M:%S');
typeset cmnd="$*"
typeset ret_code
eval $cmnd
ret_code=$?
if [ $ret_code != 0 ]; then
printf "[%s] Error : [%d] when executing command: '$cmnd'" "$dt" $ret_code
fi
}
LOG_FILE=/var/log/test-sqlconnection.errors.log
command="chronic /opt/mssql-tools/bin/sqlcmd -S <IP_ADDRESS> -d easy_dev -U myUser -P myPassword -l 2 -t 2 -Q 'SELECT @@VERSION'"
exec 3>&1 1>>${LOG_FILE} 2>&1
while [ "1" = "1" ]; do
safeRunCommand "$command"
sleep 3
done

I've created a POWERSHELL-Script, that runs on a Windows2012-server and on the SQL-Server:

FUNCTION Test-SQLConnection {
[CmdletBinding()]
Param (
[Parameter(Mandatory=$true)]
[string[]]
$Servers
,
[Parameter(Mandatory=$false)]
[Alias('LogPath')]
[string]$Path='',
[ValidateSet("String","Object")]
[string]
$OutputTo
)
BEGIN {}
PROCESS {
TRY {
Invoke-Sqlcmd -ServerInstance <IP_ADDRESS> -Database easy_prod -Username myUsername -Password myPassword -ConnectionTimeout 3 -QueryTimeout 3 -Query 'SELECT @@VERSION'
}
CATCH {
Write-Log -Message "No connection to SQL-server" -LogPath $LogPath
Write-Output "No connection to SQL-server"
}
}
}
}
function Write-Log
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true,
ValueFromPipelineByPropertyName=$true)]
[ValidateNotNullOrEmpty()]
[Alias("LogContent")]
[string]$Message,
[Parameter(Mandatory=$false)]
[Alias('LogPath')]
[string]$Path='',
    [Parameter(Mandatory=$false)]
    [ValidateSet("Error","Warn","Info")]
    [string]$Level="Info",

    [Parameter(Mandatory=$false)]
    [switch]$NoClobber
)
Begin
{
    $VerbosePreference = 'Continue'
}
Process
{
    if ((Test-Path $Path) -AND $NoClobber) {
        Write-Error "Log file $Path already exists, and you specified NoClobber. Either delete the file or specify a different name."
        Return
        }
    elseif (!(Test-Path $Path)) {
        Write-Verbose "Creating $Path."
        $NewLogFile = New-Item $Path -Force -ItemType File
        }
    else {
        }
    $FormattedDate = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    switch ($Level) {
        'Error' {
            Write-Error $Message
            $LevelText = 'ERROR:'
            }
        'Warn' {
            Write-Warning $Message
            $LevelText = 'WARNING:'
            }
        'Info' {
            Write-Verbose $Message
            $LevelText = 'INFO:'
            }
        }

    "$FormattedDate $LevelText $Message" | Out-File -FilePath $Path -Append
}
End
{
}

}
$LogPath = $PSScriptRoot + "" + $MyInvocation.MyCommand.Name -replace "ps1", "log"
while (1 -eq 1){
Test-SQLConnection -Servers <IP_ADDRESS> -LogPath $LogPath
start-sleep -seconds 5

}

This script runs every 5 seconds, too.
The irregular connection problems are only on the two linux-servers (same time are the problems!), not on the Windows2012-servers!
Linux-Webserver1 - <IP_ADDRESS>, Ubuntu 16.04, msodbcsql <IP_ADDRESS>-1, unixodbc  2.3.1-4.1, PHP7.1
Linux-Webserver2 - <IP_ADDRESS>, Ubuntu 16.04, msodbcsql <IP_ADDRESS>-1, unixodbc  2.3.1-4.1, PHP7.1
Windows-Webserver - <IP_ADDRESS>, Windows 2012 R2, Powershell 5.0
SQLserver - <IP_ADDRESS>, Windows 2012 R2, SQL 2016 Standard, Powershell 5.0
There are no activated firewalls on all of the servers. I'm connectiong over IP, not over DNS-name. All server are virtualized. We don't have any other connection problems or network problems. All the servers are in the same subnet. We have no problem by the connection from Windows-Webserver to the SQLserver, only between
Linux-Webserver1 -> SQLserver
and
Linux-Webserver2 -> SQLserver.
Please help me to solve problem.
Best regards
Stefan

Hi @stefankaifer, this sounds like a network issue - the function that returns that particular error message reads TDS packets. Have you tried using wireshark or netcat to identify the problem?

Hi @david-puglielli, sorry, but I can't explain that (network issue). Why?

Same server, same subnet, only different OS:
Windows to SQL-Server: no problems in the last 5 days.
Linux (IP: <IP_ADDRESS>) to SQL-Server (IP: <IP_ADDRESS>): random timeouts in the last 5 days.

I've installed Wireshark on the SQL-machine; if I understand the wireshark log, it can't be a network issue. Why? Please see this log:

Please see the Line No. 29314-29325. in the first lines are everything OK, but in the last not. Same ports. Why can I see the errors (No. 29322-29325) in Wireshark, if I've a network problem? I think, that the problem is on SQL-side, or not?
But in the SQL-errorlogs I can't find an issue. Can you advice me, please?
Regards,
Stefan

You're getting RST responses, so the linux client is refusing packets from the server. My guess is a configuration issue on the client side. I would try running wireshark on the client. Also, does the problem happen if you try connecting with isql (from unixODBC) instead of sqlcmd?

@stefankaifer Please run the bash script you wrote above using strace -tt. strace -tt sh <scriptname>
When a failure occurs, inspect the timestamps and look for any backward jumps. We suspect there is a bug in the way the ODBC driver calculates clock time - it may jump backwards occasionally, possibly right when sqlcmd disconnects. Please let us know if you observe this behaviour.

@david-puglielli thanks! I'm capturing on the linux-client with tshark and I wrote a isql-script, too. Since yesterday there was no connection issue with isql, but with sqlcmd again!
My isql-command:
chronic /usr/bin/isql -v SECTION_FROM_ODBC.INI USERNAME PASSWORD -b < isql.sql
There could be a timestamp issue, your're right. Please see my log of the sqlcmd-script:
[2017-07-16 21:34:50] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-pilot.HE.DIR -d easy_pilot -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''[2017-07-16 21:34:50] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-prod.HE.DIR -d easy_prod -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Error code 0x102. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. [2017-07-17 21:33:57] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-pilot.HE.DIR -d easy_pilot -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Error code 0x102. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. [2017-07-17 21:33:57] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-prod.HE.DIR -d easy_prod -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Query timeout expired. [2017-07-17 21:33:55] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-prod.HE.DIR -d easy_prod -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Error code 0x102. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. [2017-07-17 21:34:02] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-pilot.HE.DIR -d easy_pilot -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Error code 0x102. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.. [2017-07-17 21:34:02] Error : [1] when executing command: 'chronic /opt/mssql-tools/bin/sqlcmd -S db-easy-prod.HE.DIR -d easy_prod -U easy -P u5LWhfmEtF8B2TYN -l 2 -t 2 -Q 'SELECT @@VERSION''Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : Login timeout expired. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : TCP Provider: Error code 0x102. Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online..
First problem at 21:34:50, but the second at 21:33:57, the timestamp should be a later one, it couldn't be earlier!
Maybee my provider (it is a cloud vmware-server) activated the option "Time synchronisation" in the vmware-options. I'll check it.
Here is a screenshot of the tshark-capture on the linux-client:

I'll trying your posted strace-command.

That will strace the /bin/bash, which won't log anything about sqlcmd. Instead, you should run strace on sqlcmd inside the script, and have the script exit the loop when sqlcmd fails.

Thanks, v-chojas. I've modified my script, but since two days I haven't any issues (==> no strace-log). It's very strange!
I've changed only one thing: because of the above "isql"-test, I've created the file /etc/odbc.ini (it wasn't there by default installation). But I don't belive, that it is the solution. I'm investigating further...

Adding an odbc.ini file is not relevant to the problem I think. Please do keep us apprised of any further developments though!

thanks, guys. I had the problem 2 or 3 times in the last days again. I've attached the strace-output.
strace-2017-07-24-21-33-59.txt
strace-2017-07-24-21-34-19.txt
strace-2017-07-25-19-58-19.txt

@stefankaifer I believe you are still using strace on bash, you must run it on sqlcmd inside the script. There doesn't seem to be anything unusual in the logs you have provided.

@stefankaifer Have you had a chance to try strace on command as @david-puglielli suggested?

@stefankaifer do you have any update on this?

Hi guys, sorry for the late answer, but I had a lot of todos in the last weeks.
The problem is existing sometimes. My problem is, that I don't know how to run the strace command crrectly. I've read @david-puglielli's answer

@stefankaifer I believe you are still using strace on bash, you must run it on sqlcmd inside the script. There doesn't seem to be anything unusual in the logs you have provided.

but I don't know, ho to do it right. Can seomebody modify my script, please?
`#!/bin/bash
safeRunCommand() {
dt=$(date '+%Y-%m-%d %H:%M:%S');
dtlog=$(date '+%Y-%m-%d-%H-%M-%S');
LOG_FILE_STRACE="/var/log/strace-$dtlog.log"
typeset cmnd="$*"
typeset ret_code
#echo cmnd=$cmnd
eval $cmnd
ret_code=$?
if [ $ret_code != 0 ]; then
strace -tt -f -o $LOG_FILE_STRACE /opt/mssql-tools/bin/sqlcmd -S <IP_ADDRESS> -d easy_dev -U myUser -P myPassword -l 2 -t 2 -Q 'SELECT @@VERSION'
printf "[%s] Error : [%d] when executing command: '$cmnd'" "$dt" $ret_code
| tee /dev/fd/3
#exit $ret_code

fi
}
LOG_FILE=/var/log/test-sqlconnection.errors.log
command="chronic /opt/mssql-tools/bin/sqlcmd -S <IP_ADDRESS> -d easy_dev -U myUser -P myPassword  -l 2 -t 2 -Q 'SELECT @@VERSION'"
exec 3>&1 1>>${LOG_FILE} 2>&1
while [ "1" = "1" ]; do
safeRunCommand "$command"
sleep 3
done`

You are running strace correctly - the problem is that the error happens only occasionally, so if you run strace only when an error occurs, the error will probably not occur again if you run the same command again under strace.
What version of the ODBC driver are you using? I would suggest upgrading to version 17, available here. The bug that causes time to jump backwards has been fixed in version 17, so upgrading may solve your issue.

@david-puglielli Thanks. I had version 13, now I've installed version 17 on my development system. I'm observing...

Hi guys,
I think, that with the current odbc-version (17) the problem is gone! I hadn't drops in the past 2 weeks.
Thank you very much for the support!
Regards
Stefan

Glad we could help! You may also be interested to know, we will upload an updated ODBC driver 17 later this month as well.
Closing the issue.

@stefankaifer
I'm experiencing a similar issue, which might be connected to this case. Could you clarify: were you using persistent database connections (connection pooling) in your setup?

@david-puglielli I've tried upgrading to version 17 of the driver, but got:

SQLSTATE[IMSSP]: This extension requires the Microsoft ODBC Driver 13 for SQL Server to communicate with SQL Server

Does that means that v17 is not supported with sqlsrv driver for PHP? or that is not finding the driver at all?
Problem I'm getting is that after 2/3 minutes of the driver running correctly, I get timeout login for all of the connections (even if I try to reconnect again).
Tried enabling pooling and increasing CPTimeout, but nothing helped. 😟

@pmartelletti Please try adding the Driver keyword to your connection string and specifying version 17, like this:
Driver={ODBC Driver 17 for SQL Server};

This will override the default ODBC driver version.

@david-puglielli hmm.. I'm using:
'driver' => 'sqlsrv',

how should I specify the version there? sqlsrv17 won't work.

@david-puglielli  sorry, just realised you were referring to the .odbcinst.ini file.
Changed that to look something like this, in all the locations that I could find:
/etc/odbcinst.ini
/home/forge/.odbcinst.ini
/opt/microsoft/msodbcsql/etc/odbcinst.ini

to look something like this:
[ODBC]
Pooling=No

[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
#Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.9.2
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-17.0.so.0.5
UsageCount=1

but still getting from PHP the following:

Can't open lib 'ODBC Driver 13 for SQL Server' : file not found

So seems like it's still trying to use the v13 of the driver.
Any ideas?

@david-puglielli is it possible that the PHP driver only works with the ODCB Driver 13, as per this list? Or that's just listing stable releases?
https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver

That page just lists stable releases. The latest preview (5.2.0-RC) supports ODBC 17. To use ODBC 17, instead of 'driver=>'sqlsrv', use
'driver'=>"ODBC Driver 17 for SQL Server"

in your PHP script. To make sure you are using a supported version of the drivers, run
php --ri sqlsrv
php --ri pdo_sqlsrv

from the command line and verify that ExtensionVer is at least 5.2.0.


Add Scopes

Currently, code such as
if (cond) {
  let a = 3;
}
a = 10;

compiles without any problems.
In a proper environment, compilation should fail with "Undefined variable a in line 4", because a's scope is only the if-block.

First version of scope handling introduced in the latest commit.
Probably prone to bugs, but it works for now.


Add support for Windows Azure

Verify installation on Windows Azure and add support for using Blob storage for storing uploaded media files.

How do you avoid media being overwritten when you redeploy to Azure using the default setup (i.e. without the Azure Storage providers)?
What about when using multiple deployment slots on Azure websites Standard?

Well I almost never use Azure storage and never have this issue. It all depends on how you deploy your application. If you deploy directly from Visual Studio, make sure that you have selected to "delete everything" before publish, and that you're "including the App_Data folder". If you deploying using some automated service, make sure that above applies.
This really isn't a problem with Piranha CMS, it rather has to do with the deployment process.

Surely that would overwrite the app_data folder in that case?  Are you managing your content locally then publishing?
My question is about managing the content via the manager hosted on Azure and files uploaded from there.

Yes you're right, you should only include app_data the first time probably.
You can inherit the local media to create a new file based media and store uploads anywhere as long as you give the application read write access to that folder.
Regards

Clarification. You can inherit the LocalMediaProvider!

I think the issue I'm referring to is the situation where Azure re-deploys your app after it being spun-down.
In this case only the files present in the deployment package would actually get deployed.  Or have Azure changed things to persist App_Data or other local storage between instances?
Maybe I'm just getting totally confused

I haven't come around to trying out the deployment slots on Azure yet so I can't say for sure, but by just looking at it swiftly I'm guessing your assumption is right. If your using this technique for deployment you are probably better off keeping uploaded data away from the website.
This means either storing it in either the database or blob storage. In my experience they charge more for database storage than they do for blob storage, so from a cost perspective I think the Azure provider would more beneficial.
Regards


empty_label is ignored

On ModelChoiceFields you can set an empty_label instead of the default '---------'. On smart-selects the '---------' is hardcoded and empty_label is ignored

@dmoisset, where is your pull request?

@mpena2099 Why are you digging up an issue and PR that's been closed for 7 years?
His PR is here: https://github.com/digi604/django-smart-selects/pull/6
If you are also having this issue please open a new issue and/or a new PR.

@blag, because I was not sure if that pull request was accepted, and I'm having a similar problem right now.


gds_env grabs nearly everything from conda forge

I think the current installation method grabs nearly every package from conda forge, instead of pulling from conda and then grabbing only what's unavailable from conda forge.
Not sure why it's doing this, except that when you use
[channels]
  - conda-forge

the only channel that is recognized in that environment is conda forge.
I think using:
[channels]
  - defaults
  - conda-forge

will prefer to install from the main conda repositories & (if a package is not not found) install from conda-forge.

Good point. Would defaults be better? I've had issues with geo libraries being installed from there as opposed to conda-forge (they might be solved now though).

I think so? Maybe there's a way to specify which channel to install each
time, like the -c conda-forge call... Will investigate.
On Mon, Apr 30, 2018, 13:41 Dani Arribas-Bel<EMAIL_ADDRESS>wrote:

Good point. Would defaults be better? I've had issues with geo libraries
being installed from there as opposed to conda-forge (they might be
solved now though).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/darribas/gds_env/issues/4#issuecomment-385374580, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACJY8_D71_TWN9knxik70Rykxk9TQeYtks5ttvhigaJpZM4TsVho
.

--
Levi John Wolf
Lecturer in Quantitative Human Geography | University of Bristol
Fellow | Center for Spatial Data Science, University of Chicago
Associate Member | Center for Multilevel Modeling, University of Bristol
Interim Director | Q-Step Program, University of Bristol
ljwolf.org

Merged and we can see how performance is influenced! Let me know if you use this at all, very curious on feedback!

Just ran into @ocefpaf's comment on geopandas/geopandas/pull/796, so I thought I'd tag it here.
In theory, I bet this would be most stable if you either used defaults only, or used conda-forge only.  But, if there's no issues (i.e. people running into the libkea stuff), then no reason to change!

Yep. Use one or the other but not both. The current state of channel preference is broken in conda.
Note that now conda-forge ships the compilers runtimes, the only thing we did not have in conda-forge before, so using only conda-forge is the preferred solution IMO. (The other way around is OK too, remove conda-forge and use only defaults if you don't need anything from conda-forge.)
PS: the more +1 we get in this issue the higher are the changes of them prioritizing it and fixing this.

Thanks! To be honest, I have not compiled the container in a while. I built it a couple of months ago and checked it on Docker hub. I will need to update it very soon to reflect new PySAL and geopandas versions, so this will surely be useful. The conclusion thus is to do:
conda install -c conda-forge --quiet --yes

and the set of libraries?
When I start with the new build, I'll first tag the current one as stable so a working one will be available.


merge of  library into sdk

merge of library transaction builders into SDK
merge of library crypto implementation into SDK
merge of library encoders into SDK
merge of library swagger generated classes into SDK

Pull Request Test Coverage Report for Build 393

1 of 4 (25.0%)  changed or added relevant lines in 1 file are covered.
23 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-75.3%) to 3.704%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/model/account/Account.ts
1
4
25.0%

Files with Coverage Reduction
New Missed Lines
%

src/model/account/Account.ts
23
3.7%

Totals

Change from base Build 375:
-75.3%

Covered Lines:
1

Relevant Lines:
27

💛  - Coveralls

Pull Request Test Coverage Report for Build 393

1 of 4 (25.0%)  changed or added relevant lines in 1 file are covered.
23 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-75.3%) to 3.704%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/model/account/Account.ts
1
4
25.0%

Files with Coverage Reduction
New Missed Lines
%

src/model/account/Account.ts
23
3.7%

Totals

Change from base Build 375:
-75.3%

Covered Lines:
1

Relevant Lines:
27

💛  - Coveralls

It is the same as the library, to test the schema and  buffer. I am directly copying

ok,I will deal with it seriously.

Handled in other PRs


icon path - mysql

Please can you replicate the same sql icon to a mysql folder?

Sure, but I’d suggest you to do a pull request instead. Thanks!

You say follow up, correct? By Watch?

And ... ?

I'll do it for version 20.0

OK, thank you


feat: add timeout to controller.py

Requested from Daniel ~
Original Post:
Re: timeouts, we would need to adjust the timeouts on the record_with_value calls which is actually happening in the protocols methods we're calling, I think. We'd need to update the acapy-minimal-example controller to allow us to set a default timeout.

Subtle note: The get method forces it to be a int. We can either change this or leave it be, let me know! :)

LGTM!


config/prometheus: add metrics exporter for workers

Why are these changes needed?
Sample configuration for exporting metrics from ray cluster workers. This works with autoscaling and should cover new workers and remove destroyed worker pods as well

The podMonitor CRD resource works in a similar way that serviceMonitor works but instead of targeting services, it targets pods.
Prometheus example after applying this manifest
ray_raylet_mem{..., container="ray-head", ...} | ...
...
ray_raylet_mem{..., container="ray-worker", ..., pod="ray-cluster-main-worker-generic-group-h2nhg",...} | ...
...

@ulfox This is great. BTW, how do you use pod level (ray worker) metrics in your case? We considered to monitor workers in our downstream but feel there're not lots of values. I am trying to learn how you leverage those metrics?  /cc @scarlet25151

@ulfox This is great. BTW, how do you use pod level (ray worker) metrics in your case? We considered to monitor workers in our downstream but feel there're not lots of values. I am trying to learn how you leverage those metrics? /cc @scarlet25151

We currently use the worker's metrics for obersvability using grafana panels.
We check

active workers per node group to detect activity spikes on ray cluster
Scheduling status on ray workers, for example Unscheduled tasks

For example with the following query
sum(ray_scheduler_unscheduleable_tasks{ray_io_cluster="$RayCluster"}) by (Reason, pod)

We can detect waiting for resources or plasma memory spikes and then check

if it was infra related
activity related (spike)
the client is using a non-optimized code

Some additional examples of metrics we observe
sum(rate(ray_scheduler_failed_worker_startup_total{ray_io_cluster="$RayCluster"}[$__range])) by (Reason, pod)
sum(rate(ray_operation_run_time_ms{ray_io_cluster="$RayCluster"}[$__range])) by (Method, pod)
sum(rate(ray_operation_queue_time_ms{ray_io_cluster="$RayCluster"}[$__range])) by (Method, pod)
sum(rate(ray_grpc_server_req_handling_total{ray_io_cluster="$RayCluster"}[$__range])) by (Method, pod)
ray_object_directory_lookups{ray_io_cluster="$RayCluster"}

Ratio metrics
# Ratio of GRPC new / finished requests
sum(rate(ray_grpc_server_req_new_total{ray_io_cluster="$RayCluster"}[$__range])) by (Method, pod) / sum(rate(ray_grpc_server_req_finished_total{ray_io_cluster="$RayCluster"}[$__range])) by (Method, pod)

# Ratio of directory objects added / removed
ray_object_directory_added_locations{ray_io_cluster="$RayCluster"} / ray_object_directory_removed_locations{ray_io_cluster="$RayCluster"}

# Workers memory util
(1 - (ray_node_mem_available{container="ray-worker", ray_io_cluster="$RayCluster"} / ray_node_mem_total{container="ray-worker", ray_io_cluster="$RayCluster"})) * 100

Availability metrics
# [99.9] Percentile of Workers register latency
100 * (sum(rate(ray_worker_register_time_ms_bucket{le="10000.0", ray_io_cluster="$RayCluster"}[$__range])) by (pod) / sum(rate(ray_worker_register_time_ms_count{ray_io_cluster="$RayCluster"}[$__range])) by (pod))

# [99.9] Percentile of Process startup latency
100 * (sum(rate(ray_process_startup_time_ms_bucket{le="100.0", ray_io_cluster="$RayCluster"}[$__range])) by (pod) / sum(rate(ray_process_startup_time_ms_count{ray_io_cluster="$RayCluster"}[$__range])) by (pod))

@ulfox These are awesome guidances! We export the control plane grafana dashboard here. https://github.com/ray-project/kuberay/tree/master/config/grafana If the one for workers can be open sourced on your side, I think people would love it.


The XSS is possible without Laravel varidation?

When I use Laravel-Stapler without Validation, malicious file can upload public directory or storage.
An error(Symfony Componet Exception) is displayed with the screen, but I check my storage,  malicious file is exist.
It is thought that the cause of this problem happens by not confirming file mime type.
Because a similar problem happened in the version of the former "paperclip", I added a postscript to the link which seemed to serve as a reference.
Just FYI:
・Paperclip Security Release
https://robots.thoughtbot.com/paperclip-security-release
・Fix a possible security issue with spoofing
https://github.com/thoughtbot/paperclip/commit/9aee4112f36058cd28d5fe4a006d6981bd1eda57

I don't think this is an issue with this plugin; it's realy be an issue with the end-user's application application that they use this plugin in. Paperclip also/provides handles/provides mime type validation, Stapler doesn't. The only thing this plugin verifies is that the uploaded file isn't spoofed, and that validation is done via the Symfony Uploaded File Object. Aside from that, it's entirely the responsibility of the end user to verify the mime types of their files as needed by the constraints of their specific application.

Thank you for an answer. An answer is late, and I am sorry.


Documentation is missing.

What are you trying to do that isn't documented?  https://promptfoo.dev/docs/intro

I think the antivirus blocked the page, I tested it out in another PC and it worked, go figure.
I wanted to read the Guide and references, I though the website was down, but It was my connection.

Ah ok. No problem!


Board zooming

Task consists of next steps:

Extend camera to apply zoom.
Add handle for mouse scrolling to change zoom.

To reach the goal consider using:

add_system_set_to_stage()
Res<Windows>
Query

i`ll take

taken

taken

Please, add name of command.

I'll temporary untake this issue


Format docs index page.

Coverage remained the same at 100.0% when pulling e2e41cc81f3004329584d38fc419746a97acf6d2 on small_doc_improvements into 7a6fdbdb9890831de614ac51a609bd178048d434 on master.


document variables in main.bicep

Detailed Description
It would be wonderful to have bicep-docs document variables as well as usually variables will be used in main.bicep
Context
example of a Bicep variable
var stringVar = 'example value'

Possible Implementation
same way as for parameters

Thanks for your feedback. Release v1.1.0 is out and includes vars and user-defined data types & functions.
I am closing this issue.


Flakiness on dartkp-dart_precompiled caused by MallocHooks

FAILED: dartkp-dart_precompiled release_x64 co19/Language/Reference/Operator_Precedence/precedence_16_unary_postfix_t07
Expected: Pass
Actual: Crash

Crash dumps archived:
https://storage.cloud.google.com/dart-temp-crash-archive/b74dce4b-44eb-490c-9734-215420f300fd/core.15324.tar.gz
https://storage.cloud.google.com/dart-temp-crash-archive/b74dce4b-44eb-490c-9734-215420f300fd/release_x64_dart_precompiled_runtime.tar.gz
Stack trace from the dump points towards the MallocHooks:
(gdb) thread apply all bt

Thread 2 (Thread 0x7f11120ed780 (LWP 15324)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000051eee5 in dart::bin::Monitor::WaitMicros (this=0x22e8120, micros=<optimized out>) at ../../runtime/bin/thread_linux.cc:293
#2  0x00000000005029b9 in Wait (millis=0, this=<optimized out>) at ../../runtime/bin/lockers.h:40
#3  dart::bin::EventHandler::Stop () at ../../runtime/bin/eventhandler.cc:93
#4  0x0000000000500bb9 in dart::bin::main (argc=<optimized out>, argv=<optimized out>) at ../../runtime/bin/main.cc:2075
#5  0x00000000005012d9 in main (argc=36602188, argv=0x80) at ../../runtime/bin/main.cc:2103

Thread 1 (Thread 0x7f11120eb700 (LWP 15330)):
#0  0x00007f1110d0fc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f1110d13028 in __GI_abort () at abort.c:89
#2  0x0000000000506371 in dart::bin::segv_handler (signal=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at ../../runtime/bin/platform_linux.cc:27
#3  <signal handler called>
#4  0x00000000006a6ff6 in Lookup (this=<optimized out>, key=<optimized out>) at ../../runtime/vm/hash_map.h:141
#5  Lookup (this=0x23080c0, key=<optimized out>, value=<optimized out>) at ../../runtime/vm/malloc_hooks.cc:94
#6  dart::MallocHooksState::RecordFreeHook (ptr=0x22f6f60) at ../../runtime/vm/malloc_hooks.cc:274
#7  0x000000000052d705 in MallocHook::InvokeDeleteHookSlow (p=<optimized out>) at ../../third_party/tcmalloc/gperftools/src/malloc_hook.cc:505
#8  0x00000000008f7c51 in InvokeDeleteHook (p=<optimized out>) at ../../third_party/tcmalloc/gperftools/src/malloc_hook-inl.h:138
#9  tc_free (ptr=0x22f6f60) at ../../third_party/tcmalloc/gperftools/src/tcmalloc.cc:1609
#10 0x00000000005028a0 in dart::bin::TimeoutQueue::UpdateTimeout (this=<optimized out>, port=<optimized out>, timeout=<optimized out>) at ../../runtime/bin/eventhandler.cc:34
#11 0x000000000050353e in dart::bin::EventHandlerImplementation::HandleInterruptFd (this=<optimized out>) at ../../runtime/bin/eventhandler_linux.cc:211
#12 0x0000000000503c07 in dart::bin::EventHandlerImplementation::Poll (args=<optimized out>) at ../../runtime/bin/eventhandler_linux.cc:402
#13 0x000000000051e74e in dart::bin::ThreadStart (data_ptr=<optimized out>) at ../../runtime/bin/thread_linux.cc:89
#14 0x00007f1111ace184 in start_thread (arg=0x7f11120eb700) at pthread_create.c:312
#15 0x00007f1110dd337d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Seems to happen in the shutdown sequence.

Here is another instance. Seems like there is a race in shutdown introduced by MallocHooks:
(gdb) thread apply all bt
Thread 2 (Thread 0x7fc516609780 (LWP 30835)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000051eee5 in dart::bin::Monitor::WaitMicros (this=0x2dee0c0, micros=<optimized out>) at ../../runtime/bin/thread_linux.cc:293
#2  0x00000000005029b9 in Wait (millis=0, this=<optimized out>) at ../../runtime/bin/lockers.h:40
#3  dart::bin::EventHandler::Stop () at ../../runtime/bin/eventhandler.cc:93
#4  0x0000000000500bb9 in dart::bin::main (argc=<optimized out>, argv=<optimized out>) at ../../runtime/bin/main.cc:2075
#5  0x00000000005012d9 in main (argc=48161004, argv=0x80) at ../../runtime/bin/main.cc:2103

Thread 1 (Thread 0x7fc516607700 (LWP 30836)):
#0  0x00007fc51522bc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fc51522f028 in __GI_abort () at abort.c:89
#2  0x0000000000738919 in dart::OS::Abort () at ../../runtime/vm/os_linux.cc:415
#3  0x000000000064de96 in dart::DynamicAssertionHelper::Fail (this=0x7fc516606d78, format=<optimized out>) at ../../runtime/platform/assert.cc:42
#4  0x00000000007392d4 in dart::OSThread::SetThreadLocal (key=<optimized out>, value=<optimized out>) at ../../runtime/vm/os_thread_linux.cc:179
#5  0x00000000006a6fc6 in MallocHookScope (this=<optimized out>) at ../../runtime/vm/malloc_hooks.cc:37
#6  dart::MallocHooksState::RecordFreeHook (ptr=0x2df6010) at ../../runtime/vm/malloc_hooks.cc:268
#7  0x000000000052d705 in MallocHook::InvokeDeleteHookSlow (p=<optimized out>) at ../../third_party/tcmalloc/gperftools/src/malloc_hook.cc:505
#8  0x00000000008f7c51 in InvokeDeleteHook (p=<optimized out>) at ../../third_party/tcmalloc/gperftools/src/malloc_hook-inl.h:138
#9  tc_free (ptr=0x2df6010) at ../../third_party/tcmalloc/gperftools/src/tcmalloc.cc:1609
#10 0x000000000051e748 in dart::bin::ThreadStart (data_ptr=0x7873) at ../../runtime/bin/thread_linux.cc:86
#11 0x00007fc515fea184 in start_thread (arg=0x7fc516607700) at pthread_create.c:312
#12 0x00007fc5152ef37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

This dump is at:
gs://dart-temp-crash-archive/94ae66d1-f883-41ba-86ce-a51f50dbf607/release_x64_dart_precompiled_runtime.tar.gz
gs://dart-temp-crash-archive/94ae66d1-f883-41ba-86ce-a51f50dbf607/core.30835.tar.gz

uploaded https://codereview.chromium.org/2678313003/ for review


Optimize CSS z-ordering

WebKit engineer here. CodeMirror tiggers some really inefficient code paths in browser engines related to z-ordering and CSS stacking context, which causes every line in the scrolled content to get multiple composed layers (i.e. GPU textures) in some cases. For example, here's a JSFiddle, snapshotted with Safari Web Inspector's "layer borders" turned on:

Here's the same content with <div class="CodeMirror-scroll" tabindex="-1"> made a CSS stacking context by adding z-index: 0 style:

What's happening here is that in its shipping config, CodeMirror is creating an overflow:scroll element which is not a CSS stacking context. Browsers optimize scrolling by promoting overflow:scroll to compositing layers, but in so doing, must also promote other elements which are higher in paint order to compositing layers. Because of the details of how the CSS painting model works and how browsers implement compositing, this can trigger every position:relative descendant to also get composited, which causes additional memory use and computation.
There's a really simple fix; make the CodeMirror-scroll element a CSS stacking context. You could use z-index: 0 or will-change: z-index.

I can't find a case where this breaks reasonable setups, so sure, added the z-index in attached patch.

I can't find a case where this breaks reasonable setups, so sure, added the z-index in attached patch.

@smfr awesome report - and thanks for all your hard work on WebKit! would love to know how much of a performance gain you're seeing from the patch.

@smfr awesome report - and thanks for all your hard work on WebKit! would love to know how much of a performance gain you're seeing from the patch.

This will approximately halve layer-related memory use for the CodeMirror in WebKit browsers. I don't have data on the runtime benefits, but would expect them to be noticeable for larger content.

This will approximately halve layer-related memory use for the CodeMirror in WebKit browsers. I don't have data on the runtime benefits, but would expect them to be noticeable for larger content.


parameter client_id and client_secret should be deleted, when grant_type is password

parameter client_id and client_secret should be deleted, when grant_type is password
https://tools.ietf.org/html/rfc6749#section-4.3
I got a error when i was calling oauth.token().
Invalid client: cannot retrieve client credentials
I'm using express-oauth-server 2.0.0. But I think, it's aboutoauth2-server 3.0.0.
options
router.oauth = new OAuthServer({
    model: memorystore,
    grants: ['password'],
    debug: true,
    useErrorHandler: true,
    continueMiddleware: true,
    requireClientAuthentication: { password: false }
});

request body
grant_type=password&username=admin&password=admin

I'm using express-oauth-server 2.0.0. But I think, it's about oauth2-server 3.0.0.

Could you clarify what you mean by that?
The server option requireClientAuthentication is available starting with 3.0. If you're using 2.0 as you say that's probably why it's not working ;)

Now, I'm sure the bugs is at oauth2-server 3.0.0, not at express-oauth-server 2.0.0.
rfc6749#section-4.3.2 This document defined the format of request. And for example here:

 POST /token HTTP/1.1
 Host: server.example.com
 Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
 Content-Type: application/x-www-form-urlencoded

 grant_type=password&username=johndoe&password=A3ddj3w

I call the function oauth.token().
When it creating a new object of TokenHandler, it always call the function getClient(). (line 90 at token-handler.js, caller by line 75 at server.js)
getClient() calling getClientCredentials() (line 116 at token-handler.js)
request.body.client_id and request.body.client_secret is undefined. That can make an exception.
Because there is not have client_id and client_secret by password.

Look at the Authorization header. In your example from rfc, it's
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW

After base64 decode it's
s6BhdRkqt3:gX1fBat3bV

So in this example, client_id is s6BhdRkqt3 and client_secret is gX1fBat3bV

+1, same here.
Can I do request to get token just with username and password without Authorization header?

This Auth header is optional in case:
If the client type is confidential or the client was issued client
credentials (or assigned other authentication requirements), the
client MUST authenticate with the authorization server as described
in Section 3.2.1.

There is no much a different here..
if I use requireClientAuthentication, the oauth2 will get the Authorization from the body instead of header. like so:
  if (!this.isClientAuthenticationRequired(grantType)) {
    if(request.body.client_id) {
      return { clientId: request.body.client_id };
    }
  }

So are people abandoning this project? Grant password doesn't seem to work. Anyone care to fork it?

This helps me:
// Model
module.exports.getClient = function *(clientId, clientSecret) {
return { clientId, clientSecret, grants: ['password', 'refresh_token'] };
};

Try adding requireClientAuthentication and set it to false and then pass a generic name for your clientId when using the password grant
   this.oauthServer = new OAuth2Server({
            model: require('./shared/oauth.model-ts'),
            requireClientAuthentication: {password: false}
        });

what I was wrong is the example using param 'clientId' and 'clientSecret' for body.
now worked with using client_id and client_secret as mention @wizardnet972  .
thank you
url: https://github.com/pedroetb/node-oauth2-server-example
bonus:

Try adding requireClientAuthentication and set it to false and then pass a generic name (my-app) for your clientId when using the password grant
   this.oauthServer = new OAuth2Server({
            model: require('./shared/oauth.model-ts'),
            requireClientAuthentication: {password: false}
        });

This worked for me!


change np.int to np.int32

Fixes https://github.com/linkedin/FastTreeSHAP/issues/20
and other problems with np.int, np.float deprecated in NumPy 1.20

reviews https://github.com/linkedin/FastTreeSHAP/issues/20

Thanks @julnow for pointing out the outdated numpy aliases issue and making this PR! As discussed in another PR https://github.com/linkedin/FastTreeSHAP/pull/21, we prefer removing the regular builtins over using the specific numpy scalar type. Therefore we decide to merge PR https://github.com/linkedin/FastTreeSHAP/pull/21 and close this PR.


sqlFile ignores encoding parameter

Environment
Liquibase Version: 4.2.2 (seems like all 4.x versions have this behaviour)
Liquibase Integration & Version: CLI
Database Vendor & Version: PostgreSQL 12.4
Operating System Type & Version: Windows 10
Description
sqlFile always uses system encoding instead of encoding specified by change parameter.
Additional Context
Look at the source code of SQLFileChange: encoding field is not used when reading SQL source code:
                String content = StreamUtil.readStreamAsString(sqlStream);
In fact, encoding field is not used anywhere.
This problem manifests only when default encoding is not UTF-8. There are other suspicious calls of StreamUtil.readStreamAsString without encoding parameter, which can lead to other bugs.

1 on this

Hi @osobolev Thanks for writing about this issue.  If you have a suggested fix for adding encoding, it would be great if you could add a PR with the fix and we will process add it to the list of issues to process.

duplicated by https://github.com/liquibase/liquibase/issues/1760
'encoding' was used in 3.x - just see history

Hi all.
Still waiting for a fix to have encoding managed when sqlFile is used.
With last version 4.5.0 it is still not fixed.
Very bad news for applications having accents to be displayed, or managed
It is planned to be fixed or not ?
Thanks
Regards

Looks like its no longer an issue on 4.14.  The way I verified this was:
Having a ñ on my sqlfile, using encoding="ascii" and it failed (because ñ is not in ascii table).
If you still have the same issue as before, please repon the issue with the changeset that is failing if possible. Thanks!


Set --frozen-lockfile for yarn install on CI

To catch situations where a person (or a tool like renovate) hasn't updated yarn.lock before sending a PR
Tested by temporarily having a commit on this branch where package.json and yarn.lock don't match: https://app.circleci.com/pipelines/github/coda/packs-sdk?branch=dweitzman-yarn-frozen

Yeah, this came to my attention now because the renovate PRs are failing to update yarn.lock but still passed CI testing


Magnet card only gives to inventory not to me system

Latest version of this mod and ae2 and forge.
Wether I use the option to get it in inventory or me system it always goes to inventory and im not using any other magnets.

I cannot reproduce it.
a) which minecraft version are you using (1.19.2 or 1.18.2)
b) do you have the filter set (pickup and insert to me have independent filters)
c) can you insert the item manually?
potential problems may be:

the terminal can't connect to the system (system too far away, or not loaded)
the item can't be inserted (i.e. if the system is full)

It seems gui button would not actually change Magnet mode.
And Magnet Card doesn't work with AE2 build-in terminal on Forge side0
Magnet Mode only changed with hotkey.

And Magnet Card doesn't work with AE2 build-in terminal on Forge side.

which one? It doesn't work with the normal wireless terminal, only with the wireless crafting terminal

It seems gui button would not actually change Magnet mode.

I can reproduce this on 1.18.2 now

which one? It doesn't work with the normal wireless terminal, only with the wireless crafting terminal

Oh okay, now I get it!

the fix for this would be rather complicated, and since this only happens on forge 1.18.2 (which is near the end of support anyway), I won't fix this.
as a workaround, use the hotkey instead.


Readme states wrong arguments for 'before.lory.slide' event

The readme file states that the arguments of the  'before.lory.slide' event are

currentSlide, nextSlide

but actually they are

index, nextSlide

Thanks! I've struggled with this issue for hours until I found this post and change my e.detail.currentslide to e.detail.index specifically for before.lory.slide.

Also when infinite is false, still wrong values are returned. NextSlide shows -1 on "next" click.


Signature Help height too short (need to scroll)

I am not sure if this is SublimeText or LSP. The popup cuts off after about two lines.

Ubuntu, ccls
Package Control or git (both do the same)
Create a function with many long parameters.

possible duplicate of #486

Closing duplicate, please follow the linked issue  for updates :)


Appstudio update test-component-pac-frvw

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-frvw-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
13 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
19 seconds
sanity-inspect-image

✅ Succeeded
24 seconds
deprecated-base-image-check

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
53 seconds
clamav-scan

✅ Succeeded
17 seconds
sbom-json-check

✅ Succeeded
9 seconds
sanity-label-check

✅ Succeeded
9 seconds
sanity-optional-label-check

✅ Succeeded
7 seconds
show-summary

Pipelines as Code CI/test-component-pac-frvw-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
8 seconds
init

✅ Succeeded
20 seconds
clone-repository

✅ Succeeded
32 seconds
build-container

✅ Succeeded
15 seconds
deprecated-base-image-check

✅ Succeeded
12 seconds
sanity-inspect-image

✅ Succeeded
34 seconds
clamav-scan

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
9 seconds
sbom-json-check

✅ Succeeded
7 seconds
sanity-optional-label-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
7 seconds
show-summary


reCaptcha for Security-Question Based Password Recovery NOT WORKING

Describe the issue:
reCaptcha(v2) for Security-Question Based Password Recovery NOT WORKING

How to reproduce:

Create a user with Login permission and security questions in Management Console
Go to Myaccount and Forget password option.
Give username and Recover with Security Questions option
This page should display reCaptcha if answers is incorrect

Expected behavior:

Environment information

Product Version: IS 5.11.0-Beta4
OS: Linux
Database: H2
Userstore: LDAP

For Recaptcha to work in security question-based password recovery we need to enable account locking. This is the current expectation.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


PHP code sample

Hi @kalpitshah2311 , are you still working on this PR ?

Hi @kalpitshah2311 , are you still working on this PR ?

I am sorry I can't continue with this pull request further. kindly unassigned this issue


Even more __index call reduction

Same thing as #3006; this just reduces ent:__index calls in a number of functions that I didn't catch in that PR.

I still don't understand why that local is being assigned in the Initialize function

It just doesn't work whatsoever when you try to get the baseclass's function before the entity is initialized.