feat: add a "dbdev create" command

Chore
Describe the chore
(This one is up for debate but I think it would be useful especially for testing environments like GH actions)
The "dbdev install" command adds the TLE to the database but doesn't "enable" the extension. It could be useful to have a command like dbdev create pg_idkit --connection " which essentially runs:
CREATE EXTENSION pg_idkit;
Additional context
Add any other context or screenshots that help clarify the task.

How about controlling this behavior on install with a flag --create-extension on the existing install command
e.g. dbdev install ... --create-extension rather than a separating it?

also good!
we might want something for updates too (alter extension ... update to NEW_VERSION);, although I'm not sure how far we need to go down this path - this might also mean that we need something to list the current version


Scratchpad blur when nothing is in scratchpad

Hyprland Version
Hyprland, built from branch  at commit 0a78f6031c8fcc6773f136fccde3d560aaac239d dirty (). Tag: v0.29.0
Bug or Regression?
Bug
Description
When bringing up the scratchpad/specialworkspace even though it is empty the background still gets blurred out.
Expected behaviour:
Nothing should happen if the scratchpad is empty
How to reproduce
Toggle specialworkspace when it is empty
Crash reports, logs, images, videos
No response

it's on purpose like that, so you see it's open.


Adds bindings for spoke users on hub

This commit adds changes that binds the hub's clusterrole to generated
spoke users.
This is done to fix a bug where the tokenagent on the spokeclusters are
unable to watch and make channges to mirrorpeer on the hub
Signed-off-by: Vineet Badrinath<EMAIL_ADDRESS>

LGTM, tested it out and it's working

/cherry-pick release-4.10

/test integration-test

/cherry-pick release-4.11


Cloakquest3r Refactor Suggestion/Review

While browsing through LinkedIn I found this tool and it was very useful to me so I thought "why not?" and I've decided to refactor this code while trying to improve my Python skills and helping the community somehow.
Before opening a new Pull Request I'd rather ask to the authors their opinion on the actual code as I don't expect you to accept those changes without complaints or questions. So here's a breakthrough of the changes.
:warning: Disclaimer: I'm not good at coloring outputs, and I find the methods to do so extremely irritating so I didn't used many colors, but this can always be updated later on.
:warning: Disclaimer #2: I'm also not the best programmer ever, but I've tried my best to facilitate people's life, this does include the author of the code.
:spiral_notepad: Changelog:
I've made a couple of changes, so I'm splitting this in different sections.
File Structure
I've modified the project's folder structure. Having a unique and extremely big file containing different functions and parts of the code can become a real mess sometimes and, unfortunately, this is the case.
Previous Structure
CloakQuest3r/
│
├── cloakquest3r.py
├── requirements.txt
└── wordlist.txt

Updated Structure
CloakQuest3r/
│
├── cloakquest3r.py
├── requirements.txt
├── config.toml				# Author information/links, version tag & API keys
├── core/					# Core functionalities of the code
│	├── __init__.py
│	├── cloakquester.py		# Main functions
│	├── banner.py
│	└── color.py
├── docs/					# Documentation related files
│	└── LICENSE
└── wordlists/				# Out-of-the-box wordlists
	└── default.txt

Splitting the code into distinct sections enhances organization and makes it easier to find specific code segments when updating or adding new features. Cloakquest3r has been modularized, with its own dedicated module.

The banner.py contains functions related to banner and social media output, as this is not relevant for the main code to work, this allows the author to easily change the visual and/or colors as desired without touching the main code.
The config.toml file stores API keys and also other relevant information about the author and his social media links and code version.
The color.py file stores colored output related functions, to avoid unnecessary lines of code in the main file.
The default out-of-the-box tool wordlist is included inside the wordlists folder.
Templates, license, and other documentation should be stored inside the docs folder.

Code Changes
Previously created functions have been refactored, rewritten and/or renamed for simplicity of code as needed. Some functions and variables names have become clearer and shorter.

Reduced is_using_cloudflare and detect_web_server to a single function, since both do basically the same thing. The code now verifies wheter the target is behind Cloudflare before doing anything else, and only asks for confirmation if the code isn't.
The code now verifies both if both URL and wordlist file are valid. Both have their own functions for validation: is_valid_url and is_valid_file.
Two functions have been added _to_url and _to_hostname. Those basically convert the string as some functions require the URL schema (https://) while others don't.

Features
I've added some features that I thought would help users and also trying to add some features mentioned in the Contribution section of the README.md file.

Added the -w/--wordlist flag, allowing the user to set a custom wordlist for subdomain discovery.
Added the --no-bruteforce option to avoid subdomain discovery through bruteforcing.
Added the --no-security-trail option to avoid Security Trails usage for IP history discovery,
Added the --no-banner option, as this can be useful while piping the output to an external file.
Added the --force option to ignore wheter the target is using cloudflare of not.
Added a help page -h/--help using argparse module.

TODO:
Some changes are still in progress and may or may be not continued based on the author's opinion on the code.

Adding docstrings to functions, some functions already have this included but I was too lazy to add them all.
Adding typing to functions. Same as above, too lazy to add all.
More color to printed texts. Adding colors can be an issue due to the fact a simple print function can become hard to understand in the code with the brackets everywhere.
Turn this into a PyPI package?

Final Considerations
My only objectives is to help and collaborate to the code somehow while improving my coding skills, I don't expect those changes to be accepted without complaints. I'm willing to listen and react to feedback from the code author.
The refactored code is available here: https://github.com/JLowborn/CloakquesterRedone

Oh wow!! you rewrote almost the whole thing. I haven't tested all the changes yet. Quoting your comments, here's how I feel about them.

I've modified the project's folder structure. Having a unique and extremely big file containing different functions and parts of the code can become a real mess sometimes and, unfortunately, this is the case.

- I created this tool as a proof of concept I know it's not well-written which is why it just has a single file.
I am NO programmer.

Updated Structure
CloakQuest3r/
│
├── cloakquest3r.py
├── requirements.txt
├── config.toml				# Author information/links, version tag & API keys
├── core/					# Core functionalities of the code
│	├── __init__.py
│	├── cloakquester.py		# Main functions
│	├── banner.py
│	└── color.py
├── docs/					# Documentation related files
│	└── LICENSE
└── wordlists/				# Out-of-the-box wordlists
	└── default.txt

+ I like it, it’s clean.

Splitting the code into distinct sections enhances organization and makes it easier to find specific code segments when updating or adding new features. Cloakquest3r has been modularized, with its own dedicated module.

The banner.py contains functions related to banner and social media output, as this is not relevant for the main code to work, this allows the author to easily change the visual and/or colors as desired without touching the main code.
The config.toml file stores API keys and also other relevant information about the author and his social media links and code version.
The color.py file stores colored output related functions, to avoid unnecessary lines of code in the main file.
The default out-of-the-box tool wordlist is included inside the wordlists folder.
Templates, license, and other documentation should be stored inside the docs folder.

Code Changes
Previously created functions have been refactored, rewritten and/or renamed for simplicity of code as needed. Some functions and variables names have become clearer and shorter.

Reduced is_using_cloudflare and detect_web_server to a single function, since both do basically the same thing. The code now verifies wheter the target is behind Cloudflare before doing anything else, and only asks for confirmation if the code isn't.
The code now verifies both if both URL and wordlist file are valid. Both have their own functions for validation: is_valid_url and is_valid_file.
Two functions have been added _to_url and _to_hostname. Those basically convert the string as some functions require the URL schema (https://) while others don't.

- not a fan of this, it should accept the domain as a valid argument, this was the first error I got when I ran the tool forcing me to add HTTP/HTTPS

u can use something like this ?

if input_text.startswith("http://") or input_text.startswith("https://"):
    match = re.search(r'(https?://)?([A-Za-z_0-9.-]+).*', input_text)
    if match:
        link = match.group(2)
    else:
        print("Invalid URL format.")
        exit(1)

.. or u can just check here if the status code is 200 of the domain

Features
I've added some features that I thought would help users and also trying to add some features mentioned in the Contribution section of the README.md file.

Added the -w/--wordlist flag, allowing the user to set a custom wordlist for subdomain discovery.
\

- this feature was added in the last update
it asks the user for a custom wordlist before the subdomain scan,
if the user says yes it asks for a wordlist path if NO it downloads an updated wordlist(5000 subdomain)from SECLIST GitHub repo and use it..
the default wordlist just a backup which has only (700 subdomains)

go through the last update this was missing in ur code, it just uses default wordlist.

Added the --no-bruteforce option to avoid subdomain discovery through bruteforcing.

+ bruteforcing is the only way we are doing subdomain scanning which is not an efficient way we miss some subdomains, i have a new update which if u want u can add ...

import subprocess

PURPLE = '\033[1;30m'
RED = '\033[1;31m'
GREEN = '\033[1;32m'
YELLOW = '\033[1;33m'
BLUE = '\033[1;34m'
PINK = '\033[1;35m'
LBLUE = '\033[1;36m'
WHITE = '\033[1;37m'

def execute_command(command):
    subprocess.run(command, shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

url = input(f"\n{WHITE}[!] Enter domain to Enumerate subdomains: ")

print(f"{BLUE}\n[*] Subdomain Enumeration started!\n")

print(f"{GREEN}\n[+] Enumerating subdomains from subfinder ..")
execute_command(f"subfinder -d {url} -silent > sub1")
print(f"{GREEN}\n[+] Enumerating subdomains from assetfinder ..")
execute_command(f"assetfinder {url} > sub2")
print(f"{GREEN}\n[+] Enumerating subdomains from crt.sh ..")
execute_command(f"curl -s 'https://crt.sh/?q={url}' | grep '' | grep {url} | cut -d '>' -f2 | cut -d '<' -f1 | sort -u | sed '/^*/d' > sub3")
print(f"{GREEN}\n[+] Enumerating subdomains from rapiddns ..")
execute_command(f"curl -s 'https://rapiddns.io/subdomain/{url}#result' | grep ' sub4")
print(f"{GREEN}\n[+] Enumerating subdomains from bufferover ..")
execute_command(f"curl -s 'https://dns.bufferover.run/dns?q=.{url}' | jq -r .FDNS_A[] | cut -d '\\' -f2 | cut -d ',' -f2 |  sort -u > sub5")
print(f"{GREEN}\n[+] Enumerating subdomains from ridder ..")
execute_command(f"curl -s 'https://riddler.io/search/exportcsv?q=pld:{url}' | grep -Po '(([\w.-]*)\.([\w]*)\.([A-z]))\w+' | sort -u > sub6")
print(f"{GREEN}\n[+] Enumerating subdomains from jldc ..")
execute_command(f"curl -s 'https://jldc.me/anubis/subdomains/{url}' | grep -Po '((http|https)://)?(([\w.-]*)\.([\w]*)\.([A-z]))\w+' | cut -d '/' -f3 > sub7")
print(f"{GREEN}\n[+] Enumerating subdomains from omnisint ..")
execute_command(f"curl -s 'https://sonar.omnisint.io/subdomains/{url}' | cut -d '[' -f1 | cut -d ']' -f1 | cut -d '\"' -f 2 > sub8")

execute_command(f"sort sub1 sub2 sub3 sub4 sub5 sub6 sub7 sub8 | uniq | tee {url}-all_subdomains")
execute_command("rm sub*")

print(f"{BLUE}\n[*] Subdomain Enumeration Completed!\n")
num = subprocess.check_output(f"wc -l {url}-all_subdomains | awk '{{print $1; exit}}'", shell=True, text=True).strip()
print(f"{WHITE}\n[*] Found {num} subdomains for {url}\n")

print(f"{BLUE}\n[!] view the {url}-all_subdomains file for results!\n\n")

Added the --no-security-trail option to avoid Security Trails usage for IP history discovery,
Added the --no-banner option, as this can be useful while piping the output to an external file.
Added the --force option to ignore wheter the target is using cloudflare of not.
Added a help page -h/--help using argparse module.

- simply python cloakquest3r.py should show the help menu, we can also keep -h/--help

also, I didn’t like the -u URL arg, it can be there but .. it should also work the way it is now without any argument simply: python cloakquest3r.py example.com

TODO:
Some changes are still in progress and may or may be not continued based on the author's opinion on the code.

Adding docstrings to functions, some functions already have this included but I was too lazy to add them all.
Adding typing to functions. Same as above, too lazy to add all.
More color to printed texts. Adding colors can be an issue due to the fact a simple print function can become hard to understand in the code with the brackets everywhere.

- Please don’t change the output structure and colour output,
keep all the colours and the structure. I like colours and am obsessed with making things look pretty lol.
Don’t change anything there.

Turn this into a PyPI package?

- I don’t think it would be that useful to make the PyPI package !!

+ # Useful things u can add:
1. save all output in text file in the end.
2. add proxy list support, using the tool without a VPN can get ur IP banned by VIEWDNS,
because we are scrapping data from the site which is not good practice,
adding proxy support would be good improvement.

Final Considerations
My only objectives is to help and collaborate to the code somehow while improving my coding skills, I don't expect those changes to be accepted without complaints. I'm willing to listen and react to feedback from the code author.
The refactored code is available here: https://github.com/JLowborn/CloakquesterRedone

+ I am open to merging after these changes,
just make sure you don’t change the tool output look,
the tool should also work the way it is currently
I mean it should be able to also run without all the new arguments u added,
those arguments are very nice if wanna do something specific but I also like to keep things simple.
With this in mind, feel free to make changes as you see fit to improve the tool.

P.S. I am not a pro coder, so please don't take offense to my criticism. I just want to make sure it's how I like it.

About the last update, I just saw yesterday after opening this issue, I've started the refactor at least one week before.
Not offended at all, I'm glad you've took your time to read and reply to this, I'm willing to help in any improvements. 😄
The colored output is not a problem for me, I'm still thinking of a way to maintain the color scheme while facilitating the coding, until then, I think it's better to keep the colors as they were before.
The improvements are not complete, just wanted to ask your honest opinion, so thanks again for your time and patience.
I'll keep you updated about any updates or ideias!

Looking forward to seeing the progress and staying updated on any new ideas or updates. Keep up the great work! (^^)

While reading the updated code, I saw you've added a feature to download a wordlist from SecLists. Is it really necessary? I mean, if the wordlist is going to be used by the tool, a better option is to add it to the wordlists folder and use it as default in case no wordlist flag has been supplied by the user. What's your opinion on that?
By the way, I'm reversing the prints and adding the colors once more, also removed the --url flag as it wasn't really necessary and it now also accepts hostnames instead of just URLs.

It's a good idea to download the Wordlist from Seclist's GitHub repository because it's updated frequently. If you're using a Wordlist for subdomain scanning, it's essential to have an updated Wordlist to work with. However, it's not advisable to copy and paste the Wordlist from their repository to yours as it doesn't look professional, and the Wordlist won't receive any new updates.
I am not a fan of brute force subdomain scanning. It's not foolproof. We should keep this brute force option, but also provide better methods for subdomain scanning like this one.
import subprocess

PURPLE = '\033[1;30m'
RED = '\033[1;31m'
GREEN = '\033[1;32m'
YELLOW = '\033[1;33m'
BLUE = '\033[1;34m'
PINK = '\033[1;35m'
LBLUE = '\033[1;36m'
WHITE = '\033[1;37m'

def execute_command(command):
    subprocess.run(command, shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

url = input(f"\n{WHITE}[!] Enter domain to Enumerate subdomains: ")

print(f"{BLUE}\n[*] Subdomain Enumeration started!\n")

print(f"{GREEN}\n[+] Enumerating subdomains from subfinder ..")
execute_command(f"subfinder -d {url} -silent > sub1")
print(f"{GREEN}\n[+] Enumerating subdomains from assetfinder ..")
execute_command(f"assetfinder {url} > sub2")
print(f"{GREEN}\n[+] Enumerating subdomains from crt.sh ..")
execute_command(f"curl -s 'https://crt.sh/?q={url}' | grep '' | grep {url} | cut -d '>' -f2 | cut -d '<' -f1 | sort -u | sed '/^*/d' > sub3")
print(f"{GREEN}\n[+] Enumerating subdomains from rapiddns ..")
execute_command(f"curl -s 'https://rapiddns.io/subdomain/{url}#result' | grep ' sub4")
print(f"{GREEN}\n[+] Enumerating subdomains from bufferover ..")
execute_command(f"curl -s 'https://dns.bufferover.run/dns?q=.{url}' | jq -r .FDNS_A[] | cut -d '\\' -f2 | cut -d ',' -f2 |  sort -u > sub5")
print(f"{GREEN}\n[+] Enumerating subdomains from ridder ..")
execute_command(f"curl -s 'https://riddler.io/search/exportcsv?q=pld:{url}' | grep -Po '(([\w.-]*)\.([\w]*)\.([A-z]))\w+' | sort -u > sub6")
print(f"{GREEN}\n[+] Enumerating subdomains from jldc ..")
execute_command(f"curl -s 'https://jldc.me/anubis/subdomains/{url}' | grep -Po '((http|https)://)?(([\w.-]*)\.([\w]*)\.([A-z]))\w+' | cut -d '/' -f3 > sub7")
print(f"{GREEN}\n[+] Enumerating subdomains from omnisint ..")
execute_command(f"curl -s 'https://sonar.omnisint.io/subdomains/{url}' | cut -d '[' -f1 | cut -d ']' -f1 | cut -d '\"' -f 2 > sub8")

execute_command(f"sort sub1 sub2 sub3 sub4 sub5 sub6 sub7 sub8 | uniq | tee {url}-all_subdomains")
execute_command("rm sub*")

print(f"{BLUE}\n[*] Subdomain Enumeration Completed!\n")
num = subprocess.check_output(f"wc -l {url}-all_subdomains | awk '{{print $1; exit}}'", shell=True, text=True).strip()
print(f"{WHITE}\n[*] Found {num} subdomains for {url}\n")

print(f"{BLUE}\n[!] view the {url}-all_subdomains file for results!\n\n")

"It's a good idea to download the Wordlist from Seclist's GitHub repository because it's updated frequently."

The last update on that wordlist that's being downloaded was 4 years ago.

"it's not advisable to copy and paste the Wordlist from their repository to yours as it doesn't look professional."

SecLists itself is a compilation of popular wordlists, it's a common thing to share resources in cybersecurity, even SecLists itself has copies of other tool's wordlists, such as JohnTheRipper (those are not being updated neither).

Having an "updated" wordlists consists of constantly checking which are the most common and popular subdomain names, still, it doesn't have to be done every single time, it can be done once upon a semester or a year. Besides, the best practice in real  world scenarios is studying a target since you won't find, for example, a booking subdomain if you're scanning an e-commerce.

We should keep this brute force option, but also provide better methods for subdomain scanning like this one.

I think you misunderstood me, I don't think bruteforcing is a bad ideia. When I've mentioned early I've implemented a --no-bruteforce option, the reason was based on previous experiences using different tools that had a slow performance because of bruteforcing, as it wasn't optional, every time I used the tool, I had to accept the fact that it was going to take a really long time before I had results.
I'll make sure to implement this solution as well.

"Has it really been four years? 😂  I can't believe I never noticed the last commit. Anyway, you can proceed with your preferred method. You can keep multiple wordlists in the repository, with the default being the small-wordlist. I recommend using 5000 as the default. I tested the subdomain scan code a while back and found that it sometimes breaks if the wordlist is too long. But anything under 5k should work just fine."

I've added the wordlist to the folder, and also splitted this wordlist in 5 other files with different line counts.

Make sure you test those wordlists which are bigger than 5000, as I mentioned earlier it sometimes breaks if the wordlist is too long, also do we need this many wordlists?
we should just keep 3

Default one which is already in repo
5000 from Seclist
Biggest one

We should just keep 3

Default one which is already in repo
5000 from Seclist
Biggest one

It's okay, that's a good ideia!


test: investigate and document all methods that use exit fee to futher unit-test

Background
Most of our test cases use 0 exit fees. We should ensure that we also test with non-zero where applicable.
Acceptance Criteria

Compile a list of all methods that use/take exit fee. That includes all pools in x/gamm/pool-models
Create issues for unit testing each

Also add tests for hand derived values

Exit fee is now disabled. This task is obsolete


Comparison with similar libraries

What does kapp do differently than alternatives like winit or (probably the most similar) miniquad?

Good question!
kapp aims to:

Build as quick as reasonably possible (it clean builds in 1/4th the time of winit)
Correctly implement little, often neglected, details like:

Smooth window resizing
Monitor color spaces

Be very simple architecturally

It's not (yet) trying to be as robust and cross-platform as winit, but it's also not as focused and simple as miniquad, which is tightly coupled to GL. kapp is a middleground between the two.
If you compare the three today kapp is less mature. kapp has clearly missing features and numerous small bugs. kapp also doesn't support Linux yet, which is important to much of the Rust community. Sometime next year, when kapp is a little further along, I plan to release an initial version on crates.io.


feat(text): adds semibold to the public api for Text

To support then design system more - adding semi bold to the font weights allowed by the Text component.

:tada: This PR is included in version 4.12.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


feat(form): Parse URI field upon configuring step

To ease testing, this pull request is deployed here
Context
Currently, we're not parsing the URI field from the from, to, and the toD Camel processors.
Changes
This pull request adds support for the URI field in the following fashion:
Considering the following URI: timer:timer-1?period=5000&delay=5&synchronous=true, when configuring the timer component, we parse the URI using the component's syntax (in this case is timer:timerName) and set those values in the parameter object. After the parsing, the URI field is cleaned to avoid overriding the parameter's value again. The result of this parsing is: { timerName: 'timer-1', period: 5000, delay: 5, synchronous: true }.
In addition to that, we're hiding the URI field to prevent the user from manipulating it for the time being. This decision needs to be revisited later to check how can dynamic expressions be supported.
In a more practical example, given this yaml route example:
- from:
    uri: timer:timer-1?period=5000&delay=5&synchronous=true
    steps: []

after configuring the timer component, we'll end up with the following route:
- route:
    from:
      uri: timer
      steps: []
      parameters:
        period: 5000
        delay: "50"
        synchronous: true
    id: route-3338

fix: https://github.com/KaotoIO/kaoto-next/issues/274

Would it be possible to group the parameters just below the uri field instead of placing them at the end?

Yes, it's possible, we'll do that as part of the serialization improvements.

path parameter are visible only the first time that we open the config form:

Thanks, I'll check it out right away


[dev] Make HMR work on /inquiries.

Need to not only export the Relay container, but also the component itself.
See https://github.com/alloy/relational-theory/commit/2cc900ea216f61c7a4a8096bb9177a648b135129

Is this something we should make  danger rule for in the future? In Emission there's some per-component based rules already

Ooh, yeah, I like that 👌

@orta Would the rule just be export all React components by default?
I feel like it'd be too much work to check that component is wrapped in a Relay container somewhere

it's code that doesn't exist yet, we can do whatever we want 👍


Drive doesn't work

When logging back into my world my system would not work so I took some drives out and the one with all my items doesn't work. If I have that drive in the system it will not show any items once I take it out of the system everything else will show.

USE THE TEMPLATE!
Also, check out #2356 and #2362


undefined error

Output:
Starting to copy table db_test
Error backing up undefined
undefined
Error backing up db_test
[Error: Cannot initiate transfer]
Done copying table db_test
Finished backup
Error backing up undefined
undefined
Error backing up db_test
[Error: Cannot initiate transfer]
Done copying table db_test
nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/request.js:30
throw err;
^
Error: Callback was already called.
at nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/async/lib/async.js:30:31
at nodejs/dynamodb/node_modules/dynamo-backup-to-s3/lib/dynamo-backup.js:119:21
at Uploader. (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/lib/dynamo-backup.js:63:16)
at Uploader.emit (events.js:107:17)
at Response. (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/lib/uploader.js:71:17)
at Request. (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/request.js:353:18)
at Request.callListeners (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/request.js:595:14)
at Request.transition (nodejs/dynamodb/node_modules/dynamo-backup-to-s3/node_modules/s3-streaming-upload/node_modules/aws-sdk/lib/request.js:21:10)
Script :
var DynamoBackup = require('dynamo-backup-to-s3');
var backup = new DynamoBackup({
includedTables: ['db_test'],
bucket: "dynamodb-backup",
awsAccessKey: "xxxxx",
awsSecretKey: "xxxxx",
awsRegion: "us-west-1"
});
backup.on('error', function(data) {
console.log('Error backing up ' + data.tableName);
console.log(data.error);
});
backup.on('start-backup', function(tableName) {
console.log('Starting to copy table ' + tableName);
});
backup.on('end-backup', function(tableName) {
console.log('Done copying table ' + tableName);
});
backup.backupAllTables(function() {
console.log("Finished backup");
});
Modules:-
└─┬<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ │ └──<EMAIL_ADDRESS>│ └──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ │ ├──<EMAIL_ADDRESS>│ │ ├──<EMAIL_ADDRESS>│ │ └──<EMAIL_ADDRESS>│ └─┬<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>

@aavileli - can you change the error handler to:
backup.on('error', function(data) {
  console.log('Error backing up ' + data.tableName);
  console.log(data);
});

and respond with the output?

@dylanlingelbach  I think there is some issue in the s3-streaming-upload it works some time some times does not so I moved away and implemented the solution using https://github.com/mapbox/dyno and some unix paralel to upload to s3 using shell

@aavileli - sorry to hear it wasn't working for you. I wasn't able to reproduce this, and I can't tell what is going on from the information provided.
I'll close for now, but feel free to reopen if you can provide more info here.


Auth current authenticated user enhancement

Issue #, if available:
To give the correct name in Greetings for Cognito hosted UI
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #713 into master will decrease coverage by 0.03%.
The diff coverage is 71.42%.

@@           Coverage Diff            @@
##           master   #713      +/-   ##
========================================
- Coverage   87.03%    87%   -0.04%
========================================
  Files          74     74
  Lines        3495   3500       +5
  Branches      668    671       +3
========================================
+ Hits         3042   3045       +3
- Misses        436    437       +1
- Partials       17     18       +1

Impacted Files
Coverage Δ

packages/aws-amplify/src/Auth/Auth.ts
88.56% <100%> (+0.03%)
:arrow_up:

packages/aws-amplify-react/src/Auth/Greetings.jsx
92.95% <50%> (-2.64%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 257a41a...77f26b1. Read the comment docs.


Create AoATech.netkan

You'll need to remove all those "+" symbols at the start of each line, and the identifier should match the filename.
Also, are you sure this isn't Mk2CockpitExpansion moved to a new Spacedock ID? It looks like it might be, and if so, it would be best to keep the old identifier/filename.

Closing in favour of #5098


SteamVR-OSVR fails if OSVR server isn't already running

Description of issue
It seems that if the server isn't already running then OSVRTrackedDevice::activate() simply returns VRInitError_Driver_Failed right away. SteamVR-OSVR hence needs the OSVR server to be started before SteamVR.
On the other hand, everything will work correctly if you restart the OSVR server after a correct initialization.
Steps to reproduce the problem
Start SteamVR.
Start the OSVR server.
See that the status doesn't get to ready.

Are you making a feature request so that vr games can be launched directly through steam desktop client running in large mode and without osvr server running when that launch was made?
We currently plug steamvr into osvr, that is why osvr server needs to be started to launch anything from steamvr.
We need to remember any app launched from steamvr is an openvr app I.e. not an osvr app
I suspect plugging osvr into steamvr by use of openvr drivers for osvr hdk would help removing the need to start osvr server manually before launching any openvr app from steam client.

Overview

The OpenVR API provides a game with a way to interact with Virtual Reality displays without relying on a specific hardware vendor's SDK. It can be updated independently of the game to add support for new hardware or software updates.
We would not need to load osvr runtime after configuring osvr server for hmd/vr devices being used before launching steamvr or openvr apps if osvr hdk had its own openvr driver that better implements openvr features.

Start OSVR
Start steamvr
Start steamvr app from steamvr dashboard or steam desktop client.
Think of osvr server needing started first as how steamvr knows to use osvr drivers.


support linting staged source files

Change-type: minor
Signed-off-by: Somo S<EMAIL_ADDRESS>

@balena-ci I self certify!


moment.js does not load from webpack

Describe the bug
When moment.js is installed into webpack, it is not available to any scripts. It is necessary to load moment.js from an external source before it becomes available.
To Reproduce
Steps to reproduce the behaviour:

npm install moment
webpack
Load page in browser

Expected behaviour
The console log should be empty.
Desktop (please complete the following information):

OS: all
Browser: all
Version: any

Additional context
angular-moment-picker.js:156
Uncaught ReferenceError: moment is not defined
    at Object.<anonymous> (angular-moment-picker.js:156)
    at __webpack_require__ (angular-moment-picker.js:21)
    at Object.<anonymous> (angular-moment-picker.js:207)
    at __webpack_require__ (angular-moment-picker.js:21)
    at Object.<anonymous> (angular-moment-picker.js:180)
    at __webpack_require__ (angular-moment-picker.js:21)
    at Object.<anonymous> (angular-moment-picker.js:1147)
    at __webpack_require__ (angular-moment-picker.js:21)
    at angular-moment-picker.js:67
    at Object.<anonymous> (angular-moment-picker.js:70)

Thanks for opening this request.
This is a great question for Stack Overflow.


Difficulties with setup

Trying to follow the setup instructions through as per the README does not seem straightforward as things stand. (Mac OS, Mojave).

Is there an instruction missing on creating a conda causal environment from the environment.yml? The python setup.py develop instructions given won't result in this will it?
It is conda that's expected to be used to manage environments?

Assumingconda env create -n causal environment.yml needed to be run, the following issues then came up during setup:

matplotlib version specified for the environment seem not compatible with the python install for the conda causal environment on the Mac:

ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

Workaround: This is odd, as I've used matplotlib and import matplotlib.pyplot as plt without issues on other environments. I'm not 100% sure what the details of the backend issues are, but relaxing the version specified in the environment file, and upgrading matplotlib (3.1.3 at time of writing), seemed to resolve.
2. The symlinks to @natashamjaques fork of ray seem incomplete, or at least the following errors occur when running tests:
ImportError while importing test module '<my-local-source-path>/sequential_social_dilemma_games/tests/test_envs.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_envs.py:15: in <module>
    from social_dilemmas.envs.cleanup import CleanupEnv
social_dilemmas/envs/cleanup.py:5: in <module>
    from social_dilemmas.envs.map_env import MapEnv, ACTIONS
social_dilemmas/envs/map_env.py:8: in <module>
    from ray.rllib.env import MultiAgentEnv
<my-user-directory>/anaconda3/envs/causal/lib/python3.6/site-packages/ray/rllib/__init__.py:11: in <module>
    from ray.rllib.evaluation.policy_graph import PolicyGraph
...
<my-user-directory>/anaconda3/envs/causal/lib/python3.6/site-packages/ray/rllib/evaluation/tf_policy_graph.py:12: in <module>
    import ray.experimental.tf_utils
E   ModuleNotFoundError: No module named 'ray.experimental.tf_utils'

I have been trying to add further module ray symlinks (e.g. for experimental) but it just seems to reveal more and more unresolvable dependencies. Are there any plans to integrate with a stable/supported release of ray instead of the fork? I assume there remain incompatibilities.
I would post a PR to update matplotlib if I could test the consequences. But I can't currently run tests.
May relate to https://github.com/eugenevinitsky/sequential_social_dilemma_games/issues/164, I'm not sure. That one seems to have different steps in setting up ray versions.

Just an update... Adding the experimental directory from ray's rllib did seem to fix the tests. I may post a PR to each, if this seems ok?

Hi, so yes, it does seem the matplotlib version is bad. That's a good catch. Yes, you are expected to use the conda env.
Unfortunately, as for the Ray version, there isn't a version of the causal code that I've 100% proven to work. There is a branch up that has an upgrade to Ray 0.7.6 (ppo_causal_fixes) which you're welcome to check out and try!


Support for Google Analytics 4 tags

Any new Google Analytics account is not able to create the previous 'universal' properties (UA-XXXXXX-Y, AW-XXXXXXXXXX, UA-XXXXXX-X, DC-XXXXXXXX) and instead now the tracking IDs look like (G-XXXXXXXXXX).
https://blog.google/products/marketingplatform/analytics/new_google_analytics
https://support.google.com/analytics/answer/10089681
As a result Django-analytical does no longer support for any of the new google analytics integrations.

Thanks, Carlos, for reporting this problem.
Can you suggest a specific change? Do you want to try a PR?

Hi, I hope to solve this problem.
but i am new to git contribution.
can you help me to solve it?

@likecodeingloveproblems It's good to hear that! Contributors are the best people around!
It's actually simple, as for the code change:

You'd have to adjust the regular expression in templatetags/google_analytics_gtag.py, so that it allows the patterns @carlosnasillo listed above.
And, of course, also align the related tests.

The test suite should then pass. Try running tox locally. It should all be green. Then you're ready to push your changes and open a pull request.
Mastering GitHub
If you've never made a contribution on GitHub (or GitLab or Bitbucket, and the like) there are a few things you need to learn, but the good news is there are also loads of nice tutorials that explain how to do it, e.g.

http://blog.davidecoppola.com/2016/11/howto-contribute-to-open-source-project-on-github/
https://www.dataschool.io/how-to-contribute-on-github/
https://guides.github.com/activities/forking/

Go forward, step by step! Come back here to ask for help if needed! :rocket: :+1: :100:

thanks
you are really kind
1.You'd have to adjust the regular expression in templatetags/google_analytics_gtag.py, so that it allows the patterns @carlosnasillo listed above.
it is already adjusted to regular expression "^G-[a-zA-Z0-9]+$"
2.And, of course, also align the related tests.
and all test by running tox passed

@carlosnasillo Can you clarify your statement?

As a result Django-analytical does no longer support for any of the new google analytics integrations.

A G-XXXX... ID seems to be accepted by django-analytical, see PROPERTY_ID_RE in google_analytics_gtag.py.

@bittner @likecodeingloveproblems
Back when I was updating my GA tag from G3 (ie. UA-XXXXXX-Y) to the new G4 (G-XXXXXXXXXX) it was not being parsed as 'correct' by the regular expression when using GOOGLE_ANALYTICS_JS_PROPERTY_ID.
See the PROPERTY_ID_RE for that.
I hope this helps.

Has this issue been resolved? I will like to contribute to this if it is still open.

If I'm not mistaken this issue is still unsolved. Feel free to open a PR.

Alright. I'll work on this


Can a custom ICcgDomainAuthCredentials be used to provide a non gMSA account for a pod?

My company infrastructure team currently does not support gMSA for dynamically provisioned servers.
Therefore I would like to run containers under AD service accounts managed by a 3rd party credential store.
My question is, can k8s be configured to run a pod under a cred that it gets from a custom ICcgDomainAuthCredentials. As opposed to getting a credential to then query AD for the gMSA creds.
The node is also going to be a non AD windows node.

Looping in @jsturtevant as this seems to be a K8s question.

If you build your own ccg plugin you can configure the pod with the hostAccountConfig value that uses the GUID of that plugin: https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/manage-serviceaccounts#additional-credential-spec-configuration-for-non-domain-joined-container-host-use-case.
The latest gmsa webhook release has support for this field as well: https://github.com/kubernetes-sigs/windows-gmsa/pull/47
For building your own ccg plugin you can use https://github.com/microsoft/Azure-Key-Vault-Plugin-gMSA as a reference

regardless of if the pod is in kubernetes or running locally the only way to interact with the domain from a Windows container perspective is to use the gMSA: https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/manage-serviceaccounts. I believe this is due to the way that passwords are managed with containers.

@arontsang did that answer you question?  can we close this issue or do you have further questions?


Fix Hadoop Downloader Range not correct

This fixes #20110

This fixes ValueError happens when reading a hdfs of file size larger than buffer size

Added related unit test

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: "addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment "fixes #" instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

retest this please

R: @chamikaramj

retest this please

Thanks for the quick fix.
I can merge when tests pass.

Run Python PreCommit

Run Python PreCommit

Run Python PreCommit


Cannot parse Nan (but NULL in TabSeparated format)

:) CREATE TABLE hello_world.hello_world_table(f1 Nullable(String),f2 Nullable(Float64),f3 Nullable(Int64),f4 Nullable(DateTime),f5 Nullable(Float64)) ENGINE = TinyLog;
CREATE TABLE hello_world.hello_world_table
(
f1 Nullable(String),
f2 Nullable(Float64),
f3 Nullable(Int64),
f4 Nullable(DateTime),
f5 Nullable(Float64)
) ENGINE = TinyLog
Ok.
:) INSERT INTO hello_world.hello_world_table(f2,f5,f3,f1,f4) FORMAT TabSeparated NULL    7.81651003481571E8<PHONE_NUMBER>      zlwwqlbkvrksdt  NULL
INSERT INTO hello_world.hello_world_table (f2, f5, f3, f1, f4) FORMAT TabSeparated
Exception on client:
Code: 27. DB::Exception: Cannot parse NaN.: (at row 1)
Row 1:
Column 0,   name: f2, type: Nullable(Float64),  parsed text: "N"ERROR
Connecting to localhost:9000.
Connected to ClickHouse server version 1.1.54189.
FORMAT TabSeparated is default and hardcoded in Clickhouse Jdbc.

You can use \N for NULL values.

Do you have an example?
:) INSERT INTO hello_world.hello_world_table(f2,f5,f3,f1,f4) FORMAT TabSeparated \N 7.81651003481571E8<PHONE_NUMBER> zlwwqlbkvrksdt \N;
INSERT INTO hello_world.hello_world_table (f2, f5, f3, f1, f4) FORMAT TabSeparated
Exception on client:
Code: 27. DB::Exception: Cannot parse input: expected \t before:  7.81651003481571E8<PHONE_NUMBER> zlwwqlbkvrksdt \N;: (at row 1)
Row 1:
Column 0,   name: f2, type: Nullable(Float64),  parsed text: "N"
ERROR: garbage after Nullable(Float64): " 7.8165100"
Connecting to localhost:9000.
Connected to ClickHouse server version 1.1.54189.

Example data:
vjebywyypgqorighfaw,,898999574,,3.008874029012975E8
tjqevznxclm,,2079703005,,1.1591825521454709E9
aawethajnahzxgxzzdu,,1461070703,,7.682462476872058E8
feubrhhqaja,,1078610460,,8.908589890974745E8
hnjurnrlpcjxfa,,469686467,,6.0313867905135885E7
bqzqaphgvjj,,1133548469,,1.046040289828672E8
zcpozsgfctj,,949290936,,1.710776620861809E9

What choises are available to import it?

Sure:
:) create table t1(f1 Nullable(String),f2 Nullable(Float64),f3 Nullable(Int64),f4 Nullable(DateTime),f5 Nullable(Float64)) engine = TinyLog;

CREATE TABLE t1
(
    f1 Nullable(String),
    f2 Nullable(Float64),
    f3 Nullable(Int64),
    f4 Nullable(DateTime),
    f5 Nullable(Float64)
) ENGINE = TinyLog

Ok.

0 rows in set. Elapsed: 0.015 sec.

:) insert into t1(f2,f5,f3,f1,f4) FORMAT TSV \N 7.8     1111    aaa     \N

INSERT INTO t1 (f2, f5, f3, f1, f4) FORMAT TSV

Ok.

1 rows in set. Elapsed: 0.001 sec.

:)

The problem is with ';' character: because the remainder of the query after 'FORMAT TSV' is fed into the TabSeparated parser as is, the semicolon is parsed as part of the data, not as the end of query.

What choises are available to import it?

CSV parser does not recognize empty string as a NULL value, so you will have to manually insert '\N' where you want NULLs to be.

Thanks for your help.

We should implement empty_as_null, null_string_as_null and similar options.


versewise navigation with arrow keys.

Fixed versewise navigation.

@samueljd, some major issues I come a cross please watch the video: https://www.awesomescreenshot.com/video/24074014?key=527b3d4f9cbc75cb4bba070ae6665e5f

Unable to type in the text editor
Unlock the navigation it is still auto-synced to the book and the chapter
Navigation number shows wrong in the unlocked mode
Insert Verse, Chapter, Footnote and Cross-reference fetch the verse from the resources
Highlight feature connected to only one Bible reference in the reference pane not to the others

@samueljd , this PR behaved differently in two projects.  1-4 output from old project and 5-7 are from newly created project.

FR_LSG - is an aligned Bible,
EN_UDB - is a regular Bible - extra highlight on of Galilee
TA_ULB - is a regular Bible - Verse number color got changed
On the tWords working properly

AR_NAV - RTL contents verse number color changed
Except FR_LSG, all the other references are not in sync when clicked

7. Except FR_LSG, all the other references are not in sync when arrow key moved

@vipinpaul this PR can be merged. All the requested fixes working as expected.
Fixed:

Versewise navigation with arrow keys
Navigate on same verse multi click

Known issue: Navigation pannel not working properly.


cargo build --all --all-features in a non-virtual workspace ignores features not present in the root crate

I've got a couple of projects (example) that are workspaces with a root crate and another member in an ffi subdirectory. The ffi member has an ffi-headers feature that uses cbindgen to generate C/C++ headers.
I expect to be able to run cargo build --release --all --all-features and have all crates in the workspace built with all their features. This was working in Rust 1.25.0, I was able to run that command and see C/C++ headers get generated.
I upgraded to the Rust 1.26.0 release and now the above command no longer generates headers, it's as if the feature in the ffi crate is being ignored (the issue title may be a little inaccurate, I haven't verified this is what's actually happening). If I run cargo +1.25.0 build --release --all --all-features, everything works as expected.
This might be related to #5362 and #5364, but they're more about --package than --all so I thought I'd file a new issue. If I run cargo +nightly build --release --all --all-features, that doesn't work, but cargo +nightly build --release --all --all-features -Z package-features does.
My current version of cargo is:
> cargo --version
cargo 1.26.0 (0e7c5a931 2018-04-06)

Running with the 1.25.0 toolchain:
> cargo +1.25.0 --version
cargo 0.26.0 (41480f5cc 2018-02-26)

Running with nightly:
> cargo +nightly --version
cargo 1.27.0-nightly (9e53ac6e6 2018-05-07)

Thanks for the report! Would it be possible to minimize this a bit perhaps to be easier to investigate?
@ehuss you may also be interested in this issue

Yes, I've uploaded a minimal example repository here. Running it (with cargo versions unchanged from what I gave above):
PS C:\...\cargo-features-issue> cargo +1.25.0 build --release --all --all-features
   Compiling cargo-features-issue v0.1.0 (file:///C:/.../cargo-features-issue)
    Finished release [optimized] target(s) in 0.44 secs
PS C:\...\cargo-features-issue> .\target\release\member.exe
I have member-feature active!
PS C:\...\cargo-features-issue> cargo build --release --all --all-features
   Compiling cargo-features-issue v0.1.0 (file:///C:/.../cargo-features-issue)
    Finished release [optimized] target(s) in 0.66 secs
PS C:\...\cargo-features-issue> .\target\release\member.exe
I do not have member-feature active!

Ok thanks for the reduction! I can indeed reproduce this locally now. I've bisected this regression to 7de30dd21a6b78c718c9ee88b661bd0575542884 (part of https://github.com/rust-lang/cargo/pull/5012, cc @infinity0)
I believe this should be fixed in https://github.com/rust-lang/cargo/pull/5556

Sorry about that, thanks for catching. In general I think it's an anti-pattern to have enums with options like { AllSettings: {records}, Default, Preset1, Preset2 } etc, I've seen far too much of this in various rust code. It would be better to have a single {records} type with functions called default() preset1() etc that return presets.

I just pulled and built 22c0f22f5621f1b68558d80a0966d073b9b68932 and can confirm it doesn't have the issue, thanks!


5/5 - Canary workflow arm64 support

Description:
After #189 update the sample apps terraform to consider architectures, we can add the architecture check to the Canary workflow as well.
Link to tracking Issue:
N/A
Testing:
TBD
Documentation:
N/A

Latest manually executed Canary Test was mostly successful: https://github.com/aws-observability/aws-otel-lambda/actions/runs/1742516034
No issues related to the work we have done for the ARM layer. Instead it seems we are just at some IAM limits for roles created. I deleted a few and otherwise this PR should be good to merge.
NOTE: Since there is an issue with the Lambda, I will remove ARM64 from the matrix and leave another PR open to add it in later when it is fixed.


Crash on MD5 section possible conflict

Check List
Thanks for considering to open an issue. Before you submit your issue, please confirm these boxes are checked.

[ X] I have read the wiki page and cheat sheet, but there is no information I need.
[ X] I have searched in existing issues, but did find a same one.
[ X] I want to reoprt a problem instead of asking a question. It'd better to use kingfisher tag in Stack Overflow to ask a question.

Issue Description
I am currently using my switfyhash md5 hashing library and after a random number from 10-4000 photos being fetched by a ImagePrefetcher it crashes on the stringproxy.md5.getter see attached screenshot. I cannot figure out what if any possible conflict there is and am still investigated. I was just wondering if you might have an idea as to what could be going wrong. Thank You. I am hashing md5 within the completion handler if that helps at all.
What
[Tell us about the issue]
Reproduce
[The steps to reproduce this issue. What is the url you were trying to load, where did you put your code, etc.]
Other Comment
[Add anything else here]

I noticed that you are fetching a lot of images and it might be a problem of the stack. The completion stack is too large and it might be a stack overflow.
Could you confirm that this issue only happens when you have a large number of images?
If so, a possible quick workaround is breaking down your images into smaller set and download it with multiple prefetchers.

I removed the whole prefetched code out from a dispatch in background task and it is running a lot better and hasn't crashed yet however I'm noticing what appears to be socket errors in writing data. they aren't crashing the app but I am wondering If I'm running through everything too quickly? I might have to break the images into smaller sets as the though did cross my mind thank you. and yeah there is a lot of images being fetched. Thanks for the fast reply.

@onevcat  I had the same problem. It seems the recursion is too deep when prefetching thousand of image.
The easiest solution was to add an DispatchQueue.main.async  in reportCompletionOrStartNext.

@cl3m Good catch on it. I will try it soon.

The prefetcher stack overflow issue should be fixed in 4.4.1. (preparing a release now.)


Data Normalisation

The vast majority of depths lie within the range [0, 300]:

Only 8 pixels have depths that exceed this (0.0000003456858%). I think normalising by dividing by 300 and clipping to [0, 1] is sufficient. This has the advantage that the depth labels are still linearly scaled, and interpretation is easy.
Do we want to alter the distribution? For example, make it uniform or Gaussian?

Thanks for looking at this - nice analysis. For now lets not alter the distribution? Seems like a highly skewed learning problem, looking at the peak and distr, will be interesting if learnable as is?

Wait, are there functions to get rid of the pixels past 300 depth? I meant more as in the distribution after what you said.

The depths > 300 are now clipped to 300


ci: sonarcloud setup

Get SonarCloud using a monorepo setup, which can handle the individual services in this repo.  Best guess for config in this PR, but will be updated when our issue with BCDevOps is completed.
Request/issue

Thanks for the PR!
Deployments, as required, will be available below:

Frontend
Vehicles
Dops

Please create PRs in draft mode.  Mark as ready to enable:

Analysis Workflow

After merge, new images are promoted to:

Merge Workflow

@cberg-aot Force merging to trigger a set of Sonar Cloud base runs in main!


Wrap widget is missing WrapCrossAlignment.baseline.

Wrap widget's WrapCrossAlignment is missing WrapCrossAlignment.baseline.
In fact, there is a TODO comment in the code:
enum WrapCrossAlignment {
  start,
  end,
  center,
  // TODO(ianh): baseline.
}

This is a core widget, so this should be implemented.

https://github.com/flutter/flutter/blob/757c3add1470b54fbc7dab8993dc14c7163fde13/packages/flutter/lib/src/rendering/wrap.dart#L51

@marcglasberg
You are right, my bad. Updated labels as applicable.

I will add it but can you tell me description that how baseline will work?
@marcglasberg

@viralkachhadiya When the axis is horizontal, the widgets should be aligned vertically by their baselines. It's the same as Row:
  /// Place the children along the cross axis such that their baselines match.
  ///
  /// Because baselines are always horizontal, this alignment is intended for
  /// horizontal main axes. If the main axis is vertical, then this value is
  /// treated like [start].
  ///
  /// For horizontal main axes, if the minimum height constraint passed to the
  /// flex layout exceeds the intrinsic height of the cross axis, children will
  /// be aligned as close to the top as they can be while honoring the baseline
  /// alignment. In other words, the extra space will be below all the children.
  ///
  /// Children who report no baseline will be top-aligned.
  baseline,

any updates on this?

please update this important widget .. thank u


Hide language selection in studio, if just one language is installed

I mean the language selection above text input fields. It's useless, if I have only one language installed.

It's still good indication what language should be entered.


[Feature Request]Additional priority levels and default level

OpenTasks currently has four priority levels, defaulting to the lowest one, "none". The task list in Business Calendar 2 allows for five levels, defaulting to the middle one. Would it make sense for OpenTasks to also offer five levels and possibly default to the middle one (or allow the user to configure the default level)?

I don't see it as 4 levels, but as 3 levels (high, medium and low) and "unspecified" (which I don't see as a priority level). This is what the clients I know about use (Thunderbird/Lightning, Apple).
We'll definitely build something to set a default level.

Ok, I like that it is compatible with Thunderbird/Lightning which I also use. I wonder how Business Calendar 2 synchronizes its five levels with TB/L (and OT)?

Does BC2 sync to TB/L or OT? I've never tried it.
RFC 5545 (iCalendar) specifies 9 priority levels (plus 0, meaning "undefined") with 1 being the highest priority and 9 being the lowest priority.
We map these values just like TB does, 1-4 -> "High", 5 -> Medium, 6-9 -> Low.

Understand, I have not used the task list in BC2. BTW, I use owncloud as my backend.


Changed dependsOnNot + dependsOnNullOrZero fix + dependsOnNot fix

Added ability to set dependency for a field "being NOT equal to a list of values"
Fixed check for dependsOnNullOrZero to work properly (error when picking undefined or null)
For 'dependsOnNot' - changed 'strict check' to 'check with coercion' cause all form values are always strings (error when checking for integer values, integers strict check with string equivalents)

This would need to account for the loose comparison recently fixed which could mean significant complexity to ensure that we are comparing each array value without type.

That being said I think we need to split features/bugfixes. Adding a brand new potentially polarizing feature like an array to dependsOnNot shouldn't be grouped with a simple-ish bug fix. Also adding this new feature for dependsOnNot, really should be able to extend to dependsOn which we don't have here either. So perhaps 2 PR's split by new feature and bugfix. Be sure to check older PR's as well to confirm that we aren't just doubling work.

Ok, will split fixes and new feature.
Firstly I added DependsOnNotArray method, then think that theres no such situations when will be created error while converting one-value property to one-item array. So theres no need to create separated method...

Be sure to check older PR's as well to confirm that we aren't just doubling work.

I dont need to add features from other PRs if they are there, but not merged?

Be sure to check older PR's as well to confirm that we aren't just doubling work.

I dont need to add features from other PRs if they are there, but not merged? But why they are not merged?

I mean in relation to the features you are adding and bugs that you are fixing. Not specifically you, but I've been noticing that PR's are being submitted when there's already a PR to fix a particular bug/add a feature. DependsOn array support has been a hot topic and has had at least 4-5 PRs over time for the same thing.

DependsOn array support has been a hot topic and has had at least 4-5 PRs over time for the same thing.

I saw "DependsOn array" PR, that interesting feature, but not in need for me at the moment.
I needed dependsOnNot which can check few values at the moment, so I thought that transfer this values as array - the best way to solve my needs.


Inject proxy in repos using 'proxy' config setting

See https://github.com/projectatomic/atomic-reactor/pull/461

Some remarks inline, otherwise looks good. I'm wondering whether we should call the parameter something else than "proxy" to avoid confusion in case we use some other kind of proxy in the future? Something like "yum_proxy"?
Github says there are conflicts, can you please rebase?

Something like "yum_proxy"?

Yeah, sound a bit more accurate

LGTM, merged.


Add missing 404 response to Swagger

Most middleware throw a 404 in case something is not found e.g. a Repo that is not existing. But most API endpoints don't include the 404 response in their documentation. This PR changes this.

I was unable to create a backport for 1.20. @JakobDev, please send one manually. :tea:
go run ./contrib/backport 27038
...  // fix git conflicts if any
go run ./contrib/backport --continue


wip: new(ci): add sanitizers run with asan and ubsan

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area CI

What this PR does / why we need it:
Part of https://github.com/falcosecurity/falco/issues/2930 . Will remove WIP when we're sure it's working
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [LucaGuerra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 0.38.0

Love this, as always @LucaGuerra ! Great job!


Retrieve server certificate

Is there a simple way to retrieve the certificate presented by the server for manual checking?
 curl --insecure -vvI gives you pretty much everything, and I'm looking for something similar (or certificate-specific) to allow parsing of self-signed certificates, for instance.
Note - I've tried certinfo(true), but can't find any way to output the information, and ssl_verify_host(false) doesn't seem to help.
Thanks.

The counterpart to CURLOPT_CERTINFO is to call curl_easy_getinfo(easy, CURLINFO_CERTINFO, ptr) in order to get the information that the first method call enables. Currently we don't provide safe Rust bindings to this API, though this is something we could add.

Is there a simple way to retrieve the certificate presented by the server for manual checking?

Note that CURLOPT_CERTINFO does not necessarily give you access to the actual certificate, but rather metadata about the certificate that the TLS engine chooses to return. Some TLS engines don't return anything. I would not use this API to do manual certificate validation, especially because by the time you get access to this info, curl has already connected to the server and started performing the transfer.
In the C API, the way to customize cert validation is to use CURLOPT_SSL_CTX_FUNCTION, which should comes with the significant caveats that (1) it only works for certain TLS engines, (2) requires you to know specific details about the TLS engine's API, and (3) is very unsafe and easy to cause security vulnerabilities. For these reasons it isn't really something we'd want to expose at all in safe Rust bindings.

The counterpart to CURLOPT_CERTINFO is to call curl_easy_getinfo(easy, CURLINFO_CERTINFO, ptr) in order to get the information that the first method call enables. Currently we don't provide safe Rust bindings to this API, though this is something we could add.

Is there a simple way to retrieve the certificate presented by the server for manual checking?

Note that CURLOPT_CERTINFO does not necessarily give you access to the actual certificate, but rather metadata about the certificate that the TLS engine chooses to return. Some TLS engines don't return anything. I would not use this API to do manual certificate validation, especially because by the time you get access to this info, curl has already connected to the server and started performing the transfer.
In the C API, the way to customize cert validation is to use CURLOPT_SSL_CTX_FUNCTION, which should comes with the significant caveats that (1) it only works for certain TLS engines, (2) requires you to know specific details about the TLS engine's API, and (3) is very unsafe and easy to cause security vulnerabilities. For these reasons it isn't really something we'd want to expose at all in safe Rust bindings.

Closing for now, since adding bindings to CURLINFO_CERTINFO wouldn't help in the scenario raised, and I do not think it is possible to create a safe API around CURLOPT_SSL_CTX_FUNCTION. If you really need to use CURLOPT_SSL_CTX_FUNCTION then you can use the unsafe API by using Easy::raw and the curl-sys crate.

Happy to open a new issue, but I was also looking for CURLINFO_CERTINFO bindings today. Not to validate the certificate, but to record information it contained (at the time of the request).


Call to an undefined method Database\Factories<MyModel>::trashed()

Larastan Version: 2.6.4
--level used: 6

Description
I got warning about using the trashed state method, that is defined in Illuminate\Database\Eloquent\Factories\Factory
Laravel code where the issue was found
$user = \App\Models\User::factory()->trashed()->create();

I even tried to annotate the method in my UserFactory but this does not solve the issue. The only way I figurate out to solve the issue (without ignoring the line) is to define a custom state that calls parent trashed, but of course is not that cool

Still not working.

laravel: v10.33.0
laratan: 2.6.4

@ednar28, can you provide a minimum reproducible example? I am not seeing this error:
https://phpstan.org/r/6146e0f3-0358-4188-92d5-267754a55a78

this is my phpstan.neon. I'm using larastan lvl 9

That's not an MRE...and the phpstan link I posted above is running on level 9

Thanks, and the user model?

I thought it was a problem of my User model (since it extends another class) but I did the same on another model (that extends the lavarel base model) Factory and still the same issue.

So the following passes on Larastan 2.4.6 level 9:
<?php

namespace Tests\Larastan;

use Illuminate\Database\Eloquent\Factories\Factory;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\SoftDeletes;
use Illuminate\Database\Seeder;

class User extends Model
{
    use HasFactory;
    use SoftDeletes;

    protected $guarded = [];

    protected $hidden = ['password'];

    protected $table = 'users';

    protected static function newFactory(): UserFactory
    {
        return UserFactory::new();
    }
}

/** @extends Factory<User> */
class UserFactory extends Factory
{
    protected $model = User::class;

    public function definition(): array
    {
        return [
            'name'     => fake()->name(),
            'email'    => fake()->email(),
            'password' => 'password', // set unencrypted
        ];
    }
}

class UserSeeder extends Seeder
{
    public function run(): void
    {
        User::factory()
            ->count(10)
            ->trashed()
            ->create();
    }
}

I have the same issue as darfijt, but only in the context of my Pest tests. PHPStan picks up on the ->trashed() method just fine in other contexts like seeders, etc.
I don't have time to make a full reproduction, but I can say that I'm on larastan 2.7.0, and more or less have the same key elements as your passing code, namely:
    use HasFactory;
    use SoftDeletes;

in the model and
/** @extends Factory<MyModel> */

in the factory.
I originally didn't have a definition for protected static function newFactory set at first, but even when I added it, it didn't change the outcome inside Pest tests.


FEATURE: Respect ignoring / muting / blocking DM user preferences

This commit covers several areas of functionality. In core,
a user can do the following:

Ignore users - This makes it so the user cannot be cannot
PM'd, notified via mention, or invited into a topic by the target
user and also hides the target user's posts from topics
Mute users - This makes it so the user cannot be cannot
PM'd, notified via mention, or invited into a topic by the target
user
Disable PMs - This makes it so the user cannot be PM'd by
anyone
Enable PMs from allowed users - This makes it so the user cannot
be PM'd by anyone except those the user allows

This commit applies all these rules to chat:

When a user is ignoring or muting a target user, the target user
cannot create a new direct message channel with them
When a user is ignoring or muting a target user, the target user's
mentions of the user will be suppressed, including when the target
user mentions a group the user is part of, or @all, or @here
When a user is preventing PMs from target user, the target user
cannot create a new direct message channel with them
When a user is ignoring a target user, the target user's messages
in any channel will be hidden and collapsed, with an option to un-collapse
the message, similar to core's "view hidden posts" functionality
When a user is ignoring a target user, the target user's new sent messages
will not cause the unread dot to appear in channel names, and will always be
hidden

What happens concerning replying indicator? Will I see it in a channel with someone I muted? What happens in a channel with 3 people where one is muted by only another one?

What happens concerning replying indicator? Will I see it in a channel with someone I muted?

@jjaffeux replying indicator still appears, not sure of the behaviour in topics if you have ignored someone there, will check.

What happens in a channel with 3 people where one is muted by only another one?

If I am Alice and I have muted Bob and we are in a 3 person DM with Joe, then Joe will see Alice and Bob's replies and get green indicators for them, but Alice will only see Joe's replies and get a green indicator for them

I found one bug:

User A ignores User B
User A IS NOT TRACKING a conversation with User B (ie: User B is not appearing in the sidebar)
User B sends me a message, the User B appears in the sidebar with a green dot

We shouldn't show the greendot for sure
Might be debatable, but I would say it shouldn't appear in the sidebar

Based on the above, I think there are three gaps we should close.

If the message channel already exists, and they have untracked it, then I agree, it should not become tracked.
If it is still tracked, then green dots should not appear.
Furthermore I think we should prevent User A from adding User B to a message if they already have them muted (we already do the reverse, but preventing this makes sense to me too).

Thanks @jjaffeux ! Definitely more refinements to come on this :+1:


Add new nodejs module

This PR adds a node installer module. It is powered by NVM. By default it installs the latest version of node and makes that the default. But the user can also provide a list of node versions, they like to install and also a custom default node version as well.
I'm not 100% sure about the naming. Other possible names for the module:

nvm
node
nodejs

What you you prefer?
Note: node installs the latest version of node.

@mafredri @matifali Would appreciate, if you guys could have a look at this. Thanks!

Thanks for the review! I think fixed all issue but one (see comment)

Hmm very strange, I pushed changes to my branch and even rebased onto the latest main, but somehow github does not update this PR with the latest commit 😕
https://github.com/TheZoker/coder-modules/commits/add-nvm-module/
@mafredri Any idea why this could be?

@mafredri Alright, seems like the Github sync job has caught up. Can you check if the script is ok now? Thanks!

@mafredri Since we set verified to false for now, I think this issue should now be resolved, right?

Thanks for the reviews and the merge 🎉


Readme: Add how to use repo-specific git hooks

Thank you for your contributions. I am closing this in favor of #58 which automates the setup of prepare-commit-msg githook


static_assert causes Boost.JSON integration to fail

I have written an example on how to integrate Boost.JSON with this library. However, the following static assert:
static_assert(
    details::is_valid_json_types<typename json_traits::value_type,
                                 typename json_traits::string_type,
                                 typename json_traits::object_type,
                                 typename json_traits::array_type>::value,
    "must staisfy json container requirements");

in jwt.h line 1872 causes every set_* function to error when creating a JWT.
The produced error is:
In template: static_assert failed due to requirement 'is_valid_json_object<boost::json::value, std::basic_string<char, std::char_traits<char>, std::allocator<char>>, boost::json::object>::value' "object_type must be a string_type to value_type container"

If I remove that static assert then I am able to successfully create a token. I have also been able to verify the resulting token using a separate library.
Before submitting a pull request to add this example to the code base, I would appreciate some discussion/insight as to why this is occurring.
Here is the complete example of the Boost.JSON integration.
#define JWT_DISABLE_PICOJSON

#include <iostream>
#include <boost/json.hpp>

#include "jwt.h"

namespace json = boost::json;           // from <boost/json/src.hpp>

struct boost_json_traits {
    // Type Specifications
    using value_type = json::value;      // The generic "value type" implementation, most libraries have one
    using object_type = json::object;    // The "map type" string to value
    using array_type = json::array;      // The "list type" array of values
    using string_type = std::string;      // The "list of chars", must be a narrow char
    using number_type = double;       // The "precision type"
    using integer_type = int64_t;       // The "integral type"
    using boolean_type = bool;         // The "boolean type"

    // Translation between the implementation notion of type, to the jwt::json::type equivalent
    static jwt::json::type get_type(const value_type &val) {
        using jwt::json::type;

        if (val.is_object())
            return type::object;
        if (val.is_array())
            return type::array;
        if (val.is_string())
            return type::string;
        if (val.is_double())
            return type::number;
        if (val.is_int64() || val.is_uint64())
            return type::integer;
        if (val.is_bool())
            return type::boolean;

        throw std::logic_error("invalid type");
    }

    // Conversion from generic value to specific type
    static object_type as_object(const value_type &val) {
        if (val.is_object()) return val.as_object();
        throw std::bad_cast();
    }
    static array_type as_array(const value_type &val) {
        if (!val.is_array()) return val.as_array();
        throw std::bad_cast();
    }
    static string_type as_string(const value_type &val) {
        if (!val.is_string()) return val.as_string().c_str();
        throw std::bad_cast();
    }
    static number_type as_number(const value_type &val) {
        if (!val.is_double()) return val.as_double();
        throw std::bad_cast();
    }
    static integer_type as_int(const value_type &val) {
        if (val.is_int64()) return val.as_int64();
        if (val.is_uint64()) return val.as_uint64();
        throw std::bad_cast();
    }
    static boolean_type as_bool(const value_type &val) {
        if (!val.is_bool()) return val.as_bool();
        throw std::bad_cast();
    }

    // serialization and parsing
    static bool parse(value_type &val, const string_type& str) {
        val = json::parse(str);
        return true;
    }
    static string_type serialize(const value_type &val) {
        return json::serialize(val);
    }
};

int main() {
    std::string pub_key = "YOUR PUB KEY HERE";
    std::string priv_key = "YOUR PRIV KEY HERE";

    auto token = jwt::create<boost_json_traits>()
            .set_issuer("iss")
            .set_type("JWT")
            .set_issued_at(std::chrono::system_clock::now())
            .set_expires_at(std::chrono::system_clock::now() + std::chrono::seconds{3600})
            .set_payload_claim("token_use", "access")
            .sign(jwt::algorithm::rs256(pub_key, priv_key, "", ""));

    std::cout << "token:\n" << token << std::endl;
}

Yep, I had attempted to fix the verification https://github.com/prince-chrismc/jwt-cpp/tree/boost-traits

I have never used boost json (or any boost library except asio a while back), but I think (actually I am pretty sure) the reason it fails is the following (from boost docs, highlight by me):

Instances of the object type are associative containers holding key and value pairs, where the key is a string_view and the mapped type is a value.

jwt-cpp expects a std::string as key which is why it fails to compile. The reason why everything still works fine is because std::string_view is implicitly convertible to std::string and has a very similar interface. So on the first time it hits any other function call or assignment, its converted to std::string and everything just works from there.
Given this, I think it's probably safe to change https://github.com/Thalhammer/jwt-cpp/blob/ac0424b115721e4066d2fb99f72ba0cd58759882/include/jwt-cpp/jwt.h#L1850 to use std::is_convertible instead of std::is_same, which should get rid of the error without being too lax.

I am not sure it's that easy, because the verification makes a lot of assumptions like adding to shrink strings.
Boost.JSON has an internal implementation below C++17... which is not convertible to std::string and does not have the same API.
See https://github.com/boostorg/json/blob/5b45854bb0422d9a37d846c018535cc1004e1449/doc/qbk/07_faq.qbk#L61

string also implements an improved interface
that replaces extraneous overloads with ones that
use string_view

As OP mentions, creation is easy but they did not try to verify it with jwt-cpp

If I remove that static assert then I am able to successfully create a token. I have also been able to verify the resulting token using a separate library.

From the tests I have done the verification is a lot harder than it looks. You can check this commit https://github.com/prince-chrismc/jwt-cpp/commit/85d59657ad779ec9a8c6e953adadb4d5ee50597a and this one too https://github.com/prince-chrismc/jwt-cpp/commit/d7585d70a6ce2a2916c6453977a434891dcd497c to see how I tricked it to working along with the short comming I found.

Is support Boost.JSON standalone with C++17 requirement a good enough solution?

https://github.com/boostorg/json/blob/351603c9b2dee7a80c8433841b50f7294131b25a/doc/qbk/03_06_conversion.qbk#L340
I got it working here
https://github.com/prince-chrismc/jwt-cpp/commit/a236d470fd98f91066fae607f637a31e1b3e7c84#diff-be2369ece29c0945fc04ffe13a39767b7841fc2fabd3df209a26b3bb381a2fb2R1849-R1851


Release 0.9.3

This patch prepares the release of the minor version 0.9.3. It's forked off 5c7881c45c1b9cb35e8c9417f09521f7a517b8cf and includes a cherry-pick of 48b5931f07c0dfdb96ee5fa2724e9b3689358441.
Changelog
Parser
Even though this is a minor release, it changes changes how some inputs are parsed. These changes are made to comply with the CommonMark and GitHub Flavored Markdown specifications.
First, pulldown-cmark now supports strikethrough with a pair of single tildes in addition to double tildes, if the strikethrough option is enabled. For example, ~this~ is now equal to ~~this~~ and will be parsed like ~this~.
The second change fixes a bug in the parser where a specific edge case with brackets and inline code ([`]`]) was incorrectly parsed as a link. This change should not affect most users of pulldown-cmark.
impl Debug for Everything
All public types in the pulldown-cmark library now implement the Debug trait.
PRs included in this release

https://github.com/raphlinus/pulldown-cmark/pull/604
https://github.com/raphlinus/pulldown-cmark/pull/611
https://github.com/raphlinus/pulldown-cmark/pull/566
https://github.com/raphlinus/pulldown-cmark/pull/621
https://github.com/raphlinus/pulldown-cmark/pull/643

cc @Martin1887 @raphlinus

Published, thanks.


installation instructions

fisher metro

should be fisher install metrofish/metrofish

sorry, was looking at old git tree


Example to run listNet

Is there any example on how to run the ListNet module?

Also is the code of Listnet correct?

The code of Listnet is correct, it is a vanilla implementation of the formula (2) in the paper Learning to Rank: From Pairwise Approach to Listwise Approach.
As for the example, you could modify the local config scripts/local_config.json and run the example script scripts/run_in_docker.sh, remembering about building the docker image first (docker build -t allrank:latest . from the main directory).

Thank you :)
On Mon, 2 Aug 2021, 23:23 Radosław Białobrzeski, @.***>
wrote:

The code of Listnet is correct, it is a vanilla implementation of the
formula (2) in the paper Learning to Rank: From Pairwise Approach to
Listwise Approach
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2007-40.pdf
.
As for the example, you could modify the local config
scripts/local_config.json and run the example script
scripts/run_in_docker.sh, remembering about building the docker image
first (docker build -t allrank:latest . from the main directory).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/allegro/allRank/issues/38#issuecomment-891217461, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABHWVPK3AKLYRLHXSSWZJMTT23LQTANCNFSM5BF6NVEA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.


🐛: Results API endpoints throws 500 error

Contact Details
No response
Location
runBiosimulations
What happened?
Using either the API or web browser, we get 500 errors when getting results. This is caused by the HSDS returning a 404, but it is not clear why this is not being handled or retried properly. This might be due to retry logic being set up not to retry 404s which we do not want in this case, since intermittent issues with the s3 bucket will cause 404s from HSDS
What made it happen?
Try to load visualizations, call results API with data
Version
Production (Deployed on a .org domain), Development (Deployed on a .dev domain)
What browsers are you seeing the problem on?
No response
Relevant log output
Error: Request failed with status code 404
    at createError (/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/app/node_modules/axios/lib/adapters/http.js:293:11)
    at IncomingMessage.emit (node:events:402:35)
    at IncomingMessage.emit (node:domain:475:12)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json',
      Authorization: 'Basic *********',
      'User-Agent': 'axios/0.24.0'
    },
    params: { domain: '61da6510837f3f112d0b6a4e.results' },
    withCredentials: true,
    method: 'get',
    url: 'http://hsds:80/datasets',
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'hsds',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(async_id_symbol)]: 1055047,
      [Symbol(kHandle)]: [TCP],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'GET /datasets?domain=61da6510837f3f112d0b6a4e.results HTTP/1.1\r\n' +
      'Accept: application/json\r\n' +
      'Authorization: Basic *******' +
      'User-Agent: axios/0.24.0\r\n' +
      'Host: hsds\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype],
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/datasets?domain=61da6510837f3f112d0b6a4e.results',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 404,
      statusMessage: 'Not Found',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://hsds:80/datasets?domain=61da6510837f3f112d0b6a4e.results',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 10,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'hsds',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://hsds:80/datasets?domain=61da6510837f3f112d0b6a4e.results',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      authorization: [Array],
      'user-agent': [Array],
      host: [Array]
    }
  },
  response: {
    status: 404,
    statusText: 'Not Found',
    headers: {
      'content-type': 'text/plain; charset=utf-8',
      'content-length': '14',
      date: 'Mon, 24 Jan 2022 17:40:04 GMT',
      server: 'Python/3.8 aiohttp/3.7.4.post0',
      connection: 'close'
    },
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      params: [Object],
      withCredentials: true,
      method: 'get',
      url: 'http://hsds:80/datasets',
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [Socket],
      _header: 'GET /datasets?domain=61da6510837f3f112d0b6a4e.results HTTP/1.1\r\n' +
        'Accept: application/json\r\n' +
        'Authorization: Basic *******' +
        'User-Agent: axios/0.24.0\r\n' +
        'Host: hsds\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/datasets?domain=61da6510837f3f112d0b6a4e.results',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'hsds',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: '404: Not Found'
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}
[Nest] 1  - 01/24/2022, 5:40:04 PM     LOG [DefaultFilter] Object:
{
  "error": [
    {
      "status": "500",
      "title": "Internal Server Error",
      "detail": "An unexpected error occurred",
      "meta": {
        "time":<PHONE_NUMBER>355,
        "url": "/results/61da6510837f3f112d0b6a4e/simulation.sedml%2Freaction_fluxes?includeData=true"
      }
    }
  ]
}

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This is being caused by the "db" and "domains.json" folders not being present on the prod uconn bucket. I have no idea how this data was lost, but the dataservice will not work without it.
@jonrkarr this is unfortunately going to make any demos impossible. I'll try to fix this asap, but we would need to rerun all the examples and projects on prod to get this working. Thoughts?

I think i found the issue. This commit https://github.com/biosimulations/deployment/commit/35dbcb38da7b3ee2ae9793e5c530567a1687e3fe actually fixed the loading of the configuration of HSDS properly. Before this, the config was not being loaded, but was using the default value of "biosimdev". This means that all the older data for the prod branch is also on the biosimdev bucket.

This has been resolved for the existing simulations. We need to rerun the simulations and projects on the next deployment to production and update the deployment to use the correct buckets

Seems like the buckets need to be inspected to ensure that data is being written to the correct locations.
I have at least one example working on my machine

Yup, this is fixed now for all the current examples. I switched the prod hsds to use the dev uconn bucket which is where the data is


Remove multiple SNO install complexity from SNO install playbooks

Currently the SNO install playbooks allow you to install more than one SNO concurrently while the bare-metal playbook only installs one cluster at a time. I think we should reduce complexity of the SNO install playbooks to match the same structure as bare-metal install playbooks.  If you want to install a second SNO, the user needs to actively manage their inventory file either by comment/uncommenting SNOs or creating additional inventory files.  What do you think? @Noreen21 and @sferlin

I am in two minds about this issue. Even though I understand that this would help get the structure aligned with bare metal installments, the ability to be able to install multiple SNOs simultaneously without majorly tweaking anything is of great value to me (and hopefully others). It majorly reduces the environment build time when installing and applying all DU configs on multiple SNOs. The only drawback being that haproxy support is currently limited to only a single SNO from bastion, which can be overcome by use of ssh tunnels, so documenting this solution for when managing multiple SNOs from a single bastion would make life easier.

Part of this issue was completed with #499, there is still more work to do to converge the sno deployment into mno so they can share more roles and common tasks etc.


"Error: Cellect is unavailable"

I am trying out Project "Season Spotter Alpha3"
http://demo.zooniverse.org/panoptes-front-end/#/projects/mkosmala/Season+Spotter+Alpha3
and when I click on a Workflow to load Subjects, I get the error message:
Error: Cellect is unavailable

I do not have this trouble with very similar projects. For example:
http://demo.zooniverse.org/panoptes-front-end/#/projects/mkosmala/Season+Spotter+Mu2
(select Workflow "4_ComplexAMOS")
works fine.
The only major difference between these two projects is that the former has almost 200 subjects and the latter has just 10.
Both projects are private=false, so you should be able to try them out.

Looks like the issue is that Cellect can't load the Workflow quickly enough and the task gets killed.

@edpaget Can you prioritize this bug? I'm read to do alpha testing, but can't do it if subjects don't load.

It was a DB error in Cellect-Panoptes, I've restarted manually and it resolved the issue. Possibly related to https://github.com/zooniverse/cellect_panoptes/issues/24 but could have been related to post stargazing infrastructure changes.
Closing for now, will revisit if we can reproduce on demand.


OkHttpServices::getIteratedResourceImpl ignores given mimetypes

https://github.com/marklogic/java-client-api/blob/9803cca4cfdeecde4b9d8e1ab0396e45814a2b02/marklogic-client-api/src/main/java/com/marklogic/client/impl/OkHttpServices.java#L3078
Method getIteratedResourceImpl(...) takes a String-array of accepted mimetypes, but never uses them.

The fix will be just to refactor at the start of a release cycle to remove the superfluous parameters.

Removing dead parameters is good for code quality, but this fix is a low priority because it is entirely internal.

Better late than never on this cleanup - see #1465 . Will include this in 6.0.0 so we can remove it at some point, unless we decide it's a bug that the argument exists in 6 methods in ResourceServices.