[bug] Scorecard action not reporting binary artifacts in the repo

flutter/packages contain a binary artifact https://github.com/flutter/packages/blob/main/packages/rfw/example/wasm/logic/calculator.wasm but scorecard action is not reporting it(https://deps.dev/project/github/flutter%2Fpackages).
It seems like the current version is from the end of 2020 is there a new release scheduled? is this caused by the action not using the latest version of scorecard?

\cc @diogoteles08

Is there an easy way to verify what Scorecard version is used by the latest scorecard-action version?

Any artifacts will have the scorecard version used in them:
e.g. https://github.com/ossf/scorecard/actions/runs/4556801547
The release notes usually include the scorecard version in the changelog. Or you can look at the go.mod file at the tag in question: https://github.com/ossf/scorecard-action/blob/v2.1.3/go.mod
v2.1.3 has been released which has this fix. Feel free to re-open if you continue to observe the behavior after updating.

Thanks @spencerschrock for the new release. Dependabot already started rolling the new version to the flutter repositories.


Added Yandex Cloud S3 output

Signed-off-by: Narek Tatevosyan<EMAIL_ADDRESS>
What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config

/area outputs

/area tests

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: nar3k
To complete the pull request process, please assign keisukeyamashita after the PR has been reviewed.
You can assign the PR to them by writing /assign @keisukeyamashita in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@nar3k: There is not a label identifying the kind of this PR.
Please specify it either using /kind <group> or manually from the side menu.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Welcome @nar3k! It looks like this is your first PR to falcosecurity/falcosidekick 🎉


Fix nullable types not throwing ts errors

#48 indicates there's a problem with the typescript set up where it doesn't alert if a property getter is called on a nullable variable.
// b is Vec2 | null
const a = b.x + 1 // this should throw ts-error, but it doesn't

That behavior depends on flag strict: false in tsconfig.json. I don't know the reasons why it's false, but it disable some validations in TS. To have just null checking you can enable another flag - strictNullChecks: true


Possible incompatibility between LuckPerms and Chunky Border on a FabricMC 1.20.1 server

Description
Recently, my server has been crashing with this error: https://mclo.gs/oKIqdqv, which seems to suggest a conflict between LuckPerms and Chunky Border on a FabricMC server. Here is my server log: https://mclo.gs/d3ByMZT.
Reproduction Steps
Install LuckPerms mod and Chunky Border mod on a FabricMC 1.20.1 server and wait for the error to happen.
Expected Behaviour
LuckPerms and Chunky Border should not be conflicting with each other and should not be causing the server to crash.
Server Details
Minecraft 1.20.1 with Fabric Loader 0.14.21
LuckPerms Version
v5.4.98
Logs and Configs
Server log: https://mclo.gs/d3ByMZT
Crash report: https://mclo.gs/oKIqdqv
Extra Details
No response

reproduce with only LP and chunky installed


Fix bugs of losing selected items when adding new items. Fix bugs in …

…search box and sorter. Fix bug of not refreshing after edit. Rearrange UI and add inline-insertion.

@dlwangyilin There is still a bug here. If I have two sessions, and then I make an edit to one of them, the room of one of them will show up as NO SUCH DATA. The right data is there in the DB, though.


migrate roles from outline

In some pages populated from outlines (such as https://www.tbrc.org/?locale=en#!persons/tulkus ) there's more data than in the records (ex: P999 is associated with R8LS12876 there but not in the person record of P999), but we don't migrate this data. We should do so. @xristy do you have the RIDs of the corresponding outline(s)?

well, some tiny progress: https://www.tbrc.org/xmldoc?rid=O4JW33827 says
<outline:browser class="StaticContentBrowser" funct="tulku-lines" module="persons/tulkus/"/>

but... not sure where that leads...

In O4JW33827 the reference:
<outline:browser class="StaticContentBrowser" funct="tulku-lines" module="persons/tulkus/"/>

Is to /db/tbrc/tbrc-web-content/persons/tulkus/tulku-lines_{bo,en}.xml.
The file is pretty much what you see on the screen.

The
View visualized graph: R8LS13053

Will display the graph of the lineage rooted in P5981 but they take some seconds to render. Some  look sort of like caterpillars.

apps/pgraph.jsp is located on tbrc.org server at ~/browser/tomcat/webapps/apps/pgraph.jsp.


[#136455813] Next gen

https://www.pivotaltracker.com/story/show/136455813

LGTM

LGTM too :slightly_smiling_face:


ConfigProperty#GUI_DATA_DIR default should be "pageyaml"

The current default value for the mentioned ConfigProperty as of v1.0.0 is GUIDataDir.
The 1.1.0-SNAPSHOT archetype example project pom.xml uses the value pageyaml.
As a result, a person must explicitly set -DSELION_GUI_DATA_DIR=pageyaml or <property name="GUIData" value="pageyaml" /> in a testng suite file to use the generated source that comes with the archetype
I propose SeLion change to pageyaml everywhere for the 1.1.0 release

I have had a change of heart.. Closing this issue.


Problems with SearchLight

Dear developers,
It's more than 3 days I'm trying to use your package without any good result since I'm not able to reproduce the examples in the README.
My problems are:

in the example for JSON, the result web page is the following:
[{"title":"The Best We Could Do","author":"Thi Bui"},{"title":"Evicted: Poverty and Profit in the American City","author":"Matthew Desmond"},{"title":"Believe Me: A Memoir of Love, Death, and Jazz Chickens","author":"Eddie Izzard"},{"title":"The Sympathizer!","author":"Viet Thanh Nguyen"},{"title":"Energy and Civilization, A History","author":"Vaclav Smil"}],
is that correct?
-I'm trying to use the db features of SearchLight  of paragraph
Accessing databases with SeachLight models
but when I run the command
SearchLight.Configuration.load_db_connection()
I get
Dict{String,Any} with 0 entries
I'm in the route of the app, I activated the connection by pkg> activate . but I cannot have your result of 3 entries. What can be the problem?

Anyway I think you should post somewhere the app folders for all the examples and enumerate each one with an identifier so it's easier to see if we do some errors in copying the readme lines of code. I also saw an authentication app in one of the examples folders but I couldn't fine any reference to it in the readme. I think this creates confusion.
I had also a lot of issues with package compatibilites between versions, I don't know why after some time I have an error and I have to remove all the packages and readd them again, today I did it at least 4 times. I think you should spend some lines of the readme about the packages required with the correct links for the repositories.
At the end I didn't understand what happens if I would like to deploy and distribute my webapp, is it possible to do it and in case it is possible can I open the index.html of the app  in local in a pc without Julia installed?
Thanks
Giada

Thank you for your prompt reply,
I'm using Julia 1.0 and Windows 10.
This mornign I tried to create a new_app and  load it, this is what I have -.-"
WARNING: replacing module App.
WARNING: replacing module Toolbox.
WARNING: using Toolbox.VoidTaskResult in module Main conflicts with an existing identifier.
WARNING: using Toolbox.TaskResult in module Main conflicts with an existing identifier.
ERROR: LoadError: invalid redefinition of constant App
Stacktrace:
 [1] top-level scope at none:0
 [2] eval(::Module, ::Any) at .\boot.jl:319
 [3] top-level scope at none:0
 [4] include at .\boot.jl:317 [inlined]
 [5] include_relative(::Module, ::String) at .\loading.jl:1041
 [6] include(::Module, ::String) at .\sysimg.jl:29
 [7] include(::String) at .\client.jl:388
 [8] top-level scope at none:0
 [9] eval(::Module, ::Any) at .\boot.jl:319
 [10] #load_app#4(::Bool, ::Function, ::String) at C:\Users\giada\.julia\packages\Genie\jGA6Q\src\RE
PL.jl:75
 [11] load_app(::String) at C:\Users\giada\.julia\packages\Genie\jGA6Q\src\REPL.jl:75
 [12] top-level scope at none:0
in expression starting at C:\Users\giada\Documents\Julia App\new_app\genie.jl:43

I restarted Julia, removed all the packages (SearchLight,WebSockets,HTTP,Genie,SQLite,Flax) and installed them again, I still have this error...
I think that the problem of incompatibilities is given by WebSockets and HTTP and according to which order I install one or the other I have errors, it's very difficult to explain.
I tried all my best but I cannot use Genie anymore.
Can you please post the precise order in which all the required packages must be installed with the commands so I can try for the last time?
Thank you very much
Giada

UPDATE: I did it again and now I can at least create the app, I'll try again to do everything and see if I can arrive to the point of the SerachLight example and I'll let you know.

in the JSON section I cannot use the function  respond, this is the error I get
Distributed.RemoteException(1, CapturedException(UndefVarError(:respond), Any[])) in C:\Users\giada\.julia\packages\Genie\jGA6Q\src\AppServer.jl:60
When  try to pass to the SearchLight section and I try to load the app I have this error

Starting Genie in >> DEV << mode

WARNING: both Loggers and SearchLight export "Loggers"; uses of it in module Main must be qualified
ERROR: LoadError: UndefVarError: Loggers not defined
Stacktrace:
 [1] load() at C:\Users\giada\Documents\Julia App\booksApp\src\App.jl:204
 [2] top-level scope at none:0
 [3] include at .\boot.jl:317 [inlined]
 [4] include_relative(::Module, ::String) at .\loading.jl:1041
 [5] include(::Module, ::String) at .\sysimg.jl:29
 [6] include(::String) at .\client.jl:388
 [7] top-level scope at none:0
 [8] eval(::Module, ::Any) at .\boot.jl:319
 [9] #load_app#4(::Bool, ::Function, ::String) at C:\Users\giada\.julia\packages\Genie\jGA6Q\src\REP
L.jl:75
 [10] load_app(::String) at C:\Users\giada\.julia\packages\Genie\jGA6Q\src\REPL.jl:75
 [11] top-level scope at none:0
in expression starting at C:\Users\giada\Documents\Julia App\booksApp\genie.jl:46

then I restart Julia and now I have again the problem of the invalid redefinition of constant App.
I think I'll give up and wait for new updates of the package/s, I hope my issues can help you in finding bugs in the package especially in the links between packages.
Let me know if you have suggestions to avoid my errors.
Thank you again
Giada

@giadasp

most likely you don't have using Genie.Renderer. It is Genie.Renderer which provides the respond method.

let me get back to you on this - I'm gonna run a test on Win 10.

I have it in the controller file.

Can you please post the comments of the controller file here?

Also the output of julia> versioninfo() and of pkg> st. Thanks!

julia> versioninfo()
Julia Version 1.0.1
Commit 0d713926f8 (2018-09-29 19:05 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-7300U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = "C:\Users\giada\AppData\Local\atom\app-1.32.0\atom.exe" -a
  JULIA_NUM_THREADS = 2

pkg> st
   Status `C:\Users\giada\.julia\environments\v1.0\Project.toml`
  [c7e460c6] ArgParse v0.6.1
  [c52e3926] Atom v0.7.8
  [a076750e] CPLEX v0.4.1
  [336ed68f] CSV v0.4.2
  [3da002f7] ColorTypes v0.7.5
  [861a8166] Combinatorics v0.7.0
  [34da2185] Compat v1.3.0
  [1a78104a] DBI v0.0.0 #master (https://github.com/JuliaDB/DBI.jl)
  [a93c6f00] DataFrames v0.14.1
  [31c24e10] Distributions v0.16.4
  [53c48c17] FixedPointNumbers v0.5.3
  [c41d0e25] Flax v0.0.0 #master (https://github.com/essenciary/Flax.jl)
  [c43c736e] Genie v0.1.0 #master (https://github.com/essenciary/Genie.jl)
  [708ec375] Gumbo v0.5.1
  [77172c1b] HttpCommon v0.5.0
  [682c06a0] JSON v0.19.0
  [4076af6c] JuMP v0.18.4
  [e5e0dc1b] Juno v0.5.3
  [f28f55f0] Memento v0.10.0
  [39ec1447] Millboard v0.2.0
  [49dea1ee] Nettle v0.4.0
  [4d1e1d77] Nullables v0.0.8
  [bac558e1] OrderedCollections v1.0.2
  [9b87118b] PackageCompiler v0.5.0
  [91a5bcdd] Plots v0.20.6
  [a223df75] Reactive v0.8.2
  [189a3867] Reexport v0.2.0
  [295af30f] Revise v0.7.12
  [0aa819cd] SQLite v0.7.0
  [340e8cb6] SearchLight v0.1.0 #master (https://github.com/essenciary/SearchLight.jl.git)
  [aa65fe97] SnoopCompile v0.3.1
  [60ddc479] StatPlots v0.8.1
  [2913bbd2] StatsBase v0.25.0
  [30578b45] URIParser v0.4.0
  [b8865327] UnicodePlots v0.3.1
  [104b5d7c] WebSockets v1.0.3+ #master (https://github.com/JuliaWeb/WebSockets.jl.git)
  [ddb6d928] YAML v0.3.2
  [8bb1440f] DelimitedFiles

What do you mean for comments?
anyway this is the code of BooksController.jl
# app/resources/books/BooksController.jl
module BooksController
using Genie.Renderer

struct Book
  title::String
  author::String
end

const BillGatesBooks = Book[
  Book("The Best We Could Do", "Thi Bui"),
  Book("Evicted: Poverty and Profit in the American City", "Matthew Desmond"),
  Book("Believe Me: A Memoir of Love, Death, and Jazz Chickens", "Eddie Izzard"),
  Book("The Sympathizer", "Viet Thanh Nguyen"),
  Book("Energy and Civilization, A History", "Vaclav Smil")
]

function billgatesbooks()
 #=
  response = "

Bill Gates' list of recommended books

      $( mapreduce(b -> "
$(b.title) by $(b.author)", *, BillGatesBooks) )

  "
=#
#for json
json!(:books, :billgatesbooks, books = BooksController.BillGatesBooks)

end
module API

using ..BooksController
using JSON

function billgatesbooks()
  #JSON.json(BooksController.BillGatesBooks)
  respond(JSON.json(BooksController.BillGatesBooks), "application/json")

end

end

end

Sorry, not "comments", I meant "contents" :)
Ok, so using Genie.Renderer should also be placed inside the API module, per the sample code in the README.
module API

using ..BooksController
using Genie.Renderer ### <=== this
using JSON

function billgatesbooks()
... more code here...

The output from pkg> st shows the packages installed at system level. Genie apps manage their own dependencies, using their own Project.toml file. So something is not right there. The idea is that:

you cd into the app's dir
using Genie
Genie.REPL.load_app()
This automatically loads the app's project, including its dependencies.
Here is the output from my console on Win 10, in a newly created app, after also adding SearchLight, per the README:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.1 (2018-09-29)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> cd()

julia> cd("justatestapp\\")

julia> using Genie

julia> Genie.REPL.load_app()
 _____         _
|   __|___ ___|_|___
|  |  | -_|   | | -_|
|_____|___|_|_|_|___|

Starting Genie in >> DEV << mode

(justatestapp) pkg> st
    Status `C:\Users\adrian\justatestapp\Project.toml`
  [c43c736e] Genie v0.1.0 #master (https://github.com/essenciary/Genie.jl)
  [cd3eb016] HTTP v0.6.14
  [39ec1447] Millboard v0.2.0
  [4d1e1d77] Nullables v0.0.8
  [5fb14364] OhMyREPL v0.3.0
  [295af30f] Revise v0.7.12
  [340e8cb6] SearchLight v0.1.0 #master (https://github.com/essenciary/SearchLight.jl)
  [ddb6d928] YAML v0.3.2

(justatestapp) pkg>

Here is what I did up to this point:
1 - started from scratch in a VM -- I just deleted the .julia folder and wiped off all the packages.
2 - installed latest Julia
So I had a clean Julia setup.
I started a Julia REPL. Then added Genie per the README. Then using Genie and Genie.REPL.new_app(...). This creates the new app and its project for managing the dependencies and activates the app's environment. Now you can add SearchLight. The important thing is that once you create a Genie app you need to load the project to work in its environment. For this you have to be in the app's dir and either call using Genie and Genie.REPL.load_app() which bootstraps everything for you (doing pkg> activate . among other things).
More details about dependency management in Julia v1 here: https://docs.julialang.org/en/v1/stdlib/Pkg/

If you still can't sort it out we can chat on Gitter for a few minutes

Sorry, I didn't read your last messages, I'll try and I'll let you know. Thank you very much for your patience!

I've add all the needed versions of the packages by hand after loaded the app and mow my pkg> st is like yours and the app works. I still don't know how it should be the result of api/v1/bgbooks, also with respond is like the one I posted in the first message. Is that correct (again)?
I cannot delete again Julia and reinstalled it because it will take too much time, I need a lot of packages for my work and now I don't have time to reinstall everything. I'm sorry that I cannot test the genie project.toml for you.
I'll go on with the readme examples and tell you if I find other issues (and I hope I don't have other surprises with the packages :/ )
Thank you again.
Giada

What does this output?
julia> SearchLight.Configuration.load_db_connection()

The right one:
Dict{String,Any} with 3 entries:
  "config"   => nothing
  "database" => "db/books.sqlite"
  "adapter"  => "SQLite"

One question:
how can I go back to julia> if I'm in genie> ?

Please try updating to latest Genie pkg> up and let me know how it goes.

Closing this due to lack of activity - feel free to reopen if the problem still persists.


loadRelations from collections.

Hi,
During the development of my project, i've seen the need of load relations from multiple objects at the same time. This code helps with coding several forEachs and some promises chaining.
In the PR, i've done some tests, check it out! :)
Regards

stale


E2E Tests Functional

CLI Refactored & Functional
Client Refactored & Functional
Client Automated Testing
Various bug fixes
SimpleStorage CLI demo functional
SimpleStorage dApp almost functional (web-worker source file isn't being bundled correctly)

To test locally:

Clone branch
nvm use && yarn at root
yarn build to watch all packages be built
yarn test to watch all tests be built
Head over to ./demos/simple-storage/README.md and follow the instructions for the CLI demo

Yep thanks for pointing that out @morrigan, going to leave it in the tree for now so that everyone doesn't have to configure this themselves upon cloning the code-base. These types of env-config options will handled when this issue is spec'd out: https://github.com/Web3-API/prototype/issues/95

Yep thanks for pointing that out @morrigan, going to leave it in the tree for now so that everyone doesn't have to configure this themselves upon cloning the code-base. These types of env-config options will handled when this issue is spec'd out: https://github.com/Web3-API/prototype/issues/95


Upgraded tfjs-node version to latest

Addresses #3238

Add link to release notes in the commit?

Thanks! I was looking into the build errors locally. Glad they're unrelated.
Relevant release notes (besides trivial version bump): https://github.com/tensorflow/tfjs-node/releases/tag/v1.2.2


Pressing return on live-search shows an error

Fixed by adding a hidden and disabled submit input to stop return on search navigating away from page.


New sql database policy shortterm backup retention policy

Adding Azure Policy for Azure SQL Databases to deploy short-term backup retention policy if it does not exist.
Short-term retention policy is used to address point-in-time retention ability for near-term recovery of Azure SQL databases. It can be set to a number between 7-35 days for this retention policy. The default backup retention period under this policy is 7 days but it varies depending upon your pricing tier.
https://learn.microsoft.com/en-us/azure/azure-sql/database/temporal-tables-retention-policy?view=azuresql

@microsoft-github-policy-service agree

Please update PR content:

Effect Parameters for Policies should always be called "effect". Your naming is great if the parameter is at an Initiative level
Default effect should be "AuditIfNotExists" -> this could be argued.
Policy does not have a name. Add "name": ""

Please update PR content:

Effect Parameters for Policies should always be called "effect". Your naming is great if the parameter is at an Initiative level
Default effect should be "AuditIfNotExists" -> this could be argued.
Policy does not have a name. Add "name": ""

Have changed the parameter to 'Effect', and default effect to "AudtiIfnotExists".
"displayName" is present in the policy as required by the policy definition. The name is optional and usually only defined upon assignment.


[FLINK-30959][table][doc-zh] Improve the documentation of UNIX_TIMESTAMP function for different argument formats

What is the purpose of the change
This pull request improves the documentation of UNIX_TIMESTAMP function for different argument formats for doc-zh.
Brief change log

Modify the system functions doc-zh

Verifying this change
This change is a document change without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@ruanhang1993 Could we can create a new JIRA issue to track the translation for all sql function?  I found some content of these function is outdated.

Thanks for reviewing @leonardBang .
I have opened a new issue and I will raise a new PR for this.


Import Error: No module named 'worker'

  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 365, in execute
----------
----------
----------
    from worker import CeleryWorkerThread
ImportError: No module named 'worker'

I am getting above error when using this package with integration test framework behave. I am new to celery stuff and want to employ it in my automated tests.
One more error I was getting prior to above error is -
ImportError: from celerytest import CELERY_TEST_BACKEND_CONFIG, CELERY_TEST_CONFIG

So I set that in my config app like below -
CELERY_TEST_BACKEND_CONFIG = ""
CELERY_TEST_CONFIG = ""

Am I missing some kind of config here? Any kind of similar suggestions would be really helpful. Thanks :)

Any updates?


Missing python file

Hello!
I was trying to use your instructions but failed :(
It seems that the file
https://raw.githubusercontent.com/Jesssullivan/USBoN/master/QEMU_Raspian.py
does not exist any more.
Any help?

Hey, yeah- for now I'd do the following from a shell to get running quickly with raspbian via qemu:
# make sure you've got qemu-arm and wget-
# for ubuntu:
sudo apt-get install qemu-system-arm -y
# or osx:
brew install qemu wget

# snag the kernel bits:
wget https://raw.githubusercontent.com/dhruvvyas90/qemu-rpi-kernel/master/kernel-qemu-4.14.79-stretch -O kernel-qemu-4.14.79-stretch
# (this one may take a while):
wget https://raw.githubusercontent.com/dhruvvyas90/qemu-rpi-kernel/master/versatile-pb.dtb -O versatile-pb.dtb

# get an image:
wget http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15/2018-11-13-raspbian-stretch-lite.zip

# unzip the zipped image:
unzip "2018-11-13-raspbian-stretch-lite.zip"

# convert to qcow for qemu goodness:
qemu-img convert -f raw -O qcow2 2018-11-13-raspbian-stretch-lite.img stretchlite.qcow2 && qemu-img resize stretchlite.qcow2 +8G

# launch from qcow:
qemu-system-arm -kernel kernel-qemu-4.14.79-stretch -cpu arm1176 -m 256 -M versatilepb -dtb versatile-pb.dtb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfsrtype=ext4 rw" -hda stretchlite.qcow2

Cheers!
-Jess

Hmm, going through it appears this repo is quite broken.  Sorry mate!

Thanks!
It properly works if I follow the steps.
What if you bring this information to the frontpage?
I was looking for the last 2 days a way to do this on my mac, and these instructions were the first to actually work.
Also, what about the files that got downloaded from dhruvvyas90 ? Maybe these files will rotten (like similar examples...) What if you keep a backup to your repository?
Just my thoughts :)

Thanks- updated the master for now.  Cheers!


Application/json validator is the geojson validator

Description
The complexvalidator for json type files is the geojson validator, whichl raises an error for valid json files.
Environment
Master on Ubuntu 16.04.
Steps to Reproduce
There is yet no test to exercise this issue.
Additional Information

I'm not sure what the SIMPLE validation actually does:
passed = data_input.data_format.mime_type in {mtype, FORMATS.JSON.mime_type}

In which situations would it actually catch an error not already checked by the BasicComplex.data_format setter ?
Shouldn't it compare mtype with data_input.data_format.mime_type ?

I was also wandering about the simple validator. My guess was that it should check if the detected mimetype of the input is in the allowed mimetypes.
@jachym Can you put some light on this :)

fixed by PR #403.
"simple validator" discussion moved to #407.


Fix an overflow in SSL_get_shared_ciphers().

Fixes an overflow in SSL_get_shared_ciphers().
The return type of strlen() is size_of. If the length of c->name is over 2^31, the strcpy() in Ln.3236 will trigger a overflow. Furthermore, SSL_get_shared_ciphers is an exported functions. This represents a security issue for end user code that calls this function directly.

Also this would be IMO acceptable with CLA: trivial. Could you please amend the commit message to include CLA: trivial in the commit message body on a separate line?

This PR has the label 'hold: cla required' and is stale: it has not been updated in 30 days. Note that this PR may be automatically closed in the future if no CLA is provided.  For CLA help see https://www.openssl.org/policies/cla.html

This PR has the label 'hold: cla required' and is stale: it has not been updated in 61 days. Note that this PR may be automatically closed in the future if no CLA is provided.  For CLA help see https://www.openssl.org/policies/cla.html

This PR has been closed.  It was waiting for a CLA for 90 days.


allow setting of windows StartType and ServiceType

ServiceType is necessary to create interactive services
StartType helps with #103

Coverage remained the same at 7.561% when pulling 307c535da271f7e8294654d1c43ebf35519b8c6d on boyvinall:windows into 615a14ed75099c9eaac6949e22ac2341bf9d3197 on kardianos:master.

Can you add some documentation for these options?

These look really useful. I am looking into a problem similar to #103 for my service.

Sorry for going quiet last couple of days. Intending to add the docs here soon.


Switch AppVeyor CI to use conda env / requirements.yml

[ ] closes #1127
[ ] tests added / passed
[ ] passes git diff upstream/master | flake8 --diff
[ ] whatsnew entry

@shoyer here I'm reusing existing requirements files.  Is this along the lines of what you were looking for in #1127?
I think this should solve the AppVeyor test failures in #1252, as it should install version 1.2.7 of netCDF4, rather than version 1.2.4.

@shoyer it's entirely possible I'm in over my head here, as I'm not a Windows user, so I can't test this setup locally.  It looks as though using the existing requirements files (which install more than what was specified in the original appveyor.yml configuration file) was probably not a good idea.
If it's a matter of simply creating new requirements files for the Windows environments to mirror what was installed in the original appveyor.yml file, i.e. pip, pytest, numpy, pandas, scipy, netCDF4, matplotlib, and dask, (this time using the conda-forge channel) then I can probably handle that, but anything more is guesswork on my end, so I'd be fine closing this PR.  I was hoping for a straightforward fix.

If it's a matter of simply creating new requirements files for the Windows environments to mirror what was installed in the original appveyor.yml file, i.e. pip, pytest, numpy, pandas, scipy, netCDF4, matplotlib, and dask, (this time using the conda-forge channel) then I can probably handle that

Yes, this is what I would recommend. It's just a little easier to maintain if we keep the requirements separate from the install script.

Cool, I think I have things working now.  Please let me know if you'd like me to make any changes.

Looks great, thanks!


P

"@supabase/supabase-js": "2.32.0"
cli version 1.84.1
https://github.com/supabase/supabase/issues/16427
The recent changes mean that useSupabaseAuthClient is now replaced with useSupabaseClient
this module is currently on @supabase/supabase-js 2.31, so I am using pnpm overrides
I'm not sure if this is a problem with supabase or this nuxt module

https://github.com/stlbucket/vue-nuxt-supa-login-sanity-check
pretty sure this is now in my code somewhere.  the above strips down to vue and nuxt separately bare-bones
closing this issue

https://supabase.nuxtjs.org/get-started#clientoptions
this was due to the new default setting for flowType here


Inexplicably, for seemingly no reason at all, I start glowing randomly.

Whenever I'm just walking or flying, and sometimes even when I just look in a certain direction, I start glowing. I can't find a fix or a reason my shader would do this. I'm on 1.20.2

I have no influence on shaders. Contact the shader dev and or Iris/Optifine, depending on what you are using.


feat: Initial Commit

This is my first CDK construct.
Reference Tutorial - https://www.youtube.com/live/kUfSoFy4Mgg

branch deleted


yaml.DeleteProperty create a malformed yaml

Hi
Thanks for this wonderful proyect :)
I did another test with org.openrewrite.yaml.DeleteProperty and rewrite-maven-plugin:4.13.0
My recepy is
---
type: specs.openrewrite.org/v1beta/recipe
name: com.yourorg.AddDependencyExample
displayName: Add Maven dependency example
recipeList:
  - org.openrewrite.yaml.DeleteProperty:
      propertyKey: com.company.other.name
      coalesce: true

Source yaml is
com:
  company:
    other:
      name: project-test
      logging:
        kafka:
          uniqueTopic: a-topic
#TODO bug pending          topic: a-topic #This property disappear
          properties:
            one.other: true

I did mvn rewrite:run
[INFO] --- rewrite-maven-plugin:4.13.0:run (default-cli) @ test-openrewrite ---
[INFO] Using active recipe(s) [com.yourorg.AddDependencyExample]
[INFO] Using active styles(s) []
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by nonapi.io.github.classgraph.utils.ReflectionUtils$StandardReflectionDriver$1 (file:/C:/Users/n129586/.m2/repository/io/github/classgraph/classgraph/4.8.123/classgraph-4.8.123.jar) to method sun.management.RuntimeImpl.getInputArguments()
WARNING: Please consider reporting this to the maintainers of nonapi.io.github.classgraph.utils.ReflectionUtils$StandardReflectionDriver$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Validating active recipes...
[INFO] Parsing Java main files...
[INFO] Parsing Java test files...
[INFO] Parsing YAML files...
[INFO] Skipping properties files because there are no active properties recipes.
[INFO] Skipping XML files because there are no active XML recipes.
[INFO] Skipping Maven POM files because there are no active Maven recipes.
[INFO] Running recipe(s)...
[WARNING] Changes have been made to src\main\resources\config\application.yml by:
[WARNING]     org.openrewrite.yaml.DeleteProperty
[WARNING] Please review and commit the results.

then I get a malformed yaml
com.company.other.logging.kafka:
  uniqueTopic: a-topic
g pending          topic: a-topic #This property disappear
          properties.one.other: true

The complete example is in https://github.com/ruben-garciapariente/test-openwrite/tree/62825927e1a11268dfdb1f261cb9ac12dc56858e
Thanks & regards

Hey @ruben-garciapariente -- after having merged this PR (https://github.com/openrewrite/rewrite/pull/1132), I tried out the example you provided in https://github.com/ruben-garciapariente/test-openwrite/blob/62825927e1a11268dfdb1f261cb9ac12dc56858e/src/main/resources/config/application.yml and the result looks more appropriate now:
Before:
com:
  company:
    other:
      name: project-test
      logging:
        kafka:
          uniqueTopic: a-topic
#TODO bug pending          topic: a-topic #This property disappear
          properties:
            one.other: true

After:
com.company.other.logging.kafka:
  uniqueTopic: a-topic
#TODO bug pending          topic: a-topic #This property disappear
  properties.one.other: true

This ought to be things taken care of for you. If you try out the latest changes and things aren't to what you're expecting, definitely reach out again and let us know. Thanks again for logging this, it helps 👍


@UnitOfWork not always committing transaction after call

Env Details
Dropwizard version: 2.0.25
Hibernate version: 5.5.8
DB: Postgres 9.6 and 13.4 on AWS RDS.
Java version: 11
Postrgres JDBC driver version: 42.3.1
Hibernate Entity definition
@Id
@Column(name = "id", updatable = false, nullable = false)
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
.......
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || Hibernate.getClass(this) != Hibernate.getClass(o)) return false;
Issue issue = (Issue) o;
return id != null && Objects.equals(id, issue.id);
}
@Override
public int hashCode() {
	return getClass().hashCode();
}

Details
I have an App that receives two sequential rest calls.
The first issues a single "delete where xx = 'yy'" on the entity definition above.
The second does processing and on-demand inserts to the same entity.  Since the entity has the @GeneratedValue(strategy = GenerationType.IDENTITY) Hibernate issues an insert directly after the saveOrUpdate call.
The issue, however, is that the delete call sometimes gets issued to the db near the end of the second rest call.
This causes the recent inserts to be deleted.
It doesn't happen every time - which I think is down to how many sets of these calls are being done.
Main Steps taken to try and resolve

Tried different dropwizard version(initial version had Hibernate 5.5.2) and jdbc driver version.  No difference.
Added a flush after the executeUpdate statement.  No effect.

Workaround found
If I manually manage the transaction and use @UnitOfWork(transactional = false) and manually call transaction.commit() the problem is fixed.
Is it possibly a configuration issue?
Seems like a bug.  Any thoughts on the issue?

The issue you describe sounds mostly like a phantom read problem in the deleting transaction. As the PostgreSQL documentation describes, the default isolation level in PostgreSQL is Read Committed. With this configuration, phantom reads can occur.
Please try setting the transaction isolation to Serializable to exclude an isolation violation. This can be done in the Dropwizard database configuration with the property defaultTransactionIsolation.

I don't think it's an isolation issue. As there are no read calls.  There's two sql queries (in separate non-concurrent transactions):
delete from
insert into.

As you described in the issue definition, the REST calls and therefore the managed transactions interleave. But also when the transactions commit at different times, the database scheduler may schedule them interleaved.
If the insert call is early in the transaction, the tuple gets inserted into the index. When the delete call happens later, it sees the newly inserted entry in the index while identifying the tuples it should delete. As the delete query must find the tuples to delete, it must read the index and therefore can suffer of phantom reads.
I'm not sure, on which side the bug is as interleaved calls are hard to reproduce and test. But I just wanted to make sure it's not a database issue.

No, the rest calls are sequential.
Hibernate issues the delete statement to the db too late (seen via hibernate logging showing the delete statement).
Sorry, for the confusion - should have explicitly said hibernate issues the delete too late.

Ah, okay. Can you frequently reproduce this, i.e. set up a test?

It doesn't happen too frequently in our Env. I'd say it happens like once in every three or so then almost always occurs every time after that first instance.
I can see about trying to create a simple test to isolate further on Mon or so.

@chris-cynation I set up a basic test with calls for deleting and inserting entities: https://github.com/zUniQueX/dropwizard/commit/f78643fb158c5a1cb861d288d6076b2bc4464d03. Is this code comparable to your situation?
In this test I cannot reproduce reordering of the db calls with many iterations (here 500). However, if done the requests asynchronously from the client and therefore interleaved, the delete likely happens after the insert call.

Hey,
Apologies, been busy.
I've tried to reproduce but can't as well. I don't think creating it in a similar docker is going to make a difference as I'm running the same dependencies and Java version.
I'll keep an eye out for other instances of the problem.
I've got a workaround where I create a transaction and commit manually.

Just to clarify: could it be, that your REST calls get invoked asynchronously in any way?

No.
It's possible other rest calls are occurring during that time period that are unrelated. e.g. different endpoint and entity.
The caller of the rest calls is a JS app that does an await.
I can confirm it's synchronous by looking in the logs.

I also checked that a return type is specified on the REST endpoint as that can cause issues.

Okay, thanks for clarifying this.
This seems to be hard to fix then. Since the tests cannot directly reproduce this situation, I'm not completely sure, that this is a framework issue.
The only difference I can state without further code is that the transaction commit happens later, if the transaction is managed by Dropwizard, as the Jersey processing happens before. But the commit should happen before your next call is triggered, if the calls are sequential.

Yeah, I've tried to reproduce it with a complete test setup which is very very similar - but no luck.
I'll close the ticket now.
If I come across the issue again I'll open a new ticket with hopefully the ability to reproduce what's going now.


nvm not found

When I run nvm --version, upterm (0.4.1) throws Upterm: command "nvm" not found.. It works in the default Ubuntu terminal.

I think this question has been raised before in #975, you can use n as an alternative.

Thank you. I'keeping this open (unless the maintainers want to close it) because, even if there is an alternative, nvm is quite popular and could be nice to be able to use it.

Yes, I am more concerned about virtualenv issue but it seems having the same problem. If they can solve virtualenv issue then nvm problem can resolve also.

any new updates how to use nvm?


now alias fails for AWS Route 53

> test.<MY_DOMAIN>.com is a custom domain.
> Verifying the DNS settings for test.<MY_DOMAIN>.com (see https://zeit.world for help)
> Error! Unknown error: Error: Verification required: Please add the following TXT record on the external DNS server: _now.<MY_DOMAIN>.com: eddcd0.................a153d9dda9fd92482150

The error message is clear, but how exactly do I configure the text record? Do I add it for the top-level domain or subdomain? Is the _now.<MY_DOMAIN>.com included in the text record?

The top-level domain and only the token in the record.

this doesn't work for me...
I have added TXT record for a top domain .ai and in value I have put the token from that error message but still cannot verify

Can you correctly see the the value with e.g dig: dig _now.yourdomain.ai TXT +short
E.g. with my personal domain:
$ dig _now.jarmoisotalo.com TXT +short
"fb4bac214f44487a21b80d5b1a72cf2a0e0fca63df902021c8d09a42ad0a04a1"

Yes


progressive loading on .jpg

http://php.net/manual/en/function.imageinterlace.php

I don't like the idea of progressive jpeg by default.


patroni: 4.0.3 -> 4.0.4

Things done

patroni: format
patroni: 4.0.3 -> 4.0.4

Changelog: https://github.com/patroni/patroni/blob/v4.0.4/docs/releases.rst
cc @de11n @invokes-su

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 359334

x86_64-linux

:white_check_mark: 2 packages built:

patroni
patroni.dist

aarch64-linux

:white_check_mark: 2 packages built:

patroni
patroni.dist

x86_64-darwin

:x: 2 packages failed to build:

patroni
patroni.dist

aarch64-darwin

:white_check_mark: 2 packages built:

patroni
patroni.dist

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 359334

x86_64-linux

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist

aarch64-linux

:white_check_mark: 2 packages built:

patroni
patroni.dist

x86_64-darwin

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist

aarch64-darwin

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 359334

x86_64-linux

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist

aarch64-linux

:white_check_mark: 2 packages built:

patroni
patroni.dist

x86_64-darwin

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist

aarch64-darwin

:white_check_mark: 6 packages built:

patroni
patroni.dist
python311Packages.python-etcd
python311Packages.python-etcd.dist
python312Packages.python-etcd
python312Packages.python-etcd.dist


Update web-deployment.yaml

incorrect secret key name, should be postgresql-password

What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[ ] DCO signed
[ ] Chart Version bumped
[ ] Variables are documented in the README.md

Can you change issue title to reflect charts name, also chart's version needs a bump


Option to check & allow existing port usage in jest-dev-server?

 It'd be useful to allow developers and CI machines to have a local server running that was started before tests run. That way, if we want to run several runs of tests, we don't have to start the server up each time: the tests can run immediately. We also wouldn't have to manually close our servers before running tests
Can the dev server config logic have a flag to check for the existing port being used, and not start the server if so?
I'd be very open to adding this in. :)

Still getting EADDRINUSE with userdPortAction: "ignore"?
$ ./node_modules/.bin/jest
Determining test suites to run...[0622/101431.683618:ERROR:gpu_process_transport_factory.cc(1039)] Lost UI shared context.

DevTools listening on ws://<IP_ADDRESS>:58544/devtools/browser/e60a6086-5262-48a9-872b-98cd5a037db7
events.js:137
      throw er; // Unhandled 'error' event
      ^

Error: listen EADDRINUSE <IP_ADDRESS>:8083
    at Object._errnoException (util.js:1003:13)
    at _exceptionWithHostPort (util.js:1024:20)
    at Server.setupListenHandle [as _listen2] (net.js:1366:14)
    at listenInCluster (net.js:1407:12)
    at doListen (net.js:1522:7)
    at process._tickCallback (internal/process/next_tick.js:152:19)
    at Function.Module.runMain (module.js:703:11)
    at startup (bootstrap_node.js:193:16)
    at bootstrap_node.js:617:3
error Command failed with exit code 1.

Same experience as @anton164 with 'ignore' option here

usedPortAction must be define in server section.

Not sure I understand the response/issue-close here...in my case I believe usedPortAction was defined in server section? Before I defined usedPortAction explicitly the default option of ask did not  function as described in documentation either.

Yupp, I'm experiencing the exact same issue as Nathan.
On Sat, Jun 23, 2018, 10:28 Nathan Wingert<EMAIL_ADDRESS>wrote:

[image: jp_server_port]
https://user-images.githubusercontent.com/30574536/41807446-62c2b33a-768c-11e8-8fee-70b2c33054e7.png
Not sure I understand the response/issue-close here...in my case I believe
usedPortAction was defined in server section? Before I defined
usedPortAction explicitly the default option of ask did not function as
described in documentation either.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/smooth-code/jest-puppeteer/issues/71#issuecomment-399652106,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABSWacnbsJNnm0zqtmBxgOuIl3XP2GfUks5t_fxFgaJpZM4UVaYD
.

@neoziro im also having the same problem here. my usedPortAction IS defined in the server section of the configs, but the option is just ignored, same as @nwingert posted.


No reveal buttons but need check buttons

First of all, incredible work! This has been awesome ride. I wonder if you are familiar with any python implementations of dense crossword filling algorithms (take a list of words and fit them on a X by X grid)?I have seen a few but don't think they do a good job.
Now the main issue:
I want to tell the player if they have done anything wrong, but using the Check buttons. But I don't want to reveal the answer. Obviously they can look into the source code to see that, but that is besides the point. I saw an option to replace all the letters of the puzzle with Os and that disables Reveal buttons. But that would disable check buttons too, right? How would the puzzle check for correctness when the ground truth is missing?

Thanks!
Sorry, not familiar with the filling implementation that you're looking for.
You can easily hide the "Reveal" buttons and do any other customizations by using a customizeExolve() function.
For example, this 3x3 puzzle will not render the Reveal* buttons:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<link rel="stylesheet" type="text/css" href="https://viresh-ratnakar.github.io/exolve-m.css?v1.47"/>
<script src="https://viresh-ratnakar.github.io/exolve-m.js?v1.47"></script>
<title>Exolve: Reveal Buttons Hidden</title>
</head>
<body>
<script>
function customizeExolve(puz) {
  puz.revealButton.style.display = 'none';
  puz.revealAllButton.style.display = 'none';
}
createExolve(`
exolve-begin
  exolve-title: Exolve: Reveal Buttons Hidden
  exolve-width: 3
  exolve-height: 3
  exolve-grid:
    ACE
    R.V
    TOE
  exolve-across:
    1 Top card (3)
    3 Foot part (3)
  exolve-down:
    1 Creative work (3)
    2 The night before (3)
exolve-end
`);
</script>
</body>
</html>


fasta-to-gfa conversion

Thank you for making this tool available. I found when I converted a fasta file to gfa, the scaffold number increased in the output file (a few scaffolds turned to multiple 'segments'). I was wondering if there is a flag that we can use to keep the output file having the same scaffolds as the input fasta file?

hi @YanWangTF
Thanks for reaching out. Can you provide an example? Scaffolds are supposed to be represented as multiple segments (S lines) connected by gaps (J lines) in GFA. This is what the format specifies: http://gfa-spec.github.io/GFA-spec/GFA1.html and it makes sense, segments should not have N bases.
Best

Hi @gf777, thank you for the clarification! I used the gfastats to convert a genome assembly (scaffolded using HiC data) from FASTA to GFA, which worked smoothly. While I was using Bandage to draw the graph, it produced more scaffolds (nodes) on the graph. For example, it has 22 scaffolds while showing 27 nodes. Or do you have an alternative tool to visualize the GFA that draws scaffolds instead of segments or that presents the links between segments in the meantime? Thank you.

hi @YanWangTF
Have a look at https://github.com/asl/BandageNG , the currently maintained "nextGen" version of bandage. It supports also J lines

Thank you @gf777! The BandageNG worked perfectly with the output from gfastats this time with links and merging options. Much appreciate your help with this.


Guides - Add Decision Tree to Shimming Doc

It can be difficult to know how to shim a dependency since there are multiple methods and considerations to take into account. I built a decision tree to help me. Might be useful to have in the docs.

shim-decision.pdf

Great idea. Do you have the original file somewhere? Easier to tweak to fit the style better. 👍

It's an omnigraffle doc. Do you want that? The PDF I attached should be vector based, so it should work in Illustrator.

Yeah. If I remember right, you might be able to get svg export in the newest version (I have it). Anyway, push somewhere and I'll convert and hook it up with the site.

A different approach would be to incorporate https://github.com/knsv/mermaid and keep the source of the diagrams in a simple text file.
Can we write a loader for mermaid graphs which is executed when including those graphs in the markdown file?

👍 for mermaid-loader. It would probably have to run before any markdown bits and return reference to the generated image.

The markdown-loader does not call other loaders for including images?

It likely does, but that does not help as there's no markdown-loader in the current setup.

Ah, the markdown processing is in utilities/markdown.js? Could we call mermaid in the renderer.image function? The drawback brings in a somewhat unrelated process.

@simon04 You would need to intercept the block containing mermaid syntax and then process through mermaid, but that's the place. A better, but also more intensive solution, would be to rework Antwar content pipeline so that it's entirely webpack based. Then it would be all loaders. That's something for me to think about as it might not be entirely trivial.

Using @rouzbeh84's example from #469 I created the chart from above using mermaid. I'm going to close this now as all we really have to do is push #469 forward at some point and then we can include this (also this discussion just ended up being about Mermaid and implementation anyhow). I'll link this issue there and I'm going to add a checklist in the main description for what we have and what steps are left.

For reference:

The styling still needs some work but that's listed as one of the steps in #469.


Fix links in README.md

Some of the links in README.md to sections of documentation point to blobs. These need to be replaced with links within the hosted documentation.

Fixed in 155fbab


What are experience (xp) points for?

Currently just to give a feeling of progression and for us to be able to restrict certain gear to higher level players. Later, it's for more stuff.


rework forced_allow_origin_value option to always set the header

Fix #66 (cache issues)
Because the forced_allow_origin_value option is only useful for the cache issues... but it doesn't solve it yet.
Note: As @Seldaek said, we could also set a Vary: Origin header to workaround the cache issues instead of merging this PR (if the cache system supports it).

@Seldaek I updated the PR, now it won't change anything for those who don't use the forced_allow_origin_value option.
Could be a v1.5.1 don't you think so? 😄

Thanks, 1.5.1 is out https://github.com/nelmio/NelmioCorsBundle/releases


[OneNote] Sign in error: typeerror cannot read properties of undefined (reading 'value')

After a correct login to my  M635 account, I face a TypeError blocking the enumeration of my Notebook  :
typeerror cannot read properties of undefined (reading 'value')

I don't have the skills to debug but I am happy to assist if needed. Did not find any similar issue in bounty.
Occurs on Mac and Win11

Hello I experiment the same with macbook m3 (16-inch, Nov 2023) , I have HAR capture and console logs I can provide. HAR is quite sensitive so let me know.
Sonoma 14.4.1 (up to date when writing this)
tested with edge,chrome,safari
deactivated ublock, and few unrelated plugin, reduced tracking prevention from strict to balanced
nothing did it

Can confirm error on Linux and Windows 10 too.

Same error for me on Firefox on Windows 10. Auth worked initially, however the Importer did not import properly; subsequent attempts to auth again then failed afterwards.

Console log below, it's microsoft returning 429 (too many requests) after 5+ sign-ins. Importer doesn't seem to store and reuse the login token between imports even when pressing the "back" button after an import.
OneNotes api is throttled to 5 concurrent requests per app per user: https://learn.microsoft.com/en-us/graph/throttling-limits#onenote-service-limits
Failed to load resource: the server responded with a status of 429 ()
An error has occurred while fetching an resource:
{
    "error": {
        "code": "20166",
        "message": "The app has issued too many requests on behalf of this user in a short time period.",
        "innerError": {
            "date": "2024-04-19T14:39:21",
            "request-id": "c3bc1789-bd97-485b-8dfe-3bf943047d2e",
            "client-request-id": "c3bc1789-bd97-485b-8dfe-3bf943047d2e"
        }
    }
}
An error occurred while we were trying to sign you in. Error details:  TypeError: Cannot read properties of undefined (reading 'value')
    at Mr.showSectionPickerUI (plugin:obsidian-importer:214:1374)
    at async Mr.authenticateUser (plugin:obsidian-importer:214:716)

This bug would be that the error isn't properly displayed to the user. The actual solution would be #117

I'm not even getting one import to work before getting this error. My issue does not to be related to a timeout

I'm not even getting one import to work before getting this error. My issue does not to be related to a timeout
I have the same output as @fjacquet , but without a glance at it working at all.
so the error

{"An error has occurred while fetching an resource:",{"error":{"code":"10008","message":"One or more of the document libraries on the user or group's OneDrive contains more than 5,000 OneNote items (notebooks, sections, section groups) and cannot be queried using the API. Please make sure that none of the user or group's document libraries contains more than 5,000 OneNote items. Browse to this page for more information: https://blogs.msdn.microsoft.com/onenotedev/2016/09/11/onenote-api-calls-fail-with-a-large-number-of-items-in-a-sharepoint-document-library/","innerError":{"date":"2024-04-26T00:11:47","request-id":"81b24599-d7a6-4ed7-b569-036d3dfb4826","client-request-id":"81b24599-d7a6-4ed7-b569-036d3dfb4826"}}}

to note : " the user or group's OneDrive contains more than 5,000 OneNote items .. cannot be queried ..."
=> https://blogs.msdn.microsoft.com/onenotedev/2016/09/11/onenote-api-calls-fail-with-a-large-number-of-items-in-a-sharepoint-document-library
In essence I agree with @GregHib , pointing out the limitation and a link to the above would be a solution AFAIK.
now I need to find or make something to split my object across multiple users :D

Same issue on Chrome and MacOS 10.13. I've imported twice successfully and from the third time this error occurred. I've been logged in my OneNote account all time.

Same issue here when authing through an Opera browser - some data imports, then it consistently fails.  Retrying just leaves me with this error and no further hope of importing...

Importer just updated to 1.6.4 and the same error remains...

I'm experiencing the same problem and can confirm that my console also says it is due to a 429 (Too Many Requests) error.
It occured each time after the import of a rather large notebook that failed halfway through (also because of too many requests). After waiting a day and trying again, the login worked but I got to the same point again after trying to import a larger notebook.
I'm on Windows / Chrome.
Is there a solution or a workaround to this problem? Doesn't seem like there's a whole lot one can do as an end user and it's kinda frustrating.

Is there a solution or a workaround to this problem? Doesn't seem like there's a whole lot one can do as an end user and it's kinda frustrating.

In my case I split my notebook into smaller sections and imported the sections one at a time, with a break each time I started getting 429 errors.

@varju Oh alright, tysm for the tip! Have you found out what roughly the maximum amount of pages is that can be handled without throwing an error? And by splitting do you mean just split the selection of the import or literally split the notebook?

@varju Oh alright, tysm for the tip! Have you found out what roughly the maximum amount of pages is that can be handled without throwing an error? And by splitting do you mean just split the selection of the import or literally split the notebook?

In my case I only had one large notebook with several large sections. I rearranged my sections into chunks of about 50 notes each. I think I was able to import 2 or 3 of these sections before it started rate limiting again. Larger than 50 and the initial section query tended to time out, aborting the import with a different hidden error.
Problems were compounded for me because it turns out many of my notes had never synced to the cloud, causing 404s during import. I had to manually force syncing on each note, likely consuming more of my quota.

i also have this problem, meaning I cannot import my notes and therefore not use and switch to Obsidian.
Wouldnt it be in the interest of Obsidian too, to solve this problem after nearly 3/4 of a year???

Unfortunately, there is nothing we can do about the Microsoft limitations on how fast we can import notes through the API, but we can at least make the messaging and retries a bit better.
I've got some changes on a branch that anyone is welcome to try out and report back how it goes:

Allow remembering your signed in account
Renew access token if imports take longer than the 1 hour lifetime of the token
Add option to skip previously imported notes (only considers notes imported since this update)
Add improved error messaging when rate limiting is occurring.

Please give it a try and let me know if you encounter any issues. You can try it by replacing the main.js in your vault/.obsidian/plugins/obsidian-importer with this one: main.zip

I started testing while everything is working.

Can you please elaborate on what you mean by "authorization fails"? Is that after hitting the rate limit? Does it fail every time?

Oh excellent, thank you for testing!

There are errors "429", also noticed the text, not all was copied, in several notes, but I think there is no point in debugging and testing.

Duplicates of notes and files are being created due to my mistake. I ran the import at home and at work, on different computers. If I had used just one, I think it would have been fine. I'll test it later this week. I have a lot of notes and files attached to them, approximately 7,000 files and 900 notes (rough estimate). By the end of the import, there were 16,000 files with duplicates. I didn't count the notes, and not all notes were duplicated, but some were duplicated multiple times.
The script is good anyway, despite my mistake.)))
Thank you for the development and the time spent.

Thanks very much for continuing to test, this is very helpful. Yeah, as you discovered the tracking of which files were imported is local to the instance of Obsidian because I didn't want to add an indicator into the files themselves.
I'll go ahead and release this update, and if there are specific issues that come up afterwards we can track them in a new issue.

Closing with 1.6.11


ntfs-3g-mac 2021.8.22

Signed-off-by: Yurii Kolesnykov<EMAIL_ADDRESS>

Thanks much, @yurikoles and @EricFromCanada !


SVG icons doesn't show images inside

Godot version
4.2-dev6
System information
Godot v4.2.beta1 - Windows 10.0.22621 - Vulkan (Forward+) - integrated AMD Radeon(TM) Graphics (Advanced Micro Devices, Inc.; 31.0.12044.3) - AMD Ryzen 3 5300U with Radeon Graphics (8 Threads)
Issue description
Checking changes on the Godot v4.2.dev6, also just downloaded beta1 and the same, I advertised that the icon I used in my addon was not rendered correctly.  The problem was that this SVG contained a raster image inside and some vectors over.  The raster image (body) disappeared from the rendered SVG.

The SVG was created with Inkscape, and it was correctly rendered with the last stable release of Godot 4.1.
This is how it is rendered correctly with 4.1.

Steps to reproduce
I attached a minimal project, but, really, you only have to get the SVG from there.
Just put the .svg as the icon of the project, and you will see the bad render even in the project manager window.
Minimal reproduction project
a minimal project to reproduce
svgtest.zip

Please check in https://www.thorvg.org/viewer to confirm whether this is a ThorVG regression.

It seems to be rendered perfectly there:

the picture section is detected and rendered.

This was one of the features I requested for godot 4.0 will poke around the code and from thorvg. I suspect a jpeg code oath was modified. @hermet

the robot-body is desrcibed with png image:
    <image
       width="213.51875"
       height="192.88126"
       preserveAspectRatio="none"
       xlink:href="data:image/png;base64,iVBORw0KGgoAAAA ...

Probably, the godot disabled the png loader of the thorvg in building.

Yea the png loader was disabled, I'll fix it in 0.11.1

... I suspect a png or jpeg code path was modified.

No the png/jpg loader was disabled because of size complaints, I will re-enable it in 0.11.1


Struct data type

This pull requests has

Supports STRUCT data type
Refactor the test for row/column decoder
Fix fragile integration test temporarily

Sometimes test passes, sometimes doesn't
I couldn't find out the root cause of failure, but looked like t.Parallel() causes this. So removed it temporarily.

Mike, could you take a look at this pull request?

Thanks Mike!


useAuthState initialising not working

Hi,
I've been trying the useAuthState hook and looks like the initialising variable is not working.
Here is the code:

export default function Login() {

  const { initialising, user } = useAuthState(firebaseAuth());
  const login = () => {
<EMAIL_ADDRESS>'test123');
    };
  const logout = () => { firebaseAuth().signOut(); };

    if (initialising) {
      return (

          <Spinner marginX="auto" marginY={50} />

      )
    }
    if (user) {
      return (

Current User: {user.email}

          <Button onClick={logout}>Log out</Button>

      );
    }
    return (
      <Button onClick={login}>Log in</Button>
    )
  }

The login/logout works fine, but I've confirmed that the initialising block is never reached.
Thanks in advance,
Geraldo

Thanks for flagging - this was an error on my part when refactoring to use the underlying useReducer hook rather than useState.
I've just published v0.3.2 which fixes the problem.
Please let me know how you get on in general and whether you have any feedback on how well they work, what's missing, that sort of thing...

Sorry for the delay. Thanks a lot, it's working great for me.


Thoughts on stdin for interactive prompts?

This is a really cool project!
I was looking at it to see if it could help make @aws-amplify/cli testable.
The problem I see with the Amplify CLI is that it's interactive with prompts.
Before I found shellac, I was working on a tool that did the following:

Read a markdown file (e.g. README.md) and extracted the shell code fences.
Create a /tmp directory and execute these shell scripts sequentially.
Finally, a test assertion validates if the resulting output matched expectations.

The problem I ran into was trying to document the commands in a scriptable way:
$ amplify init
Enter a name for the project: nextamplified

# All AWS services you provision for your app are grouped into an "environment"
# A common naming convention is dev, staging, and production
Enter a name for the environment: dev

Then, following this block would be a comment:
<!-- shell:input
nextamplified[ENTER]
dev[ENTER]
-->

Did you have any thoughts on how shellac could support testing interactive CLIs?

No but that's exactly where I want to take this project!
Have you seen any of these projects?

exactly
bats
Expect
cram
aruba for prior art.

Of these, Expect seemed the most geared towards interactivity, but almost too much so. Once I realised I could use function callbacks inside a JS tagged template DSL I felt it was worth inventing something new.
I do love the idea of a README test file but in the end I find Jest files readable enough so didn't pursue it. And I guess that's the biggest concept I'm exploring right now—if you use all of JS's tooling & a DSL, how good can you make the API?
One thing I've tried to do so far is avoid any inherent testing concerns inside Shellac, even though that's the whole point. But being able to drop into JS land in order to call expect for me feels really powerful.
Happy to toss around suggestions, so far I think Shellac has proved itself to be a great idea, keen to see it expand to cover more use cases.


[FEATURE] Anthropic: prompt caching

Implement prompt caching in Anthropic integration

Hi @langchain4j , Could you provide some more details regarding this feature.

@Devansh-Rastogi hi, I've added a link

Hey @langchain4j  assign this issue to me.. will implement this feature..

@Devansh-Rastogi awesome, thank you! Do you already have a design in mind?

Keep in mind that Gemini also has prompt caching, so any design should work for both!
Google Gemini cache documentation @
https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-create

@Devansh-Rastogi awesome, thank you! Do you already have a design in mind?

Yes, actually I was thinking of creating a separate base class for cache messages implementing ChatMessage interface.. and that could be handled in different tools like gemini, anthorphy etc providing separate implementation for each of them as the way to use the cache is different in  various AI tools..

@langchain4j and @stephanj do you have any other suggestions for implementation.. please do share

The cache with Anthropic is only valid for 5 minutes, not sure about Gemini.  Auto re-populating the cache when expired would probably be a nice (advanced) feature?

@Devansh-Rastogi not sure I understand your proposal, could you please provide more details? Some pseudo code would also be great

The cache with Anthropic is only valid for 5 minutes, not sure about Gemini. Auto re-populating the cache when expired would probably be a nice (advanced) feature?

The gemini by default has ttl of 60 minutes. But can be extended based on a parameter

@Devansh-Rastogi not sure I understand your proposal, could you please provide more details? Some pseudo code would also be great

Sure will provide the pseudo code in sometime

Some extra info about the two prompt caches

From what I read in the documentation, the cache is still in beta and to use it you have to add the anthropic-beta header.
curl --request POST \
  --url https://api.anthropic.com/v1/messages \
  --header 'Content-Type: application/json' \
  --header 'anthropic-beta: prompt-caching-2024-07-31' \
  --header 'anthropic-version: 2023-06-01' \
  --header 'x-api-key: KEY' \
  --data '{
	"model": "claude-3-haiku-20240307",
  "messages" : [ {
    "role" : "user",
    "content" : [ {
      "type" : "text",
      "text" : "Hi",
      "cache_control": {"type": "ephemeral"}
    } ]
  } ],
  "max_tokens" : 1,
  "stream" : false
}'

In the implementation, do I leave the anthropic-beta field without it or add it as a header option?

Hi @Claudio-code, you can set anthropic-beta header like this:
AnthropicChatModel.builder()
    ...
    .beta("prompt-caching-2024-07-31")
    .build();

To implement cache on system message need change implementation because now it concat all system message on string, but to send cache need send array like showed documentation

Finished implementation now we can send system message with cache and user messages with cache

Partially implemented by @Claudio-code in https://github.com/langchain4j/langchain4j/pull/1826.
Currently, caching of system message and tools is possible in 0.36.0: https://docs.langchain4j.dev/integrations/language-models/anthropic#caching

Please  be aware that caching of user messages is not supported yet!


Treat Egress Channel as NOT_CONNECTED on InvalidChannelException

A channel that was resolvable at connection time can subsequently become unresolvable which causes a cascade of errors when a node becomes leader or the cluster is coming up after shutdown.
 io.aeron.exceptions.RegistrationException: errorCodeValue=1 java.net.UnknownHostException: could not resolve endpoint address: xxx.host:9020
    at io.aeron.ClientConductor.onError(ClientConductor.java:169)
    at io.aeron.DriverEventsAdapter.onMessage(DriverEventsAdapter.java:85)
    at org.agrona.concurrent.broadcast.CopyBroadcastReceiver.receive(CopyBroadcastReceiver.java:120)
    at io.aeron.DriverEventsAdapter.receive(DriverEventsAdapter.java:56)
    at io.aeron.ClientConductor.service(ClientConductor.java:718)
    at io.aeron.ClientConductor.awaitResponse(ClientConductor.java:762)
    at io.aeron.ClientConductor.addExclusivePublication(ClientConductor.java:407)
    at io.aeron.Aeron.addExclusivePublication(Aeron.java:253)
    at io.aeron.cluster.ClusterSession.connect(ClusterSession.java:130)
    at io.aeron.cluster.ConsensusModuleAgent.becomeLeader(ConsensusModuleAgent.java:1113)
    at io.aeron.cluster.Election.leaderTransition(Election.java:653)
    at io.aeron.cluster.Election.doWork(Election.java:209)
    at io.aeron.cluster.ConsensusModuleAgent.doWork(ConsensusModuleAgent.java:272)
    at org.agrona.concurrent.AgentRunner.doDutyCycle(AgentRunner.java:268)
    at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:161)
    at java.base/java.lang.Thread.run(Thread.java:844)

This was the simplest way I could see to solve this.
I wonder whether the behaviour might be handled instead by the MediaDriver which could continue to create a 'null publication' that never connects?

I think we need to consider some sort of timeout on this so that it is remove eventually, like it does in the initial connect case. For example, if ingress is still working then keepalive is not applied if egress is not connected.

I was working on the basis that the session would be timed out anyway because there would be no messages would be coming in.

We need to consider if there is so it is covered.


DEV: remove autogenerated names

remove autogenerated owner and signal names
add unique id to owners and signals
remove componentName to instead use the name prop (component owners can be identified by the fact that props field exists)

My goal is to separate the autogenerated ("c-1-1-1-1-1...", "s-951856283") names from explicit user names, as the user names are the only ones I want to display to users. But the generated names occupy the same field.
The individual owners can be identified by a simple id, which I'm doing now, but it can be added here as a replacement for the generated names.
Alternatively id and name could still be merged. e.g. with a format: <id>:<name>. But that's just serialization, so probably a task for the devtools.
The hashValue and serializeGraph functions are interesting but do not have a use now.
They can still be called from the DEV object if necessary.
Not sure if I'm not missing something here... especially around stores, as I haven't gotten to them in the devtools so I don't know yet what's important there.
So it'll be useful for me, but the details probably will change in the future.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

21 of 23 (91.3%)  changed or added relevant lines in 3 files are covered.
17 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-1.2%) to 87.685%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

packages/solid/store/src/mutable.ts
4
5
80.0%

packages/solid/store/src/store.ts
4
5
80.0%

Files with Coverage Reduction
New Missed Lines
%

packages/solid/src/reactive/signal.ts
17
87.75%

Totals

Change from base Build<PHONE_NUMBER>:
-1.2%

Covered Lines:
1238

Relevant Lines:
1340

💛  - Coveralls

How pressing is this? I feel like I should do this at a clean break like 1.6 version rather than sneak it in right now. Maybe nothing here is too destructive. Just it would be best to point at a specific minor than than some random patch version.

Not pressing at all, those are just names
tbh I just wanted to get my thinking out and see how it fits in the solid code

I'm closing this for now so that the state persistance for HMR can be implemented first. I'll look at the names again after that.


Freeze UI

The code in this line takes a very long time to execute
https://github.com/chooyan-eng/crop_your_image/blob/7bf38421e5469b212e9d7a2dc574d29f8f6a4f4f/lib/src/crop.dart#L192
I think it must be async or move to Isolate.

@DrobyshevAlex
Thank you for your suggestion!
I'll try this to enhance this package.

I've done this issue on #46 and published as updated version 0.6.0.


Migrate to ESLint

TSLint is deprecated in 2019 as mentioned in https://github.com/palantir/tslint/issues/4534

I can take this one

@ozylog Go ahead brother 👍


whether to reduce loss when tracking step/epoch training loss

I am confused that if i want to track the true loss of each training step/epoch, should I use accelerator.gather_for_metric() like this script did for reducing among processes?

however, this example script just simply add loss at each step to get epoch loss, can someone help me figure out which method is correct or why different procedures are used to train loss and eval metric seperately? Will averaging loss across all processes make a difference to see whether the training of NN is going well?

Moreover, if i use multi gpus & gradient_accumulation, the the total_batch_size will be batch_size * num_processes * gradient_accumulation_steps, so the correct formula to get loss of each step should be loss = accelerator.gather_for_metric(loss).mean().item()?

I also see someone use loss = accelerator.gather(loss.repeat(batch_size)).mean().item() link , is it just an old verison&bug to fix?

I would be very appreciate if someone can answer me these questions!

Regarding your first question, I think you're confusing two things. In the first script that you link, the loss is not gathered. What's being gathered are predictions and targets to calculate the evaluation metrics, which are not being used for updating the model, just for reporting metrics. When you have those, you should gather them if you want to have the same metrics on all processes. When it comes to the loss, it may differ on each process, since the training is distributed among the processes.

True loss will always be the same across processes after .backward() iirc. You can check this by printing the loss at each step. They should be identical because DDP will average all of them across processes when doing the backward pass

True loss will always be the same across processes after .backward() iirc. You can check this by printing the loss at each step. They should be identical because DDP will average all of them across processes when doing the backward pass

Hi muellerzr, I have done some experiments. and it turns out that each process does a batch of training separately, thus the loss is not identical across processes. Is it normal? Could you please take a look at  my code？
I use 2 GPU, bath_size=256, accelerate==0.25.0; the code is
`def train(net, accelerator, optimizer, scheduler, train_loader):
for i, (x, y) in enumerate(tqdm(train_loader, disable=not accelerator.is_local_main_process)):
output = net(x)
loss1 = torch.nn.CrossEntropyLoss(reduction='mean')(output,y)
loss2 = torch.nn.CrossEntropyLoss(reduction='sum')(output,y)
print('before bp '+str(loss1)+' '+str(loss2))
optimizer.zero_grad()
accelerator.backward(loss1)
print('after bw '+str(loss1)+' '+str(loss2))
optimizer.step()
scheduler.step()
print('after op '+str(loss1)+' '+str(loss2))
        print('gather: ' +str(accelerator.gather((loss1,loss2))))

        # do something else
        if i==0:
              break

    return net, accelerator

net, optimizer, train_loader, val_loader, scheduler = accelerator.prepare(
net, optimizer, train_loader, val_loader, scheduler)
net, accelerator, scheduler = train(net, accelerator, optimizer, scheduler, train_loader)
`
and the results are
before bp tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>) before bp tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>) after bw tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>) after bw tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>) after op tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>) after op tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>) gather:(tensor([0.8187, 0.8804], device='cuda:0'), tensor([209.5794, 225.3893], device='cuda:0')) gather:(tensor([0.8187, 0.8804], device='cuda:1'), tensor([209.5794, 225.3893], device='cuda:1')).

I am confused that if i want to track the true loss of each training step/epoch, should I use accelerator.gather_for_metric() like this script did for reducing among processes?

however, this example script just simply add loss at each step to get epoch loss, can someone help me figure out which method is correct or why different procedures are used to train loss and eval metric seperately? Will averaging loss across all processes make a difference to see whether the training of NN is going well?
Moreover, if i use multi gpus & gradient_accumulation, the the total_batch_size will be batch_size * num_processes * gradient_accumulation_steps, so the correct formula to get loss of each step should be loss = accelerator.gather_for_metric(loss).mean().item()?
I also see someone use loss = accelerator.gather(loss.repeat(batch_size)).mean().item() link , is it just an old verison&bug to fix?

I would be very appreciate if someone can answer me these questions!

Hello, I am also very confused about your questions after googling and viewing the limited documents😭. Have you figured out the problems?

I haven't figure out a thorough answer yet. if you want to get an average loss across different processes, then loss = accelerator.gather_for_metric(loss).mean().item() should be used. However, I think it's not need to do so if you just want to see if your training is going well correctly, you can just record loss in the main process after each train epoch, and here is a good example you can follow  ( hope this may help u, if there is any mistake, welcome to point out :)

True loss will always be the same across processes after .backward() iirc. You can check this by printing the loss at each step. They should be identical because DDP will average all of them across processes when doing the backward pass

Hi muellerzr, I have done some experiments. and it turns out that each process does a batch of training separately, thus the loss is not identical across processes. Is it normal? Could you please take a look at my code？
I use 2 GPU, bath_size=256, accelerate==0.25.0; the code is
def train(net, accelerator, optimizer, scheduler, train_loader):
        for i, (x, y) in enumerate(tqdm(train_loader, disable=not accelerator.is_local_main_process)):
            output = net(x)
            loss1 = torch.nn.CrossEntropyLoss(reduction='mean')(output,y)
            loss2 = torch.nn.CrossEntropyLoss(reduction='sum')(output,y)
            print('before bp '+str(loss1)+' '+str(loss2))
            optimizer.zero_grad()
            accelerator.backward(loss1)
            print('after bw '+str(loss1)+' '+str(loss2))
            optimizer.step()
            scheduler.step()
            print('after op '+str(loss1)+' '+str(loss2))

            print('gather: ' +str(accelerator.gather((loss1,loss2))))

            # do something else
            if i==0:
                  break

        return net, accelerator

net, optimizer, train_loader, val_loader, scheduler = accelerator.prepare(
                        net, optimizer, train_loader, val_loader, scheduler)

net, accelerator, scheduler = train(net, accelerator, optimizer, scheduler, train_loader)

and the results are
before bp tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>)
before bp tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>)
after bw tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>)
after bw tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>)
after op tensor(0.8187, device='cuda:0', grad_fn=<NllLossBackward0>) tensor(209.5794, device='cuda:0', grad_fn=<NllLossBackward0>)
after op tensor(0.8804, device='cuda:1', grad_fn=<NllLossBackward0>) tensor(225.3893, device='cuda:1', grad_fn=<NllLossBackward0>)
gather:(tensor([0.8187, 0.8804], device='cuda:0'), tensor([209.5794, 225.3893], device='cuda:0'))
gather:(tensor([0.8187, 0.8804], device='cuda:1'), tensor([209.5794, 225.3893], device='cuda:1'))

same outputs as yours, from the result, it seems that loss is still different across processes after .backward(), but gradient will be same across processes after .backward()

I haven't figure out a thorough answer yet. if you want to get an average loss across different processes, then loss = accelerator.gather_for_metric(loss).mean().item() should be used. However, I think it's not need to do so if you just want to see if your training is going well correctly, you can just record loss in the main process after each train epoch, and here is a good example you can follow link ( hope this may help u, if there is any mistake, welcome to point out :)

The loss that recorded by total_loss += loss.detach().float() is the loss of main process's training right? Is it alright to get only partial loss instead total loss?


Merge partners DB with diapers DB

Partially resolves #2787.
Description
This PR does the following:

Adds all partner table schemas into the diaper database.
Adds a feature flag to enable changing the source of the partner models to point to the diaper database.
Adds a script to run which will copy all data from the partner database to the diaper database.

The rollout plan should be as follows:

Deploy this commit to staging. Smoke test that everything still works as expected.
Take the app down and put it in maintenance mode.
Run the script to copy the data.
Bring the app back up and flipp the single_database feature flag on.
Validate that the app is still working as expected, particularly in terms of partner login and functionality.
If all is well, repeat the steps with production.

Once this is deployed and works, we can drop the partner database entirely and remove the feature flag.
Type of change

Infrastructure change

How Has This Been Tested?
Local and unit tests.

Merged in main for testing on Heroku.

@dorner nice work! What is left for this objective?

@dorner maybe we can pair through this and setup some dates to deploy this out  today.


Token Store Support

Token Types by string
Create/Revoke/Expiration
Token Schema:
Id
Value
Type
Identity
1 TOTP key per user
N recovery codes
Per App
N Single Auth code
N Access token
N Refresh token

@blowdart @divega
Feels like the proper key is ID + type, as opposed to requiring unique ID always, but I'm not entirely convinced...thoughts?
Which leads us to an API that looks something like:
   public interface ITokenStore {
        // Create or updates the token (would we need to split these apart into two apis?)
        Task StoreAsync(string id, string type, string value, CancellationToken cancellationToken);
        Task<bool> RemoveAsync(string id, string type, CancellationToken cancellationToken)
        Task<string> GetValueAsync(string id, string type, CancellationToken cancellationToken)
   }

   // Constraints would be non null id, type and values (but empty would be considered valid)
   // GetValue would return null for tokens that don't exist

It depends. What is this id? Is it a key borrowed from some other entity or is it just added to this entity to become the key? If the latter then it should be sufficient by definition.

If we go that route with the id being the key, maybe we should use a POCO
public class IdentityToken {
   public string Id { get; set; }
   public string Type { get; set; }
   public string Value { get; set; }
}

Task StoreAsync(IdentityToken token)
Task<bool> RemoveAsync(string id)
Task<IdentityToken> GetAsync(string id)

For usage, I'm imagining we don't need any extra complexity like a manager, we'd just implement the store in Identity.EF, and it would get registered as part of AddEntityFrameworkStores, and consuming it would be up to the developer for 1.1, since we don't have any features that currently use this store (until the templates are updated)

Could you still answer what the ID represents?

Could you still answer what the ID represents?

That's sort of the question I am asking. Do we want to make the id the unique key or part of the key. We are deciding what the contract is for consumers.
It basically boils down to whether we want to make it easy to partition based on token type or not. If we say is must be unique across all token types. We don't even need token type anymore. We can just say that's something that should be in the payload as well since we won't have any APIs that use the type.

ID would be a unique key probably. Nothing you query on. Then there's the property bag too.
We would need token type though, unless you make the token type part of the property bag, because, for example, we'd want to say "Give me all the TOTP reset tokens for user X"

Having the type alone won't help with that since we won't be able to query using user id since that's part of the bag.

For example, we'd want to say "Give me all the TOTP reset tokens for user X"

Well true, but you'd get the unique token IDs from the property bag by looking up the user property, then backtrack up.

The main difference between the token server and user ids is here, where the token id is enough for them, while user tokens have the additional filters of type + owner that we will need to query by.

For identity could use these the token store just as a bag with another table in charge of managing token types/payloads.
So maybe: ITokenStore for the generic get/set/delete of ID/value token pairs.
IUserTokenStore which layers on UserId, TokenType, TokenId which we can use to build our recovery code/whatever we want

We've decided to roll the authenticator info into the existing UserAuthenticationTokenTable


Why are AuthRole and UnauthRole required for hosting?

** Which Category is your question related to? **
init and hosting
** What AWS Services are you utilizing? **
s3 and cloudfront
** Provide additional details e.g. code snippets **
I'd like to use AWS Amplify only for hosting (at least for now) and would like to avoid unnecessary extra resources getting created. I noticed that the generated cloudformation templates and the config files generated by init refer to AuthRole and UnauthRole, which refer to Cognito, which I'm not using. I attempted to remove them manually from the cloudformation file, but when running amplify publish, they get re-created. Is there a good way to remove them entirely, or a particular reason why they're needed? Thanks!

Hi @ncknuna ,
You are right, the hosting category does not using the two roles.
The CLI adds those two roles during init, and keep them there, because they are used by other categories that manage aws resources to be consumed by the Amplify library, such as storage, api and analytics. Currently, the CLI does not provide tools to remove them.

@UnleashedMind Thanks for the quick reply! In that case, can this ticket turn into a feature request to only create those roles when adding a category that requires them?

@ncknuna Almost all categories need this auth and unauth role for managin permissions except "Hosting" I beleive, so as an opinionated approach we decided to provision the auth and unauth role (which users are not charged for), during initialization so that all the common categories could share it whenever required or whenever added. We took this decision based on the 80/20 rule, considering the fact that majority of our users would be using at least one of the categories other than hosting that would require auth/unauth roles for authentication.
If you have an app which just requires hosting, then I beleive these roles are redundant but if you add any other category in the future, these roles would be utlized to provide auth/unauth access to your app. That being said, we're looking it ways as to how we can optimize for use-cases like yours and have a plan for it in our roadmap.

@kaustavghosh06 I understand, and I very well may have done the same thing if I was in your shoes :)
In my case, I'm using Auth0 as a third party auth provider (largely because of a lot of the limitations specified here). Until generic OIDC providers are supported, I likely won't be adding any other categories via Amplify, and instead following tutorials like this one to integrate with appsync or other services.
Glad to hear that there is a plan in the roadmap somewhere (even if it's not soon), and thanks for being so responsive!

Hi guys im using only API module of Amplify, but the process create this roles, UnauthRole and authRole, i explored this roles and they dont have any policy attached... this is normal?
If i add other service as Auth with cognito this files change?
Thanks!!

Hey @ncknuna :wave: is this feature request something you're still looking for?

We moved off of Amplify a while back, so no, we don't need this anymore. I still think it's a useful feature, but not one that I need personally


Feature/include merged refs

Contains added features:

Hardcoded x:Type/x:Static checks replaced with actual namespace (Now looks for http://schemas.microsoft.com/winfx/2006/xaml namespace in root's attributes)
Adds an optional parameter to include merged ResourceDictionary links from resources (otherwise we can't preserve loading order without doing additional changes in some cases)
Seeks for Markdown syntax in attribute values when namespace should be renamed to fix them.

Hello, thanks for having a time for looking into it.
About 2 (including merged references) feature:
I am tried to solve an issue with missing resources in case like this:

After merging xamls in both Assembly1/Assembly2 the resources from Assembly1 are missing from a scope of Assembly2 generated file if they are not included as merged dictionary on current level (inside generated file) or place resource with this key directly in App.xaml. Including generic.xaml reference from Assembly1 in App.xaml in merged dictionaries doesn't help 😔
I suspect it creates a duplicate copy of dictionary in this way, so i've added this option as optional parameter (false by default) as it's specific case. Without it we'll forced to manual edit generated file.
Also, to be clear: this option only keeps references on ResourceDictionaries which aren't processed by XamlCombine
About 3
Personally, i like your idea of renaming namespaces in automatic way (works smooth for our projects). Would be nice to keep it as optional flag if you decide to change logic in the future 😀

I will merge your changes, but refactor it slightly afterwards.
Could you provide a sample project that showcases why we need the merged dictionary option?
I still don't like that being added as it's quite dangerous and should be solved the way Fluent.Ribbon handles it.
Maybe i can show that on your provided sample. ;-)

Thanks. I'll try to provide sample solution on this week.

Thank you for merging. Will this version be available as prerelease on NuGet?

Will put it on nuget tomorrow.

@GeertvanHorrik just pushed to NuGet. Should be available soon.


Add Loose option to ignore non-fatal formatting errors

ref: https://github.com/Songmu/axslogparser/issues/17

Coverage increased (+0.04%) to 95.041% when pulling f37d952bbbabac3847f8bc28f5a2c82ab0e0eb0d on motemen:loose into 31874ad7dc8d453b39e3494f8979cedfaa386199 on Songmu:master.

Thank!

Thanks 😄


import/no-unresolved error with 'node:' URLs (Node.js ESM)

ECMAScript modules in Node.js support importing Node.js builtin modules using node: URLs (see the documentation), for example:
import fs from 'node:fs/promises';

Using such URLs causes eslint-plugin-import to raise an import/no-unresolved error.

What version of node are you using?
If you're seeing this error in your editor, then what version of node is your editor using?
a) this plugin does not explicitly support native ESM, it expects transpiled ESM, which means that import is really require
b) node:fs/promise is only a core module for require in node ^14.18 || >= 16.

I'm using Node.js v17.3.0. I'm assuming my editor (Code - OSS) is using the same.
It tested it and it actually seems that the error only shows up in the editor, but not when I run eslint from the command-line.
I didn't know that this plugin does not support native ESM. I'm using @typescript-eslint/parser, although I see the error both with TypeScript and with JavaScript files. In my tsconfig.json I have "module": "esnext" set, but maybe @typescript-eslint/parser ignores that and transpiles it to require anyways.
Maybe my eslint editor plugin is somehow misconfigured, although I don't know what setting it could be.

vscode has an embedded node, which i believe is v12.
The command line is the source of truth, so it's a problem you'll have to fix in your editor.
TS also isn't native ESM, it's a transpiler (which is better than native anyways).

Just found out, in vscode one can set "eslint.runtime" setting to "node" to force eslint execution in locally installed version of node (see explanation for this option here). Requires full restart of vscode.

Duplicate of #2035. Duplicate of #2248.


Question about validation

Hello, first thanks for your pytorch version of deeplab v3+, I try it at last days of 2018, and I got a perfect model whose performance is great both on train and validation datasets, which means their loss are steadily decreasing at my first try. However, when I give it a second try, train loss of the new model still decreased but the validation loss went up and down sometimes.
Below, is the first try performance.
train

validation

Below, is the second try performance.
train

validation

ps, the dataset of two try is definitely the same!! So, do you have any idea of this?

did you used the syncd bn option？

did you used the syncd bn option？

I don't think I used it cause I just have one gpu to train it.

@CeciliaPYY  How to get kinds of  figures ,can you give me a copy? thanks


Release 0.1.1

@JuliaRegistrator register(branch=release)

Registration pull request created: JuliaRegistries/General/18420
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.1 -m "<description of version>" 1392b73e2c5e3dc3d2d5d8dbde8ec4cd7c10742d
git push origin v0.1.1