Move /whisk.system/samples from openwhisk to openwhisk-catalog

Issue #4 is a prereq.
In general, we are are moving actions that appear in the "/whisk.system/samples" namespace form the current repo. "openwhisk-openwhisk" to this repo. "openwhisk-catalog".
By convention, we will:
Create top-level “package” named directories (1-per logical sample, same functional sample)
For each unique language implementation for sample package, create a subdirectory named for the language in which that language impl. will reside (e.g., “javascript”, "swift", “python”, java”, etc.)
    so it would look like: openwhisk/catalog/packages/samples/ <action name> / <language name> / <xxx>.<language extension>
    for example: "openwhisk/catalog/packages/samples/hello/swift/hello.swift"

Specifically, create known sample directories (and language specific subdirs.) for the following packages:
“countdown”
    create “javascript” -> “countdown.js”
    Rodric indicates code is not working properly but could and needs to be fixed.
"curl"
    create “javascript” -> “curl.js”
    May need testing/fixing?
"echo"
    move echo.js to /whisk.system/utils package, remove from “samples” directory, move to “utils” directory.
    move catalog install logic to change package namespace and directory locations.
"greeting"
    create “javascript” - > “echo.js”
“hello”
    Needs subdirs, for javascript, python, java, swift (others?)
    rename impl. files for actions/feeds to remove any language references (e.g., “helloJava.java” should ne just “hello.java”) as a new convention
    Python “hello.py” would come from the "blue" sample named “helloPython.py”…
    we will do this on a different issue # after this initial move.
“helloAsync”
    create “javascript” -> “helloAsync.js”
“httpGet”
    create subdir. “swift” to host “httpGet.js”
“invoke”
    create subdir “swift” to host invoke.swift
"printParams"
    create “javascript” - > “printParams.js”
"trigger"
    create ”swift” -> “trigger.swift”
    Note: may need testing/fixing
"wordcount"
    create “javascript” - > “wc.js”
    rename wc.js to wordcount.js
    move wcbin.js to wordcount javascript directory
    merge wcbin.js (i.e., wordcount binary) into wordcount.js
    provide new optional parameter to return binary result

Please also move the following actions currently in "openwhisk-openwhisk/catalog/utils" to their own directories under "openwhisk-catalog/packages/samples/".
cat.swift
head.swift
split.swift

Be sure to coordinate "delete" from the current repo. once these are moved (see Issue #2) as this move is noted there is well.

Refer to openwhisk-catalog: https://github.com/openwhisk/openwhisk-catalog/issues/3
Refer to blue: https://github.ibm.com/BlueMix-Fabric/bluewhisk/issues/2037

Move implies removing the corresponding artifacts in this repo. Are you covering the cleanup in some other issue?


Add Supported Regions

Hi, seems that for now only some regions are supported for Defender EASM. Can you please add warning to guide how to start that this temporary limitation is in place? At least I hope that this is temporary. Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ad120c2f-c1d6-dfcd-e393-74214c1473bf
Version Independent ID: 3808fad7-df74-2005-a309-83b726eae4ef
Content: Deploying the Defender EASM Azure resource
Content Source: articles/external-attack-surface-management/deploying-the-defender-easm-azure-resource.md
Service: security
GitHub Login: @danielledennis
Microsoft Alias: dandennis

@Kazzan
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.


[doc] Remove duplicate paragraph in storage_engine.adoc

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:
<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

This PR applies to 4.1, 4.0, and 3.11. Does not apply to 5.0.


Add Model::find method

Resolves #25

Coverage remained the same at 100.0% when pulling e22e7d2956d13cad479e0e5ddb5415922e537951 on soft-retrieval-methods into e52fc0fac0c49df3e085cdcae0dffc842d7f090c on master.

Coverage remained the same at 100.0% when pulling e22e7d2956d13cad479e0e5ddb5415922e537951 on soft-retrieval-methods into e52fc0fac0c49df3e085cdcae0dffc842d7f090c on master.

Coverage remained the same at 100.0% when pulling e22e7d2956d13cad479e0e5ddb5415922e537951 on soft-retrieval-methods into e52fc0fac0c49df3e085cdcae0dffc842d7f090c on master.

Coverage remained the same at 100.0% when pulling e22e7d2956d13cad479e0e5ddb5415922e537951 on soft-retrieval-methods into e52fc0fac0c49df3e085cdcae0dffc842d7f090c on master.

Coverage remained the same at 100.0% when pulling e22e7d2956d13cad479e0e5ddb5415922e537951 on soft-retrieval-methods into e52fc0fac0c49df3e085cdcae0dffc842d7f090c on master.


upgrade from non-gpg rvm to signed version

Given a rvm version was installed before gpg signing was introduced
When we request a new version through puppet
Then the gpg key should be imported before fetching the new version.

This seemed to have worked in my setup through luck or so…
Thanks!


Add PSF CoC

I've added the PSF CoC in absence of having a favorite CoC of my own. If anyone has a preference, I'm more than happy to use that instead. I know the Contributor Covenant and Citizen Code of Conduct are somewhat popular options.
Alternately, we can modify the PSF one to suit our needs. One thing it lacks by default is information about how to report potential violations, for example.
We should also likely include that information on the site itself.
Consider this PR the start of that conversation.

Agreed. I'm good with using the PSF CoC. One request:

@nkantar because the PSF CoC lacks information on how to report CoC violations, can you add a note to raise this question/topic the next time we talk to Phyllis?


Infinite loop occurs

While I conduct multiple simulation with Pybamm, I find that the simulation generates an infinite loop and causes the python program to be halted.

The script stops here (no response)

Can you try using CasadiSolver("fast with events") instead?

for me, ("fast with events") doesn't work. I tried three different solver options and "old safe" is working with an warning.
"safe": SolverError: Maximum number of decreased steps occurred at t=0.0. Try solving the model up to this time only or reducing dt_max (currently, dt_max=81.88051713742692).
"fast": SolverError: .../casadi/interfaces/sundials/idas_interface.cpp:569: IDASolve returned "IDA_CONV_FAIL". Consult IDAS documentation.
"old safe": working with an warning
psetup failed: .../casadi/interfaces/sundials/idas_interface.cpp:827: Calculating Jacobian failed

Please provide an entire script that can be run to test this

Here is the script and required data.
On Wed, May 26, 2021 at 11:15 AM Valentin Sulzer @.***>
wrote:

Please provide an entire script that can be run to test this
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pybamm-team/PyBaMM/issues/1493#issuecomment-848856146,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOPG4HUKVTOZEZ2RCUTO2U3TPUF7XANCNFSM45PQUH5Q
.

I don't see it

Can you please say the solution to this in case you discussed it

Hey, I have the same issue. Is there a solution to this problem?

Please provide a minimum working example and the full error log, so we can look into it.

Thank you for taking the time. I have attached a minimal working example and full error log.
import pybamm
import matplotlib.pyplot as plt
import numpy as np

model = pybamm.lithium_ion.DFN(
    {
        "lithium plating": "partially reversible",
        "lithium plating porosity change": "true",  # alias for "SEI porosity change"
    }
)

param = pybamm.ParameterValues("OKane2022")

cycle_number = 1000
exp = pybamm.Experiment(
    ["Hold at 4.2 V until C/100",
    "Rest for 4 hours",
    "Discharge at 0.1C until 2.5 V",  # initial capacity check
    "Charge at 0.3C until 4.2 V",
    "Hold at 4.2 V until C/100",]
    + [("Discharge at 1C until 2.5 V",  # ageing cycles
    "Charge at 0.3C until 4.2 V",
    "Hold at 4.2 V until C/100",)] * cycle_number
    + ["Discharge at 0.1C until 2.5 V"],  # final capacity check
    period="5 minutes"
)

sim = pybamm.Simulation(model, parameter_values=param, experiment=exp)

# safe solver
safe_solver = pybamm.CasadiSolver(atol=1e-3, rtol=1e-3, mode="safe", dt_max=100)

sol = sim.solve(solver=safe_solver)

The error message is:
At t = 299.979 and h = 4.56806e-09, the corrector convergence failed repeatedly or with |h| = hmin.
2023-08-07 08:57:26.287 - [ERROR] callbacks.on_experiment_error(224): Simulation error: Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDASolve returned "IDA_CONV_FAIL". Consult IDAS documentation.
ERROR:pybamm.logger:Simulation error: Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDASolve returned "IDA_CONV_FAIL". Consult IDAS documentation.

Using the following solver works for me:
safe_solver = pybamm.CasadiSolver(mode="safe")

Thank you for your answer. I ran the model with your solver. Unfortunately, I still get the following error message after 319/1000 cycles.
"""
At t = 299.973 and h = 1.45273e-09, the corrector convergence failed repeatedly or with |h| = hmin.
2023-08-07 19:28:09.253 - [ERROR] callbacks.on_experiment_error(224): Simulation error: Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDASolve returned "IDA_CONV_FAIL". Consult IDAS documentation.
ERROR:pybamm.logger:Simulation error: Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDASolve returned "IDA_CONV_FAIL". Consult IDAS documentation.
"""
Do you have an idea why this happens? From the IDAS documentation, it is unclear to me what causes the error IDA_CONV_FAIL.

That's strange, mine ran all 1000 cycles. Check that you are using the most recent version of PyBaMM and Casadi.

By separately installing the Casadi Solver, the new version got installed, and it works now - thank you.
%pip install casadi
%pip install pybamm

same problem, and try all above methods, and then still failed.
code like this. please help me. thank you very much
import pybamm
import matplotlib.pyplot as plt
model = pybamm.lithium_ion.DFN(
{
"SEI": "solvent-diffusion limited",
"SEI porosity change": "true",
"lithium plating": "partially reversible",
"lithium plating porosity change": "true",
"calculate discharge energy": "true",
}
)
parameter_values = pybamm.ParameterValues("OKane2022")
def graphite_ocp(sto):
u_eq = (
0.6379
+ 0.5416 * pybamm.exp(-305.5309 * sto)
+ 0.044 * pybamm.tanh(-(sto - 0.1958) / 0.1088)
- 0.1978 * pybamm.tanh((sto - 1.0571) / 0.0854)
- 0.6875 * pybamm.tanh((sto + 0.0117) / 0.0529)
- 0.0175 * pybamm.tanh((sto-0.5692) / 0.0875)
)
return u_eq
def LFP_ocp(sto):
u_eq = (
3.4323
- 0.8428 * pybamm.exp(-80.2493 * (1 - sto) ** 1.3198)
- 3.2474e-6 * pybamm.exp(20.2645 * (1 - sto) ** 3.8003)
+ 3.2482e-6 * pybamm.exp(20.2646 * (1 - sto) ** 3.7995)
)
return u_eq
def negative_electrode_OCP_entropic_change(sto, c_s_max):
dEdT = (
2.8957
- 5.2972 * pybamm.cos(pybamm.arctan(1/(sto2) - 1))
+ 4.0370 * pybamm.cos(2 * pybamm.arctan(1/(sto2) - 1))
- 2.5304 * pybamm.cos(3 * pybamm.arctan(1/(sto2) - 1))
+ 1.2717 * pybamm.cos(4 * pybamm.arctan(1/(sto2) - 1))
- 0.4884 * pybamm.cos(5 * pybamm.arctan(1/(sto2) - 1))
+ 0.1301 * pybamm.cos(6 * pybamm.arctan(1/(sto2) - 1))
- 0.1086 * pybamm.cos(7 * pybamm.arctan(1/(sto**2) - 1))
)
return dEdT
def positive_electrode_OCP_entropic_change(sto, c_s_max):
dEdT = (
-0.3538 * sto ** 8
+ 1.3902 * sto ** 7
- 2.2585 * sto ** 6
+ 1.9635 * sto ** 5
- 0.9872 * sto ** 4
+ 0.2886 * sto ** 3
- 0.0463 * sto ** 2
+ 0.0032 * sto
- 1.9186e-5
)
return dEdT
def electrolyte_diffusivity_arrhenius(c_e, T):
D_c_e = (
5.2969e-20 * c_e ** 3
- 2.7984e-16 * c_e ** 2
+ 2.8086e-13 * c_e
+ 3.4857e-10
)
E_D_c_e = 16500
arrhenius = pybamm.exp(E_D_c_e / pybamm.constants.R * (1 / 298.15 - 1 / T))
return D_c_e * arrhenius
def electrolyte_conductivity_arrhenius(c_e, T):
sigma_e = (
3.2948e-10 * c_e ** 3
- 1.6463e-6 * c_e ** 2
+ 2.2550e-3 * c_e
+ 2.5977e-2
)
E_sigma_e = 4000
arrhenius = pybamm.exp(E_sigma_e / pybamm.constants.R * (1 / 298.15 - 1 / T))
return sigma_e * arrhenius
def cation_transference_number_arrhenius(c_e, T):
t_plus = (
5.7052e-17 * c_e ** 5
- 4.7979e-13 * c_e ** 4
+ 1.2203e-9 * c_e ** 3
- 1.1791e-6 * c_e ** 2
+ 2.8155e-4 * c_e
+ 3.5182e-1
)
E_t_plus = 16500
arrhenius = pybamm.exp(E_t_plus / pybamm.constants.R * (1 / 298.15 - 1 / T))
return t_plus * arrhenius
def thermodynamic_factor_arrhenius(c_e, T):
dlnfdlnc = (
-4.4566e-10 * c_e ** 3
+ 1.5644e-6 * c_e ** 2
- 7.4218e-5 * c_e
- 3.1908e-2
)
E_dlnfdlnc = 16500
arrhenius = pybamm.Exp(E_dlnfdlnc / pybamm.constants.R * (1 / 298.15 - 1 / T))
return dlnfdlnc * arrhenius
def graphite_diffusivity(sto, T):
D_ref = 6e-14
E_D_s = 3.5e4
arrhenius = pybamm.Exp(E_D_s / pybamm.constants.R * (1 / 298.15 - 1 / T))
return D_ref * arrhenius
def LFP_diffusivity(sto, T):
D_ref = 4e-13
E_D_s = 20000
arrhenius = pybamm.Exp(E_D_s / pybamm.constants.R * (1 / 298.15 - 1 / T))
return D_ref * arrhenius
def graphite_electrolyte_exchange_current_density(c_e, c_s_surf, c_s_max, T):
c_e_0 = pybamm.Parameter("Initial concentration in electrolyte [mol.m-3]")
k = 6
m_ref = k * 2 / (c_s_max * c_e_0 ** 0.5)
E_r = 20e3
arrhenius = pybamm.Exp(E_r / pybamm.constants.R * (1 / 298.15 - 1 / T))
return m_ref * arrhenius * c_e ** 0.5 * c_s_surf ** 0.5 * (c_s_max - c_s_surf) ** 0.5
def LFP_electrolyte_exchange_current_density(c_e, c_s_surf, c_s_max, T):
c_e_0 = pybamm.Parameter("Initial concentration in electrolyte [mol.m-3]")
k = 2
m_ref = k * 2 / (c_s_max * c_e_0 ** 0.5)
E_r = 53e3
arrhenius = pybamm.Exp(E_r / pybamm.constants.R * (1 / 298.15 - 1 / T))
return m_ref * arrhenius * c_e ** 0.5 * c_s_surf ** 0.5 * (c_s_max - c_s_surf) ** 0.5
parameter_values["Thermodynamic factor"] = thermodynamic_factor_arrhenius
parameter_values["Cation transference number"] = cation_transference_number_arrhenius
parameter_values["Electrolyte conductivity [S.m-1]"] = electrolyte_conductivity_arrhenius
parameter_values["Electrolyte diffusivity [m2.s-1]"] = electrolyte_diffusivity_arrhenius
parameter_values["Negative electrode OCP [V]"] = graphite_ocp
parameter_values["Positive electrode OCP [V]"] = LFP_ocp
parameter_values["Negative electrode OCP entropic change [V.K-1]"] = negative_electrode_OCP_entropic_change
parameter_values["Positive electrode OCP entropic change [V.K-1]"] = positive_electrode_OCP_entropic_change
parameter_values["Electrode height [m]"] = 1
parameter_values["Negative electrode Bruggeman coefficient (electrode)"] = 1
parameter_values["Positive electrode Bruggeman coefficient (electrode)"] = 1
parameter_values["Lower voltage cut-off [V]"] = 2.5
parameter_values["Upper voltage cut-off [V]"] = 3.65
parameter_values["Open-circuit voltage at 0% SOC [V]"] = 2.5
parameter_values["Open-circuit voltage at 100% SOC [V]"] = 3.65
parameter_values["Initial concentration in negative electrode [mol.m-3]"] = 766.88
parameter_values["Initial concentration in positive electrode [mol.m-3]"] = 19058
parameter_values["Maximum concentration in negative electrode [mol.m-3]"] = 31507.0
parameter_values["Maximum concentration in positive electrode [mol.m-3]"] = 21190
parameter_values["Negative electrode active material volume fraction"] = 0.67038
parameter_values["Positive electrode active material volume fraction"] = 0.58407
parameter_values["Negative electrode porosity"] = 0.269
parameter_values["Positive electrode porosity"] = 0.336
parameter_values["Separator porosity"] = 0.42
parameter_values["Negative electrode thickness [m]"] = 5.3e-5
parameter_values["Positive electrode thickness [m]"] = 8.5e-5
parameter_values["Separator thickness [m]"] = 1.6e-5
parameter_values["Negative particle radius [m]"] = 8.1e-6
parameter_values["Positive particle radius [m]"] = 1.25e-6
parameter_values["Negative electrode diffusivity [m2.s-1]"] = graphite_diffusivity
parameter_values["Positive electrode diffusivity [m2.s-1]"] = LFP_diffusivity
parameter_values["Negative electrode exchange-current density [A.m-2]"] = graphite_electrolyte_exchange_current_density
parameter_values["Positive electrode exchange-current density [A.m-2]"] = LFP_electrolyte_exchange_current_density
parameter_values["Electrode width [m]"] = 0.47
var_pts = {"x_n": 5, "x_s": 5, "x_p": 5, "r_n": 30, "r_p": 30,}
pybamm.set_logging_level("NOTICE")
cycle_number = 1000
experiment = pybamm.Experiment(
["Rest for 1 hours", "Discharge at 0.1C until 2.5 V", "Charge at 1C until 3.65 V", "Hold at 3.65 V until C/100",]
+ [("Discharge at 1C until 2.5 V", "Charge at 1C until 3.65 V", "Hold at 3.65 V until C/25",)] * cycle_number
+ ["Discharge at 0.1C until 2.5 V", "Charge at 1C until 3.65 V", "Hold at 3.65 V until C/100",],
termination="70% capacity"
)
simulation = pybamm.Simulation(model, parameter_values = parameter_values, experiment = experiment, var_pts = var_pts)
solution = simulation.solve(pybamm.CasadiSolver(mode="safe without grid", return_solution_if_failed_early=True))

using pybamm>23.5 will output log like they said. it might be a bug.

By separately installing the Casadi Solver, the new version got installed, and it works now - thank you.
%pip install casadi
%pip install pybamm

hello! I met the same problem, could you tell me your casadi and pybamm version? Thank you!

By separately installing the Casadi Solver, the new version got installed, and it works now - thank you.
%pip install casadi
%pip install pybamm

hello! I met the same problem, could you tell me your casadi and pybamm version? Thank you!

Look at this: https://github.com/ArnaudOlt/dissertation_pybamm


[feature] Centralize django-channels dependency and upgrade to latest version

Let's centralize the django-channels dependency in this package, we'll need to upgrade the code of the following modules to support channels 4.1:

django-loci
controller
network-topology
notifications

@nemesifier  I want to take this issue.

Sounds good @himanshu-sharmav, start by upgrading this module and then at least another module.

great will do that.


KAFKA-4752: Fixed bandwidth calculation

@dguy could you have a look? Thanks.

@dguy ok

@guozhangwang this is ready to go. Thanks.

One minor comment, otherwise LGTM.

Thanks @guozhangwang


Make notes-after-punctuation also work on non-note but superscript styles

Hi,
I'd like to have notes-after-punctuation also move non-note styles, if they are using superscript.
For example, AMA style has superscript references but it is in-text (this is a slightly modified version but it gets the idea across):

If I switch it to note I get undesired behaviour where the in-text citation is a footnote that then contains a superscript reference to the bibliography(!)

From:

To:

Then:

... hence my suggestion. I think this would be a matter of adding a case to isNote along the lines of:
isNote (Cite _ [Superscript _]) = True

FWIW it took me a long time to figure out why my notes weren't being moved after the puncatuation...

I tried out the change on a local version of the package and it works:

So... would you accept a PR that does this? 🙂


Make the text in the interval text field (StringVar) a Label instead

Move Label above the Text field.

Fixed by PR.


Create Screen Cast of V3 for Users

Please create 2 videos for V3:

How to Use V3
How to Convert Presets to YAML

@bdon the second video would have to be created before the launch so we can email existing V2 users to give them time to convert their presets


Feature/basic ve ocean

Add basic veOcean support:

lock, deposit_for, increase_lock, increase_amount, withdraw (veOcean)
claim (veFees)
setAllocate & setBatchAllocate (veAllocation)

Integration test are failing, but they are not touched in this PR
2022-09-05 19:03:01 ad081f7e6841 ocean_provider.routes.compute[68] ERROR Invalid container values provided on algorithm. Check image, tag and checksum.

Integration test are failing, but they are not touched in this PR
2022-09-05 19:03:01 ad081f7e6841 ocean_provider.routes.compute[68] ERROR Invalid container values provided on algorithm. Check image, tag and checksum.

Yep, most likely the checksum we set for the ubuntu container is no longer available on docker hub. Will quickly check


Problems with a lot of data via SSE on many clients

Description
We are working on an application that utilizes cowboy to send CloudEvents via SSE. I am currently working on performance tests for said application. When trying to run these performance tests I noticed some peculiar behaviour (100k CloudEvents (each 1kb) sent to n clients, connected via HTTP 1).
In the test, I the client just waits to receive 100k messages and checks how long that takes.
When connecting with about 10 clients, everything works as expected and all is good. But when I connect with more than 15 clients, things start to go sideways very quickly.
Problems
We get these events via a Kafka consumer. Sometimes, clients would receive all 100k messages as soon as they have been produced. As soon as the client received 100k messages, it would then disconnect. One would assume that when one client disconnects, the other clients receive the messages quicker than they did before, but one would assume wrong. The contrary is the case and the clients actually take longer to receive the full 100k messages the more clients are done receiving their messages.
When connecting with 100 clients, the slowest client takes 3051s to receive the full 100k messages.

When inspecting the application state with observer, I saw that the message queues of cowboy_clear:connection_process/4 were ginormous.

Related
I saw both #1376 and #1402. I checked out from master to check if the fix actually worked for our system. Unfortunately, it didn't. The only change I could observe were a, sort-of, QoS like behaviour when sending.
Of course, it might just be that our SSE implementation is flawed. https://github.com/Accenture/reactive-interaction-gateway/blob/master/lib/rig_inbound_gateway_web/v1/sse.ex
Fix
I think a good solution would be implementing the same backpressure mechanism for HTTP 1 as was done for HTTP 2 in #1402.
Does that sound like a plausible solution to you?
Thanks!

There's no flow control in HTTP/1.1 so I'm not sure what you want me to implement.

@essen I know. But the main problem here is that our MsgQ's are overflowing (see 2nd image) and we sorta need a fix for that. Any ideas on that matter?

I'm not sure why you end up with so many messages considering you should get stuck waiting for Cowboy's ack when streaming the response body, at least on master. See https://github.com/ninenines/cowboy/blob/master/src/cowboy_req.erl#L871

So if stream_events blocks, the high number of queued-up messages could mean there's a problem within our application. Still, the performance of the SSE loop handler seems really bad: see below for an example that sends a MB of random events. With a client running on localhost I get only ~120 kB/s on my machine. Any thoughts on this?
defmodule MySSE do
  @behaviour :cowboy_loop

  require Logger

  @alphabet Enum.to_list(?a..?z) ++ Enum.to_list(?0..?9)
  @chunk_size_b 1_000
  @total_size_b 1_000_000

  @impl :cowboy_loop
  def init(req, state) do
    req =
      :cowboy_req.stream_reply(
        200,
        %{
          "content-type" => "text/event-stream; charset=utf-8",
          "cache-control" => "no-cache",
          "access-control-allow-origin" => "*"
        },
        req
      )

    send(self(), :run)
    {:cowboy_loop, req, state}
  end

  @impl :cowboy_loop
  def info(:run, req, state) do
    {time, n_streamed_bytes} = :timer.tc(&stream_data/1, [req])
    IO.puts("Streamed #{n_streamed_bytes / 1_000} kB in #{time / 1_000_000} seconds.")

    finalize(req)

    {:stop, req, state}
  end

  # We generate @total_size_b of data by streaming random @chunk_size_b strings.
  defp stream_data(req, n_streamed_bytes \\ 0)

  defp stream_data(req, n_streamed_bytes) when n_streamed_bytes < @total_size_b do
    event = random_string() |> to_server_sent_event()
    send_via(event, req)

    event_as_string = Jason.encode!(event)
    stream_data(req, n_streamed_bytes + byte_size(event_as_string))
  end

  defp stream_data(_, n_streamed_bytes), do: n_streamed_bytes

  defp random_string, do: for(_ <- 1..@chunk_size_b, into: "", do: <<Enum.random(@alphabet)>>)

  defp to_server_sent_event(data) when byte_size(data) > 0 do
    %{
      data: data,
      event: "data_generated"
    }
  end

  defp send_via(event, cowboy_req) do
    :cowboy_req.stream_events(event, :nofin, cowboy_req)
  end

  defp finalize(cowboy_req) do
    :cowboy_req.stream_events(%{}, :fin, cowboy_req)
    Logger.debug(fn -> "Finalized SSE stream" end)
  end
end

I don't really know what your code is doing. It doesn't look like you need a loop handler if you're going to just loop yourself. Other than that, I don't know what the code is doing exactly, but if you're creating random strings by choosing characters randomly one by one at runtime, that's probably your issue.
Cowboy has a test that sends 32MB of data streaming (not events, but it's not formatting the event that's gonna slow things down this much anyway). It takes between 0.9s and 3.5s depending on configuration, while other tests are running in parallel.
So I would say either there's something wrong with the test, or there's something wrong with the environment. If it's the latter then perhaps there's something I can do about it, but at this point I can't       say.

@essen you're right, the example wasn't really minimal, sorry about that. I've updated the example, removing the parts that aren't needed (including the randomizing part) and added some comments. These are the numbers I get:

n_total_bytes
n_data_bytes
kB/s
server time [s]
client time [s]

100000
10
0.179
10.84
>5min

100000
100
5.89
4.1
>5min

100000
1000
5500
2.45
17.3

1000000
10
-
out of memory (total-vm:21145076kB)
-

1000000
100
0.646
44.45
>5min

1000000
1000
60
23.27
>5min

Executed on: Intel i5-4590S @ 3.00GHz / 4 cores, 16G total memory
Some observations:

There is only one combination that lets the client finish in a reasonable amount of time.
The real-world-ish use case of many small messages does not work (but maybe the machine is just too small).
The server always finishes way before the client does, which is why we thought it'd be about message queues and missing backpressure of some sort. If the messages aren't on Cowboy's queues at that point, where could they be?

Again there's not much I can do or look at here, please provide an Erlang example I can easily run if you want me to look at it.
Note that on localhost depending on the environment the socket buffers can be really big and if the client is too slow to consume them then the data may just accumulate there.

I rewrote the test in Erlang. I am using Cowboy 2.6.3 and jsone 1.5.1
sse_handler.erl
-module(sse_handler).
-behaviour(cowboy_loop).

-export([init/2, info/3]).

%% The size of a message's data value in bytes:
-define(n_data_bytes, 1000).
%% The total transmission size in bytes:
-define(n_total_bytes, 100000000).

init(Req0, State) ->
    Headers = #{
        "content-type" => <<"text/event-stream; charset=utf-8">>,
        "cache-control" => <<"no-cache">>,
        "access-control-allow-origin" => <<"*">>
    },

    %% Set up the connection:
    Req = cowboy_req:stream_reply(200, Headers, Req0),

    self() ! run,

    {cowboy_loop, Req, State}.

info(run, Req, State) ->
    %% Run stream_data and measure execution time:
    {Time, N_streamed_bytes} = timer:tc(fun() -> stream_data(Req) end),

    io:format("Streamed ~w kB in ~w seconds.", [N_streamed_bytes, Time / 1000000]),

    %% Close the SSE connection:
    cowboy_req:stream_events(#{}, fin, Req),

    %% ..and stop the loop handler:
    {stop, Req, State}.

duplicate(Str, Times) ->
    string:join(lists:duplicate(Times, Str),"").

stream_data(Req) ->
    stream_data(Req, 0).

%% Sends at least @n_total_bytes of data through the SSE stream.
stream_data(Req, N_streamed_bytes) when N_streamed_bytes < ?n_total_bytes ->
    Event = #{data => duplicate("X", ?n_data_bytes)},
    cowboy_req:stream_events(Event, nofin, Req),

    %% Estimate the message size (lower bound) and count it to the number of sent bytes:
    Event_as_string = jsone:encode(Event),
    io:write(Event_as_string),
    stream_data(Req, N_streamed_bytes + byte_size(Event_as_string));
stream_data(_, N_streamed_bytes) ->
    N_streamed_bytes.

mysse_app.erl
-module(mysse_app).
-behaviour(application).

-export([start/2]).
-export([stop/1]).

start(_Type, _Args) ->
    Dispatch = cowboy_router:compile([
        {'_', [{"/", sse_handler, []}]}
    ]),
    {ok, _} = cowboy:start_clear(my_http_listener,
        [{port, 8080}],
        #{env => #{dispatch => Dispatch}}
    ),
    mysse_sup:start_link().

stop(_State) ->
	ok.

mysse_sup.erl
-module(mysse_sup).
-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
	supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
	Procs = [],
	{ok, {{one_for_one, 1, 5}, Procs}}.

I rewrote the test in Erlang. I am using Cowboy 2.6.3 and jsone 1.5.1
I am testing with cURL like so curl -X GET 'localhost:8080' -H 'Expect:'.
sse_handler.erl
-module(sse_handler).
-behaviour(cowboy_loop).

-export([init/2, info/3]).

%% The size of a message's data value in bytes:
-define(n_data_bytes, 1000).
%% The total transmission size in bytes:
-define(n_total_bytes, 100000000).

init(Req0, State) ->
    Headers = #{
        "content-type" => <<"text/event-stream; charset=utf-8">>,
        "cache-control" => <<"no-cache">>,
        "access-control-allow-origin" => <<"*">>,
        "connection" => <<"keep-alive">>
    },

    %% Set up the connection:
    Req = cowboy_req:stream_reply(200, Headers, Req0),

    self() ! run,

    {cowboy_loop, Req, State}.

info(run, Req, State) ->
    %% Run stream_data and measure execution time:
    {Time, N_streamed_bytes} = timer:tc(fun() -> stream_data(Req) end),

    io:format("Streamed ~w kB in ~w seconds.", [N_streamed_bytes, Time / 1000000]),

    %% Close the SSE connection:
    cowboy_req:stream_events(#{}, fin, Req),

    %% ..and stop the loop handler:
    {stop, Req, State}.

duplicate(Str, Times) ->
    string:join(lists:duplicate(Times, Str),"").

stream_data(Req) ->
    stream_data(Req, 0).

%% Sends at least @n_total_bytes of data through the SSE stream.
stream_data(Req, N_streamed_bytes) when N_streamed_bytes < ?n_total_bytes ->
    Event = #{data => duplicate("X", ?n_data_bytes)},
    cowboy_req:stream_events(Event, nofin, Req),

    %% Estimate the message size (lower bound) and count it to the number of sent bytes:
    Event_as_string = jsone:encode(Event),
    stream_data(Req, N_streamed_bytes + byte_size(Event_as_string));
stream_data(_, N_streamed_bytes) ->
    N_streamed_bytes.

mysse_app.erl
-module(mysse_app).
-behaviour(application).

-export([start/2]).
-export([stop/1]).

start(_Type, _Args) ->
    Dispatch = cowboy_router:compile([
        {'_', [{"/", sse_handler, []}]}
    ]),
    {ok, _} = cowboy:start_clear(my_http_listener,
        [{port, 8080}],
        #{env => #{dispatch => Dispatch}}
    ),
    mysse_sup:start_link().

stop(_State) ->
	ok.

mysse_sup.erl
-module(mysse_sup).
-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
	supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
	Procs = [],
	{ok, {{one_for_one, 1, 5}, Procs}}.

I don't really understand. Your duplicate/2 function is still doing a lot of pointless operations. If you really want to test Cowboy you need to use literals, like <<"aaaaaaaaaaaaaaaaaaaaaaa...">>. Macros can help build large strings at compile time. Then there's the jsone:encode/1 which seems entirely unnecessary and not at all representative of how much data was sent? SSE doesn't use JSON.
Anyway after fixing this, and using the following handler:
%% Feel free to use, reuse and abuse the code in this file.

%% @doc Hello world handler.
-module(toppage_h).

-export([init/2]).

-define(data, <<"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX">>).
%% The size of a message's data value in bytes:
-define(n_data_bytes, 1000).
%% The total transmission size in bytes:
-define(n_total_bytes, 100000000).

init(Req0, Opts) ->
    Headers = #{
        "content-type" => <<"text/event-stream; charset=utf-8">>,
        "cache-control" => <<"no-cache">>,
        "access-control-allow-origin" => <<"*">>,
        "connection" => <<"keep-alive">>
    },

    %% Set up the connection:
    Req = cowboy_req:stream_reply(200, Headers, Req0),

    {Time, N_streamed_bytes} = timer:tc(fun() -> stream_data(Req) end),

    io:format("Streamed ~w kB in ~w seconds.", [N_streamed_bytes, Time / 1000000]),

    %% Close the SSE connection:
    cowboy_req:stream_events(#{}, fin, Req),

	{ok, Req, Opts}.

stream_data(Req) ->
    stream_data(Req, 0).

%% Sends at least @n_total_bytes of data through the SSE stream.
stream_data(Req, N_streamed_bytes) when N_streamed_bytes < ?n_total_bytes ->
	Data = ?data,
    Event = #{data => Data},
    cowboy_req:stream_events(Event, nofin, Req),

    %% Estimate the message size (lower bound) and count it to the number of sent bytes:
    stream_data(Req, N_streamed_bytes + ?n_data_bytes);
stream_data(_, N_streamed_bytes) ->
    N_streamed_bytes.

With your command I get:
Streamed 100000000 kB in 5.527956 seconds.
curl -X GET 'localhost:8080' -H 'Expect:'  0,05s user 0,48s system 8% cpu 6,194 total

But curl prints all the events in the terminal. Terminals are super slow, so let's pipe the curl output to a file instead:
Streamed 100000000 kB in 1.079534 seconds.
$ time curl -X GET 'localhost:8080' -H 'Expect:' > /tmp/out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 96.1M    0 96.1M    0     0  89.0M      0 --:--:--  0:00:01 --:--:-- 89.0M
curl -X GET 'localhost:8080' -H 'Expect:' > /tmp/out  0,30s user 0,46s system 70% cpu 1,089 total
$ ls -l /tmp/out
-rw-r--r-- 1 essen essen 100800001 16 oct.  14:19 /tmp/out

1 second is pretty good for streaming 100MB of events. The culprit was therefore curl which probably does a receive+print loop and printing to the terminal is slower than sending/receiving data, at least for localhost.

Even when piping curl into /tmp/out or /dev/null, it is still taking about 16-20 seconds. I can assure you that it is not my machine as it is plenty powerful.

If you think Cowboy is slow then you should demonstrate that using only Cowboy. If you add JSON on top, then it might be the JSON code that's slow, not Cowboy. If you do duplicate operations, then it might be those. And so on. When benchmarking something you have to make sure it's not the benchmark tool or other external software that slows things down.
As far as I can see, whatever your issue is (and I'm not saying there isn't one because obviously something is going on), it does not appear to be in Cowboy. Cowboy can stream 100MB of events in 1 second. And like I said there's a 32MB stream test running fine as well (except on Windows when using TLS, but that's probably because the CI VM is misconfigured or something). So as far as Cowboy is concerned the ticket is closed unless you can show otherwise.


Return false in check_auth() is the deleted flag is true on the account

https://github.com/GoogleDeveloperExperts/experts-app-backend/commit/71f4f407457e5120943a67e69a2177f5a1313b5b


Create mixer CLI

For testing, debugging, API validation, etc., it would be nice to generate a simple CLI that can be used to send requests to a mixer instance.

I've checked in an initial cut of a client. This sets the basic scaffolding. As more of the mixer comes online, I'll continue extending the client accordingly.


Omniauth for the consumers

So earlier this year when I started working on OAuth consuming for Jux, I chose oauth-plugin over omniauth because a) at the time we only needed authorization (as opposed to authentication), and b) oauth-plugin handles the persisting of tokens for us, which we can use for API calls.
Since then I've submitted a few patches to oauth-plugin, but it seems like an endless struggle to keep up with the constantly growing number of OAuth providers.  It seems especially silly, since the folks at Intridea are already doing so.
While the two projects have slightly different goals, it seems like there's a lot of overlap functionality that could be shared and benefit both areas, much in the way that ActiveModel was abstracted from ActiveRecord and is now used in many non-SQL ORM's.
My initial thought is that oauth-plugin's consumer functionality could be largely handled by omniauth, and then just do the persisting of the tokens after the OAuth sequence is complete.  I like that oauth-plugin doesn't assume it's responsible for user authentication, so I need to make sure that's possible while using omniauth under the hood.
I really just wrote this down to solicit feedback - will probably start digging into the code soon.  I'm hoping @mbleigh and @sferik will chime in with their thoughts, too :-)
Thanks!

@pelle @mbleigh @sferik any thoughts?

It seems to me that oauth-plugin is two projects in one: a provider and a consumer. The consumer side essentially feels like an OAuth-only version of OmniAuth. My vote would be not to have the consumer side powered by OmniAuth but to remove the consumer side and either simply deprecate it or put it in its own project to grow separately.
That being said, I've never used this library so I should have no influence on its direction.

oauth-plugin is a value-add over omniauth on the consumer end in that it automatically persists the token and provides a [SERVICE_NAME]/client/[ENDPOINT], which takes out a lot of boilerplate if it's being used to interact with multiple APIs.  Very good point that acting as a consumer vs. a provider are two very different use cases, though.  Here is my ideal landscape:

omniauth: handles authentication and/or authorization
(project 1): persists tokens received through Omniauth, and provides a consistent API wrappers and optional controller endpoints for accessing via the client
(project 2): an OAuth provider

I have on my todo list to split the oauth plugin into 2 separate projects.
But OmniAuth does seem to have a well maintained list of providers.
I haven't looked at home OmniAuth, but it seems like it would be useful to be able to add omniauth strategies to the oauth plugin configuration.


Please take a look through the existing issues to see if your problem has already been reported and reply to that rather than creating a new thread :)
Description
So I have been playing on a world and got the point where I could go find a stronghold. I went to where the stronghold was according the the ender eyes and it just pointed to a huge set of ravines that spawned in each other. After getting frustrated I switched to creative and just went to town with my fist. There was no stronghold. I started a new world to see if I had gotten unlucky and the same thing happens. Repeat this several times I did.
How to reproduce

Standard Forge Server
No other third-party mods
Start a world and go find a fortress through use of Ender Eyes.

Provide versions and logs pls


Create a new workspace called products

Version info:

Scale on master (27th October 2016)
Vagrant 1.8.7.dev
ansible 2.3.0

I am trying to deploy the example landsat recipe within the ansible init-examples.
However it is expecting a workspace called "products" to already exist!
How can I create a workspace in Scale? Can I do this via a POST request to the api/v3/workspaces/? Can someone point me at an example of this.
Currently it seems that I have no workspaces as when I send the following GET request I get a count of zero.
GET request to http://scale-master/api/v3/workspaces/

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

GET request to http://scale-master/api/v3/version/

{
    "version": "3.1.0"
}

To create a new workspace you can use the existing ansible yml file at:
ansible/roles/init-examples/tasks/main.yml
Below the comment is on line 2 of this yml file:
# configure workspaces
Add the logic:
# Use template to create the products workspace.
- name: Configure products workspace from template
  tags: init-examples
  template: src=products_ws.yml.j2 dest='/tmp/products_ws.yml'

- name: Create the products workspace
  tags: [ init-examples, example-database ]
  command: /scale/bin/goscale ws create -d /tmp/products_ws.yml
  become: true
  environment:
    DOCKER_REGISTRY: "{{ example_docker_registry | replace('/', '') }}"
    SCALE_URL: 'http://{{ mesos_master_ip }}/api/v3'

Then send a GET request to your workspaces RESTful end-point:
GET request to http://scale-master/api/v3/workspaces/

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 2,
            "name": "products",
            "title": "products",
            "description": "Product storage",
            "base_url": "http://<IP_ADDRESS>:8080/products",
            "is_active": true,
            "used_size": null,
            "total_size": null,
            "created": "2016-11-09T06:14:28.764007Z",
            "archived": null,
            "last_modified": "2016-11-09T06:14:28.764044Z"
        }
    ]
}

Hope someone else finds this useful.


Quantization for V100

Similar to #1252 , do we have any plans for supporting V100. For now I can see that the place need to be modified is ldmatrix instruction and m16n8k16, as an example we may need to load the matrix manually and perform the mma in a smaller size, for example, maybe we need something similar to these
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ == 700
          // Manually loading each fragment, ldmatrix only available on sm_75 and after
          __asm__ __volatile__(
              "ld.shared.b16 %0, [%4];\n"
              "ld.shared.b16 %1, [%4 + 2];\n"
              "ld.shared.b16 %2, [%4 + 4];\n"
              "ld.shared.b16 %3, [%4 + 6];\n"
              : "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[0]),
                "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[1]),
                "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[2]),
                "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[3])
              : "r"(addr)
          );
#else
          __asm__ __volatile__(
            "ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16"
            "{%0, %1, %2, %3}, [%4];\n"
            : "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[0]), "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[1]), "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[2]), "=r"(((unsigned *)(B_shared_warp + (ax1_0 * 8)))[3])
            : "r"(addr)
          );
#endif

and
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ == 700
      for (int m_idx = 0; m_idx < 2; m_idx++) {
          for (int k_idx = 0; k_idx < 4; k_idx++) { // original K was 16
            unsigned *A_addr = &(((unsigned *)(A_shared_warp))[m_idx * 2 + k_idx * 8]); // 2 elements per 8x4 block, adjusted for K
            for (int n_idx = 0; n_idx < 2; n_idx++) {
              unsigned *B_addr = &(((unsigned *)(B_shared_warp + (j_0_4 * 8) + (n_idx * 4)))[k_idx * 8]); // adjusted for K
              float *C_addr = &(((float *)(C_warp + (j_0_4 * 8) + (m_idx * 4)))[n_idx * 4]);
              __asm__ volatile(
                  "mma.sync.aligned.m8n8k4.row.col.f32.f16.f16.f32 "
                  "{%0,%1,%2,%3,%4,%5,%6,%7}, {%8,%9}, {%10,%11}, "
                  "{%12,%13,%14,%15,%16,%17,%18,%19};\n"
                  : "=f"(C_addr[0]), "=f"(C_addr[1]), "=f"(C_addr[2]), "=f"(C_addr[3]),
                    "=f"(C_addr[4]), "=f"(C_addr[5]), "=f"(C_addr[6]), "=f"(C_addr[7])
                  : "r"(A_addr[0]), "r"(A_addr[1]), "r"(B_addr[0]), "r"(B_addr[1]),
                    "f"(C_addr[0]), "f"(C_addr[1]), "f"(C_addr[2]), "f"(C_addr[3]),
                    "f"(C_addr[4]), "f"(C_addr[5]), "f"(C_addr[6]), "f"(C_addr[7])
              );
            }
          }
        }
#elif defined(__CUDA_ARCH__) && __CUDA_ARCH__ == 750
        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k8.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5}, {%6}, {%7, %8, %9, %10};\n"
            :  "=f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[0]), "r"(((unsigned *)(A_shared_warp + 0))[1]), "r"(((unsigned *)(B_shared_warp + (j_0_4 * 8)))[0]), "f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "f"(((float *)(C_warp + (j_0_4 * 8)))[3]));
        }

        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k8.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5}, {%6}, {%7, %8, %9, %10};\n"
            :  "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[0]), "r"(((unsigned *)(A_shared_warp + 0))[1]), "r"(((unsigned *)(B_shared_warp + ((j_0_4 * 8) + 4)))[0]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3]));
        }

        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k8.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5}, {%6}, {%7, %8, %9, %10};\n"
            :  "=f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[2]), "r"(((unsigned *)(A_shared_warp + 0))[3]), "r"(((unsigned *)(B_shared_warp + (j_0_4 * 8)))[1]), "f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "f"(((float *)(C_warp + (j_0_4 * 8)))[3]));
        }

        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k8.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5}, {%6}, {%7, %8, %9, %10};\n"
            :  "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[2]), "r"(((unsigned *)(A_shared_warp + 0))[3]), "r"(((unsigned *)(B_shared_warp + ((j_0_4 * 8) + 4)))[1]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3]));
        }
#else
        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, {%10, %11, %12, %13};\n"
            :  "=f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "=f"(((float *)(C_warp + (j_0_4 * 8)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[0]), "r"(((unsigned *)(A_shared_warp + 0))[1]), "r"(((unsigned *)(A_shared_warp + 0))[2]), "r"(((unsigned *)(A_shared_warp + 0))[3]), "r"(((unsigned *)(B_shared_warp + (j_0_4 * 8)))[0]), "r"(((unsigned *)(B_shared_warp + (j_0_4 * 8)))[1]), "f"(((float *)(C_warp + (j_0_4 * 8)))[0]), "f"(((float *)(C_warp + (j_0_4 * 8)))[1]), "f"(((float *)(C_warp + (j_0_4 * 8)))[2]), "f"(((float *)(C_warp + (j_0_4 * 8)))[3]));
        }

        {
          __asm__ __volatile__(
            "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32"
            "{%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, {%10, %11, %12, %13};\n"
            :  "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "=f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3])
            : "r"(((unsigned *)(A_shared_warp + 0))[0]), "r"(((unsigned *)(A_shared_warp + 0))[1]), "r"(((unsigned *)(A_shared_warp + 0))[2]), "r"(((unsigned *)(A_shared_warp + 0))[3]), "r"(((unsigned *)(B_shared_warp + ((j_0_4 * 8) + 4)))[0]), "r"(((unsigned *)(B_shared_warp + ((j_0_4 * 8) + 4)))[1]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[0]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[1]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[2]), "f"(((float *)(C_warp + ((j_0_4 * 8) + 4)))[3]));
        }
#endif
      }

Yes, it would certainly be good if we could have it implemented for V100. It will just not be that fast because you need several more instructions, but probably still much better than FP16 for speed.

Yes, it would certainly be good if we could have it implemented for V100. It will just not be that fast because you need several more instructions, but probably still much better than FP16 for speed.

Yes, this naive implementation I posted here is not efficient as I tested, more data movements and matrix multiplications make it much slower than expected. BTW I wonder why we don't use nvcuda::wmma here for better compatability across different architechtures and readability.

BTW I wonder why we don't use nvcuda::wmma here for better compatability across different architechtures and readability.

I am by no means a CUDA expert, but my understanding is that PTX is faster. However, if you are up to the task, you could try creating a new kernel by replacing the PTX instructions with wmma.

TRT-LLM had been released, and it support INT4/INT8 on Volta(SM70) architecture GPU.

@gesanqiu while the README says it works, that's sadly not the case for GPTQ and AWQ, see: NVIDIA/TensorRT-LLM#200. Same result with Turing. Only the plain int4 mode works, which is largely undocumented, and I guess for good reason. It appears to use a single scaling factor per tensor, as described here. I don't know if this quantization strategy has a name, but after trying a few examples, I'd call it poor.

Hi @Fangzhou-Ai where need modify some file for support v100 gpu? regards

Hi @jesulo, there are many places need to be modified, mostly for those instructions involving memory loading ldmatrix (like this line) and GEMM m16n8k8 for Turing arch and m16n8k16 for Ampere arch, examplese can be found here and here.

I just tested and gptq 4bit works on v100, but not 8bit (says no executor image). is this expected? i expected both to work

hope it comming soon~

Looking  forward


Moving test out of init, Fixes #4

Installing Pandas on a raspberry pi is very very slow, we dont need to load Pandas just to use Ballpark

Fixes #4

+1 for this fix.

+1

Love this library and had the same issue. I've forked and applied this exact patch. Just replace:
# ballpark
<EMAIL_ADDRESS>
in your requirements.txt


Auto-Map Dialog not Centered

In the import template configuration, the auto-map popup is no longer centered.

Issue resolved.


Cal reward, reward ui

Pull Request Test Coverage Report for Build 142

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.212%

Totals

Change from base Build 140:
0.0%

Covered Lines:
89

Relevant Lines:
97

💛  - Coveralls


PhpStorm ignores spaces during writing

Hi,
for example, when I need to write a space character in the command definition - between {} characters, PhpStorm does not display this space. He 's blocking her. I have to get out of the current line and after that move the cursor back.
This only happens while editing latte files.
For exmaple: {foreach$gamesas$game}

I don't think I fully understand your problem. Can you make a video with your problem? Or write me on Slack.

I don't think I fully understand your problem. Can you make a video with your problem? Or write me on Slack.

I probably have same issue. When I write latte templates, I just type {foreach as normal, than press spacebar and nothing happend. And next letter i press is right after foreach, so result looks like: {foreach$games
When i want write something like {foreach $games as $game}, I write {foreach $games, , , , and than I can type $games. It's realy annoying.
When I disable Latte plugin it works  fine, but whitout Latte plugin :man_shrugging:  I did NOT change any my setting and don't know how to fix it, I tried to change some setting after, but nothing helped. Thanks for any idea.
BTW - what slack workspace?

I probably have same issue. When I write latte templates, I just type {foreach as normal, than press spacebar and nothing happend. And next letter i press is right after foreach, so result looks like: {foreach$games
When i want write something like {foreach $games as $game}, I write {foreach $games, , , , and than I can type $games. It's realy annoying.
When I disable Latte plugin it works  fine, but whitout Latte plugin :man_shrugging:  I did NOT change any my setting and don't know how to fix it, I tried to change some setting after, but nothing helped. Thanks for any idea.
BTW - what slack workspace?

I had the same problem with last release. Can you try install EAP version? It fixed problem for me. Or you can just wait for JetBrains approval of just released 1.1.1 version.

BTW - what slack workspace?

Search here for "slack" keyword and invite yourself https://pehapkari.cz

I had the same problem with last release. Can you try install EAP version? It fixed problem for me. Or you can just wait for JetBrains approval of just released 1.1.1 version.

BTW - what slack workspace?

Search here for "slack" keyword and invite yourself https://pehapkari.cz

I have/had similar problem in html unable to write space after tag name when I wanted continue with class attribute.

I have/had similar problem in html unable to write space after tag name when I wanted continue with class attribute.

For me it got fixed by adding this repo https://plugins.jetbrains.com/plugins/eap/list to custom repo list and updated plugin to 1.1.1-RC2.

For me it got fixed by adding this repo https://plugins.jetbrains.com/plugins/eap/list to custom repo list and updated plugin to 1.1.1-RC2.

For me it got fixed by adding this repo https://plugins.jetbrains.com/plugins/eap/list to custom repo list and updated plugin to 1.1.1-RC2.

That works for me! thanks!

For me it got fixed by adding this repo https://plugins.jetbrains.com/plugins/eap/list to custom repo list and updated plugin to 1.1.1-RC2.

That works for me! thanks!

Well, I take it as if it works in 1.1.1-RC2. 1.1.1 stable will soon be approved by JetBrains.

Well, I take it as if it works in 1.1.1-RC2. 1.1.1 stable will soon be approved by JetBrains.


Fix: example queued_interceptor_csrftoken.dart

Provide a more suitable example for queued_interceptor_csrftoken.dart

New Pull Request Checklist

[x] I have read the Documentation
[x] I have searched for a similar pull request in the project and found none
[x] I have updated this branch with the latest main branch to avoid conflicts (via merge from master or rebase)
[ ] I have added the required tests to prove the fix/feature I'm adding
[ ] I have updated the documentation (if necessary)
[x] I have run the tests without failures
[ ] I have updated the CHANGELOG.md in the corresponding package

Additional context and info (if any)
This example adds 2 different QueuedInterceptorsWrapper and the last one handles onError to update csrf token.

fixes #2128

I'll check the type exceptions in test_suite_test.dart

The failing tests should be random, caused by the httpbun, fyi.

Someday we'd expect to have a unit test with token updates.

I'll also be on the lookout for cases to write unit tests for token refresh.


allow syncing entries to google calendar

is it possible to allow the logs to be synced into discrete events into google calendar or 3rd party scheduling tools ?

There is no such feature yet but you could export the logs as CSV and then process them as you please.


Adding library way

@jtimberman @someara

@tas50 want to take a look at this?

too much clicking

@acaiafa Haven't had time to really test this but it's missing a bunch of memcached options that were added in 1.8.0.

@stonith pretty simple fix. I can add them directly OR you can you show me a README for these additional memcached options?
I can always add in a simple options_collector and that will solve this also.

@acaiafa https://github.com/chef-cookbooks/memcached lists the additional options.

Ok i mean thats a small subset of options. I will push this up later. Can you just examine the bulk of the work and see if it fits.

Anthony,
I think the spirit of this PR has been implemented in the new release.  Users still have the opportunity to install memcached under the system system's init system, but the old definition has been replaced with a 12.5 custom resource that behaves as it should.  The mix of attributes and definition is gone, which makes it more of a true library cookbook.  There is a current restart bug that I'm working to track down, but let me know if this solves the need for a library cookbook.
-Tim


dawson-three-tuts

Note proposed tags for "Installation Guide for OCI Monitoring." "Ansible" seems reasonable enough, but unsure how you may want to format "data-visualization."

@ttscoff Made suggested revisions. I do believe this can be merged, but just wanted to run by you before doing is myself

Actually, @snafuz Can this be merged? I think this advocate content and I'm still unclear if they've approved.

@ttscoff Updated the shell prompts so this should be good to merge when we're able to, whenever that'll be ¯_(ツ)_/¯


Add automatic docs deployment to GitHub Pages

Objective
Partially address #407 by setting up automated deployments of bevy's API reference to GitHub Pages.
Solution
Set up a GitHub Actions workflow that builds the docs on every push to main and pushes a new commit to a gh-pages (or docs branch).
A few smaller additions to better address #407:

A top level redirect was added to take "docs.bevyengine.org" directly to the bevy crate docs.
A GitHub Pages CNAME file for supporting a publicly viewable domain instead of github.io
A robots.txt file is added to disable all search engine crawlers that respect it from crawling it (avoid having conflicting Google search results)
A .nojekyl file to speed up deployments since there is no Jekyll templating in the output.

This may require configuration of the  GITHUB_TOKEN of the CI to successfully run this.
Followup
For this to completely resolve #407, a subdomain of https://bevyengine.org/ needs to be set up to point to the CNAME location. This is initially set to "docs.bevyengine.org".

Great! This will close bevyengine/bevy-website#181.

A message on top could say "Development build docs. Click for the latest release docs", or something like that.
Alternatively, we can have a dropdown menu like docs.rs to choose between development and release docs.

Added a sed command that changes the icon out for something more noticeable., and added a header.html shim to force Google not to index, even when linked from other sites. Other search engines should basically stop at the robots.txt.

Before more people go looking: the new icon is coming in bevyengine/bevy-website#234

With https://github.com/bevyengine/bevy-website/pull/234 merged, @mockersf, any other thing that's needed? It's a little hard to notice right now, but it should be enough to distinguish between it and docs.rs. I could add a sticky banner as suggested earlier, but --html-before-content seems to shove it in a row-oriented flexbox due to the navbar. Not familiar enough with HTML/CSS to make it actually stick above everything.

This should be good to go now.
The target domain/url is something that is up to Cart. Maybe it should not live on the website / under a bevy engine.org url as we try to keep the website only for the released version? Maybe a subdomain would be enough to differentiate?
I think the url where it would be deployed without custom domain would be https://bevyengine.github.io/bevy, that could help with having website for release, GitHub for in development?

Yeah I'm ready to move forward on this. I do think that docs.bevyengine.org is the wrong name for "in development / unstable" docs. I do think a github url is a reasonable way to correlate the context to "in development", but its worth calling out that github also hosts our "stable releases and branches". We might be able to make it even more clear with a custom url. unstable-docs.bevyengine.org, dev-docs.bevyengine.org, dev-api.bevyengine.org etc.
I think I like the dev-docs prefix, for symmetry with docs.rs. I am interested to hear what everyone else thinks.

I agree with the dev-docs subdomain. It seems recognizable at first glance, shorter than unstable-docs, and doesn't look like a REST API subdomain, unlike dev-api.

Cool lets roll with that. I'll get the domain set up.

We might actually need to merge this pr first. Can you update the domain?

bors r+


Update CLI to match support for the new --include-node-modules flag

This flag is available for the gcloud beta functions deploy command.

This flag has changed in latest gcloud (FYI)
--include-ignored-files


Support rewriting Hadamard edge

It would be nice if zxlive supports rewriting the Hadamard edge:

ZXLive supports custom rewrites so you can add this as a custom rewrite in the Rewrite menu

Oh, I did not know that. Thank you!


Minify XML

I have some XML files that could be minified, such as my sitemap.xml, rss.xml, and styles.xsl. It would be nice if these were minified as well.
Maybe could use something like https://github.com/kristian/minify-xml?

Hey, nice feature! I'll consider adding it. Thank you!


references class "Doctrine\ODM\MongoDB\UnitOfWork" but no such service exists.

I,
when i try to use injection dependency into a controller in Symfony4 i have this error :
Cannot autowire service "App\Repository\UserRepository": argument "$uow" of method "Doctrine\ODM\MongoDB\DocumentRepository::__construct()" references class "Doctrine\ODM\MongoDB\UnitOfWork" but no such service exists.
My dependency is a Service (UserService) who has a  repository injection (UserRepository).  This UserRepository extends DocumentRepository.
Is it normal because i found a solution but may be i make mistake ?
Thank

I think this is still the state of autowiring, https://github.com/doctrine/DoctrineMongoDBBundle/issues/443#issuecomment-385347539.

@SenseException is right, closing as autowiring stuff is specific to Symfony bundle and as such needs to be implemented there.

Ok, for me i want use autowiring for injecting service easly. I add that to my services.yml :
`
mongodb.odm.unit_of_work:
class: Doctrine\ODM\MongoDB\UnitOfWork
autowire: true
Doctrine\ODM\MongoDB\UnitOfWork<EMAIL_ADDRESS>mongodb.odm.mapping.class_metadata:
    class: Doctrine\ODM\MongoDB\Mapping\ClassMetadata
    autowire: true
    arguments:
        $documentName: 'StdClass'
Doctrine\ODM\MongoDB\Mapping\ClassMetadata<EMAIL_ADDRESS>
mongodb.common.event_manager:
    class: Doctrine\Common\EventManager
    autowire: true
Doctrine\Common\EventManager<EMAIL_ADDRESS>

@libressence I'm pretty sure you're better of aliasing existing Symfony services instead of recreating them. What you've done is error prone and will probably result in issues further down the road.

@libressence at a glance this seems like a bad idea. For instance the EventManager you have created manually will not be the same as the one DocumentManager is using internally, so it won't have same listeners subscribed to events. ClassMetadata is also wrong, usually there is no point in having one that maps \stdClass. All in all you should use factory notation for services and get their instances from DocumentManager to ensure consistency:
mongodb.common.event_manager:
    class: Doctrine\Common\EventManager
    factory: ['@ref_to_your_dm', 'getEventManager']
Doctrine\Common\EventManager<EMAIL_ADDRESS>
I can't recall if it's true for all services but many of them will already be available in the Container so you will only need to alias interfaces:
Doctrine\Common\EventManager: '@doctrine_prolly_has_event_manager_registered`.

For their name please consult command like debug:container in your application.

Ok, thank @malarzm and @Steveb-p i will do it ! Hope that will work :)

If i can has a complete example for it work, that will be very great ! Because i search how to do what you say @malarzm but i don't understand exactly. Impossible to use mongodb on symfony for now

It's not impossible to use ODM with Symfony, it's just not possible to use autowiring. Use old way of mapping services and you'll be fine, here's chapter on factories.

Ok @malarzm :). I know it's possible, i just wanted to use new injection service system of symfony.
But ok... I will not use this for the moment...
Thank

@libressence It's not about not being able to use the injection to other services/controllers, it's about the autowiring (which basically means Symfony detects that only 1 service of a particular class exists and automatically assumes it's the correct one. Or more accurately, service is named exactly as the class you're requesting to inject).
Here's a sample that should be helpful:
App\Repository\UserRepository:
    factory: ["@doctrine_mongodb", getRepository]
    # class: App\Repository\UserRepository # Note that this can be omitted since classname equals service name, which is what we want
    arguments: # Note that this argument is passed to factory method (getRepository) not the service instance
      - App\Document\User

Then you can use the injection/autowiring feature in other services/controller methods (if your Symfony is configured to do so).


[BC] Bug when calculating dBA if frequency normalization (elec_order/mech_order) is selected

Hi all,
In W.get_along("freqs->elec_order", unit="dBA"), the normalization seems to be applied to the frequency vector that is also used to calculate A-weighting, resulting in inconsistent dBA.
Regards,
Emile

Solved in last release


iOS not showing Now Playing Information

On iOS the information about the current playing audio doesn't show. It only shows the Url of the media and the play/pause button.

make sure you change your audio provider to CordovaMediaProvider in the ionic-audio.module

@area809  yep, found the issue, it was actually loading the WebProvider... changed it properly to use the CordovaMediaProvider and now it is working as expected. Thank you

@area809 @walalm  sorry can one of you tell me how to change the audio provider? I am having the same issue. I am currently using the html5 audio player to play audio in my ionic app, do I need to change to cordova-plugin-media? Or is this just a setting change somewhere? Thanks!

@area809 @walalm sorry can one of you tell me how to change the audio provider? I am having the same issue. I am currently using the html5 audio player to play audio in my ionic app, do I need to change to cordova-plugin-media? Or is this just a setting change somewhere? Thanks!

Change it to use the cordova plugin media

Cool thanks, will try that!
On Wed, Sep 19, 2018, 7:45 PM Waldy Almonte<EMAIL_ADDRESS>wrote:

@area809 https://github.com/area809 @walalm https://github.com/walalm
sorry can one of you tell me how to change the audio provider? I am having
the same issue. I am currently using the html5 audio player to play audio
in my ionic app, do I need to change to cordova-plugin-media? Or is this
just a setting change somewhere? Thanks!
Change it to use the cordova plugin media
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/homerours/cordova-music-controls-plugin/issues/128#issuecomment-423021949,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKw35BthaR498_RRPmn5FQsiDHkmefSfks5ucwEygaJpZM4UyqRZ
.


Add a lightweight http client

C++ Requests is a simple wrapper around libcurl
inspired by the excellent Python Requests project.
It supports fetch_content and is easy to integrate into Velox so it perfectly
matches the  Velox's hermetic building philosophy.
This is the first step to add PrestoQueryRunner in Velox.

CI is red, it seems to be not related to this PR.
-- Configuring done (60.6s)
CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "velox_exec" of type SHARED_LIBRARY
    depends on "velox_codegen" (weak)
  "velox_codegen" of type SHARED_LIBRARY
    depends on "velox_exec" (weak)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries.
CMake Generate step failed.  Build files cannot be regenerated correctly.

I thought about this PR a bit more. SimpleHttpClient.* is indeed very simple and it is not clear whether it is needed at all. How about we just add a couple small test cases that illustrate how to use cpr and leave it at that. Let's also not add disabled test that talks to Presto. It is not clear what's the value in checking that in.

@mbasmanova Got it, will remove the SimpleHttpClient.* and just add a CprHttpClientTest.cpp to illustrate how to use cpr.

CI is red, it seems to be not related to this PR.
-- Configuring done (60.6s)
CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "velox_exec" of type SHARED_LIBRARY
    depends on "velox_codegen" (weak)
  "velox_codegen" of type SHARED_LIBRARY
    depends on "velox_exec" (weak)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries.
CMake Generate step failed.  Build files cannot be regenerated correctly.

This is caused by the libcpr/cpr set BUILD_SHARED_LIBS ON, which must be OFF in Velox. I set it OFF in cpr.cmake, another option is to add a git patch file and apply it through FetchContent_Declare in cpr.cmake.
cc @mbasmanova @kgpai @assignUser

@duanmeng My one concern with the test is that requires open access to internet and most places test runners might be closed off from the general internet - would it be feasable to setup a small local server and have it communicate with that ?
cc: @mbasmanova

We can also just DISABLE_ the test.

@duanmeng My one concern with the test is that requires open access to internet and most places test runners might be closed off from the general internet - would it be feasable to setup a small local server and have it communicate with that ?
cc: @mbasmanova

@kgpai This test case is an illustration purpose (see comment ), Could we remove this success get, only test the failure scenario?

@duanmeng Maybe add this to examples or just create it as a binary in tests like the fuzzer expression verifier so that its used only for adhoc testing.
What do you think @mbasmanova ?

We can also just DISABLE_ the test.

Got it, done! cc @mbasmanova @kgpai

@duanmeng Please ping me when the build failures are resolved and I will start the merge process. Looks good otherwise.

@duanmeng Please ping me when the build failures are resolved and I will start the merge process. Looks good otherwise.

@kgpai Sure, thanks a lot.

It seems to actually be an issue on cpr's side by using a deprecated callback https://github.com/libcpr/cpr/issues/932 so the issue is that our containers haven a too new version of curl... 🤦 As I understand it this PR addition is a testing utility right? In that case we could run it in a build that does not build any other dependencies that also need curl like linux-build (which was building fine before we forced the system version).

@assignUser Hi Jacob, thank you for your help, it is much appreciated.

As I understand it this PR addition is a testing utility right?

Yes, it is intended to be used in velox/exec/test by AggregationFuzzer.

we could run it in a build that does not build any other dependencies

Do you mean to put the set(cpr_SOURCE BUNDLED), resolve_dependency(cpr) into the velox/exec/tests/CMakeLists.txt?

Do you mean to put the set(cpr_SOURCE BUNDLED), resolve_dependency(cpr) into the velox/exec/tests/CMakeLists.txt?

We could do that but I think I prefer to have all dependencies optional or not in the root cml so we have a better overview. There I would gate it with testing && velox_minmal or something like that? the tests using cpr would ofc need to be gated similarly.

prefer to have all dependencies optional

Do you mean sth like the following?
if(${VELOX_BUILD_BENCHMARKS})
  set(VELOX_BUILD_TESTING OFF)

if(VELOX_BUILD_TESTING)
  set(cpr_SOURCE BUNDLED)
  resolve_dependency(cpr)
endif()

BTW, I used to use non-system curl (cpr's curl 8.4.0) but also encountered the issue https://github.com/curl/curl/issues/11692.
https://app.circleci.com/pipelines/github/facebookincubator/velox/37362/workflows/c8c0f36c-783b-498c-9c6e-f83f7d8ae73e/jobs/249983
-- SSL auto detect: Using OpenSSL.
-- Configuring built-in curl...
-- Using CMake version 3.17.5

_deps/cpr-src/include/cpr/util.h:26:36: error: 'CURL_PROGRESSFUNC_CONTINUE' was not declared in this scope
   26 |     static_assert(cancel_retval != CURL_PROGRESSFUNC_CONTINUE);

Something like that but also checking if any of the other features that need curl are build (namely adapters) but as you also have the issue with the build in curl version the problem might be unfixable from our side? I would defenetly recommend engaging in the linked issue above to create visibility for the maintainer.
That callback was apparently removed around 7.61 or so so it's not suprising that it happens with that 8.x version. But it seems suprising that a version that is not compatible with the library in question would be used and not throw an error in their CI?
Also not specifically related to this PR or you but: I have seen the "squash into one commit and force push" workflow often in velox PRs but I dislike it as it makes it much harder to see what actually changed between reviews which forces me to look at all of the changes again only to discover that it was just a linting change or w/e. Additionally it obfuscates previous CI run and makes it hard for me to find the correct runs/logs as I can't use the :x: ui next to the old commit anymore. But again not specific to you and I will mention it in slack or so ^^

Hm I think we can just patch it out using the fetch content PATCH step (see folly or gflags resolve_dependency for examples) because as far as I can see the statix assert can never be true even if that value was defined?
    const int cancel_retval{1};
    static_assert(cancel_retval != CURL_PROGRESSFUNC_CONTINUE);

https://github.com/curl/curl/blob/4855debd8a2c1cbd0b0dbbb8319b1743c4644873/include/curl/curl.h#L232
#def in older curl version
https://github.com/curl/curl/blob/4855debd8a2c1cbd0b0dbbb8319b1743c4644873/include/curl/curl.h#L232

it much harder to see what actually changed between reviews which forces me to look at all of the changes again only to discover that it was just a linting change or w/e. Additionally it obfuscates previous CI runs and makes it hard for me to find the correct runs/logs as I can't use the ❌ ui next to the old commit anymore.

@assignUser Yes, this is a nice point. Indeed squash and force push bring more burden to the reviewer. We should avoid this (except for rebase that needs force push) as much as possible.

Hm I think we can just patch it out using the fetch content PATCH step (see folly or gflags resolve_dependency for examples) because as far as I can see the statix assert can never be true even if that value was defined?

@assignUser Got it, I will have a try, thanks a lot.

@kgpai The CI is successful now, with the great help from @assignUser


Evaluation of iterables in generator expressions uses wrong scope

In Python, this works:
it = (1,2,3)
l = list(it for it in it)

In current Cython, this raises an error at runtime as the iterable inside of the generator expression refers to the loop target variable, not the name in the outer scope. (So the iterable happens to be None.)
The name of the iterable (or rather the entire iterable expression) must be evaluated in the outer scope.
Actually, it's even different - CPython simply pre-evaluates the outermost iterable when creating the generator. Example (Py3.2):
>>> x = [range(10)]
>>> g = (i for x in x for i in x)
>>> list(g)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> x = [range(10)]
>>> g = (i for i in range(5) for x in x)
>>> list(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <genexpr>
UnboundLocalError: local variable 'x' referenced before assignment

So, basically, the outermost iterable should be evaluated into a closure variable of the generator, instead of letting the generator evaluate it itself. This:
>>> x = [range(10)]
>>> g = (i for x in x for i in x)

should be transformed into
x = [range(10)]

def _gen(_outer_it):
    for x in _outer_it:
        for i in x:
            yield i

g = _gen(x)

Note that this also applies to list comprehensions with language_level=3.
Migrated from http://trac.cython.org/ticket/600

@scoder thanks for the cleanup--I see now how the bug described in this ticket is still a problem too.  I'll see if I can do something about it...

Apparently the original attempt to fix this was also reverted in a3d4461252315caeef3477399c49b4e9843cd479, which is part of my confusion as well.

I'd be happy to have more people look into this, but be "warned", it's not entirely trivial. Cython looks at the iterable in several places in order to optimise loops. Moving the iterable out of the generator too early will prevent these optimisations. Moving it out too late will lead to incorrect declarations inside and/or outside of the generator. This is one of the bugs that have been languishing for a bit of a reason.

Thanks for the admonition.  That will help me understand better what the problem is.

Nominating this as a release blocker for Cython 3.0 to keep it in sight, although it's not really a breaking change.


fix download of epsonscan2 source files

Running cmake . -B build should download the epsonscan2 source files, but it just creates a empty file with the name epsonscan2-<IP_ADDRESS>-1.src.tar.gz inside the build folder.
The problem is that Epson is checking the user agent to verify that a "real" user is requesting a download.
This command without a user agent returns a 403 Forbidden error.
wget https://support.epson.net/linux/src/scanner/epsonscan2/epsonscan2-<IP_ADDRESS>-1.src.tar.gz

This command containing a user agent downloads the file as expected.
wget --user-agent="Mozilla/5.0" https://support.epson.net/linux/src/scanner/epsonscan2/epsonscan2-<IP_ADDRESS>-1.src.tar.gz

I added the user agent header to the download command and with this modification the download is working again. Without this change the user agent defaults to the curl user agent (e.g curl/7.81.0), which probably all get blocked by Epson.

Thanks for the PR!
But it looks like that user agent isn't sufficient to bypass the user-agent filter.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36

Seems to work on my end even when sent from a macbook

If I try to download the file it sometimes works and sometimes it doesn't (returns 403 error). The behaviour is realy flaky.
Probably Epson doesn't only check the user agent.
I think it just makes more sense to host the file on a own mirror server.
curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" https://support.epson.net/linux/src/scanner/epsonscan2/epsonscan2-<IP_ADDRESS>-1.src.tar.gz

Newer versions seem to be hosted on another server that works fine: https://download3.ebz.epson.net/dsc/f/03/00/15/17/69/0ef02802c476a6564f13cac929859c394f40326a/epsonscan2-<IP_ADDRESS>-1.src.tar.gz
However, I can't seem the current version hosted there. So maybe we just update

I'll work on https://github.com/janrueth/es2button/pull/6 which won't require to download the files as it vendors a stripped down version of the headers already

Should not be required anymore with the recent changes. Closing this.
Thanks anyway for raising and working on this!


Document how to specify local MySQL db

DATABASES['default'] =  dj_database_url.config(default="mysql://myname:mypass@localhost:3306/db_name") works.

@trodicaro did you just remove the dict DATABASES = { } ?
in your settings.py you just added this line?
DATABASES['default'] = dj_database_url.config(default="mysql://myname:mypass@localhost:3306/db_name")

This is an old issue closing it off. If you feel this is still a requirement please create a new ticket.


Portfolio suggestions

You have done excellent work so far, but we have a couple of suggestions that could make your code better
Kindly suggest you avoid using multiple h1 to improve accessibility and SEO.
We suggest you when you create a repo, you can use the tool to create a License. This way will be easier to make changes when you need it,

Thanks for the feedback, all h1 except the main was changed to h2.


Patch for  type

Does the patch work for the object type?
I tried this method but was unsuccessful...
Request:
[
  {"op": "replace", "path": "number", "value": 86632}
]

Handler
[HttpPatch]
public IActionResult Patch([FromBody] JsonPatchDocument patchDoc)
{
      var jsonData = """"
            {
                 "number": 1111
            }
             """";

      var obj = JsonSerializer.Deserialize<object>(jsonData)!;

      patchDoc.ApplyTo(obj);

      return Ok(patchDoc);
}

Exceptions
he target location specified by path segment 'number' was not found.

Good question, I havent tested this behavior.

Yeah this doesn't work because System.Text.Json deserializes object as JsonElement type
You could however Deserialize jsonData as JsonNode
using System.Text.Json.Nodes;

[HttpPatch]
public IActionResult Patch([FromBody] JsonPatchDocument patchDoc)
{
      var jsonData = """"
            {
                 "number": 1111
            }
             """";

      var obj = JsonSerializer.Deserialize<JsonNode>(jsonData)!;

      patchDoc.ApplyTo(obj);

      return Ok(patchDoc);
}

Maybe that suits your need?

Thank you for your answer. Will try to with it.


Add lib.mkDefault in services.xserver.videoDrivers = [ "amdgpu" ];

I get a very weird problem
There is such a line of code in common/gpu/nvidia/default.nix
services.xserver.videoDrivers = lib.mkDefault [ "nvidia" ];
But in common/gpu/amd/default.nix which serves a similar purpose. It is
 services.xserver.videoDrivers = [ "amdgpu" ];
This means that if you use prime, that is, if you import these two files at the same time, services.xserver.videoDrivers = lib.mkDefault [ "nvidia" ]; will override services.xserver.videoDrivers = [ "amdgpu" ];.
Since the parameters provided to mkOverride are not the same, the smaller one is overwritten. So we need to keep them in sync
This issue should cause problems with all configurations using hybrid graphics, as the nvidia proprietary drivers won't be installed, but it doesn't seem to cause the system to completely crash, as nouveau will be automatically installed instead.
I checked the commit record, but it seems that the earliest commit on both sides is like this. It seems to be a very serious and very common problem but no one seems to be trying to fix it. I don’t know if there is any other purpose, but it makes common/gpu/nvidia/prime.nix completely useless.
Description of changes
Add lib.mkDefault in services.xserver.videoDrivers = [ "amdgpu" ];
Things done

[x] Tested the changes in your own NixOS Configuration
[x] Tested the changes end-to-end by using your fork of nixos-hardware and
importing it via <nixos-hardware> or Flake input

bors merge


Update README.md with link to GitHub's Saved replies feature

Hi, I googled "github canned responses", and only found links to your extension on the first page of results. When I saw your note in the README that this extension was superseded, it took me some more googling to find the GitHub Saved replies feature.
To help other users find this Saved replies feature, I added a direct link to the README (also fixed the markdown title and added a few more lights for symmetry).
(instead of https://help.github.com/articles/working-with-saved-replies/, you could also link to https://github.com/blog/2135-saved-replies, but in my experience the GitHub blog entries tend to get out of date after a while, so it's probably better to link to the actually documentation)

Thanks!


Does not correctly tear itself down

This codepen demonstrates: https://codepen.io/kphillips86/pen/orYpPM?editors=1011

https://github.com/canjs/can-stache-define-element/tree/teardown
branch with a test and partial fix


Application-specific mappings?

Is this possible with the current version? And if not, is it on the roadmap?
I started using the original Karabiner as it was suggested on the NoMachine support forums as a workaround for the described problem.
I use NoMachine from a MacBook to a remote Linux machine, and it results in the Command and Control keys being flipped resulting in inconsistent behavior between which keys function as Command/Control in the local and remote machines.

See #137

Probably should be closed as a duplicate.


Ryan McWeeny Review ticket

Individuals
Team+Indi+Teacher
Runtime
Issues
Key Commit(s)
Analytics

Ryan

Link
Video
Frontend  Backend
Ryan McWeeny

Individual Review Jishnu grading Ryan

[x] Individual should show that they were key contributor and example to team.  This includes their participation in ideas, plans, creating individual issues, providing code commits to project, crossover grading participation, being on task and positive example in the classroom.

Individuals Video, Issue(s), Commits(s)

[x] Video, includes Web demo of key contribution to project, 1 minute
[x] Issue(s) that show plans/progress to team objectives
[x] Highlights of key commit(s) in Issues, summarizes code contributions
[x] Review GitHub analytics for key commits in each weeks during the project, shows consistent participation for 3 weeks

Freeform comment

Very frequent commiter and works diligently based on the commit history shown on his profile
Good Video going through the actual code and showing what he did and talking about it
Shows the runtime live and the part he worked on and shows it while demonstrating

Final Score
0.8 - meets all standards


feat: PoC: Keyring backend for pip auth with ADO feeds

This PR implements a custom keyring backend that interfaces with ~/ado-auth-helper to provide authentication when pip needs to access an ADO feed. It is in the form of a Python package, named codespaces_artifacts_helper_keyring (open to renaming it).
Background
The artifacts-helper codespace feature provides authentication helpers for npm, yarn, etc. For many of these package managers, we simply use shell alises and environment variable substitution to inject the authentication tokens prior to calling the underlying package manager. However, this approach does not work with pip, namely because it has multiple entrypoints (e.g. python -m pip) and not all of them will respect shell aliases.
Luckily, pip has builtin support for keyring. Keyring is a framework that supports providing multiple authentication backends. In this PR, we implement a custom keyring backend to be used against ADO package feeds when the ~/ado-auth-helper tool is detected.
Inner Workings
The package works by providing a custom keyring.backend.KeyringBackend. This will automatically be picked up by pip and the keyring package when our codespaces_artifacts_helper_keyring is installed.
When our backend is called we:

Check whether we support the package index that is being accessed (e.g. does it live on dev.azure.com, etc?)
Check whether ~/ado-auth-helper is executable.
Call the auth helper and capture the token from stdout.
Pass the credentials to keyring, which will pass it to pip.

Usage
Usage on the target machine will require these prerequisites:
a. https://github.com/microsoft/ado-codespaces-auth is installed and ~/ado-auth-helper is executable
b. keyring is installed
c. the feed being accessed has a domain matching dev.azure.com, etc
To install the codespaces_artifacts_helper_keyring package:
cd src/artifacts-helper/codespaces_artifacts_helper_keyring

# PDM is used to manage the project
$ pip install 'pdm>=2.14'

# Install dependencies and build the package
$ pdm build

# Install package + deps with pip
$ pip install dist/codespaces_artifacts_helper_keyring-*.whl

To use the keyring backend
# Optionally set the pip package index globally
# The index should not contain any `username:password` details
pip3 config set global.index-url https://org.pkgs.visualstudio.com/ORG/_packaging/FEED/pypi/simple

# Use the feed via normal pip operations
pip3 download yapf  # Using global package index
pip3 download yapf -i <package index>  # Specify custom index

Testing
I've included some unit tests that are based on existing tests over at https://github.com/microsoft/artifacts-keyring. I've also added tests to exercise our process of calling the auth helper tool. The tests would be parameterized against multiple Python versions using nox.
I haven't extensively tested it yet! I've manually confirmed it's working with the latest version of all dependencies on Python 3.11.
Assumptions

I've assumed that the returned credentials will always be a valid JWT with the payload containing the username in upn or unique_name. Although, we can probably just pick another username to authenticate against ADO and it will still work.
I've assumed that ~/ado-auth-helper is always the location of the helper tool.

Considerations
Packaging and Distribution
How do we want to package this? These are the approaches that I can think of. Are you guys able to maintain it over the long term?

build and install from source

this might pollute the codespace's dev environment if we aren't careful

build the package on CI and release it on PyPI:

this might be considered a separate project and we might have to get approval for the release of a new publicly available pip package

build the package on CI and release on GitHub releases at Releases · microsoft/codespace-features (github.com)

installation would need to curl the github api to determine the latest release, download the wheel, and install it

wget latest release location
pip install codespaces_artifacts_helper_keyring.whl

build the package on CI and release on GitHub releases

Creating a release and installing from that makes the most sense. Will dependencies auto install? Is this installed globally or specific to the repo that is being worked on? As you said, we do not want to pollute anyone's repository.

Check whether we support the package index that is being accessed

How is this check done? The URL for the feed people use can vary. For example pkgs.dev.azure.com is the default but many teams use the older visualstudio.com URL and there may even be others.

Is this installed globally or specific to the repo that is being worked on? As you said, we do not want to pollute anyone's repository.

It can be installed globally to provide ADO auth for the entire codespace. If the user only wants authentication for a specific project, I believe they can create a virtual environment and only install our keyring backend inside that environment.

I do not think we need to complicate this. Codespaces is a very specific scenario and this feature is optional, and including Python support will be an optional setting of the feature. So if someone installs this feature AND says to setup pip then I think we can safely just do that. As long as a global installation is likely to work then that is likely what we should do.
When the install.sh script from this feature is executed when a Codespace is being built it is run as the root user. If we install this with that user, will it be available to the Codespace user or do we need to run the install as that user? You can see in the existing script where it does this so that the bash aliases it adds are added for the user.

So if someone installs this feature AND says to setup pip then I think we can safely just do that. As long as a global installation is likely to work then that is likely what we should do.

Sounds good!

When the install.sh script from this feature is executed when a Codespace is being built it is run as the root user. If we install this with that user, will it be available to the Codespace user or do we need to run the install as that user? You can see in the existing script where it does this so that the bash aliases it adds are added for the user.

If we install it as the root user, it will work system-wide. However, it's generally best practice to avoid installing pip packages as root, explained by this warning when running pip operations as root:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead.

There's a risk of conflict between our package's dependencies installed via pip and those installed by the system's package manager. For instance, the 'requests' package is available both as a Debian package and a pip package. If both are installed, pip could overwrite the one installed with apt (or vice versa).
So, I think it's best if we stick to running the install for the remote user only and leave the root environment alone (unless the consumer of the devcontainer feature explicitly specifies remoteUser or containerUser to be root). What do you think?

I think next steps are to get this merged. I would like to see a workflow added that builds the package and uploads it as an artifact to the build. I can help with the latter if you need it but having a workflow do the build would be best if you do.
Once we have this I will just manually add a Release with the build attached. Then I can start working on changing the feature to download and install from the release as the user.

I've added a CI workflow for linting, formatting, testing, etc for now. I'll work on the release workflow this week!

Okay done! I've added a release workflow in release-keyring.yaml. It will trigger when a tag matching feature_artifacts-helper_* is created, so it can build the package and upload it as an artifact.
This might be handy, here's the process I'm using to install and test it:
# Fetch artifact from GitHub releases
WHEEL_URL=$(
  curl -L \
        -H "Accept: application/vnd.github+json" \
        -H "X-GitHub-Api-Version: 2022-11-28" \
        https://api.github.com/repos/delilahw/codespace-features/releases/latest |
       jq -r 'first( .assets[] | select( .name | endswith("whl") ) ).browser_download_url'
)
wget -O package.whl "$WHEEL_URL"

# The wheel package is required to install wheels
python3 -m pip install wheel

# Install the package and its dependencies
python3 -m pip install package.whl

# Delete wheel file
rm package.whl

# Connect to Azure feed and download something
python3 -m pip download yapf -i "https://pkgs.dev.azure.com/<org_name>/_packaging/<feed_name>/pypi/simple"

I'm pretty happy with the changes now, so I'm taking the PR out of draft. Let me know if there's any more feedback y'all have, @markphip and team!
Also @embetten, who's maintaining artifacts-keyring
and artifacts-credprovider, has kindly offered her time as an additional reviewer for the keyring backend implementation. Could you please give it another glance? The only "real" changes I've made since our last discussion are additional tests added in b909288.
Thanks all 😇


Add method(s) on MotifCountsArray for compositional metrics

Similar to pairwise_distance, except on the MotifCountsArray class.
It should compute either Jensen-Shannon divergence, Jensen-Shannon metric, or an Aitchison distance. Of course, only the latter is truly a compositional measure but (at this point) I cannot think of a better name.

Fixed in 45e183eb8edde8a95c05091a8d5c9895f45d7ab7


Use SNAPSHOT artifacts from maven for dependencies

Coming from https://github.com/opensearch-project/opensearch-plugins/issues/45, use SNAPSHOT Maven dependencies from aws.oss.sonatype.org. See https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#building-within-the-opensearch-build-system for details.

this can be closed by https://github.com/opensearch-project/anomaly-detection/pull/286