Clustered Lighting

Description
System must take as input:

List of active light indices as SSBO
PRoperty Pool of Lights
ZBuffer or Hi-Z buffer (needs to be of the padded to PoT but not stretched variety) = need to merge @Przemog

Output must be:

Some sort of a data structure allowing us to retrieve light index lists for a given tile/cluster
Must be bounded in average number of lights to process per pixel

Example:

Sponza or some other scene lit with Forward+
between 10k and 4M lights (compensate amount with shorter range)

Testing:

Fast development, Vulkan advised
But final system must be able to run on GL and GLES on Intel HD Graphics

Light Struct
struct nbl_glsl_ext_ClusteredLighting_SpotLight
{
vec3 position;
float outerCosineOverCosineRange; `cos(outerHalfAngle)/(cos(innerHalfAngle)-cos(outerHalfAngle))`
uvec2 intensity; // rgb19e7 encoded
uvec2 direction; // xyz encoded as packSNorm2x16, w used for storing `cosineRange`
};

All lights shall be spotlights (if you want a point light, make it have an outer and inner cosine of -1).
Description of the related problem
Attenuation
Computed as
const float radiusSq = ...; // will be constant for all lights in the scene and settable globally via a define
const nbl_glsl_ext_ClusteredLighting_SpotLight light = ...;

const vec3 fromLight = wpos-light.position;
const float lenSq = dot(fromLight,fromLight);
const float attenuation = 0.5f*radiusSq*(1.f-inversesqrt(1.f+radiusSq/lenSq));

// at some point I'll encode the direction better, allowing me to squeeze a a windowing coefficient into 16 bits freed up
const vec2 dir0 = unpackSNorm2x16(light.direction[0]);
const vec2 dir1 = unpackSNorm2x16(light.direction[1]);
const vec3 spotDir = vec3(dir0.xy,dir1.x);
const float spotHalfAngle = dot(spotDir,fromLight)*inversesqrt(dot(spotDir,spotDir)*dot(fromLight,fromLight));

const float innerCosine= dir1.y;
const float spotPenumbra = clamp(spotHalfAngle*innerCosine-light.outerCosineOverCosineRange,0.f,1.f);

return attenuation*spotPenumbra;

as per https://www.youtube.com/watch?v=wzIcjzKQ2BE
Solution proposal
Shading budget
All we care about is keeping the amount of ALU ops and memory ops summed across all subgroups below a global threshold.
This translates to keeping the sum of light index lists lengths across all tiles/clusters below a soft (performance) as well as a hard (storage) threshold.
Use R32_UINT format for the cluster pointer texture and partition the bits between offset and count.
Tiling/Clustering granularity
In Deferred Shading, a whole workgroup (usually 16x16) maps to a screen tile, it is therefore extremely efficient to bin lights at the granularity of tiles and preload all the necessary lights into shared memory.
In Forward+ the story is similar, as primitives' fragments must be blended in draw order, which enforces an execution order at the macro scale, even on tilers. This means that subgroups will try to stick to the same triangle, or at least the same drawcall.
4-size aligned units are going to rasterize quads which themselves are aligned to even pixel coordinates.
We're mostly going to be targetting Forward+
This is why most fine tiles/clusters covering 16x16 pixel footprints make sense. You can however give a choice of 32x32 or even 64x64 tiles.
To promote stable cluster rejection of lights, do the clustering in world space (not frustum space)..
Light-Cluster Assignment
Main focus is stability for this reason, do not slice up the view frustum into clusters.
Instead create a 3D clipmap (nested grids) around the camera, this way if lights are dropped because of shading budget, the light popping will be stable.
If we make a 3D texture and atlas it with bricks of size KxKxK (suggestion, K=4), we can run compute dispatches to have 1 workgroup test 1 light against every single voxel (map invocation to voxel).
would be a good idea to warp the voxels logarithmically
linear = mix(a,b,percentage);
logarithmic = exp2(mix(log2(a),log2(b),percentage))

Then we can take the innermost 12.5% and subdivide them and repeat the process.
Initial assignment as a linked list could work nicely in order to keep worst case memory budgets down and avoid having to do two culling passes over the same list of lights and grid, interrupted by a prefix sum (do not want). However at the very end a compaction and linearization would be required (turn linked list into flatpacked arrays).
So instead of having each cluster traverse a light BVH, we have each light traverse a cluster BVH.
Light Rejection (Budgeting)
To achieve storage of light lists in one SSBO, while a 3D texture points to it with R32_UINT, we require a fundamental limit on the "length" of the light list of a cell. Lets call this limit L, we can set it even lower to improve performance.
A worst case scenario would be 4M lights, which would leave us with 10 bits to store the count of lights per cluster (L=1023).
If we know that a cell cannot store more than L light references, then we know that during the build (subdivision) process, no cell should have a linked list longer than L*2^subdivisionLevelsRemaining.
Also if we want to not blow the linked list reserved capacity (or overall end capacity) we should be probably keeping a minheap of all the lights.
Additional context
See:
https://github.com/buildaworldnet/IrrlichtBAW/issues/216
for previous ideas.
https://www.zora.uzh.ch/id/eprint/107598/1/a11-olsson.pdf
http://www.humus.name/Articles/PracticalClusteredShading.pdf

Spotlight Culling
Basically you need Cone and Sphere vs. AABB culling first. The cases split into 2
In both cases you first want to test the sphere.
Convex
After testing sphere, test that the cone is not fully outside the AABB.
Concave
When the spotlight's half angle > 90 degrees it flips inside out, then you need to test that the AABB is not fully inside the cone.
https://www.zora.uzh.ch/id/eprint/107598/1/a11-olsson.pdf

https://www.intel.com/content/www/us/en/developer/articles/technical/forward-clustered-shading.html

https://www.jonmanatee.com/blog/2015/1/20/improved-spotlight-culling-for-clustered-forward-shading.html

Old idea for constructing the Light BVH
start by dispatching a NumLights invocations, then test each light against 2x2 frustums (each with its own custom far value), add the light to a per-frustum bucket/list
we can duplicate each light reference 0 to 4 times
we can also keep track of how many light references overall we have, and do an indirect dispatch
if we tag each light reference with the frustum (i,j) coord, we can process all the light references for a lower level of the hierarchy in a single dispatch
and we can rinse and repeat until we hit the lowest level
and this should be pretty fast
we could actually tag the light references with (i,j,level) [unique Tile ID]
and they could go onto a append buffer and no special bucketing or linked listing necessary
then after we're all done, we can sort the light references by TileID, prefix sum and write the (offset,lightcount) into a texture that matches the tile hierachy
then we have what we want, a contiguous list of lights thats easy to iterate in Forward+ and easy to coopeartively load into shared mem in Deferred/VisBuffer

Newer idea for Light AS
We frustum cull and then sort the lights by importance, scratch memory needed to do this is O(numLights)
we then take the first min(KTileCount,LightCount) most important lights
then we subdivide the frustum into 2x2 (or 4x4 or 8x8) sub frustums and output uint4/uint16/uint64 per light, with each bit enabled/disabled depending on if the light intersects the subfrustum
then we Scan (prefix sum courtesy of @achal ) these per-sub-frustum bits to obtain an offset table and we scatter the lightID[i] into memory location subFrustumIDKsubFrustumTileCount+prefixSum[i] but only if prefixSum[i]<KsubFrustumTileCount
this lets us fill the per-frustum lightID buckets with the lights that intersect it, but in a stable ordering (most important first!)
the only question that remains now, given that we'll use this with Forward+
is the choice of the frustum hierarchy shape

How to produce a compact list with a fixed size memory pool and not having to recull the lights
Use the pool to give you linked list nodes (or segmented array).
In your grid cells keep an offset to forward linked list head, as well as a counter of total nodes in the list.
Exclusive Prefix Sum the cell's light counts, these are now your light list offsets.
Go through the original linked list and rewrite it linearly to the buckets.
P.S. You can use atomicCompSwap on the head pointer and late-record the next offset into your newly allocated linked list node, to allow for multiple invocations to append nodes to the same cell's linked list (probably suboptimal though).

3 ways of Light to Cluster Assignment
There's two ways of doing a traversal.
Local (stack based)
We assign one invocation to one testee, then traverse all tree nodes to test against.
Global (partition based)
We start with a list of testees and a tree node, 1 invocation per testee.
We filter this list into a separate list for each of the node's children.
Repeat the process. Multiple nodes not sharing subtrees can be processed in parallel.
Light BVH
BVH of Lights is constructed from AABBs (preferably BVH8 or 16).
An invocation is fired out for each cluster and we traverse the BVH to find lights which intersect the cluster AABB.
The intersection is recorded according to previous post.
Because lights are written to Clusters, its best to traverse locally via stacks (avoid atomics and multiple invocations writing the same cluster).
Cluster BVH (Preferred)
Really an octree or a clipmap/cascade in our case of world space clustering.
Flat list of lights is taken and pruned & partitioned amongst the clusters.
The intersection is recorded according to previous post.
Because lights are written to Clusters, its best to traverse globally via partitioning (avoid atomics and multiple invocations writing the same cluster).
Assigning invocations to lights and traversing cluster BVH would be a nightmare of work balancing and atomic updates
I prefer this option because lights are very likely to move and need the BVH updating frequently, whereas worldspace cluster BVH just follows camera position, and keeps a constant topology.
Light and Cluster BVH
Light BVH is constructed as in the first option.
Then we traverse both trees at the same time.
Instead of a cluster tree node containing a direct list of light IDs, it would contain a list of light BVH nodes which are roots of disjoint branches.
Every traversal step, you either split the light BVH (turn small list into bigger list while retesting) or split the cluster BVH (partition the list while culling against smaller nodes). In both cases you'd map invocation to light BVH node ID.
A good heuristic would be to split whichever BVH node is larger.


#15698 Improve support for structs

Now structs are also compatible with the output of psql.
Several test cases:
create type pair as (a int[], b int[]);
select array[row(array[1,2,3], array[4,5,6])::pair, row(array[1,2,3], NULL)::pair]::pair[];

create type a as (a text);
select row(null)::a, row('')::a, row('"')::a, row('()')::a;

Pretty edge case, but this for example will give incorrect result
create type aa as (a text, b text);

select array[row(null, null)::aa, null];

@Serge Do you think if this should be fixed, or works as expected?

Verified
Opened a different ticket for the edge case #16822


Changing 'merge' Tasks to be 'resolved'

This is concerning the request in the issue tracker https://storyboard.openstack.org/#!/story/2001432 .

Thank you for contributing to openstack-infra/storyboard-webclient!
openstack-infra/storyboard-webclient uses Gerrit for code review.
If you have never contributed to OpenStack before make sure you have read the
getting started documentation:
http://docs.openstack.org/infra/manual/developers.html#getting-started
Otherwise please visit
http://docs.openstack.org/infra/manual/developers.html#development-workflow
and follow the instructions there to upload your change to Gerrit.


Spec for #line_labels_for behavior

There's a new cool feature merged into master in #418: you can opt to add labels onto the GitHub issues lines of the changelog.
That behavior (the creation of the line) lacks an RSpec test.
Here is the (private) method that has the behavior to test.
https://github.com/skywinder/github-changelog-generator/blob/master/lib/github_changelog_generator/generator/generator_generation.rb#L154

Related #443


Allow searching for specific speakers

As the system gets more and more users it can become a bit annoying to find a specific speaker to either promote them to admin or check out their talks.
Right now it just searches through their first and last name, i'm not sure if it should be expanded to also include things like their bio/email/twitter etc.

I have fixed the Yoda conditionals. I'm not entirely sure how to go about testing this though.

@BackEndTea If I were writing a test to go with this code, I think it would be something like this:

create a DB table with the list of speakers in it
add the 'search' variable to the request
execute that controller method (should be lots of examples of doing this in our code base)
verify in the output that you see only the names of the speakers you are expecting

I'll wait for #501 since that makes some changes to the database tests before adding the tests. So i'm closing this for now

@chartjes I changed it around a little bit and moved this functionality to the User Mapper. It makes more sense for it to be there.
Plus i also wrote the unit tests for it.

@BackEndTea I can help resolve this, if you need. I just did you a dirty and sent a pretty rediculously-large PR.

@mdwheele I don't mind fixing it myself, but i probably won't get around to that until monday

Thanks for making the changes @BackEndTea


Desktop - When uploading a receipt via the request money flow, it shows as Pending

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?: y
Reproducible in production?: n
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @mallenexpensify
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1697243512619269
Action Performed:

Next to the compose box, click the + action button and request money
Upload receipt as part of the Scan flow
Notice Pending shows at top, during SmartScan process and after it's done SmartScanning

Expected Result:
Pending to not show
Actual Result:
When uploading a receipt via the request money flow, it shows as Pending
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

https://github.com/Expensify/App/assets/93399543/e12f5803-3764-4ab4-858d-d6731ad9d7dc

View all open jobs on GitHub

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

It seems to be a normal behavior, the pending will disappear as stated in the message. this new behavior was added here
https://github.com/Expensify/App/pull/29201

also fixed
Payment for c+ handled here https://github.com/Expensify/App/issues/29626 closing


Examples should not have both JSON and YML

This will just cause duplication of effort.
I suggest JSON only since that will increase adoption.
i.e. ActiveStandard/callExample.json and ActiveStandard/callExample.yml

We agreed in the meeting on 2019-02-19 to abandon YAML. We will use JSON Schema to document the specification. Closing this issue as resolved.


bugfix: LiteralSubject match function reports incorrect mismatch

The LiteralSubject match logic in match.go gives us false-positive detected changes.
This happens when we are working with complex subject values (using the hex encoded DN values with =#).
The go-representation of the AttributeTypeAndValue is different for the generated and parsed values, while the binary representation is identical for both.
This issue has appeared after we fixed support for these more complex DN values in https://github.com/cert-manager/cert-manager/pull/6761.
Manual test:
Apply the following resource before and after this PR:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
spec:
  secretName: example-com-tls

  literalSubject: ST=example,C=US,O=#04024869
  dnsNames:
    - example.com

  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

Before: controller keeps creating CertificateRequests thinking the created CertificateRequest does not match the Certificate resource.
After: controller creates the CertificateRequest once and stops (still returns an error since it does not support the provided #04024869 value)
Kind
/kind bug
Release Note
NONE

/retest
flake: I think node must have been shutdown


stealing focus?

I have not been able to determine it with certainty yet, but I am getting the idea that the CodeGlance plugin steals focus when opening PHPStorm or even a document. Could that be true? Any thoughts on this?

what is the stealing focus mean?

Oh sorry, I mean I open PHPStorm, then go to another window (browser, file manager, whatever), and PHPStorm is brought back into focus, hiding what I was doing in the other window and even taking keyboard entries. It sometimes happens too when I open a file and switch to another program. Sometimes the IDE is so busy (indexing etc.) it just takes a while.

CodeGlance will not stealing focus.and i use idea,didn't meet this situation


Move access-path filtering into ModuleNameLookup (out of lookupValue)

Removes duplicated logic from the implementations of FileUnit::lookupValue, and simplifies the interface to ModuleDecl::lookupValue, where everyone was passing an empty (non-filtering) access path anyway except during actual lookup from source code. No functionality change.

apple/swift-lldb#1978
@swift-ci Please smoke test


Custom resource packs

Implements custom resource pack support.
Adds a Client method to set resource pack, and a ClientEvent for listening to resource pack status updates from clients.

I updated the conway.rs example to show a working custom resource pack, for demonstration's sake. Let me know if you'd like this taken out of the example or for me to make a new example instead.
I've also noticed that we cannot set a client's resource pack immediately after they are created (in the same tick). I'd assume that deferring the call until the next tick would solve this issue, but am unsure if that is possible at the moment (something like Bukkit's BukkitRunnable.runTask())

As @dyc3 said, resource pack tests should go in a separate example.
You can use the mediafire link for now, but you should add the resource pack to assets/. In a separate commit we will change the URL.

I'll just leave the URL as localhost in the new example, and open another PR to change it when the pack.zip has been merged into main.

What happens if the client receives the resource pack packet multiple times in the same tick? Does the new resource pack prompt immediately override the old prompt? In that case I think it might be better to have an Option<ResourcePackS2c> on Client rather than a Vec<ResourcePackS2c> if there is no observable difference.

Yeah, the new resource pack prompt overrides the old one on the client immediately, and the client sends back a "Failed" C2s for the old prompt, which means that Option<ResourcePackS2c> is probably the way to go.


Your example from the docs doesn't work

Error: The property "name" is required on a content type.
Error: The property "name" is required on the field "name".
For the code from your docs:
const dog = migration.createContentType('dog');
const name = dog.createField('name');
name.type('Symbol').required(true);

Please fix it

This issue had no activity for some time now, please feel free to create a new one if you still have the issue


Relative dependencies not found

I have a nested project with a main module Foo and an example directory that makes use of that module. The example directory has its own Project.toml and Manifest.toml which is created by running ]dev .. from withthin example/.
That is, the project has the following layout:
❯ tree
.
├── examples
│   ├── main.jl
│   ├── Manifest.toml
│   └── Project.toml
├── Manifest.toml
├── Project.toml
└── src
    └── Foo.jl

Content of src/Foo.jl:
module Foo.jl
export foo

foo() = "foo"
end

Content of examples/main.jl
using Foo: foo

print(foo())

When editing main.jl (even with nvim launched in examples), coc-nvim fails to find Foo; that is, I'm getting a missing reference. Accordingly, auto-completion and "go-to-definition" features fail.
In contrast, when I open examples/main.jl in VSCode, Foo is found and I can also jump to the source code.
I suspect that this has something to do with the way that coc-julia detects the package root. It seems that coc-julia is using the top-level {Project,Manifest}.toml here.

Interesting data-point: I can trick coc-julia if I touch .git in example/. I guess julia-vscode just considers the .git directory as the project root?

A somewhat clean fix would be o just set the coc rootPatterns as documented here. For now, I just added autocmd FileType julia let b:coc_root_patterns = ["Project.toml", ".git"] to my vimrc. We should either document this somewhere or see if we can set the variable in the extension somehow.


Bumping up react-loader version to 2.4.5 for react 16 support

Done...

Coverage remained the same at 100.0% when pulling 96a0144f0d556a834b7c8f9b3804b0aeada3dc40 on Alameli:react-loader-version-bump into c08c50a30117492fa9d4bc72d262dc8379ae7546 on yury-dymov:master.

Thank you :)


Error running mrtrix.TrcGen on my machine - ?memory issue

Hello
I was wondering if someone could help me. I've tried using qsiprep for the first time and was doing a test run. I've successfully managed to preprocess a scan but during the recon phase I get the following error (see below). I'm running it on Ubuntu 20.04 and have about 12GB free of memory. I have googled the error and noticed from other issues from fmriprep that this may potentially be an issue with memory? If so, do you know any way to get around this on my local machine?
Cheers
Meher

[Node] Running "tractography" ("qsiprep.interfaces.mrtrix.TckGen"), a CommandLine Interface with command:
tckgen -act /tmp/work/qsirecon_wf/sub-A00008326_mrtrix_singleshell_ss3t/anat_ingress_wf/create_5tt/sub-A00008326_desc-preproc_T1w_5tt.mif -algorithm iFOD2 -backtrack -crop_at_gmwmi -maxlength 250.000000 -minlength 30.000000 -samples 4 -nthreads 7 -output_seeds out_seeds.nii.gz -power 0.330000 -seed_dynamic /tmp/work/qsirecon_wf/sub-A00008326_mrtrix_singleshell_ss3t/recon_wf/ss3t_csd/_dwi_file_..qsiprep-output..sub-A00008326..ses-ALGA..dwi..sub-A00008326_ses-ALGA_space-T1w_desc-preproc_dwi.nii.gz/intensity_norm/sub-A00008326_ses-ALGA_space-T1w_desc-preproc_dwi_wm_mtnorm.mif -select 10000000 /tmp/work/qsirecon_wf/sub-A00008326_mrtrix_singleshell_ss3t/recon_wf/ss3t_csd/_dwi_file_..qsiprep-output..sub-A00008326..ses-ALGA..dwi..sub-A00008326_ses-ALGA_space-T1w_desc-preproc_dwi.nii.gz/intensity_norm/sub-A00008326_ses-ALGA_space-T1w_desc-preproc_dwi_wm_mtnorm.mif tracked.tck
210728-13:38:55,723 nipype.workflow INFO:
	 b''
210728-13:38:55,723 nipype.workflow INFO:
	 b''
exception calling callback for <Future at 0x7fdb2e849ef0 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fdb2e849828 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

It's possible this is a memory issue. Have you tried setting --nthreads 1? you may also need to edit the pipeline json file to create fewer streamlines

Thanks so much for replying Matt! It didn't work. I ended up running the modules individually. I was just testing qsiprep out and hopefully when I get a pipeline on AWS working I can use it properly.
Really appreciate your time to respond to my query!


build: remove CFLAGS from add_swift_library

This is in preparation for migrating to Swift support in CMake.  That
will allow us to use exported targets to track the dependencies properly
across projects.

@swift-ci please test


Default to itimer profiler when perf profiler isn't available

Today, it defaults to wall time profiler since d78e0b3a517593ced9af0fbaa073105f379c228d. This commit doesn't make it clear why that change of behavior, especially on Linux.
It is more useful to switch to itimer as it's behavior is closer to perf sampling interval.

Hi @luhenry,
What is the motivation to switch to itimer? In my observation, it's the other way round: wall engine gives more accurate profile (which is closer to perf_events) comparing to itimer. Distribution of itimer signals isn't always fair between threads, especially on macOS, where they seem highly biased towards system calls (#184). On Linux, itimer is not that bad, but still produces skewed results sometimes (#272).
Despite its name, wall engine is not only for wall-clock profiling. It can also act as a CPU profiler by choosing those samples that hit runnable threads only.

perf is overall a CPU profiler, aka it's sampling what happens when on CPU. itimer (as used in async-profiler) is similar since it's also using a CPU clock and will trigger a signal once X ms has been spent executing on CPU.
wall clock however is not a CPU profiler in the same way. It will only sample threads which are currently running on CPU, but it won't sample every X ms of _on-CPU execution time but every X ms of wall time.
That difference is what concerns me, and which could potentially introduce bias in what we end up measuring. For example, what happens if a method ends up executing every 10 ms, while the sampling interval is also every 10 ms. If this method executes for 1 ms, then itimer and perf will get it right and only sample every 10 executions, but what's the guarantee wall will not sample every invocation of the method or, even worst, never sample any invocation of the method because it's just out of sync. Is it something you've observed in practice?

OK, I understood your concern. It makes sense, and I agree that the scenario you describe is theoretically possible. If itimer signals were fair, making this a default engine would be a good choice. The problem is, they are not fair at all (#184, #272). So changing the default engine to itimer does not necessary make results better. On macOS, itimer is almost always worse. On Linux, it depends. Furthermore, ITIMER_PROF is not always available, e.g. on WSL 1. Therefore I'm not convinced that we should prefer itimer to wall/cpu.


ENH: add a "name" field to DataFrame, consistent with Series

It would be logical to add a name field to DataFrame, similar to Series. Then when you construct a Panel it can try to use the name, in the same way as when you construct a DataFrame from Series.
It would also be useful when loading/saving DataFrames to files and DBs, for example at the moment to save to HDF you always have to add a key and for SQL a table name, it would be logical to attach these to the DataFrame itself as its name parameter.

dupe of #447
long discussed. It IS possible, just would have to be done carefully.
The primary argument against have a DataFrame having named attributes that propogate is kind of moot for this simple type of attribute, which could be propogated similarly to the Series.name attribute.


Load performance

Add load_performance method (similar to  load_score)

Codecov Report

Merging #96 (373fb1e) into develop (fb9e76f) will increase coverage by 0.02%.
The diff coverage is 81.57%.

@@             Coverage Diff             @@
##           develop      #96      +/-   ##
===========================================
+ Coverage    77.33%   77.36%   +0.02%
===========================================
  Files           48       49       +1
  Lines         8317     8353      +36
===========================================
+ Hits          6432     6462      +30
- Misses        1885     1891       +6

Impacted Files
Coverage Δ

partitura/io/importmidi.py
83.39% <ø> (ø)

partitura/io/__init__.py
64.17% <77.27%> (+5.48%)
:arrow_up:

tests/test_load_performance.py
86.66% <86.66%> (ø)

partitura/__init__.py
100.00% <100.00%> (ø)

partitura/io/importmatch.py
73.58% <0.00%> (+0.12%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb9e76f...373fb1e. Read the comment docs.


Add mysql creds propagation

Description

Addresses https://github.com/youtube/vitess/issues/3044.
The core of the changes are located in pool.go and dbconn.go

Tests

Unit and integration
Local cluster

I just glanced over this. Will get back in again later for a more detailed review.
What I had in mind was a dirtier implementation where the dbconnpool API wouldn't have to change. You define this additional user info as global flags in dbconnpool, and if it matched, it would just use the info to create the connection using the global info.
The advantage of the 'dirty' approach is that the changes will be minimal and will be easier to undo in the future. Your current approach is cleaner.
I'm not fully convinced that we'll undo this soon. So, I'm on the fence.

Yeah - I was thinking about that too. I was debating myself around adding the flags there as you mentioned in the issue, but found this more consistent with how everything is done.
I don't have strong opinions about it. Happy to update the PR and set the dirty approach if you think that will make it easier for us in the future.

@sougou thanks for the feedback. Updated!

LGTM


chore(deps): update dependency commons-codec:commons-codec to 1.12

This PR contains the following updates:

Package
Change

commons-codec:commons-codec
1.11 -> 1.12

Renovate configuration
:date: Schedule: At any time (no schedule defined).
:vertical_traffic_light: Automerge: Disabled by config. Please merge this manually once you are satisfied.
:recycle: Rebasing: Whenever PR is stale, or if you modify the PR title to begin with "rebase!".
:no_bell: Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

              Apache Commons Codec 1.12 RELEASE NOTES

The Apache Commons Codec team is pleased to announce the commons-codec-1.12 release!

The Apache Commons Codec package contains simple encoder and decoders for
     various formats such as Base64 and Hexadecimal.  In addition to these
     widely used encoders and decoders, the codec package also maintains a
     collection of phonetic encoding utilities.

Changes in this version include:

New features:
o Add Percent-Encoding Codec (described in RFC3986 and RFC7578)  Issue: CODEC-240. Thanks to Ioannis Sermetziadis.
o Add SHA-3 methods in DigestUtils  Issue: CODEC-251. Thanks to Gary Gregory.

Fixed Bugs:
o B64 salt generator: Random -> ThreadLocalRandom  Issue: CODEC-252.
o Wrong value calculated by Cologne Phonetic if a special character is placed between equal letters  Issue: CODEC-250. Thanks to Alex Volodko.
o ColognePhoneticTest.testIsEncodeEquals missing assertions  Issue: CODEC-246. Thanks to Oscar Luis Vera Pérez.

Changes:
o Update from Java 7 to Java 8  Issue: CODEC-253.

Have fun!
-Apache Commons Codec team


[WIP] Physics tests: Bullet

Motivation and Context
Test installation of Bullet physics and habitat-sim with Bullet physics in circleCI.
Test python exposed dynamics with various physics implementations (only Bullet currently).
How Has This Been Tested
Local (MacOS) and devfair (Ubuntu) pytest.
Types of changes

[ ] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] Testing

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

I'd like to get this test (especially the build part) into master before the release. I'll write a separate PR to add assertions and additional tests once we have force/torque caching and more interesting time-stepping schemes.


Step 1 awk invalid char in expression

Hello, I'm trying to assemble some reads with minimap and miniasm. Minimap seems to have worked, but my process is failing on miniasm:
[M::mm_idx_gen::0.010*2.97] collected minimizers
[M::mm_idx_gen::0.022*2.77] sorted minimizers
[M::main::0.022*2.77] loaded/built the index for 25 target sequence(s)
[M::main] max occurrences of a minimizer to consider: 7
[M::main] Version: 0.2-r124-dirty
[M::main] CMD: minimap -Sw5 -L100 -m0 Gap_Support/Contig17_1_1011970_pilon|arrow.gap.3/reads.fa Gap_Support/Contig17_1_1011970_pilon|arrow.gap.3/reads.fa
[M::main] Real time: 0.035 sec; CPU: 0.082 sec
[M::main] ===> Step 1: reading read mappings <===
awk: '/^S/{print(">"$2"
awk: ^ invalid char ''' in expression

Any pointers on how to get around this would be appreciated. Could it have something to do with the headers in my fasta?

Sorry, I think the awk error is actually coming from my own code, as I've got an awk wrapper to extract Fasta sequences from the GFA output of Miniasm. However, I'm still not sure if Miniasm is actually working? None of the reads were assembled:
Gap_Support/Contig498_pilon|arrow.gap.11:
total 512
-rw-rw-r-- 1 dbrowne dbrowne     0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne     0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne 39826 May 12 00:06 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne   372 May 12 00:06 reads.paf.gz

Gap_Support/Contig498_pilon|arrow.gap.3:
total 18432
-rw-rw-r-- 1 dbrowne dbrowne        0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne        0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne  6323839 May 12 00:04 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne 11544864 May 12 00:06 reads.paf.gz

Gap_Support/Contig498_pilon|arrow.gap.5:
total 512
-rw-rw-r-- 1 dbrowne dbrowne     0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne     0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne 16932 May 12 00:04 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne    20 May 12 00:06 reads.paf.gz

Gap_Support/Contig498_pilon|arrow.gap.6:
total 512
-rw-rw-r-- 1 dbrowne dbrowne    0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne    0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne 6864 May 12 00:04 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne  402 May 12 00:06 reads.paf.gz

Gap_Support/Contig498_pilon|arrow.gap.7:
total 1024
-rw-rw-r-- 1 dbrowne dbrowne      0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne      0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne 393547 May 12 00:05 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne  48777 May 12 00:06 reads.paf.gz

Gap_Support/Contig498_pilon|arrow.gap.9:
total 512
-rw-rw-r-- 1 dbrowne dbrowne    0 May 12 00:06 assembly.fa
-rw-rw-r-- 1 dbrowne dbrowne    0 May 12 00:06 assembly.gfa
-rw-rw-r-- 1 dbrowne dbrowne 4521 May 12 00:05 reads.fa
-rw-rw-r-- 1 dbrowne dbrowne  229 May 12 00:06 reads.paf.gz

Moreover, I never see anything in my output past "Step 1" from Miniasm:
[dbrowne@ada3 2017.05.09_Jelly2_Testing]$ head -n 30 OUT_JELLY2_TESTING_v22
[M::mm_idx_gen::0.010*2.97] collected minimizers
[M::mm_idx_gen::0.022*2.77] sorted minimizers
[M::main::0.022*2.77] loaded/built the index for 25 target sequence(s)
[M::main] max occurrences of a minimizer to consider: 7
[M::main] Version: 0.2-r124-dirty
[M::main] CMD: minimap -Sw5 -L100 -m0 Gap_Support/Contig17_1_1011970_pilon|arrow.gap.3/reads.fa Gap_Support/Contig17_1_1011970_pilon|arrow.gap.3/reads.fa
[M::main] Real time: 0.035 sec; CPU: 0.082 sec
[M::main] ===> Step 1: reading read mappings <===
awk: '/^S/{print(">"$2"
awk: ^ invalid char ''' in expression
[M::mm_idx_gen::0.003*6.60] collected minimizers
[M::mm_idx_gen::0.007*4.68] sorted minimizers
[M::main::0.007*4.67] loaded/built the index for 7 target sequence(s)
[M::main] max occurrences of a minimizer to consider: 7
[M::main] Version: 0.2-r124-dirty
[M::main] CMD: minimap -Sw5 -L100 -m0 Gap_Support/Contig17_1_1011970_pilon|arrow.gap.6/reads.fa Gap_Support/Contig17_1_1011970_pilon|arrow.gap.6/reads.fa
[M::main] Real time: 0.012 sec; CPU: 0.039 sec
[M::main] ===> Step 1: reading read mappings <===
awk: '/^S/{print(">"$2"
awk: ^ invalid char ''' in expression
[M::mm_idx_gen::0.003*8.09] collected minimizers
[M::mm_idx_gen::0.006*5.31] sorted minimizers
[M::main::0.006*5.29] loaded/built the index for 13 target sequence(s)
[M::main] max occurrences of a minimizer to consider: 9
[M::main] Version: 0.2-r124-dirty
[M::main] CMD: minimap -Sw5 -L100 -m0 Gap_Support/Contig26_pilon|arrow.gap.12/reads.fa Gap_Support/Contig26_pilon|arrow.gap.12/reads.fa
[M::main] Real time: 0.010 sec; CPU: 0.036 sec
[M::main] ===> Step 1: reading read mappings <===
awk: '/^S/{print(">"$2"
awk: ^ invalid char ''' in expression

I suppose it's possible that simply none of the sequences were able to be assembled by Miniasm, but that seems unlikely.

Hi,
Is it possible that the pipes in your file names are causing the trouble? Otherwise, I've found setting -f0.0 to work around gaps, where (simple) repeats may be more common.


Using HTML tags in strings inside mustache braces

For some reason using HTML tags in strings inside triple mustache doesn’t work as expected.
Example:
{{{ item.price_for == 'item' ? 'items' : 'm2' }}}

In this case it will output to HTML everything including triple braces.
P.S. For double mustache it has the same issue.

Note Vue templates are parsed as actual HTML. The  tag in your template is parsed as an actual tag and breaks up your expression. As a rule of thumb, you should never include tags inside mustache expressions, because Vue is not a string-based template language. Use either a filter or a computed property for this case.

@yyx990803  小右哥，请教一个问题：我想在数据库的某个字段中插入一个html tag，像这样:"今日流量: 888"，然后这个字段在页面里通过vue渲染出来的，但是在页面里并不会像我期待的那样换行，而是原封不动的渲染出来字符串。我该怎么做呢？


Dynamic routing..

Would there be an interest in extending the product to support dynamic routing?
What I mean is that the routing table is external and dynamic to the running proxy application, and it periodically refreshes its internal lookup table without a restart.
In the NGINX world, this would require a new config and a SIGHUP.
We have a scenario where our routing data is actually in REDIS and used as a shared cache.
We'd like to run multiple copies of rust-rpxy as a proxy array to a set of origin servers.  Each origin server has a specific piece of content, but this is not known beforehand.
The content is placed on the origin ad-hoc, and then a reference to the content pointing to the origin IP is stored in Redis.
So if I'm looking for content of /content/what, and "what" was uploaded into origin <IP_ADDRESS>, then our redis would have some lookup table where "content/what" maps to "<IP_ADDRESS>".
This redis content is updating constantly, but you would only need to query it periodically to insure the routing is current.
We'd like to use rpxy in some way to load this dynamic lookup table so as requests come in, it knows the proper server IP to proxy to in order to serve the request.
We could build an application to query our Redis server and write a new .toml config file for each rpxy we have running, but this becomes cumbersome, and we'd still need the ability to update the config with a sighup to avoid downtime.
What would be better is if it could query the redis server directly, then we only have to configure the rpxy to pull it's configuration (which is the same for every instance) from a central place.
pull instead of push.
Let me know your thoughts.

@fairbairn
Hello! Thanks for your feature request and comment.
First of all, I think 'hot-loading'/'dynamic-loading' of routing configuration is technically possible. Indeed, certificates can be hot-loaded in rust-rpxy, and routing information can be served similarly to that.
I understand your situation. Honestly, it is quite tough to directly support Redis in rust-rpxy from the viewpoint of maintenance, and such feature a bit out-of-concept of rust-rpxy. In my opinion, rust-rpxy should be simple as possible. But I believe hot-loading feature of configuration (maybe simply from file or via RestAPI?) is nice and worth to consider for future implementation.


Get auth fails on single and double JWT when not logged in

Hi, running some tests with a postman test runner I found a bug in the GET::/api/access endpoint when the user is not logged in or hasn't requested an access token(double JWT), according to documentation in those cases the response should be:
{         "id": 40,         "name": "guest",         "loggedIn": false, }
what im getting instead is
{     "errorcode": "access_token_invalid",     "error": "Access Token invalid." }
Link to the test runner
https://www.postman.com/elbedroom/workspace/appapi/collection/10237763-1f332748-23b5-4077-8dd3-64bb3fce4e9e?action=share&creator=10237763

Sorry just found it was the postman runner that was resolving empty collection variables as {{refresh_token}}, thus sending Auth headers like 'Bearer {{refresh_token}}' and causing the error, tested without those malformed headers and it works fins


[cluster_test] Introduce event! macro for structured events

Instead of using kinesis for receiving events, this diff proposes alternative:
In short, this diff introduce event! macro that can be used in code to emit json-structured events.
Example (from event_processor.rs):
event!("committed", "block": committed_block.id());

Events can be fetched by using GetEvents method on debug interface.
Only last 1000 events are persisted, so node is not going to OOM if no-one collect events.
Most recent events are persisted and old events are purged when more then 1000 events are 'stuck' in queue,
circular buffer is used to achieve this.

Add timestamp to log entry
Add HashValue::short_str() and use it to properly format commit json
Add TODOs


boards: arm: stm32f746g_disco: Use specified USB serial

The 'west flash' command allows specifying the port where the target is attached via the '--serial' option, allowing users to set the USB serial port for flashing. However, the 'stm32f746g_disco' script file currently ignores the _ZEPHYR_BOARD_SERIAL variable set by this option, preventing effective port specification.
This commit fixes it by correctly setting the openocd adapter serial when _ZEPHYR_BOARD_SERIAL variable is set, enabling proper USB serial port specification during flashing.
For example, if one builds the hello_world sample for the board in question with:
$ west build -p always -b stm32f746g_disco samples/hello_world
and have more than one USB serial port available:
gromero@gromero6:~/zephyrproject/zephyr$ pyocd list
  #   Probe/Board      Unique ID                  Target
-------------------------------------------------------------------
  0   STLINK-V3        000C00284741500320383733   ✖ stm32u585aiix
      B-U585I-IOT02A

  1   STM32 STLink     066CFF485153826687133653   n/a
gromero@gromero6:~/zephyrproject/zephyr$

and tries to flash to the specific target attached to port #1 by using the Unique ID / serial, it fails:
gromero@gromero6:~/zephyrproject/zephyr$ west flash --serial 066CFF485153826687133653
-- west flash: rebuilding
ninja: no work to do.
-- west flash: using runner openocd
-- runners.openocd: Flashing file: /home/gromero/zephyrproject/zephyr/build/hello_world/zephyr/zephyr.hex
Open On-Chip Debugger 0.11.0+dev-00725-gc5c47943d (2023-03-14-22:08)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
066CFF485153826687133653
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
Info : DEPRECATED target event trace-config; use TPIU events {pre,post}-{enable,disable}
Info : clock speed 2000 kHz
Info : STLINK V3J8M3 (API v3) VID:PID 0483:374E
Info : Target voltage: 3.299800
Warn : UNEXPECTED idcode: 0x0be12477
Error: expected 1 of 1: 0x5ba02477

FATAL ERROR: command exited with status 1: /home/gromero/zephyr-sdk-0.16.0/sysroots/x86_64-pokysdk-linux/usr/bin/openocd -s /home/gromero/zephyrproject/zephyr/boards/arm/stm32f746g_disco/support -s /home/gromero/zephyr-sdk-0.16.0/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts '-c set _ZEPHYR_BOARD_SERIAL 066CFF485153826687133653' -f /home/gromero/zephyrproject/zephyr/boards/arm/stm32f746g_disco/support/openocd.cfg '-c init' '-c targets' -c 'reset init' -c 'flash write_image erase /home/gromero/zephyrproject/zephyr/build/hello_world/zephyr/zephyr.hex' -c 'reset run' -c shutdown
gromero@gromero6:~/zephyrproject/zephyr$

With this patch applied the result is:
gromero@gromero6:~/zephyrproject/zephyr$ west flash --serial 066CFF485153826687133653
-- west flash: rebuilding
ninja: no work to do.
-- west flash: using runner openocd
-- runners.openocd: Flashing file: /home/gromero/zephyrproject/zephyr/build/hello_world/zephyr/zephyr.hex
Open On-Chip Debugger 0.11.0+dev-00725-gc5c47943d (2023-03-14-22:08)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
066CFF485153826687133653
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
Info : DEPRECATED target event trace-config; use TPIU events {pre,post}-{enable,disable}
Info : clock speed 2000 kHz
Info : STLINK V2J25M14 (API v2) VID:PID 0483:374B
Info : Target voltage: 3.238421
Info : [stm32f7x.cpu] Cortex-M7 r0p1 processor detected
Warn : [stm32f7x.cpu] Silicon bug: single stepping may enter pending exception handler!
Info : [stm32f7x.cpu] target has 8 breakpoints, 4 watchpoints
Info : starting gdb server for stm32f7x.cpu on 3333
Info : Listening on port 3333 for gdb connections
    TargetName         Type       Endian TapName            State
--  ------------------ ---------- ------ ------------------ ------------
 0* stm32f7x.cpu       hla_target little stm32f7x.cpu       running

target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x080011e0 msp: 0x20011100
Info : device id = 0x10016449
Info : flash size = 1024 kbytes
auto erase enabled
wrote 32768 bytes from file /home/gromero/zephyrproject/zephyr/build/hello_world/zephyr/zephyr.hex in 1.017129s (31.461 KiB/s)

shutdown command invoked
gromero@gromero6:~/zephyrproject/zephyr$

Thanks @gromero that's looks great, but is there any way to make it a little bit more generic ? From what I see it would have to be added in each openocd.cfg file

Yeah, to be honest, I was wondering the same thing. I need to investigate if there is another more generic way to solve it. I think patch is not wrong in the sense that other boards are using the same approach but I agree it's annoying to have to add this check per board script.
Before I investigate it deeper, is this (--serial option) the correct way to handle flashing and debugging when multiple targets are attached to a host? I vaguely recall you saying you have multiple targets attached to your dev host.

Before I investigate it deeper, is this (--serial option) the correct way to handle flashing and debugging when multiple targets are attached to a host? I vaguely recall you saying you have multiple targets attached to your dev host.

In the configuration where I need to play with several boards on the same host, I'm using twister. This is not much usable for pure development purpose as with west. I guess this --serial option is the right option in west context

@FRASTM Hi. Mind giving a second review for this simple change, please? :-)


Differentiate between an object's own BlendMode and its inherited BlendMode, so if a parent's blendmode is updated multiple times, each update is passed to the children instead of only the first one.
Prioritize the inherited BlendMode over the object's own BlendMode for closer resemblance to flash.

Thank you
I have updated things to something to similar to your approach, but I have fixed the issue where an object's blendMode property returns a value different than what a user set.
Here's the commit:
https://github.com/openfl/openfl/commit/0e321560a6b3095fc873ee0b1b0b4725756e5ad2
This doesn't fully resolve multiple layers of blending, but should (I think) help make things work better for you :smile:

I don't expect it to be perfect, just wanted something better than what was there. Thanks!


Change highlighted code

The previous code was:
if (courseID != null)
    {
        ViewData["CourseID"] = courseID.Value;
        viewModel.Enrollments = viewModel.Courses.Where(
            x => x.CourseID == courseID).Single().Enrollments;
    }

and it has changed to:
if (courseID != null)
    {
        ViewData["CourseID"] = courseID.Value;
        var selectedCourse = viewModel.Courses.Where(x => x.CourseID == courseID).Single();
        await _context.Entry(selectedCourse).Collection(x => x.Enrollments).LoadAsync();
        foreach (Enrollment enrollment in selectedCourse.Enrollments)
        {
            await _context.Entry(enrollment).Reference(x => x.Student).LoadAsync();
        }
        viewModel.Enrollments = selectedCourse.Enrollments;
    }

So I think the highlighted portions should change!

@badersur  looks great, thanks!

Welcome :)


Free-form C structure entry mechanism

Before closing #4229 there was a small discussion about adding a C-struct text mode editor that would make it easier to copy-paste and duplicate chunks between elements and in and out of the program.
There @ghidra1 said that the team had brainstormed about it in the past.
Personally, I think the all-or-nothing way one has to import and export all their existing headers/types in one go can be severely improved or complemented. When importing you need to do a lot of prepping and dial-tuning, checking that it isn't in read-write mode, and even if it succeeds it replaces the whole data block so you essentially need to move the imported data somewhere after the import and use a dummy/transient/disposable one for staging (in the likely case you are doing it in pieces).
Sometimes I change the enum or struct externally and it would be painless to do a direct copy-paste. Or the other way around, exporting small pieces of fields, or cloning them between data types, as well as quickly refactor in Ghidra by moving them around. It would make things more interoperable.
There is already something very similar for function signatures, and it works very well, so I feel like this would improve everyone's workflow.
Let me know what you think, and thanks for reading.

There is already something very similar for function signatures, and it works very well, so I feel like this would improve everyone's workflow.

If you are referring to the Function Editor's parser this is know to have many issues and does not leverage the C Parser.

When importing you need to do a lot of prepping and dial-tuning, checking that it isn't in read-write mode, and even if it succeeds it replaces the whole data block so you essentially need to move the imported data somewhere after the import and use a dummy/transient/disposable one for staging (in the likely case you are doing it in pieces).

I am completely confused by this statement and uncertain how this applies to using the C Parser - assuming that is what you were referring to.

One of the biggest obstacles with using the C Parser for re-parsing Ghidra datatypes is the fact that our type naming is not C-compliant and in many cases can drag in namespace and template constructs that the C Parser can not handle.  In addition, the C Parser can not be used for manipulating non-packed structures which are frequently incomplete in their definition and precise with component placement.

I am completely confused by this statement and uncertain how this applies to using the C Parser - assuming that is what you were referring to.

I'm talking about Ghidra data type archives, and how you need to manually toggle the read-only mode back and forth, and how a successful import seems to overwrite the whole archive. Which if you are incrementally updating or appending a single data type in a new header forces you to make temporary files to copy them over.
That's how I have been using it so far, at least.

If you are referring to the Function Editor's parser this is know to have many issues and does not leverage the C Parser.

I don't know the underlying system, it may be different and using an incomplete parser with many holes. But it certainly saves a lot of time. One can always revert on weird results.
Even if it starts small as a way of importing and exporting enums and maybe just importing simple structs without roundtrip it would come in handy. As you said, no matter what there's currently no good way of bridging this functionality.
To give some examples of workflow, I use 010 Editor all the time to reverse file format structs and apply binary templates. Even if I need to manually tweak padding/dummy bytes afterwards it would be a big improvement whenever I need to share common data type stuff. Having an immediate skeleton on Ctrl+C, Ctrl+V.

After reading the manual again (it's been a while) seems like at least the Preview Window in the Data Type Manager can be used to quickly copy parts of enums using a C-ish syntax, and the values are neater and even hex-encoded, unlike when exporting as C-header. Having 4000 constants in there with the preview pane opened brings the whole program to a crawl, but that's a different issue. If you want some example, try importing this in Ghidra and see how it fares every time that pane gets refreshed.

Another seemingly buggy feature is the fact that after completing Parse to File... for some already-loaded GDT one seemingly needs to essentially reopen or reload said file for the updated contents to show. This is something that threw me off the first time around, thinking that it just wasn't working, but just using Open for Editing seems to do the trick to refresh it.
I would definitely like a warning about potentially losing any preexisting GDT contents after using Parse to File.... I don't think that's clear enough when it warns about overwriting it. I initially assumed it would merge the types and categories already there.

I don't know why when asked about overwriting GDTs it can't figure out that the lock is local and owned by the current tool instance and proceed anyway. Seems more user friendly, it's going to essentially rewrite it anyway.

And this is just cosmetic, but when pasting directly a quoted path like "C:\Users\User\Documents\github\gforce-tools\gforce_hashcodes.enum.h" in the Choose Source Files dialog it seems to only strip the initial "s; leaving the trailing ones there. For example, this is what happens when you start to add extra trailing quotes.

We've had this on the queue for a while adding it to our GP tickets to do soon.

@emteere   Do you foresee this being incorporated into structure merges (update/commit to archive and in VT) or just creation/editing?  Hopefully somewhat the same logic and its just a matter of squeezing into the UI once the hard part is done.


Make sure that default_arg is not passed rather than checking for fal…

…sity
if not default_arg:
    ...

fails when default_arg can be assigned and still be false (implementing a __bool__ or __len__ etc). You dont want to overwrite the passed values.

Good catch @vrthra
Merging it right away!


C# wrapper for SAT solver

Question:
Is the C# wrapper for the SAT solver planned to be released any time soon?
Context:
I have a python script working very nicely with the SAT solver but need to integrate it into a C# application.
After reading #417 I followed the example in examples/test/testsat.cs to convert my python code, but when I add several variables to the linear constraint I get the error Unable to parse Google.OrTools.Sat.CpSolverResponse protocol message., it seems to work less variables.
Also I don't know how search for all the solutions in C#.

Hi,
Good news, @lperron has worked on C# SAT Solver and started to port most examples on it...
Please fetch the last master revision and give it a try ;)

Hi,
I would like to tell you that I am still encountering the same error message:  Unable to parse Google.OrTools.Sat.CpSolverResponse protocol message. I also don't encounter the error if I use less variables.
I am using Google.OrTools.6.7.4973. Is this a known issue? Is there any known workaround for this?
Thanks.

Hi,
Attached with this comment is the code sample that exhibits the problem. It is a VS2012 solution. Pardon me if the code is messy because I am still learning.
Thanks you.
JspPm_ParseError.zip

I ran from source,
Horizon: 75
Statistics

solve status : Optimal

makespan     : 25

conflicts    : 45

branches     : 547

wall time    : 0.012902394 ms

Protocol 0 Task 0 Machine 100 Start 0 End 3 Duration 3
Protocol 0 Task 1 Machine 201 Start 3 End 4 Duration 1
Protocol 0 Task 2 Machine 301 Start 4 End 6 Duration 2
Protocol 0 Task 3 Machine 201 Start 6 End 10 Duration 4
Protocol 0 Task 4 Machine 100 Start 10 End 15 Duration 5
Protocol 1 Task 0 Machine 101 Start 1 End 4 Duration 3
Protocol 1 Task 1 Machine 201 Start 5 End 6 Duration 1
Protocol 1 Task 2 Machine 301 Start 7 End 9 Duration 2
Protocol 1 Task 3 Machine 200 Start 9 End 13 Duration 4
Protocol 1 Task 4 Machine 101 Start 13 End 18 Duration 5
Protocol 2 Task 0 Machine 101 Start 10 End 13 Duration 3
Protocol 2 Task 1 Machine 200 Start 13 End 14 Duration 1
Protocol 2 Task 2 Machine 301 Start 14 End 16 Duration 2
Protocol 2 Task 3 Machine 200 Start 16 End 20 Duration 4
Protocol 2 Task 4 Machine 101 Start 20 End 25 Duration 5
Protocol 3 Task 0 Machine 101 Start 4 End 7 Duration 3
Protocol 3 Task 1 Machine 200 Start 7 End 8 Duration 1
Protocol 3 Task 2 Machine 301 Start 9 End 11 Duration 2
Protocol 3 Task 3 Machine 201 Start 11 End 15 Duration 4
Protocol 3 Task 4 Machine 100 Start 15 End 20 Duration 5
Protocol 4 Task 0 Machine 101 Start 7 End 10 Duration 3
Protocol 4 Task 1 Machine 201 Start 10 End 11 Duration 1
Protocol 4 Task 2 Machine 301 Start 11 End 13 Duration 2
Protocol 4 Task 3 Machine 201 Start 15 End 19 Duration 4
Protocol 4 Task 4 Machine 100 Start 20 End 25 Duration 5
Therefore, I do not know the source of the bug.
I suspect a protobuf version mismatch. We will investigate.
Laurent Perron | Operations Research |<EMAIL_ADDRESS>| (33) 1 42 68 53
00
Le dim. 24 juin 2018 à 22:22, Haukinger<EMAIL_ADDRESS>a écrit :

Any idea when this bugfix will show up in the nuget package?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/or-tools/issues/552#issuecomment-399784813,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKj17YjN-MUiksox0ymImXlH19_4eWQdks5t__UGgaJpZM4RQolp
.

@lperron It is strange that you can run the code in JspPm_ParseError.zip without any error.
I have double-checked that the code is still causing problem on my laptop. By the way, I am on Windows 10 64-bit latest update as of today with Intel Core i7 7500U and 12 GB of RAM if that helps.
Try to increase the number of data, by doubling the loop count and see if you can reproduce the issue.
Replace
for (var i = 0; i < 5; i++) // this is having problem
With
for (var i = 0; i < 10; i++) // this is having problem
If you suspect that it is protobuf mismatch, then probably there is a problem with or-tools package in Nuget.

I'm experimenting with the job-shop-sample (link), and until now, I just increased the number of jobs and machines looking for the limitations of the solver. With 15 machines and 9 jobs I get the error.
I've tried both protobuf 3.5.1 as in the nuget package and the current 3.6.0 version. Windows 7 x64 on i7-3520M here.

Any news on this one? Will this fix make it to a 6.7 version? Or do we have to wait for 6.8? You have a timeline for this?


Make the METS Transformer work with Born Digital METS files

Currently, the METS transformer assumes (until now, correctly) that all METS content represents digitised material associated with a Sierra Record.
There are some key differences between Sierra METS and Born Digital METS.
There are also significant similarities.
Make the transformer capable of working with Born Digital METS files

Derived from #2476

METS is one of those XML formats with IDREF all over the place, and the existing transformer doesn't care.  For example, it just assumes that the descriptive metadata in the first section applies to the whole document, and therefore individually to all of the files in the structMap.
This is probably fine for the digitised works, but I don't have a great deal of confidence that the assumptions made for digitised METS all hold true for born digital METS.
One of the reasons METS allows multiple dmdSecs is so that different objects in the package may be associated with different metadata.

This may need a bit of further investigation/chat with Tom about what assumptions I can make, rather than having to write a full METS parser.

In this first step at least, the output of the transformer for BD METS should provide no different extra data than the digitised METS.  Once we have started ingesting it, then we may wish to extract different fields.

Step 1: [https://github.com/wellcomecollection/catalogue-pipeline/pull/2494](Stop premature normalisation) - METS URLs were being turned into DLCS ones in the class that should be non-judgmentally pulling data from the XML files.  This made it a bit tricky to reuse the FileReferences code from the existing METS transformer.
After I did this, I found a few other bits that made reuse/extension a little tricky (in no particular order):

Make the test suite a bit more comprehensive WRT Float elements. Not strictly necessary, as in the end, I just did a lift-and-shift on the relevant part of the code, but this is a sensibly isolated bit of work that really should be there anyway.
Titles are optional.  Archivematica mets files don't seem to have anything that really constitutes a title, so I've just made it an empty string for now.  When we get some more examples, we can see if we want to extract something, but I expect it will just be ignored by the merger anyway.
An odd two-stage split when creating a MetsData instance Partly, I just found this irritating, but this also made it easier to create the InvisibleMetsData companion object.  I think this can probably go further,  I don't think we need to return a MetsData and then call toWork on it later, but too many tests rely on the two steps, so I didn't want to rock the boat there too much, otherwise I'd never finish.

Finally: Actually extract Archivematica-style METS data.  This one is still a draft, and it builds on all the above, so it's probably best to leave it until the others are all on main and I can see the wood for the trees.  Unfortunately, this one grew to a bit of a large atomic change, as there's quite a lot that broke the old way as I was getting on with it.  I might be able to make it a bit less unwieldy next week.

https://github.com/wellcomecollection/catalogue-pipeline/pull/2492


CrossfadeDrawerLayout below status bar and App bar

@mikepenz. Thanks for this great library, its indeed helpful.. my issue is that i can't figure out how to make the cross fade drawer layout show behind the ActionBar/APP Bar. I tried using
.withDrawerLayout(R.layout.crossfade_drawer).withRootView(R.id.miniDrawer).withRootView(R.id.drawer_container) But i was getting: "Relative layout" cannot be cast into drawerlayout.
I went through previous issues to see if i could find similar question but couldn't find one. i also tried using the .withBelowStatusBar still problem persist.
Below is my cross fade drawerlayout.xml
<android.support.v7.widget.Toolbar
    android:id="@+id/toolbar"
    android:layout_width="match_parent"
    android:layout_height="?attr/actionBarSize"
    android:background="?attr/colorPrimary"
    android:elevation="4dp"
<EMAIL_ADDRESS><EMAIL_ADDRESS>/>
<!-- the layout which will contain (host) the drawerLayout -->
  <FrameLayout
      android:id="@+id/drawer_container"
      android:layout_width="match_parent"
      android:layout_height="match_parent"
      android:layout_below="@id/toolbar">

<com.mikepenz.crossfadedrawerlayout.view.CrossfadeDrawerLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:id="@+id/material_drawer_layout"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:fitsSystemWindows="true" />

  </FrameLayout>

Thanks in anticipation.
Regards

@kennyung6 please have a look on the CustomContainerActivity of the sample app. It shows exact the same use-case just with a normal DrawerLayout
https://github.com/mikepenz/MaterialDrawer/blob/develop/app/src/main/java/com/mikepenz/materialdrawer/app/CustomContainerActivity.java

Thanks @mikepenz


Css Printing Rules

Expected behavior
I'm expected that a long text, auto breaking page. If I print the HTML page, the result is correct (3 pages), but if I convert in PDF with phantomJS, all zoom out in one page.
Actual behavior
The HTML rendered in one page and when printing zoom out all
Steps to reproduce the behavior

Create a page with css rules (

@page {
size: A4;
margin: 190mm 132mm 367mm 190mm;
}
@media print {
    html,
    body {
        width: 210mm;
        height: 297mm;
    }
}

) and long paragraph (more than one page)
2. Run phantomjs on server with previous page

### Environment

- OS: OSx (El Captain)
- Node version (`node -v`): 6.2.0
- package version: 2.1.2

This seems to be related to phantomjs and not this module. I suggest trying it without node.

Thanks for response!
I've tested all with phantomjs and, with correct settings, all works!
Now I've copied this settings in node, but doesn't works as I'm expext. This is my code (node):
`phantom.create().then(ph => {
_ph = ph;
return _ph.createPage();
}).then(page => {
_page = page;
    _page.paperSize = {
        format: 'A4',
        margin: {
            top: '1.9cm',
            left: '1.9cm',
            right: '1.32cm',
            bottom: '1.32cm'
        },
        footer: {
            height: '1cm',
            contents: _ph.callback(function (pageNum, numPages) {
                return "
- " + pageNum + " -
";
            })
        }
    };
    return _page.open('http://www.google.com');
}).then(status => {
    if (status != 'success')
        return;

    return _page.render(pathFile);
}).then(content => {
    var stat = fs.statSync(pathFile);
    fs.readFile(pathFile, function (err, data) {
        res.setHeader('Content-Length', stat.size);
        res.setHeader('Content-Type', 'application/pdf');
        res.setHeader('Content-Disposition', 'inline; filename=report.pdf');
        res.send(data);
    });

    _page.close();
    _ph.exit();
}).catch(err => {
    console.log(err);
    phInstance.exit();
});`

Wait that's not how you set paperSize. You have to use property method. Look at the readme

Ok ok... Now works!
For the future:
I've replaced:
_page.paperSize = {};
with:
page.property('paperSize', {...});
Thank you! And sorry to don't understand README

@salcam I added a warning so this is more obvious. See 5d7d7ec103c4c68e27d12948b3b126d63586a955

Related https://github.com/amir20/phantomjs-node/pull/477

It's perfect!!! Thanks a lot


test: cookies controlling 1st party storage

2701: cookies control access to blah blah. Does this control 1st party storage. Test required.

[NOTE] This also controls access to 3rd party Web Storage, IndexedDB, Cache API and Service Worker Cache

Note: With Web Extensions it certainly blocked Extensions using IDB (exhibit A: uBO, uM, VM, Stylus). It also blocked them using localStorage as well (see Disconnect test I did a few days ago). Not sure if that means they were treated as first party.
Feel free to test and post STR and outcomes
PS: reminder about HW-A and gfx.direct2d.disabled

Simple Test No.1
The Setup:

Nilla profile, FF59.0.1: I allowed 1st party cookies only. I cleared all Site Data (from about:preferences#privacy), I cleared history, i.e time range everything, and all check boxes checked (ctrl-shift-del). I checked I had zero cookie exceptions, and I had no cookies. A bit overkill, nilla profile, but hey, ZERO data

The Control:

load github, open Storage Inspector (Shift-F9, make sure your github page has focus)

cookies, yes
localStorage, yes (2 items)
IDB, no (this is handy for tests later on re santizing, hence why I mention it)

The Test:

under options, block ALL cookies
scrub all data (see setup), repeat control test

cookies, no
localStorage, no

The Picture:

I need a test site that only does IDB (not youtube, that uses SWs to push its storage). I'll test YT later

When I get time I will test SW's->IDB which is what Youtube does

That's all I ever thought blocking cookies protected me from as 1st party: localStorage and IDB. My assumption was based on experience (eg see issues with extensions) etc. I never did any tests.
I also never thought about service workers (because they are disabled) or appCache (which until now was disabled). I have done some interesting tests with appCache .. see next post

PS: I am not sure if these tests are definitive, so poke holes in any procedural flaws or assumptions etc
Simple Test No.3 (appCache)
Site: https://devdocs.io/ (note: secure context required for appCache these days, can;t be bothered looking up which version it kicks in)
Side Quest:

set offline-apps.allow_by_default to false and the site prompts to "allow storing data"
There is no option to remember - allow or deny. I clicked allow
There is no change to site permissions panel info.
Reload the page, it remembers. Clear everything except site permissions, reload, it still remembers.
Clear site permissions and it forgets
So maybe a bit of a bug re permissions panel info, but ONLY if Store data in persistent storage is meant to relate to this

The Control:

allowed 1st party cookie, scrub all data, load site, "allow storing data" if prompted
Storage Inspector (which does not display info on appCache) shows

cookie, localStorage, IDB = yes
cache storage, session storage = no

profile/OfflineCache/index.sqlite

lots of devdocs entries

AppCache Sanitizing Tests:

both Site Data>Clear All Data and clear everything (ctrl-shift-del) wiped the appCache (yes I did separate tests)

The Test:

scrubbed all data, blocked 1st party cookie, loaded site

profile/OfflineCache/index.sqlite showed lots of devdoc entries

So cookies do not control 1st party appCache - that's what the prompt is for

@earthlng : does this answer your question?

The NOTE is correct, but it also happens to be that cookies (for now) also control 1st party localStorage and IDB.
AppCache, which I tested because we flipped it, is not controlled by cookies, but the sanitizing and prompts work, so we're covered there.
As for SW cache, I doubt a cookie controls it as 1st party - not even sure how to test it TBH. We don;t even claim it's covered as 1st party. Thoughts?

Can I close this now? PS: I want to ignore the HW-A thing!

Related to: https://github.com/ghacksuserjs/ghacks-user.js/issues/390#issuecomment-376639248

How does this work?

https://devdocs.io/offline

Each page is cached as a key-value pair in IndexedDB (downloaded from a single file).
The app also uses AppCache and localStorage to cache the assets and index files.

I was testing appCache (since we made it inactive in order to make Site Data in the UI work). I tested four things:

sanitizing on close = same as clear recent history = works
clearing site data button = works
if cookies were involved - not used
the prompt = works

Not sure by what you mean "how does this work?" I don't care if the site works or not (or how it works), I was just testing persistent local data

nicely done, thx! So cookies also controls some but not all storage. I assume that means when cookies are blocked storage is also blocked for both 1st and 3rd party. When cookies are allowed for 1st-party only then 3rd party storage should still be blocked.
The UI does now also reflect that better than it used to ie it no longer just says "cookies" but "Cookies and Site Data". If you toggle between "Accept cookies and site data from websites (recommended)" and "Block cookies and site data" you can see that "Accept third-party cookies and site data" also changes from Always to Never and vice-versa.
Pref-wise, selecting "Block cookies and site data" sets network.cookie.cookieBehavior to 2.
"Accept cookies and site data from websites (recommended)" sets "Accept third-party cookies and site data" to "Always" which means network.cookie.cookieBehavior = 0, and to achieve 1st-party only (= network.cookie.cookieBehavior = 1) you then have to set "Accept third-party cookies and site data" to "Never".
Site Data in this context (ie blocked by the cookie pref) definitely covers localStorage + IDB, might cover service-workers cache and maybe some of the other storage behind QuotaManager like asmjs-cache but does definitely not include appCache (aka offline cache).

 Having modified network requests, wide open to man in the middle attacks would be really bad. In Firefox, Service Worker APIs are also hidden and cannot be used when the user is in private browsing mode.

the fact that they're disabled in PB means that there's definitely something wrong with them, IMHO.

  This is rather vague, everywhere else we're using the item names in 2700's header. Is Web Storage -> localStorage in this case?

well there's a ServiceWorkers cookbook and a Service Worker Demo. Testing the latter shows that SW also uses the storage folder. In my case it went under storage/default/. It also created a serviceworkers.txt file in the profile top directory. Cleaning OWD removed the subfolder and also cleaned the txt-file.
about:serviceworkers + about:debugging#workers might also be useful when testing SWs

I just struggling with how to indentify where the SW cache actually is or how its marked as SW cache. I mean, Youtube uses SW's to write data to IDB. But IDB data is not SW cache, is it, it's IDB.
Maybe SW cache means data stored using already existing storage types but set by a SW? It's a bit over my head TBH

https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers

if you use AppCache and SW on a page, browsers that don’t support SW but do support AppCache will use that, and browsers that support both will ignore the AppCache and let SW take over.

Note: localStorage works in a similar way to service worker cache, but it is synchronous, so not allowed in service workers.

Note: IndexedDB can be used inside a service worker for data storage if you require it.

SW cache goes into a cache subfolder under storage/default/http.../ and IDB into idb. IDK if it always goes into default though or if it can also go into permanent or temporary.

PS: I think we should move to v60-beta before changing info on storage, site data (which I know is not in ESr52.x), sanitizing etc - because there have been way too many changes, and I am definitely not up to testing in ESR, just adds more complexity
Am happy to do a release and move to 60-beta now - just :+1: if you want

about that, why don't we switch it around and mark it -alpha first and then -beta later for the pre-release used for the changelog? that would make a lot more sense IMHO

So...

eg today: 59-> 60-alpha and start applying 59->60diffs etc
eg just after stable lands: when all diffs done etc -> 60-beta and do an beta release

no more -alpha releases

weeks later

change to 60 and do a legacy-end-of-life release
immediately after that I then move to next alpha
maybe use -final as its only going to be live for 10 minutes

So we're always in a state of alpha or beta, which is technically true, we're always tweaking it. Did I follow that right?
-alpha applying new stuff, -beta ready for use after stable lands, -final for archived releases

yes, more or less. Not necessarily "immediately after that I then move to next alpha" because it can wait until we do a first commit with some actual changes rather than just change the version. I wouldn't do -final either, just the version without -alpha or -beta is fine.

Not necessarily "immediately after that I then move to next alpha" because it can wait until we do a first commit with some actual changes

That's problematic. The work-flow, reminder, trigger (for me) is doing the release, otherwise, days/weeks later with the next commit, it's too easy to forget.
IDK, this seems like overkill, adding a third version build type, one of which is effectively only used for a single point-in-time. Would you be disappointed in me if we just left it at status quo - I feel its much simpler to follow?

SW cache test?
Test Site: https://www.youtube.com/watch?v=nfWlot6h_JM   #TayTay :heart:

I wait for the 15 sec advert to finish and then stop the vid after a few seconds of play so nothing else loads
to view SW info use about:serviceworkers (also if you want about:debugging#workers & profile/serviceworkers.txt)

Control Test

1st party cookies only, SW cache enabled
close all tabs, scrub all data, load TayTay
Site Data says 200 KB

open Storage Inspector on YT tab
yes: cookies, local storage, IDB
reload Site Data: now shows 119 KB (probably something expired)

SW: one YT registered SW

Test A

1st party cookies only, disable SW cache dom.caches.enabled=false
close all tabs, scrub all data, load TayTay
Site Data says 176 KB

open Storage Inspector on YT tab
yes: cookies, local storage, IDB
reload Site Data: now shows 119 KB

SW: one YT registered SW

Test B

no cookies, SW cache enabled
close all tabs, scrub all data, load TayTay
Site Data says 0 bytes

open Storage Inspector on YT tab
no: cookies, local storage, IDB
reload Site Data: now shows 128 KB
64 KB each for YT and tpc.googlesyndication

its in the IDB storage/default folders

SW: nothing!!
Conclusion: this must be a clear cut example of SW cache data?

Test C

no cookies, disable SW cache
close all tabs, scrub all data, load TayTay
Site Data says 0 bytes

open Storage Inspector on YT tab
no: cookies, local storage, IDB
reload Site Data: now shows 0 bytes

SW: nothing!!

Conclusion (so far):

I used YT as a test because I knew it used SWs to load the comments section. Probably not the best test site, and don't forget sites can code for fallbacks to blockage. AFAICT, cookies have nothing to do with SW cache and it is solely controlled by dom.caches.enabled which makes sense.

Slightly OT: 2711 doesn't belong in 2700s which is about web, not chrome - I think we should stick it with other 2600's extension stuff
/* 2711: clear localStorage and UUID when an extension is uninstalled

^^ Test B is weird: no registered SWs (can they unregister themselves?), but we got some data in IDB folders and we know IDB is blocked by the cookie block. I can't quite work it out. If its SW cache then there must be a SW? :headspining:

Not necessarily "immediately after that I then move to next alpha" because it can wait until we do a first commit with some actual changes

That's problematic. The work-flow, reminder, trigger (for me) is doing the release, otherwise, days/weeks later with the next commit, it's too easy to forget.

I said "not necessarily". If you prefer to do it immediately afterwards that's fine too.

Yup. I get that :) I'd have to do it immediately or I could forget (and being out of sync would highly annoy me as a failure of the system/methodology). Even if I wait til the next commit, to me it's still just an extra version for a single state of the project: the one-off legacy archive. I get that it's more in line with version-naming conventions etc and software build environments.
While I appreciate that the user.js is always a work in progress, I don't like calling the version after the alpha release, a beta. Its not really a beta (to me), its ready to go for that version, we've finished that version's diffs. Every change after that is almost always just improvements (links, revamps) and version+1 diffs getting in ahead of time.
I can tell you really want this. Right? if I stuck with the status quo, would you be:
 [ ] mad
 [ ] really mad
 [ ] leave the project
 [ ] upset
 [ ] disappointed
 [ ] understanding
 [ ] forgiving
 [ ] keep talking me into it
 [ ] still love me

You can tick as many boxes as you like


pjsip: add pythonSupport option

Description of changes
Currently the pjsip package doesn't expose Python bindings to use the pjsua2 module. This PR attempts to add this when pythonSupport is enabled. The resulting package has site-packages available:
$ ls -1 result/lib/python3.10/site-packages/
pjsua2-2.13-py3.10.egg-info
_pjsua2.cpython-310-x86_64-linux-gnu.so
pjsua2.py
__pycache__

I could import pjsua in python:
$ PYTHONPATH=/nix/store/73cf30ypmhy4dp8k8qp4z53giwqfz6cq-pjsip-2.13/lib/python3.10/site-packages nix run .#python3
Python 3.10.9 (main, Dec  6 2022, 18:44:57) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pjsua2
>>>

Note that I enabled pythonSupport by default, but I don't know what the policy is for such options.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

It's less objectionable enabling python support by default if you move all the python-related files to a separate python output.
It's also often useful to add a reference to a python3-overridden variant of it in python-packages.nix so that it's available to any python version rather than just the global default python3.

Hmm, that's a good point. I have looked into building the python package on its own, but it relies on other parts of pjprojects build and private header files. I'll look into this again to see whether there are ways to make a separate python derivation without rebuilding pjproject as well.

I have incorporated the suggestions. @risicle could you have another look?


Refactor GS API bindings

Description

Checklist

[ ] added release notes to Unreleased section in CHANGELOG.md, if user facing change

References

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I don't like how we have a ticket to plan this and then there's lots of code already that I have to parse again  ^^`
Short look only rn, as I actually tried to get another hour of sleep but had another idea to solve this, but looks like you add code to the generic client to handle specific cases? I think we talked about this before.. it's not gonna be maintainable with all the different behaviors we have in different APIs, that's why it was hooks in the first place
Our problem is the hooks being attached to the objects and working on the Object they get via the receiver, right?
What if we moved all the hooks away from the object types itself and have them optionally implement an interface "client overrides" where we have those hooks instead?

they would get the object via an argument, so extending objects via embedding still works
we still have custom logic per object type if needed but not in the generic client itself
we can share lots more code since e.g. all of GS has a single "client override" class they can all use
it makes the object types a bit cleaner  conceptually - they are what's transferred via API request, not the logic for that
existing objects could just return themselves as their "client overrides" object to still have the hooks directly on themselves (with slightly changed interface), easing migration to this pattern

Ah so you are essentially doing those things here:

add a way to flatten linked objects to their identifier

implemented here with reflection in the generic client
implementing this with (generated) code in the Objects would be better imho, reflection can get slow and this is something we should be able to do at build-time rather easily?

remove the need for hooks for GS State
remove the need for hooks to deal with GS APIs returning the object data on DELETE

I still think this does not belong into the generic client and should work good with a "client override" class instead of hooks on all the affected Object types

Maybe we should just have Encode and Decode methods on the "client overrides" interface, getting a context.Context, io.Reader and Object to decode into - they could have different encoding/decoding logic per operation and even call out to the Object's Encode/Decode which is generated to handle e.g. this identifier flattening?

implementing this with (generated) code in the Objects would be better imho, reflection can get slow and this is something we should be able to do at build-time rather easily?

I did a benchmark and while flattening has a relative high overhead compared to just json encoding, considering the unit of the extra time needed is not a valid concern for an API client IMHO.
BenchmarkEncodeWithFlatten-8              134679              9044 ns/op
BenchmarkEncodeWithoutFlatten-8          5170754               247.3 ns/op

Implementing this with generated code would probably not be as trivial as it appears with the needed support for embedded structs.

remove the need for hooks to deal with GS APIs returning the object data on DELETE

I still think this does not belong into the generic client and should work good with a "client override" class instead of hooks on all the affected Object types

With the plan to eventually move everything to GS I think it'd be better located in the client core than overriding it for every resource.


Measure Impact with Analytics

As far as I know we don't measure any analytics, which means that we don't know how many people are using the site thus the impact of the project.
A few months ago, through some of 0xb10c's projects, I stumbled upon umami, a simple, self-hosted, open-source alternative to Google Analytics. The setup is very easy and they even have a simple one-click deployment to Heroku which we could use with a free account to initially validate the idea. After having the server it just needs a tracking script that we can easily add to our Hugo setup.
The cool think we umami is that

After you setup your umami instance you can have unlimited websites and separate accounts. That's how I tested that this is working for bitcoin transcripts by using my own umami setup to get analytics from a local instance of this repository.
You can create a shareable link in which everyone can see your website statistics. This is the link showing the analytics for my aforementioned demo.

You can see examples of umami at

The footer of https://bitcoin-dev.blog/ which also has a link for their publicly accessible statistics.
The footer of https://miningpool.observer/ which just mentions the use of umami.

If we proceed with this idea, the implementation is the easiest part (minor code changes + 1 repository secret for the tracking code). We just need to figure out the logistics of the self-hosted setup, which as I said, we can avoid with a new account in Heroku.

I understand the urge to want to track, but I guess the question is for what purpose? What would the use of this data be?

 This is an open repository of transcripts were each contribution is help for a collective effort of aggregating a valuable resource for the benefit of the community. I think that being somehow aware of how the community uses that resource gives an inside for future steps and improvements.
In my mind, the data have two purposes. (1) Incentive for contributors, a way to see that their work has impact (people are reading their transcripts) thus continue contributing. (2) A way to measure the results of future changes/actions.
My hypothesis is that more audience results to more contributors thus a better website and greater impact.
That's more of a personal note, but if I'm dedicating time to something I want it to have the max possible impact, if it doesn't I'll try to improve it. But this needs data. You need data to confirm that people are using the website. And if that's not the case, you need data to validate future actions.


[ServiceBus] Unification of Receiver and SessionReceiver

design: session receiver lock renew

should we raise error when user call renew_lock on the session receiver

if so, should we just rename it to renew_message_lock to be more clear

https://github.com/Azure/azure-sdk/issues/1884#issuecomment-714252397

In case of sessions, the receiver.session.renew_lock() renews the lock on the sessions. The receiver.renew_lock(message) method was proposed to work the same way. This has push back from Richard, Harsha and Ramya as summarized below. Python team to revisit this decision
Having renew_lock(message) renew the session lock is misleading. Losing lock on message is a minor inconvenience as Service Bus is expected to provide a atleast once guarantee not atmost once. But losing on session is much more serious and requires users to recreate the receiver to attempt to get a lock on the session or decide to move on to the next session
Having renew_lock(message) renew the session lock can lead people to call it on every message which results in unnecessary renew locks on the session under the hood

@yunhaoling @KieranBrantnerMagee - I'm okay with conceding on this one.
The rest of the SB crew aligned with your perspectives, and additionally had not given due consideration to the "user choosing to let a message expire in the session scenario means the loss of that session".

receiver would have renew_message_lock method, and it would raise error called on the session receiver would go into the settlement PR


rvm install fails on server

When trying to install Ruby as the Ubuntu user the following occurs:
Checking requirements for ubuntu.
Installing requirements for ubuntu.
mkdir: cannot create directory ‘/usr/share/rvm/log/1664326928_ruby-3.1.2’: Permission denied
tee: /usr/share/rvm/log/1664326928_ruby-3.1.2/update_system.log: No such file or directory
Updating system....
Error running 'requirements_debian_update_system ruby-3.1.2',
please read /usr/share/rvm/log/1664326928_ruby-3.1.2/update_system.log
Requirements installation failed with status: 1

When I try to view that log file, it is almost universally missing. It seems /usr/share/rvm is owned by root which might be wrong or need to be fixed. Otherwise not sure what the issue is here.

Miraculously this worked when attempting to reproduce today.


⚠ ClamAV has degraded performance

In c80e1ee, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 68 ms

Resolved: ClamAV performance has improved in 5f082f6 after 6 minutes.


Can not use Spyder 6 with old conda (4.8.3)

Issue Report Checklist

[ ] Searched the issues page for similar reports
[ ] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[ ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[ ] Could not reproduce inside jupyter qtconsole (if console-related)
[ ] Tried basic troubleshooting (if a bug/error)

[ ] Restarted Spyder
[ ] Reset preferences with spyder --reset
[ ] Reinstalled the latest version of Anaconda
[ ] Tried the other applicable steps from the Troubleshooting Guide

[ ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
kernel doesn't open when using old version of conda (4.8.3, works with 4.9)
What steps reproduce the problem?

Install Spyder 6 with conda 4.8.3
Try to use a kernel

What is the expected output? What do you see instead?
Paste Traceback/Error Below (if applicable)

An error occurred while starting the kernel
usage: conda‑script.py [‑h] [‑V] command ...
conda‑script.py: error: unrecognized arguments: ‑‑no‑capture‑output

Versions

Spyder version: 6.0.0
Python version: 3.14
Qt version:
PyQt version:
Operating System name/version: windows

Hi @impact27 thanks for the report! So probably we need to handle the case of people having older conda versions, right @ccordoba12 @mrclary ? Or maybe at least better detect the error and prompt users to update conda >=4.9 ?

@impact27, how do you have Python 3.14?
Why would the conda version matter?

Or maybe at least better detect the error and prompt users to update conda >=4.9?

I think this is the only thing we can do because we can't activate envs with conda versions that don't support --no-capture-output. The problem is this:

We know use conda run to activate envs, instead of the old scripts we used before. That mechanism is more robust and better supported upstream by conda.
@impact27, since you changed receiving the kernel's stdin/stdout streams to pipes, we need a way to let that info pass through the conda run process to get it in Spyder. And that's precisely what --no-capture-output does.

A fix shouldn't be that hard. Just checking that conda‑script.py: error: unrecognized arguments: ‑‑no‑capture‑output is in the kernel error message and change it for another text that is more meaningful.