delete old activations doesn't free disk space

the delete in couchdb is an update actually, so it will increase the usage of disk space, and while the purge is not implemented in couchdb right now, looks like we have no way to free the disk space
I do a test to insert 10 millions documents to couchdb, and delete them later, below is my test result:
before insert:
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
7.5M    ./data
36M    .

after insert 10 millions documents:
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
2.1G    ./data
2.2G    .

after create the activations/byDate view:
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
4.6G    ./data
4.6G    .

after delete all test documents(the used space increase)
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
6.9G    ./data
6.9G    .

after compact db
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
4.5G    ./data
4.5G    .

after compact view
root@bf83420db0ec:/opt/couchdb# du -h --max-depth=1
64K    ./bin
5.8M    ./erts-6.2
40K    ./etc
19M    ./lib
244K    ./releases
3.2M    ./share
0      ./var
2.3G    ./data
2.3G    .

So even after compact the db and view, the usage of disk size doesn't reduce a lot(it is bigger instead)
I know this is related to couchdb, but I wonder if there are someone has some experience to solve this problem.
This can be a problem in the production environment as the used disk size will keep growing

@cbickel do we have a compaction script?

Even after doing a purge follow by a compact, disk size will not shrink as docs are never deleted.
_purge
Purging documents does not remove the space used by them on disk. To reclaim disk space, you should run a database compact (see /db/_compact), and compact views (see /db/_compact/design-doc).

with compat you are removing old revisions, so not a complete delete
_compat
Writes a new, optimised, version of the database file, removing any unused sections from the new version during write. Because a new file is temporarily created for this purpose, you may require up to twice the current storage space of the specified database in order for the compaction routine to complete.
Removes old revisions of documents from the database, up to the per-database limit specified by the _revs_limit database parameter.

If you want to really delete docs, for example old activations that you don't care about them after certain period you would need to use a new db, and replicate docs you really want to keep.
There are a few blogs around the web that talk about this: https://eclipsesource.com/blogs/2015/04/20/how-to-finally-delete-documents-in-couchdb/
For whisk db holding docs for entities like actions, then purge and compat are more effective, deleting old history.

yes, it seems _purge  will work well, but _purge is not implemented by now: https://github.com/apache/couchdb/issues/965
and replace the activations db need about 1 second to finish, this may cause error during that period

Ah thanks for pointing out the _purge item, I thought I saw purge long time ago in some couchdb docs in 1.x maybe I confused them with compat
What 1 second are you talking about and what errors?
In production we do blue/green swap to avoid downtime period and avoid errors.

ok, I understand now, thanks a million!
1 second is mean: we delete the old activation database and create a new one using same name, this need about 1 second to finish, and during that time the activation database is not exist, that is the error.
By using blue/green deployment we can avoid that error

Oh never delete the active db, create a new db with a new name, once you move all components to the new db, then you can go back and delete the old db and start reclaiming disk space.

ok, I get that now, thanks again 😄

@csantanapr How did you manage such procedures without any service shutdown?

@style95  You have to replicate the documents of your current database (e.g wsk_activations) to a second database (e.g. wsk_activations1).
When the replication is finished, you can change db_whisk_activations in ansible to the new database activations1.
After a redeployment of the system, the new database will be used and you can delete the old database.

@cbickel Could you redeploy your system under high load as well?
Do you have logics for zero downtime deployment or blue-green deployment?

As Carlos mentioned, we have a blue-green deployment.
The load will go to blue, while green will be updated. After the update is complete we route the load to green and update blue.

below is my thought with rolling update deployment:

create continuous replication of activations database with selector:

{
   ...
   "selector": {
      “start": {
        "$gte": "1512630323343"
      }
    }
   ...
}

create new replication from new database to old one, then the replication is bidirectional

configure new deployment using new activation database

start deployment, the new component will use new database, and old ones still use old database, as the two database are replicated between each other, so it's ok to use different database

after deployment is over, the old database is useless, we can delete it

In opensource OpenWhisk, blue-green deployment is not supported yet, right?

blue-green should be supported, but everyone would have specific requirements based how they deploy.
I don’t have a problem if someone wants to contribute a doc sectio/paragraph  in the ansible readme capturing a simple/based blue-green scenario. The end results boils down having two environments definitions and some type of top level nginx/router/gateway to drain and switch traffic.

another problem is: the growth of disk usage may be very fast, it can be hundreds of GB per day if the tps up to 10,000(this value is not high if cluster hosting many services), we don't want to redeploy frequently, right?

It could certainly grow - the metadata is small (< 1KB) but the result and logs could be MBs large in all. We are doing and discussing the following to reduce the load on the db: 1. Stop creating rule activation records and only create trigger activations if they cause actions to execute, 2. Possibly drop the trigger activation altogether in the future, 3. Retain logs in a seperate store with a much shorter retention and defer to a stack like ELK for longer longevity.


sqruff: init at 0.11.0

Description of changes
This only adds x86_64 linux for now.
Notice that we can't build this from source, as it uses 1.81 nightly features.

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 332976 run on x86_64-linux 1

1 package failed to build:

sqruff

GitHub has an interesting UX. Sorry about that!

Result of nixpkgs-review pr 332976 run on x86_64-linux 1

1 package built:

sqruff

@cafkafk could you please add a comment to sqruff package that outlines which Rust 1.81 nightly features are used that prevent this package from being built. This would be helpful so that someone can revisit this package in the future and build from source once the dependent Rust features are available in nixpkgs' Rust package.

@cafkafk could you please add a comment to sqruff package that outlines which Rust 1.81 nightly features are used that prevent this package from being built. This would be helpful so that someone can revisit this package in the future and build from source once the dependent Rust features are available in nixpkgs' Rust package.

...can't you just add it after merge? feels like this keeps dragging on with unimportant nitpicks, updating should be enough.

I honestly already have this locally build from source with crane, and I prefer using that... I'll upstream another time, perhaps when we have a toolchain that is ready to package it.


ISSUE-94 Update documentation to explain how to access a specific branch from github locally(other than main)

Update documentation explaining how you can access a specific branch. (Useful for testing code when doing code reviews from pull requests).
https://www.freecodecamp.org/news/git-clone-branch-how-to-clone-a-specific-branch/

git clone [URLTORepository]
git checkout  [commithash]
commit hash looks like this "45ef55ac20ce2389c9180658fdba35f4a663d204"


perf(map): optimize MapRef to reduce more copy

We can use (ListRef<'a>, ListRef<'b>) to replace MapRef. Note that ListRef is actually a generalized ref over an Array, not specific to list type.
Btw, in arrow-rs, the Array types are actually ref types. And so ListArray::iter's elements are just ListArray, and MapArray iterates over StructArray.
But our arrays are owned types, which made it harder to do MapRef properly here.
But it's indeed not that simple, as we cannot get the underlying (ListRef<'_>, ListRef<'_>) from a ListRef<'_>. It's Arc<ArrayImpl> vs &'a ArrayImpl thing.
Similarly, we cannot build ListRef<'_> from the tuple too. This will make it more useless. e.g.,
fn append_n(&mut self, n: usize, value: Option<ListRef<'_>>)

Perhaps we should refactor ListRef to use Arc<ArrayImpl>

Fix: the OP is wrong. ListRef<'a> -> (ListRef<'a>, ListRef<'a>) is completely OK.
https://github.com/risingwavelabs/risingwave/pull/17690#discussion_r1696481562

Should we switch to arrow-rs? 🤪

This is technically possible and I agree with it. But there will be endless migration work. 🤪

It seems our array implementation has flaws to support such kind of optimizations. Should we switch to arrow-rs? 🤪

Why? 😕 Can't we make it a enum to support both cases? Just like the idea of Cow.


generator: *string attribute value is ignored

Right now templ.Attributes silently ignores *string during rendering. It would be helpful if it had a case for *string and if the value is not nil render it

Thanks for raising this! And thanks @whalenrp for the code contribution.
For now, I've marked this as needs decision, to allow for a discussion on the expected behaviour. I think I agree with your implementation... what to you think @a-h ?

That looks sensible to me, I've merged it in. Thanks for that.


field.id doesn't work for templates

Trying to print out the value of both the hashid and the underlying id in a django template.
field > hashid value
field.id > empty
field.hashid > empty

I'm unable to duplicate this. You must have something else going on between the Model and the template, such as resolving the field with str(field) or something similar. For example, I just put this in the sandbox code:
{{ author.name }} ({{ author.id.id }}/{{ author.id.hashid }})

and it properly printed out entries such as:
Arthur C Clarke (1/3586d4894bae7)
Frederik Pohl (1000/85e7469824da6)

in the template output.
I would need a test case that duplicates this problem to debug. At least I need to know exactly what settings are enabled for the field, and see some code in the view where it fetches the records and passes them to the context of the template.


[Backport v3.1-branch] Backport 46195 to v3.1 branch

Backport #46195 to v3.1 branch

@chen-png Could you file an issue for this and link it?

@chen-png Could you file an issue for this and link it?

thanks done.
but I closed it by mistake, and it couldn't allow me to reopen it, so I created a new PR https://github.com/zephyrproject-rtos/zephyr/pull/51638 to replace this.


New urlMatcher param type : optionalint

Allows state to be defined with an optional parameter that is returned
as int if specified.
Fixes #1920

You're complecting generalized behavior with a data type.

I'm sorry, I don't understand your comment, I've fixed my dev and now it works for me but I'll make it "ok for integration" if I know what I have to do

I've found a cleaner solution and explained it in #1920


LXD failed to start daemon : unable to load network lxdbr0 with converting NULL to string error

Required information

Distribution: Ubuntu
Distribution version: 18.04.6
The output of "lxc info" or if that fails: Fails with error: Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory

Kernel version: 4.15.0-159-generic
LXC version:
LXD version: 4.19
Storage backend in use: ZFS

Issue description
I was trying to list containers with lxc list when the unix.socket error appeared. I tried to restart the lxd daemon using systemctl restart snap.lxd.daemon but after a few seconds, the daemon stops.
Information to attach

[ ] Pertinent snippet from /var/snap/lxd/common/lxd/logs/lxd.log

t=2021-10-17T03:06:54+0800 lvl=info msg="Initializing networks"
t=2021-10-17T03:06:54+0800 lvl=eror msg="Failed to start the daemon" err="Failed to load network \"lxdbr0\" in project \"default\": sql: Scan error on column index 2, name \"description\": converting NULL to string is unsupported"
t=2021-10-17T03:06:54+0800 lvl=info msg="Starting shutdown sequence" signal=interrupt

UPDATE networks SET description = "" WHERE name = "lxdbr0"

Thank you, this workaround did the trick. LXD successfully starts now.


DBZ-649 Support for heterogenous arrays

https://issues.jboss.org/browse/DBZ-649
This PR provides the following functionality

Before it encodes arrays it checks whether all elements are of the same type
If all elements are of type document it makes a union of all fields of all elements to create the schema valid for all elements. It also checks that the fields of elements do not change types accross elements
Introduces a new array encoding that encodes array as struct as is case for BSON - see array in document http://bsonspec.org/spec.html Such encoding is able to handle arbitrary array values.

@gunnarmorling The test is fixed - the schema struct should be optional. Regarding schema assertion - I just expanded what was already present. We can extract the schema assetion code from postgres module and make it shared.
Regarding the removing Envelope suffix, the explanations is here https://github.com/debezium/debezium/pull/505

Because we are ripping out part of Envelope - so I think it make sense to name the schema after the collection only.

@jpechane, having some trouble when testing DBZ-649 with Avro:
org.apache.avro.SchemaParseException: Illegal initial character: 0
    at org.apache.avro.Schema.validateName(Schema.java:1147)

I.e. we'd have to name those indexing fields differently.

@jpechane Any update on this one?

@gunnarmorling According to Avro spec _<number> seems to be vali field name so I am prefixing the indices with underscore.

Rebased and applied. Thanks, @jpechane! IIUC, you said you're working on a doc PR already, right?
Btw. in the course of testing, I ran into one more related issue: DBZ-710.


feat!(share/p2p): extend proto with ignoreMaxNamespace field

Overview
Closes #2427
Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

https://github.com/celestiaorg/celestia-node/issues/2427#issuecomment-1630265675

Codecov Report

Merging #2450 (4dad051) into main (beaf6db) will decrease coverage by 0.24%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2450      +/-   ##
==========================================
- Coverage   53.28%   53.04%   -0.24%
==========================================
  Files         156      156
  Lines        9915     9908       -7
==========================================
- Hits         5283     5256      -27
- Misses       4179     4196      +17
- Partials      453      456       +3

Impacted Files
Coverage Δ

share/p2p/shrexnd/client.go
45.52% <0.00%> (ø)

share/p2p/shrexnd/params.go
12.50% <ø> (ø)

share/p2p/shrexnd/server.go
52.66% <0.00%> (-0.36%)
:arrow_down:

... and 7 files with indirect coverage changes

@Wondertan , https://github.com/celestiaorg/celestia-node/issues/2427#issuecomment-1630389538

This PR should be closed as we have implemented own nmt proto definition in the nmt repo.


Implement Dark and Light Mode Toggle

Is your feature request related to a problem? Please describe.
Users desire the ability to toggle between dark and light modes within the application for enhanced readability and reduced eye strain. Currently, this feature is lacking, resulting in a less customizable and potentially discomforting user experience.
Describe the solution you'd like
Integrate a simple toggle switch within the application interface, allowing users to effortlessly switch between dark and light modes. Dark mode should feature darker color schemes for improved readability, particularly in low-light environments.
Additional context
The integration of a dark/light mode toggle aligns with modern user preferences and usability standards. Providing this feature enhances user satisfaction and improves overall usability. Additionally, it can be implemented seamlessly within the existing application interface, adding value to the user experience without significant disruption.

pls assign the label,
i would like to work on it, pls assign this task to me

@thevijayshankersharma I will Implement it using react, tailwind CSS where there is a toggle button for changing mode by making a separate file where I will write code and then using tailwind implement it in whole project. Kindly assign this task to me

@thevijayshankersharma @8deepakkumar8 @Isha1233  this issue has been resolved


Investigate startup perf regression in #performance bot

Refs https://github.com/microsoft/vscode/issues/97638#issuecomment-649495641
@jrieken lets start with tracing chrome and our own js code in the main process to eliminate any app specific issues.

Chrome startup trace - code-insiders <path-to-vscode> <path-to-vscode-package.json> --trace-startup --trace-startup-duration=20 , this will run the trace for 20seconds and will generate a file chrometrace.log. Perform this for both code and code-insiders.

App main process JS profiling - code-insiders <path-to-vscode> <path-to-vscode-package.json> --inspect-brk and visit chrome://inspect > open devtools for node, this will automatically attach the debugger. take cpu profile until the app finishes the startup loading. Attach the logs from both code and code-insiders

If both these logs can't identify the problem then we can go down a more sophisticated yet effective ETW tracing.

There you go. This is today insider which running on electron 7 and last insiders using 8

b16.zip (electron 7.x)
792.zip (electron 8.x)

Some observations from the profiles,
Electron 8 :

I couldn't find a equivalent notifyReady call in Electron 7 but for consistency the crash reporter in the render gets initialized via ipc to the main just before the workbench is finished, comparing the timings

Electron 7 :

Comparing the readiness of shared process
Electron 8 :

Electron 7

:

Electron 8 numbers don't reflect the actual poor startup numbers we are seeing on the performance channel, @jrieken the test seems to be done on a fresh user data directory, should we try checking numbers from the user data directory of the perf bot ?

@jrieken the test seems to be done on a fresh user data directory,

Unsure about that, maybe some random changes that we also sometimes see with the bot. Or the fact that this is run manually, e.g with screen on and not triggered by the task scheduler? Also, do we have a clue how the user data dir would impact this? This would be bad because it affects all users, right?

Unsure about that, maybe some random changes that we also sometimes see with the bot.

The tests are done on the user data dir C:\Users\Administrator\Desktop\electron8-slow\data-dir , is this one the bot always uses ?

Also, do we have a clue how the user data dir would impact this? This would be bad because it affects all users, right?

this is unlikely the issue here, chromium is pretty cautious about blocking calls that can happen on the UI thread during startup, most of the file IO operations will be delegated as tasks to a threadpool https://chromium.googlesource.com/chromium/src.git/+/master/docs/threading_and_tasks.md. If something weird goes on here the chrome tracing profile data will reveal it. I don't see such a case with the provided traces.

e.g with screen on and not triggered by the task scheduler?

this is an interesting one, can you share the task scheduler script used, I would like to test on my machine.

@bpasero shared the task scheduler script on slack, will play with it tomorrow and get back.

Or the fact that this is run manually, e.g with screen on and not triggered by the task scheduler?

Imported the scheduler script, only adjustment was the trigger action being set to workstation lock. The final numbers were equally good as the numbers seen previously on my machine.
@jrieken can you get a chrome trace from running the perf script via the task scheduler, also lets try using the default data dir used by the perf bot
Perf Canary.zip

Not sure if this helps, but I just noticed a new electron trace event category in the release notes for 8.4.0: https://github.com/electron/electron/releases/tag/v8.4.0

@bpasero the screenshots with category electron I posted in https://github.com/microsoft/vscode/issues/101083#issuecomment-654507246 are based on those events.

@deepak1556 The zip are traces for 1eda45d590d0bc9e8db4ac610705f9b804b2c9df (exploration). One run triggered manual, e.g starting the perf-script from powershell, and once triggered by task scheduler - which then doesn't render a UX. That might help drill into this
traces.zip

Thanks for the traces,
with Electron 8 there are repeated blocks of RenderThreadImpl::EstablishGpuChannelSync calls blocking the renderer executions, which mostly means the GPU process has been crashing constantly on the system and the renderer is failing to establish a channel.
Electron 8:

Electron 7:

I am not sure if this is a bug in chromium trying to connect with so many retries, I was under the impression it would disable gpu after certain failed tries but doesn't seem to be.
@jrieken as a next step,

can you get the output of --verbose, most likely I will need to provide a debug build to get more useful logs from GPU
run the perf bot with --disable-gpu
Get a trace with --disable-gpu

Closing this as we know that this is happening only in the "run vscode without UX" case. The underlying cause is unknown tho

I would like to investigate the underlying issue, certainly seems like a bug in chromium or if its the predefined behavior would be good to understand why, can we keep this open for debt work ?


Fix exception if no code is selected to inspect

Fixes #598

Let's ship a patch release! :shipit:


App cannot be deployed on device when NS is built as a dynamic framework

Xcode targets for sample and TestRunner apps cannot be deployed on device when NativeScript is built as a dynamic framework.

Fixed by #953


Check-in missing helper.c.

File was grabbed from latest sdist on Hackage.

This seems necessary. Ping.


change shell var MINION_SCOPES to user setable

@mbforbes @roberthbailey The scope of this grew slightly because we cannot pass a bash array as an environment variable, so I had to refactor and change MINION_SCOPES to a string for GCE and GKE. So actually MINION_SCOPES was not even customizable in GCE previously. So this may mean this feature needs to wait until post 1.0, but hopefully it is helpful at some point. Until it is merged, I will probably maintain my own branch.
Because bash arrays may not be environment variables
(see: https://stackoverflow.com/questions/5564418/exporting-an-array-in-bash-script)
we have to change the MINION_SCOPES array to a string

prefers aliases instead of full URLs for scopes

tested under GKE, needs GCE testing
Close #10458

ok to test

The change LGTM. One suggestion: since it wasn't configurable except via file before and is now, this is a pseudo-breaking change and would let us rename to NODE_SCOPES (we might as well get rid of the word minion).
I think it's as good a time as any to make the change, FWIW.

+1 for renaming MINION_SCOPES to NODE_SCOPES.

Change LGTM. Will add to the 1.0-candidate list for triage.

@zmerlynn @roberthbailey Thanks. I had previously considered that name change but afaik the name issue isn't quite decided so I left it as is. I defer to you Googlers; I have no problem adding a commit to change the name if that's where kubernetes is being piloted.
https://github.com/GoogleCloudPlatform/kubernetes/issues/1111
https://github.com/GoogleCloudPlatform/kubernetes/issues/10284#issuecomment-115148372

@chrishiestand We discussed today and decided to hold this feature for post-1.0 (which is actually pretty soon at this point). Hopefully it isn't too much trouble for you to maintain it in your own branch for the next couple of weeks. Thanks.

@roberthbailey Yup, that better keeps in the spirit of the feature freeze. Thanks, I'm sure it won't be much trouble at all to wait a short period of time.

Thanks for understanding. Please ping after the 1.0 release is cut if you notice that this hasn't been merged.

Since it's post 1.0 release I figured I'd ping on this. Does it need a fresh rebase against master?

Yes, this needs a rebase!

rebased

LGTM.

@k8s-bot ok to test


Cow: Watch keyofcreation.com

Cow requests the watch of the watch_keyword keyofcreation\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Approved by VLAZ in Charcoal HQ


Do not set cranelift feature by default for integration testing

Singlepass now runs on 64 bit Windows, such that non-singlepass use cases became rare enough to ~ignore~ stop optimizing for them.
Closes #649

We'll get back to this once there is singlepass support for Windows


unlink() calls fail in BaseFileHelper.php on Windows hosted Nanobox environment

What steps will reproduce the problem?
On my Nanobox environment which is running in VirtualBox but ultimately is hosted on a Windows host, unlink() related calls are failing within: https://github.com/yiisoft/yii2/blob/master/framework/helpers/BaseFileHelper.php#L406-L408. This is happening within Craft CMS 3 in several file situations, reported here:

https://github.com/craftcms/cms/issues/3990
https://github.com/craftcms/cms/issues/4031

I am using the WINDOWS_FS environment variable because the file share Nanobox points to is a Windows FS and is required otherwise I get various file mutex issues with file locking.
If I remove the unlink() call in 406-408, everything works. If I also change the line to:
return @unlink($path);

This also allows it to work, but it seems it's more of a hack. I don't know if the problem is due to the fact Yii is essentially running within a Unix-like system but the file share Nanobox points to is a Windows filesystem, so it's a bit of a mix match.
What is the expected result?
File delete operation should be successful and removed without any error.
What do you get instead?
PHP exception with unlink(). Text file is busy/permission denied depending on file operation performed.
Additional info

Q
A

Yii version
2.0.16

PHP version
7.1

Operating system
Windows 10 64-bit

Looks like the same story as https://github.com/yiisoft/yii2/issues/16603.
Related discussion: https://github.com/yiisoft/yii2/pull/16604#issuecomment-410453360
Can you give some context what you're doing with this helper?

I'm not directly doing anything with it. It's what Craft CMS is using. Generally the file operations that fail are removing temp files generated from various operations, because they are being tried with unlink() condition highlighted. However for a Windows FS this condition doesn't look like it should be tried in this way, but the fact that the $isWindows test, won't actually be true in my case, because the PHP directory separator will return / (as Nanobox is running in a Linux VM) but the app path is being hosted on a Windows FS mountpoint.
Ultimately, I think it all falls down to how Nanobox is mounting the app path:
https://github.com/nanobox-io/nanobox/issues/462
I cannot get the netfs option to work, so I have to fallback to native. If the mountpoint was netfs, things would probably work a lot better, but as of now I have never got Nanobox to mount properly with it. Despite looking at all available resources and configurations that say this "works".
I guess this is kind of an edge case given the mixture of a Linux VM running on a Windows FS.

I'm not directly doing anything with it. It's what Craft CMS is using.

Then what Craft is doing with it? Can you share stack trace of this error?

Sure, here's an example exception thrown:
2019-03-13 13:12:00 [-][43903][51b97db60ebb60aebd1239f015978078][error][yii\base\ErrorException:2] yii\base\ErrorException: unlink(/app/storage/runtime/temp/example.jpg): Text file busy in /app/vendor/yiisoft/yii2/helpers/BaseFileHelper.php:407
Stack trace:
#0 /app/vendor/yiisoft/yii2/helpers/BaseFileHelper.php(407): ::unlink()
#1 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(812): yii\helpers\BaseFileHelper::unlink()
#2 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): craft\controllers\AssetsController->actionDownloadAsset()
#3 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): ::call_user_func_array:{/app/vendor/yiisoft/yii2/base/InlineAction.php:57}()
#4 /app/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams()
#5 /app/vendor/craftcms/cms/src/web/Controller.php(109): craft\controllers\AssetsController->runAction()
#6 /app/vendor/yiisoft/yii2/base/Module.php(528): craft\controllers\AssetsController->runAction()
#7 /app/vendor/craftcms/cms/src/web/Application.php(297): craft\web\Application->runAction()
#8 /app/vendor/craftcms/cms/src/web/Application.php(561): craft\web\Application->runAction()
#9 /app/vendor/craftcms/cms/src/web/Application.php(281): craft\web\Application->_processActionRequest()
#10 /app/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest()
#11 /app/web/index.php(21): craft\web\Application->run()
#12 {main}

A file is written to the temp download folder, but unlink fails causing an exception.

https://stackoverflow.com/questions/16764946/what-generates-the-text-file-busy-message-in-unix
https://stackoverflow.com/questions/41251652/php-cant-unlink-file-after-fclose

Thanks. I understand why the error can be thrown, but I believe the discussion referenced here: https://github.com/yiisoft/yii2/pull/16604#issuecomment-410453360 about the whole $isWindows condition is something this relates to.
Currently in the BaseFileHelper.php, the condition is being done on PHP's directory separator value by the looks of it. Given this is a Unix-like VM ultimately reading/writing to a Windows FS, this condition will never be true, because based on that condition, it isn't Windows, so unlink() is tried.
https://github.com/yiisoft/yii2/blob/master/framework/helpers/BaseFileHelper.php#L404-L408
However, given I am telling the file mutex that this is Windows system with an env var, surely, the same logic should be used across all the areas. @brandonkelly, discussed this in is proposal, given $isWindows = DIRECTORY_SEPARATOR === '\\'; is used elsewhere, but in this case this is what's causing the problem. My PHP directory separator won't look like Windows, because it isn't, only the underlying FS which the Nanobox environment points to is, if that makes sense.

I have managed to get a project running under the Nanobox netfs mount type which pretty much resolves all the issues highlighted. It is ultimately a underlying FS problem in the way the app directory is mounted and performs various actions, it would seem file locking and other areas are pretty sensitive when using the VirtualBox driver.
However, I would still suggest there is an argument to make the $isWindows condition the same across the various areas and not using the PHP directory separator as the condition. If the isWindows option is being used in the file mutex, shouldn't this be the flag in other places as discussed previously?

Probably but since that's quite rare use case, I don't think we'll get to implementing it anytime soon.

Fair enough. At least it's documented if someone else comes across this!


Adding an adapter for formality transfer with GYAFC

Trained using the english GYAFC formality dataset, for zero-shot multilingual classification (tested with hindi so far). Best results after lower-casing, detokenizing and removing trailing punctuation from input. To be used with a language adapter like model.active_adapters = Stack("en", "gyafc").

Hi @calpt, just pushed some changes adding task and subtask files.


How to handle significant whitespace?

Hi! I'm interested in exploring how to use this parser system for a language that uses indentation to indicate scoping - which, notably, is not something a raw LR parser (or any strict context-free parser) can handle.
Normally, this would be handled in the lexer by consuming all the space/tab indentation in the lexer and emitting INDENT/DEDENT tokens for when the indentation is increased/decreased, respectively.
I'm puzzled as to how this works in the example py.lang grammar? The only mention of whitespace there seems to be immediately discarding it. I also don't see any mention in the technical paper on arxiv as to how this is handled.

The relevant directive is ws_sig, as in:
    mode body ws_sig {

This will tell langcc that, even though the lexer may pass over whitespace, it should still have special machinery for converting the passed-over whitespace into newline/indent/dedent tokens. Please note that the whitespace machinery will have generally "Python-like" characteristics: e.g., backslashes at the end of lines will cause line continuation; and whitespace characters that occur between matching pairs of brackets ([], {}, ()) will be exempted.
Also, I have just pushed some changes that should significantly improve whitespace handling, so if you are working with significant whitespace, be sure to pull a fresh copy.

Aha! That explains it! Do you have any plans to allow customizing this behavior in the future?

No plans as of yet, but it's not out of the question. How specifically would you want to customize it?

The haskell family of languages use a somewhat different version of whitespace-significant indentation than python - in particular, where the column of the first char of an "inner" line must be greater than the column of the "head" token on the "parent" line - and the "inner" line is considered a child of the inner-post parent for which this rule holds.
This has two practical effects on the look/feel of the code:

indentation doesn't follow "tab stops" - but is specified precisely to the column
different amounts of indenting on the next line can lead to very different meanings, by parenting that line to different parts of the former line

I'm afraid this isn't quite enough detail for me to pinpoint how your desired behavior differs from the "Python-like" interpretation. Could you specify precisely when Newline, Indent, and Dedent tokens should be emitted (and how many), in your desired behavior? Also, would \ still serve as a line continuation, and would the interior of matching braces ([], {}, ()) still be exempt from whitespace processing?

I will think about this, though it may not be something I can get to in the near future. One problem is that we still want to have the declarative, regex-based tokens spec in order to compile to an efficient DFA implementation for the lexer. So whatever Turing-complete language the lexer uses, that language should be restricted to the "action" part of the lexer mode definition, with the tokens still matched by regex.

Please note that the whitespace machinery will have generally "Python-like" characteristics: e.g., backslashes at the end of lines will cause line continuation; and whitespace characters that occur between matching pairs of brackets ([], {}, ()) will be exempted.

How would we override this for grammars which have non pythonic but still significant whitespace?

How would we override this for grammars which have non pythonic but still significant whitespace?

It depends on the nature of the significant whitespace. If, like Python, it is still based on inserting Newline/Indent/Dedent tokens at appropriate points, then it may not be too difficult to modify the code to support it. If it is not of that form, it would probably be much more difficult. Fundamentally, the DFA-based lexer spec language of langcc is not designed for arbitrary significant whitespace, and so any solution is going to be somewhat hacky.

In my case I was hoping to have YAML-ish map declaration but delimited by {}, so newline/indent/dedent would definitely be sufficient for that

Can you say a bit more about the desired behavior in this case? How does it differ from the current behavior with ws_sig?

My goal was to parse something like:
obj = {
  # fields obj.a, obj.b
  a: 0
  b: 1
  c:
    # nested fields obj.c.regular, obj.c.plusplus
    regular: 2
    plusplus: 9000
}

but if whitespace characters that occur between matching pairs of brackets ([], {}, ()) will be exempted then this would not be possible since the indents I'm looking at lie between a matching pair of {}.

I am not quite sure I understand your example language. If the inner object is also a dictionary, then, to be consistent, shouldn't it also have braces?
obj = {
  a: 0
  b: 1
  c: {
    regular: 2
    plusplus: 9000
  }
}

In any event, I have just pushed some changes that should allow customization of the delimiters in ws_sig mode (see grammars/py.lang, grammars/test/ws_sig_02_brackets.lang for examples) -- let me know if that works for your use case.


Allow to return from actionCreator without action

Curious, what is the use case? Usually you'd want to exit early when some state-dependent condition is true, but to read the state, you'd need to use the callback form, from which you can already exit early.

Yes, I'm already understood that.
I receive problem when started new action with console.log and no return statement.

Could be closed


Update Twitter Icon to X Logo Across Bootstrap Website

Description:
The current Twitter icon used on the website should be updated to reflect the new X branding. The icon appears outdated and should be replaced with the X logo to stay consistent with Twitter's rebranding.
Steps to Reproduce:

Visit https://getbootstrap.com/
Locate the Twitter icon in the header media section.
Observe that the Twitter icon is still being used instead of the X icon.

Expected Behavior:
The Twitter icon should be updated to the X icon to match the latest branding.
Actual Behavior:
The Twitter icon is currently being displayed instead of the X icon.
Screenshots:

Proposed Solution:
Update the icon to reflect Twitter's new X branding by replacing the existing Twitter icon with the X icon.

i would like to work on this issue!!

great. i will try


[Automated] Merge patch branch with the master

[Automated] Daily syncing of patch branch content with the master

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.9 out of 10 committers have signed the CLA.:white_check_mark: mindula:white_check_mark: malinthar:white_check_mark: MaryamZi:white_check_mark: DamithaSenevirathne:white_check_mark: dulajdilshan:white_check_mark: SumayyaZiyad:white_check_mark: tharindulak:white_check_mark: SasinduDilshara:white_check_mark: IMS94:x: Sumayya ZiyadSumayya Ziyad seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


improvement: exception handler should respond JSON on API endpoint

In Laravel ExceptionHandler, the function shouldReturnJson() determines whether generate JSON response by Accept header.
But, sometimes the client may forget add Accept: application/json, if the request is successful, server responds JSON normally, But if the request is failed (4xx, 5xx), the server responds a webpage.
Obviously, when the endpoint is an API endpoint, to respond a webpage doesn't make sense.
So, just check current route is an API endpoint and respond JSON.

Laravel 11.x provides Rendering Exceptions as JSON

Possible solutions:

Laravel 11.x Rendering Exceptions as JSON
this issue won't fix on laravel-restful-utils, downstream developer must update bootstrap/app.php for each laravel project

Override shouldReturnJson() in src/Exceptions/Handler.php
detect the current route's middleware group's name "api". Downstream developer just update their laravel-restful-utils


Feature suggestion: Allow entering URL of songs/albums/playlists somewhere

One of the best things about GPM is the ability to link songs or albums or playlists directly to friends, for sharing music. However I can't find any way to enter a URL I have been linked in the desktop player.
My suggestion (if I haven't just missed how to do this) is to load the page for URLs pasted into the search bar, or have some sort of popup that you can just paste URLs into.
Thanks!

+1

In the next release there will be a menu option and a shortcut to do this.
"Control / Command + Shift + G"

Hey, excellent news!
When you say menu option, does that mean like an option to change the keyboard shortcut? Ctrl+Shift+G is a bit odd, I love would to be able to rebind it to a more standard shortcut like Alt+D or Ctrl+L
Thanks a bunch for implementing this so quickly!

Ctrl+Shift+G is the stand "go to" shortcut.  (At least that's what my experience has been).  Neither "Alt+D" or "Ctrl+L" make any sense to me?  Accelerators (shortcuts) make more sense to users if the letter they are pushing is relevant to the command.  So Goto uses G


panel 16*32 and example P6_32x16_1_4_ScanPanel.ino

Just wondering..can this QuarterScanMatrixPanel be configured to be used with pointers.
I defined my whole sketch with matric pointer.. but in the example it shows a direct approach.
My definition:
// placeholder for the matrix object
MatrixPanel_I2S_DMA *dma_display = nullptr;

// object
dma_display = new MatrixPanel_I2S_DMA(mxconfig);

Now i can code the output to the display like this:
dma_display->drawPixelRGB888(x,y,0,0,0);
however, with the Quarterscanmatrixpanel class i cannot define it with pointer like i did, or can I?

You'll need to refactor the example as it was written before the library implemented a pointer based approach as well. Feel free to raise a pull request.
Should be as simple as replacing the .'s with ->'s in most cases.

replacing the .'s with ->'s in most cases.  that part i understand. However, those are a lot of changes to my code for only one type of panel...to keep it universal for all panels....i would like to keep using that pointer. That way all i need to add is something in the trend of
#ifdef 32x16
// setup QuarterScanMatrixPanel
..
..
#endif

Maybe there is someone who can help me with rewriting that QuarterScanMatrixPanel function?
How about you leave this topic open for a few days to see if anyone offers some help and if not...close it as "not an issue"

who can help me with rewriting that QuarterScanMatrixPanel function?

No need, you can use it as-as.
SOmething like this:
MatrixPanel_I2S_DMA *dmaOutput = nullptr;
QuarterScanMatrixPanel * display = nullptr;
void setup() {
dmaOutput = new MatrixPanel_I2S_DMA();
display = new QuarterScanMatrixPanel(*dmaOutput);
}
void loop() {
  display->fillScreen(display->color444(0, 0, 0));
}


Enable grid prefixing via autoprefixer options object

To verify grid prefixing I swapped out flexbox for grid in App.css.
yarn build with no autoprefixer changes:

yarn build with autoprefixer grid option enabled:

This fixes #4017

@ai does this have negative implications other than it doesn't provide full support in older browsers?
i.e. will this not break support in browsers that support grid? is it safe to turn on for everybody?

This option should not be enabled by default. Developer must understand limits before using it.
I suggest to add postcss.config.js support. In this case developers will be able to redefine CRA PostCSS’s plugins and plugin's options. It will be there best way to enable Grid.

I think it's reasonable to leave it on here. In our case, the user doesn't set up autoprefixing themselves. If they were, it makes sense for the option to be off because Autoprefixer is expected to be correct.
However in our case autoprefixing happens "behind the curtains". So you don't expect it to work in 100% of cases because you don't think about it that much in the first place. Therefore, I think it makes sense that we do best effort to support IE here. Worst case — your layout is just as broken as it would be if we left it off. You'd have to test it in IE regardless.
I agree it would be bad for Autoprefixer to make it on by default because it would hurt its brand as a "correct" tool. But in our case I think it's okay because I can't see the harmful scenario it attempts to prevent. We can add it to User Guide in any case.

It could a another solution. We can add /* autoprefixer grid: on */ control comments.

Interesting. That would be good I think. We could document that and leave it off then.

@Timer it could not automagically apply to any CSS. CSS should be written with limits in mind.
Here is a good guide about the limits:
https://css-tricks.com/css-grid-in-ie-css-grid-and-the-new-autoprefixer/
This is why calling grid: true on some node_modules code is not the best idea.
Also, it is a popular way to exclude node_modules from Babel and PostCSS.
But I agree that we may have a cases when we need to run PostCSS and Babel on npm packages. But I do not see a way to resolve it without adding postcss.config.js support.

Will grid prefixes get stripped if imported when we have the flag set to false? This sounds like unacceptable behavior to me for node_modules.

Nope. Prefixes will not be stripped, if IE 11 is in Browserslist.

OK, we won't merge this and will document /* autoprefixer grid: on */ instead. Is there an estimated timeline on this feature?

Yeap. I added the task to our Cult of Martians https://cultofmartians.com/tasks/autoprefixer-grid-comment.html
I think it will be done in next week (maybe two).

/* autoprefixer grid: on */ support was added to Autoprefixer https://github.com/postcss/autoprefixer/pull/1138


help Merge transformer with other layers

Hi,
I would like to create a new model that uses the transformer encoder and decoder. However, it also needs to include layers in between and the decoder needs to use not only the output of the transformer but also the output of the other layers. Do you have some advice or guide on the best way to do this? Perhaps generate a new T2T model? also, it's not clear for me where would I define the graph of the model?
Thank you,
Daniela

Yes, create your own subclass of T2TModel. See the definition of Transformer as a guide. You typically only need to override body.

Hi rsepassi,
Thank you for your answer. I created a new T2TModel subclass following the definition of Transformer and decorate it with registry.register_model. I imported it in init.py and  I try to run it using the --t2t_usr_dir flag. In my user dir I have the following files:
queries.py (problem definition)
query_transformer.py (submodel)
__init__.py (with from . import queries and from . import query_transformer)
However, I'm getting the following error: LookupError: Model query_transformer never registered.  Available models.
Do you happen to know the common errors I could be making or do you have any suggestion of a T2TModel class besides Transformer I could use as guide for my problem?
Thank you.

please see the example_usr_dir in tensor2tensor/test_data and try to match
its general structure
On Thu, May 10, 2018 at 12:39 PM Daniela Solís<EMAIL_ADDRESS>wrote:

Hi rsepassi,
Thank you for your answer. I created a new T2TModel subclass following the
definition of Transformer and decorate it with registry.register_model. I
imported it in init.py and I try to run it using the --t2t_usr_dir
flag. In my user dir I have the following files:
queries.py (problem definition)
query_transformer.py (submodel)
init.py (with from . import queries and from . import
query_transformer)
However, I'm getting the following error: LookupError: Model
query_transformer never registered. Available models.
Do you happen to know the common errors I could be making or do you have
any suggestion of a T2TModel class besides Transformer I could use as guide
for my problem?
Thank you.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensor2tensor/issues/762#issuecomment-388162544,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEGW49_6ziPEk2Mx1ezE1Uk6y4oj_i1ks5txJeJgaJpZM4TvEfE
.


GH_TOKEN not set

Builds are failing https://travis-ci.org/jupyterlab/jupyterlab_app/jobs/260434845
#87 was merged without setting GH_TOKEN
@lucbouchard1

Helpful links:
https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
https://docs.travis-ci.com/user/encryption-keys/

Fixed. First release will be published soon.


How activate the IMU frame on Lidar MRS1000

I am working on integrating the MRS1000 LiDAR with ROS2 Humble, and in addition to point clouds, I also want to utilize the IMU. With the correct configuration, I have the topics for both point clouds and IMU data.
My question concerns the frames for these two types of data. The point clouds are published on the 'cloud' frame, and the IMU data is on the 'imu_link' frame (as indicated in the topics). How can I activate or create the frame for the IMU, because when I visualize the frames, there is nothing displayed for 'imu_link'?

Thanks for your feedback. Point cloud messages of type sensor_msgs/PointCloud2 are published on topic "/cloud" with frame id "cloud". IMU messages of type sensor_msgs/Imu are published on topic "/sick_mrs_1xxx/imu" with frame id "imu_link". You can view IMU messages with rostopic echo -p /sick_mrs_1xxx/imu or visualize them with rviz by adding topic "/sick_mrs_1xxx/imu" and setting option "Fixed Frame" to "imu_link". Use a static_transform_publisher like rosrun tf static_transform_publisher 0 0 0 0 0 0 base_frame imu_link 100 to link imu frames to another base frame.

ok i see.
Thanks


Update NewServer call

#86  👈
feature/proof-of-stake

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @lazynina and the rest of your teammates on  Graphite


Support for non-python implementations

Hi, I have a question about using this framework w/ a non-python ANN implementation.
It looks like this is mostly a fork of ann-benchmarks. So the only option for using outside of Python is to hack together a client/server setup, as has been done for a few algos in ann-benchmarks. This obviously handicaps and complicates non-python implementations, as it introduces costs of context-switching, serialization, and data transfer among processes.
I asked and was told early on by project organizers that the big-ann challenge would support non-python implementations:

So I'm wondering if there has been progress here, or any idea of how it might work?
It seems like it wouldn't be terribly difficult to refactor the code so that the containers executed by runner.py can have any entrypoint, e.g., a program in another language. The interface between runner and algorithm would then simply be some standard file format for inputs and nearest neighbor results. If that sounds like a good idea I can try to implement it. Otherwise maybe we can use this ticket for discussing alternatives.
Thanks
-Alex

Alex,
Thanks for bringing up this issue again here.
One of the primary advantages to the evaluation framework is that it
standardizes and normalizes the process by which all participants are
measured.  As an evaluator for the challenge, it certainly makes my job
easier compared to a situation where each participant submitted a different
code-base.
That said, I understand your concern about the "overhead" of merging
different language run-times.
Do you have an estimate of the percentage of this "overhead" compared to
the latency of a large batch query search set?  For example, the query set
for Deep-1B has 10,000 vectors.
If we could figure out a way to account for this "overhead," would that
alleviate your concerns about this?
On Wed, Aug 18, 2021 at 6:51 AM Alex Klibisz @.***>
wrote:

Hi, I have a question about using this framework w/ a non-python ANN
implementation.
It looks like this is mostly a fork of ann-benchmarks. So the only option
for using outside of Python is to hack together a client/server setup, as
has been done for a few algos in ann-benchmarks. This obviously handicaps
and complicates non-python implementations, as it introduces costs of
context-switching, serialization, and data transfer among processes.
I asked and was told early on by project organizers that the big-ann
challenge would support non-python implementations:
[image: image]
https://user-images.githubusercontent.com/8015228/129908445-a5515724-0e5c-4b63-b81a-27e63c50b3a9.png
So I'm wondering if there has been progress here, or any idea of how it
might work?
It seems like it wouldn't be terribly difficult to refactor the code so
that the containers executed by runner.py can have any entrypoint, e.g., a
program in another language. If that sounds like a good idea I can try to
implement it. Otherwise maybe we can use this ticket for discussing
alternatives.
Thanks
-Alex
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/harsha-simhadri/big-ann-benchmarks/issues/20, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADL6CKR7UJ2SBS2B23B223T5O3FRANCNFSM5CMDBVAQ
.

Hi George, thanks for the quick response.

One of the primary advantages to the evaluation framework is that it
standardizes and normalizes the process by which all participants are
measured.  As an evaluator for the challenge, it certainly makes my job
easier compared to a situation where each participant submitted a different
code-base.

I understand the concern in adding complexity.
To clarify a bit, I'm proposing two changes:

instead of implementing a specific Python class/interface, algorithms implement a specific CLI, which could even be identical to the Python CLI
instead of keeping inputs/outputs (vectors/neighbors) in memory in the Python runtime, the CLI reads/writes the vectors/neighbors using the file format already used for datasets in the big-ann challenge.

The CLI still runs in a docker container, so we can just as precisely measure the container running time, the CPU/mem used, etc.
The Dockerfile installs an artifact containing the CLI, fundamentally no different than the way the Dockerfiles currently install a Python library. If anything, this actually removes a fair amount of code and complexity from the benchmarks repo.

Do you have an estimate of the percentage of this "overhead" compared to
the latency of a large batch query search set?

I'll re-run the ann-benchmarks this weekend and get back to you with a number.

If we could figure out a way to account for this "overhead," would that
alleviate your concerns about this?

Maybe, but it's still a more complex implementation. I'll have a better answer once I re-run those benchmarks.

Thanks for the suggestion. I don't know why you would need some client/server setup instead of just using subprocess calls to support any cli language?
I am happy with providing the filenames of the groundtruth files to the search operations such that you can write a very simple wrapper that just uses subprocess calls. I think this is what Matthijs' comment referred to.  You could write the results to disk (or keep them in memory and pipe), and then transform them in the get_results methods to the format expected by the framework. If the overhead of doing that is too large, we could split up the search phase into "prepare" and "search" so that we minimize the overhead.
I am very concerned about putting all the effort of file management/file preparation/result file writing onto the client side. Morevoer, once we go done this path, all implementations will become black boxes that will be difficult to check (and re-use for others).

I'm confused about how subprocess calls would work. I don't want to start a new instance of some program for every vector. That would be even more wasteful than a client/server setup. I want to boot up a program that reads vectors from a file in the standard competition format and writes results to a file in the standard competition setup.

I made a demo PR over in https://github.com/alexklibisz/ann-benchmarks/pull/1 to demonstrate the overhead of a client/server model.
Copying from the PR:
To summarize: the total overhead for a model w/ recall@100 = 0.91 on the Fashion Mnist dataset is 40%.
Unless I've made some error (please check my math and details in the PR), I think this result shows there is clear value in allowing language-specific benchmark implementations.

Sorry for the late response @alexklibisz, I've been on vacation.

The dockerfile for the non-python algo installs an executable and sets it as the docker entrypoint. The executable should implement the same CLI as the one in the run_from_cmdline method in runner.py.

The runner program starts the container with the standard CLI args.

The executable basically re-implements the logic from the run method from runner.py and produces an hdf5 file with the same properties.

Sure, that would work after changing some of the command line arguments, I'm still not sure that it's worth replicating all the logic.
What I meant above is that the python runner and your implementation just communicate via stdin/stdout; from the python runner's side (>>> stdout, <<< stdin) this could look like this:
>>> BUILD_INDEX <datasetname> <path_to_dataset> # build index, if necessary
<<< DONE BUILD_INDEX
>>> LOAD_INDEX  <datasetname> <path_to_dataset> # load index
<<< DONE LOAD_INDEX
>>> PREPARE <path_to_queries> # load query set from file [maybe unfair advantage over the actual wrappers because they usually copy code as well]
<<< DONE PREPARE
5 times do:
>>> QUERY
<<< DONE QUERY # measure running time
Take best running time, or median,
>>> LIST RESULTS
<<< 1 neighbor_1, neighbor_2, ....
<<< 2 neighbor_1, neighbor_2, ...

Your process is run exactly once.
I could implement a driver for this next week.

Hmm. Thanks for explaining it. I don't think I see how stdin/stdout is better than having a CLI that mimics the python runner and runs end-to-end. Managing a stdin/stdout event loop makes the program more complicated compared to just parsing CLI params and running. The container overall is more complicated because it requires coordinating two processes. And it could still cheat. For example, it could go ahead and start running queries before it emits "DONE PREPARE". It's a tricky problem, but I'm still thinking a CLI-based abstraction is simpler to implement and requires the same trust (via code review) model.
So far I've been able to make some progress on a branch that just mimics the CLI and runs all the way through in ann-benchmarks (i.e., Erik's project). The trickiest part was honestly just getting HDF5 to work from Java. Once that's stable I'll try the same pattern on the big-ann-benchmarks, which luckily have a much easier file format to work with from the JVM.

I think that it would be much easier to write and verify wrappers with the stdin/stdout approach. (I expect the wrapper will be smaller than your efforts to write the hdf5 file correctly, without replicating any of the other necessary logic.) I'll happily review your code and write another wrapper that follows my suggestion once you are happy with yours.
Also ann-benchmarks has support for what I outlined above, but it's a bit more difficult to implement because of the file reading. It's documented here https://github.com/erikbern/ann-benchmarks/tree/master/protocol and used, for example, here https://github.com/maumueller/mih.

You're right that it's quite difficult to replicate the runner and get the details correct. It's not really a lot of code, but there's some subtlety to how the metrics are computed. I started seeing good results today after maybe ~30 hours total getting this to work the past few weeks. It does make a big difference; even without much optimization on the JVM side, so far it's ~1.5-2x faster than the client/server model.
So I'm starting to agree your idea of passing some minimal commands back and forth and reading the data from a file will be overall simpler. Or maybe the grass just looks greener now that I've tried to re-implement the runner.
The only part that seems a bit over-complicated is the last step, where it prints the results to stdout. Is there a reason the non-python process couldn't just write the results to a file?
So instead of:
>>> LIST RESULTS
<<< 1 neighbor_1, neighbor_2, ....
<<< 2 neighbor_1, neighbor_2, ...

We would do:
>>> WRITE_RESULTS <path_to_results>

Let me try to make a demo wrapper next Monday or Tuesday and then we can see how well it works.

So instead of:
>>> LIST RESULTS
<<< 1 neighbor_1, neighbor_2, ....
<<< 2 neighbor_1, neighbor_2, ...

We would do:
>>> WRITE_RESULTS <path_to_results>

I'm not sure what you mean by results here, because the whole hdf5 file will be constructed from the runner. This is really just meant to report the nearest neighbors retrieved for the queries, nothing else. I am not sure I see much of a different on your side if you somehow have to write them to a file, or just iterate through them and print them on stdout. This is not meant to be part of the timings.

I'm also working on a wrapper that uses basically this exact command/response format via JSON-over-HTTP instead of stdin/stdout. So it's not passing data around, but just commands with pointers for where to find the data on the local filesystem. It's also not trivial, but probably the least bad format I've tried so far. I started on stdin/stdout but found I was basically re-implementing an http client/server model. I've set the ann-benchmarks implementation aside for now. Hopefully will be ready for a PR early next week.

Closing this now that #35 is merged.


Github page for landing

It would be pretty good for new users - yes, os x end-users - if we have a landing page with below items.

Description for keycastr
Document link (to wiki)
Download link (to release)
Issue link (to here)
License information
Contributors and history of keycastr

Yep that. I just landed on the github project because someone I know starred it recently. And I have no clue what this does, and can't dive deep right now. A short description would be great!

We definitely need a more robust set of documentation.  We don't have a very good elevator pitch right now, and we don't have documentation for a lot of the things that people are tripping up on, like how to move the visualizers around.  Setting a 1.0 milestone - it's not done until this is done!


autolibs option does not support 3, 4

Mode 3 (packages) and 4 (enable) are not supported by this gem. https://rvm.io/rvm/autolibs

more details please

got the pull request submitted that fixes this, I believe. I didn't actually check if there were tests

It looks like there are tests, but nothing specific for autolibs

closing in favor of #108


Magnitude correction

This version has the frame time correction and the stmag/vegamag changes.
This includes issues #89, #90, and #92.

This pull request needs to be closed because I had to resync my local copy of the master to the up to date version.  Pull request #94 puts in the same changes as were originally in this request.


ci: publish amd64 and arm64 images

#713  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


starting with cartographer - provide imu

Hello,
I have problems getting cartographer to work. There are no error messages, but the robot frame (base_link) does not move in the map and only rotates in place. I have provided IMU data.
I have one lidar sensor in the front of a car and an IMU in the rear which is rotated 90° in yaw axis (see *urdf)
Now i try to get cartographer to build me a map. Will it even work in an outdoor environment?
This is the link to my modified .urdf, .lua and launch files:
Folder
I have run the rosbag_validate and it gives me the following result:
File
I cannot really say what that tells me. Is my bag okay?
I hope that you could help me. Thank you.
EDIT:
provided launchfiles (demo_honda_2d.launch, honda_2d.launch) with remapping of /adma_imu to /imu
and the corresponding configuration files.
Screenshot:

This thread has not seen activity in a while. Please reopen if you have further questions.

Hi, @cschuet @gaschler @davenass @SirVer
I met similar issue above. I have one lidar sensor and an IMU in the rear which is rotated 90° in yaw axis. However, the robot frame (base_link) does not match with map_frame and odom_frame. The detai is present in https://github.com/googlecartographer/cartographer_ros/issues/1455.
Thanks in advance.


Implement infinite bounds in Relay

The goal is treat explicitly infinite bounds in RelayProblem.
How to encode it ? How to avoid the duplication of variable ?

Towards this, a small test program would help and we can fix bugs from there.

Transfered to wish list in wiki
https://github.com/siconos/siconos/wiki/Wish-list-and-enhancement


patch executable not found when cross compiling for QNX

Build process fails when cross compiling for QNX because patch executable not found

fixed cross compiling cmake external projects by:

exporting some of the variables
adding the rest of the variables in the toolchain file
setting set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) which allows cmake to search the host file system for programs.

Now only a toolchain file is needed when cross compiling external projects.
I still need to do some testing on actual hardware. I will do that tomorrow on an aarch64 board.


A question re use in Wordpress

Not sure how or where I ask a question as opposed to raising an issue, so my apologies for probably doing it in the wrong place..
I would very much like to have this work within my Wordpress Website.
The iframe is used by our booking engine and they have everything setup on their end.. I'm reasonably confident I have things set up according to instructions here, but resizing does not seem to work.. I have asked our bookings people about this and they say they were not able to get this working in Wordpress.
page currently trying to get working http://dev.bell-lodge.nz/book-motel/
Many Thanks for any help
Rick Malkin
NZ

I don't the first thing about WordPress I'm affraid. However, this article might be of some help.
http://b-l-i-n-k.com/automatically-adjust-cross-domain-iframe-height-dynamic-content-wordpress/
If not my advise is to ask a question on StackoverFlow and give as much detail as possible.


query for relevant text snippets

Hi,
Is it possible to just get a list of closest nodes to the prompt? Like search for "green car" and just see a list of the closest matching text fragments
I see there's a function called "get_nodes_and_similarities_for_response" but not sure how it can be called when using GPTSimpleVectorIndex

Was just looking into this! I wonder whether similarity_search() from LangChain can be used here 🤔
It's useful to see the source text sometimes. Will be convenient to have this easily accessible in gpt_index
And for reference, with LangChain one way to do it is:
docsearch = FAISS.from_texts(texts, embeddings)
query = docsearch.similarity_search(<similar to text here>)
for item in query:
    print(item)

You mean only get the top k results and not the answer? Here is how I do it:
def query_index(question, index_path, top_k):
    # Load index
    index = GPTSimpleVectorIndex.load_from_disk(index_path)

    # Query index
    response = index.query(question, response_mode="no_text", similarity_top_k=top_k, verbose=True)

    # Format source text
    relevant_source_text = []
    for node in response.source_nodes:
        similarity = node.similarity
        text = node.source_text
        relevant_source_text.append({similarity, text})

    return relevant_source_text

So if you set the response_mode to "no_text" it will not use completions to answer it and you will be able to just access the top k results via response.source_nodes.

That works, thank you!

You mean only get the top k results and not the answer? Here is how I do it:
def query_index(question, index_path, top_k):
    # Load index
    index = GPTSimpleVectorIndex.load_from_disk(index_path)

    # Query index
    response = index.query(question, response_mode="no_text", similarity_top_k=top_k, verbose=True)

    # Format source text
    relevant_source_text = []
    for node in response.source_nodes:
        similarity = node.similarity
        text = node.source_text
        relevant_source_text.append({similarity, text})

    return relevant_source_text

So if you set the response_mode to "no_text" it will not use completions to answer it and you will be able to just access the top k results via response.source_nodes.

Thanks! 😃


Initializer

Is it still necessary to add an event listener for push notification in an Ember initializer if I use Cordova Shims? Also, would you mind sharing what your initializer looks like in general when building Cordova and Ember? :)

There is an example on how to use Push Notifications in the README. Creating a service in an initializer, as described there, should be sufficient. The event listeners that are documented in PushPlugin's README will be created by the service.
Check out the README and let me know if anything should be more clearly documented :)

Cool will do, thanks!


Download data buttons does not work

Right now the callback is very complicated, can all be abstracted with the df.to_csv method from pandas

dash-bootstrap-components must be added to the req of the package

The download button does now working.


custom-validators

hello,
Custom validators look great, check schemaform.io/examples/custom-validators.html.
But, the form definition in this example is javascript, not JSON.
The problem is that I store forms in database serialized as string ...
Any idea ?
Thank you !
Philippe

You can just encode the JSON into a javascript object. Actually I think this is done automatically by $http.get...

Please re-open if you do not feel the question has been answered sufficiently.


Pagination buttons and table filter datepicker popover are competing on the z-index with the selected page popping through

Package
filament/tables
Package Version
v2.8.3
Laravel Version
v8.78.1
Livewire Version
v2.5
PHP Version
PHP 8
Bug description

Just a little thing. The selected pagination page button is showing up through popovers on the filter popup.
For some reason it has z-10 class applied. Not sure if that is intentional or not, but without it works fine.

Steps to reproduce
No response
Relevant log output
No response

Closed by #1186.


js back and refresh operations

I added a flag to the TestScripter class to toggle headless in my upcoming patch, otherwise looks good.

Cool, i was confused this morning when firefox wasn't opening yet the tests were running and I didn't know how you wanted to handle that


add "meter" - a new type of metric and metric handler

motivation: create a new type of gauge that is separate from recorder in a backwards compatible way
changes:

add "meter" - a new type of metric and metric handler pair that is similar to gauge
add increment and decrement to meter
add default implementation of meter based on recorder for backwards compatibility
add tests

cc @ FranzBusch @ktoso

Lgtm :)


Some NOTIFICATION_* constants collide by sharing the same integer value

Godot version
4.0.2-stable
System information
N/A
Issue description
Godot uses NOTIFICATION_* constants across different classes to identify different types of lifecycle callbacks. Since they are defined in different classes, they are not enums, but resolve to int in GDScript. At the moment, some of these constants share the same value, although they mean different things.
Examples I've found (not necessarily exhaustive):
30:   CanvasItem.NOTIFICATION_DRAW               Node.NOTIFICATION_NODE_RECACHE_REQUESTED      Window.NOTIFICATION_VISIBILITY_CHANGED
32:   CanvasItem.NOTIFICATION_ENTER_CANVAS       Window.NOTIFICATION_THEME_CHANGED
41:   Control.NOTIFICATION_MOUSE_ENTER           Node3D.NOTIFICATION_ENTER_WORLD
42:   Control.NOTIFICATION_MOUSE_EXIT            Node3D.NOTIFICATION_EXIT_WORLD
43:   Control.NOTIFICATION_FOCUS_ENTER           Node3D.NOTIFICATION_VISIBILITY_CHANGED
44:   Control.NOTIFICATION_FOCUS_EXIT            Node3D.NOTIFICATION_LOCAL_TRANSFORM_CHANGED
50:   Container.NOTIFICATION_PRE_SORT_CHILDREN   Skeleton3D.NOTIFICATION_UPDATE_SKELETON

It can be argued that this may not be an issue as long as those collisions happen in different inheritance branches (i.e. never directly between a base and derived class).
However, the first example above even violates that: both CanvasItem and Window inherit from Node, so there are simultaneously two different notification constants of the same value in scope.
30:
  CanvasItem.NOTIFICATION_DRAW
  Node.NOTIFICATION_NODE_RECACHE_REQUESTED
  Window.NOTIFICATION_VISIBILITY_CHANGED

This can be problematic in the virtual function Object::_notification(int what), which can be overridden by the user. There is no way to distinguish such notifications at the moment.

The open question is whether only this specific bug should be fixed, or collisions should be avoided in general. Given a 32-bit-integer, it's probably reasonable to never have overlapping values; yet backwards-compatibility (mostly with GDExtension) might be a factor in the decision.
Steps to reproduce
See above.
Minimal reproduction project
N/A

NOTIFICATION_NODE_RECACHE_REQUESTED was removed, fixing the worst conflict.
The other duplications are less problematic, as they appear in disjoint inheritance branches. Is the decision to keep them as-is?

I think so. For 5.0 we should look into a system that avoids such overlaps, but for now it would be worth it IMO to break compat in those key notifications if there's no actual problem with reusing the same number in separate inheritance chains.


Saved Values Not Displaying On Initial Open

Opening a record every time requires a refresh in order to display the saved values in the picker. Either a browser refresh or Dynamics 365 'Refresh' button is required. Refreshing with an API call does not work.

Are you using the latest version?

Yes, I'm using <IP_ADDRESS>

maybe lets do an offline call, you can setup something and invite me<EMAIL_ADDRESS>


Out of memory

The firmware is stable, thank you very much! But there is enough memory to control 266 LEDs. Maybe you can simplify the code somewhere? It would be very nice to drive 300 LEDs.

I just compiled SR ESP8266 for SP511E, I didn’t write it.  The memory issue is due to the way ESP8266 manages pins and the fact that they used pin 3 as the data pin. If you switch to using pin 1 or 2, it will use less memory. To do that you’ll need to follow divsys’s guide for hacking the hardware on the WLED discourse.

Share the link for hacking the hardware on the WLED discourse, please

https://wled.discourse.group/t/sp511e-upgrade-project/3753


Windows Wavenet asnumpy Timeout

Description
Wavenet produces output that when trying to convert it to numpy fails.
I identified the pull that introduced this bug: https://github.com/awslabs/gluon-ts/pull/727 see https://github.com/awslabs/gluon-ts/pull/727/checks.
(The preceding pulls, including the most recent before this in the commit history https://github.com/awslabs/gluon-ts/pull/728 do not have this error: https://github.com/awslabs/gluon-ts/pull/728/checks.)
Examples:

https://github.com/awslabs/gluon-ts/pull/772/checks?check_run_id=606008914

  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\_pytest\python.py", line 166, in pytest_pyfunc_call
    testfunction(**testargs)
  File "D:\a\gluon-ts\gluon-ts\test\model\wavenet\test_model.py", line 40, in test_accuracy
    accuracy_test(WaveNetEstimator, hyperparameters, accuracy=0.7)
  File "D:\a\gluon-ts\gluon-ts\test\model\conftest.py", line 100, in test_accuracy
    evaluator=Evaluator(calculate_owa=statsmodels is not None),
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\evaluation\backtest.py", line 224, in backtest_metrics
    ts_it, forecast_it, num_series=maybe_len(test_dataset)
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\evaluation\_base.py", line 178, in __call__
    for ts, forecast in it:
  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\tqdm\std.py", line 1127, in __iter__
    for obj in iterable:
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\model\predictor.py", line 317, in predict
    num_samples=num_samples,
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\model\forecast_generator.py", line 204, in __call__
    outputs = prediction_net(*inputs).asnumpy()
  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\mxnet\ndarray\ndarray.py", line 2535, in asnumpy
    ctypes.c_size_t(data.size)))

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
test\model\wavenet\test_model.py
##[error]Process completed with exit code 1.

https://github.com/awslabs/gluon-ts/pull/773/checks?check_run_id=605821548

File "D:\a\gluon-ts\gluon-ts\test\model\conftest.py", line 101, in test_accuracy
    calculate_owa=statsmodels is not None, num_workers=num_workers
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\evaluation\backtest.py", line 224, in backtest_metrics
    ts_it, forecast_it, num_series=maybe_len(test_dataset)
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\evaluation\_base.py", line 179, in __call__
    rows.append(self.get_metrics_per_ts(ts, forecast))
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\evaluation\_base.py", line 333, in get_metrics_per_ts
    forecast.quantile(lower_q.value),
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\model\forecast.py", line 631, in quantile
    q = self.distribution.quantile(mx.nd.array([level])).asnumpy()[0]
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\distribution\binned.py", line 218, in quantile
    _, states = F.contrib.foreach(step, probs, start_state)
  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\mxnet\ndarray\contrib.py", line 224, in foreach
    outs, states = body(eles, states)
  File "d:\a\gluon-ts\gluon-ts\src\gluonts\distribution\binned.py", line 216, in step
    return zeros_batch_size, (cdf, idx)
  File "<string>", line 58, in broadcast_add
  File "c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\mxnet\_ctypes\ndarray.py", line 114, in _imperative_invoke
    stype=out_stypes[0])

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
test\model\wavenet\test_model.py ..
##[error]Process completed with exit code 1.

Since you might know better what needs to be fixed, could you please look into it @jaheba ?

This issue also occurs when running the wavenet test_model.py on MacOs, increasing the timeout for this test allows it to finish the run successfully.

Ive looked into this a bit.
The wavenet estimator fails due to a general timeout error caused by PyTest killing the function due to it taking too long to execute (>30 seconds).
This happens specifically to WaveNet very often due to the hyperparameter batches_per_epoch being set too high for it to consistently finish within pytests time limit.
Previous to commit #727 the test for WaveNet did not run for the constant dataset but only for the synthetic dataset. Due to the increased size of the constant dataset (~60% larger than the synthetic) and batches_per_epoch being the same, the test times out and the thread is killed.
I see two solutions:

Increase the timeout for this test using @pytest.mark.timeout(60)
lower the batch size for example as: batches_per_epoch=1

which one do you prefer?
@jaheba @lostella
I personally prefer setting it to 1 since we are likely to replace the accuracy check with a more comprehensive accuracy regression test anyhow.

I think we can try option 2, but maybe setting batches_per_epoch to half of what it is now?

Setting it to 5 seems reasonable, it might break the already existing accuracy test though, should I handle that in a seperate pull request or in the same one?
Running it at batches_per_epoch=5 causes us to save around 8 seconds, so this should be enough time wise.

Setting it to 5 seems reasonable, it might break the already existing accuracy test though, should I handle that in a seperate pull request or in the same one?

The same PR works, you can try to figure out what a "good threshold" is. And if batches_per_epoch=5 is not low enough for some reason, you can put 4 to further reduce the chance of timeouts.

Also should I revert the change that was made in #777 ?
I.e. should I allow this to be run on windows again since it should work faster?
I would say yes and if it is not lowered enough I will simply lower the threshold again?
Whats your take on this?

Yes sure, we should re-enable what we are skipping


Discussion: Reducing complexity and reducing required dependencies

I think a lot of the recent Xcode 8 Overcoat build pains  ( see the heroic efforts of @Iyuna with #144 )  come from the fact that Overcoat adds nested dependencies to (potentially) unwanted frameworks -- each dependency adds its own build environment requirements, complexity, and just as importantly -- more build time.
I imagine some people don't necessarily want Reactive Cocoa or Promise Kit linking into their projects.  Others are only interested in Reactive Cocoa -- not Promise Kit, and some will want the reverse.  But in general, adding the dependencies to the core Overcoat project seems add additional frustration for anyone that wants to consume Overcoat in it's purest form -- as a convenience framework for object-mapping AFNetworking responses with Mantle.
I'm not exactly clear on the best way to implement a more modular Overcoat framework in the modern era of package management (CocoaPods, Carthage, eventually Swift PM).  Ideally, whatever solution is one that more easily allows someone to opt-in to the accessories and their nested dependencies, but I think it's an important effort worthy of some time investment.
Maybe the answer is really to have separate GitHub repositories (and thus separate frameworks) for each optional accessory?

Overcoat+PromiseKit
Overcoat+ReactiveCocoa
Overcoat+ThenKit  --- Maybe @rodrigo-lima ?

Thoughts from the community or @sodastsai ?  Do you think splitting things up will add more hassle than it's worth?

@grgcombs Don't worry, I was able to fix everything.


gitignore: add a few more

Add *.dyn_hi and *.dyn_o after discovering they exist.

Hmm, in which circumstance do these files exist, and do you know where they're from/what the contents of them are?

I got them on my server while doing a few builds to get arata set up on a test network.
On branch master
Your branch is up-to-date with 'origin/master'.

Untracked files:
(use "git add <file>..." to include in what will be committed)

src/Arata.dyn_hi
src/Arata.dyn_o
src/Arata/Config.dyn_hi
src/Arata/Config.dyn_o
src/Arata/DB.dyn_hi
src/Arata/DB.dyn_o
src/Arata/Helper.dyn_hi
src/Arata/Helper.dyn_o
src/Arata/Message.dyn_hi
src/Arata/Message.dyn_o
src/Arata/Plugin/Load.dyn_hi
src/Arata/Plugin/Load.dyn_o
src/Arata/Protocol/Charybdis.dyn_hi
src/Arata/Protocol/Charybdis.dyn_o
src/Arata/TS6.dyn_hi
src/Arata/TS6.dyn_o
src/Arata/Types.dyn_hi
src/Arata/Types.dyn_o
src/Dated.dyn_hi
src/Dated.dyn_o
src/Main.dyn_hi
src/Main.dyn_o

nothing added to commit but untracked files present (use "git add" to track)


Use Node 18 >= 18.14.2

Node 18.14.0 and 18.14.1 include npm 9.3.1, which has a breaking change related to how local dependencies are installed: https://github.com/nodejs/node/issues/46542
This breaks pipeline js - template - ci:
https://dev.azure.com/azure-sdk/public/_build/results?buildId=2206943&view=logs&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=2102385d-609d-5572-64d2-932661c7902f&l=30
When installing Node 18, we should require >= 18.14.2 to avoid this isssue.

Closing in favor of https://github.com/Azure/azure-sdk-tools/issues/5536


[5.7] Redis events (simplified)

This is a reboot of #22961 without the clutter of #23225 and #23235.
It adds a QueryExecuted event to Redis connections. It's pretty much identical to what the database component is doing regarding events.
The only breaking change is that RedisManager constructor has changed and now requires a Application instance, just like DatabaseManager does.
My use case is: We're using the Server Timing API in production to continuously measure performance. Another interesting use case would be Debugbar support to get insights into Redis calls during development.
Just like in the database component, Redis events are enabled by default, but may be disabled in a service provider:
class AppServiceProvider extends ServiceProvider
{
    public function boot()
    {
        Redis::disableEvents();
    }
}

What this PR might be lacking is an easy way to disable Redis events in the AppServiceProvider:
class AppServiceProvider extends ServiceProvider
{
    public function boot()
    {
        Redis::disableEvents();
    }
}

But since this is on the 5.7 branch, I though this could be added in a separate PR to keep this one short and simple.

What this PR might be lacking is an easy way to disable Redis events in the AppServiceProvider:

For me this is essential to move forward (parts of my comments from https://github.com/laravel/framework/pull/22961#issuecomment-361747799 and https://github.com/laravel/framework/pull/22961#issuecomment-363070187 still apply).
As in: it should really default to off by default unless required and I rather see a clear solution this is possible/default.

@mfn: I've updated the PR. Whether or not events are enabled by default, I'll leave up to Taylor. For now I copied what the database component is doing.

Thanks!


Bump helm-tool to v0.4.2

https://github.com/cert-manager/helm-tool/releases/tag/v0.4.2

/approve


Create video tutorials (relavance, pre-install, install, use, integrated)

script part 1 and 2

Introduction to BigBang

What is a mailinglist?

A way to be in conversation with a group of people by using asynchornous technology that works on low-bandwith Internet connections
Used for software projects as well as Internet governance

Why is it important?

One function of mailinglists is that archives are kept
Not just an archive for archive's sake, but also to document decisionmaking. Because of the important of software and Internet infrastructure, these documents could be seen as public policy documents, and analyzed as such
When these archives are publicly accessibl

Where is it used?

Mailinglists have been around since the late 1960. There are two main software packages to administer mailinglists: Mailman and Listserv. These are the ones that can be analyzed with BigBang.
There are also other mailinglists such as Google Groups - but these are not supported by BigBang, in part because Google makes it very hard to access them.

What can BigBang do with mailinglists?

BigBang can do many things, but it has three basic functionalities: basic statistics, network analysis, and discourse analysis
Statistics

Counting people that post to a mailinglist (over time)
Counting ppls tenure
Counting emails (total + list)
Counting threads, longest thread

Network analysis

Affiliation of actors
Centrality
How networks form within a list and between list
Explore modularity and fragmentation in communities

Discourse analysis

Word trends over time
Key word occurence
Most used word per person or per all users
Special word analysis

How to install BigBang

on Ubuntu

it's easy on linux
If you run Mac, the instructions are very similar, check the BigBang website
If you run Mac or Windows you can always choose to either run Ubuntu in a virtual Machine, or with a persistent live USB. Especially if you're not very comfortable with the command line yet, this is a safe way to experiment and not mess up your production environment.

Instructions on setting up a virtualbox or use the LiveUSB (and make it persistent)

https://www.linuxuprising.com/2019/08/rufus-creating-persistent-storage-live.html

Installing goes through a couple of steps. I will not just go through the steps, but also seek to explain them. If you're just looking for the commands, look at the website.

Install Anaconda

Anaconda is a virtual environment on your systems that keeps track of which versions of files you're using. Software is pretty much always built on other software, but if that software changes, it might break something. This is why it is important to try to keep the versions consistent. This is what Anaconda does for is. So let's go the the anaconda website, download the package and install it.
Then add Anaconda to path - do this else the conda and source activate commands will not work!

Install git

Git is a version management control sytems. It is the interface with the systems where we develop BigBang. You need this in order to download BigBang.

Now open a terminal and first type:

sudo apt install git
then download bigbang:

git clone https://github.com/datactive/bigbang.git

Install BigBang

Now we will need to install all the software needed for BigBang to run and setup an environment in which BigBang will run. For this we will run an install script

type: bash conda-setup.sh

Congrats - you have now installed BigBang
If there are any issues with this installation process - feel free to send an email to the BigBang mailinglist<EMAIL_ADDRESS>

How to use BigBang

Get data
What question do you want to ask?
Start a Jupyter notebook
edit the notebooks variables
Play
see data, export data, use data (also with Gephy etc)

is this finished, @nllz ? Or is some part of it finished in the linked video from the current README?

Sorry about that - indeed finished it with https://www.youtube.com/watch?v=JWimku8JVqE


Tries to install PhantomJS 4 times and then finishes script, without error. But it is not installed.

When composer tries to run the installPhantomJS method, it just tries to download phantomjs 4 times unsuccessfully. It does not error. The binary is not installed int he correct location.
I'm guessing that it just can't save the file for some reason and it's trying over and over again?
> PhantomInstaller\Installer::installPhantomJS
  - Installing phantomjs (2.1.1)
    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: 100%

Hey Wes!
Thank you for reporting the issue.
Which version of phantomjs-installer are you using?
What you are describing is not reproducible for me, but must be close to the main download() function: https://github.com/jakoch/phantomjs-installer/blob/master/src/PhantomInstaller/Installer.php#L191-L203
I wonder why it tries to download over and over gain. I would expect the downloader to retry 3 times, then fail, see Composer/Downloader/FileDownloader L135
If it just can't save the file for some reason, it should throw an \UnexpectedValueException (see Composer/Downloader/FileDownloader L159), which then bubbles up to the installer, where the try-catch block should catch it as Exception.
You should get an error/exception.

+1
I get the same problem:
> PhantomInstaller\Installer::installPhantomJS
  - Installing phantomjs (2.1.1)
    Downloading: Connecting...    Downloading: 0%               Downloading: 5%    Downloading: 10%    Downloading: 15%    Downloading: 20%    Downloading: 25%    Downloading: 30%    Downloading: 35%    Downloading: 40%    Downloading: 45%    Downloading: 50%    Downloading: 55%    Downloading: 60%    Downloading: 65%    Downloading: 70%    Downloading: 75%    Downloading: 80%    Downloading: 85%    Downloading: 90%    Downloading: 95%    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: Connecting...    Downloading: 0%               Downloading: 5%    Downloading: 10%    Downloading: 15%    Downloading: 20%    Downloading: 25%    Downloading: 30%    Downloading: 35%    Downloading: 40%    Downloading: 45%    Downloading: 50%    Downloading: 55%    Downloading: 60%    Downloading: 65%    Downloading: 70%    Downloading: 75%    Downloading: 80%    Downloading: 85%    Downloading: 90%    Downloading: 95%    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: Connecting...    Downloading: 0%               Downloading: 5%    Downloading: 10%    Downloading: 15%    Downloading: 20%    Downloading: 25%    Downloading: 30%    Downloading: 35%    Downloading: 40%    Downloading: 45%    Downloading: 50%    Downloading: 55%    Downloading: 60%    Downloading: 65%    Downloading: 70%    Downloading: 75%    Downloading: 80%    Downloading: 85%    Downloading: 90%    Downloading: 95%    Downloading: 100%
  - Installing phantomjs (2.1.1)
    Downloading: Connecting...    Downloading: 0%               Downloading: 5%    Downloading: 10%    Downloading: 15%    Downloading: 20%    Downloading: 25%    Downloading: 30%    Downloading: 35%    Downloading: 40%    Downloading: 45%    Downloading: 50%    Downloading: 55%    Downloading: 60%    Downloading: 65%    Downloading: 70%    Downloading: 75%    Downloading: 80%    Downloading: 85%    Downloading: 90%    Downloading: 95%    Downloading: 100%
```

Ubuntu 16.04 LTS

I couldn't reproduce the issue (using Composer 1.2.1 + phantomjs-installer 2.1.1-p07).
Please add the additional catch block after Line 193 - to check, if the exception isn't bubbling up.
} catch (UnexpectedValueException $e) {
   $io->error($e-getMessage());
}

I am using https://github.com/jonnnnyw/php-phantomjs which depends on phantomjs-installer 2.1.1.
After finding this: https://stefanzweifel.io/posts/laravel-forge-php7-and-bzip2/ and installing bz2.
And requiring phantomjs-installer 2.1.1-p07, it works fine.
Thanks

Thanks for the feedback. Glad you got it working.
I forgot about the bz2 issue. The problem was mentioned in https://github.com/jakoch/phantomjs-installer/pull/28#issuecomment-208331812
I will send a PR to php-phantomjs to raise the version to 2.1.1-p07.
@t202wes Does an update to 2.1.1-p07 solve the issue for you, too?

I have already sent a PR to php-phantomjs :)

"two is better than one" :) Closed mine.

Issue seems to be resolved. Closing.


AVRO-2308: Use Java StandardCharsets

This seems like a nice change, but it may degrade perfomance.
I tried this years ago, but it slowed down some operations by a solid 20%, so reverted the change.  There are benchmarks from the time somewhere in the avro JIRAs.
Were performance tests done along with this?
On newer JVMs, maybe this is no longer the case.  IIRC this was for Java 7.
I recall the reason was that the string form got to the fast-path faster, and the extra work was masked  an unlikely to be taken branch hint (e.g. unlikely the charset doesn't parse).

I would have to run my own tests to verify, but:
https://blog.codecentric.de/en/2014/04/faster-cleaner-code-since-java-7/
On Mon, Mar 18, 2019, 6:00 PM Scott Carey<EMAIL_ADDRESS>wrote:

This seems like a nice change, but it may degrade perfomance.
I tried this years ago, but it slowed down some operations by a solid 20%,
so reverted the change. There are benchmarks from the time somewhere in the
avro JIRAs.
Were performance tests done along with this?
On newer JVMs, maybe this is no longer the case. IIRC this was for Java 7.
I recall the reason was that the string form got to the fast-path faster,
and the extra work was masked an unlikely to be taken branch hint (e.g.
unlikely the charset doesn't parse).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/avro/pull/437#issuecomment-474118351, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AL_vEz9oa6B0Olc87rpfKzT2hN2eWyLUks5vYAyPgaJpZM4aVB7u
.

Yeah, I did some research and digging in the history.  The issue was with Java 6 and prior, where it was faster with the raw string than the charset.

Great. Thanks for tracking it down!
On Tue, Mar 19, 2019, 5:49 PM Scott Carey<EMAIL_ADDRESS>wrote:

Yeah, I did some research and digging in the history. The issue was with
Java 6 and prior, where it was faster with the raw string than the charset.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/avro/pull/437#issuecomment-474598089, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AL_vE78CJDRP2dO-8cIx_g0-qB_h2lnUks5vYVtLgaJpZM4aVB7u
.

It will be great once we have #476 in we will be able to measure such regressions.

btw, any help with review/tests of that PR will be greatly appreciated. We need to have some infrastructure to try to catch performance regressions.