Absolutely horrible Lucene performance with JDK 23 (Lucene 9.11.1 and 10.0.0)

Description
I am using Lucene in my app for several years happily with JDKs up to 22.
My use case searches through film data and Lucene can return fairly huge result sets to my app - which as of now never was a problem.
I upgraded my app to JDK 23.0.1 on my MacBook Air macOS 15.01 16GB RAM.
openjdk version "23.0.1" 2024-10-15
OpenJDK Runtime Environment (build 23.0.1+13)
OpenJDK 64-Bit Server VM (build 23.0.1+13, mixed mode, sharing)

and started to notice horrible Lucene performance.
With the following code snippet I do query my results:
var reader = DirectoryReader.open(list.getLuceneDirectory());
final var searcher = new IndexSearcher(reader);
final var docs = searcher.search(finalQuery, list.size());
final var hit_length = docs.scoreDocs.length;

var storedFields = searcher.storedFields();
// the for loop takes ages with JDK23...
for (final var hit : docs.scoreDocs) {
	var docId = hit.doc;
	//storedFields.prefetch(docId); //<-- doesn't change anything
	var d = storedFields.document(docId, INTEREST_SET); //<-- this takes ages
	//filmNrSet.add(Integer.parseInt(d.get(LuceneIndexKeys.ID)));
}

In the explored use case Lucene always returned the expected 558333 hits out of 802k documents.
99% of the app runs take 5.7 seconds to get the result. However when I am lucky 1% of the app runs do get the same result back in 756 milliseconds.
If Lucene is delivering fast, it will stay fast, if it is slow it will remain slow. I have no idea how this is triggered.
I moved from NRTCachingDirectory to MMapDirectory but the performance remained bad. Tried some other stuff from internet - same result.
I switched back to JDK 22.
openjdk version "22.0.2" 2024-07-16
OpenJDK Runtime Environment (build 22.0.2+11)
OpenJDK 64-Bit Server VM (build 22.0.2+11, mixed mode, sharing)

The same source which performed just horrible with JDK23 was consistently fast with JDK 22:
Search took: 763.8 ms

I am using the following flags for the JVM:
-ea
-XX:+UseShenandoahGC
-XX:ShenandoahGCHeuristics=compact
-XX:+UseStringDeduplication
-XX:MaxRAMPercentage=50.0
--enable-native-access=ALL-UNNAMED
--add-modules jdk.incubator.vector

I removed --enable-native-access=ALL-UNNAMED and --add-modules jdk.incubator.vector for testing purposes but the performance remained bad with JDK23.
I made the same tests with JDK23 on my Windows 11 AMD Ryzen 4900H 16GB RAM laptop. There I get the results back in 13.63 seconds with JDK 23. JDK 22 does the same consistently in 2.1 seconds.
Switching between Lucene 9.11.1 and 10.0.0 made no difference, always shitty performance with JDK 23 both on macOS and windows. Consistent performance with JDK 22.
Version and environment details
macOS 15.01
MacBook Air 16GB RAM
Windows 11 23H2 / Ubuntu Linux 24.10
AMD Ryzen 4900H 16GB RAM
Lucene 9.11.1
Lucene 10.0.0
JDK 23.0.1
JDK 22.0.2

I made some tests with Ubuntu 24.10:
JDK 23: 9.9 seconds
JDK 22: 1.4 seconds

@derreisende77 do you have profiling of the two different runs? Maybe through async-profiler? It would be interesting to see where the time is being spent.

@benwtrent I have JProfiler but I am not really experienced in using it - or profiling at all.
I made two runs on macOS and made screenshots from the hotspot page.
JDK23:

JDK22:

The highlighted line in JDK22 image marks the function from where I posted the code snippet at the beginning.
The StoredFields.document ine above corresponds to the var d = storedFields.document(docId, INTEREST_SET); line.
What I have seen during the profile run:

ArrayUtil.growExact takes a lot more time on JDK23 than on JDK 22.
UnicodeUtil.UTF16toUTF8 calls were created during the index creation phase and take almost the same time on JDK 22 and 23.

HTH

@benwtrent I made several tests on macOS with JDK 23 and 23.0.1 from Liberica and Azul. I always ran into the performance problem.
I switched from Shenandoah GC to ZGC with
-XX:+UseZGC
-XX:+ZGenerational
-XX:ZUncommitDelay=5
-XX:+ZUncommit
-XX:SoftMaxHeapSize=4g

and so far was unable to trigger the performance problem. Lucene behaves as performant as before with JDK 23

Is there also no regression if you use the default garbage collector? If so, this looks like a regression with Shenandoah.

I have no problem with default GC as well.
I downloaded a Windows Amazon Corretto 23 nightly build from today and I don't have a problem with Shenandoah anymore. So I guess it is a Shenandoah problem in JDK 23 and 23.0.1 that will be fixed in a later 23 release.


Azure Bastion refactor update

Azure Bastion refactor update

Updated code as per review comment

removed auto_scale_vm_count, , unnecessary file deletion and some more updates


Update readme

Please update the readme file

This is a very confusing way to learn.  We need two windows, minimum.  1 with instructions.  2 with actual github.  Doing this all from one window with buttons just doesn't work.


Error while setting up ecowater_softener platform for sensor

Similar to #43 but the error logs don't match up, and I'm still able to login at wifi.ecowater.com unlike last time when the URL changed for ~1 day.
Running v3.1.0 in Home Assistant Container 2024.1.6.  Seeing issue since earlier this morning, ~9:30 CST.  Restarting integration and restarting home assistant does not appear to restore functionality.
Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:360
Integration: Sensor (documentation, issues)
First occurred: 10:01:13 AM (3 occurrences)
Last logged: 12:31:38 PM

Error while setting up ecowater_softener platform for sensor
Traceback (most recent call last):
  File "/config/custom_components/ecowater_softener/coordinator.py", line 50, in _async_update_data
    data_json = await self.hass.async_add_executor_job(lambda: ecowaterDevice._get())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/ecowater_softener/coordinator.py", line 50, in <lambda>
    data_json = await self.hass.async_add_executor_job(lambda: ecowaterDevice._get())
                                                               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ecowater_softener/ecowater.py", line 59, in _get
    json_data = json.loads(data.text)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1220)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 300, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/ecowater_softener/coordinator.py", line 83, in _async_update_data
    raise UpdateFailed(f"Error communicating with API: {e}")
homeassistant.helpers.update_coordinator.UpdateFailed: Error communicating with API: Extra data: line 2 column 1 (char 1220)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 360, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/ecowater_softener/sensor.py", line 107, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 272, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: Error communicating with API: Extra data: line 2 column 1 (char 1220)

Same issue since 3 hours ago. I am
able to access through the mobile app (iOS) and also at wifi.ecowater.com.

It looks like a temporary problem with the api. I just recharged the integration and now it works OK again.
I tried recharging the integration some some hours ago and it didn’t work. But now it worked.

@nanderson97651 Did this issue resolve itself? It sounds like the API had an issue but figorr is reporting that it resolved itself.

Yes, I didn't see this error anymore. Probably it was a maintenance issue related to the API. So once the maintenance ended ... the API came back to life again.

Thanks! In that case, I'll close this issue.
👍


Fixes incorrect import suggestions in FN016004. Closes #1484

Fixes incorrect import suggestions in FN016004. Closes #1484

Coverage remained the same at 100.0% when pulling 7dda83b799dde4fa43f7152cd29ffaaf4833f979 on waldekmastykarz:fix-1484 into 4206056a040cb4660b711bbf2e81092b4e8affc1 on pnp:dev.

Merged manually


fixes #122

@tillahoffmann
As discussed in the issue #122 , here's my fix.
I've also added a test.

Codecov Report

Merging #123 (d3e0651) into master (70055bc) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #123      +/-   ##
==========================================
- Coverage   83.36%   83.33%   -0.04%
==========================================
  Files          22       22
  Lines         529      528       -1
  Branches       36       35       -1
==========================================
- Hits          441      440       -1
- Misses         70       71       +1
+ Partials       18       17       -1

Impacted Files
Coverage Δ

testcontainers/selenium.py
92.59% <100.00%> (+3.30%)
:arrow_up:

testcontainers/neo4j.py
93.75% <0.00%> (-6.25%)
:arrow_down:

testcontainers/core/container.py
80.23% <0.00%> (+1.16%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70055bc...d3e0651. Read the comment docs.


Try to fix EVGA 1.0.6

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

Microsoft Reviewers: Open in CodeFlow

is this another instance of the URL bug? It also happens with the links in the FileZilla manifests, see #7841.

is this another instance of the URL bug? It also happens with the links in the FileZilla manifests, see #7841.

I think so. I can download the file via the link in #7841 .

Ashwini, can you see why the service is getting a permission error.

Investigating, will update once I find anything.

/azp run


aws-sam-cli: 0.16.1 -> 0.19.0

Motivation for this change
Saw a weird error while reviewing another pr
Actions:

python2 -> python3
version bump
made python2 only libs optional
added longDescription

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[X] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Notify maintainers
cc @andreabedini
@FRidh  because it's most a python package.

Rebased on top of master, and applied changes.

@GrahamcOfBorg build aws-sam-cli

@worldofpeace @marsam rebased on top of master, applied changes 6 weeks ago :)

@jonringer We'll see about integrating. And commit bit soon :rofl:

@GrahamcOfBorg build aws-sam-cli

[3 built, 1 copied (0.1 MiB), 0.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/66125
5 package were build:
aws-sam-cli python27Packages.aws-lambda-builders python27Packages.serverlessrepo python37Packages.aws-lambda-builders python37Packages.serverlessrepo

master is now at 0.22.0, closing


remove temporary variable

it looks like this was mostly copied from the documentation, so I think we're okay to remove the temporary variable and simplify this a little.

Thanks.


RMT API no wait

Hi,
I'm currently porting https://github.com/Xinyuan-LilyGO/LilyGo-EPD47 to rust using esp-hal. I managed to implement a basic working poc.
While replicating the RMT behavior, I noticed that there are some hardware specific operations that are rather timing sensitive. The original code uses rmt_write_items with wait_tx_done set to false.
I was able to replicate this behavior, but I don't think the solution is very elegant.
As long as I can wait for the tx to complete, I'm able to reuse the TxChannel using the result of SingleShotTxTransaction::wait().
If I can't wait due to timing constraints, I have to re-create the channel the next time I want to use it. That feels rather inefficient and happens quite often (once for each row of the epd).
Hence I was wondering if it would make sense to add something like
pub fn nowait(self) -> C to SingleShotTxTransaction (or channel(self))
This would allow to reuse the channel without waiting for the tx.
I guess it would also make sense then to add something like is_busy() or is_done() to the channel (Not sure what would happen if I transmit more data while another tx is still in progress).
Disclaimer
I'm neither a hardware nor a rust expert. This is basically my second rust project and was meant as a learning project. So there might be a chance I'm missing one or two things..
Best Regards,
Frido

Thanks for raising this issue. I'd say yes something like this would make sense for the case when the amount of data is less (or equal) than the FIFO size.
In that case we could just fill the FIFO and the use can check the status before starting another write.
For that we should probably introduce a new type like SingleShotTxTransactionFittingFifo and a new function like ``transmit_fitting_fifo` (I don't like that naming - just to illustrate things)

I was curious and took a crack at it:

I added transmit_single_block to TxChannel
Add SingleShotTxTransactionSingleBlock with wait and no_wait
Add is_busy to TxChannelInternal and expose it in TxChannel

I tested it and it seams to work on my ESP32S3
Here is my fork: https://github.com/esp-rs/esp-hal/compare/main...fridolin-koch:esp-hal:main
Some notes:

Not sure about the naming, I went with SingleBlock since otherwise you would transfer multiple blocks/chunks of bytes, maybe "chunk" is better
The is_busy implementation is a wild guess. I tried !is_done() but that didn't work (just hung forever). After browsing through the docs, I found esp32s3::rmt::ch_tx_status::STATE_R which sounded promising. I consulted the technical reference manual (p. 1396 / 1412 ) regarding possible values, but found no info. After doing some experiments (i.e. retrieving the value after wait and no_wait), the value was either 1 (no_wait) or 0 (wait). Hence my guess 1 == busy/transmitting.

In case this is at least somewhat useful, I'm happy to send it as a PR :)

Related, I'm also playing with the RMT driver. In my case, I want it to take an iterator, #1768 - this way I can translate data for the bits of LED colors without needing a large buffer.
I wonder if that would be a simpler solution for your use case of sending a large amount of data via RMT? I'm also looking into async support of the iterator, so you don't have to fully stop while sending from the iterator.
(my in-progress work is https://github.com/esp-rs/esp-hal/compare/main...tschundler:esp-hal:main)
The challenge is that if the iterator can't generate data fast enough to keep the RMT buffer filled, it will just loop through whatever is in the buffer, resending it. So if you want to use the iterator to send and then pause as you build the next line, you'd need to fill the RMT's ring buffer with transmitting all 0s or something.

Ok, reading your ed047tc1.rs, what I'm proposing is probably not useful for how you are using RMT.
I think you can do what you want without needing anything new. But I can see where single-shot / no wait is handy.
With current RMT, You could use store the channel or the transaction both in your rmt struct. If there is a transaction, .wait() on it to get the channel at the start of the new pulse, where you busy loop now.
enum TXCh {
   #[default]
   None,
   Channel(TxChannel),
   Txn(SingleShotTxTransaction),
}

impl TXCh {
  fn take(&mut self) -> Result<TxChannel, error> {
    match mem::take(self) {
      None => panic!("very broken"),
      Channel(ch) => Ok(Ch)
      Txn(txn) => txn.wait()
    }
  }
}

If you do your design, maybe the busy wait should be in the hal rmt driver, since wouldn't any user need to use that?

I wonder if that would be a simpler solution for your use case of sending a large amount of data via RMT? I'm also looking into async support of the iterator, so you don't have to fully stop while sending from the iterator.

I'm not sending a large amount of data. The rmt is used to control the gate clock (as far as I understand the datasheet/schematics). So I'm only sending 2 pulses which should result in 2 x 32bit written to the rmt ram (Which should be 48x32bit per channel, at least that's how I understand the code / technical reference).
Async was also my first approach, but that messed up the timing completely and resulted in garbage output on the screen, so I discarded that idea. I can't rule out that it was due to an implementation error on my side.

The challenge is that if the iterator can't generate data fast enough to keep the RMT buffer filled, it will just loop through whatever is in the buffer, resending it. So if you want to use the iterator to send and then pause as you build the next line, you'd need to fill the RMT's ring buffer with transmitting all 0s or something.

Probably because Wrap Tx Mode is enabled? (See pp. 1399 of the technical reference). Maybe try disabling it?

With current RMT, You could use store the channel or the transaction both in your rmt struct. If there is a transaction, .wait() on it to get the channel at the start of the new pulse, where you busy loop now.

That sounds like a good solution! Thanks!


App secret in GraphAPI constructor

If the Facebook app has Require App Secret enabled, this SDK becomes useless - all requests are going to fail.
There should be an option for providing a secret key in the GraphAPI(**kwargs) constructor

This is my proposal. https://github.com/pythonforfacebook/facebook-sdk/pull/213

Closing because #213 seems like a good solution to this issue. Further discussion should take place there.


LDAP Integration

Description
I am aware that version 0.12.0 will include auth reworks and the possibility of multi user support. Being able to incorporate into an LDAP database would be really powerful for larger groups of people that may need different home pages for what they do. Of course this wouldn't be easy to implement and would need to also implement LDAP standards with group perms and other things part of that system.
Relates to #623 in that it provides a centralized and secure user access from a single point.
Priority
Medium (Would be very useful)

That could be great!

Implemented via #1497


application won't start: "sh: 1: electron: not found"

I'm on Linux Mint 17 which is based on Ubuntu 14.04. I ran the installation steps (along the way I had to install npm, even though I'd thought I already had it installed):
git clone https://github.com/zulip/zulip-electron
cd zulip-electron/
npm install
sudo apt-get install npm
npm start

I got this error:
$ npm start

><EMAIL_ADDRESS>start /home/sumanah/test/zulip-electron
> electron .

sh: 1: electron: not found
npm ERR! weird error 127
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR! not ok code 0

I read /usr/share/doc/nodejs/README.Debian but I'm not sure whether there's anything I can do:
nodejs for Debian
=================

packaged modules
----------------

The global search path for modules is
/usr/lib/nodejs

Future packages of node modules will use that directory,
so it should be used wisely.

user modules
------------

Node looks for modules in ./node_modules directory first;
please read node#modules documentation carefully for more information.

Node does not look for modules in /usr/local/lib/node_modules,
where npm put them.
Please read npm-link(1) of npm package, to understand how to properly
use npm-installed modules in a project.

Note that require.paths is not supported in future node versions.
See also node(1) for more information about NODE_PATH.

nodejs command
--------------

The upstream name for the Node.js interpreter command is "node".
In Debian the interpreter command has been changed to "nodejs".

This was done to prevent a namespace collision: other commands use
the same name in their upstreams, such as ax25-node from the "node"
package.

Scripts calling Node.js as a shell command must be changed to instead
use the "nodejs" command.

I can copy and paste my install log from installing npm if that helps.

Duh. I used apt to install npm but then after doing so I didn't go back and actually run npm install! Now I've done so and all is well.

We should document the need to install nodejs-legacy on Ubuntu/Debian systems.


What's wrong? I have opened the VPN in global mode.

curl: (7) Failed to connect to api.openai.com port 443: Operation timed out
Your request to Open AI API failed:

@CaiXunchao I am closing this issue as it is a failure that is not caused by the script itself. I hope you figured it out.


Stalls after login

waiting for locator("a").locator("text=Returns & Orders") to be visible

If the page doesn't load it's probably Amazon detecting you are automation.  Jump through all of the hoops for logging in and getting to that page, and the script will continue when it detects the link that you mentioned.


Update Free Membership.js

When I buy membership it is free

vhgvgv

This is code, which is obfuscated, and not updatable


set bos directory to an env variable

Set the bos directory to an env variable OR .bos (default) and replaced the hard coded value everywhere I found.

Will have to change the path joining lines everywhere in the code, since it won't be join(homedir(), home, config.json) etc anymore. Is that ok?

I think it can still do join, but instead of join(homedir(), home, file) it would be join(homePath({}), file)? and homepath is going to return either the env var path or join(homedir(), home)

Right yeah. Will make the changes tomorrow.

Can you check now?

I merged it but made some conflicting changes


set INTEL CPU frequency

can i use this repository on windows platform to set frequency of intel cpu ?

Hi @nikitavedre
The power library currently only works on linux platform.
Currently we have no plans to support windows


Wrong negative Temperature

The lib display wrong negatives temperatures. The type of the uncomp_temp is uint32_t and it must be int32_t.
For solve it must be change the structure:
/*! @name Uncompensated data structure */
struct bmp280_uncomp_data {
int32_t uncomp_temp;
int32_t uncomp_press;
};
And the function with:
int32_t bmp280_comp_temp_32bit(int32_t uncomp_temp, struct bmp280_dev *dev)
best regards

I believe this is a duplicate of this issue : #17


Non-Default worlds are inaccessible...

Recently, all of the Multiverse worlds located on my server, not including world, world_nether, and world_the_end, have become inaccessible. The files to the worlds still appear and contain every thing including the data, playerdata, and whatnot, but they don't appear when you type '/mv list'. Additionally, I use GGserver's server managing panel. Not sure if that has to do with anything...

Use latest versions of everything to see if the issue persists, then pastebin your log file here.

By everything what do you mean? Plugins? I'm not much of a guru on this stuff,

Nevermind, problem resolved! If just had to re-import my worlds! But thanks for responding anyways, I appreciate it

Sorry for not being specific enough with what I meant, I meant Multiverse plugin versions, and server version. Thanks for listing that as something that fixes it :)


Enhanced the contact section

Fixes Issue🛠
Closes #754
Description👨💻
I changed the contact form background to match the theme.
Type of change📄

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)

How this has been tested✅
I have tested it on my own local server.
Checklist✅

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added demonstration in the form of GIF/video file
[x] I am an Open Source Contributor

Screenshots/GIF📷

@Rakesh9100 please have a look

@Rakesh9100 There was a conflict, I have fixed it now check my PR

Congrats on getting your PR merged to the repo under GSSOC 2k24 🥳 @ayush-848


spoilers: Style headers in Markdown-neutral way.

This PR corrects for some over-zealous CSS introduced in #29715:

It no longer assumes a paragraph for the spoiler header, which assumption lead to a misplaced spoiler arrow previously.
It no longer touches possible margin-left or -right values, such as may be used to style list items.

CZO discussion

Screenshots and screen captures:

Before
After

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks @karlstolley!


AE Conflict of Interest

Having checked through the authors and against the ACM Guidance on CoIs we have identified a conflict in relation to @jsndyks work with one of the authors on Dagstuhl Seminar 23381.
@jsndyks to withdraw at this stage (before 'responding to the authors with initial submission email' - no decisions made, no reviewers assigned).Alternative AE to be appointed.

Invited a replacement AE with no COI


chore: update package.json with @vueuse/head peer version

Description
fix @vueuse/head unmet peer

Didn't see https://github.com/antfu/vite-ssg/pull/317


Mojo::Reactor::EV: Timer failed: Transaction already destroyed

Mojolicious version: 7.91
Perl version: 5.20.1
Operating system: Linux / CentOS 6 / x86_64

CORE
  Perl        (v5.20.1, linux)
  Mojolicious (7.91, Doughnut)

OPTIONAL
  Cpanel::JSON::XS 4.04+  (4.04)
  EV 4.0+                 (4.22)
  IO::Socket::Socks 0.64+ (n/a)
  IO::Socket::SSL 2.009+  (2.058)
  Net::DNS::Native 0.15+  (0.18)
  Role::Tiny 2.000001+    (2.000006)

This version is up to date, have fun!

Actual behavior
Another episode similar to #1247 and #1235.

in #1247 we had Mojo::Reactor::EV: Timer failed: Connection already closed
here we have Mojo::Reactor::EV: Timer failed: Transaction already destroyed

This is what I see in a logfile:
[13:00:00 2018] [error] Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value
[13:00:00 2018] [error] Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value
[13:01:00 2018] [warn] Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[13:01:00 2018] [warn] Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[13:01:00 2018] [error] Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value
[13:02:00 2018] [error] Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value
[13:02:00 2018] [warn] Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[13:02:01 2018] [error] Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value

As mentioned before we set up UserAgent's timeouts like:
$app->ua->max_connections(100)->inactivity_timeout(60)->request_timeout(0)->connect_timeout(10);

Looking at the log time stamps (troubles appears in approx. 1 min period) it seems that it might be related to inactivity_timeout(60).
Currently I do not have a full stack trace yet.
Expected behavior
On 7.82 the same applications code run without any error like this (or similar).
Steps to reproduce the behavior
I do not have a simple failing test case. On the other hand this one seems to fail much faster on our trouble making site than #1247.
I'll try to investigate more.

The "Transaction already destroyed" error is not a real error, just a hint that you're doing something wrong. https://mojolicious.org/perldoc/Mojolicious/Guides/FAQ#What-does-Transaction-already-destroyed-mean

And "Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value" should not be possible anymore. Please make sure you are really testing with Mojolicious 7.91. If you're really still getting that error, i'm afraid we are back to needing a test case to replicate the problem.

If this turns out to be a real bug again, i think we might have to consider reverting https://github.com/kraih/mojo/commit/fd65c44c3d9b8d3a2242a1521ba08905a811d004. Mojolicious being this unreliable for so long is simply unacceptable.

IMO when the scenario from the URL happens I see an error like Mojo::Reactor::EV: I/O watcher failed: Transaction already destroyed (at least this I saw during my stress testing) but important is that this appears in the log just once and does not affect any future requests. Which is fine.
The trouble I have is that under some circumstances we hit Mojo::Reactor::EV: Timer failed: Transaction already destroyed but it is not just a single message, it start complaining into a logfile repeatedly until I restart the application.
Here is 20 min fragment (the number 2265/2266 is worker PID - so in this case 2 workers out 8 got into a strange state):
[Thu Aug  9 12:41:02 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:42:03 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:43:00 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:44:00 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:44:01 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:45:01 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:46:00 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:46:03 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:47:02 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:47:15 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:49:00 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:49:01 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:50:01 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:50:05 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:51:02 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:51:06 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:52:02 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:53:01 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:53:52 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:54:01 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:55:00 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:55:13 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:56:01 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:57:00 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:57:02 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:58:04 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:59:00 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 12:59:40 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 13:01:00 2018] [warn] 2265 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.
[Thu Aug  9 13:01:00 2018] [warn] 2266 Mojo::Reactor::EV: Timer failed: Transaction already destroyed at /usr/local/perl520/site/lib/Mojolicious/Plugin/DefaultHelpers.pm line 123.

It is also possible that we do the non-blocking app->ua related stuff in a way that is not according the best practice. But the code in question did not change at least 1 year and worked well up to 7.82.
Ad version: it really is the latest 7.91 (today's) - the app was properly reloaded after mojo upgrade (in fact the server was rebooted).
Ad "Mojo::IOLoop::Stream::HTTPClient:25 Can't call method "isa" on an undefined value" it is there and it happens basically at the same time moment as "Mojo::Reactor::EV: Timer failed: Transaction already destroyed".

I've pulled the trigger. https://github.com/kraih/mojo/commit/61f6cbf22c7bf8eb4787bd1014d91ee2416c73e7

OK, meanwhile I have tried to compare logfiles after running our testsuite on 7.82 vs. 7.91 and found some other error messages with 7.91 that did not occur in 7.82 like:
[Thu Aug  9 16:33:57 2018] [error] Can't call method "_write_content" on an undefined value at /usr/local/perl520/site/lib/Mojo/IOLoop/Stream/HTTPServer.pm line 54.

But now it is probably not so interesting.
I have deployed 61f6cbf on our trouble making site.

I assume the lack of complaints means that 7.92 works for everyone?

Yes, works nice for me. Thank you very much for your time.


add typescript typings

Fixes #5
Pull request adds typescript definitions only. I used the naming currentState instead of baseState as this seems to be intention of #4

Thanks a lot!


Use error objects

This PR ensures that uploadcare-node passes error instances to callbacks when bad things happen, instead of numbers or plain objects.
My use case:
In Hapi, you call reply() and pass it your response object or an Error instance. If you give it an error, it'll automatically respond to the client with a HTTP 500 along with the error message. For any other type it'll JSON.stringify it and respond with a HTTP 200.
My problem was that uploadcare was returning a 401 as the error object for the callback, and I was giving that directly to hapi, e.g.:
uploadcare.store(xxx, function (err) {
  if (err) return reply(err)
})

It meant that Hapi was responding with a HTTP 200, and a 401 number as the JSON body! This PR fixes this issue.

Thanks @alanshaw
Pushed in v0.4.1


PySpark dependency

Cause2e is a lightweight package except for the dependency on PySpark. Can we make this optional, given that most users will not really need it?

PySpark is not explicitly imported anywhere, so we can probably remove the dependency. Users who pass a pyspark session when initializing a StructureLearner or Estimator already have an installation of pyspark, so there won't be an error.


Stuck at Fetching Origin Entitlements

I messed up a line while working on the fork and it crashed, after crashing it got stuck at fetching origin entitlements, plz help (o;TωT)o

After further testing it has become apparent that the program configured a proxy on my pc, which either didn't notice was there or it got misaligned during the crash, that was the reason behind the inability to fetch origin entitlements later on, the proxy is great while the software works as intended, but if for any reason whatsoever it crashes/is forcefully closed, it will block every request later on, I use a hardened version of Firefox, that's why I could bypass the proxy, but you should consider making an script to deactivate the proxy, end users may not have the know-how to deactivate it themselves, or add a check to see if any of the tasks are taking too long, so you can just prompt the user with a reset button instead of making the app do an infinite loop.
Man, gotta love those infinite loop bugs, seem to happen to both of us as of late 😆

In the forum topic, I have written this in the usage instructions:

Note: When you run DreamAPI, it automatically configures your system to use DreamAPI as its proxy server. And when you shutdown DreamAPI, it disables the proxy server usage. This means that if the DreamAPI has ended unexpectedly (for instance it crashed or you have killed its process from the task manager) then you would need to manually disable the usage of proxy server in the system settings in order to restore connectivity. Or you can simply run DreamAPI again and properly shut it down.

As it has been previously made apparent, I don't read what I should 🤣

Alright, let me know if there is anything else unclear or could be improved

The check I suggested sounds like a good idea, some other goof may stumble upon a random crash at some point and get all their connections blocked by the proxy.

Released a new version which adds some error handling logic to the fetching: https://github.com/acidicoala/dream-api/releases/tag/v1.1.2


[WIP] Add online label smooth (OLS) loss support

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Add OLS support for better performance compared with original label smooth. In addition, the OLS loss works better under the noisy label case.
Modification
A new loss named online label smooth (OLS) following standard PyTorch loss format is provided. A new hook to plant the OLS loss to the mmcv-style training pipeline is provided. Related configs are given
Use cases (Optional)
The OLS loss could be activated with the following script in mmclassification project
python3 -m torch.distributed.launch \
    --nproc_per_node=8 \
    --master_port=23333 \
    tools/train.py configs/resnet/resnet50_b32x8_ols_imagenet.py \
    --launcher pytorch

Checklist
Before PR:

[ ] Pre-commit or other linting tools are used to fix the potential lint issues.
[ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
[ ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
[ ] The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

[ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
[ ] CLA has been signed and all committers have signed the CLA in this PR.

Hi, thanks for your contribution, would you like to update your PR with the latest master branch ?


Vulnerability fix for PackerBuildV0

Task name: PackerBuildV0
Description:
Documentation changes required: (Y/N)
Added unit tests: (Y/N)
Attached related issue: (Y/N) https://dev.azure.com/mseng/AzureDevOps/_componentGovernance/33/alert/11126?typeId=30110
Checklist:

[x] Task version was bumped - please check instruction how to do it
[x] Checked that applied changes work as expected
[x] node make.js build --task PackerBuildV0
[x] node make.js test --task PackerBuildV0

/azp run


README: Fix quoting in for loop.

The quoting made the for loop iterate only once.
Background

Checklist

[ ] Git commit messages conform to community standards.
[ ] Each Git commit represents meaningful milestones or atomic units of work.
[ ] Changed or added code is covered by appropriate tests.

Thanks!


treewide: migrate packages to by-name hierarchy

Description of changes

The ones maintained by @AndersonTorres
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

I think it was discussed at NixCon that we should move to the new schema using automation (where possible) to avoid the overhead of many PRs. You also did some cleanup so in this case this is probably fine.

@ofborg build pkgsLLVM.mksh mksh

Yeah so I will work on updating https://github.com/NixOS/nixpkgs/pull/211832 next, which is the automated mass migration. I wouldn't want people to have to do this manually to save on both reviewer and contributor time.
I think it's fine if you happen to clean up the code or update the versions anyways though. Since you did the work here already I guess it's fine, but others should definitely not be encouraged to do this manually until the automated migration is done (it doesn't work automatically for all packages) :slightly_smiling_face:

OK, I will stop this thread for now and concentrate in the version bumps.

@ofborg build xorriso dosbox-staging

Are you intending on having this reviewed and merged after all? Or are you using this as a testing ground before splitting it into smaller cleanup PRs? If so this should probably be turned into a draft.


invalid json response - still working?

Hey i saw your project but get the following error:
FetchError: invalid json response body at https://reiseauskunft.bahn.de/bin/query.exe/dn?tbpMode=1&date=%2B1&getstop=1 reason: Unexpected token < in JSON at position 0
Is it still working or broken since the latest update of the db-website?
Thx/Danke!

I am sorry to hear that. Cant check my datalogger right now, but I will next week.

Yes, I do have the same issue. I opened an issue here: https://github.com/derhuerst/db-prices-cli/issues/4

as there was no real progress i rewrote the data collection part quick and dirty here
This is still work in progress

I reworked it pretty much, check the updated repo, should run again


Tensorflow has not supported operator [MaxPoolWithArgmax] with name [resnet_model/MaxPoolWithArgmax].

Platform :arm centos
Python version:3.7.5
Source framework with version (like Tensorflow 1.4.1 with GPU):tensorflow 1.15
when tried ”mmconvert -sf tensorflow -iw resnet50_910to310.pb --dstNodeName resnet_model/final_dense -df caffe -om tf_resnet --inNodeName input_data --inputShape 224,224,3”，i get the erros as below:
2020-08-14 22:21:28.678883: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/tensorflow/tensorflow_frozenparser.py:175: The name tf.train.export_meta_graph is deprecated. Please use tf.compat.v1.train.export_meta_graph instead.
Tensorflow has not supported operator [MaxPoolWithArgmax] with name [resnet_model/MaxPoolWithArgmax].
Traceback (most recent call last):
File "/usr/local/python3.7.5/bin/mmconvert", line 8, in
sys.exit(_main())
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/_script/convert.py", line 102, in _main
ret = convertToIR._convert(ir_args)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/_script/convertToIR.py", line 120, in _convert
parser.run(args.dstPath)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/common/DataStructure/parser.py", line 22, in run
self.gen_IR()
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/tensorflow/tensorflow_frozenparser.py", line 397, in gen_IR
self.rename_UNKNOWN(current_node)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mmdnn/conversion/tensorflow/tensorflow_frozenparser.py", line 459, in rename_UNKNOWN
assert False
AssertionError
any one help me to figure out what is the problem?  thanks .

I am having something similar and I guess mmdnn doesn't recognize the operators created with Keras API rather than pure TensorFlow API

I am having something similar and I guess mmdnn doesn't recognize the operators created with Keras API rather than pure TensorFlow API


Don't get XML completion in project file when you have another language installed

Installed English & Japanese, don't get IntelliSense when editing project file:

@tmeschter Talk to @RaulPerez1 about how this thing is registered. Thought there was another bug on this but I cannot find it.

Confirmed that if switch back to English, then I get intellisense.

#1719 is the bug you're thinking about and has a workaround. There's a bug on the VS side tracking the bug (it's a setup issue)

Dup'ing against: https://github.com/dotnet/roslyn-project-system/issues/1719


Implement reworked admin-connections

Worth noting that when we make the jump over to this over the old admin-connections, the other modules that rely on shared connection data will need minor updates according to the updated structure of messages.

Corresponding ACA-Py Plugin PR: https://github.com/hyperledger/aries-acapy-plugin-toolbox/pull/34


Add kfunc support

Adding support to specify kfunc/kretfunc attach points.
The 'kfunc' attach point is a bpftrace 'code name' for
function trampolines, which is a new way to attach BPF
program to kernel functions.
The trampolines are closely binded with BTF kernel data,
and that allows them to know the name and type of arguments.
We are able to use function arguments by name as standard
bpftrace variables with '$' prefix', like:
  # bpftrace  -e 'kretfunc:fget { printf("fd %d\n", $fd); }'
  Attaching 1 probe...
  fd 3
  ...

We are also able to access return value via $ret variable
(together with kfunc function arguments), like:
  # bpftrace -e 'kretfunc:fget { printf("fd %d name %s\n", $fd, str($ret->f_path.dentry->d_name.name));  }'
  Attaching 1 probe...
  fd 9 name _journal.cpython-37m-x86_64-linux-gnu.so
  ...

As I already mentioned and presented above the kretfunc has access
to function arguments as well and could speed up some scripts,
that need function arguments at the return probe.
The kfunc and kretfunc is also a starting point for using BTF
powered argument passing to kernel helpers, which would allow
us to do fast filepath helper, that operates on 'struct file'
pointer directly. I have the code ready both for kernel and
bpftrace and it will follow once this change in.

Nice! Very useful to have.
I'm not convinced by the use of "hidden" variables for this purpose. I can see this cause confusion, e.g. when you start using bpftrace scripts between kernel versions and the kernel function signature change starts clashing with your existing . It also introduces another way to access args. We already have arg0..9, retval for k/uprobes and the args struct for tracepoints.
Maybe we can use args for this too? E.g. kretfunc:fget {@=args->fd } instead of kretfunc:fget { @=$fd }.

I'm not convinced by the use of "hidden" variables for this purpose. I can see this cause confusion, e.g. when you start using bpftrace scripts between kernel versions and the kernel function signature change starts clashing with your existing . It also introduces another way to access args. We already have arg0..9, retval for k/uprobes and the args struct for tracepoints.
Maybe we can use args for this too? E.g. kretfunc:fget {@=args->fd } instead of kretfunc:fget { @=$fd }.

ok, that seems like good idea, should not be a problem to do that

Awesome, was actually just thinking about this. (Haven't looked at code changes yet)
cc @4ast .

Would be interesting to see if we can do something about https://github.com/iovisor/bpftrace/blob/master/src/main.cpp#L64 with kfunc. Alexei says setup should be faster but teardown cost is not yet profiled.

Would be interesting to see if we can do something about https://github.com/iovisor/bpftrace/blob/master/src/main.cpp#L64 with kfunc. Alexei says setup should be faster but teardown cost is not yet profiled.

hum, do you mean BPFTRACE_MAX_PROBES?

hum, do you mean BPFTRACE_MAX_PROBES?

Yeah, I think main.cpp changed and my link points to something else now

the test failures seem ok.. there's clang format check,
which would force me to change code I did not changed,
so I think it's ok
and there's following one:
tar: invalid tar magic
The command '/bin/sh -c tar xvf /bcc.tar.gz' returned a non-zero code: 1

I re-pushed and now it's only the clang error.. so we should be good ;-)

rebased..

other than the issue around listing, LGTM.
Porting tcpconnect was quite easy,  s/$sk/args->sk/ and it works :)

it's again only clang format failures

Can we access a return value  if there is a function that use ret as an argument name?

Can we access a return value if there is a function that use ret as an argument name?

damn..  you're right :-
how about using 'retval' as we do for kprobe/uprobe ? I'll check on that
thanks

Looks like the listing doesnt fully work yet, it only lists with a regex present now
[vagrant@localhost bpftrace]$ sudo bpftrace --version
bpftrace v0.9.4-146-g1940-dirty
[vagrant@localhost bpftrace]$ sudo bpftrace -l 'kfunc:*' | wc -l
21863
[vagrant@localhost bpftrace]$ sudo bpftrace -l  | grep -c kfunc
0

The -lv from the reference guide doesnt work either.

Can we access a return value if there is a function that use ret as an argument name?

damn.. you're right :-
how about using 'retval' as we do for kprobe/uprobe ? I'll check on that

Does accessing arguments make sense for a kretfunc? Or does the kernel somehow store the input args until the return probes have been evaluated?

Looks like the listing doesnt fully work yet, it only lists with a regex present now
[vagrant@localhost bpftrace]$ sudo bpftrace --version
bpftrace v0.9.4-146-g1940-dirty
[vagrant@localhost bpftrace]$ sudo bpftrace -l 'kfunc:*' | wc -l
21863
[vagrant@localhost bpftrace]$ sudo bpftrace -l  | grep -c kfunc

oops. sry, will fix

The -lv from the reference guide doesnt work either.

dtto

Can we access a return value if there is a function that use ret as an argument name?

damn.. you're right :-
how about using 'retval' as we do for kprobe/uprobe ? I'll check on that

Does accessing arguments make sense for a kretfunc? Or does the kernel somehow store the input args until the return probes have been evaluated?

it's actually one of the nice kfunc/trampoline features, that you can
access function arguments in return handler, because there's plenty
of tools that capture function arguments and process them in the return
handler.. so you could use just one probe instead of two

Using retval seems good.

I found two problems:

reg() is allowed to use in kfunc and kretrunc. This should be checked in the semantic analyzer.

% sudo ./src/bpftrace -e 'f:vfs_open { printf("%d\n", reg("ip"));  }' -v
BTF: using data from /sys/kernel/btf/vmlinux
Attaching 1 probe...

Error log:
0: (bf) r6 = r1
1: (b7) r1 = 0
2: (7b) *(u64 *)(r10 -16) = r1
last_idx 2 first_idx 0
regs=2 stack=0 before 1: (b7) r1 = 0
3: (79) r1 = *(u64 *)(r6 +128)
invalid bpf_context access off=128 size=8
processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

Error loading program: kfunc:vfs_open

probe builtin does not work.

% sudo ./src/bpftrace -e 'f:vfs_open { printf("%s\n", probe);  }' -v
BTF: using data from /sys/kernel/btf/vmlinux
terminate called after throwing an instance of 'bpftrace::WildcardException'
  what():  Wildcard matches aren't available on probe type 'kfunc'
zsh: abort      sudo ./src/bpftrace -e 'f:vfs_open { printf("%s\n", probe);  }' -v

sudo ./src/bpftrace -e 'f:vfs_open { printf("%s\n", probe);  }' -v

ok, reg needs to be banned for kfunc.. for probe, I guess it'd be
better to make the expansion working now

sudo ./src/bpftrace -e 'f:vfs_open { printf("%s\n", probe);  }' -v

ok, reg needs to be banned for kfunc.. for probe, I guess it'd be
better to make the expansion working now

after some more thought, I'm not enabling kfunc expansion at the moment:

expansion is done in code generation and I'll need it already in semantic analyser,
to get all the arguments.. so not small factoring would be needed
we'd need to check that all the expansed functions have the same arguments,
which in my opinion won't be the case for most of the kernel functions, so
I don't think it's too useful feature
if really needed we could do that in separate PR

I'm not enabling kfunc expansion at the moment:

I think that is fine for now.

we'd need to check that all the expansed functions have the same arguments

Indeed, if we want to attach the same kfunc BPF program to multiple functions, then functions must have the same arguments. On the other hand, bpftrace supports attaching a single program to multiple functions that have different arguments by creating a BPF program for each function. e.g., :
% sudo ./src/bpftrace -e 'tracepoint:syscalls:sys_enter* { @[probe] = count(); }'
Attaching 332 probes...
^C

(though in this case, I believe there is no need to create a BPF program for each function)
I think this is useful. For example, we would be able to create kfunc version of vfscount.bt:
% sudo ./src/bpftrace -e 'f:vfs_* { @[probe] = count(); }'
stdin:1:1-8: ERROR: Failed to resolve kfunc args.
f:vfs_* { @[probe] = count(); }
~~~~~~~

(By the way, printing like "wildcard expansion is not supported" would be easy to understand the rror.)

I think this is useful. For example, we would be able to create kfunc version of vfscount.bt:
% sudo ./src/bpftrace -e 'f:vfs_* { @[probe] = count(); }'
stdin:1:1-8: ERROR: Failed to resolve kfunc args.
f:vfs_* { @[probe] = count(); }
~~~~~~~

aaah right.. when we are not using args->xxx we don't need that check ;-)
I'll do the wildcards expansion for kfunc in separate PR

(By the way, printing like "wildcard expansion is not supported" would be easy to understand the rror.)

ok, changing to this:
# ./src/bpftrace -e 'f:vfs_* { @[probe] = count(); }'
stdin:1:1-8: ERROR: wildcard expansion is not supported for kfunc/kretfunc probes
f:vfs_* { @[probe] = count(); }
~~~~~~~

Looks good to me. We can make improvements later. kfunc is very interesting!

I am tempted to merge this and do the 0.10 release. This one is still has some improvements/issues but those can be done in a later version too. The basics look good to me and this one is getting quite big.
This is mostly a new code path too, so getting bad regressions into the release is unlikely too (I don't see any).
@danobi  any objections?

@danobi any objections?

Nope, I think this should be fine to merge for 0.10.0.

Can you rebase again? Master was broken for a bit.


Is the voc_2012_trainaug.json and voc_2012_val.json files obtained by converting the voc's xml annotations to json format?

The VOC2012 dataset doesn't provide json annotation files, so I'm wondering if the json files needed in the model are generated additionally? If so is it possible to provide the corresponding generated files?

Thanks for your attention.
I have placed the json files in this link, please feel free to use it. Link

Thanks a lot .

🎉Feel free to reopen if you meet some questions~

Hi, I'm having another problem during training. There is a new issue in training about the number of elements in the output of the model with the following error message.
4%|███▉                                                                                | 1940/45000 [14:02<5:01:53,  2.38it/s][Dec26-18-30-01_serverA_step][resnet50_voc.yaml][Step 1941 / 45000]
loss: 1.431148, lr: 0.000250 time: 0.434069, eta: 5:11:31
bag_loss: 0.257181, pcl_loss: 0.739873, cls_loss: 0.191624, iou_loss: 0.127284
4%|███▉                                                                                | 1941/45000 [14:03<5:11:52,  2.30it/s]
Traceback (most recent call last):
File "./tools/train.py", line 464, in
main()
File "./tools/train.py", line 432, in main
net_outputs = model(**input_data)
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/～/CIM-main/lib/nn/parallel/data_parallel.py", line 108, in forward
outputs = [self.module(*inputs[0], **kwargs[0])]
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/～/CIM-main/lib/modeling/model_builder.py", line 187, in forward
using_CIM=self.using_CIM[i])
ValueError: too many values to unpack (expected 3)
I have built the environment by following the INSTALL.md. My torch, cuda, and python versions are as follows:
torch：1.10.0+cu111
cuda：11.1
python：3.6.13
Looking forward to your answer.

Hi, I'm having another problem during training. There is a new issue in training about the number of elements in the output of the model with the following error message. 4%|███▉ | 1940/45000 [14:02<5:01:53, 2.38it/s][Dec26-18-30-01_serverA_step][resnet50_voc.yaml][Step 1941 / 45000] loss: 1.431148, lr: 0.000250 time: 0.434069, eta: 5:11:31 bag_loss: 0.257181, pcl_loss: 0.739873, cls_loss: 0.191624, iou_loss: 0.127284 4%|███▉ | 1941/45000 [14:03<5:11:52, 2.30it/s] Traceback (most recent call last): File "./tools/train.py", line 464, in main() File "./tools/train.py", line 432, in main net_outputs = model(**input_data) File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/～/CIM-main/lib/nn/parallel/data_parallel.py", line 108, in forward outputs = [self.module(*inputs[0], **kwargs[0])] File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/～/CIM-main/lib/modeling/model_builder.py", line 187, in forward using_CIM=self.using_CIM[i]) ValueError: too many values to unpack (expected 3)
I have built the environment by following the INSTALL.md. My torch, cuda, and python versions are as follows: torch：1.10.0+cu111 cuda：11.1 python：3.6.13 Looking forward to your answer.

My bad. It seems like a mistake I made when cleaning up my code? You can fix it quickly, just change this line of code. Link

Hi, I'm having another problem during training. There is a new issue in training about the number of elements in the output of the model with the following error message. 4%|███▉ | 1940/45000 [14:02<5:01:53, 2.38it/s][Dec26-18-30-01_serverA_step][resnet50_voc.yaml][Step 1941 / 45000] loss: 1.431148, lr: 0.000250 time: 0.434069, eta: 5:11:31 bag_loss: 0.257181, pcl_loss: 0.739873, cls_loss: 0.191624, iou_loss: 0.127284 4%|███▉ | 1941/45000 [14:03<5:11:52, 2.30it/s] Traceback (most recent call last): File "./tools/train.py", line 464, in main() File "./tools/train.py", line 432, in main net_outputs = model(**input_data) File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/～/CIM-main/lib/nn/parallel/data_parallel.py", line 108, in forward outputs = [self.module(*inputs[0], **kwargs[0])] File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/～/CIM-main/lib/modeling/model_builder.py", line 187, in forward using_CIM=self.using_CIM[i]) ValueError: too many values to unpack (expected 3)
I have built the environment by following the INSTALL.md. My torch, cuda, and python versions are as follows: torch：1.10.0+cu111 cuda：11.1 python：3.6.13 Looking forward to your answer.

My bad. It seems like a mistake I made when cleaning up my code? You can fix it quickly, just change this line of code. Link

Thank you for replying so quickly.

Hi, I got the contents of the Dec26-23-17-22_serverA_step folder after I finished training. The folder address for testing is Mar22-00-46-22_user-Super-Server_step and the test requires Mar22-00-46-22_user-Super-Server_step/ckpt/model_step44999.pth to exist, so I'm putting Dec26 -23-17-22_serverA_step folder to get the model weights to Mar22-00-46-22_user-Super-Server_step.
Now the prompt needs the file Mar22-00-46-22_user-Super-Server_step/test/model_step44999/detections.pkl. Can you tell me which file the detections.pkl file corresponds to in the output or how to get this file. Below is the error message:
INFO evaluation.py: 143: Called with args:
INFO evaluation.py: 144: Namespace(cfg_file='./configs/resnet50_voc.yaml', dataset='voc2012sbdval', output_dir=None, result_path='./Outputs/resnet50_voc/Mar22-00-46-22_user-Super-Server_step/test/model_step44999/detections.pkl', set_cfgs=[], total_process=24)
args.result_path: ./Outputs/resnet50_voc/Mar22-00-46-22_user-Super-Server_step/test/model_step44999/detections.pkl
Traceback (most recent call last):
File "tools/evaluation.py", line 147, in
assert os.path.exists(args.result_path), 'result_path doesnot exit'
AssertionError: result_path doesnot exit

the prompt needs the file Mar22-00-46-22_user-Super-Server_step/test/model_step44999/detections.pkl. Can you tell me which file the detections.pkl file corresponds to in the output or how to get this file. Below is the error message:
INFO evaluation.py: 143: Called with args:
INFO evaluation.py: 144: Namespace(cfg_file='./configs/resnet50_voc.yaml', dataset='voc2012sbdval', output_dir=None, result_path='./Outputs/resnet50_voc/Mar22-00-46-22_user-Super-Server_step/test/model_step44999/

You need to update these four values cfg_file, output_file, dataset, iter_time according to your needs. This is the rule for all bash files. https://github.com/ZechengLi19/CIM/blob/33f695de12a04e3a9da434bccf21824d20a6ee7d/scripts/eval_CIM.sh#L1-L5
In your case, you need to change .bash file as
# Editable
cfg_file=./configs/resnet50_voc.yaml
output_file=./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step
dataset=voc2012sbdval
iter_time=model_step44999

Maybe there are still some problems with the above changes, because I don’t know your file structure, so you’d better write output_file as an absolute path.

Thanks, I've changed the output_file path of the .bash file, but I'm still not sure about . /Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detections.pkl file where it comes from, it is this path result_pkl=${output_file}/test/$ {iter_time}/detections.pkl corresponding to detections.pkl.
Here is the full error message:
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
INFO json_dataset.py:  71: /home/～/CIM-main/data/VOC2012/annotations/voc_2012_val.json
INFO subprocess.py:  87: detection range command 0: python /home/～/CIM-main/tools/test_net.py --range 0 725 --cfg
./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detection_range_config.yaml --set TEST.DATASETS '("voc_2012_sb
dval",)' TEST.PROPOSAL_FILES '("./data/cob/voc_2012_val.pkl",)' --output_dir ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/m
odel_step44999 --load_ckpt ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/ckpt/model_step44999.pth
INFO subprocess.py:  87: detection range command 1: python /home/～/CIM-main/tools/test_net.py --range 725 1449 --c
fg ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detection_range_config.yaml --set TEST.DATASETS '("voc_2012
_sbdval",)' TEST.PROPOSAL_FILES '("./data/cob/voc_2012_val.pkl",)' --output_dir ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/tes
t/model_step44999 --load_ckpt ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/ckpt/model_step44999.pth
INFO subprocess.py: 127: # ---------------------------------------------------------------------------- #
INFO subprocess.py: 129: stdout of subprocess 0 with range [1, 725]
INFO subprocess.py: 131: # ---------------------------------------------------------------------------- #
INFO test_net.py:  93: Called with args:
INFO test_net.py:  94: Namespace(cfg_file='./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detection_range_con
fig.yaml', dataset='', load_ckpt='./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/ckpt/model_step44999.pth', load_detectron=None, m
ulti_gpu_testing=False, output_dir='./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999', range=[0, 725], set_cfgs
=['TEST.DATASETS', '("voc_2012_sbdval",)', 'TEST.PROPOSAL_FILES', '("./data/cob/voc_2012_val.pkl",)'], vis=False)
single GPU
Traceback (most recent call last):
File "/home/～/CIM-main/tools/test_net.py", line 123, in
merge_cfg_from_file(args.cfg_file)
File "/home/～/CIM-main/lib/core/config.py", line 678, in merge_cfg_from_file
yaml_cfg = AttrDict(yaml.safe_load(f))#.load(f))
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/init.py", line 125, in safe_load
return load(stream, SafeLoader)
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/init.py", line 81, in load
return loader.get_single_data()
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/constructor.py", line 51, in get_single_data
return self.construct_document(node)
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/constructor.py", line 55, in construct_document
data = self.construct_object(node)
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/constructor.py", line 100, in construct_object
data = constructor(self, node)
File "/home/～/tools/miniconda3/envs/cim/lib/python3.6/site-packages/yaml/constructor.py", line 429, in construct_undefined node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:utils.collections.AttrDict'
in "./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detection_range_config.yaml", line 1, column 1
Traceback (most recent call last):
File "tools/test_net.py", line 169, in
check_expected_results=True,flag=cfg.transform_mode)
File "/home/～/CIM-main/lib/core/test_engine.py", line 142, in run_inference
all_results = result_getter()
File "/home/～/CIM-main/lib/core/test_engine.py", line 119, in result_getter
multi_gpu=multi_gpu_testing,flag=flag
File "/home/～/CIM-main/lib/core/test_engine.py", line 169, in test_net_on_dataset
args, dataset_name, proposal_file, num_images, output_dir
File "/home/～/CIM-main/lib/core/test_engine.py", line 223, in multi_gpu_test_net_on_dataset
args.load_ckpt, args.load_detectron, opts
File "/home/～/CIM-main/lib/utils/subprocess.py", line 107, in process_in_parallel
log_subprocess_output(i, p, output_dir, tag, start, end)
File "/home/～/CIM-main/lib/utils/subprocess.py", line 145, in log_subprocess_output
assert ret == 0, 'Range subprocess failed (exit code: {})'.format(ret)
AssertionError: Range subprocess failed (exit code: 1)
INFO evaluation.py: 143: Called with args:
INFO evaluation.py: 144: Namespace(cfg_file='./configs/resnet50_voc.yaml', dataset='voc2012sbdval', output_dir=None, result_path='./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detections.pkl', set_cfgs=[], total_process=24)
args.result_path: ./Outputs/resnet50_voc/Dec26-23-17-22_serverA_step/test/model_step44999/detections.pkl
Traceback (most recent call last):
File "tools/evaluation.py", line 147, in
assert os.path.exists(args.result_path), 'result_path doesnot exit'
AssertionError: result_path doesnot exit

detections.pkl is generated by https://github.com/ZechengLi19/CIM/blob/33f695de12a04e3a9da434bccf21824d20a6ee7d/scripts/eval_CIM.sh#L12-L16
Please check version of PyYAML, because I found this error occur
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:utils.collections.AttrDict'


Duplicate test file in package folder returns an unhandled error message

Description:
Bad, sad error when we have duplicate test file in the package folder. Need to handle the error properly.
$ ballerina build
Compiling source
    jerad/foo:0.0.1

Compiling tests
    jerad/foo:0.0.1
error:jerad/foo:0.0.1\main_test.bal:8:10: redeclared symbol 'beforeSuiteFunc'
error:jerad/foo:0.0.1\main_test.bal:15:10: redeclared symbol 'beforeFunc'
error:jerad/foo:0.0.1\main_test.bal:26:10: redeclared symbol 'testFunction'
error:jerad/foo:0.0.1\main_test.bal:34:10: redeclared symbol 'afterFunc'
error:jerad/foo:0.0.1\main_test.bal:42:10: redeclared symbol 'afterSuiteFunc'
ballerina: Oh no, something really went wrong. Bad. Sad.

There should be a file named "ballerina-internal.log" in the current directory.
If you are able to share with us the code that broke Ballerina then
we would REALLY appreciate if you would report this to us:
go to https://github.com/ballerina-platform/ballerina-lang/issues and
create a bug report with both this log file and the sample code.

We thank you for helping make us better dancers.

Steps to reproduce:

Run ballerina init -i and follow steps

$ ballerina init -i
Create Ballerina.toml [yes/y, no/n]: (y) y
Organization name: (jerad)
Version: (0.0.1)
Ballerina source [service/s, main/m, finish/f]: (f) m
Package for the main : (no package) foo
Ballerina source [service/s, main/m, finish/f]: (f) f

Ballerina project initialized

Duplicate test folder inside the package folder and rename it to something else
foo/sample/sample.bal

Run ballerina build

Affected Versions:
0.981.1
OS, DB, other environment details and versions:
Windows 10
Suggested Labels (optional):
[Component/Testerina] [Type/Bug]

Yes tested on Ballerina 0.981.2-SNAPSHOT. And the issue is fixed.


Refactored get_plugin_aliases function, also added one test case

This change is

Coverage increased (+0.06%) to 72.136% when pulling da4a37cdaef02e3a24de3f189d6f7284dbd8c2e5 on MichaelYusko:refactor/change_get_plugin_aliases into 099d8dc3ca2e70f900cd06b8647d3ab412241006 on openprocurement:ea_core_master.


tuic in v2rayng

I want to use Tuic in windows trough V2rayNg  but when use "scan QR code" it's brings nothing.
Is it a way to add it from "add a custom configuration server" in V2rayNg?

maybe your client is not updated.
I recommend you to use nekoray or Furious.


chore(main): release 1.6.4

:robot: I have created a release beep boop
1.6.4 (2023-11-29)
Bug Fixes

Github workflow (9452092)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/agence-adeliom/adeliom-webcomponents/releases/tag/v1.6.4 :sunflower:


Unable to compile or fetch from unpkg.com

The unpkg.com link produces a 500 Server Error and compiling hangs at puppeteer:
$ yarn install
yarn install v1.17.0
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
[-/5] ⠈ waiting...
[-/5] ⠈ waiting...
[-/5] ⠈ waiting...
[-/5] ⠈ waiting...
[5/5] ⠈ puppeteer

Is this still happening? I can access this just fine<EMAIL_ADDRESS>If it's blocked or something you could try these links:

<EMAIL_ADDRESS><EMAIL_ADDRESS>

It works now!


[FEATURE] A scroll to top button on all the  pages

Description
Here I speak of a small component which could be rendered on all the pages, now when the user has scrolled down, instead of manually scrolling down, we could add a feature that causes a small scroll-up button to appear on the bottom right corner of the page when the navbar is out of view so now if the user wants to go back to the top of the page, instead of scrolling up manually, we the user can just click on this button and reach back up smoothly.
Refer to the following video for a better understanding of the feature I speak of, I implemented this in another r project
https://github.com/tamalCodes/Milan/assets/95973644/106cf25d-09e9-4bad-9b97-cebd22297dcc
Screenshots
https://github.com/tamalCodes/Milan/assets/95973644/f8395dd8-bfa7-4adb-8d1b-763e4f02dcec
🥦 Browser
Google Chrome
Checklist ✅

[X] I checked and didn't find similar issue
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (blank for no)

I would like to work on this issue under gssoc'23

This has been assigned to #833 , please do wait - if they are not working on it by the next 5 days - I will revert back to you.


Type conversion for arrays

Hey - thanks again for this library.
This branch implements conversion of n-dimensional arrays going into and out of the database, with the help of some modified parsing code ripped out of pgjdbc.
I don't know whether this is a feature you're interested in incorporating, however either way I'd very much appreciate some feedback on how best to integrate it, either as a feature or a separate library.
High level stuff:
There's an object used for both insertion and retrieval, PgArray, which is given either an array (for insertion), which it converts to a string using a pretty simple and predictable couple of methods, or is given a string (for parsing) and an OID, and returns an Object[] (arbitrarily nested), where the leaves are the result of delegating the element parsing to DataConverter.
That's the main sticking point as far as integration - the array converter's constructor accepts a DataConverter (which is constructed vanilla in the tests, just to get the thing working), but ideally we want it to have access to the pool's converter, so it can parse arrays of types which are themselves converted by user code.
I wrote this for use from Clojure/postgres.async, but touching DataConverter seemed like the wrong thing to do, and I think I may be missing something as far as how, in Java, it would practical to have the .get which happens during the CLJ map conversion correctly invoke ArrayConverter.  Obviously if this feature ends up in your library, it would be possible to add more clauses to the switch in toObject
Anyway, I'm super open minded about making changes, etc. - so please let me know if you can imagine any problems.
Todo:
It'll be a quick change, but I think strings are constructed in a few places without passing in a codec, which could cause problems.  Will revisit tomorrow.
More notes:
The largest changes I made to the pgjdbc utilities are as follows:

The JDBC stuff queries the database for type-specific array delimiters before parsing.  I wanted to avoid doing that at all costs.  I'm also unable to determine in what situations the literal delimiter for an array would be anything other than a comma, so I'm mostly not worrying about it.  I checked the catalog in a few varied PG installations for type delimiters, they're all commas, and the only references I can find online are about alternative delimiters with can be used when parsing/constructing strings from/to arrays, which is something different.
I removed pretty much everything from buildArray - including the facility for constructing primitive arrays, and for declaring the arrays as anything but Object[].  Without having a mapping from OIDs to converter return types (including types constructed by custom converters), I think the only other approach to this would be to defer array creation until we can reflect the return type.  I'm open-minded about this.
PG 8.3+ supports NULL as an array element.  I removed the version checking, so the NULL will go to the DB regardless of its version.  It works fine both in-and-out post-8.3.

Is there anything I can do to make this easier for you to digest?

Sorry for the delay, I really am interested in getting this merged.
Here are some ideas:

Method for getting a single-dimension array from Row could be <T> T[] getArray(String column, T typeClass).
Both single- and multi-dimensional arrays could be accessed with get. Example: int[][] ids = row.get("ids", int[][].class).
Array support could be added into DataConverter as a core type (not via converter)
The parsing code looks pretty heavy and complex, I might consider special casing parsing of String arrays and handling other cases with a simpler code path.

DataConverter could maybe look something like this:
// DataConverter.java
public <T> T toArray(Optional<Class<T>> dimensionalArrayType, Oid oid, byte[] value):
    switch(oid) {
        case INT2_ARRAY:
        case INT4_ARRAY:
        case INT8_ARRAY:
            return ArrayConversions.toArray(dimensionalArrayType, oid, value, NumericConversions::toLong);

ArrayConversions would then handle parsing of (nested) '{' and '}' and calling the supplied function to create an object from the string value. The dimensionalArrayType parameter (eg. int[][].class) can be missing since the clojure version doesn't care about types.
How does this sound to you?

Hey - I'll need to re-familiarize myself with the code, but it sounds good to me.  I'll take a look at making the changes this week.

I updated this to be in line with your comments.  Thanks a bunch for the feedback.
Some notes/clarifications:

The implementation uses boxed types everywhere for representing values retrieved from Postgres, due the permissibility of null values in Postgres arrays.  You could make an argument for not using Java arrays at all, and using a generic collection - however in general I've erred on the side of not altering pgjdbc's fundamental implementation strategy.

I've adjusted DataConverter to do as you suggested, and added clauses for the handling of arrays of all of the currently-supported types - with the exception of bytea[].  As the converter for that function returns a byte[], and buildArray gets its depth cues only from the databases values, this would have to be handled with some pretty ugly special-casing, I believe.

The type passed to DataConverter.toArray, e.g. Long[][] amounts basically to a runtime assertion about the return values of the converter which was delegated to for individual elements.  I implemented your example above of delegating INT2, INT4 and INT8 elements to NumericConversions::toLong, however this means that users must call getArray(..., Long[].class), even though it may not be immediately obvious that an INT2 would map to a long (because it doesn't for single values).  Maybe the types should be symmetric?

Internally, the element-converter method reference (passed to ArrayConversions) is typed as returning Object.  I think this could only be more specific if toArraywas parameterised over the type of the elements, as well as the type of the array, but I don't think it matters.

Class.isArray is the clause in fromObject which determines whether to use ArrayConversions - accordingly, primitive arrays can be inserted.

The only point I did not address was an unquoted code path for buildArrayList.  If/when everything else makes sense, I can have a look at simplifying it.

I'm far from comfortable with Java's type system when things get complicated, so feel free to tell me how weird this is.

P.S. isArray() as the fromObject guard is too specific, if we want to be able to insert regular collections.  There could be another clause which looks at interfaces.

This looks good.

Change the numeric types to be symmetric. Delegating all numeric types to long would have worked with primitive types, but since we have to used boxed types (I forgot NULLs in arrays) all numeric types must be handled separately.
Use UTF-8 with String::getBytes

Commit the fixes and this is ready for merge.
I will probably at some point revisit the decision of using byte[] as the internal data format, the getBytes/fromBytes shuffing is pretty ugly but there is no way around it in the current api.

Done.  I added a few more tests for the changes.

Merged, thanks!


make clean-launch-dev removes the dash service, but not the deployment or the pod(s)...

...and make launch-dev doesn't recreate any dash-related resources
I think this has to do with the way things are tagged (suite=pachyderm?)
My inclination is that if make launch-dev isn't going to deploy the dash (which i think we said it shouldn't), then make clean-launch-dev shouldn't remove it

Is this done or ever going to be done?

AFAIK this has not been done, and still should be. I don't think it should take long for someone who has a better understanding of how this deploy/undeploy code works and how we group pachyderm parts of the system (not sure if there's anything other than suite=pachyderm?). It may just be a matter of adding the right tag in the right place in the deploy manifest

Just ran into weirdness from this again:
$ make clean-launch-dev
# check that kubectl is installed
which kubectl
/home/gabriel/.local/opt/kubernetes/bin/kubectl
pachctl deploy local -d --dry-run | kubectl  delete --ignore-not-found -f -
kubectl  delete rc -l suite=pachyderm
No resources found
kubectl  delete svc -l suite=pachyderm
No resources found
$ kubectl delete all -l suite=pachyderm  # manually delete remaining pods
pod "dash-3498131396-mjg3h" deleted
deployment "dash" deleted

@jdoliner is this something that's easy to fix? Is there something that you're doing with the other pach pods that makes them go away that I could be doing with this dash pod?

This seems to be fixed now, just tested it.


Can you please update README.txt to how to run this repo

@abizovnuralem Can you please update README.txt to how to run this repo

Ofc!


Distinguish observations from management actions

https://github.com/riparias/rato-occurrences/issues/69 is partly linked to the query we use to trigger a GBIF download..
The mapping of rato vzw data can be improved. But one thing is sure: we should check that the download query is good enough to catch the difference between occurrences describing observations (to be imported/shown in our early warning tool) and the others describing management actions and so not be imported/shown.

Related to riparias/rato-occurrences#86 and riparias/rato-occurrences#77