Propagate trace ID to database queries

I initially opened this issue against Spring Sleuth, but after some discussion we decided it belonged over here instead. Let me know if that's wrong, though.
We've found that when troubleshooting slow queries it can sometimes be difficult to definitively link a given query back to the operation that initiated it. This is especially true for DBAs that troubleshoot problems using a database-first perspective, where they hunt down slow queries using the database's profiling tools instead of application logs.
Do you think it makes sense to prepend the trace ID as a comment on queries? That would allow database queries to be tied back to a trace ID, which in turn would link them back to the operation that produced them. For example:
-- X-B3-TraceId: b763b53c5c6ba343
SELECT * FROM foo WHERE baz = true;

Obviously, if there's a better way to associate metadata with a query than a literal comment then I'd advocate going in that direction. Please let me know what you think, and thanks for maintaining this library.

Brave currently relies on https://github.com/p6spy/p6spy to trace database calls (some discussion #824 took place to see if https://github.com/r2dbc/r2dbc-proxy would be a better fit but that's beside the point). I guess such functionality would belong more there, providing a hook for brave to call and set the trace/span id. https://github.com/p6spy/p6spy/tree/master/src/main/java/com/p6spy/engine/wrapper provides a bunch of wrappers, so it seems doable to modify the statement. Have you checked that prepending a comment is portable across databases and will not break any queries ?
Definitely a nice idea !

I would probably advise against using X-B3- as that is a bit too specific to a propagation format which might not be in use. If this is using correlation, I'd suggest traceId and spanId like our MDC integration. On that note, I suppose I agree with the implementation suggestion of @jorgheymans (using a database wrapper)

Have you checked that prepending a comment is portable across databases and will not break any queries ?

I have not, but that's definitely a good idea! My naive view was that the comment syntax was consistent across databases and couldn't break anything. But you're right, that's something we'd want to make sure of before making this change.

one funny thing is that in some areas comments themselves end up as statements (I've seen this in mysql). Which means we have to be careful not to trace the comments :P
At least in the beginning, and possibly forever, I think this should be an opt-in filter based approach so that sites can practice getting it right. It might be small enough to copy/paste until such experience accrues.

Any value in not doing a generic approach like using the comment field, but rather trying to hook into the specific auditing mechanisms of each database type directly ? Just thinking out loud ...

@jorgheymans Yeah, that would be awesome! I mentioned in my initial comment that if there was a better way to tie metadata (like a trace ID) to a SQL query then I'm all for going with that approach.
What specific auditing mechanism were you thinking of, here? I'm mostly familiar with Postgres which (AFAIK) doesn't have one, but if others do then that would be a really good integration.

mysql binlog can log sql comment

@brothertao can you give a java example using mysql interceptor? I think this is a neat idea

ysql interceptor? I thin

@adriancole i use mybatis plugin to interceptor dml sql,add trace id to sql comment
u also  can intercept sql in jdbc api i think

if you have a sample project please link it back. It is possible we can do this, but we'd have to choose what to add. Probably traceId=abcd because this decoupled from propagation. Ex in logs we use the traceId key.
I've found something recent about a system who uses yaml for payload because it permits comments. They apparently use comments for traceIds also. I will try to find it.
In any case this seems a correlation feature and without any special field (JDBC wouldn't know but maybe mysql does? no idea), a practice of opt-in on comments might make sense. Probably as you said a filter as people will likely be careful to not add comments to.. comment statements :D
but stepping back, please post an example of what works for you. I think this is basically special-casing what can already be done with filters.

   // add traceid to sql comment
    String sql = boundSql.getSql();

    if (mappedStatement.getSqlCommandType() != SqlCommandType.SELECT) {
      sql = binlogTraceService.generateTraceComment(sql);
    }

@brothertao  what is the literal SQL you would generate.. curious

like this:
/*__trace_start__{"traceId":"bd59aad5c8bc0895e6e1ef831b615cf7","userId":0}__trace_end__*/ insert into
 msg_log (`id`,`data`) values (?,?)

thanks @brothertao and how do you use this data on the other side? grep? or something different?

like this:
https://dba.stackexchange.com/questions/101651/show-sql-statements-from-mysql-binlog-gtid

Some database systems create and cache query plans based on the query matching and it's not certain they're sophisticated enough or willing to take the performance hit to do a semantic match skipping comments that are dynamic. Just something to keep in mind.

right, if someone did something like this it would have to be opt-in. even our cassandra propagation is opt-in. https://github.com/openzipkin/brave-cassandra/blob/master/cassandra-driver/src/main/java/brave/cassandra/driver/CassandraClientTracing.java#L140
I'm setting this help-wanted as personally won't be doing this feature

Some database systems create and cache query plans based on the query matching and it's not certain they're sophisticated enough or willing to take the performance hit to do a semantic match skipping comments that are dynamic. Just something to keep in mind.

Just trace update, insert, delete


[BUG] Additional virtual machine and images created during zero downtime upgrade should be deleted after completion

Describe the bug

Additional virtual machine and images created and remains in harvester-system namespace after zero downtime upgrade

Additional virtual machine under harvester-system namespace

Additional images created unser harvester-system namespace

To Reproduce
Steps to reproduce the behavior:
Do zero downtime upgrade from v1.0 to latest master head, steps refer to #1022

Prepare 3 nodes harvester with 1.0.0 GA
Prepare latest master head iso
Upload iso file to http or file server
ssh to harvester controller node
Create the versions.yamlcat > versions.yaml <<EOF
apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: 1.0.2
  namespace: harvester-system
spec:
  isoChecksum: 2765d89e5e8024347c81d36cc84b7168bf358a3a2f4da0c63164ad834a94bcbbaa13af4900aad4caa0a27550e4b4db5bf2c4726daa88664378e0c581a79ad2cb
  isoURL: http://<IP_ADDRESS>/harvester/harvester.iso
  minUpgradableVersion: 0.3.0
  releaseDate: "20211231"
  tags:
  - dev
  - test
EOF

Run kubectl create -f versions.yaml
Refresh harvester page
Click the Upgrade button
Click the green circle to display upgrade status page
Check the status and wait until zero downtime upgrade finish

Expected behavior

Additional virtual machine and image created during zero downtime upgrade should be deleted from harvester virtual machine and images page.
Support bundle

supportbundle_9f470b6f-44de-4cf9-891e-9bce00060bd7_2022-01-21T10-27-47Z.zip
Environment:

Harvester ISO version: v1.0.0 GA upgrade to v1.0 master head
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): 3 nodes harvester on provo bare machine

Verified fixed on v1.0.1-rc2. Close this issue.
Result
After one click upgrade from v1.0.0 to v1.0.1-rc2

The additional virtual machine and image created for upgrade have both been removed

Test Information

Test Environment: 3 nodes harvester on provo bare machine
Harvester version: v1.0.0 to v1.0.1-rc2

Verify Steps
One click upgrade guide
https://github.com/harvester/docs/blob/42598035cfe79856f7e159d4c47c4d7d29178900/docs/upgrade/automatic.md

Prepare 3 nodes harvester with 1.0.0 GA
Prepare latest master head iso
Upload iso file to http or file server
ssh to harvester controller node
Create the versions.yaml    rancher@node1:~> sudo -i

node1:~ # cat > version.yaml <<EOF
apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: v1.0.1-rc2
    namespace: harvester-system
  spec:
    isoChecksum: 2e8233e99589f735253255ccbe5ed039a37000f6766233fdb2dba5255180d27c8f8116548c8ba5b1b46a4dd605a3ffbe0284e59a912a50c5a6586957fa4e71d3
    isoURL: https://releases.rancher.com/harvester/v1.0.1-rc2/harvester-v1.0.1-rc2-amd64.iso
    releaseDate: "20220331"
  EOF

  node1:~ # kubectl create -f version.yaml
  version.harvesterhci.io/1.0.1-rc2 created

Run kubectl create -f versions.yaml
Refresh harvester page
Click the Upgrade button
Click the green circle to display upgrade status page
Check the status and wait until one click upgrade finish

Check the virtual machine page
Check the images page


Release v0.2.0

This will include breaking changes to the wallet creation / import APIs and the shielding APIs. It will include Unified Address support. It will not include Orchard support.

changing to 0.2.0 because I'm adding breaking changes to support type-safe Memos
https://github.com/zcash-hackworks/zcash-light-client-ffi/pull/27


feat: add tests

This adds in unit tests for all services - and should serve as a good base for people to write tests on.

codecov seems to be broken at the moment, so here is the coverage reported inside of rider:


Content picker search results are always empty when setting content type on field settings

When restricting content picker search settings to a specific content type, no results are ever returned.

Not sure but, as I remember, you may need to check the above include in the index option

@jtkech I tried your suggestion, but it doesn't make any difference. My understanding of the "include in index" field is that it determines if the data for that field is indexed as part of the content item. I don't think it has anything to do with the content picker search functionality.

There should also be an Admin Index - I think that's the one that is used in the content picker search.

@dcinzona I don't have an admin index. You should be able to pick which index you want to use using the drop-down list. The search works against my Site index as long as I don't specify a content type.
I've stepped through the code in the debugger and there's a point in the Orchard.Search.Controllers.ContentPickerController where it adds a search filter if a content type has been specified in the field settings.
foreach (string type in contentTypes.Select(x => x.Name)) {
    builder.WithField("type", type).AsFilter();
}

If this line gets called, no results are returned from Lucene. I don't know why though. It looks correct to me, but I don't have much experience with Lucene.

Well something is definitely off here.  I added a new index and had it set to index a certain content type.  I then added a new content picker field to the another content type and specified said index and specific content type.  I then search and the results are listed as blog posts?  Not the content type I selects (Employee and Department).

@dcinzona I tried changing the content type I was specifying and am now getting back results of a completely different type as well. I'm not sure I understand the pattern here but it's definitely not working as intended.

I thought I've already done this, can't test it right now, other work this night, I will test it tomorrow.

just tried it, it works but with a quite recent dev branch. I needed to specify the Content Type Id (not the Display Name). Which version, maybe you need this committed PR #5640 that solves not only a route issue but also a search index issue and a type filtering issue...
Best

When you say you need to specify the content type ID, do you mean the int value id or the type name? In my example Employee is the type name as well as the display name.
On Nov 28, 2015, at 7:39 PM, jtkech <notifications<EMAIL_ADDRESS>wrote:
just tried it, it works but with a quite recent dev branch. I needed to specify the Content Type Id (not the Display Name). Which version, maybe you need this committed PR #5640https://github.com/OrchardCMS/Orchard/pull/5640 that solves not only a route issue but also a search index issue and a type filtering issue...
Best

Reply to this email directly or view it on GitHubhttps://github.com/OrchardCMS/Orchard/issues/6094#issuecomment-160350984.

It is labelled content type ID on the dashboard, but you're right it's the technical content type name (not the display name if different)
Best

On 1.9.x, I was able to get things working by adding the call to NotAnalyzed as explained in PR #5640.
foreach (string type in contentTypes.Select(x => x.Name)) {
    builder.WithField("type", type).NotAnalyzed().AsFilter();
}

Does it make sense to submit a patch for this on the 1.9.x branch?


cppflow model prediction by gpu

My model (h5 format) can be predicted using GPU, but when I convert it to pb format, I can't call GPU to predict using CppFlow. What should I do?
I set up the following configuration in the model.py file
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

Cppflow (libtensorflow to be exact) automatically uses your GPU if it is available. Please check the error logs to see if you have incompatible driver/cudatookit.
You can try export CUDA_VISIBLE_DEVICES=0, but it is often not necessary.

thanks a lot，my problem has been solved


Half duplex test fails at stream->Write(request) when the message size is very large

I tried 20MB and the client fails to write. Attached is the full log:
client_log.txt
Tested in cloud-to-cloud scenario.
@fengli79

The interop server needs to increase the max recv message size (default to 4MB).

After setting both server side send and receive message size to INT_MAX, server throws error about handshake failure.
E0405 14:25:45.715066   94651 chttp2_server.c:83] Handshaking failed: {"created":"@1491427545.714862658","description":"Socket closed","fd":42,"file":"third_party/grpc/src/core/lib/iomgr/tcp_posix.c","file_line":253,"target_address":"ipv6:[::1]:60737"}

I tried adding the following to the client
channel_args.SetMaxReceiveMessageSize(INT32_MAX);
channel_args.SetMaxSendMessageSize(INT32_MAX);
But it still failed.

I could not reproduce.
I noticed that there are multiple CreateTestChannel used in the interop test. Did you update the channel_args on all of them?

It's working now. I didn't propagate the channel_args properly in the last try.


[Newton] RO-2451 Clone openstack-ansible in ansible-lint

In #2483, we drop the use of submodules but did not update tox.ini to
handle the clone of openstack-ansible.  The reason this passed the lint
check on #2483 is because it first checked out master, did a submodule
update, then checked out that commit and proceeded to run tests.  At
that point, the submodules had already been cloned and therefore
ansible-lint worked as expected.
This commit simply updates ansible-lint to clone openstack-ansible,
check out the correct commit, and then proceed as it did before.
Note that we also update how we handle submodules in .travis.yml.  This
isn't an issue in master for now (since we dropped submodules), but it
currently affects other branches since we checkout a differing commit
and then never update submodules.

recheck_cit_trusty_leapfrogupgrade_swift

This will most likely need to be backported to mitaka and liberty.

recheck_cit_trusty_leapfrogupgrade_swift

recheck_cit_trusty_leapfrogupgrade_swift

recheck_cit_trusty_leapfrogupgrade_swift

recheck_cit_trusty_leapfrogupgrade_swift

To document, this isn't needed on mitaka and below.  They have a much simpler .travis.yml and https://github.com/rcbops/rpc-openstack/pull/2534 passed a travis lint without issue.


Add the ability to change the bot's prefix

Pull request checklist:

[x] CHANGELOG.md was updated, if applicable

Yes I fucked up, accidentally used the fix-pleblist branch instead of the correct one

If it tells me to reformat again, I'm gonna kill someone

@RAnders00 what should I do here: https://github.com/alazymeme/pajbot/blob/master/pajbot/web/routes/api/commands.py#L161 in order to "import" the prefix (using the bot definition or whatever it's called fdm)

from flask import current_app as app

log_msg = f'The {app.bot_prefix}{command.command.split("|")[0]} command has been updated from "{old_message}" to "{new_message}"'

And in web/__init__.py, add a config parsing entry like you added in bot.py, this time call it app.bot_prefix = ... similar to how the other existing ones are already done:
https://github.com/pajbot/pajbot/blob/master/pajbot/web/__init__.py#L115

@RAnders00 Check the above commit, is this required in every file that I cannot "import" bot?

In module files the bot is available via self.bot.
The only pöaces you need to do it through app is in the web process which is in the .py files in the web directory.

@pajlada You haven't commented on this, so I just have to tag you and ask for your review


T-SQL Parser as part of Microsoft.SqlServer.Management.SqlParser

https://sqlblogcasts.com/blogs/sqlandthelike/archive/2012/03/13/parsing-t-sql-the-easy-way.aspx
It's a T-SQL Parser. Not sure if it'll be helpful for us internally but maybe?

I actually see a use case for it: parsing a tsql file to find if it contains any 'dangerous' statements (server level management, global roles management, etc). Might be useful in automated deployments.


CR for #24

Signed-off-by: Edward Z. Yang<EMAIL_ADDRESS>

Stack from ghstack (oldest at bottom):

(to be filled)


texinfoInteractive: skip test instead of expecting fail

This test fails, but only in some environments (glibc). Skip the test outright instead of running it and expecting it to fail.
This fixes pkgsMusl.texinfoInteractive.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Apparently SKIP_TESTS does not work, so I'll leave it with XFAIL_TESTS and make it conditional

Had to change checkFlagsArray to checkFlags, otherwise it being [] breaks the build


STORE command returning unexpected status using Exchange

I'm trying to get store of \Deleted flag working with Microsoft Exchange. I've adopted the code from demo1, I get back logging like this:

--- STORE ---
imap: unexpected completion status ("PQZGK7 NO Command received in Invalid state.")
Logout reason: Microsoft Exchange Server 2010 IMAP4 server signing off.
Close reason: end of stream
Connection closing (flush=false)

I've tried browsing the source code and I'm not getting very far with that.

The server is telling you that the STORE command cannot be executed in the current connection state. I can't tell you why that is. Maybe one of the preceding commands failed or maybe the mailbox isn't selected with read-write access. Enable raw logging, as in the demo, to see exactly what is being sent and received.

Maxim,
Thanks for the reply. You were exactly right to suggest I didn’t have the mailbox open with read-write access.
I do have a different problem and if you are willing to take a quick look, I would appreciate it. I have gotten intermittent connection errors, coming on the imap.DialTLS call. I've been trying to get our MS Exchange support team to look up the logs to see what is going on. Here is the transaction/message (I didn't have the raw logging turned on in this case and cannot regenerate the error now):
Connected to <IP_ADDRESS>:993 (Tag=FDMIL)
S:  (WSARecv tcp <IP_ADDRESS>:5386: An existing connection was forcibly closed by the remote host.)
Close reason: protocol error
Connection closing (flush=false)
Greeting error: WSARecv tcp <IP_ADDRESS>:5386: An existing connection was forcibly closed by the remote host.
panic: WSARecv tcp <IP_ADDRESS>:5386: An existing connection was forcibly closed by the remote host.
Thanks.
Kevin Buchs   Cloud Planning Services   Phone:<PHONE_NUMBER>
Mayo Clinic   200 1st. St SW   Rochester, MN 55905 http://mayoclinic.org
From: Maxim Khitrov<EMAIL_ADDRESS>Sent: Thursday, July 16, 2015 8:10 PM
To: mxk/go-imap
Cc: Buchs, Kevin J., M.S.
Subject: Re: [go-imap] STORE command returning unexpected status using Exchange (#11)
The server is telling you that the STORE command cannot be executed in the current connection state. I can't tell you why that is. Maybe one of the preceding commands failed or maybe the mailbox isn't selected with read-write access. Enable raw logging, as in the demo, to see exactly what is being sent and received.
—
Reply to this email directly or view it on GitHubhttps://github.com/mxk/go-imap/issues/11#issuecomment-122141753.

Hard to say without raw logging. If it's intermittent, perhaps your server has connection rate limit enabled: https://technet.microsoft.com/en-us/library/bb123712(v=exchg.150).aspx

exec cmd, err = c.Select("INBOX", true) error:
imap: unexpected completion status ("VSUTE7 NO EXAMINE The login is not safe! Please update your mail client: http://mail.163.com/dashi")


SDK-JS not consumed in the browser

Hi Team,
We have tried to bundle the consent string sdk JS using webpack but it doesnt work.
could yu please provide any guide or documentation on how to do that.
we want to use it for our web application . that will run on client side(browser).
Kind regards

You don't just npm install and then require("consent-string")? That's what I'm doing, there shouldn't be any special "bundling" instructions.

Thanks for your response,
That's exactly what I have done, but it still show "undefined" in console as error.
do you have a live example for live JS-SDK how it should be called and consumed in the browser ?

Here is exactly my issue : https://stackoverflow.com/questions/50316762/issue-bundeling-a-library-using-webpack-to-be-consumed-in-browser
could you please take a look

On your stack overflow question someone answered it.  If you webpack bundle something it is in an enclosed scope.  This is not an issue with the consent string sdk so I'm going to close the issue.


As a developer, I want to upload the resume to application and receive the result in JSON so that it can be used to display the information.

Create upload page @tanmaysharma2001


xpdf: 4.04 -> 4.05

Fixes a bunch of CVEs (but not all of them apparently).
Changes:
https://forum.xpdfreader.com/viewtopic.php?t=43343
4.05 (2024-feb-08)
------------------
Added the '-overwrite' option to pdftohtml.
Added the 'ignoreWrongSizeToUnicode' xpdfrc setting.
Added the loadSession and saveSession commands, and the 'Load last
  session' menu item.
Added code to automatically save and restore the xpdf session under
  control of a session manager.  This has not been thoroughly tested
  yet.
Added the zoomScaleFactor xpdfrc setting.
Added the zoomValues xpdfrc setting.
Added a 'smart case' option for search in xpdf.
Added the '-custom' flag to pdfinfo.
Added a color/gray/mono switch to the 'save image' dialog.
Added the separateRotatedText xpdfrc setting.
Added the '-meta' flag to pdftohtml.
Added the allowLinksToChangeZoom xpdfrc setting.
Added the 'uses JavaScript' output to pdfinfo.
Implemented pattern stroking of text.  Also fixed the various
  combinations of filling/stroking with color/pattern + clipping, some
  of which weren't being handled correctly.
Pdftops now (re)compresses any uncompressed or RLE-compressed images.
On an out-of-memory error, the command line tools now exit with an
  "out of memory" message, rather than an exception message.
Add code to pdfimages to extract images from tiling patterns.
Pdftops can now embed external 8-bit OpenType CFF fonts.
Fixed a corner case in the text extractor related to characters drawn
  at extremely large coordinates.  [Thanks to elvadisas for the bug
  report.]
Fixed an integer overflow in the transparency group code.  [Thanks to
  elvadisas for the bug report.]
Modify Annots::Annots() to skip annotations that have been turned into
  AcroFormFields -- invalid Widget-type annots will now be rendered as
  annots.
Added a missing integer overflow check in the JBIG2 decoder.  [Thanks
  to sangjun for the bug report.]
Added some sanity checks to the JBIG2 decoder.  [Thanks to sangjun and
  ycdxsb for the bug reports.]
Tiling patterns that use non-Normal blend modes can't be cached.
Fixed a bitmap size sanity check in the JBIG2 decoder.  [Thanks to Han
  Zheng (NCNIPC of China, Hexhive), for the bug report.]
Fixed a missing bounds check in FoFiType1C::convertToOpenType (used in
  pdftohtml).  [Thanks to cyth for the bug report.]
Fixed a use-after-free bug in pdftohtml.  [Thanks to FeRDNYC for the
  bug report.]
Merged aconf2.h into aconf.h; corrected the cmake config settings for
  paths; added the BASE14_FONT_DIR config option.  [Thanks to FeRDNYC
  for the suggestions.]
Fixed a missing check for a zero-length index in the CFF (Type1C) font
  parser.  [Thanks to Yuhang Huang (NCNIPC of China), Han Zheng

  (NCNIPC of China, Hexhive), Wanying Cao, Jiayu Zhao (NCNIPC of
  China) for the bug report.]
Add an object loop check to Catalog::countPageTree().
The DCT decoder wasn't checking for an SOF before the first SOS.
  [Thanks to cyth for the bug report.]
The inline image decoder was skipping to end-of-stream in the wrong
  stream object.  [Thanks to cyth for the bug report.]
Fixed a bug in the JPEG 2000 decoder when nLayers > 1 and the
  'termination on each coding pass' flag is set.
Removed the #pragma interface/implementation stuff (which is outdated
  and useless at this point).
Fixed a bug in the ICCBased color space parser that was allowing the
  number of components to be zero.  (CVE-2023-2662)  [Thanks to
  huckleberry for the bug report.]
Added checks for PDF object loops in AcroForm::scanField()
  (CVE-2018-7453, CVE-2018-16369, CVE-2022-36561, CVE-2022-41844),
  Catalog::readPageLabelTree2() (CVE-2023-2663), and
  Catalog::readEmbeddedFileTree() (CVE-2023-2664).
The zero-width character problem can also happen if the page size is
  very large -- that needs to be limited too, the same way as
  character position coordinates.  (CVE-2023-3044) [Thanks to jlinliu
  for the bug report.]
Add some missing bounds check code in DCTStream.  [Thanks to Jiahao
  Liu for the bug report.]
Fix a deadlock when an object stream's length field is contained in
  another object stream.  (CVE-2023-3436) [Thanks to Jiahao Liu for
  the bug report.]
Correctly handle tiling patterns with negative step values.
Ignore overprint in soft masks (to match Adobe's behavior).

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Build failed on darwin.

OfBorg does not build insecure packages and I do not have a Darwin system at hand. Do you have a trace of the error you could provide?

I know this is marked insecure, but do we want to support old versions? Currently when a new version comes out the old version's url gets changed (an "/old " is added) so it no longer builds. A fix to this e.g. would be to change the source url to:
    urls = [
      "https://dl.xpdfreader.com/xpdf-${version}.tar.gz"
      "https://dl.xpdfreader.com/old/xpdf-${version}.tar.gz"
    ];

p.s. Closes https://github.com/NixOS/nixpkgs/issues/288848

I know this is marked insecure, but do we want to support old versions? Currently when a new version comes out the old version's url gets changed (an "/old " is added) so it no longer builds. A fix to this e.g. would be to change the source url to:
    urls = [
      "https://dl.xpdfreader.com/xpdf-${version}.tar.gz"
      "https://dl.xpdfreader.com/old/xpdf-${version}.tar.gz"
    ];

Done, thanks.

OfBorg does not build insecure packages and I do not have a Darwin system at hand. Do you have a trace of the error you could provide?

xpdf> [ 98%] Building CXX object xpdf-qt/CMakeFiles/xpdf.dir/moc_XpdfViewer.cpp.o
xpdf> In file included from /tmp/nix-build-xpdf-4.05.drv-0/source/xpdf-qt/XpdfWidgetPrint.cc:22:
xpdf> In file included from /nix/store/piyj3b701x60a8w1d8dfqrl1f2nc94nw-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:60:
xpdf> In file included from /nix/store/piyj3b701x60a8w1d8dfqrl1f2nc94nw-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFPropertyList.h:17:
xpdf> In file included from /nix/store/piyj3b701x60a8w1d8dfqrl1f2nc94nw-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:16:
xpdf> /nix/store/piyj3b701x60a8w1d8dfqrl1f2nc94nw-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1189:83: error: expected ','
xpdf>     kCFURLBookmarkCreationWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0))  API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 11 ), // Mac OS X 10.7.3 and later, include information in the bookmark data which allows the same sandboxed process to access the resource after being relaunched

Thanks, I adjusted the change, it should be good now.


feat: Implement RAG

This is a work-in-progress pull request that adds RAG implementation to the Express server side.
It's composed these pieces:

A one-time script scripts/loadDocIntoVectorDb.js that loads user document into a FAISS vector store in a server directory, for example, ./vectorStore;
Config the directory of the vector store in the config/config.ts;
RAG APIs are written in src/server/vectorStoreHandler/index.ts;
An endpoint /query-vector is written for testing purpose. This endpoint will be removed when RAG is included in the AAC users' message processing.

The remaining work includes:

Write tests for the vector store handler;
Add documentation.

This pull request is based on another pull request https://github.com/inclusive-design/adaptive-palette/pull/17 that should be reviewed and merged first.

Thanks for the review, @klown. All suggestions are addressed. Ready for another look.

Thanks for the review, @klown. All addressed and ready for another look.


Sonic the Anonymous Hedgehog: Watch bravolaw.vn

Sonic the Anonymous Hedgehog requests the watch of the watch_keyword bravolaw\.vn. See the MS search here and the Stack Exchange search here.

bravolaw\.vn has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Seen in this TP: https://metasmoke.erwaysoftware.com/post/235628

Sorry. This watch was applied through a separate !!/watch.


schema annotation should add cachedImplicit

The @schema annotation should also add an implicit definition with shapeless.cachedImplicit (or equivalent) to the companion object (thus optimizing compile time for the uses of the Reified instance).
Problems:

[x] If the companion object already contains implicit definition(s), they could conflict with the added one. [Solution: this should be detected, and in these cases, the automatic one shouldn't be added.]
[x] We can't detect already existing conflicting implicits (because it seems that we cannot typecheck in an annotation macro). [Solution: add the implicit unconditionally and document it.]
[ ] The implicit defined in the companion object will conflict with the ones in the Reified  companion object.

TODO:

[ ] take a look at Exported[_] in circe
[ ] take a look at LowPriority and Orphan in shapeless


Implementing BFS

Hi guys,
I'm trying to implement the BFS-Level algorithm defined in this presentation  but I'm facing several problems regarding any pairs semiring and nothing values. I'm quite new to the language as well but this is the implementation i tried
#A is the input matrix, s is the source node, and n is the number of nodes
function bfs_level(A, s, n)

    #frontier init
    frontier = GBVector{Bool}(n)
        for i = 1:n
            frontier[i] = false
        end

    #init result vector
    distance = GBVector{Int64}(n)
        for i = 1:n
            distance[i] = 0
        end

    #putting source node inside the frontier
    frontier[s]= true

    print("starting from source node\n\n")
    for level = 1:n
            print(frontier)
            distance[frontier] .= level
            frontier = mul(frontier, A, Semirings.ANY_PAIR, mask=distance, desc=Descriptors.C)
            print(distance)
    end
    end

i tried the algorithm against this input
#0, 1, 1, 1, 0, 0, 0,
#0, 0, 0, 0, 0, 0, 0,
#0, 0, 0, 0, 0, 0, 0,
#0, 0, 0, 0, 0, 1, 0,
#0, 0, 0, 0, 0, 0, 0,
#0, 0, 0, 0, 0, 0, 1,
#0, 0, 0, 0, 1, 0, 0,

matrix =  GBMatrix([[0, 0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [0, 0, 0, 0, 0, 0, 1] [0, 0, 0, 1, 0, 0, 0] [0, 0, 0, 0, 0, 1, 0]])
bfs_level(matrix, 1, 7)

the Any_Pairs semirings should return 0, 1, 1, 1, 0, 0, 0 according to the matrix above but i'm getting [nothing, 1, 1, 1, 1, 1, 1]
I think I didn't understand quite well how to implement masking even if I succeded in implementing the Bellman Ford algorithm. Sorry again to bother you for this kind of issues

No bother at all! Thanks for trying it out, and sorry it's not a smooth experience yet.
I have some ideas, and there's definitely some fixes to make on my end. Give me a day or two and I'll post a working solution that works on the #master branch.
*Note in particular that indexing with GBVectors isn't working correctly right now. But I'll get it fixed!

Thanks for your answer and don't worry! I see it's still in an early phase but definetely keep going because it's interesting to use this library! I'll wait for the fix and I'll try to implement other algorithms in the meanwhile

I actually managed to solve the problem and to implement the algorithm:

Converted the input matrix to Bool values (true and false)
Used the LOR_LAND semirings instead of ANY_PAIR
Finally and most important I deleted the mask and the desc from the mul operation, it seems there is a problem computing the output of a mask in the following operations but should be because of Julia, nothing values can cause problems when used as input to other operations (even in print statements sometimes)

I probably didn't cover some operation properly, which means it falls back to Julia, and Julia doesn't like nothing values in arrays.
I spent today figuring out some memory issues, but I'll get the original working asap!

Thanks for the update!
PS=I looked at the code and the assign operation is not implemented, is there a work around for this at the moment? I need to compute an assign + mask operation

assign! should be there. Line 423 of matrix.jl. You should just be able to use setindex! though.
A[1:5,1:2:10, mask=B] = C should work. Note that uses subassign not assign. So you're mask will be the size of C not the size of A. If you need the mask to be the size of A you have to use assign!.

Thanks again!

Hey man thanks for the tip, i think I found another problem on MIN_FIRST semiring, this time to compute parents inside the BFS. unfortunately this time I didnt' managed to find a workaround like the last time

#A is the input matrix, s is the source node, and n is the number of nodes
function bfs_parents(A, s, n)

    index = GBVector{Int64}(n)
        for i = 1:n
            index[i] = i
        end
    print(index)

    #wavefront vector -> w[s]=source node insrted in wavefront
    f = GBVector{Int64}(n)
    f[s] = 1

    #parent vector -> p[s]=1 because the source is already visited
    p = GBVector{Int64}(n)
    p[s] = 0

    print("INIZIO\n\n")
    for i = 1:n-1
            f = mul(f, A, Semirings.MIN_FIRST, mask=p, desc=Descriptors.RSC)
            p = f[:, mask=f]
            f = index[:, mask=f]

    end

    print(p)

end

And my input (which is the same represented in the photo below) (source)
matrix =  GBMatrix([[0, 0, 0, 1, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 1, 1] [1, 0, 0, 0, 0, 0, 1] [0, 1, 0, 0, 0, 0, 1] [0, 0, 1, 0, 1, 0, 0] [0, 1, 0, 0, 0, 0, 0]])
bfs_parents(matrix, 1, 7)

The min first operation in the semiring should return [nothing, 1, 0, 1, 0, 0, 0] but instead it returns [nothing, 1, 1, 1, 1, 1, 1]. Thanks again for your patience

Why would you want it to be [nothing, 1, 0, 1, 0,0,0]? You want it to be [nothing, 1, nothing, 1, nothing, nothing, nothing], otherwise you've got essentially a dense matrix. Note GraphBLAS sparse matrices treat structural zeros as nothing not 0 since you could have a sparse matrix with materialized 0 values.
matrix =  GBMatrix([[0, 0, 0, 1, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 1, 1] [1, 0, 0, 0, 0, 0, 1] [0, 1, 0, 0, 0, 0, 1] [0, 0, 1, 0, 1, 0, 0] [0, 1, 0, 0, 0, 0, 0]]) this is probably your biggest problem.
This is a full matrix of mostly zeros, not a sparse one. You can either do:
using SparseArrays
matrix =  GBMatrix(sparse([[0, 0, 0, 1, 0, 0, 0] [1, 0, 0, 0, 0, 0, 0] [0, 0, 0, 1, 0, 1, 1] [1, 0, 0, 0, 0, 0, 1] [0, 1, 0, 0, 0, 0, 1] [0, 0, 1, 0, 1, 0, 0] [0, 1, 0, 0, 0, 0, 0]]))

Or more idiomatically:
matrix = GBMatrix([4,1,4,6,7,1,7,2,7,3,5,2], [1,2,3,3,3,4,4,5,5,6,6,7], ones(Int64, 7))

which is the cooordinate form.
GraphBLAS treats zeros as actual numbers, which is not what SparseArrays does. I'm making a package that will behave like SparseArrays, but for graph algorithms you don't want this anyway.
I'm going to write a BFS example this weekend, I'm just working on getting the next version out at the same time. So bear with me! :)

Thank's man this tip solved my issue and i think this was the issue I had with the original post as well! I'm gonna try it tomorrow on the original algorithm i posted to check! I thought that 0 was equivalent to nothing and this was my problem! One more thing, citing your answer a couple of days ago regarding the assign command
A[1:5,1:2:10, mask=B] = C
in the single parents bfs implementation i used this command and it worked like a charm
f = index[:, mask=f, desc=Descriptors.S]
basically it was for mapping index values (1, 2, 3, 4, 5 ecc...)  to non-nothing f values to f (both index and f are vectors)
Then i tried to implement the multi bfs parents algorithm, to compute more parents istances of different nodes at the same time. When i try to do the same assignment but with 2 matrices i get an error, both F and index are 3*7 matrices in this case with the same purposes of the line before (mapping from index to f non-nothing values)
F = index[:, :, mask=F, desc=Descriptors.S]
example
[1, 2, 3, 4, 5, 6, 7; 1, 2, 3, 4, 5, 6, 7; 1, 2, 3, 4, 5, 6, 7] and [1, nothing 1, nothing, nothing ... ; nothing, 1, nothign ... ; 1, nothing, nothing... ]
result [1, nothing, 3, nothing, nothing ... ; nothing, 2, nothing, 1 ... ; 1, nothing, nothing ...]

 I'll add it for the next release!

Thanks again man! I'll try without transposing and see how it goes. I completed about 13/14 algorithms using the library till now and it was smooth, now I'll try to optimize them using the sparse keywords on the input matrices

Hey man, I'm back from holiday and as promised I made the pull request. Feel free to review it on your free time


vanishing npcs

Hi,
i have some trouble with your plugin. I just created some npcs, after i logged out and came back several hours later, they´re disappeared. it also happens, when i just log out for some seconds or go too far away (maybe when the chunk is not loaded?)
I´m on Version 1.11, SpongeForge (sponge-build 2004, forge-build 2195)
List of all plugins used:
CommandBooks
DynMap
EconomyLite
holograms
Keys
LuckPerms
MMCReboot
Nucleus
PlayerShopsRPG
ProjectPortals
RedProtect
sleepvote
walletdrop
WorldEdit

This seems to happen more often. This should not be a problem anymore in the version that is comming soon, as the NPC's will no longer be stored in the world's NBT data.

This should no longer be an issue in the new version (v2.0.0)/


PC-NONE: fix cookies text

Approved, but can you raise tech debt to think about what cookie durations we actually want? Thanks.


multiple consumer in single process

Can I have two  HighLevelConsumer in the same process?
I tried to do it this way:
Two consumers are bind to two different clients. They also subscribe to different topics and labeled in different groups.
The strange thing is, one can fire a 'message' event when new message arrive, but the other never. I seems that the other never gets messages from socket ( I don't see the callback getting invoke )

You shouldn't have any problems. Make sure you are using version 0.2.30. There are issues with 0.3 that haven't been merged yet.

Switched to 0.2.30, still no luck

Please post code that demonstrates this issue.

I have the same problem with Consumers.
Two consumers A,B, different clients. Each subscribed to different topic. First created consumer A start to consume messages, B does not. If I change creation order from A, B to B, A then  consumer B start to consume messages, A does not.

same issue for 0.3.1
However 0.2.3 work for me.

Guys, is this issue ever fixed? I'm using 0.4.0, issue still remains.
I also tried 0.2.30, FailedToRebalanceConsumerError happens with code NODE_EXISTS[-110] sometimes, and when there's no this Error, only 2nd consumer get the message. @hyperlink

@yukuan1988  please take a look at this PR. Does this solve the issue for you?
If not we will need to get some test code to replicate the issue.

Sorry, @hyperlink , I'm getting confused. Which comment should I take? It seems that all the solutions are to modify the library, I wonder whether it's a good approach. Some more details are really appreciated!

@yukuan1988 could you post your code here?  We need to see a test code to replicate the issue you're describing.

Hi @hyperlink my bad, i should have attached my code earlier. And here it goes
kafka.zip
and the console log:
**** trying to connect to kafka located at osboxes:2181
**** connected to osboxes:2181
++++ producer is ready
++++ sending message to topic test_topic_1
++++ sending message to topic test_topic_2
---- topic subscribed: test_topic_1
---- topic subscribed: test_topic_2
++++ null { test_topic_1: { '0': 2 } }
++++ null { test_topic_2: { '0': 2 } }
----1 Thu Jun 30 2016 13:36:25 GMT+0800 (Malay Peninsula Standard Time) test-message 1
---- close consumer before shutdown
---- close consumer before shutdown

@yukuan1988 just glanced at the code. I recommend you create new client for every consumer created. Don't share the client object between consumers. Also if you want each consumer listening to the same topic to receive the message you should give a unique groupId otherwise kafka believes it to be the same group and only the leader in the group will receive that message. Good luck.

@hyperlink I've also tried different groupId for different topic, no luck. I think if for one client and one topic, only one consumer would receive the message, at least it should be documented.


Puppeteer old Headless deprecation warning

I'm receiving the following message when running Pageres:
    In the near future `headless: true` will default to the new Headless mode
    for Chrome instead of the old Headless implementation. For more
    information, please see https://developer.chrome.com/articles/new-headless/.
    Consider opting in early by passing `headless: "new"` to `puppeteer.launch()`
    If you encounter any bugs, please report them to https://github.com/puppeteer/puppeteer/issues/new/choose.

(node:22972) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.```

A workaround for now can be passing launchOptions headless: 'new'....
await new pageres({
  launchOptions: {
      headless: 'new'
  }
})


Button style - Save to Calendar

Hi Peter,
Doing a great job.
I've just noticed on both the Chrome version and the Android app that the style of the "save to calendar" button in programming is not consistent with the other buttons for Summit.

Thanks again for all your hard work.
Adam
2nd Eltham Sea Scouts

Thanks, are you using the Android version too now? Was the install process ok?
Which version was this installed from? In 2.3.0 in Chrome seems to be ok and 2.2.0 is ok on Android. You can tell the version from the logo, the version number came in with 2.2.0 so if there is no version is it likely 2.1.0 or older.

Chrome = 2.3.0
Android = 2.2.0
Both do not have the correct style, as I see above in your pics.
Yes using the Android version now, too - the installation process was easy

Here is a screenshot of the Android output.

Note covered Activity ID

I have just created a new plan, and it has the correct styling when the plan has not been concluded on both Android and Chrome.
So the incorrect button style is when the plan has been concluded.

Oh OK, I hadn't looked at a concluded plan, I can confirm the issue now. Thanks for the info.

Fixed in V2.4.0, release pending.


Remove names from solverapi results

address this issue: https://relationalai.atlassian.net/browse/RAI-16587
see also https://github.com/RelationalAI/raicode/pull/16451

@zengjian-hu-rai I do think we need to keep the names around, unless we want to rely on clients having the input model present as well. Otherwise, we don't know which value corresponds to which variable.
We can create a top-level field, either called names, but maybe better called variables to match the input. Each result in results does not need to enumerate the variables again as they are the same.

@bachdavi the user has to pass in the model to some of the other extraction-related relations, so maybe it would be ok to rely on the input model. Hm I'll play with this today to see how it feels.

Yeah variables already exist within the model relation, so i thought it was unnecessary to pass it back in the result?@bachdavi

@zengjian-hu-rai Riiight! Okay fair enough :)

can I please leave to you @bachdavi to merge this one along with the corresponding PRs in rai-solver-service and rai-code, once you guys are comfortable?
https://github.com/RelationalAI/rai-solver-service/pull/44
https://github.com/RelationalAI/raicode/pull/16451


Master build is failing [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:<IP_ADDRESS>8:sonar (default-cli) on project camel-lsp-server: Unable to read /home/travis/build/camel-tooling/camel-language-server/target/jacoco.exec to determine JaCoCo binary format.: EOFException

[WARNING] Property 'sonar.jacoco.reportPath' is deprecated (JaCoCo binary format). 'sonar.coverage.jacoco.xmlReportPaths' should be used instead (JaCoCo XML format). Please check that the JaCoCo plugin is installed on your SonarQube Instance.
[WARNING] Property 'sonar.jacoco.reportPaths' is deprecated (JaCoCo binary format). 'sonar.coverage.jacoco.xmlReportPaths' should be used instead (JaCoCo XML format). Please check that the JaCoCo plugin is installed on your SonarQube Instance.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  28.759 s
[INFO] Finished at: 2019-09-30T13:28:10Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:<IP_ADDRESS>8:sonar (default-cli) on project camel-lsp-server: Unable to read /home/travis/build/camel-tooling/camel-language-server/target/jacoco.exec to determine JaCoCo binary format.: EOFException -> [Help 1]

might be due to environment change as the code change seems completely unrelated

seems a bug with Sonar/jacoco https://support.cloudbees.com/hc/en-us/articles/205074234-SonarQube-reporting-is-failing-with-a-jacoco-error-Incompatible-version-1007

doing in 2 mvn builds, the test and teh sonar analysis is no more working... don't know why. the simple workaround is to launch both in same command.
Will need to report the same modifications on all Maven Java projects (maybe others too?)

so doing in same builds was working locally but not on CI... I'm suspecting that it was caused by previous build states...
trying to follow https://community.sonarsource.com/t/coverage-test-data-importing-jacoco-coverage-report-in-xml-format/12151 but seems the case of unit and integration test is not covered (in doc)

build no more failing at the cost of not having code coverage anymore. Created https://github.com/camel-tooling/camel-language-server/issues/313 to restore it

verified with master branch


[BUG] PRIVACY.md includes license terms that result in Amethyst being non-free software

Describe the bug
PRIVACY.md includes the following:

You cannot use Amethyst to submit Objectionable Content to relays. Objectionable Content includes, but is not limited to: (i) sexually explicit materials; (ii) obscene, defamatory, libelous, slanderous, violent and/or unlawful content or profanity; (iii) content that infringes upon the rights of any third party, including copyright, trademark, privacy, publicity or other personal or proprietary right, or that is deceptive or fraudulent; (iv) content that promotes the use or sale of illegal or regulated substances, tobacco products, ammunition and/or firearms; and (v) illegal content related to gambling.

This means that Amethyst is non-free software, contrary to the MIT License specified in LICENSE.
Beyond that, the license text is also unclear:

"sexually explicit", "obscene", "violent", etc. by what standards/guidelines?
"unlawful", "illegal", "regulated", etc. under which jurisdictions?

Such ambiguities make compliance with the license terms difficult, even if I did consent to them (which I do not; what I choose to send to or receive from a relay is solely the business of myself and the relay).
The paragraph following it is also concerning:

We reserve the right to modify this Privacy Policy at any time. Any modifications to this Privacy Policy will be effective upon our posting the new terms and/or upon implementation of the new changes on the Service (or as otherwise indicated at the time of posting). In all cases, your continued use of the app after the posting of any modified Privacy Policy indicates your acceptance of the terms of the modified Privacy Policy.

The ability to introduce new potentially-non-free license terms to which all users of all versions of Amethyst are automatically bound without any apparent mechanism to notify users and obtain explicit consent means that even if Amethyst is currently free software, that could change at any time without warning.
To Reproduce
Steps to reproduce the behavior:

Open the app for the first time
Click on the "terms of use" link
Observe the non-free license terms quoted above

Expected behavior
Removal of the offending license terms (or explicit permission for derivative works to remove them), or clarification in LICENSE and elsewhere that Amethyst is not actually subject to the MIT License (and consequent removal from e.g. F-Droid's default repositories).
Additionally, pointing the "terms of service" link to the MIT License (or whatever other license) as it exists in the actually-installed version of the app would alleviate concerns around users of older app versions being unexpectedly bound by new license terms for which they had no opportunity to refuse consent.
Video and Screenshots
N/A
Device (please complete the following information):

Phone Brand/Model [e.g. Pixel 7 Pro]: Unihertz Titan
Android Version [e.g. 33]: 10
App Version [e.g. v0.20.3]: v0.32.3

Bounty (in Bitcoin sats) offered for a solution
N/A

For reference, there is an issue over at F-Droid to handle this situation: https://gitlab.com/fdroid/fdroiddata/-/issues/2973

@vitorpamplona that basically makes Amethyst non-free, which means we'd have to unlist it at F-Droid. While I can see why you don't wish Amethyst being used that way (and I'm with you there), forbidding/restricting its use in such a way is taking away freedom.
By the way: to regulate those things is the task of the law. One shouldn't use any means to do "illegal stuff" – but it's not in our powers to enforce that.

I am confused. Why are we mixing the license with the terms of use? These two files are separate legal matters. The Privacy is used by the Play Store to manage the distribution of the executables. The MIT license relates to the source code only.

Why are we mixing the license with the terms of use?

Whether or not software is FOSS is dependent on the full set of conditions under which users can use/modify/distribute the software.  That would include terms of use which users are required to accept before they can use the app.

The Privacy is used by the Play Store to manage the distribution of the executables.

Gotcha; I'm guessing you're referring to this requirement?

Privacy policy
Adding a privacy policy to your app's store listing helps provide transparency about how you treat sensitive user and device data.
The privacy policy must, together with any in-app disclosures, comprehensively disclose how your app collects, uses, and shares user data. This includes the types of parties with whom it’s shared. You should consult your legal representative to advise you of what is required.

For apps that request access to sensitive permissions or data (as defined in the user data policies): You must link to a privacy policy on your app's store listing page and within your app. Make sure your privacy policy is available on an active URL, applies to your app, and specifically covers user privacy.
For apps that target children: You must link to a privacy policy on your app's store listing page and within your app, regardless of your app's access to sensitive permissions or data. Make sure your privacy policy is available on an active URL, applies to your app, and specifically covers user privacy. Note that even apps that do not access any personal or sensitive user data must still submit a privacy policy.

Add a privacy policy

Open Play Console and go to the App content page (Policy > App content).
Under "Privacy Policy," select Start.

Note: If you’ve previously added a privacy policy and want to make changes, you’ll see and select Manage instead of start.

Enter the URL hosting the privacy policy online.
Save your changes.

It doesn't seem like the paragraphs at issue are required to be part of the privacy policy for the app to be compliant.  Was the Play Store rejecting the app without those paragraphs present?
In any case, is that something that can be controlled with a build flag, so that Play Store builds get the Play-Store-required privacy policy while F-Droid builds get the intended MIT license?  Alternately: can acceptance of the privacy policy be made optional, and/or can it be moved to an "About" page (or similar) in-app instead of being part of the out-of-box experience?

Yes, those were specifically added to comply with software distribution laws worldwide from the Play Store reviewers. I am surprised F-droid doesn't require the same thing. Rights to use the source are different from the rights to use the distributed binaries (which in some cases I am liable for, in some others F-Droid itself is liable).
In other words, the MIT license removes any author liability from the misuse of the code. But when the author is also providing the system as binaries (which is an additional service in every jurisdiction I know of), there are many other legal issues that the source code license won't cover.
And I don't know about you, but I am not comfortable allowing people to use the Play Store version or the FDroid version for these activities written in the Privacy statement. Most of them are local crimes that should not happen anyway.
This has nothing to do with the source code license, which people can still download, compile and use in nefarious ways.

This has nothing to do with the source code license, which people can still download, compile and use in nefarious ways.

So it does not apply to the APK produced and distributed by F-Droid. Maybe you could phrase it that way? I vaguely remember we had a comparable case not that long ago, and solved it by such a rephrasing.

I updated the PRIVACY.md to clarify that the terms of use is for the distributed applications and not the source code.
Let's hope it helps

@vitorpamplona so what did change, apart from "Amethyst" => "the Amethyst app for Android"? I guess changing "you cannot" to "you should not" would help more (and would be more true as well, as technically "cannot" is not correct even :wink:).

No, the first line now reads:

Effective as of Jan 20, 2023 for the distributed applications in the Play Store and the F-Droid Catalogue

Ah, so now it explicitly applies to F-Droid. Forgive me if I ask if that's not just the wrong direction taken? I thought exactly that was what we wanted to avoid? You wrote:

the terms of use is for the distributed applications and not the source code.

Now someone (here: F-Droid) takes your source code, which is under a FOSS license and thus got the 4 freedoms granted – and you say

for the distributed applications in the Play Store and the F-Droid Catalogue

thus taking one of the 4 freedoms away again. Explicitly. That way you force F-Droid to take your app down, as it violates the inclusion criteria – because you contradict the very license (MIT) you've applied to the source and thus (at least partly) render it moot. Quote from the MIT (emphasis mine):

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so

"Without restriction" doesn't match "You cannot use Amethyst to …" IMHO. If you apply that to the APK you distribute at Play, that's one thing – but you cannot apply it to what a 3rd party compiles of the MIT licensed code. That's absurd, if I may say so.
Apologies if this sounds harsh, no offense meant – I just try to clarify why that is, IMHO, a no-go.

You are still confusing distribution (the key word in the new sentence) with the source code license.
F-droid is not distributing source code. It's distributing a binary as a service. That has nothing to do with the openness of the source code.

So you say "to deal in the Software without restriction" does not apply to dealing with the software? If I can deal with the source code without restriction, how does that not include the binaries I produce from it? Sure that it's me confusing things here? I admit I'm not a lawyer – but "without restriction" sounds pretty clear to me. Dealing with the source *without restriction" also includes compiling it. And distribution is explicitly mentioned.
Besides, it's not only my personal understanding (as you can see by the other participants here). I won't argue that anymore – I just tried to find a solution both sides can live with. Unfortunately that doesn't seem to be possible – so I'm afraid F-Droid will have to remove Amethyst then. But that's not my personal decision either, it's for the F-Droid team to decide. Feels sad, though – after all the work you, your team and I invested to get your app in. It's your app, so the decision of the terms of course is yours. Still, I'm disappointed. I had a little hope there, but that's gone now.
Nevertheless, I wish you the best for you and your project. Maybe one day… well.

Yep you can compile it and distribute it. You cannot use the version we compile and distribute for whatever you want. Very different legal frameworks between the two.

Does f-droid require that the code is FOSS or that the distribution is FOSS? (I have never ever seen something requiring distribution to be FOSS). Those are not the same thing.

We pass all of these requirements: https://f-droid.org/en/docs/Inclusion_Policy/#:~:text=All applications in the repository,application from the published source.
I don't know where the restriction to publish on f-droid comes from.

All applications in the repository must be Free, Libre and Open Source Software (FLOSS)

Applications. And note it says "FLOSS", which includes the L for "libre". Don't confuse that with "code openly available". You've decided for "reproducible builds" – so after F-Droid builds the APK and compares the results with yours, it indeed ships yours when the two match – which gives extra trust for those using the app (F-Droid checked it, you signed it) and the possibility of cross-updates (should one channel fail, e.g. your app get blocked at Play, they can seamlessly update from another).
But yes, F-Droid requires that everything is F/LOSS. Even down to the tools used to create the APK. That also includes the software used to maintain the repo (server), and the website (client). Can all be found at GitLab under https://gitlab.com/fdroid
But I'm tired discussing that if the outcome is already clear, we're both wasting our time now Vitor. You won't change your mind, and F-Droid won't change its rules.

We match all of that, @IzzySoft It's not about changing my mind... I am happy to do so.

What else do you want me to add to the Privacy to make this right?

As pointed out above: it's Google who forces this upon you, so please apply that only there. And as I already suggested: turning the "you cannot" into "you should not" would make the problem vanish. But I cannot tell if that would satisfy Google. If not, I personally would blame it on them (trying to monopolize again; I'm pretty sure they know about this implication). In that case, if your intro says

Effective as of Jan 20, 2023 for the distributed applications in the Play Store

(and leave F-Droid out there), it can only be applied to Play, so the sting has been removed as well. I guess one of the two should do, both together I'm pretty sure should do (at least that would totally convince me – but again, IANAL :wink:)

Thanks for chiming in "over there", @vitorpamplona! I'll leave further discussion to the team then (especially as I won't have time to participate in it for the next week or so, being bound to other tasks). :crossed_fingers: for a good outcome!

Thanks Izzy.. I hope we can keep it there. I just need some clarity on who goes to jail when a user uploads illegal content through the version shipped by F-Droid.

I understand your position Vitor. But have you ever heard that a car maker was held responsible when one of their cars was involved in, say, a bank robbery? The contracts don't have such a clause there. To me it should be clear that, whenever you use anything to do something illegal, blame is on the evildoer, not on the tool. So "You cannot use Amethyst to…" to me even sounds like the wrong angle. The point is, you should not do these things anyway, at all. And it's not the software developers task to take care for that.
But again, I understand your point.
So the question "who goes to jail" is for a lawyer to answer I'm afraid. And would be a question for the other 4k apps at F-Droid as well; I didn't hear (but neither check) that any of those established such a TOS. :man_shrugging:

But have you ever heard that a car maker was held responsible when one of their cars was involved in, say, a bank robbery

Social media is different. There are regulations in place for marketplaces and apps that capture and share users content.
It's similar to banks. If you are making illegal transactions, your bank is indeed liable.

Yes yes, that's what I mean by "IANAL". Exactly that part is what my mind refuses to grasp, it's different rules depending on the "medium": why would the software developer be held responsible for the use of their tool – but not the factory producing kitchen knives?
But btw, the bank is liable for different reasons (I'm working for one, so I get that stuff "indoctrinated" twice a year): it is a service (not an app). So it's not the "app maker" (here: of the banking app) who's held responsible in such a case, but the service provider (bank) which must keep eyes open for "patterns" indicating illegal transactions¹. It won't be held responsible if it had no means to detect a certain fraud – only if it had but did not.
So in the case here: you as the developer of the app have no means to control its usage – those who run the relays might have. So if at all, it would be on the relay operators – not on the app developers. But again, IANAL…
¹ "fraud detection", e.g. a credit card normally only used for few big payments suddenly used for massive shopping sprees; an account that usually receives salaries of monthly 2k suddenly receives multiple 1k payments a month and immediately spends 90% of that to some accounts abroad, etc.

it's different rules depending on the "medium": why would the software developer be held responsible for the use of their tool – but not the factory producing kitchen knives?

It's not about the developer, but the shipper. The problem with us is that the shipper and the developer are the same. So the confusion.
When you make an app available in a market, you are "servicing" that market and must comply with local laws as a "service" provider. Putting code on GitHub is not servicing a market, but placing an app in a store or a catalog is the definition of servicing the market.   Thus all these other requirements.
For instance, F-Droid is technically required to evaluate every app's content ratings in many jurisdictions like the US and Europe, and in case of Europe, make sure they comply with GDPR. They don't do this right now, but if they have any legal staff, that would be one of the first things to solve.

It's not about the developer, but the shipper.

So the downtown store selling the knives gets sued? :thinking: Or the website operator where the banking app (not the banking service) is hosted? Never heard that. But then again, IANAL – and there might be a place where they have such crazy rules.

They don't do this right now

Sure? You were there when I reviewed your app :wink: You couldn't publish a single web browser if you were held responsible for the websites visited with it. And guess what: yes, we had "legal contact" in the past, where the other side claimed "illegal activities". That was either defended/repelled – or at worst the app was taken down when justified. Full stop. :man_shrugging:
Oh, and to fill the last gap: where the app intended especially for illegal use, the case would of course be different – and a weapon seller of course is obliged to take certain precautions as well, as a tobacco shop owner is :wink:

@vitorpamplona It sounds like:

You're under the impression that by distributing binaries, you're now liable for illegal content generated by the users of those binaries.
To avoid liability, you wish to shift liability to the users of those binaries by forcing them to agree to a Terms of Use prior to using the binaries, saying that they won't do illegal things, and they also won't do a bunch of other legal things.
This results in the binaries not being open source. When folks point out that this is not how liability in software works (browsers for example), you make the argument that social media apps have special rules and regulations.

Many of the things you're saying I am unable to verify:

Web browsers are not social media apps to regulators because domain names are registered with the government and thus separate the liability from the browser into the domain name owner.

Where would I look to see if this is true? I know the last part isn't true in the United States (Section 230), but I can't falsify the rest because I don't know who "regulators" are.

When you make an app available in a market, you are "servicing" that market and must comply with local laws as a "service" provider. Putting code on GitHub is not servicing a market, but placing an app in a store or a catalog is the definition of servicing the market. Thus all these other requirements.

What additional requirements?  Where are they written?

Social media is different. There are regulations in place for marketplaces and apps that capture and share users content.

What have you seen that makes you say this? What are the regulations?

Thanks Izzy.. I hope we can keep the app on f-droid. I just need some clarity on who goes to jail when a user uploads illegal content through the version shipped by F-Droid.

In the United States, users are responsible for user-generated content.

As a point of comparison, Infinity (another social media app on F-Droid - in this case for reddit) does not include such prohibitions in the privacy policy linked within the app's Settings page.  If Infinity doesn't need them, then I'd be surprised if Amethyst does.

I am primarily concerned with things with a line or two that invalidate Section 230 or similar laws worldwide. For example, in the US, a set of laws called FOSTA/SESTA basically says that anything having to do with "assisting, facilitating, or supporting sex trafficking" means that Section 230 immunity doesn't apply. So, I want to add that specific sentence (you cannot "assist, facilitate, or support sex trafficking" through the use of the app) to the app's terms of use. The current terms are basically a compilation of those things.
To be clear, I am not a lawyer. But I have had some guidance on this in the past.

As a point of comparison, Infinity (another social media app on F-Droid - in this case for reddit) does not include such prohibitions in the privacy policy linked within the app's Settings page. If Infinity doesn't need them, then I'd be surprised if Amethyst does.

Thank you for this reference. It feels like they are quite loose on their terms.
But I did notice that they say: "These Services do not address anyone under the age of 13". To me, this means that people under 13 cannot use the app, which from the first post, would break their FOSS status. But I hope we are beyond this point now.

FOSTA/SESTA are tricky, since the definition of an interactive computer service could be construed to mean a browser.
But I think you're a step removed there. The interesting section is this:

Whoever, using a facility or means of interstate or foreign commerce or in or affecting interstate or foreign commerce, owns, manages, or operates an interactive computer service (as such term is defined in defined in section 230(f) the Communications Act of 1934 (47 U.S.C. 230(f))), or conspires or attempts to do so, with the intent to promote or facilitate the prostitution of another person shall be fined under this title, imprisoned for not more than 10 years, or both.

This reads to me as saying you'll be liable if:

You own, operate, or manage an online service, and
You have intent to promote or facilitate prostitution of another person

The definition of interactive computer service is based on a pretty weird definition of "access software provider" that could be read to include Amethyst (and a browser!):

(4) Access software provider
The term "access software provider" means a provider of software (including client or server software), or enabling tools that do any one or more of the following:
(A) filter, screen, allow, or disallow content;
(B) pick, choose, analyze, or digest content; or
(C) transmit, receive, display, forward, cache, search, subset, organize, reorganize, or translate content.

So I can see why this might be a concern.  But the second clause ("intent to promote or facilitate prostitution...") seems like a very tough bar to meet in for something like Amethyst, since Amethyst could be used with any relay. How would one show any such intent with such a setup? Terms of Use along these lines feel like they would be something relays would want to think about, rather than authors of clients.
FWIW, it seems like FOSTA-SESTA has only been used once since it was signed in 2018. It seems most of the cases in the study linked from that article were prosecuted on standard trafficking charges.

Another reference: Hacki (a client for Hacker News, on F-Droid) also includes an EULA which does not (as far as I can tell) include non-free license terms.  It does include the worrying provision that said EULA can be changed at any time, but unless/until that mechanism is used to introduce non-free terms, I don't think that would be an issue.

Closing this discussion since we found a way to "dual license it" (play and fdroid) months ago.


Add requires_new to ActiveRecord to use savepoints

https://trello.com/c/CrtqBpmu/793-fix-bug-with-basepath-reservation-small

(This is an interesting quirk; perhaps it's worth explaining in the commit message?)

I don't understand exactly why even without this change, the tests that cover this use savepoints. Saying that, I think that means that there is room for improvement in those tests.


Add contact's case number

This issue has been fixed in https://github.com/AY2122S1-CS2103-W14-2/tp/pull/62.


fix: use bigint for amounts in postgres

The amount in sats can be higher than the the INT type in postgres (2^32)
Solution:

add the big_int property (simialr with timestamp_now)

Note:

in Postgres int8 is an alias for BIGINT
expected_amount: BIGINT
timeout_block_height: INT

Closes #956


[Question] Variations of VRP

Is there a overview of all the VRP's you want to implement into the optimization package?
Are you planning to provide a more flexible solution to let people build their own constraints?

There is no planned flexible solution, the purpose of this library is two-fold:

Implement a few very common cases.
Provide tools for developers to implement their own specifics.

This comes down to VRP's with vehicles that can have:

Max travel weight (time/distance/other).
Max capacity, think weight.
Fixed start location or not.
Fixed end location or not.

We also plan to support:

U-turn prevention on all variants.
Time windows on all variants.

This means this out-of-the box support is going to be there for:

TSP
TSP-TW
STSP
STSP-TW
VRP
VRP-TW

The ones in bold are currently implemented, most of the work is going on in a refactoring effort reflecting everything that was learned over the past years.


[Task] While sending is pending, the SendForm shows no expiration date

When I create a transaction with expiration date and confirm it, the SendForm while loading shows that the expiration date is none

Apparently the pull request that would close this issue was merged


Civ 386 upgrade library i18next

JIRA link
CIV-386, CIV-387 and CIV-388
Change description
Updated I18next library
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[X] No

Closing this pr as this is not relevant now


Creating Subdivisions inside Adminclaims with "worldedit we" not working.

Title says it all.

Steps to reproduce

Create Adminclaim
Select smaller area inside with worldedit
/gd mode subdivide
/gd claim worldedit we
Experience Error "I'm sorry, but there was an error performing this command."

Worldedit used: worldedit-forge-mc1.12.2-6.1.10-dist.jar
Luckperms used: LuckPerms-Sponge-5.2.20.jar
Griefdefender used: griefdefender-sponge-1.5.2.jar
Sponge used: spongeforge-1.12.2-2838-7.3.1-RC4079.jar
Forge used: forge-1.12.2-<IP_ADDRESS>38-universal.jar

Error in Console
Better formatting: https://hastebin.com/walucutoce.md
[00:33:03] [Server thread/INFO] [nucleus]: colorspace_ ran the command: /gd mode subdivide
[00:33:11] [Server thread/INFO] [nucleus]: colorspace_ ran the command: /gd claim worldedit we
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF] Exception in command: claimwe
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF] java.lang.reflect.InvocationTargetException
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at java.lang.reflect.Method.invoke(Method.java:498)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.RegisteredCommand.invoke(RegisteredCommand.java:152)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.BaseCommand.executeCommand(BaseCommand.java:576)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.BaseCommand.execute(BaseCommand.java:512)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.RootCommand.execute(RootCommand.java:99)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.SpongeRootCommand.executeSponge(SpongeRootCommand.java:96)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at co.aikar.commands.SpongeRootCommand.process(SpongeRootCommand.java:64)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at org.spongepowered.common.command.SpongeCommandDispatcher.process(SpongeCommandDispatcher.java:355)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at org.spongepowered.common.command.SpongeCommandManager.process(SpongeCommandManager.java:360)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.command.ServerCommandManager.func_71556_a(SourceFile:665)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.NetHandlerPlayServer.func_147361_d(NetHandlerPlayServer.java:962)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.NetHandlerPlayServer.func_147354_a(NetHandlerPlayServer.java:941)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.play.client.CPacketChatMessage.func_148833_a(SourceFile:37)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.play.client.CPacketChatMessage.func_148833_a(SourceFile:9)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at org.spongepowered.common.event.tracking.phase.packet.PacketPhaseUtil.onProcessPacket(PacketPhaseUtil.java:219)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.PacketThreadUtil$1.redirect$zlf000$impl$redirectToPhaseTracker(SourceFile:540)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.network.PacketThreadUtil$1.run(SourceFile:13)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.util.Util.func_181617_a(SourceFile:529)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at org.spongepowered.common.SpongeImplHooks.onUtilRunTask(SpongeImplHooks.java:325)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.server.MinecraftServer.redirect$zjj000$onRun(MinecraftServer.java:4682)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:723)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:397)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:668)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at java.lang.Thread.run(Thread.java:823)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF] Caused by: java.lang.NullPointerException
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at com.griefdefender.claim.GDClaim$ClaimBuilder.build(GDClaim.java:2860)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at com.griefdefender.internal.provider.GDWorldEditProvider.createClaim(GDWorldEditProvider.java:398)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         at com.griefdefender.command.CommandClaimWorldEdit.execute(CommandClaimWorldEdit.java:50)
[00:33:11] [Server thread/ERROR] [griefdefender]: [ACF]         ... 30 more

Tested this too, only with newer Spongeforge, it doesn't matter what parent claim type is, subdivision at all using worldedit selection causes this error.

Thanks for the testing/clarification!
I adjusted title and issue


Visualise distribution of numbers of cases per region / subregion (example: Germany)

show how histogram of incidence numbers per region changes over time
Draft idea in tools/pelican/content/ipynb/daily-distribution-new-cases-germany.ipynb

The same idea has been taken up by the Welt at https://www.zeit.de/wissen/gesundheit/coronavirus-echtzeit-karte-deutschland-landkreise-infektionen-ausbreitung: they only show the green bars to the left: the number of Landkreise that have 0 infections as a function of time. Bitmap attached.
This would be  a good first thing to compute (to explain what we are doing), and then to move on to the histograms, and the 3d view of that.

Made some progress (branch: daily-distribution-new-cases-germany, in oscovida/tools/pelican/content/ipynb/daily-distribution-new-cases-germany2.ipynb):

For entertainment another visualisation attempt:


Implement user logins and sessions

User Stories

As a user, I want to be able to log in and have my session remembered if I refresh the page

Users can now log in and out session remembered if a user refresh the page.


ci: revert changes related to Benchmark CI

What does this PR introduce?

PR Checklist

[ ] Wrote Tests
[ ] Wrote & Generated Documentation (readme/natspec/dodoc)
[ ] Ran npm run lint && npm run lint:solidity (solhint)
[ ] Ran npm run format (prettier)
[ ] Ran npm run build
[ ] Ran npm run test

Task linked: DEV-9462 Fix CI to run benchmark from Fork

Revert #840


There are some js to c/wasm transpilers but:

          There are some js to c/wasm transpilers but:

they only supports a small subset of js
https://stackoverflow.com/a/5193132/4011614

Very, very tricky --- Javascript is a heavily dynamic language where pretty much everything can be changed at run time: object properties, functions, types, etc.
As such it maps very badly onto C. Any Javascript translator would have to be able to cope with such things, which means it would have to translate the Javascript into machine code at run-time - which makes it a JIT, which we're already using.

So a transpiler that supported every js feature would still be slower overall.
https://www.assemblyscript.org is very similar to js/ts but would also require rewriting or substantially modifying all libs, in which case using another language is most likely a better choice
Originally posted by @j4k0xb in https://github.com/j4k0xb/webcrack/issues/23#issuecomment-1806459574
found this:
https://www.wasm.builders/gunjan_0307/compiling-javascript-to-wasm-34lk

.
(you can keep commenting on the previous issue btw 😅)
javy uses quickjs under the hood which is a javascript interpreter that can be run in wasm. The javascript itself doesn't get compiled to wasm.
I used it before to evaluate some parts of the obfuscated code before but the reason was for sandboxing and not speed.
As you can see v8 is roughly 30-100x as fast: https://bellard.org/quickjs/bench.html

what about bytenode?

I haven't evaluated if it would fully cover your needs, but if you haven't already, you might like to look at tree-sitter / web-tree-sitter / etc:

My suggestion here would be a rather major change, so it's not worth looking into too deeply unless it turns out there is no good/efficient way to manage this with the current AST parsers/etc; but one thing I stumbled across/was thinking about the other day was that the swc / tree-sitter / etc Rust/etc parsers can apparently be used from JS apps; and how that might allow us to run the unminify process much faster and/or in a potentially more memory efficient way:
These links/resources probably aren't exhaustive; but figured I would share them as a starting point in case this was a path that was worth looking into at some stage:
tree-sitter / web-tree-sitter

https://github.com/tree-sitter/node-tree-sitter

Node.js bindings for tree-sitter

https://www.npmjs.com/package/web-tree-sitter

https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web

WebAssembly bindings to the Tree-sitter parsing library

https://crates.io/crates/tree-sitter-javascript

https://github.com/tree-sitter/tree-sitter-javascript

JavaScript and JSX grammar for tree-sitter. For TypeScript, see tree-sitter-typescript.

swc / @swc/wasm-web / swc_ecma_parser

https://swc.rs/

https://swc.rs/docs/usage/core#parse
https://swc.rs/docs/usage/wasm

Some examples of using @swc/wasm-web to parse code to an AST in AST Explorer(s)

https://github.com/fkling/astexplorer/blob/master/website/src/parsers/js/swc.js#L3
https://github.com/sxzz/ast-explorer/blob/main/composables/language/javascript.ts#L74-L109

https://github.com/swc-project/swc/discussions/3713

Running a plugin with @swc/wasm-web

https://rustdoc.swc.rs/swc/index.html

https://rustdoc.swc.rs/swc_ecma_parser/
https://rustdoc.swc.rs/swc_ecma_transforms_base/fn.resolver.html
https://rustdoc.swc.rs/swc_ecma_minifier/eval/struct.Evaluator.html

https://www.christopherbiscardi.com/how-to-print-a-javascript-ast-using-swc-and-rust

How to print a JavaScript AST using SWC and Rust

https://blog.logrocket.com/writing-webpack-plugins-rust-using-swc/

Writing webpack plugins in Rust using SWC for faster builds

https://github.com/swc-project/swc/discussions/3254

Q: How to manipulate nodes+parent_nodes in AST & generate the tree back to JS code?

https://play.swc.rs/

The output can be set to AST

ast-grep

https://ast-grep.github.io/reference/api.html

ast-grep currently has an experimental API for Node.js

Etc

https://github.com/rustwasm/wasm-pack

This tool seeks to be a one-stop shop for building and working with rust- generated WebAssembly that you would like to interop with JavaScript, in the browser or with Node.js. wasm-pack helps you build rust-generated WebAssembly packages that you could publish to the npm registry, or otherwise use alongside any javascript packages in workflows that you already use

https://rustwasm.github.io/wasm-pack/book/

Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/35#issuecomment-1815952802

that the swc / tree-sitter / etc Rust/etc parsers can apparently be used from JS apps

I experimented a while ago with using rust parsers and converting back to the babel AST format: https://github.com/coderaiser/swc-to-babel
The parsing is faster but unfortunately transferring all the objects to the JS context has so much overhead that its slower than babel. So it should most likely be entirely JS or a native language, not mixed.
Another WIP attempt with https://github.com/meriyah/meriyah + https://github.com/j4k0xb/estree-to-babel/tree/perf could speed up parsing from 1.5s to 900ms for a large file
Not sure if its worth it since parsing takes up like 2% and transforms 98% of the time

How did you go about transfering the objects back to JS out of curiosity?

Its serializing it with serde_json::to_string and deserializing with JSON.parse.
Idk how to benchmark how long the native parsing vs serializing takes, here's the combined time for a 1.2MB bundle (380k nodes):

SWC parse: 959 ms (bindings.parseSync 491 ms + JSON.parse 466 ms)
babel parse: 442 ms
meriyah parse: 171 ms

or alternatively, maybe a way to keep the actual AST objects within the rust side of things, but be able to manipulate them from JS functions/similar?

Definitely a possibility, but again the whole project would have to be rewritten in rust

sometimes it might take an unreasonable amount of time and/or memory to try and unbundle them

Do you have examples where this happens? The current time/memory usage is not that unreasonable imo:

1.2MB webpack bundle: 1.1s unpack, 6s total, 450mb memory
7.2MB obfuscated browserify bundle: 0.3s unpack, 14s total, 1.1GB memory
8MB webpack bundle: 4s unpack, 17s total, 1.2GB memory

to use a rust/etc based tool to first extract them into individual module files (as an example), and then potentially process them in a later 'unminify' step

I also thought about something similar where transforms could be performed in parallel for each module.
How would that work with other kinds of scripts though? For example for deobfuscation its necessary to unminify everything before, a script could contain no bundle or a bundle in a deeply nested part of the code

(I thought I replied to this the other week, but apparently got distracted somewhere along the way.. 😅)

here's the combined time for a 1.2MB bundle (380k nodes)

@j4k0xb True true, that doesn't look too bad all in all then.

Definitely a possibility, but again the whole project would have to be rewritten in rust

@j4k0xb Yeah.. that would be less than ideal.
I don't know how possible it is, or if there are existing libraries/etc that would make it easier, but my original thought in this area was if it was possible to basically 'describe the transformation to be made' on the JS side of things, and then pass that through to the rust side for it to do the 'heavy lifting'.
If that were possible, then it should remove the need to convert the whole project to rust, and also would remove the performance penalty of needing to pass the whole AST back and forth across the rust/JS boundary.
The 2 ways I was initially pondering that this might be possible were:

a rust library that exposes an 'AST transformation DSL', such that the JS side could use that DSL to describe the transformations, then pass those to the rust side of things to actually process/run
a rust library that can run/interpret some basic JS code within rust itself, and use that to apply the transformations (that are still written in JS)

There was also one more idea I was pondering, though not sure if it would be impractical due to needing to cross back and forth across the rust/JS boundary (this idea is less thought through than the others above; and realistically could probably work with/alongside them, particularly the DSL idea):

somehow do the main 'AST walking'/modification on the rust side of things, but be able to specify when it should pass a smaller subset of the AST back to the JS side for more specific processing

Do you have examples where this happens?

@j4k0xb I don't personally have specific examples, and the one I was thinking of was using wakaru rather than webcrack, so it might not even apply here; but here are the details from that issue:

I have a 5Mb sized file that needs to be processed, I've tried running it directly in node and it takes up to 21G of memory space in some transform processing.
Originally posted by @StringKe in https://github.com/pionxzh/wakaru/issues/35#issue-1987139845

I tried the sample with the new CLI.
It took ~2GB and 60 seconds to pass for unpacker (and failed to unpack it into multiple files😞; I will check further!).
Generated 1 modules from main.ba3b216f.js to out (61,265.4ms)

For unminify, it took 3.4 hours... to finish the whole process. Memory usage is from 100MB ~ 1GB ~ 2GB depending on the rule.
I will improve the CLI to write files more frequently during the unminify process, and add a --perf flag for recording the time and memory usage.
Originally posted by @pionxzh in https://github.com/pionxzh/wakaru/issues/35#issuecomment-1818082000

The following may also be of interest:

I wonder if there are any methods that could be used to optimize that in a way that would speed things up; or alternatively, maybe a way to keep the actual AST objects within the rust side of things, but be able to manipulate them from JS functions/similar?

Actually that's how ast-grep works. Here is a quote from Benchmark TypeScript Parsers: Demystify Rust Tooling Performance:

Tree-sitter and ast-grep avoid serde overhead by returning a tree object rather than a full AST structure. Accessing tree nodes requires invoking Rust methods from JavaScript, which distributes the cost over the reading process.

Originally posted by @pionxzh in https://github.com/pionxzh/wakaru/issues/35#issuecomment-1838763473

Actually that's how ast-grep works.

Oh true. I was thinking it probably did, particularly after skimming the source the other day (Ref), but I wasn't 100% sure still.

Here is a quote from Benchmark TypeScript Parsers: Demystify Rust Tooling Performance:
tree-sitter and ast-grep avoid serde overhead by returning a tree object rather than a full AST structure. Accessing tree nodes requires invoking Rust methods from JavaScript, which distributes the cost over the reading process.

That's super interesting and neat to know. For future reference, here's a link to the non-medium-paywalled version of that article (Ref) It was a really interesting read in general!
The results of the benchmarks for synchronous performance were pretty interesting, I definitely didn't expect swc to perform so poorly in general, nor for babel to perform unexpectedly so much better on the medium-sized file compared to most other things (I wonder what made it perform so well there?). It was also interesting to see that ast-grep consistently beat tree-sitter on it's own, despite it using tree-sitter as it's parser.

It was also really interesting in the async parsing to see just how much ast-grep seems to dominate everything; again seeming to perform way better than the tree-sitter parser it's built on top of:

These notes towards the end were also interesting/worth paying attention to:

Native Parser Performance Tricks
tree-sitter & ast-grep' Edge
These parsers manage to bypass serde costs post-parsing by returning a Rust object wrapper to Node.js. This strategy, while efficient, can lead to slower AST access in JavaScript as the cost is amortized over the reading phase.
ast-grep's async advantage:
ast-grep's performance in concurrent parsing scenarios is largely due to its utilization of multiple libuv threads. By default, the libuv thread pool size is set to four, but there's potential to enhance performance further by expanding the thread pool size, thus fully leveraging the available CPU cores.

Cool to see this note at the very end aligns with one of the thoughts I had too!

Shifting Workloads to Rust: The creation of a domain-specific language (DSL) tailored for AST node querying could shift a greater portion of computational work to the Rust side, enhancing overall efficiency.

Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/35#issuecomment-1839675606

If you get deeper into benchmarking/optimisation of the Rust side of things, this was a good read:

https://medium.com/source-and-buggy/data-driven-performance-optimization-with-rust-and-miri-70cb6dde0d35

Which I summarised the main points as:

The tl;dr of that blog seemed to be that they first used flamegraph, which was mildly useful, but not specifically detailed enough:

https://github.com/flamegraph-rs/flamegraph

A Rust-powered flamegraph generator with additional support for Cargo projects! It can be used to profile anything, not just Rust projects!

And then moved on to using miri and a less-obvious feature of it to generate a far more detailed trace, that could then be analysed within Google Chrome's DevTools performance tab:

https://github.com/rust-lang/miri

An experimental interpreter for Rust's mid-level intermediate representation (MIR)

https://github.com/rust-lang/miri#miri--z-flags-and-environment-variables

-Zmiri-measureme=<name> enables measureme profiling for the interpreted program. This can be used to find which parts of your program are executing slowly under Miri. The profile is written out to a file inside a directory called <name>, and can be processed using the tools in the repository https://github.com/rust-lang/measureme

https://github.com/rust-lang/measureme#crox

crox turns measureme profiling data into files that can be visualized by the Chromium performance tools.

https://github.com/rust-lang/measureme/tree/master/crox#readme

Originally posted by @0xdevalias in https://github.com/ast-grep/ast-grep/issues/144#issuecomment-1839767054


Create conversion option if needed

All messages are now supposed to be shown in UTF-8. If a message is actually Base64 it should be able to be converted. See old GUI for the changes.

[ ] Inform Martijn of this feature

More information for when implementing frontend tests:
https://github.com/ibissource/ibis-ladybug/issues/128
The test webapp has a link “Message encoded using Base64” which will generate a report to open with Cypress. With the first checkpoint the Base64 button can be used and with the other checkpoints some special characters (“ë©” and “😊”) can be tested