Rewite with rust-s3 and caching

This merge requests provide:

update to datafusion master branch
drop aws-sdk-rust in favor of much simpler rust-s3
Tested with self-hosted S3 servers: MinIO,  SeaweedFS, s3-server (https://crates.io/crates/s3-server) (no unit tests)
caching buffers
concurrently pre-feching metadata for all parquet files during listing
invalidate cached parquet metadata when file last modification changed
Tested with partitioned tables (no unit tests)

Caching
Motivation why caching is needed is because current datafusion implementation is not optimal for network file operations:

datafusion fetch and parse metadata for every parquet file 2 times during every query (3 time if you count registering table). Once during creating physical plan and second while executing it.
datafusion sometimes requests very small file chucks (I got a lots of 14 bytes queries)
datafusion executing plan don't read file in sequence. It's start from beginning for every column.

Results of very unscientific tests on local machine with my example SQL query:

Query using filesystem took 0.5 seconds
Query using S3 without cache took 25 seconds
Query using S3 with default cache options took 0.8 seconds

Why not aws-sdk-rust?
I don't know how AWS works, units test on my local machine took several minutes to complete (with success). I think it was because of IMDS timeouts. I couldn't find how to disable it, so I have chosen much simpler implementation of S3 client.
I had also problems with rust_s3 and other S3 self-hosted server. I created MRs for fix it: https://github.com/durch/rust-s3/pull/267 and https://github.com/datenlord/s3-server/pull/114
What can be done next?

check etag/last modification in every file query and return error when file has been changed during query execution.

thank you very much for the contribution @mateuszkj 👍
CC @seddonm1 (as the original core author) and @houqp (has helped with reviewing changes / design here and has experience with delta-rs which i believe connects to s3 with rusoto).
To confirm the primary reason for switching to rust-s3 is performance?
Indeed rust-s3 does look much lighter, has significant usage, appears to be actively maintained (release a month ago), and it seems it might enable GCS as well.  however, im a little concerned about moving from official aws implementation.  i think we know that the official implementation will continue to be maintained and enhanced moving forward while its unclear what what the status of 3rd party libraries will be (i.e. rusoto going into maintenance mode https://github.com/rusoto/rusoto).  my specific concern being we will have to rewrite again if the same thing happens to this library.  can you provide any insight on this?
regarding the cache - can you please include a more implementation details and a description of how specifically it is used?  Could we perhaps start with just using the cache functionality with aws-sdk-rust?
regarding the IMDS timeout - do you have an example in other languages of how to disable it that we could use as reference?  and rust-s3 allowed disabling IMDS timeout?
regarding testing other s3 servers - if it isnt too much work would you be able to run CI against them using docker files (it looks like seaweed makes that easy)?  I think this would be a great value add to the project regardless and would make comparing results easier.
Given the size of the change i will need some time to review the code - so please bare with me as i will likely need at least a few days to review and test.
thank you again for the contribution.

thanks for your fast response @matthewmturner
After second thought I think this MR should not be merged. It can be done better. Can you mark this MR as draft?.
Moving cache mechanism to different crate
Now I consider that caching mechanism could be moved to generic ObjectStore implementation in another crate. Then in constructor we could pass any implementation of direct ObjectStore.
for example
let s3_file_system = S3FileSystem::new(...).await?;
let s3_cached_file_system = Arc::new(ObjectStoreCache::new(s3_file_system, ObjectStoreCacheOptions::from_envs()?));
runtime_env.register_object_store("s3", s3_file_system.clone());

with cache in different crate, we could have two S3 client implementations or other remote storage systems.
Cache description
First let look how datafusion read files without cache.
To disable cache set environment variables:
export RUST_LOG=info,datafusion_objectstore_s3=debug
export DATAFUSION_S3_MIN_REQUEST_SIZE=0
export DATAFUSION_S3_USE_METADATA_PREFETCH=false

Below is example of querying data using SQL simillar to:
SELECT
  s.col3_text,
  s.col4_int
FROM stops s
WHERE s.col1_time >= CAST('2022-03-02 00:00:00') AS Timestamp)
  AND s.col1_time < CAST('2022-03-03 00:00:00') AS Timestamp)
  AND s.col2_text = 'xxx'
ORDER BY s.col1_time;

Query uses 4 columns and filters by time and some text field. I have multiple *.parquet files in my table (89 files). I greped only one for clarity.
# Regier table. datafusion read metadata of every *.parquet file to collect schemas.
2022-04-10T18:28:13.751524Z direct id: 25 file: s3://data-2022-02.parquet start: 982029 len: 65536, rlen: 65536
2022-04-10T18:28:13.760407Z direct id: 25 file: s3://data-2022-02.parquet start: 951802 len: 30227, rlen: 30227

# datafusion read metadata of every *.parquet file to create phisical plan
2022-04-10T18:28:17.134288Z direct id: 203 file: s3://data-2022-02.parquet start: 982029 len: 65536, rlen: 65536
2022-04-10T18:28:17.150239Z direct id: 203 file: s3://data-2022-02.parquet start: 951802 len: 30227, rlen: 30227

# datafusion read metadata of every *.parquet file to execute phisical plan.
2022-04-10T18:28:19.187678Z direct id: 290 file: s3://data-2022-02.parquet start: 982029 len: 65536, rlen: 65536
2022-04-10T18:28:19.203489Z direct id: 290 file: s3://data-2022-02.parquet start: 951802 len: 30227, rlen: 30227

# datafusion read data from first column (iterdate over row groups)
2022-04-10T18:28:19.223254Z direct id: 290 file: s3://data-2022-02.parquet start: 220862 len: 14, rlen: 14
2022-04-10T18:28:19.235615Z direct id: 290 file: s3://data-2022-02.parquet start: 225370 len: 14, rlen: 14
2022-04-10T18:28:19.248903Z direct id: 290 file: s3://data-2022-02.parquet start: 229878 len: 14, rlen: 14
2022-04-10T18:28:19.257919Z direct id: 290 file: s3://data-2022-02.parquet start: 234386 len: 14, rlen: 14
2022-04-10T18:28:19.271822Z direct id: 290 file: s3://data-2022-02.parquet start: 238894 len: 14, rlen: 14
2022-04-10T18:28:19.284739Z direct id: 290 file: s3://data-2022-02.parquet start: 462074 len: 14, rlen: 14
2022-04-10T18:28:19.298708Z direct id: 290 file: s3://data-2022-02.parquet start: 466582 len: 14, rlen: 14
2022-04-10T18:28:19.313049Z direct id: 290 file: s3://data-2022-02.parquet start: 471090 len: 14, rlen: 14
2022-04-10T18:28:19.325878Z direct id: 290 file: s3://data-2022-02.parquet start: 475598 len: 14, rlen: 14
2022-04-10T18:28:19.338194Z direct id: 290 file: s3://data-2022-02.parquet start: 480106 len: 14, rlen: 14
2022-04-10T18:28:19.350122Z direct id: 290 file: s3://data-2022-02.parquet start: 704027 len: 14, rlen: 14
2022-04-10T18:28:19.362426Z direct id: 290 file: s3://data-2022-02.parquet start: 708535 len: 14, rlen: 14
2022-04-10T18:28:19.374202Z direct id: 290 file: s3://data-2022-02.parquet start: 713043 len: 14, rlen: 14
2022-04-10T18:28:19.386540Z direct id: 290 file: s3://data-2022-02.parquet start: 717551 len: 14, rlen: 14
2022-04-10T18:28:19.400017Z direct id: 290 file: s3://data-2022-02.parquet start: 722059 len: 14, rlen: 14
2022-04-10T18:28:19.412182Z direct id: 290 file: s3://data-2022-02.parquet start: 947346 len: 14, rlen: 14

# datafusion read data from second column (iterdate over row groups)
2022-04-10T18:28:19.424140Z direct id: 290 file: s3://data-2022-02.parquet start: 220981 len: 14, rlen: 14
2022-04-10T18:28:19.436095Z direct id: 290 file: s3://data-2022-02.parquet start: 225489 len: 14, rlen: 14
2022-04-10T18:28:19.448831Z direct id: 290 file: s3://data-2022-02.parquet start: 229997 len: 14, rlen: 14
2022-04-10T18:28:19.461895Z direct id: 290 file: s3://data-2022-02.parquet start: 234505 len: 14, rlen: 14
2022-04-10T18:28:19.474107Z direct id: 290 file: s3://data-2022-02.parquet start: 239013 len: 14, rlen: 14
2022-04-10T18:28:19.488363Z direct id: 290 file: s3://data-2022-02.parquet start: 462193 len: 14, rlen: 14
2022-04-10T18:28:19.503114Z direct id: 290 file: s3://data-2022-02.parquet start: 466701 len: 14, rlen: 14
2022-04-10T18:28:19.515595Z direct id: 290 file: s3://data-2022-02.parquet start: 471209 len: 14, rlen: 14
2022-04-10T18:28:19.527923Z direct id: 290 file: s3://data-2022-02.parquet start: 475717 len: 14, rlen: 14
2022-04-10T18:28:19.539998Z direct id: 290 file: s3://data-2022-02.parquet start: 480225 len: 14, rlen: 14
2022-04-10T18:28:19.553284Z direct id: 290 file: s3://data-2022-02.parquet start: 704146 len: 14, rlen: 14
2022-04-10T18:28:19.566201Z direct id: 290 file: s3://data-2022-02.parquet start: 708654 len: 14, rlen: 14
2022-04-10T18:28:19.578198Z direct id: 290 file: s3://data-2022-02.parquet start: 713162 len: 14, rlen: 14
2022-04-10T18:28:19.589092Z direct id: 290 file: s3://data-2022-02.parquet start: 717670 len: 14, rlen: 14
2022-04-10T18:28:19.602429Z direct id: 290 file: s3://data-2022-02.parquet start: 722178 len: 14, rlen: 14
2022-04-10T18:28:19.615445Z direct id: 290 file: s3://data-2022-02.parquet start: 947465 len: 14, rlen: 14

# datafusion read data from third column (iterdate over row groups)
2022-04-10T18:28:19.628900Z direct id: 290 file: s3://data-2022-02.parquet start: 221621 len: 14, rlen: 14
2022-04-10T18:28:19.642048Z direct id: 290 file: s3://data-2022-02.parquet start: 226129 len: 14, rlen: 14
2022-04-10T18:28:19.656429Z direct id: 290 file: s3://data-2022-02.parquet start: 230637 len: 14, rlen: 14
2022-04-10T18:28:19.669087Z direct id: 290 file: s3://data-2022-02.parquet start: 235145 len: 14, rlen: 14
2022-04-10T18:28:19.681621Z direct id: 290 file: s3://data-2022-02.parquet start: 239653 len: 14, rlen: 14
2022-04-10T18:28:19.696082Z direct id: 290 file: s3://data-2022-02.parquet start: 462833 len: 14, rlen: 14
2022-04-10T18:28:19.708401Z direct id: 290 file: s3://data-2022-02.parquet start: 467341 len: 14, rlen: 14
2022-04-10T18:28:19.721417Z direct id: 290 file: s3://data-2022-02.parquet start: 471849 len: 14, rlen: 14
2022-04-10T18:28:19.733059Z direct id: 290 file: s3://data-2022-02.parquet start: 476357 len: 14, rlen: 14
2022-04-10T18:28:19.745427Z direct id: 290 file: s3://data-2022-02.parquet start: 480865 len: 14, rlen: 14
2022-04-10T18:28:19.758242Z direct id: 290 file: s3://data-2022-02.parquet start: 704786 len: 14, rlen: 14
2022-04-10T18:28:19.770439Z direct id: 290 file: s3://data-2022-02.parquet start: 709294 len: 14, rlen: 14
2022-04-10T18:28:19.783198Z direct id: 290 file: s3://data-2022-02.parquet start: 713802 len: 14, rlen: 14
2022-04-10T18:28:19.795203Z direct id: 290 file: s3://data-2022-02.parquet start: 718310 len: 14, rlen: 14
2022-04-10T18:28:19.808062Z direct id: 290 file: s3://data-2022-02.parquet start: 722818 len: 14, rlen: 14
2022-04-10T18:28:19.827283Z direct id: 290 file: s3://data-2022-02.parquet start: 948105 len: 14, rlen: 14

# datafusion read data from four column (iterdate over row groups)
2022-04-10T18:28:19.840199Z direct id: 290 file: s3://data-2022-02.parquet start: 223858 len: 14, rlen: 14
2022-04-10T18:28:19.853255Z direct id: 290 file: s3://data-2022-02.parquet start: 228366 len: 14, rlen: 14
2022-04-10T18:28:19.866894Z direct id: 290 file: s3://data-2022-02.parquet start: 232874 len: 14, rlen: 14
2022-04-10T18:28:19.878996Z direct id: 290 file: s3://data-2022-02.parquet start: 237382 len: 14, rlen: 14
2022-04-10T18:28:19.892441Z direct id: 290 file: s3://data-2022-02.parquet start: 241890 len: 14, rlen: 14
2022-04-10T18:28:19.906870Z direct id: 290 file: s3://data-2022-02.parquet start: 465070 len: 14, rlen: 14
2022-04-10T18:28:19.925541Z direct id: 290 file: s3://data-2022-02.parquet start: 469578 len: 14, rlen: 14
2022-04-10T18:28:19.939221Z direct id: 290 file: s3://data-2022-02.parquet start: 474086 len: 14, rlen: 14
2022-04-10T18:28:19.951420Z direct id: 290 file: s3://data-2022-02.parquet start: 478594 len: 14, rlen: 14
2022-04-10T18:28:19.964162Z direct id: 290 file: s3://data-2022-02.parquet start: 483102 len: 14, rlen: 14
2022-04-10T18:28:19.977930Z direct id: 290 file: s3://data-2022-02.parquet start: 707023 len: 14, rlen: 14
2022-04-10T18:28:19.992568Z direct id: 290 file: s3://data-2022-02.parquet start: 711531 len: 14, rlen: 14
2022-04-10T18:28:20.005620Z direct id: 290 file: s3://data-2022-02.parquet start: 716039 len: 14, rlen: 14
2022-04-10T18:28:20.016705Z direct id: 290 file: s3://data-2022-02.parquet start: 720547 len: 14, rlen: 14
2022-04-10T18:28:20.028415Z direct id: 290 file: s3://data-2022-02.parquet start: 725055 len: 14, rlen: 14
2022-04-10T18:28:20.040173Z direct id: 290 file: s3://data-2022-02.parquet start: 950342 len: 14, rlen: 14

What we see in logs:

I got 70 requests above single file. 2744 request for total of 89 files in my table.
I got 3 times request for the same metadata for every parquet file.
During creating physical plan metadata from all parquet files are read in sequence on single thread.
Datafusion always perform listing files before reading metadata during creating physical plan
Datafusion first iterate over columns then row groups.
I got lot's of small requests with 14 bytes.
file data-2022-02.parquet have data from another month, despite this datafusion reached this parquet file. I think I have statistics in my *.parquet file.

My understanding to implement cache which was:

Store metadata for all parquet files in RAM. So, we don't have to read it from S3 server every single SQL query.
Read metadata for all parquet files in parallel/concurrently. Listing files looks like good place for it. It's async and is done every time datafusion create physical plan.
Invalidate metadata cache when file last modification time changed.
Use multiple cache buffers per file, so when datafusion read next column there is good change we already have that data (my implementation is not idea here)

CI with other self-hosted S3 servers
About running CI with other self-hosted S3 servers is great idea. I would like to add it but not for now but I don't want to promise when or if.  Now i want to focus on different things. I will consider it when (and if) i will move cache mechanism to different crate.
aws-sdk-rust nad rust-s3
About why I dropped aws-sdk-rust. I don't have experience with AWS. I don't have a clue what IMDS is and why is required (?). When I run unit tests on my linux machine I got logs with IMDS timeout for every S3 request (somehow unit tests passed successfully but took long time to complete).  Maybe I have to install something or disable (?). After spending haft of day on this problem i just rewired it to rust-s3 which worked at first try and it was very fast.
One of drawbacks of rust-s3 is that only one person can publish to crates.io.
Maybe we need two S3 implementations, one for AWS services with official aws-sdk-rust crate and another for self-hosted S3 servers without AWS complexity. Or maybe IMDS complexity can be disabled with aws-sdk-rust.
Why I need this changes
My personal intention is to run datafusion on premise cluster and store parquet files on different machines. I could move in future away from S3 protocol in favor of seaweedfs internal protocol or use ballista. S3 protocol looks like good transition for me from distributed  storage systems for datafusion.

Thanks for the detailed write up @mateuszkj - it makes sense.
I think that bringing up the cache on the main datafusion repo would be a good starting point to see what the community thinks about this.  For example there is an active HDFS community using datafusion as well (https://github.com/datafusion-contrib/datafusion-objectstore-hdfs) and perhaps this could benefit them.  In general there is a lot of focus on improving performance - and parallelism within datafusion (for example https://github.com/apache/arrow-datafusion/issues/2199) so i think its a good time to bring it up.  I can help with creating that if you would like.
Regarding CI and testing with other S3 providers - I can work on this.  I think its a valuable addition to the project regardless and should help with whatever changes come from this.
I will look into reproducing the IMDS issue.  I think once we have a better understanding of it then we can consider whether it makes sense to use alternative s3 implementations.
Thanks working with me on this!

@tustvold thank you for the insight - very helpful.  personally, i am interested in learning about and implementing these types of performance improvements. so if you could draft some issues outlining what you are looking to accomplish i would be happy to try and help.

@tustvold No rush - I have a couple things I'll need to close before being able to start on that anyway. Thanks again.

Hi everyone. I wrote the base implementation so any blame can be directed at me 😉.
I think moving away from the official AWS SDK would be a mistake as they are developing a strong Rust practice so I would expect it to be better maintained than the others going forward (maintaining a library for someone else's API is a bit of a thankless task).
This was always a bit of a hacky implementation until Async datasources are properly implemented. It is good the IOx team is onboard as I hope we can arrive on one canonical implementation - that deals with all of the issues above. That means any caching ideas would be good to incorporate. 👍

@seddonm1 No blame being directed - we wouldn't be having this conversation without your efforts.

As @tustvold mentioned, IMDS won't work for custom S3 implementations. If this is the problem that's causing the slow down, then it's likely that the official S3 client is not as bad as we observed here. It is usually used when a default credential chain is configured for the client, then it will use IMDS to discover what AWS environment the client is in and what methods to use to fetch the IAM credential for S3 access. The client might use IMDS to discover which region it's in as well. As a quick test, you should be able to turn it off by using a static client credential by hard coding the access keys and region. If that doesn't work, I recommend filing an issue in the upstream official AWS repo, it should be a fairly straight forward fix.

Another place that I think caching could help is at the object store level to perform look ahead scan, i.e. fetch the next chunk of data from S3 while the system is performing compute on the previous chunk.


fix: deps security updates

This PR addresses several high-severity security vulnerabilities found when running npm install.
List of commands ran and logs:
npm install
npm warn deprecated<EMAIL_ADDRESS>This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated<EMAIL_ADDRESS>Use @eslint/config-array instead
npm warn deprecated<EMAIL_ADDRESS>Rimraf versions prior to v4 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>Use @eslint/object-schema instead
npm warn deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported

added 1164 packages, and audited 1165 packages in 13s

331 packages are looking for funding
  run `npm fund` for details

14 vulnerabilities (2 moderate, 12 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
npm audit fix --force
npm warn using --force Recommended protections disabled.
npm warn audit Updating react-email to 1.10.1, which is a SemVer major change.
npm warn audit Updating react-email to 1.10.1, which is a SemVer major change.
npm warn audit Updating @aws-sdk/s3-presigned-post to 3.658.1, which is outside your stated dependency range.
npm warn audit Updating @aws-sdk/s3-presigned-post to 3.658.1, which is outside your stated dependency range.

added 143 packages, removed 134 packages, changed 44 packages, and audited 1174 packages in 13s

331 packages are looking for funding
  run `npm fund` for details

# npm audit report

fast-xml-parser  <4.4.1
Severity: high
fast-xml-parser vulnerable to ReDOS at currency parsing - https://github.com/advisories/GHSA-mpg4-rc92-vx8v
fix available via `npm audit fix --force`
Will install<EMAIL_ADDRESS>which is outside the stated dependency range
node_modules/fast-xml-parser
  @aws-sdk/core  3.529.1 - 3.620.1
  Depends on vulnerable versions of fast-xml-parser
  node_modules/@aws-sdk/core
    @aws-sdk/client-s3  3.529.1 - 3.620.1
    Depends on vulnerable versions of @aws-sdk/client-sso-oidc
    Depends on vulnerable versions of @aws-sdk/client-sts
    Depends on vulnerable versions of @aws-sdk/core
    Depends on vulnerable versions of @aws-sdk/credential-provider-node
    node_modules/@aws-sdk/client-s3
    @aws-sdk/client-sso  3.529.1 - 3.620.1
    Depends on vulnerable versions of @aws-sdk/core
    node_modules/@aws-sdk/client-sso
      @aws-sdk/credential-provider-sso  3.529.1 - 3.620.1
      Depends on vulnerable versions of @aws-sdk/client-sso
      node_modules/@aws-sdk/credential-provider-sso
        @aws-sdk/credential-provider-ini  3.529.1 - 3.620.1
        Depends on vulnerable versions of @aws-sdk/credential-provider-sso
        node_modules/@aws-sdk/credential-provider-ini
        @aws-sdk/credential-provider-node  3.529.1 - 3.620.1
        Depends on vulnerable versions of @aws-sdk/credential-provider-ini
        Depends on vulnerable versions of @aws-sdk/credential-provider-sso
        node_modules/@aws-sdk/credential-provider-node
          @aws-sdk/client-sso-oidc  3.529.1 - 3.620.1
          Depends on vulnerable versions of @aws-sdk/client-sts
          Depends on vulnerable versions of @aws-sdk/core
          Depends on vulnerable versions of @aws-sdk/credential-provider-node
          node_modules/@aws-sdk/client-sso-oidc
            @aws-sdk/client-sts  3.529.1 - 3.620.1
            Depends on vulnerable versions of @aws-sdk/client-sso-oidc
            Depends on vulnerable versions of @aws-sdk/core
            Depends on vulnerable versions of @aws-sdk/credential-provider-node
            node_modules/@aws-sdk/client-sts

next  14.0.0 - 14.2.9
Severity: high
Next.js Cache Poisoning - https://github.com/advisories/GHSA-gp8f-8m3g-qvj9
fix available via `npm audit fix --force`
Will install<EMAIL_ADDRESS>which is outside the stated dependency range
node_modules/next

10 high severity vulnerabilities

To address issues that do not require attention, run:
  npm audit fix

To address all issues, run:
  npm audit fix --force
npm audit fix --force
npm warn using --force Recommended protections disabled.
npm warn audit Updating next to 14.2.13, which is outside your stated dependency range.
npm warn audit Updating @aws-sdk/client-s3 to 3.658.1, which is outside your stated dependency range.

added 38 packages, removed 58 packages, changed 38 packages, and audited 1154 packages in 6s

330 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

I'm not a fan of downgrading react-email to 1 when it's on version 3.  I'll need to look into this.


chore(deps): update @babel/traverse

https://github.com/bcgov/nr-fom/security/dependabot/145
https://github.com/bcgov/nr-fom/security/dependabot/146
https://github.com/bcgov/nr-fom/security/dependabot/147
https://github.com/bcgov/nr-fom/security/code-scanning/68
@MCatherine1994 @ianliuwk1019 Any chance you understand why the API tests are failing?
Update: pulled out API traverse upgrade.

Thanks for the PR!
Any successful deployments (not always required) will be available below.

api
admin
public

Once merged, code will be promoted and handed off to following workflow run.

Main Merge Workflow

Replaced by https://github.com/bcgov/nr-fom/pull/545.


fix wine build error with debian buster

All Submissions:

[x] Have you ensured there aren't other open Pull Requests for the same update or change?
[x] Have you created a new branch for your changes and PR from that branch and not from your master branch?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?

Changelog for this PR:

added missing depencies on Buster

pls wait with merge, until i update it to bullseye :)

This shouldn't be necessary since base is now built for bullseye and the wine image is already successfully built?

This shouldn't be necessary since base is now built for bullseye and the wine image is already successfully built?

this pr update to Bullseye. Now Buster repos are used in published image

This shouldn't be necessary since base is now built for bullseye and the wine image is already successfully built?

this pr update to Bullseye. Now Buster repos are used in published image

Bullseye is used as far as I can see
https://github.com/parkervcp/yolks/blob/master/wine/latest/Dockerfile#L16

These 2 line should be in the Dockerfile
RUN     wget -O- -q https://download.opensuse.org/repositories/Emulators:/Wine:/Debian/Debian_11/Release.key | apt-key add -
RUN     echo "deb http://download.opensuse.org/repositories/Emulators:/Wine:/Debian/Debian_11 ./" | tee /etc/apt/sources.list.d/wine-obs.list
They contain additinal packages for wine


Consider ignoring raw reference

Some XML doesn't contain separate reference fields (e.g. title, year etc). They are currently counted as mismatches for the field evaluation. Consider ignoring them (as per GROBID end2end evaluation).

There is now a reference_text field that looks at the raw reference. Although it may still be worth excluding evaluation on other fields if they are not present in the target XML.


The call to LexActivatorNative.GetLicenseMeterAttribute raises a TypeError exception: this function takes 3 arguments (2 given)

https://github.com/cryptlex/lexactivator-python/blob/ab0b45747cc58eb07807ebe612764cd986818b49/cryptlex/lexactivator/lexactivator.py#L302

Thanks for reporting. The issue is fixed now.


Replace rustc_serialize dependency

Replace rustc_serialize dependency with a base64 and a hex crate
dependency. rustc_serialize provides a lot of additional (serialization)
capability which we don't utilize, so it makes sense to use smaller crates with
the specific features we need to reduce compile times and resulting binary
sizes.
Closes #467.

Looks like I forgot about the rustc_serialize usages in S3. I'll fix those up and update the PR.

Both base64 and hex seem to be pretty quiet.  Anyone have thoughts if it's because they're stable or if they're being neglected?  Rustc_serialize is still being actively developed.
Maybe we can do some checks to see what kind of size differences we get in compiled binaries with this PR.  And compile times.

I'll reiterate I would prefer to keep using rustc_serialize instead of dependencies that haven't seen updating, including accepting long standing PRs.  (Which I find hilarious with how old some of our PRs are 😉 )

Yeah I agree. If the dependencies were updated more I'd push for this harder but as-is I think we can close this pending better crates in the ecosystem.


Choice of GPL

What is the reason for the strong copyleft license? It's not clear how this applies to non-software artefacts such as the ones in this repo. Are the excel files here considered source?
For non-code repos I usually use a CC-BY license or CC-0 waiver, I would recommend this from mixs

Thank you Chris for your interest in the Genomic Standards Consortium MIxS
standards.
If you are looking to collaborate with the GSC, please contact us with a
project proposal.
Regards,
Lynn
On Thu, Mar 7, 2019 at 2:59 PM Chris Mungall<EMAIL_ADDRESS>wrote:

What is the reason for the strong copyleft license? It's not clear how
this applies to non-software artefacts such as the ones in this repo. Are
the excel files here considered source?
For non-code repos I usually use a CC-BY license or CC-0 waiver, I would
recommend this from mixs
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GenomicsStandardsConsortium/mixs/issues/22, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEIeDeUVfJA9V9Z3snvYwzQle3QJVQfcks5vUW-rgaJpZM4bj_tb
.

--
Lynn M. Schriml, Ph.D.
Associate Professor
Institute for Genome Sciences
University of Maryland School of Medicine
Department of Epidemiology and Public Health
801 W. Baltimore St., Room 659
Baltimore, MD 21201
P:<PHONE_NUMBER> |  F:<PHONE_NUMBER>
<EMAIL_ADDRESS>


Schema engine error - prepared statement "s0" already exists

Describe the Bug
I've forked the latest version of the project and I'm trying to run it on Vercel. I have a brand new database on Supabase.
I've followed the guide here https://umami.is/docs/guides/running-on-vercel and added the required DATABASE_URL environment variable. However, when the build runs on Vercel it fails with the error ERROR: prepared statement "s0" already exists
Database
PostgreSQL
Relevant log output
[15:26:50.039] Running "yarn run build"
[15:26:50.222] yarn run v1.22.19
[15:26:50.266] $ npm-run-all check-env build-db check-db build-tracker build-geo build-app
[15:26:50.669] $ node scripts/check-env.js
[15:26:50.945] $ npm-run-all copy-db-files build-db-client
[15:26:51.306] $ node scripts/copy-db-files.js
[15:26:51.423] Database type detected: postgresql
[15:26:51.431] Copied /vercel/path0/db/postgresql to /vercel/path0/prisma
[15:26:51.666] $ prisma generate
[15:26:52.158] Prisma schema loaded from prisma/schema.prisma
[15:26:52.839]
[15:26:52.839] ✔ Generated Prisma Client (v5.17.0) to ./node_modules/@prisma/client in 181ms
[15:26:52.839]
[15:26:52.840] Start by importing your Prisma Client (See: http://pris.ly/d/importing-client)
[15:26:52.840]
[15:26:52.840] Tip: Want to react to database changes in your app as they happen? Discover how with Pulse: https://pris.ly/tip-1-pulse
[15:26:52.840]
[15:26:53.288] $ node scripts/check-db.js
[15:26:53.391] ✓ DATABASE_URL is defined.
[15:26:54.043] ✓ Database connection successful.
[15:26:54.855] ✓ Database version check successful.
[15:26:56.793] Error: Schema engine error:
[15:26:56.793] ERROR: prepared statement "s0" already exists
[15:26:56.793]
[15:26:56.793] ✗ Command failed: prisma migrate deploy
[15:26:56.794] Error: Schema engine error:
[15:26:56.794] ERROR: prepared statement "s0" already exists
[15:26:56.794]
[15:26:56.794]
[15:26:56.810] error Command failed with exit code 1.
[15:26:56.810] info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
[15:26:56.826] ERROR: "check-db" exited with 1.
[15:26:56.841] error Command failed with exit code 1.
[15:26:56.842] info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
[15:26:56.864] Error: Command "yarn run build" exited with 1
[15:26:57.523]

Which Umami version are you using? (if relevant)
No response
Which browser are you using? (if relevant)
No response
How are you deploying your application? (if relevant)
Vercel

I still don't know why this doesn't work with Supabase but I've managed to get it working with a Vercel Postgres database.


how can i use different language

How can i use different language?
"Swipe to refresh"
"Release to refresh"
"Refreshing complete"

Unfortunately, the text in the default head view provided in the library can not be modified currently. You'd better create your own refresh head layout (see create-custom-head-refresh-view) to support different languages.


Update version: Google.Chrome.Dev version 117.0.5911.2

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/114777&drop=dogfoodAlpha

@wingetbot run

Installer Verification Analysis Context Information:
Specified hash doesn't match.
Actual hash:   D72E708F6DA5040448373B02AC8D49256B5F5BCBE4F1672E8A67EF008FC6A8E6
Expected hash: 4961b18339af691f12ee3938597435af3c2dc61a51bad6e6ed2a796c50828134
Specified hash doesn't match.
Actual hash:   CC59EE124485424D8D18444542E57CCCB7B5E4399923B734C10784EA1D3A762C
Expected hash: 782a09ffed2247a83f7e9c70d38431d1a4be1a1fb87e3fa2d5081328d69d5167

Reset PR


[SYSTEMML-2078] Support for global variables

Hi, I have cleaned up and closed the last pull request for this issue because I put the old commits in it. And I open this new PR for it.
In this PR, I add the support for global variables and some builtin global constants(PI, INF, NaN).
Thanks for review

Can an Admin verify this patch?

thanks for the cleanup @EdgarLGB - in general, this is a good start. However, I would recommend to split the handling of builtin constants and global variables into two PRs and start with the builtin constants.
The current approach prepends assignment statements such NaN = 0.0/0.0 which does work but could lead to weird behavior because users are still able to assign NaN=7 afterwards - this will likely be very confusing. I would follow R's semantics here and disallow the assignment to NaN and similar constants altogether (R returns the error "Error in NaN = 7 : invalid (do_set) left-hand side to assignment"). You can realize this restriction either by extending the ID specification in our ANTLR grammar (e.g., dml.g4) or by extending the validation of assignment and multi-assignment statements. The latter is perhaps the simplest and thus best approach. Once this is done, you could easily replace NaN, PI, INF with const identifiers during syntactic parsing without the need for variable handling.

Thanks for the review @mboehm7
I would close this PR and create a new splitted PR for builtin constants.


Configure: avoid novel '//' operator.

Configure starts with 'require 5.10.0', but if executed by older perl
it fails with "might be runaway multi-line // string" instead of
naturally expected "Perl v5.10.0 required--this is only v5.x.y".

Hmmm, chicken or egg, which comes first?  require checks the version number at run-time, perhaps another answer is to use use instead, since it checks at compile-time.
It would be quite horrible if we have to avoid perfectly legitimate 5.10.0 syntax just because we aren't doing the version check right!

Like that then :-)

We should probably review the version checks we do elsewhere...


INT_R.IMUX43.LOGIC_OUTS23 solution is unstable

https://github.com/SymbiFlow/prjxray-db/commit/b2c6e1c6c1693204fbaa1ff974670d4401f66579#diff-bba4931f2676a2911f064194d8cfc0e3L1919
-INT_R.IMUX43.LOGIC_OUTS23 21_30 !22_30 !23_30 24_30 !25_30
+INT_R.IMUX43.LOGIC_OUTS23 20_00 21_30 !22_30 !23_30 24_30 !25_30

Just did a fresh build and it appears to still be happening?
@@ -1916,7 +1916,7 @@ INT_R.IMUX43.FAN_BOUNCE3 20_30 !22_30 23_30 24_30 25_30
 INT_R.IMUX43.FAN_BOUNCE5 20_30 22_30 !23_30 24_30 25_30
 INT_R.IMUX43.LOGIC_OUTS1 21_30 22_30 !23_30 24_30 25_30
 INT_R.IMUX43.LOGIC_OUTS13 21_30 !22_30 23_30 24_30 25_30
-INT_R.IMUX43.LOGIC_OUTS23 20_00 21_30 !22_30 !23_30 24_30 !25_30
+INT_R.IMUX43.LOGIC_OUTS23 21_30 !22_30 !23_30 24_30 !25_30
 INT_R.IMUX43.EE2END1 19_31 !22_30 !23_30 !24_30 25_30
 INT_R.IMUX43.EL1END2 19_31 !22_30 23_30 24_30 25_30
 INT_R.IMUX43.ER1END1 18_31 22_30 !23_30 24_30 25_30

commit 2830c4ab88edd4984acea9ac1d53f3f9be89dc8e
Merge: 7c7f446 98cf09b
Author: litghost<EMAIL_ADDRESS>Date:   Thu Feb 7 15:43:08 2019 -0800

    Merge pull request #624 from mithro/kmaster

    infra: Improve the diff output.

I saw that we now have INT_R.IMUX43.LOGIC_OUTS23 21_30 !22_30 !23_30 24_30 `!25_30 in our database. I tried running the fuzzer several times and didn't see the bit 20_00 show up for this feature, so I wonder whether or when we can consider this solution stable.

Didn't see the problem in any of the runs I did lately. I guess we could close this issue?


Min/max randomized count

Describe the feature you have in mind.
It'd be nice if you had the option to set a min/max for the TradeIngredient count and the code would pick a number within the given range.
Does this solve a particular problem?
It brings the mod's functionality closer to vanilla trades, as some trades do have quantity ranges.

You should be able to do that using the minecraft:set_count function, you can find documentation about it on the wiki

Awesome, thanks for the info. :)


⚠ GitLab Server has degraded performance

In 6c72992, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11243 ms

Resolved: GitLab Server performance has improved in 6f2dfed.


Shuffle State for Websocket API broken when selecting a specific song from playlist

OS:
Windows 10
Issue Descriptions:
ALL_SHUFFLE payload and playback.toggleShuffle() not working as intended.
Shuffle state does not update if you select in individual song in a playlist or album or hit the play button in the top right on a playlist/albums landing page. If you select to play the whole playlist/album from the icon in on the bottom right before you select it or select a song from the song tab it works as intended.
Steps to Reproduce:

Open websocket
Open playlist
Select any song and play it
Websocket is now receiving bad shuffle data and any shuffle commands do not work as intended

On further inspection it sends the ALL_SHUFFLE payload with a value of NO_SHUFFLE when this happens and then never updates again. If you use the playback.toggleShuffle() command  it does not toggle shuffle on and off but instead does the same as if you pressed the shuffle button at the top of the playlist/album which shuffles all songs in the current playlist and picks a new song to play.
My guess is that google changed an id around in the webpage and the automatic error detection figured it was still okay.
I should note that the shuffle button in GPMDP is still correct it is just not reported correctly in the Websocket API.

I looked into this and I don't know why you still have this problem.  What version of GPMDP are you on?

Latest, let me try reinstalling

Yeah it is still doing it.

@tjhrulz Theme color isn't documented for a reason (let's call it a private API).  The volume not being documented is just a bit of missing docs.  (PR's welcome) 😆


Add License to the repository

I am thinking to add MIT License
Are there any suggestions regarding?

MIT license is suitable, go for it.

Okay sure
I'll also update the readme file with the license info as well

Sure

https://github.com/alexa-dev-hub/alexa-community-jaipur/pull/10
created a PR

As the PR is merged I am closing this issue


New federation support and Cognito user pools

Have you considered making Cognito user pools into a federated service like Google and Facebook? Cognito supports AWS generated UIs now. If you do that there is no need for the user dialogs in the react app any more.
Please don't use the Cognito self-hosted UI that also supports Google/Facebook. That UI makes the federated users in the User Pool, not the Federation Pool.  Doing it that way causes a Google login from the User Pool UI and a mobile login via the Federation Pool to end up with two different Cognito IDs.
If you use the Cognito User Pool UI just for Cognito users and leave the current amplify Google/Facebook login code alone, everything is fine for mobile integration (ie app generated via Mobile Hub).
Here's a demo login page I made using the AWS User Pools hosted UI:
https://digispeaker.auth.us-east-1.amazoncognito.com/login?client_id=71hfsop6tn8sr85kqg088emdkr&redirect_uri=https%3A%2F%2Flocalhost%3A3000%2Fredirect.html&response_type=token&scope=openid profile email&state=cjanc9fdf0000325kmnz959zq

ok but can aws-amplify enable embedded UI for cognito FB/google ?
I don't want the hosted UI because it's 1 more click for the user and an exit from our domain.

Thank you for the feedback @jonsmirl we recently pushed out support for federation via custom ui with Amazon Cognito Federated Identities; this is currently only supported in React and Google/Facebook, further support is coming. The docs are here.

@mlabieniec docs link is broken. Where can I find out how to use this?

@brianfoody all docs are now on github.io site:
https://aws.github.io/aws-amplify/media/authentication_guide.html#6-federated-identity


Performance issues

I've had issues with atom performance working with elm for a while. Atom is just really slow, especially when switching tabs for instance. I tried to capture a performance log when switching tabs, and it seems like it's got to do with elmjutsu.

I've attached the performance if that helps:
Profile-20180406T151438.json.zip
Not sure what i can do to debug this really.

Hi @smucode, can you give me your Elmjutsu settings?

I have disabled everything but hyperclick.

Have you tried it with just Elmjutsu installed?  (You can rename ~/.atom, then restart Atom.)
Also, do you have lots of .elm files in the project or huge .elm files?

I still experience the problem with only elmjutsu, language-elm and atom-ide-ui. When i cmd+click a around, atom freezes and the profiles seems to indicate it's the indexer.

With dependencies we have about 3.6k elm files.

How many files do you have in your project (without dependencies)?  Was switching tabs still slow with just those packages installed?

About 200 files without dependencies. I haven't experienced slow tab switching, no. I'll update if I experience it again.

Thanks!
As for Hyperclick/Go To Definition, there are cases where it will be slow (related to record fields, IIRC).  I'll look at it again when I have time :D

Yeah, i'm having all the same issues. Might be related to some other elm atom plugin i'm using (elm-format, elm-lens, linter-elm-make). Seems to be fairly OK when I restart atom, but after some use I notice ~3-5 second delays switching tabs, navigating the cursor around using the keyboard, selecting lines using the keyboard, and just general usage. The performance profiler always seems to be hinting towards the elmjutsu indexer. When I disable elmjutsu the lag goes away.

If it's not much of a hassle, can you give a screen capture (like a gif or video)?  There might something that's not obvious.  You can also send me a direct message on Elm Slack (same username).  Thanks!

As discussed on Slack, this is related to the Hyperclick support.  The workaround is to uncheck Enable Hyperclick for now.  A fix will be available in the next release.

This should be fixed in the latest version (v.8.0.0).  Thanks to @smucode for the help in debugging!


Trees, ores, etc. don't generate

In all biomes I have looked at, there have been no trees, no ores, no tall grass/flowers. This even happens between y=0 and 255. This sounds to me like the biome decorator is not working properly (since it adds every single one of those things).
Edit: Added images.

Note: I am in a plains biome in the third image, but that greener area in the center of the image is Forest/ForestHills.

Known issue. Trees and ores are features, and features are generated in a different part of the pipeline. We're still working on it.

Thanks! I didn't know that it was being worked on, so I reported it.


Line endings conversion

This PR will close #34. We'll use only standard line ending format in future.

Best change ever! Hope it will make our git-blame even more prettier!

It will make contributing and core review easier, because otherwise these changes anyway would pollute our repository inside any random patches (check https://github.com/codingteam/naggum/pull/30/files#diff-0a20fc0b7ff77a3a7944ca5177f2e86fL1 as an example). So this change should be made as early as possible. Your irony is inappropriate here.

I'm always serious in PRs. Your remark is inappropriate here. Goat confirms. :goat:

Now I'm decoupled enough from the state of mind in which I've created this PR. The time has come, I'm starting to review it myself.


Fix unescaped File.separator on Windows environment (butterknife-gradle-plugin).

On Windows environment the following statement throws an exception:
packageForR.replaceAll('\\.', File.separator)

Caused by: java.lang.IllegalArgumentException: character to be escaped is missing
	at java_lang_String$replaceAll$8.call(Unknown Source)
	at butterknife.plugin.ButterKnifePlugin$_apply_closure1$_closure2$_closure3$_closure4.doCall(ButterKnifePlugin.groovy:30)
	at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:554)
	at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:535)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
	... 70 more

After the proper escape of the \\ file separator, the butterknife-gradle-plugin can run without any errors.

Thanks!


Fix resources link

The original link was a fasttrack resource link, it was then changed to a link that does not have the resources mentioned, so replacing it with a link to several template resources
closes https://github.com/MicrosoftDocs/OfficeDocs-SharePoint/issues/3051

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@JoanneHendrickson Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

@scanum can you help me get this merged? thanks

@tiburd tina can you help me with the merging?


CLOUDSTACK-6181: Add 'root disk size' field in instance wizard for KVM templates

This commit includes three changes:
(1) Revert commit 13bf1ec5da9891eca98ef5dab61bf74b6db9530b to show the 'root disk size' field
(2) Set the default value from '1' to empty. This also changes the default value of customized service offerings from '1' to empty.
(3) show 'root disk size' field only when users choose a KVM template, as cloudstack supports deployvm with rootdisksize on KVM only.

the jenkins failure is due to an old and unrelated licensing issue.

LGTM based on code and function- manual test (see screen print)

@wilderrodrigues @remibergsma You guys have any use for this too? (hint intended)

@ustcweizhou @DaanHoogland Will test this soon. Can you please add that the size that one needs to specify is in GB? Didn't look so I had to look that up ;-)

LGTM, the UI shows fine and the value is passed to the API as shown in screenshot. See comment above, please add "GB" after the field.

LGTM

@ustcweizhou please add the "GB" label after the string, then we can merge this. Thanks.

add (GB) to the label. Please test it.

cc @remibergsma requires merging on 4.6 and consequent fwd merges


name-claim-fast can be used to snipe

It seems the function name-claim-fast can be used for sniping. Once a name-register transaction appears, an actor can initiate a name-claim-fast transaction upon seeing the non-hashed fqn.

Hey! @dartman10, name-register can not be snipped.
We do a check on name-register to see if the preorder happened before the name-claim-fast, that way even though the name-claim-fast happens and mints the name, as soon as the name-register happens it will transfer the name to the correct owner who preorder it first


updating folder location and a little typo.

Thank you!

Would it make sense to keep the path without the version?  @helto4real

Would it make sense to keep the path without the version? @helto4real

Yea I will change this next major version. The reason I had the version number so you can potentially run two versions at the same time without having to move all code to new version at once. But I guess this can still be configured by users if they want this functionality.


Typos in docs/proposals/container-image-vulnerability-adaptor.md

Describe the bug
Typo is picutre
Expected behavior
Typos picutre should be changed into picture

I fixed the typo

https://github.com/kubescape/kubescape/pull/678#issue-1349466893

@RohitYandigeri where?

Kindly check the current content
And typo is there

Kindly check the current content
And typo is there

Can you please close your PR as a newbie I want to merge the PR

as a newbie I want to merge the PR

@RohitYandigeri I wish you best of luck :)


Migrate auth-proxy to CentOS

This PR migrates auth-proxy from Debian to CentOS.  Debian and CentOS configure Apache quite differently, so there's a bit more change there than you may expect.
There is still a bit of clean up to do here, but I do believe everything is working.  I can say that the auth-proxy test suite all passes!
I am going to leave the WIP flag up for now while I do a bit more testing, but I think this is pretty close.

Taking off the WIP.  Open for review.


Draft: Modular Health Reporting

Draft - for discussion purposes only (for now)

@derailed Thanks for the feedback, I think I covered (most) of these in my latest PR.


[Bug] Update to remove healthCheck section in TargetGroupPolicy cannot trigger controller to disable Target Group healthCheck

For example, If I have a full connectivity setup gateway, route, service, deployment as well as TargetGroupPolicy.
I applied the following TargetGroupPolicy first with healthcheck:
apiVersion: application-networking.k8s.aws/v1alpha1
kind: TargetGroupPolicy
metadata:
  name: test-policy
spec:
  targetRef:
    group: “”
    kind: Service
    name: my-http-server
  protocol: HTTP
  healthCheck:
    enabled: true
    intervalSeconds: 5
    timeoutSeconds: 1
    healthyThresholdCount: 3
    unhealthyThresholdCount: 2
    path: “/”
    port: 9000
    protocol: HTTP
     protocolVersion: HTTP2
     statusMatch: “200”

Then I hope to delete the health check config by remove the healthCheck section:
apiVersion: application-networking.k8s.aws/v1alpha1
kind: TargetGroupPolicy
metadata:
  name: test-policy
spec:
  targetRef:
    group: “”
    kind: Service
    name: my-http-server
  protocol: HTTP

But controller actually didn't invoke lattice api to update to disable the health check
( Explicitly set healthCheck: enabled: true could work)

Thank you for pointing this out. This needs a fix for sure..
One thing to note is disabling health check may not be the correct behaviour here, because health check is enabled by default for non-GRPC target groups. The right behaviour would be setting health check config to Lattice default instead.
Current workaround: as mentioned in the description, explicitly set health check instead of removing the policy.


[SPARK-2827][GraphX] Add collectDegreeDist to compute the distribute of vertex degrees in GraphOps

This is from #4399.

@andrewor14 Done. We'll wait for tests done.

Test build #48448 has started for PR 10521 at commit 99776c9.

Test build #48448 has finished for PR 10521 at commit 99776c9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48448/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #48451 has started for PR 10521 at commit 99776c9.

Test build #48451 has finished for PR 10521 at commit 99776c9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48451/
Test FAILed.

retest this please

@maropu you might need to merge the latest master. My PR (https://github.com/apache/spark/pull/10509) non-deterministically broke the build, and has since been reverted. Sorry about that.

@hvanhovell Ah, okay. I'll rebase this pr.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48479/
Test FAILed.

Test build #48480 has started for PR 10521 at commit 4a3d588.

Test build #48480 has finished for PR 10521 at commit 4a3d588.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48480/
Test PASSed.

retest this please

Test build #48649 has started for PR 10521 at commit 4a3d588.

Test build #48649 has finished for PR 10521 at commit 4a3d588.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48649/
Test PASSed.

@andrewor14 ISTM it's ready to merge.

@andrewor14 ping

@andrewor14 ping

@andrewor14 ping

@maropu I actually do not maintain GraphX code. You need to ping @ankurdave instead.

@andrewor14 okay cc: @ankurdave

Test build #50534 has started for PR 10521 at commit c7435c1.

Test build #50534 has finished for PR 10521 at commit c7435c1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50534/
Test FAILed.

Test build #50542 has started for PR 10521 at commit ec0d62b.

Test build #50542 has finished for PR 10521 at commit ec0d62b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50542/
Test PASSed.

@ankurdave ping

Test build #56882 has started for PR 10521 at commit 2bbf57e.

Test build #56882 has finished for PR 10521 at commit 2bbf57e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56882/
Test PASSed.

Merged build finished. Test PASSed.

I think the GraphX features seems freeze, so I'll close this.


Add FreeBSD uptime detection with libc

This is for the most part the implementation from sysinfo that uses the libc crate to make the CTL_KERN and KERN_BOOTTIME syscalls. Taking this code snippet should not be an issue as sysinfo is licensed under the same license as this project.

I saw this: https://github.com/Macchina-CLI/libmacchina/blob/main/src/freebsd/mod.rs#L8
Could it be used?

@CarterLi yup, thank you for pointing that out, the sysctl crate makes the implemetation much cleaner and avoids taking code from sysinfo.

@Gobidev Do you have a physical FreeBSD machine to test with? Can you verify if the battery readout works for you?

@CarterLi I don't, I have been testing using a headless VM so far. I also don't have a spare laptop to set one up

I too used a virtual machine when porting to FreeBSD.
Have you had any problems with the battery readout?

I just doubt if anyone really had tested it.

Yep, just me. I can't really remember the details since it was a long time ago.

I just doubt if anyone really had tested it.

Yep, just me. I can't really remember the details since it was a long time ago.

I know it compiles. However, how can you make sure it will return the correct values? VM usually doesn't have a battery, at least for mine.

However, how can you make sure it will return the correct values? VM usually doesn't have a battery, at least for mine.

I'll reinstall FreeBSD and give it another go to verify its correctness.


WIP: Update README in preparation for 0.9.0 release

Which issue does this PR close?

N/A
Rationale for this change

Update README to reflect recent improvements.
What changes are included in this PR?

General update of README
Add roadmap (was previous in datafusion repo)

Are there any user-facing changes?

No

@mingmwang @avantgardnerio Let me know if there is anything you would like to add

I'll go ahead and merge this since there have been no objections in 10 days. Feel free to PR to make additional changes!


Add support for Path pattern to Dir.glob

With this change, Dir.glob can receive patterns as Path. I initially thought it might be best to only accept strings (hence this was not included in #9153). But it's actually useful to receive Path because glob patterns only support forward slashes as separators. If the method only accepts String but you have a Path, you would likely call Dir.glob(path.to_s). But that won't work for windows paths with backslash separators. By passing the Path directly to Dir.glob it can make sure to convert the path to a posix path, so it uses the expected separators.

Definitely a nice addition. I was going to propose something like this to resolve https://github.com/crystal-lang/shards/pull/401#discussion_r434793029


Found broken /docs/integration/langchain link. Please fix!

thank you, fixed in https://github.com/langfuse/langfuse-docs/commit/cca5a7319d0f39f9a87aea7ee99a4a41394ca517


ENH: Fix gpu cudf to_csv

What do these changes do?

delete unsupported params in cudf.datafram.to_csv
Related issue number

Fixes #795
Check code requirements

[ ] tests added / passed (if needed)
[x] Ensure all linting tests pass

Could you please add unit test in /home/lwz/xorbits/python/xorbits/_mars/dataframe/datasource/tests with @support_cuda decorator.


Resolve Warnings

we have too many warnings, this will prevent us from getting more important warnings.
lets either fix these warnings or config our editors to ignore these warnings.

Based on the most recent run of the project today, we didn’t encounter any warnings after executing the following command:
dotnet run --project src/FluentCMS


RBAC guide mentions obsolete authz bypass via insecure port

This is a Bug Report

Problem:
Using RBAC Authorization states:

If your API server runs with the insecure port enabled (--insecure-port), you can also make API calls via that port, which does not enforce authentication or authorization.

However, there is no such option in current Kubernetes. At least, I couldn't see it. As far as I can tell, the mechanisms that would work are:

to perform the bootstrap when authenticated as a member of system:masters.

Because this is an API server restriction, you can't configure an extra authorizer during initial bootstrap (such as a webhook authorizer). However, once any principal is bound to a ClusterRole that is allowed to escalate, it's all good and further bootstrapping can take place as that cluster role.
Proposed Solution:
Remove that item from the page.
Page to Update:
https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Additional Information:
/language en
/sig auth

/assign

/triage accepted

@dhanraj-druva I see that you haven't worked on this for long, so I stepped up and did this. Thank you for your interest 🙂
/unsassign @dhanraj-druva
/assign

/unassign @dhanraj-druva


ImportError: fused_layer_norm_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol

🐛 Bug
ImportError: /usr/local/lib/python3.8/dist-packages/fused_layer_norm_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE when fine-tuning bart_large model using translation task with the latest v0.11.0
To Reproduce
Steps to reproduce the behavior (always include the command you ran):

Run cmd using a custom dataset:

python -m torch.distributed.launch --nproc_per_node=4 \
    --nnodes=1 --node_rank=0 \
    --master_addr=$ip \
    --master_port=12345 \
    $(which fairseq-train) $DATA \
    --max-tokens $MAX_TOKENS \
    --task translation \
    --source-lang iq --target-lang oq \
    --truncate-source \
    --layernorm-embedding \
    --share-all-embeddings \
    --share-decoder-input-output-embed \
    --reset-optimizer --reset-dataloader --reset-meters \
    --required-batch-size-multiple 1 \
    --arch bart_large \
    --criterion label_smoothed_cross_entropy \
    --label-smoothing 0.1 \
    --dropout 0.1 --attention-dropout 0.1 \
    --weight-decay 0.01 --optimizer adam --adam-betas "(0.9, 0.999)" --adam-eps 1e-08 \
    --clip-norm 0.1 \
    --lr-scheduler polynomial_decay --lr $LR --total-num-update $TOTAL_NUM_UPDATES --warmup-updates $WARMUP_UPDATES \
    --fp16 --update-freq $UPDATE_FREQ \
    --skip-invalid-size-inputs-valid-test \
    --find-unused-parameters;

See error

Traceback (most recent call last):
  File "/usr/local/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/usr/local/lib/python3.8/dist-packages/fairseq_cli/train.py", line 535, in cli_main
    distributed_utils.call_main(cfg, main)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/distributed/utils.py", line 354, in call_main
    distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/distributed/utils.py", line 328, in distributed_main
    main(cfg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/fairseq_cli/train.py", line 94, in main
    model = fsdp_wrap(task.build_model(cfg.model))
  File "/usr/local/lib/python3.8/dist-packages/fairseq/tasks/translation.py", line 369, in build_model
    model = super().build_model(cfg, from_checkpoint)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/tasks/fairseq_task.py", line 335, in build_model
    model = models.build_model(cfg, self, from_checkpoint)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/__init__.py", line 106, in build_model
    return model.build_model(cfg, task)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/transformer/transformer_legacy.py", line 133, in build_model
    return super().build_model(cfg, task)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/transformer/transformer_base.py", line 97, in build_model
    encoder = cls.build_encoder(cfg, src_dict, encoder_embed_tokens)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/transformer/transformer_legacy.py", line 143, in build_encoder
    return super().build_encoder(
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/transformer/transformer_base.py", line 119, in build_encoder
    return TransformerEncoderBase(cfg, src_dict, embed_tokens)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/models/transformer/transformer_encoder.py", line 79, in __init__
    self.layernorm_embedding = LayerNorm(embed_dim, export=cfg.export)
  File "/usr/local/lib/python3.8/dist-packages/fairseq/modules/layer_norm.py", line 32, in LayerNorm
    return FusedLayerNorm(normalized_shape, eps, elementwise_affine)
  File "/usr/local/lib/python3.8/dist-packages/apex/normalization/fused_layer_norm.py", line 166, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.8/dist-packages/fused_layer_norm_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Code sample

Expected behavior
This works fine on 0.10.2, but I would like to start using --ddp-backend fully_sharded mode thus switching to (unreleased) 0.11.0
Environment

fairseq Version (e.g., 1.0 or main): 0.11.0
PyTorch Version (e.g., 1.0): 1.10.0
OS (e.g., Linux): Linux x86_64
How you installed fairseq (pip, source): pip3 install of v0.11.0
Build command you used (if compiling from source): pip3 install --no-dependencies . (PyTorch installation details below)
Python version: 3.8.12
CUDA/cuDNN version: 11.3
GPU models and configuration: Tesla V100-SXM2-16GB
Any other relevant information: Using Apex fork: https://github.com/szhengac/apex using --global-option="--cpp_ext" --global-option="--cuda_ext"

Additional context
Build as part of a Docker image base on nvidia/cuda:11.3.0-devel-ubuntu20.04.
Compiling PyTorch from source:
DS_BUILD_OPS=1 TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0 7.5 8.0" pip3 install -v . --global-option="build_ext" --global-option="-j$(nproc)"
Not sure, but it might be related to https://discuss.pytorch.org/t/undefined-symbol-when-import-lltm-cpp-extension/32627

I believe it's because of ABI, same issue as in https://github.com/pytorch/pytorch/issues/13541.
It appears I had GLIBCXX_USE_CXX11_ABI/_GLIBCXX_USE_CXX11_ABI set to 0 for both PyTorch and Fairseq, but Apex had 1.
Rebuilding with GLIBCXX_USE_CXX11_ABI/_GLIBCXX_USE_CXX11_ABI set to 1 everywhere didn't help.
Trying with 0 everywhere.

Found the problem, one of Fairseq's dependencies in v0.11.0 branch, torchaudio was pulling the latest torch and uninstalling my locally built from source one.

@clumsy How did you end up fixing this?

@ggrantrowberry I had to use a specific version of PyTorch I compiled for my hardware from source. So I had to install Fairseq with --no-deps making sure I have all the dependencies installed beforehand.

hi, @clumsy  I have the same promblem. So you reinstall all of your environment, and when you was installing the fairseq, you use the --no-deps command?

pip uninstall apex works for me.

pip uninstall apex对我有用。
thanks! That also works for me!


fix(tabs): fix tab headers resize/scroll issue

Closes carbon-design-system/carbon-components-angular#
The tab headers component doesn't handle view resize event and reset scroll properly, which can cause display issues on view resize. This PR fixes that using ResizeObserver.
Changelog
New

{{new thing}}

Changed

{{change thing}}

Removed

{{removed thing}}

I have read the DCO document and I hereby sign the DCO.

recheck

Could you remove the service import (line 18) & injection in the constructor (line 231):
import { EventService } from "carbon-components-angular/utils";
protected eventService: EventService,

Done, thanks.


URL Update Not Reflected After Modification

Current Behaviour
the url change wasn't properly reflected.
Screenshot

Expected Behaviour
URL should be updated.
Steps To Reproduce

Login to https://portal.int.catena-x.net/connectorManagement
Click on Managed connectors
Click on details
change the url click on update.
close and again click on details

Hi @MaximilianHauer
i tested this issue it's working fine. Can you close this issue.
Attached Screenshots for reference


[Bug] TextField: Check layout priority between the textfield and its addons

https://docs.google.com/spreadsheets/d/1yt1YLzNsM_RJK9Ac1u2pcHA5i_kNvFpWvYYdXQhXaCA/edit#gid=139525665&range=3:3

Not a bug, it's how UIKit works, the user can define how it should act should they want to


Translate this page error

If you select the 'Members' tab, an error message is displayed asking if you want to translate the page.

Tried this in Firefox and Chrome but don't get this error??


Add method for creating unique logging directory names

One issue is that if two logging directories are created simultaneously, they may conflict with each other. To prevent this, we should add something like UUID to prevent directories from having the same name.

Resolved in #e55ba7b85f20faf413ffd26e6dcc391056c564c2


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target-new/pull/1 is merged into live branch.
Auto archive content to https://github.com/OPS-E2E-PPE/docs-archive-test-target-new.git

Docs Build status updates of commit d37d244:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

archvieSourceFolder/TOC.yml
:white_check_mark:Succeeded

archvieSourceFolder/content/someContent.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Cardano Node Installation process: add cabal update

Checklist

[x] I have read the How to Contribute.
[x] I have run yarn build after adding my changes without getting any errors.

Updating documentation or Bugfix
The documentation regarding the installation of the Cardano node could benefit from a minor update. Specifically, there is a crucial step missing: running "cabal update" before executing the "cabal configure --with-compiler=ghc-8.10.7" command. It's easy for users to miss the warning and this leading to confusion when the node build doesn't proceed as expected.

Already added to docs/get-started/installing-cardano-node.md in #1008 but for some reason got missed in this page (it might still have been in the development branch at the time).
@rdlrt @os11k I would normally do cabal update immediately before cabal build but maybe it's better the way it's presented here, e.g. if people are using more than one version of GHC (personally I don't, so haven't investigated). 🤔

I'd imagine configure will be (more often than not) a once-off...and should always occur prior to cabal update , I'd also go with executing it right before build.


⚠ Bofrost.at has degraded performance

In feaccf6, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2690 ms

Resolved: Bofrost.at performance has improved in b314013 after 8 minutes.


Bars Wont stay in place.

I copied Axarva's dotfiles completely to replicate her setup, and then configured i3-gaps because I don't like xmonad, though it worked properly on xmonad, when I tried I3, it broke an all the widgets lined up on top rather than being separated.

sounds like something to bring up with axarva then, no? There might be some issues with i3 window handling, but I'm afraid without any further details I can't really debug this -- if you can share a minimum reproducible example I'd be able to take a look

Would you like my config files for eww and i3 then or what would you consider to be minimal? I don't really think it's a problem with axarva's dot's since it worked properly in xmonad. I think it could be more of a problem with i3-gaps, as every eww widget I have used has done something wrong, either won't display right, or it sticks to my screen and won't remove even with eww close-all

I3 definitely has some specific issues -- I don't know If I can actually help all that much here specifically, i'll just need to get more deeply into the specifics of how i3 breaks the ewmh spec, I'm afraid

Bspwm seems to work well for most people, and is relatively easy.
I do think I've seen it at least mostly work on i3 too, tho, so something is definitely weird. I'll need to so some specific testing there....


Ion mobility optimisations for branch remove_mass_list

Optimised ims mass detection:
First run of mass detection:

Second run of mass detection:

Left for you: fix mzxml import due to immutable data points in SimpleScan

You can submit this PR against master now..


osd-11717 Onv command for osdctl

Adds "osdctl network verify-egress" subcommand which integrates with osd-network-verifier
To test locally,
Clone this branch.
Build osdctl binary by running make build.
Select a subnet and get AWS credentials for it.
run AWS_ACCESS_KEY_ID=XXX AWS_SECRET_ACCESS_KEY=XXXX AWS_SESSION_TOKEN=XXX . /dist//osdctl network verify-egress --subnet-id= --region=

This PR has intricate dependency on onv accommodation for profile flag. To resolve that, I've split this PR in 2 parts. HEre's one that creates ONV command. Once profile flag is added to ONV, there will be a part 2 with addition of profile flag in this code.
This PR should be reviewed on its own. The done criteria for this card is to have egress command run on osdctl.

/cc @clcollins


opencr tf publisher - no messages received on topic

Issue template

Hardware description: opener 1.0
RTOS: micro_ros_arduino
Installation type: micro_ros_setup
Version or commit hash: humble

Steps to reproduce the issue
upload example code micro-res_tf_publisher.ino to opencr board, connect it to ubuntu virtual machine with micro-res-agent running
Expected behavior
micro-ros-agent recognise board, sets up publisher and tf messages are published on it
Actual behavior
micro-ros-agent recognise board, sets up publisher, but no message are published. The error light on the board is not flashing.
Additional information
publisher, subscriber and service examples work as expected. Have tried the imu demo code from the opencr hardware examples (that just prints ascii to the serial port) and data is printed out, so the imu is working. It is the original board with the MPU9250 imu on it if that makes any difference

Which is the output of the micro-ROS Agent using the flag -v6 in the case the communication is failing?

here is example with -v6
[1708007808.595466] info     | TermiosAgentLinux.cpp | init                     | Serial port not found. | device: /dev/ttyACM0, error 2, waiting for connection... [1708007808.772748] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3 [1708007810.372695] info     | Root.cpp           | create_client            | create                 | client_key: 0x3BA8E7D4, session_id: 0x81 [1708007810.372820] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0x3BA8E7D4, address: 0 [1708007810.373251] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 19, data:  0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00 [1708007810.375171] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 56, data:  0000: 81 80 00 00 01 07 2E 00 00 0A 00 01 01 03 00 00 1F 00 00 00 00 01 01 20 17 00 00 00 6D 69 63 72 0020: 6F 5F 72 6F 73 5F 61 72 64 75 69 6E 6F 5F 6E 6F 64 65 00 00 00 00 00 00 [1708007810.390030] info     | ProxyClient.cpp    | create_participant       | participant created    | client_key: 0x3BA8E7D4, participant_id: 0x000(1) [1708007810.390345] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 14, data:  0000: 81 80 00 00 05 01 06 00 00 0A 00 01 00 00 [1708007810.390494] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 01 00 00 00 80 [1708007810.391891] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 01 00 00 00 80 [1708007810.391947] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 72, data:  0000: 81 80 01 00 01 07 3E 00 00 0B 00 02 02 03 00 00 30 00 00 00 06 00 00 00 72 74 2F 74 66 00 00 01 0020: 20 00 00 00 74 66 32 5F 6D 73 67 73 3A 3A 6D 73 67 3A 3A 64 64 73 5F 3A 3A 54 46 4D 65 73 73 61 0040: 67 65 5F 00 00 01 00 00 [1708007810.392126] info     | ProxyClient.cpp    | create_topic             | topic created          | client_key: 0x3BA8E7D4, topic_id: 0x000(2), participant_id: 0x000(1) [1708007810.392389] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 14, data:  0000: 81 80 01 00 05 01 06 00 00 0B 00 02 00 00 [1708007810.392615] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 02 00 00 00 80 [1708007810.393585] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 02 00 00 00 80 [1708007810.393635] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 24, data:  0000: 81 80 02 00 01 07 10 00 00 0C 00 03 03 03 00 00 02 00 00 00 00 00 00 01 [1708007810.393814] info     | ProxyClient.cpp    | create_publisher         | publisher created      | client_key: 0x3BA8E7D4, publisher_id: 0x000(3), participant_id: 0x000(1) [1708007810.394105] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 14, data:  0000: 81 80 02 00 05 01 06 00 00 0C 00 03 00 00 [1708007810.394245] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80 [1708007810.394479] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80 [1708007810.395559] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x3BA8E7D4, len: 36, data:  0000: 81 80 03 00 01 07 1C 00 00 0D 00 05 05 03 00 00 0E 00 00 00 00 02 01 00 03 00 01 20 0A 00 00 00 0020: 00 00 00 03 [1708007810.396066] info     | ProxyClient.cpp    | create_datawriter        | datawriter created     | client_key: 0x3BA8E7D4, datawriter_id: 0x000(5), publisher_id: 0x000(3) [1708007810.396276] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 14, data:  0000: 81 80 03 00 05 01 06 00 00 0D 00 05 00 00 [1708007810.396438] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80 [1708007810.695186] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80 [1708007810.895827] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80 [1708007811.095965] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80 [1708007811.296387] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80 [1708007811.496766] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80 [1708007811.697718] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x3BA8E7D4, len: 13, data:  0000: 81 00 00 00 0B 01 05 00 03 00 03 00 80

Does it works if you publish fake numbers instead of reading from the IMU?

I tried putting fake values but no observable difference – set them quaternion to 0,0,0,1
Could try doing the same but comment out all the imu code
From: Pablo Garrido @.>
Date: Thursday, 15 February 2024 at 14:41
To: micro-ROS/micro_ros_arduino @.>
Cc: Nick Weldin @.>, Author @.>
Subject: Re: [micro-ROS/micro_ros_arduino] opencr tf publisher - no messages received on topic (Issue #1678)
Does it works if you publish fake numbers instead of reading from the IMU?
—
Reply to this email directly, view it on GitHubhttps://github.com/micro-ROS/micro_ros_arduino/issues/1678#issuecomment-1946233804, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAUVCSZR52YGDPVKAQ6HIADYTYNCHAVCNFSM6AAAAABDKEE4KWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBWGIZTGOBQGQ.
You are receiving this because you authored the thread.Message ID: @.***>

Yes please, let's discard the IMU problems by commenting the code out

no_imu.txt
micro_ros_tf_publisher_no_imu.txt
Here ia
the Arduino code with imu commented out (changed the extension to .txt to enable upload
and the terminal produced from that

could you determine where is the board freezing?

so it executes line
// create executor   RCCHECK(rclc_executor_init(&executor, &support.context, 1, &allocator));
but does not return form the function call in the if statement after that
if(!micro_ros_utilities_create_message_memory(       ROSIDL_GET_MSG_TYPE_SUPPORT(tf2_msgs, msg, TFMessage),       &tf_message,       (micro_ros_utilities_memory_conf_t) {})     )   {     error_loop();   }
it doesn't get to the error call, or the line after the if statement

So what if you try to remove the call to micro_ros_utilities_create_message_memory and initializes the message memory manually as explained here: https://docs.vulcanexus.org/en/latest/rst/tutorials/micro/memory_management/memory_management.html#message-memory

The page you linked to seems to make sense, but I don't understand how this relates to what has been done in the example code - why is there are is a sequence of transforms - a tf message only contains 1 transform, and only the 1st element in the micro-ros code is assigned numerical values anyway, the other one seems to be just being used to provide the second frame, but from the ros tf transform stamped message definition I would expect to have a header, a string for child frame and a transform, but the code is referring to a message that has two transforms, which each have a header inside, which then appears to have some of the structure like the ros message.
I looked in the library to see if there were files that defined things like a .msg definition in ros, but couldn't find anything that simple to refer to, so a bit stuck now as to how this works in practice

Look for the .h file that defines the type you are using. And recursively checking each member you will find which ones you need to initialize, as explained in the tutorial I attached. Tomorrow I will check this in more detail.

Thanks for your help. That's what I was looking at - I hadn't looked closely enough at the definition in the tf_message file - I see where the transforms element comes from now. My confusion now is why the tf_message is being used rather than just transform_stamped - is it a micro-ros specific reason ? Creating a message with two transform_stamped objects seems it would increase the memory footprint, and not sure why its needed. All the tf2 tutorial examples for publishing a transform just use transform stamped.

So I'm seeing if a simple TransformStamped version will work - the board now returns successfully from the line
if (!micro_ros_utilities_create_message_memory(
ROSIDL_GET_MSG_TYPE_SUPPORT(geometry_msgs, msg, TransformStamped),
&transform_stamped,
(micro_ros_utilities_memory_conf_t){})) {
error_loop();
}
but doesn't get past the next line - tried each of these separately and they both failed
transform_stamped->header.frame_id =
micro_ros_string_utilities_set(transform_stamped->header.frame_id, "/panda_link0");
transform_stamped->child_frame_id =
micro_ros_string_utilities_set(transform_stamped->child_frame_id, "/inertial_unit");
so will look today at replacing those with manual assignment from the example you linked to

using
strcpy(transform_stamped->header.frame_id.data, "/inertial_unit");
transform_stamped->header.frame_id.size = strlen(transform_stamped->header.frame_id.data);
compiles, but board does not get past it

have you assigned memory to transform_stamped->header.frame_id.data with a malloc?

no - I thought the micro_ros_utilities_create_message_memory that was now succeeding was doing that
will try it now

added malloc and child_frame_id - the micro_ros_string_utilities_set function then runs on board as well as manually setting data and size.
code now seems to be running to the end of loop, topic is showing up as geometry_msgs/msg/TransformStamped, but still no messages arriving in ros2 topic echo
attached current code on board and terminal -v6 - the terminal has stopped printing out data at the bottom of the document, it is not continuously printing recv_message
terminal.txt
current.txt

Your micro-ROS client is still not sending data to the Agent

not sure what to do now
I moving away from the example as written - but it did not work for me - is this something that needs fixing, or have others got it to work as it is.
is using the more complex message with two transform objects necessary or does add additional for an example when trying to get things working and understanding what is going on in the code

so I found some example code here https://answers.ros.org/question/416189/cannot-send-transform-stamped-sequence-using-micro-ros-arduino-library-on-esp32/
that works on the board once its adjusted to not use wifi - I will work over the weekend to get it working with with the imu, time and publishing a TransformStamped in a way that is closer to the way the current example is structured. Would you be interested in that as a fix or alternative example ?

Please create a PR with a fix and we will evaluate if it worth to split it in different example in a further step

so went down a bit of a rabbit hole - seems the troubleshooting that I had done at the start was misleading, and so went down the wrong path, none of the changes to allocation of memory produced messages, so on a whim tried switching the message so it wasn't a pointer, replaced -> with . in assignments and passed it to the publisher with & - it worked.
I thought I had tried that right at the beginning but can't have done it right. Will sort out a pr for just that change, one for then refactoring the message type to just a transform_stamped with all the frames set, and add the code for syncing time so that the tf displays more reliably in rviz

Ok, so can we close?

closing pr to follow


Cant run tests

import projekt in IntelliJ
run "mvn clean install -Pproprietary-deps"
Error "Could not resolve dependencies for project org.testcontainers:testcontainers:jar:1.2.2-SNAPSHOT: Failure to find fakejar:fakejar:jar:0 in ..."

Hi @arnzel,

You don't have to use Maven to run the tests, you can just run them from IntelliJ as normal JUnit tests
Why do you use proprietary-deps profile?

Hi, thank you for your answer.

-> I read it in the documentation (Topic 'Contributing')

I just ran ./mvnw clean install -Pproprietary-deps in current master and it seems to work. Could you try again using the maven wrapper script?
Or as an alternative try running the tests (or Maven goals) directly from within IntelliJ.
Still we should update the docs in this point I think, since are building the project in Travis with mvn -B -P !build-extras test.

For command line usage, look in the .travis.yml file and follow the instructions in there, specially the commands shown by install: and script:.


chore(main): release 1.0.13-beta.1

:robot: I have created a release beep boop
1.0.13-beta.1 (2024-10-06)
Bug Fixes

experimenting with release (ec2393b)
experimenting with release v2 (403e618)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v1.0.13-beta.1
:sunflower:


Check key at startup not during runtime in ch-app

Feature Request
Which Areas Would Be Affected?
e.g., DPF, CI, build, transfer, etc.
CH-APP
Why Is the Feature Desired?
Are there any requirements?
CH-APP checks private key only during runtime first request not during startup. So it makes it difficult to check if the CH-APP started successfully.
Solution Proposal
If possible, provide a (brief!) solution proposal.
Check during startup

Implemented a check on startup, that opens the private key file and checks if the key is useable as a private key.


Add swift support

This PR adds support of OpenStack Swift for the image registry operator.

/test e2e-aws-operator

/test e2e-aws-operator

/test e2e-aws-operator

/test e2e-aws-upgrade

/test e2e-aws-upgrade

/lgtm


Add order parameter to toggleSort method to control ascending/descending

It would be useful to be able to define which direction you want to sort the column by.
toggleSort('columnName') // Sort flip flip
toggleSort('columnName', false) // Sort DESC
toggleSort('columnName', true) // Sort ASC

That's a good idea, I'd accept a PR for this but probably won't get to it soon.

Cool. Pull request submitted 👍

Merged in #22


Updates

@Xaenalt this is a raw version of the dockerfile I used to build ovms with nvidia plugin

Looks good, I force pushed a few things, can you rebase onto HEAD for that branch so I can see the changes easier


B 21533

Agility ticket
Summary
Is there anything you would like reviewers to give additional scrutiny?
this article explains more about the approach used.
Verification Steps for the Author
These are to be checked by the author.

[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] Have the Agility acceptance criteria been met for this change?

Verification Steps for Reviewers
These are to be checked by a reviewer.

[ ] Has the branch been pulled in and checked out?
[ ] Have the BL acceptance criteria been met for this change?
[ ] Was the CircleCI build successful?
[ ] Has the code been reviewed from a standards and best practices point of view?

Setup to Run the Code

Instructions for starting storybook
Instructions for starting the MilMove application
Instructions for running tests

How to test

Access the
Login as a

Frontend

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, Edge).
[ ] There are no new console errors in the browser devtools.
[ ] There are no new console errors in the test output.
[ ] If this PR adds a new component to Storybook, it ensures the component is fully responsive, OR if it is intentionally not, a wrapping div using the officeApp class or custom min-width styling is used to hide any states the would not be visible to the user.
[ ] This change meets the standards for Section 508 compliance.

Backend

[ ] Code follows the guidelines for Logging.
[ ] The requirements listed in Querying the Database Safely have been satisfied.

Database
Any new migrations/schema changes:

[ ] Follows our guidelines for Zero-Downtime Deploys.
[ ] Have been communicated to #g-database.
[ ] Secure migrations have been tested following the instructions in our docs.

Screenshots

Test coverage on client failed.
Last run: Tue 17 Dec 2024 09:14:07 PM UTC
Refer to this report to see coverage details for the client app.
Documentation for test coverage can be found at this link.

Warnings

:warning:

Files located in legacy directories (src/shared or src/scenes) have
been edited. Are you sure you don’t want to also relocate them to the new file structure?
View the frontend file org ADR for more information

:warning:
This PR does not include changes to unit tests, even though it affects app code.

  Generated by :no_entry_sign: dangerJS against e77cb580c05bf491a8a3e53e5f23c430d394b2c7

Test coverage on server failed.
Last run: Wed 18 Dec 2024 12:28:05 AM UTC
Refer to this report to see coverage details for the server app.
Documentation for test coverage can be found at this link.


undefined method 'get' for Parse::Installation:Class

I am using
installation = Parse::Installation.get "objectId"
as said in the tutorial. It worked but now I am getting this error.
I don't know what's going on with this.
Anyone have any idea?

Hi @origds, can you give more info about that? Which version of parse-ruby-client are you using? Did you manage to fix it?

I am using parse-ruby-client 0.3.0. I don't have more details about the error. I just follow the gem documentation, and it worked. But now I am getting the error: undefined method 'get' for Parse::Installation:Class

@origds parse-ruby-client 0.3.0 doesn't have that method. You need to use the github version.
Replace your entry in the Gemfile with
gem 'parse-ruby-client', git: 'https://github.com/adelevie/parse-ruby-client.git'

there will likely be some differences, so check the documentation if you're having further problems.

Thank you! :+1: @rhymes

I still have the problem, change the Gemfile and updated it, but I still have version 0.3.0

Can you post your Gemfile and Gemfile.lock? Or you can just remove your Gemfile.lock and then run bundle install
btw @origds can you please not open a issue for the same problem :) ?

I'm having this problem too, I have the master branch installed, I'm trying to assign a custom key named 'alias' and I'm getting the following error:
undefined method 'alias' for Parse::Installation:Class

@MarkMurphy this is not the same problem as the op though.
Only the default (see https://parse.com/docs/rest/guide/#push-notifications-uploading-installation-data) fields have attribute accessors, for the rest you have to you take advantage of the fact the Installation object is a Hash, so:
installation = parse.installation.tap do |i|
  i.device_token = device_token["device_token"]
  i.device_type = "ios"
  i.channels = device_token["tags"]
  i["alias"] = device_token["alias"]
end

gotcha, thanks


:bug: Fixed maintaining archived newsletter subscriptions for members

refs #16355
-archived newsletters will no longer be unsubscribed when saving member

@allouis Any idea why the snapshot needed changing here? I don't get why my changes would result in a change in the context.

@9larsons Looks like you've fixed a bug - we didn't always load the initialMember before, but now we do. That piece of code throws a NotFound error with a custom message, I'm guessing elsewhere in the flow a NotFound error is thrown with a generic message and that was what the snapshot used before!


Consignment tracking phase 1

This is a placeholder ticket for KungFu Panda's work on consignment tracking.
link to PR to come.

The code is here: https://github.com/SAP/cloud-commerce-spartacus-storefront/compare/feature/PANDA-3071

As you know, we are implementing the consignment tracking feature based on existing code of order details page. Regarding this working model, we have 4 concerns:

Unit test: our changes will introduce impacts on the existing unit test
Maintainability: as we introduced changes on the existing code, will it bring a bit more difficulties for troubleshooting in the future?
Threats modeling: I suppose you've already done a threats modeling for existing feature, then we introduced our changes, need we do another one?
Ngrx: As we aligned before, we needn't to consider the ngrx implementation for now. However, we are coding based on existing implementation, need we also complete the ngrx things?

Another proposal is you provide enough extensibility, e.g. outlet, so that we can define our component and introduce our or override your page fragment.
The CONS: more than one places are handling a same page, and it may make customer confused when they want to introduce their functionalities.

The related jira item is https://jira.hybris.com/browse/PANDA-3071

We have done a POC of this feature so close this issue now. I will create another issue for implementation.

Close this issue since POC is done. I will create another issue for release