Use wait_for_genesis_commit in benchmarks.

Currently, the benchmarks after #1784 use a fixed time to wait for genesis commit. Genesis commits in a reasonable time, however, usage of wait_for_genesis_commit in benchmarks almost always results in a timeout.

Example:


Add Plane animation to leaderboard scene

Code for animation moves plane to correct location. See issues below.
Issues:
-Score box is not visible with plane animation.
-Plane speeds up to final destination. Should slow down.

See #82


dbt run fails

Struggling to get this working. I think we have a pretty vanilla Intercom setup and we created our connector in Dec.
Here is the output from running dbt run locally. We use FiveTran connected to RedShift.
I've tried playing around with disabling some of the vars, but no luck.
Completed with 3 errors and 0 warnings:

Database Error in model stg_intercom__contact_history_tmp (models/tmp/stg_intercom__contact_history_tmp.sql)
  relation "stg_intercom__contact_history_tmp__dbt_backup" already exists
  compiled SQL at target/run/intercom_source/models/tmp/stg_intercom__contact_history_tmp.sql

Database Error in model stg_intercom__conversation_tag_history_tmp (models/tmp/stg_intercom__conversation_tag_history_tmp.sql)
  relation "intercom.conversation_tag_history" does not exist
  compiled SQL at target/run/intercom_source/models/tmp/stg_intercom__conversation_tag_history_tmp.sql

Database Error in model stg_intercom__company_tag_history_tmp (models/tmp/stg_intercom__company_tag_history_tmp.sql)
  relation "intercom.company_tag_history" does not exist
  compiled SQL at target/run/intercom_source/models/tmp/stg_intercom__company_tag_history_tmp.sql

Hi @dselman!
Thank you for opening this issue. Looking through your issue description I had a few quick questions:

Do you have conversation_tag_history and company_tag_history tables in your Intercom Schema?
Would you be able to share the variables you are declaring to hopefully help with any confusion on how to successfully deactivate those models if you do not use them.
The first error indicates you already have a table named stg_intercom__contact_history_tmp in your schema. Is that true?

conversation_tag_history and company_tag_history  are both missing from RedShift.

Thanks for sharing @dselman! Since you do not have the conversation or company tag tables, you can disable them from the dbt project. You can add the below variable config to your dbt_project.yml file which will disable the conversation_tag and company_tag models in both the Intercom source and transform package:
vars:
  intercom:
    using_company_tags: False
    using_conversation_tags: False
  intercom_source:
    using_company_tags: False
    using_conversation_tags: False

Let me know if you still encounter an error after applying this change.

That didn't appear to work:
vars Config:
vars:
  mailchimp:
    using_automations: false #disable if you do not have the automation_email, automation_email, or automation_recipient_activity tables
    mailchimp_campaign_recipients: false
  intercom:
    using_company_tags: False
    using_conversation_tags: False
  intercom_source:
    using_company_tags: False
    using_conversation_tags: False

Error:
Database Error in model stg_intercom__company_tag_history_tmp (models/tmp/stg_intercom__company_tag_history_tmp.sql)
  relation "intercom.company_tag_history" does not exist
  compiled SQL at target/run/intercom_source/models/tmp/stg_intercom__company_tag_history_tmp.sql

Database Error in model stg_intercom__conversation_tag_history_tmp (models/tmp/stg_intercom__conversation_tag_history_tmp.sql)
  relation "intercom.conversation_tag_history" does not exist
  compiled SQL at target/run/intercom_source/models/tmp/stg_intercom__conversation_tag_history_tmp.sql

@dselman my apologies another customer encountered this same issue and we have a pull request open that fixes the disabling of these tmp tables in the source package if they are not used.
We will try to get this pull request approved and merged soon so you can successfully run this package and use your modeled Intercom data. I will let you know once the PR is merged!

@dselman our Intercom v0.1.2 release is now live! This release fixes the table disable variables bug you were encountering as well as some additional updates as well.
You can now update your package dependency to install the most recent Intercom release.
packages:
  - package: fivetran/intercom
    version: 0.1.2

Let me know if you have any questions!

That fixed it. Thank you very much! 🚀

@dselman Can you set up Fivetran to run a dbt run --full-refresh and see if that works?

Hi @dselman did Kristin's recommendation fix your issue? If not, let us know and we can help further. Otherwise we can close this issue.
Thanks!

Hi @dselman I assume the issue is no longer relevant and has been resolved. As such I will close this issue. If that is not the case, and the issue still persists, please open a new issue and we may work further from there.
Thank you!


Scripts & processes: importing and exporting csv's

This PR adds REST api support for the following 2 CLI scripts:

metadata-export
metadata-import

Because these imports use files (both as input and output), some REST changes were needed, these REST changes can be found here: https://github.com/DSpace/Rest7Contract/pull/99

I'm curious to hear why we want to use Bitstream for temporary files.
We might want to add a metadata field to these Bitstreams:  dspace.scope or the like.

@tdonohue @mwoodiupui
Given your feedback above I would suggest we do the following:

Bitstreams that are attached to a process will not be stored in the assetstore, but rather in a separate "temp assetstore".
The path for this temp assetstore will be configurable in the dspace.cfg / local.cfg folders
With configuration you will be able to change the storeService for this temp assetstore (so you could have your temp assetstore on the local drive, but your actual assetstore on S3, or both on S3, or both local, ...)
Whenever we delete a file which is attached to a process the file will be removed from the drive (as opposed to flagged as deleted)

More general information about bitstreams decision
Because we are now working with a rest service in which a file is uploaded & needs to be processed in a thread we need to store a file on the server for processing purposes. Once the process ends we also want to make files available for download (which can only be done by storing them on the server & offering up a rest endpoint to download them).
Instead of creating a separate system to read/write files to the filesystem (not even mentioning the rest which would need a separate endpoint) we decided it would be best to reuse existing code that does just this. An additional benefit of having the input files a bitstream if the process fails, the user can retrieve the actual file that was actually used as input. Another benefit was that if we attach these files as bitstreams (instead of temp files) is that we can link to the process endpoint. If a server reboot is triggered for whatever reason the files can be downloaded after & are only removed once the process itself is removed.
We might also consider an automatic cronjob (or something similar) that will automatically clear out processes after X of time has passed, but this would be something for a follow up PR.
Regarding other points of feedback

CSV's that are attached a process won't be indexed in solr because the discovery script doesn't loop over files it wants to index, it loops over items & indexes the files that are attached to those items. As these bitstreams are attached to another object it won't be indexed.
The only consumer that is currently triggered by creating a bitstream is the RDFConsumer which checks for bundles before it does anything: https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/rdf/RDFConsumer.java#L119 as these bitstreams have process as a parent this won't do anything.
Changing the "process" schema to "dspace" isn't an issue.

@KevinVdV : thanks for the updated description.  I'm OK with storing these temporary files as Bitstreams provided they can be...

Easily distinguishable/separated from normal Bitstreams.  Storing them in a separate assetstore is useful for this. However, I'd also like us to consider whether we need to subclass Bitstream to create something that is a TemporaryBitstream (or similar).  I'm slightly considered about the fragility of using a metadata field (dspace.process.type or similar) as the distinguishing factor.  What would happen if a user accidentally reused this dspace.process.type field for a real Bitstream?  I'm not sure whether we should consider a new table column to flag whether a Bitstream is "temporary"?
Cleaned up / deleted separately from normal Bitstreams.  This relates back to distinguishing them as different & having them in a separate assetstore.  I like your suggestion to delete them immediately after a process completes.  However, as you note, I think we'll need a new ./dspace cleanup flag to cleanup temporary bitstreams only (in situations where they were not successfully deleted previously, or the process errored out).  So, for example, sites may choose to run ./dspace cleanup --temporary (or maybe ./dspace cleanup --processes) more frequently to remove temp files or failed processes.  But, they may wish to run ./dspace cleanup (to remove deleted bitstreams) on a less frequent basis, depending on local policies about bitstream deletion.

Finally, we'll need to be very careful with how best to implement processes that require many files to succeed.  For example, an AIP restoration process may end up needing hundreds or thousands of AIP Zip files.  Obviously, attempting to store all those AIPs at once (even temporarily) could be like storing your entire repository a second time (if you are performing a full site restore).  So, we may want to ensure a single process can "release" (i.e. delete) Bitstreams it is finished with immediately (even if the process is not yet completed). This would essentially mean ensuring a single process can only use one file at a time even in large AIP restores.  So, processing a giant AIP restoration would take up a lot less local space as each AIP would be processed one by one.

Instead of creating a separate system to read/write files to the filesystem

I must be missing something.  The JRE already provides java.io.*.

What would happen if a user accidentally reused this dspace.process.type field for a real Bitstream?

We would all point to the documentation which clearly states that the dspace schema is internal to DSpace and you use it at your peril.  (Probably the REST layer should not expose fields from internal namespaces.)

@tdonohue @mwoodiupui
To be clear bitstreams are attached to the process using a separate table (https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V7.0_2019_06_14__scripts-and-process.sql#L30) so we can clearly distinguish these bitstreams, not just using metadata
The actual delete of the files is managed by the ProcessService delete method: https://github.com/DSpace/DSpace/pull/2648/files#diff-cc6245f1fe98265fee952abb3eed15aaR149 it isn't hard to alter the logic here to remove the actual bitsteam row & file from the HDD instead of flagging a bitstream record as deleted.
As we need to have the files stored on the DSpace server regardless (for import / export) it makes more sense to use bitstreams as:

temp files wouldn't be linked to the actual process database record & would be needlessly fragile
if we do a csv export the file needs to be on the drive until the user is done with the file, how would the system know ?

Concerning the actual deletion of the files a few things come to mind:

If angular wanted it could trigger a process delete right after download (which in turn will clear the bitstream)
We create an internal DSpace cron that will delete processes after X amount of time after completion

With bitstreams linked to the process DSpace remains in control of the files & can always keep track of them.
A bit more on AIP import/exports:

This currently isn't possible using the UI in xmlui, only SAF imports & exports are available (SAF exports are also kept on the server right now as they are not immediatly triggered)
As the AIP import/export is a CLI thing the process table & bitstreams aren't used for the CLI
If we were to support for this these files would have to be stored to the HDD on the server regardless, so better to have them as bitstreams where we have the control to clear them out.
Regarding the point above, AIP doesn't delete files it already processed right now anyway

"...create an internal DSpace cron...."  Please, no.  Why not use the task scheduler which is offered by every modern operating system?  DSpace only need provide a tool which knows whether one of these objects is still in use, or can be presumed unused due to age, or whatever.  The sysadmin can tell the OS when to run it.

Permit me to summarize and then explain my uneasiness:  this feels like we are hijacking Bitstream for some completely different and ill-fitting use.
Everybody already knows what a Bitstream is:  a representation of a document which is a permanent member of the DSpace content model.  Now we are to have two completely different kinds of Bitstream.  The new kind represents an ephemeral file, which is not in Bundles, not contained by an Item, and not findable by navigating the content model hierarchy.  They have a  different lifecycle and a duration ranging from seconds to hours, not decades.  They do not need extensible metadata and thus do not need to be DSOs.  They are contained by process executions, not Items.  They do not need long-term fixity checks.  They are working files, not content objects -- they are produced, consumed, and discarded in short order.  Many of the characteristics of Bitstream are useless or even annoying with regard to these files.
I propose that the inputs and outputs of Scripts&Processes are a distinct data type.  They should be represented by a distinct class in memory.  They should be persisted to a separate table in the database (and such a table is already defined by this PR, needing only minor adjustment).  Their content should be stored as files in a place separate from model assets.  We do not need the complicated assetstore path scattering -- we will not be holding hundreds of thousands of these for years.  This will be no more fragile than Bitstream since the association is backed by the database.
In short, these working files are almost, but not quite, entirely unlike Bitstreams, and I feel that to conflate the two invites confusion and unnecessary complexity.

@mwoodiupui it may help you to compare this with collection logos. They're also not a representation of a document, don't need metadata, ….
Bitstreams are not designed to be limited to be item resources, it's just their most common purpose.
The only new thing about this use of bitstreams is the expected duration to keep them available. Kevin did propose a solution to ensure they can be stored in a different assetstore if desired, and to be able to perform cleanup only on these bitstreams if desired.

@benbosman : I disagree slightly with the comparison to Collection/Community logos.  While you are correct those are a different "type" of Bitstream from ones attached to an Item, they still are something that is semi-permanent in nature -- so running the checksum checker on them, and other similar BItstream-level activities is reasonable.
When we start to talk about Bitstreams that are temporary, that is a new thing.  These temporary Bitstreams need to behave differently in that they must be ignored by other Bitstream-specific system processes, like the checksum checker, curation tasks, etc.  We also may or may not want them returned from the /api/core/bitstreams REST endpoint. They might need a separate endpoint so as to not imply these are archived or managed similar to normal Bitstreams... in addition, their endpoint might require a read-only reference back to the script they are associated with.
So, to clarify, I don't have concerns about storing these temporary files similar to Bitstreams (in the database and perhaps in the assetstore or some other similar area, though as Mark points out they need not be in the assetstore necessarily).  However, they need to be represented differently so that existing Bitstream processes will automatically ignore them without having to clutter existing code with things like if not a temporary Bitstream, do something

@tdonohue It seems the checksum checker is indeed an exception which doesn't care whether the bitstreams are linked to an item (and thus also handles collection logos). Other scripts (curation tasks, filter-media, … are item based and only retrieve the bitstreams from the items)
It would indeed make sense to ensure the checksum checker will not process these bitstreams, we'll propose a solution for that

@benbosman : Just a minor clarification, there are other areas of the code that simply look for any Bitstream -- so, in your solution, I would suggest not assuming the checksum checker is the only thing to "fix".  For example, Event Consumers can also be configured to do something specific to Bitstreams...here's one example: https://github.com/DSpace/DSpace/blob/master/dspace/config/dspace.cfg#L687
There may be other areas of the code like this too...I haven't done a thorough check myself to be honest.
Essentially, the message here is that reusing the Bitstream object for temporary files may have unintended consequences for any code that processes Bitstreams directly (including if a site or service provider created a custom plugin that works directly against Bitstreams without going through Items/Bundles first).

All,
I talked with Lieven & @benbosman about how to move this PR forward, and between the three of us we've come to general agreement on direction (feedback is welcome).

First, keep in mind that this PR is built based on the existing (already merged) backend PR: https://github.com/DSpace/DSpace/pull/2529 which established a link between processes and bitstreams via a processes2bitstreams table.
Bitstreams are quite literally just files + metadata, which require contextual information to understand them. Some Bitstreams may be archived (often permanent or semi-permanent), while others are unarchived (potentially temporary or at least less permanent).  However, it's the context of the Bitstream that describes the type of Bitstream. Quite literally, we already have a number of types of Bitstreams:

Bitstreams in the ORIGINAL bundle : These are archived and usually permanent (though they may occasionally be deleted).
Bitstreams in the TEXT bundle: These are technically "archived" but usually semi-permanent. May be recreated by filter-media.
Bitstreams in the THUMBNAIL bundle.  Again, "archived" but usually semi-permanent.  May be recreated by filter-media.
Bitstreams which are logos (attached to a Community or Collection).  Technically "unarchived" (not attached to an archived item). Usually permanent or semi-permanent, but can be replaced whenever needed.
Bitstreams which are the output of Processes (new concept).  Unarchived, and usually either temporary or semi-permanent.  While currently export processes are more temporary in nature (and associated with a single user account), it's plausible that a user might (eventually, post 7.0) want to share an export report with someone else.

Because Bitstream require context to understand their usage and permanency (see above), at the REST API level, the list of all Bitstreams should never be accessible directly.  So, the main endpoint (GET /api/core/bitstreams) should be updated to return a 405 error (like the /bundles endpoint and others).  In other words, Bitstreams will only be findable through the associated Bundle, Community, Collection and Process endpoints.
All of the above means that individual scripts (whether internal server-side scripts or external client-side calls) should NEVER simply query for all Bitstreams and perform actions across them all. Without the context around the Bitstream, it's impossible to understand if it's permanent, semi-permanent or temporary.

By removing this capability from the REST API, this (soon) won't be possible for client-side tools.
Atmire reviewed all server-side tools and found that only the CheckSum Checker accesses Bitstreams directly (without going through an associated object). All other tools minimally go through Bundle (in order to filter out logo Bitstreams).

In summary, this PR can move forward with the direction of using the /api/core/bitstreams endpoint for Bitstreams related to Processes.  However, it needs to update that endpoint to return a 405 when listing all Bitstreams (as that can no longer be allowed).  Additionally, the Checksum Checker should be updated (possibly in a follow-up PR) to only validate the checksum of Bitstreams attached to Bundles, Communities or Collections.

I'm fine with reusing the bitstream object to store such temporary information trusting the information that they can be stored in a separate assetstore and that it is still possible to retain in the assetstore deleted "real" bitstream longer if needed.
To clarify, my change request about this PR https://github.com/DSpace/DSpace/pull/2648#pullrequestreview-362247628 is only related to the way that process and script are managed in Spring. We need to assure that, as this facilities will be accessed via REST, don't use a single thread model. It should be not too difficult to fix that with a proper refactoring.
Just a remark for all of us, I hope that from this discussion we will learn one more time that no new features come without extra, unexpected cost (in discussion, review, coding, later bug fixing, etc) so we should really stick to the rule to implement things in DSpace 7 as they were in previous version and delay changes as much as possible for later but now we are in the middle of the sea so we need to go to the destination.

@mwoodiupui : Apologies for asking you directly on GitHub (but it looks like you are out today). I wanted to be sure you are reasonably OK with the direction proposed in my comment above, as you expressed previous concerns.  In today's meeting, we briefly discussed this and there were no other objections raised at that time.
I will admit, I initially agreed with your analysis that it seems a tad odd to use Bitstream to represent these temporary / non-permanent export files.  But, after further thought and discussion, I've realized these export files are not much different than Bitstreams in the TEXT Bundle, which are also temporary export files created for the purpose of full text indexing.  I also have come around to the point that, depending on the type of export, sometimes these process bitstreams might not be entirely temporary.  For example, a user might kick off a larger CSV metadata export that they later want to download to a different computer. Or maybe they want to share an export with a different user.  These exports still may not be permanent, but they may need to live longer than a single request.
While you are correct that Bitstreams need not be used for Process exports, I think it simplifies the codebase at this time. As you noted, the alternative would be to create a new type of object/class, with a new database structure and new REST API endpoint, and that object type would still need ResourcePolicies attached (to access restrict it to the user who exported the content).  That said, in a future release (v8 or later) we could very well decide to refactor temporary-ish export files (perhaps also include the TEXT bundle) into a different object/class.  But, before that, I'd like a much deeper discussion about "types of bitstreams" to better draw the line between various file objects/classes, as I'm not yet certain it's even just two types (e.g. logos & thumbnails might be a third type that are stored simply for the UI & also just semi-permanent).  (All that said, maybe another resolution is to create a JIRA ticket to remind us to re-discuss the "types of Bitstreams" post-7.0?)

@tdonohue Sorry, I was travelling that week and the email thread scrolled out of view before I got back, so I'm just now seeing your question.
I won't continue to resist the current plan.  We might revisit the design later.  I classify bitstreams a bit differently, though, and I offer my view for later consideration.  Up to now, all Bitstreams have represented files which are meant to be retained indefinitely.  They are objects which are effictively permanent, even though some may be regenerated at need.  All are incorporated into model objects (not necessarily Items).
The new type of Bitstream is meant to be created, consumed and discarded, not necessarily in a single operation, but within a short period.  The new type are not part of any model object.
(I find it useful to think of three file lifecycles:  (1) scratch files, which are created and destroyed by a single procedure; (2) products, which are left on a shelf to be collected and carried away as soon as it is convenient; and (3) reference files, which remain in the store for long periods to be used by the same system which creates them.  In my mind, the new Bitstream type represents product files, while all previous types represent reference files.  This is what gave rise to my objection.)

@abollini The "DSpaceRunnable" class is now split up between a "DSpaceRunnable" & a "ScriptConfiguration" class. The "ScriptConfiguration" class are the singleton beans & the actual runnables are created by java when we want to start a process.
@tdonohue The bitstream findAll() method has been disabled.

@KevinVdV and @Raf-atmire : Just a quick note.  I see some updates have been made here recently, but I also still see unaddressed feedback of mine in the MetadataExport and MetadataImport classes with regards to improper Context management.  Please make sure to thoroughly review all prior feedback, and let me know once you feel it's all been addressed.  Thanks!

@tdonohue I believe we have processed all of your feedback, can you please re-review this PR ?

as noted in today meeting I'm satisfied by the current changes so when the others feedback is addressed as well, it can be merged. I haven't found the test suggested here
https://github.com/DSpace/DSpace/pull/2648#pullrequestreview-362247628
to verify that multiple scripts can run in parallel without interference, if this is the case and implement it would be too costly can I ask please to just create a ticket with the above note as a suggestion for a future improvement?

@KevinVdV and/or @Raf-atmire : Let me know once this PR has been updated / resynced with the Contract changes from this week: https://github.com/DSpace/Rest7Contract/pull/99  (At a minimum, I know the metadata field needs renaming to dspace.process.filetype)  I'll gladly review this once it is ready.
I also agree with @abollini that we should make sure to address the outstanding question around running multiple scripts in parallel.  Either we need a test to prove it works already, or we need a JIRA ticket to complete this task at a later date (in which case I can schedule it for a later beta).  Let me know which approach you'd like to take here.

@tdonohue We implemented the changes in the rest contract. The integration tests for parallel process execution testing are best done later on, I created the following JIRA ticket for it: https://jira.lyrasis.org/projects/DS/issues/DS-4528


feat: support new sepolia testnet [APE-1494]

What I did
Adds support for the new Sepolia testnet
How I did it
How to verify it
Checklist

[x] Passes all linting checks (pre-commit and CI jobs)
[ ] New test cases have been added and are passing
[x] Documentation has been updated
[x] PR title follows Conventional Commit standard (will be automatically included in the changelog)

related: https://github.com/ApeWorX/ape-infura/pull/68


Feature/training data pipeline

Description
This pull request is for the training data pipeline; This consists of a GUI style application (CLI to come) that allows for great control and far better results when preparing data for training on custom models.
--- Loom Video ---
https://www.loom.com/share/2f6741ce6470402fa19bf59f6952340d
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Change Status

[ ] Complete, tested, ready to review and merge
[x] Complete, but not tested (may need new tests) ** --- Includes some tests --- **
[ ] Incomplete/work-in-progress, PR is for discussion/feedback

Has This Been Tested

[x] Yes
[ ] No
[ ] Not necessary

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] There are no merge conflicts

This is a massive PR and deserves scrutiny. I'm surprised by the lack of inline comments. This is a common problem with large PRs, it's too big to hold in our heads so it's much harder to review line by line. Without evidence of a detailed review, it will make the approving comments equivalent to 🤞🏻
Kudos to @peterger8y for the inline comment.
@techthumb1 in the future do not hold back. Add inline comments in places you think deserve documentation (docstrings).

Great work Tasha. I reviewed your code during the collaboration with you. Your work is clean and I liked your idea of importing code as a package to be used in any file. This was very helpful in making multiple files that were manageable and easy to understand.


interleavedBuffer from meshoptimizer glb for Trimesh

meshoptimizer  use interleavedBuffer for the glb optimation
this needs to be conferted to BufferAtrributes as input for the Trimesh Shape
https://codesandbox.io/s/trimesh-test-meshopt-2l4wt?file=/src/index.js
( load the unoptimized glb works fine loadModel("../public/diamond.glb");)

works with this  interleavedBuffer to non-interleavedBuffer
https://codesandbox.io/s/trimesh-test-meshopt-2l4wt?file=/src/index.js


Fix:SQS md5 calculation for custom string data type.

Fix:SQS md5 calculation for custom string data type.
https://github.com/localstack/localstack/issues/2976

Coverage decreased (-0.3%) to 94.144% when pulling 9c2ba8c5f2711188bf6488788e6112b8f0ba31b2 on usmangani1:LISSUE2976 into ebb1c6bd6881a9d3e69c39755ba16bbb8c439439 on spulec:master.


Problems about resuming from checkpoint for finetune_with_lora

Hi, thanks for your great job. I found some errors when I wanted to continue my LoRA finetuning.
09/04/2024 15:54:23 - INFO - accelerate.accelerator - Loading states from output/tulu_v2_dolly_openorca_1M_v2_64_7B_lora/step_15600
09/04/2024 15:54:23 - INFO - accelerate.accelerator - Loading DeepSpeed Model and Optimizer
[rank0]: Traceback (most recent call last):
[rank0]:   File "/homes/gws/ypwang61/Research/LLM/open-instruct/open_instruct/finetune.py", line 682, in <module>
[rank0]:     main()
[rank0]:   File "/homes/gws/ypwang61/Research/LLM/open-instruct/open_instruct/finetune.py", line 573, in main
[rank0]:     accelerator.load_state(checkpoint_path)
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/accelerate/accelerator.py", line 3064, in load_state
[rank0]:     model.load_checkpoint(input_dir, ckpt_id, **load_model_func_kwargs)
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2759, in load_checkpoint
[rank0]:     load_path, client_states = self._load_checkpoint(load_dir,
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2809, in _load_checkpoint
[rank0]:     sd_loader = SDLoaderFactory.get_sd_loader(ckpt_list, checkpoint_engine=self.checkpoint_engine)
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/state_dict_factory.py", line 43, in get_sd_loader
[rank0]:     return MegatronSDLoader(ckpt_list, version, checkpoint_engine)
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/state_dict_factory.py", line 193, in __init__
[rank0]:     super().__init__(ckpt_list, version, checkpoint_engine)
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/state_dict_factory.py", line 55, in __init__
[rank0]:     self.check_ckpt_list()
[rank0]:   File "/homes/gws/ypwang61/miniconda3/envs/oi/lib/python3.10/site-packages/deepspeed/runtime/state_dict_factory.py", line 168, in check_ckpt_list
[rank0]:     assert len(self.ckpt_list) > 0
[rank0]: AssertionError

This is what I have in the step_15600 directory:
adapter_config.json  adapter_model.safetensors  README.md

Hi! The code doesn't currently support resuming training when doing LoRA training. We should add support for this (often internally we usually just full-finetune). Feel free to help with adding, otherwise it might take a little time to get to this, due to some upcoming deadlines. I'll leave the issue open to track this.


Null states and Chain info for nft

I have read the CLA Document and I hereby sign the CLA

I have read the CLA Document and I hereby sign the CLA

recheck


Swapping items

Hi there,
is there any way of swapping items around?

Please submit an issue with a properly detailed question


Message: 'Version on server is not -1 of what you supplied: Server: 1, Client req: 1'

The client throws this error on the uploadEPUB method: Message: 'Version on server is not -1 of what you supplied: Server: 1, Client req: 1',
Everything was working fine until a few days ago. Perhaps this could be change to the server.
  const deviceToken = "<token>"
  const client = new Remarkable({ deviceToken })
  await client.refreshToken()

  const epub = fs.readFileSync("./file.epub")
  try {
    return await client.uploadEPUB(name, uuid, epub)
  } catch (error) {
    console.error(error.message)
  }```

I hit this error as well

you have to change the uuid, otherwise it thinks that the file already exists and it won't overwrite it, change the UUID and it'll work (it did in my case at least)


Chore: Only await futures

:scroll: Description
Fixes #870
By awaiting only futures, the performance can be marginally improved.
I did my best to find all instances, but I'm not sure I got all.
:bulb: Motivation and Context
My motivation is the strive for a better performance.
:green_heart: How did you test it?
There are no changes in functionality, just in the implementation. So the existing tests cover these changes.
:pencil: Checklist

[x] I reviewed submitted code
[ ] I added tests to verify changes
[x] No new PII added or SDK only sends newly added PII if sendDefaultPii is enabled
[x] I updated the docs if needed
[x] All tests passing
[x] No breaking changes

:crystal_ball: Next steps

@ueman

I did my best to find all instances, but I'm not sure I got all.

I guess you covered most of them, maybe missing are BeforeSendCallback, BeforeSendTransactionCallback, searching for FutureOr<help as well.


Support for pointing to gists

Quicker than setting up an entire github repository!

Should this be in repo2docker?

Nope - repo2docker only cares about getting a git repository. Since gists are already git repositories, we don't need any additional work in repo2docker.

gotcha - in that case, perhaps it seems like this is just a matter of making sure URLs to gists work + documenting for users how to do this...doesn't sound like we need to change any machinery in general, no?
What about the URL structure? I think gists follow the same username/gist-id convention so we could just treat gist-id as a repo name for url purposes...though gists don't have things like "branches". Alternatively we could have a separate spec for that (e.g. instead of /v2/gh/ we have /v2/gist). WDYT?

will be closed by #306

@choldgraf i think this issue should have been closed or ?


Improved support for x-zally-ignore

Dropped recently added SwaggerIgnoresExtension (adding a second parameter to check methods was pretty ugly).
RuleValidator uses a new Context to contain everything a check might need to know:

SwaggerRuleValidator uses new SwaggerContext.
JsonRuleValidator uses new JsonContext.

Check methods can optionally take a Context parameter instead of model root.
Context implements x-zally-ignore logic applicable to vendor extension locations.
SwaggerContext offers validatePaths(..) utility to implement x-zally-ignore aware path based validation.
See LimitNumberOfSubresourcesRule for demonstrates use of new utility method.
More utility methods could be added to iterate over operations, parameters, responses etc but I suggest they evolve as needed and allow #446 to be closed.
Added AvoidXZallyIgnoreRule to explicitly highlight and discourage x-zally-ignore usage.

Codecov Report

Merging #668 into master will increase coverage by 0.25%.
The diff coverage is 94.91%.

@@             Coverage Diff              @@
##             master     #668      +/-   ##
============================================
+ Coverage      83.5%   83.76%   +0.25%
- Complexity      829      831       +2
============================================
  Files           151      153       +2
  Lines          2231     2254      +23
  Branches        315      315
============================================
+ Hits           1863     1888      +25
- Misses          229      232       +3
+ Partials        139      134       -5

Impacted Files
Coverage Δ
Complexity Δ

...ver/src/main/java/de/zalando/zally/rule/Context.kt
100% <100%> (ø)
2 <2> (?)

...n/java/de/zalando/zally/rule/JsonRulesValidator.kt
71.42% <100%> (-8.58%)
3 <1> (-2)

.../main/java/de/zalando/zally/rule/SwaggerContext.kt
100% <100%> (ø)
13 <13> (?)

...ally/rule/zalando/LimitNumberOfSubresourcesRule.kt
100% <100%> (+12.5%)
2 <1> (-12)
:arrow_down:

...src/main/java/de/zalando/zally/rule/JsonContext.kt
100% <100%> (ø)
5 <5> (?)

...src/main/java/de/zalando/zally/rule/RulesPolicy.kt
100% <100%> (ø)
5 <2> (+1)
:arrow_up:

...ava/de/zalando/zally/rule/SwaggerRulesValidator.kt
66.66% <100%> (-22.23%)
3 <1> (-3)

.../main/java/de/zalando/zally/rule/RulesValidator.kt
88.23% <84.21%> (+0.73%)
18 <9> (+1)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df7bba0...db5c914. Read the comment docs.

@maxim-tschumak, @netme: Would appreciate some review on this. Personally I think the new utility methods should make most rules much simpler to implement and also buys x-zally-ignores support into the bargain. Additionally I think AvoidXZallyIgnoresRule is the perfect way to inform users of x-zally-ignores usage.

Thanks for this proposal @roxspring! I find it elegant, but I do have some concerns:

Users don't know which of the rules support the "ignore on path level" functionality.
If we go this direction, we should avoid having multiple APIs for the Check (in this case we should stick to Context as the only possible parameter of the Check function). This also means we have to adjust all Checks asap (which would fix 1.).
Maybe we can simplify the Check interface. (important since our customers have to implement this interface)
(minor): toMessages method should be renamed, check(Path), validate, etc. would be better)

AvoidXZallyIgnoresRule is a great idea!
@mfellner @tkrop @netme Please share your thoughts!

Glad you like it @maxim-tschumak - hope others do too!
I had intended (1) to be addressed by (2) in time and should have stated that somewhere - certainly wanted to avoid any more merge requests where I'm forced to touch every single rule! validate(..) sounds better for (4) - think I tried it and found it confusing but will have another go at it today and maybe fall back to check(..) if that seems clearer.
Are you able to expand on (3) though? I'm not clear what you're wanting to simplify.

Are you able to expand on (3) though? I'm not clear what you're wanting to simplify.

I don't have a concrete proposal so far. The implementer of a check has to:

know the signature of the validate method (SwaggerContext -> Violation?), and
know the signature of the validatePaths method (String, (Swagger, String, Path) -> List<String> -> Violation)

This is a bit too much to implement "just" a check on a swagger object. I hope we can simplify this for the check implementers.

[..] move the description into the @Check annotation [..]

I was also thinking about this. For now, we use Check descriptions as Violation texts. We can separate these things:

Static Check description in the annotation (answering the question "What's wrong?")
Violation text is used to motivate a change (answering the question "How to fix?")

This should not be a part of this particular change though.

That makes sense. We've been using some very similar extension methods for writing rules in-house for a couple of months now and I guess we've been valuing the clarity of the result over the ease of getting started without copying another.
An alternative approach I mentioned before (Oct/Nov timeframe?) is to move the model traversal into SwaggerRuleValidator and then use reflection to invoke checks matching a handful of signatures along the lines below.

validateSwagger: (Swagger) -> Violation?
validatePath: (Swagger,String,Path) -> Violation?
validateOperation: (Swagger,String,Path,HttpMethod,Operation) -> Violation?
validateParameter: (Swagger,String,Path,HttpMethod,Operation,Parameter) -> Violation?
validateResponse: (Swagger,String,Path,HttpMethod,Operation,String,Response) -> Violation?

Implementers would still have a few signatures to remember but only one set, and maybe those could be provided in interfaces to help people get started. Perhaps I should prepare an alternate PR based on this approach and see which we prefer?

I remember your proposal @roxspring :) It's probably the best way to go. Please let's wait for others' opinions.

I agree with @maxim-tschumak that the API for rule authors should be as simple as possible. However the approach of creating multiple adaptor methods inside Context (validatePath, validateOperation, etc.) makes the API more complicated imho.
I like the idea of having a Context to abstract away some common logic for each rule and to (eventually) only pass this context to each rule. In fact, we wanted to do the same thing to implement https://github.com/zalando/zally/issues/204 and support both Swagger 2 and OpenAPI 3.
But I also think that giving rules the raw Swagger object is cleaner and more maintainable than providing special accessor methods inside Context. This and the introduction of a callback-based API also makes implementing rules more complicated than before, imo.
In https://github.com/zalando/zally/issues/689 we suggest traversing Swagger ahead of time to create a lookup map to get a JSON pointer for any child node of Swagger. This traversal can also be used to check if a child node has the method vendorExtension, check wether x-zally-ignore is set, and completely remove the respective sub-tree of the Swagger object.
That way ignored parts of Swagger would already be removed/nulled ahead of time, before running the checks.
@roxspring, what do you think about this alternate approach?

I agree with @maxim-tschumak that the API for rule authors should be as simple as possible. However the approach of creating multiple adaptor methods inside Context (validatePath, validateOperation, etc.) makes the API more complicated imho.

Totally agree that the new rule author should need to understand minimal API contracts. I think we're also all in agreement that x-zally-ignore processing should be happening centrally; outside of the rules if at all possible.
I think the bit where we perhaps disagree is that my team have found that repeating big chunks of traversal logic from rule to rule is boring, error-prone and makes reading the code harder. To resolve this I'd like to abstract away the traversal logic of "for each response of each operation of each path" into a single readable expression, and the (possibly more advanced) rule author to focus directly on their rule logic.
My suggestion to address this is to allow a set of API check signatures, starting with validate(Swagger):Violation?, but potentially expanding to:

validate(Swagger):Violation?
validate(Swagger,String,Path):Violation?, -
validate(Swagger,String,Path,HttpMethod,Operation):Violation?, 0 -
validate(Swagger,String,Path,HttpMethod,Operation,String,Response):Violation? etc.

Admittedly the final parameter will often be all that's of interest, but sometimes one/more of the other parameters is needed for context, and when implementing any unused parameter can simply be named _ for clarity. A side benefit of the abstracted traversal is that the paths can be prefixed in a uniform manner, such that all path related violations have paths beginning "paths: /the/path: " or whatever is agreed upon.
I guess we could try and query the model using JSON pointer expressions to hide some of the traversal but I'm not clear how to include the benefits of contextual model elements or uniform paths, and it has the cost of developers having to learn JSON pointer (which I failed at with a 5min attempt earlier today).

I like the idea of having a Context to abstract away some common logic for each rule and to (eventually) only pass this context to each rule. In fact, we wanted to do the same thing to implement #204 and support both Swagger 2 and OpenAPI 3.
But I also think that giving rules the raw Swagger object is cleaner and more maintainable than providing special accessor methods inside Context. This and the introduction of a callback-based API also makes implementing rules more complicated than before, imo.

Agree that the callback approach over-complicates things, and direct access to the Swagger root object is much easier for rule developers to use.

In #689 we suggest traversing Swagger ahead of time to create a lookup map to get a JSON pointer for any child node of Swagger. This traversal can also be used to check if a child node has the method vendorExtension, check whether x-zally-ignore is set, and completely remove the respective sub-tree of the Swagger object.
That way ignored parts of Swagger would already be removed/nulled ahead of time, before running the checks.

Now that's an idea that hadn't occurred to me! You'd have to take a deep copy of the entire model before pruning (or re-parse), so that the original model can be used for the next rule. I'm not too clear how a JSON pointer make this easier but I'm sure you have a plan. My initial reaction is that perhaps additional copying and pruning might burn resources unnecessarily - but it's definitely worth exploring!!

@roxspring thank you for your feedback on this matter, it is really helpful! Let me try to list the different concers that we've started to discuss across multiple issues:

cleaner implementation of x-zally-ignore
the notion of Context as a wrapper/adapter/decorator for Swagger
abstracting away redundant traversal of Swagger in each rule
OpenAPI 3 support with fallback support for Swagger 2
generating violation paths automatically instead of manually in each rule
using only valid RFC-standard JSON pointers as violation paths
overloading/abusing Violations attributes to store additional information

(1.) So far we've said that it can either be realised by:

having abstract adapter methods to access nodes inside Swagger that filter out ignored nodes
traversing and pruning the Swagger object ahead of time

(2.) We agree it's useful for both (3)., (4.) and maybe (5.).
(3.) A helpful change than can help making rule implementations much simpler and also support (4.)! However it might break backwards compatibility or we need to update all rules?
(4.) We definitely want OpenAPI 3 support asap, but still want to support Swagger 2 as well. (2.) and (3.) can help with this feature.
(5.) Another helpful change to make rule implementations easier. It can be either achieved by

an implementation for (3.) where each adapter method also constructs the correct path
an ahead-of-time traversal as explained in https://github.com/zalando/zally/issues/689

(6.) This is currently not enforced in any way but (5.) could implement it that way. Valid JSON pointers would be more useful to clients, e.g. the web client.
(7.) We need to rethink the Violation interface or what a rule should return.
About your last comment:

Now that's an idea that hadn't occurred to me! You'd have to take a deep copy of the entire model before pruning (or re-parse), so that the original model can be used for the next rule. I'm not too clear how a JSON pointer make this easier but I'm sure you have a plan. My initial reaction is that perhaps additional copying and pruning might burn resources unnecessarily - but it's definitely worth exploring!!

My expectation would be that a x-zally-ignore should disable all rules and checks so we would need to prune Swagger only once and not keep an unmodified copy around.
JSON pointers won't be useful for the zally-ignore feature, I hope the listing above better explains the different concerns.

A potential issue with the suggested solution for (3.) of having multiple different validate method signatures is that some rules still need to get different properties from different places inside Swagger. For example, SecureWithOAuth2Rule.
In those cases the method signature would have to be validate(Swagger):Violation? and the rule would still need to implement all the necessary traversal. @vadeg and I thought that a better solution to solve (3.) could be either a plain old adepter or a fancy Kotlin DSL.

Great summary of where the discussion is at @mfellner 👍 - I agree with all the points you numbered. Let me add (8.) and respond to your comments regarding (3.):
(8.)

My expectation would be that a x-zally-ignore should disable all rules and checks so we would need to prune Swagger only once and not keep an unmodified copy around.

My expectation is different. I was expecting to be able to ignore specific rules at lower locations. I want my team to be obeying as many of the rules as possible, and am ok with them temporarily using x-zally-ignore: [SomeNewRule] on a subtree that can't meet the new rule yet for legacy reasons, but I don't want them to then accidentally fall out of compliance with all other rules.
(3.)

A potential issue with the suggested solution for (3.) of having multiple different validate method signatures is that some rules still need to get different properties from different places inside Swagger. For example, SecureWithOAuth2Rule.

That's precisely the point of the various signatures, you provide the node in the model that needs validating preceeded by it's ancestry which might be needed for context. e.g. checkUsedScopesAreDefined(..) would operate at the Operation level but would need the root Swagger context to lookup defined scopes.
    @Check(severity = Severity.MUST)
    fun checkSecurityDefinitions(_: Swagger, securityDefinition: SecuritySchemeDefinition): Violation? {
        // validate specific definition
    }

    @Check(severity = Severity.SHOULD)
    fun checkPasswordFlow(_: Swagger, securityDefinition: SecuritySchemeDefinition): Violation? {
        // validate specific definition
    }

    @Check(severity = Severity.MUST)
    fun checkUsedScopesAreDefined(swagger: Swagger, _: String, _: Path, _: HttpMethod, op: Operation): Violation? {
        val definedScopes = getDefinedScopes(swagger)
        // validate specific operation
    }

(3.)

@vadeg and I thought that a better solution to solve (3.) could be either a plain old adepter or a fancy Kotlin DSL.
I haven't played with Kotlin DSLs so can't really comment, other than to say I'm curious how it would work out!

I have just played a little bit with Kotlin and I have an idea to implement simple and flexible API for Swagger and OpenApi. I think by simplifying traversal logic we would not need several validate methods for all rules and can leave it up to specific rule.
For example:
        val parseResult = OpenAPIParser().readLocation(url, null, ParseOptions())
        val root = parseResult.openAPI

        // Get all operations having tag "super-pets" from all paths equals "/pets"
        val allOperations = root.paths { (name, _) ->
            name == "/pets"
        }.operations { it ->
            it.tags.contains("super-pets")
        }

        println("All /pets operations: ${allOperations.all().size}")

        // Get all responses from all operations above
        val allPetsResponses = allOperations.responses()
                .flatMap { it -> it.entries }
        println("All responses for all /pets operations: ${allPetsResponses.size}")

I have just slightly updated OpenApi (former Swagger class) with additional methods (thank you, Kotlin) and now I can traverse more flexible rather than before.
Also, I can join methods into chains, implement filtering, grouping and etc.
I think if we can simplify traversal logic by implementing extensions/delegates and etc. then we can keep one validate method and traverse much easier.
This
   @Check(severity = Severity.MUST)
    fun checkSecurityDefinitions(_: Swagger, securityDefinition: SecuritySchemeDefinition): Violation? {
        // validate specific definition
    }

can be rewritten like this
   val root = //Swagger object init
   val violations = root.securitySchemes { name, scheme ->
        //validate specific definition
        Violation("Something wrong with security")
   }
    // Do something with violations...

Pros

All traversable logic is located "close" to the source.
More traversable callbacks can be added.
Everything can be checked at compile time.
No "magic" with multiple validate methods.

Cons

Need carefully implement callbacks.
Rule remains simple.
Less map -> flatMap(map -> map -> flatMap) inside rules.

Thank you for your answers, @roxspring!

My expectation is different. I was expecting to be able to ignore specific rules at lower locations.

Yes, I can see that now. It makes sense to have a more granular ignore-logic. In that case it would be inefficient to traverse and prune Swagger for every rule. However we then need to be able to detect wether a certain rule operates on a sub-tree of Swagger with a x-zally-ignore attribute matching that specific rule at its root.
I could imagine doing that by creating another data structure during an initial traversal pass that maps Swagger node objects to (already parsed) x-zally-ignore values. Then we can use this map to check wether a node should be ignored. The map would probably need to have entries not only for the root node of sub-tree (the one with x-zally-ignore) but also all of its children...

That's precisely the point of the various signatures, you provide the node in the model that needs validating preceeded by it's ancestry which might be needed for context. e.g. checkUsedScopesAreDefined(..) would operate at the Operation level but would need the root Swagger context to lookup defined scopes.

@Check(severity = Severity.MUST)
fun checkUsedScopesAreDefined(swagger: Swagger, _: String, _: Path, _: HttpMethod, op: Operation): Violation? {
    val definedScopes = getDefinedScopes(swagger)
    // validate specific operation
}

I am not convinced that we would gain much by this though. Although we can directly access op we still need to have some method getDefinedScopes to get the defined scopes from the root. And the other parameters need to be ignored with _.
This would also not be very intuitive and discoverable to write as opposed to the type-system assisted way of accessing properties on the Swagger object directly. However the problem of verbose and cumbersome traversal with flatMap etc. remains. Another question that I have: would the RulesValidator call a method like checkUsedScopesAreDefined(swagger: Swagger, _: String, _: Path, _: HttpMethod, op: Operation) multiple times, essentially swa * str * pat * meth * op times (those variables being the number of possible attribute combinations)?
That's why I think @vadeg's approach of simplifying the access and traversal of Swagger properties is more promising. Given that with the approach of special validate methods we would still need "helper" methods for accessing some properties, why not reduce the whole solution to just making this access easier and more concise?

I know this PR has ended up mostly in discussion, but have you looked at the changes proposed? I find them to be very similar to those proposed by @vadeg, with just a couple of differences:

The methods operate on a Context object rather the model root, thereby allowing the additional metadata essential for implementing x-zally-ignore functionality (i.e. the id of the current rule)
The methods also handle the tedious construction of Violations.
Possibly @vadeg might be advocating dropping support for multiple checks within a rule which would be a shame IMHO.

In-house we've been using similar extension methods (on Swagger, no ignore support, constructing Violations) for a couple of months and find rules much cleaner to read/write/maintain with them.
I note also that @mfellner's first comment on this thread objected to complicating things with a "callback-based API" and yet this is exactly what's being approved of from @vadeg. I don't call this out to complain but to highlight that I think we've now all traveled the same mental journey around the problem-space and seem to be approaching full circle! Hopefully this means we are closer to identifying the best (aka least-worst) route forward! I think the options on the table, applied to a more interesting deep iteration (worst case?) are as follows:
I think @vadeg proposes (1.):
@Check(severity=Severity.MUST)
fun validate(root: Swagger) =
    root.responses { pattern, path, method, operation, status, response ->
        when {
            status != "200" -> null // some filtering based on context
            response.schema == null -> Violation("Schemas should be specified", /* path construction */)
            else -> null
        }
   }

vs possibly the similar (1b.)?:
@Check(severity=Severity.MUST)
fun validate(root: Swagger) =
    root.paths { pattern, path ->
        path.operations { method, operation ->
            operation.responses { status, response ->
                when {
                    status != "200" -> null // some filtering based on context
                    response.schema == null -> Violation("Schemas should be specified", /* path construction */)
                    else -> null
                }
            }
        }
   }

vs as in my PR (2.)
@Check(severity=Severity.MUST)
fun validate(context: SwaggerContext) =
    context.validateResponses("Schemas should be specified") { pattern, path, method, operation, status, response ->
        when {
            status != "200" -> null // some filtering based on context
            response.schema == null -> /* path construction */
            else -> null
        }
   }

vs as previously proposed (3.)
@Check(severity=Severity.MUST)
fun validate(root: Swagger, pattern: String, path: Path, method: HttpMethod, operation: Operation, status: String, response: Response) =
    when {
        status != "200" -> null // some filtering based on context
        response.schema == null -> Violation("Schemas should be specified", /* path construction */)
        else -> null
    }

Additionally @maxim-tschumak and I also pondered variations where the violation description is static and pulled out to the annotation too, e.g.: (4.)
@Check(severity=Severity.MUST, description="Schemas should be specified")
fun validate(root: Swagger, pattern: String, path: Path, method: HttpMethod, operation: Operation, status: String, response: Response) =
    when {
        status != "200" -> null // some filtering based on context
        response.schema == null -> Violation(/* path construction */)
        else -> null
    }

Personally I like (1.) but find it viable only if we pre-process the model for x-zally-ignore, and so probably lean towards (2.) as the most readable (especially if we move the description to the annotation as in (4.)).
I do also recognize though that (3.) might be the easiest to understand for new rule authors not starting from a cut & paste, but I question how many won't start from a cut & paste and whether we really ought to optimize for them!?

Sorry - failed to address a couple of questions:

Another question that I have: would the RulesValidator call a method like checkUsedScopesAreDefined(swagger: Swagger, _: String, _: Path, _: HttpMethod, op: Operation) multiple times, essentially swa * str * pat * meth * op times (those variables being the number of possible attribute combinations)?

Yes, that's would be my intention, probably doing some aggregations of Violations where rule/severity/description match.

That's why I think @vadeg's approach of simplifying the access and traversal of Swagger properties is more promising. Given that with the approach of special validate methods we would still need "helper" methods for accessing some properties, why not reduce the whole solution to just making this access easier and more concise?

Totally agree that "easier and more concise" is achieved but it doesn't address the underlying problem of local x-zally-ignore processing.

Thank you, @roxspring for detailed response.
Actually, I am not advocating. I am totally fine with your solution. It works for paths and it is fine. The problem is that now every rule author should remember that path should be retrieved only using Context.validatePaths methods. But there are still Swagger object accessible and no one prevents from accessing swagger.paths and get unfiltered paths. I think that is error-prone because:

authors should know about this behaviour and prevent using Swagger object for paths directly.
reviewers should know about this behaviour and prevent authors to use Swagger object for paths directly.

Actually, we were thinking how to apply x-zally-ignore not only to path but in general. For example there is a rule SnakeCaseInPropNameRule and it works with

definitions
paths
schemas
parameters in definitions
parameters in path
responses

To all of them x-zally-ignore can be applied as well. That means I might have several more methods on context like validateDefinitions, validateSchemas, validateSomethingElse. All of these method has only one goal - filter out all entities marked with x-zally-ignore for specific rule.
We are thinking are there some other options how it can be done.

I think the bit where we perhaps disagree is that my team have found that repeating big chunks of traversal logic from rule to rule is boring, error-prone and makes reading the code harder. To resolve this I'd like to abstract away the traversal logic of "for each response of each operation of each path" into a single readable expression, and the (possibly more advanced) rule author to focus directly on their rule logic.

How many of these methods we would need to define? Of course a lot of rules working with paths but they all work in a different way. For example MediaTypesRule currently works with produces/consumes but in API 3.0 it will work with path/responses or components/responses,
Rule SnakeCaseInPropNameRule and CommonFieldTypesRule works with all objects.
I have an impression that in the end we will end up with one method per model entity.
Our ideas, your ideas - all of them have pros and cons. Currently we are trying different ways how we can solve these problems (or can it be solved in general). I think together we can implement proper solution.

The problem is that now every rule author should remember that path should be retrieved only using Context.validatePaths methods. But there are still Swagger object accessible and no one prevents from accessing swagger.paths and get unfiltered paths

Yes, this is a fair point, and one which I'd largely been ignoring.

How many of these methods we would need to define? ... I have an impression that in the end we will end up with one method per model entity.

Yes, this is a fair accusation too.
OK, so I suspect the only real way to truly enforce that rule XYZ properly ignores elements with x-zally-ignore: [XYZ] is to prune the object model before handing it to the rule to process. Each pruned variation of the tree nodes would have different hashCodes throughout and so any Map<Object,JSONPointer> would likely need to be rebuilt for that pruned variation too. Our assumption is that this would be wasteful of time/memory.
Clearly the pathological case here is that somebody submits a swagger definition with every enabled rule being ignored for different subtrees of the model - and perhaps this burns more time/memory than we'd like - but if it still completes then maybe that's not too bad? I would hope that the normal case would be to be ignoring a handful of rules at worst, likely clustered around similar subtrees - in which case there should be lots of opportunities to reuse trees and caches.

@roxspring @mfellner , some more ideas came to my mind except those that @roxspring already suggested:
Option 1: Decorators
Wrap every object class with some kind of decorator which can filter object based on extensions.
Pros

Clear logic

Cons

Original hierarchy will be duplicated with decorators hierarchy.
Very time consuming and error-prone

Option 2: AOP
Use AOP (f.e. AspectJ) to pre-process Swagger data model and attach pointcuts to specific places (like Spring is doing with beans).
Pros

Whole model and be processed with few lines of code.

Cons

Unexpected behaviour. It would be hard to understand why it works like that only looking at the code.

Option 3: Filter violations
Build a map of ignored paths for each rule. Pass whole Swagger model to each rule (like it works right now) because only rule knows what it needs for checks. Receive violations from rules like JSON pointers and exclude violations which should be ignored using ignored paths.
Pros

Easy to implement because we will have JSON pointers map for another reasons.

Cons

Rules will be executed even if it is not needed.

In SecureWithOAuth2Rule.checkUsedScopes() we validate Path objects conform with what's found in SecuritySchemeDefinitions. One thing I like about filtering the input model the rule sees (whether pre-pruning, decorating, aspects) is that you could use x-zally-ignore in the SecuritySchemeDefintions and effect the results. As far as I can think through, filtering output Violations is unable to achieve the same.
Not sure that this theoretical benefit translates into any practical benefit though, so perhaps filtering the results is the sane way forward. It's certainly seems a lot more practical with the JSON pointers than it is currently with unstructured strings.

@roxspring @mfellner I have created #692 as one of the possible solutions how to ignore model instances based on vendor extensions.

Thank you everybody for the detailed inputs! It can feel cumbersome to discuss these topics for so long however it is hopefully going to pay off if we get it right.
Personally, I would prefer solutions that do not need more 3rd party libraries or that introduce yet another programming concept to Zally (e.g. aspects). I'm even skeptical using things from Spring because imo it should only be used on the web service layer of the application.
Ideally we find a solution that only uses vanilla Java and Kotlin features.
In the meantime I was able to implement the Swagger "model tree walker" (https://github.com/zalando/zally/blob/openapi3/server/src/test/java/de/zalando/zally/util/OpenApiWalkerTest.kt) that makes it easy to get JSON pointers and x-zally-ignore information for any node in the Swagger (or OpenAPI) object.
To me, this in combination with the "telescoping" validate methods approach that @roxspring has suggested and/or a light-weight and native variant of @vadeg's proxies looks the most promising to me. @tkrop also chimed in and said that we wouldn't be opposed to the "telescoping" validate methods either.
But to move forward I believe we need a small incremental improvement. The OpenApiWalker would allow for an intermediate solution, evaluating x-zally-ignore after the fact:

change the Violation interface to only have two properties

description: String
node: Object (the violating node in Swagger/OpenApi)

RulesValidator reads the violation and handles it

looks up the JSON pointers (paths) in the node map based on the node in the Violation
checks wether the node is ignored by looking into the node map as well
discards the Violation or transforms it and passes it on

pros

small API change
rules become immediately simpler: no more paths/pointers generation, no more "ignore" checking

cons

doesn't solve the Swagger traversal issue (yet)
rules are executed even if they should be ignored

Personally I think this could be a very promising approach because if we at least believe that changing the Violation interface in this way is right (because it allows us to make rules simpler since RulesValidator has more information to play with after a Violation is generated). It would give us some immediate benefits while being a relatively small code change and would also be forwards compatible with future improvements, even ignoring parts of Swagger ahead of time and simpler traversal.
What do you think, @roxspring, @vadeg, @tkrop, @maxim-tschumak, @netme?

Thank you everybody for the detailed inputs! It can feel cumbersome to discuss these topics for so long however it is hopefully going to pay off if we get it right.

Agreed - thanks all!❤
Definitely agree that small incremental improvements are ideal wherever possible, and the proposed plan seems to deliver that: Formalizing the Violations/Results around something like JSON pointer seem to be a positive step forward, independent of x-zally-ignore support.
A few minor questions/issues with the plan:

If each Violation is to be associated with exactly one node, then presumably an unstated part of your plan is to let all the rules return a list of Violations rather than just one? (RulesValidator should already support this)
Have you looked at getting the JsonNode based rules to play with this approach?
I have some questions/comments around the OpenApiWalker but I guess they should wait for a WIP PR from that branch (hint hint).

Thanks @roxspring! In the end our next steps also depend on wether we want to prioritise OpenApi 3 or the other topics first (personally I would prefer the former).

If each Violation is to be associated with exactly one node, then presumably an unstated part of your plan is to let all the rules return a list of Violations rather than just one?

 Thanks for pointing that out.

Have you looked at getting the JsonNode based rules to play with this approach?

The OpenApiWalker should work with JsonNodes too in principle but I haven't tried it yet.

I have some questions/comments around the OpenApiWalker but I guess they should wait for a WIP PR from that branch (hint hint).

OK! Let's discuss here: https://github.com/zalando/zally/pull/693

@mfellner nice plan!

checks wether the node is ignored by looking into the node map as well
discards the Violation or transforms it and passes it on

I am not sure that it will work in all cases. @roxspring provided good example when it would not work because filtering should be done before executing rule but not after.

Personally, I would prefer solutions that do not need more 3rd party libraries or that introduce yet another programming concept to Zally (e.g. aspects). I'm even skeptical using things from Spring because imo it should only be used on the web service layer of the application.

Yes, but this is very small and narrowed aspect :) Spring dependency is optional. I did it because it was faster to show. Spring can be safely replaced by cglib dependency. And yes, this is another 3rd party library. I think in compare to effort/value it brings more value with less effort rather than other solutions.

Ideally we find a solution that only uses vanilla Java and Kotlin features.
Ideally yes and I also vote for this. But, unfortunately, I have not found anything useful in Kotlin. Java has Proxies (as you mentioned in #692 ) but it does not work with classes. Only with interfaces (that is one of the major reasons that cglib exists actually).

I would be happy if someone will find better solution for this except cglib.

Thanks for this important discussion @roxspring @vadeg @mfellner !
I agree with the plan described by @mfellner and think this is the best way to go. It's maybe not the nicest solution from the technical perspective, but this trade-off is very reasonable in our situation: this is a nice-to-have feature and we also want to discourage (#686) its extensive usage. Let's not over-engineer the solution :)

@maxim-tschumak , @mfellner , @vadeg thanks a lot for your comments.
@roxspring should we close this PR?

Yup. Long been clear that this PR wasn't going in but wanted to keep it open while the conversation was useful.


[Bug]: Next.js and rules_ts are incompatible for custom tsconfig.json options

What happened?
Next.js is currently trying to overwrite the tsconfig.json at runtime with its "suggested" values. Some options are unfortunately incompatible with rules_ts, specifically the configuration noEmit must be false or undefined for the rules to work but Next.js attempts to overwrite it with the value true enabling it.
In the js_run_devserver the attribute grant_sandbox_write_permissions can be enabled as a workaround to let Next.js do its magic. Unfortunately, this is not possible in js_run_binary rule.
The options are therefore the following:

Embed in the srcs the tsconfig.json with noEmit disabled. This will fail as Next.js attempts to overwrite the read-only file.
Embed in the srcs the tsconfig.json with noEmit enabled. This will fail as rules_ts require output files to be generated (noEmit must be disabled).
Do not embed in the srcs the tsconfig.json and let Next.js generate a default one during runtime. This is a suboptimal option as no custom configuration can be defined (this is the current approach in the Next.js example in this repository).
Embed in the srcs a runtime tsconfig.json with noEmit enabled, other compile options must be compatible with Next.js enforced options. Reference a second, different tsconfig.build.json during build time (i.e. in the ts_project definitions under the tsconfig attribute) which is as close as possible to the runtime configuration. This is suboptimal but the best solution found so far.

NOTE: Next.js does not allow disabling overwriting the tsconfig.json.
NOTE: Next.js does not allow to specify a different tsconfig.json, it must be in the root of the project being compiled.
Keywords:

Error: EROFS: read-only file system, open
cannot be built because the 'noEmit' option is specified in the tsconfig

Example
BUILD.bazel
load("@aspect_rules_ts//ts:defs.bzl", "ts_config")
load("@aspect_rules_js//js:defs.bzl", "js_run_binary", "js_run_devserver")
load("@internal_npm//:next/package_json.bzl", next_bin = "bin")

# See the following pages for examples, issues, and documentation:
#
# - https://github.com/aspect-build/bazel-examples/blob/main/next.js/apps/alpha/BUILD.bazel
# - https://github.com/aspect-build/bazel-examples/blob/main/next.js/bazel/next.bzl
# - https://github.com/aspect-build/bazel-examples/issues/226

next_bin.next_binary(
    name = "next_js_binary",
    visibility = ["//visibility:public"],
)

ts_config(
    name = "tsconfig",
    src = "tsconfig.build.json",
    visibility = ["//applications/crossrefs-website:__subpackages__"],
)

ts_config(
    name = "tsconfig-runtime",
    src = "tsconfig.json",
    visibility = ["//applications/crossrefs-website:__subpackages__"],
)

srcs = [
    # ...
]

data = [
    ":next.config.js",
    ":next-env.d.ts",
    ":tsconfig-runtime",
    "//:node_modules",
]

# `next build` creates an optimized bundle of the application
# https://nextjs.org/docs/api-reference/cli#build
js_run_binary(
    name = "build",
    srcs = srcs + data,
    args = ["build"],
    chdir = package_name(),
    out_dirs = [".next"],
    tool = ":next_js_binary",
)

# `next dev` runs the application in development mode
# https://nextjs.org/docs/api-reference/cli#development
js_run_devserver(
    name = "dev",
    args = ["dev"],
    chdir = package_name(),
    command = "../../node_modules/.bin/next",
    data = srcs + data,
)

# `next start` runs the application in production mode
# https://nextjs.org/docs/api-reference/cli#production
js_run_devserver(
    name = "start",
    args = ["start"],
    chdir = package_name(),
    command = "../../node_modules/.bin/next",
    data = data + [":build"],
)

tsconfig.build.json
    "compilerOptions": {
        "noEmit": false,
        "paths": {
           // paths...
        },
       // compilerOptions...
    },
    // exclude, include, ...
}

tsconfig.json
    "compilerOptions": {
        "noEmit": true,
        "paths": {
           // paths...
        },
       // compilerOptions...
    },
    // exclude, include, ...
}

pages/BUILD.bazel
load("@aspect_rules_ts//ts:defs.bzl", "ts_project")

ts_project(
    name = "pages",
    tsconfig = "//:tsconfig",
    # options, data, srcs, deps, ...
)

Unlike js_run_devserver which creates its own custom sandbox where it can control the permissions of the files, js_run_binary rule runs in the bazel output tree where Bazel controls the permissions of the files. It would be not idiomatic to change the permissions of the files there and it could cause issues with other actions that are running in parallel and also using the tsconfig.json file.
The least hacky fix here would be to patch Next.js in your repository to disable the tsconfig writes. I've seen this done at a client site.

@gregmagolan I believe that Next.js should have addressed this issue a long time ago, I found dozens of closed PRs and issues mentioning the nightmare of dealing with Next.js actively overwriting values in that tsconfig.json. I find it pretty sad that all attempts to solve this have been discarded, in multiple iterations they considered only printing a warning instead of rewriting the file, that would be the optimal solution here.
For context I added a working example of how to use two almost identical tsconfig.json that allow for custom options with noEmit as the only difference. My advice for anyone adopting this workaround is to keep the two configurations as close as possible.

Yeah Bazel can only do what the underlying tool allows. I'd suggest filing more issues with Next.js so they know their approach causes incompatibilities. I think they can default to this sort of user-friendly behavior but should have a flag to turn it off so their tool can be composed into other build systems.
The next.js example moved to https://github.com/bazelbuild/examples/tree/main/frontend - please file an issue there if you think there's something we should change in the example code .- thanks


added other message dependencies

The message generation were still missing dependencies of geometry_msgs  std_msgs. With these changes, crazyswarm2_interfaces should build properly.

related to a part of issue #1

Thanks!


Allow TAB, CR and LF in a query string

Motivation:
The validation logic in PathAndQuery is still too restrictive. It
should allow %09, %0A and %0D (TAB, LF and CR) which can be
included in a query string part, e.g. a user can enter a multi-line
text in a form.
Modifications:

Allow TAB, CR and LF in a query string.

They are still unallowed in a path part.

Result:

Innocent web form submissions are not rejected.

Thanks for swift reviews. Will cut a release tomorrow morning. :sob:

Just wondering, are we reinventing a really complicated wheel with this method?

Maybe.. I hope this is the last hole to close and it leaves less concern of path and query sanitization to users.


Sugarcane Leaf Disease Detection

Pull Request for DL-Simplified 💡
Issue Title : Sugarcane Leaf Disease Detection

Info about the related issue (Aim of the project) : The main goal of this project is to develop a machine learning model capable of accurately classifying different diseases in sugarcane leaves.
Name: Atharv Pal
GitHub ID: atharv1707
Email ID<EMAIL_ADDRESS>Idenitfy yourself: GSSoC'24 Contributor

Closes: #456
Description:
In this pull request, I have thoroughly analyzed and processed the dataset through the following steps:
Exploratory Data Analysis (EDA):

Conducted EDA to gain a comprehensive understanding of the dataset.
Utilized data visualization techniques to uncover patterns and insights.
Applied data augmentation to enhance the diversity of the training data.

Data Pipeline:

Built a robust data pipeline to streamline the data preprocessing and preparation process, essential for training the models.

Model Training:
Trained multiple models to compare their performance based on accuracy metrics:

VGG16
InceptionV3
MobileNetV2
Custom CNN model

Ensemble Learning:

Implemented an ensemble learning approach to combine the strengths of different models. This approach significantly improved the overall performance and accuracy.

Evaluation:

Evaluated the models using classification reports and confusion matrices to ensure the quality of the results.

This comprehensive approach has resulted in improved model performance and accuracy. I hope that these contributions are valuable to the project.
Type of change ☑
What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested? ⚙
The dataset was split into three parts with a 60:20:20 ratio for training, validation, and testing. Separate data generators were created for each subset. After training each model, I evaluated its performance against the testing dataset to ensure accurate training and robust performance. This approach confirmed that the models were correctly trained and capable of performing well on unseen data.
Checklist: ☑

[ x] My code follows the guidelines of this project.
[ x] I have performed a self-review of my own code.
[ x] I have commented my code, particularly wherever it was hard to understand.
[ ] I have made corresponding changes to the documentation.
[ ] My changes generate no new warnings.
[ ] I have added things that prove my fix is effective or that my feature works.
[ ] Any dependent changes have been merged and published in downstream modules.

Hey @abhisheks008! I've implemented the changes based on your suggestions. Could you please review them? Thanks!

hey @abhisheks008  I have deleted the file. Though the indentation seemed fine, I have edited the file suitably so that the issue doesn't come up again. Please review the changes


Resizing window does not update the number of balloons

Way too many balloons!
When it resizes, it should rerun the balloon count calculation and control the balloon population to match. Spawn more balloons or don't respawn popped/missed balloons, depending on the diff.

TODO already written


Black particles

I would like to change the particle colour to black. I changed the page.tsx's first div to the following to allow for testing:
<div className="flex flex-col items-center justify-center w-screen h-screen overflow-hidden bg-gradient-to-tl from-white via-zinc-600/20 to-black">
Next, in the drawCircle() function in the particles.tsx script, I modified this line:
context.current.fillStyle = `rgba(0,0,0,${alpha})`;
Any other color seems to work except this one. No particles appear at all.

In global.css:
body {
  background-color: white !important;
}


Updated READme.md

This change can help clarify that there are three different files presented in the demo to help better orient new users.

nice work. thanks!


Convert Table Data to JSON file

To reduce the massive size of index.html, I think it best to convert the Table Data to a separate JSON file, that DataTables will load.
this should convert the current table to JSON, with the exception of the NBT column
this should detail all that's needed for DataTables to load the JSON

f80b6ac7c66ababf52c7fd7b3d467362ccca4234
16dcfda73c1a8aa1e4912e2829397d48c234e61f
f80b6ac7c66ababf52c7fd7b3d467362ccca4234


Missing whitespace on page /cli_install.html and please use sudo in build_binary_release.sh

Make the build a bit nicer for end-user:
In https://ydf.readthedocs.io/en/latest/cli_install.html, please add a space in INSTALL_DEPENDENCIES=1 BUILD=1./tools/build_binary_release.sh to make it INSTALL_DEPENDENCIES=1 BUILD=1 ./tools/build_binary_release.sh. This makes the build instruction a bit smoother...
And, then, please use sudo in that script when invoking apt. I indeed do not want to run ./tools/build_binary_release.sh as root..

See https://github.com/google/yggdrasil-decision-forests/pull/60 for a fix for sudo.

Hi, thank you for reporting this. I'll add a fix for both problems soon. It's a tiny bit more complicated than your PR, since we sometimes run this script in environments without sudo, so I won't approve the PR and just fix the issue in a change of my own.

Thank you for taking this PR suggestion into account. I'll wait for your correction.
When you say that you sometimes run this script without sudo, do you mean that you run it as root on some, say, cloud environment ?

Mostly the build docker: tools/build_binary_release_in_docker.sh


Couldn't connect to node

Hi all,
i'm trying to run the cardano-wallet against my mainnet node but always get the following message:
[cardano-wallet.network:Warning:13] [2021-07-12 11:17:02.46 UTC] Couldn't connect to node (x87). Retrying in a bit...
I use the following comand to start the wallet:
docker run  inputoutput/cardano-wallet:2021.6.11 serve --node-socket=/home/thomasm/Cardano/cardano-node/node-ipc/mainnet/node.socket --mainnet
Here are also the first few lines of output after i start the wallet:
thomasm@pop-os:~/Cardano/wallet$ docker run  inputoutput/cardano-wallet:2021.6.11 serve --node-socket=/home/thomasm/Cardano/cardano-node/node-ipc/mainnet/node.socket --mainnet [cardano-wallet.main:Info:4] [2021-07-12 11:14:17.30 UTC] Running as v2021-06-11 (git revision: 1532352c75cfe02015873c4c82b0d76eb798dfd8) on x86_64-linux [cardano-wallet.main:Info:4] [2021-07-12 11:14:17.30 UTC] Command line: /bin/cardano-wallet serve --node-socket=/home/thomasm/Cardano/cardano-node/node-ipc/mainnet/node.socket --mainnet [cardano-wallet.main:Debug:4] [2021-07-12 11:14:17.30 UTC] ServeArgs {_hostPreference = Host "<IP_ADDRESS>", _listen = ListenOnPort 8090, _tlsConfig = Nothing, _nodeSocket = CardanoNodeConn "/home/thomasm/Cardano/cardano-node/node-ipc/mainnet/node.socket", _networkConfiguration = MainnetConfig, _database = Nothing, _syncTolerance = SyncTolerance 300s, _enableShutdownHandler = False, _poolMetadataSourceOpt = Nothing, _tokenMetadataSourceOpt = Nothing, _logging = LoggingOptions {loggingMinSeverity = Debug, loggingTracers = Tracers {applicationTracer = Const (Just Info), apiServerTracer = Const (Just Info), tokenMetadataTracer = Const (Just Info), walletEngineTracer = Const (Just Info), walletDbTracer = Const (Just Info), poolsEngineTracer = Const (Just Info), poolsDbTracer = Const (Just Info), ntpClientTracer = Const (Just Info), networkTracer = Const (Just Info)}, loggingTracersDoc = Nothing}} [cardano-wallet.application:Info:4] [2021-07-12 11:14:17.30 UTC] Wallet backend server starting. Using /home/thomasm/Cardano/cardano-node/node-ipc/mainnet/node.socket. [cardano-wallet.application:Info:4] [2021-07-12 11:14:17.30 UTC] Node is Haskell Node on mainnet. [cardano-wallet.token-metadata:Notice:4] [2021-07-12 11:14:17.30 UTC] No token metadata server is configured. [cardano-wallet.token-metadata:Notice:4] [2021-07-12 11:14:17.30 UTC] No token metadata server is configured. [cardano-wallet.token-metadata:Notice:4] [2021-07-12 11:14:17.30 UTC] No token metadata server is configured. [cardano-wallet.token-metadata:Notice:4] [2021-07-12 11:14:17.30 UTC] No token metadata server is configured. [cardano-wallet.main:Info:4] [2021-07-12 11:14:17.31 UTC] Wallet backend server listening on http://<IP_ADDRESS>:8090/ [cardano-wallet.pools-engine:Info:23] [2021-07-12 11:14:17.31 UTC] Chain following starting. Requesting intersection using 0 checkpoints [cardano-wallet.network:Notice:13] [2021-07-12 11:14:17.55 UTC] Couldn't connect to node (x2). Retrying in a bit... [cardano-wallet.network:Notice:15] [2021-07-12 11:14:17.55 UTC] Couldn't connect to node (x2). Retrying in a bit... [cardano-wallet.network:Notice:26] [2021-07-12 11:14:17.56 UTC] Couldn't connect to node (x2). Retrying in a bit... [cardano-wallet.network:Warning:13] [2021-07-12 11:14:17.81 UTC] Couldn't connect to node (x3). Retrying in a bit...
First i thought that maybe the socket-path is wrong, but i use the same path when starting DB-Sync and there it works without problems. A little confusing is, that even if i put a wrong socket-path i don't get any different output (apart of the path). It would be great that i get a message that no socket-path has been found or something similar.
Is there something else i miss or should do to make this work?
Thanks very much,
Thomas

Hi, I believe you need to use https://docs.docker.com/storage/bind-mounts/ to access node.socket that is on your local system, e.g.:
$ docker run --network host --rm \
-v /path/to/your/node.socket/dir/on/local/system:/ipc \
inputoutput/cardano-wallet:2021.6.11 serve --port 8091 --node-socket /ipc/node.socket --mainnet

Have a look at our https://github.com/input-output-hk/cardano-wallet/blob/master/docker-compose.yml which provides cardano-walet and cardano-node docker containers with compatible versions. You can start wallet + node from there on mainnet too:
wget https://raw.githubusercontent.com/input-output-hk/cardano-wallet/master/docker-compose.yml
NETWORK=mainnet docker-compose up


For discussion 2022-08-29

Links in markdown (http addresses versus file system addresses).

Discussed.


Add pipeline action to ensure a Go dependency is at least at a specified version

Problem
In order to fix vulnerabilities in a Go project, we've been inserting imperative commands into a package's Melange pipeline like this (taken from here):
pipeline:
  # ...

  - runs: |
      # ...

      # Mitigate GHSA-vvpx-j8f3-3w6h
      go get<EMAIL_ADDRESS>
This addresses the vulnerability, but there are a handful of issues with this approach:

Once the upstream project is using a version of the Go module that's later than the specified version (v0.7.0 in this example), this command will start downgrading project dependencies, thereby potentially introducing vulnerabilities and breaking behavior for end users.
Removing this snippet from a package config is manual and error-prone.

Proposed solution
It'd be great to have a declarative and reusable pipeline action to handle this scenario. I'm envisioning something like this:
pipeline:
  # ...

  - uses: go/set-module-minimum-version
    with:
      project-dir: /<...>/terraform
      dependency: golang.org/x/net
      minimum-version: v0.7.0

The behavior here would be improved: this pipeline action would determine if the Go project is currently using a version of the specified dependency below the specified minimum-version, and if so, it would do the go get .... Otherwise, it would leave the dependency version as is (i.e. at or above this version).
Further, this would allow us to add functionality in the future to programmatically determine which of these actions were no longer needed, and we could programmatically remove these YAML blocks to keep the config clean.

N.B.: This kind of pipeline action may be beneficial to other ecosystems beyond Go.

This would also get us closer to automated remediation workflows where we can programmatically insert this YAML block into existing configs.

I'm new to melange so maybe I might have missed something, but I would have expected support for versioned package dependencies out of the box.
We're considering using melange to package all our projects into apks and push them into a private registry. Other projects would reference other packages together with a distinct semantic version that we can track automatically using renovate.
As for the proposed solution above, why not just use what is common practice in other languages like for instance rust/cargo or javascript/npm where you express compatible versions with *,~ or ^ syntax together with the package name?

I think we might be on different pages. This issue isn't about apk versions, or which versions of apks are suitable as dependencies for other apks. This issue is talking about language ecosystem dependencies (specifically, Go modules), which are not packaged by Melange, and how to provide a sane way to manipulate the version of a Go module referenced by the Go project we're building in Melange.

I understand that this is about go modules, but I was thinking that the ecosystem could use one approach and get versioned package dependencies kind of 'for free'. Managing versions for go modules can't be that different from apk versions, or am I wrong?
Anyway, we would need versioned dependencies. Should I open another issue?
thx

Yeah I think an issue for this would help! It'd be great to better understand the perceived benefits of the idea and how it might work in practice. Thanks!

Ref: https://github.com/chainguard-dev/melange/commit/ece24c2854a2956bf48c539b312cde654765e6d5#diff-e9720019c7b001a10aa87ca89ef44a50a3942cfb22ea8c16ea3dd009d9992a0c

Nice!


Add missing onPartnerAdImpression

According to this forum post from Unity, the ad impression is tracked on their end when the ad is played: https://forum.unity.com/threads/ad-request-and-impressions.540565/#post-5832385


Optional Ability Properties

This PR allows for the nextAbilityResult to have an optional message parameter.
Acceptance Testing is also added to account for this feature.

closes #118


📖 DOC: Improvement

📖 DOC: Comment added for onDateChange in the example code.

Coverage remained the same at 97.368% when pulling f8aa250cdba60af18649040d7147cde237fb3d35 on waahab:master into 2a209bd4f11a84ff4543fbde930fed60dcbc0f68 on xgfe:master.


Using slots singleLabel & Vue 3

Slot singleLablel in vue 3 not working!!!
when is a fix coming?
thank you

Duplicates #1776


[GEP-18] Adapt kubeconfig secrets (gardener{-internal}, kube-proxy)

How to categorize this PR?

/area security
/kind enhancement
/merge squash
What this PR does / why we need it:
This PR adapts the kubeconfig secrets (gardener{-internal}, kube-proxy) to contain the entire CA bundle.
Which issue(s) this PR fixes:
Part of https://github.com/gardener/gardener/issues/3292
/invite @timebertt @BeckerMax
Release note:

NONE

/retest

/assign


Correcting Chinese translation errors in Qt

modify 狗 which in chinese means dog to 佩 which in chinese means pe

I tested this and it works fine. I checked several translations and they all seem sensible (though I don't read Chinese). I think this should target a new development branch; once it's reparented, I think this is safe to merge.
However, it would be sensible to get native Chinese speaker review on the text.

im a native Chinese speaker


[docs] add return value from actions cleos output explanation and samples

[docs] add return value from actions cleos output explanation and samples in cleos reference documentation

The rtmp action output, returning a default tmp object, output will look like this:
-to-
The rtmp action returns a default tmp object as json:

When it will not be possible to decode the return value, for example when ABI doesn’t contain information about an action return value, output will look like this:

to -
The rtmp action output when no ABI is available to decode the return value will look like this:

If the length of the return value string is more than 100 chars the string will be truncated to the first 100 chars and three dots ... will be added at the end. It's similar to what cleos does to action parameters output.
-to -
If the length of the return value string is more than 100 chars the string will be truncated to the first 100 chars and three dots ... will be added at the end.


Attribute validation rules

Lunar version: ^1.0.0-alpha
Laravel Version: ^11.0
PHP Version: ^8.2
Database Driver & Version: Using Laravel Sail image mysql/mysql-server:8.0

Expected Behaviour:
In 0.8 it was possible to add a validation rule to an attribute.
The same should be possbile in 1.0.0-alpha
Actual Behaviour:
Right now it doesn't use the rule when using the attribute.
Steps To Reproduce:

A fresh Lunar 1.0.0-alpha installation.
Create an attribute group eg. "SEO".
Create an attribute on the group eg. "Meta description" with the rule max:50
Use the attribute on a product and save the product. (Should validate using max:50 but it doesn't)

Example of 1.0.0-alpha rule setup

Example of 0.8 rule setup

Example of 0.8 validation

@ThorNissen does this https://github.com/lunarphp/lunar/pull/1798 fix for you?

@ThorNissen does this #1798 fix for you?

I'm currently not able to give it a test.
But looking at the PR it seems to do the job 👍🏻


Converting BigNumber into int256

Hi there,
Assuming, the code can inspect big number, and if ensured it fits into int256 - how to covert it?
I didn't find any example.

You can just use     abi.encodePacked(bn);
I added this to my version of the library for debugging purposes, but it's not necessay.
function toBytes(BigNumber memory n) public pure returns(bytes memory r){
return n.val;
}

Hi there,
Assuming, the code can inspect big number, and if ensured it fits into int256 - how to covert it?
I didn't find any example.

Hi, the conversion functions are indeed lacking. I will take a look at adding more, as it seems there is some demand for this.