Ensure the command size from the client does not exceed the cmdBuffer…

… capacity.
This resolves a buffer overflow that is the result of trusting the
size of the TPM command provided by the client in the TpmCmdServer
thread. The resourcemgr recv's the size as a uint32, and then blindly
reads however many bytes the user specified into the cmdBuffer.
The cmdBuffer in the resourcemgr is a fixed size determined by the
TPM (through a call to GetCapability). This is a classic OWASP "top
10" data validation bug:
https://www.owasp.org/index.php/Data_Validation
We resolve this by comparing the size of the command provided by the
client to the maxCmdSize variable. At the time of initialization,
the resourcemgr queries the TPM for the maximum command buffer size
that it supports and sets maxCmdSize to this value. If the client
attempts to send a command buffer larger than this size, the
resourcemgr will now send a response buffer indicating that
the buffer available is insufficient:
TSS2_TCTI_RC_INSUFFICIENT_BUFFER.
This resolves #291
Signed-off-by: Philip Tricca<EMAIL_ADDRESS>

@gwei3: can I get a quick review of this when you get a chance? I can provide example exploit code if you'd like to repro.

It is a obvious bug, and the fix looks good.


Updating the Analytics Rules and Hunting Queries for Azure Firewall Solution to support Resource Specific logs.

Required items, please complete
Change(s):

Updated the below Hunting Queries to support Resource Specific Logs:
- Azure Firewall - First Time Source IP to Destination Using Port.yml
- Azure Firewall - First time source IP to Destination.yml
- Azure Firewall - Source IP Abnormally Connects to Multiple Destinations.yml
- Azure Firewall - Uncommon Port for Organization.yml
- Azure Firewall - Uncommon Port to IP
Updated the below Analytic Rules to support Resource Specific Logs:
-  Azure Firewall - Abnormal Deny Rate for Source IP.yml
-  Azure Firewall - Abnormal Port to Protocol.yml
-  Azure Firewall - Multiple Sources Affected by the Same TI Destination.yml
-  Azure Firewall - Port Scan.yml
-  Azure Firewall - Port Sweep.yml
-  SeveralDenyActionsRegistered.yml
Added a new workbook for Azure Firewall Solution to support Resource Specific Logs

Reason for Change(s):

To support the resource specific logs

Version Updated:

Yes

Testing Completed:

Yes

Checked that the validations are passing and have addressed any issues that are present:

Yes

@v-atulyadav Metadata has been added for both the templates. Please help merging this PR. Thank you.

Hi @shabaz-github, please take a look at @devikamehra's comments. Thanks

Hi @devikamehra, please check fixes from @shabaz-github. Thanks


[stable/dokuwiki] Improve notes to access deployed services

What this PR does / why we need it:
As described on https://github.com/kubernetes/kubernetes/pull/59809, now kubectl port-foward resolves service port to target port. Therefore, we can access the services deployed as 'ClusterIP' using a simple port-forward.
This PR improves the NOTES.txt so this new method is the one recommended to access the application when using 'ClusterIP'

/ok-to-test
/lgtm


Add freetype to brew install pre-reqs

When running pip install, it error-ed out because I didn't have FreeType installed. This was the specific message:
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DFREETYPE_BUILD_TYPE=system -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_ft2font_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D__STDC_FORMAT_MACROS=1 -Iextern/agg24-svn/include -I/Users/brandonhigh/.asdf/installs/python/3.9.0/lib/python3.9/site-packages/numpy/core/include -I/Users/brandonhigh/.asdf/installs/python/3.9.0/include/python3.9 -c src/checkdep_freetype2.c -o build/temp.macosx-10.15-x86_64-3.9/src/checkdep_freetype2.o
    src/checkdep_freetype2.c:3:6: error: "FreeType version 2.3 or higher is required. You may set the MPLLOCALFREETYPE environment variable to 1 to let Matplotlib download it."
        #error "FreeType version 2.3 or higher is required. \
         ^
    src/checkdep_freetype2.c:9:10: fatal error: 'ft2build.h' file not found
    #include <ft2build.h>
             ^~~~~~~~~~~~
    2 errors generated.
    error: command '/usr/bin/clang' failed with exit code 1

After brew install freetype, I was able to pip install with no issues.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Could this PR be somehow related to this issue -> https://github.com/duo-labs/cloudmapper/issues/803 ?

@w0rmr1d3r I believe so.

LGTM, sorry for the delay.


Fix bug to correctly build Packet-Out message for OpenFlow 1.5

@rizard
@geddings
Problem:
While building out Packet-Out message in OpenFlow 1.5 environment (i.e., BBDP/LLDP packet-out message), those Packet-Out messages are not built correctly.
Solution:
This is because "in_port" attribute was replaced with a "match" in OpenFlow 1.5, therefore we should use OFMessageUtils.setInPort to correctly build up packet-out messages.

Looks good!

Thanks for the review!


reindex process slows after some hours when Postgres vacuuming falls behind

After some time, queries against LOGICAL_RESOURCES slow down. After a couple of hours, the reindex status update on this table ends up dominating the overall processing time. It turns out this is a byproduct of doing so many updates to this table and the way Postgres handles those updates. The update leaves behind dead records which need to be cleaned up by the vacuum process. This is supposed to be run in the background but it looks like the automatic vacuum can't keep up.
After stopping the reindex and run a manual vacuum the performance recovers (from 30s per call to 10s):
VACUUM (ANALYZE,VERBOSE) fhirdata.logical_resources;

After restarting reindex, the time slowly creeps up again, so this has to be repeated.
Investigate other solutions which avoid the need to perform an update against an indexed column. One possible solution is to copy all the rows needing to be processed into a working queue table and delete each row from there as it is processed. This table probably does not even require an index, because processing can be done in row order (somewhat arbitrary, but that is OK).

options:

make vacuuming more aggressive
manual (code-driven vs db-driven) vacuuming
other?

need investigation as too which is best

The LOGICAL_RESOURCES table will still be updated as part of reindex after #2524 but we don't expect it to cause the same contention as what we were seeing from updating the tstamp.

After trying to reindex a large postgres db using the new client-driven reindex behavior, I don't think this one is fully addressed.
It ran for over a day, but after some amount of time we ran out of disk space.  I don't know if it was vacuuming related or just that the space was needed for the new tables/indices in 4.9.0.
However, after increasing disk capacity, client-driven reindex repeatedly failed with error like:
2021-07-11 22:37:13.879 fhir-test-server-86b656757d-9drdx fhir-test-server SEVERE inserting parameters
org.postgresql.util.PSQLException: ERROR: multixact "members" limit exceeded
  Detail: This command would create a multixact with 2 members, but the remaining space is only enough for 1 member.
  Hint: Execute a database-wide VACUUM in database with OID 16478 with reduced vacuum_multixact_freeze_min_age and vacuum_multixact_freeze_table_age settings.
  Where: SQL statement "SELECT 1 FROM ONLY "fhirdata"."parameter_names" x WHERE "parameter_name_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"

I found the following command from the postgresql forums for finding tables with the oldest xids:
SELECT oid::regclass, relminmxid, mxid_age(relminmxid) FROM pg_class WHERE relminmxid <> '0' ORDER BY mxid_age(relminmxid) DESC;

Our observation_resource_token_refs was on the top of this list and so I performed a manual vacuum of that table:
vacuum fhirdata.observation_resource_token_refs
The next morning I noticed that the reindex response times were much faster.

Discussed with Robin and we think that we can further mitigate this issue by setting the properties mentioned at https://ibm.github.io/FHIR/guides/FHIRPerformanceGuide#412-tuning-auto-vacuum on the postgresql tables.

PR https://github.com/IBM/FHIR/pull/2628

Ran the latest reindex of 79 million resources, and it finished without error in 37 hours. this was with the latest fixes for vacuum settings. I am closing this issue based on last run.


Assigning User object to changed_by attribute gives Error

models.py
class BaseModel(models.Model):
    createdAt = models.DateTimeField(default=timezone.now)
    id = models.CharField(primary_key=True, max_length=50, default=string_uuid)

    class Meta:
        abstract = True

class User(BaseModel):
    firstName = models.CharField(max_length=200)

class UserTrackMixin(models.Model):

    changed_by = models.ForeignKey(User, null=True, on_delete=models.DO_NOTHING)

    @property
    def _history_user(self):
        return self.changed_by

    @_history_user.setter
    def _history_user(self, value):
        self.changed_by = value

    class Meta:
        abstract = True

class Service(BaseModel, UserTrackMixin):
    name = models.CharField(max_length=200)
    serviceType = models.CharField(max_length=200)

    history = HistoricalRecords()

I am trying to associate user changing the service object using the way proposed here
But, when i am trying to assign user object to the attribute changed_by to Service object it gives error as mentioned below.
u1=User.objects.first()
<User: User object (07ab8565-26c7-4bfa-a92e-e12bf924160a)>
s._history_user = u1
s.save()

ValueError: Cannot assign "<User: User object (07ab8565-26c7-4bfa-a92e-e12bf924160a)>": "HistoricalService.history_user" must be a "User" instance.

I think i found the issue.
The issue was with my code.
I didn't have the variable AUTH_USER_MODEL defined in the settings file due to which history assumes 'auth.User' as the default user as can be seen here which is not the user model i am using as i have defined my custom user model.


As a user, I want to be able to edit my patient's particulars when their situation changes

All attributes of the Patient class can now be edited by edit commands. In the future, we can consider to make certain particulars uneditable (i.e. NRIC). This issue is closed.


Update order of import statements to reflect code

It's small, but the docs currently depict the mysite/urls.py import statements in the opposite order of how they appear in the code.
As this appears to be a trivial change, I have not created a corresponding Trac ticket. Thank you!

@timgraham As a docs change, should I backport this to stable/2.0.x? (Thanks!)

@carltongibson I love isort! In this case I didn't use it, but did suspect that of the two import orders, that in the code (alphabetical) was likely the one to follow. Thanks!


Add Better ServiceType support to MariaDB

This commit adds logic to better support the NodePort and LoadBalancer
service types. Annotations can be used by the LoadBalancer service to
make the load balancer internal only. The nodePort parameter can be used
to specify the exact port that NodePort (and LoadBalancer) uses.

/ok-to-test
/lgtm
thanks @apeschel

@prydonius It's not clear to me what the failure is here, do you have any additional insight?

@apeschel @prydonius Here's the story.... it seems when the chart was updated to version 2.0.1 (or 2.x in general) the tests were not passing. Not sure what happened but all mariadb updates fail now due to test issues.

@apeschel would you mind rebasing off of master to pickup the fixed tests and making sure to increment the chart version.

@mattfarina Updated

thanks @apeschel!


Driver Analysis

Model 1: Recovery Years

[x] model fit
[x] sensitivity analysis
[x] sketch figures

Model 2: greater than 80% recovered

[x] model fit
[x] sensitivity analysis
[x] sketch figures

We need to do a sensitivity analysis on the NDVI model to better explain how current NDVI reflects past fire history. The the driver analysis focuses on the recovery period.


[Build] Functional tests intermittently fail with OOM

Running .\build fails on the project with the following stack trace:
      System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
      Stack Trace:
            at System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb)
            at System.Reflection.Internal.NativeHeapMemoryBlock..ctor(Int32 size)
            at System.Reflection.Internal.StreamMemoryBlockProvider.ReadMemoryBlockNoLock(Stream stream, Boolean isFileS
tream, Int64 start, Int32 size)
            at System.Reflection.PortableExecutable.PEReader..ctor(Stream peStream, PEStreamOptions options, Nullable`1
sizeOpt)
            at Microsoft.CodeAnalysis.ModuleMetadata.CreateFromStream(Stream peStream, PEStreamOptions options)
         D:\work\katana\mvc\src\Microsoft.AspNet.Mvc.Razor\Compilation\RoslynCompilationService.cs(191,0): at Microsoft.
AspNet.Mvc.Razor.RoslynCompilationService.<>c__DisplayClass3.<CreateMetadataFileReference>b__4(String _)
            at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
         D:\work\katana\mvc\src\Microsoft.AspNet.Mvc.Razor\Compilation\RoslynCompilationService.cs(187,0): at Microsoft.
AspNet.Mvc.Razor.RoslynCompilationService.CreateMetadataFileReference(String path)
         D:\work\katana\mvc\src\Microsoft.AspNet.Mvc.Razor\Compilation\RoslynCompilationService.cs(168,0): at Microsoft.
AspNet.Mvc.Razor.RoslynCompilationService.ConvertMetadataReference(IMetadataReference metadataReference)
         D:\work\katana\mvc\src\Microsoft.AspNet.Mvc.Razor\Compilation\RoslynCompilationService.cs(142,0): at Microsoft.
AspNet.Mvc.Razor.RoslynCompilationService.GetApplicationReferences()
            at System.Lazy`1.CreateValue()
            at System.Lazy`1.LazyInitValue()
            at System.Lazy`1.get_Value()

Requires investigating to identity if this is a failure due to the way functional tests are set up or if it's something that could also affect runtime.

@pranavkm says you have a PR coming for this

@pranavkm did a94d5223e1748b24a023fffbb8852066a028f40f fix this issue?


WebExtension Addon

It is used to transfer links & cookies from browser, since WebExtension is used in both Firefox & Chrome.
API docs:
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Native_messaging

@Rictusempra Thanks for your advice. Browser extension for capturing downloads is on my to-do list. I plan to write extension as long as I have completed main features of Photon. For now, you can try capturer extension for aria2 like Aria2-Integration and chrome-aria2-integration, since these extensions are also Photon compatible.

@alanzhangzm
Just a hint: Opera & Edge also use webext with little differences


Create table unique index parameters problem fixed

          did     int,
          name    varchar(40),
          UNIQUE (name) WITH (fillfactor=70)
        )
        WITH (fillfactor=70);

Failure/Error: it { is_expected.to eq oneline_query }
   expected: "CREATE TABLE distributors (did int, name varchar(40), UNIQUE (name) WITH (fillfactor=70)) WITH (fillfactor=70)"
        got: "CREATE TABLE distributors (did int, name varchar(40), UNIQUE (name)) WITH (fillfactor=70)"

@emin100 Thanks for the contribution!
For deparser changes, we would actually need to add this in the C library (https://github.com/pganalyze/libpg_query), and then update the source here. Do you want to open a PR there? You can add the test case to the deparse_tests.c file.

I sent PR to libpg_query.
PR: https://github.com/pganalyze/libpg_query/pull/193

@lfittl I opened a new PR to libpg_query repo.
PR: https://github.com/pganalyze/libpg_query/pull/193

fixed another deparse problem
SQL: SHOW ALL
expected: "SHOW ALL"
got: "SHOW SESSION ALL"
libpg_query PR: https://github.com/pganalyze/libpg_query/pull/193

SQL:
CREATE PROCEDURE insert_data(a integer, b integer)
LANGUAGE SQL
BEGIN ATOMIC
  INSERT INTO tbl VALUES (a);
  INSERT INTO tbl VALUES (b);
END;

ERROR for "CREATE PROCEDURE insert_data(a int, b int) LANGUAGE sql BEGIN ATOMIC INSERT INTO tbl VALUES (a); INSERT INTO tbl VALUES (b); END"
  deparse: unpermitted node type in a_expr/b_expr: 231

libpg_query PR:  https://github.com/pganalyze/libpg_query/pull/195

From the history here it looks like this has been fixed in libpg_query, so I'll close this.


Added B31 ID and texture lists, as well as a level.dat file

These should be helpful. What exactly do they do?

Actually, these aren't correctly generated. Let me remove this and I'll do it with a proper commit


Add log to report final count of triples

RDFReader reports ingested triples in multiples of 10000 which could be confusing for user.
The fix is to report on the final count of ingested triples at the end.
Results look like following:

20/07/30 11:07:29 INFO contentpump.RDFReader: Ingested 454 triples from file:///.../semantics.rdf
20/07/30 11:07:29 INFO contentpump.RDFReader: Ingested 10000 triples from file:///.../semantics.xml
20/07/30 11:07:30 INFO contentpump.LocalJobRunner:  completed 100%
20/07/30 11:07:31 INFO contentpump.RDFReader: Ingested 17481 triples from file:///.../semantics.xml
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: INPUT_RECORDS: 180
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 180
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 180
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 0
20/07/30 11:07:31 INFO contentpump.LocalJobRunner: Total execution time: 2 sec

The PR looks good. Please add bug number when merging the change.
Thanks!

unit tests passed. regression look fine.


export fails if file has dots

Hello everyone,
I've crossed with a weird issue. When trying to export (to kappa) the file setting01.plasmid.without.negativefeedback.py pysb.export fails with
Error in model script:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/pysb/export/__main__.py", line 53, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python3.6/dist-packages/pysb/export/__main__.py", line 37, in main
    model_module = __import__(model_name)
ModuleNotFoundError: No module named 'setting01'

while the renamed file setting01_plasmid_without_negativefeedback.py is exported without problems. The problem arises in python2 and 3.
Thank you in advance if you solve this small issue

I think the general issue (not related to PySB) is that if you have dots in your filename and then try to import from it. Python interprets the dots as submodule separators, hence the error: ModuleNotFoundError: No module named 'setting01' which is the part before the first dot.

Yes, it's generally not advised to use dots in filenames for python code for this reason. How have you been importing this model from your other scripts? I suspect that this would not work either:
from setting01.plasmid.without.negativefeedback import model

and this is a Python issue, not a PySB one.


Docs.MS Release Updates for azure-resourcemanager

Update docs metadata and targeting for release of azure-resourcemanager

Docs Build status updates of commit 11a21fd:
:clock10: Full build: the average full build time is 279 min(s) 9 sec(s), this is based on the last 923 build(s) for this repository.

Docs Build status updates of commit 11a21fd:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-services/resourcemanager-readme.md
:white_check_mark:Succeeded
View (azure-java-stable) View (azure-java-preview) View (azure-java-legacy)

package.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Add Atlas Launch Vehicle Pack [Launchers Pack] from Kerbal Stuff

This pull request was automatically generated by Kerbal Stuff on behalf of Kartoffelkuchen, to add Atlas Launch Vehicle Pack [Launchers Pack] to CKAN.
Please direct questions about this pull request to Kartoffelkuchen.

This mod has a dependency on Bob Fitch's Old School Fairings (http://forum.kerbalspaceprogram.com/index.php?/topic/119171-105-old-school-fairings/), which has not yet been added to CKAN.


Refactor updates

Move to the new Sensu SDK
Move to Go modules
Move from Travis to GitHub actions
Add options to support secure API connections
Add support for using API key in place of username/password
Closes #2
Closes #4
Closes #5

Looks good, Todd!  I just finished uploading my patches for SSL, API KEY and now I see yours :)
I also added a Curl style "-k" for trusting self-signed certs and a --url to specify https://host:8443 easier than -h host -p 8443 --secure
I'll wait for your pull request to be approved and base a new patch on them.


TI SmartRF Flash Programmer v2 will not flash file CC2652R_coordinator_20230507.hex

have an Electrolama zig-a-zig-ah! using TI SmartRF Flash Programmer v2 to flash this dongle.  Flashing CC2652R_coordinator_20211217.hex works OK.  However flashing CC2652R_coordinator_20230507.hex ended up in a fail
>Initiate access to target: COM4 using 2-pin cJTAG.
>Reading file: C:/_userdata/Raspberry/ZigBee/CC2652R_coordinator_20211217.hex.
>Start flash erase ...
>Erase finished successfully.
>Start flash programming ...
>Programming finished successfully.
>Start flash verify ...
>Skip verification of unassigned page: 21.
>Skip verification of unassigned page: 22.
>Skip verification of unassigned page: 23.
>Skip verification of unassigned page: 24.
>Skip verification of unassigned page: 25.
>Skip verification of unassigned page: 26.
>Skip verification of unassigned page: 27.
>Skip verification of unassigned page: 28.
>Skip verification of unassigned page: 29.
>Skip verification of unassigned page: 30.
>Skip verification of unassigned page: 31.
>Skip verification of unassigned page: 32.
>Skip verification of unassigned page: 33.
>Skip verification of unassigned page: 34.
>Skip verification of unassigned page: 35.
>Skip verification of unassigned page: 36.
>Skip verification of unassigned page: 37.
>Skip verification of unassigned page: 38.
>Skip verification of unassigned page: 39.
>Skip verification of unassigned page: 40.
>Skip verification of unassigned page: 41.
>Skip verification of unassigned page: 42.
>Page: 0 verified OK.
>Page: 1 verified OK.
>Page: 2 verified OK.
>Page: 3 verified OK.
>Page: 4 verified OK.
>Page: 5 verified OK.
>Page: 6 verified OK.
>Page: 7 verified OK.
>Page: 8 verified OK.
>Page: 9 verified OK.
>Page: 10 verified OK.
>Page: 11 verified OK.
>Page: 12 verified OK.
>Page: 13 verified OK.
>Page: 14 verified OK.
>Page: 15 verified OK.
>Page: 16 verified OK.
>Page: 17 verified OK.
>Page: 18 verified OK.
>Page: 19 verified OK.
>Page: 20 verified OK.
>Page: 43 verified OK.
>Verification finished successfully.
>Reset target ...
>Reset of target successful.

>Initiate access to target: COM4 using 2-pin cJTAG.
>Reading file: C:/_userdata/Raspberry/ZigBee/CC2652R_coordinator_20230507.hex.
>Unknown record type: 3.
>Reset target ...
>Reset of target successful.

What can I do to update my zzh!

Exactly the same here!

Solution


chore(master): release 0.15.0-alpha

:robot: I have created a release beep boop
0.15.0-alpha (2024-01-17)
Features

new (c105ce1)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/mafqla/yalis-blog/releases/tag/v0.15.0-alpha :sunflower:


Second example in README not working

I'm trying out altair-transform, and when I run the second example, I still get the MaxRowsError even with transform_chart(chart). Running extract_data(chart) doesn't work either; it gives me the input data unchanged.
Strangely, the first example works: extract_data(line) returns the aggregated dataframe.
altair v3.2.0
altair-transform v0.1.0

Sorry for the confusion: the second example only works with the master branch of the package. The package is under active development, so the released version is missing a lot of features.

Gotcha. Thanks!


Fixtures shouldn't allow shift+enter to create new paragraphs

I want to create single only editable element.
I already set ContentEdit.PREFER_LINE_BREAKS to true;
But if user press Shift+Enter, he can anyway create another block, wrapped into p tag.
It breaks selections, it requires a p tag styling etc.
How can i make Shift+Enter acts same as Enter? Or completely disable it?

@koutsenko it sounds like you want to use fixtures:
<div
    data-fixture
    data-ce-tag="p"
    data-name="snippet:{{ snippet.id }}:heading"
    class="snippet-cta__heading"
    >
    Enter content...
</div>

@anthonyjb , sorry for delay. I tried to add data-fixture attribute both to parent div and content p tag, and added test function to init method. Still can add new p tag via shift+enter. Where can i read docs about fixtures? Thanks.

shift+enter don't add a new paragraph actually, but it create a line break ().
To me, it sound as a normal behavior, (it's possible you want to use line break in a fixture).
If you want to prevent line break in fixture, you could do something like that in CSS :
[data-fixture] br { display:none; }

Should do the job


Add VM as provision source for VM Template

@rawagner, I guess this is rightfully rotten. Please re-open otherwise.

no, we still want to be able to 'convert' VM to VM template
/remove-lifecycle rotten
/reopen


Incorrect/Striped output on ESP32-S3 with SIMD when decoding 4:2:0 images

Thanks for this amazing jpeg decode lib.
I encountered some issues while trying to use SIMD to decode JPEG images on the ESP32-S3.
Without SIMD:

With SIMD:

Code snippet (I'm using RGB888)
  Jpeg.openFLASH((uint8_t *)src, data_length, NULL);
  Jpeg.setFramebuffer(buffer);
  Jpeg.setPixelType(RGB8888);
  int code = Jpeg.decode(0, 0, 0);
  if (code == 0) {
      LOGE(TAG, "JPEG decode error: %d", Jpeg.getLastError());
      return NULL;
  }
  Jpeg.close();

Based on my tests, this issue only occurs with RGB888, while RGB565 works fine.

I tested with a solid red image and printed out the pixel values of the first row. Here's what I got:
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FE 00 00 FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF
FF 00 4A FF

As you can see, every 8 pixels in each row, there are 8 incorrect values.
I suspect there might be an issue with this assembly code.
.rgba8888_output:          # generate RGBA 32-bit pixels
  movi.n a9,8       # shift 8 bits
  wsr.sar a9
  ee.vsl.32 q3,q3   # shift green over 8 bits
  ee.orq q7,q7,q3   # combine red and green
  ee.vcmp.eq.s16 q1,q1,q1  # create FFs
  ee.xorq q2,q2,q2  # create 00s
  ee.vzip.8 q2,q1   # create FF00 for Alpha
  ee.orq q2,q2,q6   # combine blue + alpha
  ee.vzip.16 q7,q2  # create RGB8888 pixels
  ee.vst.128.ip q7,a5,16  # store 8 x RGB8888 pixels = 32 bytes
  ee.vst.128.ip q2,a5,16
  addi.n a6,a6,-16         # restore pointer to start of 16-bit constants
  ee.vld.l.64.ip    q0,a2,0   # load right 8 Y values into Q0
  ee.movi.32.q q1,a3,0        # restore second 4 values of Cb
  ee.movi.32.q q2,a4,0        # restore second 4 values of Cr
  addi.n a8,a8,-1
  bnez.n a8,.convert_420_loop
  retw.n

Here's my complete test code:
#include <M5Unified.h>
#include <Arduino.h>
#include "JPEGDEC.h"
#include "testjpg.cc"

extern "C" void app_main()
{
	delay(2000);

    M5.begin();
	M5.Lcd.fillScreen(GREEN);

	Jpeg.openFLASH((uint8_t *)test, sizeof(test), NULL);
	printf("JPEG Image width: %d, height: %d\n", Jpeg.getWidth(), Jpeg.getHeight());

	uint8_t* buffer = (uint8_t*)heap_caps_aligned_alloc(2, Jpeg.getWidth() * (Jpeg.getHeight() + 2) * 4, MALLOC_CAP_8BIT | MALLOC_CAP_INTERNAL);

	Jpeg.setFramebuffer(buffer);
	Jpeg.setPixelType(RGB8888);
	int code = Jpeg.decode(0, 0, 0);
	if (code == 0) {
        printf("JPEG decode error: %d\n", Jpeg.getLastError());
        return;
    }
	Jpeg.close();

	for (int i = 0; i < 128; i++) {
		printf("%02X ", buffer[i]);
		if ((i+1) % 4 == 0) {
			printf("\n");
		}
	}
	printf("\n");

	for (int y = 0; y < Jpeg.getHeight(); y++) {
		for (int x = 0; x < Jpeg.getWidth(); x++) {
			uint8_t *pixel = &buffer[(y * Jpeg.getWidth() + x) * 4];
			M5.Lcd.drawPixel(x, y, M5.Lcd.color565(pixel[0], pixel[1], pixel[2]));
		}
	}
	// M5.Lcd.pushImage(0, 0, Jpeg.getWidth(), Jpeg.getHeight(), (uint16_t*)buffer);
}

Thank you for discovering this bug. I found and corrected the error. The new code is pushed the repo (not released yet). Please let me know you have confirmed the fix.

Thank you for the quick fix, after testing, the new code works well :)
I have another small question, if the buffer size for setFramebuffer is wh4, it causes out-of-bounds memory access.
I took a quick look at the source code and want to confirm one thing: does the image height have to be a multiple of 8?

If you're using the S3, then the buffer needs to be a multiple of 16 bytes. For maximum speed, I write a whole Q register at a time (16 bytes).

Sorry, there was an oversight in the previous test. I tried a different image with content (not solid color), and I found that the left side is decoded incorrectly, like this:

Test image:

This is due to the image not being a multiple of 8 pixels wide. I can either force the output to have space for the extra pixels (widen the pitch) or not use the S3 SIMD in that case. Which do you prefer?

I prefer to widen th pitch.
But, the width of the second image is 200px, which is a multiple of 8, the result is still incorrect.

I should have said "MCU width" and not multiple of 8. For images with horizontal color subsampling, the MCU width is 16 and so any fraction above will cause the rest of the MCU to be written.


fix(remix-dev): omit Resource Routes from client build

If resource routes don't have any client-safe exports, the entire thing gets included as a browser entry point (export * from ${file}). This can expose sensitive information.
Fix is to only include routes as entry points that have at least one browser safe export.
Fixes #1828

This is great, thank you! Let's add a test into integration/compiler.ts for this and get it merged.
In the test you have access to the internal build.assets and could inspect them to ensure the resource route is not included.
I'll add if you're not up for it :) Just lmk.

Ah I forgot about a case here. Sometimes you want to add a route that just redirects, like a "/calendar" route. Maybe you want that to just redirect to "today"
export function loader() {
  return redirect(`/calendar/${getTodayParam()}`);
}

By removing the route completely Remix in the browser can't know it should be calling a loader that will then redirect. If it's a normal anchor then it all works fine because the code is called on the server and the redirect is sent.
For typical Resource Routes, none of this is a problem because you don't <Link> to them (except like a PDF but we already require apps to use <Link reloadDocument> for those).
So we have a few choices here:

Require apps to export default function() { return null } to signify that this is not a resource route. If you <Link> to a route, it's part of the UI so you must export a UI component to gets included in the client bundles.

This is nice because it's explicit. You link to UI routes, so you export components for UI routes. The definition for a resource route is "not part of the UI" so it lines up nicely
<Link prefetch> will work
This is annoying because you have to kinda know more about Remix than I'd like people to have to know.

When a <Link> is clicked to a route that isn't in the route manifest in the browser, do a normal browser navigation instead of a Remix navigation.

This is nice because I think it'll generally Just Work™.
It's a bummer that client state will be lost (scroll positions, focus, react state, etc)
We'd probably need to do some more defensive stuff with <Link prefetch>, today I think it explodes (we should probably do this work anyway though)

Include resource routes in the manifest but exclude them from the bundles. This will enable a link click to call the loader and get the redirect.

This is nice because it will generally Just Work™
You won't lose client state
The problem here is what do we do when the loader doesn't redirect, but returns a PDF or something and we called fetch on it expecting JSON.

Again, this isn't a general resource route concern, the only case we're concerned about is linking to routes that exist only to redirect elsewhere--which is actually quite common.
I think I want to do (1), but with some nice error messaging. When a link is clicked that's not in the manifest, Remix throws a 404 because nothing matched. We can add a warning to the console about the nuance of "Resource Routes" vs. "UI Routes that just Redirect". In development we could even send a resource route manifest to give an even better message about how the resource route exists so add a default export if it's there to redirect, or add <Link reloadDocument> if it sends a PDF or whatever, but that <Link> on its own doesn't know what to do w/ that route.

To summarize, there are two valid times to <Link> to a resource route:

Actual Resource Route: Link to a downloadable resource like a PDF. In these cases, use a normal <a href> or <Link reloadDocument>
UI Route Redirect: Linking to a route that redirects to some other route dynamically. In this case, you are required to export default function() { return null } to make it part of your UI.

From there, it's a task for Remix to help steer you the right direction when you <Link> to a resource route or don't export a default component for the UI redirect route.

Actually, in development we could warn the app developer to tell Remix the intent of the link:
You linked to a "Resource Route" from your UI, please add `<Link reloadDocument>` for the browser to be able to view or download this resource.

If it's not a resource route, please add a default export to your route so that it gets included in the UI. If it's just there to redirect, use `export default function() { return null }`.

But in production we can do (2) and make it (usually) Just Work™. When we match the routes and find nothing, we do a window.reload. If it's a resource route, the browser will view/download it. If it's a redirect, they'll get redirected. If it doesn't exist, they'll get a real 404.
This way Remix steers app developers to the optimal UX before they deploy (don't lose client state, don't try to fetch PDFs in a client transition expecting JSON), but if the developer misses the messaging, the app will probably do the right thing.

TIL about the subtleties of the route manifest :) I agree that 1) seems like the most pragmatic approach. Let me take a stab at adding that warning to <Link /> and see about getting that test to pass.

I think the old test is wrong and should warn instead. I'm still not totally sure what we want here, but I'm thinking:

For development builds, add resource routes to the manifest.
In Link, when process.env.NODE_ENV === "development" we can matchRoutes and see if it matches anything.

If it matches nothing, console.warn that the route is not found, go add it to app/routes/
If it matches a resource route, console.warn that they're linking to a resource route and should use reloadDocument or add a default export (the error message from my earlier comment)

In production and dev, when we matchRoutes before calling the client side transition, if the route is not found, do a window.reload at the new URL and then they'll either get:

a 404
a redirect from a route w/o a component that redirects
the resource they linked to (like a PDF or whatever)

This way we steer the developer to providing the right semantics to Remix, but fallback to the Right Thing™.
We can only do the Right Thing™ automatically at the cost of losing client state and the benefits of client transitions. So we do the warnings in dev to get the developer to provide the right semantics so that the app doesn't have to make that tradeoff in production.

@rossipedia if you're going to take on this larger scope, we can ditch the tests in the gists-app/tests and do them all in integration/resource-routes-test.ts

Would be happy to! Just need to ramp up a bit on how <Link /> works. Will try to get to it this evening.

@ryanflorence I'm looking into this, and I'm assuming that for this bit:

In production and dev, when we matchRoutes before calling the client side transition,

You're referring to here and here? Just wanna make sure I'm grokking where it needs to go.

First one yes, but case "fetcher" doesn't matter, that code path isn't involved with links or resource routes. Thank you for digging in to this!

@ryanflorence if I'm understanding things correctly, we're basically altering the default "not found" behavior, right? In essence, we're removing the client-side 404 behavior by deferring it to be strictly server handled?
Instead of window.reload after the transition finishes, couldn't we achieve the same by doing a location.href = createPath(location); instead of calling into handleNotFoundNavigation()?
Just wanna make sure my logic is sound before I go ripping into a bunch of tests that have to be updated :)

@ryanflorence ok probably not worth running the workflow since I didn't change anything in the tests, I wanted to mainly just push up what I had that I think is what you outlined above and make sure that the approach is solid before I start fixing up the tests.
The changes were a bit more involved since adding "resource routes" in dev mode meant changing types in a handful of places, but the meat of the new changes are in components.tsx and transition.ts.
Lemme know if you want me to split this up into more granular commits 👍

Lemme know if you want me to split this up into more granular commits

Ok I went ahead and did this anyways :)

@rossipedia Could you please rebase your branch onto latest dev & resolve conflicts?


ENH: support DANDI_ALLOWED_LOCALHOST_URLS env var to point  staging if needed

Ref: https://github.com/dandi/dandi-archive/issues/961
I have decided to not hardcode gui-staging. one because the fix is targetting
specifically dandisets testing, and otherwise AFAIK is not needed elsewhere.
If something wants to upload to staging, they can also adjust their CI to point
to whatever instances they upload to
@jwodder -- do you think this would work if we set env var to point to staging in testing of dandisets?

Why are we even restricting the published URLs in the first place?  Don't we want to eventually support publishing on any arbitrary instance?

Why are we even restricting the published URLs in the first place? Don't we want to eventually support publishing on any arbitrary instance?

well -- my thinking is that even in the bright future of distributed DANDIs, we might want to ensure that uploaded metadata does target that instance instead of some other or central dandiarchive.org. That is why probably it does make sense to restrict

@yarikoptic But it seems likely that people would want to validate metadata of assets from other instances, without knowing the instance URL, and/or against assets from multiple instances in a single Python process, all of which would be harmed by restricting the URL in the schema.

@yarikoptic - this looks good to me (in theory). btw, the cli tests take a long time to run. would be worthwhile taking a closer look.
@jwodder - i think we need to have a clear idea within our team of what we mean by urls, especially what it means for dandi as a project. despite decentralized stores, we may still have a central entity minting DANDI id's (just like dois are minted by particular agencies). we may consider being a registrar of decentralized instances, but we haven't really discussed the details.
on a more practical front, the only thing that can publish with an actual retrievable and followable doi is on the main instance. while the staging instance mints dois that are in the test doi server.
i believe there is a separate issue where we have talked about closing loops completely.

dandiarchive redirector, gui, doi
staging redirector, staging gui, test doi.

the piece that makes this a bit complicated is that there is only a single identifiers.org that resolves DANDI:, and therefore we would also need a different prefix DANDIstaging: (but identifiers.org may not allow us to register this - we will have to check).
perhaps we can attempt to set up these independent loops of dandi with an alteration of dandischema to allow different prefixes.

@satra , hm, not yet sure I would buy into the "DANDI registrars" idea.  I would have preferred more of a "freedom of git" approach toward decentralization. So overall, in the long run I think I would slide with @jwodder in making URLs unrestricted at the level of schema, and rather rely on tests for a specific instance (like e.g. dandi-cli knows with which instance it corresponds with) to ensure consistent URLs being used.
@satra If you feel strong about keeping schema "restrictive" in validation, then let's I guess proceed with this PR if @jwodder could confirm that it would be sufficient for us to resolve the failing tests situation.  But if you start to slide with us in making them less restrictive, then may be let's at once offload URL testing into unittests of dandischema (where/if it knows which instance it talks to) and dandi-cli?

ok, in the last commit I implemented refactoring to "opt-in" to have archive URL "hardcoded".
It will be taken from the environment variable DJANGO_DANDI_WEB_APP_URL
if specified, as it is now set in the teraform configuration:
$> git grep DJANGO_DANDI_WEB_APP_URL
terraform/api.tf:    DJANGO_DANDI_WEB_APP_URL                     = "https://dandiarchive.org"
terraform/staging_pipeline.tf:    DJANGO_DANDI_WEB_APP_URL                     = "https://gui-staging.dandiarchive.org"

so with this change we would get deployed setups ensuring that metadata
does have corresponding to that deployment metadata.
Note that  Resource.repository  is also set based on the
DJANGO_DANDI_WEB_APP_URL environment variable.
It is a datatype for CommonModel.relatedResource . To be generic, I do not
think we even should impose any specific default value there.
@jwodder and @dchiquito -- do you foresee or could test for any problems which could be potentially be brought up with this change?  Would it resolve the https://github.com/dandi/dandi-archive/issues/961 ?

as you can see in the staging terraform, we already have a mismatch in instance type. we should really create at least a redirect from staging.dandiarchive.org --> gui-staging.dandiarchive.org, if not a full redirector service just like production

I don't mind a redirect/tuning up terraform for staging. All those activities wouldn't need changes in this PR AFAIK

@yarikoptic - this PR is fine. it's just that this again seems like a short term patch, not fixing the separation problem. this can be merged so cli can work, but then it would be useful if we actually fixed the set of problems related to this issue, including introducing the idea of an instance prefix in the schema.

Indeed this is not the "optimal", but I think it is more than just a "fix" but rather an enhancement to support multiple instances (#76) and needs more thought!

dandi-cli is failing with obnoxious diff
>       assert data == BareAssetMeta(**data_as_dict)
E       AssertionError: assert BareAsset(id=...sameAs=None)]) == BareAsset(id=...sameAs=None)])
E         Full diff:
E         - BareAsset(id='dandiasset:0b0a1a0b-e3ea-4cf6-be94-e02c830d54be', schemaKey='Asset', schemaVersion='0.6.2', name=None, description=None, contributor=None, about=None, studyTarget=None, license=None, protocol=None, ethicsApproval=None, keywords=['test', 'sample', 'example', 'test-case'], acknowledgement=None, access=[AccessRequirements(id=None, schemaKey='AccessRequirements', status=<AccessType.OpenAccess: 'dandi:OpenAccess'>, contactPoint=None, description=None, embargoedUntil=None)], url=None, repository=AnyHttpUrl('https://dandiarchive.org/', scheme='https', host='dandiarchive.org', tld='org', host_type='domain', path='/'), relatedResource=None, wasGeneratedBy=[Session(id=None, schemaKey='Session', identifier='XYZ789', name='XYZ789', description='Some test data', startDate=datetime.datetime(2020, 8, 31, 15, 58, 28, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=72000))), endDate=None, wasAssociatedWith=None, used=[Equipment(id=None, schemaKey='Equipment', identifier='probe:probe04', name='Ecephys Probe', description=None)])], contentSize=69105, encodingFormat='application/x-nwb', digest={<DigestType.dandi_etag: 'dandi:dandi-etag'>: 'e455839e5ab2fa659861f58a423fd17f-1'}, path='/test/path', dateModified=None, blobDateModified=None, dataType=None, sameAs=None, approach=None, measurementTechnique=None, variableMeasured=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='cell01', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='cell'), assayType=None, anatomy=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='slice02', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='slice'), assayType=None, anatomy=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='tissue03', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='tissuesample'), assayType=None, anatomy=None, wasDerivedFrom=None, wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=[Participant(id=None, schemaKey='Participant', identifier='a1b2c3', altName=None, strain=None, cellLine=None, vendor=None, age=PropertyValue(id=None, schemaKey='PropertyValue', maxValue=None, minValue=None, unitText='ISO-8601 duration', value='P170DT12212S', valueReference=PropertyValue(id=None, schemaKey='PropertyValue', maxValue=None, minValue=None, unitText=None, value='dandi:BirthReference', valueReference=None, propertyID=None), propertyID=None), sex=SexType(id=None, schemaKey='SexType', identifier=AnyHttpUrl('http://purl.obolibrary.org/obo/PATO_0000384', scheme='http', host='purl.obolibrary.org', tld='org', host_type='domain', path='/obo/PATO_0000384'), name='Male'), genotype='Typical', species=SpeciesType(id=None, schemaKey='SpeciesType', identifier=AnyHttpUrl('http://purl.obolibrary.org/obo/NCBITaxon_1234175', scheme='http', host='purl.obolibrary.org', tld='org', host_type='domain', path='/obo/NCBITaxon_1234175'), name='Cyperus bulbosus'), disorder=None, relatedParticipant=None, sameAs=None)])

E         + BareAsset(id='dandiasset:0b0a1a0b-e3ea-4cf6-be94-e02c830d54be', schemaKey='Asset', schemaVersion='0.6.2', name=None, description=None, contributor=None, about=None, studyTarget=None, license=None, protocol=None, ethicsApproval=None, keywords=['test', 'sample', 'example', 'test-case'], acknowledgement=None, access=[AccessRequirements(id=None, schemaKey='AccessRequirements', status=<AccessType.OpenAccess: 'dandi:OpenAccess'>, contactPoint=None, description=None, embargoedUntil=None)], url=None, repository=None, relatedResource=None, wasGeneratedBy=[Session(id=None, schemaKey='Session', identifier='XYZ789', name='XYZ789', description='Some test data', startDate=datetime.datetime(2020, 8, 31, 15, 58, 28, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=72000))), endDate=None, wasAssociatedWith=None, used=[Equipment(id=None, schemaKey='Equipment', identifier='probe:probe04', name='Ecephys Probe', description=None)])], contentSize=69105, encodingFormat='application/x-nwb', digest={<DigestType.dandi_etag: 'dandi:dandi-etag'>: 'e455839e5ab2fa659861f58a423fd17f-1'}, path='/test/path', dateModified=None, blobDateModified=None, dataType=None, sameAs=None, approach=None, measurementTechnique=None, variableMeasured=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='cell01', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='cell'), assayType=None, anatomy=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='slice02', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='slice'), assayType=None, anatomy=None, wasDerivedFrom=[BioSample(id=None, schemaKey='BioSample', identifier='tissue03', sampleType=SampleType(id=None, schemaKey='SampleType', identifier=None, name='tissuesample'), assayType=None, anatomy=None, wasDerivedFrom=None, wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=None, sameAs=None, hasMember=None)], wasAttributedTo=[Participant(id=None, schemaKey='Participant', identifier='a1b2c3', altName=None, strain=None, cellLine=None, vendor=None, age=PropertyValue(id=None, schemaKey='PropertyValue', maxValue=None, minValue=None, unitText='ISO-8601 duration', value='P170DT12212S', valueReference=PropertyValue(id=None, schemaKey='PropertyValue', maxValue=None, minValue=None, unitText=None, value='dandi:BirthReference', valueReference=None, propertyID=None), propertyID=None), sex=SexType(id=None, schemaKey='SexType', identifier=AnyHttpUrl('http://purl.obolibrary.org/obo/PATO_0000384', scheme='http', host='purl.obolibrary.org', tld='org', host_type='domain', path='/obo/PATO_0000384'), name='Male'), genotype='Typical', species=SpeciesType(id=None, schemaKey='SpeciesType', identifier=AnyHttpUrl('http://purl.obolibrary.org/obo/NCBITaxon_1234175', scheme='http', host='purl.obolibrary.org', tld='org', host_type='domain', path='/obo/NCBITaxon_1234175'), name='Cyperus bulbosus'), disorder=None, relatedParticipant=None, sameAs=None)])

in test_metadata2asset due to expected now repository=AnyHttpUrl('https://dandiarchive.org/' -> repository=None which is expected given the change -- by default model doesn't know which repository an asset belongs.
We could in principle "fill it in" by dandi-cli (although we would first need to get staging redirector and provide it in https://github.com/dandi/dandi-cli/blob/HEAD/dandi/consts.py#L116 and then channel all the way into upload) upon upload but most logical IMHO would be to mutate metadata by API server upon receiving it since it is the ultimate location aware of itself. WDYT @satra and @dandi/dandiarchive folks?
I have pushed increment to DANDI_SCHEMA_VERSION (only a patch) and added minor version boost to library (so dandi-cli / server would consciously upgrade to support it) and we could proceed with a merge or should we decide first on what should fill in repository?
note: repository is read-only, but I assume that the readOnly is only for web UI and dandi-cli/api would have no difficulty in changing it, right? ;)

repository is read-only

we strip out many of the readonly pieces on the server side and inject them there. repository is one of those things i believe. so the user cannot override them through a POST/PUT. if repository is not being populated, it should.

repository is read-only

we strip out many of the readonly pieces on the server side and inject them there. repository is one of those things i believe. so the user cannot override them through a POST/PUT. if repository is not being populated, it should.

I see
dandiapi/api/models/version.py:            'repository': settings.DANDI_WEB_APP_URL,

so may be it does, dunno. @dchiquito @AlmightyYakob - could you please check on dandi-archive end if repository is populated? or should we just proceed with the merge and try it out in staging with upgrade of dandischema? ;)

so may be it does, dunno. @dchiquito @AlmightyYakob - could you please check on dandi-archive end if repository is populated? or should we just proceed with the merge and try it out in staging with upgrade of dandischema? ;)

I don't know if I fully understand what you're asking, but both the production and staging instance have the DJANGO_DANDI_WEB_APP_URL environment variable (and thus the DANDI_WEB_APP_URL django setting) set. For the record, those values are set here and here, respectively.

@yarikoptic - yes that line adds the repository field to the metadata.


Update _pointset.py

Update from_charisma

This pull request is duplicated by #617 which has the encoding mocker test issue resolved.  Currently inclined to close this pull request without merging.

Closing due to replacement with #617


K3s becomes unresponsive after flooding mariadb backend with specific SQL query

Environmental Info:
K3s Version:
k3s version v1.24.8+k3s1 (648004e4)
go version go1.18.8
Node(s) CPU architecture, OS, and Version:
Linux rancher01 4.18.0-425.3.1.el8.x86_64 https://github.com/rancher/rancher/issues/1 SMP Wed Nov 9 20:13:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Rocky Linux release 8.7 (Green Obsidian)
Cluster Configuration:
K3s (Rancher) cluster with MariaDB as a datastore
2 node k3s (rancher) nodes with 2 node MariaDB nodes (Active Backup mode) with Load Balancer.
More Info:
a) k3s nodes with 4 CPU and 8GB RAM
b) Rancher version: v2.7.9
c) Installation option (Docker install/Helm Chart): Helm
d) Cluster Type (Local/Downstream):
The cluster type is Downstream/custom, using Rancher k3s hosted on VSPHERE
Describe the bug:
All of a sudden, k3s (rancher) nodes race to OOM with more resource utilization of CPU and RAM. As a workaround, we have increased (step by step in increments) up to 12 CPU and 16GB RAM, and that too didn't help. We tried rebooting the nodes (as a hard reboot) as the only way the nodes will be responding and post-reboot it will stay responsive for a maximum duration of 10 - 15 minutes and then it will go into unresponsive (OOM). The load will peak to the maximum.
a) No changes to the cluster in recent times (apart from upgrading the rancher upgrade a month back from 2.6.13 to 2.7.9).
b) No changes in the network and MariaDB too.
From the logs, we see the errors are "Slow SQL" as When checked on the DB side there are no issues (in terms of any errors or latency). From the logs, we can see that one of the specific SQL queries (provided below) keeps getting flooded and we believe that could be the issue and have no idea why these keep getting into.
====
SELECT *
FROM (
SELECT (
SELECT MAX(rkv.id) AS id
FROM kine AS rkv), (
SELECT MAX(crkv.prev_revision) AS prev_revision
FROM kine AS crkv
WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value
FROM kine AS kv
JOIN (
SELECT MAX(mkv.id) AS id
FROM kine AS mkv
WHERE
mkv.name LIKE '/registry/events/%'
AND
mkv.id <= 179546257 AND
mkv.id > (
SELECT MAX(ikv.id) AS id
FROM kine AS ikv
WHERE
ikv.name = '/registry/events/kube-system/kube-scheduler.17a3eaad9d69528e' AND
ikv.id <= 179546257)
GROUP BY mkv.name) AS maxkv
ON maxkv.id = kv.id
WHERE
kv.deleted = 0 OR
0
) AS lkv
ORDER BY lkv.theid ASC
LIMIT 501;
When checked on the system logs, we found the below:

Jan 5 10:21:57 rancher01 k3s[1765]: time="2024-01-05T10:21:57Z" level=info msg="Slow SQL (started: 2024-01-05 10:21:55.867128457 +0000 UTC m=+8.985828695) (total time: 1.946224228s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? AND mkv.id <= ? AND mkv.id > ( SELECT MAX(ikv.id) AS id FROM kine AS ikv WHERE ikv.name = ? AND ikv.id <= ?) GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 1000 : [[/% 179522524 /registry/events/kube-system/ccm.17a3e5d67395b87a 179522524 false]]"
Jan 5 10:21:59 rancher01 k3s[1765]: E0105 10:21:59.546311 1765 available_controller.go:524] v1beta1.metrics.k8s.io failed with: failing or missing response from https://xx.xx.x.xx:10250/apis/metrics.k8s.io/v1beta1: Get "https://xx.xx.x.xx:10250/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jan 5 10:21:59 rancher01 k3s[1765]: time="2024-01-05T10:21:59Z" level=info msg="Slow SQL (started: 2024-01-05 10:21:57.818731597 +0000 UTC m=+10.937431864) (total time: 1.944465114s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? AND mkv.id <= ? AND mkv.id > ( SELECT MAX(ikv.id) AS id FROM kine AS ikv WHERE ikv.name = ? AND ikv.id <= ?) GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 1000 : [[/% 179522524 /registry/events/default/rancher01.17a3e70d8068bdac 179522524 false]]"
Jan 5 10:22:01 rancher01 k3s[1765]: time="2024-01-05T10:22:01Z" level=info msg="Slow SQL (started: 2024-01-05 10:21:59.768305618 +0000 UTC m=+12.887005863) (total time: 1.936865322s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? AND mkv.id <= ? AND mkv.id > ( SELECT MAX(ikv.id) AS id FROM kine AS ikv WHERE ikv.name = ? AND ikv.id <= ?) GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 1000 : [[/% 179522524 /registry/events/kube-system/kube-scheduler.17a3e8742932ebc8 179522524 false]]"

Moreover, stopping the k3s service would make the node responsive, and no issues there.
We have other rancher clusters with the same configuration for various environments and no issues there. Kindly help us to eradicate this issue to get the cluster UP and running.
Steps To Reproduce:

Installed K3s:

Expected behavior:
Actual behavior:
Additional context / logs:
Updated above.

I don't see any unusual queries here, everything you're showing as examples are normal for kine.
What do you get from running the following query against your database:
SELECT name, COUNT(*) FROM kine GROUP BY name HAVING COUNT(*) > 5000;

You might also check the k3s log for messages with the word 'compact' in them; check to see if compaction is occurring successfully and if so how many rows it is deleting. See https://github.com/k3s-io/kine/issues/213#issuecomment-1717143914 for an example.

Can you reproduce this on a newer release? 1.24 is quite old and EOL upstream and there have been quite a few changes since then

I don't see any unusual queries here, everything you're showing as examples are normal for kine.
What do you get from running the following query against your database:
SELECT name, COUNT(*) FROM kine GROUP BY name HAVING COUNT(*) > 5000;

You might also check the k3s log for messages with the word 'compact' in them; check to see if compaction is occurring successfully and if so how many rows it is deleting. See k3s-io/kine#213 (comment) for an example.

Thanks @brandond for your immediate response. Much appreciated!
Below is the outcome of what we got today.

a) 5000 Count Query
MariaDB [prod_rancher]> SELECT name, COUNT() FROM kine GROUP BY name HAVING COUNT() > 5000;
Empty set (0.099 sec)
b) Max Count Query
SELECT name, COUNT() FROM kine GROUP BY name HAVING COUNT() > 100;
+----------------------------------------------------------------------------------+----------+
| name                                                                             | COUNT(*) |
+----------------------------------------------------------------------------------+----------+
| /registry/configmaps/cattle-fleet-local-system/fleet-agent-lock                  |      168 |
| /registry/configmaps/cattle-fleet-system/fleet-controller-lock                   |      167 |
| /registry/configmaps/cattle-fleet-system/gitjob                                  |      167 |
| /registry/configmaps/kube-system/cattle-controllers                              |      167 |
| /registry/configmaps/kube-system/k3s                                             |      168 |
| /registry/leases/cattle-fleet-local-system/fleet-agent-lock                      |      168 |
| /registry/leases/cattle-fleet-system/fleet-controller-lock                       |      168 |
| /registry/leases/cattle-fleet-system/gitjob                                      |      168 |
| /registry/leases/cattle-provisioning-capi-system/controller-leader-election-capi |      170 |
| /registry/leases/kube-system/cattle-controllers                                  |      168 |
| /registry/leases/kube-system/k3s                                                 |      168 |
| /registry/leases/kube-system/k3s-cloud-controller-manager                        |      170 |
| /registry/leases/kube-system/kube-controller-manager                             |      170 |
| /registry/leases/kube-system/kube-scheduler                                      |      170 |
+----------------------------------------------------------------------------------+----------+
14 rows in set (0.096 sec)

This shows no event names appearing more than 5000 times and then narrows it down to show you the few that are the biggest.
c) Here are the K3s logs with compact - not much that is all I got. This repeats. I ignored 'Slow SQL' lines since we already posted it. I see no nothing sensitive ?

Jan 11 12:15:24 rancher01 k3s[13056]: time="2024-01-11T12:15:24Z" level=error msg="error while range on /registry/events/kube-system/ccm.17a3f0f6d3bebf6c\x00 /registry/events/: rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"
Jan 11 12:15:24 rancher01 k3s[13056]: {"level":"warn","ts":"2024-01-11T12:15:24.918Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c56380/kine.sock","attempt":0,"error":"rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}

@brandond  Moreover, can you please let me know if this is normal or not - "absence of COMPACT word in k3s logs"

You're on an old and very end-of-life version of k3s that was released in November of 2022. We increased the verbosity of the COMPACT messages in https://github.com/k3s-io/k3s/issues/6958 in February of 2023. There have been many other improvements since then, I would work on upgrading ASAP to see if your issues are resolved.

Hi @brandond,
Thanks. We took some time to come back as we attempted the k3s upgrade.

We have upgraded to 1.25.16k3s4. Post upgrade, the issue continues to exist and we see the below COMPACT messages.

We have upgraded our LABS and staging environment to 1.25.16k3s4 and that continues to perform well and we are facing this issue only on this environment.

Feb  7 11:46:10 rancher01 k3s[3176]: time="2024-02-07T11:46:10Z" level=info msg="COMPACT compactRev=179665707 targetCompactRev=179666707 currentRev=179671315"
Feb  7 11:46:11 rancher01 k3s[3176]: time="2024-02-07T11:46:11Z" level=info msg="COMPACT deleted 984 rows from 1000 revisions in 927.850619ms - compacted to 179666707/179671315"
Feb  7 11:46:11 rancher01 k3s[3176]: time="2024-02-07T11:46:11Z" level=info msg="COMPACT compactRev=179666707 targetCompactRev=179667707 currentRev=179671322"
Feb  7 11:46:12 rancher01 k3s[3176]: time="2024-02-07T11:46:12Z" level=info msg="COMPACT deleted 999 rows from 1000 revisions in 943.444778ms - compacted to 179667707/179671322"
Feb  7 11:46:12 rancher01 k3s[3176]: time="2024-02-07T11:46:12Z" level=info msg="COMPACT compactRev=179667707 targetCompactRev=179668225 currentRev=179671327"
Feb  7 11:46:13 rancher01 k3s[3176]: time="2024-02-07T11:46:13Z" level=info msg="COMPACT deleted 516 rows from 518 revisions in 889.87316ms - compacted to 179668225/179671327"
Feb  7 11:51:10 rancher01 k3s[3176]: time="2024-02-07T11:51:10Z" level=info msg="COMPACT compactRev=179668225 targetCompactRev=179669225 currentRev=179673586"
Feb  7 11:51:11 rancher01 k3s[3176]: time="2024-02-07T11:51:11Z" level=info msg="COMPACT deleted 800 rows from 1000 revisions in 886.395801ms - compacted to 179669225/179673586"
Feb  7 11:51:11 rancher01 k3s[3176]: time="2024-02-07T11:51:11Z" level=info msg="COMPACT compactRev=179669225 targetCompactRev=179670225 currentRev=179673593"
Feb  7 11:51:12 rancher01 k3s[3176]: time="2024-02-07T11:51:12Z" level=info msg="COMPACT deleted 920 rows from 1000 revisions in 928.743941ms - compacted to 179670225/179673593"
Feb  7 11:51:12 rancher01 k3s[3176]: time="2024-02-07T11:51:12Z" level=info msg="COMPACT compactRev=179670225 targetCompactRev=179671225 currentRev=179673598"
Feb  7 11:51:13 rancher01 k3s[3176]: time="2024-02-07T11:51:13Z" level=info msg="COMPACT deleted 993 rows from 1000 revisions in 900.661558ms - compacted to 179671225/179673598"
Feb  7 11:51:13 rancher01 k3s[3176]: time="2024-02-07T11:51:13Z" level=info msg="COMPACT compactRev=179671225 targetCompactRev=179671327 currentRev=179673601"
Feb  7 11:51:14 rancher01 k3s[3176]: time="2024-02-07T11:51:14Z" level=info msg="COMPACT deleted 102 rows from 102 revisions in 917.342484ms - compacted to 179671327/179673601"

As stated above, the issue continues to exist with nodes peaking to high load (as high consumption of the resources). OOMs continue to occur as before, as the nodes get stalled.

Moreover, below the phrase which continues to exist, even after the upgrade.

 AND mkv.id <= ? AND mkv.id > ( SELECT MAX(ikv.id) AS id FROM kine AS ikv WHERE ikv.name = ? AND ikv.id <= ?) GROUP BY mkv.name) AS maxk
v ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 501 : [[/registry/events/% 179668560 /registry/events/d
efault/rancher01.17a3f4f4de873d2e 179668560 false]]"
Feb  7 11:51:41 rancher01 k3s[3176]: time="2024-02-07T11:51:41Z" level=error msg="error while range on /registry/events/default/rancher01.17a
3f4f4de873d2e\x00 /registry/events/: rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"
Feb  7 11:51:41 rancher01 k3s[3176]<EMAIL_ADDRESS>terceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00724d500/kine.sock","attempt":0,"error":"rpc error:
code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
Feb  7 11:51:41 rancher01 k3s[3176]: I0207 11:51:41.862947    3176 trace.go:219] Trace[2003738464]: "List(recursive=true) etcd3" audit-id:ad1
8ad12-a732-4e5d-a74c-78b9bf013de8,key:/events,resourceVersion:,resourceVersionMatch:,limit:500,continue:eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6M
Tc5NjY4NTYwLCJzdGFydCI6ImRlZmF1bHQvcmFuY2hlcjAxLjE3YTNmNGY0ZGU4NzNkMmVcdTAwMDAifQ (07-Feb-2024 11:51:18.854) (total time: 23007ms):
Feb  7 11:51:41 rancher01 k3s[3176]: Trace[2003738464]: [23.00798235s] [23.00798235s] END
Feb  7 11:51:41 rancher01 k3s[3176]: I0207 11:51:41.863041    3176 trace.go:219] Trace[1398538296]: "List" url:/apis/events.k8s.io/v1/events,
user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:ad18ad12-a732-4e5d-a74c-78b9bf013de8,client:xx.xx.xx.xx,accept:application
/json,protocol:HTTP/2.0 (07-Feb-2024 11:51:18.854) (total time: 23008ms):
Feb  7 11:51:41 rancher01 k3s[3176]: Trace[1398538296]: [23.008076417s] [23.008076417s] END
Kindly check and suggest a fix.

Hi @brandond,
Thanks. We took some time to come back as we attempted the k3s upgrade.

We have upgraded to 1.25.16k3s4. Post upgrade, the issue continues to exist and we see the below COMPACT messages.

We have upgraded our LABS and staging environment to 1.25.16k3s4 and that continues to perform well and we are facing this issue only on this environment.

Feb  7 11:46:10 rancher01 k3s[3176]: time="2024-02-07T11:46:10Z" level=info msg="COMPACT compactRev=179665707 targetCompactRev=179666707 currentRev=179671315"
Feb  7 11:46:11 rancher01 k3s[3176]: time="2024-02-07T11:46:11Z" level=info msg="COMPACT deleted 984 rows from 1000 revisions in 927.850619ms - compacted to 179666707/179671315"
Feb  7 11:46:11 rancher01 k3s[3176]: time="2024-02-07T11:46:11Z" level=info msg="COMPACT compactRev=179666707 targetCompactRev=179667707 currentRev=179671322"
Feb  7 11:46:12 rancher01 k3s[3176]: time="2024-02-07T11:46:12Z" level=info msg="COMPACT deleted 999 rows from 1000 revisions in 943.444778ms - compacted to 179667707/179671322"
Feb  7 11:46:12 rancher01 k3s[3176]: time="2024-02-07T11:46:12Z" level=info msg="COMPACT compactRev=179667707 targetCompactRev=179668225 currentRev=179671327"
Feb  7 11:46:13 rancher01 k3s[3176]: time="2024-02-07T11:46:13Z" level=info msg="COMPACT deleted 516 rows from 518 revisions in 889.87316ms - compacted to 179668225/179671327"
Feb  7 11:51:10 rancher01 k3s[3176]: time="2024-02-07T11:51:10Z" level=info msg="COMPACT compactRev=179668225 targetCompactRev=179669225 currentRev=179673586"
Feb  7 11:51:11 rancher01 k3s[3176]: time="2024-02-07T11:51:11Z" level=info msg="COMPACT deleted 800 rows from 1000 revisions in 886.395801ms - compacted to 179669225/179673586"
Feb  7 11:51:11 rancher01 k3s[3176]: time="2024-02-07T11:51:11Z" level=info msg="COMPACT compactRev=179669225 targetCompactRev=179670225 currentRev=179673593"
Feb  7 11:51:12 rancher01 k3s[3176]: time="2024-02-07T11:51:12Z" level=info msg="COMPACT deleted 920 rows from 1000 revisions in 928.743941ms - compacted to 179670225/179673593"
Feb  7 11:51:12 rancher01 k3s[3176]: time="2024-02-07T11:51:12Z" level=info msg="COMPACT compactRev=179670225 targetCompactRev=179671225 currentRev=179673598"
Feb  7 11:51:13 rancher01 k3s[3176]: time="2024-02-07T11:51:13Z" level=info msg="COMPACT deleted 993 rows from 1000 revisions in 900.661558ms - compacted to 179671225/179673598"
Feb  7 11:51:13 rancher01 k3s[3176]: time="2024-02-07T11:51:13Z" level=info msg="COMPACT compactRev=179671225 targetCompactRev=179671327 currentRev=179673601"
Feb  7 11:51:14 rancher01 k3s[3176]: time="2024-02-07T11:51:14Z" level=info msg="COMPACT deleted 102 rows from 102 revisions in 917.342484ms - compacted to 179671327/179673601"

As stated above, the issue continues to exist with nodes peaking to high load (as high consumption of the resources). OOMs continue to occur as before, as the nodes get stalled.

Moreover, below the phrase which continues to exist, even after the upgrade.

 AND mkv.id <= ? AND mkv.id > ( SELECT MAX(ikv.id) AS id FROM kine AS ikv WHERE ikv.name = ? AND ikv.id <= ?) GROUP BY mkv.name) AS maxk
v ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 501 : [[/registry/events/% 179668560 /registry/events/d
efault/rancher01.17a3f4f4de873d2e 179668560 false]]"
Feb  7 11:51:41 rancher01 k3s[3176]: time="2024-02-07T11:51:41Z" level=error msg="error while range on /registry/events/default/rancher01.17a
3f4f4de873d2e\x00 /registry/events/: rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"
Feb  7 11:51:41 rancher01 k3s[3176]<EMAIL_ADDRESS>terceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00724d500/kine.sock","attempt":0,"error":"rpc error:
code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
Feb  7 11:51:41 rancher01 k3s[3176]: I0207 11:51:41.862947    3176 trace.go:219] Trace[2003738464]: "List(recursive=true) etcd3" audit-id:ad1
8ad12-a732-4e5d-a74c-78b9bf013de8,key:/events,resourceVersion:,resourceVersionMatch:,limit:500,continue:eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6M
Tc5NjY4NTYwLCJzdGFydCI6ImRlZmF1bHQvcmFuY2hlcjAxLjE3YTNmNGY0ZGU4NzNkMmVcdTAwMDAifQ (07-Feb-2024 11:51:18.854) (total time: 23007ms):
Feb  7 11:51:41 rancher01 k3s[3176]: Trace[2003738464]: [23.00798235s] [23.00798235s] END
Feb  7 11:51:41 rancher01 k3s[3176]: I0207 11:51:41.863041    3176 trace.go:219] Trace[1398538296]: "List" url:/apis/events.k8s.io/v1/events,
user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:ad18ad12-a732-4e5d-a74c-78b9bf013de8,client:xx.xx.xx.xx,accept:application
/json,protocol:HTTP/2.0 (07-Feb-2024 11:51:18.854) (total time: 23008ms):
Feb  7 11:51:41 rancher01 k3s[3176]: Trace[1398538296]: [23.008076417s] [23.008076417s] END

Kindly check and suggest a fix.

Thank you Brandon for your updates, we appreciate it!
We have increased the DB node resources from 4 vCPUs / 8GB RAM to 16 vCPUs / 16 GB RAM but we are still
seeing the logs with SQL queries taking too long to complete. However, we found some other log messages that could help.

Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: ---"Listing from storage done" 10388ms (11:52:39.106)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: [10.396998053s] [10.396998053s] END
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: ---"Listing from storage done" 10416ms (11:52:39.136)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: [10.423875526s] [10.423875526s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: ---"Listing from storage done" 10179ms (11:52:49.331)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: [10.185408167s] [10.185408167s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: ---"Listing from storage done" 10202ms (11:52:49.384)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: [10.208759022s] [10.208759022s] END
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: ---"Listing from storage done" 15820ms (11:53:05.242)
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: [15.826846383s] [15.826846383s] END

We are not sure what listing from storage refers to it and this might be related to /var/lib/rancher/k3s and the advice that you provided in this ticket.
https://github.com/k3s-io/k3s/issues/6661
Since it takes too long to complete.
Moreover, this jump in IOPS occurs and then OOM happens (there was no data before because rancher nodes were shut down). Below is the graph screenshot referencing that.

Thank you Brandon for your updates, we appreciate it!
We have increased the DB node resources from 4 vCPUs / 8GB RAM to 16 vCPUs / 16 GB RAM but we are still
seeing the logs with SQL queries taking too long to complete. However, we found some other log messages that could help.

Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: ---"Listing from storage done" 10388ms (11:52:39.106)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: [10.396998053s] [10.396998053s] END
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: ---"Listing from storage done" 10416ms (11:52:39.136)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: [10.423875526s] [10.423875526s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: ---"Listing from storage done" 10179ms (11:52:49.331)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: [10.185408167s] [10.185408167s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: ---"Listing from storage done" 10202ms (11:52:49.384)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: [10.208759022s] [10.208759022s] END
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: ---"Listing from storage done" 15820ms (11:53:05.242)
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: [15.826846383s] [15.826846383s] END

We are not sure what listing from storage refers to it and this might be related to /var/lib/rancher/k3s and the advice that you provided in this ticket.
https://github.com/k3s-io/k3s/issues/6661
Since it takes too long to complete.
Moreover, this jump in IOPS occurs and then OOM happens (there was no data before because rancher nodes were shut down). Below is the graph screenshot referencing that.

Thank you Brandon for your updates, we appreciate it!
We have increased the DB node resources from 4 vCPUs / 8GB RAM to 16 vCPUs / 16 GB RAM but we are still
seeing the logs with SQL queries taking too long to complete. However, we found some other log messages that could help.
#############
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: ---"Listing from storage done" 10388ms (11:52:39.106)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[1272443639]: [10.396998053s] [10.396998053s] END
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: ---"Listing from storage done" 10416ms (11:52:39.136)
Feb  8 11:52:39 rancher01 k3s[2400]: Trace[109032895]: [10.423875526s] [10.423875526s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: ---"Listing from storage done" 10179ms (11:52:49.331)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1916448722]: [10.185408167s] [10.185408167s] END
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: ---"Listing from storage done" 10202ms (11:52:49.384)
Feb  8 11:52:49 rancher01 k3s[2400]: Trace[1332769888]: [10.208759022s] [10.208759022s] END
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: ---"Listing from storage done" 15820ms (11:53:05.242)
Feb  8 11:53:05 rancher01 k3s[2400]: Trace[1187522670]: [15.826846383s] [15.826846383s] END
##################
We are not sure what listing from storage refers to it and this might be related to /var/lib/rancher/k3s and the advice that you provided in this ticket.
https://github.com/k3s-io/k3s/issues/6661
Since it takes too long to complete.
Moreover, this jump in IOPS occurs and then OOM happens (there was no data before because rancher nodes were shut down). Below is the graph screenshot referencing that.

Which process specifically is OOMing? It sounds like there is some client that is putting excessive load on the apiserver, and that is in turn passed along to the datastore in the form of additional queries.
Can you enable audit logging and figure out what is making all these requests?

Thanks again.
We have enabled the audit log on the rancher nodes and captured the audit logs. The same situation of the node gets into a race condition as well with OOM.
We do see that the code shows all RequestReceived arrays where userAgent repeats more than 1000 times.
We have used the "jq" filter to get parsed. We have concatenated all the audit logs from rancher01 into one log and here is now the overall picture.
Had put a condition for requestURI to start with "/api" since there are requests to the API server with different URI portions but they all start with "/api":

jq 'select(.stage == "RequestReceived" and select(.requestURI | startswith("/api")))' audit_all.log | grep userAgent | sort | uniq -c | awk '{if ($1 > 1000) {print $0}}' | sort -rk1
68659   "userAgent": "rancher/v0.0.0 (linux/amd64) kubernetes/$Format",
5615   "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/leader-election",
4081   "userAgent": "fleetagent/v0.0.0 (linux/amd64) kubernetes/$Format",
3670   "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/system:serviceaccount:kube-system:generic-garbage-collector",
30276   "userAgent": "webhook/v0.0.0 (linux/amd64) kubernetes/$Format",
3024   "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/system:serviceaccount:kube-system:resourcequota-controller",
1873   "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247",
1866   "userAgent"<EMAIL_ADDRESS>(linux/amd64) k3s/ddda2476",

Hope this will be helpful to progress further.

You're sorting alphabetically instead of numerically, if you sort by the actual count it's pretty clear that the requests are coming from rancher and the rancher webhook; they're an order of magnitude above everything else. I would probably open an issue in rancher/rancher.
68659 "userAgent": "rancher/v0.0.0 (linux/amd64) kubernetes/$Format",
30276 "userAgent": "webhook/v0.0.0 (linux/amd64) kubernetes/$Format",
5615 "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/leader-election",
4081 "userAgent": "fleetagent/v0.0.0 (linux/amd64) kubernetes/$Format",
3670 "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/system:serviceaccount:kube-system:generic-garbage-collector",
3024 "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247/system:serviceaccount:kube-system:resourcequota-controller",
1873 "userAgent": "k3s/v1.25.16+k3s4 (linux/amd64) kubernetes/ddda247",
1866 "userAgent"<EMAIL_ADDRESS>(linux/amd64) k3s/ddda2476",

Thanks @brandond for all your time and your great help. As per your recommendation, we have opened a new request in rancher.
Thanks again.

Closing this in favor of tracking in rancher/rancher, we can reopen if an issue is identified that needs to be fixed here in k3s.

Hi @brandond ,
Thanks for making the closure.
Yesterday, we increased the memory from 16GB to 32GB and started the k3s service on both nodes. For the initial time of 1 hour, we noticed the steep UP of memory, and network transaction in the interface connecting to the DB host and I/O. then after some amount of time, utilization dropped. Also, the events count was more than 3 Lakhs in count. As time progressed, events started to clean on their own and so the memory usage has come down to 20% and this is the present state. No crashes and GUI access is much better. We have kept in surveillance now.


The lack of test cases with Half Precision Floating-Point in the master branch

https://github.com/KhronosGroup/OpenCL-CTS/blob/58cf793fdb2d8786d772ab1fd7cedae6cf9cafdd/test_conformance/half/main.cpp#L56

As we can see in the master branch, only vload/vstore is tested in the “half” test group，without test cases for Half Precision Floating-Point in conversion/math/common/geometric/relational, etc.
The device info “CL_DEVICE_HALF_FP_CONFIG” is also not used in master branch for testing half denormal.

This is a duplicate of #831

This is a duplicate of #831


no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type

Plugin Version or Commit ID
v0.82(latest)
Unity Version
2020.3.30
Your Host OS
window 10 Professional
Target Platform
Android
[Windows Only] Visual Studio C++ and Windows SDK Version
No response
[Linux Only] GCC/G++ and GLIBC Version
No response
[Android Only] Android Build Tools and NDK Version
30.0.3
NDK : 23.0.7272597
[iOS Only] XCode Version
No response
Command Sequences
python build.py build --android arm64 --android_ndk_api_level 21 -vv
Log
DEBUG: C:/_bzl/hmjgtv3e/external/org_tensorflow/third_party/repo.bzl:122:14:
Warning: skipping import of repository 'pybind11' because it already exists.
INFO: Build options --copt, --cpu, --crosstool_top, and 6 more have changed, discarding analysis cache.
ERROR: C:/users/pedro/downloads/mediapipeunityplugin-master/mediapipe_api/java/com/github/homuler/mediapipe/BUILD:10:14: While resolving toolchains for target //mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android_android_lib: no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type
ERROR: Analysis of target '//mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android' failed; build aborted:
INFO: Elapsed time: 0.288s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 167 targets configured)
Traceback (most recent call last):
File "C:\Users\pedro\Downloads\MediaPipeUnityPlugin-master\build.py", line 436, in
Argument().command().run()
File "C:\Users\pedro\Downloads\MediaPipeUnityPlugin-master\build.py", line 155, in run
self._run_command(self._build_android_commands())
File "C:\Users\pedro\Downloads\MediaPipeUnityPlugin-master\build.py", line 54, in _run_command
return subprocess.run(' '.join(command_list), check=True, shell=shell)
File "C:\Python39\lib\subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
Additional Context
what may be the problem?
problem is that on mac monterrey 12.3 & window, it cause same error log.
what I want is to implement objectron example on android device with unity.

Set ANDROID_HOME and ANDROID_NDK_HOME.
Please also note that Bazel does not support NDK >= r22 yet.
cf. https://github.com/homuler/MediaPipeUnityPlugin/wiki/Installation-Guide#android-configuration
By the way, using Docker is far easier to build mediapipe_android.aar.
https://github.com/homuler/MediaPipeUnityPlugin/wiki/Installation-Guide#docker-linux-container

I'm trying to run build.py for android following the document on my M1 Mac. I installed android studio and installed build tools 30.0.3 and also ndk 20.1.5948944 and also set ANDROID_HOME and ANDROID_NDK_HOME in my ~/.zprofile and I can also see them set when I run printenv. but when I run "python3 build.py build --android arm64 -vv" to build for android I get the following errors:
ERROR: /Users/smtabatabaie/UnityWorkSpace/MediapipeUnity_homuler2/MediaPipeUnityPlugin2/mediapipe_api/java/com/github/homuler/mediapipe/BUILD:10:14: While resolving toolchains for target //mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android_dummy_app: no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type
ERROR: Analysis of target '//mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android' failed; build aborted:
I'd appreciate it if you can help me with this.

I'm trying to run build.py for android following the document on my M1 Mac. I installed android studio and installed build tools 30.0.3 and also ndk 20.1.5948944 and also set ANDROID_HOME and ANDROID_NDK_HOME in my ~/.zprofile and I can also see them set when I run printenv. but when I run "python3 build.py build --android arm64 -vv" to build for android I get the following errors:
ERROR: /Users/smtabatabaie/UnityWorkSpace/MediapipeUnity_homuler2/MediaPipeUnityPlugin2/mediapipe_api/java/com/github/homuler/mediapipe/BUILD:10:14: While resolving toolchains for target //mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android_dummy_app: no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type
ERROR: Analysis of target '//mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android' failed; build aborted:
I'd really appreciate it if you can help me with this.

I'm trying to run build.py for android following the document on my M1 Mac. I installed android studio and installed build tools 30.0.3 and also ndk 20.1.5948944 and also set ANDROID_HOME and ANDROID_NDK_HOME in my ~/.zprofile and I can also see them set when I run printenv. but when I run "python3 build.py build --android arm64 -vv" to build for android I get the following errors:
ERROR: /Users/smtabatabaie/UnityWorkSpace/MediapipeUnity_homuler2/MediaPipeUnityPlugin2/mediapipe_api/java/com/github/homuler/mediapipe/BUILD:10:14: While resolving toolchains for target //mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android_dummy_app: no matching toolchains found for types @bazel_tools//tools/android:sdk_toolchain_type
ERROR: Analysis of target '//mediapipe_api/java/com/github/homuler/mediapipe:mediapipe_android' failed; build aborted:
I'd really appreciate it if you can help me with this.

I could fix it by downloading and running setup_android_sdk_and_ndk.sh, it's a bash script from mediapipe repository which sets android SDK and android ndk.


Path to non-existing key on GCS fails to be handled as optional input

Summary
Preamble
What happened/what you expected to happen?
Before we start, around 2/3 of issues can be fixed by one of the following:

Have you double-checked your configuration? Maybe 30% of issues are wrong configuration.

Yep, other downloaded artifacts are fine.

Have you tested to see if it is fixed in the latest version? Maybe 20% of issues are fixed by this.

Nope, but the code in question hasn't been touched for 16 months.

Have you tried using the PNS executer instead of Docker? Maybe 50% of artifact related issues are fixed by this.

Nope, but this doesn't seem relevant as I believe the cause is the GCS storage client failing to make a file or raise an error.
Description
I expected an optional input to not cause an error when it does not exist on the local filesystem.
What happened was, the optional input caused: executor error: rename /argo/inputs/artifacts/ambient-cache.tmp /argo/inputs/artifacts/ambient-cache: no such file or directory.
When using Google Cloud Storage as an artifact repository, passing optional outputs/inputs between steps may cause an unexpected failure in the case that an optional output was not created in one step, and thus the key doesn't exist on GCS, but it is attempted to be passed as an optional input to a subsequent step. In the case that the key doesn't exist, an empty array skips a loop execution and fails to raise/catch an error for a missing optional artifact.
In this line, the errors.CodeNotFound is not raised because the listByPrefix method called by artDriver.Load returns an empty list and skips any other operations on the file.
I pulled out the methods from the argo-workflows/workflow/artifacts/gcs package into a gist here to demonstrate the problem. The listByPrefix method may return an empty list from the GCS API, which means that the loop in which downloadObject is called never executes. Thus, none of the file I/O errors are ever raised from inside downloadObject. The line in question is here.
It seems like the simplest fix would be to raise an error in the case of an empty array from listByPrefix that will be sent up the stack to be handled by the executor.
Diagnostics
👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we'll go around in circles asking you for it:
What Kubernetes provider are you using?
GKE
What version of Argo Workflows are you running?
v3.1.0-rc14
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Emissary
Did this work in a previous version? I.e. is it a regression?
Unknown.
Are you pasting thousands of log lines? That's too much information.
Nope
Workflow
Our workflows that experience this have hundreds to thousands of lines, and seem like they would be more distracting than helpful here. I believe this issue is specific to the GCS storage adapter rather than the workflow itself. I'm happy to try and craft a minimal workflow if that does seem useful though. The gist that is linked above reproduces the failure case.

Failing Workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  namespace: argo
  generateName: gcs-test-
  labels:
    workflows.argoproj.io/container-runtime-executor: emissary
    disposable: 'true'
spec:
  entrypoint: gcs-test
  templates:
    - name: gcs-test
      inputs:
        artifacts:
          - name: my-art
            path: /my-artifact
            optional: true
            gcs:
              bucket: pollination-public
              key: blobs/argo-test/not-exists.txt
              serviceAccountKeySecret:
                name: gcs-creds
                key: serviceAccountKey
      container:
        image: ubuntu:latest
        command: [sh, -c]
        args: ["echo 'no artifact :('"]

Working Workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  namespace: argo
  generateName: gcs-test-
  labels:
    workflows.argoproj.io/container-runtime-executor: emissary
    disposable: 'true'
spec:
  entrypoint: gcs-test
  templates:
    - name: gcs-test
      inputs:
        artifacts:
          - name: my-art
            path: /my-artifact
            optional: true
            http:
              url: ''
      container:
        image: ubuntu:latest
        command: [sh, -c]
        args: ["echo 'no artifact :('"]

Misc.
Thank you for working on argo! Overall, it has been really great to work with. :smile_cat:

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@tymokvo Is this issue happening consistently?
Can you share the wait  container logs and controller logs? Can you provide sample reproducible workflow yaml?

@sarabala1979 thanks for the quick reply! I updated the issue with a failing workflow using GCS and, for comparison, one that succeeds using HTTP.
Wait container logs:
Error from server (BadRequest): container "wait" in pod "gcs-test-4gtk4" is waiting to start: PodInitializing

Controller logs:
time="2021-07-07T01:40:55.860Z" level=info msg="Processing workflow" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:40:55.864Z" level=info msg="Get configmaps 200"
time="2021-07-07T01:40:55.864Z" level=info msg="resolved artifact repository" artifactRepositoryRef="argo/#"
time="2021-07-07T01:40:55.866Z" level=info msg="Get configmaps 200"
time="2021-07-07T01:40:55.867Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:40:55.867Z" level=info msg="Pod node gcs-test-4gtk4 initialized Pending" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:40:55.872Z" level=info msg="Create events 201"
time="2021-07-07T01:40:55.882Z" level=info msg="Create pods 201"
time="2021-07-07T01:40:55.883Z" level=info msg="Created pod: gcs-test-4gtk4 (gcs-test-4gtk4)" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:40:55.889Z" level=info msg="Update workflows 200"
time="2021-07-07T01:40:55.890Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=61169066 workflow=gcs-test-4gtk4
time="2021-07-07T01:40:56.188Z" level=info msg="Watch workflows 200"
time="2021-07-07T01:40:58.742Z" level=info msg="Get leases 200"
time="2021-07-07T01:40:58.745Z" level=info msg="Update leases 200"
time="2021-07-07T01:41:03.749Z" level=info msg="Get leases 200"
time="2021-07-07T01:41:03.752Z" level=info msg="Update leases 200"
time="2021-07-07T01:41:05.884Z" level=info msg="Processing workflow" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.887Z" level=info msg="Get configmaps 200"
time="2021-07-07T01:41:05.888Z" level=info msg="Pod failed: Error (exit code 1): open /argo/inputs/artifacts/my-art.tmp: no such file or directory" displayName=gcs-test-4gtk4 namespace=argo pod=gcs-test-4gtk4 templateName=gcs-test workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.888Z" level=info msg="Updating node gcs-test-4gtk4 status Pending -> Error" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.888Z" level=info msg="Updating node gcs-test-4gtk4 message: Error (exit code 1): open /argo/inputs/artifacts/my-art.tmp: no such file or directory" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.890Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.890Z" level=info msg="Updated message  -> Error (exit code 1): open /argo/inputs/artifacts/my-art.tmp: no such file or directory" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.890Z" level=info msg="Marking workflow completed" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.890Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.890Z" level=info msg="Checking daemoned children of " namespace=argo workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.897Z" level=info msg="Create events 201"
time="2021-07-07T01:41:05.899Z" level=info msg="Update workflows 200"
time="2021-07-07T01:41:05.900Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=61169135 workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.902Z" level=info msg="archiving workflow" namespace=argo uid=ee1bbe7a-5b72-46b2-b1b2-6803b2049533 workflow=gcs-test-4gtk4
time="2021-07-07T01:41:05.904Z" level=info msg="Create events 201"
time="2021-07-07T01:41:05.907Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/gcs-test-4gtk4/labelPodCompleted
time="2021-07-07T01:41:05.908Z" level=info msg="Create events 201"
time="2021-07-07T01:41:05.916Z" level=info msg="Patch pods 200"
time="2021-07-07T01:41:05.929Z" level=info msg="Patch workflows 200"
time="2021-07-07T01:41:05.930Z" level=info msg="archiving workflow" namespace=argo uid=ee1bbe7a-5b72-46b2-b1b2-6803b2049533 workflow=gcs-test-4gtk4

@sarabala1979 I have patched our fork of argo with the change that I proposed here and can confirm that it solves the issue. Should I open a PR or is there more info I can provide?

Maybe fixed by #6393. Re-open if not.


Register new and modified files with the server and local cache

Upon detecting a new or modified file on the filesystem (based on comparison to local PouchDB cache), file is immediately registered with the backend server. If backend responds without error, local cache for the file is updated.
No longer using a single 'status' per file, but storing the status associated with each transfer destination.
GraphQL queries have been centralised in a single location.
Added a test to check that new files are added to the local cache.
Also updates some npm packages.

Looks like Github isn't re-pulling force pushed branches anymore (I squashed all the AppVeyor fix commits) - might want to squash this on merge.


Cannot download SVG icons from Carbon or Design Language websites

Detailed description
I cannot download SVG icons from the Carbon or Design Language websites:

https://www.ibm.com/design/language/iconography/ui-icons/library/
https://www.carbondesignsystem.com/guidelines/icons/library/

When I click on the download SVG icon, it appears to download but the file has a size of Zero bytes and appears to be empty.
Additional information
Screenshot:

@joshua-vaughn investigating, thanks for opening the issue!

Verified that it's working now, thanks!


Removed octree related project files and change sscanf_s to sscanf

Please pay attention to modifications done to RaymarchUtils.cpp, I just simply commented out the GenerateOctreeForVolume_RenderThread line, and i'v encountered some runtime bugs when testing the project.
DICOM import worked, but not in runtime mode, several random crashes all related to octree shaders?
Anyway, looking forward to contribute in this project😀

By the way, this modified version can run successfully on my Mac with Unreal Engine 5.4.4.