BUG: Twistlock instance stuck at InProgress and failing status checks

Description
The twistlock instance is responding to ssm commands with InProgress and then failing to run the scan. The instance itself also eventually reports that the  'Instance reachability check failed'. Typically this has been resolved by stopping the instance, starting the instance, waiting for the 2 status checks to pass, then re-starting the Twistlock action.
This morning, the Twistlock instance was restarted twice after a failure last night, the scan rerun performed twice and the result is:
Waiter CommandExecuted failed: Max attempts exceeded. Previously accepted state: For expression "Status" we matched expected path: "InProgress"

Waiter CommandExecuted failed: Max attempts exceeded. Previously accepted state: For expression "Status" we matched expected path: "InProgress"

Waiter CommandExecuted failed: Max attempts exceeded. Previously accepted state: For expression "Status" we matched expected path: "InProgress"

Waiter CommandExecuted failed: Max attempts exceeded. Previously accepted state: For expression "Status" we matched expected path: "InProgress"

Waiter CommandExecuted failed: Max attempts exceeded. Previously accepted state: For expression "Status" we matched expected path: "InProgress"
"InProgress"
""
Error: Process completed with exit code 255.

Eventually the instance failed its reachability status check.

[ ] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

Steps to Reproduce

Stop the utility-twistlock-scanner instance
When fully stopped, start the utility-twistlock-scanner instance
Wait for the instance to report Running and that the two status checks have passed
Re-run this Twistlock job
Observe that the action fails with the Waiter CommandExecuted failed: never reaching the expected status and exiting with error code 255 after 5 attempts
Wait some period of time and observe that the reachability status_check fails

Workaround
Is there something we can do to work around this issue in the meantime?
No - there is no workaround at present.
Impact/Urgency

High - Twistlock scans our app container for security vulnerabilities.
Expected Behavior

[ ] cron job is running that deletes the /var/log/messsages  daily
[ ] Increase memory (terraform) - double it.
[ ] When the app is deployed to dev, then the Twistlock scan runs and completes.
[ ] Follow-on spike ticket created to determine a better path moving forward (see comment below for suggestions

QA Considerations

[ ] Validate that when the app is deployed to perf, then the Twistlock scan runs and completes.

Additional Info & Resources
'
There are Cloudwatch logs for the Twistlock EC2 which should be reviewed to find any smoking guns.
From the errors.log:
2023-07-19 13:41:15 ERROR [New @ errors.go.59] [ssm-agent-worker] [MessageService] [Association] [EngineProcessor] [OutOfProcExecuter] [43fb267c-f4d1-4f2d-959b-041a43de4339.2023-07-18T14-07-46.567Z] received timedout signal!
2023-07-19 13:41:15 ERROR [messaging @ master.go.131] [ssm-agent-worker] [MessageService] [Association] [EngineProcessor] [OutOfProcExecuter] [43fb267c-f4d1-4f2d-959b-041a43de4339.2023-07-18T14-07-46.567Z] messaging worker encountered error: document worker timed out

From the failed action on the Perform Twistlock scan, before it starts printing out the WaiterCommands
Run INSTANCE_ID=$(aws ssm get-parameter --name /utility/twistlock/instance | jq '.Parameter.Value' -r)
  INSTANCE_ID=$(aws ssm get-parameter --name /utility/twistlock/instance | jq '.Parameter.Value' -r)
  COMMAND_ID=$(aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --instance-ids $INSTANCE_ID \
    --parameters '{"commands":[
        "#!/bin/bash",
       <redacted>
1>/dev/null", "exit $STATUS_CODE"
      ]}' \
    --output text --query "Command.CommandId")
  n=0
  until [ "$n" -ge 5 ]
  do
    aws ssm wait command-executed --command-id $COMMAND_ID --instance-id $INSTANCE_ID && break
    n=$((n+1))
  done
  RESULT=$(aws ssm list-command-invocations \
    --command-id $COMMAND_ID \
    --details \
    --query "CommandInvocations[].CommandPlugins[].{Status:Status,StatusDetails:StatusDetails,ResponseCode:ResponseCode,Output:Output}")
  echo `jq '.[].Status' <<< $RESULT`
  printf '%b\n' "$(jq '.[].Output' <<< $RESULT)"
  exit `jq '.[].ResponseCode' <<< $RESULT`
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    <redacted>

AWS Understanding Command Statuses
AWS Send Command output - info on timeouts - we never reach a timeout but might be useful
AWS Troubleshooting SSM Agent

From Twistlock system log:

Yesterday, Lucas terminated the instance and re-deployed (since this EC2 uses terraform).
Instance went into a failed status check today after the GutHub Action successfully ran.
Restarting instance - storage is 36GiB so will look for any logs that can be removed if able to connect.

Questions:

should we put an alarm on this or monitor storage
if logs are knocking this over, do we need to extend the storage or run a cron that removes logs?
can we instead run twistlock through a lambda? we would need to put the API container that it scans somewhere, possibly a self-hosted runner?
can we ditch twistlock in favor of the native AWS container scans?
AWS Image Scanning
Container scanning updates in Amazon ECR private registries using Amazon Inspector

Coverage and pricing

Now that you have an idea of the new scanning alternatives available for Amazon ECR, let’s address the questions of coverage and costs. At the moment, Amazon ECR provides CVE scanning for OS packages for most common Linux distributions, including Debian, Ubuntu, and Amazon Linux. Please refer to the supported OS and programming languages documentation for an up-to-date listing.

Basic scanning is available with no additional charge to Amazon ECR customers. Each container image may be scanned once per 24 hours with basic scanning. This limit includes the initial scan on push, if enabled, and any manual scans.

Enhanced scanning is provided and billed through Amazon Inspector. All accounts new to Amazon Inspector are eligible for a 15-day trial to evaluate the service and estimate its cost. During the trial, all eligible container images pushed to Amazon ECR are continually scanned at no cost.

Pricing for enhanced scanning has two tiers. The cost for container images scanned initially on-push to Amazon ECR is $0.09 per image scan. Each re-scan for container image in Amazon ECR configured for continuous scanning is $0.01 per image re-scan.

Twistlock scan completed on a deploy to Perf and status checks are still 2/2. Stable for the moment, but we need a plan here.

TL fell over again on the next scan. Restarting to find logs and start removing them.

Some logs in here that are candidates for deletion, specifically the messages one.
-rw-------.  1 root   root            5556423 Jul 20 18:22 messages

@ldraney with @k-macmillan 's green light, I deleted the /var/logs/messages file.
We need to run a cron job to delete that file once a week until we come up with a LT solution.
Twistlock scanned a few times without falling over, so we are good until this log fills up again.

On two twistlock scans today, the server failed a status check (the scans were also run at the same time, in different environments.
Stopping and restarting to see how big the messages log is and to re-run those scans.

Messages grew overnight:
-rw-------.  1 root   root            4458611 Jul 21 16:45 messages

Documentation on AWS CodeScan and Amazon Inspector
cc @k-macmillan @mjones-oddball @ldraney

Twistlock has fallen over again, restarting and deleting the messages file when it comes back up.

-rw-------.  1 root   root            6060854 Jul 21 20:29 messages

That's only 5-6mb. I wouldn't expect issues? Is there anything interesting in there?

@k-macmillan the output is all piped to CloudWatch, there are some interesting things in there, and (this is a guess) whatever is writing to that file is trying to open it and struggling to do so to add another line. That is not that big of a file as far as storage is concerned, but memory might be limiting the write operation.... which is why most log files are configured to roll over to a new file at a certain size (say, 1MB).

Twistlock failed 1/2 status checks over the weekend. Restarting.

Just waiting for Twistlock to restart:

Cron job set up to delete twistlock logs every 24 hours:
sudo crontab -e

I tested running twistlock and resource utilization was non-critical.
Here is a video of running twistlock with top open

Twistlock running successfully:

I'm trying to connect to the instance but the screen is now blank:

I'm stopping the instance to restart and get in and try a:
df -h

to see how much disk space we are utilizing.

df -h gives us:

And I tested twistlock again and saw that memory goes down to 49mb out of 200, (so we are at 75% memory utilization).  It might make the difference to increase the ec2 size at this point.
Also I wonder if there is something with the twistlock.yml action using the CLI to communicate with the ec2 that is somehow overwhelming the EC2... IDK.

Twistlock System Requirements (geared toward the main server, which the twistcli connect to, does not specifically speak to what an instance running twistcli should be scoped to)
Twistlock documentation on twistcli

I'm increasing to t3.medium:

Also, since t3 is a burstable instance type that uses credits, it may be that we simply ran out of credits, which would severely limit the twistlock performance.

Latest video shows t3.medium at 75% usage

So the cause is likely too small an ec2 for the twistlock:
I gathered the following data in the t3.medium:

Not Under Load:

Total Memory: 3977792 KiB
Used Memory: 958796 KiB
Total Used Memory = Used Memory + Buffers/Cache = 958796 KiB + 789184 KiB = 1747980 KiB
Percentage of memory used = (Total Used Memory / Total Memory) * 100 = (1747980 / 3977792) * 100 ≈ 43.94%

Under Load / running Twistlock:

Total Memory: 3977792 KiB
Used Memory: 1134212 KiB
Total Used Memory = Used Memory + Buffers/Cache = 1134212 KiB + 1991764 KiB = 3125976 KiB
Percentage of memory used = (Total Used Memory / Total Memory) * 100 = (3125976 / 3977792) * 100 ≈ 78.59%

Therefore, the memory utilization when not under load is roughly 43.94%, and under load, it is approximately 78.59%.
Additional Test - Running Twistlock Twice Simultaneously

Total Memory: 3977792 KiB
Available Memory: 104488 KiB
Used Memory: 1202640 KiB
Buffers/Cache: 2670664 KiB
Percentage of memory used = ((Used Memory + Buffers/Cache) / Total Memory) * 100 = ((1202640 KiB + 2670664 KiB) / 3977792 KiB) * 100 ≈ 97.37%

The test of running Twistlock twice simultaneously shows that memory utilization spikes to approximately 97.37%.
Please, let me know if you need further information or testing.

re-deployed ec2 t3.medium and redid the cronjob:

opening the PR now

I'm not sure what to put on this follow-on spike ticket for twistlock.
Cris started a doc and I added to the end.  It sounds like we'll finish that doc and send it to Mel and Bev. I’m waiting to confirm this course of action with Cris and whoever before closing this ticket.
Meanwhile, I think the best course of action is to leave Twistlock until our better solution is approved and available.

@ldraney Can you make sure @k-macmillan approves before any plans go to me or Bev to approve? Is it a cost thing that we'll need to approve?

From Kyle:

Lucas compare the cost of running t3.medium 24/7/365 vs the advanced scan costs. Try and grab how many scans we do per week and multiply it by 52 (a week will be more accurate than a day).
TBH engineer costs to maintain Twistlock are likely greater than any cost for scanning.

Documentation for replacing Twistlock

@ldraney  to add costs to document and create new ticket, then we can close this one out.

Twistlock failed 1/2 status checks again today and failed to run a scan. Stopping, restarting, and logging in to check things.

restarted, it came back up, re-tried scan, that worked.

From Lucas

Well in the last fifteen minutes we got 1k errors saying VAEC network is unreachable.  Twistlock is set up on the AMI, and VAEC may have set up their own server
The way Twistlock works is it collects information about the packages and binaries in the container image and sends this data to the Prisma Cloud Console for analysis.  I'm guessing VAEC set up their own server for security reasons.
So, Twistlock may be failing due to reasons outside our control.  Another reason to move away from it, if I'm right.

Logs are now in DD, some patterns note that the failure is on some heartbeat to logstash/
Jul 27 HH:mm:ss
 ip-10-247-142-206 *
: yyyy-MM-ddTHH:mm:ss.SSSXXX
#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:145#011Attempting to reconnect to backoff(async(tcp://aws-logstash-west.<redacted>)) with [1-210]
 reconnect attempt(s)

Jul 27 HH:mm:ss
 ip-10-247-142-206 *
: yyyy-MM-ddTHH:mm:ss.SSSXXX
#011ERROR#011[publisher_pipeline_output]#011pipeline/output.go:154#011Failed to connect to backoff(async(tcp://aws-logstash-west.<redacted>)): lookup aws-logstash-west.ecs.vaec.va.gov on <IP_ADDRESS>:53: dial udp <IP_ADDRESS>:53: connect: network is unreachable

Info on audit logs
https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/audit/logging
Unable to connect to session manager, restarting.

Lucas quadrupled the audit_backlog_limit

Twistlock fell over again today.

ROI document for switching to AWS Inspector

Story created for implementing AWS Inspector

Next steps:

@k-macmillan @mjones-oddball @cris-oddball to review #1396
@k-macmillan to enable advanced scanning in AWS today to immediately replace Twistlock
@ldraney to disable twistlock GitHub action from running on deploy today.


Adding option to use TLS in broker admin client.

Motivation
Bug fix for #2830
Modifications

Renamed replicationTlsEnabled to brokerClientTlsEnabled since all broker clients (admins included) share the same brokerClient* params.
Used brokerClientTlsEnabled to determine if admin client should use TLS
Add Auth params and TLS options to Admin Client

Result
Fixes #2830

@merlimat - addressed your comments


Prevent users from being able to change their ABN in the seller application

This PR is related to:
https://github.com/AusDTO/dto-digitalmarketplace-frontend/pull/898
https://github.com/AusDTO/dto-digitalmarketplace-api/pull/733
Because ABN is provided on user signup now, this change will prevent the user from being able to change their ABN when editing their profile (but still allows admin accounts to do so).

This PR may need to be merged and deployed after the two other PRs, because once this is deployed users will be unable to save the ABN value in their application, so any user who starts an initial application before the other PRs are deployed may not already have their ABN in the application and won't be able to save their ABN value.
I suggest we deploy the frontend and API PRs, and monitor the initial application data for when the initial applications started before the ABN signup change are submitted. For those that hang around and may be submitted at a much further date, we may need to instruct ops how to handle these (e.g. manually adding the ABN to the application in the admin portal).

@tobiasbrown lets close this ticket off?


Add new domain for SibSUTIS

When switching to remote leaning, our university acquired a new domain for organizing work in Google services. The redirection from the new domain (sibguti.ru) to the old one (sibsutis.ru) is also configured.

@BorisBochkaryov  Siberian State University of Telecommunications and Information Sciences

@BorisBochkaryov  Siberian State University of Telecommunications and Information Sciences

@BorisBochkaryov Thank you for the update! Pull request merged.

@BorisBochkaryov Thank you for the update! Pull request merged.


AutoGluon Text training error due to data passing

When training AutoGluon Text (nightly) with a large file (2M rows), I get an error
distributed.core - ERROR -<PHONE_NUMBER> exceeds max_bin_len(2147483647)
Xingjian: " I’m suspecting this line: https://github.com/awslabs/autogluon/blob/ae5dd91e8892c6c8edd87e42442b8007fc9d60c5/text/src/autogluon/text/text_prediction/models/basic_v1.py#L[…]11, in which we are passing the train_data. This may potentially be large. "

@sflender Would I know if the fix works or not?

@sflender Would I know if the fix works or not?

the fix worked, the model is training. thanks!

the fix worked, the model is training. thanks!


redirect_uri invalid error with AuthTokenRetriever on macOS

I tried running the AuthTokenRetriever on macOS (Big Sur) and I kept getting an error in the opened browser for "redirect_uri invalid". When I modified Program.cs to output the AuthURL before sending it I noticed that redirect_uri was being set to 'http://<IP_ADDRESS>:8080/Reddit.NET/oauthRedirect' instead of 'http://localhost:8080/Reddit.NET/oauthRedirect'.
So if anyone sees the redirect_uri invalid error, just make sure you use 'http://<IP_ADDRESS>:8080/Reddit.NET/oauthRedirect'
I also updated all the .csproj files to use TargetFramework = "net7.0" instead of .net core 2.1. I'm not sure if that has anything to do with the error.

So you're saying the browser was changing the host?  I don't know much about Macs so I'm not much help here, but it sounds like you found a workaround.  Back in the 5.x days of PHP, I remember the php_mysql library had a weird bug where you had to specify <IP_ADDRESS> instead of localhost but that was ages ago.


Add module to read recob::Wire back into dataprep.

The dataprep modules read raw data or digits and write recob::Wire. It would be useful if that structure could be fed back into dataprep, in particular, so dataprep could be used to further process the output of the Wirecell reco module. Slavic (@vgalymov) has recently requested this for his studies.
The most likely resolution here will be to modify one or add to the modules in the dataprep module directory.

That is a good question. Originally I envisioned two separate workflows workflow where one runs through digits with some set of dataprep tools and then reruns with wires with some set of tools (e.g., ROIs display and storage). Then offline one would try to compare the two results in some way. The benefit of being able to run a combination of the two at the same time might perhaps be that it would allow for an eventual possibility to make a tool where the two information could accessed and used.

I was distracted this week preparing to present talks on the new channel status tool/service and did not make ny progress here. But this remains a high (but not highest?) priority issue.

Slavic, sorry I have been distracted with vacation and many meetings but am back looking at this and hope to get something in place next week. LMK if you are no longer interested.

It is not at all urgent, but I think would be a good feature to have available eventually.


Support other return types than number in sort callback

Currently, sort(fish, f => f.weight) only works if weight is a number. Would be nice if it supported strings too ✨

Hello @danmichaelo do you want to sort alphabetically or something like this?

Thanks! Would be nice if there was linking between the documentation for those two functions, I can add a PR for that


Becatrue/118/change bus to channel

This PR will close #118.
I changed all buses to channels, and set channel names as dbch and logch`.

I updated it. @kangz12345


Set up automated testing

We really need at least some basic tests for core functionality that we can automate with pytest and travis. This will also help with detecting issues as jax changes, by running tests against multiple versions of all the dependencies: https://docs.travis-ci.com/user/languages/python/#dependency-management
Here's a basic template: https://docs.python.org/3/library/unittest.html
and an example: https://github.com/f0uriest/GASTOp/blob/master/tests/test_fitness.py

Closing this for now that travis is set up and working, though we still obviously need to design more tests to exercise more parts of the code.


Need support added for new navigation

The extension doesn't work with the new navigation turned on GitLab
https://about.gitlab.com/blog/2023/05/01/gitlab-product-navigation/

Thanks for highlighting the issue, looking into it

@aridastidar
Couldn't exactly reproduce the issue, can you attach a video/screenshot for the issue?

+1 :)
Some update?

seems that this extension is dead :(
Someone can advise alternative working with new ui?

@clementbrunel launched a new release, can you try and lemme know if it works?

Apologies for the delay, but was busy with my job

Hi !
Thanks for your update !
Works now in our<EMAIL_ADDRESS>


GALAXY MANGA: Couldn't reach problem

Source information
ar.galaxymanga 1.4.30
Source language
Arabic
Steps to reproduce
When I try to browse the manga list  this error showing to me

Expected behavior
The list of manga/manhwa should be listed
Actual behavior
When I try to browse the manga list  this error showing to me

Tachiyomi version
0.15.3
Android version
Android 10
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.15.3.
[X] I have updated all installed extensions.
[X] I have tried the troubleshooting guide.
[ ] I have admitted that I am a clown by having checked this box, as I have not read these acknowledgements.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I will fill out all of the requested information in this form.

Cannot reproduce.

The issue still happening to me. What should do I do ?


[FEATURE] Re-ordering tags with Drag-and-Drop

Is this a unique feature?

[X] I have checked "open" AND "closed" issues and this is not a duplicate

Is your feature request related to a problem/unavailable functionality? Please describe.
I was updating my profile by adding some tags. I added all the tags. I just felt like re-ordering all the programming languages and organize them together. I went ahead to drag and re-order the tags but discovered the feature hasn't been implemented yet🙄.
We can enhance user experience by implementing a drag-and-drop feature that will allow users to organize tags in an order that makes sense to them, aligning with their preferences or the way they prioritize certain skills or interests without having to delete ones that they have already entered.

Proposed Solution
A solution similar to the following video that I am recorded from Upwork would be great as far as implementation is considered
https://github.com/EddieHubCommunity/BioDrop/assets/72060440/14c6ab56-a6ed-4c5f-8285-b68a5949d71a
Screenshots
No response
Do you want to work on this issue?
No
If "yes" to above, please explain how you would technically implement this
No response

This is a cool idea @akash19coder and will greatly help folks organize.  I am going to open this for discussion first as although you linked the potential UI, I am curious what packages or approach are best for implementation here in your opinion. Any thoughts? (Anyone is welcome to offer some ideas)

Hi, I think we can use either react-dnd or react-beautiful-dnd to implement this feature. (I have not used these packages personally as of yet )(I am new in open source learning and discussions so sorry for any inconvenience).

Hi, I think we can use either react-dnd or react-beautiful-dnd to implement this feature. (I have not used these packages personally as of yet )(I am new in open source learning and discussions so sorry for any inconvenience).

Thanks for starting the conversation! We are all learning together for sure.  I have heard react-dnd mentioned with next before as well, but personally have no experience with it so will wait for others to weigh in before we unlock this issue.

Great discussion 👍 Please note we already have react-sortablejs in the project for drag and drop sorting, I suggest we use the same library and do not add more

Thanks Eddie for the insight, I agree there is no use to import one more library for the same. I would love to give this a try and learn something new along the way if I can be assigned this issue. 😁

Thanks Eddie for the insight, I agree there is no use to import one more library for the same. I would love to give this a try and learn something new along the way if I can be assigned this issue. 😁

@Mohit-karwasra thank you for your enthusiasm, however as you are a new contributor to this project we ask that you choose issues where the points range from 1-3 (this is 5 points).

Can I take it up?

@Abhishek-90 I will assign this to you

I have added the drag and drop functionality to the profile tags. react-sortablejs also had a SwapPlugin which I used to swap items in a list. Please find below a screen record of the new behavior of profile tags.
https://github.com/EddieHubCommunity/BioDrop/assets/43419831/484ac22d-caa1-4391-b9db-cbbeb4687197

This is issue has two things to be taken care of:

Wrapping the tags with react-sortablejs to add a drag and drop functionality.
Once the wrapping is done, we need to modify the index of the tags as well.
Add the newly sorted to database.

The 2nd point above was not needed afterall. The SwapPlugin took care of it.
Raising a PR for the issue.

Hello, I have added the changes required to make the tags to swap amongst themselves. However, while testing I encountered this below error in the video.
https://github.com/EddieHubCommunity/BioDrop/assets/43419831/8b8c3479-fed1-460a-9a50-328abf3fbdc3
The reason for this error is in the below code snippet, from TagsInput.js file.
https://github.com/EddieHubCommunity/BioDrop/blob/e3767d0fca2e0ba37d4c5159c972283388cf8c6c/components/tag/TagsInput.js#L61C7-L77C12
As the input field is part of

 tags as a list item. This is ul tag is changed to ReactSortable tag as part of solution for this issue, refer below code snippet.
https://github.com/Abhishek-90/BioDrop/blob/af3d35ddb741374a3c0f6fa4228c99682352c0b9/components/tag/TagsInput.js#L70
Solution:  Move the Input field out of the
 tag and make the Tags appear below the field.

TagsInput component is being used on the events page as well, hence due to changes in the component the tags of the events are also having the drag-&-drop and swap functionality. Do we need that or we need to find a different solution.
Solution 1: We can create another component for the TagsInput one will have Drag and drop functionality and other won't(one used on events page)
Solution 2: We can add the props named isSwapable which if true we used ReactSortable instead of ul tag.


[Suggestion] Keep loaded model separate from context

I would like to use the shared library for serving an LLM through an inference endpoint that can serve multiple requests with varying temperature and max token settings. At the moment the LlamaModel seems to be merged with the context, and the only way to set the temperature or max tokens again would be by closing / freeing the model and loading it again with the correct configuration.
I would like to keep the model loaded though, and allocate separate contexts with different configurations per incoming completion request. Is that possible using the llamap.cpp API, and would you consider making changes to java-llama.cpp for that?

Yes, I think that would be a nice improvement. One thing I am working on anyway is to separate model specific and inference specific parameters. Currently, these can only be changed when constructing a LlamaModel.
I was thinking to overload LLamaModel#generate(String) with LLamaModel#generate(String, InferenceParameters). This way I can also enable to pass a context.

The requirements of this issue are implemented since a few versions, so I'll close this ticket. Let me know if you still have questions.


[DEV-1785] update base images' conda versions, consolidate RUN steps

This PR makes two connected changes:

Update the conda version our base images use to 4.9.2, and add a conda update conda step to both to keep it updated (for DEV-1785).
While adding the conda update conda calls, I added them to preexisting RUN steps so that we don't create additional layers (meaning our overall image is smaller) - and in the process noted that there were additional RUN steps in both that could be combined for consistency (and further minor size reduction). Small change - happy to split it out if we think there's good reason to.

Task linked: DEV-1785 [images] [saturn] miniconda image is more than a year old


When using * it gets styled inline everywhere

The usage of:
* { box-sizing: border-box; }
<html xmlns="http://www.w3.org/1999/xhtml" style="box-sizing: border-box;">
<head style="box-sizing: border-box;">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" style="box-sizing: border-box;">

Like it was suppose to? When you add it like that on the CSS file its doing that also.
Is this and issue ?


Running multiple times with pymc3 backend throws an error

When running infer multiple times with the pymc3 backend we get an error. It seems to be related to re-defining the pymc3 model more than once.

I have tried to fix the issue with redefining the model by using the del keyword to explicitly remove the model object after running infer. I am not sure if this is the way to go

fixed

We re-open as it doesn't work with the newest modifications

Remember that this could have problems with concatenate

Fixed in #92

Fixed by @wasowski in #112


Support for alternate 'idAttribute'

Many of our models use a custom 'idAttribute' to represent the model's id per Backbone's documentation.  I just realized there is an issue with Backbone-Associations and this feature after some digging.  On line 163 you set the idKey to the idAttribute (which defaults to 'id'):
idKey = currVal && currVal.idAttribute,

Then on line 230 you check:
if (currVal && data[idKey] && currVal[idKey] === data[idKey]) {

When you're using the default idAttribute of 'id' this works fine - as data['id'] does a lookup on the id attribute of the model.  But when using a custom idAttribute - eg. ProductId - data['ProductId'] always returns undefined.  I believe an alternate way to check would be:
if (currVal && data.attributes[idKey] && currVal.attributes[idKey] === data.attributes[idKey])

which should correctly resolve for both standard ids and custom idAttributes.
Thanks!

I found this problem persists in the latest version (0.6.2). I did not inspect the code thoroughly, but I see the line commited to fix this error changed. get the following error when my Model lacks an key named 'id'. Even when idAttribute is set. If I set both 'id' and idAttribute, I still get the error.
Uncaught TypeError: currVal._set is not a function
    at child.<anonymous> (backbone-associations.js:328)
    at underscore.js:76
    at Function._.each._.forEach (underscore.js:182)
    at child._setAttr (backbone-associations.js:260)
    at child._set (backbone-associations.js:233)
    at child.set (backbone-associations.js:197)
    at setAttr (backbone.stickit.js:267)
    at child.listener (backbone.stickit.js:177)
    at executeBound (underscore.js:758)
    at HTMLSelectElement.<anonymous> (underscore.js:771)


Plotting single channel image with show_sample results in incorrect plot output, with out without masks

🐛 Bug
Describe the bug
When I plot a sample that has a single channel image and masks, PIL is used in the background I think and it incorrectly converts the single channel image to a color image instead of keeping it in greyscale. The show sample func also adds a false border as seen below.
To Reproduce
I can't provide the underlying data, but this is what my train record looks like for the single channel dataset
import icevision
parser = icevision.parsers.COCOMaskParser(annotations_filepath="../../data/cv2_transfer/instances_slicks_test_v2.json", img_dir="../../data/tiled_image_slicks_test_v2")
train_records = parser.parse(data_splitter=icevision.data.SingleSplitSplitter())
record = icevision.Dataset(train_records[0])[0]
record

BaseRecord

common:
	- Filepath: ../../data/tiled_image_slicks_test_v2/S1A_IW_GRDH_1SDV_20200729T034859_20200729T034924_033664_03E6D3_93EF_vv-image_tile_0.png
	- Img: 512x512x3 <np.ndarray> Image
	- Record ID: 0
	- Image size ImgSize(width=512, height=512)
detection:
	- masks: [<icevision.core.mask.RLE object at 0x7fd1ccba2760>]
	- mask_array: <icevision.core.mask.MaskArray object at 0x7fd1ccb98190>
	- Is Crowds: [1]
	- Class Map: <ClassMap: {'background': 0, 'infra_slick': 1, 'natural_seep': 2, 'coincident_vessel': 3, 'recent_vessel': 4, 'old_vessel': 5, 'ambiguous': 6}>
	- Labels: [6]
	- Areas: [17586]
	- BBoxes: [<BBox (xmin:256.0, ymin:275.0, xmax:512.0, ymax:512.0)>]

and to show the plot that looks incorrect
icevision.show_samples([record],display_label = True,
    display_bbox=True,
    display_mask=True)

Expected behavior
Plot the image in greyscale, with colored masks on top
Screenshots

with labels:

Desktop (please complete the following information):

ubuntu 20.04

Additional context
I'm running icevision .12.0
Plotting the file directly with scikit-image .imshow looks as expected. In general I prefer scikit-image to using PIL because PIL makes too many assumptions about the data for my use cases (that it is rgb, that it is of a certain data type, doesn't support float32 fully https://github.com/python-pillow/Pillow/issues/1888)

Hey, is your image 8bit or 16bit in grayscale values? We have a set of grayscale record defaults for the purpose of using 16bit grayscale images (https://github.com/airctic/icevision/blob/master/icevision/core/record_defaults.py). It should be enough to just overwrite the template_record function https://github.com/airctic/icevision/blob/0e42d3d76731d0e70b1486ded0e0d498126c61d0/icevision/parsers/coco_parser.py#L113 of the COCOMaskParser.

Closing due to inactivity

I think this should be re-opened. one of the components that builds the grayscale default record seems to have been commented out and it is non trivial to find all the imports to set this tempalte record function up
https://github.com/airctic/icevision/blob/master/icevision/core/record_components.py#L453
ideally, the COCOMaskParser would handle reading grayscale or n-channel imagery by either using the grayscale record or the other record.

I tried to replace the commented out MasksRecordComponent with what looks like a newer component: InstanceMasksRecordComponent. but this doesn't work
class COCOMaskParserGrayscale(COCOMaskParser):
    def template_record(self) -> BaseRecord:
        record = super().template_record()
        record.add_component(GrayScaleRecordComponent())
        record.add_component(InstancesLabelsRecordComponent())
        record.add_component(BBoxesRecordComponent())
        record.add_component(InstanceMasksRecordComponent())
        return record

parsing doesn't fail, but show_results does
show_records(train_records[0:3], ncols=3, class_map=class_map)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [10], in <cell line: 2>()
      1 get_ipython().run_line_magic('matplotlib', 'inline')
----> 2 show_records(train_records[0:3], ncols=3, class_map=class_map)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/visualize/show_data.py:151, in show_records(records, class_map, display_label, display_bbox, display_mask, ncols, figsize, show, **draw_sample_kwargs)
    127 def show_records(
    128     records: Sequence[RecordType],
    129     class_map: Optional[ClassMap] = None,
   (...)
    136     **draw_sample_kwargs,
    137 ) -> None:
    138     partials = [
    139         partial(
    140             show_record,
   (...)
    149         for record in records
    150     ]
--> 151     plot_grid(partials, ncols=ncols, figsize=figsize, show=show)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/utils/imageio.py:141, in plot_grid(fs, ncols, figsize, show, axs_per_iter, **kwargs)
    138     raise ValueError("axs_per_iter has to be greater than 1")
    140 for f, ax in zip(fs, axs):
--> 141     f(ax=ax)
    143 plt.tight_layout()
    144 if show:

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/imports.py:88, in partial.__call__(self, *args, **kwargs)
     87 def __call__(self, *args, **kwargs):
---> 88     return self._partial(*args, **kwargs)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/visualize/show_data.py:56, in show_record(record, class_map, display_label, display_bbox, display_mask, display_keypoints, ax, show, figsize, **draw_sample_kwargs)
     44 def show_record(
     45     record,
     46     class_map: Optional[ClassMap] = None,
   (...)
     54     **draw_sample_kwargs,
     55 ) -> None:
---> 56     img = draw_record(
     57         record=record,
     58         class_map=class_map,
     59         display_label=display_label,
     60         display_bbox=display_bbox,
     61         display_mask=display_mask,
     62         display_keypoints=display_keypoints,
     63         **draw_sample_kwargs,
     64     )
     65     show_img(img=img, ax=ax, show=show, figsize=figsize)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/visualize/draw_data.py:383, in draw_record(record, class_map, display_label, display_bbox, display_mask, display_score, display_keypoints, font_path, font_size, label_color, mask_blend, mask_border_thickness, color_map, prettify, prettify_func, return_as_pil_img, exclude_labels, include_only)
    362 def draw_record(
    363     record,
    364     class_map: Optional[ClassMap] = None,
   (...)
    381     include_only: List[str] = None,
    382 ):
--> 383     sample = record.load()
    384     return draw_sample(
    385         sample=sample,
    386         class_map=class_map,
   (...)
    402         include_only=include_only,
    403     )

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/core/record.py:66, in BaseRecord.load(self)
     64 def load(self) -> "BaseRecord":
     65     record = deepcopy(self)
---> 66     record.reduce_on_components("_load")
     67     return record

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/core/components/composite.py:83, in TaskComposite.reduce_on_components(self, fn_name, reduction, **fn_kwargs)
     81 results = {}
     82 for task, composite in self.task_composites.items():
---> 83     result = composite.reduce_on_components(fn_name, reduction, **fn_kwargs)
     84     results[task] = result
     86 return results

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/core/components/composite.py:132, in Composite.reduce_on_components(self, fn_name, reduction, **fn_kwargs)
    130 results = []
    131 for component in self.components:
--> 132     results.append(getattr(component, fn_name)(**fn_kwargs))
    134 if reduction is not None and len(results) > 0:
    135     out = results.pop(0)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/core/record_components.py:227, in GrayScaleRecordComponent._load(self)
    226 def _load(self):
--> 227     img = open_gray_scale_image(self.filepath)
    228     self.set_img(img)

File ~/work/.ice-env/lib/python3.9/site-packages/icevision/utils/imageio.py:69, in open_gray_scale_image(fn)
     67     img = open_dicom(str(fn))
     68 else:
---> 69     img = PIL.Image.open(str(fn))
     71 img = np.dstack([img, img, img])
     72 img = img.astype(np.float32)

File ~/work/.ice-env/lib/python3.9/site-packages/PIL/Image.py:2975, in open(fp, mode, formats)
   2972     filename = fp
   2974 if filename:
-> 2975     fp = builtins.open(filename, "rb")
   2976     exclusive_fp = True
   2978 try:

FileNotFoundError: [Errno 2] No such file or directory: 'None'


Make SAI sstable components immutable

When a SAI index is rebuild, create new index files rather than overwriting existing files, making SAI index components
files essentially truly immutable (once initially written).
To do this, SAI index components have a new parameter, their "generation": for a given version, the component with the
highest generation is the most recently written one. That generation is added to the filename, but with the special case
that the initial generation (for a version), generation 0, is special cased and not added to the filename, and this for
backward compatibility. This means in particular that filenames are completely unchanged by this patch up until one
triggers an index rebuild that does not also correspond to a version change.
One implication is that on a rebuild, old SAI index components are not deleted anymore (previously, they were deleted at
the end of the rebuild, and the rebuild was re-creating new versions of the same files, essentially overwriting the file
but in 2 steps) and will remain on disk. The intent here is that this pave the way for allowing index "live" rebuild,
that is rebuilding an index without making the index un-queriable during the rebuild. However, this patch does not
implement such "live" rebuild, leaving it as a follow up. As of this patch, the old index component files are also never
deleted, though over time, compaction will get rid of them by virtue of replacing the sstables that have those
components.
Technically, the generation added is not incremented "per component", but rather per "group" of components, where
"group" means (for a particular sstable) either 1) all the per-sstable components or 2) all the components of a
particular index. As an example, suppose the components for an sstable are (where 1 is the generation, and including
only a subset of real components to keep the example shorter):
<sstable X>-SAI+CA+1+GroupMeta.db
<sstable X>-SAI+CA+1+TokenValues.db
<sstable X>-SAI+CA+1+PrimaryKeyTrie.db
<sstable X>-SAI+CA+1+GroupComplete.db
<sstable X>-SAI+CA+myIndex+Meta.db
<sstable X>-SAI+CA+myIndex+TermsData.db

In this example, the per-sstable group is complete, but not the group for myIndex (missing at least the
-ColumnComplete component). Also, the per-sstable group is at generation 1, while the myIndex group is at generation
0.. If a rebuild is triggered, the created components will be (assuming CA is the current version and again using a
subset of the actual components for simplicity):
<sstable X>-SAI+CA+2+GroupMeta.db
<sstable X>-SAI+CA+2+TokenValues.db
<sstable X>-SAI+CA+2+PrimaryKeyTrie.db
<sstable X>-SAI+CA+2+GroupComplete.db
<sstable X>-SAI+CA+1+myIndex+Meta.db
<sstable X>-SAI+CA+1+myIndex+TermsData.db
<sstable X>-SAI+CA+1+PostingLists.db
<sstable X>-SAI+CA+1+ColumnComplete.db

In other words, the generation for the per-sstable group and the myIndex one are independent, but the generation are
linked within a group (the <sstable X>-SAI+CA+1+PostingLists.db file is at generation 1 like the rest of that group
and not 0, despite there being no generation 0 file before the rebuild).
The reason generation are linked "by group" is that 1) we need to at least link the generation to the "completion
marker" components to the rest of the component of the group this is a marker for, and linking the whole group as above
feels like the cleaner solution and 2) linking the generation of all the SAI components of a given sstable would
prevent us from being able to rebuild a single index without rebuilding the per-sstable components/other indexes.

Some technical details on the changes here for reviewers sake.
The core of the changes of this patch is IndexDescriptor, which handles the SAI components for an sstable, but that file is change fairly drastically. The rest of the changes is by and large propagating the changes to that file to all the places in the code where it is used. And that's that's probably the right place to start looking at for a review.
This patch changes the fact that there can be only a single file for a given sstable, version, component type and index "context", by adding the additional generation. This has the implication that we need to distinguish a bit more about the components for when we write a new index, from the current "active" components of an sstable. In the former case in particular (writing new index), the code used to be able to not care about what components previously existing, but with this patch, we need to know what exists and potentially bump the generation accordingly.
Additionally, as explained in the description above, the generation is linked between components of the same "group", but the existing code didn't had any abstraction for that notion of "group" of components (the notion always existed in a sense, but only abstractly), which made it really hard to make the intended changes. So the bulk of the changes in this patch is really to introduce this notion of "group" of components, where each sstable has a per-sstable "group", for all its per-sstable components, and per-index groups that, for each index, groups the per-index components of that index.
This notion of group of components in the patch is the ComponentGroup class (it is an interface and its only concrete implementation is a private subclass in IndexDescriptor in practice). It is mostly used through it's 2 sub-interfaces, ComponentGroup.Writer and ComponentGroup.Reader, whose goal is to make it more explicit in the code when we're writing new components, versus when we just read existing ones. With that, IndexDescriptor basically tracks the ComponentGroup active for an sstable, and most of the methods of IndexDescriptor are moved to ComponentGroup (which in some cases saves some duplication of code between per-sstable and per-index stuffs).
Another slight change to IndexDescriptor is that it always tries to "discover" the components on disk: previously, there was an IndexDescriptor.createNew(...) method that was used when building index and wasn't checking anything existing on disk because even if something did (because it's a rebuild), we were overwriting things anyway. With this patch however, we need to know what exists, if anything, to bump the generation if appropriate.
The "discovering" of the existing components of a sstable also has to change. Before this patch, we never "scanned" the disk for this: there is only a finite and fairly small number of versions and component kind in existence, so we just tried all combination and looked if the corresponding file existing. With generations, we can't really do that anymore, because in theory there is no saying what the most recent generation may be (in practice, it's never going to be big, there is only so may rebuild a sstable may be part of in its lifetime, but it's still hard to cap it completely). So this patch change the discovery to actually scan the sstable directory for all the files belonging to the sstable it is a descriptor of, parse the file names to see which ones are SAI components and what they are, and go from there. This discovery is in IndexDescriptor.populateComponents, but it rely on being able to parse a SAI component file name and that's added in Version.tryParseFilename (because the exact format of file names actually depends on the version).
As a minor aside, this probably make the creation of IndexDescriptor instances a tad more costly that before, and the existing code had a tendency of calling IndexDescriptor.create even when a reachable instance already existed so the patch change a few of those.
Lastly, I'll point out that this patch adds a class to represent a specific component, that is one that directly map to a single file on disk, and the patch calls this IndexComponentInfo. I'm only mildly a fan of that name, and in a perfect word I'd have call this just IndexComponent, but that name is taken for "the enum of all the kind of index components there is". I'd personally be in favor of renaming that enum to IndexComponentKind, and then use IndexComponent for the IndexComponentInfo of this patch, but I didn't do it in this patch because I wasn't sure if this would be "controversial". But if reviewers also thinks this is worthy renaming, happy to add it.

Lastly, I'll point out that this patch adds a class to represent a specific component, that is one that directly map to a single file on disk, and the patch calls this IndexComponentInfo. I'm only mildly a fan of that name, and in a perfect word I'd have call this just IndexComponent, but that name is taken for "the enum of all the kind of index components there is". I'd personally be in favor of renaming that enum to IndexComponentKind, and then use IndexComponent for the IndexComponentInfo of this patch, but I didn't do it in this patch because I wasn't sure if this would be "controversial". But if reviewers also thinks this is worthy renaming, happy to add it.

+1 to renming. How about IndexComponentType or ComponentType?

How can we roll this out to a live cluster ?
I imagine that the procedure is:

update the code on all the writers to a version that contains this change
wait for the system to settle
update the compactors with this code and enable the feature

In case of problems the first version of the files is always there (because we don't delete the first version when rebuilding the indexes) so it is safe to rollback the writers to a code that doesn't support this feature (but they will see the old version of the files, before the rebuild).
Is this correct ?

Is this correct ?

Yes.

As mentioned on slack, the code moved to use the TOC as a mean to find the current generation of SAI components to use, and this to avoid having to scan the whole sstable directly every time as that can be slow for CNDB. But to avoid relying on the presence of the TOC too much, the previous code that scan disk is still there as a fallback if the TOC doesn't exists: because it's only a fallback for hopefully exceptional case (we lost the TOC somehow), having it be maybe a bit slow at times shouldn't be a problem.
But it does mean that if the TOC exists, then that is now relied on to know if SAI components exists or not, so hopefully existing TOC files are not broken (I don't think they are, but just pointing out that the increased reliance on them is kind of new).
I just pushed those last changed, so will wait to see if there is CI failures, but otherwise I think I've addressed all review feedback and any known issues.

I know this was approved before, but needed to make a few changes for the sake of the cndb, so re-requested review to have a look on the last changes (I've kept them in separate commit so only those probably need a look). There changes basically do the following:
a reloading method is added to re-discover things from disk. That's not used in C* proper (outside of the test I added) because any change to components is done by the done through methods of `IndexDescriptor', but in cndb, components are modified "externally" by compactor, so we need this to tell writers to re-check.
I've change a bit when IndexDescriptor does component discovery from disk. Historically, the per-sstable components were looked up when creating the IndexDescriptor instance and then each per-index "group" was looked up lazily, the first time any method for the index context was used. And so far I had kept that. But that lazy loading also means it's not entirely clear when we do discover components: it could well happen at any place were we check anything on per-index components, which is a lot of place, and in the cndb, with NSS, that's not great because things can change in remote storage concurrently of what any particular writer is doing, but a read of the TOC will go to NSS and see whatever the last version is in remote, so that makes it really hard to reason about. Tl;dr, I've changed the code so that IndexDescriptor always loads everything eagerly when it is created. It means we need to pass the existing index to the ctor, but that's easy enough. As it happens, it's also a bit more efficient as it means we only read the TOC once, and not once per "group" of components (obviously doesn't matter too much, but well, still technically more efficient).
The case of rebuilding indexes that were at an older version to a newer one was slightly broken (we were not saving the correct components in the TOC because the wrong version was used at one point) so this fixes it and add a test for it.


Json MappedBy

It should be possible to specify a surrogate object for JSON I/O which shall be related to the object that shouldn't be I/O'd directly via a RareObjectMapper mapping.
e.g. NOTE(MyType, Json::MappedBy) as a class or field annotation; perhaps find some way to signal that a default surrogate exists for JSON (this is tricky, as I've found no way to aggregate a list of reflected objects; so sadly this may require a specialization in the global scope or some such mechanism.

#84


feat(hbase): bypass server for hbase writing  hugegraph-loader (BETA)

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Feature Description (功能描述)
1、Support write vertex/edge directly to KV storage
2、Just support customString and customNumber ID now;
The current version of the client feature has been launched internally.  continue to submit loader gen HFile feature!
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

loader part code for #347, #332


A11y_AzureToolsStorageExplorer_ConfigureSoftDeletePolicy_ScreenReader: NVDA announces entire content present in screen when we tab to shift NVDA focus to "What is Soft Delete?" link.

“Check out Accessibility Insights! Identify accessibility bugs before check-in and make bug fixing faster and easier.”
GitHubTags: #A11y_AzureToolsStorageExplorer; #A11yMAS; #A11YTCS; #DesktopApp; #MAS1.3.1; #ScreenReader; #NVDA; #JAWS; #Win32;
Environment Details:
Application Name: Azure Tools Storage Explorer
Application Version: 1.10.0
Windows10
Repro Steps:

Launch Storage Explorer
Navigate to "Open Connect dialog" pane and Sign in to Azure.
Go to "Manage Accounts" pane and Select the subscriptions you will use. Once selected, click on "Apply" button.
Navigate to "Blob Containers" of the selected subscription, right click and Select "Configure Soft Delete Policy" and hit enter.
Start NVDA.
Navigate to "What is Soft Delete?" link.
Verify NVDA announces entire content present in screen when we tab to shift NVDA focus to "What is Soft Delete?" link.

Actual:
Issue1: While navigating on the screen using Tab key for the first time, NVDA announces entire content present in screen when we tab to shift NVDA focus to "What is Soft Delete?" link.

Same issue is observed for "Enable Soft Delete" radio button in this Scenario.
Same issue is observed in "JAWS" also.

Issue2: And also for second time While navigating on the screen using Tab key, NVDA announces "dialog cancel button What is Soft Delete?" when NVDA focus is on "What is Soft Delete?".
Same issue is observed in "JAWS" also.
Same Issue1 is observed in below Scenarios:

SAS URL
Here in this Scenario NVDA announces entire screen content when Shared Access Signature screen gets opened.

Same Issue2 is observed in below Scenarios:

Delete Popup
Here in this Scenario NVDA announces dialog title when shift focus from delete control to cancel control.
Rename blob
Here in this Scenario NVDA announces dialog title when shift focus from New name edit box to Rename control.
Update Access Tier
Here in this Scenario NVDA announces entire screen content when Press Tab key to shift focus to "Learn More" link.
Same issue is observed in "JAWS" also.
Set Container Public Access Level
Here in this Scenario NVDA unnecessary announces other content present on the screen when Tab to shift keyboard focus to "No Public Access" radio button.
Same issue is observed in "JAWS" also.
Undelete Selected
Here in this Scenario NVDA unnecessary announces other content present on the screen when Tab to shift keyboard focus to "Cancel" control for the first time.

Expected:
Issue1: NVDA should not announce entire content present in screen when we tab to shift NVDA focus to "What is Soft Delete?" link, Instead NVDA should announce only link name with description.
Issue2: And also for second time While navigating on the screen using Tab key, NVDA should not announce" dialog cancel button What is Soft Delete?" when NVDA focus is on "What is Soft Delete?", Instead NVDA should announce only link name with description.
UserImpact:
Screen reader users gets confusion if screen reader start announcing entire content of the screen at a time when shift screen reader focus to link.
Recommendation:
https://microsoft.sharepoint.com/teams/msenable/mas/Pages/browse-fixes.aspx
MAS Reference:
https://microsoft.sharepoint.com/teams/msenable/_layouts/15/WopiFrame.aspx?sourcedoc={54f28d1f-a2d1-4dcd-84e1-5c9b87e8aba4}
Attachment for Reference:
A11y_AzureToolsStorageExplorer_ConfigureSoftDeletePolicy_ScreenReader.pptx

@Suraj1200499, I can confirm this bug on NVDA. But I also tried Jaws and cannot reproduce this issue there. The screen reader doesn't announce the whole page when navigating using Jaws. Could you please show me a recording with the speech history from Jaws to let me see the exact repro steps for Jaws?

This is the default behavior of NVDA. NVDA enters a web page in browse mode by default. It will read the entire page the first time it gets loaded. There is an option for this "Say all on loaded" behavior and is turned on by default.
The reason that some dialog doesn't have this issue is that we gave the initial focus to an element which is considered to be a control element by NVDA. In this case, NVDA automatically enters the focus mode and don't read the entire page.
Ryan Shugart, the accessibility expert suggest us not to change the default behavior of the screen reader. We are going to mark this issue (and the equivalent ones for other screen readers) as won't fix.

@JasonYeMSFT
The bug is By design, As per our process By design bug will be validated by our SME's. So I had put this bug in their note. Once I get a reply from them, I will update the comment.

We got reply from SME to close the bug.
Hence Closing the bug.

GithubTags:#Closed;

GitHubTags:#BM-TCS-StorageExplorer-Win32-Sep2019;


Migrate bus to the spring cloud function based messaging

For Spring Cloud Bus >= 2.2.x
without this change Spring Cloud Bus will disable any functional configuration which means that a Stream application using Spring Cloud Stream's functional configuration will not work! If you're using the legacy (@EnableBinding) approach - everything will work fine.

duplicates #227


When I'm trying to load marzipano.js in ionic framework it says "GET http://localhost:8100/marzipano.js net::ERR_ABORTED 404 (Not Found)". Is there any possibility to add marzipano in Ionic Framework?

When I press CTRL + Link over  in VSCode i'm able to open it.
I even tried : type="text/javascript">
Thanks in advance!♥

What happens when you go to http://localhost:8100/marzipano.js in your browser?

Thanks for your response.
I'm new to angular and ionic.
Help me to resolve this.
♥Thanks♥

The issue seems to be that your development server isn't configured to serve the marzipano.js file. I'm not familiar enough with Angular to be able to help. Sorry.


DangerousJourneys Mythus sheet updates

Changes / Comments
Update to Rev 0.96
Added Psychogenics and Powers.
Added Bestiary (monster/NPC) view
Added Heka automatic computation
Added an upgrade script to detect when an older character is being opened for the first time in this latest revision, to ensure new variables are properly set
Included spreadsheet to assist in organizing character/weapon/armor/Casting/characters into strings for import.
Roll20 Requests
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

[ x] Does the pull request title have the sheet name(s)? Include each sheet name.
[x ] Is this a bug fix?
[x ] Does this add functional enhancements (new features or extending existing features) ?
[x ] Does this add or change functional aesthetics (such as layout or color scheme) ?
[x ] If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
[ ] If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Hello,
I am unable to merge your pull request because it has failed our validation process. Please consult the following link for information on our character sheet development process works. https://roll20.zendesk.com/hc/en-us/articles/360037773393-Building-Character-Sheets

Darn.  Don't know how those quotes snuck into the JSON.  Will re-update.
Not sure from messaging what has failed on the html.  But will try to figure it out.


Added titles to home page and about us link to footer

nice job @Ehnvik!
This PR won't be merged, there's no sense. But everything looks great!


ECMPS Test Files Group 35

./src/utils/selectors/monitoringPlanEvalData.test.js
./src/utils/selectors/monitoringPlanFormulas.test.js
./src/utils/selectors/monitoringPlanFuelData.test.js
./src/utils/selectors/monitoringPlanLEEQualifications.test.js

Private Zenhub Image
Private Zenhub Image
Private Zenhub Image
Private Zenhub Image

PR: https://github.com/US-EPA-CAMD/easey-ecmps-ui/pull/1225


robot.http - invalid array length Allocation failed - process out of memory

This might be more of a coffeescript question, but I am trying to build a hubot script that pulls down train departure times from real time trains. I'm using robot.http to request some JSON from their API, but for certain requests returning ~300 lines of JSON the hubot crashes and I get the following error:
<--- Last few GCs --->

   36578 ms: Scavenge 984.4 (1023.4) -> 984.4 (1023.4) MB, 0.2 / 0 ms (+ 5.6 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
   36934 ms: Mark-sweep 984.4 (1023.4) -> 601.0 (640.9) MB, 356.2 / 0 ms (+ 5.8 ms in 2 steps since start of marking, biggest step 5.6 ms) [last resort gc].
   37190 ms: Mark-sweep 601.0 (640.9) -> 601.0 (640.9) MB, 256.4 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1261ab9e3ac1 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/Users/Jack/git/hal-slackbot/scripts/realtimetrains.coffee:31] [pc=0x213d10c979ab] (this=0x1261ab904189 <undefined>,err=0x1261ab904101 <null>,httpRes=0x1b4a5d0e97f9 <an IncomingMessage with map 0x51550d70841>,body=0x1b4a5d0ea2d9 <Very long string[17847]>)
    2: /* anonymous */ [/Users/Jack/git/hal-slackbot/node_modules/scoped-http-client/src/index.j...

FATAL ERROR: invalid array length Allocation failed - process out of memory
Abort trap: 6

The hubot script giving me this error is:
module.exports = (robot) ->

# user = <user_name>
# pass =  <password>

    robot.respond /trains from (.+) to (.+)/i, (res) ->
        from = res.match[1]
        to = res.match[2]
        rtt_call =<EMAIL_ADDRESS>        robot.http(rtt_call)
        .header('Accept', 'application/json')
        .get() (err, httpRes, body) ->
            res.send body
            data = null
            try
                data = JSON.parse body
                if data.services != null
                    res.send "Next trains from #{from} to #{to}:"
                    i = 0
                    while i < data.services.length
                        train = data.services[i]
                        if i < 5
                            train.locationDetail.platform
                            response = "The #{train.locationDetail.origin[0].description} to #{train.locationDetail.destination[0].description}"
                            response += ", from platform #{train.locationDetail.platform}"
                            response += ", departs at #{train.locationDetail.realtimeDeparture}"
                            res.send response
                            i++
                else
                    res.send "Sorry, there's no trains today"
            catch error
                res.send "Ran into an error parsing JSON :("
                return

Is there a way of perhaps cropping the data returned by the http call? Although perhaps I am doing something wrong – it seems odd to me that 300 lines of JSON is too much for the bot to handle?
Any suggestions would be greatly appreciated.

Is this issue still a problem? Or did you find a way through it in the while?

No, always seemed to cause a crash - sort of abandoned the project as a result. Are you experiencing the same issues?

No I was just beginning to clean up issues systematically so we can see clearer :)
But I already had to deal with very big payloads and it never crashed for me.

Likely my poor programming skills ;) lets close the issue to help things move along.


Add as struct

Merges Errol's change for results as struct.

The two MacOS checks fail but the workflow file from upstream allows for them to fail:
          # db: A DB's brew package name in macOS case.
          #     Set a name "db<EMAIL_ADDRESS>when using an old version.
          # MariaDB lastet version
          # Allow failure due to the following test failures that rarely happens.
          # https://github.com/brianmario/mysql2/issues/1194
          - {os: macos-latest, ruby: '2.6', db: mariadb, allow-failure: true}
          # MySQL latest version
          # Allow failure due to the issue #1194.
          - {os: macos-latest, ruby: '2.6', db: mysql, allow-failure: true}


Add support for font colors in tracks

Actual Behaviour
Currently, Track model does not contain font color
Expected Behaviour
Add support for font color in model and UI both and add corresponding migration in realm database
The font color must be HTML compliant
Would you like to work on the issue?
No

sure will work on it after the json api work. please assign me this.

@iamareebjamal
i think this work can be integrated with the issue regarding implementing json api.
i think this can be closed then.

Not really, this is more of a UI thing than model thing.

title seems inappropraite i think. support for font-color is added for tracks but not used right.
something like "using font-color attribute for track layout".
anyways should i start working on it?

Font color is not in model yet and neither is used in UI. That's why this title. Yes, you can work on it, but occasionally check out and contribute to the JSONAPI project

my main focus is on that right now. sure.


Add to Gymnasium's third-party list of environment

Hey, I just came across your project and it looks really cool
Would you be interested in adding the project to gymnasium's list of third-party environments?
https://gymnasium.farama.org/environments/third_party_environments/

Hello,
I would greatly appreciate it if you could add this project to gymnasium's third-party list.
Here's a quick status update on the project:
I'm currently working on creating a video that explains the theoretical background of the modeling approach, specifically focusing on the Disjunctive Graph approach for the Job Shop Problem.
The style of the video will be inspired by 3blue1Brown's educational content.
Once it's completed, I'll include it in the README.md for easy access.
Additionally, I've conducted some research on the Job Shop Problem using this environment, although the results have not been published yet.
As soon as they are, I will provide a link in the README.md to the code I used for the experiments.
Best Regards,
Alex

No worries, when you are ready, could you submit a PR to Gymnasium modifying the docs/envs/third-party.md file with the project's details


Nakamoto signer docs Updates

Use the following template to create your pull request
Description
Updated User guide section in regards towards running a sBTC Signer

Added new page "Nakamoto Signer Get Started"
this includes the latest planned configuration set up for running a signer, text describing how to run a custom signer if needed
Added a page describing the lack of signer in Developer release

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Features/RFC

hey guys, thank you for the amazing work! i have a few feature ideas for auth module....

setLastLoggedIn hook
socket presence (current authenticated/subscribed userIds for Resource—must be using sessions)
register custom hooks to be run on auth events—onLogin 'onLoginSuccess' etc

is it possible to implement any of above without adding functionality to feathers-authentication?
if not, i'm happy to contribute! the setLastLoggedIn hook is pretty straightforward, would need some direction on socket presence :)

@justingreenberg Sounds great!  We would love the help.
I think you could implement the lastLoggedIn on two services.  (Related to the third question, you can use hooks on auth services.)  I'd put one on the auth/local service as an after hook, and another on the users service, which would probably make the most sense as a before hook for when new login details are being saved to the user during an OAuth login.
For socket presence, both socket.io and Primus keep a list of connected users, which is used here: https://github.com/feathersjs/feathers-socket-commons/blob/e357f8d7d98f5ec11d9c687d1cba51899079852f/src/events.js#L79
and passed from here:  https://github.com/feathersjs/feathers-socket-commons/blob/e357f8d7d98f5ec11d9c687d1cba51899079852f/src/index.js#L15.  I'm not sure if it will contain the user information you need, but that's the first place I'd check.


Allow drag-clicking before symbol lookup

This makes it possible for touch and graphics tablet users to do symbol lookups in the code editor by ctrl+clicking on a symbol. (Previously this wasn't possible because drag events invalidated the click).
As a consequence, starting a selection, hitting ctrl and releasing the mouse over a symbol triggers the lookup. Please let me know if this behavior is considered a bug, so that I can find a cleaner solution (probably involving some sort of maximum distance from the selection start).

Please let me know if this behavior is considered a bug

Yeah this introduces a bug, see #32232

I don't think this is the same bug. I can't replicate selection or clicking on the minimap to open the documentation. Can you elaborate on how to replicate the bug?

Apologies should have been clearer, I meant releasing the mouse at the end of a selection going to help would be a bug.

I added a fix for the unwanted behavior: The lookup is only canceled when the user drag-clicks more than one character.

Whoops, that was the wrong branch. Sorry!

I resolved the conflict, this should be ready to merge.

This is still desired, if anyone wants to take a look at it.

Thanks!


Large files do not render correctly

If I open a file with more 1000 lines it does not not display correctly after a certain point it only shows blank lines.
Also for these large files widget uses a lot of CPU and RAM.

I saw that the problem of lines is fixed by adding height to the GutterStyle property. The performance is good for me, but the situation worsens when typing.

It seems already discussed in other issues, like #255 , where I found this https://github.com/flutter/flutter/issues/128575


Hugo fails to display changes in browser

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.60.1/extended linux/amd64 BuildDate: unknow

Does this issue reproduce with the latest release?
I'm assuming so, I just rebuilt and reinstalled Hugo from the AUR to try and fix this issue.
Issue
I have been using the following method to write pages locally for months without any issues.

I run hugo server on a local machine
Open localhost:1313 in my browser
See changes reflected on page when I edit the files

Now, the files are not updating. I edit them and I see the console output of Hugo detects the changes, but they are not updated on the browser. It only seems to be new files as well. I edited the contents of a pre-existing file and the changes were shown, but the file I created today won't even show up.
I tried running with hugo server --disableFastRender but this did not help

Did you figure something out here? Just a guess here, but I guess it has to be related to your site or there would be many more people piling on here. I am on 0.61/0.62 and it's working well. If you have your site in a public repo I could see if anything jumps out at me. Maybe try updating to the current release too? Is the livereload.js script getting appended to your page with the correct port?

Yeah i'm not too sure what happened tbh. I was giving someone a demo on how simple hugo was to use and it just... didn't work.
I wouldn't have been suspicious if not for the fact it was doing the same thing on my desktop at home. Not sure what to say, it seems like it fixed itself within a few hours.


investigate hhvm build failure

investigate why hhvm builds fail and fix them. we started ignoring the fail in #11

Looks like failures are triggered by Puli.

Confirmed, puli does not support HHVM ATM: https://github.com/puli/cli/blob/master/.travis.yml#L22

Puli is no longer a dependency to this bundle. We should consider rewrite our tests so they are not dependent of Puli.


[BUG] Provider BTP terrraform init

Is there an existing issue for this?

[X] I have searched the existing issues

What type of issue are you facing
bug report
Describe the bug
When we have launched  the terraform init we have this error:
Initializing the backend...
Initializing provider plugins...

Finding sap/btp versions matching "~> 1.6.0"...
╷
│ Error: Failed to install provider
│
│ Error while installing sap/btp v1.6.0: could not query provider registry for registry.terraform.io/sap/btp: failed to retrieve authentication checksums for provider:
│ the request failed, please try again later: Get "https://github.com/SAP/terraform-provider-btp/releases/download/v1.6.0/terraform-provider-btp_1.6.0_SHA256SUMS": tls:
│ failed to verify certificate: x509: certificate signed by unknown authority

Expected Behavior
Expected behavior is described in step 1 https://developers.sap.com/tutorials/btp-terraform-get-started.html
Steps To Reproduce
No response
Add screenshots to help explain your problem
No response
Additional context
No response

@jizquga Can you please check your network and firewall settings. The error you are encountering seems to be caused by the Terraform CLI not being able to make a call to the Terraform registry which gets forwarded to GitHub.
I retried this first step and the provider could be installed, so there seems to be something in the setup on your local machine that causes this issue

I think the error is related with the client certificate that we are using, I don't know if you can send me the client certificate we need to have configure in our side.
Thanks and regards

@jizquga This is something I cannot support you with. Please contact your IT support for supporting on that issue.


changed match validation regex to start username charcter only with word

Closes: #1
Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[x] Tests pass
[x] Appropriate changes to README are included in PR

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.


[Wallet] Set default gradle properties

Description
Set default gradle properties (JVM memory and enabling parallel).
This is so the app compiles fine even if the developer didn't set GRADLE_OPTS in their environment.
Tested
App compiles fine.
Other changes
N/A
Backwards compatibility
Yes.

Will this overwrite the values we set in CI? circle has different limits so just want to make sure it'll still work okay

@cmcewen apparently gradle.properties have precedence over GRADLE_OPTS according to https://docs.gradle.org/current/userguide/build_environment.html


(minor) formatting in IntellIJ?

just a quick question - any tips on how to configure IntelliJ (CLion) so that it formats code inside the html! macro? Thanks!

Hi @yatesco!
Sorry for the late response.
Unfortunately I don't have an answer here, as I don't use JetBrains IDEs. But I think it's unlikely as Maud's syntax is entirely custom. You'd need a bespoke formatter plugin I believe.
LMK if you have any further questions!


Documentation: Enhancement Of Contributing.md

The contributing.md file is not in a structured manner. Also some sections are missing. I can enhance it in a well structured manner with proper details and some additional sections also that will make the contributors easier to contribute. Please can you assign me this task?
Wanted to do the following changes :-

Enhance the project structure
Add need help with the basics section
Add project structure section
Add additional method of contributing through GitHub desktop section
Add good coding practices section
Add the help and support section
Add the pull request section
Add issue report process section

Can you please mention the URL of the file in which you want to update these changes?
@milansamuel609

This one
https://github.com/abhisheks008/DL-Simplified/blob/main/CONTRIBUTING.md

Cool go ahead. Quick reminder: this type of issue will only be labelled as Level 1.

Cool go ahead. Quick reminder: this type of issue will only be labelled as Level 1.

Sure