error happens during ipv6 network rename

E, [2018-01-17T03:44:40 #29964] [task:90] ERROR -- DirectorJobRunner: Attempted to compare a version 4 CIDR with a version 6 CIDR.
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/netaddr-1.5.1/lib/cidr.rb:726:in `contains?'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/manual_network_subnet.rb:109:in `is_reservable?'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:149:in `block (2 levels) in find_network_and_subnet_containing'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:149:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:149:in `find'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:149:in `block in find_network_and_subnet_containing'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:148:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:148:in `find_network_and_subnet_containing'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/ip_provider/ip_provider.rb:95:in `reserve_existing_ips'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_network_reservations.rb:66:in `add_existing'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_network_reservations.rb:25:in `block in create_from_db'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_network_reservations.rb:24:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_network_reservations.rb:24:in `create_from_db'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/network_reservation_repository.rb:10:in `fetch_network_reservations'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_repository.rb:18:in `fetch_existing'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_plan_factory.rb:38:in `desired_existing_instance_plan'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:242:in `create_existing_instance_plan'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:187:in `create_existing_instance_plan_with_az'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:155:in `block (2 levels) in create_instance_plan_based_on_existing_ips'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:149:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:149:in `block in create_instance_plan_based_on_existing_ips'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:144:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:144:in `create_instance_plan_based_on_existing_ips'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:67:in `block in place_existing_instance_plans'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:66:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:66:in `place_existing_instance_plans'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/static_ips_availability_zone_picker.rb:25:in `place_and_match_in'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/plan.rb:28:in `assign_zones'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/placement_planner/plan.rb:14:in `create_instance_plans'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/instance_planner.rb:18:in `plan_instance_group_instances'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/assembler.rb:64:in `block in bind_models'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/assembler.rb:61:in `each'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/assembler.rb:61:in `bind_models'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:79:in `block (2 levels) in perform'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/event_log.rb:105:in `advance_and_track'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:73:in `block in perform'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/lock_helper.rb:13:in `block in with_deployment_lock'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/lock.rb:79:in `lock'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/lock_helper.rb:13:in `with_deployment_lock'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/update_deployment.rb:59:in `perform'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:99:in `perform_job'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:34:in `block in run'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh_common-0.0.0/lib/common/thread_formatter.rb:49:in `with_thread_name'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/job_runner.rb:34:in `run'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/base_job.rb:10:in `perform'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:36:in `block in perform'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/bosh-director-0.0.0/lib/bosh/director/jobs/db_job.rb:83:in `block (3 levels) in run'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/eventmachine-1.2.5/lib/eventmachine.rb:1076:in `block in spawn_threadpool'
/var/vcap/packages/director/gem_home/ruby/2.4.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Closing and reopening to create tracker story.


[UI] Migrate Meshery Result Dialog Component to Functional Component with MUI v5 Styling and Sistent Component use

Current Behavior
This MesheryResultDialog Component is a class-based component and it uses MUI v4's styling. This needs to be updated for improved performance and consistency.
Desired Behavior

[ ] migrate class-based component to a functional component
[ ] use Sistent components
[ ]  migrate MUI v4 styling to MUI v5 supported styling

Acceptance Tests
The component must be updated without introducing any breaking changes in functionality or theming across all instances where it is used.

Contributor Guides and Handbook

🎨 Wireframes and designs for Meshery UI in Figma (open invite)
🖥 Contributing to Meshery UI
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

Hello @SAHU-01 assign this to me I'll complete it asap

Hello @SAHU-01 can you please assign this to me ?

@SAHU-01 I have worked with issue like this, I have created a PR for that,
Please assign this to me.

Hey, @leecalcote @SAHU-01 . Please assign this issue or any other issue related to class based components to functional components or migration of mui as I have set up the codebase locally. Thanks

@Karan-Palan Thank you for contributing, please go ahead!


[Sample] Policies for nginx-ingress vulnerabilities CVE-2021-25745 and CVE-2021-25746

Problem Statement
CVE-2021-25745: Ingress-nginx path can be pointed to service account token file
Issue Details
A security issue was discovered in ingress-nginx where a user that can create or update ingress objects can use the spec.rules[].http.paths[].path field of an Ingress object (in the networking.k8s.io or extensions API group) to obtain the credentials of the ingress-nginx controller. In the default configuration, that credential has access to all secrets in the cluster.
This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2021-25745.
Affected Components and Configurations
This bug affects ingress-nginx. If you do not have ingress-nginx installed on your cluster, you are not affected. You can check this by running kubectl get po -n ingress-nginx.
Multitenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Affected Versions
<v1.2.0
Fixed Versions
v1.2.0-beta.0
v1.2.0
Mitigation
If you are unable to roll out the fix, this vulnerability can be mitigated by implementing an admission policy that restricts the spec.rules[].http.paths[].path field on the networking.k8s.io/Ingress resource to known safe characters (see the newly added rules, or the suggested value for annotation-value-word-blocklist).
Detection
If you find evidence that this vulnerability has been exploited, please contact<EMAIL_ADDRESS>Additional Details
See ingress-nginx Issue #8502 for more details.
Acknowledgements
This vulnerability was reported by Gafnit Amiga.
CVE-2021-25746: Ingress-nginx directive injection via annotations
Issue Details
A security issue was discovered in ingress-nginx where a user that can create or update ingress objects can use .metadata.annotations in an Ingress object (in the networking.k8s.io or extensions API group) to obtain the credentials of the ingress-nginx controller. In the default configuration, that credential has access to all secrets in the cluster.
This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2021-25746.
Affected Components and Configurations
This bug affects ingress-nginx. If you do not have ingress-nginx installed on your cluster, you are not affected. You can check this by running kubectl get po -n ingress-nginx.
Multitenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Affected Versions
<v1.2.0
Fixed Versions
v1.2.0-beta.0
v1.2.0
Mitigation
If you are unable to roll out the fix, this vulnerability can be mitigated by implementing an admission policy that restricts the metadata.annotations values to known safe (see the newly added rules, or the suggested value for annotation-value-word-blocklist).
Detection
If you find evidence that this vulnerability has been exploited, please contact<EMAIL_ADDRESS>Additional Details
See ingress-nginx Issue #8503 for more details.
Acknowledgements
This vulnerability was reported by Anthony Weems, and separately by jeffrey&oliver.
Solution Description
Sample policy that implements
https://github.com/kubernetes/ingress-nginx/blame/main/internal/ingress/inspector/rules.go
Sample policy that implements
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#annotation-value-word-blocklist
Example "Good" Resource
No response
Example "Bad" Resource
No response
Other Comments
No response
Slack discussion
No response
Troubleshooting

[X] I have read and followed the documentation AND the troubleshooting guide.
[X] I have verified the policy does not exist in the samples library.

See this below. Seems to work on all cases I can come up with. Needs polish, annotations, appropriate names, and a good description.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ingress-contains
spec:
  background: false
  validationFailureAction: enforce
  rules:
  - name: cve-2021-25745
    match:
      any:
      - resources:
          kinds:
          - Ingress
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: Equals
        value: CREATE
    validate:
      message: "Ingress paths cannot use forbidden strings."
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'load_module') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'lua_package') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'_by_lua_') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'location') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'root') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'proxy_pass') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'serviceaccount') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'{') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'}') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'alias') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'/etc/') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.spec.rules[].http.paths[].path.contains(@,'/var/run/secrets') }}"
            operator: AnyIn
            value:
              - true
  - name: cve-2021-25746
    match:
      any:
      - resources:
          kinds:
          - Ingress
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: Equals
        value: CREATE
    validate:
      message: "Ingress annotations cannot use forbidden strings."
      deny:
        conditions:
          any:
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'load_module') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'lua_package') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, '_by_lua') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'location') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'root') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'proxy_pass') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'serviceaccount') }}"
            operator: AnyIn
            value:
              - true
          ## If the check for {, }, ', and " are in place, this will fail validation since the API server
          ## will add the kubectl.kubernetes.io/last-applied-configuration containing these chars since
          ## it represents the entirety of the object as JSON.
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, 'alias') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, '/etc') }}"
            operator: AnyIn
            value:
              - true
          - key: "{{ request.object.metadata.annotations.values(@)[].contains(@, '/var/run/secrets') }}"
            operator: AnyIn
            value:
              - true

Some Ingress scaffolding you can use:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-bad-ingress
  annotations:
    foo: foo
    bar: /foo/bar
spec:
  rules:
  - host: https-good.good.com
    http:
      paths:
      - path: /cart
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80
  # - host: https-good.bad.com
  #   http:
  #     paths:
  #     - path: '/foo/bar/a_by_lua_boo'
  #       pathType: Prefix
  #       backend:
  #         service:
  #           name: service1
  #           port:
  #             number: 80


Thanks

This is not a real issue, but a way of me saying thank you for this lib. ❤
This lib helped so much in my daily job, more than you can't imagine.
I really appreciate the work @NARKOZ and everyone did here.
Let's show our gratitude to this awesome lib 🙇

Thanks


Translate WCMP vendor’s shop page

Hi,
We are running WooCommerce and use Hyyan WooCommerce Polylang Integration to translate products, categories and so on which works perfectly.
We can’t translate WC Marketplace vendor’s store page. I have been in contact with WCMP. This is there answer:
“Our vendor shop page is like any other taxonomy page i.e. category/tag etc. So, can you please check with polylang support, and ask them why this page is not translating properly.”
Is this something that you can help with?

this is a free plugin bridging woocommerce and polylang.
it does not claim to support WC Marketplace or any other plugin (it can't, its free).
If you need polylang translation support for WC Marketplace then you could write your own customisations or ask WC Marketplace or Polylang to do so.


Unsupervised learning in SnipsNLUEngine

Is there a way to train snips-nlu in a unsupervised way?

Hi @manishprasadthapliyal, what do you mean by "unsupervised" way ?
What would be the inputs of the NLU and what output would you expect ?

Closing this one since there's no activity on it


DACT-2100: Bug-not-returning-to-CYA

JIRA: https://companieshouse.atlassian.net/jira/software/c/projects/DACT/boards/492?selectedIssue=DACT-2100
Fixes issue of not returning to CYA through change links

Might be worth adding a unit test?


Feat: UNT-T27821 Create Coloured Neumorphic views demo screen

Add Coloured NeumorphicView demo screen
Make demo screen scrollable by wrapping it in ScrollView.
Decrease the spacing between "Go to SwiftUI Demo" button and "heart" button in Main.storyboard.
Add inner shadow width parameter in innerShadow extension for View.
Remove default values from InnerShadowModifier code.
Added customization options for inner shadow offset, blur radius, width, and outer shadow settings for improved neumorphic styling.
Added offDarkShadow and offLightShadow Colours in NMToggleStyle.swift.

LGTM


Search: Design search bar with filters

I don't think we should have filters on the search bar (other than "Category") because:

If there are more than 5 search results, we should link to the Full Search Results page to facilitate easier filtering and parsing of the results
If there are less than 5 results, filtering becomes redundant


Should there be some testing of the debug version during RC QA testing?

In https://github.com/phetsims/coulombs-law/issues/70 @jessegreenberg and I found an assertion that was only triggered in the build version, so it only happened when running a sim built with assertions left in (debug version or --mangle=false option). Do we think that it would be good for QA to test on the debug version for RC.
I don't think it warrants another item to the list. Perhaps QA can choose "randomly" one block of their matrix to test with the debug version each full RC. If they encounter performance issues, they can compare it will the normal built sim.
@KatieWoe @ariel-phet how does this sound? Marking for dev meeting to see it if it worth it.

I'd like to make sure I have a better understanding of what the debug version is and how its different, but in general I don't necessarily have a problem with this.

The difference between versions are that in the debug version (a) assertions are left in, and (b) the file isn't mangled/minified so that it is easier to read (hence it may be slower).

Please see this thread: "Should QA tests be able to test with assertions enabled?" https://github.com/phetsims/tasks/issues/722

The last comment there from @phet-steele:

QA knows the debug versions exist, and they know when to use them 👍

If that is true then all is good here. The main reason I brought up this issue was because phetsims/coulombs-law#70 was the first time that I saw an issue only triggered with assertions in built mode.
@KatieWoe do you feel like you have a good grasp on when/how to use the debug version? Anything else for this issue?

@zepumph I don't really.  It hadn't been brought up until now so I'm just starting to look into it.  Hopefully I'll be able to ask someone to walk me through it this week and we'll be good after that though.

The more important issue here is that the built version is somehow different enough to trigger an assertion failure.  Do we understand why that is the case?

The more important issue here is that the built version is somehow different enough to trigger an assertion failure.

I don't understand what this means.  What is the built version different from?  We are talking about comparing the built version to the built debug version.  The hypothesis is that QA team should be running some tests with the debug version and ?ea to catch issues that would perhaps silently fail in the main built version.

In https://github.com/phetsims/coulombs-law/issues/70 we found that the buildt version is different from the requireJS version, and we don't know why yet.
If the QA team tests the debug version, https://github.com/phetsims/coulombs-law/issues/70 and other issues like it will be caught in the future.

Rather than adding additional testing to QA's plate, I think we'd be better off understanding and fixing the root cause of the difference between requirejs and built versions.  Before adding to QA tests, consider adding to CT tests.

Discussion from 10/11/18
@KatieWoe wondered if the debug version requires the ?ea query param. Devs said that assertions are enabled by default.
@KatieWoe says that @phet-steele would test with the debug version in the background with ?fuzzMouse testing.
We discussed that if a QA issue is found while testing a built version, the debug version could be used to find more information upstream. @KatieWoe said that she would put the debug version on a machine and run in the background that won't take any additional time. She will make a note of this in the QA handbook.


Cannot convert value "1648551970,87108" to type "System.Int32". Error: "Input string was not in a correct format."

Hi ! I had this issue:
Cannot convert value "1648551970,87108" to type "System.Int32". Error: "Input string was not in a correct format."
when calling Add-JiraDeploymentInformation in a local powershell session.
turned out to be this line in the code:
updateSequenceNumber     = (Get-Date -UFormat "%s");
I changed it to:
updateSequenceNumber     = [math]::Round((Get-Date -UFormat "%s"));
not sure if this is a local setting that makes it that strange none int number.
Also in term of the readme that says " you can call this locally" - there is also
deploymentSequenceNumber = [Int]($env:BUILD_BUILDID);
that actually is a pipeline parameter and should look different :)
Using Win10, Powershell 5.1
Apart from that! So happy you made this! So sad that MS didn't manage to integrate with Jira AND Bitbucket correctly with their extensions.

ah, our build agents run much newer version of powershell where this isn't an issue


added condition to bucketclass schema field based on isNoobaa

fixes https://bugzilla.redhat.com/show_bug.cgi?id=2175685

/test odf-console-e2e-aws

/test all

/retitle Bug 2175685: added condition to bucketclass schema field based on isNoobaa

/approve
will lgtm once we will get all the acks on the BZ...

/lgtm

/cherry-pick release-4.13

/cherry-pick release-4.13-compatibility

/retitle added condition to bucketclass schema field based on isNoobaa
now we have 4.13 branches, we need to add BZ number to the title for backport PRs now... not for master...
once backport PRs are up (after this master PR is merged), add the BZ number to the title of both PRs and make sure both PRs are added to "Links" section of the BZ... after that we can move BZ to "POST" and need to ask for "qa_ack" on the BZ... once done we can merge the backport PRs and move BZ to MODIFIED...

/retest-required


Caleb hill rest api docs

This PR adds a redoc docker container, moves the rest_api docs from the core directory to the 1.2 directory, and fixes the markdown issues found in those docs.

Both a rest_api.md and rest_api/ directory co-exist after this PR and the empty rest_api.md is linked to in some places.


Switch from bundle to bundler in Gemfile

Whilst working on Libraries.io I noticed that this project depends on the bundle gem rather than bundler, this pull request fixes the typo.

Thanks!

The upstream repo (https://github.com/MoOx/compass-recipes) seems to have the same issue and perhaps some newer commits then this fork...

Opened one on upstream too https://github.com/MoOx/compass-recipes/pull/106


random order of dataset files

Detailed Description
Could we mix up the order of the dataset. We could make this random
This means if we are running two satellite consumers, then they will pull different files.
There are other ways to do it, like a queue, but this mixing up of datasets I think is a quick win
Context
-- it takes quite long (15 mins) sometimes for satellite consumer to run
Possible Implementation
add shuffle here ish - https://github.com/openclimatefix/Satip/blob/main/satip/app.py#L174

what do you reckon to this @jacobbieker

Yeah, this sounds good to me. If the satellite data also isn't coming in every 5 minutes, maybe we delay how often we run it? Could also then reduce the amount of duplicate work.

I'll open a PR for the randomness, seems sensible to do anyway.


Preserve annotation for nested types

Bug
Please complete:

OS: macOS
Python version import sys; print(sys.version): 3.7.3
Pydantic version Master: https://github.com/samuelcolvin/pydantic/commit/5015a7e48bc869adf99b78eb38075951549e9ea7

Please read the docs and search through issues to
confirm your bug hasn't already been reported before.
The following snippet fails with
TypeError: The type of B.seq differs from the new default value; if you wish to change the type of this field, please use a type annotation

import pydantic

from typing import List

from pydantic import BaseModel

class A(BaseModel):
    seq: List[int] = [0, 1, 2]

class B(A):
    seq = [4, 5]

@dmontagu I'd definitely expect it to pass. How about running A.__fields__['seq'].validate() on the new default value set at B instead of comparing the inferred and the previously annotated types?
This way it could ensure that the new default value is compatible with the previous type hints, without the need of advanced inferring.

@kszucs I think that is a good idea for an approach, but it might have the following problem: what if you convert an attribute originally annotated as a string to have a default value that is a float? I think that should probably be an error, but this change would cause it to be allowed (and it would be parsed to a string implicitly).
The current behaviour allows the following:

class A(BaseModel):
    member: str = 3.5

assert type(A().member) is float
# but
assert type(A(member=3.5).member) is str

So I think the consistent behaviour would be to treat the default value as the input for the validation, thus allowing the coercion from float to string.  I suppose this would be the easiest too.

Another alternative would be to modify the behavior to check whether the elements of the list/dict/etc. actually match the annotation. It might involve special casing for List/Dict/etc., but even still might be preferable given how commonly used those types are.
Couldn't we reuse the validation system for this purpose?
A third alternative could be modifying the error message to just say that a type annotation must be added for the field (rather than the potentially confusing error that the inferred type was wrong). Given this would happen as soon as the class is created, I think this might be acceptable even if the exact logic is a little opaque.
Then modifying the default value without also repeating the annotation would not been possible for nested types.
I'd prefer either the first or the second one. Please correct me if I'm wrong, but both seem to be implementable by the validation system, the only difference is that the first allows the coercion whereas the second does not.
@kszucs @samuelcolvin Thoughts?

Another alternative would be to modify the behavior to check whether the elements of the list/dict/etc. actually match the annotation.

This isn't currently possible with the validation system since it's designed to parse, not validate, so it "tries to coerce and errors if it can't" rather than "checking the types are correct".
The problem is that currently (unless you use validate_all = True) any default value is allowed - parsing/validation is not implemented on the default values.
Thus whatever we do here, it'll be slightly confusing compared to what we have at present.
I think the solution might be to just look at the outer type, currently lists without type annotations will cause the field to just require a list, eg:
class Foo(BaseModel):
    x = [1, 2, 3]

print(Foo.__fields__)
#> {'x': <Field(x type=list default=[1, 2, 3])>}

Foo(x=['a', 'b'])  # ok

Foo(x=1)  # error: value is not a valid list

In other words we don't do any clever checks to work out what type of list [1, 2, 3] - we just decide it's List[Any] since you wouldn't be bothered to add an annotation.
I would say we do roughly the same thing with inheritance: check the new default is a list/dict/set but not investigate it's contents.

Thus the following would be fine:
from typing import List
from pydantic import BaseModel

class A(BaseModel):
    seq: List[int] = [0, 1, 2]

class B(A):
    seq = ['x', 'y']

Class B_Alternative(BaseModel):
    seq: List[int] = ['x', 'y']

B.seq would be of type List[int] with the (rather confusing, but acceptable default of) ['x', 'y'], basically the same as B_Alternative.

Yes, I think the first error should remain, but not the second.
This should not be that difficult:

It involves only inspecting the type of the default, not any nested subtypes.
This is the same logic we already use to infer the type of a field without an annotation.

I'm sure you're not being blockheaded.

Is the idea that if the annotated type is a container, we should only check for equality of the outermost type (e.g., the origin)?

Yes, exactly. But I haven't looked into this, I was just thinking out loud about a way to find a sensible compromise on this.
How about replacing inferred.type_ != fields[var_name].type_ with something like:
def annotation_matches_type(annotation_type: Type, value_type: Type):
    return annotation_type == value_type or getattr(annotation_type, '__origin__', None) == value_type

?

Okay, but I still think a function that fixes these cases and a few others wouldn't be that long or difficult to implement.

For what it's worth, the @kszucs approach would probably handle all these cases neatly

Of calling validate()?
But wouldn't that confuse people much more? Eg. validators being called at load time and field defaults being validated in a few very specific circumstances but not otherwise.
I want to avoid confusing edge cases for those new to pydantic.

I've thought about this a lot and I don't see an elegant solution that works in all cases:
I think there are two options:

We require type annotations for all fields that override a field on a parent class
we implement a "best attempt" annotation_matches_type and modify the error message to more strongly encourage a type annotation; something like it wasn't possible to confirm that the type of {value!r} matched {annotation_type}, please use an explicit type annotation on "{field_name}"

I'd prefer option 2.

I'm happy with the current behaviour on main, not particularly inclined to change it, please open a new issue if you have a suggested change in v2.


TextInput with Drop-Down List

In the Tabris Android App there is a Drop Down List showing the last IP's to connect the smartphone with the server running on the pc.
I couldn't find the source code.  How can I use this functionality in my App?
Thanks for helping me on this strange issue:))

Would a SearchAction work?

The search action is exactly what I want but I can not use it as an normal widget of a page. It is only usable in the head of a navigation view, isn't it?

That is correct.  You'll have to stay tuned for #1247 or implement your own functionality by listening to the focus and blur events of the TextInput.

OK thank you I close this issue because of #1247


Check Headlamp on AKS

Description
As initially reported in #136  , we should check how well Headlamp plays with AKS.

Headlamp is tested on AKS and it works the way we intend it to.
We can close this issue.
@joaquimrocha

Headlamp is tested on AKS and it works the way we intend it to.
We can close this issue.
@joaquimrocha


HA Restart stalls if MA playing

What version of Music Assistant has the issue?
2.0.0b85
The problem
If you are playing music with MA and attempt to restart HA, the restart process stalls, MA doesn't shutdown, and everything just hangs there until you stop music playback some other way (Android GHome Notification, in my case).
How to reproduce

Stream music from MA to a speaker (Google Home Max, in my case)
Request HA to restart itself
See HA begin shutdown and stall while music continues to play
Leave like this as long as you'd like, then stop playback on the speaker some other way (HA interface is offline at this time)
As soon as you stop playback, HA shutdown completes and restarts immediately

Relevant log output
No response
Additional information
Factors involved:
Google Home Max Speaker
YT Music Streaming
Crossfade enabled
Continued playing through 2 full songs before I manually stopped playback and HA restarted
What version of Home Assistant Core are your running
2024.1.5
What type of installation are you running?
Home Assistant Container
On what type of hardware are you running?
Linux

Interesting as playback is supposed to just continue....

Please provide more details of your installation.

Interesting as playback is supposed to just continue....

What do you mean? Isn't MA supposed to restart along with HA, since it is an Add-On?

Please provide more details of your installation.

Can you be more specific?

I don't know how you have installed your system. If you were running HAOS then HA and MA are installed in separate docker containers (as are all other addons) and therefore restarting HA (in its container) shouldn't affect MA. The connection between the two containers is the integration.
Hence why we would be interested in exactly how you have HA installed and how you have MA installed.

I don't know how you have installed your system. If you were running HAOS then HA and MA are installed in separate docker containers (as are all other addons) and therefore restarting HA (in its container) shouldn't affect MA. The connection between the two containers is the integration.
Hence why we would be interested in exactly how you have HA installed and how you have MA installed.

Thank you for clarifying. It's HAOS running in OVA, MA was installed via HACS, which would imply it is containerized, as you mention.
I am still testing. MA is playing YT Music on a speaker and I asked HA to restart. webGUI disconnects and my shutdown automation fired (flashes a lamp). This time, I let it go and waited...well over 5 minutes later, HA finally completed its restart. This is nearly 5x longer than restarting HA without MA running.
I doubt this is a resource issue. MA container itself is only using 3-4% CPU and 15-18% RAM at most...it normally hovers around 0.5% CPU and 6% RAM outside of a sync interval. Resource usage on the VM doesn't spike above about 60% during an HA restart in either case. I have the VM rather oversized to maintain "snappy" response times for more complicated automations. So, I'm not sure what holds it up when MA is playing music.
I'll see if I can find anything noteworthy in HA's logs.

Look in the supervisor logs. There may be something going on with the integration

I can't find anything even remotely related in Supervisor logs and replicating it hasn't panned out 100% (more like 70%), so I'm going to chalk it up as a fluke unless/until I have something more concrete.


Test gap: verify behavior of SDS when CA gRPC is terminated (citadel restart, etc)

Both IstioCA and GoogleCA initialize the gRPC in main() - there is a TODO on both about
reconnecting, but it doesn't seem to be implemented. For example citadel/client.go:69
sendRetriableRequest is retrying the request - but it is not reconnecting. It doesn't appear to have any other recovery.
Killing citadel can be used to reproduce it.

Note this does not affect SDS for ingress.

but it is not reconnecting

The gRPC client is normally automatically reconnecting with the default options, which is the case here.
Have you observed that it doesn't reconnect?

Sorry, I didn't get to grab the logs.
Confirmed that grpc is reconnecting with backoff - and there is the backoff on SDS code itself.  It may be just a timing issue.

Updated the title - I think the remaining issue is to understand the timing and config options around retries/backoff.  And remove the comment around Dial(), I got confused by it and stopped looking for other causes.

https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md - seems 2 minute is the max backoff, with 20 sec min. On top of this there is the retry/backoff in the sds code.

Things change a lot since the issue is filed. @myidpt @JimmyCYJ could you confirm whether there still some AI here?


Use Java 11 everywhere

is this raising min sdk?

Making it consistent was my intention, but now I'm seeing that I inadvertently raised it when I added the convention plugin (I pulled the convention plugin from the paging repo). Will make consistent at 24,

Thanks for reverting. Maybe add release notes for last version just in case


Support dragging across transformed iframes

Follow on to #1510 for transformed iframes in 37cd54072169e68ae5c505c20e613b14dad8aaa9.
If reviewing before #1510 is merged, please just review the final commit for diff.
Tasks

[ ] Rebase once #1510 merged
[ ] Tidy stories
[ ] Add changeset

Closing in favour of #1510


Particles2D weird behaviour with scale randomisation and curve

Godot version: fe841fe
OS/device including version: Windows 7 - HD 6950
Issue description:
Particles will not scale down to the correct size if I use a curve along with a random value.
It seems as though the curve is ignoring the random part.
Steps to reproduce:

Create a particle object with a particle material.
Set scale random to any non-zero value (I used 1).
Create a scale curve, going from 1 to 0.
See that some particle sizes do not fully reach 0.

This can be checked in:
https://github.com/godotengine/godot/blob/master/scene/resources/particles_material.cpp
https://github.com/godotengine/godot/blob/master/scene/2d/cpu_particles_2d.cpp
https://github.com/godotengine/godot/blob/master/scene/3d/cpu_particles.cpp
will probably need fixing in those places

I'm not sure what the issue is. I'm seeing the particles all approach zero at the end.

@pwab is the behavior in the gif different from what you saw?

Hmm... I still don't see a "bug" per se. Seems to me that if a particle starts from scale 20 to 0 over lifetime 8 it'll give a fading effect to the human eye. If a particle starts from scale 1 to 0 over lifetime 8, it'll fluctuate at the small scale for awhile and just dissapear at the end. I'm guessing scale=0 doesn't play well in general and it's gonna fluctuate the pixel too with the small floats. Maybe there needs to be an alpha curve too that logarithmic ally fades when approaching zero.

I can see your point but the fact that particles at scale 1 will

fluctuate at the small scale for awhile and just dissapear at the end

in contrast to particles at scale 20 could lead to confusion for the users. The question for me is: Why won't the scale 20 particles disappear at the end all at once (and instead create this fading away effect)?
Or maybe I ask in another way: How to achieve the fading effect with a randomized scale curve? An alpha or color curve would be interesting here but then the small particles will fade away all at once at the end which is different from the effect seen in the last gif (particles reach their size 0 at different times).


SlickGrid Sorting on Grouping

hey all,
I have used the Grouping capabilities integrated in the framework with good results so far:
Except when i want to sort DESC the group field:
       onClick: () => this.view.setGrouping(
          [{
            getter: 'SessionStartDate',
            formatter: x => x.value + ' (' + x.count + ' items)'
          }

Items are always sort ASCending order.
I did try using in the xyzGrid.ts :
    protected getDefaultSortBy() {
      return [myRow.Fields.SessionStartDate + " desc"];
    }

(syntax is correct but field is never displayed desc sort)
Anyone could manage this ?

Could manage to make it work using a comparer :
        onClick: () => this.view.setGrouping(
          [{
            getter: 'SessionStartDate',
            formatter: x => x.value + ' (' + x.count + ' items),
            comparer: function comparerDesc(a, b) {
              var x = a, y = b;
              return (x == y ? 0 : (x > y ? 1 : -1));
            }
          }


SIMD: misc instructions

This completes the implementation of SIMD proposal for both
the interpreter and compiler(amd64).
This also fixes #210 by adding the complete documentation
over all the wazeroir operations.
arm64 compiler will follow in subsequent PRs.
part of #484

ok passed simd_**_arith.wast, simd_**_arith2.wast, simd_f32.wast, simd_f64.wast spectests with both engines... will implement more!

ok simd_i16x8_q15mulr_sat_s, simd_int_to_int_extend, simd_i64x2_extmul_i32x4, simd_i32x4_extmul_i16x8 and simd_i16x8_extmul_i8x16 passed...

ok passed all spec test with the interpreter, meaning that it is fully compatible with WebAssembly 2.0 draft!

ok passed all tests for both engines... backfilling comments and unit tests...

ok finished the tests for spectest.go... marked this for ready!


[BUG-python/deployment] DELETE /api/datasets/{dataset_name} returns 200 even if dataset doesn't exist

Describe the bug
When calling rg.delete to try to delete a dataset that doesn't exist, it doesn't raise an error because the DELETE /api/datasets/{dataset_name} endpoint returns 200 status code even if the dataset doesn't exist.
Stacktrace and Code to create the bug
import argilla as rg

rg.delete(name="invented", workspace="my-workspace") # doesn't raise error

Expected behavior
The endpoint returns a 404 and rg.delete raises an error.

Environment:

Argilla Version [e.g. 1.0.0]: 1.19.0
ElasticSearch Version [e.g. 7.10.2]: not relevant
Docker Image (optional) [e.g. argilla:v1.0.0]: not relevant

It's not a bug, we do this on purpose.


CLI doesn't provide feedback when referencing a non existing server id

in earlier versions, CLI would return an indicative message saying that you are pointing to a server id that does not exist in the config:
export JFROG_CLI_LOG_LEVEL=DEBUG
jfrog rt dl path/to/folder/ repo/path/in/artifactory/ --server-id test
[Error] Server ID 'test' does not exist.
In the latest version 1.33.1, this message isn't thrown even in debug log level

Fixed by https://github.com/jfrog/jfrog-cli/pull/586.
Will be available in the next version.


[1.2]  bluetooth broadcast interval of 1.2 seconds on linux

Reproduction steps
I am currently building a matter environment on the Linux platform and using Raspberry Pi to connect using chip-tool. However, there is a problem where I am prone to connection failure during the Bluetooth connection phase when using chip tool. It takes about 5 times to successfully connect, but after successfully connecting, I found that it can be paired normally. I checked the Bluetooth properties broadcasted and found that the broadcast interval is 1283ms, which I think is abnormal, When I tried to adjust the broadcast interval, I couldn't find the location to adjust it. My system uses a solution like GDbus+Bluez to set up Bluetooth broadcasting. May I ask if you have a deeper understanding of the reason for this connection failure, or can someone tell me how to adjust the Bluetooth broadcasting interval

Bug prevalence
alway
GitHub hash of the SDK that was being used
181b0cb14ff007ec912f2ba6627e05dfb066c008(HEAD, tag: v<IP_ADDRESS>)
Platform
other
Platform Version(s)
1.2
Type
Platform Issue
Anything else?
No response

@kagami555 please provide a reproduction script for this: how can I reproduce this issue using a linux-compiled chip-tool and example application?


[SPARK-22169][SQL] support byte length literal as identifier

What changes were proposed in this pull request?
By definition the table name in Spark can be something like 123x, 25a, etc., with exceptions for literals like 12L, 23BD, etc. However, Spark SQL has a special byte length literal, which stops users to use digits followed by b, k, m, g as identifiers.
byte length literal is not a standard sql literal and is only used in the tableSample parser rule. This PR move the parsing of byte length literal from lexer to parser, so that users can use it as identifiers.
How was this patch tested?
regression test

cc @hvanhovell @gatorsmile

Test build #82322 has started for PR 19392 at commit 9a11231.

Test build #82322 has finished for PR 19392 at commit 9a11231.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82322/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #82326 has started for PR 19392 at commit 9a11231.

retest this please

Test build #82328 has started for PR 19392 at commit 9a11231.

Test build #82326 has finished for PR 19392 at commit 9a11231.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82326/
Test FAILed.

Test build #82328 has finished for PR 19392 at commit 9a11231.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82328/
Test PASSed.

Test build #82452 has started for PR 19392 at commit d8548a8.

LGTM pending Jenkins

Test build #82452 has finished for PR 19392 at commit d8548a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82452/
Test PASSed.

Thanks! Merged to master.
Could you backport it to 2.2?

hmm, it's not a bug fix but a nice-to-have feature, do we want this in spark 2.2?

OK


repochecker: Fix Tumbleweed case

When using with Tumbleweed, the urls are different no updates are needed.

LGTM


Issue with while (true) loop inside notification.handler.js

There is a while loop in notification.handler.js that behaves a bit unpredictable in different environments. I can't reproduce the issue locally but in our production environment it occurs at every call to the function.
https://github.com/Adyen/adyen-commercetools/blob/97bee4eaee73f7e6cc7cf8c70b32eae9e5a3eaee/notification/src/handler/notification/notification.handler.js#L88
Issue description
The while loop does not actually wait for the await statement inside the try block, leading to potentially huge amount of logs. We have this running in azure functions and one of invocations of this function had around 30 thousand logs of Update payment with key..... Running the function locally does not produce the issue but when i deploy a fix to our staging environment it seems to fix the excess logging.
Solution proposal
Instead of a while (true) {} loop just wrap everything in a recursive function that you call in the catch block if maxRetry isn't reached yet. Something like this:
async function updatePaymentWithRepeater(
  payment,
  notification,
  ctpClient,
  logger,
) {
  const maxRetry = 20
  let currentPayment = payment
  let currentVersion = payment.version
  let retryCount = 0
  let retryMessage
  let updateActions
  const repeater = async () => {
    updateActions = await calculateUpdateActionsForPayment(
      currentPayment,
      notification,
      logger,
    )
    if (updateActions.length === 0) {
      return;
    }
    logger.debug(
      `Update payment with key ${
        currentPayment.key
      } with update actions [${JSON.stringify(updateActions)}]`,
    )
    try {
      /* eslint-disable-next-line no-await-in-loop */
      await ctpClient.update(
        ctpClient.builder.payments,
        currentPayment.id,
        currentVersion,
        updateActions,
      )
      logger.debug(
        `Payment with key ${currentPayment.key} was successfully updated`,
      )
      return;
    } catch (err) {
      const moduleConfig = config.getModuleConfig()
      let updateActionsToLog = updateActions
      if (moduleConfig.removeSensitiveData)
        updateActionsToLog =
          _obfuscateNotificationInfoFromActionFields(updateActions)

      if (err.statusCode !== 409) {
        const errMsg =
          `Unexpected error on payment update with ID: ${currentPayment.id}.` +
          `Failed actions: ${JSON.stringify(updateActionsToLog)}`
        throw new VError(err, errMsg)
      }

      retryCount += 1
      if (retryCount > maxRetry) {
        retryMessage =
          'Got a concurrent modification error' +
          ` when updating payment with id "${currentPayment.id}".` +
          ` Version tried "${currentVersion}",` +
          ` currentVersion: "${err.body.errors[0].currentVersion}".`
        throw new VError(
          err,
          `${retryMessage} Won't retry again` +
            ` because of a reached limit ${maxRetry}` +
            ` max retries. Failed actions: ${JSON.stringify(
              updateActionsToLog,
            )}`,
        )
      } else (
				repeater()
			)
      /* eslint-disable-next-line no-await-in-loop */
      const response = await ctpClient.fetchById(
        ctpClient.builder.payments,
        currentPayment.id,
      )
      currentPayment = response.body // eslint-disable-line prefer-destructuring
      currentVersion = currentPayment.version
    }
  }
  return repeater()
}

Hey @joakimmillen,
We have validated your request and it is correct. We have used the code you have provided, slightly refactored it, and tested it. It is working correctly. This will be included in the next release.
Best regards.


Spiketrain behavior on rescale changed

The behavior when rescaling a SpikeTrain changed in 0.7.0 proposed, and is now very counter-intuitive. The change surfaces in a number of tests in Elephant.
Behavior 0.5.2 (t_start and t_stop are rescaled accordingly):
In [1]: import quantities as pq
In [2]: import neo
In [3]: a=neo.SpikeTrain([1,2,3]*pq.ms,t_start=0*pq.s,t_stop=10*pq.h)
In [4]: b=a.rescale(pq.s)
In [5]: print b
[0.001 0.002 0.003] s
In [6]: print b.t_start
0.0 s
In [7]: print b.t_stop
36000.0 s

Behavior 0.7.0 (t_start and t_stop are rescaled to millisecond):
In [1]: import quantities as pq
In [2]: import neo
In [3]: a=neo.SpikeTrain([1,2,3]*pq.ms,t_start=0*pq.s,t_stop=10*pq.h)
In [4]: b=a.rescale(pq.s)
In [5]: print b
[0.001 0.002 0.003] s
In [6]: print b.t_start
0.0 ms
In [7]: print b.t_stop
36000000.0 ms

Related to https://github.com/NeuralEnsemble/elephant/pull/189

Further investigation yields that in 0.7 when rescaling the spiketrain, this calls the __new__ of a new Spiketrain instance, which in turn uses the dimensionality of the times property of the original object to set that of t_start and t_stop, i.e., if the spike train a above would have times in hourse, t_start in the new object would be in hours.
In 0.5.2, the rescale() method of Spiketrains has this code
if self.dimensionality == pq.quantity.validate_dimensionality(units):
    return self.copy()
spikes = self.view(pq.Quantity)
obj = SpikeTrain(times=spikes, t_stop=self.t_stop, units=units,
                         sampling_rate=self.sampling_rate,
                         t_start=self.t_start, waveforms=self.waveforms,
                         left_sweep=self.left_sweep, name=self.name,
                         file_origin=self.file_origin,
                         description=self.description, **self.annotations)

that is replaced in 0.7.0 by
obj = super(SpikeTrain, self).rescale(units)

The 0.7.0 version is simpler and nicer, but there must be something in the 0.5.2 version we should put back in -- I just don't see what yet...

see #614

@apdavison Great, that is a simple solution to solve the issue, and also helps with Elephant's unit tests

close by #614


DCOS: Windows custom script extension to pass exit code on

Fixing DCOS Windows agent deployment. Windows custom script extension should propagate exit status.

/approve
/lgtm


Increase OSX file descriptor limit during restore

Workaround https://github.com/dotnet/cli/issues/809

I think we have a fix for this. It might not be needed. @moozzyk ?

Yes. NuGet folks committed the fix. I am not sure if cli moved to this version of NuGet yet though.

FWIW dotnet 1.0.0-beta-001491 appears to have the fix. no more osx restore crashes :)


confusion_matrix should use cast rather than convert to tensor for size

This does not work:
>>> cm = tf.contrib.metrics.confusion_matrix(
...     [1,2,3], [4,5,5], num_classes=tf.constant(3, dtype=tf.int32))
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/metrics/python/ops/confusion_matrix_ops.py", line 84, in confusion_matrix
    indices=indices, values=values, shape=shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 981, in __init__
    shape = convert_to_tensor(shape, name="shape", dtype=dtypes.int64)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 628, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 571, in _TensorTensorConversionFunction
    % (dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype int64 for Tensor with dtype int32: 'Tensor("confusion_matrix/pack:0", shape=(2,), dtype=int32)'

where as this does:
tf.contrib.metrics.confusion_matrix(
    [1,2,3], [4,5,5], num_classes=tf.constant(3, dtype=tf.int64))

This seems related: https://github.com/tensorflow/tensorflow/pull/2187
Machine:
 bazel version
INFO: Reading 'startup' options from /root/.bazelrc: --batch
Extracting Bazel installation...
Build label: 0.3.0
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jun 10 11:38:23 2016<PHONE_NUMBER>)
Build timestamp:<PHONE_NUMBER>
Build timestamp as int:<PHONE_NUMBER>

 git rev-parse HEAD
58b37cf745c6c30d44878ddf5987c4283ed12c3c

ls -l /usr/lib/x86_64-linux-gnu/libcud*
lrwxrwxrwx 1 root root       29 Aug 12 05:11 /usr/lib/x86_64-linux-gnu/libcudnn.so -> /etc/alternatives/libcudnn_so
lrwxrwxrwx 1 root root       17 Jun 10 13:07 /usr/lib/x86_64-linux-gnu/libcudnn.so.5 -> libcudnn.so.5.1.3
-rw-r--r-- 1 root root 60696704 Jun 10 13:07 /usr/lib/x86_64-linux-gnu/libcudnn.so.5.1.3
lrwxrwxrwx 1 root root       32 Aug 12 05:11 /usr/lib/x86_64-linux-gnu/libcudnn_static.a -> /etc/alternatives/libcudnn_stlib
-rw-r--r-- 1 root root 59715990 Jun 10 13:07 /usr/lib/x86_64-linux-gnu/libcudnn_static_v5.a

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.7.5 locally

Looks like a fix is on its way...

Closing, assuming that the fix was committed. Feel free to open again if the problem persists in more recent versions.


Clean up worker code

What changes are proposed in this pull request?
Clean up worker code
Why are the changes needed?
Code quality
Does this PR introduce any user facing changes?
na

@jja725 can you solve the conflicts?

alluxio-bot, merge this please


"Start recording" action checks if already recording

When user starts recording on an endpoint that is already recording, an internal server error message is displayed. Instead, the start recording action should check whether the endpoint is already recording, and when it is, it should display information message about recording in progress and allow the user to stop the recording with the stop icon.
Current error message:

Let's add the following descriptions for error cases:
409 - a recording is already in progress
anything else - the endpoint does not support AppMap recording

@jansorg please see the error message in Dustin's comment.


Preset changes LUT, LUT does not change preset

When selecting a preset item in the combobox, the LUT is changed accordingly (i.e. if one selects "Vascular II", LUT is automatically set to "Red Vessels".
When changing the LUT though, preset does not change.
This implies that, when reloading a scene, preset has to been reloaded before LUT.
Is it by design (same behaviour on all workspaces)?

I think a preset, is a specific LUT with a certain Window/Level value, not sure, but It is by design.


restart out of memory with 500GB data

Ledis use 100% memory (16GB) when restart with 500GB data files in data path.
After change the data path to another empty dir, ledis runs well.
Does ledis load data to memory when restart?

@CocaCola183
Ledis don't load data to memory.
Do you use RocksDB engine? Can you give me the LOG of RocksDB and ledis's log?

@siddontang
Engine is leveldb and there is no startup log. Do you need access log?

leveldb will have a LOG in the leveldb data path.


Eliminate ChildAdmissionController after dropping support for Kueue 0.6

The ChildAdmissionController was a hacky workaround for Kueue 0.6 not correctly recognizing external frameworks.
We contributed upstream enhancements (https://github.com/kubernetes-sigs/kueue/pull/2059) that are included in Kueue 0.7 that make this workaround obsolete.   Once we can drop support for Kueue 0.6 from the main branch, we should remove this controller and related configuration and documentation (including controller architecture description in webpages).
Need to wait for #161 to be merged to main before doing this.

#185 updates our controller architecture description for Kueue 0.7


Get pushing unknown route on stack when directly accessing urls on web

Get is pushing an unknown route in the middle of the stack. It occurs when directly accessing the app using a dynamic url on a new chrome tab.
To Reproduce
If you have the following route structure: /foo/:id/bar
And access the app with the following link: host/#/foo/123/bar, the resulting navigator stack will be:

HomeScreen (/)
UnknownRoute (/foo which doesn't exists)
FooScreen (/foo/:id)
BarScreen (/foo/:id/bar)

Expected behavior
Get should check when pushing screens to the stack if they have the /route/:id format to avoid pushing non existing routes.
Flutter Version:
1.18.0-11.1.pre
Get Version:
2.13.1

Please, update get version and use getPages rather namedRoutes.

this problem still persists


Add is_binary, is_text, and is_archive fields to the CodebaseResource model #75

Signed-off-by: tdruez<EMAIL_ADDRESS>

@JonoYang could you review the changes?


Build error with Swift 4.2.1

What did you do?
Building 4.3.0 with Carthage
swift --version
Apple Swift version 4.2.1 (swiftlang-1000.11.42 clang-1<IP_ADDRESS>)

What did you expect to happen?
Successful build
What actually happened instead?
Build failed with
Build settings from command line:
    CARTHAGE = YES
    CLANG_ENABLE_CODE_COVERAGE = NO
    CODE_SIGN_IDENTITY =
    CODE_SIGNING_REQUIRED = NO
    GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = NO
    ONLY_ACTIVE_ARCH = NO
    SDKROOT = iphoneos12.1
    SKIP_INSTALL = YES
    STRIP_INSTALLED_PRODUCT = NO

note: Using new build system
note: Planning build
note: Constructing build description
Build system information
error: SWIFT_VERSION '5.0' is unsupported, supported versions are: 3.0, 4.0, 4.2. (in target 'SwiftyJSON iOS')

Build system information
error: SWIFT_VERSION '5.0' is unsupported, supported versions are: 3.0, 4.0, 4.2. (in target 'SwiftyJSON iOS')

Environment

Carthage: 0.32

Project that demonstrates the issue
Any, Add Cartfile and try to build

4.3.0 is compatible with Swift 5.0, you need to switch to <= 4.2.0

I have the same issue

@ahmedomeruofk The workaround is to fix the version in Cartfile to 4.2.0

@ahmedomeruofk I have the same issue and I don't know how to fix the version. could you help?

I   have the same issue  but don't  know how to do it


Fix engine connection break when starting onboarding from a fresh install

This fixes #4215, which is a regression that has been introduced to onboarding, and is reproducible by starting onboarding from a fresh install with no user-level settings, or by resetting the onboarding status before opening a project in the desktop app (resetting the settings from the app does not reset the onboarding status).
This behavior was not detectable in the browser app, so I've added a test that verifies the engine connects in an electron playwright test.
I tweaked the way setupElectron's appSettings parameter works, making it not blow away the default settings but rather just override beyond them, because I didn't want to lose the "clean dir" behavior we have through it. That required me to update just one other test that assumed the onboarding dialog would be needs dismissed during its test because it overrode a different setting and blew away the defaults. I know we'll migrate to fixture-based tests but this seems like better behavior for now.

Also fixes #4231


mumble: use our celt library

cc @viric @geerds

Hello. Thank you for the PR.
Can you ellaborate why this is better than using the celt included in mumble? Did you find any problem?

@viric No, this is more of "avoid in-tree libraries when we have one in repositories" change -- I've noticed and done this occasionally while looking for other (fixed in master it seems) bug.

In general, in-tree libraries seem like a hack to work around the lack of a good dependency management system, which I think we can provide. They just grow closure sizes, increase build time, decrease resilience against major bugs in dependencies, and don't provide much value as far as I can tell. The main thing I can see in favor of them is that they produce a more predictable (and tested by others) artifact.

:+1: if it works

I have no preference in using a nix celt. Feel free to merge it, those who
have any preference. :)
On Fri, Jul 31, 2015 at 05:26:35AM -0700, lethalman wrote:

:+1: if it works

Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/pull/9065#issuecomment-126674815

--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key D4831A8A - https://emailselfdefense.fsf.org/

Tested this with old 1.2.3 server (which forces CELT), works nice. Merging.


Fully reverse engineered CMOS data checksum and parity scripts and parameters

I had a bit too much fun and got carried away and wrote this script, might be useful but not sure anymore.
This script attempts to brute force the start and end offsets for suspected checksums in the Hydro Thunder CMOS data. Currently only very basic SUM8, SUM16, SUM24 and SUM32 are implemented and tested.
Current findings: SUM8 and SUM16 return many matches for valid checksums across all the images however so far this is just random chance of them adding up right, none of the start and end locations that I looked at made sense. SUM24 and SUM32 do not return many results, most are again not in nonsensical locations. In addition, checksum byte differences in the data sample images appear not to reflect the behavior I would expect from a SUM checksum.
The current most likely checksum at 0x75BE66F to 0x75BE672 (data sample .img offset) I have not been able to map to anything or figure out what algorithm it uses, but it looks like it could be multiple checksums.
This code needs a fair bit more work to be an exhaustive search, but it’s leaning me more towards it not being a super easy SUM(x) across clearly defined regions of the CMOS data otherwise this code would have found it instantly (unless I screwed up my SUM(x) implementation, again) CRC16 is probably next on my list of guesses.
But I’m wondering if the name of the function that does this  cmos_CopyAllGlobalStructs2Cmos() is trying to tell us something about the format, have they actually designed a memory map to properly store all this (if I had 3GB I would have aligned and separated this a lot better to make my life easier especially when it came to adding that extra 11 bytes in 1.01b) or have they just dumped variables from the game engine into here. If so it would likely mean terminators, separators and other in-between data are not included in the checksums, we would have to extract all the values and strip the garbage making those brute force searches infeasible without a very good understanding of every variable and their bytes.
Performance to expect:
~500 k checksums per second (at uint16 size), takes about 15 to 18 seconds to exhaustively scan all contiguous variations of the ~5,000 byte cmos area.
TODO

[x] Fix bug before merge
[x] Find that extra DWORD/LSB error (LSB parity calculation)
[x] Modify the CMOS data (both areas) in an image with new high scores and use this newly calculated checksum
[x] Upload modified .imgs
[x] Test modified image on production cabinet
[x] Make functions return useful values and usable in other contexts
[x] Cleanup code

I spent all day today toggling every single setting and imaging the cmos data after. I'm going to work out what exactly everything maps too later today. But I can tell you now that they are clearly dumping raw data and that it is sloppy. In the menu of the game "Master Volume" and "Rumble Volume" are both displayed as adjustable percentages. In the data "Master Volume" is stored as a single byte u8. "Rumble Volume" is stored as a 4 byte float, for literally no reason at all. The wheel and throttle calibration points are a min, mid, max set for each that are shown as 0-255 in the menu, but each value is also stored as a 4 byte float. So what should be 6 bytes is balooned to 24. The "Wait for Operator" option added to the network menu in 1.01b's setting is also very close to the end of the settings area rather than near the other network settings. So I think they are linearly writing variables from RAM after they were added to the code or something similar.
Also, you should look at the discord server. Someone got the EXE open in Ghidra and has some strong clues on how the checksum works. I haven't processed what they found yet but it looks like the checksum method may be identifiable.

I think my scripts found the checksum solution, but I have been known to screw up checksums several times in the past so don't get your hopes up too much.
The evidence of success is in HTchecksumAnalysis.py the output is as below. The important part is Checksum abs diff between midway being all zeros excluding the ones that aren't NO DIFF, there's an extra DWORD somewhere, I think it intentionally breaks the checksum to force using one of the two CMOS areas.
...

  39 [./DataSamples/CF-Data-Set-AllTracks.img]
Checksum for first cmos copy     : 72 53 07 14 | 71 53 07 14<PHONE_NUMBER>0001110101001101110010
Checksum for second cmos copy    : 72 53 07 14 | 71 53 07 14<PHONE_NUMBER>0001110101001101110010
Checksum abs diff between copies : 00 00 00 00 | 00 00 00 00<PHONE_NUMBER>0000000000000000000000 NO DIFF, CMOS AREAS IDENTICAL
Checksum abs diff between midway :               01 00 00 00

  40 [./DataSamples/CF-Data-Set-Force.img]
Checksum for first cmos copy     : 9e 53 07 14 | 9d 53 07 14<PHONE_NUMBER>0001110101001110011110
Checksum for second cmos copy    : 9e 53 07 14 | 9d 53 07 14<PHONE_NUMBER>0001110101001110011110
Checksum abs diff between copies : 00 00 00 00 | 00 00 00 00<PHONE_NUMBER>0000000000000000000000 NO DIFF, CMOS AREAS IDENTICAL
Checksum abs diff between midway :               01 00 00 00

  41 [./DataSamples/CF-Data-Set-P-On.img]
Checksum for first cmos copy     : 74 53 07 14 | 73 53 07 14<PHONE_NUMBER>0001110101001101110100
Checksum for second cmos copy    : 74 53 07 14 | 73 53 07 14<PHONE_NUMBER>0001110101001101110100
Checksum abs diff between copies : 00 00 00 00 | 00 00 00 00<PHONE_NUMBER>0000000000000000000000 NO DIFF, CMOS AREAS IDENTICAL
Checksum abs diff between midway :               01 00 00 00

  42 [./DataSamples/CF-Data-Set-Free1st.img]
Checksum for first cmos copy     : 6e 53 07 14 | 6d 53 07 14<PHONE_NUMBER>0001110101001101101110
Checksum for second cmos copy    : 6e 53 07 14 | 6d 53 07 14<PHONE_NUMBER>0001110101001101101110
Checksum abs diff between copies : 00 00 00 00 | 00 00 00 00<PHONE_NUMBER>0000000000000000000000 NO DIFF, CMOS AREAS IDENTICAL
Checksum abs diff between midway :               01 00 00 00

  43 [./DataSamples/CF-Data-Race-4.img]
Checksum for first cmos copy     : 00 9d a9 22 | 00 9d a9 22<PHONE_NUMBER>1010011001110100000000
Checksum for second cmos copy    : fe 9c a9 22 | fe 9c a9 22<PHONE_NUMBER>1010011001110011111110
Checksum abs diff between copies : 02 00 00 00 | 02 00 00 00<PHONE_NUMBER>0000000000000000000010 LOWER CHECKSUM   <--
Checksum abs diff between midway :               00 00 00 00 Valid Checksum!

  44 [./DataSamples/CF-Data-Race-3.img]
Checksum for first cmos copy     : c6 05 89 31 | c6 05 89 31<PHONE_NUMBER>0010010000010111000110
Checksum for second cmos copy    : c4 05 89 31 | c4 05 89 31<PHONE_NUMBER>0010010000010111000100
Checksum abs diff between copies : 02 00 00 00 | 02 00 00 00<PHONE_NUMBER>0000000000000000000010 LOWER CHECKSUM   <--
Checksum abs diff between midway :               00 00 00 00 Valid Checksum!

  45 [./DataSamples/CF-Data-Race-slow.img]
Checksum for first cmos copy     : 93 8d c8 aa | 92 8d c8 aa<PHONE_NUMBER>0010001000110110010011
Checksum for second cmos copy    : 91 8d c8 aa | 90 8d c8 aa<PHONE_NUMBER>0010001000110110010001
Checksum abs diff between copies : 02 00 00 00 | 02 00 00 00<PHONE_NUMBER>0000000000000000000010 LOWER CHECKSUM   <--
Checksum abs diff between midway :               01 00 00 00
                                 |  Midway's   | My checksum
                                 |  Checksum   | SUM32 little<PHONE_NUMBER>

Checksum parameters
checksum_algorithm = 'SUM32'
checksum_endian = 'little'
checksum_seed = 0xFEDCBA94         # Result of of -0x0123456b
checksum_start_offset = 0x075be677   # Result of 0x75BE660 + 0x17
checksum_stop_offset =  0x075bfb76   # This is just a guess, it's probably shorter or longer but I think it's only white space after
checksum_span = 5376               # bytes, just the result of the above stop offset

TODO

[ ] Find that extra DWORD
[ ] Modify the CMOS data (both areas) and use newly calculated checksum
[ ] Test modified image on production cabinet

Those have been modified to have all 'WIZ' for high scores.

hydro-thunder-wiz-highscores-2.zip

https://youtube.com/shorts/b6midkL9Ya4
This has now been confirmed working!
A +1 to the LSB of the checksum validated by HTchecksumManualCheck.py seems to work reliably. Both sets of data must be replaced by the new data based on testing as well.

I made a few changes to this thing, mainly:
Better HTchecksumManualCheck.py
You can now also call it with a path argument python3 HTchecksumManualCheck.py './DataSamples/Pay.img'
Plus it just generally looks better an does both CMOS blocks
python3 HTchecksumManualCheck.py './DataSamples/Pay.img'

Checking supplied offsets point to start of CMOS areas in image:
  Image # 0 PASS : Found [01 00 00 00 98 ba dc fe] @ 0x75be663 [./DataSamples/Pay.img]
  Image # 0 PASS : Found [01 00 00 00 98 ba dc fe] @ 0x75ee663 [./DataSamples/Pay.img]

                                  |     Old      |    Newly     |
                                  |    Stored    |  Calculated  |
                                  |   Checksum   |   Checksum   |  Checksum Area Span
----------------------------------|--------------|--------------|-----------------------
CMOS area #0 Checksum @ 0x75be66f | e4 23 14 26  | e3 23 14 26  | 0x75be677 → 0x75bfb76
CMOS area #1 Checksum @ 0x75ee66f | e4 23 14 26  | e3 23 14 26  | 0x75ee677 → 0x75efb76

Reworked HTchecksumUtils.py to be useful in other scripts
It’s far from perfect but you can do stuff programmatically like bellow with it, calculating a new checksum and writing it the image all from your own script without having to deal with all the other rubbish behind the scenes. Not sure what your end goal is but I imagine this might be useful in achieving it.
This example for CMOS block [0]
>>> from HTchecksumUtils import *
>>> verifyImageHeaders(['./DataSamples/SN33-Real.img'])
  Image # 0 PASS : Found [01 00 00 00 98 ba dc fe] @ 0x75be663 [./DataSamples/SN33-Real.img]
>>> readChecksum('./DataSamples/SN33-Real.img', cmos.base_offsets[0] + cmos.checksum.rel_offset).hex(' ')
'ea 75 09 e9'
>>> calculateChecksum('./DataSamples/SN33-Real.img', cmos.base_offsets[0] + cmos.area.rel_offset)[0].hex(' ')
'ea 75 09 e9'

...
 < modified .img >
...

>>> calculateChecksum('./DataSamples/SN33-Real.img', cmos.base_offsets[0] + cmos.area.rel_offset)[0].hex(' ')
'10 76 09 e9'
>>> new_checksum = calculateChecksum('./DataSamples/SN33-Real.img')[0]  # Default is block [0]
>>> writeChecksum('./DataSamples/SN33-Real.img', new_checksum , cmos.base_offsets[0] + cmos.checksum.rel_offset)
>>> readChecksum('./DataSamples/SN33-Real.img', cmos.base_offsets[0] + cmos.checksum.rel_offset).hex(' ')
'10 76 09 e9'

I think that's enough for this script to be useful so I'll leave it at this.
If I get the time I'll look into that LSB error more, it's probably an odd/even parity bit, but for now just make two images and add 1 to the checksum, it's probably a 50/50 the first one works.

The +1 LSB error isn't consistent but that 01 at the start of the header is static, scripts that run verifyImageHeaders() will exit with an error if they don't find 01 00 00 00 98 ba dc fe at the start of every CMOS block. There's something else that must change with the data, I just checked odd/even (when uint32 interpreted) parity tonight and it’s not that, couldn’t find a correlation.
It doesn’t seem to be correlated with the CMOS blocks being different either, this is my understanding of the checksum differences at the moment:

Data in CMOS blocks is IDENTICAL
Data in CMOS blocks is DIFFERENT

Unknown Case 1 (our checksums are accepted)
Both checksums identical and valid
Each block's checksums different but both are valid (Always 2nd LSB diff so one is +1 other is -1, none at the between value)

Unknown Case 2 (game resets or does weird stuff)
Both Midway's checksums are identical but 1 LSB HIGHER than ours
Each block's checksums different but both Midway's are +1 LSB higher than ours (same +1/-1 LSB apart thing)

If I have time tomorrow I might try to dig further through HYDRO.EXE and see if there’s any obvious bit mask for some sort of parity check. If we’ve managed to narrow the checksum down from 4,294,967,296 possible values to 2, I think we can find this last bit somewhere.
HTchecksumBruteForceFind.py was mostly for when I thought it was either four SUM8 or two SUM16 checksums and I was trying to find where the different regions of the CMOS data were, but I ran it on SUM32 for completeness and noticed by chance when it would fail to find anything the last checksum it did for the full span of the CMOS area was off from the correct checksum by the same value between all 8 image files but with the LSB error on some. I wrote HTchecksumAnalysis.py after that and saw the massive correlation, subtracted the difference as a seed and it started calculating correct checksums (less the LSB error). The header does contain a suspiciously similar byte sequence to the seed but the first byte is different, I thought maybe it was clearing the checksum and calculating it for the whole CMOS block header included but I couldn’t get that to work with 0x0 for the checksum, does work if you put different seed in place of the checksum, but our seed is equivalent to whatever they are doing before checksumming the data and is easier to implement.
I agree you can safely remove all the extra bits at the start of the image, when I reworked HTchecksumUtils.py I made it use relative offsets from the CMOS header so it can be at any absolute offset you want, but I’m not sure what offset you’d want to start the image at, you could start at the first bit of the first CMOS block?
I would agree in not pursuing a fancy GUI for this, I think the venn diagram for “people who don’t know how to use a CLI script” and “technical people who have bought a 1999 PC arcade machine and want to change/backup the high scores and have extracted the HDD and adapted it from IDE to a modern system” don’t overlap at all, likely no one would use it. But furthermore, this script is for a 1999 arcade machine, I would not be surprised if someone another 24 years in the future has to use this script to restore their settings or high scores, trying to run an old Python 3.10 script from 2023 will be annoying enough, one with lots of dependences for pyqt would probably be worthless to them, I tried to do everything in HYchecksumsUtils.py using built-ins and no 3rd party packages for this reason.
With that said I think it should still have some nicely formatted CLI output, I always like that in CLI scripts so you know what’s going on and if it was successful rather than scripts that just output a mess of data or nothing at all. Probably doesn't need to nicely display all the data, people can just use a spreadsheet program to view the output CSV.

My inclination right now is to overwrite both data blocks to be identical right now and use a +1 LSB. I haven't been able to fully test this (I will tomorrow) but that seems to be the most consistent method to pass so far. I'm working on adding an option right now to allow picking which data block to work with so the user can decide which to use. I don't think there will be a way to tell which is the newer one though as it would likely require decoding audit info which I will not be doing.
My plan is to make the "official" data block size 8,192 bytes starting from the first 01. The audit data from the hard drives of my used machines is about 1680 bytes which made the total data size somewhere around 5300 bytes, but I'm going to overshoot a bit just to be sure. I'm avoiding the term "CMOS" in everything I'm doing because I feel it confuses what and where the data is. I'm guessing they had a plan at some point to either use the BIOS ROM or maybe a chip on the Diego board to hold settings. But other than the references from reverse engineering the code we wouldn't come up with that name and it's never shown to the users. So I think it's just best to avoid it so it isn't confused with anything going on with the PC side of the arcade hardware. I'm trying to use "raw" in all of those places instead to indicated that it is not decoded data.
There may be an easy solution here for a pretty CLI output. I'll make all the parameters for the different sections just take an optional filename as a value. Then it can pretty print by default and only output CSV data to the filenames provided. The GUI idea can just be axed. Like you said, it really won't benefit the people who will likely use this tool.

The latest commit adds the parity check implementation for the previously unknown LSB error, it now calculates checksums for the supplied images perfectly 100% of the time, no messing around with +1 LSB!
...
  45 [./DataSamples/CF-Data-Race-slow.img]
Checksum for first cmos copy     : 93 8d c8 aa | 93 8d c8 aa<PHONE_NUMBER>0010001000110110010011
Checksum for second cmos copy    : 91 8d c8 aa | 91 8d c8 aa<PHONE_NUMBER>0010001000110110010001
Checksum abs diff between copies : 02 00 00 00 | 02 00 00 00<PHONE_NUMBER>0000000000000000000010 LOWER CHECKSUM
Checksum abs diff between midway :               00 00 00 00<PHONE_NUMBER>0000000000000000000000 Valid Checksums!
                                 |  Midway's   | My checksum
                                 |  Checksum   | SUM32 | little | 94 ba dc fe | 100% of Checksums Match (46)

The machine seems to know which block is newer by looking at the blocks of data after the two CMOS blocks that contains the Steven E. Ranck string, looking at the decompiled functions that write this data it appears to write a 64-bit timestamp from the RDTSC counter (uptime, not a datetime). Whichever block has the biggest uptime (written latest) and has a valid checksum seems to be used. This seems to be solely for power loss protection on an incomplete write so we can just ignore this if we duplicate the CMOS data into both blocks and leave the timestamps untouched in that extra block.
8,192 bytes is enough for all the data that thing could possibly write. But I will note it looks like in the code they’ve hardcoded each CMOS block as 0x7d00 (32,000) bytes long spaced 0x30000 (196,608) bytes apart, but they’re not using anywhere near that much of it so doing 200 kB of 00 read/writes would just be pointless and spend the CF card endurance, reading and checksuming on 8,192 bytes is good.
I agree CMOS is a terrible name for it, really implies they were going to use CMOS BIOS/RTC user data space but gave up on it either for performance, space or complexity reasons and just reused the code and dumped it to the HDD. I really wonder if there’s a 1.00a or 0.9X build of Hydro Thunder (or others in the Thunder series) that actually do save data to real CMOS. Calling it ‘raw data’ is probably a good call, but still mention it as being internally called “cmos” in the technical documentation.
Also, there is an ATMEL EEPROM IC on the Diego PCB as U10 (guess, can’t find a good picture of it) and but it’s used in HYDRO.EXE for some settings (maybe cabinet style?) but it’s referred to in functions as  _diego_io_write_eeprom rather than cmos so I don't think it was intended for this data.

Awesome! Being parity makes sense why it was intermittent, I'll get that added into the main code as well. Last night I got it to a functional state aside from that where it can import/export CSVs for times/splits and the whole raw data blocks from a complete drive image of arbitrary size(Although I gave up on checking to only write diffs so far). Getting parity added should make it "beta" worthy after I clean up the code some more and rewrite the documentation for what the new parameters do.
Uptimes make perfect sense, I had a suspision that was what those later bits of data were but I still haven't been able to put the time into cracking open the EXE myself to look into it. I was trying to think of how it would know which of the two data blocks was the newest one and couldn't see if being possible without some kind of outside data.
I don't like that I couldn't get a solid answer on the real data size but the 8,192 bytes does seem to work. And yeah, it would wasteful to do the full size they allotted to it. I'm still pretty sure they are starting at one point in memory and just moving a pointer while writing to the drive. So it not an intentional size thing.
I'm settling on a name like "Data Blocks" and have added an option to select which "block" you import/export from. It would be interesting to know more about the development history of the machine for sure to know why it was called that. I will add a mention about it being called cmos somewhere in the TECHNICAL.md file I've been working on for sure. I've been mostly writing that out to get my thoughts sorted on how to write the program and get notes for the video on this.
Ah, I wonder if U10 is the chip I've heard needs dumped on the Diego board. I'll have to take a look at dumping that some time. The manual mentions that it will use a dip switch setting from that board as the default for the cabinet size though. But that wouldn't need an EEPROM.

After trying to make much broader changes to the stored track times, I've run into what may be another parity bit or different check because I can reliably get the game to reset even with the new parity bit. I have manually figured out what the correct value should be for this test case though.
To reproduce:
Use these files: https://github.com/AkBKukU/HydroThunder-TimeTool/blob/main/DataSamples/test-case.tar.gz
And this commit: https://github.com/AkBKukU/HydroThunder-TimeTool/commit/310b4185be4be83304788087c1b3310c8a8764b4
Running the following command uses the current checksum/parity scheme as is, it produces a raw data block with an invalid checksum for the game:
python3 ht-time.py -r free-0.img --times times-bad.csv --write_raw mod-times.img --lsb_offset 0
Your Manual Check tool will give it a pass with these results:
$ python3 HTchecksumManualCheck.py mod-times.img

Checking supplied offsets point to start of CMOS areas in image:
  Image # 0 PASS : Found [01 00 00 00 98 ba dc fe] @ 0x0 [mod-times.img]
  Image # 0 PASS : Found [01 00 00 00 98 ba dc fe] @ 0x0 [mod-times.img]

                                  |     Old      |    Newly     |
                                  |    Stored    |  Calculated  |
                                  |   Checksum   |   Checksum   |  Checksum Area Span
----------------------------------|--------------|--------------|-----------------------
CMOS area #0 Checksum @ 0xc | e4 23 14 16  | e4 23 14 16  | 0x14 → 0x1513
CMOS area #1 Checksum @ 0xc | e4 23 14 16  | e4 23 14 16  | 0x14 → 0x1513

If I adjust the LSB by +2 then the game will accept it:
python3 ht-time.py -r free-0.img --times times-bad.csv --write_raw mod-times.img --lsb_offset 2
As a baseline you can run the following with the checksum/parity as is and it will be valid:
python3 ht-time.py -r free-0.img --times times-good.csv --write_raw mod-times.img --lsb_offset 0
The only difference between times-good.csv and times-bad.csv is changing R to B on the initials in the top score for Thunder Park. times-good.csv was based on the stock times and adding current world record times one at a time until the error occurred. These other changes must be made as well in times-bad.csv only changing the single R to B does not make the checksum invalid.

Test-images.zip
Here are the data block images I made on stream.

pre-thunder.img [Works] : Before adding the Thunder Park time
post-thunder-2lsb.img [Works] : Before adding the Thunder Park time with a +2 LSB
post-thunder-0lsb.img [Fails] : Before adding the Thunder Park time with unchanged LSB per checksum+parity

post-thunder-2lsb.zip
Correct 2LSB offset post-thunder

thunder-time.zip
Time set with post-thunder loaded

thunder-time-no-audit.zip
Removed audit data

Before I sleep and forget this. The only thing I can think of is it has to be a rounding thing. You put in a time that isn't rounded the same, the game reads and rounds it, the time changes, then it does the checksum which ends up 2 LSB higher. Solution may be to round all user inputs with the formula I put in that other issue. Idk I didn't think too hard about it I sleep now.

I was having a similar thought right after stream but what you are speculating could make sense. If it is running the checksum on the section of RAM in the game and it automatically processes the floats I could see that happening.
The only thing is that earlier I was saying I got it down to a point where I could change just a letter in an initial. So I don't know if the floats are it. We might even be finding an edge case bug, who knows.
I'll trying messing with it some more another day still to see what I can find. I think a good next step might be cutting the times to just tenths of seconds to see if it can load arbitrary data like that. It will be annoying if emulating the game becomes necessary to monte carlo test this with random data until we find a pattern. It may also be time to see if I can get in contact with the original developer to see if they have any insight.

Currently those algorithms predict checksums with 100% accuracy, and I cannot find anything that would argue it is not implementing some other undiscovered feature of the checksum so I’m betting this a materialization of the Centiseconds Rounding Bug Part 2: Electric Boogaloo and will continue this discussion in the issue for it.

Issue fixed in pull #8.
This branch's code was experimental and is not useful for the final ht-time.py implementation so won't be merged.


Added missing placeholder for "format()"

This doesn't give any error... but doesn't print the list neither!

Ehm... I must admit that only now I saw there was a "v0.3.2dev" branch with much more recent commits.
(I must also admit that I am not exactly a fan of how this project uses branches...)
This pull request (as well as the other I just made) is unchecked/untested in v.0.3.2dev (because of https://github.com/dr-leo/pandaSDMX/issues/22 )

Hi Pietro,
thanks a lot for all this.
I don't think my branch policy, if any, is acceptable by any standard. I
have a blind spot here as I am reluctant to delve into git. Locally I
work with Mercurial. This is no excuse, I know. But I had other
priorities so far.
Maybe I should commit to master only and tag each release rather than
open a branch unless I fiddle with stuff. So far I would have been
better of without any of those branches...
Leo
Am 04.11.2015 um 23:39 schrieb Pietro Battiston:

Ehm... I must admit that only now I saw there was a "v0.3.2dev" branch
with much more recent commits.
(I must also admit that I am not exactly a fan of how this project
uses branches...)
This pull request (as well as the other I just made) is
unchecked/untested in v.0.3.2dev (because of #22
https://github.com/dr-leo/pandaSDMX/issues/22 )
—
Reply to this email directly or view it on GitHub
https://github.com/dr-leo/pandaSDMX/pull/19#issuecomment-153892389.

Done.

Il giorno mer, 04/11/2015 alle 21.53 -0800, dr-leo ha scritto:

[...]
Maybe I should commit to master only and tag each release rather than
open a branch unless I fiddle with stuff.

That would be great!
Then versioned branches in principle can make sense (e.g. you want to
avoid experimental changes in "master" to affect a version you're about
to release, but still polishing, so the two must diverge), but in your
repo, currently, version-specific branches... never diverge! So they
are of no use, as far as I can tell.
(and in any case, I think it would be better to avoid giving the same
name to branches and tags)
Pietro