Call fixup for pointers to member functions

It seems that the Metrowerks Codewarrior uses a special helper function to perform calls made via the pointers to member functions (ptmfs).
Basically, a single ptmf is represented by a struct with sizeof == 12 bytes where all needed info is stored. The semantics of the call are:

Load the ptmf struct into 3 fpu regs (4+4+4 == 12)
Push it to the stack
Setup the this pointer
Put the PTMF's stack address to the t9 reg
Call __ptmf_scall
__ptmf_scall fetches the ptmf from the stack via t9, grabs the function address and unconditionally jumps there.
(It doesn't matter where the ptmf comes from - either grabbed from .data or constructed in runtime)

Right now I'm not happy with the decompilation results as Ghidra doesn't show that's a function member call, doesn't show the ptmf setup and doesn't show the arguments. I would like to fix this if possible. Inlining didn't work (the callee got batchered) so I'm currently hoping a call fixup can be applied.
Can I make a custom calling convention / call fixup that would force Ghidra to properly detect the ptmf stack setup as described in the list above? If so, where can I look for examples? I personally want Ghidra to show that's a ptmf call somehow and apply the data types for the args and the return value.
res = mw_ptmf( /*some deduced ptmf info*/ , args); instead of res = __ptmf_scall(auStack44) (and it's the only place where auStack44 is used).

This could be modelled by making __ptmf_scall have a variable number of arguments, and a PTMF structure pointer. However, to do that, you'd need to use "Custom Storage" to set the PTMF pointer to register t9. Unfortunately, using both "varargs" and "Custom Storage" on the same function confuses the decompiler (see #4334, which has been in triage for over a year now :cry:).
The specific data types for the arguments would depend on the PTMF being executed, right? So making a custom calling convention that hardcodes the data types would probably not be a good solution. You should be able to create a custom calling convention that takes the first argument in t9 and then the remaining arguments "as usual". This is done in the .cspec file. I couldn't find any documentation in the code, but going by this comment, it seems it's as easy as copying the default calling convention, giving it a new name and inserting lines similar to this directly after the <input> tag:
  <pentry minsize="1" maxsize="8">
      <register name="t9"/>
  </pentry>

If you do this, you shouldn't need "Custom Storage", and can probably just set the __ptmf_scall function to use your new calling convention, and set it to varargs.

Can I make a custom calling convention / call fixup that would force Ghidra to properly detect the ptmf stack setup as described in the list above? If so, where can I look for examples? I personally want Ghidra to show that's a ptmf call somehow and apply the data types for the args and the return value.

Yes. In the code browser go to edit -> Options for Program Name->Specification Extensions. There you can add your own call fixups.
You can export an existing one as an example. You'll probably want to reformat it though. Note that if the body is multiple lines you may need to use <![CDATA[fixup body]]>

...
You can export an existing one as an example. You'll probably want to reformat it though. Note that if the body is multiple lines you may need to use <![CDATA[fixup body]]>
...

Exporting an existing calling convention doesn't give register names etc. in the xml, it just spits out numeric values!

...
You can export an existing one as an example. You'll probably want to reformat it though. Note that if the body is multiple lines you may need to use <![CDATA[fixup body]]>
...

Exporting an existing calling convention doesn't give register names etc. in the xml, it just spits out numeric values!

I meant for a call fixups. I didn't realize it was asking about both. The exporting of call fixups works at least.

@LukeSerne
Before making this issue I used the void __thiscall __ptmf_scall(void *this,...) signature with varargs and now decided to experiment. void __stdcall __ptmf_scall(void *this,PTMF *ptmf) with the custom storage for the second argument works for the ptmfs that accept no args. As you said, the changes got reverted after I added the ellipsis:

Unfortunately, using both "varargs" and "Custom Storage" on the same function confuses the decompiler

So yeah, confirmed that...
But you also said I should try a custom calling convention, so my question is... If I "hardcode" this in a, say, __mw_ptmfcall convention, then apply it to __ptmf_scall and make it variadic... Would it "unconfuse" the decompiler? (Because if it doesn't, then why bother?)

@LukeSerne Before making this issue I used the void __thiscall __ptmf_scall(void *this,...) signature with varargs and now decided to experiment. void __stdcall __ptmf_scall(void *this,PTMF *ptmf) with the custom storage for the second argument works for the ptmfs that accept no args. As you said, the changes got reverted after I added the ellipsis:

Unfortunately, using both "varargs" and "Custom Storage" on the same function confuses the decompiler

So yeah, confirmed that...
But you also said I should try a custom calling convention, so my question is... If I "hardcode" this in a, say, __mw_ptmfcall convention, then apply it to __ptmf_scall and make it variadic... Would it "unconfuse" the decompiler? (Because if it doesn't, then why bother?)

Use a call fixup.
__sdivsi3.xml.txt
__aeabi_uldivmod.xml.txt
NOTE: I make no guarantees to the accuracy of the attached examples.

Load the function pointer and make it a call

But this is exactly what __ptmf_scall is doing. Wouldn't I just reimplement inlining? (And we know inlining didn't work)

Load the function pointer and make it a call

But this is exactly what __ptmf_scall is doing. Wouldn't I just reimplement inlining? (And we know inlining didn't work)

No it is not. Your call fixup should just implement the call.
t9 = fn_ptr;
call [t9];

It's not that easy.
First of all, I have just noticed that I didn't tell that __ptmf_scall also adjusts the this argument to properly deal with virtual functions.
Second of all...yeah... virtual functions and non-virtual functions are handled differently.
The whole point of it being a compiler-generated helper function is that it's not trivial!
I'm from my phone right now, but afaik the meaning of __ptmf_scall is:
If vtable offset is -1, it's a non-virtual call and we can grab the address from the last 4 bytes of the struct.
Else we increment this by the value in the first 4 bytes, grab the vtable offset and take the address from there.
This is why I don't understand how inlining would be any different from an accurate call fixup...

It's not that easy. First of all, I have just noticed that I didn't tell that __ptmf_scall also adjusts the this argument to properly deal with virtual functions. Second of all...yeah... virtual functions and non-virtual functions are handled differently. The whole point of it being a compiler-generated helper function is that it's not trivial!
I'm from my phone right now, but afaik the meaning of __ptmf_scall is: If vtable offset is -1, it's a non-virtual call and we can grab the address from the last 4 bytes of the struct. Else we increment this by the value in the first 4 bytes, grab the vtable offset and take the address from there.
This is why I don't understand how inlining would be any different from an accurate call fixup...

You don't need the real address. The goal is to show the function call without the noise. Take the pointer to member function, move it to t9 and then call [t9]. The result will be an indirect function call without the implementation details, which is what you want.

You don't need the real address.

If we follow your logic, I also don't want the real addresses for normal calls (and don't want proper decompilation).
Real life example of a virtual ptmf values: {0x0, 0xA8, 0xC}. Be my guest and explain how I would benefit from a call to 0xC?
The vtable start is at *(this + 0x0 + 0xC) and the function I need is at *(vtable_start + 0xA8).
Real life example of a nonvirtual ptmf values: {0x0, -1, 0x08837960}. Here the address is 0x08837960, because the second value is negative.


autoGeneratedHeaders key appears as a git change

autoGeneratedHeaders value fluctuates between null and []

Is this related - #24944?


Improved dat create output

Just noticed a few grammatical & spelling errors.
Having removed these errors, the project will feel a little more legit to a newcomer 😄

Oh jeez, thank you! Have read it way to often to see those.

Haha no worries, gets the best of us.


Update to latest napari and pydantic

Allow Viewer and RegistrationViewer to work on latest napari version

Will require removing the varying gene point sizes, since this feature was removed in later versions of napari

@reillytilbury what do you think about removing the variation in gene point sizes? Do you think that would be okay? I think the fact that we can choose z plane as well as z plane thickness in the Viewer is sufficient control to understand what genes are in what z plane.

Hi @paulshuker. I think that this is a feature we would really like to keep. This seems like the only way that we can actually keep the z-thickness slider, which is really helpful when data on a single z-plane is sparse.

Oh really, would the thickness slider not continue to work?

Nope, the z-thickness slider simply increases the size of the markers (in z).

Oh I see, but what we could do is just add or remove into the plot when the z thickness varies?

Hi @paulshuker. Yes, there are definite workarounds. Currently we only make spots at the initialisation of the Viewer object. The first n_anchor_spots spots refer to the anchor, then all following spots are the OMP spots.
Creating spots just once should be simpler for spot selection, as the masks we need for each gene will be constant.
I think that the best thing to do is to stay with it in its current form, but it seems like we are going to need to revamp the viewer in a pretty big way so maybe we can redo this?

I actually changed this slightly so that the first spots are anchor spots * 2 for the gene probabilities as well as the anchor spots, then OMP is appended on top of them.
Refactoring our Viewer so that the plot update is handled by generating a new napari image layer each time will make our code much easier to manage in the future and add new features. I am not concerned about speed because napari is always working to speed up their Viewer and when the Viewer is run the spot information is kept in memory.

To update napari Reilly and I think it will be necessary to refactor the z thickness code so that points are switched off and on based on their z coordinates and the selected z thickness. This will remove the need for varying point sizes. This would remove the anisotropic point sizes which is not in the latest napari version.

Done and z-thickness sliders were added with manual event functions in main results viewr and icp point cloud viewer.


Implement expr.

Implemented as follows:

Usage: expr EXPRESSION
  or:  expr OPTION

      --help     display this help and exit
      --version  output version information and exit

Print the value of EXPRESSION to standard output.  A blank line below
separates increasing precedence groups.  EXPRESSION may be:

  ARG1 | ARG2       ARG1 if it is neither null nor 0, otherwise ARG2

  ARG1 & ARG2       ARG1 if neither argument is null or 0, otherwise 0

  ARG1 < ARG2       ARG1 is less than ARG2
  ARG1 <= ARG2      ARG1 is less than or equal to ARG2
  ARG1 = ARG2       ARG1 is equal to ARG2
  ARG1 != ARG2      ARG1 is unequal to ARG2
  ARG1 >= ARG2      ARG1 is greater than or equal to ARG2
  ARG1 > ARG2       ARG1 is greater than ARG2

  ARG1 + ARG2       arithmetic sum of ARG1 and ARG2
  ARG1 - ARG2       arithmetic difference of ARG1 and ARG2

  ARG1 * ARG2       arithmetic product of ARG1 and ARG2
  ARG1 / ARG2       arithmetic quotient of ARG1 divided by ARG2
  ARG1 % ARG2       arithmetic remainder of ARG1 divided by ARG2

  STRING : REGEXP   [NOT IMPLEMENTED] anchored pattern match of REGEXP in STRING

  match STRING REGEXP        [NOT IMPLEMENTED] same as STRING : REGEXP
  substr STRING POS LENGTH   [NOT IMPLEMENTED] substring of STRING, POS counted from 1
  index STRING CHARS         [NOT IMPLEMENTED] index in STRING where any CHARS is found, or 0
  length STRING              [NOT IMPLEMENTED] length of STRING
  + TOKEN                    interpret TOKEN as a string, even if it is a
                               keyword like 'match' or an operator like '/'

  ( EXPRESSION )             value of EXPRESSION

Beware that many operators need to be escaped or quoted for shells.
Comparisons are arithmetic if both ARGs are numbers, else lexicographical.
Pattern matches return the string matched between \( and \) or null; if
\( and \) are not used, they return the number of characters matched or 0.

Exit status is 0 if EXPRESSION is neither null nor 0, 1 if EXPRESSION is null
or 0, 2 if EXPRESSION is syntactically invalid, and 3 if an error occurred.

Environment variables:
	* EXPR_DEBUG_TOKENS=1   dump expression's tokens
	* EXPR_DEBUG_RPN=1      dump expression represented in reverse polish notation
	* EXPR_DEBUG_SYA_STEP=1 dump each parser step
	* EXPR_DEBUG_AST=1      dump expression represented abstract syntax tree

The operations marked as not implemented ( match/2, substr/3, index/2, length/1 ) are correctly parsed as prefix-operations with specified arity.
They are just missing implementation of their actions: string operations.

Thanks


Doesnt seem to play nice with google cloud functions?

When using this package in google cloud functions, we get errors with the function not terminating.
In google cloud console the function is forcibly terminated after 15 minutes with the error "Error: Process exited with code 143 "
Even tho the process has completed, there is something still "hanging"
I tracked it down to exit_hook. Using patch package to remove it from process-logs.ts seemed to solve it for me....
Is exit_hook needed if sendImmediate is true?
Should this issue be opened in exit_hook instead?
Thanks!

exitHook is two major versions behind since the next major is ESM only. It is probably some bug that may have been corrected later.
I'll look into seeing if there's an alternative or I just fork the thing.

I've pushed a new 3.0.0 version that updates the exit hook to the latest version, but also only enable it if sendImmediate is false.
let me know if it fixes your issue. If not, please open an issue on:
https://github.com/theogravity/datadog-transports/issues
as I've moved the code there.


Replace outdate library underscore.string by voca to fix emoji issue

Currently metalsmith-better-excerpts plugin has the problem with emoji symbols because of the issue in unescapeHTML function of underscore.string library. The issue connected to String.fromCharCode function usage (see code) and detailed explained on stackoverflow question Printing emojis with JavaScript and HTML.
Unfortunately, as I see from GitHub repository of underscore.string project it did not support from 2016. So creating the issue to it is senseless.
Fortunately, exist almost identical voca project. And just by replacing underscore.string by voca (with minimal code update) emoji symbols start processed correctly.

Ups sorry forget about tests. I will revise errors and update my pull request shortly.

Thanks for the changes. It's already merged but it seems that i have to fix the travis config.
I will look into that asap, so the new version can be released.

Released.


raise_solidity_error_on_revert

File "/home/ubuntu/.local/lib/python3.8/site-packages/web3/_utils/method_formatters.py", line 580, in raise_solidity_error_on_revert
raise ContractLogicError('execution reverted')
web3.exceptions.ContractLogicError: execution reverted
Getting hit when I use uniswap.make_trade(***)

I think it is a good idea to search for your error code before creating an issue.

Should have searched other issues first, and not enough info to resolve anyway. Closing.


Semantic release pushing only package.json to npm

Hi
Everything is working on Travis, testing, building, release new version on GIthub... BUT:
The released package on NPM only has package.json and README.
I'm not sure if it's a configuration that I'm missing in typescript-library-starter, a bug in semantic-release... Or probably just me missing something! But it's definitely driving me crazy and I can't figure out why my package is not correctly deployed to NPM.
So I tried to do some debug on Travis and I've added ls -a right before the yarn run semantic-release.
The result is normal:
$ ls -a
.
..
bin
coverage
.editorconfig
.git
.gitignore
node_modules
.nojekyll
package.json
README.md
rollup.config.ts
.rpt2_cache
src
tools
.travis.yml
tsconfig.json
tslint.json
yarn.lock

So all my files are here and my .gitignore contains the following:
node_modules
coverage
.nyc_output
.DS_Store
*.log
.vscode
.idea
dist
compiled
.awcache
.rpt2_cache

The folder supposed to be deployed is bin. So as long as I have my package.json and the bin folder it should be ok (it's a global package, thus the bin...).
You can take a look into Travis logs here: https://travis-ci.org/maxime1992/pulse-issues
If you want to see the code, it's all on Github here: https://github.com/maxime1992/pulse-issues
Thanks for any help on that :+1: and congrats for this starter which is really nice :smile:!

By default, as you can see in the package.json https://github.com/alexjoverm/typescript-library-starter/blob/master/package.json#L10, the dist folder is the one published. Whatever you need to be published, you'll need to include it in files property.

Thank you :pray:, completely missed that part...


Bug Report: Plugin Fails with Brotli Compressed JavaScript Files

Describe the bug
Traefik fails to serve the replaced page when the response is a Brotli-compressed JavaScript file (whether it is text/javascript or application/javascript). The transformation is visible in the Docker logs, and the replacement works well when the response encoding is gzip.
Relevant Config
Static Config:
providers:
  docker: {}
  file:
    directory: "/etc/traefik/dynamic"
    watch: true
log:
  filePath: "/logs/traefik.log"
  maxAge: 1
  level: DEBUG
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
api:
  dashboard: true

experimental:
  plugins:
    traefik-plugin-rewritebody:
      moduleName: "github.com/the-ccsn/traefik-plugin-rewritebody"
      version: "v0.1.3"

Dynamic config:
http:
  middlewares:
    rewrite-to-githubassets:
      headers:
        customRequestHeaders:
          Host: "github.githubassets.com"
    my-rewrite-body:
      plugin:
        traefik-plugin-rewritebody:
          # Keep Last-Modified header returned by the HTTP service.
          # By default, the Last-Modified header is removed.
          lastModified: true

          # Rewrites all "foo" occurences by "bar"
          rewrites:
            - regex: "global-create-menu"
              replacement: "testReplacement"

          # logLevel is optional, defaults to Info level.
          # Available logLevels: (Trace: -2, Debug: -1, Info: 0, Warning: 1, Error: 2)
          logLevel: -2

          # monitoring is optional, defaults to below configuration
          # monitoring configuration limits the HTTP queries that are checked for regex replacement.
          monitoring:
            # methods is a string list. Options are standard HTTP Methods. Entries MUST be ALL CAPS
            # For a list of options: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods
            methods:
              - GET
              - POST
            # types is a string list. Options are HTTP Content Types. Entries should match standard formatting
            # For a list of options: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types
            # Wildcards(*) are not supported!
            types:
              - text/html
              - text/css
              - text/xml
              - text/javascript
              - application/x-www-form-urlencoded
              - application/json
              - application/javascript

  routers:
    dashboard:
      rule: Host(`traefik.docker.localhost`)
      service: api@internal
      entryPoints:
        - websecure
        - web
      tls:
        domains:
          - main: "docker.localhost"
            sans:
              - "*.docker.localhost"
          - main: "domain.local"
            sans:
              - "*.domain.local"

    git-asset:
      rule: Host(`githubassets.docker.localhost`)
      service: githubassets
      entryPoints:
        - websecure
        - web
      middlewares:
        - rewrite-to-githubassets
        - my-rewrite-body
      tls:
        domains:
          - main: "docker.localhost"
            sans:
              - "*.docker.localhost"

  services:
    githubassets:
      loadBalancer:
        servers:
          - url: "https://github.githubassets.com/"
        passHostHeader: true
tls:
  certificates:
    - certFile: "/etc/certs/local-cert.pem"
      keyFile: "/etc/certs/local-key.pem"

Expected behavior
The Brotli-compressed JavaScript file should be served with the replaced content as specified in the regex replacement.
Actual behavior
The Brotli-compressed JavaScript file is not served correctly. Instead, the response body is empty. The transformation appears to be successful in the Docker logs.
Logs
Traefik log file (Level: DEBUG):
2024-07-24T18:50:57Z DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 8bc98d0a1f3fcd91
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/hash.go:118:9: panic: github.com/andybalholm/brotli.backwardReferencePenaltyUsingLastDistance(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/h5.go:115:21: panic: github.com/andybalholm/brotli.FindLongestMatch(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/backward_references.go:41:49: panic: github.com/andybalholm/brotli.createBackwardReferences(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/encode.go:752:21: panic: github.com/andybalholm/brotli.encodeData(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/encode.go:1135:6: panic: github.com/andybalholm/brotli.encoderCompressStream(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/writer.go:70:5: panic: github.com/andybalholm/brotli.writeChunk(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/vendor/github.com/andybalholm/brotli/writer.go:106:12: panic: github.com/andybalholm/brotli.Close(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/compressutil/compressutil.go:106:10: panic: github.com/the-ccsn/traefik-plugin-rewritebody/compressutil.compressWithBrotli(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/compressutil/compressutil.go:52:9: panic: github.com/the-ccsn/traefik-plugin-rewritebody/compressutil.Encode(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=
2024-07-24T18:50:57Z ERR fmt/print.go:305 > plugins-storage/sources/gop-1717797873/src/github.com/the-ccsn/traefik-plugin-rewritebody/httputil/response_writer.go:92:18: panic: github.com/the-ccsn/traefik-plugin-rewritebody/httputil.SetContent(...) module=github.com/the-ccsn/traefik-plugin-rewritebody plugin=plugin-traefik-plugin-rewritebody runtime=

Docker's console log:
2024-07-25 04:50:57 reverse-proxy-1  | Rewrite-Body | DEBUG2024/07/24 18:50:57 restricted.go:50: Starting supported request: &{GET /assets/global-create-menu-cd6a29cd32f9.js HTTP/2.0 2 0 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7; Accept-Encoding: gzip, deflate, br, zstd;] Accept-Encoding:[deflate, gzip, br, zstd] User-Agent:[curl/7.81.0] X-Forwarded-Host:[githubassets.docker.localhost] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[3f5e8853da66] X-Real-Ip:[<IP_ADDRESS>]] 0xc019118480 <nil> 0 [] false github.githubassets.com map[] map[] <nil> map[] <IP_ADDRESS>:47904 /assets/global-create-menu-cd6a29cd32f9.js 0xc000bd69a0 <nil> <nil> 0xc0132f8b10 <nil> [] map[]}
2024-07-25 04:50:57 reverse-proxy-1  | Rewrite-Body | DEBUG2024/07/24 18:50:57 restricted.go:50: Response body: "use strict";(globalThis.webpackChunk=globalThis.webpackChunk||[]).push([["global-create-menu"],{90119:(e,_,s)=>{var o=s(657),r=s(79315);(0,o.t)("global-create-menu",{Component:r.G})}},e=>{var _=_=>e(e.s=_);e.O(0,["react-lib","vendors-node_modules_github_mini-throttle_dist_index_js-node_modules_primer_octicons-react_di-b40d97","vendors-node_modules_primer_react_lib-esm_Box_Box_js","vendors-node_modules_primer_react_lib-esm_Button_Button_js","vendors-node_modules_primer_react_lib-esm_TooltipV2_Tooltip_js","vendors-node_modules_primer_behaviors_dist_esm_focus-zone_js","vendors-node_modules_clsx_dist_clsx_m_js-node_modules_primer_react_node_modules_primer_octico-c56103","vendors-node_modules_primer_react_lib-esm_ActionList_index_js","vendors-node_modules_primer_react_lib-esm_ActionMenu_ActionMenu_js-node_modules_primer_react_-5b2420","vendors-node_modules_primer_react_lib-esm_FeatureFlags_FeatureFlags_js-node_modules_primer_re-d6531f","ui_packages_react-core_create-browser-history_ts-ui_packages_safe-storage_safe-storage_ts-ui_-682c2c","ui_packages_react-core_register-partial_ts-ui_packages_global-create-menu_GlobalCreateMenu_tsx"],()=>_(90119)),e.O()}]);
2024-07-25 04:50:57 reverse-proxy-1  | //# sourceMappingURL=global-create-menu-ef686f2af525.js.map
2024-07-25 04:50:57 reverse-proxy-1  | Rewrite-Body | DEBUG2024/07/24 18:50:57 restricted.go:50: Transformed body: "use strict";(globalThis.webpackChunk=globalThis.webpackChunk||[]).push([["testReplacement"],{90119:(e,_,s)=>{var o=s(657),r=s(79315);(0,o.t)("testReplacement",{Component:r.G})}},e=>{var _=_=>e(e.s=_);e.O(0,["react-lib","vendors-node_modules_github_mini-throttle_dist_index_js-node_modules_primer_octicons-react_di-b40d97","vendors-node_modules_primer_react_lib-esm_Box_Box_js","vendors-node_modules_primer_react_lib-esm_Button_Button_js","vendors-node_modules_primer_react_lib-esm_TooltipV2_Tooltip_js","vendors-node_modules_primer_behaviors_dist_esm_focus-zone_js","vendors-node_modules_clsx_dist_clsx_m_js-node_modules_primer_react_node_modules_primer_octico-c56103","vendors-node_modules_primer_react_lib-esm_ActionList_index_js","vendors-node_modules_primer_react_lib-esm_ActionMenu_ActionMenu_js-node_modules_primer_react_-5b2420","vendors-node_modules_primer_react_lib-esm_FeatureFlags_FeatureFlags_js-node_modules_primer_re-d6531f","ui_packages_react-core_create-browser-history_ts-ui_packages_safe-storage_safe-storage_ts-ui_-682c2c","ui_packages_react-core_register-partial_ts-ui_packages_testReplacement_GlobalCreateMenu_tsx"],()=>_(90119)),e.O()}]);
2024-07-25 04:50:57 reverse-proxy-1  | //# sourceMappingURL=testReplacement-ef686f2af525.js.map
2024-07-25 04:50:57 reverse-proxy-1  | Rewrite-Body | WARNING2024/07/24 18:50:57 restricted.go:50: Recovered from: reflect: call of reflect.Value.Uint on int Value

Reproduction Steps
The error occurs both when using a browser and when using the following curl command:
curl --compressed \
  --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" \
  --header "Accept-Encoding: gzip, deflate, br, zstd" \
  --insecure \
  -v \
  https://githubassets.docker.localhost/assets/global-create-menu-cd6a29cd32f9.js

According to my config file, It should replace text "global-create-menu" with "testReplacement".
The file's originally url is https://github.githubassets.com/assets/global-create-menu-cd6a29cd32f9.js. Although the file path is obfuscated, but it can be reproduced when accessing other file.
Server (please complete the following information):
-	OS: Ubuntu 22.04.3 LTS
-	Traefik Version: 3.1
-	Plugin Version: v0.1.3
Additional context
The cert I used here was self-signed. I want to set up traefik as a reverse proxy to an external website.
The replacement works well when the response encoding is gzip. The issue seems to be specifically with Brotli encoding.

Hi, it seems the bug is caused because there's something wrong with yaegi itself: https://github.com/andybalholm/brotli/issues/49
I will investgate into this later as soon as I've got some spare time. (Sorry for late responding, what's more, go newbie here, if you have any idea about solving this bug, please tell me :)

since the issue of yaegi has been fixed, the plugin should work with latest yaegi build, you can now give it a try :)


Compilation Error

I'm trying to build MGL, following instructions from build.md, but through the process I'm getting a missing header error. Concretely:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/monitor.c:30:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/internal.h:331:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/platform.h:42:
/Users/ilya/D/cpp/MGL/glfw/src/cocoa_platform.h:32:10: fatal error: 'MGLContext.h' file not found
#include "MGLContext.h"
         ^~~~~~~~~~~~~~
In file included from /Users/ilya/D/cpp/MGL/glfw/src/context.c:30:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/internal.h:331:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/platform.h:42:
/Users/ilya/D/cpp/MGL/glfw/src/cocoa_platform.h:32:10: fatal error: 'MGLContext.h' file not found
#include "MGLContext.h"
         ^~~~~~~~~~~~~~
In file included from /Users/ilya/D/cpp/MGL/glfw/src/input.c:30:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/internal.h:331:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/platform.h:42:
/Users/ilya/D/cpp/MGL/glfw/src/cocoa_platform.h:32:10: fatal error: 'MGLContext.h' file not found
#include "MGLContext.h"
         ^~~~~~~~~~~~~~
In file included from /Users/ilya/D/cpp/MGL/glfw/src/init.c:30:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/internal.h:331:
In file included from /Users/ilya/D/cpp/MGL/glfw/src/platform.h:42:
/Users/ilya/D/cpp/MGL/glfw/src/cocoa_platform.h:32:10: fatal error: 'MGLContext.h' file not found
#include "MGLContext.h"
         ^~~~~~~~~~~~~~
1 error generated.
1 error generated.
1 error generated.
make[3]: *** [src/CMakeFiles/glfw.dir/context.c.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [src/CMakeFiles/glfw.dir/monitor.c.o] Error 1
make[3]: *** [src/CMakeFiles/glfw.dir/init.c.o] Error 1
1 error generated.
make[3]: *** [src/CMakeFiles/glfw.dir/input.c.o] Error 1
make[2]: *** [src/CMakeFiles/glfw.dir/all] Error 2
make[1]: *** [src/CMakeFiles/glfw.dir/rule] Error 2
make: *** [glfw] Error 2

you need to tell XCode to link GLFW against the missing headers (which can be found in MGL/include), like this:

I've stumbled against the same problem. It happened on running build_external script, not building the XCode project. I assume that OP meant the same.
I'm also assuming that adding headers in 'build phases' won't resolve this. So I decided just to hardcode absolute paths to MGLContext.h and MGLRenderer.h in cocoa_platform.h and mgl_context.m. It helped, but the way is very rude.


Prepare for integration into PyCharm

[x] Python 2.4
[x] Python 2.5
[ ] Python 3.1
[ ] Jython 2.5
[ ] IronPython?
[ ] testCount?
[ ] Unicode characters in strings, do not use str(x)

Handling on pycharm side now


added sans-serif to font-family to display correctly on Linux

In Firefox on Linux (Ubuntu 22.04, as of writing the most recent LTS release) the theme appeared with serif text. Investigation showed the fonts Lucida and Verdana aren't available on this machine and the default (serif) was picked.
This change adds sans-serif as a last font suggestion to the font-family line.

This is my first public pull-request, so any guidance you can spare is greatly appreciated.
For example: I took the liberty to edit the minor version number in version and wcms-modules.json. Don't know if this is good practice or an inconvenience.

This is my first public pull-request, so any guidance you can spare is greatly appreciated.
For example: I took the liberty to edit the minor version number in version and wcms-modules.json. Don't know if this is good practice or an inconvenience.


v0.25.2 doesn't reliably run preload scripts in Webviews

Our last big blocker for moving to Chrome 42 is that we're seeing the preload script not execute reliably - on initial load it doesn't execute at all, and refreshing the page will allow some of them to execute but not all of them.
I'll dig into renderer/lib/init.coffee to see if I can get more actionable information than "It doesn't work Halp!"

The command line parameters of webview's renderer process would be very helpful, you can get it via process.argv in webview's devtools.

Hm, more weirdness - this is almost a 100% repro on OS X, but on Windows I haven't seen it yet. Here's process.argv on OS X, sure enough the preload script isn't there:
process.argv
["/private/var/folders/_c/hwfdpfg168s0tl7dlqzfrf_h0000gn/T/slack-build/Slack.app/Contents/Frameworks/Slack Helper.app/Contents/MacOS/Slack Helper", "--type=renderer", "--js-flags=--harmony", "--no-sandbox", "--enable-deferred-image-decoding", "--lang=English", "--enable-pinch-virtual-viewport", "--enable-delegated-renderer", "--num-raster-threads=2", "--channel=60816.7.559197518"]

I take it back, it's appearing in the login screen periodically, here's a Windows example:

--type=renderer --js-flags=--harmony --no-sandbox --enable-deferred-image-decoding --lang=en-GB --enable-pinch --device-scale-factor=1 --font-cache-shared-mem-suffix=2376 --enable-pinch-virtual-viewport --enable-delegated-renderer --num-raster-threads=1 --channel="2376.5.1597965734\645508156" /prefetch:673131151

Strangely enough, I'm getting more render processes than I'm expecting - with a single WebView on screen:

From the command line parameters Electron failed to append custom switches for webview's renderer process, can you work out a reproducible example? The simplest code seems to work:
var w = new WebView;
w.setAttribute('preload', 'file:///tmp/test.js');
w.src = 'http://google.com';
document.body.appendChild(w);

@zcbenz What happens if you do like 10 of those in a row? It seems to be timing-related, because I can affect how likely it will happen with opening devtools at the right time

@zcbenz I'll work on getting a repro for this tomorrow

Openning 10 webview in the same time works for me.

Ok! So it appears that this is related somehow to setting the user agent at the wrong time. I'm trying to get a failing test to prove it

It seems like a super reliable repro is:
var w = new WebView;
w.setAttribute('preload', 'file:///tmp/test.js');
w.src = 'http://google.com';
w.addEventListener('did-start-load', function() { w.setUserAgent("CustomUserAgent 1.0.0") });
document.body.appendChild(w);

It's difficult to find a way to call setUserAgent before the page loads correctly - calling it immediately will throw an exception, call it too late and it doesn't get applied. The only way that I've reliably found is to send an IPC message from the very top of the preload script to the renderer.

This is fixed by #1587


Disable Animation on Tab Bar Item press

Some users can sometimes find transition animation annoying when they navigate using the tabBarController. (when they press tab bar items)
Is there a way to disable animation on tabBar(_ tabBar: UITabBar, didSelect item: UITabBarItem) but enable it on both left and right swipe gesture?

Check release 1.3.1. You can disable it anywhere. Check the app for an example on how to use it.


tools: sync ratbagd.py from Piper - drop async commit

ratbagd now implements commit as normal always-successful call with a
potential Resync signal later if something failed on the device.
PR just so I can have an extra pair of eyes on the other changes here, thanks!

As in Piper, this should probably also now listen to the Resync signal to notify the user of any failure. Do you want to add that to this PR right away?

As in Piper, this should probably also now listen to the Resync signal to notify the user of any failure. Do you want to add that to this PR right away?

Nah, don't have the time for this atm. It's a bit tricky now anyway, the Resync signal works nice for piper but not so much for ratbagctl which is supposed to exit immediately. We'd have to put a forced wait in for a few seconds just in case it may fail but that makes ratbagctl less useful.


Bugfix: Removed moderator rights to edit posts; added test

Problem Description
Currently a user with moderation rights can edit anyone's posts. As discussed in the Communication Team Meeting, this behaviour is unwanted.
Changes

Removed moderator rights to edit other's posts
Created unit test

Steps for testing

Login as an instructor (has moderation rights)
Long press another users post
See that "Edit" is not available anymore
Press on a post of yourself
"Edit" is still possible

I think the UI test would definitely make sense; I'm not sure, though, if we should include them as part of this PR. The changes were discussed in the last meeting and seem to be urgent. I would include them in the upcoming release and create an issue for the UI tests. Furthermore, there seems to be an ongoing issue with the RobolectricRunner and the modal bottom sheet, which is another reason to postpone the UI tests.


Add Rudbeck

Creating this for a friend
School website: http://www.rudbeck.se/

Sollentuna kommun is the municipality, but our emails is hosted by the municipality

@MatthewHaskett Please provide us with an URL of a page or another proof (.pdf or a screenshot are OK) showing that the university recognizes the domain which you are submitting as an official email domain for the students.

Friend sent this video, not sure if it works.

@MatthewHaskett Yes, this is fine. An official instruction would be better, but given that the domain definitely belongs to a municipality, the poof works. Pull request merged. Please remind the friend that free educational license for JetBrains tools is for high and higher schools students, i.e. those who are over 13 of age. Thank you!


Improve BackupBucket controller test

How to categorize this PR?

/area testing
/kind flake
/priority critical
What this PR does / why we need it:
Improve the BackupBucket extension controller test by

increasing some wait timeouts
using a new health check function ExtensionOperationHasBeenUpdatedSince to ensure that reconciliation was started instead of statically waiting 2s
using ExpectWithOffset(1, ... in verification helper func

Which issue(s) this PR fixes:
Ref #2915
Special notes for your reviewer:
Release note:

NONE

Thanks @timebertt, appreciate it!


⚠ IMAP has degraded performance

In 720dead, IMAP (https://news.ycombinator.com) experienced degraded performance:

HTTP code: 200
Response time: 1172 ms

Resolved: IMAP performance has improved in 8660555 after 5 minutes.


AMBARI-23845 Manage Ambari UI style fixes

How was this patch tested?
Tested manually

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/2277/
Test PASSed.


[Feature request] Add two buttons to the player's selection window

Button "Select all" amd "Select none", while playing you have select all 5 players each time you share something, would be great to select them all or select none if you want to reset the selection.
Thanks for your work!

Yep, that's definitely something I've intended to add 👍

Great! Thank you!


add_missing_levels does not recreate missing levels

I used the add_missing_levels option to recreate the pyramid structure of DICOM file but it did not recreate the pyramid structure.
If I look into the code, in this line:
https://github.com/imi-bigpicture/wsidicom/blob/90671211ddb87096d19952d5e80a7199dda51b67/wsidicom
/file/wsidicom_file_target.py#LL79C1-L79C44
it seems that new_levels is set to "None" and then it never enters the "if" check later on.

Hi @nstatho,
Your observation is correct. Additionally the target-object is not used as a context in the WsiDicom.save() method so any new pyramid levels would be left open.
I have a draft for a fix here #100, but would like to test it more before merging.

@nstatho Can you try the fix in the PR #100?

I tried it and it seems to work. I tested it on several DICOMs and all except one gave an error. The error I got from that one DICOM seems to be related to that one specific file.
This is the error:
Frame count 2040 != Fragments 2023. Fragmented frames are not supported

Yes it worked with all except one indeed. I'll send you the file on a separate email to take a look.
If I view the file in PACS, you see black and white squares on certain magnifications so I think the file itself has an issue.

Yes it worked with all except one indeed. I'll send you the file on a separate email to take a look.
If I view the file in PACS, you see black and white squares on certain magnifications so I think the file itself has an issue.

The second and third layer seems to be missing tiles on the bottom row. For example, dataset.NumberOfFrames returns 2040 but len(dataset.PerFrameFunctionalGroupsSequence)  returns 2023.


Terraform 0.12 - Error: "features": required field is not set

Platform: macOS 10.15.3
Terraform Version: v0.12.23
AzureRM Provider: v2.1.0
Warning: Quoted type constraints are deprecated

  on .terraform/modules/emptyDisk/Azure-terraform-azurerm-manageddisk-518e2b3/variables.tf line 48, in variable "tags":
  48:   type        = "map"

Terraform 0.11 and earlier required type constraints to be given in quotes,
but that form is now deprecated and will be removed in a future version of
Terraform. To silence this warning, remove the quotes around "map" and write
map(string) instead to explicitly indicate that the map elements are strings.

Error: "features": required field is not set

This is related to #2


Create an animated logo screen for videos

Hey there! I'm looking forward to recording the talks tonight! Could someone throw together a page on the site that I can use as a title screen for the videos? I'm thinking basically the animated background of the site with a large centered "Donut JS" title and logo. I just need it up at a URL and I can screencap it from there! 🎉

@aaronpk I can probably do this. When do you need it? By tonight?

I probably won't have time to cut the videos until next week, so no rush. Unless we're livestreaming tonight, in which case yes.

@paulcpederson bump! I'm planning on finishing the videos this weekend!

does running document.querySelector('.constrain').remove() in a javascript console on the homepage work for you?

@aaronpk @paulcpederson @jden Thank you for your ideas and input on this, and I'm very sorry for the delay. Here is a special ‘splash screen’ with just the logo and sprinkles, as requested: http://donutjs.club/splash/

Thanks! This will be great for the videos as well as for the live events!


Add common crypto utilities used by ccip.

These utilities are used widely for CCIP security. They are also useful in general for applications that need to validate or create CCIP execution proofs.

I guess CI is probably failing because I used a fork. I'll re-open this PR once I have repo permissions.

closing in favor of #527


When using PackOnBuild in multitargeting project, only one TFM is packed

Given a project:
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net472;netstandard2.0</TargetFrameworks>
  </PropertyGroup>

  <PropertyGroup>
    <PackageId>TestNuGetizer</PackageId>
    <Description>Test Package</Description>
    <PackOnBuild>True</PackOnBuild>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NuGetizer" Version="*" />
  </ItemGroup>
</Project>

Running build will result in a package that only contains netstandard2.0 libs.
Reported by @mauroa

This is related to the way we set up the BuildDependsOn when PackOnBuild == true. Currently, we just run Pack after build, regardless of whether the build was single or multi-targeting. This causes the Pack to run once for each TF, causing each build to overwrite the previous package and therefore leaving only the last TFM built as the package contents.
We need to account for this inner build situation and not Pack in that case, only when doing an outer build (or non-multitargeting build).


⚠ Bofrost.at has degraded performance

In 6d908d4, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2665 ms

Resolved: Bofrost.at performance has improved in 1134f7a.


Support interpolation of waypoints for Omnidirectional and Ackermann steering

Extend VehicleTraits to support ackermann steered vehicles. Update Holonomic class.
Update Interpolate::positions()

One of the simplest approaches can be using Dubin curves, they basically deconstruct navigation between two arbitrary waypoints in a set of three of any of the following actions:

Turn left at minimum turning radius
Turn right at minimum turning radius
Go straight at maximum speed

However, while they are simple and easy to implement @codebot mentioned that a more realistic model would apply a constant angular acceleration to simulate a steering wheel being turned at constant speed, since a Dubin curve involves every turn being at minimum turning radius that would either create the risk of tipping over vehicles or force them to slow down a lot even for wide bends.
Thus I think we could have the planner model a vehicle through the following parameters:

Maximum linear velocity / acceleration
Minimum turning radius
Maximum angular acceleration

Then deconstruct the trajectory into a series of bends / straight parts following Dubin curves that, instead of being all at maximum turning radius, are at constant angular acceleration. We should probably apply a scaling to the linear speed to make it inversely proportional to the angular speed so that vehicles will slow down as they reduce the bending radius but still take wide bends at relatively high speed.
Afterwards it's a question of what the waypoints information coming from the planner should look like.
I think it could be useful if the planner output two new fields for each waypoint:

Target linear velocity
Turning direction (left / right / straight).

The most accurate (but probably most complex) bend model would split each turn into three points and probably look something like:

Entrance: with a target velocity to enter the bend at (to make sure that we execute it as fast as possible) and the direction (i.e. R). After this point the vehicle will start decelerating at -maximum_linear_acceleration (or other acceleration value) and turning right at maximum_angular_acceleration.
Mid point, where we start to exit the turn, hence apply maximum_angular_acceleration in the other direction and maximum_linear_acceleration (or other acceleration value)
Exit, where we plan to be having an angular velocity of zero and a target linear velocity.

Where I say other acceleration value I mean that probably we don't want vehicles to break to their maximum capability in the middle of a bend since it is a common cause of loss of control. We could just output a target velocity at the entrance, middle and end of the bend and do linear interpolation through a constant linear acceleration model with deceleration when entering and acceleration while exiting the bend to calculate the speed throughout.
This modelling implies an infinite jerk trajectory since we are changing both angular and linear acceleration instantly at the three points of the bend, if we want continuity at the acceleration level we can have a maximum_linear_jerk and maximum_angular_jerk parameter but imho that might be unnecessarily complex.

Here's a video detailing the curves plus more manoeuvers like reversing, and how to plan for those.
This begs the question as to how much rmf_traffic is going to handle with regards to the planning for such vehicles. For example, the ackermann steering plugin right now, given 3 waypoints that make up a bend for a right/left turn, breaks it down to simple CSC manoeuvers. Should this act of breaking down waypoints into CSC manoeuvers be done in rmf_traffic?
Tagging in @mxgrey

how much rmf_traffic is going to handle with regards to the planning for such vehicles.

My expectation/hope is that we can provide a "close enough" representation of these motions using the existing piecewise cubic spline format that rmf_traffic::Trajectory is already based on. The first step would be creating some utilities for generating piecewise cubic splines that reasonably approximate the Reeds-Shepp curves (ideally with some parameter for error threshold so users can decide how they want to trade-off between accuracy and waypoint density).
After we can generate the right kinds of trajectories, the next step would be supporting it in the planner. I'm thinking we can define a pure abstract interface for an "Interpolator" that can be used by the Planner class. The interface for Interpolator would be something (very roughly) like
class Interpolator
{
public:

  struct Branch
  {
    std::size_t arrival_waypoint;
    std::vector<rmf_traffic::Route> route;
  };

  virtual std::vector<Branch> branch_from(
    const rmf_traffic::agv::Graph& graph,
    std::size_t waypoint_index) const = 0;
};

Essentially the job of Interpolator would be to consider a waypoint on a navigation graph and tell the planner what the different options are for an agent that starts at that point. We can create an implementation of this interface for the existing straight-line steering type and a separate implementation for the new Reeds-Shepp steering type. Users would provide one implementation or the other to set what kind of steering the want. In the future other implementations of the interface could be developed to support yet more types of steering (e.g. holonomic omnidirectional).


BigInt browser support

BigInt isn't supported in a lot of browser versions, would you consider polyfilling it or using another method like https://gist.github.com/diafygi/90a3e80ca1c2793220e5/?

The adoption number is 90% and is growing every day. One year ago Safari didn't support bigints.
No, I won't change it. I develop other cryptographic libraries that use Bigints. Polyfilling bigints take 6 thousand lines of code. I specifically developed the libs to not depend on bigints.

Thanks for the reply, we've solved it by using our own adaptation without use of BigInt.


feat: add global image settings for all images in the EG chart

This PR exposes a top-level global image configuration in the helm value file for images used in the EG helm chart.
global:
  images:
    envoyGateway:
      image: "private-hub/envoyproxy/gateway-fips:abcdef12"
      pullPolicy: Always
      imagePullSecrets:
      - name: "secret1"
      - name: "secret2"
    ratelimit:
      image: "private-hub/envoyproxy/ratelimit-fips:test"

Future work to support air-gapped envirionment:

Support pullPolicy and imagePullSecrets for ratelimit
Support  custom image setting for statsd-exporter
Support custom image setting for envoy

Those will be addressed in follow-up PRs.
part of: https://github.com/envoyproxy/gateway/issues/3049

@zhaohuabing what happens what the image is set in top level as well as deployment, which one wins ? I'd vote for most specific one (deployment) wins

@zhaohuabing what happens what the image is set in top level as well as deployment, which one wins ? I'd vote for most specific one (deployment) wins

Deployment level wins. Ideally we should remove the deployment level config. I keep it for back compatibility.


There is Limitation in features of control plane! I Suggest remove limits!

In new version of APK we have control plane but with limited features.
I request that in the next version please release a new version with full feature please. thank you.
I refer this problem: unsupported features

Hi @feyzollahi ,
Are you looking for any specific feature from the unsupported feature list?

Hi @feyzollahi ,
Are you looking for any specific feature from the unsupported feature list?

Hi @pubudu538
Yes. One of the important features is Soap API and also Soap to Rest API. I recommend and request to you dear leaders of WSO2 APIM to create a version that have complete control plane.


Consider adding executor_traits::when_all

then_execute can only take a single dependency, but we often need to write tasks that have many dependencies. There needs to be a simple way for executor users to introduce a join point that is also visible to executors.
One could in principle implement when_all by doing a bulk then_execute and having agents individually wait on futures. However, we should avoid solutions which require agents to work at the level of futures.

Should probably allow the user to choose the type of the returned future somehow.


Workspace Project Titles

Underscores are shown on the tree: projects with a title like 'test piece' show as 'test_piece'. we shouldn't see the underscore there.

I don't get this one? We now allow _ and I only see the _ in the tree when it is in the title?? Has this been fixed @torinfo @thexerteproject ?

I didn't change anything yet. I'll check again.

Do we need to? I dont see what the issue is... if you add underscores to the name then they show as they should... Is that not what we want? People have asked for non alphabetic chars there in the past.

The problem is that _ is replacing spaces in project names. It's fine for the underscore to be in there when it's supposed to be but not when it's meant to be a space.

Hmm, thats strange. I played around with this for a while yesterday and couldn't reproduce this behaviour but I was perhaps looking in the wrong place.
Just to get this straight, we are talking about the new workspace and changing the project name via the properties? Or does it only happen if you include spaces in the name when you create the project?

When creating a new project. I'm not sure what happens when you change it, I don't know if I've tried that

Ok, I'll take a look at this today too. I'm not 100% sure I created a new one with spaces becausevi got into the habit of not doing it... should be an easy win since it looks like everything still works if you change it afterwards...

Hi John I just pulled the latest code and tested this. Did you only test with two word titles by any chance? What seems to be happening now is a space is retained and not replaced by underscore between the first two words but not subsequent words. e.g. "my test" works but "my test project" becomes "my test_project" This happens when cerating a new project or when renaming an existing project.

Thanks Ron, yes looking back I only have test LOs with two words and forgot that javascript replace(for the tree) works differently from php replace (for the properties panel)...
Should be ok now...


[Backport to v0.6] Upgrade rabbitmq to v3.8.9

Task #1935. There are other tasks for backporting recent changes #2163 #2164 #2165 #2166

/azp run

/azp run

/azp run

/azp run

/azp run


Fix: Warning: config:terminal_title has been replaced by install_status and is ignored

Describe the bug
After the last update from spack mainline, we see this warning when we run spack concretize:
==> Warning: config:terminal_title has been replaced by install_status and is ignored

To Reproduce
See above
Expected behavior
Remove warning by identifying and changing legacy code in spack-stack (after the 1.6.0 release)
System:
any
Additional context
n/a

We must have fixed this when we pulled in spack develop again end of January:
heinzell@JCSDA-L-18146:~/work/spack-stack/spack-stack-remove-upp [brew-arch64]> grep -Rie terminal_title .
./spack/lib/spack/spack/schema/config.py:            "properties": ["terminal_title"],
./spack/lib/spack/spack/schema/config.py:            "message": "config:terminal_title has been replaced by "


_.get definition for ObjectWrappers

Fixing for ImplicitObjectWrapper
Adding for ExplicitObjectWrapper

How is the ExplicitObjectWrapper being use right now?

@unional The ExplicitObjectWrapper didn't have a definition for get. Therefore, the following code erroneously reported an error:
const value = _.chain(obj).get('prop').value();
// error TS2339: Property 'get' does not exist on type 'LoDashExplicitObjectWrapper<any>'.

The definitelytyped definition also reported an error, albeit a different one:
error TS2339: Property 'value' does not exist on type '{}'.

I mean I see that you add the interface but I don't see it is being used (from your change).

@unional I didn't add a new interface, it already existed, but didn't have a definition for .get yet. See: https://github.com/types/npm-lodash/blob/master/index.d.ts#L5976
However, I have been doing some more work and realized that my definition is actually wrong for the ImplicitWrapper, so I'll cancel this pull request.


Debug issue

Getting the following error while trying to run the C# code of powerforensics
A project with an output type of class library cannot be started directly
In order to debug this project,add an executable solution to this project which references the library project.
Need a solution to fix this issue

If you are trying to debug PowerForensics you want to follow the instructions in this article http://www.powershellmagazine.com/2014/04/08/basics-of-writing-a-powershell-module-with-c-part-2-debugging/.

On Feb 16, 2016, at 12:11 AM, suminvr<EMAIL_ADDRESS>wrote:
—
Reply to this email directly or view it on GitHub.

Thanks.That fixed the issue


Draft: feat(controller): implement instance logic

Overral explanation of our work :
Implement routes for Instance:

[x] GET /instance//<instance_id> : Return one instance by id
[x] DELETE /instance//<instance_id> : Delete one instance by id
[x] GET /instance/ : Return all instances with filter
[x] POST /instance/ : Instanciate an instance from a workload_id

Architecture
The code is divided into three parts :

Service : It implements all the logic to interact with etcd ( get , put ...) and grpc (create_instance)
Controller : Contains the route and functions they call
Model : It implements all models used by the two other parts

Tests
See this issue for the implementation of the tests
Used libraries
serde_json : serialise and deserialise Instance to JSON
grpc : send request to the scheduler

Commit sha: eb65253, Author: Maxime, Committer: Alexandre Gomez; Expected "Maxime<EMAIL_ADDRESS>but got "GridexX<EMAIL_ADDRESS>


Add Gradle Dependency Submission action  to GitHub Workflow

gradle-dependency-submission

Gradle Dependency Submission action failing to submit result, looks too time consuming to deal with


Editor integration for new blending

Added GUI controls for enabling / disabling the new terrain blending and modifying the blendDepth as described in #712 .
Watch the new controls in action here

@lukaspj Hello! (:
Thanks again for this nice work.
I'm working on this, especially to profile it and evaluate the render cost.
I see a for loop in the shader that could be unrolled since the iteration number is known at compile time. Other improvement could be done.
I've also already ported it to OpenGL.
I'll give updates progressively and make probably a new PR, for the final one.
Thank you!


Fix release 0.7

Fixes: #679

This should not break PR testing (which will see here). It should fix also the release-0.7 branch, but I have no way of testing that other then merging this PR, merging into release-0.7 and then re-running the release job.


Better organization of the source code

In fact there are two problems.

How to manage the dependency between different modules after splitting the code.
How to package these code (with or without compression) and distribute it in a single file.

We have several options. If we prefer JavaScript stuff, we can use browserify or webpack. Or we can simply use sprockets. The pro of the first choice is that these tools are more powerful and can really understand dependency, with the cost of additional code at runtime.

How to manage the dependency between different modules after splitting the code.

I don't think there's any point in distributing separate modules so whatever's the best/most convenient tool for compiling to a single file. I definitely think we should be writing it in ES6 though.

How to package these code (with or without compression) and distribute it in a single file.

We should be as flexible as possible without imposing too much of a maintenance burden. So at a minimum we should publish an npm module and source gem like we do for Turbolinks 5 - bower is debatable but if it's easy to automate that as part of the publishing process then we should probably do it.

I definitely think we should be writing it in ES6 though.

Does that mean that we should compile code using Babel? Some polyfills may add unnecessary code to generated file.
Browserify or Webpack will also add extra code. I'll do an experiment to check how much overload they will add.

Webpack will add an extra 218 bytes while browserify will add 498 bytes. So choose webpack. :smile:


French translations

You might want to enjoy those :fr: translations.

That's great!! just merged, thank you very much.
Best regards.

 Thank you for sharing your work.


Implement missing entities

Define Entities for Complete Ledger System
To ensure a complete and modular ledger system, we need to extend the existing structure by transitioning some of the data currently stored in settings into dedicated objects. These objects will have their own set of accessors and mutators. The new objects to be introduced are:

Assets (formerly precision):

Revaluation (formerly fx_adjustment):

Price History

Details

Each object type will have its own accessors and mutators.
The system should treat these objects consistently, ensuring that their data is properly read, modified, and stored according to the same principles as existing ledger entities.

Revaluations

Overview: Revaluations are understood as rules that specify when to adjust a particular account and to which account adjustments should be booked. These are handled in the form of a DataFrame.

Questions:

There are differences between the column schemas described in the canvas and constants.py. Specifically, the presence of account, credit, and debit columns together in the schema creates some confusion. Which schema should we follow?
Should we define an ID column, or use an account-date pair as the unique identifier?

To update the scope, the current schema is used in the validate_accounts() method here and in the complete_ledger() method here.
 REVALUATION_SCHEMA_CSV_CANVAS = """
 column,             dtype,                mandatory
 date,               datetime64[ns],       True
 account,            string[python],       True
 account_to_book,    Int64,                True
 description,        string[python],       True
 """

 REVALUATION_SCHEMA_CSV_CURRENT = """
 column,             dtype,                mandatory
 date,               datetime64[ns],       True
 account,            string[python],       True
 credit,             Int64,                True
 debit,              Int64,                True
 description,        string[python],       True
 """

Assets

Current State: Currently, precision values for currencies are stored in settings, for example:

   CURRENCY_PRECISION = {
       "AUD": 0.01,
       "CAD": 0.01,
       "CHF": 0.01,
       "EUR": 0.01,
       "GBP": 0.01,
       "JPY": 1.00,
       "NZD": 0.01,
       "NOK": 0.01,
       "SEK": 0.01,
       "USD": 0.01,
   }

This data is currently only used here.

Proposed Change: The precision settings should be converted into a DataFrame with the following schema:
ASSETS_SCHEMA_CSV = """
column,             dtype,                mandatory
ticker,             string[python],       True
increment,          Float64,              True
date,               datetime64[ns],       False
"""

Identifiers: Ticker-date pairs should be used as unique identifiers for assets.

Accessor/Mutator Implementation: The implementation of accessor and mutator methods for assets is expected to be straightforward. Below is a sketch of the methods:

    @abstractmethod
    def assets(self) -> pd.DataFrame:
        """Retrieves all asset entries.

        Returns:
            pd.DataFrame: DataFrame with columns `ticker` (str), `increment` (float),
                          and `date` (datetime64[ns], optional).
        """

    @abstractmethod
    def add_asset(
        self,
        ticker: str,
        increment: float,
        date: pd.Timestamp
    ) -> None:
        """Adds a new asset entry.

        The unique identifier is a combination of `ticker` and `date`.

        Args:
            ticker (str): Identifier for the asset (e.g., stock ticker).
            increment (float): Increment value representing the asset unit.
            date (pd.Timestamp): Date of the asset entry.
        """

    @abstractmethod
    def modify_asset(
        self,
        ticker: str,
        increment: float,
        date: pd.Timestamp
    ) -> None:
        """Updates an existing asset entry.

        The unique identifier is a combination of `ticker` and `date`.

        Args:
            ticker (str): Identifier for the asset.
            increment (float): Increment value for the asset.
            date (pd.Timestamp): Date of the asset entry.
        """

    @abstractmethod
    def delete_assets(
        self,
        tickers: List[str],
        dates: List[pd.Timestamp],
        allow_missing: bool = False
    ) -> None:
        """Removes asset entries.

        The unique identifier is a combination of `ticker` and `date`.

        Args:
            tickers (List[str]): List of asset tickers to remove.
            dates (List[pd.Timestamp]): List of dates corresponding to the tickers to remove.
            allow_missing (bool, optional): If True, no error is raised if an asset entry is
                                            not found. Defaults to False.
        """

Questions:

From Architect Alex's perspective, it would be beneficial to have a method that can find a single asset by ticker and optionally by date. What do you think about adding this?

Price History
It is now well described starting from this line, and I believe it is aligned with our concept. No changes are needed here.
Question:

Currently, no modifier (mutator) is available for price history. Is this intentional, or should we consider adding one?

Reveluation updates:
 REVALUATION_SCHEMA_CSV_CURRENT = """
 column,             dtype,                mandatory
 date,               datetime64[ns],       True
 account,            string[python],       True
 credit,             Int64,                True
 debit,              Int64,                False
 description,        string[python],       True
price, Float64, False
 """

If debit is not specified use the same as credit.
Introducing new column "price" since some systems require that

Issue Title 1: Using Unique Combinations of Columns as Identifier for Entities
Problem Description:
In our system, we manage data entities where each entry is identified by a unique combination of columns. For the assets schema, the unique combination is the pair of ticker and date. For other schemas, such as prices, the unique combination includes additional columns like currency.
The assets schema is defined as follows:
ASSETS_SCHEMA_CSV = """
    column,             dtype,                mandatory
    ticker,             string,               True
    increment,          float64,              True
    date,               datetime.date,        False
"""

Possible Solutions for Uniqueness Check:

Using pandas' concat() and unique():

Concatenates columns to form a unique combination and checks for uniqueness using pandas unique().
Example:combined_current = pd.concat([df['ticker'], df['date'].astype(str)]).unique()

Using numpy.unique():

Uses numpy.unique() to efficiently find unique combinations, but requires conversion of mixed data types (like string and datetime) to homogeneous types (e.g., strings).
Example:unique_values = np.unique(df[['ticker', 'date']].astype(str).values, axis=0)

Using Python set() and pandas merge():

This method uses Python's set() for efficient uniqueness checks and pandas merge() to update rows based on differences.
This approach is flexible for mixed data types and provides clarity for adding, updating, and deleting records.

Chosen Solution:
We chose the Python set() + pandas merge() approach for both the Assets and Prices schemas due to the following reasons:

Efficiency: set() provides fast operations for checking uniqueness across multiple columns (ticker, date, and currency where necessary).
Clarity: merge() enables clean and maintainable logic for updating or adding rows.
Scalability: This method handles large datasets effectively and can be extended to multiple schemas.
Flexibility: This solution works for both schemas with minimal adjustments, making it easier to maintain consistent code.

Issue Title 2: Proposal for Mutator Methods (Add, Modify, Delete) for Assets
Problem Description:
We need to implement mutator methods (add, modify, delete) for the assets schema. Each asset entry is identified by a unique combination of ticker and date. The date field is of type datetime.date and can be optional (None). The methods should handle these combinations properly, ensuring efficient and reliable asset operations.
The assets schema is defined as follows:
ASSETS_SCHEMA_CSV = """
    column,             dtype,                mandatory
    ticker,             string,               True
    increment,          float64,              True
    date,               datetime.date,        False
"""

Method Proposals:

Add Method (add_asset):

Ensures the unique combination (ticker, date) does not already exist before adding a new asset.
Example:def add_asset(self, ticker: str, increment: float, date: datetime.date = None):
    current_state = self.assets()
    new_entry = set([(ticker, date)])
    current_entries = set(zip(current_state["ticker"], current_state["date"]))

    if new_entry.issubset(current_entries):
        raise ValueError("The asset with this unique combination already exists.")

    new_row = pd.DataFrame({
        "ticker": [ticker],
        "date": [date],
        "increment": [increment]
    })

    self.assets = pd.concat([current_state, new_row], ignore_index=True)
    return new_row

Modify Method (modify_asset):

Modifies the increment value for an existing asset, ensuring the unique combination (ticker, date) exists before modification.
Example:def modify_asset(self, ticker: str, increment: float, date: datetime.date = None):
    current_state = self.assets()
    entry_to_modify = set([(ticker, date)])
    current_entries = set(zip(current_state["ticker"], current_state["date"]))

    if not entry_to_modify.issubset(current_entries):
        raise ValueError("The asset with this unique combination does not exist.")

    current_state.loc[
        (current_state["ticker"] == ticker) &
        (current_state["date"] == date),
        ["increment"]
    ] = [increment]

    self.assets = current_state
    return current_state

Delete Method (delete_asset):

Deletes an asset based on the unique combination (ticker, date).
Example:def delete_asset(self, ticker: str, date: datetime.date = None):
    current_state = self.assets()
    entry_to_delete = set([(ticker, date)])
    current_entries = set(zip(current_state["ticker"], current_state["date"]))

    if not entry_to_delete.issubset(current_entries):
        raise ValueError("No asset found with this unique combination.")

    current_state = current_state[~(
        (current_state["ticker"] == ticker) &
        (current_state["date"] == date)
    )]

    self.assets = current_state
    return entry_to_delete

Why This Solution:

Efficiency: Set-based logic simplifies the process of checking for unique combinations.
Clarity: The code is streamlined and easy to maintain, with consistent logic across all methods.
Handling of Optional Date: The date field is handled properly, even when it's None.

Here is the example for Price entity:
Here's the set of methods, based on the unique combination of ticker, currency, and date in the Prices Schema:
Prices Schema Definition:
PRICE_SCHEMA_CSV = """
    column,             dtype,                mandatory
    ticker,             string,               True
    date,               datetime.date,        True
    currency,           string,               True
    price,              float64,              True
"""

Method Implementations:

Add Method (add_price):

def add_price(self, ticker: str, date: datetime.date, currency: str, price: float):
    """Adds a new price entry, ensuring the unique combination (ticker, date, currency) does not already exist."""
    current_state = self.prices()

    # Create sets of current entries and the new entry to be added
    new_entry = set([(ticker, currency, date)])
    current_entries = set(zip(current_state["ticker"], current_state["currency"], current_state["date"]))

    # Check if the new price entry already exists
    if new_entry.issubset(current_entries):
        raise ValueError("The price entry with this unique combination already exists.")

    # Add the new price row
    new_row = pd.DataFrame({
        "ticker": [ticker],
        "date": [date],
        "currency": [currency],
        "price": [price]
    })

    self.prices = pd.concat([current_state, new_row], ignore_index=True)
    return new_row

Modify Method (modify_price):

def modify_price(self, ticker: str, date: datetime.date, currency: str, price: float):
    """Modifies an existing price entry, ensuring the unique combination (ticker, date, currency) exists."""
    current_state = self.prices()

    # Create sets for current entries and the entry to be modified
    entry_to_modify = set([(ticker, currency, date)])
    current_entries = set(zip(current_state["ticker"], current_state["currency"], current_state["date"]))

    # Check if the price entry to modify exists
    if not entry_to_modify.issubset(current_entries):
        raise ValueError("The price entry with this unique combination does not exist.")

    # Modify the price row
    current_state.loc[
        (current_state["ticker"] == ticker) &
        (current_state["currency"] == currency) &
        (current_state["date"] == date),
        ["price"]
    ] = [price]

    self.prices = current_state
    return current_state

Delete Method (delete_price):

def delete_price(self, ticker: str, date: datetime.date, currency: str):
    """Deletes a price entry based on the unique combination (ticker, date, currency)."""
    current_state = self.prices()

    # Create sets for current entries and the price entry to be deleted
    entry_to_delete = set([(ticker, currency, date)])
    current_entries = set(zip(current_state["ticker"], current_state["currency"], current_state["date"]))

    # Check if the price entry to delete exists
    if not entry_to_delete.issubset(current_entries):
        raise ValueError("No price entry found with this unique combination.")

    # Delete the price row
    current_state = current_state[~(
        (current_state["ticker"] == ticker) &
        (current_state["currency"] == currency) &
        (current_state["date"] == date)
    )]

    self.prices = current_state
    return entry_to_delete

Mirror Method (mirror_prices):

Aligns the current prices with the target state by adding, modifying, or deleting entries as necessary.

def mirror_prices(self, target: pd.DataFrame, delete: bool = False):
    """Aligns the prices with a desired target state using set() for uniqueness and merge for updating."""
    if target is not None:
        target = target.copy()
    target_df = self.standardize_prices(target)
    current_state = self.prices()

    # Delete prices not in the target if the delete flag is set
    if delete:
        to_delete = set(zip(current_state["ticker"], current_state["currency"], current_state["date"])).difference(
            set(zip(target_df["ticker"], target_df["currency"], target_df["date"]))
        )
        self.delete_prices(to_delete)

    # Add new prices
    new_prices = set(zip(target_df["ticker"], target_df["currency"], target_df["date"])).difference(
        set(zip(current_state["ticker"], current_state["currency"], current_state["date"]))
    )
    to_add = target_df.loc[target_df.apply(
        lambda row: (row["ticker"], row["currency"], row["date"]) in new_prices, axis=1
    )]
    for row in to_add.to_dict("records"):
        self.add_price(
            ticker=row["ticker"],
            currency=row["currency"],
            date=row["date"],
            price=row["price"],
        )

    # Update modified prices
    both = set(zip(target_df["ticker"], target_df["currency"], target_df["date"])).intersection(
        set(zip(current_state["ticker"], current_state["currency"], current_state["date"]))
    )
    left = target_df.loc[target_df.apply(
        lambda row: (row["ticker"], row["currency"], row["date"]) in both, axis=1
    )]
    right = current_state.loc[current_state.apply(
        lambda row: (row["ticker"], row["currency"], row["date"]) in both, axis=1
    )]

    # Merge left and right to identify which rows need updating
    merged = pd.merge(left, right, on=["ticker", "currency", "date"], how="outer", indicator=True)
    to_update = merged[merged["_merge"] == "left_only"]

    # Modify prices that have changed
    for row in to_update.to_dict("records"):
        self.modify_price(
            ticker=row["ticker"],
            currency=row["currency"],
            date=row["date"],
            price=row["price"],
        )

    # Return statistics
    stats = {
        "pre-existing": len(current_state),
        "targeted": len(target_df),
        "added": len(to_add),
        "deleted": len(to_delete) if delete else 0,
        "updated": len(to_update),
    }
    return stats

Key Points:

Mirror Method: Aligns the current prices with a target DataFrame by adding new entries, updating existing ones, and optionally deleting entries that are not in the target.
Set-Based Logic: Ensures efficient uniqueness checks based on the combination of ticker, currency, and date.
Consistency: All methods follow a similar pattern for handling price entries.

Issue Title: Optimizing Account Range Overlap Checks in Revaluation Methods
Problem Description:
In the current implementation of mutator and mirror methods for the Revaluation Schema, a common challenge is handling account ranges effectively. Accounts in this schema can be represented as ranges (e.g., 1000+1020:1025), which introduces complexity when checking for overlaps between existing entries and proposed changes.

Proposed Solutions:
1. Expand Accounts Without Iterating Rows:

Description: Expand the account ranges into individual accounts and compare those accounts for overlaps using vectorized operations, avoiding iterrows().
Benefits: Simplifies the comparison logic and avoids row-by-row iteration.
Drawbacks: May still require significant memory for large datasets but avoids the inefficiencies of iterrows().

Example:
def add_revaluation(self, date: pd.Timestamp, account: str, credit: int, debit: int = None, description: str = "", price: float = None):
    """Adds a new revaluation entry after expanding and checking account ranges for overlaps."""
    current_state = self.revaluations()

    # Expand the current accounts into individual accounts for comparison
    expanded_current = current_state["account"].apply(lambda acc: set(self.account_range(acc)["add"])).explode()
    expanded_target = set(self.account_range(account)["add"])

    # Check for overlaps in a vectorized way
    overlap_mask = (current_state["date"] == date) & expanded_current.isin(expanded_target)

    if overlap_mask.any():
        raise ValueError("Overlapping account ranges found.")

    # Add the new revaluation entry
    new_row = pd.DataFrame({
        "date": [date],
        "account": [account],
        "credit": [credit],
        "debit": [debit],
        "description": [description],
        "price": [price]
    })

    self.revaluations = pd.concat([current_state, new_row], ignore_index=True)

2. Range Intersection Using Vectorized Logic:

Description: Use vectorized operations to check for intersections between account ranges directly in the add_revaluation method, avoiding the need for row-by-row processing.
Benefits: Efficient range comparison while avoiding the inefficiencies of iterrows().
Drawbacks: Maintaining custom logic for range comparison.

Example:
def add_revaluation(self, date: pd.Timestamp, account: str, credit: int, debit: int = None, description: str = "", price: float = None):
    """Adds a new revaluation entry by checking for range intersections in a vectorized way."""
    current_state = self.revaluations()

    # Vectorized range intersection check
    proposed_accounts = set(self.account_range(account)["add"])
    expanded_current = current_state["account"].apply(lambda acc: set(self.account_range(acc)["add"]))
    overlap_mask = (current_state["date"] == date) & expanded_current.apply(lambda x: not proposed_accounts.isdisjoint(x))

    if overlap_mask.any():
        raise ValueError("Overlapping account ranges found.")

    # Add the new revaluation entry
    new_row = pd.DataFrame({
        "date": [date],
        "account": [account],
        "credit": [credit],
        "debit": [debit],
        "description": [description],
        "price": [price]
    })

    self.revaluations = pd.concat([current_state, new_row], ignore_index=True)

3. Treat Account Ranges as Regular Columns:

Description: Treat the account range as a regular column and perform direct comparisons between ranges as strings. Use set-based logic similar to how uniqueness is handled in other contexts.
Benefits: Simplifies implementation by avoiding any special handling for account ranges.
Drawbacks: May lead to inaccuracies if ranges are not treated properly as individual accounts.

Example:
def add_revaluation(self, date: pd.Timestamp, account: str, credit: int, debit: int = None, description: str = "", price: float = None):
    """Adds a new revaluation entry using set-based logic for account ranges."""
    current_state = self.revaluations()
    new_entry = set([(date, account)])
    current_entries = set(zip(current_state["date"], current_state["account"]))

    if new_entry.issubset(current_entries):
        raise ValueError("The account range already exists for this date.")

    new_row = pd.DataFrame({
        "date": [date],
        "account": [account],
        "credit": [credit],
        "debit": [debit],
        "description": [description],
        "price": [price]
    })

    self.revaluations = pd.concat([current_state, new_row], ignore_index=True)

Sample of Methods for Better Visibility:
Here’s a sample of the methods that require optimization in handling account ranges for better visibility:
Add Method (add_revaluation):
def add_revaluation(self, date: pd.Timestamp, account: str, credit: int, debit: int = None, description: str = "", price: float = None):
    """Adds a new revaluation entry, ensuring no overlap between current and proposed account ranges."""
    current_state = self.revaluations()

    # Get account range for the proposed entry
    proposed_accounts = set(self.account_range(account)["add"])

    # Expand all current accounts into ranges and check for overlaps
    expanded_current = current_state["account"].apply(lambda acc: set(self.account_range(acc)["add"]))
    overlap_mask = current_state["date"] == date
    overlap_found = expanded_current.apply(lambda x: not proposed_accounts.isdisjoint(x)) & overlap_mask

    if overlap_found.any():
        raise ValueError("The proposed account range overlaps with an existing entry.")

    # Add the new revaluation entry
    new_row = pd.DataFrame({
        "date": [date],
        "account": [account],
        "credit": [credit],
        "debit": [debit],
        "description": [description],
        "price": [price]
    })

    self.revaluations = pd.concat([current_state, new_row], ignore_index=True)
    return new_row

Modify Method (modify_revaluation):
def modify_revaluation(self, date: pd.Timestamp, account: str, credit: int, debit: int = None, description: str = "", price: float = None):
    """Modifies an existing revaluation entry, ensuring no overlap between the modified and existing account ranges."""
    current_state = self.revaluations()

    # Get account range for the proposed modification
    proposed_accounts = set(self.account_range(account)["add"])

    # Expand all current accounts into ranges and check for overlaps, excluding the current entry
    expanded_current = current_state["account"].apply(lambda acc: set(self.account_range(acc)["add"]))
    overlap_mask = (current_state["date"] == date) & (current_state["account"] != account)
    overlap_found = expanded_current.apply(lambda x: not proposed_accounts.isdisjoint(x)) & overlap_mask

    if overlap_found.any():
        raise ValueError("The modified account range overlaps with an existing entry.")

    # Modify the entry
    current_state.loc[
        (current_state["date"] == date) & (current_state["account"] == account),
        ["credit", "debit", "description", "price"]
    ] = [credit, debit, description, price]

    self.revaluations = current_state
    return current_state

Delete Method (delete_revaluation):
def delete_revaluation(self, date: pd.Timestamp, account: str):
    """Deletes a revaluation entry based on the unique combination (date, account)."""
    current_state = self.revaluations()

    # Delete the row
    current_state = current_state[~(
        (current_state["date"] == date) &
        (current_state["account"] == account)
    )]

    self.revaluations = current_state
    return account

Mirror Method (mirror_revaluations):
def mirror_revaluations(self, target: pd.DataFrame, delete: bool = False):
    """Aligns the revaluations with a desired target state, ensuring no overlap in account ranges."""
    if target is not None:
        target = target.copy()
    target_df = self.standardize_revaluations(target)
    current_state = self.revaluations()

    # Delete entries not in the target if the delete flag is set
    if delete:
        to_delete = set(zip(current_state["date"], current_state["account"])).difference(
            set(zip(target_df["date"], target_df["account"]))
        )
        for date, account in to_delete:
            self.delete_revaluation(date, account)

    # Efficiently check for overlaps and add new revaluations
    for _, row in target_df.iterrows():
        proposed_accounts = set(self.account_range(row["account"])["add"])

        # Vectorized check for overlaps
        expanded_current = current_state["account"].apply(lambda acc: set(self.account_range(acc)["add"]))
        overlap_mask = current_state["date"] == row["date"]
        overlap_found = expanded_current.apply(lambda x: not proposed_accounts.isdisjoint(x)) & overlap_mask

        if not overlap_found.any():
            self.add_revaluation(
                date=row["date"],
                account=row["account"],
                credit=row["credit"],
                debit

=row["debit"],
                description=row["description"],
                price=row["price"]
            )

    return {
        "pre-existing": len(current_state),
        "targeted": len(target_df),
        "added": len(target_df) - len(current_state),
        "deleted": len(to_delete) if delete else 0,
    }

Prices Implementation Observation
In reviewing the prices implementation, I noticed we have two methods for standardizing data:

standardize_price_df(): returns a single DataFrame.
standardize_prices(): returns a dictionary of DataFrames.

In both the StandaloneLedger and MemoryLedger, the _prices variable currently stores a dictionary of DataFrames, as it’s standardized via standardize_prices(). Could you confirm that storing this as a dictionary, rather than a single DataFrame, is the intended design?
For assets, I can either implement a similar method, which would streamline the precision() logic, or I could hardcode the necessary adjustments directly within precision() for now.

precision() Method Logic
When date is Specified:

Latest Date on or Before Specified Date: Retrieve the latest available row where ticker matches and date is on or before the specified date. If found, return the increment value.
Ticker with Only NaT Dates: If the ticker exists but all date entries are NaT, return the single available increment associated with NaT, regardless of the specified date.
No Prior Observation Available: If no rows are available with dates on or before the specified date, and there’s no default NaT row, raise an error.

When date is None:

Set date to Today’s Date: Default to today’s date if no date is provided.
Follow the Same Logic as Above.

Example Scenarios
Sample DataFrame (assets_df)

ticker
date
increment

USD
2023-01-01
0.01

USD
2023-06-01
0.02

EUR
2023-05-01
0.005

EUR
NaT
0.01

JPY
NaT
1.00

Test Cases

Specified Date with Latest Date on or Before Specified Date

Input: ticker='USD', date='2023-04-01'
Output: 0.01

Specified Date with Exact Match

Input: ticker='USD', date='2023-06-01'
Output: 0.02

Ticker with Only NaT Dates

Input: ticker='JPY', date='2023-06-01'
Output: 1.00

No Prior Observation Available

Input: ticker='EUR', date='2023-01-01'
Output: Raises ValueError

Date is None (Defaults to Today)

Input: ticker='USD', date=None (assuming today is 2023-06-01)
Output: 0.02

Ticker with NaT Date Only and Date is None

Input: ticker='EUR', date=None (assuming today is 2023-10-01)
Output: 0.01

Implementation Steps

Filter Rows by Ticker: Start by filtering the DataFrame to include only rows where ticker matches the specified ticker.

Sort by Date: Sort the filtered DataFrame by date in ascending order. This way, any NaT values will be sorted to the beginning, making them the "oldest" entries.

Filter Rows by Date Condition:

If a date is specified, keep only the rows where date is less than or equal to the specified date.
If date is None, set it to today's date and apply the same filter.

Select the Newest Record:

Take the last row from the filtered DataFrame after sorting. This row will have the latest date on or before the specified date, ensuring that:

If all date values for the ticker are NaT, the first entry (NaT entry) will be selected.
If no entries meet the date criteria, raise an error since no increment value is available for this ticker.

Challenges

Varying Schemas: Each entity has its own set of columns needing validation.
Different Storage Mechanisms: Entities are saved differently (e.g., in-memory variables, CSV files, YAML).
Standardization Needs: Each entity must ensure data conforms to its schema via a standardize method.
Avoiding Code Duplication: Need a generic solution that minimizes repetitive code across entities.
Unique Identifiers: Ability to specify columns that make up the unique identifier for each row.

Possible option
Implement a flexible architecture using design patterns and composition, focusing on:

Abstract Base Class (BaseEntity): Contains generic methods (add, modify, delete) that handle unique identifiers.

Storage Strategy Pattern: Defines a StorageStrategy interface with concrete implementations (e.g., CSVStorage, InMemoryStorage).

Composition: Each entity class uses a StorageStrategy for persistence, separating storage logic from entity logic.

Standardization Logic: Each entity implements a standardize method tailored to its schema.

Unique Identifiers Handling: Entities specify the columns that constitute the unique identifier, used in generic methods.

Example of Implementation Steps
1. Define the StorageStrategy Interface
from abc import ABC, abstractmethod
import pandas as pd

class StorageStrategy(ABC):
    @abstractmethod
    def save(self, df: pd.DataFrame):
        pass

    @abstractmethod
    def load(self) -> pd.DataFrame:
        pass

2. Implement Concrete Storage Strategies
Example: CSV Storage
class CSVStorage(StorageStrategy):
    def __init__(self, file_path: str):
        self.file_path = file_path

    def save(self, df: pd.DataFrame):
        df.to_csv(self.file_path, index=False)

    def load(self) -> pd.DataFrame:
        try:
            return pd.read_csv(self.file_path)
        except FileNotFoundError:
            return pd.DataFrame()

Example: In-Memory Storage
class InMemoryStorage(StorageStrategy):
    def __init__(self):
        self._data = pd.DataFrame()

    def save(self, df: pd.DataFrame):
        self._data = df.copy()

    def load(self) -> pd.DataFrame:
        return self._data.copy()

3. Define the BaseEntity Abstract Class
from abc import ABC, abstractmethod

class BaseEntity(ABC):
    def __init__(self, storage_strategy: StorageStrategy):
        self.storage = storage_strategy
        self._data = self.storage.load()

    @abstractmethod
    def standardize(self, df: pd.DataFrame) -> pd.DataFrame:
        pass

    @abstractmethod
    def unique_id_cols(self) -> list:
        pass

    def save(self):
        self.storage.save(self._data)

    def load(self):
        self._data = self.storage.load()

    def add(self, **kwargs):
        df_new = pd.DataFrame([kwargs])
        df_new = self.standardize(df_new)
        mask = self._get_mask(df_new.iloc[0])

        if mask.any():
            raise ValueError("The entity with this unique combination already exists.")

        self._data = pd.concat([self._data, df_new], ignore_index=True)
        self.save()

    def modify(self, identifier: dict, updates: dict):
        mask = self._get_mask(identifier)
        if not mask.any():
            raise ValueError(f"Entity with identifier {identifier} not found.")

        for key, value in updates.items():
            self._data.loc[mask, key] = value

        self._data = self.standardize(self._data)
        self.save()

    def delete(self, identifier: dict, allow_missing: bool = False):
        mask = self._get_mask(identifier)

        if mask.any():
            self._data = self._data[~mask].reset_index(drop=True)
            self.save()
        elif not allow_missing:
            raise ValueError(f"Entity with identifier {identifier} not found.")

    def _get_mask(self, identifier: dict) -> pd.Series:
        mask = pd.Series([True] * len(self._data))
        for col in self.unique_id_cols():
            value = identifier.get(col)
            if pd.isna(value):
                mask &= self._data[col].isna()
            else:
                mask &= self._data[col] == value
        return mask

4. Implement Specific Entities
Example: Asset Entity
class AssetEntity(BaseEntity):
    def __init__(self, storage_strategy: StorageStrategy):
        super().__init__(storage_strategy)
        self.schema = {
            'ticker': 'string',
            'date': 'datetime64[ns]',
            'increment': 'float64',
        }

    def standardize(self, df: pd.DataFrame) -> pd.DataFrame:
        for column, dtype in self.schema.items():
            if column not in df.columns:
                df[column] = pd.NA
            df[column] = df[column].astype(dtype)
        return df

    def unique_id_cols(self) -> list:
        return ['ticker', 'date']

    def assets(self) -> pd.DataFrame:
        return self._data.copy()

    # Matching original interface methods
    def add_asset(self, ticker: str, increment: float, date: pd.Timestamp = None):
        self.add(ticker=ticker, increment=increment, date=date)

    def modify_asset(self, ticker: str, increment: float, date: pd.Timestamp = None):
        identifier = {'ticker': ticker, 'date': date}
        updates = {'increment': increment}
        self.modify(identifier, updates)

    def delete_asset(self, ticker: str, date: pd.Timestamp = None, allow_missing: bool = False):
        identifier = {'ticker': ticker, 'date': date}
        self.delete(identifier, allow_missing)

Benefits of This Approach

Generic Handling of Unique Identifiers: Centralizes logic for matching records.
Code Reuse: Common methods are defined once in the base class.
Flexibility: Entities can have unique schemas and storage mechanisms.
Consistency: Entities maintain consistent interfaces.
Maintainability: Simplifies updates and additions.


updated some basic introduction of terminal

A basic introduction to what is terminal or command prompt was necessary.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Congrats @sudoshreyas on your first pull request (PR)! 🎉
Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 📝