Optimized FreeList allocator

Ticket
Link to Github Issue
Problem description
The current FreeList allocator is fairly slow. Epically under my GGML backend, it becomes unusable slow under high memory pressure. This can be mitigated by using an better optimized allocator.
Detailed benchmark can be found my GitHub repo: https://github.com/marty1885/tt-alloc-opt. In short, the new allocator is over an order of magnitude faster (~60x) for large amounts of allocations (DRAM) and at least 2x on smaller allocations (L1). I have tested the new allocator it works against my GGML backend on GS.
Due to networking issues, I don't have WH tests ready. I can't until after Thanksgiving. But still want to open the PR and start the discussion.
Parameters like table sizes and base values most likely needs to be tuned for real world workloads. Even without changes, it should be faster then the existing one by default.
2024-11-28T11:09:30+08:00
Running ./tt-alloc-opt
Run on (16 X 5132 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 16384 KiB (x1)
Load Average: 1.02, 0.90, 0.84
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------------------------------------
Benchmark                                           Time             CPU   Iterations
-------------------------------------------------------------------------------------
FreeListOpt/WorstCase                          156211 ns       155689 ns         4428
FreeListOpt/MixedAllocations                    59938 ns        59760 ns        11755
FreeListOpt/TypicalCase                         32595 ns        32492 ns        21033
FreeListOpt/Small                                3676 ns         3663 ns       179884
FreeListOpt/GetAvailableAddresses                 409 ns          407 ns      1726829
FreeListOpt/Statistics                            379 ns          378 ns      1848454
FreeListOpt/ShrinkReset                          11.7 ns         11.6 ns     60177702
FreeList[BestMatch]/WorstCase                 8430132 ns      8396434 ns           81
FreeList[BestMatch]/MixedAllocations          2219312 ns      2211079 ns          314
FreeList[BestMatch]/TypicalCase                863138 ns       860375 ns          802
FreeList[BestMatch]/Small                        7968 ns         7944 ns        85656
FreeList[BestMatch]/GetAvailableAddresses         893 ns          890 ns       779186
FreeList[BestMatch]/Statistics                   1107 ns         1103 ns       643461
FreeList[BestMatch]/ShrinkReset                  3.44 ns         3.43 ns    204268507
FreeList[FirstMatch]/WorstCase                7409143 ns      7380524 ns           95
FreeList[FirstMatch]/MixedAllocations         2283444 ns      2274426 ns          296
FreeList[FirstMatch]/TypicalCase               864990 ns       861908 ns          809
FreeList[FirstMatch]/Small                       7726 ns         7705 ns        91195
FreeList[FirstMatch]/GetAvailableAddresses        886 ns          882 ns       794174
FreeList[FirstMatch]/Statistics                  1092 ns         1088 ns       645254
FreeList[FirstMatch]/ShrinkReset                 3.47 ns         3.45 ns    202517500

The idea of a segregated list comes from the TLSF algorithm. However the intention is not to implement TLSF nor to make a real time allocator. To keep the behavior the same as the exiting FreeList implementation (L1 is small, reducing fragmentation is important) - 1) a full scan of blocks in the size class is done to determine the best place to allocate the new memory. 2) only one layer exists in this implementation's segregation list instead of two. As my synthetic benchmark indicating cache locality is more important then reduced number of blocks scanned. Though, it can be easily converted to a proper 2 layer design by changing how the index is calculated.
I have WIP optimizations that should provide an extra 5~10% throughput. But again, I wat to start the discussion early as a new allocator is a big change.
For license concerns. TT can have the file licensed under Apache 2.0. I licensed the version under 0BSD in my repo to make the process easy.
What's changed

Implemented a new FreeListOpt allocator

Same algorithm as the existing FreeList allocator under BEST mode

Produces the same results

Removes the use of linked list and shared_ptr. Everything is in SoA to maximize cache efficency
Table to store free blocks by size so large allocations won't need to look at smaller blocks
Hash table to convert addresses back into blocks. No more list walk
Same memory coalescing as the existing allocator
Metadata blocks are reused internally to minimize memory allocation

Tests for the new allocator.
dump_block new accepts ostream instead of ofstream

Checklist

[ ] Post commit CI passes
[ ] Blackhole Post commit (if applicable)
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[ ] New/Existing tests provide coverage for changes

Hey Marty, this is awesome! The unoptimized allocator was something on the back of my mind for a while. Thanks for taking this on ⭐

Running tests on CI here
Would it be possible to benchmark fragmentation @marty1885? I don't think we would be worse off since the current allocator is set to use first fit.

@abhullar-tt Let me debug that. It shouldn't though since it's in best mode
instead of first.

@abhullar-tt How can I see the exact error message and cause? Like know when the allocator reported OOM? I can't find indications on the CI. I will only have access to my QB during weekend, have to wait for that. Some networking issues where I host it.
There is a known pattern causing BestFit to have more fragmentation then FirstFit. If a fixed allocation is wedged in the middle-to-side of the allocatable space, the BestFit allocator may see one half of the remaining space as smaller and dump everything there. Since technically it is a better fit.

@abhullar-tt How can I see the exact error message and cause? Like know when the allocator reported OOM? I can't find indications on the CI. I will only have access to my QB during weekend, have to wait for that. Some networking issues where I host it.
There is a known pattern causing BestFit to have more fragmentation then FirstFit. If a fixed allocation is wedged in the middle-to-side of the allocatable space, the BestFit allocator may see one half of the remaining space as smaller and dump everything there. Since technically it is a better fit.

I would start with looking at the post commit tests, for example: https://github.com/tenstorrent/tt-metal/actions/runs/12075033647/job/33674419924
I would also ensure that L1 buffers are respecting the top down allocation scheme.

@abhullar-tt can you run the CI again? I have confirmed the UT is passing and LLMs are running on my machine. I am not 100% all tests passes as I lack N150 locally.

@abhullar-tt can you run the CI again? I have confirmed the UT is passing and LLMs are running on my machine. I am not 100% all tests passes as I lack N150 locally.

Starting running some tests
Do you think you could add some benchmarks on fragmentation differences between the two allocator algos?

Do you think you could add some benchmarks on fragmentation differences between the two allocator algos?

The earlier problem stems from a bug not discovered in testing (deallocate() not working when offset !=0). Otherwise the new allocator is equivalent to the existing FreeList with the BestFit policy. I made it this way to reduce the complication of figuring out if fragmentation increased under TT's use case (which the allocation pattern is not typical).
I can add some fragmentation benchmark if it is required. What do you imagine it be like?

Do you think you could add some benchmarks on fragmentation differences between the two allocator algos?

The earlier problem stems from a bug not discovered in testing (deallocate() not working when offset !=0). Otherwise the new allocator is equivalent to the existing FreeList with the BestFit policy. I made it this way to reduce the complication of figuring out if fragmentation increased under TT's use case (which the allocation pattern is not typical).
I can add some fragmentation benchmark if it is required. What do you imagine it be like?

Awesome, I saw that a bunch of CI tests are passing now.
Re fragementation benchmark: could be similar to the tests you added to demonstrate speedup, could do a series of randomized allocations/deallocations and then query allocator statistics or buffer addresses to get a sense of the fragmentation.

NP. I wrote a quick test comparing my new allocator against the existing one. To simulate random allocation and deallocation on WH's L1. Random allocations are performed and there's a chance of deallocation of previous allocations. Measuring how many iterations has to happen until the memory is full.
Results are as shown. My allocator is indded the same as the existing FreeList with BestFit policy (and consistently better then FirstFit used now). Actually I found a bug in the existing BestFit that is sometimes messes up and doesn't pick the best block. So rarely my allocator will perform slightly better is rare cases
Benchmarking fragmentation... (number of random allocation and deallocation attempts until full)
FreeListOpt: 337
FreeList (First): 321
FreeList (Best): 337

Source code can be found here: https://github.com/marty1885/tt-alloc-opt/blob/master/fragmentation.cpp

NP. I wrote a quick test comparing my new allocator against the existing one. To simulate random allocation and deallocation on WH's L1. Random allocations are performed and there's a chance of deallocation of previous allocations. Measuring how many iterations has to happen until the memory is full.
Results are as shown. My allocator is indded the same as the existing FreeList with BestFit policy (and consistently better then FirstFit used now). Actually I found a bug in the existing BestFit that is sometimes messes up and doesn't pick the best block. So rarely my allocator will perform slightly better is rare cases
Benchmarking fragmentation... (number of random allocation and deallocation attempts until full)
FreeListOpt: 337
FreeList (First): 321
FreeList (Best): 337

Source code can be found here: https://github.com/marty1885/tt-alloc-opt/blob/master/fragmentation.cpp

Awesome! I think we can upstream this PR.

Can you merge it? I don't have write access to the repository.

Can you merge it? I don't have write access to the repository.

Yep will do!

@abhullar-tt Ping on merging this! I see some exciting features merged recently into main and wants to take advantage of them. But also I need the faster allocator.

@abhullar-tt Ping on merging this! I see some exciting features merged recently into main and wants to take advantage of them. But also I need the faster allocator.

Yep, this is on top of my list! I held off earlier this week because we had some instability in our CI.

@marty1885 do you think you could take a look at the tests failing in: https://github.com/tenstorrent/tt-metal/actions/runs/12202495523
Earlier I thought it was machine instability but on multiple attempts it looks like these tests crash with the changes here.

@abhullar-tt Thanks for finding the problem. Should be fixed with the latest commit. Please run the CI again.

@abhullar-tt Thanks for finding the problem. Should be fixed with the latest commit. Please run the CI again.

Getting closer but still some failures: https://github.com/tenstorrent/tt-metal/actions/runs/12227847709.
I think you can try to run this locally by invoking the same tests as all-post-commit-workflows.yaml

Will do. Thanks for the tips. Really wish CI runs can be open to outside developers.

@abhullar-tt Need your input. I cannot replicate the failing tests locally. For llama inference, I see the the message FileNotFoundError: [Errno 2] No such file or directory: '/mnt/MLPerf/tt_dnn-models/llama/Llama3.2-1B-Instruct' in CI. To me it looks more like a CI environment problem. Tried to run the tests myself, but ended up with an error with no documents to resolve.
======================================================= short test summary info =======================================================
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-accuracy-128-1-page_params0-paged_attention-quick] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-accuracy-128-1-page_params0-paged_attention-full] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-performance-128-1-page_params0-paged_attention-quick] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-performance-128-1-page_params0-paged_attention-full] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
=================================================== 4 failed, 30 warnings in 12.45s ===================================================

For the failing unit test eager unit tests 5 wormhole_b0 N150. I am able to pass the test hanged on CI. Test tests/tt_eager/python_api_testing/unit_testing/misc/test_reshard.py::test_reshard_with_program_cache[tt_dtype0-input_shape0-input_layout0-input_shard_grid0-input_shard_shape0-input_shard_orientation0-input_sharding_scheme0-output_shard_grid0-output_shard_shape0-output_shard_orientation0-output_sharding_scheme0] failed on my QB. But the exact error occurred even if I switch back to using the current free list allocator with first fit policy. I don't think my allocator is at fault here.
                 Always | FATAL    | Cannot set to globally allocated buffer. Circular buffer size 65536 B exceeds allocated L1 buffer bank size of 57344 B

There is a chance that the problem is N150 specific. But I only have access to N300s.

@abhullar-tt Need your input. I cannot replicate the failing tests locally. For llama inference, I see the the message FileNotFoundError: [Errno 2] No such file or directory: '/mnt/MLPerf/tt_dnn-models/llama/Llama3.2-1B-Instruct' in CI. To me it looks more like a CI environment problem. Tried to run the tests myself, but ended up with an error with no documents to resolve.
======================================================= short test summary info =======================================================
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-accuracy-128-1-page_params0-paged_attention-quick] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-accuracy-128-1-page_params0-paged_attention-full] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-performance-128-1-page_params0-paged_attention-quick] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
FAILED models/demos/llama3/tests/test_llama_model.py::test_llama_model_inference[wormhole_b0-True-8-performance-128-1-page_params0-paged_attention-full] - AttributeError: 'TtModelArgs' object has no attribute 'DEFAULT_CKPT_DIR'
=================================================== 4 failed, 30 warnings in 12.45s ===================================================

For the failing unit test eager unit tests 5 wormhole_b0 N150. I am able to pass the test hanged on CI. Test tests/tt_eager/python_api_testing/unit_testing/misc/test_reshard.py::test_reshard_with_program_cache[tt_dtype0-input_shape0-input_layout0-input_shard_grid0-input_shard_shape0-input_shard_orientation0-input_sharding_scheme0-output_shard_grid0-output_shard_shape0-output_shard_orientation0-output_sharding_scheme0] failed on my QB. But the exact error occurred even if I switch back to using the current free list allocator with first fit policy. I don't think my allocator is at fault here.
                 Always | FATAL    | Cannot set to globally allocated buffer. Circular buffer size 65536 B exceeds allocated L1 buffer bank size of 57344 B

There is a chance that the problem is N150 specific. But I only have access to N300s.

Take a look at https://github.com/tenstorrent/tt-metal/blob/main/tests/scripts/run_python_model_tests.sh#L56 it looks like there is a mode to run these llama model tests with dummy weights.
Re. the N150 failing, I will take a look at whether its flaky on main as well.

I added support for the First Fit policy. I have to say, this "solution" is not ideal. Ideally, what needs to happen is top-down and bottom up allocations should be managed by different alloactors. Which grows/shrinks as needed. This reduces both fragmentation with in the allocator itself, and simplifies management.

@abhullar-tt Can you run the CI again? The new code should work now. Would love to have the faster allocator in so I don't need to maintain a special branch with all my patches in order to develop my GGML stuff.

@abhullar-tt Can you run the CI again? The new code should work now. Would love to have the faster allocator in so I don't need to maintain a special branch with all my patches in order to develop my GGML stuff.

Ran it earlier and it passed: https://github.com/tenstorrent/tt-metal/actions/runs/12380834839
waiting on results from wider set of tests but so far looks good to merge tomorrow morning!

Passing post commit
Passing model perf


[Question] - How do you handle forms?

Hi, you can totally delete this if you think it is inappropriate but I was wondering how you handle your forms with react-static as it is static content - eg. no server side script to handle form inputs. I'm currently finished building my static website but don't have a clue on how to actually handle form posts.
Are you just sending out fetch or axios requests when somebody submits a form? Or am I missing something obvious here?

Yep, as with any other form submission in an SPA, you have to have a backend or a service to send it to. Whether this is your own api, a paid service, etc is all up to you :)

@nealoke Here are some references

https://www.netlify.com/blog/2017/07/20/how-to-integrate-netlifys-form-handling-in-a-react-app/
https://docs.google.com/forms
https://zapier.com/learn/forms-surveys/best-online-form-builder-software/

Also, if you use a CRM tool like Hubspot, they'll typically have a form component as well.


Subscription upgrade, downgrade and management features

Are there any plans to add RevenueCat's subscription upgrade and downgrade controls with the upcoming SDK 6+ update?

Hi @bartan34! Yes, that's included in version 6.0.0-beta.1. This update points to the latest changes in the iOS and Android SDKs. There are some migration docs here: https://github.com/RevenueCat/purchases-capacitor/blob/main/migrations/v6-MIGRATION.md. We are working in some public documentation right now which should be ready soon.
Please try it out and let us know if you have any issues or feedback!
Going to close this issue for now.


wide toc in v3 isn't wide

The wide table of contents has css which forces it into two columns (i.e. not wide)

.page .toc.wide {
  column-count: 2;
  column-fill: auto;
  column-gap: .9cm;
  column-width: 7.68cm;
  -webkit-column-count: 2;
  -moz-column-count: 2;
  -webkit-column-width: 7.68cm;
  -moz-column-width: 7.68cm;
  -webkit-column-gap: .9cm;
  -moz-column-gap: .9cm;
}

(The H1 above the toc has it's own rule that busts out of those two columns)
In the legacy render, the toc outline is set to span the full page width.

Wanting a two column toc with a full width heading above is a different requirement (with its own complications).

This is what a wide toc could/should look like:

This is a misunderstanding of what wide is.
wide does not necessarily mean its contents have to be single-column. It just means a block that spans two columns of the main .page.  i.e., a normal element will fit into a single .page column. A wide element will span the full page width, but can internally have 1, 2, 3, or more columns. In case that is the issue.

OK. Closing this specific issue.


feat: CoreStringExtensions

would be nice to also have tests

I think I'll add those with/after live list (in live list PR I will add the gh action for running tests as well)


Adding sophtron updates from old ucw

I'm pulling these changes into my sophtron provider tests branch and closing this


Show the image shooted on the screen

[x] Create it ImageView
[x] Show the image shooted inside it

But it still need to refactor the imageview resizing


Connect-ExchangeOnline doesn't work with PWSH Core

Hi,
When Connect-ExchangeOnline is executed within PowerShell Core 7.1.4, it throws this error:
New-ExoPSSession: C:\Users\User\Documents\PowerShell\Modules\ExchangeOnlineManagement\2.0.3\ExchangeOnlineManagement.psm1:426:30
Line |
 426 |  … PSSession = New-ExoPSSession -ExchangeEnvironmentName $ExchangeEnviro …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Could not load type 'System.Security.Cryptography.SHA256Cng' from assembly 'System.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=

Any timeline as to when this will be fixed?
Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 39fbe5a2-c138-b566-c094-82ffc3d1b734
Version Independent ID: 4d69ee5d-5a6d-a203-69fe-bf5441a8b5eb
Content: Connect to Exchange Online PowerShell
Content Source: exchange/docs-conceptual/connect-to-exchange-online-powershell.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@haneef95, the current version of the module is 2.0.5. In an elevated Windows PowerShell window (run as administrator), run the command:
Update-Module ExchangeOnlineManagement

Perhaps the error will go away?


update cdn links

jsdelivr.net has updated their url scheme.  The old style only has graphiql through version 0.10.x, but the new style supports through the latest (0.11.10 as of today).

I need this 🙏
Having history pane in graphiql with newer versions is great.

From a security perspective I think it would be good to include Subresource Integrity/SRI in the tags. An example of the risk that SRI helps prevent is JS crypto-miners among other attacks where the CDN and/or publisher has been compromised;
https://www.wordfence.com/blog/2018/02/cryptomining-javascript-supply-chain-attack/
As a result I think it's probably easier to maintain through copy-pasting the whole tag rather than templating the version. cdnjs as an example provides a "link with SRI" copy facility for most assets.


Shrink empty lines

Visual Studio has a nice extension called "Shrink Empty Lines" which enables the user to have empty lines and/or lines without alphanumeric characters shrunk in the editor to a specific extent. This way users can have more vertical space for their actual code.
https://marketplace.visualstudio.com/items?itemName=VisualStudioPlatformTeam.SyntacticLineCompression

It would be very nice to have this feature on VS Code, maybe with an extra setting to specify shrinking percentage (or just an enumeration). I would have made an extension but this is clearly something not achievable through the current Extension API.

May I bump it please?

It would be great if this is implemented in a way that will make #98410 possible, too.

Any progress on this?


[TableToolbarAdvancedFilterFlyout] Advanded filter error when columns are empty

What package is this for?

[x] React
[ ] Angular

Describe the bug
The TableToolbarAdvancedFilterFlyout fails and breaks the table when the columns are an empty array or undefined, e.g. when lazy loading columns and data.
To Reproduce
Steps to reproduce the behavior:

Go to a story using TableToolbarAdvancedFilterFlyout
Modify the story to have no columns in the columns prop
See error

Please create a reduced test case in CodeSandbox:
https://codesandbox.io/s/github/carbon-design-system/carbon-addons-iot-react/tree/next/packages/react/examples/codesandbox

Expected behavior
No error thrown
Environment/versions:

OS: [e.g. MacOS, Windows]
Browser: [e.g. chrome, safari]
carbon-addons-iot-react version: [e.g. v2.60.0]

Additional context

Add any other context about the problem here.
Specific timeline issues / requests
Do you want this work within a specific time period? Is it related to an
upcoming release?
NB: The core contributors will try to work with your timeline, but it's not
guaranteed. The earlier you make a request in advance of a desired delivery
date, the better!

@JordanWSmith15 assigning this a severity:4 since the TableToolbarAdvancedFilterFlyout is experimental


[WFCORE-3037] WildFly Elytron 1.1.0.CR2

https://issues.jboss.org/browse/WFCORE-3037
https://issues.jboss.org/browse/WFCORE-3038
https://issues.jboss.org/browse/JBEAP-11914
https://issues.jboss.org/browse/JBEAP-11916

Full integration - Windows Build 3818 outcome was FAILURE using a merge of cd412f442fe94766a8a0e0b74edd5e61959ca6ab
Summary: Execution timeout (new); tests passed: 4042, ignored: 116 Build time: 02:01:15


fix demo

one-time bindings need angular 1.3

I've updated to 1.5 (so we no longer support 1.2 on the latest release)
1.3 and 1.4 aren't supported by the angular team)


Dynamic URLs in CoreUI Sidebar

I am using the CoreUI React Free Template to build out a React frontend project. The Sidebar component uses a navigation array of objects, which contains properties that are used to create CoreUI elements. Each object in the array has a "to" property, which provides the created nav item with a link, like so:
const nav = [
{
    _tag: 'CSidebarNavItem',
    name: 'Account',
    to: '/user/:id/profile',
    icon: 'cil-user',
  }
]

The nav array is imported into a component and passed as a render prop to a CCreateElement component that in turn creates CSidebarNavItems, which are glorified React Router Links.
CCreateElement
<CCreateElement
          items={navigation}
          {...props}
          components={{
            CSidebarNavDivider,
            CSidebarNavDropdown,
            CSidebarNavItem,
            CSidebarNavTitle
          }}
        />

I am trying to use dynamic urls in the "to" property, but my URLs return static strings /users/:id/profile. How I can get these rendered elements to generate these dynamic URLs?

For anyone wondering, took me a while but if you change it to:
/users/profile/:id then you can set exact: false
const nav = [
{
    _tag: 'CSidebarNavItem',
    name: 'Account',
    to: '/user/profile/:id',
    exact: false,
    icon: 'cil-user',
  }
]


No error message on walkthrough if password doesn't meet minimum criteria

if a password isnt at least 8 chars, it will show an error. Not sure what else this ticket meant. closing


Optional parameters are not being considered as being optional

Here's an example file causing this:
swagger: '2.0'
info:
  title: Plantando API
  description: RESTful API of the Plantando project.
  version: "1.2.0"
schemes:
  - http
basePath: /api
produces:
  - application/json
paths:
  /plants:
    get:
      summary: Get all plants
      description: |
        Returns information about all plants in Plantando. The response includes the name, kind, description, owner and the godparent of the plant.
      tags:
        - Plants
      parameters:
        - name: limit
          in: query
          description: Limit the result length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: Success - an array of plants is contained in the response body
          schema:
            type: array
            items:
              $ref: '#/definitions/Plant'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}:
    get:
      summary: Return a plant
      description: |
        Returns information about a particular plant identified by its ID. The response includes the name, kind, description, owner and the godparent of the plant.
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Success - plant with the specified plant_id is contained in the response body.
          schema:
            $ref: '#/definitions/Plant'
        404:
          description: The plant with the specified ID could not be found.
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Create or update a plant
      description: |
        Creates or updates the plant with the specified ID. The plant with the provided ID will be updated if it already exist. Otherwise a new one will be create..
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: plant
          in: body
          required: True
          schema:
            $ref: '#/definitions/Plant'
      responses:
        200:
          description: Plant was updated successfuly
        201:
          description: A new plant was created
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Removes a plant
      description: |
        Removes the plant with the specified ID from Plantando.
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Plant was removed successfuly
        404:
          description: The plant with the specified ID could not be found.
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/pictures:
    get:
      summary: All pictures assigned to a plant
      description: |
        The plant pictures endpoint returns an array with URLs of all pictures assigned to the plant with the specified ID.
      tags:
        - Plant
        - Pictures
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: limit
          in: query
          description: Limit the resut length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: Success - an array of URLs of the assigned pictures is contained in the response body
          schema:
            type: array
            items:
              type: string
        404:
          description: The plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/pictures/{picture_id}:
    get:
      summary: Get a plant picture
      description: |
        Returns the plant picture with the specified ID in its original binary format.
      tags:
        - Pictures
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: picture_id
          in: path
          description: Id of the picture
          required: True
          type: string
      responses:
        200:
          description: Success - the picture with the specified ID is contained in the response body.
        404:
          description: The picture or plant with the specified picture ID could not be found.
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Create or change a plant picture
      description: |
        Assigns a new picture to a plant or updates an existing one.
      tags:
        - Pictures
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: picture_id
          description: Id of the picture
          in: path
          required: True
          type: string
      responses:
        200:
          description: Picture with the specified ID was updated
        201:
          description: A new Picture was assigned to the plant
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Delete plant picture
      description: |
        Remove a picture with the provided id from a plant.
      tags:
        - Pictures
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: picture_id
          in: path
          description: Id of the picture
          required: True
          type: string
      responses:
        200:
          description: Picture with the specified ID was deleted successfully.
        404:
          description: The picture or plant with the provided ID could not be found.
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/water_needs:
    get:
      summary: Get the information about water needs of a plant
      description: |
        Returns the information about the frequency of watering and the amount of water needed in for one watering
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Success - the water needs of the plant are contained in the response body
          schema:
            $ref: '#/definitions/WaterNeeds'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Change the water needs of a plant
      description: |
        Assigns a new water needs to a plant
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: water_needs
          in: body
          required: True
          schema:
            $ref: '#/definitions/WaterNeeds'
      responses:
        200:
          description: Water needs of the plant were successfully updated
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/sunlight_needs:
    get:
      summary: Get the information about sunlight needs of a plant
      description: |
        Returns the information about the sunlight required for the plant
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Success - the sunlight needs of the plant are contained in the response body
          schema:
            $ref: '#/definitions/SunlightNeeds'
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Change the sunlight needs of a plant
      description: |
        Updates the sunlight needs to a plant
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: sunlight_needs
          in: body
          required: True
          schema:
            $ref: '#/definitions/SunlightNeeds'
      responses:
        200:
          description: Sunlight needs of the plant were successfully updated
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/location:
    get:
      summary: Get the location of the plant
      description: |
        Returns the information about the current location of the plant with the specified ID
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Success - the location of the plant are contained in the response body
          schema:
            $ref: '#/definitions/Location'
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Change the location of a plant
      description: |
        Updates the information about the location of the plant with the specified ID
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: location
          in: body
          required: True
          schema:
            $ref: '#/definitions/Location'
      responses:
        200:
          description: Location of the plant was successfully updated
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Remove the location of the plant
      description: |
        Removes the information of the plant location and move the plant to the pool.
      tags:
        - Plants
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
      responses:
        200:
          description: Location of the plant was successfully deleted
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/waterings:
    get:
      summary: Get the waterings of the plant
      description: |
        Returns the information about the waterings of the plant with the specified ID
      tags:
        - Plants
        - Waterings
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: limit
          in: query
          description: Limit the resut length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: Success - the array of waterings is contained in the response body
          schema:
            type: array
            items:
              $ref: '#/definitions/Watering'
        404:
          description: Not found - the plant with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /plants/{plant_id}/waterings/{watering_id}:
    get:
      summary: Get the watering by id
      description: |
        Returns the information about the watering with the specified ID
      tags:
        - Plants
        - Waterings
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: watering_id
          in: path
          description: Id of the watering
          required: True
          type: string
      responses:
        200:
          description: Success - the watering information is contained in the response body
          schema:
            $ref: '#/definitions/Watering'
        404:
          description: Not found - the plant or the watering could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Add or change the watering by id
      description: |
        Updates the information about an existing watering or creates a new one
      tags:
        - Plants
        - Waterings
      parameters:
        - name: plant_id
          in: path
          description: Id of the plant
          required: True
          type: string
        - name: watering_id
          in: path
          description: Id of the watering
          required: True
          type: string
      responses:
        200:
          description: The watering information was successfully updated
        201:
          description: A new watering information was successfully created
        404:
          description: Not found - the plant with the specified ID could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /areas:
    get:
      summary: Get all areas
      description: |
        Returns the information about the areas
      tags:
        - Areas
      parameters:
        - name: limit
          in: query
          description: Limit the resut length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: Success - the array of areas is contained in the response body
          schema:
            type: array
            items:
              $ref: '#/definitions/Area'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /areas/{area_id}:
    get:
      summary: Get an area
      description: |
        Returns the information about the area with the specified ID
      tags:
        - Areas
      parameters:
        - name: area_id
          in: path
          description: The ID of the area.
          required: True
          type: string
      responses:
        200:
          description: Success - the area is contained in the response body
          schema:
            $ref: '#/definitions/Area'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Create or update an area
      description: |
        Updates an existing area with the specified ID or creates a new one.
      tags:
        - Areas
      parameters:
        - name: area_id
          in: path
          description: The ID of the area.
          required: True
          type: string
      responses:
        200:
          description: An existing area was updated
        201:
          description: An new area was created
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Remove an area
      description: |
        Removes existing area with the specified ID. All plants contained in this area will be moved to the pool and the area will be unassigned from all employees.
      tags:
        - Areas
      parameters:
        - name: area_id
          in: path
          description: The ID of the area.
          required: True
          type: string
      responses:
        200:
          description: An existing area was removed
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /users:
    get:
      summary: Get all users
      description: |
        Returns the information about the users of Plantando
      tags:
        - Users
      parameters:
        - name: limit
          in: query
          description: Limit the resut length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: Success - the array of users is contained in the response body
          schema:
            type: array
            items:
              $ref: '#/definitions/User'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    post:
      summary: Register a new user
      description: |
        Creates a new user and adds it to Plantando
      tags:
        - Users
      parameters:
        - name: signin_data
          in: body
          required: True
          schema:
            $ref: '#/definitions/SigninData'
      responses:
        200:
          description: Success - the user was created successfully
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /users/me:
    get:
      summary: Get current user
      description: |
        Returns the information about the current user.
      tags:
        - Users
      responses:
        200:
          description: Success - the current users is contained in the response body
          schema:
            $ref: '#/definitions/User'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /users/{user_id}:
    get:
      summary: Get user by ID
      description: |
        Returns the information about the users with the specified ID
      tags:
        - Users
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
      responses:
        200:
          description: Success - the users is contained in the response body
          schema:
              $ref: '#/definitions/User'
        404:
          description: The user could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Add or update user
      description: |
        Creates a new Plantando user or upates an existing one.
      tags:
        - Users
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
        - name: user
          in: body
          description: User to create or update.
          required: True
          schema:
            $ref: '#/definitions/User'
      responses:
        200:
          description: The users was updated successfully
        201:
          description: A new users was created successfully
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Delete user
      description: |
        Removes an existing Plantando user. All plants owned by this user will be moved to the pool.
      tags:
        - Users
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
        - name: user
          in: body
          description: User to create or update.
          required: True
          schema:
            $ref: '#/definitions/User'
      responses:
        200:
          description: The users was removed successfully
        404:
          description: The user could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /users/{user_id}/picture:
    get:
      summary: Get user picture
      description: |
        Returns the user picture in binary format
      tags:
        - Users
        - Pictures
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
      responses:
        200:
          description: Success - the user picture is contained in the response body
        404:
          description: The user could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    put:
      summary: Add or update user picture
      description: |
        Creates a new user picture or upates an existing one.
      tags:
        - Users
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
      responses:
        200:
          description: The users was updated successfully
        201:
          description: A new users was created successfully
        404:
          description: The user could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
    delete:
      summary: Delete user picture
      description: |
        Removes an existing user picture.
      tags:
        - Users
        - Pictures
      parameters:
        - name: user_id
          in: path
          description: The ID of the user.
          required: True
          type: string
      responses:
        200:
          description: The user picture was removed successfully
        404:
          description: The user could not be found
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
  /user/{user_id}/plants:
    get:
      summary: All plants by user
      description: |
        Returns all plants owned by the specified user.
      tags:
        - Users
        - Plants
      parameters:
        - name: user_id
          in: path
          description: Id of the user
          required: True
          type: string
        - name: limit
          in: query
          description: Limit the resut length to n items.
          required: False
          type: integer
        - name: offset
          in: query
          description: Skip first n results.
          required: False
          type: integer
      responses:
        200:
          description: An array of plants owned by the user
          schema:
            type: array
            items:
              $ref: '#/definitions/Plant'
        404:
          description: Not found - the user with the specified ID could not be found
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
definitions:
  Plant:
    type: object
    properties:
      plant_id:
        type: string
        description: Unique identifier representing a plant.
      name:
        type: string
        description: User defined name of the plant.
      species:
        type: string
        description: Species of the plant.
      description:
        type: string
        description: User defined description of the plant.
      owner_id:
        type: string
        description: ID of the owner of this plant.
      godparent:
        type: string
        description: ID of the godparent of this plant. Can be empty if no godparent is assigned
  WaterNeeds:
    type: object
    properties:
      amount:
        type: string
        description: Amount of water to be used.
      period:
        type: string
        description: How often the plant shall be watered.
  Location:
    type: object
    properties:
      area_id:
        type: string
        description: ID of the area of the location.
      details:
        type: string
        description: Details of the location within the specified area (e.g. "desk 1" or "windowsill" ).
  Area:
    type: object
    properties:
      area_id:
        type: string
        description: ID of the area.
      building:
        type: string
        description: Name of the building.
      floor:
        type: string
        description: Name of the floor.
      room:
        type: string
        description: Name of the room within the building.
  SunlightNeeds:
    type: object
    properties:
      amount:
        type: string
        description: Amount of sunlight the plant need.
  Watering:
    type: object
    properties:
      watering_id:
        type: string
        description: Id of the watering.
      user_id:
        type: string
        description: Id of the user responsible for watering this plant.
      date:
        type: string
        description: Date of the last time the plant was watered.
  User:
    type: object
    properties:
      user_id:
        type: string
        description: ID of the user.
      name:
        type: string
        description: Display name of the user.
      area_id:
        type: string
        description: Id of the area where the user is usually located.
  SigninData:
    type: object
    properties:
      username:
        type: string
        description: Name of the new user.
      password:
        type: string
        description: Password of the new user.
      email:
        type: string
        description: E-mail address of the new user.
  Error:
    type: object
    properties:
      code:
        type: integer
        format: int32
      message:
        type: string
      fields:
        type: string

For instance, limit parameter in the first call

Fixed in 0.1.7


Allow a task to be registered for each task that matches some criteria

For example, allow build logic to register a report task for each test task, without realizing either task until required.
The implementation should also work for any kind of container that supports registration.

I just encountered this as well.
In my scenario, I'm attempting to add a finalizer task to all instances of a task type I do not own/control.  The problem I've encountered is that all of the task  configuration avoidance APIs deal in TaskProvider instances, which implies that I have foreknowledge of the task name,  which in my case I do not.
Typically, a Plugin would register a task to obtain a TaskProvider instance, then use TaskCollection#named to obtain another task's TaskProvider instance, at which point the two instances can be linked with a dependsOn call without having to perform the linkage at configuration-time.
Because I do not know the task names and the task may appear in an unknown number of instances per Project, my Plugin cannot leverage this approach, since I do not know how many instances of my task I'd need to create, nor do I know the names of the tasks I need to link to.  All that remains are configuration-time approaches and workarounds, which are sub-optimal.
What I had envisioned as a hypothesized solution was a method being added to TaskCollection such that I could call project.tasks.whenRegistered to gain knowledge of the task name and type.  Perhaps whenRegistered would invoke its callback and provide TaskProvider instances which have been extended to be able to expose the type, i.e., TaskPovider#getTaskType().  At that point we'd have the name and the type and could call existing API to stay within the confines of task configuration avoidance.


Introduce attribute delayed in krnl.memset and enable N-D Matmul to 3-D MatMul for dynamic shape

In PR #1612, we disabled support for dynamic shape when rewriting N-D Matmul into 3-D Matmul due to an unknown issue.
After investigating it, I found that it was caused by krnl.memset when the input is a MemRef with affine_map. More info about the issue is in the below comment.
This patch fixes the issue by introducing an attribute, named delayed, in krnl.memset to indicate two modes of krnl.memset:

delayed = false: do memset over the original iteration space of the MemRef with affine_map.
delayed = true: do memset over the extended iteration space (the space after the MemRef is normalized) of the MemRef with affine_map.

For example, if we want to set original elements to 1.0 and padding elements to 0.0 in a tiled MemRef:
   #map_2ds = affine_map<(d0, d1) -> (d0, d1 floordiv 64, 0, 0, 31, d1 mod 64)>
   %2 = memref.alloc(%i0, %i1) {alignment = 4096 : i64} : memref<?x?xf16, #map_2ds>

  // Set all elements including padding elements to 0.0.
  krnl.memset %2, %cst_f0 {delayed = true} : memref<?x?xf16, #map_2ds>

  // Set elements in the original area (non-padding elements) to 1.0
  krnl.memset %2, %cst_f1 {delayed = false} : memref<?x?xf16, #map_2ds>

Signed-off-by: Tung D. Le<EMAIL_ADDRESS>

Convert to Draft to investigate more.

Out of curiosity, N-d matmul is just multiple 2D matmul correct? So would there be any reason not to collapse the N-dim to 3D at all times to have fewer outer loops? Or maybe it does not matter? I would think that if we try to parallelize, it would be nicer to have only the 3rd dim to parallelize.

N-d matmul is just multiple 2D matmul correct? So would there be any reason not to collapse the N-dim to 3D at all times to have fewer outer loops?

It's correct. No reason except I just do it for NNPA now. We can move the rules to a common place without any modification. I can do it in this patch too.

@AlexandreEichenberger Regarding the issue of wrong results on NNPA when 3D Matmul has dynamic shape, I summary what Tong and I discussed offline and would like to hear your thoughts.
NNPA Matmul's signature is
zdnn_status zdnn_matmul_op(const zdnn_ztensor *input_a,
                           const zdnn_ztensor *input_b,
                           const zdnn_ztensor *input_c,
                           zdnn_matmul_ops op_type, zdnn_ztensor *output);

that computes input_a * input_b + input_c. In case of 3D, shapes are:

input_a: [s, m, n], layout 3DS
input_b: [s, n, p], layout 3DS
input_c: [s, p], layout 2DS

When we lower 3D Matmul to NNPA, we need to prepare a zero-ed input_c, and the wrong result came from wrong input_c where not all elements are zero-ed.
This is our code to prepare a zero-ed input_c.
#map_2ds = affine_map<(d0, d1) -> (d0, d1 floordiv 64, 0, 0, 31, d1 mod 64)>
%input_c = memref.alloc(%i0, %i1) {alignment = 4096 : i64} : memref<?x?xf16, #map_2ds>
krnl.memset %input_c, %cst_f0 : memref<?x?xf16, #map_2ds>

There are two situations to lower the above code:

lower it to affine, then normalize MemRef: we are doing this in onnx-mlir and it produces a wrong result.
normalize MemRef, then lower to affine: this leads to the correct result.

Situation 1 led to a wrong result because only the mapped part has zero values while the other part is untouch. We think this is natural to affine_map where only part of interest is visible. In particular, below is the generated code using --convert-krnl-to-affine:
    %4 = memref.alloc(%1, %3) {alignment = 4096 : i64} : memref<?x?xf16, #map0>
    affine.for %arg2 = 0 to #map1(%1) {
      affine.for %arg3 = 0 to #map2(%1, %3) {
        affine.store %cst, %4[%arg2, %arg3] : memref<?x?xf16, #map0>
      }
    }

then --normalize-memrefs:
    %5 = memref.alloc(%1, %4) {alignment = 4096 : i64} : memref<?x?x1x1x32x64xf16>
    %6 = memref.cast %5 : memref<?x?x1x1x32x64xf16> to memref<?x?x1x1x32x?xf16>
    affine.for %arg2 = 0 to %1 {
      affine.for %arg3 = 0 to %3 {
        affine.store %cst, %5[%arg2, %arg3 floordiv 64, 0, 0, 31, %arg3 mod 64] : memref<?x?x1x1x32x64xf16>
      }
    }

It is clear that not all elements in memref<?x?x1x1x32x64xf16> are set to zero.
Now, consider Situation 2, we do --normalize-memrefs first and get:
    %7 = memref.alloc(%4, %5, %6) {alignment = 4096 : i64} : memref<?x?x1x1x32x?xf16>
    krnl.memset %7, %cst : memref<?x?x1x1x32x?xf16>

then --convert-krnl-to-memref:
   %7 = memref.alloc(%4, %5, %6) {alignment = 4096 : i64} : memref<?x?x1x1x32x?xf16>
    affine.for %arg2 = 0 to #map3(%4) {
      affine.for %arg3 = 0 to #map4(%4, %5) {
        affine.for %arg4 = 0 to 1 {
          affine.for %arg5 = 0 to 1 {
            affine.for %arg6 = 0 to 32 {
              affine.for %arg7 = 0 to #map5(%4, %5, %6) {
                affine.store %cst, %7[%arg2, %arg3, %arg4, %arg5, %arg6, %arg7] : memref<?x?x1x1x32x?xf16>
              }
            }
          }
        }
      }
    }

All elements in memref<?x?x1x1x32x?xf16> are set to zero.
In summary:

If we need to memset a zTensor, is it reasonable to use affine_map to allocate the zTensor or not? It seems it's not and we should use a raw shape instead, e.g. memref<?x?x1x1x32x?xf16>.
The semantic of krnl.memset is currently reasonable or not, meaning it resets only the mapped part.
Whether this is the issue in --normalize-memrefs in MLIR.

@AlexandreEichenberger I implemented two modes for krnl.memset as your suggestion by introducing an additional attribute in krnl.memset. Please take a look. Thanks!

Jenkins Linux amd64 Build #7062 [push] Introduce attribute dela... started at 09:33

Jenkins Linux s390x Build #7077 [push] Introduce attribute dela... started at 10:33

Jenkins Linux ppc64le Build #6170 [push] Introduce attribute dela... started at 10:34

Jenkins Linux s390x Build #7077 [push] Introduce attribute dela... aborted after 1 hr 37 min

Jenkins Linux ppc64le Build #6170 [push] Introduce attribute dela... aborted after 1 hr 37 min


vcrpy fails to record requests made in setup and teardown

If I declare a test like this:
class Example(TestCase):
    def setUp(self):
        # Network requests here
    @vcr.use_cassette()
    def test_case(self):
        # Other things, other requests

The requests made in setUp are not recorded.
This issue is not remedied by using the VCRTest unit test integration

did you try decorating the setUp?
    @vcr.use_cassette()
    def setUp(self):
        # Network requests here

@j-funk that would work if subsequent tests didn't rely on setUp creating a fresh new state with every call. If we decorate setUp, the cassette created will works with test_case1, but then if it gets used while recording test_case2 it won't work because test_case2 expects that setUp would have created a fresh user.
class Example(TestCase):
    @vcr.use_cassette()
    def setUp(self):
        # Network requests to create a user here
    @vcr.use_cassette()
    def test_case1(self):
        # Other things, other requests expecting a fresh user 1
    @vcr.use_cassette()
    def test_case2(self):
        # Other things, other requests expecting a fresh user 2

We're specifically using vcr to mock out Stripe, in case you were wondering. Thanks for the great library 💯

You have to forgive me for saying, but this seems like an issue with the way your tests/system under test are designed more than an issue with VCR.  If your tests are hermetic ie. no side effects or shared fixtures (like a DB instance), then you'll have a better time testing.
A brute force workaround would be to create a helper function like make_network_request() and call that from each test.  Then the decorator in each test will work the way you intend and you avoid most of the code duplication.

@j-funk yup, that's the strategy we're using. It just involves a bit more refactoring that we'd initially anticipated. I just wanted to raise an issue to highlight something that tripped us up in our initial consideration of VCR as a drop-in solution for our existing test suite.
It might be worth adding an example of this usage (annotating the setUp) and this potential gotcha in the docs.

I also encountered this issue when trying to write tests around the Stripe API. It's certainly an architecturally tricky one...

I tried annotating setUp, but in my testing all tests use the setUp cassette then. The test doesn't switch the cassette in the middle of the run.

A lot of changes have happened to VCRpy since this ticket was opened. As this ticket has become stale and we have a lot of old tickets that need to be groomed, I'm closing this for now to make it easier to see what is still relevant.
However if it is still needed, please feel free to re-open or create a new ticket.
Thanks! 🙏


end 2 end working example for rmb

deploy kubernetes (or use existing one)
deploy 3 pods with rmb (rust version)
builder for the rmb in docker
deploy simple cosmos chain
get example working from vlang script, redis, to cosmos chain

kristof needs to make first example code, we can further work on


URL updates

Issue:

[X] Tick here to confirm that you have run RunCodeFormatter.ps1

Description:
Removed hardcoded base url in ui pipeline configuration
Updated old repo urls to new MicrosoftDX url
Test:
Build and run Nether.Web
Run Integration tests
Check that Admin UI loads

Hi @stuartleeks, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Stuart Leeks). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;


Question: Does coinwidget calls an external api to get the bitcoin address total? And where?

I'm learning how your coin widget works and how it communicates with the blockchain. What I'm trying to figure out is how do you communicate with the blockchain as i try to do a file FIND for any other external URLs and i cant seem to find any. It would be great if you can shed some light on this as I'm still learning how to develop stuff for crypto. thanks!

lookup.php just search for blockchain.info


Adding Pinot base build/runtime image support for Amazon Corretto and MS OpenJDK

Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK

Thanks @xiangfu0 Are we going to end creating images for oracle openjdk ?

Thanks @xiangfu0 Are we going to end creating images for oracle openjdk ?

I plan to switch the default to amazon jdk then also publish ms-openjdk image.

Thanks @xiangfu0 Are we going to end creating images for oracle openjdk ?

I plan to switch the default to amazon jdk then also publish ms-openjdk image.

I see Thanks for working on this

Looks good to me

Codecov Report

Merging #10422 (30ac267) into master (dadd42c) will decrease coverage by 6.05%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #10422      +/-   ##
============================================
- Coverage     70.26%   64.21%   -6.05%
- Complexity     5269     6082     +813
============================================
  Files          2049     1999      -50
  Lines        111068   108935    -2133
  Branches      16898    16646     -252
============================================
- Hits          78045    69956    -8089
- Misses        27554    33888    +6334
+ Partials       5469     5091     -378

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
67.99% <ø> (+0.13%)
:arrow_up:

unittests2
13.85% <ø> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 525 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks for the quick work  @xiangfu0 .Just curious does the latest-11-amazoncorretto point to 0.12.1


Feature/animal module - A Miscellanea

Description
This is a small PR with a miscellaneous bag of changes that combines housekeeping with some minor updates that add bit of realism to certain methods.
Changes include:

communities now know what grid they are in and what grids are neighbouring
an initialization method to populate communities with one cohort of each kind
natural mortality functionality based on allometric longevity
energetically motivated dispersal
tests

Fixes # (issue)
Type of change

[x] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist

[x] Make sure you've run the pre-commit checks: $ pre-commit run -a
[x] All tests pass: $ poetry run pytest

Further checks

[x] Code is commented, particularly in hard-to-understand areas
[x] Tests added that prove fix is effective or that feature works

Codecov Report

Merging #300 (7982279) into develop (173c791) will increase coverage by 0.42%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #300      +/-   ##
===========================================
+ Coverage    97.04%   97.47%   +0.42%
===========================================
  Files           46       46
  Lines         2067     2096      +29
===========================================
+ Hits          2006     2043      +37
+ Misses          61       53       -8

Files Changed
Coverage Δ

...irtual_rainforest/models/animals/animal_cohorts.py
98.66% <100.00%> (+0.13%)
:arrow_up:

...al_rainforest/models/animals/animal_communities.py
97.22% <100.00%> (+6.45%)
:arrow_up:

virtual_rainforest/models/animals/animal_model.py
100.00% <100.00%> (ø)

virtual_rainforest/models/animals/constants.py
100.00% <100.00%> (ø)

virtual_rainforest/models/animals/decay.py
100.00% <100.00%> (ø)

...rainforest/models/animals/dummy_plants_and_soil.py
92.00% <100.00%> (+10.75%)
:arrow_up:

...tual_rainforest/models/animals/functional_group.py
96.55% <100.00%> (+0.25%)
:arrow_up:

virtual_rainforest/models/animals/protocols.py
93.33% <100.00%> (ø)

...ual_rainforest/models/animals/scaling_functions.py
97.05% <100.00%> (+0.90%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


exit without further info when doanloading gzipped file

def download_blob(bucket_name, source_blob_name, destination_file_name):
#Downloads a blob from the bucket.
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(source_blob_name)
print(bucket.exists(client=storage_client))
print(bucket.get_blob(blob_name=source_blob_name))
with open(destination_file_name, 'wb') as file_obj:
blob.download_to_file(file_obj=file_obj)
#blob.download_to_filename(destination_file_name)
print('Blob {} downloaded to {}.'.format(
    source_blob_name,
    destination_file_name))

download_blob('test-ml-storage', 'deepdad/sample.tar.gz', './sample.tar.gz')
AttributeErrorTraceback (most recent call last)
 in ()
20         destination_file_name))
21
---> 22 download_blob('test-ml-storage', 'deepdad/sample.tar.gz', './sample.tar.gz')
 in download_blob(bucket_name, source_blob_name, destination_file_name)
13     print(bucket.get_blob(blob_name=source_blob_name))
14     with open(destination_file_name, 'wb') as file_obj:
---> 15         blob.download_to_file(file_obj=file_obj)
16     #blob.download_to_filename(destination_file_name)
17
/opt/conda/envs/py27/lib/python2.7/site-packages/google/cloud/storage/blob.pyc in download_to_file(self, file_obj, client)
462
463         try:
--> 464             self._do_download(transport, file_obj, download_url, headers)
465         except resumable_media.InvalidResponse as exc:
466             _raise_from_invalid_response(exc)
/opt/conda/envs/py27/lib/python2.7/site-packages/google/cloud/storage/blob.pyc in _do_download(self, transport, file_obj, download_url, headers)
416         if self.chunk_size is None:
417             download = Download(download_url, stream=file_obj, headers=headers)
--> 418             download.consume(transport)
419         else:
420             download = ChunkedDownload(
/opt/conda/envs/py27/lib/python2.7/site-packages/google/resumable_media/requests/download.pyc in consume(self, transport)
99
100         if self._stream is not None:
--> 101             self._write_to_stream(result)
102
103         return result
/opt/conda/envs/py27/lib/python2.7/site-packages/google/resumable_media/requests/download.pyc in _write_to_stream(self, response)
60             response (~requests.Response): The HTTP response object.
61         """
---> 62         with response:
63             body_iter = response.iter_content(
64                 chunk_size=_SINGLE_GET_CHUNK_SIZE, decode_unicode=False)
AttributeError: exit

@deepdad Sorry this issue occurred for you. See #3841 for an explanation of why it's happening.
Simply upgrade to requests >= 0.18.0 and the error will be resolved.

Hi there,
I seem to be seeing a similar error with gzipped files, despite having an up-to-date requests.
The error:
>>> from google.cloud import storage
>>> gcs_client = storage.client.Client(myproj)
>>> bucket = gcs_client.bucket(bucket_name)
>>> for blob in bucket.list_blobs(prefix='idk'): x = blob.download_as_string()
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/name/.virtualenvs/newenv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 502, in download_as_string
    self.download_to_file(string_buffer, client=client)
  File "/Users/name/.virtualenvs/newenv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 464, in download_to_file
    self._do_download(transport, file_obj, download_url, headers)
  File "/Users/name/.virtualenvs/newenv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 418, in _do_download
    download.consume(transport)
  File "/Users/name/.virtualenvs/newenv/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 101, in consume
    self._write_to_stream(result)
  File "/Users/name/.virtualenvs/newenv/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 62, in _write_to_stream
    with response:
AttributeError: __exit__

Below is the output of pip freeze:
cachetools==2.0.1
certifi==2<IP_ADDRESS>
chardet==3.0.4
dill==<IP_ADDRESS>
enum34==1.1.6
future==0.16.0
futures==3.1.1
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-spanner-admin-database-v1==0.15.3
gapic-google-cloud-spanner-admin-instance-v1==0.15.3
gapic-google-cloud-spanner-v1==0.15.3
google-auth==1.0.2
google-cloud==0.27.0
google-cloud-bigquery==0.26.0
google-cloud-bigtable==0.26.0
google-cloud-core==0.26.0
google-cloud-datastore==1.2.0
google-cloud-dns==0.26.0
google-cloud-error-reporting==0.26.0
google-cloud-language==0.27.0
google-cloud-logging==1.2.0
google-cloud-monitoring==0.26.0
google-cloud-pubsub==0.27.0
google-cloud-resource-manager==0.26.0
google-cloud-runtimeconfig==0.26.0
google-cloud-spanner==0.26.0
google-cloud-speech==0.28.0
google-cloud-storage==1.3.2
google-cloud-translate==1.1.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.26.0
google-gax==0.15.14
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2
grpc-google-iam-v1==0.11.1
grpcio==1.6.0
httplib2==0.10.3
idna==2.6
monotonic==1.3
oauth2client==3.0.0
ply==3.8
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-spanner-admin-database-v1==0.15.3
proto-google-cloud-spanner-admin-instance-v1==0.15.3
proto-google-cloud-spanner-v1==0.15.3
protobuf==3.4.0
pyasn1==0.3.4
pyasn1-modules==0.1.4
requests==2.18.4
rsa==3.4.2
six==1.10.0
src==1.0
tenacity==4.4.0
urllib3==1.22

Thank you

#3841 has no explanation and requests is up to date
import requests
print (requests.version)
2.14.2

Sorry, I mistyped, the minimum version is 2.18.0, not 0.18.0. "Up-to-date" would be the latest version on PyPI right now (2.18.4).
Sorry for the lack of information in #3841, I had to click my way through to find the real issue (#3736).
The issue is caused by using a requests.Response object as a context manager.

thank you


Update CI

Currently, torch-dftd's CI image is missing because of our image registry transfer. This PR updates the CI condition and also includes the following updates:

update DFTD3 download link in the Dockerfile
update the CI environment (Python, CUDA Toolkit, and Pytorch)
update repository to put the CI image
update pysen (because of some environment dependency)
add flex CI project to build and push the CI image

From now on, the CI image is placed at asia-northeast1-docker.pkg.dev/pfn-artifactregistry/torch-dftd/torch-dftd-ci:torch20

Now, we can build and push the CI image by manually creating a job on torch-dftd.build-images flex CI project.
https://ci.pfn.io/torch-dftd.build-images/
I confirmed the CI passed on the image built via torch-dftd.build-images.
https://ci.pfn.io/torch-dftd.pytest/1669443/
Image digest: sha256:b9a8de801f892b515783e2167691bc8448c563417fd65889b886d17592ed93ea

I think this is review ready!

Somehow failing with "execution finished: [ABORTED] excessive memory usage is detected" on
tests/functions_tests/test_edge_extraction.py::test_calc_neighbor_equivalent ...
Basically it compares ASE behavior and pymatgen behavior, and only basics functionality of torch is used.
I'm not sure which version change caused error : ase, pymatgen, or torch.

https://github.com/pfnet-research/torch-dftd/blob/43b38b8a4349b35e95536e00fada763e2f711e19/tests/functions_tests/test_edge_extraction.py#L14-L15
https://github.com/pfnet-research/torch-dftd/blob/43b38b8a4349b35e95536e00fada763e2f711e19/torch_dftd/functions/edge_extraction.py#L11-L64

In setup.py, it is written that # Note that we require ase==3.21.1 for pytest. so this version may be necessary?

Thanks. I checked its actual memory usage with ase==3.22.1. I can see that its peak is approximately 11GB, which is much smaller than the memory size of the instance that flexCI acquired. I heard that flexCI says an exceeding memory usage when it detects that swapping occurs. I don't know if accidental swapping can happen when there is enough memory, though.
https://ci.pfn.io/torch-dftd.pytest/1669787/
I've pinned ase==3.21.1 on CI and have kicked the CI again. Do you have any preference about pinning ase==3.21.1 on CI or using the latest one if we can?
https://ci.pfn.io/torch-dftd.pytest/1669810/

Okay, then, let me keep 3.21.1 on CI at this moment.

I think this is merge-ready. Would you merge this? Thanks!


When adding C++ classes we are unable to compile

When adding C++ class files the project can not be compiled. You have to set the project properties to do so by setting compile as to default.

But then when you compile lvgl is generating compile errors;

When removing the C++ classes and setting the compiler back to Compile as C Code (/TC) it all compiles and runs fine.

How can I solve this and combine my C++ classes with lvgl in visual studio?

First, you need to compile the LittlevGL files as C code and your C++ classes as C++ code. I'm not sure if you are doing this already.
Then you will most likely run into the fact that Visual Studio's C++ compiler has issues with one of the initializer constructs used in LittlevGL. This issue has been reported before in #2; the solution there was to switch to a different IDE which includes a compiler with support for this construct.

Hi
I would like to use LV in an Arduino project which is C++ but also use the simulator to have a quicker way for creating the UI.
Is there a viable way to use the simulator together with C++ code in Visual Studio? Maybe creating 2 projects and referencing the C project from the C++ project?
Or should I forget the idea and go without the simulator?
Thanks

Take a look at the platform_io project. Hope it helps.

Thanks, but I now went for writing my own object oriented GUI system.

Okay, good luck with your project!

Fixed by https://github.com/lvgl/lvgl/pull/1711 (will be released with 7.4).


DaemonSetChecker is failing

Describe the bug
DaemonSetChecker is failing because of insufficient permission for the user
Steps To Reproduce
Apply the YAML from v2.1.0
Check DaemonSetChecker pod logs
Expected behavior
insufficient permission for the user
Screenshots

Versions

Cluster OS: Ubuntu 18.04
Kubernetes Version: 1.11
Kuberhealthy Release or build [v2.1.0]

Hey there.  What specs did you use to apply your checks.  Were there any errors when that happened?
It looks like the daemonset checker pod can't fetch a list of pods using its service account.

Hey there. What specs did you use to apply your checks. Were there any errors when that happened?
I just used this https://raw.githubusercontent.com/Comcast/kuberhealthy/master/deploy/kuberhealthy.yaml

It looks like the daemonset checker pod can't fetch a list of pods using its service account.
yes

I see.  The only question is how long ago you applied that spec, then.  What KH image are you running?  kubectl get deployment kuberhealthy -n kuberhealthy -o yaml | grep image:
I just want to make sure this isn't a problem with our specs for some people.  I know that you can fix the issue by adding a new role binding for the daemonset checker service account.

I see. The only question is how long ago you applied that spec, then. What KH image are you running? kubectl get deployment kuberhealthy -n kuberhealthy -o yaml | grep image:
I just want to make sure this isn't a problem with our specs for some people. I know that you can fix the issue by adding a new role binding for the daemonset checker service account.

image: quay.io/comcast/kuberhealthy:2.1.0
Yes, I can do the role binding for my own but why it's not working for default?

@integrii I see that there is already  service account for daemonset checker

I've ran into the same issue as mentioned above
        image: quay.io/comcast/kuberhealthy:2.1.0
       ```

Can we reopen this? Unfortunately the change in #305  does not work for me.
The daemonset checker still has insufficient permissions, but incorrectly reports "success" to the status page. So it "appears" to work, but logs indicate the resources were never created/deleted because of insufficient permissions on the kuberhealthy-daemonset-khcheck ClusterRole.
Here are the logs:

A fix for this would be to add apps to apiGroups, daemonsets to resources, and create and delete to verbs. I'm asking for this change so that I can use the helm chart. If there is a different way I should be using this, let me know.
Other details:

Using the updates from #305  and #315 that were merged today (2/20/2020)
Kuberhealthy v2.1.2
Daemonset check v2.0.1
Cluster OS: Ubuntu 18.04

Thanks for bringing this up and trying out the new fixes.
Are you running your daemonset checker in the kuberhealthy namespace?  The RBAC in our spec files describe a ds-admin role in the kuberhealthy namespace which is attached to the daemonset checker.  That ds-admin role has the permissions you describe on it and should be set in your khcheck as the service account for the daemonset checker.
https://github.com/Comcast/kuberhealthy/blob/master/deploy/kuberhealthy.yaml#L206
And just to be clear: The best way to apply specs right now is to use the helm chart in this repository (under deploy/) or via the flat files linked in the readme.

Yes, the daemonset checker in running in the kuberhealthy namespace.
Thanks for pointing out the ds-admin role. Then, I believe the issue/fix is that on Line 216 the apiGroup should be apps, not app.
I just made this change locally and it worked for me (the logs reflected going through a successful daemonset check).
Thanks for the clarification. Wanted to mention I had the latest updates. Due to the configuration of my cluster, I need to make some changes to the flat files, which are also not currently configurable in the values of the helm chart either. I was going to make a PR to address these so that my team can use the helm chart (such as adding a nodeSelector value and envvars for KH_EXTERNAL_REPORTING_URL), and I see that there is a PR out for allowing non-hardcoded images which we also need. With those changes, my team could use the helm chart, and we'd like to 🙂

I just submitted PR #318 for fixing that typo you called out...


Enabling GitHub's "Discussions" feature

Hi Andrew and contributors,
Would you consider enabling GitHub discussions?
Thank you!

Hey @alexanderankin, any particular reason why? For small projects like this one, I prefer to keep everything inside of issues because it's doesn't introduce much noise and keeps everything in one place.

sure - i am generally anxious about trying to fit my scattered thoughts into a structure, in this case i'm not really qualified to help out with this project (at least this is my first impression) but i do kind of which there was a "back of the classroom" where i could hang out and still exchange ideas.
for example - i tried out the terraform module for this project and was not able to make it work. I then looked at the source code in here and was able to put together a very simple user data script for an ubuntu ami - perhaps amazon linux is slightly different. obviously this is not the direction of the project, as youre implementing on amazon linux. however, im sure there are others who would rather make it work on ubuntu, or some other combination of circumstances, or whatever...
also - i wanted to make some kind of a post to improve the SEO of the docs page to use the RPM or whatever, as i imagined those AMIs may not be there forever and incur costs, but a source code package or .DEB/.RPM may be something operationally a lot more feasible.
additionally, i wasted almost half a day today learning about nat64, rather, trying not learn about it - i was hoping to get a public ipv6 on aws (impossible?) and do a NAT setup where i was able to talk to public ipv4's without having to pay AWS $4/mo for this myself (impossible.). So maybe part of the good that this project brings into the world is simply aggregating people interested in this stuff? and then i dont want to make an issue claiming there is something wrong with fck-nat, just want to read up on the conversations.

im sure there are others who would rather make it work on ubuntu, or some other combination of circumstances, or whatever...

We can open an interest check for this, but I'd be very curious why Ubuntu in particular is desirable for this use case. I have suspicions, but would like to hear from users.

also - i wanted to make some kind of a post to improve the SEO of the docs page to use the RPM or whatever, as i imagined those AMIs may not be there forever and incur costs, but a source code package or .DEB/.RPM may be something operationally a lot more feasible.

Not quite sure what you're looking for here? The AMIs aren't going anywhere. Are you asking for the docs to make it more clear that an RPM package is available as an alternate install method?

So maybe part of the good that this project brings into the world is simply aggregating people interested in this stuff?

Part of my hesitation to turn on discussions is simply: I don't want to operate a community for this. Not that I don't think it would be valuable, but I simply do not have the time to be responsible for it. There are plenty of networking and AWS focused communities around and I don't think this project needs to add to that.

i dont want to make an issue claiming there is something wrong with fck-nat, just want to read up on the conversations. and educate myself from the best.

It's unfortunate that these are called "issues" because that really gives the wrong impression. Most discussions can happen inside of issues. Support, feature requests, bugs, etc, can all work fine inside of issues and they're generally easier for me to manage as a solo maintainer.

totally understood, I feel the same way from the maintainers perspective - e.g. testcontainers/testcontainers-python#517

I don't want to operate a community for this

✔


Change WeatherAgent to use Openweather instead of DarkSky (which is EOL)

This is one option for fixing #2732 .  Openweather offers a free tier that's pretty generous, and they also have done some work to making porting from DarkSky reasonably easy.  It's not a perfect match, but it works pretty well.
All this PR does is remove the ForecastIO gem (which is no longer maintained and no longer needed without DarkSky) in favor of just hitting the API directly with HTTParty. There are some Openweather gems, but they're pretty rudimentary, and don't buy us much that I could see.
Changed some documentation, hopefully cleared up some errors that had crept in. Comments and suggestion welcome. I'm trying to get familiar with Huginn.

This looks really good. Maybe we wait until Aug 1 to merge?

Maybe we wait until Aug 1 to merge?

Sure! I just wanted to see whether I needed to do this differently or if I was on the right track.
(What's happening Aug 1?)

I think that's the end of the Darksky API, according to your validation error?

Oh, duh, right on. 👍

I agree in general the code looks good. What is a bit unclear to me is the date when the API is shut down. The Dark Sky blog says that it will still be available until the end of 2021. I don't have an API key myself, so I can not verify that.
If their API is really still working I would love to see this change as an addition to DarkSky until they really shut down to keep the functionality for existing users.

Yeah, you're right. The Android app were to be shutdown on August 1; the API is still functioning.  So not sure what you wanna do.  I don't see the point in creating a separate agent for 4 months; perhaps just let the PR sit until Jan 1?  Not gonna bother me. What do you guys wanna do?

I agree, we should wait to remove the Agent until the API is dead so that we don't break it for people.

Can we add OpenWeather without removing the other APIs for now?

Can we add OpenWeather without removing the other APIs for now?

That would be great,

perhaps just let the PR sit until Jan 1?

That is also fine with me if you don't feel like investing more time into the change.

Dark Sky registrations are disabled. Am I supposed to wait until the end of 2021 to implement Weather events?

No, you can add it! Just don't delete support for the older APIs while they still function.

I don't see the point in creating a separate agent for 4 months; perhaps just let the PR sit until Jan 1?

@erebor As far as I know, the API is being shut down at the end of 2021, not Jan. 1, 2021. I think it would be worth it to either create a separate agent or provide an option in the Weather agent to switch between the two APIs. Otherwise, it means I can't use the Weather agent at all until 2022...

If you'd prefer to not invest more time into the PR for now, would you mind if I or someone else took over where you left off and added OpenWeather as an option instead of a replacement?

@hmnd I would not mind at all! I just ran out of available time to spend on it, but I'm happy to see somebody else pick it up and run with it.

@hmnd Any progress on this? It would be a good idea if the agent could support multiple API options

For historical context, the weather agent originally supported two API providers, DarkSky and Weather Underground. When Weather Underground decommissioned their API I removed support for WU from the agent.

FYI I'm working on updating this diff to support both Dark Sky and OpenWeather, at least until Dark Sky's API is decommissioned. I believe I've merged the two together, but need to sort out the tests as I'm not really a Ruby developer.
Progress here: https://github.com/JohnMaguire/huginn/tree/openweather

Thanks for this jumping off point @erebor! Per @cantino and @dsander's suggestion, I implemented this as another service in the existing agent in #2900.

We can also just delete the DarkSky code since the API is gone now.


BE redesign and refactor

UI quickly tested, main functionalities do seem to work, minor fixes added, squashing the commit history cya


(GSC-G103) Function call made to an unsafe package

Description
Package unsafe contains operations that step around the type safety of Go programs and may be non-portable. Also note that unsafe package is not supported by Go 1 compatibility guidelines. Read in detail -- https://golang.org/pkg/unsafe/
Occurrences
There are 2 occurrences of this issue in the repository.
See all occurrences on DeepSource → deepsource.io/gh/da-moon/soil/issue/GSC-G103/occurrences/

I am going to close this issue as the functions pointed out are unsafe by nature.


Wrong indentation?

not just indention, but also the if pseudo syntax is inconsisten, with both if-then-else and if-else used.


check if coco/train already exists in Gradient-Based Post Training Quantization code block

Issue Type
Performance
Source
source
MCT Version
2.2
OS Platform and Distribution
No response
Python version
No response
Describe the issue
https://github.com/sony/model_optimization/blob/main/tutorials/notebooks/imx500_notebooks/keras/keras_yolov8n_for_imx500.ipynb
see title - assign this to me
Expected behaviour
see title
Code to reproduce the issue
see title

Log output
see title

Hi @alexander-sony, thank you for raising the issue.
I assigned you for it as you requested, if you have any problems feel free to reach out.
I also assigned @Idan-BenAmi since he is responsible for this tutorial.


Add Support for Cyclonedx 1.5

Add support for start scoring cyclonedx 1.5
https://cyclonedx.org/docs/1.5/json/

Please find a basic sample cdx 1.5 bom here: sbom.json

thank your @heubeck  appreciate it

v0.0.29 adds cyclonedx 1.5 support


Determine why npm publish fails

During deployment it looks like npm publish is run twice.  Determine why this is the case.
#!/bin/bash -eo pipefail
npm publish

><EMAIL_ADDRESS>prepare .
> npm run compile

><EMAIL_ADDRESS>compile /home/node/project
> tsc -p .

><EMAIL_ADDRESS>postcompile /home/node/project
> cpy ./src/types/* ./build/src/types

npm ERR! publish Failed PUT 403
npm ERR! code E403
npm ERR! You cannot publish over the previously published versions: 0.8.1. : @google-cloud/logging-bunyan

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2018-05-08T22_01_56_560Z-debug.log
Exited with code 1

This error did not occur when releasing version 0.8.2, and thus I am closing this issue.