Plugins incompatible with pytest 4, leads to ContextualVersionConflict

As can be seen in this commit, none of the plugins are compatible with pytest. This causes test suites that depend on them to fail. Consider:
setuptools master $ pip-run -q pytest pytest-virtualenv -- -c "import pkg_resources; pkg_resources.require('pytest-virtualenv')"
pytest-shutil 1.4.0 has requirement pytest<4.0.0, but you'll have pytest 4.1.1 which is incompatible.
pytest-fixture-config 1.4.0 has requirement pytest<4.0.0, but you'll have pytest 4.1.1 which is incompatible.
pytest-virtualenv 1.4.0 has requirement pytest<4.0.0, but you'll have pytest 4.1.1 which is incompatible.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jaraco/code/main/setuptools/pkg_resources/__init__.py", line 898, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/jaraco/code/main/setuptools/pkg_resources/__init__.py", line 789, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pytest 4.1.1 (/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-svi02ctd), Requirement.parse('pytest<4.0.0'), {'pytest-shutil', 'pytest-fixture-config', 'pytest-virtualenv'})

@javefang @eeaston any specific reason this plugin is not compatible with v4.x.x?

Hi all, we just merged #134 to address this, thanks

Hi all, we just merged #134 to address this, thanks

Hi, do you plan a release that includes this change in the near future?

@eeaston I think this issue can be closed.

Released in 1.6.0 - many apologies all for the delay in releasing to pypi.


Why type 'service' and 'MapController' fails to load on below url?

https://rawgit.com/allenhwkim/angular-jsdoc/master/angular-template/docs/ngmap.map.html

The service type does simply not exist. You could create the service type like so:
/**
 * @ngdoc type
 * @name service
 *
 * @description
 * Any service you could imagine.

It would then correctly link to the documentation of the type service.
As for MapController, it does exist, but it’s not correctly linked to. MapController exists in the ngmap module, the current documentation assumes it’s not in a deeper module structure. You’d need to prepend the module name like @param {ngmap.MapController} ctrl map controller to clarify that.

do I need to put service type code ? in separate file of same file were we define the doc for controller?

You can put it anywhere.

ok thanks.


Just doesnt work - update until you resize browser

I am running mamp.
Followed instructions.
Tried to access the site on an IE laptop.
Doesnt work.

Sorry update on this, it does work (in IE 8). However only after you have resized the browser window.

$(element).hide();
setTimeout(function(){
$(element).show();
}, 2);
temporary fix seems to make it work.

same issue. It works a lot of times but every now and then it doesn't, only after a resize.


Documentations v1.4 rev2

Pull Request Test Coverage Report for Build 582

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 87.275%

Totals

Change from base Build 53:
0.0%

Covered Lines:
3107

Relevant Lines:
3560

💛  - Coveralls


Add "method" option to Vips ("reduction_effort")

From documentation: "Use reduction_effort to control how much CPU time to spend attempting to reduce file size. A higher value means more effort and therefore CPU time should be spent. It has the range 0-6 and a default value of 4."
It was added in Vips 8.8.0

Oh, this actually corresponds to the cwebp "method" option!


Add missing Docblocks

More docblocks are needed for the top layer APIs that are used the most. At the very least simple param and return tags for code completion in IDEs.

I'll try some :-) (started with Auth bundle)

Thanks!


Error while running cloned website on local machine

This is a Bug Report

Problem: I tried to run the cloned website, which I forked on my Ubuntu Machine, but am getting error. I tried both running via docker and my running locally using Hugo but getting some issues in both.
Following is the error while running via docker

and this is while running locally via Hugo

I also initialized modules by running the command: make module-init

/kind support

I tried to analyze a bit. On first investigation, it seems like a valid issue. However, I do not know the root cause.
I created a new Ubuntu VM, and tried to setup k/w from scratch.
npm version: 10.2.4
go version: go1.22.1 linux/arm64
hugo version: hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/arm64 BuildDate=2024-03-20T11:40:10Z VendorInfo=snap:0.124.1
Docker Engine version: 26.0.0

On running make serve, the hugo command was stuck at:
Start building sites …
hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/arm64 BuildDate=2024-03-20T11:40:10Z VendorInfo=snap:0.124.1

I then executed hugo -v --debug -D and managed to reproduce the issue you reported. (Note that at times this command gets stuck without an output as well).
ERROR [fr] Glossary Bundle not found for language. Create at least an index.md file inside docs/reference/glossary
ERROR [fr] No glossary items found
ERROR [fr] "/docs/concepts/overview/components": "kube-apiserver" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list
INFO  build:  step render pages 2897 content 6269 duration 23.51889686s
INFO  build:  step postProcess duration 12.459µs
INFO  build:  duration 26.870210574s
Total in 26951 ms
Error: error building site: render: failed to render pages: render of "page" failed: "/home/ubuntu/website/layouts/docs/single.html:3:7": execute of template failed: template: docs/single.html:3:7: executing "main" at <partial "docs/content-page" (dict "ctx" . "page" .)>: error calling partial: "/home/ubuntu/website/layouts/partials/docs/content-page.html:6:8": execute of template failed: template: partials/docs/content-page.html:6:8: executing "partials/docs/content-page.html" at <.page.Content>: error calling Content: "/home/ubuntu/website/content/fr/docs/concepts/overview/components.md:29:1": failed to render shortcode "glossary_definition": failed to process shortcode: "/home/ubuntu/website/layouts/shortcodes/glossary_definition.html:18:19": execute of template failed: template: shortcodes/glossary_definition.html:18:19: executing "shortcodes/glossary_definition.html" at <$term_info.Content>: nil pointer evaluating resource.Resource.Content

On my local machine, I have older versions of npm, hugo, go and docker and k/w builds fine.
npm version: 10.2.3
go version: go version go1.21.5 darwin/arm64
hugo version: hugo v0.120.4-f11bca5fec2ebb3a02727fb2a5cfb08da96fd9df+extended darwin/arm64 BuildDate=2023-11-08T11:18:07Z VendorInfo=brew
Docker Engine version: 25.0.3

It could be related to one of the upgrades in the dependencies (most probably Hugo), however I cannot state for certain.

I tried to analyze a bit. On first investigation, it seems like a valid issue. However, I do not know the root cause.
I created a new Ubuntu VM, and tried to setup k/w from scratch.
npm version: 10.2.4
go version: go1.22.1 linux/arm64
hugo version: hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/arm64 BuildDate=2024-03-20T11:40:10Z VendorInfo=snap:0.124.1
Docker Engine version: 26.0.0

On running make serve, the hugo command was stuck at:
Start building sites …
hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/arm64 BuildDate=2024-03-20T11:40:10Z VendorInfo=snap:0.124.1

I then executed hugo -v --debug -D and managed to reproduce the issue you reported. (Note that at times this command gets stuck without an output as well).
ERROR [fr] Glossary Bundle not found for language. Create at least an index.md file inside docs/reference/glossary
ERROR [fr] No glossary items found
ERROR [fr] "/docs/concepts/overview/components": "kube-apiserver" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list
INFO  build:  step render pages 2897 content 6269 duration 23.51889686s
INFO  build:  step postProcess duration 12.459µs
INFO  build:  duration 26.870210574s
Total in 26951 ms
Error: error building site: render: failed to render pages: render of "page" failed: "/home/ubuntu/website/layouts/docs/single.html:3:7": execute of template failed: template: docs/single.html:3:7: executing "main" at <partial "docs/content-page" (dict "ctx" . "page" .)>: error calling partial: "/home/ubuntu/website/layouts/partials/docs/content-page.html:6:8": execute of template failed: template: partials/docs/content-page.html:6:8: executing "partials/docs/content-page.html" at <.page.Content>: error calling Content: "/home/ubuntu/website/content/fr/docs/concepts/overview/components.md:29:1": failed to render shortcode "glossary_definition": failed to process shortcode: "/home/ubuntu/website/layouts/shortcodes/glossary_definition.html:18:19": execute of template failed: template: shortcodes/glossary_definition.html:18:19: executing "shortcodes/glossary_definition.html" at <$term_info.Content>: nil pointer evaluating resource.Resource.Content

On my local machine, I have older versions of npm, hugo, go and docker and k/w builds fine.
npm version: 10.2.3
go version: go version go1.21.5 darwin/arm64
hugo version: hugo v0.120.4-f11bca5fec2ebb3a02727fb2a5cfb08da96fd9df+extended darwin/arm64 BuildDate=2023-11-08T11:18:07Z VendorInfo=brew
Docker Engine version: 25.0.3

It could be related to one of the upgrades in the dependencies (most probably Hugo), however I cannot state for certain.

The issue is in the Hugo version. Upon changing the version, it worked perfectly. Thanks for your input. I think this issue should be raised.

@arujjval Can you please mention the exact combination of versions (npm, Hugo, go, docker) that worked for you?

@adityasamant25 here, it is
npm version: 10.2.4
go version go1.22.1 linux/amd64
hugo v0.121.2-6d5b44305eaa9d0a157946492a6f319da38de154+extended linux/amd64 BuildDate=2024-01-05T12:21:15Z VendorInfo=gohugoio
Docker version: 25.0.2
but I think the issue is with Hugo's version only.

Try make container-serve whilst you have the latest main branch checked out. That works for me.
You should find that you're using Hugo 0.121.2 (see netlify.toml).

The instruction to install a specific Hugo extended version as defined in netlify.toml is mentioned in the Running the website locally using Hugo section.
However, I believe it would be helpful to move the instruction to the Prerequisites section. This would prevent readers from inadvertently installing the latest Hugo extended version (currently v0.124.1) which is incompatible with k/w.
I'll try to work on a PR for this.

/assign


Prevent FlightData overflowing max size limit whenever possible.

Which issue does this PR close?
Closes #3478
What changes are included in this PR?
This reworks the encoding/writing step of flight data messages to ensure it never overflows the given limit whenever possible (specifically, it's impossible when we can't even fit a single row + header within the limit - there are still no mechanisms for splitting a single row of data between multiple messages).
It does this by first constructing a fake IPC header, then getting that header's encoded length, and then subtracting that length from the provided max size. Because the header's size stays the same with the same schema (the only thing that changes is the value within the 'length of data' fields), we don't need to continually recalculate it.
There are more tests I'd like to add before merging this, I was just hoping to get this filed first so that I could get feedback in case any behavior seemed seriously off.
Rationale for these changes
Since we are dynamically checking array data sizes to see if they can fit within the alloted size, this ensures that they will never go over if possible. Of course, as I said before, they will still go over if necessary, but I've rewritten the tests to check this behavior (if the tests sense an overage, but decode it to see that only one row was written, they allow it as there is no other way to get the data across).
Are there any user-facing changes?
Yes, there are API additions. They are documented. As far as I can tell, this shouldn't require a breaking release, but I haven't run anything like cargo-semver-checks on it to actually verify.

It looks like CI failed due to some network flakiness - I'm going to close and reopen to try it again

I had a brief skim, couple of comments
Adding an IPC specific API to ArrayData seems a touch unfortunate.
I'm not really sure of the context for this PR, but assuming it is to better avoid the gRPC limits, I worry this may be a fools errand. There is a lot beyond the data buffers going into those payloads (e.g. metadata flatbuffers, framing protobuf, HTTP framing, etc...) and trying to account for all of this is going to be a never ending game of wack a mole. Ultimately the only solution I can see is to set the soft limit in the encoder well below the hard limit enforced by the transport.

The context of this PR is that we have set the soft limit well below the hard limit (2MB vs 4MB) and somehow a customer still managed to hit the hard limit. So @itsjunetime  is trying to improve the specificity of the size.
We (at least our team in Influx) don't directly control all layers involved in gRPC communication -- this limit is being enforced by one of the various programs / mixings installed in kubernetes to manage traffic, report on things, etc. While in theory we should be able to figure out which one and increase its limits, that will also likely be a never ending game of whack a mole.
Let me see if I can help to find a way to break this PR up into smaller, more manageable pieces, so that we can get this in / tested in a way that is reasonable to maintain

Let me see if I can help to find a way to break this PR up into smaller, more manageable pieces, so that we can get this in / tested in a way that is reasonable to maintain

One change I could make that may help with breaking this up and also with @tustvold's concern about the IPC-specific interface would be to maybe genericize the get_memory_slice_size_with_alignment over a T: MemoryAccountant (or something like that), e.g.:
enum SizeSource {
  Buffer(BufferType),
  NullBuffer,
  ChildData,
  // does it need to be more granular? idk
}

trait MemoryAccountant {
  fn count_size(&mut self, size: usize, source: SizeSource);
}

impl ArrayData {
  fn get_slice_memory_size_with_accountant<A: MemoryAccountant>(
    &self,
    acc: &mut A
  ) -> Result<(), ArrowError> {
    // ...
  }

  fn get_slice_memory_size(&self) -> Result<usize, ArrowError> {
    struct DefaultAccountant(usize);

    impl MemoryAccountant for DefaultAccountant {
      fn count_size(&mut self, size: usize, _: SizeSource) {
        self.0 += size;
      }
    }

    let mut acc = DefaultAccountant(0);
    self.get_slice_memory_size_with_accountant(&mut acc)?;
    Ok(acc.0)
  }
}

This would allows us to use it nicely with the 'alignment' accounting that we need without being too IPC-specific. It would also allow us to remove the ugly re-accounting for RunEndEncoded buffers that this PR adds in get_encoded_arr_batch_size, which would be nice.
Obviously, I'd be happy to make this change and pull it out to a separate PR (to make this PR easier to review once that separate PR is merged) if we feel like this would be a better move.

Hi @itsjunetime -- what is the next step for this PR?
Both @tustvold  and I have suggested various ways to break it into parts that are more easy to review and the parts you were able to split off we got in

https://github.com/apache/arrow-rs/pull/6725
https://github.com/apache/arrow-rs/pull/6707

This PR has now accumulated a bunch of conflicts and I am not sure what to do with it now.
To reiterate, I don't feel comfortable / able to review this PR in its current form . That isn't to say I can't read the code, but it is beyond my ability to review it in pieces and verify it won't cause some issue / regression due to the number of things it changes.
Please let us know how you want to proceed.

Marking as draft while we decide next steps -- From my perspective it isn't waiting on any additional feedback and I am trying to clear the review queue.


Add option to change brackets style for formatter

Right now the brackets style like this:
SELECT
	(
	SELECT
		thecol
	FROM
		thetable )
FROM
	dual

I would personally much prefer a brackets style like this, with a line break before the closing bracket and the added indentation. Is this a feature you could add?
SELECT
	(
		SELECT
			thecol
		FROM
			thetable
	)
FROM
	dual

This issue want to add the button option to change brackets style or Can I modify code for new expected formatter.


rm -f does not work on windows

The deletion of files in RotateFileTest.h should be changed to something like this.
Unless you want to use <filesystem> which is already in msvc
#if (defined(WIN32) || defined(_WIN32) || defined(__WIN32__)) && !defined(__MINGW32__)
  std::string removeTgz_1 = _directory + "g3sink_rotatefile_test*.gz";
  remove(removeTgz_1.c_str());

  std::string removeTgz_2 = _directory + "new_sink_name*.gz";
  remove(removeTgz_2.c_str());
#else
  std::string removeTgz_1 = std::string("rm -f ") + _directory + "g3sink_rotatefile_test*.gz";
  system(removeTgz_1.c_str());

  std::string removeTgz_2 = std::string("rm -f ") + _directory + "new_sink_name*.gz";
  system(removeTgz_2.c_str());
#endif

Thanks
Does can you use
unlink(..), remove(...) on Windows?
Else "rmdir" could be used.
For Linux unlink or remove should be used. Or something similar to:

FYI: I'm looking into addressing this by using std::filesystem and seeing if the support is good enough in Linux. (It should just work on Windows).

closed by https://github.com/KjellKod/g3sinks/pull/124


short hand path cause error

hi~ I`m newer to front end test
my problem is set cypress fold in my project directly like
project

src

api

campaign.ts

test
-e2e
-  campaign.cy.ts

what i want is reuse variebles in test files but variebles file internal import other file use path alias , this cause error test file cannot read import variebles but if import file internal import other file with no path alias is ok
code like pictures head~
thx~

Hi @RosApr, I think the issue is coming from your configurations. Can you give me more context (tsconfig, error message, etc.) or create a minimal reproduction so that I can debug it?!

I have a project structure like this
src/
|-- components/
|   `-- ...
|-- utils/
|   `-- font-size.js
|-- App.vue
`-- main.js
tests/
`-- e2e/
    `-- specs/
        `-- userprofile.cy.js
cypress.config.js
package.json
vite.config.js

// vite.config.js
export default defineConfig({
  base: '',
  resolve: {
    alias: {
      '@': path.resolve(__dirname, './src'),
      ...
    }
  },
  plugins: [
    vue(),
    vuetify()
  ],
...

// cypress.config.js
const { defineConfig } = require('cypress')
const vitePreprocessor = require('cypress-vite')
const path = require('path')

module.exports = defineConfig({
  e2e: {
    setupNodeEvents (on, config) {
      on(
        'file:preprocessor',
        vitePreprocessor(path.resolve(__dirname, './vite.config.js'))
      )
      return Object.assign({}, config, {
        specPattern: 'tests/e2e/specs'
      })
    },
    specPattern: 'tests/e2e/specs/**/*.cy.{js,jsx,ts,tsx}'
  },
...

This module and its spec
// src/utils/font-size.js
export function getCurrentFontSize () {
  ...
}

// tests/e2e/specs/userprofile.cy.js
import { getCurrentFontSize } from '@/utils/font-size'
...

Result from cypress open

The following error originated from your test code, not from Cypress.

Cannot find module '@/utils/font-size'

    at webpackMissingModule (webpack:///./tests/e2e/specs/userprofile.cy.js:18:0)
    at ./tests/e2e/specs/userprofile.cy.js (webpack:///./tests/e2e/specs/userprofile.cy.js:18:0)
    at __webpack_require__ (webpack:///webpack/bootstrap:19:0)
    at 0 (http://localhost:38886/__cypress/tests?p=tests/e2e/specs/userprofile.cy.js:206:18)
    at __webpack_require__ (webpack:///webpack/bootstrap:19:0)
    at eval (webpack:///webpack/bootstrap:83:0)
    at eval (http://localhost:38886/__cypress/tests?p=tests/e2e/specs/userprofile.cy.js:87:10)
    at eval (<anonymous>)
From previous event:
    at evalScripts (http://localhost:38886/__cypress/runner/cypress_runner.js:158706:58)
    at <unknown> (http://localhost:38886/__cypress/runner/cypress_runner.js:158715:152)
From previous event:
    at runScriptsFromUrls (http://localhost:38886/__cypress/runner/cypress_runner.js:158715:136)
    at Object.runScripts (http://localhost:38886/__cypress/runner/cypress_runner.js:158729:12)
    at $Cypress.onSpecWindow (http://localhost:38886/__cypress/runner/cypress_runner.js:147444:75)


Check and summarize issues from support tickets

Check issues in the past to summarize and categorize the issues so that we can find the way to prevent that happens in the future.

Issues

Export all PM job
RFM/soundscape job stuck

Requests

Check timestamps
Add species to database
Create playlist from large project
Change project owner


Add cv::Mat_ and cv::Scalar_ conversions

Adds the templated Mat_ and Scalar_ converters.

Hi,
Thanks a lot for this PR!
Since all conversion cvnp are tested via CI, I had to add some some tests for the new conversions:
see branch https://github.com/pthom/cvnp/tree/mat_scalar_rect
Can you please merge this branch to your PR (and maybe add some more test if you see some I may have missed)?
Once done, I think it can be merged.
PS: is cvnp in use at Humainizing Autonomy? If so, I am happy :-)

Update: I did merge your changes and added the unit tests separately:
9958e1b (mat_scalar_rect) update readme: mention cv::Mat_, cv::Rect_ and cv::Scalar_
2a985ca Add unit test for cv::Rect_ conversions
a6e311e Add unit tests for cv::Scalar_ conversions
2d396ad Add unit tests for cv::Mat_ conversions
Thanks for your contribution!

Thank you for adding the tests! I've been quite busy about to go on holiday so I didn't have the time.
To answer your earlier question, I'm integrating cvnp into some non production code ATM. We will see if it makes it into production :wink:


⚙ chore(glob): add globby config options to handle Next.js route gro…

fix: support Next.js 13 route group folders in file listing
This PR fixes an issue where Next.js 13 route group folders (e.g., (dashboard)) were not visible when using @ to specify directories. The problem occurred because globby was treating parentheses as special characters for pattern matching.
Technical Details:
Next.js 13 uses parentheses in folder names for route grouping (e.g., (dashboard))
Previously, these folders were not properly listed due to globby's default pattern matching behavior
Special characters in folder names were being interpreted as glob syntax
Changes:
Disabled brace expansion to prevent parentheses from being treated as special characters
Turned off extended glob patterns to handle special characters in paths
Disabled directory expansion to ensure proper handling of route group folder names
Impact:
Next.js 13 route group folders are now properly visible and accessible
No changes to existing file listing behavior for regular folders
Maintains all other functionality (recursive search, directory ignoring, etc.)

but seems like this kind of issue only appears in vscode?working fine with windsurf.


[Docs] Revert Meshery Architecture diagram to SVG

Current State:
The first arch diagram on this page uses a png not an svg like the others.
The docs should be using vector-based images where possible, not rasterized images. The image is of poor quality and has an errant, floating OSM icon on it.
Desired State:

Download this diagram as an SVG from the Meshery Architecture deck.
Delete .png image and replace with SVG.

Contributor Resources

Meshery documentation site
Meshery documentation source
Instructions for contributing to documentation

Looks like the issue has been fixed in #1748 @leecalcote

@bartkalanski good eye. I missed this one. @bartkalanski, it'd be good to have your 👀 on PRs like these (or any other PRs that you'd like to review).  I see in part what you mean now. I see that the issue was assigned to you. Right, ok.

@bartkalanski good eye. I missed this one. @bartkalanski, it'd be good to have your 👀 on PRs like these (or any other PRs that you'd like to review). Great way for you to engage even more, learn more, and to avoid me bottlenecking progress. 😅

I will do for sure!

Oh! I see in part what you mean now. I see that the issue was assigned to you. Right, ok.

It was, but it doesn’t matter, it’s probably better that a brand new person to Layer5 has done this!


Scheduler no value eventhough specific days schedule should be false

Got a similar issue but with false.
If there is no schedule a false should be sent, because at the moment it does not send anything so the OR logic is assuming it is true.
Example: Tuesday the time schedule only starts at 15:00 and it is currently 14:30
export.txt

OK, that makes sense I'll have a look

Does it matter if I just send the payload and miss out the event info (to save having to work out what the last event would have been)

version 0.0.6 pushed to npm with this fix


Search window disappears after typing the first character

On OS X El Capitan when I try to search, the search window immediately disappears after entering the first character and I find myself typing the rest inside the code. This does not happen on a Linux machine.

I have solved the problem - it was called by the "Auto Save" plugin


Richer team and user profiles

Feature

[x] Allow admins and team managers to create a user through the UI

[ ] Better UX in Teams and User profile

More details to come.

Closing this since it is now tracked via #4009


worldBlacklist has no effect on bed-respawns

If a bed sets a new respawnpoint in one map and there is an
entry in worldBlacklist for this world players still spawn to the bed.

You may be using another plugin (Such as essentials) which by default handles teleportation.


All the objects detected by retinafce is not in the deepstream meta

I added this line to check the retinaface detections

        if(r.score<=VIS_THRESH) continue;
        std::cout << r.score << "\n";

and also a probe to the primary detector and print the bounding box info. some objects in the metafile is missing when compared with the print statement in .so file
  def dummy_probe(self, pad, info, u_data):

        gst_buffer = info.get_buffer()
        if not gst_buffer:
            logger.error("Unable to get GstBuffer ")
            return

        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        if not batch_meta:
            return Gst.PadProbeReturn.OK

        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                continue
            try:
                source_id = frame_meta.source_id
                payload = {}
                payload['objects'] = []
                l_obj = frame_meta.obj_meta_list
                while l_obj is not None:
                    try:
                        obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                    except StopIteration:
                        continue

                    object_ = {}
                    object_["y"] = obj_meta.rect_params.top / self.frame_height
                    object_['x'] = obj_meta.rect_params.left / self.frame_width
                    object_['w'] = obj_meta.rect_params.width / self.frame_width
                    object_['h'] = obj_meta.rect_params.height / self.frame_height
                    object_['class_id'] = self.labels[obj_meta.class_id]
                    object_['trackingId'] = obj_meta.object_id
                    object_['confidence'] = obj_meta.confidence
                    logger.debug(object_)
                    payload['objects'].append(object_)

                    try:
                        l_obj = l_obj.next
                    except StopIteration:
                        break
                print('\n'.join([str(o['confidence']) for o in payload['objects']]))

            except Exception as e:
                logger.error(str(e))
            try:
                l_frame = l_frame.next
            except StopIteration:
                break

        return Gst.PadProbeReturn.OK

I added this line to check the retinaface detections

        if(r.score<=VIS_THRESH) continue;
        std::cout << r.score << "\n";

and also a probe to the primary detector and print the bounding box info. some objects in the metafile is missing when compared with the print statement in .so file
  def dummy_probe(self, pad, info, u_data):

        gst_buffer = info.get_buffer()
        if not gst_buffer:
            logger.error("Unable to get GstBuffer ")
            return

        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        if not batch_meta:
            return Gst.PadProbeReturn.OK

        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                continue
            try:
                source_id = frame_meta.source_id
                payload = {}
                payload['objects'] = []
                l_obj = frame_meta.obj_meta_list
                while l_obj is not None:
                    try:
                        obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                    except StopIteration:
                        continue

                    object_ = {}
                    object_["y"] = obj_meta.rect_params.top / self.frame_height
                    object_['x'] = obj_meta.rect_params.left / self.frame_width
                    object_['w'] = obj_meta.rect_params.width / self.frame_width
                    object_['h'] = obj_meta.rect_params.height / self.frame_height
                    object_['class_id'] = self.labels[obj_meta.class_id]
                    object_['trackingId'] = obj_meta.object_id
                    object_['confidence'] = obj_meta.confidence
                    logger.debug(object_)
                    payload['objects'].append(object_)

                    try:
                        l_obj = l_obj.next
                    except StopIteration:
                        break
                print('\n'.join([str(o['confidence']) for o in payload['objects']]))

            except Exception as e:
                logger.error(str(e))
            try:
                l_frame = l_frame.next
            except StopIteration:
                break

        return Gst.PadProbeReturn.OK

Nice work!

how can i extract the  landmarks from the NvDsObjectMeta

@Athuliva set output-tensor-meta to true, then you can extract data from original model output.

i saw this in the updated readme decode landmarks in object_user_metadata commit ID. Does that mean now we can access the landmarks from the user meta using python?

@Athuliva
Q1: yes, since it is in usermeta_data now, I follow the official cpp example so it should work...
Q2: i'm not using retinaface from tensorrtx anymore. However, they are almost the same, just some differences between post-process. you can use the old one still, all need to be done is to rewrite the postprocess codes in nvdsinfer_customparser

@Athuliva Yes, you don't need that plugin anymore. trans onnx to trt is simple, you can find many examples online. there is a simple script according to official TensorRT.
'''
Author: zhouyuchong
Date: 2024-05-30 11:15:58
Description:
LastEditors: zhouyuchong
LastEditTime: 2024-07-16 16:45:05
'''
import os
import argparse
from loguru import logger
import tensorrt as trt
import math

TRT_LOGGER = trt.Logger()

def build_engine(onnx_file_path, engine_file_path="", set_input_shape=None):
    """Takes an ONNX file and creates a TensorRT engine to run inference with"""
    network_creation_flag = 0
    network_creation_flag = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)

    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(
        network_creation_flag
    ) as network, builder.create_builder_config() as config, trt.OnnxParser(
        network, TRT_LOGGER
    ) as parser, trt.Runtime(
        TRT_LOGGER
    ) as runtime:
        config.set_memory_pool_limit(
            trt.MemoryPoolType.WORKSPACE, 1 << 28
        )  # 256MiB
        # Parse model file
        if not os.path.exists(onnx_file_path):
            print(
                "ONNX file {} not found, please run yolov3_to_onnx.py first to generate it.".format(
                    onnx_file_path
                )
            )
            exit(0)
        print("Loading ONNX file from path {}...".format(onnx_file_path))
        with open(onnx_file_path, "rb") as model:
            print("Beginning ONNX file parsing")
            if not parser.parse(model.read()):
                print("ERROR: Failed to parse the ONNX file.")
                for error in range(parser.num_errors):
                    print(parser.get_error(error))
                return None
        # set input shape
        profile = builder.create_optimization_profile()
        logger.debug("total input layer: {}".format(network.num_inputs))
        logger.debug(network.num_outputs)
        output = network.get_output(0)
        logger.debug(output.shape)
        for i in range(network.num_inputs):
            input = network.get_input(i)
        #     assert input.shape[0] == -1
            logger.debug("input layer-{}: {}".format(i, input.name))
        profile.set_shape("images", set_input_shape[0], set_input_shape[1], set_input_shape[2])
        config.add_optimization_profile(profile)
        logger.debug("build, may take a while...")

        plan = builder.build_serialized_network(network, config)
        engine = runtime.deserialize_cuda_engine(plan)
        print("Completed creating Engine")
        with open(engine_file_path, "wb") as f:
            f.write(plan)
        return engine
def main(args):
    trt_file_name = args.onnx.replace('.onnx', '_bs{}.trt'.format(args.batch))
    size = [int(x) for x in args.size.split('x')]
    input_shape = [(1, 3, size[0], size[1]), (math.ceil(args.batch/2), 3, size[0], size[1]), (args.batch, 3, size[0], size[1])]
    logger.debug("set input shape: {}".format(input_shape))
    build_engine(args.onnx, trt_file_name, input_shape)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--onnx', type=str, default='', help='onnx path', required=True)
    parser.add_argument('-s', '--size', type=str, default="640x640", help='input shape')
    parser.add_argument('-b', '--batch', type=int, default=1, help='max batch size')
    args = parser.parse_args()
    main(args=args)

and you can use command as well, just follow the official tensorrt doc.
as for the custom parser for retinaface, there is one in repo.
I strongly suggest using official tensorrt instead of tensorrtx, since there are many new features and tensorrtx doesn't make adaption, even some codes can't even work.

Q1: yes, since it is in usermeta_data now, I follow the official cpp example so it should work...

how can i extract the landmarks from user meta?  user_meta.landmarks or something like that?

@Manuja1998 cpp example, python should also work but I didn't try.

type of the meta type is NvDSMetaType.???
if user_meta and user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
     pass
else:
      print( user_meta.base_meta.meta_type)

@Manuja1998 NVDS_USER_OBJECT_META_EXAMPLE I guess.

user_meta.base_meta.meta_type is NvDsMetaType.???
I think there i no pybinding to extract this into python

can you try casting it using ctype

@Manuja1998 yes your are right, seems there is no python binding for this type.  As @Athuliva said, can you try using ctype?

I tried this
                        elif user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
                            user_meta_data = ctypes.cast(user_meta.user_meta_data, ctypes.POINTER(ctypes.c_int))

i got wrong type error. @zhouyuchong may be can you give some insights on  how to caste it to python

This worked for me
                        elif user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
                            user_meta_data = user_meta.user_meta_data
                            ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
                            ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
                            pointer = ctypes.pythonapi.PyCapsule_GetPointer(user_meta_data, None)
                            pointer = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_int))
                            for i in range(10):
                                print(pointer[i])

This worked for me
if user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
        user_meta_data = user_meta.user_meta_data
        ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
        ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
        pointer = ctypes.pythonapi.PyCapsule_GetPointer(user_meta_data, None)
        pointer = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_int))
        for i in range(10):
             print(pointer[i])

good job!!


Seperate navigation tree cache for each host (Case 166581)

This allows us to populate navigation trees on dedicated preview servers with host names like "preview.webfactory.de" under different visibility rules.
Technically speaking, at least two issues could be considered as BC breaks:

the changed location of the tree cache file
the changed constructor parameters of TreeFactory

But I'd like to argue that for practical reasons, both of them should be consivered as internal implementation details. The TreeFactory class/service was never intended to be overwritten, as suggested by the private properties ranging back to 2017. => I would tag a release as a minor version.

@mpdude Seems to work as intented and I don't know which ghost I was hunting yesterday.
Fabian and I like this approach because it is encapsulated inside the bundle without a BC (arguably 👆).
With the alternative "only one tree cache with modified PageBuildDirector or an additional BuildDirector", we would need to consider the different tree structure in the blocks in src/Resources/views/Navigation/navigationBlocks.html.twig (or the overriding Twig template in the project respectively).
A downside of this approach is that in edge cases with projects with multiple host names (e.g. www.youthpass.eu and demo.youthpass.eu) the tree is cached once per host name - but this seems to be negigible (e.g. for GBA, the tree cache is ~220 kb).

In the project where we actually implemented the "preview" solution, we chose to use different Kernel instances for different modes of operation – as suggested in https://symfony.com/doc/current/configuration/multiple_kernels.html.
That gives complete separation of all types of caches, including the HTTP level, navigation tree, routing etc. So, I think this PR should be closed.


Support OCI-compliant alternative container registry for bicep publish

Hi everyone,
with the lastest version of bicep, we can now publish bicep modules in the ACR. This works very fine:
bicep publish secure.bicep --target 'br:dirien.azurecr.io/bicep/modules/app-service-plan:v1.0'

If i want to use, for example Artifactory its not working.
➜  flw-controlplane-azure git:(initial) ✗ bicep publish secure.bicep --target 'br:schwarzit-xx-sit-aebi-playground-docker-local.jfrog.io/bicep/modules/app-service-plan:v1.0'
Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 404 (Not Found)

Content:
404 page not found

Headers:
Date: Sat, 16 Oct 2021 16:39:28 GMT
Connection: keep-alive
X-Content-Type-Options: nosniff
Strict-Transport-Security: REDACTED
Content-Type: text/plain; charset=utf-8
Content-Length: 19

   at Bicep.Core.RegistryClient.AuthenticationRestClient.ExchangeAadAccessTokenForAcrRefreshTokenAsync(String service, String accessToken, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryBlobRestClient.StartUploadAsync(String name, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.BicepRegistryBlobClient.UploadBlobAsync(Stream stream, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PushArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference moduleReference, StreamDescriptor config, StreamDescriptor[] layers)
   at Bicep.Core.Registry.OciModuleRegistry.PublishModule(RootConfiguration configuration, OciArtifactModuleReference moduleReference, Stream compiled)
   at Bicep.Core.Registry.ModuleDispatcher.PublishModule(RootConfiguration configuration, ModuleReference moduleReference, Stream compiled)
   at Bicep.Cli.Commands.PublishCommand.RunAsync(PublishArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)
[1]    11253 abort      bicep publish secure.bicep --target

It would be cool, if we had support for alternative OCP compliant registries. Most of the enterprises, have already a binary repository in place. So it could be difficult to use a different one for "just" modul sharing.

We feel that this is worth considering but difficult to prioritize right now. Upvoting this issue or adding comments would help us prioritize this.

Number 1 makes sense to me but I urge you to really consider not 'reinventing the wheel' here in terms of authentication with OCI registries. All tooling I know use some form of ' login  --username --password' format to login to the registry. Docker uses this, helm uses this, oras uses this. Even the az cli supports az acr login which then persists your acr credentials in the docker config. I strongly feel that Bicep must support some form of bicep login or bicep registry login. This might even take the credentials automatically from your current az cli session if you don't specify separate credentials, but this command should at a minimum cache the credentials separate from az cli.
If the registry credentials would be cached separate from the az cli we would already be unblocked on this, because in that case we could simply do a az login again after login to the registry.

Just to add our scenario to this thread.  We were looking to use GitHub Packages to host some modules that we wanted to be publicly available without the need to authenticate.
In this case, the az publish command appears to fail whilst trying to validate the capabilities of the registry.
az bicep publish --file mymodule.bicep --target br:ghcr.io/<org-name>/<repo-name>/mymodule:v1
Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 405 (Method Not Allowed)

Content:
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

Headers:
docker-distribution-api-version: registry/2.0
Date: Mon, 31 Jan 2022 15:03:11 GMT
X-GitHub-Request-Id: REDACTED
Content-Type: application/json
Content-Length: 78

   at Azure.Containers.ContainerRegistry.AuthenticationRestClient.ExchangeAadAccessTokenForAcrRefreshTokenAsync(String service, String accessToken, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Containers.ContainerRegistry.ContainerRegistryBlobRestClient.StartUploadAsync(String name, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.Specialized.ContainerRegistryBlobClient.UploadBlobAsync(Stream stream, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PushArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference moduleReference, StreamDescriptor config, StreamDescriptor[] layers)
   at Bicep.Core.Registry.OciModuleRegistry.PublishModule(RootConfiguration configuration, OciArtifactModuleReference moduleReference, Stream compiled)
   at Bicep.Core.Registry.ModuleDispatcher.PublishModule(RootConfiguration configuration, ModuleReference moduleReference, Stream compiled)
   at Bicep.Cli.Commands.PublishCommand.RunAsync(PublishArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

Whilst the public registry will tick the box for the unauthenticated access requirement, I can still imagine the benefits of being able to push to an intermediate registry ahead of publishing to the official registry.  For example, to support some automated testing that consumes those modules, as published.

Not stale

ORAS has a .NET client that's now in development: https://github.com/oras-project/oras-dotnet
Once it's reached a suitable stability level this could be used to interact with non-ACR registries

We'd love to use GitHub Packages too for Bicep 😄

We are still interested in having bicep modules published to GitHub Packages or other OCI compliant registries.

I'd also like to see this implemented.
In enterprise context companies usually have a private artifact repository for all types of artifacts & dependencies (Docker images, Maven packages, npm modules etc.). This is usually implemented by using JFrog Artifactory, Nexus or similar. I don't want to make an exception for Bicep modules in a way that "We fetch all the packages from our private artifact repository but Bicep modules are fetched from Azure Container Registry). This causes unnecessary architectural drift in development tooling and company platform team has to spend some effort to explain this to development teams.
So as a Platform Team DevOps Engineer I want to be able to authentiate towards private OCI compliant artifact registry and pull and push bicep templates from private registry.

Also would like to see this implimented, we are just jumping on the Bicep train, but use Jfrog Artifactory as our Registry platform. Could be a bit of a hard sell as we will need to impliment ACR just for Bicep Modules.

Honestly, I am surprised there is not a native way to reference Bicep modules stored in public GitHub Repos. Having to standup and then support ACR to share modules that are already stored on a public github repo seems like a step backwards.
I am just a network engineer learning bicep, but if I write and store my deployments in github, why can't a reference another repo's bicep module without cloning the external modules folder to my own repo.
I would like to be able to reference it using a URL ' https://github.com/Azure/bicep-registry-modules/blob/main/modules/network/virtual-network/main.bicep'
(yes, I know that the above is in the public Bicep module repo, but why can't I use the above URL?)
Rather than 'module hw 'br/public:samples/hello-world:1.0.2''

@microsoftazureaaron -- We made an intentional decision to not support GH as a source for modules as it would make it easy to accidentally take a dependency on a non-versioned asset.
That being said, feel free to file this ask in a separate issue and we can think about revisiting our decision if there is enough interest.

I would like to add a +1 for this feature. would love to switch to github packages!


Current Installer (0.9) is not compatible with Input Manager plugin

via https://twitter.com/#!/appleseedTW/status/119599581137879040/photo/1
Current version of cocoaDialogue doesn't help by the way.

It should already be solved.


Use NPM as a social network

Pros:

http restful interface
accounts

Cons:

not much discovery

Closing as this is not a priority right now


Nav control docs can be improved

Ionic version:  (check one with "x")
(For Ionic 1.x issues, please use https://github.com/ionic-team/ionic-v1)
[ ] 2.x
[ ] 3.x
[X] 4.x
I'm submitting a ...  (check one with "x")
IMPROVE THIS DOC suggestion
comment one: your doc pages include an "improve this doc" link which brings me here, yet the "I'm submitting a ..." does not list "improve this doc" as an option.
The real issue: The docs for NavController,
https://ionicframework.com/docs/api/navigation/NavController/
have an example which includes this code:
pushPage() {
      this.viewCtrl.dismiss();
      this.appCtrl.getRootNav().push(SecondPage);
    }

However,  the code getRootNav is marked as depreciated.
The docs should be updated to show the proper way to call the function.
[ ] bug report
[ ] feature request
Please do not submit support requests or "How to" questions here. Instead, please use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:

Expected behavior:

Steps to reproduce:

Related code:

insert short code snippets here

Other information:

Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
insert the output from ionic info here

Hello! Thank you for opening an issue with us!

comment one: your doc pages include an "improve this doc" link which brings me here, yet the "I'm submitting a ..." does not list "improve this doc" as an option.

The intention with that link is that you use the editor you are brought into to modify the documentation as you see fit and then submit a pull request for the documentation change. It is there to allow you to easily contribute to the project by making our documentation make sense to the end user.
If you would like to do that, great! We will have a look at the PR. It is a great way to contribute to OSS, IMO.
If not, that is totally cool too. I have this marked as a docs issue.
Thank you for using Ionic

@kensodemann , I'd love to update the docs. But it seems there is an issue about how to know which id to use : https://github.com/ionic-team/ionic-site/issues/1197

@kensodemann I created a PR to reflect this change based on using getRootNavs()[0] instead which seemed to be the most reliable non-deprecated method according to ionic-team/ionic-site#1197


Improve random color generation for parcellations without explicit color assignment

A random color is generated for brain areas if they have no explicit color assignment using the "rgb" attribute. Currently, this color generation is based only on labelIndex, which is very often identical, resulting in the same color. It would be better to include other attributes in the random color generation (for example ngId) to make sure a wider range of colors gets generated.

fixed by #856


[Question]: Running on docker WSL2

Describe your problem
Hi,
I'm on Windows 11 using docker and WSL2 Ubuntu backend.
1. I use the following config for WSL2:
.wslconfig
[wsl2]
networkingMode=mirrored

2. Then I follow the installation tutorial:
git clone https://github.com/infiniflow/ragflow/
cd ragflow/docker
docker compose up -d
time="2024-08-02T10:32:50+02:00" level=warning msg="The \"MACOS\" variable is not set. Defaulting to a blank string."
[+] Running 5/5
 ✔ Container ragflow-minio   Running 0.0s
 ✔ Container ragflow-es-01   Healthy 0.5s
 ✔ Container ragflow-redis   Running 0.0s
 ✔ Container ragflow-mysql   Healthy 0.5s
 ✔ Container ragflow-server  Running 0.0s

This works and all containers are up and running.
3. Redis connection issue
The logs show ragflow can't connect to redis (I didn't make any configuration changes, everything is default):
docker logs -f ragflow-server
[WARNING] [2024-08-01 23:06:20,843] [synonym.__init__] [line:40]: Realtime synonym is disabled, since no redis connection.
[WARNING] [2024-08-01 23:06:20,847] [synonym.__init__] [line:40]: Realtime synonym is disabled, since no redis connection.
[WARNING] Load term.freq FAIL!
Traceback (most recent call last):
  File "/ragflow/api/ragflow_server.py", line 26, in <module>
    from api.apps import app
  File "/ragflow/api/apps/__init__.py", line 97, in <module>
    client_urls_prefix = [
  File "/ragflow/api/apps/__init__.py", line 98, in <listcomp>
    register_page(path)
  File "/ragflow/api/apps/__init__.py", line 84, in register_page
    spec.loader.exec_module(page)
  File "/ragflow/api/apps/canvas_app.py", line 26, in <module>
    from graph.canvas import Canvas
  File "/ragflow/graph/canvas.py", line 25, in <module>
    from graph.component import component_class
  File "/ragflow/graph/component/__init__.py", line 17, in <module>
    from .google import Google, GoogleParam
  File "/ragflow/graph/component/google.py", line 17, in <module>
    from serpapi import GoogleSearch
ModuleNotFoundError: No module named 'serpapi'

4. Inspecting network issues, show no problems:
I checked what the IP of ragflow-redis is via this command:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ragflow-redis
<IP_ADDRESS>

Then I use curl (ping not available in ragflow-server) to see if ragflow-server can access ragflow-redis:
docker exec ragflow-server curl <IP_ADDRESS>:6379
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

So the connection works. There's no network issue to my understanding.
What could be the issue?

Fixed. Pull the dev version of image again.

I tried with the latest dev image and it works perfect now. :) Thank you


Rename chiasm/runtime to chiasm

After separating out the server side code into a separate repository, the file "runtime.js" should be moved into the "main.js" file within the Chiasm RequireJS package. This way, users of the library will see
require(["chiasm"], function(Chiasm){
  var chiasm = Chiasm(someDiv);
});

rather than what it is currently:
require(["chiasm/runtime"], function(Runtime){
  var runtime = Runtime(someDiv);
});

Done in 1f2ce4c002dc08a60865eb25a5cd8e8743470aa9


SQL Server

Hi - the functionality is available in SQL Server now with the 2019 release, page says just Azure.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7199df2d-4950-2533-791c-4fc134ce8eed
Version Independent ID: 0b52e407-d9be-0ebd-2414-6a2478d999c3
Content: sys.sensitivity_classifications (Transact-SQL) - SQL Server
Content Source: docs/relational-databases/system-catalog-views/sys-sensitivity-classifications-transact-sql.md
Product: sql
Technology: t-sql
GitHub Login: @barmichal
Microsoft Alias: mibar

@SQLDTAIT -- David, thank you for your feedback.  I am looking into this document.

@WilliamAntonRohm Hi William,  please create the necessary PR, and use the following revised markdown lines to replace the current lines, to correct the AppliesTo / Versioning situation in the indicated article.
monikerRange: ">= sql-server-2019 || = azuresqldb-current || = azure-sqldw-latest || = sqlallproducts-allversions"
---
# sys.sensitivity_classifications (Transact-SQL)

[!INCLUDE[appliesto-ss-asdb-asdw-xxx-md](../../includes/appliesto-ss-asdb-asdw-xxx-md.md)]

Thanks.  (GeneMi)
cc: @barmichal (just fyi)


Missed Data

It happened to me multiple times that I write things and then all of them disappeared

it happened again and it makes me to redo my things again and again. waist of time!


diff: Add --no-error-on-diff flag

Add a --no-error-on-diff flag to the diff subcommand so as to not
exit with an error when there are diffs present. It will still exit with
an error if the configs are not valid, so kubecfg diff can be a useful
command to run on a branch of a CI system to show diffs and still
validate the configs.

Not sure if I should have raised an issue to discuss this feature first, but this is the third time I've needed this (with a TODO in my Makefiles to do it). I find this feature useful as I can run kubecfg diff on a branch and CI shows the diffs and validates the configs, and on master it is deployed. Because I run everything from a Makefile, it is a bit hard to detect the exit(10) that you get for a difference present condition.

Not sure how to get a passing build. Travis is saying:
$ travis_setup_go
gimme: given '1.12.x' but no release for '1.12' found
gimme: given '1.12.x' but no release for '1.12' found
Failed to run gimme

that's weird, I ran this 3 days ago and it worked: https://travis-ci.org/bitnami/kubecfg/jobs/648533819?utm_medium=notification&utm_source=github_status

Not sure kubecfg needs to have a special feature of that.
What's wrong with just using the standard shell's features to deal with that, like:
kubecfg diff .... || true

?

What's wrong with just using the standard shell's features to deal with that, like:

I covered that partly in the commit message - that shell construct will hide errors that are not due to a diff. But the other reason is that you may not have a shell because you have kubecfg in a distroless image or similar. Also I avoid running things with the shell as much as possible because it introduces quoting problems and changes argv from a vector to a single string (sh -c "single string").
But the other reason I'd like this patch to be merges is I don't think kubecfg doing exactly what you ask it to do should result in an error. At least how I use it. I think it should be the default not to exit with an error but it's too late for that.

but it's too late for that.

I wonder how many people are actually depending on that behavior.
I'm all for not breaking backward compatibility, but what if a behavior is objectively wrong?
What does kubectl diff do?

What does kubectl diff do?

I'm not sure. I haven't used that. And I can't run it now because we have "security" in my cluster:
$ kubectl diff -f foo.yaml
Error from server (BadRequest): admission webhook "validation.gatekeeper.sh" does not support dry run

As for whether bugs should be fixed that change behaviour, i think it comes down to pragmatism. If people have built scripts that depend on this behaviour, you'll be breaking them if you change this. I can see kubecfg diff being used as a validation step to ensure things have not changed that should not be changed - such validation could now pass unintentionally if you changed the default exit code to be 0.

kubectl diff returns 0 on no difference and 1 on a difference.
diff returns 0 on no difference and 1 on a difference.

Makes sense.
So, there are two main reasons why one would want to run k* diff:

To log what will change when we apply/update.
To detect config drift (e.g. to ensure that what you have checked in matches what is running in the cluster)

If I understood correctly, the current behaviour (consistent with kubectl and good old unix diff) works well out of the box in the second use case but poorly in the first use case, because if all you do is to look whether the exit code is non 0 you may misinterpret unrelated failures (e.g. transient networking, configuration) as "there are differences".
Technically, the exit code can encode such difference, by using the value 1 for "there are differences" and 2 for actual errors, as unix diff and grep utilities do (Not sure if kubecfg already does that, consistently). But that means that users of  kubecfg diff would need to have a more complicated exit status check, e.g. when using it in shell scripts they should use $? (something they should so anyways when writing robust scripts anyway even when using diff and grepvand friends, but rarely people does so, probably because it's cumbersome).
What if instead of adding a flag that forces found differences to be reported as exit code 0, we explicitly address the first use case and add a --diff flag to kubecfg update ?
Perhaps to could even ask for interactive confirmation before continuing (overrideable with --yes).
(It's been a long time since the last time I used borgcfg ; I think it worked that way although I forgot the details )

$ kubecfg update --diff foo.jsonnet
<some yaml diff shown>
Update? [Yn]:

What if instead of adding a flag that forces found differences to be reported as exit code 0, we explicitly address the first use case and add a --diff flag to kubecfg update ?

That would work for me - in my CI/CD I do the diff followed by an update which is either a dry run on a branch or not if master. That would probably shave some seconds off my CI.

Perhaps to could even ask for interactive confirmation before continuing (overrideable with --yes).

As long as it can be overridden, I'm fine with that. I don't know whether it makes more sense to have -i/--interactive to add the prompt or --yes to remove it - if I'm asking kubecfg to apply an update, the intent is clear that I want it done. Does --diff change that intent more often that not?
We would also need to add the --diff-strategy flag too - I always use it with subset (I'm not sure I understand the use case for all - it seems to me to always show diffs that mean little to me).

I don't have strong feelings for whether to default to interactive or not. On one hand I do agree with you that "do what I tell you" would more naturally follow the principle of least surprise. On the other hand I don't know of there are some precedents set by sister tools.
As for diff strategy: I agree. I always wondered the same. @anguslees?

We would also need to add the --diff-strategy flag too - I always use it with subset (I'm not sure I understand the use case for all - it seems to me to always show diffs that mean little to me).

The --diff-strategy subset has some deficiencies, for example #228 , as update uses the last-applied-configuration annotation.
These leads me to the concern that if update --diff is implemented, I will always want to use that instead of diff. It will also be confusing if the diff output of update --diff differs from diff. This would be addressed if/when https://github.com/bitnami/kubecfg/issues/228#issuecomment-517731648 is done. I will comment on that issue as well.


Base font size revisions

Description
Decreases base font size globally on larger screens to 14px:

Mobile font size remains unchanged (font size: 16px)
Font size mappings sage-font-size() have been modified to reference css custom properties rather than hard-coded values. This allows us to keep the current Sass mixin and variable references in elements and objects intact
Consolidates font-face declarations and removes unused entries and files
Updated documentation

Design questions

[x] Which elements will need further size adjustments to match the smaller scale?

Apply scaling from previous Figma mockup (https://type-scale.com)
Reference: https://polaris.shopify.com/design/typography

[x] Is 992px the preferred breakpoint for "desktop" / the decreased base font size?

UXD questions

[ ] Appetite for applying custom props to other design tokens or Sass variables
[ ] Future: should we move to a type scale multiplier instead of defining sizes? see reference here as an example. Doing so may get complicated—we make extensive use of the rem() function for sizing, which can't currently be altered to support a different relative base

Screenshots

before
after

Related

#203

@monkeypox8 per your UXD questions above for posterity, here's my take:

So if Sass provides variables and CSS provides custom properties, when would we use one or the other? This PR is a great example of tasteful use of custom props in a Sass context. I'd argue alongside this PR that using Sass variables as the default choice is best, but to leverage CSS Custom Properties for cases like this where the variability of the variables is so helpful. So here the same set of variables (font sizes) needs to be applied in a responsive fashion. While Sass variables have override abilities, the Custom Properties seem to accomplish this more elegantly IMO.
I would not be opposed to a ratio scale for the font sizes as long as we keep the line height snapping to the baseline grid (which may or may not defy the point of a scale ratio depending on your perspective). I believe that keeping at least the baseline to rounded values helps ensure tighter adherence to design specs and better reinforces the baseline grid and thus vertical rhythm.

@monkeypox8 per your UXD questions above for posterity, here's my take:

So if Sass provides variables and CSS provides custom properties, when would we use one or the other? This PR is a great example of tasteful use of custom props in a Sass context. I'd argue alongside this PR that using Sass variables as the default choice is best, but to leverage CSS Custom Properties for cases like this where the variability of the variables is so helpful. So here the same set of variables (font sizes) needs to be applied in a responsive fashion. While Sass variables have override abilities, the Custom Properties seem to accomplish this more elegantly IMO.
I would not be opposed to a ratio scale for the font sizes as long as we keep the line height snapping to the baseline grid (which may or may not defy the point of a scale ratio depending on your perspective). I believe that keeping at least the baseline to rounded values helps ensure tighter adherence to design specs and better reinforces the baseline grid and thus vertical rhythm.


Deprecate chip-tool CLI and prioritize the python chip device controller as primary developer and certification tool

Problem
Currently we have two developer tool, chip-tool and python chip device controller, for device controller. For chip device controller, we see the below benefits
-- Interactive Python interface to CHIPDeviceController.cpp which is used by iOS and android app
-- Cross platform: OSX & Linux
-- Controller & Commissioner functionality beyond what the chip-tool has
--  whl package support for python wheel
To support two tools, it is a little bit painful for developer and test team.
In today's 1/12/2020 developer standup meeting, we agree to have this device controller as primary developer and certification tool beyond and would use this ticket to collect the missing stuff that chip device controller don't have but chip-tool has and further feature requests developer and test team wanna to have.
Proposed Solution
Deprecate chip-tool CLI, and missing feature support in chip device controller if necessary, and publish the existing CLI doc to github.

@andreilitvin @franck-apple @robszewczyk @rwalker-apple @bzbarsky-apple @mspang

@andreilitvin @franck-apple @robszewczyk @rwalker-apple @bzbarsky-apple @mspang

@Damian-Nordic @LuDuda FYI

@Damian-Nordic @LuDuda FYI

I'm happy if at the end of the day we have a single developer tool!
On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.
chip-tool relies on ChipDeviceand the new controller API, while the python device controller relies on the deprecated API.
chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.

I'm happy if at the end of the day we have a single developer tool!
On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.
chip-tool relies on ChipDeviceand the new controller API, while the python device controller relies on the deprecated API.
chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.

I'm happy if at the end of the day we have a single developer tool!

Thank you!

On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.

In real device, we would not run CoreBluetooh and Bluez Central C++ implementation from platfrom BLE stack, since we would not support device vs device communication upon BLE in CHIP 1.0 scope, in device bluez side, we only run Bluez peripheral C++  implementation for device oobe/pairing. So in central test side, it is fine to put ChipCoreBluetoothMgr.py and ChipBluezMgr.py in device controller python CLI.

chip-tool relies on ChipDeviceand the new controller API, while the python device controller relies on the deprecated API.

Yep, Currently android test app is using Chip Device controller, and python test program is reusing what test app did. Agree, we can migrate the new ChipDevice API @erjiaqing

chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
if involving IM PacketBuffer, for different message, we provide CheckSchemaValidatity(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) to check packet buffer contents. @erjiaqing @vivien-apple  we may check together on what useful check we can migrate to new tool.

chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

for example, if command is malformed, IM's decoder(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) would check on it,  IM is responsibie for message's format's check, and which is not device controller's responsiblity.

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.
This tool would support BLE and IP rendezvous over PASE/CASE encryption, and WiFi/Thread/Fabric Provisioning, further it provides the python interface to IM so that it can easily send invoke Command, read/write cluster attribute against device over local BLE/IP connection.

I'm happy if at the end of the day we have a single developer tool!

Thank you!

On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.

In real device, we would not run CoreBluetooh and Bluez Central C++ implementation from platfrom BLE stack, since we would not support device vs device communication upon BLE in CHIP 1.0 scope, in device bluez side, we only run Bluez peripheral C++  implementation for device oobe/pairing. So in central test side, it is fine to put ChipCoreBluetoothMgr.py and ChipBluezMgr.py in device controller python CLI.

chip-tool relies on ChipDeviceand the new controller API, while the python device controller relies on the deprecated API.

Yep, Currently android test app is using Chip Device controller, and python test program is reusing what test app did. Agree, we can migrate the new ChipDevice API @erjiaqing

chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
if involving IM PacketBuffer, for different message, we provide CheckSchemaValidatity(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) to check packet buffer contents. @erjiaqing @vivien-apple  we may check together on what useful check we can migrate to new tool.

chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

for example, if command is malformed, IM's decoder(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) would check on it,  IM is responsibie for message's format's check, and which is not device controller's responsiblity.

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.
This tool would support BLE and IP rendezvous over PASE/CASE encryption, and WiFi/Thread/Fabric Provisioning, further it provides the python interface to IM so that it can easily send invoke Command, read/write cluster attribute against device over local BLE/IP connection.

On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.

In real device, we would not run CoreBluetooh and Bluez Central C++ implementation from platfrom BLE stack, since we would not support device vs device communication upon BLE in CHIP 1.0 scope, in device bluez side, we only run Bluez peripheral C++ implementation for device oobe/pairing. So in central test side, it is fine to put ChipCoreBluetoothMgr.py and ChipBluezMgr.py in device controller python CLI.

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.
This tool would support BLE and IP rendezvous over PASE/CASE encryption, and WiFi/Thread/Fabric Provisioning,

Ok. So for this part there is not difference to what chip-tool could be since it is basically replicating a controller.

further it provides the python interface to IM so that it can easily send invoke Command, read/write cluster attribute against device over local BLE/IP connection.

So the main difference is the Python interface then :)  I guess this is an important difference for certification folks though !

On top of my head and to the best of my knowledge here are some stuffs that chip-tool contains that seems missing in the python version:

chip-tool CLI uses the BLE implementation from src/platform/. I would like to see that in the python CLI as well instead of https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/ChipCoreBluetoothMgr.py since the platform BLE stack is what is running on the device.

In real device, we would not run CoreBluetooh and Bluez Central C++ implementation from platfrom BLE stack, since we would not support device vs device communication upon BLE in CHIP 1.0 scope, in device bluez side, we only run Bluez peripheral C++ implementation for device oobe/pairing. So in central test side, it is fine to put ChipCoreBluetoothMgr.py and ChipBluezMgr.py in device controller python CLI.

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.

"-- Controller & Commissioner functionality beyond what the chip-tool has" -> Can you elaborate ? It may help me to catch some differences that I have forgotten.
This tool would support BLE and IP rendezvous over PASE/CASE encryption, and WiFi/Thread/Fabric Provisioning,

Ok. So for this part there is not difference to what chip-tool could be since it is basically replicating a controller.

further it provides the python interface to IM so that it can easily send invoke Command, read/write cluster attribute against device over local BLE/IP connection.

So the main difference is the Python interface then :)  I guess this is an important difference for certification folks though !

chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
if involving IM PacketBuffer, for different message, we provide CheckSchemaValidatity(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) to check packet buffer contents. @erjiaqing @vivien-apple  we may check together on what useful check we can migrate to new tool.

chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

for example, if command is malformed, IM's decoder(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) would check on it, IM is responsibie for message's format's check, and which is not device controller's responsiblity.

Agreed that this is not the controller responsibility to check for the message format.
I think my point was that I'm currently using chip-tool to checks if our ZAP integration works correctly, and so inside chip-tool there is a bunch of checks to validate that the response is appropriate, which let me do a bunch of scripts to quickly checks against it.
But maybe this bunch of checks is only useful for hacking CHIP but useless for other use cases...

chip-tool has been written to do various checks against the content of the PacketBuffer that is received as a response to a command. It has let me find various bugs into the generated code. I don't know what is provided by the python device controller for such thing ? I understand that the IM may changes the content of those packets though, but until it lands, this is very convenient.
if involving IM PacketBuffer, for different message, we provide CheckSchemaValidatity(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) to check packet buffer contents. @erjiaqing @vivien-apple  we may check together on what useful check we can migrate to new tool.

chip-tool returns a failure if the response if unexpected. For example, if the cluster does not exists on the endpoint, or if the command is malformed. I'm not sure what the python device controller offers for that ?

for example, if command is malformed, IM's decoder(https://github.com/project-chip/connectedhomeip/blob/master/src/app/MessageDef.cpp#L3408) would check on it, IM is responsibie for message's format's check, and which is not device controller's responsiblity.

Agreed that this is not the controller responsibility to check for the message format.
I think my point was that I'm currently using chip-tool to checks if our ZAP integration works correctly, and so inside chip-tool there is a bunch of checks to validate that the response is appropriate, which let me do a bunch of scripts to quickly checks against it.
But maybe this bunch of checks is only useful for hacking CHIP but useless for other use cases...

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.
Agree. for python device controller in mac, it is better to use the same code as ios uses.

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.
Agree. for python device controller in mac, it is better to use the same code as ios uses.

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.
-- Yep, agree, for python device controller, we should replace ChipCoreBluetoothMgr.py with the same implementation as iOS app has.

Sorry by device I mean an iOS device. chip-tool simulates a controller, like the iOS device controller. And on iOS, src/platform/darwin is the used BLE stack.
-- Yep, agree, for python device controller, we should replace ChipCoreBluetoothMgr.py with the same implementation as iOS app has.

I think this should probably be tracked as a project... as the overall task here is to have the zap-generated-python-code fit all the needs for:
• a hand rolled python implementation of CHIP, just wrap lib chip on each platform
• a nice python API that is easy to use for testing/certification
• a simple cli-tool that developers can use
• supports all the current chip-tool features (like packet verification, @vivien-apple - what else?)

Hi, I think Linux Chip-Tool is very useful for Embedded Controller Device and Not all devices support Python in particular Embedded Control device.
I hope that  it can use Linux Chip-Tool CLI for Embedded Controller Device continuously.

Hi, @wissn84
python chip device controller is particularly used in linux desktop/raspi/macbook and simulate what mobile phone's pairing behavior. May I know which embedded controller device do you use for chip-tool CLI, and for which scenario?

Hi, I think Linux Chip-Tool is very useful for Embedded Controller Device and Not all devices support Python in particular Embedded Control device.
I hope that it can use Linux Chip-Tool CLI for Embedded Controller Device continuously.

Python Device Controller as well as the chip-tool, are targeted for testing, both the chip-tool and the python script are providing bindings for human use and not for building real products for use. The python script can provide programming interface and can build complex logics.
For building CHIP controllers on embedded linux platforms, I would suggest build an app using APIs provided by the controller class (or the CHIP library) and link it into your project instead of calling chip-tool or the python controller.

Hi, I think Linux Chip-Tool is very useful for Embedded Controller Device and Not all devices support Python in particular Embedded Control device.
I hope that it can use Linux Chip-Tool CLI for Embedded Controller Device continuously.

FWIW I raised this issue in one of the SWTT meeting as a downside of deprecating native chip-tool, but nobody else in attendance seemed concerned about it at that time.

To use Python Controller, The Embedded Controller Device is needed Python 3 and Python Library.
Both will be burden in view of embedded device. As you know the device capability is small.
For the upper reason I think Linux Controller API and chip-tool should be considered.

@wissn84 I think what we're discussing here is a generic testing tool used for development and automated testing and even if we go for Python CHIP Controller, Linux Controller API will still exist and anyone will be able to implement their minimal controller application for constrained devices with little effort. I wonder if there is a use case for generic testing tool for embedded devices. My thinking is that embedded devices would rather need a controller application tailored to a specific scenario (instead of an application supporting all possible clusters/commands), but let me know if I'm wrong and please describe your needs/use case.

I'm not 100% sure what this is covering at this point, initially this was to move the certification process over to python (which is going well) - but we have no plans to delete the C++ tool, it's a great way to test the C++ API, we'll soon have an Objective C tool, to test the Objective C API. Let's close this for now, given we have plenty of issues tracking the python tool remaining work - and it's close.


[Feature Request] Allow to restore tables in a live cluster

In order for CassKop to start using the sidecar we would need to be able to restore tables in a live cluster. The restore should be done in the following steps:

restore the data in a temporary folder
truncate all the tables that gonna be restored
create hardlinks to the restored files in the target folders
do a nodetool refresh (jmx calls)
clean temporary folder

@cscetbon
The approach seems all fine in case there are no changes of a schema. What if this is the case:

in time t1 with schema s1 I do a backup
in time t2, I alter a table, e.g. I add a column c1, so this will be on schema s2
in time t3, I want to make the restore, if I apply your procedure, is not it the case that I would restore SSTables on schema s1 into (empty, as truncated) table with schema s2?

The procedure I described would only work if the schema is identical and the topology didn't change.
If topology changed, you'll need to restore it using sstableloader cause the distribution/replication can be totally different.
If the schema changed, there is no way you can restore the table unless you restore it as it was using the previous method and apply the schema change manually, but then what about the data contained in that new column in your example ?
We won't be able to resolve all cases but we can try to get as close as possible 😉
@ahmedjami don't hesitate to weigh in on the procedure I described if I forgot to mention anything.

@cscetbon yes this is what I was thinking, however, I am not sure this tool should be enriched to deal with sstableloader, I feel like we are mixing things here. Not saying your idea with sstableloader is not sound, it is just I dont see its place in this particular piece of tooling, maybe we could do something custom or similar or make it another module.
As a food for thought, sstableloader might be embedded into our CLI tool, if you check the code it is just calling some stuff underneath so we could embed it and call however, if we are restoring e.g. 5 nodes cluster, this means we would have to download all data to one place and stream it by loader back to cluster? Or data would be saved locally to that node and then we would use sstableloader just for that part and this would be repeated for each node?

I only talked about sstableloader because you asked how it should work in cases where the live restore would not work. It’s a case we should handle though but I don’t know what form it should take. I also had in mind to embed it in your tool but this is something we should talk about.
To answer your question, you don’t have to push everything to one place to load sstables, you can always ask different nodes/containers to load different sstables using sstableloader.
But I don’t think this part should be the goal of another ticket.
The current ticket is to restore tables on a live cluster where schema and topology are identical

Hi,
@cscetbon the procedure is good to restore tables without a restart.
Going through a temporary folder will be very useful with the new nodetool command " import   -d " when we will upgrade to C*4.  It will replace the actual nodetool refresh.
@smiklosovic yes "refresh" performs loading of new sstables without checking schema. So, between t1 and t3 in your example, your data will be well restored. I just tested it to be sure and it works in both cases: drop or add  column.
Refresh just needs that replication and token ownership are the same for every node. Otherwise, an sstabeloder will be required.

@ahmedjami yes, I can already incorporate this into backup tooling so it would autodetect that it runs against Cassandra 4 so nodetool import would be invoked instead of making hardlinks and refreshing so from the user's point of view it will be all transparent.
However, to truncate means what we need to execute a CQL command and this tool does not know it yet but that is just an implementation matter. You would have to be sure that you propagate all your ssl stuff to your containers or what have you, sidecar (where backup-restore is embedded into) knows how to extract these credentials dynamically from Kubernetes secrets, consult this documentation to know more (1), I will do this the same way.
(1) https://github.com/instaclustr/cassandra-operator/wiki/Accounts,-login-and-SSL#ssl-with-sidecar

@smiklosovic not really, we can use jmx https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/StorageServiceMBean.java#L447-L457

@cscetbon nice! I was looking into this interface looking for this method but I had to be blind. Thank you, this simplifies things a lot.

@cscetbon @ahmedjami I am struggiling to understand the workflow you have described at the
beginning. Lets revisit that:

restore the data in a temporary folder
truncate all the tables that gonna be restored
create hardlinks to the restored files in the target folders
do a nodetool refresh (jmx calls)
clean temporary folder

This is all nice if we are ever going to deal with one node only. Imagine your cluster is consisting of 5 nodes. To truncate a table means that we invoke respective JMX method. That is fine, but you can not truncate per node. If you have 5 nodes in a ring and you truncate on the first node, it means that data would be truncated on all other nodes. Hence if you did all these steps and you would go to restore the second node, truncating it again would truncate what you restored on the first node.
Docs on truncate say:
    /**
     * Truncates (deletes) the given table from the provided keyspace.
     * Calling truncate results in actual deletion of ALL DATA IN THE CLUSTER
     * under the given table and IT WILL FAIL UNLESS ALL HOSTS ARE UP.
     * All data in the given column family will be deleted, but its definition
     * will not be affected.

So, I would change the flow like this:

restore the data in a temporary folder on each node
truncate all the tables that gonna be restored calling truncate on the first node, this means that data to restore will be downloaded and there will be no data in each node after calling truncate
create hardlinks to the restored files in the target folders on each node
do a nodetool refresh (jmx calls) on each node
clean temporary folder on each node

If these steps would be for one node in a cluster only, that is all fine and I can execute it in one flow, however I am not sure how to do this if we have not only one node in a cluster.
There might be a flag / option if a particular restoration request sent to node / invoked by CLI should truncate a table or not and you would have to orchestrate this from your operator.

yes @smiklosovic, sorry that was implicit for the truncate. I had exactly the same idea. You can choose the first node or a random node by providing a flag. the only thing is the truncate must happen before you restore on each node. So you need to have a way to ensure it's done sequentially.

@cscetbon

So you need to have a way to ensure it's done sequentially.

That is your task here, to orchestrate this from the operator. This tool is going to be invoked per  node only, if you are going to call via Sidecar, a REST request would have to be sent. If you are invoking it via CLI, you would have to manage that again, per node.
From the implementation perspective, both approaches should same, but this orchestration of what to do when have to be done by you. This tool is "cluster agnostic", it cares about one node at a time at most.

@cscetbon one more idea,
it could be done by me but for you to ingest this whole project more easilly, I would make one "cluster restoration" endpoint in our sidecar and this endpoint would do this whole orchestration on your behalf so your only entry point to restoration would be a sidecar of any node which would act like a "coordinator".
The input of this sidecar endpoint would be (among other things) a mapping between remote backup and local node (as data are stored again per node). Basically it would be a list of restoration requests and sidecar would take care of this internally - it would internally call all other sidecars with specific restoration requests to that respective node.
How do you actually want to invoke this restoration?

Basicaly: when a truncate has been invoked by a first node/sidecar, no data files will are present on the rest of the nodes so you can check this condition before invoke a truncate.
But, this could make sense only with sequential node restore to avoid 2 truncate on the same time.
Otherwise, in order to keep the sidecar generic and responsible off all the procedure without giving the control of truncate to an external programme/operator. I vote for the second idea @smiklosovic

@ahmedjami good, I am about to finish importing strategy as discussed for Cassandra 4, once it is done, the "old" way of doing this (by hardlinks) should logically more or less same ... stay tuned!

Good to hear that @smiklosovic. you'll have to tell us a bit more about this coordination thing I'm not sure to fully understand. But I like having only one endpoint for the whole thing if you can make it work ;) I'm not sure I understand why you need this mapping. Can you elaborate ?

If it is a sidecar that acts as a coordinator you need to be sure to get everything you need to contact the right sidecar. I feel like that's what you said but I'll need to know more about it. Cause CassKop has a notion of datacenters that you don't have for instance. I just wanna be sure you contact the right sidecars. Maybe you'll tell me we just have to treat different datacenters as different groups of sidecars with one coordinator in each ?

@cscetbon interesting point about the datacenters! I will do it with that in mind. I ll try to do some proof of concept and I ll do some docs about.
What I meant, in a nutshell, is that if there is a sidecar chosen to be a coordinator a "datacenter restore request" is sent to, that very sidecar has to know what other sidercars (except itself) are in that particular datacenter.
We can be smart about this and calling other JMX method to see what other nodes are there so all other sidecars would be on same IP address but I do not know yet how stable this would be. Or we can just play it simple and a list of sidecars would be given in a restore request by operator so this burden of realizing your topology would be left on you.
I ll try to investigate the first option and restore would work just on a datacenter level, not cluser wide level, you would have to issue another restoration request for the other dc by choosing some other sidecar from there to restore that dc ... lets just keep it simple here.