[Zeppelin 1165 : improve]  Paragraph dynamic relationship executable form

What is this PR for?
Workflow process feature.
(To ensure the success of each paragraph, it is possible to run consecutively.)
Case 1
Through a dynamic form, you can execute the order in paragraph.
There is a difference with traditional methods.
Please check the following flowchart.

Case 2
In general, when run a plurality of Paragraph, it performs Note entire run.
This is a good way to run a lot of Paragraph contained in the Note.
However, the problem occurs if the Interpreter of Paragraph different.

For Paragraph each using a different type of one of the Interpreter Note but run in sequence, the end is all different.

For example, Markdown is a very fast Interpreter.
The process is completed very quickly.
This is a problem in the sequential execution Paragraph.

This feature ensures a certain execution order Notebook with each Interpreter.
Case 3
For concurrent job in the workflow ...

If the current functional design is supposed to run at the same time, as follows
It is to share the results of the job.
But if the situation need to run the job at the same time, subject to their execution flow.
** The results will have to succeed, the following paragraph will be executed. **
What type of PR is it?
Improvement
jira
https://issues.apache.org/jira/browse/ZEPPELIN-1165
How should this be tested?
Dynamic form
For usage, see the following:
https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/dynamicform.html
In the markdown type:
%md
${workflow:workflow=MyWorkflowName,NotebookId01:ParagraphId01|NotebookId02:ParagraphId02}

details explain.

You can specify the title work flow.
Simply displayed on the UI.
${workflow:workflow=
or
${workflow:workflow=MyWorkflow

title name : MyWorkflow
character a comma(,), you can start working set.
${workflow:workflow=MyWorkflow**,**

Enter your Notebook ID and Paragraph ID.
${workflow:workflow=MyWorkflow,2BS99R6QA:20160712-163556_1475991024}

In order to distinguish between the Notebook ID and Paragraph ID, characters <:> it is.
The example was set to run 20160712-163556_1475991024 paragraph with a notebook id called 2BS99R6QA.

how to get notebook id?

how to get paragraph id?

Through a pipe character( | ), you can add tasks consecutively.
${workflow:workflow=MyWorkflow,2BS99R6QA:20160712-163556_1475991024|2BS99R6QA:20160710-03221_12234323}

Note run

If you want to run to all Paragraph in  Note, you can:
NoteID : *
or
NoteID

${workflow:workflow=MyworkflowName,NotebookId}
or
${workflow:workflow=MyworkflowName,NotebookId:*}

for example
${workflow:workflow=Myworkflow,2A94M5J1Z}

Of course, it is possible to run a particular paragraph and mix.
%md
${workflow:workflow=Myworkflow,2A94M5J1Z|r|2BQWXDH1C:20160714-151024_1284650101}

Screenshots (if appropriate)

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? yes, After the merge.

@cloverhearts Is this PR WIP?

Great feature!
Can you add documents for this or please explain more detail to review?

Test description modified.

@AhyoungRyu
It has been completed.
However, if the part you want, please tell us at any time.
Thank you.

update to description.
This feature ensures a certain execution order Notebook with each Interpreter.

jira re link - ZEPPELIN-1165
https://issues.apache.org/jira/browse/ZEPPELIN-1165

@cloverhearts Thanks for the detailed description. It would be better for reviewing this cool feature 👍

Cool feature, even I wanted to have something similar. Will test it, and give more feedback, in the mean while, and referring to the gif posted in screenshot segment, can the rest of the paragraph go into pending state.
For example in this case can the last paragraph %md go into pending state while %spark was executing ?

Thank you for your opinion.
I understood as a story about running redundant during the workflow.
If the current functional design is supposed to run at the same time, as follows
It is to share the results of the job.

But if the situation need to run the job at the same time, subject to their execution flow.
Did you find this answer?

update
add operator : all paragraph (*)
and ui fixed.

If you want to run to all Paragraph in  Note, you can:
NoteID : *
or
NoteID

${workflow:workflow=MyworkflowName,NotebookId}
or
${workflow:workflow=MyworkflowName,NotebookId:*}

for example
${workflow:workflow=Myworkflow,2A94M5J1Z}

Of course, it is possible to run a particular paragraph and mix.
%md
${workflow:workflow=Myworkflow,2A94M5J1Z|r|2BQWXDH1C:20160714-151024_1284650101}

Screenshot change

Thanks @cloverhearts for the contribution and the descriptions.
And sorry for giving some negative feedbacks here.
Workflow definition
I'm not sure about workflow definition shares dynamic form syntax ${ ... }.
I think it's not a good idea access two completely different feature with one syntax.
Alternative approach
Current approach does create new syntax for workflow creation. And WorkflowManager, WorkflowJob, and so on.
I think there is much simpler way achieve the same goal, with more flexible/powerful workflow control while keeping usage simple and easy.
Currently, z.run(PARAGRAPH_ID) in spark interpreter allows run other paragraph in the same notebook.
Simply extend this method, from
z.run(PARAGRAPH_ID)   # run paragraph in current notebook

to
z.run(PARAGRAPH_ID)   # run paragraph in any notebook
z.run(NOTE_ID)        # run all paragraphs in the notebook

will achieve the same goal without introducing new syntax or WorkflowManager.
Then you can just define workflow in scala programming language in spark interpreter. e.g.
z.run("Paragraph1")
z.run("Paragraph2")
if (somecondition) {
   z.run("Paragraph3")
} else {
   z.run("Note2")
}

Which is more flexible.
And note that, z.run() feature can be implemented not only in SparkInterpreter but also in any other interpreter. Therefore, other interpreter may provide different interface (python, etc) to access this feature. And if we really want new syntax, it's even possible to create %workflow interpreter and provide custom syntax or some gui using AngularDisplay system to define and run workflow.
So, i think just improving z.run() provides more flexible, modular, easier solution.
What do you think?

z.run("Paragraph1") z.run("Paragraph2")
will "Paragraph1" and "Paragraph2" run sequencially? If "Paragraph1" failed, will ""Paragraph2"" still run?

I like the idea, but I also don't think that a new dynamic form is a good way to manage it.
I think that feature could fit better inside of the JOB page, especially since it is cross-notebook.
JOB page is currently only listing the Notebooks and the status of each paragraph, we could see it as a way to manage any workflow. (Full Notebook, or Custom Workflow)
I also wonder how can we handle the case of paragraphs that can run in parallel during one of the workflow step. (multiple sql queries for example)
There was a lot of demands to have a better 'Run all paragraphs' feature, and maybe we could later have a dropdown on the run button allowing to run multiple profile (to run some workflow related to that notebook, a bit like building different profile in Eclipse)

@Leemoonsoo
Thank you for your opinion!
Workflow definition
I actually agree with your opinion.
The reason I made  is thought that the user wants to manage the workflow Note.
I am satisfied with the usability of the workflow according to the .
However, there was doubt whether this is also no problem using the .
It is not consistent usage.
Maybe @corneadoug think is right.
(add workflow feature in job menu)
Also feel the need to do to represent the workflow Paragraph well.
<% Workflow> are a good idea.
However, it is creating a new interpreter makes a number of concerns.
Maybe  is the same problem with this issue with.
This part I will try to think more.
Alternative approach
I know of the existence of z.run.
However, z.run can not replace this function.
z.run does not guarantee the order of execution of each other for the other Paragraph interpreter.
Case 2 See the following image.

Thanks again for details opinion.

@corneadoug
Your opinion is right.
 should be added to the .
But I was thinking that it must be able to control even in Note.
 to use convenient and it is true that I am expressing the workflow.
Satisfied.
However, I'm not sure Whether this is correct.
In particular usability and consistency.
<% Workflow> @Leemoonsoo 's idea is good.
If you have any other idea, please advise me.

z.run() currently submit paragraph into each interpreter's scheduler.
Let's say there're PARAGRAPH_1, PARAGRAPH_2, PARAGRAPH_3
%md
...

%spark
...

%sql
...

When user execute
z.run(P1)
z.run(P2)
z.run(P3)

z.run(P1) submit a job to run P1 to scheduler of Markdown interpreter.
z.run(P2) submit a job to run P2 to scheduler of Spark interpreter.
z.run(P3) submit a job to run P3 to scheduler of SparkSQL interpreter.
And each scheduler from each different interpreter works independently.
That's why P1, P2, P3 runs concurrently. (and this is why 'Run all' button doesn't run paragraph sequentially)
Then we can add synchronous option, such as
val result1 = z.runSynchronously(P1)
val result2 = z.runSynchronously(P2)
val result3 = z.runSynchronously(P3)

Then we can make sure P2 is not submitted before P1 finishes, P3 is not submitted before P2 finishes.
My points are,

We can provide API to interpreter runs notebook or paragraph in the other notebook.
So user can define workflow in programming language (i.e. scala, python, etc).
It naturally gives more flexibility and power user to creates workflow, because programming language provides such things: conditional block, future, threads, and so on.
To provide sequential job execution, we can provide synchronous api such as z.runSynchronously()
In case of users, who are not familiar with programming language, want to creates workflow, we can create %workflow interpreter and invent new syntax or GUI on top of the same z.run() API internally. This is more modular and less confusing compare to run workflow with dynamic form syntax.

Thank you for your answer.
I would restart operations soon to organize your information.
I plan to expand the .
If anyone have a different opinion, please write to comment.
Thank you.

Hi, may I ask if this feature has been merged to the master?

@xiufengliu No it wasn't merged.
In merged PR you can see a message saying: "asfgit closed this in CommitHash Xdays ago"
This is an interesting feature and there was some discussions on how this feature could be handled and how to define the flow, so I think it will need a lot of changes.

@xiufengliu
Thanks for your interest in this feature.
This feature can undermine the existing problems Zeppelin used consistently.
I would like to re-implement this feature in different ways.
If you ever write a Comment about the environment, or purpose you want to use you use, we seem to be a lot of help to improve and implement this feature.

Thanks! It is a fantastic feature. I am really looking forwards to.

This feature is not merged.
This feature will be re-implemented.
However, because there are people who need it, and I hope to continue to manage the PR.
I will continue the discussion later in a new way.
Thank you.

@cloverhearts If I understand, the plan is to open a new PR for this feature?
Can it be closed then?

@corneadoug
Yes, no problem.
:)

Hi, may i ask if the new PR was open? I am looking forward to this feature

Dear corneadoug:
May I ask if this pull request work is based on the zeppelin 7.0? Are there any problems to use it for now since I saw there are some conflicting files in the above checks? Thanks
/Xiufeng

@xiufengliu
This feature is currently implemented.
I have almost done this change so we can use it as a code implementation of the paragraph.
If a fast merge is done, I think I can introduce this feature to you before 2017.
Thank you.

https://github.com/apache/zeppelin/pull/1799
I was create new pr.
and re-implementation codebased.


Help me set this up?

Hi I am currently trying to set up the plugin onto a mysql database/phpmyadmin database and I have imported the .zip into the database now I dont know where to go from here. How will the server be connected to the database how will I make the plugin on the server connect with the database and what else do I have to do!
Thank You!
PS: i know there is already a closed issue for this but I still dont understand

LiteBans and litebans-php are two entirely different things.
The web interface (litebans-php) requires the plugin (LiteBans) in order to work. The web interface will not work on its own, and the plugin does not require the web interface in order to work.
The web interface is basically an online ban-list. It's a free add-on which uses and represents data created by the plugin.
To install the plugin, you have to buy it through SpigotMC and then configure it to connect to the database. The web interface can only be used after the plugin has been installed.


feat: bump base image to BCI 15.6

Solution:
Bump base image to BCI 15.6.

harvester
harvester-webhook
harvester-upgrade
general job
skip-restart-rancher-system-agent plan

Update binaries in upgrade image.

kubectl
virtctl
wharfie

Related Issue:
https://github.com/harvester/harvester/issues/6568
Test plan:
Create a 3-node harvester cluster without error.

@mergify backport v1.4 v1.3


SecurityResponseTypeCodeSet(323): values 3 and 4 missing

Code set SecurityResponseTypeCodeSet(323): values “3” and “4” which were missing from FIX 4.4. This has now been added.

Attribute deprecated="FIX.5.0SP1" needs to be removed to reflect the pedigree at the time of FIX 4.4 publication.


[Feature]: Add light theme to the application

Implement the feature to add a light theme to this application.

Hello @anmol111pal, this issue already exists #22 ds

Hello @anmol111pal, this issue already exists #22
Then close this issue @arpittyagi102


Bump up elasticsearch instance size / storage on prod

Fixes elasticsearch issues with running out of space... for now.

@cablej merging this


feat(app): Add support for verify issuer api function in the app

Signed-off-by: Talwinder kaur<EMAIL_ADDRESS>

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (0be2a9b) 89.06% compared to head (a0b5d49) 89.10%.

:exclamation: Current head a0b5d49 differs from pull request most recent head 1fb9efa. Consider uploading reports for the commit 1fb9efa to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #683      +/-   ##
==========================================
+ Coverage   89.06%   89.10%   +0.04%
==========================================
  Files         103      103
  Lines        4150     4150
==========================================
+ Hits         3696     3698       +2
+ Misses        310      308       -2
  Partials      144      144

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Adjust FLOT to show as complex geometry according to specification.

Use "Delay" as template for circles
Marek - do segmentation
Ondra - do circles

Todo:

[ ] Segmentation - will need some algebra inspired by here: https://github.com/spatialillusions/milgraphics/blob/9cc4e608b577e61b0aacf22ba322c2b2ca4feae1/src/geometry/pointbetween.js
[ ] Half-circles separation, inspired by the Delay sign

c9ef89b features version that shows segmented half circles
TODO:

[ ] Unify bearing size
[ ] (optional) - create padding between them


Add support for new @WithSpan annotation

Fixes #359

I now have troubles with muzzle and as I still don't quite understand it, I would need some help with the following error:
> Task :instrumentation:trace-annotation:muzzle-AssertPass-io.opentracing.contrib.dropwizard-dropwizard-opentracing-0.2.2 FAILED
FAILED MUZZLE VALIDATION: io.opentelemetry.auto.instrumentation.traceannotation.TraceConfigInstrumentation$TracerClassInstrumentation mismatches:
-- io.opentelemetry.auto.instrumentation.traceannotation.MethodsConfigurationParser:26 Missing class io.opentelemetry.auto.instrumentation.traceannotation.MethodsConfigurationParser
-- io.opentelemetry.auto.instrumentation.traceannotation.TraceDecorator:72 Missing class net.bytebuddy.matcher.ElementMatchers
-- io.opentelemetry.auto.instrumentation.traceannotation.TraceDecorator:75 Missing class net.bytebuddy.matcher.ElementMatcher
-- io.opentelemetry.auto.instrumentation.traceannotation.TraceDecorator:75 Missing class net.bytebuddy.matcher.ElementMatcher$Junction


Custom English anchor for non-alphabet headings in chapter0

What does this PR do?
Fixed internal anchors of chapter0 for Left-to-Right non-alphabetical languages (Bengalese, Gujarati, Hindi, Japanese, Korean, Traditional Chinese, Simplified Chinese, with more coming) to raise awareness of the current state of anchors in the documentation. I use transformers for my side-project and wanted to contribute.
Suggestions:

For example in the vietnamese course, Giới thiệu(introduction) is converted as gii-thiu(good-natured) in the url.

Putting an english anchor should be necessary for all of the documentation regardless of language. This is because when people switch languages (from en->vi), the current system disallows them to see the same part of the documentation. This is troublesome for long api docs. Ideally, when a user makes a simple change in the language code, they will get to the section they want fast. What I am saying is:

https://huggingface.co/course/en/chapter0/1#installing-dependencies
https://huggingface.co/course/vi/chapter0/1#installing-dependencies
https://huggingface.co/course/ko/chapter0/1#installing-dependencies
should all point to the same location. This is not the case now.

Putting the burden on the translator doesn't seem right. Anchoring in English can and should be done automatically. I suggest a 2 step change to the documentation.

Update the original English Documentation to include custom anchors by default for ease of translation by future contributors. This is rather easy and I would be making another PR for this.
Fix the currently translated documentation.

May I please have some guidance on writing a fixing program for the currently translated documents?
Note: Right-to-Left languages like Arabic have a unique linking structure, and I couldn't figure out where to put the custom anchor.
Thank you in advance for your review.

Part of https://github.com/huggingface/course/issues/376
Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?
@sgugger, @lewtun, @ArthurZucker, @eunseojo may you please review this PR?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Thank you so much for your reviews @lewtun, I'm learning tons from your approach and feedback.
I'll upload a new pr with for the English documentation with the anchors attached in a moment. Please hold my beer 🍺

Thank you so much for your reviews @lewtun, I'm learning tons from your approach and feedback. I'll upload a new pr with for the English documentation with the anchors attached in a moment. Just a moment please..

Amazing, that would be great - thank you!


Pass extra params to transforms

Ensure that callback recieves extra argument and current transforms object.
When recursing to children, pass parent element as argument to transform callbacks.

manually marshaled in


fix(graphcache): set operation when updating the cache with a result

Resolves #2768
Summary
When we update the cache we aren't writing to the operations map anymore because we moved this setter from the updateDependencies loop to the relevant functions in https://github.com/FormidableLabs/urql/pull/2736.

Thank you :hugs:, this has been affecting us since we updated to the latest urql version a while back. Any chance this will make it into a release soon?


Implement workpiece entity

This adds the workpiece entity. Should be compatible with all current frontend versions.

Add workpiece entity, which belongs to a project. Migration clears nearly all data again (sorry, necessary)
For now, it is only possible to have a 1:1 relation with a project, although it is possible to support multiple workpieces per project later. But that completely changes the current workflow, as each workpiece would very likely have its own welding points.
It is not possible to delete a workpiece!
When a project is created, a workpiece is added to the project. My initial plan was that it is possible to have the option to optionally provide data for the workpiece as well with the upload of a project. But form data cannot have JSON data as well, it is not allowed by HTTP. So an empty workpiece is added to the project :( Either we have to send two requests when creating a project (meh), or just keep it empty and the user has to do it in the project in later stage (meh meh)
Create project endpoint removed, as it was only a leftover option and I don't want to support this endpoint anymore. We create projects by uploading with a file
Upload endpoint of project now also returns the workpiece! Same for a get request on the project with an ID. It returns the same as the upload endpoint now, so project + welding points + workpiece.
The returned workpiece is always WITHOUT the model file. The model file has to be requested explicitly with a special endpoint of workpiece. Why? I have no idea how big model files are, and I don't want to attach a maybe multiple MB file to every project get request
Endpoint tests for the workpiece. Skipped the other tests due to time reasons. For now the endpoints are IMHO the most important ones.

Addressed the change requests


ParquetTpchTest not reading data as expected because DuckDB written files ColumnChunks' file_offset were all 0's

Parquet spec requires each ColumnChunk's file_offset to be set correctly.  This is the Parquet cpp generated from thrift definition:
uint32_t ColumnChunk::read(::apache::thrift::protocol::TProtocol* iprot) {
  ::apache::thrift::protocol::TInputRecursionTracker tracker(*iprot);
  ...
  bool isset_file_offset = false;

  while (true) {
    xfer += iprot->readFieldBegin(fname, ftype, fid);
    if (ftype == ::apache::thrift::protocol::T_STOP) {
      break;
    }
    switch (fid) {
      case 1:
        ...
        break;
      case 2:
        if (ftype == ::apache::thrift::protocol::T_I64) {
          xfer += iprot->readI64(this->file_offset);      // This is 0 for all columns
          isset_file_offset = true;                                   // But isset_file_offset becomes true
        } else {
          xfer += iprot->skip(ftype);
        }
        break;
      ....

  if (!isset_file_offset)
    throw TProtocolException(TProtocolException::INVALID_DATA);    // If isset_file_offset is not set the file is corrupted
  return xfer;
}

By specification, the file_offset of the RowGroup is optional. However DuckDB writes it for all RowGroups but does NOT write the required file_offset of the ColumnChunk. Take Q3 for example, the metadata being parsed is in the following picture:
The table has 1 rowgroups and 5 columns. All 5 columns have file_offset = 0, but the row_group's file_offset is 4.

This may result reading wrong rowgroups. In fact Q1 would read nothing because of this.
Another issue is DuckDB does not do Dictionaries encoding properly. For TPCH, dictionary encoding is crucial.
In summary we need to change the Parquet file writer from DuckDB to Arrow.

cc @oerling @majetideepak

@yingsu00 Please add tests to https://github.com/facebookincubator/velox/blob/main/velox/dwio/parquet/tests/reader/E2EFilterTest.cpp if you want to add tests with dictionary encoding and to use the Arrow writer. There are already some tests there with dictionary encoding.
The ParquetTpchTest is currently passing. Is there an issue with the test setup that is hiding the issue you posted? Do you have a reproducer?

@majetideepak I actually made a mistake in describing the issue. There was no failure for this particular case and sorry for the confusion. This issue is mainly for ParquetTpchTest to use Arrow Parquet writer. DuckDB writes RG file_offset but not ColumnChunk file_offset which is different from the spec. Another hope was to make this test cover the multiple RG case. E2EFilterTest already covers multiple RowGroup tests. I've updated the description accordingly.

@yingsu00 I accidentally opened a new one here #6495. I will close this in favor of that since there is already some discussion about the implementation there.


change access level of Site::onBeforeWrite() to public, in alignement wi...

...th Page::onBeforeWrite()

Cheers for that!


Versioning FHIR resources

I'm curious how Hearth can/could handle different versions of the same documents. For a use case, say, for example, that a ministry of health using Hearth has to store different versions of it's facility info because they are updating records and may need to rollback.
I don't see in FHIR how it supports namespaces/domains or other ways to declare something akin to MongoDB collections that are versioned. Interested in your thoughts.

FHIR does specify that every document is versioned when it is updated. This way you can refer back to an exact version of a document at a particular point in time using the history interaction - See https://www.hl7.org/fhir/http.html#history
Hearth has full history support for all resources. Is this helpful or were you thinking of something different?

I wonder if the question is more in terms of versioning a set of records instead of the individual record. So for example the ministry of health has loaded 500 facilities as their first starting, this will be considered version 1 of facilities, which can get small updates, like their name etc, but sometime in the future a new list is drafted with say 1500 facilities which will become version 2. Some systems could then still be using version 1 and others version 2 of the list.
I suppose something like this could be done with a coded value where the system specifies the version it is in? but not sure if this is a correct way of doing something like this or even correct in general

Sorry, I was unclear and mixed up the use cases. So, one use case is versioning as noted where you can use history. Another use case is where you have whole collections of docs that are the same resource type but versioned, like @BMartinos notes. So, in that case, perhaps metadata tags could be useful. Are those supported. See https://thefhirplace.com/2015/03/17/metadata-in-fhir/ on tags.

Closing as stale.


Import operation adds header row as "data"

When I tried to use "import" operation and provided CSV file contents as payload, I noticed that header row is duplicated and added as data as well. At line number 198 of AnaplanImportOperation.java class adds header row and for loop at line 202 loops the CSV data. At line 198 the header was added and since the for loop is running from 0th element, even the header row is added as data.
Line 198 : dataWriter.writeHeaderRow(rows.get(0));
Line 202:
for (String[] row : rows) {
dataWriter.writeDataRow(row);
++rowsProcessed;
}
Proposed Fix at Line 202:
for (String[] row : rows.subList(1, rows.size())) {
dataWriter.writeDataRow(row);
++rowsProcessed;
}

Thank you for the feedback. We have this tracked as a bug and working towards applying the fix. Once QA has performed regression, we will mark it for the next release.

Thanks for the update. Do you know planned release date of v1.1.4?

Quite soon, we haven’t set a date yet because we have to update Mulesoft also after the patch is applied to master. So it depends on their release cycle as well, which could take a few weeks.
But if you’re interested in test-driving it right now, check out the ‘develop’ branch from the repository and install it from source in Anypoint studio. You might have to uninstall it later and install the published connector from Mulesoft repos, so that you can continue receiving new updates from us.


Reset sensor names after homebridge reboot

Hi,
I have a problem - after changing the name of the sensor in the home app, homebridge does not remember it - after a reboot it is restored to default to Temperature. The same problem applies to humidity sensors.
As long as I do not perform a reboot homebridge is ok.

I am writing about the configured name option

Hi, yeah I am aware of that. The issue is that there is a bug in the home app and previously after a reboot all the accessories had the same name and each of them had to be renamed automatically. So I changed that to the current behavior, which is also not optimal but a better one than before. In order to manually store the name I would need to do some re-writing of the plugin, just not sure when I will have the time to tackle that...

Ok thanks, that means this is normal behavior. Is there possibly a halfway solution to default the sensors to names such as:
Temperature Humidifier
Temperature Purifier
Or that they can be edited in the json file? Although I guess that doesn't change anything.
Another option, is it possible to delete them? E.g. by unchecking?
You know, I'm thinking of such half-hearted solutions.

Yeah for now this is a normal behavior. You can use the suppressAutoServiceCreation property if you wish to disable some of them.

Can you write me how to do it or is there any instruction on the web? I am closer to an administrator than a programmer

Just check the documentation, the property is described there.


Add ibutsu reporting to pr_check

Description
Push pr_check tests report to ibutsu from ephemeral testing
Fixes # (issue)
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I run go fmt ./... to check that my code is properly formatted
[x] I run go vet ./... to check that my code is free of common Go style mistakes

Test results are available in Ibutsu. The test run IDs are:
31fee16e-c130-4b61-9045-2f5cc8e6279a
397d55a3-4ca3-43e3-81df-e878d65f2693

Codecov Report

Merging #964 (39bb4b2) into main (986f4cb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #964   +/-   ##
=======================================
  Coverage   30.74%   30.74%
=======================================
  Files          45       45
  Lines        5435     5435
=======================================
  Hits         1671     1671
  Misses       3529     3529
  Partials      235      235

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 986f4cb...39bb4b2. Read the comment docs.

@Mergifyio refresh

@Mergifyio unqueue

@Mergifyio requeue

@Mergifyio requeue

Test results are available in Ibutsu. The test run IDs are:
abb2222f-4070-497a-bc3d-91a03b6ea99f
d950324b-4f9d-4534-9b06-983683a11d10

Test results are available in Ibutsu. The test run IDs are:
6558df64-0312-4c3b-bc0d-cd84869e1395
93f1977f-9567-4280-835f-56b69b2dd756


Feat 5630/v2

#999
#996

Should CI be green ?

Should CI be green ?

I think it will be if we merge the framework patch (#999) and then test the test.

Should CI be green ?

I think it will be if we merge the framework patch (#999) and then test the test.

This PR has both and fails CI.
Is #996 waiting for a suricata PR ?

Should CI be green ?

I think it will be if we merge the framework patch (#999) and then test the test.

This PR has both and fails CI. Is #996 waiting for a suricata PR ?

I'm rechecking. No Suricata PR is needed, I think. I must have misunderstood the expected behavior.


Thoughts on setting for daprd (in integration tests) graceful shutdown duration to 0 (which makes it 0 seconds)?

I don't think we achieve much doing that. We will have proper graceful shutdown in 1.12 anyway.


market API - execute/bid and /ask, missing credit for receival

Core 5.1.3-rc5
Tested in wallet 3.1.3; once for market/execute/ask - ask txid: 019254c5e9ad8256adefeff6b3e53db03b273c716066d89b1f45205bb580f5f2f25bc24eb01816a69ce7c9d00da88ab6b2c71e95aca318a5461a89935e2f883a (market=/NXS)
Success txid: 01300a143cf64f2b28e5c1cf7c756e2b9dbb511782e4456b71c9cf1b63e7fd4532b480b106ea2e69e02d173fe61e2de45379180cb8e57b8efc9d8254ed0d549b
token1 was debited from seller account and credited to buyer account, NXS was debited from payment account but not credited by receival account. This was tested within same sigchain, manual credit of txid did not work (error -32).
Tested also for market/execute/bid - bid txid: 016e2563ee35025b9371c06b6554b08f852d13bb14d74aafb7f374c30bb178b8e1c87b8a141998f1398cbc0b75f8e0b1816a03ec1a06f532b846bdfcbf1769da (market=/NXS)
Success txid: 01e4a618d147d9b2a9c2d3a557d4adeb78664f6c1ece54e6b81d0a13c014df0e592d4f8f447aa6f77a2393cbf1077292644de8bb2631c6bca9c45b97bf7e9caf
Same as case above; NXS was debited from payment account and credited to receival account, token1 was debited from sellers account but not credited to buyers account.

Transaction inside same sigchain not possible. Works fine between two separate sigchains.
Closed.


dynamic parameter

add a controller to dynamically update parameters:
POST.      /api/v1/workflowdefinitions/{workflowDefinitionName}/parameters/update/{valueProviderName}
requestBody:
[
    {
        "key": "parameter_key",
        "value": "parameter_value",
        "workName": "onboardingAssessmentTask"
    }
]

new field in workParameter: valueProviderName
UI can use this field to decide if this parameter needs dynamic update
example parameter in new workflow definition:
"WORKFLOW_SELECT_SAMPLE": {
        "valueProviderName": "complexWorkFlowValueProvider",
        "format": "select",
        "description": "Workflow select parameter sample",
        "type": "string",
        "required": false,
        "enum": [
          "option1",
          "option2"
        ]
      }

@RichardW98 for some reason I thought we wanted to fetch param values from third party system. Is it something we want to do it in the future?

@RichardW98 for some reason I thought we wanted to fetch param values from third party system. Is it something we want to do it in the future?

yes,  this pr is meant to fetch the values from 3rd-party system with the ValueProvider   based on UI input

lgtm


Explain how to lookup defined macros and their value.

I needed Google and Stackoverflow based upon this documentation to help me find the macro's values.

@monty241,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@mblome please review

The instructions could be clarified, but this example is a little too specific, and thus confusing. You can get to the Macros pane from any property edit dialog, not just some specific post-build property which, as near as I can tell, does not exist in VS 2017, or at least, not as described.


Add use-from-path

I don't like the name. How about skip-micromamba-download or download-micromamba: false? Or micromamba-binary: <path> where path can be an absolute path or just 'micromamba'
Alternatively to this PR we could default to checking PATH and only download if nothing was found?


iOS Compatibility

@bkueng are you familiar with how to handle this issue?
https://github.com/mavlink/qgroundcontrol/actions/runs/10761996544/job/29841807927

Not specifically. Isn't the actual problem that the sysroot is not found?
clang: warning: no such sysroot directory: 'iphoneos' [-Wmissing-sysroot]

Yeah that's probably it, I will try. For some reason I thought you couldn't include  for iOS and had to write in Objective C


[HxSideBar] SideBar dropdown remains expanded when the window is resized

SideBar dropdown incorrectly remains expanded when an option was clicked and the window is resized. However, when we start directly with the desired page without any click the component behaves correctly.
Steps to reproduce (with click):

Go to https://havit.blazor.eu/

Click Navigation/HxSidebar

Narrow the window till the point where the vertical SideBar changes to horizontal - the dropdown remains expanded - wrong

Correct behaviour (without clicking):
The SideBar collapses immediately when the window is resized.

Go directly to https://havit.blazor.eu/components/HxSidebar

Don't click anything
Narrow the window till the point where the vertical SideBar changes to horizontal - the dropdown is collapsed - correct

Expected behaviour
The situation with click and without click is the same.

This is a known issue where we do not consider that a very likely scenario and have consciously decided not to address it for the time being. If you feel urge, please describe your scenario.
Feel free to submit a PR if you see a simple maintainable solution... ;-)


Authentication problem with socketIO

I have the following code in Angular 2:
import { Injectable } from '@angular/core';
import {Parameters} from '../../../parameters';
import {AuthService} from '../security/auth.service';

declare const io;
declare const feathers;

@Injectable()
export class SocketService {

  private _app: any;

  constructor(private authService: AuthService) {
    this.connect();
    this.authenticate();
  }

  connect(): void {
    var authService: AuthService = this.authService;
    var socket: any = io(Parameters.URL);

    var app: any = feathers()
      .configure(feathers.socketio(socket))
      .configure(feathers.hooks())
      .configure(feathers.authentication({ storage: localStorage, tokenKey: 'jwt'}))
    ;

    this._app = app;
  }

  authenticate(): Promise<any> {
    return this._app.authenticate({
      type: 'token',
      token: this.authService.getToken(),
    }).then(() => {
      console.info('Authenticated!');

    }).catch(function(error){
      console.info('Error authenticating!', error);
    });
  }

  getResource(service: string): any {
    return this._app.service(service);
  }
}

authenticate() method works fine but when I try to call getResource("some service") .someMethod then it says that token is missing.
Any suggestion how to fix that?

Also the code below seems not to work (params are never set):
app.mixins.push(function(service) {
  service.before((hook) => {
    Object.assign(hook.params, {
      user: authService.getUser(),
      token:authService.getToken()
    });

    return hook;
  });
});

I'm not sure what getResource("some service") does. The token will only be set automatically on service calls that are used on the client via app.service('some service'). For any other client mechanism you will have to add it yourself.

It is working in REST as expected but using socketIO, token is not sent to the server at all.
Shall I provide you with some code?

Yes please. Maybe a repository to reproduce. I have definitely used authentication and service requests with Socket.io before.

I have created a simple featherjs app: https://github.com/zender/feathers-285
Also client app (angular2):  https://github.com/zender/feathers-client-285
REST provider is working without any problems but socketio is not sending the token.
feathers-client does not read token from local storage (I had to set it manually) - https://github.com/zender/feathers-client-285/blob/master/src/app/shared/services/rest.service.ts#L36
Also hooks are not working either on the client side: https://github.com/zender/feathers-client-285/blob/master/src/app/shared/services/rest.service.ts#L25
Thank you in advance for your support.

I have the same problem, I am logged in rest but not in socket, I get an error 'token missing'.
So it would be nice to reconnect the user to the socket with cookies.

I sign in with facebook JS SDK in react app, after I use facebook-token passport strategy, it's work but if I also use a socket connection for a chat I'm not connected, how to connect the socket after a rest connection?
On client-side I noticed that we can't connect with socket on social provider: app.authenticate('facebook', data) is disallow and app.service('/auth/facebook') work but only if app use rest.

any updates???

I would recommend using a basic feathers example that works for you, then modifying what you need on top of that.

When i deploy a feathers application, the token gets set automatically when I access through the feather services, but get a 401 when doing it any other way.

For anybody having this issue, please make sure that your socket.io connection's upgrade process has fully stabilized before you call authenticate.

Socket.IO never assumes that WebSocket will just work, because in practice there’s a good chance that it won’t. Instead, it establishes a connection with XHR or JSONP right away, and then attempts to upgrade the connection to WebSocket. Compared to the fallback method which relies on timeouts, this means that none of your users will have a degraded experience. http://socket.io/blog/introducing-socket-io-1-0/

If you call authenticate right when socket.io connects, and you haven't locked the transports to websocket, the XHR-based connection with be authenticated, but then will subsequently be lost in the upgrade to JSONP or WebSockets.  There are two workarounds to this problem.  One is to configure your socket.io connection transports to only use WebSockets, then you can always assume you've got the right connection.  The client code would look something like this:
const feathers = require('feathers/client')
const socketio = require('feathers-socketio/client');
const hooks = require('feathers-hooks');
const io = require('socket.io-client');
const authentication = require('feathers-authentication/client');

const socket = io('http://api.my-feathers-server.com', {
  transports: ['websocket']
});
const app = feathers()
  .configure(hooks())
  .configure(socketio(socket));
  .configure(authentication({ storage: window.localStorage }));

app.authenticate();

The downside to the above is that it won't work with clients that aren't compatible with WebSockets.  So, the alternative would be to make sure you authenticate after the transport upgrade has occurred:
const feathers = require('feathers/client')
const socketio = require('feathers-socketio/client');
const hooks = require('feathers-hooks');
const io = require('socket.io-client');
const authentication = require('feathers-authentication/client');

const socket = io('http://api.my-feathers-server.com', {
  transports: ['websocket']
});
const app = feathers()
  .configure(hooks())
  .configure(socketio(socket));
  .configure(authentication({ storage: window.localStorage }));

socket.io.engine.on('upgrade', function(transport) {
  console.log('transport changed');
  app.authenticate();
});

See this Stack Overflow answer for more examples.

@marshallswain @daffl maybe we should do this inside the client itself.

I think so.  It's something that you would want in every app, I think.

Ya I agree. Going to change the title on this.

Issue moved to feathersjs/feathers-authentication-client #4 via ZenHub


Add CDNJS badge in README.md

This badge will show the latest version of this library hosted in CDNJS!

Thanks! :+1:


Are Arkham Comments working with Meteor 1.3?

I apologize for opening this "issue" as it's probably just something I'm doing wrong. When I install this package all I get is the UI without any functionality. I created a test install just for Arkham, installed accounts-password and accounts-ui, pasted in...

    {{> commentsBox id=documentId}}

...and it's just an input box that doesn't do anything. I'm properly logged in with accounts but Arkham doesn't work.

You need to make sure that that some data is bound to the template in which you add the div with the comment-section class. Comment objects need to be related to another js data object.  If the div is added without any context, then documentId is null.
Therefore, it might not work if you don't have any object to which the comments are related. Maybe that's the reason why they don't appear.
_id in the example below is the primary key of the data object to which the comments refer. For example a blog post, a photo or whatever.  _id is defined in my template in which I add the comment section.

		{{> commentsBox id=_id}}

@DominikButz
In lib/collections/comments.js
// Create 'comments' database
ArkhamComments = new Mongo.Collection('arkham_comments');

// Allow logged in users access to 'comments' database
ArkhamComments.allow({
  insert: function(userId, doc) {
    return !!userId;
  },
  update: function(userId, doc) {
    return !!userId;
  }
});

In server/publications.js
//Comments database
Meteor.publish('arkham_comments', function() {
  return ArkhamComments.find();
});

//Single comments
Meteor.publish('single_arkham_comment', function(commentId) {
  return ArkhamComments.find({_id: commentId});
});

And finally:
// Subscribe to "comments" collection
Meteor.subscribe('arkham_comments');

I have also added both the insecure and autopublish packages with no luck. What is "documentId" supposed to be? I only somewhat know what I'm doing.

I'm not going to lie... only about 20% of what you said makes any sense to me.

I removed autopublish and subscribed to articleComments.
//server:
Meteor.publish('articleComments', function(docId) {
return Comments.get(docId);
});
// client:
Template.[templateName].onCreated(function() {
Meteor.subscribe('articleComments', Session.get('documentId'));
});
I set the documentId (or whatever you call it) when routing to my detail view (Session.set('documentId', this.params._id).
Still, I can see all comments but not the replies to comments...

@DominikButz I'm going to install meteor 1.2 and see if arkham comments work with that. I'll report back what happens.

Thanks, let me know if it is related to the Meteor version.

Closing as it works with the latest Meteor release <IP_ADDRESS>.


ci: lint and format with stylua

Since this repo is community contributed, many of the stylistic choices differ between users (making things look a bit messy). Additionally, some lua syntax (calling functions with table, dropping parens) can be confusing for new contributors. I feel weird nitpicking style (and it's time consuming) the diff LGTM, so I think this will reduce our maintenance burden going forward.

my bot, I feel so betrayed 😢


sizeToFit won't work if you set width (initial load)

requirements.txt:
dash-ag-grid==2.0.0a3
dash==2.8.1
Code to reproduce the error:
import dash_ag_grid as dag
import dash
from dash import html

app = dash.Dash(__name__)

columnDefs = [
    {"headerName": "Make", "field": "make", "width": 90},
    {"headerName": "Price", "field": "price", "width": 30},
]

rowData = [
    {"make": "Toyota", "model": "Celica", "price": 35000},
    {"make": "Porsche", "model": "Boxter", "price": 72000},
]

app.layout = html.Div(
    [
        dag.AgGrid(
            id="input",
            columnSize="sizeToFit",
            columnDefs=columnDefs,
            rowData=rowData,
        ),
    ]
)

if __name__ == "__main__":
    app.run_server(debug=True)

Output

Hi @Thuener
I'm not sure that's a bug.  If you set the width on the column level  it will override the sizeToFit  You could try changing it to minWidth

If you need to have the columns be proportional to the width, please use flex instead of width

Just as FYI, it worked perfectly. Thanks!


[Ubuntu 20] warden env up results in invalid argument

I followed the warden manual on how to setup and install a magento 2 project (https://docs.warden.dev/environments/magento2.html).
warden svc up starts the core without any issues. I created the example project as described in manual and tried to start the env with warden env up resulting in the following error:
Connecting traefik to exampleproject_default network
Connecting tunnel to exampleproject_default network
Connecting mailhog to exampleproject_default network
Removing exampleproject_php-fpm_1
Starting exampleproject_rabbitmq_1         ... done
Starting exampleproject_db_1               ... done
Starting exampleproject_elasticsearch-hq_1       ... done
Starting exampleproject_elasticsearch_1          ... done
Starting exampleproject_redis_1                  ... done
Recreating 3b9984dee1e8_exampleproject_php-fpm_1 ... error

ERROR: for 3b9984dee1e8_exampleproject_php-fpm_1  Cannot start service php-fpm: error while creating mount source path '/host_mnt/home/navin/Desktop/bos-local': chown /host_mnt/home/navin/Desktop/bos-local: invalid argument

ERROR: for php-fpm  Cannot start service php-fpm: error while creating mount source path '/host_mnt/home/navin/Desktop/bos-local': chown /host_mnt/home/navin/Desktop/bos-local: invalid argument
ERROR: Encountered errors while bringing up the project.

I havn't modified project's .env. I also purged all docker data and tried again without success.
What can I do?
Ubuntu 20.20
Docker: 20.10.16
docker-compose 1.29.2

Fixed by using another path on the host system (/opt in my case). This works, for whatever reason..


Don't Merge! (I need help compiling it)

I have tried as much as a I may but I cannot get sh4 cross toolchains working as much as I want. I wanted to ask you for a favour, if you could compile this branch (i.e. run make) containing gameboy color support and send me the zip file.

Never mind, it doesn't work


Invidious integration

idk if its possible but if it is possible u can add it heres invidious https://github.com/iv-org/invidious

I'm not sure what an integration from our end would look like. Indivuous may be interested in supporting the lbry:// protocol or Odysee content.


[BUG] Authors not being displayed

Describe the bug 描述你遇到的错误
After adding authors files on the "/data/authors" folder they cannot being accessed and not listed on the "authors" page.
Even with changing the "/content/authors" index files to underscore like the exampleSite did not work.
Example:
Code:

https://github.com/HabitsWay/blog/tree/main/data/authors

Website:

https://blog.habitsway.com/authors/angie
https://blog.habitsway.com/authors/

Please could you check it out? I'm not sure if is a problem with my project or a bug.
Many thanks in advance.
Expected behavior 期待的行为
The expected behavior is to display all the authors on the authors section and the individual author pages.
Screenshots 屏幕截图
No response
Build Environment 构建环境
No response
Preview Environment 预览环境

OS: MacOS
Browser: Chrome

Additional Information 补充信息
No response

Add
[taxonomies]
author = "authors"

to your config.toml

@HEIGE-PCloud many thanks for your quick response!


v2.1.9 build causes some errors

$ go build
# github.com/bsm/sarama-cluster
./config.go:132: c.Config.Metadata.Full undefined (type struct { Retry struct { Max int; Backoff time.Duration }; RefreshFrequency time.Duration } has no field or method Full)
./config.go:134: c.Config.Metadata.Full undefined (type struct { Retry struct { Max int; Backoff time.Duration }; RefreshFrequency time.Duration } has no field or method Full)
./consumer.go:825: c.client.config.Config.Metadata.Full undefined (type struct { Retry struct { Max int; Backoff time.Duration }; RefreshFrequency time.Duration } has no field or method Full)

sorry, My mistake, I use the wrong version of github.com/Shopify/sarama


Nextra3, can't imported shared .mdx if it has a reference to Callout.

In Nextra3, it appears I can no longer do the following:
pages/governance/vendor-engagement.mdx:
import DomainRegistrationSnippet from "../../snippets/domain-registration.mdx";
snippets/domain-registration.mdx:
import { Callout } from "nextra-theme-docs";

<Callout type="warning" emoji="⚠">
            Some Text Goes Here
</Callout>

Some more text here...

This used to work in Nextra2, but now in Nextra3 I get:
Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.

ChatGPT sent me down a rabbit hole of outdated information, so here I am.   This is likely not a bug, but not readily apparent "breaking change documentation".

TLDR; How do I reference a shared snippet that has calls to other Nextra components, etc. within Nextra 3?

https://the-guild.dev/blog/nextra-3?utm_source=nextra.site&utm_campaign=blog_page&utm_content=blog_link#steps-callout-tabs-cards-and-filetree-exports-were-removed
Please refer to the content at the link above!
In the “Nextra 2 -> 3 Migration Guide,” under item 2, you can see that the import method has changed.
It no longer imports from nextra-theme-docs but instead from nextra/components.
It might be helpful to review the official documentation to confirm how to call it correctly.
https://nextra.site/docs/guide/built-ins/callout#usage
import { Callout } from 'nextra/components'

<Callout type="info" emoji="ℹ">
  Today is Friday.
</Callout>

@oobg thanks!

Thanks.   I will review this after I "return to the office" in the new year.


unable to send email and generate pdf

I am working on A project with Medplum and I am facing some issues in Medplum.
I am using MedplumClient for fetching data from the Medplum.
1.When I generate report pdf with medplum's method  makePdf I am facing an error which is pdfgeneration is disabled.
2. When I send email by using sendEmail method I am facing cors error.
3. Unable to register new patient showing an error oper registration is disabled.


Sortable "paused" when dragging a svg icon on mobile

only when dragging start on a svg (event.target = svg) (no matter there's a handle button ).
only when dragging too fast.
only on mobile.

Me too , when I remove svg element ,  it works well 。

Seems like a mobile browser issue to me. If you post a jsbin I can look into it, but for now I will close this.


Pass shortcode atts to the handler

Description
In 4.0.0 the "[auth0]" shortcode no longer accepts parameters (i.e "redirect_to"), this looks like it was an accidental bug introduced during a rewrite.
Testing
Verified locally that the redirect_to shortcode now works. Also compared to v3.11 to be sure that this used to be supported.

Sounds great. Thanks @joshcanhelp!

❯ composer test
> "vendor/bin/phpunit"
Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.2.28 with Xdebug 2.6.0
Configuration: ~/Sites/wp-auth0/wp-content/plugins/auth0/phpunit.xml.dist

...............................................................  63 / 409 ( 15%)
............................................................... 126 / 409 ( 30%)
............................................................... 189 / 409 ( 46%)
............................................................... 252 / 409 ( 61%)
............................................................... 315 / 409 ( 77%)
............................................................... 378 / 409 ( 92%)
...............................                                 409 / 409 (100%)

Time: 23.19 seconds, Memory: 48.25MB

OK (409 tests, 1561 assertions)

~/Sites/wp-auth0/wp-content/plugins/auth0 4.0_fix_shortcode_atts 24s
❯ snyk test

Testing ~/Sites/wp-auth0/wp-content/plugins/auth0...

Organization:      auth0-sdks
Package manager:   composer
Target file:       composer.lock
Project name:      auth0/wp-auth0
Open source:       no
Project path:      ~/Sites/wp-auth0/wp-content/plugins/auth0
Licenses:          enabled

✓ Tested 4 dependencies for known issues, no vulnerable paths found.

Next steps:
- Run `snyk monitor` to be notified about new related vulnerabilities.
- Run `snyk test` as part of your CI/test.


[gcp][feat] Use next gen collector

Description
WIP
To-Dos

[ ] Add test coverage for new or updated functionality
[ ] Lint and test with tox
[ ] Document new or updated functionality (someengineering/resoto.com#XXXX)

Issues Fixed

Fixes #XXXX

Code of Conduct
By submitting this pull request, I agree to follow the code of conduct.

collection result of current collector:
Sent 7908/7908, 3868 nodes, 4040 edges and 0 deferred edges in 0.1121s
Collect ran for 207 seconds
> search is(gcp_resource) | count kind
gcp_project: 1
gcp_bucket: 1
gcp_disk_type: 1
gcp_network: 1
gcp_service: 1
gcp_disk: 2
gcp_machine_type: 2
gcp_instance: 2
gcp_service_sku: 3
gcp_region: 37
gcp_subnetwork: 38
gcp_route: 39
gcp_zone: 112
gcp_quota: 3626
total matched: 3866
total unmatched: 0

collection result of new collector:
Sent 11335/11335, 5579 nodes, 5756 edges and 0 deferred edges in 0.1890s
Collect ran for 162 seconds
> search is(gcp_resource) | count kind
gcp_project: 1
gcp_disk_type: 1
gcp_network: 1
gcp_disk: 2
gcp_instance: 2
gcp_machine_type: 2
gcp_sku: 4
gcp_firewall: 4
gcp_region: 37
gcp_subnetwork: 38
gcp_route: 39
gcp_zone: 112
gcp_accelerator_type: 212
gcp_interconnect_location: 294
gcp_sql_flag: 562
gcp_node_type: 598
gcp_quota: 3626
total matched: 5535
total unmatched: 0

To Do:

gcp_bucket isn't implemented yet
tagging and deletion methods
...

quick update: migrated the update_tag logic to the new collectors architecture using the GcpClient. So far, works on disks but haven't tested other types of resources yet.

Update: Can now tag and delete all resources. Buckets get emptied before deletion

Couple of issues after running first test collects:

There are currently thousands of objects below the project. Ideally there should only be regions under a project.
In AWS we add the "global" region and put every global object there, let's do the same in GCP to not pollute the project space. The old collector does that too for routes, but it's a bad behaviour that we should not continue, esp. as we get more kinds.

There is a resource kind "gcp_sql_flag" that shouldn't be a resource. When a DB is making use of certain flags we can add them as a nested property to that DB.
It makes no sense to have something like "cloudsql.enable_pglogical" or "commit_delay" which are essentially individual DB settings as a resource in the graph.

I'm getting thousands of log lines like
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-highmem-48 (902048): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-highmem-64 (902064): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-highmem-8 (902008): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-highmem-80 (902080): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-highmem-96 (902096): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-standard-128 (901128): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-standard-16 (901016): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-standard-2 (901002): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']
23-04-13 14:30:53|resotoworker-gcp|DEBUG|46803|gcp_testaccount-e2e  Unable to determine SKU(s) for gcp_machine_type n2-standard-32 (901032): ['services/6F81-5844-456A/skus/4AA7-6169-3F12 (4AA7-6169-3F12)', 'services/6F81-5844-456A/skus/AFF5-E97F-63FD (AFF5-E97F-63FD)', 'services/6F81-5844-456A/skus/CDAA-A05D-DE0B (CDAA-A05D-DE0B)', 'services/6F81-5844-456A/skus/F902-A6F0-7D1B (F902-A6F0-7D1B)']

Upon completion I get a couple of hundred errors about duplicate checksums, like
23-04-13 14:30:58|resotoworker|ERROR|46677|resoto.worker-collector_1  Duplicate checksum 6eCNl9_k6jLEPJDx82x0XA for node gcp_node_type m1-node-96-1433<PHONE_NUMBER>00) in graph!
23-04-13 14:30:58|resotoworker|ERROR|46677|resoto.worker-collector_1  Duplicate checksum LZoAjxS4SfK75ae8jE_NtA for node gcp_node_type m2-node-416-11776<PHONE_NUMBER>60000) in graph!
23-04-13 14:30:58|resotoworker|ERROR|46677|resoto.worker-collector_1  Duplicate checksum 6eCNl9_k6jLEPJDx82x0XA for node gcp_node_type m1-node-96-1433<PHONE_NUMBER>00) in graph!

I haven't yet compared the graph structure of the new vs. the old collector.

Upon completion I get a couple of hundred errors about duplicate checksums, like

Thats great feedback, thanks!
After looking into it for a few minutes it seems that we're correctly collecting nodeTypes, but linking those to regions/zones doesn't work as expected at this point. The undefined zone name then leads to the duplicate object and checksum warnings.
[nt.link.split('/')[8] for nt in collector.graph.search("name", "m1-node-96-1433") if nt.zone().name == "undefined"]
['us-east1-a', 'europe-west5-c', 'europe-west5-b', 'europe-west5-a', 'us-east7-b', 'us-east7-c']

Those are the zones that we get zonal nodeTypes for, but arent collected by resoto yet. Looking into the zone collection logic now - might be something with "hidden" or deactivated zones, based on project (i.e. some projects can't use us-east-1a)

Those are the zones that we get zonal nodeTypes for, but arent collected by resoto yet. Looking into the zone collection logic now - might be something with "hidden" or deactivated zones, based on project (i.e. some projects can't use us-east-1a)

Those zones are indeed simply not there for all projects. By removing nodeType nodes which aren't used by more specific resources, we should get rid of this problem, so I've added them to the node-removal that we already have in place for machineTypes and so on

There are currently thousands of objects below the project. Ideally there should only be regions under a project.
In AWS we add the "global" region and put every global object there, let's do the same in GCP to not pollute the project space. The old collector does that too for routes, but it's a bad behaviour that we should not continue, esp. as we get more kinds.

What kind of resources are those? Looking at a list of non-regional resources that I compiled back in March thats likely:
gcp_firewall, gcp_interconnect_location, gcp_network, gcp_service, gcp_sku, gcp_route, gcp_sql_flag, or?

For gcp_service and  gcp_sku we should even consider pulling them upwards under the Gcp cloud resource, as those are not project specific but global and shared across all Gcp customers and projects.
gcp_sql_flag are no longer collected as per another commit earleir today.
gcp_firewall, gcp_interconnect_location, gcp_network and gcp_route - Looking into linking them to a new global region now.

Note:
There is also firewallPolicies - which we arent collecting yet at all - that is not project specific, but one level up on a GCP organisation. I'd consider this one out of scope for this PR.

With 3b6394a15e353e67d02f7ec5a246ef3119113403, all non-zonal and non-regional resources are connected to a new (fake) global region resource, instead of the project.

@lloesche looking for more feedback as soon as you're able to re-run the collection on those projects please 😄

Nice, I just tested and looks pretty good.
The region global is not fake as far as Google is concerned. For many services it's part of the identifier. E.g. projects/{{project}}/global/networks/{{name}} for networks.

There is also firewallPolicies - which we arent collecting yet at all - that is not project specific, but one level up on a GCP organisation. I'd consider this one out of scope for this PR.

Are they related to the gcp_firewall resource that we are collecting? If yes, we should add them if it's not a lot of effort. If they are unrelated then I agree, let's skip them for now.
I still got a few duplicate checksums but a lot fewer than previously.
The global region seems to exist multiple times in the graph.
23-04-20 15:35:50|resotoworker|ERROR|3768341|resoto.worker-collector_1  Duplicate checksum Zt5JR3Fb0eFCJkwdnng-XQ for node gcp_region global in graph!

We must make sure that global is only being instantiated once per project. And that object instance then reused everywhere. If we create the same GcpRegion twice, even if it has the same id it's handled as a unique new object.
And then gcp_accelerator_type exists multiple times with the same checksum.
23-04-20 15:35:52|resotoworker|ERROR|3768341|resoto.worker-collector_1  Duplicate checksum ZQjsfBUlSEQABTmFWJ369w for node gcp_accelerator_type nvidia-tesla-p4 (10010) in graph!
23-04-20 15:35:52|resotoworker|ERROR|3768341|resoto.worker-collector_1  Duplicate checksum QbXgOKqIIqr6VSWbgdVJYw for node gcp_accelerator_type nvidia-tesla-p4-vws (10012) in graph!

Regarding GcpSqlFlag, we do want to collect them, but not as distinct resources. Instead they are database flags for Cloud SQL DB instances. They are attributes of the DB, sort of like the color of your car is an attribute of the car and not a distinct resource.  You buy a red car, if that makes sense.
In the same way we should collect the flags associated with a Cloud SQL instance.
I still have 421 times "Unable to determine SKU" in a single project. Is that expected?
I noticed that the id field sometimes contains the GCP internal id. In the past we said that in our unified data model id contains the string that you require to talk to the API. Whereas name contains whatever name the user chose.
reported:
  id: '2543460420143619458'
  tags: {}
  name: konvoy-e2e-bastion-1681957679-bastion
  ctime: '2023-04-20T02:29:02Z'
  instance_cores: 4
  instance_memory: 16
  instance_type: n2-standard-4
  instance_status: running

In this case, the internal Google ID is practically useless for the user. And what's currently in name is the id that you use to talk to the Google APIs.
To be consistent we should have ID contain the ID that the API expects and maybe have a google_id or something like that where we can map the google internal ID.
@aquamatthias @anjafr thoughts?

Regarding GcpSqlFlag, we do want to collect them, but not as distinct resources. Instead they are database flags for Cloud SQL DB instances. They are attributes of the DB, sort of like the color of your car is an attribute of the car and not a distinct resource.  You buy a red car, if that makes sense.
In the same way we should collect the flags associated with a Cloud SQL instance.

I had looked into this previously and surprisingly, there is no way to link the flag to a database instance, from either direction.

There is also firewallPolicies - which we arent collecting yet at all - that is not project specific, but one level up on a GCP organisation. I'd consider this one out of scope for this PR.

Are they related to the gcp_firewall resource that we are collecting? If yes, we should add them if it's not a lot of effort. If they are unrelated then I agree, let's skip them for now.

Those policies can't be collected by the current scope of the project collector, because they're on org level. So yes, that would be a bigger  chunk of work and out of scope for parity with the existing collector, that also doesn't do any of that. See more details in our trello. Just mentioned it for sake of completeness 😄

Regarding GcpSqlFlag, we do want to collect them, but not as distinct resources. Instead they are database flags for Cloud SQL DB instances. They are attributes of the DB, sort of like the color of your car is an attribute of the car and not a distinct resource.  You buy a red car, if that makes sense.
In the same way we should collect the flags associated with a Cloud SQL instance.

I had looked into this previously and surprisingly, there is no way to link the flag to a database instance, from either direction.

Isn't that simply a new property on the database instance?

Regarding GcpSqlFlag, we do want to collect them, but not as distinct resources. Instead they are database flags for Cloud SQL DB instances. They are attributes of the DB, sort of like the color of your car is an attribute of the car and not a distinct resource.  You buy a red car, if that makes sense.
In the same way we should collect the flags associated with a Cloud SQL instance.

I had looked into this previously and surprisingly, there is no way to link the flag to a database instance, from either direction.

Isn't that simply a new property on the database instance?

Sql Flags are referencing database versions, not instances - so, no 🤷🏻♀

The global region seems to exist multiple times in the graph.

fixed

And then gcp_accelerator_type exists multiple times with the same checksum.

fixed

I still have 421 times "Unable to determine SKU" in a single project. Is that expected?

This can only come from GcpDiskTypes - but I can't reproduce on my project

Isn't that simply a new property on the database instance?

Sql Flags are referencing database versions, not instances - so, no 🤷🏻♀
ETA: I want to point out that even in my small project, without any database instances, we collected over 500 sql flags (see first PR comment). Those are all just some sort of default flag, it's just clutter, really.

Yes, SQL flags are essentially just a dictionary on a DB instance with the flag itself as key and flag value as value. We could even flatten it down to be a string as key and union of base types as value. When you create a new DB instance you  configure its flags and flag values like so:

I just ran a collect against our own account and created a MySQL DB instance called testdb with some random SQL flag flush_time on it. First of all, I see the flags when I use gcloud sql instances describe testdb on the CLI:
  dataDiskSizeGb: '100'
  dataDiskType: PD_SSD
  databaseFlags:
  - name: flush_time
    value: '3600'
  deletionProtectionEnabled: true
  insightsConfig: {}

So I would think the API returns them? Or do we already collect them?
I was trying to find out the later but the collector died with the following message:
23-04-25 13:33:18|resotoworker-gcp|ERROR|94144|gcp_inbound-axon-320811  [gcp:inbound-axon-320811] Failed to collect project: Could not match the object of type "dict" to any type of the Union: GcpSqlSettings, NoneType

You can replicate this by creating a DB instance and then testing the collector code.

@lloesche hang on - it's just hidden in plain sight. Our model of GcpDatabaseInstance has a property settings, which in turn contains database_flags - so I think we're good, actually 😅

Yes I think so. That's what I half suspected and wanted to test when the collector had some issue mapping the value it encountered from the API.

I noticed that the id field sometimes contains the GCP internal id. In the past we said that in our unified data model id contains the string that you require to talk to the API. Whereas name contains whatever name the user chose.

fixed (I think)
I looked at where we individually get resources, and in all cases they use the human readable name, not the id. So I went ahead and made that the default. Nothing broke 🤞🏻

Some more collect issues:

lot's of None's - the problematic one is id but maybe others as well

23-05-03 13:59:39|resotoworker-gcp|DEBUG|3659678|gcp_braided-flow-431_1  Can not deserialize json into class GcpSqlUser: {'id': None, 'tags': {}, 'name': None, 'ctime': None, 'description': None, 'link': None, 'label_fingerprint': None, 'deprecation_status': None, 'dual_password_type': None, 'etag': '2e23234bc26e1b2ac274ccd5a0bd60d5ae5a92bf4ec244674073327d2a23a69e', 'host': 'cloudsqlproxy~%', 'instance': 'review-board', 'password': None, 'password_policy': None, 'project': 'braided-flow-431', 'sqlserver_user_details': None, 'type': None}. Error: NoneType cannot be deserialized into str

Some cyclic dependency

23-05-03 13:50:50|resotoworker|ERROR|3606164|collector_gcp  Graph of plugin gcp is not acyclic - ignoring plugin results. Cycle (<built-in method join of str object at 0x7f42629ee4f0>, ['EdgeType.default: gcp_target_vpn_gateway mcd-vpn-->gcp_forwarding_rule mcd-vpn-rule-udp500', 'EdgeType.default: gcp_forwarding_rule mcd-vpn-rule-udp500-->gcp_target_vpn_gateway mcd-vpn'])
23-05-03 13:50:50|resotoworker|ERROR|3605756|resoto.worker-collector_1  Skipping invalid collector graph <class 'NoneType'>

If someone doesn't use K8S (or any other API) we shouldn't abort the entire project.

[gcp:eng-mesosphere-dcos] Failed to collect project: <HttpError 403 when requesting https://container.googleapis.com/v1/projects/eng-mesosphere-dcos/locations/-/operations?alt=json returned "Kubernetes Engine API has not been used in project<PHONE_NUMBER>18 before or it is disabled

Some more collect issues:

lot's of None's - the problematic one is id but maybe others as well

23-05-03 13:59:39|resotoworker-gcp|DEBUG|3659678|gcp_braided-flow-431_1  Can not deserialize json into class GcpSqlUser: {'id': None, 'tags': {}, 'name': None, 'ctime': None, 'description': None, 'link': None, 'label_fingerprint': None, 'deprecation_status': None, 'dual_password_type': None, 'etag': '2e23234bc26e1b2ac274ccd5a0bd60d5ae5a92bf4ec244674073327d2a23a69e', 'host': 'cloudsqlproxy~%', 'instance': 'review-board', 'password': None, 'password_policy': None, 'project': 'braided-flow-431', 'sqlserver_user_details': None, 'type': None}. Error: NoneType cannot be deserialized into str

Some cyclic dependency

23-05-03 13:50:50|resotoworker|ERROR|3606164|collector_gcp  Graph of plugin gcp is not acyclic - ignoring plugin results. Cycle (<built-in method join of str object at 0x7f42629ee4f0>, ['EdgeType.default: gcp_target_vpn_gateway mcd-vpn-->gcp_forwarding_rule mcd-vpn-rule-udp500', 'EdgeType.default: gcp_forwarding_rule mcd-vpn-rule-udp500-->gcp_target_vpn_gateway mcd-vpn'])
23-05-03 13:50:50|resotoworker|ERROR|3605756|resoto.worker-collector_1  Skipping invalid collector graph <class 'NoneType'>

If someone doesn't use K8S (or any other API) we shouldn't abort the entire project.

[gcp:braided-flow-431_1] Failed to collect project: <HttpError 403 when requesting https://container.googleapis.com/v1/projects/braided-flow-431_1/locations/-/operations?alt=json returned "Kubernetes Engine API has not been used in project<PHONE_NUMBER>18 before or it is disabled

2. Some cyclic dependency

resolved

Some cyclic dependency

resolved

Now the dependency that resulted in the cyclic dependency is just deleted. Meaning it was wrong in the first place? Or is it missing now?

Now the dependency that resulted in the cyclic dependency is just deleted. Meaning it was wrong in the first place? Or is it missing now?

it was defined from both ends, I deleted one of the definitions only 👍🏻

I had this now run against D2iQ many times. With the latest changes it no longer throws any errors other than the "API not active" ones. We already talked about how we'll change the behaviour to skip resources for disabled APIs instead of skipping the project. But we will do so in a separate PR. I also have a gut feeling that we'll run into more issues on the delete side of things, but merging this into main will allow our GCP users to test the :edge image before we make it into a numbered release. We won't cut a release this Friday so we'll have 1.5 weeks of additional edge case testing.
So from my PoV this is good to merge.


Add workaround for highlighting matching brackets with matchparen

Rename tsxEscapeJs syntax type to tsxEscJs as a work around to
mismatching bracket highlighting.
Alias tsxEscJs to tsxEscapeJs to keep backward compatability for
styling.

The built in plugin matchparen (enabled by default on most systems)
highlights the matching paren when the cursor is over a bracket. The
matching relies on syntax types (when enabled) and is configured to
exclude any syntax types containing the words
string\\|character\\|singlequote\\|escape\\|comment from the matching
process.
This would cause a tsxEscapeJs region to mismatch highlighting for
that region and the overlapping regions bounded by braces.

Hi
Thanks for all your work on this project!
I realised that this pull request doesn't do a lot to highlight the bug it is addressing. Here are some screenshots to illustrate the different behaviour:
Mismatched - the cursor is over the brace that opens the method but the closing brace is detected as the end of the tsxEscapeJs region:

Fixed - the cursor is over the brace that opens the method and the matched brace is the one that closes the method:

The matching parenthesis functionality can be enabled / disabled using :DoMatchParen and :NoMatchParen.
The style for highlighting the matched parenthesis can be set using:
hi MatchParen cterm=none ctermbg=green ctermfg=blue

Thanks! I don't know matchparen is implemented via grepping syntax group! Thanks for pointing out!

It was a surprise for me too. It definitely wasn't the first place I looked
for why this was happening!
On Sat, 6 Oct 2018, 2:06 PM Herrington Darkholme<EMAIL_ADDRESS>wrote:

Thanks! I don't know matchparen is implemented via grepping syntax group!
Thanks for pointing out!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/HerringtonDarkholme/yats.vim/pull/85#issuecomment-427544052,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArHOsd9XN-gX5RdLJuNnE8ztiunXzlpks5uiCxWgaJpZM4XJYPN
.


fix: added replicaCount to lapi and switch to traefik.io

This enables LAPI to be HA.
Also: traefik.containo.us is being deprecated in favor of traefik.io

Hi @aleksasiriski,
You have two subjects in this PR, the first one about replicaCount (HA) is already handle in this PR : https://github.com/crowdsecurity/helm-charts/pull/112
This was also a change to be done in crowdsec https://github.com/crowdsecurity/crowdsec/pull/2506 now merged.
So could you please remove your changes about the HA and keep only the changes on the apiVersion for the bouncer.


Add option to retrieve credentials without git sync

Summary
We use gopass in direnv to inject credentials. An option to ommit git sync in order to speed up the direnv script would be great :-)
Steps To Reproduce
See https://repo.prod.meissa.de/meissa/build#try-out for an example.

I assume execution in direnv counts as interactive & terminal as this is executed in bashrc context.
I will try out the config and hope to observe a speedup in execution.


manifest mode checks binary cache before vcpkg_installed for new CMake build directory

Describe the bug
I am not sure if this is a bug or by design, so I am opening this issue to clarify. I tried caching the vcpkg_installed directory on GitHub Actions using vcpkg in manifest mode. I have a step in the GitHub Actions workflow to run vcpkg install before the CMake configure step. With vcpkg install, it first searches vcpkg_installed, then the binary cache. However running vcpkg from the CMake toolchain file during CMake configuration seems to do the opposite. In this case, vcpkg first checks the binary cache. If there is a cache miss, it rebuilds the package even if vcpkg install would not have.
Environment

OS: all
Compiler: all

To Reproduce
Steps to reproduce the behavior:

cmake -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake -S . -B build in a repository with a vcpkg.json manifest file
rm -r build ~/.cache/vcpkg (to simulate not caching these directories on CI)
ls vcpkg_installed to confirm packages are still installed
cmake -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake -S . -B build

Expected behavior
In step 4, vcpkg would not rebuild packages because they are already installed in vcpkg_installed.
Failure logs
❯ ls vcpkg_installed/x64-linux/lib
libavcodec.a   libavformat.a  libFLAC++.a  liblilv.a     libogg.a        libportmidi.a  libpostproc.a  libsndfile.a             libsoxr.a     libswresample.a  libvorbisenc.a   pkgconfig
libavdevice.a  libavutil.a    libFLAC.a    libmad.a      libopus.a       libportSMF.a   libsbsms.a     libsord.a                libsqlite3.a  libswscale.a     libvorbisfile.a
libavfilter.a  libexpat.a     libjack.a    libmp3lame.a  libportaudio.a  libporttime.a  libserd.a      libSoundTouchDll_x64.so  libsratom.a   libvorbis.a      libz.a
❯ cmake -G Ninja -DCMAKE_PREFIX_PATH=/home/be/local -DCMAKE_INSTALL_PREFIX=/home/be/local -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=~/sw/vcpkg/scripts/buildsystems/vcpkg.cmake -S . -B build
-- Found ccache: /usr/bin/ccache
Using conan: ON
-- Running vcpkg install
Detecting compiler hash for triplet x64-linux...
The following packages will be built and installed:
  * libflac[core]:x64-linux -> 1.3.3#5
  * libogg[core]:x64-linux -> 1.3.5
    libsndfile[core,external-libs]:x64-linux -> 1.0.31
  * libvorbis[core]:x64-linux -> 1.3.7#1
  * opus[core]:x64-linux -> 1.3.1#5
Additional packages (*) will be modified to complete this operation.

-- Using /home/be/sw/vcpkg/downloads/xiph-opus-5c94ec3205c30171ffd01056f5b4622b7c0ab54c.tar.gz
-- Cleaning sources at /home/be/sw/vcpkg/buildtrees/opus/src/2b7c0ab54c-f5f7ba8ec8.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/be/sw/vcpkg/downloads/xiph-opus-5c94ec3205c30171ffd01056f5b4622b7c0ab54c.tar.gz
-- Using source at /home/be/sw/vcpkg/buildtrees/opus/src/2b7c0ab54c-f5f7ba8ec8.clean
CMake Deprecation Warning at scripts/cmake/vcpkg_check_features.cmake:182 (message):
  calling `vcpkg_check_features` without the `FEATURES` keyword has been
  deprecated.

      Please add the `FEATURES` keyword to the call.
Call Stack (most recent call first):
  ports/opus/portfile.cmake:13 (vcpkg_check_features)
  scripts/ports.cmake:141 (include)

-- Found external ninja('1.10.2').
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/opus_x64-linux/lib/pkgconfig/opus.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/opus_x64-linux/debug/lib/pkgconfig/opus.pc
-- Installing: /home/be/sw/vcpkg/packages/opus_x64-linux/share/opus/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/be/.cache/vcpkg/archives/70/7098166afb6b05fbe045582ecc3d3e06e31a518eed4a05497621cf5227bc8498.zip
Building package opus[core]:x64-linux... done
Installing package opus[core]:x64-linux...
Installing package opus[core]:x64-linux... done
Elapsed time for package opus:x64-linux: 14.39 s
Starting package 2/5: libogg:x64-linux
Building package libogg[core]:x64-linux...
-- Using /home/be/sw/vcpkg/downloads/xiph-ogg-v1.3.5.tar.gz
-- Cleaning sources at /home/be/sw/vcpkg/buildtrees/libogg/src/v1.3.5-1a4243fef9.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/be/sw/vcpkg/downloads/xiph-ogg-v1.3.5.tar.gz
-- Using source at /home/be/sw/vcpkg/buildtrees/libogg/src/v1.3.5-1a4243fef9.clean
-- Found external ninja('1.10.2').
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libogg_x64-linux/lib/pkgconfig/ogg.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libogg_x64-linux/debug/lib/pkgconfig/ogg.pc
-- Installing: /home/be/sw/vcpkg/packages/libogg_x64-linux/share/libogg/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/be/.cache/vcpkg/archives/80/8021de4c10b0cfc68cf5dd7c70402fe7d8997f4cde6bcecaa2fb65e438a1ccc5.zip
Building package libogg[core]:x64-linux... done
Installing package libogg[core]:x64-linux...
Installing package libogg[core]:x64-linux... done
Elapsed time for package libogg:x64-linux: 3.866 s
Starting package 3/5: libvorbis:x64-linux
Building package libvorbis[core]:x64-linux...
-- Using /home/be/sw/vcpkg/downloads/xiph-vorbis-v1.3.7.tar.gz
-- Cleaning sources at /home/be/sw/vcpkg/buildtrees/libvorbis/src/v1.3.7-6c4ca9c573.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/be/sw/vcpkg/downloads/xiph-vorbis-v1.3.7.tar.gz
-- Applying patch 0001-Dont-export-vorbisenc-functions.patch
-- Applying patch 0002-Fixup-pkgconfig-libs.patch
-- Using source at /home/be/sw/vcpkg/buildtrees/libvorbis/src/v1.3.7-6c4ca9c573.clean
-- Found external ninja('1.10.2').
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/lib/pkgconfig/vorbis.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/lib/pkgconfig/vorbisenc.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/lib/pkgconfig/vorbisfile.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/debug/lib/pkgconfig/vorbis.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/debug/lib/pkgconfig/vorbisenc.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libvorbis_x64-linux/debug/lib/pkgconfig/vorbisfile.pc
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/be/.cache/vcpkg/archives/b2/b2cf9042767f4ea4eb911afe5c9ed0731199ad6c9c0b6c7d8fd7045c37bc175d.zip
Building package libvorbis[core]:x64-linux... done
Installing package libvorbis[core]:x64-linux...
Installing package libvorbis[core]:x64-linux... done
Elapsed time for package libvorbis:x64-linux: 5.901 s
Starting package 4/5: libflac:x64-linux
Building package libflac[core]:x64-linux...
-- Using /home/be/sw/vcpkg/downloads/xiph-flac-ce6dd6b5732e319ef60716d9cc9af6a836a4011a.tar.gz
-- Cleaning sources at /home/be/sw/vcpkg/buildtrees/libflac/src/a836a4011a-419595447f.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/be/sw/vcpkg/downloads/xiph-flac-ce6dd6b5732e319ef60716d9cc9af6a836a4011a.tar.gz
-- Applying patch uwp-library-console.patch
-- Applying patch uwp-createfile2.patch
-- Applying patch fix-compile-options.patch
-- Using source at /home/be/sw/vcpkg/buildtrees/libflac/src/a836a4011a-419595447f.clean
CMake Deprecation Warning at scripts/cmake/vcpkg_check_features.cmake:182 (message):
  calling `vcpkg_check_features` without the `FEATURES` keyword has been
  deprecated.

      Please add the `FEATURES` keyword to the call.
Call Stack (most recent call first):
  ports/libflac/portfile.cmake:31 (vcpkg_check_features)
  scripts/ports.cmake:141 (include)

-- Found external ninja('1.10.2').
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libflac_x64-linux/lib/pkgconfig/flac++.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libflac_x64-linux/lib/pkgconfig/flac.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libflac_x64-linux/debug/lib/pkgconfig/flac++.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libflac_x64-linux/debug/lib/pkgconfig/flac.pc
-- Installing: /home/be/sw/vcpkg/packages/libflac_x64-linux/share/libflac/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/be/.cache/vcpkg/archives/bb/bb80155f213126ba752f87712ab24b45eac51f7ee9f1d6a764e0b963c6fbf8a0.zip
Building package libflac[core]:x64-linux... done
Installing package libflac[core]:x64-linux...
Installing package libflac[core]:x64-linux... done
Elapsed time for package libflac:x64-linux: 12.12 s
Starting package 5/5: libsndfile:x64-linux
Building package libsndfile[core,external-libs]:x64-linux...
-- Using /home/be/sw/vcpkg/downloads/libsndfile-libsndfile-1.0.31.tar.gz
-- Cleaning sources at /home/be/sw/vcpkg/buildtrees/libsndfile/src/1.0.31-f71190aa0d.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /home/be/sw/vcpkg/downloads/libsndfile-libsndfile-1.0.31.tar.gz
-- Using source at /home/be/sw/vcpkg/buildtrees/libsndfile/src/1.0.31-f71190aa0d.clean
-- Found external ninja('1.10.2').
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
CMake Warning at scripts/cmake/vcpkg_configure_cmake.cmake:421 (message):
  The following variables are not used in CMakeLists.txt:

      PYTHON_EXECUTABLE

  Please recheck them and remove the unnecessary options from the
  `vcpkg_configure_cmake` call.

  If these options should still be passed for whatever reason, please use the
  `MAYBE_UNUSED_VARIABLES` argument.
Call Stack (most recent call first):
  ports/libsndfile/portfile.cmake:17 (vcpkg_configure_cmake)
  scripts/ports.cmake:141 (include)

-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libsndfile_x64-linux/lib/pkgconfig/sndfile.pc
-- Fixing pkgconfig file: /home/be/sw/vcpkg/packages/libsndfile_x64-linux/debug/lib/pkgconfig/sndfile.pc
-- Installing: /home/be/sw/vcpkg/packages/libsndfile_x64-linux/share/libsndfile/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/be/.cache/vcpkg/archives/fc/fcebb0c1a1b89f58fd29cd5888f133126714989e86c1f26b431cae5a1cbcbd25.zip
Building package libsndfile[core,external-libs]:x64-linux... done
Installing package libsndfile[core,external-libs]:x64-linux...
Installing package libsndfile[core,external-libs]:x64-linux... done
Elapsed time for package libsndfile:x64-linux: 27.47 s

Total elapsed time: 1.062 min

Additional context
I am trying to improve the run-vcpkg GitHub action for manifest mode.

@strega-nil-ms Can you please take a look?
Thanks.

This is intentional. The vcpkg_installed directory for CMake is in the generated build directory, so that vcpkg doesn't step on itself when configuring for different triplets.

@strega-nil
I think you assumed the following notion that is not explicit from the docs:

when using vcpkg install, the vcpkg_installed directory is sibling to vcpkg.json.
when using cmake to start vcpkg (that is, when vcpkg.json and the vcpkg's toolchain is being used) then the location is CMAKE_BINARY_DIR/vcpkg_installed.

Basically depending on how vcpkg is consumed, the target directory where all is going to be installed is different, leading to unexpected surprise to users.
It looks like the only way to avoid surprise, is to drive vcpkg to put everything in the directory that the user explicitly sets, for example by setting the environment variable VCPKG_DEFAULT_BINARY_CACHE to a valid location to store the installed bits.

setting the environment variable VCPKG_DEFAULT_BINARY_CACHE to a valid location to store the installed bits.

Just to avoid misunderstanding: the binary cache isn't the same as vcpkg_installed. It caches archives. It can contain a different set of packages than (installed in) vcpkg_installed.

the target directory where all is going to be installed is different, leading to unexpected surprise to users.

What surprises are you thinking of? The packages are relocatable, if not broken.

If you really want the vcpkg_installed directory to be at the top level, you can set it via the VCPKG_INSTALLED_DIR option to the triplet.

IIUC this doesn't mean to modify the triplet but to set the VCPKG_INSTALLED_DIR cmake parameter for the vcpkg cmake toolchain, which maps to the --x-install-root parameter of vcpkg.

the target directory where all is going to be installed is different, leading to unexpected surprise to users.

This was very surprising to me: https://github.com/lukka/run-vcpkg/pull/94#issuecomment-895922964

@dg0yt right, thanks for correcting me! I mentioned the wrong environment variable indeed: I meant the VCPKG_INSTALLED_DIR one as you correctly pointed out. let me know if my

I feel like the "steps to reproduce" and in particular the title didn't help to express the problem:
You are using two different modes, and you are surprised that they work independently.

vcpkg_installed in the source dir, created by vcpkg install next to vcpkg.json (aka Command Line Interface)
vcpkg_installed  in CMake's binary dir, created by cmake -DCMAKE_TOOLCHAIN_FILE=.../.../vcpkg.cmake ... (aka CMake Integration)

Command Line Interface makes no assumptions about CMake, and so it cannot know where your build directory will be.
For CMake Integration, it is not desirable to use the source directory for target artifacts: It needs write access to the source dir, and it doesn't scale easily for multiple builds from one source.
Maybe it would be helpful to make this more clear in the documentation.

Maybe it would be helpful to make this more clear in the documentation.

Yes, it would.


AttributeError: 'code' object has no attribute 'co_exceptiontable'

[16:03:03] The output path at output.py already exists, choosing alternative...                                                                                main.py:138
Found one: output_0.py                                                                                main.py:152
Parsing AST...                                                                                main.py:155
Executed transformer fstrToFormatSeq                                                                                main.py:116
Executed transformer intObfuscator                                                                                main.py:116
Executed transformer encodeStrings                                                                                main.py:116
Executed transformer renamer                                                                                main.py:116
Executed transformer replaceAttribSet                                                                                main.py:116
Executed transformer collector                                                                                main.py:116
Obfuscating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━  86% -:--:--
  File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/xor7/pyobf2/obfuscator/__main__.py", line 23, in <module>
    launch()
  File "/home/xor7/pyobf2/obfuscator/__main__.py", line 9, in launch
    main.main()
  File "/home/xor7/pyobf2/obfuscator/main.py", line 104, in main
    go()
  File "/home/xor7/pyobf2/obfuscator/main.py", line 157, in go
    compiled_ast = transform_source(compiled_ast)
  File "/home/xor7/pyobf2/obfuscator/main.py", line 115, in transform_source
    c_ast = t.transform(c_ast)
  File "/home/xor7/pyobf2/obfuscator/transformers.py", line 675, in transform
    return self.do_enc_pass(ast_mod)
  File "/home/xor7/pyobf2/obfuscator/transformers.py", line 651, in do_enc_pass
    main_loader = self.create_code_obj_loader(tn, compiled_code_obj)
  File "/home/xor7/pyobf2/obfuscator/transformers.py", line 389, in create_code_obj_loader
    collected_args = self.args_from_co(compiled_code_obj)
  File "/home/xor7/pyobf2/obfuscator/transformers.py", line 341, in args_from_co
    return [getattr(code, x) if not isinstance(x, tuple) else x[1] for x in self._ctype_arg_names]
  File "/home/xor7/pyobf2/obfuscator/transformers.py", line 341, in <listcomp>
    return [getattr(code, x) if not isinstance(x, tuple) else x[1] for x in self._ctype_arg_names]
AttributeError: 'code' object has no attribute 'co_exceptiontable'```

which py ver

simply an issue of skill