issues using /bin/plugin to install

Hi there,
I am currently trying to use both elasticsearch and couchbase db in a GKE kubernetes cluster. Part of this involves installing the couchbase elasticsearch plugin.
My k8s cluster currently has operational couchbase pods, and I have added elasticsearch as described at https://github.com/pires/kubernetes-elasticsearch-cluster.
The instructions I am using to install the plugin are described at http://developer.couchbase.com/documentation/server/4.0/connectors/elasticsearch-2.1/install-intro.html.
According to the above plugin instructions, I must install the plugin in each instance of ES. I enter the ES containers (sudo docker ps -a, get the ES container id, then sudo docker exec -ti <container id> sh) and then attempt to run the install instructions as described above.
When I first did this, I would get the following error:
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'HTTP_CORS_ALLOW_ORIGIN'.
I then set environment variables in each the yaml files for the ES deployments as follows:
name: HTTP_CORS_ENABLE
value: "true"
name: HTTP_CORS_ALLOW_ORIGIN
value: /https?://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/*/
I had hoped this would alleviate the issue, but then when I repeated the process I got the same error, except now the environment variable having an issue was NETWORK_HOST.
I set the env variable similarly, this time name: NETWORK_HOST and value: "_local_" as per the ES documentation.
Now I am being presented with the same error except now the variable having an issue is DISCOVERY_SERVICE.  When I check the link on https://github.com/pires/docker-elasticsearch-kubernetes regarding DISCOVERY_SERVICE, it brings me to another plugin page. When I try to install this plugin by running plugin install io.fabric8/elasticsearch-cloud-kubernetes/2.3.4, I get the same errors as when I try to install the ES-CB plugin.
Is there something I'm inherently doing wrong here? If so, what should I be doing to successfully install plugins?
Thanks for any and all help you can provide :)

This is one of the yaml files that I am currently using to try to get around this issue:

bump

I have to try and reproduce, otherwise I'm clueless.

Here's some information that I can give you for reproduction purposes:
Currently the google container engine cluster contains couchbase pods as described in https://github.com/couchbase/kubernetes.
It also contains an nginx pod that serves as the entrypoint for the cluster, as described at https://github.com/kubernetes/kubernetes/blob/master/examples/https-nginx/nginx-app.yaml.
I've added to the cluster 1 es-master pod, 1 es-client pod, and 1 es-data pod, as described in https://github.com/pires/kubernetes-elasticsearch-cluster. These pods are using elasticsearch v2.3.4 as described in their documentation.
I'm currently trying to run the couchbase elasticsearch plugin as described at https://github.com/couchbaselabs/elasticsearch-transport-couchbase. The version I'm running is version <IP_ADDRESS>, as this is the one that's supposed to be compatible with ES version 2.3.4.
The documentation at https://github.com/couchbaselabs/elasticsearch-transport-couchbase says to first install the ES Plugin on each node, using the command below:
/elasticsearch/bin # ./plugin install https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/<IP_ADDRESS>/elasticsearch-transport-couchbase-<IP_ADDRESS>.zip
However when I do this, I get the following error:
Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'DISCOVERY_SERVICE'
at org.elasticsearch.common.property.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:128)
at org.elasticsearch.common.property.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:81)
at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1179)
at org.elasticsearch.node.internal.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:131)
at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100)
at org.elasticsearch.plugins.PluginManagerCliParser.main(PluginManagerCliParser.java:61)
If you have any other questions regarding how to reproduce this, please let me know. Thanks for any and all assistance you can provide!

This looks to be a problem with the container not having the expected configuration settings for ES, not specifically a problem with the transport plugin. Are you able to successfully install other plugins? Which configuration file (elasticsearch.yml) is it actually using?

DISCOVERY_SERVICE is an environment variable populated in containers by Kubernetes itself. Obviously, said service needs to be created as per instructions in this repo. Did you do anything differently?

Hi guys, thanks for your responses.
@Branor, the yml files I've been using for this cluster can be found here: https://github.com/pires/kubernetes-elasticsearch-cluster, and the ES container they use is the one from this repo.
@pires I didn't realize that the instructions on this repo were mandatory. They only list the environment variables that may be used in replication controllers. I've since gone and tried to install the plugin shown at https://github.com/fabric8io/elasticsearch-cloud-kubernetes#kubernetes-pod-discovery, but this had issues as well.  Also looking at this specific repo, I didn't see instructions indicating that I had to create the discovery service.
I looked at the dockerfile that is used to create the ElasticSearch containers, and it runs the shell script that is supposed to instantiate the environment variables. Isn't this the step that's supposed to prevent these issues, as the image the containers are built on is supposed to have the environment variables set. I was under the impression that the environment variables could be changed in the replication controller if necessary, but that the image primarily took care of setting them up.

@tyh0 you're right. This repo is just used to build the image used in https://github.com/pires/kubernetes-elasticsearch-cluster.

So is it possibly the case that the image doesn't have the environment variables set correctly, or in such a way that the plugin is able to recognize them?

@tyh0 is the plug-in even compatible with Elasticsearch 2.3.x? Not from what I read http://developer.couchbase.com/documentation/server/4.0/connectors/elasticsearch-2.1/release-notes.html#elastic-release-notes!

@pires The link in your above comment is a bit outdated. The command to install the plugin ultimate hits this repo: https://github.com/couchbaselabs/elasticsearch-transport-couchbase
The versioning convention of the plugin is such that version <version#> of ES would correspond to version 2.<version#> of the plugin. Version 2.3.4 of ES therefore uses plugin version <IP_ADDRESS>. This is the version that I was attempting to install, and it definitely exists as can be seen at https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/tag/<IP_ADDRESS>.

@tyh0 I deleted my previous comment after I re-read your instructions. I am able to reproduce this and I'm still trying to figure out why this would happen. However, it seems that elasticsearch user, which is used to run Elasticsearch, doesn't have access to certain environment variables.

Oh awesome!  Do you have any idea of a timeframe for when this could be straightened away? I ask because I'm currently a co-op student, and my work term is ending at the end of the month. Ideally I really want to get this finished before I depart from the company.

I'm looking at it as we speak. Ping me back on monday if I didn't fix this by then.

Perfect. Thanks so much :)

I'm waiting on a new version of the Elasticsearch Kubernetes discovery plug-in in order to update to version 2.3.5 that will  include this fix.

@tyh0 it works.
$ kubectl exec -it es-master-3368190183-4ux7v sh

/ # sudo -E -u elasticsearch /elasticsearch/bin/plugin install https://github.co
m/couchbaselabs/elasticsearch-transport-couchbase/releases/download/<IP_ADDRESS>/elas
ticsearch-transport-couchbase-<IP_ADDRESS>.zip
-> Installing from https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/<IP_ADDRESS>/elasticsearch-transport-couchbase-<IP_ADDRESS>.zip...
Trying https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/<IP_ADDRESS>/elasticsearch-transport-couchbase-<IP_ADDRESS>.zip ...
Downloading .......................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/download/<IP_ADDRESS>/elasticsearch-transport-couchbase-<IP_ADDRESS>.zip checksums if available ...
Downloading .DONE
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * listen,resolve
* javax.security.auth.AuthPermission modifyPrincipals
* javax.security.auth.AuthPermission modifyPrivateCredentials
* javax.security.auth.AuthPermission setReadOnly
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed transport-couchbase into /elasticsearch/plugins/transport-couchbase

Has this been straightened away to work with ES 2.3.4 and plugin <IP_ADDRESS>? Or should I change the dockerfiles here to use 2.3.5?

Oh, I notice you've already done this. Thanks Pires! If I have any other issues with it, I'll open a new issue.

@tyh0 you need to use 2.3.5.


implement sampling

I initially thought that the sampling feature could be simplified simply by ensuring that filtering happens before sampling. However, I realise now that it is not as simple as that because a filter works on an individual basis, ie it considers just a single node at a time. But the sampler is concerned with a collection of items, so the 2 features don't work in the same logical domain. It is still true that filtering must happen before sampling because if we sample before filtering, the filter may result in fewer items that satify the sampling requirements; some of the filtered outr items could have been used in the sample set.
We could make the sampler work on an individual basis by doing the following

sampler should subscribe to decend and ascend events
the navigator should publish the number of children found of a directory (including both files and folder counts)
on descend: reset a counter. the counter denotes the number of items seen in the current directory
for each item encountered in the current directory, increment the counter. If we reach the samler count, then we start ignoring subsequent item by returning false from link.Next

The disadvantage of this approach is that when sampling is active, we still have to process the full result set for a folder and then ignore items that breach the sampling capacity. But what we could do is return skip-dir as the error, this will have the effect of short circuiting processing items that we know will be ignored as the sampler has already been satified.

A different approach has to be taken. The probem with the implementation of sampling in extendio which we trying to avoid here in traverse is that sampling and filtering are seen as 2 distinct operations and the sampler ran before filtering resulting in the potential of double filtering and the requirement for a look-ahead and caching facility. This is overkill and needs to be avoided.
We can improve things here by bringing together filetring and sampling. The solution consists of the following concepts:

filtering and sampling are integrated
apply the filter before the sampler
the guardian will not permit the filter plugin to register if sampling is also active, defined by the manifest function
the sampler plugin needs to modify the ReadDirectory hook, it will apply the filter to the read and restrict the result set according to sampling parameters
the items will flow through to the sampler link, at this stage they will be pre-filtered

I'm wondering if it is still true that filter should be exclusively handled by the sampler plugin as opposed to the filter plugin.

It might have been because of the 'last' sampler, but last is not helped by filtering in sampler
I think the sample based filtering was only considered if we couldf have applied the filter at source via the ReadDirectory plugin, but this has been confirmed for not being possible, because filters are node based and work one at a time, which is not compatible with the ReadDirectory hook

This means that we can revert back to leaving the filter plugin to apply the filter and let the sampler perform sampling. It also means that we can remove the manifest rule that prevents the filter plugin from registering it's link if sampling is active.

But we could just use the Child filter and apply that to ReadDirectory; so the sampler would use the Child filter instead of the Node filter.


Add Kobo Libra/Clara colour

This PR

fixes some trailing whitespaces
adds a colour toggle logic
adds Kobo Libra Colour
adds Kobo Clara Colour

Damn, I forked the wrong repo. Sorry, my bad. I will open the PR against he correct one.


Support EIP-712

EIP-712 Typed structured data hashing and signing (eth_signTypedData_v4) is required to perform token swap on pancakeswap.
Existing implementation using @noble/hashes https://github.com/Mrtenz/eip-712

An example on walletconnect website https://docs.walletconnect.com/advanced/multichain/rpc-reference/ethereum-rpc#eth_signtypeddata

How popular is EIP-712? Have you analyzed this?

EIP-712 is used in all token swaps on pancakeswap and uniswap. Some documentation:

https://support.uniswap.org/hc/en-us/articles/8120520483085-What-is-an-approval-transaction
https://docs.walletconnect.com/advanced/multichain/rpc-reference/ethereum-rpc

I think other decentralized exchanges use this mechanism too.
Soon we will have our own usage statistics.

Current solution https://github.com/CoinSpace/cs-evm-wallet/blob/master/lib/EvmWallet.js#L557-L583

EIP-712 has become the de facto standard for Typed Data Signing in Ethereum and EVM systems. This would definitely be a good addition to this package.
Ethers.js provides a good EIP-712 implementation but I was required to roll my own implementation of it for a private project using only noble cryptography packages without any dependency on ethers. It's essentially a drop-in replacement for the TypedDataEncoder class provided by ethers.
I'd be happy to re-purpose my implementation and raise a PR with it.
cc: @paulmillr


Fix unable to leave message android

This PR fix the bug that doesn't allow an Android user to leave a message after chatting with the initial chatbot.
The Android code has been aligned with the IOS code by removing the user identification when the chat starts.
Before
https://user-images.githubusercontent.com/11773070/140720063-702baaf6-58ff-4b39-9fd9-025c08e66ff5.mov
After
https://user-images.githubusercontent.com/11773070/140720474-3c05264b-157a-46b3-b066-7275ed7f88fe.mov

Opened in the wrong repo. Refers to https://github.com/pagopa/io-react-native-zendesk/pull/3


Timestamp not appearing in screenshot

I've set the timestamp value to true yet it still doesn't appear in my screenshots... Any idea why?

Which OS and PhantomJS version are you using? Do you use IITC? Could you post your config file (without login and password)?

This is a configuration file for Ingress Ice (github.com/nibogd/ingress-ice)                        # Insert your settings here (after '=' without spaces). Only first three settings (login, password and area link) are mandatory.

After modifying this file, save it and close the editor. Ice will start automatically.
[ice]
#> Your Google login and password
login=                                                                  password=                                                                            #> Your area link. To get it, visit http://ingress.com/intel, scroll the map to your location and press [LINK] at the top right corner. Copy that link and paste it here.
#> e. g. https://www.ingress.com/intel?ll=53.502328,49.41516&z=14                                     area=https://www.ingress.com/intel?ll=1.294717,103.782635&z=13                                        [ice-optional]                                                                                #> The delay between taking screenshots in milliseconds (1 second = 1000 milliseconds). Set to -1 to take a single screenshot and exit
delay=300000
#> Minimal and maximal portal levels to display                                                       minlevel=1
maxlevel=8                                                                                #> Width and height of the resulting screenshot                                                       width=500
height=500                                                                                #> Inject IITC? (true/false)
iitc=true                                                                                plugins=["https://secure.jonatkins.com/iitc/release/plugins/player-tracker.user.js"]                  # Read more at https://github.com/nibogd/ingress-ice/wiki/Loading-IITC-plugins                        pluginsConfig=[{"key":"plugin-linkshowdirection-mode","value":"Static near origin"},{"key":"YourKeyHere","value":""}]
#> Put a timestamp on your screenshots? (true/false)                                                  timestamp=true
#> Hide Resistance portals/links/fields, Enlightened portals/links/fields, links, fields respectively? (true/false)
hideRes=false
hideEnl=false                                                                                hideLink=false                                                                                hideField=false                                                                                # Picture format (png/jpg)                                                                            format=png                                                                                #> Log in timeout. Default is 10 000 (ms). If you have problems with signing in, increasing this may help                                                                                loginTimeout=10000                                                                                #> Directory (folder) where to save screenshots. Note the trailing slash. Relative to the current location. In linux, you can start it from any folder like this:
$ some-folder/ingress-ice/ingress-ice.sh                                                            directory=/screenshots/
#> Add timezone to timestamp? (true/false)
timezone=true
[cookies]
#> If set, will use cookies for authentication instead of login/password                              SACSID=
CSRF=
[Amazon-S3]
#> If set, will upload screenshots to Amazon S3 cloud
#> AWS S3 accessKeyId
S3Key=
#> AWS S3 secretKeyId
S3Secret=
#> AWS S3 bucket name
S3Bucket=
#> AWS S3 access control list
S3Alc=                                                                                #> Delete files after upload? (true/false)
S3Remove=false
[version]                                                                                version=43
~                                                                                ~                                                                                ~                                                                                ~                                                                                ~
I'm on CentOS 7, not sure what version of phantomJS, but it's dated April 18

Thank you, I'll look if i can do anything with it

so i took the default config file and did a re-run, and the timestamp came out. on a whim i decided to play with the screenshot width and height value, and i realised that it could be that my current setting of width=500 is causing the timestamp value not to appear.

@victaru yep, it must be that setting. Too narrow for a timestamp.
Glad that you've found out the problem! Closing the issue.


Replace stretchr/testify with self-written asserts

This PR removes the dependency github.com/stretchr/testify. It's used only in config_test.go and no need to include the whole powerful package for using only one function require.Equal.

Looks good to me and makes sense, @alexandear . Thanks again!


Fix LLJIT ignoring the CodeModel setting

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@conda-forge/llvmdev This PR adds a small patch for a bug (fixed in LLVM 19) where LLJIT would ignore the CodeModel setting specified by the user, leading to compilation failures for large JIT compiled modules.
The issue was originally reported here:
https://github.com/llvm/llvm-project/issues/88115
Here is the PR with the fix:
https://github.com/llvm/llvm-project/pull/90599

You did everything right with the rerender, but there's an unsolved issue with some components having been removed in the GHA images, which we need to reinstate here. If this situation persists, I guess we'll have to fix this in smithy at some point... In the meantime, I've just fixed it by hand.

@conda-forge/llvmdev is this ready to be merged?

Yeah, I was just waiting a little in case someone else has an opinion.

Thanks!


[Bug]: Pagination buttons are inaccessible in mobile.

Contact Details
<EMAIL_ADDRESS>What happened?
Notice the pagination buttons at the bottom. They are hidden behind the bottom menu of Obsidian.

What platform were you using?
Mobile
Version of the plugin
3.1.0
Relevant log output
No response
Are you using the latest version of the plugin in your Obsidian vault?

[X] I have verified that I am on the latest version

Are you check if there is a similar issue?

[X] I have verified that there are not similar issues

we have to support more kind of resolutions, thanks for the report !

You can create your own css snippet to support your resolution
I do not know the exact dimensions of your device so maybe you have to test some values like max height pixels or height percentage
This is my bet to fix it
@media only screen and (max-height: 768px) {
  .database-plugin__scroll-container {
    will-change: transform;
    width: 100%;
    height: 82%;
  }
}

You can create your own css snippet to support your resolution
I do not know the exact dimensions of your device so maybe you have to test some values like max height pixels or height percentage
This is my bet to fix it
@media only screen and (max-height: 768px) {
  .database-plugin__scroll-container {
    will-change: transform;
    width: 100%;
    height: 82%;
  }
}

I tried this. Didn't help since the pagination button is positioned absolutely at the bottom. Instead, it worked:
.database-plugin__pagination {
    width: fit-content;
    height: 40px;
    list-style: none;
    /* bottom: 1em; */
    /* position: absolute; */
    /* right: 0; */
    /* margin: 0; */
    padding: 0 10px 0 0;
    display: flex;
    column-gap: 2px;
    margin-top: 1em;
    margin-left: auto;
}

A good thing about this approach is it will work in all breakpoints. Even in situations like this on a desktop where the status bar overlaps the pagination buttons:

I'm glad you were able to find a solution to your resolution.
For other features I was testing your snippet but it doesn't fit with most and is completely lost in case of activating the search bar.
position: absolute is a must

You can go for a flex-based solution though. The main container can be a column flex with the height set with calc (e.g. calc(100vh - 50px), Then the table with flex: 1 to take maximum available space, and pagination bar just after that, taking the necessary space.

the problem is Obsidian's own scroll. Being able to hide the search bar and for better control of the overflow, it would not be possible to use flex without having a second scroll bar on the right when we show the bar or the size of the pagination is greater than the default.
I will wait until the official 1.1 on Mobile, maybe we are testing different sources

Another possible solution :
You can provide 4 buttons (> < << >>) in the ribbon for devices that don't have the status bar (phones). And hide these ribbon buttons on the other devices that have a status bar (desktop).
And if this solution works for you, then you can also transfer the pagination buttons to the status bar on desktop.

Once the Obsidian 1.1 is released on mobiles, I think this issue will be fixed with a css improvement of 3.1.0

Could you check if this issue was resolved with 3.2.3? I am not able to test it well on android

Could you check if this issue was resolved with 3.2.3? I am not able to test it well on android

I surely will once I get the next update of Obsidian.

With v3.2.3 of the plugin the mobile version is available again

No, this issue persists, so does the weird formatting issue which may rewrite my files with formatted text in desktop again.

I will work on this visual issue then
about the rewrite thing should not be happening with this flag https://github.com/RafaelGB/obsidian-db-folder/blob/e21e8da5d034c30140a4efd6d031fe60350b0fca/src/components/cellTypes/FormulaCell.tsx#L58

I will work on this visual issue then
about the rewrite thing should not be happening with this flag https://github.com/RafaelGB/obsidian-db-folder/blob/e21e8da5d034c30140a4efd6d031fe60350b0fca/src/components/cellTypes/FormulaCell.tsx#L58

The rewrite might not happen. However, the formula is running 4 times recursively. Weird.

It seems like links are not being rendered anymore.

Wikilinks in simplest form works. But wikilinks with custom text don't.

thanks for the testing, I will open an issue your your comment

Commit 8fd0f600
Could a possible solution. Now just for mobile devices you can do an extra bottom scroll to show the pagination


[Regression] No generated run_results.json for failed dbt runs in dbt=versionless

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
Pure Storage is reporting no run_results.json were generated for failed job runs in versionless. Their job is invoking dbt --fail-fast run and the issue started on Thu, 03 Oct 2024 11:43:48 UTC. They are using dbt API's to query for the artifacts/run_results.json file for dbt runs. Now the file is missing, which has broken their pipeline.
First instance of failing job: Run #330575227
2024-10-03 19:43:57.777911 (MainThread): 19:43:57  Running dbt... 2024-10-03 19:43:58.833166 (MainThread): 19:43:58  Registered adapter: snowflake=1.9.0-post8+eea98443b21f6f3bdc3d297b727a0a9424d54d7b 2024-10-03 19:43:59.024413 (MainThread): 19:43:59  checksum: 8d46bb42fef02cfa88ddca274a3572c32f4618d6a7cc8ae947cb24524d6f0352, vars: {}, profile: user, target: CI_PA_SANDBOX_137, version: 2024.10.258

debug-Run #330575227.log
Expected Behavior
Failing job run should have run_results
Steps To Reproduce
Unable to reproduce in sandbox account
Relevant log output
No response
Environment
- OS:
- Python:
- dbt: versionless
- snowflake=1.9.0-post8+eea98443b21f6f3bdc3d297b727a0a9424d54d7b

Which database adapter are you using with dbt?
snowflake
Additional Context
This is the existing thread about the issue: #ask-dev-metadata

@QMalcolm could you add the steps to reproduce here once you are able?


feat(deps): bump jw-epub-parser from 2.0.0 to 3.0.0

:tada: This PR is included in version 2.72.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Unable to delete a deployment

Hi,
First off, thanks for contributing the terraform operator. At this time we are currently evaluating using it as our main infrastructure provisioner. I have been able to use it successfully to create infrastructure, but am stuck on how to delete a deployment.
Some information. In my Terraform resource, I set these flags:
applyOnCreate: true
applyOnDelete: true
applyOnUpdate: true
ignoreDelete: false

I have waited more than 6 hours before deleting the terraform resource so that the pod that was created to deploy the infra has been deleted. What I have noticed is that the job has not been deleted. So when I delete the terraform resource, nothing happens and I see this in the log of the operator:
2021-02-08T11:56:04.460Z    INFO    controller_terraform    Checking if destroy task is done    {"Request.Namespace": "platform", "Request.Name": "iws-mbaum-test2"}                                            │
│ 2021-02-08T11:56:04.460Z    DEBUG    controller-runtime.manager.events    Normal    {"object": {"kind":"Terraform","namespace":"platform","name":"iws-mbaum-test2","uid":"bff70371-e0b6-4971-84e8-f1a55d48edbd" │
│ 2021-02-08T11:56:34.460Z    INFO    controller_terraform    Checking if destroy task is done    {"Request.Namespace": "platform", "Request.Name": "iws-mbaum-test2"}                                            │
│ 2021-02-08T11:57:04.463Z    INFO    controller_terraform    Checking if destroy task is done    {"Request.Namespace": "platform", "Request.Name": "iws-mbaum-test2"}

What I see earlier in the log is the following:
2021-02-08T11:56:04.444Z    INFO    controller_terraform    Secret iws-mbaum-test2-ssh-config already exists    {"Request.Namespace": "platform", "Request.Name": "iws-mbaum-test2"}                            │
│ 2021-02-08T11:56:04.460Z    INFO    controller_terraform    jobs.batch "iws-mbaum-test2" already exists    {"Request.Namespace": "platform", "Request.Name": "iws-mbaum-test2"}

This seems to confirm my suspicion that the fact the create job still exists when I try to delete the resource causes the issue I observed.
Maybe I am missing something. Can you please advice on if I did not follow the right procedure for deleting a resource.
The operator just loops forever waiting for the destroy task to be done, but no destroy task has been started.

HI @mbaumims thanks for reporting. I was able to replicate this quite easily. I'll take a look at this; it seems to be related to the changes between v0.3.0 and v0.3.1. The destroy feature still works in v0.3.0.
For a quick fix, you can revert the the container image back to v0.3.0 which should help complete the destroy process.

Hey @isaaguilar thanks for the quick feedback. I'll consider using the version you suggested. Just to understand, can I destroy infra at any time once it has been created, or does the pod and the job from the apply run need to be deleted first?

Just to understand, can I destroy infra at any time once it has been created,

Yes, destroy can run anytime after Terraform has completed. This means that tfstate is saved to your backend. The "original" pod can still exist when you run destroy.
I highly recommend that you select a backend that does state locking so just in case a pod is trying to make changes, the lock will protect against tfstate corruption.

Okay, great. Yeah, we're using S3 as our backend.

Fixed the bug in https://github.com/isaaguilar/terraform-operator/commit/e6aa74daf9e1e13272a55305eef19615d5874957 which is included in release v0.3.3.
Closing this ticket

Thanks for the quick turn around Isa. I tested your fix and it works well. One thing I noticed however, and it's not related to this bug per se, is that a plan is always run after an apply. This plan run overwrites the previously run plan, such that the original plan is no longer available. Not a big deal, but it would be nice to see the original plan.


A package that implements a libuv wrapper.

Ref: http://libuv.org/

Refer: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod.
libev is preferred over libuv because it might be easier to integrate with a TLS library, for e.g https://tls.mbed.org

Packages should be tracked in separate repos.


NPM install fails

I tried to install the ProtonMail Webinterface.
Sadly when I do: sudo npm install I always get:
git clone --template=/root/.npm/_git-remotes/_templates --mirror<EMAIL_ADDRESS>/root/.npm/_git-remotes/ssh-git-github-com-bartbutler-angular-gettext-tools-git-c40c6a4
I tried changing this to https:// in the package.json since manual https checkout works but ssh:// does not.  Sadly it still tries using ssh ...
So I just uncommented that package and it installed just fine.
Now I do: npm start and this results in the following:
> protonmail-web@ start /var/www/mail.yatekii.ch/WebClient
> grunt watch --api=dev

Running "notify_hooks" task

Running "ngconstant:dev" (ngconstant) task
Creating module proton.config at src/app/config.js...OK
Warning: Task "nggettext_extract" not found. Use --force to continue.

Aborted due to warnings.

npm ERR! Linux 4.1.15-gentoo-r1-xxxx-grs-ipv6-64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
npm ERR! node v0.12.6
npm ERR! npm  v2.11.2
npm ERR! code ELIFECYCLE
npm ERR! protonmail-web@ start: `grunt watch --api=dev`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the protonmail-web@ start script 'grunt watch --api=dev'.
npm ERR! This is most likely a problem with the protonmail-web package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     grunt watch --api=dev
npm ERR! You can get their info via:
npm ERR!     npm owner ls protonmail-web
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /var/www/mail.yatekii.ch/WebClient/npm-debug.log

And here is a cat of the .log:
nebuchadnezar WebClient # cat /var/www/mail.yatekii.ch/WebClient/npm-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info prestart protonmail-web@
6 info start protonmail-web@
7 verbose unsafe-perm in lifecycle true
8 info protonmail-web@ Failed to exec start script
9 verbose stack Error: protonmail-web@ start: `grunt watch --api=dev`
9 verbose stack Exit status 3
9 verbose stack     at EventEmitter.<anonymous> (/usr/lib64/node_modules/npm/lib/utils/lifecycle.js:213:16)
9 verbose stack     at EventEmitter.emit (events.js:110:17)
9 verbose stack     at ChildProcess.<anonymous> (/usr/lib64/node_modules/npm/lib/utils/spawn.js:24:14)
9 verbose stack     at ChildProcess.emit (events.js:110:17)
9 verbose stack     at maybeClose (child_process.js:1015:16)
9 verbose stack     at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
10 verbose pkgid protonmail-web@
11 verbose cwd /var/www/mail.yatekii.ch/WebClient
12 error Linux 4.1.15-gentoo-r1-xxxx-grs-ipv6-64
13 error argv "/usr/bin/node" "/usr/bin/npm" "start"
14 error node v0.12.6
15 error npm  v2.11.2
16 error code ELIFECYCLE
17 error protonmail-web@ start: `grunt watch --api=dev`
17 error Exit status 3
18 error Failed at the protonmail-web@ start script 'grunt watch --api=dev'.
18 error This is most likely a problem with the protonmail-web package,
18 error not with npm itself.
18 error Tell the author that this fails on your system:
18 error     grunt watch --api=dev
18 error You can get their info via:
18 error     npm owner ls protonmail-web
18 error There is likely additional logging output above.
19 verbose exit [ 1, true ]

So I actually have no clue what fails. Appart from node being the biggest mess I have ever seen ;)
I hope I am allowed to ask her :S
Thanks for any help!

The npm start fails because the package angular-gettext-tools is missing.
Have you setup a SSH key for your account? When you replaced ssh by https, did you just replaced by hand or did you take the right HTTPS URL from https://github.com/bartbutler/angular-gettext-tools?

No I used: https://github.com/bartbutler/grunt-angular-gettext. That is the module that fails to clone, not the one you mentioned. Except it's the same. Then it is wrong in the package.json.

No, it's because https://github.com/bartbutler/grunt-angular-gettext depends on the ssh version of https://github.com/bartbutler/angular-gettext-tools. So maybe adding https://github.com/bartbutler/angular-gettext-tools as an https dependency can solve you problem? If not, just add an ssh key to your Github account.

Ohh, I see, thanks a lot! It installed everything fine now :)
Unfortunately I now get this:
nebuchadnezar WebClient # npm start

> protonmail-web@ start /var/www/mail.yatekii.ch/WebClient
> grunt watch --api=dev

Running "notify_hooks" task

Running "ngconstant:dev" (ngconstant) task
Creating module proton.config at src/app/config.js...OK

Running "clean:build" (clean) task
>> 0 paths cleaned.

Running "nggettext_extract:pot" (nggettext_extract) task

Running "nggettext_compile:all" (nggettext_compile) task

Running "html2js:app" (html2js) task
Successfully converted 107 html templates to js.

Running "sass:build" (sass) task

Running "concat:build_css" (concat) task

Running "copy:build_app_assets" (copy) task
Created 16 directories, copied 190 files

Running "copy:build_appjs" (copy) task
Copied 102 files

Running "copy:build_vendorjs" (copy) task
Warning: Unable to read "vendor/jquery/dist/jquery.js" file (Error code: ENOENT). Use --force to continue.

Aborted due to warnings.

npm ERR! Linux 4.1.15-gentoo-r1-xxxx-grs-ipv6-64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
npm ERR! node v0.12.6
npm ERR! npm  v2.11.2
npm ERR! code ELIFECYCLE
npm ERR! protonmail-web@ start: `grunt watch --api=dev`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the protonmail-web@ start script 'grunt watch --api=dev'.
npm ERR! This is most likely a problem with the protonmail-web package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     grunt watch --api=dev
npm ERR! You can get their info via:
npm ERR!     npm owner ls protonmail-web
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /var/www/mail.yatekii.ch/WebClient/npm-debug.log

npm-debug.log:
nebuchadnezar WebClient # cat /var/www/mail.yatekii.ch/WebClient/npm-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info prestart protonmail-web@
6 info start protonmail-web@
7 verbose unsafe-perm in lifecycle true
8 info protonmail-web@ Failed to exec start script
9 verbose stack Error: protonmail-web@ start: `grunt watch --api=dev`
9 verbose stack Exit status 3
9 verbose stack     at EventEmitter.<anonymous> (/usr/lib64/node_modules/npm/lib/utils/lifecycle.js:213:16)
9 verbose stack     at EventEmitter.emit (events.js:110:17)
9 verbose stack     at ChildProcess.<anonymous> (/usr/lib64/node_modules/npm/lib/utils/spawn.js:24:14)
9 verbose stack     at ChildProcess.emit (events.js:110:17)
9 verbose stack     at maybeClose (child_process.js:1015:16)
9 verbose stack     at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
10 verbose pkgid protonmail-web@
11 verbose cwd /var/www/mail.yatekii.ch/WebClient
12 error Linux 4.1.15-gentoo-r1-xxxx-grs-ipv6-64
13 error argv "/usr/bin/node" "/usr/bin/npm" "start"
14 error node v0.12.6
15 error npm  v2.11.2
16 error code ELIFECYCLE
17 error protonmail-web@ start: `grunt watch --api=dev`
17 error Exit status 3
18 error Failed at the protonmail-web@ start script 'grunt watch --api=dev'.
18 error This is most likely a problem with the protonmail-web package,
18 error not with npm itself.
18 error Tell the author that this fails on your system:
18 error     grunt watch --api=dev
18 error You can get their info via:
18 error     npm owner ls protonmail-web
18 error There is likely additional logging output above.
19 verbose exit [ 1, true ]

I am not quite sure if I am really that dumb or if npm is really non-verbose :S

We should fix the dependency to be ssh (or https I guess) everywhere. Feel free @emersion :)

@Yatekii The error here is: Warning: Unable to read "vendor/jquery/dist/jquery.js" file (Error code: ENOENT). Use --force to continue.. Can you try to run node_modules/.bin/bower install? Normally this step is executed automatically right after npm install, but it seems that it's not the case for you.
@bartbutler Yes, since we don't need write permissions to clone these public repos, https is better as it doesn't require any authentication.

@emersion I guess it was because bower doesn't like the sudo I put in sudo npm install ;) There was some hint how to fix it; guess I will read that later :)
The script now ends with those four lines and then chills out:
Running "connect:watch" (connect) task
Started connect web server on http://localhost:8080
Running "delta" task
Waiting...
When I load that page I just get the page source as the html body but not actually a rendered page ... will have a look at my firewall and nginx relay config :)
Thanks a lot for your help!

@emersion I guess it was because bower disliked sudo from sudo npm install ;) We'll fix that later. For now the Webserver seems to run and the nginx-forward works too. Sadly I don't see the rendered page but It's source as the body ... have a look: https://mail.yatekii.ch/ (yes yes, the certi was ripped from another subdomain I have (will do that after I get the client running))

When I go to https://mail.yatekii.ch/ I get a rendered page.
Note that you can't use this client as a webmail without a server - and this repo doesn't include a real server. That's why you get "Sorry our server is down" when you try to login.

http://noah.wapple.ch/screens/Screen Shot 2016-06-12 at 18.44.46.png
Here is a Screenshot of what I see in Chrome. In Safari it even downloads the .html file ... Firefox is the same problem. So I have no clue how you can get a rendered page ...
Mates of mine tell me the same behvaior on their systems :(
Also, I already have a server backend. I just want a proper webclient for when I have no laptop with me :) And ProtonMail has the only sane one in both design and functionality as far as I am informed.

How do you plan to connect the webclient with your server? You can't just install the webclient and expect it to work with your IMAP/SMTP server - you need an API server between.

... lol, I was talking about the WebUI not rendering. To which you responded it renders great for you. Which it doesn't. And now you are talking about IMAP/SMTP?
Anyways, I know it's an nginx issue which I can fix myself :) Thanks for your help!

Just wanted to make sure you know that you can't use the webclient as is to read your personal email. ;-)


Improved Show and Match types

Previous version:

New version:

This was my misunderstanding of TypeScript and coercion. I had it that way originally and changed it since I thought it was effectively the same. Thanks for rectifying that.


Popover do not destroy  the content component when hiding the content

What problem does this feature solve?
I have a loading component in popover which will increase after ngOnInit triggering. when the popover close, the loading compinent will
be destroied, the loading process bar will start over again after popover opening.
there is a demo: demo
What does the proposed API look like?
Provides an API for popover , such as nzNotDestroyContent

You can extract the data source for <app-loading>,  and I think it's easy to solve.

I debugged it a it seems that Popover works correctly. When togging its visibility, nz-popover does not re-create the overlay component.
IMHO, the reason may be that you put a component in the template
<ng-template #contentTemplate>
  <app-loading></app-loading>
</ng-template>

It seems that this property is always true and hence the view get re-created every time. Could you please take a look at this?

I will have a look, thanks @wendellhu95


Homekit storage problem

Hi..
After flash the led example, in serial monitor shows that:

HomeKit: Starting server
!!! HomeKit: Failed to read HomeKit storage magic
HomeKit: Formatting HomeKit storage at 0x100000
!!! HomeKit: Failed to erase HomeKit storage
HomeKit: Resetting HomeKit storage
!!! HomeKit: Failed to reset HomeKit storage
HomeKit: Generated new accessory ID: 61:CE:30:5E:75:42
!!! HomeKit: Failed to write accessory ID to HomeKit storage
scandone
HomeKit: Generated new accessory key
!!! HomeKit: Failed to write accessory key to HomeKit storage
HomeKit: Configuring mDNS

...
hope any1 can help... thank you

Which ESP module (which hardware) are you using?

nodemcu lolin v3

Try changing HOMEKIT_SPI_FLASH_BASE_ADDR to a smaller value (e.g. 0x7A000)

I flash the esp with this command:
esptool.py -p /dev/cu.wchusbserial1420 --baud 115200 write_flash -fs 1MB -fm dout -ff 40m 0x0 rboot.bin 0x1000 blank_config.bin 0x2000 otaboot.bin
as far as I understand the HOMEKIT_SPI_FLASH_BASE_ADDR is the -fs 1MB... I've changed it to -fs 8m and I have the same problem.

No, HOMEKIT_SPI_FLASH_BASE_ADDR has nothing to do with -fs option of esptool.py. You need to recompile firmware with that flag (e.g. make HOMEKIT_SPI_FLASH_BASE_ADDR=0x7a000 all)

Thanks a lot!  that was the problem!  congratulations on the code, you are a genius!


Vue Element Syntax Error with Arrow Functions Involving Strings

Using arrow functions within Vue elements and utilizing strings inside them leads to an error, regardless of whether backticks(`), single quotes('), or double quotes(") are used.

I can't reproduce it with ST4155...4178.

<q-dialog
	:refs="() => {refController(modal.fullTool.refs, 'dialog')}"
	v-model="modal.fullToll.isOpen"
	no-esc-dismiss
	no-backdrop-dismiss
>

As Vue just embedds source.js the issue might be caused by an outdated or malformed JavaSctipt syntax.


[KOGITO-3927] Add processInstance before process start

Fix https://issues.redhat.com/browse/KOGITO-3927

jenkins retest this


Add Il2CppFieldDefinition::DeclaringType

Convenience
Discoverability
Debugging

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 1 (0.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.001%) to 28.211%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

LibCpp2IL/Metadata/Il2CppFieldDefinition.cs
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>0:
-0.001%

Covered Lines:
3352

Relevant Lines:
10110

💛  - Coveralls


cant run this on docker-compose traefik?

Hi!
I tried to run this on my traefik stack. Tried to get this work for awhile and having no clue what might be the problem.
I read the instructions on https://miniflux.app/docs/installation.html#docker
This is my setup.
Docker-compose.yml
version: '3'
services:

  miniflux:
    image: miniflux/miniflux:latest
    container_name: miniflux
    environment:
      - DATABASE_URL=postgres://miniflux:secret@db/miniflux?sslmode=disable
      - RUN_MIGRATIONS=1
      - CREATE_ADMIN=1
      - ADMIN_USERNAME=miniflux
      - ADMIN_PASSWORD=secrets

    networks:
      - web
    depends_on:
      - db
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=host:miniflux.example.xyz
      - traefik.port=8080
      - traefik.docker.network=web

  db:
    image: postgres:10.1
    environment:
      POSTGRES_USER: miniflux
      POSTGRES_PASSWORD: secret
    volumes:
      - ./miniflux-db-data:/var/lib/postgresql/data
    container_name: db
    networks:
      - web
    labels:
      - traefik.enable=false

networks:
  web:
    external: true

Docker-compose logs contains
db          | The files belonging to this database system will be owned by user "postgres".
db          | This user must also own the server process.
db          |
db          | The database cluster will be initialized with locale "en_US.utf8".
db          | The default database encoding has accordingly been set to "UTF8".
db          | The default text search configuration will be set to "english".
db          |
db          | Data page checksums are disabled.
db          |
db          | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db          | creating subdirectories ... ok
db          | selecting default max_connections ... 100
db          | selecting default shared_buffers ... 128MB
db          | selecting dynamic shared memory implementation ... posix
miniflux    | Current schema version: 0
miniflux    | Latest schema version: 23
miniflux    | Migrating to version: 1
miniflux    | [FATAL] [Migrate] dial tcp <IP_ADDRESS>:5432: connect: connection refused
db          | creating configuration files ... ok
miniflux exited with code 1
db          | running bootstrap script ... ok
db          | performing post-bootstrap initialization ... ok
db          | syncing data to disk ... ok
db          |
db          | WARNING: enabling "trust" authentication for local connections
db          | You can change this by editing pg_hba.conf or using the option -A, or
db          | --auth-local and --auth-host, the next time you run initdb.
db          |
db          | Success. You can now start the database server using:
db          |
db          |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db          |
db          | waiting for server to start....2019-07-19 07:53:52.653 UTC [36] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
db          | 2019-07-19 07:53:52.654 UTC [36] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
db          | 2019-07-19 07:53:52.654 UTC [36] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
db          | 2019-07-19 07:53:52.659 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db          | 2019-07-19 07:53:52.689 UTC [37] LOG:  database system was shut down at 2019-07-19 07:53:52 UTC
db          | 2019-07-19 07:53:52.700 UTC [36] LOG:  database system is ready to accept connections
db          |  done
db          | server started
db          | CREATE DATABASE
db          |
db          | CREATE ROLE
db          |
db          |
db          | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db          |
db          | 2019-07-19 07:53:53.258 UTC [36] LOG:  received fast shutdown request
db          | waiting for server to shut down....2019-07-19 07:53:53.261 UTC [36] LOG:  aborting any active transactions
db          | 2019-07-19 07:53:53.266 UTC [36] LOG:  worker process: logical replication launcher (PID 43) exited with exit code 1
db          | 2019-07-19 07:53:53.271 UTC [38] LOG:  shutting down
db          | 2019-07-19 07:53:53.295 UTC [36] LOG:  database system is shut down
db          |  done
db          | server stopped
db          |
db          | PostgreSQL init process complete; ready for start up.
db          |
db          | 2019-07-19 07:53:53.391 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
db          | 2019-07-19 07:53:53.393 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db          | 2019-07-19 07:53:53.399 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db          | 2019-07-19 07:53:53.423 UTC [63] LOG:  database system was shut down at 2019-07-19 07:53:53 UTC
db          | 2019-07-19 07:53:53.432 UTC [1] LOG:  database system is ready to accept connections

@thepenguinthatwants Miniflux is starting too quickly. You need to set up an extra image that will wait for port 5432 to be open, and make Miniflux depend on that. For example https://github.com/jwilder/dockerize

Thanks for the answer. I didnt even thought thats possible. No sure how to utilize dockerize but I guess its like putting miniflux to depend on something that takes more time to start to slowdown miniflux start.
Do I need to define the port 5432 on my docker-compose?

So I changed
    depends_on:
      - db-miniflux
      - dockerize

and added
  dockerize:
    image: jwilder/dockerize
    container_name: dockerize

the logs are still.
db-miniflux    | 2019-07-19 09:24:44.642 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
db-miniflux    | 2019-07-19 09:24:44.642 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db-miniflux    | 2019-07-19 09:24:44.645 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-miniflux    | 2019-07-19 09:24:44.718 UTC [20] LOG:  database system was shut down at 2019-07-19 08:07:07 UTC
db-miniflux    | 2019-07-19 09:24:44.732 UTC [1] LOG:  database system is ready to accept connections
dockerize      | Usage: dockerize [options] [command]
dockerize      |
dockerize      | Utility to simplify running applications in docker containers
dockerize      |
dockerize      | Options:
dockerize      |   -delims string
dockerize      |        template tag delimiters. default "{{":"}}"
dockerize      |   -no-overwrite
dockerize      |        Do not overwrite destination file if it already exists.
dockerize      |   -poll
dockerize      |        enable polling
dockerize      |   -stderr value
dockerize      |        Tails a file to stderr. Can be passed multiple times
dockerize      |   -stdout value
dockerize      |        Tails a file to stdout. Can be passed multiple times
dockerize      |   -template value
dockerize      |        Template (/template:/dest). Can be passed multiple times. Does also support directories
dockerize      |   -timeout duration
dockerize      |        Host wait timeout (default 10s)
dockerize      |   -version
dockerize      |        show version
dockerize      |   -wait value
dockerize      |        Host (tcp/tcp4/tcp6/http/https/unix) to wait for before this container starts. Can be passed multiple times. e.g. tcp://db:5432
dockerize      |   -wait-http-header value
dockerize      |        HTTP headers, colon separated. e.g "Accept-Encoding: gzip". Can be passed multiple times
dockerize      |   -wait-retry-interval duration
dockerize      |        Duration to wait before retrying (default 1s)
dockerize      |
dockerize      | Arguments:
dockerize      |   command - command to be executed
dockerize      |
dockerize      | Examples:
dockerize      |
dockerize      |    Generate /etc/nginx/nginx.conf using nginx.tmpl as a template, tail /var/log/nginx/access.log
dockerize      |    and /var/log/nginx/error.log, waiting for a website to become available on port 8000 and start nginx.
dockerize      |
dockerize      |    dockerize -template nginx.tmpl:/etc/nginx/nginx.conf \
dockerize      |              -stdout /var/log/nginx/access.log \
dockerize      |              -stderr /var/log/nginx/error.log \
dockerize      |              -wait tcp://web:8000 nginx
dockerize      |
dockerize      | For more information, see https://github.com/jwilder/dockerize
miniflux       | Current schema version: 23
miniflux       | Latest schema version: 23
miniflux       | [INFO] User "miniflux" already exists, skipping creation
miniflux       | [INFO] Starting Miniflux...
miniflux       | [INFO] Starting scheduler...
dockerize exited with code 2
miniflux       | [INFO] Listening on "<IP_ADDRESS>:8080" without TLS

I still get so the database system seems to shutdown.
When I try to get into miniflux.
404 page not found

You are not configuring dockerize correctly. I suggest you have a look at the docs there

version: "3"
services:
  postgress:
  ....
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
  app:
    ...
    depends_on:
      postgres:
        condition: service_healthy

ERROR: The Compose file './docker-compose.yml' is invalid because:
services.miniflux.depends_on contains an invalid type, it should be an array


Serialization of DataPreparer pipeline.

Description
The DataPreparer should be able to be serialized and deserialized. The base serialization Mixin enables this naively (i.e., by calling .serialize(deep=True) method on the DataPreparer). The output is currently a nested json object with many redundant/duplicate (i.e., column_sharer) and not so readable raw python object tag (i.e., py/). The goal is to implement OO serialization that enables custom filtering for each component during serialization such that it decides what should and should not be displayed.
For instance,

NoTransform should be hidden - this functionality may want to be backported to the base serializer mixin.
each step will need a 'column view' whereby you can see what transformer was mapped to each column, at each step in DataPreparer
column_sharer will be serialized multiple times at each substep has the shared instance of it. It should either be displayed only once or not at all.

One rule of thumb is that if there are duplication for the same instance, they should not be shown in the serialized object.
What should be included in the output
In a format that can represents nested hierarchy of the data_preprarer, the list of steps, and the transformer in each steps, we want to include all the leaf parameters (of the objects just mentioned) that have changed from their default values.
Q: Then how does user know what are the existing parameters that they can potentially tune/override if we only output changed parameters? What if they are all default values? @adithyabsk could you comment on this?
Update based on discussion with @adithyabsk :

We will also need to keep fields like _method and _class so that the deserialization method knows what method to deserialize the data and to what class it should deserialize it.

Success Criteria

DataPreparer is able to be serialized to json and yaml format.
DataPreparer is able to be deserialized back from json and yaml files.
The serialized output should be free of redundant info and human-friendly.

Subtasks

[x] Implement (or fix) the ConcreteSerializerMixin on the PreparerStep class, invoking to_json().  Spent 2 hours on this task
[x] .from_json() bug due to get_params() of data_preparer, possibly blocked by #129 . Spent 1 hour just to root cause the issue
[x] Based on the output of serialization (after we finish subtask 1), reference the V1 serialization format to design a draft version of the V2 format.
[x] Review the draft with the team.
[ ] Implement the serialization by overriding the .serialize() and .deserialize() methods in each component if certain custom filtering on what is returned by .get_params() is required. Is this blocked by #129 ? @cchoquette could you clarify?
[ ] Unit tests for new code
[ ] (lower priority) Code coverage for old code

TODO:

[ ] Read the code to understand the flow of get_params, set_params, serialize and deserialize.

Estimate

1 to 2 days for the first 3 tasks
2 to 3 days for the last 2 tasks
Total: 3 to 5 days (and by a day I mean 4 uninterrupted hours per day).

Fixed the issue with recursive serialization on PrepareStep that inherits/implements the ConcreteSerializerMixin . The _method argument was introduced into the kwargs during the call in _make_serializable(), which breaks {concrete}_serialize method (in this case dict_method) since it does not accept _method as a keyword argument.
The solution is to extract and pop the _method from kwargs and assign it to the method variable if it exists in kwargs. In this way we can pass down the serialize method recursively without breaking downstream code. This solution also avoids changing the current unit tests.
However, it also means that we have untested code branches and they need to be thoroughly tested before we can claim (de)serialization is working.

Currently, .get_params() on data preparer shows the following info, which includes a lot of info that it should not include as mentioned in #129 .  The starting point I have right now is the ColumnSharer and we are going to override the dict_serialize method by using the jsonpicker and manipulate the output there. Then we try to deserialize it back. This will give me a better understanding of how this process work.

Data Preparer deserialization does not work due to issue with "unpicklering" some data with the error:
E       IndexError: list index out of range  in .pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/jsonpickle/unpickler.py:97: IndexError

Serialization of a bare minimal data_preparer with only a data_cleaner on a dataset with 1 column is completed as follows:
{
  "cleaner_kwargs": null,
  "intent_kwargs": null,
  "engineerer_kwargs": null,
  "preprocessor_kwargs": null,
  "reducer_kwargs": null,
  "modeler_kwargs": null,
  "y_var": null,
  "steps": [
    {
      "data_cleaner": {
        "n_jobs": 1,
        "transformer_weights": null,
        "collapse_index": true,
        "_class": "CleanerMapper",
        "_method": "dict",
        "transformation_by_column_group": [
          {
            "crim": {
              "memory": null,
              "steps": [
                {
                  "Flatten": {
                    "check_wrapped": true,
                    "force_reresolve": false,
                    "keep_columns": false,
                    "name": "Flatten",
                    "should_resolve": false,
                    "transformer": {
                      "_class": "NoTransform",
                      "_method": "dict"
                    },
                    "y_var": false,
                    "_class": "Flatten",
                    "_method": "dict"
                  }
                },
                {
                  "Cleaner": {
                    "check_wrapped": false,
                    "force_reresolve": false,
                    "keep_columns": false,
                    "name": "Cleaner",
                    "should_resolve": false,
                    "transformer": {
                      "_class": "NoTransform",
                      "_method": "dict"
                    },
                    "y_var": false,
                    "_class": "Cleaner",
                    "_method": "dict"
                  }
                }
              ],
              "_class": "DynamicPipeline",
              "_method": "dict"
            }
          }
        ]
      }
    }
  ],
  "column_sharer": {
    "store": {
      "domain": {
        "crim": "NoTransform"
      }
    },
    "_class": "ColumnSharer",
    "_method": "dict"
  },
  "_class": "DataPreparer",
  "_method": "dict"
}

The main challenge is to sift through and remove all the verbose output from jsonpickler but at the same time keep all the valuable content.

Currently there are three issues:

[ ] Serialized and deserialized Scalers are different due to how params with tuple type are treated in the process. If we are using jsonpickler, tuple will be tagged with py/tuple. However, since we are not using it, that info is lost and we deserialize back, it is treated as a list instead of a tuple. This may or may not be an issue, depending on how the tuple is used. For example, for the quantile_range in the RobustScaler, it is only used to access the min/max value so tuple and list both work but it may not work in other cases.
[ ] We are not serializing the fitted field, for example, the scaler has a field center_. It's not part of the serialized object and thus nowhere to be found after deserialization.  In other words, we are only serializing and deserializing an unfitted pipeline. Is this intentional?
[ ] ColumnSharer, even after the preprocessing step, does not include the one-hot encoded categorical columns. This seems like a problem? Could this be a bug?

We are not serializing the fitted field, for example, the scaler has a field center_. It's not part of the serialized object and thus nowhere to be found after deserialization. In other words, we are only serializing and deserializing an unfitted pipeline. Is this intentional?

Yes, the idea is that there are three types of serialization:

User facing params, ie not the internal state (dict serialize)
Internal state, ie save params such as center_ in pickle
a. inline (in the serialized representation)
b. symbolic (saved to disk with a link to the saved location)

We are not serializing the fitted field, for example, the scaler has a field center_. It's not part of the serialized object and thus nowhere to be found after deserialization. In other words, we are only serializing and deserializing an unfitted pipeline. Is this intentional?

@JingJZ160 Yes, the idea is that there are three types of serialization:

User facing params, ie not the internal state (dict serialize)
Internal state, ie save params such as center_ in pickle
a. inline (in the serialized representation)
b. symbolic (saved to disk with a link to the saved location)

Thanks for the clarification @adithyabsk

Currently there are three issues:

[ ]  Serialized and deserialized Scalers are different due to how params with tuple type are treated in the process. If we are using jsonpickler, tuple will be tagged with py/tuple. However, since we are not using it, that info is lost and we deserialize back, it is treated as a list instead of a tuple. This may or may not be an issue, depending on how the tuple is used. For example, for the quantile_range in the RobustScaler, it is only used to access the min/max value so tuple and list both work but it may not work in other cases.
[x]  We are not serializing the fitted field, for example, the scaler has a field center_. It's not part of the serialized object and thus nowhere to be found after deserialization.  In other words, we are only serializing and deserializing an unfitted pipeline. Is this intentional?
[x]  ColumnSharer, even after the preprocessing step, does not include the one-hot encoded categorical columns. This seems like a problem? Could this be a bug?
[ ]  Given the way we are group columns by intent, the output can still be very verbose, especially if we have a large number of columns with the same intent. The column group is currently a list of columns joined by , so it could get very long. Are we OK with this?

For item 3:
Strangely, if I re-enable feature reducer in the DataPreparer, we have all those encoded columns again in the serialized ColumnSharer. Definitely need a deep dive to understand how it works. Turn it back on and off to compare the output.
Update:
It is because of the self.check_resolve() method of the AutoIntentMixin and FeatureEngineerer, Preprocessor and FeatureReducer all implement. Since the generated columns are created after the preprocessing step, they are populated in the column_sharer only in the feature_reducer when self.check_resolve() is invoked.
For item 1 (scroll to the right):
E         'copy': True,
E         -                                                                                'quantile_range': (25.0,
E         ?                                                                                ^
E         +                                                                                'quantile_range': [25.0,
E         ?                                                                                ^
E         -                                                                                75.0),
E         ?                                                                                ^
E         +                                                                                75.0],
E         ?                                                                                ^

For item 1, we will need to implement a customized json encoder/decoder: https://stackoverflow.com/questions/15721363/preserve-python-tuples-with-json
I'll create a new issue for this.

No longer a concern. We may add a task to simplify the whole serialized json or just create a python file to include the whole trained pipeline there like TPOT.


Machine creation fails due to pool keyservers

I get this failure a lot during machine provisioning (like 10% of the time).
I know that provisioner executes if ! type docker; then curl -sSL https://get.docker.com | sh -; fi and it's the get.docker.com script handles it, but could there be anything we can do in machine to prevent it (maybe add retries around this ssh command)? (Sorry if this is not the right repo to open the issue.)
+ sudo -E sh -c apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.vrqNYDUeXK --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
gpgkeys: key 58118E89F3A912897C070ADBF76221572C52609D can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

Error creating machine: Error running provisioning: error installing docker:
[bugsnag output redacted]

There's some improvements made to the install script, but they may not be live yet; https://github.com/docker/docker/pull/20737

@thaJeztah nice,thanks!

Going to go ahead and close this issue -- we can re-open if those improvements don't seem to help with this type of issue (I've definitely seen it before)


DecomposeIntoTimeStepsCA doc bug

Current documentation for DecomposeIntoTimeStepsCA says the only Trotter-Suzuki orders supported are one and two:
    /// ## trotterOrder
    /// Selects the order of the Trotter–Suzuki integrator to be used.
    /// Order 1 and 2 are currently supported.

Looking at the implementation, it appears though like arbitrary even orders are supported, too.
The documentation should hence be updated if higher orders are in fact supported. In case the code handling higher orders were not supported/known buggy(?), a comment to this effect should be added to the code.

@guanghaolow could you comment on this?

@jgukelberger Thanks for pointing this out. Even orders are supported. I've updated the documentation to reflect this.


everytime open vscode show 'Shell command 'code' successfully installed in PATH.' alert

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.75.0
OS Version: macOS 13.0.1

Steps to Reproduce:

close vscode and open vscode.
click file-new window

Disabled all extensions not show this.


[Bug] Interpolation in transform operation does not work

Pre-requist tasks

[X] My device drivers are up to date.
[X] The troublemaking solutions did not solved my issue.

Describe the bug
Interpolation option in transform operation return an empty image
To Reproduce
set interpolation to True when running affine_transform()
Expected behavior
it should work identically to the prototype
Screenshots
See #197

should be fixe with https://github.com/clEsperanto/CLIc/pull/310


Upgrade to Node 18

Expected Behavior
Be compatible with NodeJS v18.
Actual Behavior
Only compatible with Node v16.
Reproduction Steps

Install Node 18
Clone latest repo
Run npm run bootstrap

Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
No response
Insomnia Version
5c109ac4969bd6a9a29ce60e7fa0550a9fe40e8a
What operating system are you using?
Other (specify below)
Operating System Version
FreeBSD 13.1
Installation method
Git clone ; npm run bootstrap
Last Known Working Insomnia version
N/A

Hi! I'd like to work on this. I am a newbie but this is my understanding:

Install Node 18
Clone latest repo
Run npm run bootstrap
and after this do I send s pull request?
Thank you!

@yuhwaa  Yes. A high level overview:

remove .nvmrc
add the following to package.json:

"engineStrict": true,
"engines": {
    "node": ">=16.0.0"
 },

Hi @yonas and @yuhwaa unfortunately node version and electron version must be in sync. There is no electron build which will work with node 18 currently. The latest electron version is 20 which requires node v16.15.0. I have an open PR for it but its blocked. Can I ask what feature of node 18 would make this desirable?
https://www.electronjs.org/releases/stable

@jackkav On FreeBSD 13, the default node version is 18, although it's possible to install node 16.16.0.
It would be great to be able to build a web app only build so I can run insomnia headless on a dedicated server.

Due to current insomnia architecture a web version is still a long way off, although it would be handy. Electron is moving fast, we might see a node 18 version in the next 12 months 🤞


CIV-9482 corrected cancel page and expert name label

JIRA link
https://tools.hmcts.net/jira/browse/CIV-9482
Change description
Update cancel page to show radio button inline
Corrected experts' names to expert's names

Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Expert's name changes need to be reverted as per latest update on the ticket. https://tools.hmcts.net/jira/browse/CIV-9482


Allow to Store Vaccination Cert in Google Pay (Google equivalent of apple wallet)

Feature description
As iOS will get the wallet feature I think I would be nice Todo the same for Android users. This should be possible thru the https://developers.google.com/pay/passes
Pass API in Google pay
Problem and motivation
Quick access to the Vaccination Cert
Is this something you're interested in working on
Solid Maybe

Internal Tracking-ID: EXPOSUREAPP-7975

@5V715 Thank you for the request.
I've forwarded the feature request to the developers (ticket ID: EXPOSUREAPP-7975). We will now investigate how this feature could be implemented and report back here on Github if we get any news.

Corona-Warn-App Open Source Team

Hello everyone,
This feature request has been declined and will not be pursued further. We have decided against this feature because of the legal restrictions and drawbacks regarding the privacy of the stored data.

Corona-Warn-App Open Source Team

Hello everyone,
This feature request has been declined and will not be pursued further. We have decided against this feature because of the legal restrictions and drawbacks regarding the privacy of the stored data.
Corona-Warn-App Open Source Team

Could you elaborate a bit more on what the legal restrictions and privacy concerns were?
According to the APIs documentation, the Data is stored locally on the device?
What legal ristrictions are there ristricting this but seemingly not corona-warn-app/cwa-app-ios#2965

@DRSchlaubi https://github.com/corona-warn-app/cwa-app-ios/issues/2965 was reopened by accident. The feature was rejected on both Android and iOS for the same reasons.

Feature description
As previously requested in Digitaler-Impfnachweis/certification-apis#37, corona-warn-app/cwa-app-ios#2965 and corona-warn-app/cwa-app-android#3486. The original issue on this has been closed due to privacy concerns, that were unfortunately discussed internally, so I can't know if all of them still exists. However since the original issue on this has been closed, Google has released a new pass type in the passes API specifically, which might solves some concerns? In other open discussions about this, people have been concerned about being able to see the phones Covid certificates even if it's locked, which won't be an issue, since this certificates require a display lock to be set and the phone to be unlocked.
Documentation links

https://developers.googleblog.com/2021/06/google-updates-passes-api-to-save-covid-testing-and-vaccination-information-on-Android-devices.html
https://developers.google.com/pay/passes/guides/pass-verticals/use-cases?vertical=covid-cards
https://payments.developers.google.com/terms/aup#covid-cards
https://developers.google.com/android/reference/com/google/android/gms/pay/package-summary

Problem and motivation
It can come in quite handy to have all of your cards and passes in one place, also having an official way to do this can make use of the new Pass type which is only available to authorities which can actually create vaccination certificates
Is this something you're interested in working on
Maybe. I am not sure how much I could do as the documentation on Covid certificate is locked behind a registration form only authorities have access to, I would be able to work on the Integration in the App itself since this Documentation is public

@DRSchlaubi I've asked the dev team to reevaluate the issue based on the information you provided. We'll have to see what RKI and BSI have to say about this new development.

Corona-Warn-App Open Source Team

https://support.google.com/pay/answer/10890261 says

Add your COVID card to your Android device
On your Android device, you can store a digital copy of your COVID vaccination and test information as a COVID card.
Your COVID card will only be stored on the device that you save it on. You can manually save it on more than one device.
You don't need the Google Pay app to use this feature.
This feature is currently available for government and healthcare providers or authorised third parties in the US.
This feature is available in the US.

Is there any indication that this feature is planned to be made available in countries of the European Union?

https://support.google.com/pay/answer/10890261 says

Add your COVID card to your Android device
On your Android device, you can store a digital copy of your COVID vaccination and test information as a COVID card.
Your COVID card will only be stored on the device that you save it on. You can manually save it on more than one device.
You don't need the Google Pay app to use this feature.
This feature is currently available for government and healthcare providers or authorised third parties in the US.
This feature is available in the US.

Is there any indication that this feature is planned to be made available in countries of the European Union?

As there is no clear documentation about this, I contact the Google Pay Developer support about this. I will update this comment as soon as I have an answer

Covid passes are not yet available in every region. If you are interested in issuing vaccine cards in Germany fill out the form here, and we will collect your information and add you to our pipeline, and we will get back to you once these passes are available in your region.

Any updates on this?

@DRSchlaubi No, nothing new here, sorry.

Corona-Warn-App Open Source Team

Also discussed in https://github.com/Digitaler-Impfnachweis/covpass-android/issues/52.

Still no updated after months?


Louisiana: ReportStream to Ochsner

@JohnNKing to provide a description

We've started the app registration on fhir.epic.com and have a staging certificate that we can use for the HCO.

I've requested HL7v2 API access via email. Still waiting on that...

Prod and non-prod keys shared with Ochsner.
Next steps also sent via email:

Ochsner to request an app download via Epic for the non-production client ID above.
ReportStream should receive a notification and will approve your request.
ReportStream to upload a certificate to fhir.epic.com for authentication to your endpoint.
Ochsner will need to share the non-production authentication and results delivery endpoints (e.g. HOSTNAME.epichosted.com/FhirProxy/oauth2/token and HOSTNAME.epichosted.com/ENVIRONMENT/api/epic/VERSION/EDI/HTTP/HL7v2/NUMBER)

Blocked waiting on Ochsner. See steps required in prior comment.

Blocked pending a 3rd Party Organizational Risk Questionnaire

Blocked pending a 3rd Party Organizational Risk Questionnaire

Reviewed at Planning. @JohnNKing Could you provide an update?

Reviewed at Planning. @JohnNKing Could you provide an update?

@basiliskus Still blocked -- the 3rd Party Organizational Risk Questionnaire is pending completion by ReportStream.
I've also updated the ticket description based on our experience with Epic connectivity from UCSD.


cargo check - make possible to use "pipes" instead of files

As I understand cargo check works only with files,
so work pattern should looks like:

Enter a couple of symbols
Save file to disk
Run cargo check

A lot of saves, bad for SSD.
So it would be good if cargo check provide interface like:
cargo check --temp-file-name src/lib.rs
in such mode cargo read content of modified src/lib.rs from stdin,
and do cargo check as src/lib.rs contains data that cargo read from stdin.
This helps for example flycheck-rust: https://github.com/flycheck/flycheck-rust/issues/29

For me, the big challenge is that IDEs would likely have multiple files open for edit that need to be checked and stdin wouldn't be sufficient.  https://internals.rust-lang.org/t/jar-for-rust-single-file-crate-support-for-rustc/19576 does do some exploring for alternative ways for Cargo to communicate source buffers to rustc.


chore: increase version apsara

Increase apsara version in UI, sdk-demo and core.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 9.565%

Totals

Change from base Build<PHONE_NUMBER>5:
0.0%

Covered Lines:
10679

Relevant Lines:
111643

💛  - Coveralls


Generated paths during the output generation don't point to the right files

Generated paths during the output generation don't point to the right files.
For example the blog folder.
The styles don't load for any of the blog pages as it's trying to find the static folder inside the blog folder.
The static folder is in the parent directory, to fix this while creating any new folder all the paths inside it should start with ../ indicating that this path is located in a parent directory.

Can you provide an example of the fix that you're suggesting.
If I understand correctly the issue is that the static path isn't pointing to an absolute path so you wind up with urls not always pointing to the correct path.
This is an issue that I've noticed a few times when dealing with GH Pages where your url may not be the root path. I think the best solution is one similar to other frameworks where there is a {{"<PATH/AFTER/STATIC>" | static }} or similar pattern be added to create an absolute URL for the path.
My main concern with this is do you use that in content for blog posts or do something different.
To my knowledge you cannot pass jinja variables into your content (perhaps this is a reason to change that).

We agreed on using absolute paths for the whole website during generation we substitute with the SITE_URL var instead of relative or current implementation to account for cases when you just want to double click on the page to open it instead of using a server.


DEFAULT_KEY_HANDLE_LEN too big

for Oracle FIDO auth the DEFAULT_KEY_HANDLE_LEN is too big and gets rejected, had to lower it to 192

Thanks for the report, fixed in 941df8b3993481dc6559b373c0dcffb89d18fcee


Update action filters to avoid duplicates on Jira

Currently, we are not taking into account the resolution status of the tickets already been raised in the jql filters of the app leading to duplications.
What needs to be added is the following logic:

if a ticket has been marked as obsolete (resolution = obsolete), then it means if the same vulnerability is spotted, it's ignored
if a ticket has bene marked as resolved/done then if the same vulnerability is spotted, then we reopen it
The change has to be done here

After further investigation, this is not a bug. The mechanism to prevent duplications works as expected.


Question: Why "batch size" affects sampler results?

Problem:
Having different amount of frames severely affects output results (see screenshots below)
Disabling/Bypassing ADE_AnimateDiffLoaderWithContext generates expected outputs.
Details:

Initially was following https://civitai.com/articles/2379/guide-comfyui-animatediff-guideworkflows-including-prompt-scheduling-an-inner-reflections-guide took and took 3 - Basic Txt2Vid.json as base.
I was trying to make 1 frame as initial prompt, so later increase batch size.
But it is impossible, cause output looks like noise.
Attaching my current workflow
3 - Basic Txt2Vid_v2.json

1 frame:

1 frame (bypass AnimateDiff):

12 frames:

15 frames:

Yesterday was normal, today I also have a similar situation, update comfyui is useless

@gxbsyxh the thing the question is about here has always been this way. Are you replying to the proper issue?

@DmytroSokhach hey, here is an explanation I wrote for someone else in a different thread that asked a similar question: https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/issues/151#issuecomment-1793164770

@gxbsyxh这里的问题一直都是这样。您是否在回复正确的问题？您可以提出一个新问题并解释您所经历的情况，但如果它与少于 16 帧而没有良好的输出有关，那么这就是 AD 的本质。您也可以阅读我在下面链接的解释。
Thanks for your reply, I am still troubleshooting the problem, my problem is that ControlNet tile image is used to image, ControlNet does not work, it can be used normally yesterday, but there is noise today, Text-to-Image is normal

Thanks again, after I re-installed ComfyUI-Advanced-ControlNet, and most nodes, the function returned to normal, I also don't understand why the sudden noise

@gxbsyxh

Thanks again, after I re-installed ComfyUI-Advanced-ControlNet, and most nodes, the function returned to normal, I also don't understand why the sudden noise

Hey, I'm not using any ControlNet here. Please see workflow attached above - you can use it to reproduce issue.
I've tried reinstalling custom nodes and models, but seems it has no effect. 😕
@Kosinkadink Thank you for pointing to https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/issues/151#issuecomment-1793164770 👍
Looking forward for implementing that part, as it sound it might help in my scenario 🤞

so that AnimateDiff will use an existing latent/image as a reference

Hey @DmytroSokhach, I think you may have slightly missed the key point of the comment I linked, so I'll try to summarize here:
The 'issue' you're describing with less than 16 frames is not an issue - it's just how animatediff works. AnimateDiff achieves it's temporal consistency by diffusing multiple latents at a time. Specifically, AnimateDiff models were trained on two seconds of footage at 8fps, meaning 16 frames in total. Due to that, the further you stray from 16 frames in vanilla AnimateDiff, at some point you just get bad output. Significantly less than 16, and you get deep fried images as the model weights just can't handle it, and if more than 16 (while not using sliding window context), you'll get images that become jumpy, as the positional encoding portion of the models past 16 frames is not good.
For greater than 16 frame animation, that can be pulled off without the jumpy issue by tricking AD into working at 16 frames at a time, diffusing all images in each step via a sliding context window, where it receives context_length-amount images at a time. And currently, the only context schedule I made available is one that causes looping of elements in the output - which can be very handy in some cases, but also not desired in others. I'm working on some alternate context schedules to alleviate the looping feature, and also investigating methods to keep the output from changing too much as the sliding context moves. FYI, the Uniform Context Options are the definition of how the sliding context window should behave, at some point there will likely be more Context Option nodes that will cause different behavior/outputs.
In summary, if you change the amount of frames/latents you're diffusing over, your output will change, too, due to the nature of AD. The only way to "checkpoint" your output at the moment is to render separate AD runs, and combining them afterwards. It's also possible to take frames if one output to make them influence the frames of another AD run.
Let me know if you have any more questions!

@Kosinkadink Thank you for great explanation. I've checked the same in A1111 and it even can't produce anything "meaningful" below 16 frames. 🫤 So "min 16 frames" is rule of thumb now👍
Reliability is my main struggle : Generating even 16 frames takes "minutes", generating 200+ frames is "dozens of minutes".
Before feeding AnimateDiff, I need to make sure that waiting for 10 min would result into something meaningful, so I my thinking is following:
For txt2vid:

Generate single image (2-5 sec)
a. repeat till get acceptable result
b. Pray it will be something similar on next step 🤞
Enable AnimateDiff for 16 frames (2-3 min roughly)
a. make sure it produces acceptable result

However generating "initial image" makes no sense, cause enabling AnimateDiff produces totally random results 🫠
I wonder if it is possible to generate txt2img, and then use the same parameters for animation in txt2vid scenario?

However generating "initial image" makes no sense, cause enabling AnimateDiff produces totally random results 🫠 I wonder if it is possible to generate txt2img, and then use the same parameters for animation in txt2vid scenario?

I've been generating a still image as "seed" latent, duplicate as many frames as needed, and set denoise to 0.8-0.9 for AnimateDiff. There will be loss of details (because AnimateDiff) but the result video resembles the initial image.

I've been generating a still image as "seed" latent, duplicate as many frames as needed, and set denoise to 0.8-0.9 for AnimateDiff. There will be loss of details (because AnimateDiff) but the result video resembles the initial image.

@justanothernguyen I think part about "duplicate as many frames as needed" I kind of replace with VHS_LoadVideo from https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite it produces all necessary latent images (this is probably out of my original question about txt2vid)
My original thinking (maybe still falsy one) was like:

Take video
a. take 1st frame
b. Apply ControlNet and generate "initial image"
Enable AnimateDiff with same parameters that were tested in step 1

Expected: animation that resembles visual style of step 1
Actual: animation is good, but style is veeery close to original video, but blurry.
"set denoise to 0.8-0.9 for AnimateDiff" I don't have denoise anywhere in AnimateDiff node.
Question: Which node are you using? Or it is scenario with multiple chained KSampler's? (like in: https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved#txt2img-w-latent-upscale-full-denoise-on-upscale)

@DmytroSokhach sorry for the confusion, by AnimateDiff denoise I mean the denoise value on the KSampler that is attached to AnimateDiff (single simple KSampler, not advanced or chained).
It also sounds like you're talking about rotoscoping from another video, is that it? Maybe try using IPAdapter from the first image?

@DmytroSokhach btw here's my attempt, it's a bit weird because AnimateDiff add a reddish hue to the video
Source video:
https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/1379769/ee1a81f0-a609-4176-b0af-d369f835f1e9
Rotoscoped video, first attempt (IPAdapter style transfer from the source video):
video_00085.webm
Workflow (drop it into comfyui): video_00085
Rotoscoped video, second attempt (IPAdapter style transfer from first generated frame), the details are a bit erratic
video_00086.webm
Workflow: video_00086
The workflows are adapted from here: https://civitai.com/models/158720?modelVersionId=182480 (people posted some NSFW result so be warned)

@justanothernguyen amazing examples! After couple hours I was able to make it work with smaller amount of frames like 48 or so! 🕺
Special thank you for "rotoscope" which was a new word for me 😳 I've started only 2 weeks ago with ComfyUI and probably was reinventing the wheel.
I've tried 240 frames and got strange situation on RTX 3090Ti, which is usually pretty descent. But here while in KSampler it is extremely slow, even with 12/12 steps (see below)
Question: Do I correctly understand that those 12/12 from last KSampler's are not reported properly to console and are actually should be multiplied by amount of frame_load_cap ?
First 2 samplers were more or less usual (2.5-4 it/s):
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| **25/25** [00:06<00:00,  3.86it/s]
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| **12/12** [00:04<00:00,  2.58it/s]

But next one was 100x slower (~47s/it)
[AnimateDiffEvo] - INFO - Loading motion module improvedHumansMotion_refinedHumanMovement.ckpt
[AnimateDiffEvo] - INFO - Using fp16, converting motion module to fp16
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (240) greater than context_length 16.
[AnimateDiffEvo] - INFO - Injecting motion module improvedHumansMotion_refinedHumanMovement.ckpt version v2.
Requested to load BaseModel
Loading 1 new model
unload clone 3
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 30/30 [23:47<00:00, 47.59s/it]

And next one became deadly slow (141.69s/it)
[AnimateDiffEvo] - INFO - Ejecting motion module improvedHumansMotion_refinedHumanMovement.ckpt version v2.
[AnimateDiffEvo] - INFO - Cleaning motion module from unet.
[AnimateDiffEvo] - INFO - Removing motion module improvedHumansMotion_refinedHumanMovement.ckpt from cache
[VideoHelperSuite] - INFO - Using fallback file for long metadata: 70391/32496
[AnimateDiffEvo] - INFO - Loading motion module improvedHumansMotion_refinedHumanMovement.ckpt
[AnimateDiffEvo] - INFO - Using fp16, converting motion module to fp16
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (240) greater than context_length 16.
[AnimateDiffEvo] - INFO - Injecting motion module improvedHumansMotion_refinedHumanMovement.ckpt version v2.
Requested to load BaseModel
Loading 1 new model
unload clone 3
 42%|\u2588\u2588\u2588\u2588\u258c     | 5/12 [11:44<16:31, 141.69s/it]

@DmytroSokhach haha yeah I just learned about rotoscoping few weeks ago myself. Looking around on Reddit and CivitAi helps.
Regarding sampling time, the first 2 Ksamplers are just still image so should be straight forward.
The following Ksamplers are attached to AnimateDiff. Those had to denoise more frames in total than the number of output frames, because there is a sliding context (check the Uniform Context Option node). The larger the context_overlap, the more frames need to be denoised.
Sampling time is not always linear to number of frames anyway. You can try sampling a batch of 1 latent vs a batch of 4 latents (with AnimateDiff not plugged in)


Install sqlite3.exe as part of sqlite3 port

The sqlite3.exe command line shell is very helpful when learning or using sqlite3.  The shell source comes with the amalgamated source drop, so let's build and install it as part of the vcpkg port.

Modeled after the bzip2 port: If SQLITE3_SKIP_TOOLS is not
set, build and install sqlite3.exe as part of the sqlite3
package.  sqlite3.dll must also be copied to the tools folder
for sqlite3.exe to be able to launch.
Tested on Windows 10.0.16299.125 with VS 2017 15.5.2 using: .\vcpkg install sqlite3:x86-windows sqlite3:x64-windows

The tool feature does not work with static builds:

vcpkg install sqlite3[tool]:x64-windows-static
-- Configuring x64-windows-static
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:56 (message):
Command failed: ninja;-v
Working Directory: VCPKG_ROOT/buildtrees/sqlite3/x64-windows-static-rel/vcpkg-parallel-configure
See logs for more information:
VCPKG_ROOT\buildtrees\sqlite3\config-x64-windows-static-out.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_configure_cmake.cmake:236 (vcpkg_execute_required_process)
ports/sqlite3/portfile.cmake:20 (vcpkg_configure_cmake)
scripts/ports.cmake:72 (include)

Log:

CMake Error at CMakeLists.txt:28 (install):
install TARGETS given no ARCHIVE DESTINATION for static library target
"sqlite3".


Dataset download issue

The link provided for downloading the data.zip file is not working. I mean the link fails to download the 34 GB data. It starts download initially and popups a "connection closed error" and starts downloading from the starting, eventually not downloading complete data. Please resolve this issue. Thanks in advance!

I started hosting the data on Kaggle now and updated the data link in readme. This should allow you to download individual files.


.ir tld

Some of data is missing on parser
It's 11nd most popular tld:
https://w3techs.com/technologies/overview/top_level_domain

package main

import (
  "os"
  "github.com/likexian/whois-go"
  "github.com/likexian/whois-parser-go"
)

func main() {
  if (len(os.Args) != 2) {
    println("Domain not passed");
  }
  domain := os.Args[1];
  result, err := whois.Whois(domain)
  if err == nil {
    result, err := whoisparser.Parse(result)
    if err == nil {
        println(result.Domain.Status)

        // Print the domain created date
        println(result.Domain.CreatedDate)

        // Print the domain expiration date
        println(result.Domain.ExpirationDate)
        println(result.Domain.NameServers)

        // Print the registrar name
        println(result.Registrar.Name)

        // Print the registrant name
        println(result.Registrant.Name)

        // Print the registrant email address
        println(result.Registrant.Email)
    }
  }
}

Many of fields not parsed see raw
http://whois.nic.ir/WHOIS?name=mhf.ir

Hello @mhf-ir
Thanks for your feedback, I have fixed this issue.
Please update to the latest version and have a try.
Verified tlds please refer to: https://github.com/likexian/whois-parser-go/blob/master/examples/README.md

Seems be better:
 This is the IRNIC Whois server v1.6.2.
 Available on web at http://whois.nic.ir/
 Find the terms and conditions of use on http://www.nic.ir/

 This server uses UTF-8 as the encoding for requests and responses.

 NOTE: This output has been filtered.