netty uses unsafe.Allocatememory() cause native memory leak

Hey,
In the production environment, we found that the memory occupied by the program has been increasing. Using the top command, we found that the value of RES has been increasing, even exceeding Xmx. We use jconsole.exe to observe that the number of threads and classes is stable, and the size of heap memory and meta space are within the normal range. Therefore, we begin to suspect that the program is constantly occupying native memory and is not released.
When we use gperftools to analyze the memory allocation occupied by the program, we will find that the method unsafe.Allocatememory0 occupies a very large proportion of memory and increases over time.

This is the original PDF file generated using gperftools analyzer memory
redisson-demo_1318.pdf
redisson-demo_1450.pdf
After debugging with debug, it is found that the thread redisson-netty calls unsafe.Allocatememory().

stack info
allocateMemory:614, Unsafe (jdk.internal.misc)
<init>:122, DirectByteBuffer (java.nio)
allocateDirect:317, ByteBuffer (java.nio)
allocateDirect:645, PoolArena$DirectArena (io.netty.buffer)
newChunk:621, PoolArena$DirectArena (io.netty.buffer)
allocateNormal:204, PoolArena (io.netty.buffer)
tcacheAllocateSmall:174, PoolArena (io.netty.buffer)
allocate:136, PoolArena (io.netty.buffer)
allocate:128, PoolArena (io.netty.buffer)
newDirectBuffer:378, PooledByteBufAllocator (io.netty.buffer)
directBuffer:187, AbstractByteBufAllocator (io.netty.buffer)
directBuffer:178, AbstractByteBufAllocator (io.netty.buffer)
ioBuffer:131, AbstractByteBufAllocator (io.netty.buffer)
allocateBuffer:140, MessageToByteEncoder (io.netty.handler.codec)
write:105, MessageToByteEncoder (io.netty.handler.codec)
write:76, CommandEncoder (org.redisson.client.handler)
invokeWrite0:717, AbstractChannelHandlerContext (io.netty.channel)
invokeWrite:709, AbstractChannelHandlerContext (io.netty.channel)
write:792, AbstractChannelHandlerContext (io.netty.channel)
write:702, AbstractChannelHandlerContext (io.netty.channel)
write:120, MessageToByteEncoder (io.netty.handler.codec)
write:45, CommandBatchEncoder (org.redisson.client.handler)
invokeWrite0:717, AbstractChannelHandlerContext (io.netty.channel)
invokeWrite:709, AbstractChannelHandlerContext (io.netty.channel)
write:792, AbstractChannelHandlerContext (io.netty.channel)
write:702, AbstractChannelHandlerContext (io.netty.channel)
write:115, ChannelDuplexHandler (io.netty.channel)
write:97, CommandsQueue (org.redisson.client.handler)
invokeWrite0:717, AbstractChannelHandlerContext (io.netty.channel)
invokeWriteAndFlush:764, AbstractChannelHandlerContext (io.netty.channel)
write:790, AbstractChannelHandlerContext (io.netty.channel)
writeAndFlush:758, AbstractChannelHandlerContext (io.netty.channel)
writeAndFlush:1020, DefaultChannelPipeline (io.netty.channel)
writeAndFlush:299, AbstractChannel (io.netty.channel)
sendData:123, CommandsQueue (org.redisson.client.handler)
write:100, CommandsQueue (org.redisson.client.handler)
invokeWrite0:717, AbstractChannelHandlerContext (io.netty.channel)
invokeWriteAndFlush:764, AbstractChannelHandlerContext (io.netty.channel)
write:790, AbstractChannelHandlerContext (io.netty.channel)
writeAndFlush:758, AbstractChannelHandlerContext (io.netty.channel)
writeAndFlush:808, AbstractChannelHandlerContext (io.netty.channel)
writeAndFlush:1025, DefaultChannelPipeline (io.netty.channel)
writeAndFlush:294, AbstractChannel (io.netty.channel)
send:169, RedisConnection (org.redisson.client)
async:215, RedisConnection (org.redisson.client)
async:187, RedisConnection (org.redisson.client)
async:183, RedisConnection (org.redisson.client)
channelActive:89, BaseConnectionHandler (org.redisson.client.handler)
invokeChannelActive:230, AbstractChannelHandlerContext (io.netty.channel)
invokeChannelActive:216, AbstractChannelHandlerContext (io.netty.channel)
fireChannelActive:209, AbstractChannelHandlerContext (io.netty.channel)
channelActive:1398, DefaultChannelPipeline$HeadContext (io.netty.channel)
invokeChannelActive:230, AbstractChannelHandlerContext (io.netty.channel)
invokeChannelActive:216, AbstractChannelHandlerContext (io.netty.channel)
fireChannelActive:895, DefaultChannelPipeline (io.netty.channel)
fulfillConnectPromise:305, AbstractNioChannel$AbstractNioUnsafe (io.netty.channel.nio)
finishConnect:335, AbstractNioChannel$AbstractNioUnsafe (io.netty.channel.nio)
processSelectedKey:707, NioEventLoop (io.netty.channel.nio)
processSelectedKeysOptimized:655, NioEventLoop (io.netty.channel.nio)
processSelectedKeys:581, NioEventLoop (io.netty.channel.nio)
run:493, NioEventLoop (io.netty.channel.nio)
run:989, SingleThreadEventExecutor$4 (io.netty.util.concurrent)
run:74, ThreadExecutorMap$2 (io.netty.util.internal)
run:30, FastThreadLocalRunnable (io.netty.util.concurrent)
run:834, Thread (java.lang)

After modifying the redisson configuration so that redisson is no longer used to connect to redis, we use gperftools to observe that there is no Unsafe_AllocateMemory0, and the memory occupied by the program remains stable and will not increase,so we now suspect that netty caused the program's native memory leak
Expected behavior
We hope to get your help so that we can use redisson without increasing the memory all the time
Actual behavior
Steps to reproduce
Minimal yet complete reproducer code (or URL to code)
https://github.com/redisson/redisson/files/7531565/redisson_demo.zip

unzip redisson_ demo.zip
modify start.sh to set your redis address
execute start.sh to start the jar

Netty version
4.1.48.Final
JVM version (e.g. java -version)
redis 5.0.12
redisson 3.12.4
springboot 2.4.2
openjdk version "11" 2018-09-25
OS version (e.g. uname -a)
Linux

jstack info
jstack.txt

You can change this by using -Dio.netty.maxDirectMemory=0

You can change this by using -Dio.netty.maxDirectMemory=0

We've tried this scheme, but it didn't work.
We even tried to use System.setProperty("io.netty.noUnsafe","true")
On the premise of using -Dio.netty.maxDirectMemory=0, if there is redisson-netty, we can still find that unsafe.allocatememory0 is called by using the tool gperftools, but this will not happen when we remove redisson-netty
This is the memory map obtained by using the tool gperftools
withRedisson-netty_11151930.pdf
noRedisson-netty_11151950.pdf

Note that if you use the System.setProperty(…), you have to do so before any netty classes are loaded. These properties are read once on class-load time, and then cached.

Note that if you use the System.setProperty(…), you have to do so before any netty classes are loaded. These properties are read once on class-load time, and then cached.

how about i use -Dio.netty.noUnsafe=true?

Yes, that works.

Yes, that works.

It's very disappointing.  After many tests, we found that -Dio.netty.noUnsafe=true didn't work, and the test results are the same as this https://github.com/netty/netty/issues/11835#issuecomment-968850027

is redision shading netty by any chance ?

is redision shading netty by any chance ?

I once submitted this issue to redisson,https://github.com/redisson/redisson/issues/3945, but they said that the memory allocation is controlled by netty, so I submitted this issue here

is redision shading netty by any chance ?

Through debugging class io.netty.util.internal.platformdependent, I found that these two parameters are valid and are not overwritten by redisson

After using two parameters:

You can't see the redisson correlation in the deeper code

@caizy-2020 if you use redisson-all netty will be shaded

We use reisson,but we haven't used reisson all
<dependency>
    <groupId>org.redisson</groupId>
    <artifactId>redisson</artifactId>
    <version>3.12.4</version>
</dependency>

I have used -Dio.netty.leakDetectionLevel=paranoid, but no LEAK is found in the log. Does this mean that there is no netty memory leak?

When redisson is removed, unsafe_Allocatememory0 does not exist in the memory allocation graph, but it exists when redisson is used. According to this phenomenon, and through my debugging of the demo program, it can be determined that unsafe_Allocatememory0 is called and unreleased stack information is among the following three
stack_1.txt
stack_2.txt
stack_3.txt
All of them are called unsafe.allocatememory0 by netty
So this is very confusing. I can't think of any other causes of native memory leakage. Is it caused by the open JDK?

Did you ever ran into Problems because of this? It is possible that the resource are not cleaned because there is no reason for it. The Point of the buffer pooling is that buffer (or areas of it) can be reused so they are not deallocated as they are shared. You can use an UnpooledByteBufAllocator if you want to avoid that

Did you ever ran into Problems because of this? It is possible that the resource are not cleaned because there is no reason for it. The Point of the buffer pooling is that buffer (or areas of it) can be reused so they are not deallocated as they are shared. You can use an UnpooledByteBufAllocator if you want to avoid that

So you mean like this?
-Dio.netty.allocator.type=unpooled

@caizy-2020
I think that the leak analysis is pointing to something that's not a leak at all: Netty use native (and heap) memory pooling and that looks like a "leak" indeed, because memory will be released only on application stop.
This issue can be closed I think.

@caizy-2020 I think that the leak analysis is pointing to something that's not a leak at all: Netty use native (and heap) memory pooling and that looks like a "leak" indeed, because memory will be released only on application stop. This issue can be closed I think.

I understand what you mean, but isn't it a problem that the memory occupied is becoming larger and larger?
We found that the memory occupied by method unsafe.Allocatememory0 becomes larger and larger over time, and the memory occupied by the whole application is also increasing

We found that the memory occupied by method unsafe.Allocatememory0 becomes larger and larger over time, and the memory occupied by the whole application is also increasing

If the leak detector hasn't found anything abnormal it just means that the memory wasn't enough to cope with the load: similarly to a java heap, that grows to cope with the load and fail with OOM if the max capacity isn't enough.

but isn't it a problem that the memory occupied is becoming larger and larger?

That's a smart question indeed: we (I'm a user of Artemis, a messaging broker relying on Netty for networking layer) found that there are cases where it would be nice, under certain cicustances, that netty will return back memory (similarly to some recent changes on G1GC, Shenandoah and ZGC) to the operating system, and I'm  sure the allocator can be changed (making it to diverge from what jemalloc does) in order to do it BUT this is not implemented.
Feel free to add a RFC and I will happily try to help there, because I thunk this can be a useful feature for containerized apps :)

If the leak detector hasn't found anything abnormal it just means that the memory wasn't enough to cope with the load: similarly to a java heap, that grows to cope with the load and fail with OOM if the max capacity isn't enough.

We have monitored the Java heap memory for a long time and found no abnormality. At the same time, there has been no OOM exception in the production and test environment, because the heap memory has always been small. Now the problem in production is that the system memory is insufficient because the memory occupied by each application is gradually becoming larger, so the operating system will automatically kill other applications

Did you ever ran into Problems because of this? It is possible that the resource are not cleaned because there is no reason for it. The Point of the buffer pooling is that buffer (or areas of it) can be reused so they are not deallocated as they are shared. You can use an UnpooledByteBufAllocator if you want to avoid that

@SplotyCode Fortunately, after setting -Dio.netty.allocator.type=unpooled , no method unsafe.Allocatememory0 has been called in the memory allocation diagram obtained through gperftools , but we are still monitoring the change of memory occupied by the application
But I have another question. When we use unpool mode, will this reduce netty performance? Will it affect the memory application or release of netty?

Yeah it will reduce netty performance that's the reason this feature exists.


Expose ValidatedTransaction from transaction pool

This is required to make a tx pool wrapper for some custom transaction validation, specifically
required to make a wrapper around the LocalTransactionPool implementation (https://github.com/subspace/subspace/blob/f54881a9b5/crates/subspace-service/src/pool.rs#L232-L269).
There probably is some other nicer way to make a tx pool wrapper but this can be the quickest one with minimal changes to make it possible.
Related: https://github.com/paritytech/substrate/discussions/11520

bot merge


Remove unnecessary submodules

Running eval here: https://github.com/uwsampl/lakeroad-evaluation/actions/runs/6344398828/job/17234697699

Looks like there are still failures to resolve, see the log when it finishes.

Shoot, the logs didn't save for some reason. But unsurprisingly it seemed like most of the errors came from Verilator tasks.

I'm just going to merge this and deal with errors as they come!


Consensus and Api versioning - APIs

Currently rebased on top of #1575 to avoid merge conflict. See only the last commit.
Implement the basic types and module-related primitves to support
versioning.
These are currently not used for anything, and here only for
moving forward with the discussion from https://github.com/fedimint/fedimint/issues/1548.

@elsirion
I haven't really thought through all the details, and I hope exactly that looking at the code will make the vision and discussion more precise.
I agree that in many cases importing multiple versions of module codebase is way better idea than cramming it together glued with bunch of if statements. But then - won't majority of cases be trivial? E.g. no changes?
One thing that is not clarified (in my head and in the code) is if module will have its own counter for consensus versions and then there be a global one for core Fedimint protocol. I guess it has to be both.

I don't think the module consensus version has to be a function of the FM consensus version. Either the module needs to be adapted to work with a newer FM version (likely) then it can change its own version or not, then there is no sense in changing the module version.

Either the module needs to be adapted to work with a newer FM version (likely) then it can change its own version or not, then there is no sense in changing the module version.

That means it is a function of fm consensus version, no?
We at least should aim at not letting users make accidental mistakes.
So
fn version(core: CoreConsensusVersions) -> [ModuleConsensusVersion]

If FM changes consensus from 2 to 3, and module doesn't care then:
module_init.versions(2) == module_version(3)

If it does, then it will have it's own version change and
module_init.versions(2) + 1 == module_version(3)

By having this function we can check for the user that all the modules are compatible with FM consensus changes. Initially just bunch of checks and crashing with an error message explaining it, but in the future we might even support automated scheduled consensus changes where guardians use UI to agree on the consensus change that gets scheduled and "switched into" automatically at the designed epoch.

Maybe I'm wrong here, but in my mind a new consensus version of Fedimint is a new major code version, so the old module wouldn't compile with the new Fedimint anyway. So the
module_init.versions(2).last() + 1 == module_init.module_version(3).first()

case doesn't really exist.
I also don't get what first/last of the returned slice would be. What's the semantic of the returned array? Supported versions? But how would a version be even selected when e.g. processing an output? We don't have version fields in all our consensus relevant structs rn, are we planning on changing that?
So far I assumed we just treat a new consensus version of a module as a completely new module with its own instance id and a related kind and let clients figure out migration. I would like to have a version field in the module kind type for that.

But how would a version be even selected when e.g. processing an output?

Versions are on ModuleGen so what it means is: this module codebase can instantiate module instance for the following module consensus versions. This could be used e.g. in the Admin UI to tell user that their codbase is/isn't compatible with module consensus upgrade guardians are trying to schedule, and prompt them to upgrade their binary.
In the simplest case, a module returns just one version meaning - this code base supports only this module consensus version.

Maybe I'm wrong here, but in my mind a new consensus version of Fedimint is a new major code version, so the old module wouldn't compile with the new Fedimint anyway.

I don't think this is necessary. Initially we probably will tie them together for simplicity, but our module code API surface layer is somewhat narrow, and nicely typed.
We might at some point change some consensus rules which most (all) of the module won't care about, and they won't have to change a single line of code, other than adding the new consensus to their list of supported FM consensus version, purely as a safeguard.
On the other hand, we might want to change some module trait APIs, without actually changing consensus rules.

So far I assumed we just treat a new consensus version of a module as a completely new module with its own instance id and a related kind and let clients figure out migration. I would like to have a version field in the module kind type for that.

I guess there's an overlap here. You might be right. I need to think about about it a bit longer and run thought more scenarios for theoretical modules, and how practical it is. For a module like mint such a static migration seems very nice, but I wonder if it will always be sufficient.

Currently we have a CODE_VERSION which is stored in the ServerConsensusConfig so that if they don't match, servers will fail to join consensus. How would these new versions interact with the existing one?

Replace. We should compare consensus versions (global FM one, and each module's instance). If the code says it's compatible, we believe it.

@elsirion

I guess there's an overlap here. You might be right. I need to think about about it a bit longer and run thought more scenarios for theoretical modules, and how practical it is. For a module like mint such a static migration seems very nice, but I wonder if it will always be sufficient.

OK, so I think it's not hard to come up with examples where rolling over to new instance is not  really desireable / possible.
Let's say we have a smart contract module, Ethereum style. Let's ignore for a second if that's a bad idea to do it. Since the contracts in the existing instance interact with other contracts, this creates a strong network effect.
Migrating to another instance with new rules in slow motion is close to impossible. The module becomes it's own side-chain, not unlike any alt-coin.
Similarly if the module is storing a lot of long term backup (like a threshold encrypted-backup), a module consensus hard-fork is preferred over bothering users with migration.
So, yeah - I don't think rolling over is going to work in every case. Thought I do agree that when possible it's very elegant. The module consensus upgrade is orthogonal to it. A mint-v2 KIND becomes a completely different module kind than mint-v1 (if we do it through just different KIND string or an extra "module kind version" is implementation detail. It will be up to module developers to decide if rolling over to a new kind&instance is preferable over "module consensus hard-fork", and we will guide the user through this difference in the UIs.

Currently we have a CODE_VERSION which is stored in the ServerConsensusConfig so that if they don't match, servers will fail to join consensus. How would these new versions interact with the existing one?

Replace. We should compare consensus versions (global FM one, and each module's instance). If the code says it's compatible, we believe it.

So we could run with different binary versions potentially?  Seems like that could be error-prone.
If this is replacing CODE_VERSION can we also remove it and put these versions into the consensus config then?

So we could run with different binary versions potentially? Seems like that could be error-prone.

Eventually we have to discard the training wheels and graduate to the big league of distributed software.
Right now the stakes are low, so it is a good opportunity to learn and gain some experience.
Whenever we touch anything that smells like breaking consensus, we just increment the corresponding number and it comes to the same thing.

If this is replacing CODE_VERSION can we also remove it and put these versions into the consensus config then?

I am not doing it in this PR, but yes, that would exactly be the next step. We start comparing consensus versions between peers, and module code compatibility against the federation consensus and we error out if anything is wrong.

Sounds reasonable, will look at it again with fresh eyes tomorrow, not too opposed to land in this form anymore. I still think one code version only supporting one consensus version would be more elegant, but this flexibility is likely enough to be useful in the not too distant future to make sense.

Dev call: add test to CI to assert that versions didn't change if not declared in some way

I amended with changes corresponding to some conclusions from the discussion here.

Let's try this! Next steps are inclusion in cfg/client side checks?

I think so, yes. And we have to remember to try to increment these numbers when we break api/consensus rules. Not a big deal if we forget, but we should try.


Cluster ips policy updates

Updates to the sample policies in newly merged How to, which were not accurate. Also, minor formatting updates.

Spike has reviewed
Ready to merge

Deploy preview for calico ready!
Built with commit f0d67a173ab9265a51f89bdbadfdf1b4b9b205f6
https://deploy-preview-2761--calico.netlify.com


Catalog build fails with UglifyJs compile error

When recently changing our catalog project, the build is no longer working giving an UglifyJs error:
✖ Failed to compile.

static/catalog.1b819263.js from UglifyJs
Unexpected token: name (slug) [./node_modules/github-slugger/index.js:23,0][static/catalog.1b819263.js:43626,6]

There was no change to our project except from markdown files.
A recent version of github-slugger using ES6 seems not to be handled by Babel and breaks UglifyJs.

I have the same Problem!

When recently changing our catalog project, the build is no longer working giving an UglifyJs error:
✖ Failed to compile.

static/catalog.1b819263.js from UglifyJs
Unexpected token: name (slug) [./node_modules/github-slugger/index.js:23,0][static/catalog.1b819263.js:43626,6]

There was no change to our project except from markdown files.
A recent version of github-slugger using ES6 seems not to be handled by Babel and breaks UglifyJs.

Could you fix the problem?

Hi Guys,
You need to use npm shrinkwrap and enforce in catalog github-slugger 1.1.3 as dep :)

Hi Guys,
You need to use npm shrinkwrap and enforce in catalog github-slugger 1.1.3 as dep :)

@aranwe Could you provide more detail on how to do this please? @grossbart , @blowery Could you maybe just alter the package to only use github-slugger 1.1.3 and no higher for now?


QA GUI Defect Log

This ticket is a holder for discovered GUI defects in the QA process.
### Tasks
- [x] ⭐Dropdown showing the code not the labels.
- [ ] Parse error on `[ Aggregate_Column.Sum (expr "")]` resulted in incorrect placement within `aggregate`.
- [ ] ~No longer showing escaped text when not editing a text node. @vitvakatu~
- [ ] https://github.com/enso-org/enso/issues/9379
- [x] Dropdown selected item is indented. - #9516
- [ ] ~Showing the dropdown on a text box with values (e.g. column names) is very fiddly.~
- [ ] https://github.com/enso-org/enso/issues/9357
- [ ] Zoom steps are not sensible (100 -> 71 -> 50 ...) #9585
- [ ] https://github.com/enso-org/enso/issues/9445
- [ ] Parenetheses are sometimes not rendered in the node

Dropdown:

Textbox:

Join Video:
https://github.com/enso-org/enso/assets/4699705/768620fe-21eb-41f2-bf1e-1cb558303aab

Parse Error: [ Aggregate_Column.Sum (expr "")]

Zoom levels should be similar to Chrome's one: 500/400/300/250/200/175/150/125/100/90/80/75/67/50/33/25/10

Dropping two tasks from the list:

“No longer showing escaped text when not editing a text node.” – not a priority right now, should rather be fixed by #8753 and #8754, allowing to edit the code of the specific text literal.
“Showing the dropdown on a text box with values (e.g. column names) is very fiddly.” – seems to be fixed now.

All tasks done


replaced the API call to the registry, with the couchDB-API

Problem:
The list of ember-addons doesn't get updated.
NPM removed the API endpoint, which "npm-registry" uses to load all names of the npm packages with the keyword "ember-addon".
Solution
There is still the API for the couchDB of NPM at https://skimdb.npmjs.com/registry
I will add some code comments. I hope this PR is alright in that style.
Oh and of course I couldnt test the data-uploading, etc. I just wrote the output to a JSON file locally to check the output.

@GerritSommer Thanks for the PR. I'll test it today.

I forgot a comma and removed some spaces from the url I added.
Sorry, was my mistake

Finally merged and deployed, thanks! @GerritSommer


[SPARK-21181] Release byteBuffers to suppress netty error messages

What changes were proposed in this pull request?
We are explicitly calling release on the byteBuf's used to encode the string to Base64 to suppress the memory leak error message reported by netty. This is to make it less confusing for the user.
changes proposed in this fix
By explicitly invoking release on the byteBuf's we are decrement the internal reference counts for the wrappedByteBuf's. Now, when the GC kicks in, these would be reclaimed as before, just that netty wouldn't report any memory leak error messages as the internal ref. counts are now 0.
How was this patch tested?
Ran a few spark-applications and examined the logs. The error message no longer appears.

Test build #78462 has started for PR 18392 at commit 21b4c0b.

Test build #78462 has finished for PR 18392 at commit 21b4c0b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78462/
Test PASSed.

Test build #78471 has started for PR 18392 at commit acabec0.

BTW should this go on master first? Or is this not a problem on master for some reason?
(If this doesn't happen on master nor 2.2, please state so in the bug, otherwise, please open a PR against master.)

Test build #78471 has finished for PR 18392 at commit acabec0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78471/
Test PASSed.

I haven't got a chance yet to test it against 2.2 and the master. I will update it after running against both the builds.

Oops yeah @dhruve you need to open vs master and this can be closed.
You can just do it and see what the tests say here.

@vanzin Tested this against branch-2.2 and master and I can see the issue manifest in both.
@srowen Have opened a PR against master https://github.com/apache/spark/pull/18407
Closing this one. Once merged can be backported to 2.1 and 2.2.
Thanks for reviewing.


Fix typo

Initial checklist

[x] I read the support docs
[x] I read the contributing guide
[x] I agree to follow the code of conduct
[x] I searched issues and couldn’t find anything (or linked relevant results below)
[x] If applicable, I’ve added docs and tests

Description of changes
Fixes a minor typo in readme.md

thanks!


doing the minimum to be a contributor <3

Thank you <3 and great branch name. Please merge when you're ready and delete the branch when you're done


Environment variables in table configs are revealed

Context
Pinot allows replacing environment variables in tables configs as documented
This can be used to replace secrets configured in table config. For example to ingest from kafka cluster that has authorization one might configure:
{
  "tableIndexConfig": {
    "streamConfigs": {
      "sasl.jaas.config": "${KAFKA_SASL_CONFIG}"
    }
  }
}

Problem
While environment variable replacement works, it reveals the secret when showing table config in UI or API. This defeats the purpose of having secrets replaced with environment variables.
To reproduce

Follow startup guide:

# Clone a repo
$ git clone https://github.com/apache/pinot.git
$ cd pinot

# Build Pinot
$ mvn clean install -DskipTests -Pbin-dist

Export secret

export MY_SECRET=to_not_reveal

Start Pinot

# Run the Quick Demo
$ cd build/
$ bin/quick-start-batch.sh

Open sample table:
http://localhost:9000/#/tenants/table/airlineStats_OFFLINE
Edit its config, change metadata and save:

    "metadata": {
      "customConfigs": {
        "regular": "some-value",
        "secret": "${MY_SECRET}"
      }
    },

After saving, the secret is revealed in the same UI:

    "metadata": {
      "customConfigs": {
        "regular": "some-value",
        "secret": "to_not_reveal"
      }
    },

Check the state in Zookeeper, it is saved correctly:

"metadata": "{\"customConfigs\":{\"regular\":\"some-value\",\"secret\":\"${MY_SECRET}\"}}",

Expected behaviour
Environment variables should be replaced only internally within Pinot. They should not be replaced when stored in ZK, nor when revealed in UI or via API call.

Thanks for reporting the issue. We need to mask the value in rest API. cc @xiangfu0 @soumitra-st


Muntants survive despite tests guarding well

I've got standard Angular 6 project, created with ng new.
I've just bumped karma to "^2.0.5".
Now following mutation (providedIn "root" is substituted with ""):
import { Injectable } from '@angular/core';

@Injectable({
  providedIn: ""
})
export class LocalStorageService {

  get(key) {
    return localStorage.getItem(key);
  }

  set(key, value) {
    localStorage.setItem(key, value);
  }
}

is being reported as surviving this test:
  it('should be decorated with @Injectable annotation', () => {
    const ngInjectableDef = (<any>LocalStorageService).ngInjectableDef;
    expect(ngInjectableDef).toBeDefined();
    expect(ngInjectableDef.providedIn).toEqual('root');
  });

which is simpossible, because that code does not even compile:
ERROR in src/app/common/localStorage.service.ts(3,13): error TS2345: Argument of type '{ providedIn: ""; }' is not assignable to parameter of type '({ providedIn: Type<any> | "root"; } & ValueSansProvider) | ({ providedIn: Type<any> | "root"; } ...'.
  Type '{ providedIn: ""; }' is not assignable to type '{ providedIn: Type<any> | "root"; } & ClassSansProvider'.
    Type '{ providedIn: ""; }' is not assignable to type '{ providedIn: Type<any> | "root"; }'.
      Types of property 'providedIn' are incompatible.
        Type '""' is not assignable to type 'Type<any> | "root"'.

package.json:
{
  "name": "angular-testing-frontend",
  "version": "0.0.0",
  "scripts": {
    "coverage": "ng test --code-coverage --watch=false",
    "postcoverage": "if test ${CI_NAME+x} ; then COVERALLS_SERVICE_NAME=Codeship COVERALLS_GIT_COMMIT=$CI_COMMIT_ID COVERALLS_SERVICE_JOB_ID=$CI_BUILD_ID cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js; fi",
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "mutation-test": "stryker run",
    "lint": "ng lint",
    "e2e": "ng e2e",
    "webdriver-manager-update": "webdriver-manager update --chrome"
  },
  "private": true,
  "dependencies": {
    "@angular/animations": "^6.0.3",
    "@angular/common": "^6.0.3",
    "@angular/compiler": "^6.0.3",
    "@angular/core": "^6.0.3",
    "@angular/forms": "^6.0.3",
    "@angular/http": "^6.0.3",
    "@angular/platform-browser": "^6.0.3",
    "@angular/platform-browser-dynamic": "^6.0.3",
    "@angular/router": "^6.0.3",
    "core-js": "^2.5.4",
    "rxjs": "^6.0.0",
    "zone.js": "^0.8.26"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.7.4",
    "@angular/cli": "~6.1.4",
    "@angular/compiler-cli": "^6.0.3",
    "@angular/language-service": "^6.0.3",
    "@types/jasmine": "~2.8.6",
    "@types/jasminewd2": "~2.0.3",
    "@types/node": "~8.9.4",
    "codelyzer": "~4.2.1",
    "coveralls": "3.0.2",
    "jasmine-core": "~2.99.1",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "~2.0.5",
    "karma-chrome-launcher": "~2.2.0",
    "karma-coverage-istanbul-reporter": "~2.0.0",
    "karma-jasmine": "~1.1.1",
    "karma-jasmine-html-reporter": "^0.2.2",
    "karma-webdriver-launcher": "1.0.5",
    "protractor": "~5.3.0",
    "stryker": "0.27.1",
    "stryker-api": "0.19.0",
    "stryker-html-reporter": "0.15.3",
    "stryker-karma-runner": "0.16.5",
    "stryker-typescript": "0.12.2",
    "ts-node": "~5.0.1",
    "tslint": "~5.9.1",
    "typescript": "~2.7.2"
  }
}

The solution is to downgrade karma to "^1.7.1".

We're closing this issue for now. If it's still present in the latest version of Stryker, let us know!


Does GitVersion update your assembly/csproj automatically via the GitReleaseManager.yaml?

If so, where is that done? I do not see the GitVersion MSBuild NuGet, or the GitVersion.exe command anywhere in the project.
How is the package version incremented? What am I missing?

I notice we are using this: https://github.com/adamralph/minver
And I notice theres a build task here https://github.com/IdentityServer/IdentityServer4/blob/master/src/Directory.Build.targets#L46
But im not sure how this eventually makes it way into the final nuget package.

MinVer is doing all the magic. Check their docs.
GitReleaseManager is unrelated (it's for release notes).


feat: upgrade user account capability

:tada: This PR is included in version 0.52.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[Toolchain] Exclude UI related codes from coverage analysis

UI related codes are hard to be managed about to code quality.
This commit excludes them.
ONE-vscode-DCO-1.0-Signed-off-by: Seok NamKoong<EMAIL_ADDRESS>

This PR is no more needed, close :)


No value on changeAction

In the function taken from changeAction I'm expecting two arguments to be fullfilled: model and value or at least is what I understood from the documentation. In a checkbox only the model seem to have a value, while value is undefined. Is it normal?

What does your store look like? It could be that the initial value of your model is undefined.

You're right. This is the store:
const reducers = combineReducers({
    ...createForms({
        options: [],
    }),
    auth,
    routing: routerReducer
})
const store = createStore(reducers, {}, middlewares); // with thunk in middlewares

Then, the component looks like this (only the relevant parts)
{
this.props.payment_options.map((option, index) =>
  <Control.checkbox
  key={index}
   model={track(`options[].enabled`, {id:   option.id}), 'enabled')}
   changeAction={(model, value) => dispatch => {
   // async code....
   dispatch(actions.toggle(model))
}}
/>
)
}

and the array of options, once fetched contais object like
{
id: 1,
name: '...'
}

This looks invalid:
   model={track(`options[].enabled`, {id:   option.id}), 'enabled')}

But besides that, is the component working? Since your model value is undefined, it's going to be passed undefined as the second argument to the change action, but for a checkbox, that shouldn't matter - it's falsey so actions.toggle will just set the model to true.

Sorry, I cutted the code to simplify it. Is
model={track(`options[].enabled`, {id:   option.id})}

and
{
id: 1,
name: '...',
enabled: true/false
}

Yes, it is working. Enabled is not undefined but actually it works even if it is undedfined


Running Simple Query through Java API But not getting Result

OrientDB Version: 3.0.30
Java Version: 1.8
OS: windows 10

        try(ODatabaseSession db = pool.acquire()){
            String query = "select * from Song where out_SungBy.in.id=?";
            OResultSet rs = db.query(query,artistId);

            while (rs.hasNext()) {
                OResult item = rs.next();
                Artist ar = new Artist();
                System.out.println("count : "+ item.getProperty("Title"));
                data.add(ar);
            }
            ObjectMapper mapper =new ObjectMapper();
            JsonArray ob = new JsonArray(mapper.writeValueAsString(data));
            promise.complete(ob);
        }
        catch(Exception e){
            e.printStackTrace();
            promise.fail(e.fillInStackTrace());
        }

Trying to Run this simple Query but result set is empty, same query is working fine from OrientDB Studio.
if i change the query to  "select * from Song" than its working fine.

Hi @somnath157
I'd say out_SungBy is a collection, so you have to use CONTAINS instead of =
String query = "select * from Song where out_SungBy.in.id CONTAINS ?";

In general, I strongly discourage you from using the low level mechanics of edge representations, and use the normal graph API instead:
String query = "select * from Song where out("SungBy").id CONTAINS ?";

Thanks
Luigi

Hi @luigidellaquila
Thanks worked with using Contains.
But Facing one more issue here
out('SungBy').in is returning null while out_SungBy.in return values correctly.
is there some difference?

you don't need the .in after out("SungBy")
in fact, out("SungBy") is equivalent to out_SungBy.in

Okay and what if i have to access a parameter of the edge than this won't work so i will have to use out_SungBy.fieldName
or is there any better way?

Hi @somnath157
you can access the edge with outE("EdgeClassName") (see outE() instead of out())
Then you can access to the following vertex with inV()
So the following are equivalent
out("EdgeClass")

outE("EdgeClass").inV()

To filter based on edge properties, you can use square bracket filtering:
outE("EdgeClass")[foo > 5].inV()

I hope it helps
Thanks
Luigi

Thanks
Somnath


Personal Finance Dashboard: Comprehensive Financial Management Tool

Personal Finance Dashboard: Comprehensive Financial Management Tool
Personal Finance Dashboard, is a web-based application designed to help users efficiently manage and visualize their personal finances. It features tools for budgeting, expense tracking, savings goal monitoring, investment tracking, and financial insights. Built with HTML, CSS, JavaScript, and data visualization libraries like Chart.js, the dashboard provides a comprehensive and interactive way for users to understand and control their spending patterns and financial goals.
The project introduces a modular structure with interactive sections:

Budgeting: Users can set a monthly budget and track spending progress.
Expense Tracking: Daily expenses can be logged and categorized, providing a clear picture of spending habits.
Savings Goals: Enables users to define and monitor specific savings targets, displaying progress visually.
Investment Tracking: Allows tracking of multiple investments, displaying portfolio distribution in an intuitive format.
Financial Insights: Generates insights and recommendations based on spending patterns.

Many existing personal finance tools either lack customizable tracking for personal budgets, investments, and savings goals or require complex integrations. This project addresses the common challenge of fragmented financial data management, where users struggle to see a consolidated view of their finances across different categories. The dashboard provides an all-in-one, simplified solution for tracking and visualizing financial data in real-time.

Addresses the issue #2299

@Dkghuser resolve the conflict

sure, i will resolve and update you soon

@iamrahulmahato I've reviewd and resolved the conflict with the file.
Please check with it,
Thank you.

Can I submit a new PR, addressing the same issue, as the issue seems to be resolved again. I'll submit a new PR for this issue, within some time.

ok make a pr i am closing


Changes design for access activity, replaces with new icons

Description
Fixes #1558
Notes (out of scope, known isues, hints for reviewing code, ...)  (optional)
...
Test cases

[ ] Case 1:

Given I am the usual user
When I go to this page
Then I see new design of access activity
Then I click on header
Then I see the panel is collapsed

[ ] Case 2:

Given I am the usual user
When I go to this page
Then I see no access message and edit permissions button is enabled

[ ] Case 3:

Given I am the usual user
When I go to this page
Then I see no access message and edit permissions button is disabled

The html "title" (what is displayed when leaving the mouse for a moment) should not just be "can view" but probably "Can view: ".

It's because of html attr title (which is part of html), so need to change naming and to use more uniq name of attr. The place where we are using title currently doesn't have logic for make it full as you offered -> "Ca view: Sequence"


Combine Template with modifications (--cloud-init-user-data, --disk-size)

especially because of this:
!!IMPORTANT NOTE: At the moment, the create sub-command supposes a Network vlan1 already exists!!
But also because there might be some vms only running with special configs.
it would be cool to be able to pass all of these while using a template:
--disk-size
--cpus
--memory
--cloud-init-user-data
--cloud-init-network-data
do you have an example of the usage for how to pass this data?:
--cloud-init-user-data
#cloud-config
runcmd:

$var1 = testvalue1
wget https://mytestpagewichdeliversa.sh
chmod +x ./a.sh
./a.sh
--cloud-init-network-data
network:
version: 2
renderer: networkd
ethernets:
enp1s0:
dhcp4: no
addresses: [<IP_ADDRESS>/24]
gateway4: <IP_ADDRESS>
nameservers:
addresses: [<IP_ADDRESS>, <IP_ADDRESS>]

@belgaied2 hey there, is this smh possible or should I start a fresh project?

@belgaied2 hey there
do you think you can do something in this relation the next few days?

Hi @ne0YT ,
I am starting to work on this. I will update you soon.

Hi @ne0YT ,
This should be implemented in the commit 1dfb09c. However, no release yet. Feel free to check out the code and compile it to test it.

@belgaied2 Im not able to build it:
go build -o harvester-cli.exe
github.com/c9s/goprocinfo/linux
....\go\pkg\mod\github.com\c9s\goprocinfo@v0.0.0-20210130143923-c95fcf8c64a8\linux\disk.go:15:16: undefined: syscall.Statfs_t
....\go\pkg\mod\github.com\c9s\goprocinfo@v0.0.0-20210130143923-c95fcf8c64a8\linux\disk.go:16:17: undefined: syscall.Statfs

https://github.com/belgaied2/harvester-cli/issues/36


Memory leak issue

Hello! I’ve been using an app with Ag-Grid component and noticed that it was taking up too much server space. After doing some investigation, it seems that using the Ag-Grid component causes a memory leak.
In short, my app loads some data from csv/excel file to pandas dataframe and then uses Ag-Grid table with checkboxes to dynamically select rows.
Here’s the code snippet:
import streamlit as st
import pandas as pd
from st_aggrid import AgGrid
from st_aggrid.grid_options_builder import GridOptionsBuilder
from st_aggrid import GridUpdateMode, DataReturnMode

@st.cache
def load_data():
    my_file = 'data_source.xlsx'
    df = pd.read_excel(my_file)
    return df

data = load_data()

gb = GridOptionsBuilder.from_dataframe(data)
gb.configure_default_column(enablePivot=False, enableValue=False, enableRowGroup=False)
gb.configure_selection(selection_mode="multiple", use_checkbox=True)
gridOptions = gb.build()

response = AgGrid(
    data,
    gridOptions=gridOptions,
    enable_enterprise_modules=False,
    update_mode=GridUpdateMode.SELECTION_CHANGED,
    data_return_mode=DataReturnMode.FILTERED_AND_SORTED,
    allow_unsafe_jscode=True)

response_df = pd.DataFrame(response["selected_rows"])

I used memory profiler and it is apparent that the app does not release the memory when using AgGrid (images attached). I went through the AgGrid documentation, but didn’t find anything on this topic. Am I missing something? Does anyone encounter such issue?
Many thanks!

UPDATE: The issue has been solved by updating Python version & libraries used in the app


End of timer proc starts next timer. Went to com_ptr's for hidutil su…

WMI BIST Method Implementation
Implements a fake Built-In Self Test (BIST) functionality.
The BIST first causes the taillight to go red. Next, the taillight "reboots" by going black. Finally, the BIST status is shown by the taillight becoming green.
The existing variable WMIInstance was renamed to WMIReportInstance. There are now two WMIInstances, and I thought leaving the variable alone would cause confusion.
HidUtil was updated to drive the BIST method. The existing code is not capable of calling a WMI method. Toaster exe was ported and updated to use the modern com_ptr RAII.
Note that the first compile of HidUtil now will tell you how to install the winrt package using nuGet.
For simplicity, if a BIST is requested while a BIST is in progress, STATUS_DEVICE_BUSY will be returned.
HidUtil updated with the /bist switch. This switch causes and waits for the result from the taillight BIST.
Updated TailLight.ps1 to invoke the BIST functionality. This file just demonstrates functionality. It is useful for verifying synchronization.

@ManOnTheMountainTech The driver fails to load when I try to test your PR on my computer. I'm observing the following WinDBG errors when attaching the mouse:
TailLight: DriverEntry - WDF version built on May  2 2024 06:35:15
TailLight: PdoName: \Device\000000dd
TailLight: EvtDriverDeviceAdd: "WdfTimerCreate" failed 0xc0200210(1)
TailLight: DriverUnload.

Can you please either confirm that the PR works on your end or update the PR to fix this problem?

A challenging error...I'll be digging into what can cause this.
//
// MessageId: STATUS_WDF_INTERNAL_ERROR
//
// MessageText:
//
// An internal error has occured.
//
#define STATUS_WDF_INTERNAL_ERROR        ((NTSTATUS)0xC0200201L)

A challenging error...I'll be digging into what can cause this.
//
// MessageId: STATUS_WDF_INTERNAL_ERROR
//
// MessageText:
//
// An internal error has occured.
//
#define STATUS_WDF_INTERNAL_ERROR ((NTSTATUS)0xC0200201L)

The error seem to be triggered by the attribs.SynchronizationScope = WdfSynchronizationScopeDevice line when I test. I'm surprised that you didn't experience this problem before submitting this PR.

Closing PR since I've just merged #72 that's a cleaned up and simplified version of this proposal.

There's one case statement in the WDF source that has this error. There is a path where a fileobject is created that might fail in the WMI package. A WDF log dump might be helpful. Also, what are your driver verifier settings?


(API) Make sure IDN is properly working for any TLD

Raised https://github.com/hjr265/node-whois/pull/29 to fix incorrect passing of the punycode option

This can be closed as soon as theres a new NPM package :)

Sorry for the delay.. I have been traveling last couple of days. I will do this within the next 24 hours as soon as I settle down a little.

@hjr265 No problem, take your time! I can work with a hardcoded fix in my node_modules in the meantime. 👍


Add draft opensearch interface

This is a first draft spec for the opensearch interface

database field has been replaced with index
read_only_endpoints aren't in this relation, since they're not really relevant to opensearch.

I'm still fixing up the tests, but you can see the interface in action here.
The spec is detailed below:

opensearch_client
Usage
This relation interface describes the expected behaviour of any charm interfacing with OpenSearch or ElasticSearch charms, such as the Charmed OpenSearch Operator using the opensearch-client relation.
In most cases, this will be accomplished using the data_interfaces library, although charm developers are free to provide alternative libraries as long as they fulfil the behavioural and schematic requirements described in this document.
Direction
flowchart TD
    Requirer -- index, \nextra-user-roles --> Provider
    Provider -- username, \npassword, \nendpoints --> Requirer

As with all Juju relations, the opensearch-client interface consists of two parties: a Provider (opensearch charm), and a Requirer (application charm). The Requirer will be expected to provide an index name, and the Provider will provide new unique credentials (along with other optional fields), which can be used to access the index itself.
Behavior
Both the Requirer and the Provider need to adhere to criteria to be considered compatible with the interface.
Provider

Is expected to create an application user inside the opensearch cluster when the requirer provides the index field.

This user is removed when the relation is removed.
Is expected to apply the permissions in the extra-user-roles provided by the Requirer to this user (e.g. extra-user-roles=admin).
If multiple relations require the same index name, they should all be able to access it.

Is not expected to create an index on relation creation.

Responsibility for managing an index rests with the requirer application, including creating and removing indices.

Is expected to provide username and password fields as Juju Secrets when Requirer provides the index field.
Is expected to provide the endpoints field containing all cluster endpoint addresses in a comma-separated list.
Is expected to provide the version field describing the installed version of opensearch.
If the charm has TLS enabled (such as using the TLS Certificates Operator), it is expected to provide the CA chain in the tls-ca field as a Juju Secret.

Requirer

Is expected to provide an index name in the index field.
Is expected to manage its own index.

Indices are not created on the provider application when the relation is created. The index field exists to grant the correct permissions for the relation user, which the requirer charm uses to control its index.
This index is NOT removed from the provider charm when the relation is removed.

Is expected to have different relations with the same interface name if Requirer needs access to multiple opensearch indices.
Is expected to provide user permissions in the extra-user-roles field. These permissions will be applied to the user created for the relation.

This can be set to two values:

default: this has read-write permissions over the index that has been generated for this relation. This permission level will be applied if no value is provided.
admin: this has control over the index, including how cluster roles are assigned to nodes in the cluster.

Relation Data
Provider
[JSON Schema]
Provider provides credentials, endpoint addresses, TLS info and database-specific fields. It should be placed in the application databag.
Example
  relation-info:
  - endpoint: opensearch-client
    related-endpoint: opensearch-app-consumer
    application-data:
      index: myindex
      endpoints: <IP_ADDRESS>:9200,<IP_ADDRESS>:9200
      password: Dy0k2UTfyNt2B13cfe412K7YGs07S4U7
      username: opensearch-client_4_user

Requirer
[JSON Schema]
Requirer provides index name. This should be placed in the unit databag in at least one unit of the Requirer.
Example
  relation-info:
  - endpoint: opensearch-app-consumer
    related-endpoint: opensearch-client
    application-data: {}
    related-units:
      worker-a/0:
        in-scope: true
        data:
          index: myindex

can you also add this to the top level README?

Thanks @WRFitch!


Add banning admins name to ban reason.

 You can't have asynchronous tasks/methods inside the ban definition file without reworking the entire ban system, and it would break previous bans. This is literally the only way to do it, and you have to just request the username from wizdens api.

this is dumb ngl
thx


Use Highcharts.Options type for ChartModel.highchartsConfig

Would want to look carefully around any complexities with extra modules or things like that, but this would be a big upgrade over our current bail-out to PlainObject when it comes to getting help writing and updating chart configs.

Ah - the curse of "quick wins" strikes again...
So we deliberately don't import HC at all as a Hoist dep, to make it clear that any project / developers wishing to use it must review and properly license the library according to their terms.

Looking at highcharts available downloads, I don't see an easy path forward here -- we have gone through much trouble to have the toolkit work without highcharts, and don't think we want to give up on that.


can not parse method correctly in enum

enum PostEnum {
    public static String getEnum(String name) {
        return name + "_get";
    }
}

class A {
    public static void main(String[] args) throws Exception {
        String e = PostEnum.getEnum("a");
        __print(e);
    }
}

That's because it doesn't have any members, which makes it invalid.


Removed UML since is too outdated (#65)

Related with #51

Codecov Report

Merging #256 (f8497b2) into main (3550b32) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main     #256   +/-   ##
=======================================
  Coverage   84.92%   84.92%
=======================================
  Files          40       40
  Lines        1327     1327
=======================================
  Hits         1127     1127
  Misses        200      200

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


fix a bug at iris-ws.js

missing object key "*" for OnAll()

Could you make the same for the client.ts file in order to keep them align ?

@kataras ok i will.


publishing test results fails

Describe the bug
Request POST /repos/Marthijs-Berfelo/buddy-harmony/check-runs failed with 403: Forbidden

wrong permission raised


username/password secrets

I can't username & password to work as secrets and it doesn't look like the chart has any reference to secrets. Does this still need to be added as a feature of the chart?

Which chart are you talking about?

Which chart are you talking about?

mosquitto. Couldn't figure out how to set it up based on this

mosquitto. Couldn't figure out how to set it up based on this

Erg, this is totally irrelevant and comes from a README template I used. Sorry for that mistake.
If you want to define users for your mosquitto instance, you can use the mosquitto.users value, eg:
mosquitto:
  users: |-
    foo:$6$b5VUNl9bhQZ2oGuA$SvNhPFXAfnbsKmCarzEwFkJLOmzKBz3CV+rMyhEyS7VQT/e7pEYfWuL/taGSTSbgJYnbPSvMs8ZkZxnRwwZGFQ==

(The content of this value can be generated using mosquitto_passwd binary)

Erg, this is totally irrelevant and comes from a README template I used. Sorry for that mistake.
If you want to define users for your mosquitto instance, you can use the mosquitto.users value, eg:
mosquitto:
  users: |-
    foo:$6$b5VUNl9bhQZ2oGuA$SvNhPFXAfnbsKmCarzEwFkJLOmzKBz3CV+rMyhEyS7VQT/e7pEYfWuL/taGSTSbgJYnbPSvMs8ZkZxnRwwZGFQ==

(The content of this value can be generated using mosquitto_passwd binary)


Torus-crossing LOS

That issue was revealed by an ITER test case implemented by Gabor.
On figure01, page 1 one notices that there is part of the VOS on the opposite side of the Tokamak.
How is this possible ?
When estimating the VOS of a detector+aperture, one of the first steps that ToFu takes is to plot many LOS from the detector through the aperture, to ‘sample’ the VOS.
Now, since we are dealing with a torus, there is always a risk that a LOS will enter the torus twice : once in the poloidal cross-section of interest, and once more in the opposite poloidal cross-section (cf. figure01 page 2 where several examples of LOS are represented)
Of course ToFu can handle this, and it ‘knows’ that each LOS should enter the Torus only once, thus, it disregards the second part in the opposite cross-section.
But detector 1805-00 is in a very special configuration : it is such that some LOS can enter the opposite cross-section without having first entered the relevant one, because the detector is so close to the edge and looking up.
Thus, for these LOS ToFu does not know that they should be disregarded because the usual criterion (i.e. disregard if it is the second time they enter the torus) is not fulfilled.
To solve this problem in the former ToFu version available at IPP, two types of routines were added to General_Geom_cy.pyx (each in 1D and 2D version).
One returns the three planes and 2 points defining the forbidden area.
One check whether a points to this area.
See modified part attached => to be implemented
Debug_1805-0_Corrected.pdf
Figure01.pdf

.... and it required to add 2 keyword arguments to all occurences of the 3 methods in tofu.geom._compute.py:
The additional kwdargs are : Forbid=True and Margin=0.1

It seems the computation of the 2 points tangential to the central cylinder (S1 and S2 in the program) were computed with a mistake.
There was a sign problem in the expression of S2.
Furthermore, the expressions for V1 and V2 (the vectors perpendicular to the asscoiated planes) were unnecessarily complicated. They have been simplified (-S1 and -S2), and the sign checks after them are now unnecessary.

Forgot to correct the previous sign mistake in both functions in General_Geom_cy.pyx:

Calc_InOut_LOS_ForbidArea_1D()
Calc_InOut_LOS_ForbidArea_2D()

Done


Fix up use of chunk_size in var_inf to avoid overwriting config param…

…eter
Problem & Solution Description (including issue #)
Code Quality

[ ] My code follows the code style of this project
[ ] I have written unit tests or justified all instances of #pragma: no cover; in the case of a bugfix, a new test that breaks as a result of the bug has been added
[ ] My code contains relevant comments and necessary documentation for future maintainers; the change is reflected in applicable demos/tutorials (with output cleared!) and added/updated docstrings use the NumPy docstring format
[ ] Any breaking changes, described above, are accompanied by backwards compatibility and deprecation warnings

#126


added definitions for request-promise

definitions for https://www.npmjs.com/package/request-promise which I used in a previous project

@cglantschnig Thanks mate!


Using NGU Explore with Stable-Baselines3's MultiInputPolicy on Gymnasium-Robotics

Hi,
I'm working on a project that involves using a Stable-Baselines3 agent with a multi-input policy in a Gymnasium-Robotics environment. I'm interested in incorporating the NGU exploration method.
To handle the environment's multi-input nature, the observation space is defined as a 'space.Dict' with multiple keys.
Question:
Is it possible to use NGU Explore with Stable-Baselines3's MultiInputPolicy when the observation space is a spaces.Dict?
If so, what steps are involved in integrating NGU Explore with this setup (normalize observation and reward)?
Error Encountered:
```python
Traceback (most recent call last):
File "/home/bmahdy/Master/M_NGU/RLeXplore-main/rlexplore_with_sb3_FetchPickAndPlace.py", line 115, in
irs = NGU(envs=envs, device=device, batch_size=batch)
File "/home/bmahdy/anaconda3/envs/NGU_9/lib/python3.9/site-packages/rllte/xplore/reward/ngu.py", line 87, in init
rnd = RND(
File "/home/bmahdy/anaconda3/envs/NGU_9/lib/python3.9/site-packages/rllte/xplore/reward/rnd.py", line 78, in init
super().init(envs, device, beta, kappa, gamma, rwd_norm_type, obs_norm_type)
File "/home/bmahdy/anaconda3/envs/NGU_9/lib/python3.9/site-packages/rllte/common/prototype/base_reward.py", line 88, in init
TorchRunningMeanStd(shape=self.obs_shape)
File "/home/bmahdy/anaconda3/envs/NGU_9/lib/python3.9/site-packages/rllte/common/utils.py", line 52, in init
self.mean = th.zeros(shape, device=device)
TypeError: zeros() received an invalid combination of arguments - got (dict, device=NoneType), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
* (tuple of ints size, *, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
'''
Any insights, guidance, or code examples would be greatly appreciated.
Thanks in advance!

This is because here, the TorchRunningMeanStd is only for simple tensors, but you are using Dict observations. You may need to rewrite the TorchRunningMeanStd class to adapt to your Dict obs.


[Question] Any clues to sum up the image along the channel dimension?

For example, I have an image of size 256*256*3. Can I sum up the RGB channel and get a 256*256 image with current symbols? Thanks.

I created an op that does this (actually more than this it does sum(input_1*input_2, axis=n)). It didn't seem to generic so I didn't commit it in the main repo. I would be greate if you can remove the prod part and make a SumWithAxis op
https://gist.github.com/piiswrong/e39f7383cae9d4134f5b

you can use convolution for that, the num_hidden=1,kernel=(1,1),fix weight=1

Thanks. @tornadomeet  Do you have any idea on how to fix weight (without back-propagation update)?

@XericZephyr Set gradient req to null, eg:
req = {"fc_weight" : "null", "conv_weight" : "write"}

This only works if you are trying to sum a 4d tensor along the second dimension though

I think using convolution to do the sum is a smart idea, also a bit of a hack.
Operator to do sum (and other operate) in axis seems should be quite usefull.


no RTL support

i think it could be added using a rtl config option or just a simple button
set (--direction-var) to toggle bettwen rtl and ltr
current workaround

its not useable for my usecase keep it left to right! lol


⚠ Bofrost.be has degraded performance

In 13f37d2, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1005 ms

Resolved: Bofrost.be performance has improved in e8481a7.


--auto-gen-config can fail with "Error: empty section found in .rubocop_todo.yml"

When running Rubocop with the --auto-gen-config option, the part-built output file .rubocop_todo.yml is loaded and checked during the processing of each cop (because the generated excludes are added to any defined manually in .rubocop.yml and when rubocop.yml is loaded it follows the inheritance down to .rubocop_todo.yml).
One validation check is for an empty section.
The output file is built up cop-by-cop during this process by writing to it after each cop using puts. Ruby's caching of these writes means that sometimes only a partial image of the offences is written to the file and this partial image is what is read during the next check.
Occasionally, by coincidence, the end of the buffer is the beginning of a cop section like this:
Style/CaseEquality:
  Exclude:

and the empty Exclude section causes the validation error
Error: empty section Style/CaseEquality found in .rubocop_todo.yml

Possible solutions

Add an output.flush after each output.puts. This works and is easy to implement but is maybe slows things down a bit.
Do not write .rubocop_todo.yml while it's being built. Instead write to a temporary file and rename it after building has finished.
Add a config load option that ignores the part-built todo file.

Expected behavior
I expect rubocop --auto-gen-config to complete successfully.
Actual behavior
Instead, rubocop --auto-gen-config exits with an error status and is truncated at the end of the output buffer last written.
Steps to reproduce the problem
It's going to be difficult to reproduce this because it depends on the flushing behaviour of Ruby's IO class. It happened to me with my repository which is large and contains many offences, but it's private unfortunately.
I'm happy to help reproduce this any way I can
RuboCop version
$ rubocop -V
0.52.0 (using Parser <IP_ADDRESS>, running on ruby 2.2.2 x86_64-darwin16)

I experienced the same issue yesterday, while running rubocop for the first time on a large project, with many offenses. And I, too, suspected a puts/flush issue, but didn’t investigate as thoroughly as @dominicsayers.
Reverting back to v0.51.0 allowed me to run rubocop --auto-gen-config, so the problem is probably somewhere between the last two releases.

See #5294 for a potential solution.

Checking now with my code

OK, here's what happened using the #5294 code with my codebase.

--auto-gen-config completed without error
The generated file .rubocop_todo.yml usually has its keys in alphabetical order. In the one that was generated by #5294, the key Metrics/LineLength appears at the end, out of order. I was not expecting to see this key at all because this cop is disabled in my .rubocop.yml.
The list of exclusions for Style/IfUnlessModifier was incomplete so when I ran rubocop after generating the to-do file there were still some offences reported for this cop.

I can't say whether these two problems were caused by your changes or other issues with Rubocop post-0.52 - my guess would be it's other people's changes not yours. I can't check because the master branch won't complete successfully against my codebase.
cc @bquorning

Phew 😄
Looking at the spec changes in 9e76387952725f1a7fa54989024114549c0b7d80, I think Metrics/LineLength is now expected to be at the end of the auto-generated file.
I’ve seen the same issue, only for Naming/MethodName. I think these are separate issues though.


getBoundingBox not in local space

According to the specification getBoundingBox always returns the bounding box of the element in local space.
However, calling getBoundingBox on a group element returns a bounding box with values that are obviously in world space.

So the old getBoundingBox function was pretty confusing, it would return boxes in world space for group nodes and local space for mesh nodes. For consistency this function has been split up into getLocalBoundingBox and getWorldBoundingBox, similar to the get*Matrix functions. Both these functions are available on xml3d, group, mesh and model nodes.
getBoundingBox is now deprecated and will be removed in a future version, until then it follows the old (confusing) behavior of returning world space boxes on group nodes and local space boxes on mesh or model nodes.


sw-appcache-behavior bad function results in fail in just Firefox Stable

Library Affected:
sw-appcache-behavior
Browser & Platform:
FF Stable
Issue or Feature Request Description:
When working on https://github.com/GoogleChrome/sw-helpers/pull/375 I had a bad function call to logHelpers.log as I renamed it to logHelpers.info.
This results in an error in the browser console - function doesn't exist.
In other browsers, the tests were passing, whereas Firefox Stable was failing to pass the tests. This may be an issue if the other browsers weren't actually passing the tests correctly, OR it could simply be that only Firefox stable has an issue with failing tests.
cc @jeffposnick

I think we mentioned this elsewhere but was there any concern with deprecating and moving this behavior out to another repo vs continuing to land fixes for it? (Jeff seems fine with updating but just wanted to sanity check)

Personally, I'd favor moving to a seperate repo. Not kill it though if it still has use.

It hasn't seen a great amount of usage out in the wild from what I can see. As @jeffposnick is the owner, I'll defer to him: my suggestion is we move to a separate repo to keep focus on the newer libraries / work that will be ported but for now not deprecate/kill until clear no one is using it. Is that fair?

No longer relevant as sw-appcache-behavior has been moved.


compat+docs

@bhollis  can you test out whether this sed format works? my bash interprets
-i '' as '' being a path. which is i think technically correct. not sure if it's coded as an exception for macs.
this syntax seems clearer about its intentions, i'm hoping mac understands it
this normalizes windows file paths
also: would it be correct to say this requires globally installed babel-cli and typedoc? it seems to

Unfortunately the mac version of sed doesn't know --in-place=''.
This doesn't require globally installed babel-cli and typedoc - they're listed as dev-dependencies so as long as you run yarn start instead of ./build.sh those are available in path.

re: babel/typedoc: :heavy_check_mark:
re: sed could you try the syntax sed -i''? maybe it works without the space

Ugh, not having the space doesn't work, it doesn't understand it. I'd also be happy with solutions that didn't involve sed.

closing these in favor of just the stuff in https://github.com/DestinyItemManager/bungie-api-ts/pull/10
sed isnt really a huge priorirty

https://www.npmjs.com/package/nip


Getting error while installing mininet

Hi Friends,
I am getting following error while installing mininet.please help how to fix the below issue.My VM running on virtualbox Ubuntu 14.04 lts.
W:failed to fetch trusty release htt://in.archive.ubuntu.com  and unable to find expected entry
E:some index files failed to download.
Thank you

Linux distro is ubuntu 14.04 trusty amd64

This distro is unfortunately not supported anymore, please try using a newer version of ubuntu.
Apart from that, maybe check your /etc/apt/sources.list and make sure the urls are correct as the log says  W:failed to fetch trusty release htt://in.archive.ubuntu.com and unable to find expected entry which indicates a missing ps in https://


sokol_defines.h: fix for Android

Did manage to get something working on Android?

Yes. It worked just by modifying this part.


Change Module 7 to use SSMS instead of ADS

SSMS 20 no longer installs Azure Data Studio so change the labs in Module 7 to use SSMS

fixed


appDiagnostics Capability not required

I don't think appDiagnostics Capability is required to access AppInfo class. That is probably only required to access the AppInfo property for AppDiagnosticInfo class.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7e617ca3-d599-55db-4017-b9554bcd5c2f
Version Independent ID: e740d334-bc23-a021-e9db-7cae750a91da
Content: AppInfo Class (Windows.ApplicationModel) - Windows UWP applications
Content Source: https://cpubwin.visualstudio.com/DefaultCollection/windows-uwp/_git/winrt-api-build?path=%2fwinrt-api-build%2fwindows.applicationmodel.appinfo.yml&version=GBlive&_a=contents
Product: uwp
Technology: winrt-reference
Git Login: knicholasa
Microsoft Alias: nichola

Thanks for taking the time to let us know about this. Due to the sheer volume of feedback in our repo, we're currently prioritizing new Issues. If you believe your feedback is still actionable, let us know either directly in this thread or by filing a new Issue and we’ll work to incorporate your suggestions. Though this issue will be marked as Closed, this issue is still being tracked internally and you're welcome to leave a comment with more details. Thank you again!


Info config message showing when starting the application in another directory.

Info Message
When I'm starting my application from another directory (node src/server.js), it's showing the following info message:
info config no opencv4nodejs section found in C:\Users\User\Desktop\spot-sdk\spot-server-js\package.json

But if I'm starting it in the src directory (node server.js) it's doesn't show any info message in the console.
OpenCV version : 4.5.5
With OpenCV-contrib? (extra modules): no
OS: Windows 10

yep if no package.json is found, no message is displayed.
Is it better to display one?
maybe.

Thanks for your reply !
As I can read in the readme file, the opencv4nodejs section in the package.json's file is deprecated, so I think it doesn't make sense to have an info message to notify when there is no opencv4nodejs's section ?
I'm maybe wrong ?

I flag it as deprecated, but since this was mostly used, I will keep it functional.
My approach is building OpenCV using build-opencv command line.

Oh, I see sorry for the inconvenience I am closing this issue.
Have a good day !


.NET Analyzer fails with IDE1003 due to missing netstandard 2.0

I created a Roslyn analyzer which targeted the .NET Standard 2.0 framework. When I reference it from one of my .NET Core assemblies as an analyzer, it fails with an Analyzer assembly warning IDE1003. The message is '{DLL Path}.dll' depends on 'netstandard, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' but it was not found. Analyzers may not run correctly unless the missing assembly is added as an analyzer reference as well.
When I downgraded to a lower version of .NET Standard for the same analyzer without any other configuration changes, this error goes away. Based on this StackOverflow post, it seems like there is a whitelisted assembly missing somewhere.
Since we run with warnings as error, this is causing multiple false positives.

@michsco Can you file this bug on dotnet/roslyn instead?


No treeshakkng?

It looks like vue-color does not support tree shaking. Is this something that is inherent to the architecture of vue-color?

Sorry, got confused with another package.


NullPointerException in version 1.2.0 (com.authy.api.Resource.java)

`
StringBuilder e = new StringBuilder();
if(method.equals("GET")) {
e.append(this.prepareGet(data));
}
        URL url = new URL(this.apiUri + path + e.toString());
        HttpURLConnection connection = this.createConnection(url, method, options);
        connection.setRequestProperty("X-Authy-API-Key", this.apiKey);
        if(data.toMap().containsKey("api_key")) {//THROWS A NULL POINTER EXCEPTION
            LOGGER.log(Level.WARNING, "Found \'api_key\' as a parameter, please remove it, Authy-Java already handles the\'api_key\' for you.");
        }

        if(method.equals("POST") || method.equals("PUT")) {
            if(this.isJSON) {
                this.writeJson(connection, data);
            } else {
                this.writeXml(connection, data);
            }
        }

        this.status = connection.getResponseCode();
        answer = this.getResponse(connection);
    } catch (SSLHandshakeException var9) {
        System.err.println("SSL verification is failing. This might be because of an attack. Contact support@authy.com");
    } catch (Exception var10) {
        var10.printStackTrace();
    }

    return answer;
}

`
This line of code is throwing a NPE when trying to create a user, using your documented process (the createUser line below works fine in version 1.1.0):
`
User user =<EMAIL_ADDRESS>"405-342-5699", "57");
//eventually calls this which is what blows up as data = null
if(data.toMap().containsKey("api_key")) {
`

@tpthaler looks like it is working now


Sanitise personal and private data for dev environments

Would be nice on the database syncing from LIVE to DEV to be able to sanitize the data to replace user data and sensitive information with dummy data.
If I have any luck getting it to happen, will be in touch. Unless someone else can do it faster.

This is probably not something I'd do in Craft-Scripts, but you're definitely welcome to fork it and make the changes needed.
The reason is it'd be very hard to know in a general way what is sensitive, and what isn't... and then also I think it's important that the local dev database is an exact copy (cache tables aside) from the production db. But you may disagree!


Why do I have to initialize the deformation gradient with zeros?

Why do I have to initialize the deformation gradient with zeros, add the ones to the diagonal terms and finally add the displacement gradient? I mean I tried to write down
F = grad(u)
for i in range(3):
    F[i, i] += 1.

instead of
F = zeros_like(grad(u))
for i in range(3):
    F[i, i] += 1.
F += grad(u)

but this seems to give a different result. Honestly, I do not understand why. Do you have any explanation for that? (I took the code snippet from example 36)
In addition to that,I find the different import statements for math helpers a bit confusing - although I understand why they are necessary. I mean these imports:
from numpy import (einsum, linalg as nla, zeros,
                   zeros_like, concatenate, split as npsplit,
                   hstack, abs as npabs, arange, sqrt)
from skfem.helpers import grad, transpose, det, inv

In the beginning I did not understand why I can't use det and inv from numpy.linalg but then I realized the special structure of the arrays (3,3,n,m). I think this shape is used in order to obtain a reasonable performance with Python/NumPy 😄
Anyway, I'll experiment with the examples and I hope Github Issues are the right place to ask.
Thanks and with best regards,
Andreas

I tried it out with a debugger by adding
import pdb; pdb.set_trace()

after the lines you mention.
(Pdb) FF = grad(u)
(Pdb) FF[0, 0] += 1.
(Pdb) FF[1, 1] += 1.
(Pdb) FF[2, 2] += 1.
(Pdb) (FF-F).sum()
0.0

I don't see any difference here.

In the beginning I did not understand why I can't use det and inv from numpy.linalg but then I realized the special structure of the arrays (3,3,n,m). I think this shape is used in order to obtain a reasonable performance with Python/NumPy

This is correct. Here is some information related to the forms: https://scikit-fem.readthedocs.io/en/latest/forms.html#helpers-are-useful-but-not-necessary

In addition to that,I find the different import statements for math helpers a bit confusing - although I understand why they are necessary. I mean these imports:
from numpy import (einsum, linalg as nla, zeros,
                   zeros_like, concatenate, split as npsplit,
                   hstack, abs as npabs, arange, sqrt)
from skfem.helpers import grad, transpose, det, inv

Yeah, I think more typical would be to have import numpy as np and then use np.einsum and so on. This example was not written by me but maybe I will update it at some point.

Thanks for your investigations. I modified the mentioned lines in function F1 of example 36 but I did not change the calculation of the deformation gradient in the other functions. Then the solution did not converge anymore. I will have a look at it again tomorrow and hopefully provide some code for reproduction.

Here is the output I get with the unmodified Example 36 taken from the docs folder:
1, norm_du: 15.945909822097681, norm_dp: 5.921076520330817
2, norm_du: 5.598466873920148, norm_dp: 7.5338746879470735
3, norm_du: 1.0735307535449552, norm_dp: 3.52864022713699
4, norm_du: 0.08852224392032632, norm_dp: 0.4692554732052099
5, norm_du: 0.005475563449509868, norm_dp: 0.013662254425980746
6, norm_du: 7.064329920208044e-05, norm_dp: 0.0002627006123853513
7, norm_du: 1.5250880685127334e-08, norm_dp: 9.71235752142727e-08
8, norm_du: 2.0704928782384954e-15, norm_dp: 7.487594363769038e-15

If I modify Function F1 in it to
def F1(w):
    u = w["disp"]
    p = w["press"]
    #F = zeros_like(grad(u))
    F = grad(u)
    for i in range(3):
        F[i, i] += 1.
    #F += grad(u)
    J = det(F)
    Finv = inv(F)
    return p * J * transpose(Finv) + mu * F

the solution does not converge anymore. I get this output instead:
1, norm_du:<PHONE_NUMBER>0451476, norm_dp:<PHONE_NUMBER>4476666
2, norm_du: 90483.89305264289, norm_dp:<PHONE_NUMBER>3656483
3, norm_du:<PHONE_NUMBER>8693928, norm_dp: 76422.94872722038
4, norm_du: 2184968.3522374863, norm_dp:<PHONE_NUMBER>081902
5, norm_du:<PHONE_NUMBER>066894, norm_dp:<PHONE_NUMBER>609701
6, norm_du:<PHONE_NUMBER>993895, norm_dp:<PHONE_NUMBER>1103134
7, norm_du:<PHONE_NUMBER>372445, norm_dp:<PHONE_NUMBER>1935522
8, norm_du:<PHONE_NUMBER>2298647, norm_dp: 78772.38589782098
9, norm_du:<PHONE_NUMBER>3635778, norm_dp: 51718.8543437011
10, norm_du:<PHONE_NUMBER>12649, norm_dp: 39334.79494817416
11, norm_du:<PHONE_NUMBER>640927, norm_dp: 83827.73033857255
12, norm_du:<PHONE_NUMBER>838606, norm_dp: 54564.865595477415

So I assumed I can't "initialize" F with grad(u)...
My sckit-fem version (from pip) is
print(pkg_resources.get_distribution('scikit-fem').version)
>>> 2.5.0

and I use Python 3.8 with Anaconda on Win10.

This is a classical mistake when using numpy. I did not notice it first but now it's obvious.
One should write
F = grad(u).copy()

instead.

Oh wow, thanks @kinnala !

I made some improvements to this example here: https://github.com/kinnala/scikit-fem/pull/589/commits/f224989ee5027553281f26fa427b6cd4e38e3ec9
Sorry for doing it in a relatively unrelated branch. However, it should address everything pointed out here.


update 5.0

I modified three places

update calculateBaseProgressIncrease & calculateBaseQualityIncrease use new formula
add control limit；maxControl = originControl + 3000
update ingenuity reference teamCrafter

Best Wish

Thank you so much!

@SaltFish-X: I've made a few minor changes to your updates here: 14b793ea071826c73b4de71d5ef59ff7330a0514
It's mostly formatting and naming changes to suit my style preferences. I also rewrote the level difference factor lookup function (getLevelDif) to simplify it and make it harder for bugs to creep in later.
Please let me know if you disagree with any of these changes by commenting on that commit.

Your code is better！
it inspired me a lot。

    // Since game version 5.0, effects increasing control are capped at crafter's original control + 3000
    control = Math.max(control, s.synth.crafter.control + 3000);

Here is wrong
But You have changed in 6d797d


Importing uuid causes a Visual C++ runtime library error in EXE on Windows 7 (Trac #66)

See attached error screen grab.
This is likely related to the use of ctypes in uuid.
See http://comments.gmane.org/gmane.comp.python.pyinstaller/2639
Migrated from http://microfluidics.utoronto.ca/ticket/66
{
    "status": "closed",
    "changetime": "2014-04-17T19:39:01",
    "description": "See attached error screen grab.\n\nThis is likely related to the use of ctypes in uuid.\nSee http://comments.gmane.org/gmane.comp.python.pyinstaller/2639",
    "reporter": "cfobel",
    "cc": "",
    "resolution": "fixed",
    "_ts": "1397763541728826",
    "component": "microdrop",
    "summary": "Importing uuid causes a Visual C++ runtime library error in EXE on Windows 7",
    "priority": "major",
    "keywords": "",
    "version": "0.1",
    "time": "2012-03-12T17:41:17",
    "milestone": "Microdrop 1.0",
    "owner": "ryan",
    "type": "defect"
}

Trac update at 20120312T17:42:23: cfobel changed attachment from "" to "uuid_error.jpg"

Trac update at 20120312T18:10:48:

<EMAIL_ADDRESS>commented:

commit 2c9b39903e5dbeb2b0ff1a82652979f470a248d4
Author: Christian Fobel<EMAIL_ADDRESS>Date:   Mon Mar 12 13:56:39 2012 -0400
Add customized uuid to utility module and remov...

Add customized uuid to utility module and replace all imports of uuid with
an alias to utility.uuid_minimal.
The modified uuid does not use the ctypes module.
This change fixes #70.

<EMAIL_ADDRESS>changed resolution from "" to "fixed"
<EMAIL_ADDRESS>changed status from "new" to "closed"

Trac update at 20140417T19:39:01: ryan changed component from "core" to "microdrop"

Trac update at 20150607T08:46:11:

Slavon changed attachment from "" to "d40bd6a41276ebb3795cb50422975481.jpg"
Slavon commented:

[http://kdkraftupwnkitchen.tumblr.com/ http://kdkraftupwnkitchen.tumblr.com/]


!!! NOTE I FOUND THE ANSWER CHECK THE ANSWER: TypeError: argument of type 'bool' is not iterable on brownie run scripts/deploy.py command

Hey i am having difficulties running deploy, in the first times it was working fine and afterwards even the code that was running stopped running, what could be the error.
Please help me. Note that the same code had run before it simply stoped on my revision or maybe I removed something accidentally
here is the whole error
`INFO: Could not find files for the given pattern(s).
Brownie v1.19.2 - Python development framework for Ethereum
File "C:\Users\JC\Desktop\deveth\pythondemos\lib\site-packages\brownie_cli_main_.py", line 64, in main
importlib.import_module(f"brownie._cli.{cmd}").main()
File "C:\Users\JC\Desktop\deveth\pythondemos\lib\site-packages\brownie_cli\run.py", line 42, in main
active_project.load_config()
File "C:\Users\JC\Desktop\deveth\pythondemos\lib\site-packages\brownie\project\main.py", line 474, in load_config
_load_project_config(self._path)
File "C:\Users\JC\Desktop\deveth\pythondemos\lib\site-packages\brownie_config.py", line 222, in _load_project_config
and "cmd_settings" in values
TypeError: argument of type 'bool' is not iterable`
here is my deploy.py file
`from brownie import FundMe, MockV3Aggregator, network, config
from scripts.helpful_scripts import (
get_account,
deploy_mocks,
LOCAL_BLOCKCHAIN_ENVIRONMENTS,
)
def deploy_fund_me():
account = get_account()
if network.show_active() not in LOCAL_BLOCKCHAIN_ENVIRONMENTS:
price_feed_address = config["networks"][network.show_active()][
"eth_usd_price_feed" #fetching addresses from brownie-config file
]
else:
deploy_mocks() # imported function from helpful scripts
price_feed_address = MockV3Aggregator[-1].address # just in case we have many addresses we take the most recent/last
`fund_me = FundMe.deploy(
    price_feed_address,
    {"from": account},
    publish_source=config["networks"][network.show_active()].get("verify"),`
)
print(f"Contract deployed to {fund_me.address}")
return fund_me

def main():
deploy_fund_me()
`
and here is my helpful_scripts.py
`
from brownie import network, config, accounts, MockV3Aggregator
from web3 import Web3
FORKED_LOCAL_ENVIRONMENTS = ["mainnet-fork", "mainnet-fork-dev"]
LOCAL_BLOCKCHAIN_ENVIRONMENTS = ["development", "ganache-local"]
DECIMALS = 8
STARTING_PRICE =<PHONE_NUMBER>00
def get_account():
if (
network.show_active() in LOCAL_BLOCKCHAIN_ENVIRONMENTS
or network.show_active() in FORKED_LOCAL_ENVIRONMENTS
):
return accounts[0]
else:
return accounts.add(config["wallets"]["from_key"]) #fetching the key from brownie-config
def deploy_mocks():
print(f"The active network is {network.show_active()}")
print("Deploying Mocks...")
if len(MockV3Aggregator) <= 0: # Mocks are similated interfaces for local networks, and here we check if there is no contract that is already deployed
    MockV3Aggregator.deploy(DECIMALS, STARTING_PRICE, {"from": get_account()})
   print("Mocks Deployed!")

`
and here is my  config file
` dependencies:
- <organisation/repo>@ this is a schema to tell brownie where to download the interfaces on github

<EMAIL_ADDRESS>
compiler:
solc:
remappings:
- '<EMAIL_ADDRESS># this is to tell brownie that whenever we use @chainlink we are referring to<EMAIL_ADDRESS>dotenv: .env
networks:
goerli:
eth_usd_price_feed: "0xD4a33860578De61DBAbDc8BFdb98FD742fA7028e"
verify: True
development:
verify: False
# we can add any netwok we want or we may want to work with here
ganache-local: False
wallets:
from_key: ${PRIVATE_KEY} # from .env file
`
.env file
`export PRIVATE_KEY ='0x98a1f8269ebb99a59948b7c09079cf0a75c04bcbea3b6881e3c5d7658261ce35'
export WEB3_INFURA_PROJECT_ID = a6bd5268a2b0432cbbefe4f634f9e789
export ETHERSCAN_TOKEN = TAD6VC3ERIX1RMPK4URRJ3U3IPWQSNAAZK
this is a api token to verify a contract on etherscan otherwise you would need to manually do it`
FundMe and Mock3aggregator I used the ones Patrick used

Hey had forgotten an indentation in the config file on ganache-local
ganache-local:
    verify:    False


libSBML install fails (on Mac with M2)

Morpheus installation fails, apparently due to libSBML.
This is on Mac with Silicon M2.
==> Installing<EMAIL_ADDRESS>==> cmake . -DMORPHEUS_RELEASE_BUNDLE=ON
==> make install
Last 15 lines from<EMAIL_ADDRESS>-- Installing: /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/include/sbml/math/MathML.h
-- Installing: /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/lib/libsbml-static.a
-- Installing: /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/lib/cmake/sbml-static-config.cmake
-- Installing: /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/lib/cmake/sbml-static-config-release.cmake
-- Installing: /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/lib/cmake/sbml-static-config-version.cmake
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/build && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E touch /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/libSBML-prefix/src/libSBML-stamp/libSBML-install
[ 44%] No test step for 'libSBML'
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/build && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E echo_append
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/build && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E touch /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/libSBML-prefix/src/libSBML-stamp/libSBML-test
[ 44%] Completed 'libSBML'
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E make_directory /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/CMakeFiles
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E touch /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/CMakeFiles/libSBML-complete
cd /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML && /opt/homebrew/Cellar/cmake/3.27.7/bin/cmake -E touch /tmp/morpheusA2.3.0-20231107-77867-ebqk57/morpheus-v2.3.0_1/3rdparty/libSBML/libSBML-prefix/src/libSBML-stamp/libSBML-done
[ 44%] Built target libSBML
make: *** [all] Error 2

If reporting this issue please do so at (not Homebrew/brew or Homebrew/homebrew-core):
  https://github.com/morpheus-lab/homebrew-morpheus-versions/issues

Hi @wdeback,
could you please try the latest bugfix release Morpheus 2.3.5 of the 2.3 series? For this, you will need to have the Morpheus standard tap morpheus-lab/morpheus enabled.
brew tap morpheus-lab/morpheus
brew install morpheus

Best regards,
Diego

Bingo! That worked!
$ brew tap morpheus-lab/morpheus
==> Tapping morpheus-lab/morpheus
Cloning into '/opt/homebrew/Library/Taps/morpheus-lab/homebrew-morpheus'...
remote: Enumerating objects: 323, done.
remote: Counting objects: 100% (323/323), done.
remote: Compressing objects: 100% (151/151), done.
remote: Total 323 (delta 154), reused 296 (delta 137), pack-reused 0
Receiving objects: 100% (323/323), 42.42 KiB | 241.00 KiB/s, done.
Resolving deltas: 100% (154/154), done.
Tapped 1 formula (18 files, 71.1KB).

$ brew install morpheus
==> Downloading https://formulae.brew.sh/api/formula.jws.json
################################################################################################################################################################## 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
################################################################################################################################################################## 100.0%
Warning: Treating morpheus as a formula. For the cask, use homebrew/cask/morpheus
==> Fetching morpheus-lab/morpheus/morpheus
==> Downloading https://gitlab.com/morpheus.lab/morpheus/-/archive/v2.3.5/morpheus-v2.3.5.tar.gz
                                                    -=O=-                                                                                #       #      #   #
==> Installing morpheus from morpheus-lab/morpheus
==> cmake . -DMORPHEUS_RELEASE_BUNDLE=ON
==> make install
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /opt/homebrew
Could not symlink bin/morpheus
Target /opt/homebrew/bin/morpheus
is a symlink belonging to<EMAIL_ADDRESS>You can unlink it:
  brew unlink<EMAIL_ADDRESS>
To force the link and overwrite all conflicting files:
  brew link --overwrite morpheus

To list all files that would be deleted:
  brew link --overwrite --dry-run morpheus

Possible conflicting files are:
/opt/homebrew/bin/morpheus -><EMAIL_ADDRESS>/opt/homebrew/bin/morpheus-gui -><EMAIL_ADDRESS>==> Caveats
To start the Morpheus GUI, type the following command:

  morpheus-gui

Or add Morpheus to your Applications folder with:

  ln -sf /opt/homebrew/opt/morpheus/Morpheus.app /Applications

For more information about this release, visit: https://morpheus.gitlab.io/download/latest/
==> Summary
🍺  /opt/homebrew/Cellar/morpheus/2.3.5: 14 files, 15.9MB, built in 3 minutes
==> Running `brew cleanup morpheus`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Not sure whether it is related, but now it complains about libtiff. Morpheus depends on libtiff.5.dylib, while brew currently is on version 4.6.0: https://formulae.brew.sh/formula/libtiff
$ morpheus
dyld[95032]: Library not loaded: /opt/homebrew/opt/libtiff/lib/libtiff.5.dylib
  Referenced from: <2FD7DB12-EEBB-3FDE-9E81-6147B6D80B43><EMAIL_ADDRESS>  Reason: tried: '/opt/homebrew/opt/libtiff/lib/libtiff.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libtiff/lib/libtiff.5.dylib' (no such file), '/opt/homebrew/opt/libtiff/lib/libtiff.5.dylib' (no such file), '/usr/local/lib/libtiff.5.dylib' (no such file), '/usr/lib/libtiff.5.dylib' (no such file, not in dyld cache), '/opt/homebrew/Cellar/libtiff/4.6.0/lib/libtiff.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/libtiff/4.6.0/lib/libtiff.5.dylib' (no such file), '/opt/homebrew/Cellar/libtiff/4.6.0/lib/libtiff.5.dylib' (no such file), '/usr/local/lib/libtiff.5.dylib' (no such file), '/usr/lib/libtiff.5.dylib' (no such file, not in dyld cache)
Abort trap: 6

You now seem to have several Morpheus versions installed in parallel and there are symlinks on your system that point to Morpheus 2.3.0. This is why entering morpheus calls the binary of the<EMAIL_ADDRESS>formula instead the binary of the morpheus formula (the morpheus Homebrew formula always refers to the latest version, currently release 2.3.5). You should overwrite these symlinks as suggested by Homebrew during installation (see also our FAQ):
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /opt/homebrew
Could not symlink bin/morpheus
Target /opt/homebrew/bin/morpheus
is a symlink belonging to<EMAIL_ADDRESS>You can unlink it:
 brew unlink<EMAIL_ADDRESS>
To force the link and overwrite all conflicting files:
 brew link --overwrite morpheus

To list all files that would be deleted:
 brew link --overwrite --dry-run morpheus

Possible conflicting files are:
/opt/homebrew/bin/morpheus -><EMAIL_ADDRESS>/opt/homebrew/bin/morpheus-gui -><EMAIL_ADDRESS>
So simply entering brew link --overwrite morpheus should solve the problem.
If you like, you can also get rid of the conflicting<EMAIL_ADDRESS>with brew uninstall<EMAIL_ADDRESS>

Thanks @jdieg0! That solved it.

@wdeback Glad everything worked out. On which macOS version, by the way?
Feel free to open an issue if you have any further questions!


add context to default incremental behavior

Contributions

[X] I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates
https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models#default-behavior
What part(s) of the page would you like to see updated?
In working with a client, I discovered that the default setting of on_schema_change: ignore does not truly ignore all changes -- there is still a step of DDL that dbt executes specifically on data types to ensure that the data in the temporary relation created during an incremental run can still be merged to the target table. I think it's key to call out the what dbt is ignoring is not every change to an incremental model, it's specifically in regards to column additions and removals.
I think this is a nuance to the default behavior of an incremental model that is worth calling out in the docs!
Additional information
No response

Looks like this was addressed in the docs, so I will close this issue! Feel free to reopen, @dave-connors-3 if I'm missing something or wrong.


Add in Question Mark (?) for showing the hotkeys.

Moved to https://github.com/fastchat/web/issues/26