Unload content of external tilesets to save memory usage

As described in #739, external tilesets are currently never unloaded by cesium-native. This results in memory accumulating as tiles are loaded, eventually causing the device to run out of memory. This change allows tiles loaded from external sources to be unloaded when they're no longer used, as long as all of their child tiles are also ready to unload.

After consulting with the rest of the team, we're going to put this PR on hold for now as it's taking an unexpectedly large amount of effort to get working. In its current state the code works fine with @javagl's example above, and does load and unload Google P3DT when tested with CesiumGS/cesium-unreal#1415, but currently only loads a limited amount of tiles closest to the camera. In theory, any tile that's being visited as part of building the list of tiles to render should never be unloaded, but for some reason this only applies to the closest tiles and not to ones farther away. This likely has something to do with unloadTileContent and loadTileContent - for whatever reason, tiles outside of the immediate circle around the camera remain "Unloaded". Further investigation is needed, if anyone reading this wants to take a crack at it!

Further investigation is needed, if anyone reading this wants to take a crack at it!

I haven't beem following the details and evolution of  cesium-native, and wouldn't claim to be able to even guess the reason for certain issues. But if it is possible to describe "a minimal tileset that is sufficient for testing (and debugging) this", I could try to create such a test tileset.

What I've called doNotUnloadCount above is really more like doNotUnloadSubtreeCount. It may be safe to unload renderable content when this count is non-zero (subject to the rules around this that we already have), but it's not safe to unload an external tileset when this count is greater than 0.

I'm watching this, from a huge distance (from on an island in Indonesia, so to speak), and see quite some "reference counting" going on in various forms. And I'm wondering how much code (and how many bugs) could be avoided by just saying: Let's handle each Tile as a std::shared_ptr. Yes, this has to do this ~"expensive reference counting", and "... performance...", but ... apparently, reference counting is necessary, and doing it manually (with different places that increase/decrease the 'reference count', and different threads being involved) is error prone.
(I'm just wondering. I know, it won't happen. So 🤞 that you can sort this out in another way. But don't forget to add comments accordingly, e.g. in ViewUpdateResult, saying things like: "Thou shalt not store these Tile* pointers, because the ownership and lifetime of these is not specified")

Even putting aside costs, using a std::shared_ptr wouldn't work here. We can't destroy individual tiles, we only want to destroy entire subtrees loaded from external tilesets. So it is reference counting, but even though Tile stores the count, it's not really the individual Tile instances we're controlling the lifetime of. With std::shared_ptr, the thing counted and the thing whose lifetime is controlled are always one and the same.

Even putting aside costs, using a std::shared_ptr wouldn't work here. We can't destroy individual tiles, we only want to destroy entire subtrees loaded from external tilesets. So it is reference counting, but even though Tile stores the count, it's not really the individual Tile instances we're controlling the lifetime of. With std::shared_ptr, the thing counted and the thing whose lifetime is controlled are always one and the same.

I don't understand external tilesets well, but if the problem is that the lifetime of the external tileset structure is different than the individual tiles in it, then it seems that a shared_ptr would work. You don't really care about the precise reference count, just whether or not any references to a tile are held.
As an old Lisp guy, I get a bit jaded about performance and shared_ptr, putting complaints in the same category as complaints about garbage collection performance. If you need to track object lifetimes, then you need to pay for it somewhere. Was shared_ptr deemed to be too expensive at some point in Cesium Native's history? If so, I'd look at nearby solutions like std::make_shared and intrusive pointers before tossing the whole idea.

Some of this may be too unrelated to this PR (and a too large topic in general) to be discussed in this PR. I just threw it in here, because of the difficulties of the whole "lifetime and ownership" question that seems to be at the core here, and the attempt to solve it with manual reference counting is bound to involve intricate handling of the reference count, and will require deeeep knowledge about what is happening where on which thread.
(An aside: I also doubt that replacing manual reference counting with a shared_ptr would have a prohibitively large performance impact. In fact, I'm pretty sure that it would not have any measurable impact at all. But there's no point in arguing about this without benchmarking. So right now, I cannot contribute to a profound discussion. For that, I'd have to start with basic things, like ... figuring out what the difference between a ReferenceCounted and a shared pointer is to begin with...)
A few abstraction levels higher: I think that the "lifetime and ownership" question that is hidden in the point about 'individual tiles vs. subtrees' is crucial. But I assume that the state space, state transitions, lifetimes, and ownerships will raise enough questions to be documented incrementally. One overly specific example is that I assume that ViewUpdateResult will eventually contain a hint saying something like "The Tile* pointers in the tilesToRenderThisFrame may become invalid at time X under condition Y. Clients may not store these pointers"...
(The question about "tile-vs-subtree" may, at the bottom, involve seemingly trivial aspects like the std::vector<Tile> _children; not being a std::vector<std::shared_ptr<Tile>> _children;, but if the question of "Where to use which kind of pointer" is supposed to be sorted out more holistically, it should be done in a dedicated issue...)

I don't understand external tilesets well, but if the problem is that the lifetime of the external tileset structure is different than the individual tiles in it, then it seems that a shared_ptr would work.

There is no external tileset structure. Well, there's TileExternalContent, but there are no Tiles in it (just metadata).
It would be interesting to consider moving to a model where there's an explicit representation of an external tileset, and it owns all of the Tile instances within it. Perhaps TilesetJsonLoader is a start toward that.
But tackling that as part of this PR definitely risks turning a relatively focused (if slightly tricky and possibly error-prone) task into a major refactoring effort.

Was shared_ptr deemed to be too expensive at some point in Cesium Native's history?

No, we use shared_ptr. But not for tiles. Not so much because of the cost of shared_ptr, but because of the cost of heap allocating every Tile individually.
I'm aware of std::make_shared and we always use it. It cuts the heap allocations in half, but that's still a huge number of heap allocations in the case of Tiles.
We also have CesiumUtility::IntrusivePointer for things that are always reference counted, don't need weak_ptr, and benefit from a little bit cheaper reference counting.

the attempt to solve it with manual reference counting is bound to involve intricate handling of the reference count, and will require deeeep knowledge about what is happening where on which thread.

cesium-native's rule is that Tile instances can only ever be accessed from the main thread (or the thread that created them if they haven't been passed to the main thread yet, i.e., during loading). So there's no tricky threading issue here, at least.
A non-owning smart pointer class might be a nice elegant way to solve the "manual" part of the reference counting, though. On construction with a Tile*, it increments doNotUnloadSubtreeCount for the tile and all its ancestors. On destruction, it decrements the same. Then anything that holds a Tile pointer outside the normal flow and needs to make sure the TIle doesn't disappear (such as TileLoadInput) just has to use this smart pointer class.

I also doubt that replacing manual reference counting with a shared_ptr would have a prohibitively large performance impact. In fact, I'm pretty sure that it would not have any measurable impact at all. But there's no point in arguing about this without benchmarking.

Yeah and I agree with you on this point anyway. As I mentioned above, it's not the reference counting part that's costly, it's the memory allocation. I don't think I need to do a benchmark to be convinced that heap-allocating every Tile would be a mistake, though. In fact, I bet we'd see a small but measurable performance improvement if we could avoid heap-allocating (via a std::vector) each block of child Tiles, as we're currently doing. Avoiding unnecessary heap allocations is just good practice, particularly in code where performance matters. It's even in the C++ Core Guidelines. I'm not saying we have to obsess over it of course, sometimes dynamic allocation is the best solution.

I'd have to start with basic things, like ... figuring out what the difference between a ReferenceCounted and a shared pointer is to begin with...)

ReferenceCounted is an optional, convenience base class that can be inherited from in order to easily enable a particular type for use with CesiumUtility::IntrusivePointer.
I could try to explain the tradeoffs of IntrusivePointer versus shared_ptr, but I'll just refer you to Boost's documentation on their intrusive pointer to start (not exactly the same, but same idea and same tradeoffs):
https://www.boost.org/doc/libs/1_85_0/libs/smart_ptr/doc/html/smart_ptr.html#intrusive_ptr

But I assume that the state space, state transitions, lifetimes, and ownerships will raise enough questions to be documented incrementally.

Yes, no question, we need to be clear about the lifetime rules. We've gotten away with this with Tiles until now because Tiles, once created, were never destroyed, other than when the entire Tileset was destroyed.

cesium-native's rule is that Tile instances can only ever be accessed from the main thread (or the thread that created them if they haven't been passed to the main thread yet, i.e., during loading). So there's no tricky threading issue here, at least.

That seems to be focussed on the usage side (i.e. what is supposed to be visible for clients). One of my concerns about manual reference counting referred to the guts of the implementation. I think that it could easily happen that someone is doing something on some thread and this affects the reference counter. This does not refer to something explicit and obvious like
something.thenInWorkertThread([...] {
    someTileThatSomeoneGotFromSomewhere.decreaseReferenceCount();
}

that could easily spotted during a review. It rather referred to the "implicit" changes, in scenarios like

an issue is caused: Some resource or memory is not properly cleaned up
it has to be fixed: When a certain function is called, then something else has to happen, to "clean up/release" something

(This will affect some reference counter, down the call chain, which may not be obvious)

it is not clear or obvious which thread is calling the function, and/or which thread is allowed to touch the reference counter

With very hazy memories about some details, but roughly looking at your list above, there may be something like
Tile::upsample() {
    parent.increaseReferenceCount();
    ....
}

Which thread is calling 'upsample'? You probably know that. You could claim that "this sort of operation is always done by thread X". But it's probably not documented (and even less enforced) explicitly.
Now, there has to be a matching parent.DEcreaseReferenceCount(); call somewhere. This could be part of some finishedUpsampling() method or so. But where it's not clear or obvous which thread is calling this method either.
(All this could probably be summarized with: "See ReferenceCountedThreadSafe vs ReferenceCountedNonThreadSafe" 😉 )

The Boost documentation does not go into details about reasons for using intrusive_ptr (but mentions that one should start with shared_ptr by default). And the reasons that I found by 5 minutes of websearching always seem to refer to "performance". I know that there are general rules and best practices, and some design decisions on a high level (~"preferring stack over heap") that have to be carefully considered in view of their implications on performance and (IMHO more importantly:) lifetime of objects.
But as a baseline: I doubt any performance claim that is made without a benchmark. And when there is a benchmark, then I doubt that benchmark 😈 . Quickly looking at the first, naive, shallow search results like https://stackoverflow.com/questions/14837829/intrusive-ptr-shared-ptr-performance-tests or https://groups.google.com/g/benchmark-discuss/c/uENrnU4J3AM :

they are sometimes pretty old (comparing Boost intrusive_ptr to Boost shared_ptr might have made sense 10 years ago, but now there is std::shared_ptr, which may have wildly different performance characteristics...)
they are varying wildly in terms of the performance/timing claims that they make
important: they seem to be run single-threaded, meaning that they may be "distorted" when it comes to the impact of thread safety on overall performance of a multi-threaded application

(Whether or not intrusive_ptr is thread safe to begin with? I don't know. It can probably be derived from https://www.boost.org/doc/libs/1_85_0/libs/smart_ptr/doc/html/smart_ptr.html#intrusive_ref_counter , but I didn't look into the details....)
One reason why I didn't look into all the details: Imagine there is a Tile that has to be created, and this can be a shared_ptr and be allocated on the heap and require atomic access for reference counting. All this is expensive. But then this Tile is filled with life, by doing a network request and donwloading 5 MB of data that has to be processed and uploaded to the GPU. I'm convinced that even the most naive approach for managing tiles will (in a real application, i.e. outside the scope of a microbenchmark) change the "average FPS over 100 frames" from 42.123 to something like 41.987 or so.
Proving or disproving this with a benchmark could be difficult: One can not just introduce some #define USE_APPROACH_X true somewhere and run both versions. The structural changes all over the codebase (implied by the change from "manual reference counting and intrusive pointers" to "naive, blanket std::shared_ptr everywhere") would be so large that the results could hardly be compared objectively. But even in a very narrow sope: If there was a measurable performance difference between Tile::_children being a vector<Tile> and a vector<shared_ptr<Tile>>, then I would at least consider this as "evidence". But I'm not sure whether it's worth trying that out.

That seems to be focussed on the usage side (i.e. what is supposed to be visible for clients). One of my concerns about manual reference counting referred to the guts of the implementation. I think that it could easily happen that someone is doing something on some thread and this affects the reference counter.

I'm talking about implementation, too. What you describe can't happen as long as we follow our rule of never passing a Tile to a worker thread. That's why we have that rule.

Which thread is calling 'upsample'?

Upsampling is done in a worker, but it does not require access to a Tile.
The glTF, however, is made available to the upsample process, which is ugly and dangerous but necessary.

The Boost documentation does not go into details about reasons for using intrusive_ptr

It lists three reasons:

Some existing frameworks or OSes provide objects with embedded reference counts;
The memory footprint of intrusive_ptr is the same as the corresponding raw pointer;
intrusive_ptr can be constructed from an arbitrary raw pointer of type T*.

shared_ptr is quite complicated, by necessity. Primarily because of two design decisions: the count being stored separately from the object being controlled, and the availability of weak_ptr. An intrusive pointer is simpler. That makes it smaller and faster.

One reason why I didn't look into all the details: Imagine there is a Tile that has to be created, and this can be a shared_ptr and be allocated on the heap and require atomic access for reference counting. All this is expensive. But then this Tile is filled with life, by doing a network request and donwloading 5 MB of data that has to be processed and uploaded to the GPU. I'm convinced that even the most naive approach for managing tiles will (in a real application, i.e. outside the scope of a microbenchmark) change the "average FPS over 100 frames" from 42.123 to something like 41.987 or so.

You missed a critical detail. Tile instances are created in bulk, sometimes many thousands at a time, and only a small fraction of those, sometime later, may have content loaded for them.
As a completely unsurprising (at least to me) point of reference, at one point I tried to switch the Tile loading code from its current approach, which reads the tile hierarchy from a tileset.json into a RapidJSON Document and then creates Cesium3DTilesSelection::Tile instances for each tile represented in the Document, to a slightly different approach that reads a Cesium3DTiles::Tileset and then creates Cesium3DTilesSelection::Tile instances from those. It took somewhere around twice as long to load the tile hierarchy, adding as much as a full second to the initial load time of a Tile-rich tileset like Melbourne Photogrammetry. Why? The JSON reader for Cesium3DTiles::Tileset should be extremely efficient. However, Cesium3DTiles::Tileset uses vectors of Tiles (just like Cesium3DTilesSelection::Tile), which requires a lot of heap allocations. Heap allocations are expensive.
Believe it or not, I'm not some performance zealot trying to make your life hard by imposing questionable rules without evidence. 😀

Believe it or not, I'm not some performance zealot trying to make your life hard by imposing questionable rules without evidence. 😀

I assume that you did tests and comparisons (beyond what you already know), and the point about "many tiles being created at once (and sometimes only few of them being filled later)" is certainly a valid one: It sounds like the two factors here are "multiplicative": "Heap-vs-Stack" (with the first one being slower) and "Individual-vs-Bulk" (with the first one being slower).
I'm lacking some context, but

Cesium3DTiles::Tileset uses vectors of Tiles ...

I don't see a vector<Tile> there, and if this referred to the Tile::children, I'd have to think through why this makes a difference. A guess: Could collecting these Cesium3DTIles::Tile instances (in one huge vector) and then bulk-creating the matching Cesium3DTilesSelection::Tile instances from them have been a reasonable attempt for alleviating the problem?
And an aside: I assume that this experiment was in the context (or at least related to) the consideration to replace the whole third-party-JSON parsing with the auto-generated cesium-native JSON handling. A devil's advocate question could be: If the auto-generated structures do not allow an implementation that is as efficient as manual JSON twiddling, what would have to be changed in order to allow such an efficient implementation?

In my naive, convenient Java world, there is no real "stack", and nothing that is not a "shared pointer", so to speak. And that may be a reason for me to (always keep performance in mind, but) lean towards the "safe and simple" approach, Often, time is wasted in unexpected places. For example, for that JSON parsing layer: I've heard rumors about ~20 MB tileset.json files, where questions like "SAX-vs-DOM" and maybe even "indented-vs-non-indented" may come into play, and the overarching question would be "Couldn't you split that into '1MB + (19 x 1MB)' tilesets?" ... with the latter being external ones... leading back to the purpose of this PR (sorry for derailing that a bit...).

if this referred to the Tile::children

Yep, that's the vector of tiles I'm referring to.

A guess: Could collecting these Cesium3DTIles::Tile instances (in one huge vector) and then bulk-creating the matching Cesium3DTilesSelection::Tile instances from them have been a reasonable attempt for alleviating the problem?

Yes, that would help significantly. Switching instead to a std::vector<any_kind_of_pointer<Tile>> would be a significant step in the wrong direction, even if we also moved to a single vector to hold all the tiles.
In theory it's possible to use a custom allocator to get std::vector<any_kind_of_pointer<Tile>> semantics while keeping close to std::vector<Tile> performance, but it's tricky. Particularly if the reason you're doing that is so that you can allocate/free individual Tiles whenever you like.

I assume that this experiment was in the context (or at least related to) the consideration to replace the whole third-party-JSON parsing with the auto-generated cesium-native JSON handling.

Yes that's right.

If the auto-generated structures do not allow an implementation that is as efficient as manual JSON twiddling, what would have to be changed in order to allow such an efficient implementation?

It's not that "manual JSON twiddling" is faster. It's that RapidJSON Document is obsessively focused on avoiding heap allocations. Basically, it's fast in all the ways the naive implementation is not. And that makes it win even though it requires some manual JSON twiddling.

In my naive, convenient Java world, there is no real "stack", and nothing that is not a "shared pointer", so to speak.

Interestingly, one of the advantages of having a garbage collector - especially a compacting, generational one - is that heap allocations are much faster.

Interestingly, one of the advantages of having a garbage collector - especially a compacting, generational one - is that heap allocations are much faster.

The other "advantage" is that as a developer, you don't have to give a ... ... care about why this is the case 🙂
Even though it might be a disadvantage now - I suspect this is related to fragmentation and the bookkeeping that is necessary for non-GC'ed heap allocation. As a "Java fanboi" (looking at my name here...) I did dig far deeper into the details of the language and JVM than most other developers, but eventually, the memory management and GC has to be treated as a "black box" (and its details may change over time). Things like https://softwareengineering.stackexchange.com/a/208676 seem to be reasonable summaries, though...

It's surprisingly hard to find concise, reliable information about the performance differences of heap-vs-stack in C++.
(By "hard" I mean that there doesn't seem to be much in the first 10 search results. By "concise" I mean that there are things like stack overflow answers that talk about this, and why one is faster than the other, but not how much. And by "reliable" I mean that https://publicwork.wordpress.com/2019/06/27/stack-allocation-vs-heap-allocation-performance-benchmark/ is easy to find and concise, but too narrowly focused on a too artifical microbenchmark).
One approach for avoiding the heap costs could be some sort of "pool" - that's probably what you referred to with..

In theory it's possible to use a custom allocator to get std::vector<any_kind_of_pointer<Tile>> semantics while keeping close to std::vector<Tile> performance, but it's tricky. Particularly if the reason you're doing that is so that you can allocate/free individual Tiles whenever you like.

And the trickiness obviously is: What can you do manually that is faster than what the new/delete operators are doing?
The first thought would be: Not much, probably. Except for very narrow, special cases of allocation/deallocation patterns. But... I could imagine that for Tile, we do have such a narrow case: Tiles form a hierarchy, and I could imagine that the constraints and patterns that are implied by having a tree structure could allow for specialized/optimized implementations of such a "pool". (Not trivial by any means, and ... even more unrelated to this PR than much of the previous disucssion, but a thought...)

And the trickiness obviously is: What can you do manually that is faster than what the new/delete operators are doing?

Yep. An allocator for a single type of object is a bit simpler than a general one, so that helps. It's also possible to do the sort of tricks that RapidJSON does: https://rapidjson.org/md_doc_internals.html#MemoryPoolAllocator
So something like: give each tileset its own pool for allocating Tile instances, and once memory is allocated from that pool, never return it. The pool itself can be freed by unloading the entire tileset. (by tileset I also mean external tileset, so external tileset subtrees can be unloaded without unloading the entire top-level tileset)
Actually now that I'm looking at the stackoverflow link you posted above, it actually kind of says what I just wrote above better than I did.
By the way, you don't have to convince me that good garbage collectors are awesome. I completely agree.

From a naive, high-level point, this now sounds like something that could actutally be ... "less non-trivial" ... than many other forms of pooling: Storing the tile.children as a std::vector<Pointer<Tile>>, where these are just pointing into one huge Tile array that is maintained in the tileset. But one would certainly have to think about some details here. For example, conditions for "compacting/defragmenting" this array. Unloading some external tileset might be one of them, but depending on how this "mapping" of "pointers to array entries" is established and maintained, the challenge would then be do do this without causing the usual "GC compacing pauses" that can easily ruin every interactive experience...


Do you have plan to create a new tag?

Question

A description of what you want to know.
Last tag 4.0.4 was created more than one year ago, there are many commits and pull requests after that. Do you have plan to create a new tag so that we can use latest code?

Environment:

OS/Version: [e.g. iOS/13.3]
Starscream Version [e.g. 4.0.4]
Xcode version [e.g. 11.5]

Yes, 4.0.5 was released.


Added link to get started with Camel in general.

This is for #3580

Thanks @spatnity!

I had to revert this commit as it breaks the website build
11650d6739097c3e0e80345add0b7ef250b0acd2
[check:html     ] public/camel-quarkus/next/user-guide/index.html
[check:html     ]   1:50606  error  For links within camel.apache.org use relative links, found: https://camel.apache.org/manual/getting-started.html  camel/relative-links
I dont know the top of my head how to do relative links between the projects (try to find in some other docs how that is done) and send another PR, thanks

I think the correct link should be like:
 xref:manual::getting-started.adoc[Getting started with Camel]

Thanks we can try this after this job is done
https://github.com/apache/camel-website/actions/runs/5129187679/jobs/9228170859?pr=1006

Okay lets try with
https://github.com/apache/camel-quarkus/commit/8f72af5e8a10ba53aaee1f23fbf0587bfcd0690c


0.5.0: Python error

Hi @tillsteinbach :-)
just tried the latest version 0.5.0:
docker-compose -f docker-compose-homekit-host.yml --env-file ./myconfig.env up
And got the following error:
vwsfriend_1   | INFO:VWsFriend:vwsfriend 0.5.0 (using WeConnect-python 0.20.8)
vwsfriend_1   | INFO:weconnect:Could not use token from file /tmp/weconnect.token ([Errno 2] No such file or directory: '/tmp/weconnect.token')
...
vwsfriend_1   | INFO:weconnect:Token refreshed
vwsfriend_1   | INFO:VWsFriend:Starting database upgrade if necessary
vwsfriend_1   | Traceback (most recent call last):
vwsfriend_1   |   File "/opt/venv/bin/vwsfriend", line 8, in <module>
vwsfriend_1   |     sys.exit(main())
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/vwsfriend.py", line 142, in main
vwsfriend_1   |     connector = AgentConnector(weConnect=weConnect, dbUrl=args.dbUrl, interval=args.interval, withDB=args.withDatabase, withABRP=args.withABRP,
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/agent_connector.py", line 32, in __init__
vwsfriend_1   |     run_database_migrations(dsn=dbUrl)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/migrations.py", line 18, in run_database_migrations
vwsfriend_1   |     upgrade(config, "head")
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
vwsfriend_1   |     script.run_env()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
vwsfriend_1   |     util.load_python_file(self.dir, "env.py")
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
vwsfriend_1   |     module = load_module_py(module_id, path)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
vwsfriend_1   |     spec.loader.exec_module(module)  # type: ignore
vwsfriend_1   |   File "<frozen importlib._bootstrap_external>", line 855, in exec_module
vwsfriend_1   |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 76, in <module>
vwsfriend_1   |     run_migrations_online()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 58, in run_migrations_online
vwsfriend_1   |     connectable = engine_from_config(
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 743, in engine_from_config
vwsfriend_1   |     return create_engine(url, **options)
vwsfriend_1   |   File "<string>", line 2, in create_engine
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 298, in warned
vwsfriend_1   |     return fn(*args, **kwargs)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 548, in create_engine
vwsfriend_1   |     dbapi = dialect_cls.dbapi(**dbapi_args)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 793, in dbapi
vwsfriend_1   |     import psycopg2
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module>
vwsfriend_1   |     from psycopg2._psycopg import (                     # noqa
vwsfriend_1   | ImportError: libpq.so.5: cannot open shared object file: No such file or directory
vwsfriend_vwsfriend_1 exited with code 1

Any clue what could be wrong?
Thanks!

Hi @ChristianWeyer,
doesn't look good :(
Two questions:

was it working before 0.5.0?
what is your Docker Host architecture? Some people have trouble on ARM (e.g. Raspberry Pi)

This is a new Apple MBP with M1 chip - so yeah... could be an issue :-/

Hmm, I,m asking myself if I miss to copy some libraries from the build to the runner stage, but then it should not work at all.

... happy to try out some things - if you have a clue ... ;-)

I will try to put up an experimental branch and add the library there. Might take some days until I find the time.
If you want to try things out you could login on that image and install libpq-dev, would be a good hint if that works.

Ok, lets go :)
Can you try to replace:
image: tillsteinbach/vwsfriend:latest
with
image: tillsteinbach/vwsfriend:experimental
and give it another try?

Some days... jajajaja :-)
Very cool, thanks!
OK, here we go:
vwsfriend_1   | INFO:weconnect:Token refreshed
vwsfriend_1   | INFO:VWsFriend:Starting database upgrade if necessary
vwsfriend_1   | Traceback (most recent call last):
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
vwsfriend_1   |     return fn()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
vwsfriend_1   |     return _ConnectionFairy._checkout(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
vwsfriend_1   |     fairy = _ConnectionRecord.checkout(pool)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
vwsfriend_1   |     rec = pool._do_get()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
vwsfriend_1   |     return self._create_connection()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
vwsfriend_1   |     return _ConnectionRecord(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
vwsfriend_1   |     self.__connect()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
vwsfriend_1   |     pool.logger.debug("Error on connect(): %s", e)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
vwsfriend_1   |     compat.raise_(
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1   |     raise exception
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
vwsfriend_1   |     connection = pool._invoke_creator(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
vwsfriend_1   |     return dialect.connect(*cargs, **cparams)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
vwsfriend_1   |     return self.dbapi.connect(*cargs, **cparams)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
vwsfriend_1   |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
vwsfriend_1   | psycopg2.OperationalError: could not connect to server: Connection refused
vwsfriend_1   | 	Is the server running on host "postgresdbbackend" (<IP_ADDRESS>) and accepting
vwsfriend_1   | 	TCP/IP connections on port 5432?
vwsfriend_1   |
vwsfriend_1   |
vwsfriend_1   | The above exception was the direct cause of the following exception:
vwsfriend_1   |
vwsfriend_1   | Traceback (most recent call last):
vwsfriend_1   |   File "/opt/venv/bin/vwsfriend", line 8, in <module>
vwsfriend_1   |     sys.exit(main())
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/vwsfriend.py", line 142, in main
vwsfriend_1   |     connector = AgentConnector(weConnect=weConnect, dbUrl=args.dbUrl, interval=args.interval, withDB=args.withDatabase, withABRP=args.withABRP,
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/agent_connector.py", line 32, in __init__
vwsfriend_1   |     run_database_migrations(dsn=dbUrl)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/migrations.py", line 18, in run_database_migrations
vwsfriend_1   |     upgrade(config, "head")
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
vwsfriend_1   |     script.run_env()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
vwsfriend_1   |     util.load_python_file(self.dir, "env.py")
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
vwsfriend_1   |     module = load_module_py(module_id, path)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
vwsfriend_1   |     spec.loader.exec_module(module)  # type: ignore
vwsfriend_1   |   File "<frozen importlib._bootstrap_external>", line 855, in exec_module
vwsfriend_1   |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 76, in <module>
vwsfriend_1   |     run_migrations_online()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 64, in run_migrations_online
vwsfriend_1   |     with connectable.connect() as connection:
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
vwsfriend_1   |     return self._connection_cls(self, close_with_result=close_with_result)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
vwsfriend_1   |     else engine.raw_connection()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
vwsfriend_1   |     return self._wrap_pool_connect(self.pool.connect, _connection)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
vwsfriend_1   |     Connection._handle_dbapi_exception_noconnection(
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
vwsfriend_1   |     util.raise_(
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1   |     raise exception
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
vwsfriend_1   |     return fn()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
vwsfriend_1   |     return _ConnectionFairy._checkout(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
vwsfriend_1   |     fairy = _ConnectionRecord.checkout(pool)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
vwsfriend_1   |     rec = pool._do_get()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
vwsfriend_1   |     return self._create_connection()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
vwsfriend_1   |     return _ConnectionRecord(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
vwsfriend_1   |     self.__connect()
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
vwsfriend_1   |     pool.logger.debug("Error on connect(): %s", e)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
vwsfriend_1   |     compat.raise_(
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1   |     raise exception
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
vwsfriend_1   |     connection = pool._invoke_creator(self)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
vwsfriend_1   |     return dialect.connect(*cargs, **cparams)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
vwsfriend_1   |     return self.dbapi.connect(*cargs, **cparams)
vwsfriend_1   |   File "/opt/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
vwsfriend_1   |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
vwsfriend_1   | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
vwsfriend_1   | 	Is the server
``` running on host "postgresdbbackend" (<IP_ADDRESS>) and accepting
vwsfriend_1   | 	TCP/IP connections on port 5432?
vwsfriend_1   |
vwsfriend_1   | (Background on this error at: https://sqlalche.me/e/14/e3q8)
vwsfriend_vwsfriend_1 exited with code 1

Hmm, is the Postgres server running? Maybe it takes too much time to Startup. Let me think…

It should run:

postgresdb_1  | PostgreSQL init process complete; ready for start up.
postgresdb_1  |
postgresdb_1  | 2021-09-11 12:44:58.863 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgresdb_1  | 2021-09-11 12:44:58.863 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
postgresdb_1  | 2021-09-11 12:44:58.863 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgresdb_1  | 2021-09-11 12:44:58.865 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgresdb_1  | 2021-09-11 12:44:58.868 UTC [62] LOG:  database system was shut down at 2021-09-11 12:44:58 UTC
postgresdb_1  | 2021-09-11 12:44:58.870 UTC [1] LOG:  database system is ready to accept connections

What happens if you keep Postgres running and just restart vwsfriend?

What would be the best way to do this with docker compose?

docker-compose up -d will start up the stack but puts the processes in the background, you can then do docker-compose up vwsfriend to start just the vwsfriend container

I used this command:
docker compose -f docker-compose-homekit-host.yml --env-file ./myconfig.env up vwsfriend
And still got the exception:
vwsfriend_1  | INFO:VWsFriend:vwsfriend develop (using WeConnect-python 0.20.8)
vwsfriend_1  | INFO:weconnect:Could not use token from file /tmp/weconnect.token ([Errno 2] No such file or directory: '/tmp/weconnect.token')
vwsfriend_1  | INFO:weconnect:Token refreshed
vwsfriend_1  | INFO:VWsFriend:Starting database upgrade if necessary
vwsfriend_1  | Traceback (most recent call last):
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
vwsfriend_1  |     return fn()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
vwsfriend_1  |     return _ConnectionFairy._checkout(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
vwsfriend_1  |     fairy = _ConnectionRecord.checkout(pool)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
vwsfriend_1  |     rec = pool._do_get()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
vwsfriend_1  |     return self._create_connection()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
vwsfriend_1  |     return _ConnectionRecord(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
vwsfriend_1  |     self.__connect()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
vwsfriend_1  |     pool.logger.debug("Error on connect(): %s", e)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
vwsfriend_1  |     compat.raise_(
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1  |     raise exception
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
vwsfriend_1  |     connection = pool._invoke_creator(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
vwsfriend_1  |     return dialect.connect(*cargs, **cparams)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
vwsfriend_1  |     return self.dbapi.connect(*cargs, **cparams)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
vwsfriend_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
vwsfriend_1  | psycopg2.OperationalError: could not connect to server: Connection refused
vwsfriend_1  | 	Is the server running on host "postgresdbbackend" (<IP_ADDRESS>) and accepting
vwsfriend_1  | 	TCP/IP connections on port 5432?
vwsfriend_1  |
vwsfriend_1  |
vwsfriend_1  | The above exception was the direct cause of the following exception:
vwsfriend_1  |
vwsfriend_1  | Traceback (most recent call last):
vwsfriend_1  |   File "/opt/venv/bin/vwsfriend", line 8, in <module>
vwsfriend_1  |     sys.exit(main())
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/vwsfriend.py", line 142, in main
vwsfriend_1  |     connector = AgentConnector(weConnect=weConnect, dbUrl=args.dbUrl, interval=args.interval, withDB=args.withDatabase, withABRP=args.withABRP,
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/agent_connector.py", line 32, in __init__
vwsfriend_1  |     run_database_migrations(dsn=dbUrl)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/migrations.py", line 18, in run_database_migrations
vwsfriend_1  |     upgrade(config, "head")
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/alembic/command.py", line 320, in upgrade
vwsfriend_1  |     script.run_env()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/alembic/script/base.py", line 563, in run_env
vwsfriend_1  |     util.load_python_file(self.dir, "env.py")
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
vwsfriend_1  |     module = load_module_py(module_id, path)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
vwsfriend_1  |     spec.loader.exec_module(module)  # type: ignore
vwsfriend_1  |   File "<frozen importlib._bootstrap_external>", line 855, in exec_module
vwsfriend_1  |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 76, in <module>
vwsfriend_1  |     run_migrations_online()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/vwsfriend/model/vwsfriend-schema/env.py", line 64, in run_migrations_online
vwsfriend_1  |     with connectable.connect() as connection:
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
vwsfriend_1  |     return self._connection_cls(self, close_with_result=close_with_result)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
vwsfriend_1  |     else engine.raw_connection()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
vwsfriend_1  |     return self._wrap_pool_connect(self.pool.connect, _connection)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
vwsfriend_1  |     Connection._handle_dbapi_exception_noconnection(
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
vwsfriend_1  |     util.raise_(
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1  |     raise exception
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
vwsfriend_1  |     return fn()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
vwsfriend_1  |     return _ConnectionFairy._checkout(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
vwsfriend_1  |     fairy = _ConnectionRecord.checkout(pool)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
vwsfriend_1  |     rec = pool._do_get()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
vwsfriend_1  |     return self._create_connection()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
vwsfriend_1  |     return _ConnectionRecord(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
vwsfriend_1  |     self.__connect()
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
vwsfriend_1  |     pool.logger.debug("Error on connect(): %s", e)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
vwsfriend_1  |     compat.raise_(
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
vwsfriend_1  |     raise exception
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
vwsfriend_1  |     connection = pool._invoke_creator(self)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
vwsfriend_1  |     return dialect.connect(*cargs, **cparams)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
vwsfriend_1  |     return self.dbapi.connect(*cargs, **cparams)
vwsfriend_1  |   File "/opt/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
vwsfriend_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
vwsfriend_1  | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
vwsfriend_1  | 	Is the server running on host "postgresdbbackend" (<IP_ADDRESS>) and accepting
vwsfriend_1  | 	TCP/IP connections on port 5432?
vwsfriend_1  |
vwsfriend_1  | (Background on this error at: https://sqlalche.me/e/14/e3q8)
vwsfriend_1 exited with code 1

hmmmm, why can't it reach the database...

I think I found the problem. The homekit-host config is missing ports:       - 5432:5432
Can you use the new one I just uploaded? (remember to change to "experimental" for now until I merged that into the main branch

Looks better :-)
But when surfing to http://localhost:4000/ I get

But it says
vwsfriend_1   | INFO:VWsFriend:VWsFriend is listening on <IP_ADDRESS>:4000)
vwsfriend_1   | INFO:VWsFriend:Updating data from WeConnect

We will get there ;) I just saw that the Dockerfile may be missing an EXPOSE 4000. I'm just building a new image. Should be there in ~30min

BTW: Thanks for debugging with me. Improved the stability already a lot today!

Gerne ;-)

Updated Image should be there

I deleted all the images locally and pulled again.
Still, the same error...

Hmm, will try out some things later

If I understand the internet right docker host mode on OS X behaves a bit different than on other OS. I now also understand why. Docker is running a linux container, thus it needs to run it in a virtual machine. So probably you cannot connect to the container via localhost but some other IP address that you need to find out.
The problem is: The whole purpose of the host networking is to make multicast work (what is necessary for HomeKit). I expect that this won't work in this virtual machine anyways. so the whole purpose of the host network setup is gone.
To sum it up: You can probably only use the config without HomeKit. Or try to find a linux machine to host the docker :(

... will take some time for me to test this out ... busy :-/

No problem. If you don't mind I close this issue for now.


work around checksum mismatch during cleanvm

Default to one mirror we know is updated fast enough instead of the
redirector because some mirrors lag and cause a checksum mismatch for rpms
that keep name and size and only change content.
This happened only on Leap 15 for Rocky when using osc release based
copy to the ToTest project, probably because otherwise changes like this
don't happen often enough.
Example from the log (the received checksum was from the day before):
Warning: Digest verification failed for file 'openstack-suse-sudo-2015.2-lp150.2.1.noarch.rpm'
[/var/tmp/AP_0xqwI2Is/noarch/openstack-suse-sudo-2015.2-lp150.2.1.noarch.rpm]
expected 388d766a8213163cb1ab36c4705f8a16958a9c8f3abfd10b715df2cf996a44cf
but got  d95494a9af9ae19810c4f0a818c01b1ec09fa53458739d26c0e1a53ad16c0bcc

For my understanding - is this a missmatch of the calculated checksum for the .rpm package and the checksum that is in the ...-primary.xml.gz mentioned?

yes. Because these two files can be served by two different mirrors. For the primary.xml, the digest in the name already ensures that we always get the latest version.
downloadcontent.opensuse.org also has the advantage of being in Nuremberg, so we do not waste much external bandwidth.

yes. Because these two files can be served by two different mirrors. For the primary.xml, the digest in the name already ensures that we always get the latest version.

Isn't that a server setup failure then? Why is the nearest mirror calculated diffferently?

downloadcontent.opensuse.org also has the advantage of being in Nuremberg, so we do not waste much external bandwidth.

True as long as cleanvm runs in nbr.

Why is the nearest mirror calculated diffferently?

I don't have an example handy if it was in this case, to make sure there was no bug in the distance calculation. But the mirror redirected to is also decided on other criteria, so anything that uses the redirector must cope with it anyway.

The redirector is 'mirrorbrain' written in C and I think it uses a GeoIP DB to find your country and gives you a mirror in your country (using randomness and a mirror weight value)


[WIP] Bug 1714775: OpenAPI schema for CRD (4.1)

/retest

/lgtm

/hold
4.1.z is not fully open yet.

/hold cancel
Now have cherry pick approvals

/lgtm
Re-tagging

/cc @sudhaponnaganti @nstielau

@dmage are you proposing this for 4.1.2?  if so, can you add "4.1.2" to the whiteboard on https://bugzilla.redhat.com/show_bug.cgi?id=1714775 ?

@nstielau can you put the cherry-pick-approved label?

/hold
Our current generator is not creating top-level descriptions, leading to empty descriptions in oc explain

prior to us picking this back in 4.1.z, can we clone the bz in master and only pick once verified there?  changes like this should be verified in master before picked back in support streams.

@derekwaynecarr the bug for master is https://bugzilla.redhat.com/show_bug.cgi?id=1705752

/lgtm cancel
(given this is not fixed in master re: https://github.com/openshift/cluster-image-registry-operator/pull/293#issuecomment-501419736)

/bugzilla refresh

/bugzilla refresh

/hold cancel

@dmage I noticed that openshift/installer is floating on master in this branch. We should change Gopkg.toml so that it references the release-4.1 branch. We do not want to pull in 4.2 code here by accident.

/bugzilla refresh
/hold cancel
Is the bot broken?

This pr requires engineering team review/lgtm to move forward.

@dmage @adambkaplan is there still a plan to move this PR forward?

@bparees yes, I want to backport the CDR from master next week.

These commits can't be backported. github.com/openshift/crd-schema-gen doesn't have the release-4.1 branch, the branch release-4.2 is not compatible with sigs.k8s.io/controller-tools/pkg/crd from 4.1.
	origin-4.0-kubernetes-1.12.4: Could not introduce sigs.k8s.io/controller-tools (from<EMAIL_ADDRESS>due to multiple problematic subpackages:
	Subpackage sigs.k8s.io/controller-tools/pkg/crd does not contain usable Go code (*build.NoGoError).. (Package is required by<EMAIL_ADDRESS>Subpackage sigs.k8s.io/controller-tools/pkg/genall is missing. (Package is required by<EMAIL_ADDRESS>
The new version executed from a container complains on openshift/api types:
Copying "vendor/github.com/openshift/api/operator/v1/"* to temporary "/go/src/github.com/openshift/cluster-image-registry-operator/598764604/pkg/apis"...
/go/src/github.com/openshift/cluster-image-registry-operator/598764604/pkg/apis/types.go:49:2: must apply pattern to a string
/go/src/github.com/openshift/cluster-image-registry-operator/598764604/pkg/apis/types_network.go:164:28: map values must be a named type, not *ast.ArrayType
error running generator

Older versions of crd-schema-gen are known to be buggy.
@bparees @adambkaplan does it worth to continue?

@sttts do we have a better means of back-porting the CRD generators for 4.1.z?

The approach in https://github.com/openshift/cluster-kube-apiserver-operator/pull/562 (with a temporary GOPATH) might work for you. This decouples dependencies of the openapi generator from the project your are generating for.

Given the age of this, do we think we should just WONTFIX?

@eparis spoke with @pweil- this morning - we will close as WONTFIX as this has been addressed in 4.2

/close


gz sdf always print "Error: SDF parsing the xml failed"

Environment

OS Version:  Ubuntu 20.04
Source or binary build?
Binary, it comes with gazebo11.11

Description

Expected behavior: Expect to verify an SDF file and provide a useful error message.
Actual behavior: always complain "Error: SDF parsing the XML failed".

Steps to reproduce

Prepare an SDF file, model.sdf,

<?xml version="1.0"?>
<sdf version="1.5">
  <model name="a_simple_box">
    <static>false</static>
    <link name="link">
      <pose>0 0 1 0 0 0</pose>
      <collision name="collision">
        <geometry>
          <box>
            <size>.3 .2 .5</size>
          </box>
        </geometry>
      </collision>
      <visual name="visual">
        <geometry>
          <box>
            <size>.3 .2 .5</size>
          </box>
        </geometry>
      </visual>
    </link>
  </model>
</sdf>

to get mentioned error message.
2. Run gz sdf --verbose -k model.sdf
3. Run cp model.sdf modelv.sdf && gz sdf --verbose -k modelv.sdf` to get "Check complete".

Thanks for the issue @huweiATgithub. Since this is an SDFormat issue, would you mind creating the same issue at https://github.com/gazebosim/sdformat?


"unexpected signal during runtime execution" when setting compile time options

When I run go build with sqlite compile time flags such as CGO_CFLAGS="-DSQLITE_OMIT_AUTOINIT", go panics with unexpected signal during runtime execution.
Is it possible to set sqlite compile time flags? Is there another way that this should be done?
 But this is not guaranteed. In general, always compile from canonical sources in order to take advantage of SQLITE_OMIT_* options.

You likely need to build your own version of SQLite from scratch if you want such customization.

Is there a better way to build a custom SQLite for use with go-sqlite3?
Since the c code is in the go-sqlite3 repo and appears to be built with cgo, I assumed that directly changing the code in the repo or setting CGO_CFLAGS would be the way to do it. I also tried adding this flag to sqlite3.go in the repo with the same result.

The C code in this repo is the aforementioned amalgamation. Therefore, as per the SQLite docs, just setting CFLAGS alone is insufficient.
Either you will have to fork this repo to replace it with your own custom build amalgamation, or you may be able to get it to work with the libsqlite3 build tag by directing it to link against your custom library. I've never done it, so I cannot offer any further detail.


Add collenchyma as a dev dep so the tests can build.

LGTM. Thank you, @DiamondLovesYou

@homu r+

:pushpin: Commit 258545b has been approved by MichaelHirn

:zap: Test exempted - status


chore: Add react 19 as a peer dependency

fixes #346

[x] I have followed (at least) the PR section of the contributing guide.

@brijeshb42 - Looks like the PR needs a label in order to pass the final check.

@mnajdova - Any objections here?


jreleaser config --generate?

Is your feature request related to a problem? Please describe.
I run  jreleaser config and gets wall of:
some.value must not be blank. Configure a value using the CLI yaml DSL, or define [jreleaser.some.value] System properties, or define [JRELEASER_SOME_VALUE] environment variables, or define a key/value pair in /Users/manderse/.jreleaser/config.toml with keys [JRELEASER_SOME_VALUE]
<repeat 10-15 times for some other value:)>

Describe the solution you'd like
would like a way to have output with the right keys given the format wanted/needed.
like:
jreleaser config --generate=toml
which would then let me copy the output into my config used format.

If I understand correctly, what you expect is a file in the chosen format that contains suggested key/values for all missing settings that may be set using env vars?

Yes.
Since All My attempts to write the suggested config is failing :)

This may be doable.


[GLUTEN-5102][VL] Support cast date as timestamp in velox

What changes were proposed in this pull request?
This PR aims to support cast date as timestamp in velox. After this, we can run cast(date as timestamp) with velox.
(Fixes: #5102)
How was this patch tested?
Add new unit test.

@rui-mo would you please review this PR?

@rui-mo This PR is ready to review. At the begin, the test failures are the result that I set the expectedAnswer in checkAnswer to df.show. In spark, timestamp behaves differently in df.show and df.collect. Now use the result of df.collect for verification. The behavior of velox is consistent with that of spark

@PHILO-HE Could you help confirm? Thanks.

@rui-mo This PR is ready to review. At the begin, the test failures are the result that I set the expectedAnswer in checkAnswer to df.show. In spark, timestamp behaves differently in df.show and df.collect. Now use the result of df.collect for verification. The behavior of velox is consistent with that of spark

See the posted comment https://github.com/apache/incubator-gluten/pull/5240#discussion_r1550934905. cc @rui-mo


Add CNN vessel segmentation documentation pdf

@aylward As of now i have put the data in a private folder on MIDAS here. Should i make it public?

Perfect - yes, please make it public.
Thanks!
s
On Thu, Aug 18, 2016 at 9:49 AM, Deepak Roy Chittajallu <
<EMAIL_ADDRESS>wrote:

@aylward https://github.com/aylward As of now i have put the data in a
private folder on MIDAS here
https://midas3.kitware.com/midas/community/7. Should i make it public?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/KitwareMedical/TubeTK/pull/805#issuecomment-240728264,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARErzV4yzrWndMPW3XmR2WTqTDe-isWks5qhGLPgaJpZM4JnFzS
.

--
Stephen R. Aylward, Ph.D.
Senior Director of Operations, North Carolina, Kitware, Inc.
http://www.kitware.com and http://www.aylward.org
(919) 969-6990 x300


nvm prefix error appears in VS Code terminal only when default alias is present

Operating system and version: MacOS High Sierra 10.13

nvm debug output:

nvm --version: v0.33.6
$TERM_PROGRAM: Apple_Terminal
$SHELL: /bin/bash
$HOME: /Users/stonehippo
$NVM_DIR: '$HOME/.nvm'
$PREFIX: ''
$NPM_CONFIG_PREFIX: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)'
uname -a: 'Darwin 17.0.0 Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64 x86_64'
OS version: Mac 10.13 17A405
curl: /usr/bin/curl, curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
wget: /usr/local/bin/wget, GNU Wget 1.19.2 built on darwin17.0.0.
git: /usr/local/bin/git, git version 2.14.3
grep: /usr/local/bin/grep, grep (GNU grep) 3.1
awk: /usr/local/bin/awk, GNU Awk 4.2.0, API: 2.0 (GNU MPFR 3.1.6, GNU MP 6.1.2)
sed: /usr/local/bin/sed, sed (GNU sed) 4.4
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
cut: /usr/bin/cut,
basename: illegal option -- -
usage: basename string [suffix]
       basename [-a] [-s suffix] string [...]
basename: /usr/bin/basename,
sed: -e expression #1, char 9: Unmatched ) or \)
rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file
rm: ,
sed: -e expression #1, char 9: Unmatched ) or \)
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
mkdir: ,
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
             [-L number] [-n number [-x]] [-P maxprocs] [-s size]
             [utility [argument ...]]
xargs: /usr/bin/xargs,
nvm current: v8.8.1
which node: $NVM_DIR/versions/node/v8.8.1/bin/node
which iojs:
which npm: $NVM_DIR/versions/node/v8.8.1/bin/npm
npm config get prefix: $NVM_DIR/versions/node/v8.8.1
npm root -g: $NVM_DIR/versions/node/v8.8.1/lib/node_modules

nvm ls output:

        v6.11.5
->       v8.8.1
default -> 8.8.1 (-> v8.8.1)
node -> stable (-> v8.8.1) (default)
stable -> 8.8 (-> v8.8.1) (default)
iojs -> N/A (default)
lts/* -> lts/boron (-> v6.11.5)
lts/argon -> v4.8.5 (-> N/A)
lts/boron -> v6.11.5

How did you install nvm? (e.g. install script in readme, homebrew): Install script

What steps did you perform?

Installed nvm via the standard script in MacOS terminal, installed node 8.8.1 (default alias was created) and lts/boron, then opened VSCode (with and without extensions), started terminal.

What happened?

The VS Code terminal displayed the following error at startup:
nvm is not compatible with the npm config "prefix" option: currently set to "/usr/local"
Run `npm config delete prefix` or `nvm use --delete-prefix v8.8.1 --silent` to unset it.

This error does not appear in MacOS terminal, only the VS Code terminal. Removing the default alias in .nvm/alias removes the error message at startup (and node continues to function correctly).
The node environment does not appear to get set up correctly as shown in this run of nvm debug from the VS Code terminal when the issue appears:
nvm --version: v0.33.6
$TERM_PROGRAM: vscode
$SHELL: /bin/bash
$HOME: /Users/stonehippo
$NVM_DIR: '$HOME/.nvm'
$PREFIX: ''
$NPM_CONFIG_PREFIX: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)'
uname -a: 'Darwin 17.0.0 Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64 x86_64'
OS version: Mac 10.13 17A405
curl: /usr/bin/curl, curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
wget: /usr/local/bin/wget, GNU Wget 1.19.2 built on darwin17.0.0.
git: /usr/local/bin/git, git version 2.14.3
grep: /usr/local/bin/grep, grep (GNU grep) 3.1
awk: /usr/local/bin/awk, GNU Awk 4.2.0, API: 2.0 (GNU MPFR 3.1.6, GNU MP 6.1.2)
sed: /usr/local/bin/sed, sed (GNU sed) 4.4
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
cut: /usr/bin/cut,
basename: illegal option -- -
usage: basename string [suffix]
       basename [-a] [-s suffix] string [...]
basename: /usr/bin/basename,
rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file
rm: /bin/rm,
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
mkdir: /bin/mkdir,
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
             [-L number] [-n number [-x]] [-P maxprocs] [-s size]
             [utility [argument ...]]
xargs: /usr/bin/xargs,
nvm current: none
which node:
which iojs:
which npm: /usr/local/bin/npm
npm config get prefix: env: node: No such file or directory
npm root -g: env: node: No such file or directory

Restoring the default alias triggers the error message. In addition, it is possible to invoke nvm use default and get the environment to work. The issue occurs on at startup.

What did you expect to happen?

The error should not occur when the default alias is present, and nvm should load the version assigned to the default alias.

Is there anything in any of your profile files (.bashrc, .bash_profile, .zshrc, etc) that modifies the PATH?

No

What is pwd in the VSCode terminal? If you cd into that directory in the normal terminal, and run nvm use node, can you replicate the error?

@ljharb The error occurs regardless of the directory. For example, I can get it to happen in my home directory or in a project directory.

Gotcha, and presumably only in VSCode and nowhere else.
In VSCode terminal, what does npm config get prefix return?

In my MacOS terminal, it returns /Users/stonehippo/.nvm/versions/node/v8.8.1.
If I try it the VS Code terminal when it first starts, error: env: node: No such file or directory. I can invoke nvm use default at that point, and it's the same as above. So the issue only seems to happen right at the startup of the terminal.

@ljharb Sorry, I think the above was unclear. Once I do nvm use default in the VS Code terminal, the result of npm config get prefix becomes /Users/stonehippo/.nvm/versions/node/v8.8.1, which is what shows up in the MacOS terminal all the time.

I think i see the problem - you don't have a system node installed, but somehow you have a system npm at /usr/local/bin/npm - why is this there (without node)? If you remove it, does everything work?

@ljharb Aha! That's it. It looks like the "system" node wasn't. What is showing up is actually a node installed by brew (as a dependency from another package). I killed that node, but it looks like it left npm and npx behind. Once I killed those, the error went away.
I was able to confirm this on a second system. First, cleaned of the "system" node and npm/npx, confirmed the error was gone, then reinstalled it with brew install node. Error returned. So that was it, for sure. I just tried to install a system node with the official installer, same problem. So it's the system npm getting called first. Any ideas for preventing this, or should I go file this with the VS Code team?

I think this isn't actually an issue with either; it was an issue with your system. Installing node with brew can cause problems; brew uninstall node is supposed to remove npm along with node; if you ran brew install npm separately from node, that's another problem (npm only ever comes with node and should never be installed separately).
Either it's an issue with homebrew (npm should never exist on the system without node), or an issue with VSCode (does it use the same PATH as on your real terminal?) - I'm not really sure which.

@ljharb hmm. I agree that the leftover npm is a brew issue (it should really have gone when node is uninstalled). But I can make this happen with a legit install of the official node package. For example, here's nvm debug and nvm ls after installing a system node:

nvm debug

nvm --version: v0.33.6
$TERM_PROGRAM: vscode
$SHELL: /bin/bash
$HOME: /Users/stonehippo
$NVM_DIR: '$HOME/.nvm'
$PREFIX: ''
$NPM_CONFIG_PREFIX: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)'
uname -a: 'Darwin 17.0.0 Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64 x86_64'
OS version: Mac 10.13 17A405
curl: /usr/bin/curl, curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
wget: /usr/local/bin/wget, GNU Wget 1.19.2 built on darwin17.0.0.
git: /usr/local/bin/git, git version 2.14.3
grep: /usr/local/bin/grep, grep (GNU grep) 3.1
awk: /usr/local/bin/awk, GNU Awk 4.2.0, API: 2.0 (GNU MPFR 3.1.6, GNU MP 6.1.2)
sed: /usr/local/bin/sed, sed (GNU sed) 4.4
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
cut: /usr/bin/cut,
basename: illegal option -- -
usage: basename string [suffix]
       basename [-a] [-s suffix] string [...]
basename: /usr/bin/basename,
rm: illegal option -- -
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file
rm: /bin/rm,
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
mkdir: /bin/mkdir,
xargs: illegal option -- -
usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
             [-L number] [-n number [-x]] [-P maxprocs] [-s size]
             [utility [argument ...]]
xargs: /usr/bin/xargs,
nvm current: system
which node: /usr/local/bin/node
which iojs:
which npm: /usr/local/bin/npm
npm config get prefix: /usr/local
npm root -g: /usr/local/lib/node_modules

nvm ls

        v6.11.5
         v8.8.1
->       system
default -> 8.8.1 (-> v8.8.1)
node -> stable (-> v8.8.1) (default)
stable -> 8.8 (-> v8.8.1) (default)
iojs -> N/A (default)
lts/* -> lts/boron (-> v6.11.5)
lts/argon -> v4.8.5 (-> N/A)
lts/boron -> v6.11.5

Now that I know what to look for, I can see the legit which statements for my system node, and I am getting the error in the VS Code terminal. Remove the system node, no issues. That doesn't seem system specific. I can reproduce this on two systems: it's possible they've both got the same issue, but it doesn't seem to be brew's fault in the end.

Gotcha; in that case it's a VSCode issue, since it's differing from your system terminal. Perhaps there's VSCode configuration that sets the PATH differently, or attempts to locate node/npm?

@ljharb I think it's an issue with VSCode, as well, maybe specific to MacOS High Sierra. I've opened an issue over there, and I'm going to close this one for now. I tried the same setup in an Ubuntu VM, and no issues (obviously no brew, either). If I get a chance, I'll also try to reproduce on a Mac that's never had brew installed.
Thanks for the help.

@ljharb I was able to replicate this issue on a system that never had brew, node, VS Code or nvm on it before. I installed node via the official package, nvm + a node (which got get to default) and VS Code. With this setup, same error as before, so it seems like an interaction between nvm.sh and the VS Code terminal.

I'm reopening this one, as it seems to be related to a known issue with nvm, which is that NVM has trouble when starting up in a nested shell. @charsleysa notes that VS Code appears to do just that in it's terminal, as revealed by
echo $SHLVL     // returns 2 in VS Code terminal

So this looks like it very similar to https://github.com/creationix/nvm/issues/1652.
Here's a reference to the VS Code issue comment that points back to NVM:
https://github.com/Microsoft/vscode/issues/37296#issuecomment-341504374

I don't know if removing or uninstalling node via brew is a good idea. Especially because it's a dependency for yarn.
For example:
$ brew uninstall node
Error: Refusing to uninstall /usr/local/Cellar/node/8.9.0
because it is required by yarn 1.3.2, which is currently installed.
You can override this and force removal with:
  brew uninstall --ignore-dependencies node

a) you can install yarn via npm, or via non-brew means; b) yarn should be able to use nvm's node without brew installing node.

And you can install Yarn through Homebrew without node:
$ brew install yarn --without-node

To fix it, in addition to delete npm on /usr/local/bin,  I also have to delete node file on /usr/local/bin

I had the same issue after installing node using homebrew.
The command brew uninstall node leaves behind /usr/local/bin/npm
a quick rm -rf /usr/loca/bin/npm solved it!

I believe this has been resolved in an update to nvm itself. Please reopen if not.

Actually after removing /usr/loca/bin/npm it solved my prefix issue but I have an app that is spawning a process using node and the command is npm run dev on my terminal it works but from the app it errors out this is the error.
/bin/sh: npm: command not found So I guess the question is how do I point npm from the nvm folder to the /bin/sh?

npm is provided by node; nvm sets up the PATH so that node and npm are available. If your PATH is not set up properly, that implies a few things: that the shell giving the error isn’t sourcing nvm, but also, your app may not be spawning a child process with the correct environment.

https://www.zhaofinger.com/detail/68

I think i see the problem - you don't have a system node installed, but somehow you have a system npm at /usr/local/bin/npm - why is this there (without node)? If you remove it, does everything work?

I found that incomplete uninstall of homebrew's nodes will cause this problem, to find them all try use ls -l /usr/local/bin/ | grep node .


Signature Match when validating assertions

Lots of API improvements and new endpoints for assertions have been added in the new-front branch.
However, signature matching still fails.
@aseigler Do you see anything wrong with how I extract the public key, concatenate bytes or do the verify call?
Code is here:
https://github.com/abergs/fido2-net-lib/commit/3e1237f8fec3689f63980ac7bd129eaea405bd4d#diff-16c131d9cd22eb6badda0902b79690cfR134
(line 131-133 is just dummy test code to make to test signatures)

I wasn't able to get any of the routes to controller working, so I wasn't able to debug this. The verify call looks fine but it's hard to tell if the inputs are ok without seeing them.

	Thank you for looking in to it. Did you check out the new branch? Index.html should work (you get some 404 but they are not affecting the attestation or assertion calls.

	/Anders (from phone)

On Mon, Aug 6, 2018 at 11:17 PM +0200, "Alex Seigler"<EMAIL_ADDRESS>wrote:
I wasn't able to get any of the routes to controller working, so I wasn't able to debug this. The verify call looks fine but it's hard to tell if the inputs are ok without seeing them.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I was trying use the register user button on the login.html page, get 404 to makeCredential (on new branch). Didn't have enough time to get any further.

	I will look into it tomorrow morning and see if I messed something up.

	/Anders (from phone)

On Mon, Aug 6, 2018 at 11:46 PM +0200, "Alex Seigler"<EMAIL_ADDRESS>wrote:
I was trying use the register user button on the login.html page, get 404 to makeCredential (on new branch). Didn't have enough time to get any further.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I figured out what you are doing.  The signature is malformed.  In your test code, like 132, you will find the signature is 64 bytes long, which is correct length.  If you take the exact string for signature from assertionNoneResponse.json, feed that to StringToByteArray(), then feed the result of that to ParseSigData(), the result will work.  Like this:
var cng = CngKey.Import(publicKey, CngKeyBlobFormat.EccPublicBlob);
var pubKey = new ECDsaCng(cng);
var sig = StringToByteArray("3046022100f20793dc46905c62fdbc309506b59c7cfa4fafc42dc2e51af4e58f474f64ba78022100c550af5e6516816517ddf83706fab02fe6287c04c8747a0621949456792111b6");
var parsedSig = ParseSigData(sig);
var signatureMatch = pubKey.VerifyData(concatedBytes, parsedSig, HashAlgorithmName.SHA256);
if (!signatureMatch) throw new Fido2VerificationException("Signature did not match");
Something about how the signature was stored or how it is being retrieved from the json file is causing the problem.  I understand what you are doing now with the tests, and will write some cases.

Because of the custom JsonConvert we do not need to use StringToByteArray in the actual solution but I didn't realize I needed to perform the PareSigData.. We should probably add more docs to that function to explain why it parses it.
👍 Good find!


Enable compute accesses from the controller-0

Until now, we had to check the IP in the ansible inventory. While it
works "well" when it comes to mono-hypervisor, the ssh command is a bit
more complicated to tweak when it comes to multi-hypervisor.
As a pull request owner and reviewers, I checked that:

[X] Appropriate testing is done and actually running

recheck

/hold
some templating issues - working on it right now

/unhold


refactor: update trestle documentation webpage's Demo section to be in sync with the demo repo

Types of changes

[ ] Hot fix (emergency fix and release)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Documentation (change which affects the documentation site)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Release (develop -> main)

Quality assurance (all should be covered).

[x] My code follows the code style of this project.
[x] Documentation for my change is up to date?
[x] My PR meets testing requirements.
[x] All new and existing tests passed.
[x] All commits are signed-off.

Summary

The demo repo was recently updated. The trestle documentation webpage should also be updated.

Key links:
Before you merge

Ensure it is a 'squash commit' if not a release.
Ensure CI is currently passing
Check sonar. If you are working for a fork a maintainer will reach out, if required.

@Ma1h01 Please check the corresponding checks on section Quality assurance (all should be covered).

@AleJo2995  Quality Assurance sections are now checked

@Ma1h01 update the branch and squash and merge when you can. The other problem related with the code of conduct will be resolved in other branch. Thanks


[8.x] Add Str::studlyWords()

Description
This PR adds a new Stringable method to convert a string to Studly Words.
Purpose

There is studly() but it concatenates words together
There is title(), but it converts a series of words to ucfirst without separating words that are underscored, hyphenated, etc.

Use Case
This is super useful for displaying "base-named" PHP classes to user interfaces as titles:
// Displays: "Voice Recording Stored"
echo Str::studlyWords(
    class_basename(\App\Events\VoiceRecordingStored::class)
);

Or even taking slugs and converting them to a title:
$slug = "using-laravel-with-redis";

// Displays: "Using Laravel With Redis"
echo Str::studlyWords($slug);

Thanks for your time! No hard feelings on closure ❤

Yea absolutely! I don't mind displayableTitle() if you like it.
Here's a couple ideas as well (let me know your thoughts):

Str::human()
Str::humanTitle()
Str::readable()
Str::readableTitle()
Str::headline()

Yeah headline is a pretty good thought 👍

Ok great -- let me update the PR with the name change. I'll comment back when it's ready 👍


Add option to specify ScriptRoot for func host start action

This will allow running the functions host from other directories than cwd, which is useful for tooling scenarios where artifacts do not live in the same directory as the source code.

closed in #266


Feat: Migration to Typescript

Description
Migration to Typescript for a better IDE experience and other benefits that type can bring to us.
Details

Migration source code to .ts, and remove unnecessary jsdoc.
Add dependencies typescript, @babel/preset-typescript, and ts-loader for building .ts sources.
Add dependencies tsc-alias to transform path alias in .d.ts files under the dist/.

Test
I have tested it with Vue3 and Vue2 projects, and you can try this PR by installing the tarball:
https://we.tl/t-tYunQL182v

@DevRickLin fix:


[IMAGING-260]: Remove JavaNCSS report plugin fixing the site build

Coverage remained the same at 76.304% when pulling a2904178909e1db3157dbce131a0bf166cfbd8f5 on kinow:disable-ncss into bca1100f878f4a174cef26c9b406c1c92a54bc92 on apache:master.


Tabs ideas

Hi folks,
I'm looking for implementing tabs in my browser,so I need some ideas for that.
Please respond to the following question:
1.Where the tabs should be located? (in the navigation drawer,or above the action bar,etc.)

I really like the navigation drawer idea, the way it's also implemented right now.

@tobiasvandriessel  Thanks for your ideea! Tabs will be added soon!

Tabs are located in thhe left navigation drawer and they work well.
This issue is CLOSED!


Secret2

What
Describe the change
Why
Describe why the change was necessary
Link to Trello card (if applicable):

Closing as I believe this was part of a test that is now complete


How to deal with integration test resources?

Hello,
we are using pybuilder for pyspark based projects; we extensively use intergrationtests that launch an in memory spark cluster and run some tests using test data files.
In our current project layout we have setup a ./data folder that contains the test data and is committed to version control. We generate output data in ./target/output which we validate.
In a production setup, the root folder where to read and write data would however be the same. Moreover, the test data seems to be more logically placed at ./src/integrationtest/resources.
For that to work, we configure our pybuilder projects to first copy the integration test resources to ./target/output. For the integration tests to properly function, we should be able to set the working directory to ./target/output when running these tests.
Would something like that be feasible or complex to configure? Does pybuilder need an extension for this?
Thanks for the input!

If you're running an integration test what prevents you from os.chdir to the directory you want at the beginning of every test?

hmmm ... we are standardizing on pybuilder for our big data pipelines projects at a big bank. So, ideally, we'll be writing 1000s of tests. I prefer it in the tool or project setup rather than in each test ;-)
But, using your suggestion,  I added some tasks to the project setup using the dependents and depends decorators to be triggered before and after the integration test.
I'll turn that into an plugin to be reusable by all the projects.
Thanks for the input!

Or you can do os.chdir in base class for the said thousands of tests ;)

tests will also generate files ... that I don't want to mix with the test resources. That's why I want to have everything end up in target somewhere.


load data functionality from within a callback in chart creation

Hi;
thanks for the great library.
I want to create my charts empty (no data) and load the data afterwards. the only reason for this is the animation that the load function possess that I cant seem to get in chart creation (if I missed it please do tell). for example, using load for a guage chart post creation, will animate the value within the gauge as opposed to declaring the value in creation which will not animate at all.
personally I think a better programming practice would be to provide a 'post create' callback we can hack into in order to get a reference to the chart and call its load function. the 'onrender' does not seem to get a reference to the c3 final chart.
this is a matter of encapsulation which I think should be addressed.
if I missed something you are welcomed to comment. what I am looking for is:
var chart = c3.generate({
onChartCreate: function(theChart){//this is the function I need
theChart.load({
columns: [['data', 10]]
});
},
data: {
columns: [
['data']
],
type: 'gauge',
},
color: {
pattern: ['#FF0000', '#F97600', '#F6C600', '#60B044'], // the three color levels for the percentage values.
threshold: {
values: [30, 60, 90, 100]
}
},
size: {
height: 180
}
});

There is a "oninit" callback already:
http://c3js.org/reference.html#oninit
For further help please use the Google Group at https://groups.google.com/forum/#!forum/c3js.
Thank you!


useId hydration mismatch with dynamic and non-dynamic components nested in dynamic component

Environment

Operating System: Linux
Node Version:     v18.20.3
Nuxt Version:     3.12.3
CLI Version:      3.12.0
Nitro Version:    2.9.7
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      compatibilityDate, devtools, components
Runtime Modules:  -
Build Modules:    -

Reproduction
https://stackblitz.com/edit/nuxt-starter-ijvmk1?file=app.vue
Describe the bug
Consider this setup:

app.vue containing:

dynamic global component containing:

dynamic global component containing:

component with useId

non-dynamic component containing:

component with useId

This results in hydration mismatches due to different order of appended numeric ids:
  - rendered on server: for="n7UMpQwNU5n-1"
  - expected on client: for="n7UMpQwNU5n-0"

Additional context
No response
Logs
No response

Actually, i just realised, that in my project the second form is not a child of any dynamic component, but in a separate non-dynamic component. Nonetheless i still have the same issue. But i was not able to create a reproduction using this structure. This might be yet another bug, or related to the one above.
So, imho there are definitely some fishy things going on with useId in more complex projects using dynamic and variously nested components.

Unfortunately this is a consequence of this upstream bug: https://github.com/vuejs/core/issues/6638. In general hydration within any async component, including layouts, can cause issues until this is resolved.
Let's track in https://github.com/nuxt/nuxt/issues/27378.


fix: max height correct show in configuratoin

What does this PR do and why?
Provide a description of what this PR does. Link to any relevant GitHub issues, Notion tasks or Discord discussions.
Screenshots or screen recordings
If applicable, provide screenshots or screen recordings to demonstrate the changes.
Acceptance checklist

[ ] I have evaluated the Approval Guidelines for this PR.
[ ] (if relevant) I have tested this PR in all supported browsers.
[ ] (if relevant) I have tested this PR in dark mode and light mode (if applicable).

Current dependencies on/for this PR:

main

PR #3663   👈

PR #3664

This comment was auto-generated by Graphite.

@wraeth-eth started a stack merge that includes this pull request via Graphite.


Add descride for GetGroupsPermission flag in Gitlab connector

https://github.com/dexidp/dex/pull/2941

Thanks!


Support better editing of sensors_to_show

Editing how the dashboard should look like (via sensors_to_show) is not fun: It requires looking up sensor IDs and editing JSON in a horizontal text box.
The second problem is editing JSON. I have ideas for both here.
Picking sensors
In #1059, we already did some UI work on selecting sensors within the asset form, so this can be built on that experience. However, we decided to move away from doing this within the main edit form, as it becomes too heavy when sensor selection is involved.
This is not a pleasure to do, but also we also complicated the structure in #1119. We need to load, show, edit and send to the API a list of lists (each sublist is a plot that contains one or more sensors). Each entry should be saved as a dictionary (modern form), but also accept the earlier, simpler forms when loading.
I looked at support for editing JSON based on a schema, but I believe editing of this complex information (certainly with loading sensors) requires us to move to writing our own form.
From the user's perspective, the idea is that there is a modal dialogue on the asset page (when clicking "Edit Graphs"):

Main functionality:

Existing sensors_to_show information will be available and rendered on page load.
JavaScript/HTML will allow to edit plots, add new "plots" and to remove them. For each plot entry, sensors can be added or removed and the (optional) plot title can be edited.
On submit, the sensors_to_show JSON representation is constructed and send to the asset PATCH endpoint.
Toast messages about failure / success are shown (see Toast messages in the account editing).

Here are TODOs, each can be a PR (please link them here when they are being worked on):

[ ] Prepare sensor API for sensor lookup: Add all_accessible parameter and search filter (for the name and account name) like the asset API has. Also add a unit parameter.
[ ] Move sensors_to_show from attributes to its own JSON field on the GenericAsset data model. The data migration needs support for existing attributes to be moved over - that has to be Python code, check some earlier migrations that had custom code. Likewise in the reverse direction. Sensors_to_show should also get its own (temporary) form field in the asset editing, so this PR can stand on its own. This is also what issue #1124 is about.
[ ] Edit modal for editing asset page graphs (see description and sketch above). Put the modal JavaScript in its own file, it will be a lot. For representation of the data and managing the entries: Bootstrap Groups/Tabs might give a good way to support the styling, and we might also look into the JS support (at the bottom of that page) to read the resulting structure when the user clicks "Save".

The small sensor lookup dialogue will be re-used when we edit the flex-context and flex-model in a similar way, so it would be good to place it in a more central place, if that is possible.

More info on syntax of sensors_to_show: https://flexmeasures.readthedocs.io/en/latest/views/asset-data.html#the-asset-page-as-data-dashboard
We want to send the more elaborate one.

sensors_to_show = db.Column(MutableDict.as_mutable(db.JSON), nullable=False, default={})


Modules not found

Hello, I downloaded the repo and I'm trying to implement the code. I copied the catalog file from demo into src of project folder and I'm getting the follow errors when running:
./src/catalog/mycatalog.js
Module not found: Can't resolve './areas/**/planner-element.jsx'
Assuming it's related to the following lines in mycatalog.js:
import * as Areas from "./areas//planner-element.jsx";
import * as Lines from "./lines//planner-element.jsx";
import * as Holes from "./holes//planner-element.jsx";
import * as Items from "./items//planner-element.jsx";
Also, initially, I get an error that the imports are below this:
let catalog = new Catalog();
Which requires me to rearrange the order of the code. The folder was copied exactly, so I'm not sure why the path can't locate the files..

Is it possible the ** path is causing issues? Is there an extension or something needed to enable it's usage? Otherwise I'm not sure what's going on. If it's of any help, here the file structure:

src
app.js
catalog
mycatalog.js
areas
holes
items
lines
utils

I made a stackoverflow relating to this issue:
[(https://stackoverflow.com/questions/61597022/module-not-found-when-importing-jsx-file-off-github-using-create-react-app)]

I had similar issues.  What made the /demo folder finally compile for me was adding the following after line 51 on the /demo/webpack.config.js file
"import-glob",
I then ran npx webpack in that folder and it compiled down for me
I also added npm install babel-loader babel-preset-react at some point troubleshooting, but am not sure if that had an impact or not...
That 'import-glob' package is referenced in the babel config file, but on in webpack.  It's required to make the "**" style imports work it seems.

Did you manage to solve the issue? What did you do?

I had similar issues. What made the /demo folder finally compile for me was adding the following after ~ line 48 on the /demo/webpack.config.js file
"import-glob",
I then ran npx webpack in that folder and it compiled down for me
I also added npm install babel-loader babel-preset-react at some point troubleshooting, but am not sure if that had an impact or not...
That 'import-glob' package is referenced in the babel config file, but on in webpack. It's required to make the "**" style imports work it seems.

Awesome! Thank you


Setup script should fail gracefully if ~/.bash_profile not writeable

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Feature Request

I just installed poetry using the get_poetry.py script, ran using the curl command in the docs.
However, the script failed because my ~/.bash_profile isn't writeable (I have this deployed from version control):
Installing version: 0.12.11
  - Downloading poetry-0.12.11-darwin.tar.gz (7.14MB)
Traceback (most recent call last):
  File "<stdin>", line 859, in <module>
  File "<stdin>", line 855, in main
  File "<stdin>", line 328, in run
  File "<stdin>", line 462, in install
  File "<stdin>", line 610, in update_path
PermissionError: [Errno 13] Permission denied: '/Users/harry/.bash_profile'

This left me in a state where I wasn't sure if the script had completed or not. I made ~/.bash_profile writeable and ran it again, and it said "Latest version already installed". I still didn't trust it all went OK, so be sure I had to uninstall and re-install!
How about we make the install script continue on a failure to update ~/.bash_profile, and just instaed print this as a warning at the end of the script that the user needs to update their $PATH themselves? (I.e. the behaviour when you answer "Modify PATH variable? ([y]/n) n") This is the approach that brew takes, and was also suggested by @sdispater over on https://github.com/sdispater/poetry/issues/342#issuecomment-408470755.
I'd be happy to give the PR a shot myself..

Oddly, get-poetry.py also fails to write to ~/.bash_profile even if I don't ask it to update my $PATH...
$ python3 get-poetry.py
Retrieving Poetry metadata

Before we start, please answer the following questions.
You may simply press the Enter key to leave unchanged.
Modify PATH variable? ([y]/n) n

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

$HOME/.poetry/bin

This path needs to be in your `PATH` environment variable,
but will not be added automatically.

You can uninstall at any time with `poetry self:uninstall`,
or by executing this script with the --uninstall option,
and these changes will be reverted.

Installing version: 0.12.11
  - Downloading poetry-0.12.11-darwin.tar.gz (7.14MB)
Traceback (most recent call last):
  File "get-poetry.py", line 859, in <module>
    sys.exit(main())
  File "get-poetry.py", line 855, in main
    return installer.run()
  File "get-poetry.py", line 328, in run
    self.install(version, upgrade=current_version is not None)
  File "get-poetry.py", line 462, in install
    self.update_path()
  File "get-poetry.py", line 610, in update_path
    with open(profile, "a") as f:
PermissionError: [Errno 13] Permission denied: '/Users/harry/.bash_profile'

The bug is in https://github.com/sdispater/poetry/blob/master/get-poetry.py#L310