Pinvoked libraries somehow get broken in CoreRT

Hello there.
I'm trying to debug a pretty hairy behavior in CoreRT here. This isn't a proper issue because my time is limited and I'm struggling to make sense of what's happening, but I think it's better to use an issue instead of Gitter to log my findings and maybe get help.
I'm working on C# bindings for the PhysX library, and I have a testbed sample kinda thing which renders the debug geometry from the PhysX library, it's working fine on both .NET Framework (4.7.2) and .NET Core 2.1, and the code seems correct. However, I couldn't get that to work properly on CoreRT.
At first I thought it could be something in my pinvokes/wrappers, but after some investigation by debugging the output binary in VS, the stack traces point to the PhysX internals, most specifically some threading code (most of the time at least). At that point I've tried to make a minimal repro using only the PhysX libraries, but I simply couldn't get the problem to manifest. I then started from the other way and stripped stuff from the sample until I get something that works, that's when I noticed that for some reason the contact solving stuff in PhysX is broken, take a look at these lovely gifs :trollface::
Here's the proper expected result on CoreCLR:

Here's what happens on CoreRT:

However, the thing is, even though the error happens in PhysX, when I remove the other libraries (I was trying to make a minimal SSCCE), the thing works, here's a minimal code that does the same thing except the other libraries were removed (it plots the cube height to console): https://github.com/Alan-FGR/CoreRT_SharpPhysX_Debug and it works fine on CoreRT, and here's a branch of SharpPhysX with some kinda minimal code that doesn't work: https://github.com/Alan-FGR/SharpPhysX/tree/corert_debug, note how the code that has to do with PhysX is the same (not 1:1 but the pinvoke calls are), the project in question is the "dotnetCore" the code is all in the DebugRenderer though because I thought it could be the abstract impls :P...
I've also tried to remove PhysX and have just the rendering (bgfx debug text and whatnot), and that works fine, the thing only crashes when I'm using all the pinvoked libs at the same time, what makes me think that somehow they're conflicting with each other, like maybe they're writing on each other memories or something, I don't really know... but that makes debugging very hard, especially since I'm a total n00b at debugging interop.
I'm going to try and make some better repros, but in case some of you want to give it a try I think that's all you need, the native PhysX is an opensource project too (https://github.com/NVIDIAGameWorks/PhysX/tree/4.0) and the native library (LibSharpPhysX) was built against the latest 4.0.

hey guys, so I'm now having consistent and sensible stack traces like this:
LibSharpPhysX.dll!physx::PxDefaultSimulationFilterShader(unsigned int attributes0, physx::PxFilterData filterData0, unsigned int attributes1, physx::PxFilterData filterData1, physx::PxFlags<enum physx::PxPairFlag::Enum,unsigned short> & pairFlags, const void * constantBlock, unsigned int constantBlockSize) Line 234
	at c:\projects\physx\physx\source\physxextensions\src\extdefaultsimulationfiltershader.cpp(234)
dotnetCore.exe!Flt2LngOvf()
dotnetCore.exe!Flt2LngOvf()
PhysX_64.dll!runFilterShapeSim(physx::PxFilterInfo & filterInfo, const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & e0, const physx::Sc::ShapeSim & e1, const unsigned int fa0, const unsigned int fa1) Line 306
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(306)
PhysX_64.dll!filterRbCollisionPairSecondStage(const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & s0, const physx::Sc::ShapeSim & s1, bool kine0, bool kine1, const unsigned int fa0, const unsigned int fa1) Line 357
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(357)
PhysX_64.dll!filterRbCollisionPair(const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & s0, const physx::Sc::ShapeSim & s1) Line 550
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(550)
PhysX_64.dll!physx::Sc::NPhaseCore::runOverlapFilters(unsigned int nbToProcess, const physx::Bp::AABBOverlap * pairs, physx::PxFilterInfo * filterInfo, unsigned int & nbToKeep_, unsigned int & nbToSuppress_, unsigned int & nbToCallback_, unsigned int * keepMap, unsigned int * callbackMap) Line 574
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(574)
PhysX_64.dll!OverlapFilterTask::runInternal() Line 5745
	at c:\projects\physx\physx\source\simulationcontroller\src\scscene.cpp(5745)
PhysX_64.dll!physx::Cm::Task::run() Line 67
	at c:\projects\physx\physx\source\common\src\cmtask.h(67)
LibSharpPhysX.dll!physx::Ext::DefaultCpuDispatcher::runTask(physx::PxBaseTask & task) Line 97
	at c:\projects\physx\physx\source\physxextensions\src\extdefaultcpudispatcher.h(97)
LibSharpPhysX.dll!physx::Ext::CpuWorkerThread::execute() Line 97
	at c:\projects\physx\physx\source\physxextensions\src\extcpuworkerthread.cpp(97)
PhysXFoundation_64.dll!physx::shdfnd::`anonymous namespace'::PxThreadStart(void * arg) Line 101
	at c:\projects\physx\physx\source\foundation\src\windows\pswindowsthread.cpp(101)

it's something to do with filter data, it's a memory access violation error for one of the filters. so don't bother for now, I think I got this... i don't have much time to investigate but it's starting to look like it could be some error on my side that CoreRT won't forgive, I don't know...

nah, it's not consistent, sometimes filter errors are valid and i get this:
[Inline Frame] dotnetCore.exe!Thread::InlineReversePInvokeReturn(ReversePInvokeFrame *) Line 1232
	at e:\a\_work\600\s\corert_2607052\src\native\runtime\thread.cpp(1232)
dotnetCore.exe!RhpReversePInvokeReturn2(ReversePInvokeFrame * pFrame) Line 1397
	at e:\a\_work\600\s\corert_2607052\src\native\runtime\thread.cpp(1397)
dotnetCore.exe!Flt2LngOvf()
PhysX_64.dll!runFilterShapeSim(physx::PxFilterInfo & filterInfo, const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & e0, const physx::Sc::ShapeSim & e1, const unsigned int fa0, const unsigned int fa1) Line 306
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(306)
PhysX_64.dll!filterRbCollisionPairSecondStage(const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & s0, const physx::Sc::ShapeSim & s1, bool kine0, bool kine1, const unsigned int fa0, const unsigned int fa1) Line 357
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(357)
PhysX_64.dll!filterRbCollisionPair(const physx::Sc::FilteringContext & context, const physx::Sc::ShapeSim & s0, const physx::Sc::ShapeSim & s1) Line 550
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(550)
PhysX_64.dll!physx::Sc::NPhaseCore::runOverlapFilters(unsigned int nbToProcess, const physx::Bp::AABBOverlap * pairs, physx::PxFilterInfo * filterInfo, unsigned int & nbToKeep_, unsigned int & nbToSuppress_, unsigned int & nbToCallback_, unsigned int * keepMap, unsigned int * callbackMap) Line 574
	at c:\projects\physx\physx\source\simulationcontroller\src\scnphasecore.cpp(574)
PhysX_64.dll!OverlapFilterTask::runInternal() Line 5745
	at c:\projects\physx\physx\source\simulationcontroller\src\scscene.cpp(5745)
PhysX_64.dll!physx::Cm::Task::run() Line 67
	at c:\projects\physx\physx\source\common\src\cmtask.h(67)
LibSharpPhysX.dll!physx::Ext::DefaultCpuDispatcher::runTask(physx::PxBaseTask & task) Line 97
	at c:\projects\physx\physx\source\physxextensions\src\extdefaultcpudispatcher.h(97)
LibSharpPhysX.dll!physx::Ext::CpuWorkerThread::execute() Line 97
	at c:\projects\physx\physx\source\physxextensions\src\extcpuworkerthread.cpp(97)
PhysXFoundation_64.dll!physx::shdfnd::`anonymous namespace'::PxThreadStart(void * arg) Line 101
	at c:\projects\physx\physx\source\foundation\src\windows\pswindowsthread.cpp(101)

when they're valid they're 0 though, what makes me think that could be happening by chance (since zeroing mem is relatively common and zero objects are normally valid) so that still doesn't rule out that my code could be wrong, but coreclr/netfx they zero that so it's valid

OK guys, here's some more stuff I figured: the filter shader thing is a function pointer, so to make sure it wasn't my stuff that was wrong I made my own impl like this:
PxFilterFlags SomeFilterShader(
PxFilterObjectAttributes attributes0, PxFilterData filterData0,
PxFilterObjectAttributes attributes1, PxFilterData filterData1,
PxPairFlags& pairFlags, const void* constantBlock, PxU32 constantBlockSize)
{
	printf("filter called\n");
	pairFlags = PxPairFlag::eCONTACT_DEFAULT;
	return PxFilterFlag::eDEFAULT;
}

and that works, when collisions happen that text is printed (so we can rule out wrong pointers and such), but on CoreRT it's printed once and then the errors with the stacks as above happen.

OK guys, I figured what's the problem, the problem is declaring the function pointer as a delegate (with UnmanagedFunctionPointer attribute). If I use an IntPtr then it works fine on CoreRT! :smiley:.
I'm not sure whether that's a bug, maybe lacking impl for that attribute, or if that's just me not doing stuff correctly (as usual :trollface:).

I think the problem is that I assumed the UnmanagedFunctionPointer would actually turn a C# delegate into an actual function pointer what's not the case. The inconsistency is still there, but the code wasn't sound to begin with.


check-format-commit.sh: make awk patterns more robust; further improvements

for use with MacOS and certain flavours of Linux.
As briefly discussed at https://github.com/openssl/openssl/pull/24806#issuecomment-2403300718,
the nice new tool had trouble on certain systems where awk is unhappy with unescaped '+' symbols appearing in patterns.
On MacOS, I got
awk: illegal primary in regular expression +{3} at {3}
 source line number 3
 context is
	    >>>  /+{3}/ <<<
This commit has no files that require checking

This can be fixed by escaping the two occurrences of '+' in patterns within the script with a backslash: '\+' .
Interestingly, even on my Ubuntu and Debian Linux systems, I got
awk: line 3: regular expression compile failed (missing operand)
+{3}
awk: line 8: regular expression compile failed (missing operand)
+

There, for whatever reason, after escaping the  '+' in patterns in the same way,
I just got
This commit has no files that require checking

for whatever (commit range) parameters I tried, including ones that do have formatting nits that normally are shown.
So the tool did not exit with error but did not call check-format.pl at all and thus not show any findings.
Both issues are fixed by the first commit this PR,
which also prevents failure in case the source files happen to contain +++ or @@.

I meanwhile had a full look at the new script and suggest further minor improvements,
including simplifications, example invocations, and fixes in comment texts.
Moreover, allow for empty commit range (no parameter),
which is taken as the list of currently unstaged changes (like git diff),
which can be very handy during development (before committing).
Hope you don't mind that I piggy-back this on the existing PR?

I'm fine with the piggy-back
CI errors look unrelated

Ping for 2nd review/approval

This pull request is ready to merge

Thanks for the approvals - merged to all active branches.


read.myacc.csv: add resample and timestamp character format freedom

Add functionalities:

Resample the data if the sample rate is irregular
To handle alternative timestamp formats other than POSIX and UNIXsec. In that way the user can specify with argument rmc.format.time what the format is.

This has now been done and merged into master branch.


Review ActiveX related patch

https://github.com/transistor1/dockpanelsuite/commit/adcbf05a1483e926dc97aeb9f72f1b989e697ad9
Check if it helps.

Cherry picked as e62de372ddba420e1e82581853fe9584d3aad438

Anyone know more about the reasoning behind this patch?

You might ping @transistor1 for more details.

@billybednar, Please see https://github.com/transistor1/dockpanelsuite/issues/1
I'm using DockPanelSuite in Access SQL Editor, which is an ActiveX object.  PreFilterMessage doesn't get called, because it's not a normal window.
I'm curious to know - how does this relate to the bug you're tracking down?

@transistor1 there is an OpenGL related issue #304. I thought it might be related to this, but it seems to be irrelevant. If you also master OpenGL, you can shed some light too. Appreciate it.

@transistor1 See the two issues I just opened

I fixed an intermittent crash in my own fork a while back:. https://github.com/transistor1/dockpanelsuite/issues/4
Your case #456 sounds similar but I haven't reviewed it.
I forked DockPanelSuite because I have a special use case & don't expect that many users would need to host DPS in an ActiveX control.  I didn't expect to see my changes make it to the master.  @lextm if #401 needs to be backed out, I'm fine with that.

Thanks, @transistor1 . I have put the patch now under patch controller and disable it by default. Then most users won't be affected by its side effect.


Fail to log the uncommitted code of $CWD when run git diff

When run the example: workflow_by_code.ipynb, I got the info
[15460:MainThread](2022-10-09 21:54:33,045) INFO - qlib.workflow - [recorder.py:372] - Fail to log the uncommitted code of $CWD when run git diff
I am wondering how to avoid the fail info.
PS: the code was run on windows 10, anaconda
Thanks for help.
❓ Questions and Help
We sincerely suggest you to carefully read the documentation of our library as well as the official paper. After that, if you still feel puzzled, please describe the question clearly under this issue.

This issue has been fixed in this PR, thank you for your issue.


Edited the intern.nasa.gov link to incl. https://

(This is a duplicate of the /37 pull request but I have messed up on my end and can't figure out how to re-open that one)
Included "https://" with intern.nasa.gov link
In some instances (including GitHub), middle-mouse-clicking the link will redirect you to that page + intern.nasa.gov
For instance, https://github.com/codedex-io/blog/blob/main/blogs/a-day-in-the-life-swe-intern-nasa-brandon-lam.mdx redirects to https://github.com/codedex-io/blog/blob/main/blogs/intern.nasa.gov

Good catch!


Add HideModList mod

A few things to note:

This doesn't completely remove the modlist (Modding API text is still seen) (see example)
An IL hook was used to make it less replicable.

So this really shouldn't cause issues for speedrun verifications

This makes verification suck for modded boards and it's already difficult enough to see the text for sr verification honestly. Is there any actual reason you'd want to hide the mods you're using besides like cheating?

I've seen 2 uses cases until now

you dont want people to see what mods you have for like giving a suprise source. Could also be used when streaming ig
ModList takes up alot of space which can become annoying when dealing with menu changer (because text overlap) source

I could go for it if there was very clear indication that the mods were being hidden

so if it would be either

or with other palceholder text, though no whitespace,
or something like
Modding API: whatever
+ ## Mods

it should be good?

Could replace the placeholder with something like "QoL: v...." still, no?

Could replace the placeholder with something like "QoL: v...." still, no?

Isn't that the same as just showing full modlinks
Maybe I could just do what the mini save states (which is legal for IL runs) does which puts text on main main menu that states it is being used

I mean you could pretend to have a version of a mod by just putting in its identifier as the placeholder


[AUTOPATCHER-CORE] Upgrade mariadb to 10.11.10 none

[AUTOPATCHER-CORE] Upgrade mariadb to 10.11.10 none
Upgrade pipeline run -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=669657&view=results
buddy build -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=669659&view=results

fix build break build #669702


run with default paras failed, where should I put the models ? thanks

I download the opt-1.3b and 350m models ,but don't know where to put them .
Here is the cmd and errors:
cmd: python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu
error logs:
[INFO] [comm.py:586:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
'HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /facebook/opt-1.3b/resolve/main/tokenizer_config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6c3b2c1760>, 'Connection to huggingface.co timed out. (connect timeout=10)'))' thrown while requesting HEAD https://huggingface.co/facebook/opt-1.3b/resolve/main/tokenizer_config.json
'HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /facebook/opt-1.3b/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f6c3b2baa90>, 'Connection to huggingface.co timed out. (connect timeout=10)'))' thrown while requesting HEAD https://huggingface.co/facebook/opt-1.3b/resolve/main/config.json
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 409, in cached_file
resolved_file = hf_hub_download(
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1259, in hf_hub_download
raise LocalEntryNotFoundError(huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.

I know how to fix it :
put downloaded models to the path same to train.py


while sign-up, the link for 'FightPandemics' on 'Check your inbox' page is not working

Issue: while sign-up, the link for 'FightPandemics' on 'Check your inbox' page is not working . Refer picture below:
steps:1. Go to  https://rcv11.fightpandemics.xyz/ , Navigate to Join Now
2.Enter email,password and confirm password and click 'Join Now'
3.'Check your Inbox' page is displayed
4. click on FP link on top left
Issue: FP link is not working

Thanks @sheeba1 .
This is behaving as such for similar reasons explained in #1153. It wouldn't be easy to change (because of multiple functions depended on the same redirect logic) without refactoring to auth (which I'm working on slowly).
I'm thinking the best solution now and maybe even later is for these auth pages:

create-profile
auth/signup
auth/login
auth/check-email
auth/forgot-password

We should just make the logo non-clickable so there's confusion. And for some of the pages we don't want them to navigate away (although they still can be clicking back button).
Thoughts @indirabhowmick @vikjoshi @manualzuru ?
FYI these pages are the ones with navless headers like:

Comment from Manu, "@Sheeba Jose in most of the transactional features we don't want them to click anywhere else. Please check Amazon checkout page"
So we can make logo unclickable?

@sheeba1  Yeah that was my thought. It would be fairly easy.
Right now it's wrapped in a <Link> like such:
          <Link to="/">
            <Logo src={logo} alt="Fight Pandemics logo" />
          </Link>

Where we don't want it clickable we just omit wrapping it in a <Link>.
If this is a go then either we can reword this issue or close this and create a new one.

@joshmorel yeah you are correct. In transactional pages (Login/Signup) we want them to not be able to click on the logo so they don't leave and complete.


Request for comments/thoughts/advice on implementing Bolt Routing.

👋 Hey folks! I recently migrated my Neo4j architecture to a Causal Cluster, and I've spent the past few days working on enhancing neo4j-core to support the Bolt Routing protocol---which is just Bolt with a little service discovery, er, bolted on---so that my application will be compatible. The highly 🚧 work-in-progress 🚧 Core branch lives here, with a corresponding wrapper branch here. It's something that I'll be happy to yield back to the community when it's ready, but I'm stuck on part of the load balancing logic and I was hoping folks with deeper insight into the overall architecture of the gem might have some thoughts or insights to share.
Currently, the adapter checks a connection out from a pool at the start of every transaction; the transaction hangs on to the connection for its duration, since each transaction has to start and end at the same database instance, then releases it back to the pool. The service discovery logic to disambiguate read-type connections from write-type connections is in place, and works fine. The rub is that when a transaction starts, we issue the Cypher BEGIN statement before we know what the meat of the transaction will look like---we don't know, at that point, whether the transaction will contain exclusively read statements or whether it will include write statements (which must pass through the cluster leader).
The JavaScript adapter library drives this problem down to the consumer---it assumes all queries are write queries and should pass through the leader unless the application specifically requests a read-only transaction. I don't think that'll work here, though.
What I would love to be able to do is have the complete battery of queries available before firing them off, so that I could check for whether it was safe to send them to a read instance or whether they needed to go through a write instance. But I can't see a way to make this work in the current architecture of the gem.
If any of the maintainers, or anyone with deeper insight into the gem's inner workings, has any thoughts, I'd be only too glad to hear them. Thanks in advance!

I think I may have gotten a handle on it after taking another look with fresh eyes: https://github.com/ignota/neo4j-core/commit/818f8db149167d3a13878fbe4b55d6cd350f7778. The adapter now overrides queries, transaction, and new_or_current_transaction to assign an access mode to the transaction when queries are flowing through it, falling back on :write mode if no queries are given. Initial testing in my cluster suggests that this works acceptably well, but I'd be happy to hear suggestions for better options.

@phyllisstein This is a great work you have been doing here. Our team took over the maintenance of neo4j.rb. As a first step, we would like to produce a neo4j ruby driver that is totally analogous to the other 4 official language drivers. As a proof of concept, I have created a jruby driver based on the java driver.
I would love if you could review the manual examples at https://github.com/neo4jrb/neo4j-driver/blob/master/spec/support/dev_manual_examples.rb and gives us some feedback regarding the proposed API.
As a 2nd step we were planning to leverage the soon to be released seabolt (C driver) to implement an MRI gem to exactly the same API so both the java and c versions can be switched without any changes to the client.
However, seeing how much work you have done on the pure ruby driver and how much understanding you seem to have I wanted to ask if you would be interested to package that driver behind the proposed API in https://github.com/neo4jrb/neo4j-driver/.
Once we have a general purpose ruby driver with an official API we would take on adapting neo4j and neo4j-core to use that driver, but I would like to implement the driver completely independently from those and without a significant influence by those gems to guarantee a good separation.
Let me know what you think?

@phyllisstein @klobuczek any updates on bolt+routing in the Ruby gem?

@klobuczek , would the seabolt implementation support bolt+routing? Or would that still have to be built on top?

I'm sad to say I've moved away from Ruby and Rails for my work. Sad because I can't be all that helpful; and sad because even just skimming the Neo4j::Driver code made it clear how far it's come in the past year.
It looks as though the Seabolt-based driver does support bolt+routing: see dev_manual_examples.rb:29-49. What kind of work would be necessary to use Neo4j::Driver with Neo4j.rb, I can't really say. You might want to start in Neo4j::SessionManager.adaptor_class_by_type. Depending on the extent of API drift, it could be a quick and easy hack or a big ole mess.

@phyllisstein, thanks for that update!
In your code, you reference a bolt_routing adapter however, I cannot seem to find that code anywhere. Can you perhaps help me out there?

Those lines pull in changes I made to neo4j-core. I don't think the adapter I wrote is quite production-ready, however: there's no test suite, a lot of the idioms are borrowed from the Neo4j JavaScript driver, and the lack of support for bookmarks is pretty egregious.
What I was suggesting is that you could try dropping Neo4j::Driver in there and seeing how far you got. But after looking at its code, I can see the answer would be "not very far at all." As an alternative, you might use neo4j-core's implementation of the Bolt adapter as a starting-point for creating a very thin wrapper around Neo4j::Driver. You'd want to construct the same basic API surface, so that Neo4j.rb could consume it, but call out to Neo4j::Driver to actually interact with the database, so that you could avail yourself of its improvements.


What is the max macOS Version support?

Compatibility Catalina or Big Sur?

I am not sure I got doxed 8 months ago and no longer have the Dell Optiplex 760 do to getting removed from the internet I have no idea if it works with anything other then big sur.

you can try tho; I'd love to see if it works! If it does work let me know and I'll mark the issue solved.

internet not working in sierra and high sierra and el capitan but still can't find out if it is working on big sur

I no longer own/have this computer this project is E0L


Detection Method

Description - Add a detection method property similar to SCCM.
Software packagers do all kinds of dumb things that require custom detection.

Not placing a key in the registry at all
Including versions in the DisplayName
Not including versions in the DisplayVersion
Having a DisplayName with special characters that mess with regex (Notepad++)
Self-updating by overwriting files in Program Files, but not updating the registry with the new DisplayVersion

Proposed technical implementation details
Detection Script
Include an optional "DetectionScript" field in your package manifest that accepts a PowerShell script that returns a valid semantic version if it is installed, and Null or Empty string if it is not. The script can look at file versions and other places indicative of the actual installed version of the software.
Add regex and version capture support to the name field
For software that places its version in DisplayName and not DisplayVersion, allow a regex pattern to capture the version. I've seen a lot of LOB apps do this. For a DisplayName of "My Ancient LoB Soft Pro 18.7.123" the regex pattern could be "My Ancient LoB Soft Pro (\d+.\d+.\d+)"

For --update to work you'd need the version detection method to work without actually installing the package, because the user wants to know if there is an update available before installing it. Unless one wants to go the crazy route of installing each package in a sandbox on the server-side just in order to extract its version by running a detection script.

@denelon - Is this resolved with the ARP entry matching?

@Trenly not all the use cases above are resolved. Some of them are most likely unactionable at this time. We've been discussing some additional options, and these types of challenges are also what led to:

#2252

@Trenly not all the use cases above are resolved. Some of them are most likely unactionable at this time. We've been discussing some additional options, and these types of challenges are also what led to:

Improve Uninstall Confidence #2252

Ah, makes sense; Thank you


feat(messages): add message attachments

What this PR does 📖

add message attachments upload/download
add message keys

Which issue(s) this PR fixes 🔨

Resolve #

Special notes for reviewers 🗒
Additional comments 🎤

Found a few things when testing I wanted to mention -

It seems like messages with attachments will only send if you have text typed with it as well. Trying to send a file without adding any text will cause it to not send

https://user-images.githubusercontent.com/93608357/204061131-56f6d5d4-2d6f-4c43-a2b7-d653e7cf5f04.mov

Not sure why but it seems like most of the files will be blocked from sending

And one last thing that is more of a question, are we adding file preview in a future pr or should it be added with this pr?

We can have a different PR for the preview / design stuff, for now I think we should just try to get the ability to send all kinds of files on the system then merge this if everything else works as expected.

We can have a different PR for the preview / design stuff, for now I think we should just try to get the ability to send all kinds of files on the system then merge this if everything else works as expected.

file type filter removed

Found a few things when testing I wanted to mention -

It seems like messages with attachments will only send if you have text typed with it as well. Trying to send a file without adding any text will cause it to not send

sending.image.mov
2. Not sure why but it seems like most of the files will be blocked from sending

And one last thing that is more of a question, are we adding file preview in a future pr or should it be added with this pr?

1 and 2 are fixed


Snakemake draft

I was playing a bit with https://github.com/c-scale-community/c-scale-tutorial-snakemake
Then decided to create this draft for a snakemake workflow for HiSea.
Regarding https://github.com/c-scale-community/use-case-hisea/issues/40, Snakemake can also help with job submission to SLURM:
https://snakemake.readthedocs.io/en/stable/executing/cluster.html

Sorry, forgot to add instructions for testing:
# install snakemake
mkdir working-dir
cd working-dir
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p conda-install
source conda-install/etc/profile.d/conda.sh
conda install mamba -c conda-forge --yes
mamba create -c bioconda -c conda-forge -n snakemake snakemake-minimal --yes
conda activate snakemake

# run workflow
git clone https://github.com/c-scale-community/use-case-hisea.git
cd use-case-hisea/
git checkout --track origin/snakemake
snakemake -c1 pre_processing_boundary --use-conda --conda-frontend mamba

Fixed now the snakemake rule for pre_processing_boundary
Note that I had to manually execute:
mkdir -p data/output
mkdir -p data/output/DFMWAQ_input/

@backeb @lorincmeszaros I fixed the issue with pre_processing_boundary.
I think the best next step is for us three to have a dedicated meeting to move this forward with new steps for the workflow (if you still want to go down the snakemake route)


Does this provider support RichText?

Does the provider support richText entries?
I am trying to add this entry but getting validation error. Am I missing something or is RichText not supported?
Thank you so much for assistance
Error message:
The type of \\\\"value\\\\" is incorrect, expected type: RichText\\",\\"path\\":[\\"fields\\",\\"content\\",\\"en\\"]}]}
ContentType Field:
  field {
    id       = "content"
    name     = "Content"
    type     = "RichText"
  }

Entry Field:
field {
    id = "content"
    locale = "en"
    content = jsonencode({
      data= [],
      content= [
        {
          nodeType= "paragraph",
          content= [
            {
              nodeType= "text",
              marks= [],
              value= "This is paragraph.",
              data= [],
            },
          ],
          data= [],
        }
      ],
      nodeType= "document"
    })
  }

@demeyerthom Would appreciate your input on this

@demeyerthom Sorry for the ping, Do you have any insights on this?
If provider doesn't support Richtext, I can add a PR for it

Hi @rahulpatel596!
I ran a quick test with the same entries, and I am getting the same errors. This is because the provider value is an encoded json string (which is the type defined in the provider), but the API actually expects an unencoded json object.
I will release a fix a bit to attempt unmarshalling the values before sending them to the API.
Sidenote: your json will still fail; the data fields should be objects, not arrays. This is enforced by the api from contentful

Appreciate your help on this. Thank you so much @demeyerthom

@demeyerthom It seems like this version hasn't been deployed in terraform cloud yet.

@demeyerthom Would it be possible to look into above. It looks like github-actions release step is failing.


README, suggest append_after to ensure proper order of cleanup

It's best to have DatabaseCleaner.clean run AFTER capybara cleanup, in case capybara cleanup ends up triggering anything that could be interrupted by the database cleaner.
require 'capybara/rspec' installs an after hook for Capybara cleanup.
rspec after hooks are run in reverse order of definition, after hooks defined last are run first. So with require 'capybara/rspec' at top of file, and then later manually defining an after hook for DC cleanup... DC cleanup fires before capybara's cleanup.
Using RSpec's append_after instead of after fixes this. With so many timing issues that can go wrong with capybara, might as well eliminate one more potential one.

done

@jrochkind wonderful, thanks! :+1:


The CMake build process is failed if BUILD_SHARED_LIBS is set to ON (on Mac)

What version of gRPC and what language are you using?
C++ build version v1.34.0
What operating system (Linux, Windows,...) and version?
macos 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6<IP_ADDRESS>~1/RELEASE_X86_64 x86_64
What runtime / compiler are you using (e.g. python version or version of gcc)
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
What did you do?
cmake ../grpc/ -DBUILD_SHARED_LIBS=ON -DABSL_ENABLE_INSTALL=OFF
make -j9

What did you expect to see?
The build process is succeeded.
What did you see instead?
The build process is failed with many errors. Below is the sample output
[ 31%] Building CXX object third_party/re2/CMakeFiles/mimics_pcre_test.dir/re2/testing/mimics_pcre_test.cc.o
      bssl::internal::DeleterImpl<crypto_buffer_st, void>::Free(crypto_buffer_st*) in handshake.cc.o
      bssl::internal::DeleterImpl<crypto_buffer_st, void>::Free(crypto_buffer_st*) in handshake_client.cc.o
      bssl::internal::DeleterImpl<crypto_buffer_st, void>::Free(crypto_buffer_st*) in handshake_server.cc.o
      bssl::internal::DeleterImpl<crypto_buffer_st, void>::Free(crypto_buffer_st*) in ssl_asn1.cc.o
      bssl::ssl_cert_dup(bssl::CERT*) in ssl_cert.cc.o
      bssl::ssl_set_cert(bssl::CERT*, std::__1::unique_ptr<crypto_buffer_st, bssl::internal::Deleter<crypto_buffer_st> >) in ssl_cert.cc.o
      bssl::internal::DeleterImpl<crypto_buffer_st, void>::Free(crypto_buffer_st*) in ssl_cert.cc.o
      ...
  "_CRYPTO_BUFFER_init_CBS", referenced from:
      bssl::do_send_client_key_exchange(bssl::SSL_HANDSHAKE*) in handshake_client.cc.o
      bssl::do_read_client_certificate_verify(bssl::SSL_HANDSHAKE*) in handshake_server.cc.o
      bssl::check_leaf_cert_and_privkey(crypto_buffer_st*, evp_pkey_st*) in ssl_cert.cc.o
      bssl::ssl_cert_check_private_key(bssl::CERT const*, evp_pkey_st const*) in ssl_cert.cc.o
      bssl::ssl_on_certificate_selected(bssl::SSL_HANDSHAKE*) in ssl_cert.cc.o
      bssl::DC::Parse(crypto_buffer_st*, unsigned char*) in ssl_cert.cc.o
  "_CRYPTO_BUFFER_len", referenced from:
      bssl::ssl_verify_peer_cert(bssl::SSL_HANDSHAKE*) in handshake.cc.o
      bssl::do_send_server_certificate(bssl::SSL_HANDSHAKE*) in handshake_server.cc.o
      bssl::SSL_SESSION_to_bytes_full(ssl_session_st const*, cbb_st*, int) in ssl_asn1.cc.o
      bssl::ssl_add_cert_chain(bssl::SSL_HANDSHAKE*, cbb_st*) in ssl_cert.cc.o
      bssl::ssl_add_client_CA_list(bssl::SSL_HANDSHAKE*, cbb_st*) in ssl_cert.cc.o
      _SSL_get0_signed_cert_timestamp_list in ssl_lib.cc.o
      _SSL_get0_ocsp_response in ssl_lib.cc.o
      ...
  "_CRYPTO_BUFFER_new", referenced from:
      _SSL_CTX_use_certificate_ASN1 in ssl_cert.cc.o
      _SSL_use_certificate_ASN1 in ssl_cert.cc.o
      set_signed_cert_timestamp_list(bssl::CERT*, unsigned char const*, unsigned long) in ssl_cert.cc.o
      _SSL_CTX_set_ocsp_response in ssl_cert.cc.o
      _SSL_set_ocsp_response in ssl_cert.cc.o
      set_client_CA_list(std::__1::unique_ptr<stack_st_CRYPTO_BUFFER, bssl::internal::Deleter<stack_st_CRYPTO_BUFFER> >*, stack_st_X509_NAME const*, crypto_buffer_pool_st*) in ssl_x509.cc.o
      add_client_CA(std::__1::unique_ptr<stack_st_CRYPTO_BUFFER, bssl::internal::Deleter<stack_st_CRYPTO_BUFFER> >*, x509_st*, crypto_buffer_pool_st*) in ssl_x509.cc.o
      ...
     (maybe you meant: _sk_CRYPTO_BUFFER_new_null)
  "_CRYPTO_BUFFER_new_from_CBS", referenced from:
      bssl::do_read_certificate_status(bssl::SSL_HANDSHAKE*) in handshake_client.cc.o
      bssl::SSL_SESSION_parse(cbs_st*, bssl::SSL_X509_METHOD const*, crypto_buffer_pool_st*) in ssl_asn1.cc.o
      bssl::SSL_SESSION_parse_crypto_buffer(cbs_st*, std::__1::unique_ptr<crypto_buffer_st, bssl::internal::Deleter<crypto_buffer_st> >*, unsigned int, crypto_buffer_pool_st*) in ssl_asn1.cc.o
      bssl::ssl_parse_cert_chain(unsigned char*, std::__1::unique_ptr<stack_st_CRYPTO_BUFFER, bssl::internal::Deleter<stack_st_CRYPTO_BUFFER> >*, std::__1::unique_ptr<evp_pkey_st, bssl::internal::Deleter<evp_pkey_st> >*, unsigned char*, cbs_st*, crypto_buffer_pool_st*) in ssl_cert.cc.o
      bssl::ssl_parse_client_CA_list(ssl_st*, unsigned char*, cbs_st*) in ssl_cert.cc.o
      bssl::ext_sct_parse_serverhello(bssl::SSL_HANDSHAKE*, unsigned char*, cbs_st*) in t1_lib.cc.o
      bssl::tls13_process_certificate(bssl::SSL_HANDSHAKE*, bssl::SSLMessage const&, bool) in tls13_both.cc.o
      ...

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
See TROUBLESHOOTING.md for how to diagnose problems better.
Anything else we should know about your project / environment?
None

I'm able to reproduce this (on Mac) by running:
cd cmake/build
cmake -DBUILD_SHARED_LIBS:BOOL=ON ../..
make grpc -j2

Looks like the issue is that building of boringssl (libcrypto.dylib specifically) with BUILD_SHARED_LIBS=ON is broken (rather than the build of grpc itself).

I'm able to reproduce this (on Mac) by running:
cd cmake/build
cmake -DBUILD_SHARED_LIBS:BOOL=ON ../..
make grpc -j2

Looks like the issue is that building of boringssl (libcrypto.dylib specifically) with BUILD_SHARED_LIBS=ON is broken (rather than the build of grpc itself).

sounds like something that should be raised with boringssl then. Closing it here.


Large allocations when value size << table size, possible memory leak

We're hitting out of memory errors with high memory usage for only ~200 keys: 26.64 GiB! The keys are fairly small, no more than 8k max, so this was pretty surprising. Further, the total sum of all allocated slabs returned by Stats() only shows a small fraction of the actually in-use memory.
For one sample: 40 MiB slab total, 18 GiB container total with 134 keys on<EMAIL_ADDRESS>Of course, some of this could be our app – so I dug in more and captured profiles. 😄
Data collection
Here's top heap:
Type: inuse_space
Time: Jun 29, 2021 at 12:16pm (CDT)
Active filters:
   focus=github\.com/buraksezer/olric/internal/storage\.newTable
Showing nodes accounting for 7199.49MB, 99.87% of 7208.76MB total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context
----------------------------------------------------------+-------------
                                         7156.94MB 99.59% |   github.com/buraksezer/olric/internal/storage.(*Storage).Put /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:98 (inline)
                                           29.56MB  0.41% |   github.com/buraksezer/olric/internal/storage.New /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:52 (inline)
 7186.49MB 99.69% 99.69%  7186.49MB 99.69%                | github.com/buraksezer/olric/internal/storage.newTable /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/table.go:63
----------------------------------------------------------+-------------
                                            8.50MB   100% |   github.com/buraksezer/olric/internal/storage.(*Storage).Put /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:98 (inline)
    8.50MB  0.12% 99.81%     8.50MB  0.12%                | github.com/buraksezer/olric/internal/storage.newTable /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/table.go:51
----------------------------------------------------------+-------------
                                            4.50MB   100% |   github.com/buraksezer/olric/internal/storage.(*Storage).Put /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:98 (inline)
    4.50MB 0.062% 99.87%     4.50MB 0.062%                | github.com/buraksezer/olric/internal/storage.newTable /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/table.go:52
----------------------------------------------------------+-------------

Which points to these lines: https://github.com/buraksezer/olric/blob/7e13bd0c669b83f2b1ef1716ec8564b2e0098977/internal/storage/storage.go#L98-L99
I also ran a CPU profile for good measure, and found storage.Inuse() was taking up a large percentage of time compared to the typical HTTP traffic this app expected:
Type: cpu
Time: Jun 29, 2021 at 12:21pm (CDT)
Duration: 30.11s, Total samples = 30.52s (101.37%)
Showing nodes accounting for 30.52s, 100% of 30.52s total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context
----------------------------------------------------------+-------------
                                            28.26s   100% |   github.com/buraksezer/olric/internal/storage.(*Storage).Put /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:98
    28.23s 92.50% 92.50%     28.26s 92.60%                | github.com/buraksezer/olric/internal/storage.(*Storage).Inuse /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:333
                                             0.03s  0.11% |   runtime.asyncPreempt /usr/local/go/src/runtime/preempt_amd64.s:8
----------------------------------------------------------+-------------
                                             1.40s   100% |   github.com/buraksezer/olric/internal/storage.(*Storage).Put /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:98
     1.40s  4.59% 97.08%      1.40s  4.59%                | github.com/buraksezer/olric/internal/storage.(*Storage).Inuse /go/pkg/mod/github.com/buraksezer/olric@v0.3.7/internal/storage/storage.go:332
----------------------------------------------------------+-------------
                                             0.07s   100% |   runtime.netpoll /usr/local/go/src/runtime/netpoll_epoll.go:126
     0.07s  0.23% 97.31%      0.07s  0.23%                | runtime.epollwait /usr/local/go/src/runtime/sys_linux_amd64.s:725
----------------------------------------------------------+-------------

Steps to reproduce
I made a test and benchmark that cause this same issue (the heap profile shows the same lines affected).
Each of them have several tuning knobs (constants) you can use to experiment, if you like. I found a high table size and small value byte size triggered the issue most easily.

Click to expand for the test:
In dmap_put_test.go:
const mebibyte = 1 << 20

func TestDMap_PutMany(t *testing.T) {
	const (
		keyCount  = 1000 // Number of keys to set in 1 round.
		valueSize = 10   // Value size in bytes.
		// Cumulative value memory = keyCount * valueSize
		tableSize = 10 * mebibyte

		sampleInterval = 5 * time.Second
		statusInterval = 10 * time.Second
	)

	cfg := testSingleReplicaConfig()
	cfg.TableSize = tableSize
	db, err := newDB(cfg)
	if err != nil {
		t.Fatalf("Expected nil. Got: %v", err)
	}
	dm, err := db.NewDMap("mymap")
	if err != nil {
		t.Fatalf("Expected nil. Got: %v", err)
	}

	before := readMem()
	t.Logf("Before Memory: %.2f MiB", toMiB(before.Alloc))
	expectedMemDiff := toMiB(keyCount * valueSize)
	t.Logf("Expected memory increase of %.2f MiB", expectedMemDiff)

	ctx, cancel := context.WithCancel(context.Background())
	t.Cleanup(cancel)
	go runMemStatusTicker(ctx, t, sampleInterval, statusInterval)

	value := make([]byte, valueSize)
	for i := 0; i < keyCount; i++ {
		err := dm.Put(bkey(i), value)
		if err != nil {
			panic(err)
		}
	}
	runtime.GC()
	after := readMem()
	writePprofHeap(t)

	t.Logf("After Memory: %.2f MiB", toMiB(after.Alloc))
	actualMemDiff := toMiB(after.Alloc) - toMiB(before.Alloc)
	t.Logf("Expected memory increase of %.2f MiB, got %.2f MiB. actual/expected ratio: %.2f", expectedMemDiff, actualMemDiff, actualMemDiff/expectedMemDiff)
}

func toMiB(i uint64) float64 {
	return float64(i) / mebibyte
}

func readMem() runtime.MemStats {
	var mem runtime.MemStats
	runtime.ReadMemStats(&mem)
	return mem
}

func writePprofHeap(t *testing.T) {
	f, err := ioutil.TempFile("", "")
	if err != nil {
		t.Fatal(err)
	}
	runtime.GC()
	if err := pprof.WriteHeapProfile(f); err != nil {
		t.Fatal(err)
	}
	_ = f.Close()
	t.Log("Pprof written to:", f.Name())
}

func max(values ...uint64) uint64 {
	maxVal := values[0]
	for _, val := range values[1:] {
		if val > maxVal {
			maxVal = val
		}
	}
	return maxVal
}

func runMemStatusTicker(ctx context.Context, t *testing.T, sampleInterval, statusInterval time.Duration) {
	sampleTick := time.NewTicker(sampleInterval)
	defer sampleTick.Stop()
	statusTick := time.NewTicker(statusInterval)
	defer statusTick.Stop()

	const maxSamples = 10
	samples := make([]uint64, maxSamples)
	i := 0
	for {
		select {
		case <-ctx.Done():
			return
		case <-sampleTick.C:
			mem := readMem()
			samples[i%maxSamples] = mem.Alloc
			i++
		case <-statusTick.C:
			t.Logf("Current memory: %.2f MiB", toMiB(max(samples...)))
		}
	}
}

And the benchmark:
func BenchmarkDMap_PutMany(b *testing.B) {
	b.ReportAllocs()
	const (
		valueSize = 10 // Value size in bytes.
		tableSize = 10 * mebibyte

		sampleInterval = 5 * time.Second
		statusInterval = 10 * time.Second
	)

	cfg := testSingleReplicaConfig()
	cfg.TableSize = tableSize
	cfg.LogOutput = ioutil.Discard
	db, err := newDB(cfg)
	if err != nil {
		b.Fatalf("Expected nil. Got: %v", err)
	}
	dm, err := db.NewDMap("mymap")
	if err != nil {
		b.Fatalf("Expected nil. Got: %v", err)
	}

	value := make([]byte, valueSize)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		err := dm.Put(bkey(i), value)
		if err != nil {
			panic(err)
		}
	}
	b.StopTimer()
	runtime.GC()
	var mem runtime.MemStats
	runtime.ReadMemStats(&mem)
	b.ReportMetric(toMiB(mem.Alloc), "heap-MiB")
}

Which produced this output for me for a 1m bench time:
go test . -run '^$' -bench BenchmarkDMap_PutMany -v -benchtime=1m

goos: darwin
goarch: amd64
pkg: github.com/buraksezer/olric
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkDMap_PutMany
BenchmarkDMap_PutMany-8         22833872              4389 ns/op              3410 heap-MiB         1640 B/op         26 allocs/op
PASS
ok      github.com/buraksezer/olric     106.409s

The 3 GiB heap is telling me there's effectively a leak in there. I imagine compaction could help take care of it, but some of the memory persists long-term.
Theories
It seems this is most easily triggered when the table size is much larger than the average value size.
I've explored some possibilities so far:

Race condition when determining if a new table should be allocated

I attempted to find a data race on the affected lines but I can't seem to trigger it with go test -race. It might still be a race in the protocol somehow, which the race detector isn't catching, but I tend to trust the race detector.

Incorrect formula for determining the next table size. This one seems most likely to me, though I don't know enough about olric's table allocation to say with any certainty.

Hope that helps. I'll try and keep this issue updated if I discover anything new.

Hi @JohnStarich
Thank you for detailed analysis. I ran TestDMap_PutMany and BenchmarkDMap_PutMany and tried to debug the storage engine. It seems that the storage engine works as expected. Actually, this allocation behavior looks a bit problematic to me too but this can be caused by a misconfiguration.
tableSize = 10 * mebibyte

This is too big, if you have ~200 keys with max 8k value size. tableSize identifies that the initial allocation for a fragment of DMap on the partitions. So if you have 271 partitions on a single node, it will pre-allocate 2.7G on heap with 10M initial table size. So what is your partition count?
If your key count and size is relatively small, keeping tableSize small is a good idea. Olric will allocate more memory if you need. There is only one downside of this: if you insert too many keys (say 1M) while your tableSize is small, Olric is going to create new byte slices on the heap very frequently and this also triggers compaction. So small tableSize decreases the throughput of the cluster.
This behavior has to be documented. It's my bad. Sorry for the inconvenience. I'm also open to discuss the storage engine design.
The engine uses Golang's built-in map to store indexes and a byte slice is used for the entries.
Memory layout of a storage entry. It includes 21 bytes of metadata:
KEY-LENGTH(uint8) | KEY(bytes) | TTL(uint64) | Timestamp(uint64) | VALUE-LENGTH(uint32) | VALUE(bytes)

Index:
map[hash(key)<uint64>]offset<uint64>

Thank you @buraksezer for looking into this!
Our real table size uses the default (of 1 MiB, I believe), but I exaggerated various parts of the test and benchmark to make the problem more evident.
I'm actually in the process of testing a much smaller table size than the default to see if it helps with the leak. Unfortunately, I think that will only slightly mitigate the underlying issue.

This behavior has to be documented. It's my bad. Sorry for the inconvenience. I'm also open to discuss the storage engine design.

No worries, I appreciate your help! Thanks for the data layout, that's good to know 👍
Could you go over how tables are sized and allocated? That may help us tune olric such that it makes this much less likely.

So what is your partition count?

Just re-read and noticed your question: I'm actually not sure what our partition count is. We do have 5 embedded olric nodes, does that correlate with 5 partitions? I'll try and extract more metrics during my pre-production tests.

Just re-read and noticed your question: I'm actually not sure what our partition count is. We do have 5 embedded olric nodes, does that correlate with 5 partitions? I'll try and extract more metrics during my pre-production tests.

Partition count one of the key concepts of Olric. Olric divides the key space into partitions and maintains the partitions among the nodes. These partitions are distributed by a consistent hashing algorithm among the servers. There is no other measure to determine partition ownership. Partition count is 271 by default and directly taken from Hazelcast. You can choose any other number as partition count but prime numbers is good for fair distribution of keys among partitions.
So if you have 5 nodes and your partition count is 271, every node has ~54 partitions to host. If you have add new nodes, some of the partitions and their replicas is going to move to the new nodes. This flow is almost the same for the replicas.
When you start to insert keys to the cluster, Olric allocates 1 MiB (default tableSize) of memory on heap for every partition. So ~54 MiB of memory should be allocated initially by the nodes in your 5-nodes cluster.
I ran the benchmark many times and I still think it works as expected.
goos: darwin
goarch: amd64
pkg: github.com/buraksezer/olric
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
BenchmarkDMap_PutMany
BenchmarkDMap_PutMany-8         70533109              1482 ns/op              9194 heap-MiB           9225 heap-inuse-MiB            566 B/op          9 allocs/op
PASS
ok      github.com/buraksezer/olric     107.986s

It inserts ~70M keys to a single node. There are 7 tables initially. Olric tries to expand them a few times: the engine creates double-sized byte slices and moves everything to the new ones. If everything is okay, the old tables should be freed by the GC. But it takes time and the runtime may reuse the allocated memory. So the heap size grows over and over if you insert 70M keys in a few minutes.
This is my observation but it doesn't explain your initial case. I tried to reproduce it with olricd and olric-load but everything looks normal. ~50MB of memory usage is reported by macOS. It looks like that there is a subtle bug in the engine code. I'm on it.
Is it possible to share your Olric configuration?

@buraksezer Ah, that helps thank you! We're using the default partition count then, and we're spreading them across 6 (I misremembered – thought it was 5 earlier) kubernetes pods with the following config. (These comments were my interpretation of the config options, apologies if they're incorrect 😄)
cfg := olricConfig.New("lan")
cfg.ReplicaCount = 2                                                 // Ensure data is on at least 2 nodes, so it has a chance to survive a pod shutdown.
cfg.ReadRepair = true                                                // If DB is inconsistent (e.g. wrong replica count), attempt to repair.
cfg.ReplicationMode = olricConfig.AsyncReplicationMode               // Run write and delete operations in the background.
cfg.MemberlistConfig.ProtocolVersion = memberlist.ProtocolVersionMax // Since all clients use the same code, use the maximum protocol.
cfg.MaxJoinAttempts = int(5 * time.Minute / cfg.JoinRetryInterval)   // Retry joining for up to 5 minutes.

cfg.LogOutput = logger.Writer() // for some reason, setting Logger doesn't capture logs from memberlist
cfg.ServiceDiscovery = map[string]interface{}{"plugin": provider}

I've also tested adding:
cfg.TableSize = 1 << 18     // 256 KiB

But this doesn't seem to have any effect, sadly.

I ran the benchmark many times and I still think it works as expected.

That's good to hear. I'm struggling to reproduce these same conditions in a controlled fashion, but they do occur in the real environments very consistently. I'll keep trying to reproduce without the abnormal table sizes.
One of the odd things I've noticed in this environment is the container's memory leaps upward in large jumps:

Whenever I've taken a memory profile on those pods, they show the vast majority of memory in use was allocated by newTable(s.Inuse() * 2) 🤷

Configuration looks good to me.

One of the odd things I've noticed in this environment is the container's memory leaps upward in large jumps:

Current design of the storage engine is read optimized. It allocates large chunks of memory on the heap and inserts the key/value pairs. It will never try to allocate memory gradually.
Unfortunately, compaction is an expensive and slow operation. I have prototyped a new design and it performs better than the current one. It also significantly better for write-intensive tasks. It may be major topic for v0.5.0.
I have also fixed a few bugs in the compaction code. These bugs should not make an affect for your case. I wonder that how many keys you inserted to draw that graph. Average value size is 8k, right?

For one sample: 40 MiB slab total, 18 GiB container total with 134 keys on<EMAIL_ADDRESS>
I never managed to reproduce that. I have been inserting 1M keys (value size is 10 bytes) and it consumes ~350Mb. Are sure that you don't have a few very large k/v pairs?

Whenever I've taken a memory profile on those pods, they show the vast majority of memory in use was allocated by newTable(s.Inuse() * 2) 🤷

newTable is malloc for Olric but we don't have a free function. Compaction creates a significant pressure on memory and GC. We need to improve it but this is a significant task for a minor version bump.
By the way, I have used mmap to allocate memory outside the heap but compaction is still too expensive and slow. I have also tried to use jemalloc but it didn't make any difference.

Turns out many of these values were much bigger than I had measured before. To find the value sizes, I've wrapped a new gob serializer with metrics:
func main() {
	...
	collector := metrics.NewCollector(prometheus.DefaultRegisterer)
	cfg.Serializer = wrapSerializerMetrics(cfg.Serializer, collector)
	...
}

type metricsSerializer struct {
	serializer serializer.Serializer
	collector  *metrics.Collector
}

func wrapSerializerMetrics(s serializer.Serializer, collector *metrics.Collector) serializer.Serializer {
	if s == nil {
		s = serializer.NewGobSerializer()
	}
	return &metricsSerializer{
		serializer: s,
		collector:  collector,
	}
}

func (m *metricsSerializer) Marshal(v interface{}) ([]byte, error) {
	data, err := m.serializer.Marshal(v)
	m.collector.ObserveSerializerMarshalSize(len(data))
	return data, err
}

func (m *metricsSerializer) Unmarshal(data []byte, v interface{}) error {
	err := m.serializer.Unmarshal(data, v)
	m.collector.ObserveSerializerUnmarshalSize(len(data))
	return err
}

With this, now I have a better picture of the data size per key.
Under continuous load, my 90% quantile is >256KiB per key. I imagine in production it's much higher than this, too.
Critically, my assumption for key size was completely flawed. 😓  I'll take this as a lesson learned, and sorry for the wild goose chase to find a non-existent leak. This just looks like we're tossing way more data into olric than we have resources for. I'll find a way to prune everything we're not using to get more reasonable sizes.
Would it be worthwhile to collaborate on a Prometheus integration or similar (like more Stats())? I imagine value size metrics would be invaluable data for newcomers to easily spot issues like this.

By the way, I have used mmap to allocate memory outside the heap but compaction is still too expensive and slow. I have also tried to use jemalloc but it didn't make any difference.

Interesting, I hadn't heard of off-heap libraries like this before - very cool! As an aside: It may be harder to debug if it's outside the Go heap too, since it could be invisible to Go & pprof.
One alternative I've seen is in zap's buffer pool implementation. Essentially keeping tabs on large byte slices and mapping memory to those on-demand.

Hi @JohnStarich
I have just released Olric v0.3.9. It includes a minor fix in the compaction code to clean stale tables more effectively.

Critically, my assumption for key size was completely flawed. 😓 I'll take this as a lesson learned, and sorry for the wild goose chase to find a non-existent leak. This just looks like we're tossing way more data into olric than we have resources for. I'll find a way to prune everything we're not using to get more reasonable sizes.

Never mind. 😉 I discovered new things in this research.

Would it be worthwhile to collaborate on a Prometheus integration or similar (like more Stats())?

Absolutely. I had started an exporter several months ago but I was not able to finish it due to lack of spare time. It works as a sidecar but I'm open for an easier integration with Prometheus. Please open an issue for your feature request. 😉

I imagine value size metrics would be invaluable data for newcomers to easily spot issues like this.

Olric v0.4.x has network statistics. ReadBytesTotal would be useful to inspect the problem for this case. We may also add more stats for DMaps such as MeanValueSize or something. By the way, the latest beta version of v0.4.x is much more stable than the previous ones. So you may want to upgrade for the new Stats().

One alternative I've seen is in zap's buffer pool implementation. Essentially keeping tabs on large byte slices and mapping memory to those on-demand.

I'll take a look at this. Thank you!

Thanks @buraksezer! 👏
We're still sitting on v0.3.7 with our smaller-values fix in place. I discovered a separate issue upgrading to v0.3.8 surrounding Gets/Puts taking way longer than expected (max 5s went up to 1m before we canceled our contexts), but I'll open an issue if we come back around to it. For now, it's working! 😄

Absolutely. I had started an exporter several months ago but I was not able to finish it due to lack of spare time. It works as a sidecar but I'm open for an easier integration with Prometheus. Please open an issue for your feature request. 😉

Can do 👍 Seems like https://github.com/buraksezer/olric/issues/57 may cover it, so I'll keep tabs on it there.

Olric v0.4.x has network statistics

Perfect 💯 I'll take a look when I circle back to improvements again, hopefully soon.
Thanks again for digging into the problem with me, greatly appreciated. I'll close, since we've managed to fix it on our end too.
I'll probably need to take a deeper look at the encoder magic: looks like gob is encoding a much larger value than I initially suspected, which could cause future mishaps. But you've made it very easy to swap out the Serializer, so should be no trouble 🙂


Issue while downloading the Image from Private registry

Hi,
While downloading the image from the Private registry , I need to have the credentials supplied within the ".dockercfg" file .
Current work around is to have this copied to VM maually and that is creating a issue in the deployment.
We should have some clean way of running the docker login command as part of the deployment to authenticate against the private registry

Agreed - please see #104


My profile

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

[ ] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Done!

Thank you! Hope you remember me from twitter!

I remember you from Twitter 👍


libtensorflowlite_jni.so (offset 0x2f3000):signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xa59fdbc0

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
org.tensorflow:tensorflow-lite:0.0.0-nightly org.tensorflow:tensorflow-lite-gpu:2.3.0 org.tensorflow:tensorflow-lite-support:0.1.0
Custom code
Yes
OS platform and distribution
Andorid 13
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
Our process com.vt.tv.aipq use tensorflow-lite cause process crash:

Build fingerprint: 'Hisense/songshan-FFM/songshan:11/RTT2.220118.001/<IP_ADDRESS>:user/release-keys'
Revision: '1234'
ABI: 'arm'
Timestamp: 2023-05-06 14:36:47+0800
pid: 14794, tid: 28796, name: TFService-T  >>> com.vt.tv.aipq <<<
uid: 1000
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xa59fdbc0
r0  00000004  r1  00000020  r2  00000300  r3  a59fdbc0
r4  31a05640  r5  000002f0  r6  31a05740  r7  a59fe4c0
r8  31a056c0  r9  a5674220  r10 a59fe1c0  r11 31a055c0
ip  a59fdec0  sp  84db73e8  lr  832ded97  pc  832f08ac
backtrace:
#00 pc 000258ac  /system_ext/app/HiAIPQ/HiAIPQ.apk!libtensorflowlite_jni.so (offset 0x2f3000)
Standalone code to reproduce the issue
The Google server background statistics process crash information, unable to clarify the steps to reproduce the problem.

Relevant log output
No response

tombstone_00.zip

@monsterzlz Could you please try to upgrade to the latest TF version as the older versions are not actively supported.
Keeping extra lines in label.txt would be an issue for crashing. Please confirm the steps to reproduce the issue here.
Thank you!


ansible-galaxy install ansible-role-ovftool

Try to install:
ansible-galaxy install ansible-role-ovftool

downloading role 'ansible-role-ovftool', owned by
[WARNING]: - ansible-role-ovftool was NOT installed successfully: Content has no field named 'owner'

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

@jimb0bmij use:
ansible-galaxy install vmware.ovftool


Text area with lable animation not working correctly.

Text area with lable animate will not work correctly if the value is in text area table return into normal position.

Fixed: https://github.com/origin-space/originui/commit/ca70aa31b3be96f4e10befb9cca11f45bbd0ba20


Add: Impallari on 'Fonts' category

There are bunch of Libre typeface projects in Impallari. Wanna add on awesome-stock-resources' ‘Fonts’ category.

Perfect, thanks @young-hwa!


AuditLogRecordType  needs to be updated as I find a new record with value 50

Can you please update the audit log record type list as the Content URI is returning a record with value 50 and There is nothing mentioned against the Enum: AuditLogRecordType - Type: Edm.Int32

@KalapJo1 Thanks for heads up. Audit records with RecordType = 50 are related to the MailItemsAccessed mailbox auditing action (which requires E5 user license). For more information, see https://docs.microsoft.com/en-us/microsoft-365/compliance/mailitemsaccessed-forensics-investigations?view=o365-worldwide
I will add RecordType 50 to the schema documentation.


Default to sourceLocale when --localize=false

This looks like a regression to me in v9:
I couldn't find a way to tell the cli the default locale to inject in my app instead of the default en-US. The only method I found is using the localize flag, but enabling it changes the way the app is built and slows the process with the extra Generating localized bundles phase.
Shouldn't it inject the sourceLocale defined in the angular.json the way the --i18nLocale flag used to worked in pre v9 versions?

Hi @javiermarinros, can you give some more details on what you're trying to do?
Are you trying to make users visiting your site use a different locale by default? Are you trying to make local builds use a different locale by default? Are you trying to change the default value of LOCALE_ID injected into your app?

Hi @dgp1130, I would like to configure the default app locale (that is hardcoded to en-US) without using the localize flag.
I think this would be better explain with an example: Using this angular.json should create builds using es insteadof en-US
 "projects": {     "projectName": {       "root": "",       "sourceRoot": "src",       "projectType": "application",       "i18n": {         **"sourceLocale": "es"**       },       "architect": {         "build": {           "builder": "@angular-devkit/build-angular:browser",           "options": {             **"localize": false,**             ...

Ok, I think I'm understanding now. This was more of a historical limitation in View Engine when we didn't track sourceLocale as a separate locale, so we always used en-US. Currently there is no way to set the default to another locale without using --localize. The closest you can get is:
"projects": {
  "my-project": {
    // ...
    "i18n": {
      "sourceLocale": "es",
      "locales": {
        // ...
      }
    },
    "architect": {
      "build": {
        "options": {
          // ...
          "localize": [ "es" ]
        }
      }
    }
  }
}

Then ng serve would serve This would localize only to the Spanish language in default builds, though it still has the overhead costs you mentioned (but only for one language).
As a feature request this is pretty reasonable. It makes sense that the sourceLocale should be the default when --localize=false. To make this happen, when --localize=false, we'll need to:

Include locale data in the bundle.
Set LOCALE_ID to the sourceLocale value, so modules which inject it are accurate.

Neither of these take too long in the build process, so they should not impact build times too much. The slow part is dealing with translations, which we don't have to do here because we are using the sourceLocale. Once that happens, ng serve and ng build should use sourceLocale throughout the app by default.

Is there any ETA on this or any way someone from the outside could help? I am currently contemplating to switch the main language of my application to english because this step is almost a twentyfold increase in the buildstep when making changes: The incremental build finishes almost instantly, and the following localized bundle generation takes 10 to 20 seconds.

A v17 application will now work as explained in the above comments and use the sourceLocale for both the application locale and the available locale data. If upgrading from an earlier version to v17, also convert the project to the new build system via https://angular.dev/tools/cli/esbuild


on centos7

TASK [devops-coop.minecraft : check supported version] *************************
fatal: [<IP_ADDRESS>]: FAILED! => {"msg": "The conditional check 'ansible_distribution == 'Ubuntu' and ansible_distribution_version | version_compare('16.04', '<')' failed. The error was: template error while templating string: no filter named 'version_compare'. String: {% if ansible_distribution == 'Ubuntu' and ansible_distribution_version | version_compare('16.04', '<') %} True {% else %} False {% endif %}\n\nThe error appears to be in '/root/.ansible/roles/devops-coop.minecraft/tasks/systemd/Ubuntu.yml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: check supported version\n  ^ here\n"}

The same problem on ubuntu 22.04

the same on ubuntu 16.04

rather than just listing OS versions with no detail, in order for others to even attempt to help here, it would be good if you can provide more info:
namely:
the version of ansible you are trying to use
the playbook you are using to call this role
running the playbook with -vv, to present additional logging information.
However, the fact that this repo, appears to have not been worked on for 4-5 years, indicates that perhaps things have moved on, and the later flavours of ansible don't support it, in it's current form....
I am just another user of this repo (btw, it works fine on my CentOS 7 servers.....) but if there's actually something of diagnostic use posted in this ticket regarding the issue, it would perhaps assist with any troubleshooting....


Makefile this Binding

Just for the docs, it might be good to note that in shelljs makefiles targets, this is set to the function itself, not the main target object as some might expect.  This is because this is explicitly rebound to each target function in the duplicate prevention wrapper (make.js#L36).

We don't really support make anymore (there are dedicated node task runners like grunt and gulp). If you'd like to update the docs, feel free to change the wiki page for it.


Race condition when injecting translate pipe into component

I'm submitting a ...  (check one with "x")
[x] bug report => check the FAQ and search github for a similar issue or PR before submitting
[ ] support request => check the FAQ and search github for a similar issue before submitting
[ ] feature request

Current behavior
The translate pipe does not function properly when injected into a component. See the attached plunker (inside ngOnInit()) for an example of this race condition.
Expected/desired behavior
Depends on the answer to this question: How should ng2-translate be used inside of a component? As far as I can tell it's an undocumented use case. If injecting the service, values do not update when the language is changed. Pipe encapsulates this functionality, and is also available to be injected into components, so it would seem the logical choice. If it is indeed the right thing to use, then it should function identically to when it's used inside a template.
Reproduction of the problem
If the current behavior is a bug or you can illustrate your feature request better with an example, please provide the steps to reproduce and if possible a minimal demo of the problem via https://plnkr.co or similar (you can use this template as a starting point: http://plnkr.co/edit/tpl:btpW3l0jr5beJVjohy1Q).
http://plnkr.co/edit/kzbZ4O0VKnqiYWzhsSxI
What is the expected behavior?
Already answered this.
What is the motivation / use case for changing the behavior?
Avoid the boilerplate of having to copy and paste sections of TranslatePipe to get the desired functionality.
Please tell us about your environment:

ng2-translate version: x.x.x

ng2-translate 4.2.0

Angular version: 2.0.0-rc.X

angular 2.3.1

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Tested in chrome.

Language: [all | TypeScript X.X | ES6/7 | ES5]

Typescript

Hello,
I'm not sure I understand why you need to inject the pipe into the component ?

If injecting the service, values do not update when the language is changed.

You can subscribe to onLangChange events, isn't it what you need ?

If that's the proper use case then I'm fine with it, but injecting the translate pipe seems cleaner than having to manage that subscription in each component.

Well the thing is that if you inject the pipe in your component I have no idea if the subscriptions in the pipe are really cleaned up after the component is destroyed (when it's in the template it gets handled by angular with the OnDestroy life cycle hook).

So, basically you're left with having to do this (is it correct?):
    this.translate.get('HOME.TITLE').subscribe((value: string) => {
      this.myValue = value;
    });
    this.translate.onLangChange.subscribe((event: any) => {
      this.translateService.get('HOME.TITLE').subscribe((value: string) => {
        this.myValue = this.translateService.transform('HOME.TITLE');
      });
    });

It's quite a bit of boilerplate, and it gets a lot bigger when using multiple values.

I'm just wondering about the use case of this? What are you trying to do that's not possible with using a pipe in the template itself?

You can use "instant":
    this.translate.get('HOME.TITLE').subscribe((value: string) => {
      this.myValue = value;
    });
    this.translate.onLangChange.subscribe((event: LangChangeEvent) => {
      this.myValue = this.translateService.instant('HOME.TITLE');
      // or even:
      // this.myValue = event.translations['HOME.TITLE'];
    });

But I get what you want, some kind of observable that would get the translation immediately and then call the callback when the lang changes as well. It makes sense and would be easier than doing 2 subscriptions. I'll see how to implement that when I get the time.

@SamVerschueren I've encountered a few situations where this is useful. Since the one I'm dealing with now is the only one that comes to mind, I'll use that as an example. We have a custom select box plugin that uses ul / li structure. The selected value is set by a callback. I could keep the translation key as the value and then put it in a pipe in the template instead, but the problem is that the selected value is sometimes distances, such as "100km", but also the string "Anywhere", which must get translated.
@ocombe Thanks!

Follow up this issue in https://github.com/ocombe/ng2-translate/issues/363, closing this one as I believe it's a duplicate (the desired functionality is the same in both cases)


fix(events): add information about sustainable events

:tada: This PR is included in version 1.33.26 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


From Swinject, is it possible to have an Struct as Singleton (inObjectScope(.Container))?

Currently, I have all the services defined as Structs.
There is only one service I would like to have as a singleton (inObjectScope(.Container)). Since this service is an Struct, the scope .Container is not working. It only works if I define this service as Class. (Of course, I know the differences between Structs and Classes, and I know why this is not working).
I think, Swinject could be improved adding this capability. I think, internally, once an struct is registered with the .Container scope and it needs to be returned, it could be coping all the data to returned Struct in order to keep the same data available and simulate the Singleton behaviour.

I added a unit test (c598e7a844f9ac813116108615a7971e08e77985). Hopefully it would be an answer to this question.
If .Container or .Hierarchy is specified, a factory closure is called only once to simulate Singleton, which has only an instantiation.

I'm closing this issue because I think my comment above answers to your question. Please feel free to reopen this issue if my answer didn't catch your question correctly.


Fix 'static compilation error

Update Bevy to 0.14.2
Bump version to 0.1.3

Thanks for the contribution!


init rng offset with ground clearance param when on ground to avoid n…

…oisy init
Should solve https://github.com/PX4/ecl/pull/292#issuecomment-314708514
@jlecoeur can you please retest with this one?

I rebased and changed it to an else if instead of to ifs . Logic is still the same.

@jlecoeur So you would prefer something like
if (_control_status_prev.flags.rng_hgt != _control_status.flags.rng_hgt) {
    resetHeight();
    _hgt_sensor_offset = _params.rng_gnd_clearance;
}

right?
@priseborough what do you think?

@ChristophTobler Yes, something like that.
@priseborough to summarize, I am worried that _state.pos(2) does not estimate height when range is primary, as suggested in the doc.

I'm going to merge this one as it is a fix that better gets in sooner than later. I'm going to open an issue/enhancement for your suggestion @jlecoeur

@ChristophTobler Due to the critical nature of the estimator can we please ensure that test evidence is supplied and a code review completed before merging functional changes. I apologise that I was unable to get to this last week due to overseas travel, however there should have been either link to a log from the test of this branch or a log enabling the reviewer to verify the correct behaviour.

@priseborough You are absolutely right! I tested the two PRs but did not link the logs to this one... https://github.com/PX4/ecl/pull/295#issuecomment-315765659

I would like to get @CarlOlsson added to the list of reviewers, do you know how we can do this?

I guess he has to have merge/admin rights to this repo

He has now.


[Question] 4.1.1 crashes the app with null pointer - at Session.getSessionContext - is anything missing?

I recently upgraded to 4.1.1 from version 3.0.0 and there seems to be a crash reported from the sdk:
Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.Map ee0.e.d()' on a null object reference
       at com.snowplowanalytics.snowplow.internal.session.Session.getSessionContext(Session.java:202)
       at com.snowplowanalytics.snowplow.internal.tracker.Tracker.workaroundForIncoherentSessionContext(Tracker.java:757)
       at com.snowplowanalytics.snowplow.internal.tracker.Tracker.track(Tracker.java:629)
       at com.snowplowanalytics.snowplow.internal.tracker.TrackerControllerImpl.track(TrackerControllerImpl.java:94)

i set the up the tracker as under:
Snowplow.setup(
            application,
            remoteConfig,
            listOf(configBundle)
        ) { configurationPair ->
            Snowplow.getDefaultTracker()?.attachGlobalContextToTracker()
            -- manually attach global config since this is not possible at the moment with remoteConfig.
        }

and send the event as below:
Snowplow.getDefaultTracker()?.track(selfDescribingEvent)

Is there something i might be missing? Or may i please know why this crash is happening? I am confident this never happened with version 3.0.0

Thanks for the report! We will try to replicate it and get back to you soon.

HEy @mscwilson, we are experiencing the same issue, we were using 4.0.0 for a while, it started doing the issue, upgraded to 4.1.1 hoping it fixes it, but now there are more reports of this crash.
Its currently breaking the app, during start. Seems to be happening api 12-13 on newer devices like pixel 7.

Hi @krobert, thanks for reporting. We were able to replicate the error in PR #582 in case session tracking is paused and the tracker tracks an event. Do you by any chance pause session tracking in your app using tracker.session.pause()?

.pause()

I updated my comment with the crash report from crashlytics.
No we are not pausing the session , we receive a crash at app start onSessionUpdate callback, when we are trying to get the session id from the tracker

Thanks @krobert, that's useful!
Just a couple more questions to help us identify the problem since I think yours is slightly different than the one reported previously:

Are you also using remote configuration to configure the tracker?
Is it true that you are calling the above snippet from inside the onSessionUpdate callback? In that case, why not use the SessionState object that is passed to the callback to retrieve the information? We will try to fix also the access from the tracker object, but using the state object passed to the callback can be a quick fix for the exception that you are getting.

@matus-tomlein

yes, we configure the tracker right after, bit problematin, we have to do our own event caching mechanism
sure we can do that

Hi @dss-kenya and @krobert, the bug has been fixed in the 4.1.2 release. I hope it will help, please let us know if not.

Thank you @matus-tomlein 👍


@ashfurrow => Beginners 1

Similar, but different.

Main thrust feels right – just a few pieces of feedback – should be fun!

Agreed on all the things. Changed.

Nice!


to_df method is inheritted from base Source class and do not take additional parameters

HI,
I faced that issue when migrating Eurostat Source. to_df method is taking only self and if_empty parameters, so there is no possibility to pass directly to to_df method more arguments (I tried and I had error that to many arguments are passed to the function). Maybe I did something wrong or something changed in base Source class. I see that most of the connectors have only 2 parameters, but for example sharepoint has multiple one and it seems that it does not inherited this method from base Source class.

@adrian-wojcik that's not an issue with the library, you might need to learn OOP though to understand how inheritance works. Here's a working example of overriding the to_df() method:
from viadot.sources.base import Source

class MySource(Source):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def to_df(self, x, y, z):
        print(x)
        print(y)
        print(z)

MySource().to_df(1, 2, 3)


System.Web, System.Configuration, WebApi dependencies

After a lot of struggle I succeeded getting the BotBuilder running in a selfhosted Owin environment, which  usually is easy with WebApi. I had to learn a lot about the use of Autofac in BotBuilder, which I generally didn't like as it does lots of logic and parameter passing in non-obvious ways.
I eventually worked around those problems, but I think I encountered some bugs and issues that could be fixed or improved. The easiest way to find these issues is to remove the dependency on System.Web, and try to configure the connector without using AppSettings. I few things I found:
BotAuthentication does:
MicrosoftAppId = MicrosoftAppId ?? ConfigurationManager.AppSettings[MicrosoftAppIdSettingName ?? "MicrosoftAppId"];

A more correct implementation would be:
MicrosoftAppId = MicrosoftAppId ?? (actionContext.Request.GetDependencyScope().GetService(typeof(MicrosoftAppCredentials)) as MicrosoftAppCredentials)?.MicrosoftAppId;

BotAuthentication also has this no-no:
if (HttpContext.Current != null)
    HttpContext.Current.User = Thread.CurrentPrincipal;

I think that should be:
actionContext.RequestContext.Principal = Thread.CurrentPrincipal;

MicrosoftAppCredentials uses a System.Web.HttpRuntime.Cache (which I replaced with a MemoryCache) and inspects the HttpContext.Current.User in ShouldSetToken, which I removed (even if I didn't fully understand what that does, but it still works "on my machine").
I think it would be better to factor out the whole WebApi dependency from the Connector as it seems only required for the BotAuthentication attribute. That attribute could be distributed as a separate nuget, so that it becomes easy to make an AspNetCore implementation in another nuget.

@dandriscoll can you comment or close as appropriate?

@msft-shahins may be better able to weigh in -- he's looking at how this layer sits on frameworks like .Net Core.

We prereleased Microsoft.Bot.Connector nuget 3.5.1-alpha which adds support for asp.net core to connector SDK. Bot authentication is now implemented as a middleware: https://github.com/Microsoft/BotBuilder/blob/master/CSharp/Library/Microsoft.Bot.Connector.AspNetCore/BotAuthentication.cs#L73 and here is an example asp.net core bot: https://github.com/Microsoft/BotBuilder/tree/master/CSharp/Samples/Microsoft.Bot.Sample.AspNetCore.Echo
Feel free to comment about changes and send pull requests for missing features and bugs.


运行报错：OpenShift server failed

我的系统是ubuntu 12.04 64bit的，python版本是2.7
第一次运行，sudo python bypy.py list
报以下错误：
Refreshing, please be patient, it may take upto 60.0 seconds...
Authorizing/refreshing with the OpenShift server ...
 [17:09:22] Exception:
<type 'exceptions.ValueError'> - No JSON object could be decoded
Stack:
File "bypy.py", line 4016, in
main()
File "bypy.py", line 3995, in main
result = getattr(by, args.command[0])(*uargs)
File "bypy.py", line 2144, in list
return self.__get(pcsurl + 'file', pars, self.__list_act, (rpath, fmt))
File "bypy.py", line 1749, in __get
return self.__request(url, pars, act, 'GET', actargs, retry, addtoken, dumpex, **kwargs)
File "bypy.py", line 1728, in __request
result = self.__request_work(url, pars, act, method, actargs, addtoken, dumpex, **kwnew)
File "bypy.py", line 1599, in __request_work
if ENoError == self.__refresh_token(): # refresh the token and re-request
File "bypy.py", line 1943, in __refresh_token
result = self.__get(url, pars, self.__refresh_token_act, retry = retry, addtoken = False)
File "bypy.py", line 1749, in __get
return self.__request(url, pars, act, 'GET', actargs, retry, addtoken, dumpex, **kwargs)
File "bypy.py", line 1728, in __request
result = self.__request_work(url, pars, act, method, actargs, addtoken, dumpex, **kwnew)
File "bypy.py", line 1582, in __request_work
perr(formatex(ex))
File "bypy.py", line 433, in formatex
type(ex), ex, ''.join(traceback.format_stack()))
 [17:09:22] Maximum number (1) of tries failed.
OpenShift server failed, authorizing/refreshing with the Heroku server ...
然后我试了下 sudo pip install bypy，运行结果如下，好像也不能安装
Downloading/unpacking bypy
Downloading bypy-1.2.9.tar.gz (47Kb): 47Kb downloaded
Running setup.py egg_info for package bypy
WARNING: Can't detect encoding for stdout / stderr, assume it's 'UTF-8'.
Files with non-ASCII names may not be handled correctly.
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/root/sh/baiduwangpan/bypy/build/bypy/setup.py", line 10, in <module>
    long_desc += pypandoc.convert('HISTORY.md', 'rst')
  File "/usr/local/lib/python2.7/dist-packages/pypandoc/__init__.py", line 50, in convert
    outputfile=outputfile, filters=filters)
  File "/usr/local/lib/python2.7/dist-packages/pypandoc/__init__.py", line 68, in _convert
    raise RuntimeError('Missing format!')
RuntimeError: Missing format!
Complete output from command python setup.py egg_info:
WARNING: Can't detect encoding for stdout / stderr, assume it's 'UTF-8'.

Files with non-ASCII names may not be handled correctly.
Traceback (most recent call last):
File "", line 14, in
File "/root/sh/baiduwangpan/bypy/build/bypy/setup.py", line 10, in
long_desc += pypandoc.convert('HISTORY.md', 'rst')

File "/usr/local/lib/python2.7/dist-packages/pypandoc/init.py", line 50, in convert
outputfile=outputfile, filters=filters)

File "/usr/local/lib/python2.7/dist-packages/pypandoc/init.py", line 68, in _convert
raise RuntimeError('Missing format!')

RuntimeError: Missing format!

Command python setup.py egg_info failed with error code 1
Storing complete log in /root/.pip/pip.log

感谢反馈。回复的有点晚。
贴下你的locale输出呢？是不是没有设成UTF-8？

locale 运行结果
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

哦，locale看上去没问题。
你试一下不用sudo，直接python bypy list -dv呢？

试试bypy.py -c然后重新授权

按照 jonahfang 说的重新授权就成功了，谢谢啊

感谢 @jonahfang 的回答，发现原来的相关提示并不会输出，修改了一下提示，现在出现这种错误应该会输出相应的提示了。


Support Github Enterprise

change hostname and path for public github and enterprise github

Hey, it would be nice to get support for GHES upstream to faciliate evaluating this action. Thanks!


Feature: Optimize Clusterfile structure

What is the problem this feature will solve?
This format is not friendly enough and there are duplicate configurations

Adding a port configuration to an IP address is not a good format，and here have two ssh port config, Easily annoying
This format is more friendly
apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  name: default
spec:
  hosts:
    - {name: node1, address: <IP_ADDRESS>, port: 22, user: root, password: "Sealos@123", pk: /root/.ssh/id_rsa, arch: amd64}
    - {name: node2, address: <IP_ADDRESS>, port: 22, user: root, password: "Sealos@123", pk: /root/.ssh/id_rsa, arch: amd64}
    - {name: node3, address: <IP_ADDRESS>, port: 22, user: root, password: "Sealos@123", pk: /root/.ssh/id_rsa, arch: amd64}
    - {name: node4, address: <IP_ADDRESS>, port: 22, user: root, password: "Sealos@123", pk: /root/.ssh/id_rsa, arch: amd64}
    - {name: node5, address: <IP_ADDRESS>, port: 22, user: root, password: "Sealos@123", pk: /root/.ssh/id_rsa, arch: amd64}
  roleGroups:
    master:
      - node1
      - node2
      - node3
    worker:
      - node4
      - node[5:10]
    registry:
      - node11
  image:
  - labring/kubernetes:v1.24.0
  - labring/calico:v3.24.1

Currently, sealos does not support automatic configuration of hostnames, which is the biggest problem that prevents automated installation， This configuration should solve the problem.
If you have solution，please describe it
Adjusting the Clusterfile structure
What alternatives have you considered?
No response

Break changes, If this modification means that many parts of the system need to be adjusted, it is a big change and requires careful consideration.


decode a Dictionary

Hi i get a error when i try decode a Dictionary
public struct ShowMorePosts {
    public let posts: [STPost]
    public let showMorePosts: [String:[STPost]]
}

extension ShowMorePosts: Decodable {
    public static func decode(j: AnyObject) throws -> ShowMorePosts {
        return try ShowMorePosts(
            posts: j => "posts",
            showMorePosts: j => "show_more_posts"
        )
    }
}

Note: STPost is confirming to Decodable
I get this error Cannot convert value of type 'AnyObject' to expected argument type '[String : [STPost]]'

We need a [K: [T]]-returning overload. Think there are only 2 dictionary-related overloads at the moment, so a lot of useful ones are missing. Will look at it tomorrow. Feel free to beat me in implementing one though. Should just be to copy suiting existing dict overload, but letting the elementDecodeClosure be decodeArray(T.decode)


Global scripts are included in the index.html but, not the .angular-cli.json - Doing such throws an error

This is a question more than an issue. Couldn't find the reason googling.
My question is since angular-cli has structured in a  way that we can add all the scripts, global files and libraries in the .angular-cli.json script array or style array.
Why in this repository it's done otherwise?
I've been using Cognito and I am using all these scripts in my application. It throws an error if I follow the approach recommended by angular-cli.
I've added all these scripts in the index.html after placing them in the assets folder for now as below.
<!-- Cognito -->
  <script src="./assets/js/cognito/jsbn.js"></script>
  <script src="./assets/js/cognito/jsbn2.js"></script>
  <script src="./assets/js/cognito/sjcl.js"></script>
  <script src="./assets/js/cognito/aws-cognito-sdk.min.js"></script>
  <script src="./assets/js/cognito/amazon-cognito-identity.min.js"></script>
  <script src="./assets/js/cognito/aws-sdk-2.7.13.min.js"></script>
  <script src="./assets/js/cognito/aws-sdk-mobile-analytics.min.js"></script>
<!-- /Cognito -->

Any help on reasoning and do it in the angular-cli recommended way would be appreciated.

I've created the project before angular2 was finalized, so many of currently-documented features weren't documented or developed. So feel free to submit a PR with any updates, especially those following angular best-practices.


Bug: Double-wrapped OpenCDC records

Bug description
While the source pipeline produces a valid OpenCDC record, the destination will consume this from Kafka, assume it is raw data and create a new OpenCDC record with the source record set as rawdata in the Payload.After field. It ends up being re-wrapped in another OpenCDC record, making it double-wrapped.
Suggestions:

Add an OpenCDC unwrap processor
When producing to kafka topics on the platform, only emit the contents of payload.After

Steps to reproduce

Create a pipeline that uses a destination builtin conduit connector. example:

version: 1.0
pipelines:
  gen-to-kafka:
    status: running
    description: >
     generator to kafka
    connectors:
      source-generator:
        type: source
        plugin: builtin:generator
        settings:
          format.options: event_id:int,pg_generator:bool,sensor_id:int,msg:string,triggered:bool
          format.type: structured
          recordCount: 100
          readTime: 1000ms
      dest-kafka:
        type: destination
        plugin: standalone:kafka
        settings:
          insecureSkipVerify: true
          servers: <put local kafka brokers here>
          topic: demo-123

Make sure you produce a record that can be produced by the destination connector. Consuming that record you'll notice it double wrapped. Example:

{
  "key": "MTc3NzQ5NDEtNTdhMi00MmZhLWI0MzAtODkxMmE5NDI0YjNh",
  "metadata": {
    "conduit.source.connector.id": "dest-log-78lpnchx7tzpyqz:source-kafka",
    "kafka.topic": "stream-78lpnchx7tzpyqz-generator",
    "opencdc.createdAt": "1706028953595000000",
    "opencdc.readAt": "1706028953606997000",
    "opencdc.version": "v1"
  },
  "operation": "create",
  "payload": {
    "after": {
        "position": "NzgyNjJmODUtODNmMS00ZGQwLWEyZDAtNTRmNjA1ZjkyYTg0",
        "operation": "create",
        "metadata": {
          "conduit.source.connector.id": "source-generator-78lpnchx7tzpyqz:source",
          "opencdc.readAt": "1706028953595546000",
          "opencdc.version": "v1"
        },
        "key": "MTc3NzQ5NDEtNTdhMi00MmZhLWI0MzAtODkxMmE5NDI0YjNh",
        "payload": {
          "before": null,
          "after": {
            "event_id":<PHONE_NUMBER>,
            "msg": "string 0e8955b3-7fb5-4dda-8064-e10dc007f00d",
            "pg_generator": false,
            "sensor_id":<PHONE_NUMBER>,
            "triggered": false
          }
        }
      },
    "before": null
  },
  "position": "eyJHcm91cElEIjoiNGQ2ZTBhMjktNzAwZi00Yjk4LWEzY2MtZWUyNzZhZTc4MjVjIiwiVG9waWMiOiJzdHJlYW0tNzhscG5jaHg3dHpweXF6LWdlbmVyYXRvciIsIlBhcnRpdGlvbiI6MCwiT2Zmc2V0IjoyMjF9"
}

Version
v0.8.0

@raulb This looks to be an issue when Kafka is used as an intermediary? For example, source foo -> kafka -> destination bar?


Fixed Typo In PetClinicIntegrationTests

@programmingfire Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@programmingfire Thank you for signing the Contributor License Agreement!