Extensible persistence support (e.g. React Native, browser)?

I'm really interested in using persistent Datomic implementations across the board (on the backend on the JVM, and on the frontend in the browser or via React Native). Of course, Datomic is great on the JVM, and DataScript is all right for ephemeral (non-persistent) use cases in any JavaScript environment, but Datomish (now Mentat?) caught my eye as a possible candidate for a general JavaScript Datomic implementation with persistence (something DataScript obviously lacks).
However, it seems that Datomish is focused on NodeJS and Firefox. Is there a way to straightforwardly hook up Datomish to e.g. ReactNative's AsyncStorage or the browser's IndexedDB? I'm totally fine with contributing, but I'm looking to avoid reinventing the wheel ;)
Thanks for your help!

You're the second person I've seen ask about whether Mentat would be suitable as a persistent datom store for the web.
This answer could get very long, so I'll try to keep it short!
tl;dr: "straightforwardly hook up" — absolutely not.
There are two broad reasons why not: implementation choices and web limitations.
The proof-of-concept version of Mentat was built in ClojureScript, so in principle you could use it in a web context or anywhere else you could run JS… if you don't mind having an opaque 600KB lump of compiled JS to debug, churned out by an inscrutable 60-second build step through Closure Compiler. We didn't like that situation — no way would we be willing to support that code for hundreds of millions of users in Firefox — and so a rewrite was in order. We don't intend to spend any further time on the ClojureScript version, so you couldn't rely on bugs being fixed.
Our rewrite is in Rust, which you could try compiling with emscripten to get JS/WebAssembly that you could run in a browser. But somewhere at the bottom of the stack is SQLite, trying to memory-map files and control fsync calls. Without the deprecated WebSQL, there's no suitable foundation on the web itself.
You could try using the Rust binary as a custom component from React Native. And you could use nativeMessaging in a WebExtension. But that requires a bit more work than just dropping a JS library into your toolchain.
Mentat is quite closely tied to SQLite: it relies on a SQL engine's transactions, constraints, full-text search, and more, and it expects full control over everything for performance. A Datalog query is translated into a single SQL query for execution.
That makes Mentat a poor fit to IndexedDB, which pushes an abstraction layer of its own, and requires third-party code for FTS. React Native's AsyncStorage is itself layered on top of SQLite (or RocksDB, or flat files!), and is too simple for our needs.
One could imagine a datom store that's backed by IndexedDB or AsyncStorage, in the same way that we're backed by SQLite, but I would be worried about the performance of such a solution.
Could such a datom store share code with Mentat? Maybe. The EDN and query parsers, certainly, but the majority of the code is about implementing transacts, connection handling, and translating queries, and all of that would need to be abstracted and reimplemented… not to mention figuring out how a Rust library is sensibly shipped as a consumer of a JavaScript API.
In the longer term, a good answer might be exposing Mentat through a WebExtensions API, so you could use it instead of IndexedDB.

However, it seems that Datomish is focused on NodeJS and Firefox. Is there a way to straightforwardly hook up Datomish to e.g. ReactNative's AsyncStorage or the browser's IndexedDB? I'm totally fine with contributing, but I'm looking to avoid reinventing the wheel ;)

I started to type a message saying you might have a better experience trying to implement the DataScript DB protocol backed by IndexedDB, but then I remembered that IndexedDB has an asynchronous access pattern, which really doesn't play well with DataScript's synchronous implementation.  I do wonder if there's some middle ground where you make DataScript-in-CLJS use JavaScript generators so it can be both synchronous and pseudo-asynchronous.
Sadly, I'll concur with @rnewman -- Mentat/CLJS is not built to do what you want, and Mentat/Rust is not going to do what you want either.  Sorry!

Our rewrite is in Rust, which you could try compiling with emscripten to get JS/WebAssembly that you could run in a browser. But somewhere at the bottom of the stack is SQLite, trying to memory-map files and control fsync calls. Without the deprecated WebSQL, there's no suitable foundation on the web itself.

There is prior work that compiles SQLite to js with https://github.com/kripken/sql.js.  Details about persisting the db on a page with https://github.com/kripken/sql.js/wiki/Persisting-a-Modified-Database / http://kripken.github.io/sql.js/examples/persistent.html.

Yeah, Kripken hosts a SQLite 'file' as an in-memory byte buffer, relying on the caller to figure out when to dump a few megabytes of byte array into some kind of persistent storage, and making sure that no concurrent modifications occur.
Kripken is very cool, and It might be fine for relatively small workloads, but because everything is in memory and has to be manually persisted, at that point you might as well use DataScript — you're in the same boat, but the code is already written and is meant to be used on the web.

First of all, thank you all so much for your quick and generously long replies!
To @rnewman :
I can't say I'm on board with your reasoning for switching away from ClojureScript, as "an opaque 600KB lump of compiled JS to debug", as you say, could apply to probably many libraries of comparable size, especially ones which use a feature-rich compile-to-JS-language like ClojureScript. I'm not sure that the build/optimization step is as "inscrutable" as you say, or why 60 seconds of compilation/optimization time is an abomination when it often takes e.g. C++ code tens or hundreds of times that to achieve the same result (ignoring incremental builds, which ClojureScript also has via e.g. Figwheel). But to each his own, I suppose :) That said, I do see the raw speed advantage of Rust.
The Rust -> WebAssembly possibility sounds intriguing, but as you say, without hooking into the deprecated WebSQL API, it seems like that wouldn't achieve what I'd want to achieve.
Thanks for making me aware that Mentat is so tightly coupled to SQL. However, from what I've read (and I could be wrong), IndexedDB is pretty performant, right? I can't speak for AsyncStorage, because I haven't investigated it as much, and my guess is that it's not performance-optimized. What makes you worry about IndexedDB's performance as compared to SQLite?
Using the WebExtensions API is probably straightforward, yes, but not the route I'm looking to go for, as I'm looking to avoid plugins wherever possible.
The Rust binary as a custom component from React Native doesn't sound too terrible at all actually. Maybe a little painful / not entirely straightforward the first time around, but once everything is in place, it's just a matter of tweaking e.g. the React Native header files when/if Mentat API changes take place. It's certainly a possibility for non-browser environments.
To @ncalexan :

IndexedDB has an asynchronous access pattern, which really doesn't play well with DataScript's synchronous implementation.

That probably is semi-true, but it's entirely possible to use e.g. the following in ClojureScript:
; under the hood, use IndexedDB's callback with a promise-chan,
; then (potentially) transacts to the in-memory DataScript DB
(go (<! (persistent-transact! conn txn-data))
    ...)

in the same way one uses the following in Clojure:
@(transact conn txn-data)

What other concerns do you have about IndexedDB's asynchrony?
To @bgrins:
Thanks so much for your helpful link about sql.js! I'll look into that, even if @rnewman 's point about it having to be manually persisted still holds.

For context, I've been writing Lisps for over fifteen years, Clojure for seven or eight. I'm no stranger to bad tooling; we wrote Firefox in pre-stable Swift. Iterative development of Mentat in Clojure wasn't bad. Deploying to two or three different JS destinations with CLJS, and debugging there, was an absolute nightmare.
We had a regression traced to a minor CLJS update, for example, which is terrifying. I had to resort to bisecting, because there's no way I could figure out what was going wrong: heavy use of channels plus a compilation step or two plus inadequate source mapping meant printf debugging, and the bug only occurred in the wild, with a 70-second rebuild with each print statement…
Rust's tooling is pretty great, by comparison.

That probably is semi-true, but it's entirely possible to use e.g. the following in ClojureScript:

If you look at the DataScript implementation, you'll see that the DB protocol really needs to be synchronous.  I tried to make DS use go blocks, etc and I found it not possible.  Others have tried as well, I believe... see https://github.com/tonsky/datascript/issues/190 and other links.

IndexedDB is pretty performant, right? I can't speak for AsyncStorage, because I haven't investigated it as much, and my guess is that it's not performance-optimized. What makes you worry about IndexedDB's performance as compared to SQLite?

Although I do doubt whether implementing Mentat's ideas on top of IndexedDB would result in a library that was as fast as doing so with direct SQL, my concern is more about the depth of the layers of abstraction.
IndexedDB in Firefox is implemented as a layer (written in C++) on top of SQLite, so adding yet another layer on top for Mentat seems… wasteful. IndexedDB also doesn't give fine-grained control of e.g., result prefetching, page sizes (naturally), vacuum behaviors, and all the other techniques we expect to need. And, as previously mentioned, even though it's SQLite underneath, we wouldn't be able to get access to FTS.

@ncalexan :
Interesting. I apparently haven't dug into the internals of DataScript as deeply as I needed to in order to make an informed statement on whether it can or cannot be used asynchronously — my apologies. The link you posted is extremely useful but disheartening because what I'd really like is a JS implementation of Datomic with pluggable storage services à la, well, Datomic — except instead of e.g. DynamoDB or Cassandra, it might use e.g. IndexedDB in the browser or React-Native-compatible SQLite or whatever.
I'm not sure how to effectively proceed with a persistent implementation of Datomic in the browser without the bad options of either 1) magically cobbling together a bridge between IndexedDB and DataScript (low likelihood of straightforward progress) or 2) using a plugin for Mentat:Rust (I'm set on avoiding plugins for what I'm doing).
A persistent implementation of Datomic seems more straightforward with React Native, in which custom native modules are possible (cue Mentat:Rust), but (referencing the link you gave me) there would apparently be issues with the "database as a value" during rendering with any asynchronous persistence implementation, and my guess is that a React-Native-ready Mentat:Rust would require asynchrony just like the other options.
@rnewman :
I definitely feel your pain about debugging in ClojureScript. ClojureScript exceptions in the browser can be extremely unhelpful. React Native exceptions are essentially entirely opaque without some version of e.g. printf as you said; this is certainly also the case even with inadequately-source-mapped ClojureScript in a non-React-Native context (e.g. Node). And as to the rebuild, I thought you were talking about just the advanced compilation build stage usually used for (production) deployment. I, too, have experienced the pain of a "70-second rebuild with each print statement" — actually, probably more like 150, depending on how deeply down the dependency graph I changed something. This is even with Figwheel, which does incremental compilation, as I'm sure you know — Bruce Hauman, the creator of Figwheel, was on ScreenHero with me the other day and was shocked at how long it took. I've had to forego the convenience of having all my code auto-rebuildable and do a lein install on the source paths I wanted to exclude from auto-rebuilding. To this day, after working on the backend in Clojure for a while, I dread going back to ClojureScript land where it seems that after creating a certain critical mass of code, my progress is effectively halted by the prohibitive (even incremental) recompilation time.
I've never tried Rust, but I'm glad that you've found a good fit with it!

@rnewman :
I've never really researched why WebSQL was deprecated in favor of IndexedDB (something about wanting multiple storage implementations, i.e. not just SQLite?), but it seems a shame, because yes, all those layers of abstraction really do seem wasteful, as you say. If WebSQL weren't deprecated, would you say you might even recommend using that as Mentat's storage backend in the browser?

I can't say I'm on board with your reasoning for switching away from ClojureScript, as "an opaque 600KB lump of compiled JS to debug", as you say, could apply to probably many libraries of comparable size, especially ones which use a feature-rich compile-to-JS-language like ClojureScript. I'm not sure that the build/optimization step is as "inscrutable" as you say, or why 60 seconds of compilation/optimization time is an abomination when it often takes e.g. C++ code tens or hundreds of times that to achieve the same result (ignoring incremental builds, which ClojureScript also has via e.g. Figwheel).

600KB is the optimized version. IIRC the unoptimized single-file version was 6MB, which is about as big as Firefox 3's zip. (Devtools wasn't too happy about a few hundred thousand lines of code.) And this from an incomplete early version of the library, only 6500 lines of Clojure. I dread to think how big it would have got with a fully fleshed out query engine, good logging and performance tracing, etc.
The compiled output was also scarily bad: duplicate argument names within the same JS function, for example. And Closure Compiler is clearly intended to target a web context: it typically depends on run-time user-agent sniffing in navigator to choose how to implement features. This is fragile stuff, and not well suited to our needs.
For comparison: my machine builds all of Firefox — something like six million lines of code — in fifteen minutes, and can do an artifact build of Firefox for Android (300K lines of Java) in a minute or two. Spending 70 seconds just to produce Mentat was baffling.
The idea that we might get a crash or a bug in release and have essentially no usable stack was a strong disincentive. And if someone who's been writing Common Lisp and Clojure since some of his coworkers were in grade school can introduce subtle bugs¹ that take hours to track down, I dread to think what coworkers who are used to other languages might experience.
¹ My 'favorite' was accidentally transposing the arguments to put! inside an error handler within a go block. An unrelated change somewhere else caused an error like does not implement IChan error with no useful stack. Hooray for dynamic typing!

I've never really researched why WebSQL was deprecated in favor of IndexedDB (something about wanting multiple storage implementations, i.e. not just SQLite?), but it seems a shame, because yes, all those layers of abstraction really do seem wasteful, as you say. If WebSQL weren't deprecated, would you say you might even recommend using that as Mentat's storage backend in the browser?

Web standards mostly don't encourage standardizing a single, specific implementation; from that perspective I agree with deprecating WebSQL. If it were possible for web content to run arbitrary SQL on a database, with control over transaction boundaries, then yes, one could absolutely use it as a storage backend for Mentat.

@rnewman :

IIRC the unoptimized single-file version was 6MB, which is about as big as Firefox 3's zip.

I agree that that's crazy (though I've had ridiculous code sizes in my own experience too, like ~18MB unoptimized). That said, the "only 6500 lines of Clojure" probably required several libraries, not the least of which was the Google Closure Library. Too bad Closure's tree-shaking claims don't really hold up here, probably because of the dynamism inherent to ClojureScript.

duplicate argument names within the same JS function, for example.

I've never seen that. That example in particular is definitely fragile and scary as you say. Closure Compiler depending on "run-time user-agent sniffing in navigator to choose how to implement features" seems pretty broken to me... not sure why the maintainers haven't fixed that "small detail".
Fifteen minutes to build six million lines of code is amazing — you're right. With a comparison like that, it makes sense that 70 seconds is more than a bit much unless you're e.g. macroexpanding the world.
As to your footnote (I liked that you had a footnote by the way haha) I stood staunchly by dynamic typing for a long time until a few months ago when I finally got tired of the age-old process of looking at functions, wondering what precise shape the inputs take, looking at docstrings, analyzing the code, and experimenting in the REPL to find out "for sure". Over the course of my Clojure experience, that alone must have added up to hundreds or even thousands of hours in all. I love core.spec's approach, because it goes beyond static typing to provide guarantees that static typing, to my knowledge, can't (or at least, not without creating a combinatorial explosion of types). That said, something like Haskell's type system has always intrigued me, as finding bugs at compilation time is orders of magnitude better than finding them at runtime, or worse, in production.

seems pretty broken to me…

The CC maintainers are very change averse (each potential change gets tested internally for its Google app consumers before being approved), and CC's primary audience is multi-browser web JS, so I chalk this up to "ain't our use case", which is fair enough. But it does reinforce the feeling of not having the tools on our side!

That said, something like Haskell's type system has always intrigued me…

We've been very pleased with Swift for our iOS work; it has a very rich type system, and we've built some things with it that would have been very buggy or hard to navigate if written in JS.
Rust ain't Haskell — it chooses predictability, stack allocation (particularly of return types), and performance over expressiveness — but they're both worlds ahead of having no type checker at all.

One thing that stands out as suspect to me here: Are you really planning on compiling arbitrary datalog queries directly to SQL? My understanding is that datalog is strictly more expressive (particularly as relates to rules, recursion and such), and that this would therefor be impossible.  Given that, it seams like you'd have to implement some components of a more classical datalog system, or only support a sort of "SQL compatible" subset of datalog. Assuming the former, I imagine you could translate whatever portions of a datalog query you could into SQL, then implement the rest in terms of those intermediate results/relations using standard techniques for executing datalog queries.
With that in mind, and the comments above on problems with transactionality and asynchrony, I'm a bit confused why something more analogous to the Datomic's model of evaluation wouldn't make more sense, and open up more room for generality at the storage level?

All of the queries we've written so far are compiled into a single SQL query, yes. Modern SQL supports recursive CTEs, and SQLite supports user-defined functions, so we have a lot of flexibility… but we haven't even needed that yet.
and/or joins, filters, bindings, fulltext, and aggregates can all be translated into a non-recursive SQL query.
Some features, like pull expressions within :find queries, might be most easily implemented in a two-stage execution process. (Many simple pull expressions can also be implemented as a stream rollup on a single SQL query, of course.)
Other features like user-defined rules might require iterative or recursive query plans, using temporary tables or in-memory joins. We haven't got to that point yet.
The query executor is able to control transaction boundaries, and will have exclusive ownership of a connection, so SQLite takes care of isolation. Queries can be as asynchronous and iterative as we like, and the right result will drop out at the end.
Datomic's model is essentially: replicate index chunks over a network, and locally interrogate the index chunks to get answers. It's very memory intensive, and it's designed for the lowest common denominator of storage.
Doing so loses the principle advantage we have as an embedded data store, which is taking full advantage of SQLite and its excellent page cache, VFSes, and isolation. We absolutely could build a chunk-based system and write all of the stats, query planner, explainer, metaindex primitives, and index walks ourselves — it would take us longer to build, it would run queries more slowly, it would be buggy, and we would still use SQLite as a backing store. Perhaps one day we'll decide to spend the time on our own storage backend; I wouldn't bet on it.
(I spent several years implementing parts of AllegroGraph, which is a chunk-based tuple storage system, so I have some familiarity with this.)

On Thu, Jan 19, 2017 at 12:01 PM, Christopher Small <
<EMAIL_ADDRESS>wrote:

One thing that stands out as suspect to me here: Are you really planning
on compiling arbitrary datalog queries directly to SQL? My understanding is
that datalog is strictly more expressive (particularly as relates to rules,
recursion and such), and that this would therefor be impossible.

I cannot find a reference right now, but it is a theorem that Datalog with
appropriate recursion restrictions is equivalent to SQL with certain
recursion extensions.  We can easily target these subsets and get what we
need.

@rnewman @ncalexan Thank you both for the clarifications. I had never encountered SQL+recursion, but intuitively it makes sense that would do the trick.
I am saddened to see that things don't look so peachy for a durable, client side datomic/datascript/datomish/whatever clone coming out of Mozilla. But I'm hopeful and encouraged that your work is pushing the general model further into the mainstream.
Cheers


Update slack-app-backend to 1.38.2

About this PR
📦 Updates com.slack.api:slack-app-backend from 1.30.0 to 1.38.2
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.slack.api", artifactId = "slack-app-backend" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.slack.api", artifactId = "slack-app-backend" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #163.


improve UX of update form (review & suggestions needed)

I'll try to test more locally, but I have a question/concern about not allowing submitting when some of the montonicity concerns failed.
Say a person arrives at this form and for some reason (e.g. the previous submission had a mistake) sees cumulated values higher than ones they want to report.
Currently the only course of actions possible is to keep them at previous value regardless.
Wouldn't it be better to show a big warning if they are decreasing (as it's done now), but nevertheless allow submitting data, even if it's not consistent with previous submission? Otherwise this might lead to a bias in collected data that would be even harder to correct in the end?

very clever. i didn't think about that scenario! i will change the error to a warning. hopefully i made it very clear that the values are cumulative. thanks for that great insight @rth

Maybe let's wait for a second opinion before changing it? I might be missing something in how this form is typically filled.

i think you're entirely right in your last comment @rth
here's the new version. no disabling of the submit button now.

here's the new version. no disabling of the submit button now.

Looks good. Maybe repeat "décès (cummulés)" and "transferts (cummulés)" in the error message just to be more explicit about why this is an issue?


Section - Updating a Linked Deployment Share Using Windows PowerShell

In Section
Updating Deployment Media Using Windows PowerShell
Step 4
4. Update the deployment share using the Update-MDTDeploymentShare cmdlet, as shown in the following example:
The example says
Update-MDTLinkedDS -Path "DS002:\Linked Deployment Shares\LINKED002"
rather than
Update-MDTDeploymentShare  -Path "DS002:\Linked Deployment Shares\LINKED002"
So either your step is wrong or your example is wrong, given its the same example in section - Updating a Linked Deployment Share Using Windows PowerShell

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6fc2393d-6a81-c9c9-87bb-b40b66f26d68
Version Independent ID: fda85f4c-33ec-d0aa-b793-381d2bebb57d
Content: MDT samples - Microsoft Deployment Toolkit
Content Source: sccm/mdt/samples-guide.md
Product: configuration-manager
Technology: configmgr-osd
GitHub Login: @aczechowski
Microsoft Alias: aaroncz

Update-MDTLinkedDS -Path "DS002:\Linked Deployment Shares\LINKED002" is the example in the section - Updating Deployment Media Using Windows PowerShell.
I put the wrong title section

Hi BlackV - you're welcome to submit the edit yourself, 😄 for more info see How to contribute. Otherwise I've tagged this issue for help from the community.

Fair enough. I'll have a look later on

Thanks again for the feedback. We recently moved this content to a new GitHub repository, so this issue is no longer tracked with the original article. As part of that process, we reviewed our backlog of feedback, and made some tough decisions on items that we don't think we'll realistically ever address. If you're passionate about this specific feedback, please refile it on the current article, now under docs.microsoft.com/mem/configmgr.


Orbital Data Collection VM - Public IP Address?

In the documentation "Prepare your virtual machine (VM) and network to receive AQUA data" it states "Ensure that the VM has at least one standard public IP".
https://learn.microsoft.com/en-us/azure/orbital/downlink-aqua#prepare-your-virtual-machine-vm-and-network-to-receive-aqua-data
Is this correct given that in the Contact Profile Documentation:
https://learn.microsoft.com/en-us/azure/orbital/downlink-aqua#configure-a-contact-profile-for-an-aqua-downlink-mission
It states "Enter the Private IP address of the virtual machine you created above (VM)".

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e70189d1-5eab-f552-144c-d4b2fcb5ef23
Version Independent ID: 62ab5f8e-8512-9479-2276-469d3bce7369
Content: Schedule a contact with NASA's AQUA public satellite using Azure Orbital Ground Station service
Content Source: articles/orbital/downlink-aqua.md
Service: orbital
GitHub Login: @wamota
Microsoft Alias: wamota

@mattweale
Thanks for your feedback! We have assigned the issue to author and will provide further updates.

#please-close


include ssl port in AppDashboard /apps/json route

CI build: https://ocd.appscale.com:8080/job/Daily Build/1155/

rake now passes all tests successfully.

Thanks for updating @selfcommit
Latest build: https://ocd.appscale.com:8080/job/Daily Build/1162/


SAK-29192 Chat messages refresh correctly. updateReq set correctly.

updateReq set correctly. to null now.
Third times the charm for this one...

Thanks.


feat: add mailparse 3.1.6 release

update mailparse with release 3.1.6

:tada: This PR is included in version 1.181.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


⚠ Bofrost.be has degraded performance

In 0e7c637, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1981 ms

Resolved: Bofrost.be performance has improved in 00ac2a9 after 13 minutes.


Support for v0.5?

Is there going to be an upgrade to slate v0.5? Really hoping so. Cheers

100% upvote. These plugins look amazing! Would love to use them if they can be upgraded to v0.5


Begin migration from Guava Cache to Caffeine

Caffeine is apparently strictly superior to the older Guava Cache (and is even
recommended in lieu of Guava Cache on Guava Cache's own documentation).
This adds the relevant dependencies and switch over just a single call site to
use the new Caffeine cache. It also implements a new pattern, asynchronously
refreshing the cache value starting from half of our configuration time. For
frequently accessed entities this will allow us to NEVER block on a load, as it
will be asynchronously refreshed in the background long before it ever expires
synchronously during a read operation.

This change is

It looks like the mirror repos haven't been updated.


Make sure settings, including example.settings.json, are refactured to match settings page PR structure

The Settings Page feature made changes to the way settings are handled. As a follow up to that task, we need to check that all settings are updated in throughout our app, including the example.settings.json, to match the new structure.
Specifically:

apinf
apiUmbrella
elasticSearch
mail
contactForm
girubConfiguration

This issue is related to the review process for @elnzv issue #327

@jykae Would you be interested in this task?

@apinf/developers Who wants to volunteer for this task?

Checked. That have already been done.
See this commit:
https://github.com/apinf/api-umbrella-dashboard/commit/bd3e1e71e213186b4ec77ca937565694939a2269
@apinf/developers, please confirm and close this issue.

OK, go ahead and close.


Search SDK: Hotfixing GA SDK version to 1.1.3

We are retroactively applying the fix for this issue to the GA SDK: https://github.com/Azure/azure-sdk-for-net/issues/2434
FYI @chaosrealm @seansaleh @pablocastro

CI build failed because this branch corresponds to an older version of the build environment. Merging anyway.


conductor-server failed to start

Describe the bug
Going through the instructions from here https://conductor.netflix.com/devguide/running/docker.html
On docker-compose-up conductor-server failed to start.
Details
Nothing special. Just trying to launch a simple default configuration
To Reproduce
Execute the next commands in CLI:

git clone https://github.com/Netflix/conductor.git
cd conductor/docker
docker-compose build
docker-compose up
conductor-server container exited with the code 2
output is:
: not found10:14:09 /app/startup.sh: line 14:
: not found10:14:09 /app/startup.sh: line 16:
: not found10:14:09 /app/startup.sh: line 18:
: not found10:14:09 /app/startup.sh: line 21: nginx
: not found10:14:09 /app/startup.sh: line 22:
: No such file or directoryartup.sh: cd: line 24: can't cd to /app/libs
: not found10:14:09 /app/startup.sh: line 28:
2023-10-07 10:14:09 /app/startup.sh: line 41: syntax error: unexpected end of file (expecting "then")
2023-10-07 10:14:09 Starting Conductor server
2023-10-07 10:14:09 Running Nginx in background
2023-10-07 10:14:09 Property file: config-redis.properties
2023-10-07 10:14:09 config-redis.properties

Expected behavior
conductor-server container started successfully.
Screenshots
Additional context
Doing this on Windows 10
Docker Desktop 4.24.1 (123237)

@amid0 its related to the .gitattributes file included in the repo.
change your steps to this

git init (in your working folder)
Add a .gitattributes file with the following in it

gradlew eol=lf
*.gradle eol=lf
*.java eol=lf
*.groovy eol=lf
spring.factories eol=lf
*.sh eol=lf

git add *
git commit -m 'git attributes'
git pull --allow-unrelated-histories https://github.com/Netflix/conductor.git
cd conductor/docker
docker compose build
docker-compose up

The important part is the .gitattributes file, it leaves the line endings of the files as unchanged when pulling the source.

@Robban1980  Thank you a lot.
In such a way, I'm able to run conductor.
I think It is better to add such information to the installation documentation.


Bahri Alabey About Me Revision

Bahri Alabey about me page revision according to About Me template .

Revision of the about me page of Bahri Alabey is done. Bahri Alabey About Me


C

Create some simple project using C

Hey there!
I would like to work on this issue. Kindly assign this to me.

yes create a folder and make changes


Architectural toolkit

support for architectural elements within a-frame
scene structure to A-Frame conversion ( lots of fixes )
A-Frame elements to scene structure conversion
scene structure validation playground
minimap component
SVG floor plan export

massive!! :) will review tomorrow

I've started using the 3dio build from the architectural-toolkit branch, because it fixes some floor display problems. 🤘 Let me know please before you delete branch.

suggestions:

[ ] rename to "architecture toolkit"
[ ] add "v" property for versioning to every element
[ ] make complex array params consistent in aframe (i.e. https://github.com/archilogic-com/3dio-js/blob/0191bf01071d07477d40b3fc631dc118038b7e23/src/scene/structure/validate/by-type/polyfloor.js#L16)
[ ] gulp jshint trows warnings about undefined ÌO3D`variable in multiple files (i.e. closet.js)


Build for .NET 4.5 does not work, fails to find/restore packages

To reproduce:

git clone
open zipkin4net.sln in Visual Studio (2017)
build

I see packages.config files but they are not part of the solution. I added them to get around the build issue but for some reason does not work.

OK, this is fixed in my new PR.

OK, I am re-opening this for coordination. Can you please respond here rather than in the mother thread.

@aliostad Thanks for reporting this issue. I'm currently working on a solution where net45 is available for 2017 csproj but travis will still compile netstandard until we find a solution.
Nuget can therefore be published supporting net45 and netstandard

@aliostad It works on my VS2017 with this fix. Can you test ?

What I care about most is command line build and having nuget. Does it work on those?

I can see you closed this issue.
I checked and build works but does not create 4.5 binaries. The PR I sent was generating them. Am I missing something?

@aliostad The PR allows to generate a nuget with both net standard and net45 supported. I will release a new nuget tomorrow as I don't have a windows machine at home. Isn't that what you needed ?

Nuget published with version 0.4.1

Thanks a lot.
Fine with that PR, my point was that if you want to pick some of that work.
Yeah, I can work on a command line build for windows and .NET 4.5. I will create an issue.
Cheers

Please come back to us if you need anything related to this issue.


sub-data sections

I have been toying with LDPL for a while now, and one of the things that irks me the most is the sheer amount of variables you need to create in order to hold stuff and not override the data of one of the other methods.
my proposed solution to this is SUB-DATA sections. I am proposing that before every SUB-PROCEDURE you will be able to declare an optional SUB-DATA section. all variables declared inside a SUB-DATA section act like local variables. they are only usable inside a SUB-PROCEDURE and (optionally) are not reset after every call to the SUB-PROCEDURE (in order to be compatible with the whole "variables are persistent between scopes thing the language has going on already) . they also should not have any naming conflicts with other internal variables (of other sub-procedures).
example:
DATA:
    i is number
PROCEDURE:
    SUB-DATA:
        x is number
    SUB-PROCEDURE foo
       incr x
       display x " "
end SUB-PROCEDURE
while is less than 10 do
    call foo
repeat
#stdout should be: 0 1 2 3 4 5 6 7 8 9

tl;dr: persistent local variables

Hi there! This is something we considered at one point, but I can't remember why we didn't go through with it. @dvkt, @dgarroDC what do you think about this?

I was thinking that this could be maybe coupled with a way of passing parameters to sub-procedures, something like:
SUB PROCEDURE foo
    PARAMETERS:
      myNumber IS NUMBER
      otherNumber IS NUMBER
    LOCAL DATA:
      bar IS NUMBER
    PROCEDURE:
      ACCEPT bar
      ADD bar AND myNumber IN bar
      DIVIDE bar BY otherNumber IN bar
      DISPLAY bar CRLF
END SUB-PROCEDURE

CALL foo WITH 17 -8

What do you all think?

I was thinking that maybe this would lead to easier to implement libraries than the COBOL based model we previously proposed, @dvkt. We could just write a "library writing standard" and state that is good practice to name the global libraries inside a library as <libname>.<variablename> and create an IMPORT statement.

I was thinking that this could be maybe coupled with a way of passing parameters to sub-procedures, something like:
SUB PROCEDURE foo
    PARAMETERS:
      myNumber IS NUMBER
      otherNumber IS NUMBER
    LOCAL DATA:
      bar IS NUMBER
    PROCEDURE:
      ACCEPT bar
      ADD bar AND myNumber IN bar
      DIVIDE bar BY otherNumber IN bar
      DISPLAY bar CRLF
END SUB-PROCEDURE

CALL foo WITH 17 -8

What do you all think?

I think this would be a great addition to the language!


Sonobuoy v0.53.2 : Plugin timeout while waiting for results so there are no results

What steps did you take and what happened:
I pulled/tagged/ pushed the sonobuoy images to bcmt-registry:5000
And created the custom repo file
cat << EOF > /root/custom-repo-config.yaml
buildImageRegistry: bcmt-registry:5000
dockerGluster: bcmt-registry:5000
dockerLibraryRegistry: bcmt-registry:5000
e2eRegistry: bcmt-registry:5000
e2eVolumeRegistry: bcmt-registry:5000
gcRegistry: bcmt-registry:5000
gcEtcdRegistry: bcmt-registry:5000
promoterE2eRegistry: bcmt-registry:5000
sigStorageRegistry: bcmt-registry:5000
EOF
Then I ran sonobuoy as following:
sonobuoy run --sonobuoy-image bcmt-registry:5000/sonobuoy:v0.53.2
--kube-conformance-image bcmt-registry:5000/conformance:v1.21.0
--systemd-logs-image bcmt-registry:5000/systemd-logs:v0.3
--plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=is_control,is_edge"
--e2e-repo-config /root/custom-repo-config.yaml
--mode=certified-conformance
The test ran for couple of hours and after that i collected the results
[root@eindhoven-v331-allinone-0 ~ (Active)]# outfile=$(sonobuoy retrieve)
[root@eindhoven-v331-allinone-0 ~ (Active)]# sonobuoy results --mode detailed --plugin e2e $outfile |  jq '.  | select(.status ==                                                                       "failed") | .details'
parse error: Invalid numeric literal at line 1, column 11
[root@eindhoven-v331-allinone-0 ~ (Active)]# sonobuoy results --mode detailed --plugin e2e $outfile
e2e|Plugin timeout while waiting for results so there are no results. Check pod logs or other cluster details for more informatio                                                                      n as to why this occurred. {"error":"Plugin timeout while waiting for results so there are no results. Check pod logs or other cl                                                                      uster details for more information as to why this occurred."}[root@eindhoven-v331-allinone-0 ~ (Active)]#
[root@eindhoven-v331-allinone-0 ~ (Active)]#
What did you expect to happen:
Can't check the sonobuoy results from the tarball and i found the following error message:
{"error":"Plugin timeout while waiting for results so there are no results. Check pod logs or other cluster details for more information as to why this occurred."}
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

Sonobuoy version:
[root@eindhoven-v331-allinone-0 ~ (Active)]# sonobuoy version
Sonobuoy Version: v0.53.2
MinimumKubeVersion: 1.17.0
MaximumKubeVersion: 1.99.99
GitSHA: 981a3ffd4368600eb1a5bca3f12a251e80895d37
API Version check skipped due to missing --kubeconfig or other error
[root@eindhoven-v331-allinone-0 ~ (Active)]#

Kubernetes version: (use kubectl version):
[root@eindhoven-v331-allinone-0 ~ (Active)]# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
[root@eindhoven-v331-allinone-0 ~ (Active)]#

Kubernetes installer & version:

Cloud provider or hardware configuration:

OS (e.g. from /etc/os-release):
[root@eindhoven-v331-allinone-0 ~ (Active)]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
You have new mail in /var/spool/mail/root
[root@eindhoven-v331-allinone-0 ~ (Active)]#

Sonobuoy tarball (which contains * below)

I'm not sure what you're asking for exactly. As the message says: "Plugin timeout while waiting for results so there are no results. Check pod logs or other cluster details for more information as to why this occurred."
Have you checked those logs? Generally, this isn't a problem with sonobuoy but a problem with the cluster causing tests to run long or hang.
If you want to post your tarball I can peek at it for you but as it stands this isnt a bug as much as it is just a typical error case. The reason we timeout at all is so that we see this error and can realize there is a problem instead of the run stalling entirely, not getting the tarball to investigate, and eating up resources/time.
If you want you can specify --timeout=999999999 or something like that (check the help text to see if thats seconds or minutes) to extend the timeout. However, I think the timeout was extended a while back and isn't typically hit without an underlying problem.

Closed due to inactivity. Feel free to reopen if you had more information to debug.


The bot can't listen channels

From this morning's update (26 Jul 2015) the bot can't listen channels yet in the both versions of the game - Source 1 and Source 2.

It's my bad. Nobody can listen me in the chat channels - I don't know how and why, and why I don't have my Compendium too...


[AutoDiff] Part 1: Add support for delayed gradients in AD configuration.

(Split from megapatch #17901)
This patch adds support for delayed gradients in AD configuration and merges [seedable], [preserving_result] and [delayed] to an option set SILGradientOptions. This change is reflected on the differential operator in SIL, aka. the gradient instruction.
%0 = function_ref @foo : $(Float, Float) -> Float
%1 = gradient [source 0] [wrt 0, 1] [seedable] [preserving_result] [delayed] %0 : $(Float, Float) -> Float
// %1 has type $(Float, Float) -> (Float, @convention(thin) (Float) -> (Float, Float))

A seedable, result-preserving, delayed gradient takes the original parameters of function @foo, and returns a tuple of the original results and a thin closure that takes a seed and returns the partial derivatives.
TODO: The support for delayed gradient isn't complete. We need to change the canonical gradient options to include SILGradientOptions::Delayed, and change Differentiation pass to synthesize calls to the new canonical gradient.
Also included:

NFC: Use a llvm::BitVector to store AD parameter indices.
Other NFC cleanup.

@swift-ci Please test tensorflow

Thx. Haven't adapted the AD pass yet. Hang on a sec...

@swift-ci please test tensorflow

@swift-ci please test tensorflow

@swift-ci please test tensorflow

@swift-ci please test tensorflow

@swift-ci please test tensorflow

@swift-ci please test tensorflow

@swift-ci please test tensorflow

Can't trigger CI. Starting another PR.


[QUERY]which service can use to query device list information from IoT Hub.

Hi,
We need to get a list of devices from the IoT Hub with details like capabilities and scope. What would be the right service to use from azure java sdk?
Thanks,
Prateek

Thank you for opening this issue! We are routing it to the appropriate team for follow up.
I did a quick search and found the azure-iot-sdk-java repo which may hold the answers for this.  However, I'm unsure, so I'm sending this for Java Client team review instead of moving the issue there.

Hi @pratson20, I believe the correct SDK reference is available at https://github.com/Azure/azure-iot-sdk-java#api-reference.
/cc @myagley

@pratson20 I'm closing this issue now.  Feel free to comment and reactivate if there are follow on questions or I misunderstood the original question.  Thanks.


Unexpectedly short alignment lengths using alignall

Expected Behavior
Workflow:
cluster -> alignall -> createtsv
I noticed there are cluster members in my output with very short alignment lengths. I was expecting the aligned length to be at least 50% of the cluster representative length, but that appears not to be the case.
Current Behavior
With the evalue cutoff for alignall set to a very large number, I'm getting really short alignment lengths that make me confused about how these cluster members were assigned to the cluster representative in the first place. For example on the first line below, the aligned length of query and target for the cluster representative 1288495_4011 and cluster member 758793_4580 appear to be 4 amino acids, which is much shorter than 0.5*65 = 32.5
1288495_4011 1288495_4011 758793_4580 12 0.600 3.362E+08 78 82 120 41 45 65
1288495_4011 1288495_4011 458836_4783 12 0.227 3.362E+08 19 40 120 0 21 49
1288495_4011 1288495_4011 1777137_4763 12 0.666 3.362E+08 25 30 120 57 62 66
1288495_4011 1288495_4011 1968433_3027 13 0.363 1.888E+08 52 71 120 71 92 372
1288495_4011 1288495_4011 268404_3908 14 0.444 1.059E+08 23 31 120 166 174 379

(showing only 5 lines for brevity)
Steps to Reproduce (for bugs)
Please make sure to execute the reproduction steps with newly recreated and empty tmp folders.
mmseqs createdb protfam.fasta protfamDB
mmseqs cluster --threads 24 --min-seq-id 0.4 -c 0.5 --cov-mode 2 --cluster-mode 0 protfamDB clusterDB tmp
mmseqs createsubdb <(printf '8277758') clusterDB 1288495_4011.subdb
mmseqs alignall protfamDB 1288495_4011.subdb 1288495_4011.alnDB --comp-bias-corr 0 --cov-mode 2 --alignment-mode 3 --threads 24 -e<PHONE_NUMBER>740992
mmseqs createtsv protfamDB protfamDB 1288495_4011.alnDB 1288495_4011.alnDB.tsv
OFS="\t" awk 'FNR==NR{a[$1]=$2;next}{$3=a[$3]; print}' protfamDB.lookup 1288495_4011.alnDB.tsv > 1288495_4011_renamed.tsv
awk -F' ' '!seen[$2,$3]++ && !seen[$3,$2]++' 1288495_4011_renamed.tsv > 1288495_4011_dedupe.tsv
sort -grk6,6 1288495_4011_dedupe.tsv | awk '$2=="1288495_4011" {print $0}'

Your Environment
Include as many relevant details about the environment you experienced the bug in.

Git commit used (The string after "MMseqs Version:" when you execute MMseqs without any parameters):
Which MMseqs version was used (Statically-compiled, self-compiled, Homebrew, etc.):
anaconda:
mmseqs2                   13.45111             h95f258a_1    bioconda
For self-compiled and Homebrew: Compiler and Cmake versions used and their invocation:
Server specifications (especially CPU support for AVX2/SSE and amount of system memory):
Operating system and version:

Can you try clustering with the --cluster-reassign Parameter?
due to the cascaded cluster algorithm, clustern can end up merged that have members below the clustering thresholds. This parameter fixes the clustering afterwards.
the paarwise alignments among cluster members that alignall produces can be much more dissimilar though.

Thanks for the quick response.
I've tried clustering again with --cluster-reassign 1, but am still getting alignment lengths that appear to violate the clustering criteria, unless I'm misunderstanding something:
mmseqs cluster protfamDB clusterDB_reassign ~/scratch/tmp -s 6 --threads 24 --min-seq-id 0.4 -c 0.5 --cov-mode 2 --cluster-mode 0 --cluster-reassign 1
mmseqs align protfamDB protfamDB clusterDB_reassign alnDB_reassign -a --threads 24 --comp-bias-corr 0 -e<PHONE_NUMBER>740992
mmseqs convertalis protfamDB protfamDB alnDB_reassign aln_to_clustreps.m8
awk '$4<50{print}' aln_to_clustreps.m8 | sort -nk4,4 | head

45656_3631      1137799_3214    0.950   20      1       0       79      98      423     442     2.607E-05       51
45656_3631      217204_6534     0.809   21      4       0       79      99      695     715     1.266E-04       49
45656_3631      2571105_3757    0.761   21      5       0       79      99      695     715     3.266E-04       48
45656_3631      668_9026        0.761   21      5       0       79      99      714     734     1.155E-03       46
489_1270        489_2691        1.000   21      0       0       1       21      1       21      3.857E-05       50
33968_38        33968_2958      0.833   24      4       0       92      115     128     151     2.956E-04       48
1550024_4392    1608583_1648    0.720   25      7       0       6       30      7       31      2.660E-04       48
2527978_7844    2527978_861     0.920   25      2       0       1       25      1       25      1.690E-05       52
45656_3631      191863_3100     0.880   25      3       0       75      99      282     306     8.777E-08       58
1128665_3036    709876_1980     0.653   26      9       0       43      68      211     236     2.419E-03       45

For example, in the first row of output above, the alignment length is 20 residues for a query sequence that is 100 residues long. My clustering parameters (-c 0.5 --cov-mode 2) should prevent a target sequence w/ alignment length <50 residues from joining this cluster, right?
On a related note, I noted this line in my clustering output log:
Combining cluster mode 0 in combination with coverage mode 2 can produce wrong results.
Please use --cov-mode 2

And found this comment from a previous issue:
The cluster mode 0 (set cover) requires a bi-directional coverage criteria. If it is used with unidirectional coverage than it pick representative sequences that do not fulfill the clustering criteria anymore.
Originally posted by @martin-steinegger in https://github.com/soedinglab/MMseqs2/issues/218#issuecomment-580978693
It seems like my choice of parameters is at fault, and that I would probably have clusters that more closely obey the criteria if I used cov-mode 0 and cluster-mode 0 (set cover, query AND target coverage), or cov-mode 1 and cluster-mode 2 (CD-hit-like, target coverage). Am I understanding correctly?


FreedomTex works with 1.0.2

Yet another .netkan hack to mark that mod works with new version of ksp :+1:

I'm going to be that guy. Could you also filter the .DS_Store file?

Here you are :)


API key guide redirects to "Page Not Found"

Specifically, this link https://docs.hiro.so/api/how-to-use-api-key redirects to a 404.

https://github.com/hirosystems/docs/pull/447.


matrix-commander: 2.37.3 -> 3.5.0

Description of changes
Update, for changelog see https://github.com/8go/matrix-commander/tags .
Relevant for pacakging: switched from xdg to pyxdg

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

@benneti can you update to latest release?

merged with master and bumped to 3.5.0 all in a new clean commit

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/581


feat: include full path in the cyclic deps error

bors r+
🤖

I think this can work with a visited set if once find a cycle you go back down the stack until you find the current node.

Yeah, I like that much better, thanks for suggestion!
bors r+

This begs for an iterative implementation :smile:.


Support new design of Triton in Python examples

Hi, I think you should support the new design of Triton in Python examples of Arybo.
https://github.com/JonathanSalwan/Triton/tree/master/src/examples/python
https://github.com/JonathanSalwan/Triton/releases

That should be okay with arybo 1.1.0!


Loss of focus

Most of the time when I press CMD + SHIFT + V and try to go through available clipboards via keyboard there's no focus and the focus is lost.

Also getting this issue across a couple of Macs -- perhaps a MacOS update affected it?

Any help on that? 👍

I downloaded Clipy, which appears to be actively maintained -- and I believe is a fork of ClipMenu -- but that also has the same issue.

Where do we go from here :)


Contribution of Keylogger made in python

Its done..I'm uploading keylogger files 👍.Kindly add the lable 😊

@SANTHOSH17-DOT It was great to contribute in your repo ..I'll try to share your repo to more cybersecurity ethusiasts...Also I would like to connect with you on some linkedIn or twitter?

@SukarayamJanjua Thank you very much for your contribution. My social media profiles are available in my GitHub profile.


What should our tests do?

What is the problem?
I can see through some PR's decisions that the definitions of tests are differents. Specially about what should journey tests cover and do.
How can this be addressed?
I believe that after a discussion a simple Markdown file inside the project could keep this definition inside the project.
Who could help with this issue?
I think that @Irio @jtemporal @cuducos @anaschwendler could contribute to this discussion.
What I want from you?
Share with us here answers for 3 questions:

What should tests do and covers in general?
What should unit-tests do and cover?
What should journey-tests do and cover?

All of it focused on Serenata Toolbox context.
Labels
[question]

Taking my first guesses:

What should tests do and covers in general?

The toolbox tests should assert that given an expected input (e.g. a fixture simulating the structure of a real world a dataset) the output is what we wanted.

What should unit-tests do and cover?

Unit tests should test each function/method, isolating it from any environment or side-effect (use and abuse of mocks, for example).

What should journey-tests do and cover?

They should depart for a controlled input (fixtures) to test the integration os all the pieces: given the input it should really create the output file, for example — without depending on any external eventuality (mock the input if it comes from a remote server, for example).

What should tests do and covers in general?

The Serenata Toolbox tests should protect us from our on mistakes providing safety to move on and help us keep everything working while we keep improving our project.

What should unit-tests do and cover?

The unit-tests should cover the smallest unit of code. Unless it covers a piece of code that don't have coverage. What happens on project that do not have 100% of code coverage and is usual.

What should journey-tests do and cover?

In my vision it has two principal values:

Test real journey based on real value but mocked
Test real journey based on real integration before we deploy to assure that everything is ok with our integration.

Test real journey based on real integration before we deploy to assure that everything is ok with our integrations.

Experience has shown us that relying on external services (specially on government one I might add) is problematic: they are unstable, they change format/encoding without warning, etc.
This argument actually  goes both ways: on the one hand we must get used to see our CI red not because we messes up, but because these external servers did it. On the other hand we must be sure the toolbox works with them before releasing a new version.
What really bugs me is that the instability is way more common than unwanted changes in their side… so we'll be blocked most of the times just because of low availability of third-parties.

Experience has shown us that relying on external services (specially on government one I might add) is problematic: they are unstable, they change format/encoding without warning, etc.

There is a advantage of monitor the external service that is know as soon as possible the changes on their side.

On the other hand we must be sure the toolbox works with them before releasing a new version.

I believe that will depends on the kind of break.

If it's for instability of external service. We should move on.
If it's for format changing or encoding. We should not move on.

so we'll be blocked most of the times just because of low availability of third-parties.

We should not be stopped by instability just for really breaking changes from the external services.

There is a advantage of monitor the external service that is know as soon as possible the changes on their side.

This in in fact important… however IMHO it's not the role expected from the test suite.

This in in fact important… however IMHO it's not the role expected from the test suite.

Which alternative do we have to keep this important point and a suite that do not have external dependencies?
I can think a pipeline like:
unit-test => mocked-integration-tests => deploy => integration-tests-with-**real-external-service**
Or  have  just:
unit-test => mocked-integration-tests => deploy
and a second stuff like status.io specific for what we have as external dependency.

The toolbox today depends on third parties to work right, or at least every time that we open up a new PR we are open to a red CI because of third parties like @cuducos said:

on the one hand we must get used to see our CI red not because we messed up, but because these external servers did it.

I think the ideal question is: How to test everything without going crazy on testing?

What should tests do and covers in general?

given an input the output is what is expected

What should unit-tests do and cover?

the minimum amount of code, e.g.: a function that translates should receive a dataset in pt-br and return a dataset in en, so the test is "given the dataset a in pt-br the output of the translate function was the dataset a in en? if so then the test pass otherwise it shouldn't

What should journey-tests do and cover?
If the whole process is working. The toolbox is a pipeline it has a beginning and a path to be followed  until the end, do all of those steps being done? and working as they should?

On the matter of integration tests: Toolbox is the point that encounters "the outside world" on the project" especially considering Rosie. We definitely need some integration tests. But the integration should more raise an alert out than block us from improving our toolbox.

What should tests do and covers in general?
The tests must cover all the inputs generating expected outputs, but not only that, I think that we need to think about the behavior and use the tests to try easier ways to perform the system as it is.

What should unit-tests do and cover?
Little slices of code, as @jtemporal said.

What should journey-tests do and cover?
The behavior that toolbox should perform to other repos. But at this point I don't know if we should work for a repo that make everything works, or we still work on separate repos.

The big refactor is to fix things and make them easier, so I can't think only about toolbox when I ask myself those questions

@cuducos' comment summarizes well my view on the topic.
When removing "real integration" tests, we lose the ability to know when the Congress changes its file formats or becomes offline (which happened often in the past months, affecting our project in multiple instances). They could be replaced by a simple status page, as mentioned before, with online status (on/off) and some sanity checks on the data (e.g. is a zip file? contains a csv? has a given list of columns? has more rows than the last update?).

My conclusion with this is that even that when we have an instability from externals we should move on without this checks. Like a nice to have but not a must have.

Yes… as far as I got these checks are not part of automated tests. They're something else ; )

Archived by inactivity — anyone is free to re-open when needed ; )


How to read the sequence of images and obtained the gaussin?

I have a set of images which is a number of frames of video. How to read it and work on it? Thanks I have script and code in Matlab, but I am not expert in python so want to learn it.

Hi,
I find that the question is not very clear. However, you can have at the README to have an idea on how to use the algorithm. In particular
If you want to denoise an image sequence using the pretrained model you can execute

python test_fastdvdnet.py \
	--test_path <path_to_input_sequence> \
	--noise_sigma 30 \
	--save_path results
NOTES

The image sequence should be stored under <path_to_input_sequence>
The model has been trained for values of noise in [5, 55]
run with --no_gpu to run on CPU instead of GPU
run with --save_noisy to save noisy frames
set max_num_fr_per_seq to set the max number of frames to load per sequence
to denoise clipped AWGN run with --model_file model_clipped_noise.pth
run with --help to see details on all input parameters


WIP: Add how to for requesting generic permission

Generic permission should allow the users to request any available device permission.

yet to be released


[nnx] fix custom_vjp

What does this PR do?

Solves the issue where FwdFn is called outside of an update context. When this happens FwdFn behaves as a pure function.
Correctly passes the non differentiable states as captures instead of using global state.

Solves #4265.

Hi @cgarciae - I'm excited to try this fix!
Just wanted to mention a slight discrepancy in jax vs. nnx custom_vjp signatures that are expected:
nnx bwd:  res, (ins_g, outs_g) -> tangent
jax bwd:  res, outs_g -> tangent
The docs here seem t obe showing an example of jax.custom_vjp.

@hrbigelow this is correct! I expanded a bit more on the docs to reflect this.
BTW: thanks for the wait. I've been trying to get this transform right but its taken some rewrites to solve some edge cases.


Newsletter signup

Can we have a CTA for the newsletter where the signup field is not hidden behind a click or a tap (see image below)?
Furthermore, we had the following use-case last week: when sending out an all-staff email to get people to subscribe to the newsletter, the easiest direction was to get them to click on the 'hand icon'.
Is there a way we can embed the newsletter signup onto a unique page, like a 'Basic Page'? This way there will be a unique URL we can send people to. Thoughts?
Newsletter--unfurled:

I can implement the CTA today, two options:

The element is as it is now with the added ability to set a 'default' state. This would mean sign up input can be shown by default, but you would still be able to click the cross and go back—as in the current implementation.
The sign up input is shown with no cross/option to 'go back'. In this case, do you need text above the input or will the title suffice as a descriptor?

In either case, should there be an option for a social media CTA too?
For the dedicated page, do you have an example of what this might look like, or what other content it may contain?

CTA
@rdunk I think the first option offers the most flexibility. To confirm:

The element could be either state—the current display or the sign-up box shown by default.
If the sign-up box is on display only, the cross will take you back to the current layout.

Yes, I think it does make sense for this option to also be available on for the social icons.
Dedicated page
My feeling is the most effective solution would be to enable newsletter sign-up modules on both forms of About page; basic and documentation.
I think we could include it on both the current About and Contact Us pages for instance.

Hi @rdunk — we didn't go over this yesterday during our call but we can proceed with this after tasks relating to @micahwalter have been completed.


Various fixes docker

Codecov Report

Merging #14 (18ef9d8) into master (2eb2a29) will decrease coverage by 33.81%.
The diff coverage is 85.13%.

:exclamation: Current head 18ef9d8 differs from pull request most recent head 81ece30. Consider uploading reports for the commit 81ece30 to get more accurate results

@@             Coverage Diff              @@
##            master      #14       +/-   ##
============================================
- Coverage   100.00%   66.18%   -33.82%
============================================
  Files            3       17       +14
  Lines           68      488      +420
============================================
+ Hits            68      323      +255
- Misses           0      165      +165

Impacted Files
Coverage Δ

src/amqp_invoker.py
25.64% <0.00%> (ø)

src/function_invocable.py
36.66% <0.00%> (ø)

src/ergo_cli.py
36.50% <25.00%> (ø)

src/flask_http_invoker.py
36.84% <25.00%> (ø)

test/integration/test_amqp.py
91.46% <91.46%> (ø)

test/integration/test_http.py
95.65% <95.65%> (ø)

src/version.py
33.33% <100.00%> (ø)

test/integration/utils.py
100.00% <100.00%> (ø)

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2eb2a29...81ece30. Read the comment docs.


ROCm Support

Would love to see AMD/ROCm support. I know you guys are busy though. If I have the time I'll put in a pull request.

Glad to hear that and waiting for your PR.


Cannot create a PR: Cannot read property 'insertAdjacentElement' of null

Not sure if I'm reporting thisin the right place but when i trying to create a PR I get this console error.

also the Create Pull Request button is disabled. This works after refreshing few times

Hi, I believe this is a question about GitHub.com, the website. This issue tracker is for a command-line utility called hub, not the website itself. For reporting bugs and requesting features for GitHub the website, please go to https://github.com/contact or visit https://github.community. Thank you!


Add collection Cursed HYAIPE Punks

we dont support cursed yet

we dont support cursed yet

We acknowledge that cursed support is not available at the moment. However, we would like to request to be added to the queue for when it becomes supported. We believe that this collection can greatly contribute to testing and fostering innovation within the cursed space. Thank you for considering our request!

cool ill let it hang here until we figure out what we are gonna do


Load more transactions

@tranvictor Currently, Load more button is not implemented, What is maximum transaction we should display?

We will not support transaction history in new version so I will close this issue


feat: ng-if doesn't work with ion-content

Type: feat
Platform: all
Hi, I'm not sure whether this is intentional or not but I can't get ng-if to work with <ion-content>.
Indeed, when doing so, I get a :
Error: Failed to execute 'appendChild' on 'Node': This node type does not support this method.
The problem seems to come from the scroll, if I add <ion-content scroll="false" ng-if="vm.bool"> it works. Also, ng-show/ng-hide does work fine and I'm currently using this but I guess it would be nice to be able to use ng-if.
Another workaround is to wrap the <ion-content> in a

Greetings @clmntfrndz!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!

I have encountered the same issue with ng-if not working with ion-content. The workaround to wrap the <ion-content> in a
 worked.

Was also hit by this. I don't see any reason why this wouldn't work, looks like something that should be fixed.

Hello! Are you still experiencing this issue?

Hello! Thanks for opening an issue with us! Since it has been a little while since there was any activity on this issue i will be closing it, but feel free to respond back if your still having the issue! Thanks again!

ng-if is currently working on ion-content. However, I get an uncaught exception in the console on views using this.
TypeError: Cannot read property 'clientHeight' of undefined
    at child.initialize (vendor.js:29592)
    at child.ionic.views.View (vendor.js:27077)
    at new child (vendor.js:26041)
    at new <anonymous> (vendor.js:80196)
    at invoke (vendor.js:48994)
    at Object.instantiate (vendor.js:49002)
    at vendor.js:53558
    at prelink (vendor.js:82799)
    at invokeLinkFn (vendor.js:53194)
    at nodeLinkFn (vendor.js:52673)
    at compositeLinkFn (vendor.js:52085)
    at nodeLinkFn (vendor.js:52689)
    at compositeLinkFn (vendor.js:52085)
    at publicLinkFn (vendor.js:51960)
    at self.appendViewElement (vendor.js:79546)
    at Object.render (vendor.js:77651)

@jgw96 I can still reproduce it in version 1.2.4
Below is the sample. I added ng-if in home.html
http://plnkr.co/edit/wSqvxrnrRY7WMEmXZySi?p=preview
It will cause the createScrollbars method fail and the list in the ion-content can't be scrolled.
Error: HierarchyRequestError: DOM Exception 3
appendChild@[native code]
__createScrollbars@http://code.ionicframework.com/1.2.4/js/ionic.bundle.min.js:5564:37
initialize@http://code.ionicframework.com/1.2.4/js/ionic.bundle.min.js:4962:28
View@http://code.ionicframework.com/1.2.4/js/ionic.bundle.min.js:4500:26
child@http://code.ionicframework.com/1.2.4/js/ionic.bundle.min.js:3464:49
http://code.ionicframework.com/1.2.4/js/ionic.bundle.min.js:20938:58

Issue still perstists, newest version (Not ionic2beta)
ionic.bundle.js:25642 Error: Failed to execute 'appendChild' on 'Node': This node type does not support this method.
at Error (native)
Works on some pages, not others, very inconsistent.

having same issue:
TypeError: Cannot read property 'clientHeight' of undefined
at ionic.views.ScrollNative.ionic.views.View.inherit.initialize (http://localhost:8100/lib/ionic/js/ionic.bundle.js:6999:43)
at ionic.views.View (http://localhost:8100/lib/ionic/js/ionic.bundle.js:4484:21)
at new child (http://localhost:8100/lib/ionic/js/ionic.bundle.js:3448:44)
at new  (http://localhost:8100/lib/ionic/js/ionic.bundle.js:57540:36)
at invoke (http://localhost:8100/lib/ionic/js/ionic.bundle.js:17762:17)
at Object.instantiate (http://localhost:8100/lib/ionic/js/ionic.bundle.js:17770:27)
at http://localhost:8100/lib/ionic/js/ionic.bundle.js:22326:28
at prelink (http://localhost:8100/lib/ionic/js/ionic.bundle.js:60143:24)
at invokeLinkFn (http://localhost:8100/lib/ionic/js/ionic.bundle.js:21962:9)
at nodeLinkFn (http://localhost:8100/lib/ionic/js/ionic.bundle.js:21441:11)
Any solution please?
Basically, I want to check whether variable has value or not in the view.

@Simerjit-developer Current workaround is wrap your content in an extra div and put the ng-if there.
The wrapper can be directly inside or outside of .


Ensure added tokens are supported

+1 on this one, trying to get phi3 working on iOS and the tokenizer does not seem to recognise the added_tokens I the config, specifically the <|end|> token, so the model doesn't know when to stop.
Test code:
import Tokenizers

func testTokenizer(testString: String) async throws {
    let tokenizer = try await AutoTokenizer.from(pretrained: "mlx-community/Phi-3-mini-128k-instruct-4bit")
    let inputIds = tokenizer(testString)

    for token in inputIds {
        let decoded = tokenizer.decode(tokens: [token])
        print("\(decoded)")
    }
}

Output:
<s>
hello
<
|
end
|
>

as can be seen in the tokenizer.json on huggingface <|end|> can be found in the added_tokens, but the tokenizer does not seem to honor it and sees all parts of <|end|> as separate tokens.


installing PySM3 at NERSC

I'm having problems at installing pysm3 at nersc.
I'm using the pip install pysm3 command with the --prefix option and getting the following error:
   ERROR: Command errored out with exit status 1:      command: /usr/common/software/tensorflow/gpu-tensorflow/1.15.0-py37/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-aftj8io3/pysm3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-aftj8io3/pysm3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-aftj8io3/pysm3/pip-egg-info          cwd: /tmp/pip-install-aftj8io3/pysm3/     Complete output (5 lines):     Traceback (most recent call last):       File "<string>", line 1, in <module>       File "/tmp/pip-install-aftj8io3/pysm3/setup.py", line 51, in <module>         from astropy_helpers.setup_helpers import (     ModuleNotFoundError: No module named 'astropy_helpers.setup_helpers'     ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Have you tried the conda package?

No, I usually prefer to use pip at NERSC, but I can try with conda if you recommend so.

Do you want to also install mapsims?
On Wed, Mar 4, 2020, 07:37 Nicoletta Krachmalnicoff <
<EMAIL_ADDRESS>wrote:

No, I usually prefer to use pip at NERSC, but I can try with conda if you
recommend so.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/healpy/pysm/issues/40?email_source=notifications&email_token=AAC5Q4RPDN6ICU3YCYP6IODRFZYUPA5CNFSM4LBJFPE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENYQA5I#issuecomment-594608245,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC5Q4S2JGGR2XHYHOHD77LRFZYUPANCNFSM4LBJFPEQ
.

great! thanks!

give me a few days, I want to find a way to properly handle once and for all the case of running on the login node (where I cannot import mpi4py).

@NicolettaK for now do:
module load python/3.7-anaconda-2019.10
conda create -c conda-forge -n pysm pysm3 python=3.7 ipython
conda activate pysm
module unload python

I found the last step needed otherwise it always loads the base environment.
Then quick test:
python -c "import pysm;sky=pysm.Sky(nside=64, preset_strings=['d1']);print(sky.get_emission(12*pysm.units.GHz)[:5])"
[[ 0.05146998  0.05273027  0.06248602 ...  0.04509924  0.06567437
   0.0677657 ]
 [ 0.00080898 -0.00050239  0.00052695 ...  0.00210943 -0.00166372
   0.00063091]
 [-0.00293329  0.00108048 -0.00223178 ... -0.00236847  0.00317197
   0.00120832]] uK_RJ

It worked. Thanks!

I've got the same problem when installing pysm3 in a clean conda environment on my laptop. pip install pysm3 fails with Nico's error message.
Installing the pysm3 through conda works, but I'd like to add pysm3 as a dependency to another package that I is currently pip-installable.

Have you tried installing astropy_helpers?
On Sun, Mar 8, 2020, 04:39 Davide Poletti<EMAIL_ADDRESS>wrote:

I've got the same problem when installing pysm3 in a clean conda
environment on my laptop. pip install pysm3 fails with Nico's error
message.
Installing the pysm3 through conda works, but I'd like to add pysm3 as a
dependency to another package that I is currently pip-installable.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/healpy/pysm/issues/40?email_source=notifications&email_token=AAC5Q4XSJHY2LJYRDVLUUCDRGN7WXA5CNFSM4LBJFPE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOETU3A#issuecomment-596195948,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC5Q4U4MKHQDPF3QJI2MC3RGN7WXANCNFSM4LBJFPEQ
.

Thanks. Yes, if I installed the modules by hand it worked. However, I found that the problem was that pip was picking the system install instead of the one on the conda environment. Probably I initialized the environment with conda create -n myenv instead of conda create -n myenv python=3.7. In the latter case everything works. I hope this saves others' time...


Fix Issue3: openalea namespace generation

Fix #3 by removing flags --single-version-externally-managed.
Add six as host (used in setup.py install-requires).

Checklist

[X] Used a fork of the feedstock to propose changes
[X] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

Close #3

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Consul Service Discovery needs more documentation

ISSUE TYPE

Documentation Report

DESIRED RESULTS
All declaration examples in clouddocs refer to only a "/nodes" endpoint for consul, which is a really strange way to grab everything consul knows about and add it into a pool. What isnt documented is that you can pull the "/service/{name of service}" which is actually a logical grouping someone would use.
The documentation should highlight the available options
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/discovery.html

At the moment we do not recommend users to use Consul's service API because the "servicePort" property from that API is ignored by AS3's service discovery (see #117). We have an item in our backlog to improve support for Consul's service API, and I have added a comment there about making sure this documentation is added when we do that. Thank you for your feedback.

is that needed though? only if you had different ports on the nodes across the pool member, otherwise you specify the ports in the pool class and it works.
is there a different method to no pull in literally every node in consul to a pool? like tag/label support?

I was misunderstanding your use case. I agree now that we should document this use case, and it does not require the using the servicePort property. I have added this to our internal backlog as AT-692.

Just watched the HashiCorp + F5 joint webinar wherein Consul was used with AS3 for service discovery. It would be nice if the documentation is updated soon, as this may cause confusion about Consul support.

Documentation has been added for using the Consul service API: https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/discovery.html#service-discovery-using-hashicorp-consul-for-a-specific-service
Closing this issue.


docs: Draft of merging this with the go-ipfs contribution guidelines

https://github.com/ipfs/go-ipfs/blob/master/contribute.md

Reviewed and approved by @Stebalien and @jbenet


XREADGROUP skips messages if body is nil

Bug Report
Current Behavior
I have a capped stream and a consumer has read messages and did no acknowledge. Then those messages got trimmed, but they remain in PEL of the consumer. Lettuce seems to return items skipping like if body nil fooled it

I have 10 pending messages via redis-cli:
> xreadgroup GROUP clickhouse5 LogToCH-1 count 12500 streams LOG 0-0
1) 1) "LOG"
   2)  1) 1) "1613148827078-6"
          2) (nil)
       2) 1) "1613149395197-12"
          2) (nil)
       3) 1) "1613151931878-5"
          2) (nil)
       4) 1) "1613152648864-7"
          2) (nil)
       5) 1) "1613157152132-67"
          2) (nil)
       6) 1) "1613159691817-0"
          2) (nil)
       7) 1) "1613161933879-11"
          2) (nil)
       8) 1) "1613162164941-3"
          2) (nil)
       9) 1) "1613164598204-0"
          2) (nil)
      10) 1) "1613169116881-8"
          2) (nil)

then I run the same reading command in lettuce
15:21:40.990 [thread-1] DEBUG RedisStreamClientRead - LOG:clickhouse5:LogToCH-1><PHONE_NUMBER>989) reading Pending after 0-0
15:21:40.990 [thread-1] TRACE RedisStreamClientRead - LOG:clickhouse5:LogToCH-1: begin reading from stream 25000 items
15:21:41.160 [thread-1] TRACE RedisStreamClientRead - LOG:clickhouse5:LogToCH-1 xreadGroup returned 5 items timetaken 169 ms
15:21:41.161 [thread-1] DEBUG RedisStreamClientRead - LOG:clickhouse5:LogToCH-1><PHONE_NUMBER>989) received 5:<PHONE_NUMBER>078-6 -<PHONE_NUMBER>204-0

In the last line the log reports the first and the last read message ID
after the messages are processed and acknowledged I check the PEL in redis-cli again
> xreadgroup GROUP clickhouse5 LogToCH-1 count 12500 streams LOG 0-0
1) 1) "LOG"
   2) 1) 1) "1613149395197-12"
         2) (nil)
      2) 1) "1613152648864-7"
         2) (nil)
      3) 1) "1613159691817-0"
         2) (nil)
      4) 1) "1613162164941-3"
         2) (nil)
      5) 1) "1613169116881-8"
         2) (nil)

You can see that every even message remained in the PEL and every odd is gone.

Expected behavior/code

I would assume that the xreadgroup should return all messages like the redis-cli does. I also do not know any good way of bringing down the PEL to 0 other than doing it log2 times
Environment

Lettuce version(s): 5.3.1.RELEASE
Redis version: 5.0.5

That issue happens only when using RESP2. That's fixed now.


prometheus cannot find all k8s resouces, such as pod/services etc

I deployed prometheus with image on my cluster, and added scc privileged and admin role permission to system:serviceaccount:monitoring:default, I hoped I could see the cluster monitoring states from prometheus, but it kept reporting errors:
time="2017-07-14T10:01:50Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:207: Failed to list *v1.Pod: User "system:serviceaccount:monitoring:default" cannot list all pods in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
 time="2017-07-14T10:01:51Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to list *v1.Endpoints: User "system:serviceaccount:monitoring:default" cannot list all endpoints in the cluster" component="kube_client_runtime" source="kubernetes.go:75"
Environment
openshift 1.5
System information:
Linux 3.10.0-514.21.1.el7.x86_64 x86_64

Prometheus image version:
docker.io/prom/prometheus      latest      4da113bb6ae3        4 weeks ago         74.49 MB

Alertmanager image version:
docker.io/prom/alertmanager    latest<PHONE_NUMBER>11        4 weeks ago         17.63 MB

Prometheus DeployConfig file:

apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: prometheus
  name: prometheus
spec:
 replicas: 1
 selector:
   app: prometheus
   deploymentconfig: prometheus
 strategy:
   type: Rolling
 template:
   metadata:
     labels:
       app: prometheus
       deploymentconfig: prometheus
   spec:
     containers:
     - args:
       - -storage.local.retention=6h
       - -storage.local.memory-chunks=500000
       - -config.file=/etc/prometheus/prometheus.yml
       - -web.console.libraries=/etc/prometheus/console_libraries
       - -web.console.templates=/etc/prometheus/console
       - -alertmanager.url=http://alertmanager:9093
       image: prom/prometheus:latest
       imagePullPolicy: Always
       securityContext:
         runAsUser: 0
         privileged: true
         capabilities: {}
       name: prometheus
       ports:
       - containerPort: 9090
         name: web
         protocol: TCP
       volumeMounts:
       - mountPath: /prometheus
         name: prometheus-data
       - mountPath: /etc/prometheus
         name: prometheusconf
     dnsPolicy: ClusterFirst
     restartPolicy: Always
     nodeSelector:
       kubernetes.io/hostname: tj1-container-cloud02.kscn
     serviceAccountName: default
     securityContext:
       runAsUser: 0
       privileged: true
     volumes:
     - hostPath:
         path: /prometheus
       name: prometheus-data
     - hostPath:
         path: /etc/prometheus
       name: prometheusconf

Alertmanager DeployConfig file:

apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: alertmanager
  name: alertmanager
spec:
 replicas: 1
 selector:
   app: alertmanager
   deploymentconfig: alertmanager
 strategy:
   type: Rolling
 template:
   metadata:
     labels:
       app: alertmanager
       deploymentconfig: alertmanager
   spec:
     containers:
     - image: prom/alertmanager:latest
       imagePullPolicy: Always
       securityContext:
         runAsUser: 0
         privileged: true
         capabilities: {}
       name: alertmanager
       ports:
       - containerPort: 9093
         name: web
         protocol: TCP
       volumeMounts:
       - mountPath: /alertmanager
         name: alertmanager-data
       - mountPath: /etc/alertmanager
         name: alertmanager-config
     dnsPolicy: ClusterFirst
     restartPolicy: Always
     nodeSelector:
       kubernetes.io/hostname: tj1-container-cloud03.kscn
     serviceAccountName: default
     securityContext:
       runAsUser: 0
       privileged: true
     volumes:
       - name: alertmanager-data
         hostPath:
           path: /home/data/alertmanager
       - hostPath:
           path: /etc/alertmanager
         name: alertmanager-config

Usage questinos are best asked at https://groups.google.com/forum/#!forum/prometheus-users


pair.dx, dy, dz not available for DDrppi?

Dear sir,
I have adopted the custom weighting function procedure and done good tests with DD and DR, including the use of pair->dx, dy, dz.
However, when I tried to use dx, dy for the projected pairs, it always returns segmentation fault, despite successes in using weight_type=pair_product.
Could you please be able to shed some light on this issue?
Thanks,
Qianli

Sure, we'll be happy to take a look.  Are you using theory.DDrppi or mocks.DDrppi?
Could you post your custom weighting function?  Also, remember that pair.dx is of type weight_union_DOUBLE, so your weighting function must access pair.dx.d, pair.dx.s, or pair.dx.a, depending on which instruction set you're using.

Thanks, so I was using DDrppi from:
from Corrfunc.mocks.DDrppi_mocks import DDrppi_mocks
the custom weighting function I was using is: (showing only the DOUBLE one, AVX and SSE follows similarly with dx.a or dx.s specified etc.)
'''
static inline DOUBLE MY_GAMMA_T_DOUBLE(const pair_struct_DOUBLE pair){
DOUBLE CTWOPHI;
DOUBLE STWOPHI;
CTWOPHI = -1.0 * ((pair->dx.dpair->dx.d) - (pair->dy.dpair->dy.d))/
((pair->dx.dpair->dx.d) + (pair->dy.dpair->dy.d));
STWOPHI = -1.0 * (2.0pair->dx.dpair->dy.d)/
((pair->dx.dpair->dx.d) + (pair->dy.d*pair->dy.d));
return  CTWOPHI*pair->weights1[0].d + STWOPHI*pair->weights1[1].d;

}
'''

I don't immediately see any problem with this function.  Does the crash still happen if you use the 'fallback' kernel?
We'll probably need example code that produces the crash to debug further.  You may wish to fork the repository so we can download and compile the code with your custom weight function.

The problem doesn't occur when I used DD for 3D pairs though. I would definitely take your suggestion by forking the repository. But I haven't used the "fallback" kernel before, do you know how can I enable it?
Best wishes,

Pass the argument isa='fallback' to the DDrppi_mocks function.
Similarly, you can use 'avx' and 'sse' to test the other kernels.
On Wed, Nov 1, 2017 at 3:09 PM, qx211<EMAIL_ADDRESS>wrote:

The problem doesn't occur when I used DD for 3D pairs though. I would
definitely take your suggestion by forking the repository. But I haven't
used the "fallback" kernel before, do you know how can I enable it?
Best wishes,
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/manodeep/Corrfunc/issues/141#issuecomment-341208564,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFYPy5txgqOoWk_rwKhGG-m83sEEijwCks5syMHtgaJpZM4QOizs
.

Thanks! I will let you know what comes out!

@lgarrison Could this be related to #110 ?
@qx211 How did you install Corrfunc? via git clone or pip install

@qx211 Side-note -- you should get better performance if you rewrite the function in the following (untested) manner:
static inline DOUBLE MY_GAMMA_T_DOUBLE(const pair_struct_DOUBLE *pair){
    DOUBLE CTWOPHI;
    DOUBLE STWOPHI;
    const DOUBLE dx = pair->dx.d;
    const DOUBLE dy = pair->dy.d;
    const DOUBLE sqr_dx = dx * dx, sqr_dy = dy * dy;
    const DOUBLE inv_sqr_dx_p_sqr_dy = 1/(sqr_dx + sqr_dy);
    CTWOPHI = (sqr_dy - sqr_dx) * inv_sqr_dx_p_sqr_dy;
    STWOPHI = -2 * dx * dy * inv_sqr_dx_p_sqr_dy;

    return  CTWOPHI*pair->weights1[0].d + STWOPHI*pair->weights1[1].d;
}

Re-writing this way reduces the number of computations, keeps all the calculations in the same precision (float or double), and removes one (expensive) division operation.

@qx211 Did you get a chance to look at this?

@qx211 I am going to close this issue, please feel free to re-open if you had any trouble getting the code to work with the fallback kernel.


feat: add form submission health check metrics

Summary
Add a CloudWatch metric filter for successful and failed form submissions through the client app.
Related

https://github.com/cds-snc/platform-forms-client/issues/3590

good step


[DPE-3543] Zookeeper on Juju secrets (peer and for cross-charm relations

Zookeeper on Juju Secrets both for Peer and Cross-charms Relations.
NOTE: This PR relies on small data-platform-libs updates that are still to be merged: https://github.com/canonical/data-platform-libs/pull/138
However I think the conten here is certainly ready for further discussions (while the above gets merged)

major check note-to-myself Though pipelines confirm external relations functionality, I want to follow step-by-step that there's zero interference regarding cross-charm relations protocols (ZK event handlers vs. DP-libs event handlers)

Early version of https://github.com/canonical/zookeeper-operator/pull/129


install.sh uses unrecognized v2-install option --symlink-bindir

Description
It appears install.sh is a bit too trusting to the environment and ends up using some wrong tools from the system despite having just installed the right tool.
./install.sh
...
===== RUNNING: ghcup install-cabal
Installing cabal-install-<IP_ADDRESS> into "/home/gnezdo/codeworld/build/.ghcup/bin"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4071k  100 4071k    0     0  19.5M      0 --:--:-- --:--:-- --:--:-- 19.5M
Successfully installed cabal-install into
  /home/gnezdo/codeworld/build/.ghcup/bin

===== RUNNING: cabal update
Config file path source is default config file.
Config file /home/gnezdo/.cabal/config not found.
Writing default configuration to /home/gnezdo/.cabal/config
Downloading the latest package list from hackage.haskell.org
===== RUNNING: cabal v2-install happy-1.19.9 alex --symlink-bindir=/home/gnezdo/codeworld/build/bin
cabal: unrecognized 'v2-install' option
`--symlink-bindir=/home/gnezdo/codeworld/build/bin'
gnezdo@os-ws:~/codeworld$ which cabal
/usr/bin/cabal
gnezdo@os-ws:~/codeworld$ cabal --version
cabal-install version <IP_ADDRESS>
compiled using version <IP_ADDRESS> of the Cabal library
gnezdo@os-ws:~/codeworld$ /home/gnezdo/codeworld/build/.ghcup/bin/cabal --version
cabal-install version <IP_ADDRESS>
compiled using version <IP_ADDRESS> of the Cabal library

gnezdo@os-ws:~/codeworld$ uname -a
Linux os-ws 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3 (2019-09-02) x86_64 GNU/Linux
gnezdo@os-ws:~/codeworld$ cat /etc/issue
Debian GNU/Linux 9 \n \l

I'm mistaken, it is picking up the correct cabal binary. It's just the flag which is invalid.

If I understand the intent correctly, cabal-3 can do the same thing with
--installdir=$BUILD/bin --install-method=symlink
I can send a PR.

Okay, that sounds great!  I'll confess to knowing almost nothing about cabal v2 commands.  The option there was the result of my random fiddling to make GHCJS 8.6 work.  It may or may not even matter that it's a symlink in the first place.

One little concern that I have is we don't pin cabal repository state which is presumably responsible for suddenly starting to download a newer cabal than before. It might be something to consider to have a more reproducible setup going forward.

Yes, I understand.  CodeWorld has a lot of non-reproducible build going on.  For instance, I believe it's currently cloning GHCJS from github at head.  I'm definitely supportive of improving matters, but I haven't really sorted through everything.

Subsumed into #1181


Cart vanishes when the shell is restarted (OCC-276)

Describe the bug
If you have items in the cart, a shell restart will wipe it out. This is an issue because configuration changes, deployments can cause the shell to restart at any time, but this disrupts shopping and causes the shop to lose money.
To Reproduce

Launch the latest main (9d3d6859b15672c28c55825d6fbca4b495cad97e).
Add something to the cart.
Enable any feature. For the sake of simplicity, I enabled the harmless Background Tasks one.
Observer that after a refresh on the frontend, the contents of the cart are gone.

Expected behavior
The cart remains after a shell restart too.
Screenshots
https://github.com/user-attachments/assets/58e46923-4269-4933-898d-f56364385c1a
Jira issue

This is not a bug. The reference implementation of IShoppingCartPersistence is SessionShoppingCartPersistence. This serializes the cart contents into HttpContext.Session, which is volatile storage that's lost when the application closes.
The advantage of this temporary storage is both ease of implementation and not having to worry about invalidating and periodically cleaning up abandoned carts (otherwise abandoned carts could accidentally or maliciously eat up the site's database space). The disadvantage is what you described. However config changes and deployments shouldn't happen so often where this becomes a realistic problem in everyday use, so session storage remains a suitable default.
Web developers using OCC can implement their own IShoppingCartPersistence which uses site setting, an external database, etc for persistent storage. We may add a different implementation as a feature in the future. The added complication and the unclear demand makes me less convinced that this should be a priority. But if desired, please convert this from bug to a feature issue.

Really every user-facing feature needs to be resilient so shell restarts in an OC app (as it is with the OOTB features, like user logins remain, and you can edit content items without disruption), since a shell restart is not at all uncommon. It happens e.g. in the following scenarios:

A lot of site settings and other configuration changes restart the shell apart from turning features on or off.
On DotNest, we shut down idle tenants. If you start shopping, get up to do whatever, then come back later to finish it, you might hit a now newly starting shell.
In an actively developed app deployments and prod rollouts can happen daily, or even multiple times a day. (There are periods like this in our apps too.)

And furthermore, horizontal scaling should be possible to support as well.
I don't think the current implementation is really appropriate, then.


ddc_web_compiler.js is running during DDC startup?

I see evaluation of dart2js code in finishClasses from ddc_web_compiler.js during DDC startup.  I'd like to understand what this is doing and why, whether it only happens with dev tools open, and whether it happens on the main isolate or in parallel.
Tentatively assigning to @jacob314 since I think this might be devtools extension related code.

@jacob314 Can you comment on this?  Is this a DDC extension thing?

That is an extension thing. Alan is working on running the extension ddc_web_compiler.js in the extension's frame or on a worker which will also fix this issue.

Great, thanks!

@alan-knight is this fixed now?

This bug seems to be out of date - closing.


Package List - V1 Parity - SBOM COUNT

The image above (the red circle) should be populated with the total count of SBOM that each package has.
The only potential REST endpoint for getting the TOTAL is:
/api/v1/sbom/by-package?id={id}&limit=10&offset=0

Problem

The rest endpoint above requires the client to set a limit.  What value should I set to limit? Also setting no limits would have any performance penalties?
The UI only needs the total count without the whole list of SBOMs. How can extract only the COUNT? fetching thousand of SBOM's data when the only data required is the COUNT doesn't seem reasonable

You can set the limit to 1. You'll still get the total number of entries.
The burden is actually not to fetch that one sbom, but to count all others. If anything, then removing the count would make sense from a performance perspective.
I agree that a single call of "give me the counts for the following packages" would make sense for this.

I any case, I think a dedicated call for this makes sense. Single call, array in, array out. But it shouldn't be a blocker.