Assess the state of read_sql

I'm going through StackOverflow questions and I notice an uptick in questions about reading from SQL databases.  It might be worth having someone go through twenty recent questions or so (there are 93 currently) and summarizing their requests.
It might also make sense to look at other systems like Spark and see what approaches they do.  We might be able to learn something here.

@mmccarty and I spoke about this and he was curious in looking at this.

@martindurant and I took some time to look through the issues a do a bit of digging. Generally, the questions seem to indicate the need for comprehensive documentation discussing best practices for loading data from query results. I don't see a many articles on how to do this for dataframes, in general.
Notes:

A few issues related to confusion about the intent or usage of index_col. Why do we need it? Perhaps start with why it is an option for pd.read_sql.
Best practices for SQL Alchemy expressions
Consideration of the load on the server, indexing of column tables
Push query to server versus get data and compute in Dask (hint: bottleneck is probably the data download). Also, discuss when to leverage the database query engine versus post processing on results with Dask.
Access raw files directly
Output query to files in temp drop zone
Explanation of how you would do this with delayed when you know your own WHERE syntax (or perhaps make a function that takes such a list/generator?)
Chunk/streaming via client?

A theme among questions that I observed was "I have a more complex query than the entire table, how does dask dataframe read this query from the database?"  I got the sense that the answer was "we don't" and that we can only support queries that select columns.
I wonder if this would actually be possible for us.  We could take their existing query, slap a min/max of the index column on it to get divisions, and then apply WHERE clauses much like we do today.  Do we implement this today?  Should we?
For example what happens if the user wants to ask for SELECT name, value FROM accounts WHERE lastname == "smith"

Also +1 on a best practices or explanatory dealing with SQL database doc.  It would help reduce the stack overflow load I think.  I'd love to be able to say "I think that you're looking for this doc: ..."

the answer was "we don't"

we do, but only if they can put the query into sqlalchemy expression format, which is hard and beyond most casual users. The reason is, that slapping a WHERE clause at the end of a text query will usually fail, is getting it right depends on the query itself and on the dialect of the target server.

Ah, so yes maybe in this case a doc that explains the situation and gives people some examples on how to use SQLAlchemy will help some.  I suspect that some of these users will be motivated enough to say "OK, I guess I'll spend a few minutes trying out SQLAlchemy"

Yes, definitely clear docs... but also a "construct your own delayed functions" may well be easier for many.

This is what I tend to do but I need to be a bit smarter about specifying the types of columns in the pd.read_sql call as in some pulls a column may be empty and in another it may be datetime. Sometimes I have trouble working with these files e.g. appending them all. It's just a pain when I have ~200+ columns. Perhaps as I think about this more I could use the column dtypes from a reference file (e.g. one file which I think has the correct dtypes)
For example I may (manually) split the query into manageable file sizes.
@dask.delayed
def pull_data(year, name, con):
    file_path = "file_{}_{}.parquet".format(year, name)

    query = """
    select * from TABLE
    where YEAR = '{year}' AND NAME = '{name}'
    """.format(year=year, name=name)
    df = pd.read_sql(query, con)
    dd.from_pandas(df, npartitions=8).to_parquet(file_path, engine="pyarrow")

output = []
for year in years:
    for name in names:
        tasks = delayed(pull_data)(year, name, con)
    output.append(tasks)
dask.compute(output)

There's a bag example here which could also be tested https://stackoverflow.com/a/62821858/6046019

How to specify dtypes rather than inferring them?

I added this one to address this question.

@mmccarty and I chatted a bit yesterday on this topic and thought working with some example would be helpful:
https://gist.github.com/quasiben/08a7f291039db2b04c2e28e1a6c21e3b
@raybellwaves I'd like to use your example as well if that's ok with you and convert all of this to docs

Another hand-crafted solution, with some rationale (but should have used a meta=, and said that this won't serialise for distributed)
https://www.saturncloud.io/s/snowflake-and-dask/

Hey @martindurant, thanks, I forgot to pass meta into from_delayed when writing the article.
I put that together after helping some customers out with Snowflake and Dask.  I can summarize the main points so folks don't have to read the whole thing.

Snowflake has alternative loading methods that go to Arrow -> Pandas, so you will get less performance calling read_sq as opposed to their new methods.
StringDTypes vs Object saves memory for strings.
It is efficient to identify which columns should be Categoricals in SQL, and compute the Categorical dtypes themselves in SQL.
Analytic data warehouses have much simpler indexing schemes.  But they need to be aligned with your partitioning logic or else every partition results in a full table scan  (probably not always, but enough where it becomes a big problem).

Thanks @hhuuggoo . I hope you didn't take it as criticism, I meant my comments in parentheses as a note to myself.
Actually, it would make a lot of sense for the current read_sql_table to use unique values for partitioning rather than ranges in some cases. I wonder if there's a solid way to tell which is most appropriate.

@martindurant no worries, I'm very interested in this issue as well, will be following closely and will help out where we can.  Thanks!

@raybellwaves I'd like to use your example as well if that's ok with you and convert all of this to docs

@quasiben of course. Nice job on the gist. I was going to suggest a copy and paste workable solution (possibly in https://github.com/dask/dask-examples ?) with sql alchemy. A while ago I was trying to reproduce an issue regarding read_sql_table, I couldn't, but I documented (very basic) the table creation using sqlite https://github.com/dask/dask/issues/5988#issuecomment-597223489

I definitely hiccuped on finding the URI after only using con in pd.read_sql
Admittedly i'm using ye olde cx_Oracle
For pandas:
dnsstring = cx_Oracle.makedsn('HOST', 'PORT', service_name='SERVICENAME')
con = cx_Oracle.connect(user=u, password=p, dsn=dnsstring)

For dask:
dnsstring = cx_Oracle.makedsn('HOST', 'PORT', service_name='SERVICENAME')
uri = 'oracle://' + u + ':' + p + '@' + dnsstring


Using require instead of import

Hi,
I'm trying to get the nordnet wrapper to work, using the provided example code. As my environment doesn't seem to support 'import' I try to use 'require' instead. However I can't get it to work (although it worked on the Avanza library). Here is the code I'm using:
`//import Nordnet from 'nordnet'
//const nordnet = new Nordnet()
const Nordnet = require('Nordnet');
nordnet = new Nordnet()
`
This is the error I'm getting:
TypeError: Nordnet is not a constructor
at Object. (c:\Users\hakan\node\nordnetApi\app.js:5:11)
at Module._compile (module.js:566:14)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Function.Module.runMain (module.js:605:10)
at startup (bootstrap_node.js:158:16)
at bootstrap_node.js:575:3
Guess there is no need to say I'm a beginner at Node and js...

After two hours of googling and experimenting I solved it. "Babel assigns default exports to the default property. So if you use require to import ES6 modules, you need to access the default property:"
So the code should be:
var Nordnet = require('Nordnet').default; nordnet = new Nordnet()


Add Jeopardy "thinking" music

This is great, thanks Chris! I'll @chrisrowe when published

Awesome. Thank you!

@chrisrowe
If you've got >=<EMAIL_ADDRESS>npx should be available in your terminal & you can avoid a global install
npx npmusic -m npmusic -m thinking

# or

npm uninstall -g npmusic && npm i -g npmusic && npmusic -m thinking

If you (or any pals) have other contributions, fire away!


[Documentation] Matter 1.0 Application Cluster Specifications "Reportable" for On/Off cluster

Documentation issues

In the Matter 1.0 Application Cluster Specifications the "Reportable" quality information is missing for the OnOff attrible for the OnOff Cluster. The chip XMLs contain this as "reportable" and it also seems that some manufactorers expect it to be reportable and pairing breaks without it

Platform
other
Anything else?
No response

In Matter all attributes are "reportable" except the ones that have the C quality.

Or to be more precise, the "reportable" quality (P) in the cluster specs only affects Zigbee behavior.  In Matter, the behavior is as follows:

The priming part of subscription setup sends values for all attributes which match the given attribute paths.
After that, updates are sent for all attributes except the ones that have the "C" ("Changes Omitted") quality.

The Reportable quality definition in Matter is:

The Subscribe interaction supports all attribute data. This quality is supported by other interactions that only require attribute data with this quality to support interval or change reporting.

and "other interactions" in this case is the Zigbee attribute reporting interaction.

Thank you for the clarification!

But @bzbarsky-apple Then we still have a mismatch in specs vs chip XML files regarding "onOff" attribute of the onoff cluster! WHich is right? Spec or XML? The other should be fixed

The spec is wrong here, and I expect it will be fixed once it's merged with the Zigbee-targeted data model specs.


Automatically update website

Currently the ecosystem on the website is static. That means we manually need to rebuild the website every time we want new packages to show up.
There are two solutions to this:

make the website dynamically load the json from https://scverse.org/ecosystem-packages/ecosystem.json
Trigger a website build via github actions every time a PR gets merged into ecosystem-packages.

repository_dispatch seems a way to do this:
https://github.com/orgs/community/discussions/26323

This seems to have worked :robot: :rocket: :slightly_smiling_face:


Fixed copy paste error

While building the ros2 branch, we get a warning about reading too many bytes:
Starting >>> adma_ros2_driver
--- stderr: adma_ros2_driver
In file included from /usr/include/string.h:495,
                 from /usr/include/c++/9/cstring:42,
                 from /home/rst/AutowareAuto/src/drivers/adma_ros_driver/adma_ros2_driver/src/adma_parse.cpp:3:
In function ‘void* memcpy(void*, const void*, size_t)’,
    inlined from ‘void getmiscellaneuospoi4(const string&, adma_msgs::msg::AdmaData&)’ at /home/rst/AutowareAuto/src/drivers/adma_ros_driver/adma_ros2_driver/src/adma_parse.cpp:966:11:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34:33: warning: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ reading 4 bytes from a region of size 2 [-Wstringop-overflow=]
   34 |   return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---
Finished <<< adma_ros2_driver [21.6s]

This was probably caused by a copy paste error. This PR fixes the issue.

Thank you


python312Packages.tlds:<PHONE_NUMBER>0 -><PHONE_NUMBER>

And configure nix-update as the update script, because the default python update script does not find the tags, because it only checks for releases.
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 329431 run on x86_64-linux 1

6 packages built:

pretix
pretix.dist
python311Packages.tlds
python311Packages.tlds.dist
python312Packages.tlds
python312Packages.tlds.dist


HL-66 | Validate employee data

Description :sparkles:
Validation rules:

Working hour must be > 18 hour per week
Salary must be > 0
Vacation money and other expense must be >= 0

Issues :bug:
Testing :alembic:
Screenshots :camera_flash:
Additional notes :spiral_notepad:

Codecov Report

Merging #178 (8ddb2b1) into develop (d040408) will increase coverage by 27.69%.
The diff coverage is 96.87%.

@@             Coverage Diff              @@
##           develop     #178       +/-   ##
============================================
+ Coverage    67.51%   95.21%   +27.69%
============================================
  Files          273       74      -199
  Lines         5686     2236     -3450
  Branches       433        0      -433
============================================
- Hits          3839     2129     -1710
+ Misses        1810      107     -1703
+ Partials        37        0       -37

Impacted Files
Coverage Δ

backend/benefit/applications/models.py
93.65% <ø> (ø)

backend/benefit/applications/api/v1/serializers.py
92.74% <95.65%> (+0.40%)
:arrow_up:

...enefit/applications/tests/test_applications_api.py
99.69% <100.00%> (+<0.01%)
:arrow_up:

backend/benefit/helsinkibenefit/settings.py
93.61% <100.00%> (+0.13%)
:arrow_up:

backend/shared/shared/common/tests/conftest.py
94.44% <0.00%> (-5.56%)
:arrow_down:

backend/shared/shared/oidc/auth.py
86.84% <0.00%> (-1.32%)
:arrow_down:

...nd/shared/browser-tests/hooks/http-request-hook.ts

frontend/kesaseteli/employer/jest.config.js

...loyer/src/components/employerForm/EmployerForm.tsx

... and 197 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d040408...8ddb2b1. Read the comment docs.


SVG files not rendering 100% correctly

Hi!
first of I want to make clear that the SVG renderer works really really good
for the most part.
But I found several glitches:

If the real content in the SVG is much smaller than the "page size", kivy can't load the svg, for optimal results you have to make the page size exactly like your object.
I had one SVG with many thin lines on top of a surface, to emulate a wood look. This was rendered completely wrong. See http://paste.fedoraproject.org/160426/18780731/
I stripped the problematic lines from the SVG and it gets rendered correctly in principle, but produces artifacts below and on the side. See http://paste.fedoraproject.org/160427/18780822/
You see this especially when you rotate it.
I use the same SVG for several Scatter widges. As loading them isn't exactly instant I tried to cache the SVG and copy it for the different Scatter widgets. But that didn't work at all, the SVG would always only be displayed on one of the Scatter widgets, while the others were empty. I tried to use deepcopy too, but that wouldn't change anything. It's very important to have a way of caching/copying SVGs cheaply from widget to widget, because of the load times and efficiency.

All in all it still was a lot of fun to make an app with SVG widgets and the benefits of scaling and so on are clear.
Greetings

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/7070497-svg-files-not-rendering-100-correctly?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github).

As of 1.9dev, music.svg isn't rendering on OS/X.  It's the widest of the examples.  FYI.


UI changes trends

Trend Page UI Updates
:recycle: Current situation & Problem
There were a few issues with the Trends Page UI:

Placeholder text for the WIQ survey
No indication that the user could tap on the data cards to see a chart
Data cards would load in varying orders
Data card labels inaccurate

:bulb: Proposed solution

Removed WIQ Survey placehold text so now nothing renders
Added a small arrow icon to the Data Card to indicate to tap to see the charts
Render survey data cards in alphabetical order
Updated data card labels

:gear: Release Notes

:heavy_plus_sign: Additional Information
Related PRs
Testing
Also updated Trends UITests to reflect changes.
Reviewer Nudging
Where should the reviewer start? Where is a good entry point?
Code of Conduct & Contributing Guidelines
By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

[x] I agree to follow the Code of Conduct and Contributing Guidelines.

Can you please fix the lint issues so those tests pass?


Hope to add the theme of vim-one

I want to used the color theme ,how to add the theme

I, too, am trying to work with color scheme. I can't even get vimrc to recognize other schemes that are supposed to be included (vim-colors-solarized, vim-irblack, mayansmoke, vim-pyte) when I edit the extended.vim file, line 32. Each time I open vim up again, it gives me an error that it can't find the color scheme. Is there a place someone can point to on how to update the color scheme?

Did you already try to clone the theme in ~/.vim_runtime/my_plugins/ and configure it in ~/.vim_runtime/my_configs.vim?


Adding age to the default container

Maybe age, the encryption program, would be a good idea to be added by default in this, because some people rely on it to make their chezmoi configuration work properly.

Nice catch, the build kicked off, you should have it in a few minutes.


Please add "clint" as an acceptable guess entry word

Please add "clint" as an accepted Word Master guess entry.  Clint is a real English word (a hard or flinty rock : a rocky cliff : a projecting rock or ledge;  a crevice or gully in limestone rocks).  It is accepted in Wordle, is in the Miriam-Webster Dictionary, and is an accepted Scrabble word, but it is not an accepted guess entry in Word Master.  Is there a way to add it so that it is accepted by Word Master?  I would greatly appreciate it if you would.  BTW, I love Word Master (except for not having clint =).

Sure, I can add that


Add new package cache lock modes

The way locking worked before this PR is that only one cargo could write to the package cache at once (otherwise it could cause corruption). However, it allowed cargo's to read from the package cache while running a build under the assumption that writers are append-only and won't affect reading. This allows multiple builds to run concurrently, only blocking on the part where it is not possible to run concurrently (downloading to the cache).
This introduces a new package cache locking strategy to support the ability to safely modify existing cache entries while other cargos are potentially reading from the cache. It has different locking modes:

MutateExclusive (new) — Held when cargo wants to modify existing cache entries (such as being introduced for garbage collection in #12634), and ensures only one cargo has access to the cache during that time.
DownloadExclusive  (renamed) — This is a more specialized name for the lock that was before this PR.   A caller should acquire this when downloading into the cache and doing resolution.  It ensures that only one cargo can append to the cache, but allows other cargos to concurrently read from the cache.
Shared (new) — This is to preserve the old concurrent build behavior by allowing multiple concurrent cargos to hold this while a build is running when it is reading from the cache

Reviewing suggestions:
There are a few commits needed to help with testing which are first. The main commit has the following:

src/cargo/util/cache_lock.rs is an abstraction around package cache locks, and is the heart of the change. It should have comments and notes which should guide what it is doing. The CacheLocker is stored in Config along with all our other global stuff.
Every call to config.acquire_package_cache_lock() has been changed to explicitly state which lock mode it wants to lock the package cache in.
Context::compile is the key point where the Shared lock is acquired, ensuring that no mutation is done while the cache is being read.
MutateExclusive is not used in this PR, but is being added in preparation for #12634.
The non-blocking try_acquire_package_cache_lock API is not used in this PR, but is being added in preparation for #12634 to allow automatic gc to skip running if another cargo is already running (to avoid unnecessary blocking).
src/cargo/util/flock.rs has been updated with some code cleanup (removing unused stuff), adds support for non-blocking locks, and renames some functions to make their operation clearer.
tests/testsuite/cache_lock.rs contains tests for all the different permutations of ways of acquiring locks.

This is split off from #12634.

Could you go into why these specific cache modes were chosen?  What I'm wanting to understand is which areas of performance are we concerned about and how much this helps.  I feel like the package cache locking in cargo was already confusing (being a un-Rust-like lock) and I'm concerned this will make things harder.

Could you go into why these specific cache modes were chosen? What I'm wanting to understand is which areas of performance are we concerned about and how much this helps.

The way locking worked before this PR is that only one cargo could write to the package cache at once (otherwise it could cause corruption). However, it allowed cargo's to read from the package cache while running a build under the assumption that writers won't modify anything that it is reading. This allows multiple builds to run concurrently, only blocking on the part where it is not possible to run concurrently (downloading to the cache).
This adds a "mutate" mode that ensures that no other cargo is accessing the cache in any way while it is running. This allows garbage collection to make any modifications it wants without worrying about stomping on something another cargo might be using.
The "share" mode is just a formalization of the way cargo used to run while it was doing a build. The reason it was added is to have something to block on the "mutate" lock, to ensure that no cargo is accessing the package cache while a mutate lock is held.
I'm not sure if that answers your question. The only performance concern is to ensure that multiple builds can run concurrently without blocking one another, and this preserves that behavior.

That makes sense.  I also realized the intention last night.  I've attempted to merge your description into the PR description; feel free to clear up anything I made confusing.

:umbrella: The latest upstream changes (presumably #12751) made this pull request unmergeable. Please resolve the merge conflicts.

:pushpin: Commit 78bb7c5cb5a061d153f0a3bf5446a27bfa721a71 has been approved by epage
It is now in the queue for this repository.

Thanks for your patience in working with my feedback!

:hourglass: Testing commit 78bb7c5cb5a061d153f0a3bf5446a27bfa721a71 with merge b48c41aedbd69ee3990d62a0e2006edbb506a480...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing b48c41aedbd69ee3990d62a0e2006edbb506a480 to master...


Support for Linux and MacOS desktop versions

Description:
Greetings @hudy9x, the product seems nice as a note taking tool for developers. Was hoping to get in two suggestions to make this even more awesome.

Would it be feasible to provide offline support for Linux and MacOS operating systems.
Is there any export functionality? or perhaps share notes option that lets other developers view my notes?

@thekodeking

Would it be feasible to provide offline support for Linux and MacOS operating systems.
I apologize, but I'm unable to provide offline support for Linux and macOS systems. My primary focus is on providing online support

Is there any export functionality?
No, there isn't

or perhaps share notes option that lets other developers view my notes?
I do, but the other have to login to the app

Oh cool, thank you for answering.
Closing the issue 👍🏼


Steps for development

It will be great if development steps are added in the Readme. This way contributing can become easier. Thanks.

Sure. That's one of my top priority this week.

It's here: https://github.com/kadirahq/react-storybook/blob/master/.github/CONTRIBUTING.md


Cell types should not be clickable

@signechambers1 rdev can be tested here! https://cell-type-no-click-frontend.rdev.single-cell.czi.technology/gene-expression

@ashin-czi that link isn't working for me, I'm getting "NOT FOUND" after I log in using Github. Is it still working for you?

@signechambers1 hmmm yeah it still works for me, but I sign in with Google. Maybe that would work for you?

Ha that worked. Looks great!!!


doesn't work with jabber.otr.im

due to j.o.i sending a certificateRequest message, which ocaml-tls doesn't care about (yet, see client authentification PR https://github.com/mirleft/ocaml-tls/pull/192)

now merged!


Update dependencies to align with Istio 1.14.3

Signed-off-by: Jacek Ewertowski<EMAIL_ADDRESS>

@jewertow did you try pulling these changes into istio-operator and running its tests? I don't want us to merge this and later find out that the new version of controller-runtime breaks anything in the operator.

You're right. I will do it.


Sync current grammar

This change is

, Matt. Welcome back!
Do you want to add separate files for SAVEPOINT and RELEASE SAVEPOINT as well? What about SHOW KEYS? Not sure if that one's user-relevant, though.

Review status: 0 of 2 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Hi Matt, I have PR #5838 out which makes some improvements to the grammar around @. Just mentioning it in case you want to wait for that to go in first.

Review status: 0 of 2 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

@jseldess I'm happy to add files for whatever you want. Could you open an issue and assign it to me with specifics for what you'd like to add? I personally think all the SHOW stuff should be on a single page, but I don't care strongly.


[Core] Add tooltip to the folder view (TreeView) when the search bar is hovered to show "Search field" or something similar

Solved in https://github.com/gdt050579/GView/issues/267.


[TACHYON-671] Make tachyon.worker.block.BlockLockManager.LockRecord static

A simple fix for https://tachyon.atlassian.net/browse/TACHYON-671

Can one of the admins verify this patch?

Jenkins,  test this please

+1 LGTM

+1 LGTM

Thank @calvinjia @apc999 and @haoyuan for your review and commit.


add spaces round pipe in [conda] version badge

As discussed in https://github.com/badges/shields/pull/8169#discussion_r917443156 - this is a bit clearer

Messages

:book:
:sparkles: Thanks for your contribution to Shields, @chris48s!

  Generated by :no_entry_sign: dangerJS against b399249b7b88dca796d8370efd2b61729ead63ba


Fix an activation error issue

The activation was always flashing "Invalid activation code supplied" when user clicked the activation link in the activation email. (I've seen that even on the october official website this bug occurs)
This is because the Core User::attemptActivation() method does never return true even in case of success but false in error and null else.
The best solution should be to change the core in order to return true in case of successful activation but this is a workaround that will work even if the core function is updated accordingly

Thanks, have fixed the issue at the source


updated vivaldi-snapshot (1.0.380.2)

Closes #1622.

Merged as 274071acb9ede6f4c907e943de618f2bf23441d4.


NotificationRuntimeExceptions are not handled in the account suspension task when triggering email notification.

Describe the issue:
NotificationRuntimeExceptions are not handled in the account suspension task when triggering email notifications. There are places where the NotificationRuntimeException is thrown[1], this is not handled in the code[2].
Environment information :

Product Version: [IS 5.10.0]

[1]https://github.com/wso2-extensions/identity-event-handler-notification/blob/master/components/event-handler-notification/org.wso2.carbon.identity.event.handler.notification/src/main/java/org/wso2/carbon/identity/event/handler/notification/util/NotificationUtil.java#L321
[2]https://github.com/wso2-extensions/identity-governance/blob/master/components/org.wso2.carbon.identity.account.suspension.notification.task/src/main/java/org/wso2/carbon/identity/account/suspension/notification/task/util/EmailUtil.java#L92

similar issue : https://github.com/wso2/product-is/issues/7524

closing since this is tracked through https://github.com/wso2/product-is/issues/7524


Customize connection pool and/or queries for cluster instances

I'm looking to configure the mysql pool to handle reader and writer endpoints available for aws aurora clusters.
Basically read operations would point at one endpoint and write operations would point at another. Is there currently any configuration mechanism available for this use case?
I've looked for previously opened issues that seem related:

Seprate Read And Write Support #703
Replication - Master/Slave #138
[Help]: need figure out how to do Read/Write Splitting with mysql #1951

Its not clear to me if or how they've been resolved and if there's now a mechanism for handling that now. Any help is appreciated.

Currently not supported. I wrote a quick (and dirty) work around though until Knex can be updated.

I'm working on this right now too...
I've set up a Connection Pool and added the WRITER and READER endpoints from the Aurora cluster. Then each time I execute a query I regex it to see if it only contains selects.

+10

@richraid21 You stated in #2339 that having read/write routing inside the pool would be the most robust solution. I disagree:

A pool is, by name, a collection of similar resources. Making them dissimilar splits the pool into separate pools conceptually. Thus, it seems more natural to keep this resulting division in code and preserve the guarantee that each pool stores objects that are functionally equivalent.
Additional complexity in the pool code may bring about bugs similar to those found in generic-pool.
The pool may be used in cases where there are no concepts of "read" and "write", and it should be kept generic, not knex-specific.

I would like to propose having this above the pool, as a router/layer that possibly encapsulates multiple tagged pools and serves acquire() requests using tags:

acquire(tag:string) - returns a resource (in our case, connection) from any pool that has the given tag; tries to round-robin before appropriate pools if acquisition from one fails due to resource starvation (requires short timeouts), or uses resource availability reporting APIs and immediate acquiring in a critical section (without yielding to the main loop)
acquire(tags:string[]) - returns a resource from any pool which has all of the given tags, for example "read" and "write" simultaneously; falls back as above
acquire() - compatibility function which returns a resource using pre-configured tag or tags, according to settings stored in this router instance; might use a "default" tag and knex.js in its default (non-split) config might register 1 pool with the same tag so that by default Knex behaves as it does today
release(resource) - same as currently; does not require user to specify the original tag, as the router should remember the allocations and route the releases back to wherever the resource came from.

I also recently had a need for knex to be able to direct some table queries to a replica DB. It's d be great to see this feature added in the nearest future.
+20

@rkaw92 I think we have the same idea, I just didn't express my thoughts adequately. I agree that this should not be done inside the pool.
To clarify, if I'm understanding correctly, you are imagining a system in which:

A pool is created for each endpoint and is assigned 0..N tags
A pool contains between min and max resources (connections)

When knex requires a connection, it will call acquire(tag:string) which will then locate all pools assigned tag and retrieve a connection via round robin across those pools?
Thanks

@richraid21 That's conceptually it, yeah. Additionally, this layer should live between knex and the pools themselves so as to be re-usable outside of knex and not add any bloat.

Hey all. Not sure this is needed anymore for Aurora... The now offer a master-master serverless DB https://aws.amazon.com/rds/aurora/serverless/

@podtrackers Updated issue header to be more general, some need for this still remains .

@podtrackers I think that's a good point. As far as I can tell, Aurora serverless looks to be more useful in terms of decreasing costs for low-load, non-prod environments. We'll utilize it where we can but will probably continue to run dedicated clusters in our prod environments where the traffic is more consistent.

Having load balancing in db driver has many advantages by removing all the intermediate services (pgpool/virtual interfaces). As it was already mentioned there should be a separate pool per each DB node. But we'd also need health checks so that requests are not sent to servers that are offline (High Availability).
This will also have to be implemented per db driver, because of a need to check replication lag and not send queries to servers that are too far behind master.
Also we'd need to give API user the power to decide which queries go there, with an option to send all selects to slave servers. This could be configured per query, e.g. query.fetch({slave: true})
I'm building a similar solution for PHP, where my code will juggle PDO connections for postgres DB. If  PHP code works as intended, I'm willing to do some sort of knex implementation and share it. It would be only for postgres DB, however.
+10

I took a stab at this here: https://github.com/tgriesser/knex/pull/2847
looking for feedback on the approach taken

I wonder is there anything new about this? We are using AWS RDS and we have read replicas. We use AdonisJs as backend framework and it uses knex.js as database provider. Without read replica support, I guess we have to change our stack.

@ozziest I don't know another stack will automagically do this for you.
The "simple" way is to have a knex handle for writes and a knex handle for reads.
Alternatively, check earlier comment: https://github.com/tgriesser/knex/issues/2253#issuecomment-333715066

I have approached it by separating knex instance of read and write. However, within read and write, one may want to use multiple connection hosts vs relying on a single server and creating a new knex instance for each host is lot of manual work + waste of resources.
To counter the situation, I have created a module, that will let you use a callback function to compute the connection config at runtime and opens up the possibility to round robin between multiple connection configs https://github.com/thetutlage/knex-dynamic-connection
Lemme know if anyone has any thoughts :)

This is my two cents on the issue.
Note that I'm using Aurora PostgreSQL and maybe this is a very bad suggestion, I don't know ;)
Since aurora provide two endpoints: reader and writer (master) where the reader one load balance automatically between all read replicas, we can use only those two.
I show you how I split my queries between read and write and use the correct endpoint.
I also added a way (knex.select('*').queryContext({ useMaster: true })) to force a query to execute on the write or read endpoint.
This is very useful because sometime you can perform a write operation and a read one few ms after.
But since the replication process take time (under 100ms) your data is not guaranteed to be up to date on the replicates databases.
Let's say I "insert" a new product on a shopping cart table, and right after that use some select queries to get the new shopping cart price, I'm not guaranteed the new inserted data is present on the replicates databases yet.
So we should have a way to force select queries to be executed on the write (master) endpoint.
// First you must init two knex object for each endpoint

const knexMaster = require('knex')({
  client: 'pg',
  connection: { ...masterConnectionInfos },
  pool: { ...pollOpts },
});

const knexRead = require('knex')({
  client: 'pg',
  connection: { ...readConnectionInfos },
  pool: { ...pollOpts },
});

// and then we will use another knex object as a wrapper
// this one doesn't need a connection or pool settings
const knexWrapper = require('knex')({
  client: 'pg',
});

// We override the runner method of our wrapper client
knexWrapper.client.runner = function (builder) {

  /** here we will redirect the query on the correct knex object,
  * We use this method since this is one of the first executed
  * after your query has been built and it's still before the aquireConnection process
  **/

  /** bypass with knex.select('*').queryContext({ useMaster: true })....
  * this is to force your query to be executed on read or write endpoint
  * not sure about using queryContext for this but it seems ok to me
  **/
  if (builder._queryContext && builder._queryContext.useMaster === true) {
    return knexMaster.client.runner(builder);
  } else if (builder._queryContext && builder._queryContext.useMaster === false) {
    return knexRead.client.runner(builder);
  }

  // .toSQL() return an object or an array of object (not sure about that but this is what I found in /lib/runner.js)
  const sql = builder.toSQL();
  const useMaster = Array.isArray(sql) ? checkUseMasterMultiple(sql) : checkUseMasterSingle(sql);
  return useMaster ? knexMaster.client.runner(builder) : knexRead.client.runner(builder);
};

const checkUseMasterSingle = (object) => {
  // if object.method equals "insert", "del" or "update" then use master endpoint
  return ['insert', 'del', 'update'].includes(object.method);

  /** As a bonus (I will not provide code since it's still very poorly done on my side and kind of weird...)
  * But here you can also parse the sql query in object.sql
  * Find a way to retrieve all table names usage in the query
  * and if you find some "critical" one, let's say ("shopping_cart") then force to use master endpoint
  * This can be done if you don't want developpers to know when to force select queries on master endpoint using .queryContext({ useMaster: true })
  * Note that parsing every select queries to check for table names usage will take some times on every requests
  * So I suggest you to save a hash of the current query:  const hash = crypto.createHash('md5').update(object.sql).digest('hex');
  * and save the result (if it should use master endpoint or not) in an object, hash map or any other way
  * I think this is fine to do like this since object.sql do not contains variables, ex: select * from "table_name" where "column" in (?, ?, ?) (Note: this is not true If you use knex.raw)
  * It depends on your application and how many different queries your application have
  * Maybe you can still process / parse / transform the sql query to be more abstract and reduce the number (remove all after where condition, care with subqueries)
  **/
};

// this is just a simple loop
const checkUseMasterMultiple = (array) => {
  for (let i = 0 ; i < array.length ; i++) {
    if (checkUseMasterSingle(array[i])) {
      return true;
    }
  }
  return false;
};

// Don't forget to export the wrapper not the others two ;)
export default knexWapper;

@Frolanta Thanks for the idea. Are you sure this gonna work with Aurora pgsql with read replica? I have exactly the same stack (nodejs, aurora postgres) but I also included bookshelf.js above knex.
but overall, it worth to implement this as a feature inside Knex API

This is a real pain point for me as well.

Just confirming that @Frolanta's design seems to work. It should be baked into knex.

@jpike88 : Out of curiosity, what are the main pain points you encounter when using the "2 separate Knex pools" approach?  Ex: I'm guessing that this approach becomes problematic when you try to use bookshelf or any other ORM on top of knex.

It becomes an issue when the codebase gets so large that it's painstaking to manually split all queries between the two pools.
There's one issue with the above alternative though, it doesn't seem to play nice when I try to use the .transaction method, just says can't find includes of undefined. I'll keep messing with it

I'll finally create a separate repo for knex-related utilities today, so if anyone would volunteer to build solution similar to the one proposed above, it could be included there. It sounds a little bit too high level to be included in Knex itself.

Just to inform you that there is a small problem with the solution I provided few months ago.
This is more a problem with AWS Aurora than knex but maybe it can be useful to you.
You can't use the AWS read endpoint which load balance between your replicas.
This will not work if you have a high traffic or if your knex connection timeout is too high.
Let's say you have multiple read replicas.
Then when you create a pool using and AWS Aurora read endpoint host, you will probably only use one of your replicates.
The read host only route to one of your replica "ip" and it change every second.
So if like me your pool if almost full in less than 1sec then all your pool connections will be on the same replica.
When one client connection timeout you will have a chance to be on an other replica, but again if your client connections almost never timeout like me you will only use one replica, or maybe 80% one and 20% another.
The only solution that I found is to have an array of read pools.
Then use the AWS SDK to find every minutes if a new replica is available or have been deleted.
And create (or delete) a new pool then add it to you array.
Since you have multiple read pool, you can select one randomly or use knex.client.pool.used.length and use the pool with the minimum connections.

Without giving too much information away... how much activity are you having to deal with? Is your use case really heavy?
Aurora is becoming increasingly popular, I think this will be more of a problem for people as time goes on

@jpike88 I don't have the exact statistics but the peak is something like 1000 users at the same time.
So it's not very heavy.
We could still easily use a larger instance but I prefer to use multiple smaller replicates to reduce costs. Also our number of users is very volatil so it make more sens to use replicates and autoscaling.

@Frolanta : Just want to make sure I understand the problem you described above.
On the AWS side: it rotates the replicate IP once per second.  However, knex has a tendency to populate its connection pool all at once.  As a result, each of these connections is very likely to talk to exactly the same AWS IP.
Does that sound accurate?

If that's the issue you might want to experiment with low numbers for your pool's idleTimeoutMillis.  You can only get a new connection to the new IP if you give your pool a chance to reap idle connections and create new ones every so often.

@briandamaged Sorry if this is not very clear, but yes this is the idea.
I will try to clarify.
knex doesn't no populate all at once, it's only create a connection when needed, if there is none active. Then the connection will remain active for a time idleTimeoutMillis.
But take an example, let's say you have a pool of 5 connections with an idleTimeoutMillis of 10sec.
You start your application (so no active connection) and in less than 1sec you make 5 queries, this will create 5 connections.
So all of your pool's connections will be on the same replica / IP.
Then if your connection never reach your 10sec timeout because they are always busy then you will always use the same replica.
Sure you can increase your pool connection number or reduce idleTimeoutMillis so it will be less of an issue.
But theoretically with a very high traffic even with a high number of connection and small connection timeout (witch is no very good because connection takes time) this can result to not using all your replicas.

@Frolanta : Got it.  So yeah: it sounds like we're on the same page, then.
The reason I was asking: I've been experimenting w/ some redesigns around Knex's Transaction and Pooling logic.  So, this information is helping us figure out how some of those details should work.
From what I can tell, it looks like at least 2 separate capabilities are desired:

Provide a fancy way to automatically choose the correct pool depending upon which operations are going to be performed.  (ex: use knexMaster whenever a "write" operation will be performed)
Provide some way to alter to connection acquisition logic so that it accommodates upstream behaviors.  (Ex: acquire only 1 connection per second to accommodate AWS's IP rotation)

And, possibly a 3rd capability as well:

Provide a way to "lease" a connection for use across multiple distinct operations.  (Ex: write to the database.  Afterwards, use the same connection to read from the database to avoid timing issues caused by replication.)

Does this sound accurate?

However, knex has a tendency to populate its connection pool all at once.

Knex actually doesn't have any tendency like that.
Though it is true if there are very many concurrent queries done all at once when app starts. Knex does not pre-fill connection pool automatically (not even until the minimum connections limit), but it creates new connection only if there are not enough available connections waiting in the pool and new connection is needed.

@elhigu : Right.  I was just being a bit informal with my description.
More precisely: let's say you launch a new application that uses knex.  Immediately after launching, the application already receives demand for 100 new connections.  In this case, the pooling logic will attempt to create 100 new connections all at once, which effectively recreates the scenario that @Frolanta  was describing.
So, what I'm wondering: would it be possible to introduce a "warm-up" period of some kind?  Ex:

The application launches, but spends the first minute in "warm-up" mode.  (ie: controlled via docker, or something like that)
Meanwhile, the connection pool begins acquiring connections at a rate of 2 per second
Finally, the application leaves "warm-up" mode and becomes available.  Now its pool has been pre-populated with connections that are distributed across the IP addresses

Provide some way to alter to connection acquisition logic so that it accommodates upstream behaviors. (Ex: acquire only 1 connection per second to accommodate AWS's IP rotation)

Just want to chime-in to say that I'm unsure that this is the ideal fix to the problem.  The way I see it, there's some concern that connections aren't (re-)made often enough to get a good mix of read replicas in the case that there's a balanced cluster behind the host you're pointing at.  I think it warrants more experimentation with idleTimeoutMillis and reapIntervalMillis to be sure that the pool really can't be configured to work smoothly in this use-case.  An alternative feature to add to the pool might be to put a limit on the number of times a single connection can be acquired before needing to be reaped.

@devinivy : Agreed -- I'm not convinced that this is an ideal solution.  I'm just confirming that I understand the proposal.
More generally speaking: I've noticed a number of different bugs / feature requests lately that could potentially be reframed as "Connection Pooling Enhancements".  For example:

This current ticket
Details around how connections get reused during Transactions and Nested Transactions.  (Ex:  https://github.com/knex/knex/pull/3722#issuecomment-597955089)
Various work-arounds for operating on lower-level connections more directly.  (Ex:  https://github.com/knex/knex/issues/756#issuecomment-597617787 )
Etc.

In particular: I'm starting to wonder if there is a way to control connection pooling behaviors using a more distributed / component-based approach.  I touched upon one of the main concepts here:
https://github.com/knex/knex/pull/3722#issuecomment-597955089
The idea of "connection pool warming" can then become an extension of this concept.

If there is need to recycle connections one could pass custom validate function in pool config that fails randomly for example with 0.1% chance.
That way once in every 1000 queries a connection is forced to be renewed which shouldn't be too much of an overhead.
It should mixup where everything is connected during the lifetime of app and coupled with slow warmup read replicas should be used pretty evenly also from the start.

One can easily also use validate callback to implement counter after how many times connection can be fetched from pool before it will fail the validation as @devinivy suggested.
Both should be able to be implemented already by passing custom validate function to knex pool configuration.

@briandamaged

Provide a fancy way to automatically choose the correct pool depending upon which operations are going to be performed.  (ex: use knexMaster whenever a "write" operation will be performed)

This could be nice, but we will still need a way to choose manually.

Provide some way to alter to connection acquisition logic so that it accommodates upstream behaviors.  (Ex: acquire only 1 connection per second to accommodate AWS's IP rotation)

I'm not sure this is good idea, 1sec is really long.
Let's say a user perform an action to my API and my pool is not full, it will take maybe 1 more sec to answer :/
Also @devinivy has a point.

Provide a way to "lease" a connection for use across multiple distinct operations.  (Ex: write to the database.  Afterwards, use the same connection to read from the database to avoid timing issues caused by replication.)

The thing is that 100ms (replication time) is kind of long.
Just a very small example of the problem.
Let's say you have an api to create book.
perform /book POST (create a new book and return id)
At 0ms: the new book is in your master database
at 40ms user (browser) received the new id
at 45ms a new request to your api is made /book/:id GET (with the new id)
at 75ms your request is received on your api.
at 85ms your try a SELECT * FROM book .... on a replica but the data doesn't exist yet because the replication has not been perfomed yet.
I don't know how you can imagine a way to lease a connection for this kind of  scenario.
For the replica connection I think the solution should be something a bit like sequelize do https://sequelize.org/v5/manual/read-replication.html (Didn't really dig into it so I don't know much about it)
You should be able to provide an array of replica (read) connection params.
Then don't use AWS loadbalanced read enpoint (because it suck) but all your replicas endpoints (this is really not difficult to get with the AWS SDK)

You should have a way to add and remove replica "on the fly" (because auto-scaling is a thing).

The number of connection to a pool should be per replica or it doesn't make sens.

When a new connection is required knex should choose the least used replica in your array for the connection (the one with the less active connections).

Also, read queries should also be performed sometime and your master (so master connection should also be included during the selection of the least used connection): Your application can be 99% read queries, so if your have one master and only one replica you will not use your master properly if we don't include it.
What do you think ?

@elhigu @lorefnon @briandamaged @maximelkin  I've created https://github.com/knex/knex-utils (with placeholder content for now) for all kind of useful things that are outside of knex per se but can be useful for a wider audience. Ideas for possible features and submissions are more than welcome!

@Frolanta

This (automatic pool selection) could be nice, but we will still need a way to choose manually.

Agreed that end-users should still be able to directly access the underlying pools.  (Personally, I'm not sure how much I'd trust logic that tries to choose the appropriate pool automatically.  It seems like there would be a lot of corner-cases that could cause it to make the wrong choice)

I'm not sure this is good idea, 1sec is really long.
Let's say a user perform an action to my API and my pool is not full, it will take maybe 1 more sec to answer :/

Agreed.  I'm mostly trying to figure out if there are any oddball scenarios that we might need to be able to accommodate in the future.
Like you said: it would be much more efficient just to obtain/update the complete set of IPs periodically.  This would eliminate the need for any type of "warm-up" period.

perform /book POST (create a new book and return id) ...

Yeah -- I don't think connection leasing would be applicable in this example since it spans multiple HTTP transactions.
Connection leasing is more applicable when you have multi-step logic on the server side.  In that case, it provides a way to guarantee that the same connection will be used throughout the entire multi-step process.  (Ex: this is important when you are performing operations inside of a Transaction)

Would be useful if connection could be a function, eg:
const knex = Knex({
  client: 'pg',
  async connection() {
    /** select and return the appropriate server */
  },
})

but ended up with this not so ugly hack:

const knex = Knex({
  client: 'pg',
  connection: {},
})

const servers = [ /** list of servers */ ]
const { acquireRawConnection } = knex.client

knex.client.acquireRawConnection = async function _acquireRawConnection() {

  knex.client.connectionSettings = null

  for (let server of servers) {

    /** logic to select the server */
    if (server.hasSuperPowers) {
      knex.client.connectionSettings = server
      break
    }

  }

  if (!knex.client.connectionSettings) {
    return Promise.reject(new Error(`No read-write server detected`))
  }

  return acquireRawConnection.call(knex.client)
}

using async cause i'm selecting the server based on show transaction_read_only query result

Okay, so I've given it some thought and think we can realize read write clustering without too much effort.

We obviously need some changes to Knex.Config: I would propose a new replication key with which we can specify the cluster configuration:

interface ReplicationConfig {
  connection: string | StaticConnectionConfig | ConnectionConfigProvider,
  pool?: PoolConfig;
}

interface Config<SV extends {} = any> {
  // ... all the other options currently available
  replication: {
    write: ReplicationConfig,
    read: ReplicationConfig[],
  },
}

When the client is being initialized and a replication configuration is present it should not create a single connection pool but instead create one connection pool for the write connection and one for each read connection. This way we can provide different scheduling strategies like round robin or even a least load strategy by selecting the server that has the least current connections. In the case that the write connection should also be used for reading, the connection should be listed in both write and read, but we may consider that they use the same connection pool to lower the actual load of the write server. Also the pool configuration for the read connection would not make any sense and will be ignored.

It is necessary that the Runner passes a parameter to the client.acquireConnection() function indicating whether it needs a write or a read connection (depending on builder._method (can we rely on this?)). This parameter can be optional and by default the write connection will be used. The Client instance can then acquire a connection from its connection pools. This would also be compatible with transactions for which we will default to the write connection (we can assume that transactions include at least one write query).

It is always possible to explicitly specify that the write connection should be used for a given query using @Frolanta's approach of using the query context (which definitely is a good idea IMO).

Yet, the problem of so called sticky connections (using the same connection for one request) remains, but still I think that the developer has the option to chose the connection based on the code.

With just a few changes to the original code I was able to extend the default Client class to support read-replication which I would like to PR into the code base.
There are just a couple of questions which I would like to discuss first before creating a PR.

Is one write connection enough?
Which replication methods should be supported? This should also answer question 1. Is read-replication enough (one connection for writing, multiple connections for reading) or do we want something like Galera clustering as well?
What to do in the case of a connection problem? Should we temporarily ban/ignore a node when acquiring a new connection failed? And should we automatically try a different node on error?

Is one write connection enough?

I think that should be configurable, and considering that there are typically more than a few connections at a minimum, it shouldn't default to 1... maybe 3?

Which replication methods should be supported? This should also answer question 1. Is read-replication enough (one connection for writing, multiple connections for reading) or do we want something like Galera clustering as well?

Never heard of Galera Cluster, likely the market share of ppl using it versus standard configurations is so small it's not worth consideration.

What to do in the case of a connection problem? Should we temporarily ban/ignore a node when acquiring a new connection failed? And should we automatically try a different node on error?

Good question... my main concern with your PR is keeping scope constrained so whatever it does, it does it well and with room for configuration
I'm not a maintainer, just weighing in

@llamadeus please before trying to push that solution to knex, first implement it as an external package, which uses knex internally.   I would say that you are better off with having separate getters for read / write connections already becuase of transactions.
Those are usually needed when ever you are doing anything non-trivial. So you actually have to know what kind of connection you need already before making the queries.
I haven't seen any reason, why you couldn't implement that clustering functionality outside of the knex and I wouldn't like to see this prototyped in knex core code, since when its done, then maintenance will be in shoulders of knex core maintainers and handling bug reports etc.
With external package you can:

setup as many read / write servers as you want.
create a strategy which selects for example the one with least used connection for the next transaction (knex.client.pool provides this information)
create knex instance for query that actually doesn't have connection set yet and then write heuristics, that analyzes created query and decides if read or write connection is needed and request one for the query from you wrapper package for execution (might require that you write some knex plugin for that, if knex extension API is not powerful enough then it can be made better to support also needs of that package)

In any case the most important feature you will need is to manually request read or write transaction.

@llamadeus Let me know if there's anything I can do to help with your work.
I'm not a maintainer, just a project consumer - we implemented Knex about nine months ago and have slowly been replacing Sequelize.  However we're now to the scale where we need to implement read replicas in our Aurora cluster.  I had assumed going in that Knex supported this out of the box - I'm surprised to find that it does not.  We'll either need to help out with this issue, and get read/write replica support into Knex, or we'll have to bake our own internal solution.

@llamadeus same here, I am not a maintainer, but I've been using Knex a lot recently.
I will be happy If I can provide any help.👍

There's one shortcoming with @Frolanta's code, which has been useful to me anyway:
If you're using asyncStackTraces, and there are undefined bindings in a query, an error gets thrown at toSQL() in the wrapper. This causes an useless error as the asyncStackTraces code never had time to execute. At first I lodged #4531 because I thought it was a Knee internal issue but over time realised the real issue.
Here @Frolanta's code with my changes near the bottom:
// We override the runner method of our wrapper client
knexWrapper.client.runner = (builder) => {
	// here we will redirect the query on the correct knex object,
	// We use this method since this is one of the first executed
	// after your query has been built and it's still before the aquireConnection process

	// bypass with knex.select('*').queryContext({ useMaster: true })....
	// this is to force your query to be executed on read or write endpoint
	// not sure about using queryContext for this but it seems ok to me

	if (builder._queryContext && builder._queryContext.useMaster === true) {
		return knexMaster.client.runner(builder);
	} else if (
		builder._queryContext &&
		builder._queryContext.useMaster === false
	) {
		return knexReader.client.runner(builder);
	}

	// .toSQL() return an object or an array of object (not sure about that but this is what I found in /lib/runner.js)
	// if toSQL fails, just point at master. this is because the query won't run anyway, but stack trace will be preserved.
	let sql;
	let useMaster = true;
	try {
		sql = builder.toSQL();
		useMaster = Array.isArray(sql)
			? checkUseMasterMultiple(sql)
			: checkUseMasterSingle(sql);
	} catch (error) {
		// swallow this, it will be thrown properly in a second when Knex internally runs it
	}

	return useMaster
		? knexMaster.client.runner(builder)
		: knexReader.client.runner(builder);
};

@albertomontalesi fffuuuu

Hi guys, wee have any update on this? thanks!

Anything new to handling clustered db with knex ? For switching db automatically ?
Thanks

+10

+1

I should add more context around this workaround I did:
https://github.com/knex/knex/issues/2253#issuecomment-879182892
In that implementation it still blows away all slack outside the knex code, even with asyncstacktrace enabled... so that sucks.

Actually, found the problem. knexWrapper itself needs to have asyncStackTrace enabled for the stack to be preserved. thank god

@kibertoad @OlivierCavadenti any updates about this issue?

+100

For anyone implementing any of the workarounds suggested here:
We put together an implementation based on @jpike88 's code in this thread. That code will break transactions if you route them through the knexWrapper as is. We got around this through a somewhat brute-force method of passing all transactions to our read/write instance.
ala:
  knexWrapper.context.transaction = function(...props) {
    return knexMaster.context.transaction(...props);
  }

Seems to work just fine.

I was thinking instead of changing knex we can manage this at a higher level. Suppose we have servers R1, R2 and W.
We create a knex connection to each one of them. Give each connection a flag so to know it is only readable, only writable or read/write. We put all this connection objects into an array.
When we create a query we call a helper function to give us the desired connection. It would become something like:
 connectionHelper('r')('TABLEX').select(...)  // R1 gets selected
 connectionHelper('w')('TABLEX').insert(...)  // W gets selected

This helper function can implement round robin or random scheme to select a node (in the selected class) in each call.

+1


RFC: test(nutritionSurvey): team indexed by leader ID

Not ready to merge. Pending discussion.
Closes #22.

@johanneswilm, @readevalprint, @peritus, the team property is pending two changes:

A constraint will be added such that the teamID matches the team leader's memberID
A team is no longer static; members are chosen per-survey

I've stubbed how it may look. Note the moving of _id and _rev on team into each team.members. On the app, the team DB will be removed, in favour of a members DB.
An alternative might be to change the members type from array to object and index members by their designation, something like:
{
  "team": {
    "members": {
      "Team Leader": {
        "memberID": 1
        "firstName": "Alexei",
        ...
      },
      "Assistant": {
        "memberID": 2,
        "firstName": "Henry",
        ...
      }
    }
  }
}

Thoughts?

Hey @tlvince why don't we remove the "teamID" and just refer to the ["Team Leader"].member_id directly?
Also could you replace any keys with space to camel case or with underscores?

Sounds good. So it sounds like we can reduce it to:
{
  "team": {
      "teamLeader": {
        "memberID": 1,
        "firstName": "Alexei",
        ...
      },
      "assistant": {
        "memberID": 2,
        "firstName": "Henry",
        ...
      }
  }
}

ok, but isn't this going to be very restrictive once we have teams with 2-3 assistants? (not to think of teams with 2 team leaders).

According to https://github.com/eHealthAfrica/NutSurv-Mobile/issues/104#issuecomment-78875789, a team consists of three members; a leader, anthropometrist and assistant. We can revise the schema later if that changes?

And when you say "per survey" -- What do you mean by "survey"? There has been quite some confusion about that term. I think it was Tomasz/me who used "survey" to mean an individual interview, whereas Robert used it to describe the entire 90 survey on a specific theme (or the other way round). Will the teams change whenever they go to a new house or whenever they start a new 90-day survey cycle?

I've taken survey to mean an individual interview.

Yes, it consists of three members. But we know that is not the case often and the next version will likely require flexibility on that, so it's probably not a good idea to introduce spec changes that will make that harder, I would think. But Maybe it's OK if we just call them "assistant 1", "assistant 2", etc.

@tlvince so do the team members all sign in before each interview?

I think I would prefer having the array for team members (your first proposal) so that there won't be any problems extending it to hold several members with the same designation. But if you feel strongly about that, I guess we can always change it at a later stage, should we need to.

Is it really the case that a team can have more than one leader?

A constraint will be added such that the teamID matches the team leader's memberID

… doesn't make sense if that's the case.

/cc @cristinasantamarina

@tlvince Right now it cannot. Just think ahead of things that can happen. All we know is that the team structure is very different in each country.

No we need to just focus on on the specs we have now. We can modify the
models in the next version of the project if it ever happens.
On Mar 26, 2015 1:57 PM, "Johannes Wilm"<EMAIL_ADDRESS>wrote:

@tlvince https://github.com/tlvince Right now it cannot. Just think
ahead of things that can happen. All we know is that the team structure is
very different in each country.
—
Reply to this email directly or view it on GitHub
https://github.com/eHealthAfrica/data-models/pull/27#issuecomment-86503577
.

Team members are listed once per day, not per interview. But their data should be available for each interview they conduct.
+1 to @tlvince  Teams will only have one Team Leader. We need to make this very plausible assumption; after all the Team Leader ID is the team ID.
+1 to @readevalprint on adjusting to the spec we currently have.

OK, lets move forward and use:
{
  "team": {
      "teamLeader": {
        "memberID": 1,
        "firstName": "Alexei",
        ...
      },
      "assistant": {
        "memberID": 2,
        "firstName": "Henry",
        ...
      }
  }
}

That looks good

@readevalprint care to give this a once over?

Looks good!


Failing tests reported as a success AND a failure

I've just started using this, so apologies if I'm missing something.
When I write tests that pass, all is good and happy. However, if the tests fails then I get two runs of the test reported, one that passed and one that failed with the actual error. This doesn't actually cause problems - the overall test still failed - but it looks weird.
For example, the test:
class DemoTest : WordSpec() {
    init {
        "calculator" should {
            "perform addition" {
                (1 + 1).shouldEqual(3)
            }
        }
    }
}

Gives the following output in Maven:
Tests in error:
calculator.should perform addition(calculator)
  Run 1: PASS
  Run 2: 2 did not equal 3

Fixing the test so that it passes appears to only run the test once, and that is what I'd expect here too.

It's something to do with the way intellij lays out the tests when using a junit runner. I hope junit 5 fixes it, and its in alpha now.

I would hypothesize that the green is shown whilst the test is happening and if it fails the green is supposed to be amended to be red, but instead for this plugin only, a new line is being written instead of the old one overwritten. This doesn't happen for other plugins using the JUnit runner.

It could be that yes. The way it works in junit is you just tell it the status of a test "description". So for some reason it thinks its there twice. Might be something to do with the equality of the instance. Also there are definitely some bugs here, if you put brackets in your test names it all goes wonky. That's something other frameworks have reported and is fixed in JU5.

Would the TestNG be runner an option?

Whilst this happens in IntelliJ as well, what I'm describing here is actually in a command-line Maven build instead.

@sazzer I've setup a maven pom and used your demo test, and I get this output:
Tests in error:
should perform addition(calculator): 2 did not equal 3
Ie, just the single test. How are you getting the "run 1" "run 2" output, is that some maven plugin?

That was just Surefire. Nothing special at all.
If I get time I'll try and set up a bare-bones project and reproduce it.
On Tue, 28 Jun 2016 at 00:34 Stephen Samuel<EMAIL_ADDRESS>wrote:

@sazzer https://github.com/sazzer I've setup a maven pom and used your
demo test, and I get this output:
Tests in error:
should perform addition(calculator): 2 did not equal 3
Ie, just the single test. How are you getting the "run 1" "run 2" output,
is that some maven plugin?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kotlintest/kotlintest/issues/42#issuecomment-228907364,
or mute the thread
https://github.com/notifications/unsubscribe/AAECGJmllIU-vzZ_PzL-5Fu1SIZ4KQeTks5qQF4cgaJpZM4I8nwn
.

Thanks
On 28 Jun 2016 08:11, "Graham Cox"<EMAIL_ADDRESS>wrote:

That was just Surefire. Nothing special at all.
If I get time I'll try and set up a bare-bones project and reproduce it.
On Tue, 28 Jun 2016 at 00:34 Stephen Samuel<EMAIL_ADDRESS>wrote:

@sazzer https://github.com/sazzer I've setup a maven pom and used your
demo test, and I get this output:
Tests in error:
should perform addition(calculator): 2 did not equal 3
Ie, just the single test. How are you getting the "run 1" "run 2" output,
is that some maven plugin?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/kotlintest/kotlintest/issues/42#issuecomment-228907364
,
or mute the thread
<
https://github.com/notifications/unsubscribe/AAECGJmllIU-vzZ_PzL-5Fu1SIZ4KQeTks5qQF4cgaJpZM4I8nwn
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/kotlintest/kotlintest/issues/42#issuecomment-228969577,
or mute the thread
https://github.com/notifications/unsubscribe/AAtZGqRjKyqUMjM7OMIkwfowoMeBBHxWks5qQMkdgaJpZM4I8nwn
.

Just to second @sksamuel: I get the same here: Using maven-surefire-plugin:2.19.1 I get a passing and a failing test for e.g. ShouldSpec and BehaviorSpec.

With Maven this runs into https://github.com/apache/maven-surefire/blob/surefire-2.19.1/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactory.java#L360

I created a little test project https://github.com/mfriedenhagen/kotlintest-surefire including a failing travis build.
One thing: kotlintest spec tests are reported as errors, not as failures (failures are AssertionErrors every other exception is reported as error).

I added a passing test in UserTest and UserFunTest and another failing in UserShouldTest, all failing (or better erring) tests are counted twice.

Thanks for taking the time to do this. I can investigate and fix the issue now.

I think kjunithandler should rethrow AssertionError and the custom TestFailedException would be Vetter called TestErrorException (or just re- thrown with some context information or as RTE)
While we are at it: AssumptionViolatedException should probably be handled as "ignored" test to be consistent with JUnit.
Unfortunately I am not able to build the project (neither master nor 1.2.0) on Mac OS X (jdk7.0.72, gradle 2.14).

The MacOSX problem is this: https://youtrack.jetbrains.com/issue/KT-5256. io.kotlintest.matchers.be and io.kotlintest.matchers.Be(note the case) are defined in the same kotlin file.

That's fixed in release/1.3 (or should be!). Can you try that branch?

Partial fix in place. Replaced all TestFailedException with assertion errors and now errors are failures.

Ok so turns out junit docs are wrong. You shouldn't call fireTestFinished if the test fails, even though the docs say do it anyway. So fixed that.

At least in Intellij this still does not work properly, only the first failing test is reported as failed, a second failing and a successful are marked with a yellow stop sign.

Are you building from source @mfriedenhagen

@sksamuel: First of all thanks for your effort to provide kotlintest.
I have updated my test project at https://github.com/mfriedenhagen/kotlintest-surefire and hopefully made my expectations clearer (I assume the behavior shown by the plain JUnit implementation called JUnitTest should be the reference), each of the three test classes now has one failing, one erring and one succeeding test (total of 3 failing, 3 erring and 3 succeeding). kotlintest.version may be given as property while calling Maven.
My test setup (project name before the colon):
Given I cloned kotlintest and there ran: git checkout f35d75ff3f04376bd644e04ff55594225bbdb168; gradle -x signArchives install
And I checked out master of kotlintest-surefire
When I call (in project kotlintest-surefire):  mvn -B -q -Dkotlintest.version=1.2.0 clean test
Then I see

Total summary: Tests run: 9, Failures: 1, Errors: 5, Skipped: 0 (expectation Tests run: 9, Failures: 3, Errors: 3, Skipped: 0)
FunSpecTest and ShouldSpecTest class results show Tests run: 5, Failures: 0, Errors: 2, Skipped: 0 (JUnitTest: Tests run: 3, Failures: 1, Errors: 1, Skipped: 0)
Specs show failed as well as erring tests as errors, failing and erring tests are counted twice.

When I call (in project kotlintest-surefire): mvn -B -q -Dkotlintest.version=1.2.1-SNAPSHOT clean test
Then I see:

Total summary: Tests run: 9, Failures: 3, Errors: 0, Skipped: 0 (expectation would be Tests run: 9, Failures: 3, Errors: 3, Skipped: 0)
FunSpecTest and ShouldSpecTest class results show Tests run: 3, Failures: 1, Errors: 0, Skipped: 0 (JUnitTest: Tests run: 3, Failures: 1, Errors: 1, Skipped: 0)
So Specs do not show erring tests at all with Maven

I assume once this is running correctly with Maven it should behave correctly in IntelliJ as well :-)

The latest version from release/1.3 gives this output in intelliJ:

And in maven:
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running ShouldSpecTest
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.056 sec <<< FAILURE! - in ShouldSpecTest
should fail(This ShouldSpec)  Time elapsed: 0.006 sec  <<< FAILURE!
java.lang.AssertionError: true did not equal false

should err(This ShouldSpec)  Time elapsed: 0 sec  <<< ERROR!
java.lang.RuntimeException: ShouldSpec

Running FunSpecTest
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0 sec <<< FAILURE! - in FunSpecTest
should fail(FunSpecTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: true did not equal false

should err(FunSpecTest)  Time elapsed: 0 sec  <<< ERROR!
java.lang.RuntimeException: FunSpec

Running JUnitTest
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.011 sec <<< FAILURE! - in JUnitTest
shouldErr(JUnitTest)  Time elapsed: 0.001 sec  <<< ERROR!
java.lang.RuntimeException: JUnit
	at JUnitTest.shouldErr(JUnitTest.kt:12)

shouldFail(JUnitTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.AssertionError
	at JUnitTest.shouldFail(JUnitTest.kt:8)

Results :

Failed tests:
  FunSpecTest>TestBase.shouldEqual:16 true did not equal false
  JUnitTest.shouldFail:8
  true did not equal false
Tests in error:
  FunSpecTest.should err » Runtime FunSpec
  JUnitTest.shouldErr:12 Runtime JUnit
  ShouldSpec

Tests run: 9, Failures: 3, Errors: 3, Skipped: 0

That version is on maven central snapshots as 1.2.1-SNAPSHOT. I added to your pom:
    <repositories>
        <repository>
            <id>maven-central-snapshots</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>always</updatePolicy>
            </snapshots>
        </repository>
    </repositories>

and set the version to <kotlintest.version>1.2.1-SNAPSHOT</kotlintest.version>
And then it works with the latest changes. If you're happy with this, I'll release 1.3.0 as this is the last issue outstanding in the 1.3.0 branch.

Hello @sksamuel, I am pretty sure notifier.fireTestFinished has to be called for failing builds as well (see e.g. https://github.com/JetBrains/spek/blob/master/spek-core/src/main/kotlin/org/jetbrains/spek/junit/JUnitNotifier.kt or https://github.com/wgroeneveld/jasmine-junit-runner/pull/21/files).
With "IntelliJ IDEA (Community Edition) 2016.2 EAP Build #IC-162.1120.6, built on June 28, 2016" and the current SNAPSHOT from ossrh I got the following:

So I reenabled calling fireTestFinished in my build (from 446070492a5709eedf5548d86ec74c5f83f81298 changing if (!failed) to if (true)) and now the output in IntelliJ is correct but Surefire shows the wrong numbers and the additional PASS again. I digged deeply into this and the point is, that the failed status is stored in a ThreadLocal in Surefire and lateron retrieved (see https://github.com/apache/maven-surefire/blob/surefire-2.19.1/surefire-providers/common-junit4/src/main/java/org/apache/maven/surefire/common/junit4/JUnit4RunListener.java#L140).
kotlintest runs fireTestStarted and fireTestFinished on the main thread but  fireTestFailure on the Executor thread. Yesterday I fiddled around with collecting Future<Failure>s and then call  fireTestFailure on the main-thread again (did not finish this). Or "started" and "finished" should be called from the executor as well.
After thinking a while about TestBase.runTest I am uncertain whether it does behave correctly:

If you configured say 10 invocations and one of them fails you have no further information which invocation failed.
Coming from Maven I am always thinking from the "XML output perspective" here. fireTestStarted and fireTestFinished "render" the starting resp. ending element and fireTestFailure sets the state.
I do not see where any build tool would be able to deduct this for multiple invocations let alone doing these in multiple threads.

This is off-topic for this issue: What is the use case for multiple invocations? If you have differing input parameters you definitely want to create multiple notifier "transactions".

Invoking a single test case parallel with multiple threads is used to provoke multi threading issues.

@helmbold, of course, sorry. I got my Future solution working, see PR #55

I didn't get the same issues you do @mfriedenhagen. I don't know why, it should have failed for me like it did for you, but you could see the output of running the tests in the screenshots. But your patch makes sense, so that's merged and I've released 1.3.0-SNAPSHOT to central.

@sksamuel, all machines are different, (OS, JDK, IDE version, plugin version in the IDE, other installed plugins etc.) :-). I mostly trust CI-systems and like Docker containers because they eliminate a lot of difference here (but come with a baggage of their own). Thanks for merging my patch.

So we should be good for 1.3.0 now then


Add and view calendar events

We'll make those changes later – we need this branch to be pushed so others aren't blocked!


Base model completions API

Describe the feature or improvement you're requesting
README.md states that:

3rd Party LLMs with an OpenAI proxy ... are considered legitimate use cases.

Yet the ability to use the /completions API was removed in 312f502, making the library unusable for base model completions. To me, this seems counterintuitive: many providers (for example, TogetherAI) require the use of /completions for base models. I don't think the library should consider the /completions API deprecated, as it is well within the scope of third-party providers.
Additional context
No response

@satin-spirit My apologies. I don't use it myself, and wasn't aware others were still using the API. Feel free to add it back in with a PR, and I will accept it. I can't test it out myself, so it would be great if you could check that it works on the 3rd party servers that you care about.

No problem. Thank you for the quick reply ^^


New Syntax - Lambdas, Types, and Methods

Pull Request Description
This PR introduces working lambdas, type definitions, top-level bindings and methods for the new syntax.
Important Notes
This is done in parallel with the old syntax so nothing is broken during the process. It is also all staggeringly WIP, and will be cleaned up with the final PR that closes the associated task.
Checklist
Please include the following checklist in your PR:

[x] The documentation has been updated if necessary.
[x] All code conforms to the Scala, Java, Rust or Haskell style guides as appropriate.
[x] All code has been tested where possible.

Manager tests are majorly flaky on Windows right now. @mwu-tow


Implement optimizations found in lighthouse/webhint audits

DONE:

preload critical fonts
use the charset meta-tag and make sure it's first
add noreferrer/noopener to all _blank links
minify css for better performance

TODO:

correct wrong content-types
add missing content-types
check cache control headers
ensure text-based resources are compressed
add missing x-content-type headers

I trust you added the remaining findings to another ticket? Maybe mention a link to that ticket in the comments, or PR description.


Bots from LyoSU (stickerpack and messagesticker)

add stickerpack bot (including animated)
add converter from messages to pictures for stickers

Thanks you!


OSX: Rust tools: install 'rustfmt' and 'clippy'

Make installed Rust components the same as for Ubuntu. See also:
https://github.com/actions/virtual-environments/blob/f38833acec5cc1c3239474f8135b23c17cb72351/images/linux/scripts/installers/rust.sh
Description
See: https://github.com/actions/virtual-environments/issues/1916
Related issue:
Check list

[ ] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

Fixes: #1916
@maxim-lobanov

/azp run macos1013, macos1014, macos1015, macos110

Hi @maxim-lobanov @AlenaSviridenko , thanks for the quick reviews! Could you merge the pull-request please?

Hi @maxim-lobanov @AlenaSviridenko , thanks for the quick reviews! Could you merge the pull-request please?

Thank you for submitting.
We have a few critical releases coming to next week deployment. Don't you mind if we postpone merging until next week? We wouldn't like to include additional changes to next week deployment


Fix Gradle build on kotlin 1.7.0

also maybe upgrade:
# kx.datetime 0.3.3 is built with Kotlin 1.7, so currently not compatible with our build
kotlinx-datetime = { require = "0.3.2", rejects = "0.3.3" }

done


Buttons do not use inputButtonConfig

When specifying an InputButtonConfig,
neither the DeleteButton, the CancelButton nor the HiddenButton use that configuration.
Only the main number buttons use it.
This seems counter-intuitive.
To apply the same button configuration to the delete and cancel button, I have specify them myself, fully.
There is no way of applying the input config to the hidden buttons, because they are not reachable or definable through the screenLock call.
I would be very happy if the input button configuration that is passed to the screenLock call was applied to all the buttons.

it would also be great to be able to specify all the different theme configurations with the new theme extensions in flutter 3.0,
but its not necessary.
I would be fine with just being able to apply the parameter input button config to all buttons.

I would even be fine with the default settings if they didnt use MediaQuery to scale their size and would instead use fixed values,
because of right now, the scaling makes the buttons look very silly on larger screens.


Implement a path to Category picker

What

[ ] Implement a path to Category picker

Part of

#379
#505

Closing in favor of #944


Add Display and more From<T> implementations for StrBytes

Looking back at this, I added #[inline] to Display because that is how the std lib does it. But should I add #[inline] for the other From<T> impls (which is what the stdlib does), or get rid of them all?

Looking back at this, I added #[inline] to Display because that is how the std lib does it. But should I add #[inline] for the other From<T> impls (which is what the stdlib does), or get rid of them all?

I'd remove them, please. The #[inline] attribute is just a suggestion to the compiler and in practice these kinds of impls are almost certainly inlined regardless of its presence (or if they aren't, doesn't matter).

Removed 👍


azcopy login failing

Which version of the AzCopy was used?
Note: The version is visible when running AzCopy without any argument
azcopy version 10.6.0
Which platform are you using? (ex: Windows, Mac, Linux)
Windows
What command did you run?
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
azcopy login
What problem was encountered?
Failed to perform login command:
failed to login with tenantID "common", Azure directory endpoint "https://login.microsoftonline.com", autorest/adal/devicetoken: -REDACTED- occurred while sending request for Device Authorization Code: Post https://login.microsoftonline.com/common/oauth2/devicecode?api-version=1.0: dial tcp: lookup login.microsoftonline.com: getaddrinfow: A non-recoverable error occurred during a database lookup.
How can we reproduce the problem in the simplest way?
Run: azcopy login in a normal PowerShell window
Have you found a mitigation/solution?
I have not found a solution for running this command on my machine. When I run the same command on one of our Windows Servers, it works as expected. I have tried adding the tenant id, but that did not work either, and threw the same error.

@HunterGarza ,
Thanks for reaching out!
For the first look, it seems that the value of environment variable TENANT_ID is set incorrectly. Can you please check that once? Incorrect tenant id can lead to error "A non-recoverable error occurred during a database lookup."
One convenient way to get tenant id by logging on to your Azure Portal and navigating to Azure Active Directory. In the overview tab, you'll see your tenant information.
To learn more, check this and this
For more clarity, please don't hesitate to reach out to us.


bugfix: 支持bk_iam_skip

现在可以再本地配置BK_IAM_SKIP来跳过权限中心了

Codecov Report

Merging #46 (1d1ba9a) into V1.0.1rc (80b0908) will decrease coverage by 0.01%.
The diff coverage is 60.00%.

@@             Coverage Diff              @@
##           V1.0.1rc      #46      +/-   ##
============================================
- Coverage     53.11%   53.09%   -0.02%
============================================
  Files           177      177
  Lines          8695     8699       +4
============================================
+ Hits           4618     4619       +1
- Misses         4077     4080       +3

Impacted Files
Coverage Δ

apps/iam/handlers/permission.py
27.38% <60.00%> (-0.07%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80b0908...1d1ba9a. Read the comment docs.


Expose info fields

Fixes #40
Expose all optional info fields listed here to PBFIterator and SparkConnector:
message Info {
   optional int32 version = 1 [default = -1];
   optional int32 timestamp = 2;
   optional int64 changeset = 3;
   optional int32 uid = 4;
   optional int32 user_sid = 5; // String IDs
   optional bool visible = 6;
}

Will add more integration test when exporting PBF/XML is available. Create a ticket for adding bbox info if needed in future.
Reference:

https://wiki.openstreetmap.org/wiki/PBF_Format
https://github.com/openstreetmap/osmosis/blob/2219470cef1f73f5d1319c57149c84b398e767ce/osmosis-pbf2/src/main/java/org/openstreetmap/osmosis/pbf2/v0_6/impl/PbfFieldDecoder.java
https://github.com/openstreetmap/osmosis/blob/2219470cef1f73f5d1319c57149c84b398e767ce/osmosis-pbf2/src/main/java/org/openstreetmap/osmosis/pbf2/v0_6/impl/PbfBlobDecoder.java

Coverage decreased (-6.8%) to 86.119% when pulling 5f93ae9e06680b20a912c231835fa74901c84536 on ericsun95:master into f2b868271ca42bf797c8c8784a130d718cd67601 on simplexspatial:master.

I can not accept this PR. Looks like you can not wait for a proposal and talk from there and your design and quality creteria is too different from mine.
So in this case, looks like is not going to be possible to collaborate.
Please, no more privates messages.


Validate List of maps

No way to validate a list of maps
yaml:
childs:
  - name: foo
    age: 3
  - name: bar
    age: 5

not working schema
childs: list(map())

childs: list(include(childs))
---
childs:
  name: str()
  age: int()

Hi, thanks for using Yamale.
For your last example, don't forget to use quotes...for example:
childs: list(include('childs'))
---
childs:
  name: str()
  age: int()

As for this schema:
childs: list(map())

It validates correctly for me with the latest master and with 2.0.1. What version are you using and what is the error you're getting?

yes, the quotes made my day
thanks


Modules API design

I see a lot of implementation work in here already, could you save that work somewhere else? This PR should just be for API design


Feat: Added one-off and sweeper bot frontmatter checks

This PR adds a one-off script to update all existing PRs with an applicable comment/label related to missing/invalid frontmatter information for guide articles.  This same logic has now been added to the sweeper.js file.  These additions close issue #12.
This PR also restructures the repo in an attempt to streamline and organize the code base.
Also, all the one-off scripts now have comments at top of script explaining each script's purpose.
Added a few new libraries and updated to a newer version of octokit (15.8.0), so package.json has been updated.
The one-off script findFailures.js has been modified to work as long as we can figure out how to implement the Travis API (recently installed).
This PR implements a BREAKING CHANGE.

@raisedadead Take a look at the changes I implemented.  I tested both the one-off and the sweeper on a test run on the first 1000+ PRs and validated that the log results documented all of the relative guide frontmatter issues were identified and would be added to the PR.  Once this PR is merged, we can run another test run to validate the many changes I made did not break anything (linux-wise).


Fix strings coming from json

Currently the JSON unmarshalling leaves \uXXXX and \n\r\t sequences of characters as is, instead of replacing them with corresponding runes. I'm not sure it's a activitypub.go/jsonld problem or not.

Fixed in 9efb6bed


[AutoPR cognitiveservices/data-plane/CustomVision/Prediction/cognitiveservices/data-plane/CustomVision/Training] Update the output-folder to match azure-sdk-for-net paths

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/2529

Codecov Report

Merging #2044 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2044   +/-   ##
=======================================
  Coverage   53.06%   53.06%
=======================================
  Files        4774     4774
  Lines      119483   119483
=======================================
  Hits        63406    63406
  Misses      56077    56077

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f896c05...189a6c9. Read the comment docs.

C# only PR


Where socket.headers from version 3.0.0? )

Where socket.headers from version 3.0.0? )

https://github.com/daltoniam/Starscream#custom-headers
It has been replaced by URLRequest in the init. I felt that was more inline with standard Apple APIs.

For the lazy (which includes me so ... no shame 😄 )
Find in your code where you call
socket = WebSocket(url: serverUrl, protocols: proto)
socket.headers = headers

Replace with:
var request = URLRequest(url: serverUrl)
request.timeoutInterval = 5
for (key,value) in headers {
    request.setValue(value, forHTTPHeaderField: key)
}
socket = WebSocket(request: request, protocols: proto)


GPG issue during/after install

zopen install gpg results in message:

mkdir: FSUM6404 directory "/u/jenkins/.gnupg": EDC5129I No such file or directory.
./setup.sh 47: FSUM7343 cannot open "/u/jenkins/.gnupg/gpg-agent.conf" for output: EDC5129I No such file or directory.
See upstream package issue:
https://github.com/ZOSOpenTools/gpgport/issues/6

Expanding on the comments in the issue linked above:
After install  (zopen install gpg) , create a gpg-agent.conf file in your $HOME/.gnupg directory
pinentry-program $ZOPEN_ROOTFS/usr/bin/pinentry-tty
log-file //.gnupg/gpg-agent.log  or  $ZOPEN_ROOTFS/var/log/gpg-agent.log  possibly
gpg-agent can be restarted within a zopen env (using commands such as:
::>gpg-connect-agent --agent-program which gpg-agent reloadagent

Tried zopen install gpg and also zopen upgrade gpg. it worked fine


[DWC] Update rename credential modal

Simply updating name and/or description of credential metadata will update name/description of credential metadata without actually impacting original credential data.
Reference: Updating Credential Metadata

This is a duplicate of https://github.com/trustbloc/wallet/issues/1344

UI for renaming credential need to be implemented according to the designs.

This has been implemented


Remove one flag from formatoptions at a time

See :help remove-option-flags. setlocal formatoptions-=cro searches 'formatoptions' for exact "cro", so "c", "r", or "o" flag won't be removed if 'formatoptions' is "cor", "cr", "crto", etc.

Thanks, merged.


infiniteQuery continues to refetch data even there are no more observers

Describe the bug
infiniteQuery continues to iterate and fetch all pages even after there are no more observers to consume the data. If the infiniteQuery contains a large set of pages (i.e. 100) this causes unnecessary network load and keeps references to variables for a long time.
To Reproduce
Steps to reproduce the behavior:
Open sample in CodeSandbox

Allow all pages to finish loading (up to 15 pages).
Open Console tab (bottom right) in CodeSandbox or browser console
Sample app will print Fetched page every time an API call is simulated
Click on Toggle button to unmount the component which calls useInfiniteQuery
Notice even though there are no more observers which consume the components query the infiniteQuery continues to fetch pages in the background.

Expected behavior
After all observers (i.e. React components) which consume an infiniteQuery go out of scope the infiniteQuery should stop fetching subsequent pages.
Screenshots

Desktop (please complete the following information):

OS: all
Browser: all
Version: all

Smartphone (please complete the following information):

Device: all
OS: all
Browser: all
Version: all

Additional context
Possible code that controls this behaviour

same issue, I'm using nextjs and following the example

Gave a staleTime of 1000 * 60 * 60 but then also it's fetching

Same issue.

Same issue.


Bonsai Badge

Similar to #9 , this issue to is get a badge into the repo so that folks know where to download the asset (a la bonsai).

Working badge:

Once Shields updates:
https://img.shields.io/badge/Bonsai-Download%20Me-brightgreen.svg?colorB=89C967?logo=sensu
We should then be able to do something like:
[![Bonsai Asset - CHANGEME](https://img.shields.io/badge/Bonsai-Download%20Me-brightgreen.svg?colorB=89C967?logo=sensu)](https://bonsai.sensu.io/assets/CHANGEME/CHANGEME)

Badges are working!

I'll put in a PR to add this

Closed by #13


Bad "value" for KUBERNETES_PUBLIC_ADDRESS setting.

Replace :
KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  --region $(gcloud config get-value compute/region) \
  --format 'value(name)')

with
KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  --region $(gcloud config get-value compute/region) \
  --format 'value(address)')

The reason is that when I ran the snippet it was not returning the IP address, rather kubernetes-the-hard-way.

Both work, but I'll set it to address for consistency. This is fixed on the 1.10.2 branch.