Error combining string template formatting options

String template formatting options cannot be combined, syntax error.
Causes https://github.com/exercism/abap/issues/102
Paste into https://transpiler.abaplint.org and uncomment line to reproduce:
data h type i value 2.
data m type i value 5.
write / |{ h WIDTH = 2 }:{ m PAD = '0' }|.
"write / |{ h WIDTH = 2 PAD = '0' }:{ m }|.

#684

@pokrakam try again

this fails,
    result = |{ ls_time-hours PAD = '0' WIDTH = 2 ALIGN = RIGHT }:{ ls_time-minutes PAD = '0' WIDTH = 2 ALIGN = RIGHT }|.

reduced to below, expect output "02:02"
DATA result TYPE string.
DATA hours TYPE i.
DATA minutes TYPE i.
hours = 2.
minutes = 2.
result = |{ hours PAD = '0' WIDTH = 2 ALIGN = RIGHT }:{ minutes PAD = '0' WIDTH = 2 ALIGN = RIGHT }|.
WRITE result.

@pokrakam try again, latest version

closing, no reply


DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12

I get the following warning in pytest:
venv/lib/python3.10/site-packages/msal_extensions/cache_lock.py:7: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.version import LooseVersion

Well, I'm using the latest version fråm pypi, I do not have an extra environment to help you with testing.

You do not have to use an extra environment. Your existing environment can pull the not-yet-released version from this official repo by:
pip uninstall msal-extensions
pip install git+https://github.com/AzureAD/microsoft-authentication-extensions-for-python.git@dev

Afterwards, you can always revert back to the existing 1.0.0 version by a normal pip install msal-extensions.


dependabot[bot] Bump Roslynator.Analyzers from 4.6.0 to 4.6.1 in /webapi (#553)

Describe the bug
A clear and concise description of what the bug is.
There is a problem with the deployment of the new version, and by default it stays at Looking for your backend without any action

Duplicate of #523 and #545
This is due to the backing storage running out of indexes.
Will add more information in #545
Will also make error more obvious.


Add basic support for VS2017

This is best reviewed each commit individually. I also put a lot of information in the commit messages

@pmiossec Did you have chance to look at the commits already? For me, the only open issue is the one about the quite noisy logging right now. This can either be solved via

Tone down this logging so it only appears in the trace file
Introduce a static class type specific member to collect the data which shall be shared over all TfsHelpers of the same type (e.g. the VS2017 TfsHelpers)
Configure the TfsHelper to be a singleton in the DI container so we don't create it multiple times

Right now I am leaning to point 2, as it would be quite a small change and probably only touches the last two commits
('Add minimal VS2017 support VS2017 using a hardcoded location ...' , 'Add support for VS2017')
So I would be interested if the prior commits are structured correctly, e.g. split in such a way they are understandable or if they need further refinements or if I should tackle this differently to enable easier review

Did you have chance to look at the commits already?

Just a very light look to see if something evident stand out.
I have to spend more time on it (but time is something I lack at the moment 😢  )
Note: You should rebase to solve a conflict. I don't know if you want to do it now or later...

Thx for the int, I rebased the PR to solve the conflict in the doc/release-notes/NEXT.md file

I fixed the typos and added a commit which makes the TfsHelper a singleton.
So there is IMHO only the one unresolved comment from @spraints, which I am leaning to do differently as this will not work for VS2019 anyhow

I fixed the typos and added a commit which makes the TfsHelper a singleton.
So there is IMHO only the one unresolved comment from @spraints, which I am leaning to do differently as this will not work for VS2019 anyhow

Scratch that last commit, It fails the unit tests, at least locally. Not sure why appveyor didn't fail

Scratch that last commit, It fails the unit tests, at least locally. Not sure why appveyor didn't fail

I force pushed to fix the unit tests in the last commit.

I force pushed to fix the unit tests in the last commit.

From my end, this would be good to go in

From my end, this would be good to go in


Fix Clippy error

Modify deprecated function in chrono

Codecov Report
Patch coverage is 42.47% of modified lines.

:exclamation: Current head ac526b7 differs from pull request most recent head 5f925f5. Consider uploading reports for the commit 5f925f5 to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Files Changed
Coverage

src/producer.rs
0.00%

src/syslog.rs
0.00%

src/zeek/network.rs
93.75%

src/migration.rs
100.00%

src/migration/network.rs
100.00%

src/operation_log.rs
100.00%

src/zeek.rs
100.00%

:loudspeaker: Thoughts on this report? Let us know!.


Create gentx-c37494a68d3a75ae61638d7fd4c9b2a2076446a8.json

MY ARCHWAY_MONIKER="Martorit9nin"

Why was the summit closed? Is something wrong?


Update faq.yml

Description

As already mentioned in #ISSUE_NUMBER, this PR tackles:

...
...
...

In particular, the ...
Checklist

[ ] I followed the indications in the CONTRIBUTING
[ ] The documentation related to the proposed change has been updated accordingly (also comments in code).
[ ] Have you written new tests for your core changes, as applicable?
[ ] Have you successfully ran tests with your changes locally?
[ ] Ready for review! :rocket:

Fixes

Fixes #

@bfabio scusa tanto, questa PR annulla la precedente che chiudo


BaseSolver.solve() multiprocessing is broken on system's defaulting to spawn context for windows

Follow up to #3975

It's worth noting that this only fixes https://github.com/pybamm-team/PyBaMM/issues/3974 for darwin systems and protects linux systems from the default context change in Python 3.14. Windows systems will still require the main guard discussed in https://github.com/pybamm-team/PyBaMM/issues/3974.

Labelled the issue with high priority so that we remember to get this done before v24.5rc0

@BradyPlanden Any update on this ticket?

Removed this from blocking the release based on comments from the dev meeting and on the issue linked above


WIP Handle SIGTERM and SIGINT

Fixes #67.
For some reason this does not catch SIGINT. Need another debugging session.

Closing in favor of #72.


Update docs

Codecov Report

Merging #2 (2ad16aa) into master (3dd33a9) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #2   +/-   ##
=======================================
  Coverage   50.57%   50.57%
=======================================
  Files           5        5
  Lines         261      261
=======================================
  Hits          132      132
  Misses        129      129

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3dd33a9...2ad16aa. Read the comment docs.

Codecov Report

Merging #2 (2ad16aa) into master (3dd33a9) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #2   +/-   ##
=======================================
  Coverage   50.57%   50.57%
=======================================
  Files           5        5
  Lines         261      261
=======================================
  Hits          132      132
  Misses        129      129

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3dd33a9...2ad16aa. Read the comment docs.


Error taking gradient through ops.index_update.

This error seems to occur when the indices are computed from the input being differentiated. Here's a simple repro:
def f(x):
  y = np.zeros((3,))
  indices = np.array(x - 1, np.int32)
  y = ops.index_update(y, indices, x)
  return np.sum(y)
x = np.array([1.0, 2.0, 3.0])
grad(f)(x)

Thanks, Sam! By the way, in addition to an excellent short repro like this one, it's helpful to include the error message and traceback.

Sure thing! Here's the trace,
/usr/local/lib/python3.6/dist-packages/jax/api.py in grad_f(*args, **kwargs)
    236   def grad_f(*args, **kwargs):
    237     if not has_aux:
--> 238       _, g = value_and_grad_f(*args, **kwargs)
    239       return g
    240     else:

/usr/local/lib/python3.6/dist-packages/jax/api.py in value_and_grad_f(*args, **kwargs)
    280     f_partial, dyn_args = _argnums_partial(f, argnums, args)
    281     if not has_aux:
--> 282       ans, vjp_py = vjp(f_partial, *dyn_args)
    283     else:
    284       ans, vjp_py, aux = vjp(f_partial, *dyn_args, has_aux=True)

/usr/local/lib/python3.6/dist-packages/jax/api.py in vjp(fun, *primals, **kwargs)
    896   jaxtree_fun, out_tree = pytree_fun_to_jaxtupletree_fun(fun, in_trees)
    897   if not has_aux:
--> 898     out_primal, out_vjp = ad.vjp(jaxtree_fun, primals_flat)
    899   else:
    900     out_primal, out_vjp, aux = ad.vjp(jaxtree_fun, primals_flat, has_aux=True)

/usr/local/lib/python3.6/dist-packages/jax/interpreters/ad.py in vjp(traceable, primals, has_aux)
    103 def vjp(traceable, primals, has_aux=False):
    104   if not has_aux:
--> 105     out_primal, pval, jaxpr, consts = linearize(traceable, *primals)
    106   else:
    107     out_primal, pval, jaxpr, consts, aux = linearize(traceable, *primals, has_aux=True)

/usr/local/lib/python3.6/dist-packages/jax/interpreters/ad.py in linearize(traceable, *primals, **kwargs)
     92   in_pvals = (pe.PartialVal((None, pack(primals))),
     93               pe.PartialVal((core.AbstractTuple(tangent_avals), core.unit)))
---> 94   jaxpr, out_pval, consts = pe.trace_to_jaxpr(jvpfun, in_pvals)
     95   pval_primal, pval_tangent = unpair_pval(out_pval)
     96   aval_primal, const_primal = pval_primal

/usr/local/lib/python3.6/dist-packages/jax/interpreters/partial_eval.py in trace_to_jaxpr(fun, pvals, **kwargs)
    399   with new_master(JaxprTrace) as master:
    400     fun = trace_to_subjaxpr(fun, master, instantiate)
--> 401     jaxpr, (out_pval, consts, env) = fun.call_wrapped(pvals)
    402     assert not env
    403     del master

/usr/local/lib/python3.6/dist-packages/jax/linear_util.py in call_wrapped(self, *args, **kwargs)
    145
    146     del gen
--> 147     ans = self.f(*args, **dict(self.params, **kwargs))
    148     del args
    149     while stack:

<ipython-input-134-2ac8abff172b> in f(x)
      2   y = np.zeros((3,))
      3   indices = np.array(x - 1, np.int32)
----> 4   y = ops.index_update(y, indices, x)
      5   return np.sum(y)

/usr/local/lib/python3.6/dist-packages/jax/ops/scatter.py in index_update(x, idx, y)
    371          [1., 1., 1., 6., 6., 6.]], dtype=float32)
    372   """
--> 373   return _scatter_update(x, idx, y, lax.scatter)
    374
    375 def segment_sum(data, segment_ids, num_segments=None):

/usr/local/lib/python3.6/dist-packages/jax/ops/scatter.py in _scatter_update(x, idx, y, scatter_op)
     94           inserted_window_dims=tuple(range(len(idx))),
     95           scatter_dims_to_operand_dims=tuple(range(len(idx))))
---> 96       return scatter_op(x, stacked_idx, y, dnums)
     97     elif np._is_advanced_int_indexer(idx):
     98       # TODO(mattjj, phawkins): one of us is going to implement this case someday

/usr/local/lib/python3.6/dist-packages/jax/lax/lax.py in scatter(operand, scatter_indices, updates, dimension_numbers)
    753       operand, scatter_indices, updates, update_jaxpr=jaxpr,
    754       update_consts=consts, dimension_numbers=dimension_numbers,
--> 755       updates_shape=updates.shape)
    756
    757 def index_take(src, idxs, axes):

/usr/local/lib/python3.6/dist-packages/jax/core.py in bind(self, *args, **kwargs)
    145
    146     tracers = map(top_trace.full_raise, args)
--> 147     out_tracer = top_trace.process_primitive(self, tracers, kwargs)
    148     return full_lower(out_tracer)
    149

/usr/local/lib/python3.6/dist-packages/jax/interpreters/ad.py in process_primitive(self, primitive, tracers, params)
    250           "Forward-mode differentiation rule for '{}' not implemented"
    251           .format(primitive))
--> 252     primal_out, tangent_out = jvp(primals_in, tangents_in, **params)
    253     return JVPTracer(self, primal_out, tangent_out)
    254

/usr/local/lib/python3.6/dist-packages/jax/lax/lax.py in _scatter_jvp(primals, tangents, update_jaxpr, update_consts, dimension_numbers, updates_shape)
   3050   g_operand, g_scatter_indices, g_updates = tangents
   3051   dnums = dimension_numbers
-> 3052   assert g_scatter_indices is ad_util.zero
   3053
   3054   if g_operand is ad_util.zero and g_updates is ad_util.zero:

AssertionError:

Strangely I don't see assert g_scatter_indices is ad_util.zero in the git history for lax.py's _scatter_jvp`, and your snippet works fine for me on master. Are you on a branch?

Oh, intriguing! I guess this is a good reason to give the exact steps for the repro. This error occurs using the following invocation from OSS colab:
!pip install --upgrade -q https://storage.googleapis.com/jax-wheels/cuda$(echo $CUDA_VERSION | sed -e 's/\.//' -e 's/\..*//')/jaxlib-$(pip search jaxlib | grep -oP '[0-9\.]+' | head -n 1)-cp36-none-linux_x86_64.whl
!pip install --upgrade -q jax

However, if I use master locally then everything works. Is it possible that the version PyPi links to is out of date?
Anyway, thanks for the help!

I think this was fixed by https://github.com/google/jax/pull/1056
We just need to make a new PyPi release.

Ohh, whoops, I was looking at git blame and wasn’t able to see code that was added and then removed.


Don't run fetch_worker_metadata for scratch builds

Signed-off-by: Tim Waugh<EMAIL_ADDRESS>No unit tests because its presence will be tested properly once the new arrangement including the plugin is created.

Actually I'll close this as a more complete PR is in the works elsewhere.


Fix invalid null delegate usage attempts in media clusters

Problem
@andreilitvin  pointed out that in media cluster code quote: "Delegate null check was not valid: even if null check failed, delegate use was still attempted."
Change overview
see above
Issue(s) fixed:
20683
Testing
Run ./scripts/run_in_build_env.sh \                     "./scripts/tests/run_test_suite.py \                      --chip-tool ./out/debug/standalone/chip-tool \                      run \                      --iterations 1 \                      --all-clusters-app ./out/debug/standalone/chip-all-clusters-app \                      --tv-app ./out/debug/standalone/chip-tv-app \                   "

@lazarkov - changes in feature map should be independent of cpp error handling logic. Separate PRs for each would make more sense.

It came as an additonal request from @chrisdecenzo. I believe it's per spec to have feature map of network commissioning set to 4. @chrisdecenzo can confirm.
I understand the logic to move this change to new PR, in case we want to roll the change back. But per my understanding, this should be the default value.


feat: introduce AWS KMS types.MessageTypeRaw for AWS KMS signing operations

Hi smallstep/crypto reviewers! :wave:
This is my first contributing PR, please feel free to provide any feedback! I'm happy to change things up where required!
Name of feature:
Implement a means for callers to specify types.MessageTypeRaw in AWS KMS Sign operations.
Pain or issue this feature alleviates:
Currently, the awskms.Sign operation exclusively supports the signing of messages, where the message is expected to be a types.MessageTypeDigest.
This limitation prevents callers from being able to specify that their message is un-hashed, and thus unable to inform the AWS KMS API that the Sign operation should have a types.MessageType of types.MessageTypeRaw (RAW).
Limitations
The objective of this package, (and the other Signing packages) is to provide an implementation of the crypto.Signer interface that performs the relevant cloud based signing operations.
As a part of the crypto.Signer interface, the crypto.SignerOpts interface is include as the third and final argument to the Sign() method.
A perk of the crypto.SignerOpts interface is the promise that:
// HashFunc returns an identifier for the hash function used to produce
// the message passed to Signer.Sign, or else zero to indicate that no
// hashing was done.
HashFunc() Hash

However, within the awskms implementation of the crypto.Signer interface, this promise cannot be fulfilled, as the AWS KMS Sign operation requires a types.SigningAlgorithmSpec must be provided when performing an AWS KMS Signing operation.
AWS KMS Go SDK v2 Documentation:
// Specifies the signing algorithm to use when signing the message.
//
// Choose an algorithm that is compatible with the type and size of the specified
// asymmetric KMS key. When signing with RSA key pairs, RSASSA-PSS algorithms are
// preferred. We include RSASSA-PKCS1-v1_5 algorithms for compatibility with
// existing applications.
//
// This member is required.
SigningAlgorithm types.SigningAlgorithmSpec

The only exclusion to this limitation is when signing with ECDSA keys. As AWS does not enable the user to specify arbirtary signing algorithms, but rather the specified signing algorithm for that ECDSA key type. See The ECDSA AWS Key Spec
As such, when calling the awskms Sign() we must always provide a non-zero crypto.Hash as our crypto.SignerOpts argument.
Mitigation
To mitigate this limitation, a new AWSOptions type is introduced to the awskms package.
This AWSOptions type implements the crypto.SignerOpts interface, and allows callers to specify whether or not the message is RAW.
Additionally, it posses a crypto.SignerOpts field Options where callers can provide the underlying crypto.Hash that can be matched with the appropriate types.SigningAlgorithmSpec that is expected by the AWS KMS Sign API.
Approach
I have opted to use a more generic AWSOptions struct with the Raw field as a boolean to indicate if the types.MessageType is types.MessageTypeRaw or not. I did consider using a struct such as AWSRawMessageOptions and removing the field (since the only logical difference is the truthy value of the Raw field.) However I decided against it such that we open ourselves up to adding or removing additional options in the future.
Why is this important to the project (if not answered above):
Allow callers to have the additional flexibility of using both types.MessageTypeRaw and types.MessageTypeDigest when using the awskms Signer.
Is there documentation on how to use this feature? If so, where?
GoDoc comments on the awskms.AWSOptions struct has been introduced.
In what environments or workflows is this feature supported?
AWS Environments that leverage AWS KMS Sign operations.
In what environments or workflows is this feature explicitly NOT supported (if any)?
Non-AWS Environments.
Supporting links/other PRs/issues:
N/a

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @hslatman, by the looks of the contributors insights you might be an appropriate reviewer of this PR, please let me know if you have any feedback regarding the change 🙏

I think the approach looks good. We will look at it tomorrow and assign a reviewer.

@alexatcanva thank you for your contribution. Can you (re)check the CLA? Let me know if it's not an option for you.


get stuck when creating the fourth rtsp-sink

May I ask if this program can create up to 3 rtsp-sinks and get stuck when creating the fourth rtsp-sink, thanks!

@shancaidazf are you reporting an issue with more than 3 rtsp-sinks? Or are you just asking if DSL can support more than 3?
The description of your issue -- "get stuck when creating the fourth rtsp-sink"" -- sounds like you are reporting a bug?

Thank you very much for your reply. I want to know how to create multiple rtspsins. When I create more than 3 rtspsinks during testing, the interface will get stuck. Am I using the wrong method? What should I do? Thank you

@shancaidazf I'm able to create a pipeline with 4 rtsp-sinks and can run it on my Jetson device.  In order for me to help you further I need more information.
What device are you using: Jetson or dGPU?
What version of Deepstream are you using.
How are you defining your rtsp-sinks? What values are you using for udp_port and rtmp_port for each?
Most importantly I need logs.  Please set the following environment variables.
export GST_DEBUG=1,DSL:4
export GST_DEBUG_FILE=./dsllogs.txt

Please send me logs for both cases

3 rtsp-sinks working
4 rtsp-sinks getting stuck

i am running on RTX3070, DGPU environment. deepstream6.2.  the following is my logs.

this is the full logs. https://github.com/shancaidazf/111/blob/main/DSLLOGS.txt

@shancaidazf looks like we'll need to bring this up with NVIDIA.  According to the table here https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new you should be able to handle 5 concurrent sessions with your dGPU.
Can I ask you to get me one more log to share with NVIDIA.  Please set the log level to.
export GST_DEBUG=4,DSL:1

They will want to see more of there info and less DSL.

when i set the log level to  this :  export GST_DEBUG=4,DSL:1     the log file:

when i set the log level to this: export GST_DEBUG=1,nvv4l2h265enc:4
DSLLOGS.txt

@shancaidazf please see https://forums.developer.nvidia.com/t/errors-when-using-more-than-3-nvv4l2h265enc-plugins/256281
and try as suggested... upgrade to latest driver version

@shancaidazf can you let me know if their suggestion worked so I can close the issue with NVIDIA, thanks

@shancaidazf can you let me know if their suggestion worked so I can close the issue with NVIDIA, thanks

I just gave it a try and was surprised to find that he succeeded. Thank you very much！

firstly,thanks to your help.  howerver  i meet a new problem now.  when i try add a rtsp-source and a url-source to a pipeline and use the tilter , it doesnot work, do you know why?


select network param for Fortmatic

you can now provide an optional parameter for Fortmatic which allows developers to select the network if no param is provided mainnet is selected

Also we need to look into our prettier configs because this PR was supposed to be like 3-4 lines and its over 70

Also we need to look into our prettier configs because this PR was supposed to be like 3-4 lines and its over 70

Yes, I noticed that. I'm pretty much just using the default Prettier configuration for VS Code. Maybe you can you send me your config to add it.

I have this setting on my VS Code that adapts the Prettier rules to the project's Eslint rules
// settings.json
{
    ...
    "prettier.eslintIntegration": true,
}

This is a Typescript project however so it uses Tslint instead. Do you know if it's possible to add .prettierrc config file that simply follows the Eslint or Tslint rules instead of the default?

Hmm it doesn't seem like it's possible according to this issue.
https://github.com/prettier/prettier-vscode/issues/292
I will investigate a bit further

It looks like the solution might be to use tslint-config-prettier:
https://github.com/prettier/tslint-config-prettier

Just tested a prettier configuration to fix this @crisgarner, could you test the prettier-config branch on your side?

Fixed on commit fdcdc69


How to get all coupons with relation (redeemer)

I want to get all coupons with redeemer
like:
Coupon::with(['redeemer'])->get();
thanks.

Hi @stanley-hypo
I noticed the inconsistency in the DefinesModelRelations trait, as this trait was supposed to define relations, not the concrete models. The fix is PRed already and would ship in the next major release of the package. Delay is because it is a breaking change for those who override that method. For now, I recommend creating a separate model, and extend from MichaelRubel\Couponables\Models\Coupon, then add a method:
public function redeemerRelation(): MorphTo
{
    return $this->morphTo();
}

And you'll able to eager load it:
$coupons = Coupon::with('redeemerRelation')->get();

I hope this helps! 🙂

thanks so much


Use Redux to separate & manage the state

[x] create new branch for redux
[x] install redux dependencies?
[ ] create resusable file/class structure
[ ] separate login/postlogin state into redux actions
[ ] update new states via packets

w/ handleActions from redux-actions and update from 'react-addons-update':
export default handleActions({
    [actions.SOME_ACTION]: (state, { payload }) => update(state, {
        listings: {$set: payload}
    })
})

Also, look into combineReducers from 'react'

Should also be using redux-thunk, so that I can call dispatch from the actions directly


add cmake check for libattr

This adds a cmake find_package check for libattr, which is used for attr_copy_action when copying extended attributes.  When xattr support is enabled, libattr is marked as required.  However, this also adds compile-time checks in the code to disable libattr code if is was not found so that it could be optional rather than required.

@ofaaland , would you please review this when you get a chance?  If you have suggestions, please let me know.
In case someone doesn't have libattr installed, I protected the libattr-related lines with a HAVE_LIBATTR check.  For now, I've left the cmake check as required, but we could likely back that off to make it optional.
I also dropped the input ctx variable and replaced it with NULL since we weren't using it anyway.  To use, I think we'd want to fill in the fields of the struct with pointers to functions, as in the example here:
https://github.com/philips/attr/blob/1cc88bd4c17ef99ace22c8be362d513f155b1387/examples/copyattr.c#L89

@adammoody To be clear, the current (and intended) behavior is that libattr is required when ENABLE_XATTRS=ON?

@daltonbohning , yes, that's the goal, but we're assuming that one has libattr available.  If we find common setups that use xattrs but do not have libattr installed, I think we could do more work to make libattr optional.

@daltonbohning , yes, that's the goal, but we're assuming that one has libattr available. If we find common setups that use xattrs but do not have libattr installed, I think we could do more work to make libattr optional.

@adammoody Thanks. Luckily, so far it seems our (DAOS) setups do have libattr installed/available.


Add metadata schemas to plugin.EntrySchema

I'm working on a refactor PR to clean-up some of the EntrySchema stuff. That will make this easier to implement, so am closing this PR until that one's finished.


Add rudimentary support for DWARF5

This allows cv2pdb to process executables produced by the mingw-w64 version of GCC v11.x. The symptoms of the fixed issue would look like this:
error: cannot add line number info to module

We will most likely have to run with this patch to allow building Git for Windows henceforth, as we use the mingw-w64 version of GCC provided by the MSYS2 project, and they recently upgraded to GCC v11.x (where it seems impossible to force an older DWARF version via -gdwarf-<version> -gstrict-dwarf).
This fixes https://github.com/git-for-windows/git/issues/3488.

Thanks for your contribution. I'll have to trust you on this change, but I see that older DWARF versions should be unaffected.

Heh, I had hoped that somebody more knowledgeable than I would double-check my work :-)
Having said that, I went by the official specs and verified that it works using a freshly-built git.exe, so I am reasonably confident that it's doing what it should be doing.


BATS libexec dir on install

I installed the bats script in /usr/local (following the instructions).  The tool was not working, looked like it couldn't find the libexec path scripts.  I also had a second issue where bats tests would not find the script I was trying to run.
For the first issue (libexec) I updated the bats script (in /usr/local/bin/bats) line 55:
Old Line:
 BATS_LIBEXEC="$(abs_dirname "$0")"
New Line:
BATS_LIBEXEC="$(abs_dirname "${BASH_SOURCE[0]}")/../libexec"
This would probably work no matter if bats is installed centrally or if it is pulled into a git project since it's all based on relative path to where the bats script is installed.
The second issue was when I was running bats on scripts in my project.  The bats test script could not find my scripts for testing.  For example:
cd <my project>      --- my .sh scripts and .bats test script is here
/usr/local/bin/bats mytest.bats   -- run the bats test

It wouldn't find my script.  I updated my bats test like this:
Old:
  run myscript.sh
New:
  run ${BATS_TEST_DIRNAME}/myscript.sh
This seems to be something to do with the working directory.  I'm not sure but it might be worth while to add something like:
cd ${BATS_TEST_DIRNAME} to the bats-exec-test script, maybe between lines 57 and 58?
If the path needs to be saved, another option to save it off in a variable or use pushd / popd.
The instructions for bats seemed to imply that it would find scripts in the current working directory that bats is executed from.
Anyway, let me know if I can provide any other details.
Thanks,
Gary

This project is no longer being actively maintained by its original auther (see https://github.com/sstephenson/bats/issues/236).
You might want to migrate to the community maintained fork bats-core.


Changes in Map getAll/putAll implementation

Apply the changes in https://github.com/hazelcast/hazelcast/pull/6555 and https://github.com/hazelcast/hazelcast/pull/6480.

https://github.com/hazelcast/hazelcast-cpp-client/pull/11 implements the request, hence it can be closed.


Re-establish connections in the case of Master/Slave failover

The master-slave docs say:

The connection needs to be re-established outside of lettuce in a case of a Master/Slave failover or topology changes.

So that users don't have to implement this themselves, it'd be great if lettuce could do this transparently.

Hi @tomfitzhenry,
thanks for the ticket. Could you tell me more about your use case?
Static-Master-Slave connections are designed intentionally that way - AWS ElastiCache reports internal IP addresses and the connection point details need to be user-provided.
Regular Master-Slave connections could be self-discovering. That would lead to a crawler-like behavior in which lettuce could discover all members of the Master-Slave setup, but it comes with some challenges:

Any discovered node would live either forever or requires a time-to-live. Which one is the correct approach?
How to discover topology changes? Polling? Use connection events  (i.e. connection disconnected)?

From tests using a workaround outside of Lettuce (as suggested), it seems the following would be a great help if it could be included in Lettuce to help...

Let’s assume discovery is independent of this problem (maybe app will update nodes on query to AWS API), in this case, updating the nodes Lettuce is using would be great StatefulRedisMasterSlaveConnection.updateNodes(List<Iterable>).
Topology refresh:
Polling: regularly for disconnected nodes (e.g. Default 1/second) that the client doesn’t use until back and also for topology changes less regularly (e.g: Default 1/30 seconds), because there’s another way of identifying topology change…
Read/Master exceptions: saw exceptions about reading from master and writing to a slave, which might be good candidate for topology refresh, perhaps not on each discovery (if master is down, there will be nowhere to write to until it’s back or a node promoted) so perhaps rate limiting topology refreshes in this instance (e.g.: default 1/second) but if you think Redis can handle the extra check on per command in this situation, maybe refresh on each fail is okay

Thanks for the detailed description. lettuce provides a similar facility for Redis Cluster (listening to events during operations; adaptive topology refresh).
I think it would make sense to expose the refresh trigger API and accept a custom TopologyProvider. The MasterSlave API is built on top of these components. Node details come from inside of the TopologyProvider and should not be set externally.
Users are able to build their own TopologyProvider and can provide RedisNodeDescriptions. The only point which requires a bit more thought is signaling demand for a topology refresh.

@mp911de
Hi Mark,
Today, I'm trying to configure the Lettuce as Redis client with Master/Slave topology, but I did not find any good article talking about the configuration.
I read the documentation and is not clear how can I re-establish the connection with AWS in a failover case, like... I need to switch the roles (master/slave) between the hosts using the static topology? How can I re-create the connection? There is some easy way to do this using Spring-Boot?
Do you have some kind of an implementation?
Thanks!

Hey @jaimebrolesi
the short answer is: There is nothing available.
Longer version: AWS Elasticache Master/Slave (and Master/Slave as known from Redis, without Sentinel) do not provide any details over topology changes. There's no possibility to discover that a failover (or reconfiguration) has happened. I'm not terribly familiar with AWS, maybe AWS provides events that can be captured in such case.
Because Master/Slave changes are typically an operational task that is performed outside of Redis, we made the assumption that these things don't happen when an application is running. Changing a Master/Slave setup is basically not constrained in any way, so we can't assume that the currently connected nodes will persist upon a change. Failovers/changes require an application restart to take effect of the new configuration.

I think it's possible because AWS give us a TOPIC with the Elasticache events.
We can create a re-connection policy with INFO replication command based on failover event.
This is the harder way I guess because Lettuce will need to carry AWS SDK for SQS/SNS consumption.
OR
We can define a new topology strategy for AWS, if any command timeout for 3 times (configurable), we can start a re-connection policy using the same INFO replication command strategy from MasterSlaveTopologyProvider on getNodes() method. This is possible because AWS give to us 2 kinds of connection a loadbalanced (DNS issues) and individual one (a hostname for each node)
What do you think?! I can help with AWS explanation or coding :) hehehe.

I think we can increase visibility/provide an SPI to either trigger a refresh from outside or supply endpoint details so AWS-specific tooling can contribute to the topology/topology update. We will not introduce Cloud-specific (in this case AWS-specific) functionality to Lettuce that uses non-Redis infrastructure.

I'm investigating a similar issue to AWS elasticache but I'm also experiencing problems relating to DNS caching. In our use case it seems like hostname resolutions are cached forever regardless of the DNS resolver used because of the behaviour of SocketAddressResolver.
We're using lettuce 5.0.4 in Spring Boot 2.0.1
We setup our RedusConnectionFactory in the following way
  ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh(Duration.ofSeconds(30))
                 .enableAllAdaptiveRefreshTriggers()
                .build();
        ClientOptions clientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(topologyRefreshOptions)
                .build();
        ClientResources clientResources = DefaultClientResources.builder().dnsResolver(DnsResolvers.JVM_DEFAULT).build();

        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(clusterNodes);
        LettuceClientConfiguration lettuceClientConfiguration = LettuceClientConfiguration.builder()
                .clientResources(clientResources)
                .clientOptions(clientOptions).build();

        return new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);

(I've also tried not changing the DNS at all and using the default)
We're not using the static topology because we have a VPC connection into AWS and the member node IPs are accessible to us. Also if I understand DATAREDIS-580 and DATAREDIS-762 correctly it's currently not possible to use the static topology with spring data.
The above works fine except in the case where the IP address mapped to the hostname changes. This can be triggered manually by deleting and recreating the cluster with the same name but also the AWS docs explicitly warn that DNS mappings should not be cached and are prone to change. My experience with AWS services matches that.
The problem seems to be here in SocketAddressResolver
https://github.com/lettuce-io/lettuce-core/blob/cf42638bcb1c4ba06ae68414bff0e484907475b1/src/main/java/io/lettuce/core/resource/SocketAddressResolver.java#L83
The DNS resolution is skipped if it's already been resolved.
It's very possible I'm missing somewhere but I don't see any way to configure the connection in a way that will resolve the DNS again upon reconnection.
Is there something we can do within the bounds of spring data to resolve this? Will StatefulRedisMasterSlaveConnection improve the situation?

guys, I am dealing with this same issue, after having a AWS failover I got an exception because I can write to read instances because the master/read was discover at the beginning, is there any config to allow reconnections in this exception case? do you think we can do something similar to issue 822 trying to reconnect in case of exception based on some configuration? Do you think useful to create a jira for this? or this is something which is not going to be added?

Luis,
Whenever a failover occurs your program will need 60 seconds (time that AWS
needs to change de IP from ELB) to identify the change between the write
and read machine.
For some weird reason Java DNSResolver has problems to solve the IP change
on AWS environment, for this reason Mark developed the
DirContextDnsResolver.
Everything you can do is change the resolver for reconnection and live with
this 60 seconds of "writes" exception because like I sad is the time that
AWS needs to change de IP.
Em sáb, 16 de fev de 2019 às 11:06, Luis De Bello<EMAIL_ADDRESS>escreveu:

guys, I am dealing with this same issue, after having a AWS failover I got
an exception because I can write to read instances because the master/read
was discover at the beginning, is there any config to allow reconnections
in this exception case? do you think we can do something similar to issue
822 trying to reconnect in case of exception based on some configuration?
Do you think useful to create a jira for this? or this is something which
is not going to be added?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lettuce-io/lettuce-core/issues/338#issuecomment-464345472,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKKRtmCibMg21hWUTTedsJA6pEUvJQ39ks5vOAJMgaJpZM4JoUfv
.

@ldebello I haven't had a chance to do a failover test on the newer versions but in 5.0.4 setting dynamicRefreshSources to false and enablePeriodicRefresh to true had the effect of rediscovering the entire cluster on the configured interval.

Thanks for the detailed description. lettuce provides a similar facility for Redis Cluster (listening to events during operations; adaptive topology refresh).
I think it would make sense to expose the refresh trigger API and accept a custom TopologyProvider. The MasterSlave API is built on top of these components. Node details come from inside of the TopologyProvider and should not be set externally.
Users are able to build their own TopologyProvider and can provide RedisNodeDescriptions. The only point which requires a bit more thought is signaling demand for a topology refresh.

@mp911de Do you have any pointers on this, do you think this is still a good way to solve this issue? Wouldn't mind working on this.

The issue requires some design and this is the hard part. Writing down the code is the easy part here.

@mp911de Is this issue resolved by https://github.com/lettuce-io/lettuce-core/issues/672?

No. #672 is a Redis Cluster issue. This one is Master/Slave without Redis Cluster.

Hi @mp911de, I've been trying to read quite a bit on this (in particular #1008 and this issue) as we're having a very similar issue and have been looking for a workaround.
We're using:

ElastiCache non-clustered (1 primary, 1 replica)
Spring Boot 2.1.7
Spring Data Redis 2.1.10
Lettuce 5.1.8

We use the default Spring Boot Data Redis' autoconfiguration (with pooling) by providing host and port (so basically a RedisStandaloneConfiguration)
When a primary/replica failover occurs, where the primary doesn't die but just change role, the application is not able to recover from it. Existing connections are still connected to the former primary node and write commands fail with RedisCommandExecutionException - READONLY You can't write against a read only replica. until we restart the application.
We've been looking at ways to catch that particular exception and force the LettuceConnectionFactory to re-establish the connections but there doesn't seem to be a good way to do that.
This is the best we could find at the moment

enable testOnBorrow in GenericObjectPoolConfig
Lettuce implements BasePooledObjectFactory's  validateObject() and destroyObject() but they're never called since the Spring-created GenericObjectPoolConfig uses default values of false for testOnX fields. We were able to tweak it through a LettuceClientConfigurationBuilderCustomizer bean and set testOnBorrow to true
catch RedisCommandExecutionException and then

call resetConnection() on the LettuceConnectionFactory bean
call close() on the internal StatefulConnection

It looks something like this
try {
    // write to redis
} catch (RedisSystemException e) {
    if (e.getCause() instanceof RedisCommandExecutionException) {
        if (e.getCause().getMessage().startsWith("READONLY")) {
          final RedisConnection connection = connectionFactory.getConnection();
          connectionFactory.resetConnection();
          ((RedisAsyncCommands) connection.getNativeConnection()).getStatefulConnection().close();
        }
    }
}

After a few errors, eventually the connections are recreated and the application recovers.
 I'm also worried about unintended side-effects.
We're looking for guidance on how to best handle this scenario gracefully (manually restarting our dozen of instances is not an option)

Hi guys,
Is there any solution for this case? @mp911de , can You share your code solution to connect with AWS Redis using master and slave?
I need to configure using the same stack and didn't find good configurations using spring+lettuce.

@charlesardsilva AWS has added master endpoint and reader endpoint so you could use those dns to solve the issue.

@ldebello
The reader endpoint round-robin routing works by changing the host that the DNS entry points to.
lettuce does not load balance slave nodes because it reuses connections.
Can you solve this problem?

Currently we accepted the provide balance,it is not perfect but at least we can use master/replicas.

@charlesardsilva

Is there any solution for this case

Nothing that we could do from a Lettuce-only perspective. Lettuce requires additional information about the new (changed) topology and that is something that needs to be provided externally.
We faced a few times the requirement to refresh the topology on demand. We could provide a reloadNodes(Iterable<RedisURI> hint) method that accepts a topology hint. Master/Replica uses various strategies for topologies:

Static Master/Replica using auto-discovery
Static Master/Replica using provided endpoints
Sentinel-managed Master/Replica

We need to figure out what a reloadNodes should do for each of these cases.

@mp911de I don't know if I'm right but Lettuce uses the COMMAND INFO by default to identify who is the master and slave, right? So, I believe that is the problem. When you use the AWS ElastiCache, the IP from INFO command is an AWS internal IP. The real IP or VPC IP is unavailable on info command. For AWS the correct way to reload the nodes is to use the DNS informed on the Topology configuration.

That is exactly why we have the

Static Master/Replica using provided endpoints

mode to take user-specified endpoint addresses. Lettuce figures out the roles from the given array of endpoints.
INFO replication is used for

Static Master/Replica using auto-discovery

which is intended mostly for on-premise setups.
We do not want to integrate with any sort of Cloud provider SDK as Lettuce is a Redis client, not a swiss army cloud knife.

For AWS the correct way to reload the nodes is to use the DNS informed on the Topology configuration.

Single-node failover already works this way as the hostname is resolved upon (re)connect.

Let me see if my understanding is correct...
AWS ElastiCache behavior
AWS provides DNS CNAMEs for

A) the current master
B) the load balancer over all replicas
C) one DNS entry for each individual Redis node (one for master and one for each replica)

AWS ElastiCache will automatically reconfigure the topology based on various events (individual node failure, manual master promotion, manually adding or removing a node in the AWS web console, etc). When any of these events occur, ElastiCache will make the changes to the topology and then update the DNS CNAME records when complete.
Example use case
Let's assume we have a long running process using Lettuce to interact with the ElastiCache topology. This process has both write requirements, consistent read requirements (read from master), and high volume read requirements where stale data is acceptable (e.g. ok to read from a replica). For a short-lived  program we could probable get away with simply creating a "Static Master/Replica using provided endpoints" configuration based on DNS names for the master and replica load balancer. For a long-live program we need (1) the ability to detect a topology change event and (2) the ability to reconfigure our Lettuce client triggered by (1). Let's also assume we are not running Redis in a cluster topology, just a simple master/replica configuration. Let's also assume the client is running in AWS as well, so we can take advantage of DNS resolving to the non-public IP address for the Redis nodes.
Approach
For ElastiCache it seems like there are two possible approaches we can take.

Static Master/Replica using provided endpoints
Static Master/Replica using auto-discovery

Initialization
For initialization of (1) I think we need to pass in a static list of the DNS CNAMEs for each individual Redis node (this is (C) from up above).
For initialization of (2) we can simply pass in the DNS CNAME for the master and let Lettuce discover the topology (this is (A) from up above).
Topology change discovery
It seems like for both (1) and (2) you need to either have some logic to periodically inspect the topology to see if it has changed since initialization or catch exceptions (e.g. reading from a node that is offline or writing to a node that was change from master to replica). My understanding is that it is straightforward to catch an exception, but the ability to inspect the topology to compare to the topology at initialization time does not exist in Lettuce. Is that right?
Lettuce client update
Once a change is detected the Lettuce client needs to be updated. In a high volume, concurrent processing program where multiple threads may share Lettuce resources it can be tricky to update the client to the new topology while the underlying topology change itself (AWS performing the change) is not instantaneous. For a static redis topology it seems like you need to reinitialize a new client connection. There is no way to keep the existing Lettuce objects and tell it to re-initialize itself for a non-cluster topology. Is that right?
My understanding is that there is more advanced support in Lettuce for detecting topology changes and updating the Lettuce client accordingly for both Sentinel and Redis Cluster. For the simple Master/Replica configuration some of the Lettuce API's do not apply.
It seems reasonable for Lettuce to not build cloud-specific logic into Lettuce itself. At the same time, it seems the ElastiCache use case in non-cluster mode is a common one. Users want the ElastiCache update mechanism to play nicely with Lettuce in a concurrent processing system. At this point, it is not clear how to do that. There seems to be an impedance mismatch. Hence this open github issue.
An ideal outcome would be documentation and example code on the Lettuce wiki that demonstrates how to use Lettuce with ElastiCache for high availability failover in a non-cluster, master/replica configuration. This may or may not prompt enhancements in Lettuce itself.

Let's go back to the first problem and talk about the response to failover
use case
node 1-primary
node 2-replica
node 3-replica
after failover
node 1-replica
node 2-primary
node 3-replica
However, in the StaticMasterSlave configuration the known nodes of lettuce still know node 1 as primary.

This is because the StaticMasterSlave configuration does not support refreshing RedisNodeDescription after the initial connection.
` Static Master / Slave runs a refresh only at the very beginning since there's no trigger that indicates a topology change (in contrast to Redis Sentinel). `
This is understood.
But what if the user issues an event internally and leaves room for updates?
`` `
private StaticMasterSlaveRefreshConnections collectConnections (
List <ConnectionFuture <StatefulRedisPubSubConnection <String, String >>> connectionFutures) {
StaticMasterSlaveTopologyRefreshConnections collector = new StaticMasterSlaveTopologyRefreshConnections (connectionFutures.size ());

for (ConnectionFuture <StatefulRedisPubSubConnection <String, String >> connectionFuture: connectionFutures) {

    connectionFuture.thenCompose (connection-> {

        connection.addListener (adapter);
        return connection.async (). psubscribe ("*"). thenApply (v-> connection) .whenComplete ((c, t)-> {

            if (t! = null) {
                connection.closeAsync ();
            }
        });
    }). whenComplete ((connection, throwable)-> {

        if (throwable! = null) {
            collector.accept (throwable);
        } else {
            collector.accept (connection);
        }
    });
}

return collector;

}
`` `
I also think that catching dynamically growing quantities is not something the StaticMasterSlave strategy will do.
However, the above strategy can achieve redistribution of static nodes, and the reconnect logic for a single node can work well.

It will be great if we can have an approach like this https://github.com/luin/ioredis#reconnect-on-error, even with Spring wrappers

Hi Everyone, I am trying to write some code in my application to re-initialise the redis client after failover. Instead of catching the exception on read/write, Is there a way to intercept all read/write operations and re-initialise a new client connection? Can someone suggest if there is a way to do it ?

Right now, there's no method to apply a new topology to a StatefulRedisMasterReplicaConnection. The only thing possible is to re-obtain StatefulRedisMasterReplicaConnection.

Hi Mark,
Thanks for quick response. Indeed I am trying to re-obtain the redis
connection. But I am not in favor of obtaining the connection by
surrounding the read/write operation with a try-catch block, as it needs
code changes at every place when the read/write operation is invoked.
Instead I want to intercept all read/write methods to achieve this by using
spring aspects. Read/write operations are defined inside
DefaultValueOperations class and I can't create a bean of it to implement
the aspect as its access modifier is DEFAULT. Is there a way to intercept
the read/write operations?
Thanks in advance.
Vikram
On Fri, Apr 9, 2021 at 3:50 PM Mark Paluch @.***> wrote:

Right now, there's no method to apply a new topology to a
StatefulRedisMasterReplicaConnection. The only thing possible is to
re-obtain StatefulRedisMasterReplicaConnection.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lettuce-io/lettuce-core/issues/338#issuecomment-816582496,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARZ6CT3MKWTETTWT66XCJADTH3IGJANCNFSM4CNBI7XQ
.

Multi-node connections are always subject to mixed availability. Depending on what hosts are up/down, a connection may work for parts of the commands. There's no way to tell from the outside.

Can we do something at a unified place instead of littering all over the code base with try-catch, so that the re-initialisation will be triggered on read/write failure? Any small clue or idea will be a great help. Thanks.

Ok. Can we do something at a unified place instead of littering all over the code base with try-catch, so that the re-initialisation will be triggered on read/write failure? Any small clue or idea will be a great help. Thanks.

Using Spring, enable LettuceConnectionFactory.validateConnection so Spring hands you out a connection where a PING succeeds.

Can you explain more how LettuceConnectionFactory.validateConnection can eliminate the need for a try/catch block on a read/write failure with a StaticMasterReplicaRedisClient?

In discussion with AWS folks, the recommendation was to disable dns caching at the jvm level with the assumption that reconnection would eventually connect to whatever redis instance is behind the DNS name after failover.  So, for example, let's assume we start our MasterReplica with the following URIs:
Primary:

Endpoint: -001.**.0001.use1.cache.amazonaws.com, IP <IP_ADDRESS>

Replica

Endpoint: -002.**.0001.use1.cache.amazonaws.com, IP <IP_ADDRESS>

After failover, we have
Primary:

Endpoint: -001.**.0001.use1.cache.amazonaws.com, IP <IP_ADDRESS> (promoted)

Replica

Endpoint: -002.**.0001.use1.cache.amazonaws.com, IP <IP_ADDRESS> (replacement)

Should the reconnect mechanism work as they expect?  Eventually finding <IP_ADDRESS> as the primary and <IP_ADDRESS> as the replica, or does there need to be a full reconnection?

Ran into this -- somehow one of our app's replicas didn't catch the topology update coming from sentinel or didn't respond to it correctly. Ideally there would be a nice easy way to, if we encounter readonly exceptions, trigger a topology update and recover.

Is this issue being solved? I've just enountered such problem with ElastiCache, one replica was promoted to master (3 shards, master+2 replicas), and App couldn't write (PUT,DEL operations were failing) for 15 minutes..
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 2 second(s)
Problem was solved by redeploying the App (thus redis client was reinitialised with a new topology).
How to mitigate that problem ?

Hi all, any news on this?
I have a similar problem. I configured my spring-boot application to use the lettuce client. I have configured 6 Redis nodes on Elasticache (3 masters + 3 slaves). When a master goes down (to test a failover), the application stop working because it continues to try to connect to the old master, and the application gets a connection timeout.
Following the configuration:
spring   data:     redis:       database: 1       timeout: 10000       cluster:         nodes: master01:6379,slave01:6379,master02:6379,slave02:6379,master03:6379,slave03:6379         max-redirects: 6       ssl: true       lettuce:         pool:           max-idle: 8           min-idle: 0           max-active: 8           enabled: true         cluster:           refresh:             dynamic-refresh-sources: true             adaptive: true             period: PT1S       client-type: lettuce


Split MQTT message documentation out to its own wiki page

Right now all the information is in the Configuration topic which is way too much for one document. Keep configuration to just enabling MQTT and link to a new doc page for MQTT messages.

Closing old issues.


autoparser: tools to semi-automate parser construction

Observation table needs more work, and field matching and scoring will undergo iterative improvements. I have tried it for CCPUK/RAPID and it seems fairly accurate for the CSV -> TOML step, less so in generating the intermediate CSV mapping, which requires manual curation at this point.
Documentation

Nice. I will review this in more detail.
On Mon, Mar 13, 2023 at 3:07 PM Abhishek Dasgupta @.***>
wrote:

@abhidg https://github.com/abhidg requested your review on: #98
https://github.com/globaldothealth/isaric/pull/98 autoparser: tools to
semi-automate parser construction.
—
Reply to this email directly, view it on GitHub
https://github.com/globaldothealth/isaric/pull/98#event-8732981041, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABW7SJGZDRSEX5A4RIC2VNLW34Z3LANCNFSM6AAAAAAVZGR7ZA
.
You are receiving this because your review was requested.Message ID:
@.***>

Maybe a step further is also include column names from the source data for text searching. Although the column names information won't give you as much information as a data dictionary, it can be useful for mapping ..
I found this with the South Africa dataset for acute infections - the data dictionary was not very useful.
This migh be the case for other studies.

Good point. I will add a --infer-dictionary option to autoparser-csv to get the data dictionary from a CSV file

Is this ready for trial?

I'll add a few tests, should be ready by next week


Multi lang doesnt work

Hello,
I try to change the language but it's not working.
const invoiceIt  	= require('@rimiti/invoice-it');

invoiceIt.configure({
  language: {
    locales: ['fr', 'en'],
    directory: `${__dirname}/locales`,
    defaultLocale: 'fr'
  }
});

and within my route :
		const invoice = invoiceIt.create(recipient, emitter);

		invoice.getInvoice().toPDF().toFile(pdfPathfile).then(() => {
			console.log('PDF file created.');
			return res.json({});
		});

I did a log within your generator.js
and it's always showing 'en' as default language
version package npm:     "@rimiti/invoice-it": "^3.1.0",

Nevemind the solution was here : https://dimsolution.com/blog/2018-05-29/how-to-easily-generate-order-invoice-pdf-file/
I had to do :
invoiceIt.configure({
	global: {
		logo: 'http://placehold.it/230x70&text=xxxx',
		lang: 'fr',
		footer: {
			fr: "xxxxx"
		}
	},
});

There is a mistake within the readme

Hello @edeuxk,
As default, fr + en locales are set.
If you want to change from en to fr, you can use the lang()setter like that:
const invoiceIt = require('@rimiti/invoice-it');
invoiceIt.configure({
  language: {
    locales: ['fr', 'en'],
    directory: `${__dirname}/locales`,
    defaultLocale: 'fr'
  }
});

invoice.getInvoice().toPDF().toFile(pdfPathfile).then(() => {
			console.log('FR PDF file created.');
			return res.json({});
		});

invoice.lang('en');
invoice.getInvoice().toPDF().toFile(pdfPathfile).then(() => {
			console.log('EN PDF file created.');
			return res.json({});
		});

Regards,


New Invisible Error in Querie after migratating in V24 (Saas)

Hello, since the update to v24, we have encountered an error (which does not trigger any Business Central error) on one of our queries. After several tests, it seems that this may be related to some of our fields that are in the filter format.
For example this querrie:
`///
/// Query API-Today Total Order CP0 (ID 50001).
///
query 50001 "API-Today Total Order CP0"
{
APIGroup = 'cap';
APIPublisher = 'cap';
APIVersion = 'v1.0';
EntityName = 'todayTotalOrder';
EntitySetName = 'todayTotalOrders';
QueryType = API;
elements
{
    dataitem(salesHeader; "Sales Header")
    {

        filter(documentType; "Document Type")
        {

        }

        column(sellToCustomerNo; "Sell-to Customer No.")
        {
        }

        filter(creationDate; "Creation Date CP0")
        {

        }

        dataitem(salesLine; "Sales Line")
        {
            DataItemLink = "Document No." = salesHeader."No.";

            filter(type; Type)
            {
            }

            column(sumLineAmount; "Line Amount")
            {
                Method = Sum;
            }

        }

    }

}
var
    endDate: DateTime;
    startDate: DateTime;

trigger OnBeforeOpen()
begin

    //La startDate est à la date d'hier à 18h01:00
    startDate := CreateDateTime(CalcDate('<-1D>', Today()), 180100T);

    //La endDate est à la date d'aujourd'hui à 18h00:59
    endDate := CreateDateTime(Today(), 180059T);

    SetRange(documentType, Enum::"Sales Document Type"::Order);
    SetRange(creationDate, startDate, endDate);
    SetRange(type, Enum::"Sales Line Type"::Item);
end;

}
`
Return code 200 but the json is malformated by Business central : $homeUrl//todayTotalOrders?%24filter=sellToCustomerNo eq %273030%27 (worked fine in v23)

but if i change two filter in column (Document Type and Type) :
`
query 50001 "API-Today Total Order CP0"
{
APIGroup = 'cap';
APIPublisher = 'cap';
APIVersion = 'v1.0';
EntityName = 'todayTotalOrder';
EntitySetName = 'todayTotalOrders';
QueryType = API;
elements
{
    dataitem(salesHeader; "Sales Header")
    {

        column(documentType; "Document Type")
        {

        }

        column(sellToCustomerNo; "Sell-to Customer No.")
        {
        }

        filter(creationDate; "Creation Date CP0")
        {

        }

        dataitem(salesLine; "Sales Line")
        {
            DataItemLink = "Document No." = salesHeader."No.";

            column(type; Type)
            {
            }

            column(sumLineAmount; "Line Amount")
            {
                Method = Sum;
            }

        }

    }

}
var
    endDate: DateTime;
    startDate: DateTime;

trigger OnBeforeOpen()
begin

    //La startDate est à la date d'hier à 18h01:00
    startDate := CreateDateTime(CalcDate('<-1D>', Today()), 180100T);

    //La endDate est à la date d'aujourd'hui à 18h00:59
    endDate := CreateDateTime(Today(), 180059T);

    SetRange(documentType, Enum::"Sales Document Type"::Order);
    SetRange(creationDate, startDate, endDate);
    SetRange(type, Enum::"Sales Line Type"::Item);
end;

}
`
The result was ok :

I think a new system bug has appeared with v24. If anyone has an idea or can tell me how to report the information directly to Microsoft?
Thank you.
Kevin
### Tasks

Hello, please report this as a support request as it looks like runtime issue affecting live customers.
You can do the following:
Open a support request to CSS through the PartnerSource portal
Open a bug in Collaborate
Contact your Service Account Manager (SAM) in your local subsidiary to understand what is included in your contract as of support incident and PAH (Partner Advisory Hours). Your SAM might also direct you "step by step" how to open a support request or how to get credentials, if this is the first time for you or your company.


Explore SSO data

https://socialscience.one/blog/unprecedented-facebook-urls-dataset-now-available-research-through-social-science-one

Hi,
I read through some of their stuff. The proposal sounds very useful. They have achieved a way to bypass certain laws and rules by the GDPR, which wouldn't allow the collection of data, even if it is anonymized. Therefore they used a way only fetch the URL of a users which then (as far as I understood) is anonymized. So they actually never use any data that would help to reveal a person.
Despite the FTC and the GDPR rules being in effect, they went on collecting data with the help of Open Source applications, which led to the release of two papers which can be found here and here.
How much Data was gathered:

The dataset contains approximately an exabyte (a quintillion bytes, or a billion gigabytes) of raw data from the platform, a total of more than 10 trillion numbers that summarize information about 38 million URLs shared more than 100 times publicly on Facebook (between 1/1/2017 and 7/31/2019).  It also includes characteristics of the URLs (such as whether they were fact-checked or flagged by users as hate speech) and the aggregated data concerning the types of people who viewed, shared, liked, reacted to, shared without viewing, and otherwise interacted with these links.

The best part is, we as researchers can get access to their data. Here is the papers, which shows a overview of the data they have by now and what is needed to get access.
I decided to make a New Sheet on Google so we can gather all DATA COLLECTIONS (Open and Closed) for later elaboration.


New tag

Could you please add a new tag regarding PR #381 ?

Hi, I'm sorry to insist on this, but would this be possible?

Sorry for the delay, will see If I can fix this for you today.

Your change is merged into version 7.1.4.
Greetings Sander


2516 deduplicate attachements on compaction

This is replacing PR https://github.com/apache/couchdb-couch/pull/22

There are problems with calculating active size when document get updated or deleted.

For posterity, we were trying to make this work without requiring an on-disk format change but sadly that appears to not be possible due to the lack of information available when updating documents. Given the lack of information we can't tell if an attachment that's being updated or deleted needs to be counted in active_size.
The proper fix for this will be to add a new index to each database that contains the attachment data. Given the merge work we're going to delay this until later. Also if/when we get back to this we'll also want to investigate having attachment data stored in a separate file which would remove a lot of the gymnastics we have around couch_file:make_blocks and friends.


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-k8s-29-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 4221

Questions ?
Please refer to the Backport tool documentation


Feat/config

@marcin-nedza
I've marked this pr for myself by mistake. Could you please take a look at this and response to each comment ? In addition to that, fix eslint errors ?
This pr is also part of #4 .


Refactoring attempt for indirect-calls-executor

This PR tries to do a small refactoring for indirect-calls-executor. The main changes and reasons are listed below:

move the content of indirect_calls_executor.rs to lib.rs, as the crate has the same name already: indirect-calls-executor/src/indirect-calls-executor.rs seems unnecessary and overlaps with indirect-calls-executor/src/lib.rs
Turn the if .. paradigm to a vector of Executors to more clearly illustrate how the parentchain extrinsics are handled. Otherwise we'll have longer if else branches as more handlers are added, and readability can be significantly reduced.
Put each Executor in an individual file
We need an extra helper trait DecorateExecutor because Executor has an associated type
We try to unify the error handling and pre/post processing: IMHO the successfully handled extrinsics should always be pushed into executed_calls to be further passed into create_processed_parentchain_block_call. In the original code, only shield_funds call is pushed, call_worker not -- we think this might be wrong.

Thank you @clangenb for the review and suggestions, it makes a lot of sense. I also left a comment in #1156, as mentioned IMHO it doesn't change much in terms of the logic but does have better readability.
The question is do you still want to integrate this PR? If so I can fix the points you mentioned in the review and re-sync it. Otherwise if you don't need it for now -- simply go ahead and close it, it's completely fine :)

Thanks for asking. I don't have a strong opinion about this PR, I think it is a useful intermediate step, however, the follow-up #1156 will not be the priority. So I would say if it makes your downstream live easier, I am happy to merge the PR after the fixes, otherwise we can just close it.

if it makes your downstream live easier

It does, one less change is always better :) I'll fix those and request a re-review.

Updated some files according to the review :)


If no monitoring defined on DB all appear to be selected

When you access Service Monitoring from an instance that has no monitoring assigned it gets a list of everything.

This should not happen.
The tables should have the proper info and this is just a dev quirk.
Will check to manage in any case the empty info

From the UI perspective we could not know if the selected instance has or not info to show.

We should.
The status of the customization has to be used
If not deployed monitoring and stop or start Cannot be pressed
Inviato da Libero Mail per Android Giovedì, 10 Marzo 2016, 08:51AM +01:00 da Xavier Cases Camats <<EMAIL_ADDRESS>:

From the UI perspective we could not know if the selected instance has or not info to show.
—
Reply to this email directly or  view it on GitHub .


bug in skip-thought.ipynb

NLP-Models-Tensorflow/unsupervised-summarization/skip-thought.ipynb
bugs:
for i in range(5):
pbar = tqdm(range(0, len(middle), batch_size), desc='train minibatch loop')
for p in pbar:
should be
for k in range(5):
pbar = tqdm(range(0, len(middle), batch_size), desc='train minibatch loop')
for i in pbar:

thanks for noticing that, will going to fix that on next commit!

Fix! feel free to reopen.


Blank states: error

Story
Whenever information does not load, I get a message so I know what went wrong.
Question: Where do we expect this to happen? What types of errors do we need?
Expected result
Examples
Search / filter results will not load

Something inside a publication will not load

Copy
Title: No records found
Text: Adjust your search to find what you are looking for.
Component
https://biblio-ugent-prototype-emily-v2.vercel.app/styleguide/c-blank-slate.html#component-06-c-blank-slate-danger
Example code
Inside the card body:

Something appears to be wrong.

    Refresh the page and try again please.

    If this continues to happen, contact us.

"Internal server error" -> global problem
"Not found" -> publication not found
"Not found" -> generic path not found
Loading state..
Something went wrong

@nicolasfranck Can you give me the types of error pages we need to display (and where to show it)? I'll create a design for it.

No answers gotten, stale issue


change lockhunter.json:  $bucketsdir to $bucketdir

As the title show.

Post a screenshot of successful installation from the manifest in this PR

Didn't install successfully for me, is $bucketdir supposed to exist? it's empty when running

Yes, this is why I asked for a screenshot too. Thanks @phanirithvij!


Do not append variables missing append_dim

nc2zarr fails when appending variables that do not have the dimension indicated by append_dim, e.g. "time".
Variables that lack the append_dim dimension should be written once and from then on be excluded from appending.
Here is an example from the SST L4 GHRSST GMP source products:
2021-06-03 11:39:54,121: INFO: nc2zarr: 365 input(s) found:
  0: /neodc/esacci/sst/data/gmpe/CDR_V2/L4/v2.0/1982/01/01/19820101120000-ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-v02.0-fv01.0.nc
  1: /neodc/esacci/sst/data/gmpe/CDR_V2/L4/v2.0/1982/01/02/19820102120000-ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0--GLOB_CDR2.0-v02.0-fv01.0.nc
...
  364: /neodc/esacci/sst/data/gmpe/CDR_V2/L4/v2.0/1982/12/31/19821231120000-ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-v02.0-fv01.0.nc
2021-06-03 11:39:54,122: INFO: nc2zarr: Processing input 1 of 365: /neodc/esacci/sst/data/gmpe/CDR_V2/L4/v2.0/1982/01/01/19820101120000-ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-v02.0-fv01.0.nc
2021-06-03 11:39:54,300: INFO: nc2zarr: Opening done: took 0.18 seconds
2021-06-03 11:39:56,486: INFO: nc2zarr: Writing dataset done: took 2.16 seconds
2021-06-03 11:39:56,497: INFO: nc2zarr: Processing input 2 of 365: /neodc/esacci/sst/data/gmpe/CDR_V2/L4/v2.0/1982/01/02/19820102120000-ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-v02.0-fv01.0.nc
2021-06-03 11:39:56,607: INFO: nc2zarr: Opening done: took 0.11 seconds
2021-06-03 11:39:56,650: ERROR: nc2zarr: Appending dataset failed: took 0.02 seconds
2021-06-03 11:39:56,650: ERROR: nc2zarr: Converting failed: took 3.23 seconds
Traceback (most recent call last):
  File "/apps/slurm/spool/slurmd/job56309097/slurm_script", line 33, in <module>
    sys.exit(load_entry_point('nc2zarr', 'console_scripts', 'nc2zarr')())
...
  File "/home/users/forman/Projects/nc2zarr/nc2zarr/writer.py", line 82, in write_dataset
    retry.api.retry_call(self._write_dataset,
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/retry/api.py", line 101, in retry_call
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/home/users/forman/Projects/nc2zarr/nc2zarr/writer.py", line 98, in _write_dataset
    self._append_dataset(ds)
  File "/home/users/forman/Projects/nc2zarr/nc2zarr/writer.py", line 146, in _append_dataset
    ds.to_zarr(self._output_store,
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/xarray/core/dataset.py", line 1790, in to_zarr
    return to_zarr(
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/xarray/backends/api.py", line 1452, in to_zarr
    _validate_datatypes_for_zarr_append(dataset)
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/xarray/backends/api.py", line 1300, in _validate_datatypes_for_zarr_append
    check_dtype(k)
  File "/home/users/forman/miniconda3/envs/nc2zarr/lib/python3.9/site-packages/xarray/backends/api.py", line 1291, in check_dtype
    raise ValueError(
ValueError: Invalid dtype for data variable: <xarray.DataArray 'field_name' (fields: 16, field_name_length: 50)>
dask.array<array, shape=(16, 50), dtype=|S1, chunksize=(16, 50), chunktype=numpy.ndarray>
Coordinates:
  * fields             (fields) int32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  * field_name_length  (field_name_length) int32 1 2 3 4 5 6 ... 46 47 48 49 50 dtype must be a subtype of number, datetime, bool, a fixed sized string, a fixed size unicode string or an object

The actual error here is (I think) caused by an xarray bug which I reported here: https://github.com/pydata/xarray/issues/5224 . xarray should allow appending of this variable. But of course it's a nc2zarr bug as well, since even if xarray could append those |S1-typed variables correctly, it doesn't make sense to do so.

The actual error here is (I think) caused by an xarray bug which I reported here: pydata/xarray#5224 .

Right!


#2286 Add function of getting thread dump from agent

change response data
add test

@emeroad review please :)