refactor: [M3-6852] - FormattedAPIError, a type-safe extension of APIError

Description 📝
Today, across the Cloud Manager, errors are guaranteed to conform to the universal APIError interface. This interface comes from the @linode/api-v4 package and includes a string-based reason field.
In some cases, we want to intercept errors from the API and inject formatting for UX purposes -- for example, a <Link> is injected into error messages that mention opening a support ticket. Currently, this is done by casting the formatted reason (which is natively of type JSX.Element) into the reason string field.
Historically, this has lead to crashes when string methods are applied to formatted error messages. And since the interceptors are global, it is impossibly to confidently determine which components may receive intercepted error messages.
This PR introduces a new universal error type, FormattedAPIError, which extends APIError and is native to the linode-manager package. This new interface introduces an additional field, formattedReason, which supports formatted JSX error messages. Components or utilities that need a string-based error message can fallback to the reason field, which is now guaranteed to actually be a string.
Changes  🔄

New universal error type: FormattedAPIError
Replace APIError with FormattedAPIError across the codebase
Update documentation

Preview 📷
No visual or other user-facing changes.
How to test 🧪
Since this PR replaces APIError with FormattedAPIError across the app, it is likely cumbersome to verify all individual changes. However, the following steps may assist while reviewing this PR:

Inspect the new interface and how it is used in the app. Do you agree with this approach, the shape of the interface, and the ergonomics of using it?
Verify there are no regressions or bugs introduced in the error handling of criticals forms in the app (e.g., Linode/Volume/OBJ/etc. create forms)
Verify unit tests and E2E tests are passing

As an Author I have considered 🤔
Check all that apply

[x] 👀 Doing a self review
[x] ❔ Our contribution guidelines
[ ] 🤏 Splitting feature into small PRs
[x] ➕ Adding a changeset
[x] 🧪 Providing/Improving test coverage
[ ] 🔐 Removing all sensitive information from the code and PR description
[ ] 🚩 Using a feature flag to protect the release
[ ] 👣 Providing comprehensive reproduction steps
[x] 📑 Providing or updating our documentation
[ ] 🕛 Scheduling a pair reviewing session
[ ] 📱 Providing mobile support
[ ] ♿  Providing accessibility support

Do we want to actually prevent usage of .reason? Both are required now which feels odd to me. Can you clarify the intent?

No, reason is kept as a string fallback. There are a small number of cases that require a string (e.g., see PayPalChip).

Similarly to @mjac0bs, I wonder about the type safety of the approach. I know we had briefly discussed alternatives in the past, but I am starting to think that the global intercept is flawed in general and each error override should be handled individually to guaranty type safety (not 100% sure but this is what my instinct tells me). This PR could serve as a base to identify all cases needed to be handled if so, unless we find a way to strengthen this approach. From a developer perspective, it sure seems to leave room for errors (including new error overrides we're adding). Happy to pair to look at this at any point.

@hkhalil-akamai I'm curious what your thoughts are between this approach and replacing errors at the component level now that you've done both this PR and https://github.com/linode/manager/pull/10390.
I think my favorite solution so far would be to just preform replacements at the component level where needed to keep things simple, but I understand that it wouldn't be as hands-off as this approach.
Example:

Similar to what Alban stated, I do think having as little reliance on intercepting as possible is a good thing for keeping things straightforward and un-abstracted. While it might be a pain, maybe we'll like having things be explicit instead of intercepted.. but I'm not sure
Curious to hear how you feel  @hkhalil-akamai because you have all of the context and hands on experience with both approaches.

We can use formattedReason and fall back on reason, but whatever is returned by getErrorMap() (e.g. generalError) can be inaccurately typed as string | undefined when it could be (and, in the case of the account limit error, is) JSX -- unless we also change those types (L88, L109)... but the returned value is expected to be a string in a lot of places.

Great catch @mjac0bs (and I expect there to be more if we go this route) but I want to point out that this means the changes are doing exactly what was intended. Previously, getErrorMap was violating its contract by returning JSX in place of a string (leading to possible crashes and other negative consequences). Now, it is truly returning a string, leading to the less serious problem of not including a hyperlink, which is straightforward to solve by tweaking and refactoring the signature of getErorrMap.
@abailly-akamai I have considered the issue of type safety and I believe this is a sound approach. Our existing system deliberately breaks type safety (specifically this cast in interceptAPIError.tsx:18) shying away from a project-wide change like this one. While in some cases global intercepts may be undesirable, it is specifically through them that we can (and now do) guarantee the shape and type of errors from the API and avoid making this compromise.
@bnussman-akamai I am open to both solutions but I prefer the one in this PR for two reasons:

It is extremely difficult to comprehensively predict which errors a particular API endpoint may return (due to the way errors bubble up in the API)
This adds an additional burden to maintainers since they have to remember to format error messages when displaying them (as opposed to this approach, where they just need to remember to use .formattedReason

Curious to hear everyone's thoughts and if there is enough interest, I can bring this to cafe for another team-wide discussion.

I think this does warrant another convo just so we're on the same page and understand all options. Can we schedule 30m outside of the cafe with a select group to get to the nitty gritty (cafe tends to stay on the surface with a wider audience)?

where they just need to remember to use .formattedReason

If we can't enforce usage then perhaps this is not the strongest solution. If our primary concern is string methods crashing there's run time solutions to help with that. Anyway, would love to get the fullest picture before committing to this PR (which still has value!)

Closed for now -- will revisit in the future.


Additional installation steps and bug for CORD-19

Hi David,
In addition to following your paperetl installation instructions I had to take these steps to get rid of the following error and warning:

~$ python3
>>> import nltk
>>> nltk.download(‘punkt’)
>>> exit()

this created a directory ~/nltk_data/tokenizers/punkt
and fixed the above error
Also, the UserWarning below was eliminated as follows:
$ pip3 uninstall scikit-learn==0.23.2
$ pip3 install scikit-learn==0.23.1

Then, unfortunately after a full run the resulting articles.sqlite database came out with the Study Design fields, and Tags and Labels being NULL (see screenshots below).

Any ideas on how solve this NULL issue would be appreciated

Hi David,
Unfortunately, NLTK has different models that are installed with different installations. Thank you for sharing that you needed to install an additional model.
Regarding study design, not all documents will have a detected study design (will be 0). Only certain documents will be tagged as COVID-19.
Please take a look at: https://www.kaggle.com/davidmezzetti/cord-19-etl and https://www.kaggle.com/davidmezzetti/cord-19-analysis-with-sentence-embeddings
These notebooks on Kaggle have all the background on how this process works.

Thank you David. After examining the results more carefully, unlike the above screenshots,
I found plenty on non-NULL values (screenshots below); so the output is correct and there is no such a bug.
Yes, I have read your writings on GitHub, Kaggle, and tds as well as Ankur Mohan’s blogs about your work. Your models are very nice, I have learned a lot from them.


[5.3] [Proposal] Array values recursive.

Some time we need to fetch all values from array and we can do it by using array_values in array_walk_recursive function but sometime we need to maintain array dimensions which is not possible by using same trick. So i created a function which can do both the things by just passing 2nd parameter true or false. Also i added test case for more clarity...

No plans on this in the core.

@taylorotwell please tell me according to you is this not useful? And i also made a mistake this should go into Arr class and method name should be values.


add CORS to rest server and config

i have to add it because my frontend gave me some CORS errors. Feel free for some tests and let me know if i have something to change.

thanks for your input. i hope my changes works for you.

thanks for your input. i hope my changes works for you.

thanks for your response. maybe a little bit change is needed :smiley:

no problem i learn a lot in go now ^^


predict() with trained predictor and open questions

Again, thank you for this great time-series prediction tool. I have some open questions left, that have not been answered in my last issue:
About predicting with a trained model (not evaluating):
I have an predictor that has been trained with
estimator = DeepAREstimator(freq="d",
                            prediction_length=prediction_length
                            , context_length=context_length,
                            trainer=Trainer(epochs=1), use_feat_dynamic_real=True)
predictor = estimator.train(training_data=training_data)

now i want to predict the future. To construct the dataset i want to predict on i use all my training data and the future values of my exogenous variables.
training_data = ListDataset(
    [{"start": df.index[0], "target": df.target[:split_date],
      'feat_dynamic_real': df.feature_1[:split_date:],
      'feat_dynamic_real': df.feature_2[:split_date:],
      'feat_dynamic_real': df.feature_3[:split_date:],
      }],
    freq="d"
)
prediciton = predictor.predict(training_with_exogen_future).__next__().mean_ts

Which throws me the error:

ValueError: all the input array dimensions except for the concatenation axis must match exactly

It seems the predict method is complaining about the target variable array not having as many values as the exogenous variable in my ListDataset. Am i using the wrong method, or do i have to specify another parameter?
About DeepAREstimator parameters:
Does frequency only refer to the gap between two datapoints e.g. "D" or "H", or does it refer to the timeframe we see a pattern repeat, e.g. "7D". When i have a series of data points where 10 days lay in between, do i specify "10D" or "D"?
How do i determine the value for the context_length parameter? Do i do this with grid search?
About the backtest (make_evaluation_prediction):
The documentation says:

Removes the final window of length prediction_length of the dataset.test that we want to predict
The estimator uses the remaining data to predict (in the form of sample paths) the “future” window that was just removed

Does that mean it predicts my prediction_length at num_samples different random points in my test set to evaluate the model?
About Evaluator()
It has a parameter "seasonality". I could not find any further information in the documentation. How do i determine the best value for this parameter, and what does it actually mean in regards to my dataset?
About feature importance:
Is there anything implemented, that shows me how important/how much impact which of my exogenous variables is?
About my specific hourly data problem:
Since i actually have hourly data from 8am to 8pm, i think it is a total waste to look model each hour individually as daily data because i lose the influence of the previous hours. So what i did it i set my DeepAREstimator frequency to "H" and prediction length to 360 (15days x 24hours). Then i set the context length to 744 (31days x 4hours) because i want it to look at the last 31 days. I created a feature called store closed. The network should learn itself, that if that feature is 1, the sales are 0 as well, right?
It runs through, and the results make sense, i just want to know if i forgot something, that i did not think of.
About Hyperparameter Tuning:
How to find the optimal parameter for context_length / batch size? In my understanding batch size means how many batches of context length the epoch uses to train the network. The question here is do the batches overlap, e.g. batch 1 context of context_length 15 starts today and the other one tomorrow, or today and next one at least 15 days away? Also how do i determine how far back my model has to look? My seasonal pattern repeats every 7 days (determined by acf plot), but the time before that might also have an influence. Is context length just try and error? is there a thumb rule for batch size in relation to dataset size and context length?
Thanks in advance,
Alex

To answer one of my questions myself:
Question:

About predicting with a trained model (not evaluating)
So what i did is:

training = ListDataset(
        [{"start": df.index[0],
          "target": df.target[:split_date],
          "feat_dynamic_real": df.exogen_feature[:split_date]
          }],
        freq="d"
    )
    future = ListDataset(
        [{"start": df.index[0],
          "target": df.target[:split_date],
          "feat_dynamic_real": df.exogen_feature[:split_date + timedelta(days=prediction_length)]
          }],
        freq="d"
    )
    predictor = estimator.train(training_data=training)
    prediction = next(predictor.predict(future))

This returns me a Sampleforecast with the predicted values. What i would like to retrieve though is also the borders for the confidence interval. Are those included in the object?

Hello Alex,
sorry for not replying earlier.
The error you are seeing is actually coming numpy. Make sure that your input-data for each time-series has the correct shape (but I think you fixed it in your second example).

About DeepAREstimator parameters:
Does frequency only refer to the gap between two datapoints e.g. "D" or "H", or does it refer to the timeframe we see a pattern repeat, e.g. "7D".

Frequency is the dataset's granularity, so the time-gap between values. It does not refer to some pattern, since that's basically what we ask the model to find for us.

When i have a series of data points where 10 days lay in between, do i specify "10D" or "D"?

10D.

How do i determine the value for the context_length parameter? Do i do this with grid search?

I'm not sure how effective tuning this parameter is. I think a good default is 2x the prediction length.

Does that mean it predicts my prediction_length at num_samples different random points in my test set to evaluate the model?

No. It cuts off prediction_length values from your test-time-series and compares that against the forecast to get metrics. num_samples refers to how many forecasts are generated for each time-series.

It has a parameter "seasonality". I could not find any further information in the documentation. How do i determine the best value for this parameter, and what does it actually mean in regards to my dataset?

I think this argument is used to compare against a seasonal-naive forecast. E.g., if we have hourly frequency, how does our forecast compare against just taking last days value.
These are the default seasonalities used:
    seasonalities = {"H": 24, "D": 1, "W": 1, "M": 12, "B": 5}

The parameter allows to use a custom offset value.

Is there anything implemented, that shows me how important/how much impact which of my exogenous variables is?

No. Although I think that's an interesting topic, it's not on our roadmap.

Since i actually have hourly data from 8am to 8pm, i think it is a total waste to look model each hour individually as daily data because i lose the influence of the previous hours. So what i did it i set my DeepAREstimator frequency to "H" and prediction length to 360 (15days x 24hours). Then i set the context length to 744 (31days x 4hours) because i want it to look at the last 31 days. I created a feature called store closed. The network should learn itself, that if that feature is 1, the sales are 0 as well, right?

Sounds reasonable.
However, in my view your prediction_length and context_length values are uncomfortably large. We use lagged-values to take more data into account than what is provided in context_length. You could try and reduce the values for prediction_length and context_length, but this of course could also worsen the result.

This returns me a Sampleforecast with the predicted values. What i would like to retrieve though is also the borders for the confidence interval. Are those included in the object?

I think you could take quantiles 0 and 100 for that, or access the min and max values of the underlying data.
Still, I think you have to be cautious to not over-interpret the results and there is good reasons why p10 and p90 are common metrics.

training = ListDataset(         [{"start": df.index[0],           "target": df.target[:split_date],           "feat_dynamic_real": df.exogen_feature[:split_date]           }],         freq="d"     )     future = ListDataset(         [{"start": df.index[0],           "target": df.target[:split_date],           "feat_dynamic_real": df.exogen_feature[:split_date + timedelta(days=prediction_length)]           }],         freq="d"     )     predictor = estimator.train(training_data=training)     prediction = next(predictor.predict(future))
why target size for training and testing is same ? by looking at df.target[:split_date] in both of dataset

training = ListDataset(         [{"start": df.index[0],           "target": df.target[:split_date],           "feat_dynamic_real": df.exogen_feature[:split_date]           }],         freq="d"     )     future = ListDataset(         [{"start": df.index[0],           "target": df.target[:split_date],           "feat_dynamic_real": df.exogen_feature[:split_date + timedelta(days=prediction_length)]           }],         freq="d"     )
is this for production? where you want to use all data for training and predict future values with known exog. how to configure the dataset for evaluation ?

Hi @jaeniale-fd, @jaheba, Does this work?
For me it didn't due to shape required can't be provided by below method.
"feat_dynamic_real": df.exogen_feature[:split_date + timedelta(days=prediction_length)]
Please let me know how you made it to work? Thanks!


Add footer display, display all announcements and dismissable option configurations

Added configuration option to display announcements in the page footer.
Introduced a new config setting to display all announcements instead of one random announcement.
Introduced a new config setting to enable/disable dismissable announcements by default.

Thanks for your contribution, I'll review as time allows.


Some Time Fail to scan 1d type Barcode

When we scan 1d type bar code it take more time to scan and some time it gives me wrong values.
Frequency of giving wrong values was 5/10.
How to resolve this?
Is this not support for 1d ?

Feel free to attach a particular barcode that's having trouble, but in general 1-D barcodes can be tough to scan because of their dimensions.


Java enums incorrectly marked final by model loader

With the current mapping for Java enums, the model loader marks the enum class final, but then makes each enum value a static member subclass, which is not possible for a final class.
Either:

don't mark the enum class final, or
much better and more natural, map the enum values to value constructors of the enum class.

Fixed.


Do not allow stopping the Low Frequency clock when it's not started.

Sorry about the rustfmt changes. The important one is moving stop_lfclk into an impl for Clocks<H, L, LfOscStarted. This will allow the compiler to prevent stopping the clock when it's not started.

Overall LGTM, is there any negative effect to stopping the clock if it isn't running? Or is this aimed at preventing developer mistakes?
Also the rustfmt changes look small enough here, though in the future I would prefer if they were at least separate commits, to make the reviewing easier.
Thanks for the contribution!

I actually didn't try stopping the clock while it was stopped. I just noticed this while I was going through the manual for this MCU and figured it should probably be enforced if it's being called out.

LGTM.
Thanks for the contribution :)


Correct notes styling so that they match with the README

What does this PR do (include background context, if relevant)?
Previously, the secretless contributing docs had 'notes' that were not uniform. Some were capitalized, bolded, or not given its own line. Our aim should keep the styling uniform across the Secretless GH docs. I made the styling uniformed with the README.md docs, italics.
What ticket does this PR close?
Connected to #576
Where should the reviewer start?
CONTRIBUTING.MD
What is the status of the manual tests? N/A
Have you run the following manual tests to verify existing functionality continues to function as expected?

[ ] Manually tested K8s CRDs
[ ] Manually tested Keychain provider
[ ] Manually run the K8s demo
[ ] Manually run Kubernetes-Conjur demo with a local Secretless Broker image build of your branch
[ ] Manually run the full demo (optional)

If this feature does not have any/sufficent automated tests, have you created/updated a folder in test/manual that includes:

[ ] An updated README with instructions on how to manually test this feature
[ ] Utility start and stop scripts to spin up and tear down the test environments
[ ] A test script to run some basic manual tests (optional; if does not exist, the README should have detailed instructions)

Links to open issues for related automated integration and unit tests
Links to open issues for related documentation (in READMEs, docs, etc)
Screenshots (if appropriate)
BEFORE:

AFTER:

registry is down, will check error when able to access Jenkins


FYI SPEW

Hey.
I was just made aware your work.
Just an FYI about SPEW.
Not ready for users and so hence not on code.dlang.org.
Much more heavy weight, but may interest you.

Thanks.
Oswald is predominantly a learning project, and is designed to create windows and nothing else. SPEW is (as you said) far too heavyweight for what I want to do with Oswald. I am definitely very interested though.


As a developer, I would like a basic routing example

Type of Change

[ ] New Feature Request
[x] Documentation
[ ] Future Spec
[x] Improvement / Suggestion
[ ] Bug
[ ] Other (please clarify below)

Summary
A basic routing example would be useful to assist any developer with experience using modern javascript framework who is looking to quickly build their first application with evergreen.
Use Case
A simple routing example would be useful for most applications but one specific use case might be a website.
Code Sample
A  router example using lit-redux-router
Pros / Cons
Would certainly help with getting more people to participate in the community and build better applications.

related to #2 .
found this project, which may also be worth looking into
https://github.com/router5/router5/blob/master/README.md


Memory peaks on agent v1.19.2

Describe the bug
One group of jaeger agents (3 replicas), deployed on kubernetes. Not so loaded system - 500-700 spans per second.
Used v1.13.1 for year without any issues, upgraded all componenet to 1.19.2 (also replaced Tchannel with gRPC) and agent memory usage become unstable. Before upgrade agent instances used as low as 16 mb with 64 limit, but after upgrade memory usage peaks appeared and agent got oomkilled. I raised limits, but even 512 mb is not enough.
After few hours and tens of oomkills, I downgraded agent to 1.18.1 version and see no issues so far.
To Reproduce
Steps to reproduce the behavior:

Deploy agent v1.19.2
Send 500-700 spans per second
???
Observe memory usage peaks.

Expected behavior
Agent memory usage close to lower versions.
Screenshots

Version (please complete the following information):

OS: Kubernetes nodes based on Ubuntu 20.04 LTS, 5.4.0-39 kernel
Jaeger version: 1.19.2
Deployment: Kubernetes

What troubleshooting steps did you try?
Collected /metrics, will provide if needed.
Additional context

When you downgraded to 1.18.1, did you change any other configuration? Would you mind sharing your agent configuration?

@jpkrohling the only agent parameter used before and after downgrade is --reporter.grpc.host-port=jaeger-collector:14250.
Also played for a while after got first OOMkills with parameters, but reverted changes singe they didn't helped much:
        - --processor.jaeger-binary.server-queue-size=...
        - --processor.jaeger-binary.workers=...
        - --processor.jaeger-compact.server-queue-size=...
        - --processor.jaeger-compact.workers=...
        - --processor.zipkin-compact.server-queue-size=...
        - --processor.zipkin-compact.workers=...

I'll take a look at list of commits affecting the agent between 1.18.1 and 1.19.2, perhaps I can spot a suspect there. I'll also try to reproduce the problem here. Do you think it would be reproducible with minikube and a simple agent with 64Mi of requests , 256Mi limit? Jaeger's tracegen would probably generate enough load ;-)

@zigmund I'll need your help here. I tried to reproduce your problem but didn't succeed. I have a local minikube cluster with 20Gi of memory available, initialized with:
$ minikube start --vm-driver kvm2 --cpus 6 --memory 20480 --container-runtime=crio --addons=ingress

Using the Jaeger Operator, I provisioned a Jaeger instance with the following spec:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: agent-as-daemonset
spec:
  agent:
    image: jaegertracing/jaeger-agent:latest
    strategy: DaemonSet
    resources:
      requests:
        memory: 64Mi
      limits:
        memory: 256Mi
  storage:
    options:
      memory:
        max-traces: 100000

And here's a trace generator, creating 1 trace per millisecond:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tracegen
  annotations:
    "sidecar.jaegertracing.io/inject": "false"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tracegen
  template:
    metadata:
      labels:
        app: tracegen
    spec:
      containers:
      - name: tracegen
        image: jaegertracing/jaeger-tracegen:1.19.2
        env:
        - name: JAEGER_AGENT_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        args:
        - -pause=1ms
        - -duration=1h
        - -workers=1

The agent pod was up for the whole time, and the metrics shows me that the agent didn't even blink. I changed the Jaeger CR to use 1.18.1, to see if latest was in the same ballpark, and I couldn't spot any differences:

The first part of the graph is for an early test, where I set the tracegen to have 10 workers and no pause. I also forgot to set the pod limits and left for lunch. If I'm reading this data correctly, it shows about 200Mi of memory consumption for about 4M spans per minute.
The second part is jaegertracing/jaeger-agent:latest, with a 64Mi of requests and 256Mi of limit. The tracegen is with a pause of 1ms and 1 worker.
The third part is the same a second part, only with jaegertracing/jaeger-agent:1.18.1 as the image.
Here's how I need your help: are you able to reproduce your performance problems based on this setup? If you can't, could you perhaps then run a git bisect with the first good commit being v1.18.1 and the bad as being v1.19.2? Like:
$ git bisect start
$ git bisect good v1.18.1
$ git bisect bad v1.19.2
$ make docker # this is the easiest, but will build everything, which might take a while
$ podman tag jaegertracing/jaeger-agent:latest quay.io/jpkroehling/jaeger-agent:issue2452-step1
$ podman push quay.io/jpkroehling/jaeger-agent:issue2452-step1

Change the last two commands accordingly, to tag and push to a namespace you own. At this point, configure your agent to use the image you just pushed.
If you don't have Jaeger source code locally to run the previous steps, you should probably do this first (see CONTRIBUTING.md):
$ git clone https://github.com/jaegertracing/jaeger.git
$ cd jaeger
$ git submodule update --init --recursive
$ make install-tools

@jpkrohling I'll try to test with git bisect, but it will take some time.

It's really hard to test good steps since it could take from minutes to much more time to catch first oom. But at least already reproduced oom's at the first two steps. At the moment I'm testing the third step for an hour already and I don't know if it is enough to be certain that the step is good.

How fast did you get OOMs for the first two steps? If they are at around 15m, it might be safe to say that a commit is good if it's not crashing after 30m.

@jpkrohling at the first two steps got first ooms in few minutes.
Got compile error at 4 and 5 steps, testing 6 step at the moment. It seems that https://github.com/jaegertracing/jaeger/commit/be17169a417954a347408708182519e267618500 affected memory usage behavior in out environment.

@jpkrohling completed 6 step with hour+ test period.
be17169a417954a347408708182519e267618500 is the first bad commit
commit be17169a417954a347408708182519e267618500
Author: Pavol Loffay<EMAIL_ADDRESS>Date:   Fri Jun 26 16:11:45 2020 +0200

    Upgrade thrift to 0.13 (#2311)

    * Upgrade thrift to 0.13

    Signed-off-by: Pavol Loffay<EMAIL_ADDRESS>
    * Add context to sampling managers

    Signed-off-by: Pavol Loffay<EMAIL_ADDRESS>

cc @pavolloffay

Here is the upgrade to thrift 0.13 PR https://github.com/jaegertracing/jaeger/pull/2311.  The upgrade was done due to security issues in the old version and also it was needed in OpenTelemetry collector IIRC.
I am not sure what is causing the regression.

@zigmund are you able to get us a memory profile for the running agent? The admin port has pprof endpoints, which you can browse like this:
$ go tool pprof -http localhost:6060 http://your-agent-host:14271/debug/pprof/heap

This will get a web UI on http://localhost:6060 for browsing your agent's memory.

@jpkrohling I'll try but don't think it will help. Agent allocates large amount of memory and releases it next moment, so I have to catch that moment with profiler(?)

That's true. Ideally, you'd capture it every N minutes, so that we'd get at least one profile when the memory is ramping up. There are a few experimental tools that can help with that, but I'm really not sure about their state, like conprof (cc @brancz).
Once we understand better where this memory usage is coming from, we can probably try to reproduce it ourselves, which is the first step to fix the problem.

Conprof is not well optimized itself but used by various organizations so it's definitely worth a try :)

@jpkrohling lowered replicas to 1 and collected profile every second via forwarded port with curl:
while true; do curl http://localhost:14271/debug/pprof/heap -o `date +%s`.pb.gz; sleep 1; done

Raised limit to 2Gb to avoid oom but agent still been killed. :D
But at least I got few identical profiles right before oom. Here is one of them

Looks interesting

I'll try to reproduce it here again. Just to confirm: data is flowing from the agent to the collector, right?

@jpkrohling yes, agent sends data to collector.

@zigmund it's still quite difficult to tell what's going on. The profile you sent seems to indicate that all memory is being allocated in the ReadString from Thrift's Compact protocol, which seems to indicate that you have a span bigger than 930MiB which I doubt is the case, especially because a batch has a max size equal to the limits of a UDP packet, typically 64KiB. Either that, or quite a bunch of spans are locked in that function, which also seems unlikely.
I tried to put as much pressure as I could on the agent, but it keeps with a minimal memory usage. I don't think Elasticsearch is part of the equation here, but I used it anyway as the backend, without luck. Even my memory profile is quite different than yours: every time I get a snapshot, it's mostly showing 1MiB here or 512KiB there, which is in line with the inflight data (like the image at the end).
Could you also get us a goroutine snapshot? If we have a single call to ReadString, it would probably be apparent there.
I would also like to ask you for your exact runtime configuration, like:

where are you deploying it? Bare metal? VM? Kubernetes?
the exact command used to start your Collector (or Deployment, in case of Kubernetes, or the Jaeger CR in case you provisioned it using the Jaeger Operator)
same for the Agent
which client are you using? Jaeger Go Client? OpenTelemetry? Something else?

@jpkrohling

Could you also get us a goroutine snapshot?

Do you mean profile from /debug/pprof/goroutine?debug=2?

where are you deploying it? Bare metal? VM? Kubernetes?

Kubernetes

the exact command used to start your Collector (or Deployment, in case of Kubernetes, or the Jaeger CR in case you provisioned it using the Jaeger Operator)

Env vars passed via configmap:
SPAN_STORAGE_TYPE=elasticsearch
ES_SERVER_URLS=http://es-node1:9200,http://es-node2:9200,http://es-node3:9200
ES_NUM_SHARDS=3
ES_NUM_REPLICAS=0
ES_MAX_SPAN_AGE=24h

same for the Agent

Single argument: --reporter.grpc.host-port=jaeger-collector:14250

which client are you using? Jaeger Go Client? OpenTelemetry? Something else?

golang - https://github.com/jaegertracing/jaeger-client-go
node.js - https://github.com/jaegertracing/jaeger-client-node
php - https://github.com/jukylin/jaeger-php
nginx - https://github.com/opentracing-contrib/nginx-opentracing

Do you mean profile from /debug/pprof/goroutine?debug=2?

Yes, please

golang - https://github.com/jaegertracing/jaeger-client-go
node.js - https://github.com/jaegertracing/jaeger-client-node
php - https://github.com/jukylin/jaeger-php
nginx - https://github.com/opentracing-contrib/nginx-opentracing

Would it be possible to have one agent per client type? If we notice that it only happens with spans from one specific client, we might have a lead...

Yes, please

I'll try to catch memory eating profile with method I used before.

Would it be possible to have one agent per client type? If we notice that it only happens with spans from one specific client, we might have a lead...

I don't think it's possible since we're have tens of services pointed to same jaeger agent endpoint. I probably could configure nginx and node.js services by myself... but all other services' are out of my control.

@jpkrohling Tried to collect goroutine profile containing thrift.(*TCompactProtocol).ReadString for few hours with no luck. :(
Memory allocations can be seen via pprof for a time till GC comes, I think. But gouroutine profile have to be collected right at the moment(?).

Maybe I should make custom build with some debug logs in thrift.(*TCompactProtocol).ReadString? But I'll need help since I'm not so strong in golang.

That is generated code and is being called for almost every field in every object: it would probably generate a lot of noise. We have a log statement at debug level around this place already:
https://github.com/jaegertracing/jaeger/blob/df2582d5331f8b0931a88e0a7c4c32ef788561f0/cmd/agent/app/processors/thrift_processor.go#L120
Are you seeing it often, or does it seem stuck right before the process crashes? But just in case, I built an image for the agent which would add two new prints to the console:
diff --git a/thrift-gen/jaeger/jaeger.go b/thrift-gen/jaeger/jaeger.go
index 1e8cb9f3..dcbeb44a 100644
--- a/thrift-gen/jaeger/jaeger.go
+++ b/thrift-gen/jaeger/jaeger.go
@@ -1855,6 +1855,7 @@ func (p *Batch) Read(iprot thrift.TProtocol) error {
         if err := p.ReadField1(iprot); err != nil {
           return err
         }
+       fmt.Printf("processing batch for service %q\n", p.Process.GetServiceName())
         issetProcess = true
       } else {
         if err := iprot.Skip(fieldTypeId); err != nil {
@@ -1910,6 +1911,7 @@ func (p *Batch) Read(iprot thrift.TProtocol) error {
   if !issetSpans{
     return thrift.NewTProtocolExceptionWithType(thrift.INVALID_DATA, fmt.Errorf("Required field Spans is not set"));
   }
+  fmt.Printf("processed batch for service %q\n", p.Process.GetServiceName())
   return nil
 }

This will print the service name for a batch before processing its spans, and after it finished parsing the batch: it's reading one span from the batch that seems to be breaking, so, I expect you to eventually see one print but not the other. The hypothesis I had earlier will confirm if you end up with the name of the service for all crashes, or at least with services using the same client type.
Image with these extra prints: quay.io/jpkroehling/jaeger-agent:issue2452-extra-verbose

@jpkrohling launched agent from your image, enabled debug and see tons of logs:
{"level":"debug","ts":1600148307.8616223,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":847}
{"level":"debug","ts":1600148307.863402,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":847}
{"level":"debug","ts":1600148307.8717968,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":25111}

But don't see any lines processing batch for service...
Also we have some logs about bad spans:
{"level":"error","ts":1600148418.579937,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
{"level":"error","ts":1600148418.816928,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: *jaeger.Span field 27 read error: don't know what type: 14","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
{"level":"error","ts":1600148433.9862144,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Required field TraceIdLow is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}
{"level":"error","ts":1600148451.748264,"caller":"processors/thrift_processor.go:123","msg":"Processor failed","error":"*jaeger.Batch error reading struct: *jaeger.Span error reading struct: Required field TraceIdLow is not set","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:123\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:87"}

But they been there for long time before upgrade.

launched agent from your image, enabled debug and see tons of logs:

What's the last Span(s) received by the agent entry you see before the crash?

But don't see any lines processing batch for service...

Something might have happened between building the binary and placing it in the image. Anyway, here's a new one, which I confirmed to work: quay.io/jpkroehling/jaeger-agent:issue2452-extra-verbose-2:
$ podman run -p 6831/udp quay.io/jpkroehling/jaeger-agent:issue2452-extra-verbose-2 --reporter.grpc.host-port=<IP_ADDRESS>:14250 --log-level=debug
...
{"level":"info","ts":1600158147.7916539,"caller":"command-line-arguments/main.go:78","msg":"Starting agent"}
{"level":"info","ts":1600158147.7916827,"caller":"healthcheck/handler.go:128","msg":"Health Check state change","status":"ready"}
...
{"level":"debug","ts":1600158156.744476,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":376}
processing batch for service "jaeger-query"
processed batch for service "jaeger-query"
{"level":"debug","ts":1600158156.7448108,"caller":"reporter/client_metrics.go:177","msg":"received batch from a new client, starting to keep stats","client-uuid":"5b9222a904b62ac6"}

Also we have some logs about bad spans:

Do you know which clients/services are sending them? Those failures are happening very close to where you are seeing the memory explode. How does the metric jaeger_agent_thrift_udp_t_processor_handler_errors_total looks like? Are they all part of model="jaeger",protocol="compact", or do you see some other variants there? How about jaeger_agent_thrift_udp_server_packets_processed_total?

@zigmund, do you have any news for us? I'm really curious on how this develops :-)

@jpkrohling sorry, I'm now in business trip for few days. Will test as soon as possible.

@jpkrohling, checked metrics.
sum(rate(jaeger_agent_thrift_udp_t_processor_handler_errors_total[1m])) by (model, protocol):
{model="jaeger",protocol="binary"} | 0
{model="jaeger",protocol="compact"} | 0.06667950988310203
{model="zipkin",protocol="compact"} | 0

sum(rate(jaeger_agent_thrift_udp_server_packets_processed_total[1m])) by (model, protocol):
{model="jaeger",protocol="binary"} | 0.35553778133262237
{model="jaeger",protocol="compact"} | 73.39500099980003
{model="zipkin",protocol="compact"} | 0

Currently unable to test debug image v2, will try to do it later.

@jpkrohling, I think I found two agent-breaking services, both are php-written. Will try to catch breaking traces, but don't know how at the moment. :D

I think I found two agent-breaking services, both are php-written

That's actually great news. This means that the Agent might not be at fault here at all for common cases. We still need to figure out what type of incoming data is causing the problem, but I believe it does change the criticality of the issue.

Will try to catch breaking traces, but don't know how at the moment.

I can't help much here, but perhaps you can start by checking which operations are expected to exist vs. which ones actually exist. The missing one(s) might be worth double-checking.

start by checking which operations are expected to exist vs. which ones actually exist.

These two services are pretty large monolith php applications, traces often consist of hundreds spans, so It's near impossible to catch missing.

Alright, let us know once you have more info then, or if there's anything we can do on our side to help you.

@zigmund, were you able to find a pattern already? I'm worried that we might have a bigger issue, as bad clients should be able to bring down the agent...

@jpkrohling, unfortunately, didn't found any additional info. :(

Are you still experiencing those problems, or have you disabled tracing for the services that are causing problems?

@jpkrohling I'm using your extra verbose image for debug and reverting back to 1.18.1 for regular usage. The services' tracing is enabled.

Got it. We might have newer versions of the models/protobuf soon, which might help here. I'll ping you once we have it ready, in case you are able to try it out before we release it.

OK, I'm ready to test.

@zigmund We released 1.20 some days ago. Would you be able to try it out?

@jpkrohling checked 1.20 - same behavior as 1.19.2

Thanks for the confirmation. I'm leaving this open, but unless there's new information, there's not much we can do to reproduce and finally fix this problem.

Hi @zigmund
Thank you so much because of this issue. I'm facing to the same problem. I'm also using jukylin/jaeger-php for my project. After a long day for debugging, I figure out that the reason is about my data. It isn't all data, just only malformed data - where percentage is very small. So it is really difficult to debug if you are on production.
You can read this comment https://github.com/jaegertracing/jaeger/issues/2638#issuecomment-742592854
Hope you can resolve this issue.

Hi @zigmund
Thank you so much because of this issue. I'm facing to the same problem. I'm also using jukylin/jaeger-php for my project. After a long day for debugging, I figure out that the reason is about my data. It isn't all data, just only malformed data - where percentage is very small. So it is really difficult to debug if you are on production environment.
You can read this comment #2638 (comment). Thank you so much @jpkrohling
Hope you can resolve this issue.


[#394] Added missing exception documentation, emitted by the establis…

…hConnection method.

Good to know :)


'starkGeneratePatchDebug' not found

when run gradle :stark-sample:clean :stark-sample:starkGeneratePatchDebug，it happens

Configure project :stark-sample
==========Stark Gradle: Stark multiDexEnabled = false isGeneratePatch = true

FAILURE: Build failed with an exception.

What went wrong:
Task 'starkGeneratePatchDebug' not found in project ':stark-sample'.

starkGeneratePatch + BuildType


middleware/proxy: implement Exchanger

By defining and using an proxy.Exchanger interface we make the proxy
more generic and we can then fold back httproxy into proxy.
This overrides #463 and #473 and should make futures extensions rather
trivial

Current coverage is 38.45% (diff: 12.00%)

Merging #480 into master will increase coverage by <.01%

@@             master       #480   diff @@
==========================================
  Files           116        116
  Lines          5790       5797     +7
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           2226       2229     +3
- Misses         3381       3385     +4
  Partials        183        183

Powered by Codecov. Last update b6a2a5a...e372a38


generateServiceProviderMetadata: remove callbackUrl dependency

This change mirrors generateAuthorizeRequest's behavior of constructing callbackUrl from protocol and path if callbackUrl is not supplied.
I kept req independent from generateServiceProviderMetaData by adding a host option that defaults to localhost.

Looks good to me, can I get you to resolve the conflicts I just created, and maybe add a test case under generateServiceProviderMetadata that constructs the callback url from protocol/path/host?
Thanks!

Done!


[Bug] cookie.name.remove() not removing cookie as expected

What version of Elysia.JS is running?
1.0.7
What platform is your computer?
Darwin 19.6.0 x86_64 i386
What steps can reproduce the bug?

Set cookie

cookie.auth.set({
    path: '/',
    value: Math.random().toString(),
    maxAge: 7 * 86400,
    secure: true,
    httpOnly: true
})

Attempt to remove cookie

cookie.auth.remove() // not working

See browser Devtools > Application > Cookies has a cookie named auth

What is the expected behavior?
Expected to have the cookie removed in the client. We can do this by setting maxAge of cookie to 0.
cookie.auth.update({
    path: '/',
    value: '',
    maxAge: 0,
    secure: true,
    httpOnly: true
})

What do you see instead?
No response
Additional information
No response

What is more bewildering is that I see nothing out of place in elysia code...
https://github.com/elysiajs/elysia/blob/e8ddb2f76c273275940c0f80881931f61ab918db/src/cookies.ts#L239-L249
On second look, we should provide an option to path. Usually auth stores cookies at / path. but cookie.name.remove() does not specify the path so the cookie stays on the client.

Hi, would like to help but I'm unable to reproduce on my end using the following code.
import { Elysia } from 'elysia'

const app = new Elysia()
	.get('/', ({ cookie }) => {
		cookie.auth.set({
			path: '/',
			value: Math.random().toString(),
			maxAge: 7 * 86400,
			secure: true,
			httpOnly: true
		})
	})
	.get('/remove', ({ cookie }) => {
		cookie.auth.remove()
	})
	.get('/value', ({ cookie }) => {
		return cookie.auth.value
	})
        .listen(3000)

@SaltyAom first here are the elysia related packages on the project:
"@bogeychan/elysia-polyfills": "^0.6.4",
"@elysiajs/cors": "^1.0.0",
"@elysiajs/swagger": "^1.0.0",
"elysia": "^1.0.7",

Using Node version 18.14.2

And here is the exact code used to test the bug, it is consistently happening, this might be a Node thing but idk.
import { Elysia, t } from 'elysia'

const routes = new Elysia({ prefix: '/experiment', name: 'routes:experiment' })
  .derive(({ cookie, set }) => {
    return {
      userHasRole: async () => {
        console.log('cookie.auth: ', cookie.auth)
        console.log('cookie.auth.value: ', cookie.auth.value)
        if (cookie.auth.value) {
          return 'User is not important.'
        }
        set.status = 'Unauthorized'
        return "You seem out of place."
      },
      login: () => {
        cookie.auth.set({
          path: '/',
          value: Math.random().toString(),
          maxAge: 7 * 86400,
          secure: true,
          httpOnly: true
        })
      },
      signOut: () => {
        cookie.auth.remove() // not working
        // cookie.auth.update({
        //  path: '/',
        //  value: '',
        //  maxAge: 0,
        //  secure: true,
        //  httpOnly: true
        // })
        console.log('cookie.auth: ', cookie.auth)
        return 'User logged out.'
      }
    }
  })
  .get('/user-role', async ({ userHasRole }) => {
    return {
      message: await userHasRole()
    }
  })
  .get('/login-user', ({ login }) => {
    return {
      message: login()
    }
  })
  .get('/signout-user', ({ signOut }) => {
    return {
      message: signOut()
    }
  })

Hello, I'm on Deno, and I cannot remove cookie. I tried delete cookie["some_cookie_name"] and cookie["some_cookie_name].remove(). And it does not work.


fix: resolve compatibility issues with Java and Rust clients

The replication test failed. Is it related to the changes in this PR?

The postgresql port is provided for direct DuckDB SQL execution and FLOAT and DOUBLE is different in DuckDB. Therefore I think we should not do this replacement.

That means, even if the user use psql to connect and create a table with pg grammar, FLOAT is still recognized as F32?

I think so. Maybe we can skip the FLOAT type in the client tests. FLOAT is not in https://www.postgresql.org/docs/current/datatype.html

Good luck — you're almost there!


UTM and Referrer Data Not Being Passed

Expected Behavior
When initializing with {includeReferrer: true, includeUtm: true, includeGclid: true} and logging an event, UTM or Referrer data is not included.
Current Behavior
Events submit, but UTM and Referral data isn't included.
Steps to Reproduce

Initialize / Set up events as follows

<script type="text/javascript">
  (function(e,t){var n=e.amplitude||{_q:[],_iq:{}};var r=t.createElement("script")
  ;r.type="text/javascript"
  ;r.integrity="sha384-girahbTbYZ9tT03PWWj0mEVgyxtZoyDF9KVZdL+R53PP5wCY0PiVUKq0jeRlMx9M"
  ;r.crossOrigin="anonymous";r.async=true
  ;r.src="https://cdn.amplitude.com/libs/amplitude-7.2.1-min.gz.js"
  ;r.onload=function(){if(!e.amplitude.runQueuedFunctions){
  console.log("[Amplitude] Error: could not load SDK")}}
  ;var i=t.getElementsByTagName("script")[0];i.parentNode.insertBefore(r,i)
  ;function s(e,t){e.prototype[t]=function(){
  this._q.push([t].concat(Array.prototype.slice.call(arguments,0)));return this}}
  var o=function(){this._q=[];return this}
  ;var a=["add","append","clearAll","prepend","set","setOnce","unset"]
  ;for(var u=0;u<a.length;u++){s(o,a[u])}n.Identify=o;var c=function(){this._q=[]
  ;return this}
  ;var l=["setProductId","setQuantity","setPrice","setRevenueType","setEventProperties"]
  ;for(var p=0;p<l.length;p++){s(c,l[p])}n.Revenue=c
  ;var d=["init","logEvent","logRevenue","setUserId","setUserProperties","setOptOut","setVersionName","setDomain","setDeviceId", "enableTracking", "setGlobalUserProperties","identify","clearUserProperties","setGroup","logRevenueV2","regenerateDeviceId","groupIdentify","onInit","logEventWithTimestamp","logEventWithGroups","setSessionId","resetSessionId"]
  ;function v(e){function t(t){e[t]=function(){
  e._q.push([t].concat(Array.prototype.slice.call(arguments,0)))}}
  for(var n=0;n<d.length;n++){t(d[n])}}v(n);n.getInstance=function(e){
  e=(!e||e.length===0?"$default_instance":e).toLowerCase()
  ;if(!n._iq.hasOwnProperty(e)){n._iq[e]={_q:[]};v(n._iq[e])}return n._iq[e]}
  ;e.amplitude=n})(window,document);

  amplitude.getInstance().init("MYKEY1234", {includeReferrer: true, includeUtm: true, includeGclid: true});

  amplitude.getInstance().logEvent('Viewed Page', {
    url: window.location.pathname,
    domain: window.location.host
});
</script>

Navigate to URL with utm params (e.g. https://example.com/?utm_source=test123&utm_medium=test123&utm_campaign=test123) directly, or via referring link from another site.

Note that utm params aren't included in event data.

Environment

JS SDK Version: 7.2.1
Installation Method:

Hey @tnoetzel, UTM params aren't represented as event properties but user properties. Could you check your project if this is correct in your project?

According to the docs:
Once you set the includeUtm option to true, the JavaScript SDK will automatically pull UTM parameters from the referring URL and include them as user properties on all of the relevant events...
And
Once you set the includeReferrer option to true, Amplitude will capture the referrer and referring_domain for each session and set them as user properties on all of the relevant events...
Neither of these seem to be true. The utm/referred isn't bundled with events or posted in a separate call like setUserProperties. Best I can tell, there's no request that's sending this data, nor is it showing up in the Amplitude console.

I looked at your source code again and believe the problem lies with amplitude being incorrectly initialized. The config options are being passed into the user id param.
If it's changed to
The API reference page also has more details

Good catch! For what it's worth, it would be really helpful for future devs to have this information grouped with the rest of the documentation here: https://developers.amplitude.com/docs/javascript
The library didn't throw an error, and I had no idea that other resource even existed...


Reduce the memory footprint associated with the ProcessedDocBatch

Right now, we run the document processing and the indexing on two different actors in order to take advantage of multithreading. (Side-note: it has been observed by @PSeitz that multithreading does not actually happen due to the unstealable fifo slot. The latter will be solved once we merge.)
The document processing takes a batch of docs in the form of json strings, parse them into a TantivyDocument.
The TantivyDocument is a structured object in which every string/object field is its own allocation.
This considerably inflates the memory footprint (probably due to pointers, memory alignment) and number of allocations.
In addition, right now we keep accounting the memory associated with the TantivyDoc as being equal to the JSON document length in the in flight memory counter, which is inaccurate.
We want a solution to reduce the memory footprint and ideally measure more accurately the associated memory footprint.
A monkey patch simply serializing the tantivy doc in the docprocessor and deserializing it in the indexer proved that the
benefit was significant.
Tantivy now has a Document trait. It would be good to have an implementation that works over a serialized buffer.
Working off a zero-copy solution straight from json is interesting in the long term, but I think an intermediary solution where we replace the tantivy document with something that allows zero-copy (rkyv, something adhoc, or whatever we can come up with) would be a simple and large win.

CompactDoc is now used in quickwit, which is similar to serialized JSON in memory consumption
https://github.com/quickwit-oss/tantivy/pull/2402

In addition, right now we keep accounting the memory associated with the TantivyDoc as being equal to the JSON document length in the in flight memory counter, which is inaccurate.
We want a solution to reduce the memory footprint and ideally measure more accurately the associated memory footprint.

This should be easy to add to CompactDoc, but in the meantime the serialized JSON should be a pretty good estimate.


Data validator UI: show the number of extension record in relation to the core numbers

show the number of extension record in relation to the core numbers, e.g 45% if its less then the core or 780% if its more

This would only make sense if it was gouped by the core row id.
Closing


[GHSA-v35c-49j6-q8hq] Security Constraint Bypass in Spring Security

Updates

Affected products

Comments
Add .RELEASE suffix for a more accurate description of the affected version.

Hi @SunBK201, the affected product in this advisory does not appear to use the .RELEASE suffix. Do you have any reference links to support this change?

According to the list of versions in Maven, it should be added .RELEASE suffix.


[Feature] Add style guide links

Add style guide links for each cop to the RubyDoc documentation site.
This is great for IDE/text editor integration and means that users don't have to copy/paste RSpec/CopName into Google to find documentation. Because you guys already use RubyDoc, we basically get this for $free.99.
Rubocop uses the same approach:
https://github.com/bbatsov/rubocop/blame/master/config/disabled.yml#L29-L30

On the CLI:
bundle exec rubocop --display-style-guide
-- > C: RSpec/SubjectStub: Do not stub your test subject.

(http://www.rubydoc.info/gems/rubocop-rspec/RuboCop/Cop/RSpec/SubjectStub)

In editor (the link icon takes you to the Rubydoc documentation):

👍

👍

This is a neat idea, thank you! I'm curious though if this clobbers the main StyleGuideBaseURL value for RuboCop. Do you know?

@backus You are right, it does clobber. Let me think of a workaround (if any)

I believe the workaround would be to just have the base url in all of our cop docs. We could have a buildstep that checks this I guess

@backus looks like if you just define the entire URL, then base url config from Rubocop is ignored 😄 .

Great!

Gonna open/search for issue in Rubocop, wonder if there's a cleaner way to do it, but I hope that's not a blocker!

Not a blocker at all. This whole gem is technically a violation of public API heh. As long as something in the build confirms that all the links are using the proper base url. Seems like it would fit in nicely with the config generator

Looks great! Could you squash your commits down @maschwenk? I'll merge after that

@backus done. You know you can configure that repo-wide? That's what I do at work because I'm very lazy ;)

@maschwenk Yeah I should consider just doing that at some point. I also know some people care about squashing down their work in a certain way that gives it a clean description.
Thanks for the contribution!


Ionic 2 pull to refresh + ion-list and open navigation content disappears

Ionic version:  (check one with "x")
[ ] 1.x
[ x] 2.x
I'm submitting a ...  (check one with "x")
[ x ] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:
Currently i am using the Pull the refresh and the ion list component on a page. My content disappears when i use the pull to refresh and after that i open the navigation. Currently i am not able to setup a plunker but i will try that later.

Expected behavior:
When i use the pull to refresh and scroll down the list and then open the menu the content will not be gone.
Steps to reproduce:

Create a ion list that contains enough items so the list is scrollable.
Use the pull the refresher
Afther the refresh is done, scroll the list down.
Now open the navigation panel, the content will dissapear.

Other information:
Currently i cannot setup a plunker with RC5, i will try that later and add the link in this ticket.
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
Cordova CLI: 6.4.0
Ionic Framework Version: 2.0.0-rc.5
Ionic CLI Version: 2.1.18
Ionic App Lib Version: 2.1.9
Ionic App Scripts Version: 1.0.0
ios-deploy version: Not installed
ios-sim version: Not installed
OS: Windows 10
Node Version: v6.2.1
Xcode version: Not installed

I just noticed the same on pages without refresher, just have a list in the page, scroll down a bit and open the side.menu, the content will fully or partially disappear and will pop back when you close the window.  I only see it on iOS

For me it will not happen always when not using the refresher first. If i use the refresher the content will always disappear or partially disappear. I didnt had the time to test it on ios. Hope that they come with a fix soon :(

Any feedback from the ionic team would be nice

I have the same issue from at least RC4, may be earlier. @bolivir, have you find some temporary fix or some another solution.

I think this is related to a bug I submitted guys, see the gif in the bug report:
#9518

@daveshirman i think both tasks have a one reason. But i have this bug without virtual scroll element. I have this bug only when i use menu component with ion-refresher.

This doesn't seems like refresher problem. I do have problem without refresher.
but same problem as https://github.com/driftyco/ionic/issues/9518

I found that, the issue happens if translate: transformZ(0px); is added to .scroll-content element, and that is what ion-refresher do when activated.
If I overrided that attribute with none, everything goes ok.

I am experiencing similar problems, and I am trying to change in this section in app.scss
ion-content {   pointer-events: auto !important; }


bug: Song put on loop will "keep playing" long after the bot has been disconnected

Describe the bug
Song was played using the bot three days ago, put on repeat, and later the bot was disconnected (not sure if by itself or manually). Bot will still say it is looping the song every 12 hours or so, despite the song being only a couple minutes long.
To Reproduce

Queue a song
Loop the song
Disconnect bot manually or vacate the channel for auto disconnect, not sure what happened but I suspect former
Bot will post in channel saying song is looping still within 12-24 hours.

Expected behavior
The bot should not be ghost looping a song.
Screenshots

edit; still going

Additional context

Fixed in #1787.


Binding generation issue

I've been revisiting CppSharp in the hope that I'd be able to programatically generate bindings for libspotify. I have been able to compile the project, and it does run, however I am hitting a bit of a wall when it comes to actually setting up the library for use. libspotify relies on typedefs for opaque types, which appears to cause issues with CppSharp.
typedef struct sp_session sp_session; ///< Representation of a session
typedef struct sp_track sp_track; ///< A track handle
typedef struct sp_album sp_album; ///< An album handle
typedef struct sp_artist sp_artist; ///< An artist handle
typedef struct sp_artistbrowse sp_artistbrowse; ///< A handle to an artist browse result
typedef struct sp_albumbrowse sp_albumbrowse; ///< A handle to an album browse result
typedef struct sp_toplistbrowse sp_toplistbrowse; ///< A handle to a toplist browse result
typedef struct sp_search sp_search; ///< A handle to a search result
typedef struct sp_link sp_link; ///< A handle to the libspotify internal representation of a URI
typedef struct sp_image sp_image; ///< A handle to an image
typedef struct sp_user sp_user; ///< A handle to a user
typedef struct sp_playlist sp_playlist; ///< A playlist handle
typedef struct sp_playlistcontainer sp_playlistcontainer; ///< A playlist container (playlist containing other playlists) handle
typedef struct sp_inbox sp_inbox; ///< Add to inbox request handle

Above shows how the typedefs are written, this causes CppSharp to give the following errors (many more in practice, however all to the same effect).
Parsing libraries...
Parsed 'libspotify.lib'
Indexing library symbols...
Parsing code...
#include "..." search starts here:
#include <...> search starts here:
 ../../../libspotify/include/libspotify
 .\lib\clang\3.7.0\include
 C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include
 C:\Program Files (x86)\Windows Kits\8.1\\include
End of search list.
Parsed 'api.h'
#include "..." search starts here:
#include <...> search starts here:
 ../../../libspotify/include/libspotify
 .\lib\clang\3.7.0\include
 C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include
 C:\Program Files (x86)\Windows Kits\8.1\\include
End of search list.
Processing code...
Pass 'CppSharp.Passes.CleanUnitPass'
Pass 'CppSharp.Passes.SortDeclarationsPass'
Pass 'CppSharp.Passes.ResolveIncompleteDeclsPass'
    Unresolved declaration: sp_session
    Unresolved declaration: sp_track
...
Pass 'CppSharp.Passes.CheckIgnoredDeclsPass'
    Function 'sp_session_create' was ignored due to incomplete param
    Function 'sp_session_user' was ignored due to incomplete return decl
    Typedef 'sp_session' was ignored due to incomplete type
    Typedef 'sp_track' was ignored due to incomplete type
    Typedef 'sp_album' was ignored due to incomplete type
...
Pass 'CppSharp.Passes.CheckIgnoredDeclsPass'
    Property 'logged_in' was ignored due to ignored decl
    Function 'sp_session_playlistcontainer' was ignored due to incomplete return decl
Pass 'CppSharp.Passes.CheckFlagEnumsPass'
Pass 'CppSharp.Passes.CheckDuplicatedNamesPass'
Pass 'CppSharp.Passes.RenameRootNamespacesPass'
Generating code...
Generated 'api.cs'

The full log can be found at https://gist.github.com/benpye/b05437bfda34acf6494e . Sorry if this is an inappropriate place to ask about this, however documentation seems very light on the ground at this point.

I'm not dis-encouraging you to create bindings for libspotify, but you might be interested in this: https://github.com/torshy/torshify

Thanks, I will probably use that however, I am still interested in how I am misusing CppSharp. Up until now I hadn't found anything other than SpotiFire which is Windows only.

As you've found out our support for C is not very robust so it doesn't surprise me that we need a few tweaks here and there to make it work with the idioms in that language. This seems one of those cases.
We already have an IsOpaque property that can be used to signal this: https://github.com/mono/CppSharp/blob/d2d941638a87656c429121ef99ccc023b3dce2e7/src/AST/Class.cs#L84
You can try to set that on your types, like the outdated Clang sample does: https://github.com/mono/CppSharp/blob/10d9384ccace01fc46aa395f0a82db2ae279015e/examples/Clang/Clang.cs#L18
Though it's quite possible that the generator itself or the passes that check for ignored types need some work to let those opaque types through.

Yeah I did try setting IsOpaque, both by finding the class and the helper method, neither worked unfortunately. I believe what you said about the later passes needing to allow opaque types is true.

I wonder what your advice is on handling types that are not really declared, as in, their layout is not declared but the type is used as a pointer in a bunch of methods.
Currently CppSharp just ignores them as being incomplete and doesn't generate bindings for any method containing an argument or return type with that contain such an incomplete type.

May be I'm doing something wrong by current version of CppSharp can not read typedef definitions from another .h file :( I have to manually copy all my 'typedef' lines to every (!) file where it's used :(

Could you create another issue and post some more details about your problem? You definitely should not need to manually copy typedefs, the parsing is handled by Clang.

Hi @tritao - this is gist: https://gist.github.com/jasper22/f90d7594045a24e1ac30
but may be I did something wrong :(
P.S. If you need libELF it's here: https://github.com/jasper22/libELF

From quick glance, I think the problem is that you're trying to pass all libDWARF headers to CppSharp, I suspect some of them are not meant to be included on their own but only as part of other files. For instance, dwarf_abbrev.h does not include any other files, even though it uses types that are defined outside its translation unit.
The same will happen if you try to use those errors with a C++ compiler.

This is annoyingly still an issue, it also affects the SDL example. Any new ideas on how to get around this?

Hi @benpye, I will look into fixing the issue.

I believe this should be fixed.


Bug in next_prime

https://github.com/dignifiedquire/num-bigint/blob/f6b46ffe320b113437738e348e265c5d2b13617f/src/prime.rs#L144
The '/' should be changed to '%'

Is this the root cause of #50?


Add support for CURLFORM_STREAM

Currently it's impossible to upload multiform files that are dynamically read from something. In raw curl one seems to use CURLFORM_STREAM paired with CURLOPT_READFUNCTION for this. Would be great if this could be supported somehow.

Seems plausible to me!
Are you familiar with how this works in C? I'm curious to clarify how CURLOPT_READFUNCTION and CURLFORM_STREAM interact if you set both

The form thing has a data parameter which is pssed to readfunction as last parameter and as far as i remember overwrites what is passed there normally by default.

Ok! I think I've got an implementation at https://github.com/alexcrichton/curl-rust/pull/164, mind giving it a spin and see if it works out for you?

does this work now ? can the PR be merged ?


[Slack Status] Automatically set status based on location

Extension
https://www.raycast.com/petr/slack-status
Description
We use Slack status to communicate where in the world our distributed team is located:

I'd love to be able to automatically update my Slack status using this Raycast extension detecting which location I'm at.
Who will benefit from this feature?
Any company with a distributed team
Anything else?
No response

That's a cool idea. Would you like to run a command or have it completely automatic? The latter might be a bit unexpected.

I would like the latter, but the former would also be fine.
Makes me think if there could be a way to build Zapier into Raycast… 🤔


src/framing.c: close compile warning under windows with msbuild

Close the C4244 warning while compiling under windows with cmake+msbuild.

Thanks, applied in 6e9f7cc2f64c9e659c6ca7cde6737f5d1d564b5e


Release plan for 1.1 and 2.0

I would like to have a bit of clarity in order to have my focus targeted on certain issues.
Currently there are Milestones for 1.1 and 2.0 which both have issues marked for them.
Also there are plenty of issues that have no milestone, but might need to be fixed before either of those releases can come true.
What I have gathered from the comments, regarding PhantomJS version parity, is that the release 1.1 would support PhantomJS 1.9.x and release 2.0 would be the starting point for supporting PhantomJS 2.0.
What is not fully clear to me, is that will the support for PhantomJS 1.9.x be preserved in CasperJS 2.0?
I would like to have the release 1.1 to be released soon, so that the focus for PhantomJS 2.0, thus release 2.0 could be increased.
@n1k0 are the milestones up to date, and if not, would it be possible to have them reviewed and "locked" so that when the issues are fixed, the releases can be done?
Once there is a plan, it can be written in the documentation and users can be aware of it.

@n1k0 are the milestones up to date

Not at all.

if not, would it be possible to have them reviewed and "locked" so that when the issues are fixed, the releases can be done?

I don't have the required time to handle this workload properly (nor have the required energy as well tbh); would you want to drive this — in collaboration with @mickaelandrieu?
The idea would be to triage issues & PRs and ensure they're all qualified and if so that the proper milestone is assigned — you would decide. For PRs, code could be reviewed by both of you, and I could be pinged only if you disagree on some particular change. Then once everything is triaged and PRs are landed, I can take care of releasing new versions.
Would this work for you guys?
Note: I'll try to initiate project ownership transfer to people having more time and active usage of it than me asap, I've been a really bad maintainer for the past year.

@n1k0 for me it's ok.
What I suggest for the next 1.1 release is:

fix critical bug if exists > issues/PR usage

What I suggest for the next 2.0-dev is:

casperjs2 should be an npm module
casperjs2 should be compatible with phantomjs >2 & < 3
casperjs test api should be move on this own repository, and we should work on a better integration of libraries like mocha: in others words, casperjs2 shouldn't be a "test library" anymore

This is a lot of work and I think that we can probably release 1.1 "stable" soon, and start 2.x-dev release now with a target of 2016 for a first 2.0 "stable" release.
Note this is only suggests, and I will be very happy to accepts pros and cons about it, and also consider totaly differents suggests :)

@n1k0 ok for me too.
@mickaelandrieu sounds great, splitting into modules and use of npm
Perhaps it would be the most clear solution to create a v1.1branch which would serve only as preparation for the 1.1 release and once done, perhaps maintain few bug fixed, namely 1.1.1 and so forth.
Master branch would then start going towards 2.0, as the core casperjs modules, while the test framework could move to casperjs-test (or something similar).
All of this could be under the GitHub organisation called...? casperjs.
I will start looking at milestones and putting issues in/out. In case either/anyone sees something that is out of place, please let me know.

@paazmaya :+1: totaly agree on this point,

1.1 should be moved to his own branch
master should reflect 2.0 branch and be able to accept BC.

Let's take 2 or 3 weeks to qualify issues and pull requests to decide what will be the 1.1, and move all others to the 2.0 milestone.

Sounds good!

Let's take 2 or 3 weeks to qualify issues and pull requests to decide what will be the 1.1, and move all others to the 2.0 milestone.

See my 2 cents in depth here.
tl;dr

1.0.* old and busted (obsolete): supports phantomjs up to 1.8.* (< 1.9.0)
1.1.* next/current stable: supports phantomjs 1.9.1 and later as well as slimerjs (experimental), we are only a few steps away from that (as soon as possible, Q1/2016 the latest)
2.0.* new hotness: will support phantomjs 2 and later, possibly breaks API with 1.* versions, comes with fancy new API and modular structure -- let's see if there is enough interest / manpower to really tackle this before diving deep into it

Pretty sure this is obsoleted by the new 2.0 thread and the fact that 1.1 has been released.


Unable to resolve module

I'm getting this issue trying to implement PubNub onto my application.
r: Unable to resolve module ./views/EmojiPicker from C:\Users********\src\App.tsx:
None of these files exist:

src\views\index(.native|.android.js|.native.js|.js|.android.json|.native.json|.json|.android.ts|.native.ts|.ts|.android.tsx|.native.tsx|.tsx)

src\views\index\index(.native|.android.js|.native.js|.js|.android.json|.native.json|.json|.android.ts|.native.ts|.ts|.android.tsx|.native.tsx|.tsx)

If you need anything else added, please comment down below, hopefully we can resolve my issue.

Was fixed by putting ChatView and EmojiPicker on the same App.tsx file.


Temporary workaround for JGRP-1942. To be reverted when we upgrade to…

… JGroups 3.6.7.
Apply to master and 8.0.x

Integrated in master and 8.0.x, thanks Tristan!


[stable/redis] Replication not working

Describe the bug
A clear and concise description of what the bug is.
Version of Helm and Kubernetes:
Kubernetes v1.10.11
Helm v2.13.1
Which chart:
stable/redis
What happened:
Slaves got disconnected from master
info.replication.master_link_status=down
What you expected to happen:
Whenever I set a key in master, it should be replicated to existing slaves
How to reproduce it (as minimally and precisely as possible):
Just run helm install stable/redis --name test -- on local

Hi,
We would need a bit more details about the platform you are using. In our environment it worked without issues.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
<IP_ADDRESS>:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=<IP_ADDRESS>,port=6379,state=online,offset=308,lag=1
slave1:ip=<IP_ADDRESS>,port=6379,state=online,offset=322,lag=0
master_replid:f335276d7e32b6d0cef97b8a468682bf3c32a2d8
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:322
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:322


Support Windows

[x] Remove dependency on core library (use Cmdliner as a CLI parser); fixed with cfd222f0b1064c1de38f55756040ad4714c66c1e.
[x] Linting does not work on Windows so we skip it.
[x] Dune's mdx support does not work on Windows (reported here https://github.com/ocaml/dune/issues/3849). Switched to Cram tests at 3ac067a4735cf4c028438b66b9ed92aa6dd22760.

Fixed with 3ac067a4735cf4c028438b66b9ed92aa6dd22760 and 87566e6b83b047986b25eed1e8232801300296ff.


No index signature error

I asked this question on SO: https://stackoverflow.com/questions/57672886/using-global-filters-in-vue-with-typescript
But, 13 days have passed and nobody has the answer for my question, therefore i'm asking for a help here.

Have you tried Gitter?
https://gitter.im/Microsoft/TypeScript

@AnyhowStep would try, thanks. So it's impossible to use Object.keys() with indexes?

If you are absolutely sure Object.keys() will return (keyof T)[], then you can use this,
Object.keys(filters) as (keyof typeof filters)[]

Be aware that it might be fine in specific cases but is unsafe in general.

interface Foo {
    x : string
}

const obj = { x : "hi", y : "bye" };
const foo : Foo = obj;

//Are you expecting `["x"]` ?
const keys = Object.keys(foo);

//Prints `["x", "y"]`, not `["x"]`
console.log(keys);

Playground


No Move and Copy in Unified Inbox contrarily to given email address inbox

+13

Same issue if click on an email notification which opens the given email, usinf back button redirects to the Unified inbox, hence face the same issue.

Sometimes have ⋮ > Refile... meeting my needs.

Would be nice to warn if considering emails with different email addresses and only in this case request in what folder to move them in.


chore: retire AREnableNewCollectionsRail feature flag

This PR resolves ONYX-332
Description
This PR retires AREnableNewCollectionsRail feature flag

PR Checklist

[x] I have tested my changes on iOS and Android.
[x] I hid my changes behind a [feature flag], or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an [app state migration], or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

#nochangelog

Warnings

:warning:

✅ No changelog changes

  Generated by :no_entry_sign: dangerJS against a3b8543ccbcdec4c2ea30c58eee91573b17c477d


Transaction tab filter by date

The same filter by date range for transaction give me inconsistent result from day to day.
yesterday screenshot

today screenshot

This is really weird. Do you have any more informations ? Did the network disconnect, did you do anything between these two filters ?

Did the network disconnect? -> I don't know
Did you do anything between these two filters? ->
Nothing special
Open cutecoin on 27-04-2015 play with the new version, add contact and use the filter by date. Take a screenshot. Close cutecoin.
The next day (28-04-215) open cutecoin use filter by date play a little bit with it using different date. My goal was to take a new screenshot to illustrate the diff in the fuzzy boundary bug.
The rest is history...

Ok. Weird.
Canno't reproduce at the moment, waiting for more informations...


Added new reviwer to the pt team members

New reviewer and admin for the Portuguese team.

/lgtm
/approve


Support smooth scroll.

The same SmoothScroller as other built-in LayoutMangers can be used.

This PR fixes #264

@yaraki  could you have a look?


Add support for hiding the tab close button

Summary of the Pull Request
Adds support for the tab.showCloseButton property to themes. This accepts three values:

"always" (default): The close button acts like it does today.
"hover": The close button is always visible on the active tab. On inactive tabs, the close button only appears on mouse over.
"never": The close button is never visible. You can't close the tab with middle-click, but you can still use keyboard shortcuts to close the tab.

References

See #3327
⚠ targets #13178 ⚠

PR Checklist

[x] Closes #3335
[x] I work here
[ ] Tests added/passed
[n/a] Requires documentation to be updated - YUP

Detailed Description of the Pull Request / Additional comments
See the following two properties in WInUI that we're leveraging here.

TabViewCloseButtonOverlayMode.OnPointerOver
TabViewItem.IsClosable

One is a tabview-level property, the other is a per-tab-item property, hence why this code is a little wacky.
Validation Steps Performed
gifs below

"tab.showCloseButton": "hover":

"tab.showCloseButton": "never":

@DHowett fyi the fix we discussed was in dab4501


[issue-51] Create blue section

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix #
Test URLs:

Before: https://main--jet2holidays-portugal--aemsites.aem.live/
After: https://issue-51--jet2holidays-portugal--aemsites.aem.page/drafts/rnecsoiu/bluesection

Converting to draft, as this isn't a block but a section styling.


api for checking on status of workflow - after submitting work for execution

Is there an api call to check execution status?

Yes, GET workflow/execution/asset/{AssetId} from the WORKFLOW_API_ENDPOINT.
See also https://github.com/awslabs/aws-media-insights/blob/master/IMPLEMENTATION_GUIDE.md#workflow-api


New resource: service.systemd

Title says it all.
I've been doing some trickery to get the state to play well with enabled. Since stopping a service might still mean you need to disable or enable it. Just look at
service.systemd.delete() {
  service.systemd.update
}

Another option would be, a bit more elegant but more verbose for the user, splitting this into two resources:

service.systemd
service.systemd_at_boot (or, preferably, some better name)

This would follow the flow of waffles.resource.process better. What's your take on this?

Whoa, nice!
If I understand correctly, you're talking about how to handle both "stopped" and "disabled" elegantly? That's a situation I put off with the sysv and upstart resources, so suggestions are definitely welcome.
For Puppet, services have two attributes for this:

enabled: determines if the service should start at boot.
ensure: determines if the service should be running or not.

I looked through the code (here and here. also systemd here) and didn't see anything that makes these two mutually exclusive, so it looks like they work together (which seems counter-intuitive, but not necessarily illogical).
My take would be to follow along these lines and treat them as two separate attributes of the resource, similar to checking for the uid of a user as  well as what groups they should belong to. So in the create process, enable the service at boot if the attribute is set to true and start the service if it should be in a "running" state.
There are resource states such as "running" and "stopped" specifically for services, but maybe it's best to do away with these and just keep the standard "present", "absent", "update" states.

Agreed.

There are resource states such as "running" and "stopped" specifically for services, but maybe it's best to do away with these and just keep the standard "present", "absent", "update" states.

I like the idea. What about something like this:
We've got the usual states: present, update, absent.
Services require two extra boolean options: --started and --enabled
A service is only present or absent if both options are set correct. ($started AND $enabled)
service --state present --started true --enabled false would only be true if running and disabled. Otherwise an update needs to be called.

So in the create process, enable the service at boot if the attribute is set to true and start the service if it should be in a "running" state.

But I would have to do the same in update and delete process. What I meant by "trickery". It's like this in the PR. Only I made enabled an optional parameter, but maybe that was a bad call.

maybe boot and now would be better names.
service --state present --now true --boot true
seems to read better and somehow feels less mathematical

(just a quick side note: I'm tied up with some offline stuff, but hope to dedicate time to this, as well as the other Waffles stuff, shortly :)

but maybe that was a bad call

Not at all. Embedding rm -rf / on state change would be a bad call. This is all just exploring and finding a good approach (which can definitely change post-commit).
I like the onboot parameter name. It's a lot better than Puppet's enabled.
Thinking this through more, we want to control two aspects of a service: whether it is currently running and whether it starts on boot. We've started thinking about three different parameters to control this: state, onboot, and now.
onboot makes sense since it clearly solves one aspect of the service to control.
state and now are leftover. If now is used, what good is state? If it's just for the existence of the init script, that could be handled by state = stopped and onboot = false.
Although I've been horrible at documenting resource design, state is the one common and required parameter of all resources. So I think I might have been premature in removing the running and stopped states.

Sorry for the delay.
I did a quick refactor of service.sysv and came to the same conclusion you found: for services, it makes sense to just point everything to the "update" function and make the system consistent with the requested state.
I also really like the boolean option check -- nice work on that. I thought maybe a new option type of waffles.options.create_boolean_option would be even better. I did a quick sketch of it here.
Neither of my refactors have been tested yet. These were more to review your changes interactively.
I've made one inline comment about grep. Also, a quick nit-pick: but could the order of the functions be reordered to create, update, delete? Even if update is going to be the one doing the work.
Very nice work!

I thought maybe a new option type of waffles.options.create_boolean_option would be even better.

It is. How about waffles.options.create_state_option?

I've made one inline comment about grep.

Fixed it some days ago in my local copy. Nice catch :smile:

Also, a quick nit-pick: but could the order of the functions be reordered to create, update, delete? Even if update is going to be the one doing the work.

No problem. Will fix it before asking to merge.

How do you feel about keeping state to store the now/present state of the service (either "stopped" or "running")? And onboot to determine whether the service should start at boot time or not?

I've given it some thought. In my (principled) view there are 2 main options of dealing with this problem:

Use these 4 states: present, absent, error and update. The user can only select present or absent. I understand your argument about the "useless" state but I like the consistency of state. It's less confusing. However this means we've got to implement some boolean logic in the module since: service --state absent --now true --onboot true means that at both times the service should be disabled. service --state stopped --onboot true seems a bit more open to interpretation.

Let each resource define it's own states. This means that we can have services like this service --state persistent which will make sure it's started and enabled on boot. We could name them: persistent, started, stopped, enabled, disabled and dead. But then it would also make more sense for the package modules to have a state called: latest. And just remove the option for version as it causes headaches for developers and won't see much use.

I know both options are not what's implemented right now, but I would argue to go for option 1. It's consistent with the design of Waffles having a central resource function which does the dispatching.
I must say I think option 2 is the best but to implement it properly requires (I think) a redesign of waffles and maybe that's not worth it. Maybe I'll go ahead and spend some time coding up how I would see it implemented.

These are really good thoughts and ideas. I spent last evening thinking about this. I apologize for the length.
Option 2
I've mentioned Terraform in a few other places, which I'm a huge fan of. Terraform manages "resources" through CRUD functions. Waffles does the same thing, but that's because it was a direct copy of Terraform's style of resource management.
I quickly ran into a problem, though: Terraform builds resources from scratch and therefore only needs to add. Waffles is run on systems with existing resources and might have to remove resources. So I needed some type of way of declaring the "state" of the resource. And so state was born.
Puppet does Option 2: it overloads the ensure parameter and each resource might have different ensure values. For example, the package resource can have its version specified in ensure.
I've trained a lot of in-house people on Puppet. It's a great tool, but it has some quirks. Overloading ensure is one of them. It might make sense from a developer/engineering point of view, but not to a new user. I've found that new users end up making explicit mental notes that ensure is quirky rather than viewing ensure as something that would contain an intuitive or natural value.
A dramatic example of ensure's quirkiness is the file resource:
file { '/tmp/foo.txt':
  ensure => present,
}

file { '/tmp':
  ensure => directory,  # wat?
}

IMO, ensure, state, etc have values that can answer a simple question:
Q: What's the state of the package?
A: It's present
Q: What's the version of the package?
A: 1.3
Not:
Q: What's the state of file /tmp?
A: It's a directory.
Because of that, Option 2 is something I'd prefer to not have in Waffles.
With that in mind, I realized last night that maybe state was the wrong choice of word. Maybe the current use of state is too shallow. state could be interpreted as "the entire state of the resource, including all attributes such as version, owner, group..."
Maybe a word like status would have been better? Or maybe the word "state" is kept, but its use is restricted only to the shallow existence of the resource (present, absent, upgrade, error). I'm leaning on the latter right now. I can see the same argument made for status being shallow, but maybe I've just been thinking about this for too long :)
Option 1
And with all of that in mind, let's jump to Option 1. This definitely seems like the way to go for now, though I'm inclined to drop the now parameter. To me, now feels too event-triggered. I'd be inclined to just use "present" and "absent" as the service's valid states.
With regard to the combinations of present, absent, onboot, and not onboot, it's really the --state absent --onboot true combination that is the odd one out. I'm inclined to omit the logic to prevent this odd state and leave it as the responsibility for the user. Or maybe they want that state.
(as a side note: --state present --onboot false is something we ($work) use in production. When a hypervisor server restarts, we do not want the hypervisor software to immediately start until we can vet the server is healthy).
Option 3
For the sake of discussing all options:
state has worked well for resources like files, packages, users, etc. state has only gotten weird with services. So what if we just nuke state in the service resources? Instead, have something like:
service.sysv --name memcached --running true --onboot true

Internally, the resource would translate running into present and stopped into absent so the resource dispatcher would work as-is. This would be a bit of a hack, but a hack that the user would never see -- only resource developers.
Boolean Option

waffles.options.create_state_option

Maybe I've just been seeing the word "state" too much. What are the benefits of create_state_option over create_boolean_option? That's not a rhetorical question, either - I'm genuinely asking :)
Thank you again for putting so much thought into this. I do feel bad that I've essentially shot down Option 2 and I'm sorry about that.
Thoughts? Did I miss anything? Sorry again for the lengthy response.

Maybe I've just been seeing the word "state" too much. What are the benefits of create_state_option over create_boolean_option?

In addition to boolean option. a boolean option for true and false. state for present or absent.

Because of that, Option 2 is something I'd prefer to not have in Waffles.

I see your point, but I do still think it's the best way to go. Just try to not overload state too much...

Option 1.  This definitely seems like the way to go for now

Agreed. Although I don't like dropping the --now parameter I see your point and if there's a good rule how the corner cases are handled I do think it's acceptable. It still seems the request in it's current form
Sorry for the delay!

In addition to boolean option. a boolean option for true and false. state for present or absent.

Ah! Yes, that sounds like a great idea to me.

I see your point, but I do still think it's the best way to go. Just try to not overload state too much...

Let's go with Option 1 and revisit this one down the road; maybe make an issue for it as a reminder.
So in summary, for now:

running and stopped will be removed from the dispatcher.

state on services will control if the service is running (present) or not (absent)

onboot will control if the service is set to start upon boot.

A create_bool_option will be added to better manage boolean options.

This can be done either in this PR or a new one. I can modify the existing resources to take advantage of this where applicable.

A create_state_option will be added to handle the valid values of state.

Ditto.

Sorry for the delay!

No problem at all!


Backend web fixes

I have more locally, but working on getting a sample running for it and that runs into all sorts of other things ... so I'll be pushing more here as I go.


Add September 2022 Patch Releases

Changes:

[data/release] update patch releases for September
releases: update the patch release page
releases: add patch release dates for Nov and Dec 2022

/cc @kubernetes/release-engineering
/priority critical-urgent

/lgtm


Getting a $(...).datetimepicker is not a function error

I am getting this error when attempting to load my page. I followed the directions in how to install and looked around at other issues that had this problem but their solutions did not fix mine.
This is my application.html.erb file includes
<head>
  <title>Loadtest</title>
  <%= stylesheet_link_tag    'application.css', media: 'all', 'data-turbolinks-track' => true %>
  <%= javascript_include_tag 'application.js', 'data-turbolinks-track' => true %>
  <%= csrf_meta_tags %>

  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width = device-width, initial-scale = 1">
  <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
  <link rel="stylesheet" type="text/css" href="http://fonts.googleapis.com/css?family=Ubuntu">

</head>

This is how I call the datetimepickers
                 $('.startDateTime').datetimepicker({
                     defaultDate: new Date(startDate),
                     useUtc:true,
                     sideBySide:true
                 });
                 $('.endDateTime').datetimepicker({
                     defaultDate: new Date(endDate),
                     useUtc:true,
                     sideBySide:true
                 });

This is my application.css I know that require bootstrap isnt here but I link it in the application.html.erb file
 *= require_self
 *= require bootstrap-datetimepicker
 *= require template
 * require_tree .

Finally this is my application.js file
require turbolinks
//= require jquery
//= require jquery_ujs
//= require moment
//= require bootstrap-datetimepicker
//= require template
//= require tree .

@TrevorS

@TrevorS I have virtually the same situation and haven't found a solution other than including files manually.

I will look into this. Sorry for the super late response.

Similar error on the docs

make startDateTime is put on the input not somewhere else like a div

If you have require jquery and require bootstrap-datetimepicker on your application.js (and assuming you have the right gems installed), you shouldn’t need these:
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>