keep_alive and OLLAMA_KEEP_ALIVE not effective

What is the issue?

After Ollama starts the qwen2-72b model, if there is no interaction for about 5 minutes, the graphics memory will be automatically released, causing the model port process to automatically exit.
I want the model to continue to exist, so I tried setting OLLAMA_KEEP_ALIVE=-1 in ollama.service, and also setting keep-alive=-1 when calling the interface. However, it does not take effect.  I also tried setting keep_alive=24h with ollama run qwen2:72b --keepalive 24h, but it didn't work either.
I used nvidia-smi to check and there were no running processes.
The graphics card is NVIDIA GeForce RTX 3090 24G * 8
CUDA Version: 12.5

OS
Linux
GPU
Nvidia
CPU
Intel
Ollama version
0.1.44

It works for me. Using Ubuntu Linux:
Created a /etc/systemd/system/ollama.service.d/override.conf:
Environment="OLLAMA_HOST=<IP_ADDRESS>"
Environment="OLLAMA_MODELS=/home/lstep/workspace/Apps/OLLAMA/models"
Environment="OLLAMA_ORIGINS=*"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KEEP_ALIVE=-1"
User=myusername
Group=myusername

Then systemctl daemon-reload and restart the ollama server with systemctl restart ollama. Try to load a model (for example ollama run deepseek-coder-v2:16b-lite-instruct-q8_0.
And to check it is loaded "forever", use ollama ps which should show UNTIL forever:
$ ollama ps
NAME                                            ID              SIZE    PROCESSOR       UNTIL
deepseek-coder-v2:16b-lite-instruct-q8_0        44250301ba51    19 GB   100% GPU        Forever

Thank you for your reply. When I run the model on my machine, I set keep_alive. If I call the model interface remotely without setting keep_alive, the previously set keep_alive will become invalid. Is this scene normal?

I just tested it and I'm sorry for the feedback that keep_alive doesn't work. ollama run qwen2:72b --keepalive 24h is effective, but as I said above, when I call the interface remotely, UNTIL will change from 24 hours from now to 4 minutes from now. Is this normal?

I just tested it and I'm sorry for the feedback that keep_alive doesn't work. ollama run qwen2:72b --keepalive 24h is effective, but as I said above, when I call the interface remotely, UNTIL will change from 24 hours from now to 4 minutes from now. Is this normal?

What do you mean by "calling remotely" exactly? Using the ollama-webui?

My ollama host is <IP_ADDRESS>, port is 11434, then I use a domain name to forward the request to port 11434 of the ollama server. When calling with the domain name, the above situation will occur.
My request method is openai api, and keep_alive is not set. I tested that if I use ollama run qwen2:72 --keepalive 24h, then after the call, UNTIL will become 4 minutes from now. When using OLLAM_KEEP_ALIVE=-1, the call is normal, and UNTIL is still forever

I think there is a bug here.  The intent on this settings is if unset by the client, any prior setting should be respected, but the following shows it reset to the default ~5 minute setting:
% ollama run llama3 --keepalive 1h hi
Hi! It's nice to meet you. Is there something I can help you with or would you like to chat?

% ollama ps
NAME         	ID          	SIZE  	PROCESSOR	UNTIL
llama3:latest	365c0bd3c000	6.7 GB	100% GPU 	59 minutes from now
daniel@tater07:~$ curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "hi",
  "stream": false
}' > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   592  100   532  100    60    787     88 --:--:-- --:--:-- --:--:--   877
% ollama ps
NAME         	ID          	SIZE  	PROCESSOR	UNTIL
llama3:latest	365c0bd3c000	6.7 GB	100% GPU 	4 minutes from now

How can I use it with serve as I currently do this:
OLLAMA_HOST=<IP_ADDRESS> ollama serve

@amirvenus something like would work:
OLLAMA_HOST=<IP_ADDRESS> OLLAM_KEEP_ALIVE=1h ollama serve

Why doesn' t it keep the model in memory with this serve ? From the screenshot you can see the OLLAMA_KEEP_ALIVE is set to -1.

@xgdgsc it works for me, so I'm not sure what's going on.
OLLAMA_KEEP_ALIVE=-1 ollama serve

% ollama run orca-mini hello
 Hello there! How can I assist you today?
% ollama ps
NAME                ID              SIZE      PROCESSOR    UNTIL
orca-mini:latest    2dbd9f439647    5.9 GB    100% GPU     Forever

My best guess is your client is setting it to something.  The server setting establishes the default, but a client can still request a different value.

Same issue here, I use curl to set a particular model's keep_alive to -1, but after a call from llamaindex (without setting any keep_alive), it is reset to expire after 5 mins.
But when call from Open WebUI, the FOREVER is retained, I am not sure if Open WebUI first query existing setting and apply to its call or what.
I am not using the env var OLLAMA_KEEP_ALIVE because I don't want FOREVER to apply to all models.

I have same problem. It used to work, but now the systemctl edit ollama and there:
Environment="OLLAMA_NUM_PARALLEL=6"
Environment="OLLAMA_MAX_LOADED_MODELS=3"
Environment="OLLAMA_MAX_QUEUE=256"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_HOST=<IP_ADDRESS>"
these are not effective.
Also they are in the override.conf file.
/etc/systemd/system/ollama.service.d/override.conf:1: Assignment outside of section. Ignoring.

I'm facing the 5 minutes timeout issue on an n8n setup serving my Supabase data ingest with Ollama running from Docker Desktop on an Apple Silicon Mac. I create a container from Ollama in Docker specifying variable OLLAMA_KEEP_ALIVE with value 24h in the Environment variables.
@dhiltgen You have an unfavorable typo above. Even when setting the correct variable "OLLAMA_KEEP_ALIVE", the text-ingest task to my database is aborted when exceeding 300 seconds. I couldn't set yet another keep_alive parameter in the run command for non-generative embedding models.
People, please mind to write keep_alive with the underscore as it seems to be relevant according to the documentation. It should deal briefly with Docker Desktop setups as you don't use curl commands there.
Why is there actually a default stop parameter when you can stop running a model anytime? When I run out of RAM, I know that I just have to quit Ollama, otherwise, I want to use the running model. A 5 minute timeout is unexpected and unhelpful in regular usage!
Maybe we should reopen this issue as the title applies also to other setups that are apparently still not working.


How to Visualize metrics using Grafana

Description
This tutorial will guide user how to visualize the metrics with Grafana.

Related issues/PRs

Todos
Information about charts and statistics to Grafana.

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

Deploy preview for calico ready!
Built with commit 210a28ff92ca15346941e2752adee49015ed6a64
https://deploy-preview-3472--calico.netlify.app

This pull request is not ready yet!
I have submitted this to ask people about ideas for the dashboard.

This is a followup on last pull request "create Prometheus metrics".
I believe it's ready for review.
Thanks in advance!
@hogepodge @lxpollitt @lwr20

Grafana dashboard is asking for a password on my preview, I'm not sure what the defaults are or what the configuration is set to.
The large dashboard object isn't the way to go. We need to have this be a file that's hosted on the docs site and can be curl'ed down.

Grafana dashboard is asking for a password on my preview, we should mention the default is admin/admin.
@hogepodge : there is an info box stating what are default username and password for Grafana. maybe it would be better to bump it up a bit so user can see it more easily?

https://github.com/projectcalico/calico/blame/5f73937c97f80015165627afdf692ac347db8358/maintenance/monitor/monitor-component-visual.md#L198-L201


Installing extension is invalid on chrome store

Unfortunely the package has been redone in such a way that for local development it works fine but installing it on the chrome store provides a package is invalid error. A screenshot of the error is listed below
This makes it so users cannot install the extension

Version: Version 74.0.3729.131 (Official Build) (64-bit)
Can I get conformation that is it the same on other peoples computers?

I believe the issue is we have been packaging and distributing our local debug build. We need to be running npm build and distribute that copy. Unfortunately npm build is currently broken

I fixed the build process. I am now going to package it and upload it again. Unfortunately we need to wait 3 to 5 business days for our update to get approved.

Just saw the approved extension and it was still broken. This time it was complaining it couldn't find a certain icon file (which was different from last time when it couldn't find the background.js file)
This led me to look into the zipping of the app. (You have to add a zipped version of the app to the chrome store).
When I unzipped the app, I realized that it didn't zip and of the folders....
note to self zip app.zip build/* is different from zip -r app.zip build/
Now I tested the zipped version by unzipping it and loading it and it worked. 3 to 5 more businesses days more now to wait.

Blessed be the name. Fixed it with the previous procedure.


uchime?

I can see that the pipeline runs usearch -uchime2 which generates uchime_out.txt, but it seems that this file is never used in any way.  In particular, exclude_chims.py says in its usage note that it excludes chimeras identified by uchime and vsearch, but it never accesses the uchime file.  Am I missing something?

You are correct - this uchime_out.txt file was created for testing purposes and never used fr filtering out any sequences because UCHIME tends to identify non-chimeric sequences as chimeras. I'll remove it from the code.

Ok, thanks for the explanation!


fix(knative): Integration in error state when knative SinkBinding mode=inclusion (1.10.x branch)

https://github.com/apache/camel-k/issues/4119
This is the backport of https://github.com/apache/camel-k/pull/4120 to 1.10.x branch
Release Note
knative trait sets `bindings.knative.dev/include=true` label to the namespace when there is a SinkBinding.

I think we should backport to release 1.12 as well.


TypeError: Object of type Shuffle is not JSON serializable

Hi! I have some model output I am trying to serve with xpublish (and xpublish-host and kerchunk). This model output has compression, and when I try to access it through the server I get the following error, though I am able to just open the model output fine (pointing to the kerchunked file):
TypeError: Object of type Shuffle is not JSON serializable

Indeed, a "Shuffle" object is in the metadata, for example here is what it shows in the xpublish code for an example variable:
'Uwind_eastward/.zarray': {'compressor': {'id': 'zlib', 'level': 3}, 'filters': [Shuffle(elementsize=4)], 'chunks': [1, 362, 794], 'dtype': '<f4', 'fill_value': 'NaN', 'order': 'C', 'shape': [1, 362, 794], 'zarr_format': 2}

I'm not sure why the compressor items remained a dictionary but the filter item was converted to an object, which is what seems to cause the issue. In the metadata coming out of kerchunk the filter item looked like
"filters":[{"elementsize":8,"id":"shuffle"}]

(I'm not sure if that change in "elementsize" is important as I go back and forth between two cases I'm testing, but I can track it down if it seems relevant.)
Does anyone follow this? I am not sure how to set up an actual working example for this workflow yet.

For reference, I decompressed the file with
ncks -L 0 nwgoa_1999-02-02.nc nwgoa_1999-02-02_decompressed.nc

and I do not hit any error anymore. Is there an issue with serving compressed model output?

Unfortunately of course that didn't work. Still not sure how to handle having filters necessary in the model output metadata.


Sparse Jaccard via python interface

I have been endeavoring to get more ANN libraries working with sparse Jaccard data in ann-benchmarks. There are surprisingly few libraries that support this. I was pleased to see that NGT does have support for sparse Jaccard, however (as far as I can tell) this is only accessible via the C++ API, and not via python (which ann-benchmarks would require).
Is it possible to add sparse Jaccard support to the python interface? If it is not too challenging I could attempt to provide a PR myself, but that would likely take some time, as I have little familiarity with the code.

Sorry for the late response.
Since NGT python bindings (ngtpy) support Jaccard, I think that you only have to update the ann-benchmarks ngt file. If the data format ann-benchmarks required is from different that of ngtpy, please let me know. I will help you.

Thanks for the reply. My concern was with the input data format for the Jaccard benchmark in ann-benchmarks. It is a (74962 x 27983) sparse matrix. This can be formatted as a boolean numpy array of shape (74962, 27983) which I believe can be fed to NGT's ngtpy jaccard. Given the very high dimensionality this will be slower than it could be. I will try this none the less and get back to you with preliminary results.
It seemed that the SparseJaccard comparator can act directly on the lists of indices of non-zero elements per row -- essentially directly on the sparse data format -- which will be much faster (other algorithms that support the Jaccard benchmark dataset in ann-benchmarks essentially do this). If possible I would prefer to be able to use this approach, as it would give a much fairer comparison in ann-benchmarks.

Although I surely implemented Jaccard into NGT, I had almost forgotten that. There are two types of Jaccard for dense and sparse on NGT. However, as you mentioned, it seems that sparse Jaccard is not available on ngtpy. I will try to implement sparse jaccard into ngtpy if it is easy.

Thanks. My initial efforts with the dense matrix are mostly just timing out on the runs, so I fear it is too expensive. Hopefully the sparse jaccard version is not too challenging to make available in ngtpy. It would certainly be useful for many people, but it is definitely a less used feature, so if it is too difficult to implement, please don't invest time in it. Thanks again for all your work.


Release trigger: "Outdated target/agent is online"

When releasing to a deployment group with one or more offline targets the deployment fails. There is no mechanism to redeploy to only the offline agents when they come online again. This should be possible to automate the same way as https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts#automatically-deploy-to-new-targets-in-a-deployment-group
For more details see this post:
https://stackoverflow.com/questions/52387916

This doesn't seem to be an agent issue.  If the agent is offline it can't absorb this issue.  It would need service work.  Please log an issue at developer community


Only plot 5 chans max for plot_epochs_image

See #2750
Currently, picks=None plots all channels. This is often overwhelmingly many. However, None is also the default. Now, None picks the first 5 channels only.
@jasmainak

Where else is this a problem?

Then it'll only plot these right?

oh yeah, true. LGTM. +1 for merge.

Please update what's new

@agramfort updated.

Hm ...  in the docstring, should it be

`None`

or

None

?

merged by rebase.
thx @jona-sassenhagen


FEAT: add welcome panels

This pull-request implements a set of panels in the landing page of our documentation.

This looks great! I've somehow forgotten how to look at the doc that was generated as part of the checks. Would someone remind me for future situations where this this would be helpful. @jorgepiloto I'm so excited to see this implemented! Great job!

@PipKat If you click on "Details" close to "CI / Documentation (pull_request)"

Then a new page will open. It is dedicated to this action/check. You can click on the top left corner on "Summary".

You will see the artifacts (both html and pdf) at the bottom of the page.

This is great @jorgepiloto .


Updating the recommendation for Executor GC heuristic in case of high executor GC

Description
Adding some more recommendations in ExecutorGC heuristic when high GC is witnessed in the application.

Looks good, thanks!

@edwinalu thanks for the review. There were some merge conflicts introduced to some efforts to merge this branch with master. I have resolved the conflicts, if feasible kindly have a look through the changes and let me know if I should merge the changes.


Automated cherry pick of #1524: fix azure reserved ips

Cherry pick of #1524 on release/2.11.0.
#1524: fix azure reserved ips

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.11.0@87192b0). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             release/2.11.0   #1525   +/-   ##
================================================
  Coverage                  ?   7.68%
================================================
  Files                     ?     411
  Lines                     ?   62834
  Branches                  ?       0
================================================
  Hits                      ?    4828
  Misses                    ?   57596
  Partials                  ?     410

Flag
Coverage Δ

#aFlag
7.68% <ø> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87192b0...ebbdcf4. Read the comment docs.

/lgtm
/approve


No matching version found for<EMAIL_ADDRESS>

i do
git clone https://github.com/Microsoft/monaco-editor
cd monaco-editor
npm install

and i got err:
npm ERR! code ETARGET
npm ERR! notarget No matching version found for<EMAIL_ADDRESS>npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'monaco-editor'
npm ERR! notarget

npm ERR!      /Users/chyroc/.npm/_logs/2018-08-13T10_11_36_343Z-debug.log

npm has been having some issues with folks unable to install recent published modules<EMAIL_ADDRESS>does exist
https://www.npmjs.com/package/monaco-languages


Can the wait step wait for fractions of a second or support decimal point?

Raising this from an user email for sharing, as other users might benefit from this question.

If I want to wait 500ms, should I use
wait 0.5
or
wait 500 ms
Does your wait support decimal point?

Yes, the time is in seconds and decimal point is supported, for example in the sample flow - https://github.com/kelaberetiv/TagUI/blob/master/src/samples/1_yahoo
You can write as wait 0.5 or wait 0.5 seconds or wait 0.5s etc, TagUI will understand accordingly.


I can't reboot into CFW from caffeine exploit via PegaScape on 4.0.0 in last two days with ~60 tries so far, even though I had it working for ~2 years before

I started to have following problem: after running pegascape via reswitched news applet and tapping that text control and waiting 5 full seconds to power tap instead of the boot I just get screen turned off just like normal power tap behaviour is. While I understand that this is normal and had problems like this before, it always went away after few tries. I am right now on about 60 tries on 2 different sd cards:

64 gb FAT32, EMUmmc RAW with 12 v. firmware on EMUmmc, hekate+atmo package from sdsetup site that was working before with no problem for two weeks
new 400gb FAT32 that has todays setup from sdsetup site and later changed to deepsea package.

On both I get same things. my screen goes black, I can normally unblock my screen and I have to close my switch by holding power button for ~15secs. rinse and repeat.
the thing I did that directly precedes my problems and looks like a cause of it was me trying to run faulty backup. What I did was that on my new sd card (I put new atmo and hekate combo. so far good. did emummc (2x29gb - the most I could allocate and created RAW 1 partition). after that emuMMC properly booted into copy of my SYS one. then I tried to use 'backup' I did from my failing old card, which wasnt created by hekate tools but by just copying emummc folder from sd to my harddrive. Then on new sd removed existing emummc folder and copied 'backuped' one. After I copied that folder, did migration of emummc via tools on hekate to let it find that existing partition which went correctly and tried to boot into it. which didn't work as I only got black screen, on which I waited like 15-20 minutes and just powered off. after that it booted me straight into SYSmmc without exploit and that is where I started to have problems with restarting into hekate from pegascape.
I am long time atmo+hekate user using caffeine via pegascape option running mostly via EMUmmc. My SYSMMC is still at 4.0.0 although have few nsp installed there. My DNS is working because I have pegascape working correctly and it is the same configuration for quite a while. I was working with PegaScape without a hitch for a ~2 years with several firmware updates on emuMMC up to 12v . so far.
While it looks like my fault, can I do something about this to make it bootable into cfw again? Even something unusual like upgrading via game card to 4.0.1 or 4.1.0 or am I out of luck and have to forget about using CFWs?
Any help or even confirmation of being OOL is super appreciated!

Ok, for some reason restoring old version (from kosmos days) of atmo + hekate (5.0) that was 2 years old did the trick and I managed to boot into cfw!


Browser refuses set cookie if the size of session is larger than max-size limit 4096 allowed by the browser

PR Checklist

[✅ ] Have you followed the guidelines in CONTRIBUTING.md?
[ ✅] Have you got 100% test coverage on new code?

Minimal Reproducible Example
Run it as is. Visit / which redirects to /health. The assertion error will be raised if the cookies were refused by the browser.
import secrets

import uvicorn
from starlette.middleware.sessions import SessionMiddleware
from starlette.status import HTTP_308_PERMANENT_REDIRECT
from starlite import DefineMiddleware, get, Request, Starlite
from starlite.datastructures import Redirect

# Set size of cookie value here, currently set to 1512 bytes to fail the
# assertion.
overflow = secrets.token_hex(1512)

@get(path='/', status_code=HTTP_308_PERMANENT_REDIRECT)
async def root(request: Request) -> Redirect:
    # Fill the session with large string to exceed the size of max-size limit
    # of the cookie in browser which is 4096 bytes.
    request.session['overflow'] = overflow
    return Redirect(path='/health')

@get(path='/health')
async def health_check(request: Request) -> str:
    # If the browser refused cookie in the response of the previous route
    # handler, this assertion will fail. This assertion will only pass if the
    # size of overflow is reduced to keep the session size less than 4096
    # bytes.
    assert tuple(request.session.items()) == (('overflow', overflow),)
    return 'health checked'

app = Starlite(route_handlers=[health_check, root],
               middleware=[DefineMiddleware(SessionMiddleware, secret_key=secrets.token_hex(16))],
               debug=True)

if __name__ == '__main__':
    uvicorn.run(app)

What's Working?
Response is registered by the browser but it refuses to set cookie in the response if it exceeds the max 4096 size limit.

Seems that Flask issues a warning for same.

The size of the session is not even close to 4096 bytes. Even by setting a string of size 1512 bytes in session, when the session middleware sets it as a cookie in the response header, the set cookie field size grows to 4100+ bytes.

The size of the session is not even close to 4096 bytes. Even by setting a string of size 1512 bytes in session, when the session middleware sets it as a cookie in the response header, the set cookie field size grows to 4100+ bytes which is then rejected by the browser. Run the example, you will see the failure.

I see, have you done any debugging you can share?

Are you sure it is not this? https://github.com/encode/starlette/issues/1259
Here's the response headers for the redirect that I get with your example:
HTTP/1.1 308 Permanent Redirect
date: Thu, 25 Aug 2022 21:18:32 GMT
server: uvicorn
content-length: 0
location: /health
set-cookie: session=eyJvdmVyZmxvdyI6ICIifQ==.YwfnKQ.N4BXrcIduaYMFgt69HmI2hVBZkE; path=/; Max-Age=1209600; httponly; samesite=lax

Yes, the problem is coming from here:
https://github.com/encode/starlette/blob/31164e346b9bd1ce17d968e1301c3bb2c23bb418/starlette/middleware/sessions.py#L62-L63
Based on the example, these are the changes in its size.
>>> len(json.dumps(scope["session"]))
3040
>>> len(b64encode(json.dumps(scope["session"]).encode("utf-8")))
4056
data = b64encode(json.dumps(scope["session"]).encode("utf-8"))
>>>len(self.signer.sign(data))
4091
>>> 4091 - 3040
1051  # Increased by

I reproduced the same situation in flask with the same key value pairs in flask.session.
https://github.com/pallets/flask/blob/36af821edf741562cdcb6c60d63f23fa9a1d8776/src/flask/sessions.py#L409
>>> len(json.dumps(flask.session))
3040
>>> len(self.get_signing_serializer(app).dumps(dict(session)))
2362

It's starlette.middlware.SessionMiddleware's bug, not at starlite.
--
The response headers will show the set cookie field but the browser will silently ignore it for exceeding the max-size limit. If the assertion is failing that means the browser has ignored the cookie.

Even by setting a string of size 1512 bytes in session

>>> import secrets
>>> overflow = secrets.token_hex(1512)
>>> len(overflow.encode("utf-8"))
3024
>>>

So the original value you are setting in the session is twice as large as you say it is in the example.
But I get the same error if I set session["overflow"] = "a" which I think is to do with the starlette error I linked above.

So what should we be doing here in your opinion?

As SessionMiddleware is not the extension of Starlite, I'll upstream the fix.


Flathub release temporarily suspended

Due to issues with the pre-build process on Flathub at the moment releases on this platform are temporarily suspended.
The problem specifically is related to the generation of the generated-source.json file that is used to cache dependencies and run npm install --offline in the flatpak build process.
To generate this file you have to use flatpak-node-generator, which so far I have managed to use with some workarounds.
The problem now is that due to the updating of various Antares dependencies I can no longer get flatpak-node-generator to work.
Unfortunately, this software was last updated two years ago, and workarounds are no longer enough.
I will try to find a solution, but in the meantime it is impossible for me to release new flatpak versions.
If anyone has experience with this, and has time and willingness to help me out, I would appreciate it 💖

Hello @Fabio286 ,
Apparently, the issue with v3 has been resolved, flatpak-node-generator is not showing version errors.
However, in the tests I performed, I needed to uncomment the run: rm package-lock.json command from the create-generated-sources.yml file, to force an update of package-lock.json
I hope this helps, thanks.

Hello @Fabio286,
I think I found a solution, it's not necessary to use flatpak-node-generator.
PR: https://github.com/flathub/it.fabiodistasio.AntaresSQL/pull/13
I hope this helps.

Thanks to @carvalhods the Flathub has been restored!
@all-contributors add @carvalhods for package.

Thanks to @carvalhods the Flathub has been restored!
@all-contributors add @carvalhods for platform.


Fluentd retains excessive amounts of memory after handling traffic peaks

When setting up a simple, 2-part Fluentd (TCP -> forwarder) -> (forwardee -> disk configuration) and giving it 5 million JSON objects to process all at once, resident-set memory consumption jumps from an inital 30MB to between 200-450MB, and does not come back down after computation is complete.  This is observed using version 2.3.5-1.el7 of the TD Agent RPM package running on CentOS 7.  (The version of Fluentd in that package is 0.12.36.)
Steps to reproduce:
# define and start a simple TCP forwarder
$ cat > test-in.conf <<'EOF'
<source>
  @type  tcp
  tag    testing
  format json
  port   10130
</source>

<match **>
  @type forward
  require_ack_response true
  flush_interval 10s

  <server>
    host <IP_ADDRESS>
    port 10131
  </server>
</match>
EOF
$ td-agent --no-supervisor -c test-in.conf &

# define and start a simple forwadee that logs to disk
$ cat > test-out.conf <<'EOF'
<source>
   @type forward
   port  10131
</source>

<match **>
  @type              file
  format             json
  path               "/tmp/fluent-test"
  flush_interval     10s
  buffer_chunk_limit 16m
  compress           gzip
</match>
EOF
$ td-agent --no-supervisor -c test-out.conf &

# observe initial memory consumption
$ ps -o pid,vsz,rss,cmd | grep '[t]d-'
 4254<PHONE_NUMBER>4 /opt/td-agent/embedded/bin/ruby /sbin/td-agent --no-supervisor -c test-out.conf
 4259<PHONE_NUMBER>2 /opt/td-agent/embedded/bin/ruby /sbin/td-agent --no-supervisor -c test-in.conf

# pump a bunch of data through it
$ seq 1 5000000 | sed 's/.*/{"num": &, "filler": "this is filler text to make the event larger"}/' > /dev/tcp/localhost/10130 &

# wait for all data to flush to disk and CPU to return to idle
$ watch -n1 'ls -lt /tmp/fluent-test.* | head -5'

# observe final memory consumption
$ ps -o pid,rss,cmd | grep '[t]d-'
 4338<PHONE_NUMBER>72 /opt/td-agent/embedded/bin/ruby /sbin/td-agent --no-supervisor -c test-out.conf
 4343 1023868 461616 /opt/td-agent/embedded/bin/ruby /sbin/td-agent --no-supervisor -c test-in.conf

As you can see from the RSS numbers, each td-agent process started out around 30MB and they ended at ~290MB and ~460MB, respectively.  Neither process will release that memory if you wait a while.  (In the real-world staging system we initially discovered this on, memory consumption of the test-out.conf-equivalent configuration reached over 3GB, and the test-in.conf-equivalent was a Fluent Bit instance exhibiting a recently-fixed duplication bug.)
Reviewing a Fluentd-related kubernetes issue during our own diagnostics, we noticed that the behavior we observed seemed similar to the Fluentd behavior described there when built without jemalloc.  This led us to check if the td-agent binary we were using was in fact linked with jemalloc.  According to the FAQ, jemalloc is used when building the Treasure Data RPMs, and though we found jemalloc libraries installed on the system, we couldn't find any existence of jemalloc in the process running in memory.  Specifically, we tried the following things:
# given a td-agent process with PID 4343...

# no jemalloc shared library mentioned in the memory mapping
$ pmap 4343 | grep jemalloc
# ... so it doesn't look like it's dynamically linked

# grab the entire memory space and search it for references to jemalloc
$ gcore 4343
$ strings core.4343 | fgrep jemalloc

# ... but if it were statically linked, you'd expect to find some of these strings
$ strings /opt/td-agent/embedded/lib/libjemalloc.a | fgrep jemalloc

In short, this leads us to wonder... are the binaries invoked by td-agent actually linked with jemalloc?  If they are not, is the old memory fragmentation problem that jemalloc solved what we are observing here?  (And if they aren't, am I raising this issue in the wrong place, and if so where should I raise it?)

Did you try this setting?
https://docs.fluentd.org/v0.12/articles/performance-tuning#reduce-memory-usage

Yes, even with constraints on the oldobject factor, the problem persists.  In fact, even with more draconion restrictions on the garbage collector the problem persists, e.g.:
$ RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 \
  RUBY_GC_HEAP_GROWTH_FACTOR=1.05         \
  RUBY_GC_MALLOC_LIMIT_MAX=16777216       \
  RUBY_GC_OLDMALLOC_LIMIT_MAX=16777216    \
  td-agent --no-supervisor -c test.conf &

Coupling that output td-agent process with an input of ...
$ td-agent-bit -i tcp://<IP_ADDRESS>:10130 -t test -o forward://<IP_ADDRESS>:10131 &
$ seq 1 3000 | sed 's/.*/{"num": &, "filler": "this is filler text to make the event larger"}/' > /dev/tcp/localhost/10130 &

... sees the td-agent resident-set memory jump to around ~650MB and hold there, even long after all data has been processed.  (Note: I'm using Fluent Bit 0.11.15 here, which has a nasty data duplication problem which is useful in stressing td-agent with input in this test.)
Honestly, it seems like the real issue here is the Ruby 2.1 garbage collector; as far as I can tell, it never releases memory to the OS that it allocates, thereby damning any process that has an even momentary need for a large quantity of memory to hold on to that memory for the rest of its lifetime.  (Please correct me if I'm wrong here.)
If the Ruby gc issue cannot be fixed by some form of additional configuration, then perhaps Fluentd could use some type of backpressure mechanism to avoid ingesting input faster than it can truly process it and avoid accumulating large queues in memory?

+1

it's a stale issue. I'm closing. if you have any problem, updating fluentd and ruby might help it.

I have re-tested this issue using the latest td-agent 3.5.1 RPM package for RHEL 7.  That package includes Fluentd 1.7.4 and embeds Ruby 2.4.0.  The problem still remains exactly as it was originally reported.
Nothing has been fixed, despite the intervening 2 years.  The issue should be re-opened.
P.S. You as the Fluentd developers and project administrators collectively have the right to run your project however you see fit.  However, making claims of Fluentd's suitability for production use and its performance on your website are not consistent with this kind of callous disregard for the validity of an easily reproducible issue that has serious impacts on Fluentd's production suitability.  I gave very simple steps to reproduce this issue, and it only took me a few minutes to download the latest TD-Agent RPM, install it, and copy and paste the commands from my original report to see that the outcome remained the same.  You could have trivially done the same.  The fact that you could not be bothered to do so but instead chose to try to bury and ignore this problem speaks volumes, especially as another user indicated as recently as June 27 that it is still affecting people.  If you truly feel that is the appropriate response, then you should also remove the false claims of performance and production-suitability from the Fluentd website.

The problem still remains exactly as it was originally reported.

Thank you re-test it. then I should re-open this issue.

I'm having the same issue using td-agent 3.8.0 RPM package for Amazon2. That package includes Fluentd 1.11.1 and embeds Ruby 2.4.10. Any news here? Work in progress? Our fluentD is in production with 4 aggregators ingesting at the same time into ElasticSearch. So far stable but the memory is normalizing always close to 100% leaving just 250-300 MB free which I think doesn't have any sense... I don't know what else to test... I'm checking this since weeks, changing many configurations and different versions without any clue. Even I tried to adjust the GC variables as @joell did in the past but the behaviour is the same.
You can see here how a new aggregator doesn't release memory until be close to 100%. The unique thing we improve adding a new aggregator is reduce a little bit the CPU usage.

Our buffer size (I think is in sync with this memory issue) as I read in other threads with memory buffer the behaviour is different...

Total network I/O (when the peacks are related also with the amount of memory needed)

Please help, thanks in advance.

@cede87: As I noted in my comment on April 9, 2017 the underlying issue here appears to be in the default Ruby garbage collector and memory allocator.
The Fluentd developers could directly avoid this by applying a backpressure mechanism or spooling incoming data to disk instead of in memory.
Alternatively, the Ruby memory allocation issue can be indirectly avoided by replacing or manipulating the Ruby memory allocator.  One method is to replace the allocator with jemalloc (though different versions are reported to be more effective than others); this approach was documented as being done by the Fluentd devs, but as I noted in the original issue text it doesn't look like jemalloc is actually used in the build that produces the RPM.  Another method would be to try to manipulate the allocator's behavior through things like the MALLOC_ARENA_MAX environment variable.
A summary of the underlying problem and some of the techniques you might be able to try -- including going so far as patching the Ruby garbage collector yourself -- can be found in this article.
Best of luck.

@joell thanks for the quick reply. I did the same checks you did in the past and I could verify that we are using jemalloc with our RPM installation. So if I'm not mistaken we are not able to use MALLOC_ARENA_MAX environment variable... even so we are suffering the same problems. Any suggestion?
[root@x ~]# pmap 4057 | grep jemalloc
00007fba1a234000    292K r-x-- libjemalloc.so.2
00007fba1a27d000   2048K ----- libjemalloc.so.2
00007fba1a47d000      8K r---- libjemalloc.so.2
00007fba1a47f000      4K rw--- libjemalloc.so.2
Thanks!

@cede87: The presence of libjemalloc.so.2 indicates you might be running jemalloc 5 instead of jemalloc 3.  During a discussion about making jemalloc the default allocator for Ruby, it was noted:

jemalloc 3.6 is slow but space efficient. jemalloc 5.1 is faster but almost as bad with space as untuned glibc.

Glancing at package content listings online, it looks like you would see libjemalloc.so.1 if you were using the jemalloc 3 series, which demonstrates the small heap sizes.  You might consider rebuilding Fluentd with jemalloc 3 instead.

@joell many many thanks for your suggestions. I was able to change jemalloc version from 5.x to 3.6.0 using td-agent 3.8.0 in one server (just to test). Notice the difference.

So I can confirm the following:

td-agent 3.8.0 is using jemalloc 5.x
jemalloc 5.x is faster but almost as bad with space as untuned glibc.
downgrade jemalloc to 3.6.0 solved the memory issues.

I think FluentD developers should take a look on this.
Thanks again,

Would appreciate a solution, the memory usage has peaked and isn't falling down at all.

@cede87: The presence of libjemalloc.so.2 indicates you might be running jemalloc 5 instead of jemalloc 3. During a discussion about making jemalloc the default allocator for Ruby, it was noted:

jemalloc 3.6 is slow but space efficient. jemalloc 5.1 is faster but almost as bad with space as untuned glibc.

Glancing at package content listings online, it looks like you would see libjemalloc.so.1 if you were using the jemalloc 3 series, which demonstrates the small heap sizes. You might consider rebuilding Fluentd with jemalloc 3 instead.

How do I rebuild Fluentd with jemalloc 3? Thanks

Hi @Adhira-Deogade please follow my notes to be able to downgrade jemalloc version.

yum groupinstall "Development Tools"
wget https://github.com/jemalloc/jemalloc/releases/download/3.6.0/jemalloc-3.6.0.tar.bz2
bunzip2 jemalloc-3.6.0.tar.bz2
tar xvf jemalloc-3.6.0.tar
cd jemalloc-3.6.0
./configure
make
make install
create symbolic links: ln -s file link

Note: First delete the files in: cd /opt/td-agent/embedded/lib
rm libjemalloc.a libjemalloc_pic.a libjemalloc.so.2 libjemalloc.so
ln -s /usr/local/lib/libjemalloc.a /opt/td-agent/embedded/lib/libjemalloc.a
ln -s /usr/local/lib/libjemalloc_pic.a /opt/td-agent/embedded/lib/libjemalloc.pic.a
ln -s /usr/local/lib/libjemalloc.so.1 /opt/td-agent/embedded/lib/libjemalloc.so.2
ln -s libjemalloc.so.2 libjemalloc.so
Note: If you do ls you should see the these symbolic links:
/opt/td-agent/embedded/lib/libjemalloc.a -> /usr/local/lib/libjemalloc.a
/opt/td-agent/embedded/lib/libjemalloc.pic.a -> /usr/local/lib/libjemalloc_pic.a
/opt/td-agent/embedded/lib/libjemalloc.so -> libjemalloc.so.2
/opt/td-agent/embedded/lib/libjemalloc.so.2 -> /usr/local/lib/libjemalloc.so.1

Restart td-agent
Verify
ps -aux | grep td-agent -> Copy the ID of one process
pmap ID | grep jemalloc -> Must be libjemalloc.so.1
[root@ip-10-103-149-184 lib]# pmap 28103 | grep jemalloc
00007ff35f4f9000    276K r-x-- libjemalloc.scd o.1
00007ff35f53e000   2048K ----- libjemalloc.so.1
00007ff35f73e000      8K r---- libjemalloc.so.1
00007ff35f740000      4K rw--- libjemalloc.so.1

I hope it helps you,
Daniel

This issue should remain open until it is resolved.  It has continued to affect people since it was reported in 2017, and the current best workarounds are laborious and invasive.

This issue should remain open until it is resolved.  It has continued to affect people since it was reported in 2017, and the current best workarounds are laborious and invasive.

I repeat:
This issue should remain open until it is resolved. It has continued to affect people since it was reported in 2017, and the current best workarounds are laborious and invasive.

This issue is still affecting us. The unique solution is changing the jemalloc version. Currently is what we are doing even in production environments. Which is not the best solution.

Thank you for notifying it.

# no jemalloc shared library mentioned in the memory mapping
$ pmap 4343 | grep jemalloc
# ... so it doesn't look like it's dynamically linked

td-agent loads jemalloc by LD_PRELOAD and this environment variable is set only by init script or systemd unit file.
Launching td-agent command manually doesn't set it, this is the why you didn't see jemalloc in pmap.
https://github.com/treasure-data/omnibus-td-agent/blob/b0a421e49b123c0c77ae897fbd1f782bb226a9ed/templates/etc/init.d/rpm/td-agent#L186-L188
if [ -f "${TD_AGENT_HOME}/embedded/lib/libjemalloc.so" ]; then
  export LD_PRELOAD="${TD_AGENT_HOME}/embedded/lib/libjemalloc.so"
fi

https://github.com/treasure-data/omnibus-td-agent/blob/master/templates/etc/systemd/td-agent.service.erb#L11
Environment=LD_PRELOAD=<%= install_path %>/embedded/lib/libjemalloc.so

td-agent 3.8.0 is using jemalloc 5.x

td-agent 3 uses jemalloc 4.5.0, not 5.x
https://github.com/treasure-data/omnibus-td-agent/blob/b0a421e49b123c0c77ae897fbd1f782bb226a9ed/config/projects/td-agent3.rb#L24
override :jemalloc, :version => '4.5.0'

Hmm, I confirmed that jemalloc-3.6.0 consumes fewer memory than jemalloc 5.2.1 (td-agent 4.1.0's default) in this case:
jemalloc 3.6.0:
  27672<PHONE_NUMBER>48 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent -c test-in.conf --under-supervisor
  27685<PHONE_NUMBER>24 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent -c test-out.conf --under-supervisor

jemalloc 5.2.1:
  28148<PHONE_NUMBER>48 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent -c test-in.conf --under-supervisor
  28161<PHONE_NUMBER>96 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/td-agent -c test-out.conf --under-supervisor

You can switch jemalloc version easily by LD_PRELOAD:
$ wget https://github.com/jemalloc/jemalloc/releases/download/3.6.0/jemalloc-3.6.0.tar.bz2
$ tar xvf jemalloc-3.6.0.tar.bz2
$ cd jemalloc-3.6.0
$ ./configure --prefix=/opt/jemalloc-3.6.0
$ make
$ sudo make install
$ LD_PRELOAD=/opt/jemalloc-3.6.0/lib/libjemalloc.so td-agent

But I'm not sure it's always efficient and worth to replace.

Thanks for your opinion. I've opened an issue for td-agent: https://github.com/fluent-plugins-nursery/td-agent-builder/issues/305

I can confirm that the issue is still very much present and valid on 4.2.0 td-agent / Ubuntu Bionic. We're using it to report on Artifactory stats as part of Jfrog monitoring platform so configuration is pretty much default.
Fluentd's memory usage is creeping up a lot so as a workaround we've applied cgroup limits to it (50% of 128 RAM utilization of the host which is). It took only a few hours after restarting td-agent service for it to be OOM killed:
Nov 02 18:55:57 host.example.com kernel: ruby invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Nov 02 18:55:57 host.example.com kernel: CPU: 38 PID: 16039 Comm: ruby Tainted: P           O      5.4.0-77-generic #86~18.04.1-Ubuntu
Nov 02 18:55:57 host.example.com kernel: Hardware name: Dell Inc. PowerEdge R530/0CN7X8, BIOS 2.4.2 01/09/2017
Nov 02 18:55:57 host.example.com kernel: Call Trace:
Nov 02 18:55:57 host.example.com kernel:  dump_stack+0x6d/0x8b
Nov 02 18:55:57 host.example.com kernel:  dump_header+0x4f/0x200
Nov 02 18:55:57 host.example.com kernel:  oom_kill_process+0xe6/0x120
Nov 02 18:55:57 host.example.com kernel:  out_of_memory+0x109/0x510
Nov 02 18:55:57 host.example.com kernel:  mem_cgroup_out_of_memory+0xbb/0xd0
Nov 02 18:55:57 host.example.com kernel:  try_charge+0x79a/0x7d0
Nov 02 18:55:57 host.example.com kernel:  ? __alloc_pages_nodemask+0x153/0x320
Nov 02 18:55:57 host.example.com kernel:  mem_cgroup_try_charge+0x75/0x190
Nov 02 18:55:57 host.example.com kernel:  mem_cgroup_try_charge_delay+0x22/0x50
Nov 02 18:55:57 host.example.com kernel:  __handle_mm_fault+0x8d5/0x1270
Nov 02 18:55:57 host.example.com kernel:  ? __switch_to_asm+0x40/0x70
Nov 02 18:55:57 host.example.com kernel:  handle_mm_fault+0xcb/0x210
Nov 02 18:55:57 host.example.com kernel:  __do_page_fault+0x2a1/0x4d0
Nov 02 18:55:57 host.example.com kernel:  do_page_fault+0x2c/0xe0
Nov 02 18:55:57 host.example.com kernel:  page_fault+0x34/0x40
Nov 02 18:55:57 host.example.com kernel: RIP: 0033:0x7f1f16426b23
Nov 02 18:55:57 host.example.com kernel: Code: fe 6f 64 16 e0 c5 fe 6f 6c 16 c0 c5 fe 6f 74 16 a0 c5 fe 6f 7c 16 80 c5 fe 7f 07 c5 fe 7f 4f 20 c5 fe 7f 57 40 c5 fe 7f 5f 60 <c5> fe 7f 64 17 e0 c5
Nov 02 18:55:57 host.example.com kernel: RSP: 002b:00007f0bafa270c8 EFLAGS: 00010202
Nov 02 18:55:57 host.example.com kernel: RAX: 00007f098866df61 RBX: 00007f1ed0df8f90 RCX:<PHONE_NUMBER>26df61
Nov 02 18:55:57 host.example.com kernel: RDX:<PHONE_NUMBER>0000fc RSI: 00007f1ee3a3e800 RDI: 00007f098866df61
Nov 02 18:55:57 host.example.com kernel: RBP:<PHONE_NUMBER>1c2454 R08: 00007f0984400000 R09: ffffffffffffffff
Nov 02 18:55:57 host.example.com kernel: R10: 00007f1f14c5e100 R11: 00007f09885b4192 R12: 00007f1ee3a3e800
Nov 02 18:55:57 host.example.com kernel: R13:<PHONE_NUMBER>0000fc R14:<PHONE_NUMBER>000001 R15:<PHONE_NUMBER>26e05d
Nov 02 18:55:57 host.example.com kernel: memory: usage 65894512kB, limit 65894512kB, failcnt 29724
Nov 02 18:55:57 host.example.com kernel: memory+swap: usage 0kB, limit<PHONE_NUMBER>740988kB, failcnt 0
Nov 02 18:55:57 host.example.com kernel: kmem: usage 138544kB, limit<PHONE_NUMBER>740988kB, failcnt 0
Nov 02 18:55:57 host.example.com kernel: Memory cgroup stats for /system.slice/td-agent.service:
Nov 02 18:55:57 host.example.com kernel: anon<PHONE_NUMBER>8
                                                  file 0
                                                  kernel_stack 626688
                                                  slab 3518464
                                                  sock 0
                                                  shmem 0
                                                  file_mapped 0
                                                  file_dirty 0
                                                  file_writeback 811008
                                                  anon_thp 0
                                                  inactive_anon<PHONE_NUMBER>
                                                  active_anon<PHONE_NUMBER>6
                                                  inactive_file 0
                                                  active_file 0
                                                  unevictable 0
                                                  slab_reclaimable 557056
                                                  slab_unreclaimable 2961408
                                                  pgfault 378416214
                                                  pgmajfault 0
                                                  workingset_refault 0
                                                  workingset_activate 0
                                                  workingset_nodereclaim 0
                                                  pgrefill 946474
                                                  pgscan 243287
                                                  pgsteal 91570
                                                  pgactivate 60357
                                                  pgdeactivate 946408
                                                  pglazyfree 0
                                                  pglazyfreed 0
                                                  thp_fault_alloc 0
                                                  thp_collapse_alloc 0
Nov 02 18:55:57 host.example.com kernel: Tasks state (memory values in pages):
Nov 02 18:55:57 host.example.com kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 02 18:55:57 host.example.com kernel: [  15976]     0 15976    59526     3287   462848     6900             0 fluentd
Nov 02 18:55:57 host.example.com kernel: [  15983]     0 15983 22606458 16437845 134709248    84397             0 ruby
Nov 02 18:55:57 host.example.com kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0-1,oom_memcg=/system.slice/td-agent.service,task_memcg=/system.slice/td-agent.
Nov 02 18:55:57 host.example.com kernel: Memory cgroup out of memory: Killed process 15983 (ruby) total-vm:90425832kB, anon-rss:65744072kB, file-rss:7308kB, shmem-rss:0kB, UID:0 pgtables:131552kB
Nov 02 18:55:59 host.example.com kernel: oom_reaper: reaped process 15983 (ruby), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

+1

@ashie What is the latest status on this issue? This seems like a HUGE flaw in fluentd. The issue you linked https://github.com/fluent/fluent-package-builder/issues/305 was closed without action taken.
We are facing fluentd OOM issues in production. Please advise.

Same issue with td-agent 4.4.2 ( fluentd 1.15.3 ) on ubi image ( rhel8 ).
I cannot believe this is not fixed - surely not production ready :(

I can confirm this behavior on td-agent 4.4.1 fluentd 1.13.3 (https://github.com/fluent/fluentd/commit/c32842297ed2c306f1b841a8f6e55bdd0f1cb27f) as well.
The memory seems is not dynamically deallocated. Initial startup without traffic, memory consumption is low. After a large traffic then turn to no traffic, memory is still staying at the highest point.

This issue may relate to https://github.com/fluent/fluentd/issues/4174


system tests - adding listener overrides

what
Now that we have an abstraction for listeners, people should be allowed to pass in certain overrides for their listeners. This allows a user to override both client & interbroker listeners with a dictionary of configs.
why
There may be certain listener configs a user would want to override. Currently, to do so, they would either have to completely override all listener configs OR try to shoehorn an override based on the name of the security-protocol or an internal constant. This makes things a bit cleaner such that we handle the prefixing and simply add on some additional configs.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Let's run this with the branch builder before merging.

test results: (2 failures, 423 passing)
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2019-06-21--001.1561194134--brianbushree--listener-overrides--dc70db4/report.html

@rajinisivaram I made the requested changes. Could you take another look?
Also I am running the tests again now

out of the two test failures mentioned above, looks like one of them is also failing on trunk at the moment (http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2019-06-25--001.1561541534--apache--trunk--14d8549/report.html)

https://github.com/apache/kafka/pull/7000 is meant to fix the upgrade test. @brianbushree are all other tests passing with this PR?

http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2019-06-26--001.1561614455--brianbushree--listener-overrides--ddb024d/report.html
@ijuma all but one. however kafkatest.tests.connect.connect_rest_test seems irrelevant to this change. will double check though

http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2019-06-27--001.1561654600--brianbushree--listener-overrides--ddb024d/report.html
kafkatest.tests.connect.connect_rest_test passed


Add <template functional> syntax to section about functional components

Functional components are discussed here and according to this announcement we can now do
<template functional> instead of functional: true.
I think the docs should include this. I can make this PR later this week if the team is happy with this idea.

A small note regarding SFCs on that page linking to the vue-loader docs would be welcome. 🙂

Thanks Chris, I will do it in the next day or two!

I looked into this, it feels really out of place on this page. I feel like it fits better in [this section[(https://vuejs.org/v2/guide/render-function.html#Functional-Components) that talks about how to make a file functional.
Is there any reason you felt it fits better in the SFC area @chrisvfritz ? The reason I think it goes best in the section section is because when I was making some functional components, I was referring to that area mostly, as opposed to the SFC page or vue-loader documentation, which I rarely use.

Just a misunderstanding. 🙂  we're on the same page actually (no pun intended). When I said "that page" I was referring to the one you linked to in your original post, which is the render functions page.

Sure! Great to hear. I'll make the PR this weekend.
Thanks for your quick response, no wonder Vue has the best documentation of any framework I've used.

It's done, closing


Wording

Can some clarity be provided on the 'settings that can be Microsoft Managed' section?
The following table lists settings that can be set to Microsoft managed and whether it is enabled or disabled - does this mean the table below are the defaults, the setting that Microsoft managed applies or something else?

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bfbc994a-9eaa-6eb3-7907-9baf2e8e3b3a
Version Independent ID: 42a31644-c046-6e0e-09bc-8fbd6280641b
Content: Use Microsoft managed settings for the Authentication Methods Policy - Azure Active Directory
Content Source: articles/active-directory/authentication/how-to-mfa-microsoft-managed.md
Service: active-directory
Sub-service: authentication
GitHub Login: @mjsantani
Microsoft Alias: justinha

@danwdavey
Thanks for your feedback! We will investigate and update as appropriate.

@danwdavey
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.

@danwdavey Thanks Dan! Is this any better?
The following table list each setting that can be set to Microsoft managed and whether that setting is enabled or disabled by default.
The point is MS is going to change the value at some time from disabled to enabled.

Hi, that would be great. Customer is just unsure (and so was I) regarding its default status or what the table was referring to.

@danwdavey Thanks for providing feedback that helps improve our documentation. This issue has been resolved and we are closing the issue. #please-close


feat: add token update

Description

Adds tokens.update method for tokens client.

Testing required outside of automated testing?

[x] Not Applicable

Screenshots (if appropriate):

[x] Not Applicable

Rollback / Rollforward Procedure

[x] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 1.38.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


check if top window can be accessed before getting data from it

Type of change

Bugfix

Description of change
check if top window can be accessed before getting data from it

contact email of the adapter’s maintainer<EMAIL_ADDRESS>[x] official adapter submission

Link to Prebid.js PR https://github.com/prebid/Prebid.js/pull/7841


support x509

ISSUE TYPE

Feature Idea

ROLE VERSION
1.0.0.

CONFIGURATION
N/A
OS / ENVIRONMENT
N/A
SUMMARY
the role should support x509 certificates for TLS.
STEPS TO REPRODUCE
N/A
EXPECTED RESULTS
user can manage x509 certificates for TLS.
ACTUAL RESULTS
the role does not implement certificate management.

fixed in https://github.com/trombik/ansible-role-haproxy/commit/0e6a8b640010311b3f6b2e86bfa515b25b0033fe


Using subprocesses involves the global event loop

Hello,
I'm the maintainer of pytest-asyncio. Pytest-asyncio allows users to block access to the global event loop, and pass the event loop explicitly; this is considered by some in the community as a best-practices approach.
However I've just received a bug report this approach doesn't work with subprocess (on Unix at least).
>       raise NotImplementedError
E       NotImplementedError

/usr/lib/python3.5/asyncio/events.py:538: NotImplementedError

Two comments:

can't the event_loop instance passed in be used for this functionality, instead of the global event loop policy? If an instance of the loop is provided, in my opinion asyncio shouldn't care about any globals.
event_loop_policy.get_child_watcher() is undocumented as far as I can see.

I ran into this issue at some point, and here is the explanation:

Only the main thread can register signals, so the child watcher has to be bound to the main loop (the loop running in the main thread).
It is fine for other loops to watch subprocesses, as long as the main loop is running (so the main loop can notify the other ones using call_soon_threadsafe).

This was my conclusion:

From the design point of view, it is up to the policy to decide how the watcher is supposed to run. So if one chooses to go explicit (i.e. set_event_loop(None)), the loop should also be attached explicitly:

asyncio.set_event_loop(None)
loop = asyncio.new_event_loop()
asyncio.get_child_watcher().attach_loop(loop)

More information in issue #390 and PR #391.

Should we close this one?

Sure, I'll go through the linked issues and see what do to in pytest-asyncio.


SIMSBIOHUB-13: Lock Submitted Data and Files

Overview
Links to Jira tickets

https://apps.nrs.gov.bc.ca/int/jira/browse/SIMSBIOHUB-13

Description of relevant changes

{List all relevant changes, in particular anything that will help the reviewers test/verify this PR}

PR Checklist
A list of items that are good to consider when making any changes.
Note: this list is not exhaustive, and not all items are always applicable.
Code

[ ] New files/classes/functions have appropriately descriptive names and comment blocks to describe their use/behaviour
[ ] I have avoided duplicating code when possible, moving re-usable pieces into functions
[ ] I have avoided hard-coding values where possible and moved any re-usable constants to a constants file
[ ] My code is as flat as possible (avoids deeply nested if/else blocks, promise chains, etc)
[ ] My code changes account for null/undefined values and handle errors appropriately
[ ] My code uses types/interfaces to help describe values/parameters/etc, help ensure type safety, and improve readability

Style

[ ] My code follows the established style conventions
[ ] My code uses native material-ui components/icons/conventions when possible

Documentation

[ ] I have commented my code sufficiently, such that an unfamiliar developer could understand my code
[ ] I have added/updated README's and related documentation, as needed

Tests

[ ] I have added/updated unit tests for any code I've added/updated
[ ] I have added/updated the Postman requests/tests to account for any API endpoints I've added/updated

Linting/Formatting

[ ] I have run the linter and fixed any issues, as needed
See the lint commands in package.json
[ ] I have run the formatter and fixed any issues, as needed
See the format commands in package.json

SonarCloud

[ ] I have addressed all SonarCloud Bugs, Vulnerabilities, Security Hotspots, and Code Smells

@KjartanE handful of bugs/cod smells to address

@KjartanE One issue I found:
Setup:

I have a project with 1 survey (no other attachments)
The survey has an observation uploaded (no other attachments).
I submit the observation to biohub, successfully.

Result:

The survey page banner/chip shows it was submitted (green)
The project page banner/chip still shows the survey is unsubmitted (grey)
The project list page shows the project is unsubmitted (grey)


about auto increment version code

I want an auto-increment number as version code, It should automatically increment by 1 for each release build, what are the best practice for app-versioning?  write a wrapper ? or override overrideVersionCode function?
Thanks

app-versioning uses git info to generate versionCode.
Auto incrementing the versionCode for each release means the source of the versionCode has nothing to do with git, so you'll need to keep track of this versionCode yourself and decide when to increment it, where to store it etc, and inject it as environment variable and read it from overrideVersionCode.

Thanks for reply


How to post CSV file data into the InfluxDB server in same time series as CSV File's time series?

I would like to post data from a CSV file to InfluxDB file and the Time column of InfluxDB should match with CSV file Time column. I meant InfluxDB time series should same as CSV file's Time series.
My main motive is InfluxDB should store the data of CSV file with respect CSV file's time series.
The CSV file is shown below
Time                Energy   Power
12-26-2017 7:34:27  103691   24.838
12-26-2017 7:35:28  103693   19.525

I have used this spice code to convert CSV file's time to epoch to send the InfluxDB server
t = "2017-12-26 07:34:27"
lctime = (int(time.mktime(time.strptime(t,"%Y-%m-%d %H:%M:%S")))) *<PHONE_NUMBER> #in nano second`

and I have posted this data to InfluxDB server in JSON format.
client.write_points([
         {"measurement": "Meter1",
          "tags":{"host": "localPC",
                  "Region": "Northam"},
          "Time":lctime,
          "fields":{"Energy": e1,
                    "Power": p1,
                    }
               }
          ]

There was no issue with posting these data to InfluxDB server but when I went to check whether the correct data was written or not into Influxdb using this query in database panel.
Select * From Meter1
The influxDB showed me this "2018-01-04T05:43:41.580065574Z" time but it should have shown me  "2017-12-26 07:34:27" as same as CSV file's time.
Please tell me in which method I can match the time of CSV and InfluxDB?
Please tell me in which method I can do this thing which I want?

Use pandas pd.to_datetime to handle the date conversion


v0.1.6+5 breaks Xcode compile

Due to the recent changes in the FlutterUnityWidgetPlugin.m file, the line [GetAppController() setUnityMessageHandler: ^(const char* message) is breaking the compiling. I needed to fix it by manually download v0.1.6+4 again from the flutter page, because whatever version number I add in the pubspec.yaml file, it forces the v0.1.6+5 version.
I would appreciate if someone could look into what the problem is, as I have not much experience in Objective C and Swift.
Thanks!

@Klingens13 the XCodePostBuild.cs file was also changed in that release, have you updated it in your unity project and rebuild the project? The Unity-iPhone/Classes/UnityAppController.h should contain this on line 78 after you rebuild it:
@property (nonatomic, copy)    void(^unityMessageHandler)(const char* message);


RDK plugin does not present next aperture when 'response_ends_trial: false' and a response is made

(also reproducible by setting  'response_ends_trial: false' at line 42 in examples/jspych-RDK.html)

UPDATE - RDK plugin will keep moving along the timeline of apertures when response_end_trial: false and a response is made IF both 'choices : []' and 'correct_choice : []' for all coherence_directions in RDK timeline BUT response is logged as -1 regardless actual response - An issue with how data.correct is created?

Thanks for alerting me to the bug. There was a logical error in the after_response function, so it should be fixed by placing the clearTimeout inside the if(trial.response_ends_trial) statement.
Here is the function to be replaced (line 567):
`               //Function to record the first response by the subject
function after_response(info) {
		//If the response has not been recorded, record it
		if (response.key == -1) {
			response = info; //Replace the response object created above
		}

		//If the parameter is set such that the response ends the trial, then kill the timeout and end the trial
		if (trial.response_ends_trial) {
			window.clearTimeout(timeoutID);
			end_trial();
		}

	}; //End of after_response`

PS. I'll get back to you with the other issue you emailed me about.

Closing this as PR will handle it.


Added maven URL to project gradle

gradle 3.x.x alpha requires the google maven url for
tools like FindBugs to function properly

I merged #127 instead.


Add quick links to install wallets from the modal if they are not installed

Done, now when clicking on a non-installed wallet it will open the official site for such wallet


Duplicated values when reading from an ADC in a loop.

Hi!
Thanks for the library. I think I've found a bug, but it seems pretty strange. I'm running on an rp2040 and communicating with an ADS1115 using rustc 1.82.0-nightly. ChannelA0 is connected to 3.3V and ChannelA1 is connected to 0v. I'm using a local copy of latest version of the library from git.
It seems like reading from the ADC in a loop is causing the last value read in the loop to be duplicated. Here's the simplest reproduction I can make.
let mut count = 0;
let mut adc = ads1x1x::Ads1x1x::new_ads1115(i2c, ads1x1x::SlaveAddr::default());
loop {
    info!("Reading pot_0 ADC values");
    let pot_0 = nb::block!(adc.read(ads1x1x::channel::SingleA0)).unwrap();
    info!("0: {}", pot_0);
    info!("Reading pot_1 ADC values");
    let pot_1 = nb::block!(adc.read(ads1x1x::channel::SingleA1)).unwrap();
    info!("1: {}", pot_1);
    // Loop twice then stop
    if count == 1 {
        loop {}
    }
    count += 1;
    info!("====================loop====================");
}

Here's the output. The first time around the loop everything is fine, but the second time around the loop it seems to only be repeating the last read value from channel SingleA1.
INFO  Reading pot_0 ADC values
INFO  0: 32767
INFO  Reading pot_1 ADC values
INFO  1: 1
INFO  ====================loop====================
INFO  Reading pot_0 ADC values
INFO  0: 1
INFO  Reading pot_1 ADC values
INFO  1: 1

If the loop is allowed to run continuously, changing the voltage of the channel SingleA1 has no effect on the output. The first value that was read get's repeated.
Additionally, it seems to only happen when reading the channels in a loop, reading the same channels twice in a row is fine, and only breaks on the second time around the loop.
let mut count = 0;
let mut adc = ads1x1x::Ads1x1x::new_ads1115(i2c, ads1x1x::SlaveAddr::default());
loop {

    info!("Reading pot_0 ADC values");
    let pot_0 = nb::block!(adc.read(ads1x1x::channel::SingleA0)).unwrap();
    info!("0: {}", pot_0);
    info!("Reading pot_1 ADC values");
    let pot_1 = nb::block!(adc.read(ads1x1x::channel::SingleA1)).unwrap();
    info!("1: {}", pot_1);

    info!("Reading pot_0 ADC values");
    let pot_0 = nb::block!(adc.read(ads1x1x::channel::SingleA0)).unwrap();
    info!("0: {}", pot_0);
    info!("Reading pot_1 ADC values");
    let pot_1 = nb::block!(adc.read(ads1x1x::channel::SingleA1)).unwrap();
    info!("1: {}", pot_1);

    // Loop twice then stop
    if count == 1 {
        loop {}
    }
    count += 1;
    info!("====================loop====================");
}

INFO  Program start
INFO  Reading pot_0 ADC values
INFO  0: 32767
INFO  Reading pot_1 ADC values
INFO  1: 2
INFO  Reading pot_0 ADC values
INFO  0: 32767
INFO  Reading pot_1 ADC values
INFO  1: 1
INFO  ====================loop====================
INFO  Reading pot_0 ADC values
INFO  0: 1
INFO  Reading pot_1 ADC values
INFO  1: 1
INFO  Reading pot_0 ADC values
INFO  0: 1
INFO  Reading pot_1 ADC values
INFO  1: 1

I'll try and reproduce this on another platform to rule out hardware issues when I get some time.

The bug seems to be correlated with reading from the I2C bus too quickly in the loop or via nb::block!. The following worked as a workaround by adding a 100us delay to each iteration of the loop.
    let mut adc = ads1x1x::Ads1x1x::new_ads1115(i2c, ads1x1x::SlaveAddr::default());
    let mut channel_index = 0;
    loop {
        match channel_index {
            0 => match adc.read(ads1x1x::channel::SingleA0) {
                Ok(pot_0) => {
                    info!("0: {}", pot_0);
                    channel_index = 1;
                }
                Err(nb::Error::WouldBlock) => {}
                Err(_) => {
                    info!("Error");
                }
            },
            1 => match adc.read(ads1x1x::channel::SingleA1) {
                Ok(pot_1) => {
                    info!("1: {}", pot_1);
                    channel_index = 0;
                }
                Err(nb::Error::WouldBlock) => {}
                Err(_) => {
                    info!("Error");
                }
            },
            _ => {
                info!("Error");
            }
        }
        delay.delay_us(100u32);
    }


Application hangs after device.free_memory call

Short info header:

GFX version: 0.6.0
OS: elementary OS 5.1.5 Hera
GPU: NVIDIA GeForce GTX 2080 Ti.

Hey! I'm trying to learn gfx-hal the hard way using these tutorials. However, the question is not strictly related to tutorials, because I'm using my own implementation of some things, so the question is more general about vertex buffers.
The problem is in the cleaning up code:
device.free_memory(vertex_buffer_memory);
device.destroy_buffer(vertex_buffer);

When free_memory is called, everything hangs and I'm forced to kill the process. Note: if I draw only one frame and then instantly close application, it works okay.
What could cause free_memory to hang? Am I missing some kind of synchronization?

Getting Vulkan validation layer output would help here. Please install the VVL and run your app in debug, with the logging enabled (i.e. env_logger::init).

Thanks!
The output from the VVL is like that:
[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkFreeMemory-memory-00677<PHONE_NUMBER>)] : Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x50000000005[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkFreeMemory-memory-00677)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyBuffer-buffer-00922 (-464217071)] : Validation Error: [ VUID-vkDestroyBuffer-buffer-00922 ] Object 0: handle = 0x40000000004, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe4549c11 | Cannot free VkBuffer 0x40000000004[] that is in use by a command buffer. The Vulkan spec states: All submitted commands that refer to buffer, either directly or via a VkBufferView, must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyBuffer-buffer-00922)
    object info: (type: BUFFER, hndl:<PHONE_NUMBER>108)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroySemaphore-semaphore-01137 (-1588160456)] : Validation Error: [ VUID-vkDestroySemaphore-semaphore-01137 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa1569838 | Cannot call vkDestroySemaphore on VkSemaphore 0xc000000000c[] that is currently in use by a command buffer. The Vulkan spec states: All submitted batches that refer to semaphore must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroySemaphore-semaphore-01137)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyFence-fence-01120<PHONE_NUMBER>)] : Validation Error: [ VUID-vkDestroyFence-fence-01120 ] Object 0: handle = 0xb000000000b, type = VK_OBJECT_TYPE_FENCE; | MessageID = 0x5d296248 | VkFence 0xb000000000b[] is in use. The Vulkan spec states: All queue submission commands that refer to fence must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyFence-fence-01120)
    object info: (type: FENCE, hndl:<PHONE_NUMBER>5547)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyPipeline-pipeline-00765<PHONE_NUMBER>)] : Validation Error: [ VUID-vkDestroyPipeline-pipeline-00765 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bdce5fd | Cannot call vkDestroyPipeline on VkPipeline 0xa000000000a[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to pipeline must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyPipeline-pipeline-00765)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyRenderPass-renderPass-00873<PHONE_NUMBER>)] : Validation Error: [ VUID-vkDestroyRenderPass-renderPass-00873 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x473619ad | Cannot call vkDestroyRenderPass on VkRenderPass 0x60000000006[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to renderPass must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyRenderPass-renderPass-00873)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyCommandPool-commandPool-00041 (-1387836198)] : Validation Error: [ VUID-vkDestroyCommandPool-commandPool-00041 ] Object 0: handle = 0x55fa13b5ff60, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0xad474cda | Attempt to destroy command pool with VkCommandBuffer 0x55fa13b5ff60[] which is in use. The Vulkan spec states: All VkCommandBuffer objects allocated from commandPool must not be in the pending state (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyCommandPool-commandPool-00041)
    object info: (type: COMMAND_BUFFER, hndl:<PHONE_NUMBER>9456)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyImageView-imageView-01026<PHONE_NUMBER>)] : Validation Error: [ VUID-vkDestroyImageView-imageView-01026 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x63ac21f0 | Cannot call vkDestroyImageView on VkImageView 0x150000000015[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to imageView must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyImageView-imageView-01026)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

[2020-08-28T08:47:07Z ERROR gfx_backend_vulkan]
    VALIDATION [VUID-vkDestroyFramebuffer-framebuffer-00892 (-617577710)] : Validation Error: [ VUID-vkDestroyFramebuffer-framebuffer-00892 ] Object 0: handle = 0x55fa13901750, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xdb308312 | Cannot call vkDestroyFramebuffer on VkFramebuffer 0x3a500000003a5[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to framebuffer must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyFramebuffer-framebuffer-00892)
    object info: (type: DEVICE, hndl:<PHONE_NUMBER>5216)

So, it seems like the drop that is called after the winit event loop is done is accessing memory that is still in use by the GPU.
I have an event loop:
events.run(move |e, _, flow| {
    match e {
        Event::WindowEvent { event, .. } => match event {
            WindowEvent::CloseRequested => {
                *flow = ControlFlow::Exit;
                return;
            },
            WindowEvent::Resized(new_size) => {
                graphics.resize(new_size.width, new_size.height);
            }
            WindowEvent::ScaleFactorChanged { new_inner_size, .. } => {
                graphics.resize(new_inner_size.width, new_inner_size.height);
            }
            _ => (),
        },
        Event::MainEventsCleared => {
            window.request_redraw();
        }
        Event::RedrawRequested(_) => {
            graphics.begin_render_pass([1.0, 0.5, 0.3, 0.1]);
            graphics.draw_triangle();
            graphics.submit_render_pass();
        }
        _ => (),
    }
});

Where graphics is an object that contains all gfx-hal resources. It has a drop method:
fn drop(&mut self) {
    unsafe {
        let vertex_memory = ManuallyDrop::take(&mut self.vertex_memory);
        let vertex_buffer = ManuallyDrop::take(&mut self.vertex_buffer);

        let rendering_semaphore = ManuallyDrop::take(&mut self.rendering_semaphore);
        let submission_fence = ManuallyDrop::take(&mut self.submission_fence);

        let graphics_pipeline = ManuallyDrop::take(&mut self.graphics_pipeline);
        let graphics_pipeline_layout = ManuallyDrop::take(&mut self.graphics_pipeline_layout);
        let render_pass = ManuallyDrop::take(&mut self.render_pass);
        let command_pool = ManuallyDrop::take(&mut self.command_pool);

        let mut surface = ManuallyDrop::take(&mut self.surface);

        self.device.free_memory(vertex_memory);
        self.device.destroy_buffer(vertex_buffer);

        self.device.destroy_semaphore(rendering_semaphore);
        self.device.destroy_fence(submission_fence);

        self.device.destroy_graphics_pipeline(graphics_pipeline);
        self.device.destroy_pipeline_layout(graphics_pipeline_layout);
        self.device.destroy_render_pass(render_pass);
        self.device.destroy_command_pool(command_pool);

        surface.unconfigure_swapchain(&self.device);
        self.instance.destroy_surface(surface);
    }
}

Is there any way to wait until GPU is done with the resources?

Just note: tried the same code on Mac with Metal, worked well.

If you want something that works out of the box, you'd need wgpu-rs.
Programming gfx-hal directly is unsafe, hard, and requires a good understanding on how GPUs work. It's not something where you are encouraged to go with trial and error (since everything is unsafe, and you can easily hang the PC).
More to the question, device.wait_idle() would wait for GPU operations to complete.

Great, thanks!


Support to convert between k8s podtemplate and Jenkins

Codecov Report

Merging #52 (61ee0d8) into main (c930b66) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #52   +/-   ##
=======================================
  Coverage   71.91%   71.91%
=======================================
  Files          36       36
  Lines        2065     2065
=======================================
  Hits         1485     1485
  Misses        513      513
  Partials       67       67

Flag
Coverage Δ

unittests
71.91% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c930b66...61ee0d8. Read the comment docs.


/var/vcap/data on diego-cell needs to be a separate volume with quota support

The problem
garden and grootfs both assume that they together have exclusive control over the disk space mounted at /var/vcap/data. The property grootfs.reserved_space_for_other_jobs_in_mb (default: 15360) can be used to carve out space for other jobs.
Currently an emptyDir volume is mounted to various parts of the filesystem. I don't understand what purpose this serves, as the disk space will still be allocated on the local node storage. Yes, it survives a pod restart, but I'm not even sure this is desirable.
The real issue though is that now grootfs is going to allocate sparse files as backing store for the privileged and unprivileged stores that claim all the size except for 15360MB of the local node storage, and formats those sparse files with XFS.
There is more to this, including the way garden calculates available disk space for apps, and how it does garbage collection for the layer cache. That code assumes that /var/vcap/data/garden/depot is on the same disk as /var/vcap/data and the grootfs stores.
Potential fix
Use a PVC and mount it to /var/vcap/data. This still requires that the storageclass for this PVC implements quotas correctly. The NFS test setup used during development typically does not, so would still allocate huge stores.
This is the mechanism SCF uses to move the grootfs away from the local node disk. For some reason we found it necessary to wipe this PVC every time the container restarts. I don't know why this was necessary, but the current emptyDir volume used by kubecf would keep data between restarts and only be wiped when the pod is deleted.
Alternative
Assuming that we can require worker nodes to have sufficient disk space for the grootfs, we might be able to set a limit.disk for the pod to get a quota on the pod filesystem. That way we could get rid of the volume mount completely.
This would also provide faster storage (just like the current emptyDir setup) in case the node uses e.g. SSD drives.

For illustration here you can see the dh -h output from inside the diego-cell, and the /var/vcap/data/grootfs/store/unprivileged claims to have 419G available where as the disk itself only has 360G of free space:
/:/var/vcap/jobs/garden# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         436G   77G  360G  18% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/sda4       436G   77G  360G  18% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs            16G   12K   16G   1% /run/secrets/kubernetes.io/serviceaccount
/dev/loop0      421G  1.6G  419G   1% /var/vcap/data/grootfs/store/unprivileged
/dev/loop1      421G  462M  420G   1% /var/vcap/data/grootfs/store/privileged

@fargozhu We need to determine if this is a blocker for KubeCF 1.0.

Docs for the grootfs disk usage: https://github.com/cloudfoundry/garden-runc-release/blob/91f02154969fc94546d766573c047ad2eb7dff2f/docs/understanding_grootfs_store_disk_usage.md.

@jandubois it seems that using PVCs is making cells unstable. Specifically because a lot of storage classes are based on NFS, some with hard mounts. These cause garden and the cell to missbehave (write/close calls hanging indefinitely).
I think garden was designed with local storage in mind for garden store.

@jandubois  with the disk capacity set as below, the cell reports the expected capacity.
Based on your original comment, it also sounds like we should calculate grootfs.reserved_space_for_other_jobs_in_mb at runtime. I'm working on a patch. Did you use some automated testing to confirm the cell was consuming more than it should?
properties:
  diego-cell:
    rep:
      diego:
        executor:
          disk_capacity_mb: 40960

@viovanov The disk_capacity_mb is a rep property, which doesn't limit how much space is allocated by grootfs init-store. That size will be calculated based on the size of the filesystem mounted at /var/vcap/data unless you explicitly override it with a --store-size-bytes commandline option to grootfs init-store.
Since this is only allocating a sparse file, this may still be fine as long as you don't use too much space inside. I think part of the problem before was that the sparse file was allocated with a size larger than the partition size because a different PVC was mounted under /var/vcap/data/grootfs that was smaller than the filesystem at /var/vcap/data.
But really, you are now relying on details of how the sparse file logic is implemented by the PVC and how the xfs filesystem is managing disk blocks inside the sparse file, which sounds to me like: "Let's hope this all works out!".
E.g. I don't see any reason why a filesystem should try to re-use deleted blocks as soon as possible and not use up all available disk blocks before re-using. The whole grootfs logic relies on the filesystem reusing disk blocks as much as possible in order to avoid committing more blocks from the sparse filesystem. Except XFS doesn't know that this is a concern.
So in summary, I think overallocating the sparse file is asking for problems, even if it seems to work properly in our specific testing.
I really wish I would never have to think about this again, it is such a mess!

#1264 contains a patch to calculate sparse file size


OM 2.0: Decision around type and unit labels in Text format

There is a proposal that makes it possible for the unit and type of the metric being part of the metric/metric family identification. This issue is about considering this an option and tracking the proposal for the spec changes.
There are a few key discussion points, if we would like to accept: https://github.com/prometheus/proposals/pull/39

Should we flatten the format (use type and unit labels instead of TYPE and UNIT); mix approach or not allow those labels?
How to solve the problem of the OM 1.0 not allowing multiple metric families (essentially names)?

Sounds like this generally requires redefinition of metric family (to include type and unit).

I feel like it's a mistake to add units and type as labels in the exposition format. Since this is repeated for the whole metric family, there's no need for it.
I do see the problem that Prometheus can scrape two different targets with same metric family name but different type/unit, but this is a Prometheus problem, not the exposition 🤔

I do see the problem that Prometheus can scrape two different targets with same metric family name but different type/unit, but this is a Prometheus problem, not the exposition 🤔

This is not the biggest problem as usually target is identifiable, so those are two different series. The problem is with ONE target with e.g.
TYPE metric1 counter
UNIT metric1 seconds
HELP metric1 ...
metric1{...}
TYPE metric1 gauge
UNIT metric1 milliseconds
HELP metric1 ...
metric1{...}

It feels we could consider allowing that from 2.0, which requires storage to handle things a bit differently (e.g Prometheus putting those into label). Thanks of major version change, it might a quite clean transition. One thing to check would be: what storage/scraper won't handle it for some reasons (e.g. the internal model does not allow this). Is this fine as those storages could simply ignore second series as the worst case or is there a bigger consequence of this?

I do think the TYPE and UNIT comments are more readable than __type__ and __unit__ in all labels, which is important for the text format. If we ever considered adding other metadata (e.g. alias, or something similar in concept to OTel's instrumentation scope), adding all of them as labels would make the text format less useable over time.

From the WG group:

What would surprise users less?
Context: Different unit and types for a the metric family


Title: lightBulb Effect project

Is your feature request related to a problem? Please describe.
Hey I madelightBulb Effect project Which Is Very Good!
Describe the solution you'd like.
This Project is very Helpfull,
I Make This Project With the help of HTML, CSS & JavaScript.
Describe alternatives you've considered.
The CSS & JavaScript Code are Available in very well mannered! & Easy to understand!

Add any other context or screenshots about the feature request here.
No response

It's like a simple toggle button which is not a quite good project.

It's like a simple toggle button which is not a quite good project.

ok np :)


Fix failing tests after change in Bookmarks

When adding the consecutive_answers to the bookmarks we can run into exceptions if the bookmark as not been scheduled yet. In these cases, the value should be set to Null and this should fix the tests failing in the API.

We are not guaranteed that a bookmark has been trained and has consecutive answers. In this case, we should return null.

Not sure why tests are failing?


Add a eventlet specific selector in case eventlet has monkey patched select

Please see https://github.com/andymccurdy/redis-py/issues/1136

After testing our eventlet patch (see https://github.com/eventlet/eventlet/issues/55) the last few days, I don't think we should apply this to redis-py. The correct thing to do is to fix eventlet. (gevent btw works fine with redis-py and no patching is needed).

Interesting. Were you still seeing errors after applying your patch?


Warn if there is a missing parent element

Log a message if:

there isn't a "wrapper" element (i.e. the component has more than 1 root element)

<!-- component-one.html -->
<input unicorn:model="first_name"></input>
<input unicorn:model="last_name"></input>

the root element doesn't have children

<!-- component-two.html -->
<input unicorn:model="first_name"></input>

I should also update the docs to make it more clear, but a "wrapper" element is required for the DOM diffing algorithm to work properly. These checks should hopefully help catch invalid component templates earlier.

@cyface How does this more explicit warning in the documentation with examples look?

Yeah, I think something like that could work. I'll continue to ruminate.

this is super helpful

After re-reading the docs a few times, I realized that it doesn't explicitly make it clear that components are a Python view + HTML template. So, I explicitly split out a new views page and re-arranged a lot of the content. The overall goal is to reduce the head scratching moments that new users run into and generally clarify the concepts behind Unicorn.

line 154 in docs/source/views.md:
I noticed that you went to the effort of removing a "just" statement in templates.md. It might be worthwhile to do so here as well:
"Inheriting from unicorn.components.UnicornField is a quick way to serialize a custom class as it calls self.__dict__ under the hood."

I noticed that you went to the effort of removing a "just" statement in templates.md. It might be worthwhile to do so here as well

Thanks for pointing that out @kevinbowen777. I've been trying to remove words like "just", "simple", "quick", etc. although I'm sure they are still more in there. 😄
I also just pushed some more changes mostly around rearranging the sidebar navigation to (hopefully) make things more clear.

Typo in early part of the docs “forcing you to re-building your application”

Typo in early part of the docs

Thanks! 🚀

I'm going to merge this in, but any other feedback on doc changes would still be appreciated. 👍


exit game contract should be retrieved from contracts

Plasma contracts allow us to dynamically retrieve exitGames(uint256) for specific payment type. We should do that so that we don't have to specify exit game contract address. For reference check Itest.StandardExitClient.get_exit_game_contract_address/1

Already implemented in https://github.com/omgnetwork/elixir-omg/blob/master/apps/omg_eth/lib/omg_eth/release_tasks/set_contract.ex#L97-L108


Error rate vs. genotyping error rate

I have a slight concern/misunderstanding regarding error rate.
We have two separate error rates in pandora.

-e,--error_rate  which defaults to 0.11 for nanopore and 0.001 for Illumina.
--genotyping-error-rate (which was hidden prior to my CLI PR) which is set to 0.01.

The genotyping error rate is used for computing likelihood in the VCF whereas the first error rate is used for alignment-related tasks, estimating parameters for the kmer graph model, and de novo discovery.
Why are these error rates different? From Rachel's thesis (p65 - 5.2.2 Genotyping)

We model the mean k-mer coverage on the true allele as Poisson distributed with rate equal to the expected depth of coverage d. To account for noise, we assume that coverage on alternative alleles arises as a result of an error process with rate e.

Shouldn't this error rate also be a function of the sequencing technology used?
I.e. from p45 4.3.2 Quasi-mapping to the Index (Setting the minimum cluster size)

To account for sequencing errors, we estimate that if errors occur independently as a Poisson distribution with rate e along a sequence, the probability that a k-mer has been sequenced with no errors is exp(−ke).

I remember being confused by these two different error rates when refactoring some stuff, but I did not go deep in the investigation as you did, as I was worried about assuring the refactoring would produce the same results.
Just for complementing, this is how we compute the log likelihood:

As Michael previously stated, this e should be the error rate, which is currently always fixed at 0.01. From my understanding, this is a bug, and it is easy to fix.
If we assume ONT error rate is around 0.11, ln(0.11) = -2.20. But we have been used the fixed ln(0.01) = -4.60 . So, we have been giving more weight than we should to c_b (the total mean coverage of all other alleles), effectively pushing likelihoods down. But this has a larger effect on the likelihood of lowly covered alleles than on the highly covered alleles (which are possibly the ones chosen), so I think for ONT data this is leading to overestimated genotype confidences. I think for illumina the effect is opposite, we are underestimating genotype confidences. I could be wrong on these...
Anyway, it is hard to predict what is the impact of this fix on the paper results. My blind guess is that it won't change much, but it is just really a blind guess. It is indeed easy to fix, but it would require us to reproduce the results for the paper once again. This fix should take no more than 10 minutes (I guess), and putting everything to run should be 5 minutes, but then we have to wait for the results to be produced.
Maybe we should confirm this is indeed a bug first...

We definitely don't touch this lightly. This is exactly where the difference between our mental models and reality is most clear. Definitely do not fix this Leandro until you have weeks available for debugging. I propose we add this (leandros bug) to the list of things to return to.
Returning to Michaels issue:
I'd like to understand both the intention and the current implementation and how it has been evaluated. But I want to be free of the current super heavyweight pipelines first and evaluate on something small and controlled post paper. I've chatted a bit to Rachel this week, maybe we can have a brief 4 way call next week

There is always the possibility of evaluating this with my TB dataset instead? Much faster turn around time, and I am doing the evaluation anyway...

Interesting. I think this must be a bug. I made the commit here https://github.com/rmcolq/pandora/commit/062e4ebe84424619241c59a14582536000422700#diff-23632e736a3562808e4b2d5bf5dc59c7 to add the genotyping_error_rate parameter. It was just a constant number before this. However I did not carry it through any further. I have no idea why the error rate there should be different as the error rate used for alignment things (although I would hesitate to say that they should be the same without more brain power to think it through)

I have no idea why the error rate there should be different as the error rate used for alignment things (although I would hesitate to say that they should be the same without more brain power to think it through)

Yeah, speaking with Zam we came to this conclusion too. Requires some careful thinking...

I have been playing around with these two error rates today.
The data I am parameter sweeping with is Nanopore Mtb data. They have "truth" assemblies from PacBio CCS and the evaluation is done with varifier.
The x-axis labels indicate the filters/parameters used. The COMPASS (left-most) box is the sample's Illumina data SNP calls from the pipeline of the same name and bcftools is SNP calls for the nanopore data on that tool.
Filters/parameters key:

e = error rate
E = genotype error rate
g = minimum genotype confidence
G = maximum gaps fraction
d = minimum k-mer coverage
s = minimum strand bias percentage
I = maximum indel length

For a more comprehensive examination of a filter sweep, see here.
Recall - SNPs only

Precision - SNPs only

Recall - all variants

Precision - all variants

I've been staring at these sorts of plots all day, so it would be nice to have some fresh eyes to interpret.

what are the default values for e and E?

Is it possible to replot these only for samples with >40x depth, so we can disentangle coverage?

I think we are reaching the stage where once again we need to find our high confidence false positives and manually look through many of them in detail (VCF info, pileups, graphs). Are they enriched in specific loci, do they look confident to a human, is this all true using the PRG we had before de novo?

just deleted a comment.
So - we can achieve the same precision as compass for SNPs, if we lose 15% recall (some of this will be due to low coverage samples and not having GCP implemented)
Instead of snps/All could we see SNPs/indels?

what are the default values for e and E?

e = 0.11 (nanopore) and E = 0.01 (always)
I'll put together those other plots today

Is it possible to replot these only for samples with >40x depth, so we can disentangle coverage?

Here are the plots with samples that have coverage over 40x (which is 5/7). Note: there is a few extra filters in there now as I was concurrently working on https://github.com/mbhall88/head_to_head_pipeline/issues/48
Recall - SNPs only

Precision - SNPs only

Recall - all variants

Precision - all variants

Instead of snps/All could we see SNPs/indels?

This is a lot harder and will take a bit longer. I will have to create indel-only VCFs and then run varifier on them.