Swift compiler getting overly restrictive about code whitespaces

Motivation
I get a warning in XCode 16 about code like this:
private (set) var messagesEnabled : Bool = false
@objc (addInfo:) public func add(info : CDDeviceInfo)

Where I get "Extraneous whitespace between attribute name and '('; this is an error in the Swift 6 language mode". I feel this to be overly restrictive. It is my code, I have my style and there are mostly reasons for this style, in this case readability.
I use spaces to be better able to visually distingish elements.
For example, this is okay:
@objc public init(withSettings settings : CDPersistenceSettings)
but this is not, while keeping a whitespace at the very same position:
@objc (addInfo:) public func add(info : CDHueSyncDeviceInfo)
Same goes for this:
private var _baseUrl : String?
But not allowed:
private (set) var messagesEnabled : Bool = false
This would be valid, but affects readability for me:
private var _baseUrl : String?
private(set) var messagesEnabled : Bool = false
This looks much cleaner:
private var _baseUrl : String?
private (set) var messagesEnabled : Bool = false
The same goes for this piece of code:
activeTaskCount+= 1
where I get "'+=' is not a postfix unary operator", while
activeTaskCount += 1
AND
activeTaskCount+=1
work properly. Why? It doesn't make sense to me.
I style it that way so it is easier for me to distinguish in my assigning a value to a variable vs. CHANGING a value of a variable. For me, it is easer to oversee the "+=", if it must be written the same way I write value assignment.
Just have a look:
activeTaskCount = 1
activeTaskCount += 1
Is harder for me to distinguish than:
activeTaskCount = 1
activeTaskCount+= 1
Now your mileage may vary, and it might be different from user to user, but that is exactly my point: Let users write code the way THEY can work the most efficient way with it.
Proposed solution
The compiler is a tool that should serve the user, not the other way around. When I style certain things may way, it's my way, and obviously the compiler understands what I want to do - it just throws a warning "because you should do it that way" - but we're not at school and blindly need to follow instructions. Instead, we want to work efficiently the way WE function.
So by my perspective, this is no warning, and absolutely it should NOT be an error. The compiler is capable to compile it, so it is to be considered functional code. It should be up to me, the user, to choose how I style my code. Letting the compiler interfere at THIS level is overly restrictive, and absolutely not necessary by any means. Let users write code by their preferences and experiences.
Alternatives considered
No response
Additional information
No response

Agreed. This is restrictive and breaks already shipping code. We can't just be maintaining for these kind of syntactical nuances introduced in new versions of language.


同步错误

org.gradle.api.internal.tasks.DefaultTaskInputs$TaskInputUnionFileCollection cannot be cast to org.gradle.api.internal.file.collections.DefaultConfigurableFileCollection Possible causes for this unexpected error include:

Gradle's dependency cache may be corrupt (this sometimes occurs after a network connection timeout.) Re-download dependencies and sync project (requires network)

The state of a Gradle build process (daemon) may be corrupt. Stopping all Gradle daemons may solve this problem. Stop Gradle build processes (requires restart)

Your project may be using a third-party plugin which is not compatible with the other plugins in the project or the version of Gradle requested by the project.

In the case of corrupt Gradle processes, you can also try closing the IDE and then killing all Java processes.

本身网络问题无法下载gradle插件吧…


'Un-obsolete' XslTransform class

During discussion at the PR #4529 it was figured out that XslTransform class is not so obsolete as it seems.
More background: https://github.com/dotnet/corefx/issues/25874
While 'un-obsoleting' this class numerous notes should be removed, for example, in those articles:
https://docs.microsoft.com/en-us/dotnet/standard/data/xml/xslt-transformations
https://docs.microsoft.com/en-us/dotnet/standard/data/xml/xslt-transformations-with-the-xsltransform-class
cc: @mairaw @pjanotti

@pjanotti is this for 2.1 right?

PIng @pjanotti

Yes, this is for CoreFx 2.1.


Possible Query Limit?

This query doesn't work:
type Invoice struct {
	ID int32 `json:"id"`
	Month string `json:"month"`
	Sheet []Sheet `json:"sheet"`
	Firstname string `json:"firstname"`
	Lastname string `json:"lastname"`
	TotalHours int32 `json:"total_hours"`
}

sqlQuery := "Select students.id, (sheet -> 0 ->> 'start_time')::timestamp::date as month, sheet, firstname, lastname, total_hours from attendance join students on attendance.student_id = students.id join agencies on students.agency_id = agencies.id WHERE (sheet -> 0 ->> 'start_time')::timestamp::date >= (SELECT date_trunc('month', '2023-01-17'::date)::date)AND (sheet -> 0 ->> 'start_time')::timestamp::date <=  (SELECT date_trunc('month', '2023-01-17'::date + interval '1 month - 1 day')::date)"

var invoice []*Invoice
err := pgxscan.Select(ctx, db, &invoices, sqlQuery)

But this one does:
type Invoice struct {
	ID int32 `json:"id"`
	Sheet []Sheet `json:"sheet"`
	Firstname string `json:"firstname"`
	Lastname string `json:"lastname"`
	TotalHours int32 `json:"total_hours"`
}

sqlQuery := "Select students.id, sheet, firstname, lastname, total_hours from attendance join students on attendance.student_id = students.id join agencies on students.agency_id = agencies.id WHERE (sheet -> 0 ->> 'start_time')::timestamp::date >= (SELECT date_trunc('month', '2023-01-17'::date)::date)AND (sheet -> 0 ->> 'start_time')::timestamp::date <=  (SELECT date_trunc('month', '2023-01-17'::date + interval '1 month - 1 day')::date)"

var invoice []*Invoice
err := pgxscan.Select(ctx, db, &invoices, sqlQuery)

Is there some sort of limit to what SQL queries I can pass in? Because this SQL Query will work if I query the database directly, but not through golang.

Nevermind, it was a bug on my end, I was trying to run a SQL function on a subquery on data that didn't exist


feat: upgrade to aegir@9

Ref https://github.com/ipfs/js-ipfs/pull/485

Coverage remained the same at 96.154% when pulling 48f6897a0f92e2d20e2b6eb5d3bff5c9058a6451 on new-aegir into b55c105c75d89af9ca7b30c977de7602736f0bc7 on master.

@diasdavid ping

needs rebase. Also it has @coveralls doing that comment thing again

@diasdavid done


When a .cargo/config target-dir is set, xargo may fail

When a cargo target-dir is set, such as:
 > cat ~/.cargo/config
[build]
target-dir = "/Volumes/ramdisk/rs"

Xargo can fail, as the target directory is not where it expects, which can cause the cp_r utilities to fail unexpectedly. This was found in:  https://github.com/rust-lang/miri/issues/1285

I wonder if this is a duplicate of https://github.com/japaric/xargo/issues/195.
Also, copying what I said on the Miri side:

I doubt an absolute target-dir can work for xargo, we kind of require a clean dir to know what it is that we have to copy (everything in the target dir) -- and because otherwise cargo thinks it can be smart and skip the build of stuff that is already built, which might be why we see no build for you.

Given that xargo anyway prepares a temporary directory for the sysroot build, I think what we should do is overwrite target-dir. Is that possible?

You could put in a .cargo/config into every build you generate, because it uses the "most local" .cargo/config possible?

According to https://doc.rust-lang.org/cargo/reference/config.html#environment-variables, env vars take precedence, so we could also try setting CARGO_TARGET_DIR.

I guess part of the problem is, the user has requested you use target-dir for some reason - for me it's to reduce burn-out of my SSD due to huge and frequent builds taking write cycles so I build to a ram disk.
I think better, would be a way to make xargo work under the target dir that is set, rather than fighting to set it to something you want?

Another thought I had this morning was that this would affect distributions who attempt to package or use CARGO_TARGET_DIR in some way for their build artifacts. So I think finding a way to respect the target-dir would be really good. Maybe if you need a guaranteed clean workspace, something like target-dir/xargo.HASH (you currently do TMPDIR/xargo.HASH I think ).
Would that be a reasonable suggestion? How would cargo clean affect or work with this?

So I think finding a way to respect the target-dir would be really good.

I don't think it makes any sense. xargo builds in a temporary directory that it removes again when it is done. So how would there be any point in trying to follow local configuration about storing build artifacts elsewhere? It needs to know exactly where the build output is to process it further.

How would cargo clean affect or work with this?

xargo has its own workspace for building libstd. There is no interaction whatsoever with the workspace of your project. That's why I keep saying xargo should just overwrite the target dir.

I guess part of the problem is, the user has requested you use target-dir for some reason - for me it's to reduce burn-out of my SSD due to huge and frequent builds taking write cycles so I build to a ram disk.

In that case, shouldn't you put your TEMPDIR also on the ramdisk? If you do, xargo will build on that ramdisk even when it overwrites target-dir.

In that case, shouldn't you put your TEMPDIR also on the ramdisk? If you do, xargo will build on that ramdisk even when it overwrites target-dir.

No? My tempdir is not the same place as where I want my rust builds to go, they are seperate places.

I don't think it makes any sense. xargo builds in a temporary directory that it removes again when it is done. So how would there be any point in trying to follow local configuration about storing build artifacts elsewhere? It needs to know exactly where the build output is to process it further.

But the location where build artifacts go does matter, even if they are temporary. As mentioned, it could be in my case a ramdisk, for others a volume in a vm with different caching or performance needs, or a package distributor who in a chroot only has a readable build directory, or a different area with more capacity than my /home. Who knows? The issue here is xargo makes assumptions about my host system that are not always valid, when my host system says clearly "please build to this location".
A different way to ask this:
Is there a harm to xargo ta build to the target-dir, even if those artifacts are only temporary? Is there a real benefit to ignoring the target-dir parameter and using tmpdir?

No? My tempdir is not the same place as where I want my rust builds to go, they are seperate places.

I would just assume that if you are worried about SSD writes, surely you'd also be worried about tmpdir causing such writes?

Is there a harm to xargo ta build to the target-dir, even if those artifacts are only temporary?

I don't know how to make xargo work with a generic target-dir. Things get even worse when we cannot rely on the target-dir being empty before xargo runs -- currently, we assume everything in that target dir to be created by the sysroot build, which is how xargo does not need any special magic knowledge about sysroots to copy the right things.

Is there a real benefit to ignoring the target-dir parameter and using tmpdir?

Yes -- it means that the current, rather simple implementation strategy of xargo continues to work.
So, I can fix this bug by making xargo overwrite target-dir. Or else we can leave it unfixed as I do not know another way to fix it (nor do I have huge amounts of time to do the research for a more clever solution).

No? My tempdir is not the same place as where I want my rust builds to go, they are seperate places.

I would just assume that if you are worried about SSD writes, surely you'd also be worried about tmpdir causing such writes?

Yes, but my tmpdir is generally used by my email, not rust builds that often have multi-hundred MB binaries being written out on a high frequency ... rust is really IO intensive for builds.

Is there a harm to xargo ta build to the target-dir, even if those artifacts are only temporary?

I don't know how to make xargo work with a generic target-dir. Things get even worse when we cannot rely on the target-dir being empty before xargo runs -- currently, we assume everything in that target dir to be created by the sysroot build, which is how xargo does not need any special magic knowledge about sysroots to copy the right things.

Okay I get that but why not use the target-dir and then put in a .xargo.hash ...
IE:
Current you do:
TMPDIR/xargo.hash/<content>
Why not:
cargo-target-dir/xargo.hash/<content>

IE read the current target dir from env or the .cargo/config, and then just set your target dir to "cargo_target_dir/xargo.hash". That has the same properties you want, you know it's your directory, and clean etc, and it has the properties I want of being in the "cargo dir". I think this would be a simple way to achieve both our goals.

Is there a real benefit to ignoring the target-dir parameter and using tmpdir?

Yes -- it means that the current, rather simple implementation strategy of xargo continues to work.
So, I can fix this bug by making xargo overwrite target-dir. Or else we can leave it unfixed as I do not know another way to fix it (nor do I have huge amounts of time to do the research for a more clever solution).
Anyone else is of course welcome to come up with and implement a more clever solution if they want. :)

Is the solution above simple enough? I think simple is better than clever here :)
But thanks for taking all the time to explain this to me, I appreciate it!

Why not:
cargo-target-dir/xargo.hash/

I don't know if tmpdir, the crate we use for getting a temporary directory, supports that. It is really convenient to use, but not meant to cover weird special cases like this one. ;)
Also keep in mind that "normally", cargo-target-dir is relative (at least, the documentation explicitly says "relative to the workspace directory"). So we'd need to add some extra code to treat relative and absolute target-dirs separately. And then there is the problem that I don't know of a supported way in cargo to ask for the target-dir that is going to be used (there can be overrides on the project and user level and in the environment and whatnot).
It's certainly possible, but it's a lot of complexity. I don't agree with your assessment that this is "simple".


how to delete thread from database?

one thread was manually deleted and whenever the user messages us there is already a #DELETED-CHANNEL

Not sure what you mean by #deleted-channel. The bot would automatically close the thread in the background if its channel is deleted and make a new channel next time the user messages the bot.
However, you can use any sqlite database utility to view and edit the database. You should then be able to just delete the thread from the threads table.


Update readme.md

I propose README change, because in the new project to test this awesome package, I get the error when I run the app with the package, I get the error:
FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':app:preDebugBuild'.
> Could not resolve all task dependencies for configuration ':app:debugRuntimeClasspath'.
   > Could not resolve project :react-native-camera.
     Required by:
         project :app
      > Cannot choose between the following variants of project :react-native-camera:
          - generalDebugRuntimeElements
          - mlkitDebugRuntimeElements

and I found the solution with this solution: https://github.com/react-native-community/react-native-camera/issues/2276#issuecomment-498022704

I propose README change, because in the new project to test this awesome package, I get the error when I run the app with the package, I get the error:
FAILURE: Build failed with an exception.

What went wrong:
Could not determine the dependencies of task ':app:preDebugBuild'.

Could not resolve all task dependencies for configuration ':app:debugRuntimeClasspath'.
Could not resolve project :react-native-camera.
Required by:
project :app
Cannot choose between the following variants of project :react-native-camera:
- generalDebugRuntimeElements
- mlkitDebugRuntimeElements
and I found the solution with this solution: react-native-community/react-native-camera#2276 (comment)

I have the same problem. Have you solved it?

https://react-native-community.github.io/react-native-camera/docs/installation
Insert the following lines in android/app/build.gradle:
android {
  ...
  defaultConfig {
    ...
    missingDimensionStrategy 'react-native-camera', 'general' // <--- insert this line
  }
}

I'll be merging this one because there's no other workaround to resolve this build setting for gradle builds as it's related to react-native-camera and not directly to this. Thank you for this contributing @jchaves07


Options to set a commandPrefix in addCommands()

Description

Added possibility to set a commandPrefix in a Ok google [...] fashion
Motivation and Context

The change allow to filter out a lot of noise by only triggering when the prefix is spoken.

Fixes https://github.com/TalAter/annyang/issues/18
How Has This Been Tested?

Tested the match with pref and the no-match without the prefix
Tests : https://github.com/TalAter/annyang/pull/240/commits/cd1f7566a796584d22d0e248a463ad710b8e31bc#diff-e504a2e67b28d9ebc5fac06973ab8275R487

recognition.onresult will not add any commandPrefix
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

NB: Travis fails because of incompatibility of yarn and iojs
Node.js version v3.3.1 does not meet requirement for yarn. Please use Node.js 4 or later.

@TalAter Any thoughts on this ?


TLVDecoder: Prevent overflowing int when parsing TLV lengths

When checking a parsed TLV length, prevent input data from overflowing Int.MAX_VALUE.
Prevents the following exception (as can be seen when running the unit-test shouldDecodeTLVDataWithFourByteLengthOfAlmostIntMaxValue added in this commit):

Stacktrace was: java.lang.OutOfMemoryError: Java heap space
	at com.izettle.tlv.TLVDecoder.helper(TLVDecoder.java:99)
	at com.izettle.tlv.TLVDecoder.decode(TLVDecoder.java:35)
	at com.izettle.tlv.TLVDecoderTest.shouldDecodeTLVDataWithFourByteLengthOfAlmostIntMaxValue(TLVDecoderTest.java:200)

Ping @linnie @xiaodong-izettle

Nice work 👍

@oskarwiksten how does this relate to the other fixes? https://github.com/iZettle/izettle-toolbox/pull/198 and https://github.com/iZettle/izettle-toolbox/pull/199 ?
We might have different JVM:s that behave differently but neither shouldDecodeTLVDataWithFourByteLengthOfAlmostIntMaxValue nor shouldDecodeTLVDataWithFourByteLengthOfIntMaxValue gives a Java heap space OOM for me but a Requested array size exceeds VM limit OOM. I haven't digged into this myself but I tried fixing the Heap Space issue which at least on my machine this does not, are you running with special JVM settings to get a Heap Space issue and not Array size? Was the issue in prod a Array size or a Heap Space?
for next time it would be nice if you would copy me into the issue given that you knew I had PR:s doing the same thing.

@fredrikbackstrom, set something like -Xmx256m -Xms256m, the test will throw OOM... :)

ok, thanks for the clarification @xiaodong-izettle

NP, @fredrikbackstrom. I should have pinged you...  :)
Oskar probably run the test with Maven... :)


⚠ Bofrost.at has degraded performance

In a0f3a88, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 921 ms

Resolved: Bofrost.at performance has improved in 8adcc4a after 8 minutes.


Acceptance Criteria #7 - Display Password

WHEN the password is generated
THEN the password is either displayed in an alert or written to the page

Password is displayed in a windows alert box.


Action runs after socket.io response but the botMessage does not appear in component

We are using socket.io to get some more complicated answers from server and the messaging is socket event based.
We defined multiple actions and two of them are to implement the server side responses.
As a sample you can find code below.
All codes below are from ActionProvider.js
Socket initializer and the response event to client
const socketInitializer = async () => {
  // We just call it because we don't need anything else out of it
  await fetch("/api/socket/chatbotSocket")

  socket = io()

// Whenever a message is generated on server it reaches to this event on client as a socket message and works well
  socket.on('serverToClientMessage', (msg) => {
	  console.log('msg returned on client: ', msg)
	  addServerSideAnswerToMessages(msg)
  })
}

This action is for sending the message to server, response is not sync, it will come to the event defined above
const sendMessageToServer = async (_message) => {
    console.log('user message send to server: ', _message)
    socket.emit('clientToServerMessage', { author: 'user', message: _message })
}

This method also defined as an action, it writes the result to console and it works very well. When the event triggered this function runs with correct message
const addServerSideAnswerToMessages = (msg) => {
    const botMessage = createChatBotMessage(msg.message)
    console.log('botMessage: ', botMessage)

    setState((prev) => ({
	    ...prev,
	    messages: [...prev.messages, botMessage],
    }))
}

These two actions and the first socket listener works well and I see the result on console properly, but it does not appear on chat window.
<div>
{
React.Children.map(children, (child) => {
    return React.cloneElement(child, {
		         actions: {
			        sendMessageToServer, // Send to server
			        addServerSideAnswerToMessages, // Runs when the server responds over socket message
			        handleHello, // Client side hello
			        handlePolicy, // Client side another test
		        },
	        })
        })
    }
</div>

What could be the problem? Any one has any experience using with socket.io?
Thanks.

https://github.com/FredrikOseberg/react-chatbot-kit/issues/108
This issue may be related with the same problem with us.
Actions calling from MessageParser works properly, but when I call an action from another part related to another event, botMessage does not appear.
Wanted to share, because they could be related.

Found any solution for Action runs after socket.io response but the botMessage does not appear in component  ? @ticpoi


How to... Make Airflow fast again.

Airflow has been constantly using 90% of my server's CPU, but I was able to resolve it and bring it down to around 5-10%. I wanted to document and share it in case someone else has the same issue.
Below is everything I had to do to get airflow to work well on a standard digital ocean droplet with 2gb of ram and 1 vcpu.
If you have any other performance tuning tips, please share.
1. Scheduler Settings
AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=60 # Prevent airflow from reloading the dags all the time and set. This is the main setting that reduces CPU load in the scheduler
AIRFLOW__SCHEDULER__SCHEDULER_MAX_THREADS=1 # This should be set to (CPU Cores - 1)

2. Fix airflow 1.10 scheduler bug
The AIRFLOW-2895 bug in airflow 1.10, causes high CPU load, because the scheduler keeps looping without a break.
It's already fixed in master and will be included in airflow 1.10.1, but it could take weeks or months until its released. In the meantime this patch solves the issue:
Add this patch file to config/af_1.10_high_cpu.patch:
--- jobs.py.orig    2018-09-08 15:55:03.448834310 +0000
+++ jobs.py     2018-09-08 15:57:02.847751035 +0000
@@ -564,6 +564,7 @@

         self.num_runs = num_runs
         self.run_duration = run_duration
+        self._processor_poll_interval = 1.0

         self.do_pickle = do_pickle
         super(SchedulerJob, self).__init__(*args, **kwargs)
@@ -1724,6 +1725,8 @@
             loop_end_time = time.time()
             self.log.debug("Ran scheduling loop in %.2f seconds",
                            loop_end_time - loop_start_time)
+            self.log.debug("Sleeping for %.2f seconds", self._processor_poll_interval)
+            time.sleep(self._processor_poll_interval)

             # Exit early for a test mode
             if processor_manager.max_runs_reached():

Then create your own Dockerfile that uses this image as the base:
# Custom Dockerfile
FROM puckel/docker-airflow:1.10.0-2

# Install mssql support & dag dependencies
USER root
RUN apt-get update -yqq \
    && apt-get install -y gcc freetds-dev \
    && apt-get install -y git procps \
    && apt-get install -y vim \
    && pip install 'pymssql==2.1.1' \
    && pip install apache-airflow[mssql,slack] azure

# This fixes permission issues on linux.
# The airflow user should have the same UID as the user running docker on the host system.
# To set it automatically, run 'docker build --build-arg DOCKER_UID=`id -u` --rm -t my-airflow .'
ARG DOCKER_UID
RUN \
    : "${DOCKER_UID:?Build argument DOCKER_UID needs to be set and non-empty.}" \
    && usermod -u ${DOCKER_UID} airflow \
    && echo "Set airflow's uid to ${DOCKER_UID}"

# This fixes high CPU load in airflow 1.10
COPY config/af_1.10_high_cpu.patch /root/af_1.10_high_cpu.patch
RUN patch -d /usr/local/lib/python3.6/site-packages/airflow/ < /root/af_1.10_high_cpu.patch; \
    rm /root/af_1.10_high_cpu.patch

USER airflow

3. RBAC webserver high CPU load
If you upgraded to use the new RBAC webserver UI, you may also notice that the webserver is using a lot of CPU persistently.
For some reason the RBAC interface uses a lot of CPU on startup. If you are running on a low powered server, this can cause a very slow webserver startup and permanently high CPU usage.
I have documented this bug as AIRFLOW-3037. To solve it you can adjust the config:
AIRFLOW__WEBSERVER__WORKERS=2 # 2 * NUM_CPU_CORES + 1
AIRFLOW__WEBSERVER__WORKER_REFRESH_INTERVAL=1800 # Restart workers every 30min instead of 30seconds
AIRFLOW__WEBSERVER__WEB_SERVER_WORKER_TIMEOUT=300 #Kill workers if they don't start within 5min instead of 2min

Enabling RBAC is also buggy, but you can find instructions on how to fix it here.

thanks for link to the airflow issue, 1.10.2 is now out which fixes (2) at least. After upgrading CPU usage has gone down a lot, so not sure if the settings still need changing.

We have moved to AirFlow 1.10.2 to resolve the CPU usage, Good thing is that the issue we had, got fixed in our environment. However, we have observed that the DAG's tasks though are getting submitted and shows running on the AirFlow dashboard, but they kind of hold up with actual processing and then kind of appears to remain in the queue for about 60 seconds after that the actual execution happens. Please note that for our use case implementation

The AirFlow DAGs are not time dependent i.e. they are not 'Scheduled DAGs' but are triggered via a python code.
AirFlow v1.10.2 is being used as a single standalone instalation [executor = LocalExecutor] .

The python code watches a directory for any file(s) that arrive. For any file, it observes, the code triggers the AirFlow DAG. We get bundles of files arriving and so at any given instance, we have scenarios where multiple instance of the same DAGs are getting invoked [ code snippet provided below ]. The DAGs are triggered which in turn has a task that calls a python code to trigger a Kubernetes pod where some file related processing happens. Please find below an excerpt from the DAG code
positional_to_ascii = BashOperator(
                    task_id="uncompress_the_file",
                    bash_command='python3.6 ' + os.path.join(cons.CODE_REPO, 'app/Code/k8Job/create_kubernetes_job.py') + ' POS-PREPROCESSING {{ dag_run.conf["inputfilepath"] }} {{ dag_run.conf["frt_id"]}}',
                    execution_timeout=None,
                    dag=dag)

Once this task completes it triggers another DAGs are triggered which has task that processes data from the output of the previous DAG.
Please find below a few details of our config file parameters which may assist in assessing the root cause.
min_file_process_interval = 60
dag_dir_list_interval = 300
max_threads = 2
dag_concurrency = 16
worker_concurrency = 16
max_active_runs_per_dag = 16
parallelism = 32
sql_alchemy_conn = mysql://airflow:fewfw324$gG@someXserver:3306/airflow
executor = LocalExecutor

The DagBag parsing time: 1.305286. Please as well find below the snapshot of the command airflow list_dags -r below
-------------------------------------------------------------------
DagBag loading stats for /root/airflow/dags
-------------------------------------------------------------------
Number of DAGs: 7
Total task number: 23
DagBag parsing time: 1.305286
------------------------------+----------+---------+----------+------------------------------
file                          | duration | dag_num | task_num | dags
------------------------------+----------+---------+----------+------------------------------
/trigger_cleansing.py         | 0.876388 |       1 |        5 | ['trigger_cleansing']
/processing_ebcdic_trigger.py | 0.383038 |       1 |        1 | ['processing_ebcdic_trigger']
/prep_preprocess_dag.py       | 0.015474 |       1 |        6 | ['prep_preprocess_dag']
/prep_scale_dag.py            | 0.012098 |       1 |        6 | ['dataprep_scale_dag']
/mvp.py                       | 0.010832 |       1 |        2 | ['dg_a']
/prep_uncompress_dag.py       | 0.004142 |       1 |        2 | ['dx3_unzip_decrypt_dag']
/prep_positional_trigger.py   | 0.003314 |       1 |        1 | ['prep_positional_trigger']
------------------------------+----------+---------+----------+------------------------------

Below is the status of the airflow-scheduler service which is showing multiple processes
systemctl status airflow-scheduler
● airflow-scheduler.service - Airflow scheduler daemon
   Loaded: loaded (/etc/systemd/system/airflow-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-03-09 04:44:29 EST; 33min ago
 Main PID: 37409 (airflow)
   CGroup: /system.slice/airflow-scheduler.service
           ├─37409 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37684 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37685 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37686 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37687 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37688 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37689 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37690 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37691 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37692 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37693 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37694 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37695 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37696 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37697 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37699 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37700 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37701 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37702 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37703 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37704 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37705 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37706 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37707 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37708 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37709 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37710 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37712 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37713 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37714 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37715 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37717 /usr/bin/python3.6 /bin/airflow scheduler
           ├─37718 /usr/bin/python3.6 /bin/airflow scheduler
           └─37722 /usr/bin/python3.6 /bin/airflow scheduler

Now the fact that we have several files comming in the DAGs are constantly being fired and have ample DAG task that gets into a waiting stage. Strangely we though didnt have this issue when we were using v1.9 please advise.

I realized that in the 'airflow.cfg' file , setting the value of the 'min_file_process_interval' to zero resolved the problem I reported here.

@shanit-saha  min_file_process_interval this value should me a big number not zero for better performance .Even I am facing same issue you mentioned above .So many scheduler threads are running in background.I am not sure whether it is an expected behavior.Can you shed some light on this issue?

@anooprp : I would recommend you to go and take a look at the contents provided in this link Click Here. Moreover please elaborate your problem statement. For us moving to Airflow_v1.10.2 resolved slowness and high CPU usage.

We moved to puckel/Airflow-1.10.2 to try and resolve a poor performance we've had in  multiple environments. We are running on ECS Airflow 1.10.2 on AWS ECS. Interestingly, the CPU/mem never jump above 80%. The Airflow metadb stays very underutilized as well.
Below I've listed the configuration we're using, the DagBag parsing time plus the detailed execution times from the cProfile output of just running DagBag() in pure Python.
A few of our DAGs import a function from create_subdag_functions.py that returns a DAG we use in 12 DAGs. Most of those DAGs and their corresponding subdags only run on the hour, but 1 DAG / 3 Subdags run every 10 minutes.
max_threads = 2
dag_dir_list_interval = 300
dag_concurrency = 16
worker_concurrency = 16
max_active_runs_per_dag = 16
parallelism = 32
executor = CeleryExecutor

Some observations:

airflow list_dags -r takes a very long time too and runs off the example DAGs even though they are disabled. The time to parse for each DAG will jump around.
The duration for each DAG is inconsistent (but it only applies to our DAGs and not the examples)
There is usually a big jump in that parsing time. e.g. 5 dags will have durations < 1 and then the next 4 will have durations 20+
When we profiled the DagBag() function with cProfile we found that DagBag() spent most of its time in the airflow.utils.dag_processing.list_py_paths function  probably due to 50+ sql files in our /usr/local/airflow/dags folder
Looking through the Landing Times, the task time jumped an order of magnitude between two specific runs. I've tried to look through logs, etc and there is nothing notable between two runs. I've attached the image at the bottom. This performance loss was in Airflow 1.10.0

Solutions I've tried:

increasing/lowering max_threads
increasing/eliminating min_file_process_interval
clearing the airflow database of all DAGs and reloading
shutting down and redeploying the environment

DagBag loading stats for /usr/local/airflow/dags
-------------------------------------------------------------------
Number of DAGs: 42
Total task number: 311
DagBag parsing time:<PHONE_NUMBER>9999995
--------------------------------------------+--------------------+---------+----------+------------------------------------------------------------------------------------------------------------
--------------------------------------------+--------------------+---------+----------+------------------------------------------------------------------------------------------------------------
/dag1.py                                    | 60.576728          |       1 |       21 | ['dag1']
/dag2.py                                    | 55.092603999999994 |       1 |       28 | ['dag2']
/dag3.py                                    | 47.997972000000004 |       1 |       17 | ['dag3']
/dag4.py                                    | 22.99313           |       3 |       16 | ['dag4', 'dag4.subdag1', 'dag4.subdag2']
/dag5.py                                    | 0.67               |       1 |       21 | ['dag5']
/dag6.py                                    | 0.652114           |       1 |        9 | ['dag6']
/dag7.py                                    | 0.45368            |       1 |       26 | ['dag7']
/dag8.py                                    | 0.396908           |       5 |       40 | ['dag8', 'dag8.subdag1', 'dag8.subdag2', 'dag8.subdag3', 'dag8.subdag4']
/dag9.py                                    | 0.242012           |       6 |       38 | ['dag9', 'dag9.subdag1', 'dag9.subdag2', 'dag9.subdag3', 'dag9.subdag4', 'dag9.subdag5']
/dag10.py                                   | 0.134342           |       1 |        1 | ['dag10']
/dag11.py                                   | 0.13325            |       2 |        8 | ['dag11', 'dag12.subdag1']
/dag12.py                                   | 0.10562            |       1 |        6 | ['dag12']
/create_subdag_functions.py                 | 0.105292           |       0 |        0 | []
 example_http_operator.py                   | 0.040636           |       1 |        6 | ['example_http_operator']
 example_subdag_operator.py                 | 0.005328           |       3 |       15 | ['example_subdag_operator', 'example_subdag_operator.section-1', 'example_subdag_operator.section-2']
 example_bash_operator.py                   | 0.004052           |       1 |        6 | ['example_bash_operator']
 example_branch_operator.py                 | 0.003444           |       1 |       11 | ['example_branch_operator']
 example_branch_python_dop_operator_3.py    | 0.003418           |       1 |        3 | ['example_branch_dop_operator_v3']
 example_passing_params_via_test_command.py | 0.003222           |       1 |        2 | ['example_passing_params_via_test_command']
 example_skip_dag.py                        | 0.002386           |       1 |        8 | ['example_skip_dag']
 example_trigger_controller_dag.py          | 0.002386           |       1 |        1 | ['example_trigger_controller_dag']
 example_short_circuit_operator.py          | 0.002344           |       1 |        6 | ['example_short_circuit_operator']
 example_python_operator.py                 | 0.002218           |       1 |        6 | ['example_python_operator']
 example_latest_only.py                     | 0.002196           |       1 |        2 | ['latest_only']
 example_latest_only_with_trigger.py        | 0.001848           |       1 |        5 | ['latest_only_with_trigger']
 example_xcom.py                            | 0.001722           |       1 |        3 | ['example_xcom']
 docker_copy_data.py                        | 0.001718           |       0 |        0 | []
 example_trigger_target_dag.py              | 0.001704           |       1 |        2 | ['example_trigger_target_dag']
 tutorial.py                                | 0.00165            |       1 |        3 | ['tutorial']
 test_utils.py                              | 0.001376           |       1 |        1 | ['test_utils']
 example_docker_operator.py                 | 0.00103            |       0 |        0 | []
 subdags/subdag.py                          | 0.001016           |       0 |        0 | []
-------------------------------------------------------------------------------------------------------+--------------------+---------+----------+--------------------------------------------------

-------------------------------------------------------------------
DagBag loading stats for /usr/local/airflow/dags
-------------------------------------------------------------------
Number of DAGs: 42
Total task number: 311
DagBag parsing time:<PHONE_NUMBER>999999
------------------------------+--------------------+---------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
file                          | duration           | dag_num | task_num | dags
------------------------------+--------------------+---------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/dag1.py                      | 74.819988          |       1 |       21 | ['dag1']
/dag3.py                      | 53.193430000000006 |       1 |       17 | ['dag3']
/dag8.py                      | 34.535742          |       5 |       40 | ['dag8', 'dag8.subdag1', 'dag8.subdag2', 'dag8.subdag3', 'dag8.subdag4']
/dag4.py                      | 21.543944000000003 |       6 |       38 | ['dag9', 'dag9.subdag1', 'dag9.subdag2', 'dag9.subdag3', 'dag9.subdag4', 'dag9.subdag5']
/dag5.py                      | 18.458316000000003 |       3 |       16 | ['dag4', 'dag4.subdag1', 'dag4.subdag2']
/create_subdag_functions.py   | 14.652806000000002 |       0 |        0 | []
/dag7.py                      | 13.051984000000001 |       2 |        8 | ['dag11', 'dag11.subdag1']
/dag8.py                      | 10.02703           |       1 |       21 | ['dag5']
/dag9.py                      | 9.834226000000001  |       1 |        1 | ['dag10']
/dag10.py                     | 9.575258000000002  |       1 |       28 | ['dag2']
/dag11.py                     | 9.418897999999999  |       1 |        9 | ['dag6']
/dag12.py                     | 9.319210000000002  |       1 |        6 | ['dag12']
/dag13.py                     | 8.686964           |       1 |       26 | ['dag7']

Note: removed the example DAGs from the second output for brevity
cProfile output of from airflow.models import DagBag; DagBag():
{{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=6740
{{__init__.py:51}} INFO - Using executor SequentialExecutor
{{models.py:273}} INFO - Filling up the DagBag from

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      997  443.441    0.445  443.441    0.445 {built-in method io.open}
      198  186.978    0.944  483.629    2.443 zipfile.py:198(is_zipfile)
      642   65.069    0.101   65.069    0.101 {method 'close' of '_io.BufferedReader' objects}
     1351   45.924    0.034   45.946    0.034 <frozen importlib._bootstrap_external>:830(get_data)
     7916   39.403    0.005   39.403    0.005 {built-in method posix.stat}
      2/1   22.927   11.464  544.419  544.419 dag_processing.py:220(list_py_file_paths)
       33   18.992    0.576  289.797    8.782 models.py:321(process_file)
       22    8.723    0.397    8.723    0.397 {built-in method posix.scandir}
      412    2.379    0.006    2.379    0.006 {built-in method posix.listdir}
        9    1.301    0.145    3.058    0.340 linecache.py:82(updatecache)
 1682/355    0.186    0.000    0.731    0.002 sre_parse.py:470(_parse)
     1255    0.183    0.000    0.183    0.000 {built-in method marshal.loads}
 3092/325    0.143    0.000    0.647    0.002 sre_compile.py:64(_compile)
       59    0.139    0.002    0.139    0.002 {built-in method builtins.compile}
    25270    0.134    0.000    0.210    0.000 sre_parse.py:253(get)
    52266    0.132    0.000    0.132    0.000 {method 'append' of 'list' objects}
4210/4145    0.131    0.000    1.760    0.000 {built-in method builtins.__build_class__}

Airflow performance drop:

my min_file_process_interval is set to 0 even then my consecutive dag_runs wait for 90 seconds before starting.
How can I fix that?

@abhineetraj Do you have that issue in Airflow 1.10.3? Because I have the same issue and I do run 1.10.3.
I'm in the process of choosing a workflow engine that needs to have a clear GUI for monitoring purposes. Airflow seems to be what we need in terms of user interface, although it's very sluggish. For a simple task like the templated tutorial task (here), I expect it to finish within milliseconds.
Details below.
Here's the script I use for stress testing:
import datetime
import multiprocessing
import requests

url = 'http://airflow.localhost:8080/api/experimental/dags/tutorial/dag_runs'
headers = {
    'Cache-Control': 'no-cache',
    'Content-Type': 'application/json',
}

def test(instance, runs):
    for x in range(runs):
        payload = {
            'replace_microseconds': False,
        }

        response = requests.post(url, headers=headers, json=payload)
        print(response.text)
        if not response.ok:
            return

def run_tests(threads):
    thread_pool = []
    start_batch = datetime.datetime.now()
    print('Test: Calling Airflow example_trigger_controller_dag')
    print('Running test with {} threads'.format(threads))

    for thread in range(threads):
        thread_pool.append(multiprocessing.Process(target=test, args=(thread, 20)))
        thread_pool[thread].start()

    for thread in range(threads):
        thread_pool[thread].join()

    end_batch = datetime.datetime.now()
    return "total duration: {:7.5f}s".format(
        end_batch.timestamp() - start_batch.timestamp()
    )

if __name__ == '__main__':
    result = run_tests(threads=2)
    print(result)

The DAG/pipeline is exactly the same as the first code block in the tutorial I linked above
Here's a random log for the templated task, note the timestamps and the time in the log filename:
*** Reading local file: /root/airflow/logs/tutorial/templated/2019-06-28T07:38:16.058941+00:00/1.log
[2019-06-28 07:39:58,364] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: tutorial.templated 2019-06-28T07:38:16.058941+00:00 [queued]>
[2019-06-28 07:39:58,395] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: tutorial.templated 2019-06-28T07:38:16.058941+00:00 [queued]>
[2019-06-28 07:39:58,395] {__init__.py:1353} INFO -
--------------------------------------------------------------------------------
[2019-06-28 07:39:58,395] {__init__.py:1354} INFO - Starting attempt 1 of 2
[2019-06-28 07:39:58,395] {__init__.py:1355} INFO -
--------------------------------------------------------------------------------
[2019-06-28 07:39:58,411] {__init__.py:1374} INFO - Executing <Task(BashOperator): templated> on 2019-06-28T07:38:16.058941+00:00
[2019-06-28 07:39:58,411] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'tutorial', 'templated', '2019-06-28T07:38:16.058941+00:00', '--job_id', '1412', '--raw', '-sd', '/usr/local/lib/python3.6/site-packages/airflow/example_dags/tutorial.py', '--cfg_path', '/tmp/tmpxo1tgu19']
[2019-06-28 07:40:00,559] {base_task_runner.py:101} INFO - Job 1412: Subtask templated [2019-06-28 07:40:00,559] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=4338
[2019-06-28 07:40:01,107] {base_task_runner.py:101} INFO - Job 1412: Subtask templated [2019-06-28 07:40:01,106] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-06-28 07:40:01,716] {base_task_runner.py:101} INFO - Job 1412: Subtask templated [2019-06-28 07:40:01,714] {__init__.py:305} INFO - Filling up the DagBag from /usr/local/lib/python3.6/site-packages/airflow/example_dags/tutorial.py
[2019-06-28 07:40:01,786] {base_task_runner.py:101} INFO - Job 1412: Subtask templated [2019-06-28 07:40:01,784] {cli.py:517} INFO - Running <TaskInstance: tutorial.templated 2019-06-28T07:38:16.058941+00:00 [running]> on host 3063ba2ca0c5
[2019-06-28 07:40:01,887] {bash_operator.py:81} INFO - Tmp dir root location:
 /tmp
[2019-06-28 07:40:01,888] {bash_operator.py:90} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=tutorial
AIRFLOW_CTX_TASK_ID=templated
AIRFLOW_CTX_EXECUTION_DATE=2019-06-28T07:38:16.058941+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2019-06-28T07:38:16.058941+00:00
[2019-06-28 07:40:01,888] {bash_operator.py:104} INFO - Temporary script location: /tmp/airflowtmp_umua3_o/templatedr3ouai12
[2019-06-28 07:40:01,888] {bash_operator.py:114} INFO - Running command:

    echo "2019-06-28"
    echo "2019-07-05"
    echo "Parameter I passed in"

    echo "2019-06-28"
    echo "2019-07-05"
    echo "Parameter I passed in"

    echo "2019-06-28"
    echo "2019-07-05"
    echo "Parameter I passed in"

    echo "2019-06-28"
    echo "2019-07-05"
    echo "Parameter I passed in"

    echo "2019-06-28"
    echo "2019-07-05"
    echo "Parameter I passed in"

[2019-06-28 07:40:01,945] {bash_operator.py:123} INFO - Output:
[2019-06-28 07:40:01,947] {bash_operator.py:127} INFO - 2019-06-28
[2019-06-28 07:40:01,947] {bash_operator.py:127} INFO - 2019-07-05
[2019-06-28 07:40:01,947] {bash_operator.py:127} INFO - Parameter I passed in
[2019-06-28 07:40:01,947] {bash_operator.py:127} INFO - 2019-06-28
[2019-06-28 07:40:01,947] {bash_operator.py:127} INFO - 2019-07-05
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - Parameter I passed in
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - 2019-06-28
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - 2019-07-05
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - Parameter I passed in
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - 2019-06-28
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - 2019-07-05
[2019-06-28 07:40:01,948] {bash_operator.py:127} INFO - Parameter I passed in
[2019-06-28 07:40:01,949] {bash_operator.py:127} INFO - 2019-06-28
[2019-06-28 07:40:01,949] {bash_operator.py:127} INFO - 2019-07-05
[2019-06-28 07:40:01,949] {bash_operator.py:127} INFO - Parameter I passed in
[2019-06-28 07:40:01,949] {bash_operator.py:131} INFO - Command exited with return code 0
[2019-06-28 07:40:03,358] {logging_mixin.py:95} INFO - [2019-06-28 07:40:03,356] {jobs.py:2562} INFO - Task exited with return code 0

Hi, I'm a bit of a newbie, but I've been recently tasked to containerize Airflow for our company. I've been running into some performance issues and observed something of interest. Mainly... min_file_process_interval is documented to control when a new/updated file gets picked up by Airflow, but it also seems to control when a queued job is executed. I've tried changing this value to see if this is the case, and consistently I've found this config to be correlated to job queue times.
This is a bit of a problem at larger scale, since if you have many dags, the cost of dag refresh becomes more expensive. However, if you have many dags, you also want shorter intervals where jobs can be picked up if workers are available. Since one config controls both functionalities, you have to make a compromise somewhere, and this compromise could be a deal breaker if you are resource constrained, such as using AWS Fargate (max 4vCPU per task).
I am considering a few solutions... but before I do, has anyone scaled Airflow with 100+ dags using this fargate solution?

Hi,
I read the following:

Airflow Variables are stored in Metadata Database, so any call to variables would mean a connection to Metadata DB. Your DAG files are parsed every X seconds. Using a large number of variable in your DAG (and worse in default_args) may mean you might end up saturating the number of allowed connections to your database.
To avoid this situation, you can either just use a single Airflow variable with JSON value. As an Airflow variable can contain JSON value, you can store all your DAG configuration inside a single variable as shown in the image below

Can't find the settings for variable cache https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg

Guys - i must be missing something here with Airflow performance (version 1.10-5 and 4 as well ) .
i have made a very simple test (all with celery executor) - running the example_dag (the one with 12 bash operators that sleep random of 30 seconds)
when i run 100 instances of this dag (through rest API) - the scheduler is stuck after 3 minutes.

it does not matter if i run locally with celery executor - or in astronomer cloud with several workers.
if does not matter how much power i give to the scheduler/ workers.
i have played with every possible performance param
it looks like the scheduler just can't handle 1200 tasks fired at it at once.

anyone can help understand - either airflow is not working , or i'm missing something critical

I made another simple test - running the tutorial dag on LocalExecutor was way faster than Celery-Executor
i ran both locally with the docker compose files supplied by this repo.
version 1.10-4

I'm having an issue similar to @shlomiken, I have a simple DAG running with CeleryExecutor.
However, the scheduler stops scheduling next tasks after EXACTLY {parallelism} tasks. It doesn't matter if they're from a single DAG, two different DAG runs or parallel runs, after exactly {parallelism} tasks, the scheduler just loops over and over.
The tasks themselves are marked as success, it's as if they are completed everywhere except for the scheduler.
Restarting the scheduler causes the DAGs to start running from the same place they have stopped, just as they were supposed to

I want to add another simple test that does not succeed and cause the scheduler to halt.
running the tutorial dag , as its defined back- fill of 2015 also does not finish successfully and cause the scheduler to stop scheduling task.
i verified this both locally with celery executor (running 1 or more workers)
and on astronomer cloud (with 1 or more workers)

@shlomiken are you working with @ashb directly on this issue?

@shlomiken are you working with @ashb directly on this issue?

Sort of yes . he did some scheduler improvements that suppose to fix this , but it didn't work.
so i guess we continue to work on this.

Somehow related as I landed on this issue. I am running this on my MAC and the problem I was having was the default RAM allocation settings in the docker desktop for os X. Default is 2gb which was making everything slow and unresponsive.
Make this higher and everything is better again. Hopefully this can help someone.

@shlomiken is there a fix/workaround for this issue?

I'd recommend trying Airflow 2 Alpha, where the scheduler lag has been decreased significantly. Check out the announcement on the mailing list https://lists.apache.org/thread.html/rf34558953ba367561574c194500a34d7f3c21fe2798b173b86fc309c%40<dev.airflow.apache.org> and we expect beta in a couple of weeks, and release of Airflow 2 before end of year assuming everything goes as planned.

@ryw Thanks, but we cannot wait till EOY, since its a pressing issue for us. Lets see what we can do as a workaround. Any suggestion?

@gokhanoner Luigi inside python operator - https://issues.apache.org/jira/browse/AIRFLOW-6920

@tooptoop4 would you test your 200 sequential tasks DAG with Airflow 2 Alpha?
Would love to see how close we got to your Luigi numbers.

Some of the performance issues of the scheduler were fixed in version 10.7 - at least for our use case it was a major improvement and didn't get into the scheduler is stuck scenario.

@ryw time has not changed much :(

Yeah, would love more details on how you tested it, i.e. which Executor, what resources etc

@tooptoop4 What version were you running.
Testing on Alpha1 I see this result on the LocalExecutor:
   dag   | task_id |          start_date           |           end_date
---------+---------+-------------------------------+-------------------------------
 looper3 | op1     | 2020-10-26 15:34:48.334339+00 | 2020-10-26 15:34:48.583849+00
 looper3 | op2     | 2020-10-26 15:34:48.249266+00 | 2020-10-26 15:34:48.549773+00
 looper3 | op3     | 2020-10-26 15:34:48.274539+00 | 2020-10-26 15:34:48.538254+00
 looper3 | op4     | 2020-10-26 15:34:48.277586+00 | 2020-10-26 15:34:48.564638+00
 looper3 | op5     | 2020-10-26 15:34:48.333745+00 | 2020-10-26 15:34:48.599082+00
 looper3 | op6     | 2020-10-26 15:34:48.253816+00 | 2020-10-26 15:34:48.572588+00
 looper3 | op7     | 2020-10-26 15:34:48.267808+00 | 2020-10-26 15:34:48.542517+00
 looper3 | op8     | 2020-10-26 15:34:48.238346+00 | 2020-10-26 15:34:48.470832+00
 looper3 | op9     | 2020-10-26 15:34:48.312878+00 | 2020-10-26 15:34:48.538258+00
 looper3 | op10    | 2020-10-26 15:34:48.323253+00 | 2020-10-26 15:34:48.563398+00
 looper3 | op11    | 2020-10-26 15:34:48.253816+00 | 2020-10-26 15:34:48.558089+00
 looper3 | op12    | 2020-10-26 15:34:48.326777+00 | 2020-10-26 15:34:48.631332+00
 looper3 | op13    | 2020-10-26 15:34:48.397406+00 | 2020-10-26 15:34:48.634616+00
 looper3 | op14    | 2020-10-26 15:34:48.373018+00 | 2020-10-26 15:34:48.646307+00
 looper3 | op15    | 2020-10-26 15:34:48.366809+00 | 2020-10-26 15:34:48.59908+00
 looper3 | op16    | 2020-10-26 15:34:48.338096+00 | 2020-10-26 15:34:48.587398+00
 looper3 | op17    | 2020-10-26 15:34:50.65509+00  | 2020-10-26 15:34:51.010709+00
...

Total run time for this dag run is 31.614024s.

@ashb - How are you , just wanted to remind all that the tests we made and were problematic were mostly on celery even with 1 executor or many (back then in the test it wasn't matter)

@shlomiken :wave:
Testing on celery now.

@kaxil @ashb single pool slot as I'm testing fully sequential tasks

@tooptoop4 With a sequential dag:
 looper3 | op1     | 2020-10-26 16:32:11.125396+00 | 2020-10-26 16:32:11.208969+00
 looper3 | op2     | 2020-10-26 16:32:12.319367+00 | 2020-10-26 16:32:12.40748+00
 looper3 | op3     | 2020-10-26 16:32:13.471998+00 | 2020-10-26 16:32:13.555303+00
 looper3 | op4     | 2020-10-26 16:32:14.631682+00 | 2020-10-26 16:32:14.714908+00
 looper3 | op5     | 2020-10-26 16:32:15.837069+00 | 2020-10-26 16:32:15.942473+00
 looper3 | op6     | 2020-10-26 16:32:17.014756+00 | 2020-10-26 16:32:17.098437+00
 looper3 | op7     | 2020-10-26 16:32:18.227378+00 | 2020-10-26 16:32:18.321089+00
 looper3 | op8     | 2020-10-26 16:32:19.384662+00 | 2020-10-26 16:32:19.46615+00
 looper3 | op9     | 2020-10-26 16:32:20.580557+00 | 2020-10-26 16:32:20.666001+00
 looper3 | op10    | 2020-10-26 16:32:21.779499+00 | 2020-10-26 16:32:21.864664+00
 looper3 | op11    | 2020-10-26 16:32:22.93437+00  | 2020-10-26 16:32:23.027736+00
 looper3 | op12    | 2020-10-26 16:32:24.083956+00 | 2020-10-26 16:32:24.16761+00
 looper3 | op13    | 2020-10-26 16:32:25.273135+00 | 2020-10-26 16:32:25.359319+00
 looper3 | op14    | 2020-10-26 16:32:26.424024+00 | 2020-10-26 16:32:26.516319+00
 looper3 | op15    | 2020-10-26 16:32:27.61346+00  | 2020-10-26 16:32:27.699136+00
 looper3 | op16    | 2020-10-26 16:32:28.808298+00 | 2020-10-26 16:32:28.893568+00
 looper3 | op17    | 2020-10-26 16:32:29.995106+00 | 2020-10-26 16:32:30.078256+00
 looper3 | op18    | 2020-10-26 16:32:31.191905+00 | 2020-10-26 16:32:31.276655+00

So the task lag is about 5x better in 2.0.0alpha -- it's still slower than I'd like even if it is much much faster than 1.10

what config did u change to get the gap between tasks of around 1 second?

@Eyal Bar<EMAIL_ADDRESS>- FYI
On Mon, Oct 26, 2020 at 6:40 PM Ash Berlin-Taylor<EMAIL_ADDRESS>wrote:

So the task lag is about 5x better in 2.0.0alpha -- it's still slower than
I'd like even if it is much much faster than 1.10
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/puckel/docker-airflow/issues/233#issuecomment-716673857,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH2LRNKEKGQEPA7PGVWU6FLSMWRAXANCNFSM4FV7YL4Q
.

--
This e-mail, as well as any attached document, may contain material which
is confidential and privileged and may include trademark, copyright and
other intellectual property rights that are proprietary to Kenshoo Ltd,
its subsidiaries or affiliates ("Kenshoo"). This e-mail and its attachments
may be read, copied and used only by the addressee for the purpose(s) for
which it was disclosed herein. If you have received it in error, please
destroy the message and any attachment, and contact us immediately. If you
are not the intended recipient, be aware that any review, reliance,
disclosure, copying, distribution or use of the contents of this message
without Kenshoo's express permission is strictly prohibited.

No config changed - I used the new 2.0.0 alpha 2

FYI we've also made it easy to run Airflow 2 alpha locally using the astro CLI.

Install the CLI:

curl -sSL https://install.astronomer.io | sudo bash -s -- v0.21.0

Create an Airflow project w/ astro dev init

Change Dockerfile to:

FROM astronomerio/ap-airflow:2.0.0-1.dev3-buster-onbuild

You can install packages in requirements.txt using this format:

apache-airflow-providers-google
apache-airflow-providers-snowflake
apache-airflow-providers-http
apache-airflow-providers-postgres

If you have some dependency conflicts you can designate what version to use
with this format:
requests<2.24.0
idna<2.10

I had to do this for snowflake-connector-python package to work, for
example...

Then run astro dev start to fire everything up locally w/ Docker Compose

Note: you need to have Docker installed on your computer as well for this
to work.


ReaderT asks*K variants

This implements the ability to run an effect using the various From* type-classes after asking for something from the environment.
I'm happy to add some tests if they seem needed, but I figured that since it's mainly just composition and type-signatures they might not be too important.

Thanks @TylorS


expand data shown on stats page

Some things we might want to add:

[ ] Number of (unique?) sequences
[ ] Number of OTUs
[ ] Number of biomes represented

Removing easy-fix and milestone. The complexity of this issue is that currently we do not keep track of any of the sequences or OTUs statistics that were requested in the DB and to start displaying them we will need to:

Have a patch that counts the number of seqs and otus in the hdf5 files
Have a place to store them, perhaps the settings table?
Display those values

Perhaps the counter can be part of https://github.com/biocore/qiita/issues/814

I think this issue has been superseded by https://github.com/biocore/qiita/issues/1425. Open again if you think otherwise.


Convert StaticStrings that refer to a #filePath to Strings so that they are Codable

Preparation for a follow-up PR that makes Event conform to Codable.
Motivation:
Some data structures need to conform to Codable so that they can be transmitted between the test process and connected processes.
Modifications:
For consistency I converted all StaticStrings that refer to a #filePath to Strings, not just the ones I expect need to become Codable.
Result:
This change enables us in a future PR to make certain types conform to Codable.
Resolves rdar://116629725

@swift-ci please test

@swift-ci please test

Would it make sense to wait for my XCTIssue stuff to land, then just delete the #if in XCTS?

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

Don't forget to change this in XCTestScaffold.swift:
      XCTSourceCodeLocation(
        filePath: String(describing: sourceLocation._filePath),

Actually, you should search the package for anything using _filePath and get rid of any redundant String(describing:) calls you find. :)

Thanks, done!

@swift-ci please test

@swift-ci please test

Since CI is broken right now due to https://github.com/apple/swift/issues/69096, I double checked that this is building and testing successfully on macOS, Jonathan is looking into Linux.

Linux building and testing was successful as well.


Failed with quick orders

Quick orders are not working, even though I did exactly what you (@alexylem)  explained in https://www.openjarvis.com/content/snowboy#commandes-rapides
Things I did :

recorded my new hotword in snowboy website.
tested it.
downloaded the file which is in .pmdl format with file name all in lower_case .
placed it in stt_engines/snowboy/resources/
restarted openjarvis as well as rpi too.

jarvis didn't recongnize the second trigger word or quick word
Did I miss something in the above step?
Is there any changes to be made in snowboydecoder as you mentioned earlier in topic #15 ?

initialize snowboydecoderwith trigger + allpmdl

Btw awesome project @alexylem  , appreciate you're work. Thanks for making it open-source.

same issue,
I opened a ticket since 9 January  and no one answers

Maybe @wikijm or @Oliv4945 might help, they both are very helpful to me.
@wikijm is looking into all the issues and trying to give quick solutions, as I observed recently. He might help you too i guess..


update VRS annotation batch script for v4 exomes

There are a few changes implemented to make the run:

add function for the output path of VRS in v4 exomes;
Delete other v3 input and output paths in the dictionary;
put retry in init_job_with_gcloud to avoid Hail Batch transient errors, suggested by Dan King;
add bgzip in --run-vrs step so import_vcf could be run faster;
5. move import_vcf to --annotate-original step because this is faster to run in Query-on-Spark (QoS);
6. remove checkpoint since we know where the final folder of the VRS annotations
remove delete_temps because files got removed even if the jobs weren't all successful;
change exclude duplicate so it applies to only one input file;
use gnomad-tmp-4day in working-bucket, so the intermediate files can be automatically deleted);
10. Clarify in the docstring that the script has to be run once on QoB, once on QoS.

update 2023-08-16:

add test argument to test on 200 partitions;
put annotate-original back to original design, now with hailctl batch submit we can run both run-vrs and annotate-original in one go

@klaricch annotations.py is identical to main now, I couldn't merge main into this branch because it's been there too long.

@klaricch I made changes to get rid of vrs-io-finals, but you can't test with hailctl batch submit yet (because the docker image needs to be updated with the annotations.py).


Added 2.5rem width to class rz-slot-hour-header with max-width: 576px…

Css fix for mobile resolution in Scheduler module to fix with issue #1445

Thanks for the suggested fix, @thekim1! 🙏🏻  The issue however is still reproducible on screen sizes larger that 576px and would require a deeper CSS change. Also setting a fixed width might not work well in different themes / design systems. Closing this PR now and working on a universal fix.


ESP-IDF Wi-FI and Mesh Stacks Guides and Application Frameworks (IDFGH-2872)

Hello,
I would like to share with you the work I performed for my bachelor’s degree thesis in computer engineering, consisting of the following two comprehensive guides of the ESP-IDF Wi-Fi and Mesh stacks as of version 3.1 and the two derived application frameworks which, through the use of the menuconfig utility, allow the customization of most of the aspects of Wi-Fi and mesh networking in an application.
The Wi-Fi and Mesh stack guides consist of a total of 45 and 88 pages in .pdf format respectively, the following images showing some excerpts:

The two related application frameworks consist of 5928 lines of code, and while I understand that the code formatting doesn't strictly conform to the style guidelines on contributions, I feel confident of its clarity and comprehensibility, which are also enhanced by the comprehensive code documentation.
Thank you in advance for your time examining my work.
Riccardo Bertini

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for your interest on ESP-MESH, I will check your code and give you feedback soon.

@RickyDenton Thanks for the PR, since this is quite old, and we already have the mesh network examples, so we won't consider merging this PR.

Thanks for contribution again.


.trash on mac

Partly bug / partly question(?)
Don't crucify me, at the moment the output of .trash is:
  This is probably a custom directory on your system.
  You can move the file to XDG_DATA_HOME/trash.

My question is: If that is also relevant to macOS, or is it Linux only? If it's Linux only, it needs to be fixed.
On macOS, the .trash folder is the user's system-trash folder.
Thanks

If I understand correctly, you can't move the directory to XDG_DATA_HOME on macOS?
If that's the case, there should definitely be a note about this in the output.

I'm not 100% sure, but I couldn't find any documentation on how to move the folder. As it's treated as a special folder under MacOS, I think it can not be moved.

Thank you, I've added a note in cda9beb, if someone finds a way to move this directory, please open a PR or let me know!

Good job. I think that will be it for now. If anybody will find a solution in the future (if there is any) we will update it.


When doing multi hit skills, if one hit crits and the other is a non-crit, damage numbers overlap

See image for example:

Example skill used, Dual Strike (Warrior Axe offhand skill 4) or Whirling Axe (skill 5).

This seems to come and go for me, with restarts seemingly fixing it.


Static keyword highlighting is wrong

There is wierd 'static' keyword highlighting in .jsx react component in ES6 syntax (https://facebook.github.io/react/docs/react-component.html).
Specs: Windows 7 SP 1 (also Ubuntu 16.10), VSCode 1.8.1, 1.9.0.
Tested on a lot of Color Schemes (sublime-babel supported and others).

Please try the static branch I just pushed. Let me know if that resolves it for you without introducing any other new issues.
This issue seems to also happen in the original Sublime Text grammar that this is based on.

@joshpeng Everything works just fine without any other issues, thank you a lot :)


[question] Defining env with not bounded state

Hi,
I am working with a custom environment where the state boundaries are not known (they are generated based on an external lstm internal state and can oscillate a lot).
In order to set the environment boundaries with spaces.Box(low=..., high=..., shape=..., dtype=np.float32), I see three options:

Run the data for a certain period of time, find a local max and min, and normalize the state based on those values (it is not assured to get values higher than 1 or lower than 0).
Define huge boundaries so you assure it is likely to be within the limits.
Not defining the boundaries?

What is the recommendation in this case?
Thanks in advance.

Duplicate of https://github.com/hill-a/stable-baselines/issues/820
(there are also other issues related to that if you search more)

Sorry for the duplicate, did not read enough


Karte Does not compile with destination Apple Vision Pro

Karte does not compile with destination 'Apple Vision Pro' (Destination Apple Pro /Designed for iPad is no problem.
Problem is this code in enum Karte:
    if #available(iOS 10.0, *) {
        UIApplication.shared.open(url, completionHandler: nil)
    } else {
        UIApplication.shared.openURL(url)
    }

UIApplication.shared.openURL(url) isn't available in visionOS

Hey @GerdC, thanks for the report!
@lightsprint09 as you recently added visionOS to the list of supported platforms, do you happen to know how this can be used there?

I cloned Karte, and the cloned version compiles due to this fix
https://github.com/kiliankoe/Karte/pull/37

which is in a separate branch and not part of the main branch.

The change was merged into main, but I forgot to push a tag with that change. I just pushed 4.2.0, so SwiftPM should pull that in by default now :)

I confirm that it works now, thanks :-)
This is the second of the packages I use where just a release was missing :-)

Awesome, glad to know it now works! If you're willing to share what you're working on that uses this on visionOS, I'd be super curious! :)

Just looking what happens if I ran my tourist app natively.
Two more missing libraries that don't compile yet, before I know :-)
With destination "Apple Vision Pro (Designed for iPad)" it already runs nicely. https://attractions-map.com/en/barcelona_overview

Oh, very cool! Thanks for sharing!


Feature request(Provide README)

Is your feature request related to a problem? Please describe.
Should provide a completion README
Describe the solution you'd like
Help people to know the project.
Describe alternatives you've considered
[-] nothing
Additional context
[-]nothing

Yes, I will complete the document of launcher. As for the problem of the whole warehouse, I think it is a concluding thing, and it needs to wait until all functions are completed


Location is not preserved across views

Version and Platform (required):

Binary Ninja Version: 3.0.3233 Personal
OS: macOS
OS Version: 11.6.4

Bug Description:
When I have a file open to the Disassembly Graph view and I have a particular address selected then change the view to Linear Disassembly, the location that appears is different than the one highlighted in the previous view.
Steps To Reproduce:
See text above and video below.
Expected Behavior:
The current location in the binary should be retained across view changes within a pane.
Screenshots:
https://user-images.githubusercontent.com/607452/159508377-12d1edef-299d-44d8-9eb9-c386ed7deac3.mp4
Additional Information:
All user settings were disabled when creating the video above.

Original issue was reported via email by a different user.
I can reproduce the issue by following the steps shown in the screenshot with 3.0.3280-dev Enterprise.
I cannot reproduce the behavior by using the spacebar or the Command Palette option "View in Linear Disassembly".
I suspect we just need to change the behavior of the drop-down to do whatever the Command Palette option is doing.

Fixed in 3.0.3315-dev.


#31 Fails to build with Xcode 13

Fix compilation error updating SwiftSyntax

Signed-off-by: Enrique Garcia<EMAIL_ADDRESS>

Does the fatalError block in File.swift need to be updated to indicate Swift 5.5, as per the adjacent comment?

Yes, you're right, in fact I should have read the comment in the line above my change.
I would add the change ASAP.

I've think of another approach to support more than one swift version at a time.
I've tested it with Xcode 13 and Xcode 12.5.1, both compiling without errors.
What do you think?

I love this – thank you so much!


Adjust import of StationPropertyValueRenderer

In https://github.com/iTwin/presentation-rules-editor/pull/173 await was added before import("../experimental/StationPropertyValueRenderer.js") in ITwinJsApp.tsx. This seems to work when running with start option, but does not work with production build.
Decided to remove the EXPERIMENTAL_STATION_VALUE_RENDERER value, as it always has a true value and we never change it. This allows us to remove the await and import it normally.

Have you tried changing build target to es2022? https://vite.dev/config/build-options#build-target

Have you tried changing build target to es2022? https://vite.dev/config/build-options#build-target

Yes, it didn't help.

Closing in favor of https://github.com/iTwin/presentation-rules-editor/pull/175


[wit2ts] wit2ts fails if return type is List

So I am not sure if this is actually a bug on the JSON generation with wasm-tools
However,
If you try to generate TS for the following code
package wyatt-herkamp:archiveexplorer;

interface types {
	variant file-type {
        file,
        directory
	}
    record file{
        name: string,
        ftype: file-type,
        size: u64
    }
}
world archiveexplorer {
	use types.{ file, file-type };

	export open-archive: func(path: string) -> list<file>;
}

wit2ts silently errors out with no message. After some digging. I learned that after parsing the result from wasm-tools it creates an error of Error: Unsupported owner [object Object]
Now I can generate the Typescript by returning a single record. So export open-archive: func(path: string) -> file
I don't know how WIT and WASI work enough to go further than this.

@wyatt-herkamp thanks for the issue.
I improved the error reporting and fixed the qualified name generation for world owners.

Will create a new version mid next week.

New version is out. Latest is: 0.1.0-pre.13


[EUWE] When the record is saved to the database, the timestamps are truncated

Reload the record so that the timestamps saved in the database match the in memory timestamps

@simaishi This was added by https://github.com/ManageIQ/manageiq/pull/16426


Create and validate a build pipeline for the swift collection

Checked in a dev-pipeline along with other artifacts in the kabanero-piepline repo that will build an appsody swift repo and publish the image to docker hub.  We will be adding security tasks to this pipeline in the near future.

For now the same dev-pipeline works the same for all the featured collections.


install-to-filesystem does not work when targeting LVM

Host system is Fedora 39 Server.
Using the tier1 Fedora ELN image, I created a custom variant that had bootc installed from Copr:
FROM registry.gitlab.com/centos/cloud/sagano/fedora-boot-tier-1:eln
RUN curl -L -o /etc/yum.repos.d/bootc.repo https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/repo/fedora-rawhide/rhcontainerbot-bootc-fedora-rawhide.repo
RUN sed -i 's|\$releasever|rawhide|' /etc/yum.repos.d/bootc.repo
RUN rpm-ostree install bootc
RUN ostree container commit

Built it successfully with sudo podman build -t sagano-bootc-copr:latest -f Containerfile.sagano .
Tried install-to-filesystem:
$ image=localhost/sagano-bootc-copr:latest
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs
ERROR No filesystem uuid found in target root

Hmm...looks like it might be SELinux as the culprit:
$ sudo ausearch -m avc
----
time->Fri Nov  3 12:03:42 2023
type=AVC msg=audit(1699027422.008:673): avc:  denied  { nosuid_transition } for  pid=3425 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c54,c375 tcontext=system_u:system_r:install_t:s0:c54,c375 tclass=process2 permissive=0

Similar failure using CentOS Stream 9.
$ cat Containerfile.sagano
FROM registry.gitlab.com/centos/cloud/sagano/centos-boot-tier-1:stream9
# Not sur why I can't just use the copr repo, but it doesn't show as an
# enabled repo when I copy it to /etc/yum.repos.d ¯\_(ツ)_/¯
RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit

$ image=localhost/sagano-cs9-bootc-copr:latest
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs
ERROR Inspecting filesystem /target: findmnt /target failed: ExitStatus(unix_wait_status(256))
$ sudo ausearch -m avc
----
time->Fri Nov  3 16:07:22 2023
type=PROCTITLE msg=audit(1699042042.505:752): proctitle=2F746D702F2E746D70376174694E4800696E7374616C6C2D746F2D66696C6573797374656D002D2D7461726765742D6E6F2D7369676E61747572652D766572696669636174696F6E002D2D6B6172673D636F6E736F6C653D74747953302C3131353230306E38002D2D7265706C6163653D616C6F6E6773696465002F74617267
type=PATH msg=audit(1699042042.505:752): item=0 name="/lib64/ld-linux-x86-64.so.2" inode=34460089 dev=00:32 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:container_file_t:s0:c264,c980 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1699042042.505:752): cwd="/"
type=EXECVE msg=audit(1699042042.505:752): argc=6 a0="/tmp/.tmp7atiNH" a1="install-to-filesystem" a2="--target-no-signature-verification" a3="--karg=console=ttyS0,115200n8" a4="--replace=alongside" a5="/target"
type=SYSCALL msg=audit(1699042042.505:752): arch=c000003e syscall=59 success=yes exit=0 a0=55bdd5e15820 a1=55bdd5ddfca0 a2=55bdd5e16ca0 a3=8 items=1 ppid=35181 pid=35183 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4 comm=".tmp7atiNH" exe="/tmp/.tmp7atiNH" subj=system_u:system_r:unconfined_t:s0:c264,c980 key=(null)
type=SELINUX_ERR msg=audit(1699042042.505:752): op=security_bounded_transition seresult=denied oldcontext=system_u:system_r:unconfined_t:s0:c264,c980 newcontext=system_u:system_r:install_t:s0:c264,c980
type=AVC msg=audit(1699042042.505:752): avc:  denied  { nosuid_transition } for  pid=35183 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c264,c980 tcontext=system_u:system_r:install_t:s0:c264,c980 tclass=process2 permissive=0

RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit

(this is unrelated to the bug, but what you really want is more like:
RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm && \
  rpm-ostree install bootc.rpm && rm bootc.rpm && ostree container commit

This will avoid leaking the RPM (as a file) into the container image and ensure that unnecessary content is properly squashed.

Similar failure using CentOS Stream 9.

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy.  (When you first said "Using c9s" I thought you meant for the host, not the container)

Hmm...looks like it might be SELinux as the culprit:

Right.  We've been trying really really really hard to make this work.  We have https://github.com/containers/bootc/issues/83 which is related.
For https://github.com/containers/bootc/pull/78 I finally just gave up and in that mode did a setenforce 0.  (Background: for reasons similar to this, Anaconda has always run in setenforce 0 mode)
Can you check that setenforce 0 beforehand works?  Bear in mind that that this command is effectively replacing the OS, and setenforce 0 is a transient operation for the soon-to-go-away OS.
It's really tempting to do that unconditionally but I'm trying to Do It Right as much as possible.  I'm not yet sure why it works in some paths but not others.
cc @rhatdan

Oh actually this has nothing to do with selinux...we're already detecting this situation and running setenforce 0 internally already.  The problem is LVM

Can't we run the tool that is attempting to install as install_t or unconfined_t?  Is this bootc?  We could just transition to an unconfined role. Looks like there is a type
install_exec_t
Which transition to install_t, which is allowed to do anything including put labels that don't exist yet, I believe.

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy.  (When you first said "Using c9s" I thought you meant for the host, not the container)

I amended my comment to indicate I was using a c9s host, even if it is still an LVM issue.

@cgwalters Hi! I run into the same issue on ARM VM in beaker. Some of ARM VM in beaker are deployed as LVM.

@cgwalters Do you think we need to escalate this issue a little bit? Azure RHEL on-demand images are using LVM. And I just tried to do bootc install on a RHEL9.3 public image on Azure, which lead to exactly the same problem here.

Ah, good find.  So...I don't think we need to escalate this right now, let's just document the places that it works and the limitation on LVM.
The real fix for the "in-place" will be https://github.com/containers/bootc/pull/78 which will default to rewriting the filesystem to not be LVM (i.e. it will default to what the container image specifies).
(We still do want to support configuring LVM in this scenario of course, but that gets into its own trickiness)


Ensure nested event handlers work correctly

If element A contains element B, and they both declare (say) an onclick handler, and the user clicks on B, then you'd expect both event handlers to run.
Currently they do both try to run (first B, then A, as the event bubbles up), but with a render cycle between them. The render cycle might dispose the handler for A before we try to run it, which would lead to a crash.
We should avoid this issue by capturing the list of all event handler IDs to be invoked, pass them all to .NET atomically, and have it run all the corresponding delegates before any render cycle occurs.

This may be related. If B defines a custom event handled in A, A will receive the event, but not re-render even if ShouldRender is true. It doesn't crash. Example:
CompA.cshtml:
@using System
<button onclick=@OnClick>CompA</button>
<CompB BClick=@BClicked />
BClicked time: @_t

@functions
{
    DateTime? _t;

    private void OnClick()
    {
        Console.WriteLine("button clicked in CompA");
    }

    private void BClicked()
    {
        Console.WriteLine("BClicked, ShouldRender=" + ShouldRender());
        _t = DateTime.Now;
    }

}

CompB.cshtml:
@using System
@using Microsoft.AspNetCore.Blazor.Components
<button onclick=@OnClick>CompB</button>

@functions
{
    [Parameter]
    private Action BClick { get; set; }

    private void OnClick()
    {
        Console.WriteLine("button clicked in CompB");
        BClick?.Invoke();
    }
}


Image coercions

@ayush-1506 Thanks indeed for this. I will try to look at this as soon as I can. In the mean time, can you please change the base to "dev" branch?
In case others are interested in reviewing, the context is https://github.com/alan-turing-institute/MLJFlux.jl/pull/33#issuecomment-616251504

Codecov Report

Merging #33 into master will decrease coverage by 0.86%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #33      +/-   ##
==========================================
- Coverage   99.07%   98.20%   -0.87%
==========================================
  Files           7        7
  Lines         216      223       +7
==========================================
+ Hits          214      219       +5
- Misses          2        4       +2

Impacted Files
Coverage Δ

src/convention/coerce.jl
98.48% <100.00%> (-1.52%)
:arrow_down:

src/schema.jl
96.96% <0.00%> (-3.04%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ca446a...9220fcb. Read the comment docs.

@ablaom Thanks for the review. Everything seems to work fine now. Let me know if you still need me to change the base to dev branch.

Closing in favour of https://github.com/alan-turing-institute/MLJScientificTypes.jl/pull/34


Fast writer

Starting with the newer version of parquet-python as of early October, I have repeated some of the ideas in this fork from last year to produce a high-performance, pandas- and dask-oriented reader and writer of parquet.
A brief readme can be found here and the set of features I aim to implement in issue #1 . New issues with comments and suggestions are also welcome, as well as a conversation below.
High-level aims are to support perfomant reading of parquet, including selection of only some columns and only some row-groups and handling the many-file format produced by spark/hive, and being able to transparently read parquet from alternative file-system backends such as s3fs and hdfs3. The writer should, similarly, produce parquet files that are usable by spark and other tools, although I don't anticipate enabling all encoding options.
I am posting this PR now to gain feedback from interested people, particularly @jcrobak , who worked on the base parquet-python project. Some specific questions for you: this work here is substantially different from what currently exists in the main fork, and I'd like you to consider whether a merger is possible. In particular, this work now depends on numpy, pandas and numba, and is designed to allow parallelism between row-groups using dask dataframes.

ccing @jcrist @hussainsultan @jreback @seibert for review

looks awesome @martindurant esp like the use of numba jit classes. I didn't really check but you prob need a min version for pandas/numba on this?

Yes, I believe so - I am working with most recent conda versions, and haven't checked how many versions back I can go.

I am posting this PR now to gain feedback from interested people, particularly @jcrobak , who worked on the base parquet-python project.

Thanks! This looks really cool.

Some specific questions for you: this work here is substantially different from what currently exists in the main fork, and I'd like you to consider whether a merger is possible. In particular, this work now depends on numpy, pandas and numba, and is designed to allow parallelism between row-groups using dask dataframes.

This is a great question! Your work certainly has a different set of goals (performance+integration with the python data tools) than mine historically has. Specifically, I've made parquet-python as light-weight as possible so that it's easy to integrate. Adding deps on numpy/pandas/numba may make that more complicated for what I see is the primary use case (a quick and dirty way to dump the first few hundred records in a file).
With that said, we do overlap on a lot of goals that you've mentioned in  https://github.com/martindurant/parquet-python/issues/1. One area that's different: I've spent some time working on support for nested schemas, arrays, dicts, etc. And it's unclear to me (I see you have some ideas) how that fits with a dataframe model.
So, I see a few ways forward:

incorporate your code into my parquet-python codebase, but adding numpy/pandas/etc as optional dependencies. If those deps exist, the optimized version is transparently used (unless the schema is too complex)
Same as 1, but use a different namespace/set of functions for the dataframe codebase.
Keep the codebases separate—if useful, your code code depend on mine for common functionality (but I'm not sure that's useful).
Completely merge the codebases. I've asked some of the folks using parquet-python of their opinions in https://github.com/jcrobak/parquet-python/issues/46

Look forward to discussing!

@mrocklin

I'm very interested in identifying sorted columns that we can use as the index
The parquet metadata says whether any columns (there can be more than one) are sorted per row_group, and if there are statistics, we can infer whether the row_groups form a progression of values. Thus we might, in some circumstances, be able to establish if a column is sorted throughout the data, and so use it as the index and set divisions.

performance relative to disk-io speeds
I can look into it, but I started to become pleased when the snappy decompression started to be a significant portion (1/4-1/3) of the time budget.

how we identify hot-spots in numba code.
They were coded as python first, of course, but I don't know how to profile within the numba jit-ed functions. If we get that close to the metal, we can get help.
For the moment, I am sure that there are more gains to be made by eliminating numpy and pandas operations to do with concat and type-casting. Maybe @jreback will spot some easy wins. One example: dictionary encoding is stored as int32 and so loaded into an int32; but in pandas it's always int64. More clever preallocating should be used.

Regarding options for integration with @jcrobak main fork I find myself somewhat favoring option 3 keeping the codebases separate with explicit imports if it makes sense.
My reasoning for this is that I expect @martindurant 's version here to evolve fairly quickly over the next few months.  Demand for this is very high among Dask users at the moment.  I suspect that this code churn will probably not be acceptable to existing users of @jcrobak 's version.  I would prefer to see development be able to proceed quickly for a while.  Perhaps after things calm down we could reassess.  In the mean time we can work to make some of the more difficult dependencies, like llvmlite, pip installable.

@jcrobak , I think it may be reasonable to go for Option 3 for two reasons: a) attempting a merger for the amount of changes I have made, but keeping the original functionality (with its fewer dependencies) working unaltered seems unfeasible, or at least very hard, and b) I would like to be able to distribute this code to interested people for feedback, and that would be best done using a conda recipe in a separate fork.
That doesn't mean that a merger is off the cards - this can be a temporary measure - and of course your feedback will continue to be very welcome.


Fixed docker run command

With Docker 1.10, at least:
invalid reference format: repository name must be lowercase

@tj-corrigan - As this is a README update only, I'm merging this in.  Looks like at least going back to docker 1.8 it's required lowercase.


Remove LocalAuthority model

This is no longer needed as the source of truth for these is in Local
Links Manager.
Follow up to #389
For https://trello.com/c/opAHOMr4/443-4-of-4-delete-old-ldg-import-into-publisher-and-related-code-3

👍  Merge when ready


Major release 1.0.0

[x]  Visibility callbacks for entering and exiting
[x] Lock in first measures (avoid firing threshold callback for 0 measurement)
[ ] Maestro tests or jest tests

Just going to merge now and publish 1.0 from main after updating docs.  want to test the new functionality bbbbbb


Clean up sphinx warnings.

Helps with https://github.com/googlegenomics/readthedocs/issues/83

LGTM


#151948528 api tests need fixing

What does this PR do?
It fixes and completes API tests
Description of Task to be completed?
Fix tests in test_admin.py
Fix tests in test_badge.py
Fix tests in test_check_model.py
Fix tests in test_create_check.py
Fix tests in test_ensuretriggers.py
Fix tests in test_list_checks.py
Fix tests in test_notify.py
Fix tests in test_pause.py
Fix tests in test_ping.py
Fix tests in test_sendalerts.py
What are the relevant pivotal tracker stories?
#151948528

Coverage decreased (-0.1%) to 80.03% when pulling 67c210e31e7f2d91e196f93e21f769d0c1a3bbfa on ch-api-tests-need-fixing-151948528 into befb7dee19f4cc8743375f06769e66014b8a12a7 on develop.

Coverage increased (+0.8%) to 80.888% when pulling a978f6690e20b3e16cf1d099e875d6842ba0e015 on ch-api-tests-need-fixing-151948528 into befb7dee19f4cc8743375f06769e66014b8a12a7 on develop.


visitor matchmaking inspired by https://grip.events/

visitor could select keywords describing their interest

badge color OR icon = branch


Give more output formats than just CSL JSON

E.g. BibTeX and RIS would be nice, or just formatted references.
Lars Willighagen did some work on building an API around Citation.js that might come in handy: https://twitter.com/rintzezelle/status/921356848305004545 and https://debug-citeproc-js.glitch.me/docs

See also https://citation.js.org/api/tutorial-input_types.html and https://citation.js.org/api/tutorial-output_options.html

Solved with Citation.js v0.4.0-3, now offer CSL JSON, BibTeX, RIS, and APA-formatted output.


Drop the spec URL for Offline web applications

This change drops the spec URL for the "Offline web applications" feature — because that entire section has now been removed from the HTML standard.

Mh, wouldn't there be value in having a deprecated spec linked? Like the snapshot before it was removed?

A specification URL of some kind is required, perhaps https://www.w3.org/TR/2011/WD-html5-20110525/offline.html#offline would be a better alternative?

A specification URL of some kind is required, perhaps https://www.w3.org/TR/2011/WD-html5-20110525/offline.html#offline would be a better alternative?

I don’t

A specification URL of some kind is required, perhaps https://www.w3.org/TR/2011/WD-html5-20110525/offline.html#offline would be a better alternative?

WFM — amended the patch here

Thanks!


Arrowing down through cells does not put the cursor in the correct location

Have two cells with more than one line each
Arrow down to the second
Expected:
Your cursor is at the top on the second cell
Actual:
Your cursor is on the bottom of the second cell.
Doesn't seem to be a recent regression, but this worked at some point

Works two versions ago. Recent regression.

Validated. Looks good now.


Use pod monitor

This configures the operator to be scraped using a pod monitor instead of the old scheme

Your PR says it makes use of a PodMonitor while you're actually using a ServiceMonitor? IMHO we can drop the service and really just use a PodMonitor as you proposed for the etcd-kubernetes-resources-count-exporter.

I tried but it does not work because this pod is using hostnetwork:true

But that's probably more an issue because of not using CiliumNetworkPolicies, right? Like, plain NetworkPolicies can not be used for allowing traffic to host network.

Hmm thé issue I had was really related to kyverno policy exceptions but you're free to try a Pod monitor instead :) I don't have the bandwith to do them all "properly", I really just need to get rid of the old annotations asap:)

Sure :)

It's fixed @Gacko