Bent prongs

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This was also the case at our YRV site. Both were the 80 cm depth. So from now on that will be the first thing I try to check and fix.

Hi Deann,
What is the soil like at 80 cm at your locations? Is this very rocky? Close
to the bedrock or parent material? Will this be a frequent occurrence? In
PA, in the foothills, for example, they have had some struggles at this
depth. Or do you think this was a learning curve?
Just curious.
On Wed, Jul 21, 2021 at 11:01 PM deannrp @.***> wrote:

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This was
also the case at our YRV site. Both were the 80 cm depth. So from now on
that will be the first thing I try to check and fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3LJLNDPAP5GRKEMD27FGDTY6CXLANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC  27695
phone:<PHONE_NUMBER>

It’s silty clay texture dominated by smectite that’s the problem, and a learning curve. From now on if the data looks bad we will pull it, straighten the prongs, push it back in, and watch the data for a few days before contacting you.
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 8:56:48 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs (#148)
This email originated from outside of K-State.
Hi Deann,
What is the soil like at 80 cm at your locations? Is this very rocky? Close
to the bedrock or parent material? Will this be a frequent occurrence? In
PA, in the foothills, for example, they have had some struggles at this
depth. Or do you think this was a learning curve?
Just curious.
On Wed, Jul 21, 2021 at 11:01 PM deannrp @.***> wrote:

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This was
also the case at our YRV site. Both were the 80 cm depth. So from now on
that will be the first thing I try to check and fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3LJLNDPAP5GRKEMD27FGDTY6CXLANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885656977, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQMGYDRCOIXUT5I3TCXIPIDTZFYKBANCNFSM5AZEI2RA.

Do you have the wavelength guide tool for accurate distance between the
prongs?
On Fri, Jul 23, 2021 at 10:08 AM deannrp @.***> wrote:

It’s silty clay texture dominated by smectite that’s the problem, and a
learning curve. From now on if the data looks bad we will pull it,
straighten the prongs, push it back in, and watch the data for a few days
before contacting you.
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 8:56:48 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs
(#148)
This email originated from outside of K-State.
Hi Deann,
What is the soil like at 80 cm at your locations? Is this very rocky? Close
to the bedrock or parent material? Will this be a frequent occurrence? In
PA, in the foothills, for example, they have had some struggles at this
depth. Or do you think this was a learning curve?
Just curious.
On Wed, Jul 21, 2021 at 11:01 PM deannrp @.***> wrote:

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This was
also the case at our YRV site. Both were the 80 cm depth. So from now on
that will be the first thing I try to check and fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148>,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AL3LJLNDPAP5GRKEMD27FGDTY6CXLANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885656977>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AQMGYDRCOIXUT5I3TCXIPIDTZFYKBANCNFSM5AZEI2RA

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885665306,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3LJLNSD37UCVE4VNZBLPLTZFZWRANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC  27695
phone:<PHONE_NUMBER>

Yes
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 11:57:18 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs (#148)
This email originated from outside of K-State.
Do you have the wavelength guide tool for accurate distance between the
prongs?
On Fri, Jul 23, 2021 at 10:08 AM deannrp @.***> wrote:

It’s silty clay texture dominated by smectite that’s the problem, and a
learning curve. From now on if the data looks bad we will pull it,
straighten the prongs, push it back in, and watch the data for a few days
before contacting you.
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 8:56:48 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs
(#148)
This email originated from outside of K-State.
Hi Deann,
What is the soil like at 80 cm at your locations? Is this very rocky? Close
to the bedrock or parent material? Will this be a frequent occurrence? In
PA, in the foothills, for example, they have had some struggles at this
depth. Or do you think this was a learning curve?
Just curious.
On Wed, Jul 21, 2021 at 11:01 PM deannrp @.***> wrote:

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This was
also the case at our YRV site. Both were the 80 cm depth. So from now on
that will be the first thing I try to check and fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148>,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AL3LJLNDPAP5GRKEMD27FGDTY6CXLANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885656977>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AQMGYDRCOIXUT5I3TCXIPIDTZFYKBANCNFSM5AZEI2RA

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885665306,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3LJLNSD37UCVE4VNZBLPLTZFZWRANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885772321, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQMGYDVJCNSPSQLUYM2LAA3TZGNO5ANCNFSM5AZEI2RA.

It's worth a shot! Thanks for troubleshooting this. Do you have extra TDR's
if you need them? Send a pic of the prongs if you think of it?
On Fri, Jul 23, 2021 at 1:37 PM deannrp @.***> wrote:

Yes
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 11:57:18 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs
(#148)
This email originated from outside of K-State.
Do you have the wavelength guide tool for accurate distance between the
prongs?
On Fri, Jul 23, 2021 at 10:08 AM deannrp @.***> wrote:

It’s silty clay texture dominated by smectite that’s the problem, and a
learning curve. From now on if the data looks bad we will pull it,
straighten the prongs, push it back in, and watch the data for a few days
before contacting you.
DeAnn Presley
Professor, Soil Science
Kansas State University
Work cell:<PHONE_NUMBER>

From: Sarah Seehaver @.>
Sent: Friday, July 23, 2021 8:56:48 AM
To: precision-sustainable-ag/On-farm-Protocols @.>
Cc: Deann Presley @.>; Author @.>
Subject: Re: [precision-sustainable-ag/On-farm-Protocols] Bent prongs
(#148)
This email originated from outside of K-State.
Hi Deann,
What is the soil like at 80 cm at your locations? Is this very rocky?
Close
to the bedrock or parent material? Will this be a frequent occurrence? In
PA, in the foothills, for example, they have had some struggles at this
depth. Or do you think this was a learning curve?
Just curious.
On Wed, Jul 21, 2021 at 11:01 PM deannrp @.***> wrote:

Pulled the "bad" sensor at OOA tonight and the prongs were bent. This
was
also the case at our YRV site. Both were the 80 cm depth. So from now
on
that will be the first thing I try to check and fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148
,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AL3LJLNDPAP5GRKEMD27FGDTY6CXLANCNFSM5AZEI2RA

.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<

https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885656977

,
or unsubscribe<

https://github.com/notifications/unsubscribe-auth/AQMGYDRCOIXUT5I3TCXIPIDTZFYKBANCNFSM5AZEI2RA

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885665306
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AL3LJLNSD37UCVE4VNZBLPLTZFZWRANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC 27695
phone:<PHONE_NUMBER>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885772321>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AQMGYDVJCNSPSQLUYM2LAA3TZGNO5ANCNFSM5AZEI2RA

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/precision-sustainable-ag/On-farm-Protocols/issues/148#issuecomment-885794385,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3LJLIX5WTSFHVXQIXUHPDTZGSGRANCNFSM5AZEI2RA
.

--
Sarah Seehaver Eagen
NCSU Department of Crop and Soil Sciences
Southern Cover Crops Council Program Manager
Campus Box 7620
101 Derieux Pl
Raleigh, NC  27695
phone:<PHONE_NUMBER>


Make sure writes are thread safe

We are using Puma in a threaded environment. SQLite gest unhappy if you try to commit from 2 threads. Do we need a monitor around write uses, or does Sequel have a way of handling this natively? Do we need to reduce the DB connection pool to 1 when using SQLite?

Reducing the pool to 1 is a good way of ensuring that we will not have race
conditions, but it will also generate a lot of contention.
What about this? https://www.sqlite.org/sharedcache.html
On Wed, Oct 28, 2015, 21:37 Mike Virata-Stone<EMAIL_ADDRESS>wrote:

We are using Puma in a threaded environment. SQLite gest unhappy if you
try to commit from 2 threads. Do we need a monitor around write uses, or
does Sequel have a way of handling this natively? Do we need to reduce
the DB connection pool to 1 when using SQLite?
—
Reply to this email directly or view it on GitHub
https://github.com/bundler/gemstash/issues/42.

Create a separate connection for writes, use a Monitor around any code that uses it. This pattern already exists in the bundler-api codebase. :+1:

Isn't this exactly what database transactions exist for?

I suggest balancing the benefits of having more than one connection on the pool but having to use a monitor or any other kind of synchronization around it. I'm afraid that synchronizing that will hurt more than help. Synchronization is done with system calls, after all.
Maybe it is better to use only one connection and touch it only if we start suffering from that?

@segiddins Exactly.
Should we run some tests before changing the code? I think that this threading issue should be handled by the database layer, maybe we need to tweak things a bit, but I rather not have specific SQLite cases hidden in the code.
Why don't we pull, say, 10,000 gems from actual rubygems and build the import script to inject them all at once and see how gemstash behaves and if it is still functional?
Still, this 10,000 gems case seems quite an isolated edge case as I don't expect people from doing this every day.

OK, so I would think transactions would solve this problem, except I use transactions whenever I'm dealing with a thread sensitive multi-statement database interaction, and yet I got this error in a project I work on:
E, [2015-10-17T09:15:31.970378 #12401] ERROR -- : Error while [REDACTED]: (ActiveRecord::StatementInvalid) SQLite3::BusyException: database is locked: [REDACTED SQL UPDATE]
  ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `step'
  ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:108:in `block in each'
  ...
  ~/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
  ~/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/transactions.rb:220:in `transaction'
  ~/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
  ...

I created a test script to replicate this using Sequel (like we use in Gemstash) rather than ActiveRecord and Rails from my project:
# Gemfile
source "https://rubygems.org"
gem "sequel"
gem "sqlite3"

# migrations/01_create_tmp_table.rb
Sequel.migration do
  change do
    create_table :tests do
      primary_key :id
      String :value, size: 255, null: false
    end
  end
end

# threading.rb
#!/usr/bin/env ruby
require "sequel"
db_path = File.expand_path("../test.db", __FILE__)
db = Sequel.connect("sqlite://#{db_path}")
Sequel.extension :migration
Sequel::Migrator.run(db, File.expand_path("../migrations", __FILE__), use_transactions: true)

100.times do
  threads = []

  100.times do |i|
    threads << Thread.new do
      db.transaction do
        db[:tests].insert(value: "Value: #{i}")
        db[:tests].insert(value: "Value: #{i}")
      end
    end
  end

  threads.each &:join
end

At first nothing happened (I had slightly different values for how many times to insert etc), and I thought maybe I misunderstood my original problem. Then, after running it a bunch (maybe this problem only occurs once you have a certain size of dataset), I got this:
$ ./threading.rb
~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:108:in `step': SQLite3::BusyException: database is locked (Sequel::DatabaseError)
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:108:in `block in each'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:107:in `loop'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/statement.rb:107:in `each'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:158:in `to_a'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:158:in `block in execute'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:95:in `prepare'
	from ~/.rvm/gems/ruby-2.2.2/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:134:in `execute'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/adapters/sqlite.rb:187:in `block (2 levels) in _execute'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/logging.rb:33:in `log_yield'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/adapters/sqlite.rb:187:in `block in _execute'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/connection_pool/threaded.rb:99:in `hold'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/connecting.rb:249:in `synchronize'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/adapters/sqlite.rb:178:in `_execute'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/adapters/sqlite.rb:149:in `execute_insert'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/dataset/actions.rb:966:in `execute_insert'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/dataset/actions.rb:336:in `insert'
	from ./threading.rb:14:in `block (4 levels) in <main>'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/transactions.rb:134:in `_transaction'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/transactions.rb:108:in `block in transaction'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/connecting.rb:249:in `block in synchronize'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/connection_pool/threaded.rb:103:in `hold'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/connecting.rb:249:in `synchronize'
	from ~/.rvm/gems/ruby-2.2.2/gems/sequel-4.27.0/lib/sequel/database/transactions.rb:97:in `transaction'
	from ./threading.rb:13:in `block (3 levels) in <main>'

@pcarranza: I don't think we should have issues with 10,000 gems during import, it's more about concurrent writes (though size of the dataset seems to help trigger the issue)... so multiple pushes happening at the same time, or maybe someone adding a user authentication while someone else is pushing a gem (come to think of it... that might be problematic with SQLite, considering that is a multi-process scenario, but it is probably such a slim chance to cause a problem, so maybe better to ignore that case). In my original project, we were dealing with maybe a dozen things or so happening around the same time. Earlier in the project, it wasn't happening, only once we had a larger dataset.

Sqlite should pretty much never be used in multiprocess scenarios.

@segiddins the above script I wrote is multithreaded though, not multiprocess (and Gemstash is multithreaded too... but you could run actions while the server is running that touch the database... we could deal with that by blocking those actions if the server is running and SQLite is in use, though)

Right, that is really interesting. My take is that SQLite will basically deadlock in a heavy concurrent write scenario. The good part is that it will probably survive a heavy write scenario without corruption.
The only situation where I can imagine this scenario is by having a service that is used by a lot of clients that are writing a lot; or when importing those 10k gems while trying to write something or authorise a token.
The former case should be handled with a more beefy SQL server implementation (postgres, mysql, whatever), the later case, even if we insisting in using SQLite (because I assume that later on the use is going to be mostly reads: way more pulling than pushing and we do not want to spend maintaining a service) we could just handle with an exponential backoff retry.
I can see the challenge from the engineering perspective as it is always fun to scale things that should not be capable of doing it (by manually managing synchronisation in this particular case). I just wonder if it just makes sense to let the user know that he needs an infrastructure that matches the kind of use that he is going to do: SQLite for simple tasks with low concurrency, a real service for a real use where a lot of users will be hammering the service. And then just let the service manage the load.
For what is worth, probably the size of the dataset matters because the SQLite will need to load the indexes from the file to then insert the new value and write it back, and while doing this it is basically locking the whole table, which is going to be worse as the dataset grows.
This is usually handled gracefully by services because the indexes are in memory all the time... until you hit the point where the database needs to swap out the index, in which case you start seeing a degraded database, and it is not fun. Or is it?

Okay, I think we've gotten a little bit off track here. Let me see if I can summarize and address each issue in this ticket separately:
Issue 1: Writes need to be thread-safe, e.g. Puma using more than one thread to serve requests should not possibly corrupt the database.
Issue 2: Trying to write from two threads at exactly the same time raises an SQLite3::BusyException.
Issue 3: SQLite3 may not be suitable for large-scale deployments of gemstash.
Issue 1 is already solved by SQLite3 itself. It's not possible to corrupt the database by using the same connection from multiple threads.
Issue 2 is a symptom of SQLite's reduced concurrency support, and can be fixed by simply ensuring that writes happen one at a time. SQLite will do this automatically if you increase the busy timeout so that conflicting writes simply wait for their turn rather than raising right away. If we want to do it ourselves, we can add a mutex around write operations so that only execute one write operation a time.
Issue 3 is unclear to me—even a very large gemstash is unlikely to have multiple, critical simultaneous database write operations. But let me be extremely clear: no deadlocks are happening, and it should not even be possible to get this codebase into a state where it deadlocks.
The implication of issue 3 is that it is somehow possible to "scale up" gemstash by switching it from sqlite3 to postgres. However, gemstash is an extremely poor candidate for scaling via adding more web servers that share a database because it depends on the filesystem as well. If you're using gemstash, and you have so many users writing so many gems that sqlite3 can't save them all, you will need to scale your filesystem—you can't just switch to postgres and say the problem's fixed.
Sorry for the novella. tl;dr: sqlite3 handles concurrency without danger, and we can prevent errors while using it with a tiny amount of work. there is a limit to how many gems we can save per second, but that limit is based on writing multi-megabyte .gem files to the filesystem, not on any sqlite3 limits.

Thanks for the analysis @indirect!
I think it's also worth noting that (assuming not NFS) SQLite supports multi-process access. This means that for the ultra rare case of adding a gem at the same time as adding a new authorization key, SQLite will be fine.
Also, it seems Sequel already uses a 5 second timeout by default.
One more thing, I tweaked my script to have max_connections: 1 and the busy error completely goes away.
Based on all this info, I'm inclined to say we should have max_connections: 1 when SQLite is in use. There doesn't seem to be a significant speed problem with that (it actually seemed to be faster on heavy writes), and our database usage is so light that I doubt this would be a problem anyways. This would mean no need for locks, so no chance for deadlocks.
We could also go a max_connections: 1 write only connection used for any inserts or updates, and a regular connection pool for reads... for non-SQLite DB, the 2 would just be references to the same thing. I don't know if the busy issue would crop up between write and read uses, but it sounded like no.

Since even 30 or 60 second timeouts can't provide a guarantee against busy exceptions, I think that only allowing 1 connection seems fine.

:+1:


get help from docstring

What does this PR do?
Get help message of argparse from docstring.
➜ python3 pl_examples/basic_examples/mnist.py --help
usage: mnist.py [-h] [--batch_size BATCH_SIZE] [--logger [LOGGER]]
                [--checkpoint_callback [CHECKPOINT_CALLBACK]]
                [--default_root_dir DEFAULT_ROOT_DIR]
                [--gradient_clip_val GRADIENT_CLIP_VAL]
                [--process_position PROCESS_POSITION] [--num_nodes NUM_NODES]
                [--num_processes NUM_PROCESSES] [--gpus GPUS]
                [--auto_select_gpus [AUTO_SELECT_GPUS]]
                [--tpu_cores TPU_CORES] [--log_gpu_memory LOG_GPU_MEMORY]
                [--progress_bar_refresh_rate PROGRESS_BAR_REFRESH_RATE]
                [--overfit_batches OVERFIT_BATCHES]
                [--track_grad_norm TRACK_GRAD_NORM]
                [--check_val_every_n_epoch CHECK_VAL_EVERY_N_EPOCH]
                [--fast_dev_run [FAST_DEV_RUN]]
                [--accumulate_grad_batches ACCUMULATE_GRAD_BATCHES]
                [--max_epochs MAX_EPOCHS] [--min_epochs MIN_EPOCHS]
                [--max_steps MAX_STEPS] [--min_steps MIN_STEPS]
                [--limit_train_batches LIMIT_TRAIN_BATCHES]
                [--limit_val_batches LIMIT_VAL_BATCHES]
                [--limit_test_batches LIMIT_TEST_BATCHES]
                [--val_check_interval VAL_CHECK_INTERVAL]
                [--flush_logs_every_n_steps FLUSH_LOGS_EVERY_N_STEPS]
                [--log_every_n_steps LOG_EVERY_N_STEPS]
                [--accelerator ACCELERATOR]
                [--sync_batchnorm [SYNC_BATCHNORM]] [--precision PRECISION]
                [--weights_summary WEIGHTS_SUMMARY]
                [--weights_save_path WEIGHTS_SAVE_PATH]
                [--num_sanity_val_steps NUM_SANITY_VAL_STEPS]
                [--truncated_bptt_steps TRUNCATED_BPTT_STEPS]
                [--resume_from_checkpoint RESUME_FROM_CHECKPOINT]
                [--profiler [PROFILER]] [--benchmark [BENCHMARK]]
                [--deterministic [DETERMINISTIC]]
                [--reload_dataloaders_every_epoch [RELOAD_DATALOADERS_EVERY_EPOCH]]
                [--auto_lr_find [AUTO_LR_FIND]]
                [--replace_sampler_ddp [REPLACE_SAMPLER_DDP]]
                [--terminate_on_nan [TERMINATE_ON_NAN]]
                [--auto_scale_batch_size [AUTO_SCALE_BATCH_SIZE]]
                [--prepare_data_per_node [PREPARE_DATA_PER_NODE]]
                [--amp_backend AMP_BACKEND] [--amp_level AMP_LEVEL]
                [--distributed_backend DISTRIBUTED_BACKEND]
                [--automatic_optimization [AUTOMATIC_OPTIMIZATION]]
                [--hidden_dim HIDDEN_DIM] [--learning_rate LEARNING_RATE]

optional arguments:
  -h, --help            show this help message and exit
  --batch_size BATCH_SIZE
  --logger [LOGGER]     Logger (or iterable collection of loggers) for
                        experiment tracking.
  --checkpoint_callback [CHECKPOINT_CALLBACK]
                        Callback for checkpointing.
  --default_root_dir DEFAULT_ROOT_DIR
                        Default path for logs and weights when no
                        logger/ckpt_callback passed. Default: ``os.getcwd()``.
                        Can be remote file paths such as `s3://mybucket/path`
                        or 'hdfs://path/'
  --gradient_clip_val GRADIENT_CLIP_VAL
                        0 means don't clip.
  --process_position PROCESS_POSITION
                        orders the progress bar when running multiple models
                        on same machine.
  --num_nodes NUM_NODES
                        number of GPU nodes for distributed training.
  --num_processes NUM_PROCESSES
  --gpus GPUS           number of gpus to train on (int) or which GPUs to
                        train on (list or str) applied per node
  --auto_select_gpus [AUTO_SELECT_GPUS]
                        If enabled and `gpus` is an integer, pick available
                        gpus automatically. This is especially useful when
                        GPUs are configured to be in "exclusive mode", such
                        that only one process at a time can access them.
  --tpu_cores TPU_CORES
                        How many TPU cores to train on (1 or 8) / Single TPU
                        to train on [1]
  --log_gpu_memory LOG_GPU_MEMORY
                        None, 'min_max', 'all'. Might slow performance
  --progress_bar_refresh_rate PROGRESS_BAR_REFRESH_RATE
                        How often to refresh progress bar (in steps). Value
                        ``0`` disables progress bar. Ignored when a custom
                        callback is passed to :paramref:`~Trainer.callbacks`.
  --overfit_batches OVERFIT_BATCHES
                        Overfit a percent of training data (float) or a set
                        number of batches (int). Default: 0.0
  --track_grad_norm TRACK_GRAD_NORM
                        -1 no tracking. Otherwise tracks that p-norm. May be
                        set to 'inf' infinity-norm.
  --check_val_every_n_epoch CHECK_VAL_EVERY_N_EPOCH
                        Check val every n train epochs.
  --fast_dev_run [FAST_DEV_RUN]
                        runs 1 batch of train, test and val to find any bugs
                        (ie: a sort of unit test).
  --accumulate_grad_batches ACCUMULATE_GRAD_BATCHES
                        Accumulates grads every k batches or as set up in the
                        dict.
  --max_epochs MAX_EPOCHS
                        Stop training once this number of epochs is reached.
  --min_epochs MIN_EPOCHS
                        Force training for at least these many epochs
  --max_steps MAX_STEPS
                        Stop training after this number of steps. Disabled by
                        default (None).
  --min_steps MIN_STEPS
                        Force training for at least these number of steps.
                        Disabled by default (None).
  --limit_train_batches LIMIT_TRAIN_BATCHES
                        How much of training dataset to check (floats =
                        percent, int = num_batches)
  --limit_val_batches LIMIT_VAL_BATCHES
                        How much of validation dataset to check (floats =
                        percent, int = num_batches)
  --limit_test_batches LIMIT_TEST_BATCHES
                        How much of test dataset to check (floats = percent,
                        int = num_batches)
  --val_check_interval VAL_CHECK_INTERVAL
                        How often to check the validation set. Use float to
                        check within a training epoch, use int to check every
                        n steps (batches).
  --flush_logs_every_n_steps FLUSH_LOGS_EVERY_N_STEPS
                        How often to flush logs to disk (defaults to every 100
                        steps).
  --log_every_n_steps LOG_EVERY_N_STEPS
                        How often to log within steps (defaults to every 50
                        steps).
  --accelerator ACCELERATOR
                        Previously known as distributed_backend (dp, ddp,
                        ddp2, etc...). Can also take in an accelerator object
                        for custom hardware.
  --sync_batchnorm [SYNC_BATCHNORM]
                        Synchronize batch norm layers between process
                        groups/whole world.
  --precision PRECISION
                        Full precision (32), half precision (16). Can be used
                        on CPU, GPU or TPUs.
  --weights_summary WEIGHTS_SUMMARY
                        Prints a summary of the weights when training begins.
  --weights_save_path WEIGHTS_SAVE_PATH
                        Where to save weights if specified. Will override
                        default_root_dir for checkpoints only. Use this if for
                        whatever reason you need the checkpoints stored in a
                        different place than the logs written in
                        `default_root_dir`. Can be remote file paths such as
                        `s3://mybucket/path` or 'hdfs://path/' Defaults to
                        `default_root_dir`.
  --num_sanity_val_steps NUM_SANITY_VAL_STEPS
                        Sanity check runs n validation batches before starting
                        the training routine. Set it to `-1` to run all
                        batches in all validation dataloaders. Default: 2
  --truncated_bptt_steps TRUNCATED_BPTT_STEPS
                        Truncated back prop breaks performs backprop every k
                        steps of much longer sequence.
  --resume_from_checkpoint RESUME_FROM_CHECKPOINT
                        To resume training from a specific checkpoint pass in
                        the path here. This can be a URL.
  --profiler [PROFILER]
                        To profile individual steps during training and assist
                        in identifying bottlenecks.
  --benchmark [BENCHMARK]
                        If true enables cudnn.benchmark.
  --deterministic [DETERMINISTIC]
                        If true enables cudnn.deterministic.
  --reload_dataloaders_every_epoch [RELOAD_DATALOADERS_EVERY_EPOCH]
                        Set to True to reload dataloaders every epoch.
  --auto_lr_find [AUTO_LR_FIND]
                        If set to True, will make trainer.tune() run a
                        learning rate finder, trying to optimize initial
                        learning for faster convergence. trainer.tune() method
                        will set the suggested learning rate in self.lr or
                        self.learning_rate in the LightningModule. To use a
                        different key set a string instead of True with the
                        key name.
  --replace_sampler_ddp [REPLACE_SAMPLER_DDP]
                        Explicitly enables or disables sampler replacement. If
                        not specified this will toggled automatically when DDP
                        is used. By default it will add ``shuffle=True`` for
                        train sampler and ``shuffle=False`` for val/test
                        sampler. If you want to customize it, you can set
                        ``replace_sampler_ddp=False`` and add your own
                        distributed sampler.
  --terminate_on_nan [TERMINATE_ON_NAN]
                        If set to True, will terminate training (by raising a
                        `ValueError`) at the end of each training batch, if
                        any of the parameters or the loss are NaN or +/-inf.
  --auto_scale_batch_size [AUTO_SCALE_BATCH_SIZE]
                        If set to True, will `initially` run a batch size
                        finder trying to find the largest batch size that fits
                        into memory. The result will be stored in
                        self.batch_size in the LightningModule. Additionally,
                        can be set to either `power` that estimates the batch
                        size through a power search or `binsearch` that
                        estimates the batch size through a binary search.
  --prepare_data_per_node [PREPARE_DATA_PER_NODE]
                        If True, each LOCAL_RANK=0 will call prepare data.
                        Otherwise only NODE_RANK=0, LOCAL_RANK=0 will prepare
                        data
  --amp_backend AMP_BACKEND
                        The mixed precision backend to use ("native" or
                        "apex")
  --amp_level AMP_LEVEL
                        The optimization level to use (O1, O2, etc...).
  --distributed_backend DISTRIBUTED_BACKEND
                        deprecated. Please use 'accelerator'
  --automatic_optimization [AUTOMATIC_OPTIMIZATION]
                        If False you are responsible for calling .backward,
                        .step, zero_grad. Meant to be used with multiple
                        optimizers by advanced users.
  --hidden_dim HIDDEN_DIM
  --learning_rate LEARNING_RATE

Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
[ ] Did you make sure to update the documentation with your changes?
[x] Did you write any new necessary tests?
[x] Did you verify new and existing tests pass locally with your changes?
[x] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

[x] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Hello @louis-she! Thanks for opening this PR.

In the file tests/utilities/test_argparse_utils.py:

Line 6:5: E122 continuation line missing indentation or outdented
Line 41:5: E122 continuation line missing indentation or outdented

Do see the Hitchhiker's guide to code style


Close mobile sidebar when selected file

Close mobile sidebar when selected file

Thanks for the change @datlechin !


STORM-633. Nimbus - HTTP Error 413 full HEAD if using kerberos authentication.

+1 (since this is just a doc update, we probably don't need to wait for additional review)


Adapt KHeader parameters

Suggestion:
Rename cmodify_acq_info to ._apply
and move it to the acq info class.
It does pretty much the same as the _apply on a torch.module - apply a function to the tensor fields.
We could,to be more consistent,also name it apply as it does in place modification.
Soon(tm) we will most likely need this kind of apply also on the other objects for the fancy indexing..

Alternatively (or additionally) the modify acq info should be changed (dependence injections):
Move the is dataclass condition up.
Do the other path on else
Now, in the modify functions you can do a match case on what kind of object you get.
For most modifications, rotation should behave the same as a tensor -  and you can just index into it in the modify function.

(Ps: sorry for the spelling, hope you can guess what I try to say. I can only communicate using  my phone and one hand at most, as my main job still seems to be a heating pad.. )

The apply function works nicely. I moved it to DRAFT because we should first merge  #417 and then we can remove the special case for SpatialDimension in KDataSplitMixin and KDataSelectMixin

Something is not right with the orientation.
Siemens saves the orientation as a quaternion. This is then translated to read_dir, slice_dir and phase_dir in siemens_to_ismrmrd here:
https://github.com/ismrmrd/siemens_to_ismrmrd/blob/cda30e2f4fbace05dce666a781e20a11eb28c7e6/main.cpp#L1139C14-L1139C46
using
https://github.com/ismrmrd/ismrmrd/blob/4037c41fcdcb8870234baaeb46ce4848aac1579a/libsrc/ismrmrd.c#L667
To me it seems that the signs are not necessarily correct or I am misunderstanding what these directions are supposed to represent.

Something is not right with the orientation.
Siemens saves the orientation as a quaternion. This is then translated to read_dir, slice_dir and phase_dir in siemens_to_ismrmrd here:
https://github.com/ismrmrd/siemens_to_ismrmrd/blob/cda30e2f4fbace05dce666a781e20a11eb28c7e6/main.cpp#L1139C14-L1139C46
using
https://github.com/ismrmrd/ismrmrd/blob/4037c41fcdcb8870234baaeb46ce4848aac1579a/libsrc/ismrmrd.c#L667
To me it seems that the signs are not necessarily correct or I am misunderstanding what these directions are supposed to represent.

How and why did you notice? Where is the issue in your opinion?

How and why did you notice? Where is the issue in your opinion?

I noticed that when I do ismrmrd-slice-phase-readout-dir -> mrpro quaternion -> ismrmrd-slice-phase-readout-dir then I don't get the same result as I put in.
Siemens is a quaternion which corresponds to a rotation matrix with columns phase - readout - slice encoding. If I take that into consideration (latest commit) I get the same out as I put in from the ismrmrd file. But I guess this still does not fit to our slice - phase - readout convention...

Maybe we can discuss UserParameters again I we make misc more private and unsupported

I f'ed this up. sorry.
@ckolbPTB can you create a new PR from the unit_conversion_utils branch to main?

test

New PR: #506


Import Store Data option is missing inside the Modal after Reset App

Bug Description
After Resetting my App, I'd like to do either of these:

Create a new account (Already Present)
Add existing account (Already Present)
Import Store Data (Not Present) in case I have multiple accounts.

Steps to Reproduce

Open Account Manager
Click on DEV button on the right top corner.
Export Store Data ( Note: First of all export and save your data, otherwise I'll not be responsible for any loss.)
Now click on Reset App, this will erase all your info
New modal opens with the option to either create a new account or add a single signing key in case I already have an account signing key.

Expected behavior
Modal contains Import Store Data option also inside the Add Existing Account option
Actual behavior
Import Store Data option missing
Screenshots/Recordings

OS: Windows 10

Account Number
0f92e975101eb96cea80dce430d74efd3172fa229377818d467ded85dcf2ab6c

This issue can also be fixed by adding X (cross) to close the Get started modal. So that after closing that modal users can go to DEV and then import the data.


How to add listerner for TableView cell or row change ?

I want when i clicked TableView row or cell, something like rowchanged event, print
rowID of selected row or table cell content to console by sout statement.
Thanks

Hello, you can either add a ListChangeListener to table view items, or implement a custom cell factory that print the row index / item of the current cell.
Regards,

Thank's for your help
i can do by this code
    usersTreeTableView.setRowFactory(tv->{
        JFXTreeTableRow<UserRecursive> row = new JFXTreeTableRow<>();
        row.setOnMouseClicked(event -> {
                UserRecursive rowData = row.getItem();
                System.out.println(rowData.id);
        });
        return row ;
    });

Thank's for your help
i can do by this code
    usersTreeTableView.setRowFactory(tv->{
        JFXTreeTableRow<UserRecursive> row = new JFXTreeTableRow<>();
        row.setOnMouseClicked(event -> {
                UserRecursive rowData = row.getItem();
                System.out.println(rowData.id);
        });
        return row ;
    });


Documentation comparing Clump and Stitch

@williamboxhall This is the initial draft. Do you remember other main diferences?

Coverage remained the same when pulling c48a1f4bed628d28b9c4249a202405cc986eb83b on stitch-comparison into 3db1baf46a8ebd1ceada595651fefbee1f6ef46c on master.

maybe exists - doesn't exist :P

@williamboxhall The text is ready for review.

Coverage remained the same when pulling 0d8e7abe014ca756b4750554185dd1dffb2eafa0 on stitch-comparison into 41a33105bc3fcd3d58a6512b9a5856be9211d0f1 on master.

Coverage remained the same when pulling 06fe415d06584ddb8a3c99dd6b6376cd6327c049 on stitch-comparison into 41a33105bc3fcd3d58a6512b9a5856be9211d0f1 on master.

Looks really great man :+1:


[mirai] add tag annotations to libra-crypto crate

Motivation
This commit adds documentation to the code and helps reduce the false positives produced by MIRAI so that newly introduced problems would be easier to find. Some of the annotations make use of tag analysis to track validity of public keys at compile time. (Note that #3846 explained the importance of public key validation.)
Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan
cargo xtest libra-crypto

@hermanventer @huitseeker @mimoo This commit could be a first step to apply tag analysis.

A quick reply to the first two questions in https://github.com/libra/libra/pull/5525#pullrequestreview-464472325:

I expect tags on trait methods are nowhere near checkable yet, is that correct?

If you mean that we annotate trait methods with contracts like "the result must have a validated tag," I think currently a possible way is to use the design by contracts crate with the contract_trait attribute.

is it possible to run cross-crate checks yet?

Yes. I'm also trying to annotate and analyze some other crates that make use of libra-crypto, such as safety-rules.

A reply to the last two questions in https://github.com/libra/libra/pull/5525#pullrequestreview-464472325:

while I have you here, how about adding tag checks to each element of the public key iterator argument  of batch_verify?

Currently MIRAI does not support quantification in preconditions (e.g., "all the keys in the input have the Validated tag").
We could instead use verify! macros, but they would probably flag many false positives.

if we're being super ambitious, those checks straightforwardly extend to MultiEd25519PublicKey, and both modules could share the same set of tag declarations (see my suggestion to open a module for those in PR comments)

Added annotations to MultiEd25519PublicKey.
Regarding https://github.com/libra/libra/pull/5525#discussion_r468105758:
Added conditional compilation to guard the usage of const_generics by both mirai and debug_assertions.
Is it still necessary to add an extra crate feature like mirai-tag-analysis as a third condition?

:exclamation: Invalid command

Bors help and documentation

Bors is a Github App used to manage merging of PRs in order to ensure that CI is always green. It does so by maintaining a Merge Queue. Once a PR reaches the head of the Merge Queue it is rebased on top of the latest version of the PR's base-branch (generally master) and then triggers CI. If CI comes back green the PR is then merged into the base-branch. Regardless of the outcome, the next PR is the queue is then processed.
General

This project's Merge Queue can be found here.
This project requires PRs to be Reviewed and Approved before they can be queued for merging.
Before PR's can be merged they must be configured with "Allow edits from maintainers" enabled. This is needed for Bors to be able to update PR's in-place so that Github can properly recognize and mark them as "merged" once they've been merged into the upstream branch.

Commands
Bors actions can be triggered by posting a comment which includes a line of the form /<action>.

Command
Action
Description

Land
land, merge
attempt to land or merge a PR

Canary
canary, try
canary a PR by performing all checks without merging

Cancel
cancel, stop
stop an in-progress land

Cherry Pick
cherry-pick <target>
cherry-pick a PR into <target> branch

Priority
priority
set the priority level for a PR (high, normal, low)

Help
help, h
show this help message

Options
Options for Pull Requests are configured through the application of labels.

          Option
Description

Indicates that the PR is high-priority. When queued the PR will be placed at the head of the merge queue.

Indicates that the PR is low-priority. When queued the PR will be placed at the back of the merge queue.

Before merging the PR will be squashed down to a single commit, only retaining the commit message of the first commit in the PR.

@bors-libra /land

:exclamation: Invalid command

Bors help and documentation

Bors is a Github App used to manage merging of PRs in order to ensure that CI is always green. It does so by maintaining a Merge Queue. Once a PR reaches the head of the Merge Queue it is rebased on top of the latest version of the PR's base-branch (generally master) and then triggers CI. If CI comes back green the PR is then merged into the base-branch. Regardless of the outcome, the next PR is the queue is then processed.
General

This project's Merge Queue can be found here.
This project requires PRs to be Reviewed and Approved before they can be queued for merging.
Before PR's can be merged they must be configured with "Allow edits from maintainers" enabled. This is needed for Bors to be able to update PR's in-place so that Github can properly recognize and mark them as "merged" once they've been merged into the upstream branch.

Commands
Bors actions can be triggered by posting a comment which includes a line of the form /<action>.

Command
Action
Description

Land
land, merge
attempt to land or merge a PR

Canary
canary, try
canary a PR by performing all checks without merging

Cancel
cancel, stop
stop an in-progress land

Cherry Pick
cherry-pick <target>
cherry-pick a PR into <target> branch

Priority
priority
set the priority level for a PR (high, normal, low)

Help
help, h
show this help message

Options
Options for Pull Requests are configured through the application of labels.

          Option
Description

Indicates that the PR is high-priority. When queued the PR will be placed at the head of the merge queue.

Indicates that the PR is low-priority. When queued the PR will be placed at the back of the merge queue.

Before merging the PR will be squashed down to a single commit, only retaining the commit message of the first commit in the PR.

/land

@huitseeker :exclamation: This PR is still missing approvals, unable to queue for landing

/land


Compatibility with python 3.12

Hey, nice project =)
I noticed that the numpy version for this project is not compatible with newer python versions (3.12)
The current numpy version for this project (1.24.4) tries to access the distutils folder from python, but it was discontinued in 3.12:
"It was deprecated in Python 3.10 by PEP 632 “Deprecate distutils module”. For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils." What is new? (python 3.12)
When I try to install through poetry I get the following error:

It would be nice to have support to newer versions =)

@leomariga Update on this issue: 3.12 is now supported as part of ASB v4.2.2, live on PyPI!
pip install --upgrade aerosandbox[full]


3273 galxe campaign subgraph cred deployment

Description

Type of change

[ ] Docs change / dependency upgrade
[ ] Configuration / tooling changes
[ ] Refactoring
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Requires changes in customer code

High-level change(s) description - from the user's perspective

Related Issue(s)
Fixes
Related pull request(s)

This overlaps with: #3379

It overlaps quite a bit however there are some fundamental differences. This PR stores a unique LP entity with an increasing/decreasing balance based on adding or removal. The other PR simply stores all events. We might need both at the end of the day. I am working with @liu-zhipeng on reconciling these.

This will be integrated with #3379


Document and code are not synchronized！

For example, writing according to the document, Custom Validators Section is not correct.
Following is the right code for importing Custom Validators
import {withParams} from 'vuelidate/lib/params'
export const isHello = withParams({type: 'isHello'}, value => value === 'hello')

This didn't work for me. I am kind of clueless about how to create custom validators.


Can not create secret

Component
web-ui
Describe the bug
Creating a new secret using the web-ui results in the following error: : user not authorized
Can anyone confirm this?
Steps to reproduce

Log in the web-ui
Go to settings -> secrets -> add secret
add a secret, e.g. name mail and some random value like abc
click add secret
see error message : user not authorized. Also, the secret is not added.

Expected behavior
no error message, adding a secret works
System Info
`{"source":"https://github.com/woodpecker-ci/woodpecker","version":"2.8.0"}`

Additional context
No response
Validations

[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]

Can anyone confirm this?

I can't on next-21755bef4e

After trying a few times on different days, it is like the behavior @zc-devs mentioned. Works sometimes, sometimes not.

Closing, as i have no access to server logs


References

This is an obvious but probably the single most important issue. When writing any text using external sources we should cite it and add it to the references. While we can just add the citations at the end of writing I think it will be much much easier if we do it while writing since it will be much more difficult to do it later. At the very least we should keep the references saved somewhere with a note in which section and paragraph they were used. For now we can store them locally for individual use but we can try to think of a better way to store them so that we all have access to them.
I've noticed that the section "Introduction/Reinforcement Learning" has already been written but there are no citations or references. I don't know whether they were simply not added yet or if the text does not require them but since we haven't explicitly talked about this yet I wanted to make sure.
Also another obvious reminder -- don't copy/paste entire articles from other sources especially without citation.

TODO: add references to environments' documentation when describing third-party environments


View full message

https://github.com/Start9Labs/embassy-os/issues/888

Did this ever get merged?


allow DSTs in ghost code (prototype)

Context
Since we use Rust's typechecker, we have to obey its requirements regarding Sized. However, these requirements stem from the need to compile the code, so in principle, it should be fine to remove such requirements in ghost code.
Currently, ghost code does invoke the usual rustc Sized checks, after which the Sized checks are then ignored in VIR. The result is a bit of a mismatch between the source code and the underlying logic. More precisely, there are a number of problems:

If the user wants to make their own pointer type, e.g., MyRc<V> with spec fn view(&self) -> V, it will be impossible to instantiate the pointer type with a DST.
Since Sized is used everywhere in ghost code, even where it's not necessary, it would be wasteful to encode the Sized trait bound - and therefore, currently, we don't. However, this means that we can't write any proof lemmas or axioms that rely on a Sized bound (like V: Sized ==> size_of<&V> == word_size). The currently accepted workaround proposes that we create a new concept to be mean "type is sized" that is encoded. However, the result is confusing to work with.
You have to add Sized bounds for trait params, which isn't terrible but is slightly annoying when the trait is only for spec purposes.

There's also some more motivation in this discussion by @jaybosamiya: https://github.com/verus-lang/verus/discussions/377
This PR allows DSTs in ghost code. I propose we make ?Sized the default (e.g., through the macro) in ghost code.
Implementation
Unfortunately, even with all the unstable features currently available, there's no way to turn off all the Sized requirements (.e.g, that return values cannot be Sized). I considered a number of hacks that involved query-overriding, but they didn't pan out.
Soooo ... I'm proposing to fork the rustc_hir_typeck crate, as done in this PR. I mean, we've already forked 2 crates, what's 1 more?
Due to rustc's query framework, it's pretty easy to inject the modified verus_rustc_hir_typeck. The only modification I have to make to the crate is deleting the lines of code related to Sized checks. We don't even have to distinguish ghost from exec code here, because we'll end up using the unmodified typechecker when compiling the code anyway.
(also this would let us fix that annoying as nat error message)
Todo

[ ] make ?Sized the default in ghost code (by adding ?Sized bounds with the macro)
[ ] ensure the providers are injected for the non-erased pass and the lifetime pass, but not for the erased executable pass
[ ] tests

At a high level, I'm very reluctant to fork rust_hir_typeck: we would become even more dependent on its internals, and on the way the relevant queries are defined (which may change); it would also mean that updating to newer rustc's requires a significant rebase (where the diff we need may be significantly different as a result of refactors in rust_hir_typeck).

we've already forked 2 crates,

These are small crates, and it was in part a way to speed up the move to unmodified rustc: ideally I'd prefer we found a way to remove the need for these forks too, rather than forking more.

@tjhance are you okay with me closing this for now?


How to set os.environ[DEM_PATH] correctly?

In FAQ I read

if I want to use the default setting as the DEM of SNAP, how can I find the specific path?

If you want to use i.e. COPDEM-30, just type:
import os
os.environ[DEM_PATH] = "Copernicus 30m Global DEM"

But I just saw that the default is not GETASSE30 anymore (it was when COPDEM-30 wasn't existing or was buggy).
I will change that 😄
You can find that piece of code with a custom DEM [here](https://eoreader.readthedocs.io/en/v0.17.0/notebooks/SAR.html#load-bands)

I updated the documentation of SAR notebook to add a paragraph about your question 😉

To load DEM-related bands, you need to provide a DEM on your own as specified here.
(I cannot use SNAP DEMs as the software downloads the tiles on the spot...)
You may have more information about that in the DEM notebook.

However, your current bug seems to come from another source, could you create another issue with full log trace ?

OK, I create a new one right now.


⚠ Bofrost.de has degraded performance

In be831da, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1483 ms

Resolved: Bofrost.de performance has improved in 0e6d2ec.


Add documentation for series-file path for influx_inspect dumptsi

URL for relevant page?
https://docs.influxdata.com/influxdb/v1.5/tools/influx_inspect/#influx-inspect-dumptsi
What products and version are you using?
1.5.2
The series-file parameter is required in order for the dumptsi command to work.  Add documentation to describe that parameter and it's function.

Fixed in this PR https://github.com/influxdata/docs.influxdata.com/pull/1627


[NTR] colon macrophage

Note - please check that the term does not already exist (check OLS: https://www.ebi.ac.uk/ols/ontologies/cl)
Preferred term label:
colon macrophage
Synonyms
macrophage of colon
Definition (free text, please give PubMed ID in format PMID:XXXXXX)
Macrophage that is located in the colon.
Parent cell type term (check the hierarchy here https://www.ebi.ac.uk/ols/ontologies/cl):
macrophage http://purl.obolibrary.org/obo/CL_0000235
What is the anatomical structure that the cell is a part of? Please check Uberon: https://www.ebi.ac.uk/ols/ontologies/uberon
colon http://purl.obolibrary.org/obo/UBERON_0001155
Your nano-attribution (ORCID)
0000-0003-3440-1876
Any additional notes or concerns

Note to self: 'appendix macrophage' currently reasons under 'colon macrophage', because CL hasn't ingested the Uberon change by which appendix is no longer part_of colon (because Uberon hasn't released yet).

[ ] Double-check after Uberon release.

@sfexova
CL_0009038 (colon macrophage)
(leaving ticket open until https://github.com/obophenotype/cell-ontology/issues/1123#issuecomment-873976856 is addressed)


Corrupt entries in FS on Windows Side

I have a Windows directory that I, in my infinite wisdom, have shared between Windows, Linux and WSL.
Recently I got this-
[~/.ivy2/cache/com.typesafe.play/twirl-parser_2.10/docs]ls -al
total 215M
drwxrwxrwx 0 root root 4.0K Aug 22  2016 .
drwxrwxrwx 0 root root 4.0K Mar 16 00:16 ..
-rwxrwxrwx 1 root root 423K Aug 21  2016 twirl-parser_2.10-1.0.2-javadoc.jar
[~/.ivy2/cache/com.typesafe.play/twirl-parser_2.10/docs]du -h
213M	.
[~/.ivy2/cache/com.typesafe.play/twirl-parser_2.10/docs]

I'm noting this in case you guys want to look into it, I currently have no problems. This btw, happened when i was running on WSL. The shared directory's actual location is Windows.

@SRGOM - will you please fill out our issue template?

@benhillis Oops updated.
On second thoughts, while I'm  almost certain that the directory was never inside lxss folder, I don't know if this issue helps you in any way since the instructions to reproduce are absent. Feel free to close if this is distracting. I only wanted to note so you can keep an eye out.

I think they need to be able to reproduce it.


[decomp] make bg decompile

Pull Request Test Coverage Report for Build 969580650

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.0006%) to 69.34%

Totals

Change from base Build 966606184:
0.0006%

Covered Lines:
36664

Relevant Lines:
52876

💛  - Coveralls


Specify transform in package.json for browserify support

If you use the "browserify.transform" feature, it will make it easier for those who also use browserify at the app level to use your npm package. The two changes required to your package.json are: add a dependency on reactify (you currently only have a dev dependency), add at the below line.
"browserify": { "transform": [ ["reactify",{"es6":true}] ] }

You may also wish to consider using babelify instead. It is based on the babel project which is replacing the react-tools project that reactify is based on. If you do, you can use this line instead (as well as a dependency on babelify):
"browserify": { "transform": [ "babelify" ] }

Great idea. I shall update and commit in the morning. I thought I was using babelify, weird. I shall also switch that out too!
Thanks for the tip.

Turns out, by doing both #12 and #13, it now does extra unnecessary work. Since you targeted the precompiled code using main in package.json, browserify no longer needs babelify, but still applies it due to #12. I suggest removing the browserify.transform field in package.json.


Added organisational unit administration

Assigning to @simong for review

When you:

click jeditable A (don't change anything)
click jeditable B (don't change anything)
click jeditable A again

you get a notification saying "Organisational unit successfully updated"

When you update a part the notification should say "Part updated", similarly for subjects and courses.
Ideally, we don't expose the "organisational unit" terminology anywhere

When you create a part under a subject we can just hide the select box as it only contains the part value anyway

Follow-up done, back to @simong for review

Good stuff, this'll really help helpdesk. 1 thing remaining really

One smallish thing: If you rename the course aaa to Yeah it will be sorted to the bottom. Maybe the scroll should be adjusted as well?

Follow-up done. Back to @simong

LGTM, I'll be merging this all into our grasshopper-ui repository later


ArkOS Beta RG351MP - Incorrect battery indicator

At first I thought it was a problem with my console but then I realized that what happens is that the indicated battery % is not correct in the ArkOS beta. I put the original SD with EmuELEC and it shows 100% battery, while with ArkOS it is "stuck" at 33.

%

In the latest update for this beta, we're attempting to use the internal power management controller to give a more accurate measure of battery.  This works well for the OGA devices and related clones.  It should hopefully work well for the Anbernic devices as well.  What has to happen so that the controller understands the correct parameters of the battery, you need to do a complete drain of the battery then recharge to full in order for the battery indicator to be correct using the internal power management controller.
The battery measure used in the stock card as well as what was provided in source goes by voltage which is wildly incorrect as witnessed with the RG351P/M/V.  For instance, if you drain the battery on stock then plug it into a power source, it will immediately show 100%.

@christianhaitian
OS: ArkOS 2.0 Test 1.7
I tested the problem with displaying the battery charge for several days, here are my observations.

I tried to completely discharge the battery, and then fully charge it - the charge display is stuck at 22%.
To fix the display functionality, I had to disconnect the battery and reconnect it. Charge display 100%.
After that, the charge only drops. For example, I discharged the console to 50%, then I charge it for a couple of hours, the charge will remain at 50%.

Thanks for this feedback.  I just released update 1.8 that will revert the battery measurement back to the same method used for the other rg351 devices.  It may not be accurate but it seems to be the best solution available for now.


Control Plan Count Variable control_plane_vm_count addition to variables.tf for AWS

Signed-off-by: archups<EMAIL_ADDRESS>
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
To handle Control Plane count via terraform variables for AWS and refer to this variable value instead of the hardcoded count = 3 in the main.tf

Does this PR introduce a user-facing change?:

None

/assign @xmudrii @kron4eg


[Sema] Preserve compatibility for weak self with @_implicitSelfCapture

In 5.10 we warned on this:
func bar(@_implicitSelfCapture _ fn: @escaping () -> Void) {}

class C {
  func foo() {
    bar { [weak self] in
      foo()
    }
  }
}

But with the implicit self rework (#70575), this accidentally became an error. Fix it to ensure we continue to warn until Swift 6 mode.
rdar://128941797

@swift-ci please test source compatibility

Source compat is a UPASS, which is expected, that's what uncovered this issue

@swift-ci please test


Avoid importing yt unnecessarily

See commits

Coverage decreased (-0.05%) to 74.774% when pulling 5aaff45c712b0d7f4e7b3324d1c2ef553e003baa on keflavich:yt_import into 9818f02c126af5f7e94030ee93f73919807b8112 on radio-astro-tools:master.


FlatList with column does not display completely the last item of each row

Please provide all the information requested. Issues that do not follow this format are likely to stall.
Description
My issue is that I have a screen with margin on each size (about 24). The screen contain a FlatList with 3 columns,
each items have a size of Dimensions.get('screen').width / 3 and a height of 105. And the box in an item have a size of 105 on 105.
I have 2 same flatlist with a view between them to separate (design issue).
On android it fit perfectly but on some iOS device the last item of each row is cropped.

React Native version:
System:
OS: macOS 11.2.3
CPU: (8) x64 Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz
Memory: 1.03 GB / 16.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 14.13.0 - /usr/local/bin/node
Yarn: 1.22.4 - /usr/local/bin/yarn
npm: 6.14.8 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
Managers:
CocoaPods: 1.10.1 - /usr/local/bin/pod
SDKs:
iOS SDK:
Platforms: iOS 14.4, DriverKit 20.2, macOS 11.1, tvOS 14.3, watchOS 7.2
Android SDK: Not Found
IDEs:
Android Studio: 4.2 AI-202.76<IP_ADDRESS>51085
Xcode: 12.4/12D4e - /usr/bin/xcodebuild
Languages:
Java: 15.0.2 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.1 => 17.0.1
react-native: 0.64.0 => 0.64.0
react-native-macos: Not Found
npmGlobalPackages:
react-native: Not Found
Steps To Reproduce

Create a flatlist
Set each item width to Dimensions.get('screen').width / 3 and a height of 105
Set a child in the item with a size of 105/105

Expected Results
I would like the last box of each row to be perfectly displayed on Iphone 12 emulator
Snack, code example, screenshot, or link to a repository:
The render for the flatlist look like this :
<View style={{ width: Dimensions.get('screen').width, height: 105 }}>
    <View style={{ width: 105, height: 105 }}>
    </View>
 </View>

The FLatList have no style only the col={3} props and keyExtractor.

The reason you are having issues is because you are using defined widths and different devices are going to have different widths. You should use Layout with Flexbox attributes instead.


CreateDistrict modal shows up after modifying district name

After clicking on "Save" in the Modify district screen, create district modal shows up.

Fixed this. The state for the modal was getting toggled in the modify district action creator. Removed it.


lecture04

made by Rakhlinskii Vlad

Coverage decreased (-3.1%) to 46.541% when pulling c11ff4ce56742508e127d14861426fc8e6de136a on vic0din:lecture04 into c0938b61d9a9c2f82f0ba2dfb4dc492717119c93 on rybalkinsd:lecture04.

не всё -+


feat: Add Home button after quiz ends

Related Issue
When any quiz ends, the user doesn't get any navigation ideas, so a home button has been added.
Fixes: #574
Describe the changes you've made
Added a Home button after quiz ends
Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.

Screenshots
Original

Updated

@Amit366 kindly review my PR, before the project submission deadline. Thanks :D


Reinstall ADE extension as last step of the restore process

Current:
"For Linux VMs, reset the ADE extension so the data disks are open and mounted."
Please change the word reset to reinstall to avoid confusions, customers are confused with the use of the word reset:
"For Linux VMs, reinstall the ADE extension so the data disks are open and mounted."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 54252e63-be57-2c49-15fc-b9dc7fdda1b3
Version Independent ID: 30777b2a-604d-ce18-c891-4d272632f039
Content: Back up and restore encrypted Azure VMs - Azure Backup
Content Source: articles/backup/backup-azure-vms-encryption.md
Service: backup
GitHub Login: @dcurwin
Microsoft Alias: dacurwin

#assign:dcurwin

Thanks for the correction. The document has been updated, and should be published soon.
#please-close

@dcurwin Thanks for super quick response. 👍


Would be nice to have .navigatorKey on VRouter

It seems we need to use VNester in order to pass a nav key to the VRouter, is this the case? If so, would be nice to be able to pass it to the top level VRouter element.
Additionally, there seems to be some delay before the Navigator is created. I would like to use the key to gain access to a context that can be used to showDialogs globally. Something like:
final GlobalKey<NavigatorState> _navigatorKey = GlobalKey();
initState(){
  scheduleMicroTrack(() => print(_navigatorKey.context))
}

build => VRouter(routes: [
            VNester(
              path: null,
              navigatorKey: _navigatorKey,
              nestedRoutes: [VWidget(path: ViewRoutes.Splash.path, widget: SplashView())],
              widgetBuilder: (c) => c,
            ),)

Is there another better way to get a context that is below the Navigator/Overlay?

Essentially I want to be able to call showDialog() from any of my views, without having to actually pass context along for the ride.
I'm having trouble actually getting that context when using VRouter. Passing in a key to VNester seems to work, but is easily broken, as not all routes are inside my Nester (for example, SplashPage is not nested by the shared app scaffold)
If the first route shown is not inside a Nester, it seems the nav key is not assigned, and I get a runtime exception.

Several things:

Not being able set the root navigation key is a mistake. I did have this option before and must have missed it during some refactoring
Using showX is a bit of an anti pattern is not recommanded (even if it's fully supported), see this comment and discussion if you are interested

I will fix point 1 though, I'll update you here!

Thx! Not sure I would call it an anti-pattern

I don't actually want most of my dialogs to be associated with a route name
Most users in web don't expet that Backbtn closes dialogs
It's more boilerplate, especially when doing common things like OkCancel dialogs that have a response, nicer to just roll with the devil we know here.

It's certainly a cool pattern though and we'll keep it in mind!

navigatorKey can now be specified in as of 1.1.4+7.
You are right, anti-pattern is not the right word, there are definitely valid cases! From my point of view I think a Dialog is ok, but a huge BottomSheet containing information should be part of the path.
Feel free to close if navigatorKey works as intended after your tests !

Thanks @esDotDev for the issue and @lulupointu for looking into adding navigatorKey for I was also intending to showDialog globally.
However, according to my experience, the navigatorKey does not seem to work to my intentions with it. At where the navigator key was being accessed for showing a dialog, the currentState and currentContext turned out to be null.
Any advice to this? Any response would be appreciated.
Just for your information, here below are the widgets, keys and the VRouter configuration that were used in my testing:
widget_keys.dart
class WidgetKeys {
  static final navKey = GlobalKey<NavigatorState>();
}

main.dart
void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return VRouter(
      debugShowCheckedModeBanner: false,
      navigatorKey: WidgetKeys.navKey,
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      routes: [
        VWidget(
          path: '/home',
          widget: MyHomePage(),
        ),
        VWidget(
          path: '/anotherPage',
          name: 'anotherPage',
          widget: AnotherPage(),
        ),
        VRouteRedirector(
          path: ':_(.*)',
          redirectTo: '/home',
        ),
      ],
    );
  }
}

home_page.dart
class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key}) : super(key: key);

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Home Page'),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            TextButton(
              child: Text('Next page!'),
              onPressed: () {
                context.vRouter.pushNamed('anotherPage');
              },
            ),
          ],
        ),
      ),
      floatingActionButton: ShowDialogFab(),
    );
  }
}

another_page.dart
class AnotherPage extends StatelessWidget {
  const AnotherPage({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Another Page'),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            TextButton(
              child: Text('Go home!'),
              onPressed: () {
                context.vRouter.push('/home');
              },
            ),
          ],
        ),
      ),
      floatingActionButton: ShowDialogFab(),
    );
  }
}

show_dialog_fab.dart
class ShowDialogFab extends StatelessWidget {
  const ShowDialogFab({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return FloatingActionButton(
      onPressed: () {
        final navKeyContext = WidgetKeys.navKey.currentState!.overlay!.context;

        showDialog(
          context: navKeyContext,
          builder: (context) => AlertDialog(
            title: Text('I am a dialog'),
            content: Text('Triggered with NavKey context'),
            actions: <Widget>[
              TextButton(
                child: Text(
                  'OK',
                ),
                onPressed: () => Navigator.of(context).pop(),
              ),
            ],
          ),
        );
      },
      tooltip: 'Trigger alert dialog',
      child: Icon(Icons.notifications),
    );
  }
}

That's because I'm stupid 🤦
Anyway that's fixed in 1.1.4+9
Also you can just use WidgetKeys.navKey.currentContext! no need to go after the overlay.

That's fast!
Alright, I can confirm that using showDialog globally with navigator key works!
Big up to you, overall I really like this library! @esDotDev please confirm if it worked for you too 😄

Works great, ty!


[MINOR][SQL] Mark StreamSinkProvider and StreamSourceProvider as stable

What changes were proposed in this pull request?
This change marks the StreamSinkProvider and StreamSourceProvider
interfaces as stable so that it can be relied on for compatibility for all of
Spark 2.x.
These interfaces have been available since Spark 2.0.0 and unchanged
since Spark 2.1.0. Additionally the Kafka integration has been using it
since Spark 2.1.0.
Because structured streaming general availability was announced in
Spark 2.2.0, I suspect there are other third-party integrations using it
already as well.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hi, @granthenke . Please file an Apache SPARK JIRA issue. This is worth for that.
And, you had better send<EMAIL_ADDRESS>with that JIRA issue link.

cc @tdas and @zsxwing

Thank you @dongjoon-hyun. I have created a Jira, updated the subject here, and sent a summary email to the dev mailing list.

If I understand the progress of DatasourceV2 it sounds like there was a bit of a redesign and that DatasourceV2 wouldn't be available until Spark 3.0.0. If that is the case marking this as stable in the mean time should be minimal impact. Additionally using new interfaces for DatasourceV2 would allow for a much smoother transition for those who adopted structured streaming early.
My context into DataSourceV2 comes from these locations:

Pull Request: https://github.com/apache/spark/pull/22547
Umbrella Jira: SPARK-25390
API Design Doc: here

Perhaps @cloud-fan could comment.

AFAIK DSV2 could effect 3.0 only. On rest of branches can be done, though can potentially make backports harder.

Another reason to use new Interfaces for DatasourceV2 is precisely to make any backports of bugs in earlier code easier.

Since there won't be a 2.5.x release, I don't think changing this make any sense. We don't change APIs between maintenance release (such as 2.4.1).

Thanks @zsxwing, thats good to know.
Note: The Unstable annotation does imply that it can be changed between maintenance releases. It implies this because the Evolving annotation claims it can only change from one feature release to another and unstable is more permissive than that. Perhaps the documentation could be updated.

@zsxwing Would you mind closing the Jira as Wont-Fix and add a comment with this reason?

@granthenke . So, if you want to stability only on branch-2.4, yep. +1 for closing this PR and JIRA issue.

@dongjoon-hyun I want it for anything in the 2.x series. This is what the stable annotation would indicate. Since @zsxwing said there would be no 2.5 and maintenance releases won't break it, it sounds like that is the case.

Thanks for the confirmation. Then, I'll close this for you. :)


Fork node-slug and rename to mollusc

So I just messed up. Ignore this.
That said, I've forked this project to mollusc. For those who are interested, I am currently triaging and merging PR's.

So I just messed up. Ignore this.

@simison I see, you were just referencing it.  The real author is @Zertz


gnome.evince: Fix cross

Description of changes
Needs build -> build & build -> host pkg-config.
Closes #259759, CC @evrim for testing
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux (cross)
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@OfBorg build pkgsCross.aarch64-multiplatform.gnome.evince


[Text Shortcuts] Incorrect List ordered: all the lines are 1

Extension
https://www.raycast.com/koinzhang/text-shortcuts
Description
This command all the time works non-properly: when I select the text and try to make it "List ordered", all the lines start from "1." and not 1., 2., 3. etc.
Who will benefit from this feature?
No response
Anything else?
No response

@raycastbot Please assign this issue to me.


Fixed overflow issure, I think deep clone operation one time is en…

…ough, what do you think?

A model will probably contain a number of mesh objects, this break would result in only one of those being copied over, would it not?

Also, it's lacking {} braces around that if statement which would actually cause it to exit out after the first iteration.

You are right, that is my misunderstanding.


Implemented counting sort

Implemented counting sort algorithm, placed it under "Sorting"

Thanks for contributing.
For the next time, can you run gulp or gulp build before you push it?
Since files in /public directory are automatically built, they should not be changed unless you changed stylesheet or javascript files.

Sure thing. I tried doing that the past couple of times, but it looks as though 15+ files are changed after I run gulp even though I didn't touch them. Am I supposed to commit all of those modified files every time I make a change?

Never mind, this may not be your fault.
I will figure out how to automatically build those files when merging.

@parkjs814  https://libraries.io/npm/javascript-gulp-precommit
not sure but might help

Actually gulp should build after we merge the pull request, not before pushing it.
I don't know what to do hmm..

@parkjs814 We could use travis.

@TornjV Isn't travis only for the deployment to server?

@TornjV might be on to something: https://gist.github.com/domenic/ec8b0fc8ab45f39403dd

Mhm that can be one possible solution.
I guess it is time to move to AWS or Heroku, there are too many limitations as well as this issue.

@parkjs814, so for future commits, should I run gulp before pushing, or just push files that I explicitly changed?

@jarettgross,
I figured out that there will be no difference whether you run gulp or not.
Don't worry about this issue, and just send a pull request.
I will care about it.
Sorry for the inconvenience.


Zoomed image isn't displayed the second time

Found this while testing the app on iPhone X simulator, it seems that ios fails the second time you tap a ZoomedImage (from a PR or a profile screen).
I was also able reproduced it on iPhone 7 and 8 simulators.
Can someone with a real device confirm this?
Error logged the second time:
2017-09-19 20:48:04.617844+0100 GitPoint[24593:16853987] [Unknown process name] CGAffineTransformInvert: singular matrix.

Found this freshly opened PR that fixes the bug in react-native-photo-view repo : https://github.com/alwx/react-native-photo-view/pull/103

I did this component, then I did not have the opportunity to check on iOS, although it worked, but probably no one checked if the image opens the second time, but it's good that there is a PR that fixes it, only the author of this library does not actively take PR's, so we'll have to wait, or we can make a fork of this library..

Merged \o/ https://github.com/alwx/react-native-photo-view/commit/3d36136ca2cd4c39270541ffa7e951a8eed92008
Fix should be present in 1.5.1

Oh, this is very cool, then we just need to update the version of this dependency.


[JS, Dotnet] Choice isEnabled + inlineAction, Input.File, Input.Color

Related Issue

Fixes #5852

Description

[JS, Dotnet] Add support for isEnabled property on Choice in ChoiceSetInput
[JS, Dotnet] Add support for inlineAction (ToggleVisibility) inside Choice in ChoiceSetInput (multiSelect and unique selection)
[JS, Dotnet] Add support for Input.File
[JS, Dotnet] Add support for Input.Color
[JS] Updated submit action result on adaptivecards-designer-app

Please note that i used base64 string to send the file, i'm pretty sure i'm wrong on that, so please, may i have an explanation on how you see how it should be done. I have updated the dotnet package because i need it to be accessible on a BotFramework chatbot app. Also, i don't know if i did everything that is expected for pull-request like this one, please notify me if anything is wrong and how i should do it to satisfy requirements, maybe i can have some information on how to make a way for these updates to go through WebChat and BotFramework Emulator ?
Sample Card

Sample card for unique selection expanded or compact :

{
  "type": "AdaptiveCard",
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
  "version": "1.3",
  "body": [
    {
      "id": "ChoiceSet",
      "type": "Input.ChoiceSet",
      "choices": [
        {
          "title": "Choice1",
          "value": "Choice1",
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "FileInput",
                "isVisible": true
              },
              {
                "elementId": "ColorInput",
                "isVisible": false
              }
            ]
          }
        },
        {
          "title": "Choice2",
          "value": "Choice2",
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "ColorInput",
                "isVisible": true
              },
              {
                "elementId": "FileInput",
                "isVisible": false
              }
            ]
          }
        }
      ],
      "value": "",
      "isMultiSelect": false,
      "style": "expanded"
    },
    {
      "id": "FileInput",
      "type": "Input.File",
      "isVisible": false
    },
    {
      "id": "ColorInput",
      "type": "Input.Color",
      "isVisible": false
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "title": "Submit"
    }
  ]
}

Rendering :

Sample card for multiselect :

{
  "type": "AdaptiveCard",
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
  "version": "1.3",
  "body": [
    {
      "id": "ChoiceSet",
      "type": "Input.ChoiceSet",
      "choices": [
        {
          "title": "Choice1",
          "value": "Choice1",
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "FileInput",
                "isVisible": true
              }
            ]
          }
        },
        {
          "title": "Choice2",
          "value": "Choice2",
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "ColorInput",
                "isVisible": true
              }
            ]
          }
        }
      ],
      "value": "",
      "isMultiSelect": true,
      "style": "expanded"
    },
    {
      "id": "FileInput",
      "type": "Input.File",
      "isVisible": false
    },
    {
      "id": "ColorInput",
      "type": "Input.Color",
      "isVisible": false
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "title": "Submit"
    }
  ]
}

Rendering :

Sample card with isEnabled on Choice :
{
  "type": "AdaptiveCard",
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
  "version": "1.3",
  "body": [
    {
      "id": "ChoiceSet",
      "type": "Input.ChoiceSet",
      "choices": [
        {
          "title": "Choice1",
          "value": "Choice1",
          "isEnabled": false,
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "FileInput",
                "isVisible": true
              }
            ]
          }
        },
        {
          "title": "Choice2",
          "value": "Choice2",
          "inlineAction": {
            "type": "Action.ToggleVisibility",
            "targetElements": [
              {
                "elementId": "ColorInput",
                "isVisible": true
              }
            ]
          }
        }
      ],
      "value": "Choice1",
      "isMultiSelect": true,
      "style": "expanded"
    },
    {
      "id": "FileInput",
      "type": "Input.File",
      "isVisible": true
    },
    {
      "id": "ColorInput",
      "type": "Input.Color",
      "isVisible": false
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "title": "Submit"
    }
  ]
}

Rendering :

How Verified
I have tested it in using adaptivecards-designer-app for nodejs package, AdaptiveCards.Sample.WPFVisualizer for the dotnet package and also tried with local npm package in WebChat and BotFramework Emulator local execution.
Microsoft Reviewers: Open in CodeFlow

@TemoeBOUISSOU - thanks so much for the PR! In order to take new features, we need to understand how the features will work on every platform (Desktop, Mobile, etc) as well as understand the schema changes that drive the features. Can we engage further to spec out how each feature will work on every platform and go through our process to get these to move forward? Also, where are you looking to have these features rendered (ex: a custom app? Teams? Outlook?)?
I see that our contribution guidelines are not well documented and are not very clear, so I apologize for that. I'll work on clarifying those to make sure this is clarified going forward.

Just to clarify, for now, i have updated only Dotnet (Libraries + WPF visualizer) and nodeJs (adaptiveCards + designer but only submit action result). These were parts that i was needing to update to make these behaviours working on WebChat and BotFramework-Emulator. For the needs, i need it to be available on BotFramework-Emulator in order to have a development environment ready for these behaviours, and also, i need it on WebChat because i am using BotFramework SDK (and so adaptivecards dotnet library) on a Dotnet server to expose a bot building cards with these behaviours.
I hope that i was clear enough on that part, do not hesitate if you have any other question. I was not planning on updating every platform as i was not aware on the dev process, but i can manage to make it working for every platform if needed. I didn't look at all projects inside the repository for now, so from here and what i have explained here, can you confirm what projects are missing in order to make the PR fulfill my needs and requirements.

@TemoeBOUISSOU quick update - we're still discussing this and will get back to you next week.

Hello, i was in holidays i couldn't work on it lately, i'm sorry for that ! Thank you for your answers, i tried to fix things like you explained but i'm having trouble on the isDesignMode method from Choice class.
I have found where the problem is but don't know how i should manage that problem. I would have to change something that is applied generically to all Action :

That is why i let the isDesignMode implementation on the Choice class for now otherwise i wouldn't be able to test my code because of the error.

@TemoeBOUISSOU ah, I see. The problem actually goes well beyond isDesignMode(). ActionProperty.parse currently assumes that the object (sender) on which the action is being parsed is a CardElement instance, which is no the case here (it's a Choice). It is necessary for an Action instance to have a parent of type CardElement for things to work properly. Choice is not a CardElement, so the let parent = <CardElement>sender; cast in ActionProperty.parse actually hides a bigger bug where the parsed action will have a parent of type Choice.
Your initial implementation worked because:

You artificially added the isDesignMode() method on Choice
Your handler for Choice.inlineAction duplicated the ToggleVisibility logic and was able to get to the target elements via the ChoiceSet's parent

But that's a workaround, we need an actual solution. That solution likely involves:

Updating the Choice class so it has an owner property of type ChoiceSet
Making sure that all choices added to a ChoiceSet (either during parsing or in code via the API) have their owner property set properly to that ChoiceSet. That probably means changing the actual type of ChoiceSet.choices from Choice[] to a class that extends Array<Choice> and handles setting owner while maintaining backward compatibility with the current model in order to not break existing applications
Not using the ActionProperty class and instead creating a new ChoiceInlineActionProperty whose parse method would look like this:

parse(sender: SerializableObject, source: PropertyBag, context: SerializationContext): Action | undefined {
    let parent = (<Choice>sender).owner;

    return context.parseAction(
        parent,
        source[this.name],
        this.forbiddenActionTypes,
        parent.isDesignMode());
}

That last bit isn't as generic as I would like, but it's clean. There might be a better way.

@TemoeBOUISSOU ah, I see. The problem actually goes well beyond isDesignMode(). ActionProperty.parse currently assumes that the object (sender) on which the action is being parsed is a CardElement instance, which is no the case here (it's a Choice). It is necessary for an Action instance to have a parent of type CardElement for things to work properly. Choice is not a CardElement, so the let parent = <CardElement>sender; cast in ActionProperty.parse actually hides a bigger bug where the parsed action will have a parent of type Choice.
Your initial implementation worked because:

You artificially added the isDesignMode() method on Choice
Your handler for Choice.inlineAction duplicated the ToggleVisibility logic and was able to get to the target elements via the ChoiceSet's parent

But that's a workaround, we need an actual solution. That solution likely involves:

Updating the Choice class so it has an owner property of type ChoiceSet

Making sure that all choices added to a ChoiceSet (either during parsing or in code via the API) have their owner property set properly to that ChoiceSet

That probably means changing the actual type of ChoiceSet.choices from Choice[] to a custom collection class that handles setting owner whenever a choice is added
Problem is that will likely be a breaking change

Not using the ActionProperty class and instead creating a new ChoiceInlineActionProperty whose parse method would look like this:

parse(sender: SerializableObject, source: PropertyBag, context: SerializationContext): Action | undefined {
    let parent = (<Choice>sender).owner;

    return context.parseAction(
        parent,
        source[this.name],
        this.forbiddenActionTypes,
        parent.isDesignMode());
}

That last bit isn't as generic as I would like, but it's clean.
This is all off-the-top-of-my-head and not thought through though. There might be a better way.

Can you please elaborate on how i do the custom collection ? I tried with SerializableObjectCollectionProperty but i can't manage to get the right parent in the parsing. Also i tried with CustomProperty<PropertyBag>, i managed to get it to be parsed (but it's not calling ChoiceInlineActionProperty and i don't know how to), but the parent is missing a lot of properties, like i'm having a serialization problem. Can you please help me @dclaux ? That's the only thing i'm missing right now to make it work as you said.