Calling ICollection<>.Contains() with a sub query doesn't work

Have a query of this shape:
from p in context.Products
where p.OrderDetails.Contains(context.OrderDetails.FirstOrDefault(orderDetail => orderDetail.Discount == 0.1))
select p;

EF will throw an Argument Exception:
System.ArgumentException : The item expression ('{from OrderDetail orderDetail in value(Microsoft.Data.Entity.Query.EntityQueryable1[Northwind.OrderDetail]) where (Convert([orderDetail].Discount) == 0.1) select [orderDetail] => FirstOrDefault()}') is no ConstantExpression, it is a SubQueryExpression. Parameter name: expression 	at Remotion.Linq.Clauses.ResultOperatorBase.InvokeExecuteMethod(MethodInfo method, Object input) 	at Remotion.Linq.Clauses.ResultOperatorBase.InvokeGenericExecuteMethod[TInput,TResult](IStreamedData input, Func2 genericExecuteCaller)
at Remotion.Linq.Clauses.ResultOperators.ValueFromSequenceResultOperatorBase.ExecuteInMemory(IStreamedData input)
Query\EntityQueryModelVisitor.cs(430,0): at Microsoft.Data.Entity.Query.EntityQueryModelVisitor.ExecuteResultOperator[TSource,TResult](IEnumerable`1 source, ResultOperatorBase resultOperator, StreamedSequenceInfo streamedSequenceInfo)
at lambda_method(Closure , QuerySourceScope )

Tried this on latest build and it no longer throws, but it does return the incorrect results.
Full code I used is
using System;
using System.Collections.Generic;
using Microsoft.Data.Entity;
using System.Linq;

namespace ConsoleApp1
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using (var db = new ProductContext())
            {
                db.Database.EnsureDeleted();
                db.Database.EnsureCreated();

                db.Products.Add(new Product
                {
                    Name = "Crackers",
                    OrderDetails = new List<OrderDetail>
                    {
                        new OrderDetail
                        {
                            Discount = 0.1
                        }
                    }
                });

                db.Products.Add(new Product
                {
                    Name = "Cheese",
                    OrderDetails = new List<OrderDetail>
                    {
                        new OrderDetail
                        {
                            Discount = 0.2
                        }
                    }
                });

                db.SaveChanges();

                var query = from p in db.Products
                where p.OrderDetails.Contains(db.OrderDetails.FirstOrDefault(orderDetail => orderDetail.Discount == 0.1))
                select p;

                foreach (var item in query)
                {
                    Console.WriteLine(item.Name);
                }
            }
        }
    }

    public class ProductContext : DbContext
    {
        public DbSet<OrderDetail> OrderDetails { get; set; }
        public DbSet<Product> Products { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Database=Products;Trusted_Connection=True;");
        }
    }

    public class OrderDetail
    {
        public int OrderDetailId { get; set; }
        public double Discount { get; set; }
        public int ProductId { get; set; }
        public Product Product { get; set; }
    }

    public class Product
    {
        public int ProductId { get; set; }
        public string Name { get; set; }
        public List<OrderDetail> OrderDetails { get; set; }
    }
}

Generated SQL is
SELECT [p].[ProductId], [p].[Name]
FROM [Product] AS [p]
WHERE (
    SELECT 1
    FROM [OrderDetail] AS [o]
    WHERE [p].[ProductId] = [o].[ProductId]
) = 1

And all products are returned from the query (i.e. the filter is not applied in-memory either).

In #4297 queries with Contains were updated to generate SQL with the IN syntax. This works for scalar values, but navigations may involve composite keys.
The provided query tests equality on a floating point value, which tends to have unexpected results, so a better example could be something like this:
    from p in context.Products
    where p.OrderDetails.Contains(context.OrderDetails.FirstOrDefault(orderDetail => orderDetail.Quantity == 1))
    select p;

The current implementation generates the following SQL:
SELECT [p].[ProductID], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
WHERE (
    SELECT CASE
        WHEN (
            SELECT TOP(1) 1
            FROM [Order Details] AS [orderDetail]
            WHERE [orderDetail].[Quantity] = 1
        ) IN (
            SELECT 1
            FROM [Order Details] AS [o]
            WHERE [p].[ProductID] = [o].[ProductID]
        )
        THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)
    END
) = 1

This generates incorrect results because the IN does not correlate the data between the two subqueries.
To correlate these subqueries, the keys of boths Order Details rows must match, but this table has a complex key and a subquery in an IN cannot select more than one column, which means continuing without restructuring the query would require some kind of work-around for navigations like
SELECT [p].[ProductID], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
WHERE (
    SELECT CASE
        WHEN (
            SELECT TOP(1) CONCAT([o].[OrderID], '_',  [o].[ProductID])
            FROM [Order Details] AS [o]
            WHERE [o].[Quantity] = 1
        ) IN (
            SELECT CONCAT([orderDetail].[OrderID], '_', [orderDetail].[ProductID])
            FROM [Order Details] AS [orderDetail]
            WHERE [p].[ProductID] = [orderDetail].[ProductID]
        )
        THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)
    END
) = 1

An alternative implementation would be to switch from using IN to EXISTS which would allow for a query more like:
SELECT [p].[ProductID], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
WHERE (
    SELECT CASE
        WHEN (
            Exists(
	            select 1 from (
				    SELECT TOP (1) [o].[OrderID], [o].[ProductID]
                    FROM [Order Details] AS [o]
			        WHERE [o].[Quantity] = 1
			    ) as [t0] INNER JOIN (
				    SELECT [orderDetails].[OrderID], [orderDetails].[ProductID]
			        FROM [Order Details] AS [orderDetails]
                    WHERE [p].[ProductID] = [orderDetails].[ProductID]
				) as [t1]  on [t0].[OrderID] = [t1].[OrderID] AND [t0].[ProductID] = [t1].[ProductID]
            )
        )
        THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)
    END
) = 1

I suspect that IN is a non-starter for composite keys. The concatenation will most likely make the query non-sargable.
Also, the CASE should be unnecessary and it should be possible to avoid the JOIN in the subquery, e.g. something like this:
SELECT [p].[ProductID], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
WHERE EXISTS (
        SELECT NULL
        FROM (
                SELECT TOP (1) [o].[OrderID], [o].[ProductID]
                FROM [Order Details] AS [o]
                WHERE [o].[Quantity] = 1) as [t1]
         WHERE   on [p].[OrderID] = [t1].[OrderID] AND [p].[ProductID] = [t1].[ProductID])

I think we could also use a query based on JOIN instead of EXISTS:
SELECT [p].[ProductID], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
        INNER JOIN (
                SELECT TOP (1) [o].[OrderID], [o].[ProductID]
                FROM [Order Details] AS [o]
                WHERE [o].[Quantity] = 1) as [t1] ON
                        [p].[ProductID] = [t1].[ProductID]

This should also work:
SELECT [p].[ProductId], [p].[Discontinued], [p].[ProductName], [p].[UnitsInStock]
FROM [Products] AS [p]
WHERE (
        SELECT 1
        FROM (
                SELECT TOP (1) [o].[ProductID]
                FROM [Order Details] AS [o]
                WHERE [o].[Quantity] = 1) AS [t1]
         WHERE  [p].[ProductID] = [t1].[ProductID]) = 1;

Had a discussion with @divega on the implementation for this issue. The eventual goal is to have simple SQL (i.e. the single join between Products and Order Details). To get to the final implementation, there are a few intermediate steps that need to be taken:

Implement a solution that produces correct output (i.e. the exists with inner join example I proposed) - This should be sufficient to address the high priority data corruption issue.
Resolve #4304 - Optimization to remove extra casting logic from the query
Open a new issue to eliminate the inner join - Both inner select statements are on the same table, should be able to combine the clauses into a single subquery.

Created #4389 to track inner join optimization


XML or JSON

For a payment initiation transaction, the ASPSP may support the coding of the payment data using XML or using JSON or both.

Fullur stuðningur við JSON (Mandatory), XML stuðningur valkvæmur.


add wagmi/view 2.x (#484)

add wagmi/view 2.x

renovate

@0xPr0f do you think we can verify the contract or change it for one that's verified?

@glitch-txs yes, we can, i am working on the contract verification right now

running into some trouble with verification, i used remix for the contract and remix passed verification but not showing on etherscan, i want to setup foundry for the verifiaction and potentially redeploy it

@glitch-txs  updated address here 0xc0453e6C996346B9FD712AE4b29ec7EB80806E1b on sepolia


pacman Issue: SSL certificate problem

Describe the issue
Looks like SSL certificate issues
➜ sudo pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirrors.kernel.org : SSL certificate problem: unable to get local issuer certificate
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from mirrors.kernel.org : SSL certificate problem: unable to get local issuer certificate
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from mirrors.kernel.org : SSL certificate problem: unable to get local issuer certificate
error: failed to update community (download library error)
error: failed to synchronize all databases

Looking at this error, it seems others have resolved it by disabling ipv6.. however I cannot find any kernel modules to disable this, and it seems that it's ignoring systemd networking  (i haven't installed any systemd workaround)
To Reproduce
run pacman
Expected behavior
pacman to work
Enviroment:

Windows build number: 1909 18363.1198
Security Software: ZScaler
WSL version 1/2: 2
ArchWSL version <IP_ADDRESS>
ArchWSL Installer type zip/appx/other clean/update [e.g. appx update] appx
Launcher version 20100500

What's interesting is my other pc, with windows 20H2 doesn't have these issues.. I backed up it's copy of arch, and installed it over this one, and still the issue persists.

Just uninstalled arch and reinstalled, same issue.

i installed ubuntu, and found the same issue, this time with curl..., i installed the curl .pem and ran update-ca-certificates which doesn't exist in arch, i found update-ca-trust but it didn't work

I changed my mirror, looks like something was wrong with mirrors.kernel.org's certs

Ok
Thank you for your info.

Thank you for your info.
But are you using ZScaler's TLS inspection?
Please try other https communication.

The TLS-inspected communication is re-encrypted with another CA certificate.

Thanks, yeah it's a work computer, and I don't have access to any of the zscaler configs.. however, I've seen another thread elsewhere, where someone noted disabling his firewall solved the issue.
I have something to now go to our security team with... Thanks :D I'll update this if disabling tls inspection on zscaler solves the issue.

Ok.
If necessary, please refer to the link below.
https://wiki.archlinux.org/index.php/User:Grawity/Adding_a_trusted_CA_certificate#System-wide_–_Arch,_Fedora_(p11-kit)

Ok.
However, there was something I forgot to say.
Apparently, update-ca-certificates doesn't apply in Arch.
If necessary, please refer to the link below.
https://wiki.archlinux.org/index.php/User:Grawity/Adding_a_trusted_CA_certificate#System-wide_–_Arch,_Fedora_(p11-kit)

That last bit solved the issues when I used the corp certs!
closing

I'm glad to hear that.
I thought it should be documented.


Git commands added

Added the common git commands to be explained with demo

@saurabh1002 will open another PR with @MOLOCH-dev 's changes too, closing this PR


vimPlugins.deoplete-emoji: Drop plugin

Motivation for this change
Upstream repository https://github.com/fszymanski/deoplete-emoji has been deleted.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 139620 run on x86_64-linux 1

1 package built:

spacevim


updater: Add support for Ubuntu Vivid Core and ignore Vivid PhoneOverlay

Reacts to https://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/revision/10488

LGTM


Cleanup middleware

Favor standard library in favor of negroni and clean up mux router initialization.

lgtm

@dgonyeo

lgtm


comment tweaks

@bors r+

:pushpin: Commit 0ec710b25e2a56a0aed1793fd83dcb6ef6402b71 has been approved by RalfJung
It is now in the queue for this repository.

:sunny: Test successful - checks-actions
Approved by: RalfJung
Pushing 4d6a5729fb4a76f214c7a91ce2f78ef426bd4f23 to master...


Convert isinstance checks on GateOperation to use op.gate

First attempt at fixing #3556

Will reopen with git authors fixed


Cannot start server

Is it a bug?

Hi, @NdaruAjaa,
Are you using files downloaded from Poggit? If you download from Poggit and get a similar error, please let me know.

Yes I downloaded from poggit and error if I download from github release no errors occur

Thank you for the help


Documentation: using Ribbon without Eureka may require extra configuration (ribbon.eureka.enabled=false)

The chapter in the documentation describing how to use Ribbon without Eureka makes the assumption Eureka is NOT on the classpath - hence the RibbonEurekaAutoConfiguration is not triggered.
However, it might be interesting to say the Ribbon/Eureka integration can be explicitly disabled using the property:
ribbon.eureka.enabled = false

The example given in the documentation won't work if that property is not specified and Eureka is available on the classpath (even if disabled by eureka.instance.enabled=false).
The same is true for people who want to configure Zuul with a static list of servers as described in the "Embedded Zuul Reverse Proxy'. The following example won't work out of the box:
zuul:
  routes:
    users:
      path: /myusers/**
      serviceId: users

 users:
  ribbon:
    listOfServers: example.com,google.com

Since Eureka is included in the Zuul packaging, the integration with Ribbon must be disabled explicitly. The example should be:
 zuul:
  routes:
    users:
      path: /myusers/**
      serviceId: users

 ribbon:
   eureka:
    enabled: false

 users:
  ribbon:
    listOfServers: example.com,google.com

I don't want to disable ribbon completely, because there could be a situation where some paths use eureka, and others don't.

So it takes code, but it is possible
@SpringCloudApplication
@RibbonClients({
		@RibbonClient(name = "users", configuration = UserConfig.class)
})
public class Application {
	public static void main(String[] args) {
		SpringApplication.run(SidecarApplication.class, args);
	}

	@Configuration
	static class UserConfig {

		private String name = "users";

		@Bean
		@ConditionalOnMissingBean
		public IClientConfig ribbonClientConfig() {
			DefaultClientConfigImpl config = new DefaultClientConfigImpl();
			config.loadProperties(this.name);
			return config;
		}

		@Bean
		ServerList<Server> ribbonServerList(IClientConfig config) {
			ConfigurationBasedServerList serverList = new ConfigurationBasedServerList();
			serverList.initWithNiwsConfig(config);
			return serverList;
		}

	}
}

Indeed. Using the ribbon.eureka.enabled=false property would disable Eureka for every Ribbon instance - this is true. And currently, the only way to have it "per service" is using a custom configuration per-service as you described here above. I was even wondering if such configuration couldn't be triggered by the presence of the ribbon.listOfServers property... Do you think it is worth creating another issue to host that feature?
However, my point was that the examples given in the documentation are not complete enough and will not work out of the box - especially the last one for Zuul and the statically registered routes. Don't forget that Ribbon will be integrated with Eureka by default within Zuul - so a DiscoveryEnabledNIWSServerList server list will be used and not a ConfigurationBasedServerList. The former will not consume the ribbon.listOfServers property.

@brenuart Another issue for per service eureka or not would be nice.  I'll update the docs, since it doesn't work as it is.  I think it used to before we changed to @RibbonClients.

can i create this part:
 users:
  ribbon:
    listOfServers: example.com,google.com

dynamically in my source code? The listOfServers depends on database configuration.

@matzegebbe I believe you can return a bean of type ServerList<Server with the list of servers.

@matzegebbe Have you solved this problem?  How to read listOfServers from database and can be dynamically to reload. thanks!

@shengnian this issue was closed a while ago, if you are still having a problem open another issue with a description of your issue and a sample to reproduce it


Accept any content type as json

bors merge

FYI, this is temporary and will be improved in v0.22

bors merge

Bors merge


Navbar

I made the navbar without the routing yet. Should we merge it to the landing-page branch?

I changed the folder name again


Add integration tests for life cycle

Implemented integration tests.
First of all, I implemented tests for the container lifecycle and the create command, referring to opencontainers/runtime-tools.
See tests/README.md for how to run the integration tests.
The flow of test execution is not to run cargo test directly, but to first generate a binary and then run it via sudo.
I am currently trying not to put it on the CI, but will do so if I can.
ref: #56

@minakawa-daiki
Thank you for the PR. I'll review it later. I'm a little busy so it may be later.

I'd like to ask about binary generation for tests. Currently, the binary obtained by cargo test --no-run has a hash value and cannot be uniquely identified.
The binary obtained by cargo test --no-run currently has a hash value and cannot be uniquely identified, so as discussed in the following issue, we parse the json with --message-format=json to get the binary filename.
I can't think of a better idea at the moment, but what do you think?
https://github.com/rust-lang/cargo/issues/1924
https://github.com/rust-lang/cargo/issues/3670

@minakawa-daiki
I think it would be better to make it independent from youki's main code, like oci-spec. And how about writing it in main instead of the test folder as a test tool instead of test? What do you think?

I agree with @utam0k. It would be best to split this out into another module. I would be really interested in contributing in this area, as I think we'll want to start building out our own test suite for cgroups v2. Maybe we could even make this not a cargo test at all, and so something similar to the oci runtime-tools and use the test anything protocol or similar. This way we can essentially just run the built tests which can invoke a new build of the project and test it. Food for thought.

I apologize for the delay in my response. It's certainly better to separate it as a testing tool, and I've reminded myself that there's no need to force it to be included in cargo test. I'm a little busy with my personal life and will deal with this a little later, but I'll try.

@minakawa-daiki Let me know if you need any help with this. I think if the effort to break this out into it's own tool is too high that perhaps we allow this change in, and when I have time I can move this into it's own crate which will serve as a place to build out our own integration testing tool. What do you think @utam0k?

@minakawa-daiki Let me know if you need any help with this. I think if the effort to break this out into it's own tool is too high that perhaps we allow this change in, and when I have time I can move this into it's own crate which will serve as a place to build out our own integration testing tool. What do you think @utam0k?

@tsturzl
I would like to hear the opinion of the author of this PR and make a decision. If he is busy, I think that's fine.

Hey, I was checking the test anything protocol which is used by oci-spec validation tests, and rust has testanything crate which provides producer of data for tests as per TAP protocol. I'm not sure how much maintained and active it is, but that might be useful. We can use that as the part of our main program, and take path of youki executable and bundle etc.as cmd options. Then our binary will run all the tests that are in oci-validation tests, written in rust, and produce output using testanything. Maybe that way we won't need to create a complete tool for testing right now, instead just port validation suite, which will be used by us to validate the changes. @minakawa-daiki @tsturzl ?
Also, If you decide to port the tests to rust in some way, let me know , I would like to help.

@tsturzl @utam0k
Thank you for your concern!
I'm currently writing the code to migrate using oci_spec as a reference. I'll be able to submit a pull request after this, and I hope you'll take a look at that code and decide for yourself!

@YJDoc2
I think testanything is a very good idea!
I think we can use this to implement it once, and if testanything is not active, we can make it a policy to replace the implementation ourselves later. Thanks!

@utam0k @tsturzl @YJDoc2
I wrote out the integration test as a test tool.
Since the integration-test directory already exists, it is temporarily named youki-integration-test.
As an implementation, I used testanything that @YJDoc2 taught me. As the first test, I've added tests for the container lifecycle and the create command.
Please refer to the Readme.md for more information on how to use the tests.
I hope you will review if this is a good policy for now.

@minakawa-daiki I ran the test according to the README, and it failed. Is there something wrong with this?
 sudo ./youki_integration_test
# Start Create comand test suite
1..6
ok 1 Create a new container test
# This operation must create a new container.
ok 2 Execute state test
# This operation must state the container.
ok 3 Execute start test
# This operation must start the container.
ok 4 Execute state test
# This operation must state the container.
ok 5 Execute kill test
# This operation must kill the container.
not ok 6 Execute delete test
# This operation must delete the container.
# End Create comand test suite
# Start Create comand test suite
1..3
ok 1 create with no ID test
# This operation MUST generate an error if it is not provided a path to the bundle and the container ID to associate with the container.
ok 2 create with ID test
# This operation MUST create a new container.
ok 3 create with an already existing ID test
# If the ID provided is not unique across all containers within the scope of the runtime, or is not valid in any other way, the implementation MUST generate an error and a new container MUST NOT be created.
# End Create comand test suite

Would it be possible , that instead of relying on panic as indication of test failure, we take the result of spawned process, along with its stderr, and accordingly set status and error message? That way we can have more information on why the test failed.
The process ExitStatus implements fmt , so it can convert the exit code of the process to readable string, and we might be able to use it as an error message.

I agree with @YJDoc2. I think that would provide more context. And in one of @utam0k's review comments he mentions using the anyhow crate that we use heavily in youki already. I think that could give you a lot of useful tools for dealing with errors, and should be pretty simple to add.

@utam0k I pulled this PR and ran tests as stated, several times, but all tests give ok for me consistently. :thinking: :thinking:

As for the error messages, I'll change the implementation to use anyhow.
Please wait for a while as we will respond to other reviews as they come in.
The reason why the test fails is not yet known, so we are investigating.

@utamoku Corrected the reviews we received except for the hard coding of the youki command.
I have commented on the hard coding of the youki command separately, so please check it.

Sorry, It had nothing to do with this PR, so I sent out a PR to fix it.
https://github.com/containers/youki/pull/165

@utam0k I pulled this PR and ran tests as stated, several times, but all tests give ok for me consistently. thinking thinking


Hyperdrive Replication Issues

This is a few issues, but may be related:
Discovery Key Issues
I had an issue adding the hyperdrive archive key as a string to the database. The discovery key ended up different if I used a string vs buffer.
var coreArchive = require('hypercore-archive')('/dir')

var archive = drive.createArchive()
var archiveKey = archive.key

coreArchive.add(archiveKey) // okay
coreArchive.add(archiveKey.toString('hex')) // not okay

If add should accept strings then we just need to fix that. Or make sure its documented that only buffers are accepted.
Metadata + Content
(EDIT: this worked nevermind!)
EDIT Also: But they ended up in two different directories. Is that how it is supposed to work?

Metadata + Content
Yes thats exactly how its supposed to work :)

strings are also accepted now


Is there any way to make my nvim-cmp panel transparent?

I use default settings and seems dont work? Nvim-8.0.0dev

I'm a bit confused by the question. Why do you think this plugin has anything to do with nvim-cmp?

sorry, i thought this plugin can let my neovim floating windows transparent like readme.md . My english is bad lol.

So is there any ways can make my floating windows transparent?

Most windows use the :help winblend option to set transparency. For nvim-cmp, you may need to set :help pumblend instead


Hit space to RESPAWN does not work

The hit space to re-spawn unconscious screen only works if you respawn throught the vanilla process (press ESC and then select the RESPAWN option)
P.S. Tested with ACE in DEVELOP branch in MP

Tested also in SP (Altis mission) and didn't worked.

which version did you tested this?
To me it works correctly in  Version: develop 8c0ea9b
Mod: CBA, TFAR
Multiplayer dedicated server
Side: tested both vs East and vs West

On master branch not in the develop one.
Better to test the develop version?

yes it's way ahead

Hi again, tested again in develop (SP) version and respawn is working but not 100%.
Killed again but inside a vehicle and cannot click on space bar. After that clicked on respawn on general arma menu and get back to the new game & load mission menu.

oh yes, that was happening in Warlords of the Pacific... i fixed it there by adding a "moveout" i'll check in the code if i understand where to place it

So, do we agree that this only happens when a player is killed on a vehicle?


Add link to About CC section in following places

?action=browse
?action=search
text for should be something like "What is Creative Commons? Learn more[link]"
?action=new
text should be something like "Learn more about the licenses"

Added in next version. This is currently below the list of licenses so it's immediately above the "about CC" ink in the footer and linked to the same location.
If this is not OK I can move the link higher, or remove it and rename "About CC" in the footer to "What Is Creative Commons" or "About Creative Commons".


Screenshots for finishing quests have incorrect filenames

Screenshot gets saved as "ds with My Arm.png"

Many quests have this issue, I think another approach has to be taken to name screesnhots unless we want to add specific cases for each quest.

https://oldschoolrunescape.wikia.com/wiki/Category:Quest_reward_scroll_images
Looked at a few of them, it's a total crapshoot what format any given quest will use. Specific cases for each quest might be the way to go here

Horror from the Deep
The Giant Dwarf
Recruitment Drive
The Feud (kind of)

Is some of the quests I came across in my screenshots directory. As I understand without looking at the code, the first "You have completed " length characters of the quest sentence is chopped and the rest is appended to screenshot name text. Maybe a way could be implemented to check for exceptions like these.

I'm adding the help wanted tag, as this is very likely some extra/different widget text present for certain quests that's causing this. Anyone who hasn't completed some of the quests listed in the comment above and having some knowledge of how the screenshot plugin code gets quest names can help determine why it is these quests are having the beginnings removed in error.

As was pointed out by kittenparry, it seems the filename is determined with
String text = client.getWidget(WidgetInfo.QUEST_COMPLETED_NAME_TEXT).getText();
fileName = "Quest(" + text.substring(19, text.length() - 1) + ")";
which means it only expects quests complete screens with the "You have completed {Quest_name}!" format at the top.
Unfortunately, Jagex isn't entirely consistent with the text at the top.
Luckily however, the Wiki provides a screenshot of every quest completed screen, so I've compiled a list of the quests with exceptions to the format that would generate this error in the filename.

The Slug Menace (no period or exclamation mark at the end)

Mourning's End(s?) Part 2

Swan Song

Rag and Bone Man II

Every Recipe for Disaster subquest and endquest except for the first one

The Ratcatchers (the closing quote will be cut off)

Icthlarin's Little Helper
Lunar Diplomacy
The Feud

Forgettable Tale of a Drunken Dwarf

Recruitment Drive

The Giant Dwarf

One Small Favor

In Search of the Myreque

Between a Rock...

Horror from the Deep

Lucien and armadyl versions of Temple of Ikov

Making Friends with My Arm

Thank you for compiling this huge list, must have taken some time.
Fake completion of the Hazeel Cult also results in a screenshot being saved named Quest(... completed the Hazeel Cult Quest) DATE.png. I guess if an exception could be implemented for the rest, maybe it could also be implemented to ignore this quest completion screen.

I've been running some test scripts in Python and I propose a solution that uses two regexes to capture the quest name from the image. But before I share sample code, I want to address what I think is the bigger question among this entire thread, which is "To what extent do we care about matching the quest name to wiki name?" Above replies have cited many quests for one of multiple reasons, from totally missing the quest name to having an extra character, but no mention of any guideline for what the output should be (e.g. match wiki name exactly, 1-2 char difference is okay, 1 word difference is okay).
The proposed mechanism carries its own exceptions, so I want to discuss guidelines with this mechanism. The first question is, is any exception worth the extra code to handle them? An easy, yet suitable answer is "no", which means the base solution is roughly 9-ish lines of code. The probable answer, though, is "yes", and in that case the second question is, which exceptions should be handled and how? I've listed a sample of 8 exceptions I came across in my testing so I and others can decide whether it is appropriate to handle them. Exception means a not 100% match with wiki name, which is the strictest definition.
Guide for table: Each row is a category of exception, with example quest and description. I compare the output of current RL code, and of 2-3 approaches I created. Option 1 uses the base solution. Option 2 uses the base solution plus extra code to refine the output. Option 3 uses the base solution but modified regexes, which affects more than that category of cases. Only 2 quests are double-counted (see row #5).

Quest Name
Quest Description
Current RL Code
Proposed Code (Option 1)
Proposed Code w/Extra Code (Option 2)
Proposed Code w/Modified Regex (Option 3)

(#1) Hazeel Cult
case-specific
Quest(... completed the Hazeel Cult Quest)
Quest(Hazeel Cult) *Duplicates this but RL avoids overwrite
Quest(Hazeel Cult partial completion)
N/A

(#2) Rag and Bone Man II
case-specific
Quest( completed Rag and Bone Man)
Quest(Rag and Bone Man) *Duplicates this but RL avoid overwrite
Quest(Rag and Bone Man II)
N/A

(#3) Making Friends with My Arm
Differs from wiki name by 1+ words (5-count)
Quest(ds with My Arm)
Quest(Made Friends with My Arm)
Quest(Making Friends with My Arm)
N/A

(#4) Recipe for Disaster/Defeating the Culinaromancer
RFD Subquests (10-count)
Quest(u have defeated the Culinaromancer)
Quest(Culinaromancer)
Quest(Recipe for Disaster - Culinaromancer) *(2a)
Quest(Recipe for Disaster - Defeating the Culinaromancer) *(2b)

(#5) Doric's Quest
Contains "Quest" in wiki name (5-count + "Another Cook's"(#4) + "Legends"(#3))
Quest(Doric's Quest)
Quest(Doric's)
Quest(Doric's Quest)
Quest(Doric's Quest) *but a quest like Demon Slayer becomes Quest(Demon Slayer Quest)

(#6) The Golem
Differs from wiki name by word "The" (14-count)
Quest(the Golem quest)
Quest(Golem)
Quest(The Golem)
Quest(the Golem) *but a quest like Demon Slayer becomes Quest(the Demon Slayer)

(#7) Lost City
Contains wiki name plus extra words (4-count)
Quest(the Lost City Of Zanaris Quest)
Quest(Lost City Of Zanaris)
Quest(Lost City)
N/A

(#8) Mourning's End Part II
Differs from wiki name by 1-2 chars (12-count)
Quest(Mourning's Ends Part )
Quest(Mourning's Ends Part 2)
Quest(Mourning's End Part II)
N/A

Consider each row independently in deciding which of the options is the most appropriate, and consider the tradeoff between simplicity of code and accuracy of output. As it seems to be beyond a simple patch, I wanted to share a (lengthy) description first before sharing code. That said, I've tested all cases in Python and have an idea of how to implement each option for each exception, but I'm new to the RL code and less fluent in Java, so I'd appreciate some pointers. Also it's not clear to me how this could be effectively tested in practice other than legitimately completing the relevant quests. (Edit: I found ScreenshotPluginTest but I'm still comprehending the mechanics in case I should add to that file.) If needed, I could go into further detail about any aspect of this proposal, or how any specific quest is handled.


RtlCloneUserProcess no longer works after WSL??

Some people report that. Unfortunately, RtlCloneUserProcess is the only way to implement POSIX fork semantics on win32. That is extremely important for a stable ABI.

I have not seen that reported anywhere, could you provide a link?
I know RtlCloneUserProcess is used by CYGWIN but WSL gives you a proper fork().
I don't think that takes away or conflicts with RtlCloneUserProcess though.

RtlCloneUserProcess

https://gist.github.com/juntalis/4366916
I can ensure the RtlCloneUserProcess is bugged now. The process never returns and even the process manager could not see the process.
https://social.msdn.microsoft.com/Forums/en-US/4ade520c-0899-4600-9b80-6202d87c16ef/rtlcloneuserprocess-behavior-changed-after-build-14393?forum=windowsgeneraldevelopmentissues
Can you use the proper fork Linux syscall for none WSL process?

https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process-overview/
This shows that in Windows NT Kernel, fork does exist since its beginning. I think it is time to expose it as a real win32 API so that C++ standard library could use that a portable implementation for future process library.

Can you use the proper fork Linux syscall for none WSL process?

As far as I know, no.

I would like to work on a stable ABI to allow POSIX programs to run directly on windows without NTDLL.dll.

WSL is the new POSIX compatibility layer for Windows, so simply target your applications to Linux and test on WSL.

This shows that in Windows NT Kernel, fork does exist since its beginning

I understand but I don't think that is up to the WSL team.

WSL is the new POSIX compatibility layer for Windows, so simply target your applications to Linux and test on WSL.

The problem is that WSL is not enabled by default.
It does not reduce binary bloat. I would like to reduce attack surface of Return oriented programming. The shared library is a historical bloating.

It is unclear what you want to achieve. WSL has nothing to do with RtlCloneUserProcess. That function is exported from ntdll.dll and its a wrapper of NtCreateUserProcess which is a NT syscall. User should not use undocumented functions. The POSIX fork functionality in WSL1 is implemented in LxCore.sys kernel mode driver, written from scratch, is 100% closed source.

NtCreateUserProcess

AFAIK, NT kernel does have a full working fork syscall. It was designed that way.
So, NtCreateUserProcess can emulate fork()?

WSL is not enabled by default.

WSL is easy enough to enable and is getting easier.

It does not reduce binary bloat.

Docker Desktop built their new version on WSL2 with a distro image. If size is a big concern then you can download an import or build your own mini distro image for WSL.

The shared library is a historical bloating.

What shared library? If you use WSL your libraries are handled by the distro. Ubuntu has 5+ years of support on the LTS releases we push to the Store. That is a pretty solid basis to build on. You could also build your application statically.

NT kernel does have a full working fork syscall.

NT does not have a UNIX-style fork. The nuances between the NT, Linux, and UNIX handling of forks, threads, and processes can all be easily Googled.

NtCreateUserProcess can emulate fork()?

Still not sure what problem you are trying to actually solve here. We might be able to help if we knew.

It is unclear what you want to achieve. WSL has nothing to do with RtlCloneUserProcess. That function is exported from ntdll.dll and its a wrapper of NtCreateUserProcess which is a NT syscall. User should not use undocumented functions. The POSIX fork functionality in WSL1 is implemented in LxCore.sys kernel mode driver, written from scratch, is 100% closed source.

Some questions to ask you about NtCreateUserProcess. I want to create a process with POSIX at-series semantics to avoid TOCTOU security vulns. I have successfully done that with files and other APIs. However, the NtCreateUserProcess does not seem to allow me to pass a directory handle to that API. why?

How can I pass directory handle or the image file handle or the section handle to NtCreateUserProcess? I want to implement POSIX at() semantics (takes a directory file descriptor and then use that for pathing) to avoid TOCTOU vulns. Is that possible I can pass the section handle or the image filehandle or the directory handle (+ relative path to that directory) to the NtCreateUserProcess?
Should I still do something to tell csrss.exe we created the new process with NtCreateUserProcess?

@expnkx
RtlCloneUserProcess hasn't stopped working. The problem was the code you're using was outdated.

https://gist.github.com/juntalis/4366916

You need to fix these 3 issues for that code to work again:

RtlCloneUserProcess is missing the __stdcall calling convention which is required.
Zero the structure and initialize the size before calling RtlCloneUserProcess:

memset(&process_info, 0, sizeof(process_info));
process_info.Size = sizeof(process_info);

The type for SECTION_IMAGE_INFORMATION has changed and needs to be updated with the latest type found here:
https://github.com/processhacker/phnt/blob/e9c8121f41b00cd24e31eb466c9795785096b764/ntmmapi.h#L424-L470

After making those 3 changes the code correctly forked and created a file with the string "ok".
You can probably close this ticket now 👍

@expnkx
RtlCloneUserProcess hasn't stopped working. The problem was the code you're using was outdated.

https://gist.github.com/juntalis/4366916

You need to fix these 3 issues for that code to work again:

RtlCloneUserProcess is missing the __stdcall calling convention which is required.
Zero the structure and initialize the size before calling RtlCloneUserProcess:

memset(&process_info, 0, sizeof(process_info));
process_info.Size = sizeof(process_info);

The type for SECTION_IMAGE_INFORMATION has changed and needs to be updated with the latest type found here:
https://github.com/processhacker/phnt/blob/e9c8121f41b00cd24e31eb466c9795785096b764/ntmmapi.h#L424-L470

After making those 3 changes the code correctly forked and created a file with the string "ok".
You can probably close this ticket now 👍

@expnkx
RtlCloneUserProcess hasn't stopped working. The problem was the code you're using was outdated.

https://gist.github.com/juntalis/4366916

You need to fix these 3 issues for that code to work again:

RtlCloneUserProcess is missing the __stdcall calling convention which is required.
Zero the structure and initialize the size before calling RtlCloneUserProcess:

memset(&process_info, 0, sizeof(process_info));
process_info.Size = sizeof(process_info);

The type for SECTION_IMAGE_INFORMATION has changed and needs to be updated with the latest type found here:
https://github.com/processhacker/phnt/blob/e9c8121f41b00cd24e31eb466c9795785096b764/ntmmapi.h#L424-L470

After making those 3 changes the code correctly forked and created a file with the string "ok".
You can probably close this ticket now 👍

Another question. How do you execve after RtlCloneUserProcess?

@expnkx
RtlCloneUserProcess hasn't stopped working. The problem was the code you're using was outdated.

https://gist.github.com/juntalis/4366916

You need to fix these 3 issues for that code to work again:

RtlCloneUserProcess is missing the __stdcall calling convention which is required.
Zero the structure and initialize the size before calling RtlCloneUserProcess:

memset(&process_info, 0, sizeof(process_info));
process_info.Size = sizeof(process_info);

The type for SECTION_IMAGE_INFORMATION has changed and needs to be updated with the latest type found here:
https://github.com/processhacker/phnt/blob/e9c8121f41b00cd24e31eb466c9795785096b764/ntmmapi.h#L424-L470

After making those 3 changes the code correctly forked and created a file with the string "ok".
You can probably close this ticket now 👍

Another question. How do you execve after RtlCloneUserProcess?

How do you execve after RtlCloneUserProcess?

You can only execve native executables (/subsystem:native) and only by using system calls such as NtCreateUserProcess?
execve in the CRT calls CreateProcessW which requires csrss/win32k support which is using cached addresses for some objects (e.g. activation contexts) which no longer exist because they're owned by win32k and since its a new process that can't access them it causes an access violation in the clone.
That's also why cloning other win32 processes on Windows has always had issues - objects like windows/gdi/fonts/bitmaps/brushes etc... are not 'real' handles to objects which exist in the executive object namespace that can be duplicated by cloning the process but rather 'fake' objects allocated by a kernel driver (win32k).
Cloning doesn't copy objects created by drivers (e.g. win32k) that end up cached deep inside usermode DLLs (kernelbase/user32/imm32) for things like windows/gdi/fonts etc... which will also cause access violations because those objects and addresses don't exist or can't be accessed by the newly cloned process.
That's also why that gist sample you linked called AllocConsole() because it had to ask condrv.sys to recreate objects for the console that were cached by the CRT and were only valid for the original parent process. Kernel drivers don't have any support for duplicating/cloning objects and are never notified about cloning, so any objects they've created for the process don't get cloned.
Native processes and console processes only ever use system calls for everything (similar to linux processes) and ever use executive objects so cloning them works. To better illustrate; If the process has an object reference to something outside this blue box then cloning won't work: https://user-images.githubusercontent.com/1306177/105578892-73f94580-5dd7-11eb-9f11-646f4e0b9c9b.png
So provided you only clone native/console processes and only use native system calls you'll be fine.

How do you execve after RtlCloneUserProcess?

You can only execve native executables (/subsystem:native) and only by using system calls such as NtCreateUserProcess?
execve in the CRT calls CreateProcessW which requires csrss/win32k support which is using cached addresses for some objects (e.g. activation contexts) which no longer exist because they're owned by win32k and since its a new process that can't access them it causes an access violation in the clone.
That's also why cloning other win32 processes on Windows has always had issues - objects like windows/gdi/fonts/bitmaps/brushes etc... are not 'real' handles to objects which exist in the executive object namespace that can be duplicated by cloning the process but rather 'fake' objects allocated by a kernel driver (win32k).
Cloning doesn't copy objects created by drivers (e.g. win32k) that end up cached deep inside usermode DLLs (kernelbase/user32/imm32) for things like windows/gdi/fonts etc... which will also cause access violations because those objects and addresses don't exist or can't be accessed by the newly cloned process.
That's also why that gist sample you linked called AllocConsole() because it had to ask condrv.sys to recreate objects for the console that were cached by the CRT and were only valid for the original parent process. Kernel drivers don't have any support for duplicating/cloning objects and are never notified about cloning, so any objects they've created for the process don't get cloned.
Native processes and console processes only ever use system calls for everything (similar to linux processes) and ever use executive objects so cloning them works. To better illustrate; If the process has an object reference to something outside this blue box then cloning won't work: https://user-images.githubusercontent.com/1306177/105578892-73f94580-5dd7-11eb-9f11-646f4e0b9c9b.png
So provided you only clone native/console processes and only use native system calls you'll be fine.

How do you execve after RtlCloneUserProcess?

You can only execve native executables (/subsystem:native) and only by using system calls such as NtCreateUserProcess?
execve in the CRT calls CreateProcessW and requires csrss/win32k support which is using cached addresses for some objects (e.g. activation contexts) which no longer exist because they're owned by win32k and since its a new process that can't access them it'll cause an access violation in the clone.
That's also why cloning other win32 processes on Windows has always had issues - objects like windows/gdi/fonts/bitmaps/brushes etc... are not 'real' handles to objects which exist in the executive object namespace that can be duplicated by cloning the process but rather 'fake' objects allocated by a kernel driver (win32k).
Cloning doesn't copy objects created by drivers (e.g. win32k) that end up cached deep inside usermode DLLs (kernelbase/user32/imm32) for things like windows/gdi/fonts etc... which will also cause access violations because those objects and addresses don't exist or can't be accessed by the newly cloned process.
That's also why that gist sample you linked called AllocConsole() because it had to ask condrv.sys to recreate objects for the console that were cached by the CRT and were only valid for the original parent process. Kernel drivers don't have any support for duplicating/cloning objects and are never notified about cloning, so any objects they've created for the process don't get cloned and have to be recreated.
Native processes and console processes only ever use system calls for everything (similar to linux processes) and only ever use executive objects so cloning them works. To better illustrate; If the process has an object reference to something outside this blue box then cloning won't work: https://user-images.githubusercontent.com/1306177/105578892-73f94580-5dd7-11eb-9f11-646f4e0b9c9b.png
So provided you only clone native/console processes and only use native system calls you'll be fine.

I would like to create process at directory_file just like execveat(). How to achieve this? Unfortunately win32 does not provide this facility, it has to be done directly at NT level.
https://github.com/expnkx/fast_io/blob/56769a8175148f03941ea16fed650e6dc7f9569a/include/fast_io_hosted/process_revamp/nt.h#L84

How do you execve after RtlCloneUserProcess?

You can only execve native executables (/subsystem:native) and only by using system calls such as NtCreateUserProcess?
execve in the CRT calls CreateProcessW and requires csrss/win32k support which is using cached addresses for some objects (e.g. activation contexts) which no longer exist because they're owned by win32k and since its a new process that can't access them it'll cause an access violation in the clone.
That's also why cloning other win32 processes on Windows has always had issues - objects like windows/gdi/fonts/bitmaps/brushes etc... are not 'real' handles to objects which exist in the executive object namespace that can be duplicated by cloning the process but rather 'fake' objects allocated by a kernel driver (win32k).
Cloning doesn't copy objects created by drivers (e.g. win32k) that end up cached deep inside usermode DLLs (kernelbase/user32/imm32) for things like windows/gdi/fonts etc... which will also cause access violations because those objects and addresses don't exist or can't be accessed by the newly cloned process.
That's also why that gist sample you linked called AllocConsole() because it had to ask condrv.sys to recreate objects for the console that were cached by the CRT and were only valid for the original parent process. Kernel drivers don't have any support for duplicating/cloning objects and are never notified about cloning, so any objects they've created for the process don't get cloned and have to be recreated.
Native processes and console processes only ever use system calls for everything (similar to linux processes) and only ever use executive objects so cloning them works. To better illustrate; If the process has an object reference to something outside this blue box then cloning won't work: https://user-images.githubusercontent.com/1306177/105578892-73f94580-5dd7-11eb-9f11-646f4e0b9c9b.png
So provided you only clone native/console processes and only use native system calls you'll be fine.

I would like to create process at directory_file just like execveat(). How to achieve this? Unfortunately win32 does not provide this facility, it has to be done directly at NT level.
https://github.com/expnkx/fast_io/blob/56769a8175148f03941ea16fed650e6dc7f9569a/include/fast_io_hosted/process_revamp/nt.h#L84

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

Can you join discord https://discord.gg/vMKhB9Q to teach me how to do this? NtCreateUserProcess is too hard for me to use correctly.

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

Can you join discord https://discord.gg/vMKhB9Q to teach me how to do this? NtCreateUserProcess is too hard for me to use correctly.

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

extern "C"
NTSTATUS
NTAPI
NtCreateUserProcess(
__out PHANDLE ProcessHandle,
__out PHANDLE ThreadHandle,
__in ACCESS_MASK ProcessDesiredAccess,
__in ACCESS_MASK ThreadDesiredAccess,
__in_opt POBJECT_ATTRIBUTES ProcessObjectAttributes,
__in_opt POBJECT_ATTRIBUTES ThreadObjectAttributes,
__in ULONGLONG ProcessFlags,
__in ULONGLONG ThreadFlags,
__in_opt PRTL_USER_PROCESS_PARAMETERS ProcessParameters,
__inout PVOID CreateInfo,
__in_opt PVOID AttributeList
) noexcept;

I do not know where you can put section handle into this API. The API is too chaotic.

I would like to create process at directory_file just like execveat()

You need to pass the directory handle in the OBJECT_ATTRIBUTES and your nt_create_section function doesn't do that.
Consider this example:
HANDLE fileHandle;
HANDLE sectionHandle;
UNICODE_STRING fileName;
OBJECT_ATTRIBUTES objectAttributes;
IO_STATUS_BLOCK isb;

RtlInitUnicodeString(&fileName, FileName); // <--- filename relative to directory
InitializeObjectAttributes(
    &objectAttributes,
    &fileName,
    OBJ_CASE_INSENSITIVE,
    DirectoryHandle, // <--- Parent directory
    NULL
    );

status = NtCreateFile(
    &fileHandle,
    FILE_READ_DATA | FILE_EXECUTE | SYNCHRONIZE,
    &objectAttributes,
    &isb,
    NULL,
    FILE_ATTRIBUTE_NORMAL,
    FILE_SHARE_READ,
    FILE_OPEN,
    FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT,
    NULL,
    0
    );

status = NtCreateSection(
    &sectionHandle,
    SECTION_QUERY | SECTION_MAP_EXECUTE,
    NULL,
    &size,
    PAGE_READONLY,
    SEC_IMAGE,
    fileHandle
    );

NtCreateUserProcess(..., sectionHandle, ...);

extern "C"
NTSTATUS
NTAPI
NtCreateUserProcess(
__out PHANDLE ProcessHandle,
__out PHANDLE ThreadHandle,
__in ACCESS_MASK ProcessDesiredAccess,
__in ACCESS_MASK ThreadDesiredAccess,
__in_opt POBJECT_ATTRIBUTES ProcessObjectAttributes,
__in_opt POBJECT_ATTRIBUTES ThreadObjectAttributes,
__in ULONGLONG ProcessFlags,
__in ULONGLONG ThreadFlags,
__in_opt PRTL_USER_PROCESS_PARAMETERS ProcessParameters,
__inout PVOID CreateInfo,
__in_opt PVOID AttributeList
) noexcept;

I do not know where you can put section handle into this API. The API is too chaotic.

I do not know where you can put section handle into this API. The API is too chaotic.

Sorry, I didn't check the types... NtCreateUserProcess doesn't use the section handle while NtCreateProcess/NtCreateProcessEx does.... You would have to use NtCreateProcess/NtCreateProcessEx instead since it uses the object_attributes.

NtCreateUserProcess is too hard for me to use correctly.

It's going to constantly break because it always changes... You can instead use the documented method which won't break:
int execveat(int dirfd, char *pathname
dirfd is a handle so you can query them for the path:
TCHAR Path[MAX_PATH];
DWORD status;

status= GetFinalPathNameByHandle(dirfd, Path, MAX_PATH, VOLUME_NAME_DOS);

strcat/wcscat the "Path" with "pathname" then simply call CreateProcess... That method won't ever break and there's samples online:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlea#examples
or
https://docs.microsoft.com/en-us/windows/win32/memory/obtaining-a-file-name-from-a-file-handle

I do not know where you can put section handle into this API. The API is too chaotic.

Sorry, I didn't check the types... NtCreateUserProcess doesn't use the section handle while NtCreateProcess/NtCreateProcessEx does.... You would have to use NtCreateProcess/NtCreateProcessEx instead since it uses the object_attributes.

NtCreateUserProcess is too hard for me to use correctly.

It's going to constantly break because it always changes... You can instead use the documented method which won't break:
int execveat(int dirfd, char *pathname
dirfd is a handle so you can query them for the path:
TCHAR Path[MAX_PATH];
DWORD status;

status= GetFinalPathNameByHandle(dirfd, Path, MAX_PATH, VOLUME_NAME_DOS);

strcat/wcscat the "Path" with "pathname" then simply call CreateProcess... That method won't ever break and there's samples online:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlea#examples
or
https://docs.microsoft.com/en-us/windows/win32/memory/obtaining-a-file-name-from-a-file-handle

dirfd is a handle so you can query them for the path:
TCHAR Path[MAX_PATH];
DWORD status;

status= GetFinalPathNameByHandle(dirfd, Path, MAX_PATH, VOLUME_NAME_DOS);

strcat/wcscat the "Path" with "pathname" then simply call CreateProcess... That method won't ever break and there's samples online:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlea#examples
or
https://docs.microsoft.com/en-us/windows/win32/memory/obtaining-a-file-name-from-a-file-handle

bad idea, the goal of putting dir handle here is to avoid TOCTOU security vuln and that is my goal here. Your stuff is how TOCTOU comes from.

dirfd is a handle so you can query them for the path:
TCHAR Path[MAX_PATH];
DWORD status;

status= GetFinalPathNameByHandle(dirfd, Path, MAX_PATH, VOLUME_NAME_DOS);

strcat/wcscat the "Path" with "pathname" then simply call CreateProcess... That method won't ever break and there's samples online:
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlea#examples
or
https://docs.microsoft.com/en-us/windows/win32/memory/obtaining-a-file-name-from-a-file-handle

bad idea, the goal of putting dir handle here is to avoid TOCTOU security vuln and that is my goal here. Your stuff is how TOCTOU comes from.

the goal of putting dir handle here is to avoid TOCTOU

The disassembly for NtCreateProcess doesn't appear to use RootDirectory anywhere so you're not going to be able to use it for this purpose.

Your stuff is how TOCTOU comes from.

It's demonstration code and doesn't have error checking? You need to copy/probe the variables into your own memory allocation to prevent those sort of issues.

the goal of putting dir handle here is to avoid TOCTOU

The disassembly for NtCreateProcess doesn't appear to use RootDirectory anywhere so you're not going to be able to use it for this purpose.

Your stuff is how TOCTOU comes from.

It's demonstration code and doesn't have error checking? You need to copy/probe the variables into your own memory allocation to prevent those sort of issues.

You clearly do not understand what is TOCTOU.

You clearly do not understand what is TOCTOU.

Please enlighten me how it applies here?


Takes too long (18 secs)

This action simply finds the path to msbuild using the vswhere tool - why on earth it's taking 18 seconds? Is it because of the slow file system on the Github-hosted runner?

Acknowledging this issue.  I'll look into this with runner team.

@timheuer thanksa lot, appreciate! At the moment I'm simply hardcoding the path.
I suspect it's because of the overall file-system warm up in a "just booted" Azure instance

vswhere itself is very fast, but I also see times when building it that running it the first time encounters a full Windows Defender scan. While you may be right about the warm up, I wouldn't expect it to be this slow unless, perhaps, Defender is compounding that. For hosted agents, I wonder if Defender should maybe only run for the workspace directories. They are ephemeral, so it's not like another tenant could corrupt someone else's host.

While resolving an issue I was having fixing another bug, I found it's the globbing that is the problem. I had to move when the globbing happens for compatibility. vswhere returned a value almost right away, but the globbing (now done outside vswhere, since globbing in vswhere is pretty new in Visual Studio) took a while. On my dev machines with ssds, this only takes about 1s. I imagine the drive type/configuration of the hosts is more of a factor here.
An alternative would be the older way. I'll make the changes in the PR I'm working on for #10.


Additional modes

Requires Node 8, but since Node 6 has reached end of life, most npm libraries should now require Node 8.

LGTM, but can you now rebase?

done. ready to merge

Thanks!

Thanks for the merge :-)
Any chance of a release? I can't figure out how to depend on the github master directly since the library needs to be built before it can be used.

Yeah I think releasing after the weekend would work best.

@Fleker I'd really appreciate a release -- I want to improve our functional tests and due to the way this project is built, I can't depend on the github from npm :-(
Is there a way to depend on a project of this style from github without waiting for a release?

v0.3.0 is published!


[Candidate 1v1 Echo ZH v1] Map uses houses with missing fire ports

Observed behavior
Map uses houses with missing fire ports. Soldiers shoot from 0,0,0.
Expected behavior
Ranked Maps use houses with good fire port setups.
Screenshots

Not a blocker bug, because terrain around houses is very flat and there appears to be no practical detriment.

If houses gets destroyed by own fire firing at enemy I would agree, but as you stated "no practical detriment" I would like to leave that.

As designed.


how to test inject_into_file behavior

Hi, I'm struggling when testing an inject_into_file, My problem is that I can't generate my template file inside my dummy/tmp directory before running my generator that modifies that file. The file that I want to change is generated by another generator. Does ammeter has a clean way to test this?

Hi @joseramonc, have you found your solution ? I'm having the same problem. if you guys can have a look at this issue in my project ?

@alain-andre sorry, I skipped that problem and haven't tried to resolve it again yet, so I still haven't found a solution.


Scala.js 1

The following build error is suspicious:
[error] (catsJS / Test / fastOptJS) java.io.IOException: Failed to deserialize /home/travis/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_sjs1_2.13/2.1.0/cats-core_sjs1_2.13-2.1.0.jar:/cats/syntax/package$hash$.sjsir

Codecov Report

Merging #243 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #243   +/-   ##
=======================================
  Coverage   99.51%   99.51%
=======================================
  Files           7        7
  Lines         415      415
  Branches       42       42
=======================================
  Hits          413      413
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8eef85...3c576ee. Read the comment docs.


fix(auth): Configure Auth0 SDK to use local storage and rotating refr…

CHANGES:

Configure Auth0 SDK to use the new Rotating Refresh Token on the client side.
Use localstorage for cache location instead of memory, to avoid blocking third-party cookies.

Auth0 has built-in reuse detection - if a refresh token is leaked and used, a subsequent exchange
will kick in the detection and the entire “family” of refresh tokens will be invalidated.
(With the Rotating Refresh token approach, it’s okay to store the refresh and access tokens in local storage)

(source: https://auth0.com/docs/tokens/refresh-tokens/refresh-token-rotation/use-refresh-token-rotation#example)

:tada: This PR is included in version 1.8.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


UmlsEntityLinker issue with proxy?

I am using this code for UmlsEntityLinker:
import scispacy
import spacy
#import displacy
from scispacy.abbreviation import AbbreviationDetector
from scispacy.umls_linking import UmlsEntityLinker
import os
os.environ['http_proxy'] = "http://proxy.xxx:5555"
os.environ['https_proxy'] = "https://proxy.xxx:5555"

#
nlp = spacy.load('en_core_sci_sm')
linker = UmlsEntityLinker(resolve_abbreviations=True)

nlp.add_pipe(linker)

However, it seems there is an issue with Proxy even though I am using the same Proxy commands for other tasks without any problem:
Your CPU supports instructions that this binary was not compiled to use: SSE3 SSE4.1 SSE4.2 AVX AVX2
For maximum performance, you can install NMSLIB from sources
pip install --no-binary :all: nmslib
Traceback (most recent call last):
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/connection.py", line 506, in _connect_tls_proxy
    ssl_context=ssl_context,
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 429, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/xx/anaconda3/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/home/xx/anaconda3/lib/python3.7/ssl.py", line 850, in _create
    self.do_handshake()
  File "/home/xx/anaconda3/lib/python3.7/ssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1045)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /ai2-s2-scispacy/data/linkers/2020-10-09/umls/tfidf_vectors_sparse.npz (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1045)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "linker.py", line 12, in <module>
    linker = UmlsEntityLinker(resolve_abbreviations=True)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/scispacy/linking.py", line 85, in __init__
    name=linker_name
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/scispacy/candidate_generation.py", line 223, in __init__
    linker_paths=linker_paths, ef_search=ef_search
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/scispacy/candidate_generation.py", line 133, in load_approximate_nearest_neighbours_index
    cached_path(linker_paths.tfidf_vectors)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/scispacy/file_cache.py", line 36, in cached_path
    return get_from_cache(url_or_filename, cache_dir)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/scispacy/file_cache.py", line 112, in get_from_cache
    response = requests.head(url, allow_redirects=True)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/data/home/xx/scispacy/sci-env/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='s3-us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /ai2-s2-scispacy/data/linkers/2020-10-09/umls/tfidf_vectors_sparse.npz (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1045)')))

downloading the  "/ai2-s2-scispacy/data/linkers/2020-10-09/umls/tfidf_vectors_sparse.npz" using pip and the defined Proxy is done without any issue.

Are you able to run this line https://github.com/allenai/scispacy/blob/2290a80cfe0948e48d8ecfbd60064019d57a6874/scispacy/file_cache.py#L112 on that file with your proxy?
Maybe this answer works? https://stackoverflow.com/a/60352993/10459457

Are you able to run this line https://github.com/allenai/scispacy/blob/2290a80cfe0948e48d8ecfbd60064019d57a6874/scispacy/file_cache.py#L112 on that file with your proxy?
Maybe this answer works? https://stackoverflow.com/a/60352993/10459457

After using fresh environment, the proxy issue gone. However, now I go this issue:
Traceback (most recent call last):   File "linker.py", line 12, in <module>     linker = UmlsEntityLinker(resolve_abbreviations=True)   File "/home/xx/anaconda3/lib/python3.7/site-packages/scispacy/umls_linking.py", line 68, in __init__     self.candidate_generator = candidate_generator or CandidateGenerator()   File "/home/xx/anaconda3/lib/python3.7/site-packages/scispacy/candidate_generation.py", line 129, in __init__     self.umls = umls or UmlsKnowledgeBase()   File "/home/xx/anaconda3/lib/python3.7/site-packages/scispacy/umls_utils.py", line 47, in __init__     raw = json.load(open(cached_path(file_path)))   File "/home/xx/anaconda3/lib/python3.7/json/__init__.py", line 296, in load     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)   File "/home/xx/anaconda3/lib/python3.7/json/__init__.py", line 348, in loads     return _default_decoder.decode(s)   File "/home/xx/anaconda3/lib/python3.7/json/decoder.py", line 337, in decode     obj, end = self.raw_decode(s, idx=_w(s, 0).end())   File "/home/xx/anaconda3/lib/python3.7/json/decoder.py", line 353, in raw_decode     obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 79986649 (char 79986648)

Were you able to resolve this? Could you see if you can load the json file just in a ipython notebook or something?

Unfortunately still the same issue. I can load and work with json files.
I want to share my environment:
Spacy 2.3.5
Scispacy 0.2.4
en_core_sci_sm 0.2.0 also tried 2.5.0 and 0.3.0

with scispacy 0.2.4 you should be using the 0.2.3 models. Could you try that? The way to figure this out is to go to the tag for the release you are using and check the links in the readme (https://github.com/allenai/scispacy/tree/release-v0.2.4#available-models)

and you probably need spacy 2.2.1 for scispacy 0.2.4 as well

(from checking the requirements file on that tag)

Unfortunately still the same issue. I also checked your post in: https://www.kaggle.com/daking/extracting-entities-linked-to-umls-with-scispacy and tried to use packages versions but no success.
Here the full stack with the list of packages/versions I have:
`/data/home/fsa/Prodigy/Linker/linker-env/lib/python3.7/site-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.20.3 when using version 0.22.2. This might lead to breaking code or invalid results. Use at your own risk.
UserWarning)
/data/home/fsa/Prodigy/Linker/linker-env/lib/python3.7/site-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator TfidfVectorizer from version 0.20.3 when using version 0.22.2. This might lead to breaking code or invalid results. Use at your own risk.
UserWarning)
Traceback (most recent call last):
File "linker.py", line 12, in
linker = UmlsEntityLinker(resolve_abbreviations=True)
File "/data/home//Prodigy/Linker/linker-env/lib/python3.7/site-packages/scispacy/umls_linking.py", line 68, in init
self.candidate_generator = candidate_generator or CandidateGenerator()
File "/data/home/fsa/Prodigy/Linker/linker-env/lib/python3.7/site-packages/scispacy/candidate_generation.py", line 129, in init
self.umls = umls or UmlsKnowledgeBase()
File "/data/home//Prodigy/Linker/linker-env/lib/python3.7/site-packages/scispacy/umls_utils.py", line 47, in init
raw = json.load(open(cached_path(file_path)))
File "/home//anaconda3/lib/python3.7/json/init.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, kw)
File "/home//anaconda3/lib/python3.7/json/init.py", line 348, in loads
return _default_decoder.decode(s)
File "/home//anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/**/anaconda3/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 79986649 (char 79986648)

The installed package:
Package            Version

awscli             1.20.5
blis               0.4.1
botocore           1.21.5
catalogue          1.0.0
certifi            2021.5.30
charset-normalizer 2.0.3
colorama           0.4.3
conllu             4.4
cymem              2.0.5
docutils           0.15.2
en-core-sci-sm     0.2.4
idna               3.2
importlib-metadata 4.6.1
jmespath           0.10.0
joblib             1.0.1
murmurhash         1.0.5
nmslib             2.1.1
numpy              1.21.1
pip                21.1.3
plac               0.9.6
preshed            3.0.5
psutil             5.8.0
pyasn1             0.4.8
pybind11           2.6.1
pysbd              0.3.4
python-dateutil    2.8.2
PyYAML             5.4.1
requests           2.26.0
rsa                4.7.2
s3transfer         0.5.0
scikit-learn       0.22.2
scipy              1.7.0
scispacy           0.2.4
setuptools         39.0.1
six                1.16.0
spacy              2.2.1
srsly              1.0.5
thinc              7.1.1
threadpoolctl      2.2.0
tqdm               4.61.2
typing-extensions  <IP_ADDRESS>
urllib3            1.26.6
wasabi             0.8.2
zipp               3.5.0
`
For the sklearn warning, I installed the version:
DeprecationWarning: np.intis a deprecated alias for the builtinint. To silence this warning, use intby itself. Doing this will not modify any behavior and is safe. When replacingnp.int, you may wish to use e.g. np.int64ornp.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.


better handling of zero and negative values on log-scale axes

Present negative data as log(abs(y)) on an inverted scale below the normal log scale graph.   Furthermore, set the yscale minimum to min(abs(y)+dy)/2 to protect against zero counts in the data.  This will be useful for specular and background subtracted data.
For spin asymmetry data a linear scale would be better even if yscale is set to log.  Maybe the plot info could carry a flag indicating that yscale should be set by the data rather than the UI?

Plotting negative data on a log scale plot is likely to be confusing to people (most?) who aren't familiar with the practice.
I'm happy to use the 'ytransform' from the plot options, though - if it's present I'll use it, otherwise things work as they do now.

Piping through the xtransform and ytransform is implemented and used with spin asymmetry - see d05794d


ignore_langs list is not updatable

According README I try to edit ignore_langs, but ignore_langs is not updated:

Using with config section in 'packer` – not work (because reference already on other table);

  use({
    'LostNeophyte/null-ls-embedded',
    config = function()
      require('null-ls-embedded').ignore_langs = {
        ['*'] = { 'comment', 'vim' }, -- don't format `comment` in all languages
        markdown = { 'inline_markdown' }, -- don't format embedded `inline_markdown` in `markdown` files
      }
    end,
  })

Using with setup section in 'packer` – throws error (because plugin not exist yet);

  use({
    'LostNeophyte/null-ls-embedded',
    setup = function()
      require('null-ls-embedded').ignore_langs = {
        ['*'] = { 'comment', 'vim' }, -- don't format `comment` in all languages
        markdown = { 'inline_markdown' }, -- don't format embedded `inline_markdown` in `markdown` files
      }
    end,
  })

Just put code from readme to init.lua - also not work

Please, add setup function for setting (similar as in most other plugins).

it should be require('null-ls-embedded').config.ignore_langs

the readme is wrong

Yes, that work.

I'll add a config function though, I'm doing a bigger refactor


UserProfile: add a new field, create API to store and retrieve information.

Add a new field, create API to store and retrieve information.

Test cases for API 'Edit profile' are covered in this issue.


Added stuffs

Added combobox, buggy color picker but work a bit
Fixed choose team when join a  map
Re-init font on screen size changed
sv_pure bypass stuffs
and I don't remember

nvm I'm new to git pull so ignore this but u can read the code?


[Jenkins Pipeline in K8s cluster] ERROR: Slack notification failed. See Jenkins logs for details.

When running this code
stage('test slack') {
    container('web-build') {
        slackSend (channel: 'web-builds', color: '#00FF00', message: 'hello')
    }
}

I get this error
run slackstepsend, step false, desc false
Slack Send Pipeline step configured values from global config - baseUrl: true, teamDomain: true, token: true, channel: false, color: false
ERROR: Slack notification failed. See Jenkins logs for details.

In the Jenkins logs I see hundreds of lines of an unreadable log dump. How do I go about debugging this?

I think this is the error causing me trouble, looks to be on the k8s side of things.
Cannot list namespaces: Failure executing: GET at: https://<IP_ADDRESS>/api/v1/namespaces.
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. namespaces is forbidden: User "system:serviceaccount:build:cjoc" cannot list namespaces at the cluster scope.

that looks like a kubernetes error?
need logs related to the slack plugin

Turns out the Kubernetes error was irrelevant.
The problem I was having turned out to be that the global configuration was not inherited to the pipeline, so everything needs to be defined explicitly.
This works for me:
WithCredentials([string(credentialsId: 'theSlackToken', variable: 'slackCredentials')]) {
    slackSend (channel: '@johnsmith', message: 'hello there', color: '#3eb991', failOnError: true, teamDomain: 'myteamsubdomain', token: slackCredentials)
}

More explicit error messages would be helpful in the future

global configuration does go to the pipeline
There must be a problem with your global configuration


Add RSMPI_STATUSES_IGNORE

Where did this commit go? o.o

Ah, sorry, I seem to have dropped this unintentionally yesterday, when I pushed the fix for the CI script. Could I ask you to make another PR?

Done: https://github.com/bsteinb/rsmpi/pull/19


--embed-subs causing specific videos to have video length/padding issues

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
United States
Provide a description that is worded well enough to be understood
I had downloaded videos yesterday with YT-DLP and YT-DLG, so I decided to watch them today. However, I found that two episodes of the series I downloaded had video length issues. That is, the videos were showing themselves to be longer than they actually were and that when they reached the REAL cutoff point they froze up or crashed the playback depending on the method I used to play them (MPV and VLC.
The two videos I have found this strange problem with are Shephard's Mind Episode 1 and Shephard's Mind Episode 2.
Thinking it might have been a problem with YT-DLG, as I've sometimes had a bug or two with it in the past, I downloaded the video using the same variables I had been using with YT-DLP alone. However, the downloaded videos had the same problems as the ones downloaded previously. Wondering what was going on, I decided to test the various possibilities by taking out various variables to see what exactly was the issue.
From my testing, I believe that there is some sort of issue with --embed-subs, as every other combination seems to produce videos without any video length issues, while any variations that contain --embed-subs have them. I'm pretty assured that only when YT-DLP tries to EMBED the subtitles do issues occur, as I used --all-subs and they downloaded just fine, leaving the separated video completely fine.
What I'm REALLY confused by is that this appears to ONLY be happening with the aforementioned videos, as the other videos that I downloaded from the same series didn't seem to have this issue. For example, Shephard's Mind Episode 3 didn't have issues when using --embed-subs. I don't have any clue why the aforementioned videos would be any different from the others, maybe they're simply so old YT-DLP is having issues with them?
Attached to this will be my full collection of outputs from YT-DLP in a ZIP, with the specific output from the test where I ONLY used --embed-subs being the one featured below in this issue report.
Sorry in advance if I somehow formatted this incorrectly, but hopefully I at least gave enough information to help point in the right direction of why this is occurring.
Outputs from YT-DLP.zip
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', '--embed-subs', 'https://www.youtube.com/watch?v=GNaJX6gOYMA']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds [9073ae645] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-114630-g591e27d1e7-20240403 (setts), ffprobe N-114630-g591e27d1e7-20240403
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.31.0, sqlite3-3.35.5, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1809 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds)
[youtube] Extracting URL: https://www.youtube.com/watch?v=GNaJX6gOYMA
[youtube] GNaJX6gOYMA: Downloading webpage
[youtube] GNaJX6gOYMA: Downloading ios player API JSON
[youtube] GNaJX6gOYMA: Downloading android player API JSON
[youtube] GNaJX6gOYMA: Downloading m3u8 information
[info] GNaJX6gOYMA: Downloading subtitles: en
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] GNaJX6gOYMA: Downloading 1 format(s): 244+251
[info] Writing video subtitles to: Shephard's Mind Episode 1 [GNaJX6gOYMA].en.vtt
[debug] Invoking http downloader on "https://www.youtube.com/api/timedtext?v=GNaJX6gOYMA&ei=LD8PZvTEHJCO_9EP3caToAg&caps=asr&opi=112496729&xoaf=4&hl=en&ip=<IP_ADDRESS>&ipbits=0&expire=1712300444&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cei%2Ccaps%2Copi%2Cxoaf&signature=62EB45D69BA0D1F0BF179A57CC8C537D6ADEF51C.274AD317F391C72E06AF681F82141D53D53AE13B&key=yt8&lang=en&fmt=vtt"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: Shephard's Mind Episode 1 [GNaJX6gOYMA].en.vtt
[download] 100% of    9.06KiB in 00:00:00 at 130.44KiB/s
[debug] Invoking http downloader on "https://rr4---sn-ab5l6nrl.googlevideo.com/videoplayback?expire=1712296845&ei=LD8PZpyCO9-n_9EP346fuAY&ip=<IP_ADDRESS>&id=o-AM4LuqrSlZwIVr16BjqkrOvMYaVi-NGYNEhPN-t-cin2&itag=244&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=OD&mm=31%2C29&mn=sn-ab5l6nrl%2Csn-ab5sznld&ms=au%2Crdu&mv=m&mvi=4&pl=24&initcwndbps=710000&vprv=1&svpuc=1&mime=video%2Fwebm&gir=yes&clen=33778234&dur=598.466&lmt=1496519193853681&mt=1712274296&fvip=2&keepalive=yes&fexp=51141542&c=IOS&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRQIhAMMqdV-2YSO25JyfdxDqzLCDg3OCiVkueRiEjkI_pP7gAiAIV0RhLN19pUROQek1xBnqZr2_p174afdz4r-Rp8-Bdg%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ALClDIEwRAIgGGXEkPyyRKmTIOST6C8WXAVCsKyL3j02kIum3hOvCdYCIA2h3xI_5lZg3nGmkXWjd90MDDDtJ7PMkR7XueWjkQpR"
[download] Destination: Shephard's Mind Episode 1 [GNaJX6gOYMA].f244.webm
[download] 100% of   32.21MiB in 00:00:21 at 1.47MiB/s
[debug] Invoking http downloader on "https://rr2---sn-ab5sznld.googlevideo.com/videoplayback?expire=1712296845&ei=LT8PZvygBoKF_9EP_ueioA4&ip=<IP_ADDRESS>&id=o-ABEPf9eO3iqOdC1Wre13pbEKeB88WPXTC56jmeDt3K_Q&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=OD&mm=31%2C29&mn=sn-ab5sznld%2Csn-ab5l6nrl&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=728750&spc=UWF9f4D6VFfEy0EeR8ZK6hy_q-VncZX2W__J&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=10076170&dur=598.561&lmt=1496517559027508&mt=1712274296&fvip=4&keepalive=yes&fexp=51141541&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRQIhAIVw2HtnrBMd-Uz4aKaONo60xsju7-BvTBw_QwuWuHp9AiBNyYLBixzhBkZbfdD88nyHxJwUHBSl4WgW1_O37Ps-tA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ALClDIEwRQIhAMfB7NPQg1jGYoxK0hYxTPxn_PAgksqTB1GGbhSE2m61AiAk2J_0-ClZc4B5wN67HNR9giuqbvqNYOWg0PrCItVOvQ%3D%3D"
[download] Destination: Shephard's Mind Episode 1 [GNaJX6gOYMA].f251.webm
[download] 100% of    9.61MiB in 00:00:06 at 1.42MiB/s
[Merger] Merging formats into "Shephard's Mind Episode 1 [GNaJX6gOYMA].webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].f244.webm" -i "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].f251.webm" -c copy -map "0:v:0" -map "1:a:0" -movflags "+faststart" "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].temp.webm"
Deleting original file Shephard's Mind Episode 1 [GNaJX6gOYMA].f244.webm (pass -k to keep)
Deleting original file Shephard's Mind Episode 1 [GNaJX6gOYMA].f251.webm (pass -k to keep)
[EmbedSubtitle] Embedding subtitles in "Shephard's Mind Episode 1 [GNaJX6gOYMA].webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].webm" -i "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].en.vtt" -map 0 -dn -ignore_unknown -c copy -map "-0:s" -map "1:0" "-metadata:s:s:0" "language=eng" "-metadata:s:s:0" "handler_name=English" "-metadata:s:s:0" "title=English" -movflags "+faststart" "file:Shephard's Mind Episode 1 [GNaJX6gOYMA].temp.webm"
Deleting original file Shephard's Mind Episode 1 [GNaJX6gOYMA].en.vtt (pass -k to keep)

Seems related to (or possibly the same issue as) #6918
Things to test:

does downloading mp4 formats (e.g. -S vcodec:h264,res,acodec:aac) instead of webm solve the problem?

does converting the subs from vtt to srt (e.g.  --convert-subs srt) before embedding solve the problem?

Sorry I took so long to report back, go side-tracked by things.
I did your suggestions and the results are:

The MP4 variables seemed to make the subtitles work, even with all the other variables at the same time (besides -f "bv+ba/b" of course).
Converting the subs from vtt to srt didn't seem to do anything as the video length was still far longer and had the same broken aspects as previously described.

I've attached the outputs for you to look at.
Second Set of Outputs.zip

Does -fix_sub_duration help ffmpeg command help? I've been using that in my branch as it fixes a fatal failure to merge when the subs have timing issues.

I don't know if that command is meant to go before or after --embed-subs, so I tested it both ways.
Regardless of it being before or after, it didn't seem to help.
Here are the outputs of the tests.
Third Set of Outputs.zip

I don't know if that command is meant to go before or after --embed-subs, so I tested it both ways.

It's an ffmpeg command not yt-dlp

Ah, sorry. I haven't really used ffmpeg itself ever, just used it with other programs that make use of it like YT-DLP.
How exactly would I go about utilizing this feature?
I tried using it on a download of the video that has the error, but I'm guessing with how nothing occurred besides it giving me a "At least one output file must be specified" error that I'm going in the wrong direction.
FFMPEG Issues.zip

Ah, sorry. I haven't really used ffmpeg itself ever, just used it with other programs that make use of it like YT-DLP. How exactly would I go about utilizing this feature?

I'm not sure about after downloading the video, when I encountered the issue in the past and it left the files unmerged (since yt-dlp considered it fatal) I fixed the sub file itself like this: ffmpeg -fix_sub_duration -i "The Faraway Paladin - S1E1 - The Boy from the City of the Dead - 1.en-US.ass" "test.ass"

Alright, so thanks to your example I was able to use ffmpeg properly this time. However, after doing the fixing and adding them to the video using MKV Toolnix the results were the same broken lengths as before.
As always, here are the outputs, plus the resulting subtitles since they're so small anyway.
Subtitle Results and Outputs.zip

Alright, so thanks to your example I was able to use ffmpeg properly this time. However, after doing the fixing and adding them to the video using MKV Toolnix the results were the same broken lengths as before.

According to the log you sent, you didn't use -fix_sub_duration at all.

You are completely right, this is why I shouldn't do things when I'm sleepy.
After actually doing things properly, the MKV that was created from the video and subtitles through MKVToolnix doesn't have the length and other issues.
Here are the results and outputs (hopefully with no mistakes this time):
Proper Subtitle Fixing Output and Results.zip
After thinking about it, I wonder if this is some issue with trying to embed multiple subtitles at the same time or something. The first two videos mentioned previously have more than one subtitle option, while the the third video has only one subtitle option and is created just fine by YT-DLP. But I don't know if that's really the problem since, in the past, I've downloaded videos with many subtitle options and they didn't mess up the video. In any case, I hope something within all these outputs helps in some manner.

Hmm... Well I don't know if yt-dlp can enable the option by default on all subtitles as this is the negative impact of it: this option will delay the output of all data until the next subtitle packet is decoded which can increase latency
For ass in mp4 it should be fine though since yt-dlp currently has a warning still about that and fatally breaking is a lot worse than length/padding issues I'd say.
As for detecting the issues in your case and only applying it if there is issues, I don't think that's really possible? Though ffmpeg recently added a new -fix_sub_duration_heartbeat in 7.0 which might help with the latency.
For now though it would be very helpful if it was added as a yt-dlp command.

this option will delay the output of all data until the next subtitle packet is decoded which can increase latency

Not relevant for us as I understand. We can add it to EmbedSubtitlesPP if it solves the issue.


error depends with rails 5.1.1

Use bundle
Bundler could not find compatible versions for gem "rails":
  In Gemfile:
    rails (~> 5.1.1)

    kms was resolved to 0.9.0, which depends on
      rails (~> 5.0.0)

Hello.
Thanks for reporting. Yes, at this moment Rails 5.0 supported, but this milestone will include Rails 5.1 support.

@denegny, you could try master branch, dependency was updated