Using global temporary table in copy activity not working

The documentation states that:
> ...in Azure Data Factory, you can create a pipeline with a Copy activity chained with a Stored Procedure activity. The former copies data from your source store into an Azure SQL Database temporary table, for example, ##UpsertTempTable, as the table name in the dataset.
Can you provide more information about how to get the Copy activity to copy data into a global temporary table.
I have tried setting the Sink Table to ##EmployeeStaging. I have also tried adding a pre-copy script to create the temp table, and also set the SqlWriterTableType to a table type that has been defined. However I always get the error  "System.InvalidOperationException,Message=Cannot access destination table...".
If possible can you provide the json source for the CopyToTempTable activity.
Here is what I am using:
{
    "name": "CopyToTempTable",
    "type": "Copy",
    "dependsOn": [],
    "userProperties": [],
    "typeProperties": {
        "source": {
            "type": "SqlServerSource"
        },
        "sink": {
            "type": "AzureSqlSink",
            "SqlWriterTableType": "EmployeeTableType"
        },
        "enableStaging": false
    },
    "inputs": [
        {
            "referenceName": "SourceDatabase",
            "type": "DatasetReference",
            "parameters": {
                "SourceTableName": "Employee"
            }
        }
    ],
    "outputs": [
        {
            "referenceName": "AzureDataWarehouse",
            "type": "DatasetReference",
            "parameters": {
                "SinkTableName": "##EmployeeStaging"
            }
        }
    ]
}

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 09dd1d03-ae28-c5a2-7c64-2f51675f6b09
Version Independent ID: dac74d68-2105-a029-1f49-d444c0a252f4
Content: Copy data to or from Azure SQL Database by using Data Factory
Content Source: articles/data-factory/connector-azure-sql-database.md
Service: data-factory
GitHub Login: @linda33wj
Microsoft Alias: jingwang

@patrickgalbraith Thank you for your query. We will investigate and update this thread further.

@himanshusinha-msft @shashishailaj Any updates guys?

@patrickgalbraith : We need to understand that when we create a global temp table ( its accessible to all the session  ) , but it gets created in the tempdb and not in the regular DB  . So you need to have the required permission on the on the tempdb to read that , but if you try to read the table from a store proc it will work fine .
We are going ahead and closing this issues for now , you can always re-open the issue if you think that's appropiate .

@himanshusinha-msft I am reading it from a stored procedure, and the user I am using is the database admin that Azure created. The issue is with writing the data to the temp table. This is what your docs say:

The former copies data from your source store into an Azure SQL Database temporary table, for example, ##UpsertTempTable, as the table name in the dataset.

This is what I have done with the output of the copy activity:
    "outputs": [
        {
            "referenceName": "AzureDataWarehouse",
            "type": "DatasetReference",
            "parameters": {
                "SinkTableName": "##EmployeeStaging"
            }
        }
    ]

If you have got this working can you provide an example.

When you are trying to copy data to the ## table , you can do the following .
1 . Create the table ( named tabletype below)

2. Create a proc on the sink side . We are passing the TableType as a parameter , The SQL code you can change as per your logic .
CREATE PROCEDURE [dbo].TestProc
@TABLE1 dbo.tableType readonly
AS
BEGIN
MERGE ##Test1Destination AS target
USING @TABLE1 AS source
ON (target.id_new = source.id )
WHEN NOT MATCHED THEN
Insert (id_new) VALUES (source.id);
END
3. Now got to the Sink tab and point to the proc which we created and also the TableType from Step#1 and name of the parameter as named in the proc ( in our case its table1 )

This worked for me , let me know if you have any query .

@himanshusinha-msft What you have done is "Option 2" from https://docs.microsoft.com/en-us/azure/data-factory/connector-sql-server#upsert-data.

You also can choose to invoke a stored procedure within the copy activity. This approach runs each row in the source table instead of using bulk insert as the default approach in the copy activity, which isn't appropriate for large-scale upsert.

I am wanting to do "Option 1". Which should look like this:

@himanshusinha-msft Copying the data to the temp table works. But accessing it from the subsequent stored procedure doesn't work.

Sorry but I really don't get why the global temporary table is immediately dropped after it's created from a stored procedure. Creating User defined table types it's only necessary to support ADF?
Doing everything from sql managment, using the same user as configured  in data factory, the temp table is  created and  filled correctly. in ADF it's dropped as soon as it's created.
Here is the same problem found on stack overflow: https://stackoverflow.com/questions/56119601/how-to-create-temp-tables-in-sql-to-be-used-in-several-adf-activities

@patrickgalbraith : Apologizes for delay , unfortunately it just works fine for me . Adding the screenshot and also add the simple proc which it is calling .
Create Proc SetData
as
Insert into ##Test1Destination(id_new) values (1000)

@himanshusinha-msft No luck on my end. Can you export the code for the pipeline that you are using.

I've been trying to accomplish the same, but have pinned the underlying issue due to scoping. The global temp table, while available to all users, persists only as long as the original session remains open.  (See this link for temp table scoping.) The main points are:

A local temporary table created in a stored procedure is dropped automatically when the stored procedure is finished.
All other local temporary tables are dropped automatically at the end of the current session.

This is why the above recommendations fail because it's split across two separate sessions that ADF invokes. One for the initial INSERT and a second to execute the linked PROC.
To prove the issue, I opened a session in SSMS and created the global temp table there. I then left the session open and ran the ADF activities. It worked without any issue. If I close the session (and moved the create statement into the COPY job), the ADF SPROC activity will fail as it's unable to find the table referenced.
So the work around I created was creating an "Import" table that just persists forever. I know this defeats the purpose of utilizing the temp table, but I cannot find a practical way of persisting the session within ADF between two different activities. Further, this is complicated by other comments I read where session limits have a 2 hour timeout. So 1) you can't just leave a permanent session open that created the temp table and 2) if your downstream processing takes longer, the temp table would get dropped when the session hits the timeout limit.
This is why I created the other ticket, trying to get more information on the recommended approach on handling the creation of the temp table. The major benefit I have for using the temp table is that the initial INSERT into the temp table skips the transaction logging. So I would only have 1 log per record rather than 2 (initial insert into import table and then merge statement).

@patrickgalbraith : I did revisited my pipeline and I was able to repro the issue . I have reached out to the internal team and we will update you once we hear from them .
@jeffjwu : We agree with all the inputs , while reproing the issue we did followed one of the many scenarios which you have mentioned . As mentioned before we have escalated the issue and will update the thread once we hear back from the team.

@himanshusinha-msft Thanks for the update.
@jeffjwu Great overview of the issue. At the very least the documentation needs to be updated since what is being recommended is not possible.

The major benefit I have for using the temp table is that the initial INSERT into the temp table skips the transaction logging. So I would only have 1 log per record rather than 2 (initial insert into import table and then merge statement).

This is exactly why I am trying to do this with temp tables. This becomes important when you have millions of records to insert (something which ADF should handle).

@patrickgalbraith  @jeffjwu  .
The global table is active when the connection is active , once the connection is lost SQL drops the ## table. You can test it against SSMS directly.
“New Query” -> Run “createtemptable” -> Close Query & “New Query”  ->  Run “createtemptable”  It always success in SSMS.
That is by design. . We won’t keep connection between 2 activities.
If you use a real table instead of temporary table. Then you will get the expected result.
The suggestion is don’t used temporary table in ADF if the data need more than 1 activities to access.
@linda33wj : It looks like this document needs some update , please review that and make the updates accordingly .

I have raised this as a suggestion for Azure here https://feedback.azure.com/forums/270578-data-factory/suggestions/38287108-persist-global-temporary-tables-between-activities
For anyone reading this that might want his feature please upvote that suggestion.

@papatrickgalbraith I came across the same problem as you, and while the proposed request to persist the session (and implicitly the global temporary tables) across multiple activities will solve the problem, I think it's rather unlikely to be implemented. A more reasonable request in my opinion is to have a postCopyScript for the copy activity, the same way we have a preCopyScript. This way you could use the preCopyScript to create the temporary table (directly or using a stored procedure), then the activity to insert directly into the temporary table, and the postCopyScript to merge the data (directly or using a stored procedure).

I looked, and there is already a request for that https://feedback.azure.com/forums/270578-data-factory/suggestions/34617748-post-copy-script-in-copy-activity
So please support it!

@eugenpodaru Great point. I have posted a link to the feature request you mentioned on the one I raised.

@linda33wj can u update documentation and provide sample ADF code as well

@himanshusinha-msft can you post sample code here. The solution is not working local/global temp tables.

@himanshusinha-msft It seems like this was identified as incorrect doco in August. Here's a more recent stackoverflow question from someone who is still being misled by it. https://stackoverflow.com/questions/59649463/data-factory-copy-to-temporary-table

This approach will solve this issue: https://github.com/scoriani/azuresqlbulkupsert

Thanks all for the feedback and sorry for the incorrect doc earlier. The section has been updated to clarify the current supported capabilities. We will continue evaluating the need to further smooth it. Please suggest/vote on https://feedback.azure.com/forums/270578-data-factory site.
#please-close


Rename Strategy to a more intuitive name

The term "Strategy" for OpenAI and Llama strategies may not be intuitive or understandable. Consider renaming it to something more descriptive and aligned with its purpose.

@CesarJimenezVilleda02 Just had a thought: how about FunctionGenerationFactory instead? it's still pretty similar but i think factory is clearer than strategy (i.e. which LLM "factory" produced the function)


Folder sharing and search over folder shares

People would be able to share files by sharing entire folders and would be able to search files over peoples shares. Direct Connect protocol has this ability but why not to include it into Matrix?
Example use case: A group of travellers create Matrix room to communicate with each others. During their travel everybody takes a lot of photos. As people want to share their photos they can use Matrix client to share photo folders and everyone download the photos from shared folders.
In advanced versions the clients could make thumbnails from shared photos and users can just click thumbnails of photos they want to download.
Some file standard should be developed to handle file indexes of shared folders. Then search function just searches over these index files. DC++ is popular open source client that already does this. Perhaps some ideas for implementation could be taken from DC++ source code. Why not also make the implementation Direct Connect compatible? Then we could have Matrix-Direct Connet bridges.

This sounds like a client feature. Matrix clients can already send media into rooms - it's up to the client how to represent that.

But shouldn't there be some standard way for clients to do this?

I'm not totally clear on what part needs standardizing here.

If servers would handle searching over people's shares. This would need support from protocol I think.

If clients would make a list of the files that people share, there should be some standard format for such a file. Clients would send this file into Matrix server and server would return search results based on the file lists it has. If the files are changed the clients need to update the list into the server.

That wouldn't be e2ee friendly.

Why not? Files could be transferred straight from user to user. Server just handles searching for the files.

Because leaking filenames is not something e2ee communications do.

This would be based on user's choice of course.

This to me sounds more like representing rooms as file systems :laughing:  . Which I'd be intrigued to see.

There would be option to turn file sharing on or off when you set up your own Matrix server. Besides the conversation log the servers that are also file sharing servers would keep log of the files that people share. If there is a file sharing room with people on many servers, the servers could sync the file lists periodically or search for files could be forwarded between the servers.
If file sharing features are turned off from the server, the servers won't answer the search requests or store logs of people's shares.

I think this should just be done over WebRTC without involving the server as much, its not like the server can do anything if the specific device which has the file is down at the time.

People should be also able to create rooms where they can require everyone to share certain amount of data. There could some kind of attribute: file_sharing_room: (true|false). You can only create this room if the file sharing is turned on for the server.

I think there would be demand for this feature. Besides photo sharing there could be audio and video sharing communities. Audio and video sharing rooms could create list of all the shared files in the room and you could stream audio and video from users.

There should be also standard for hashing to ensure integrity of the files.


rendering properties which have leading or trailing space in markup

Hey,
Nustache doesn't seem to render {{ propertyName }} (space before & after property name) but does render {{propertyName}} (no spaces). Was this a designed behavior? It would be nice if spaces can be present for readibility & also the JS renderer works just fine with spaces.
Thank You,
Vish

Any updates on this issue?
Thank You,
Vish

Sorry about dropping the ball. Added a unit test and fix. Please re-open if you find a problem.


Add Indian holidays to the Holidays Calendar

Add Indian holidays please. I've added just three of them for now.
@skcript/backend

http://www.officeholidays.com/countries/india/tamil_nadu/2016.php

http://www.officeholidays.com/countries/india/2016.php

http://www.timebie.com/calendar/india2016.php


[Packagist] Update Packagist service to use v2 api

Packagist deprecated the original packagist.org/p/username/package endpoint in favor of v2 packagist.org/p2/username/package endpoint. Because of this, new packages aren't being found using v1.
This PR updates the Packagist service to use the new endpoint.

Warnings

:warning:
This PR modified service code for packagist but not its test code. That's okay so long as it's refactoring existing code.

Messages

:book:
:sparkles: Thanks for your contribution to Shields, @GeoffSelby!

  Generated by :no_entry_sign: dangerJS against 09937506ec8550efe673da71a50e6d3207c380cd

Thank you for the PR @GeoffSelby! The changes seem to have broken some of the badges, more details can be found in the CI output (e.g. https://app.circleci.com/pipelines/github/badges/shields/6648/workflows/ba687673-7fbf-4cb5-8579-5a699f2b9972/jobs/143021).
Could you please take a look when you get a chance? Let us know if you have any questions

Thanks, @calebcartwright! Looks like it was the validation schema causing problems. Should be good when these tests wrap up.

@chris48s for some reason I feel like I recall you have some more hands on PHP experience than others (certainly more so than myself) so do you have any background on what we were doing before vs. the changes with the new API?

I'll try and have a proper look over this in the next couple of days and see if I have anything useful to add
Also thanks for picking this up @GeoffSelby :+1:

I did start looking at this one tonight and I've made some notes but I still need to spend a bit more time on it before I can post something useful.

I'm kind of hesitant to bring this one up: I don't want to get too bogged down in the 1% case because I think fixing this for the 99% case (packagist.org) is the priority and that is currently broken or producing unexpected behaviour for some users. That said, this transition is slightly complicated by the fact that the packagist integration allows the user to specify a custom (self-hosted) registry using the ?server= param.
If we completely drop compatibility with the v1 API and move to only allowing the v2 API to be used, if anyone is currently using this with a self-hosted registry that is running an older version of self-hosted packagist, their badges are going to break.
I suspect this use-case is probably super-rare for shields.io. Maybe some self-hosting users use it too (but we don't support setting credentials for private packagist, so maybe not..) Presumably someone somewhere is using this though: https://github.com/badges/shields/pull/3986
I think with one of the other services (sonar? nexus?) we support multiple API versions and we try the most recent first then fall back to the second (but allow you to explicitly specify the version). This situation is slightly complicated by the fact that it seems like the old API is retained but serves stale data so falling back by default might not be a great idea either :/ I think my view on this is we should just optimise to work for packagist.org and other custom registries that maintain feature parity with packagist.org. I think a fallback is going to further complicate an integration which already has a few pain points to serve a use-case which may turn out to be hypothetical in any case.

@chris48s I have added the changes you mentioned regarding the PHP version support badge and license badge.

@GeoffSelby sorry I've not had a chance to get to this one yet. I've been a bit over-committed this week and I've been slightly putting this review off as I know I need to find a decent block of time to read over it properly but I'm not ignoring it. I will find some time for this over the weekend.

I think ideally you would run the MetadataMinifier::expand logic on the response before validating the schema, and then you can restore the required()

i thought about that too but I don’t think it would be possible with the way Shields makes a request since it’s shared with other services.

I suggest we override _requestJson(), like this:
class BasePackagistService extends BaseJsonService {
  async _requestJson({ schema, url, options = {}, errorMessages = {} }) {
    const mergedOptions = {
      ...{ headers: { Accept: 'application/json' } },
      ...options,
    }
    const { buffer } = await this._request({
      url,
      options: mergedOptions,
      errorMessages,
    })
    const compressedJson = this._parseJson(buffer)

    // decompress the response here
    const expandedJson = this.decompressResponse(compressedJson)

    return this.constructor._validateJson(expandedJson, schema)
  }
}

then because we apply the schema after decompressing we can make the things we need .required(). That will allow us to simplify the code for individual services by removing all the manual error handling/validation code and calls to decompressResponse().
On the subject of validation:

Is version guaranteed to exist? It looks like we're not handling the case where it is missing at least. If so, lets make it .required() in the schema(s)
Is it possible for a package on packagist to have zero releases, only dev branches (i.e: ...json returns no objects but ..~dev.json returns some). We could get away with assuming that a call to /p will give us at least one usable version when they were all mixed together, but maybe now we need to handle this case for /p2?

Is it possible for a package on packagist to have zero releases, only dev branches (i.e: ...json returns no objects but ..~dev.json returns some).

It is possible for either .json to be empty as long as no tag exists. For ~dev.json to be empty is very unlikely and would indicate a broken package most likely, I wouldn't really care about handling this gracefully but it is an eventuality.
If there is a package version object present then yes the version field will always be present.

@GeoffSelby have you had a chance to read through @chris48s's latest review comments? It would be nice to get this landed. 😉

Given that there hasn't been any activity for a while, I'm going to close this pull request. Feel free to reopen it if you get a chance to look into it again.
If someone else wants to pick this up, you can simply fetch the commits by running git fetch origin pull/6508/head:pr-6508. Address the pending review comments and open a new pull request with a co-authored trailer included in the commit message to give attribution to the original author.


The QA confidence score by your script

Hi, thanks for the nice work. I have a question about the QA confidence score.
According to your paper, the QA confidence score should be the probability of the
true answer according to the QA model. However, in the predict function of qa_models.py of your source code, the prob is the probability of the *predicted answer. I wonder which definition of QA confidence score should we use?

Hi, it is the probability of the predicted answer.
In the paper we wrote: "Complementary to the F1 score, we propose to also consider the confidence of the QA system for its retrieved answer. This corresponds, for each triplet, to the probability of the true answer according to the QA model."
The formulation of the last sentence is probably confusing, it should rather read "This corresponds, for each triplet, to the probability - according to the QA model - of the predicted answer to be correct."

Thanks!


chore : add a a special hapi-base-version so different versions of hapi-base and hapi-structures can be used (#7519)

cherry-pick over from main

(left the hapi.version upgrade from 4.1.0 -> 4.2.0 here


Misc GH workflow tweaks

Only run when relevant files have changed.
Explicit timeout.
Only run coverage workflow which needs secret on main repo.

AFAICT this PR should work OK. However PRs for workflow often can't be tested properly until they have been merged.

Hm, the problem with this is that those actions are set as required checks, so running them conditionally will make it impossible to merge PRs that don't trigger them. Is there a particular reason you think it's important to only run on PHP file changes?

Only because there is no point in running PHP specific workflow when there are no PHP changes.

Yeah, if they're missing they block the merge.


Update broker image used in tests

ConnectorsTest have been failing several times failing to connect to the remote broker. Hoping a new broker will improve that, and maybe we should also consider to use an on-cluster client and turn of TLS.

why should we consider turning of TLS?

@famartinrh mainly because I was seeing ssl connect errors for the test client and it was a bit hard to know if it was a client, broker or openshift issue.


Missing linker entry for math library in c

I manually added "-lm" at the end of the build command and it worked.  Couldn't work out the best place in the Makefile though!

Probably line 42. Anyway, I expect you to include these files into your own project and use whatever you’re using to build it.

OK np cheers again!


Proxying already secure images doesn't work with integrated Camo.

Proxying an image that is already https doesn't actually work. I have no clue why.

It does parse the url correctly, so I don't think that's the issue.

@yariplus you are using the integrated camo, right? Because it works for me with external camo on minora.io/community/test-https-image

Only happens when you don't provide the fullchain in SSL configs.


Follow redirects is not working properly

Hi,
I have one request that should automatically follow redirects, but Rest Assured is not doing it. Even if I explicit ask to follow redirects, it's not working. Below is my code:
		SessionFilter sessionFilter = new SessionFilter();

		// Create the request body
		HashMap<Object, String> body = new HashMap<Object, String>();
		body.put("session_token", sessionToken);

		// Call the API to generate the user session
		given()
			.header("Content-Type", "application/json")
			.body(body)
			.filter(sessionFilter)
			.redirects().follow(true)
		.when()
			.post(baseURL)
		.then()
			.statusCode(200)
			.log().all();

And here is the response log:
HTTP/1.1 302 Found
Cache-Control: no-cache no-store max-age=0 must-revalidate private s-maxage=0
Content-Security-Policy: frame-ancestors 'none';
Content-Type: text/html; charset=utf-8
Date: Tue, 11 Aug 2020 17:48:46 GMT
Expires: 0
Location: https://ologindev.onelogin.com
P3P: CP="CAO DSP COR CURa ADMa DEVa OUR IND PHY ONL UNI COM NAV INT DEM PRE"
Pragma: no-cache
Set-Cookie: persistent=eyJ0eOBJjkbOUH1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1OTcwODE3MjZ9.lXO1NuGmUFRcoPLD510x8zkiEIa_Jpj90NkNQh9t5fs; path=/; expires=Mon, 10-Aug-2020 17:48:46 GMT; secure; HttpOnly; SameSite=None
Set-Cookie: subdomain=ologindev; domain=.onelogin.com; path=/; expires=Wed, 11-Aug-2021 17:48:46 GMT; secure; HttpOnly; SameSite=None
Set-Cookie: onelogin.com_user=452765bea89465d5165a2d66c389f775c8629; domain=.onelogin.com; path=/; expires=Fri, 11-Sep-2020 17:48:46 GMT; secure; HttpOnly; SameSite=None
Set-Cookie: sub_session_onelogin.com=BAh7EjoUc2FtbF9zZXNzaW9uX2lkIilkNDY4MzUwMC1iZTI4LTAxMzgtNDczZi0wNjJjY2ZhZGQ2YzQ6C2RldmljZTA6EGZyb21fb3BlbmlkMDoQX2NzcmZfdG9rZW4iyfyfbuybfUYY3RlZF9sb2dpbjA6D2NyZWF0ZWRfYXRJdToJVGltZQ1xHR6AczzrwgY6H0BtYXJzaGFsX3dpdGhfdXRjX2NvZXJjaW9uRjoJdXNlcmkEUZ94AzoVc2FtbF9zZXJ2aWNlX2lkc1sAOghMb0EwOg1wYXNzd897jkhKJHkjWxhc3RfcmVxdWVzdEl1OwsNcR0egHI868IGOwxGOhRsb2dpbl9jb21wbGV0ZWRUOg9zZXNzaW9uX2lkIik4MjdkMzAxMy0zODBjLTRkMTUtOWNiOS1mNDQ4N2NkYmJiMDM%3D--1e9f6c937ae77881613e5d27d6d2751d7e4df4d3; path=/; secure; HttpOnly; SameSite=None
Status: 302 Found
Strict-Transport-Security: max-age=63072000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Request-Id: 5F32D9FD-B3E49BE5-F12E-0A090584-01BB-177FD4-1703
X-Xss-Protection: 1; mode=block
Content-Length: 99

You are being     redirected
.

Calling the same API through Postman or browser, I can see all the 5 redirects and last status Status Code 200

Found the issue, it's not in Rest-Assured: "If the 302 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued."

@pehguerra : Hi, I'm facing the same issue for a POST request. Getting 302 as a response through RestAssured but getting 200 OK from Postman. Tried setting the follow redirect to true as well as false but no luck. Can you enlighten me what should I do in this case?

Hi @pehguerra , I'm facing a similar issue for a POST request. Getting 302 as a response through RestAssured but getting 200 OK from Postman. Tried setting the follow redirect to true as well as false but no luck. Can you enlighten me what should I do in this case?


Properly emulate Indirect JMP page boundary bug

If the 16-bit argument of an indirect JMP is located between 2 pages (0x01FF and 0x0200 for example), then the LSB will be read from 0x01FF and the MSB will be read from 0x0100. This is an actual hardware bug in early revisions of the 6502 which happen to be present in the 2A03 used by the NES.
The simple solution is to check for the indirect JMP opcode during parsing and alter the argument.

Resolved as of 93f5f8298e9471356495f8fad3ed773b75dcf722.

Solution was incorrect, fixed in 186c72460de09e37b6ba95ad35826fde3f4c8bcf.


Revert the commits of apis module in 5.0.0-beta

#4485

Coverage decreased (-0.07%) to 47.178% when pulling 170fc7ebf507dad3ec76f26d3f3663ef59912609 on aaron-ai:pr into cd24a244248db32819474328672868a7ff6ee1f2 on apache:5.0.0-beta.


wss web socket creation issue

I got - closed connection -1 , draft org.java_websocket.drafts.Draft_17@7559967 refuses handshake , false - then changed the code as below:
Socket creation coode
webClient = new APICWebClient(new URI(getWebSocketUrl(currentApic.IPAddress, port)), new Draft_17());
webClient.connect();
Constructor:
public APICWebClient(URI serverURI, Draft draft) {
super(serverURI, draft);
SSLContext sslContext = null;
try {
    sslContext = SSLContext.getDefault();
} catch (NoSuchAlgorithmException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}
setWebSocketFactory(new DefaultSSLWebSocketClientFactory(sslContext));
logger.info("Socket object created");

}
Draft_17. I get : closed connection -1 , , true. Any help here. This happens during socket creation.

Seems you're missing quite a lot of code. Take a look at the Example: https://github.com/TooTallNate/Java-WebSocket/blob/master/src/main/example/SSLClientExample.java

I dont have keys with me. I use only user name and password. I have token generated after login via other API. It is possible to establish the connection with token or user/password ?

How do i say don't look for any certificate ?

Need to create sslcontext like below. it skips the certificate. I was successfully able make a connection without certificate
SSLContext sslContext = SSLContext.getInstance("SSL");
// set up a TrustManager that trusts everything
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {
System.out.println("getAcceptedIssuers =============");
return null;
}
        public void checkClientTrusted(X509Certificate[] certs,
                        String authType) {
                System.out.println("checkClientTrusted =============");
        }

        public void checkServerTrusted(X509Certificate[] certs,
                        String authType) {
                System.out.println("checkServerTrusted =============");
        }

} }, new SecureRandom());


other JS libs

Is there a package handling also pubKey/privKey generation and tx encoding/signing for Polkadot?

The keying handles signing/keypairs, the API can contruct any transaction - https://polkadot.js.org/api/start/keyring.html


Garbled filename after running onnx-mlir on a model

OS: Ubuntu 18.04 64-bit
Commit id: f5f336db
Steps to reproduce:
I downloaded a model from https://github.com/onnx/models and ran the follow command:
onnx-mlir --EmitONNXIR model.onnx. Only model.mlir was outputted while model.onnx.mlir was missing.
Secondly, after running onnx-mlir --EmitLLVMIR model.onnx, model.onnx.mlir was nowhere to be found. Instead, a file with a garbled filename showed up. The content of the file was fine though.
It appears to be an issue with the way the concatenation is done here :https://github.com/onnx/onnx-mlir/blob/137ce767e6bf25b0fa09f34c84d36393f8a408a0/src/MainUtils.cpp#L110-L111
If I change this to something like below, the filename corruption goes away.
string tempFilename = filename + extension;
freopen(tempFilename.c_str(), "w", stderr);

Let me know what you think. Thanks.

Ran into the same problem. I tried the suggested fix and it works for me. It is a correct fix.

@kwu91 Will you submit your fix or do you want me to do that for you?

@kwu91 Will you submit your fix or do you want me to do that for you?

Sure I can submit a PR. Thanks


[Sema] [CSDiag] Fixes incorrect diagnostic when using keypath with tuples

This PR fixes an issue where an incorrect diagnostic would be emitted when using key path with tuples.
Resolves SR-9213.

@theblixguy once you fix this small newline nit I mentioned, I'll run the CI for you.

@theblixguy once you fix this small newline nit I mentioned, I'll run the CI for you.

Done! @xedin

@swift-ci please smoke test

@swift-ci please smoke test

@xedin Seems like it can be merged now 🎉

@theblixguy Thank you!


wrong credentials error with cn phone account

[custom_components.sonoff] Login failed! Please check credentials!
sonoff:
username: '+86134'
password: '13453'
scan_interval: 60
grace_period: 600
api_region: 'as'
entity_prefix: True
debug: False

sonoff:
  username: 'your phone number without country code +86'
  password: '13453'
  scan_interval: 60
  grace_period: 600
  api_region: 'as'
  country_code: '86'
  entity_prefix: True
  debug: False

This should work

@Lisheng2016 there is no “country_code” configuration option, stop spreading false fixes
Sent with GitHawk

This might be related to API "as" host down.

@Trace-star you still have the same problem? can you login in the mobile app?

I have the same trouble.


Screen goes black - CUDA_ERROR_UNKNOWN  on TensorFlow-GPU 1.5

Machine specs:
Operating System: Windows 7 64 bits
TensorFlow version: 1.5.0
Python version: 3.6 (Anaconda)
CUDA/cuDNN version: CUDA 9.0, cudNN 7.0.5
GPU model and memory: Quadro K2000 2GB
Note: GPU is also driving the display.
While testing, my demo runs well until the screen goes black for a short instant of time and it crashes with the following error message:
2018-02-27 11:45:23.275524: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructi
ons that this TensorFlow binary was not compiled to use: AVX
2018-02-27 11:45:23.533539: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with pro
perties:
name: Quadro K2000 major: 3 minor: 0 memoryClockRate(GHz): 0.954
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.79GiB
2018-02-27 11:45:23.552540: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow dev
ice (/device:GPU:0) -> (device: 0, name: Quadro K2000, pci bus id: 0000:01:00.0,
 compute capability: 3.0)
[I 11:45:53.864 NotebookApp] Saving file at /Mask_RCNN/demo.ipynb
2018-02-27 11:46:41.003970: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran
 out of memory trying to allocate 2.07GiB. The caller indicates that this is not
 a failure, but may mean that there could be performance gains if more memory is
 available.
2018-02-27 11:46:52.468626: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\stream_executor\cuda\cuda_driver.cc:1080] failed to synchronize the
 stop event: CUDA_ERROR_UNKNOWN
2018-02-27 11:46:52.584632: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\stream_executor\cuda\cuda_timer.cc:54] Internal: error destroying C
UDA event in context<PHONE_NUMBER>8FE360: CUDA_ERROR_UNKNOWN
2018-02-27 11:46:52.585632: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\stream_executor\cuda\cuda_timer.cc:59] Internal: error destroying C
UDA event in context<PHONE_NUMBER>8FE360: CUDA_ERROR_UNKNOWN
2018-02-27 11:46:52.585632: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\stream_executor\cuda\cuda_dnn.cc:2289] failed to set stream for cud
nn handle: CUDNN_STATUS_MAPPING_ERROR
[I 11:47:24.287 NotebookApp] KernelRestarter: restarting kernel (1/5), keep rand
om ports
WARNING:root:kernel 134641e2-3f62-4019-9536-be72010310fa restarted

I tried for several days different ideas from different issues:
Checked carefully the video tutorial from youtube, nvidia installation manual, etc.
Reinstalled cuda, cudnn
Downgrade tf, cuda, cudnn
Could you suggest any method to detect what is wrong and how to solve the problem?
Post-data:
Code works fine on tensorflow without GPU.

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
Bazel version
Exact command to reproduce


bower version is old

Can you update the current version to bower?

i didn't saw that it is available as version 2 alpha. Sorry


KAFKA-3682 ArrayIndexOutOfBoundsException thrown by

SkimpyOffsetMap.get() when full
Limited number of attempts to number of map slots after the internal
positionOf() goes into linear search mode.
Added unit test
Co-developed with @mimaison

Hi may anyone take a look at this fix ?

Hi may anyone take a look at this please? @ijuma ? thanks !

I don't know this code well so maybe @junrao can review this.

Thanks for the patch. LGTM. Just a minor comment.


Filter out NULLS in aggregation function

The queries like COUNT(columnA) should only count rows which have non-null values.
e.g.
A column int0 has 17 rows out of which 6 have nulls
in pinot COUNT(column) will return 17
but it should ideally return 11.

This is supported by filter clauses which will be released in 0.10.0. Making it the default behaviour is another question.

Just saw it. https://github.com/apache/pinot/pull/7916 has these changes. I think that should suffice for now.


Newly added questions break jquery

There are two things that do not happen on newly added questions:

Propagation is not stopped, meaning the collapsible expands after clicking the edit/delete buttons.

Clicking the edit button no longer triggers the modal from for editing an unanswered question.

@h12 is this still an issue?

@Faithsend It would be if we were using Ajax to add new questions. Since we're just re-loading the whole thing on the submission of a new question, it's working fine.


Do not block intentional duplicates

Firefox has built-in ability to create duplicate tabs. I do not know which English term is used for it, but I suppose it’s called “Clone tab”.
With your addon installed this features is blocked in the moment. This behaviour started with one of the latest updates, but because coincidentally both Firefox and your addon had updates I cannot say whether it is caused by the addon or a change in Firefox. (In fact, I tested two other similar addons, and both blocked intentional cloning, too.)
I am not a developer and do not know, how a solution could look like. Being a built-in feature there must be some API or whatever it is called in this case.

there must be some API or whatever it is called in this case.

Sadly there is not AFAIK.
Duplicating a tab via the manual option can currently not destinguished.
And i dont think mozilla will extend the addon API to contain event which allow to monitor user actions.  I've already seen some examples where they intentionally dont implement these features, because this might be used to track/monitor user behaviour.
So for the time being there is nothing i can do about this. Sorry.

As a workaround, i can currently only suggest that you use the toolbar icon to temporary disable the duplication prevention and after cloneing click it again to re-enable it.

I've checked "Tab Deduper" addon, and it has an option of "Keep Explicitly Duplicated Tabs", which does what it says - it will not close the tabs duplicated by you. So might be possible to just ask that addon's dev what method they used.

I'll take a look at what they are doing. Thanks for the hint.

@WhiteSp1rit
"Tab Deduper"'s manual duplication detection seems incomplete/faulty.
It only seems to  work when i manually duplicate the active tab.
If i try to duplicate any inactive tab, it closes the duplicated one.
Going by the description on the option/setting
Tabs created by the “Duplicate Tab" option will not be automatically removed.
this is seems incorrect.
Can you confirm this observation?

@Speravir
I thought of one other workaround and added it to version 1.7.4. (for testing)
The tab/tabstrip context menu now contains a "Force Duplicate Tabs" entry.

@WhiteSp1rit
This new entry should behaves as expected when duplicating unfocused or multiple selected tabs, as the original duplication mechanic does.
If i'll find a better solution, i'll update the addon.

@Speravir

but now you could consider hiding the default menu entry

Addons can not manipulate the context via their api, but firefox has some files you can edit to make such modifications.
I'd suggest asking on https://reddit.com/r/FirefoxCSS and you will very likely get some help on how to hide (and/or rearange) the context menus.
Hope this info helps.


Script don't work with non-latin symbols

For example, https://ru.pinterest.com/minnecom/自宅でアウトドア気分おうちキャンプ-indoor-camping/ doesn't work:
Traceback (most recent call last):
  File "D:\repos\pinterest-downloader\venv\lib\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\Starfer\AppData\Local\Programs\Python\Python310\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Starfer\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\Starfer\AppData\Local\Programs\Python\Python310\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\repos\pinterest-downloader\pinterest-downloader.py", line 1858, in <module>
    run_direct_main()
  File "D:\repos\pinterest-downloader\pinterest-downloader.py", line 1850, in run_direct_main
    return run_library_main(args.path, args.dir, args.thread_max, args.cut
  File "D:\repos\pinterest-downloader\pinterest-downloader.py", line 1719, in run_library_main
    fetch_imgs( board, slash_path[-2], slash_path[-1], None, False
  File "D:\repos\pinterest-downloader\pinterest-downloader.py", line 1367, in fetch_imgs
    data = r.json()
  File "D:\repos\pinterest-downloader\venv\lib\site-packages\requests\models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 Abort.

Process finished with exit code 0

brotli to fix r.json() encoding error in line 1366
Please rerun python3 -m pip install --upgrade -r requirements.txt  to install new dependency brotli.
Thank you so much for your contribution! We appreciate the time and effort you've put into this pull request.


Adding Automata ContextFree

Due to https://github.com/paritytech/substrate/pull/9805

Is this the only one that was added since in between? I think there was more?

The rest have been added. I added them to the previous PR each time I merged from master on the substrate PR. None have been missed.


3d console Sprig: D1 doesn't light up in real life

It used to, but it's currently commented out in the firmware

This is such a small cosmetic issue, I doubt we should allow ourselves the cycles to fix this (i.e. we should prioritize maaaany other things).  Closing this issue as a result.


First Assignment COM 411

OKPANI GOODNESS MGBODICHIMMA
OKPANIGOODNESSMGBODICHIMMA.pdf

Second Assignment
Write a PHP program to output your name, registration number, state of origin, local government area and Age


Remove Google Analytics

Would you be willing to remove this Google Analytics 'functionality' from this theme? There are better alternatives if you must keep tabs on users' activity.
Some reputable alternatives can be seen here: https://switching.software/replace/google-analytics/

But the theme user can simply not use it, why should I remove it completely?

I think you should offer safer alternatives to your users, to protect them from a corporate monopoly that bends over backwards to steal their personal information and freedom.

i second this

I would love to merge a PR if you add one or more alternatives, rather than remove an optional feature.

I can work with that.


New console log warnings and suggestions on flaky tests?

Hey there,
I started seeing new type of warning logs as I generate touch events. However it doesn't seem to affect the functionality. Do you know why this started happening?
2022-07-22 14:38:11.471364-0400 IntegrationTestHost[36563:3464107] [EventDispatcher] Found no UIEvent for backing event of type: 1; contextId: 0x9DE5CC17; Event will not be dispatched
2022-07-22 14:38:11.473324-0400 IntegrationTestHost[36563:3464107] [EventDispatcher] Found no UIEvent for backing event of type: 1; contextId: 0x9DE5CC17; Event will not be dispatched
2022-07-22 14:38:11.491951-0400 IntegrationTestHost[36563:3464107] [EventDispatcher] Found no UIEvent for backing event of type: 1; contextId: 0x9DE5CC17; Event will not be dispatched

Another question I have is how are you guys handling the flakiness for Hammer in testing. Although not very often, my tests which rely on using Hammer shows flakiness in CI build machines. Do you encounter similar problems? Do you have any tips on how to mitigate it?
Thanks in advance @gabriellanata

Yeah we're also seeing those warnings. I searched around a bit to see if anyone had seen something like that but couldn't fine anything. Deeper research would probably require some disassembling which I don't have time for now. They don't seem to affect functionality as you said, so maybe wee can ignore them for now.
Regarding flakiness, we haven't experienced many issues. In most cases they've been related to the specific view being tested and we can improve the test itself to make it more resilient. But honestly, from what I've seen, you're probably doing more complex tests than we are 😅

Thanks for your input 👍


Fixed 'Mixed Content' error from Google Fonts

The stylesheet makes an http:// request to Google Fonts. When serving the site over HTTPS, this causes a "Mixed Content" error:
Mixed Content: The page at 'https://[my site]/' was loaded over HTTPS, but requested
an insecure stylesheet 'http://fonts.googleapis.com/css?family=Lato:300,400,700,300italic'.
This request has been blocked; the content must be served over HTTPS.

While the "Add this code to your website" box on Google Fonts only suggests http://, my fix is also a Google-approved solution.

Thanks.


transform: detect duplicate queries/replies and log only once by indicating the number of repetitions

implemented in pull request https://github.com/dmachard/go-dns-collector/pull/282


Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/

This is a new error message for me, and I checked the GCP status page in case it was transient but don't see any active issues.
 I am getting the following error, but only on some tasks within this job (while others within the same job are launching and succeeding):
Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n")

This strikes me as an odd message, because (1) I'm running these machines without a public IP, and so (2) I'm only using gcr.io Docker instances with them. I don't know why they'd be pointing to registry-1.docker.io.
If this isn't a dsub issue, I can point this to the mailing list. (And if there is a quick way for me to get a flavor for "dsub issue vs not dsub issue", just let me know so I can self-triage.)
Thanks!

Can you double-check the task, perhaps pull up the underlying API operation?
Unless I am misunderstanding the error, that it is trying ["docker" "pull" "bash"] indicates that it is trying to pull the bash image from Dockerhub.
Note that I just saw this same problem for the first time earlier this week for a workflow run by Cromwell under Terra. In that case, it dockerhub was throttling the connections to pull down the ubuntu:latest image.

So, a bit interesting. Here is the full output from gcloud alpha lifesciences operations describe<PHONE_NUMBER>174961621. It strikes me as... odd that it seems to try (and fail) to connect to registry-1.docker.io when all of the images are from gcr.io, not dockerhub (at least the ones that are explicitly mentioned in this output below).

gcloud alpha lifesciences operations describe<PHONE_NUMBER>174961621
done: true
error:
  code: 14
  message: 'Execution failed: generic::unavailable: wrapping host binaries: pulling
    image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]:
    exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/:
    net/http: request canceled while waiting for connection (Client.Timeout exceeded
    while awaiting headers)\n")'
metadata:
  '@type': type.googleapis.com/google.cloud.lifesciences.v2beta.Metadata
  createTime: '2021-03-31T19:13:53.704845Z'
  endTime: '2021-03-31T19:30:38.561799866Z'
  events:
  - description: Worker released
    timestamp: '2021-03-31T19:30:38.561799866Z'
    workerReleased:
      instance: google-pipelines-worker-157f9eda0bbba01e17a66d3fd85b15fb
      zone: us-central1-f
  - description: 'Execution failed: generic::unavailable: wrapping host binaries:
      pulling image: retry budget exhausted (10 attempts): running ["docker" "pull"
      "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)\n")'
    failed:
      cause: 'Execution failed: generic::unavailable: wrapping host binaries: pulling
        image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]:
        exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/:
        net/http: request canceled while waiting for connection (Client.Timeout exceeded
        while awaiting headers)\n")'
      code: UNAVAILABLE
    timestamp: '2021-03-31T19:30:37.949807010Z'
  - description: Worker "google-pipelines-worker-157f9eda0bbba01e17a66d3fd85b15fb"
      assigned in "us-central1-f" on a "custom-4-8192" machine
    timestamp: '2021-03-31T19:14:03.234988572Z'
    workerAssigned:
      instance: google-pipelines-worker-157f9eda0bbba01e17a66d3fd85b15fb
      machineType: custom-4-8192
      zone: us-central1-f
  labels:
    dsub-version: v0-4-4
    job-id: maf-filter--james--210331-151350-57
    job-name: maf-filter
    task-id: '6'
    user-id: james
  pipeline:
    actions:
    - commands:
      - -c
      - |
        set -o errexit
        set -o nounset
        set -o pipefail

        function get_datestamp() {
          date "+%Y-%m-%d %H:%M:%S"
        }

        function log_info() {
          echo "$(get_datestamp) INFO: $@"
        }

        function log_warning() {
          1>&2 echo "$(get_datestamp) WARNING: $@"
        }

        function log_error() {
          1>&2 echo "$(get_datestamp) ERROR: $@"
        }

        function gsutil_cp() {
          local src="${1}"
          local dst="${2}"
          local content_type="${3}"
          local user_project_name="${4}"

          local headers=""
          if [[ -n "${content_type}" ]]; then
            headers="-h Content-Type:${content_type}"
          fi

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
            if gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
          exit 1
        }

        function log_cp() {
          local src="${1}"
          local dst="${2}"
          local tmp="${3}"
          local check_src="${4}"
          local user_project_name="${5}"

          if [[ "${check_src}" == "true" ]] && [[ ! -e "${src}" ]]; then
            return
          fi

          # Copy the log files to a local temporary location so that our "gsutil cp" is never
          # executed on a file that is changing.

          local tmp_path="${tmp}/$(basename ${src})"
          cp "${src}" "${tmp_path}"

          gsutil_cp "${tmp_path}" "${dst}" "text/plain" "${user_project_name}"
        }

        while [[ ! -d /google/logs/action/6 ]]; do
          mkdir -p /tmp/continuous_logging_action
        log_cp /google/logs/action/4/stdout "${STDOUT_PATH}" /tmp/continuous_logging_action "true" "${USER_PROJECT}" &
        STDOUT_PID=$!
        log_cp /google/logs/action/4/stderr "${STDERR_PATH}" /tmp/continuous_logging_action "true" "${USER_PROJECT}" &
        STDERR_PID=$!

        log_cp /google/logs/output "${LOGGING_PATH}" /tmp/continuous_logging_action "false" "${USER_PROJECT}" &
        LOG_PID=$!

        wait "${STDOUT_PID}"
        wait "${STDERR_PID}"
        wait "${LOG_PID}"

          sleep 60s
        done
      containerName: logging
      entrypoint: /bin/bash
      environment:
        LOGGING_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6.log
        STDERR_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6-stderr.log
        STDOUT_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6-stdout.log
        USER_PROJECT: ''
      imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
      runInBackground: true
    - commands:
      - -c
      - |+
        #!/bin/bash

        set -o errexit
        set -o nounset
        set -o pipefail

        function get_datestamp() {
          date "+%Y-%m-%d %H:%M:%S"
        }

        function log_info() {
          echo "$(get_datestamp) INFO: $@"
        }

        function log_warning() {
          1>&2 echo "$(get_datestamp) WARNING: $@"
        }

        function log_error() {
          1>&2 echo "$(get_datestamp) ERROR: $@"
        }

        mkdir -m 777 -p "/mnt/data/script"
        mkdir -m 777 -p "/mnt/data/tmp"
        mkdir -m 777 -p "/mnt/data/workingdir"

        echo "${_SCRIPT_REPR}"     | python -c 'import ast
        import sys

        sys.stdout.write(ast.literal_eval(sys.stdin.read()))
        '     > "/mnt/data/script/command.sh"
        chmod a+x "/mnt/data/script/command.sh"

        for ((i=0; i < DIR_COUNT; i++)); do
          DIR_VAR="DIR_${i}"

          log_info "mkdir -m 777 -p "${!DIR_VAR}""
          mkdir -m 777 -p "${!DIR_VAR}"
        done

      containerName: prepare
      entrypoint: /bin/bash
      environment:
        DIR_0: /mnt/data/input/gs/bucket/path/prevalent_disease_af_or_nonnsr
        DIR_1: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2
        DIR_2: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2
        DIR_3: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2
        DIR_COUNT: '4'
        _META_YAML_REPR: "\"create-time: 2021-03-31 15:13:50.570367-04:00\\ndsub-version:\
          \ v0-4-4\\nenvs:\\n  MAF: 1E-2\\n  PVAL: '1'\\njob-id: maf-filter--james--210331-151350-57\\\
          njob-name: maf-filter\\nlogging: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/\\\
          nscript-name: command.sh\\ntask-ids: 1-6\\ntasks:\\n- inputs:\\n    file:\
          \ gs://bucket/path/prevalent_disease_af_or_nonnsr/bolt.imputed.tsv.gz\\\
          n  logging-path: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6.log\\\
          n  outputs:\\n    count_file: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.n_snps\\\
          n    filtered: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz\\\
          n    filtered_tbi: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz.tbi\\\
          n  task-id: '6'\\nuser-id: james\\n\""
        _SCRIPT_REPR: "\"# Must be run on an image that has bgzip and tabix installed\\\
          n\\n# Screen out erroneous SE and ChiSq values ($12>0 && $15>0)\\n# And\
          \ require that MAF and P be within your specified range\\ngunzip -c ${file}\
          \ | awk -v maf=${MAF} -v pval=${PVAL} -F $'\\\\t' 'NR == 1{print $0}; NR\
          \ > 1 && $12>0 && $15>0 && ($16 <= pval) && (($7 > maf) && (1-$7 > maf))\
          \ {print $0}' > output.file\\n\\n# For X, skip the header and append to\
          \ what we already produced\\n# Disabled for now since not produced by REGENIE\\\
          n# gunzip -c ${file_x} | awk -v maf=${MAF} -v pval=${PVAL} -F $'\\\\t' 'NR\
          \ > 1 && $12>0 && $15>0 && ($16 <= pval) && (($7 > maf) && (1-$7 > maf))\
          \ {print $0}' >> output.file\\n\\n# Count the SNPs\\necho $(( $(wc -l <\
          \ output.file) - 1)) > ${count_file}\\n\\n# BGZip\\nbgzip -c output.file\
          \ > ${filtered}\\n\\n# Tabix. This is BOLT-specific formatting\\ntabix \\\
          \\\\n    -s 2 \\\\\\n    -b 3 \\\\\\n    -e 3 \\\\\\n    -S 1 \\\\\\n  \
          \  ${filtered}\\n\""
      imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
      mounts:
      - disk: datadisk
        path: /mnt/data
    - commands:
      - -c
      - |+
        function get_datestamp() {
          date "+%Y-%m-%d %H:%M:%S"
        }

        function log_info() {
          echo "$(get_datestamp) INFO: $@"
        }

        function log_warning() {
          1>&2 echo "$(get_datestamp) WARNING: $@"
        }

        function log_error() {
          1>&2 echo "$(get_datestamp) ERROR: $@"
        }

        function gsutil_rsync() {
          local src="${1}"
          local dst="${2}"
          local user_project_name="${3}"

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
            if gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
          exit 1
        }

        function gsutil_cp() {
          local src="${1}"
          local dst="${2}"
          local content_type="${3}"
          local user_project_name="${4}"

          local headers=""
          if [[ -n "${content_type}" ]]; then
            headers="-h Content-Type:${content_type}"
          fi

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
            if gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
          exit 1
        }

        function log_cp() {
          local src="${1}"
          local dst="${2}"
          local tmp="${3}"
          local check_src="${4}"
          local user_project_name="${5}"

          if [[ "${check_src}" == "true" ]] && [[ ! -e "${src}" ]]; then
            return
          fi

          # Copy the log files to a local temporary location so that our "gsutil cp" is never
          # executed on a file that is changing.

          local tmp_path="${tmp}/$(basename ${src})"
          cp "${src}" "${tmp_path}"

          gsutil_cp "${tmp_path}" "${dst}" "text/plain" "${user_project_name}"
        }

        set -o errexit
        set -o nounset
        set -o pipefail

        for ((i=0; i < INPUT_COUNT; i++)); do
          INPUT_VAR="INPUT_${i}"
          INPUT_RECURSIVE="INPUT_RECURSIVE_${i}"
          INPUT_SRC="INPUT_SRC_${i}"
          INPUT_DST="INPUT_DST_${i}"

          log_info "Localizing ${!INPUT_VAR}"
          if [[ "${!INPUT_RECURSIVE}" -eq "1" ]]; then
            gsutil_rsync "${!INPUT_SRC}" "${!INPUT_DST}" "${USER_PROJECT}"
          else
            gsutil_cp "${!INPUT_SRC}" "${!INPUT_DST}" "" "${USER_PROJECT}"
          fi
        done

      containerName: localization
      entrypoint: /bin/bash
      environment:
        INPUT_0: file
        INPUT_COUNT: '1'
        INPUT_DST_0: /mnt/data/input/gs/bucket/path/prevalent_disease_af_or_nonnsr/bolt.imputed.tsv.gz
        INPUT_RECURSIVE_0: '0'
        INPUT_SRC_0: gs://bucket/path/prevalent_disease_af_or_nonnsr/bolt.imputed.tsv.gz
        USER_PROJECT: ''
      imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
      mounts:
      - disk: datadisk
        path: /mnt/data
    - commands:
      - bash
      - -c
      - |
        export TMPDIR="/mnt/data/tmp"
        cd /mnt/data/workingdir

        "/mnt/data/script/command.sh"
      containerName: user-command
      entrypoint: /usr/bin/env
      environment:
        MAF: '1E-2'
        PVAL: '1'
        count_file: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.n_snps
        file: /mnt/data/input/gs/bucket/path/prevalent_disease_af_or_nonnsr/bolt.imputed.tsv.gz
        filtered: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz
        filtered_tbi: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz.tbi
      imageUri: gcr.io/ukbb-analyses/ubuntu:2020-12-22
      mounts:
      - disk: datadisk
        path: /mnt/data
    - commands:
      - -c
      - |+
        function get_datestamp() {
          date "+%Y-%m-%d %H:%M:%S"
        }

        function log_info() {
          echo "$(get_datestamp) INFO: $@"
        }

        function log_warning() {
          1>&2 echo "$(get_datestamp) WARNING: $@"
        }

        function log_error() {
          1>&2 echo "$(get_datestamp) ERROR: $@"
        }

        function gsutil_rsync() {
          local src="${1}"
          local dst="${2}"
          local user_project_name="${3}"

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
            if gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${user_project_flag} -mq rsync -r "${src}" "${dst}""
          exit 1
        }

        function gsutil_cp() {
          local src="${1}"
          local dst="${2}"
          local content_type="${3}"
          local user_project_name="${4}"

          local headers=""
          if [[ -n "${content_type}" ]]; then
            headers="-h Content-Type:${content_type}"
          fi

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
            if gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
          exit 1
        }

        function log_cp() {
          local src="${1}"
          local dst="${2}"
          local tmp="${3}"
          local check_src="${4}"
          local user_project_name="${5}"

          if [[ "${check_src}" == "true" ]] && [[ ! -e "${src}" ]]; then
            return
          fi

          # Copy the log files to a local temporary location so that our "gsutil cp" is never
          # executed on a file that is changing.

          local tmp_path="${tmp}/$(basename ${src})"
          cp "${src}" "${tmp_path}"

          gsutil_cp "${tmp_path}" "${dst}" "text/plain" "${user_project_name}"
        }

        set -o errexit
        set -o nounset
        set -o pipefail

        for ((i=0; i < OUTPUT_COUNT; i++)); do
          OUTPUT_VAR="OUTPUT_${i}"
          OUTPUT_RECURSIVE="OUTPUT_RECURSIVE_${i}"
          OUTPUT_SRC="OUTPUT_SRC_${i}"
          OUTPUT_DST="OUTPUT_DST_${i}"

          log_info "Delocalizing ${!OUTPUT_VAR}"
          if [[ "${!OUTPUT_RECURSIVE}" -eq "1" ]]; then
            gsutil_rsync "${!OUTPUT_SRC}" "${!OUTPUT_DST}" "${USER_PROJECT}"
          else
            gsutil_cp "${!OUTPUT_SRC}" "${!OUTPUT_DST}" "" "${USER_PROJECT}"
          fi
        done

      containerName: delocalization
      entrypoint: /bin/bash
      environment:
        OUTPUT_0: filtered
        OUTPUT_1: filtered_tbi
        OUTPUT_2: count_file
        OUTPUT_COUNT: '3'
        OUTPUT_DST_0: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz
        OUTPUT_DST_1: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz.tbi
        OUTPUT_DST_2: gs://bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.n_snps
        OUTPUT_RECURSIVE_0: '0'
        OUTPUT_RECURSIVE_1: '0'
        OUTPUT_RECURSIVE_2: '0'
        OUTPUT_SRC_0: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz
        OUTPUT_SRC_1: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.tsv.gz.tbi
        OUTPUT_SRC_2: /mnt/data/output/gs/bucket/path/maf-filter/p-1-maf-1E-2/outputprevalent_disease_af_or_nonnsr.n_snps
        USER_PROJECT: ''
      imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
      mounts:
      - disk: datadisk
        path: /mnt/data
    - alwaysRun: true
      commands:
      - -c
      - |+
        set -o errexit
        set -o nounset
        set -o pipefail

        function get_datestamp() {
          date "+%Y-%m-%d %H:%M:%S"
        }

        function log_info() {
          echo "$(get_datestamp) INFO: $@"
        }

        function log_warning() {
          1>&2 echo "$(get_datestamp) WARNING: $@"
        }

        function log_error() {
          1>&2 echo "$(get_datestamp) ERROR: $@"
        }

        function gsutil_cp() {
          local src="${1}"
          local dst="${2}"
          local content_type="${3}"
          local user_project_name="${4}"

          local headers=""
          if [[ -n "${content_type}" ]]; then
            headers="-h Content-Type:${content_type}"
          fi

          local user_project_flag=""
          if [[ -n "${user_project_name}" ]]; then
            user_project_flag="-u ${user_project_name}"
          fi

          local attempt
          for ((attempt = 0; attempt < 4; attempt++)); do
            log_info "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
            if gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}"; then
              return
            fi
            if (( attempt < 3 )); then
              log_warning "Sleeping 10s before the next attempt of failed gsutil command"
              log_warning "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
              sleep 10s
            fi
          done

          log_error "gsutil ${headers} ${user_project_flag} -mq cp "${src}" "${dst}""
          exit 1
        }

        function log_cp() {
          local src="${1}"
          local dst="${2}"
          local tmp="${3}"
          local check_src="${4}"
          local user_project_name="${5}"

          if [[ "${check_src}" == "true" ]] && [[ ! -e "${src}" ]]; then
            return
          fi

          # Copy the log files to a local temporary location so that our "gsutil cp" is never
          # executed on a file that is changing.

          local tmp_path="${tmp}/$(basename ${src})"
          cp "${src}" "${tmp_path}"

          gsutil_cp "${tmp_path}" "${dst}" "text/plain" "${user_project_name}"
        }

        mkdir -p /tmp/logging_action
        log_cp /google/logs/action/4/stdout "${STDOUT_PATH}" /tmp/logging_action "true" "${USER_PROJECT}" &
        STDOUT_PID=$!
        log_cp /google/logs/action/4/stderr "${STDERR_PATH}" /tmp/logging_action "true" "${USER_PROJECT}" &
        STDERR_PID=$!

        log_cp /google/logs/output "${LOGGING_PATH}" /tmp/logging_action "false" "${USER_PROJECT}" &
        LOG_PID=$!

        wait "${STDOUT_PID}"
        wait "${STDERR_PID}"
        wait "${LOG_PID}"

      containerName: final_logging
      entrypoint: /bin/bash
      environment:
        LOGGING_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6.log
        STDERR_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6-stderr.log
        STDOUT_PATH: gs://bucket/path/maf-filter/p-1-maf-1E-2/dsub-logs/maf-filter--james--210331-151350-57.6-stdout.log
        USER_PROJECT: ''
      imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
    resources:
      regions:
      - us-central1
      - us-east1
      - us-west1
      virtualMachine:
        bootDiskSizeGb: 10
        bootImage: projects/cloud-lifesciences/global/images/w20210322-0300-rc0060586ad3-0000-2431-9437-089e0828bff8stable
        disks:
        - name: datadisk
          sizeGb: 50
          type: pd-ssd
        labels:
          dsub-version: v0-4-4
          goog-pipelines-worker: 'true'
          job-id: maf-filter--james--210331-151350-57
          job-name: maf-filter
          task-id: '6'
          user-id: james
        machineType: custom-4-8192
        network:
          usePrivateAddress: true
        nvidiaDriverVersion: 450.51.06
        serviceAccount:
          email: default
          scopes:
          - https://www.googleapis.com/auth/bigquery
          - https://www.googleapis.com/auth/compute
          - https://www.googleapis.com/auth/devstorage.full_control
          - https://www.googleapis.com/auth/genomics
          - https://www.googleapis.com/auth/logging.write
          - https://www.googleapis.com/auth/monitoring.write
          - https://www.googleapis.com/auth/cloud-platform
        volumes:
        - persistentDisk:
            sizeGb: 50
            type: pd-ssd
          volume: datadisk
    timeout: 604800s
  startTime: '2021-03-31T19:14:03.234988572Z'
name: projects/783282864357/locations/us-central1/operations/4679186047174961621

Hi James,
I agree. The error message does not make sense in the context of that operation.
All actions use a gcr.io imageUri, either:

imageUri: gcr.io/google.com/cloudsdktool/cloud-sdk:294.0.0-slim
imageUri: gcr.io/ukbb-analyses/ubuntu:2020-12-22

And yet it would seem that the Pipelines API is trying to pull the bash docker image for dockerhub.
I have filed a bug with the HCLS team.

Thanks!

I heard back from the Pipelines team that they are aware of the problem and are looking to remove the dependency.
No specific timelines.
Do you have a sense of frequency in which you hit this?

Hard to say though it seems to be becoming more frequent. For example, I'm hitting it now (well, not waiting for all the retries - but noting that machines seem to be stuck in state VM starting (awaiting worker checkin) while trying to run a 'hello world' program on a 4-core machine). VM starting (awaiting worker checkin) seems to be the state that precedes the retry failure.

For what it's worth, I am not seeing a difference between google-v2 and google-cls-v2 providers. I also don't see any errors on the GCP status page to indicate that there is some global problem. I've tried E2 and N1 machine types with similar results. Currently I don't seem to be able to launch jobs unless I remove the --use-private-address flag, but a couple of days ago I was eventually able to. These aren't systematic observations, just anecdote, but sending them along in case they make something 'click' for someone else.

I am also consistently experiencing this issue running jobs with internal IPs, pulling from gcr.io. I just launched a batch of 20 jobs and all failed with this same error. Thanks to @carbocation for raising the issue.

Latest message from the pipelines team is that they've heard from a few people the very same issue, and are working to have it fixed as soon as possible.

From Pipelines team: A fix is in the process of rollout. us-central1 and europe-west2 are fixed, while rollouts to remaining locations are in progress.

We were hitting this issue thru cromwell in us-central1 and can confirm we haven't seen it since Google told us there was a fix mid-Friday.

Going to close this as the Google team has fixed the issue.


ztunnel log printing order problem

Bug Description
When using ambient's version 2022.9, I found that ztunnel's logs are printed in this order:
outbound tunnel -> outbound capture listener -> capture outbound (no waypoint proxy)

But in reality, the order of execution of the listener should be like this:
outbound capture listener -> capture outbound (no waypoint proxy) -> outbound tunnel
It looks like there is a problem.
Version
2022.9 version ambient

Additional Information
No response

See https://preliminary.istio.io/latest/blog/2023/rust-based-ztunnel/, Ztunnel is now implemented based on Rust. Please try the new version to see if you have more questions.

You can grab a 1.18.- build from our daily builds or you can build directly from master branch in the istio/istio repo


New pandas.DataFrame.attrs feature possibly useful for metadata such as root or label

When reviewing #232, I initially got tripped up and did not notice the difference in the similar kwargs label and labels, and got worried that the same kwarg was being passed to MCMCSamples or NestedSamples. However, as it turns out they are different, one being the (tex) labels for the columns, the other being the label/name of the entire dataframe.
I have come across a potentially interesting stackoverflow answer mentioning the pandas.DataFrame.attrs dictionary attribute (new as of pandas 1.5.0, note that it is flagged as "experimental"). This seems like the right place to put the remainders of our "metadata" to ensure the information persists even when copying/pickling/dropping/etc. What do you think?
@williamjameshandley, do you know whether/how good anesthetic currently manages to retain the label attribute when copying/pickling/dropping/etc.?

I hope this is relevant to the discussion: something I noticed when working on #235 was that the $\LaTeX$ labels are lost when Samples are sliced to a WeightedLabelledSeries, e.g. (noting the number of square brackets)
import anesthetic as ac
ns = anesthetic.read_chains("anesthetic/tests/example_data/pc")

ns["x0"].get_labels()
# array([0.00000000e+000, 1.08840734e-308, 4.83001654e-301...

ns[["x0"]].get_labels()
# array(['$x_0$'], dtype=object)

perhaps pandas.Series.attrs could store the $\LaTeX$ label?

Regarding the TEX labels, these are not so much metadata of the entire dataframe, but, more precisely, each is connected to one corresponding column. As such I really like the current implementation. It also renders beautifully in jupyter notebooks!!!
To get the Tex label of one specific column, this would be the suggested way:
ns.get_label('x0')

That said, it would be nice to have the sliced Samples retain the TEX label.
Does this affect 1D plots?
Is this a bug @williamjameshandley?
We could consider using the pandas.Series attribute name to take on the TEX label rather than the column name, since the column name is only needed in dataframes for indexing, but becomes obsolete in a series.

...
Does this affect 1D plots?
...

I believe so, e.g. calling ns.x0.plot.hist_1d(), I don't think $x_0$ is accessible at the level of HistPlot._make_plot(), because by that point self.data is the just a series. Hopefully that makes sense.

Related to #253

Also related to #303


Expose more stuff and add cabal file

Not having a .cabal file checked in breaks the source-repository-package feature of cabal.project files.

Create a separate PR for each topic please.

For the Map instance, I'm fine with it but I believe it requires a major version bump since it may cause breakage, so I'd rather combine that with another breaking change.
The Cabal file thing I don't totally understand since I always use Stack -- is this something that happens when you're using pure Cabal and you want to pull a package from a remote repo?
I'm not ready to expose those constructors until we finish our conversation on issue #7.

* The Cabal file thing I don't totally understand since I always use Stack -- is this something that happens when you're using pure Cabal and you want to pull a package from a remote repo?

Yes. It shouldn't be assumed that users use stack.


[OP] add BilinearSamplingOp and GridGeneratorOp

BilinearSamplingOp
BilinearSamplingOp has two input : data and grid. It apply bilinear sampling to data, according to grid.
Mathematical detail
output[batch, channel, y_dst, x_dst] = G(data[batch, channel, y_src, x_src)
x_dst, y_dst enumerate all spatial locations in output
x_src = grid[batch, 0, y_dst, x_dst]
y_src = grid[batch, 1, y_dst, x_dst]
G() denotes the bilinear interpolation kernel.

Users can use GridGeneratorOp to generate grid.
Users can also design CustomOp to manipulate grid. I guess the CustomOp would not be time-comsuming, since it only involves 2D grid calculation.

GridGeneratorOp
It support affine and warp.

If transform_type is affine, input data should be affine matrix :  (batch, 6)
If transform_type is warp, input data should be optical flow: (batch, 2, Y, X)

We may also need to emphasize that the grid needs be normalized to [-1, 1] before feeding into the BilinearSamplingOp

@sxjscience Is this the same to the previous warpop?

how about we just call it GridSpatialTranformer which is more similar to cudnn naming.

Since you have cudnn version could you add a consistency test in tests/python/gpu/test_operators_gpu.yp

@piiswrong I think we can use "BilinearSampler" to make it consistent with Torch and MatConvNet https://github.com/qassemoquab/stnbhwd, https://github.com/vlfeat/matconvnet/blob/master/matlab/%2Bdagnn/BilinearSampler.m

Looks good. Could you also add a consistency test in tests/python/gpu/test_operator_gpu.py?


Treap Algorithm

Is your feature request related to a problem? Please describe.
The Treap algorithm, also known as the Randomized Binary Search Tree, combines properties of both binary search trees and heaps. It maintains the binary search tree property based on the keys of the nodes and the heap property based on the priorities assigned to the nodes.
Describe the solution you'd like
1.) Define a node structure with key and priority attributes.
2.) Implement methods for insertion, deletion, and rotation operations.
3.) For insertion, recursively find the appropriate position based on key values and priorities.
4.) Rotate nodes as necessary to maintain the heap property during insertion.
5.) For deletion, recursively find the node to be deleted based on key values.
6) Rotate nodes as necessary to maintain both the binary search tree property and the heap property during deletion.
7.) Perform in-order traversal to process the nodes in sorted order.
8.) Create an instance of the treap and use the methods to perform operations.
9.) Utilize the treap for efficient search, insert, delete, and traversal operations with expected logarithmic time complexity
Additional context

@Kumar-laxmi please assign this to me under SSOC'23. I'm ready to contribute in all 4 languages

Assigned!
@Devchawla2608 : C, C++, Python and Java


Center alignment broken in lists

Not sure if this a bug or actually an expected behavior. It seems the width of a numbered list is not always 100% but instead restricted to the maximum width of its lines.
Therefore, if you don't have a long (>=100%) line in a numbered list, your math blocks in that list will not be placed at the horizontal center of the page. These math blocks are instead centered relative to the the parent numbered list. This is some what conterintuitive (at least from a previous LaTeX user's perspective).
With a long line:

Without a long line:

To reproduce:
= Question 1

+ #set enum(numbering: "(a)")
  + The constructed sequent is given as follows:
    $ q arrow p, r arrow not p, q and r tack.long ((p and q) arrow not r) or ((p and r) arrow not q) $
    //The constructed sequent has assumptions $X={A_1,A_2,A_3}$ and a conclusion $C$ where:
    Without the long line:
    $ A_1 &= q arrow p\
      A_2 &= r arrow not p\
      A_3 &= q and r\
      C space &= ((p and q) arrow not r) or ((p and r) arrow not q)
    $

A quick workaround would be finding some line and wrap it with #box(width: 100%).

For anyone else who is encountering this issue. I added the following to my template to resolve this issue: show enum: a => box(a, width: 100%)

For anyone else who is encountering this issue. I added the following to my template to resolve this issue:
show enum: a => box(a, width: 100%)

For anyone who has long lists and has content that crosses page boundaries, use block instead of box.

Wrapping the list with box(width: 100%, ...) didn't work for me, I had to wrap the equations in a box and set alignment to center:
#show math.equation.where(block: true): e => [
    #box(width: 100%, inset: 0em, [
        #set align(center)
        #e
    ])
]

Should we make enum and list's width 100% by default?

Then you can't have a list in an automatically sized block anymore.

Wrapping the list with box(width: 100%, ...) didn't work for me, I had to wrap the equations in a box and set alignment to center:
#show math.equation.where(block: true): e => [
    #box(width: 100%, inset: 0em, [
        #set align(center)
        #e
    ])
]

I find that this will make the following content a new paragraph, maybe use #block to replace #box is better.

This issue was mentioned in https://forum.typst.app/t/how-to-make-nested-bulleted-lists-where-content-includes-centered-equations/846/3.
Wouldn't this be enough?
#show math.equation.where(block: true): block.with(width: 100%)

Strange that changing the width of the existing block element doesn't change anything:
#show math.equation.where(block: true): set block(width: 21cm)

https://github.com/typst/typst/blob/022f34c43a2fb3084b93500163a601105ab582a4/crates/typst/src/math/equation.rs#L163-L175

That would break lists in automatically sized containers.

That would break lists in automatically sized containers.

Are you referring to the top snippet or the bottom one?

The top one. Sorry I was on my phone and too lazy to quote.

That would break lists in automatically sized containers.

Can we use width as a parameter of list to meet the both?

I think that wouldn't be a great UX as you couldn't easily adjust the width parameter from list syntax.
The issue can be fixed just with changes to the internals of the layout engine.

probably add a "scope" parameter would be fine


Engine failing to roll to Framework on dart fix test errors

Started following the Dart -> Engine roll here https://github.com/flutter/engine/pull/42657
https://ci.chromium.org/ui/p/flutter/builders/try/Linux framework_tests_slow/12784/overview
Failed when applying fixes to /b/s/w/ir/x/w/flutter/packages/flutter_test/test_fixes/flutter_test/binding/live_test_widgets_flutter_binding.dart
Expected:
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'package:flutter_test/flutter_test.dart';

void main() {
  final binding = LiveTestWidgetsFlutterBinding.ensureInitialized();
  binding.runTest(
    () async { },
    () { },
  );
}

Actual (original code was unchanged):
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'package:flutter_test/flutter_test.dart';

void main() {
  final binding = LiveTestWidgetsFlutterBinding.ensureInitialized();
  binding.runTest(
    () async { },
    () { },
    // Changes made in https://github.com/flutter/flutter/pull/89952
    timeout: Duration(minutes: 30),
  );
}

@eyebrowsoffire noticed the commit https://dart.googlesource.com/sdk.git/+/fb0b1393583544867703f124766d5dc300e4bffa, which by eyeball seems to be a likely culprit. cc @bwilkerson @scheglov

cc @DanTup

I'll try and repro locally, but I think this might be coincidentally fixed in the change I just pushed to https://dart-review.googlesource.com/c/sdk/+/308080.
In the bad commit above, I added an optimisation to skip some work if there are not any fixes. I did this by calling an existing function hasFixes(), but that's not checking DataDriven fixes (it checks FixProcessor.nonLintMultiProducerMap but we also have BulkFixProcessor.nonLintMultiProducerMap).
This means data driven fixes would work only if the diagnostic that triggers them happened to also have a different (non-data-driven) correction producer. I noticed this because I applied some additional optimisations and it broke some data-driven tests for the same reason.

I've confirmed the fix included in https://dart-review.googlesource.com/c/sdk/+/308080 will fix this:
Current dart-lang/sdk master:
PS C:\Dev\Google\flutter\packages\flutter_test\test_fixes\flutter_test\binding>  & 'C:\Dev\Dart SDKs\Nightly\bin\dart' C:\Dev\Google\dart-sdk\sdk\pkg\dartdev\bin\dartdev.dart fix .\live_test_widgets_flutter_binding.dart --dry-run
Computing fixes in live_test_widgets_flutter_binding.dart (dry run)...
Nothing to fix!

With https://dart-review.googlesource.com/c/sdk/+/308080
PS C:\Dev\Google\flutter\packages\flutter_test\test_fixes\flutter_test\binding>  & 'C:\Dev\Dart SDKs\Nightly\bin\dart' C:\Dev\Google\dart-sdk\sdk\pkg\dartdev\bin\dartdev.dart fix .\live_test_widgets_flutter_binding.dart --dry-run
Computing fixes in live_test_widgets_flutter_binding.dart (dry run)...

1 proposed fix in 1 file.

live_test_widgets_flutter_binding.dart
  deprecated_member_use_from_same_package • 1 fix

To fix an individual diagnostic, run one of:
  dart fix --apply --code=deprecated_member_use_from_same_package .\live_test_widgets_flutter_binding.dart

To fix all diagnostics, run:
  dart fix --apply .\live_test_widgets_flutter_binding.dart
PS C:\Dev\Google\flutter\packages\flutter_test\test_fixes\flutter_test\binding>

The fix has landed in https://github.com/dart-lang/sdk/commit/f1e3bfcef54a5880b1e489855bf601dfe8bee9a3

We can try to roll in the fix in https://github.com/dart-lang/sdk/commit/f1e3bfcef54a5880b1e489855bf601dfe8bee9a3 after the Dart SDK is rolled incrementally following the fix to the tool test discussed in https://github.com/flutter/flutter/issues/128546.

I think this fix landed in flutter/engine in https://github.com/flutter/engine/pull/42716 and is being rolled here in https://github.com/flutter/flutter/pull/128627 (that PR is all green except for the current build status, so I think that confirms the fix solved the issue).
(So I think this can be closed once https://github.com/flutter/flutter/pull/128627 is merged).

It's actually this engine -> framework roll that contains the fix: https://github.com/flutter/flutter/pull/128643
If that lands succesfully, we can close this issue.

Oops, yeah - not sure what I looked at there, it clearly wasn't in the one I linked.
Thanks!

Looks like the roll was a success! I think we can close this as fixed.


feat: backport supports cts and mts config (#8729)

PR for v3: #8729

Description

Additional context

What is the purpose of this pull request?

[ ] Bug fix
[x] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the Commit Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[x] Ideally, include relevant tests that fail without this PR but pass with it.

Thanks for backporting the PR. We normally don't merge back feat PRs. Maybe we could wait a bit to see how it works in v3 before merging it. We are trying to keep v2 as stable as possible.


Support API: adapters for fetching problem reports, orgs with problem reports

These are endpoints added in alphagov/support-api#32

closing this for now because there aren't any clients consuming this information and the API may be changing soon.


ToolWindowFactory: NPE when opening a project

Describe the bug
Opening a new project results in a NullPointerException:
java.lang.NullPointerException: Cannot invoke "com.intellij.ui.content.Content.getComponent()" because the return value of "com.intellij.ui.content.ContentManager.findContent(String)" is null
	at com.amos.pitmutationmate.pitmutationmate.ui.ToolWindowFactory$Util.updateReport(ToolWindowFactory.kt:62)
	at com.amos.pitmutationmate.pitmutationmate.activity.LaunchActivity.execute$lambda$2$lambda$1(LaunchActivity.kt:34)
	at com.intellij.openapi.application.TransactionGuardImpl.runWithWritingAllowed(TransactionGuardImpl.java:209)
	at com.intellij.openapi.application.TransactionGuardImpl.access$100(TransactionGuardImpl.java:22)
	at com.intellij.openapi.application.TransactionGuardImpl$1.run(TransactionGuardImpl.java:191)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:843)
	at com.intellij.openapi.application.impl.ApplicationImpl$4.run(ApplicationImpl.java:463)
	at com.intellij.openapi.application.impl.RwLockHolder.runWithEnabledImplicitRead(RwLockHolder.kt:75)
	at com.intellij.openapi.application.impl.RwLockHolder.runWithImplicitRead(RwLockHolder.kt:67)
	at com.intellij.openapi.application.impl.ApplicationImpl.runWithImplicitRead(ApplicationImpl.java:1430)
	at com.intellij.openapi.application.impl.FlushQueue.doRun(FlushQueue.java:82)
	at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:124)
	at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:44)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:792)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:733)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:761)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:695)
	at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$12(IdeEventQueue.kt:589)
	at com.intellij.openapi.application.impl.RwLockHolder.runWithoutImplicitRead(RwLockHolder.kt:44)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:589)
	at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:72)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:355)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:354)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:793)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:354)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:349)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:1014)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:106)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:1014)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(IdeEventQueue.kt:349)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:848)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:391)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

To Reproduce
Steps to reproduce the behavior:

Open a project

Desktop (please complete the following information):

OS: Linux
IntelliJ IDEA 2023.3.2 (Community Edition)
Build #IC-233.13135.103, built on December 22, 2023

I also noticed this issue. I could only reproduce it once. Does it occur more frequently for you?
It showed me that the NPE occurs in this line
for (file in this.reportDirectory.listFiles()!!) {
Maybe the reportDirectory is null? So we have to add a check before that?

A simple solution for the issue you mentioned would be:
val reportWindowContent = toolWindow.contentManager.findContent(PiTestReports.TITLE) ?: return
val reportWindow = reportWindowContent.component

But I noticed a similiar issue when opening a project in a new window. Unfortunatley I could only reproduce it once.
It showed me that the NPE occurs in this line of the RunArchiver for (file in this.reportDirectory.listFiles()!!) {
Maybe the reportDirectory is null? So we have to add a check before that?

The RunArchiver exception is - as far as i can see - not related to this, maybe create a separate issue.

A simple solution for the issue you mentioned would be:
val reportWindowContent = toolWindow.contentManager.findContent(PiTestReports.TITLE) ?: return
val reportWindow = reportWindowContent.component

I think it is caused by the launch activity launching "too early". Having it run "later" would in my opinion be a better fix and would also solve some other inconsistencies with the plugins-error-message

Concerning the RunArchiver excpetion you are right it was not related to that. I noticed that it appears whenver the pitest run ended in an excpetion because then no files are created so this.reportDirectory.listFiles() is null. I fixed that in #211.
About the launch activity I agree with you but I thought we are doing that through this line ApplicationManager.getApplication().invokeLater

I thought we are doing that through this line ApplicationManager.getApplication().invokeLater

As i understand it invokeLater just makes sure it runs on the AWT thread.
The issue i think is this:

When the user clicks on the tool window button, the createToolWindowContent() method of the factory class is called, and initializes the UI of the tool window. This procedure ensures that unused tool windows don't cause any overhead in startup time or memory usage: if a user does not interact with the tool window, no plugin code will be loaded or executed.
Source

So the tool window has never been created, yet we invoke the update method on it...

If you have time today we can also discuss it via discord.

Closing this as it has been resolved in #216


Creation of git tags

If a git tag were to be created, chessboard.js would be supported by cdnjs, which would be great for the library. (See here: https://github.com/cdnjs/cdnjs/issues/12943#issuecomment-431529802 )
Edit: I feel like I should be elaborating on why git tags are nice. For one, by being on cdnjs, (which requires git tags) chessboard.js gets automatically hosted, minified, and cache-configured on Cloudflare's edge network, which is sweet. Besides that, though, it lets you add npm hooks, which might help with deployment for some users.

What tag(s) are needed for this?

I think the version tags.

Yes. If you set up tags properly, and have an interest in getting chessboard.js on npm, you could follow these steps to add both set up tags, and do everything needed to setup autoupdate hooks for an npm package.

I tagged Commit #4e21c9bbfb as "v0.3.0"
Will be sure to do this in the future.

FYI - I just published v1.0.0 on npm and added links to the unpkg CDN urls on the website